Thanks for all the responses so far. I understand it is complicated, but maybe it could be an idea to explore, if the devs were up to it. A PVP server could have its own "tweaks" to skills and stuff, as if it was another different "world".
Also no one would be forced to join the server, so PvE people would be intact!