As has been said several times, the game is as difficult as you can make it. I play when I can play, sometimes that's maybe an hour max and sometimes I get to clock real long time. Either way, when I play I want enjoyment, first then challenge. If I am not enjoying the mission, the content or simply that moment, I step back. Grinding +4 enemies solo takes forever and is like boring, challenge or not. Yet even on lowest difficulty settings, some characters I have are challenged yet I am finding enjoyment.
And that is what I think is the miracle of this game. The very fact that so many different people with different play styles and power arrangements and yada yada, can play the same content and have a different experience than the person next to them is amazing. The game should not be altered at a fundamental level to be harder or more challenging but if you can broaden the ability to customize difficulty, that is awesome. The devs seem to get this, regardless of minor quibbles with X or Y.