With all other things being the same in my build (77.5% recharge bonus with the sets I have right now), hasten vs no hasten is a two second difference in recharge in the majority of my attacks. Foot Stomp is up in 6s vs 8s, Fireball is up in 8s vs 10s, KO Blow is up in 8s vs 10s. Not a big difference at first glance, but it adds up. With hasten, I can hit three Foot Stomps and two fireballs in 18 seconds vs 24 seconds if I wasn't using hasten.
The bigger benefit is long recharge powers like Dull Pain (+Heal +Max HP), which, with hasten, is up within two seconds of it running out, vs being down for a full 40 seconds without hasten running (again, with my current sets.) . Also there's Rage, that gives me a 2 minute +80% damage buff. With hasten it recharges in 1:10 vs 1:28 without, which means with hasten up, I can have it double stacked for +160% damage bonus for a full 50 seconds, vs 32 without hasten. Of course Rage does have a crash and you deal almost no damage for 10 seconds when it wears off (except you can use Judgement Incarnate and prestige powers like Sands of Mu/blackwand/nemesis staff that are not affected by the crash, which is what I do to keep my damage output going).
It's a great power to have to both increase your damage/healing output, and generally speed things up, which is why so many people use it. A lot of MMOs tend to have downtime between mobs, waiting for things to recharge. Reducing that, in my opinion, is a good thing. However, everybody plays differently, and I can understand the frustration that some people can feel when the rest of the team speeds off ahead of them and they're always having to play catch up (one of the main reasons my wife doesn't want to play).
At the end of the day, it is your character, and you can play it absolutely however you like. I like it though.