I always accept, and almost always pick it up, character concept be damned. From what I read on the Wiki page for it, there's only one that gives you a debuff, The Fool:
This fortune stands for Thoughtlessness, Folly and Frivolity. Your To-hit has been slightly reduced, however your Movement rate and Recharge rate have improved.
Note: Although The Fool debuffs your tohit, the effect on your hit chance is minor [-3% to-hit]
Still, even that one also gives you +100% Movement Rate, +15% Recharge
I doubt they ever really affect things too much, but I have seen teams throw them around like lucky rabbits feet after a team wipe, ha ha.