Let's see if I can explain this without making it more confusing...
If you solo a mob and it gives 100 XP for defeating it, then you get 100 XP.
If you duo the mob, but are not teamed, you get a proportional account of XP to the damage that you do, so, do 50% damage, get 50 XP
If you team, you get a proportional amount of XP to what the team does in damage divided equally among the team, plus a modifier based on the team size. So, if you're in a 2 man team, you'd get 50 XP plus whatever the modifier for a 2 man team is.