What's with the lag every time someone joins?

On the PS3, at least, it's very rare to get a full lobby right from the beginning, and the game will generally start with anywhere from 2 or 3 empty slots to 8 empty slots.  In a game like Untamed, it wouldn't matter, but in a game where you control both the rider and the bike, it can really screw up a whole race.  There's been multiple times where I've been close to passing another rider, and someone will join right when I'm trying to turn or just correct my bike, and then once the lag is over, my bike is sideways and I'll drop a position or two.  I'll regain those positions, then someone else joins and I'm back where I started.  Then there's my favorite, 5 people joining at once.  Now, it doesn't seem like anyone else racing has a problem, as they just continue, so is it just me?  Or does this happen to everyone?