Who is hosting the games? Is it you or the person / people you are playing? If somebody else is hosting (especially if it is the same person) then it could be there end that is the problem. One game hubby was playing which didn't have servers hosting it, if one of his friends hosted, then it was as you described, but if we hosted it was fine for both players.
Currently with Talk Talk business (nee Freedom 2 Surf).
Switching to Sky.