Sounds a lot like QOS would help. Failing that, find out if the person using Skype is the one hosting the conference call; the host of the conference call is the one communicating with all other call participants, thus requiring hefty chunks of upstream. See if someone else can be the one to start the call, that might help out.
Equally, it could be that your bandwidth requirements fall within what your line is capable of, BUT your ISP is sluggish in the evenings - what ISP are you with?
If only one of you is gaming does the line perform OK? Are you also certain that nobody is torrenting or otherwise battering the connection?
Was on Xilo C&W LLU on MRCEN, Now Be* on MRARD.