Is the phone crackling when you get these low speeds? Maybe someone else may come along with ideas. Other than checking your setup and maybe getting EE to send a new router to test it could be congestion which again is an EE prob.
Tim talktalkbusiness.net & freenetname TTB Router, Asus RT-AC68U and ZyXEL VMG1312-B10A Bridge on 80/20 Meg Fibre Speed Test