Are you on 40, 55 or 80 DS product?
Openreach minimum throughput rates for these are 15/20/30 (from the DSLAM to the L2Switch in the handover exchange).
If you're falling below this (remember other things on your network may be using the connection too) then it's more likely either congestion on their link to the L2Switch, backhaul congestion, or another fault somewhere else.
I'm assuming no weirdness with your Sync in all this (i.e. interference between those hours causes your line to re-sync at a poor rate).
Network Architect - Syscomm