As discussed above I have been switched to a new backhaul, and indeed bandwidth have returned to good levels, even higher than they were before. I have to say the technical help was very good, they returned to me by phone. There was a hiccup (the internet was down for a few hours in the morning) but after I called it was swiftly solved.
So no matter what time of day it is the same, so I do not think this is congestion. Have bought a new modem (Draytek Vigor 130), which is synced a bit higher at about 76Mbps, to see if it was the old HG612 giving up, but no, the results where the same.
There was an overnight outage on the 2nd-3rd of May and ever since then I have been seeing random connection failures where some connections time out. Hasn't impacted my speed, but I do find myself having to refresh a page or restart an app to get something to load.
The BQM graphs look normal, so I suspect it's something upstream, but I haven't been able to discern any consistent pattern.