BTw latency spikes on barely saturated connections

I've got a peculiar issue on my connection but seem to not be alone: Article

I'm currently on a FTTC connection synch'd at ~79000 Kbps/~18500 Kbps. Speed isn't the issue here though as speedtests show that I can fully saturate my connection.

I've noticed since the start of 2019 I've been getting latency spikes when just browsing the Internet or streaming Netflix/Amazon/YouTube/BBC/etc... It doesn't seem to be tied to using a service in particular like the article above suggests but just using the connection in general. I've notice that using more than 8 Mbps causes these spikes of 100-250ms, and the spikes only happen when initially downloading data. If there's a constant stream of data over a long period, the latency spikes do not appear, just only when initially starting a download. This makes online gaming extremely difficult when streaming and not really when just downloading. The latency spikes do not happen when using the upload at all.

The article suggests that this is a BT Consumer issue however, I've recently moved away from BT as I was an employee until this month so dealt with it because of the discount. I'm now on Cerberus Networks with BT Wholesale backhaul and are still experiencing this issue which leads me to believe this is a problem with BT Wholesale and not BT Consumer.

My set-up is an EdgeRouter-12 and Vigor 130 Modem but I've already ruled out any hardware issues as I've tried connecting directly to the modem and using the BT Smart Hub which made no difference. I'm monitoring my bandwidth using SNMP on my EdgeRouter on the PPPoE connection to rule out any devices which may be using bandwidth which haven't been accounted for but that just shows I'm using ~2500 KBps whilst watching Netflix at 4K with a minimum latency of 9ms, average latency of 60ms and consistent peaks of 250ms.

The current fix is to set-up a smart queue on the EdgeRouter and limit downstream bandwidth to 8 Mbps but I might as well downgrade to ADSL at that point and save some money!

I'm connected to the Birmingham Wednesbury PoP (acc-aln1.bm-wed.21cn-infra.bt.net). I was wondering if there's anyone in the surrounding area or just in general who are on BTw backhaul and are having similar issues?

The initial data download have you profiled the amount of data at a high time resolution i.e. its possible that for streaming the initial filling of the buffer may approach line speed and cause a short latency blip

This is particularly so if streaming 4K that is encoded at 20,000 Kbps, the streams may or may not arrive constantly but be delivered in a more bursty fashion, which can lead to intermittent latency variations. i.e. holding a 5 second play buffer which gets refilled when buffer is down to 3 seconds.

The stream cannot fill the line speed as I have tried limiting devices downstream speed to 16 Mbps using FQ Codel, this still causes latency spikes when initially streaming/downloading but the downstream speed is in fact limited, this is no where near saturating the line. After a second, latency stabilises to the idle latency times (9ms), even when the download is still ongoing.

I've attached a ping below by SSHing to my router and running the ping command to bypass the smart queue in case that causes the latency to increase. This is running the fast.com speedtest which shows the smart queue working as the download is maxed at 16 Mbps. You can see the latency spikes initially then, returns back to normal whilst the speedtest is still ongoing.

PING pingbox1.thinkbroadband.com (80.249.99.164) from 83.151.231.190 pppoe0: 56(84) bytes of data.
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=1 ttl=58 time=184.7 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=2 ttl=58 time=48.9 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=3 ttl=58 time=9.7 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=4 ttl=58 time=9.8 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=5 ttl=58 time=10.4 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=6 ttl=58 time=10.7 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=7 ttl=58 time=10.0 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=8 ttl=58 time=10.1 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=9 ttl=58 time=10.6 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=10 ttl=58 time=10.3 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=11 ttl=58 time=10.4 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=12 ttl=58 time=10.0 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=13 ttl=58 time=10.8 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=14 ttl=58 time=9.5 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=15 ttl=58 time=10.7 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=16 ttl=58 time=9.9 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=17 ttl=58 time=10.6 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=18 ttl=58 time=10.7 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=19 ttl=58 time=10.9 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=20 ttl=58 time=10.2 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=21 ttl=58 time=10.1 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=22 ttl=58 time=9.6 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=23 ttl=58 time=9.8 ms
64 bytes from pingbox1.thinkbroadband.com (80.249.99.164): icmp_seq=24 ttl=58 time=10.0 ms

Cerberus are currently moving me to GEA FTTC through TTB as they think that will solve the issue.

No issues on my BT line - I'm in the Highlands:

https://www.thinkbroadband.com/broadband/monitoring/...

Thanks. Judging by posts on the BT forums, this mainly seems to be affecting people in the Midlands region. It's a shame as I was going to upgrade to FTTPoD but I don't want to risk it if the connection can't be used for any latency sensitive applications as I get the spikes when just browsing websites.

I left a download running between 1pm-2pm with smart queue enabled to stop buffer bloat. You can see on the BQM that the latency barely spikes when saturating most of the connection constantly. Compare to 9pm-12am where Netflix was streamed, latency spikes >100ms and when browsing the internet between 11:30am-1pm. This doesn't seem right to me.

BQM
Bandwidth

Edited by dmk001 (Tue 31-Dec-19 16:08:44)

My line has just switched to TTB backhaul and the latency spikes have disappeared! Seems like there is something wrong with BT Wholesale in the Midlands area which they refuse to acknowledge.

BQM

Just thought I'd post to this rather than starting a new thread. This has been happening to me for months and seems to be affecting a handful of people who use ISPs running on BTW. Have any others been experiencing this issue? Just one stream, even a 360p YouTube video and my ping shoots up to a constant 170ms+ on BQM. But if I start a download going at 8MB/s or an upload going at 2.1MB/s the ping goes up to about 40ms. So a fully saturated FTTC connection has a lower ping than a barely saturated one whilst streaming one video 🤔