You can add me to the affected connections too.

In reply to a post by E300:

Yes does seem to be another bad blip although doesn't follow the same pattern we've seen before where it's stuck around for 30+ minutes, so they might just be running some tests or switching things in and out. Having said that it's not good if they are doing tests with customers on the connection during the working day. Maybe they can swap you to the other gateway the rest of us are on?

They may be getting on top of this. As you say, the 2pm spike is different from the others and perhaps represents maintenance work. Since then we've had 3 hours and counting of stability on the gateway:

https://www.thinkbroadband.com/broadband/monitoring/...

Service status update so hopefully @shortshrift27 your connection should be good now.

We have now carried out the maintenance to apply a permanent fix for the latency issues experienced by some broadband customers. The network is currently operating normally. We are closely monitoring network performance and will continue to do so for the next 48 hours.

To optimise the network further, we will be carrying out planned maintenance on

Saturday 07/11/2020 04:00 AM to 05:00 AM

They have found the issue and those of us moved to the other gateway will be moved back Saturday morning during the maintenance window.

Yup, right now it's looking promising!

I’d just like to thank @Mr Saffron and the thinkbroadband crew for providing such useful BQM tools for us to helpfully diagnose our connections (and provide substantive and helpful information to Cerberus Networks!).

Big thanks Andrew!

I second that, without tools such as these it would be very much harder for us the customer to push towards a resolution.

The troublesome gateway pings look good now, no sign of the repeating latency issue, be interesting to hear what it was in the end. Looking forward to getting back on the original gateway now as IPv6 currently on this substitute gateway is useable but not brilliant, seems to be congestion that ramps up during the working day. Although seems better today, presumably because offices are shut with the lock down and more are working from home on their residential service.

In reply to a post by Pheasant:

I’d just like to thank @Mr Saffron and the thinkbroadband crew for providing such useful BQM tools for us to helpfully diagnose our connections (and provide substantive and helpful information to Cerberus Networks!).

Big thanks Andrew!

I'll third that.

I've been really bad all Day, lots of Yellow and very slow on the speedtest

https://www.thinkbroadband.com/broadband/monitoring/...

In reply to a post by lincsat:

I've been really bad all Day, lots of Yellow and very slow on the speedtest

https://www.thinkbroadband.com/broadband/monitoring/...

Your issue appears to be something different to the original problem and you have lots of packet loss as well.

What gateway do Cerberus have you connected to? Is the chart pinging IPv4 or your IPv6 address? Are the pings direct to your router?

FWIW, I enabled the TBB BQM for my line yesterday, both v4 and v6.

What I see is:
- lots of yellow on the IPv6 address
- occasional yellow spikes on the IPv4 address (coinciding with similar on the IPv6)

If I understand the BQM service description correctly, the yellow line gives the maximum RTT out of 100 pings, so in effect it's the 99th percentile.

This doesn't match with my own measurements. For example, trying 500 pings to Google's IPv6 DNS over 100 seconds:

Text
1 23 45 6	root@nuc1:~# ping6 -q -i0.2 -c500 2001:4860:4860::8888 PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes  --- 2001:4860:4860::8888 ping statistics ---500 packets transmitted, 500 received, 0% packet loss, time 100035ms rtt min/avg/max/mdev = 4.783/5.166/5.388/0.081 ms

You can see average 5.166 and maximum 5.388 - this is extremely flat and consistent.

However, if I do the same to 2a02:68:1::164 which is the address I see the TBB BQM measurements coming in from:

Text
1 23 45 6	root@nuc1:~# ping6 -q -i0.2 -c500 2a02:68:1::164 PING 2a02:68:1::164(2a02:68:1::164) 56 data bytes  --- 2a02:68:1::164 ping statistics ---500 packets transmitted, 500 received, 0% packet loss, time 100193ms rtt min/avg/max/mdev = 3.964/5.349/33.999/3.083 ms

And again:

Text
1 23 45 6	root@nuc1:~# ping6 -q -i0.2 -c500 2a02:68:1::164 PING 2a02:68:1::164(2a02:68:1::164) 56 data bytes  --- 2a02:68:1::164 ping statistics ---500 packets transmitted, 500 received, 0% packet loss, time 100193ms rtt min/avg/max/mdev = 4.040/5.709/38.884/3.948 ms

There are your spikes: peaks in the region of 34-39ms.

Repeating the Google test again:

Text
1 23 45 6	root@nuc1:~# ping6 -q -i0.2 -c500 2001:4860:4860::8888 PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes  --- 2001:4860:4860::8888 ping statistics ---500 packets transmitted, 500 received, 0% packet loss, time 100059ms rtt min/avg/max/mdev = 4.677/5.170/6.801/0.137 ms

Still flat as a pancake.

Therefore, it seems likely that the current BQM spikiness issue is either with the transit/peering between Cerberus and TBB's upstream ISP (NetConnex AS21396), or with the TBB monitoring.

Seeing some IPv6 BQM results for non-Cerberus endpoints would help, so I've just added one for the Google IPv6 DNS address: live graph here and it looks very flat so far.

For me, traceroute to Google and to TBB both goes via the same LINX peering subnet, 2001:7f8:4:1::X:X. It's possible the return traffic is taking a different path though.

This may or may not be the same issue as the one seen last week, where people were affected by serious loss and latency in real-world applications (e.g. VOIP). Given that Cerberus users' BQM graphs were flat before, it seems likely that it's a lesser version of the same issue.