Mid-evening blip

For many days many of us have been seeing a mid-evening latency bump. Latency is leaping up to 80ms from the normal 10 or 15ms. The bump shows up best in Thinkbroadband's ping monitor. TBB Ping Meter Many people I know, including me, are having effects upon their service. e.g buffering and erratic game play. Speedtests, in my case, show <30Mb instead of my usual 104Mb.



The odd thing about the bump is that it happens very abruptly around 7.00pm and ends abruptly around 11.00pm. This is not just normal congestion, tailing in and tailing off gradually as folk arrive home from work and later go to bed. Someone is 'throwing a switch'. It doesn't usually occur at weekends, but last night was the first Friday it has happened on.



I want to know what is going on. VM appears to be stepping down the network, reducing our level of service, nation-wide. They may have a good reason for doing this. eg managing congestion or upgrading work. Despite many requests I have not had a satisfactory explanation from VM.


If the problem were to continue I feel that VM would be in breach of contract, especially because the mid evening is when people like me use the internet for video, downloading and gaming. That is what we chose VM for.

Is the latency bump visible on pings to other hosts beyond ours, i.e. not just one of the Virgin Media peering links?

What bit rate are videos that are buffering?

Andrew

This isn't going to be congestion anywhere as the onset and cutoff are far too sharp for that. Here's mine from yesterday - My Broadband Ping

It's something that VM are trying out - unsuccessfully and hopefully temporarily.

It could also be a break somewhere.

We've seen similar patterns on some Be and Talk Talk lines when there is a break and they've had to re-route traffic.

The slow ramping up and then drop back to normal is the pattern we see.

Matt

I've been seeing similar results recently. Yesterdays graph is here http://www.thinkbroadband.com/ping/share/fa73541d123...

Any one with Virgin Media is seeing the same issues, nothing similar with other providers at this time

Same here in Doncaster on VM 30Mb. A little blip on Monday evening then I've had the bump every night since.
Typical graph

In reply to a post by uno:

It could also be a break somewhere.

Considering it is happening at about the same time most nights it isn't any accidental breakage. As I said already it is VM doing something deliberately - probably testing some new traffic management with kit that cannot cope with the load.

Offering unlimited on the local pipes VM have is a recipe for technical disaster. Notoffering it with the groth in availability of FTTC is a recipe for business disaster. VM are caught between a rock and a hard place. Once every VM area has FTTC they really will be up that muddy creek and will be needing some technological magic to even remain afloat (imo).

Anyone got any latency data on different routes, as if this is not just one peering link, but all output from Virgin Media that is affected then it is very very bad

I get the same, seems to of started about a week ago.
And always between about 8 and 10 at night.
Am in Newton Abbot, Devon. An on VM 10M. Don't do gaming or massive downloads just normal e-mailing and web browsing.

Here's my graph from Friday

My Broadband Ping

In reply to a post by MrSaffron:

Anyone got any latency data on different routes, as if this is not just one peering link, but all output from Virgin Media that is affected then it is very very bad

It looks like the bump may be starting again although it's usually clear weekends.

Here is a pathping to bbc.co.uk. If there is a bump tonight I'll do another one for comparison. If not I'll do one when the bump does re-appear

C:\Windows\System32>time
The current time is: 18:05:55.40
Enter the new time:

C:\Windows\System32>pathping bbc.co.uk

Tracing route to bbc.co.uk [212.58.241.131]
over a maximum of 30 hops:
0 X3990-I5 [192.168.0.141]
1 cpc9-nrte25-2-0-gw.8-4.cable.virginmedia.com [86.1.125.1]
2 nrth-core-2b-ae2-2562.network.virginmedia.net [213.106.254.97]
3 nrth-bb-1b-ae8-0.network.virginmedia.net [213.105.159.209]
4 tele-ic-4-ae0-0.network.virginmedia.net [62.253.174.18]
5 pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
6 * * *
Computing statistics for 125 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 X3990-I5 [192.168.0.141] 0/ 100 = 0% |
1 27ms 0/ 100 = 0% 0/ 100 = 0% cpc9-nrte25-2-0-gw.8-4.cable.virginmedia.com [86.1.125.1] 0/ 100 = 0% |
2 14ms 0/ 100 = 0% 0/ 100 = 0% nrth-core-2b-ae2-2562.network.virginmedia.net [213.106.254.97] 0/ 100 = 0% |
3 15ms 0/ 100 = 0% 0/ 100 = 0% nrth-bb-1b-ae8-0.network.virginmedia.net [213.105.159.209] 0/ 100 = 0% |
4 17ms 0/ 100 = 0% 0/ 100 = 0% tele-ic-4-ae0-0.network.virginmedia.net [62.253.174.18] 0/ 100 = 0% |
5 16ms 0/ 100 = 0% 0/ 100 = 0% pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
Trace complete.

C:\Windows\System32>

===

A bump has developed. Pathping to bbc.co.uk at ~ 20:00

C:\Windows\System32>time
The current time is: 19:57:32.65
Enter the new time:

C:\Windows\System32>pathping bbc.co.uk

Tracing route to bbc.co.uk [212.58.241.131]
over a maximum of 30 hops:
0 X3990-I5 [192.168.0.141]
1 cpc9-nrte25-2-0-gw.8-4.cable.virginmedia.com [86.1.125.1]
2 nrth-core-2b-ae2-2562.network.virginmedia.net [213.106.254.97]
3 nrth-bb-1b-ae8-0.network.virginmedia.net [213.105.159.209]
4 tele-ic-4-ae0-0.network.virginmedia.net [62.253.174.18]
5 pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
6 * * *
Computing statistics for 125 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 X3990-I5 [192.168.0.141]
0/ 100 = 0% |
1 16ms 0/ 100 = 0% 0/ 100 = 0% cpc9-nrte25-2-0-gw.8-4.cable.virginmedia.com [86.1.125.1]
0/ 100 = 0% |
2 14ms 0/ 100 = 0% 0/ 100 = 0% nrth-core-2b-ae2-2562.network.virginmedia.net [213.106.254.97]
0/ 100 = 0% |
3 17ms 0/ 100 = 0% 0/ 100 = 0% nrth-bb-1b-ae8-0.network.virginmedia.net [213.105.159.209]
0/ 100 = 0% |
4 17ms 0/ 100 = 0% 0/ 100 = 0% tele-ic-4-ae0-0.network.virginmedia.net [62.253.174.18]
0/ 100 = 0% |
5 15ms 0/ 100 = 0% 0/ 100 = 0% pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]

Trace complete.

C:\Windows\System32>

===

so that path looks OK....

Edited by kwikbreaks (Sun 14-Oct-12 20:03:51)

In reply to a post by MrSaffron:

Any one with Virgin Media is seeing the same issues, nothing similar with other providers at this time

andrew I have seen the affect on traffic to my .nl server's and I get low speedtests at the same time from namesco, it seems to affect youtube, sky go and xbox live also.

Personally to me looks like another VM underinvestment area somewhere.

it defenitly does not affect everything but its affecting more than just tbb monitoring.

Edited by Chrysalis (Sun 14-Oct-12 21:37:24)

This so-called blip affects the upstream according to my observation. Chrys has observed that it may be VM under-investment. But it happened too suddenly for that to be the case. Did they roll out a Cisco IOS update across the estate (Routers) and misapplied it - or something like that? Do these symptoms appear where the Motorola BSRs are installed?

My BT Infinity graph shows no such symptoms.

It's the silence from VM that annoys me more than the blip.

Everyone on VM's cable service is getting it, not on their adsl though. Nor for any other ISP's.

Has happened every weekday from 8pm onwards and slightly different times this weekend.

Check out http://craigswebsites.co.uk/ping/ to easily see the VM graphs altogether and other isp's

I can easily see all the BQM graphs I need, but need to see some data to show it is more than just one link leaving virgin media that is a problem

We're using VM cable at 10MB and I've been running the Broadband monitor since mid September. High latency has been recorded here in Stevenage (Herts) every evening since Tues 9th October. Usually accompanied by visibly downgraded service on both Internet streaming video and catch-up TV (may be unconnected). Seems that the unmissable isn't unmissable in the evenings these days.

It would be nice if VM didn't keep acting like nothing is wrong.

Being on BBC Watchdog last Wednesday complete with BQM graphs shown means most people will know about it

FWIW I'm seeing the same on a TBB graph on a VM BigRedInternet 100 Meg fibre connection which definitely implies that it's not just cable customers but something more fundamental in VMs network / peering arrangements

It looks like we are beginning to get explanations from VM at last.

"Mark_Wilkin wrote:

F002199132 - Hi we've identified an issue with our peering links to a company called Atrato who we publically peer with in Amsterdam and London. They were pushing abnormal amounts of traffic through our public peering links at LINX, overloading them. So we've shut down both of our peering links to Atrato at LINX which should resolve this issue. Meanwhile we're going to continue to investigate the root cause of this problem. Could you please continue to give us feedback on this so we can be sure the issue is resolved.

Thanks"

VM help and support forum

This may explain the problem. I await the 7.00pm deadline.

Meanwhile I am not sure why tracert to pinbox1 at TBB, during the blip, should show such high latency at the TBB end whereas a tracert to the BBC shows no issues. Perhaps the VM connection to TBB is particularly susceptible to this network problem. This may expalin the large graphic size of the blip on the TBB quality monitor . However there is no doubt about the global effects on the VM service customers during the blip. eg, high latency, poor gaming performance, buffering and reduced up and down streams.

There was a direct link from Virgin to the BBC so unlikely to be any issues.

As such an ISP has many different routes out of it and should be monitoring these particularly if they have business users or hosting services, as a bad peering link can cause more problems.

In reply to a post by Suspishio:

This so-called blip affects the upstream according to my observation. Chrys has observed that it may be VM under-investment. But it happened too suddenly for that to be the case. Did they roll out a Cisco IOS update across the estate (Routers) and misapplied it - or something like that? Do these symptoms appear where the Motorola BSRs are installed?

My BT Infinity graph shows no such symptoms.

It's the silence from VM that annoys me more than the blip.

easily explained, VM probably did a routing change causing a link to get overloaded. Its known VM run their links close to saturation its a business policy for them.