FTTC latency spike for the second evening running

For the second evening running, I've had a massive latency spike on an almost idle Zen FTTC connection. It's far worse today than yesterday - I currently have ~55ms unexpected extra latency to the Zen gateway.

BQM for 24/09/2013
BQM for 25/09/2013 (ignore the packet loss this afternoon - I was working on the rack here and had to take the router down for several short periods, though the BT Openreach modem remained on)


Local monitoring (pinging the Zen gateway every second) confirms this is not a BQM artefact and that the connection is almost idle (between 0-1000kbit/s down and 0-150kbit/s up).

I've tried dropping the PPP session, but keep getting reconnected to "dsl1.wh-man". The Zen portal shows I've retained my 79999/20000 sync speeds and it went back to my usual ~10ms fast path latency last night, so I doubt the explanation is to do with the VDSL2 link between here and the cabinet. Even if I'm right that this isn't an issue with the VDSL2 link, I appreciate that there are multiple possible explanations to do with the Zen, BT Wholesale and local BT Openreach networks.


Can anyone from Zen throw any light on what is happening? Is anyone else seeing anything similar?

It may be pure coincidence, but I'm a little uneasy that I've unknowingly signed up for a much more contended service when I moved to Unlimited Fibre 2.

_{Edit: added links to local monitoring graphics}

Edited by deleted (Thu 26-Sep-13 00:50:50)

I've just had a chat with Zen technical support.


The agent confirmed that there's no errors showing on the VDSL2 link to the cabinet, which tends to confirm my suspicion that the problem did not originate there. He also confirmed that there's no routing or prioritisation differences between customers on the old metered products and the new unlimited ones - the difference is purely one of marketing and billing. It is pure coincidence that this problem occurred shortly after moving to an unlimited tariff.

Zen won't receive some relevant contention reports from their suppliers until tomorrow, which hampers their understanding of where the problem is. The network operations people have been alerted to this occurrence, there's a link to this thread on the internal ticket, and I've promised to keep this thread updated over the next few days. Zen have also added some sort of monitoring to my connection so they're aware of what is happening.


The problem is not just a busy router showing higher ICMP latency. 'Real' traffic is affected - my NTP servers typically have 0.15 to 0.8ms jitter to external servers, but this shot up to several tens of ms at 9pm last night. The latency on my IPv6 first hop, which heads over a SixXS tunnel terminating at gblon2 (the protocol 41 tunnel traffic typically routes via Zen's BGP peering with Goscomb at Telehouse North) showed a similar increase in latency to my IPv4 ICMP first hop latency.

The BQM graphics use local time, not UTC. The other graphics, which originate from my in-house servers, also use local time.


The most likely explanation is link saturation somewhere. This could be:

• on the BT Openreach network between the FTTx cabinet and the GEA aggregation node, which I'd expect to be at my local Flitwick (SMFK) exchange as there's no GEA-FTTx on any surrounding exchanges
• on the BT Wholesale network (there's no Zen PoP at my exchange and the Zen portal confirms I'm on WBMC)
• on the Zen links from the BT Wholesale network
• on the Zen network between BT Wholesale and the gateway I'm connected to

The latter two options seem unlikely. Zen have a philosophy of having spare capacity, and Zen didn't seem aware of any saturation problems.


The BT Openreach network may be to blame, though this seems the less likely of the remaining two options.

Virgin offered broadband here several years before the first ADSL was available. A lot of people are still on Virgin, judging by the SSIDs of the wireless networks I can see. FTTx has only been available here for 9 months and ADSL2+ speeds are fairly good (around 14Mbit/s downstream is achievable), though the consumer ISPs are marketing FTTx heavily and a lot of their customers may have switched to fibre products.

My understanding is that there's usually plenty of spare fibre in the FTTx network, though I appreciate not all of it is lit. It might be that a fibre fault has caused traffic to my cabinet to take a different route which is becoming saturated in the evening.


The problem seems most likely to have originated somewhere in the BT Wholesale network. Time will tell!


There was a brief spike on my graphs to 290ms latency to the Zen gateway just before 11:00 local today, 29/09/2013, though it would be conjecture to ascribe any significance to it. BQM didn't pick anything up at that time.

I'll post further graphics if the problem recurs. If it does not recur, I'll confirm that in writing.

In reply to a post by David_W:

The problem seems most likely to have originated somewhere in the BT Wholesale network. Time will tell!

I've had suspicions that the BTwholesale WBC network has congestion/contention and never had anyone able to get anywhere with identifying it. You might be the only one!

I'm with Plusnet on a 100/15Mbps FTTP connection (to Bradwell Abbey, Milton Keynes).

The support guys at Plusnet think it is an issue with Bradwell Abbey (major BT node). The evidence seems to suggest that.

Here is my ping graph: http://www.thinkbroadband.com/ping/share-large/a9870...

I normally get 3-4ms pings to bbc.co.uk and these are anything from 30-50ms at the moment. Speeds are also way down (getting 17-20Mbps down from my usual 88-91Mbps). It's a little strange as my SamKnows reports were showing no speed drops but they must have not been running the tests during the 20:00-22:30 peak time.

There's certainly something in common between our problems - the shapes of our BQM graphs are the same. As both our graphs show, it's at it again this evening.

For any support personnel reading, here's my BQM for 26/09/2013 and my local monitoring for the 10 hour period including tonight's event.

I dropped my PPP session at 21:37 and got reconnected to "dsl7.wh-man" (so I changed gateway this time). As I expected, the change of gateway made no difference.


The commonality between our problems rules out the ISP and ISP links to BT Wholesale, as we're using different ISPs. It all but rules out the BT Openreach network local to me, as we're on different exchanges about fifteen miles apart and I can't think that the GEA aggregation nodes for Flitwick and Bradwell Abbey exchanges are the same.

It now seems almost certain to be a BT Wholesale network issue.

_{Edit: fixed typo, s/Openworld/Openreach/g (thanks jchamier), updated local monitoring graphic to show end of event}

Edited by deleted (Thu 26-Sep-13 23:04:53)

In reply to a post by David_W:

the BT Openworld

I think you mean Openreach. (Openworld was the old name for the ADSL ISP now just called Broadband and part of BT retail division.).

In reply to a post by jchamier:

In reply to a post by David_W:

the BT Openworld

I think you mean Openreach. (Openworld was the old name for the ADSL ISP now just called Broadband and part of BT retail division.).

I do indeed mean BT Openreach - it's been a long day and that was a brain boo-boo.

I remember dial-up BT Openworld...

In reply to a post by David_W:

it's been a long day and that was a brain boo-boo.

I remember dial-up BT Openworld...

I remember my parents thinking of getting Openworld 512k ADSL and being put off by the requirement for the USB modem back in 2000/2001. I had been on NTL cable over a year by then and they were jealous.

I'm getting exactly the same shaped 'hump' on my BQM on Plusnet FTTC in Northampton (Weston Favell exchange).

Hi David,

We've reported it to BTW for the Plusnet users we've seen, I've sent the case number over to Zen too. All the users we've seen so far are on fibre (both FTTC and FTTP) and on the Milton Keynes node. Some routers will let you see the BTW BRAS you are connected to on the event logs, all the ones we've seen so far are bras-redXX-2401.mqd.21cn-infra.bt.net. so you might be able to check if you're in the same pattern.