Zen internet FTTP customer experiencing severe disruption

Hi, so I discovered this forum after trying to learn a bit more about how the OpenReach/BT FTTP network works and I apologise in advance that this is a bit of a long one. For reference, I used to teach networking commercially for a company called QA Training and although I've forgotten a lot of it, I'm reasonably technical and I like to think I'm ok at troubleshooting but I feel like I'm going mad with this one! I'm also a seasoned solaris/freebsd/linux guy so I've done a load of iperf testing on my local network to prove that's all behaving correctly.

We've been a Zen FTTP customer for a few years now - got FTTP just before lockdown and it has on the whole been excellent. A few months ago though we started to notice issues. I noticed some lag on web pages - nothing too bad - probably only 250ms here and there but it led me to do some speedtests which is when I discovered I was getting around 8Mbps down and 110Mbps up. We're on a 900/100 contract so the 110 is us hitting the cap.

VERY long story as short as possible - this has persisted for months now with numerous openreach engineers, escalation to 2nd and 3rd line openreach teams (it's with (I _think_) the national infrastructure at the moment) and something has definitely changed in the past few days because the connection has started fluctuating wildly between 900 down and 0 down and a fair bit inbetween. If I use the speedtest app or cli tool on a PC that's connected to via a gigabit card to a managed gigabit switch then 5 tests in a row can produce 5 wildly different results. I appreciate that remote server load and contention/etc may play a part in that but we're talking 900 to 4 to 20 to 600 to 0.1 for example. My ping times are always around 11ms and the last test I did was 1.80/102.72.

Tested with 2 different routers (a Unifi security gateway - my normal router and the Zen supplied Fritzbox) and also tested with a gigabit network card connected directly to the ONT using PPPoE auth. In the past, those test results haven't varied significantly from hour to hour - the changes up and down in line speed have been quite slow and the upload is almost always hitting the capped speed.

ONT replaced twice, originally a Huawei, then a Nokia, then another Huawei. I _think_ they said the line tests at -16dB - whatever it is, it's apparently excellent. Clearly there's no issue with the fibre between our house and the exchange because it wouldn't work in one direction and not the other. Apparently the line concentrator (or something like that - it's hard to get definitive information 2nd hand from zen - but I believe it's the bit of equipment that 32 fibres go into) has been replaced at the exchange along with something else.

So, here's the bit I'm hoping someone can help me with. The very first engineer that came out to me tested our line and had an error that he'd never seen before. He spoke to the 2nd line team and they didn't appear to know what it meant either. Happily I got a photo of the test error on his phone screen and the error is "Ethernet TCC parameters breaching thresholds on CP Handover Port". Having googled it, I've learned that TCC is an layer 2 encapsulation protocol - I would guess that's what's used to tunnel our traffic to Zen's network.

The first engineer repeated the test a bunch of times until that error didn't come back and called it "fixed" - thanks! It seems to be an intermittent fault and a number of the more senior engineers that I feel like I'm becoming friends with have seen the error on the line since. Yesterday a friendly engineer rang me because he had overheard his boss talking to his colleague about our house and wanted to know what the current situation was which was very nice of him.

So finally - my actual questions. Does anyone know where in the chain this Ethernet TCC encapsulation would happen? Is there a specific piece of equipment that might be causing those issues?

My understanding is that our fibre goes into the Sutton Coldfield exchange if that rings any bells with anyone.

It might be a more generic problem than you believe. Much discussion about similar goings-on here. If they are connected you are the first to report a specific point of failure beyond the broad-brush problem.

sounds very like the issues I had - always solid 110 up, often dropping to much less than than expected download - sometimes even below a meg and packet loss.

I jumped though various hoops for technical support, but they were just reading from a script and didn't seem to understand what they were asking for (I spent a decade working in telecoms/networking) In my case there may have been equipment issues in the exchange, but I suspect there were other issues in there network

Bottom line - if I get problems and the same poor level of support again I will take the hit leaving before contact up and jump ship as I have very low faith in their wiliness and ability to fix anything

In reply to a post by nononoyesno:

So finally - my actual questions. Does anyone know where in the chain this Ethernet TCC encapsulation would happen? Is there a specific piece of equipment that might be causing those issues?.

This is the Openreach side of the network - in their own words "GEA-FTTP will provide an 'always-on' Virtual LAN (VLAN) between the Optical Line Termination (OLT) equipment in a BT exchange and each Optical Network termination (ONT) device in the End User premises."

Between the Openreach OLT equipment in the exchange and the customer ONT - Ethernet frames are encapsulated into GPON frames (GPON encapsulation method [GEM] frames) for transmission on the GPON network. Ethernet frames are the 'payload' within the GEM frames on the optical network.

From the customer handover point (the copper Ethernet port on the ONT) and the ISP/CP handover point - the other end of the Cablelink - it is pure 802.3 Ethernet.

Refer to SIN 506 for the nitty gritty and light bedtime reading of the Openreach standards around FTTP.

that's really interesting thanks - I'll go and read!

That's interesting but my experience of their support is quite different to yours by the sounds of it. The people on first line have really seemed to know what they're talking about compared to other ISPs I've encountered - one of the guys I talked to on first line was even a CCNA!

Yes, they ask you to prove things that may seem unnecessary sometimes but I understand that they will just be told by the next level up that they didn't do what they were supposed to if they haven't proved x/y/z. Not to mention that they have to deal with "the public" all of the time and must get incredibly bored of the same stupid mistakes being made by customers time and again.

Equally, the openreach engineer who turned up for the 4th call out to my house first asked "have you tried a different ethernet cable". I gave him chapter and verse and "yes, of course I have despite knowing that can't be the problem"!

> but I understand that they will just be told by the next level up that they didn't do what they were supposed to if they haven't proved x/y/z

And there lays the problem - if the next layer up doesn't understand two different routers from two different manufactures are giving the same issue so doing a third test with PC isn't going to give a different result then it doesn't matter how good first level support are, especially as this seems a common problem for the ISP.

Even now if I do a speed test against their preferred speed tester I'm only seeing 50 to 60% of my expected download speeds

to put it in perspective - I was with Plusnet for 7 years previously, I had several issues over that time (only 1 Plusnet related - the others due to line into house) - I spent more time trying to get one fault fixed than I did all the rest before

nononoyesno

This is nothing to do with the PON.

The TCC protocol may be used on the Cablelink between the PON and the Zen network or deeper in the Zen network from their PE router.

see https://www.juniper.net/documentation/us/en/software... for some technical description that is opaque to me!

Edit:- Slightly better description https://www.yumpu.com/en/document/read/30802279/conf...

The TCC protocol has to have matching configurations at both ends of the circuit to work correctly and is sounds like the one your data is travelling over has some mismatches between the ends. As it is only your download affected I suspect the Zen device is overloading the OR device parameters and thereby data is being lost. Zen could be 'overbooking' the agreed bandwidth or sending too many packets ( many small packets rather than fewer larger packets). Impossible to tell from outside the Zen network.

There have been other reports of odd faults within the Zen network when people have been moved from BTW backhaul to Zen backhaul. This may be another symptom of the issues with slightly more detail than we have seen before. You can find the thread(s) on the forum
https://forums.thinkbroadband.com/zen/f/4718211-re-s...

Edited by kitcat (Fri 16-Dec-22 12:44:43)

Did you ever get this resolved as getting same error on Vodafone