zen Network going down hill

Over the last few weeks zen's network has intermittently suffered packet loss, then it seemed to rectify itself. Maybe some work has been done on parts of the network, but you never get an explanation on Zen's status page. Anyway today it has reared it's ugly head again. I am currently thinking about changing to another provider due to this. I run quite a few VOIP applications using (fire and forget) UDP and the recent packet loss impacted the reliability of these. They are Amateur Radio oriented connecting various radio sites with UDP VOIP.

This is the today snapshot of the ping status :-
https://www.thinkbroadband.com/broadband/monitoring/...
Although not too bad, it has been much worse over previous weeks.

Has anybody else been seeing this, I am in the northwest Near Lancaster.

Dave

Unfortunately, I concur with some strange goings on over at Zen recently.

Joined Zen during Dec 2020 as a FTTP 900 (OR) customer. Performance was always very good, albeit with strange gateway and routing issues occasionally (well documented in this forum). Usually a reboot of my router would bring back in line. Nevertheless, connection was always solid in terms of throughput, give or take 5% (+~850Mbps).

On 14th December 2021 the connection dropped for a short period at around 1AM and my throughput dropped dramatically to anywhere between 250Mbps and 600Mbps. Most notably is that my latency to the first external hop went from around 8ms to 2ms. Which is of course one positive.

Anyway, I raised a ticket with Zen who were trying to do the usual palming off, lots of back and forth. It wasn't until I broke out my monitoring tools and gave reports and graphs that they admitted that on 14th Dec at 0100AM, they had in fact moved my connection from "off-net" to "on-net". I'm not entirely sure what that means in the grand scheme of things but what I do know is that the connection has been really poor since then. A quote from their email below.

"Thanks for your email and providing me with those screenshots. On the 14th December you were moved over to Zen equipment so what we call from off-net to an on-net Zen service. I initially thought it could be related to this so spoke to our networks team who have done a few checks and are unable to find issues on our side as of yet. They have confirmed the backhaul is fine and not contended so should not really be a cause of concern."

Anyway, it's with their network team currently. I'm hopeful they'll find a resolve in the coming week to this otherwise I'll be pushing for a move without penalty.

Update from Zen this morning
-----------------------------------------
"This is very likely a customer side or an Openreach issue., All diagnostics need to be done with the customer and an investigation from Openreach side and fault management if that does not improve the situation. DCoE would need to be contacted if going through an Openreach engineer".

I think highly unlikely given the circumstances and events, but will jump thrpugh the hoops until they admit it's their end.

Edited by Spudster (Tue 21-Dec-21 09:52:46)

Spudster

Offnet would be via a non Zen backhaul, most likely BTwholsale. On-Net would be Zen backhaul, which says they have just built/bought their own backhaul circuit to the FTTP Headend exchange.

Whilst the suspected BTW. backhaul is likely to be 10Gb (or bigger) the Zen one depends on what they have built/bought. it may be that it is not big enough for the customers they have using it. Or they just run at a higher contention. This all depends on what people are doing with their connection, three or more people constantly speed testing at 900Mb will show up whilst 300 plus browsing will not make a dent in a 1Gb backhaul. .

The Ping would have dropped as it is no longer routed through as many switches, so now much more direct.

Edited by kitcat (Tue 21-Dec-21 13:26:57)

Thanks, I appreciate the explanations provided and makes a lot of sense. As soon as Zen admitted the change I knew it was going to be a backhaul or configuration issue on the network they'd plonked me on. But Unfortunately, Zen are sticking to their guns and wasting Openeach's time by arranging a visit to my premises in the coming days. Pointless exercise in my view given the circumstances but will have to follow their procedure.

Apologies for hijack OPs post btw! I think we're loosely in agreement here that the network at Zen is going through some changes that are a cause for concern.

Edited by Spudster (Tue 21-Dec-21 14:39:53)

This may come across as a dumb question but are you certain you're getting packetloss? I have no idea on your router/modem configuration nor did you mention whether you were on FTTP / FTTC. I'm sure you're far more knowledgeable than me when it comes to tracking disruptions but I thought I would throw the question out there regardless. Why did I even think to bring this up? Well, if you're using the Zen supplied FRITZ!Box 7530 those TBM stats may not be entirely accurate. My TBM can be fine sometimes then all of a sudden I start seeing lots of red packets, then they'll clear up again after a few days. I have no idea what you're using to track packetloss but I know when I start pinging random websites my latencies are always stable, well unless I'm downloading, but that's different.

Late edit - Forgot to mention are you seeing CRC, ES, SES build ups in your router stats? If not it as to be a Zen issue somewhere.

In reply to a post by Spudster:

Unfortunately, I concur with some strange goings on over at Zen recently.

Joined Zen during Dec 2020 as a FTTP 900 (OR) customer. Performance was always very good, albeit with strange gateway and routing issues occasionally (well documented in this forum). Usually a reboot of my router would bring back in line. Nevertheless, connection was always solid in terms of throughput, give or take 5% (+~850Mbps).

I have had similar intermittent routing / gateway issues with Zen since signing up. When you say routing / gateway you mean PPP disconnections and authentications right? I'm on FTTC though not FTTP. My event logs on the router just display loads of errors when this random event occurs. These dropouts are not maintenance related either, they tend to happen between 16:00 and 19:00, with one occuring at 15:02. Unfortunately a power cycle doesn't often bring the connection back up for me. I need to factory reset and pray a connection is established, even then it can take upwards of an hour. What's weird in my case is this PPP drop only happens every 2-3-4 weeks. The only thing to ever break the chain is when I have restarted the router before a significant PPP dropout occurs. The DSL and router uptime remains in sync, so it's a bizarre issue. I have recently been sent out a replacement router to test and they would like me to use the wires supplied and to be tested in the test socket with a microfilter. My current setup is with the dsl connected to the MK4 faceplate which is the mastersocket with no extensions so I can't see that being the issue. I will have to see how things develop.

Edited by deleted (Tue 21-Dec-21 15:59:03)

I cannot say I've noted any issues but appreciate that the performance other people are seeing may vary.

In reply to a post by radar:

Over the last few weeks zen's network has intermittently suffered packet loss, then it seemed to rectify itself. Maybe some work has been done on parts of the network, but you never get an explanation on Zen's status page. Anyway today it has reared it's ugly head again. I am currently thinking about changing to another provider due to this. I run quite a few VOIP applications using (fire and forget) UDP and the recent packet loss impacted the reliability of these. They are Amateur Radio oriented connecting various radio sites with UDP VOIP.

Yes, Zen has been getting progressively worse and it has been impacting Zen Wholesale customers too of late. Latency and weird behaviour generally and throughput issues on single threads come back from time to time.

In reply to a post by therioman:

In reply to a post by radar:

Over the last few weeks zen's network has intermittently suffered packet loss, then it seemed to rectify itself. Maybe some work has been done on parts of the network, but you never get an explanation on Zen's status page. Anyway today it has reared it's ugly head again. I am currently thinking about changing to another provider due to this. I run quite a few VOIP applications using (fire and forget) UDP and the recent packet loss impacted the reliability of these. They are Amateur Radio oriented connecting various radio sites with UDP VOIP.

Yes, Zen has been getting progressively worse and it has been impacting Zen Wholesale customers too of late. Latency and weird behaviour generally and throughput issues on single threads come back from time to time.

Bloody awfull at times Packet Loss last night was 50% [censored]

I am losing patience now

https://www.thinkbroadband.com/broadband/monitoring/...

Dave

What time was football online last night? Particularly Arsenal v Sunderland.

In reply to a post by radar:

In reply to a post by therioman:

In reply to a post by radar:

Over the last few weeks zen's network has intermittently suffered packet loss, then it seemed to rectify itself. Maybe some work has been done on parts of the network, but you never get an explanation on Zen's status page. Anyway today it has reared it's ugly head again. I am currently thinking about changing to another provider due to this. I run quite a few VOIP applications using (fire and forget) UDP and the recent packet loss impacted the reliability of these. They are Amateur Radio oriented connecting various radio sites with UDP VOIP.

That looks like some form of interference, have you got access to the line stats? the latency isn't increasing by the look of that graph Doesn't look like congestion
Yes, Zen has been getting progressively worse and it has been impacting Zen Wholesale customers too of late. Latency and weird behaviour generally and throughput issues on single threads come back from time to time.

Bloody awfull at times Packet Loss last night was 50% [censored]

I am losing patience now

https://www.thinkbroadband.com/broadband/monitoring/...

Dave

Yeah it's very frustrating, especially with no information as to what's going on.

I had several outages during video conference calls, terrible timing.

Last Thursday for example

You've posted the wrong link - around 8pm tonight all the packet loss will have disappeared from the graph. You need to share the graph for 21st.

In reply to a post by jelv:

You've posted the wrong link - around 8pm tonight all the packet loss will have disappeared from the graph. You need to share the graph for 21st.

https://www.thinkbroadband.com/broadband/monitoring/...

Dave

I suffered those sort of BQM's for far too long. I saw sense in the end and not had one red spot on my 8db latency graph since.
System Up-Time 51D 20H 36M 31S
Date/Time Wed Dec 22 21:17:15 2021
https://www.thinkbroadband.com/broadband/monitoring/...

Edited by professor973 (Wed 22-Dec-21 21:19:16)

In reply to a post by therioman:

Yes, Zen has been getting progressively worse and it has been impacting Zen Wholesale customers too of late. Latency and weird behaviour generally and throughput issues on single threads come back from time to time.

I'm on Zen FTTC with BTw backhaul.
My BQM looks OK as do my Ookla speedtests, but in the last week or so I've noticed a lot of buffering when using streaming services. Assumed it was something my end although I've not figured out what. I've flashed the stock Fritzbox 7530 firmware back on instead of OpenWRT to exclude that as the culprit.

I cannot say that I have noticed any issues with my connection.

My graph from 21/12/2021 is https://www.thinkbroadband.com/broadband/monitoring/....