Slow speed after GEA migration

Hi all

I'm on the 900 FTTP package which as far as I'm aware was working fine until about a week ago when it looks like a GEA migration happened.

At this time, my neighbour's connection (on Zen 500 FTTP) just stopped working and they're having a hell of a time trying to get things back up and running. The openreach engineer that attended said he had a load of them to visit that day and he wouldn't be able to fix any of them as they are failed migrations.

I seem to have escaped the issues, or so I thought, as mine reconnected immediately, but I have since noticed that my speed has been absolutely shocking at some times and at other times is very low (tests lower than 10Mbps download at times, regularly lower than 100Mbps).
Tests are both over wifi and ethernet and I have rebooted the router (the Zen supplied fritzbox).
Occasionally I do get a reasonable test result (over 600Mbps). Upload test results are always over 100Mbps which is totally fine.

Considering that before this, test results were always consistently above 800mbps over ethernet (and around 500-600 over wifi), this is definitely a big degradation in service.

I've seen posts about this on various forums online (including this one) both complaining about speed and failed migrations following these migrations. I wonder why these seem to be so problematic and how I'm best tackling this, as I'd like my previous performance back!

Any advice? Has anyone got their original performance back after this migration?

Edit:
Further testing I've just done... fast.com and speedtest.net servers all seem to max out at about 200Mbps download at the moment. This includes trying some previously known good speedtest.net servers (vodafone etc). Zen's own speedtest.net server is particularly poor giving only about 60Mbps!
Any server I've tried on Speedtest.net seems to be like this, except Cloudlfare, which happily zips along as if nothing is wrong... https://www.speedtest.net/result/12960718313
What's going on?
Voda test: https://www.speedtest.net/result/12957202599
Zen test: https://www.speedtest.net/result/12957582043
Speedtest.net auto selection: https://www.speedtest.net/result/12960697398

Edited by FakeJake (Tue 29-Mar-22 13:14:17)

“Migration” implies a move or change to your service - a new provider or a speed regrade on an existing service. A migration should only affect one customer. By the sounds of it, you have neither requested nor had any sort of “migration” but have noticed a distinct step change in your connection performance.

There may be a wider issue on the PON that serves both you and your neighbour. Not necessarily related to any migration per se.

Really the only way forward is for both of you and any other affected neighbours is to chase this up with your respective ISPs and ensure that it properly logged as a service affecting fault. This at least gets the timer ticking for any compensation and should get Openreach on the case.

In reply to a post by Pheasant:

There may be a wider issue on the PON that serves both you and your neighbour. Not necessarily related to any migration per se.

My guess here would be a failed move from BT Wholesale backhaul to Zen's own network. It seems to be happening a fair bit.

Not covering themselves in glory are they! Advice remains: purely within Zen’s gift to resolve, be it backhaul changes, core network, access network…yada yada. The OP can just pile on the pressure

They really aren't!

I had a look at their pricing, at 330/50, 550/75 and 1000/110 Zen are only a few pounds under the likes of Cerberus or Aquiss.

I don't think there's any reason to choose Zen these days, if I had to rely on Openreach FTTP as a primary circuit Cerberus and Aquiss would be the front-runners. A&A are good but I'd struggle to justify the increased cost.

Hi, please feel free to PM me your details and I'll get the issue looked into as a matter or urgency.

If you've already logged the issue with our support team could you also let me know the case number?

We've recently unbundled some more exchanges allowing us to cut out the middle man (e.g. BT Wholesale). A GEA migration, is process of us migrating your connection away from the wholesaler's infrastructure directly to our own. This typically results in an improvement in service or at worst no change. You should never see the degradation you've described, so something would seem to have gone wrong, we'll get that sorted for you ASAP.

In reply to a post by ajays:

We've recently unbundled some more exchanges allowing us to cut out the middle man (e.g. BT Wholesale). A GEA migration, is process of us migrating your connection away from the wholesaler's infrastructure directly to our own.

Hi ajays

Out of interest do the Zen migrations team notify customers when this type of work is going to happen and what measures do they put in place to test and ensure customers are not negatively affected by their changes?

I don't believe I was informed. The only way I know something happened was after I checked the "My orders" section of my account. In there it shows an order for a GEA migration at a cost to me of £0.

Previous orders in there are when I've migrated to a different package (when the 900 package became available).

In reply to a post by ajays:

Hi, please feel free to PM me your details and I'll get the issue looked into as a matter or urgency.

If you've already logged the issue with our support team could you also let me know the case number?

We've recently unbundled some more exchanges allowing us to cut out the middle man (e.g. BT Wholesale). A GEA migration, is process of us migrating your connection away from the wholesaler's infrastructure directly to our own. This typically results in an improvement in service or at worst no change. You should never see the degradation you've described, so something would seem to have gone wrong, we'll get that sorted for you ASAP.

I'll send you a PM shortly. I haven't logged it yet as I was interested in what others may have offered as advice to get this fixed quickly.

Thank you for offering to get this looked into

PM sent.
I've also noticed that my IPv6 is gone... Hopefully this can be easily and quickly fixed

Have you done a tracert to check which gateway you are connecting to? They changed my FTTC to connect to Manchester (even though my head exchange is in Dorset) and I lost IPv6 (and latency doubled). They changed me back to London and IPv6 was restored.

Yes the speed is poor on a variety of PoPs. Both Manchester and London. I did notice that the Manchester one doubled my first hop latency, despite being very close to Manchester geographically.

Zen called me and we ran through a thorough set of troubleshooting steps to no avail. Then shortly afterwards a fault manager called me and said that I'd be getting an Openreach engineer visit soon. I doubt they'll find anything physically wrong here but you never know.

Thanks to Andrew on here for getting the ball rolling though.

In reply to a post by FakeJake:

I'd be getting an Openreach engineer visit soon.

Did they try temporarily reversing the migration as that would seem the most logical next step and is probably step 1 in the dummies guide to fault finding.

In reply to a post by FakeJake:

.
I've also noticed that my IPv6 is gone... Hopefully this can be easily and quickly fixed

That apparently always happens with a GEA migration with Zen.
You just need to fill in the IPV6 form again or send them a quick message and they will enable it again.

So why don't they simply check and fix it themselves? Aren't they supposed to be a class act?

In reply to a post by pluralist:

So why don't they simply check and fix it themselves? Aren't they supposed to be a class act?

They used to be a class act, but I'm not so sure that they are any longer. They seem to want to change from a niche ISP to a mass market one.

In reply to a post by dect:

Hi ajays

Out of interest do the Zen migrations team notify customers when this type of work is going to happen and what measures do they put in place to test and ensure customers are not negatively affected by their changes?

I'll take from your silence that Zen neither notify their customers of this service affecting change nor do any testing to ensure their customers are not negatively impacted.

They don't. Mine was migrated on the 7th it seems (you'll have an entry in the orders section of the zen portal) and I was never notified. My bqm graph though shows the latency doubled a few days later after the usual gateway change. I'm honestly thinking of leaving zen with all this rubbish over their manchester/london gateways.

How can this be considered normal on an FTTP 900 package?

https://www.thinkbroadband.com/broadband/monitoring/...

This was today:

Speedtest: https://www.speedtest.net/result/12971030032.png
Traceroute:

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 218 ms 217 ms 222 ms lo0-0.bng3.wh-man.zen.net.uk [51.148.77.130]
3 240 ms 239 ms 252 ms lag-7.p2.wh-man.zen.net.uk [51.148.73.14]
4 266 ms 275 ms 309 ms lag-3.p1.thn-lon.zen.net.uk [51.148.73.136]
5 316 ms 314 ms 321 ms lag-1.br1.thn-lon.zen.net.uk [51.148.73.153]
6 373 ms 311 ms 268 ms 72.14.223.28
7 * 23 ms 22 ms 209.85.249.149
8 21 ms 22 ms 22 ms 142.251.54.31
9 20 ms 20 ms 20 ms dns.google [8.8.8.8]

Speedtest: https://www.speedtest.net/result/12971085484.png
Traceroute:

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 13 ms 11 ms 8 ms lo0-0.bng2.ixn-lon.zen.net.uk [51.148.77.129]
3 9 ms 8 ms 8 ms lag-4.p1.ixn-lon.zen.net.uk [51.148.73.89]
4 8 ms 8 ms 8 ms lag-2.p1.thn-lon.zen.net.uk [51.148.73.132]
5 8 ms 7 ms 8 ms lag-1.br1.thn-lon.zen.net.uk [51.148.73.153]
6 9 ms 8 ms 8 ms 72.14.223.28
7 10 ms 10 ms 10 ms 209.85.249.149
8 7 ms 8 ms 8 ms 142.251.54.31
9 8 ms 7 ms 8 ms dns.google [8.8.8.8]

A difference of 20ms -> 8ms

Edited by zebb_edi (Thu 31-Mar-22 14:20:59)

This looks fine to me. I currently achieve about 40mbps down, 110mbps up.

At least my uploads are ok!

The GEA migration seems ok for me. It's their gateway disparity that is daft.

Thanks for the pointer to the order on the portal. I've now found an order for "Zen Fibre GEA Migration" for my FTTC line showing as "Fulfilled on 22/03/2022". Like you I wasn't notified. The connection dropped in the early hours the following day and I immediately found I'd lost IPv6 and contacted them to complain. Just after 10am they obviously reversed the order. All can be seen, including the affect on latency here: IPv4 Quality Monitor

Edited by jelv (Thu 31-Mar-22 16:53:30)

Update:
Openreach chap turned up at expected time, ran a few tests, called their office to check profiles etc were all correct and basically said there was nothing wrong physically here and handed it back to Zen.

Fault manager at Zen rang me a few hours later and explained about the migration and that they were handing it to the networking team to investigate further.

In reply to a post by FakeJake:

Update:
Openreach chap turned up at expected time, ran a few tests, called their office to check profiles etc were all correct and basically said there was nothing wrong physically here and handed it back to Zen.

Fault manager at Zen rang me a few hours later and explained about the migration and that they were handing it to the networking team to investigate further.

No surprise the Openreach engineer found no issue on their side, sadly a lot of ISPs have been for years throwing faults over to Openreach as a first line of support and this may have been acceptable when it was ADSL and VDSL type products but with FTTP it really isn't the best move. This issue clearly is with the change that they made so its clearly now back with them. If it was me I would be pushing for them to revert you back to the pre-migration configuration until they can get to the bottom of this issue.

Edited by deleted (Fri 01-Apr-22 19:04:13)

I was GEA migrated on 20th March at 3:15am - Just at the time my internet has started running very slowly, and with massive slow-ups, intermittency etc. I contacted Zen, and they have changed my router (twice!) - surprisingly with zero difference. The issue manifests as an OK PPPOE connection as far as the router is concerned (there is a Fritz Speedtest you can run under the diagnostics page on the UI). The problem is LAN-side, not WAN-side, or at least it appears that way.... I did try using an ancient Apple Time Capsule, and it is radically better (a 200Mbps downlink, that is reliable, compared to the Fritz giving about 100 at BEST) The real problem seems to be as being discussed - either routing being upset now (WAN side), or possibly MTU? The real issue is that Zen KNOW of this issue, yet are trying to fob it off (that's how it feels) The reality is that everyone is having problems after the GEA migration, and NOT after changing their router or home LAN configs etc. Allegedly Zen expert contacting me tomorrow, and will do home visit if required, but I am starting to smell nasty rodents on this issue.

In reply to a post by SteveBushell999:

The problem is LAN-side, not WAN-side, or at least it appears that way....

You say you think your problem is LAN side? what makes you think it is? if your issue is in fact LAN side then I personally can't see your issue being caused by the GEA migration.

Edit: cleaned post up so hopefully it makes sense

Edited by deleted (Sun 03-Apr-22 21:49:19)

Hi, yes, it is odd that it manifests as LAN side, but let's examine.....

Firstly, the problems all started the day after my (unrequested) GEA migration occurred, bit of a coincidence as all was fine for a year previously, and it then immediately goes bad, and stays that way????

OK, why LAN-side? All my measurements are over less than 10 metre runs of CAT5e (specified to 1Gbps up to 100Mtrs) - I don't use WiFi for any tests as it has latency, bandwidth issues at distance, jitter, co-channel occupancy, possible interference etc etc). Below are what I see after the magnificent migration....

All tests boiled down to short CAT5e to the ONT, into the router, and a single CAT5e out - directly to the test machine - no other hardware or routers involved.

1) Fritz router number 1, the one originally supplied - FAST.com LAN gives 120 Mbps at best, always did give 500Mbps

2) Fritz router no 1, running the Fritz inbuilt diagnostics speed test (in Germany, so latency is 35mS) gives 500Mbits down.

3) Loan Fritz router - exactly same as above

4) Loan CityFibre Technicolor router - same connectivity and tests, but no ability to perform WANside Speedtest

5) connect Mac or Windows box over same LAN cable as before, but direct to ONT and establish PPPOE connection to Zen and 500Mbps achieved

I back all these tests up with observations using USENet downloads, which are usually very good here, at consistently around 45MBps for the last year (yes, MegaBytes, I do know the difference!).... After the migration, all I can achieve is more like 1.5MBytes / sec. APART that is when I test using a direct PPPOE connection, whence it is just fine - as before 45MBps

I must add, that I have not quoted uplink speeds as these are pretty consistent across all the above at around 60Mbps.

So, that's why I say the problem 'appears' LAN-side, but the quotes are for a reason! Obviously something changed at exactly the time of the GEA migration, and it has seemingly affected the throughput of 3 different routers, but there are other possibilities of course, MTU, routing, and I am sure there may be others). The perverse thing is that an ancient Apple Time Capsule used as a PPPOE router gives considerably better performance than the Fritz of Technicolor.

Go figure !!!

Hi Steve

If it was me I would stick to the key point when talking to Zen that the issue started immediately after the GEA migration (unlikely to be a coincidence). if you give Zen away out by saying it appears to be LAN side they will take that with both hands and you may never get the issue resolved.

Let's see what is said, I only found out about the GEA part at the weekend, so will keep that up my sleeve for now! Don't worry, I 'know' where the fault comes from, but I am a Software Performance Tester by job, so tend to report things as I see them, but will stand my ground if BS comes my way ! The thing is, there must be quite a lot of people affected (many of who don't understand / know of the problem), and I am trying to improve awareness so the matter can be properly solved.

OK

In one of my earlier posts I did ask the Zen rep (ajays) on here directly if they notified their customers about this GEA migration and he chose not to reply (even though he has been online since) so they are clearly trying to do it under the radar so in the future when more customers become away of issues they are not linked to this migration.

Is anyone that is impacted by slow speed issues following a migration to Zen On net within an hours or so's drive of Manchester? If you are and would be willing to host an engineer from our Network team for a hour or two in the next day or too, could you drop me a message?

Cheers,
Brandon.

Core Network Design, Zen.

Could you help Brandon?

Hi Brandon, I am in Winsford, and available....

In reply to a post by dect:

Could you help Brandon?

Hi, your account doesn't accept private messages. Could you send me your contact details in a PM?

Cheer, Bran.

In reply to a post by hazey_flakes:

Hi, your account doesn't accept private messages. Could you send me your contact details in a PM?

Cheer, Bran.

Sorry no point as I can't help, am trying to rally those who may be able too

I have sent PM with my details....

GEA migration aside, is it possible to also investigate the disparity between the manchester and london gateways? The latency is quite a difference. Or is there nothing that can be done about that?

At present the gateway is purely based on whichever gateway responds fastest to the pppoe initialisation request from the CPE. We are making a change soon which will make this more deterministic and you'll generally (non fault conditions) be connected to a BNG pool closer to your location.

Post deleted by hazey_flakes

Thanks. That would be a welcome improvement.

ps) For me based in South Glos on the Almondsbury headend exchange and recently GEA migrated, I actually think the service is lower latency if I get on to the london gateway than it was prior to being migrated. It seems to have improved the connection. I don't know if that's possible or i'm imagining it.

Edited by zebb_edi (Mon 04-Apr-22 12:20:36)

In reply to a post by hazey_flakes:

Is anyone that is impacted by slow speed issues following a migration to Zen On net within an hours or so's drive of Manchester? If you are and would be willing to host an engineer from our Network team for a hour or two in the next day or too, could you drop me a message?

Cheers,
Brandon.

Core Network Design, Zen.

Hi Brandon - sending you a message now as I'm basically just outside Chester which isn't too far from Manchester

Thank you to everyone who has messaged me. I hopefully have enough offers that I can run the additional checks that I'll need. For everyone is that is affected could I ask that you do raise the faults though the normal Technical Support channels if you have not already done so?

Regards,
Brandon.

Do you know if the originally supplied Fritzbox received a firmware update coincidental with the reduced downstream throughput.

I ask because by default these updates are enabled by and supplied from the manufacturer, though I think you can change the settings for busy times or notify-only or not at all.

It's possible that a later firmware combined with migrated config could result in some built-in QoS being active, which could favour low latency over sustained download throughput for example.

With asymmetric connections and a sequential test, during the upload test there is often plenty bandwidth for the ACKs (returning downstream) such that QoS does not need to kick in.

Otherwise I'd wonder if it was something else affecting PPPoE hardware acceleration (as a computer doing the PPPoE itself generally has more than enough CPU to handle it purely in software while a low power appliance relies on some kind assistance - may be called fast path or offload processing for example.

In my case, the slow speeds persist even when I've plugged a laptop into the ONT directly (bypassing the router)

Hi, no, there had been no FritzBox update, my router was at 7.29 (since Nov 2021), and the loan Fritz had same version. I also tried 7.30 beta, and that was exactly the same.

You are correct that boxes like my Time Capsule don't have enough horsepower to do 1Gb routing, but equally the problem is not that simple. It appears that most routers show packet loss over my connection (Time capsule and direct PPPOE seem much better for some as yet undiscovered reason). Zen are coming to me to run a line test tomorrow, so let's see what that shows.

I suspect that the router issue I'm seeing is somehow a symptom of the problem rather than the cause, it's just too incredible that 3 different routers are similarly faulty, and in the case of mine that it stopped working properly at EXACTLY the same time as my GEA migration?

Is/are any of your LAN cables one that was in use before the migration? If so, replace it with a brand new one. If you don't know which, replace all.

Also check all ethernet sockets for contamination including the ONT, before inserting a new cable. Air-blower clean perhaps. Similarly check and clean the plugs on the cables.

It does sound very much like a cabling fault to me.

Well this could be fun.

Tomorrow morning a Zen network engineer and an Openreach engineer are both going to be here.

Best get the kettle on. Also popcorn? 😁

Sorry, not cables!

Plug same cable into router that I use into the ONT, and no cables have changed in any case. I repeat !!! ALL this only occurred simultaneously with my GEA migration..... I changed no cables during that time, nor any of my equipment.

As I say on another post, it strongly seems like the 'router issue' is a red-herring, and is simply a symptom rather than the cause.

What the cause is? I'll publish as soon as I know, Zen engineer coming here tomorrow, don't think they'd do that if they suspected cabling! Or that 3 different routers suddenly fail in the same way !

In reply to a post by FakeJake:

Well this could be fun.

Tomorrow morning a Zen network engineer and an Openreach engineer are both going to be here.

Best get the kettle on. Also popcorn? 😁

I hope the three of you can work it out

It took a bit of pushing by those posting in this thread but Zen do now seem to be taking the GEA migration issue seriously so fingers crossed for a resolution or at least a plan of how too.

It's disappointing because this is really the sort of thing they should have been able to pick up on, just looking at the utilisation of the customer circuits before and after the migration would have shown that there was a problem.

In reply to a post by FakeJake:

Well this could be fun.

Tomorrow morning a Zen network engineer and an Openreach engineer are both going to be here.

Best get the kettle on. Also popcorn? 😁

I'm not sure if you meant to reply to my post about the cables, or just replied to the latest post. But it was a reply to Steve Bushell. (Remember this forum has a threaded system linking post and reply ).

I haven't thought much about your problem, sorry.

We shall see .

hmmm..... not sure about anyone else, but my Zen visit tomorrow has just been delayed to next week...... I will update as I know more

Same here. No popcorn tomorrow then

Maybe they think they've found the problem and need a few days to confirm and fix. So no point in sending people out. Particularly network ones.

In reply to a post by FakeJake:

Same here. No popcorn tomorrow then

Would have been nice if they had given you a reason when they delayed the diagnostic visit, it would have certainly prevented any negative comments resurfacing on here about the issue.

Sadly one step forward and two steps back in the pursuit of good customer service

Working in IT myself, sudden unexpected site visits can be delayed for all sorts of reasons. I'm not looking into it too much

In reply to a post by FakeJake:

sudden unexpected site visits can be delayed for all sorts of reasons.

Very true but in this instance you don't know what that reason is as no one has bothered to tell you why, personally I don't like being kept in the dark but clearly not everyone is the same.

Nobody likes to be in an information vacuum, I do accept that this sort of a problem can be very hard to isolate, and then to fix, but we reported this quite a time ago, and there is no real update (aside from we will send someone out..... sometime). For my part, my internet speed is down at less than 50% of what I pay for for the past 2 1/2 weeks, which is inconvenient to say the least as my job is performance testing IT systems.

I am waiting to see what eventually transpires, but it does seem to be dragging on with little update as to what is happening.

No ISP likes bad publicity and thats exactly what is happening through this thread, they know this site is highly read even if people don't post and when people are looking at changing ISP they remember these sorts of problems as no one wants to move and have a protracted issue like this.

If some members here who have the issue are OK to wait an extra week to then maybe get someone round to investigate then thats fine but I would have liked to know why they was delaying it. Zen may think they know what the issue is or may be just putting it on the back burner for a week. No one knows and thats the problem.

Or, to put it another way..... Can we be migrated back to how it was before this unrequested migration? That seems the easiest solution?

In reply to a post by SteveBushell999:

Or, to put it another way..... Can we be migrated back to how it was before this unrequested migration? That seems the easiest solution?

Totally agree that would be the logical thing to do until they find and resolve the issue.

The other thing is are they still migrating customers even though they haven't sorted this issue, lets hope not.

Totally agree dect !,

"The other thing is are they still migrating customers even though they haven't sorted this issue, lets hope not."

- it would sound a rather poor choice to migrate anyone else until this issue is a) identified and b) resolved.

I do know that on the evening of the 5th (after zen replied to this topic) both mine and my neighbours connections (who are GEA migrated) dropped for a period of time. So weather they were doing tests/checks or configuration changes I don't know?

Obviously doesn't help you if you do still have problems, but maybe they are doing things still behind the scenes. And I agree it would have been courteous to at least notify you.

https://www.thinkbroadband.com/broadband/monitoring/...

Purely a guess but I'm wondering if they were testing the Openreach part of the link to rule it out

https://www.ispreview.co.uk/index.php/2021/11/openre...

In reply to a post by SteveBushell999:

Or, to put it another way..... Can we be migrated back to how it was before this unrequested migration? That seems the easiest solution?

Not sure if yours was different but it looks like a migration takes about 2 weeks to happen. I don't think that's going to get things working any faster.

Update: Openreach guy attended (same guy as last time). He replaced the ONT this time. Ran more tests. Unsurprisingly this didn't seem to make a difference.

After I complained about losing IPv6 and doubled latency it took about an hour to be switched back to London instead of Manchester.

In reply to a post by FakeJake:

migration takes about 2 weeks to happen. I don't think that's going to get things working any faster.

I don't believe it would take anything like that to migrate a customer back, I would put about an hour on it if you want my opinion.

2 weeks is how long it takes BT to migrate between the two networks. The actual work involved may be very little, but the lead times are 10 working days.

In reply to a post by Geordish:

2 weeks is how long it takes BT to migrate between the two networks. The actual work involved may be very little, but the lead times are 10 working days.

Not sure your doing an equivalent comparison, but hey ho lets say its 2 months or 2 years and all sit here in the dark.

Yes, perhaps, but I have now been very nearly 2 weeks with messed up internet due to this migration..... On the plus side, Zen network engineer visiting me on Monday, so maybe there is hope..... I tried the original Fritz router again last night, and it's the same, so for now am back with the Apple one which gives way better results (albeit 1/2 speed). Not exactly all sunshine and roses though....
https://www.thinkbroadband.com/broadband/monitoring/...

Thats it for me, good luck on Monday

Yes it's Monday for me too.

Here's my graph https://www.thinkbroadband.com/broadband/monitoring/...

Post deleted by CarlTSpeak

If Zen place an order today to migrate a line back to the BT wholesale network it will take 10 working days to complete. I’m not sure that it can be expedited.

Hi FakeJake, do I assume you are still using the Fritz router? If so, what happens if you run their diagnostics Speedtest, do you get full speed then? I do, which is why I raised the possible router issue..... The rest seems like ancient history !
Actually, the difference between both of our broadband traces is really interesting... You get loads of packet loss consistently, and I don't (using my router) but did when I used Fritz.... This is becoming curiouser and curiouser... The drop at about 4pm yesterday on my graph is where I retried the Fritz router for a few minutes....

Edited by SteveBushell999 (Thu 07-Apr-22 15:07:49)

I believe the "packet loss" is just the fritzbox not responding to every ping. Apparently a known quirk. I've got no problems with packet loss as far as I'm aware.

Not sure how I can run a speed test from the router itself... Where do I find that?

Login to Fritz!Box..... Diagnostics...... Function...... Run that, and one of the options (a link) is the option to run a Speedtest (its the only link) - sadly it's in German, but I get full upload / download speed doing that - Of course, that is a WAN measurement, that isn't reflected LAN side, but it does 'prove' that the PPPOE connection can work at full speed..... Then you will be down the crazy Rabbit Hole that I am in.....

It's something like this http://avm.de/service/zack-der-speedtest-fuer-ihre-b...

PM me if you need help

Edited by SteveBushell999 (Thu 07-Apr-22 16:00:12)

Ok, when i ran this test, it's just a browser based speed test page, which runs the test from the browser you're using, not the router itself.

Update: Zen network guy visited with some serious test kit and ran some tests for about an hour connected directly to the ONT (new one changed by Openreach) to eliminate all parts of my network, including their router.
Confirmed the problem that I've been seeing is definitely present.
He said that interestingly that the problem seems to be present on TCP but not UDP (or maybe not as bad on UDP).
Should hear more once they've analysed their packet captures that they've taken from here, the exchange and other points further along the network.

Hi,

Zen guy just left here, similar observations, but equally no obvious conclusions as yet. Also seeing some odd packet behaviour, but they weren't immediately sure where (on the route) was causing it. I thought Dario said my issue was UPD ok, but TCP issue - which makes more sense? Perhaps I misheard.

I am wondering if something odd is happening to NAT packets? But I have no proof, just a hunch? It's still interesting as NAT isn't just one way of wrapping stuff up (see WIKI) which may explain why I see different behaviour between different routers? More research needed!

FakeJake

Been lurking on the thread for some time as had nothing of value to input.

But interesting if TCP is affected but UDP is not. There are network settings that can prioritise one protocol over the other. But even without these UDP is faster as it is connectioness whereas TCP is slower but more reliable due to retransmission.

Intuition says there is a box that is struggling and causing many transmission requests slowing everybody affected down significantly using TCP , UDP will not be affected as much but may be losing packets.

Somewhere in the new core routeing is a poorly router (or more than one) may even be a design flaw in a new set of boxes that only becomes apparent when loaded up enough, (pain to find when doing pre go live testing) . .

.

Yes what you've said makes sense and is a possible explanation as to why there's a difference seen when using TCP or UDP.

The hope is that once they've had a proper look at the packet captures, they'll be able to point the finger at what/who is causing the issue and will have data to back that up. I'm looking forward to hearing more.

In reply to a post by SteveBushell999:

I am wondering if something odd is happening to NAT packets? But I have no proof, just a hunch? It's still interesting as NAT isn't just one way of wrapping stuff up (see WIKI) which may explain why I see different behaviour between different routers? More research needed!

The equipment out in the network won't know anything is NATted - that happens on your router and that's the end of the process.

CGNAT?

Not on Zen

... we are led to believe.

What about Cityfibre?

It's not really a "led to believe" thing, CGNAT is easy to spot.

pluralist

If you are looking for more exotic things to go wrong it is more likely something to do with DPI and legal intercept if, as likely, Cityfibre are now big enough to have to implement this for the spooks. (Zen may also be at the size to have to implement things)..

There are several ways to do it that will interfere with throughput noticeably. Occam's Razor suggests core router problems are more likely ( Unless all the people with an issue are using VPNs which could overload the intercept point if the VPN is being intercepted which is a possibility since 24th Feb! )

My bet is still on the BGW between CF and Zen having an issue

Edited by kitcat (Mon 11-Apr-22 20:09:58)

Cityfibre have nothing to do with my connection so I would doubt it's anything to do with them. My connection is delivered via Openreach FTTP.

In reply to a post by kitcat:

My bet is still on the BGW between CF and Zen having an issue

The thread is about speed issues after a GEA backhaul migration from BTW/TTB to Zens own backhaul.

Seeing as that's only relevant over Openreach then the chances of it having anything to do with CityFibre are somewhere between zero and nil.

In reply to a post by jpm:

It's not really a "led to believe" thing, CGNAT is easy to spot.

To reply to the replies to my post throwing CGNAT into the ring, thanks to all.

After a working life which involved much hunting for weird happenings, and given Zen's apparent refusal for weeks even to accept there was a problem, I've just started throwing "outside possibilities" into the ring. In the hope it might spark a memory or thought in one of those here who I accept are far more up-to-date than I on networking.

I'm happy to be shot down each time , but more than once in the past I have hit on something that has been simply forgotten as a possibility because others have gone too deep because the obvious is expected to have been ruled out by first-line.

Zen seemed to be working on the premise the fault had to be at the user premises because only the LAN seemed to be affected.

Since my post, the UDP v TCP question has been raised and looks to me to be very relevant.

FakeJake

Apologies mixing threads up. It is not the FTTP part that is causing the issue but a Core or backhaul network issue somewhere. The move to the Zen network has changed the routeing to a different set of boxes some of which appear to be having an issue.

Not sure if ZEN builds it's own network or rents it from others I got the impression from elsewhere that they may be renting some of it from Cityfibre (Rather than BTW) .

Waiting with baited breath.... No new yet....Guess same for you all?

An update from Zen:

I’m still analysing the data we gathered from the testing. I have no immediate conclusions, but I have a couple of areas to investigate further.

The tests do show packet loss on the UDP tests, and poor performance on the TCP tests. I’m analysing the TCP data in more depth as that does have some areas of concern for me.

So.... We must wait....

The data we captured was very useful. We'll now be having further conversations with Openreach. I believe not all data sent into the GEA Cablelink link is getting to the server attached to the ONT.
We'll need to discuss that in more depth.

Thanks for the extra clarity Brandon. It's all very interesting, and I am really keen to understand what actually happened. In the meantime, I will continue using my Apple router as it's a lot better than the Fritz (I tried that one again earlier today)

So it looks like the GEA link is to blame? I guess that makes sense as this wouldn't have been in the picture prior to migration. (I say knowing that I know nothing about how any of this works)

Hope you guys can get this sorted quickly now that you have something to go on.

In the meantime, please can I have my IPv6 back? It hasn't worked since this migration happened. Happy to email the IPv6 team if it's more appropriate🙂

Edited by FakeJake (Wed 13-Apr-22 16:28:29)

FakeJake

As GEA links are site (and equipment) specific is sounds as if some configuration parameters are being set incorrectly across the network. As it has only been reported affecting Zen links it sounds like Zen configuration.

Wondering if (downstream) packet size is incorrect or a rate setting is too high that leads to packets being discarded by the (OR) edge router.

GEA bandwidth sending more than the capacity they have brought could lead to this, eg 10Gb port but only brought 2Gb capacity, bursts to 2.1Gb and loses 0.1Gb as the GEA will take the extra but the Edge router will police it down. by dropping packets.

This is a case of the more you know, the less you realise you know. Thanks for the info 👍

In other news, what happened at midnight? https://www.thinkbroadband.com/broadband/monitoring/...

I know that it's been Easter etc, but wondering if anyone has any updates on this as the situation hasn't changed from my perspective?

Perhaps we will get compensation in the form of 50 Easter eggs.

I'll have to do a lot of hill walking to work those off.

Performance same here. I emailed the IPv6 team to get that working again, which it now is.

OK, so 'some' good news for you at least!

It's getting closer to the 30 days OFCOM gave them, but I don't want to quit Zen, who have been good most of the time I used them. However, going back to (say) BT would at least solve this problem I think. I just wish we knew what the problem was, and why we can't just be switched back to pre GEA if it can't be fixed????

Realistically I think that should be an option. It worked fine before migration so un-migrating would "fix" it.

I imagine Zen want to get this working though. It can't be cheap to unbundle an exchange.

The 30 day Ofcom thing doesn't affect me. I'm out of contract anyway. I'll happily sign another contract once it's fixed. Mostly because it will actually lower my bill by £5/pm.

The IPv6 guys have always replied within a few hours to me which is good.

Edited by FakeJake (Wed 20-Apr-22 16:06:25)

If only it really were just cost ! Normally I have a passionate hatred of BT 'OpenJOKE', but on this occasion I am not so sure. BT had a pound of flesh out of me when I moved here, tried to charge me over £1000 because they couldn't understand that the place was FTTP not FTTC, they were actually very expensive, and I hated the offshore 'support'.

All I want is what I seemed to have, but now have not got, and no idea when / if I will have again. Seems simple to me! Yes, I get the costs of falling back etc, but we didn't ASK for this did we? Even if we had been, would the risks have been pointed out?

I am a software performance tester by trade, and do understand all the vagaries here, but the question around was this ever properly tested BEFORE we were made Guinea Pigs does arise.....

At least you have IPV6.... I settle for fixed IPV4 and Zen hosting my domain pointing here so I can run email server, other servers, log in remotely, get customer trusts to my IP for running external performance tests etc etc

Perhaps their multi-vendor core network strategy has become something of an Achilles heal as part of the backhaul migrations....

https://youtu.be/MGTkxrIQXKQ

"we don't have any congestion on the network and we never lose a packet"....words that have come back to haunt them here it seems.

Thank you Pheasant!

Good to hear Richard Shaw promising 'We never lose a packet' - Hmmmmm

It is a worrying portent that they aspire(ED) to, I wonder what the response is now?
Please, can we have a response? The GEA 'Upgrade' seems a botched, and untested one, not happy ! Additionally, the response isn't as quick and transparent as I would like for sure. Why the cover up? If it is a c**k up, then simply undo it please?

No longer a happy bunny.

Richard works for ZZoomm now, so maybe the promise moved with him

I assume Brandon must be on leave, I PM'd him with some info over a week ago where I have seen similar issues to this before with more than one ISP interworking with PON networks (not just Openreach).

My suspicion is that now Zen are directly connecting to Openreach there is an issue with traffic exceeding the policing parameters Openreach define in SIN506 (Table 1). When via BT Wholesale or similar that side ensures they meet these.

BNGs can be terrible at ensuring they don't exceed the burst parameters and the PON equipment does normally police per user hard to the limits defined. When I have seen this the packet loss Brandon mentioned seeing and the resulting performance you are seeing all matches.

Hopefully Brandon or a colleague can get things moving soon, I've indicated I'm happy to provide more info on the issues I've seen elsewhere by PM to help resolve this.

Hi I'm still waiting on a response from OR. All the BNGs confirm with SIN506. When we first discussed these issue with OR it was believed that it was because the burst sizes in SIN506 didn't work with all their GPON vendors, and they published new updated guidelines in February. We obviously updated our settings then. Unfortunately that doesn't appear to be the only issue.

When we have run tests with them, they report they don't see any discards on their network. If we were bursting too fast they would expect to see discards.

The data we took from the site visits appears to show packet loss between us handing traffic over to the Cable-link and the traffic arrive out of the ONT to our test server.

Thanks Brandon, most of the TLA's there needed a Google search, but I sort of understand what you are saying. So, from your perspective you 'seem' to be saying packet loss is the culprit, and it is between the 'Cable Link' (exchange?) and the ONT, so that is the last part of the journey I guess. What I don't see is how that part has suddenly and coincidentally gone bad due to the GEA migration (especially as many people are affected rather than just me).

I assume that OR own this bit of the circuit, but that doesn't tie up with seeing FAST.COM tests at full speed over PPPOE at my ONT, and I suspect that OR may take a similar stance. So, what will the way forwards be? I can wait seemingly forever for a 'fix' or not.... and then maybe cut across to a different (Non-GEA) isp, or ask that I am switched back to how things were from Zen - when things worked just fine?...

I would appreciate a real positive way forwards as this seems stalled.

Has anybody actually been fixed of this problem? I am wondering as the 30 days OFCOM gives to isp's ends very shortly for me, and I am considering what to do.

I don't see any real progress towards a fix, and wonder if I should start to find an alternate ISP as ongoing, the approx 25Mbits I currently see is simply nowhere near the 500Mbps advertised for my package.

Currently considering Shell Broadband, wondering if anyone has any experience of them? Or any other ideas of course?....

I would really appreciate my connection speed back too.

Is there any way we can escalate this further?

Since I'm out of contract I could move to another provider but Zen's whole setup works very well for me (or it did from joining until this migration issue) and nothing else would really fit my requirements as well as Zen, unless I spend more than Zen prices, which are already higher than the big providers (not really something that's on the table with the cost of living situation).

Edited by FakeJake (Tue 26-Apr-22 11:43:40)

In reply to a post by SteveBushell999:

Has anybody actually been fixed of this problem? I am wondering as the 30 days OFCOM gives to isp's ends very shortly for me, and I am considering what to do.

Hi Steve

I stepped away from commenting in this thread on 7th April as it was clear then that this major issue was going to drag on for some time and there is little appetite here (not you) to apply pressure to Zen to come up with an interim solution in the mean time.

In Brandon's last post there is an impression that this issue is within the Openreach network but even a layman knows that its extremely unlikely to be, this issue was introduced when the GEA migration occurred and other ISP's using the same Openreach infrastructure with their own cable-links are not having the same issue.

In reply to a post by hazey_flakes:

The data we took from the site visits appears to show packet loss between us handing traffic over to the Cable-link and the traffic arrive out of the ONT to our test server.

When you have Brandon saying below it doesn't sound positive, if that was you or I we would be on the blower every day and would escalate the issue all the way to the top as a priority but from what I am seeing they are siting on their hands hoping Openreach will tell them what they are doing wrong.

In reply to a post by hazey_flakes:

I'm still waiting on a response from OR.

I would recommend to everyone think seriously about staying with Zen if this is the level of service you get from them when a major issue occurs.

Hi Dect,

I fear that this is going to end in tears, it just doesn't feel like anything is happening, I totally agree with your comments about awaiting an OR response. To some extent I don't really care 'where' the problem lies, other than an overall technical interest, I just want things to work like they did before the UNASKED FOR migration.

I think other people may have different reasons to need Zen to fix, but for my part, it's the support that I used to get from Zen before I moved here that keeps me, but it seems that isn't happening now (PLEASE 'surprise me'). Other than that, I simply need bandwidth and fixed IP, those can be had from other suppliers more cheaply it seems, which is why the Shell offer of £49.99 a month for 900Mb down seems reasonable (plus £15 one off payment for fixed IP) - Still a saving over Zen.

I fully agree that as this issue affects multiple clients that there doesn't seem to be the level of urgency I would hope for or expect. I don't WANT to leave, but I am seriously unimpressed that after a month I am still left with a really BAD connection, averaging just 5% of what I pay for.

Edited by SteveBushell999 (Tue 26-Apr-22 12:42:09)

Are you working an open support case regarding this? Not from the perspective of getting it fixed quicker, but if you want to exit a contract or need to escalate further then Zen are unlikely to consider posting in here to be an approved way of raising issues to them.

Yes ! See below from Zen (dated 30 March) - NB fault first raised 23rd March, and my 'migration' was 21st March

Thanks for contacting Zen Internet regarding your performance issues.

Our Technical Support team have now identified that there is a performance issue with the connection continuously performing below the 'Guaranteed Minimum Download Speed' we provided in the confirmation email we sent to you.

In line with OFCOM guidelines Zen now have 30 days to investigate and resolve the performance issue. If the issue can't be resolved within the 30-day period, you may terminate our agreement without penalty but the charges for your use of the services up to the point of termination won't be refunded.

The issue has now been passed to our Fault Management team who endeavour to contact you within the next working hour to discuss the details and provide some initial timescales for the fault progression.

Please accept our sincere apologies for the issue and rest assured that Zen will now be working to resolve the problem.

Kind Regards,
Zen Technical Support

Edited by SteveBushell999 (Tue 26-Apr-22 14:23:08)

In reply to a post by SteveBushell999:

In line with OFCOM guidelines Zen now have 30 days to investigate and resolve the performance issue. If the issue can't be resolved within the 30-day period, you may terminate our agreement without penalty but the charges for your use of the services up to the point of termination won't be refunded.

So with consistently severe throughput speed for over a month they reckon no compensation is due? Hmmmm.

To paraphrase: "You don't get what you pay for".

In reply to a post by pluralist:

So with consistently severe throughput speed for over a month they reckon no compensation is due? Hmmmm.

If compensation was due this fault would have never dragged on this long. Its a [censored] show from Zen and most of their customers I would expect would prefer what they paid for rather than this [censored] service and a month plus of compensation payments.

Compensation is over and above a refund for the service, I think Zen would find it very difficult to argue that they don't owe you a refund for the month of disruption if you wanted to pursue a claim.

This is getting well beyond a joke. I know that Zen people read this (Hi Brandon) surely it's obvious that we are all well and truly [CENSORED] off. This has gone on beyond long enough, it's obvious that there is no intention to simply switch us back to what used to work just fine, and that Zen no longer give a damn, but instead just blame OR hoping that it's their fault (which it clearly isn't). I USED to be a big fan of Zen, and have recommended them in the past, but the total lack of any progress on this issue is speaking very loudly to me. For me, it's deadline Friday - the 30 days OFCOM give, and then I'm off - it wouldn't be so bad if we were kept in the loop as to what is happening, but the total lack of communication suggests maybe we ARE in the loop, and NOTHING is happening?

Refunds, compensation, whatever, I just want service, that I pay for (still do!) nothing more or less. On this issue, unless something changes tout de suite, it's not possible to recommend them - put simply, they have unilaterally ruined my connection, and done nothing about righting the obviously untested action (evidence being the number of people [CENSORED] off). My job is a software tester, and it's as clear as day that this has NOT been adequately tested, and we are just guinea pigs, unpaid UAT testers, who aren't even able to attend a defect triage...... MUSHROOMS - (kept in the dark, and fed on bull waste products).

Please surprise me Zen...... Last chance ! I am happy to delete this post if things are resolved in a timely manner. We don't pay nearly £700 a year each for this kind of 'service' !

Edited by SteveBushell999 (Tue 26-Apr-22 19:16:09)

I just can't understand why they don't just switch you back to keep you happy whilst they investigate, or if they need a real world connection that's experiencing the issue, so they can eventually prove they've fixed it, agree with you to keep it as is in exchange for offering you a very sweet deal (free service, refund + compensation).

I know a forum isn't an official mechanism for support, but this is now out there and it isn't exactly giving them a glowing review is it.

I actually thought it was a really good show recently to see them responding on here as it showed a real improvement to them caring about the level of service they offer, but obviously not. I sent them an official ticket months ago about the gateway issue and never ever received a reply.

Edited by zebb_edi (Wed 27-Apr-22 09:07:37)

Chalk this up to bad experience! Possibly seek to get your compensation, but put your energy into moving forward with a different provider.

There are plenty of other (very) decent providers over Openreach FTTP. Good luck.

I think that's sound. I am doing research on moving, in case no resolution by Friday, was looking at ShellEnergy, but they seem to get terrible reviews, so instead currently looking at Cuckoo, they offer 1Gb and can supply fixed ip for less than I pay for 500Mb, and get good reviews. BUT.... It's still not too late Zen, please at the very least contact us and say what's really going on? Saying we are waiting on OR is just not enough.

In reply to a post by zebb_edi:

I just can't understand why they don't just switch you back to keep you happy whilst they investigate, or if they need a real world connection that's experiencing the issue, so they can eventually prove they've fixed it, agree with you to keep it as is in exchange for offering you a very sweet deal (free service, refund + compensation).

Any half decent company would have at least one ONT setup in their office for each of the pre and post GEA migration setups, this would have allowed them to fully test throughput before going live but I think Zen have been caught with their trousers around their ankles as this change has clearly not been fully tested before being put live.

This issue I strongly suspect is a Zen wide issue for GEA migrated customers and possibly new customers as well, sadly novice Zen users won't even know they have got this issue and I suspect Zen won't be actively telling customers about it either.

I've been GEA migrated and so has my neighbour and our connections appear to be working better than ever. My latency on speedtest.net on the london gateway dropped from ~17ms to 8ms.

I have also noticed that my xbox which used to only hit a maximum of 550Mbps now goes to the full 900Mbps. This started occurring around the date I was migrated so I'm fairly sure it was that rather than an azure/ms change.

In reply to a post by zebb_edi:

I've been GEA migrated and so has my neighbour and our connections appear to be working better than ever. My latency on speedtest.net on the london gateway dropped from ~17ms to 8ms.

I have also noticed that my xbox which used to only hit a maximum of 550Mbps now goes to the full 900Mbps. This started occurring around the date I was migrated so I'm fairly sure it was that rather than an azure/ms change.

This is interesting

Here are the BQM's from both mine and my neighbours connections on the day we were migrated.

https://www.thinkbroadband.com/broadband/monitoring/...

https://www.thinkbroadband.com/broadband/monitoring/...

Wish I could say the same here. Migration happened overnight 21 March, I noticed next day (It's kindof OBVIOUS!) Complained 23rd, Sent first one then another router, no better, got OFCOM email 30th, had Zen perform tests here 6th April, I heard from them a couple of days later saying they were 'looking in to it', since then only communication has been via this forum - ie next to nothing, and no actual update.

That's why I am so unhappy !

This was 13th April, sorry I forgot.... really reassuring that after 14 days they have no conclusion !!!
Good morning Steve,

I’m still analysing the data we gathered from the testing. I have no immediate conclusions, but I have a couple of areas to investigate further.
Sorry to be a bit vague, but I need to be sure that what I report is factually correct, so I’ll be getting further input from our own engineers, and quite likely from Openreach.
The tests do show packet loss on the UDP tests, and poor performance on the TCP tests. I’m analysing the TCP data in more depth as that does have some areas of concern for me.

Bran.

I am starting to wish that Zen would say they won't do anything and I can leave now, that way at least I could get another supplier on the case now rather than Monday.

Edited by SteveBushell999 (Wed 27-Apr-22 12:39:40)

In reply to a post by SteveBushell999:

I think that's sound. I am doing research on moving, in case no resolution by Friday, was looking at ShellEnergy, but they seem to get terrible reviews, so instead currently looking at Cuckoo, they offer 1Gb and can supply fixed ip for less than I pay for 500Mb, and get good reviews. BUT.... It's still not too late Zen, please at the very least contact us and say what's really going on? Saying we are waiting on OR is just not enough.

Re ShellEnergy; this article at ISPreview was just posted up after your reply…

https://www.ispreview.co.uk/index.php/2022/04/shell-...

Yeah, saw quite a few like that! So looking at Cuckoo now

Well...... At least I (at last) have a response from Zen, Apparently there is a high level meeting in the morning, and I will hear what is happening (or at least a response to my concerns) after that. I will let you all know what the outcome is when I hear, but for now I am giving Zen the right to an official reply. I am just happy that it is at least escalated to board level now.

I do hope you get some sort of an acceptable solution and goodwill from Zen. Good luck.

Alway good to get an update, even if it's via proxy 🙂

In reply to a post by SteveBushell999:

Apparently there is a high level meeting in the morning, and I will hear what is happening (or at least a response to my concerns) after that.

Really hope a clear action plan comes out of this meeting to get this major issue resolved. The meeting sounds positive but after the talking stops progress needs to start.

Well looks like there's now an article on ispreview about it.

https://www.ispreview.co.uk/index.php/2022/04/isp-ze...

I still seem to be one of the "Small Minority" - or maybe Zen have forgotten to tell me I need to report this again.... Still waiting to hear from them after big meeting they had this morning..... Oh, and it's afternoon now.....

Is everyone else (ANYONE) actually fixed now?

I was called earlier and asked to email through some more speed test results and also asked for my availability for engineer visits.

That's all I know at the mo.

No communications to me though ! Zero, zip, nix, nought, nada....SO far..... Apart from a promise to contact me after the big meeting this morning.....

Perhaps they don't understand the problem with your connection, but do mine, and they can't actually fix it?????

Edited by SteveBushell999 (Thu 28-Apr-22 12:57:11)

In reply to a post by SteveBushell999:

Well...... At least I (at last) have a response from Zen, Apparently there is a high level meeting in the morning, and I will hear what is happening (or at least a response to my concerns) after that. I will let you all know what the outcome is when I hear, but for now I am giving Zen the right to an official reply. I am just happy that it is at least escalated to board level now.

It's probably not a coincidence that this is being escalated the same day that Mark @ ISPReview contacted them for comment on the issue, which is a shame.

Hi Steve, I've just sent you an email.

Anyone else that is still experiencing issues, please feel free to drop me a message with your details and I'll make sure your case is flagged.

In reply to a post by ajays:

Anyone else that is still experiencing issues, please feel free to drop me a message with your details and I'll make sure your case is flagged.

Your absence over the last month from this thread has been extremely noticeable, your customers have been desperate for info on what is happening and you are no where to be found then suddenly todays claim its been fixed you're front and centre again.

Where is Zen Customer Service when their customers need them. I'm sure people will remember this.

In reply to a post by dect:

Where is Zen Customer Service when their customers need them. I'm sure people will remember this.

I will definitely be leaving Zen when my contract with them ends. I do not have a problem with my connection at the moment, but their customer service has gone completely downhill.

OK, so pretty well a full day with about 4 or 5 different Zen people, going over exactly the same ground that I originally reported, and with exactly the same results.....

I pay for 500 down plan

I have just been on the phone to ‘Chris’ from Zen for 45 minutes, and he has all the data on the various tests, but in summary:

Using PPPOE direct to the Mac 500 down / 75 up – this was with constant cabling to downstairs, all that is eliminated, and all cables Cat5e
Apple router 220 down / 50-75 up – this will be the TimeCapsule hardware limit – look on the web…..
Fritz about 19 down / 20 up – as a LAN based test, WAN based (AVM/Fritz diagnostic) shows 500/75

I KNOW this all sounds like it is the router, BUT…. Have tried 2 Fritz boxes, and 2 firmwares, reset to factory defaults, and all are the same, as it the Technicolor exactly the same

I know this all changed over the night 20-21 March, which is ‘coincidentally’ when the GEA migration happened, it has been constantly bad ever since. There were no router upgrades any time near then, and no other hardware changed overnight here.

It is crystal clear that as far as PPPOE into the ONT is concerned that everything works fine, so I really doubt Openreach will play ball.

ANYONE would say ‘it’s the router’, but honestly? 3 different ones?

Is the NAT encapsulation different in the TimeCapsule? - or what is ‘different’ when I use that?

WHY oh WHY can’t we just go back to how it was before it was messed up? The OBVIOUS change, get rid of GEA? OK, if it isn’t that, then we have other things to look at I guess?

Edited by SteveBushell999 (Thu 28-Apr-22 16:58:22)

If your issue is the router (which has been replaced at least once) surely that would have been easily spotted by the Zen network engineer when he/she visited on 11th April? or am I expecting too much?

In reply to a post by dect:

If your issue is the router (which has been replaced at least once) surely that would have been easily spotted by the Zen network engineer when he/she visited on 11th April? or am I expecting too much?

When they came, they tested into ONT, and I have tried 3 routers tried with different firmware revisions.... It's not the router!!!

In reply to a post by dect:

In reply to a post by ajays:

Anyone else that is still experiencing issues, please feel free to drop me a message with your details and I'll make sure your case is flagged.

Your absence over the last month from this thread has been extremely noticeable, your customers have been desperate for info on what is happening and you are no where to be found then suddenly todays claim its been fixed you're front and centre again.

Where is Zen Customer Service when their customers need them. I'm sure people will remember this.

To be fair to Zen, despite this not being a Zen support channel both ajays and hazey_flakes from Zen have posted in this thread over the last month. It seems like there has been ongoing contact between Zen and customers away from the forum. It is probably right that Zen's support team deal direct with customers to fix issues rather than on a public forum.

What is interesting is that you can actually get the full bandwidth when directly connected to the ONT via the Mac.

I hadn’t appreciated that before you mentioned it today.

The Apple router is simply too old, not enough grunt. So discount using that box completely. That’s previously been acknowledged as deficient with PPPoE throughput on high speed broadband connections.

Then your left with the Frtizbox and Techicolour which are exhibiting poor throughput over the LAN, but reporting full bandwidth at least WAN-side for the Fritz - so definitely not all faulty hardware. Without diagnostics on the router it’s hard to say where the fault lies - firewall/NAT or PPPoE weirdness.

In reply to a post by JakeDog:

To be fair to Zen, despite this not being a Zen support channel both ajays and hazey_flakes from Zen have posted in this thread over the last month. It seems like there has been ongoing contact between Zen and customers away from the forum. It is probably right that Zen's support team deal direct with customers to fix issues rather than on a public forum.

Ajays last post in this thread before today was on 29th Mar (its 28th April today !!) when I asked him a question that he never replied too and I agree hazey_flakes has made several posts but my post was about ajays going AWOL not hazey_flakes.

Apologies, I wasn't ignoring you, I just hadn't spotted your question.

I hadn't posted on this thread as my technical collogues had picked up the issue.

Please always feel free to contact me with a PM if you need an answer to a question that I've not spotted.

In response to your earlier question, it all depends on what's changing. For a change of backhaul we typically wouldn't send individual notifications.

This is definitely weird and not the behaviour I've seen on my connection.

Mine is slow no matter what's connected to the ONT.

Update: there's been a flurry of activity today, I presume following this meeting.

Both ajays and hazey_flakes have been in touch directly. A change was made to my connection but this appears to not have made things any better.

Test results were sent back to them.

I've been offered a 100% refund on my service until the issue is resolved, and a Samknows Whitebox so that the connection can be continuously monitored whilst Zen make changes to resolve the issue. I think this is fair enough for now (and I presume backdated to when the issue started).

The fault team told me that they were "re-opening" my case as it had been closed due to inactivity. I'm not sure why this happened as I never hinted that the issue was resolved. If I missed an email somewhere, I was never chased for a reply nor warned/notified that my case was being closed. I'm actually more disappointed in that than the technical issue I think as it seems such an easy thing to get right. This may explain why I haven't been hearing much.
Definitely a "lessons learnt" for Zen I think.

Seems like everyone is happy then, Zen suddenly announce at midday today on ISPReview that the issue is resolved, Steve has still got an issue and so have you (your ticket was conveniently closed) and both of you participated in the on site testing so should have been used to test and confirm the fix. All swept under the carpet today after a high level meeting at Zen, just perfect.

Edited by deleted (Thu 28-Apr-22 22:45:37)

Yes, that's why it's all so confusing. The Apple router actually works fine (within the limitations of what the ancient hardware can do - maxing out around 200Mbits, so better than nothing for now.

Two different Fritz's (and also a firmware update, so almost like 3 different ones), and the Technicolor all the same, and BAD. I do have my suspicions re: NAT shaping, or something like that, but it also disconnects really badly as well.

People rightly moaning about lack of support for nearly a month, but they do seem to have woken up now, so cooperating for now, however my patience isn't infinite, and I don't want to turn into an unpaid UAT tester for Zen (I do software performance testing as a job, and really, testing stuff for fun gets somewhat boring!)

Let's stay positive, and see what the day brings, Hazy Flakes asking me to run some more tests today, which I will do after my usual round of Teams meetings this morning

I hear you. Sometimes however it’s easier to go with the path of least resistance….

If it was me: I’d be using my own capable router, and put the ISP supplied garbage (let’s not mince words) in the cupboard.

If your still not getting te full whack then beat them up over it for sure. Otherwise I’m not sure I could stand being their continued guinea pig whilst they fault find whatever is holding things up with their own CPE.

In reply to a post by Pheasant:

If it was me: I’d be using my own capable router, and put the ISP supplied garbage (let’s not mince words) in the cupboard.

I don't think the Fritz is that bad actually, it's worked totally fine for a year, and things only bad after the migration, so it's just not the router! I have a mesh set up, and the router was OK. So, the Apple one for now.....

I agree. At least things seem to be moving now.

I've forwarded my concerns regarding the support situation to them as it just seems like a weird operational misfire to close tickets without trying to at least confirm with the end-user.

And I think the comments that suggest that Zen have unilaterally declared that the issue is fixed aren't exactly fair. They've said it appears to be fixed for the vast majority of users, and we have had confirmation on this thread that it's definitely not a universal issue.

Whilst this information doesn't help me or Steve, I don't think it's fair for us forum users to assume that everyone on Zen is currently suffering either. At the end of the day there's no way we can know who's suffering with this issue unless they post on here, or Zen spill the details (I wouldn't expect them to go further than they already have on ISPReview). All we currently know is that it's me and Steve. We can guess that other Zen FTTP customers on our respective exchanges may also be having the issue, but this is just a guess.

The ISP supplied router is usually the first thing I replace but the fritzbox they supply is actually decent and I've been happily using it since I joined Zen. My only complaint is that it isn't WiFi 6 capable, so the usable bandwidth via WiFi tops out at around 500ish Mbps.

Perhaps I should ask for a free upgrade when they bring out a newer replacement 😉

In my case the speed is still low no matter what's connected to the ONT so I'm not blaming the fritzbox at all. It's not missed a beat until this migration so it's clearly capable and not at fault.

I even use the built in DECT base station with Sipgate and a gigaset handset so that I have a free local phone number, just in case I ever need it. It seems to work perfectly and was easy to set up. Not many routers i know of can do this, though I guess it's probably going to get more common with the move towards digital voice services as standard.

I only have the 500Mb package at the moment, so the WiFi limit isn't an issue, but I do have mesh repeaters, so get good speeds all over the house and garden. And in any case, I really avoid Wifi as it's not good enough for work, it's ok on the phone / tablet, and when people come to stay

Certainly differences with PPOE between us it seems, so maybe there is hope of a fix, if not a common one.

I use the same setup of dect / sipgate as you, yes, it's a handy alternative to the mobile, I put £20 of credit on Sipgate, and have barely used a few pence in the last year.

Just running Brandons PPPOE tests, but all looks fine, will publish results on here when complete.... Here they are !

Time of completion
Download speed
Upload Speed
Loaded latency
Unloaded Latency
Notes
10:16:39
510
59
294
10
All test to speedtest.net with default settings
10:18:17
510
69
263
11

10:22:01
520
65
286
10

10:27:07
600
66
228
10

10:30:36
570
67
241
11

10:32:42
620
64
213
11

10:34:47
600
68
224
10

10:37:15
610
67
176
10

10:39:32
660
63
163
10

10:45:25
560
67
156
10

10:47:07
560
70
174
10

10:50:14
230
69
23
11
Apple router

In reply to a post by FakeJake:

I think the comments that suggest that Zen have unilaterally declared that the issue is fixed aren't exactly fair. They've said it appears to be fixed for the vast majority of users, and we have had confirmation on this thread that it's definitely not a universal issue.

Whilst this information doesn't help me or Steve, I don't think it's fair for us forum users to assume that everyone on Zen is currently suffering either. At the end of the day there's no way we can know who's suffering with this issue unless they post on here, or Zen spill the details (I wouldn't expect them to go further than they already have on ISPReview). All we currently know is that it's me and Steve. We can guess that other Zen FTTP customers on our respective exchanges may also be having the issue, but this is just a guess.

Some people have a short memory about the lack of progress over the last month but thats fine, really happy for Steve as he has been the person that kept this thread going during that time and his possibly one of the main reasons (+ Mark at ISPReview) why the recent push from Zen to resolve the issue. All the best guys.

Steve gets full speed when connected directly to the ONT, so his issue artists to be something different. He may have kept the thread alive but his issue looks like it has little to do with the thread.

If you get full speed connected to the ONT, to me that means it's the router, or the cable between the ONT and the router.
A change of GEA backhaul shouldn't make any difference to how an individual router performs.

I've no idea how Zen can proceed with such a fault having already sent a 2nd Fritzbox.

In reply to a post by dect:

Seems like everyone is happy then, Zen suddenly announce at midday today on ISPReview that the issue is resolved, Steve has still got an issue and so have you (your ticket was conveniently closed) and both of you participated in the on site testing so should have been used to test and confirm the fix. All swept under the carpet today after a high level meeting at Zen, just perfect.

ISP Review journalist needs to actually investigate instead of taking ISP statements as gospel? That dude still thinks Leicester has coverage by 3 FTTP providers lol.

In reply to a post by j0hn83:

I've no idea how Zen can proceed with such a fault having already sent a 2nd Fritzbox.

AND a different router tested as well (Technicolor, also known good, and supplied by Zen), as well as Zen observing speed reducing from 500 to about 8 Mbps on their own test runs runs a few weeks ago... Of course it's not THAT simple! J0hn !

(Below results of Zen's runs, connected directly to the ONT using PPPOE, but with an intermediate run testing UDP latency when they visited, all tests only minutes apart)

brandon@PerfSonarColo:/home/engineer# iperf3 -c 82.68.31.104 -Z
Connecting to host 82.68.31.104, port 5201
[ 5] local 51.148.36.227 port 44992 connected to 82.68.31.104 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 75.8 MBytes 636 Mbits/sec 0 32.0 MBytes
[ 5] 1.00-2.00 sec 60.0 MBytes 503 Mbits/sec 0 32.0 MBytes
[ 5] 2.00-3.00 sec 61.2 MBytes 514 Mbits/sec 0 32.0 MBytes
[ 5] 3.00-4.00 sec 61.2 MBytes 514 Mbits/sec 0 32.0 MBytes
[ 5] 4.00-5.00 sec 60.0 MBytes 503 Mbits/sec 0 32.0 MBytes
[ 5] 5.00-6.00 sec 61.2 MBytes 514 Mbits/sec 0 32.0 MBytes
[ 5] 6.00-7.00 sec 60.0 MBytes 503 Mbits/sec 0 32.0 MBytes
[ 5] 7.00-8.00 sec 61.2 MBytes 514 Mbits/sec 0 32.0 MBytes
[ 5] 8.00-9.00 sec 61.2 MBytes 514 Mbits/sec 0 32.0 MBytes
[ 5] 9.00-10.00 sec 60.0 MBytes 503 Mbits/sec 0 32.0 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 622 MBytes 522 Mbits/sec 0 sender
[ 5] 0.00-10.26 sec 617 MBytes 504 Mbits/sec receiver
iperf Done.


But when we reran the same test a short while later the throughput dropped to ~ 8Mbps;

brandon@PerfSonarColo:/home/engineer# iperf3 -c 82.68.31.104 -Z
Connecting to host 82.68.31.104, port 5201
[ 5] local 51.148.36.227 port 45020 connected to 82.68.31.104 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.89 MBytes 15.8 Mbits/sec 5 19.7 KBytes
[ 5] 1.00-2.00 sec 949 KBytes 7.78 Mbits/sec 7 11.2 KBytes
[ 5] 2.00-3.00 sec 1.30 MBytes 10.9 Mbits/sec 0 33.8 KBytes
[ 5] 3.00-4.00 sec 949 KBytes 7.78 Mbits/sec 5 7.03 KBytes
[ 5] 4.00-5.00 sec 570 KBytes 4.67 Mbits/sec 6 8.44 KBytes
[ 5] 5.00-6.00 sec 570 KBytes 4.67 Mbits/sec 5 14.1 KBytes
[ 5] 6.00-7.00 sec 759 KBytes 6.22 Mbits/sec 3 21.1 KBytes
[ 5] 7.00-8.00 sec 1.11 MBytes 9.33 Mbits/sec 3 23.9 KBytes
[ 5] 8.00-9.00 sec 949 KBytes 7.78 Mbits/sec 5 19.7 KBytes
[ 5] 9.00-10.00 sec 1.48 MBytes 12.4 Mbits/sec 1 35.2 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.4 MBytes 8.74 Mbits/sec 40 sender
[ 5] 0.00-10.02 sec 10.0 MBytes 8.39 Mbits/sec receiver

My previous post shows what happens when I retest over a 30 minute period, and it is consistently good (as far as PPPOE into ONT is concerned), but obviously different to what Zen see !!!

The problem is complex! Why the sudden drop? and NOT simply a router change will fix it (so it seems), perhaps NAT related, perhaps UDP, none of us know, but quite simply, NOT that simple! Sorry!

Oh, and by pure coincidence,,,, Everything was stable and working for a year..... and 'Magically' all went wrong (and STAYED wrong) from the exact same night that GEA migration occurred..... Coincidence??? I rather think NOT!

Edited by SteveBushell999 (Fri 29-Apr-22 23:31:26)

It can't be NAT related. The router handles NAT. Nothing beyond the router even see's NAT, it cannot impact speed.

Steve - did they run any longer iperf tests? These are just short 10 second tests.

Post deleted by dect

Edited by deleted (Sat 30-Apr-22 16:53:19)

That's all the info that I have from them, but there is a crystal clear difference between the two sessions isn't there? My tests weren't iperf (although I have offered the services of my server running it to Zen).

Zen are trying another OpenReach change on Tuesday, and coming with a bunch of routers on Wednesday. Can't see how the routers will fix the behaviour that Zen have captured, but I am all ears!

Regarding 'it can't be NAT' which someone published, see the capabilities of Fortinet / Cisco / BlueCoat / Juniper / Palo Alto. They all can do deep packet inspection, and specifically throttle NAT traffic - Just Google "NAT traffic shaping". In addition, not all NAT's are equal (there are quite a few ways of bundling things up - (Wiki Network Address Translation for more details) - It could be that Apple wrap things up differently perhaps?, they DO tend to go their own way.... I am NOT saying that's the problem, merely that it could be, and clearly that it's NOT a simple problem, and to be honest nothing makes any real sense at the moment, and I an not green, I have worked in IT for over 40 years!

At least things are being done - or at least tried, which I am grateful for, and maybe, one of the approaches may guide us towards the actual problem. I remain sceptical that the GEA migration is not in some way to blame, as it was just hours after that was applied (when I got up, still unaware that anything had been done) that I noticed the glaringly obvious sudden impact on performance.

Edited by SteveBushell999 (Sat 30-Apr-22 12:44:47)

I'll be receiving a Zen Internet parcel from DPD tomorrow.

I guess this will be the whitebox.

Edit: it is the whitebox. Further tests have been done and a call between Zen and Openreach is scheduled today to try and progress further.

Edited by FakeJake (Tue 03-May-22 15:06:06)

Well....... I've just had Zen here for 4 1/2 hours, and in a nutshell:

1) They connect windows laptop over PPPOE - it's rubbish - maybe 100Mbps, but dropping packets like crazy
2) They try Technicolor router and Fritz 7300AX router, exactly the same as I see, 120Mbps, and dropping when downloading anything big
3) They confirm connecting my Mac mini over PPPOE it works fine, full speed, no drops.
4) They confirm my observations using the Apple TimeCapsule router, gives 220Mbps, and looks reasonably OK

Teleconference with Zen and Openreach, there is no conclusion, apart from saying they will push through migrating me back to pre GEA config - but WHY? What's wrong?

Essentially, the operation and speed of Mac devices is WAY different to Fritz, Technicolor, and a Windows laptop..... Go figure !!!

Well. That’s that then eh….

You just need to have a network made up exclusively of Apple computers - even for routing duties it would seem!!

What a coup for Apple. What a disaster for Zen.

In reply to a post by Pheasant:

Well. That’s that then eh….

You just need to have a network made up exclusively of Apple computers - even for routing duties it would seem!!

What a coup for Apple. What a disaster for Zen.

It does rather seem that way, and I DID consider that very possibility, but Apple internet sharing doesn't allow for port forwarding, and I need that for mail server, and Plex, and SERVARR etc etc Of course it's 'possible' but a faff, and I don't have a Mac with 2 network ports any more, and the USB3 network adapter I have doesn't support anything later than about El Capitan....

By the sound of things, OpenReach are as mystified as are Zen, but there was one little nugget..... I don't know if I heard right, but a suggestion that IPV6 works, and it's IPV4 that gets clobbered.... Perhaps I need to investigate that.....

In reply to a post by SteveBushell999:

I don't know if I heard right, but a suggestion that IPV6 works, and it's IPV4 that gets clobbered.... Perhaps I need to investigate that.....

No, that was just me trying to understand how the speedtest on the fritzbox was getting full speeds, when no other test does. So my on own fritzbox it give the url http://avm.de/service/zack-der-speedtest-fuer-ihre-b...

running that speed test gave full speeds on the test on the newer AX fritzbox that out engineer tried. But no other speed test did. When I packet captured a test on my home connection to see whether that was a UDP or TCP test (as UDP we generally can get to line rate), I saw that it was TCP, but also running IPv6. But your connection doesn't have V6, so that is not a reason for the test at your home to run full speed. Should have been obvious that it would run over V6 where its available as AVM have always been good at supporting it.

Ah, thanks for the clarity Brandon, so not worth trying to reconfigure everything here to IPV6, which would be a pain in any case! Especially as I need the IPV4 DNS entry to point to my domain anyway !

OK, well, it was a straw floating by, and asking to be clutched at !

I am as eager as anyone here to find out what will make it work, and the whole Apple versus the rest of the world is clearly interesting to say the least! Perhaps if I make my Mac dual boot via Bootcamp, so the same hardware is involved, but it runs Windows, might that give us any more info?

Edited by SteveBushell999 (Tue 03-May-22 18:05:57)

In reply to a post by SteveBushell999:

I am as eager as anyone here to find out what will make it work, and the whole Apple versus the rest of the world is clearly interesting to say the least! Perhaps if I make my Mac dual boot via Bootcamp, so the same hardware is involved, but it runs Windows, might that give us any more info?

It might tell us if the issue is hardware or software. In our conversations one of our Tech Support Engineers mentioned that in the early days of testing FTTP we had an issue with slow speeds on windows that was not there when running a a live boot of Ubuntu on the same PC. I'm assuming that was difference in the TCP stack, though still trying to find any documents they have from the time to see what the underlying cause was.
I'll email you later with a update of where we are, and what our next steps are. Just got to get my boy sorted for bed first.

Yes, the TCP stack thing came to my mind too, guess it will eliminate that..... I do have a spare Mac mini here, will look at dual booting prospects for that later tonight, I think it has two hard drives in it, so may be simple. I do have to say that almost certainly the Fritz and Technicolor routers are likely to be running some port of Linux, but I will look at the possibility of dual boot Mac anyway.

I actually think that my confusion with all this is being exonerated! But I simply need a solution. The other very worrying thing is that simply getting a result on Fast.com is no reliable indicator of real world large file transfer speeds!

Good luck with getting your son to bed, my days of all that are long gone... More's the pity !

Edited by SteveBushell999 (Tue 03-May-22 18:54:44)

The other very worrying thing is that simply getting a result on Fast.com is no reliable indicator of real world large file transfer speeds!

Very unreliable Steve - you may as well use a random number generator 😂

To be clear…. All my statements (in the past) that ‘it’s fine on PPPOE’ actually mean ‘It’s fine on PPPOE ON A MAC’ - and that (it seems) is somewhat significant, as it’s no indication that it’s fine on anything else? This is a Rabbit Warren ! And each hole leads into a mire, full of tar and treacle. I can’t stop thinking that it was all FINE before GEA migration… MAYBE that is a coincidence, but it’s one hell of one.

In reply to a post by hazey_flakes:

In reply to a post by SteveBushell999:

I am as eager as anyone here to find out what will make it work, and the whole Apple versus the rest of the world is clearly interesting to say the least! Perhaps if I make my Mac dual boot via Bootcamp, so the same hardware is involved, but it runs Windows, might that give us any more info?

It might tell us if the issue is hardware or software. In our conversations one of our Tech Support Engineers mentioned that in the early days of testing FTTP we had an issue with slow speeds on windows that was not there when running a a live boot of Ubuntu on the same PC. I'm assuming that was difference in the TCP stack, though still trying to find any documents they have from the time to see what the underlying cause was.
I'll email you later with a update of where we are, and what our next steps are. Just got to get my boy sorted for bed first.

There is differences in a few ways between Windows and Apple's OS, the auto tuning aspect of the receive window, the congestion window configuration and the implementation of cheats to get by slow start. The latter doesnt conform to RFC but is practice now commonly used by the big vendors. I think there is a lot of similarities to BSD's network stack on Apple's OS if I remember correctly but I am not a Apple OS user so have no actual direct experience of it.

Well, this is getting extremely interesting, and I think we are getting quite near to both a solution and an understanding.

My connection to the ONT is declared 'good' because my Mac sees 560 down / 70 up (the up never seems to be the issue). OpenReach therefore uninterested. However, anything else (router, Windows PC etc) sees zillions of disconnects, and reports speeds of between 15-130Mbps down. Oh, and to back up my claims on the Mac, it gives correct Fast.com speeds, as well as downloading from UseNet at believable speeds (350-500Mbps).

Of course, as soon as anything but a Mac (TimeCapsule) router gets in the middle, then everything goes awry. So, does Zen and Openreach support Macs? Or put another way, do they support Windows? !!!

There are other twists to this, around running a dual boot Mac with Bootcamp, where LANside, I see speeds of 160+Mbps on OSX compared to around 25Mbps on Windows.

I foresee the Mac versus Windows wars restarting!

And, regarding differences, OSX came from BSD, but has evolved, and so does Windows, but also has evolved. I suspect that those evolutions may well be different.

Upshot is that it seems I am to be UN-GEA migrated.... soon.

I will update

I suppose you need to really wait until your are back on BTW backhaul to see what difference that makes and whether your issues are coincidental or causal to that change.

Totally agree, that will be the final thing, but whatever that outcome, the issues around OR declaring lines OK when tested on Mac vs Windows remain don't they?

It’s been generally well known for some time that the increasing prevalence of high speed broadband connections (above 500 Mbps or so) often highlights the relative inadequacies in (especially older) networking in windows-based machines.

Linux distros and MacOS tend to have less issues with their network stack and also suffer from less complications and additional processor load as a result of how some AV software further complicate and interfere in the windows network stack. This is why many ISP support teams call for booting into say Ubuntu to take at least that variable out of the equation.

Fine.... EXCEPT for the fact that the support guy who came yesterday brought his shiny, tested, recent Windows 10 machine, and it was rubbish! Around 19Mbps - OK, so there is a 'fault' - perhaps, but it shows that even a modern Windows box is way worse (as far as network performance and real-world downloads etc) are concerned. So, is what I see on the Mac 'right'? or what was seen on Windows yesterday, or what is seen on a (presumably) Linux kernel router right? What? !!!

It's not ideal !

In reply to a post by hazey_flakes:

It might tell us if the issue is hardware or software. In our conversations one of our Tech Support Engineers mentioned that in the early days of testing FTTP we had an issue with slow speeds on windows that was not there when running a a live boot of Ubuntu on the same PC. I'm assuming that was difference in the TCP stack, though still trying to find any documents they have from the time to see what the underlying cause was.

That may be a case that I had open that the Texh Support Engineer remembers. I could get nowhere near 900 Mbps when my laptop was running Windows, but could when I booted it from a live boot of Ubumtu. I never managed to find out why this was the case though.

Do you have an intel-based Mac that you can dual boot into W10?

Edited by Pheasant (Wed 04-May-22 11:47:24)

Yes, quite a few, but they are all Parallels VM's, so not sure that would prove anything as they'd be through the Mac TCP stack.. I do have an i7 Mini, but no windows on it, but I think there is a second SSD in the box, so could put bootcamp on that.... If I can find it !

I did run some tests on an old Mac mini 2009 core 2 duo and Windows 7 via Bootcamp, but connected through the apple time capsule, those tests showed a considerable difference between OSX and Windows (Circa 150Mbps on OSX compared to 25Mbps using Windows) - I need to find the i7 one, and put on W10

Yikes !


Order Items
Order Item ID Product Price Billed to Activation date* Status
2440509 Broadband Network Migration Destination Supplier: BT Wholesale zen424315@zen £0 Steve Bushell n/a Work in progress, since 04/05/2022 10:42

Wait for KCI-2 - IN PROGRESS FOR 2 HOURS

I predict you'll be back to full speeds once migrated back.

I've got my Samknows box plugged in now. It is apparently testing away.

I've been told that Openreach want to investigate further in their own labs with Zen's equipment. This is being facilitated by both parties. Zen will continue testing in their lab too.

Apparently it looks like packet loss, smells like packet loss, feels like packet loss, but the equipment both sides doesn't report any packet loss...

Let's wait and see! But I concur, packet loss seems the near certain culprit. VERY interesting results here using Mac, even Bourne out trying a hackintosh (Windows laptop running OSX) - OSX is much better than Windows at working under these conditions. That's why I saw full speed on PPPOE with my Mac mini (I believe), and on that basis, the connection was declared good...... I am pretty confident that in reality it ISN't good (as Zen engineer saw terrible speeds on his Windows laptop on PPPOE yesterday. !

It will be very interesting to follow what happens your side.

I'm wondering if it has anything to do with the 802.1p (called pbit on Fritzbox), which is effectively the priority of the circuit.

Maybe way off, but worth looking into.

It could be, but it was the same with the Technicolor router they sent, as well as a total of 3 Fritz!boxes all allegedly correctly configured...... I am running with my TimeCapsule as a router until I get migrated back to how things were, and can see if Fritz works or not then

Thanks for trying! I will let you know the results

Maybe overnight....

Work in progress, since 04/05/2022 10:42

Wait for KCI-2 - IN PROGRESS FOR 7 HOURS

In reply to a post by SteveBushell999:

Yes, quite a few, but they are all Parallels VM's, so not sure that would prove anything as they'd be through the Mac TCP stack.. I do have an i7 Mini, but no windows on it, but I think there is a second SSD in the box, so could put bootcamp on that.... If I can find it !

I did run some tests on an old Mac mini 2009 core 2 duo and Windows 7 via Bootcamp, but connected through the apple time capsule, those tests showed a considerable difference between OSX and Windows (Circa 150Mbps on OSX compared to 25Mbps using Windows) - I need to find the i7 one, and put on W10

Need to run W natively on the apple hardware using Bootcamp. Would be interesting to see if the fault could be replicated in this scenario. But as you may be back on BTW backhaul by the morning it may be a moot point.

Need to run W natively on the apple hardware using Bootcamp

I know! I have built a windows 10 box on an i7 Mac mini, so the capability is there, but I need to work out PPPOE under Windows (I HATE Windoze!)

Think it may end up as a moot point, but my migration is now 11 hours old, and still not complete, so maybe....

My migration took 2 weeks so you might be waiting a while. Have a look on your order history and see how long yours took to complete

My migration was same day:

Order date: Monday, 21 March 2022
Order time: 03:15
Order Items
Order Item ID Product Price Billed to Activation date* Status
2423670 Zen FTTP GEA Migration zen424315@zen £0 Steve Bushell n/a Fulfilled on 21/03/2022

But migrating back? ......

Work in progress, since 04/05/2022 10:42

Wait for KCI-2 - IN PROGRESS FOR 1 DAY

Highly variable then. Maybe Openreach do them in batches?

We can but speculate.

OH DEAR !!!

Just tried my i7 BootCamp Mac running windows 10 on PPPOE..... It is showing 500 down and 70 up
The un-migration is still shown as in progress
I retried the Fritz!Box and it is still useless

My confidence has just taken a BIG dent

...Hmmm. Had a sneaking suspicion this may turn out.

ONT faults are exceptionally rare (thankfully) - but did they ever swap yours out?

Nope!

I'm just building a few router / firewall solutions based on my Mac hardware, but it's really time consuming.... Smoothewall is the latest I am trying

Perhaps not such a crazy idea to have it swapped it for a fresh one.

When Zen came to visit did they indicate if any other customers are experiencing your specific fault?

Best wait for the un-migration to be completed else you're going to tie yourself up in knots which may prove to be unnecessary.