Slow speed after GEA migration

In reply to a post by SteveBushell999:

The second line is now up and running as about midday yesterday.

I assume that the new line is on BTW and my original one is on BTW as well for now?

It’s early days, but……

I was getting solid 500Mbps from the old line, the new one was installed, and I tested it at 500Mbps as well…

Now the rather more worrying news. After an hour or two the original line dropped back to 100Mbps, and then similarly the new one.

I swapped around the ONT connections so I am now on the ‘new’ line, and it was fine (500Mbps), before dropping back to 100Mbps after a while.

I powered down both ONT’s, and then got a full 500Mbps on both lines, followed by the new line dropping back to 100Mbps after a few hours.

I have since powered off and on the new ONT, and have 500Mbps back on both connections.

I am monitoring the situation, but it ‘seems’ that both of the lines may now be unstable???

I know that the new line apparently went back to an 8 port Huawei box at the end of my cul-de-sac (if that is any help) – no idea where the existing line egresses.

Update, both running OK as at 22:50 Thursday. I updated Zen this afternoon, but both Brandon and Andy are on vacation....

In essence, it 'feels' like there is something really flakey here, but time will tell, perhaps it's all just retraining or whatever....

Steve

Could it have been the Ethernet connection to the ONT dropping down from 1G to 100Meg randomly for some reason?

Could it have been the Ethernet connection to the ONT dropping down from 1G to 100Meg randomly for some reason?

ANYTHING is possible, can't say I checked much as the first day it was all installed I was out much of the day (my birthday), but somehow it all stabilised, and we seem good now. Zen have sent out a SamKnows box for them to monitor the original (now GEA) line, that box arrives tomorrow. For now, both lines currently behaving well, but I have seen all sorts of issues in the last nearly 5 months, so anything is possible.

Jesus

This 'incredibly long thread' has been going on an 'incredibly long time' - We have ALL tried very hard to keep all interested parties informed as the situation unfolds, and I suspect that the origami is very far from over. I note that poster isn't even on Zen! Hopefully they find these shenanigans of passing amusement.....For 'us' though, it's been of very real, and impactful interest. Guess you can't please all of the people all of the time

For my part, I am finding it quite interesting that my original line re-migrated seems much better than it was back in June before I got switched back to BTW, wondering if either something else has changed, or simply disturbing the connection physically has done something? Time will tell..... (or not!)

I split this sub-thread off here:
https://forums.thinkbroadband.com/zen/t/4709246-slow...
..which is closed due to length

Post deleted by SteveBushell999

I pruned one of the August sub-threads.. you can quote from the original thread in the new one irrespective of where it was split.

seb

I don't think there should have been any real physical disturbance to the connection. You'll be on a different CBT port for the new connection. I'm not actually sure they physically move anything at the exchange when you move between GEA and BTW backhaul (certainly nothing at the fibre PON end will have moved).

It's a shame in a way that you moved off GEA originally It's possible now that perhaps it was a difference in the provision process somehow, or that they've made in the intervening time changes to the GEA setup at your exchange or elsewhere that would have fixed your original link at some point in the intervening period had you been still connected to it. Maybe they've swapped out some exchange hardware, who knows. (They've tried swapping out a bunch of SFPs at my exchange it seems in recent days).

So are you running both connections all the time, or just swapping the ethernet cable between ONTs to check each one? It seems if they wanted some really useful info they'd put each of them on the same profile as you had (maybe with separate logins) and stick a samknows box on each of them, and then be able to compare like for like at exactly the same point in time.

There certainly was physical disturbance to both fibres when the new line was put in, both at the house end and where it joins onto the 'pit' at the end of my drive. I agree, it would have been cleaner to leave me on GEA, but I actually need connectivity to work, so that route had to be followed! It is also more than possible that some exchange equipment has changed, there have been occasional outages suggesting other changes are afoot.

So are you running both connections all the time, or just swapping the ethernet cable between ONTs to check each one?

I am running both simultaneously, but the 'new' line has my main stuff on it, and the GEA (old) line has just a MacBook Pro running occasionally Fast.com's etc. I would certainly have liked to be party to Zen's testing schedule (that's my job actually) we could have had a proper plan to establish a proper baseline, and move things on one step at a time in a methodical way, but there you go, not my call. Will be just one SK Box on the old line, and I await their requirements for testing. I have two profiles, the one on the old line is Vanilla, and my real profile has my own fixed IP and domains pointing to it (to ensure business as usual), but swapping is trivially easy. I really hope that the old line can be coaxed into giving the same symptoms I saw (and so did Zen).

So, my SamKnows box is now running, and my old line has been GE Migrated back.... What does it look like? SamKnows box output Seems pretty darned stable doesn't it? So, what's different? Any comments anyone?

I wonder what the expected range of the packet loss is, and whether the non-GEA connection is any better.

I wonder if 0.0033% might still be significant for some applications. 1 in 30,000 packets going missing. Doesn't sound like a lot, but could be significant.

I have some suspicion that something may be lossy somewhere, as I have noted occasional increased packet loss on TBB graphs which seems to be gateway-dependent in severity, though all gateways appear worse for me than pre-GEA for throughput and general ping latency jitter. It may not be the gateway per se, but some relationship with exactly how each route to the gateway behaves and jives with TCP congestion mechanisms.

It is always possible that if there is a gateway relationship that you get into a more working gateway just by luck of the draw when then PPPoE session is connected. You can work out which gateway you are on by looking at the first Zen hop on a traceroute to anywhere.

If our issues are related, and what I see is the current "state of the art" for GEA, then I'm really not sure how much you could expect to see with a 500/70 connection as my connection mostly tests around the 500 mark (though it's a 900), with the exception of some select routes which will test at 900 all day long.

It's possible I guess that fixes have gone in that get up to generally around the 500 mark on the GEA system, which would appear to fix your issues, but leave me falling short with my 900/115 connection.

I also note that a symptom for me is that some limited selection of routes do test at close to line rate, which of course isn't ideal, as it is always possible that the test location that Samknows uses could be on one of those good routes...

I do now have a dedicated case manager for the issue at Zen, but I've not really heard anything useful back in the last week. I'm only about 12 days off this timing out to being a month's issue from me giving notice that I intend to exit contract if they don't get it back to how it was (per their logging system against my account) so I guess the ball is in their court to resolve.

Looks like I'm going to have my very own Samknows box sent out, will be interesting to see how one behaves on my connection. Though I'm still cautious as to exactly how you can interpret the results given questions over where the test servers route to, it will nevertheless be at least another data point.

Well, my line is totally stable now latest SamKnows output I have no real idea what has changed, but I think Brandon is back on Monday, so perhaps he can shed some light. I have a nagging feeling that somewhat like FakeJake it's 'fixed itself', which is a real worry.

Out of interest, what do the other two tabs show (the Realspeed tests and the details page?).

I'm going to be interested to see what the Samknows box makes of my line, hopefully it will arrive this week and I can make some progress with this. I've done a bit of testing with thei Samknows web based test and it seems to be middle of the road on my connection at around 700Mbps, whereas Zen's own Speedtest server rarely measures that high, and some other test servers are close to line rate.

The variability between destinations since migration (and in particular the slow speed to Zen's own test server) I find particularly confusing still.

Nothing to see really! realspeed and details I don't see much difference in results between Zen's Speedtest, Speedtest.net (with Zen as a target) and Fast.com. - obviously single threaded is more variable and lower than multi-threaded tests, but multi pretty consistent here - across both of my connections at the moment.

Thanks. Have you tried running the realspeed tests - do they give you any interesting information?
I'm looking forward to receiving my Samknows box now to have a look at how it behaves on my line.

Which Fritz router are you using it with - still the 7530?

The real speed tests won't run (because of the router See here) I am using the 7530 ax on the test line, and a 7530 on my BTW line. I have seen some other comments on different forums about many other issues people are having with Zen, can't say that there's anything obviously wrong here now, but I did have a couple of months of terrible issues, and trying wacky ideas. The SamKnows box seems to indicate nothing awry with the original line GEA migrated - but there certainly was a problem, and even Zen confirmed it was present.

Hmmm.... not 'quite' all sweetness and light - Latest latency So, the latency is up by 30% .... Why? Nothing changed here, except the weather perhaps?

Maybe you're connected via a Manchester gateway - have you had any disconnections?

Was your Samknows box from Samknows themselves or from Zen? I was sent one by Zen, but they didn't actually give me any details to connect to it or the dashboard, and are connecting to it themselves to monitor. So I'm pretty much in the dark as to what it is reporting aside for the odd snippet of info forwarded by them.

In reply to a post by SteveBushell999:

Hmmm.... not 'quite' all sweetness and light - Latest latency So, the latency is up by 30% .... Why? Nothing changed here, except the weather perhaps?

Manchester gateway increases my latency up to 16/17ms from 5 to 6 via London. Spoke to Zen about it but they don't consider it to be a problem. I am manually reconnecting to London if required to get round it.

In reply to a post by choppersrock:

Manchester gateway increases my latency up to 16/17ms from 5 to 6 via London. Spoke to Zen about it but they don't consider it to be a problem.

That would drive me crazy, what ever happened to 'Customer is King'

In reply to a post by dect:

In reply to a post by choppersrock:

Manchester gateway increases my latency up to 16/17ms from 5 to 6 via London. Spoke to Zen about it but they don't consider it to be a problem.

That would drive me crazy, what ever happened to 'Customer is King'

Yes it is soo annoying .

I sent them logs and info clearly showing the difference between the 2 gateway's. It was never a problem historically as I monitor my latency and have done for years. Couple of weeks ago it started changing to Manchester everytime I had to reboot following an update or power outage etc to my router. Up went the latency so I started forcing the ppoe connection until I got back to London (closer to me than Manchester).

I got a couple of email responses both starting that the latency is not a problem and won't be investigating it any further.

So minimal router reboots now till end of contract which is some time away as they clearly won't be spending any time on it.

Edited by choppersrock (Sun 14-Aug-22 10:24:56)

Its a shame they can't give customers a way of setting it via a web interface

>Automatic (default)
>London
>Manchester

But clearly they ain't interested in giving their customers the best possible experience when using their service. I would hate to have my gateway the other end of the country.

I'm on static ip anyway and London is closer so why not just set mine to London as location wise that is the closest.

Least I can work round it for the remainder of the contract. You fibre are about to go live in our area so would like to try that eventually as it's symmetrical with fixed ip for extra £5.

Edited by choppersrock (Sun 14-Aug-22 10:50:11)

In reply to a post by choppersrock:

I'm on static ip anyway and London is closer so why not just set mine to London as location wise that is the closest.

I was suggesting a solution to help everyone who wants to set their gateway not a solution for just you

In reply to a post by dect:

In reply to a post by choppersrock:

I'm on static ip anyway and London is closer so why not just set mine to London as location wise that is the closest.

I was suggesting a solution to help everyone who wants to set their gateway not a solution for just you

I get that lol

And, it's getting worse.... See latency and disconnections here I am going via a London gateway

steve@Steves-MBP ~ % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 fritz.box (192.168.178.1) 1.305 ms 0.608 ms 0.481 ms
2 lo0-0.bng4.thn-lon.zen.net.uk (51.148.77.132) 14.196 ms 14.587 ms 18.589 ms
3 lag-14.p2.thn-lon.zen.net.uk (51.148.73.96) 13.085 ms
lag-14.p1.thn-lon.zen.net.uk (51.148.73.94) 12.277 ms
lag-14.p2.thn-lon.zen.net.uk (51.148.73.96) 12.075 ms
4 lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 51.451 ms
lag-2.br2.thn-lon.zen.net.uk (51.148.73.155) 12.389 ms 12.152 ms
5 72.14.223.28 (72.14.223.28) 13.022 ms 11.950 ms
72.14.217.190 (72.14.217.190) 12.928 ms
6 * * 74.125.242.97 (74.125.242.97) 14.479 ms
7 108.170.234.221 (108.170.234.221) 11.718 ms
dns.google (8.8.8.8) 11.435 ms
209.85.252.181 (209.85.252.181) 11.646 ms
steve@Steves-MBP ~ %

No effect on the bandwidth, but clearly something is happening behind the scenes? I am not quite so paranoid about a few milliseconds of latency as some of you seem to be, but one would at least expect it to be relatively stable, and not on a seemingly permanent upwards curve? (compared to the steady 10 or so mS I was getting constantly a week or so ago anyway...

Edited by SteveBushell999 (Mon 15-Aug-22 10:19:52)

In reply to a post by SteveBushell999:

And, it's getting worse....

No effect on the bandwidth, but clearly something is happening behind the scenes? I am not quite so paranoid about a few milliseconds of latency as some of you seem to be, but one would at least expect it to be relatively stable, and not on a seemingly permanent upwards curve? (compared to the steady 10 or so mS I was getting constantly a week or so ago anyway...

The Samknows plotter is pretty useless for showing changes in latency. It has a single plot for each day vs 100 plots per minute on the TBB BQM.

I'd recommend setting up a BQM or if you already have 1 set up then I'd recommend posting it alongside the samknows latency plotter.

You really can't tell anything at all from the samknows plotter

You really can't tell anything at all from the samknows plotter

But the disconnections still apply, and the fact that it is consistently worse latency (averaged) in any way you like than it was (70% worse) - But I take the general point nonetheless, I am more posing a question than making a statement of fact.

Have you tried the speed test command line tool for Windows. I have tried the Linux one which for me was inconsistent but the most reliable was the Windows version from the command prompt. I find server id 34948 to be very consistent on latency and overall speed.
I just used speedtest -s 34948
Might be worth a go.

How does this compare to the non migrated line?

Have you tried the speed test command line tool for Windows.

I have tried all manner of speed tests, all of them are pretty consistent, right now, my problem is not that the speed (up or down) is a problem. I am not sure how much of this (monstrous) thread you have followed, but back in March, I was GEA migrated, that caused a massive drop in download (not upload!) speeds, there were various issues relating to different TCP stacks and how they handled disconnections (these being introduced because of the GEA migration) - eventually, Zen migrated my line back to BTW, and all was fine. Since then, they installed a second line (which I am currently using for work etc) - that is on BTW. Then Zen migrated back (GEA) on my original line, and that is the one we are currently monitoring and speaking about - the hope was that migrating back the original line would display the original fault, but as mysteriously as for other people, it seems to have 'gone away' - at least as it originally manifested itself. The overall drift in latency is just something that I am noticing, speeds are all constant up and down on both lines, and 'some' of the disconnections are explicable, but certainly not all of them.

How does this compare to the non migrated line?

There's no SamKnows box on the non-migrated line, but aside from the disconnections, the up/down speeds are very similar, and the latency seems to be around 11mS unloaded. All way better than I had March -> June, but actually pretty similar to my other line now, but there are some possible nasty portents there for the original line, but then again, perhaps not.

I really surprised they didn't give you a SamKnows box for each as they could have learnt a lot from the comparisons.

In reply to a post by dect:

I really surprised they didn't give you a SamKnows box for each as they could have learnt a lot from the comparisons.

Me too, I am a tester by trade, and I would have approached the whole thing very differently, right from the start! I really don't think there has been much in the way of methodical testing going on, just reactive ad-hoc stuff. I am absolutely convinced that much of this could have been avoided.

As a side note, both lines currently route via London gateways too. But I am far from certain that actually matters.

In reply to a post by SteveBushell999:

both lines currently route via London gateways too.

As was said by 'choppersrock' Zen ain't interested in helping those customers who want to go through their nearest gateway.

There's no obvious way to set up BQM with the router which Zen supply (fritzbox 7530) due to some odd undefeatable ICMP ping throttling that is setup. And unfortunately it's almost a pre-requisite to be using the Zen router to get very far with support.

My 7530 works perfectly with a BQM.
It's only if I try setup a 2nd BQM that the Fritzbox starts blocking ICMP.

If you can't setup a single BQM then perhaps Zen have their own monitoring running on your line.
There's many users who have successfully setup a BQM with a Fritzbox 7530.

Edited by j0hn83 (Mon 15-Aug-22 16:35:34)

In reply to a post by j0hn83:

My 7530 works perfectly with a BQM.
It's only if I try setup a 2nd BQM that the Fritzbox starts blocking ICMP.

If you can't setup a single BQM then perhaps Zen have their own monitoring running on your line.
There's many users who have successfully setup a BQM with a Fritzbox 7530.

Ah, ok. Well, they're going to be getting some superb results from that, then...

Zen do have their own latency reporting tool as they used it against my opnsense box. They decided there were no issues and sent me a chart but I reminded them the gateway was connected to London and not Manchester but this was ignored....

Using it without discussing with me would seem particularly silly though, as they can't get useful results out of a 7530 router if another location is also pinging it given the throttling. I know from my chart it is mostly red with the occasional ping getting through, which would imply they're going to be seeing some packet loss they might not be expecting...

In reply to a post by SteveBushell999:

In reply to a post by dect:

I really surprised they didn't give you a SamKnows box for each as they could have learnt a lot from the comparisons.

Me too, I am a tester by trade, and I would have approached the whole thing very differently, right from the start!

Did you get sent the Samknows box by Zen support - and did set it up so you coud get your own results from it? I still don't have any access to the testing that the Samknows box Zen support sent me has done.

Having said that, on the basis of the results they've seen it sounds like I'm going to be migrated off Zen's network and back to BTW. It looks like from the Samknows info they're seeing an above average level of TCP re-transmissions on the downstream, and while it's not definitely cause and effect of the below par speed I get since migration, it has good legs I believe that the issues will go away once migrated. My performance went from basically perfect pre-migration (>900Mbps to almost any tester) to more often 550-750Mbps, with the occasional 900Mbps.

Will be interesting to see if it does change.

Out of interest, when you were migrated away from Zen GEA back to BTW, did you get sent an email like the following:
"We're sorry that you're leaving us and we'd like to thank you for being a customer.

Broadband Service: xxxxxxxxxxxx
Username: xxxxxxxxx@zen
We can confirm that your cancellation date will be: 31 August 2022."

It then goes on to detail how much I'll owe them for leaving early (nice!).

Finally ending with "Please note, if you're moving house or have recently upgraded/changed your service, don't worry, you can ignore this email. We have everything in hand and are processing your request.".

So you're none the wiser really as to whether this is a system error, or if you're actually getting cut off at the end of the month, or what. A bit of a mess really, sounds like a system in need of a bit of an overhaul.

Hi, Zen sent me the details for the SamKnows box, and I can log in to it. I have access to the past results, and all I can say is that the speeds are pretty constant, I am getting some disconnections, and the latency is increasing over time (Circa 10 mS -> 17mS).

When I was migrated back to BTW, there was no cancellation, only a Zen order for Migration, no cancellations. It does sound as though you are having issues with billing / contracts, who are you dealing with? Aside from sometimes all going quiet, I have had no real problems dealing with Andy and Brandon. It will be interesting if anything is said re: my disconnections or the latency climbing, but I am actually using my BTW line, and the Zen GEA one is just used by Zen at the moment, apart from some random speed tests, and occasional 70GB downloads just to validate it looks good to me.

As a matter of interest, I got a letter from ZZOOMM who will be cabling up this area soon, offering symmetrical packages right up to 2Gbits up/down, but you have to get a business package to get fixed ip, and it isn't super cheap (£129 for that business package or £99 on the home package with first 6 months of 24 month contract free) - Will be interesting to monitor that......

Hi Steve hope all is good mate, not convinced they will ever get to the bottom of this and its been going on for so long now that those who were originally interesting in the problem here are getting fewer and fewer.

Indeed! Agree with all those comments.... Pity that people are getting hung up on peripheral issues to the main thrust of this, but there you go. Steve

Congrats all on the second thread (we have had a thread, yes, but what about second thread?)

I've not been about for a while as this forum started to take up all my spare time 😢

Just a quick message to say that the connection has continued to be good. I haven't noticed any blips or slowdowns.

I should probably email Brandon with the news. He asked me to monitor the connection for a while... Well it's been quite a while now... Oops 😊

Interestingly my samknows box always complains of disconnections and sometimes a bit of packet loss, but if you look at the TBB BQM for those times, everything is fine. Dodgy ethernet cable going to the samknows box maybe?

I'm not really familiar with Samknows boxes, do they ping out from your property where TBB Quality Monitors ping in towards your router? could that be the difference?

It's OK, seems that everyone losing interest in favour of complaining of the service, and worsening of ping times. Both of my lines seem pretty good at the moment, but the GEA one does have latency that's variable, and there are disconnects and some packet loss (seemingly due to the disconnections). The whole thing is really fizzled out as it 'mended itself' - which is a real concern I think, as that which suddenly becomes good can equally suddenly become bad surely?

Regarding the SamKnows box, I really don't like it that it is behind the actual PPPOE connection (LAN-side of the FritzBox), so therefore not 'really' measuring the connection, but more what the FritzBox presents as the facts, but that's something for others to discuss, where is the actual 'boundary' of the connection? Don't forget that Zen say 'try a computer using PPPOE, and tell us what happens' as being the criteria when a connection is bad, NOT what you are getting out of the FritzBox! According to what I saw when it was bad, then there would have been a very different picture dependent on what router I used.

For interest - Using BTW line:

steve@Steves-Mac-mini ~ % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 fritz.box (192.168.1.254) 0.914 ms 0.372 ms 0.219 ms
2 vt1.cor2.lond2.ptn.zen.net.uk (51.148.72.24) 8.973 ms 9.005 ms 9.360 ms
3 lag-9.p2.ixn-lon.zen.net.uk (51.148.73.208) 9.418 ms 9.236 ms 9.221 ms
4 lag-2.p2.thn-lon.zen.net.uk (51.148.73.138) 9.289 ms
lag-2.p1.thn-lon.zen.net.uk (51.148.73.132) 9.404 ms 9.346 ms
5 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 9.517 ms 9.519 ms
lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 9.128 ms
6 72.14.223.28 (72.14.223.28) 9.695 ms 9.742 ms 9.909 ms
7 * * *
8 dns.google (8.8.8.8) 9.373 ms 9.214 ms 9.721 ms
steve@Steves-Mac-mini ~ %

Using Zen GEA Line:

steve@Mini-2011 ~ % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 fritz.box (192.168.178.1) 1.164 ms 0.647 ms 0.459 ms
2 lo0-0.bng4.thn-lon.zen.net.uk (51.148.77.132) 16.402 ms 14.426 ms 15.581 ms
3 lag-14.p2.thn-lon.zen.net.uk (51.148.73.96) 18.625 ms
lag-14.p1.thn-lon.zen.net.uk (51.148.73.94) 11.833 ms 16.582 ms
4 lag-2.br2.thn-lon.zen.net.uk (51.148.73.155) 15.408 ms
lag-1.br2.ixn-lon.zen.net.uk (51.148.73.169) 11.184 ms
lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 12.452 ms
5 72.14.217.190 (72.14.217.190) 15.977 ms 11.527 ms
72.14.223.28 (72.14.223.28) 12.967 ms
6 74.125.242.97 (74.125.242.97) 12.263 ms 12.369 ms *
7 dns.google (8.8.8.8) 12.043 ms 12.230 ms
216.239.63.219 (216.239.63.219) 13.097 ms

Those tests a few seconds apart, seems BTW is better all the way?

Edited by SteveBushell999 (Fri 19-Aug-22 15:19:13)

In reply to a post by SteveBushell999:

It's OK, seems that everyone losing interest in favour of complaining of the service, and worsening of ping times. Both of my lines seem pretty good at the moment, but the GEA one does have latency that's variable, and there are disconnects and some packet loss (seemingly due to the disconnections). The whole thing is really fizzled out as it 'mended itself' - which is a real concern I think, as that which suddenly becomes good can equally suddenly become bad surely?

As you have a GEA and non-GEA connection, an interesting test you could do on your mac from the command line is to get a large file on each connection and see what the single-threaded performance is like.

I've been comparing my GEA line to another user's BTW line and I have much worse single threaded performance, but only to certain servers...(see some of the results from this post onwards... https://forum.kitz.co.uk/index.php/topic,27091.msg45... Speed tests may deploy variable numbers of threads, so depending on the extent of such issues you might see the impact on a speedtest, or might not.

You could try the following command lines from your Mac without having to install anything:

curl --output /dev/null http://ipv4.download.thinkbroadband.com:80/512MB.zip

and:

curl --output /dev/null http://slackware.uk/slackware/slackware-iso/slackwar...

These will be single threaded downloads. I see the thinkbroadband download capped around 35-40MB/sec, whereas the slackware one can achieve full line rate up to 100MB/sec. I have to have many multiple threads to the thinkbroadband site before I approach line rate. Whereas another user with a BTW Zen 900 connection finds his single thread to Thinkbroadband's download achieves line rate, which starts to appear to show some difference between the two.

To get multi-threaded downloads from command line in OSX you could use axel, but you have to install that via installing homebrew first. There is a tutorial here: https://macappstore.org/axel/

Command line for single thread TBB download with Axel:

rm 512MB.zip ; time axel -U Chrome -n 1 -v -a -o 512MB.zip http://ipv4.download.thinkbroadband.com/512MB.zip

To increase the number of threads, change -n 1 to -n 40, for instance, for 40 threads instead of one. I have to use 40 threads for the TBB download to approach line rate on my 900Mbps connection, which seems ludicrous.

The other download via Axel can be grabbed with this command line:

rm bigfile ; time axel -n 1 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...

Note axel barfs if the file is alread there and you try to overwrite it, hence why I have the rm first to remove the downloaded file.

Anyway, if you get a moment, it will be interesting to see if you see a difference between your GEA and BTW connections via this method. Please also note which gateway you are on for each (the second address on the traceroute, eg vt1.cor2.lond2.ptn.zen.net.uk or lo0-0.bng4.thn-lon.zen.net.uk from your examples).

Edited by jimbof (Sun 21-Aug-22 13:43:15)

Hi, and thank you Jimbof, now things get REALLY interesting. I had realised the multithread thing a while ago, and experimented on Fast.com (you can change the thread count with that speedtester) - The only conclusion I came to was the more is better, but that was when it was all broken.

Right, firstly BTW

steve@Steves-Mac-mini ~ % curl --output /dev/null http://ipv4.download.thinkbroadband.com:80/512MB.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 512M 100 512M 0 0 54.4M 0 0:00:09 0:00:09 --:--:-- 57.8M
steve@Steves-Mac-mini ~ % curl --output /dev/null http://slackware.uk/slackware/slackware-iso/slackwar...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3605M 100 3605M 0 0 35.9M 0 0:01:40 0:01:40 --:--:-- 48.4M
steve@Steves-Mac-mini ~ % curl --output /dev/null http://slackware.uk/slackware/slackware-iso/slackwar...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3605M 100 3605M 0 0 35.8M 0 0:01:40 0:01:40 --:--:-- 50.1M
steve@Steves-Mac-mini ~ % rm bigfile ; time axel -n 1 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
rm: bigfile: No such file or directory
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

[100%] [..........................................................................................................................................] [ 19.0MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 3:10 minute(s). (19412.85 KB/s)
axel -n 1 -v -a -o bigfile 4.32s user 17.92s system 11% cpu 3:10.58 total
steve@Steves-Mac-mini ~ % rm bigfile ; time axel -n 40 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

[100%] [.................................................................................................................................................] [ 53.9MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 1:06 minute(s). (55232.45 KB/s)
axel -n 40 -v -a -o bigfile 7.36s user 25.87s system 49% cpu 1:07.43 total
steve@Steves-Mac-mini ~ % rm bigfile ; time axel -n 2 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

Threads finishing etc....
[100%] [.................................................................................................................................................] [ 31.4MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 1:54 minute(s). (32127.28 KB/s)
axel -n 2 -v -a -o bigfile 4.20s user 18.34s system 19% cpu 1:54.99 total
steve@Steves-Mac-mini ~ % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 fritz.box (192.168.1.254) 1.400 ms 0.329 ms 0.285 ms
2 vt1.cor2.lond2.ptn.zen.net.uk (51.148.72.24) 9.044 ms 11.268 ms 8.875 ms
3 lag-9.p2.ixn-lon.zen.net.uk (51.148.73.208) 9.095 ms 9.392 ms 9.613 ms
4 lag-2.p2.thn-lon.zen.net.uk (51.148.73.138) 9.493 ms
lag-2.p1.thn-lon.zen.net.uk (51.148.73.132) 9.331 ms 9.445 ms
5 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 9.510 ms 9.272 ms
lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 9.239 ms
6 72.14.223.28 (72.14.223.28) 9.644 ms 9.679 ms 9.587 ms
7 * * *
8 dns.google (8.8.8.8) 9.783 ms 8.963 ms 13.665 ms
steve@Steves-Mac-mini ~ %

NOW, for GEA migrated line

steve@Mini-2011 ~ % curl --output /dev/null http://ipv4.download.thinkbroadband.com:80/512MB.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 512M 100 512M 0 0 59.9M 0 0:00:08 0:00:08 --:--:-- 60.7M
steve@Mini-2011 ~ % curl --output /dev/null http://slackware.uk/slackware/slackware-iso/slackwar...

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
steve@Mini-2011 ~ % curl --output /dev/null http://slackware.uk/slackware/slackware-iso/slackwar...

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3605M 100 3605M 0 0 60.6M 0 0:00:59 0:00:59 --:--:-- 60.7M
steve@Mini-2011 ~ % rm bigfile ; time axel -n 1 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
rm: bigfile: No such file or directory
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

[100%] [....................................................................................................................................................] [ 58.7MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 1:01 minute(s). (60136.08 KB/s)
axel -n 1 -v -a -o bigfile 7.10s user 30.68s system 61% cpu 1:01.48 total
steve@Mini-2011 ~ % rm bigfile ; time axel -n 40 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

Threads finishing etc....
[100%] [....................................................................................................................................................] [ 50.5MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 1:11 minute(s). (51762.57 KB/s)
axel -n 40 -v -a -o bigfile 6.79s user 34.00s system 55% cpu 1:13.74 total
steve@Mini-2011 ~ % rm bigfile ; time axel -n 2 -v -a -o bigfile http://slackware.uk/slackware/slackware-iso/slackwar...
Initializing download: http://slackware.uk/slackware/slackware-iso/slackwar...
File size: 3.5209 Gigabyte(s) (3780542464 bytes)
Opening output file bigfile
Starting download

Connection 0 finished
Connection 1 finished
Connection 1 finished
Connection 0 finished
[100%] [....................................................................................................................................................] [ 49.7MB/s] [00:00]

Downloaded 3.5209 Gigabyte(s) in 1:12 minute(s). (50864.11 KB/s)
axel -n 2 -v -a -o bigfile 6.73s user 29.95s system 50% cpu 1:13.23 total
steve@Mini-2011 ~ % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 fritz.box (192.168.178.1) 1.420 ms 0.591 ms 0.449 ms
2 lo0-0.bng4.thn-lon.zen.net.uk (51.148.77.132) 13.838 ms 14.579 ms 15.640 ms
3 lag-14.p2.thn-lon.zen.net.uk (51.148.73.96) 12.920 ms
lag-14.p1.thn-lon.zen.net.uk (51.148.73.94) 12.315 ms 12.554 ms
4 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 12.958 ms 12.358 ms
lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 12.906 ms
5 72.14.223.28 (72.14.223.28) 12.365 ms
72.14.217.190 (72.14.217.190) 12.838 ms 11.754 ms
6 108.170.246.129 (108.170.246.129) 12.785 ms
74.125.242.97 (74.125.242.97) 14.753 ms *
7 108.170.234.231 (108.170.234.231) 12.503 ms
dns.google (8.8.8.8) 11.412 ms
142.250.215.127 (142.250.215.127) 11.709 ms
steve@Mini-2011 ~ %

Conclusions? Both connecting via London, single thread very poor on BTW, but better gets good at 40 threads. As for GEA: - Well, seems I have more raw bandwidth available, and that it is better with much less threads - what's your view?

Steve

Edited by SteveBushell999 (Sun 21-Aug-22 15:33:00)

Your GEA connection is almost the complete opposite of mine.
You have to be a little bit careful with the Slackware download as sometimes it resolves to a slower server; I was mostly using it to prove my link was good as it very often found a server that would saturate 900mbps in a single thread.

It's interesting that your TBB downloads are slower via BTW than GEA.
It's worth trying axel on the TBB download too, with one and with several threads, and seeing how that behaves. I've not used the curl method much, I only looked it up as I saw you were on a mac but figured you might not want to install axel, but thanks for installing it. Probably better using axel than curl from now on, now you have it. Note the command line to use axel to download the TBB files is a bit different as it needs to spoof a Chrome useragent, otherwise the requests get denied (probably too many people trying to make automated tests using axel).

I'm not really sure what conclusions can be drawn at the moment if any...

It's weird! Firstly, I'm fine with Brew, I use it for other things like TVHeadend etc. OK, so....

BTW

steve@Steves-Mac-mini ~ % rm bigfile ; time axel -n 1 -v -a -o bigfile http://ipv4.download.thinkbroadband.com:80/512MB.zip
Initializing download: http://ipv4.download.thinkbroadband.com:80/512MB.zip
File size: 512 Megabyte(s) (536870912 bytes)
Opening output file bigfile
Starting download

[100%] [.................................................................................................................................................] [ 28.1MB/s] [00:00]

Downloaded 512 Megabyte(s) in 18 second(s). (28801.14 KB/s)
axel -n 1 -v -a -o bigfile 0.58s user 2.34s system 15% cpu 18.233 total
steve@Steves-Mac-mini ~ % rm bigfile ; time axel -n 40 -v -a -o bigfile http://ipv4.download.thinkbroadband.com:80/512MB.zip
Initializing download: http://ipv4.download.thinkbroadband.com:80/512MB.zip
File size: 512 Megabyte(s) (536870912 bytes)
Opening output file bigfile
Starting download

Connection 27 finished
Connection 13 finished
Connection 14 finished
Connection 18 finished
Connection 11 finished
Connection 7 finished
Connection 11 finished
Connection 7 finished
Connection 11 finished
Connection 16 finished
Connection 15 finished
Connection 7 finished
Connection 13 finished
Connection 26 finished
Connection 18 finished
Connection 8 finished
Connection 1 finished
Connection 0 finished
Connection 24 finished
Connection 19 finished
Connection 15 finished
Connection 13 finished
Connection 26 finished
Connection 24 finished
Connection 19 finished
Connection 13 finished
Connection 8 finished
Connection 15 finished
Connection 0 finished
Connection 14 finished
Connection 33 finished
Connection 26 finished
Connection 22 finished
Connection 16 finished
Connection 29 finished
Connection 19 finished
Connection 1 finished
Connection 8 finished
Connection 24 finished
Connection 13 finished
Connection 12 finished
Connection 3 finished
Connection 14 finished
Connection 18 finished
Connection 28 finished
Connection 33 finished
Connection 26 finished
Connection 6 finished
Connection 37 finished
Connection 7 finished
Connection 15 finished
Connection 17 finished
Connection 23 finished
Connection 22 finished
Connection 16 finished
Connection 19 finished
Connection 29 finished
Connection 4 finished
Connection 27 finished
Connection 35 finished
Connection 8 finished
Connection 1 finished
Connection 24 finished
Connection 38 finished
Connection 39 finished
Connection 10 finished
Connection 13 finished
Connection 32 finished
Connection 9 finished
Connection 21 finished
Connection 14 finished
Connection 12 finished
Connection 36 finished
Connection 0 finished
Connection 25 finished
Connection 28 finished
Connection 30 finished
Connection 18 finished
Connection 3 finished
Connection 20 finished
Connection 34 finished
Connection 2 finished
Connection 11 finished
Connection 5 finished
Connection 31 finished

Downloaded 512 Megabyte(s) in 8 second(s). (60069.10 KB/s)
axel -n 40 -v -a -o bigfile 1.04s user 3.67s system 52% cpu 8.936 total
steve@Steves-Mac-mini ~ %

Compared to GEA:

steve@Mini-2011 ~ % rm bigfile ; time axel -n 1 -v -a -o bigfile http://ipv4.download.thinkbroadband.com:80/512MB.zip

rm: bigfile: No such file or directory
Initializing download: http://ipv4.download.thinkbroadband.com:80/512MB.zip
File size: 512 Megabyte(s) (536870912 bytes)
Opening output file bigfile
Starting download

[100%] [....................................................................................................................................................] [ 59.9MB/s] [00:00]

Downloaded 512 Megabyte(s) in 8 second(s). (61333.31 KB/s)
axel -n 1 -v -a -o bigfile 1.06s user 4.68s system 66% cpu 8.676 total
steve@Mini-2011 ~ % rm bigfile ; time axel -n 40 -v -a -o bigfile http://ipv4.download.thinkbroadband.com:80/512MB.zip

Initializing download: http://ipv4.download.thinkbroadband.com:80/512MB.zip
File size: 512 Megabyte(s) (536870912 bytes)
Opening output file bigfile
Starting download

Connection 13 finished
Connection 35 finished
Connection 26 finished
Connection 33 finished
Connection 2 finished
Connection 16 finished
Connection 24 finished
Connection 18 finished
Connection 37 finished
Connection 20 finished
Connection 27 finished
Connection 22 finished
Connection 0 finished
Connection 11 finished
Connection 26 finished
Connection 17 finished
Connection 1 finished
Connection 38 finished
Connection 2 finished
Connection 18 finished
Connection 12 finished
Connection 4 finished
Connection 19 finished
Connection 10 finished
Connection 28 finished
Connection 8 finished
Connection 16 finished
Connection 31 finished
Connection 23 finished
Connection 20 finished
Connection 27 finished
Connection 3 finished
Connection 5 finished
Connection 14 finished
Connection 9 finished
Connection 7 finished
Connection 21 finished
Connection 39 finished
Connection 30 finished
Connection 36 finished
Connection 15 finished
Connection 13 finished
Connection 34 finished
Connection 33 finished
Connection 32 finished
Connection 6 finished
Connection 29 finished
Connection 25 finished
Connection 37 finished
Connection 35 finished
Connection 24 finished
Connection 22 finished
Connection 0 finished
Connection 1 finished
Connection 11 finished
Connection 18 finished
Connection 26 finished
Connection 2 finished
Connection 12 finished
Connection 17 finished
Connection 19 finished
Connection 38 finished
Connection 8 finished
Connection 4 finished
Connection 10 finished
Connection 28 finished
Connection 16 finished
Connection 31 finished
Connection 23 finished
Connection 27 finished
Connection 20 finished
Connection 14 finished
Connection 9 finished
Connection 7 finished
Connection 3 finished
Connection 5 finished
Connection 21 finished
Connection 39 finished
Connection 36 finished
Connection 13 finished
Connection 33 finished
Connection 15 finished
Connection 32 finished
Connection 34 finished
Connection 6 finished
Connection 25 finished
Connection 29 finished
Connection 30 finished
Connection 0 finished
Connection 37 finished
Connection 1 finished
Connection 35 finished
Connection 22 finished
Connection 24 finished
Connection 11 finished
Connection 18 finished
Connection 26 finished
Connection 2 finished
Connection 12 finished
Connection 17 finished
Connection 19 finished
Connection 38 finished
Connection 8 finished
Connection 10 finished
Connection 16 finished
Connection 4 finished
Connection 28 finished
Connection 9 finished
Connection 20 finished
Connection 14 finished
Connection 7 finished
Connection 31 finished
Connection 3 finished
Connection 23 finished
Connection 27 finished
Connection 5 finished
Connection 39 finished
Connection 21 finished
Connection 36 finished
Connection 13 finished
Connection 33 finished
Connection 32 finished
Connection 15 finished
Connection 34 finished
Connection 6 finished
Connection 25 finished
Connection 29 finished
Connection 30 finished
Connection 0 finished
Connection 37 finished
Connection 1 finished
Connection 35 finished
Connection 22 finished
Connection 24 finished
Connection 11 finished
Connection 18 finished
Connection 26 finished
Connection 2 finished
Connection 12 finished
Connection 17 finished
Connection 19 finished
Connection 38 finished
Connection 8 finished
Connection 10 finished
Connection 16 finished
Connection 4 finished
Connection 28 finished
[100%] [....................................................................................................................................................] [ 55.6MB/s] [00:00]

Downloaded 512 Megabyte(s) in 9 second(s). (56902.82 KB/s)
axel -n 40 -v -a -o bigfile 1.26s user 5.70s system 67% cpu 10.375 total
steve@Mini-2011 ~ %

They are certainly 'different' - does it matter? Do I care? Aside from the latency.....
BTW:dns.google (8.8.8.8) 8.764 ms 8.166 ms 8.745 ms Versus GEA:dns.google (8.8.8.8) 11.974 ms 12.504 ms

I'm not really sure what conclusions can be drawn at the moment if any...

Damn right ! .... But why multiple Connection finished output?

Edited by SteveBushell999 (Sun 21-Aug-22 16:57:51)

In reply to a post by SteveBushell999:

It's weird! Firstly, I'm fine with Brew, I use it for other things like TVHeadend etc. OK, so....

>snip<

Damn right ! .... But why multiple Connection finished output?

Maybe the connection closed partway through and had to be re-opened for the GEA run.
It's definitely odd that our patterns are quite different. It would seem for those two tests at the moment your GEA connection is more performant than your BTW one, which fills me with a little dread as mine is due to be migrated back from GEA to BTW on 31st... Oh well.
I guess there may be local factors affecting things differently - perhaps the BTW links are worse round your neck of the woods.

BTW, I was only doing 40 connections as for me that was the level at which I got close to saturating the link with the TBB test files. It may be counter-productive in some scenarios to have such a large number of threads.

I guess at least I have a growing armoury of metrics that I can run prior to migration and post migration to evaluate how things are looking.

Well, in the 'real world', there isn't much difference between the connections that I have now, sure, I can see different numbers in the tests, but overall, the situation is chalk and cheese to the woes I had with my first GEA migration, using the Fritz router I was down to about 10-100Mbits download, and it was dropping connections all the time, but fair-enough, that must have been some (unexplained!) fault, which mysteriously has now fixed itself.

I agree, there needs to be some trepidation with any migration going by what a few people have seen, but even then (allegedly) we are in a minority, although I don't think there is any active testing of the migration, only a reliance on users complain ing, and I wonder how many would even notice - most people use WiFi all the time through six solid walls and get about 15Mbps in my experience, they don't even know what UTP is, and freak out at the prospect of using it.

Happy to run any other tests you fancy, not sure where you are that you experience different GEA / BTW connectivity, I'm in Cheshire, and it seems a good number of the people affected are in this neck of the woods.

Yes, I'm thinking think I may have jumped in the sharkpool slightly now with not much more than a steak-flavoured bathing suit in pushing for the migration back to BTW, but anyway...

Your performance did sound atrocious, mine merely a bit annoying.

I'm in Norwich City Centre. Can almost see the telephone exchange from my house.

Thanks for running the tests, if anything else occurs to me I'll let you know.

Ah yes, Norwich... I used to live in Ranworth, and later in Ditchingham near Bungay. Struggled to get 40Mb down over FTTC there! How things have changed..... Yes, when it was broken, it was terrible, I endured that for 3 months.... Let's hope your migration back to BTW goes as well as mine did.

Migrated back to BTW today.

Connection went down shortly after midnight, had to get Zen support involved to get it back working as there was some problem at their end with the radius setup so authentication was failing. Was eventually back up by 10am, by which point the family were about to do me in.

Headline news - performance is now back to pre-migration level and comfortably hitting line rates to any where that is capable it seems.
Some results

Significant minus points to Zen's billing system. After the initial email about cancellation charges and being told not to worry about it, I received a further email with an invoice attached for £££'s of pounds cancellation, and then the next day received an email telling me this amount was being taken via direct debit.

Managed to speak to someone after the Bank Holiday weekend and they cancelled that payment from being taken (apparently). The proof of that pudding will be in the eating...! We will see.

If I was being charitable I'd say the billing issues may be down to migration back to BTW not being a common case they would be having to deal with. If I was feeling a little less charitable I'd say hang about, I've spent best part of 2 and a half months here wasting time back and forth over something that seems to be an issue either with Zen's network (or how their network is implemented within their 3rd party suppliers). All resulting from a change to my connection that they pushed through that I absolutely did not request on my line.

Anyway, I'm glad it's back performing how it should, I'll be considering my position very carefully in a few months when the contract is up, as though we've got there in the end I can't help feel it was painful and didn't actually start to move along until I asked to pull the plug on the contract.

Edited by jimbof (Wed 31-Aug-22 13:15:54)

Well, at least there is a positive side there, although the billing seems odd.No really update from my end, both connections seemingly generally OK, but the GEA one does have more disconnects than I would expect I think (see here) Am watching the Zzoomm installation people at work near here, not sure how long before their symmetrical FTTP becomes active at up to 2Gbits here....

Yes, I'm glad to be back up to full speed, though a little jaded by the experience.

Have you looked at any of the other stats that the Samknows box collects? If you look on the Analytics page of the Samknows portal there are loads of other tests that it is running that might be more sensitive to differences between the connection than the multithreaded connection test that is reported as the headline number.

For instance, the difference on the Netflix downloader test is massive on mine, having gone from hovering around the 450-550Mbps mark (and being very variable) to being almost rock solid at ove 900Mbps.

I do think there is a chance they've made improvements which mean the issues are masked mostly if you "only" have a 500Mbps line. I say that as most of the tests I've run have given results in excess of 500Mbps, so it stands to reason that if that is a limitation it would only become really obvious on the lines that are contracted at the highest rate.

You can see the stepchange in behaviour from the migration in the following grabs from the Samknows Analytics, and how big the difference is between the Samknows multithreaded HTTP test and the Netflix test...
Samknows results

Here are some more pre- and post- migration traces. Migration happened on 31st August.
Most notable traces for me seem to be the Netflix download and also the RTP jitter, both of which are massively improved on BTW vs Zen GEA. Some minor differences elsewhere on latencies etc (some in Zen GEA's favour). I wonder if you can see a difference on the RTP jitter traces between the two connections?

Some More traces

My Samknows box is going back now and I'm going back to my Ubiquiti router.

Hmmm, all very interesting, and in general similar to what I see, except in one major respect.... See Here for my disconnections they are getting worse, and I am sure there is something 'wrong' FTTP shouldn't behave like that surely?

As you can see, my up and downloads are pretty stable, but there are quite lengthy and repeated disconnections that may well have had something to do with the problems (I have informed Zen of this behaviour). Sorry, but SamKnows box only on my migrated line, so can't comment on a comparison, but the BTW line seems fine.

Looks like you have some pretty massive problems with something. No, FTTP should not behave like that.

Looks like you have some pretty massive problems with something

Yes, that was my reading of the situation both 'then' and now.As the line is migrated to GEA and shows this behaviour, I am wondering if Zen (Andrew / Brandon?!) fancy switching it back with the monitor on, and seeing what happens? Seems like a sensible next step to me? I got nothing like these disconnections from day zero when this was reinstated as you can see

Maybe of interest, I have just been put back onto the original backhaul after 7 weeks of horror following a GEA migration.

Sadly, Zen have not been good at answering phone calls or emails, now I have to try to sort out their billing errors.

Not exactly what I had expected from Zen.

7 weeks of horror following a GEA migration

Oh dear! I wish I could say that your experience was an isolated case, but there seem to be a few.... I suffered for 2 1/2 months with download at a max of 100Mbps (MAX!) and multiple disconnections, sometimes only 10Mbps real world speeds (500Mbps advertised)- if you have a really huge quantity of time to waste / spare, read the history of this, but FakeJake, myself, and many others have a similar story. Mine only cured by migrating back to BTW, but Zen reinstated my original line, and migrated again back to GEA, and as you will see, it seems to have reverted to a terrible state once again. There is clearly a problem somewhere, could start diagnosing your problem, but it will probably follow a familiar path...... Does PPPOE connection to a pc work fine..... (RED HERRING! it's how the TCP stack deals with disconnections! - with a router or a PC, doesn't matter!) -- then it just gets worse...... Have you tried ... this, that, and rubbing crystals on the cables.... Whatever.... The problem is plain and obvious. Being in the NorthWest seems to be the hotspot of problems, but not just there, reports in Norfolk and elsewhere too.... Regarding Zen responses, it's been mixed, a couple of us are quieter than we perhaps might otherwise be. Good luck !

North Wales here.

Now back on Openreach AFTER 70 days.

Fingers crossed. Seems OK

In reply to a post by Rhynchelma:

Maybe of interest, I have just been put back onto the original backhaul after 7 weeks of horror following a GEA migration.

Sadly, Zen have not been good at answering phone calls or emails, now I have to try to sort out their billing errors.

Not exactly what I had expected from Zen.

Did you also have a load of guff to do with cancellation charges appear on your account? I think mine is all sorted now, took 3 or 4 phone calls though.

Ironic them sending emails (and even a snail mail letter!) about cancellation charges; if I billed them for my hours spent trying to get to the bottom of an issue of their making in migrating me I'd probably not have to pay them for the next year... I've not even asked them to credit me for the poor performance over the last 2 months.

I wonder if the GEA network is generally poor performing and just most customers either don't realise or accept the fobbing off (I got fed up of the amount of times people said over the phone to me that they wouldn't be able to do much if it was above the minimum guarantee, you can't expect full speed / you're getting great performance / etc). On the other hand these could be isolated incidences due to particular kit. But the more reports I'm seeing from different parts of the country the more I'm beginning to doubt that...

This whole issue with Zen (starting way back in March of this year) is absolutely shocking. Having track this thread from the beginning I still can't believe Zen haven't got a grip of the issue yet.

I am trying to sort the cancellation charges out - fingers crossed

I think that BT backhaul costs them money. That was mentioned when they agreed to move me back. maybe they just can't fix their own, but it's "good enough" for many/most. Or edge cases. No real idea.

But it's far from good. I really suppose I will move away as soon as I can, unless things look up.

Mind you, the grass is always greener.

Edited by Rhynchelma (Thu 08-Sep-22 00:08:42)

In reply to a post by Rhynchelma:

I think that BT backhaul costs them money.

I would certainly think so but I am surprised they raised that point as my response to that would have been, "you're only having to migrate me back because you can't sort your own backhaul". I do wonder how hard they are trying now.

In reply to a post by Rhynchelma:

the grass is always greener.

For me it is, I'm now on a well known provider with mid-range costs and no issues.

I'll give them a little longer.

I don't know why more people who have been migrated and are having issues don't leave penalty free, as Zen are breaching their own contract by giving you a significantly worse service with no notification.

My contract was terminated without charge after I raised this with them post GEA migration.

I'm now with a new provider and have no more issues.

I can only speak for myself, but I am keen to properly understand what has gone wrong, and as long as I have a working line (as well as a broken one - possibly....) then it is easier to keep my fixed IP (needed for work), and domains with Zen. I did get a LOT of grief, but the offer was made for me to terminate without penalty, I have decided to be positive and try to assist. I must say though that with people apparently continuing to be migrated, and no satisfactory explanation as to the cause of all this, I am somewhat saddened and disheartened.

You have the patience of a Saint Steve

In reply to a post by S2KIP:

I don't know why more people who have been migrated and are having issues don't leave penalty free, as Zen are breaching their own contract by giving you a significantly worse service with no notification.

My contract was terminated without charge after I raised this with them post GEA migration.

I'm now with a new provider and have no more issues.

I was certainly thinking about it. However, they offered another investigation/possible fix.

Eventually, the time that a new connection would take stopped me.

I will be keeping a close eye.

Ihave serious doubts if I will still be with them when the contact renews.

In reply to a post by Rhynchelma:

I have serious doubts if I will still be with them when the contact renews.

Same.

How many people are we actually talking about here though? I >think< reported affected have barely covered the digits on one hand so far.

In my case they agreed to migrate me back before their official month timed out. I suppose if I'd wanted to I could have hard-balled them into letting me go as I wasn't going to be migrated back within the elapsed month of the issue being logged officially, but I'm limited as to how much of my time it's worth arguing the toss over this given my connection basically works ok.

I'm probably going to be leaving once my contract is up next year; the whole indeterminism of the performance based on which particular gateway you've been connected to, the sedentary pace of dealing with the issue I've seen, the threat of a future migration back to a GEA network that might or might not be fixed at some point, are all weighing heavily against them for me.

Edited by jimbof (Fri 09-Sep-22 17:18:46)

We never really did get to the bottom of how many, I really do suspect that huge numbers are unaware as they probably put it down to their WiFi connection (only us geeks use wired I believe) I may be wrong though.... It is only a few that chat on here about it, but I doubt we'll ever know the full extent, but affected people keep crawling out of the woodwork, and I believe people continue to get migrated without anyone understanding the nature of the problem.

To be fair, 'eventually' Zen have taken my case seriously, and that seems to have happened to everyone who shouts loudly enough, not the ideal way though.

There do seem to be wider issues around service and billing, but for me it wasn't SO (!) bad.....

For now, much the same as everyone it's wait and see, I was with Zen FTTC in Norfolk for about 8 years, and never had any real reason to complain, and I believe that this is mostly a NorthWest based problem, but not actually just so.

I am about to make a change to my network so that the GEA line gets rather more use (will point my Servarr data through it, so that will be quite heavy), it will be interesting to see what that does to disconnections etc.

If you have a connection that isn't disconnecting I can't see why in your right mind you would use the disconnecting connection for anything much, why do it to yourself... It's not going to get any better through using it a bit more so it just sounds like a hiding to nothing much.

Personally in your situation I would take the Samknows box and put it on the non-GEA line and see how that is behaving, to prove that you have a significant difference between the two, and you really do have one good line... Worst case you might actually have an issue on your non-GEA line too, that you're not really being alerted to, given your GEA line appears to speedtest OK.

Ah, but it's not totally that simple! The GEA line I have is only for testing (supplied by Zen), and the data I want to put through it is hardly 'critical' - won't put my my work or emails/domains etc through it yet.

I am really keen to get to the bottom of the real problem, and if I can help, then that's what I will do. I do know that my BTW line is good, as everything returned exactly to normal when that went in, but to really see what the GEA line is doing, I want to see something like 'real-world' traffic and look at what happens.

A quick look at SamKnows data shows things not quite as bad as since 7th September(number of disconnects and duration), no idea why though.

I restate that when things were bad here, it was nearly unusable, obviously not quite like that now, but trying to chase up what may be actually behind those problems.

I am really keen to get to the bottom of the real problem, and if I can help, then that's what I will do. I do know that my BTW line is good, as everything returned exactly to normal when that went in, but to really see what the GEA line is doing, I want to see something like 'real-world' traffic and look at what happens.

Your perseverance is admirable. I guess driven in part by your professional background in software testing and wish to understand what’s broken and why. You may never find out.

I suspect that this issue is a problem on the margins, but deeply rooted within the core network that Zen have deployed that serves their new GEA backhaul.

As you know internet fora can become echo chambers of self resonating doom - and I agree it is almost impossible to say (on here) how widespread and deep this problem is, nor how many of their customers are actually affected. It might be less than a dozen. It might be hundreds. It might be thousands.

The only folks that know are Zen networks team and so far it’s been extremely quiet.

Yes, but while it is frequently disconnecting though it is very obviously broken, Zen should already have more than enough to investigate that if they want, so really you would be better served yourself using the resource you have (time with the Samknows box in your possession) checking that your good line really is good, than trying to further investigate just how broken the bad line is.

One test which might be worth doing is swapping the ONTs between the cables in case one of the cables has a bad termination, as unless they run on a different PON you should find the GEA and non GEA connections follow the ONT and not the fibre. You might only want to do that under discussion with Zen though, and proceed with care as Pew pew laser beams.

If the customer drops are coming from the same CBT, then they will be on the same PON. Openreach don’t to my knowledge serve the same CBT with different light. It’s always the same PON.

Saying that I recall that one of the posters on the precursor to this thread had immediate neighbours on a different PON (using ECI gear as I recall) from an earlier partial FTTP deployment in the neighbourhood. Then there was a later connectorised CBT deployment on Huawei gear for the remaining properties.

Can’t recall if it was Steve though.

I guess my overarching point outside of ONT experimentation is recalling Steve's previous messages on this, he was under the impression that the GEA line was now fine, but I'd say the Samknows box is saying anything but fine with that level of disconnections. If the same level of "fine" is applied to the non-GEA line, that also might not be great. Hence why it's worth investigating the non-GEA line in more detail as that is the one he's probably going to be using longer term, while Zen are fully engaged. It is possible they are both "a bit less rubbish" than previous, below the threshold of being noticeable much in normal use.

I have wondered for some time why the so called experts at Zen haven't sent Steve a Samknows box for each line so they have second by second comparisons.

Yep follow your logic, but as each ONT is tied to a particular service (and hence backhaul) you would literally only be 'testing' the difference between the drop cables from the street (OK optical path from the output of the splitter to be pedantic). Which you could do but I think the probability of finding the issue there is quite remote.

The disconnections list Steve shared looked pretty grim. I think there's a reasonable chance they could be down to a physical connection. But it could be the whole PON, or it could just be the GEA link.

For reference, for my connection on GEA, this was the reported disconnections by the samknows box, and at least one lot of them was me playing. So I guess the point here is disconnections don't look to be an expected thing on GEA.

https://photos.google.com/share/AF1QipMWbRVSne4nu5U0...

This was the following 7 days for Steve:
https://prnt.sc/X2qFgxmyt_er

Mine was already off GEA by then, so I can't share comparable days. If Steve clicks the "Month" button on the dashboard he could show whether he was having disconnections prior to the 1st of Sept.

Edited by jimbof (Sat 10-Sep-22 17:51:26)

They were very keen to get mine back off me. I expect they don't have many of them. But i agree, it just seems a bit bonkers to go to all the cost of putting in a second line without having monitoring on both...!

Need him to confirm if they replaced the ONT, cant recall. That would be my first port of call for physical port disconnections (Ethernet) and would be insisting on this if this if so.

Didn't Jake's issue appear to be resolved by a mystery card placement somewhere in the Openreach network without the need for him to be un-migrated. I would have expected him to be back here if his issue had come back.

I don't think the Samknows has any way to know if they are actually physical port disconnections. It's on a switch port on the Fritzbox, one stop away from the ONT ethernet port. The logs on the Fritzbox might offer up some clues, as it does log the PPPoE sessions coming and going.

Yep. Was referring to Steve’s connection, rather than Jake

There well could be multiple things in play. Not necessarily one common thread.

Edited by Pheasant (Sat 10-Sep-22 20:33:24)

Knowing whether the disconnection is physical (Ethernet port flapping for example) is key to understanding whether the ONT could be at fault. If it’s a PPPoE disconnect that would be transparent to the ONT.

You want the month? Here it is My beef is that the whole 'testing' of this is a farce in real ISTQB / ISEB terms.... It is best described as 'exploratory' testing, without any baseline, any pass / fail criteria, no 'requirements', no objective with the testing, zero good practice TBH.

So, I am happy to do as they ask, and to share that here, I agree, any element from the RJ45 backwards could be to blame, but there is zero real attempt to isolate it, save from a retrospective look back from their end, and a snapshot of the problem observed from this end.

At the risk of repeating myself.... they continue to roll this out to other (unknowing) victims, with no understanding of the 'problem' ..... THAT is the biggest issue.

The process of testing is very well known, and documented for example It's not rocket science, but it's something that needs an understanding of what one is trying to achieve, some (sensible) pass/fail criteria (50% of advertised bandwidth...? REALLY? !) You need a clear plan of what you do before go-live (to mitigate failure risks), and after that, some warranty plan to catch stuff that was forgotten????

Of course, there is a huge caveat...... ALL testing is 'risk based testing', and it seems that the appetite for risk here is somewhat higher than the industry norms, the drawbacks? 1) Loss of reputation, 2) Probability of negative publicity.... It's again less than rocket science, speed and cost of deployment versus the two big negatives above.

I could go on and on, but there is simply no real science being applied here, even grabbing some 'try this' thing isn't a systematic process that will get the root cause. Seems there isn't a 'Eureka' moment to be had, simply guesswork.

Oh, and end comment.... A proper systematic test approach to all this would have taken FAR less time than from March to now, have caused less grief, cost less, and everyone would have a transparent view of any test results to reassure them that this had been tackled correctly, even if they had a minor problem, they would have the reassurance that things had been tackled properly.

Edited by SteveBushell999 (Sat 10-Sep-22 22:22:30)

Which indicates that as far as the corporate view from Zen is concerned it is not a real problem - just a few nerds who are not typical of most of their customer base shouting the odds. Most of the people paying their bills to Zen are not aware in their everyday use of the network of any problems so where is the business case to spend any money to find a solution?

To anyone with any sort of scientific or technical background that is a deeply unsatisfactory view of how things should be done but for anyone with an arts, liberal arts, media studies or accountancy background a totally acceptable way forward.

Edited by GonePostal (Sun 11-Sep-22 01:09:13)

In reply to a post by GonePostal:

Which indicates that as far as the corporate view from Zen is concerned it is not a real problem - just a few nerds who are not typical of most of their customer base shouting the odds. Most of the people paying their bills to Zen are not aware in their everyday use of the network of any problems so where is the business case to spend any money to find a solution?

To anyone with any sort of scientific or technical background that is a deeply unsatisfactory view of how things should be done but for anyone with an arts, liberal arts, media studies or accountancy background a totally acceptable way forward.

I sadly understand what you saying as some car manufactures when finding issues with their cars sometimes take the decision that a few possible deaths is not a good enough reason to spend millions on a vehicle recall.

Edited by deleted (Sun 11-Sep-22 09:50:43)

Yes, so it's very fail-y.

With that level of disconnections I really wouldn't be bothering to put anything else on that network. It's very obviously broken, and Zen have more than enough there with the Samknows box on the line to be trying to address it if they wanted to. You're not doing yourself any favours moving anything else onto the line, just inconveniencing yourself, and I can't see any evidence they'll care any more for the extra data it might give - they've got plenty of data there already to work through if they wanted. The only testing that would make any sense for you to do to that GEA line would be things which have potential to improve or change its behaviour, which sticking a server running on it has absolutely no chance of improving matters. So things like a replacement ONT, cable replacement, swapping ONTs between the different fibre downleads, swapping routers, etc.

I agree with the continued roll-out question. I've been migrated back, but I doubt anyone else has been at my exchange, and I doubt very much I'm the only one who was affected at my exchange as I can't believe there is any bit of kit that would be unique to me on Zen (I don't think I had a dedicated physical port wired to them, for instance, I think with FTTP it is all done via switch / OLT configuration / VLANs / etc).

In reply to a post by dect:

as some car manufactures when finding issues with their cars sometimes take the decision that a few possible deaths is not a good enough reason to spend millions on a vehicle recall.

Such as Tesla.

OK, so, my disconnections continue. I have written to Zen, and got a rather 'interesting' email back (typos corrected):

Hi Steve,

We’ve not (investigated this), though we are talking extensively with our Vendors, as we do have some problems, which other Probes have also picked up. Yours seems a little different, as we don’t see disconnections like this on other affected circuits, normally just reduced throughput when connected to some of our gateways (with no consistency around which gateways).

Andrew, are we able to get Tech Support to look at this as a standard fault, and see if there is an issue on the OR side?

I'm sure some of you will want to comment on this as it's the first real admission on the problem widely existing, and also an admission that there ARE routing problems with the gateways..... Over to you

Intuitively I would agree that your specific disconnect issue looks more like a “standard fault” - my guess ONT fault.

The ongoing but inconsistent / sporadic, reduced throughput issue - looks on the other hand like a deeper core network fault / interoperability issue with OR L2S & Zen backhaul or weird edge case fault higher up the network stack. Where though, how though….

I mentioned my case above.

I had a new ONT fitted (it's fine but why not try it), three Openreach visits all which seemed to indicate that the line to the "exchange" was fine.

Since the reversion to BT's backhaul, I'm back to "perfect" performance. 900+, 100+, low ping, near zero dropped packets.

And the billing seems sorted.

I am somewhat hesitant to say that it's good now, but it seems so.

Still not happy with their poor responses to emails/phone, and the fact that it took 70 days to sort out.

That's very interesting. After 2 weeks of all gateways looking great on my BTW migrated line, I'm seeing significant variability between gateways again for some routes.. None of them are as bad as they were on the GEA network - they range between Zen speedtests at 750Mbps up to line rate, whereas before usually Zen would test around 550Mbps. Very annoying though, just doesn't feel like a robust and well-engineered service.

I've lost interest in contacting Zen over it now. I'm just going to keep an eye on the performance, skip gateways where necessary, and high-tail out of here once my contract is done.

Seems Zen will be happy to see the back of those customers that know there is an issue, they will still have a whole heap of customers who are totally unaware of the issues and are still willing to pay them for what could be considered by some a degraded service.

I've been following this thread for some time as FTTP should be appearing here in the next 12 months and I am currently a very satisfied Zen FTTC customer and have been since 2014.

My line had a GEA migration earlier this year.

If Zen has 'issues' in its premium tier FTTP products, which other providers should I look at instead and which providers are the updaters of this thread considering moving to as I need to start doing my own research as I am completely out of touch with various providers at the moment as I've been happy with my service as it can deliver max speeds at any time that I have noticed and haven't looked at switching

Well, I was a very satisfied Zen FTTC customer for around 10 years when I lived in Norfolk, but we moved up to Cheshire, and I was delighted to have FTTP.... Originally I used BT business, but it was expensive, and they wouldn't offer more than 300Mbps down in those days, so I switched back to Zen here. All was well pre my (UNASKED for) GEA migration, but since.... It was a disaster for 3 months before I insisted on going back to BTW rather than GEA. That made everything OK, but they kept my original line as well here (I have two FTTP connections), and the GEA line continues to be disconnection city (see the screenshots I shared). Who to go to? What to do? I am staying with Zen for now, fixed IP and domain services sway it, as well as continuity, but watching the ZOOMM installation locally which offers symmetrical speeds up to 2Gbps up/down, but will it be reliable????? I am waiting to see. There are many providers, and I guess everyone has their favourites, Zen WERE the gold standard for SME's I think, but they seem to be going through a bad patch (they aren't helping themselves IMHO)

Edited by SteveBushell999 (Mon 19-Sep-22 22:04:11)

In reply to a post by Spudgun:

If Zen has 'issues' in its premium tier FTTP products, which other providers should I look at instead and which providers are the updaters of this thread considering moving to as I need to start doing my own research as I am completely out of touch with various providers at the moment as I've been happy with my service as it can deliver max speeds at any time that I have noticed and haven't looked at switching

I moved to Aquiss, I couldn't be happier with them.

I moved from Zen to File Sanctuary two months ago, and have had no issues whatsoever.

In reply to a post by dect:

Didn't Jake's issue appear to be resolved by a mystery card placement somewhere in the Openreach network without the need for him to be un-migrated. I would have expected him to be back here if his issue had come back.

Yep. Still 100% fine here on GEA line

Can't actually believe it. My performance is back down at around half line rate with faster pings.
/me checks Zen portal...
Can't even make this up. They've gone and migrated me back to ZEN GEA again.
What a joke.

Does that now explain your outage yesterday?

In reply to a post by jimbof:

Well, got through on technical support, they did "something" to fix it and it started working again.

In reply to a post by jimbof:

Can't actually believe it. My performance is back down at around half line rate with faster pings.
/me checks Zen portal...
Can't even make this up. They've gone and migrated me back to ZEN GEA again.
What a joke.

I am not sure that GEA/BTW now makes that much difference but since the cause of the poor performance/slow speeds has never been admitted to by Zen we can only guess about what the cause is.

It seemed to be cured by an Openreach visit, (who said they found nothing), but I suspect that they sorted an exchange card issue.

Alternatively, it may have been the Zen Load Balancing, (or should we just call it a Trafic Management System), even though I have read that Zen do not officially Traffic Manage was the cause of poor performance/low speeds.

The other day I felt that the performance was poorer than it was and when I checked the Zen Broadband Status site I noticed that there had been planned maintence a few days before for my line which had affected area. - Perhaps, Openreach were faffing about? - It did not say.

Since then the speed test have not been as good as they were, over 500 mbps but sometimes not the usual 900 mbps but presently it is not the same on all ooklla test sites. eg. When using the TNP ltd Manchester test site the ookla speed tests are consistently between 400 mbps and 600 mbps and the Ookla speeds reported by the test sites in Sheffield are consistenly over 900 mbps. - (At present the overall browning performance is OK and when it was bad, earlier in the year, even the browsing was dire).

I would like to know why the varoius Ookla Test Site results are so different and if these test sites somewhat match the Gateways that Zen use.

I am out of contract and every so often my finger hovers over the order BT 900 mbps button but I don't know if BT are better.

Sadly I think yes.

I had a similar outage with not being able to authenticate a PPPoE session when they migrated me from GEA to BTW. It appears the authentication stuff is specific to BTW vs GEA, and they have to move some setting or whatever over when your connection moves.

Twice now this hasn't seemed to happen properly. Once from GEA -> BTW, and now this time from BTW -> GEA. Seems like something is held together with sticky backed plastic, to lose service twice for such a dumb reason with something that you'd figure should be automated.

I can't help but feel there are some things about how the whole migration between the services happens that are note very well handled, as evidenced by my recent experiences.

Well, for me here in Norwich City Centre, GEA vs BTW makes a huge difference. I don't know if that is a local issue - perhaps the GEA links in the area are contended or somehow faulty or poorly operating - but it's happened twice now for the same line that the performance drops to a certain level for most speed tests when on the GEA network vs the BTW network.

I can't see how any BT visit here would do anything for my issue. And I'm not really sure there is even likely to be anything wrong at the exchange end of the fibre, given that performance on BTW is fine. I'm pretty sure whatever is going wrong is either in the GEA network, or the link from the OLT to the GEA network.

My speedtests on BTW are reliably over 900mbps (there is a proviso that some of the gateways don't seem to manage quite that high). With Zen GEA they're pretty much always pegged at the 500-550mbps level except to a few places - for example, on Zen GEA speedtests to some particular server from VoiceHost (Norwich) are at closer to line rate.

I'm kind of losing interest in the analysis of what or why exactly bits of their network seem to perform so poorly, and I think I'm at the point where I just want to migrate off to somewhere else. There are too many variable factors here at play in just exactly what performance you end up with - whether you get gateway'd into London or Manchester, whether your connection is migrated or not migrated, exactly what gateway you end up on, etc. It's just a poor show.

Edited by jimbof (Thu 29-Sep-22 11:54:35)

I've been seeing issues this week (central Beds) - on 900 but speed tests show a few mbps, but uploads are always around line max - resetting OTN seemed to cure it for a while.

Looking at quality monitor I've have setup on here I'm started seeing some packet loss late August, had a total loss on 30 August (had to reset router) which then gave very slow link till I reset it again next day. This week packet loss seems much higher

This sounds highly reminiscent of my experience when originally migrated to GEA, have you tried a direct PPPOE connection... It may show bad still though, but I had really variable results depending on the stack connected to the ONT (Fritz BAD, Windows 10 different to Server 2016 - same hardware, Mac apparently good) But that was a bit false as it depends on how each TCP stack handles disconnections (it seems). - I am assuming that you have been GEA migrated? Upload was always unaffected.

Edited by SteveBushell999 (Fri 30-Sep-22 15:30:02)

Portal Order history suggests I was migrated back in May and haven't really noticed any issues until this week

Not tried direct PPPOE yet and using my own router (a tri-band Asus) - currently it seems to be behaving itself

In reply to a post by Kenneth:

I was migrated back in May and haven't really noticed any issues until this week

One thing this may highlight, once you have been GEA migrated to Zen backhaul you are never safe from these issues.

I've been released from contract now by Zen, so off to pastures new. Hopefully greener.

Recommendations for FTTP 900 providers (not on Zen backhaul for sure!!!) with strong single thread and multi thread performance very much welcomed. Need ipv4 and a static address, no other frills.

I think I may have resolved my issues by enabling DoS protection on router (I think off due to it rate limiting of pings), it was almost immediate improvement on latency spikes on TBB monitor and the dropped packets reduced with a a few hours.- so not sure I can blame that on GEA

however where as when I first had the line Speed tests would show 900 mbs - I am now down to 750 max, and if I use the Zen servers 450

In reply to a post by Kenneth:

however where as when I first had the line Speed tests would show 900 mbs - I am now down to 750 max, and if I use the Zen servers 450

That's exactly the performance hit I saw from GEA, and it was absolutely repeatable. You could try speedtest to Voicehost's Norwich speedtest.net server if you can select it. That one tests at almost line rate for me on GEA still.

If you can't select it from the website, you might be able to in the speedtest standalone tester.

My router has built in speed tester provided by Ookla - which is same as zen-internet.speedtestcustom.com as linked from Zen support pages -

Zen London is one of the options and gave me just less than 450 an hour or so ago, the one labelled as dedicated London gives around 750

tracert says I going through several thn-lon.zen.net.uk (which would make sense for my location)


I'm not sure I'll be staying at contact end - but at it seems working OK at this point I live with it as most of the time I'm on WiFi and only one machine that can max the line out over wifi

In reply to a post by Kenneth:

My router has built in speed tester provided by Ookla - which is same as zen-internet.speedtestcustom.com as linked from Zen support pages -

Zen London is one of the options and gave me just less than 450 an hour or so ago, the one labelled as dedicated London gives around 750

I have noticed that the speed test performance gives vastly different results based upon which Ookla Test Site is used.

eg. When it uses the TNP ltd Manchester test site the ookla speed tests are around 500 mbps and and the speeds when the Ookla Test Sites in Sheffield are used are over 900 mbps.

I have wondered if the varoius Ookla Test Sites corrospond in some way with the Zen Gateways but I have no ides if that is the case.

In reply to a post by Kenneth:

I'm not sure I'll be staying at contact end - but at it seems working OK at this point.

Here is the rub of the matter.

I am already out of contract with Zen but I like Zen and would prefer to stay with them in the hope of consistently good service from now on, (plus the supposedly ongoing lifetime price guarantee), makes me reluctant to leave Zen while they are now mostly quite good but every so ofter my finger still hovers offer the order button for another ISP.

I do not need s static IP: BT 900 mbps are in pole position.

I have also considered Sky 900 mbps Gigafast but the Sky Website only offers 500 mbps to my address.

Talktalk seem cheaper but there is something about them that I do not like.

I don't know if BT or Sky would be any better.

If anyone leaves before me kindly let the board know how you get on elsewhere.

My Zen fixed price was £63 per month, so at the moment really not losing anything by moving away.

BT business is available with a static IP for about £56.50 per month after £80 cashback, or £59 with Halo backup.

I've been considering one of the monthly providers as I could try them and quickly work out if their performance is good. I think Cuckoo on Talktalk Business network for instance is £56 or so with a static IP.

Sky only offering 500Mbps has to be a sign of a lack of capacity in their network, so I'd imagine it is a steer clear warning if you want top performance.

Considering AAISP, but it is "reassuringly expensive"... And the 900Mbps service is a relatively recent addition, running on newly developed hardware.

In reply to a post by jimbof:

I've been released from contract now by Zen, so off to pastures new. Hopefully greener.

Recommendations for FTTP 900 providers (not on Zen backhaul for sure!!!) with strong single thread and multi thread performance very much welcomed. Need ipv4 and a static address, no other frills.

Aquiss is who I went with, they're brilliant and use BTW backhaul.

Thanks. I've just gone with AAISP, which went live this morning. Initial blip with me being connected to their older LNS that aren't quite capable of FTTP900, but their tech guys fixed it within an hour of me emailing them.
Now getting the fastest speedtest results ever to Zen's own speedtest server. On GEA it was capped around 450-550Mbps, on BTW with Zen it was around 910-915Mbps, now hitting 934Mbps:
https://www.speedtest.net/my-result/d/480464900

I've had to phone Zen's support team on Tuesday as it was getting really painful (kept putting it off as didn't need hassle at moment) - after going through the reset OTN etc with no joy, they've put monitors on and I disabled ThinkBroadband quality monitor so it didn't clash. - and Tuesday Eve/Wednesday Morning - was seeing 923mbs - but by yesterday afternoon (about time schools kick out) it had dropped 0.55 mbps down load (still 110 upload) guess how painful browsing was - later in evening seemed OK

This evening back to 923 - best until Tuesday was around 850 but normally 700 (with Zen server being 400 to 500)

I only moved to FTTP as the copper into house kept giving grief

A long way from being knowledgeable about the subject. Had very similar problems. Huge number of dropped packets. BQM should show this.

Maybe see if you had a recent GEA Migration. Orders on your Zen page. They moved quite a few recently, I believe.

I had endless problems on Zen's backhaul, Completely resolved by moving back to BT wholesale backhaul.

0.55Mbps? 550kbps? Or a typo?

not a typo it is below a mbs, i.e. only slightly faster than when I got ADSL back in 2001 - it's been that low several times this week

This morning Ookla reported 207Mbs from ZENs London server but 921 from dedicated London servers

From that, and if your 0.55Mbps was on the Zen in-house speed tester, then it (the tester) simply isn't up to the job.

Indeed. I wouldn’t place much credence in an on-net speed test server either - you want at least a few hops to an independent network to check the routed path to the internet is performant as well as on-net resource.

I wouldn't be so quick to dismiss it. At least for me the symptoms of my poor GEA connection were that many speedtest servers (inc Zen's own) would test at half line rate or less, while a small number would test at line rate. This was totally remedied by switching back to BTW, and later, after Zen put me back on GEA AGAIN, by moving away to AAISP.

There is something fruity about the Zen GEA network at least for some folk (myself included). I ended up on it twice, and twice it was rubbish, but would hit line rate to some places.

If it can be half rate, I'm sure it could be worse than that if whatever circumstances made it half rate for me, were worse somewhere else.

it was using Ookla (router has built in speed tests that uses them) to Zen, selecting other severs were giving marginally better results. Browsing the web was painfully slow, images were loading slowly (like back to dialup days) - I've no doubt it was somewhere near what I was experiencing.

Does Zen still frequent these forums? Rock solid service for years then they had a fault a few weeks ago and since then the connection has performed at about 25% of its capability.

Hope I don’t need to leave as they have served me well for about 18 years

Edited by rbw91 (Mon 07-Nov-22 21:17:46)

In reply to a post by rbw91:

Does Zen still frequent these forums?

Contact Andrew @ Zen

Check your account orders page on the Zen portal to see if you have been migrated. For me the initial fault (at the time of migration) plus reduced performance after was a sign that the migration had happened.

They did something in the exchange and mine seems back to normal and I'm seeing some speed tests 920 , though since yesterday Ookla is showing servers from Manchester

Interesting!!!

What exchane are you on and what gateway are you connected too?

Traceroute gives

3 lo0-0.bng5.thn-lon.zen.net.uk (51.148.77.133) 6.950 ms 6.929 ms 6.955 ms
4 lag-15.p1.thn-lon.zen.net.uk (51.148.73.98) 5.865 ms 5.891 ms 6.873 ms
5 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 8.072 ms lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 8.015 ms lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 8.054 ms
6 82.71.254.134 (82.71.254.134) 23.776 ms 6.234 ms 20.910 ms
7 132.185.248.51 (132.185.248.51) 7.057 ms 7.020 ms 7.011 ms
8 132.185.254.166 (132.185.254.166) 7.021 ms 6.997 ms 6.951 ms
9 132.185.254.1 (132.185.254.1) 6.907 ms 6.882 ms 6.857 ms
10 ae0.er01.telhc.bbc.co.uk (132.185.254.109) 6.859 ms 6.836 ms 9.212 ms

I'm in central Beds - about 45 miles north of London,

Hops 4/5/6 are interesting.

My guess is 4 and 5 are diagnostic checks, but look what happens on hop 6 with two out of three showing silly high values.

Not carried forward to later hops but suggest to me a peak load risk at 82.71.254.134, Zen at Telehouse London.

Hops 4, 5 and 6 are all just routers. Nothing unusual just segmenting the network. Hop 6 is an Internet facing router potentially handling over 900,000 routes so has other things on its digital mind than responding to traceroutes and it's very desirable that it rates them a low priority.

Edited by XGS_Is_On (Wed 09-Nov-22 20:12:08)

I've been a regular on these forums since 2006 with over 81k posts at least half of them on the tech side. Though not much on tech these days as almost all xDSLx questions have already been answered many times and I can't be bothered keeping up to date on FTTP and all the non-Openreach FTTH providers.

So I know all that !

Hence my "Not carried forward to later hops" obviously referring to hop 6. (There's a clue for you that I'm not ignorant about interpreting traceroutes).

I stand by my comment. What are the "Lag-15, Lag 2, Lag1, and Lag2 " names about, and why are there two different ones in hop5? Have you ever seen that before, except possibly on Zen since their downhill march began? I have never before seen a three-router hop so they are up to something unusual. Like trying to find out where their problem is so they can do something about it. They clearly haven't known for a long time.

Given that oddity and the long-running hassle Zen have been having over speeds and latency for well over a year, with the hop 6 router being the only Zen one that shows that serious 67% hiccup, perhaps you should think deeper. 82.71.254.134 is the only one of theirs in that route that decides it hasn't the time to respond normally to two of the three pings it got from that user.

Note that it didn't drop the first and third. It queued them both for over 20ms.

Edited by pluralist (Wed 09-Nov-22 21:50:58)

It’s nothing sinister…

LAG is an abbreviation for link aggregation.
They are simply splitting the traffic over multiple links to increase capacity.

The trace route on hop5 went over two links (likely the same switch, different ports), your computer sends 3 ICMP packets in total per hop.

Nothing untoward, pretty standard and becoming more common to see it in trace routes.

I know we send three pings per hop . But thanks for the LAG definition. I expect I haven't seen it in these forums due to my "virtual dropout" from the tech side of the site.

A quick skim of the wiki page proved interesting, given the apparent variety of different implementations and so on over the years. Still present even now.

Two example extracts jumped out at me:

- In addition to the IEEE link aggregation substandards, there are a number of proprietary aggregation schemes including Cisco's EtherChannel and Port Aggregation Protocol, Juniper's Aggregated Ethernet, AVAYA's Multi-Link Trunking, Split Multi-Link Trunking, Routed Split Multi-Link Trunking and Distributed Split Multi-Link Trunking, ZTE's Smartgroup, Huawei's Eth-Trunk, and Connectify's Speedify.[11] Most high-end network devices support some form of link aggregation. Software-based implementations – such as the *BSD lagg package, Linux bonding driver, Solaris dladm aggr, etc. – exist for many operating systems.

- Link aggregation offers an inexpensive way to set up a high-speed backbone network that transfers much more data than any single port or device can deliver. Link aggregation also allows the network's backbone speed to grow incrementally as demand on the network increases, without having to replace everything and deploy new hardware.

I know it's a big jump from those, (I accept I could be talking rot now), but to me they suggest Zen techies haven't fully mastered how that all works in their own backhaul. There's a huge area for hard-to-detect errors and incompatabilities.

Nearly all the user problems on it reported here after migration by Zen from BTW to their own seem to be fixed by moving them back. It's quite possible that the two networks feed into dedicated gateways. ((Two groups in London and two at Manchester).

Edit: I was too late to edit the offending post .

Edited by pluralist (Wed 09-Nov-22 23:55:23)

In reply to a post by pluralist:

I've been a regular on these forums since 2006 with over 81k posts at least half of them on the tech side. Though not much on tech these days as almost all xDSLx questions have already been answered many times and I can't be bothered keeping up to date on FTTP and all the non-Openreach FTTH providers.

So I know all that !

Okay.

In reply to a post by pluralist:

Hence my "Not carried forward to later hops" obviously referring to hop 6. (There's a clue for you that I'm not ignorant about interpreting traceroutes).

You said you thought it meant that device may be operating near capacity. It isn't. Traceroutes go to the management plane on the router if they hit it with TTL of 1. That management plane is usually powered quite differently from the forwarding plane traffic goes through and prioritises converging routing tables over sending an ICMP TTL expired. It potentially throttles and rate limits them too: Control Plane Policing.

In reply to a post by pluralist:

I stand by my comment. What are the "Lag-15, Lag 2, Lag1, and Lag2 " names about, and why are there two different ones in hop5? Have you ever seen that before, except possibly on Zen since their downhill march began? I have never before seen a three-router hop so they are up to something unusual.

They are Link Aggregation Group 15, 2, and 1 on their chassis respectively. Two or more physical interfaces combined into a logical, link aggregated interface.

There are two different ones in hop 5 because it's talking to hop 4 across multiple links using Equal Cost Multipath - ECMP. This is normal. Presumably using a single larger LAG between the two wasn't technically possible so 2 routed interfaces, ECMP, are the solution. This is all perfectly reasonable.

In reply to a post by pluralist:

Like trying to find out where their problem is so they can do something about it. They clearly haven't known for a long time.

Doesn't seem to be much to see here. All pretty basic and pretty standard stuff. May be routers with loads of 10G ports but limited to 2 or 4 members in a LAG so two of them, ECMP to share load, you've a nearly functionally equivalent solution to a larger LAG.

In reply to a post by pluralist:

Given that oddity and the long-running hassle Zen have been having over speeds and latency for well over a year, with the hop 6 router being the only Zen one that shows that serious 67% hiccup, perhaps you should think deeper. 82.71.254.134 is the only one of theirs in that route that decides it hasn't the time to respond normally to two of the three pings it got from that user.

The routers behind 82.71.254.134 talk to it and receive a subset of routes. As I said before that router is on the edge of Zen's Internet facing infrastructure and will probably have 900,000+ IPv4 routes and however many IPv6 to watch via its eBGP sessions. The routers inside in Zen's network do not need a full table. Their workload in terms of manipulating routing tables is going to be substantially lower. This is the same story for every Internet facing router. They have visibility of the mess that is the Internet and try and clean it up and present a far more pleasant view to the routers connected to them: you don't advertise a full 900,000+ entry routing table to every other device on the network. They just need to know how to reach the router with the knowledge 😊

In reply to a post by pluralist:

Note that it didn't drop the first and third. It queued them both for over 20ms.

Yes. CoPP alongside prioritisation for management plane resources. The traceroute may have landed while the CPU was processing some BGP route withdrawals and new advertised paths. It got to the traceroute when it was ready. The CoPP may also only allow a certain amount of traceroute responses with a fixed period and then throttle the responses until they're good to go.

Edited by XGS_Is_On (Thu 10-Nov-22 00:43:46)

In reply to a post by pluralist:

A quick skim of the wiki page proved interesting, given the apparent variety of different implementations and so on over the years. Still present even now.

I know it's a big jump from those, (I accept I could be talking rot now), but to me they suggest Zen techies haven't fully mastered how that all works in their own backhaul. There's a huge area for hard-to-detect errors and incompatabilities.

Well, there isn't a huge area there. If they are running LACP it negotiates how traffic is going to be balanced between the members of the lag. If they are doing something else they can monitor results. They can even set the LAG statically and specify load balancing policy there.

Given you only found out what a LAG is on here a few minutes ago then went to Wikipedia and are unaware of how to monitor one (basically just check how balanced the links are, verify how the decision is being made on which link to send a particular flow and adjust - different headers can be hashed which will produce slightly difference balancing - it's probably a tad presumptuous to assume that Zen engineers don't know how to connect LAGs together.

LAGs links die, aren't being used for some reason or one of the slave links is in strife. Every link is still visible on its own, it's 2 layer 2 links sharing an IP address and hashing deciding which of the two L2 links to use. It's not complicated.

Aggregation |= Segregation. They are complete opposites.

Maybe you have since looked up LAG in rather more detail than I did. There's no way that could be a typo.

What is perfectly reasonable is not necessarily perfectly implemented. You don't seem to be aware how the tiniest error can be made or incompatibility can be present across such a complex technology. Or how the tiniest component in electronics can develop an intermittent fault or be outside "guaranteed" tolerances and only show up under a very rare combination of circumstances in your 900,000 combinations of circumstances.

Guess what! Skimmed milk masquerades as cream. (You may need to google that).

I'm glad I never had a person with such belief in the infallibility of themselves, others or equipment as yourself working for me in IT.

Edit: Early morning error reading the post I replied to, so the first two paragraphs were irrelevant. I stand by the rest of my post.

Edited by pluralist (Thu 10-Nov-22 11:53:38)

In reply to a post by XGS_Is_On:

It's not complicated.

In which case explain why Zen engineers (and their equipment suppliers) haven't solved the problem(s) with their own backhaul over such a long period. Yet the BTW backhaul doesn't have problems with the same Zen customers either before migration to their own or reversion back to BTW.

Perhaps they need you to pop in for an afternoon and sort it out for them.

"It's not complicated". Yes it is.

In reply to a post by pluralist:

Aggregation |= Segregation. They are complete opposites.

Maybe you have since looked up LAG in rather more detail than I did. There's no way that could be a typo.

What is perfectly reasonable is not necessarily perfectly implemented. You don't seem to be aware how the tiniest error can be made or incompatibility can be present across such a complex technology. Or how the tiniest component in electronics can develop an intermittent fault or be outside "guaranteed" tolerances and only show up under a very rare combination of circumstances in your 900,000 combinations of circumstances.

Guess what! Skimmed milk masquerades as cream. (You may need to google that).

I'm glad I never had a person with such belief in the infallibility of themselves, others or equipment as yourself working for me in IT.

No use of 'segregation' in my post. 'Segmentation' in an earlier one as that's part of what routers do and why we have subnet masks.

Your suggesting I'm arrogant having claimed Zen may have made a mistake with basic functionality having just read about it on Wikipedia is beyond comedy. The appeal to authority you made regarding how long you'd been on the forum for set the scene and you just ran with it.

May I suggest using Wikipedia or similar to find the difference between control planes and forwarding planes as a start rather than the appeal to forum authority and indignance at being challenged?

As clearly I know nothing and am being arrogant for disagreeing I'll defer to yourself, your Wikipedia research and the Dunning-Kruger effect.

In reply to a post by XGS_Is_On:

In reply to a post by pluralist:

Aggregation |= Segregation. They are complete opposites.
...
I'm glad I never had a person with such belief in the infallibility of themselves, others or equipment as yourself working for me in IT.

No use of 'segregation' in my post. 'Segmentation' in an earlier one as that's part of what routers do and why we have subnet masks.

Correct. That part of my post now struck out but I stand by the rest of it. See the edit line added there. However aggregation as helpfully pointed out by devonkev (unlike yourself) is still a different matter.

Your suggesting I'm arrogant having claimed Zen may have made a mistake with basic functionality having just read about it on Wikipedia is beyond comedy.

You are the comedian.Your grammar and punctuation is terrible. There are at least three very different ways of construing that sentence. As it stands the most obvious reading is that I suggested you claimed Zen may have made a mistake. (That's if we accept your opening "Your" as "You're").

I made no such suggestion. However I do see the insulting (for the second time) intention of what you were trying to express.

The appeal to authority you made regarding how long you'd been on the forum for set the scene and you just ran with it.

My post did not claim authority, just stated the period of time and experience during which I had neve seen an LAG-labelled hop. Particularly one with three routers in the same hop. Given your poor writing I suppose your misunderstanding is to be expected. I do know and have for many years known everything you said in the post it was a reply to. Really elementary stuff. But nothing whatsoever to do with hop 6.

May I suggest using Wikipedia or similar to find the difference between control planes and forwarding planes as a start rather than the appeal to forum authority and indignance at being challenged?

Now you are simply being childish because you are losing every time you have a go at me bar my one sleepy slip-up.

As clearly I know nothing and am being arrogant for disagreeing I'll defer to yourself, your Wikipedia research and the Dunning-Kruger effect.

Well, you do seem to have a bit of course-work knowledge about networking, but how deep and that is and how much experience and capability in complex problem diagnosis and solving is a different question altogether.

The Zen people I'm sure know far more than you, and yes you are arrogant. Particularly in saying that any particular Link Aggregation strategy and implementation is simple. No network designer can know everything that might be presented to the system by legitimate users! It's easier to guard against deliberate threats by hackers, and that's hard enough.

Maybe in another five or ten years you'll begin to realise how little you currently know about reality.

Going to take a few minutes to just fill out a bit on the other posts I've made on this topic briefly rather than getting into a urinating contest with the forum technical Gods.

If you, as a Zen user, have seen ae in a traceroute you've gone over a LAG - Juniper equipment, and others, name LAGs aggregated Ethernet. I remember seeing this plenty when I used Zen retail.

Cisco call them Port Channels, so links with 'po' and a number often indicate Cisco LAGs.

All these do is, when a packet needs to use the link, basically do a bit of maths and work out which physical link to use, that's it. When there are 2 links it's essentially nothing more than distilling the packet headers down to odd or even. Where there are 4 it's turning those headers into a number, dividing by 4 and counting the remainder, then using link 0,1,2 or 3. Once that's done every packet in that connection will use the same link. Different connections will use different links,so the slow speed will be inconsistent. Anything, whether GEA, BTWholesale or TalkTalk, that goes over those links will experience the same exact issues. If the two sides of the LAG choose different links it doesn't matter - these are cables connecting the same chassis and likely same line cards either side that are probably running next to each other and physically cable managed together in a bundle.

TL;DR it's extremely unlikely to be anything to do with that.

ECMP is nothing more than having multiple routes to the same place and using all of them. This can and does create asymmetry and jitter. However, how much jitter do you reckon is possible on links between the same chassis in the same building, probably the same line of racks and don't you think if there were issues like that you'd see them in the following hops?

Traceroute: https://www.thousandeyes.com/learning/glossary/trace...

Traceroute most commonly uses Internet Control Message Protocol (ICMP) echo packets with variable time to live (TTL) values. The response time of each hop is calculated.

Time To Live is sent starting at 1, then 2, and so on until you reach the destination. Most devices that receive this and aren't tunneling your packets will count the TTL down when they receive the packet. If the TTL becomes zero they will send this to their control plane or a slow routing path to be dealt with, as it requires some action from the device itself above just forwarding it.

https://traceroute.home.blog/category/general-networ... has a nice quote:

In terms of configuration the control plane should be considered as an interface though which any traffic destined for the device must pass. This traffic can enter through any physical interface, but before it is processed it passes through the control plane “interface”.

Packets with TTL 1 hitting the router enter the control plane or a 'slow' data plane as they can't go further. They must be dealt with by this device and need to have an ICMP TTL expired message generated. The time it takes to generate and send this message and for it to arrive back at you is your latency for that hop in the traceroute. Where the TTL is above 1 the router will send it on through the fast data plane / forwarding path which in the case of these routers is a high speed, high bandwidth very low latency Application Specific Integrated Circuit routing and switching fabric.

There are, of course, buffers in between where this data hits the router and the CPUs that handle the control plane and slow data plane. When these buffers fill, or if there is policing in place to drop everything that isn't answered immediately, the traceroute is dropped. Until then packets wait for the CPU to service them and send out the ICMP TTL expired message. This is both right at the bottom of the priority list for the CPU and will in itself be throttled to protect the system so you've policing both entering the control plane in the first place and within generating the ICMP messages. This is to protect the router from having its CPU drained.

That set of CPUs likely have some work to do to handle the huge routing table. They probably export telemetry, handle generation of alerts, etc, etc. As long as nothing is seen after that hop in a traceroute the higher latency isn't an issue.

https://hal.inria.fr/hal-01111190/document

The main problem that arises when making use of TTL-limited probes is that ICMP feedback from routers is often neither instantaneous nor entirely reliable. Indeed, as the generation of ICMP error messages takes place in the slow path of the data plane, manufacturers and operators impose a low priority on it, in order to minimize the overall load on routers. Other internal tasks mostly related to the control plane, like route computation and management operations, might take precedence over it, especially when resources are shared between slow path and control plane

If nothing is seen after the hops using LAGs / ECMP it's nothing to do with that either.

That's it. I appreciate the frustration, this thread is huge, but don't get sidetracked with this stuff. ANY issues with the LAGs, ECMP or the higher ping response coming from the Zen edge router would, if relevant, show throughout the traceroute down. It really is that simple - you'd see latency, loss or jitter throughout. You don't.

I appreciate I don't have 40,000+ posts in this forum's technical section however if you are really, really bored you can read https://www.rfc-editor.org/rfc/rfc7747.html for why some edge devices have higher latency when being pinged and the truly excruciating https://www.rfc-editor.org/rfc/rfc4098 for control plane / forwarding/data plane stuff. I have no intention of reading either.

The issues almost certainly relate to Zen's Plexus network, hence why some people on it are fine, others aren't, even when they're going across the dreaded ECMP LAGs.

Coffee break over. Cheers.

EDIT: Just to reiterate, Pluralist, I am not reading your posts here since I responded to the last one, it being abundantly clear it's a waste of both my time reading them and yours writing them. This post is for the benefit of those actually interested in this and to avoid support tickets flying into Zen because people see 'lag' in a traceroute and think it's a problem. I'm sure if you look hard enough look through some documentation you can find something to nitpick, I've simplified a ton, however quoting documentation on something you only realised existed yesterday says a lot more about you than it does me so your call whether you waste your time trying to be 'right' or keep that to The Park.

EDIT 2: It's overly simplistic to not mention that there are 2 data plane forwarding paths in many routers, the fast path and the slow one, and in some cases the slow forwarding path, usually sharing resources with the control plane, handles traceroute responses. In others it goes to the control plane itself, it depends, but for clarity should be mentioned and cuts off some pendaticism even though the end result is the same.

Edited by XGS_Is_On (Thu 10-Nov-22 14:00:20)

I assume each flow is kept to the same port for the duration of the flow?

Yes.

For instance:

https://support.huawei.com/enterprise/en/doc/EDOC110...

Load Balancing in LACP Mode section.

LACP only supports flow-based balancing. Once the calculation is done a flow is sticky to that link as that result isn't changing. Into fast path on its way that goes.

Thats good to hear, so I am not sure what the issue would be with LAG then, it would be similar to RSS and the like.

Me either. The only time I've seen issues the symptoms were some links in the LAG not being used and the remaining ones congesting as a result, such as when software is limited to using 4 links in a LAG but allows 8 to be configured. Things like that alongside extremely basic configuration errors that would result in some or even all of the LAG not coming up and worst case leaving a single link alive as fallback to congest.

This is pretty obvious in its effects right down the route after the congestion point but I'll leave it to the expert to comment further.

Post deleted by XGS_Is_On

Tagging this on the end. Apparently all now resolved after…ahem “2 weeks”. Wishful thinking perhaps?

https://www.ispreview.co.uk/index.php/2022/11/uk-isp...

In reply to a post by Pheasant:

Apparently all now resolved after…ahem “2 weeks”. Wishful thinking perhaps?

I'm sure all Zen customers will be able to get a goodnights sleep tonight knowing Zen have spoken 😎🤣

So they were not bothered by retail/SOHO customers' complaints. It was only when their Wholesale customers kicked up that they bothered to look into it properly.

However - the explanation given by their Mark Howarth looks a little too simple to me, given they are two different networks that needed rebalancing between them. Surely at least their retail customers are on one or the other? With reversion to BTW solving their problems?

Very detail-light indeed. It's a shame most of us who noticed the issues have since migrated off Zen GEA or to other providers, guess we will have to take their word for it!

Given claim it was ongoing for a couple of weeks (Vs the experience of folk here of heading up for half a year) you can colour me sceptical...

Edited by jimbof (Sat 19-Nov-22 11:23:53)

In reply to a post by jimbof:

Very detail-light indeed. It's a shame most of us who noticed the issues have since migrated off Zen GEA or to other providers, guess we will have to take their word for it!

Given claim it was ongoing for a couple of weeks (Vs the experience of folk here of heading up for half a year) you can colour me sceptical...

Is this problem still going on? I have been a FTTC customer with Zen for years, I had a regrade to FTTP Zen 500 today but speeds are 200Mbit and vary wildly. I am unsure what backbone I am on.

I can say that a friend 10 minutes away in the same town gets full speed without issues.

I spoke to Zen and they made me test from the ONT with a CAT5e cable (I tried 10 cables) and I can't get anything over 220Mbit, with one result of 250Mbit, now they say they can't help me because I hit the minimum speed guarantee so I am really annoyed as I only get 50% of what I pay for and they don't care.

Am I better of just moving in the cooling off period?

I don't know if it is still ongoing as I left. Performance has been great since leaving, with the same equipment at my end.

You might want to check the terms of the cooling off. It's usually done from point of order (not point of install), and they usually set the install date to be beyond the end of the cooling off (so you can only cancel penalty free before install).

If this is still going on - and they are still trotting out the "above minimum guarantee" bull by means of justification for not doing anything - then I'd be trying to get and recommending loudly to folk via any channels open (eg Trustpilot reviews, etc) not to go with them.

I seem to remember someone having exactly the same problem a few weeks ago. IIRC it turned out nobody had updated the user's line settings to reflect/match the new Openreach service.

Edit: Found the thread. Not the same reported problem but similar. The upstream Zen profile for the line was still at 70Mbps.

If your upstream is limited that could affect downstream. Or maybe the downstream profile hasn't been updated either.

(Openreach require all providers to have a downstream limiter like the Plusnet one, to prevent data from the provider being sent faster than the line is set to by Openreach. If it is sent faster then packet loss and retransmission happens).

Edited by pluralist (Fri 02-Dec-22 12:15:22)

@wizz

I spoke to Zen and they made me test from the ONT with a CAT5e cable (I tried 10 cables) and I can't get anything over 220Mbit, with one result of 250Mbit,

Not entirely sure this is profile related.

Did Zen ask you to test using a Linux boot image rather than Windows?

What machine are you testing with? Could be your machine holding things back if it’s knocking on a bit or running antivirus/spamware protection which can hold things back network wise, especially at higher bit rates.

Hi,

So yes I have tried two different laptops directly connected to the ONT. I only have windows devices unfortunately. I also moved a desktop machine up near the ONT and plugged that in. All these devices are not that old. one laptop is a I5 8th Gen the other is a 10th Gen I7 with an NVME drive, the desktop is a little older.

In all tests the speeds are between 150 to 250Mbit and its so unstable, no test is the same, its hard to keep track of the changes in speeds, 1 test will be the bare minimum speed of 250 and then the next it will be in the low 100's

I could boot ubuntu and test but I don't see why I should, operating system should make very little difference and if it does something is messed up elsewhere outside of my network


They have now escalated the case and have said I am not getting the minimum speed required but it's taken a 2 hour wait on hold and then 45 minutes to get anywhere! Terrible service.

I don't think they have left me on an old profile, prior I had FTTC 70down 20up and got it never dropped below that, so now I get great 70Mbit upload but down is no where near the 500

In reply to a post by Pheasant:

@wizz

I spoke to Zen and they made me test from the ONT with a CAT5e cable (I tried 10 cables) and I can't get anything over 220Mbit, with one result of 250Mbit,

Not entirely sure this is profile related.

Did Zen ask you to test using a Linux boot image rather than Windows?

What machine are you testing with? Could be your machine holding things back if it’s knocking on a bit or running antivirus/spamware protection which can hold things back network wise, especially at higher bit rates.

Edited by Wizz_The_Great (Fri 02-Dec-22 17:12:39)

In reply to a post by Wizz_The_Great:

I could boot ubuntu and test but I don't see why I should, operating system should make very little difference

You don't have to but it's a very good test for ruling out your own hardware.

Operating system can make a dramatic difference in throughput. Older machines can hit gigabit on Linux while barely hitting half that on Windows.

A Linux boot image provides a blank operating system which can also rule out software firewalls, anti-virus, anti-malware etc or even unknown malicious software on your system.

It does sound from the specs like it's not your hardware at fault but there are good reasons why it is suggested as a troubleshooting step.
This forum has had a considerable number of slow speed complaints from people convinced it wasn't their own hardware at fault and when advised to try it they received full speed on a Linux boot iso. That's become more common since gigabit take up had increased.
It's usually down to old hardware, security software or a combination of both.

You could be running the same security software on both machines which is affecting throughput.

and if it does something is messed up elsewhere outside of my network

Not at all.
If a Linux boot disc gets full speed then it would mean the issue is within your network.
I have more than 1 Windows machine (admittedly older) with a Gigabit port that can't get near gigabit on Windows. They don't hit gigabit on Linux either but they are significantly faster than on Windows.
I couldn't blame my provider for that.

The cooling off period starts the day you placed your order and not the day the service went live.
For many orders this means the cooling off period is over by the time service starts.

The likes of BT & Virgin start the 14 days from the service starting but that's their own decision. The consumer protection regulations start when you place the order and is what most providers use.
I'm not sure about Zens cooling off period.

Zen's cooling of period does start from date of order not date service went live so unfortunately you may be outside of this now.

As someone who has worked in ISP tech support I can confirm that I have seen instances where speeds have been low on windows but on the same hardware they have hit 900Mb plus on Linux. Windows really isn't the best OS and this would rule out an issue with Windows networking/drivers and antivirus software. It wouldn't take too long to load Ubuntu Live USB and do a few tests and wouldn't require any permanent changes to you PC.

If after this you are still getting low speeds it gives you more ammo to through at Zen in proving the issue is not at your side.

Well I took both laptops to my friends house 10 minutes away who has Zen 500 and got perfect speeds on this hardware on his line so that answers that question!

How can the cooling off period start when you order? They didn't install the line until day 16 into my house, that makes no sense!

In reply to a post by johnno101:

Zen's cooling of period does start from date of order not date service went live so unfortunately you may be outside of this now.

As someone who has worked in ISP tech support I can confirm that I have seen instances where speeds have been low on windows but on the same hardware they have hit 900Mb plus on Linux. Windows really isn't the best OS and this would rule out an issue with Windows networking/drivers and antivirus software. It wouldn't take too long to load Ubuntu Live USB and do a few tests and wouldn't require any permanent changes to you PC.

If after this you are still getting low speeds it gives you more ammo to through at Zen in proving the issue is not at your side.

It's a cooling off period to decide if the contract you signed was right for you, it's not designed as a way to trial the service and then cancel it.

How can you make that decision without trying the service first, it still doesn't make any sense. The 14 day cooling off period is then basically useless at protecting the consumer.

Edited by Wizz_The_Great (Mon 05-Dec-22 10:44:05)

Pretty much and made even worse by the enforced delay on migrations.

Because that isn't what the cooling off period was ever intended for - it's not meant as a try-and-buy where the provider incurs £100+ of installation charges and then you cancel without penalty.

If the service isn't performing then consumer law that has existed for a long time outside of Ofcom's own regulations will protect you.

If you are using the same hardware and Ethernet cables then at least you have ruled this out and this would suggest the issue definitely is with Zen. Out of curiosity do you know if you are using Zen's own GEA network or on BT Wholesale or TalkTalk Business?

Most of the issues in the thread were for people using Zen's own GEA network, BT wholesale lines were not affected.

I am unsure, because it was a regrade from FTTC to FTTP the order area on the Zen portal doesn't state anything.

I asked the technical person from Zen on the phone and he didn't know but said "I think you are on BTW" but that doesn't mean anything if he didn't know for sure.

Either way they booked an engineer for Wednesday but I doubt the engineer will be able to fix it if its Zen's backend, they have sent me an email confirming I am not getting the minimum speed and if it can't be fixed in 30 days I can get out of the contract so that's something.

Can't recommend Zen anymore, speed is only getting worse as the days go on as well

In reply to a post by johnno101:

If you are using the same hardware and Ethernet cables then at least you have ruled this out and this would suggest the issue definitely is with Zen. Out of curiosity do you know if you are using Zen's own GEA network or on BT Wholesale or TalkTalk Business?

Most of the issues in the thread were for people using Zen's own GEA network, BT wholesale lines were not affected.

If the engineer gets there and finds no physical fault it may be worth asking him to phone Openreach DCoE and get them to check the build and perform a rebuild. Zen should have already told them to do this but this may help if the build isn't correct and get you on the right profile.

When did your friends connection go in? When did yours go live?

I'd be amazed if Zen were doing FTTP migrations onto the BT Wholesale platform simply because it's far more expensive than using their own network. They don't seem to care that their own network seems to be broken. Chances are if your friend has been a customer for longer than a year then they are on BT Wholesale.

Have you tried using your friends PPPoE details and they try using yours?

I have not tried my PPPoE details at my friends yet but he has only just had it installed a week ago and I have had mine over a year so this makes sense that I'm on BTW and he is on Zen's platform.

I will try to do this but my connection is always in use by someone in the house so its a bit difficult

In reply to a post by jpm:

When did your friends connection go in? When did yours go live?

I'd be amazed if Zen were doing FTTP migrations onto the BT Wholesale platform simply because it's far more expensive than using their own network. They don't seem to care that their own network seems to be broken. Chances are if your friend has been a customer for longer than a year then they are on BT Wholesale.

Have you tried using your friends PPPoE details and they try using yours?

In reply to a post by jpm:

When did your friends connection go in? When did yours go live?

I'd be amazed if Zen were doing FTTP migrations onto the BT Wholesale platform simply because it's far more expensive than using their own network. They don't seem to care that their own network seems to be broken. Chances are if your friend has been a customer for longer than a year then they are on BT Wholesale.

Have you tried using your friends PPPoE details and they try using yours?

There is some link between PPPoE details and whether you are on GEA or BTW - Zen have to make some server change when you move between them. I know this because twice I was migrated and the server change didn't happen, which left me without a working connection (as the authentication was set up on the other network still).

I'm not sure if that is the only thing tying the accounts to the service though, and whether it is possible to (if all other things are equal) move between services.

The OP could try to push Zen to sending one of the Samknows boxes to you to see how that behaves, I don't doubt it would also likely record poorer than expected performance. Mine was night and day when moving between Zen and BTW networks.

A symptom I noticed of the poor performance on the Zen GEA network was that most speedtest.net servers were poor, but there were a small number that were almost at line speed still. Swish fibre was one with OK performance, Voicehost (Norwich) was another, be interesting to know if any such difference is observed. Zen's own server was poor, as were most others. It's quite easy on the web interface or command line tool to specify particular servers.