Traceroute gives

3 lo0-0.bng5.thn-lon.zen.net.uk (51.148.77.133) 6.950 ms 6.929 ms 6.955 ms
4 lag-15.p1.thn-lon.zen.net.uk (51.148.73.98) 5.865 ms 5.891 ms 6.873 ms
5 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 8.072 ms lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 8.015 ms lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 8.054 ms
6 82.71.254.134 (82.71.254.134) 23.776 ms 6.234 ms 20.910 ms
7 132.185.248.51 (132.185.248.51) 7.057 ms 7.020 ms 7.011 ms
8 132.185.254.166 (132.185.254.166) 7.021 ms 6.997 ms 6.951 ms
9 132.185.254.1 (132.185.254.1) 6.907 ms 6.882 ms 6.857 ms
10 ae0.er01.telhc.bbc.co.uk (132.185.254.109) 6.859 ms 6.836 ms 9.212 ms

I'm in central Beds - about 45 miles north of London,

Hops 4/5/6 are interesting.

My guess is 4 and 5 are diagnostic checks, but look what happens on hop 6 with two out of three showing silly high values.

Not carried forward to later hops but suggest to me a peak load risk at 82.71.254.134, Zen at Telehouse London.

Hops 4, 5 and 6 are all just routers. Nothing unusual just segmenting the network. Hop 6 is an Internet facing router potentially handling over 900,000 routes so has other things on its digital mind than responding to traceroutes and it's very desirable that it rates them a low priority.

Edited by XGS_Is_On (Wed 09-Nov-22 20:12:08)

I've been a regular on these forums since 2006 with over 81k posts at least half of them on the tech side. Though not much on tech these days as almost all xDSLx questions have already been answered many times and I can't be bothered keeping up to date on FTTP and all the non-Openreach FTTH providers.

So I know all that !

Hence my "Not carried forward to later hops" obviously referring to hop 6. (There's a clue for you that I'm not ignorant about interpreting traceroutes).

I stand by my comment. What are the "Lag-15, Lag 2, Lag1, and Lag2 " names about, and why are there two different ones in hop5? Have you ever seen that before, except possibly on Zen since their downhill march began? I have never before seen a three-router hop so they are up to something unusual. Like trying to find out where their problem is so they can do something about it. They clearly haven't known for a long time.

Given that oddity and the long-running hassle Zen have been having over speeds and latency for well over a year, with the hop 6 router being the only Zen one that shows that serious 67% hiccup, perhaps you should think deeper. 82.71.254.134 is the only one of theirs in that route that decides it hasn't the time to respond normally to two of the three pings it got from that user.

Note that it didn't drop the first and third. It queued them both for over 20ms.

Edited by pluralist (Wed 09-Nov-22 21:50:58)

It’s nothing sinister…

LAG is an abbreviation for link aggregation.
They are simply splitting the traffic over multiple links to increase capacity.

The trace route on hop5 went over two links (likely the same switch, different ports), your computer sends 3 ICMP packets in total per hop.

Nothing untoward, pretty standard and becoming more common to see it in trace routes.

I know we send three pings per hop . But thanks for the LAG definition. I expect I haven't seen it in these forums due to my "virtual dropout" from the tech side of the site.

A quick skim of the wiki page proved interesting, given the apparent variety of different implementations and so on over the years. Still present even now.

Two example extracts jumped out at me:

- In addition to the IEEE link aggregation substandards, there are a number of proprietary aggregation schemes including Cisco's EtherChannel and Port Aggregation Protocol, Juniper's Aggregated Ethernet, AVAYA's Multi-Link Trunking, Split Multi-Link Trunking, Routed Split Multi-Link Trunking and Distributed Split Multi-Link Trunking, ZTE's Smartgroup, Huawei's Eth-Trunk, and Connectify's Speedify.[11] Most high-end network devices support some form of link aggregation. Software-based implementations – such as the *BSD lagg package, Linux bonding driver, Solaris dladm aggr, etc. – exist for many operating systems.

- Link aggregation offers an inexpensive way to set up a high-speed backbone network that transfers much more data than any single port or device can deliver. Link aggregation also allows the network's backbone speed to grow incrementally as demand on the network increases, without having to replace everything and deploy new hardware.

I know it's a big jump from those, (I accept I could be talking rot now), but to me they suggest Zen techies haven't fully mastered how that all works in their own backhaul. There's a huge area for hard-to-detect errors and incompatabilities.

Nearly all the user problems on it reported here after migration by Zen from BTW to their own seem to be fixed by moving them back. It's quite possible that the two networks feed into dedicated gateways. ((Two groups in London and two at Manchester).

Edit: I was too late to edit the offending post .

Edited by pluralist (Wed 09-Nov-22 23:55:23)

In reply to a post by pluralist:

I've been a regular on these forums since 2006 with over 81k posts at least half of them on the tech side. Though not much on tech these days as almost all xDSLx questions have already been answered many times and I can't be bothered keeping up to date on FTTP and all the non-Openreach FTTH providers.

So I know all that !

Okay.

In reply to a post by pluralist:

Hence my "Not carried forward to later hops" obviously referring to hop 6. (There's a clue for you that I'm not ignorant about interpreting traceroutes).

You said you thought it meant that device may be operating near capacity. It isn't. Traceroutes go to the management plane on the router if they hit it with TTL of 1. That management plane is usually powered quite differently from the forwarding plane traffic goes through and prioritises converging routing tables over sending an ICMP TTL expired. It potentially throttles and rate limits them too: Control Plane Policing.

In reply to a post by pluralist:

I stand by my comment. What are the "Lag-15, Lag 2, Lag1, and Lag2 " names about, and why are there two different ones in hop5? Have you ever seen that before, except possibly on Zen since their downhill march began? I have never before seen a three-router hop so they are up to something unusual.

They are Link Aggregation Group 15, 2, and 1 on their chassis respectively. Two or more physical interfaces combined into a logical, link aggregated interface.

There are two different ones in hop 5 because it's talking to hop 4 across multiple links using Equal Cost Multipath - ECMP. This is normal. Presumably using a single larger LAG between the two wasn't technically possible so 2 routed interfaces, ECMP, are the solution. This is all perfectly reasonable.

In reply to a post by pluralist:

Like trying to find out where their problem is so they can do something about it. They clearly haven't known for a long time.

Doesn't seem to be much to see here. All pretty basic and pretty standard stuff. May be routers with loads of 10G ports but limited to 2 or 4 members in a LAG so two of them, ECMP to share load, you've a nearly functionally equivalent solution to a larger LAG.

In reply to a post by pluralist:

Given that oddity and the long-running hassle Zen have been having over speeds and latency for well over a year, with the hop 6 router being the only Zen one that shows that serious 67% hiccup, perhaps you should think deeper. 82.71.254.134 is the only one of theirs in that route that decides it hasn't the time to respond normally to two of the three pings it got from that user.

The routers behind 82.71.254.134 talk to it and receive a subset of routes. As I said before that router is on the edge of Zen's Internet facing infrastructure and will probably have 900,000+ IPv4 routes and however many IPv6 to watch via its eBGP sessions. The routers inside in Zen's network do not need a full table. Their workload in terms of manipulating routing tables is going to be substantially lower. This is the same story for every Internet facing router. They have visibility of the mess that is the Internet and try and clean it up and present a far more pleasant view to the routers connected to them: you don't advertise a full 900,000+ entry routing table to every other device on the network. They just need to know how to reach the router with the knowledge 😊

In reply to a post by pluralist:

Note that it didn't drop the first and third. It queued them both for over 20ms.

Yes. CoPP alongside prioritisation for management plane resources. The traceroute may have landed while the CPU was processing some BGP route withdrawals and new advertised paths. It got to the traceroute when it was ready. The CoPP may also only allow a certain amount of traceroute responses with a fixed period and then throttle the responses until they're good to go.

Edited by XGS_Is_On (Thu 10-Nov-22 00:43:46)

In reply to a post by pluralist:

A quick skim of the wiki page proved interesting, given the apparent variety of different implementations and so on over the years. Still present even now.

I know it's a big jump from those, (I accept I could be talking rot now), but to me they suggest Zen techies haven't fully mastered how that all works in their own backhaul. There's a huge area for hard-to-detect errors and incompatabilities.

Well, there isn't a huge area there. If they are running LACP it negotiates how traffic is going to be balanced between the members of the lag. If they are doing something else they can monitor results. They can even set the LAG statically and specify load balancing policy there.

Given you only found out what a LAG is on here a few minutes ago then went to Wikipedia and are unaware of how to monitor one (basically just check how balanced the links are, verify how the decision is being made on which link to send a particular flow and adjust - different headers can be hashed which will produce slightly difference balancing - it's probably a tad presumptuous to assume that Zen engineers don't know how to connect LAGs together.

LAGs links die, aren't being used for some reason or one of the slave links is in strife. Every link is still visible on its own, it's 2 layer 2 links sharing an IP address and hashing deciding which of the two L2 links to use. It's not complicated.

Aggregation |= Segregation. They are complete opposites.

Maybe you have since looked up LAG in rather more detail than I did. There's no way that could be a typo.

What is perfectly reasonable is not necessarily perfectly implemented. You don't seem to be aware how the tiniest error can be made or incompatibility can be present across such a complex technology. Or how the tiniest component in electronics can develop an intermittent fault or be outside "guaranteed" tolerances and only show up under a very rare combination of circumstances in your 900,000 combinations of circumstances.

Guess what! Skimmed milk masquerades as cream. (You may need to google that).

I'm glad I never had a person with such belief in the infallibility of themselves, others or equipment as yourself working for me in IT.

Edit: Early morning error reading the post I replied to, so the first two paragraphs were irrelevant. I stand by the rest of my post.

Edited by pluralist (Thu 10-Nov-22 11:53:38)

In reply to a post by XGS_Is_On:

It's not complicated.

In which case explain why Zen engineers (and their equipment suppliers) haven't solved the problem(s) with their own backhaul over such a long period. Yet the BTW backhaul doesn't have problems with the same Zen customers either before migration to their own or reversion back to BTW.

Perhaps they need you to pop in for an afternoon and sort it out for them.

"It's not complicated". Yes it is.