Routing via Manchester (but a bit different this time)

Since switching to cityfibre I've been mostly connected to one of the Manchester BNGs (despite living very much in the south, and orders of magnitude closer to London than Manchester).

Now we all saw this before on BT FTTC and you could just bounce and more often than not end up on a London PPPoE endpoint and solve all your problems.

But for me at least on cityfibre, that's not been the case. I was first connecting to Manchester BNGs and I could see that (as would make sense) the BNG first hop was 8ms and then almost everything of course routes via London and had a 15ms ping.

So, after a while of waiting for it to fix itself, I sat down and bounced the connection until I got a telehouse north BNG connection. Great, right?

Well, no. It seems that either cityfibre or Zen are routing me to Manchester behind the scenes. Now, first hop is the telehouse north BNG with 15ms RTT.

I was wondering if anyone else is seeing this. I was also wondering if a factory reset of the ONT might do something (seems like a long shot, but perhaps it stores some cityfibre node it connects to).

It's minor in the scheme of things, and I'm not THAT bothered. But of course if I could get a lower RTT I'd not say no.

Did you ever manage to get this sorted as I think it's happened to me now? Last night the router dropped my CityFibre Zen connection briefly at around 2:30am and now I'm connecting via Manchester despite being in Berkshire.

You can see the result on the BQM graph below. I tried rebooting the ONT and router a couple of times at around 11am.

My Broadband Ping

I have VPN connections to two locations in Finland and today the connection speed to the two locations has dropped from about 45Mbps to 25Mbps. Not sure if that's just a coincidence.

I'm afraid not. Also, it seems they've made it much much much less likely to end up on a London BNG if it sees you as coming in from Manchester.

I put in a support ticket about it. Mainly because, with more people connecting at these speeds if it's quite common it's really going to put a lot of pressure on their internal connection between Manchester and London. Almost all of my traffic goes from my location (not far from Heathrow airport) to Manchester and then back down to telehouse and out to the internet.

I got a really good detailed response, essentially saying they're aware of the problem and there's not much they can do about it right now. I replied thanking the guy for the detailed response. At which point I got a rather obtuse reply telling me that 20ms is "fine" and I shouldn't be complaining. Someone that clearly didn't read the content of my ticket at all. Since if they did, they would have seen I was more concerned with the traffic beting routed up/down the country on already congested links unnecessarily.

So, support at Zen has become a bit chequered, it seems. You either get excellent, or pretty much dire responses now, it seems.

I've tried the following to see if it helped. None did.
- Reset PPP session until getting London based BNG. As mentioned before, this just results in the same ping, but it looks like you're directly connected to London. I get a better speedtest (always >900) this way. But no difference in ping.
- Disconnect router and ONT for 1hr+. This resulted in no change.
- Resetting ONT using the reset pinhole. This resulted in no change.

This seems to be a cityfibre/zen problem that, I'm sure they MUST be noticing since my speedtests vary wildly when connected to Manchester. Anything from 600-900 on downstream but generally 900+ on upstream. It really suggests their internal connectivity is being impacted by this problem. But if I do manage to get onto a London BNG (as I say, it is pretty much almost impossible last time I tried), I get more consistent speeds. But, I suspect now the load is somehow on cityfibre's end?

Like I say, right now, it's a minor problem. 20ms IS a decent latency and nothing to complain about, even in FPS games. But, the longer term impact as they sign up more customers, if there's many being routed this way is my concern. And as I say I already see it in speed test results.

In reply to a post by agent_r00t:

they would have seen I was more concerned with the traffic beting routed up/down the country on already congested links unnecessarily.

Who says those links are "already congested"? Do you have any evidence to back that up?

Maybe they have dual 400G circuits and it's peaking at 100G of traffic(*), in which case, they're not at all worried about traffic over that link. Indeed, given there are LNSes in Manchester and most of the Internet is reached via London, I would expect that link to be very closely monitored and over-provisioned.

(*) I just made up those figures, as an example

Well, if they create bigger circuits to deal with routing everyone in the south via Manchester, I'd lose the respect I have for Zen. It's a totally nonsensical way to deal with the situation.

It's not about how much bandwidth they have, it's that they're needlessly routing customers in the south to North and then back south again.

I would expect though, that given the rest of my comment where I make it clear that when on the London BNGs I get a distinctly higher average speed than when connected via the Manchester ones. That would suggest there might be some level of congestion involved.

Actually, here's an example. These tests were done back to back.

Vodafone Manchester: https://www.speedtest.net/result/16319556697 (908/826)
Zen London: https://www.speedtest.net/result/16319559565 (626/884)

Edited by agent_r00t (Fri 31-May-24 12:58:24)

And yet the faster speedtest shows a longer ping time.

If the speed are slow in the evening but fast in the middle of the night, that would be an indication of congestion. If that's not the case, then the problem probably lies elsewhere.

I've seen people reporting issues of low-level packet loss on the Zen backhaul network, and I don't know if those have been resolved.

I did notice that. But, what we don't know is whether the latency is added in Zen's network, or the target network. E.g. is how much of my route that seems to be going South->North->South->North is in Zen, and how much is within peering networks or within the speedtest target's network.

I could probably spy the IP being used and see that in a traceroute.. Well actually I could see MY side of the route in a traceroute.

What we do know is that the speedtest to Zen's own server in London is considerably slower than speedtests to other networks in Manchester. With me, based in the South. Which, is mainly what this was about if you'll recall.

If the speed are slow in the evening but fast in the middle of the night, that would be an indication of congestion. If that's not the case, then the problem probably lies elsewhere.

Well actually I find Zen is more congested during work hours. I think a lot of us that are remote workers use Zen. Probably explains it.

Edited by agent_r00t (Fri 31-May-24 14:10:40)

I'm in the north-west MCR 1st hop is 7ms London is 12 -13ms but more consistent and less jitter , this latency is the same via BT WBMC and tt wholesale, Zen's GEA has improved routing from the n'west or for me at least The 13msto 1st hop London has been consistent with other isp's like Plus not and UKOnline (Now sky) apart from BE iicr Zen's old routing was 20+ms to London, no MCR option

Edited by tommy45 (Fri 31-May-24 22:10:09)

I don't think there's anything wrong with the routing or latency from Manchester. I'm just not sure why they (or cityfibre, or both) are routing me via Manchester in the first place, when I'm 20 miles away from London, in the westerly direction.

Just seems a bit weird and wasteful to do so. If their only peering point with cityfibre is in Manchester, it might make sense. But the other commenter above shows their latency shift from what would make sense for someone in the south routing via London to the same situation I have. So it seems they do have a presence with cityfibre in the south. It's just not being used in my case, for some unknown reason.

It looks like City Fibre are terminating your connection at Manchester. It doesn’t look like they are load distributing, nor do we know if they’ll fail you over to London (if it is failover, it’s unlikely reconnecting will help during normal operations).

Once you have reached the Zen network in Manchester, it looks like the first preference for PPPOE is Manchester and London as a failover. You’ve managed to get it to failover, but that’s still going from your location —> Manchester —> London, which leaves you in a worse situation.

Although the majority of stuff will still breakout via London, they do have a CDN in Manchester and they may even have some transit.

In an ideal world you would route via London, from what you have said. It would be worthwhile to send an e-mail to see if it possible to have some more sensible routing. In the meantime, your best bet is to have your pppoe termination in Manchester.