IPv6 Broken?

Sometime after 01:00 this morning odd things are occurring with IPv6

Trying to connect to a number of networks via IPv6 appears to fail and the traceroute goes cold before it leaves Zen's network...

https://tools.keycdn.com/ipv6-ping

1 <1 ms <1 ms <1 ms <My local gateway>
2 38 ms 21 ms 27 ms lo0-0.bng4.thn-lon.zen.net.uk [2a02:8010::15b]
3 10 ms 9 ms 9 ms lag-14.p1.thn-lon.zen.net.uk [2a02:8010:0:700::4c]
4 * * * Request timed out.
5 * * * Request timed out.


Other routes appear normal (like bbc.co.uk) and I can reach other hosts via other networks without any issues.
No changes this end, and at that time of the morning looks rather like PEW.

Tracing route to bbc.co.uk [2a04:4e42::81]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms <My local gateway>
2 39 ms 20 ms 18 ms lo0-0.bng4.thn-lon.zen.net.uk [2a02:8010::15b]
3 10 ms * 10 ms lag-14.p1.thn-lon.zen.net.uk [2a02:8010:0:700::4c]
4 87 ms 101 ms 10 ms 2001:7f8:4::d361:1
5 9 ms 9 ms 9 ms 2a04:4e42::81

Is anyone else experiencing similar issues?

I have not tried jumping gateways as yet and it doesn't appear I've be reconnected over night.

This is looking like a routing issue somewhere. Trace into Zen [2a02:8010::15b / lo0-0.bng4.thn-lon.zen.net.uk]

Frankfurt

Start: 2024-09-28T21:18:10+0000
Loss Snt Last Avg Best Wrst StDev
1.|-- fd42:10::1 0.0% 4 0.1 0.1 0.1 0.2 0.0
2.|-- 2001:2000:3080:11e5::1 75.0% 4 0.6 0.6 0.6 0.6 0.0
3.|-- 2001:2034:1:6b::1 0.0% 4 1.0 1.1 1.0 1.4 0.2
4.|-- 2001:2034:1:be::1 0.0% 4 8.9 8.8 8.6 8.9 0.1
5.|-- 2001:2034:1:7a::1 0.0% 4 15.7 15.7 15.7 15.7 0.0
6.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0

But works from

Amsterdam

Start: 2024-09-28T21:18:11+0000
Loss Snt Last Avg Best Wrst StDev
1.|-- 2a03:b0c0:2::15 0.0% 4 0.7 1.5 0.7 2.5 0.9
2.|-- 2a03:b0c0:fffd::1e 0.0% 4 2.3 1.2 0.6 2.3 0.7
3.|-- 2a03:b0c0:fffe::54 0.0% 4 1.0 2.4 1.0 3.8 1.4
4.|-- 2001:7f8:4:1::32ed:1 0.0% 4 12.4 12.9 11.0 16.3 2.3
5.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
6.|-- 2a02:8010::15b 0.0% 4 13.1 16.8 13.1 26.1 6.2

Edited by IamQ (Sat 28-Sep-24 22:20:17)

Signed up to say I'm seeing similarly weird things. WireGuard tunnel over IPv6 to Germany dropped around the same time you report. Weirdly, I can reach Hetzner, a German hosting provider, but not Deutsche Telekom. Traceroute looks fine but no traffic passes, and ICMP ping doesn't work either which is bizarre.

Oh thanks for letting me know. I'll try and look more into it tomorrow and may drop support an email

It's one for Zen technical support: clearly they have a problem with their IPv6 transit, which they've not picked up by their own monitoring.

FWIW, it's worth bearing in mind that the BBC is not IPv6-enabled. "bbc.co.uk" has an IPv6 address, but when you connect to it with a web browser all it returns is a redirect to "www.bbc.co.uk", and that does *not* have an IPv6 address.

Has been reported to the 1st line support team and directly to the IPv6 team

I'm not seeing anything where I cannot outright connect over ipv6. But, definitely for the last few days, some (maybe all, not sure) sites that are ipv6 enabled seem noticeably slower. If I set them to use ipv4 they are fine.

The plugin "IPvFoo" for Chrome browser (I don't know if it's available for others) gives a very useful indication of whether IPv4 of IPv6 is being used for the man page, and for any assets referenced within the page.

In reply to a post by candlerb:

The plugin "IPvFoo" for Chrome browser (I don't know if it's available for others) gives a very useful indication of whether IPv4 of IPv6 is being used for the man page, and for any assets referenced within the page.

Oh, I know that I can check. But I've only spotted it happening on two sites so far and I've not been actively checking otherwise.

I notice refreshing on here is sometimes delayed longer than usual, and ipv6 is used here. So maybe 3 sites.

It does seem intermittent too.

Edit: Right now it's back to super fast over ipv6 on all the sites I was seeing slowness.

Edited by agent_r00t (Wed 02-Oct-24 13:40:33)

My ipv6 only SIP registration has been failing (sip6.voipcheap.com). Zen support have not got back to me yet .
It's working fine on my FIL's community fibre connection, so I know it's Zen IPV6 at fault

And has been escalated to the infrastructure team & Zen have confirmed many more reports of problems have started coming in relating to v6 issues.

Edited by IamQ (Thu 03-Oct-24 17:16:40)

Can confirm, let them know after the postings here:

This issue does indeed look to be our end, though we will need to escalate this internally to our Network Operation Centre. I will raise a query with them and get back to you as updates are made available.

It has been broken on and off for a few years now.

Would be nice if they at least acknowledged an issue publicly.

In reply to a post by NICK_ADSL_UK:

It has been broken on and off for a few years now.

I've never had weird issues like this - mine has thankfully just worked since the day it was enabled. Problem now though for me is I have v6 (only) servers and devices now which I can't communicate with.

Indeed - I was thinking that last night.

At one point support where trying to make me jump through the diagnostic hoops which was flatly rejected on my part!

Maybe a couple of 1 start trust pilot reviews will prompt them into action!

I received an email this morning. They fault has been logged under their service alerts now: https://servicealerts.zen.co.uk/active/5/8543

Appears there been an update - things appear to be working again for me at least but haven't had a chance to test a number of sites and services.

And as of midnight last night its even more broken with many v6 networks now no longer reachable

Yes, routing has failed again. I tried reporting it via Zen's online chat but they responded that IPv6 issues can be handled only via email with a 2-day target response time. They then suggested that IPv6 isn't enabled on my account, despite it having worked on my account for years.

Not an impressive response.

It's really bad today. I had a lot of stuff running extremely slowly when trying to do anything network. It's because some networks (including google when I tried just now) are not reachable. So because there is a working ipv6 network it has to time out every single try before dropping back to ipv4 making things extremely painful.

I disabled ipv6 for the first time in a long time!

They have updated the original alert on the website and did reply to my email from this morning

They've replied to me, too, but not with any substantive information, merely that the issue has been passed on and that they haven't yet updated the service alert. It's disappointing that Zen can't be more open about what's happening and that they still apparently regard IPv6 as a niche protocol beyond the knowledge of their first-line support staff.

I haven't noticed any IPv6 issues up until last night, and indeed this morning I found no IPv6 connectivity. Fired off an email to Zen and it's been dead all morning, but just as I logged in and started writing this reply my pings are coming back in, so it looks like I'm back online.

My IPv6 connectivity is now working again, so the outage was reasonably short today. I'd still welcome some technical detail about what happened.

Edited by ashdown (Tue 15-Oct-24 11:52:06)

Indeed - I got the same useless reply.
Looks like things came back around 11:30 looking at some long running processes.

It did highlight an odd issue though with one of my apps - it kept trying to connect out via v6 even though it was ending in a failure and would never try the v4 addresses (which was working) (Ruby net:http)

In reply to a post by IamQ:

Indeed - I got the same useless reply.
Looks like things came back around 11:30 looking at some long running processes.

It did highlight an odd issue though with one of my apps - it kept trying to connect out via v6 even though it was ending in a failure and would never try the v4 addresses (which was working) (Ruby net:http)

There is an algorithm called "Happy Eyeballs" that's supposed to fix that. Web browsers all implement it and I think the networking stack in Android and iOS does too but command line apps like Ruby typically don't, they are very dumb and try the first address given to them and if that times out then it times out.