HO blocking Cloudflare DNS

It appears HyperOptic are blocking / started having trouble peering with Cloudflare DNS at
https://1.1.1.1

This has been working for the better part of a year (10 months) and only started this morning. This is extremely disappointing, as I use DNS over HTTPS to Cloudflare. There is *not* an issue with Cloudflare itself as I can query it from my phone, other networks.

A traceroute shows (gateway masked below by me):

pi@pi:~ $ traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 172.16.44.1 0.486 ms 0.307 ms 0.398 ms
2 xxx.39.89.yy.bcube.co.uk (89.39.xxx.yy) 6.752 ms 6.829 ms 7.133 ms
3 172.16.28.148 (172.16.28.148) 13.499 ms 13.638 ms 13.765 ms
4 172.17.12.242 (172.17.12.242) 2.570 ms 2.724 ms 2.881 ms
5 172.17.8.148 (172.17.8.148) 1.819 ms 2.518 ms 2.461 ms
6 172.17.13.22 (172.17.13.22) 6.086 ms 172.17.13.49 (172.17.13.49) 2.472 ms 2.497 ms
7 172.17.13.24 (172.17.13.24) 6.019 ms 172.17.12.35 (172.17.12.35) 4.761 ms 172.17.13.24 (172.17.13.24) 2.520 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Results of ping:
pi@pi:~ $ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
18 packets transmitted, 0 received, 100% packet loss, time 17651ms

Is this a way to force Cloudflare users to use HO's DNS? Or is this a peering problem?

TIA

Edited by deleted (Mon 26-Aug-19 16:54:17)

If it's only started this morning, what have Hyperoptic said when you reported it?

it works fine for me though, I can get to it in 8 hops and it responds to ping.

I can also trace to 1.1.1.1, perhaps something was wrong with the set up?

traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 *my router* (192.168.1.1) 1.220 ms 0.824 ms 0.802 ms
2 xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 2.347 ms 2.045 ms 2.074 ms
3 172.16.xxx.xxx (172.16.xxx.xxx) 1.794 ms 1.940 ms 1.685 ms
4 172.16.xxx.xxx (172.16.xxx.xxx) 7.844 ms 2.542 ms 2.357 ms
5 172.16.xxx.xxx (172.16.xxx.xxx) 5.797 ms 2.491 ms 2.225 ms
6 lonap.as13335.net (5.57.81.75) 2.249 ms 5.979 ms 2.362 ms
7 one.one.one.one (1.1.1.1) 1.464 ms 1.569 ms 1.652 ms

There was nothing wrong with my setup.

I had opened a ticket with HO and they responded today to say there was a routing problem which they have now fixed.

pi@pi:~ $ traceroute -n 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 172.16.44.1 0.511 ms 0.414 ms 0.489 ms
2 89.39.xxx.yyy 5.392 ms 5.755 ms 5.974 ms
3 172.16.28.148 2.850 ms 3.078 ms 3.236 ms
4 172.17.12.242 2.459 ms 2.576 ms 2.804 ms
5 172.17.8.148 2.260 ms 2.367 ms 2.448 ms
6 172.17.13.22 6.219 ms 172.16.25.232 8.066 ms 7.994 ms
7 172.16.25.234 8.498 ms 7.549 ms 7.809 ms
8 1.1.1.1 1.479 ms 1.405 ms 1.501 ms

Hey there,

Thanks for hanging in there while we investigated and resolved this issue.

If you have any other questions, feel free to get in touch.

Thanks @Hyperoptic_CS.

Contrary to blocking CF, it appears Hyperoptic are now private peering with Cloudflare (or maybe they are hosting a cache node?).

Previously the route to CF went through lonap, whereas now it appears to directly drop to CF off HO's network. Could you please elaborate?

TIA

From Manchester it all looks fine to me, and is still clearly heading out of the network, via IX Manchester to Cloudflare.

traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 192.168.34.254 (192.168.34.254) 1.931 ms 1.968 ms 1.283 ms
2 88.98.247.xyz.bcube.co.uk (88.98.247.xyz) 5.706 ms 5.699 ms 6.937 ms
3 172.16.23.106 (172.16.23.106) 6.641 ms 3.757 ms 3.169 ms
4 172.16.27.124 (172.16.27.124) 2.929 ms 3.559 ms 3.226 ms
5 172.16.30.164 (172.16.30.164) 3.980 ms 4.464 ms 2.918 ms
6 172.16.17.81 (172.16.17.81) 2.936 ms 5.350 ms 3.325 ms
7 ixmanchester.as13335.net (195.66.244.71) 3.355 ms 4.104 ms 2.708 ms
8 one.one.one.one (1.1.1.1) 1.717 ms 2.014 ms 1.943 ms

Hrm, not any more.. seems Hyperoptic have either broken some config or are deliberately not peering at LINX Manchester (or IX Manchester as it used to be known) with as many parties.. instead going via GTT transit for a lot more!

~ traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 192.168.1.254 (192.168.1.254) 1.083 ms 0.606 ms 0.623 ms
2 IP.bcube.co.uk (IP) 5.610 ms 5.090 ms 4.808 ms
3 172.16.23.106 (172.16.23.106) 2.770 ms 2.524 ms 1.875 ms
4 172.16.27.124 (172.16.27.124) 2.013 ms 3.193 ms 1.926 ms
5 172.16.30.164 (172.16.30.164) 5.786 ms 2.424 ms 2.530 ms
6 ae0.cr1-man3.ip4.gtt.net (87.119.120.169) 2.394 ms 1.467 ms 1.113 ms
7 ae10.cr1-man1.ip4.gtt.net (213.200.125.98) 1.524 ms 1.440 ms 1.600 ms
8 cloudflare-gw.cr1-man1.ip4.gtt.net (141.136.99.138) 1.887 ms 2.188 ms 2.177 ms
9 one.one.one.one (1.1.1.1) 1.588 ms 1.364 ms 1.307 ms



~ traceroute www.manchester.ac.uk
traceroute to mhn.mc.man.ac.uk (130.88.101.57), 64 hops max, 52 byte packets
1 192.168.1.254 (192.168.1.254) 0.930 ms 0.537 ms 0.510 ms
2 IP.bcube.co.uk (IP) 5.116 ms 5.469 ms 4.452 ms
3 172.16.23.106 (172.16.23.106) 3.089 ms 15.527 ms 8.487 ms
4 172.16.27.124 (172.16.27.124) 6.221 ms 8.311 ms 1.920 ms
5 172.16.30.164 (172.16.30.164) 5.950 ms 2.478 ms 2.650 ms
6 ae0.cr1-man3.ip4.gtt.net (87.119.120.169) 1.309 ms 19.366 ms 1.293 ms
7 et-10-1-0.cr0-dub2.ip4.gtt.net (141.136.107.49) 12.094 ms 12.092 ms 11.902 ms
8 dln-b1-link.telia.net (62.115.45.198) 11.942 ms 12.205 ms 12.214 ms
9 ldn-bb4-link.telia.net (62.115.120.10) 17.706 ms 14.435 ms
ldn-bb3-link.telia.net (62.115.119.242) 14.335 ms
10 ldn-b7-link.telia.net (62.115.138.151) 14.063 ms 14.076 ms 14.052 ms
11 jisc-ic-345130-ldn-b7.c.telia.net (62.115.175.107) 14.611 ms 14.695 ms 14.745 ms
12 ae24.londtt-sbr1.ja.net (146.97.35.193) 14.427 ms 14.341 ms 14.315 ms
13 ae28.londtw-sbr2.ja.net (146.97.33.62) 14.322 ms 14.546 ms 14.527 ms
14 ae31.lowdss-sbr1.ja.net (146.97.33.29) 17.917 ms 17.504 ms 17.637 ms
15 ae29.leedaq-sbr2.ja.net (146.97.33.49) 20.396 ms 20.261 ms 22.090 ms
16 ae25.presab-rbr1.ja.net (146.97.38.46) 22.776 ms 22.899 ms 22.677 ms
17 ae28.mancrh-rbr1.ja.net (146.97.78.69) 20.647 ms 28.375 ms 23.523 ms
18 universityofmanchester.ja.net (146.97.169.2) 24.857 ms 21.082 ms 21.263 ms
19 130.88.249.169 (130.88.249.169) 21.906 ms 21.372 ms 21.220 ms
20 gw-rh.cnw.its.manchester.ac.uk (130.88.249.162) 21.762 ms 21.498 ms 21.970 ms

^^ all the way via Dublin and London before being handed to ja.net, rather than their previous peering in Manchester.


~ traceroute www.fastly.com
traceroute to prod.www-fastly-com.map.fastly.net (151.101.37.57), 64 hops max, 52 byte packets
1 192.168.1.254 (192.168.1.254) 1.228 ms 0.532 ms 0.516 ms
2 IP.bcube.co.uk (IP) 5.849 ms 5.451 ms 4.492 ms
3 172.16.23.106 (172.16.23.106) 2.676 ms 2.322 ms 1.830 ms
4 172.16.27.124 (172.16.27.124) 2.737 ms 1.861 ms 1.711 ms
5 172.16.30.164 (172.16.30.164) 6.259 ms 2.516 ms 2.366 ms
6 79.141.39.245.available.above.net (79.141.39.245) 1.266 ms 1.226 ms 1.405 ms
7 ae3.mpr3.lhr3.uk.zip.zayo.com (64.125.31.160) 7.366 ms 8.170 ms 7.315 ms
8 ae27.cs1.lhr15.uk.eth.zayo.com (64.125.30.234) 12.708 ms 12.562 ms 15.066 ms
9 ae2.cs1.ams10.nl.eth.zayo.com (64.125.29.16) 13.099 ms 12.469 ms 12.837 ms
10 ae3.er1.ams1.nl.zip.zayo.com (64.125.31.105) 12.333 ms 13.368 ms 11.848 ms
11 * * *
12 * * *

^^ off to Amsterdam via Zayo transit.

~ traceroute azure.microsoft.com
traceroute to l-0007.l-msedge.net (13.107.42.16), 64 hops max, 52 byte packets
1 192.168.1.254 (192.168.1.254) 1.449 ms 0.520 ms 0.520 ms
2 IP.bcube.co.uk (IP) 5.418 ms 4.853 ms 4.971 ms
3 172.16.23.106 (172.16.23.106) 2.338 ms 2.145 ms 2.173 ms
4 172.16.27.124 (172.16.27.124) 2.308 ms 5.321 ms 5.353 ms
5 172.16.30.164 (172.16.30.164) 5.830 ms 2.336 ms 2.304 ms
6 ae0.cr1-man3.ip4.gtt.net (87.119.120.169) 1.212 ms 2.456 ms 1.761 ms
7 et-10-1-0.cr0-dub2.ip4.gtt.net (141.136.107.49) 12.588 ms 12.643 ms 11.907 ms
8 microsoft-gw.ip4.gtt.net (46.33.92.70) 14.194 ms 14.200 ms 15.153 ms
9 *


Despite having 50gbit of capacity (source: https://www.peeringdb.com/ix/583 ) at LINX Manchester, it seems they're not using it as much as before.

Wonder why not?

Edited by deleted (Fri 29-Nov-19 19:36:25)