Peering / network throughput issues? (March 2019)

I've been discussing with Hyperoptic support for a couple of months now after I noticed really low throughput to both London and Manchester destinations - on a gig line, we're talking a max of about 300 Mbit/sec per thread, more often lower than that.

A few weeks ago, all my routes to Manchester servers started going in to GTT then off to either Dublin then London, or straight to London. This was even to providers who were peering at LINX Manchester, and previously had a five- or six-hop route from my connection to the far end. A handful of servers even went to Telia and were more sensibly routed, but still incurred a massive speed penalty.

European destinations are still terrible, but at the moment I can't even max out the line with speed tests to servers I know are hosted on excellent networks with more than enough bandwidth to satisfy a gigabit line.


I started making notes of this poor performance and passed to HO support, and in fairness they responded quickly and did thinks like pass MTRs and speed test results to GTT.

It now seems some of the worst routing nonsense has stopped, but I'm still getting very poor speeds from speed tests and VPN tests, much lower than I was in January.

What's more, throughput appears to either be being capped or suffering from core network contention - I'm starting to wonder if the Manchester part of their core network is starting to become overwhelmed. At present, I can't even max out the Hyperoptic speedtest (which is hosted on Fastly, so should be quick!)

Any other HO customers care to share their observations from speedtests, VPN usage etc?

Edited by christopherwoods (Fri 08-Mar-19 01:00:17)

I've noticed that there's a load of split routing going on in their network too. My parents are on idnet, tracing from my Manchester Gigabit hyperoptic line goes via GTT MCR, GTT London then to Level3 then to idnet (presumably idnet buy their transit from L3).
So packets I send to idnet go via transit, fair enough, but then running a trace from my parents to my Hyperoptic IP goes from idnet to LINX London and into Hyperoptic's core network in London, before heading up to Manchester.

I contacted support about this & they said this was how the network was designed to operate.


I have been curious about peering etc, so based on my own meandering testing, the following providers are currently peering with HO in Manchester. Note, this list isn't exhaustive of course. Comments are what these networks host, in case anyone reading doesn't already know:
* Akamai (Lots of streaming and websites incl BBC iPlayer)
* Limelight (Lots of streaming incl BBC iPlayer)
* Cloudflare (Lots of websites in general, including this one! Although, the Manchester peering doesn't mean that every site on CF's network comes from the Manchester POP; some still are in Dublin and so use GTT Manchester -> GTT Dublin to get to CF.)
* Janet (so mirrorservice.org and pretty much all the universities - pings to manchester.ac.uk take 2.6msec!)
* Microsoft (Office361 and counting, Azure, Skype, Xbox downloads etc)
* Apple
* Talktalk ISP - useful if you're transferring data to a friend who's on that network
* Zen ISP - ditto.

Notably, there's one major provider at IX Manchester who doesn't seem to be peering, at least, I can't find any evidence of such...Amazon AWS (so you use the London CloudFront endpoint etc). Despite Amazon having 100Gbit into IX Manchester, they don't seem to be peering here with HO. Same split routing as with idnet appears to be happening; out via transit, return goes to LINX London.

A colleague and I have done a few speed tests, we're both on static IPs but different subnets - either that, or how their network is segmented or routed, is causing massive speed disparities.

For example, my colleague:
http://speedtest.uk2.net/speedtest.1024mb = 72MB-80MB/sec
https://speed.hetzner.de/1GB.bin = 10MB/sec (surprisingly low given his Hetzner VPN gives super-fast speeds)

http://ams-speedtest-1.tele2.net/10GB.zip = 105MB/sec
http://kst5-speedtest-1.tele2.net/10GB.zip = 88MB/sec
http://fra36-speedtest-1.tele2.net/10GB.zip = 75MB/sec

Me:
http://speedtest.uk2.net/speedtest.1024mb = 32 MB/sec but fluctuates to mid-20s
https://speed.hetzner.de/1GB.bin = 9.2 MB/sec peak, though with multiple running each ends up at ~8 MB/sec


http://ams-speedtest-1.tele2.net/10GB.zip = caps at 18 MB/sec v4, just under 20 MB/sec v6
http://kst5-speedtest-1.tele2.net/10GB.zip = caps at 7.8 MB/sec v4, same v6
http://fra36-speedtest-1.tele2.net/10GB.zip = caps at 13.7 MB/sec v4, 13.2 MB/sec v6


Netflix routing also seems to result in Netflix CDN considering our traffic as American, and serving US CDN nodes - and in peak time, IPv4 to Netflix for me has been unusable, sub-1 megabit.

Interestingly, using IPv6, the connection is still sometimes considered as being "US" (perhaps due to sometimes being routed via NTT or another provider normally carrying US traffic, or actually routing to the US before reaching Netflix?) but throughput is easily hundreds of megabits per second, as you'd expect.


Routes sometimes are completely illogical, as you've said I sometimes see traffic going from GTT to Telia or Level3 and then heading across the Irish sea, back to London, into Europe then back the same route, which is utterly nonsensical. No amount of route costing can explain that, particularly when GTT can happily carry London-Manchester traffic.

It's also odd how Amazon haven't peered with HO at Manchester, perhaps we're just not on a big enough ISP volume-wise.


I've also recently noticed slowdowns to servers which are peered directly at LINX Manchester; TNP, Boundless Networks, Exascale, all are slowing down. I can get better speeds from FAELIX - about as direct as possible via LINX Manc. But even single-threaded connections to them are about 60% of what they should be. Per-connection speeds appear to have recently reduced quite a lot, no explanation why.

Well if it helps, my IP is in the 88.98.247.0/24 block.

I have noticed that Netflix strangeness when going to fast.com, it shows the datacentre that Netflix uses there & gives a speedtest. Not looked at the routing to that.

Sometimes I do see stuff going to Dublin then being handed over to other Tier 1 ISPs there before going to London. I can sort of understand that; if we ignore the map of the world and look at the GTT network, they use the Hibernia cable running from Manchester via Southport to Dublin (and then across the pond to USA). So if HO pass the traffic to GTT in Manchester which needs to go to (say) Telia, they might have two choices of location to hand this traffic over; Dublin and London; Dublin might be the better choice given network load etc. The fact that Telia are promptly going to take that traffic back to (say) London is somewhat out of GTT's hands. I think, at least... routing is a bit of a dark art to me!
I've never seen any more odd routes, the 'strangest' i've seen has been just going to Dublin first before handing over from GTT there.

I think I've got ipv6 disabled on my connection at the mo, but here are the speeds I got for the following links, just lately, using curl -o /dev/null <link>

http://mirror.bytemark.co.uk/gentoo/releases/amd64/1... (IX Manchester)
Average: 66MB/sec

https://speed.hetzner.de/1GB.bin
Average: 16.8MB/s, it started slow then sped up to 70MB/sec toward the end of the file...

http://speedtest.uk2.net/speedtest.1024mb
Avg: 70MB/sec but for the final 50% of the file it was transferring at 93MB/sec, so it was just a bit slow to start

http://ams-speedtest-1.tele2.net/10GB.zip
Average: 99MB/sec, speedy!

http://kst5-speedtest-1.tele2.net/10GB.zip
88MB/sec

http://fra36-speedtest-1.tele2.net/10GB.zip
76MB/sec

Seems that I get similar-ish speeds to your colleague. What IP blocks are you both in, and are either of you using the CGNAT system?

..Looking at the linx stats page for Manchester, https://portal.linx.net/stats/lans#man1 - I think it might be worth looking at around 4am. (Well, okay, getting cron to look around then!)

It does seem strange, though, that both IX Manchester and GTT-transit endpoints are slowing down. Would this be indicative of a router in Hyperoptic running close to its maximum, rather than any of the individual links being busy per se?

Possibly unrelated, possibly absolutely related, I play games with nVidia Geforce Now and it sometimes reports packet loss... but then some other users on that service also lament the poor performance, so it could be that nVidia are struggling & I'm just also a witness to their difficulties.

Hi,

I'm Chris' colleague mentioned above.

Whilst most of my speeds are decent depending on the server , for example Steam/Origin downloads tend to max out my pipe - I'm mainly having major issues with Netflix CDN peering. Fast.com as mentioned above always defaults to a US server and most of my devices can barely get out of SD 480p streaming (both wired and wireless!).

I'm in the 88.98.250.0/24 block

Steam and a good number of game store downloads are massively multiple thread i.e. data comes from many locations at the same time. The aim of doing this is to avoid almost all congestion effects.

Hi Christopher,

We can rely on our own speed test as a reference because it utilises our servers.

When it comes to the routing, we've been checking this with our Network Team and we've asked GTT Upstream why do they prefer Dublin instead of London for servers destinated to London and still expecting their reply. But we can only guarantee 1Gbit in Hyperoptic network and all other speed tests outside of Hyperoptic network are not reliable. Also, testing speed test via VPN can affect the speed because of geographic distance, VPN server congestion, encryption latency.

If you have any further questions, please let us know.

In reply to a post by Hyperoptic_CS:

Hi Christopher,

We can rely on our own speed test as a reference because it utilises our servers.

When it comes to the routing, we've been checking this with our Network Team and we've asked GTT Upstream why do they prefer Dublin instead of London for servers destinated to London and still expecting their reply. But we can only guarantee 1Gbit in Hyperoptic network and all other speed tests outside of Hyperoptic network are not reliable. Also, testing speed test via VPN can affect the speed because of geographic distance, VPN server congestion, encryption latency.

If you have any further questions, please let us know.

Thanks for your reply


Yep, speed.hyperoptic.com is showing a reduction of at least 100 mbit/sec, usually more, even at midnight. Tests done on multiple PCs, multiple browsers, even multiple routers (which I've confirmed are gigabit capable). It's been noticable for a few weeks now. Not even the Hyperoptic test can break 900 megabits down, in multi-thread mode, to its own test server... Another test just now: 802 down, 922 up, at 1am. Where's the other 160 megabits gone?

When it was first installed, I remember getting at least 700 mbits via NordVPN Manchester servers, now I can't get much above 300 from almost anywhere. I've also done extensive tests to get the optimum OpenVPN settings with the least complex algorithms (so least overhead), I'm pretty sure the VPN connection is not causing much speed loss, perhaps 1-5% at most.


The recent Netflix problems seemed to be due to either your transit provider or Netflix misclassifying the origin of Hyperoptic IPs and routing them to the 'wrong' place.

I had a reply from CS yesterday - IPv4 traffic now appears to be served from LHR (London) servers, however it's still 50/50 whether you get a server which can do 230 Mbps, or a server which can only do 25 Mbps.

Again, this appears to be because, despite the Netflix LHR cluster now serving IPv4 clients, sometimes it's still being routed via New York. IPv6 traffic is still being incorrectly handled, I think - I provided all this info back to CS on email.

IPv4 speeds to other locations still suck, frankly. I can only get 30 Mb/sec to various London UK2 servers on a 1 gigabit/sec pipe, which means VPN traffic is dismally slow. Interestingly, UPload speeds to that server are about 2x as fast.

Example:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.2.1 - 0 | 11 | 11 | 0 | 0 | 0 | 0 |
| my.static.ip.gateway - 0 | 11 | 11 | 1 | 1 | 2 | 2 |
| 172.17.4.58 - 0 | 11 | 11 | 2 | 7 | 27 | 27 |
| 172.16.29.10 - 0 | 11 | 11 | 1 | 1 | 1 | 1 |
| 172.16.22.154 - 0 | 11 | 11 | 1 | 1 | 1 | 1 |
| ae2-775.cr1-man1.ip4.gtt.net - 0 | 11 | 11 | 0 | 0 | 1 | 1 |
| xe-1-0-3.cr0-lon1.ip4.gtt.net - 0 | 11 | 11 | 7 | 7 | 11 | 11 |
| 92.60.249.46 - 0 | 11 | 11 | 7 | 7 | 7 | 7 |
| 83.170.70.238 - 0 | 11 | 11 | 7 | 7 | 8 | 8 |
| 31.24.231.254 - 0 | 11 | 11 | 7 | 7 | 8 | 7 |


And even to speedtest.net servers physically in Manchester, peered at LINX Manchester, using multiple connection tests, I can only get mid-800s down - and in some cases really low upload (140, 340...):

https://www.speedtest.net/result/8130666501
https://www.speedtest.net/result/8130669214

| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.2.1 - 0 | 47 | 47 | 0 | 1 | 22 | 0 |
| my.static.ip.gateway - 0 | 47 | 47 | 1 | 3 | 24 | 2 |
| 172.17.4.58 - 0 | 47 | 47 | 2 | 3 | 19 | 2 |
| 172.16.29.10 - 0 | 47 | 47 | 1 | 13 | 45 | 21 |
| 172.16.22.154 - 0 | 47 | 47 | 1 | 24 | 71 | 1 |
| tnp-manc-rtr01.tnp.net.uk - 0 | 47 | 47 | 0 | 3 | 33 | 2 |
| speedtest.tnp.net.uk - 0 | 47 | 47 | 0 | 1 | 21 | 0 |



Boundless Networks is just about able to deliver 900 at peak, but it still dropped:
https://www.speedtest.net/result/8130671841


| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.2.1 - 0 | 7 | 7 | 0 | 3 | 25 | 0 |
| my.wan.ip.gateway - 0 | 7 | 7 | 1 | 5 | 27 | 1 |
| 172.17.4.58 - 0 | 7 | 7 | 2 | 5 | 27 | 2 |
| 172.16.29.10 - 0 | 7 | 7 | 1 | 11 | 27 | 21 |
| 172.16.22.154 - 0 | 7 | 7 | 31 | 31 | 33 | 31 |
| 1ge-man1.boundlessnetworks.uk - 0 | 7 | 7 | 0 | 5 | 20 | 0 |
| 146.255.104.131 - 0 | 7 | 7 | 1 | 1 | 6 | 1 |


Vodafone's Manchester test server also seems routed via London,
speedtest-manchester.vodafone.co.uk.prod.hosts.ooklaserver.net

https://www.speedtest.net/result/8130677135
150 down, 15 up!

1 <1 ms <1 ms <1 ms 192.168.2.1
2 2 ms 3 ms 4 ms my.ip.gateway
3 3 ms 2 ms 2 ms 172.17.4.58
4 2 ms 1 ms 1 ms 172.16.29.10
5 1 ms <1 ms 1 ms 172.16.22.154
6 <1 ms <1 ms <1 ms ae2-775.cr1-man1.ip4.gtt.net [77.67.123.169]
7 7 ms 11 ms 7 ms et-0-0-43.cr10-lon1.ip4.gtt.net [89.149.139.1]
8 7 ms 7 ms 7 ms ae9-xcr1.lns.cw.net [195.2.22.217]
9 7 ms 7 ms 7 ms ae9-xcr1.lsh.cw.net [195.2.25.169]
(at this point the C&W network drops ICMP)

Weirdly, Vodafone's London test server also gives rubbish downstream, but amazing upstream (213 down, 659 (!) up on a single thread test);

speedtest.vodafone.co.uk.prod.hosts.ooklaserver.net
https://www.speedtest.net/result/8130681434

1 <1 ms <1 ms <1 ms 192.168.2.1
2 34 ms 19 ms 11 ms my.ip.gateway
3 20 ms 20 ms 6 ms 172.17.4.58
4 1 ms 1 ms 1 ms 172.16.29.10
5 1 ms 1 ms 1 ms 172.16.22.154
6 <1 ms <1 ms <1 ms ae2-775.cr1-man1.ip4.gtt.net [77.67.123.169]
7 7 ms 20 ms 18 ms et-0-0-43.cr10-lon1.ip4.gtt.net [89.149.139.1]
8 7 ms 7 ms 8 ms ae9-xcr1.lns.cw.net [195.2.22.217]
(C&W drops ICMP from here)

I also see this performance reflected to some VPN servers, including my own private servers and corporate servers -- all on reliable, premium multi-gigabit connections - quite slow down, but much faster up, sometimes 2x or 3x as fast. Throughput reaches a limit and stays there, it doesn't vary up and down at all.


It's weird how GTT and in some cases Level3 appear to be spoiling speeds to and from the wider Internet, despite Hyperoptic's network being gigabit-capable. I do remember seeing 950 Mbit/sec when I first had my line first installed. Speeds seem to have reduced slightly.

I have a dedicated fibre to wherever the local pop is - I think I'm still the only person in my small block of half a dozen flats with Hyperoptic, so I don't think local congestion is an issue.


Some more examples in single thread mode to replicate VPN experience.

Vodafone London
https://www.speedtest.net/result/8130703581

B4RN in Lancaster, 570/405, 580/457...
https://www.speedtest.net/result/8130704658
https://www.speedtest.net/result/8130705989

FAELIX in Manchester is the other way round from Vodafone and Boundless Networks - 771/161, 770/260... It definitely feels like routing and upstream transit is the problem here.
https://www.speedtest.net/result/8130707092
https://www.speedtest.net/result/8130708526

Fibrenest in Manchester is slow, 284/92 and 309/96, again looks like throughput is sometimes hitting a ceiling and suffering from bad upstream bandwidth. Traceroutes indicate the traffic for that Manchester server is going via London once it gets to GTT's network...
st-2.fibrenest.com.prod.hosts.ooklaserver.net
https://www.speedtest.net/result/8130713350
https://www.speedtest.net/result/8130714507

See how the bandwidth looks,
https://i.imgur.com/Lygegp8.png

The first few tests are to Vodafone London, the others are to Manchester and Lancaster servers, the last is to Fibrenest - all geographically close, so should have fairly direct traffic routes. FAELIX is only one hop away from Hyperoptic's network -- and so is TNP, which used to get great speed results but which are now really bad.



Basically it's disappointing, paying for a gigabit connection but rarely being able to actually use anything like the speeds it should be able to support. It's a premium product and I expect it to have premium onward connectivity to let me utilise it fully.

I understand the complexities of maintaining high speed traffic over the Internet, and the difficulties bandwidth providers face, but parts of the network seem too congested, and there's problems in other areas. I had to produce demo videos and lots of test results to get the Netflix problem seriously investigated.


GTT seems a bit incapable of providing sufficient bandwidth to most other networks in the UK, never mind US or Europe. I've done dozens of tests to different places which should give great speeds, and nearly always they suffer.

The first time I began to complain about poor speeds to Manchester and London servers, eventually I got a response passed back from GTT which basically said "internet speeds are not guaranteed, ICMP isn't a reliable indicator of network quality." - I accept that, I've done network engineering so I understand this. It seems they didn't even take the feedback of poor routing and overall low speeds seriously. I'm blaming them for that, not Hyperoptic, but it's annoying as a customer.

I have friends in Spain, France and Norway on gigabit fibre connections who have none of these problems - they get rapid, reliable connectivity to the UK, US and mainland Europe. I don't understand what's so hard about providing connection that can achieve at least 50% of its stated line speed most of the time, unless there's a fundamental issue with the third party bandwidth providers.


What's been most frustrating though is the lack of transparency from Hyperoptic regarding any potential issues.

If the Hyperoptic network is struggling to cope with a recent influx of new customers, or connectivity partners (GTT, Telia, Level3 etc) need to scale up their bandwidth, please tell the customers!

If work is being done behind the scenes to improve the available bandwidth and connection quality for customers, I'd like to hear about it. If I knew when things were going to improve, I would tolerate slower speeds for a while - at the moment there's no updates given, not even a formal network status page like many other ISPs.



I'm not trying to rip on Hyperoptic for the sake of it. I'm hoping this is all constructive feedback. I hope the info I've provided can be used to improve the network. Hopefully someone can kick your transit providers into gear to fix whatever capacity issues there are outside of the core Hyperoptic network - and get peak speeds back up to what we'd expect them to be.

I still have great enthusiasm for Hyperoptic's products and recommend them to everyone, full fibre is miles ahead of the rest of the market. Customer support have been incredibly helpful with various technical queries, just like in the Be* days! The reliability and consistency of bandwidth to the wider Internet just needs some improvement.

Gaming pings are absolutely killer though... 1ms! 0ms! Nearest I've experienced to a LAN, including the early days of Telewest broadband. The gaming experience on Hyperoptic is pretty rad.

Edited by christopherwoods (Fri 22-Mar-19 01:25:00)

"But we can only guarantee 1Gbit in Hyperoptic network and all other speed tests outside of Hyperoptic network are not reliable."

I can understand that - you of course have got no control over whether the destination has got sufficient capacity to supply 1gbps of traffic, and you've no visibility of the network which isn't yours.

However, one question then - in the past month, have any of the connections from the Hyperoptic network out to the internet (such as your links to GTT, IX Manchester, LINX 1, LINX 2 or Lonap) had drastically less than 1gbps of available capacity?

I ask this because you could be running your internal speedtest with plenty of capacity, but we don't know whether your external links are running red hot or if they're quiet.

For an example, look at IDNet's network map: https://www.idnet.com/about/our_network.php - at the time of writing (ok, it's only 11am on a weekday morning), things look pretty quiet; do you have a similar page anywhere? It would certainly help in situations like this.