WeFibre (Telcom) varied bandwidth

I've currently got an FTTP service with WeFibre (Telcom) which is sold as a symmetrical 1Gbps service. Whilst for the most part this has been okay, I seem to get really mixed results when it comes to speeds.

To give an example of how varied this can be, I ran a number of speedtests across a variety of servers from both my connection here, and additionally from an Azure VM to rule out limitations with the speedtest server(s) themselves.

One speedtest server (51737 - Exa Networks Limited, Bradford) regularly performs well on this line and gave a result of 870/920.

On the contrary, 21221 - FibreNest, Manchester is barely able to achieve VDSL speeds (for the download at least) with a result of 40/912. Similarly, 3504 - TNP Ltd, Manchester returned similar figures at 64/911. I tested a few other Manchester based servers and all of them returned similar results of around 80-90Mbps down and 860Mbps up.

I wanted to rule out the servers themselves just being overstressed and so ran the same test from an Azure VM, which probably isn't best test case but does confirm the servers are more than capable of delivering plenty more than my connection is capable of. Indeed, the FibreNest server which could barely break past 40Mbps on my connection was suddenly able to turn a cool 8526/6988 from the VM.

I also ran an iperf3 server on the VM and tested that from my PC, first with a simple single threaded test using the command 'iperf3 -c <vm_ip_addr> -R' for which the results were... not great. The average reported bandwidth was an almighty 1.11Mbps, yikes. I then tried using a parallel stream parameter and initially set this to 4, which improved things slightly to 32.7Mbps. Strangely 8 threads was worse at 8.64Mbps, 16 threads was 28.4Mbps, and 32 threads was significantly improved at 386Mbps. 64 threads managed 744Mbps and 128 threads was 786Mbps. I don't know how many parallel streams you would normally expect to use to test a 1Gbps connection, but 64-128 seems like an awful lot.

For more general use cases such as game downloads on Steam, I generally find it tends to top out at around 500Mbps give or take.

Given the extreme variation between results and the fact that several Manchester based servers returned very similar results, would this suggest a bandwidth constraint with peering or routing somewhere? Telcom are based in Manchester and peeringdb suggests they have a 20G link at LINX Manchester (https://www.peeringdb.com/net/8678).

Is there anything else that would be worth checking to try and get a better understanding of the [potential] limitations I seem to be running into?

What you want to do is find 3 'fast' and 3 'slow' locations (speed tests aren't the best example but they'll do too).. and compare the traceroutes - ideally both ways (from you to remote site and also back). You can get a trace from us back to you on the 'tools' page.

Also, have you made sure you're consistently using IPv4 and you don't have an "IPv6 is slow, IPv4 is fast" situation you're unaware from.

FWIW you can use the files on download.thinkbroadband.com for download tests you control if it helps for our side. Make sure trace is to download.thinkbroadband.com (www.thinkbroadband.com is on Cloudflare)

seb

In reply to a post by seb:

What you want to do is find 3 'fast' and 3 'slow' locations (speed tests aren't the best example but they'll do too).. and compare the traceroutes - ideally both ways (from you to remote site and also back). You can get a trace from us back to you on the 'tools' page.

Also, have you made sure you're consistently using IPv4 and you don't have an "IPv6 is slow, IPv4 is fast" situation you're unaware from.

FWIW you can use the files on download.thinkbroadband.com for download tests you control if it helps for our side. Make sure trace is to download.thinkbroadband.com (www.thinkbroadband.com is on Cloudflare)

seb

Thanks Seb

I don't believe WeFibre currently support IPv6 and so everything should be using IPv4 I believe.

I've tried the 5GB file from your downloads site and this ran at a rather unimpressive 950KiB/s. This was true in both the browser, and also from WSL using the curl command. I tried again a little while later and things were better at around 25-30MiB/s for a little while, before it eventually crashed down to 200KiB/s.

TBB speedtest results were also pretty woeful, albeit better: https://www.thinkbroadband.com/speedtest/17182368435...

The traceroute from my network and the traceroute from the TBB tools page are below.

Tracing route to download1.thinkbroadband.com [80.249.99.148]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 15 ms 7 ms 6 ms 100.64.0.1
3 7 ms 7 ms 8 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 7 ms 7 ms 8 ms cr1.ukf1.man--cr1.man2.telcom.network [185.108.168.150]
5 16 ms * 15 ms ae100.edge2.Manchesteruk2.Level3.net [195.50.120.173]
6 13 ms 19 ms 16 ms ae2.3215.edge7.lon1.neo.colt.net [171.75.8.71]
7 22 ms 19 ms 13 ms ae4.londra31.lon.seabone.net [195.22.209.222]
8 12 ms * 13 ms ae30.londra32.lon.seabone.net [195.22.209.220]
9 12 ms 12 ms 12 ms 149.3.183.85
10 12 ms 12 ms 12 ms download.thinkbroadband.com [80.249.99.148]

1 thinkbroadband-gw9.core-rs2.thw.ncuk.net 15 15 1ms 1ms 2ms
2 te1-52-37.core-rs1.thw.ncuk.net 15 15 1ms 1ms 1ms
3 xe-0-1-6-38.edge-rt1.thn.ncuk.net 15 15 0ms 1ms 17ms
4 149.3.183.84 15 15 0ms 2ms 8ms
5 149.3.183.83 15 15 0ms 11ms 42ms
6 level3.londra31.lon.seabone.net 15 15 0ms 2ms 11ms
7 ae1.6.edge2.man2.neo.colt.net 15 15 5ms 7ms 20ms
8 195.50.120.174 15 15 5ms 5ms 5ms
9 cr1.man2--cr1.ukf1.man.telcom.network 15 15 5ms 5ms 5ms
10 185.205.175.202.default.telcom.network 15 15 5ms 5ms 5ms
11 45.150.xxx.xxx.ipv4.telcom.network 15 15 12ms 13ms 17ms

With regards to good speedtest results, I was able to get decent performance from the following servers:

* Exa Networks Limited - Bradford (51737): 853/862
* Zen Internet - London (40788): 676/837
* IDNet - London (48505): 844/846

Generally the results of overwhelmingly poor, with most servers tested showing similar results to those below. I didn't get a single test above 100Mbps other than the few above which happily ran at 700-900Mbps.

* aql - Leeds (10580): 78/733
* Quickline Communications - Leeds (41597): 36/821
* YouFibre - London (55137): 33/832
* Hyperoptic - London (14679): 36/841
* Hyperoptic - Manchester (40628): 55/920
* TNP Ltd - Manchester (3504): 88/918
* Wildcard Networks - Newcastle (4058): 58/789

I also ran these tests from my phone to rule out any issues with my PC, but widely achieved similar results aside from the fast servers which were ultimately capped at around 250-300Mbps due to WiFi limitations.

What did stand out is that the Zen and IDNet servers in London didn't seem to encounter any issues, whilst YouFibre and Hyperoptic seemed to be limited to around 30-40Mbps.

Traceroute: Zen Internet (London - Fast)

Tracing route to speedtest02a.web.zen.net.uk [51.148.82.21]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 7 ms 100.64.0.1
3 7 ms 7 ms 7 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 8 ms 8 ms 8 ms cr1.ukf1.man--cr1.tcw1.man.telcom.network [185.108.168.146]
5 8 ms 7 ms 8 ms cr1.tcw1.man--cr1.man1.telcom.network [185.108.168.152]
6 8 ms 7 ms * 195.66.244.28
7 13 ms 13 ms 13 ms lag-2.br1.ixn-lon.zen.net.uk [51.148.73.195]
8 13 ms 13 ms 14 ms speedtest02a.web.zen.net.uk [51.148.82.21]

Traceroute: IDNet (London - Fast)

Tracing route to ookla.idnet.net [212.69.36.27]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 7 ms 100.64.0.1
3 15 ms 38 ms 7 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 12 ms 12 ms 12 ms cr1.ukf1.man--cr1.thn1.ldn.telcom.network [185.108.168.143]
5 14 ms 12 ms 12 ms cr1.thn1.ldn--cr2.ldn1.telcom.network [185.108.168.158]
6 18 ms 50 ms * idnet-gw-a.lonap.net [5.57.80.132]
7 12 ms 12 ms 12 ms telehouse-gw11.idnet.net [212.69.63.75]
8 13 ms 12 ms 12 ms 212.69.36.27

Traceroute: YouFibre (London - Slow)

Tracing route to speedtest-1.london.network.youfibre.com [45.92.46.45]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 7 ms 100.64.0.1
3 7 ms 7 ms 7 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 7 ms 13 ms 11 ms cr1.ukf1.man--cr1.tcw1.man.telcom.network [185.108.168.146]
5 8 ms 8 ms 7 ms cr1.tcw1.man--cr1.man1.telcom.network [185.108.168.152]
6 10 ms 9 ms * 195.66.244.173
7 * * * Request timed out.
8 16 ms * * 45.92.46.106
9 12 ms 12 ms 12 ms 45.92.46.45

Traceroute: TNP Ltd (Manchester - Slow)

Tracing route to speedtest.tnp.net.uk [5.61.120.37]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 7 ms 100.64.0.1
3 7 ms 7 ms 7 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 7 ms 8 ms 8 ms cr1.ukf1.man--cr1.tcw1.man.telcom.network [185.108.168.146]
5 7 ms 7 ms * cr1.tcw1.man--cr1.man1.telcom.network [185.108.168.152]
6 31 ms 20 ms 7 ms tnp-manc-rtr01.tnp.net.uk [195.66.244.39]
7 7 ms 7 ms 7 ms speedtest.tnp.net.uk [5.61.120.37]

Traceroute: Exa Networks Ltd (Bradford - Fast)

Tracing route to speed.exa.co.uk [82.219.4.110]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 7 ms 100.64.0.1
3 7 ms 7 ms 7 ms 185.205.175.201.default.telcom.network [185.205.175.201]
4 9 ms 8 ms 8 ms cr1.ukf1.man--cr1.tcw1.man.telcom.network [185.108.168.146]
5 7 ms 7 ms * cr1.tcw1.man--cr1.man1.telcom.network [185.108.168.152]
6 8 ms 7 ms 7 ms ixman-gw1.man-tcw.exa.net.uk [195.66.244.10]
7 8 ms 8 ms 8 ms brad-br-sr1.net.exa.net.uk [82.219.45.90]
8 9 ms 8 ms 8 ms 82.219.4.110

I don't see anything obvious between the routes for good and bad test servers, but perhaps someone might spot something I'm missing.

Hi, the traffic is routing through a provider called Telcom.

They have two 10Gb links into LINX Manchester, this is likely either load balanced or failover.

If it's load balanced, either a link is congested or there is a fault. My suspicion is a fault, as LINX don't like any links running at full capacity. If it's running in failover, the issue is within LINX.

The problem is, that's a rather big assumption that the problem is outside of Wefibre's network. They could also be running load balanced links within their network, again the exact same issue could be occurring.

If you haven't already, open a support ticket

In reply to a post by devonkev:

Hi, the traffic is routing through a provider called Telcom.

They have two 10Gb links into LINX Manchester, this is likely either load balanced or failover.

If it's load balanced, either a link is congested or there is a fault. My suspicion is a fault, as LINX don't like any links running at full capacity. If it's running in failover, the issue is within LINX.

The problem is, that's a rather big assumption that the problem is outside of Wefibre's network. They could also be running load balanced links within their network, again the exact same issue could be occurring.

If you haven't already, open a support ticket

Thanks, I believe WeFibre is a subsidiary of Telcom and so I would imagine they share the same core network, but I could be wrong on this.

I wanted to collate some more details on the issue before raising it as I know when I've questioned speeds in the past they just came back with the usual "it's a contended service" and tell you to update the router firmware.

I've raised it with them so I'll wait and see what they come back with.

In reply to a post by Toonshorty:

I wanted to collate some more details on the issue before raising it as I know when I've questioned speeds in the past they just came back with the usual "it's a contended service" and tell you to update the router firmware.

Just be glad you are not dealing with Vodafone. A couple of years back they had a black hole on their network which was dropping traffic to certain Google IP addresses as a result of an error in the peering routing with them. It was easy to demonstrate, easy to reproduce and, unquestionably, easy to fix.

It took them *weeks* and the few of us that were technically competent were literally pulling our hair out trying to get through their technical support teams to actually speak to someone who knew which way up to place a keyboard on their desk. When every single call to every single tier of support elicited the same response - "Your wifi is not working? Have you rebooted your computer?" - it's hard to keep your cool on the phone. I've never known anything like it and I have a special place on my dartboard of shame reserved for the company with what I consider to be the worst technical support function on the planet.

You have some great data here and I hope you have better luck in getting it to the right person than I ever did with Vodafone!

(ps. Voda's problem ended up being some decommissioned peering kit within their network that had not been properly removed from their routing. Should have been a trivial fix, especially given the dozen or so IT professionals providing them with detailed information of what the problem was!)

Interesting that Vodaphone has brain dead tech support as an ISP. My mobile is with Voxi, a Vodaphone brand. More than once it has taken a period of weeks to get them to address a cell tower which has switched itself off. All kinds of stupidities, like not having the correct post code for the tower. We are in a rural area, so only have the one tower available.

Typical stupidities are being told to go into the mobile settings to change things when they have already been told that the phone works perfectly well when we are down the road in the nearest town. They won't accept this as proof that the phone is OK. Another - after several hours of stupidity - is offering a phone back from a manager, when the problem is no connection.

Nobody who answers the phones at large companies is given the training to understand what they are saying, and they aren't empowered to actually raise anything directly to a network team. There's like a firewall between the customers and the networks team, and the support agents are on the customer side of that wall. If anything gets fixed as a result of people reporting it then it's more than likely a result of lots of complaints about the same thing finally triggered a response other than "nobody else has reported a fault", someone important at the company was inconvenienced by it, or it reached a threshold to trigger an alert somewhere.

Yeah, pretty much this. In the end, the only thing that got us anywhere was half a dozen of us IT professionals all raising formal complaints to Vodafone about their dreadful service and the fact that noone would even acknowledge the fault. This, finally, got someone to send an email to the right person who literally fixed it in an hour. He even commented that it would have been fixed quicker "had he known about it earlier"....arrrrrgh!

In reply to a post by daern:

Yeah, pretty much this. In the end, the only thing that got us anywhere was half a dozen of us IT professionals all raising formal complaints to Vodafone about their dreadful service and the fact that noone would even acknowledge the fault. This, finally, got someone to send an email to the right person who literally fixed it in an hour. He even commented that it would have been fixed quicker "had he known about it earlier"....arrrrrgh!

Yep. My mobile problem only got fixed one time - after several weeks - when I sent in a formal complaint complete with pictures of the cell tower including shots of the site reference no and Vodaphone labelling. It was fixed next day. Funny that.