Reasonable Single Threaded Performance BT Fibre 900

Hi,

I've recently had BT Full Fibre 900 installed.

Narrowed down the issue I'm seeing to single threaded peformance when downloading files.

When testing single threaded downloads using speedtest files from OVH (FR, 10Gbit, UK, 1-2Gbit), Leaseweb (NL), Hetzner (DE, 1Gbit, 1Gbit+), NFORCE (NL), and AWS S3 (UK, IE), I'm seeing around 20MB/s.

When testing the exact same speedtest files at the exact same hosts and DC's, using 8 threads, I saturate the line.

When using web based speedtests (such as thinkbroadband and speedtest.net) that offer the option of using a single thread, then using multiple threads, I see the same speeds repeated, slow single thread, fast multithread.

Testing from my Hetzner server (1Gbit up/down, DE, using FTPS), and brand new OVH server (10Gbit down/1-2Gbit up, UK, using HTTP), I see the same performance.
Previously with Virgin Media, I was able to saturate the 600Mbit line doing the same test over FTPS on the Hetzner server.

I've reached the end of the line with BT, they've issued me a deadlock letter, refuse to deal with the issue, and were completely useless beforehand.

Can anyone with BT Fibre 900 check their single threaded performance for me, and does anyone know what is commonly reached?

OVH
Leaseweb
Hetzner
NFORCE

Edited by kst (Thu 26-Aug-21 10:35:19)

In reply to a post by kst:

When using web based speedtests (such as thinkbroadband and speedtest.net) that offer the option of using a single thread, then using multiple threads, I see the same speeds repeated, slow single thread, fast multithread.

What sustained and burst single thread perf are you seeing?

I only have a BT 300Mbps service.

Running multithread it will give 320-330 Mbps. Going to single thread it drops very slightly and tends to be 310-320 so, not really noticable.

BT Full Fibre 900 here, lowest single threaded seen was 610Mbits/s, highest 790Mbits/s. Similar results on both IPv4 and IPv6. Multithreaded is normally close to 900.

What does your TBB speed test graph look like? That can give clues on what the issue might be.

It is probably congestion somewhere. I moved to FTTP 900 down via Openreach and on single thread tests got around 300Mbit/sec on Thinkbroadband, and multi-threaded around 800 and this was consistent for around 7 months. On speedtest.net I could pick some sites that easily achieved > 700M on single threaded tests, so it was what ever route it took to Thinkbroadband that was busy.

Then suddenly some extra capacity was added somewhere and Thinkbroadband is now > 600M on single threaded and > 900 on multi-thread tests, this is similar speeds to many tests I've seen for FTTP 900, so that busy link must have been upgraded.

When you say 20MB/sec do you mean Mega bytes, so 160Mbit/sec which if yes isn't unusable and many things we do on the Internet ends up using multiple threads to get at the data anyway.

There are other things which will be better and arguably more important than raw speed with being on FTTP rather than Virgin such as jitter, latency and packet loss, have you set up a BQM here and compared the two services? You might also have IPv6 now that Virgin don't do, this may or may not be important to you of course.

I'd put money on your speed issues resolving themselves in the coming months, there will be someone somewhere that knows a certain link is busy and plans will be in place to upgrade it. You might change to another ISP and find speeds the same if the busy link is close to you and you are just passing over the same route.

Edited by E300 (Thu 26-Aug-21 13:15:36)

What sustained and burst single thread perf are you seeing?

Single-thread, graphed:
ovh_1gbit_speedtest_file_singlethreaded_graphed.png
oshi.at/yfMjgN

leaseweb_1gbit_speedtest_file_singlethreaded_graphed.png
oshi.at/ziPhgT

hetzner_1gbit_speedtest_file_singlethreaded_graphed.png
oshi.at/zBggAc

Single-threaded tests, Firefox:

hetzner_1gb_speedtest_file_singlethreaded.png
oshi.at/hjgQQJ

leaseweb_1gb_speedtest_file_singlethreaded.png
oshi.at/sxjuUh

nforce_1gb_speedtest_file_singlethreaded.png
oshi.at/iniNMH

nvidia_graphics_card_drivers_singlethreaded.png
oshi.at/eKKVFu

ovh_1gb_speedtest_file_singlethreaded.png
oshi.at/EgFXjW

ovh_private_server_1gb_speedtest_file_singlethreaded.png
oshi.at/AXAdtZ

speedtest_net_singlethreaded.png
oshi.at/gRPRzs

windows_iso_download_singlethreaded.png
oshi.at/mPGFTW

windows_iso_download_singlethreaded_2.png
oshi.at/ESqVRD

Multi-threaded tests, 8 threads, graphed:

hetzner_private_server_1gb_speedtest_file_multithreaded.png
oshi.at/uDoqRn

leaseweb_speedtest_file_multithreaded.png
oshi.at/HJeQur

nforce_speedtest_file_multithreaded.png
oshi.at/eaaXcU

nvidia_graphics_card_drivers_multithreaded.png
oshi.at/GYuRNJ

ovh_private_server_1gb_speedtest_file_multithreaded.png
oshi.at/mwpGFM

ovh_speedtest_file_multithreaded.png
oshi.at/eKKVJB

speedtest_net_multithreaded.png
oshi.at/jHCLdf

windows_iso_download_multithreaded.png
oshi.at/xNujGj

windows_iso_download_multithreaded_2.png
oshi.at/jqkLMP

Edited by kst (Thu 26-Aug-21 22:15:59)

I only have a BT 300Mbps service.

Running multithread it will give 320-330 Mbps. Going to single thread it drops very slightly and tends to be 310-320 so, not really noticable

Single-threaded downloads are peaking higher than that, but sustained below.

What does your TBB speed test graph look like? That can give clues on what the issue might be.

Late last night:
thinkbroadband_singlethreaded_and_multithreaded.png
oshi.at/voYpZP

Right now, 22/08 22:19
thinkbroadband_singlethreaded_and_multithreaded_2608_2219.png
oshi.at/yyeiNx

Edited by kst (Thu 26-Aug-21 22:21:52)

Something wrong there. Are you testing directly wired up to the ONT (bypass any BT routers etc)

I can hit between 700-800 Mbps on single thread on Openreach FTTP using TTB. Through all my network.

Ping times seem on the high side too. I'd expect around half that (although not if your hitting a server in London from Scotland say)

Then suddenly some extra capacity was added somewhere and Thinkbroadband is now > 600M on single threaded and > 900 on multi-thread tests, this is similar speeds to many tests I've seen for FTTP 900, so that busy link must have been upgraded.

Realistically I'm seeing much slower single-threaded performance than the other couple people that responded.

When you say 20MB/sec do you mean Mega bytes, so 160Mbit/sec which if yes isn't unusable and many things we do on the Internet ends up using multiple threads to get at the data anyway.

Big B luckily, or unluckily in this case as BT are useless.

have you set up a BQM here and compared the two services?

I have not, and don't have the VM service anymore.

You might also have IPv6 now that Virgin don't do, this may or may not be important to you of course.

I do indeed!

I'd put money on your speed issues resolving themselves in the coming months, there will be someone somewhere that knows a certain link is busy and plans will be in place to upgrade it. You might change to another ISP and find speeds the same if the busy link is close to you and you are just passing over the same route.

I'm hoping so.

My complaint with BT reached deadlock within hours.
I've submitted a complaint through the ombusdman shortly ago.

Thanks for the response

Something wrong there. Are you testing directly wired up to the ONT (bypass any BT routers etc)

I'm glad someone agrees.

I've bypassed the BT router for testing and saw no difference, went straight to the ONT and set up a PPPOE connection on the computer I was testing from.

I can hit between 700-800 Mbps on single thread on Openreach FTTP using TTB. Through all my network.

Yeah I'm not getting anywhere near that on a single thread, normally around 20MB/s.

Ping times seem on the high side too. I'd expect around half that (although not if your hitting a server in London from Scotland say)

Ping will be a bit higher as I am going NE Scotland->London or France or Germany or Netherlands for the tests.

How would you proceed with a complaint like this?
I've had no luck with BT, or Openreach.

I documented what has happened so far here: https://community.bt.com/t5/BT-Fibre-broadband/Slow-...

The first post will be confusing, as I now know it's single-threaded performance that is the issue, but I thought I had ruled that out using a windows build of aria2 and trying 8 threads, which didn't work at all. I figured it out later.

Edited by kst (Thu 26-Aug-21 22:35:28)

Have you tested using another machine/NIC combo?

There may be other congestion, and I know that some folks in the past (not necessarily on BT Retail though!) who were experiencing sustained slowdowns, have requested from their service provider that BTW move their connection to a different SVLAN, which had less congestion.

I documented what has happened so far here: https://community.bt.com/t5/BT-Fibre-broadband/Slow-...

If I were you and the others are fine I would maybe refer to this thread as it is a very reputable site forum with a variety of experts from all over then country and world.
I always noticed twitter is a good place to complain when there is an issue as things seem to get fixed then. Up to you but it worked with Virgin media for me once, got an engineer appointment next day (for them that's about how long it would normally take to get off the hold music)

In reply to a post by Ad_G:

BT Full Fibre 900 here, lowest single threaded seen was 610Mbits/s, highest 790Mbits/s. Similar results on both IPv4 and IPv6. Multithreaded is normally close to 900.

What does your TBB speed test graph look like? That can give clues on what the issue might be.

Also 900 here, this speedtest below is typical: Had it for a year or so, and single thread has improved a bit, used to be 200 at one point.

Your images are not working for me, but I took a look at them on the BT side.

As you have a ramp up followed by a peak and then it dips back that is tending to imply congestion or very slight packet loss somewhere on the path. Are you able to run a probe to you servers to see if you can see any sign of packet loss? Pathping etc might help but as modern routers tend to de-prioritise ping it can give a false result.

One other thought as you mentioned being in NE Scotland (probably explains your latency being twice mine), have you looked at TCP window size configuration? The three things that are generally the case of single TCP thread are generally congestion, packet loss and latency/window size. With your latency are you maxing the window size? If so whilst there is still bandwidth free the server is waiting for ACKs before sending any more data.

From a quick check if you have a TCP window of 64Kbyte then then with 23ms of latency you are at around 22Mbits/s. Worth a look a the settings on your machine/server and if possible look at some packets to see what the window size is running at during the download.

What you can hit in another location is not relevant to someone else. Like I said in my post I could only hit around 300Mbits/sec on single threaded tests to Thinkbroadband for many months with IDNet, it was actually a bit slower than Cerberus 330 service I left on single thread tests, until all of a sudden it's over double that, and nothing changed my end, not even a drop in PPP. Links get upgraded, links get congested.

It is a CONTENDED service, no one is guaranteed 900Mbits/sec. If the ISP considers the speeds below whatever guarantees they sell them at, then at least you can leave and try another ISP.

I would suggest to the OP to make sure to perform tests via IPv4 to get a more like for like with Virgin, as IPv6 has a bit more overhead or might not be so well optimised on their system/router for some reason, and there is an IPv4 speed test button on Thinkbroadband.

If everything is okay on their system no amount of our help or suggestions is going to resolve their problem. We don't pay for an uncontended service with these products, slow downs are all part of it, hence the guaranteed minimums are way lower than 900Mbits/sec.

Those speed tests seem to be showing congestion somewhere. Try clicking on the Analysis button and see what it says about buffer bloat, and compare with an IPv4 test which I've always found gives me the faster numbers on speed tests.

You are buying a contended service, some people are lucky and get top speeds most or all the time, some of us go over more congested routes. Yours seems particularly congested but I'm sure it will just suddenly resolve itself as new capacity gets added. Virgin media has these problems quite often as well, just you didn't, horses for courses.

Maybe see if you can agree with the ISP to give it 60 days and if things are not improved then they allow you to still leave and cancel the contract.

Definitely worth setting up a BQM to make sure you have no packet loss going on, and this also may show up the congestion and times it is happening.

Edited by E300 (Fri 27-Aug-21 10:06:02)

In reply to a post by E300:

I would suggest to the OP to make sure to perform tests via IPv4 to get a more like for like with Virgin, as IPv6 has a bit more overhead or might not be so well optimised on their system/router for some reason

IPv6 actually has less overhead because there's no NAT involved and the connection is completely end-to-end. I see this reflected in latency too (although that could also be down to different routing...).

In reply to a post by E300:

What you can hit in another location is not relevant to someone else. Like I said in my post I could only hit around 300Mbits/sec on single threaded tests to Thinkbroadband for many months with IDNet, it was actually a bit slower than Cerberus 330 service I left on single thread tests, until all of a sudden it's over double that, and nothing changed my end, not even a drop in PPP. Links get upgraded, links get congested.

It is a CONTENDED service, no one is guaranteed 900Mbits/sec. If the ISP considers the speeds below whatever guarantees they sell them at, then at least you can leave and try another ISP.

I would suggest to the OP to make sure to perform tests via IPv4 to get a more like for like with Virgin, as IPv6 has a bit more overhead or might not be so well optimised on their system/router for some reason, and there is an IPv4 speed test button on Thinkbroadband.

If everything is okay on their system no amount of our help or suggestions is going to resolve their problem. We don't pay for an uncontended service with these products, slow downs are all part of it, hence the guaranteed minimums are way lower than 900Mbits/sec.

I know it’s contended. I’m not stupid. There is precedence, as you probably are aware as a former Cerberus customer, if folk requesting and getting moved to another SVLAN. Whether BT Retail will take that up with Openreach is debatable as they aren’t a business focussed ISP. But it’s something that could be explored.

In reply to a post by aidanh:

In reply to a post by E300:

I would suggest to the OP to make sure to perform tests via IPv4 to get a more like for like with Virgin, as IPv6 has a bit more overhead or might not be so well optimised on their system/router for some reason

IPv6 actually has less overhead because there's no NAT involved and the connection is completely end-to-end. I see this reflected in latency too (although that could also be down to different routing...).

That depends on your definition of overhead.
What exactly do you mean by less overhead because of no NAT?

When discussing packet overhead IPV6 has a larger header overhead at 40 bytes compared to IPV4.

The router doesn't have to keep track of loads of state for every single connection in order to translate your WAN address (e.g 203.0.113.56) to a private address on your LAN (e.g 192.168.1.22).

In reply to a post by aidanh:

IPv6 actually has less overhead because there's no NAT involved and the connection is completely end-to-end. I see this reflected in latency too (although that could also be down to different routing...).

I mean overhead as in how much of the packet sent is usable data, IPv6 has a larger header, so each packet carries less data, of course it also depends on MTU in play how exactly they might compare but like for like you get a bit less throughput on IPv6. Using IPv4 was just to rule out any oddity with IPv6 that might exist for the OP who had been all IPv4 on Virgin, i.e. compare like with like as much as possible to try and troubleshoot.

NAT shouldn't add any overheads (it doesn't make the packet headers larger) and whilst in theory a router might have to do a little bit more with NAT so affecting latency, it's doing it with smaller numbers, and NAT has been around long enough that hardware and software is very well optimised. IPv6 still requires the router to route and a firewall to keep state about the packets and it has to do this with bigger numbers, so really it's swings and roundabouts.

I've never seen any different in latency between the two myself, pinging the old UK favourite bbc.co.uk gives me 6ms on IPv4 via NAT and 6ms on IPv6.

In reply to a post by aidanh:

The router doesn't have to keep track of loads of state for every single connection in order to translate your WAN address (e.g 203.0.113.56) to a private address on your LAN (e.g 192.168.1.22).

Not true if your router is also a firewall, it still has to keep state and remember each connection. I can see all the states in pfSense and there are stacks of IPv6 connections it is following in order to allow incoming for anything originating from my network whilst deny unsolicited traffic.

If it's a stateful firewall, then yes. Pfsense is a stateful firewall but a stateless firewall is also possible which doesn't maintain any state at all and only looks at the packet header.