FTTP reasonable single thread expectations - what's yours?

Wondering what kind of single thread performance folk are seeing on 900/115 FTTP.

I just got Zen 900/115 FTTP installed.

Not sure really what are reasonable expectations for single threaded performance on FTTP. What is everyone else seeing (Both on Zen and other providers).

Using multithreaded testers I don't seem to have any big issue saturating the link. But, for instance, Fast.com will only saturate if you go above the default 8 threads to something higher than 20. Ookla gets there by itself when set to multithreaded (eventually, it takes a while to climb. Though the Ookla speed tests in single threaded mode, and TBB own single threaded test, both seem to peak around 50-60mbps, which is about in line with the Fast.com result.

Of course my sense of scale is totally distorted by the now massive headline speed. 50-60mbps wasn't far off my total connection capability for FTTC (several times greater for upload, actually), so in a way I feel in some ways I may be griping over nothing, but at the same time if there is extra performance I should reasonably (or even unreasonably!) be getting, then I'm keen to tune things up to the maximum possible, or hound for increased traffic priority, better routing, etc.

I plan on trying PPPoE from a gigabit laptop in Linux direct to the ONT to eliminate as much as possible any router / Windows OS issues, though I've already disabled all the router QOS/IDS stuff as that was knobbling the connection significantly due to CPU constraint (Ubiquiti USG 4 Pro).

Pings are great (ignore any cruft at the beginning of the graph, it was up and down a bit while I was setting up, and getting rid of the QOS stuff).
https://www.thinkbroadband.com/broadband/monitoring/...

Thanks for any insight!

Definitely you should do the direct PPPoE test into the ONT with Linux.

TCP is very sensitive to packet loss. There's a formula for how packet loss and RTT can limit the throughput of a single TCP stream:
https://en.wikipedia.org/wiki/TCP_tuning#Packet_loss

Given a 10ms RTT, then a 50Mbps limit could be caused by packet loss of around 0.05% (1 in 2000 packets):

1460/(0.01*sqrt(0.0005))*8 = 52,234,570

Your TBB test doesn't show any obvious packet loss, but then I don't think it's sensitive enough to be able to see that. You'd need a more sensitive tool like owamp (e.g. from the perfsonar toolkit), and that requires a remote partner to test against. (PS toolkit typically sends 10 test packets per second, i.e. 36000 per hour)

Unfortunately, even if you can confirm it directly, such low-level packet loss is hard to diagnose and fix. It could be in the upstream provider's network, or borderline light levels into the ONT, or in your own kit (which is why ruling that out is important).

I once had a higher level of packet loss on an Easynet leased line - about 0.2% - and it look a long time to persuade them it was on their side. Eventually they fixed it by moving me to a different switch port at their side. An FTTP service is much more limited in what they could do.

One other suggestion: it's useful to do different tests at different times of day and night. If the problem is consistent then it's unlikely to be a provider capacity issue.

Thanks for the pointers, very much appreciated. I'll probably try a couple of different laptops I have with different ethernet hardware, too.

Eliminating the equipment I have here as much as possible seems the sensible first step, but I still don't have any sense of scale. Is it realistic to expect a single tester thread instance on a remote server to be able to hit FTTP line rate? Or is reasonable expectation much lower? Are folk achieving much higher than my current results on Zen or other ISPs?

Without QOS enabled, with the very highest loads you could sometimes note yesterday low level packet loss for the duration of the test on the TBB graph, though I don't believe that is anything unusual, and is just what should happen I believe where there is no QOS in play - eventually you run out of bandwidth and have to drop packets, and some of them inevitably end up being the TBB packets?

Using your example of 1 in 2000 packets; I guess if totally random drops, on a 1 ping per second test, the sensitivity should be there? You'd expect almost 2 dropped pings per hour by law of averages, unless the drops only happen under heavy load conditions (which is totally reasonable I guess if some bit of gear is breaking under load). I think a single dropped ping does register as a dot on TBB? Seems like not a single drop in 14hrs of "normal" use.

The light level (at least for instantaneous measurement) seems fine, the OR engineer said they were about -13dB, with the level leaving the exchange being -11dB and the max being -26dB or so (I guess this was supposed to be dBm perhaps, but the engineer just said dB and I'm not fully across the tech for FTTP).

In reply to a post by jimbof:

Is it realistic to expect a single tester thread instance on a remote server to be able to hit FTTP line rate? Or is reasonable expectation much lower? Are folk achieving much higher than my current results on Zen or other ISPs?

My home line is 300/50 on Cerberus. Here is an iperf3 test to a server in a data centre on a 1G link - this is inherently single-threaded unless you ask for parallel connections (-P).

Text

1 23 45 67 89 1011 1213 1415 1617

iperf3 -c XXXXXXXX -R -4 ...[ ID] Interval           Transfer     Bitrate [  6]   0.00-1.00   sec  35.3 MBytes   296 Mbits/sec[  6]   1.00-2.00   sec  36.5 MBytes   306 Mbits/sec [  6]   2.00-3.00   sec  36.5 MBytes   306 Mbits/sec[  6]   3.00-4.00   sec  36.5 MBytes   306 Mbits/sec [  6]   4.00-5.00   sec  36.6 MBytes   306 Mbits/sec[  6]   5.00-6.00   sec  36.5 MBytes   307 Mbits/sec [  6]   6.00-7.00   sec  36.5 MBytes   306 Mbits/sec[  6]   7.00-8.00   sec  35.9 MBytes   301 Mbits/sec [  6]   8.00-9.00   sec  21.3 MBytes   179 Mbits/sec[  6]   9.00-10.00  sec  35.2 MBytes   295 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval           Transfer     Bitrate         Retr [  6]   0.00-10.00  sec   348 MBytes   292 Mbits/sec  157             sender[  6]   0.00-10.00  sec   347 MBytes   291 Mbits/sec                  receiver

I'm fortunate that the end-to-end RTT from here to that server is around 5ms - and as you can see from the formula, the lower the RTT is, the less sensitive TCP is to packet loss. Router at this end is Mikrotik RB4011iGS+RM.

You're right that 1-in-2000 packet loss would show as ~2 packets lost per hour, which may or may not be visible on those graphs. However if the problem *were* to do with bit error rate, then packet loss would be proportional to packet size. Large TCP packets (1500 bytes) would be affected more than small test pings (typically under 100 bytes).

I don't think light levels in the FTTP network are a likely problem. It's not impossible that a backhaul link somewhere is degraded due to faulty optics or dirt. But like I say, (a) it's very hard to diagnose, and (b) you'd need to rule out your own side first before taking this any further.

There is another test you can do: do a speed test to some server much further away, say in the USA. With a 100ms round-trip time, then you'd expect single-threaded TCP throughput to be 10 times lower than for a 10ms RTT, for a given level of packet loss.

OK, well it doesn't seem that complicated. Looks like every Windows PC I have here sucks for single threaded performance on Ookla. They all appear to work great in Linux.
Here is one example:

Linux single thread mozilla 843mbps
Windows single thread chrome 63mbps
Windows single thread mozilla 62mbps

I wonder what it is that is rubbish...

I did notice that if you use internet explorer it produces better results on the think broadband test.
I tested this using microsoft azure and it seems the thinkbroadband speedtest servers cap off at around 4 gigs but I can only get 2 Gigis regularly, while some of the speedtest.net servers seem to handle over 10Gigs a second download it just depends which ones.
Speedtest.net result

Speedtest result server 2



I forgot to mention there is a 50 Gig link to the azure server I am on.

Edited by RR_The_IT_Guy (Thu 15-Apr-21 18:28:51)

For your reference zen's speedtest server is slightly slower than the others but still achieves 8gig's a sec

ZEN server


Single Thread test

In reply to a post by jimbof:

I wonder what it is that is rubbish...

Some ideas:
* Third party security product
* Network card driver version/supplier
* Settings on the network card adaptor

Do you have similar speed problems copying files from one PC to another over network file sharing? (e.g. mapped drives) or using FTP? I can max out my LAN easily copying files from my desktop PC to my Synology NAS. Win10's Task Manager is good to see network traffic and it graphs as well, so you can spot any drops and recovers.

I thought i would suggest the software that a lot of people i know use and its free its called netstress by nuts about nets, here is a link

You download it on two pc's and it maxes out the link if you set it up in the settings, I know it's useful to identify issues as It has helped me to identify bottlenecks in the past so I would strongly suggest trying it.

In reply to a post by RR_The_IT_Guy:

I thought i would suggest the software that a lot of people i know use and its free its called netstress by nuts about nets, here is a link

We know the hardware is good because a different OS booted from USB worked, so it is a case of working out which part of the Windows system on that machine is slowing down.

Stress and diagnostic tools across a LAN are great, there is also https://totusoft.com/lanspeed mentioned on these forums recently. (I've not used).

I find copying large files (my weekly backup) is quite a good way to check I'm getting over 900 Mbps from my gigabit LAN.

Doing:

netsh interface tcp set global autotuninglevel=normal

appears to fix the performance for single threaded speed tests.
Now, I wonder why it was set to "disabled", I have no idea.

Thanks, at least I know now these servers are not fundamentally limited for single thread performance. It does look to be an OS config issue.

Probably some software that thinks it knows better than the OS vendor!

At a guess; some software I'm about to find out no longer works properly...

So it looks like yet another reason to hate Hamachi, that one of my clients insists on using!
https://community.logmein.com/t5/LogMeIn-Hamachi-Dis...
GRRRR.

I also heard of the one you're suggesting however did not like it as I think there were a few issues when i was trying to use it.

I find copying large files (my weekly backup) is quite a good way to check I'm getting over 900 Mbps from my gigabit LAN.

I would agree but it's worth remembering that you may be backing up onto a NAS or server of some description with multiple HDD's or SSD's potentially even cached on M.2 SSD's which would get you those speeds but I know I have a Synology 2 Bay NAS and with 2 WD RED 6TB HDD's and I would never get that kind of speed reading or writing onto it. I know my computer can more than handle 5Gbps even 6 over network to writing on the M.2 SSD's.

One suggestion I would make might be to try to check for updates as one windows update, I have had in the past made the computer think 100% usage was whenever it thought the connection maxed out when I could see on the Switch but also testing in Netstress that was only getting 200 when a week later it was back to the actual max after a load of updates.

Although I don't suggest the latest feature update as it seems to be causing a lot of Blue screens and is deleting system files which has caused me to reimage the same PC 3 times in one week so make sure to backup first if you risk windows update, I'm sure we all know what i mean.

We know the hardware is good because a different OS booted from USB worked, so it is a case of working out which part of the Windows system on that machine is slowing down.

I guess there is an answer that any windows user would say to that, once a solution is found a week later there will be another update that breaks it in another way again. Just like the blue screen printer driver issue last month.

I'm not sure if anyone has said this but has he tried to uninstall the network driver and manually override the install by trying to find the driver beforehand or even doing a force driver update as I know that's worked for me before.

All of this windowsness is why I have a Mac for network testing as I know that this is so unlikely to happen against windows where network's don't seem to be supported except as a fun joke to tell linux users.

In reply to a post by RR_The_IT_Guy:

I thought i would suggest the software that a lot of people i know use and its free its called netstress by nuts about nets, here is a link

You download it on two pc's and it maxes out the link if you set it up in the settings, I know it's useful to identify issues as It has helped me to identify bottlenecks in the past so I would strongly suggest trying it.

I couldn't get NetStress working on my main PC. Said something about shared adapter.