Reducing speed when downloading.

Hi,

I’m trying to solve/understand what’s going wrong with my new FTTP connection (900/112) from Plusnet.

In a nutshell the problem is that when I download a file that takes a while to download I get the high speed I would expect for a short while followed a very specific lower speed of 41.2MB/s for another short while before it goes back up to the expected higher speed, constantly and predictably lowering and raising the speed at regular intervals and at a set capped speed when going lower. (Please see bellow image, taken while using ubuntu 22 with a direct PPOE connection to the ONT).

Image: https://imgur.com/a/zFga3iD

After a lot of messing about trying to solve the problem with the Plusnet router attached, an opnsense router attached, an edge router X attached and now finally just plugging a ubuntu 22 computer directly into the ONT and configuring a PPOE connection directly to it I feel like the problem lies with the ONT and beyond or an inherent problem with PPOE rather than anything my side as they all produce the same results using different computers, OS’s and network cards (Intel - both onboard and dedicated nic, realtek and atheros)

It’s not just the tele2 file test I’ve tried downloading by the way. It happens with any large file I try to download from multiple file mirrors for many Linux distribution ISO’s and other places I could find that had large files I could download. It also happens with torrents as well and manages to lower the speed to the exact same reduced speed of 41.2MB/s.

The computers I’m downloading too also have crucial and Samsung nvme drives in them so I would like to think it’s not them that are slowing the download down. Going computer to computer across my LAN lets me download a file at full 1000mbps speeds so everything works fine in that regard and proves the system/network/drives are more than capable of handling the those kind of speeds at a consistent high rate.

Having tried to read and understand other posts with similar issues both on these forms and elsewhere would I be right in thinking it’s a PPOE issue or maybe this thing called ‘bufferbloat’?

I feel like I can’t do much more than using an ubuntu 22 computer with direct PPOE connection to the ONT. It’s an I5-3570 with 16gb ram. I feel like it should be fast enough to cope with the speed. I can try using ubuntu 22 and a direct PPOE connect to the ONT with an i9-10900k if people think it might help but that system would require a more involved dismantling of what I’ve already got it setup as and I felt like the i5-3570 should be perfectly capable?

As far as I’ve discovered, so far, everything else is fine and standard speed tests come back great. It’s just the real world application of downloading something at a decent consistent speed, which is what I would want/expect a 900mbps connection to be able to do.

For reference: https://www.thinkbroadband.com/speedtest/16877970415...

So my question is, is there something I’m missing or something else I can try to solve this problem or is this just a limitation of the ONT and the technology used by Plusnet to make the connection and is considered ‘normal’?

If there is nothing I can do I could live with that. I just need to know for sure first as i don’t want to pay for a 900mbps connection only to receive around 353mbps half of the time I’m downloading something when there actually is a solution to stop this happening.

Any help appreciated.

Were you with Plusnet before the move to FTTP ?

Do they still hold those speed profiles for their customers which are on their system ?

Your old FTTC give you a 41meg connection perchance ?

Just a random guess …

I was with Plusnet before I switched to their FTTP connection.

How would I find this out about the speed profiles?

I used to connect at around 61mbps on my FTTC connection giving around 7MB/s download speeds.

I believe it is/was shown on some online ‘customer profile’ or portal type thing.

My theory is scotched though as the speeds you report aren’t a match for those previously received via FTTC.

Someone else will have other suggestions shortly I’m sure.

Not sure what is doing it but I am afraid your theory was squashed from the first image posted. It states "41.2MiB" - that is 329.6Mb/s and so speeds that aren't possible on FTTC.

It is an odd and consistent pattern that suggests that PlusNet may be doing shaping for sustained high speed downloads but if they are then it would seem unnecessary and unfair unless they have capacity issues.

That is an interesting one. Suggestions:

- do a traceroute to the target host, during the "fast" periods and the "slow" periods, and see if there's a difference
- using tcpdump, look at the TTL of the returned packets, during the "fast" periods and the "slow" periods, and see if it changes

Those would be indications of routes changing within Plusnet's network.

Next:

- capture traffic to a file using tcpdump (tcpdump -i pppoe0 -nn -s0 -w ppp.pcap) and analyse it using tcptrace. This isn't for the faint-hearted, but gives you low level details about packet loss, retransmissions, out-of-order packet delivery etc.
- to rule out the issue with writing to SSD (and I *have* seen SSDs which slow down when they heat up), you could run an iperf3 test, although that will also require an endpoint in the cloud somewhere, like a VPS. There is also a speedtest CLI you could try: but you'd need to see if you can configure it to make it run over a longer period, like 60 seconds, to reproduce the behaviour you're seeing.
- install netdata to see instantaneous stats on your Linux box, including network interface traffic. (Netdata is a monitoring tool which collects data at 1-second intervals by default. These days, they try quite hard to get you to signup for Netdata Cloud. Ignore this; netdata works just fine standalone. Point your browser at http://127.0.0.1:19999 to see all the graphs)

Maybe after all that, you will have some more clues.

Have you checked your MTU is correct?
As you have access to various routers that support it, have you tried using SQM e.g Cake or fq_codel? You will lose a little of the speed but should result in a much more consistent connection if it is bufferbloat related.

Do you have a BQM setup (mine is in the sig) and if so what does the connection look like?

Where are you downloading the file to? Is this an SSD cache behaviour issue?

Not sure what is doing it but I am afraid your theory was squashed from the first image posted. It states "41.2MiB" - that is 329.6Mb/s and so speeds that aren't possible on FTTC.

Fair enough

In reply to a post by Zarjaz:

Not sure what is doing it but I am afraid your theory was squashed from the first image posted. It states "41.2MiB" - that is 329.6Mb/s and so speeds that aren't possible on FTTC.

Fair enough

Thanks for the replies. Took a while to have the time to do the suggestions given as well as understand then implement them.

First off the traceroute:

Here are the results, first from when the speed was high:

traceroute to 90.130.74.153 (90.130.74.153), 30 hops max, 60 byte packets
1 _gateway (195.166.130.255) 7.061 ms 7.053 ms 7.237 ms
2 84.93.253.127 (84.93.253.127) 10.660 ms 10.656 ms 84.93.253.123 (84.93.253.123) 10.650 ms
3 195.99.125.134 (195.99.125.134) 10.646 ms 195.99.125.142 (195.99.125.142) 10.642 ms 195.99.125.138 (195.99.125.138) 10.841 ms
4 core6-hu0-0-0-26.faraday.ukcore.bt.net (195.99.127.70) 10.837 ms core6-hu0-3-0-26.faraday.ukcore.bt.net (109.159.252.142) 10.634 ms core5-hu0-4-0-26.faraday.ukcore.bt.net (109.159.252.144) 10.829 ms
5 core5-hu0-7-0-35.faraday.ukcore.bt.net (62.6.201.246) 10.620 ms 166-49-209-132.gia.bt.net (166.49.209.132) 10.819 ms *
6 166-49-209-132.gia.bt.net (166.49.209.132) 10.811 ms t2c3-et-7-3-0.nl-ams2.gia.bt.net (166.49.195.174) 17.291 ms 166-49-209-132.gia.bt.net (166.49.209.132) 11.351 ms
7 * * *
8 * * ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 17.465 ms
9 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 17.847 ms ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 18.397 ms 18.393 ms
10 * ams-core-2.bundle-ether48.tele2.net (130.244.38.232) 18.385 ms 18.381 ms
11 ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 19.578 ms !X * *

And when it was low:

traceroute to 90.130.74.153 (90.130.74.153), 30 hops max, 60 byte packets
1 _gateway (195.166.130.255) 6.405 ms 6.396 ms 6.580 ms
2 84.93.253.123 (84.93.253.123) 6.782 ms 6.571 ms 84.93.253.127 (84.93.253.127) 6.773 ms
3 core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 6.922 ms 6.765 ms 195.99.125.142 (195.99.125.142) 68.380 ms
4 core5-hu0-7-0-26.faraday.ukcore.bt.net (195.99.127.32) 6.910 ms 6.907 ms core6-hu0-4-0-26.faraday.ukcore.bt.net (109.159.252.146) 6.544 ms
5 166-49-209-132.gia.bt.net (166.49.209.132) 7.671 ms 6.895 ms core5-hu0-7-0-35.faraday.ukcore.bt.net (62.6.201.246) 6.532 ms
6 t2c3-et-5-1-2.nl-ams2.gia.bt.net (166.49.208.217) 11.332 ms 11.670 ms 166-49-209-132.gia.bt.net (166.49.209.132) 7.284 ms
7 * * t2c3-et-5-1-2.nl-ams2.gia.bt.net (166.49.208.217) 12.866 ms
8 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 73.860 ms * *
9 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 73.848 ms 73.844 ms ams-core-2.bundle-ether48.tele2.net (130.244.38.232) 13.226 ms
10 * * ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 13.924 ms
11 ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 13.938 ms !X * *

Next up Iperf3 results:

Command used server side was this: iperf3 –s -f M –p #####
Command used client side was this: iperf3 –c xxy.xxy.xxy.xxy -f M –R –t 350 –p #####
Hopefully they were the right commands to use.

The below image link shows a screen shot of the server side of things while client side was downloading. Still gives same problem.

https://imgur.com/a/TkN7Fx5

As for the tcpdump thing I tried to do it but didn’t understand what it was i was doing. Below is a screen shot of some results. No idea if they are any good though. Took them whilst doing an iperf3 test, client side.

https://imgur.com/a/JpSdp3q

Installing netdata gave nice visual representation of the data being downloaded but I already had that with system monitor? Was I meant to use it to do something specific with it?

Moving on to my MTU settings, here they are: https://imgur.com/a/oV3LLbh

I’m about to go start figuring out SQM but here is a BQM when I had the Plusnet router attached for a full day. Ignore the first hours red, twas me causing it.

https://www.thinkbroadband.com/broadband/monitoring/...

As to where I’m downloading to, over time I’ve downloaded to both ssd’s and nvme drives on different computers via different OS’s. Using an nvme drive at the moment. How would I tell if it’s a cache issue?

Current results are using this system spec with a direct PPPOE connection to the ONT:

I9-10900k
32GB Ram
Z490-E Gaming
970 Evo Plus 500GB

could use wget and send the file to NULL to rule out any local storage performance issues

ie.

wget http://hgd-speedtest-1.tele2.net/100GB.zip -O NUL

In reply to a post by Icgaln:

First off the traceroute:

That's interesting: hops 2 to 5 changed. You may find it's just doing load balancing ("equal cost multipath"), which is normally fine. But if you do lots of tests and the traceroute changes correlate completely with the low/high speed period, then it points to a problem in Plusnet's network which you can raise with them.

It's also good if you can traceroute in the reverse direction: starting from a VPS say, traceroute to your outside public IP. That could also be switching back and forth - and indeed is more likely to affect the throughput in the Internet-to-you direction. If *those* traceroute changes correlate with the low/high speed changes, that's a smoking gun. You can use "mtr" instead of "traceroute" to get a more real-time view.

If iperf3 shows the same high/low speed profile, then it rules out SSD issues, since iperf throws away the data rather than writing it anyway.

Netdata was just intended to give you a real-time view of this, which you can look at either while doing a speedtest or while doing regular activity, such as downloading a large file. It can also highlight other possible bottlenecks, like for example the CPU on your box going up to 100% for some other, unrelated reason (e.g. an antivirus kicking in). But if you are doing this testing under Ubuntu, that is very unlikely to be something that you see.

I did a NUL test anyway just to see what it would do.

Here are the results when the speed was high: https://imgur.com/uHaq4EL
And when they were low: https://imgur.com/z1d9pde

Makes me feel confident now it isn’t a hard drive issue.

As to the trace routing and hop changing I’ve found the hops change whether I download or not so it made it very hard to figure out if they specifically change in correlation to the speed changing when downloading something as I get the same results if I’m not download something anyway.

Here are some examples of three traceroutes from my server to my plusnet IP without downloading anything. I was in a creative mood so did a spread sheet and colored them to highlight the different hops taken: https://imgur.com/abh8IEW

And here’s when I was doing an iperf3 test using mtr like was suggested: https://imgur.com/XyJHZfw

Using mtr confused me a little as it didn’t seem to give the same number of hops as traceroute does so I did a traceroute at the same time and got this: https://imgur.com/iL1Dqqo

For some reason the mtr wasn’t registering some of the hops given in the traceroute like:
be-3.cr02.ams-01.nl.leaseweb.net or core2-hu0-16-0-0.southbank.ukcore.bt.net.

This lead me down a googling path of understanding that mtr seems to use ICMP by default and traceroute uses udp so I then tried: mtr –u xxy.xxy.xxy.xxy.xxy and: mtr -T xxy.xxy.xxy.xxy.xxy and got these results:
https://imgur.com/s4Omp7D
https://imgur.com/BMKxFgF

Seeing this has gotten me excited because the 50% packet loss makes me think this could be my problem. What I find even more interesting is that I got these results with nothing being downloaded.

So is this the problem? Looks like one of the hops on the plusnet network has packet loss for both tcp and udp traffic?

That shows that one node is not responding directly to being pinged.
The node did route all the traffic to the subsequent nodes.
Packet loss is zero from all subsequet nodes
Only loss that persists to the end node would be significant

Indeed. Many routers de-prioritise responding to ICMP, or rate limit it, or block it entirely.

Bah! I got all excited for nothing

I also tried setting up opnsense with SQM but it didn’t seem to do anything. I used this guide: https://forum.opnsense.org/index.php?topic=7423.0 and set it up like this, was this right?

https://imgur.com/9qv3vR0
https://imgur.com/EgAWopT
https://imgur.com/eI1bLP3
https://imgur.com/VAeQRU7
https://imgur.com/PYXhEyr
https://imgur.com/dctw6K0

Is there anything else I can try?