Thanks for the replies. Took a while to have the time to do the suggestions given as well as understand then implement them.

First off the traceroute:

Here are the results, first from when the speed was high:

traceroute to 90.130.74.153 (90.130.74.153), 30 hops max, 60 byte packets
1 _gateway (195.166.130.255) 7.061 ms 7.053 ms 7.237 ms
2 84.93.253.127 (84.93.253.127) 10.660 ms 10.656 ms 84.93.253.123 (84.93.253.123) 10.650 ms
3 195.99.125.134 (195.99.125.134) 10.646 ms 195.99.125.142 (195.99.125.142) 10.642 ms 195.99.125.138 (195.99.125.138) 10.841 ms
4 core6-hu0-0-0-26.faraday.ukcore.bt.net (195.99.127.70) 10.837 ms core6-hu0-3-0-26.faraday.ukcore.bt.net (109.159.252.142) 10.634 ms core5-hu0-4-0-26.faraday.ukcore.bt.net (109.159.252.144) 10.829 ms
5 core5-hu0-7-0-35.faraday.ukcore.bt.net (62.6.201.246) 10.620 ms 166-49-209-132.gia.bt.net (166.49.209.132) 10.819 ms *
6 166-49-209-132.gia.bt.net (166.49.209.132) 10.811 ms t2c3-et-7-3-0.nl-ams2.gia.bt.net (166.49.195.174) 17.291 ms 166-49-209-132.gia.bt.net (166.49.209.132) 11.351 ms
7 * * *
8 * * ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 17.465 ms
9 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 17.847 ms ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 18.397 ms 18.393 ms
10 * ams-core-2.bundle-ether48.tele2.net (130.244.38.232) 18.385 ms 18.381 ms
11 ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 19.578 ms !X * *

And when it was low:

traceroute to 90.130.74.153 (90.130.74.153), 30 hops max, 60 byte packets
1 _gateway (195.166.130.255) 6.405 ms 6.396 ms 6.580 ms
2 84.93.253.123 (84.93.253.123) 6.782 ms 6.571 ms 84.93.253.127 (84.93.253.127) 6.773 ms
3 core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 6.922 ms 6.765 ms 195.99.125.142 (195.99.125.142) 68.380 ms
4 core5-hu0-7-0-26.faraday.ukcore.bt.net (195.99.127.32) 6.910 ms 6.907 ms core6-hu0-4-0-26.faraday.ukcore.bt.net (109.159.252.146) 6.544 ms
5 166-49-209-132.gia.bt.net (166.49.209.132) 7.671 ms 6.895 ms core5-hu0-7-0-35.faraday.ukcore.bt.net (62.6.201.246) 6.532 ms
6 t2c3-et-5-1-2.nl-ams2.gia.bt.net (166.49.208.217) 11.332 ms 11.670 ms 166-49-209-132.gia.bt.net (166.49.209.132) 7.284 ms
7 * * t2c3-et-5-1-2.nl-ams2.gia.bt.net (166.49.208.217) 12.866 ms
8 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 73.860 ms * *
9 ams13-agg-1.bundle-ether4.tele2.net (91.129.14.30) 73.848 ms 73.844 ms ams-core-2.bundle-ether48.tele2.net (130.244.38.232) 13.226 ms
10 * * ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 13.924 ms
11 ams-core-2.bundle-ether8.tele2.net (130.244.38.232) 13.938 ms !X * *

Next up Iperf3 results:

Command used server side was this: iperf3 –s -f M –p #####
Command used client side was this: iperf3 –c xxy.xxy.xxy.xxy -f M –R –t 350 –p #####
Hopefully they were the right commands to use.

The below image link shows a screen shot of the server side of things while client side was downloading. Still gives same problem.

https://imgur.com/a/TkN7Fx5

As for the tcpdump thing I tried to do it but didn’t understand what it was i was doing. Below is a screen shot of some results. No idea if they are any good though. Took them whilst doing an iperf3 test, client side.

https://imgur.com/a/JpSdp3q

Installing netdata gave nice visual representation of the data being downloaded but I already had that with system monitor? Was I meant to use it to do something specific with it?

Moving on to my MTU settings, here they are: https://imgur.com/a/oV3LLbh

I’m about to go start figuring out SQM but here is a BQM when I had the Plusnet router attached for a full day. Ignore the first hours red, twas me causing it.

https://www.thinkbroadband.com/broadband/monitoring/...

As to where I’m downloading to, over time I’ve downloaded to both ssd’s and nvme drives on different computers via different OS’s. Using an nvme drive at the moment. How would I tell if it’s a cache issue?

Current results are using this system spec with a direct PPPOE connection to the ONT:

I9-10900k
32GB Ram
Z490-E Gaming
970 Evo Plus 500GB

could use wget and send the file to NULL to rule out any local storage performance issues

ie.

wget http://hgd-speedtest-1.tele2.net/100GB.zip -O NUL

In reply to a post by Icgaln:

First off the traceroute:

That's interesting: hops 2 to 5 changed. You may find it's just doing load balancing ("equal cost multipath"), which is normally fine. But if you do lots of tests and the traceroute changes correlate completely with the low/high speed period, then it points to a problem in Plusnet's network which you can raise with them.

It's also good if you can traceroute in the reverse direction: starting from a VPS say, traceroute to your outside public IP. That could also be switching back and forth - and indeed is more likely to affect the throughput in the Internet-to-you direction. If *those* traceroute changes correlate with the low/high speed changes, that's a smoking gun. You can use "mtr" instead of "traceroute" to get a more real-time view.

If iperf3 shows the same high/low speed profile, then it rules out SSD issues, since iperf throws away the data rather than writing it anyway.

Netdata was just intended to give you a real-time view of this, which you can look at either while doing a speedtest or while doing regular activity, such as downloading a large file. It can also highlight other possible bottlenecks, like for example the CPU on your box going up to 100% for some other, unrelated reason (e.g. an antivirus kicking in). But if you are doing this testing under Ubuntu, that is very unlikely to be something that you see.

I did a NUL test anyway just to see what it would do.

Here are the results when the speed was high: https://imgur.com/uHaq4EL
And when they were low: https://imgur.com/z1d9pde

Makes me feel confident now it isn’t a hard drive issue.

As to the trace routing and hop changing I’ve found the hops change whether I download or not so it made it very hard to figure out if they specifically change in correlation to the speed changing when downloading something as I get the same results if I’m not download something anyway.

Here are some examples of three traceroutes from my server to my plusnet IP without downloading anything. I was in a creative mood so did a spread sheet and colored them to highlight the different hops taken: https://imgur.com/abh8IEW

And here’s when I was doing an iperf3 test using mtr like was suggested: https://imgur.com/XyJHZfw

Using mtr confused me a little as it didn’t seem to give the same number of hops as traceroute does so I did a traceroute at the same time and got this: https://imgur.com/iL1Dqqo

For some reason the mtr wasn’t registering some of the hops given in the traceroute like:
be-3.cr02.ams-01.nl.leaseweb.net or core2-hu0-16-0-0.southbank.ukcore.bt.net.

This lead me down a googling path of understanding that mtr seems to use ICMP by default and traceroute uses udp so I then tried: mtr –u xxy.xxy.xxy.xxy.xxy and: mtr -T xxy.xxy.xxy.xxy.xxy and got these results:
https://imgur.com/s4Omp7D
https://imgur.com/BMKxFgF

Seeing this has gotten me excited because the 50% packet loss makes me think this could be my problem. What I find even more interesting is that I got these results with nothing being downloaded.

So is this the problem? Looks like one of the hops on the plusnet network has packet loss for both tcp and udp traffic?

That shows that one node is not responding directly to being pinged.
The node did route all the traffic to the subsequent nodes.
Packet loss is zero from all subsequet nodes
Only loss that persists to the end node would be significant

Indeed. Many routers de-prioritise responding to ICMP, or rate limit it, or block it entirely.

Bah! I got all excited for nothing

I also tried setting up opnsense with SQM but it didn’t seem to do anything. I used this guide: https://forum.opnsense.org/index.php?topic=7423.0 and set it up like this, was this right?

https://imgur.com/9qv3vR0
https://imgur.com/EgAWopT
https://imgur.com/eI1bLP3
https://imgur.com/VAeQRU7
https://imgur.com/PYXhEyr
https://imgur.com/dctw6K0

Is there anything else I can try?