In reply to a post by LIGHTFAST:

Thanks for the info very interesting to hear. I do work for a business telecom provider and connected lots of gamma and zen connections but I haven't come across this issue on fttp. I've just swapped the router out for a draytek 2766. Had the same result with packet loss. If I bandwidth limit the connection to 440mb the packet loss stops. I do see the occasional 1x 1000ms+ response in the mix which is odd but no packet loss.

Can't imagine the Plusnet guy coming will have any idea on this one. Unless they can escalate it.

Just out of intrest, on the PC side, what speeds do you see in task manager when the limit is enforced, is it more than 440 or does it not respect it?
I have noticed the issue you raise, if under bandwdth limit as it doesn't adhear to it continuously for some UDP game updates.

For me it's not a big issue as I limit each device and VLAN well under the gig symmetric to stop one device or VLAN using it all.

The 'engineer' visit is probably from a useless outfit called Circet who Plusnet use to sort out complicated issues like not connecting the router up correctly so they will be a complete waste of time. Might be worth telling them not to bother.

Not plusnet so can't really help in this regard, but I did do a bit of tuning on my own OPNsense firewall when we switched to Youfibre 1000/1000 earlier this year in order to reduce the impact of multiple connections on the overall connection latency.

In effect, I apply some shaping rules to cap the overall bandwidth slightly below the line speed to ensure that there is typically additional bandwidth available for other services. This is visible on a speedtest when you look at the contended vs uncontended latency figures which for me are typically 3ms (uncontended) and maybe 6-10ms (contended). Without shaping, the contended latency can easily jump over 100ms or more which means that a single heavy user can significantly impact other activities in the house:

https://www.speedtest.net/result/16926798950
(it's the blue "download latency" figure - 10ms here)

With shaping disabled, the speed is slightly higher, but the contended latency goes through the roof and would certainly have other users complaining about performance if you were maxing out the connection:

https://www.speedtest.net/result/16926805762

There is a small, peak throughput cost to this, but in the real world it's well worth doing. Perhaps something available to you in your openwrt router? Perhaps also worth checking the performance of your router too, to be sure it's not the cause. Mine can easily shift 10G+ so I know that horsepower is not a limitation here.

TCP will attempt to find the fastest the path can manage therefore packet loss under full load is basically guaranteed to be happening somewhere. QoS and other AQM systems manipulate the flows to take advantage of this behaviour. (I literally make my living selling managed networks that make use of these technologies).

The issue you see is that the connection is under buffered + the LNS server cant enter and exit the dropping state quickly enough, which; combined with the bursty nature of the CDN traffic (which is probably an artifact of efficiency/power saving measures on the CDN side called generic segementation offload) leads to an odd situation where you cant use your full connection speed when downloads from certain sources are happening.

There is no local network issue happening. The issue is at the LNS within the providers data centre. Do not order an engineer to your house for this!

To fix this one of the following needs to happen:

1) The CDN be less bursty, which could be achieved by the game download systems using less TCP threads for their download or adjusting parameters on the CDN.
2) The buffer in the ISP network be increased to a size such that it can accommodate the CDN traffic bursts
3) The LNS at the ISP be tweaked to allow it to enter and exit the dropping state more rapidly, so it can still pass the full line rate even when dropping heavily

The only realistic option in my opinion is 2 because the CDNs are a law unto themselves. However as yet nobody I've ran into; apart from select people high up at BT group really understand whats going on here. End users just blame the provider for being 'slow' however ultimately as you can see the situation is far more complicated. This isnt an end user router throughput issue, not an issue with PPPoE vs IPoE (DHCP) etc. This is simply a case of the bottle kneck being under buffered.

The major headache here though is that the providers are running at such huge scale that providing say 50ms buffer for a 1Gbps service needs 15MB of ram, or thereabouts, plus overheads and I suspect the buffer capacity for the LNS is extremely expensive. That coupled with the race to the bottom price point means they cant/wont do it. Bit of a tricky one?!

Thanks for the help. I've told plusnet its likely a waste of time sending the in house engineer and asked if it would be better to raise a ticket with their network team but they dont seem to be able to deviate from the script. that if the line test passes its an in house problem even though i've sent images and explained the issue.

I wonder if an upgrade to 900mb would help?

Really is a pain and im about 5 months into a 2 year contract

Slower services have the same buffer size as far as I am aware, in effect meaning their buffer is larger so for this purpose the slower services are actually 'better'. On the plus side, with the gig (sold as 900m) services your 100+GB game downloads in about 15 minutes so not the end of the world.

I had the same experience trying to get support through the business as usual processes on this. Thankfully I had contacts in other departments for various reasons.

Out of interest what service did you subscribe to with your altnet previously?

Did have 500/500 on brsk which was much better even with CGNAT. Then when moving house I ordered plusnet (no other altnets around). They put a SOGEA in as a temp line until some civils could be done for FTTP (which got installed after a month or 2).

I think agoodm explained it very well.

CDN's can be quite aggressive as I expect they constantly dealing with "its too slow " complaints, so threads get bumped up and the window size gets pumped up to a very high value.
CDN's also bring the internet closer to us, but there is a downside, a low RTT in my opinion aggravates this problem, because steam allows you to choose your mirror, I found as an example choosing a mirror further way was quite effective at mitigating the problem.

In my personal experience I had the problem quite bad when I was on VDSL, I never resolved it in what I consider a proper way, but instead did things like apply a speed limit in the steam download client, traffic shaped downloads on my games consoles, and eventually started traffic shaping my ack's on steam. What I noticed on steam is if you restrict the download speed, it still remained very bursty as agoodm described, so if I as an example set it to 30mbit/sec, instead of being constantly around 30mbit/sec it would burst to 60, then stop for a second then burst to 60 again, so it averaged out slower, but when it bursted, it was prone to harming the quality of my connection. I discovered shaping the ack's upstream would naturally slow it down to a more constant speed by reducing the congestion window on the CDN sending the data. The most effective shaping was using an intermediate VPN and shaping the output to myself from that. I also late on artificially added latency on dummynet for steam and console traffic.

I had all sorts of potential causes of the problem including the very small buffers from the ISP or backhaul provider, but of course none of this was ever proven, it would be very difficult to do so. However one day when I had issues on the VDSL connection I started running my entire lan from 4G over my phone and noticed the behaviour completely changed, it no longer had these symptoms, and I could push the connection to 99% with less of an issue than if I was pushing the VDSL to 80%. Once I witnessed this I started the process of changing to a difference fixed line connection.

I then moved to gig1 on Virgin Media which is carried out over DHCP not PPPoE, and had much more bandwidth, VM do buffer a lot more data as well. The problems completely vanished, I removed all configurations related to managing downstream traffic, and only left a basic upload QoS in place.

I am now on a gigabit both ways connection to CityFibre via AAISP, initially I had a regression although it was nowhere near as bad as VDSL was, I then moved to more powerful hardware so the PPPoE wasnt saturating the CPU and made sure I had working max-mss headers. The connection also uses baby jumbo frames to allow a 1500 byte MTU, all these changes, its now almost as good as things were on VM. It is also worth mentioning I cant fully saturate the connection from a single device due to having a gigabit LAN, this may be helping things, as it kind of acts as a built in QoS that keeps a bit of unutilised bandwidth at all times.

Since gigabit is harder to saturate, I think that alone would improve things, I do think providing you provide enough hardware grunt to drive the PPPoE and make sure you dont have fragmentation issues, then upgrading to a faster connection alone might kill off the problem.

Because of my previous VDSL issues is why I took an interest in your post.

Edited by Chrysalis (Sat 26-Oct-24 18:15:51)

The memory for buffers on the Nokia SRs lives on the line cards and is shared and contended: when it runs out packets get dropped however I can't see it being full at 2pm on a weekday so that seems unlikely.

Plusnet punters signing up any time recently would've had the LAC and LNS the same way EE, BT Consumer and BT's business services run - the BT Wholesale Nokia SR in the headend exchange doing everything.

Can see a reasonable amount of buffer there, 10ms or so. It's behaving like either the other side is super inconsiderate with tons of TCP flows using an aggressive stack, UDP with poor or no congestion control over the top or there's a configuration mismatch.

Another thing that comes to mind is if OP is near a BT Wholesale POP where there are CDN caches so the latency to those is really low: they will respond super fast to loss and ramp the windows back up really fast from the tiny round trip times.

I've no idea if it'd be useful but be interesting to see the IP Profile the BT Wholesale speed checker shows for this to make sure it's correct, and to rebuild the service on the BT Wholesale network then if still bad the Openreach OLT. The rate limiting happens in two places, egress on the SR facing the Openreach network then egress on the Openreach OLT PON port and the SR should be taking care of it: no Openreach customer should ever be sending more to the OLT than the customer is provisioned for.

Edited by XGS_Is_On (Sat 26-Oct-24 18:16:50)