How often are you maxing your fibre, and how?

This thread is born out of some curiosity. I noted a 4k stream buffering quite badly a couple of days back. A check on my 62-ish megabit connection showed it was saturated. It remained saturated until I stopped streaming.

I have a router that produces various statistics and confirmed what was going on. There were 3 devices streaming, 1 browsing and an XBoxOne downloading. There were 4 active users. The XBOne is the heaviest device for usage, its owner streams and downloads demos and games a fair bit.

Usage is heavily skewed towards peak period. The connection is consuming around 750GB downstream a month, up from less than 500 last year, so the CAGR on my connection isn't exceptional and it will probably be over the 1TB a month mark next year.

Is it just me that is finding my above-average FTTC insufficient at times?

I have noticed exactly the same issue.

I have BT TV and it was buffering a lot the other night. Granted I was watching a recording and recording 2 more in background but none were 4k channels. At the same time, the wife was on the net doing some work and I was on BBC News.

I sync at about 64Mbps so wouldn't expect the above to be saturating my connection so much but it did seem to be. Haven't had the time to do any more in-depth analysis but I will try to this weekend.

That's easily managed by having the xbox user set a load of games to download before going to bed?

Even 3 Netflix 4K streams should only utilise about 45-50mbps.

Perhaps you should use QOS to prioritise outgoing ACKs and maybe implement FQ-codel

It's kinda the reason why I have my own connection. Before when I wanted to download I would make the wife's IPTV stream buffer. So I agree that it's getting more restrictive as streaming quality increases and the speeds do not.

In reply to a post by BatBoy:

Perhaps you should use QOS to prioritise outgoing ACKs and maybe implement FQ-codel

As it was downstream congestion prioritising acknowledgements wouldn't really have helped, that only fixes upstream.

Changing queuing strategies wouldn't have really been helpful as all the significant traffic was single flow TCP and time insensitive, removing buffer bloat wouldn't have done much. My stream was getting a fair share of the downstream pipe, there were just too many other demands on it for that fair share to be adequate for my application.

My only real option would've been to either prioritise my own devices above others or to use something layer 7 aware that prioritises traffic from YouTube, etc, above other traffic or recognises XBoxOne downloads and deprioritises them. From the point of view of TCP they would all look pretty similar.

In reply to a post by lee111s:

That's easily managed by having the xbox user set a load of games to download before going to bed?

Even 3 Netflix 4K streams should only utilise about 45-50mbps.

I can manage it easily enough but the question was more about if I were somehow unusual in being on a 'superfast' connection and having to manage it.

TL;DR I shouldn't have to manage it.

The stream was a YouTube 4k stream, a busy one, so was averaging well over the sustained 15-20Mb that has been mentioned.

So you definitely don't use QOS or FQ_Codel? I wonder why you're having buffering problems then.

In reply to a post by BatBoy:

So you definitely don't use QOS or FQ_Codel? I wonder why you're having buffering problems then.

Not sure I understand. How would upstream QoS or a mechanism for combating buffer bloat and prioritising small packets help me get more bandwidth downstream when competing with similar flows? The flows were all TCP, all long lived, so would've all ended up in the same queue without some additional configuration.

The only thing that would work as far as I'm aware is prioritising either streaming video or my own machine.

I'd say prioritise video over downloads for sure

We have a line which sync at full speed 80/20 and in our experience two or three teenagers gaming on console or PC can quite easily max it out. Steam game update downloads use a huge amount of bandwidth and seem to be released fairly frequently. Couple that with one or two streaming TV in the evenings and it is easy to max out bandwidth.

I've found that the Amazon Fire TV will use huge amounts of bandwidth if you let it (I dont and limited it at the router). Netflix seems to be far less data hungry.

In reply to a post by lee111s:

I'd say prioritise video over downloads for sure

It wasn't super-effective last time. Will look into it.

The 'advanced' QoS on the router seems to use some DPI secret sauce from Trend Micro rather than using iptables to mark and something else to police.

admin@RT-AC87U:/tmp# ip route show table 0 | grep mark
admin@RT-AC87U:/tmp#

https://github.com/RMerl/asuswrt-merlin/blob/master/...
https://github.com/RMerl/asuswrt-merlin/blob/master/...

The definitions haven't been updated in ages.

I don't think you'll see much with ip route.

"prioritise video over downloads " easier said than done - but then still possible if you have full control I guess.

Shaping from the wrong end of the bottleneck is not ideal. You have to set the rate 5-10% lower than line rate just to get anything to work.

Plusnet in theory would prio video over bulk, but on their new network QOS seems to be disabled.

Even when it did work, xbox was a bit special, in that game downloads/updates still got prio = the only time my BQM on old network showed spikes/blocks of yellow/blue was if the line was being maxed by the xboxes.

In reply to a post by Ignitionnet:

The only thing that would work as far as I'm aware is prioritising either streaming video or my own machine.

I agree ... but, as the downstream congestion happens in, or before, the DSLAM, surely the prioritisation has to happen there. Whatever place has to queue or discard packets - your ISP, if they limit throughput to the IP profile (as BT wants them to), or the DSLAM.

I'm therefore not sure how it could identify streams to your machine as being the ones in need of priority.

This would then work provided your streaming uses adaptive bitrates when packets get dropped - which, I guess, iPlayer, Netflix, NowTV do. Does that happen to YouTube 4k streams?

Plusnet limits throughput, and applies prioritisation to streaming. I haven't seen any problems with streaming with them ... but we probably aren't close to filling our 80/20 line.

In reply to a post by WWWombat:

I agree ... but, as the downstream congestion happens in, or before, the DSLAM, surely the prioritisation has to happen there. Whatever place has to queue or discard packets - your ISP, if they limit throughput to the IP profile (as BT wants them to), or the DSLAM.

I'm therefore not sure how it could identify streams to your machine as being the ones in need of priority.

This would then work provided your streaming uses adaptive bitrates when packets get dropped - which, I guess, iPlayer, Netflix, NowTV do. Does that happen to YouTube 4k streams?

Plusnet limits throughput, and applies prioritisation to streaming. I haven't seen any problems with streaming with them ... but we probably aren't close to filling our 80/20 line.

Not really. Drop TCP packets at the router's WAN port it'll cause TCP congestion control to kick in and reduce throughput on those flows. In the case of UDP of course there's not much that can be done but as streaming now is pretty much all TCP it's fine.

My router is Linux based and a 'standard' technique is to use IP tables to mark traffic then police it with tc.

Inbound shaping is, of course, never going to be as good as outbound shaping, which is part of the reason I don't bother as a matter of routine. When I tried it previously as mentioned it didn't really work so well so I just switched it off.

If the XBox traffic is considered to be gaming by the DPI engine in the router even when it's actually downloads that changes the game somewhat. Either way I'm not obsessing too much about shaping, that was brought up by another poster, I was just curious as to how many people are bumping up against their broadband's capacity.

If I really, really wanted to control everything I would of course use SD-WAN, VPN or some other tunnel and control traffic outbound from the remote end towards my home. I have an SD-WAN appliance that I can use to control traffic that has far more buttons to press and knobs to twiddle than the Asus if it comes to it, I use it to connect to work but can certainly place it inline on my connection and use its QoS.

Birthday of XBox One owner = broadband struggling. With games at 50GB+ it would be ideal if it weren't faster to order them from Prime Now than to download them.

http://www.thinkbroadband.com/ping/share/c8a87c2339b...

In reply to a post by legume:

Plusnet in theory would prio video over bulk, but on their new network QOS seems to be disabled.

FWIW Plusnet QOS for me is back. They seem to have changed it a bit, in that the TOS/DSCP marks are different.

There seems to be 3 classes, game/voip/ping top, then unmarked, which seems to be everything else apart from p2p which is marked and I assume lowest prio. So the statement above about video over bulk won't really apply to normal downloads anymore.

your usage seems massive compared to mine.

In terms of when my line is saturated?

If I stream youtube or netflix it can momentarily hit high utilisation for short bursts, but it never reaches 100% and its not sustained, aside from the start to build up a buffer.

What I consider true line saturation is steam downloads (and battle.net but this is very infrequent compared to steam), as someone else also pointed this out on the pfsense forums, the way steam downloads work is abusive, typically the downloads come from a server in the same region so very low rtt, and yet steam feel the need to have in excess of 30 download threads, on a line which has no QoS, the affect is significant, lots of packet loss on non steam traffic and probably a increase of latency well above 50ms from a base of under 10ms. The latency I could live with during the downloads but not the packet loss. Steam does not allow you to reduce the tcp thread count. I found if I throttle within the steam app to e.g. 50% speed, it doesnt stay at 50% but just bursts to full line rate, rests, bursts again to full line rate and so on, this will still cause small amounts of packet loss. When I tried to mitigate it via traffic shaping on the router it worked ok'ish but not great on all shapers I tried, until I tried HSFC on pfsense, that one can actually manage steam traffic quite well.

Steam downloads is probably the hardest thing I have had to shape on my home connection, its most definitely worse than torrents. Although I dont really torrent at home, but when I have done its far less abusive of the TCP protocol than steam.

HSFC can get rid of the vast majority of packet loss during downloads from steam with it only limiting to about 90% of line rate which is very impressive compared to all other shapers I have tried. Latency tends to vary between about base of 9ms and about 20ms with HSFC during steam downloads also. Pretty much any other download activity is handled without a sweat by HSFC. I find codel + fairq best for managing upstream traffic tho which is far easier to shape.

For the month of February I used 203 gig traffic. My own usage is definitely going up but I am nowhere near the TB range.

I also think I could happily get by on a 40/10 package, and probably only have 80/20 due to the geek in me.

Edited by Chrysalis (Tue 21-Mar-17 15:46:23)

I suspect my personal usage isn't that different from yours but the larger household makes the difference.

Concurrency is the killer app.