In reply to a post by Banger:

Point is, is this missing segment lark normal or peculiar to my line. If my line is dropping segments is this an ISP problem or overly officious billion router.

I also see the same behaviour on my connection downloading large test files. I see a slight slow down during the dup-acks, and then a recovery. We are also seeing the same thing on both your routers (right? according to pcap files, but the symptoms are more noticeable on the Billion). So I say this is normal TCP behaviour. What might not be so normal is how the Billion is dealing with (or not dealing with) this TCP behaviour.

The network chap at work says that TCP has a mechanism which aims to double its transfer speed during the initial stages of the connection (and during it too). In other words, TCP is always trying to grab as much speed as possible during the transfer. The mechanism(s) it uses are based both on measuring packet delay between peers, the window size (sliding window position and all that malarkey), and also, dropped packets (missed segments). So, what is happening is the remote server you are downloading from is trying to push the speed up beyond your line rate, at which point, your side detects a missed segment (the modem either dropped this packet, or it never reached you from the ISP), and then coughs and splutters until things are re-negotiated and sorted out (until the next time this same event occurs). It seems the Billion is worse at dealing with this issue than the SpeedTouch - now the question is, why?

It could be that inside the Billion, their routing module (or what ever you want to call it) has a harsh policing policy that just drops packets if things get too fast, where as, possibly in the SpeedTouch, its routing module is a bit more intelligent and prefers to shape packets first (delaying them where possible) before harshly dropping them (policing). That's my guess. And it is purely a guess.

Good that we seem to have narrowed down the mystery a bit though.

Edited by deleted (Sat 11-Feb-12 12:38:31)

I see well that makes sense but brings us back to the billion being a bit harsh on dropped segments. The workaround for me on downloads is to use multiple streams eg 10 so one stream having a hissy fit is hardly noticeable, eg speed drop is a 10th of line speed. That's sorted.

Only problem still remaining is HD video streams which can be as high as 3.5 mbps and dont like the router stopping mid flow as this causes pauses and the never ending (seems like) circle. Need to think of a workaround as the alternative speedtouch with it's fondness of rebooting during video stream isn't an option.

You have a PM.

Cheers got it.

The plot thickens - bare with me.

I have held the same sync on my line since before Christmas. This was just after the changes BT made on 21CN connections to no longer follow stepped BRAS values, but instead, follow (in real-time) the precise line rate of the line. So back when that happened, I set my AA profile to be 99% of BT BRAS and did a re-sync, this showing up on the portal:

24 Dec 2011 10:35:34 Update Updated TxMargin=100->99
16 Dec 2011 20:10:26 Line rate now 12692000 -BT-

Great I thought. Well, now I've just forced another re-sync, obtained a slightly faster sync, and this has now shown up:

Today 14:49:08 Tx rate (adjusted) 0 to 11102483 -Auto-

So, I suspect initially, AA were not factoring in the ADSL overhead when placing a limiter their side on my broadband. If you do the maths on the above figures, 11102483 is 99% of 11214629.29... which is 87.9% of the actual sync speed reported by my modem, 12755 Kbps (87.9% ADSL over head is about right, yes? I've seen this figure being quoted around).

So the up shot of all this is that for the past 2 months, I have, on occasions, been sent data faster than what my line can handle. Meaning dropped packets, which confirms what I said yesterday when I tested the download files and noticed I was seeing dup-acks and slight periods of reduced download speed, recovering seconds later (much like yourself, but not as prominent). This was because of inbound dropped packets (probably dropped at the BT BRAS limiter, not AA).

I have just re-run the downloads and they are now absolutely full speed, as far as what is reported in the browser progress bar. I also took a pcap whilst doing this. I'm still seeing dup-acks, but when they do occur, they are for literally bursts lasting fractions of a second (4 ms to be precise) but if you compare that to the bursts you get on your Billion pcap, some last as long as 300 ms.

The drastic improvement for me at least is that now AA are applying their shaper their side. In theory, there is no bottle-neck downstream on my connection any more since the data sent should always be 99% of my capable line speed (factoring in ADSL over head) and also, more intelligent shaping from the AA shaper (rather than the dumb policer BT BRAS).

As for your setup, I can't comment (Vivaciti hold the answers). They should apply limits their side as it's the "proper" thing to be doing.

Anyway, it confirms, at least for me, that somewhere, some how, pretty harsh policing is happening on the traffic when you are using the Billion and nicer, smoother shaping is happening with the SpeedTouch.

Finally, with regards to your multiple TCP streams being "better", if you think about this logically, of course they will be. If you are downloading across 4 parallel TCP streams, the back log of data in each connection pipe* is going to be a quarter of what it normally would be at full speed. You will still get pauses on each of these 4 connections, but they will be less, and probably not for as long (and so probably not as noticeable).

* pipe - I like to use the term pipe here as this is essentially how to think of a TCP session. It is a pipe, with a certain cross-sectional area (your capable transmission speed) of a certain length (the time it takes to send / receive packets to / from your peer).

Edited by deleted (Sat 11-Feb-12 16:38:18)

Interesting. I still think the Billion is just too sensitive to overspeed if what u are saying is correct. Your right Vivaciti needs to let me know what, if any shaping they are using, will be Monday for that now. I do know there is some sort of shaping because when I first got connected ITV player was chronic as well as 4oD. They adjusted my profile and sorted it so was getting 2 mbps streaming on those Players, BBC was unaffected. I was using the speedtouch then so may not have noticed any problems on port 80.

Your explanation also fits with my fiddling in the router, forcing ADSL1 and 2 modes at lower line rate as well as applying QoS of 50% in the router. If this shaping takes place after the modem then it will still stall so that would fit.

I've also been attacking Wireshark trying to learn how to use it, finally converting timestamps into something I can understand. But while doing this I found two TCP Analyze Window Full packets in a capture of a single stream I did from the TBB 100mb file. Why would the window be full? Repeating the excersize on a multi-stream download, there was only one TCP Window Full. But on the first capture the timestamp of the first Window full roughly coincided with the pause. Some more googling needed.

The shaping issue (if Vivaciti are shaping) might possible explain why Billion can't reproduce the problem (maybe) - do you even know what ISP they use? Are they even based in the UK? Do they even use the same exchange kit? So many variables there, it's not surprising they can't reproduce it.

Definitely have a chat with Vivaciti for sure. Another related question - are the sync speeds between the SpeedTouch and the Billion substantially different? Does the Billion grab a faster sync? If so, how much faster?

Edited by deleted (Sat 11-Feb-12 16:43:55)

Yes the Speedtouch grabs around 17 meg and the Billion 19 meg. Have a feeling the Billion Engineers are in China but pretty sure Billion UK tested the box.

Been reading about Silly Window Syndrome and this would account for Window Full segments and pauses. Nagle's algorithm is supposed to get rid of SWS so if that wasn't implemented that would account for Full Window packets and pauses. But the curious thing is the Window full in the capture was sent by the Apache server to the Billion, so the Billion sits and twiddles it's thumbs until the server has emptied it's buffer as far as I can tell.

Upstream is 1.2mbps so wouldnt have thought that would overload a TBB server unless the Billion is being aggressive in sending ACK packets too fast. It's just odd how it's so random.

Will see what Viva say.

In reply to a post by Banger:

Yes the Speedtouch grabs around 17 meg and the Billion 19 meg. Have a feeling the Billion Engineers are in China but pretty sure Billion UK tested the box.

Been reading about Silly Window Syndrome and this would account for Window Full segments and pauses. Nagle's algorithm is supposed to get rid of SWS so if that wasn't implemented that would account for Full Window packets and pauses. But the curious thing is the Window full in the capture was sent by the Apache server to the Billion, so the Billion sits and twiddles it's thumbs until the server has emptied it's buffer as far as I can tell.

Upstream is 1.2mbps so wouldnt have thought that would overload a TBB server unless the Billion is being aggressive in sending ACK packets too fast. It's just odd how it's so random.

Will see what Viva say.

I'm still looking at the dump you posted - they are always a pain especially large ones and I am working with tcpdump = text only in linux. I do have tcptrace which makes graphs out of dumps and at first sight it looks like a policer or full buffer is causing multiple losses causing the backoffs. These tend to appear worse when viewed from an app as the data just appears to stop even though it is still coming down the line. When the missing packets get retransmitted a big chunk of data then gets passed to the app (because at application level tcp provides in order byte stream).

As to why it happens only with the billion - I have no idea. It may be something as simple as the higher sync making things different.

One dump can be misleading - I just tried some and they all looked different.

What is the Window full you refer to ( I am not using wireshark so haven't seen that).

Your Billion has nothing to do with tcp as such so doesn't send acks. It's windows that handles that - I notice you have set a > 1meg window, and windows just blindly uses it, so on a close server you would expect to fill any buffers and get loss. I accept this doesn't explain why only the Billion is affected.

Here's a screenshot of the tcptrace throughput graph - note the scale doesn't go down to 0. You can see that it is getting full speed then gets knocked back (by multiple loss) recovers and repeats. This is sort of normal, but in this case it seems there is too much loss/backoff. The large window you set probably doesn't help, but then you may need it for distant servers.

Edit forgot www on url
http://www.andyqos.ukfsn.org/tpt.png

Edited by deleted (Sat 11-Feb-12 21:39:49)

I can't say how AA do things, but IME with entanet on a BT line it's the ISP that polices and BT BRAS that buffers.

Policing if set up right can be acceptable - in someways nicer for latency. When on 20CN it seemed that bras buffers were about 700ms which meant I had to ingress shape.

They seem to be a bit shorter now, but I can still lag myself out if I try. I don't know if this is due to isp policer being closer to BT rate now that profiles are gone or whether the buffers really are smaller (as some would argue they should be - buffer bloat).