Massive Latency problems

Target Name: www.bbc.co.uk
IP: 212.58.246.91
Date/Time: 03/06/2013 20:29:47 to 03/06/2013 20:29:56

1 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms home.gateway.home.gateway [192.168.1.254]
2 25 ms 131 ms 29 ms 292 ms 51 ms 16 ms 26 ms 18 ms 24 ms 18 ms lo0-central10.ptn-ag01.plus.net [195.166.128.190]
3 16 ms 15 ms 16 ms 16 ms 15 ms 31 ms 16 ms 15 ms 15 ms 16 ms link2-central10.ptn-gw02.plus.net [84.93.248.130]
4 46 ms 15 ms 15 ms 15 ms 15 ms 18 ms 15 ms 15 ms 18 ms 15 ms xe-1-2-0.ptw-cr02.plus.net [212.159.1.30]
5 16 ms 15 ms 16 ms 34 ms 19 ms 19 ms 15 ms 15 ms 15 ms 16 ms ae2.ptw-cr01.plus.net [195.166.129.4]
6 15 ms 16 ms 16 ms 16 ms 15 ms 15 ms 15 ms 15 ms 15 ms N/A kingston-gw.thdo.bbc.co.uk [212.58.239.6]
7 * * * * * * * N/A N/A N/A [-]
8 16 ms 17 ms 58 ms 53 ms 16 ms 24 ms 17 ms 16 ms 27 ms N/A ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
9 17 ms 27 ms 28 ms 17 ms 18 ms 16 ms 18 ms 18 ms 16 ms N/A [132.185.255.164]
10 16 ms 16 ms 16 ms 16 ms 16 ms 16 ms 16 ms 15 ms 16 ms N/A bbc-vip012.cwwtf.bbc.co.uk [212.58.246.91]

Ping statistics for www.bbc.co.uk
Packets: Sent = 10, Received = 10, Lost = 0 (0.0%)
Round Trip Times: Minimum = 15ms, Maximum = 16ms, Average = 15ms


lo0-central10.ptn-ag01.plus.net hitting 500ms

Edited by Stanman_24 (Mon 03-Jun-13 20:31:28)

ahhh good.

Plusnet still have the overloaded pipe problem then where they have to disconnect users and let them reconnect on less congested connections I assume?

If the pipe was overloaded/congested - in terms of throughput to/from other places - then I'd expect to see some increased ping latencies from all other places, not just PN's own edge router.

PN themselves have said that ICMP packets are treated as titanium level traffic in their traffic management nodes. I'd expect them to transit through the routers on the fastest possible speeds.

However, an ICMP ping packet targeted *at* one of the edge routers is something different - it needs to be processed and answered from within the router, rather than being transited by the router. A subtle difference.

So a delay in a packet that was targeted at the edge router might indicate that it has other processing going on, independent of "just" transit routing, and the ICMP response was delayed for those reasons.

I'd be more concerned if packets that transit the edge routers were being delayed.

Agreed.

It is only if the step at a node is then continued to most or all of the later nodes that there is a problem.

If the particular node is (mostly) the only one with a high value that simply means it has better things to do, i.e. passing genuine traffic through, than respond to somebody pinging it. Which is why many are configured not to respond at all - the ones that time out.

Edit:-

Ping statistics for www.bbc.co.uk
Packets: Sent = 10, Received = 10, Lost = 0 (0.0%)
Round Trip Times: Minimum = 15ms, Maximum = 16ms, Average = 15ms

Looks pretty good to me.

Edited by RobertoS (Mon 03-Jun-13 22:37:10)

look at my bqm

That's more meaningful . Have you tried a router disconnect t see if you get a different gateway? Also, what is being done on your connection?

Which product are you one? A "new" unlimited, or a legacy one?

unlimited

you will see i did a resync as my latency dropped about 3 ms

hopefully nothing major

I did wonder where that drop in the base latency had come from .

On FTTC re-sync's can be so quick as not to drop the session at the ISP. If I force one I always make sure I break the connection for at least 20 seconds before starting the re-sync.

There is another reason for my doing this this as well. The PlusNet Current line speed doesn't get updated if the session isn't dropped, so the "new" BT IP Profile doesn't get reflected into it. That can cause issues on below 80 (or 40) Mbps connections.

A quick scan of the plusnet forums Icame accross this TBBQM in a related thread , looks like congestion setting in somewhere Not looking good
And was Plusnet Gateway ptw-ag04

Edited by tommy45 (Tue 04-Jun-13 01:58:45)

From what I can see, ptn-ag01 and ptw-ag04 were looking fine last night. No significant latency or dropped packets on our monitoring.

I'll have a bit of a dig and see if I can see anything further.

We did add another 4.5Gb/s of capacity to WBC yesterday - though the balance is usually pretty good these days.

tonight my connection has been good as gold

very happy despite yesterday's hiccup

doing it again tonight

Target Name: www.bbc.co.uk
IP: 212.58.246.92
Date/Time: 08/06/2013 23:00:50 to 08/06/2013 23:00:59

1 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms 3 ms home.gateway.home.gateway [192.168.1.254]
2 117 ms 94 ms * * 222 ms 440 ms 183 ms N/A 78 ms N/A lo0-central10.ptn-ag01.plus.net [195.166.128.190]
3 21 ms * 21 ms 19 ms 19 ms 19 ms 20 ms N/A 20 ms 20 ms link11-central10.ptn-gw01.plus.net [84.93.248.148]
4 21 ms * 19 ms 20 ms 19 ms 20 ms 20 ms N/A 19 ms 20 ms xe-1-2-0.ptw-cr01.plus.net [212.159.1.28]
5 19 ms 19 ms 19 ms 20 ms 19 ms 19 ms 18 ms N/A 20 ms 20 ms kingston-gw.thdo.bbc.co.uk [212.58.239.6]
6 * * * * * * * N/A N/A N/A [-]
7 20 ms 21 ms * 21 ms * 21 ms 21 ms 21 ms 21 ms 21 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
8 21 ms 22 ms 21 ms 21 ms 25 ms 21 ms 24 ms 23 ms N/A 23 ms [132.185.255.164]
9 21 ms * 21 ms 20 ms * 19 ms 20 ms N/A 20 ms 20 ms bbc-vip013.cwwtf.bbc.co.uk [212.58.246.92]


having to resync to sort the problem out, i was using bbc iplayer as well
Ping statistics for www.bbc.co.uk
Packets: Sent = 9, Received = 7, Lost = 2 (22.2%)
Round Trip Times: Minimum = 19ms, Maximum = 21ms, Average = 20ms

Edited by Stanman_24 (Sat 08-Jun-13 23:06:14)

Hi Stan,

I also noticed something similar when doing the same route:

Tracing route to bbc.co.uk [212.58.253.67]
over a maximum of 30 hops:

1 10 ms 99 ms 99 ms 192.168.1.254
2 236 ms 168 ms 127 ms lo0-central10.ptn-ag01.plus.net [195.166.128.190
]
3 15 ms 15 ms 14 ms link10-central10.ptn-gw02.plus.net [84.93.248.14
6]
4 14 ms 14 ms 14 ms xe-1-2-0.ptw-cr02.plus.net [212.159.1.30]
5 15 ms 14 ms 14 ms ae2.ptw-cr01.plus.net [195.166.129.4]
6 15 ms 15 ms 14 ms kingston-gw.thdo.bbc.co.uk [212.58.239.6]
7 * * * Request timed out.
8 16 ms 15 ms 15 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
9 15 ms 15 ms 15 ms 132.185.255.156
10 16 ms 16 ms 16 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Trace complete.

Did a quick second test and slightly worse:

241 ms 273 ms 225 ms lo0-central10.ptn-ag01.plus.net [195.166.128.190]

Edited by deleted (Sun 09-Jun-13 02:56:08)

Don't worry about the response time of that gateway. The fact that the routers beyond it report in a fast time shows that the delay is non impacting.

Edited by deleted (Mon 10-Jun-13 09:09:41)

In reply to a post by Stanman_24:

doing it again tonight

having to resync to sort the problem out, i was using bbc iplayer as well
Ping statistics for www.bbc.co.uk
Packets: Sent = 9, Received = 7, Lost = 2 (22.2%)
Round Trip Times: Minimum = 19ms, Maximum = 21ms, Average = 20ms

I'd ignore the traceroute times unless there's a big jump in all times after a certain hop. Routers are often much better at routing packets through them than responding to traceroute requests.

I'd be much more worried by the 22.2% packet loss to the BBC. That's a real problem! TCP links should not really drop many packets at all. My 80/20 FTTC ink typically does not lose any packets in normal use.

In reply to a post by bsdnazz:

I'd be much more worried by the 22.2% packet loss to the BBC. That's a real problem! TCP links should not really drop many packets at all. My 80/20 FTTC ink typically does not lose any packets in normal use.

What's a TCP link please, Sir?

The bit that goes with IP - TCP/IP.

In reply to a post by RobertoS:

The bit that goes with IP - TCP/IP.

I'm aware of what TCP is, I wanted to know what a TCP link was.

look at my bqm

pings are in the low 40's to bbc.co.uk

[censored] is going on !

Call it late night wording on my part. Think connection.

The TCP protocol does not work well with more than a little packet loss.

TCP will actively detect droped packets as part of its congestion control mechanism. Losing the odd packet (I guess something like 1 in 1000 to 1 in 100) is generally not a problem but 22% packet loss will cripple a connection.

The OP, rather than worrying about slow ICMP responses by routers when using traceroute should be much more worried by 22% packet loss when pinging the BBC.

Edited by bsdnazz (Tue 11-Jun-13 12:47:08)

You need to be careful trying to put a percentage on such a small sample. It isn't particularly representative. Your statement was based on 2 packets missing.

It is fairly straightforward to cause at least 1 missing packet manually. All you have to do is hit ctrl-c after a packet has been sent out, but before the answer has been received. On 200ms response times, that would have a 1-in-5 chance, or 20%. On 20ms response times it would be a 1-in-50 chance, or 2%.

And packet loss when pinging an innocent third-party, who might be deliberately choosing to drop packets anyway, isn't the best way to show packet loss over your own link. The BQM does a much better job of informing you about that, because you are in control of both the link and the ping target (your own router).

Stanman's BQM isn't showing any packet loss; I don't see even 1% during the evening peak time. I don't think packet loss is an aspect that ought to worry him

Comparing his graph to the ones published by Plusnet for a group of users, it seems to have been quite a busy night over there. (You'll have to compare soon, as all the graphs are live, and the peaks are gradually disappearing. Robertos - yours looks bad, given that even the minimum latency was raised!)

But you're right that 22% packet loss is likely to be crippling to TCP throughput.

TCP might change the number of packets in flight as part of congestion control, but I've never heard of it actively dropping a packet. Given that its job is to ensure everything is received, that would be plain daft.

Yep - not a great night for graphs, mine included!

http://www.thinkbroadband.com/ping/share/152e8cdd67a...

In reply to a post by WWWombat:

TCP might change the number of packets in flight as part of congestion control, but I've never heard of it actively dropping a packet.

I guess I'm not much better in the morning either! I've edited my post to change 'actively drop' to 'detect dropped'.

Your point about a 9 packet test being too small a sample size is a good one. I usually send 100 packets (ping -c 100 XXX or ping -n 100 XXX depending on OS).

Thanks for the heads-up. It does look rather worse than usual, considering I never do anything to cause even the normal variation at that time of day.