Very weird latency issue - Any ideas?

Hi guys,

Recently we have been having an issue on our connection whereby our latency increases around 10 times during the evening. It is probably being caused due to nodes giving high latency within BT Retails' network however there is a very peculiar problem; when pinging some IPs, the latency is fine despite going through the same nodes with a high ping.

Just to show you the latency issue, here is a TBB graph running on the BT Retail Termination Router's IP I am being routed through, not my connection:

http://www.thinkbroadband.com/ping/share/f093d99862b...

Here is a tracert from last night when pinging bbc.co.uk (This is the ping I typically get to most of the internet):

Tracing route to bbc.co.uk [212.58.253.67]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 9 ms 217.32.141.2
3 9 ms 8 ms 8 ms 217.32.140.222
4 13 ms 13 ms 13 ms 217.41.216.154
5 153 ms 151 ms 154 ms 31.55.164.69
6 14 ms 13 ms 15 ms 31.55.164.107
7 14 ms 14 ms 13 ms acc1-10GigE-0-0-0-5.bm.21cn-ipp.bt.net [109.159.
248.74]
8 22 ms 23 ms 23 ms core1-te0-15-0-15.ealing.ukcore.bt.net [109.159.
248.40]
9 18 ms 18 ms 17 ms acc1-10GigE-0-5-0-4.l-far.21cn-ipp.bt.net [109.1
59.254.104]
10 158 ms 159 ms 168 ms 194.74.65.42
11 * * * Request timed out.
12 20 ms 20 ms 20 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
13 21 ms 21 ms 21 ms 132.185.255.156
14 144 ms 149 ms 150 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Trace complete.

As you can see my ping to 217.32.141.2 (2nd hop) rules out anything wrong with my connection (Also the fact the TBB Meter gets a high ping to that IP also rules my connection out).

Now for the weird part:

Tracing route to www-vip.telhc.bbc.co.uk [212.58.251.195]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 9 ms 217.32.141.2
3 10 ms 9 ms 10 ms 217.32.140.222
4 14 ms 13 ms 12 ms 213.120.161.34
5 150 ms 157 ms 158 ms 31.55.164.69
6 14 ms 14 ms 14 ms 31.55.164.107
7 14 ms 13 ms 13 ms acc1-10GigE-0-5-0-5.bm.21cn-ipp.bt.net [109.159.
248.98]
8 22 ms 23 ms 23 ms core2-te-0-4-0-4.ilford.ukcore.bt.net [109.159.2
48.6]
9 194 ms 136 ms 139 ms peer2-xe3-0-1.telehouse.ukcore.bt.net [109.159.2
54.235]
10 162 ms 161 ms 159 ms 194.74.65.42
11 * * * Request timed out.
12 * * * Request timed out.
13 18 ms 18 ms 19 ms ae0.er01.telhc.bbc.co.uk [132.185.254.109]
14 20 ms 20 ms 20 ms 132.185.255.140
15 21 ms 20 ms 20 ms www-vip.telhc.bbc.co.uk [212.58.251.195]

Trace complete.

As you can see the ping to that BBC IP is going through the same nodes with the high latency (Takes a different route between hop 5 and 10) however the ping to the last hop is normal on this last tracert.

Can anyone think of what could be causing this? Surely if my connection goes through the same node on two tracerts, the latency after the node on both should be the same?

I've been in contact with BT over this but so far I'm having to rule out my connection as the problem. I'm just intrigued as to what the problem could be.

The high latency has been happening for quite a few months now and gaming is a nightmare.

Also I thought I might mention, my throughput seems fine despite this high latency issue happening.

Can anyone offer an explanation?

What's a reverse trace show?

https://mebsd.com/ipv6-ping-and-traceroute

Nick

You seem to be hitting 2 different servers at the bbc and one may prioritise ping responses higher than the other or may be less busy. The hop part way through is irrelevant as the responses are fine after it which just suggests it does not prioritise responding to pings.

Edit: to be clear there is no error your end, bt or at the bbc - it is just the priorities the boxes place on responding to a ping as pings are relatively unimportant and the servers/routers have more important things to do. Unless you are seeing actual problems don't get hung up about tracerts as they are only helpful in trying to diagnose real problems and only if read correctly.

Edited by ian72 (Tue 02-Apr-13 10:36:15)

In reply to a post by ian72:

You seem to be hitting 2 different servers at the bbc and one may prioritise ping responses higher than the other or may be less busy. The hop part way through is irrelevant as the responses are fine after it which just suggests it does not prioritise responding to pings.

If that is the case then, why does it affect the latency between TBB's Firebrick Quality Monitor and the 2nd hop as shown in the graph I linked in the first post?

I'll try a reverse tracert tonight.

Sorry, majored on the tracert rather than the graph

The graph looks either like you are doing some big downloads during the evening or that bt has some major contention in your area. Are the tracerts from the evening when you are seeing the high latency?

Ignore the 2 points in the trace as they are irrelevant as everything after them is fine so can't be causing the slow response. Ideally need a trace to the TBB servers that are doing the BQM.

Edited by ian72 (Tue 02-Apr-13 10:57:32)

Used to see the ping tsunami (hump) too often with Be/02 It looks like a congested link somewhere ,and because it is also showing on the TBBQM is isn't going to be the BBC probably within the BT network, or a peering transit link, Im suprised others aren't seeing similar, unless this was local to you

Edited by tommy45 (Tue 02-Apr-13 10:53:35)

It wouldn't matter if I was doing any downloads because that graph is monitoring the BT Termination Node (The 2nd hop from my connection), not my connection as I mentioned.

Yes those tracerts are from the evening. Here is one taken just now (11:08am):

Tracing route to www-vip.cwwtf.bbc.co.uk [212.58.253.67]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 8 ms 217.32.141.2
3 8 ms 8 ms 8 ms 217.32.140.222
4 13 ms 13 ms 13 ms 217.41.216.154
5 13 ms 13 ms 13 ms 31.55.164.69
6 14 ms 13 ms 14 ms 31.55.164.107
7 13 ms 13 ms 13 ms acc1-10GigE-0-0-0-5.bm.21cn-ipp.bt.net [109.159.
248.74]
8 20 ms 23 ms 23 ms core1-te0-15-0-15.ealing.ukcore.bt.net [109.159.
248.40]
9 18 ms 18 ms 17 ms acc1-10GigE-0-5-0-4.l-far.21cn-ipp.bt.net [109.1
59.254.104]
10 19 ms 19 ms 19 ms 194.74.65.42
11 * * * Request timed out.
12 20 ms 19 ms 20 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
13 21 ms 20 ms 21 ms 132.185.255.156
14 20 ms 20 ms 20 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Trace complete.

What is/are the IP(s) for the TBB Servers that are doing the BQM (Firebrick appliances)?

OK, is there any reason to think it is anything other than prioritisation on the node? During the evenings it will be busier sending traffic all over and therefore would make sense for it to put responding to pings at the bottom of the list and therefore would go up. If you do a BQM graph to your actual router then is that ok? I think you are getting concerned over something that is most likely irrelevant to any real world performance.

I'm not getting concerned over something that's totally irrelevant.

Am I wrong in assuming it's not acceptable for a near idle 21CN connection to have a 160ms latency casually during the evening? Especially when before 3-4 months ago this problem wasn't occurring?

To put it into more perspective; the much higher latency we receive during the evenings is even higher than the latency we get if we use up all available bandwidth on our connection and then ping a website on a normal day. During the evenings, the performance of the connection when gaming (Not downloading) is worse than that of having the connection hammered.

When gaming and using real time applications, having an eight to tenfold increase in ping can really make a difference and we have all found that having this higher ping is very annoying in the evenings. It isn't just affecting the ICMP ping protocol either; the performance does show when using the connection for things like gaming and Skype.

No, using a BQM graph on my actual router results in the near exact same pattern as that graph I have already linked. I'll post my new connection BQM graph when the problem happens along with the other graph so you can see (I have a dynamic IP which has changed from the last update to my personal connections' graph).

What's the diff in circumstances between the 2 TRACERTs you posted (other than the IPs arrived at)?

Or are you saying these are 2 random TRACERTs taken about the same time?

BT network congestion. You're not alone.

In reply to a post by XRaySpeX:

What's the diff in circumstances between the 2 TRACERTs you posted (other than the IPs arrived at)?

Or are you saying these are 2 random TRACERTs taken about the same time?

The two tracerts in my first post were done at the same time to different IPs (Both belonging to bbc.co.uk).

Edited by chris6273 (Wed 03-Apr-13 05:28:58)

In reply to a post by chris6273:

I'll try a reverse tracert tonight.

Did you do one - if so, what did that show?

Nick

In reply to a post by chris6273:

The two tracerts in my first post were done at the same time to different IPs (Both belonging to bbc.co.uk).

If you mean you predetermined the IPs to trace, rather letting the DNS decide the IP from the domain name, then those 2 TRACERTs don't prove anything. The faster could well be the duty server at that time for BBC and the slower one just a standby server. Their only diff was their time to respond.

In reply to a post by XRaySpeX:

In reply to a post by chris6273:

The two tracerts in my first post were done at the same time to different IPs (Both belonging to bbc.co.uk).

If you mean you predetermined the IPs to trace, rather letting the DNS decide the IP from the domain name, then those 2 TRACERTs don't prove anything. The faster could well be the duty server at that time for BBC and the slower one just a standby server. Their only diff was their time to respond.

It is a mixture of both. For the first post I had to manually specify the faster (Lower latency) one however the slower, higher latency IP was resolved from bbc.co.uk.

Even if they don't prove anything the problem is definitely still there according to BT's own asset. Here is a BT Speedtest showing my speed and latency during the problematic evening which every BT Customer is advised to use instead of speedtest.net:

http://imageshack.us/a/img22/3250/speedtest293.png

Does that help narrow things down? It can't be anything to do with the BBC's servers if BTs' Test Server is giving a high latency reading.

I'll do a reverse tracert a bit later when the problem is at its usual worst.

Edited by chris6273 (Wed 03-Apr-13 20:00:27)

In reply to a post by Lethe:

In reply to a post by chris6273:

I'll try a reverse tracert tonight.

Did you do one - if so, what did that show?

Nick

Sorry, I totally forgot to do one last night.

Here is one I have just done:

1 [AS8468] gi3-4.telford-dc1.core.enta.net (78.33.39.217) 0.372 ms 0.359 ms 0.199 ms
2 [AS8468] te4-4.wolv-te.core.enta.net (87.127.236.18) 0.822 ms 0.826 ms 0.666 ms
3 [AS8468] te4-4.stoke.core.enta.net (87.127.236.190) 1.447 ms 1.447 ms 1.292 ms
4 [AS8468] te1-6.man.core.enta.net (87.127.236.193) 2.852 ms 2.854 ms 2.695 ms
5 [AS8468] te5-1.telehouse-north0.core.enta.net (87.127.236.109) 9.560 ms 9.564 ms 9.559 ms
6 [AS2856] 194.74.65.57 (194.74.65.57) 13.777 ms 10.510 ms 11.751 ms
7 [AS2856] core2-te0-3-0-7.ilford.ukcore.bt.net (109.159.254.158) 17.818 ms 17.060 ms 16.115 ms
8 [AS2856] acc1-10GigE-0-7-0-0.bm.21cn-ipp.bt.net (109.159.248.3) 15.331 ms 15.335 ms 15.333 ms
9 [AS2856] 109.159.248.113 (109.159.248.113) 15.954 ms
[AS2856] 109.159.248.103 (109.159.248.103) 15.490 ms
[AS2856] 109.159.248.101 (109.159.248.101) 15.178 ms
10 [AS2856] 31.55.164.106 (31.55.164.106) 15.807 ms 15.800 ms 15.644 ms
11 [AS2856] 31.55.164.72 (31.55.164.72) 15.799 ms 15.804 ms 15.643 ms
12 [AS2856] 217.41.216.233 (217.41.216.233) 132.372 ms 136.870 ms
[AS2856] 217.41.216.153 (217.41.216.153) 19.862 ms
13 * * *

Looks like the hop in bold is the one giving a high latency figure in a reverse tracert. Does this help?

Those high figures in the intermediate nodes mean very little. In every case the response from the next node is "normal" which suggest that the nodes you have highlighted are moving data on and not holding it up in both directions.

In reply to a post by chris6273:

I'm not getting concerned over something that's totally irrelevant.

With traceroute you need to consider what you're measuring and what that means. Generally, traceroute results are unreliable to use for end to end latency measurement because they are greatly dependant on the response times of the supervisor software in the routers.

Routers are generally optimised to transmit packets through them and can be relativity slow to process errors such as packets exceeding their hop count. traceroute works by sending packets with an increasing hop count and getting an error response back from each router in turn as the router discovers the packet it receives has exceeded its hop count. The time taken by the router to do this is not a good measure of latency of packets travelling through the router.

In your traceroute output

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 9 ms 217.32.141.2
3 9 ms 8 ms 8 ms 217.32.140.222
4 13 ms 13 ms 13 ms 217.41.216.154
5 153 ms 151 ms 154 ms 31.55.164.69
6 14 ms 13 ms 15 ms 31.55.164.107
7 14 ms 14 ms 13 ms acc1-10GigE-0-0-0-5.bm.21cn-ipp.bt.net [109.159.
248.74]
8 22 ms 23 ms 23 ms core1-te0-15-0-15.ealing.ukcore.bt.net [109.159.
248.40]
9 18 ms 18 ms 17 ms acc1-10GigE-0-5-0-4.l-far.21cn-ipp.bt.net [109.1
59.254.104]
10 158 ms 159 ms 168 ms 194.74.65.42
11 * * * Request timed out.
12 20 ms 20 ms 20 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
13 21 ms 21 ms 21 ms 132.185.255.156
14 144 ms 149 ms 150 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Hop 5 took 153ms but hop 6 took only 14ms. Time cannot have gone backwards by 139ms between hop 5 and 6 so something else must be going on. Chances are the supervisor software in hop 5 was in no hurry to respond to the traceroute packet reachnig its maximum hop count.

You need to measure the end to end latency using ping and see how that varies.

In reply to a post by chris6273:

I'm not getting concerned over something that's totally irrelevant.

You are confused by the significance of TRACERT results. Its primary purpose is to show the route taken to the final target. Only the final timing is of interest. If you are only interested in the timing then PING is sufficient.