Very weird latency issue - Any ideas?

Hi guys,

Recently we have been having an issue on our connection whereby our latency increases around 10 times during the evening. It is probably being caused due to nodes giving high latency within BT Retails' network however there is a very peculiar problem; when pinging some IPs, the latency is fine despite going through the same nodes with a high ping.

Just to show you the latency issue, here is a TBB graph running on the BT Retail Termination Router's IP I am being routed through, not my connection:

http://www.thinkbroadband.com/ping/share/f093d99862b...

Here is a tracert from last night when pinging bbc.co.uk (This is the ping I typically get to most of the internet):

Tracing route to bbc.co.uk [212.58.253.67]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 9 ms 217.32.141.2
3 9 ms 8 ms 8 ms 217.32.140.222
4 13 ms 13 ms 13 ms 217.41.216.154
5 153 ms 151 ms 154 ms 31.55.164.69
6 14 ms 13 ms 15 ms 31.55.164.107
7 14 ms 14 ms 13 ms acc1-10GigE-0-0-0-5.bm.21cn-ipp.bt.net [109.159.
248.74]
8 22 ms 23 ms 23 ms core1-te0-15-0-15.ealing.ukcore.bt.net [109.159.
248.40]
9 18 ms 18 ms 17 ms acc1-10GigE-0-5-0-4.l-far.21cn-ipp.bt.net [109.1
59.254.104]
10 158 ms 159 ms 168 ms 194.74.65.42
11 * * * Request timed out.
12 20 ms 20 ms 20 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
13 21 ms 21 ms 21 ms 132.185.255.156
14 144 ms 149 ms 150 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Trace complete.

As you can see my ping to 217.32.141.2 (2nd hop) rules out anything wrong with my connection (Also the fact the TBB Meter gets a high ping to that IP also rules my connection out).

Now for the weird part:

Tracing route to www-vip.telhc.bbc.co.uk [212.58.251.195]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 9 ms 217.32.141.2
3 10 ms 9 ms 10 ms 217.32.140.222
4 14 ms 13 ms 12 ms 213.120.161.34
5 150 ms 157 ms 158 ms 31.55.164.69
6 14 ms 14 ms 14 ms 31.55.164.107
7 14 ms 13 ms 13 ms acc1-10GigE-0-5-0-5.bm.21cn-ipp.bt.net [109.159.
248.98]
8 22 ms 23 ms 23 ms core2-te-0-4-0-4.ilford.ukcore.bt.net [109.159.2
48.6]
9 194 ms 136 ms 139 ms peer2-xe3-0-1.telehouse.ukcore.bt.net [109.159.2
54.235]
10 162 ms 161 ms 159 ms 194.74.65.42
11 * * * Request timed out.
12 * * * Request timed out.
13 18 ms 18 ms 19 ms ae0.er01.telhc.bbc.co.uk [132.185.254.109]
14 20 ms 20 ms 20 ms 132.185.255.140
15 21 ms 20 ms 20 ms www-vip.telhc.bbc.co.uk [212.58.251.195]

Trace complete.

As you can see the ping to that BBC IP is going through the same nodes with the high latency (Takes a different route between hop 5 and 10) however the ping to the last hop is normal on this last tracert.

Can anyone think of what could be causing this? Surely if my connection goes through the same node on two tracerts, the latency after the node on both should be the same?

I've been in contact with BT over this but so far I'm having to rule out my connection as the problem. I'm just intrigued as to what the problem could be.

The high latency has been happening for quite a few months now and gaming is a nightmare.

Also I thought I might mention, my throughput seems fine despite this high latency issue happening.

Can anyone offer an explanation?

What's a reverse trace show?

https://mebsd.com/ipv6-ping-and-traceroute

Nick

You seem to be hitting 2 different servers at the bbc and one may prioritise ping responses higher than the other or may be less busy. The hop part way through is irrelevant as the responses are fine after it which just suggests it does not prioritise responding to pings.

Edit: to be clear there is no error your end, bt or at the bbc - it is just the priorities the boxes place on responding to a ping as pings are relatively unimportant and the servers/routers have more important things to do. Unless you are seeing actual problems don't get hung up about tracerts as they are only helpful in trying to diagnose real problems and only if read correctly.

Edited by ian72 (Tue 02-Apr-13 10:36:15)

In reply to a post by ian72:

You seem to be hitting 2 different servers at the bbc and one may prioritise ping responses higher than the other or may be less busy. The hop part way through is irrelevant as the responses are fine after it which just suggests it does not prioritise responding to pings.

If that is the case then, why does it affect the latency between TBB's Firebrick Quality Monitor and the 2nd hop as shown in the graph I linked in the first post?

I'll try a reverse tracert tonight.

Sorry, majored on the tracert rather than the graph

The graph looks either like you are doing some big downloads during the evening or that bt has some major contention in your area. Are the tracerts from the evening when you are seeing the high latency?

Ignore the 2 points in the trace as they are irrelevant as everything after them is fine so can't be causing the slow response. Ideally need a trace to the TBB servers that are doing the BQM.

Edited by ian72 (Tue 02-Apr-13 10:57:32)

Used to see the ping tsunami (hump) too often with Be/02 It looks like a congested link somewhere ,and because it is also showing on the TBBQM is isn't going to be the BBC probably within the BT network, or a peering transit link, Im suprised others aren't seeing similar, unless this was local to you

Edited by tommy45 (Tue 02-Apr-13 10:53:35)

It wouldn't matter if I was doing any downloads because that graph is monitoring the BT Termination Node (The 2nd hop from my connection), not my connection as I mentioned.

Yes those tracerts are from the evening. Here is one taken just now (11:08am):

Tracing route to www-vip.cwwtf.bbc.co.uk [212.58.253.67]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms edge1-r-0-0-1.evansnet.co.uk [10.5.14.254]
2 8 ms 8 ms 8 ms 217.32.141.2
3 8 ms 8 ms 8 ms 217.32.140.222
4 13 ms 13 ms 13 ms 217.41.216.154
5 13 ms 13 ms 13 ms 31.55.164.69
6 14 ms 13 ms 14 ms 31.55.164.107
7 13 ms 13 ms 13 ms acc1-10GigE-0-0-0-5.bm.21cn-ipp.bt.net [109.159.
248.74]
8 20 ms 23 ms 23 ms core1-te0-15-0-15.ealing.ukcore.bt.net [109.159.
248.40]
9 18 ms 18 ms 17 ms acc1-10GigE-0-5-0-4.l-far.21cn-ipp.bt.net [109.1
59.254.104]
10 19 ms 19 ms 19 ms 194.74.65.42
11 * * * Request timed out.
12 20 ms 19 ms 20 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
13 21 ms 20 ms 21 ms 132.185.255.156
14 20 ms 20 ms 20 ms www-vip.cwwtf.bbc.co.uk [212.58.253.67]

Trace complete.

What is/are the IP(s) for the TBB Servers that are doing the BQM (Firebrick appliances)?

OK, is there any reason to think it is anything other than prioritisation on the node? During the evenings it will be busier sending traffic all over and therefore would make sense for it to put responding to pings at the bottom of the list and therefore would go up. If you do a BQM graph to your actual router then is that ok? I think you are getting concerned over something that is most likely irrelevant to any real world performance.

I'm not getting concerned over something that's totally irrelevant.

Am I wrong in assuming it's not acceptable for a near idle 21CN connection to have a 160ms latency casually during the evening? Especially when before 3-4 months ago this problem wasn't occurring?

To put it into more perspective; the much higher latency we receive during the evenings is even higher than the latency we get if we use up all available bandwidth on our connection and then ping a website on a normal day. During the evenings, the performance of the connection when gaming (Not downloading) is worse than that of having the connection hammered.

When gaming and using real time applications, having an eight to tenfold increase in ping can really make a difference and we have all found that having this higher ping is very annoying in the evenings. It isn't just affecting the ICMP ping protocol either; the performance does show when using the connection for things like gaming and Skype.

No, using a BQM graph on my actual router results in the near exact same pattern as that graph I have already linked. I'll post my new connection BQM graph when the problem happens along with the other graph so you can see (I have a dynamic IP which has changed from the last update to my personal connections' graph).

What's the diff in circumstances between the 2 TRACERTs you posted (other than the IPs arrived at)?

Or are you saying these are 2 random TRACERTs taken about the same time?