FTTC providers w/ sensible routing that don't touch traffic

Hi all,

I recently jumped on the FTTC bandwagon signing up to BT a couple of months ago. While they've been OK, I'm not entirely happy with the throttling side of things and particularly unhappy with routing. I'm in south-west London and to get to other London servers I'm being routed via Sheffield - here's part of a traceroute to my IP:

11 1 ms 1 ms 1 ms 166-49-168-102.eu.bt.net [166.49.168.102]
12 12 ms 11 ms 11 ms core1-te-0-4-0-14.ilford.ukcore.bt.net [62.172.1
13 6 ms 6 ms 6 ms acc2-10GigE-10-0-0.sf.21cn-ipp.bt.net [109.159.2
14 9 ms 9 ms 9 ms 109.159.251.228
15 9 ms 9 ms 9 ms 217.41.169.108
16 10 ms 9 ms 9 ms 217.41.169.198
17 13 ms 13 ms 13 ms 213.120.181.141
18 19 ms 19 ms 19 ms my-super-secret-ip-goes-here

Hops 17-18 show that I'm on fastpath (6ms change) - but hop 13 I think indicates sheffield (sf.21cn-ipp.bt.net).

Just wondering if it's possible to know whether I'm likely to get the same shenanigans happening with other ISPs? I'm frankly a bit confused by BT's network so not sure how much of it is BT retail and how much is BT wholesale.

I'm looking for an ISP which is a little bit more 'enthusiast' focused after previously being with Be - which means no messing about with traffic and actually seeing things like the above as a problem, and not working as intended.

With BT's soon to be announced (to me) price rises, this seems like a good opportunity to move somewhere else.

On the shortlist is AAISP, Zen and potentially Plusnet. BT's usage monitor suggests I use about 80GB a month on average so anything 100GB plus should do the job. Sky isn't an option as I've done the pay for 12 months line rental in advance thing with BT.

So main questions:
- Will the above be fixed by moving ISP, or will it just be hidden from a traceroute?
- Comments on AAISP/Zen/Plusnet or others for an 'enthusiast'/gamer?

Edited by deleted (Wed 29-Aug-12 15:29:24)

How are you going to get out of the 18 month contract?

In reply to a post by Zadeks:

How are you going to get out of the 18 month contract?

Quoting from BT's price change FAQ linked on the tbb homepage:

However if you contact us directly within 10 calendar days of receiving our price change notification you may cease your services without penalty.

I'm no expert on tracert, but surely the route itself is irrelevant? Doesn't tracert just ping each router in turn in the route? So a straight ping would have been the 19ms from your target at that's the slow one.

To me the main indicator there of fast path is the 6ms at hop 13. The difference between two hops doesn't mean anything so far as I know.

Hop 11 is wierd though?

It might be a bit confusing because the traceroute is from a server in London to my home IP address - ie, in reverse to what you might expect!

My explanation is:

Before hop 10 is related to the server that I'm tracerouting from. Since we still have 1ms at hop 11, it's not relevant.
Hop 11 is where we enter the BT network
Hop 12 is more BT network, potentially in Ilford. Looks a bit weird, but subsequent hops make sense so we can ignore it
Hop 13 is in Sheffield. Based on the difference between hop 11 and hop 13 (5ms) this suggests roughly 5ms between Sheffield and London.
Hops 14-16 are mysteriously unnamed.
Hop 17 is the last hop before traversing the VDSL link. The difference between the known sheffield hop and this hop is about 7ms. It seems reasonable to assume that this and maybe hops 14-16 are traffic making its way back down to London to my home.
Hop 18 is my home IP. The difference between 17 and 18 is 6ms, which is consistent with ADSL/VDSL with fastpath turned on. This also suggests that this hop is in London.

As I understand it, tracert hops are not cumulative. You don't add them together to get the round trip time. Each line is the ping from source and return from the router at that hop.

So your logic is wrong, and incidenatlly hop 11 is wierd. It means from source the ping to there was 1ms.

Traceroute is quite a funky program and creates a stream of 3 packets. Each set of 3 has the same 'TTL' and it increases for each set.

The TTL IP parameter tells routers how many hops the packet can make before it has made too many and sould be dropped and a notification sent back. Normally the TTL is set to 32 (I think) and is decremented by each router the packet travels through until it gets to 0.

The first set of packets with a TTL of 1 reach the first router which decrements the TTL counter to 0 and sends back a notification. traceroute reports on this router then increments the TTL to 2 and repeats. These packets make it to the next router and the process repeats until the reporting router is the target device.

Wikipedia has more...

Correct - the ping to hop 11 from the source system was 1ms.

And if hop 17 takes 13ms from system, and hop 18 takes 19ms from the source system, then the time to traverse the link between 17 and 18 is 6ms? I'm not sure what the confusion is!

In reply to a post by driz:

Correct - the ping to hop 11 from the source system was 1ms.

And if hop 17 takes 13ms from system, and hop 18 takes 19ms from the source system, then the time to traverse the link between 17 and 18 is 6ms? I'm not sure what the confusion is!

Therefore the time taken to traverse the link between 12 and 13 was minus 5ms.

In reply to a post by bsdnazz:

Traceroute is quite a funky program ....

Thanks. I remember reading similar a while ago now you describe it. Just doesn't seem to have stuck in my brain .

-5 seconds? No, more traceroute funkyness!

The reported time included the time taken for the reporting router to realise it needs to report a packet with a zero TTL and send a ICMP packet reporting this. Routers often do this as a low priority task and so can take a while.

If you want a better guide to the response time then you need to ping each of the routers in the route that traceroute reports. You'll still be at the mercy of the routers ability to respond to the ping as opposed to route packets through the router.

Also, there's the whole issue about the return path not necessary being the same as out outbound path! Your delay could be in a router that's used on the return path and is not reported by traceroute.

If you're able to do a traceroute from both ends of a link you may find the routes are different.

Blinding me with science there, though I do follow you.

The thing is, I'm trying to explain to the OP in simple terms that he has got it wrong about his whole opening post. I think what you are telling me amounts to confirmation of that. So I was debunking his assertation that hop 17 = 16ms and hop 18 = 19ms means that a packet takes 6ms to get from router 17 to router 18. He hasn't got that yet. By highlighting the drop between hops 12 and 13 I show that is nonsense.

He is worried about the route taken, when the highest ping is from his target. The route taken, and the intermediate pings, (unless of course there is a persistent slow one and nearly all following it are above it), is irrelevant. It's not as if it routes through telia then to NY and then back to London.

As for tracert demonstrating that he is on fast path, you will know more than me. Maybe a number of 6ms ones show that? Maybe not. I'm just trying to think logically about it.

What do you think of the hop 11? I've only seen 1ms at hop 1, when the machine triggering the tracert reaches its own router.

In reply to a post by RobertoS:

The thing is, I'm trying to explain to the OP in simple terms that he has got it wrong about his whole opening post. I think what you are telling me amounts to confirmation of that. So I was debunking his assertation that hop 17 = 16ms and hop 18 = 19ms means that a packet takes 6ms to get from router 17 to router 18. He hasn't got that yet. By highlighting the drop between hops 12 and 13 I show that is nonsense.

I haven't got it wrong - here's a traceroute from my home to BBC. On Be fastpath, I was able to ping bbc.co.uk at 5ms. With BT FTTC, also fastpath, it's around 20ms probably because I'm routed via sheffield. In any event, there's something funny going on.

1 <1 ms <1 ms <1 ms 192.168.1.254
2 6 ms 5 ms 5 ms 217.32.147.97
3 6 ms 5 ms 6 ms 217.32.147.126
4 11 ms 10 ms 11 ms 213.120.181.182
5 11 ms 10 ms 11 ms 217.41.169.199
6 11 ms 12 ms 11 ms 217.41.169.109
7 11 ms 16 ms 10 ms acc2-10GigE-9-3-0.sf.21cn-ipp.bt.net [109.159.251.223]
8 18 ms 23 ms 23 ms core2-te0-0-0-6.ealing.ukcore.bt.net [109.159.251.159]
9 19 ms 19 ms 19 ms peer2-xe9-1-0.telehouse.ukcore.bt.net [109.159.254.114]
10 21 ms 21 ms 21 ms 194.74.65.42
11 * * * Request timed out.
12 58 ms 17 ms 19 ms ae1.er01.rbsov.bbc.co.uk [132.185.254.46]
13 21 ms 20 ms 20 ms 132.185.255.60
14 18 ms 18 ms 18 ms 212.58.241.131

Also, there's no need to explain things to me in simple terms - as complicated as possible would be good, please!

The reason it's 1ms at hop 11 is because the traceroute is from a server in London which is well connected.

Edited by deleted (Wed 29-Aug-12 22:53:55)

You can't be sure you're on fastpath unless you can see the router stats.

My understanding of fastpath is that it's an ATM protocol and thus only applies to the VDSL and ADSL part of the link. Basically ATM assumes that the physical path is reliable and does not do any error checking and expects the higher level protocols (TCP & IP) to do the error checking.

While the tracerotue is intersting the times are tricky to interpret because they are a mix of next work delay AND the routers ability to report packets dropped due to TTL being zero.

The important figure is the end to end ping time. Ping times to intermediate points give a better indication of network delays than traceroute times.

VDSL2 is PTM not ATM but if you can see the router stats you can tell if your connection is fastpath or not.

Right - I can make sense of that.

The difference seems to be simply that you are into the BT Wholesale WBC system, and the only way you avoid that on FTTC is Sky or TalkTalk. Be LLU centres around London and it doesn't surprise me that you had lower latency there to your secret IP, or to the Beeb.

The routing through Sheffield, (if it is Sheffield), is irrelevant - note that in your opening post you get 11ms to Ilford. You get to "Sheffield" on 6ms, and from then on a few at 9ms. The last two hops are where the problem lies, and may be better another time.

bsdnaz has given the technical description of tracert. I can follow what he says, a (not very) complicated technical overview of how it works, which confirms what I've been saying to you at a non-technical level.

The problem is essentially - I have 5ms to my first (non-router) hop, but 18ms to the BBC, so for some reason it is taking 13ms to cross London. There is no universe in which it should take 13ms to cross London. Sheffield is only irrelevant if you don't want to try and figure out why this is happening!

Using traceroute as a diagnostic tool shows the hop ending with .sf.21cn-ipp.bt.net - which I believe to be in Sheffield. Googling for 21cn-ipp.bt.net comes up with others' traceroutes which include hops such as .bm.21cn-ipp.bt.net (Birmingham) and .l-far.21cn-ipp.bt.net (London-Faraday) which correlates well with this list: http://www.kitz.co.uk/adsl/21cn_nodes.htm. Reading both forward and reverse traceroutes suggests (not directly, but mostly) about 6-7ms between London and Sheffield, which is within the bounds of reality given the distance involved.

So the semi-obvious conclusion is that for some reason my traffic is sent up to Sheffield before coming back down to London, resulting in higher latency than should be expected.

I would like this to change, and ideally would like <10ms to London based servers.
I do not think this is an unreasonable expectation. Waving it away as the vagaries of the BT WBC network is not a very satisfying answer. If the reason I'm sent to Sheffield and back is BT retail specific then that''s good, because it means that I can switch to another ISP and not have this problem. If it's BT WBC specific (maybe my exchange or cabinet is configured like this?) then it's bad, because switching to a BT wholesale ISP won't do anything.

I do not have the option of switching to Sky as they require line rental to be with them and I've already paid for that up front. Hence - the thread!

Edited by deleted (Thu 30-Aug-12 09:37:20)

While some latency is lost in the WBC network, I think this is more to do with how BT Infinity and BT Retail operates its WBC network, rather than an overall WBC issue e.g.

1    <1 ms    <1 ms    <1 ms  home.gateway.home.gateway [192.168.0.1]
2    18 ms    18 ms    19 ms  telehouse-gw4-lo1.idnet.net [212.69.63.98]
3    18 ms    18 ms    19 ms  telehouse-gw6-gi4-400.idnet.net [212.69.63.246]

4    17 ms    18 ms    18 ms  telehouse-gw5-gi4-400.idnet.net [212.69.63.245]

5    19 ms    19 ms    18 ms  rt-lonap-b.thdo.bbc.co.uk [193.203.5.91]
6     *        *        *     Request timed out.
7    18 ms    19 ms    19 ms  ae1.er01.rbsov.bbc.co.uk [132.185.254.46]
8    19 ms    19 ms    19 ms  132.185.255.60
9    19 ms    19 ms    19 ms  212.58.241.131

In short it is probably not as great because BT Retail contract their network to BT Wholesale so it uses more network than other providers, and thus bounces around due to load balancing/redundancy.

If it helps diagnostics along with Andrews trace, this is on one of our FTTC lines in Essex.

Tracing route to bbc.co.uk [212.58.241.131]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.1.1.254
2 9 ms 8 ms 8 ms lns16.the.dsl.enta.net [188.39.0.22]
3 8 ms 8 ms 8 ms gi1-6.the.dist.dsl.enta.net [188.39.0.21]
4 8 ms 8 ms 8 ms te2-2.telehouse-east3.dsl.enta.net [78.33.141.81]
5 8 ms 8 ms 8 ms te5-2.telehouse-east.core.enta.net [62.249.192.121]
6 11 ms 8 ms 8 ms bbc-linx.pr01.thdow.bbc.co.uk [195.66.224.103]
7 * * * Request timed out.
8 9 ms 9 ms 9 ms ae1.er01.rbsov.bbc.co.uk [132.185.254.46]
9 9 ms 13 ms 11 ms 132.185.255.60
10 9 ms 8 ms 8 ms 212.58.241.131

Trace complete.

Edited by vivaciti (Thu 30-Aug-12 10:06:06)

And to double check, thats entanet using WBC too.

In reply to a post by RobertoS:

In reply to a post by driz:

Correct - the ping to hop 11 from the source system was 1ms.

And if hop 17 takes 13ms from system, and hop 18 takes 19ms from the source system, then the time to traverse the link between 17 and 18 is 6ms? I'm not sure what the confusion is!

Therefore the time taken to traverse the link between 12 and 13 was minus 5ms.

This can be caused by different packet handing and traffic prioritisation along the route and at the destination - with in this case hop 12 passing packets between 11 and 13 faster than it dealing with packets "destined" for itself.

The way traceroute works is by always sending to the final destination but with a limited hop count: 1 first time, then 2, then 3, ... - so when you get a reply back (unless that reply is from the destination) it is actually an error response (hop count would be exceeded if I passed this further) which is why the node handles them differently to when it is simply forwarding the original packet on or forwarding the error report (which it doesn't know/care is an error: it is just a packet to be routed with a valid hop count) back.

Thanks, but we've been through all that. If you follow through what leads up to what you quote from me, I am saying it to debunk his assertation that the time to traverse 17-18 is 6ms, by saying something that logically follow from his belief but which is clearly absurd.

Therefore his belief is wrong.

I don't think he has grasped that even now.

If you are looking for an ISP that does absolutely nothing with your traffic including DNS (many replace NXDOMAIN errors with references to their search/advert web server for instance) then you'll need to pay a bit extra I'm afraid, to use the likes of AAISP, Zen, or their ilk.

AAISP goes a little further than most in that they are one of the few that don't even subscribe to the IWF blacklist. If using AAISP you can reduce the bandwidth cost of your ~80-100Gb considerably if you can schedule large downloads for between 0200 and 0600 when downstream transfer is metered at 5% of what it is at other non-peak times. Obviously that doesn't help is most of your downstream activity is interactive (live TV/film streaming for instance). Upstream transfer is never metered on most of the more expensive ISPs.

Oops, I thought I was replying to him there. Must have got myself lost in the thread a bit and should have read the rest in ore detail before replying!

In reply to a post by RobertoS:

Thanks, but we've been through all that. If you follow through what leads up to what you quote from me, I am saying it to debunk his assertation that the time to traverse 17-18 is 6ms, by saying something that logically follow from his belief but which is clearly absurd.

Therefore his belief is wrong.

I don't think he has grasped that even now.

Perhaps we are not understanding each other.

Hop 18 is my home router. Hop 17 is whatever one might usually refer to as my 'first hop' if you were doing a traceroute from your home out to the wider internet. 17 and 18 correspond to 1 and 2 in my second traceroute.

It is reasonably well established that FTTC fastpath (and ADSL fastpath) introduces about 5-6ms latency.

I am definitely _not wrong_ about this.

An errant value midway through a traceroute does not prove my premise incorrect - that particular router may deprioritise ICMP traffic. Or maybe it's related to MPLS. BT's network is frankly a bit of a mystery to me, which is why I'm a) trying to understand it and b) trying to move somewhere else.

Thanks both - very useful.

Ah, now I see where you are going wrong .

The direction is relevant. You just implied that it isn't. Any one, or a combination of a few adding 1ms or so, of the routers that tracert went through could have caused the extra 6ms.

It is reasonably well established that FTTC fastpath (and ADSL fastpath) introduces about 5-6ms latency.

Really?

In reply to a post by dspillett:

If you are looking for an ISP that does absolutely nothing with your traffic including DNS (many replace NXDOMAIN errors with references to their search/advert web server for instance) then you'll need to pay a bit extra I'm afraid, to use the likes of AAISP, Zen, or their ilk.

AAISP goes a little further than most in that they are one of the few that don't even subscribe to the IWF blacklist. If using AAISP you can reduce the bandwidth cost of your ~80-100Gb considerably if you can schedule large downloads for between 0200 and 0600 when downstream transfer is metered at 5% of what it is at other non-peak times. Obviously that doesn't help is most of your downstream activity is interactive (live TV/film streaming for instance). Upstream transfer is never metered on most of the more expensive ISPs.

I like the look of AAISP and seem to be the closest philosophically to what I'm looking for BUT their weekday usage rates make it untenable. We use Netflix et al pretty regularly and the idea of severely restricting daytime internet usage on a bank holiday or perhaps if someone had a day off work is just a little too constrictive.

Zen seem to be the next best choice, I think.

In reply to a post by RobertoS:

Ah, now I see where you are going wrong .

The direction is relevant. You just implied that it isn't. Any one, or a combination of a few adding 1ms or so, of the routers that tracert went through could have caused the extra 6ms.

It is reasonably well established that FTTC fastpath (and ADSL fastpath) introduces about 5-6ms latency.

Really?

You're not really claiming anything except that I'm going wrong. Except that I'm not going wrong.

Is it really just a coincidence that hop 1 to 2 going out (6ms change) is the same as hop 17 to 18 (6ms) coming in?

OR, does it show time taken to traverse that link?

Another example, using BT's looking glass from London to a server in New York - http://lg.bt.net/

This is an extract:
5 ae-59-224.ebr2.London1.Level3.net (4.69.153.141) [AS 3356] 0 msec
ae-56-221.ebr2.London1.Level3.net (4.69.153.129) [AS 3356] 0 msec
ae-58-223.ebr2.London1.Level3.net (4.69.153.137) [AS 3356] 0 msec
6 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) [AS 3356] 80 msec
ae-42-42.ebr1.NewYork1.Level3.net (4.69.137.70) [AS 3356] 68 msec
ae-43-43.ebr1.NewYork1.Level3.net (4.69.137.74) [AS 3356] 76 msec
7 *
ae-91-91.csw4.NewYork1.Level3.net (4.69.134.78) [AS 3356] 68 msec
ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) [AS 3356] 76 msec
8 ae-33-80.car3.NewYork1.Level3.net (4.69.155.133) [AS 3356] 68 msec 72 msec
ae-23-70.car3.NewYork1.Level3.net (4.69.155.69) [AS 3356] 96 msec
9 PILOSOFT-IN.car3.NewYork1.Level3.net (4.53.88.10) [AS 3356] 68 msec 72 msec 68 msec

Hop 5 is in London. Hop 8 is in New York. It is relatively well known that it takes about 70ms to cross the atlantic from London to NY. Coincidentally, this is the difference between these hops.

As to really? Well, yes.

I know a couple of people who use Zen and would be happy to recommend them so you can add that to your pool of anecdotal evidence!

A&A's daytime rates can be a pig, but for me it works out OK: there is only me and the cat and she doesn't use the Internet much, the extra when I have a day off isn't really a problem and if it was it would be spread out (due to the way carry-over/-under works) so I'd not end up paying extra. But if you regularly have someone at home in the day (you work from home and/or have a family) it would me much more of a concern.