FTTP Performance / Zen Internet

After years of struggling with broadband from a local WISP, OpenReach finally pulled their finger out and we got FTTP installed last week with Zen. Ignoring the shockingly bad customer service and install process we were finally there.

However I've been really disappointed with the performance of the connection and its actually worse than my WISP service for latency and jitter.

What should I expect from a 150/30 FTTP service? At the moment getting 90/30'ish but with latency's of around 35/40ms.

Don't get me wrong, not shocking but without the copper for the ISP to blame I was hoping for better.

Are you testing with a wired connection - where in the country are you as latency increases the further you are from London typically. I live in London and get 2-3ms ping to bbc.co.uk and get the full 330/30 from my Cerberus FTTPoD connection.

PING bbc.co.uk (151.101.128.81): 56 data bytes
64 bytes from 151.101.128.81: icmp_seq=0 ttl=61 time=2.653 ms
64 bytes from 151.101.128.81: icmp_seq=1 ttl=61 time=2.769 ms
64 bytes from 151.101.128.81: icmp_seq=2 ttl=61 time=2.706 ms

--- bbc.co.uk ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.653/2.709/2.769/0.047 ms

Edited by brookheather (Mon 13-May-19 22:20:32)

Yes testing on wired GbE.

I'm in Lincoln so really not that far. Getting about 40ms to BBC.

I get around 5ms from Birmingham on FTTP via BT.

Pinging bbc.co.uk [151.101.64.81] with 32 bytes of data:
Reply from 151.101.64.81: bytes=32 time=5ms TTL=57
Reply from 151.101.64.81: bytes=32 time=6ms TTL=57
Reply from 151.101.64.81: bytes=32 time=5ms TTL=57
Reply from 151.101.64.81: bytes=32 time=6ms TTL=57

Ping statistics for 151.101.64.81:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 5ms, Maximum = 6ms, Average = 5ms

Trace route:

Tracing route to bbc.co.uk [151.101.192.81]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ubnt [192.168.1.1]
2 * * * Request timed out.
3 * * * Request timed out.
4 6 ms 7 ms 7 ms 31.55.187.180
5 6 ms 6 ms 6 ms core2-hu0-8-0-5.southbank.ukcore.bt.net [195.99.127.186]
6 7 ms 6 ms 6 ms peer7-et-3-1-5.telehouse.ukcore.bt.net [109.159.252.236]
7 6 ms 6 ms 6 ms peer5-te0-9-0-32.telehouse.ukcore.bt.net [195.99.126.81]
8 5 ms 5 ms 5 ms 151.101.192.81

Trace complete.

Edited by max360 (Mon 13-May-19 23:13:14)

Something sounds wrong

What are Zen saying about the speeds? And got a link to a speed test result e.g. from https://www.thinkbroadband.com/speedtest

They are saying its normal and are not interested at all.

Their service really has been shocking. I would love to leave if I could find a way.

Post deleted by dan_miles86

Need to see the speed test - Zen use different backhaul providers at different exchanges so it may be they one on yours is performing badly and swapping things around may help.

https://www.thinkbroadband.com/speedtest/15573354328... this is someone on the 330 Mbps FTTP service

So they can work well, just needs to get to someone who will realise that speeds you are getting seem slow.

Thanks, that's really helpful. That was what I was hoping for!

Who is your provider if you don't mind me asking?

Thanks, thats going to be the trick.

TBH the speed doesn't bother me too much, the latency and jitter does. Seem to be getting quite a few dropped packets too.

I've set up a BQM now as well so will see what that looks like tomorrow. A red spike aleady isnt looking promising!

Great to hear from others to know it doesn't seem right though. Gives me confidence to put more pressure on them.

https://www.thinkbroadband.com/broadband/monitoring/...

C:\Users\milesd2>tracert bbc.co.uk

Tracing route to bbc.co.uk [151.101.64.81]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.50.1
2 20 ms 20 ms 85 ms vt1.cor1.lond2.ptn.zen.net.uk [51.148.72.23]
3 31 ms 37 ms 30 ms ae-17.agg3.lond2.ptn.zen.net.uk [51.148.73.32]
4 20 ms * 20 ms 195.66.225.91
5 20 ms 20 ms 20 ms 151.101.64.81

Trace complete.

Some spiking is possible if for example you download or upload a lot e.g. a speediest so best guide is leaving it idle over night (even switching wireless off) and seeing how stable it is then

The jitter looks very low compared to say VIrgin Media your few spikes look most like usage at present.

20ms first hop looks reasonable for Lincoln. FTTC can be fairly low ping across the VDSL2 segment and for the majority of people it is the regional routing that is the issue these days i.e. FTTP is not the magic answer to low latency unless you had problems due to interleaving etc on VDSL2

Edited by MrSaffron (Mon 13-May-19 20:13:16)

In reply to a post by dan_miles86:

They are saying its normal and are not interested at all.

Their service really has been shocking. I would love to leave if I could find a way.

That's not sound likely from Zen Internet as they always the best service provider, I be shocked if they don't wanna to know your FTTP problem with slow speed. Hope Zen Internet will ask BT Wholesale to put u on different SVLAN.

Edited by adslmax (Mon 13-May-19 23:56:54)

In reply to a post by brookheather:

Are you testing with a wired connection - where in the country are you as latency increases the further you are from London typically. I live in London and get 2-3ms ping to bbc.co.uk and get the full 330/30 from my Cerberus FTTPoD connection.

Agreed, I am about 5 miles (by road) from Telehouse in the docklands (where it goes as far as I am aware) and I get between 1.7 and 2.8ms (on average 2.6ms) ping to the BBC Site.

[ON PC]
ping -4 bbc.co.uk

Pinging bbc.co.uk [151.101.128.81] with 32 bytes of data:
Reply from 151.101.128.81: bytes=32 time=2ms TTL=57
Reply from 151.101.128.81: bytes=32 time=1ms TTL=57
Reply from 151.101.128.81: bytes=32 time=1ms TTL=57
Reply from 151.101.128.81: bytes=32 time=2ms TTL=57

Ping statistics for 151.101.128.81:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 2ms, Average = 1ms

I think those 1ms on my results are more close to 2ms due to it rounds down, but the last time I checked on my Linux box it shows the 2 decimal points and its close to 2ms.

So if the OP is in London they "should" be getting the same or close to it.

Paul

In reply to a post by PaulKirby:

So if the OP is in London they "should" be getting the same or close to it.

The OP is in Lincoln not London.

What matters most is the variability in RTT, rather than the absolute value. If it's 20ms consistently, then I'd expect it's just a speed-of-light limit based on the distance travelled (*). However if it varies widely, then that's a sign of congestion. The lowest RTT observed is the path delay, and the variation is queuing delays. What RTT is observed at 3am?

(*) Having said that, if the backhaul path is over fibre, 20ms ought to get you to Scotland. RTT increases roughly 1ms per 100km (light travelling at 200km/sec in fibre). For comparison: London to AWS in Ireland is about 10ms.

There's another thing you can do. If you have a Virtual Private Server in the cloud somewhere, you can find your home IP address (whatsmyip.org) and then traceroute back to that address. RTT is the total delay A->B->A, and sometimes the A->B path is very different to the B->A path.

In reply to a post by candlerb:

In reply to a post by PaulKirby:

So if the OP is in London they "should" be getting the same or close to it.

The OP is in Lincoln not London.

What matters most is the variability in RTT, rather than the absolute value. If it's 20ms consistently, then I'd expect it's just a speed-of-light limit based on the distance travelled (*). However if it varies widely, then that's a sign of congestion. The lowest RTT observed is the path delay, and the variation is queuing delays. What RTT is observed at 3am?

(*) Having said that, if the backhaul path is over fibre, 20ms ought to get you to Scotland. RTT increases roughly 1ms per 100km (light travelling at 200km/sec in fibre). For comparison: London to AWS in Ireland is about 10ms.

There's another thing you can do. If you have a Virtual Private Server in the cloud somewhere, you can find your home IP address (whatsmyip.org) and then traceroute back to that address. RTT is the total delay A->B->A, and sometimes the A->B path is very different to the B->A path.

I'm in Inverness yet getting ~20ms on FTTP, so the OP being in Lincoln (ie much closer to London) should be getting around half of that, unless the OP's fibre circuit takes a scenic route around the British Isles.

OP, if Zen aren't being helpful over the phone, I would message their TBB rep, ajays (Andrew) on these forums who should be able to investigate further.

C:\>ping bbc.co.uk

Pinging bbc.co.uk [151.101.0.81] with 32 bytes of data:
Reply from 151.101.0.81: bytes=32 time=20ms TTL=58
Reply from 151.101.0.81: bytes=32 time=21ms TTL=58
Reply from 151.101.0.81: bytes=32 time=20ms TTL=58
Reply from 151.101.0.81: bytes=32 time=20ms TTL=58

Ping statistics for 151.101.0.81:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 20ms, Maximum = 21ms, Average = 20ms

In reply to a post by candlerb:

In reply to a post by PaulKirby:

So if the OP is in London they "should" be getting the same or close to it.

The OP is in Lincoln not London.

Yeah you are right, I was half asleep when I replied, all I saw in my head was London, my bad.

Paul

I am wondering if they are having routing issues and being routed to a further Internet Exchange.

Maybe one of the Manchester Internet Exchanges or the one in Leeds might of been better for them instead of a London one.

Paul

I wonder if the OP has tried a different router .... ?

In reply to a post by PaulKirby:

I am wondering if they are having routing issues and being routed to a further Internet Exchange.

Maybe one of the Manchester Internet Exchanges or the one in Leeds might of been better for them instead of a London one.

Paul

Nah. Across the BT Wholesale network, I presume, to Telehouse North then straight across LINX London according to their trace

1 <1 ms <1 ms <1 ms 192.168.50.1
2 20 ms 20 ms 85 ms vt1.cor1.lond2.ptn.zen.net.uk [51.148.72.23] << Telehouse North
3 31 ms 37 ms 30 ms ae-17.agg3.lond2.ptn.zen.net.uk [51.148.73.32]
4 20 ms * 20 ms 195.66.225.91 << LINX LON1 / Juniper LAN
5 20 ms 20 ms 20 ms 151.101.64.81

Might be taking a rather circuitous route across the BT Wholesale network, mind.

In reply to a post by Zarjaz:

I wonder if the OP has tried a different router .... ?

Yeah, that too, not too sure if Zen are hiding some hops, but 20ms from the router to the next hop seems a bit too high.

Or even done a complete power recycle of the ONT and Router to see if that helped.

Paul

The 85ms on the third test of line 2, and the whole of line 3, plus the timed out entry on line 4 suggest to me that the two Zen routers are getting some short-term queues of heavy traffic so not handling the pings well.

Not even passing them through occasionally?

In reply to a post by Ignitionnet:

In reply to a post by PaulKirby:

I am wondering if they are having routing issues and being routed to a further Internet Exchange.

Maybe one of the Manchester Internet Exchanges or the one in Leeds might of been better for them instead of a London one.

Paul

Nah. Across the BT Wholesale network, I presume, to Telehouse North then straight across LINX London according to their trace

1 <1 ms <1 ms <1 ms 192.168.50.1
2 20 ms 20 ms 85 ms vt1.cor1.lond2.ptn.zen.net.uk [51.148.72.23] << Telehouse North
3 31 ms 37 ms 30 ms ae-17.agg3.lond2.ptn.zen.net.uk [51.148.73.32]
4 20 ms * 20 ms 195.66.225.91 << LINX LON1 / Juniper LAN
5 20 ms 20 ms 20 ms 151.101.64.81

Might be taking a rather circuitous route across the BT Wholesale network, mind.

My point was they are closer to Manchester or Leeds then they are to London.

Also not too sure if Zen hides a lot of the hops (or only display hops that respond) but I am liking the 5 hops to the BBC Site

Paul

In reply to a post by RobertoS:

The 85ms on the third test of line 2, and the whole of line 3, plus the timed out entry on line 4 suggest to me that the two Zen routers are getting some short-term queues of heavy traffic so not handling the pings well.

Not even passing them through occasionally?

The lack of packet loss to the end destination alongside zero evident jitter, all 3 pings coming back with the same round trip, suggests to me that there aren't issues there. Ping response from a router takes a completely different path to the one forwarding them uses and should be ignored as a general rule. The only time it's relevant is when all subsequent hops show the same condition of loss / high latency.

I know/knew what you are saying but hold to what I said too.

You are correct in that the anomalous behaviour is not relevant to data throughput, but it does point to those two routers having more important things to do at those times than respond to pings.

That�s the point of the test as opposed to a straight ping test.

As you say, a consistent rise at and following a given step shows a congestion/pinch point that could affect real traffic, so in general you are right. But in this case we have a delayed response spanning three hops, all involving the two Zen routers in London, then clearing. I believe that in this particular case those delays were relevant.

Traceroute goes along the control plane not the forwarding plane. It may be treated with a low priority if there are other more pressing tasks such as pretty much anything, it may be rate limited, it may be blocked entirely. Not sure what the point is of the test if the anomalous behaviour isn't relevant to throughput but should be noted.

I find it more likely if the home install is clean that this is an issue on the BT Wholesale network with the customer's connection taking a very circuitous route to Zen's LNS.

Anyway here's Zen's agg3.lond2.ptn.zen.net.uk:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 51.148.73.32, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Here's cor1.lond2.ptn.zen.net.uk

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 51.148.73.31, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Both pinged from the LINX LAN, both seem healthy. I fully imagine latency increases when they are converging their routing tables but, as you said, this isn't relevant to throughput.

LOL
Despite your blinding with science type post, I think it suggests we are in fact now in agreement.

For instance, I started from the assumption that the timeout at 4 20 ms * 20 ms 195.66.225.91 << LINX LON1 / Juniper LAN was due to something prior to that, specifically to do with zen. I wouldn't expect that LINX routers are going to timeout on a ping. It's far more likely it was dropped in one direction or the other in the "zen network".

I also noted that the displayed transfer within LON2 is zen to zen.

Hop 3 is relatively normal, I agree that. The third entry of 85ms on hop 2 could be that router, or something earlier. You suggest it was earlier.

However I find this repeated opinion, "an issue on the BT Wholesale network with the customer's connection taking a very circuitous route to Zen's LNS" rather vague. Are you blaming BT Wholesale itself, or zen's rented WBMC MSIL capacity/option? Have we in fact established this is routing via BTW anyway, not zen's own backhaul?

Not so much in agreement. I'm not remotely bothered by the 85ms latency and aren't going to claim it was caused by anything prior to it as there's no evidence of that, especially within the poster's BQM.

Wasn't blinding with science, or trying to at least, just stating how things work.

I am confident that the FTTP is over the BT Wholesale network because, as far as I know, Zen don't have any FTTP CableLinks at this time outside of perhaps some trials.

I am not blaming any subset of the route between the customer and Zen's network because there's no data to support either claim above the other.

Anyway with that back to the day job

EDIT: https://www.thinkbroadband.com/broadband/monitoring/...

No spikes around time of traceroute anywhere near 85ms. Packets are getting forwarded okay by the looks with maybe a touch of peak time strain.

Edited by deleted (Tue 14-May-19 14:03:09)

Looking at their BQM

Looks like something went on around 1:45 to 2pm today and there was a drop in latency at around 2:10pm

Of course no idea what the change was, maybe different router or a routing tweak from Zen or wholesale team

Prior to this change the overnight looks as clean as I'd expect with the yellow block between 9pm and 11pm looking like actual usage of the connection.

Think I've seen their speed test while doing the daily rounds, and something no one has suggested but might explain speed problems...

So to original poster double check the speeds that the Ethernet interfaces are connecting at, since speed test is nailed to what you would get for a port running at 100 Mbps.

Worth mentioning that an RJ45 cable or port with one pin missing can downgrade the link from 1G to 100M. So can using a "crossover" cable instead of a straight-through one.

In reply to a post by RobertoS:

For instance, I started from the assumption that the timeout at 4 20 ms * 20 ms 195.66.225.91 << LINX LON1 / Juniper LAN was due to something prior to that, specifically to do with zen. I wouldn't expect that LINX routers are going to timeout on a ping. It's far more likely it was dropped in one direction or the other in the "zen network".

The IP you see isn't a Linx router. It is an IP of a Linx member and will be the Linx member's device. In this case, Fastly, who provide a CDN to BBC.

The Linx member can apply whatever policy they want to their interface, some place very restrictive limits on diagnostic protocols like ICMP so that it can concentrate on the more important job of routing as Ignitionnet as mentioned in a more technical way. Some Linx members don't respond to ICMP at all.

The reverse route may not even be the same from Fastly to the end end user, so the "loss" seen may actually have been caused by a completely different path having an issue outside of the Linx, Zen or Fastly network.

Matt

Heh! Hehe! Matt . Thanks. The plot thickens.

That'll teach me not to take labels inserted in capitalised bold by someone I realise knows more technically than I do too literally . Next time we have anything similar I will check for this sort of thing .

As for the concentrating on doing the real job rather than bother with "user" diagnostics, that I fully understand. As I explained to Ignitionnet. It's a basic feature of tracert that we all learn very quickly.

I still believe that zen does have some bottleneck problems, as evidenced by more and more postings by their users over the last year or so. The fact my interpretation of hop 4 is probably wrong does not rule that out.

In particular the single 85ms of three in hop 2 and the single timeout surrounded by the 20ms ones for hop 4, the router in question, I suggest do point the finger at either the zen hop 2 router or something earlier in the zen network. As Ignitionnet himself propounds.

Edited by RobertoS (Wed 15-May-19 00:28:32)