High ping times intermittently with PlusNet

Apologies for the length of this post but there's quite a lot of history to an ongoing problem that I need some assistance with. Thanks for taking the time to read through this and I hope most of it makes sense.

Summary of the issue:
- problem started in June 2014 with high ping times affecting keyboard lag when logged into an office VPN (50 miles away) while working from home and poor online gaming performance (being kicked from ARMA 2 Epoch Mod servers for high ping times in excess of 1000ms) and my kids reporting "bad lag" on Xbox games such as Halo 4/Titanfall.
- plusnet switched me to a static IP address in June to run TBB ping graphs which showed high ping times with no real pattern to when or for how long it would last (could be an hour or two, or all night or all day, or not at all during some days or nights).
- BT Openreach sent two engineers in July with the first suspecting REIN but not finding any other real problems. The second engineer a few days later discounted REIN and tried to fix some anomalies with the polarity of the internal phone extensions.
- the problem seemed to resolve itself in July sometime after the second engineers visit but things we'd tried before that point that didn't resolve the problem:
+ plugged into master socket instead of using extensions
+ new PN Technicolor router tested instead of my usual Billion BiPac 7800N
+ replaced ADLS filter
+ wireless access point disabled and only one laptop connected via ethernet (and no machines connected at all over night)
+ updated firmware on Billion router
- problem returned in early September and PN agreed to send a third BT Openreach engineer (visit 25th September). BTO engineer James discounted REIN after lengthy investigation and replaced the master socket and wiring from the dropline. As a result we have a speed increase and no errors on the line but the problem is still there intermittently.
- issue is still happening as of midnight 26th September ... last 24 hours ping graph:
http://www.thinkbroadband.com/ping/share/eca5f1f948c...
- live ping graph:
http://www.thinkbroadband.com/ping/share/1c294b6d6f6...
- this broadband connection is in a rural area and though we're close to the exchange but on aluminium wiring to the exchange only (no cabinet). Some work has been done over the summer to bring FTTC to parts of the village.


Recent stuff:
BT Openreach were onsite for a few hours yesterday to investigate the intermittent high ping issue and resolve the CRC errors that were being reported on the line. The high ping issue was happening at the time but no major sign of REIN was found. One power adaptor in the house was identified as producing some REIN but once unplugged the problem persisted and no other REIN evidence was found. The BTO engineer agreed to put in a brand new master socket and re-wire from the drop line as some wiring between the drop line and master went through the roof space with the internal extension wiring which is known to be poor quality (as we were losing some sync speed and seeing some CRC errors when connected to the extension sockets). Now connected to the new master socket there are no CRC errors and sync speed is a much better 17351kbps (SNRM Up 6.2/Down 5.9, Line attenuation Up 10.6/Down 20.0). In the evening after the engineer's visit I was able to get the best pingtest.net results I've ever seen on my broadband connection but unfortunately from midnight the TBB ping graphs show that the problem is not resolved with lots of thick blue areas on the graph showing high ping times again.

Example of a good tracert after the engineer's visit:
25/09/2014 - 23:25:21.93
Tracing route to uwsp-webdev1.warwicknet.com [94.125.132.15] over a maximum of 30 hops:
1 1 ms 2 ms 1 ms home.gateway.home.gateway [192.168.1.254]
2 14 ms 13 ms 14 ms lo0.10.central10.pcl-bng02.plus.net [195.166.130.151]
3 20 ms 18 ms 16 ms irb.10.pcl-cr01.plus.net [84.93.249.81]
4 20 ms 16 ms 13 ms ae1.ptw-cr01.plus.net [195.166.129.0]
5 14 ms 15 ms 15 ms gi3-5.the-scr1.warwick.net.uk [195.66.224.113]
6 15 ms 16 ms 16 ms gi2-1-252.uwsp-scr1.warwick.net.uk [94.125.133.209]
7 17 ms 17 ms 17 ms uwsp-webdev1.warwicknet.com [94.125.132.15]

and a tracert when the problem is happening (exactly the same path but very different timings - and sometimes each hop can take over a second):
26/09/2014 - 8:18:00.16
Tracing route to uwsp-webdev1.warwicknet.com [94.125.132.15]over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.1.254
2 142 ms 206 ms 17 ms lo0.10.central10.pcl-bng02.plus.net [195.166.130.151]
3 43 ms * 39 ms irb.10.pcl-cr01.plus.net [84.93.249.81]
4 380 ms 519 ms 204 ms ae1.ptw-cr01.plus.net [195.166.129.0]
5 98 ms 127 ms 101 ms gi3-5.the-scr1.warwick.net.uk [195.66.224.113]
6 328 ms 25 ms 26 ms gi2-1-252.uwsp-scr1.warwick.net.uk [94.125.133.209]
7 253 ms 17 ms 288 ms uwsp-webdev1.warwicknet.com [94.125.132.15]

another tracert:
26/09/2014 - 9:52:14.39
Tracing route to uwsp-webdev1.warwicknet.com [94.125.132.15] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.1.254
2 * * * Request timed out.
3 2259 ms * 2331 ms irb.10.pcl-cr01.plus.net [84.93.249.81]
4 2643 ms 429 ms 32 ms ae1.ptw-cr01.plus.net [195.166.129.0]
5 746 ms 92 ms 23 ms gi3-5.the-scr1.warwick.net.uk [195.66.224.113]
6 244 ms 198 ms 926 ms gi2-1-252.uwsp-scr1.warwick.net.uk [94.125.133.209]
7 23 ms 184 ms 451 ms uwsp-webdev1.warwicknet.com [94.125.132.15]

Questions for anyone who has managed to get this far:
Is there anything Plusnet should or could be investigating from their end to diagnose the problem?
Are there other Plusnet customers that have had similar issues or customers of other ISPs?
Does anyone have other suggestions of a possible cause since REIN, router, microfilters, quality of the line and internal wiring have been ruled out?

Any Plusnet types able to confirm or refute that the hops where the high latency is being seen are on the PN network ?

As Zarjaz says, the first thought is that your problem is within the Plusnet network. Then I did a bit of hunting around and thinking, and now I doubt if it is.

For a start, if it really was within Plusnet we would have many people here on about it. Second, looking at this page there is only one with anything resembling your pattern - Bobsta.

Are you connecting wired or wireless to your router, and have you any Apple devices connecting to it?

Edited by RobertoS (Fri 26-Sep-14 21:56:38)

Have you ever changed the ADSL cable between the router and master socket?

Yes, the ADSL cable changed each time I switched between the Billian BiPac and the Technicolor routers. Most of the time I'm using the WiFi but often I'll switch to ethernet when I've moved the router to go directly into the master socket - this hasn't had any impact on the issue. I've also tested disabling the wifi access point with no changes to the symptoms. There's only one iPhone in the household and there doesn't appear to be any correlation between the problem occurring and the phone being connected - most days the phone isn't here. Most of the devices we have are Android phones/tablets or Windows laptops, but as I mentioned switching off the wireless access point doesn't resolve the issue.

Current status is that PN ran more line tests yesterday and now believe that the SNR might be too low and that the connection speed is too high and that a line reset/retrain is required, which they've done. I've now got to wait 10 days for that to process to complete but this SNR too low/speed too high idea doesn't really make any sense knowing the history of the problem. Will have to wait and see though I guess.

I think it unlikely that the current SNR level has any thing to do with why you are seeing this latency issue so a DLM reset probably wont make any favourable difference , Now if someone said the exchange or backhaul from it,
I would think more likely, either that or something is configured incorrectly remotely to you

Edited by tommy45 (Tue 30-Sep-14 11:16:44)

Hey Dom ! So what have they done now ?? Had it hard reset to 9db maybe ? What's the downstream SNR showing now ? It was 12db at 13 meg at first, and the issue was apparent then also, so that kind of scotches their theory.

As Tommy suggests, I reckon it's a backhaul issue .... how you get PN to address this, I don't know. As far as I am aware your exchange isn't awash with similar faults.

Have you tried sending a PM to one of their reps who posts on here, including the PN ticket ref in the message ?

Malicious traffic or malicious software inside your network can cause router overload on a cheap home router giving those sort of results.

Try unplugging the router from your network (so it's just the only thing connected) for a period and see if the graph stops showing high latency during that period.

Failing that I would try to find another plusnet user or other users from different providers on the same exchange and see if they also have the problem.

If it's not you then could be the whole exchange or just your area with the fault.

Post deleted by tommy45

Edited by tommy45 (Wed 01-Oct-14 13:12:17)

He meant disconnect the router from the LAN, (and wireless links).

Hence why i deleted the post

I didn't know you had deleted it while I was drafting mine, and having seen you had done I thought mine added useful stuff for the OP.

Hey Zarjaz, that's exactly the point I've tried to make to PN - if the SNR has only been low since last Thursday, how is it possible that a low SNR could be the cause of the problem.

These are the router stats from yesterday morning:

Upstream 1155
Downstream 19099
SNR Margin(Upstream) 6.6
SNR Margin(Downstream) 3.1
Line Attenuation(Upstream) 10.6
Line Attenuation(Downstream) 20.0

The following is a link to the PN forums where the issue is also being discussed - http://community.plus.net/forum/index.php/topic,1286... and there are PN customer relations reps responding on that thread.

Hi all, thanks for contributing on this issue.

I've been looking for something "rogue" on my network since this problem started. I've had the wifi disabled on many occasions and I have tried three different routers (a cheap TP-Link, my usual Billion BiPac 7800N and the Technicolor router that plusnet kindly sent to try to help diagnose the problem). Some sort of torrent s/w causing the issue was suggested early on but I cannot find anything within my LAN that could be responsible.

here is what I suggest.

leave it for 24 hours with wifi off and NO devices connected, just the router online so tbb can ping.

if the issue remains in my opinion its congestion.

I would find it a very bizarre line issue to cause that, the stats you posted on plusnet's forum have a low error rate and errors would typically cause packet loss not high latency.

just had a look at your graph, you really need to have all cables disconnected of router for 24 hours and make sure wifi is off, as that graph looks like you saturating your own connection.

Edited by Chrysalis (Wed 01-Oct-14 18:18:00)

It would appear that the profile is now on 3db, hence the now 19 meg sync, it was 17 meg at 6db ....... so it's been sped up, or DLM has decided that your line is so pukka (cough) that it's gotta be super stable for a fair while for that to happen ....... and that ain't the case.

Is there any chance you might follow Chysalis' advice, just to 100% prove it ain't your kit ?

I'm not sure I understand the need for 24 hours with no wifi and no kit connected ... that's not really practical but I've had the wifi disabled overnight on a few occasions with no ethernet connections and also during the day with one laptop connected (a laptop that is not always connected and for example was not on the LAN today and is not usually connected overnight) and none of that has resolved the issue at those times. If the source of the problem was something on my own LAN it couldn't possibly be using the wifi and there is only ever really one laptop that I use via ethernet when I've disabled the wifi. I honestly wish it was something as simple as some kind of rogue torrent or something else saturating my own broadband as I'm pretty confident I'd have found the source by now.

Wifi was disabled and no devices attached via ethernet to the router overnight (12 till 7) but the ping graph this morning shows the thick blue columns. Hopefully that discounts anything on the LAN from being the cause.

Zarjaz, do you think BTO would have anything to investigate at the exchange on this issue? There was been work in the village to bring FTTC to parts of it so is it possible that something isn't working 100% where my EO line reaches the exchange?

If it's not REIN or the house internal wiring or something on the LAN and PN have so far found nothing out of the ordinary at their end, what else could it be?

Zarjaz, do you think BTO would have anything to investigate at the exchange on this issue? There was been work in the village to bring FTTC to parts of it so is it possible that something isn't working 100% where my EO line reaches the exchange?

The recently enabled FTTC cabinets, 1 and 2 should be having no affect on your circuit at all. As for an Openreach bod being able to prove anything, well the usual array of testing equipment has nothing to look at packet loss.

Is there any way PN can have you put on a different pipe/virtual path back to the exchange ?

Playing about with the SNRM is having no impact on the problem. PN are talking about a "lift and shift" now to see if that helps.

Who knows, but if the issue is on the 'pipe' between the exchange and the POP site, the lift and shift won't fix it. Worth a guess though i suppose.

Heard some gossip though, possible network rearrangement on the EO stuff in Mortimer. Once through a cab, then FTTC, maybe ..... who knows.

That's the kind of gossip I'd like to hear more of ... any timescales with those rumours?

In reply to a post by dompedro:

That's the kind of gossip I'd like to hear more of ... any timescales with those rumours?

How long has this problem been happening and how long have they spent so far rolling out FTTC do you estimate?

It's possible that someone somewhere has been a complete pillock and screwed up a setting in the exchange on the 21CN kit which also transports FTTC traffic for BTw customers - settings such as a QoS policy or something to that degree. Wouldn't be the first time.

Do you have any other performance issues such as throughput? How does that look? Do web pages load like normal?

any timescales with those rumours?

Sadly, no.

Your graphs have bad packet loss, not just high latency, I was surprised this wasn't mentioned.

There also does seem possibly to be a pattern, but I'm only looking at 2 graphs. 3 broad worst periods: 1-4am, 9-11am, 1-4pm. How are the rest of the dates?

The problem started in mid-June but I cannot be sure when the work on the FTTC started at the exchange. The first (and only) cabinet so far to be switched to FTTC happened in August.

Speeds generally aren't affected by the problem. My speedtest.net runs are usually pretty decent even when the latency is really high. So for the most part webpages load pretty quickly.

I've poured over days and days worth of ping graphs to look for patterns (particularly when it was believed this might be caused by REIN). If there is a pattern to it then it's often worst from about midnight or the early hours and it can still be an issue until midday and sometimes later. Unfortunately this is only sometimes a "pattern" so not really something that could be used to diagnose a cause. I've picked a few random dates and shared the BQMs below:

10th September

16th September

24th September

2nd October

Yes there is packet loss but practically speaking isn't that just a ping that has not replied within a reasonable amount of time (4 seconds on Windows though I don't know what kit is running the BQM - Firebrick) and therefore just higher latency?

In reply to a post by dompedro:

Yes there is packet loss but practically speaking isn't that just a ping that has not replied within a reasonable amount of time (4 seconds on Windows though I don't know what kit is running the BQM - Firebrick) and therefore just higher latency?

lost pings (packet loss) when the connection is being maxed out is expected, so the small amount of red is normal. The main question is why you're getting such significant amount of blue and yellow, given your green is stable. If you compare to mine (also on plusnet) you can see that when I'm just using web etc you get almost nothing on the graph, but when I upload/download (or stream video e.g. Netflix) then the latency increases as the connection is used.

As you've already tried turning everything off and its made no difference, it _could_ be that your particular router just returns pings in its own time. The only way to compare would be for someone else using the same router to post, or for you to try a different router.

Another test you could do is set up a third party ping test: F8lure has their own system and also a Firebrick but they are on different networks to Thinkbroadband's. Could be worth setting one up for a few days to see if you get any different data. http://fruk.net/index.php?fruk=f8lure

Plusnet sent a Technicolor router for me to test with but it made no difference to the problem. I also have an old TP-Link that I've tested with.

I'm also running RouterStats to ping "out" from my network and seeing similar graphs really. Tracert also shows that there are problems when the ping graphs are at their worst. It might be worth pointing out that I noticed that there was a problem with ping and latency and then started monitoring the ping graphs in order to try to diagnose whether there was a pattern to it. The real problems are that using a VPN connection into machines in my work's office is terribly laggy and online gaming is badly affected by latency and high ping times.

In reply to a post by dompedro:

Plusnet sent a Technicolor router for me to test with but it made no difference to the problem. I also have an old TP-Link that I've tested with.

Okay, that's interesting, and rules out what I thought.

I'm also running RouterStats to ping "out" from my network and seeing similar graphs really. Tracert also shows that there are problems when the ping graphs are at their worst. It might be worth pointing out that I noticed that there was a problem with ping and latency and then started monitoring the ping graphs in order to try to diagnose whether there was a pattern to it. The real problems are that using a VPN connection into machines in my work's office is terribly laggy and online gaming is badly affected by latency and high ping times.

Yes if you've got high latency that varies a lot (jitter) then even protocols that cope with high latency can't adapt, and it makes VPN and gaming hard

Edited by jchamier (Wed 08-Oct-14 13:35:39)

The first (and only) cabinet so far to be switched to FTTC happened in August.

Not quite correct Dom, cab 2 (opposite Budgens) is live too.

Well the lift and shift apparently happened on the 7th ... unfortunately it hasn't made any difference to the symptoms of the problem.

The plusnet fault ticket is dragging on and on. Currently waiting for BTO to approve a TPM order.

So if you are awaiting a TPM, then you've NOT had a lift and shift previously

Why do BTO need to approve it ? They just do what the service providers request.

Hi dompedro,

Sorry to hear about the issue. From reading this thread it sounds like things are going in the right direction but I'm happy to cast a fresh pair of eyes on the details if need be.

Feel free to drop me a PM with your username or a ticket ID.

Adam, thanks - I've just dropped you a PM. Honestly, I can't really agree that this is going in the right direction. The fault ticket usually sits untouched for several days at a time and no-one is really actively working on it even though I've been told that the issue would be assigned to one agent at plusnet who would personally handle the case.

Zarjaz, I was told that the "lift and shine" just moved my connection to another port on the same rack and that was carried out on the 7th October at 3.30pm. I'm being told that the TPM moves my connection to a different rack entirely and initially Plusnet's "supplier" had rejected it and refused to escalate it. As far as I can tell PN are still waiting on the TPM but in the meantime BT Wholesale have suggested that this could be caused by "SHINE". It sounds to me like someone at BTW hasn't read any of the history of this issue at all!

Thought I'd share a couple of recent snapshot BQMs ... one shows the 17th October where unusually there was a very good period of ping times overnight:

http://www.thinkbroadband.com/ping/share/d31ac15388a...

and then this is what I've had to deal with today and this is becoming the "norm":

http://www.thinkbroadband.com/ping/share/89a4dd64181...

Zarjaz, I was told that the "lift and shine" just moved my connection to another port on the same rack and that was carried out on the 7th October at 3.30pm. I'm being told that the TPM moves my connection to a different rack entirely

That is correct, but highly unusual for this just to be done off the bat, usually the result of a visiting SFI engineer (you know the type ) requesting it and carrying it out as part of the visit. Are you able to confirm that you lost sync around 3:30 on the 7th ? A proper TPM will give you a fair length of no service.

Yes the connection did drop on the 7th around that time. Looks like the TPM was pushed through eventually and work must have started yesterday morning ... I'm hoping that by tomorrow morning it'll be complete. Fingers crossed that the TPM does resolve the problem.

I'm rooting for ya Dom !

The TPM seems to have done the trick and my BQM is now a very different looking thing:

http://www.thinkbroadband.com/ping/share/1c294b6d6f6...

No latency and no high pings since the TPM was completed on Thursday afternoon.

Well that's good news then Dom, and all in all, nothing a poor old engineer could have identified or requested to be done. How did Plusnet get round to requesting this for you ?

On the plus side, having your house wiring issue sorted has produced a nice speed increase

After the lift and shift hadn't helped, PN escalated things back to BT wholesale who initially rejected the TPM requests several times. When BT said that they suspected SHINE (and then REIN), PN rejected those suggestions and pointed out the history of the issue and the top quality work of one BTO engineer who'd investigated REIN in September. BT then agreed to arrange the TPM.