Packet Loss of 40%

Following what I think must have been BT maintenance at 00.25hrs on the 24th February, my line dropped from attainable speeds of 85D and 23U to 67.4D and 18U. A look at my Line data shows no major issues for nearly 3 years prior to this 25% fall. . Associated with this has been frequent bursts of latency up to 150ms and, from 10am today, almost constant packet loss of 40%. To date, Zen's support has been lack lustre: i.e., little we can do as the speed is still above 55 (the impacted figure?).

I am not sure what to do next? I am considering migrating to AAISP but I suspect that the fault would just migrate with me unless I pay for a separate copper pair (new line).

I should add that my Fritz!Box 7490 log suggests that the issue is backhaul related. There are no errors showing on the downstream.

Hi, Please can you PM me your details and I'll get this looked into for you ASAP!

In reply to a post by ajays:

Hi, Please can you PM me your details and I'll get this looked into for you ASAP!

PM sent. Your colleagues have all the details including graphs and screenshots.The issue was first reported last week.

Any thoughts on the likely cause of this on my BQM?

Here

Ouch.

No expert in this, but the text on the graph in the BQM FAQ that most closely matches your graph has the following cation against it:

This graph is showing consistent packet loss between 2% and 15%, which would making gaming or Voice over IP (VoIP) across the connection very difficult. The minimum latency is higher than average, which suggests a permanently congested link somewhere between the broadband router and our BQM Pingbox. The fact the blue area is still narrow suggests the packet loss is not due to the user downloading on the connection, but some issue in a core network.

Not sure how relevant it is to you though as your minimum ping looks to be a steady 12ms.

In reply to a post by jaydub:

Ouch.

No expert in this, but the text on the graph in the BQM FAQ that most closely matches your graph has the following cation against it:

This graph is showing consistent packet loss between 2% and 15%, which would making gaming or Voice over IP (VoIP) across the connection very difficult. The minimum latency is higher than average, which suggests a permanently congested link somewhere between the broadband router and our BQM Pingbox. The fact the blue area is still narrow suggests the packet loss is not due to the user downloading on the connection, but some issue in a core network.

Not sure how relevant it is to you though as your minimum ping looks to be a steady 12ms.

Thanks. Zen, I am afraid, are offering no practical help or support whatsoever. I am bitterly disappointed with their response given the time that I have been with them. I am now looking at switching ISP - possibly to a LLU provider or to AAISP.

Stating the obvious but clearly a cyclic pattern repeating every 5hrs or so.

I wish BQMs could be tagged with metadata like, ISP, Exchange, cabinet to help trouble shoot problems like this.

e.g. If you could see other BQMs on your cabinet or exchange and they are the same it helps to narrow down where the problem might be and helps expose under performing ISP's if others using same ISP are seeing similar patterns.

My BQM was better than yours prior to the 24th February with an up time in weeks, and max attainable speeds of 85D and 24U. There is also an ongoing PCP change which as I have alluded to before may be a factor.

That is an ugly BQM - bouts of huge packet loss, implying that either there is some defect or a congested link somewhere. Considering the timing, it looks more like a defect such as failed equipment or possibly a configuration error; I wouldn't expect severe congestion in the middle of the night.

There is no general problem with Zen, as can be seen from the IPv4 and IPv6 BQMs in my signature. I'm on Zen backhaul, not BT Wholesale.

I am on BT backhaul but the Zen Portal shows that I am due a regrade to GEA. My Fritz!Box 7490 diagnostics is showing occasional internet line interruptions which I assume might be associated with the packet loss.

If it's a BT Wholesale (BTW) FTTC connection, you could ask for a Lower Threshold Breach (LTB) fault to be raised:

http://www.ispreview.co.uk/index.php/2012/12/when-sl...

"The 25% Rule

A lesser known fact is that of the 25% rule, which usually applies to services that initially sync up at 15Mbps Downstream or above. According to BTWholesale, �if the line rate drops by more than 25% over a 14 day continuous period then a fault can be reported�. In practice it helps to have a vigilant ISP on your side for this one as they�re best placed to provide BT with the necessary data."

EDIT: 85Mbps minus 25% is 64Mbps... at 67Mbps you're just over the threshold. But if diagnostics show 40% packet loss provable to the FTTC line this should still be raised to BT Wholsale as a line fault.

EDIT: Assuming this is a BT Wholesale connection there's a whole section on packet loss at:

https://www.btwholesale.com/assets/documents/Orders_...

From the decrease in sync speed and increase in packet loss it sounds like a copper line issue and not backhaul ('hot VP') related. A line test *should* pick up any physical line faults. Caveat: it's impossible to be conclusive about this without access to ISP diagnostic results from line tests, logs etc

Edited by deleted (Tue 07-Mar-17 11:21:18)

In reply to a post by Pontefract:

If it's a BT Wholesale (BTW) FTTC connection, you could ask for a Lower Threshold Breach (LTB) fault to be raised:

http://www.ispreview.co.uk/index.php/2012/12/when-sl...

"The 25% Rule

A lesser known fact is that of the 25% rule, which usually applies to services that initially sync up at 15Mbps Downstream or above. According to BTWholesale, �if the line rate drops by more than 25% over a 14 day continuous period then a fault can be reported�. In practice it helps to have a vigilant ISP on your side for this one as they�re best placed to provide BT with the necessary data."

EDIT: 85Mbps minus 25% is 64Mbps... at 67Mbps you're just over the threshold. But if diagnostics show 40% packet loss provable to the FTTC line this should still be raised to BT Wholsale as a line fault.

EDIT: Assuming this is a BT Wholesale connection there's a whole section on packet loss at:

https://www.btwholesale.com/assets/documents/Orders_...

From the decrease in sync speed and increase in packet loss it sounds like a copper line issue and not backhaul ('hot VP') related. A line test *should* pick up any physical line faults. Caveat: it's impossible to be conclusive about this without access to ISP diagnostic results from line tests, logs etc

I wandered down to the PCP and there are two guys working on the transfer of lines to the new PCP. It would seem that they tried some time to install new blocks in the old cabinet but after completing two columns they just gave up and installed a new PCP next to the VDSL box.

My line has yet to be transferred across: it should happen by Friday. The wiring in the old box is a tangled mess with the engineer having to move bundled aluminium and copper lines to find the line that he is trying to move across. Very kindly, the engineer has re-jumpered my line after he thought that he had detected a break under the insulation. The DSL has re-synched but I am still getting packet loss. I was told that they have had a number of SFI visits to fix broadband faults. I just wonder how those customers got their ISP to play ball: I failed.

How many BQM/Ping monitors are running on your line?

Did you know when Fritz!Boxes see too many pings, they don't reply and you see this cyclic pattern of packet loss. Perhaps Zen have a monitor running?

Do you have an IPv6 address you could monitor instead.

In reply to a post by bet_here:

How many BQM/Ping monitors are running on your line?

Did you know when Fritz!Boxes see too many pings, they don't reply and you see this cyclic pattern of packet loss. Perhaps Zen have a monitor running?

Do you have an IPv6 address you could monitor instead.

Interesting that you should ask. I was just running a single BQM. After today's fix, the baseline latency has fallen but the packet loss has returned. I set up an IPv6 BQM an hour or so ago and it is showing 100% packet loss. That said, both the IPv4 and the IPv6 speedtests are fine and the Fritz!Box error is nothing out of the ordinary. To date, no stuttering on streaming or on FaceTime. Zen say that they are seeing more packet loss than is showing on the BQM.

BQM v6

So Zen have a monitor running too. Suspend your monitors and ask Zen if theirs clears up.

In reply to a post by bet_here:

So Zen have a monitor running too. Suspend your monitors and ask Zen if theirs clears up.

Zen seem happy to receive BQMs from me and they haven't mentioned the possibility of a conflict. IPv6 is showing 100% packet loss.

It's a 'feature' of the Fritz!box.

If you (and/or Zen) run more than 1 monitor, then you will see high or complete packet loss on one or all of the graphs.

In reply to a post by bet_here:

It's a 'feature' of the Fritz!box.

If you (and/or Zen) run more than 1 monitor, then you will see high or complete packet loss on one or all of the graphs.

I would have thought that Zen Support would know that as they supply Fritz!Boxes as a VDSL modem/router. I should have added that I did not off the IPv4 BQM for a time last night and the IPv6 still showed solid red. I have turned it off again so I will see what happens .

Edited by lexden16 (Wed 08-Mar-17 10:40:54)

If you didn't know about the BQM, would you have said you're seeing performance issues from generally using your connection?

In reply to a post by AndyHCZ:

If you didn't know about the BQM, would you have said you're seeing performance issues from generally using your connection?

Yes, a 25% drop in the max attainable speed after 3-4 years of stability suggests a performance issue. Ten days of high latency as detected by speed tests (+150ms+) and now packet loss suggests that there is an issue. I appreciate that most people would regard 67D as a good FTTC connection but from where I am sitting I can see the guys working on the 2 PCPs and underground cable.

Until now, I have rarely looked at the BQM - my line just worked. The fact that the deterioration occurred after a midnight event suggests, to me, that it is maintenance related. The ongoing PCP change is a factor that isn't helping. In fairness to the guys, they are trying to sort out a mess not of their making.

In my experience packet loss isn't a facet of a 'line fault', it is always equipment (somewhere) or off in magic internet land.

I would have stood by that apart from the cab changeover they are doing, so time to take the lads some tea and biscuits and ask if, at present, your pair is teed to both cabs. Even then this is straw clutching.

Does trace route show where the loss may be coming from ?

[edit]Now I think of it, if it were teed to both a remote test would be showing bridge tap, and I suspect Zen might have picked this up already.

Edited by Zarjaz (Wed 08-Mar-17 11:54:26)

In reply to a post by Zarjaz:

In my experience packet loss isn't a facet of a 'line fault', it is always equipment (somewhere) or off in magic internet land.

I would have stood by that apart from the cab changeover they are doing, so time to take the lads some tea and biscuits and ask if, at present, your pair is teed to both cabs. Even then this is straw clutching.

Does trace route show where the loss may be coming from ?

[edit]Now I think of it, if it were teed to both a remote test would be showing bridge tap, and I suspect Zen might have picked this up already.

I have agreed with Zen that it makes no sense to look at the issue again until the PCP work is complete which should be the case by COP Friday. The guys at the PCP have been extremely helpful - even to the point of re-jumpering my connection in the old cabinet. As I posted earlier, they say that they have had some SFI engineer visits to resolve issues as wires keep breaking. This would have been the case last Wednesday had I just contacted Zen and told them that I had the VDSL carrier but no internet connection. I suspect that many who do not have access to their stats haven't a clue what might have been happening to their lines. Perhaps too much information is a bad thing.

I am still considering a move to AAISP which with my usage would cost me £5 a month more. This is the second occasion in 3 years where Zen Support has proved to be less supportive than I would have expected. The previous incident involved a loss of PPPoE authentication. Eventually, a SFI engineer was called: he took one look at some F!B logs, and a simple authentication test using a laptop and no router that I had set up, and he identified a radius fault between BTW and BTOR. After a telephone call, a lift and switch and DLM re-set, he was on his way in 20 minutes.

In my experience packet loss isn't a facet of a 'line fault', it is always equipment (somewhere) or off in magic internet land.

My experience of this is very different; I've seen many line faults where packet loss has been a symptom.

Could you post your latest BQM? Do you have a copy of your line test from Zen?

My experience of this is very different; I've seen many line faults where packet loss has been a symptom.

How were these faults resolved, and what kind of line faults were they ?

In reply to a post by AndyHCZ:

Could you post your latest BQM? Do you have a copy of your line test from Zen?

This is my latest snapshotBQM

It has been turned off for a couple of hours to see whether anything changed on the IPv6 BQM. It didn't: packet loss is 100%.

This forum page took 23 seconds to load on a wired LAN.

My speed tests are fine for the profile that I am now on. (65227/18171).

I had a disconnect for 5 mins at 12.01. My profile fell a notch but my line attenuation has increased from 13/13 to 16/14, SNR 6/6 so I am wondering whether my line is now via the new PCP. It is raining and the guys are in their igloos so I don't want to bother them.

Fritz!Box errors are also very low after 50mins Fritz!Box 3 ES 0 MSES CRC 0.6/M last 15 mins O

Central Exchange 0 0 0 0

Perhaps things are on the up.

If you can access the internet and you're seeing 100% packet loss on the v6 BQM, then it's almost certainly the router blocking ICMP requests.

To me, that BQM does not show a line fault. If there was an underlying issue with the line causing severe packet loss, then you would not expect to see such a low minimum/average latency on your line (looks around 12ms for you). The router would be reporting errors and the BT line check would be showing there are errors on the line and DLM keeps intervening.

It's definitely worth pressing Zen for a copy of the BT line test results. This contains useful information about how your line is performing and how DLM is intervening.

How were these faults resolved, and what kind of line faults were they ?

Mostly physical line degradation of some sort or water ingress. Usually resolved by an SFI visit.

The number of observations this is based on is large - my day job has involved resolving DSL faults at various ISPs on and off since about 2003.

In reply to a post by Pontefract:

How were these faults resolved, and what kind of line faults were they ?

Mostly physical line degradation of some sort or water ingress. Usually resolved by an SFI visit.

The number of observations this is based on is large - my day job has involved resolving DSL faults at various ISPs on and off since about 2003.

Useful to know. Thank you.

The number of observations this is based on is large - my day job has involved resolving DSL faults at various ISPs on and off since about 2003.

Ah well I'll bow to your superior judgement then.

In reply to a post by lexden16:

I would have thought that Zen Support would know that as they supply Fritz!Boxes as a VDSL modem/router. I should have added that I did not off the IPv4 BQM for a time last night and the IPv6 still showed solid red. I have turned it off again so I will see what happens .

Earlier you said "Zen say that they are seeing more packet loss than is showing on the BQM". If they are running a monitor on your line too, then that is what you are seeing on your BQM

Login to http://fritz.box/html/capture.html
Capture 30 secs. or so of traffic on the internet interface, and view the file in Wireshark. Filter on ICMP or ICMPv6 and you'll see if Zen are pinging your router.

More down than up at the moment: 3 re-synchs in the last 15 minutes including one with a downstream SNR of minus 4. I am beginning to hope that the line just gives up altogether.

So it's not just packet loss you are experiencing ....

I am getting more confused by the minute. I have now had 5 re-synchs in the past 40 or so minutes. The profile fell to 50 and stayed there for 10 mins. I then saw Carrier J43. Following a further re-synch, the profile has returned to 66.4/17.7. Line attenuation has increased to 17/15 and the SNR is now 6/9 with a Carrier of A43.

I assume that there is more to this than 2 men working in the PCPs or in an adjacent underground link.

You really need Zen to do a line check to see what it's coming back with.

It's not inconceivable that this might be a router/modem issue.

In reply to a post by AndyHCZ:

You really need Zen to do a line check to see what it's coming back with.

It's not inconceivable that this might be a router/modem issue.

The Law of Probabilities suggests that this afternoon's series of 8 re-synchs was related to the PCP work. My connection has now been as steady as a rock since 4.10pm which was probably about the time when they were stuffing wiring et al back into the old PCP, and getting ready to knock off. Zen still say that my line has no issues. I accept that they can only report what they see.

But your BQM is still showing around 40% packet loss. A connection with this level of packets being lost would be next to useless.

I am surprised Zen cannot see anything. If you lost sync 8 times this afternoon, you would be flagged up on the MTBR.

Zen still say that my line has no issues. I accept that they can only report what they see.

Do they see the 40% packet loss if they ping the WAN ip of your router?

If you get the same result with two different routers (if you can borrow one or have one available) tested when connected to your NTE5 master test socket that at least rules out equipment and internal phone wiring your side as the cause.

Eleven re-connects yesterday which stopped at 4.10pm as the guys left the PCP site. My connection was steady overnight until 07.08am this morning when DLM kicked in and reduced my downstream speed by 9Mbps with latency 8m and iNP.

Update

I have just walked past the PCP to find two of my neighbours complianing about frequent dis-connections; loss of speed etc. I overhead one lady saying that they had two lines with different ISPs and both connections were badly affected. The words 'aluminium and cabinet' were mentioned frequently by the engineers. Obviously, this change of cabinet wasn't well planned. Why, for example, haven't the ISPs been told?

Edited by lexden16 (Thu 09-Mar-17 15:38:47)

Update:

The PCP men have gone - hopefully, never to return. My router log shows 44 disconnects in total since the beginning of this month. Not surprisingly, DLM has kicked and I have a profile of 67/20 with max attainable speeds of 67/23. The error rate is low but webpages are often slow to load. My latest BQM is:here

My line is due to change from BT backhaul to Zen's own system tomorrow (according to another ISP) so I will see if that changes anything.

In reply to a post by lexden16:

My line is due to change from BT backhaul to Zen's own system tomorrow (according to another ISP) so I will see if that changes anything.

How would another ISP know this?

In reply to a post by AndyHCZ:

In reply to a post by lexden16:

My line is due to change from BT backhaul to Zen's own system tomorrow (according to another ISP) so I will see if that changes anything.

How would another ISP know this?

They picked up on the fact that there was a broadband cease order showing for my line on the BT Availability Checker. As I hadn't initiated a cease, they concluded - correctly - that it was most likely a re-grade which, after checking in the depths of the Zen Portal, proved to be the case. The line was moved from WMBC to GEA just after midnight on the 15th, and my downstream is back at 78 - still about 5Mbps less than was the case before this nonsense started.

I would be extremely surprised (and concerned) if ISPs were not informed.

I see planned engineering work all the time, affecting anything from a few lines, up to hundreds of lines and more. For the most part, PEWs do not affect end users or if they do, they result in a very brief loss of disconnection in the early hours. I think is why nearly all ISPs do not report this work.

I am not aware of a single ISP that would automatically report every single planned engineering work affecting specific circuits to end users. There are a variety of sources of PEW and I think it's quite a big task for any ISP to be able to input the multiple daily updates and then output this as meaningful and useful information for end users.

In reply to a post by AndyHCZ:

I would be extremely surprised (and concerned) if ISPs were not informed.

I see planned engineering work all the time, affecting anything from a few lines, up to hundreds of lines and more. For the most part, PEWs do not affect end users or if they do, they result in a very brief loss of disconnection in the early hours. I think is why nearly all ISPs do not report this work.

I am not aware of a single ISP that would automatically report every single planned engineering work affecting specific circuits to end users. There are a variety of sources of PEW and I think it's quite a big task for any ISP to be able to input the multiple daily updates and then output this as meaningful and useful information for end users.

I can assure you that according to Zen, they were not informed of this planned engineering work and it has never appeared on any of the PEW status reporting sites. Indeed, the very nice guys that I spoke to on site said that there should only be one disconnection per line. When I chatted to them a few days later, they admitted that they had found aluminium in the old cabinet that was giving them a lot of break of service issues ( which, in turn, meant that they were receiving a lot of complaints about BB disconnections from my neighbours).

I wasn't suggesting that the ISP should inform end users but, in my view, as I had raised an issue with them, I would have expected them to contact BTOR. All Zen would say is 'we do not have many customers on your exchange.'

Not quite sure how this showed up as a cease. A cease should only happen if the underling WLR/MPF is cancelled or the FTTC service is stopped.

It should have been a migration or possibly, a C/S-VLAN move (I am not sure how BTWholesale deal with this). If that's the case, it would have shown on the systems and stopped anyone else from placing an order.

Which exchange are you on?

If I was in Zen's position, I would then be making a formal complaint to Openreach as it's a breach of the SLA for FTTC.

Edited by deleted (Thu 16-Mar-17 13:04:38)

In reply to a post by AndyHCZ:

Not quite sure how this showed up as a cease. A cease should only happen if the underling WLR/MPF is cancelled or the FTTC service is stopped.

It should have been a migration or possibly, a C/S-VLAN move (I am not sure how BTWholesale deal with this). If that's the case, it would have shown on the systems and stopped anyone else from placing an order.

The words used were:

There is currently a Broadband cease pending or in progress against this line. This cease is due to complete by 15th March 2017. This will not stop you ordering Broadband from a new supplier, but it will delay the provision of the new Broadband service.

It is interesting that (a) Zen does not pro-actively inform its customers off this backhaul change and (b) support couldn't tell me when the re-grade would go through. No big deal; that said, others have been worried about this in the past as posted on this site. Indeed, even I joked with my family that we might be in for a few internet free days if it was a genuine cease order.

And what the Lord Giveth, the Lord Taketh Away:

BQM Here

I still haven't got a clue what was going on. There are no events showing in my system log.

Hi lexden16,

I started having a packet loss issue around the same date, and I also use a Fritz!Box 7490. Two engineer visits and no faults could be found; the JSU showed the line was perfect.

I heard about another Fritz user with the same symptoms, and I checked and noticed that Fritz!OS 06.80 was released around that time. I used the DSL option to use the previous DSL formware, and since then my errors have disappeared and my line rate has restored: it took DLM a couple of weeks but I'm back to where I was. I've avoided upgrading to Fritz!OS 06.83 since it doesn't mention and DSL fixes.

You may not have the same issue, but downgrading the DSL firmware might be worth a shot.

In reply to a post by DougM:

Hi lexden16,

I started having a packet loss issue around the same date, and I also use a Fritz!Box 7490. Two engineer visits and no faults could be found; the JSU showed the line was perfect.

I heard about another Fritz user with the same symptoms, and I checked and noticed that Fritz!OS 06.80 was released around that time. I used the DSL option to use the previous DSL formware, and since then my errors have disappeared and my line rate has restored: it took DLM a couple of weeks but I'm back to where I was. I've avoided upgrading to Fritz!OS 06.83 since it doesn't mention and DSL fixes.

You may not have the same issue, but downgrading the DSL firmware might be worth a shot.

The timings don't work for me, and I did back revert to 6.52 for a day or so but the graphs still looked awful. I have now been on 6.83 for a few days so I am not convinced that it is linked to the firmware. I may be wrong so I have sent AVM a full diagnostics test.

In reply to a post by AndyHCZ:

Not quite sure how this showed up as a cease. A cease should only happen if the underling WLR/MPF is cancelled or the FTTC service is stopped.

It should have been a migration or possibly, a C/S-VLAN move (I am not sure how BTWholesale deal with this). If that's the case, it would have shown on the systems and stopped anyone else from placing an order.

It's classed a cease of WBMC as far as BTWholesale are concerned and a provide of GEA even though it is, in reality, a WBMC>>GEA migration I had the same message appear in the Wholesale checker myself last year, but it doesn't give a message when there is a GEA >>> wbmc migration

In reply to a post by tommy45:

In reply to a post by AndyHCZ:

Not quite sure how this showed up as a cease. A cease should only happen if the underling WLR/MPF is cancelled or the FTTC service is stopped.

It should have been a migration or possibly, a C/S-VLAN move (I am not sure how BTWholesale deal with this). If that's the case, it would have shown on the systems and stopped anyone else from placing an order.

It's classed a cease of WBMC as far as BTWholesale are concerned and a provide of GEA even though it is, in reality, a WBMC>>GEA migration I had the same message appear in the Wholesale checker myself last year, but it doesn't give a message when there is a GEA >>> wbmc migration

Customer confusion/concern could be eliminated if the ISP sent out an explanatory e-mail rather than just post the migration as an order in the customer's portal. Looking at the portal is not something that I do that often given that my weekly router e-mail has all the connection information that I need.

In reply to a post by lexden16:

The timings don't work for me, and I did back revert to 6.52 for a day or so but the graphs still looked awful. I have now been on 6.83 for a few days so I am not convinced that it is linked to the firmware. I may be wrong so I have sent AVM a full diagnostics test.

I had horrendous problems a few months ago following a Fritz.box update. This was on a 7390, but the principle is the same.

Normally the connection is through the Openreach-supplied Huawei modem, but after the firmware update, the Fritz.box couldn't establish a connection through it. It would work (after a fashion) using its own modem, but only slowly and unreliably.

After much faffing about and downgrading/upgrading firmware, I found the solution. I backed up all the settings of the Fritz.box, then did a factory reset, and restored all but the settings related to the internet connection. Problem solved!

I very strongly advise you to try the same. Seems that update following update can leave the Fritz.box in a bit of an iffy state.