Gamma Telecoms - own their own cabs / DSLAMS?

I've got 3 sites on an industrial estate, using Zen, BT and Gamma for ISPs. All FTTC.

Zen and BT lines experience 10+% packet loss. The Gamma line is fine.

Do Gamma Telecoms have their own cables / cabs / DLSAMs / points of presence?

Do Gamma Telecoms have their own cables / cabs / DLSAMs / points of presence?

Nope.
The DSLAM is very very unlikely to be the cause of the packet loss. It would take a very high number of Error Seconds (DSL errors) to cause packet loss. The line management would turn on interleaving and add Forward Error Correction between the DSLAM and your modem if there were that many Error Seconds.

There can be a number of reasons for the packet loss and it can often be pinned down. It could be an issue with BT Wholesale who provide the backhaul for BT and sometimes Zen.
Gamma use Talktalk Business for backhaul so might be why that line isn't seeing any packet loss.

What modem/routers do each of the connections use?
Do you get the packet loss all the time, to any site/server/IP?

I recommend setting up a BQM for each of the connections. It's a great way to monitor a line every second of the day and clearly shows packet loss.

https://www.thinkbroadband.com/broadband/monitoring/...

Sometimes the Zen control panel tells you who the backhaul provider is but that info is gone for a lot of customers now. They use a mix of BT Wholesale, Talktalk Business or on some exchanges their own network called Plexus. They might be willing/able to change the backhaul provider.

Edited by j0hn83 (Fri 18-Mar-22 03:03:17)

Thanks j0hn83, I was curious, because Gamma Telecoms are described as having their own access network, but I was under the impression that pretty much all the holes and poles and openreach.

Regarding the packet loss, it's constant both in and out, with a very noticeable improvement between ~3-7AM.

quality monitor screengrab: https://photos.app.goo.gl/9sdXswbSQzG4mqSb6
(Both sites show very, close matching on the graphs)

Two sites on Zen and BT, FTTC, IPs 88.97.13.173 & 81.133.250.29, sites are about 150 meters apart. Both have suffered 5-20% packet loss for around 6 weeks. Copper status and VDSL statistics are great, great speeds, no drops, few errors etc. Many many Pair Quality Tests have been conducted.

Lift and shift has been carried out for want of a better idea.

Running traceroute and pathping, the loss appears to occur on the last hop to the premises - ie from outside you can ping all the hops except the fault premises without loss

Engineers have also ruled out CPE, hot VP, electrical interference (REIN, SHINE

20% packet loss is insanely high. What are Zen saying is the cause?

In reply to a post by BlueComp:

Running traceroute and pathping, the loss appears to occur on the last hop to the premises - ie from outside you can ping all the hops except the fault premises without loss

The last hop you speak of is the last L3 hop on the public routed Internet, usually a PPPoE access concentrator which will be in a datacentre somewhere, many physical links away from the customer premises. So the problem could be anywhere in that backhaul network or the DSLAM itself.

As your Gamma line is fine, it could indicate a problem with BT Wholesale backhaul, but it could also be a line card issue in the DSLAM. You say a lift and shift was attempted to rule out individual port issues, but if the new port was still on the same card it would explain this behaviour.

Obviously you need to escalate with your ISPs, but I'd be looking at a) trying to find out which backhaul Zen are using and b) asking some friendly neighbouring businesses if they have IT people you can get to check their premises.

If Zen are using their own backhaul and the problem is appearing on both Zen and BT lines, then it's likely a line card... or something really obscure such as Zen buying their backhaul from BT and it sharing infrastructure with the path that BTw is backhauling on. But I'd expect those paths to be monitored and this fault to have been picked up on. A line card fault could fly under the radar.

The same thought process applies if you can identify packet loss on a neighbour using TalkTalk backhaul, for example, or identify a clean BT Wholesale line from the same DSLAM.

I know, 10-20% loss is nuts, but the crazy thing is that the users barely notice! They're using VOIP and not complaining.... beats me how that works...

Nobody so far has a clue what the cause is, but I'm only now a month in starting to get past level 1 support / Pair Quality Testers...

We figured that a problem with the DSLAM would show up in errors on the DSL, ie lost data frames, CRC errors, high ES / SES, rather than packet loss?

Bluecomp

They should start to notice above 10% loss. Below that you don't. I looked into it around 2010 when BT first started trialling VOIP.

The Codec makes a difference but using g711 the packets are so small and there are so many of them the losing a regular sequence hardly affects the call quality. Losing a whole block is noticeable so 4 packets in a row you get a blip in transmission. Losing every 10th packet you do not notice at all.

In reply to a post by BlueComp:

We figured that a problem with the DSLAM would show up in errors on the DSL, ie lost data frames, CRC errors, high ES / SES, rather than packet loss?

Disclaimer: I'm not familiar with BT's equipment... however, this isn't necessarily true.

All the error types you mention would show up on your modems if the problem was on the DSL side of the line card. If the problem is on the data path between the line card and the DSLAM backplane/CPU etc, that wouldn't show up as the DSL connection is still functioning normally.

"Wholesale have completed their investigation of the network from the exchange back to Wholesale kit and confirmed no faults but agree they can see the packet loss. They have handed over the investigation to Openreach to do an end to end investigation of their network now."

Thanks for the update, sounds like an interesting one.

In reply to a post by ft247:

Thanks for the update, sounds like an interesting one.

It is an interesting fault for sure and I'm learning a lot.

Unfortunately it's impacting one of my users quite badly, poor bloke!

Update on the fault:

Spoke to the two senior openreach engineers who visited today. They make a very convincing case that the issue is not in OpenReach domain - it's not a copper fault, DSL statistics are absolutely fine and successive lift-and-shifts have effectively ruled out a fault in a port or linecard on the DSLAM.

They advise that the only troubleshooting step not carried out is a 'soft cease and reprovide' which will need to be requested by BT.

My feeling is that Wholesale need to look at their network harder.

I think openreach will soon be refusing to visit for this fault as it's a waste of everybody's time.

So in summary.... the saga continues

Essentially the openreach guys said that from the DSLAM there's a fibre link to the exchange which terminates on an openreach 'ADVA' and everything beyond that is out of openreach domain, ie BT Wholesale's problem.

Does that sound about right?

Technically I believe it is Openreach's problem to get the packets out of a layer 2 switch (which may or may not be made by ADVA, or be the same box that the DSLAM fibre terminates on) onto a fibre its way to BT Wholesale's equipment.

But it sounds like you might have had people who knew what they were talking about, at least regarding the operation of the DSLAM - if they've made sure that one of the port shifts was to a different line card and concur that the DSL is fine, that makes it pretty unlikely that the DSLAM is at fault.

The only other thing I'd question would be whether the DSLAM has one fibre link to the exchange or multiple - I have heard anecdotally that busy DSLAMs receive extra gigabit links. I don't know how the traffic is balanced across these links - if it's round-robin then this isn't the cause of the fault, as it would affect every customer, if it's a cruder method there could be something there if one of the optics is failing. But this is less likely than a failing optic between L2S and BT Wholesale, which is my guess at this point. BT Wholesale may have multiple in that exchange, and I believe customer circuits are assigned to specific cablelinks, rather than a LAG being formed.

There's a lot of speculation here - hopefully someone with deeper knowledge of the exchange end can tell us more. But if I were BT Wholesale I'd look at whether your problem circuits all share a cablelink in the exchange.

Is the Zen site on BT Wholesale? Can you try the Gamma PPP credentials out there? Someone who knows more than me could probably say whether BTW customers are segregated from each other and whether this is likely to work or not.

In reply to a post by jpm:

Is the Zen site on BT Wholesale?

This is quite possibly the common thread.

VDSL2 layer / copper loop sounds Iike it all checks out OK. What’s common upstream - possibly cablelink(s) if Zen are using BTW backhaul? Line card on the L2S at the exchange?

There could be a over-utilisation issue on common SVLAN? as well. OP might be a able to request a move to a different BTW SVLAN.

Pheasant

Unlikely to be SVLAN issue as these are easily visible to BTW and the ISP..More likely a card on the L2S at the exchange or the distant end of the link at the Core Node. I spent some time trying to identify packet loss on a link many years back and could only pin it down when the card went completely off and the loss stopped due to switching to the resilient card. Tracking back identified the issue to the exact time the card went OOS.

It is very difficult to track when the packets are just dropped in the switch so are never counted on a link and don't show as lost in any stats. Only way we thought about was pulling 'control' cards in the switch when it was suspected to force a switch to the resilient card.

Any issue like this should be affecting every customer on that BTW switch. card, so there should be lost of people complaining. ..

In reply to a post by kitcat:

Pheasant

Unlikely to be SVLAN issue as these are easily visible to BTW and the ISP

You're having a laugh.
I've been on more hot SVLANs with BT Wholesale than I've had hot dinners.

Providers also can't see BT Wholesales SVLAN utilisation.

Just Google "BT Wholesale hot SVLAN" and you will be able to find days worth of reading that will show just how odd your comment above is.

Edited by j0hn83 (Thu 31-Mar-22 21:35:15)

john83

Zen will have a separate SVLAN for it's contracted capacity and can see it's utilisation. BTR will be in different one. To get a 10% packet loss the VLANs would have to be massively over subscribed on both.

In reply to a post by j0hn83:

Just Google "BT Wholesale hot SVLAN" and you will be able to find days worth of reading that will show just how odd your comment above is.

Tried a search to see if I could learn what youse are on about...

No results found for "BT Wholesale hot SVLAN".
Suggestions:
Make sure all words are spelled correctly.
Try different keywords.
Try more general keywords.
Try fewer keywords.

Did you search with quotes round the phrase? If so try again without.

I got this many results searching that phrase

About 46,300 results (0.36 seconds)

Thanks Dan

In reply to a post by kitcat:

Pheasant

Any issue like this should be affecting every customer on that BTW switch. card, so there should be lots of people complaining. ..

It's a nightmare. Unfortunately I'm not and for reasons won't be listed as a technical contact at either of the other sites I've tested (Zen and Gamma), so I haven't been able to chase up those lines of enquiry.

I suspect that the issue is affecting a lot of customers - I hit another one the second place I asked after I suspected it was a widespread issue. Fact is, it seems that for 95% of customer applications, they simply don't notice if you throw away 5-10% of their data!! One openreach engineer pointed out that if it was a residential area they'd have all the gamers beating them up over it...

https://i.imgur.com/94ZFNFj.png

Does anyone else think it's odd that the loss improves so drastically between 3AM and 7AM? I know the standard answer is 'that's traffic related', but is traffic really that much lower at 3AM than 2AM?

Are the BT/Zen in contract? could they be moved to Gamma?

Thanks
Dan

In reply to a post by BlueComp:

https://i.imgur.com/94ZFNFj.png

Does anyone else think it's odd that the loss improves so drastically between 3AM and 7AM? I know the standard answer is 'that's traffic related', but is traffic really that much lower at 3AM than 2AM?

Problem may only manifest beyond a certain level of traffic / utilisation. As said problem looks like it is the BTW backhaul domain, rather than Openreach / DSL side of the connections.

If other customers aren’t screaming about the losses, then it could be a long wait before BTW put the effort in to trace the root cause of the fault. It could be intermittent and quite difficult to otherwise diagnose.

Simple solution (technically) would be to request a change of backhaul provider. Whether you can sort that at a commercial/contractual level is another matter.

Anyway the upshot was that everyone at BT got bored of talking to me about it and after another 3-4 months the problem suddenly disappeared overnight and never came back. I assume that a piece of kit somewhere in the wholesale network got replaced / rebooted / updated and stopped throwing data away.

After 1 year 8 months???

Edited by Nervous (Tue 12-Dec-23 20:22:39)