Gamma Telecoms - own their own cabs / DSLAMS?

I've got 3 sites on an industrial estate, using Zen, BT and Gamma for ISPs. All FTTC.

Zen and BT lines experience 10+% packet loss. The Gamma line is fine.

Do Gamma Telecoms have their own cables / cabs / DLSAMs / points of presence?

Do Gamma Telecoms have their own cables / cabs / DLSAMs / points of presence?

Nope.
The DSLAM is very very unlikely to be the cause of the packet loss. It would take a very high number of Error Seconds (DSL errors) to cause packet loss. The line management would turn on interleaving and add Forward Error Correction between the DSLAM and your modem if there were that many Error Seconds.

There can be a number of reasons for the packet loss and it can often be pinned down. It could be an issue with BT Wholesale who provide the backhaul for BT and sometimes Zen.
Gamma use Talktalk Business for backhaul so might be why that line isn't seeing any packet loss.

What modem/routers do each of the connections use?
Do you get the packet loss all the time, to any site/server/IP?

I recommend setting up a BQM for each of the connections. It's a great way to monitor a line every second of the day and clearly shows packet loss.

https://www.thinkbroadband.com/broadband/monitoring/...

Sometimes the Zen control panel tells you who the backhaul provider is but that info is gone for a lot of customers now. They use a mix of BT Wholesale, Talktalk Business or on some exchanges their own network called Plexus. They might be willing/able to change the backhaul provider.

Edited by j0hn83 (Fri 18-Mar-22 03:03:17)

Thanks j0hn83, I was curious, because Gamma Telecoms are described as having their own access network, but I was under the impression that pretty much all the holes and poles and openreach.

Regarding the packet loss, it's constant both in and out, with a very noticeable improvement between ~3-7AM.

quality monitor screengrab: https://photos.app.goo.gl/9sdXswbSQzG4mqSb6
(Both sites show very, close matching on the graphs)

Two sites on Zen and BT, FTTC, IPs 88.97.13.173 & 81.133.250.29, sites are about 150 meters apart. Both have suffered 5-20% packet loss for around 6 weeks. Copper status and VDSL statistics are great, great speeds, no drops, few errors etc. Many many Pair Quality Tests have been conducted.

Lift and shift has been carried out for want of a better idea.

Running traceroute and pathping, the loss appears to occur on the last hop to the premises - ie from outside you can ping all the hops except the fault premises without loss

Engineers have also ruled out CPE, hot VP, electrical interference (REIN, SHINE

20% packet loss is insanely high. What are Zen saying is the cause?

In reply to a post by BlueComp:

Running traceroute and pathping, the loss appears to occur on the last hop to the premises - ie from outside you can ping all the hops except the fault premises without loss

The last hop you speak of is the last L3 hop on the public routed Internet, usually a PPPoE access concentrator which will be in a datacentre somewhere, many physical links away from the customer premises. So the problem could be anywhere in that backhaul network or the DSLAM itself.

As your Gamma line is fine, it could indicate a problem with BT Wholesale backhaul, but it could also be a line card issue in the DSLAM. You say a lift and shift was attempted to rule out individual port issues, but if the new port was still on the same card it would explain this behaviour.

Obviously you need to escalate with your ISPs, but I'd be looking at a) trying to find out which backhaul Zen are using and b) asking some friendly neighbouring businesses if they have IT people you can get to check their premises.

If Zen are using their own backhaul and the problem is appearing on both Zen and BT lines, then it's likely a line card... or something really obscure such as Zen buying their backhaul from BT and it sharing infrastructure with the path that BTw is backhauling on. But I'd expect those paths to be monitored and this fault to have been picked up on. A line card fault could fly under the radar.

The same thought process applies if you can identify packet loss on a neighbour using TalkTalk backhaul, for example, or identify a clean BT Wholesale line from the same DSLAM.

I know, 10-20% loss is nuts, but the crazy thing is that the users barely notice! They're using VOIP and not complaining.... beats me how that works...

Nobody so far has a clue what the cause is, but I'm only now a month in starting to get past level 1 support / Pair Quality Testers...

We figured that a problem with the DSLAM would show up in errors on the DSL, ie lost data frames, CRC errors, high ES / SES, rather than packet loss?

Bluecomp

They should start to notice above 10% loss. Below that you don't. I looked into it around 2010 when BT first started trialling VOIP.

The Codec makes a difference but using g711 the packets are so small and there are so many of them the losing a regular sequence hardly affects the call quality. Losing a whole block is noticeable so 4 packets in a row you get a blip in transmission. Losing every 10th packet you do not notice at all.

In reply to a post by BlueComp:

We figured that a problem with the DSLAM would show up in errors on the DSL, ie lost data frames, CRC errors, high ES / SES, rather than packet loss?

Disclaimer: I'm not familiar with BT's equipment... however, this isn't necessarily true.

All the error types you mention would show up on your modems if the problem was on the DSL side of the line card. If the problem is on the data path between the line card and the DSLAM backplane/CPU etc, that wouldn't show up as the DSL connection is still functioning normally.

"Wholesale have completed their investigation of the network from the exchange back to Wholesale kit and confirmed no faults but agree they can see the packet loss. They have handed over the investigation to Openreach to do an end to end investigation of their network now."