Is this latency normal?

This is a new BT FTTP connection, 500Mb/s. The speed is being delivered when tested via an ethernet connection, but the latency shown in the BQM screenshot (link below) worries me. Elsewhere I have Virgin FTTP connection which shows virtually no yellow. I am using a Draytek 2926, not the BT device.

Thoughts much appreciated.

https://www.thinkbroadband.com/broadband/monitoring/...

Swap the Draytek for the BT router and test again over 24 hours.

If I remember correctly, the TBB yellow line shows the maximum latency out of 100 pings.

You could get intermittently high latency if you are filling the link with traffic, because the pings have to sit behind other packets in a queue. Whilst filling 500Mbps download 24x7 may be unlikely, filling the outbound 70Mbps is possible, for example if you are backing up terabytes of data to a cloud service. Can you get graphs of interface counter stats from your Draytek which show how much traffic you are actually sending in and out?

You really want to be looking at latency when the line is as idle as possible. You could try turning off the wifi and unplugging all your LAN clients for an hour or two, and seeing if the line goes flat.

Remember that these are pings *from* TBB *to* your router's WAN interface. Many routers deprioritise responding to pings, and some will intentionally drop pings when more than a few per second arrive. However (a) if the router isn't busy there's no particular reason for it to wait 200ms, (b) there's no apparent packet loss.

Other ideas:

You can try sending 100 pings outbound to some well-known endpoint, e.g. 8.8.8.8, and checking the maximum latency on the response. Do this several times.

Have you tried the BT-supplied router? This would rule out an issue with the Draytek responding to pings.

I take your point but sadly not so simple, I lose necessary VPN connections, load balancing etc etc. I will see if I can do it over the weekend when SWMBO isn't using it for Teams during the day!

Thanks for the advice, I will try to set up a realistic test and revert if I am still seeing the issue. During the day the line is used pretty intensively for videoconferencing (although there's enough bandwidth for that not to be a factor, surely) but the issue seems to persist overnight when there is nothing going on.

In reply to a post by nieldm:

but the issue seems to persist overnight when there is nothing going on.

Swap to the BT router when everyone has gone to bed and swap back first thing in the morning. At least you will have a few hours snapshot during what is as you say an otherwise quiet time.

Videoconferencing won't be anywhere near the limit of your link.

Really you want to be doing SNMP monitoring of the WAN interface on the Draytek, so you can generate graphs of traffic in/out. If you're using Windows I don't know what's the best/easiest tool for this, but maybe someone else can suggest.

(I use prometheus+grafana but that's a sledgehammer to crack this nut)

Then you can be sure whether the link really is idle overnight or there's some background traffic going on that you weren't aware of.

The Draytek's can be a bottleneck with PPPOE due to firewall usage.

https://www.draytek.co.uk/products/business/vigor-29...

Performance
Firewall: Up to 400Mb/s
IPSec VPN: Up to 80Mb/s
SSL VPN: Up to 45Mb/s

So you need a router capable of 500Mbs over PPPOE to make full use of your connection. No idea if this has an effect on pings. My 2925 has a restriction of 300Mbs over the firewall.

It's not just PPPoE, the VPN performance is also a bottleneck which may explain the poor BQM.

...but Speedtest is reporting the full 500Mb/s. I posted about this, that the router was reporting speeds higher than the quoted firewall throughput, on the Draytek forum last week. It seems quite a few people are seeing the same thing, so it seems Draytek are underreporting the throughput. That said, if you look here - https://www.draytek.com/products/vigor2926/#specs - (the US site), the throughput figures for the 2926 are different and more in line with what I am seeing.

...but this isn't the issue.

Something badly wrong there, my BQMs from N.Ireland in sig

Pings are low priority traffic so if you are using a device at the limits of the hardware then it would be no surprise to see an increase in ping reply times.

That doesn't look like what's happening with your connection unless you have been fully throttling the line 24/7.
The high ping peaks on the BQM are consistent and you wouldn't expect that when the connection is idle.

As suggested by others your 1st troubleshooting action should be trying an alternative router.
Until you rule out your own hardware you shouldn't contact the ISP.

You could try QoS - https://www.draytek.co.uk/archive/kb/kb_qos.html

In reply to a post by smouty:

You could try QoS - https://www.draytek.co.uk/archive/kb/kb_qos.html

If the router's CPU is overburdened then that will make things worse.

In reply to a post by candlerb:

In reply to a post by smouty:

You could try QoS - https://www.draytek.co.uk/archive/kb/kb_qos.html

If the router's CPU is overburdened then that will make things worse.

Possibly but it also limits the bandwidth so could even out.

I don't think the router is overburdened, CPU usage is showing at <50%, often in single figures.

I will replace the Draytek with the BT hub when I get a chance and report back.

By all means swap the BT router to prove that there is no underlying issue with the new service.

Bear in mind that unless you notice real performance impacts, then a lovely flat BQM graph may be a bit of vanity chasing, especially if your router doesn't look to be working flat out and is otherwise passing traffic normally at full rated speed and there are no impacts you can quantify or measure (dropping packets etc.). It may simply be deprioritising ping.

For example here is todays BQM for the gateway router at my service provider. It doesn't look particularly pretty, but its perfectly 100% normal. Its just the router doesn't give a hoot about ping - nor should it - it's got more important things to do ... like route our traffic

https://www.thinkbroadband.com/broadband/monitoring/...

Edited by Pheasant (Thu 11-Nov-21 17:55:52)

"..a lovely flat BQM graph may be a bit of vanity chasing"

This is, of course, a fair point. The router certainly isn't flat out and I just looked at the router traffic graph for the past 6 hours, which shows virtually no activity whatsoever. I am not particularly worried about it, hence the rather casual "is this normal?" question....as I said in the original post, everything is working fine and at full speed.

Rather off-topic, but if anyone is interested in Drayteks, there was a very good webinar on Tuesday about their routers, the speeds, hardware acceleration, etc - https://www.draytek.co.uk/our-solutions/videos/drayt...

"..a lovely flat BQM graph may be a bit of vanity chasing"

It is to a point.
Another test you can do is this one - http://www.dslreports.com/speedtest

It is a bit more 'real world' as bufferbloat can be an issue.

In reply to a post by nieldm:

I will replace the Draytek with the BT hub when I get a chance and report back.

Just take note that the BT Hub doesn't support replying to pings therefore no BQM support.
Putting the Draytek behind the BT Hub to reply to the BQM is not exactly ruling out the Draytek.

Ideally you want another router that replies to the BQM to be run for an hour.

Having connected the Homehub to attempt to check the latency, all I see is the red graph of death. It seem the device isn't usable with BQM? I can ping the IP address from the command line....

In reply to a post by nieldm:

Having connected the Homehub to attempt to check the latency, all I see is the red graph of death. It seem the device isn't usable with BQM? I can ping the IP address from the command line....

Did you read the post above yours?

Are you pinging the external IP and where from?

Edited by smouty (Sun 14-Nov-21 09:55:08)

In reply to a post by nieldm:

Having connected the Homehub to attempt to check the latency, all I see is the red graph of death. It seem the device isn't usable with BQM? I can ping the IP address from the command line....

Did you read the post above yours?

In reply to a post by nieldm:

Having connected the Homehub to attempt to check the latency, all I see is the red graph of death. It seem the device isn't usable with BQM? I can ping the IP address from the command line....

Oops, apologies not having one, completely forget that the BT S'Hubs don't do ICMP response on the WAN side...

I had not seen the post above mine, I only noticed them when I read yours asking if I had... my apologies.

Using DSLReports speedtest things don't look too bad (with a Route Policcy ensuring all traffic goes via BT) - http://www.dslreports.com/speedtest/70040780

The BQM is still ugly though and I don't have another router I can test with, sadly. As someone noted higher up, contacting BT when I am not using a HH and haven't eliminated the Draytek isn't going to be a nourishing experience so it feels like a bit of a catch 22.

Having got my teeth into this it now feels like I need to get to the bottom of things....does anyone have any other suggestions?

You could always go routerless: plug your PC or laptop directly into the ONT and configure it as a PPPoE client. Ensure that incoming pings are enabled in your software firewall.

+1. For testing purposes.

Edit - just don’t know how performant Windows based firewall would be - suppose could always load Ubuntu etc from bootable USB stick and do it that way.

Edited by Pheasant (Tue 16-Nov-21 15:56:35)

In reply to a post by nieldm:

Using DSLReports speedtest things don't look too bad (with a Route Policcy ensuring all traffic goes via BT) - http://www.dslreports.com/speedtest/70040780

Having got my teeth into this it now feels like I need to get to the bottom of things....does anyone have any other suggestions?

Why is the speed so low? I thought you were on a 500mbit line.

Do you have a spare PC you can stick a dual nic in and run OPNSense/pfsense?

Well spotted. I am indeed on a 500Mb/s line and I get that when I connect via ethernet, that test was done on wifi. It's an old house with thick walls but that said, I took the trouble to run ethernet to various points and use Ubiquiti access points, so it's not usually that slow.

You are overestimating my knowledge of these matters, but I will research what OPNSense/pfsense is as I do have a spare laptop (or a Pi?) I could deploy.

The other suggestion, that I use this laptop to bypass the Draytek, can also be done but it will have to be when I am not in danger of a SWMBO explosion....

In reply to a post by smouty:

In reply to a post by nieldm:

Using DSLReports speedtest things don't look too bad (with a Route Policcy ensuring all traffic goes via BT) - http://www.dslreports.com/speedtest/70040780

Having got my teeth into this it now feels like I need to get to the bottom of things....does anyone have any other suggestions?

Why is the speed so low? I thought you were on a 500mbit line.

Do you have a spare PC you can stick a dual nic in and run OPNSense/pfsense?

Are you able to cary out a more local test with that site? Its telling me its using servers in Switzerland - the London Linode servers are offline apparently.

Speed testing to .ch isn't going to be ideal, although testing via Wifi will of course be the biggest potential blocker...

In reply to a post by Pheasant:

Speed testing to .ch isn't going to be ideal, although testing via Wifi will of course be the biggest potential blocker...

I get full speed on BT 80/20 but agree that wifi is the issue here most likely.

I am trying to set up a Zabbix server running on my Qnap NAS and am hoping I could get a little help setting up SNMP monitoring. I am right at the outer limits of my 'skills' here...

I have everything working, information is being displayed in the Zabbix dashboard. I used a template I found online (https://github.com/bitfinity-nl/zbx-draytek) which is great, but I don't think it covers the OIDs I need per this document - https://www.draytek.com/support/knowledge-base/5517.

Am I correct that, in order to test the latency, I need to include OIDs from the ICMP section, 1.3.6.1.2.1.5.x? If so, is it as simple as cutting and pasting the XML and making changes to the tags where required? Or am I missing the point completely?

Help would be greatly appreciated, I have the bit between my teeth and don't want to let go!

1.3.6.1.2.1.5 is the ICMP MIB. It *won't* tell you anything about latency. It just counts the number of ICMP messages sent and received by the router. See https://datatracker.ietf.org/doc/html/rfc4293#sectio...

Measuring latency needs to be done with active probes. Some routers can do this, e.g. Cisco has a feature called ip sla. I don't know if the Draytek has a similar feature, but I think it's unlikely.

You'd probably be better off running some latency-measuring tool directly on your monitoring host.

Edited by candlerb (Thu 18-Nov-21 09:03:25)

In reply to a post by nieldm:

Help would be greatly appreciated, I have the bit between my teeth and don't want to let go!

So you are measuring the result but not treating the cause??
Every test has shown the same symptoms no?

I would be looking to measure CPU usage if possible to eliminate that.

Edited by smouty (Thu 18-Nov-21 08:45:16)

As I said, I am at the outer limits of my knowledge here, hence me asking for some guidance although I am aware many of you have given me more than enough help already!

CPU usage, as far as I can see, isn't an issue at all. It's typically in single figures and I have never seen it above 30%. That said, it is something Zabbix can monitor and graph over time, so I will keep and eye on it.

What do Draytek support say about this? If they’ve de-prioritised ICMP response, then no amount of spare CPU cycles will help.

They're unlikely to *delay* ICMP responses. But the yellow line shows 99th percentile, and I suppose it's possible it's going to sleep for 200ms now and again.

There aren't any power-saving features that are enabled, are there?

A firmware update and a full reset back to scratch may resolve matters.

That's not a bad idea, will give that a try.

So i had a chance to do precisely this, a PPPOE connection to my Macbook. The results here:

https://www.thinkbroadband.com/broadband/monitoring/...

...are interesting. For some reason I couldn't keep it stable hence the red parts but the results on the right show the times it was connected to the laptop. It seems the Draytek may well be at 'fault' in some way. I asked on the Draytek forum and the people there seem to think that ICMP de-prioritisation is a myth, but as they said, they don't have the source code.

I have just done a factory reset and firmware reload....so let's see what happens.

Nothing wrong with your FTTP connection - its 100% your kit (router)

Anyhow. see how the FW update /factory reset goes.

So there is progress. The results after the reset and firmware reload can be seen from 7pm last night. This morning I remembered I had an EdgerouterX in a drawer that I had tried to use but was too complicated for a simpleton like me. Firing it up and connecting the BT line gave me the following results, with the ERX results starting at 11.30am.

https://www.thinkbroadband.com/broadband/monitoring/...

So yes, 100% it's the Draytek, but the question is why and what am I doing to make it perform so badly IF they arent throttling ICMP. As I think I said, you can use the QoS functionality to set ICMP to Class1, which I tried with no difference. I have also excluded it from being a cable issue given the results on my laptop and the ERX.

I will ask again on the Draytek forum and post anything interesting here.

I see no evidence that the Draytek is "performing badly".

What you see is that when the line is idle, 1 in 100 ping responses is being delayed by ~200ms. This is quite possibly because the router has gone into some power-saving mode due to lack of anything interesting to do.

Unless you have evidence that it's actually impacting your use of the Internet, then just ignore it.

Permit me to rephrase. I am interested in understand why the Draytek is reporting latency in the way it is.

What does bother me is that I am only getting ca. 300Mb/s even when connected directly to the router or even directly to my laptop via PPPOE, but that's off topic and something I can work on with BT with the HH connected.

In reply to a post by nieldm:

Permit me to rephrase. I am interested in understand why the Draytek is reporting latency in the way it is.

What does bother me is that I am only getting ca. 300Mb/s even when connected directly to the router or even directly to my laptop via PPPOE, but that's off topic and something I can work on with BT with the HH connected.

That generally tends to point to an issue with the laptop - either the NIC/adapter is not up to snuff or there is something in the OS/AV/software stack that's holding things back. 500 Mbps should be easily achievable up and down (I know you have an asymmetric service) on a relatively modern machine.

In reply to a post by nieldm:

I am interested in understand why the Draytek is reporting latency in the way it is.

Fair enough: that's probably one to take up with Draytek support, given that you now have plenty of evidence to support your observations.

In reply to a post by nieldm:

What does bother me is that I am only getting ca. 300Mb/s even when connected directly to the router or even directly to my laptop via PPPOE, but that's off topic and something I can work on with BT with the HH connected.

Indeed; however that could limitations of your end client device under the load of a browser-based speedtest. Trying a different client device, and/or booting your device from an Ubuntu Live USB stick, will help to rule that in our out.

This is relatively recent, I have had reliable 500+ when directly connected to this machine in the not too distant past.

Sounds slightly odd. Typically you shouldn’t see that sort of variation in headline speed on a capable setup.

If it’s a windows box, some third party and VPN software can fudge up the networking stack and cause these sorts of issues.

What's your Firewall Syslog showing?

I've had a quick look to see if anyone else has monitoring, who has a Draytek, however I can't find anything.
I'm trying to see if this is expected behaviour or if something else is going on.

"What's your Firewall Syslog showing?"

There's nothing in the syslog originating from the firewall per the "Firewall" filter.

Are you running VPNs *through* the Draytek or terminating tunnels on the Draytek?

From what I can see, it seems the router is working as intended.

What would I do?

If everything is working as intended in the office and no one is complaining, I would leave it alone and concentrate on making money .

I would also read this page and make sure hardware acceleration was working, leaving it on auto. Make sure Data Flow Monitor, QoS and Bandwidth Limit is disabled. This will give you the best chance of making use of the 500Mb.
https://www.draytek.com/support/knowledge-base/5308