|
|
|
I have a fibre leased line (100Mbit over 1Gbit bearer) provided by ITS.
The gateway is a DrayTek 2927, connected via Gigabit Ethernet to a Cisco C1117-4P (managed by ITS), connected via fibre to an ADVA FSP 150-GE102Pro.
When I run a speed test, it always gives me ~94Mbit up and down - should I be getting the full 100Mbit? When I run a speed test on my 500/70 FTTP service at home, I get 500Mbit down or more and about 68Mbit up.
We are based in North Wales, but our Microsoft 365 Admin Center reckons we're based in Hatton (south of Birmingham), whereas an IP WHOIS Lookup on our IP range has an address of Hatton in Chester (CH3).
Everything works fine, apart from downloading anything from OneDrive/SharePoint. Whether I'm downloading via the OneDrive app on Windows or via the browser, the download rate is always 600kb/s to 800kb/s. I am also getting the same speeds using a standalone Linux server (on-premise) that's connected to the same network as our Windows PCs. The upload rate is about 5mb/s, it's just the download rate that's suffering.
Microsoft so far have suggested it's a problem with our network, but I've troubleshooted this issue with everything turned off: Firewalls, QoS, etc.
It doesn't matter if I use 1.1.1.1 DNS or the ISP DNS servers either.
Any advice would be greatly appreciated.
|
|
|
When you send or receive data over an internet connection there are packet overheads. The overheads are additional bits which have to be transmitted and so use some of the band width. 94Mbps will be about the maximum usable data that you can send or receive on a 100Mbps connection.
Michael Chare
|
|
|
|
OK, but then why does my FTTP service at home achieve the full rate on the speed test?
Also, the MTU on the DrayTek WAN is set to 1500. Would decreasing it resolve the issue with OneDrive/SharePoint?
|
|
Register (or login) on our website and you will not see this ad.
|
|
|
|
Set the IP address on your laptop to the same public IP and gateway as you have entered into the DrayTek, plug it into the Cisco managed router in place of the DrayTek device, and test again.
|
|
|
OK, but then why does my FTTP service at home achieve the full rate on the speed test?
Possibly because your home FTTP provider accounts for packet overhead when calculating your actual allowed bandwidth vs advertised bandwidth, whereas your leased line provider doesn't.
|
|
|
|
A few minutes ago, I tried downloading a large file from our OneDrive for Business service, and it downloaded at 10MB/s - the full download rate of the connection. A few seconds later, I tried downloading the same file, and the download rate did not exceed 800KB/s.
To rule out it being a Windows thing, I tried downloading the same file via one of our Linux servers, which is connected to the same leased line connection as my computer, only to see the same slow download rate.
So I tried to VPN into my home router and download the file via that connection instead, and it downloaded at the full 10MB/s rate. I tried to download the file via the VPN a few more times, and every time it downloaded at the full rate.
Disconnecting from the VPN and downloading the same file via the leased line, and it is still downloading at up to 800KB/s.
Does this suggest the IP traffic originating from our leased line is being throttled?
|
|
|
What service do you use? Is it a hosted MS 365 type of thing?
If so you will be connecting to your nearest entry point to the MS Azure world.
I cannot help specifically but can only report many issues getting a stable connection for MS Teams going via ZScaler.
OPNSense on Topton J4125 - SWISH Fibre 900
PiHole/AdGuard home - Unifi for Wifi
|
|
|
|
Could you just do the test that involves bypassing the DrayTek router entirely? There's still too many things in play to make any sort of guess where the issue might be.
|
|
|
|
if the op has the chance - maybe connect to the ont directly .....
|
|
|
|
Re getting pegged at 94 Mbps, suggests a 100BaseT connection with a standard MTU of 1500 and Ethernet frame size of 1518 bytes.
Ideally the WAN connection and LAN connections should be set to larger jumbo frames (typically an MTU of 9000 bytes) to getting better maximum throughput close to 99% of the port raw bitrate, due to less framing overheads.
|
|
|
if the op has the chance - maybe connect to the ont directly .....
It’s a DIA connection via a managed router - rather than FTTP with ONT, could still go direct into the Cisco LAN connection though.
|
|
|
if the op has the chance - maybe connect to the ont directly .....
It’s a DIA connection via a managed router - rather than FTTP with ONT, could still go direct into the Cisco LAN connection though.
ahh, i've not seen a leased line termination. Anyhoo In my head i'd rule out hardware first and then software (any form of bloatware).
|
|
|
On your live network traffic, find the ip it is downloading from, check if its the same for VPN and direct, and do some diagnostics, traceroutes etc.
The internet is complicated, its likely CDN/routing related.
|
|
|
|
Sorry I've not replied lately. I haven't had chance to try bypassing the internal network and connecting directly into the Cisco, however I wanted to share something I just noticed.
When I download a 300MB ZIP file from OneDrive, it downloads at ~700Kb/s. But part way through, if I try downloading the same file or another file from OneDrive, the second download starts off much slower, such as 50Kb/s, and the first download's download speed reduces by whatever speed the second download is downloading at.
So for example, if download #1 is downloading at 700Kb/s and I start download #2, and download #2 is downloading at 50Kb/s, then download #1 will now be downloading at 650Kb/s.
It's as if the bandwidth is capped and every download is sharing one lot of bandwidth.
I just ran another test, downloading the same file via my normal browser, and the same file via an InPrivate browser session. The result was the 2 downloads were downloading at the same speed, each at ~700Kb/s. But as soon as I initiated another download in either session, the download speed of this additional download was deducted from the download speed of whichever session I initiated it within.
So it looks like there's a bandwidth restriction occurring on a session-level.
Could this still be due to my networking hardware or is it likely to be a Microsoft thing?
|
|
|
|
Most upload/download sites have limits. A commentator that I follow got himself a 10G link as an experiment. His upload to YouTube were severely below that. OTOH, he could to several other things at the same time. The link was fine, just that many servers seem to have limits.
Some sites are "wide open" but OneDrive and GoogleDrive do not seem to be one of those. In my experience, of course.
|
|
|
|
As you're paying for a managed leased line, what do ITS say about this?
|
|
|
I'm guessing it is managed. ITS are the carrier but Abzorb are our ISP. ITS provided the Cisco router, whilst we provide our own router (DrayTek 2927). However, the Cisco (C1117-4P) is running version 17.5.1a and the latest version available is 17.11.1:
https://www.cisco.com/c/en/us/support/ios-nx-os-soft...
Could the Cisco firmware be to blame? I guess I'll have to connect a computer directly into the Cisco and see what happens with OneDrive. If I do this and the same issue occurs, then I take it that means it's not our equipment but something between the Cisco and Microsoft?
|
|
|
|
If you run a 1G ethernet link with 1500 byte frames, you will get 942Mbps of usable "payload" throughput (excluding the TCP, IP and ethernet headers).
Similarly, on a 100M ethernet link you will get 94Mbps of usable "payload" throughput.
In your case, there are several ethernet connections involved: your laptop into your Draytek, Draytek into Cisco, and Cisco into ADVA. If any of these is set to 100M instead of 1G then you'll see the 94M limit. Sometimes this can happen with a faulty ethernet cable or broken pin in a connector: 100M ethernet only needs four of the 8 pins, but gigabit needs all 8.
However, the fact that you have an ADVA NTE rather than an ONT implies that it's likely an Openreach fibre tail, and it could be that the ADVA port has been hard-set to 100Mbps. This means that ITS are unable to over-provision you slightly, as otherwise I'd expect them to do.
None of this explains your poor Sharepoint speeds though. If the only problem is when communicating with Microsoft and no other sites, then the problem is likely an issue with network traffic routing. Talk to your ISP and work through it with them, and they will escalate to ITS if necessary.
Connecting directly to the Cisco will at least rule out the Draytek as the problem.
|
|
|
|
Did you resolve this problem?
I'm having a very similar issue also with an ITS provided leased line with the same CPE. Probably a very similar setup albeit at 1Gbps.
|
|
|
Not resolved yet, although I finally had the chance to test by connecting my laptop directly into the Cisco and running some speed tests, which were surprisingly poor:
https://www.speedtest.net/my-result/d/e364fa74-6172-...
https://www.speedtest.net/my-result/d/a1a50038-b930-...
Whilst connected to the Cisco, I also tried download a file from OneDrive and still only managed a download rate of about 700KB/s.
Surely this suggests our internal company network is not to blame and the ISP needs to investigate further? I reached out to the Cisco community and asked about the firmware version of our Cisco router (17.5.1a) and they suggested updating it may resolve the issue, but the ISP has said this is a tested version on their network and that they take caution when selecting firmware to avoid introducing bugs. But that firmware version is an "ED" type (early deployment), so does that mean it's a bit like a beta / release candidate?
Edited by davidenco (Tue 04-Jul-23 18:40:07)
|
|
|
|
Resolve directly with the ISP. The service should reliably deliver it’s stated throughout to most well known speed test servers (e.g. Ookla). If not then it’s their remit to resolve. You don’t need to play CCiE on this.
|
|
|
|
You do need to play CCiE if the provider denies all knowledge of any problem and insists that its on your side of the network. It becomes a case of having to prove otherwise and this isn't aways straightforward. Speedtests are great at masking nuanced problems which in my case were only evidence when analysing single TCP connections. I had to get handy with iperf and wireshark to highlight retransmissions arising from dropped or delayed packets but only in one direction and from certain destinations / routing hops. An SME without skilled network admins would never be able to do that and would be stuck between wondering if its the ISP or the cloud service provider at fault, or indeed their own problem LAN side.
Improvements have been made by them making an alternation to the routing of my connection, but I'm still seeing a certain amount of UDP packet loss in one direction which is hampering use of DTLS based services.
I suggest the OP reaches out to me by message as I can share support ticket details.
|
|
|
|
By all means work collaboratively with your supplier to resolve. However there is no need (nor is it a great idea) to effectively fudge about with a router that is under the control of the service provider.
I’ve currently got a packet loss issue on a gig EAD that I’m working to resolve too. You’re not alone in this.
|
|
|
Not sure if referring to my post - I haven't touched the ISP's router.
They have already upgraded my CPE to a higher Cisco firmware version as part of their troubleshooting but made no difference. As I mentioned, for the most part, the TCP throughput problems I had were resolved by them making a routing alteration on the core network.
I have no doubts whatsoever that the EAD circuit, CPE and everything my side of the rack is working perfectly well. It's an onward back-haul issue only the ISP can resolve.
I can't confirm to davidenco what Cisco version they upgraded to. I can't login to the device for obvious reasons, I have no creds and I don't intend to start plugging a console cable into it. It broadcast its previous version by CDP/LLDP but it stopped doing so since they updated it.
Edited by neildmb (Tue 11-Jul-23 18:59:49)
|
|
|
Did you resolve this problem?
I'm having a very similar issue also with an ITS provided leased line with the same CPE. Probably a very similar setup albeit at 1Gbps.
After a week of further troubleshooting on my part, I discovered I can resolve the issue by connecting to a VPN over the Leased Line after which I am then able to download at the full rate.
I reported this back to our ISP who relayed it to ITS, and I have just been advised by the ISP that ITS has admitted there is a problem with their core server and that this has been the case for several weeks and that we're not the only customer affected.
ITS are working on it and I'm told the issue should be resolved tomorrow at some point.
|
|
|
The route change hasn't resolved our issue.
ITS are now claiming the reason why we are getting such poor download speeds is because they do not have direct peering with Microsoft, and therefore the traffic is going over the Internet between our gateway and Microsoft. Those are their words.
Does that sound right?
Edited by davidenco (Thu 13-Jul-23 16:15:21)
|
|
|
|
If they believe that's the problem then there's nothing stopping them peering with Microsoft. They are, after all, a business connectivity provider and Microsoft are more than 50% of the cloud productivity market.
|
|
|
|
I'm more inclined to believe that that's complete BS.
You're talking about a 100Mbps circuit not 100Gbps.
If that can't be back-hauled reliably the service isn't fit for purpose, certainly not when sold as business class uncontended etc etc.
I use systems on networks all over the world with a variety of connections some colo some leased lines and I can pull e.g. an ISO off Microsoft's website at sustained wirespeed in every case except my own leased line where it flounders at around 100-200Mbps.
Very convenient of an ISP to sit there saying its off network / out of their hands when the rest of the net is working just fine.,
|
|
|
|
Tell them it’s not fit for purpose / doesn’t meet the minimum throughout SLA and that you’re therefore cancelling the contract if they don’t completely resolve within 14 days.
That usually causes them to sit up and pay attention.
If not go shopping. They’re not the only provider over Openreach tails at the end of the day.
|
|
|
The route change hasn't resolved our issue.
ITS are now claiming the reason why we are getting such poor download speeds is because they do not have direct peering with Microsoft, and therefore the traffic is going over the Internet between our gateway and Microsoft. Those are their words.
Does that sound right?
They claim they have an open peering policy. Ask them to set up a peering arrangement with Microsoft.
For reference, I just tried a few quick tests from https://www.azurespeed.com. Latency , download and upload are all very respectable:
Latency to Azure UK South (London) is about the same as other major sites. So for me situated today in London it's showing as 0-1ms average. UK West (Cardiff) is similar (1-2ms average). Very quick.
Download and upload speeds are running at between 40% and 85% of the connection bandwidth. Snapshot.
Get ITS on the case.
|
|
|
Get ITS on the case.
That's blissfully optimistic thinking.
I'm not far off just lighting up another pair of fibres in the nice cable Openreach installed.
Got any alternate provider recommendations?
|
|
|
|
Just a quick update, our ISP has now said they cannot add direct peering with Microsoft, but apparently no one else has reported as having an issue, which seems odd as that would suggest it's our end, except I've proven the issue occurs from the Cisco router onwards with all our kit disconnected. But just because something isn't reported doesn't mean it's not happening elsewhere.
Unfortunately, I have noticed a lot of other Microsoft services we're using over our leased line are affected, such as downloading Microsoft Teams direct from Microsoft servers. Also, downloading the connectivity testing tool (23MB file size) from connectivity.office.com downloads at a maximum of 80KB/s, whereas on my Zen connection at home it downloads almost instantly.
Our other office runs on Zen FTTC and is connected to our main office (ITS leased line) via VPN, and by routing all Microsoft traffic from our main office to the other office over the VPN gives us faster download rates than routing traffic via ITS, albeit with a much higher latency.
Annoyingly, when we had the leased line installed a year ago, Openreach said there were no plans to rollout FTTP round here until December 2026. Well, they've been working on our area for the past few months and I spoke to one engineer last week who said our area will have it before the end of the year! Bloody typical!
|
|
|
|
Random brain dump moment/alarm...
I absolutely detest when providers brush off problems because 'nobody else has reported them'. Almost nobody reports issues unless its completely broken so its a flawed argument to begin with. What matters here is that you perceive there to be a problem and it is their job to disprove you; not pass the buck.
Can you replicate the issue using an Azure VM and iperf3? If you can then you can create packet captures at both ends of a short test and compare these against the results of packet captures created from 'good' end points that you control and the issue may stick out like a saw thumb. You could also complete low rate udp tests to figure out if there is packet loss along the path causing the issue. If you cant replicate the issue other than from the service in question then you could still create packet captures of say the connectivity testing tool which might show you where the fault lies.
|
|
|
Can you visit the following URL:
https://www.speedguide.net/analyzer.php
What's the MTU and MSS reported? This should be run from a machine having this issue.
|
|
|
|
You say the connectivity tool was slow to download, what result did it give when you ran it? Maybe this is something relatively simple like your location resolving to somewhere thousands of miles away so you're using the wrong edge locations.
|
Pages in this thread: 
Print Thread
