Truespeed: high ping time to Microsoft Azure server

I play regularly in an online band with a group of friends (we're scattered around the south of England and meet in person a couple of times a year, otherwise it's entirely online). We're using Jamulus, which is free open source software that is optimised for low latency audio transfer using a client-server model. Servers can be set to private or public, with public servers being advertised via a central directory servers.

We use a private cloud Linux server set up on the Microsoft Azure platform - this appears to be physically in the Telehouse complex in London. Most of us are using FTTP of various flavours (I use Giganet over CityFibre, others ISPs over Openreach) and in general we all get single figure ping times to the server. Our guitarist has recently moved over from BT FTTC to Truespeed FTTP - on nearly all tests (eg 8.8.8.8, bbc.co.uk, Ookla) and the public Jamulus servers he gets slightly shorther ping times than with FTTC but on the one server only (the Microsoft one we're playing on) his routing seems completely different and his ping has gone up from 9 to about 35ms. I don't have a traceroute for his path, but he tells us it looks different.

1. Any idea why the routing to one server only would be different? Do ISPs use a different backhaul for certain paths?
2. Is there anything we can do about it? I can't imagine Truespeed being interested in spending much time on this. We've done the obvious router restarts etc.

The provider likely has peering links to a number of Internet peering providers. The path taken will depend on the destination server, what network it is on and where it is located. So, quite possible that different networks will follow different paths from the provider.

Truespeed might be able to optimise the routing but it depends on how flexible they are and whether there are any other better options (the network *should* optimise itself but it could be that they are forcing some destinations down non-optimal paths).

Are you routing via the Microsoft Global Network or via the Internet?

https://learn.microsoft.com/en-us/azure/virtual-netw...

Try swapping it to see if anything improves.

In reply to a post by Stargazer99:

1. Any idea why the routing to one server only would be different? Do ISPs use a different backhaul for certain paths?
2. Is there anything we can do about it? I can't imagine Truespeed being interested in spending much time on this. We've done the obvious router restarts etc.

Yes ISPs have different paths to the same destination. Yes, the routing is different because of the change of ISP - Truespeed are probably sending your guitarist's traffic via Frankfurt or Amsterdam because they don't connect directly with Microsoft and the carrier they're using doesn't connect with Microsoft directly or indirectly in the UK.

No, there isn't much you can do about it. Your guitarist should change ISP. I don't think Truespeed's number of paths to the outside world is abundant with options. He could always ask them about it and see if they've another way to reach the server.

In reply to a post by XGS_Is_On:

In reply to a post by Stargazer99:

1. Any idea why the routing to one server only would be different? Do ISPs use a different backhaul for certain paths?
2. Is there anything we can do about it? I can't imagine Truespeed being interested in spending much time on this. We've done the obvious router restarts etc.

Yes ISPs have different paths to the same destination. Yes, the routing is different because of the change of ISP - Truespeed are probably sending your guitarist's traffic via Frankfurt or Amsterdam because they don't connect directly with Microsoft and the carrier they're using doesn't connect with Microsoft directly or indirectly in the UK.

No, there isn't much you can do about it. Your guitarist should change ISP. I don't think Truespeed's number of paths to the outside world is abundant with options. He could always ask them about it and see if they've another way to reach the server.

Start with giving us a traceroute

But also ask Truespeed - they might well be able to adjust something in the routing.

Truespeed seem quite reliant on transit providers, they don't seem to use either of the two internet exchanges, which any serious ISP would use.

Hopefully they'll add this inexpensive option soon, so they'll have direct access via the exchange to Microsoft and many other providers.

And just as I post about it the problem goes away on its own! Our band had a session yesterday evening, and our guitarist's latency to the Microsoft cloud server has dropped to 5ms without any intervention. I was wondering if this might eventually happen as it was such an outlier. Thanks to everyone for the suggestions.

In this case our guitarist doesn't have a great deal of choice - the only options where he lives are Openreach FTTC, which has been a bit noisy at times, and Truespeed, who have recently connected his street.

Edited by Stargazer99 (Tue 16-May-23 09:19:58)

I don't think Azure will be hosted in Telehouse. They may have a network edge there but I doubt the servers are there due to cost.
What are the routes like from and to each location?

seb

In reply to a post by seb:

I don't think Azure will be hosted in Telehouse. They may have a network edge there but I doubt the servers are there due to cost.
What are the routes like from and to each location?

seb

It’s a good point Amazon, Apple, Google, Microsoft et al don’t exactly advertise where they plop their various mega datacentres.

Here's the route from Truespeed (see linked image - note that the ping time is now back where it should be):
Link to image


My own routing (from Giganet/CityFibre) looks a lot simpler...
Tracing route to jam.********.uk [51.104.x.x]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.lan [192.168.1.1]
2 5 ms 5 ms 5 ms thn-bn-3.giga.net.uk [37.48.224.36]
3 5 ms 5 ms 5 ms 217-168-248-67.m12solutions.net [217.168.248.67]
4 6 ms 6 ms 6 ms lonap01.msn.net [5.57.81.17]
5 7 ms 6 ms 6 ms ae28-0.icr02.lon22.ntwk.msn.net [104.44.239.121]
6 *

Edited by Stargazer99 (Tue 16-May-23 21:16:41)

In reply to a post by Pheasant:

It’s a good point Amazon, Apple, Google, Microsoft et al don’t exactly advertise where they plop their various mega datacentres.

I think it's known within the industry. I certainly know where some other sites are but I don't recall the London ones. I just know Telehouse is more an interconnect site than a hosting one (to the extent we pay a lot more to host in there)

seb

In reply to a post by Stargazer99:

Here's the route from Truespeed (see linked image - note that the ping time is now back where it should be):
Link to image


My own routing (from Giganet/CityFibre) looks a lot simpler...
Tracing route to jam.********.uk [51.104.x.x]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.lan [192.168.1.1]
2 5 ms 5 ms 5 ms thn-bn-3.giga.net.uk [37.48.224.36]
3 5 ms 5 ms 5 ms 217-168-248-67.m12solutions.net [217.168.248.67]
4 6 ms 6 ms 6 ms lonap01.msn.net [5.57.81.17]
5 7 ms 6 ms 6 ms ae28-0.icr02.lon22.ntwk.msn.net [104.44.239.121]
6 *

What sort of latency are you seeing? Also what about jitter? if you ping for a minute what's the average vs worst?
Above looks ok.

I know the latency on VoIP before people notice is quite high - I'm not sure what it's like for interactive music where you have to co-ordinate rather than speak one at a time .. I imagine it's a bit smaller.

The latency on hops on traces is a but unreliable (as routers don't respond to pings with high priority) e.g. from us:


Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. ncuk-gw.core-rs3.thdo.ncuk.net 0.0% 48 0.8 0.7 0.6 0.9 0.0
2. xe-1-3-0-36.edge-rt5.thdo.ncuk.net 0.0% 47 0.6 1.5 0.4 31.5 4.7
3. ae11-11.edge-rt1.thn.ncuk.net 0.0% 47 0.8 0.7 0.4 4.4 0.6
4. lonap01.msn.net 0.0% 47 29.7 3.0 1.3 36.1 6.4
5. ae28-0.icr02.lon22.ntwk.msn.net 0.0% 47 1.7 5.9 1.4 60.0 10.7
6. ???

..the variation for worst is quite high.. you can ping to your server though and see what that looks like..

seb

UK West is out in Newport on the old LG site, UK South is Slough.

In reply to a post by jpm:

UK West is out in Newport on the old LG site, UK South is Slough.

I thought I knew Slough but not if it's Equinix or one of the others out there. I doubt it's Iron Mountain

seb

My ping is very consistent - completely rock solid at 5-6ms for bbc.co.uk and 8.8.8.8 - it wasn't my own broadband that I was concerned about. The guitarist in our online band also gets similar results for bbc.co.uk and 8.8.8.8, again very stable. The problem he was having was a ping of 35ms for the Microsoft Azure server - this seems to have been a temporary glitch in the routing as it's fine now.

Tor playing online music you ideally need <30ms total round trip time including buffering and delays introduced by equipment - this equates to a ping of around 15-18ms or less typically, so the application is pretty demanding. For some more demanding types of music a total delay of <20ms is desirable, so a ping to the server no more than of 5-8ms. This is achievable if you're all in the same general part of the UK with FTTP, and most of our band members have FTTP and just about achieve this.

In reply to a post by Stargazer99:

My ping is very consistent - completely rock solid at 5-6ms for bbc.co.uk and 8.8.8.8 - it wasn't my own broadband that I was concerned about. The guitarist in our online band also gets similar results for bbc.co.uk and 8.8.8.8, again very stable. The problem he was having was a ping of 35ms for the Microsoft Azure server - this seems to have been a temporary glitch in the routing as it's fine now.

Tor playing online music you ideally need <30ms total round trip time including buffering and delays introduced by equipment - this equates to a ping of around 15-18ms or less typically, so the application is pretty demanding. For some more demanding types of music a total delay of <20ms is desirable, so a ping to the server no more than of 5-8ms. This is achievable if you're all in the same general part of the UK with FTTP, and most of our band members have FTTP and just about achieve this.

I think this makes for an interesting case study about why latency matters. You don't happen to have any video/audio recordings of when you had problems?

Routing is dynamic and there are multiple reason sit may change so it's usually not simply down to the provider having an issue but possibly the path to a destination. The larger companies will be very slow to admit problems, usually because staff on the front line just son't know about issues and say there aren't any.