Three is finally working properly OMFG.

After months of frustrating random timeouts/retries when loading pages, Three seems to have fixed it (at least from where I sit).

Holy mother of dog.

Dunno if this is network-wide, or confined to certain masts: They sent me an SMS a few days ago, saying they'd be working on my mast (I connect to cell 3143426 / eNB 12279). Things were pretty ropey over the weekend, but from Monday morning onwards, I've not noticed a single timeout/retry loading a web page.

One thing I did notice—and I think this is a change—is the true MTU has dropped to 1400. So if you have, say, 1440 or 1500 set, large packets will fragment. Best to drop it down to 1400: on my B535 it's at /html/content.html#mobileconnection or you can set it from the API tab in LTE H-Monitor.

(this is using the default APN of 3internet)

I don't want to jinx it but it appears alright for me currently too. Snappy responses of most websites i visit, including thinkbroadband. Before i'd have to wait 30 plus seconds or constantly refresh until it popped up. Could not view forums on threaded view without waiting either. Was just too time consuming. I just hope its permanent...Please be permanent

Is it really that bad, 30 seconds to load a basic website?!

Surely 3 customers would be complaining a hell of a lot if this was universal?

It was, yes. Happened with ispreview too. Didn't happen with all websites though. Probably the reason why everyone using three isn't complaining about it. I don't think their average customer is browsing thinkbroadband or ispreview. Most social media sites seem to be fine, or were for me. Youtube, netflix and amazon video were all working for me when it was happening as well. If they didn't i'm sure it would make national news, or fixed long before it made the news. I wish i could be more specific to the sites i had problems with but it become so normal that i stopped paying attention.

Though i did have problems with some games. Would try to connect to game servers and would fail multiple times before connecting. Sometimes having to quit the application to rinse and repeat until i connected. Maybe some folks just put it down to gaming on a 4G connection and didn't bother complaining. And complaining to Three was useless. They'd allow you to cancel the contract, as long as they got their router back

Just to add. I have no idea if this was universal. I know many others were/are facing the same issue. Perhaps its an issue for a small number of customers and that's the reason its taking/took so long to fix. No idea what the cause was/is

Edited by deleted (Wed 09-Sep-20 01:17:23)

Yeah, I found it would happen randomly across any website. It looked like the initial TCP or TLS negotiation sometimes got messed up. The Chrome tab would sit there for about 20 seconds, time out and immediately automatically retry, which worked.

One likely guess is that something was fragmenting then re-assembling packets at a layer below IP. But it occasionally got it wrong, causing this problem. The fact that the MTU seems to have changed at the same time as the problem going away is … suspicious.

Edited by richi (Wed 09-Sep-20 12:59:29)

In reply to a post by richi:

Yeah, I found it would happen randomly across any website. It looked like the initial TCP or TLS negotiation sometimes got messed up. The Chrome tab would sit there for about 20 seconds, time out and immediately automatically retry, which worked.

Same here when I was still on Three, it impacted sites pretty much randomly (though with sites that make more individual TCP/TLS connections being more likely to be affected).

You can get around the page hanging issue by using cloudflare warp beta on your pc.

Interestingly, all the content on these forums is delivered through Cloudfare.

In reply to a post by RobertoS:

Interestingly, all the content on these forums is delivered through Cloudfare.

I gather Cloudflare have some really high percentage of internet sites, over 50%, now protected on their platform.

The Warp service is about DNS encryption, there are other choices. E.g. you can use DNS-over-HTTPS in many browsers, and some routers (e.g. my ASUS) offer DNS over TLS. If Three's problem was broken DNS, then bypassing the ISP makes sense, which is what Cloudflare Warp would do.

Services such as Quad 9 also support DNS over TLS and DNS-over-HTTPS.

(I'm a fan of DNS over TLS, and dislike DNS over HTTPS, as DNS should be handled by the operating system or local network hardware, not the application layer).

Cloudflare Warp is a full VPN service, all your traffic is tunnelled through Cloudflare

I prefer other VPNs myself.

Three’s problem isn’t DNS, I always use my own DNS, not the ISP’s.

Cloudflare carry 5-10% of web traffic. Akamai are the largest CDN with ~30%

I have set 1400MTU on my Mikrotik SXT and 3 is snappy again!
Lets hope it stays like this..

In reply to a post by Appy:

Cloudflare carry 5-10% of web traffic. Akamai are the largest CDN with ~30%

Don't cloudflare do a lot more than Akamai though in terms of protection (e.g. DDos) which the other CDNs don't offer (yet?). I thought they were bigger than that!

I think the volume of traffic on Akamai may be down to Microsoft and Apple using them. Plus Sophos. Though apparently they have over 1.4m customers according to Wikipedia.

Cloudfare apparently claims to support over 12m domains/websites.

It looks to come down to whether we compare throughput or sites.

In reply to a post by kams19:

I have set 1400MTU on my Mikrotik SXT and 3 is snappy again!
Lets hope it stays like this.

Great news; thanks for the confirmation.

I must say, when certain websites stutter, I do wonder if the problem has come back, but that behaviour is notably different (Chrome doesn't auto-reload the site via a timeout page).

In reply to a post by andynormancx:

Three’s problem isn’t DNS, I always use my own DNS, not the ISP’s.

Agreed: The timeout issue was not a DNS problem. Hostnames were resolving fine, but the random failures manifested as timeouts—either when making TCP connections or negotiating TLS.

In reply to a post by buggerlugs:

You can get around the page hanging issue by using cloudflare warp beta on your pc.

That's nice, but no use on my Chromebook, Amazon Fire, Harmony Hub, YouView DVR, etc. And I'm not putting on any of our phones, given the reports of overheating and sucking batteries dry.

Anyway, it looks like the problem's fixed, so no need to cover it up any longer.

In reply to a post by jchamier:

(I'm a fan of DNS over TLS, and dislike DNS over HTTPS, as DNS should be handled by the operating system or local network hardware, not the application layer).

Right on, brother. I use Simple DNSCrypt as a UI to dnscrypt-proxy, with DoH disabled.

I've just phoned Three to un-cancel my new 5G broadband & so far, testing seems good. I've not been able to replicate the horrible browsing experience that I was seeing the other day. I'll continue to test and update here in a few days.

Great news. Are you able to configure your router's MTU?

Yes, I set that to 1400. I'll experiment over the weekend with Wireshark to see what higher MTU's do.

My B311 is still on 1500, and I've had the problem for weeks. It has cleared up for me as well, just before I was going to try 1400.

In reply to a post by RobertoS:

My B311 is still on 1500, and I've had the problem for weeks. It has cleared up for me as well, just before I was going to try 1400.

Great news.

To be clear: I'm not saying the problem clears up by changing the router's MTU. It's that the network's true MTU changed at the same time as the problem was fixed. So for best results, set your router to that.

Detail: Previously, the default MTU of 1440 didn't cause frags on a ping test (ping -f -l 1412 richi.uk), and neither did 1500 (-l 1472). Now the maximum ping payload size is 1372, meaning the MTU is 1400.

In reply to a post by richi:

To be clear: I'm not saying the problem clears up by changing the router's MTU. It's that the network's true MTU changed at the same time as the problem was fixed. So for best results, set your router to that.

Interesting, the advice on DSLreports and other sites for over 10 years has always said to never change the router, always change the PC. Otherwise you just cause the router more work, as it has to buffer packets, split them into the smaller size, and retransmit. Many home routers are just not up to this task, so you lose speed.

That advice was before the Windows XP era, when Path MTU Discovery became the norm, with automatic adjusting MTU for each connection.

We used to have lots of tools for PCs to change MTU, but now the majority of home usage is from streaming media on TV boxes, tablets, and mobile phones, there is no way to change the MTU on these devices. Path MTU is doing its job.

Why change the router?

The B311 default is 1500, and if you go to change it you get a strong warning you should only do so if really necessary.

From what has gone on, I think it likely that a rogue MTU got into the core network somewhere, possibly in multiple places, and it took a while to find and sort it.

In reply to a post by jchamier:

streaming media on TV boxes

Ironically streaming is one area that the Three network problems doesn't really impact. Because when you're streaming you aren't typically repeatedly starting lots of TCP/TLS sessions you don't get the same stalls you do when just browsing the web.

Through all my issues with Three streaming just kept working ok.

Not for me it didn't.

Could be, but worse than that: I'm guessing there was a bug where sometimes the frags wouldn't get combined properly, the payload was corrupted but the TCP checksum gets recalculated against the corrupt data.

In reply to a post by jchamier:

Interesting, the advice on DSLreports and other sites for over 10 years has always said to never change the router, always change the PC. Otherwise you just cause the router more work

Yeah but by changing the router's setting, you're matching the local MTU to the maximum PMTU, which should make it easier on the cheap plastic router, not harder.

Or I might be overthinking it.

By all means hardcode your Windows or macOS stack as well, but that's probably just gilding the lily (and as you say, you can't do it on most of your other devices at home).

In reply to a post by RobertoS:

Not for me it didn't.

Me neither. YouTube is noticeably smoother and more consistent now.

I guess we're not dealing with a single issue then.

With my problems it was very consistent over several months:

- random websites needing lots of TCP connections would often stall
- streaming connections were basically never impacted (there was never any buffering on several hours of video streaming each evening)
- VPN connections were never impacted

Edited by andynormancx (Sat 12-Sep-20 17:25:44)

In reply to a post by richi:

Yeah but by changing the router's setting, you're matching the local MTU to the maximum PMTU, which should make it easier on the cheap plastic router, not harder.

Or I might be overthinking it.

I don’t think there is any point changing the router. The PMTU may be higher or lower as it is the combination of networks (not just the ISP) that your traffic routes through, to get to the service you are after.

By all means hardcode your Windows or macOS stack as well, but that's probably just gilding the lily (and as you say, you can't do it on most of your other devices at home).

I also don’t see the benefit changing the router. PMTU will handle the networks.

Performance is still good. Websites load quickly and no sign of the previous stalling issues. One odd thing though - three days ago I was getting solid 80Mbps upload however, over the weekend, 18Mbps is the max I can get.

In reply to a post by andynormancx:

I guess we're not dealing with a single issue then.

Might not be as cut'n'dried as this though. Some streaming services open more than one parallel connection—YouTube for example.

In reply to a post by jchamier:

The PMTU may be higher or lower as it is the combination of networks (not just the ISP) that your traffic routes through, to get to the service you are after.

I'm confused. How can it be higher?

In reply to a post by richi:

How can it be higher?

Higher than your test, as your target will be a different network / host.

In reply to a post by jchamier:

In reply to a post by richi:

How can it be higher?

Higher than your test, as your target will be a different network / host.

But on the assumption that 1400 is now the limit for DF packets within Three's core, PMTU Discovery surely will cap at 1400, no?

In reply to a post by richi:

But on the assumption that 1400 is now the limit for DF packets within Three's core, PMTU Discovery surely will cap at 1400, no?

It should yes, but if that is temporary and becomes 1470, then why limit your modem? Let the system work it out.

In the 1990s and early 2000s we used to set the MTU on our PCs and laptops to 1400 because the PMTU didn't exist in Win98 and similar. There was always strong advice to never change the router, but to set the MTU on the PC/laptop.

In reply to a post by richi:

But on the assumption that 1400 is now the limit for DF packets within Three's core, PMTU Discovery surely will cap at 1400, no?

Why are you making that assumption?

in the immortal words of Neil Armstrong
"That's one small step for man, one giant leap for mankind."

In reply to a post by jchamier:

but if that is temporary and becomes 1470, then why limit your modem?

Fair point. I note that the Huawei equipment supplied by Three is pre-set to 1440.

Edited by richi (Mon 14-Sep-20 12:55:19)

In reply to a post by RobertoS:

Why are you making that assumption?

I do get the point you're making, but I have tested ICMP DF with several well-connected endpoints.

Edited by richi (Mon 14-Sep-20 12:55:51)

In reply to a post by popeye123:

in the immortal words of Neil Armstrong
"That's one small step for man, one giant leap for mankind."

ikr! How many months has it been since the first reports of the problem? Six at least.

In reply to a post by richi:

In reply to a post by jchamier:

but if that is temporary and becomes 1470, then why limit your modem?

Fair point. I note that the Huawei equipment supplied by Three is pre-set to 1440.

As I said, mine is set to 1500 by Three, who have full updating access so they would change it if they thought that was needed.

just wondering if anyone else has had a similar experience?

I had months of three not working very well - slow page loads and timeouts even though the upload and download speeds were great.

Last week I saw a massive improvement as many people on here reported.

One week on I've noticed that although page loads are still fine, streaming appears to time out after 20-30 minutes. I've experienced this on sonos (using amazon music, tune in and Spotify) and have also had the same happen on a laptop using streamed radio.

The connection is fine but streaming seems to get interrupted by the network.

In reply to a post by LondonCalling:

just wondering if anyone else has had a similar experience?

No problems here with:
- YouTube.com on Chrome/Windows 10
- YouTube app on gen 1 FireTV
- Radioplayer app on Android 10 phone
- BBC Sounds app on Android 10 phone (radio live or replay)
- Google Play Music on Windows
- Amazon Prime Music on Windows
- Facebook video on Chrome/Windows 10 and in Android 10 app

Despite other forumites' advice that it's pointless, you could try changing the router's MTU to 1400—as a diagnostic step, if nothing else.

Looks like Three got the hint of whats wrong from your post
IN any case, three subscribers are happy if the network remains like this!

In reply to a post by RobertoS:

As I said, mine is set to 1500 by Three, who have full updating access so they would change it if they thought that was needed.

Curious. My B535 was supplied in mid-July on a 30-day contract, set to 1440.

I can confirm 1500 did not report ICMP fail with DF when I tested before Monday (Sept 7), but did fail when tested after (including just now).

Edited by richi (Mon 14-Sep-20 15:07:23)

My 5G Huawei is also 1440

In reply to a post by kams19:

Looks like Three got the hint of whats wrong from your post
IN any case, three subscribers are happy if the network remains like this!

I know plenty of people were hammering on them since before I started having problems back in May.

I did spend some considerable time and effort sending them results of tests, as requested by the techs. It became clear that things were much improved when using a Huawei router, rather than the TP-Link I was using (but the random stalls weren't fixed). On my todo list is to dust off my TP-Link and see if whatever they've changed has fixed it for that hardware too.

I know, but even streaming that uses multiple streams doesn’t make dozens of TCP connections in a fraction of a second like loading a complex (but typical) webpage.

I still see this issue on 4G, but not with 3G, which makes me thing they still need to make this change throughout the core. It's not a cell capacity issue either, as 4G speed tests easily achieve 50-60mbps.

3's mixed bag of vendors for the 3,4 adn 5G RAN might have something to do how long it took them to identify where this issue was.

Location: Essex.

Edited by deleted (Tue 15-Sep-20 09:02:20)

In reply to a post by Anonymous1337:

I still see this issue on 4G, but not with 3G

That's consistent with what I was seeing, too. When the problem went away for me, it was just after the signal from my mast went up/down a lot over the weekend. And Three sent me a text to say "we're working on your local mast." That's why I assume the fix needs to be rolled out mast by mast (or perhaps by area).

"ping -f -l 1412 richi.uk" now does not cause frags. Perhaps they made some other fix?

In reply to a post by Appy:

"ping -f -l 1412 richi.uk" now does not cause frags. Perhaps they made some other fix?

Not here. Still 1400.

Did you set that on your router? Did you increase it again before you tested?

In reply to a post by Appy:

Did you set that on your router? Did you increase it again before you tested?

Thanks for asking. Yes. And reset the stack. And rebooted just in case.

Edited by richi (Wed 16-Sep-20 13:54:13)

Hmm behaviour has changed again. I can no longer ping your host "richi.uk with anything bigger than 1372 but I can ping 8.8.8.8 with 1472. I wonder if it depends on the route that gets picked?

Not here. I did some traceroutes and my first identifiable hob is to 172.17.70.69, but that only likes 1400.

In reply to a post by richi:

Not here. I did some traceroutes and my first identifiable hob is to 172.17.70.69, but that only likes 1400.

Of course what MTU the router's management card itself responds on, can be completely different to the MTUs it passes through itself.

and today...

Pinging richi.uk [216.239.32.21] with 1472 bytes of data:
Reply from 216.239.32.21: bytes=68 (sent 1472) time=30ms TTL=113

How odd

And tonight it seems to have suffered a regression. I'm getting frequent issues with web pages loading, same as before

I think we can safely say that the title of this post is highly mistaken. 12 months in, I can assure you from personal experience with Three that it'll never work properly.

Same here. Started looking into 4g alternatives a few hours ago and noticed that some websites were stuttering. The exact same way as they were before. Great!

A core network upgrade overnight perhaps?

I hope so. I've not been home most of the day or evening so unsure when today it started. Funnily enough, thinkbroadband is working fine, whereas before it was not. I'll check back in tomorrow or a few days from now if its sorted.

I've just done one of my late night indoor speed tests. Clocked more than 20Mbps higher than ever before on both Ookla and thinkbroadband.

I did have some very short hiccups on Sunday and Monday. Sudden loss of signal for very short periods.

I too have had a few hiccoughs over recent days. It's really hard to tell because they're so random.

Weird event at 7:54 am when the router reconnected—to a cell whose beam is pointing in the opposite direction, then reconnected again to the correct cell, 45 seconds later. Almost as if they're reconfiguring the gear again.

PMTU to well-known sites still 1400.

I've also experienced some hiccups as well - two evenings in a row the download speed went from c280Mbps down to being so slow that FaceTime video dropped out. Last night the download speed was between 6 and 20Mbps for an hour or so.

I've also noticed that slow page loads have come back again - it's intermittent but still enough that I'll likely cancel my Three subscription in the next month or so. Ultimately if Three can't even maintain good network performance on their newly upgraded 5G sites it's probably only going to get worse as more 5G devices are used.

In reply to a post by richi:

Weird event at 7:54 am when the router reconnected—to a cell whose beam is pointing in the opposite direction, then reconnected again to the correct cell, 45 seconds later. Almost as if they're reconfiguring the gear again.

Eight hours on and I've had no problems loading any sites.

Still bad today. I can replicate from the command line ...

If I repeatedly open SSL connections, I eventually get this:-

openssl s_client -servername forums.thinkbroadband.com -connect forums.thinkbroadband.com:443
CONNECTED(00000003)

A tcpdump shows the connection has been established and the initial TLS request from me to website acknowledged by the far end at a tcp level but then no further traffic gets passed. It then times out.

I wonder if this someone to do with Three content filter? When I started my contract 2 weeks ago, there was no content filtering and then a few days days ago it started filtering the "adult" sites so I contacted them and got them to turn it off. Since then the stalling issue has returned.

I've taken a few more tcpdumps of various sites. They all behave the same...

1) 3 way handshake works fine
2) client sends Hello, far end ACKs the tcp packet
3) <no more ssl traffic>
4) far end sends FIN exactly 127 seconds later

This is the exact same story for many websites that I've tested. Akamai, Cloudflare,etc all the same. They are unlikely to behave in exactly the same way unless there's something in between us and them that's the common factor - e.g. a transparent filtering proxy

In reply to a post by Appy:

e.g. a transparent filtering proxy

Always possible, and I think Smarty uses one, we didn't think Three used one.

I can provoke a hung SSL connection to news.tweaknews.eu on port 443 but I can't provoke it on port 563 using the same method to the same server, and almost certainly the same application, just different ports. There's got to be something in between filtering traffic.

news.tweaknews.eu seems to be down

https://downforeveryoneorjustme.com/news.tweaknews.eu

But tweaknews.eu works fine for me. (On Three).

Edit: Actually https://tweaknews.eu

Edited by RobertoS (Wed 23-Sep-20 19:16:23)

It's up and running, but it's an nntp service not a website

Post deleted by RobertoS

Post deleted by jchamier

In reply to a post by Appy:

It's up and running, but it's an nntp service not a website

Strange to run a secure NNTP through 443, normally services use 563, or just overlay TLS on 119.

Having used an ISP for years that used transparent proxies (and then gave up), I know that anything over 443 that isn't HTTP protocol can inadvertently get blocked.

If you are using Smarty, try a Three SIM. If you are on Three, then ditch them and try Vodafone, O2 or EE instead.

They only support nntp over 443 for "compatibility" for when your ISP blocks 563. However, it gives us a unique opportunity to test 443 vs 563.

In reply to a post by Appy:

They only support nntp over 443 for "compatibility" for when your ISP blocks 563. However, it gives us a unique opportunity to test 443 vs 563.

Its not necessarily a good test, as transparent proxies are common in corporates.

I've run OpenVPN in HTTPS mode through a transparent proxy before now

Deffo no corporate proxy here

I have deleted my previous post, as it was pointed out to me by a good friend that my recommendation suggested I was ignorant that nntp was a protocol just like the more ones that I'm familiar with.

I did realise that but didn't think that through deeply enough. In particular, I was ignorant of its purpose and implementation.

My apologies.

However I do recall many years ago some ISPs blocking usenet feeds. AIUI for the same reason as parental controls were introduced.

Might it be possible that when they inexplicably set parental filters on your connection they also set up a similar block on nntp access?

In reply to a post by RobertoS:

However I do recall many years ago some ISPs blocking usenet feeds. AIUI for the same reason as parental controls were introduced.

Many ISPs shut down their own usenet service, but NNTP is not only used for usenet. There are plenty of private NNTP servers used for discussion. Many ISPs stopped supporting usenet as the binary groups were mostly being used for piracy and terabytes of data an hour were being posted.

Might it be possible that when they inexplicably set parental filters on your connection they also set up a similar block on nntp access?

They've not blocked NNTP, they're stopped NNTP over the secure web port (443) working. I gather NNTP works over the default unencrypted port 119.

You're mixing up NNTP with a test the poster is trying to determine if the ISP is using a transparent proxy that is faulty (or overloaded)

You're mixing up NNTP with a test the poster is trying to determine if the ISP is using a transparent proxy that is faulty (or overloaded)

Yes exactly. Forget NNTP - it works fine. I'm simply opening a secure connection on port 563 & port 443 to the same server running the same software. Port 563 succeeds every time and port 443 fails sometimes, hence my deduction that something within Three is interfering with port 443 traffic. If it were just this one server, then I'd assume there was a problem with this one server but it seems to affect any secure site I try.

In reply to a post by Appy:

Yes exactly. Forget NNTP - it works fine. I'm simply opening a secure connection on port 563 & port 443 to the same server running the same software. Port 563 succeeds every time and port 443 fails sometimes, hence my deduction that something within Three is interfering with port 443 traffic. If it were just this one server, then I'd assume there was a problem with this one server but it seems to affect any secure site I try.

In theory if you do that to an HTTPS page (e.g. https://www.google.com) you should get an alert from the browser, as you can't recreate the certificate from the original page. Essentially a "Man-in-the-middle" attack. Corporates can do this as they manage the computer, but ISP's normally can't, or they get lots of helpdesk tickets.

They're not actually "accepting" the connection and pretending to be the far end because, as you say, that would be very visible. I think all they're doing is inspecting the IP address that you're connecting to and also the "server name indication" in the SSL Hello request. I guess this service is overloaded which is why we're seeing problems.

I've asked Three to turn off the parental controls but this service is probably still inspecting my traffic. There'll still be some sites that I'm not allowed to connect to. https://en.wikipedia.org/wiki/Web_blocking_in_the_Un...

@Appy & @jchamier

Maybe I should keep out of the way!

It's not just parental controls: There are some sites blocked regardless. The system appears to work by snooping on DNS requests, then proxying/blocking connections to the IP addresses of suspicious domains.

Solution: Use DoH or DNSCrypt. You also should use the 3internet APN (so if you're a Smarty user, you're SoL).

Edited by richi (Thu 24-Sep-20 12:50:28)

In reply to a post by RobertoS:

Maybe I should keep out of the way!

It got quite low level. Not many people dig into their ISP at this depth

I'm not convinced that it's got anything to do with DNS.

I've been running a test all day which opens a SSL connection on port 443 vs port 563 on news.tweaknews.eu and then times how long it takes. I've then plotted the times on a chart. You'd expect to see the same data but port 443 has lots more times where it stalls & times out (at 127 seconds). Note, the stalling problem does not seem to be an issue today in general usage, unlike yesterday where it was very noticeable and very intrusive.

https://www.wheep.co.uk/443v563-240920.PNG

Edited by Appy (Thu 24-Sep-20 15:59:50)

Those 443 times look like powers of 2 or if your 127 is consistent (2ⁿ - 1) throughout. Is the halfway one 63 seconds?

Even if true, I'm not sure how relevant it is, but might be to people who know what they're talking about.

Edited by RobertoS (Thu 24-Sep-20 18:26:13)

In reply to a post by jchamier:

In reply to a post by RobertoS:

Maybe I should keep out of the way!

It got quite low level. Not many people dig into their ISP at this depth

That is a kind interpretation.

Yes, some are 63 seconds. Here's some lines from the log file. First column is unix timestamp, 2nd is SSL connection time. Once again, only on port 443 not port 563

1600950488.41534,63.3658871650696,
1600951155.57224,63.3058650493622,
1600954998.48472,63.344251871109,
1600962240.69723,63.3089048862457,
1600965814.82116,63.3274939060211,
1600968861.71342,63.3499038219452,

In reply to a post by Appy:

I'm not convinced that it's got anything to do with DNS.

It hasn't. I wasn't talking about the TLS negotiation problem. I was replying the point about website filtering.

The DNS snooping is how they decide which TCP connections to block or at least reroute via a proxy. Because they can't possibly proxy everything!

The block/grey lists are based on hostnames or URLs, not IP addresses (but they might cache the associated IPs). So they snoop/proxy the DNS packets (even if you don't use Three's DNS) and intercept the connection to decide whether to block it.

The design comes from the IWF child-porn filter, which ISPs solemnly promised would never suffer mission-creep and become a way of blocking other content, uh-uh, no way, pinky swear, Scout's honour (I was in one of those meetings when they said that).

Then the entertainment industry got a load of injunctions, forcing ISPs to ban certain "unlawful" torrent sites, so guess what happened. And then, drip-drip-drip, we are where we are today.

In reply to a post by richi:

And then, drip-drip-drip, we are where we are today.

Unless you use A&A:
https://www.aa.net.uk/broadband/

We do not have, in our network, any equipment installed to filter access to any part of the public Internet for our customers as a whole. We will give 12 months notice if we ever add any such filtering.

This claim relates to the passing or normal unicast IPv4 and IPv6 packets to and from the public Internet based on the appropriate standards and RFCs. This means packets can be dropped because a link is full or there is a technical fault, or because they are malformed in some way, or clearly spoofed or incorrect source addresses. In the case of some sort of attack we can take steps to manage that. Only packets actually addressed to your IP addresses will get to you, and similarly only packets from you that are from your IP addresses will get to the Internet (BCP38). We don't control the rest of the Internet and so cannot bypass corporate or national firewalls or filtering outside of our network. However, we aim to deal with peers and carriers that have similarly open policies where possible.

Hmm seems a little far fetched to me, but I'll explore the possibility. I have DoH set up on my network but not using it at the moment. I'll reconfigure to see if that solves the issue. Personally, I think they are trying to inspect everything on port 443 and port 80 which is why we have issues.

In reply to a post by Appy:

Hmm seems a little far fetched to me, but I'll explore the possibility. I have DoH set up on my network but not using it at the moment. I'll reconfigure to see if that solves the issue. Personally, I think they are trying to inspect everything on port 443 and port 80 which is why we have issues.

Are you on smarty or three?

In reply to a post by Appy:

Hmm seems a little far fetched to me, but I'll explore the possibility. I have DoH set up on my network but not using it at the moment. I'll reconfigure to see if that solves the issue. Personally, I think they are trying to inspect everything on port 443 and port 80 which is why we have issues.

Again: I am NOT suggesting that DoH/DNScrypt will fix the TLS timeout problem. I was replying to the question about not using the filter.

Indeed, I know it won't, because I've been using DNScrypt for months.

Three 5G with a H112-370 router. This is a new 5G area outside of London, so this is with Three and not Three Broadband. I know, right? Amazing but apparently Three Broadband is London only & different to Three Broadband outside London. I don't get it either.

Edited by Appy (Thu 24-Sep-20 20:48:26)

In reply to a post by Appy:

Three 5G with a H112-370 router. This is a new 5G area outside of London, so this is with Three and not Three Broadband. I know, right? Amazing but apparently Three Broadband is London only & different to Three Broadband outside London. I don't get it either.

I think Three Broadband (formerly Relish) is now being folded into the main network, which is probably a good thing.

Three advertise they are building "Real 5G" but they seem to have problems implementing it.

Exactly. The decision I need to make over the next 2 weeks is whether to keep Sky Broadband which has been reliable and "just works" even though it's a little slow (60meg) or do I keep Three 5G which is fast (600meg) but has issues with such basic things like loading web pages. This feels like the early days of cable broadband with Blueyonder when I was in the first region that went live. Remember them? That must have been almost 20yrs ago now.

In reply to a post by Appy:

This feels like the early days of cable broadband with Blueyonder when I was in the first region that went live. Remember them? That must have been almost 20yrs ago now.

I do (see .sig) as I was with NTL's cable trial, which went live the year before. 512kbps for £40/month, but they managed 480kbps mostly. Then trial stopped, and they launched full product in summer 2000.

NTL used transparent proxies, Inktomi Traffic Caches, which they hoped would reduce their peering / transit costs. They usually failed and caused no end of complaint, it was why I left cable, and was on FTTC for a long time (as well as moving home). Went back (to VM) last Nov as I needed faster upload.

In reply to a post by RobertoS:

Those 443 times look like powers of 2 or if your 127 is consistent (2ⁿ - 1) throughout.

After running it overnight, I see an interesting spread of delay times.

1, 3, 7, 15, 31, 63, 127

e.g. (2nd column is the delay time)

1601021619.23777,1.19222784042358,
1601021638.14017,3.16593384742737,
1601021626.77635,7.1832971572876,
1600949488.43863,15.1755349636078,
1601021882.33607,31.2504329681396,
1601021485.15224,63.2639911174774,
1601021725.92307,127.290504932404,

How remarkable! I wonder why we see this?

Edited by Appy (Fri 25-Sep-20 09:26:14)

In reply to a post by Appy:

After running it overnight, I see an interesting spread of delay times.
1, 3, 7, 15, 31, 63, 127
…
How remarkable! I wonder why we see this?

Looks like exponential backoff, which might imply it's a side-effect of packet shaping or some other sort of congestion mitigation. Albeit stupidly implemented.

Edited by richi (Fri 25-Sep-20 14:11:38)

Given that most computers work in binary, in "words" of 4; 8; 16; 32 or 64 bits, those numbers all being powers of two, the maximum (unsigned) number in each is what is represented by all the bits being set. The value of 4 bits set is 15. And so on. 127 is the maximum value of 8 bits set.

Data transmission is generally these days, for our purposes, in 8-bit words, commonly called bytes, which is what we all refer to. (In other hardware types there can be many other word and byte sizes).

This is why I latched onto that pattern in your graph, including some of the shorter verticals, which is what you have confirmed. Some process appears to be being triggered as each successive bit of a counter is set, and for some reason that process falls over and sometimes breaks the transmission.

What and why, and whether it is intentional or a programming bug, is what we don't know.

I wonder if someone coded a second for the initial delay, when they meant a millisecond?

In reply to a post by richi:

Looks like exponential backoff, which might imply it's a side-effect of packet shaping or some other sort of congestion mitigation. Albeit stupidly implemented.

It's more likely to be an exponential backoff for some retry - e.g. talking to a database or some other associated service. If the first request fails, try 1 sec later, then 3 secs, etc etc until it gives up at 127 secs.

That would make sense.

In reply to a post by jchamier:

In reply to a post by richi:

But on the assumption that 1400 is now the limit for DF packets within Three's core, PMTU Discovery surely will cap at 1400, no?

It should yes, but if that is temporary and becomes 1470, then why limit your modem? Let the system work it out.

In the 1990s and early 2000s we used to set the MTU on our PCs and laptops to 1400 because the PMTU didn't exist in Win98 and similar. There was always strong advice to never change the router, but to set the MTU on the PC/laptop.

Normal MTU discovery relies on ICMP, since so many devices block ICMP it quite often doesnt work, not a good idea to rely on it.

For this reason some OS's have a new TCP based variant which will work but usually is "disabled" by default. It also only fixes TCP not UDP.

e.g.here is the documentation for the linux tunable that controls it.

tcp_mtu_probing (integer; default: 0; since Linux 2.6.17):

This parameter controls TCP Packetization-Layer Path MTU Discovery. The following values may be assigned to the file:

0 Disabled

1 Disabled by default, enabled when an ICMP black hole detected

2 Always enabled, use initial MSS of tcp_base_mss./quote]

Hi All,

Long time no post to this thread... but can anyone who is STILL on three (like myself), confirm that it appears this issue has finally been fixed?

I use Smarty for my primary internet, and it appears I no longer need to use a VPN. Speeds have actually improved overall and they haven't done any work at my local cell as far as I can see.

https://forums.thinkbroadband.com/mobilebroadband/f/...

Perhaps it is is possible for Three to improve their network... I always thought they just ran it further into the ground.

No its not fixed, still have the issue with hanging webpages periodically. Which always coincide with times of excessive high mast contention. I've had RAN Capacity on my mast for over 12 months, Sub 2Mbps after 6pm daily, reported it around 50 times, promised it'll be addressed too many times to count.

The reality is Three are not interested in fixing any issues.

In 3 months my contract ends and I will not be staying with Three.

No, still not fixed. I'm convinced it's to do with their content filtering. I see varying degrees of latency when connecting to HTTPS websites (port 443). On rare occasions I see HTTPS cert warnings in the browser where the wrong cert has been presented (from some entirely different site!!). All these issues can be bypassed by using a proxy such as Cloudflare WARP, however it's not convenient or even possible to run that on every single device on the Network (such as TV's or Google Home, etc).

Has anyone else noticed this problem (or something very similar) returning in the last week? It isn't affecting all websites but seems to be particularly affecting Google and Youtube of the websites that I regularly visit. Like last time, using a VPN seems to get around the problem.

Yes, perhaps a month ago. But it comes and goes.

Mind you, I find it performs better using the three.co.uk APN, so I only use 3internet when I really need an externally-reachable IP address.

In reply to a post by buggerlugs:

No its not fixed, still have the issue with hanging webpages periodically. Which always coincide with times of excessive high mast contention. I've had RAN Capacity on my mast for over 12 months, Sub 2Mbps after 6pm daily, reported it around 50 times, promised it'll be addressed too many times to count.

The reality is Three are not interested in fixing any issues.

In 3 months my contract ends and I will not be staying with Three.

I'm not convinced its high contention.

The reason I say that is my local tower recently got 5G. Had no problems at all for the first month but over the last couple of weeks I started to get page timeouts and network errors. The strange thing is, during the same period an actual download will do 400-600Mbit, so clearly not a contended mast (though upstream being still 4G definitely can be).

When I was on Three 4G the problem there was CLEARLY contention as I would see packet loss and ping spike to insanely high levels. Then again, 4G reception is very rough in the house as theres several houses and trees between me and the mast, which strangely 5G seems to handle better.

Incidentally I tried the MTU tweak and although at first it did seem to help, right now its just as bad.

Can confirm however that changing the APN makes a HUGE difference, suddenly the problem went away and I hit my highest download speed yet.

Edited by alexatkin (Sun 05-Sep-21 03:24:22)