ISP blocklists

I had an issue with my Vodafone broadband this week, where Vodafone did a DNS based man-in-the-middle hijack of a couple of sites and put the traffic through an Internet Watch Foundation proxy server. I posted about that on this forum here.

Now, I understand why they were doing it, but what I don't understand is why other ISPs weren't affected. The sources on the Wikipedia article about IWF aren't great, but it says that 95% of UK ISPs use IWF's blacklist, and that domains end up on it only if the whole domain is for hosting illegal stuff. Is that info just outdated, now that most big sites have moved to HTTPS?

Are people with other ISPs seeing sites intermittently blocked, or is this a Vodafone speciality?

It depends on how the IWF block is implemented. All large ISPs block IWF-listed content but the actual method of the blocking is down to the individual ISP. From what you say, it looks like Vodafone's blocking system needs improvement.

And there is no IWF-run proxy server, every IWF proxy server (if that's the blocking method the ISP selected) will be operated by the individual ISPs.

Edited by Oliver341 (Sat 03-Jun-17 13:49:34)

Are other ISPs using better methods? Google doesn't find reports from others, so perhaps the difference is only in how quickly sites are added and removed?

With HTTPS, the only way I can see selective URL blocking work without blocking entire domains, is if the ISPs asked customers (or the government asked OS or browser manufacturers) to install a Big Brother root certificate that then allows them to impersonate any site anywhere. When Wikipedia was on the blocklist for a week almost a decade ago, ISPs took no responsibility on their poor implementations, and IWF got the blame.

In reply to a post by MarcuT:

Are other ISPs using better methods?

Presumably, since we don't see issues reported much anymore. There are periodic complaints about image hosting sites such as imgur which are probably caused by IWF filters, but in general the incidence is low.

I can proxy https pages but without becoming the end point, in which case there is no MITM interception so to speak.

The dns servers redirect IWF listed names to a proxy server (possibly done even on the routing level for ip addresses to get round people using 3rd party dns servers), then the proxy server will either serve proxied content (probably a block page) or simply pass on the traffic depending on the url been requested.

So when I use my own squid server it can redirect https traffic and I get no certificate issues.

So basically a proxy in the middle can intercept the url, and if it passes it on, then the original host at the other end will receive it untampered, it will then send back the reply to the proxy server which passes it back to the end user, the proxy server will only see encrypted content tho and not know what it is, but it can still pass it back to the user.

From all this a block page I would expect to generate a certificate warning, but not any traffic that comes from the actual proper provider of the domain.

I am not entirely sure at the time of this post, if the proxy server only sees the domain or the full url of the https request as I think but not sure that https requests start only with the domain name and the url is requested after the handshake, but not entirely sure.

That's the thing: in HTTPS/TLS the full URL is sent only after the handshake. By then everything in the session is sent encrypted. The handshake includes the host name in plain text, required for the server to be able to pick a matching certificate with virtual hosts or SNI, but that's it. Apart from GCHQ level backdoors, it's not possible to decrypt HTTPS sessions without the private key of the endpoint, so an ISP proxy has no way of peeking into the communication. Otherwise there'd be no added security with using end-to-end HTTPS. Once the handshake fails, the browser doesn't even get to the point of requesting the full URL, and that's where the idea of filtering an encrypted protocol ends before it even got started.

Squid works with HTTPS for proxying, but it does it in blind mode and can make choices only based on the host name and IP. It can't get into any of the HTTPS headers to treat requests selectively. Wireshark is a good tool for figuring this out, and Squid's logs as well. Testing tools for tampering HTTPS traffic include an installation step of placing a Certificate Authority certificate on the client, which for testing purposes may or may not be the same machine as the server. Squid supports faking certificates as well, but the man-in-the-middle certificate needs to be added on the client machine the same way. Once that's done, you won't see anything different in the browser, unless you dig into the certificate and look at the certificate path.

Having ISPs suggest that users ignore security warnings goes completely against every other security recommendation banks etc are making, of checking the info in the browser's address bar. Customers of those ISPs instantly become more likely phishing victims for scammers.

In reply to a post by MarcuT:

Having ISPs suggest that users ignore security warnings goes completely against every other security recommendation banks etc are making, of checking the info in the browser's address bar. Customers of those ISPs instantly become more likely phishing victims for scammers.

Safiri, which is the default on iPad, doesn't help by not showing the URL unless you tap on the displayed name.

Are there ISPs that don't use the IWF list at all? I've read conflicting posts that using it is mandatory, or only if the ISP has over X customers, or that it's entirely voluntary for all ISPs.

The Open Rights Group has a list of ISPs per number of sites being blocked, but it probably isn't fully up to date or doesn't detect all types of blocking. Plusnet for instance is shown as having 0 blocks in place, but with the image they're trying to make for themselves in their ad campaigns, that can't be true. Must depend on the definition of 'block'.

AAISP

Uno.