Outage in East Sussex

Sky broadband and (incoming & outgoing) calls have been out for the whole of the East Sussex coast since 3pm yesterday: https://servicestatus.sky.com/#/uk/issues - over 20 hours now. It's affected major towns like Hastings and Eastbourne as well as lots of smaller towns.

Apparently someone dug through a single cable/duct.

Perhaps it's time ISPs and infrastructure providers are mandated to provide adequate resiliency to provide fail-over in such situations?

Edited by Oliver341 (Thu 06-Jul-17 11:39:41)

In reply to a post by Oliver341:

Sky broadband and (incoming & outgoing) calls have been out for the whole of the East Sussex coast since 3pm yesterday: https://servicestatus.sky.com/#/uk/issues - over 20 hours now. It's affected major towns like Hastings and Eastbourne as well as lots of smaller towns.

Apparently someone dug through a single cable/duct.

Perhaps it's time ISPs and infrastructure providers are mandated to provide adequate resiliency to provide fail-over in such situations?

Not when people are wanting to pay a fiver a month for unlimited broadband.

Not when people are wanting to pay a fiver a month for unlimited broadband.

+1

And who will be blamed by the press? Almost certainly not the actual culprits.


in my area, BT lost ten of thousands of voice , ADSL and VDSL circuits along with large numbers of fibre links which affected O2, Vodafone, Three, EE and many others. Yes, some had backup routes but they were over loaded ...

Who did the press criticise and where was all the flak directed? yes, at BT and not at Network Rail who could no be bothered to confirm the exact location of a know BT duct and put a 450mm auger straight into it, taking of all the connections.

You cannot account for cowboys and their actions! As already said - additional resilience can be built in, but would you pay extra on your phone line rental, broadband connection and mobile phone to cover the cost?

There has been a lot of damaged cables affecting multiple services and CPs. I expect they will be working into this evening and beyond

In reply to a post by MHC:

You cannot account for cowboys and their actions! As already said - additional resilience can be built in, but would you pay extra on your phone line rental, broadband connection and mobile phone to cover the cost?

Given how critical these services are to people, that's a discussion worth having. Especially when incoming and outgoing calls are knocked out for elderly/vulnerable people who may not have adapted well to mobile communication devices.

Sadly most people will not appreciate the importance of network resilience until they are affected, at which point they may well recognise the importance of paying a bit more for a service with more reliability.

It's back up now (for me), the downtime was 23 hours.

Edited by Oliver341 (Thu 06-Jul-17 16:28:58)

OFCOM are not interested and I would suggest 99% of people would say no as soon as a cost was mentioned.

Would I pay £1 extra on my line rental, £1 on broadband and 50p on each mobile account each month? Probably yes ...

And you are lucky at 23 hours - the outage here was around 5 days for me as a business user and up to 10 days for others.

In reply to a post by MHC:

OFCOM are not interested and I would suggest 99% of people would say no as soon as a cost was mentioned.

It's a shame OFCOM isn't interested, personally I think a situation where hundreds of thousands of customers can be disconnected from their phone and broadband services for a day (or days in your case) by a single point of failure shouldn't be tolerated in 2017. Regulations increase prices across many industries, but that doesn't mean the regulations should be scrapped if they have value.

In reply to a post by Oliver341:

It's a shame OFCOM isn't interested, personally I think a situation where hundreds of thousands of customers can be disconnected from their phone and broadband services for a day (or days in your case) by a single point of failure shouldn't be tolerated in 2017.

Problem is you are not living in the real world. Breakdowns, cable damage, equipment failures occur on a daily basis across the country. And how much duplication do you want. At some point they get close to one another - just look at Apollo 13. Have you the cash to pay for all the additional wiring, ducting etc etc. required for zero day?

What are your back up plans? Probably nil. Like so many other people.

My back up plans? If the internet goes down it would be a connection to the mobile network. It wouldn't be ideal but as a get around, it would work for me. Of course, if the power goes off, well I could go get a mobile generator too but that would be a cost just for the odd occasion and a bit overkill - probably wouldn't start when wanted anyway. I could live without the internet for a few days without too much hassle, power maybe not. For cooking and hot water I need electricity. I'd also get bored cooking off a gas ring bought for camping!

And no, I don't want to pay any more for my internet.

In reply to a post by Oliver341:

Given how critical these services are to people, that's a discussion worth having.

Which services are the critical ones?

Ofcom are somewhat interested, and place some requirements on a telco:

In reference to www.ofcom.org.uk/__data/assets/pdf_file/0021/36192/general_conditio...:

General condition 3 is "PROPER AND EFFECTIVE FUNCTIONING OF THE NETWORK", and specifies:

�The Communications Provider shall take all necessary measures to maintain, to the greatest extent possible:

(a) the proper and effective functioning of the Public Communications Network provided by it at all times, and

(b) in the event of catastrophic network breakdown or in cases of force majeure the fullest possible availability of the Public Communications Network and Publicly Available Telephone Services provided by it, and

(c) uninterrupted access to Emergency Organisations as part of any Publicly Available Telephone Services offered.�

That's for voice, though, not data.

So the intention is that a telco has to keep the voice network running to the fullest possible availability even in catastrophic circumstances or when pleading force majeure.

I'm not convinced that Sky or TalkTalk follow this principle. Not if a fence post can take out the service for 32,000 subscribers.

The last clause is the strongest - making emergency service access (999) available uninterrupted - and is clearly not happening.

I agree that there's a discussion worth having here. Just where does Ofcom draw the line over condition 3?

As you say, though, most people don't appreciate the importance of resilience ... including, it seems, Sky's network planners. Joe Public likely just assumes that Sky and TalkTalk provide the same resilience as BT for voice, and that broadband resilience follows alongside the voice service.

Around 300 fibres and around 3000 copper cut. Is it possible that Sky are running 32000 customers via a single rented fibre.

In reply to a post by 69bertie:

Problem is you are not living in the real world. Breakdowns, cable damage, equipment failures occur on a daily basis across the country. And how much duplication do you want. At some point they get close to one another - just look at Apollo 13. Have you the cash to pay for all the additional wiring, ducting etc etc. required for zero day?

Yet, in the same "real world," BT have put protection and duplication in place, at various levels. That "single point of failure" shouldn't happen within the PSTN from your local exchange building (housing either a local exchange, or a remote concentrator) and upwards.

It should be impossible to lose the whole exchange, even if some components fail, or the software fails, or power fails. It should be impossible for the exchange to lose connectivity to the rest of the network, even if there's a cable failure to one route, or a failure at one of its parent exchanges.

This level of resilience was designed-in from years of experience of "living in the real world". At least if you are a WLR subscriber, rather than an LLU subscriber.

LLU operators, it seems, can get away without bothering. Sky, it seems, lost 32 exchanges with this.

Otherwise, you're right: there is a "real world" balance to be figured out.

The current access network isn't resilient, so driving a lorry into a PCP is catastrophic for 300 lines. I find that more palatable than 32,000 lines. I don't find "hundreds of thousands" acceptable at all.

What level do you draw the line at?

In reply to a post by 69bertie:

What are your back up plans? Probably nil. Like so many other people.

1. Wired backup phone
2. UPS, keeping router, WiFi AP, and VoIP equipment running
3. A battery-based 4G MiFi device with PAYG SIM for rapid deployment
4. A spare router that will use the MiFi device over USB, for longer-term deployment, and to allow access via ethernet and AP.
5. Pre-wired to get a VM connection up and running in relatively minimum time

We've employed 1, 2 and 3 and 4 at times.

I'd not really considered choice of ISP as a matter for "backup" before, but in light of outages like this (and recent Sky fails) I guess use of WLR and WBC makes for a more secure backhaul too.

News report in case anyone is interested: http://www.independent.co.uk/life-style/gadgets-and-...

The report is inaccurate in the outage start though, it was 3 pm yesterday and not 5:30 am today. Sky talk and broadband users on WLR3 only exchanges did not experience the voice outage.

Edited by Oliver341 (Thu 06-Jul-17 22:04:50)

In reply to a post by Oliver341:

It's a shame OFCOM isn't interested, personally I think a situation where hundreds of thousands of customers can be disconnected from their phone and broadband services for a day (or days in your case) by a single point of failure shouldn't be tolerated in 2017.

You made me wonder ...

The most recent "catastrophic" event that hit wider networks was the flooding in Yorkshire 18 months ago, that took out York's exchange, a Vodafone police (101) call centre in Leeds, and Airwave Tetra sites in both.

In reference to www.yorkshirepost.co.uk/news/crime/police-in-flood-hit-yorkshire-wa...:

"One thing you don�t want when dealing with a critical incident is a failure of your communication network, and we had it in all sorts of different areas, all at the same time."

It is an interesting read, and has obviously made all sorts of groups think through their backup plans.

BT had this comment:

In reference to www.yorkshirepost.co.uk/news/crime/police-in-flood-hit-yorkshire-wa...:

�This is the first time the building has flooded and it was not previously thought to be in an area at risk.

...

We have already begun a full risk assessment of our York exchange to identify how better protect it in the event of further flooding on this scale and are engaging with the Government via the Electronic Communications Resilience and Response Group (EC-RRG) to better understand how flooding impacts the sector and what we can learn from incidents like York.�

Interesting, then, to see that the government has an "Electronic Communications Resilience and Response Group" that BT talk to. Perhaps Sky and TalkTalk need to be invited...

In reply to a post by WWWombat:

Interesting, then, to see that the government has an "Electronic Communications Resilience and Response Group" that BT talk to. Perhaps Sky and TalkTalk need to be invited...

Interesting, thanks. I also wasn't aware that WLR3 is inherently more resilient that the voice services of LLU providers. Food for thought.

Finding the government reference led me onto some other interesting sites.

In the end, I found one from UK government:
Cabinet office guidance on Telecom Resilience:
Enhancing the resilience of communications.

That, in turn, pointed out:

The Electronic Communications Resilience and Response Group (EC-RRG) promotes the availability of electronic communications infrastructures in the UK and provides an industry emergency response capability through ownership and maintenance of the National Emergency Plan for Telecommunications.

Which gives us a link to that National Emergency Plan for Telecommunications.

The Centre for Protection of National Infrastructure has a Good Practice Guide on Telecoms Resilience.

The Scottish Government has a Resilient Telecommunications Network Strategy

It does make you think don't it, who shout the loudest, and demand compensation the strongest, and yet seem to keep their end of the bargain in the cheapest fashion ?

It's a poor model for running a company providing what is the crux of so much modern life.

So who cut the cable? When utilities do digs you see them marking out where the cables are prior to doing anything to stop this exact situation. And cut in 7 places! My god

Seems it was marked incorrectly by whoever and the posts were placed accordingly when errecting the hoarding. I susspect laying the blame will be difficult

In reply to a post by WWWombat:

the flooding in Yorkshire 18 months ago, that took out York's exchange,

Looking more at the York floods, I found an article that shows BT has an "emergency response team", complete with vans with satellite links.
https://www.yorkmix.com/news/york-floods-day-3-david...

Looking online, I find their webpages:
https://www.btplc.com/civilresilience/Whatwedo/index...

That's the kind of setup I expect from a telco that is truly aiming to meet Ofcom's general condition:

The Communications Provider shall take all necessary measures to maintain, to the greatest extent possible:

(b) in the event of catastrophic network breakdown or in cases of force majeure the fullest possible availability

And Sky can't even figure out an alternative route for 32 exchanges? C'mon, guys!

In reply to a post by WWWombat:

And Sky can't even figure out an alternative route for 32 exchanges? C'mon, guys!

I can't speak for Sky but when TalkTalk suffered a major cable break in North Scotland ~ 2 years ago, my TalkTalk Business FTTC connection in the Highlands was routed via Aberdeen for a week or so instead of the usual routes direct to Glasgow/Edinburgh. As a result my download speed decreased to ~ 20 mbps instead of the usual 75 meg and pings went from 25ms to 40ms, but once the cable was fixed speed went back up. So TalkTalk at least appear to have some sort of redundancy in their core network up here in North Scotland. Of course alternative routing may give you a slower connection with increased latency but its better than having no connectivity at all...

Edited by deleted (Sat 08-Jul-17 11:14:33)

Oliver

WLR3 just resells BT service ( Provided at Wholesale OFCOM set rates) so uses all the normal BT resilience from the concentrator site making complete loss of service unlikely, although there will be loss of capacity..

There are always exceptions like losing the main exchange building as at York or very small exchanges that only have one duct route out, ( Usually less than 2000 customers) .

On Broadband I believe ( but don't know) BT provides Dual backhaul from its head end sites so as to not lose all customers and capacity if one route is lost. Some customers will lose service until they are switched to the alternate route but this should be quick to do. Other operators do not all do this.

The old saying " You get what you pay for" applies. BT is not allowed by OFCOM to compete on price so attempts to on service, but people like cheap and cheerful until something goes wrong.

Yes, I have to say I was surprised voice and broadband for 32 exchanges could be knocked out by damage to a single duct. One can only assume that single points of failure will cover ever larger areas in future if price/profit is the only factor with no consideration for resiliency.

Lets be honest the uptime over a year for that area will be well in excess of 99%, there is no way to absolutely be sure 999 calls are uninterruptible and I suspect if Sky did business connections their network would be far more resilient like the above mentioned Talk Talk Business story. But that is why Business phone lines and broadband is more expensive you are paying for better response times and resilience. If you need better than what Sky offer, pay for it.

was wondering if its my issue, as my broadband has been down since 5am sunday morning, however in my case the voice is dead and the modem has no sync so an openreach issue just similar timescales.

all ofcom care about is competition, and low prices. They wont support policies that increase the costs of providing a service, so yeah its no wonder they not interested.

End of the day, consumer broadband has no SLA, no uptime requirements, its all provided on a best efforts basis.

I see it as my own responsibility to have resilience, and right now I am making this post on my backup which is my EE 4G connection.

this is from sky community

We are aware that some customers in and around the East Midlands area may be experiencing issues with their phone and broadband services - we're sorry about this. Engineers are working hard to fix the problem.