www.plus.net down ? email probs ?

I'm getting prompted repeatedly with the 'Enter Network Password' windows dialogue box when trying to send email and www.plus.net seems to be down so I can't chec the service status page - are there problems at the moment ?

I'm having the same problems too.

Portal maintenance is taking place from 01:30 to 06:00 today. A service status post was issued (what is now) yesterday afternoon.

Any reason why keep getting this error when trying to log into webmail?

ERROR: Connection dropped by IMAP server.

07:34. There is a recorded message on their help line advising e-mail down

Hi there,

We had a power outage at one of our Sheffield data centres. It's likely to have affected most if not all Sheffield based services to some degree.

Kind Rgds,

So what happened to the backup power supply you have at your data centres and why did it not fail-over to the backup data centre as it was supposed to.

What's the point of having a UPS and a backup data centre when neither actually work!

Hmmm!!!

I have a good few mailboxes and some are 'connecting' OK but others are not??? This is using Outlook Express as my email client.

The prime question is for me why are some OK and others not and is there an expected time for resolution for the repair?

I only just walked in, so I'm still getting up to speed on what happened. We'll certainly explain the nature of the power outage in more detail later once the residual problems are fixed and the data centre owners have provided their report. The site in question is our third party managed data centre in Sheffield and I believe the issue affected all users of the facility.

I believe the issues outstanding with some mailboxes relate to database re-mounting issues, and it's being worked on along with the other stuff that's down. Non graceful power-downs are always a nightmare to recover from... It looks like a database issue is also affecting service status propagation, so we will make sure further updates are published on the community site shortly.

Ian

This post explains what happened
http://community.plus.net/forum/index.php/topic,61274.0.html

In reply to:

---

Hi all.

Posting a new thread to discuss this as the original is getting a bit long now.


Last night our 3rd party data centre suffered a total power outage.

We have found out this morning that the owners of the data centre had contractors in doing some maintenance. They failed to notify us of this work.

During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre. That action caused all power to all services to die including the generaters and ups. Failover couldnt work as the power to everything was killed. We believe that this was human error however we are awaiting a full report from our suppliers.

Having said that, not all services can be automatically failed over. DB's etc have to be failed over manually to ensure that data integrity is maintained. As we were unaware that this work was taking place we did not have any one on standby to react. Thankfully our incident management processes and call out system kicked in very effectively. We had engineers on site within about 20 mins and 97% of services were back up and running in around 4 hours.

We experienced intermittent timeouts with IMAP and webmail for a period but this was identified and resolved. This was due to the servers coming back on-line before the network to the other data centre. This meant that half of the servers failed to mount the storage correctly but the servers in the other data centre were working correctly.

I'll reiterate that this was a catastrophic failure which we believe was caused by human error. All the resiliency in the world could not have prevented this situation. However the swiftness of our reaction ensured that this was resolved in super quick time.

We currently have a case open with our suppliers pursuing this issue. We will not be able to report fully on their findings due to the nature of our contract with them but we will share as much information as we possibly can.

Where we did fall down on this one was the lack of call out for the Comms team and the fact that due to this we failed to communicate effectively. We could have used the UserGroup site and TBB to post information, but as we were all blissfully unaware that this had happned until I got in at 0800 this morning we failed to communicate effectively. I am dealing with that aspect internally this morning.

I'll post with updates as soon as I get them.

Best Regards

Mark Kelly
Plusnet Comms and Referrals Manager | The UserGroup Forums

---

In reply to:

---

So what happened to the backup power supply you have at your data centres and why did it not fail-over to the backup data centre as it was supposed to.

What's the point of having a UPS and a backup data centre when neither actually work!

---

Questions I remember being asked 12 months ago. The replies, along the lines of, "systems have been put in place to ensure this won't happen again."

Apparently this data centre was not one of PlusNets but a 3rd party and thus, PN had no control over what happened, they just have their kit in there.

There are still some serious questions that need to be asked as to how this could have happened and more importantly what they (the 3rd party) are going to do to stop it happening in future.

If the power down was of the type where the 'big red fire kill switch' is hit then everything should power off. Thats the point of these things, to kill all power to all devices instantly.

Pretty much when anything goes wrong at BT PN, it's blamed on the slap dash procedures of some pesky third party.

try reading what happened in Oldjim's post,

In reply to:

---

During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre. That action caused all power to all services to die including the generaters and ups. Failover couldnt work as the power to everything was killed. We believe that this was human error however we are awaiting a full report from our suppliers.

In reply to:

---

you can have all the procedures in the world, whether slap dash or not, still doesn't stop human error, humans make mistakes

Paul

Plus Net - maxDSL - BYW Pro package
Draytek Vigor 280VG running 2.7_E38 firmware

Perhaps the human error was actually in developing a "failsafe" procedure which didn't take account of the risks that a single engineer could snip through the wrong wire and paralyse the whole thing?

Procedures and policies are generally put in place *precisely* to avoid the possibility of mass outages as the result of human error. They were not in this case.

I and others can well remember the "human error" that lost me 4 years worth of email from BT PN servers as I was using their advertised IMAP system - I believe that procedures are now in place to ensure that "never happens again", so clearly human error is something that CAN be mitigated against - it just seems to take catostrophic occurances to galvanise people into action.

>>Perhaps the human error was actually in developing a "failsafe" procedure which didn't take account of the risks that a single engineer could snip through the wrong wire and paralyse the whole thing? <<

Hi James. Long time no speak.

Providing a safety feature in a Data Centre to isolate power, "EPO" Emergency Power Off, is a necessary evil for obvious reasons. That is part of the design of modern data centres. Having this in place is not human error. Where it gets used in the wrong circumstances is.

As we have stated. The data centre is owned and run by a third party. The engineers were from another third party. We had no knowledge that this work was taking place and this, along with the resulting actions is what we are persuing with the owners of the Data Centre.

Or the human error was the data house didn't understand what the contractor was going to do, which is a common failing.

Unfortunately we don't know what the contractor was there to do, he may have been there to work on the failsafe system, or a part of it. Because of this we can't say whether procedures etc were there or not, they probably are in place, but either not used, not understood or have errors in them.

I'm afraid, due to my background in process safety, I take statements like 'never happens again' related to procedures with a pinch of salt, humans make mistakes, whether that is in writing the procedure, failing to follow the procedure in the correct order, just not understanding it but trying to muddle through or just not using it/forgetting its there. Human error can be reduced but it can't be complete mitigated against, there is some research by a chap called Kirwan called 'Guide to Practical Human Reliability Assessment', a person doing a simple task that requires care is likely to make an error once in a hundred times.

I'm afraid we are just not that reliable, especially if there is a stressful situation

In reply to:

---

Providing a safety feature in a Data Centre to isolate power, "EPO" Emergency Power Off, is a necessary evil for obvious reasons.

---

The problem is how it is activated. Is it like a fire alarm and house in a case where physical intervention is required to operate?. Are there many access points, are they protected as above?.
Is there any laid down proceedure to be followed for "killing all the power"?.

In reply to:

---

During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre.

---

Suggests the contractors decided to do so without reference to the Data Centre staff. Rather a lax way of operating don't you think?.

<<Suggests the contractors decided to do so without reference to the Data Centre staff. Rather a lax way of operating don't you think?. >>

I have no idea what they did wingco and I certainly wont speculate on that aspect.

Rest assured that we have expressed our displeasure.

difficult to say, the reality could well be that the contractors thought the data centre knew what was going to happen, especially if they were there to test the emergency system. or someone lent on the button! seen that before or there was a lack of knowledge perhaps they thought the switchovers would work before the power down.