In reply to:

---

So what happened to the backup power supply you have at your data centres and why did it not fail-over to the backup data centre as it was supposed to.

What's the point of having a UPS and a backup data centre when neither actually work!

---

Questions I remember being asked 12 months ago. The replies, along the lines of, "systems have been put in place to ensure this won't happen again."

Apparently this data centre was not one of PlusNets but a 3rd party and thus, PN had no control over what happened, they just have their kit in there.

There are still some serious questions that need to be asked as to how this could have happened and more importantly what they (the 3rd party) are going to do to stop it happening in future.

If the power down was of the type where the 'big red fire kill switch' is hit then everything should power off. Thats the point of these things, to kill all power to all devices instantly.

Pretty much when anything goes wrong at BT PN, it's blamed on the slap dash procedures of some pesky third party.

try reading what happened in Oldjim's post,

In reply to:

---

During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre. That action caused all power to all services to die including the generaters and ups. Failover couldnt work as the power to everything was killed. We believe that this was human error however we are awaiting a full report from our suppliers.

In reply to:

---

you can have all the procedures in the world, whether slap dash or not, still doesn't stop human error, humans make mistakes

Paul

Plus Net - maxDSL - BYW Pro package
Draytek Vigor 280VG running 2.7_E38 firmware

Perhaps the human error was actually in developing a "failsafe" procedure which didn't take account of the risks that a single engineer could snip through the wrong wire and paralyse the whole thing?

Procedures and policies are generally put in place *precisely* to avoid the possibility of mass outages as the result of human error. They were not in this case.

I and others can well remember the "human error" that lost me 4 years worth of email from BT PN servers as I was using their advertised IMAP system - I believe that procedures are now in place to ensure that "never happens again", so clearly human error is something that CAN be mitigated against - it just seems to take catostrophic occurances to galvanise people into action.

>>Perhaps the human error was actually in developing a "failsafe" procedure which didn't take account of the risks that a single engineer could snip through the wrong wire and paralyse the whole thing? <<

Hi James. Long time no speak.

Providing a safety feature in a Data Centre to isolate power, "EPO" Emergency Power Off, is a necessary evil for obvious reasons. That is part of the design of modern data centres. Having this in place is not human error. Where it gets used in the wrong circumstances is.

As we have stated. The data centre is owned and run by a third party. The engineers were from another third party. We had no knowledge that this work was taking place and this, along with the resulting actions is what we are persuing with the owners of the Data Centre.

Or the human error was the data house didn't understand what the contractor was going to do, which is a common failing.

Unfortunately we don't know what the contractor was there to do, he may have been there to work on the failsafe system, or a part of it. Because of this we can't say whether procedures etc were there or not, they probably are in place, but either not used, not understood or have errors in them.

I'm afraid, due to my background in process safety, I take statements like 'never happens again' related to procedures with a pinch of salt, humans make mistakes, whether that is in writing the procedure, failing to follow the procedure in the correct order, just not understanding it but trying to muddle through or just not using it/forgetting its there. Human error can be reduced but it can't be complete mitigated against, there is some research by a chap called Kirwan called 'Guide to Practical Human Reliability Assessment', a person doing a simple task that requires care is likely to make an error once in a hundred times.

I'm afraid we are just not that reliable, especially if there is a stressful situation

In reply to:

---

Providing a safety feature in a Data Centre to isolate power, "EPO" Emergency Power Off, is a necessary evil for obvious reasons.

---

The problem is how it is activated. Is it like a fire alarm and house in a case where physical intervention is required to operate?. Are there many access points, are they protected as above?.
Is there any laid down proceedure to be followed for "killing all the power"?.

In reply to:

---

During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre.

---

Suggests the contractors decided to do so without reference to the Data Centre staff. Rather a lax way of operating don't you think?.

<<Suggests the contractors decided to do so without reference to the Data Centre staff. Rather a lax way of operating don't you think?. >>

I have no idea what they did wingco and I certainly wont speculate on that aspect.

Rest assured that we have expressed our displeasure.