User comments on ISPs
  >> PlusNet plc


Register (or login) on our website and you will not see this ad.


These posts have been archived and can no longer be replied to or modified.
Pages in this thread: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | (show all)   Print Thread
Standard User channel
(committed) Thu 03-Aug-06 11:34:45
Print Post

Alas, the email has gone for good


[link to this post]
 
700Gb of email flushed down the toilet, never to return, it would seem.

From the PUG (corrected from PUNG MrSaffron) site: http://usergroup.plus.net/forum/index.php/topic

As with my initial posting on the loss of customer email data - http://usergroup.plus.net/forum/index.php/topic,2472.0.html, what follows is an open and honest account of the events surrounding the attempt to recover the data.

Firstly you need to understand that in an attempt to recover the data swiftly, the engineer who deleted the 3 volumes in the first place swiftly followed up his error by immediately trying to create a volume of the same size as the 1st of the volumes in the same place. This is an old sysadm "trick" that on some file systems could have revealed the lost data, however in this case, it did not work, and in fact caused us more problems, as you will see later on in this account.

Within 2 hours of the data being deleted, a data recovery company had been contacted and within 3 hours the NAS was in transit to them. By 14:00 on that day the specialists were racking the NAS and began the process of copying all the 1's and 0's from our equipment to their own. This is standard operating procedure for anyone working in the field of data recovery, and is simply about ensuring that there is always an untouched copy of the information in case something further goes wrong while working on the recovery. Due to the volume of data that was being dealt with, the copy took until the early hours of the following morning.

At that point, based on their initial investigations, the data recovery specialists set the expectation with us, that we would recover some of the data, possibly not all of it though, and that it could take 4-5 days. From that point forward we have a tale of increasing woe as each new deadline set by the data recovery people was broken as they discovered more and more problems. In the following paragraphs I will briefly cover off the main problems that have been encountered.

The Sun NAS that we had selected for the mail storage platform is the first series of products to emerge from Sun since their purchase of StorageTek, and as such does not run the usual Sun OS of Solaris. It uses StorageTek's own proprietary OS which is a heavily modified FFS2 (Fast File System 2). The modifications are all about increasing the performance of the system to ensure enterprise level performance.

As the kit is fairly new to market, the data recovery specialists had not worked on this specific OS before, though they do have a lot of experience with NAS's in general, Therefore they had to significantly rewrite the tools that they use for analysing and recovering data. They utilised their engineering departments in both the UK and the US to work around the clock to achieve a re-worked set of tools.

Apart from the tools issue, the proprietary OS, uses the 1st volume it has access to, to store the master inode table. For more information about inodes, take a look at the Wikipedia article http://en.wikipedia.org/wiki/Inode Essentially this is the table that tells the system where all the other files on the system are. As I mentioned earlier, the PlusNet engineer involved, had attempted to recover the data by creating a volume of the same size in the same place as the 1st volume. That action more than any other has caused us the most issues. By creating a new volume, the existing inodes were wiped and all data that was on that volume was essentially gone. Without that master inode table and with no knowledge of where the system stored it's back-up copy of this table, it has proved very difficult to work out what the data on the relatively undamaged 2nd and 3rd volumes actually is.

We have received a partial file list from the 2nd and 3rd volumes. This list amounts to a list of inodes and the data in them, not the list of complete files. Without even a partial directory structure it becomes vastly more complex to work out which inodes are associated with which other inodes and therefore piece together the complete files. Without the data on the 1st volume we do not believe we are ever going to get the directory structure. Without the directory structure it becomes vastly more complex trying to work out which file from the partial list belongs to which user.

So, here we are, it is almost a month since the 700GB of email and mailing lists were lost and we still have no recovered data to return to you. This is of course upsetting for us, and even more so for the customers whose data has been affected. The longer we wait for the equipment to be returned to us the greater the risk we run of hitting other capacity issues that we know are ahead of us, and we do not feel that we can justify any longer a wait, and still be taking the appropriate action for our customers.

When it is implemented the new platform will provide us and you with a vastly scalable, site resilient mail storage set-up, with 6 hourly checkpoints to ensure we can roll back the majority of the changes that happen on the system within a four hour period.

What this really means is that we are currently arranging for the return to us, of the Sun NAS head unit and Disk Arrays, so that we can push forward with the implementation of the new email storage platform.

--------------------------------------------------------------------------------
Simon Day
Network Improvement Consultant

[MrSaffron fixed broken links]

___________________________________
PLUS NET - The Shaping Of Things To Come
"If you don't like it here it's very simple for you to find another ISP."
....................Neil Armstrong
Head of Marketing, Plus Net plc
15 September 2005

Edited by MrSaffron (Thu 03-Aug-06 16:04:24)

Standard User stephen_f2s
(fountain of knowledge) Thu 03-Aug-06 12:04:48
Print Post

Re: Alas, the email has gone for good


[re: channel] [link to this post]
 
I see the Einstein who wiped the data in the first place still hadn't gathered he wasn't the sharpest knife in the draw, so he thought he'd try some heroics. Nice.

stephen_f2s
ex-freedom2surf employee
Standard User caesar_salad
(experienced) Thu 03-Aug-06 12:06:37
Print Post

Re: Alas, the email has gone for good


[re: stephen_f2s] [link to this post]
 
According to the Plus.net staff who used to post on here apparently this engineer was someone who had lots of technical knowledge in storage areas so probably thought that the trick might have worked.

Sometimes you win,sometimes you lose..

=========================

Plus.net: "We can't predict the future"

Plus.net: Beware of The Leopard

My spelling mistakes are all my own


Register (or login) on our website and you will not see this ad.

Standard User stephen_f2s
(fountain of knowledge) Thu 03-Aug-06 12:11:34
Print Post

Re: Alas, the email has gone for good


[re: caesar_salad] [link to this post]
 
In reply to:

apparently this engineer was someone who had lots of technical knowledge in storage


Which could mean he knew what the write-protection tab was for on a 3.5" floppy and he carries a USB pendrive with his keys.

stephen_f2s
ex-freedom2surf employee
Standard User channel
(committed) Thu 03-Aug-06 12:13:00
Print Post

Re: Alas, the email has gone for good


[re: caesar_salad] [link to this post]
 
Agreed. I don't get any pleasure out of knocking the engineer. I'm sure he did what he thought was best given the circumstances he found himself in. I also note that the only reason anyone can pick fault with the engineer is because Plus Net have been very upfront and honest about what actually happened; given their open explanation, I think it is a bit harsh to then use this information to simply have a go at an individual.

Obviously, I am gutted at the loss of my email. I think PN have had a good go at getting it back, and I give them credit for that. But the back up systems should have been such that we should never have got to this point in the first place.

___________________________________
PLUS NET - The Shaping Of Things To Come
"If you don't like it here it's very simple for you to find another ISP."
....................Neil Armstrong
Head of Marketing, Plus Net plc
15 September 2005
Standard User caesar_salad
(experienced) Thu 03-Aug-06 12:15:53
Print Post

Re: Alas, the email has gone for good


[re: stephen_f2s] [link to this post]
 
I'm not gong to get involved in the semantics of what can be judged as to what Plus.net posted but sadly for some people the end result is the same and their emails are gone.

They personally don't care if the engineer was a grade one so and so but they believe that Plus.net had a duty of care for looking after their customers' needs.

I can see this thread being pasted on the Register again..

=========================

Plus.net: "We can't predict the future"

Plus.net: Beware of The Leopard

My spelling mistakes are all my own

Edited by caesar_salad (Thu 03-Aug-06 12:30:00)

Standard User stephen_f2s
(fountain of knowledge) Thu 03-Aug-06 12:17:57
Print Post

Re: Alas, the email has gone for good


[re: caesar_salad] [link to this post]
 
In reply to:

I can see this thread being pasted on the Regsiter again..


It's flogging a dead horse.

stephen_f2s
ex-freedom2surf employee
Standard User caesar_salad
(experienced) Thu 03-Aug-06 12:22:16
Print Post

Re: Alas, the email has gone for good


[re: stephen_f2s] [link to this post]
 
Who? The Register or Plus.net??

At least it will bring some closure to anyone who has been keeping up to date with the news story on the Reg.

They could use this as a decision tool for when they decide to connect to ADLS for the first time or migrate from another provider.

=========================

Plus.net: "We can't predict the future"

Plus.net: Beware of The Leopard

My spelling mistakes are all my own

Edited by caesar_salad (Thu 03-Aug-06 12:23:34)

Standard User stephen_f2s
(fountain of knowledge) Thu 03-Aug-06 12:24:33
Print Post

Re: Alas, the email has gone for good


[re: caesar_salad] [link to this post]
 
I mean just the email issue. If anyone reported on it ... I can't see it being "news" to anyone.

stephen_f2s
ex-freedom2surf employee
Standard User caesar_salad
(experienced) Thu 03-Aug-06 12:29:37
Print Post

Re: Alas, the email has gone for good


[re: stephen_f2s] [link to this post]
 
It's "news" that in the sense that the email is all but lost... at least when the data was being looked at there was a faint hope that they could salvage some of it.

=========================

Plus.net: "We can't predict the future"

Plus.net: Beware of The Leopard

My spelling mistakes are all my own
Pages in this thread: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | (show all)   Print Thread

Jump to