Calamity!
I’ve been around long enough–and seen enough disastrous computer crashes–that I have a pretty careful backup strategy. I’ve been the nagging voice of nagginess when my friends have lost data due to hardware failures.
“You should ALWAYS make backups of your data. You never know when you computer will take a permanent siesta.”
I commit all of my source code to a subversion repository on my local machine, and I regularly take snapshots of that development machine, which I keep on an external hard drive. In case of a failure I’d suffer two or three days of lost data. A week, tops.
Lately, my computer’s primary hard drive has been making funny noises, like a tiny helicopter flying into a narrow fjord, deep within my laptop. So I’ve been extra diligent about backups.
On Friday, it finally died.
During the boot-up process, it went into a ten-minute chkdisk loop before ultimately succumbing to a horrible BSOD.
After that… nada.
No problem. I bought a new drive and prepared to transfer everything over. After installing the operating system onto the new drive, I connected the backup drive (in its own external USB enclosure) and listened, horrified, as it made a few pathetic clicks and whirring noises.
Dead.
Huh. That’s weird.
This drive had been chugging along without complaint just a few hours beforehand, and suddenly, it’s a complete brick. The operating system doesn’t recognize it at all. It may as well be a can of sardines.
I fiddled with the connections, finally taking apart the enclosure, swapping the cables. Still nothing. The drive is definitely kaput.
Keep in mind, this is not just my source code. This is my entire life. Software projects, art projects, letters, stories, email archives, proposals, invoices, tax documents, from the last seven or eight years.
I felt like my house had burned down.
Panicked now, I connected the old laptop disk, to see what I could salvage.
As it turns out, there was still a lot there.
Even though the operating system wouldn’t boot, and some entire directory hierarchies had vanished in a puff of corruption, I was able to salvage nearly all of my personal documents (in fact, as far as I can tell, all of my photos and documents survived).
But the source code is riddled with holes. Entire projects are just completely gone. (And about a third of my MP3 collection. The horror!)
The application analytics project–which I’ve been working on since June–is particularly decimated. From a total of probably 30,000 lines of code, I’d say about 25% has vanished. The entire server project is nowhere to be found. (The entire embeddable library project and about 90% of the reporting client GUI survived.)
I was able to recover an old, crusty snapshot of the server code (from September) by manually slogging around in the raw subversion data files.
Also completely gone: a set of ActionScript support libraries, with data structures and algorithms for the charts & graphs in the reporting client. Luckily, at the end of December, I started the process of moving my source code–starting with this particular library–to a repository on a remotely hosted (and independently managed) server.
Thank God for small miracles, right? At least if I could recover that code, I’d save myself the trouble of reimplementing about three or four thousand lines of code.
So I tried doing a check-out, but guess what!
There’s nothing there.
Browsing remotely with Tortoise SVN, the server happily responds that the repository is empty. And the web SVN interface says that I don’t have access to the repository. Which is odd… since it also lists a record of my most recent check-ins.
Wouldn’t it be funny if that repository had somehow evaporated?
Is someone playing a joke on me? Am I being punk’d?
I’m going to send my (two!!) failed hard drives to a data recovery company tomorrow. Maybe they can do better than me at recovering some of this lost files. Yikes.
Anybody want to make a small wager that the data recovery facility will burn to the ground the moment my FedEx package arrives?
January 13th, 2009 at 3:08 am
Just remember to try SpinRite before sending those drives to a data recovery company. I can bet you will have your data back.
http://www.grc.com/sr/spinrite.htm
January 13th, 2009 at 7:44 am
That’s nasty. You made me recheck my backup scripts. One of them wasn’t working because the backup service had stopped. You’ve suffered, but you’ve ultimately done good for the world, because I bet I’m not the only one to check my backups.
For the record I use pair of mirrored disks in an old PC as a file server, with a nightly backup to a external-usb mounted hard disk from the mirrored disk. My laptop synchronises important files to the file server nightly (photos, documents, etc). I swap the external hard disk out with an identical copy weekly, and the not-in-use disk goes in a little wooden box to a relatives house, to be kept in a top drawer. This gives me 5 versions of my files at various states of concurrency (1 laptop HD, 2 x server HD’s, 2 x backup Hds, one of which is an off-site version)
I still manage to accidentally delete files and lose things, though. Perhaps I should put another one in there somewhere. The price of storage these days means you should have copies everywhere.
January 13th, 2009 at 8:44 am
You should definitely get Jungledisk or something similar. An external drive isn’t backup at all. What if your house burns down or you get robbed? The external drive right next to the computer goes to. Jungledisk and the related Amazon service are criminally cheap, get them!
January 13th, 2009 at 12:16 pm
Wow, 2 drives. If I were you, I’d have a professional investigate the electrical situation where I work and live. The only time I’ve experienced multiple drive failure like this was due to a surge. For my office, I use external hard drives as well – I never really think about a double whammy! I’ll be adding a 3rd backup pronto.
January 13th, 2009 at 2:01 pm
Yeah. Who’d have thought two drives would fail within a day of one another?
When I was in the process of removing the first drive from the laptop, I actually thought to myself “You don’t have a backup anymore. Your data only exists in one location.”
I fully believe in the idea of never being without a backup. The tricky thing, which had never occurred to me before, is that, when your original device fails, the backup becomes the sole instance of your data, and you’re left without a backup.
So, the only way to *always* have a backup–even during the process of recovery–is to always have *two* backups.
January 13th, 2009 at 4:39 pm
Talk about bad luck, sorry to hear it, here’s hoping the recovery folks can assist.
I don’t know if you are a Windows user… if so, I really would suggest looking into building or buying a Windows Home Server.
Not only does it do an automatic nightly backup of your PC’s, but also preserves backups over time so that you can easily recover files (or an entire system) back to the way they were yesterday, last week or even last month.
It also works as a great central repository for documents, photos, music and movies, and when multiple hard drives are added to its storage pool (USB, Firewire, IDE or SATA), you can enable duplication on a per share basis (a single check box) which instructs the underlying system to make sure that files in that share exist on two hard drives instead of one.
Heck, given it is Windows under the hood, you could also run your Subversion server on it.
In full disclosure I am kinda partial to the product as I work on it at Microsoft… but then it is the reason I came to this company.
January 13th, 2009 at 5:36 pm
I would think two drives would fail within a day of one another. Not just general Murphy’s Law pessimism, but lots of things like brownouts, power spikes, fire, theft, water damage, and rampaging children can destroy a coupe hard drives.
I’ll chime in as another fan of JungleDisk, and I’m sorry for your data loss.
January 13th, 2009 at 10:36 pm
I feel your pain Benji. Been there done that. I now label all my drives with the month and year I purchased them – and I try to stagger the purchases and use different brands to avoid a potentially bad batch from a single vendor – and I plan to rotate them after about 3 years. I back up my backups now on both hard disk as well as optical media (dual layer DVDs, planning to migrate to blue-ray). I also use shadow protect for my backups – best software there is. It will do a bare metal restore on a new disk, or even completely different machine direct from your backups. You can schedule incremental backups as frequently as every 15mins. This is a great saviour if you accidentally overwrite something from earlier in the day. Can’t recommend it enough.
January 14th, 2009 at 3:47 am
Strangely enough the hard drive on my 3 years old laptop crashed a couple of weeks ago. For some miraculous reasons I was testing my very own Online Backup product that works on top of Amazon S3 and I backed up the folder with my most important documents. I wish I backed up all of them! Once the disk crashed I restored my files having development team to fix some issues along the way and I could continue my work in a couple of hours. This sounds like promotion and it is really is, but it really helps to backup up your data regularly offsite. It also worth mentioning that a repair shop asked me for $1 000 to get my data back from the hard drive and they will need about a week to work with it … and you have to visit their shop losing another couple of hours.
If you want to sign up for a beta please visit http://cloudberrylab.com/default.aspx?id=39
Or if you are sophisticated enough you can try our CloudBerry Explorer freeware that helps you to copy files to Amazon S3.
http://cloudberrylab.com/default.aspx?id=7
October 28th, 2009 at 5:09 pm
[...] having multiple forms of backup. If you think backing up to a single USB disk is enough you should read this. If you are relying purely on an third party online backup service you should consider what would [...]