What do you do in terms of personal backups? Or perhaps I should ask if you do personal backups at all? I mean, at work we all probably do some backups to cover our asses. If you work in IT you probably deal with all kinds of backups. If you do not, you probably still keep redundant copies of your work just in case. No one wants the boss ripping them a new orifice because of a random technology failure. Unless of course you are like my users, and you can charmingly convince the management that not making laptops resistant to coffee spills, falling off the table and being stepped on (executed in quick succession) is a huge failure on the part of the IT department.
But what do you do for your personal computer? Here is my setup:
- I use Mozy for my off-site backup.
- As a secondary precaution, I have Mozy client also back up my data to an external drive
- Finally, I mirror that external drive to another drive using unison (rsync like tool)
This means that I have redundant 4 copies of my important data. One is completely off-site meaning even if my house spontaneously explodes one day, I still have a good shot at getting my data back. Granted, I heard some unflattering things about Mozy restore process but so far it has been working fine for me. Of course I never had to use it to restore more than one or two files at the time so maybe I haven’t seen it at it’s worst. But unless something really, really crazy happens I still have my data backed up locally to two drives.
I’m certain someone will want to post a comment along the lines of:
“But Luke, that second drive is surely an overkill, isn’t it?”
Nope. No it’s not. Hard drives die. This is what I always tell my students: it is not a question of “if”, it’s a question of “when” your hard drive is going to fail. Magnetic drives have moving components that are prone to wear and tear. Other than the fans, the hard drive is the only hardware piece in your computer that is in constant motion. After years of constant use, it will fail. Solid state drives do not have moving parts, but each memory cell on these devices has an upper limit of writes it can support after which it breaks. No matter what you do, your media won’t last forever.
External drives seem to be more susceptible to this than the internal ones. I have a drawer full of drives too prove this. Here are a few interesting ways in which I killed external drives:
- Death by vertical realignment – I had the drive standing upright. I laid it on the side while it was still on. It went “SKREEEEEE, KA-TUNK, KA-TUNK, KA-TUNK, SKREEEEEE” and never worked again.
- Death by spatial reallocation – I properly disconnected it, unplugged it, carefully moved it to another room, plugged it back in, and watched it die.
- Death by pen – I put a pen on a shelf above the hard drive laying flat on my desk. The pen rolled off and fell down about a foot, landing on the hard drive and causing a head crash.
- Death by rocking – I accidentally knocked bumped an upright standing drive while reaching for something. It did not fall over – it just rocked in place a bit and settled down, only to emit the worst screeching noise I have ever heard immediately afterwards
- Death by coffee – coffee cup, spill, dead hard drive – you know the drill
- Death by sneezing – a rather violent sneeze caused me to bump into my desk, thus causing an upright standing drive fall on it’s side. Head crash.
- Death by dust – there as one drive that I saved something on, then disconnected it and put it away for about a year or two. It got dusty. Never worked again.
- Death by attrition – about four of my external drives just died of old age. They worked for a few years, and then just stopped and refused to turn on, or be detected one day. They didn’t all fail on the same day, but they all failed.
I currently have four external drives connected to my home machine, and one of them is an older LaCie drive that is in it’s death throes. How do I know? Because every time I power it down, it takes it about 10 minutes to spin up to a point where and OS can detect it. Sometimes I have to power cycle it a few times to even get it do that. It also makes funny noises. I use it as a external /tmp folder for large files I really don’t care about.
I admit that maybe I’m just very unlucky with external drives. Maybe I emit some sort of hard-drive killing radiation. The drives inside my computers usually last quite a bit, but the small rectangular boxes die like flies in my house. I have seen so many die for no reason I no longer trust them to preserve my data. That’s why I mirror my backups on two. Because I’m painfully aware either of these drives can drop at a moment’s notice.
I’m not saying you should do the same but… Actually, you should do the same. A little bit of paranoia goes a long way when it comes to backups. Storing your data in a single location is reckless and foolish.
If you are a roaming laptop user, who refuses to be tied down to a desk, get a network attached storage device. They are not that expensive. I’m currently using something called Seagate GoFlex Home and it is actually pretty neat. The web based UI is pretty useless, but once you set up an account it works fine as a network drive that can be shared across multiple computers. It also comes preconfigured to work with Apple Time Machine (this is how I back up my MBP at the moment – no mirroring yet because it does not have that much important stuff on it). If you don’t have much space for attaching network devices and you don’t mind spending a little extra for a shiny box with an Apple symbol Airport Extreme is quite nice – especially for Mac lovers. It can double up as a Wireless router and a NAS, and it also works with Time Machine out of the box.
How do you back up? What are your suggestions in terms of devices and setups? What backup software do you use at home? How do you set up clueless friends and relatives?
This is my usual drill:
1) I use Dropbox for my most important and current things.
2) My photos I keep on my internal hard drive and in sync on a separate external HD. I also started keeping the SD cards I use on the camera intact. When the card is full, I just buy a new one.
3) I used to, every 1 to 3 years, backup the new things I produced/saved over these years on DVDs. I am now doing this with USB sticks.
Here is the One True Personal Backup Plan:
Step 1: Save all your data not on your computer but on a four drive NAS with at least one drive redundancy. This protects against hard drive failures and, to a very limited extend, viruses (due to the fact that your NAS will likely be running on Linux). I mostly like Synology, but QNAP, ReadyNAS, and UnRaid may also be good solutions from what I’ve heard.
Step 2: Now backup your data. RAID is not a backup. If you choose a Synology NAS, you can set the second NAS to automatically backup the first periodically. I’m sure the others have similar functionality, or you can use something like rsync.
Step 3: now that you’ve backed up your data, you need to ask: is that enough? Even if you recover your data, you’ll still need to reinstall your system from scratch I. The event your computer fails or your system is infected with a virus. That might be a significant amount of work, depending on your setup, so you might want to consider doing a full image of your system so that you can do a bare metal restore. Unfortunately, there is no simple, cross platform solution for this. I haven’t figured out a perfect solution here so I’m open to suggestion. CloneZilla might be ideal if you use multiple OS’s, but it’s fiddly. DriveImageXML is reportedly good for windows (as is windows backup, from what I’ve heard), and Carbon Coby Cloner is reportedly good for OSX. If you use Linux, you’re smart enough to use CloneZilla.
Step 4: Now that you have hard disk redundancy, scheduled backups, and full images you need an offsite backup for both your data and your images. Ideally, you would use something like CrashPlan, which appears to be much more reliable than a service like Mozy and can even be free (it allows you to backup to a friend’s computer instead of or in addition to ther cloud backup service). However, CrashPlan doesn’t work well with a NAS last I checked. Since offsite backups are only for total catastrophies, they don’t need to be frequent. Set a reminder on your calendar and once a year (or once every six months or whatever), do a manual backup of your data and an image of your system. Save those to an external hard drive using a USB enclosure (I like Thermaltake BlacX, but there are lots of choices), wrap the hard drive in a static proof bag, and take it to your office, safe dosit box, or family members house. Don’t just overwrite it the next time you do an offsite backup: have multiple hard drives and rotate them, so that you have backups to your backup.
Step 5: You now have a comprehensive and mostly automated backup solution for your computers. But increasingly a lot of our data lives in the cloud or on non-PC devices like our phones. Are you comfortable with that? Neither am I. So make a list of your cloud and device data repositories. When you do your annual offsite backup, manually backup these sources as well to your NAS. Ideally, think not just about data but also passwords (I use LastPass) and bookmarks or RSS lists.
Any weaknesses on is approach?
I should probably give more thought to backing stuff up. My desktop has a big RAID5 drive (4 physical discs) but I’ve heard “RAID isn’t a backup” often enough. This is, the vast majority of what’s on there is media files which would be… well, not trivially easy, but not impossible to replace or much of a hardship to be without.
I have a second copy of a lot of that on an external drive, and the non-media component of my personal files is mirrored on a handful of smaller discs in that same desktop. Need to sort it out, but nothing’s in immediate danger except via the “house blows up” scenario.
I have no organized backup plan, but nearly all of the data I care about is replicated on my laptop, desktop, and vps via git. I don’t bother with disk imaging because it’s faster for me to reinstall my OS than to recover an image.
Also, with regard to disk failure. Raid is a much more economical solution to that than backups if you have enough data to justify it. Backups and offsite backups are for dealing with unpredictable and catastrophic failures, but a good raid setup will predictably deal with drive failure without the interruption of backup recovery.
@ astine:
And what happens when a bug in your RAID software introduces data corruption, which it then faithfully replicates to your other disks( as happened with early Windows Home Servers)? What happens if a virus infects your files, and the virus is instantly replicated to your other disks? Hard drive failure is not the only non-catastrophic cause of data loss. RAID is not a backup.
Only just finished setting mine up the other day actually. I use CrashPlan to back up all my computers to my local server/NAS (which has raid-Z to protect against at least one disk fialure), and also to CrashPlan’s cloud. Works pretty nicely so far, though obviously I haven’t had much time to test it. Probably will add an external or two into this scheme as well and keep them at the office for a third level of backup.
@ Greg:
That is classified as “unpredictable and catastrophic failure,” or at least unpredictable. I don’t believe that I stated that raid was a backup but that backup was a uneconomic solution to predictable hdd and ssd failures. This is relevant if you read the original post where Luke is having to deal with multiple and frequent hdd failures. This is liable to happen when you make frequent full-disk backups to commodity drives as they aren’t designed for that kind of use.
@ astine:
Fair enough. As in everything, your risk management strategy depends on your risk assessment. To me, software bugs, data corruption, accidental overwrites and viruses are common enough that I would not classify them as unpredictable, but that’s a judgment call. You have to weigh the value of your data against the cost of protecting against those risks multiplied by the estimated probability of the risks, and everyone has to make that calculation for themselves.
I have a linux box with a RAID 5 array for personal pictures and videos and use dropbox for all important stuff. My wife has Mozy for all the spreadsheets, etc on her laptop. It isn’t perfect by any stretch- but it works for me.
A couple years ago I was luck to stumble across datastorageunit.com during its early adopter stages, so I scored unlimited space for $3/month. All they give you is an account for sftp/rsync, shell, etc. access and you have to do the rest. The terms may have changed since I signed up, so read carefully.
So I set up a nice bunch of rsync scripts and anacron and poof I have daily mirroring. It seems like a pretty small operation, but if they’re honest (and I believe they are, based on their handling of a few failures) they have regular offsite backups of their own servers, and their uptime is good. Plus they actually tell you what’s going on, like “we had major hard drive failures, so we’re copying everything from our secondary server onto a new machine”
@ reacocard:
How did you get CrashPlan to backup to your NAS? Last I checked it still had trouble backing up to network shares.
Documents/Pictures/important configuration files/mail — laptop/desktop bi-directional sync with home server using unison.
Homeserver — incremental backups of all synced docs/media/configurations/mail to 2nd internal hard drive. That backup also copied to external hard drive and to a 2nd external hard drive in a separate room (server is in the garage)
Offsite — complete backup from homeserver using duplicity (encrypted — excluding the rdiff-backup directory because duplicity does it’s own differential backup).
I save package lists and crontabs as well. Reinstalling Arch with that + /etc and /home configuration files is very fast.
Scott
Almost everything I consider important is versioned with Git — my blog, projects, etc. Everything personal I version with Git is on GitHub. Therefore almost everything personally important is backed up to GitHub. :-) It’s the Linus Torvalds method of backup.
Cloning a repository creates a full backup from GitHub and I’ve got clones of everything on multiple computers, at home and in my office at work. Also, people have forked my best stuff on GitHub, so those are completely out of my control. It’s all very safe, I think.
The latest Show Me Your Desktop had me reconsider how I was managing my configuration dotfiles (I mentioned KDE being bad for this), so I have since set up versioning for that too. Before, I had a pile of dotfiles I’d manually copy over to a new machine, and they were “backed up” just by being on so many different computers (some of them not connected to any public networks).
Greg mentioned having to re-install a system from scratch being time-consuming, so it should be images and backed up. Not for me! Throw in the Debian netinst CD, go through a basic install (easy), and once I boot into my system I apt-get my favorites. Then I clone my .emacs.d.git and my dotfiles.git and I’m back to my entire comfortable, productive configuration within one minute — including my editor, shell, window manager, PGP keys, browser, wallpapers, and terminal emulator. It’s seriously like I’m using the same computer everywhere I go. My dotfiles.git being rather new but I’ve already tested it successfully on several machines.
Do you know how you back up a Debian system? Easy! :-) (Assuming you didn’t install any debs manually.)
dpkg --get-selections > backup.txtJust hand that text file list to the new system and it will install all the same software again. You can use debsums to locate your local configuration file changes, too. However, I’ve never used either of these for backups, because installing things on the fly as I need them works just fine for me. Backing up my OS isn’t something I need. My dotfiles capture all the important deviations from the defaults.
So, that leaves some non-versioned important files. This amounts to essentially just my wedding photos. Really, that’s it. I have a full copy on my home computer, my wife’s computer, and a friend’s computer who offered to let me store some offsite backups on his computer (in exchange for some space on mine). Several family members have full copies too. If there were more pictures that I cared about, I’d just upload them to some popular album-sharing website, or to S3, and keep a local copy. That’s enough for me, as long as I can pull down a full copy easy — for when I lose my local copy.
I still have a ton of data other than this, of course, but it’s all widely published things: games, films, music, etc. All this stuff is available easily through *ahem* torrents, so you don’t need to worry about backing it up, even when you’ve bought it by legal means.
All that so far doesn’t cost me anything. I’m frugal … or maybe I’m just a cheap bastard.
For work, I’ve learned from other people that the official backup service we use is worthless. There are horror stories where people have lost almost everything because they couldn’t get their stuff out of the backup system. It’s also Windows-only, so I’m not even running it. :-) Again, everything significant is versioned in Git and everyone involved has a clone (a full backup). I personally keep a clone of each on both my main computers, my thumb drive, and I keep a bare clone on my network share “U drive”. No GitHub since this is work stuff, obviously. There’s nothing quite “offsite” in all this, but it is distributed across several buildings. If somehow all these buildings were destroyed at once I’d probably be dead anyway, so I’m not concerned.
I know all this stuff works very well because I had two hard drive failures in the last year: one at work and one at home. I also very recently suffered data storage failure of sorts (software-only) with the extremely-buggy Ubuntu encrypted home directory feature (DON’T USE IT!). Despite this, not a single byte of data was lost and I was able to continue along with new hardware as if nothing had happened.
One last note: with all these networked machines and the cloning, someone malicious could potentially gain access to a machine and use all this network access to trash backups on other machines (i.e. they got onto my main desktop and used it to access GitHub and destroy all my repositories by force-pushing empty branches). In the case of Git clones, it’s all done over SSH and my SSH keys are all encrypted by different passphrases, so there is a layer of protection there, too. They’d need to observe my decrypted key or passphrase in addition to gaining access to my machine. I also clone by http:// or git:// (i.e. read-only) when I know I don’t need to push commits back out, so there’s no trashing the remote repository in these instances. Finally, my home and work computers aren’t directly accessible to each other — so not all my machines are accessible from one location.
Geeze, I need to start making blog posts out of these huge comments. Luke, if you ever became a cop and then had to investigate me, all you need to do is propose a related technical question and I’ll spill my guts on the subject.
I’m much less… rigorous with my backups than everyone else it seems. All of the files I actually care about/couldn’t replace are backed up to Dropbox, and I’ll probably switch to Google Drive whenever they release a client for Linux. I have an external drive with much more space than my laptop, so I write backups to there.
Since my triple booting macbook has a fragile partition structure and hybrid mbr, I use image backups. These are really just there in case I break something in one of the OS’s with my habit of tinkering without knowing what I’m doing. I make an image of the whole drive and images of my mac and windows partitions every 4 or 5 months — that’s about how often they see meaningful changes. All of my files are in my Linux /home partition except my music, but that’s mirrored onto my phone.
I make an image of my /home partition as an actual backup about once a month. I recently decided I wanted even more frequent backups of my /home partition, and for now I’m just using deja-dup which shipped with Fedora because I’m too lazy to look for a better solution.
For images, I decided that the simplest solution was to pipe dd into pbzip2. I haven’t settled on a solution for the full HD image, but I think I like booting to Knoppix.
Since my external is so much larger than my laptop’s hdd, I have some less important files there. Those are in turn duplicated on another older/smaller external drive. Since I add new files to there infrequently, I just manually copy them.
My solution is to not care all that much about my data. :P No, seriously, for my personal stuff I mostly don’t care. But I’ve never really been that into pictures / video, and any projects I care enough about end up on the work machine as well. And I figure, if something catastrophic happens to both my home and work, my computer files really aren’t going to be my biggest concern.
However, I have no idea what to do for work. It’s a mess; I’m supposed to save all my work on the server, but it’s a POS re-purposed desktop, and trying to actually work on files from there in AutoCAD is just… painful. But I don’t know what I can do for syncing / version control when I’m dealing with 1) Mostly files in a non-text format [dwg, pdf] and 2) Have no authority whatsoever to touch / even suggest anything be done on the server.
Simple but so far it worked:
All the pictures/videos are copied manually on 1 external drive, 1 home file server and my laptop (always do that before erasing card).
Personal work (programming, etc) – tgz that goes on external drive, home server and another server that I keep at work (I have an understanding to keep one small personal server here)
The rest (documents, mails, settings, etc) – iBackup on external drive.
Went thru 2 laptop hdd crashes and 1 external hdd crash; nothing important lost so far.
In recent years, I’ve had 3 hard drive failures. Of which one was the backup drive which failed four days after my media drive did. In short, I lost all my music, movies and whatnot twice and I’ve learned to live with it. All the data that’s really important to me is located on Dropbox, Google Drive, SendIt or whatever.
I always think: will I need this data in three months? Yes: put it online. No: just put it on that disk you’ve got lying there.
I have 50GB Dropbox plan which is also at 3 of my computers at a time. Also, I backup my work files with Amazon S3 via Arq but that is still free because there’s not much space occupied. I back up my Macbook Air with Time Machine to external drive and I have a 2TB NAS which works as 1TB mirror drive in my home network where everything else is backed up again. I also keep my photos in private sets on Flickr ‘just in case’ but there’s nothing important there.
I also got into practice of using Bitbucket for my own projects, again, ‘to be on the safe side’. All of this means that I’m screwed if there’s no internet connection :)
I guess I am old school. Yep, I actually use tape at home. I can’t tell you how often I have had to dig up a ten to fifteen year old tape to recover data at my job. As long as you keep it in a magnetic field free environment, it should be able to reliably keep your data safe for half a century. There is only one trick to this, keeping a device that will be able to recover that data later. Tape drives can stay operational in cold storage for long periods of time as well, unlike traditional disk drives. The only problem I am starting to se e is the evolution of the communication path. SCSI used to rule and was pretty much backward compatible (lets ignore that thing call high voltage differential)@ Craig A. Betts:
I have this little script called dbup.sh which is executed daily. It calls bup.sh (creates backups in .tar format), copies .tar files to ~/Dropbox/bup/, removes over two revisions old backups and executes dboxsync (syncs my dropbox). It has worked pretty well as my backups are just 180MB in size.
Picasa, Google Drive, Google Music, and the Android/Chrome bookmarks that sync up so I don’t lose them all again. That covers my off-site, and availability on my phone/tablet/laptop.
I have two external drives, one of which is about ten years old and still plugging along, though it’s slow as hell, planning to get a 1TB Passport Essential USB 3.0 and the Nomad case soon for my in the safe backup and occasional SneakerNet needs.
Also getting a Tonido Plug for backup and streaming purposes. From what I gather that should work with the Roku, GoogleTV, PS3, phone, tablet, and multiple OSes on desk/laptop computers.
Additional backups go on the MicroSDHC cards, 32GB for under $20 and small enough to fit several in a credit card slot, and I have a Corsair Flash Survivor for the whole run over by a Jeep and still works rugged storage, though I need to replace it now that they have 32GB USB 3.0 versions for $37.
The only problem I have is that I still haven’t taken the time to set up any automatic backups so all of that is done manually. I’ll do it someday, but it seems that anything I find that works well on WIN7 sucks on Linux, and all the pretty toys for Linux are not available for Windows.
For starters I have my Documents and project (mostly programming) folders on Dropbox, but that’s mostly for historical reasons. I actually almost lost some data (managed to restore from another source) once due to some failed Dropbox syncing, so I very much do not trust it for backups.
Now, all my backups are handled on my Debian server. It has a samba share which I use to store everything important, which is synced to another drive several times a day using rsnapshot. It’s an awesome rsync-based tool which, apart from just backups, also gives me several points in time to go back to without using any significant amount of extra space. I also sync my Dropbox folder to the server so it gets backed up aswell.
On top of this I also have an old external 320GB LaCie drive connected to the server, to which a subset of data deemed important enough (mostly stuff that’s hard or impossible to reproduce) is synced every night. I consider this a black box of sorts, something I can grab if the house is on fire, or take with me when I leave town for more than a few days. Just in case.
For off-site backup I use Amazon EC2. I have small Linux instance with, currently, a 50GB EBS drive attached. I’m a bit paranoid about storing my data off-site though, so naturally it’s encrypted. On semi-regular intervals I mount an encfs filesystem on a local “staging” drive and sync my super-important data to it, followed by running a script to start my EC2 instance, sync the encrypted data to it, and stop it.
I’ve often considered getting some sort of storage device I can always carry, preferrably in my wallet, but have yet to find anything convenient and not really expensive with enough space to make it worthwhile. Granted, I’ve never really looked *that* hard. Maybe I will now.
Regarding bare metal restores, since I don’t really store anything important on my clients the only real concern is reinstalling the OS. For that I have a Debian server that handles PXE bootup and netinstalling Windows 7, which is my client OS of choice.
…datastorageunit.com actually seems to be exactly what I was looking for before settling on EC2. Why couldn’t I find that before?!
Sounds like you’ve got a pretty good regimen in case of hard drive failure. A four drive backup is exactly the kind of things that we tell our customers to do, but you can almost hear their eyes rolling in their head when they hear this.
The cloud backups are now so easy to use that I don’t understand why people aren’t using them on a mass basis. It saves expensive data recovery service charges in the long run, believe me! ;)
4 bay esata external hard drive enclosure. 4 truecrypted 3.5 inch hard drives and a single 2.5 inch hard drive (2.5 inch for carrying with me day to day). Incremental backup once a week. Rent a seedbox. Encrypt and dropbox anything absolutely critical.
Unlike most people, i like to have my movies, music and tv shows backed up which requires this setup. I spent a lot of time grabbing all that stuff and don’t have to hunt for and wrangle it all together again.
NAS solution doesn’t work for me since I’m on the move occasionally.