[…] Sync is great. I have been using it ever since Scott recommended it in the quintesential backup thread of 2013. It even made it onto my big list of essential tools. It […]

[…] Moral of the story: BACKUPS ARE GOOD! […]

I do them manually. Every saturday morning, before breakfast. It’s become sort of a ritual. I know i only skipped one in the last year, because i make hardlinked snapshots named by date, and that one time that i missed was when i was on holiday. But then again, i am a very rigid and routinish person. The worst that has ever happened to me on a saturday morning was waking up 2 hours late because i had had 2 beers on friday and stayed up beyond midnight.

In addition to this, i use a similar script (again rsync plus dated, hardlinked snapshots) that i regularly invoke upon critical folders that i am currently working in, and it makes a second snapshot to the house of my parents way outside the nuclear blast zone. I practically do than one after each paragraph i write, and i now have 1000 versions of my manuscript.

this is what i use, maybe someone will find it useful:

#!/bin/bash
# TODO: Description

exit 0


I don’t have enough fingers on my hand to count the number of people saying “oh, noes, I lost everything I was working on the last 6 months!”. Dude, you have Dropbox, if anything, just use that!

Myself, I’ve considered buying a small NAS for some time now, since the amount of data I store grows steadily and is starting to be a bit big, scattered everywhere and unwieldily for Dropbox.

And since I really like distributed storage solutions (Git, Dropbox,…), there are a few Dropbox-like alternative you can use, too — we tried Sparkleshare and ownCloud at work, but both have a lot of issues, rendering them pretty much unusable at scale. Seafile sounds interesting, but right now we seem to be headed towards the one built-in in our Synology NAS.

I’ve also started considering backing up my GMail mails with OfflineIMAP. I’m still not really comfortable with it since, at heart, it’s not really a proper backup solution but rather a sync mechanism. I use `rdiff-backup` to make periodic snapshots of my mails, but if you have a better solution, I’m listening!

Luke Maciak wrote:

@ Scott Hansen:
TIL about BTSync! Thanks! I’m messing around with it right now. Does it work over firewalls? It claims it does. I guess I’ll find out because the one we have at work is a nasty beast (I can’t even ssh out unless it’s to approved machines).
One downside of this over say Dropbox is that it doesn’t sync if one of the machines is asleep. I dropped some files in my BTSync folder on my MacBook and I’m trying to sync them to my work computer, but it’s not working because the Mac is likely in sleep mode now. Dropbox syncs via cloud so it doesn’t matter if machines are live or not.
Still, this is pretty cool. I’ll definitely have some good uses for it.

There’s some info about working around restrictive firewalls using the tracker server and even a relay server for btsync. I try to keep that stuff turned off when possible, but sometimes you need it! I had to resort to using a VPN connection to privateinternetaccess.com over port 443 from my work because of the massive port blockage going on there. This actually lets btsync work to sync the laptop to my home machines and my phone!

I have an always-on home server and a cheap VPS that I run btsync on. Sort of acts like the Dropbox server + redundancy. The biggest portion of what I sync is media (books, music, video) from laptops to home server. Smaller sync folders that also exist on my phone and the VPS are for syncing phone camera pix, keepass database, ssh keys, wallpapers and dotfiles.

For you late comers here…btsync is NOT a backup solution! Everything sync’d with btsync gets formally backed up with rdiff-backup and Crashplan!

Scott

@ Ethan:

Time machine is great for local backup. Dropbox and iCloud are a bit limited in scope though (as in you don’t want to use them to store your uncompressed vacation pictures for example). A robust cloud backup solution like crashplan/mozy/carbonite might give you an extra piece of mind. :)

Also consider scripting/automating your “manual backups” because you will probably forget them right before you need them. True story: my coworker used to religiously back-up all her work to a thumb drive at the end of the day. Last week she had to get like 6GB of data from the client so she deleted all her backups to make space for it. Since the drive was almost full, and she didn’t feel like cleaning it up, she did not do her manual back up routine that evening. Next morning the machine would not boot.

We ran the diagnostic on the HD and it lit up red – bad sectors everywhere. Of course the volume was encrypted with Checkpoint FDE so to get any data off of it we had to decrypt it which was of course impossible with cyclic redundancy disk errors popping up everywhere. A live recovery CD from Checkpoint would also BSOD when trying to mount the faulty drive. Checkpoint support dudes were like:

¯\_(ツ)_/¯

To make a long story short – she lost all her w0rk, and also bunch of personal stuff (pictures, documents, etc..) that she was storing on the company laptop for whatever reason and never thought about backing up to an external drive.

Robert wrote:

The one day you forget to put in the new tape is the day before the server crashes.

This is so true it is not even funny. :P

@ opti:

Haha, I did something very similar. At one point I decided to “upgrade” windows by means of a clean install on a new bigger hard drive, and then transfer my data from the old drive. This is on a desktop mind you, so I just plugged the new drive, popped in the Windows XP CD and started installation. Few hours and 27 reboots later (cause it’s Windows XP) I go to my 2nd drive to transfer data, saved games and etc and Windows is like “LOL, not formatted”.

So yeah, installed Windows on the wrong drive. All data lost.

At first I was like: ⊙▃⊙
Then I was like: (╥﹏╥)

@ Scott Hansen:

TIL about BTSync! Thanks! I’m messing around with it right now. Does it work over firewalls? It claims it does. I guess I’ll find out because the one we have at work is a nasty beast (I can’t even ssh out unless it’s to approved machines).

One downside of this over say Dropbox is that it doesn’t sync if one of the machines is asleep. I dropped some files in my BTSync folder on my MacBook and I’m trying to sync them to my work computer, but it’s not working because the Mac is likely in sleep mode now. Dropbox syncs via cloud so it doesn’t matter if machines are live or not.

Still, this is pretty cool. I’ll definitely have some good uses for it.

I have been adhering to this philosophy for quite some time now:

* Btsync (used to be unison) to keep media files, coding projects and dotfiles in sync between my laptop and server.
* Backup script to backup cron jobs, package lists and the family calendar (saved from Google Calendar)
* Rdiff-backup for nearly complete incremental backups of the laptop and server to separate hard drive on the server (/mnt/backup)
* Duplicity for encrypted incremental backups (not including rdiff-backup data) of /mnt backup to a another external hard drive (/mnt/backup2)
* Headless Crashplan on the server to save /mnt/backup (I used to sync the duplicity backups /mnt/backup2 to a shared web host, but they finally decided they didn’t like that)
* Manual upload of all family pictures to Smugmug

I’ve nagged wife and kids for years to save their stuff to their network folders on the server….yeah right :-) Just yesterday I heard without sympathy the “I loaned my flash drive with my big school project on it to my sister and she lost it….”

Scott

And now for a grand horror story!

Recently my computing center experienced the jackpot of system failures. Our center does high performance computing in academia, so there’s quite a bit (1-2 PB) of expensive data we store (I suppose you could argue most centers store expensive data). Well, we had this nasty little combination of drives/controllers/firmware that caused a catastrophic loss of data on a couple disk trays that stored the home directories for about 75% our research groups. Supposedly this particular failure has only happened one other time in a production scenario, and we were just LUCKY enough to experience it. Thankfully we did weekly backups with nightly incrementals, but it took three weeks to finish restoring and verifying the data from tape, mostly because the center can’t afford redundant filesystems. And of course, it put research effort at a standstill while all that important data was tied up, which is pretty bad when all of your funding comes from research grants. But that isn’t the complete story.

See, this story is also about how I almost lost all of my personal data. After the aforementioned event, I was like, “Oh SHI— none of my data is backed up”, so I got down to business and set up automatic file syncs, remote backups, physically isolated copies, whatever. I had everything in place to do it except for a couple of drives I wanted to format to ext3 (so that I could get reasonable IO performance out of my raspberry pi file server). Well, one of those drives was where I had single copies of data, and it was late at night and I was being stupid, and accidentally turned that drive into EXT from NTFS and started formatting, thinking I was working on the other nonessential drive. Thankfully, I was alert enough to realize that I had made a major mistake and stopped it, but I had done enough damage that I couldn’t repair the partition. And of course, that was my only copy of certain data that I actually wanted. By this time it was already about 2:00 AM and I was in a panic, and ended up staying up the rest of the night recovering the data and shuffling stuff around on my main desktop so I had enough room to recover the data. Fortunately (and I’m assigning a 1/100,000,000 probability on how lucky I was), I got everything back that I wanted. It took me three days and a fair part of my sanity to get it back though, and I have exactly zero desire to do that again. I now have four copies of everything essential at all times. BACK YOUR DATA UP NAO.

Dropbox adds some nice version control, but isn’t really a backup. Same with iCloud, except without the version control.

I use Crashplan, Time Machine, ‘throw the most important files on a thumb drive when I think about it,’ and a full system image that I update periodically. The last two are not serious backups, just an absolute last resort. I don’t consider even a single-tape backup system to be very good. The one day you forget to put in the new tape is the day before the server crashes. (Seen it with my own eyes.)

I’ve never encountered anyone who regretted having too many backups, but plenty of people on the verge of tears from not having enough.

This is an excellent piece. I totally agree with everything it says. My “backup” is currently dropbox, iCloud, and TimeMachine. I also use several other hard drives for a manual “backup” for the really important stuff. Do you consider this to be sufficiently safe?