[Discuss] On-site backups revisited - rsnapshot vs. CrashPlan

[gaf at blu.org (Jerry Feldman)]

On 02/20/2013 04:40 PM, Rich Braun wrote:
> I wrote last month a query about CrashPlan free peer-to-peer software from
> Code42.  I failed to get satisfaction from the vendor, even though the CEO of
> Code42 made a response, you can view the thread at
> https://crashplan.zendesk.com/entries/64160-How-do-I-request-a-full-integrity-check
> ; he didn't follow up any further though.
>
> I am developing an alternative strategy based on suggestions from BLU.  Here's
> what I posted at the CrashPlan forum about that:
>
> I haven't yet found a suitable replacement for CrashPlan (peer-to-peer) off
> the shelf, but here's the strategy I'm using going forward:
>
> * Set up a central backup server using rsnapshot which can easily
>   be set up to make incremental filesystem backups similar to
>   CrashPlan's peer-to-peer mechanism
> * Supplement rsnapshot with a script to make sha256sum checksums of
>   the archive contents, stored in a simple db table
> * Craft a monitoring script to warn me in case the archive files no
>   longer match checksums, and to warn when backups are incomplete
>   or stale
> * Make a tool that makes it more obvious to me whether a given local
>   directory or computer is being backed up
>
> That's all I really wanted CrashPlan's peer-to-peer software to do, but it's
> hard to find out what it's actually doing under the covers.  For on-site
> backups, I don't need some of the other features that CrashPlan provides: 
> encryption, de-duplication, the convenient UI.  But I do urgently need
> monitoring that goes beyond CrashPlan's weekly status emails, along with
> integrity checks that I control and understand.
>
> I /think/ I'm still happy with the paid remote-site backup service but I have
> to supplement or replace my local backups as noted above.
>
> ---
> I'm not sure how aggressive I have to be with the integrity checking -- I've
> actually never had a known instance of a file getting corrupt -- but I figure
> it's worthwhile for a long-term archive.  Have any of you found or developed
> tools for this part of it, in particular doing it in conjunction with
> rsnapshot or another similar tool?
>
> Setting up rsnapshot is fairly easy, though at some point I want to write up
> and post a better how-to for the benefit of future users.  In particular the
> two-step process of "sync" and "rotate" isn't well-documented in the places I
> looked online, and you really want to have a separate script (beyond what cron
> does by itself) to invoke the rotation methods.
>
>
Rich,
I'm not sure what you really mean by sync and rotate in rsnapshot context,.
First rsnapshot creates an exact copy of the directory tree that you are
backing up. Each of these backups are labeled .0, .1, nnn
Where hourly.0 is the most current. In a simple case, let's say that you
are running a daily backup and want to keep 10 copies. In rsnapshot
context these could be named hourly or daily. So you would have daily.0
through daily.9.
The .0 and .1 trees would be identical except for any files that have
changed since the last backup. Each identical file is a hard link. So,
when you kick off the daily script, the first thing it does is to rename
daily.9 to a unique __delete name, then it renames 8 to 9, 7 to 8 until
it hits. It then runs rsync to create .0 using the --link_dest to point
to daily.1.

The bottom line is that the .0 directory is effectively an incremental,
but also a full directory tree. At home I have 6 hourlies, 7 dailies, 4
weeklies and 3 monthlies. Since you are using hard links the only unique
files you have in any directory trees are those files that have changed.
The rotation is done automatically, The __delete directory is not
deleted until the backup is created successfully. I back up the BLU mail
server daily onto my home machine and rsnapshot sends me email that the
backup was successful or not.

Setting up a checksum of a directory tree is pretty easy, You could keep
them in a flat file at the same level where you archive is, so after
hourly.0 is complete, set up an hourly.0.checksum. Just make sure you
rotate it when rsnapshot rotates the directories.

Also, rsnapshot has a decent logging mechanism. Unfortunately, rsnapshot
assumes a Unix/Linux file system (hence hard links), so you really could
not use it to back up a Windows file system in the same way.


-- 
Jerry Feldman <gaf at blu.org>
Boston Linux and Unix
PGP key id:3BC1EB90 
PGP Key fingerprint: 49E2 C52A FC5A A31F 8D66  C0AF 7CEA 30FC 3BC1 EB90