[Discuss] lvm snapshot cloning

[markw at mohawksoft.com (markw at mohawksoft.com)]

> On Oct 25, 2011, at 7:51 PM, markw at mohawksoft.com wrote:
>>
>> The snapshot has no effect on the master, and yes, we've already said
>> and
>> we already know it is a weakness in LVM that if you don't extend your
>> snapshots you lose them. This can be mitigated by monitoring and
>> automatic
>> volume extension.
>
> You missed it.  This isn't about what happens to master.  It's what
> happens to b when a disappears.  If master<-a<-b and a disappears due to
> reaping then b becomes useless.  Or b is reaped, too.  Either way you're
> dealing with data loss.  This is why LVM will not do what you originally
> asked about.

Actually, in LVM 'a' and 'b' are completely independent, each have their
own copy of the COW data. So, if 'a' gets nuked, 'b' is fine, and vice
virca.

If you know the behaviour of your system, you could allocate a large
percentage of the existing volume size (even 100%) and mitigate any risk.
You would get your snapshot quickly and still have full backing.


>
> Monitoring has problems.  If the volume fills up faster than the monitor
> polls capacity then you lose your data.  If the volume fills up faster
> than it can be extended then you lose your data.  If the volume cannot be
> extended because the volume group has no more extents available then you
> lose your data.  Like I wrote at the start: LVM will quite happily bite
> your face off.

Allocate more space at the beginning. It will never be bugger than the
original volume.

>
> Now, to address your most recent question:
>
> How do I back up a 1TB disk.  Think about this: how do you intend to do a
> restore from this backup?  The most important part of a backup system is
> being able to restore from backup in a timely fashion.

That is all dependent on whether or not we are recovering from a
"disaster" i.e. main data loss, or for access to a previous incarnation
within a working environment.

In a "working environment" it is possible to use LVM snapshots, backed
with enough storage, of course, to receive the deltas from where it is to
where it should be.

In a disaster, well, brute force will save the day.
>
> I have in production a compute server with two 8TB file systems and a 9TB
> file system, all sitting on LVM volumes.  I have an automated backup that
> runs every night on this server.  It's an incremental file system backup
> so I'm only backing up the changes every night.  This is, as you might
> expect, quite faster than trying to do full backups of 25TB every night --
> which I can't because it would take three days to do it.

This works when you have access to the file system. If you expose an LVM
volume with something like iSCSI, you best hope that Linux has a file
system driver for it.

Even so, you will be backing up full files. In a system with large files,
like a database server, you'll be backing up a lot of data that doesn't
need to be backed up.

>
> On smaller capacity volumes, in the several hundred GB range, I use
> rsnapshot to do incremental file snapshots to a storage server.  Again, I
> don't back up the raw disk partitions every time.  I only back up the
> changed files.

Most file formats don't change too much. This is, of course a
generalization, with a number of exceptions. I refer back to the database
example.

>
> In both cases -- and in fact with all my backups -- they are file level
> backups.  The reason being that if I need to restore a single file or
> directory then I don't have to rebuild the entire volume to do so.  I can
> restore as little or as much as I need to recover from a mistake or a
> disaster.

I can understand this, however, snapshots work too. One of the utilities I
am writing allows you to clone a snapshot.

Assuming you always have one snapshot that is a point in time reference.
You can clone that snapshot and then apply diffs to get back to a specific
point in time.

While it is a judgement call, it is not clear which is more efficient. In
the case of a database server, it may take much longer to restore a full
file than it does to apply diffs.

>
> Suppose the case of a live volume that needs to be in a frozen state for
> doing a backup.  Database servers are prime examples of this.  Here, I
> would freeze the database, make a snapshot of the underlying volume, and
> then thaw the database.  Now I can do my backup of the read-only snapshot
> volume without interfering with the running system.  I would delete the
> snapshot when the backup is complete.

That is the most direct method of using snapshots, yes.

>
> If I were using plain LVM and ext3 for my users' home directories then I
> would do something similar with read-only snapshots.  There would be no
> freeze step, and I would keep several days worth of snapshots on the file
> server to make error recovery faster than going to tape or network
> storage.  As it is, I use OpenAFS which has file system snapshots so I
> don't need to do any of this and users can go back in time just by looking
> in .clone in their home directories.  I still have nightly backups to tape
> for long-term archives.

Seems complicated.

>
> Now, time to poke holes in your proposal.  I have a physics graduate
> student doing his thesis research project on a shared compute server along
> with a dozen others.  They collectively have 7.5TB of data on there.  This
> is a real-world case on the aforementioned compute server.  Said student
> accidentally wipes out his entire thesis project, 200GB worth of files.
> It's 9:30 PM and he needs his files by 8am or he fails his thesis defense,
> doesn't graduate and I'm looking for a new job.
>
> With my file level backup system I can have his files restored within a
> couple of hours at the outside without affecting anyone else's work.

Obviously, there are pros and cons to various approaches. You approach is
only faster if you know that the file in questions was recently modified.
If that is the case, you lucked out. What happens if the source of the
calculations, which is also needed, hasn't been modified in some time?

>
> With your volume level backup system I would spend the night on Monster
> looking for a new job.  The problem with it is that I can't restore
> individual files because it isn't individual files that are backed up.
> It's the disk blocks.  I can't just drop those backed-up blocks onto the
> volume.  Here:
>
>   master->changes->changes->changes
>        \->backup
>

> If I dumped the backup blocks onto the volume then I'd scramble the file
> system.  Restoration would require me to replicate the entire volume at
> the block level as it was when the backup was made.  This would destroy
> all the other researchers' work done in the past however many hours since
> that backup was made.  I would fire myself for gross incompetence if I
> were relying on this kind of backup system.  It's that bad.

totally wrong!!!

lvcreate -s -n disaster -L1024G /dev/vg0/phddata
(my utility)
lvclonesnapshot /dev/mapper/vg0-phdprev-cow /dev/vg0/disaster

This will apply historical changes to the /dev/vg0/disaster, the volume
may then be used to restore data.

>
> It gets worse.  What happens when the whole thing fails outright?  Total
> disaster on your 1TB disk.  Now it's not just 29 minutes to restore last
> night's blocks.  It's two hours to restore the initial replica and then 30
> minutes times however many deltas have been made.  Six deltas means 5
> hours to do a full rebuild.  I can do a complete restore from TSM or
> rsnapshot in half that time, maybe less depending on how much data was
> actually there.

You have a similar issue with file system backups. You have to find the
last time a particular file was backed up.
>
> Data point: It takes ~19 hours to restore 7.5TB from enterprise-class
> Tivoli Storage Manager over 1GB Ethernet to a 12x1TB SATA (3Gb/s) RAID 6
> volume.  I had to do it this past spring after the RAID controller on that
> volume went stupid and corrupted the whole thing.

Yes, and it should be MUCH faster!!! I agree.
>
> --Rich P.
>
> _______________________________________________
> Discuss mailing list
> Discuss at blu.org
> http://lists.blu.org/mailman/listinfo/discuss
>