Re: Re: Perfomance problem backupscript (more details and tests)

On Thu, 2005-10-06 at 10:33, [email protected] wrote:

> How does GNU-tar detect if a file vanished between
> two backup levels ?

--listed-incremental also turns on an option to itemize each
directory's contents on the tape along with a flag for whether
the tape contains a copy of the file.  When restoring with the
-G option, it removes any file currently in the directory that
is not in the listing (i.e. was not present at the time that
incremental was taken). 

> To my theory, star does this when joining updated
> directories with the file bodies during restore.
> Those which cannot be attached to any directory are prone
> to removal, i guess.
> But how does GNU-tar remove obsolete files at restore
> time ? Does it delete the whole directory before restoring
> the new layer ?

No, just the files that weren't there at the time of the
incremental run.  And it is optional.


> >> Googling for "tar" "listed-incremental" i find problem
> >> reports about VFAT ... SMBFS ... FAT.
> > 
> > I don't think it's obvious at all that inode numbers
> > would change if you mount/unmount a device even if
> > they are faked by computation.  Does this only happen
> > if you remove a device, mount it elsewhere and modify
> > it, then remount in the original location for the
> > gnutar incremental or does it happen on any remount.
> 
> No idea. It's google hearsay. But very common one.
> I use ext2 and reiserfs on disk.

It is probably very common now to use usb drives (both flash
and real) and swap among systems.  I'd expect synthetic inode
numbers to change if the contents are modified between mounts.

> But i was pointed to SMBFS when asking my users about
> their opinion on an inode-based content test.

I'm not sure this is a bigger issue than what happens
'under' the mount points.  If I arbitrarily mount different
volumes on top of an existing directory between fulls
and incrementals, what should an incremental tar contain?

> > On unix-like filesystems with valid ctimes all you need
> > is the ctime of ordinary files since that catches any
> > possible change - including ownership and permissions
> > which your method appears to miss.  The problem is a
> > renamed directory containing old files.
> 
> I see. Old mtime of directory, old mtime+ctime of files,
> no reference model of the tree to lookup wether the
> files have been present under that address previously.
> 
> Who takes care of the outdated copy of that directory
> at restore time ?

Gnutar will delete the old one because it won't be listed
in the contents of the parent directory.


> > Actually, my first choice in backups these days is
> > backuppc http://backuppc.sourceforge.net/ which
> > can use several methods including rsync to transfer
> > the copy and keeps compressed copies online with hardlinks
> > for all duplicates so you can store about 10x what you
> > would expect. 
> 
> Loveable concept for hard disk backups.

I seriously recommend it.  Toss a huge ide drive on a linux box
that doesn't do anything at night (I use my desktop machine),
set it up and ignore it until you need an old copy of something.
I haven't done a tape restore since setting mine up a couple
of years ago although I still make tapes and ship offsite for
disaster recovery.


> > Any filesystem with native unix semantics must have
> > fixed inodes and the ability to create hardlinks by
> > having any number of directory entries pointing to the
> > same inode. 
> 
> Users (the experienced and skilled ones) told me
> that they are uncomfortable with any reference to
> inodes.
> That is because inodes form a directed graph but the
> user model of data storage is a tree. Hardlinks are
> handy but introduce severe fundamental problems.

You don't want hardlinks of directories, but hardlinks
of files are fundmental to unix filesystem semantics.

> > Backup methods must be able to re-create
> > these hardlinks - which isn't easy because all they
> > have in common is the inode number.
> 
> Interesting point.

Normally during a restore, all files get new inode numbers
so the restore process has to keep a table mapping the
old/new numbers for everything with more than one link
and search for matches on each new one.  I haven't found
anything that does this efficiently.

>  I would agree to :
> "For situations where hardlinks do matter, backup methods
>  must be provided which allow restoring of hardlinks."

Rephrase that to 'where unix files matter'.  All unix files
exist as hard links.  The link count can vary from 0 to some
large number.  Having a link count of 1 is a common case
but by no means special.

> The problem with hardlinks is that they cannot be represented
> in common formats like ISO-9660. Offer a desktop user the 
> choice of having the personal backup in ISO-9660 or in tar
> and you will see a 90% majority for ISO. It's so handy
> for picking files after small mishaps.

Archival storage formats like tar/cpio/dump exist for a reason.
ISO-9660 doesn't represent unix owner/group/permissions/setuid/setgid
either.  Omit them on a restore and your box won't work.

> I delegate those tasks to the backup formats which i offer:
> ISO-9660, afio and star. I will have to check wether
> star and afio do plug hardlinks together on restore.

Any system that expects to restore a working unix box
should deal with hard links.  Look, for example, at the link
count of /bin/zcat.  You may not break things immediately if
you treat each link as a new file other than wasting space, but
subsequent updates may eave files unchanged when the contents should
have been replaced.

> > Tar only cares if you do --listed-incrementals, and then it
> > only cares that the directories have the same inode
> > number on each pass (or the fake computed equivalent)
> > for the directories.  If they don't match, the failure
> > mode is to copy everything which is probably the best
> > way to fail.
> 
> So GNU-tar not only ignores the existence of inode-problematic
> filesystems 

No, it relies on the semantics of stat().

> but also the existence of SuSE 9.0 ReiserFS ???
> (It is the _default_ filesystem of the toadies.)

If the filesystem says a file changed, it is taken in an incremental.
If your filesystem doesn't reliably note changes, then you should
not trust incrementals.

> > All unix filesytems have ctime values that are updated
> > with any inode change. 
> 
> When i started my own backup project back in 1999 
> Linux did not change ctime on rename. I was not amused
> but reading  man 2 stat  told me why this was the fact.
> There was only ext2 and ext2 is rustily based on inodes.

Well, linux wasn't really unix, especially back then. 

> > 
> > ReiserFS doesn't completely follow unix semantics.
> 
> Here you can see that the defense line of demanding
> ctime to behave against old traditions cannot be upheld.
> 
> Actually the ctime behavior which you demand (and which i
> would actually prefer, too) contradicts all old man pages.

I've never seen ctime described as anything but inode change
time.  What man page says otherwise?

> It is an underground quirk introduced to cover a flaw
> in the inode based filesystem model. 

No - that was the definition from the start.  However,
atomic rename() was a subsequent addition that allows the
confusion about the requirement.  Originally, the only
operations were link() and unlink() and a rename operation
was two steps that each necessarily affected the link count
in the inode.  Rename() as a system call was an afterthought
(a good one, but the implementation did not need to break the
filesystem design).

> But such patchwork is not reliable. You are too likely to
> encounter something that is done correctly and does not
> play well with the shabby quilt.

Place the blame where it belongs.  The original design and
specification was solid.

> >> "The field st_ctime is changed by writing or by setting
> >>  inode information (i.e., owner, group, link count, mode,
> >>  etc.)."
> >> "Moreover, st_mtime of a directory is changed by the
> >>  creation or deletion of files in that directory."
> >>
> >> "Unfortunately, however, the change was not pushed
> >>  into 2.3, and so 2.4 and now 2.6 have a ext2 rename()
> >>  implementation that does not update inode ctime."
> > 
> > Rename is kind of a special case.  The inode in question
> > doesn't necessarily change because a directory entry pointing
> > to it changes. 
> 
> Exactly. No inode change. Dire, but such are the rules.

So don't use incrementals on such filesystems.  And blame the
rename() implementation for not matching the original unix
semantics.

> > Most implementations would do it by creating
> > a new link with the new name, then removing the old one, and
> > changing the link count in the inode requires a ctime change.
> 
> This i would call a bug if i compare it with the behavior
> prescribed by the man pages.
> It is a nice bug, i have to confess.

It was the original unix method.  Apparently whoever did the
current rename() has forgotten their roots and the fact that
the original unix filesystem semantics had no possible way to
rename a file without changing ctime.

> > Rsync against a complete on-line copy should always
> > be able to detect any change...  And backuppc can do
> > it against compressed copies.
> 
> That comes near to my own methods which are less
> bandwidth conserving than those of rsync, though. I only
> do whole files. On the other hand, i can leave out many
> content checks by safe timestamp considerations. (Plus i
> can make use of inode numbers if they are reliable.)

One of rsync's options is to trust the directory/inode info
so if you have a match on name, timestamps, modes, length, etc.,
it doesn't bother reading the file at all.  Another mode makes
a pass reading the file at both ends but only exchanging block
checksums unless there is a mismatch.  Backuppc uses the former
for incrementals, the latter for fulls which are much slower but
don't use a lot more bandwidth.

> dump was a pain in the ass forever :((
> dump and DAT are the reason why i began to develop own
> backup software. :))

It's real problem is just that it is tuned to a specific filesystem
type and just doesn't deal with things like converting directories
into mount points.

> Thanks for the hint with the hardlinks. I confess not to
> have ever thought about refiddling those at restore time.

I'd like to find something that handled them efficiently.  For
example, my backuppc archive which is about 100 gigs of real
disk space but would be around 900 without the compression and
linking is basically impossible to copy with any of the normal
files tools.   I have built it on a software raid mirror and
periodically sync to different external drives, which takes a
couple of hours.  A tar | tar kind of copy would take many days
due to the inefficient way the links are resolved.  I'd imagine
your backup technique would take days for even the first copy
if it reads each instance of the links separately to save a
checksum entry for it.

-- 
   Les Mikesell
    [email protected]