Re: Perfomance problem backupscript (more details and tests)

On Tue, 2005-10-11 at 06:31, Joerg Schilling wrote:

> > Perhaps different constraints would be a better description. 
> 
> I don't see any proof that the constraints are different.

GNUtar backups including incremental runs can traverse
mount points and can have arbitrary exclusions.  The
backup operator does not need to know the filesystem
topology and mount structure or whether it changed between
runs.  The restore operator does not need to know
the filesystem structure of the source of the archive
or whether it matches the target structure. Restoring
renamed files in an incremental run does not depend
on having restored the content to the same filesystem
in a prior restore.   These are _big_ differences in
practical usage.

> > > -	GNU tar depends on the inode numbers of directories and on
> > > 	ctime/mtime
> >
> > Yes, but if they don't match, gnutar includes a new copy in the
> > incremental.
> 
> And whay do you believe that this should help?

I'd prefer any mistake to be in the 'safe' direction - that is
to include the files. The assumption is that a directory that has
the same device and inode number as it did in a previous run is not
new or renamed. If they are different, all contents are taken in
the incremental.  Thus any inconsistency in inode numbering or
mounting will result in the backup including the related files,
not accidentally omitting any.  Including any questionable files
is particularly harmless with a wrapper like amanda that does
size estimates, since it has the option of doing a full ahead
of the planned schedule if it sees that it won't save much space
by doing an incremental.

> It has absolutely no advantage. Please first try to understand how the
> star incremental restore works.

I'm pretty fuzzy on that.  Can you describe how a file renamed between
runs gets the new name hooked to the old content in the restore, and
all of the constraints that have to be met for the old content to
exist in the right place?  Suppose you make a level 0 of a filesystem
containing / and /home, then mv a directory containing old files
from somewhere on /tmp to somewhere in /home (which amounts to a
rename), do an incremental, then restore these to a new machine
where / and /home are different filesystems.   I don't understand
how the old files under the mv'd directory will ever exist on
the /home filesystem.

> Could you tell us why you write this in a discussion where you like to
> convince us why GNU tar includes features that star is missing?

It would be really nice if star offered an option like GNU tar to
make incremental backups that are not limited to single filesystems
and do not require a restore of the previous, possibly unknown
named file to enable the restore of file renamed in the incrementals.
Then star could replace GNU tar as well as dump - and we could stop
talking about GNUtar bugs...

> Ufsdump is extremely sensitive against active filesystems because it first
> reads the inode array _before_ starting to archive directories and then
> file contents.
> Star and GNU tar are less sensitive because they only depend on the 
> set of names in a directory not to shrink during the time this dir
> is archived. 

You are probably right. I'm just having trouble visualizing how
a renamed file avoids being connected to the wrong thing if
concurrent filesystem activity re-uses its inode.   With GNUtar
it only depends on being able to open the file with the name
found in the directory which is a pretty small time window.

> However, if you like to make a reliable backup from a life machine,
> you need to use a filesystem snapshot anyway.

This rules out all of my machines, although I think a reasonable
set of LVM2 tools will be in the next update to Centos so perhaps
eventually this will be usable.

> > GNUtar only cares about device/inode numbers on directories and
> > only checks them against the saved file entry to know if the entire
> > contents are needed or not (to catch old files under renamed
> > directories).  It doesn't care about inodes on files at all except
> > to track hard links on files with more than one name.
> 
> So you like to use the statistical probbability as a gauge?

Everything in the real world is a statistical probability.  That's
why we make the backups in the first place.  If software never had
bugs, if people never made mistakes, if hardware never broke, we
wouldn't have to bother.  What matters is whether you can restore
in spite of all those. 

-- 
  Les Mikesell
    [email protected]