Re: Re: Perfomance problem backupscript (more details and tests)

[email protected] wrote:

> as we are discussing incremental backups :
>
> Do i perceive it right that both star and GNU-tar rely on
> the persistence of inode numbers within filesystems ?

Yes

> Googling for "tar" "listed-incremental" i find problem
> reports about VFAT filesystems which obviously do not
> preserve inode numbers after umount. There seem to be
> similar problems with SMBFS. Not to speak of FAT.

This is incorrect, these people seem to confuse the problems
of a specific impleentation with the filesystem.

All filesystems need to have inode numbers asiciated with
all files in order to be useful in a POSIX environment.

The way this is implemented may be useful or not. In case
that a specific implementation does not retain inode numbers
across reboots or remounts, the related implementation is 
useless.

BTW: this is what we are currently planning to fix for ISO-9660 and Solaris.
Solaris and many other OS that "copied" ideas from the first UNIX
ISO-9660 implementation on SunOS-4.0 does not use the same "inode" number
for all names of a hard linked ffile.

> This is darn slow in comparison to an inode-number check,
> but it is not bound to full featured Unix filesystems
> and it is not affected by filesystem boundaries.
> (Thanks to inspiration by star, i added an alternative
> inode-number check which runs fast and reliable on ext2
> and reiserfs. Other types have still to be evaluated
> but i guess it is ok with most locally attached 
> filesystems on Linux.)

Nobody should use a NON-POSIX filesystemimplementaion for 
every day work, so only POSIX compliant implementations
need to be backed up.

> Nevertheless:
> Can it really be that the big archive programs ignore the
> existence of filesystems which do not comply to the
> inode-semantics of classical Unix ?
> Shouldn't there be some fallback provided in order to
> handle such cases ?

Every filesystem I know and I can think of allows the implementation
of inode numbers (note that POSIX calls them "file serial number").

So your question should be: Can it really be that people
work on an OS that implements non-read-only filesystems in a way that
violates POSIX?


> The ctime trick of my current Linux ext2 is a nice try 
> but i am sure that it can be fooled by using other 
> operating systems (or just other Linux versions) on the 
> same filesystem.

Do not OS that do not follow standard semantics...

> On my SuSE 9.0 the situation is nicely mixed :
>
> mv changes ctime on ext2 but not on ReiserFS.

Then make a bug report against ReiserFS.


> The ctime-quirk seems to be dependend on the type of
> filesystem and on the version of Linux. It has been

It is a depenent of the quality of a FS _implementation_.

> patched into the code without revising documentation.
> It might get patched out of the code as well.
> http://sourceforge.net/mailarchive/forum.php?thread_id=3726620&forum_id=6379 :
> "Unfortunately, however, the change was not pushed
>  into 2.3, and so 2.4 and now 2.6 have a ext2 rename()
>  implementation that does not update inode ctime."

This is not really needed. Needed (for star) is that a rename
of a file changes st_mtime for the related directories.

> Can confusion be any bigger ? 
> So the semantics of ctime is nothing that could be relied on
> for portable incremental backup software.

Star does not rely on it....

Note that if it would, it would not backup a file that
has been extreacted by star from an old tar archive. 
st_mtime would be old and star would (in this case) not
archive file data but only file meta data because st_ctime
would be new.

As a negative side effect, star needs to archive file content
for files that have been renamed (although the extract code
would correctly deal with only a set of file meta data).


> > Joerg Schilling wrote :
> > But the problem is that GNU tar claims to support this kind of backups but
> > has too many bugs with incremental backup/restore to make it useful. 
>
> That's why i prefer to do most of the incremental work
> at backup time and not at restore time. At restore time
> i just have to bang away files that vanished between 
> levels. This is done by a prepared script which can be
> reviewed and eventually edited by the user.
> The incremental volumes are complete in themselves and 
> independent of other volumes or of previous levels.
> The goal is to be very transparent and easy to review at
> backup time. The price is a fat tree model on disk and
> substantial runtime of the backup's preparation.

You need to do more at restore time. Star needs to:

-	Rename renamed files and directories

-	Remove files that did change the type between two incrementals

-	Renove all files that have been rmoved between two incrementals


> Lucky the person who can test the restore and make a
> real comparison of original and copy. (Not easy to
> achieve if the disk is nearly full.)

This is what I am doing every day with star since more than a year.
Star did not create any problems since 200 days.


> > If you rename a populated directory, star creates an archive that is a few kb
> > in size for an incremental backup.
>
> Is my understanding of star correct that in this case
> it only records the directory with its pairs of names and inodes
> but not the files' content (which is unchanged) ? 
> That at restore time, the content already got restored by
> a previous level and then gets attached to these newer
> directory entries ?

Correct.

> If so, then it is as elegant as it is daring.

Sorry it would only be daring in case that the implementation of star
would be buggy. Wht do you believe is the reason why I did think 3 months
about the incremental restore algorythm before I did start to write the first
line of the implementation?


> (Was there ever an answer to 
>  http://lists.berlios.de/pipermail/star-users/2005-September/000438.html
> ?)

Looks like someone did use an outdated star version to create the archive or
like the _partial_ dump did cause the problems because the person did not
archive enough of the files.

Note that there is a guarantee that star is able to correctly restore things
only if you create a _full_ level 0 or higher level dump.

The minimum star version for _creating_ correct incremental dumps is 
star-1.5a61.


> > If you a populated directory, GNU tar creates an archive that has the size
> > of the accumulated content of that dir.
>
> That seems to be more like my own method.
> Fat but simple.

It is extremely dumb....

Jörg

-- 
 EMail:[email protected] (home) Jörg Schilling D-13353 Berlin
       [email protected]		(uni)  
       [email protected]	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily