Re: Perfomance problem backupscript (more details and tests)

Hi,

as we are discussing incremental backups :

Do i perceive it right that both star and GNU-tar rely on
the persistence of inode numbers within filesystems ?

Googling for "tar" "listed-incremental" i find problem
reports about VFAT filesystems which obviously do not
preserve inode numbers after umount. There seem to be
similar problems with SMBFS. Not to speak of FAT.


> Les Mikesell wrote :
> and it seems there is no other choice.

One of my own methods for incremental backup resembles 
the approach of an intrusion detection system rather than
to rely on the intestinies of particular filesystem types.
I.e. there is a tree model of the backup area with
name and checksum of each single file.

This is darn slow in comparison to an inode-number check,
but it is not bound to full featured Unix filesystems
and it is not affected by filesystem boundaries.
(Thanks to inspiration by star, i added an alternative
inode-number check which runs fast and reliable on ext2
and reiserfs. Other types have still to be evaluated
but i guess it is ok with most locally attached 
filesystems on Linux.)


Nevertheless:
Can it really be that the big archive programs ignore the
existence of filesystems which do not comply to the
inode-semantics of classical Unix ?
Shouldn't there be some fallback provided in order to
handle such cases ?


The ctime trick of my current Linux ext2 is a nice try 
but i am sure that it can be fooled by using other 
operating systems (or just other Linux versions) on the 
same filesystem.

On my SuSE 9.0 the situation is nicely mixed :

mv changes ctime on ext2 but not on ReiserFS.

man 2 stat still says :
"The field st_ctime is changed by writing or by setting
 inode information (i.e., owner, group, link count, mode,
 etc.)."
The name of a file is not an inode information.
Renaming of files is rather covered by :
"Moreover, st_mtime of a directory is changed by the
 creation or deletion of files in that directory."

The ctime-quirk seems to be dependend on the type of
filesystem and on the version of Linux. It has been
patched into the code without revising documentation.
It might get patched out of the code as well.
http://sourceforge.net/mailarchive/forum.php?thread_id=3726620&forum_id=6379 :
"Unfortunately, however, the change was not pushed
 into 2.3, and so 2.4 and now 2.6 have a ext2 rename()
 implementation that does not update inode ctime."

Can confusion be any bigger ? 
So the semantics of ctime is nothing that could be relied on
for portable incremental backup software.


On one hand it is pleasant to see that my first approach
is still the only one that covers all filesystem types
and mount situations. 
On the other hand it is unsatisfying that i cannot
advise the much faster inode method or even a merely
time based method as general default.

So, if somebody could refute my above considerations,
i would be 50% sad and 50% happy.


> Joerg Schilling wrote :
> But the problem is that GNU tar claims to support this kind of backups but
> has too many bugs with incremental backup/restore to make it useful. 

That's why i prefer to do most of the incremental work
at backup time and not at restore time. At restore time
i just have to bang away files that vanished between 
levels. This is done by a prepared script which can be
reviewed and eventually edited by the user.
The incremental volumes are complete in themselves and 
independent of other volumes or of previous levels.
The goal is to be very transparent and easy to review at
backup time. The price is a fat tree model on disk and
substantial runtime of the backup's preparation.

Lucky the person who can test the restore and make a
real comparison of original and copy. (Not easy to
achieve if the disk is nearly full.)


> If you rename a populated directory, star creates an archive that is a few kb
> in size for an incremental backup.

Is my understanding of star correct that in this case
it only records the directory with its pairs of names and inodes
but not the files' content (which is unchanged) ? 
That at restore time, the content already got restored by
a previous level and then gets attached to these newer
directory entries ?

If so, then it is as elegant as it is daring.

(Was there ever an answer to 
 http://lists.berlios.de/pipermail/star-users/2005-September/000438.html
?)


> If you a populated directory, GNU tar creates an archive that has the size
> of the accumulated content of that dir.

That seems to be more like my own method.
Fat but simple.


> [...] the new file could get the same inode number
> as the old one.

With usual filesystems that new file should have a new
timestamp ... hopefully ...


Have a nice day :)

Thomas