Re: Perfomance problem backupscript (more details and tests)

Hi,

>> I wrote :
>> Do i perceive it right that both star and GNU-tar rely on
>> the persistence of inode numbers within filesystems ?
>
> Les Mikesell wrote :
> 
> I think star relies on unix-like filesystem.  Tar does
> use device/inode combinations to track directories
> traversed by the full runs.

How does GNU-tar detect if a file vanished between
two backup levels ?
To my theory, star does this when joining updated
directories with the file bodies during restore.
Those which cannot be attached to any directory are prone
to removal, i guess.
But how does GNU-tar remove obsolete files at restore
time ? Does it delete the whole directory before restoring
the new layer ?


>> Googling for "tar" "listed-incremental" i find problem
>> reports about VFAT ... SMBFS ... FAT.
> 
> I don't think it's obvious at all that inode numbers
> would change if you mount/unmount a device even if
> they are faked by computation.  Does this only happen
> if you remove a device, mount it elsewhere and modify
> it, then remount in the original location for the
> gnutar incremental or does it happen on any remount.

No idea. It's google hearsay. But very common one.
I use ext2 and reiserfs on disk.
But i was pointed to SMBFS when asking my users about
their opinion on an inode-based content test.


> On unix-like filesystems with valid ctimes all you need
> is the ctime of ordinary files since that catches any
> possible change - including ownership and permissions
> which your method appears to miss.  The problem is a
> renamed directory containing old files.

I see. Old mtime of directory, old mtime+ctime of files,
no reference model of the tree to lookup wether the
files have been present under that address previously.

Who takes care of the outdated copy of that directory
at restore time ?


> Actually, my first choice in backups these days is
> backuppc http://backuppc.sourceforge.net/ which
> can use several methods including rsync to transfer
> the copy and keeps compressed copies online with hardlinks
> for all duplicates so you can store about 10x what you
> would expect. 

Loveable concept for hard disk backups.


> Any filesystem with native unix semantics must have
> fixed inodes and the ability to create hardlinks by
> having any number of directory entries pointing to the
> same inode. 

Users (the experienced and skilled ones) told me
that they are uncomfortable with any reference to
inodes.
That is because inodes form a directed graph but the
user model of data storage is a tree. Hardlinks are
handy but introduce severe fundamental problems.


> Backup methods must be able to re-create
> these hardlinks - which isn't easy because all they
> have in common is the inode number.

Interesting point.

I would agree to :
"For situations where hardlinks do matter, backup methods
 must be provided which allow restoring of hardlinks."

The problem with hardlinks is that they cannot be represented
in common formats like ISO-9660. Offer a desktop user the 
choice of having the personal backup in ISO-9660 or in tar
and you will see a 90% majority for ISO. It's so handy
for picking files after small mishaps.

I delegate those tasks to the backup formats which i offer:
ISO-9660, afio and star. I will have to check wether
star and afio do plug hardlinks together on restore.


>> Nevertheless:
>> Can it really be that the big archive programs ignore the
>> existence of filesystems which do not comply to the
>> inode-semantics of classical Unix ?
> 
> Tar only cares if you do --listed-incrementals, and then it
> only cares that the directories have the same inode
> number on each pass (or the fake computed equivalent)
> for the directories.  If they don't match, the failure
> mode is to copy everything which is probably the best
> way to fail.

So GNU-tar not only ignores the existence of inode-problematic
filesystems but also the existence of SuSE 9.0 ReiserFS ???
(It is the _default_ filesystem of the toadies.)


>> Shouldn't there be some fallback provided in order to
>> handle such cases ?
> 
> If the filesystem in question doesn't provide any way
> to tell what has changed, there's not much hope for
> a good incremental method.

But intrusion detection is an alternative method.
I removed from that principle anything that refers to
malicious activities. I rely on unfaked mtime and ctime
but i do not rely on ctime reflecting name changes.


> All unix filesytems have ctime values that are updated
> with any inode change. 

When i started my own backup project back in 1999 
Linux did not change ctime on rename. I was not amused
but reading  man 2 stat  told me why this was the fact.
There was only ext2 and ext2 is rustily based on inodes.


>> On my SuSE 9.0 the situation is nicely mixed :
>> mv changes ctime on ext2 but not on ReiserFS.
> 
> ReiserFS doesn't completely follow unix semantics.

Here you can see that the defense line of demanding
ctime to behave against old traditions cannot be upheld.

Actually the ctime behavior which you demand (and which i
would actually prefer, too) contradicts all old man pages.
It is an underground quirk introduced to cover a flaw
in the inode based filesystem model. 
But such patchwork is not reliable. You are too likely to
encounter something that is done correctly and does not
play well with the shabby quilt.


>> "The field st_ctime is changed by writing or by setting
>>  inode information (i.e., owner, group, link count, mode,
>>  etc.)."
>> "Moreover, st_mtime of a directory is changed by the
>>  creation or deletion of files in that directory."
>>
>> "Unfortunately, however, the change was not pushed
>>  into 2.3, and so 2.4 and now 2.6 have a ext2 rename()
>>  implementation that does not update inode ctime."
> 
> Rename is kind of a special case.  The inode in question
> doesn't necessarily change because a directory entry pointing
> to it changes. 

Exactly. No inode change. Dire, but such are the rules.


> Most implementations would do it by creating
> a new link with the new name, then removing the old one, and
> changing the link count in the inode requires a ctime change.

This i would call a bug if i compare it with the behavior
prescribed by the man pages.
It is a nice bug, i have to confess.

But it will have to be emulated intentionally if there are no
real inodes to be linked and unlinked. As you can see with ReiserFS.
Even ext2 obviously does rename without performing link/unlink
in a way that affects ctime. The feature was added later.


> Do you catch other inode-only changes (owner/permissions, etc.).
> Can you reconstruct hard links? 

Yes.
Of course i first refer to the mtime and ctime stamps.
If both indicate that the file is old, then i'm not done but
have to make sure that the file address already existed at the
last backup level. (See renamed directories et.al.)
If the hosting directory's mtime is young, then i have to assert
unchanged file content. Either by checksum comparison or by
comparison of the inode number (assuming that either inode number
or time stamps of a file address change with any possible content
change).
This content test is obsolete if renaming changes ctime.

(I also check for changes of filetype and size.)


> Rsync against a complete on-line copy should always
> be able to detect any change...  And backuppc can do
> it against compressed copies.

That comes near to my own methods which are less
bandwidth conserving than those of rsync, though. I only
do whole files. On the other hand, i can leave out many
content checks by safe timestamp considerations. (Plus i
can make use of inode numbers if they are reliable.)


> Dump/restore used the same technique forever, but it is
> fairly intimate with the filesystem and it will fail
> horribly if you convert a directory to a mount point
> between runs.

dump was a pain in the ass forever :((
dump and DAT are the reason why i began to develop own
backup software. :))


Thanks for the hint with the hardlinks. I confess not to
have ever thought about refiddling those at restore time.


Have a nice day :)

Thomas