Re: Perfomance problem backupscript (more details and tests)

Hi,

>> ... persistence of inode numbers ... VFAT ...
>> ... problems with SMBFS. Not to speak of FAT.
> 
> This is incorrect, these people seem to confuse the problems
> of a specific impleentation with the filesystem.

In real life, i have to live with implementations.
As much as i would wish that standards would suffice.


> Nobody should use a NON-POSIX filesystemimplementaion for 
> every day work, so only POSIX compliant implementations
> need to be backed up.

Ok. You cater for the reasonable people and i'll
take care of the other 6 billion. :))


> (note that POSIX calls them "file serial number").

A valuable hint.
I would be glad to read a clear sentence from a valid
POSIX document which guarantees _persistence_ and not
only uniqueness of those numbers.

http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html
"3.175 File Serial Number
A per-file system unique identifier for a file."

One thing is clear: they are not "unique" in the sense of
"never issued twice". With some bad intent one could claim
that they have to be unique only in the sense of "no two
concurrent identical numbers allowed".

Up to now, i failed to find a clarifying statement.
(Maybe it's because i gave the control freaks a fake
 phone number and a throw-away mail address ?)


> So your question should be: Can it really be that people
> work on an OS that implements non-read-only filesystems in a way that
> violates POSIX?

The answer is trivial : Yes, they do. They always will.


> > The ctime-quirk
> This is not really needed. Needed (for star) is that a rename
> of a file changes st_mtime for the related directories.

Be praised for that.

ctime does not help with the mentioned directory move 
situation anyway.


> You need to do more at restore time. Star needs to:
> 
> -	Rename renamed files and directories
> -	Remove files that did change the type between two incrementals
> -	Renove all files that have been rmoved between two incrementals

Before restoring the backup level content, the user removes
vanished file addresses and those which changed file type.
This is done by a script in the last volume of that level.
(One has to extract it, of course.)
Then the volume content is copied to disk. It contains all
files where not the same content was stored under the same
address as at the time of the previous backup level.


>>  If so, then it is as elegant as it is daring.
> 
> Sorry it would only be daring in case that the implementation of star
> would be buggy. Wht do you believe is the reason why I did think 3 months
> about the incremental restore algorythm before I did start to write the first
> line of the implementation?

As stated some time ago :
Within its self-implied restrictions your incremental method
is of supreme quality. I would give up my complete own work
if i could live with those restrictions.

 
> > (Was there ever an answer to 
> >  http://lists.berlios.de/pipermail/star-users/2005-September/000438.html
> > ?)
> 
> Looks like someone did use an outdated star version to create the archive or
> like the _partial_ dump did cause the problems because the person did not
> archive enough of the file.

Here you should consider to enforce safer conditions.
You warn of partial dumps, they really seem to be problematic,
but they are still possible, people do them, finally
the blame falls on star. 

I believe to understand why your method is suitable mostly
for a single whole filesystem.
I also believe that a tree model of the partial backup area
at backup time would be able to lift that restriction.
It would have to relate file addresses to type, device number,
and inode number. star would lookup files in order to detect
sneak-ins and sneak-outs.
This is expensive, i confess. But unusable backups are more
than expensive.

GNU-tar seems to employ a very sparse model and relies on ctime.
I employ one that records any single file and need no ctime
change on rename. (Bug or not, one has to live with it.)


> Note that there is a guarantee that star is able to correctly restore things
> only if you create a _full_ level 0 or higher level dump.

Obviously you need to post a bigger warning sign
for those who wait for disaster first and then ask for
the reason of this warning.
Or maybe some electrical shocks would be helpful.


> > That seems to be more like my own method.
> > Fat but simple.
>
> It is extremely dumb....

Is it ?

Let me assume a three level star backup of a complete filesystem,
where the data content of a file was recorded at level=0 and the
most recent name change was recorded at level=2 :
  How do i retrieve my single file under its _recent_ name without
  doing a full restore (which might overextend my disk) ?

I read the man page for that question. No success, i fear.

With this situation i expect the problem that the level=0
backup cannot know about the recent name and thus does not
unpack the data content (which still has the old name).
When level=2 gets unpacked, there is no data installed
under the old name on disk. Thus the directory entry of the
recent name cannot be attached to its file body.

A solution for that seems not impossible - but challenging.
Did you already implement one ?
(Like looking up the inode number at higher levels first
and then subsequently scanning lower levels for the according
file body.)

The restore procedure as in star's man page works from lower
levels to higher levels, though. With that sequence, i fail
to imagine a successful restore of a single renamed file.
At least if i would not know its old name and the file's
owner (who already spoiled it once) is unable to remember
that old name either.


>> Users (the experienced and skilled ones) told me
>> that they are uncomfortable with any reference to
>> inodes.
> Then they are not experienced enough....

To believe that would be a mistake. 
They are just cautious not to open any encapsulation
without need.

> Many programs need to deal with st_dev/st_ino pairs because this
> is the only way to prevent killing files by copying them to itself.

But for that inode number reference one only needs the
weakest form of uniqueness (i.e. no concurrent identical numbers).
For incremental backup we need more. 


>> Hardlinks are
>> handy but introduce severe fundamental problems.
> Irrelevent. The inode is just a file serial number and the usage
> is the usage for a file serial number.

Example:
Les has problems to find tools which copy large
hardlink structures swiftly.


>> ([ReiserFS] is the _default_ filesystem of the toadies.)
> What should this mean?

The green SuSE toad ("die Kroete") advises people to use
ReiserFS. A substantial amount of german Linux installations
follow this advice.


> [regarding rename and updating of ctime]
> POSIX is unambiguous here:
> Upon successful completion, rename() shall mark for update the st_ctime and 
> st_mtime fields of the parent directory of each file.

Urm. I read "of the parent directory of each file".

Les and i were talking about the ctime of the
single file and not about its directory's timestamps.
"fields" is plural, therefore means both time stamps and
relates itself to "parent directory".
So it talks about the ctime of the directory.

What POSIX document and what paragraph does state above
sentence ? (Not that i wouldn't take your word for it
but maybe someday i could need the exact source of that 
quote.) ... aha, man rename, IEEE Std 1003.1-2001 ... 
thanks google and thanks Joerg i found in
http://mordor.tengry.com/programming/standarts/susv3/functions/rename.html
under said sentence :
" APPLICATION USAGE

 Some implementations mark for update the st_ctime field of renamed
 files and some do not. Applications which make use of the st_ctime
 field may behave differently with respect to renamed files unless
 they are designed to allow for either behavior."

So, from the POSIX point of view, this is clear.
For all practical purposes it ends as a draw. 

At least in this aspect Linux is not offically broken. ~:o)


Have a nice day

Thomas