Re: Re: Perfomance problem backupscript (more details and tests)

[email protected] wrote:

> >> ... persistence of inode numbers ... VFAT ...
> >> ... problems with SMBFS. Not to speak of FAT.
> > 
> > This is incorrect, these people seem to confuse the problems
> > of a specific impleentation with the filesystem.
>
> In real life, i have to live with implementations.
> As much as i would wish that standards would suffice.

OK, but you have the chance to make a bug report if you fins bugs and
you are able to select a better OS in case you find that the maintainers
of the OS you use do not do a good job.


> > Nobody should use a NON-POSIX filesystemimplementaion for 
> > every day work, so only POSIX compliant implementations
> > need to be backed up.
>
> Ok. You cater for the reasonable people and i'll
> take care of the other 6 billion. :))

If you believe that there are Linux 6 billion Linux users....


Solaris does it right and Win32 with a newer Cygwin does it right too.
The only real problem with Microsoft NT filesystems is that you need a 
64 bit number to represent the file serial numbers which was not POSIX
compliant on 32 bit CPUs before POSIX.1-2001 came out.


> > (note that POSIX calls them "file serial number").
>
> A valuable hint.
> I would be glad to read a clear sentence from a valid
> POSIX document which guarantees _persistence_ and not
> only uniqueness of those numbers.

A simple experiment in mind would allow you to prove this:

If you unmount/remount a NFS based filesystem on a running client system,
the numbers are not allow to change because otherwise the unique ID
of a file on that fs would change. The same restrictions apply when the
server is rebooted.

Now make your conclusions....

>
> http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html
> "3.175 File Serial Number
> A per-file system unique identifier for a file."
>
> One thing is clear: they are not "unique" in the sense of
> "never issued twice". With some bad intent one could claim
> that they have to be unique only in the sense of "no two
> concurrent identical numbers allowed".

A pair of st_ino/st_dev uniquely identifies a file for the lifetime
of the file. For dealing with reused inode numbers, NFS defines
the file generation number is used but this is beyond the scope of
POSIX.


> > So your question should be: Can it really be that people
> > work on an OS that implements non-read-only filesystems in a way that
> > violates POSIX?
>
> The answer is trivial : Yes, they do. They always will.

Well, but if people know that there are limitations caused by the 
implementations that are not needed, then people will think different.


> > You need to do more at restore time. Star needs to:
> > 
> > -	Rename renamed files and directories
> > -	Remove files that did change the type between two incrementals
> > -	Renove all files that have been rmoved between two incrementals
>
> Before restoring the backup level content, the user removes
> vanished file addresses and those which changed file type.
> This is done by a script in the last volume of that level.
> (One has to extract it, of course.)
> Then the volume content is copied to disk. It contains all
> files where not the same content was stored under the same
> address as at the time of the previous backup level.

Be careful, the algorithm is not trivial and it took me nearly half a 
year in order to get it 100% correct.


> > > (Was there ever an answer to 
> > >  http://lists.berlios.de/pipermail/star-users/2005-September/000438.html
> > > ?)
> > 
> > Looks like someone did use an outdated star version to create the archive or
> > like the _partial_ dump did cause the problems because the person did not
> > archive enough of the file.
>
> Here you should consider to enforce safer conditions.
> You warn of partial dumps, they really seem to be problematic,
> but they are still possible, people do them, finally
> the blame falls on star. 

The problem is _extremely_ rare. This is why it took more than half a year
to find it out.


> I believe to understand why your method is suitable mostly
> for a single whole filesystem.

No, the method works for partial dumps if you do not ignore the
contraints and it could be extended to support dumps that cross
filesystem boundaries without a need to extend the already documented
archive format.

It just makes no sense to add support for it before the implementation
has been verified with the set of features that is supported
by other well known dump systems like ufsdump/ufsrestore.


> I also believe that a tree model of the partial backup area
> at backup time would be able to lift that restriction.
> It would have to relate file addresses to type, device number,
> and inode number. star would lookup files in order to detect
> sneak-ins and sneak-outs.
> This is expensive, i confess. But unusable backups are more
> than expensive.

This is why I warn to use GNU tar for backups...
GNU tar does not correctly restore incrementals and GNU tar 
has problems to read back multi volume backups.


> > Note that there is a guarantee that star is able to correctly restore things
> > only if you create a _full_ level 0 or higher level dump.
>
> Obviously you need to post a bigger warning sign
> for those who wait for disaster first and then ask for
> the reason of this warning.
> Or maybe some electrical shocks would be helpful.

The man page of star is sufficient. As long as people believe that 
GNU tar is a useful backup system, I see no reason to extend the
warnings.



> Let me assume a three level star backup of a complete filesystem,
> where the data content of a file was recorded at level=0 and the
> most recent name change was recorded at level=2 :
>   How do i retrieve my single file under its _recent_ name without
>   doing a full restore (which might overextend my disk) ?

You would need a database that includes information on the files
that are included in every backup.

A future version of star may be able to write such a data base 
by using a new option. You could do it today if you run star -v and
catch stdout.

No backup system that I am aware of allows you to retrieve a file
if you don't know the name that was in use for the time you like
to restore.

> With this situation i expect the problem that the level=0
> backup cannot know about the recent name and thus does not
> unpack the data content (which still has the old name).
> When level=2 gets unpacked, there is no data installed
> under the old name on disk. Thus the directory entry of the
> recent name cannot be attached to its file body.

This is not true. As st_mtime is usually set to an arbitrary
value, star (as well as ufsdump) needs to archive the file
content also if just st_ctime did change. Star however ony 
needs to archive the single directory that was renamed and not
the whole directory tree under the new name like GNU tar.


> >> Users (the experienced and skilled ones) told me
> >> that they are uncomfortable with any reference to
> >> inodes.
> > Then they are not experienced enough....
>
> To believe that would be a mistake. 
> They are just cautious not to open any encapsulation
> without need.

??? 

> >> Hardlinks are
> >> handy but introduce severe fundamental problems.
> > Irrelevent. The inode is just a file serial number and the usage
> > is the usage for a file serial number.
>
> Example:
> Les has problems to find tools which copy large
> hardlink structures swiftly.

???? Just use star.


> >> ([ReiserFS] is the _default_ filesystem of the toadies.)
> > What should this mean?
>
> The green SuSE toad ("die Kroete") advises people to use
> ReiserFS. A substantial amount of german Linux installations
> follow this advice.

If ReiserFS is broken, then people should make bug reports.

> > [regarding rename and updating of ctime]
> > POSIX is unambiguous here:
> > Upon successful completion, rename() shall mark for update the st_ctime and 
> > st_mtime fields of the parent directory of each file.
>
> Urm. I read "of the parent directory of each file".

Correct! This is what star meeds ;-)


> Les and i were talking about the ctime of the
> single file and not about its directory's timestamps.
> "fields" is plural, therefore means both time stamps and
> relates itself to "parent directory".
> So it talks about the ctime of the directory.

Irrelevent for star.


>  Some implementations mark for update the st_ctime field of renamed
>  files and some do not. Applications which make use of the st_ctime
>  field may behave differently with respect to renamed files unless
>  they are designed to allow for either behavior."
>
> So, from the POSIX point of view, this is clear.
> For all practical purposes it ends as a draw. 
>
> At least in this aspect Linux is not offically broken. ~:o)

Check the web pages from the OpenGroup to find a long list why Linux
is officially broken.

Jörg

-- 
 EMail:[email protected] (home) Jörg Schilling D-13353 Berlin
       [email protected]		(uni)  
       [email protected]	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily