Re: Perfomance problem backupscript (more details and tests)

Hi,

> > Yes. But if i would strive for NT filesystem access from
> > a 32-bit OS where ino_t happens to be 32-bit ...
> > ... you see the temptation to stretch POSIX to the max
> > in order to achieve a bearable solution ?
> 
> The 32 bit ino_t relation for a 32 bit OS is UNIX specific but not 
> required by POSIX.1-2001 or later.

I understand that a _new_ 32-bit operating system version is
not bound to sizeof(ino_t)==4.
But any _old_ instance of a 32-bit Linux which is running
in real life will be unable to accomodate with a filesystem
that uses 64-bit file serial numbers.
So there will be a certain pressure to do such a 64-to-32
translation which according to Dirichlet's drawer principle
(aka pigeon hole) can hardly be collision-free by nature
but only by expensive implementation add-ons. And only if there
are less than 4 billion inodes in the filesystem.


> This seems to be an interesting aspect of ZFS on Solaris too.
> Let us ask the developers what happens if you create a lot of
> files on a ZFS partition :-)

Looks like the problems are lurking just below the horizon.
At least we have to watch out carefully for any attempt to
sacrifice file serial number lifetime persistence.

Does ZFS issue small inode numbers as long as there is a
small number of files ? Maybe you get the first number > 32 bit
long before the filesystem hosts more than 4 billion files.


> > > Note that POSIX _requires_ ctime to be updated on a rename.
> >
> > No, it doesn't.
> 
> If definitely _requires_ ctime to be updated on a rename.
> 
> Upon successful completion, rename() shall mark for update the st_ctime and 
> st_mtime fields of the parent directory of each file.

But that is about the parent directory.

Les, GNU-tar and i refered to the renamed file's own
timestamps. These are mentioned too in the same man page
as your above quote. And their ctime is allowed to change
or to stay unaltered at the implementor's discretion.

http://www.opengroup.org/onlinepubs/009695399/functions/rename.html
Next section after EXAMPLES. It is declared an "informative section"
so i understand the undecisive text of section APPLICATION USAGE as 
"You are not _obliged_ to mess it up, but you may".

Probably the sentences are a compromise resulting from a
long discussion. Possibly somebody implemented a use for not
updating ctime and fiercly defended the same view which i 
can deduce (against my will) from my old manual by S.R.Bourne
of 1983.


> The word "shall" seems to be unambiguous for me....

Agreed. But it is refering to the parent, not to the renamed victim.


> > - my own concept which i developed from intrusion detection
> >   ideas.
> 
> I forgot, how does your method work?

It usually relies on the parent directory timestamp
rule as of POSIX rename(). If the timestamps of a file
are old but timestamps of a directory above are new,
then the file's content is checked against a recorded
checksum.
Reasons to avoid the checksum test and to assume the file
as young enough are : young timestamp,
non-existence in recorded tree model, changed type,
changed size.
No file is assumed old unless all its parent directories
are old or unless its checksum has been verified. The
parent directory assumption may be disabled if the user
is very mistrusting.


Meanwhile i can replace the checksum test by a dev_t,
ino_t test which is fast enough so i can give up even
the assumption of correct parent directory timestamps.
For (partial or complete) backups of a single filesystem
one may safely disable the device number test.

I am aware of possible dev_t problems and even of
potential inode persistency problems. So i will continue
to offer the checksum method but i will probably make
the inode method the default.
It works on all classical Linux disk filesystems and
saves a lot of computing time. 

A multi-filesystem backup needs device numbers mainly if
it has to deal with an agile mount structure. If everything
is always mounted at the same addresses, then files cannot
easily move from one filesystem to the other without updating
their timestamps. (But this is not defended by POSIX rename(2).
Despite there is an error EXDEV defined for cross filesystem 
link attempts, these aren't absolutely outruled. Volatile
device numbers combined with cross-filesystem links seem to
be quite a challenge for the implementation, though.)

Whenever the user encounters a system with unstable
device and inode numbers, the effect will become obvious
by much too fat incremental backups.
Nevertheless i will urge people to use the checksums
if they are in doubt about devices and inodes.
(I ponder about a honeypot detector for bad inode systems.)


> Inode numbers should be persistent as long as the implementation is
> a decent one.

I would agree to call any implementation "indecent" 
that violates lifetime persistence of file serial numbers.

I am not ready, nevertheless, to leave behind users
who fell victim to such an implementation. (My BSD
license does not allow to ban the _implementors_ of such
stuff from using scdbackup, i fear.)


> If you depend on Linux, you suffer from the inability of Linux
> to have persistent device IDs for something like USB. Note that this
> is nust another reason for _not_ trying to backup a logical file tree
> that contains sub mounts as Les likes to mandate.

That is a consistent position. It is quite narrow, but
it makes sense.

That's why i am astonished that you offer partial incrementals
with star, although you know that it opens the door for
gaps in the backup's data content. This pierces the
solid consistency of star's approach, which is quite
fool-proof elsewise.


> With GNU tar and an attempt to achive a logical file tree that crosses
> mount points, you will fail if st_dev is not persistent over the time
> of the incremental backups.

Without knowing any details i would assume to get rather
an oversized backup than one which leaves out necessary
content. (But of course, one can construct misunderstandings
which really would exclude a young file from backup.)


Have a nice day :)

Thomas