Re: Perfomance problem backupscript (more details and tests)

Les Mikesell <[email protected]> wrote:

> On Tue, 2005-10-11 at 06:31, Joerg Schilling wrote:
>
> > > Perhaps different constraints would be a better description. 
> > 
> > I don't see any proof that the constraints are different.
>
> GNUtar backups including incremental runs can traverse
> mount points and can have arbitrary exclusions.  The
> backup operator does not need to know the filesystem
> topology and mount structure or whether it changed between
> runs.  The restore operator does not need to know
> the filesystem structure of the source of the archive
> or whether it matches the target structure. Restoring
> renamed files in an incremental run does not depend
> on having restored the content to the same filesystem
> in a prior restore.   These are _big_ differences in
> practical usage.

As the star tape dump headers include the mount point,
I see no problem to handle the backups. In special when
you use a high level software that helps you.

And if you did read the description of the star incremental
backup/restore process, you should know that you do not depend on
restoring previous incrementals with star if you only like to
restore single files.

So you still did not send a proof that GNU tar has different constraints.
You did however show that you don't know enough about the star or ufsdump
method.

So de-facto, there are absolutely NO differences except that the
star method has been verified to work reliable while tests show that
GNU tar fails in many cases.



> > > > -	GNU tar depends on the inode numbers of directories and on
> > > > 	ctime/mtime
> > >
> > > Yes, but if they don't match, gnutar includes a new copy in the
> > > incremental.
> > 
> > And whay do you believe that this should help?
>
> I'd prefer any mistake to be in the 'safe' direction - that is
> to include the files. The assumption is that a directory that has
> the same device and inode number as it did in a previous run is not
> new or renamed. If they are different, all contents are taken in
> the incremental.  Thus any inconsistency in inode numbering or
> mounting will result in the backup including the related files,
> not accidentally omitting any.  Including any questionable files
> is particularly harmless with a wrapper like amanda that does
> size estimates, since it has the option of doing a full ahead
> of the planned schedule if it sees that it won't save much space
> by doing an incremental.

Looks like you did not understand the GNU tar incremental method either....

Please note that in case that inode numbers are not persistent, 
a renamed directory with GNU tar (in case it works as advertized and does not
show the bugs I reported) results in the following:

-	The old directory and its whole content is removed from 
	the restoreed filesystem.

-	As the new incremental backup is _missing_ the directory content
	(as it does not have a newer time stamp than the last backup),
	GNU tar is unable to restore any of the files that are inside
	the renamed directory.

Conclusion: If inode numbers are persistent, star is able to restore
renamed directories correctly by only archiving the renamed dir, GNU tar
would need to include the _whole_ directory content in this case and it
will do.

But if inode numbers are _not_ persistent, GNU tar will not include
the content of a dir and thus fail the same way as star does.

In other words: If inode numbers are non-persistent, GNU tar and star
as well as ufsdump fail. If inode numbers are persistent, GNU tar may work
but needs to archive a lot more data.



> > It has absolutely no advantage. Please first try to understand how the
> > star incremental restore works.
>
> I'm pretty fuzzy on that.  Can you describe how a file renamed between
> runs gets the new name hooked to the old content in the restore, and
> all of the constraints that have to be met for the old content to
> exist in the right place?  Suppose you make a level 0 of a filesystem
> containing / and /home, then mv a directory containing old files
> from somewhere on /tmp to somewhere in /home (which amounts to a
> rename), do an incremental, then restore these to a new machine
> where / and /home are different filesystems.   I don't understand
> how the old files under the mv'd directory will ever exist on
> the /home filesystem.

Please read the old mails again and ask me again only in case you 
are unable to understand the algorithm.

But it should be obvious that both ufsrestore and star will not work
this way but may be enhanced to do so by copying the renamed directory
tree from one filesystem to another.

NOTE: /tmp is usually (at least since 18 years) on a separate mounted
swap based filesystem and thus not part of /home (which is just a loopback
mount of /export/home). Also note that no knowlegeable admin will put
the OS and the /home dirs on the same partition.


> > Could you tell us why you write this in a discussion where you like to
> > convince us why GNU tar includes features that star is missing?
>
> It would be really nice if star offered an option like GNU tar to
> make incremental backups that are not limited to single filesystems
> and do not require a restore of the previous, possibly unknown
> named file to enable the restore of file renamed in the incrementals.
> Then star could replace GNU tar as well as dump - and we could stop
> talking about GNUtar bugs...

I am sorry, but this feature does not make sense as there are more
important features waiting for an implementation _and_ as it is possible
to handle your wishes by just using star the right way.


> You are probably right. I'm just having trouble visualizing how
> a renamed file avoids being connected to the wrong thing if
> concurrent filesystem activity re-uses its inode.   With GNUtar
> it only depends on being able to open the file with the name
> found in the directory which is a pretty small time window.

The same applies to star.....
and GNU tar has the same other problems as star in case that a directory
that has been schanned for a current incremental is renamed to an unscanned
part of the FS.


> > However, if you like to make a reliable backup from a life machine,
> > you need to use a filesystem snapshot anyway.
>
> This rules out all of my machines, although I think a reasonable
> set of LVM2 tools will be in the next update to Centos so perhaps
> eventually this will be usable.

Well, filesystem snapshots now are state of the art since ~ 6 years.
If your OS still soes not support them, you should consider a change.


Jörg

-- 
 EMail:[email protected] (home) Jörg Schilling D-13353 Berlin
       [email protected]		(uni)  
       [email protected]	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily