Re: Perfomance problem backupscript (more details and tests)

Les Mikesell <[email protected]> wrote:

> On Wed, 2005-10-05 at 19:18, Joerg Schilling wrote:
>
> > > > This is with gnutar 1.13.25.
> > >
> > > I just reported a bug in incremental restore in 1.15.1.  I would suggest
> > > that you upgrade to 1.15.1 if you want to trust gnu tar with incrementals.
> > > The restore problem I found manifests itself by leaving certain files
> > > around after the restore (it fails to delete symlinks that don't point to a
> > > real file).
>
> Are these created in --listed incremental mode?

The problems I did see did happen in this mode.

Note that the main problem with GNU tar inrementals is that GNU tar does not
store enough file meta data in the archive and for this reason the method
used by star cannot be used by GNU tar.

If you rename a populated directory, star creates an archive that is a few kb
in size for an incremental backup.

If you a populated directory, GNU tar creates an archive that has the size
of the accumulated content of that dir.

 
> > There are many other problems with incrementals using GNU tar.
> > GNU tar does absolutey not grok the case when a directory is renamed......
>
> You must use --listed-incremental mode (which amanda does).  Can you
> show a test case where this fails?

Every (even simple) testcase should cause the problem.....

A few years ago, I did set up a testcase for ufsdump comparison between Solaris
and FreeBSD and I did run a GNU tar test for fun. The restore test failed with 
GNU tar. 

Last year in August, I finished a 3 month reflection period on incremental
restore techniques and started to add incremental restore support to star.
Note that incremental backup is implemented for a longer time already.
After 3 hours of hacking, I set up a first test case and star did pass it.
GNU tar did not pass this simple test.....and as my experience with the GNU tar
maintaines is that it makes no sense to report bugs as they don't fix them,
I did not report. 

Let me check whether I am able to find the test script....

-	I did set up a new empty filesystem.

-	I did copy /usr/bin/* to a new directory /bin in that fs

-	I did unpack my archive typetest.tar

-	I did call cp /etc/termcap termcap

-	I did copy the whole Solaris /devices tree to /devices in that fs.

-	:> file

-	mknod dev c 1 1

-	mkdir tests/

-	:> tests/file_to_remove

-	mkdir dir/

-	cp /etc/termcap dir/termcap

-	:> file_for_chmod

-	Then I did create a level 0 backup.

-	Then I did run this script:

#!/bin/sh

mv bin bin.old
mkdir bin
cp /opt/schily/bin/c* bin

mv file file.old
mkdir file

mv dir dir.old
cp /etc/termcap dir

rm -f dev
cp /etc/group dev

rm tests/file_to_remove

cp /etc/group .

cat /etc/group >> termcap

chmod 600 file_for_chmod

Then I did make a level 1 incremental.

/*--------------------------------------------------------------------------*/
This was followed by a levl 0 restore and a level 1 restore on a new
and empty fs. Then a star -c | star -diff -v test was run.

Star did pass, GNU tar did fail.

BW: The first test with star in a real world system did show a lot of problems.
Many of them have been caused by the fact that removing a file and then
creating a new one may look as if the old file was renamed _and_ did change
the type at the same time because the new file could get the same inode number
as the old one.



> > GNU tar incrementals have been created 12 years ago but none of the bugs have 
> > been fixed until very recently (past 1.15.1). This is why I don't understand the
> > amanda people.
>
> What bugs, and what other choice have they had for the last 12 years.
> What choice do they have now for incrementals on backup runs that
> cross filesystem boundaries?

There was of course ufsdump/ufsrestore since 1981.

Crossing mount points in backups is not something that I would call the right 
way to go.


> > With star, I make an incremental backup _and_ restore every night for Berlios
> > since a year.
>
> I've been using amanda/gnutar for 10 years or so.  I'd be happy to
> find a better alternative, but I don't see any other choice that
> will schedule the right mix of many fulls and incrementals to fill
> the tapes and also get the right files in the incrementals.

How many real desaster recoveries did you do with the so created archives?


> > Note that I know of no case where people did run similar tests with GNU tar.
> > Making incremental backups does not show you the problems you get with 
> > incremental restores.......
>
> The only bug I've ever seen is that if an ordinary file is removed and
> replaced with a directory of the same name between a full and subsequent
> incremental run, the incremental restore of the directory will fail. 
> This is not fatal, though, since you see the error and can still
> manually remove the directory and repeat the restore successfully.  And
> this was some time ago - it may be fixed now.

What do you do if thousands of sich replacements took place?

Jörg

-- 
 EMail:[email protected] (home) Jörg Schilling D-13353 Berlin
       [email protected]		(uni)  
       [email protected]	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily