Re: Perfomance problem backupscript (more details and tests)

On Thu, 2005-10-06 at 05:32, Joerg Schilling wrote:
>  
> > > There are many other problems with incrementals using GNU tar.
> > > GNU tar does absolutey not grok the case when a directory is renamed......
> >
> > You must use --listed-incremental mode (which amanda does).  Can you
> > show a test case where this fails?
> 
> Every (even simple) testcase should cause the problem.....

OK, where am I going wrong:
---
TESTDIR=my-tar-test
DATA="a b c d"
mkdir $TESTDIR
cd $TESTDIR
mkdir test rest
cd test
for i in $DATA
do
mkdir $i
for j in $DATA
do
echo "$DATA" >$i/$j
done
done
tar --listed-incremental=../tarlist -cvf ../tar0.out .
# note directory rename here
mv b e
tar --listed-incremental=../tarlist -cvf ../tar1.out .
cd ../rest
tar -Gxpvf ../tar0.out
tar -Gpxvf ../tar1.out
cd ..
diff -r test rest
---
Mine are identical - rest/b and contents are gone, rest/e and
contents are present.  And it would work the same if I let this
test cross a mount point.  (And yes, on my distribution, probably
one of the most popular in the world, /bin/tar is really gnutar).

> A few years ago, I did set up a testcase for ufsdump comparison between Solaris
> and FreeBSD and I did run a GNU tar test for fun. The restore test failed with 
> GNU tar. 

Is it fair to compare this week's star against a few years's old gnutar?
Even the old bug of not removing an old file before creating a directory
of the same name seems to be fixed now.  For example if you
touch e
to create a file before the first tar above
and rm it before the
mv b e
which makes the new directory of the same name, it is still
working.


> > What bugs, and what other choice have they had for the last 12 years.
> > What choice do they have now for incrementals on backup runs that
> > cross filesystem boundaries?
> 
> There was of course ufsdump/ufsrestore since 1981.

If you dumped from a ufs filesytem and restored to a ufs filesystem
with exactly the same mount points.

> Crossing mount points in backups is not something that I would call the right 
> way to go.

Depending on why the original died and what I wanted to have when I got
done rebuilding, changing filesystem types and/or mount points might be
a good thing.  Certainly the ability to make those decisions at restore
time is a good thing. 

> > I've been using amanda/gnutar for 10 years or so.  I'd be happy to
> > find a better alternative, but I don't see any other choice that
> > will schedule the right mix of many fulls and incrementals to fill
> > the tapes and also get the right files in the incrementals.
> 
> How many real desaster recoveries did you do with the so created archives?

I think there was only one that involved multiple level restores.  The
amanda estimates allow it to bump the levels up/down according to what
will fit on the tape so often you'll have a recent level 0 available
if you allowed enough tape capacity to catch occasional large changes.
I don't remember any particular real-world problems.

> > The only bug I've ever seen is that if an ordinary file is removed and
> > replaced with a directory of the same name between a full and subsequent
> > incremental run, the incremental restore of the directory will fail. 
> > This is not fatal, though, since you see the error and can still
> > manually remove the directory and repeat the restore successfully.  And
> > this was some time ago - it may be fixed now.
> 
> What do you do if thousands of sich replacements took place?

It does seem to be fixed now, but if not I'd capture the error output to
a file, convert the failed directory items to a script that removes
those files, then repeat the restore.  I don't consider a file that
is replaced by a directory to be any more likely than a directory
being replaced by a mount point, and being able to restore the data
after removing a file is much less of a problem than completely
missing the contents of a mount point you expected to be traversed.

I may be able to live with the restrictions on filesystems and 
exclusions, but it is very unlike tar and I don't think there is
enough in your current documentation to really understand what
to expect.

-- 
  Les Mikesell
    [email protected]