Re: Perfomance problem backupscript (more details and tests)

On Tue, 2005-10-18 at 09:15, Joerg Schilling wrote:

> > > The tar archive definition did not change since then, so why should
> > > the same thing be useful now but not useful in 1988?
> >
> > It isn't useful for backups now.  There is no way to specify all files
> > that existed at the time an incremental was taken so that others can
> > (optionally) be deleted during a restore.  Otherwise you will accumulate
> > all files that ever existed when you restore a series of incrementals
> > and run out of disk space.  You might as well use cpio.
> 
> Are you shure that you know what you are talking about?
> 
> First: you are confusing backups in general with "incremental backups".

Backups without incrementals won't work for me because they won't
fit on the tapes.  

> Second: even "incremental backups" are different from incremental restores.
> 	Star did support incremental backups quite a long time before it started
> 	to support incremental restores.

'Support' for incrementals should mean that it includes old files under
renamed directories during an incremental regardless of the location
during the full run.  Star can miss them if you move out of an excluded
area.

Restoring incrementals must have the option to delete files that were
not present at the time of the incremental run or you will accumulate
all the garbage that was deleted between the full and incrementals and
probably run out of space.

> cpio was not in the original POSIX.1-1988 proposal and made it into POSIX
> very late under pressure from AT&T. cpio is a very bad archive format.
> We should not start to talk about it for this reason.

So you only like some parts of POSIX?

> > Just enough to offset your hint that Solaris is the One True Way.
> 
> The way you use your words rather shows that it is _you_ who is offsetted 
> against Solaris without being able to give facts that prove this.

All I can do is describe my experience. Admittedly with old hardware
and old/buggy software.

> > I started at a company that already had the old Sparc, IPX, and IPC
> > boxes in 1996.  They were slow, buggy, and crashed regularly, and
> > _just_ an OS update would have cost more than faster pentium boxes.
> 
> It is obvious that you did never really run such boxes or that you have been
> a bad admin. 

I didn't install or configure them.  I just restarted them when people
said the services weren't working and put up with sales calls about
upgrades we couldn't afford.

> Between 1984 and 1993, I did work at a company that did use many PCs _and_ that
> was the second biggest OEM for Sun machines (#1 was Kodak) between 1985 and 
> 1990. We did sell approx. 20% of all Sun 3-50s that have been build.
> Even later, I had the fun to compare and before 1996 I would never have even
> thought about using a PC instead of a real Sun machine.

This was 1996.  Admittedly I was replacing old Suns with new PC's, but I
came out ahead and except for a few people who used those big
non-multisync monitors as space heaters all anyone saw was a speedup.

> Until 1995, PC have been very flaky and did crash frequently while Sun's did
> run approx. a year without a reboot.

Several of these Sun's crashed or rebooted at least monthly.  They
didn't have a lot of RAM and I suspected a memory leak in the OS.
I've forgotten what version of sendmail was on them but it was far
behind what Linux distributions had at the time and was known to be
buggy.

> Regarding speed: do not judge from compile times or from the numbers of 
> benchmarks. It is a lot more expensive to compile and optimize code for sparc
> processors than doing the same for intel but this does not make sparc slow.

You need to compare dollars to dollars not just processor to processor.
Spend X amount on a Sun, then take the money you save by using a free
OS on a generic pentium box to add more/faster drives and a ton of RAM,
and then compare real world speeds.  That's the way you would compare
different brands of the same item and the only meaningful way to
compare different items.  After all, that's why we are running unix-like
OS versions to hide the hardware differences...

> Intel on the other side is a clever company and did influence the C standard
> in a way that helped to make benchmarks look nice on Intel based systems.

Most of these machines weren't CPU bound anyway.

> It seems that you should first try to read a bit about Sysv..... SVr4 
> (introduced in 1990 - 15 years ago) of course includes NFS support because it
> is build on top of a SunOS-4.x kernel.

Which came from AT&T SysVr2 with TCP and utilities from BSD bolted on.
All of the things we think of as 'like unix' were already in SysVr2,
although the OSI-based networking was a little different and RFS was
very different from NFS.

> > Like I said, I'd love to see an alternative that can do all the same
> > things.
> 
> For all my use, I see star as the best way to go. I use it on a dayly base 
> since 1984. This is long before GNU tar even exists.....

I'm not sure you understand the added value of amanda. If you have a
big tape drive and one or a few filesystems it might not make much
difference for you.  However, if you have many filesystems and a
tape drive that can only manage to squeeze in a few full runs among
mostly incrementals, amanda is almost the only thing that can make
sure you get at least an incremental every day and a full of each
filesystem within a defined tape 'set'.  It will use the lowest
incremental levels possible that can accomplish this. It also
optionally indexes all the contents so for a restore you can tell it the
machine, filesystem, file, and date you want and it will tell you which
tape to insert.  I'm not interested in switching to a system that
does not offer these features.  The current star might do the basics,
but I'm not sure how amanda/star would deal with indexing such that
it knows what tapes you need and in which order to restore an
old file that was under a directory renamed before an incremental.

> > > Wrong: star is a drop in replacement for tar.
> >
> > Star is not a replacement for tar in any usage that is not precisely
> 
> WRONG: please first read the man pages for both star and tar and 
> try to avoid to again confusing GNU tar with tar.....

Perhaps you are confusing my system with Solaris. On my GNU/Linux system
tar is GNU tar and there is no proprietary or BSD version. 

> > I don't care about command line syntax, except that compatibility would
> > help with integration into existing tools.  The showstopper is the
> > unnecessary marriage to filesystem boundaries and inability to pick up
> > contents of directories renamed across excluded/included areas.
> 
> It does not make sense to start implementing the GNU tar bugs just to be
> 100% compatible with GNU tar.

Do it any way you like, as long as you can match the feature of
including old files under renamed directories regardless of their
original location.  If they exist in the included part of the
incremental run, they must exist after a restore.  I don't see
any way to claim that omitting them is correct under any circumstances.

> So you did not get it as you still follow the NIH syndrom....
> What you call "adapt packages to their own conventions" is nothing but 
> a deviation from useful UNIX rules just because they apply to the NIH
> syndrom.

There have been no UNIX rules since sysv and bsd split in the way
daemons are started at boot time and there is no sense in pretending
otherwise.   Packages must follow the distribution conventions or
things won't work.

> > > OK, but why do you believe do GNU tar maintainers ignore bug reports
> > > for serious problems?
> >
> > Perhaps no one supplies a simple script to reproduce the errors.
> 
> I did! But after a few years, I did of course stop wasting my time
> with bug reports to the GNU tar maintainers. It makes no sense if
> they do not show any reaction on bug reports.

Well, now you have a second chance since the Linux distribution
maintainers fix bugs too.   Post it to the RedHat or fedora bugzilla
and let them fix it or push it upstream.  Nobody really like the days
when you had to build your own distribution from source but times have
changed and you can take advantage of the help.

> > Agreed, in theory, but it doesn't matter in practice.  What matters is
> > that you have a program that works.   When the standard committees
> > provide reference code or standardize on existing, available reference
> > code then everything is fine.  When they make up something that sounds
> > nice but doesn't exist, it likely never will and everyone that tries
> > will end up with something different.
> 
> ???

This is where you were supposed to point out the value of the open spec
by listing all of the correctly interoperating versions that have been
developed following the spec - and how I don't have to worry about any
differences among them because the spec covers everything needed for
incremental backups to always work correctly even when the code base
for the backup tool varies wildly from that used to restore.

> > > > their own maintenance on packages.  However, you are missing the point
> > > > here.  I don't want to defend GNUtar, I want to replace it.  I just
> > > 
> > > You can do it with star.
> >
> > I can repeat the things that star can't do as often as you can pretend
> > that it can...
> 
> I don't understand you here: you did already agree that what you like may
> be done with the current star if you just change the way you run your
> backup. Again: you want a pink pill that cures a disease and I have a 
> pill that cures the disease but it is yellow.

If star 'just worked' with amanda, I might be able to switch because
my existing backups happen to be matched to filesystem boundaries
now (but I'd have to check the exclusions).   I'm also somewhat
interested in using it as the transport for backuppc if it would
get acls right.  Backuppc currently only uses the timestamp for
tar incremental operations, so only the rsync transport is exact.

However, I really don't want to miss old files under renamed
directories, even if they would not have been included in a
previous run, and I'd like them to restore correctly across 
mount points.  A backup operator can't and shouldn't have to know
if star's conditions have been met.  And a restore operator shouldn't
have to know if the target mount topology matches the source.
These missing features amount to more than a color difference.  They
will lose data under many likely conditions. 

Permitting backups including incrementals to start at arbitrary
directories and cross arbitrary mount points, and dealing with
conversions of directories to mount points between runs are important
features too, but I'll grudgingly agree that those are issues that can
(painfully) be managed by an operator as they would have to be with
dump.  I appreciate the fact that GNUtar handles these situations for
me, but could live without them.

-- 
  Les Mikesell
    [email protected]