Re: Perfomance problem backupscript (more details and tests)

On Fri, 2005-10-14 at 05:04, Joerg Schilling wrote:
> >
> > Posix 1.x was less than useful.  It specified nothing that really
> 
> I am sorry, but I have the impression that you just don't know POSIX.1-1988 :-(

Just enough to know it didn't match SysV at the time.

> The tar archive definition did not change since then, so why should
> the same thing be useful now but not useful in 1988?

It isn't useful for backups now.  There is no way to specify all files
that existed at the time an incremental was taken so that others can
(optionally) be deleted during a restore.  Otherwise you will accumulate
all files that ever existed when you restore a series of incrementals
and run out of disk space.  You might as well use cpio.

> > > Other tar's did start to support POSIX.1-1988 around 1990 (this includes
> > > Solaris - the most posular UNIX).
> >
> > UNIX?  Is that still around?  I gave up on Solaris when it would have
> 
> Are you trolling?

Just enough to offset your hint that Solaris is the One True Way.


> > cost more for a software update from the horribly buggy 2.4 to something
> > that worked than it did to replace the hardware with faster pentium
> > boxes running a free version of Linux.  I've never had a reason to
> > reconsider that decision.
> 
> You _are_ trolling!

Not about this.

> In 1993 (the time when Solaris-2.4 was recent), Linux was completely unuseable
> except fot playing. At that time, Pentium CPUs have been _boringly_ slow....
> 1994 was the first time, when Intel did come close to the speed of a 1998
> Sparcstation-II and 1995 was the first time when Intel (with a few too
> expensive for real use systems) didcome to the same speed as Sparc.
> PCs of that speed still have been a lot less reliable and even more expensive
> than Sun systems.

I started at a company that already had the old Sparc, IPX, and IPC
boxes in 1996.  They were slow, buggy, and crashed regularly, and
_just_ an OS update would have cost more than faster pentium boxes.
I don't even want to think about what it would have cost to get them
up to equivalent RAM.

> Since 5 years, Solaris-8 is free for personal & research/educational use.
> Since half a year, Solaris-10 is free for any kind of use (even commercial).

> Wake up and come back to the real world..... What do you believe where Linux
> takes nearly all of its new ideas from?

AT&T SysVr4, the same place Solaris got them except for NFS. But AT&T
didn't know how to price things for the consumer market either.

> > > GNU tar was created 1989 from PD/SUG-tar by breaking POSIX.1-1988 compliance 
> > > of the original source :-(
> >
> > And thus it became usable.
> 
> WRONG :-(
> 
> This incompatible hack was the cause for various incompatibility problems
> wwith the real world and still is as GNU tar (even the latest version)
> by default still creates non-POSIX archives.

Like I said, I'd love to see an alternative that can do all the same
things.

> > > This is a problem of the obviously missing maintenance for Amanda.
> > > As Amanda claims to support ufsdump/ufsrestore, it should take about
> > > 2 hours (for a person who knows the Amanda source) to add a different
> > > CLI string for a _very_ similar interface.
> >
> > Agreed - it could now replace dump easily on the backup side. It's not
> > a drop-in replacement for tar since it will not work in all the same
> > situations.  So you'd need a backup side driver like dump's and a
> > restore-side more like tar - not sure where indexing falls.
> 
> Wrong: star is a drop in replacement for tar.

Star is not a replacement for tar in any usage that is not precisely
tied to filesystem boundaries.  Dump already has that constraint so
star could replace dump for amanda usage.

> >
> > The 1% that is impossible is the problem.  If it were just syntax we
> > could use a shell wrapper.  And neither the GNU or stock getop() works
> > with old tar syntax.  That's the one thing that should have been broken
> > on purpose by the first usable version and everyone would have forgotten
> > it ever existed by now.
> 
> I am sorry, but do not expect me to implement GNU getopt() bugs just for the
> fun of being 100% GNU tar compatible. I am using my getargs() which is
> more flexible that GNU getopt() and even older.

I don't care about command line syntax, except that compatibility would
help with integration into existing tools.  The showstopper is the
unnecessary marriage to filesystem boundaries and inability to pick up
contents of directories renamed across excluded/included areas.

> > There's that UNIX word again.  I expect my software to be fixed to use
> > the conventions of my distribution, regardless of the originality of
> > the developer.  Packagers for other distributions should do the same.
> > I'd prefer that anything built from a developer's tarball land
> > entirely under /usr/local, but I don't expect any agreement about that.
> 
> I expect distributions to adopt to the usual conventions instead of
> suffering from the NIH syndrom like many Linux distributions do.

There are no 'usual' conventions for unix-like systems and never have
been since the BSD modifications.  As a result, OS distributions have
to adapt packages to their own conventions.

> > > So you don't use GNU tar frequently and don't see the bugs?
> >
> > Apparently I don't do anything that triggers them.
> 
> OK, but why do you believe do GNU tar maintainers ignore bug reports
> for serious problems?

Perhaps no one supplies a simple script to reproduce the errors.

> > I have no problem with GNUtar using a unique format.  I don't foresee
> > any situation where using GNUtar to extract/restore would present a
> > problem, given source availability.   I would be interested in any
> > repeatable way to produce GNUtar archives that GNUtar won't read
> > itself.
> 
> GNU is not the center of the workd, if you talk about archive standards,
> international standard commitees are the right people to look at and not
> the FSF.

Agreed, in theory, but it doesn't matter in practice.  What matters is
that you have a program that works.   When the standard committees
provide reference code or standardize on existing, available reference
code then everything is fine.  When they make up something that sounds
nice but doesn't exist, it likely never will and everyone that tries
will end up with something different.

> Following Open Standards is more important than being open source.....

I'll believe that when I see bug free code being written independently
to perfectly match these Open Standards.  I don't expect that in
my lifetime.   Given a choice between using the same code with some
known defects and using an assortment of different versions that
almost certainly include new and unknown bugs, I'll usually stick to
the devil I know.  I'll change my mind when I switch to a platform
that can't compile C code, I suppose.

> > > Now tell me please: do you trust in a tar implementation that is maintained
> > > in way that ignores bug repports for showstopper bugs for more than 10 years?
> >
> > Compatibility with ever-changing standards committee decisions isn't a
> 
> I encourage you to edocate yourself about the work of the POSIX standard
> commitee before you spread FUD on them.....

Have they spec'd a usable incremental backup system?  Has someone
implemented it exactly to specs?

> > their own maintenance on packages.  However, you are missing the point
> > here.  I don't want to defend GNUtar, I want to replace it.  I just
> 
> You can do it with star.

I can repeat the things that star can't do as often as you can pretend
that it can...

> This looks like you suffer from a disease and you like to get pink
> pills to cure them but I have only blue pills.
> 
> Star is able to do the backups even for your needs. But please 
> believe me that I will not implement any hacky new feature that
> does not even allow to really do more then you could do now,
> while star is in a state where the current development cycle
> is in a "die away" state. 

Dump could do my backups if I wanted to follow the same restraints.
I use tar because I don't want to be restricted to filesystem
or exclusion boundaries and I don't want the restore operator to
have to know or care anything about the source topology.  Noting
new/renamed directories in a way that can traverse devices is
not 'hacky'.

> And _before_ I consider to implement a different backup method 
> that does not really give new features, I will need to add support
> for extended attribute files in order to support all Solaris features.

I don't understand why you think an incremental that spans device
and exclusion boundaries would not be a new feature for star.

-- 
   Les Mikesell
     [email protected]