Re: Re: Perfomance problem backupscript (more details and tests)

On Thu, 2005-10-06 at 15:48, [email protected] wrote:

> The man page of stat says nothing about persistence of
> inode numbers.

At the time that was written, inode numbers were physical
things in the filesystem - and in fact real inode numbers
still are.  If the things on your filesystem don't persist,
you don't have much to back up.  The problem is that people
want to pretend that certain filesystems are unix-like when
they aren't, so stat() claims something is an inode number
that isn't.

> > I've never seen ctime described as anything but inode change
> > time.  What man page says otherwise?
> 
> We do not agree about wether  link()/unlink()  is the
> mandatory method to rename a file. I see it as a deliberate
> implementation decision and rather as an accident if an
> intermediate change of the inode's link count updates ctime.

At the time the ctime specification was written, there was
no way to rename a file or modify any attribute that needed
to be saved without modifying the inode.  This was not
an accident, and observing ctime changes was the designed-in
way to detect files that needed to be copied in incremental
backups.  Blame whoever you want for not maintaining that
mechanism in not-quite-unix-like systems today, but when
gnutar first used ctime it was a reliable mechanism.

> > Place the blame where it belongs.  The original design and
> > specification was solid.
> 
> It left the question open and two groups of squatters
> occupied the empty space.
> 
> If renaming affects the inode, then it affects ctime.
> If renaming only affects the directory, then ctime has
> to stay unchanged.

But, whoever removed the ability to detect renames at
incremental backup time should have provided a different
way. 

> The hardlink detection and restoring problem looks quite
> expensive in terms of computing time and memory. At least
> if there are lots of such shared inodes.

I'd guess the files average 10 links each in my backuppc tree,
but I think the problem is that since it is unusual to have a
large number of hardlinks the programs that handle them haven't
bothered to optimize the lookup at all.  They probably do a 
linear scan of a table to match each one.  Throw a few million
at them and they just crawl, even though that's not a big
number for things using hashed or indexed lookups.  It's a
recurring topic on the backuppc mailing list as people with
a 100 gig drive realize they need to move to something bigger
and wonder why it is taking days to copy.

-- 
   Les Mikesell
     [email protected]