Re: suggestion: do not keep files in tar archives.

[On 16 Dec, @16:09, Boris wrote in "Re: [hdup-user] suggestion: do ..."]
> On Tue, 2003-12-16 at 09:40, Miek Gieben wrote:
> > that this has bugs, or didn't work over the network. Whatever the
> > reason I have
> > since done this myself in 'hdup'.
> 
> I'm curious what drove this decision -- I think it may slow things down
> since you have to copy far more data between the processes.

I don't know anymore... :P... but wait it's coming back... I think it
had something to do with incremental backups and zipping them. Doing
that with tar gives you I full dump... 

I'm as said, i'm not sure and I haven't looked at it since, maybe newer
version of tar do it correct now.

About the copying, the data has to be copied anyway, tar forks off
the gzip process, unless they use shared memory (unlikely) it is
just copied from one process to the other.

> > ah, that makes sense. But then the fact of encryption remains, but
> > this can also be done on a per file basis... hmm....
> 
> You could use afio's -P option to run your own script rather than gzip
> for compression.  The script could compress and encrypt.  It might be
> super slow, though, to do so many separate encryptions.

Don't know, depends on the encryption I guess. I see no theoretical reason
why encrypting 1 gig of data should be any slower then encrypting 10 times
100 meg of data. (excluding process startup - which could be significant
in a 2 gig archive with a lot of files).

> 
> > I don't want to do 1), because that would mean hdup isn't backward
> > compatible anymore, and I also have issues with 2)...
> 
> As long as the default tar behavior is unchanged, you haven't lost
> backward compatibility, no matter what you do with afio archives.

True, but still not very nice and clean.

> > My question is this: if you gzip afio archives in this way, will there still
> > be a speed benefit compared to tar? You will need the entire archive before
> > being able to unzip it....
> 
> For the .tar.bz2 files I have, about 99% of the CPU time is taken by the
> decompression, and only 1% for the identification and extraction of a
> file (verified by running time on bunzip2|tar).  (If I was restoring a
> complete disk rather than just the single-file restore I tested, then
> some of that time would have to go into disk writes.)

So how would that change if the compression was done first ala afio. Something
like cat filelist | gzip | tar xf -.... Damn that cannot work because 'tar'
no longer knows the filename of the file in filelist... Ok, tar and afio 
files need to be handled differently.

For encryption everything will stay the same (encrypt the ENTIRE archive). 

> If the backups are uncompressed, the bottleneck is probably going to be
> data transfer rate (over USB or a slow network), so there would be a
> great benefit of not having to read the entire file.  The advantage of
> afio is entirely that it does compression of the individual files rather
> than of the whole archive.  It would probably be slightly slower than

Exactly, that's THE difference. 

Blackie: could you verify that? ie. using no compression would make tar
and afio run at the same speed. If so, I will (try to) add afio to hdup.

> tar if you are restoring an entire disk, and will certainly compress
> less well since it can't take advantage of duplications between multiple
> files.
> 
> You could probably get equivalent speed, but even less compression, by
> using 'zip'.  It would have the advantage of wide availability.
> 
> Bng
> 

grtz
      Miek
--
fingerprint = E1EB 29B8 8FA2 2923 62B8  0A2B 64B8 F15C 7764 AB4B
http://miek.nl/about.html