Re: suggestion: do not keep files in tar archives.

On Tue, 2003-12-16 at 09:40, Miek Gieben wrote:


> Afio can, just as tar, gzip the archive on the fly. Now, with tar it
> turned out
> that this has bugs, or didn't work over the network. Whatever the
> reason I have
> since done this myself in 'hdup'.

I'm curious what drove this decision -- I think it may slow things down
since you have to copy far more data between the processes.

> ah, that makes sense. But then the fact of encryption remains, but
> this can also be done on a per file basis... hmm....

You could use afio's -P option to run your own script rather than gzip
for compression.  The script could compress and encrypt.  It might be
super slow, though, to do so many separate encryptions.

> I don't want to do 1), because that would mean hdup isn't backward
> compatible anymore, and I also have issues with 2)...

As long as the default tar behavior is unchanged, you haven't lost
backward compatibility, no matter what you do with afio archives.

> My question is this: if you gzip afio archives in this way, will there still
> be a speed benefit compared to tar? You will need the entire archive before
> being able to unzip it....

For the .tar.bz2 files I have, about 99% of the CPU time is taken by the
decompression, and only 1% for the identification and extraction of a
file (verified by running time on bunzip2|tar).  (If I was restoring a
complete disk rather than just the single-file restore I tested, then
some of that time would have to go into disk writes.)

If the backups are uncompressed, the bottleneck is probably going to be
data transfer rate (over USB or a slow network), so there would be a
great benefit of not having to read the entire file.  The advantage of
afio is entirely that it does compression of the individual files rather
than of the whole archive.  It would probably be slightly slower than
tar if you are restoring an entire disk, and will certainly compress
less well since it can't take advantage of duplications between multiple
files.

You could probably get equivalent speed, but even less compression, by
using 'zip'.  It would have the advantage of wide availability.

Bng