Re: suggestion: do not keep files in tar archives.

Miek Gieben <[email protected]> writes:

| [On 16 Dec, @16:09, Boris wrote in "Re: [hdup-user] suggestion: do ..."]
| > On Tue, 2003-12-16 at 09:40, Miek Gieben wrote:
| > > that this has bugs, or didn't work over the network. Whatever the
| > > reason I have
| > > since done this myself in 'hdup'.
| > 
| > I'm curious what drove this decision -- I think it may slow things down
| > since you have to copy far more data between the processes.
| 
| I don't know anymore... :P... but wait it's coming back... I think it
| had something to do with incremental backups and zipping them. Doing
| that with tar gives you I full dump... 
| 
| I'm as said, i'm not sure and I haven't looked at it since, maybe newer
| version of tar do it correct now.
| 
| About the copying, the data has to be copied anyway, tar forks off
| the gzip process, unless they use shared memory (unlikely) it is
| just copied from one process to the other.
| 
| > > ah, that makes sense. But then the fact of encryption remains, but
| > > this can also be done on a per file basis... hmm....
| > 
| > You could use afio's -P option to run your own script rather than gzip
| > for compression.  The script could compress and encrypt.  It might be
| > super slow, though, to do so many separate encryptions.
| 
| Don't know, depends on the encryption I guess. I see no theoretical reason
| why encrypting 1 gig of data should be any slower then encrypting 10 times
| 100 meg of data. (excluding process startup - which could be significant
| in a 2 gig archive with a lot of files).
| 
| > 
| > > I don't want to do 1), because that would mean hdup isn't backward
| > > compatible anymore, and I also have issues with 2)...
| > 
| > As long as the default tar behavior is unchanged, you haven't lost
| > backward compatibility, no matter what you do with afio archives.
| 
| True, but still not very nice and clean.
| 
| > > My question is this: if you gzip afio archives in this way, will there still
| > > be a speed benefit compared to tar? You will need the entire archive before
| > > being able to unzip it....
| > 
| > For the .tar.bz2 files I have, about 99% of the CPU time is taken by the
| > decompression, and only 1% for the identification and extraction of a
| > file (verified by running time on bunzip2|tar).  (If I was restoring a
| > complete disk rather than just the single-file restore I tested, then
| > some of that time would have to go into disk writes.)
| 
| So how would that change if the compression was done first ala afio. Something
| like cat filelist | gzip | tar xf -.... Damn that cannot work because 'tar'
| no longer knows the filename of the file in filelist... Ok, tar and afio 
| files need to be handled differently.
| 
| For encryption everything will stay the same (encrypt the ENTIRE archive). 
| 
| > If the backups are uncompressed, the bottleneck is probably going to be
| > data transfer rate (over USB or a slow network), so there would be a
| > great benefit of not having to read the entire file.  The advantage of
| > afio is entirely that it does compression of the individual files rather
| > than of the whole archive.  It would probably be slightly slower than
| 
| Exactly, that's THE difference. 
| 
| Blackie: could you verify that? ie. using no compression would make tar
| and afio run at the same speed. If so, I will (try to) add afio to hdup.
I'm not sure what you want me to do.
I may however add that I do not compress files, as this is very dangerous
just one bit error on your hard disk, and the 32Gb file is useless (at
least as I see it)

Anyway, it seems like hdup fits peoples need, and from Boris Goldowsky's
mail, it indicates that rsync-backup would fit mine, so feel free to ignore
me ,-)

Cheers
Jesper.