Re: [rdiff-backup-users] Re: more info on 25gig files

On Fri, 2005-05-06 at 11:49 -0700, Ben Escoto wrote:
> >>>>> Donovan Baarda <[email protected]>
> >>>>> wrote the following on Fri, 06 May 2005 15:47:10 +1000
[...]
> Ok, so by sqrt you mean sqrt rounded to the nearest 512, with a
> minimum of 512?  Is 512 a magic number in some way?

512 is only a magic number in that it is 2^5, and computers tend to like
^2 numbers. I don't think it matters exactly... and I'm not sure what
rsync rounds to for it's blocksizes, if it does at all.

Hmm, just dug up some old emails, and it looks like it rounds to the
nearest 16. I'm not sure what the minimum is though.

The minimum size exists because at some point you start to hit the point
of diminishing returns... each block has a 12 byte strong+weak sum in
the signature. You should pick a minimum block size based on your
minimum acceptable signature size compared to the file size... I would
think 64 would be OK (ie, signature is 12/64 the size of the file).

> > So yeah, I'd definitely switch to the sqrt thing, you will get a lot
> > less grief as files get larger.
> 
> By grief you mean in terms of running time and/or memory usage?  Or
> are using too-large block sizes actually incorrect, and not just
> inefficient?

Time and memory mainly, though it also contributes to the probability of
a blocksum collision. In rsync, which used truncated md4's for the
strong sum, the probability was getting pretty close to 100% for >1G
files before the changes that tweaked the blocksum size and blocksize.
librsync uses a full md4sum, so the probability is neglegable, but it
still increases as the filesize increases.

In rsync, blocksum collisions are detected and handled because it also
uses a whole-file sum. In librsync, there is no whole-file sum, so
collisions are not detected, and you can get silent corruption.
Fortunately, librsync's much larger strong sum makes this a very low
probability. Eventually we should add a whole-file sum to rdiff, but you
should also use larger blocksizes for larger files to be on the safe
side.

Note that even if we do add a whole-file sum to rdiff, it will only
allow you to detect when corruption occurs. rsync has the added luxury
of being able to re-try the transfer using different checksum seeds
several times before falling back to a whole-file transfer.

-- 
Donovan Baarda <[email protected]>
http://minkirri.apana.org.au/~abo/



-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20