Re: [rdiff-backup-users] Re: more info on 25gig files

>>>>> Donovan Baarda <[email protected]>
>>>>> wrote the following on Sat, 07 May 2005 11:38:49 +1000

Ok thank you for the advice, I've changed the blocksize to

	if file_len < 4096: return 64 # set minimum of 64 bytes
	else: return long(pow(file_len, 0.5)/16)*16

as you recommended.

> > By grief you mean in terms of running time and/or memory usage?  Or
> > are using too-large block sizes actually incorrect, and not just
> > inefficient?
> 
> Time and memory mainly, though it also contributes to the probability of
> a blocksum collision. In rsync, which used truncated md4's for the
> strong sum, the probability was getting pretty close to 100% for >1G
> files before the changes that tweaked the blocksum size and blocksize.
> librsync uses a full md4sum, so the probability is neglegable, but it
> still increases as the filesize increases.

I was aware of some of these issues, but I thought it worked the other
way around.  If the block sizes are too small, there will be too many
blocks, and 2 of them will have a non-trival chance of matching.  But
if the blocksize is too big, and there are too few blocks, why would
this increase the chance of collision?


-- 
Ben Escoto

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCfGdf+owuOvknOnURAogHAJ9F8cqsOLwyxQWS2VjURS8RZOOFFQCfUHqo
CEVayFBrDVhHC42b8CW2lx4=
=ehpq
-----END PGP SIGNATURE-----