Re: [rdiff-backup-users] Re: more info on 25gig files

On Fri, 2005-05-06 at 23:59 -0700, Ben Escoto wrote:
> >>>>> Donovan Baarda <[email protected]>
> >>>>> wrote the following on Sat, 07 May 2005 11:38:49 +1000
> 
> Ok thank you for the advice, I've changed the blocksize to
> 
> 	if file_len < 4096: return 64 # set minimum of 64 bytes
> 	else: return long(pow(file_len, 0.5)/16)*16

I'm not sure I like that code... try this instead;

  return max(64, int(file_len ** 0.5) & ~0xf)

or, if you find ** 0.5 to be a confusing sqrt;

  from math import sqrt
  return max(64, int(sqrt(file_len)) & ~0xf)

> I was aware of some of these issues, but I thought it worked the other
> way around.  If the block sizes are too small, there will be too many
> blocks, and 2 of them will have a non-trival chance of matching.  But
> if the blocksize is too big, and there are too few blocks, why would
> this increase the chance of collision?

Sorry if I confused things... more blocks (ie small block sizes)
increase the chance of errors... not the other way around. It was rsync
that originally had fixed block sizes, and hence too small blocks for
large files.

Your original proposal of max(512,filesize/2000) would have had even
larger blocks than max(64,sqrt(filesize)). This would have had the
interesting side effect of fixed sized signatures for files larger than
1MB. The only problem this would have had is the delta's would have been
larger than they needed to be. There might have also been some speed
consequences, but exactly what they would be depends on too many factors
to predict.

-- 
Donovan Baarda <[email protected]>
http://minkirri.apana.org.au/~abo/



-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20