Re: more info on 25gig files

On 30/06/05 22:46, Donovan Baarda wrote:
>>One thing that's different between rsync and naive use of librsync (as
>>done by rdiff and, I think, rdiff-backup) is that the latter doesn't
>>have a strong full-size integrity check. With librsync's default 4-byte
>>checksums, using a small block size on a large file makes the
>>probability of silent corruption disturbingly non-negligible.
> 
> Actually, librsync uses 8 byte strong checksums, which combined with the
> 4 byte rollsum, gives you 12 bytes of checksum for each block (but you
> can only really count on 8 bytes for maliciously crafted data). This
> makes it comfortingly negligable unless you have huge files and a
> stupidly small blocksize.

Sorry, I meant 8. But as for huge and stupid, how about, say, a 4GB file
and 1K block size? Doesn't seem unreasonable if you're backing up a
large database with many small local changes. That gives you 2^22
blocks, hence 2^44 pairs of blocks, hence (for random data) a
one-in-a-million chance of a corruption.

As for the rollsum, I don't trust it even for non-malicious data. It's
too easy to believe it will be fooled by some natural structure in the
data. And as for malicious settings, well, last time we considered those
we saw corruption can occur with probability 1...


> The disturbing part is that any corruption that does occur is silently
> ignored.

Yes. Do you think the integrity testing should be done by librsync, or
by every client app (with a prominent notice that they should)?

  Eran


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click