Re: Proposal for variable length extents

On May 30, 2006  16:38 -0400, Benjamin LaHaise wrote:
> tytso wrote:
> > Sure, but the solution to that isn't to use inefficient encodings, but
> > rather to add checksums.  The reason why I didn't make that part of
> > the proposal is because my current thinking is we should be adding a
> > CRC-16 to cover the entire inode, and a CRC-32 to cover the entire
> > leaf block in an extent tree.  So I agree, but I would handle it
> > outside of the extent encoding proposal.
> 
> I would suggest using a 64 bit checksum.  CRC is very expensive to calculate 
> on most CPUs as the inner loop has a data dependant load from a lookup table, 
> while checksums are much easier to update for partial block changes.  Given 
> that we're trying to catch the most obvious corruption, a checksum is
> probably  enough.  Sorry for not commenting sooner.

Interesting.  Do you have any good algorithms for computing non-obviously-weak
checksums efficiently?  I agree that we likely don't care about crypto strong
checksums, but rather blatant corruption.  We need some form of feedback
or we can lose out on some kinds of corruption if we just do a simple sum.

The other thing that is critical in any checksum or CRC is to include the
physical data location in the sum/CRC in order to catch the case where the
right data is written to the wrong part of the disk (e.g. 2TB aliasing bug
just recently found).

My thoughts were just to use the lib/crc32 or possibly cryptoapi and if
CPUs get h/w support for CRC32 we all win.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.