Re: 16 bit checksums limit dar's potential

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ryan Thomas wrote:
> The analysis on error rates assumes that errors are random and
> uncorrelated.  But in reality that will not be the case. If a bit flips
> on a magnetic disk then the 8 physically neighboring bits on are suspect
> as well. 

I agree,

> Your error rate analysis doesn't cover correlated errors.  And
> note that encryption doesn't eliminate error correlation, it just
> spreads out the effect of the bit flip.

a single bit flip within a byte is always detected by the CRC algorithm.
It is also always detected between if flipped bits are from two adjacent
bytes. For other situation, when flipped are separated by more than 2
bytes, it has 1 chance on 16 not to be detected. I guess the bit flip
never occurs between bits that are much distant from each others.

>  For archive formats, it's
> probably better to be overly conservative which is why I was pointing
> out that the 16 bit CRC is too small.

OK, increasing  from 16 bits to 32 bits will increase the capability to
detect errors, but the situation will stay the same, dar will still
still only detect some but not all corruption (yet more than before) but
will not be able to recover them.

> 
> I want to use dar as a real archival tool for terabytes of data.  By
> archival, I mean that the original data will be removed from the system.
> Once this is done, all I have to go on for data integrity is what's in
> the DAR file and perhaps the contents of an auxiliary hash or par file.

You must understand that dar has just some basic integrity mechanisms
(CRC will be brought up to 32 bits it will stay basic anyway, in
particular for very large files within an archive). Like many other
application using files, it has not redundancy mechanism that let it
repair a damaged archive (take gzip, tar, pkzip, ar, an so on, modify
some bits of the file they produced, they will report corruption but
will not be able to overcome it).

Instead of having 32 bits CRC in suggest having CRC (which is computed
file by file) proportional to length of files, with a minimum size of
let's say 32 bits. Assuming 32 bits is enough to "protect" 4 GB of data
the CRC would increase by 32 bits for each supplementary 4 Gbyte of
data. For example, a file smaller than 4GB would have a CRC of 32 bits,
a file above or equal to 4G and below 8 GB would have a CRC of 64 bits
and so on. Would it address your need?

> And because the dar file and auxilliary file may get out of sync, I
> really want a single file solution.  My idea was to store 128 or 256
> byte checksums for every internal file of the archive as well as an
> overall checksum of checksums for the archive. That way a very
> high-confidence measure of the files integrity can be had without access
> to auxilliary data or original files.

I repeat, whatever is the size of the checksum it will never let you
repair a damaged archive. You need specific tools like par to achieve
this goal.

> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHoYzapC5CI8gYGlIRAq7/AJ9CryC2Tgri6cI5deKwrPXDv5QUoQCdGkCb
6Ebrv0s8bq6RX9AI3EAdP1I=
=Z5W/
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/