Re: Asking for comments on a dar's future implementation

Le 06/02/2010 12:42, Denis Corbin a écrit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear all,
>
> Well, its my turn to ask for help about dar! :-)
>
> As you may known, current implementation of dar (releases 2.3.x) make
> use of checksum to detect data corruption. However, as it has been
> stated, the fixed length of checksum used is not adapted for arbitrary
> large files. I am thus currently working on improving that point and am
> trying to define a function that can provide, given the size of a file,
> the total size of the checksum to use for that file.
>
> However, in one hand, I lack trusty information on the average I/O error
> rate of different media to base this implementation on. And, in another
> hand, I have found necessary and interesting to get your points of view,
> as you are (most probably) the future users of this feature.
>    
Hi,

    What about using a rolling checksum algorithm here ?
It doesn't have to be dependent on file size, and would allow very fast 
file difference computation (I highly advise you to read about rolling 
checksum).
The basic idea of the rolling checksum is that for every file you're 
checksuming, the file is analysed as a sliding windows (for every new 
byte read, it changes the checksum, and the new checksum contains the 
signature of all the previous bytes, so it's possible to locate where a 
byte was corrupted/changed, while with a classical checksum you can't).
By the way, using rolling checksum would allow rsync-like difference 
computing and this would really reduce the file size to the exact 
modified bytes in incremental archiving.

I don't think there is a plausible relation between error rate and hash 
size. Whatever the size, hash collision are very rare.
However the probability of collision can't be related to the media error 
rate.

Best regards,
> If you are interested in participating in this discussion, I invite you
> to subscribe to dar-discussion[1] mailing-list which for that point is
> more appropriated than dar-support mailing-list, and provide there you
> return and/or expectations on this feature. Note that you only need to
> subscribe if you plan to send your contribution to this discussion.
> Second point: be advised that your first email will be moderated, which
> may takes some hours to complete, further ones will not be if you
> conformed to the chart of the list. Last Point: any email used for
> sending in this mailing-list is hidden from archives and web-interface.
> Only other persons subscribed at the time of the posting can see them.
>
>
> Thanks for your help.
>
> Regards,
> Denis.
>
> [1] https://lists.sourceforge.net/lists/listinfo/dar-discussions
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFLbVWSpC5CI8gYGlIRAh9bAJ48lN4/dPQr9tlFSv5qVeVN5JcmkwCfevCF
> 1Wh88mwLluv4GPvy08hQD2A=
> =/cyI
> -----END PGP SIGNATURE-----
>
> ------------------------------------------------------------------------------
> The Planet: dedicated and managed hosting, cloud storage, colocation
> Stay online with enterprise data centers and the best network in the business
> Choose flexible plans and management services without long-term contracts
> Personal 24x7 support from experience hosting pros just a phone call away.
> http://p.sf.net/sfu/theplanet-com
> _______________________________________________
> Dar-discussions mailing list
> Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
>
>
>    



------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com