Re: some thoughts of integrity tests for v 4 ("RsyncCsumCacheVerifyProb")

hi Jeffrey,

Am 23.03.2011 um 19:35 schrieb Jeffrey J. Kosowsky:

> Fresel Michal - hi competence e.U. wrote at about 18:59:12 +0100 on Wednesday, March 23, 2011:
>> hi
>> 
>> i was thinking about the current state of integrity-tests ("RsyncCsumCacheVerifyProb")
>> 
>> on compressed files this means 
>> # decompress
>> # create checksum of decompressed
>> # compare cached checksums
>> 
> 
> If checksum caching is on then the checksum is stored at the end of
> the compressed file. So the file does *not* need to be decompressed.
> Technically, the checksum is only added to the file the 2nd time the
> file is encountered which I imagine is probably due to the fact that
> the native rsync algorithm only transmits the block and full file
> checksums when the file already exists (otherwise perhaps only the
> full file checksum is transmitted)

i was writing about "RsyncCsumCacheVerifyProb" not the cache
When rsync checksum caching is enabled (by adding the --checksum-seed=32761 option to $Conf{RsyncArgs}), the cached checksums can be occasionally verified to make sure the file contents matches the cached checksums. This is to avoid the risk that disk problems might cause the pool file contents to get corrupted, but the cached checksums would make BackupPC think that the file still matches the client.
on compressed files this starts with "decompress ...." as described above?

>> what about:
>> on first run
>> # create checksum of compressed file = checksum_compressed_1
>> # decompress
>> # create checksum of decompressed
>> # compare checksums (of cache)
>> # if checksums (of cache) are the same - recreate checksum of compressed (just in case we had some disk issue meanwile) = checksum_compressed_2
>> # compare checksum_compressed_1 to checksum_compressed_2
>> # if the same - save checksum of compressed file for future use
>> 
>> the next run we just need to checksum the compressed file as (the internal cached checksums) were already verified above
>> 
>> results in faster rechecks afterwards:
>> -) no need to decompress 
>> -) checksum-calculations on compressed files is faster (less MB's to calculate)
> 
> Seems like a lot of extra work since Craig's clever implementation
> leverages existing rsync checksum calculations and gives the block and
> full file md4 checksums for free.

it's just for speedup of the following
With checksum caching enabled, there is a risk that should a file's contents in the pool be corrupted due to a disk problem, but the cached checksums are still correct, the corruption will not be detected by a full backup, since the file contents are no longer read and compared. To reduce the chance that this remains undetected, BackupPC can recheck cached checksums for a fraction of the files. This fraction is set with the $Conf{RsyncCsumCacheVerifyProb} setting. The default value of 0.01 means that 1% of the time a file's checksums are read, the checksums are verified. This reduces performance slightly, but, over time, ensures that files contents are in sync with the cached checksums.

We just need to checksum the compressed file as the blocks and full-file of uncompressed are already verified 

Greetings 
Mike

------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software 
be a part of the solution? Download the Intel(R) Manageability Checker 
today! http://p.sf.net/sfu/intel-dev2devmar

_______________________________________________
BackupPC-devel mailing list
[email protected]
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/