Re: The Verification Treadmill

On 15/02/2024 20:05, Robert Nichols wrote:
> On 2/15/24 09:47, Dominic Raferd wrote:
> ...snip...
>> So the only way to be confident about *all* the data in a repository 
>> is to use 'rdiff-backup verify' to verify each and every backup 
>> session in each repository; and this includes verifying the current 
>> 'mirror' session (even though it is held in the clear in the 
>> repository). This needs to be done with reasonable frequency to 
>> ensure that backed-up data has not deteriorated (e.g. through media 
>> bitrot).
>
> That's the way I do it. My verification is done in conjunction with my 
> periodic (~weekly) sync of my primary backup archives to separate 
> media. I verify all of the new levels that are being synced plus at 
> least one more level to ensure that the new levels mesh properly with 
> the ones already synced.
>
>> All of which takes a lot of computing power and time, much of which 
>> is duplication of effort (because, as stated above, the verification 
>> of the earliest session in a repository confirms the integrity of all 
>> later versions of files that it contains, but it is not possible to 
>> exclude these files from re-verification for more recent sessions).
>
> Actually that is not sufficient to verify the intermediate levels. 
> Let's say one block of a reverse-diff file for backup level -3 gets 
> corrupted. That's going to cause a verification failure for level -3. 
> But, if a diff for level -5 replaces that same block in the file, then 
> level -5 and all previous levels will verify correctly. Only levels -3 
> and -4 will fail.
>
> There is no substitute for verifying each and every level of the 
> backup archive. I have a script that does verification of 8 levels in 
> parallel on a system with a lot of memory. Because those threads are 
> for the most part all reading the same files, all but the first get 
> that data from the kernel's buffer cache and do not incur any I/O 
> delay. I find that 8 threads in parallel execute almost as fast as a 
> single thread. I have 64GB of RAM to play with, and my machine isn't 
> doing much else while I'm sync-ing backups, so YMMV. Trying to do this 
> on a Raspberry PI would be an entirely different story.
>
Thanks Bob for your insights. I have a similar strategy using my 
timedicer-verify script 
(https://www.timedicer.co.uk/programs/help/timedicer-verify.sh.php, 
currently being tweaked for compatibility with rdiff-backup 2.2+), and 
then (for backup to secondary media) my timedicer-mirror script. But I 
can't help thinking that there should be a better way: rdiff-backup 
should offer a single-action 'complete repository verification' which 
would avoid unnecessary duplication of effort and confirm that the whole 
repository is OK. Another one for the wish list perhaps?

Until then, I am interested in your parallel processing approach. 
Presumably you start 8 parallel rdiff-backup verify sessions for 
datetime points -1 to -8 (and then, when they are all complete, -9 to 
-16, -17 to -23...)? And you run 8 in parallel because your CPU has 8 cores?