Re: questions about --retry-on-change and --ref

On Mon, 26 Apr 2021 23:41:17 +0200
Denis Corbin <[email protected]> wrote:

> On 26/04/2021 18:49, imschmeg wrote:
> > On Sun, 25 Apr 2021 22:08:37 +0200
> > Denis Corbin <[email protected]> wrote:
> >   
> >> ...  
> [...]
> > 
> > I'm going to use a lock around the backup processes so that this
> > kind of overlap doesn't occur.
> > 
> > One other question comes to mind that is somewhat related.  Suppose
> > that a backup is broken slightly (a bit flipped, for instance).
> > Will using that backup as a reference with dar --ref to make an
> > incremental or differential backup detect the breakage and fail?   
> 
> it depend where the corruption took place. If it took place in the
> catalogue, which is used for differential backup, dar will complain.
> You will have to fallback to repair the backup (-y option) or use
> --sequential-read with --ref for dar read it sequentially.
> 
> if corruption took place in the middle of a file's data or EA, dar
> will not see it for this operation as it is not used for
> differential/incremental backup
> 
> > I am
> > using par2 with dar, so I can potentially fix such breakage, but I
> > have noticed that par2 verify is quite slow on large archives, so I
> > don't run it every time prior to using a backup as a --ref (my plan
> > is to run par2 repair once a month).  
> 
> You should not have to repair not even "verify" a backup unless dar
> reports a error. CRC is calculated on any part of the archive, there
> is still a theorical possibility for double corruption that precisely
> compensate and make the CRC stay valid, but that's very improbable.

Isn't it worth running par2 repair on backups that would otherwise not
be used at all for long periods of time (but are still on-line)?
Wouldn't that help prevent the accumulation of errors that would
otherwise eventually exceed par2's ability to repair?

Also, in my tests, par2 repair takes about the same time as dar does to
test the backup, as both are mostly IO bound tasks with about the same
amount to read. So it seems like a waste of time to only run par2
repair after a dar test fails.  Unless that dar test is part of some
other operation one is going to do anyway, and is similarly capable
of detecting errors to par2.

> 
> > What I'm most afraid of is some
> > breakage in a --ref archive that gets propagated to subsequent
> > incremental backups without detection by dar, and that such breakage
> > cannot be fixed with par2 repair running on those incremental
> > backups.  
> 
> well, still theoretically possible but even less probable, probability
> to such even is less than wining the first price at loto, I guess :)
> 
> par2 would fail repairing if the corruption was important (not a few
> bits that compensate for the CRC to stay valid), in that case dar
> would also report the problem.
> 
> > So, I am hoping that dar does at least a quick checksum check of its
> > --ref argument backup. The man page does not specify an error code
> > for this case, though, unless this case counts as part of exit code
> > 5. 
> 
> There is Checksum (CRC) on each file's data (CRC size increases with
> file's data), same with EA, metadata, and archive structure including
> the tape marks, catalogue, archive header and slice header.
> 
> To be noted that compression as well as encryption are both very
> sensible to corruption, most of the time long before CRC can be
> triggered, the deciphering engine will fail and dar will propagate the
> error, then the decompression engine, if by any chance the first
> missed the corruption (very improbable) or if the archive was not
> encrypted.

Oh, I didn't expect the checksum or CRC to be such that it would be
less sensitive than the encryption or compression it is guarding.

In that case, I will create isolated catalogues, and create par2 files
for them (I noticed that dar's par2 option does not create par2 files
for catalogues, so I will call par2 create directly). I will run par2
repair on a catalogue immediately before using it as a reference.  I
have tested par2 repair on catalogues, and it is very fast because
catalogues are so small in comparison to their associated backups.

Thanks!