Re: Corrupted output on errors

On 21/11/2021 22:10, John Goerzen wrote:

[...]

>>> This archive was generated by dar 2.6.13 on a Raspberry Pi running ext4 and unpacked
>>> by 2.7.2 on x86_64 running ZFS.
> 
> I'm going to add that I have now also experienced this on a sender that
> is x86_64 running dar 2.6.13 atop ext4.  The receiver is 2.7.2 on x86_64
> with ZFS in both cases.

OK, this is not related to a different CPU

> 
>>
>> This error can come from a corrupted transfer between hosts or from dar
>> itself (bug). Could you thus:
> 
> The dar file itself was piped to zstd and then gpg (for encryption)
> prior to transmission, so I think it is highly unlikely that corruption
> occurred in transit.

I agree, this is not likely the problem neither,

> 
>> - check that the backup testing (dar -t <backup>) succeeds with dar
>> 2.6.13 on Raspberry?
> 
> I didn't happen to have the file from the Raspberry still around, but I
> did have the one from the x86_64 sender that triggered the same kind of
> issue, so I tried it there:
> 
> After decrypting the file:
> 
> zstdcat foo.dar.zst | dar --sequential-read -t -
> 
>  --------------------------------------------
>  204381 item(s) treated
>  0 item(s) with error
>  0 item(s) ignored (excluded by filters)
>  --------------------------------------------
>  Total number of items considered: 204381
>  --------------------------------------------
> 
> (without --sequential-read, I got an error about corrupted data; though
> I note I did use --sequential-read on the command I gave)

yes, this is still interesting to know the error message reported when
testing the backup without --sequential-read option, as it gives some
information about the backup format or misformatted issue. And already
it pushes away any filesystem related issues (ZFS).

Is it possible to get the output of the non sequential-read mode testing
adding -va option (this will make a lot of data)?  [dar -t <backup> -va]

Of course, you can send it to me directly (compressed of course,
encrypted if you want --- you should be able to find my gpg key easily).
 If you do not want to expose the whole backup filename, you can scrub
them, and eventually provide only a tens of lines before and after the
error.

Can you also confirm that you are not using temporary files to compress
and cipher the backup?

By the way, note that you have probably better security using the
embedded ciphering of dar as it adds a salt (argon2), and elastic
buffers at the beginning and at the end of the ciphered data (which are
variable amount of random data), this greatly reduce the risk of code
book attack/clear text attack.... and dar can cipher using several
threads even when outputing to pipe.

> 
>> - make a sha1 sum of the dar backup on Raspberry?
>> - check this sha1 sum against the dar backup on the x86_64
> 
> These backups were made by nightly automated scripts, which didn't save
> the files on the source machine, but due to the wrapping in zstd and
> gpg -- plus integrity checks at the transport layer -- I think
> corruption in transit is highly unlikely.
> 
>> - test the backup on x86_64 (dar -t <backup>) using dar 2.7.2 ?
> 
> Same result as on the source.

OK

> 
> You also asked about if there was any other output from dar prior to the
> error.  There was only this line:
> 
> No terminal found for user interaction. All questions will be assumed a negative answ
> er (less destructive choice), which most of the time will abort the program.

OK, that does not help much

> 
> Thanks!
> 
> - John
> 
>>

Cheers,
Denis