Re: Questions about binary deltas and an internal error

On Thu, Mar 09 2023, Denis Corbin wrote:

> On 09/03/2023 01:29, John Goerzen wrote:
>> Hello,
>
> Hello John,

Hi Denis!

Thank you for the reply.

>> So I was a little bit at a loss as to just when/how delta signatures can
>> be used.  I guess I would need to have enough space locally to store a
>> temporary copy of the entire backup, then isolate the catalog from it,
>> then send it along?  I'm not sure I can really rely on having that much
>> space available.
>
> Well, you can also proceed to catalogue isolation at the backup
> destination host, assuming you have dar available there too.

Unfortunately, that isn't practical, since I don't have a guaranteed way
to get it back to the system being backed up in time for the next
backup.

So let me give you the commands you were asking for, and also be very
clear about the two dar crashes.

Crash #1, on extraction: 100% reproducible on an archive containing
delta diffs, but only when --sequential-read is used on data piped in on
stdin.  When the backup file is first written to disk and then processed
without --sequential-read, it does not occur.

Crash #2, on creation: it was 100% reproducible with one particular
isolated catalog basis file, but generating a catalog file that causes a
crash seems more difficult.  Sadly I realized about 10 seconds after
deleting the problematic one that I should have saved it for your
inspection.

So I'm generating it with:

Running dar -q -c - -A /var/local/dar/cat -R / --empty-dir
--retry-on-change 0 --compression=zstd:4 --delta sig --exclude-delta-sig
baksnap/matrix/synapse/storage/media-store --delta-sig-min-size 1M -P
run -P tmp -P proc -P sys -P dev -P var/tmp -P var/cache/apt/archives -P
media -P mnt -P cdrom -P var/cache/apt -P var/lib/mlocate -P
var/lib/apt/lists -P matrix-data -P matrix -P var/log/journal -P
var/spool/nncp -P var/local/dar -P var/cache/apt -P mythmount -P
var/lib/docker -P */no-backup -P home/*/.cache -P var/log/*

dar -q -c - -A /var/local/dar/cat -R / \
   --empty-dir \
   --retry-on-change 0 \
   --compression=zstd:4 \
   --delta sig \
   --exclude-delta-sig baksnap/matrix/synapse/storage/media-store \
   --delta-sig-min-size 1M \
   -P run -P tmp -P proc -P sys -P dev -P var/tmp -P var/cache/apt/archives -P media -P mnt \
   ... a bunch more -P omitted ...
   -P 'var/log/*' \
| pee --no-ignore-sigpipe --no-ignore-write-errors \
   cat \
   "dar -q --sequential-read -A - -C \"$DARMETA/$THISCAT\" --compression=zstd:4 --delta sig --exclude-delta-sig baksnap/matrix/synapse/storage/media-store --delta-sig-min-size 1M" \
   nncp-file....

So, "pee" takes the output from dar and pipes it to two commands: cat,
and dar -C.  The file at $DARMETA/$THISCAT is renamed to
/var/local/dar/cat.dar.1 after successful completion, to be used for the
next run.

Extractions are using:

dar --dirty-behavior no-warn --alter=unix-sockets -w -x /nncp-bakfs/tmp/dartmp-file -R /bakfs/other-backups/dar/...

previously, I was using --sequential-read -x -

Is there any other info I could provide that would be helpful?

Thanks!

- John

>
>>
>> BUT... it actually worked!  Maybe!  It generated delta-managed dar
>> files.
>>
>> But on extraction, I got:
>>
>> ---- exception type = [BUG] ----------
>> [source]
>> #011File filesystem_tools.cpp line 573 : it seems to be a bug here
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(_ZN6libdar4EbugC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x13d) [0x7fe3c659311d]
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(+0xa5bd8) [0x7fe3c64f9bd8]
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(_ZN6libdar18filesystem_restore5writeEPKNS_10cat_entreeERNS0_Ut_ERbS6_S6_S6_+0x780) [0x7fe3c65aace0]
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(_ZN6libdar14filtre_restoreERKSt10shared_ptrINS_16user_interactionEERKNS_4maskES7_RKNS_9catalogueERKNS_4pathEbbbbbRNS_10statisticsES7_bNS_17comparison_fieldsEbbbRKNS_11crit_actionENS_23archive_options_extract7t_dirtyEbbRKSt3setINS_10fsa_familyESt4lessISN_ESaISN_EEb+0xf55) [0x7fe3c65c2d95]
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(_ZN6libdar7archive9i_archive10op_extractERKNS_4pathERKNS_23archive_options_extractEPNS_10statisticsE+0x3e9) [0x7fe3c65d69e9]
>> #011stack dump : /lib/x86_64-linux-gnu/libdar64.so.6000(_ZN6libdar7archive10op_extractERKNS_4pathERKNS_23archive_options_extractEPNS_10statisticsE+0x122) [0x7fe3c653d3d2]
>> #011stack dump : dar(+0x46bd3) [0x56396ff80bd3]
>> #011stack dump : dar(+0x4f0d0) [0x56396ff890d0]
>> #011stack dump : dar(+0x22b71) [0x56396ff5cb71]
>> #011stack dump : /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fe3c6048d0a]
>> #011stack dump : dar(+0x22cea) [0x56396ff5ccea]
>> [most outside call]
>> -----------------------------------
>>
>> INTERNAL ERROR, PLEASE REPORT THE PREVIOUS OUTPUT TO MAINTAINER
>>
>> That error is:
>>
>>                  if(!patcher.has_patch_base_crc())
>>                      throw SRC_BUG; // s_delta should have a ref CRC
>>
>> This is dar 2.7.8 on Debian x86_64.
>>
>> Are there other solutions?  Am I missing something here?
>
> Thanks for feedback. I will have a deeper look at this soon, but it
> seems the generated backup does not hold the checksum for the file in
> the state it should have before applying the binary patch. Thus dar
> cannot check the file is in good status for applying the patch.
>
> If you have a simple scenario of backup creation (with all options used)
> that leads to this condition at extraction time, it would help a lot.
>
>
>>
>> Thanks!
>>
>> John
>>
>
> Thanks in advance,
> Denis