Re: Questions about binary deltas and an internal error

On 09/03/2023 01:29, John Goerzen wrote:

Hello John,

I took some time to find the problem and have now a clearer view of what 
succeeds:

First, you miss-read the documentation (!) and I did also after you :)

[...]

> 
> dar --sequential-read -A - -C new... --delta sig --include-delta-sig "*"
> 
> But then I see that this might not work either, because:
> 
> "delta signature transfer is not possible when the archive of reference
> is read in sequential mode"

This previous sentence applies to the backup process not to isolation. I 
initially thought the problem was around this but in fact there is no 
problem here, everything works as expected: you can isolate a backup 
using sequential-read and delta sig at the same time.

[...]
> 
> But on extraction, I got:

As you mentioned on your following email this error does only appear 
when restoring in --sequential-read mode.

> 
> ---- exception type = [BUG] ----------
> [source]
> #011File filesystem_tools.cpp line 573 : it seems to be a bug here
[...]

the root cause is a design error (!)

for a given file under backup, when a delta patch is computed the data 
is stored in that order in the resulting backup:

- inode metadata (copy of what is also present in the catalog and know 
at that time, owneship, permission, file name, ... but the storage size 
is not yet know for example at that time and is not present here)
- file data, or here in the context of delta patch, the binary patch
- CRC on the previous field (file data)
- base CRC (the expected CRC the file should have before applying the patch)
- eventually a delta signature (signature of the new file state, for 
recursive binary delta)
- result CRC (the expected CRC of the file once patched)
- eventually EA...
- eventually FSA...

for those curious about the overall archive format, things are detailed 
here:

http://dar.linux.free.fr/doc/Notes.html#archive_structure

Now sequentially reading a backup  the base CRC is not yet read when the 
file's data (here the patch data) is about to be read.

Before applying the patch dar tries to check the base CRC which has not 
yet been read which trigger the exception you caught. Thus storing the 
base CRC before the file's data should solve the issue.

I'm looking how to do that properly as the inode metadata is dropped by 
some object type (cat_inode) while the delta signature relative fields 
are written by some other object type (cat_delta_signature) and file's 
data by yet another object (cat_file)...

I also have to take into account archive backward compatibility even if 
the restoration of delta patch in sequential read mode has probably 
never worked... and I also must understand whether it has even worked 
(in which case why has it passed the already stuffed testing phase, see 
"make check"), or whether this has changed upon new feature addition or 
other code modification...

In the meanwhile, the workaround you have found to not restore in 
sequential-read is correct and the method you plan to use creating a 
isolated catalogue in sequential read with delta signature using pee is 
good.

isolated catalogue do not suffer of this problem as all the data is 
stored in a catalogue which is loaded into memory before use.

Cheers,
Denis