Re: time impact of the delta signature

Hello Denis,

> Il 22 febbraio 2019 alle 21.55 Denis Corbin <[email protected]> ha scritto:
> 

[...]

> Implementing this feature, things got more complicated than expected
> due to the fact that when comparing an archive contents to filesytem
> the following situation may arise:
> - - archive contains only a delta signature
> - - comparison is done by computing the delta signature of the file
> present on filesystem, which result depends on the block size used and
> must chosen the same as the one used for the signature in the archive.
> 
> So I had to store the signature block size information inside the
> archive, which implied incrementing the archive format to be able to
> read older archive that do not have this information present (2048 is
> assumed in that case). This had some consequences on archive
> repairing, merging and isolation over just archive creation.

[...]

When I proposed the simple solution of just overriding the default length with a user-selectable size, I did make some tests and didn't find any problem with mixed delta signature blocksize. Maybe there exist corner-cases that I did not consider?

For instance I built the standard 2.6.2 dar version, and performed a full backup (with signature) and isolate the catalogue with signature.
Then I built a patched dar (changed the default signature block-size, in this example to 6001 bytes).
If I make a small difference on the file on the filesystem and perform the incremental archive (with delta sig) based on the previous isolated catalogue (that was constructed with the default 2048 blocksize), the resulting incremental archive is perfectly produced. Also a further patching produce a correct incremental archive.

I also noticed that, as expected, the first incremental archive size (without signature) is slightly larger than 2048 B, whereas the second incremental archive size (without signature) is slightly larger than 6001 B (i.e. the granularity of the signature block is visible in the produced output).

I attach a script that describes the operations performed (it is not automatic, for instance patching dar and rebuilding should be done manually).

I thought that the block-signature size (and also there may be other parameters like blake2 algorithm, a manual specification of the strong-len size) was entirely managed by librsync without any need to store specific data in the dar archive file. If I was wrong, at this points, also the other parameters I cited may need to be also stored by dar.

Best,
Andy

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support

# ================ build and install dar standard version 2.6.2

darVersion="dar-2.6.2"
cd ~/build/dar
tar -xzf ${darVersion}.tar.gz
cd ~/build/dar/${darVersion}
./configure --prefix=/usr --enable-mode=64 --disable-ea-support --disable-dar-static --disable-build-html
make
sudo make install-strip
cd /usr/bin/
sudo chmod 4755 dar dar_*
cd ~/build/dar/${darVersion}

# ================ create a 64 MB random file for testing

cd ~/test_dar/deltasig
mkdir orig
dd if=/dev/urandom of=orig/sample.bin bs=64M count=1 iflag=fullblock

# backup
dar -c "orig" -R "orig" -g "sample.bin" --delta sig -va
# isolate catalogue
dar -C "orig.cata" -A "orig" --delta sig -va

# now "orig.cata" is the isolated caralogue containing the delta signature (2048 blocksize)

# go back to dar:
# patch manually the file [...]/src/libdar/generic_rsync.cpp
# with the following difference:
# ---
# +	    job = rs_sig_begin(6001, 0, RS_MD4_SIG_MAGIC);
# ---
# i.e. now the signature is 6001 bytes
# then build and install
#...

# now go back to the test environment:
cd ~/test_dar/deltasig
mkdir patch1
cp -ar orig/sample.bin patch1/sample.bin
# patch the file
printf '\x31\xc0\xc3' | dd of=patch1/sample.bin bs=1 seek=1MB count=3 conv=notrunc
# ( diff orig/sample.bin patch1/sample.bin ) Binary files orig/sample.bin and patch1/sample.bin differ
# create incremental archive patch1
dar -c "patch1" -A "orig.cata" -R "patch1" -g "sample.bin" --delta sig -va

# the patch file is corretly produced and if I omit signature, its size is slightly more than 2048 (the original block size of the checksum)
# I can test
dar -x orig -va
diff sample.bin orig/sample.bin
# OK
dar -x patch1 -va
diff sample.bin patch1/sample.bin
# OK

# another patch
mkdir patch2
cp -ar patch1/sample.bin patch2/sample.bin
# patch the file
printf '\xcc\xaa\x00' | dd of=patch2/sample.bin bs=1 seek=40MB count=3 conv=notrunc
# create incremental archive patch2
dar -c "patch2" -A "patch1" -R "patch2" -g "sample.bin" --delta sig -va

# the patch file is corretly produced and if I omit signature, its size is slightly more than 6001 (the new block size of the checksum)
# I can test
dar -x patch2 -va
diff sample.bin patch2/sample.bin
# OK