Re: time impact of the delta signature

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 24/02/2019 14:56, [email protected] wrote:
> Hello Denis,
> 
>> Il 22 febbraio 2019 alle 21.55 Denis Corbin <[email protected]>
>> ha scritto:
>> 
> 
> [...]
> 
>> Implementing this feature, things got more complicated than 
>> expected due to the fact that when comparing an archive contents
>> to filesytem the following situation may arise: - - archive
>> contains only a delta signature - - comparison is done by
>> computing the delta signature of the file present on filesystem,
>> which result depends on the block size used and must chosen the
>> same as the one used for the signature in the archive.
>> 
>> So I had to store the signature block size information inside the
>>  archive, which implied incrementing the archive format to be
>> able to read older archive that do not have this information
>> present (2048 is assumed in that case). This had some
>> consequences on archive repairing, merging and isolation over
>> just archive creation.
> 
> [...]
> 
> When I proposed the simple solution of just overriding the default 
> length with a user-selectable size, I did make some tests and
> didn't find any problem with mixed delta signature blocksize. Maybe
> there exist corner-cases that I did not consider?

You are correct, the signature is used the same by librsync whatever
is the block size used to built it. The point as explained is when
comparing an archive with filesystem, if a file only contains a delta
signature: dar reads the file data on the filesystem and computes a
delta signature to compare it with. If you do not know the block size
used for the signature you have in the archive, you will not be able
to build a correct signature to compare with and thus to know whether
the file differs or not between the archive and the filesystem.

This is really a corner case, I agree, but I could not let dust in the
corner, so I slightly modified the archive format for that reason only.

> 
> For instance I built the standard 2.6.2 dar version, and performed
> a full backup (with signature) and isolate the catalogue with 
> signature. Then I built a patched dar (changed the default
> signature block-size, in this example to 6001 bytes). If I make a
> small difference on the file on the filesystem and perform the
> incremental archive (with delta sig) based on the previous isolated
> catalogue (that was constructed with the default 2048 blocksize),
> the resulting incremental archive is perfectly produced. Also a
> further patching produce a correct incremental archive.

this is correct. You do not compare an archive (or isolated catalogue)
against filesystem having only delta signature for some files.

> 
> I also noticed that, as expected, the first incremental archive
> size (without signature) is slightly larger than 2048 B, whereas
> the second incremental archive size (without signature) is
> slightly larger than 6001 B (i.e. the granularity of the signature
> block is visible in the produced output).
> 
> I attach a script that describes the operations performed (it is
> not automatic, for instance patching dar and rebuilding should be
> done manually).
> 
> I thought that the block-signature size (and also there may be
> other parameters like blake2 algorithm, a manual specification of
> the strong-len size) was entirely managed by librsync without any
> need to store specific data in the dar archive file. If I was
> wrong, at this points, also the other parameters I cited may need
> to be also stored by dar.

you are not wrong. Though if in the future some option let the user
specify the algorithm for delta signature they should also be dropped
beside the signature itself in the archive.

I've also looked at librsync internal data structures to read the
block size from it, but it implies to load the whole signature in
memory very early in the dar operations (which has some significant
impact on libdar) and nothing warranties that the implementation will
not change in the future. So I avoid to use something else than the
official librsync API and added the information beside the librsync
signature.

> 
> Best, Andy
> 

Regards,
Denis
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlx0SJsACgkQCDGwvQPY
sYKs0hAApFGhaV9rCPT2hfOo+wuRyWNTXbsEAV5rP4vFc81Bdlm8+C/ZByqFy1x0
4lWNZEBQHt+NoEQy+t0cHbxxvX+kenmBaTmzizO23EAVfybz40qj4VvBUpdeHmge
b0h/fP4uDu7Rvv51PZ8oX1YO9UUkzae71wWyH2+cQUGpTds71EunEcG+Me54CrTU
gGaWFb7iNsnOUShpidAl5voySqFlQgqv/9hj4DfaY+50bwbum8P34b7y4OrAXfRQ
GeiHX7FJ7RL9ctEwJ53HlZHbz84Hij1X3cX1QwbwVrNWCi/NVmgPQ/JFdxCXhKF4
cTV7Zaqf/WfXDh0h2cv2E0wmGNtCNxv9NfEWt1wbCKvykjgZt9MJ+cjehUh/mPyB
2mN99FgSDJsQElWIa5qjDx5yGYQimjyQBobSK1nskLJljymMANV1/iybZh+mwGtX
jHieovlzhsU4gSOhjuXrRLrCyeWHJn7cEdKcmz7NJ/bYW6KsfQDY5v+j5F96K4sP
y1efyMEDaNVIEgsflZAmmjq/QCH7jkqJjU6xzj0zIchQQB4V3izi8YfSo36w7Pth
tqIwo0ggUpM9rWaCmSZ/Re5rD9GJwGkEu7A7dI5EC+rM2jKagib/tROyV6tZtISn
V9plNOnkD9nIXAeVz1ySL3fvjl/GtSkrSp2wz/K97pfUeS9EEy8=
=HFBY
-----END PGP SIGNATURE-----