Re: time impact of the delta signature

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 14/02/2019 12:24, andycapo--- via Dar-support wrote:
> Hello Denis

Hello Andy,

> 
>> [...] the delta signature algorithm is implemented in librsync
>> library that libdar relies on throught its API. Thus there is
>> little thing I can do about it from libdar... and splitting the
>> files per block on which to calculate the delta signatures will
>> make the thing very complicated in dar archive format as you will
>> have to detect block boundary (adding marks in the data flow for
>> example), protect them against corruption, remove these marks
>> from the data, stick block of file data toghether in order to
>> restore a given file... [...]
> 
> In reality I'm not asking to change dar's internal architecture,
> but only the parameter of the delta checksum mechanism, when
> calling librsync. I made a quick study of the problem and my
> hypotesis revealed to be correct. I propose you a solution to
> improve the signature speed dramatically (the execution time of the
> signature is minimal instead of the huge overhead encountered). Let
> me say that I use the delta diff only if the files are bigger than,
> say 100 MB. The granularity of the difference (i.e. the minimal
> chunk to be written in delta file) for my application is, let me
> say, 4 MB (whereas default value is 2048 bytes).

OK, sorry for my misunderstanding!

> 
> In practice, for me it is sufficient to patch the file
> [...]/src/libdar/generic_rsync.cpp with the following difference: 
> --- #ifdef RS_DEFAULT_STRONG_LEN job =
> rs_sig_begin(RS_DEFAULT_BLOCK_LEN, RS_DEFAULT_STRONG_LEN); #else //
> should use RS_BLAKE2_SIG_MAGIC in place of RS_MD4_SIG_MAGIC // but
> not compatible with librsync < 1.0 -	    job =
> rs_sig_begin(RS_DEFAULT_BLOCK_LEN, 0, RS_MD4_SIG_MAGIC); +	    job
> = rs_sig_begin(1024*1024*4, 0, RS_MD4_SIG_MAGIC); #endif

OK, I see


> --- i.e. by substituting the RS_DEFAULT_BLOCK_LEN with my own datum
> 1024*1024*4.
> 
> I tested and signature, delta diff and extraction of incremental
> delta diff works properly (and really fast).
> 
> Surely the BLOCK_LEN could be dynamically adjusted to the file
> size. But for now this value simply works well.

If I correctly understand [1], using block size equal to the file size
reduce CPU but remove the ability to do delta patch ("[...] file copy
if changed"). It is suggested to rather use "sqrt(filesize)"

And another drawback is memory consumption which is increased
proportionally to this block size. This may be a problem when
computing delta signature for huge files. So I will avoid linking the
block size to the file size but increase the default value to 4 MB as
you suggest and probably add an option to let the user define this
value if needed.

> 
> Regards Andy
> 
> 

Thanks for your feedback,

Best Regards,
Denis

[1] https://sourceforge.net/p/librsync/mailman/message/480886/
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlxlysMACgkQCDGwvQPY
sYIhVBAArJCJoZL4xhFimVvsZdXmjntZwyEQJtAjLomKw9jsrzNfcAsoOpdmmGTb
80/tIXLfd9gbjBowbtU/fWKfH9aDfVKePF0QrfkJxdBBCKid6PG6hQQ3KTJfoiPg
Ezy21xjZQGjd2u1lZ2USYNgoIYPRf1Aj29sbEX7Ynv4jFv2a8kYlL85KEuJlq3Z7
afy6qrWETDuqRb77mWT2I1qAqT6PleZ/8eS9kbrfqUcTa9JMY2sjGU/DdTRGDwMK
s3dXD02MJW0BhPYgMB2agXYPxwR0vOqmrow7kIzHDD3i0vdA4piA4/f1Xfr2LmfN
ac7bFzOpOpRCEjg8g18IpKRb2TGGY84nf00OghQD8UrBeNjBauE0XcI4whM3h886
d6VCxW1Ra56UPCpCkD4YUo2ZG5G4H7RysQFNGkdMqAfEXvCnRJ6Vhp5YRgJ0QFCA
BeqgsZoS7MXbwMOKll2Tj4XDsE5/BSCUpE9Iv/4FCEi57bxMVDjVWV+YY4qqgaFY
vBU7ASctyF/SG414zQcP/u5ZrpJCTJOZ17iFx779jUhnCLlSUQ32+QBOkDRQHy4m
DVoZFlyBYDxscLxcef65xOemZ7n9v3ThZg0BTbhUfdCWEdvvPJO3uV1Y8uBla3Uv
QfLp1P4W25ZgEeW4FDaFXpZ/N4G0VD4f6nPqz0dB3/Tnl1+0jWU=
=89WZ
-----END PGP SIGNATURE-----