Re: time impact of the delta signature

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 23/02/2019 10:46, [email protected] wrote:
> Hello Denis,
> 
>> Il 22 febbraio 2019 alle 21.55 Denis Corbin <[email protected]>
>> ha scritto:
> [...]
>> 
>> Anyway, interim release 2.6.3.RC1 is available with a new option
>> to set the block size to use to build the signature. GIT is also
>> up to date on branch_2.6.x (both github and sourceforge)
>> 
>> ftp://ftp.dm3c.org/dar.linux.free.fr/Interim_releases/
>> 
>> The default behavior is unchanged (2048 block size). if you want
>> to change this default, in place of --delta sig use --delta
>> sig:<blocksize> in your case "--delta sig:4M"
>> 
> 
> Thank you, for my usage on virtual machines images backup it is
> just perfect. By the way, also I do upload dar backups on the
> cloud, but as I showed with the tests, with the default signature
> block size, the dar signature time is longer than the cloud upload
> time for files larger than 5 GB.

Correct. However there is also the storage requirement you pay for on
the cloud to take into account. Doing some tests with signature block
size of 4 MB, I found that a lot of average size files had delta patch
hundred time larger than with the default block size (2 KB)...
depending on what you backup (large VM disks, outlook mailboxes or at
the opposite a plethora of average size files) the storage size is
significantly larger with 4 MB than with the default 2 KB even taking
into account the larger delta signature generated by the later. For my
own usage CPU load is not an issue, it takes longer to run in
background, but at least I can have larger backup history on a fixed
amount a cloud storage :-)

> I'm using a librsync version 1.0 (default package as found in
> opensuse 15.0), maybe latest versions are more efficient. Will try
> build 2.0.2.

Yep, this is something I have to test too...

> 
> [...]
>> 
>> At a moment I intended to reserve low values like 2 and 3 as
>> square root and cube root of the filesystem but as file size are
>> stored as infinint (libdar internal type) I would have to
>> implement those mathematical function for this class... and on
>> the other hand computing these mathematical function for each
>> file would cost CPU more than necessary.
>> 
>> Any idea is welcome, though :)
>> 
> 
> An efficient implementation of the approximated square root
> (rs_long_sqrt), here: https://bit.ly/2GXqpeV You may be interested
> in taking a look to this patch to librsync: https://bit.ly/2GLlsXB
> in particular commit https://bit.ly/2GGjXcS

thanks for that, will look at it :)

> 
>> 
>> Regards, Denis
> 
> Best regards and thank you again! Andy
> 
> 

Regards,
Denis
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlx0REwACgkQCDGwvQPY
sYIQzBAAvp+O5QlyPyggquU8uzkZPxt2Yw2qmuRzQvn+e+mHfl68ovBK/G9X4gos
9aVkWcnEBBgJRc6FseFlEfmDx5geW6Mmu4MBiLotOXI4H1N1HpIXVXdmRP65naF2
j0dM2+zMaEI54m9sqKqg/Ec3KDYEzX5pbRTMbjX4JpR+QCbzqe527SdJLQiwFTp4
f66nmgZ5Qcd2PGGn0Xsp60/aAMI7IZDGVCuTR9lUu9YpDTomiHA7AiLo2lRq/96A
9JodJl0UKpOOOz2Exw3ucrOOx/n5Bc+49KdKp4X6Fi6CeFB7YjYEKz2t82ysWTs3
XDjxqkwy8uz26XlFpo+6PibMtnvv75tZ6bj+kQWoI8yMbnK4h+EcSzPvyIvA7Ram
Id0n8NPBrjLEPvYYAQR07N7s7zz/GnSxJ+BDTdeRUT5KrC1HpYKVLaJSVQH5X+ZJ
5zH6nU7nOhwZeFiFbiQdQ+6JqTyzhF+iOPIHzkxlsU4z1V9pGt8GO1Skn/r3a0uP
lqmYIvoU6Hidro236OfV0aZ3lTvFowDR9xb4Mvz10bO9iemN2Zpg+J1K6TWxz3VN
PvYW5ev4yDdmyNPiEqeiJafvTe2ce9wH1mXiuLCjiXIwwkpLZVGnz7NeS0e+T3Xv
Cb8HCoiuoEaqGi8mVEFgD1/GP1YmzN8rqYf+wbI4QpxNAyZ+L0A=
=tWXh
-----END PGP SIGNATURE-----