Re: time impact of the delta signature

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 26/02/2019 11:56, [email protected] wrote:
> Hello Denis, thanks for the detailed explanations.
> 
> Due to the heterogeneity of usages also I'm planning to use
> different signature block length. The possibility to have a
> user-selectable block size is surely a good option: if I archive
> virtual machine virtual disks images (this is my need at the
> moment) I can use a large signature block, but when I will have the
> need to archive small files, even the default 2048 B block length
> may be suboptimal (i.e. too large).
> 
> As you pointed previously you are considering enriching the
> option: "--delta sig:N" where N is the size of the delta signature,
> with the further option that, if N is small, a "variable blocksize"
> feature activates. May I give you the following suggestion?
> 
> I would like to see the following option (in compact form):
> 
> "--delta sig:N[:F]"
> 
> N can be any positive integer representing the signature block
> length. If N has a small value (1, 2 or maybe even 3), then the
> variable size activates. In this case the optional parameter F
> (real number, default 1.0) may be present.
> 
> In those circumstances, mathematically, if the filesystem size is
> S, the block-len size will be computed by dar as:
> 
> block-len size = F * S^(1/N) (with a minimum value, say 256 B -
> that I see also in the librsync proposal for variable blocksize).
> 
> If I use N=1, then the blocksize will be simply proportional to the
> filesize ( S^(1/1)=S ). Example: "--delta sig:1:0.0001" it means
> that the blocksize will be 0.0001 times the filesize.
> 
> If N=2 the blocksize will be proportional to the square root of the
> filesize ( S^(1/2)=sqrt(S) ). Example: "--delta sig:2" or "--delta
> sig:2:1" it means that the blocksize will be the square root of the
> filesize (suggested value).
> 
> Example: "--delta sig:2:4" it means that the blocksize will be four
> times the square root of the filesize (faster execution, but larger
> delta diffs).
> 
> If it is considered useful also the cube root, then it could be
> activated an option N=3.
> 
> For larger N values, N simply represents the constant blocksize
> (that you already implemented in dar-2.6.3.RC1).
> 
> It is just a little suggestion. I think that this kind of option
> could give sufficient flexibility on the variety of usages.
> 

I agree on the fact that having a field carrying different roles
(block size and function) is not ideal in term of use and understanding.

Thus since that time, I enriched my thought with the function
parameter not being a number but a real name and keep having on or
more parameters as number. something like this:

- --delta sig:<function>:<num>[:...]

where function could be "log", "square_root", "cube_root", "linear",
"fixed", while <num> would an integer number

- --delta sig:linear:10000

would correspond to your --delta sig:1:0.0001
in other word block size being 1/10000 of file size

- --delta sig

would equal to

- --delta sig:2048
which would be the same as:
- -delta sig:fixed:2048

for backward compatibility

following your notation with
- --delta sig:square:10

we would have

block-len-size = (S^2)/10

Doing that way is simpler to understand and simpler to sparse as well
in dar. The underlying libdar API would also be more readable that way.

This idea is not completely stabilized, I adopted you <num> parameter
and 'linear' function :-)

> 
>>> I'm using a librsync version 1.0 (default package as found in 
>>> opensuse 15.0), maybe latest versions are more efficient. Will
>>> try build 2.0.2.
>> 
>> Yep, this is something I have to test too...
> 
> 
> Tested, but from a performance point of view, I didn't notice any
> improvement.

good to know that, thanks for feedback!

> 
> Thanks, Andy.
> 
> 
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlx1lj8ACgkQCDGwvQPY
sYJLWRAAnBkUP4uFLaGKZt0F2i1n3QzwrB4TC8XDQRofJl2PPuMFxji2/7cd1mTu
nysUK+ffscX0FESbQUIUMDyAxZyFPQnjzOZ7IK3N/mczzTX6+5yD6Yo8tCix0w1m
CBSJxzwwDnDszRbh4A24j7C6ACH11nzfYQVcH6qANll5oATxFmsuQ94+5yCOr0e4
BrUY2czkrzidPnmISe4+cRn1fUs2861Zkyaccc3Pogg1xEhu/hJonuMilhlpnPdC
N+WGPA8KotFqqMUSAGNGlxdQhH5RcG+DBfkRul16tiritCMkPM6i1SL0Am3vMmAw
/DElwkZi1bt3zkV5tfCjsHxe92grnb9aqMHIHweoJLIu+TDuWZ8emepRS3FmS5TZ
zwKBfriYw9IF2H5hFmLWWJZUbS5SIWkL5Bx+7p3Bq/QPiVDt4jmLI7MDMQicq7ey
RvwkHCFHqz0pB2yPZpO7ZaLJV3MjRTtSaE/SZBEBJIzl9c4FvqWv0KiWLHuD1/Bd
nH3lEPQ0OZkobouK+gbKF2svPKfB/vK80P5B/fwzJEwpCd2JgGxeEMIgLu+P021W
5DctiFiNh135K8ZLVAe8N1lHmFMOOr71npKhG5ijfJmLhf27HLBHw/jOBSV3Muls
iechCfuS3qwky89m76UQKjKzREkOKrpyMGxBgpqbV/wPh4VX4TI=
=fjCh
-----END PGP SIGNATURE-----