Re: time impact of the delta signature

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 16/02/2019 14:11, [email protected] wrote:
> Hello,

Hello Andy,

> 
[...]
> 
> And another drawback is memory consumption which is 
> increasedproportionally to this block size. This may be a problem
> when computing delta signature for huge files.
> 
> Interesting discussion the [1] you cited. I see that memory
> compsumption is defined as
> 
> “[...] Memory = blocksize + 16x(filesize/blocksize) [...]”
> 
> I assume the units of the coefficient "16" is Bytes. So the memory
> usage increases also on the number of checksum blocks (that are
> defined as filesize/blocksize: but each block occupies only 16
> bytes? doesn't the block checksum size increase with block size?).
> 
> Anyway, if the equation is correct, the equation tells us that
> there exist, for each file size, a blocklenght that guarentees the
> minimal memory compsumption, i.e. if the block is too small or too
> large we encounter a memory overhead.
> 
> The "memory-optimal blocksize" BLOCK_LEN is:
> 
> BLOCK_LEN_MEMOPT = 4 x sqrt(filesize)

memory or CPU reduction may not match all needs when some (like me)
want incremental backup storage minimization whatever it cost as CPU
(less data to send over Internet to a public cloud storage you pay for).

Anyway, interim release 2.6.3.RC1 is available with a new option to
set the block size to use to build the signature. GIT is also up to
date on branch_2.6.x (both github and sourceforge)

ftp://ftp.dm3c.org/dar.linux.free.fr/Interim_releases/

The default behavior is unchanged (2048 block size). if you want to
change this default, in place of --delta sig use --delta sig:<blocksize>
in your case "--delta sig:4M"

Implementing this feature, things got more complicated than expected
due to the fact that when comparing an archive contents to filesytem
the following situation may arise:
- - archive contains only a delta signature
- - comparison is done by computing the delta signature of the file
present on filesystem, which result depends on the block size used and
must chosen the same as the one used for the signature in the archive.

So I had to store the signature block size information inside the
archive, which implied incrementing the archive format to be able to
read older archive that do not have this information present (2048 is
assumed in that case). This had some consequences on archive
repairing, merging and isolation over just archive creation.

Well now after manual testing it works fine, I will run the long test
script (make check) soon. In the meanwhile you can test and provide
feedback here.

At a moment I intended to reserve low values like 2 and 3 as square
root and cube root of the filesystem but as file size are stored as
infinint (libdar internal type) I would have to implement those
mathematical function for this class... and on the other hand
computing these mathematical function for each file would cost CPU
more than necessary.

Any idea is welcome, though :)

> 
> (I get this result by imposing that the first derivative of memory
> with respect to blocksize is zero and then solving for the
> blocksize).
> 
> In order to be more efficient in speed, it may be useful to
> consider a larger blocksize, let's say a factor 10 with respect to
> the value computed of BLOCK_LEN_MEMOPT.
> 
> So I will avoid linking theblock size to the file size but
> increase the default value to 4 MB as

Finally, no, the default value has not been changed, but you can
change it at will with this new feature as explained previously

[...]
> 
> Regards
> 
> Andy
> 

Regards,
Denis
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlxwYbIACgkQCDGwvQPY
sYLADg/+IObZGRY2oUp8rhP2oafxZ/ILlLQmothY5vx5flwc2N372SKMVX4G9iGf
RbUmYjojF1jm1Q8HxHiX7jWhF8A2XPrLoI/fdz3m0A6r2phE+oaDFk/i7yd9CcRg
3P3C5feeZTHngrF5ayYK9WZb2zbABW2Q2bHZkByjp0b5MDD07VOyIkkA2Dcuc1Kq
2dQFHm+Ta1QtWtENGxXHAoh/2PoLCNOv5U6b5oBSZthi2DtX4S3zHv5tWAtVnnJY
H4+LJtom2lenCIuAjnzCs6HbNv9252ydKubwRdjQNHz4KW5VS+8QBMtiBzVQbDzo
PCaHHv0xRAYlHMYw/FTsWN1KaUOgQ7yEXErDLNm6PP05WmQXvC6NeATmbYBJxyZ0
dBaFsAI9mFC+qOlyuudKM5+Rlkb2LJe+lYxC8TTxEgTkFWdYCfJM0flDDIIO2vTR
SuVutJVUS1/n0BvpA9b0EihKXG7y0BT4lPo/ve56OUzy7wKv7AmQ6lE2xArakiEr
c6WSu8I/KrQ/wPFM8Zfo3Vy9Qpl/z6neSlQWsJouQNrhrP2TrRqEMt2uOlKmb+DL
gq4gdGxVeTejClVvevUE1o6roAZNQinJSNT58i2pDaOua7xcSQ1oA6XjCy/23wh1
Vn4K/vsZsDWPKKf8g7i0HlKzhB8cFKCd8RnG4dNlMZ/2cMTD10I=
=ucBU
-----END PGP SIGNATURE-----


_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support