Binary delta in 2.6.0
gulikoza via Dar-discussions <dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org> Tue, 25 Dec 2018 21:26:02 +0100
| Newsgroups | gmane.comp.sysutils.backup.dar.general |
|---|---|
| Message-ID | <[email protected]> |
This is a multipart message in MIME format.
--===============7227505164832671095==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0001_01D49C98.6FF07A60"
Content-Language: sl
This is a multipart message in MIME format.
------=_NextPart_000_0001_01D49C98.6FF07A60
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
Hi Denis,
Congratulations on dar 2.6 release :-)
Thank you for all your hard work all these years!
I really enjoy working with dar, especially because of your attention to
detail which makes dar very versatile tool.
Unfortunately I didn't really have the time yet to test the binary delta
feature before the final 2.6.0 release, but it's just in time to make my
yearly full backups with the signatures now and start testing binary deltas
on production, haha ;-)
Just to let you know, there is an issue compiling dar 2.6 in RHEL-7 due to
gcc 4.8.5:
In file included from i_archive.hpp:35:0,
from archive.cpp:31:
erreurs.hpp:110:15: error: function 'libdar::Egeneric::niveau&
libdar::Egeneric::niveau::operator=(libdar::Egeneric::niveau&&)' defaulted
on its first declaration with an exception-specification that differs from
the implicit declaration 'libdar::Egeneric::niveau&
libdar::Egeneric::niveau::operator=(libdar::Egeneric::niveau&&)'
niveau & operator = (niveau && ref) noexcept = default;
.
(^ and so on, basically all the prototypes with noexcept = default)
I understand that gcc 4.8.5 has a very limited (buggy) support for c++11,
but unfortunately I guess this means dar 2.6 will not be available in
EPEL-7.
It should probably also be stated in the documentation that gcc 4.8 is no
longer supported (if this was the intention).
I was able to compile it with devtoolset-7 (gcc 7.3.1), create a RPM (along
with statically linked libthreadar) and the resulting RPM seems to work fine
for now on vanilla Centos 7 (without the newer gcc and libs installed).
Looking forward, I see that dar is using RS_DEFAULT_BLOCK_LEN which is just
2048 bytes. If the signatures will be switched to Blake2, this means the
signature will be 36 bytes (256-bit blake2 + 4 bytes crc32) for each 2k
block. This might be a bit too much for larger files and I was thinking that
instead of having (yet another) command line option for that, dar might try
to optimize the block_len based on file size. I don't see the need for 2k
signatures on multi-gigabyte files, where 8k or 16k signature might be just
ok and a lot smaller. I have a proof of concept patch for that already, for
instance with the 2GB file the blake2 catalog size is reduced from 36MB to
9MB with 8k blocks (default dar catalog with md4 signature would be around
18MB for the same file). I can send the patch to github if needed ;-)
And to nitpick a bit, the man page is not mentioning [Delta] in the [data]
displayed fields. :-)
Thanks again and Happy Holidays!
Regards,
gulikoza
------=_NextPart_000_0001_01D49C98.6FF07A60
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type content=
=3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros=
oft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DSL link=3Dblue vlink=
=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span lang=3DEN-US=
>Hi Denis,<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:=
p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> =
;</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Congratulations o=
n dar 2.6 release :-)<o:p></o:p></span></p><p class=3DMsoNormal><span lang=
=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-U=
S>Thank you for all your hard work all these years!<o:p></o:p></span></p><p=
class=3DMsoNormal><span lang=3DEN-US>I really enjoy working with dar, espe=
cially because of your attention to detail which makes dar very versatile t=
ool.<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p>&nbs=
p;</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Unfortunately I =
didn’t really have the time yet to test the binary delta feature befo=
re the final 2.6.0 release, but it’s just in time to make my yearly f=
ull backups with the signatures now and start testing binary deltas on prod=
uction, haha ;-)<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-=
US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Just=
to let you know, there is an issue compiling dar 2.6 in RHEL-7 due to gcc =
4.8.5:<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p>&n=
bsp;</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>In file includ=
ed from i_archive.hpp:35:0,<o:p></o:p></span></p><p class=3DMsoNormal><span=
lang=3DEN-US> &=
nbsp; from archive.cpp:31:<o:p></o:p></span><=
/p><p class=3DMsoNormal><span lang=3DEN-US>erreurs.hpp:110:15: error: funct=
ion 'libdar::Egeneric::niveau& libdar::Egeneric::niveau::operator=3D(li=
bdar::Egeneric::niveau&&)' defaulted on its first declaration with =
an exception-specification that differs from the implicit declaration 'libd=
ar::Egeneric::niveau& libdar::Egeneric::niveau::operator=3D(libdar::Ege=
neric::niveau&&)'<o:p></o:p></span></p><p class=3DMsoNormal><span l=
ang=3DEN-US> niveau & operator =3D (nivea=
u && ref) noexcept =3D default;<o:p></o:p></span></p><p class=3DMso=
Normal><span lang=3DEN-US>…<o:p></o:p></span></p><p class=3DMsoNormal=
><span lang=3DEN-US>(^ and so on, basically all the prototypes with noexcep=
t =3D default)<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US=
><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>I unde=
rstand that gcc 4.8.5 has a very limited (buggy) support for c++11, but unf=
ortunately I guess this means dar 2.6 will not be available in EPEL-7.<o:p>=
</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>It should probably=
also be stated in the documentation that gcc 4.8 is no longer supported (i=
f this was the intention).<o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3D=
EN-US>I was able to compile it with devtoolset-7 (gcc 7.3.1), create a RPM =
(along with statically linked libthreadar) and the resulting RPM seems to w=
ork fine for now on vanilla Centos 7 (without the newer gcc and libs instal=
led).<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p>&nb=
sp;</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Looking forward=
, I see that dar is using RS_DEFAULT_BLOCK_LEN which is just 2048 bytes. If=
the signatures will be switched to Blake2, this means the signature will b=
e 36 bytes (256-bit blake2 + 4 bytes crc32) for each 2k block. This might b=
e a bit too much for larger files and I was thinking that instead of having=
(yet another) command line option for that, dar might try to optimize the =
block_len based on file size. I don’t see the need for 2k signatures =
on multi-gigabyte files, where 8k or 16k signature might be just ok and a l=
ot smaller. I have a proof of concept patch for that already, for instance =
with the 2GB file the blake2 catalog size is reduced from 36MB to 9MB with =
8k blocks (default dar catalog with md4 signature would be around 18MB for =
the same file). I can send the patch to github if needed ;-)<o:p></o:p></sp=
an></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p>=
<p class=3DMsoNormal><span lang=3DEN-US>And to nitpick a bit, the man page =
is not mentioning [Delta] in the [data] displayed fields. :-)<o:p></o:p></s=
pan></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p=
><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p><p cla=
ss=3DMsoNormal><span lang=3DEN-US>Thanks again and Happy Holidays!<o:p></o:=
p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></spa=
n></p><p class=3DMsoNormal><span lang=3DEN-US>Regards,<o:p></o:p></span></p=
><p class=3DMsoNormal><span lang=3DEN-US>gulikoza<o:p></o:p></span></p></di=
v></body></html>=
------=_NextPart_000_0001_01D49C98.6FF07A60--
--===============7227505164832671095==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--===============7227505164832671095==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Dar-discussions mailing list
Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/dar-discussions
--===============7227505164832671095==--