Re: diff compression

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi guys,

the delta diff feature is now quite stable and will be part of release
2.6.0 near the end of this year if all goes as expected (I use it
since last august for my own use and testing)

In the meanwhile if you want to play with it, have a look at the
branch "delta-diff" under GIT. This branch receives all fixes from
2.5.x and 2.4.x releases while it is not modified by new features
(developed in the master branch)

Note however that due to other new features done after delta diff
there is no warranty the generated archives will be readable by future
releases, including 2.6.0. Thus it is fine playing with this code for
backup but not for long term archiving. And just in case, keep a copy
of the source code you have been using to create your delta-diff backups
.

Last, you will need librsync 0.9.7. As reported by Gulikoza libdar
source code does not compile against more recent version of librsync.
I will fix that before 2.6.0

Regards,
Denis

Le 15/01/2016 =E0 21:01, Denis Corbin a =E9crit :
> On 11/01/2016 20:24, gulikoza wrote:
>> Hello,
> =

> Hello,
> =

> =

>> I've been diagnosing why I got differential dar backup almost
>> the same size as full after updating dar from 2.3 to 2.4 (yes I =

>> know...). I've come up with some discussion about dar&hard links
>> on cifs share, which was my setup as well =

>> (http://sourceforge.net/p/dar/mailman/message/26577785).
> =

>> Now (this is a bit off-topic, but bear with me for a bit more),
>> I figured it out the problem is because cifs share will produce
>> files with hard link count =3D 0, which was apparently fixed in
>> later kernels (see samba bug report here: =

>> https://bugzilla.samba.org/show_bug.cgi?id=3D9346, which makes me =

>> wonder, should there be some way to handle this?). As hard link =

>> count changed after I have upgraded entire system from Centos6
>> to Centos7 (relevant kernel commit: =

>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit
/fs/cif
>
>> =

> =

> s?id=3D74d290da476f672ad756634d12aa707375d3564d), dar produced a
> differential
>> backup of almost the same size as full backup and is probably
>> not related to dar version change...
> =

>> Back on topic...suppose a copy of a directory (that's included
>> in daily backup) is made. Old directory probably has files that
>> have already been stored in some previous full backup...A
>> worrying admin might run 'hardlink' over both old&new directories
>> to save both server space and backup size. But - every inode is
>> now changed in the old directory as well because hard link count
>> has increased. Dar will store every file in a new data structure
>> that stores hard links (if a differential backup is made). No
>> backup space is actually saved (1 copy of file data is stored in
>> this new diff backup in either case, even if hardlink was not
>> run)...
> =

> =

> I did a test with dar 2.5.3:
> =

> STEP 1: doing a full backup =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =

> ### >dar -l full [Data ][ EA  ][FSA][Compr][S]| Permission | User
> | Group | Size  | Date                 |    filename - =

> -----------------------------+------------+-------+-------+-------+---
- ----------------------------+------------
>
> =

[Saved]       [-L-][   1%][ ]  drwxr-xr-x   denis       maison  952
> kio Fri Jan 15 20:29:17 2016        DIR1 [Saved]       [-L-][
> 1%][X]  -rwxr-xr-x   denis       maison  952 kio Wed Nov 19
> 20:27:30 2014        DIR1/bash [Saved]       [-L-][   1%][ ]
> drwxr-xr-x   denis       maison  1 Mio Fri Jan 15 20:29:19 2016
> DIR2 [Saved]       [-L-][   1%][ ]  drwxr-xr-x   denis       maison
> 952 kio Fri Jan 15 20:29:21 2016        DIR2/SUB [Saved]
> [-L-][   1%][X]  -rwxr-xr-x   denis       maison  952 kio Wed Nov
> 19 20:27:30 2014        DIR2/SUB/bash [Saved]       [-L-][   1%][X]
> -rwxr-xr-x   denis       maison  952 kio Wed Nov 19 20:27:30 2014
> DIR2/bash
> =

> =

> as seen above the three bash files are copies of the same file
> (there is no star '*' before the permission, these are not hard
> linked inodes).
> =

> STEP 2: running hardlink(1) =3D=3D=3D=3D=3D=3D=3D=3D=3D
> =

> STEP 3: differential backup #1 =3D=3D=3D=3D=3D=3D=3D=3D=3D after having r=
un
> hardlink(1), now performing a differential backup based on
> 'full.1.dar':
> =

> ### >dar -l diff [Data ][ EA  ][FSA][Compr][S]| Permission | User
> | Group | Size  | Date                 |    filename - =

> -----------------------------+------------+-------+-------+-------+---
- ----------------------------+------------
>
> =

[     ]       [-l-][     ][ ]  drwxr-xr-x   denis       maison  0
> Fri Jan 15 20:29:17 2016        DIR1 [     ]       [-L-][-----][ ]
> *-rwxr-xr-x   denis       maison  952 kio Wed Nov 19 20:27:30 2014
> DIR1/bash [Saved]       [-L-][     ][ ]  drwxr-xr-x   denis
> maison  0 Fri Jan 15 20:29:44 2016        DIR2 [Saved]       [-L-][
> ][ ]  drwxr-xr-x   denis       maison  0 Fri Jan 15 20:29:44 2016
> DIR2/SUB [     ]       [-L-][-----][ ] *-rwxr-xr-x   denis
> maison  952 kio Wed Nov 19 20:27:30 2014        DIR2/SUB/bash [
> ]       [-L-][-----][ ] *-rwxr-xr-x   denis       maison  952 kio
> Wed Nov 19 20:27:30 2014        DIR2/bash
> =

> None of the bash file got saved but only the directories where =

> hardlink(1) has modified the filesystem are [Saved]. Here dar has =

> recorded the hard linked nature of the "bash" files (see the *)
> but did not saved it as it had not changed since the full backup.
> =

> STEP 4: modifying the inode =3D=3D=3D=3D=3D=3D=3D=3D
> =

> Last, now I manually modify the bash binary (changing a byte value =

> using vbindiff), then doing a new differential backup:
> =

> STEP 5 : differential backup #2 =3D=3D=3D=3D=3D=3D=3D=3D
> =

> ### >dar -l diff2 [Data ][ EA  ][FSA][Compr][S]| Permission | User
> | Group | Size  | Date                 |    filename - =

> -----------------------------+------------+-------+-------+-------+---
- ----------------------------+------------
>
> =

[     ]       [-l-][     ][ ]  drwxr-xr-x   denis       maison  0
> Fri Jan 15 20:29:17 2016        DIR1 [Saved]       [-L-][   1%][X]
> *-rwxr-xr-x   denis       maison  952 kio Fri Jan 15 20:33:59 2016
> DIR1/bash [     ]       [-l-][     ][ ]  drwxr-xr-x   denis
> maison  0 Fri Jan 15 20:29:44 2016        DIR2 [     ]       [-l-][
> ][ ]  drwxr-xr-x   denis       maison  0 Fri Jan 15 20:29:44 2016
> DIR2/SUB [Saved]       [-L-][   1%][X] *-rwxr-xr-x   denis
> maison  952 kio Fri Jan 15 20:33:59 2016        DIR2/SUB/bash =

> [Saved]       [-L-][   1%][X] *-rwxr-xr-x   denis       maison
> 952 kio Fri Jan 15 20:33:59 2016        DIR2/bash
> =

> As expected dar saved the modified inode, and as it is linked
> three times to the filesystem(hard link), the three "bash" files
> are flagged as Saved, but the archive only contains one copy of the
> bash file:
> =

> ### >la diff2.1.dar A/DIR1/bash -rwxr-xr-x 3 denis maison 975488
> janv. 15 20:33 A/DIR1/bash -rw-r--r-- 1 denis maison 957995 janv.
> 15 20:44 diff2.1.dar
> =

> Yes, even without compression, the resulting archive is smaller
> than a single copy of bash! This is due to the sparse file
> detection [X] as the bash binary seems to be a sparse file (1% of
> its length could be suppressed) large enough to compensate the
> archive overhead (catalogue, slice header, archive header, CRC,
> tape marks, etc.)
> =

> ### >dar -l diff2 -q Archive version format               : 09 =

> Compression algorithm used           : none Symmetric key
> encryption used        : none Asymmetric key encryption used
> : none Archive is signed                    : no Sequential reading
> marks             : present Catalogue size in archive            :
> 373 bytes User comment                         : N/A
> =

> Archive is composed of 1 file(s) File size: 957995 bytes The global
> data compression ratio is:
> =

> CATALOGUE CONTENTS :
> =

> total number of inode : 4 saved inode           : 1 distribution of
> inode(s) - directories        : 3 - plain files        : 1 -
> symbolic links     : 0 - named pipes        : 0 - unix sockets
> : 0 - character devices  : 0 - block devices      : 0 - Door
> entries       : 0 hard links information - number of inode with
> hard link           : 1 - number of reference to hard linked
> inodes: 3 destroyed entries information 0 file(s) have been record
> as destroyed since backup of reference
> =

> Here above we can see the archive contains 3 directories and a
> single plain file (where from 4 inodes) this plain file is
> hardlinked three times.
> =

> =

>> Will delta compression take this into account?
> =

> this will not change the behavior that define which file to save
> or not, but it will change how it is saved.
> =

>> Inode type has changed from normal file to hard linked file, but =

>> data is the same - so ideally, it should only store a very small =

>> delta?
> =

> inode type has not changed, this is still a plain file for dar,
> hard linked or not this is the same for dar. The fact an inode is =

> hardlinked will only avoid dar saving it again if it has already
> been met in another directory. At restoration time dar will rather
> build a hard link to the existing or already restored inode than
> restoring a copy of the inode (unless cross filesystem restoration
> is met in which case dar issues a warning and duplicate the
> inode).
> =

>> Have you considered in the algorithm how delta is stored if
>> inode changes type?
> =

> only plain file are subject to delta compression, if inode type
> change there is no point to consider delta compression.
> =

> =

>> Regards, gulikoza
> =

> =

> =

> =

> Cheers, Denis.
> =

> ----------------------------------------------------------------------
- --------
>
> =

Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month =

> Monitor end-to-end web transactions and take corrective actions
> now Troubleshoot faster and improve end-user experience. Signup
> Now! =

> http://pubads.g.doubleclick.net/gampad/clk?id=3D267308311&iu=3D/4140 =

> _______________________________________________ Dar-discussions
> mailing list Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org =

> https://lists.sourceforge.net/lists/listinfo/dar-discussions
> =

> =

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIVAwUBWHveSAgxsL0D2LGCAQgS/g//fUH3lzwhKx50VTxpmxOiprPAMNnS+Usz
6uo5XMiHkg9BYwLyn9J/dOKbPqEEYXZ6YF9fUlA2Tk2+07jT9xb6cgH1SQCE34dl
tf0W5cnXdJ7G7AlP3tfoCgpXJVrUn/3311dGvcn3HHxXm+nAjauyG3HUzcFw43eN
pVfaUi/Xo2FV1df78fL+asyi5VLYDDCXT1zHnpf/lCdgcez3/g0Ss0xfQ6bQ3gav
+tjZta7BcN+R673qayzHQwwAvMZ41UKNR6FD/HyAhxR4PQLiB0OhdRnsSCFpp6tt
FQ87eFBCRByfQUMnYpZVv84rk9KcskSOS3TINWOf9SiPC2/5eG6vZxNQRV3IjOHr
W9IsdnZE8uX7gL+Kk2aRMRjpGDo/PbrTHVXZ253gyIClqKMj3W4LdGHm+4en1Qw7
gfjkiOH6JqeKTouX/7obKuga9aiD9igNhSSpvDvtHUBt5hTiPGywdQUg+Y6y7wwH
FdKmD4IJ+hFZKJ8ZOTRkjW6eU/o+8LLeJCxtJFaXZI5JMZ3RjlD8VETSUx9N/6Be
i20x5Mn6DOhnpr/2dX52XkYIlPReXzTjXQBxBGBTypWUYY5O0XxUI8XJ+ccgVCNn
4blobTsHpGL3F/p6DfPNDqdG1EtAmx0OiRLMuWnvf7bew+UgrdET2yiwOQ01tYvV
q/RoDWiKgcA=3D
=3DUvh2
-----END PGP SIGNATURE-----

---------------------------------------------------------------------------=
---
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi