Re: diff compression
Denis Corbin <[email protected]> Fri, 15 Jan 2016 21:01:45 +0100
| Newsgroups | gmane.comp.sysutils.backup.dar.general |
|---|---|
| Message-ID | <[email protected]> |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 11/01/2016 20:24, gulikoza wrote:
> Hello,
Hello,
>
> I've been diagnosing why I got differential dar backup almost the
> same size as full after updating dar from 2.3 to 2.4 (yes I
> know...). I've come up with some discussion about dar&hard links on
> cifs share, which was my setup as well
> (http://sourceforge.net/p/dar/mailman/message/26577785).
>
> Now (this is a bit off-topic, but bear with me for a bit more), I
> figured it out the problem is because cifs share will produce files
> with hard link count = 0, which was apparently fixed in later
> kernels (see samba bug report here:
> https://bugzilla.samba.org/show_bug.cgi?id=9346, which makes me
> wonder, should there be some way to handle this?). As hard link
> count changed after I have upgraded entire system from Centos6 to
> Centos7 (relevant kernel commit:
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/cif
>
>
s?id=74d290da476f672ad756634d12aa707375d3564d), dar produced a differential
> backup of almost the same size as full backup and is probably not
> related to dar version change...
>
> Back on topic...suppose a copy of a directory (that's included in
> daily backup) is made. Old directory probably has files that have
> already been stored in some previous full backup...A worrying admin
> might run 'hardlink' over both old&new directories to save both
> server space and backup size. But - every inode is now changed in
> the old directory as well because hard link count has increased.
> Dar will store every file in a new data structure that stores hard
> links (if a differential backup is made). No backup space is
> actually saved (1 copy of file data is stored in this new diff
> backup in either case, even if hardlink was not run)...
I did a test with dar 2.5.3:
STEP 1: doing a full backup
============================
### >dar -l full
[Data ][ EA ][FSA][Compr][S]| Permission | User | Group | Size |
Date | filename
-
-----------------------------+------------+-------+-------+-------+-------------------------------+------------
[Saved] [-L-][ 1%][ ] drwxr-xr-x denis maison 952
kio Fri Jan 15 20:29:17 2016 DIR1
[Saved] [-L-][ 1%][X] -rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR1/bash
[Saved] [-L-][ 1%][ ] drwxr-xr-x denis maison 1 Mio
Fri Jan 15 20:29:19 2016 DIR2
[Saved] [-L-][ 1%][ ] drwxr-xr-x denis maison 952
kio Fri Jan 15 20:29:21 2016 DIR2/SUB
[Saved] [-L-][ 1%][X] -rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR2/SUB/bash
[Saved] [-L-][ 1%][X] -rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR2/bash
as seen above the three bash files are copies of the same file (there
is no star '*' before the permission, these are not hard linked inodes).
STEP 2: running hardlink(1)
=========
STEP 3: differential backup #1
=========
after having run hardlink(1), now performing a differential backup
based on 'full.1.dar':
### >dar -l diff
[Data ][ EA ][FSA][Compr][S]| Permission | User | Group | Size |
Date | filename
-
-----------------------------+------------+-------+-------+-------+-------------------------------+------------
[ ] [-l-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:17 2016 DIR1
[ ] [-L-][-----][ ] *-rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR1/bash
[Saved] [-L-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:44 2016 DIR2
[Saved] [-L-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:44 2016 DIR2/SUB
[ ] [-L-][-----][ ] *-rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR2/SUB/bash
[ ] [-L-][-----][ ] *-rwxr-xr-x denis maison 952
kio Wed Nov 19 20:27:30 2014 DIR2/bash
None of the bash file got saved but only the directories where
hardlink(1) has modified the filesystem are [Saved]. Here dar has
recorded the hard linked nature of the "bash" files (see the *) but
did not saved it as it had not changed since the full backup.
STEP 4: modifying the inode
========
Last, now I manually modify the bash binary (changing a byte value
using vbindiff), then doing a new differential backup:
STEP 5 : differential backup #2
========
### >dar -l diff2
[Data ][ EA ][FSA][Compr][S]| Permission | User | Group | Size |
Date | filename
-
-----------------------------+------------+-------+-------+-------+-------------------------------+------------
[ ] [-l-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:17 2016 DIR1
[Saved] [-L-][ 1%][X] *-rwxr-xr-x denis maison 952
kio Fri Jan 15 20:33:59 2016 DIR1/bash
[ ] [-l-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:44 2016 DIR2
[ ] [-l-][ ][ ] drwxr-xr-x denis maison 0
Fri Jan 15 20:29:44 2016 DIR2/SUB
[Saved] [-L-][ 1%][X] *-rwxr-xr-x denis maison 952
kio Fri Jan 15 20:33:59 2016 DIR2/SUB/bash
[Saved] [-L-][ 1%][X] *-rwxr-xr-x denis maison 952
kio Fri Jan 15 20:33:59 2016 DIR2/bash
As expected dar saved the modified inode, and as it is linked three
times to the filesystem(hard link), the three "bash" files are flagged
as Saved, but the archive only contains one copy of the bash file:
### >la diff2.1.dar A/DIR1/bash
- -rwxr-xr-x 3 denis maison 975488 janv. 15 20:33 A/DIR1/bash
- -rw-r--r-- 1 denis maison 957995 janv. 15 20:44 diff2.1.dar
Yes, even without compression, the resulting archive is smaller than a
single copy of bash! This is due to the sparse file detection [X] as
the bash binary seems to be a sparse file (1% of its length could be
suppressed) large enough to compensate the archive overhead
(catalogue, slice header, archive header, CRC, tape marks, etc.)
### >dar -l diff2 -q
Archive version format : 09
Compression algorithm used : none
Symmetric key encryption used : none
Asymmetric key encryption used : none
Archive is signed : no
Sequential reading marks : present
Catalogue size in archive : 373 bytes
User comment : N/A
Archive is composed of 1 file(s)
File size: 957995 bytes
The global data compression ratio is:
CATALOGUE CONTENTS :
total number of inode : 4
saved inode : 1
distribution of inode(s)
- directories : 3
- plain files : 1
- symbolic links : 0
- named pipes : 0
- unix sockets : 0
- character devices : 0
- block devices : 0
- Door entries : 0
hard links information
- number of inode with hard link : 1
- number of reference to hard linked inodes: 3
destroyed entries information
0 file(s) have been record as destroyed since backup of reference
Here above we can see the archive contains 3 directories and a single
plain file (where from 4 inodes) this plain file is hardlinked three
times.
>
> Will delta compression take this into account?
this will not change the behavior that define which file to save or
not, but it will change how it is saved.
> Inode type has changed from normal file to hard linked file, but
> data is the same - so ideally, it should only store a very small
> delta?
inode type has not changed, this is still a plain file for dar, hard
linked or not this is the same for dar. The fact an inode is
hardlinked will only avoid dar saving it again if it has already been
met in another directory. At restoration time dar will rather build a
hard link to the existing or already restored inode than restoring a
copy of the inode (unless cross filesystem restoration is met in which
case dar issues a warning and duplicate the inode).
> Have you considered in the algorithm how delta is stored if inode
> changes type?
only plain file are subject to delta compression, if inode type change
there is no point to consider delta compression.
>
> Regards, gulikoza
>
>
>
Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIVAwUBVplQKQgxsL0D2LGCAQIdcRAAxU+Df71TOtttnjwVmS2rmnWqb+mBEARu
3PQIJDQ1fgkcT+WOSgNwhoA6pjo/2V9cTXs7rkU/hAs0xgBPoZdbTZCOrOonFHcn
+jjNktX+i258DRXotzMGcuaJoVYqE91ferJbWL+SXLNiPv3Q4Cs/BstEqOqL0TzL
fXwhveVvhw/vGHWqAqXRT/D2vMqZFcR5CImwM2LdnUZaFDtrjI8W84XNgTCD/Bht
CPQMpCo4E6y0sVsfCs3iLZLJ2/OqhCqNrLxhfw8kkAcXNO6hhyyUHf9LnY+JlY8R
uGlzFDYdZR0qMQcNB20C/6WFKzgWVWtAiQKlvv0zOy6qMjditNbyB71cxxwtaZ6z
vbqJImy+1QwWLKtO6Ps0CLij4WsO0g1e4kZ7lf/BFQQsm28r86qUzMFSseZJ5xwn
aH+tp4Zrc1hiHZW6h/QAY9c3Y3xeBguehi4QZzfdmnTzHy55T3+d89V4Vwk/qk50
/92CSBvCS5MUUS1R3qWRwJrj0EqF3gvtSdIy5fmlEulw/1rRWtZ0ONAEAJxhWHkn
yNDS8Fb9VgAarICB8Qo7t5I0xFCcY0LcnuJM4+uienD2wz9+8stRQBx3U9b7frFE
GPrc1LH1Fz8QiQ4S6pmmSjas4xEu7Gianb/QWl4Cpy+n9t35KEAE7yzFTrMuQ6Bk
JnEa1K2UC+U=
=DyRq
-----END PGP SIGNATURE-----
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140