Re: diff compression

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/04/2015 13:48, J. Roeleveld wrote:
> On Thursday, April 30, 2015 12:10:18 PM gulikoza wrote:
>>> From: J. Roeleveld Sent: Thursday, April 30, 2015 8:59 AM

[...]

>> 
>> 1. checksums would be optional and dependent on file size 2.
>> checksum for a 8GB file is about 20MB which is negligible
>> compared to the data it can save
> 
> What are the chances for checksums being identical for 2 different
> versions?

even if the chance are very low it is not null and I would not like to
have my data not saved while it should or wrongly delta-restored when
it should not, due to checksum collision. If the probability of
checksum collision is 'n' the probability that a collision occurs in
an archive is rawly equivalent to n*m where m is the number of
checksums [OK for clarity, the probability of 'A or A' is "2n - n^2"
where n is the probability of A and both events A are independent but
have the same probability. As n<1, n^2 << 2n, thus P(A or A) ~ 2n and
thus P(A ou A ou... A) ~ n*m]. Considering all files a backup
contains, the amount of backup done daily with dar in the word, if
such collision succeeds once in the following years, it would be
possible to say that dar does not work properly and that this is by
design... So yes, if there is such checksum based comparison in
libdar, this will not be activated by default and if the user
activates it a disclamer/warning will show.

> 
>>> Binary diffs?
>> 
>> See https://github.com/librsync/librsync
>> 
>>> Which means more memory usage and more CPU required. I already
>>> have 1 filesystem that can't be backed up because that server 
>>> has insufficient resources to let DAR finish.
>> 
>> CPU is cheap, I/O is not. Storing only delta saves I/O and
>> (possibly) network bandwidth. Computing additional block
>> checksum, when the file is already being read is cheap.
> 
> My backups are handled by the fileserver. It has more then
> sufficient bandwidth on disk I/O and network. The CPU and memory is
> sufficient for managing the filesystems and most of the backup.
> There is no room for expansion to accomodate it.
> 
> I am using this as an example why the comment above depends on the
>  environment.

Right, this will be the user choice depending on its most restrictive
constraint to select or not delta compression, whether he/she is
richer in CPU resource, network bandwidth or storage space. :)

> 
>> I am aware however of DAR RAM requirement. That's why the
>> signature needs to be dumped as soon as it's computed and not
>> stored in the catalog. A 50 - 100MB more RAM for a multi GB file
>> is again negligible, a pagecache for the file will consume much
>> more than that. And again, it should be optional to produce
>> checksums at all...
> 
> In order to compare the signature, it needs to be added to the
> catalog. I use an extracted catalog for incrementals to avoid
> having to store all the backups online. The extra data for these
> checksums will need to be in the catalog.
> 
> I currently have 1936 catalogs active, which is a total of 7.1GB of
> storage. How much would that increase?

It would increase depending on the block size you would select, the
smaller the more blocks, the more signatures, the bigger the isolated
catalogue. But no change if you do not select delta compression :-)

> 
>>>> What do you think? Is it doable for 2.5 :-)
>>> 
>>> Personally, I'd prefer it not be added.
>> 
>> Again, it should be optional. You have the option of using it or
>> not (if it does not provide any advantage...)
> 
> If it can be disabled reliably, I'd have no issues with it.

You won't have to disable it, you will have to enable it :) for the
reason of checksum collision risk I explained above.

> 
>>> With what you propose, merging incrementals will require a full
>>> restore and consequently a full backup.
>> 
>> Not necessarily...a merge operation can simply store 2 deltas for
>> the file consecutively. Not the most efficient, but doable. And
>> I'd be perfectly happy, if dar refuses to merge such archives at
>> first. Any additional functionality can be added later...
> 
> I'm curious about how Denis feels about this?
> 
> -- Joost
> 

Cheers,
Denis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBVUUF4AgxsL0D2LGCAQJfZA/+PCeHRWh+e77JH++ece9a+6Z0aBYaMxkU
6uTczhe5hL3rYcJt0LD/VmNXgB9DFO1fkFnnIVtOnUQCziNp1O5z/YL+3mpR/Q5I
jW4FQYx1ONL0j+88g0ALxEWsaxwjTcysVefEhBy6n4O7bNUi+Gp3i7Hp5KoBvhkw
irCW0tRLW5d4a9EmxtESjJ/bXbltvYvBsRz/Ddy6RcVCMOwBKYm1TqAessTk/lbt
/6N7udCr8L10C7Ai1Cc9V5Onv3NHLG0udYh9hJDWxOBtOZxWujFtYJ3M1Vo9KJ+c
YU/pO+KMvH4qs/BeQgl3RL1BJomPikS7jmvaJz/4qnA5Bre1+4hRRm1QgBWhfp5U
lQCb0vYKXL7aaofxwk4BCnsq3MT/sSambfRmBkJA80PF39q++30VlydBHijN8n6d
nSxP9Vceq44AMkUXNv+z+q0SouJF5PJuD9iBABdP4KHfj5x7r9ioUyw0IWusbin4
ErpgfTkxFaFClcrR+pJlba2tElzBNqM+osVnZfFO43hBL4ZbiSWqfVbwZNXL5KeK
c9NbTQqUTHyh+gSOzYEDP2LqZOTbftPvZwY3on9h39khj3rzKXLLXY2AENlTw+pj
FaAwWRIyYjGJLAm43bdTuoWYkMAlCJsHJ1A2onr901gm/1F/wJnWJ7ejvaL3Ic5T
D7QFtTWvSoI=
=irM8
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y