Re: diff compression

> From: J. Roeleveld
> Sent: Thursday, April 30, 2015 8:59 AM
>
>Which IMAP server do you use?
>
> With Cyrus and regular DAR incrementals I get the following sizes:
> Full backup = 24.8 GB
> Total for all 22 incrementals = 13.36 GB
>
> That is, on average, less than 1 GB / incremental.

I mostly use Dovecot with mbox files. Sure, switching to Maildir would make
for better incremental backups, but this is not just mbox vs. Maildir
discussion...the IMAP server was just an example of a real world situation.

> PST files are a horrible idea. 1 bit broken and the whole file ends up
being 
> corrupted.

And you would recommend to Microsoft Outlook users that they switch their
mail archive for the past 20 years to...?
Exactly why an incremental delta backup is efficient (both in space and time
to create it). Believe it or not, my librsync solution will backup 10GB+ PST
faster than copying it directly over 1Gb network... My current (compressed)
delta archive takes less space than one single uncompressed file...

> Those checksums will increase the size of the archive files and you end up
with 
> either the same size increase you are trying to avoid, or a bigger chance
that 
> the checksums will not always be unique.

1. checksums would be optional and dependent on file size
2. checksum for a 8GB file is about 20MB which is negligible compared to the
data it can save

> Binary diffs?

See https://github.com/librsync/librsync

> Which means more memory usage and more CPU required.
> I already have 1 filesystem that can't be backed up because that server
has 
> insufficient resources to let DAR finish.

CPU is cheap, I/O is not. Storing only delta saves I/O and (possibly)
network bandwidth.
Computing additional block checksum, when the file is already being read is
cheap.

I am aware however of DAR RAM requirement. That's why the signature needs to
be dumped as soon as it's computed and not stored in the catalog.
A 50 - 100MB more RAM for a multi GB file is again negligible, a pagecache
for the file will consume much more than that. And again, it should be
optional to produce checksums at all...

>> What do you think? Is it doable for 2.5 :-)
>
> Personally, I'd prefer it not be added.

Again, it should be optional. You have the option of using it or not (if it
does not provide any advantage...)

> With what you propose, merging incrementals will require a full restore
and 
> consequently a full backup.

Not necessarily...a merge operation can simply store 2 deltas for the file
consecutively. Not the most efficient, but doable.
And I'd be perfectly happy, if dar refuses to merge such archives at first.
Any additional functionality can be added later...

Regards,
gulikoza


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y