Re: diff compression

On Thursday, April 30, 2015 12:10:18 PM gulikoza wrote:
> > From: J. Roeleveld
> > Sent: Thursday, April 30, 2015 8:59 AM
> >
> >Which IMAP server do you use?
> >
> > With Cyrus and regular DAR incrementals I get the following sizes:
> > Full backup = 24.8 GB
> > Total for all 22 incrementals = 13.36 GB
> > 
> > That is, on average, less than 1 GB / incremental.
> 
> I mostly use Dovecot with mbox files. Sure, switching to Maildir would make
> for better incremental backups, but this is not just mbox vs. Maildir
> discussion...the IMAP server was just an example of a real world situation.

How big do those files become? And how certain are you that there is no 
corruption in them?

> > PST files are a horrible idea. 1 bit broken and the whole file ends up
> > being corrupted.
> 
> And you would recommend to Microsoft Outlook users that they switch their
> mail archive for the past 20 years to...?

Something more reliable.
I have seen far too many corruptions with PST files. Storing emails as seperate 
files (maildir) makes sense in a lot of ways.

20 years ago I was using MS Outlook. Lost far too much email in that period.
The only reason my archive dates back 20 years is because I stopped using it.

But that is off-topic.

> Exactly why an incremental delta backup is efficient (both in space and time
> to create it). Believe it or not, my librsync solution will backup 10GB+
> PST faster than copying it directly over 1Gb network... My current
> (compressed) delta archive takes less space than one single uncompressed
> file...
> > Those checksums will increase the size of the archive files and you end up
> > with either the same size increase you are trying to avoid, or a bigger
> > chance that the checksums will not always be unique.
> 
> 1. checksums would be optional and dependent on file size
> 2. checksum for a 8GB file is about 20MB which is negligible compared to the
> data it can save

What are the chances for checksums being identical for 2 different versions?

> > Binary diffs?
> 
> See https://github.com/librsync/librsync
> 
> > Which means more memory usage and more CPU required.
> > I already have 1 filesystem that can't be backed up because that server
> > has insufficient resources to let DAR finish.
> 
> CPU is cheap, I/O is not. Storing only delta saves I/O and (possibly)
> network bandwidth.
> Computing additional block checksum, when the file is already being read is
> cheap.

My backups are handled by the fileserver.
It has more then sufficient bandwidth on disk I/O and network.
The CPU and memory is sufficient for managing the filesystems and most of the 
backup. There is no room for expansion to accomodate it.

I am using this as an example why the comment above depends on the 
environment.

> I am aware however of DAR RAM requirement. That's why the signature needs to
> be dumped as soon as it's computed and not stored in the catalog.
> A 50 - 100MB more RAM for a multi GB file is again negligible, a pagecache
> for the file will consume much more than that. And again, it should be
> optional to produce checksums at all...

In order to compare the signature, it needs to be added to the catalog.
I use an extracted catalog for incrementals to avoid having to store all the 
backups online.
The extra data for these checksums will need to be in the catalog.

I currently have 1936 catalogs active, which is a total of 7.1GB of storage.
How much would that increase?

> >> What do you think? Is it doable for 2.5 :-)
> > 
> > Personally, I'd prefer it not be added.
> 
> Again, it should be optional. You have the option of using it or not (if it
> does not provide any advantage...)

If it can be disabled reliably, I'd have no issues with it.

> > With what you propose, merging incrementals will require a full restore
> > and consequently a full backup.
> 
> Not necessarily...a merge operation can simply store 2 deltas for the file
> consecutively. Not the most efficient, but doable.
> And I'd be perfectly happy, if dar refuses to merge such archives at first.
> Any additional functionality can be added later...

I'm curious about how Denis feels about this?

--
Joost

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y