Re: diff compression

On Thursday, April 30, 2015 08:09:50 AM gulikoza wrote:
> Hello,
> 
> I love dar, but it has become increasingly difficult to manage backups with
> it as filesize grows.
> Doing an incremental backup of a moderately busy imap server will create the
> same size backup as a full backup although in reality, probably not that
> much has changed.

Which IMAP server do you use?

With Cyrus and regular DAR incrementals I get the following sizes:
Full backup = 24.8 GB
Total for all 22 incrementals = 13.36 GB

That is, on average, less than 1 GB / incremental.

> I have read the existing discussions about xdelta usage (and sourceforge
> req. 138 and 122) and understand some of the concerns.
> I have done some research last year with librsync
> (https://github.com/librsync/librsync) and modified it a bit to create delta
> backups of PST Outlook files on my Windows machines. I believe the same
> concept could be used with dar, with a rather simple extension.

PST files are a horrible idea. 1 bit broken and the whole file ends up being 
corrupted.

> A new layer 'delta' compression would be added. When creating archives, dar
> would also store signatures (block checksums) of the files sufficiently
> large (new option: min-size to store signature along with the file). Now,
> the exact location of the signatures would need to be determined. They are
> too big to store in the catalog, but there would be a need to create an
> isolated 'catalog + signatures' dar from the full archive. Perhaps they
> could be escaped and stored after the file data, much like the escape
> catalog, along with the sig offset stored in the catalog?

Those checksums will increase the size of the archive files and you end up with 
either the same size increase you are trying to avoid, or a bigger chance that 
the checksums will not always be unique.

> So when dar is creating a backup, it stores file data (or the diff already!)
> in the backup and is creating the signature in RAM. After the file data,
> the signature is dumped into dar.

Binary diffs?

> Doing an incremental backup, dar reads signatures from the reference
> archive, computes and stores the delta (instead of full data) along with the
> signature of the file currently on the filesystem. No incremental backup
> needs original file data, only the signature stored in the reference
> archive. If the signature is not present, it stores full data the same as
> now ('null' delta compression :-)).

Which means more memory usage and more CPU required.
I already have 1 filesystem that can't be backed up because that server has 
insufficient resources to let DAR finish.

> What do you think? Is it doable for 2.5 :-)

Personally, I'd prefer it not be added.
With what you propose, merging incrementals will require a full restore and 
consequently a full backup.

--
Joost

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y