diff compression

Hello,

I love dar, but it has become increasingly difficult to manage backups with
it as filesize grows.
Doing an incremental backup of a moderately busy imap server will create the
same size backup as a full backup although in reality, probably not that
much has changed.

I have read the existing discussions about xdelta usage (and sourceforge
req. 138 and 122) and understand some of the concerns.
I have done some research last year with librsync
(https://github.com/librsync/librsync) and modified it a bit to create delta
backups of PST Outlook files on my Windows machines. I believe the same
concept could be used with dar, with a rather simple extension.

A new layer 'delta' compression would be added. When creating archives, dar
would also store signatures (block checksums) of the files sufficiently
large (new option: min-size to store signature along with the file). Now,
the exact location of the signatures would need to be determined. They are
too big to store in the catalog, but there would be a need to create an
isolated 'catalog + signatures' dar from the full archive. Perhaps they
could be escaped and stored after the file data, much like the escape
catalog, along with the sig offset stored in the catalog?

So when dar is creating a backup, it stores file data (or the diff already!)
in the backup and is creating the signature in RAM. After the file data, the
signature is dumped into dar.
Doing an incremental backup, dar reads signatures from the reference
archive, computes and stores the delta (instead of full data) along with the
signature of the file currently on the filesystem. No incremental backup
needs original file data, only the signature stored in the reference
archive. If the signature is not present, it stores full data the same as
now ('null' delta compression :-)).

When extracting, if the catalog data indicates 'delta' compression, dar
renames the original file present on the filesystem and begins building the
extracted file from the renamed file and stored delta (librsync requires
random access to 'reference file', so it cannot be stored inside reference
dar anyway). When finished, the crc is compared to the stored crc. If it
doesn't match (either corrupted archive of wrong file present on the
filesystem), inform the user, delete the 'garbage created' and rename back
the original file that was present on the system. I don't see much need to
further complicate this process. Delta compression would not be turned on by
default and restoring an incremental backup already requires some knowledge
from the user about the current state of filesystem and which incremental
archive to start restoring (because of deleted files...). 'I'm sorry, but
crc error has occurred while restoring your file, because you have used
delta compression and not provided the right reference file' seems perfectly
ok by me :-)

I do my backups mostly in 3 layers. Full yearly backup is stored on some
safe location (NAS + tape) and at least one copy moved to an off-site
location.
Then I do weekly or 14-day incremental backups. These would not need to be
delta compressed. Each day, an incremental backup is made that references
the last weekly backup. These backups I would like to have delta compressed.
This way, a mailbox could be restored with the last weekly backup + last
daily backup delta and this is precisely what I usually do for files that do
not change as often as a mailbox (there is a very high probability that some
file that is currently in use will be present in the last weekly backup as
it has probably changed since beginning of the year...). This reduces backup
storage requirement significantly (only 1 archive in 14 days stores full
file data, instead of each daily backup).

What do you think? Is it doable for 2.5 :-)

Regards,
gulikoza


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y