Re: diff compression

> From: J. Roeleveld
> Sent: Thursday, April 30, 2015 1:48 PM
>
> How big do those files become? And how certain are you that there is no 
> corruption in them?

One particular mail server has 132 files in /var/spool/mail totaling about
80 GB.
The users themselves have additional 70 GB of files in their mail
directories in home.

The combined incremental backup of just these two is 50 - 60 GB per day!
So far, there hasn't been any corruption problems on this setup.

>> 1. checksums would be optional and dependent on file size
>> 2. checksum for a 8GB file is about 20MB which is negligible compared to
the
>> data it can save
>
> What are the chances for checksums being identical for 2 different
versions?

Identical in what sense? Librsync uses rolling crc for quick identification
and a strong checksum (used to be md4, now they have switched to 256-bit
BLAKE2 hash AFAIK) for positive identification. I'd say that for 2
*different* blocks of a few hundred KB to a few MB (let's say 2 or 4MB block
size for really large files) to have identical both the weak crc and strong
256-bit hash is quite insignificant. Dar before 2.4 used a 32-bit crc for an
entire file, not just a small block...

> My backups are handled by the fileserver.
> It has more then sufficient bandwidth on disk I/O and network.
> The CPU and memory is sufficient for managing the filesystems and most of
the 
> backup. There is no room for expansion to accomodate it.
> 
> I am using this as an example why the comment above depends on the 
> environment.

DAR spends most of the time compressing data. If it only needs to compress a
smaller delta, that's a win?
If all the files are small, nothing extra is done.

> In order to compare the signature, it needs to be added to the catalog.
> I use an extracted catalog for incrementals to avoid having to store all
the 
> backups online.
> The extra data for these checksums will need to be in the catalog.
> 
> I currently have 1936 catalogs active, which is a total of 7.1GB of
storage.
> How much would that increase?

Depends on how much large files you have, that you need signatures to have
stored for.
If you only have maildir e-mails, you won't store any checksums because
either the entire file changes or it doesn't and any delta doesn't help.
That's why there would be a command line option for some threshold, when to
start storing checksums (for instance, only files > 100 MB).
If the catalogs contain only a few large files, then some additional space
for the checksums in the catalog would be much less than an entire file
stored in the incremental backups....

As I've said, it wouldn't be practical to store checksums in the catalog.
There would need to be an intermediate format of an isolated catalog +
checksums (without the data). When/if they're no longer needed, dar should
be able to transform such an archive to a purely isolated catalog.

gulikoza


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y