Re: Suggestion: delta-exclusion = compress-exclusion

On Wed, 12 May 2021 20:33:53 +0200
Denis Corbin <[email protected]> wrote:

> On 11/05/2021 23:14, imschmeg wrote:
> > It seems very likely to me that all of the masks used for
> > compress-exclusion should also be passed in with
> > --exclude-delta-sig as well.  It might be useful to have a single
> > option that makes this happen.  I'll add a delta-exclusion option
> > to my darrc.  
> 
> I don't think that all files that we usually exclude from compression
> cannot leverage binary delta.

Yes, not a 100% guarantee.  But that's true of compression itself -
some of those files in the compress-exclusion set will be compressible.

I think of delta-exclusion (and compress-exclusion) as a guide -
one can always add back in particular files where the guide does not
apply.  But most of the time, the guide works well.

> 
> If you have too much exclusion masks for either compression or binary
> delta, note that you can rely on DCF file, where you can copy an past
> the possibly long list of masks twice, searching/replacing the -Z by
> -} in the second copy and feeding the resulting file to dar thanks to
> -B option.

I put them in my darrc file, duplicating all of the -Z lines to -}

> 
> > 
> > It might also be useful to exclude read-only files from deltas, but
> > I don't see how to do that easily.
> >   
> 
> I don't understand the point: Dar does not modify files under backup,
> thus read-only files can benefit from binary delta as well as
> read-write files.

In my experience, a file with read-only permission will either never be
altered, so that the initial calculation of delta sigs is wasted.  Or
it will only very rarely be altered, so that the amortized savings from
delta compression will be small across many backups.  Of course, there
are many other reasons to give a file read-only permission, including
merely making it hard to delete or accidentally modify.

However, in most cases, the largest of these files have extensions that
would fall under the delta-exclusion masks (for instance, .iso files).

In my particular case, I follow a full backup with a sequence of
incrementals, all with extracted catalogs. This means that the delta
signatures get duplicated many times.  Meaning that the cost of delta
signatures becomes significant for very large files that either never
change or change only very rarely.

> 
> But anyway, if you want to do some fancy things while dar does not
> provide the exact way to exclude/include what you want, there is a
> simple solution:
> 
> use the find command to select files and feed its output to dar by
> mean of --include-from-file or --exclude-from-file
> 
> In your case:
> 
> find / -perm -002 -o -perm -020 -o -perm -0200 \
> 	-o -print > listing.txt
> dar --exclude-from-file listing.txt ... other options...

Thanks for that idea!  I like this better than the EA-tagging idea we
discussed with sockets.  There is a TOCTOU race possibility with using
find prior to dar, but not very likely in my case because I'm doing the
backups at logout.