Re: n00b questions about using dar for archiving wikis

On 19/07/2021 08:02, griffin tucker wrote:
> i'll try to be as comprehensive as possible so that i don't miss anything.
> 
> i want to make decremental backups of 7tb worth of wikis (text only)
> provided by wikimedia, each wiki dump in a separate dar archive

OK

> 
> wikidumps are updated each month. and from what i've read in the dar
> tutorial, if i understood it correctly, i'll have to write a script to do
> decremental backups, which shouldn't be a problem.

Well you may need a script to manage rotation and eliminate oldest
decremental but doing a decremental backup by itself is juste calling
dar with the adhoc options.

> 
> the 7tb is compressed, and uncompressed, being text, is much, much larger,
> and using btrfs transparent compression leaves a lot to be desired when it
> comes to stability (i'm starting again after losing data after a very brief
> power-loss).

There is several things I don't understand here, transparent compression
is "transparent" to the user and applications. Thus the fact btrfs
stores files compressed and make them available as if they were not
compressed does not change anything from applications standpoint and
user point of view. When a power-loss occurs, data that was read may
should not be affected less or more may it be stored compressed or not, no?

> 
> i've tried using other backup tools such as rdiff-backup, however it seems
> i would have to have all the files available uncompressed, as opposed to
> adding each file one-by-one - does dar support adding one file to the
> archive at a time, or did i misunderstand the tutorial?

here too, I don't understand why the fact if rdiff-backup and other need
data uncompressed (something to be confirmed by the way) and why this is
said to be the opposite of adding files one-by-one to the backup (to me
these are two independent things).

Maybe you mean that rdiff-backup better performs its binary delta when
it is fed with uncompressed data (which is obviously true)? But anyway,
btrfs should not provide compressed data to any application: when
rdiff-backup reads data, btrfs should provide it uncompressed
(transparently decompressing it the from storage).

At this level dar will not behave differently. The advantage of dar is
to have to possibility to compress files per files, and eventually for
some file you desire, backup them uncompressed (if they are already
compressed). Compressing per file also give better fault tolerance as a
corruption taking place in a file data will not affect other compressed
files. The drawbacks is the slightly less good compression ratio.

> 
> some of the xml files are split into groups of pages, such as the english
> wiki, because they would otherwise be very large files. sometimes a copy of
> the recombined file is provided along with the split pages files, but for
> the large english wiki meta-history file, for example, a recombined copy is
> not provided, so writing a script to recombine it shouldn't be too
> difficult (using cat and grep, etc.).

Dar has several features here that can help:
- first the filtering mechanism that gives you the possibility to ignore
recombined files and only consider the version split into groups (or the
opposite if you prefer to keep only the later and ignore the split
versions).
- the backup-hook-execute can trigger a script of your own before and
after saving any particular file you desire, providing as parameters the
filename, path, and some other interesting fields.
> 
> i want the decremental backup each month to only store the changes to the
> large files, saving a large amount of space. does dar support only backing
> up the difference of the file, as compared to simply storing a complete
> copy of the same file when they're different?

Yes, it does but you seem to make a confusion between two different
features:
- decremental/incremental/differential backup
- binary delta

For the first features:
- Decremental backup (the latest backup is a full backup and oldest
contain difference from the latest), it implies to convert the previous
full backup (the one that was before the latest you just created) into a
decremental backup, this is time/CPU/IO consuming but speeds the
recovery time as you just have to restore a single full backup (the
latest) upon disaster.
- Incremental (and its cousin the differential backup) have the full
backup as the oldest and recent backup have only files that changed
since the backup of reference (for differential this reference is the
first backup, for incremental this is the latest backup, and you can
also combine both of them over time, but I will lost you if I develop
that part).

On the other hand the binary delta relies on librsync and makes a patch
of file (text or binary) to only record the part of the file that changed.
When this feature is not activated and a file has been seen as modified
since the previous backup, it is saved as a whole in the new backup
eventually compressed by one of the many compression algorithms and
compression level dar provides (while unchanged files are not resaved at
all).

Hopefully you can combine binary delta with incremental and differential
backup. But as of today binary delta is not supported (yet?) with
decremental backup.

> 
> aside from multiple backups, is there any data redundancy built into dar,
> yet? (i'm worried about bitrot)
> 
> 
yes, there is redundancy in dar backups to detect corruption but not to
repair data. For reparing data, dar relies and integrates with Parchive
which is optimum for this type of job.