Re: n00b questions about using dar for archiving wikis

On Tue, 20 Jul 2021 at 05:18, Denis Corbin <[email protected]> wrote:
>
> On 19/07/2021 08:02, griffin tucker wrote:
> > i'll try to be as comprehensive as possible so that i don't miss anything.
> >
> > i want to make decremental backups of 7tb worth of wikis (text only)
> > provided by wikimedia, each wiki dump in a separate dar archive
>
> OK
>
> >
> > wikidumps are updated each month. and from what i've read in the dar
> > tutorial, if i understood it correctly, i'll have to write a script to do
> > decremental backups, which shouldn't be a problem.
>
> Well you may need a script to manage rotation and eliminate oldest
> decremental but doing a decremental backup by itself is juste calling
> dar with the adhoc options.

the end goal is an interactive/options bash script that downloads the
latest wiki(s) and decompresses the files one-by-one (to save disk
space) then adding each decompressed file to it's associated dar
archive, then uploading the dar archive to a cloud service (or
moving/rearranging a folder monitored by a cloud upload service)

> >
> > the 7tb is compressed, and uncompressed, being text, is much, much larger,
> > and using btrfs transparent compression leaves a lot to be desired when it
> > comes to stability (i'm starting again after losing data after a very brief
> > power-loss).
>
> There is several things I don't understand here, transparent compression
> is "transparent" to the user and applications. Thus the fact btrfs
> stores files compressed and make them available as if they were not
> compressed does not change anything from applications standpoint and
> user point of view. When a power-loss occurs, data that was read may
> should not be affected less or more may it be stored compressed or not, no?

sorry, i meant that i _won't_ be using btrfs. i was uncompressing
archives on a btrfs volume when a power loss occurred, and tried to
repair the volume, but the process to repair it kept on abandoning
without a discernible reason in logs.

i'm reluctant to use zfs due to linus torvalds not recommending it, so
instead i'll use ext4 without transparent compression. i may run into
disk space problems when recombining decompressed files, but i'll look
into that later (i might use fuse as a last resort).

> >
> > i've tried using other backup tools such as rdiff-backup, however it seems
> > i would have to have all the files available uncompressed, as opposed to
> > adding each file one-by-one - does dar support adding one file to the
> > archive at a time, or did i misunderstand the tutorial?
>
> here too, I don't understand why the fact if rdiff-backup and other need
> data uncompressed (something to be confirmed by the way) and why this is
> said to be the opposite of adding files one-by-one to the backup (to me
> these are two independent things).
>

the files must be decompressed from their .bz2 archives in order to
provide disk space-efficient deltas.

what i meant is that rdiff-backup required all files to be made
available before adding to it's 'archive', where with dar i can
decompress a file, add it to a dar archive, then delete the
decompressed file, then repeat for each file.

> Maybe you mean that rdiff-backup better performs its binary delta when
> it is fed with uncompressed data (which is obviously true)? But anyway,
> btrfs should not provide compressed data to any application: when
> rdiff-backup reads data, btrfs should provide it uncompressed
> (transparently decompressing it the from storage).

i explained poorly, sorry. i won't be using btrfs, but the wiki dumps
are compressed archives (.bz2 and .gz)

> At this level dar will not behave differently. The advantage of dar is
> to have to possibility to compress files per files, and eventually for
> some file you desire, backup them uncompressed (if they are already
> compressed). Compressing per file also give better fault tolerance as a
> corruption taking place in a file data will not affect other compressed
> files. The drawbacks is the slightly less good compression ratio.

i was unaware of this, good to know. so, if i decompress all files in
a wiki, then add them all to a dar archive (not one-by-one) i will get
better compression?

> >
> > some of the xml files are split into groups of pages, such as the english
> > wiki, because they would otherwise be very large files. sometimes a copy of
> > the recombined file is provided along with the split pages files, but for
> > the large english wiki meta-history file, for example, a recombined copy is
> > not provided, so writing a script to recombine it shouldn't be too
> > difficult (using cat and grep, etc.).
>
> Dar has several features here that can help:
> - first the filtering mechanism that gives you the possibility to ignore
> recombined files and only consider the version split into groups (or the
> opposite if you prefer to keep only the later and ignore the split
> versions).
> - the backup-hook-execute can trigger a script of your own before and
> after saving any particular file you desire, providing as parameters the
> filename, path, and some other interesting fields.

i want the recombined files and to skip the split files

> >
> > i want the decremental backup each month to only store the changes to the
> > large files, saving a large amount of space. does dar support only backing
> > up the difference of the file, as compared to simply storing a complete
> > copy of the same file when they're different?
>
> Yes, it does but you seem to make a confusion between two different
> features:
> - decremental/incremental/differential backup
> - binary delta
>
> For the first features:
> - Decremental backup (the latest backup is a full backup and oldest
> contain difference from the latest), it implies to convert the previous
> full backup (the one that was before the latest you just created) into a
> decremental backup, this is time/CPU/IO consuming but speeds the
> recovery time as you just have to restore a single full backup (the
> latest) upon disaster.
> - Incremental (and its cousin the differential backup) have the full
> backup as the oldest and recent backup have only files that changed
> since the backup of reference (for differential this reference is the
> first backup, for incremental this is the latest backup, and you can
> also combine both of them over time, but I will lost you if I develop
> that part).

the features i want are decremental (latest backup easily accessed)
with deltas, because with each dump, all the files are the same names
with different content, aside from large wikis that split large files,
which i will run a script to recombine into the same named files - the
reason for doing deltas is simply to save disk space.

>
> On the other hand the binary delta relies on librsync and makes a patch
> of file (text or binary) to only record the part of the file that changed.
> When this feature is not activated and a file has been seen as modified
> since the previous backup, it is saved as a whole in the new backup
> eventually compressed by one of the many compression algorithms and
> compression level dar provides (while unchanged files are not resaved at
> all).
>
> Hopefully you can combine binary delta with incremental and differential
> backup. But as of today binary delta is not supported (yet?) with
> decremental backup.

i should be able to write a bash script to do decremental with deltas
instead (by decompressing the dar archives/diffs, then reversing the
order of the wiki dumps and doing incremental?) and a new set every 12
months,

otherwise, to keep things simple, i may just run differential or
incremental backups with the delta feature. considering it's 7tb to
manage each month, cpu time could indeed be an issue.

what do you recommend? incremental or differential? i want to save
disk space, and in the long term, disk space may not be an issue
anymore as drive prices drop (but it still might be if machines start
adding information to wikis, such as wikidata)

>
> >
> > aside from multiple backups, is there any data redundancy built into dar,
> > yet? (i'm worried about bitrot)
> >
> >
> yes, there is redundancy in dar backups to detect corruption but not to
> repair data. For reparing data, dar relies and integrates with Parchive
> which is optimum for this type of job.

i'll look into this, and i'll simulate bit rot once i get this up and running.