Re: n00b questions about using dar for archiving wikis [apologies]

i didn't mean to upset you, i don't want you to do any development
unnecessarily.

i just wanted to know how i would use dar.

in any case, thanks for your time.

On Tue, 20 Jul 2021 at 16:00, griffin tucker
<darsuporpt340-DSTKDfZ7/[email protected]> wrote:
>
> On Tue, 20 Jul 2021 at 05:18, Denis Corbin <[email protected]> wrote:
> >
> > On 19/07/2021 08:02, griffin tucker wrote:
> > > i'll try to be as comprehensive as possible so that i don't miss anything.
> > >
> > > i want to make decremental backups of 7tb worth of wikis (text only)
> > > provided by wikimedia, each wiki dump in a separate dar archive
> >
> > OK
> >
> > >
> > > wikidumps are updated each month. and from what i've read in the dar
> > > tutorial, if i understood it correctly, i'll have to write a script to do
> > > decremental backups, which shouldn't be a problem.
> >
> > Well you may need a script to manage rotation and eliminate oldest
> > decremental but doing a decremental backup by itself is juste calling
> > dar with the adhoc options.
>
> the end goal is an interactive/options bash script that downloads the
> latest wiki(s) and decompresses the files one-by-one (to save disk
> space) then adding each decompressed file to it's associated dar
> archive, then uploading the dar archive to a cloud service (or
> moving/rearranging a folder monitored by a cloud upload service)
>
> > >
> > > the 7tb is compressed, and uncompressed, being text, is much, much larger,
> > > and using btrfs transparent compression leaves a lot to be desired when it
> > > comes to stability (i'm starting again after losing data after a very brief
> > > power-loss).
> >
> > There is several things I don't understand here, transparent compression
> > is "transparent" to the user and applications. Thus the fact btrfs
> > stores files compressed and make them available as if they were not
> > compressed does not change anything from applications standpoint and
> > user point of view. When a power-loss occurs, data that was read may
> > should not be affected less or more may it be stored compressed or not, no?
>
> sorry, i meant that i _won't_ be using btrfs. i was uncompressing
> archives on a btrfs volume when a power loss occurred, and tried to
> repair the volume, but the process to repair it kept on abandoning
> without a discernible reason in logs.
>
> i'm reluctant to use zfs due to linus torvalds not recommending it, so
> instead i'll use ext4 without transparent compression. i may run into
> disk space problems when recombining decompressed files, but i'll look
> into that later (i might use fuse as a last resort).
>
> > >
> > > i've tried using other backup tools such as rdiff-backup, however it seems
> > > i would have to have all the files available uncompressed, as opposed to
> > > adding each file one-by-one - does dar support adding one file to the
> > > archive at a time, or did i misunderstand the tutorial?
> >
> > here too, I don't understand why the fact if rdiff-backup and other need
> > data uncompressed (something to be confirmed by the way) and why this is
> > said to be the opposite of adding files one-by-one to the backup (to me
> > these are two independent things).
> >
>
> the files must be decompressed from their .bz2 archives in order to
> provide disk space-efficient deltas.
>
> what i meant is that rdiff-backup required all files to be made
> available before adding to it's 'archive', where with dar i can
> decompress a file, add it to a dar archive, then delete the
> decompressed file, then repeat for each file.
>
> > Maybe you mean that rdiff-backup better performs its binary delta when
> > it is fed with uncompressed data (which is obviously true)? But anyway,
> > btrfs should not provide compressed data to any application: when
> > rdiff-backup reads data, btrfs should provide it uncompressed
> > (transparently decompressing it the from storage).
>
> i explained poorly, sorry. i won't be using btrfs, but the wiki dumps
> are compressed archives (.bz2 and .gz)
>
> > At this level dar will not behave differently. The advantage of dar is
> > to have to possibility to compress files per files, and eventually for
> > some file you desire, backup them uncompressed (if they are already
> > compressed). Compressing per file also give better fault tolerance as a
> > corruption taking place in a file data will not affect other compressed
> > files. The drawbacks is the slightly less good compression ratio.
>
> i was unaware of this, good to know. so, if i decompress all files in
> a wiki, then add them all to a dar archive (not one-by-one) i will get
> better compression?
>
> > >
> > > some of the xml files are split into groups of pages, such as the english
> > > wiki, because they would otherwise be very large files. sometimes a copy of
> > > the recombined file is provided along with the split pages files, but for
> > > the large english wiki meta-history file, for example, a recombined copy is
> > > not provided, so writing a script to recombine it shouldn't be too
> > > difficult (using cat and grep, etc.).
> >
> > Dar has several features here that can help:
> > - first the filtering mechanism that gives you the possibility to ignore
> > recombined files and only consider the version split into groups (or the
> > opposite if you prefer to keep only the later and ignore the split
> > versions).
> > - the backup-hook-execute can trigger a script of your own before and
> > after saving any particular file you desire, providing as parameters the
> > filename, path, and some other interesting fields.
>
> i want the recombined files and to skip the split files
>
> > >
> > > i want the decremental backup each month to only store the changes to the
> > > large files, saving a large amount of space. does dar support only backing
> > > up the difference of the file, as compared to simply storing a complete
> > > copy of the same file when they're different?
> >
> > Yes, it does but you seem to make a confusion between two different
> > features:
> > - decremental/incremental/differential backup
> > - binary delta
> >
> > For the first features:
> > - Decremental backup (the latest backup is a full backup and oldest
> > contain difference from the latest), it implies to convert the previous
> > full backup (the one that was before the latest you just created) into a
> > decremental backup, this is time/CPU/IO consuming but speeds the
> > recovery time as you just have to restore a single full backup (the
> > latest) upon disaster.
> > - Incremental (and its cousin the differential backup) have the full
> > backup as the oldest and recent backup have only files that changed
> > since the backup of reference (for differential this reference is the
> > first backup, for incremental this is the latest backup, and you can
> > also combine both of them over time, but I will lost you if I develop
> > that part).
>
> the features i want are decremental (latest backup easily accessed)
> with deltas, because with each dump, all the files are the same names
> with different content, aside from large wikis that split large files,
> which i will run a script to recombine into the same named files - the
> reason for doing deltas is simply to save disk space.
>
> >
> > On the other hand the binary delta relies on librsync and makes a patch
> > of file (text or binary) to only record the part of the file that changed.
> > When this feature is not activated and a file has been seen as modified
> > since the previous backup, it is saved as a whole in the new backup
> > eventually compressed by one of the many compression algorithms and
> > compression level dar provides (while unchanged files are not resaved at
> > all).
> >
> > Hopefully you can combine binary delta with incremental and differential
> > backup. But as of today binary delta is not supported (yet?) with
> > decremental backup.
>
> i should be able to write a bash script to do decremental with deltas
> instead (by decompressing the dar archives/diffs, then reversing the
> order of the wiki dumps and doing incremental?) and a new set every 12
> months,
>
> otherwise, to keep things simple, i may just run differential or
> incremental backups with the delta feature. considering it's 7tb to
> manage each month, cpu time could indeed be an issue.
>
> what do you recommend? incremental or differential? i want to save
> disk space, and in the long term, disk space may not be an issue
> anymore as drive prices drop (but it still might be if machines start
> adding information to wikis, such as wikidata)
>
> >
> > >
> > > aside from multiple backups, is there any data redundancy built into dar,
> > > yet? (i'm worried about bitrot)
> > >
> > >
> > yes, there is redundancy in dar backups to detect corruption but not to
> > repair data. For reparing data, dar relies and integrates with Parchive
> > which is optimum for this type of job.
>
> i'll look into this, and i'll simulate bit rot once i get this up and running.