Re: Error exits on busy systems

On 10/07/2021 04:00, John Goerzen wrote:
> I should give you a little more background on what I'm doing.
> 
> I've been switching my network to asynchronous backups using NNCP. I've
> written a blog series about it here:
> 
> https://changelog.complete.org/archives/tag/nncp

Very interesting article! The population of the last century that was
using computers and their networks was massively composed of computer
scientists. Today, the huge majority or users is not concerned about the
"how" nor the "why", but just want to "use" and don't care to be abused.
So in the numbers, the initial population and more broadly the
population of people concerned by the "why" and the "why not" has less
and less chances to fight against the tendency of centralization you
criticize, which IMHO takes its root in the will to control and in the
financial appetite to gather information (data) on people by big
companies you know, and value back this information for themselves and
their own customers. ...Even my calculator application on my mobile
requests network access, Crazy!

I would not be surprised if one day, UUCP, NNCP (thanks to pointing us
to that, by the way), ToR and other decentralized networks would be
qualified as supporting terrorism, leading their users to be
"criminalized" and tracked down (see P2P fate, some providers even
forbid this protocol, while it can be used for legal things).

And I love your remark on the github's dark mode! ;^)

> 
> In particular, I covered ZFS-based backups over NNCP here:
> 
> https://changelog.complete.org/archives/10186-more-topics-on-store-and-forward-possibly-airgapped-zfs-and-non-zfs-backups-with-nncp>
> 
> However, some of the machines I deal with can't run ZFS.  They include
> some older workstations, various Raspberry Pis, and so forth.  None of
> them are using LVM, either, and in some cases, bandwidth is heavily
> constrained (eg, 4G only).  So what I need is a bandwidth-efficient
> backup that doesn't require the destination to be online and reachable
> at the moment the backup is taken. That instantly rules out most of the
> "modern" backup solutions that are based on rsync, or others such as
> borg, duplicity, duplicati, etc.
> 
> GNU tar has a listed-incremental mode, but it has some rather
> pathological bugs that I'm sure you're aware of.  That left dar and
> rdup.  All of dar's power with catalog munging and such made it a
> natural fit.  I have been backing up one system with dar for a few
> months now, but have found it somewhat fragile and have been
> investigating why.
> 
> One property of NNCP (which is conceptually similar to UUCP if you ever
> used that)

Yes, I've been using NNTP over UUCP in the '90s :) with the "Path:"
field at the top of articles filled with a long list band ('!')
separated list of NNTP servers...

> is that it does not guarantee ordering of packet processing,
> though it does guarantee processing only one packet at once, and will
> retry failed packets.  With ZFS, this was ideal; a zfs receive will
> simply fail if it lacks the necessary intermediate snapshots.  But with
> dar, I needed some scripting.
> 
> What I'm doing is this...
> 
> (assuming it's not the first backup, which must of course always be a
> full one)
> 
> On the source machine, I run:
> 
> dar -q -c - --aux "$DARMETA/$THISCAT" -A "$DARMETA/$PREV" -R / \
>  --empty-dir \
>  --retry-on-change 20:104857600 \
>  -P [...] \
>  | zstd -4 - \
>  | gpg --compress-algo none --cipher-algo AES128 -e -r [...] \
>  > "$TMPDIR/darfile"

Some remarks:

* if you don't care having dirty files, you could just avoid wasting
space in backups and network transfers by using "--retry-on-change 0".
(and use "--dirty-behavior no-warn" at restoration time as you mentioned
in your previous email).

* You could use AES256 from within dar instead of piping to gpg (see -K
option). It adds more protection thanks to the use of salt and Key
Derivation Function (argon2 since 2.7.0) and having your saved data
protected against clear text attack by the addition of elastic buffers
(more details on that topic here:
http://dar.linux.free.fr/doc/Notes.html#strong_encryption)

* you can also avoid piping to zstd by using dar's -zzstd:4 option. The
advantage is that dar will be able to read/restore files after a
corruption which is not possible if you encrypt the whole backup at once
(which is a main drawback of tar and one of the main reason of existence
of dar)

* using integrated compression and encryption preserves you from the
sequential reading/writing mode and can lead to very important speed
gains at restoration time when only a few files from a large backup have
to be restored. Beside this, not all features are compatible with
sequential read mode. Note last, that you can still produce a dar backup
to its stdout and read it in sequential or direct access mode, the
structure of the backup is the same.

last, since 2.7.0, encryption (which relies on libgcrypt) can leverage
several threads (same thing for compression), see --multi-thread option.
Though you will need libthreadar (easy to compile but not yet available
in many distro). But if network is slow in the middle, multi-threading
may not be very interesting though...

* Data reduction on the wire seems important in you use case, you might
consider using binary delta with dar (see --delta options). Also adding
forward error correction (and thus avoiding re-transmission of a
corrupted set of data) would be valuable, for this I would suggest the
great Parchive program (I have no participation and no personal interest
in it, it is just a greatly valuable software).

> 
> Then if it succeeds, remove $DARMETA/$PREV.1.dar, and then pipe it over
> to nncp-file for transmission.  Encoded in the filename is both the name
> of the previous catalog and the name of this one.

just to let you know, you can also leverage the --user-comment to store
some metadata in each backup (but it's always stored in clear text). And
quickly read this information from the backup header/trailer by mean of
'dar -l <backup> -aheader' (also works with sequential read mode and
stdin)...

> 
> On the receiving side, the backup server that is airgapped or nearly so,
> before unpacking is attempted, we compare the previous catalog name with
> what it had recorded as the previous catalog name (just a touch in a
> darmeta directory).  If they match, it proceeds with:
> 
> gpg -q -d | zstdcat -T0 | dar --sequential-read -w -x - -R "/$DESTFS"
> 
> If that works, we touch a new file with this catalog name and remove the
> old one.
> 
> So it is very important that I know programmatically at every step if
> the dar command succeeds, because otherwise I have a risk of the backups
> becoming out of sync.
> 
> So in this way, dar becomes essentially an asynchronous rsync.

Even more "rsyncish" if you activate binary delta feature, which relies
on... librsync (!)

> 
> Once I get this all sorted out, I'll be publishing a recipe for it too.

Nice! Thank you!

> 
> - John
> 
> 

Cheers,
Denis