Re: Error exits on busy systems

I should give you a little more background on what I'm doing.

I've been switching my network to asynchronous backups using NNCP. 
I've written a blog series about it here:

https://changelog.complete.org/archives/tag/nncp

In particular, I covered ZFS-based backups over NNCP here:

https://changelog.complete.org/archives/10186-more-topics-on-store-and-forward-possibly-airgapped-zfs-and-non-zfs-backups-with-nncp

However, some of the machines I deal with can't run ZFS.  They 
include some older workstations, various Raspberry Pis, and so 
forth.  None of them are using LVM, either, and in some cases, 
bandwidth is heavily constrained (eg, 4G only).  So what I need is 
a bandwidth-efficient backup that doesn't require the destination 
to be online and reachable at the moment the backup is taken. 
That instantly rules out most of the "modern" backup solutions 
that are based on rsync, or others such as borg, duplicity, 
duplicati, etc.

GNU tar has a listed-incremental mode, but it has some rather 
pathological bugs that I'm sure you're aware of.  That left dar 
and rdup.  All of dar's power with catalog munging and such made 
it a natural fit.  I have been backing up one system with dar for 
a few months now, but have found it somewhat fragile and have been 
investigating why.

One property of NNCP (which is conceptually similar to UUCP if you 
ever used that) is that it does not guarantee ordering of packet 
processing, though it does guarantee processing only one packet at 
once, and will retry failed packets.  With ZFS, this was ideal; a 
zfs receive will simply fail if it lacks the necessary 
intermediate snapshots.  But with dar, I needed some scripting.

What I'm doing is this...

(assuming it's not the first backup, which must of course always 
be a full one)

On the source machine, I run:

dar -q -c - --aux "$DARMETA/$THISCAT" -A "$DARMETA/$PREV" -R / \
  --empty-dir \
  --retry-on-change 20:104857600 \
  -P [...] \
  | zstd -4 - \
  | gpg --compress-algo none --cipher-algo AES128 -e -r [...] \
  > "$TMPDIR/darfile"

Then if it succeeds, remove $DARMETA/$PREV.1.dar, and then pipe it 
over to nncp-file for transmission.  Encoded in the filename is 
both the name of the previous catalog and the name of this one.

On the receiving side, the backup server that is airgapped or 
nearly so, before unpacking is attempted, we compare the previous 
catalog name with what it had recorded as the previous catalog 
name (just a touch in a darmeta directory).  If they match, it 
proceeds with:

gpg -q -d | zstdcat -T0 | dar --sequential-read -w -x - -R 
"/$DESTFS"

If that works, we touch a new file with this catalog name and 
remove the old one.

So it is very important that I know programmatically at every step 
if the dar command succeeds, because otherwise I have a risk of 
the backups becoming out of sync.

So in this way, dar becomes essentially an asynchronous rsync.

Once I get this all sorted out, I'll be publishing a recipe for it 
too.

- John