Re: Error exits on busy systems

On Sat, Jul 10 2021, Denis Corbin wrote:

> On 10/07/2021 04:00, John Goerzen wrote:
>> I should give you a little more background on what I'm doing.
>>
>> I've been switching my network to asynchronous backups using 
>> NNCP. I've
>> written a blog series about it here:
>>
>> https://changelog.complete.org/archives/tag/nncp
>
> Very interesting article! The population of the last century 
> that was

Thank you!  And that was only the most recent in the series ;-)

Some others that may be relevant to this audience include:

https://changelog.complete.org/archives/10160-how-why-to-use-airgapped-backups
https://changelog.complete.org/archives/10163-rehabilitating-asynchronous-communication-with-nncp-a-cross-between-tor-ssh-and-uucp
https://changelog.complete.org/archives/10219-a-simple-delay-tolerant-offline-capable-mesh-network-with-syncthing-optional-nncp

> criticize, which IMHO takes its root in the will to control and 
> in the
> financial appetite to gather information (data) on people by big
> companies you know, and value back this information for 
> themselves and
> their own customers. ...Even my calculator application on my 
> mobile
> requests network access, Crazy!

Sigh, that IS nuts.  Yes, I have noticed the dramatic difference 
between me growing up with a TRS-80 (had to learn BASIC) and kids 
now growing up with tablets.  Though I don't know if programming, 
as a percentage of the population, is any more niche today; it 
certainly is more niche as a percentage of computer owners.

> I would not be surprised if one day, UUCP, NNCP (thanks to 
> pointing us
> to that, by the way), ToR and other decentralized networks would 
> be
> qualified as supporting terrorism, leading their users to be
> "criminalized" and tracked down (see P2P fate, some providers 
> even
> forbid this protocol, while it can be used for legal things).

Not farfetched; we've seen the FBI try to demonize encryption in 
the media before.  Though I hope that by now that ship has sailed 
too far on this.

> And I love your remark on the github's dark mode! ;^)

What can I say, that one made me feel curmudgeonly for a month :-)

[ some snips ]

> Some remarks:
>
> * if you don't care having dirty files, you could just avoid 
> wasting
> space in backups and network transfers by using 
> "--retry-on-change 0".
> (and use "--dirty-behavior no-warn" at restoration time as you 
> mentioned
> in your previous email).

Thank you; I will be doing that.

> * You could use AES256 from within dar instead of piping to gpg 
> (see -K
> option). It adds more protection thanks to the use of salt and 
> Key
> Derivation Function (argon2 since 2.7.0) and having your saved 
> data
> protected against clear text attack by the addition of elastic 
> buffers
> (more details on that topic here:
> http://dar.linux.free.fr/doc/Notes.html#strong_encryption)

I really like all the random access features of dar.  It has been 
quite the long-standing weakness in tar, particularly when applied 
to very large archives.  And the alternatives (zip, 7z, etc) 
aren't trustworthy to make a high-fidelity copy of the POSIX 
metadata.

Having said that, in this particular use case, I am immediately 
extracting the archive to a ZFS dataset on the backup server, and 
using stream mode saves an extra unnecessary copy from being 
written to disk, so I'll probably stick with it here.  Still, I 
need to think through the possibilities better (eg, the FUSE 
interface could be quite nice).  I've occasionally looked into 
alternatives (eg, ISO9660/UDF, SquashFS, etc) .  Those all have 
some drawbacks.

> * you can also avoid piping to zstd by using dar's -zzstd:4 
> option. The
> advantage is that dar will be able to read/restore files after a
> corruption which is not possible if you encrypt the whole backup 
> at once
> (which is a main drawback of tar and one of the main reason of 
> existence
> of dar)

Very true.   Part if the reason I'm using zstd | gpg here is that 
it's also what I'm doing with other data sources (particularly 
zfs).

I should note, incidentally, that strictly speaking, it's 
unnecessary, since NNCP already provides end-to-end encryption and 
verification.  However, I prefer to use it just as an extra guard 
against a bug in NNCP, or to permit me to use untrusted transit if 
I ever need to.

> last, since 2.7.0, encryption (which relies on libgcrypt) can 
> leverage
> several threads (same thing for compression), see --multi-thread 
> option.
> Though you will need libthreadar (easy to compile but not yet 
> available
> in many distro). But if network is slow in the middle, 
> multi-threading
> may not be very interesting though...

I generally prefer to use packages at the versions in Debian 
(though that's not a hard requirement).  Debian stable has 2.6.2 
and testing has 2.6.13, but someone did upload 2.7.1 to 
experimental, which is a good sine.

> * Data reduction on the wire seems important in you use case, 
> you might
> consider using binary delta with dar (see --delta options). Also 
> adding
> forward error correction (and thus avoiding re-transmission of a
> corrupted set of data) would be valuable, for this I would 
> suggest the
> great Parchive program (I have no participation and no personal 
> interest
> in it, it is just a greatly valuable software).

parchive/par2 is a really nifty piece of software!  In this 
particular use case, NNCP guarantees accurate and complete 
transmission already, so I'm not over concerned with it (and I'm 
unpacking the archive on the destination).  The source systems are 
low-end (eg, Raspberry Pis with MicroSD cards) and the destination 
is a 12TB backup datastore that has a 12TB offline twin for 
redundancy.

Sadly the Debian packages are built without librsync; see 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=918075 (and my 
comment from January asking for it to be added).  I have had some 
thought to using rdiff across the entire archive.  This would have 
the benefit of being the absolute most bandwidth-efficient, as it 
would handle a file being renamed or moved.  But it is somewhat of 
a hassle to implement.

Thanks,

John