Re: Couple of questions regarding dar

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 16/05/2020 17:19, Nils Privat wrote:
> Am Do., 14. Mai 2020 um 20:21 Uhr schrieb Denis Corbin 
> <[email protected] <mailto:[email protected]>>:
> 
> Hello,
> 
> dar_split has received some enhancements (available on git/master)
> :
> 
> -c option let dar_split automatically stop after <count> tape read,
> so dar will also stop automatically once the end of the <count>
> tape will be reached. -b let you reduce the block size used to
> read/write this seems to be the cause of the "Error while reading
> from pipe: Cannot allocate memory" when reading from pipe a too
> large amount at once. -r let you rate limit the operation done by
> dar_split... which is probably not needed here.
> 
> these will be released in 2.7.0 in a few months.>
> 
>> Wow, you are fast :)

"few" does not mean how much ;) I would not assume I would be that
fast, thing will depend on my spare time.

>> I did some performance testing with the different
>> compress-algo's, just in case some users are interested in (see
>> table below or https://pastebin.com/raw/K9w4VRf3).

you do not mention the compression level? By the way, zstd can have up
to level 22 while other protocols support only up to level 9.

> 
>> Note that the files i tried to compress are not very compressible
>> in general but that the new z-std algorithm got comparatively
>> good compression rates very fast.
> 
>> Two question regarding encryption: 1) As you can see i use
>> aes256. The archive with aes256 is a little bigger than without
>> encryption, is that expected?

Yes, this is expected. This is caused by the elastic buffers
more about that at
http://dar.linux.free.fr/doc/Notes.html#strong_encryption
and
http://dar.linux.free.fr/doc/Notes.html#archive_structure

>> 2) For all my tests i use and Intel Xeon E5-1650v3 with AES-NI
>> support. During the backup with encryption, the CPU utilization
>> (one core) is significantly higher than without encryption
>> (without enc. 20%@ 1-Core and with enc. 70-95%@ 1-Core). Does dar
>> using the build-in AES-NI hardware instructions?

Dar relies on libgcrypt library to perform encryption. The use of any
acceleration or hardware related interface is eventually done by
libgcrypt.

> If yes, is it correct that even with hardware
>> supported AES instructions the CPU utilization is so high?

I would bet that libgcrypt is not using hardware instructions, for
some reason.

libdar does not disable hardware encryption support which is enabled
by default in libgcrypt (if libgcrypt compilation has been done
accordingly), and has support for intel-aesni which corresponds to
your hardware.

>> Because dar is single threaded (and multi-thread support is
>> experimental)

correct but the overhead of multi threading management is eating CPU
cycle and introduces synchronization delay, at a point that
performance is not much better than using single thread.

>> there is no much more computing power left for compression (even
>> with new zstd-1). So in my testing the write throughput by dar is
>> slower than my tape writing rate (around 100 mb/s). Is there
>> anyway to tweak that a bit?

you could check if your system has not been configured for libgcrypt
not to use your hardware encryption, or if not, you could check the
impact of disabling hardware acceleration as described here:

https://gnupg.org/documentation/manuals/gcrypt/Hardware-features.html

> 
>> Last question: Till you official release 2.7.0, can you provide
>> dar_static builds from current git/master? That would be
>> awesome.

Sorry I will not. git/master is development code and will still have
its archive structure changing before release 2.7.0. I will not lose
time to handle different intermediate format and backward
compatibility before official release, hope you'll understand.

By the way, I would not use git/master for production backup, even if
it seems to compile nicely and work flawlessly. It has not received
intensive testing, by mean of running a routine that takes a week or
so and usually let to identify a few bugs.

If you do so in spite of my advise, I would recommend taking care of
preciously storing a copy of the source code you used to build dar, as
forward or backward compatibility of the archive/backups generated by
dev version is not possible to guaranty with from a dev version.

> 
>> Thanks
> 
[...]

Cheers,
Denis

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl7BkB4ACgkQCDGwvQPY
sYI1hxAAnTisYcabkqK0xHi4dSqZnelsfV85jdRAfEfCHtKRVm63F2MPCLxNWTye
VQKZW87YGVF8SYqAGJlGKHQH3LaL6wtWuR2kILF3813TtFLVnWmvCPXv977np6yF
K6SOG/YdyF1hKdPKsas9ZA9ftsnfh8nFkb1bf3UvikQxJmKhQdrazeBXloWKENw+
bhaL1OUjkfH3wiZnDVbg8IZpdK02XdUsIvBqz4j1SWjCuT5mJfFzEpqgoRRHRUoD
O62xmHrUrmYoGBzNAHoLsgsw88lkJNJo4a+0DDdLxD/7u44Ay9/Xia6C/alfgPV4
6H5Fm/QLWlYeIBavvqS/ZHVohMVUDsG1BBwJiq3CoBl/Jt+zqNgB+u2qdJbirIk1
rTeKoMMN9ZCix9UTP2ZA2GMqUg+6EH0Z4fwvMaN+PrtE3qYoEj3YziBh7thre5T4
hctAA48PH7zZRvMSzAtxWliTITo+L35fvh2HoFWeu14WL4mWdu7Q2xJTNnwGCwc2
sdwLtpCfO80uuLVbRdrDuTt5X55v4Hy0dV44tc1rY5kfq2xmZIu3Cp45mb7pi9/P
IFArNE9yNMLtWb3BcQcRYT8OGk76HKjXQZM3pFg8XZFbBBzKpEiG+s+7AeZ7hboi
rL72qd1SWAKuSagY0huBapGzxnvD7agZv3qGGQTrg3WL4aSrm80=
=C2NA
-----END PGP SIGNATURE-----