Re: Couple of questions regarding dar

Am Do., 14. Mai 2020 um 20:21 Uhr schrieb Denis Corbin <[email protected]>:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hello,
>
> dar_split has received some enhancements (available on git/master) :
>
> - -c option let dar_split automatically stop after <count> tape read, so
> dar will also stop automatically once the end of the <count> tape will
> be reached.
> - -b let you reduce the block size used to read/write this seems to be
> the cause of the "Error while reading from pipe: Cannot allocate
> memory" when reading from pipe a too large amount at once.
> - -r let you rate limit the operation done by dar_split... which is
> probably not needed here.
>
> these will be released in 2.7.0 in a few months.
>
>
Wow, you are fast :)
I did some performance testing with the different compress-algo's, just in
case some users are interested in (see table below or
https://pastebin.com/raw/K9w4VRf3).

Note that the files i tried to compress are not very compressible in
general but that the new z-std algorithm got comparatively good compression
rates very fast.

Two question regarding encryption:
1) As you can see i use aes256. The archive with aes256 is a little bigger
than without encryption, is that expected?
2) For all my tests i use and Intel Xeon E5-1650v3 with AES-NI support.
During the backup with encryption, the CPU utilization (one core) is
significantly higher than without encryption (without enc. 20%@ 1-Core and
with enc. 70-95%@ 1-Core). Does dar using the build-in AES-NI hardware
instructions? If yes, is it correct that even with hardware supported AES
instructions the CPU utilization is so high? Because dar is single threaded
(and multi-thread support is experimental) there is no much more computing
power left for compression (even with new zstd-1). So in my testing the
write throughput by dar is slower than my tape writing rate (around 100
mb/s). Is there anyway to tweak that a bit?

Last question:
Till you official release 2.7.0, can you provide dar_static builds from
current git/master? That would be awesome.

Thanks



Compression results:
+------------------------+--------------------------+----------------------------+----------------------------+
| compression/encryption | time (write to dev/null) | dar archive size in
bytes  | Size reduce/increase in %  |
+------------------------+--------------------------+----------------------------+----------------------------+
| no_comp_no_enc         | 0:4.115s                 | 1456463357
     | 0% (base)                  |
+------------------------+--------------------------+----------------------------+----------------------------+
| no_comp_aes256         | 0:7.582s                 | 1458752081
     | 0,2%                       |
+------------------------+--------------------------+----------------------------+----------------------------+
| gzip-1_aes256          | 0:44:643s                | 1426866177
     | -2,0%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| gzip-3_aes256          | 0:44:626s                | 1426425169
     | -2,1%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| lzo-1_aes256           | 0:57:710s                | 1433101825
     | -1,6%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| lzo-6_aes256           | 0:59:550s                | 1432353137
     | -1,7%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| xz-1_aes256            | 4:40:928s                | 1410025825
     | -3,2%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| xz-3_aes256            | 7:50:528s                | 1401800513
     | -3,8%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| bzip2-1_aes256         | 2:48:092s                | 1437747793
     | -1,3%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| bzip2-3_aes256         | 2:45:918s                | 1431399329
     | -1,7%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-1_aes256          | 0:9:297s                 | 1423153505
     | -2,3%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-3_aes256          | 0:9:853s                 | 1409851473
     | -3,2%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-4_aes256          | 0:10.166                 | 1409513025
     | -3,2%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-5_aes256          | 0:16.955                 | 1406733649
     | -3,4%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-6_aes256          | 0:18:520s                | 1406436225
     | -3,4%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-11_aes256         | 0:34:045s                | 1402641505
     | -3,7%                      |
+------------------------+--------------------------+----------------------------+----------------------------+
| zstd-16_aes256         | 2:46:208s                | 1402221009
     | -3,7%                      |
+------------------------+--------------------------+----------------------------+----------------------------+




> I hope it will help,
> Regards,
> Denis
>
> On 14/05/2020 13:57, Denis Corbin wrote:
> > On 12/05/2020 21:26, Nils Privat wrote:
> >> Hello,
> >
> > Hello Nils,
> >
> >
> >> first of all i want to thank you for your time and help.
> >
> > and thank you for the tests your did:
> >
> >> i upgraded to v.2.6.9 and run the same script and got same
> >> results, so the "old" dar version was not the problem.
> >
> > OK, good to know
> >
> >> I also tried the script with putting in "mf -f $TAPE eof"
> >> between the dar creations, but that doesnt help too. Instead i
> >> got an "empty" file between the dar full backup and diff-backup
> >> (full backup start at file number=0, and diff backup at file
> >> number=2 because i manually forced an eof in between). So it
> >> seems like dar/dar_split/the tape itself is writing an eof after
> >> a successful backup....
> >
> > Strange to me. But OK, I assume the operating system adds such EOF
> > mark when closing the filedscriptor used to write data to tape.
> >
> >
> >> Second i run the all the commands you suggested (copy back the
> >> dar backup from tape to file) and got no errors - listing was
> >> fine and testing too (see commands and output here:
> >> https://pastebin.com/raw/UUNqjWtB). It even showed the "[---
> >> REMOVED ENTRY ----]" correctly.
> >
> > OK that's indicates the archive is properly written do tape, good.
> >
> >
> >> Yes, for all the tests i use the sametape (no
> >> ejecting/reloading/swichting the tape). I just used dar_split
> >> because for future backups i want to use multiple tapes. I
> >> though it was not a problem to use dar_split even for a single
> >> tape. For identifying the issue i skip the use of dar_split. See
> >> output/testing without using dar_split here:
> >> https://pastebin.com/3xZ6VTxa
> >
> > the error message you get from dar "FATAL error, aborting
> > operation: Error while reading from pipe: Cannot allocate memory"
> > is reported due to a failed read() system call, and the operating
> > system justifies the error by "Cannot allocate memory" (!) I'm
> > confused, the system is provided with a memory allocated place to
> > copy data to... so it has not to allocate any memory to provide
> > the requested bytes for reading. The only possible explanation I
> > have is that the mount of data requested at once is too large for
> > the underlying driver (?)
> >
> >
> >
> >> So without using dar_split it seems everything is working
> >> correctly, altought directly redirect the tape to dar (dar -l -
> >> --sequential-read < /dev/nst0) was not working for me, but that
> >> could be an blocksize specific thing. But as you can see without
> >> dar_split i got the correct output from dar --test and also the
> >> restore from diff-backup deletes files (which was both not the
> >> case in my testing with dar_split in my previous mail). So what
> >> to do next?
> >
> > I will see how to adapt/enhance dar_split to let you specify more
> > parameters there like a block size per system read/write call. The
> > work done by mbuffer is perturbed by dar_split except the rate (-R
> > option) at which data is sent.
> >
> >
> >> Maybe my explanation regarding the 4) point in the previous mail
> >> was a bit unclear (english is not my mother language): I
> >> understand that dar_split is printing "No more data available
> >> from source, please do something!" and its waiting for manual
> >> input of the user on what to do next (for e.g. switching tapes).
> >
> > correct
> >
> >> Pressing ctrl-c does not always exists gracefully (see
> >> script/output in previous mail) and pressing return/enter
> >> results in dar_split reading further from tape (which is correct
> >> in case of a tape switching).
> >
> > OK, you have dar and dar_split both sending characters to
> > terminal. Dar_split reads from tapes and stick them toghether to
> > its stdout, it does not interpret its content. Upon reading an End
> > of File, it shows this message "... please do something!", which
> > should maybe be changed to something clearer... English is neither
> > my mother tong ;^) dar_split cannot this know whether this is the
> > last tape or not and asks until there is no more reader on of its
> > standard output. In particular, in sequential-read mode dar will
> > read up to the EOF. So until dar_split exit and thus expects your
> > CTRL-C to be received.
> >
> > but this CTRL-C can also be sent to dar espetially if both dar and
> > dar_split use the same terminal. So this makes dar interrupt the
> > operation and sending you this message: "Received signal: Interrupt
> > / Archive delayed termination engaged".
> >
> > To cope with that, you could do the following using two different
> > xterm:
> >
> > term1: mkfifo dar_spit_to_dar dar_split split_input $TAPE >
> > dar_split_to_dar
> >
> > term2: dar -l - --sequential-read < dar_split_to_dar
> >
> > eventually adding mbuffer somewhere in the middle if you need.
> >
> > this way, hitting CTRL-C in term1 once all tape have been read
> > will avoid the terminal sending the INT signal to dar as well as
> > dar_split. So you will see in term2 the exact dar output without
> > mixed messaged from dar_split
> >
> >> In my specific case, without tape switching and appended other
> >> backups/data on the same tape, it results in dar_split reading
> >> the next file on the same tape and prints messages like:
> >
> >> # ERR <ROOT>/1.rar : Skipping backward is not possible on a pipe
> >> # A problem occurred while reading this archive contents:
> >> Skipping backward is not possible on a pipe # Final memory
> >> cleanup... # Some files are corrupted in the archive and it will
> >> not be possible to restore them
> >
> > this message comes from dar, which expectes to read a single
> > sliced archive and get in fact a new archive concatenated to the
> > archive he was reading. This flow of structured data is not
> > expected by dar and leads to this error which may leads to
> > different error (elastic buffer relative error below is at
> > encryption layer), and the previous takes place at the overall
> > archive structure.
> >
> >
> >> or
> >
> >> # Final memory cleanup... # Error while listing archive
> >> contents: elastic buffer incoherent structure
> >
> >
> >> Thanks
> >
> >
> >
> >
> > Regards, Denis
> >
> >
> > _______________________________________________ Dar-support mailing
> > list [email protected]
> > https://lists.sourceforge.net/lists/listinfo/dar-support
> >
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl69jBQACgkQCDGwvQPY
> sYKWnRAAg7WB4yGPdwFh/KRXKn7R3gVdLaOp8rnPGzZVGLgu60DRZqCiDdYbH4TA
> ypFJd1CszqOgs/ih1Ytn2NOwc+ETBQNbG3CxuW1S4DB6URvcu2UjZv18TqLyU6TS
> Bum3hXyj4lKDnAtpNOZOjnquu6IHahftCplFwO+Hw8xu/E/1ixXCarq5ULykzOzV
> 4szGzq8LXRWT6go17z9myCZPWA86v/KOj/9CyUBRPn3aUnAtBIUtOCl3Q6VcWbb/
> Jt7LscPZqL+Xf8XVjbI2FcW+1VcPZFX/trOO9PvHD8hVxp/scDP48dD/I5/3Y56H
> qJgba7Y7UretuOpLBkgKiOQCACrHWqy5M1epWhukTfSRY8ay/pXhYvbZmlWOqpH6
> JAdnKTKZwzClSbH7gzXgLRwZRnbC07qWO7zhX8zCjEaKZ0iwEiZb9Y1YhCTF5EvE
> mAgjPeTXhUp0QGM/5XXLmqvpvvLrbwlWXlnywg3+Mt6Fq9s5IE60AmKCikmCMImh
> SP6kVgIzZQ0uMX/ajBhD4fREGMCiCEP5HDIcgrZt9HJc7ltP0UD4jQikAC07z00+
> qB9iJJep1PzwznVRmwvoHKjEYk8Zls+SuxYuA077OshyzwDSTG7oF7JzAvFCIVC0
> dy2YdvAs/x7NV7f2MAI/I1WINn/x2cAEJrPVm96SxuTUlUW3GXI=
> =UCTu
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Dar-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-support
>

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support