Re: Couple of questions regarding dar

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello,

dar_split has received some enhancements (available on git/master) :

- -c option let dar_split automatically stop after <count> tape read, so
dar will also stop automatically once the end of the <count> tape will
be reached.
- -b let you reduce the block size used to read/write this seems to be
the cause of the "Error while reading from pipe: Cannot allocate
memory" when reading from pipe a too large amount at once.
- -r let you rate limit the operation done by dar_split... which is
probably not needed here.

these will be released in 2.7.0 in a few months.

I hope it will help,
Regards,
Denis

On 14/05/2020 13:57, Denis Corbin wrote:
> On 12/05/2020 21:26, Nils Privat wrote:
>> Hello,
> 
> Hello Nils,
> 
> 
>> first of all i want to thank you for your time and help.
> 
> and thank you for the tests your did:
> 
>> i upgraded to v.2.6.9 and run the same script and got same
>> results, so the "old" dar version was not the problem.
> 
> OK, good to know
> 
>> I also tried the script with putting in "mf -f $TAPE eof"
>> between the dar creations, but that doesnt help too. Instead i
>> got an "empty" file between the dar full backup and diff-backup
>> (full backup start at file number=0, and diff backup at file
>> number=2 because i manually forced an eof in between). So it
>> seems like dar/dar_split/the tape itself is writing an eof after
>> a successful backup....
> 
> Strange to me. But OK, I assume the operating system adds such EOF 
> mark when closing the filedscriptor used to write data to tape.
> 
> 
>> Second i run the all the commands you suggested (copy back the
>> dar backup from tape to file) and got no errors - listing was
>> fine and testing too (see commands and output here: 
>> https://pastebin.com/raw/UUNqjWtB). It even showed the "[--- 
>> REMOVED ENTRY ----]" correctly.
> 
> OK that's indicates the archive is properly written do tape, good.
> 
> 
>> Yes, for all the tests i use the sametape (no 
>> ejecting/reloading/swichting the tape). I just used dar_split 
>> because for future backups i want to use multiple tapes. I
>> though it was not a problem to use dar_split even for a single
>> tape. For identifying the issue i skip the use of dar_split. See 
>> output/testing without using dar_split here: 
>> https://pastebin.com/3xZ6VTxa
> 
> the error message you get from dar "FATAL error, aborting
> operation: Error while reading from pipe: Cannot allocate memory"
> is reported due to a failed read() system call, and the operating
> system justifies the error by "Cannot allocate memory" (!) I'm
> confused, the system is provided with a memory allocated place to 
> copy data to... so it has not to allocate any memory to provide
> the requested bytes for reading. The only possible explanation I
> have is that the mount of data requested at once is too large for
> the underlying driver (?)
> 
> 
> 
>> So without using dar_split it seems everything is working 
>> correctly, altought directly redirect the tape to dar (dar -l - 
>> --sequential-read < /dev/nst0) was not working for me, but that 
>> could be an blocksize specific thing. But as you can see without 
>> dar_split i got the correct output from dar --test and also the 
>> restore from diff-backup deletes files (which was both not the
>> case in my testing with dar_split in my previous mail). So what
>> to do next?
> 
> I will see how to adapt/enhance dar_split to let you specify more 
> parameters there like a block size per system read/write call. The 
> work done by mbuffer is perturbed by dar_split except the rate (-R 
> option) at which data is sent.
> 
> 
>> Maybe my explanation regarding the 4) point in the previous mail 
>> was a bit unclear (english is not my mother language): I
>> understand that dar_split is printing "No more data available
>> from source, please do something!" and its waiting for manual
>> input of the user on what to do next (for e.g. switching tapes).
> 
> correct
> 
>> Pressing ctrl-c does not always exists gracefully (see 
>> script/output in previous mail) and pressing return/enter
>> results in dar_split reading further from tape (which is correct
>> in case of a tape switching).
> 
> OK, you have dar and dar_split both sending characters to
> terminal. Dar_split reads from tapes and stick them toghether to
> its stdout, it does not interpret its content. Upon reading an End
> of File, it shows this message "... please do something!", which
> should maybe be changed to something clearer... English is neither
> my mother tong ;^) dar_split cannot this know whether this is the
> last tape or not and asks until there is no more reader on of its
> standard output. In particular, in sequential-read mode dar will
> read up to the EOF. So until dar_split exit and thus expects your
> CTRL-C to be received.
> 
> but this CTRL-C can also be sent to dar espetially if both dar and 
> dar_split use the same terminal. So this makes dar interrupt the 
> operation and sending you this message: "Received signal: Interrupt
> / Archive delayed termination engaged".
> 
> To cope with that, you could do the following using two different
> xterm:
> 
> term1: mkfifo dar_spit_to_dar dar_split split_input $TAPE >
> dar_split_to_dar
> 
> term2: dar -l - --sequential-read < dar_split_to_dar
> 
> eventually adding mbuffer somewhere in the middle if you need.
> 
> this way, hitting CTRL-C in term1 once all tape have been read
> will avoid the terminal sending the INT signal to dar as well as
> dar_split. So you will see in term2 the exact dar output without
> mixed messaged from dar_split
> 
>> In my specific case, without tape switching and appended other 
>> backups/data on the same tape, it results in dar_split reading
>> the next file on the same tape and prints messages like:
> 
>> # ERR <ROOT>/1.rar : Skipping backward is not possible on a pipe
>> # A problem occurred while reading this archive contents:
>> Skipping backward is not possible on a pipe # Final memory
>> cleanup... # Some files are corrupted in the archive and it will
>> not be possible to restore them
> 
> this message comes from dar, which expectes to read a single
> sliced archive and get in fact a new archive concatenated to the
> archive he was reading. This flow of structured data is not
> expected by dar and leads to this error which may leads to
> different error (elastic buffer relative error below is at
> encryption layer), and the previous takes place at the overall
> archive structure.
> 
> 
>> or
> 
>> # Final memory cleanup... # Error while listing archive
>> contents: elastic buffer incoherent structure
> 
> 
>> Thanks
> 
> 
> 
> 
> Regards, Denis
> 
> 
> _______________________________________________ Dar-support mailing
> list [email protected] 
> https://lists.sourceforge.net/lists/listinfo/dar-support
> 
> 
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl69jBQACgkQCDGwvQPY
sYKWnRAAg7WB4yGPdwFh/KRXKn7R3gVdLaOp8rnPGzZVGLgu60DRZqCiDdYbH4TA
ypFJd1CszqOgs/ih1Ytn2NOwc+ETBQNbG3CxuW1S4DB6URvcu2UjZv18TqLyU6TS
Bum3hXyj4lKDnAtpNOZOjnquu6IHahftCplFwO+Hw8xu/E/1ixXCarq5ULykzOzV
4szGzq8LXRWT6go17z9myCZPWA86v/KOj/9CyUBRPn3aUnAtBIUtOCl3Q6VcWbb/
Jt7LscPZqL+Xf8XVjbI2FcW+1VcPZFX/trOO9PvHD8hVxp/scDP48dD/I5/3Y56H
qJgba7Y7UretuOpLBkgKiOQCACrHWqy5M1epWhukTfSRY8ay/pXhYvbZmlWOqpH6
JAdnKTKZwzClSbH7gzXgLRwZRnbC07qWO7zhX8zCjEaKZ0iwEiZb9Y1YhCTF5EvE
mAgjPeTXhUp0QGM/5XXLmqvpvvLrbwlWXlnywg3+Mt6Fq9s5IE60AmKCikmCMImh
SP6kVgIzZQ0uMX/ajBhD4fREGMCiCEP5HDIcgrZt9HJc7ltP0UD4jQikAC07z00+
qB9iJJep1PzwznVRmwvoHKjEYk8Zls+SuxYuA077OshyzwDSTG7oF7JzAvFCIVC0
dy2YdvAs/x7NV7f2MAI/I1WINn/x2cAEJrPVm96SxuTUlUW3GXI=
=UCTu
-----END PGP SIGNATURE-----