Re: Couple of questions regarding dar

Am Do., 14. Mai 2020 um 13:58 Uhr schrieb Denis Corbin <[email protected]>:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 12/05/2020 21:26, Nils Privat wrote:
> > Hello,
>
> Hello Nils,
>
> >
> > first of all i want to thank you for your time and help.
>
> and thank you for the tests your did:
> >
> > i upgraded to v.2.6.9 and run the same script and got same results,
> > so the "old" dar version was not the problem.
>
> OK, good to know
>
> > I also tried the script with putting in "mf -f $TAPE eof" between
> > the dar creations, but that doesnt help too. Instead i got an
> > "empty" file between the dar full backup and diff-backup (full
> > backup start at file number=0, and diff backup at file number=2
> > because i manually forced an eof in between). So it seems like
> > dar/dar_split/the tape itself is writing an eof after a successful
> > backup....
>
> Strange to me. But OK, I assume the operating system adds such EOF
> mark when closing the filedscriptor used to write data to tape.
>
> >
> > Second i run the all the commands you suggested (copy back the dar
> > backup from tape to file) and got no errors - listing was fine and
> > testing too (see commands and output here:
> > https://pastebin.com/raw/UUNqjWtB). It even showed the "[---
> > REMOVED ENTRY ----]" correctly.
>
> OK that's indicates the archive is properly written do tape, good.
>
>
Thats the most important point! So i guess i can use dar (maybe without
dar_split) to write my data to tape and can restore it without problems.
Thanks.


> >
> > Yes, for all the tests i use the sametape (no
> > ejecting/reloading/swichting the tape). I just used dar_split
> > because for future backups i want to use multiple tapes. I though
> > it was not a problem to use dar_split even for a single tape. For
> > identifying the issue i skip the use of dar_split. See
> > output/testing without using dar_split here:
> > https://pastebin.com/3xZ6VTxa
>
> the error message you get from dar "FATAL error, aborting operation:
> Error while reading from pipe: Cannot allocate memory" is reported due
> to a failed read() system call, and the operating system justifies the
> error by "Cannot allocate memory" (!)
> I'm confused, the system is provided with a memory allocated place to
> copy data to... so it has not to allocate any memory to provide the
> requested bytes for reading. The only possible explanation I have is
> that the mount of data requested at once is too large for the
> underlying driver (?)
>
>
Maybe i can get rid of the error with the new '-b' option in dar_split
(specify blocksize), will test it.


>
> > So without using dar_split it seems everything is working
> > correctly, altought directly redirect the tape to dar (dar -l -
> > --sequential-read < /dev/nst0) was not working for me, but that
> > could be an blocksize specific thing. But as you can see without
> > dar_split i got the correct output from dar --test and also the
> > restore from diff-backup deletes files (which was both not the case
> > in my testing with dar_split in my previous mail). So what to do
> > next?
>
> I will see how to adapt/enhance dar_split to let you specify more
> parameters there like a block size per system read/write call. The
> work done by mbuffer is perturbed by dar_split except the rate (-R
> option) at which data is sent.
>
>
Awesome, thanks. What exactly do you mean with "perturbed"? I though
mbuffer between dar and dar_split does not hurt?! The reason i use mbuffer
is because '-P 80 -m 8G', so that the mbuffer can fill up a big buffer so
that dar_split gets an continuous stream to write ( i am using dar's
encryption and compression, which is sometimes slower than write speed to
tape, so with mbuffer i can buffer it). Do you dont recommend using it?


> >
> > Maybe my explanation regarding the 4) point in the previous mail
> > was a bit unclear (english is not my mother language): I understand
> > that dar_split is printing "No more data available from source,
> > please do something!" and its waiting for manual input of the user
> > on what to do next (for e.g. switching tapes).
>
> correct
>
> > Pressing ctrl-c does not always exists gracefully (see
> > script/output in previous mail) and pressing return/enter results
> > in dar_split reading further from tape (which is correct in case of
> > a tape switching).
>
> OK, you have dar and dar_split both sending characters to terminal.
> Dar_split reads from tapes and stick them toghether to its stdout, it
> does not interpret its content. Upon reading an End of File, it shows
> this message "... please do something!", which should maybe be changed
> to something clearer... English is neither my mother tong ;^)
> dar_split cannot this know whether this is the last tape or not and
> asks until there is no more reader on of its standard output.
> In particular, in sequential-read mode dar will read up to the EOF. So
> until dar_split exit and thus expects your CTRL-C to be received.
>
> but this CTRL-C can also be sent to dar espetially if both dar and
> dar_split use the same terminal. So this makes dar interrupt the
> operation and sending you this message: "Received signal: Interrupt /
> Archive delayed termination engaged".
>
> To cope with that, you could do the following using two different xterm:
>
> term1:
>         mkfifo dar_spit_to_dar
>         dar_split split_input $TAPE > dar_split_to_dar
>
> term2:
>         dar -l - --sequential-read < dar_split_to_dar
>
> eventually adding mbuffer somewhere in the middle if you need.
>
> this way, hitting CTRL-C in term1 once all tape have been read will
> avoid the terminal sending the INT signal to dar as well as dar_split.
> So you will see in term2 the exact dar output without mixed messaged
> from dar_split
>
>
I didnt know mkfifo yet, but thanks for the tip to not mixing messaged from
dar and dar_split.


> > In my specific case, without tape switching and appended other
> > backups/data on the same tape, it results in dar_split reading the
> > next file on the same tape and prints messages like:
> >
> > # ERR <ROOT>/1.rar : Skipping backward is not possible on a pipe #
> > A problem occurred while reading this archive contents: Skipping
> > backward is not possible on a pipe # Final memory cleanup... # Some
> > files are corrupted in the archive and it will not be possible to
> > restore them
>
> this message comes from dar, which expectes to read a single sliced
> archive and get in fact a new archive concatenated to the archive he
> was reading. This flow of structured data is not expected by dar and
> leads to this error which may leads to different error (elastic buffer
> relative error below is at encryption layer), and the previous takes
> place at the overall archive structure.
>

It seems reasonable that dar prints errors when manually forcing dar to
reading further reading after an EOF / end of tape and following data no
longer belongs to the first stream. So with the mkfifo tip i can just
abroad (CTRL-C) dar_split without canceling dar gracefully.


> >
> > or
> >
> > # Final memory cleanup... # Error while listing archive contents:
> > elastic buffer incoherent structure
> >
> >
> > Thanks
> >
> >
> >
>
> Regards,
> Denis
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl69MiYACgkQCDGwvQPY
> sYLrRw//dchxP4WmhEqjgGj2DmLyMj2CL3nWyqIbs0lKN7i6IUS5f6DFALm8xYq2
> eCIVmgRcFqemEoo4KSzm6VrCPWG1RAgQuEb6C6isOajZ0NdvK4r5p7lbyZWkrapj
> jif0XIuHI0p55qnwuob9qN7ZClTL4EidimvAMFLC2SGFvjwmazxTsyk9XK6jZbjS
> PgZwWtFlPrRLBTD6E84+l4laYLOUf8cAYToM5fRzZ2PwdBQWTJv3s5JAqbFFowAh
> fo/pMgxRDnd7kl7b/TYyYV0FeP11lYWQ0ocLvN3qQipDkqNUPMt1xdcutrLEA8Ap
> W1TNU7/OuCsGqcnA4icdGJpZ+RaXG288auxmN1I7eFUgV0i8HsXHpbFT7Xx66j3i
> 6W/uAbRbjs4oO6f9BnNzGXyPlycsfiP5Udz3yu8WQEJ4QJVFOA706aGmavyED4iV
> qtBpxwxR/NuTQqzbMeldqwL5Tmtd8MzsX48K1Zv9Z1fZCPHzV6z1+aQ98QUfOQbK
> 6eI6Ce8Hdc3L3KpQe3jHAJ4RJ+HrDmTvoncQwPGN3R8HCrOImBT50N4nzBIZ+5jI
> ZqBrJ1kZVj3XgckyWb3geXd0NGyM7fudkYXLFoSKGRLCbMZ0U6K4eB1AkWHsy4/q
> 7BSqnhLlHDc11eUYAzwCDfGPyerzdBpFtb4FPhbFAomE5QNC63U=
> =YtfU
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Dar-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-support
>

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support