Re: Couple of questions regarding dar

Hello,

first of all i want to thank you for your time and help.

i upgraded to v.2.6.9 and run the same script and got same results, so the
"old" dar version was not the problem.
I also tried the script with putting in "mf -f $TAPE eof" between the dar
creations, but that doesnt help too. Instead i got an "empty" file between
the dar full backup and diff-backup (full backup start at file number=0,
and diff backup at file number=2 because i manually forced an eof in
between). So it seems like dar/dar_split/the tape itself is writing an eof
after a successful backup....

Second i run the all the commands you suggested (copy back the dar backup
from tape to file) and got no errors - listing was fine and testing too
(see commands and output here: https://pastebin.com/raw/UUNqjWtB). It even
showed the "[--- REMOVED ENTRY ----]" correctly.

Yes, for all the tests i use the same tape (no ejecting/reloading/swichting
the tape). I just used dar_split because for future backups i want to use
multiple tapes. I though it was not a problem to use dar_split even for a
single tape. For identifying the issue i skip the use of dar_split. See
output/testing without using dar_split here: https://pastebin.com/3xZ6VTxa
So without using dar_split it seems everything is working correctly,
altought directly redirect the tape to dar (dar -l - --sequential-read <
/dev/nst0) was not working for me, but that could be an blocksize specific
thing. But as you can see without dar_split i got the correct output from
dar --test and also the restore from diff-backup deletes files (which was
both not the case in my testing with dar_split in my previous mail). So
what to do next?

Maybe my explanation regarding the 4) point in the previous mail was a bit
unclear (english is not my mother language): I understand that dar_split is
printing "No more data available from source, please do something!" and its
waiting for manual input of the user on what to do next (for e.g. switching
tapes). Pressing ctrl-c does not always exists gracefully (see
script/output in previous mail) and pressing return/enter results in
dar_split reading further from tape (which is correct in case of a tape
switching). In my specific case, without tape switching and appended other
backups/data on the same tape, it results in dar_split reading the next
file on the same tape and prints messages like:

# ERR <ROOT>/1.rar : Skipping backward is not possible on a pipe
# A problem occurred while reading this archive contents: Skipping backward
is not possible on a pipe
# Final memory cleanup...
# Some files are corrupted in the archive and it will not be possible to
restore them

or

# Final memory cleanup...
# Error while listing archive contents: elastic buffer incoherent structure


Thanks





Am Mo., 11. Mai 2020 um 21:24 Uhr schrieb Denis Corbin <[email protected]>:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Well,
>
> I would first upgrade to the latest release (2.6.9 today), release
> 2.6.2 is more than one year old! Several bugs (in particular one fixed
> in 2.6.5) could be the cause of the problem.
>
> beside that, I'm surprised that you do not need to use "mt -f
> /dev/nst0 eof" right after dropping a dar backup to file... well I
> assume it would not hurt doing so. If there is no EOF seen while
> reading the archive, this may explain why dar_split ends reading the
> tape and asks for the next one.
>
> I would forcibly add eof make using the magnetic tape (mt) command.
>
> If using 2.6.9 does not change the problem then:
>
> can you copy back the dar backup from tape to file:
> mt -f /dev/nst0 rewind
> cat < /dev/nst0 > full.1.dar
> dar -l full
> dar -t full
> cat < /dev/ns0 > diff.1.dar
> dar -l diff
> dar -t diff
>
> if the backup is correct once copied back to disk as showed above,
> can you confirm that all backups take place on the same tape? (I could
> not see any message from dar_split while both backup have been created).
>
> If so, could you just use dar directly to tape (without dar_split)
> mt -f /dev/nst0 rewind
> dar -c - ... > /dev/nst0
> mt -f /dev/nst0 eof
> mt -f /dev/nst0 rewind
> dar -l - --sequential-read < /dev/nst0
> mt -f /dev/nst0 rewind
> dar -t - --sequential-read < /dev/nst0
>
> Cheers,
> Denis
>
> On 10/05/2020 00:46, Nils Privat wrote:
> >
> > In Addition my testing with dar --test with tape:
> > https://pastebin.com/LTBkPX6G
> >
> > Am Sa., 9. Mai 2020 um 23:53 Uhr schrieb Nils Privat
> > <[email protected] <mailto:[email protected]>>:
> >
> > Hello, thanks for your answer. I am using dar v2.6.2 (from debian
> > repo).
> >
> > Maybe it helps when i am explain my testing in more detail:
> >
> > A) First i test dar without tape, just with regular archive files.
> > From my testing everything seems to work fine, see my script /
> > log: https://pastebin.com/bS02nTZz As you can see i created a full
> > backup, then i renamed a folder and removed a file. After that i
> > created a differential backup and 'dar --list' is correctly showing
> > the renamed folder as new files and the "REMOVED ENTIES". Lastly i
> > restored the full backup and it correctly restored all full-backup
> > files. Then, after restoring the differential backup, the file
> > '1.rar' and dir 'A' are deleted and the new dir 'Z' is restored.
> > Everything like expected - works!
> >
> >
> > B) Now, I used the same files and mainly the same script, but the
> > target was a tape. See script with tape:
> > https://pastebin.com/KiK5YnSn (Of course I have adjusted the
> > commands for the use of tape accordingly).
> >
> > Some of my questions regarding the script and command outputs: 1)
> > The listing from catalogue is like expected (see lines 274-301).
> > Why the listing from tape is so different (see lines 315-418)? 1a)
> > Why i got "Error while listing archive contents: elastic buffer
> > incoherent structure" (line 336)? (My guess is the following:
> > dar_split successfully read the content from the tape and after
> > reaching EOF on tape, dar_split stops and prints "no more data
> > available". If i am not changing the tape and just pressing ENTER,
> > dar_split tries to read further and prints the mentioned error.
> > (FYI: this error message doesnt occur without using encryption
> > after pressing ENTER). If my assumption is correct, that dar_split
> > tries to read further, why it does not just print the content of
> > the next file (in my case the differential backup, which is the
> > next file after the full backup). 1b) Why i do not see the "REMOVED
> > ENTIES" when i am listing from tape but from catalogue/regular
> > archive files (see line 395)? 2) Why restoring the differential
> > backup from tape does not deleted files (see line 457-482)? I my
> > first example A), they were deleted. 3) How do i interpret the
> > testing in line 484-522 ? 4) More like a general question: No
> > matter if i read, test or restore a backup from tape, if dar_split
> > reaches and EOF or the end of the tape, it prints "No more data
> > available from source, please do something!" and the user must
> > interact. If we to not want to change or reposition the tape
> > because all data is read, that is the right way to exit the
> > commands? "CTRL-C" print messages like "Archive delayed
> > termination engaged. Disabling signal handler, the next time this
> > signal is received the program will abort immediately." which does
> > not seem the right way and pressing ENTER results in other messages
> > like "elastic buffer incoherent structure" because dar_split seems
> > to read further.
> >
> > Thank you for your help in advance. Still hoping to use dar as my
> > primary tape backup tool :)
> >
> > Am Di., 5. Mai 2020 um 21:49 Uhr schrieb Denis Corbin
> > <[email protected] <mailto:[email protected]>>:
> >
> > On 03/05/2020 11:45, Nils Privat wrote:
> >> Hello,
> >
> > Hello,
> >
> >
> >> much thanks for your answer.
> >
> >> During testing a few new questions have arised.
> >
> >> 1) To list archives i run:
> >
> >> *dar_split split_input $TAPE | dar --list - --sequential-read*
> >
> >> *Output:* * * *[Data ][D][ EA  ][FSA][Compr][S]| Permission |
> >> User | Group | Size |          Date                 |
> >> filename*
> >
> > *--------------------------------+------------+-------+-------+-------
> >
> >
> - --+-------------------------------+------------*
> >
> >
> > *[Saved][ ][Saved][-L-][ 100%][X]  -rwxrwxr--   test test1 MioThu
> > Apr 27
> >> 17:47:59 2017A.rar* *..* *[Saved][ ][Saved][-L-][ 100%][X]
> >> -rwxrwxr--   test test1 MioThu Apr 27 17:47:59 2017E.rar* *No
> >> more data available from source, please do something!* *Press
> >> return when ready to continue or hit CTRL-C to abort* *
> >
> > I guess you have to change of tape, then hit return for dar_split
> > read the next part of the archive
> >
> >> * *No more data available from source, please do something!*
> > *Press
> >> return when ready to continue or hit CTRL-C to abort*
> >> *^CReceived signal: Interrupt* *Archive delayed termination
> >> engaged*
> > *Disabling
> >> signal handler, the next time this signal is received the
> >> program will abort immediately* *[Saved][ ][Saved][-L-][
> >> 100%][X] -rwxrwxr--   test test1 MioThu Apr 27 17:47:59
> >> 2017B.rar*
> >
> >
> >> As you can see listing is working but i worry about that first
> >> it says "No more data available from source, please do
> >> something", then after pressing CTRL-C or enter one last entry of
> >> the listing is still printing "B.rar". Is that normal? Why not
> >> all items are listed together?
> >
> > this is the drawback of using tapes, as tape per nature provide
> > sequential read/write operations (OK you can rewind and fast
> > forward a tape, but as you have no offset information, this is not
> > efficient to randomly read or write data whenever possible). So you
> > have to provide the whole archive for dar to read its whole
> > content. If the archive has been split over several tapes you will
> > have to provide all tapes in the same order (and assume the are
> > rewinded at the position where you have started writing the
> > achive).
> >
> > You still have the possibility to isolate a archive (on fly or
> > afterward) to ease the listing as well as differential/incremental
> > backu ps
> >
> >
> >
> >> 2) Question regarding differential archives on tapes:
> >
> >> I am using this command to create a full backup to tape:
> >
> >> *dar --create - \* *--verbose=all \* *--compression gzip:3 \*
> >> *--alter=no-case \* *-Z "*.zip" -Z "*.rar" -Z "*.gz" -Z "*.7z"
> >> -Z "*.iso" \* *-X ".DS_Store" \* *--key aes256:MyKey \*
> >> *--on-fly-isolate /root/dar_catalog_full \* *--fs-root
> > ${MY_PATH} |
> >> mbuffer -P 80 -m 4G -R 140M | dar_split split_output ${TAPE}*
> >
> >> For differential backup i use the same command but with:
> >
> >> *-A /root/dar_catalog_full \* *--on-fly-isolate
> >> /root/dar_catalog_d1 \*
> >
> >> Do i have to run *'mt -f $TAPE eod'* or 'eof' between each new
> >> backup? Or is dar positioning the tape after a backup already to
> >> the end and is writing an EOF mark so i just can append a new
> >> backup?
> >
> > dar and dar_split does not know about magnetic tape (mt) specific
> > EOF marks (this is media specific). So yes, you should add after
> > one using 'mt' after the end of an archive, espetially if you plan
> > to add another backup or file after this one on the same tape.
> >
> >
> >> INFO: After creating the first backup i checked the position:
> >> *mt -f $TAPE status* *File number=1, block number=0,
> >> partition=0.* So it seems like dar has written and EOF to file 0
> >> and now i am on position/file=1, but executing *'mt -f $TAPE eod'
> >> *then shows: *File number=1, block number=-1, partition=0.*
> >
> > first this is dar_split that writes to the tape special device.
> > And double checking dar_split code, once all bytes are written,
> > dar syncs (should be useless for tape but it does not hurt) then
> > just closes the file it was writing to. This should not set a mt
> > EOF mark AFAIK.
> >
> >
> >
> >> 3) Question regarding encryption:
> >
> >> I create encrypted archives with the command above, rewind the
> >> tape, and list via:
> >
> >> *dar_split split_input $TAPE | dar --list - --sequential-read
> >> --key aes256:MyKey*
> >
> > for reading you don't need to use --key option (but you can) if
> > you prefer to have your password visibile to all on process
> > command line ;)
> >
> >
> >> After pressing ENTER I got: *_Error while listing archive
> > contents:
> >> too large elastic buffer or elastic buffer incoherent
> >> structure_* (I am not getting this message when i am pressing
> >> CTRL-C after listing.)
> >
> > this is either a wrong password, a wrong offset of the tape, or
> > some other incoherence between the creation time and reading time.
> >
> > I've just tested this using a named pipe (mkfifo) to emulate a
> > tape and could create and read an ecrypted archive without
> > problem. This is not likely to be a bug, which version are you
> > using by the way?
> >
> >
> >> If I am not using encryption/compression at all and do the same
> >> thing, i dont get the Error either by pressing ENTER or CTRL-C.
> >> It just exists the listing normally back up shell. Why am i
> >> getting the error?
> >
> >> Much thanks for your help.
> >
> >
> > [...]
> >
> > Cheers, Denis
> >
> >
> > _______________________________________________ Dar-support mailing
> > list [email protected]
> > <mailto:[email protected]>
> > https://lists.sourceforge.net/lists/listinfo/dar-support
> >
> >
> >
> > _______________________________________________ Dar-support mailing
> > list [email protected]
> > https://lists.sourceforge.net/lists/listinfo/dar-support
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl65pmAACgkQCDGwvQPY
> sYKeWw//XlpCIxSwG2AZfk4JW3yS0yGEkU8PJV7dXS0++AdQan4eLPROlEEDEH79
> Z1DBICQhCx7IHv05WYC4UU9i60uzYdtcbWZy47/rFrpNil1lm0KsPSYdW9nSITq8
> roAFMBZX3Xg726LfAmzxhd+j/mDlV0VT/9f5bHojDNg/PZw8mXdAmm/Brol3oH5S
> BFNw/3rAdbnOKH/5uM/u2M6vSXwzmPodGCb/ML4AT2tdxjC/YbUqkDUmmdID87YS
> PLtUIexz2ePOGhdTi+Zi3I4md5IAkTb2H7F4aDL+bf7CdIQXWIjt4eqM78juGn8s
> BnALp/03kKScoF0UEqIuM1yKepppn5c3mBBdOuHuHx3E6Fd/sj61oZRHKVHlu/xo
> YprfqhD9GjSFWbo/T2kq8o9/OnsyBSn4kWyv/cDhv/Ct54CtnDYANfgNDkDl0Vkv
> CIU6181g2AgaMdwo/80VXg0y/IkP7ibA7zvt2yNl4G8Y0BJXQrbT28zMu7LD6QH0
> QZV2zJyw1+TPBYZM779472Oxxt+4y4VOMRuQhAk4LYP3ygiKtlS6H/i1UVtIYZgw
> 6zDdMfH8f0pqRkM1MQy1BRGqkyCtfDK7wd8sE+JfJPdecknf0k4oDeV9SyaB5EM0
> nutkI6qZjVdGfVJxqxxm8OhfSdFCN4xgqJ8tH0KdXPw9biHLfP4=
> =fGp7
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Dar-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-support
>

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support