Re: Multiple slices on LTO6 tape

Dear Denis, 





I am afraid we are running in a circle from my first email.

But perhaps now after explaining many things we could find the problem in  
my misunderstanding ;-)

And maybe the reference to this mail thread will help other people to 
understand better the behaviour of dar.





I will try to summarize what I currently know from the dar docs ( and some 
discussions on mailing list.

(please try to confirm my thoughts - perhaps it may be good point to expand 
on current FAQs a liitle more)





1) dar includes some special sequences (you call them tape marks even if it 
is on disk)  (see http://dar.linux.free.fr/doc/Features.html under Data 
Protection)




"It can also make use of tape marks that are used inside the backup for 
sequential reading as a way to overcome catalog corruption. The other vital 
information is the slice layout which is replicated in each slice and let 
dar overcome data corruption of that part too. As a last resort, Dar also 
proposes a "lax" mode in which the user is asked questions (like the 
compression algorithm used, ...) to help dar recover very corrupted archives
and in which, many sanity checks are turned into warnings instead of 
aborting the operation."




So there is even the layout of all slices written in each slice. Thus dar 
knows what is needed

(and if missing it asks for it)


2) at other places is written (also in a previous mails) that the catalogue 
is in the  last slice.

But when I want to list its contents in sequential mode - it requires the 
first one !




the -al option is intended for a lax mode, where I expect the ability to 
read what is presented from any slice (here is probably my misunderstanding 
- I expect the tape marks and special sequences allow to read the part of 
the directory on given slice)

But when I try it (see below) it fails....




Also in the FAQ question "
Why dar became much slower since release 2.4.0?"




are introduced escape sequence  - IS IT DIFFERENT from the TAPE MARK  ?  
Later it is written 


-at option, which suppress "tape marks" (just another name for escape 
sequences),

So probably it is the same.

So it is explained 





"Espetially (here is the typo) to be able to read dar backup through pipes 
in sequential mode, dar inserts so-called "escape sequence" to know for 
example when a new file starts. This way dar can skip to the next mark upon 
backup corruption or if the given file has not to be restored."




OK - so are those escape sequences used just for jumping over the bad part 
or is there written as well the name and attributes of the file which 
follows ????

In not - what is written and where to allow the listing of the catalogue in 
the sequential mode ????








I have made various experiments - just without the tape 


Having dir /home/skoda about 700MB I created 3 slices  on disk (not on home 
but /mnt...




dar -c myback -R /home/skoda -s 300M

I have three slices myback.1.dar, myback.2.dar and myback.3.dar

Now I will do 





dar -l myback  - OK the listing is here fastly

dar -l myback -0   (I expect -0 is --sequential-read  - I am lazy to write 
it ;-)

The dir is given but slowly   - I expect now dar uses the tape marks 
(special sequences ) to construct the catalogue.




NOW I move 2 slices somewhere - leaving only myback.1.dar available

dar -l myback  complains about  the last slice 





dar -l myback 
The last file of the set is not present in file:///mnt/work/skoda/ANTARES-
2022/DAR-TEST , please provide it. [return = YES | Esc = NO]




I will copy it in another terminal window, press enter and the listing is OK

(Please note, I did not need the slice 2 for listing)




Now I keep only slice 3 in my working directory

dar -l myback    gives immediately the listing 





while the sequential mode is missing the slice 1 


dar -l myback -0
myback.1.dar is required for further operation, please provide the file. 
[return = YES | Esc = NO]





When I copy it there it continues in the listing but after it requires the 
slice2 


03:37 2022    DAR/dar-2.7.5/src/libdar/.libs/fichier_libcurl.o


...


[Saved][ ]       [---][ 100%][X]  -rw-r--r--   skoda    tape    105 kio    
Mon May 30 21:03:37 2022    DAR/dar-2.7.5/src/libdar/.libs/i_entrepot_
libcurl.o
[Saved][ ]       [---][ 100%][X]  -rw-r--r--   skoda    tape    308 kio    
Mon May 30 21:03:38 2022    DAR/dar-2.7.5/src/libdar/.libs/delta_sig_block_
size.o
myback.2.dar is required for further operation, please provide the file. 
[return = YES | Esc = NO]




When added this piece it finishes the listing OK




So far good   - in non sequential mode dar goes to the last slice and reads 
all the contents (I mean catalogue) here .

In sequential it needs all slices to to gather information about files and 
probably does not read the catalogue on the last slice.

-------------------------------------------------------------



Then we have part  about -Tslices 





It is very shortly mentioned in the FAQ named 


Why cannot I test, extract file, list the contents of a given slice from a 
backup?




It seems to be useful only when a subtree of the backup is needed to tell 
you what slices should be involved.  Without the filter - e.g. -g subdir 


It seems redundant - as it returns " all  displayed  files have their data 
in slice range ... And ALL slices are said here.




But it works (for me ) only in NON-SEQUENTIAL mode. This implies it needs 
ALWAYS the LAST slice.

On disk it is easy 


dar -l myback -Tslice  -g fortape





It lists all files from "fortape" directory and tells me  "All displayed 
files have their data in slice range [2]"




If trying with sequential mode 





dar -l myback -Tslice -0 -g fortape   (the fortape is 


It crashes 





Slice(s)|[Data ][D][ EA  ][FSA][Compr][S]|Permission| Filemane
--------+--------------------------------+----------+-----------------------
------
Final memory cleanup...
FATAL error, aborting operation: slicing focused output is not available in 
sequential-read mode



(Please note I have  all 3 slices available so can do easily 


dar -l myback  -0 -g fortape

and even all backup list 


dar -l myback  -0




So my conclusion - Tslices will not work in sequential mode and so cannot be
used on tapes or any pipe through std input output (-)




---------------------

Now the objection to the last part of your mail (below) 





"last, thanks to the lax mode you can, in a way operate data per slice 
(listing/restoring/...),"




I do not understand the term "per slice" 


And with following statement I am going back in cycle to previous mails and 
discussion.




 I CANNOT DO ANYTHING WITH a SINGLE SLICE even in lax mode (-al) IN CASE IT 
IS NOT THE LAST OR FIRST




(in sequential - I need the first to start with in non-sequential the last 
file to read from).




But if using the tapes (or pipelines) I MUST USE sequential mode. So even 
the tape with the last slice will not help me.

I will simulate the input from sequential pipe without tape now:





dar -l -  -0 -al <myback.3.dar LAX MODE: Failed to read the archive header's
format version.
LAX MODE: Please provide the archive format: You can use the table at 
http://dar.linux.free.fr/pre-release/doc/Notes.html#Dar_version_naming to 
find the archive format depending on the release version, (for example if 
this archive has been created using dar release 2.3.4 to 2.3.7 answer "6" 
without the quotes here):





First the link http://dar.linux.free.fr/pre-release/doc/Notes.html does not 
exist ! So I can only geuss what to type if I have 2.,7.5 


I tried 6 or 7 . Always I get 





LAX MODE: Unknown compression algorithm used, assuming data corruption 
occurred. Please help me, answering with one of the following words "none", 
"gzip", "bzip2", "lzo", "xz", "zstd" or "lz4" at the next prompt:



I do not have compression so answer is "none"




It follows a number of lines 


LAX MODE: ........


LAX MODE: found unknown catalogue entry, assuming data corruption occurred. 
Skipping to the next entry, this may lead to improper directory structure 
being restored, if the corrupted data was a directory
Final memory cleanup...
Error while listing archive contents: Cannot open catalogue: incoherent 
catalogue structure





If doing the same on slice 2 I get 





dar -l -  -0 -al <myback.2.dar 





LAX MODE: Unknown compression algorithm used, assuming data corruption 
occurred. Please help me, answering with one of the following words "none", 
"gzip", "bzip2", "lzo", "xz", "zstd" or "lz4" at the next prompt:none
Unknown crypto algorithm used in archive, ignoring that field and simply 
assuming the archive has been encrypted, if not done you will need to 
specify the crypto algorithm to use in order to read this archive
Final memory cleanup...
FATAL error, aborting operation: Badly formed "infinint" or not supported 
format





It does not ask for archive format and answers the "none" itself




Only if I read the first slice it lists part of the content (I do not need -
al here - the result is the same) 





To wrap up: I am still confused what happens here - But I guess that it is 
the proof that the individual slices have allegedly thge escape sequences 
(Tape marks) embedded, but dar is not  able to operate with them . So IMHO 
the only hope is the solution of the symlink after you succeed to repair the
behaviour.

I would suggest to allow the reading of the exact filename of the slice with
-l   (not the generic) after using some special option  (as -lslice) that 
would combine the serial and lax or something similar.




I have also noticed that the output of dar is not compatible with other 
linux commends as I cannot redirect the listing to e.g. word count 


ls -l backup |wc -l Probably the output is not stdout ...

It would be nice to have this standard unix filter behaviour.




And BTW  on the doc page 


http://dar.linux.free.fr/doc/Tutorial.html

is bad link going to nowhere from  


section 9  "restoration with dar "





Best regards Petr 








" 
But if by "reading" you just mean listing the content per slice, there 
is already the possibility to do this: 

dar -l <backup> -Tslice 

last, thanks to the lax mode you can, in a way operate data per slice 
(listing/restoring/...), but that's not a straight forward solution and 
stays a last resort option. 

Cheers, 
Denis 
"