Re: Multiple slices on LTO6 tape

Dear Denis,




thank you for an  excellent support. After I have returned from a short 
vacancy I have found not only the answers to my questions but also a new FAQ
item . Excellent.

I will check all of this and ask (for sure) more questions .

Best regards,




Petr Skoda



---------- Původní e-mail ----------
Od: Denis Corbin <[email protected]>
Komu: [email protected]
Datum: 13. 5. 2022 22:26:22
Předmět: Re: [Dar-support] Multiple slices on LTO6 tape 
"Petr, 

please have a look at the new FAQ here: 
http://dar.linux.free.fr/doc/FAQ.html#tapes 

let me know if you find any inaccuracy or met any problem using LTO with 
dar, we will update the FAQ accordingly. 

Cheers, 
Denis 

Le 13/05/2022 à 18:36, Denis Corbin a écrit : 
> Le 13/05/2022 à 10:40, Petr Skoda a écrit : 
>> Dear DAR experts, 
> 
> Hi Petr, 
> 
>> 
>> I am getting stuck with usage dar on LTO6 tapes. After reading a lot of 
>> documentation and also archive of this list (and FAQ of course) 
>> I am not able to find solution for my setup. 
>> 
>> I have a machine called lto with connected LTO6 SAS tape (linux 10). 
>> Using the examples from your docs using netcat and dar_xform 
>> I have created from the remote machine multiple slices called 
>> datas.1.dar datas2.dar datas.3.dar. The first two are 2.5 TB (exactly 
>> 2500000000bytes large the no 3 is smaller. 
>> 
>> I have saved each slice to one tape using  dd if=datas.1.dar 
>> of=/dev/nst0 bs=256k etc... 
>> I still have the 3 slices on my staging disk (taking almost 7TB) . But I 
>> am not sure how to list the content , check the consistency and finally 
>> extract without needing such large space in the future. 
> 
> in the mode you have been using, if you want to list the backup content, 
> dar will only need the last slice. This will lead you to read the whole 
> 3rd slice from tape. Better you can use instead an isolated catalogue of 
> the backup on tape (something you can do on-fly or afterward). An 
> isolated catalogue is usually small an does not require to be backed up 
> to tape as you can recreate one from a backup at any time. 
> 
> example to create a isolated catalogue: 
> dar -C isolated -A backup -z ... 
> 
> If you want to extract a single file from backup, dar will ask the last 
> slice, then the slice where the file/tape to restore resided. But if you 
> restore with the help of an isolated catalogue, dar will only need the 
> slice where is located the file to restore. 
> 
> example to restore with the help of a catalogue 
> dar -x backup -A isolated ... 
> 
> 
>> I would like 
>> simply to restore only some files in the future getting instruction 
>> which tape to use and than extract  Ideally using some pipes combined 
>> with dd if=/dev/nst0 bs=256k |  dar something -x - 
>> However dar requires all slices together - I am not able to use only 
>> e.g. datas.2.dar to list only its contents. 
> 
> You could better use dar+dar_split instead, if this suits your need. The 
> advantage is that it will not need you any extra storage to restore or 
> list the backup, but will act a bit like tar, reading the tapes from the 
> first toward the last up to the point the file's metadata and data to 
> restore are reached. 
> 
> Note that you cannot use dar_split at reading time if you have not used 
> it at creation time. when using dar+dar_split, dar creates a single 
> slice, slice that dar_split (as you can guess from its name) splits over 
> different tapes. Thus, at reading time, dar expects a single slice and 
> not the concatenation of two or more slices, which is what dar_splits 
> does at reading time (concatenating the content of several tapes). 
> 
> However you can convert a splited backup to a single sliced backup using 
> dar_xform and then use dar_split: 
> 
> dar_xform backup - | dar_split split_output /dev/tape 
> 
> note that if you had an isolated catalogue from 'backup' it stays a 
> valid isolated catalog for the single sliced backup generated by 
> dar_xform, that dar_split has written to several tapes. 
> 
>> 
>> When starting with dar I expected that in sequential mode with some 
>> flags (even the -al does not work here) I will be able to scan what is 
>> on given slice and get something from it. 
> 
> you can get file content per slide using the -Tslice option while 
> listing an multi-sliced backup (or an isolated catalogue from this 
> backup). But you lose this ability when using dar with dar_split as from 
> dar stand point, there is only one slice. 
> 
>> But now having more tapes I am even not able to check what is on the 
>> tape (if it is slice 2 I am not able simply to use 
>> dd if=/dev/nst0 bs=256k |dar -0 -l  -   to see just for orientation part 
>> of the listing of content. 
> 
> there is not really a table of content per slice, a sliced backup is 
> still a coherent backup with an table of content at the end of the slice 
> set. 
> 
>> 
>> IMHO many people fighting with LTO storages will find dar soon by 
>> googling and then will be wondering exactly as I am how to use it. 
> 
> So I understand, I have to add some FAQ or the like about LTO use with 
> dar, right? 
> 
>> All tutorials are about small files but when you get to TBs and having 
>> not space for storing all slices  (say 10-20TB of full backup) 
> 
> Dar is not expecting to have all slices available at any time, you can 
> use -p option to pause after each (n) created slice(s) and do what is 
> needed, like move the produced slices to tape then remove it, before 
> having dar continuing its work. At reading time, dar will ask for the 
> missing slice and pause, you can then obtain it from tape and let dar 
> continue once the slice is available. 
> 
> Of course you can automate this by mean of -E option, so dar will not 
> pause and you can batch the whole backup process restoration process (if 
> you have a robot to physically change tapes... ;) ...) 
> 
> 
>> everything is slow and time consuming (just listing from my 7TB 3 slice 
>> the content without sequential mode takes several  minutes  to appear)  
> 
> 
> What is long is the time to copy (tape and disk I/O). dar_split should 
> obiously gives faster execution time as this intermediate step is removed.

> 
> Isolated catalogues can still help for listing content (and is also nice 
> to have to backup the internal catalog of a backup, in case of media 
> corruption). 
> 
> 
>> and learning the correct options is almost imposible using tape and 
>> pipes by trial . 
> 
> you can use named pipes and loopback mounts to simulate small tapes and 
> validate your dar commands and backup scenario: 
> 
> In a first terminal: 
> 
> for x in 1 2 3 ; do 
> dd if=/dev/zero of=tape$x bs=1024 count=11264 
> mkfs.ext4 tape$x 
> done 
> 
> mkfifo pipe 
> mkdir mnt 
> 
> mount -o loop tape1 mnt 
> cat < pipe > mnt/data 
> 
> 
> In a second terminal: 
> 
> dar -c - ... | dar_split split_output pipe 
> 
> When the tape1 filesystem is full, dar_split pauses, you can then 
> change the "tape" from the first terminal: 
> 
> (the 'cat' command has ended) 
> umount mnt/data 
> mount -o loop tape2 mnt 
> cat < pipe > mnt/data 
> 
> then hit return in the second terminal to let dar_split continue 
> 
> and so on. 
> 
> 
> 
>> So I would appreciate a new FAQ entry explaining how to use LTO tapes 
>> together with dar  for doing full and incremental backup when not having 
>> place to feed all slice to a disk to be able to work with such large 
volume. 
> 
> Yes, you are right, this is missing in FAQ, I will add this. In the 
> meanwhile I hope these explanation will let you go forward 
> 
>> (of course the final goal is to have tape on lto machine but extract 
>> through pipes (and perhaps mbuffer) to remote machine back. 
> 
> with dar_split you do not need mbuffer if you just want to rate limit 
> the throughput (see its -r option) 
> 
>> 
>> 
>> Best regards, 
>> 
>> Petr Skoda 
>> 
>> 
> 
> Cheers, 
> Denis 
> 
> 
"