Re: Multiple slices on LTO6 tape

Petr,

please have a look at the new FAQ here:
http://dar.linux.free.fr/doc/FAQ.html#tapes

let me know if you find any inaccuracy or met any problem using LTO with
dar, we will update the FAQ accordingly.

Cheers,
Denis

Le 13/05/2022 à 18:36, Denis Corbin a écrit :
> Le 13/05/2022 à 10:40, Petr Skoda a écrit :
>> Dear DAR experts,
> 
> Hi Petr,
> 
>>
>> I am getting stuck with usage dar on LTO6 tapes. After reading a lot of
>> documentation and also archive of this list (and FAQ of course)
>> I am not able to find solution for my setup.
>>
>> I have a machine called lto with connected LTO6 SAS tape (linux 10).
>> Using the examples from your docs using netcat and dar_xform
>> I have created from the remote machine multiple slices called
>> datas.1.dar datas2.dar datas.3.dar. The first two are 2.5 TB (exactly
>> 2500000000bytes large the no 3 is smaller.
>>
>> I have saved each slice to one tape using  dd if=datas.1.dar
>> of=/dev/nst0 bs=256k etc...
>> I still have the 3 slices on my staging disk (taking almost 7TB) . But I
>> am not sure how to list the content , check the consistency and finally
>> extract without needing such large space in the future.
> 
> in the mode you have been using, if you want to list the backup content,
> dar will only need the last slice. This will lead you to read the whole
> 3rd slice from tape. Better you can use instead an isolated catalogue of
> the backup on tape (something you can do on-fly or afterward). An
> isolated catalogue is usually small an does not require to be backed up
> to tape as you can recreate one from a backup at any time.
> 
> example to create a isolated catalogue:
> dar -C isolated -A backup -z ...
> 
> If you want to extract a single file from backup, dar will ask the last
> slice, then the slice where the file/tape to restore resided. But if you
> restore with the help of an isolated catalogue, dar will only need the
> slice where is located the file to restore.
> 
> example to restore with the help of a catalogue
> dar -x backup -A isolated ...
> 
> 
>>  I would like
>> simply to restore only some files in the future getting instruction
>> which tape to use and than extract  Ideally using some pipes combined
>> with dd if=/dev/nst0 bs=256k |  dar something -x -
>> However dar requires all slices together - I am not able to use only
>> e.g. datas.2.dar to list only its contents.
> 
> You could better use dar+dar_split instead, if this suits your need. The
> advantage is that it will not need you any extra storage to restore or
> list the backup, but will act a bit like tar, reading the tapes from the
> first toward the last up to the point the file's metadata and data to
> restore are reached.
> 
> Note that you cannot use dar_split at reading time if you have not used
> it at creation time. when using dar+dar_split, dar creates a single
> slice, slice that dar_split (as you can guess from its name) splits over
> different tapes. Thus, at reading time, dar expects a single slice and
> not the concatenation of two or more slices, which is what dar_splits
> does at reading time (concatenating the content of several tapes).
> 
> However you can convert a splited backup to a single sliced backup using
> dar_xform and then use dar_split:
> 
> dar_xform backup - | dar_split split_output /dev/tape
> 
> note that if you had an isolated catalogue from 'backup' it stays a
> valid isolated catalog for the single sliced backup generated by
> dar_xform, that dar_split has written to several tapes.
> 
>>
>> When starting with dar I expected that in sequential mode with some
>> flags (even the -al does not work here) I will be able to scan what is
>> on given slice and get something from it.
> 
> you can get file content per slide using the -Tslice option while
> listing an multi-sliced backup (or an isolated catalogue from this
> backup). But you lose this ability when using dar with dar_split as from
> dar stand point, there is only one slice.
> 
>> But now having more tapes I am even not able to check what is on the
>> tape (if it is slice 2 I am not able simply to use
>> dd if=/dev/nst0 bs=256k |dar -0 -l  -   to see just for orientation part
>> of the listing of content.
> 
> there is not really a table of content per slice, a sliced backup is
> still a coherent backup with an table of content at the end of the slice
> set.
> 
>>
>> IMHO many people fighting with LTO storages will find dar soon by
>> googling and then will be wondering exactly as I am how to use it.
> 
> So I understand, I have to add some FAQ or the like about LTO use with
> dar, right?
> 
>> All tutorials are about small files but when you get to TBs and having
>> not space for storing all slices  (say 10-20TB of full backup)
> 
> Dar is not expecting to have all slices available at any time, you can
> use -p option to pause after each (n) created slice(s) and do what is
> needed, like move the produced slices to tape then remove it, before
> having dar continuing its work. At reading time, dar will ask for the
> missing slice and pause, you can then obtain it from tape and let dar
> continue once the slice is available.
> 
> Of course you can automate this by mean of -E option, so dar will not
> pause and you can batch the whole backup process restoration process (if
> you have a robot to physically change tapes... ;) ...)
> 
> 
>> everything is slow and time consuming (just listing from my 7TB 3 slice
>> the content without sequential mode takes several  minutes  to appear) 
> 
> 
> What is long is the time to copy (tape and disk I/O). dar_split should
> obiously gives faster execution time as this intermediate step is removed.
> 
> Isolated catalogues can still help for listing content (and is also nice
> to have to backup the internal catalog of a backup, in case of media
> corruption).
> 
> 
>> and learning the correct options is almost imposible using tape and
>> pipes by trial .
> 
> you can use named pipes and loopback mounts to simulate small tapes and
> validate your dar commands and backup scenario:
> 
> In a first terminal:
> 
> for x in 1 2 3 ; do
>   dd if=/dev/zero of=tape$x bs=1024 count=11264
>   mkfs.ext4 tape$x
> done
> 
> mkfifo pipe
> mkdir mnt
> 
> mount -o loop tape1 mnt
> cat < pipe > mnt/data
> 
> 
> In a second terminal:
> 
> dar -c - ... | dar_split split_output pipe
> 
> When the tape1 filesystem is full, dar_split pauses, you can then
> change the "tape" from the first terminal:
> 
> (the 'cat' command has ended)
> umount mnt/data
> mount -o loop tape2 mnt
> cat < pipe > mnt/data
> 
> then hit return in the second terminal to let dar_split continue
> 
> and so on.
> 
> 
> 
>> So I would appreciate a new FAQ entry explaining how to use LTO tapes
>> together with dar  for doing full and incremental backup when not having
>> place to feed all slice to a disk to be able to work with such large volume.
> 
> Yes, you are right, this is missing in FAQ, I will add this. In the
> meanwhile I hope these explanation will let you go forward
> 
>> (of course the final goal is to have tape on lto machine but extract
>> through pipes (and perhaps mbuffer) to remote machine back.
> 
> with dar_split you do not need mbuffer if you just want to rate limit
> the throughput (see its -r option)
> 
>>
>>
>> Best regards,
>>
>> Petr Skoda
>>
>>
> 
> Cheers,
> Denis
> 
>