Re: Dar on LTO

Dear Petr,
I thought that my use case is exotic... ;)

For now I cannot help with dar itself, as I pointed out I'm on the beginning of that journey, but I believe that I could help with some ideas.

From what I understand you're facing two major problems:
1. Packages are large (2.5TB) thus difficult to handle
2. Tape managment is a pain without knowledge what is stored on which tape

1)
Maybe I'm missing something but I wonder, if there is a reason why you're using packages of exact tape size?
I mean, I would pack that data into something much smaller i.e. 100GB per package/slice. It won't hurt much the tape nor performance yet it will allow you to better manage that data.

2)
I assume that your tapes have some kind of marks or even barcodes as they do in tape libraries.
Again dar_manager could be helpful in this case, because the database is showing you the path of archive.
The idea is:
What will happen if you will create symbolic link to tape device, named after the tape ID (or barcode or whatever) then write archive to that symlink? When you'll perform a `dar_manager -B "$database" -A "$dar_archive"` in dar_manager listing you would see a path pointing to archive, and my guess is it will show you the symlink name which would be your tape ID. By doing so you will be able to easily search for data with dar manager, and even more easily determiny which archive on which tape owns the file. 
Even if the symlink trick won't work, you can use the `dar_manager -p` option and rename the path in a manner that will help you identifying the tape. In either case, you will be able to discover data without touching tape, and identify correct tape that you need.

Best regards,
Patryk.

On 23/12/21 06:37AM, Petr Skoda wrote:
[-- Type: text/plain; charset=utf-8, Encoding: quoted-printable, Size: 5.2K --]
> Dear Patryk, John and Denis
> 
> 
> 
> I am continuing from the mail originally under another subject .
> 
> 
> 
> 
> So my problem with LTO is such:
> 
> 
> 
> 
> I have Tandberg LTO 6 (2.5TB uncompressed) without robot (only frive)
> connected to SAS card in a older server (Xeon E5-2620 v3 @ 2.40GHz 24 cores,
> 128GB RAM) . I have a number of zfs RAIDs (RAID1) on HDD with about 40TB of
> science (astronomical) data. - results of some machine learning experiments,
>   tens of millions of small files - spectra and also larger images - tens of
> MBs large )
> 
> 
> 
> 
> I am also making mirror of big spectra surveys - so 10+ mil of files about
> 100kB large are added 2-3 x per year...
> 
> 
> 
> 
> When I tried to make a full backup of this wealth it was challenging ...
> 
> The logical idea was to use combination of tar and dd ... I tried to split
> the big tar to 2.5TB slices etc ... The machine with LTO was on another 
> computer connected by 1GB net... 
> 
> I wanted to compute md5 sum and store it as the first file . All of this 
> gradually failed (impossible to finish the backup during several days ....
> just computing md5 took a week) 
> 
> 
> 
> 
> Then I have discovered dar .  I tried to use direct pipes on IP ports to
> send the data to the machine with LTO ... slow - without the big cache
> impossible.
> 
> 
> 
> 
> So final results of my experiments (I will shorten it) :
> 
> the LTO drive must be on the local computer. There must be a big cache
> (several TB) on RAID 0 (to speed the transfer ) ....
> 
> All the data must be prepared on that cache and then written to the tape as
> a single file (the machine should not seek small files ) ...
> 
> 
> 
> 
> Another problem is the block size .... after some tests it seems that 32MB
> or larger is needed (if using tar cvf the the default is 10240 bytes ...   
> recommended parameter tar cvfb 512 is minimum - or dd if=file of=/dev/nst0
> bs=256k ) but the larger the better ...
> 
> 
> 
> 
> The compression is a question ....  apparently the compression switched on
> LTO (must be special version of mt task supporting compression) can cope 
> with the data flow easilly - and dar_split (thanks Denis  for the FAQ about
> it) can detect the end of media and wait for human intervention.
> 
> 
> 
> 
> So  far I had to end my experiments (temporary)  creating a big dar archives
> on the RAID0 cache (20TB) and then send it to tape using the command 
> 
> 
> 
> 
> cat mybigbackup.1.dar | dar_split split_output /dev/nst0 
> 
> 
> 
> 
> and I read it 
> 
> 
> 
> 
> dar_split split_input /dev/nst0 | dar -l - -0 
> 
> 
> 
> 
> (-l is for listing - checking) 
> 
> 
> 
> 
> I tried to play also with slices (for John) and compress them by gzip bzip
> etc ... 
> 
> Very slow .... 
> 
> Every backup was split into 2.5TB slices (I have switched the LTO
> compression off) and the slices were directly written by dd 
> 
> I also played with mbuffer  - but it seems not to help in cases when you
> want to seek files ... When the file (2.5TB) is on cache, it makes no sense,
> as the dd directly to tape seems to be faster ....
> 
> 
> 
> 
> I did not yet use dar for differential backups (I use it only for full large
> backups) 
> 
> so far. I would like to play with dar_manager  (I already tried to play with
> catalogues).
> 
> 
> 
> 
> But now I would like to return to the dar again in a more advanced usage 
> pattern.
> 
> My wish is to have a number of slices and the separate catalogue, which will
> tell me which slices (i.e tapes) I should use to get a small subset - e.g.
> directory with one observing night ...
> 
> 
> 
> 
> So the no-last-slice would be nice - but it does not work in sequence mode .
> ..
> 
> In addition - having a lot of tapes with different contents (e.g. part
> written by dd, tar  ..) it would be excellent to be able to read as a first
> file of each tape  and see it is a dar slice number xxx from large backup
> named (e.g. /home/bigdata) .
> 
> But as I understand it is currently not possible to read from any slice of
> dar the metadata directly.
> 
> 
> 
> 
> So this is in short my current experience with LTO and dar.
> 
> 
> 
> 
> I must also emphasize that I do not want to have tape written by some
> enigmatic format of any tool as bacula. Simple after years I will load the
> tape, use a dd if=/dev/nst0 of=file bs=512 (or larger) . And from the file I
> will see the format, contents etc ....
> 
> That would be optimal.
> 
> 
> 
> 
> All the recommendations about better housekeeping are nice but it practice
> you have tens of tapes after years and to remember what is on it is
> difficult. The catch is in LTO function - if you do not know how to read it
> (even the correct block size) the tape may look as empty ... gives reading
> errors etc ....
> 
> 
> 
> 
> So I am looking for getting advice how to optimally use dar for storing 
> large amount of data  (so far all tricks etc work only on small backups, but
> the 2.5TB file which is being written in a optimal way about   140MB/s 
> (with compression 220 MB/s peak) takes about 4 hours / slice - it is time 
> consuming to try various dar options, combinations of mbuffer etc ....
> 
> 
> 
> 
> I have also notice that dar_split can replace mbuffer (Denis claims it) and
> has even buffer size a speed limitation parameters ....
> 
> But my experience says it is best to leave it on default.
> 
> 
> 
> 
> Best 
> 
> 
> 
> 
> Petr
> 
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
> 

[-- Alternative Type #1: text/html; charset=US-ASCII, Encoding: quoted-printable, Size: 6.0K --]