Dar on LTO

Dear Patryk, John and Denis



I am continuing from the mail originally under another subject .




So my problem with LTO is such:




I have Tandberg LTO 6 (2.5TB uncompressed) without robot (only frive) 
connected to SAS card in a older server (Xeon E5-2620 v3 @ 2.40GHz 24 cores,
128GB RAM) . I have a number of zfs RAIDs (RAID1) on HDD with about 40TB of 
science (astronomical) data. - results of some machine learning experiments,
  tens of millions of small files - spectra and also larger images - tens of
MBs large )




I am also making mirror of big spectra surveys - so 10+ mil of files about 
100kB large are added 2-3 x per year...




When I tried to make a full backup of this wealth it was challenging ...

The logical idea was to use combination of tar and dd ... I tried to split 
the big tar to 2.5TB slices etc ... The machine with LTO was on another 
computer connected by 1GB net... 

I wanted to compute md5 sum and store it as the first file . All of this 
gradually failed (impossible to finish the backup during several days .... 
just computing md5 took a week) 




Then I have discovered dar .  I tried to use direct pipes on IP ports to 
send the data to the machine with LTO ... slow - without the big cache 
impossible.




So final results of my experiments (I will shorten it) :

the LTO drive must be on the local computer. There must be a big cache 
(several TB) on RAID 0 (to speed the transfer ) ....

All the data must be prepared on that cache and then written to the tape as 
a single file (the machine should not seek small files ) ...




Another problem is the block size .... after some tests it seems that 32MB 
or larger is needed (if using tar cvf the the default is 10240 bytes ...    
recommended parameter tar cvfb 512 is minimum - or dd if=file of=/dev/nst0 
bs=256k ) but the larger the better ...




The compression is a question ....  apparently the compression switched on 
LTO (must be special version of mt task supporting compression) can cope 
with the data flow easilly - and dar_split (thanks Denis  for the FAQ about 
it) can detect the end of media and wait for human intervention.




So  far I had to end my experiments (temporary)  creating a big dar archives
on the RAID0 cache (20TB) and then send it to tape using the command 




cat mybigbackup.1.dar | dar_split split_output /dev/nst0 




and I read it 




dar_split split_input /dev/nst0 | dar -l - -0 




(-l is for listing - checking) 




I tried to play also with slices (for John) and compress them by gzip bzip 
etc ... 

Very slow .... 

Every backup was split into 2.5TB slices (I have switched the LTO 
compression off) and the slices were directly written by dd 

I also played with mbuffer  - but it seems not to help in cases when you 
want to seek files ... When the file (2.5TB) is on cache, it makes no sense,
as the dd directly to tape seems to be faster ....




I did not yet use dar for differential backups (I use it only for full large
backups) 

so far. I would like to play with dar_manager  (I already tried to play with
catalogues).




But now I would like to return to the dar again in a more advanced usage 
pattern.

My wish is to have a number of slices and the separate catalogue, which will
tell me which slices (i.e tapes) I should use to get a small subset - e.g. 
directory with one observing night ...




So the no-last-slice would be nice - but it does not work in sequence mode .
..

In addition - having a lot of tapes with different contents (e.g. part 
written by dd, tar  ..) it would be excellent to be able to read as a first 
file of each tape  and see it is a dar slice number xxx from large backup 
named (e.g. /home/bigdata) .

But as I understand it is currently not possible to read from any slice of 
dar the metadata directly.




So this is in short my current experience with LTO and dar.




I must also emphasize that I do not want to have tape written by some 
enigmatic format of any tool as bacula. Simple after years I will load the 
tape, use a dd if=/dev/nst0 of=file bs=512 (or larger) . And from the file I
will see the format, contents etc ....

That would be optimal.




All the recommendations about better housekeeping are nice but it practice 
you have tens of tapes after years and to remember what is on it is 
difficult. The catch is in LTO function - if you do not know how to read it 
(even the correct block size) the tape may look as empty ... gives reading 
errors etc ....




So I am looking for getting advice how to optimally use dar for storing 
large amount of data  (so far all tricks etc work only on small backups, but
the 2.5TB file which is being written in a optimal way about   140MB/s  
(with compression 220 MB/s peak) takes about 4 hours / slice - it is time 
consuming to try various dar options, combinations of mbuffer etc ....




I have also notice that dar_split can replace mbuffer (Denis claims it) and 
has even buffer size a speed limitation parameters ....

But my experience says it is best to leave it on default.




Best 




Petr