Re: Dar on LTO

Denis Corbin <[email protected]> Fri, 22 Dec 2023 21:23:34 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

On 21/12/2023 06:37, Petr Skoda wrote:
> Dear Patryk, John and Denis

Hi Petr, Hi all,

> 
> I am continuing from the mail originally under another subject .
> 
> So my problem with LTO is such:
> 

[...]

> So  far I had to end my experiments (temporary)  creating a big dar 
> archives on the RAID0 cache (20TB) and then send it to tape using the 
> command
> 
> cat mybigbackup.1.dar | dar_split split_output /dev/nst0
> 
> and I read it
> 
> dar_split split_input /dev/nst0 | dar -l - -0
> 
> (-l is for listing - checking)
> 
> I tried to play also with slices (for John) and compress them by gzip 
> bzip etc ...
> Very slow ....

I don't remember all the details of your discussion last year, please 
apologies if what follows is a duplicate:

lz4 as well a zstd seems to be much faster compression algorithm than 
Bzip2 and gzip2, though they both request CPU resource, of course. You 
can mitigate this to fit your available CPU power by reducing the 
compression level for any of these algorithms. Last, another option with 
dar 2.7.x is the possibility to compress per block to get parallel 
compression. Thus, you can leverage more or even all the cores of your 
processors for compression (and encryption). See -G option.

> Every backup was split into 2.5TB slices (I have switched the LTO 
> compression off) and the slices were directly written by dd
> I also played with mbuffer  - but it seems not to help in cases when you 
> want to seek files ... When the file (2.5TB) is on cache, it makes no 
> sense, as the dd directly to tape seems to be faster ....
> 
> I did not yet use dar for differential backups (I use it only for full 
> large backups)
> so far. I would like to play with dar_manager  (I already tried to play 
> with catalogues).
> 
> But now I would like to return to the dar again in a more advanced usage 
> pattern.
> My wish is to have a number of slices and the separate catalogue, which 
> will tell me which slices (i.e tapes) I should use to get a small subset 
> - e.g. directory with one observing night ...
> 
> So the no-last-slice would be nice - but it does not work in sequence 
> mode ...

in sequential read mode, dar reads from the beginning up to the end of 
the archive, exactly the same way as tar does. The last slice is thus 
always asked at the end of the process. The no-last-slice is only 
interesting in direct access mode.

> In addition - having a lot of tapes with different contents (e.g. part 
> written by dd, tar  ..) it would be excellent to be able to read as a 
> first file of each tape  and see it is a dar slice number xxx from large 
> backup named (e.g. /home/bigdata) .
> But as I understand it is currently not possible to read from any slice 
> of dar the metadata directly.

What is lacking here is not a feature in dar but a filesystem. 
"/home/bigdata" is not stored in the slice when relying on disks, it is 
part of each file's metadata, which data-structure (directory tree, 
inodes, extended attributes, and so on) is provided by a filesystems.

Without filesystem (like LTFS) it is still possible to write a text file 
at the beginning of a tape describing what follows after the next tape 
mark(s). This would be equivalent to writing stickers on the tapes and 
would have to be done manually and outside dar/tar/dd or any other tool.

> 
> So this is in short my current experience with LTO and dar.
> 
[...]

thank you Petr for sharing your experience
> 

Regards,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)