Re: Huge archive and last slice

Hi Petr,

A disclaimer - I have not used dar with tape libraries, but I have run
numerous other systems atop them in the past.

Some thoughts -

When using dar's fixed slices, that allows a lot of features such as
detached catalogs that can pinpoint which slice(s) a file are on.
dar_manager can use those and provide a database which you can use to
identify which tape you need.  If you use dar_split, you will be forced
to read each backup sequentially -- including all tapes up until the
point where it's stored -- for even small restores.

Since dar uses fixed-size slices, that implies that you would be better
off using dar's built-in compression, since otherwise you will be unable
to predict how big each slice should be (since compression at the
hardware level effectively means that every tape has a different and
unpredictable storage capacity).

You're on the right track with mbuffer.  If you have lots of RAM on your
system, you may be able to use a large buffer and a high -P to help with
shoeshining.  That high -P tells mbuffer to not start writing until the
buffer is nearly full (as opposed to its default of starting at once).
I am unsure if this applies after buffer exhaustion, but I hope it does.
The point wouldn't be to eliminate shoeshine completely, but to reduce
it to acceptable levels.  If you could guarantee, say, 40GB of solid
writes at once, that may be a good step.

Another option is to spool the data to disk, and then despool to the
tape.  Bacula/Bareos do this.  You can pass a script to --execute that
would handle the mechanics of writing a slice to the drive, and then
either prompting the user to change tapes or operating the changer
appropriately.  This makes the assumption that your disk source is fast
enough with sequential reads to saturate the tape (a mbuffer may be of
some use here too).  Often times, reading many small files or highly
fragmented files may be far slower than just streaming a big file out to
tape.

- John


On Mon, Dec 18 2023, Petr Skoda wrote:

> Dear Patryk,
>
> you have mentioned the tape library and cache system in relation to using dar.
> Would you mind to describe us more about your setup, configuration of dar , parameters of its
> command line or perhaps scripts you are using for handling the large amount of data ?
>
> Until recently I supposed I was the only one who tried to use dar together with LTO ULTRIUM
> drives. I have made a lot of experiements but so far I am not fully satisfied with the dar
> functionality on tapes. e.g. the sequential mode recovery from slices, whether to use fixed
> slices or  dar_split, if it makes sense the parameters for limiting data flow etc. (dar_split
> -b -r)
> I am also not sure whether it is better to use the LTO HW compression or not ... . I have made
> a lot of acoustic tests (hearing the LTO drive in open computer on my desk during various
> operations, and it seems the answer is not easy (to compress or not) .... Even the large
> buffers done by mbuffer will not prevent shoe-shining of the tape .
> So I am still looking for the best practice of using LTO  (drive or library) together with dar
> to assure optimal (easy, fast) storage of tens of TB from big arrays (well based on zfs RAIDs).
> So I hope that not only myself would benefit a lot from you optimal big data handling use case
> ....
>
> Thanks a lot
>
> Petr Skoda
> Astronomical Institute of the Czech Academy of Sciences.
>
> ---------- Původní e-mail ----------
> Od: Patryk Bełzak <[email protected]>
> Komu: need help compiling or using dar ? This mailing list is for you.
> <[email protected]>
> Datum: 13. 12. 2023 9:29:51
> Předmět: Re: [Dar-support] Huge archive and last slice
>
>  I guess it's also worth to mention `dar_manager`.
>  I'm also dealing with PB of scentific data in Hierarchical Storage Management solution, and
>  I'm considering scenario where I'll build a database on some nfs (so it could be accessed
>  by all systems I need, and getting some sort of high availability).
>  By doing so, searching for files and extracting them wouldn't require to recall large
>  amounts of data from tape library to cache filesystem.