Re: Dar on LTO

Hi,

I just thought I'd add my 2 cents here.

Let me start from a different place... Bacula/Bareos are backup programs
designed for tape backup.  They both always spool backup data to disk
before writing it to tape.  For several reasons, but mainly because of
the shoeshining problem.  Now Bacula/Bareos are much more heavyweight
programs, designed for scheduling backups of many hosts on a network
with a centralized catalog.  They're great for that, but if you don't
need that, dar's featureset is a lot nicer.

So, why don't you spool the dar backups to disk?

Now, you may not have enough space to spool the entire backup.  But you
could use the slice options to define a maximum spool size.  Then, add
--execute to your dar command to point to a script.  That script would
then flush a given slice to tape.  It could be as simple as dd, or as
complex as mbuffer.  Then after it's written to tape, the script would
delete it from disk and continue with the next slice.

This will make you wind up with multiple tape files for a single dar
backup.  There are pros and cons to that.  The cons are that you may
have a more complex tape structure, particularly if you were planning to
put multiple backup sessions on one tape.  On the other hand, you will
have an easy way to seek to a given slice, so it may make partial
restores faster.  You can use --execute to interact with mt to seek to
particular tape files and such during restore automatically.  You don't
need to use dar_split; just use dar --slice.

Now then, other advice...

How many GBs can you give mbuffer's buffer?  If you can use -m 10G or -m
20G, that would help reduce shoeshining.  Use -P 99 and mbuffer won't
start writing until the buffer is 99% full, minimizing stop/start
cycles.  Adding a lot of RAM to the system, and using mbuffer -m 40G -P
99 may well reduce shoeshining to acceptable levels easily enough.

I don't see why slowing down the writes to tape would help.  I'd think
that would be harmful.

Also, turn off hardware compression and use dar compression.  Then you
can exactly predict how much space remains on a tape.

If you get into really complicated tape mechanics, then you may be
better suited for something like Bacula/Bareos.  They automate these
things, including splitting backups over tapes, tape recycling schemes,
etc.  The cost is a lot more heavyweight system; you need a SQL database
to hold their catalog (their version of dar_manager, which is required
for them to operate).  You've got to set up and manage daemons, etc.
But if you are talking multi-tape daily backups or fitting multiple
backups on a single tape and keeping track of it all, maybe it's worth
it.

If, on the other hand, you're preparing an archive -- that is, a
long-term storage copy of the data that maybe you refresh once a year --
dar is going to be nicer and gives you a lot more flexibility on
manipulating the archive.


John

On Tue, Sep 17 2024, [email protected] wrote:

> Dear Petr, Denis,
>
> I've followed in 2022 your thread about using DAR for backups on tape
> with much interest.
> But never got past the experimental stage.
>
> As you I like to pick it up again.
> And although I've slightly less demanding jobs (only 2x 8Tb compressed
> and a few million files)
> I still need several tapes to make a full backup and that will increase
> in future.
>
> One part of my data consists of a lot of hard-links made by 'dirvish' in
> differential mode.
> That makes for a very slow data rate using tar for backup to tape with
> shoe shining of the drive as consequence.
> I think DAR is a modern equivalent of tar and I like the catalogue
> option to mentioning one so I really want to use it.
>
> I have the same hardware (Tandberg LTO6) with a BTRFS raid1 and a ZFS
> raid1 on spinning rust.
> In my experiments I noticed that going beyond tar -b256 didn't gave any
> improvements.
> Using mbuffer I also tried a lot of different buffer sizes but because
> of the data part with a lot of hard-links
> it always emptied the buffer and the tape drive stopped. And big buffers
> take longer to refill so all in all longer backup times.
> The maximum data rate of this drive (400Mb/s compressed) is awful and
> only reachable with SSD's or better (and probably big files).
>
> For now I'm thinking of piping dar into mbuffer something like (very
> rudimentary) :
>
> /usr/bin/mbuffer <lot of options> -R160M -H -f -o /dev/st0 < <
> (/usr/local/bin/dar -c - -zlzh4 -R${rootpath} -@${catalogue}
> |/usr/local/bin/dar_split split_output )
>
> I can remember that there was a discussion 'End of Tape' was not
> reaching dar_split this way because of mbuffer.
> I like to use mbuffer because of it's logs showing the data rate.
> There are remarkable obervations to make looking at those logs and would
> regret loosing that possibility.
>
> As you can see I compress the data outside the drive and limit the write
> speed to 160Mb/s. Perhaps I will have to lower this even more.
> The idea is that by compressing it beforehand and using a buffer shoe
> shining is reduced to a minimum.
>
> My question is is there a good example how to use mbuffer, dar and
> dar_split in combination with a tape drive ?
> Is it possible to limit the size of dar_split chunks ?
>
> Thanks and with kind regards,
>
> Jonathan.