Re: Dar on LTO

Denis Corbin <[email protected]> Wed, 18 Sep 2024 22:04:03 +0200
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 17/09/2024 00:37, [email protected] wrote:
> Dear Petr, Denis,

Hi Jonathan,

> 
> I've followed in 2022 your thread about using DAR for backups on tape 
> with much interest.
> But never got past the experimental stage.
> 
> As you I like to pick it up again.
> And although I've slightly less demanding jobs (only 2x 8Tb compressed 
> and a few million files)
> I still need several tapes to make a full backup and that will increase 
> in future.
> 
> One part of my data consists of a lot of hard-links made by 'dirvish' in 
> differential mode.
> That makes for a very slow data rate using tar for backup to tape with 
> shoe shining of the drive as consequence.
> I think DAR is a modern equivalent of tar and I like the catalogue 
> option to mentioning one so I really want to use it.
> 
> I have the same hardware (Tandberg LTO6) with a BTRFS raid1 and a ZFS 
> raid1 on spinning rust.
> In my experiments I noticed that going beyond tar -b256 didn't gave any 
> improvements.
> Using mbuffer I also tried a lot of different buffer sizes but because 
> of the data part with a lot of hard-links
> it always emptied the buffer and the tape drive stopped. And big buffers 
> take longer to refill so all in all longer backup times.
> The maximum data rate of this drive (400Mb/s compressed) is awful and 
> only reachable with SSD's or better (and probably big files).
> 
> For now I'm thinking of piping dar into mbuffer something like (very 
> rudimentary) :
> 
> /usr/bin/mbuffer <lot of options> -R160M -H -f -o /dev/st0 < 
> <(/usr/local/bin/dar -c - -zlzh4 -R${rootpath} -@${catalogue} 
> |/usr/local/bin/dar_split split_output )

I'm not sure of the syntax: mbuffer << (dar | dar_split)
I would have naively thought about that instead:
	dar -c - ...| mbuffer ... | dar_split split_output /dev/st0

and at reading time:
   dar_split split_input /dev/st0 | mbuffer ... | dar -c - --sequential-read

> 
> I can remember that there was a discussion 'End of Tape' was not  > reaching dar_split this way because of mbuffer.

I couldn't find this discussion in the mailing-list archive... I have 
probably overlooked it somewhere. But if dar_split is in direct 
"contact" of the device inode (/dev/st0 here) this should not occur, 
while still having mbuffer controlling the speed of the byte flow to or 
from the tape, as above.

What you refer to may rather be the fact that dar_split, when reading 
from a tape, has no intelligence about the dar format and does not know 
when the archive has been completely read, so it will still wait for the 
next tape. Thus, you had to interrupt it manually when dar has finished. 
There is now a -c option to dar_split thanks to which you give the max 
number of tape to read from. And once this number of tape has been 
exhausted, dar_split ends nicely.

> I like to use mbuffer because of it's logs showing the data rate.
> There are remarkable obervations to make looking at those logs and would 
> regret loosing that possibility.

to my point of view, mbuffer is not incompatible with dar nor dar_split. 
So if you need mbuffer for its reach feature set, just use it with dar 
and dar_split. Still you may need to configure dar_split about the block 
size used to read or write data (see its -b option) according the the 
expected block size supported by your device.

> 
> As you can see I compress the data outside the drive and limit the write 
> speed to 160Mb/s. Perhaps I will have to lower this even more.
> The idea is that by compressing it beforehand and using a buffer shoe 
> shining is reduced to a minimum.

note about compression: since release 2.7.0 dar can perform compression 
leveraging multiple CPU cores (using multi-threading). But as 
compression is still performed file per file, the performance gain will 
depend on the average file size under backup and in consequence the 
compression block size used: Selecting a too small compression block 
will lead to a thread overhead that can lead to poor compression 
performances, while selecting a too large compression block will only 
lead files larger than this block size to use more than one CPU core, 
but compression performance should be slightly equivalent to the one you 
get not using multi-threading (see -z and -G options for details).

> 
> My question is is there a good example how to use mbuffer, dar and 
> dar_split in combination with a tape drive ?
> Is it possible to limit the size of dar_split chunks ?

It is possible to limit the amount of byte per system call (= size of 
the block the device will have to read or write at once, see -b option 
as mentioned above) and to limit the byte rate (see -r option). But if 
you plan to use mbuffer, you should avoid limiting dar_split rate, this 
will be done by mbuffer being piped between dar and dar_split as 
decribed above. And with mbuffer you would still have all the 
stats/indicators you like.

note that I have no more tape in my environment (and this for decades! 
dar came from that need to efficiently rely on disks :) ) Thus, I let 
those having more experience with tape drives to amend or complement my 
suggestions here.

> 
> Thanks and with kind regards,
> 
> Jonathan.
> 
> 
> 

Regards,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)