Re: Using dar for archives to optical discs

Dear John and Denis,




I am really glad that John had opened the problem of large backup on 
multiple media . In fact I have been watching the ongoing discussion from 
time to time , but as I understand most of the problems with housekeeping of
multiple slices is being gradually solved by updates of DAR. 


I would like to revive again my original idea (discussed here about a year 
ago) of using LTO tapes or any serial-only access media in general.

I have made a year ago a full backup of about 30TB of astronomical images 
using dar_split and dar_xform but after some experiments I am not happy with
the current situation. And so I would like to use the discussion about 
optical media storage to motivate you to think about the tapes again.




Denis already answered a lot of my suggestion last year , but most of the 
solutions were quite difficult to use  in practice.

So let me just  quickly summarize :




The LTO connected to SAS interface needs a disk array to feed it (I use zfs)
and it can give 160MiB/s without compression and about 220 MiB/s with 
compression on (on my data) - it is for my LTO6 . 


But the problem is you may not simply use dar or tar on disk and feed it to 
tape directly - 


like e.g. 


date&&dar -c - -R /usr  |dar_split split_output /dev/nst0&& date




The tapes always waits and then runs in start stop mode many times, so it is
being destroyed and the storage takes days (e.g. 2.5TB of  smaller images - 
several tens of million take days to store. In theory you can store 2.5 TB 
in 4 hours ... 


So it is absolutely necessary to send a data to a dar file  and then use 
directly dd (and mbuffer) to send to tape.




like 





cat betelgeuse-opt.1.dar|dar_split split_output /dev/nst0

or even adding mbuffer

cat betelgeuse-opt.1.dar||mbuffer -P90 -m 4G |dar_split split_output /dev/
nst0










The problem here is that the dar file may be much larger then 2.5TB and you 
don't have more space in array for multiple slices....

All the manipulation takes hours so it is difficult to test it ...

If you play with small samples (several GBs) - everything looks smooth but 
on large files the trape becomes "tired" and stop , wait and restarts 





So the optimal way I see is to split dar to multiple slices exactly 2.5 TB 
large and when it is ready to send it to tape and continue creating next 
slice .

In case of compression (which sometimes helps sometimes not) the dar_split 
is the only way ... 




But what I want to suggest to solve is the housekeeping of the dynamically 
on-the fly created slices.

I will have a bunch of tapes but there is no easy way to recognize which one
is of given slice and even that it contains a dar backup.

I know Denis has a good explanation why it is not possible currenly and 
namely for dar_split (as the system does not know where it is - it is just a
stream of data)




However I would like to have a hope that it will be possible one day to do 
this:




I know which slice "filename.number.dar"  is on which tape . and when 
randomly entering given tape I will be able to identify that it is a dar 
file slice number n

I will be able to create a catalog .(either on-the-fly or just reruning all 
tapes through (using tape robot) 





I will be able to find in the catalog given directory - it will tell me what
tapes I need to restore it.

I will use dar with only required  slices  to restore the given directory 
(just putting say 2 3 tapes in a robot and run some script.




So basically all the functionality is almost here but the nature of dar 
storage (catalog on the last slice etc... no clean signature on each 
slice..) prevents to use tapes in this dream scenario (i do not complain - 
just I am dreaming about ideal solution)

As you see my requirements are exactly the same as John's (below) but on 
serial device ...




Best regards, Petr Skoda






---------- Původní e-mail ----------
Od: John Goerzen <[email protected]>
Komu: [email protected]
Datum: 31. 5. 2023 18:17:52
Předmět: [Dar-support] Using dar for archives to optical discs
"Hello, 

As noted on my blog at 
https://changelog.complete.org/archives/10500-recommendations-for-tools-for-
backing-up-and-archiving-to-removable-media 
, I am working on a project for archiving data to optical media. The 
data set to be archived is several TB, and the optical media is 100GB at 
largest. Beyond the initial archive, I also want to be able to easily 
update the archived data set. 

dar is one of the two tools I'm seriously considering for this. 

I am envisioning a situation like this: 

The initial full archive run will be split into dar slices, and will 
cover many discs. 

Future archive runs will likely be a differential against the full 
archive. (Maybe some incrementals against a differential as well.) 

In any case, that much is clear enough. 

Now, the question of restoration arises. dar_manager can ingest these 
archives (or isolated catalogs). Is dar_manager slice-aware? In other 
words, can I get it to tell me, for a given restoration, "I will be 
needing slices 5, 3, and 10 of backup A and slice 2 of backup B"? 

One other question... when creating a differential backup, can dar 
efficiently store a rename (noticing the inode number is the same but 
now has a different name or is in a different directory), or will all 
renames cause files to be stored anew? 

Thanks! 

- John 

"