Re: Using dar for archives to optical discs

On 31/05/2023 17:54, John Goerzen wrote:
> Hello,

Hello,

> 
> As noted on my blog at
> https://changelog.complete.org/archives/10500-recommendations-for-tools-for-backing-up-and-archiving-to-removable-media
> , I am working on a project for archiving data to optical media.  The
> data set to be archived is several TB, and the optical media is 100GB at
> largest.  Beyond the initial archive, I also want to be able to easily
> update the archived data set.
> 
> dar is one of the two tools I'm seriously considering for this.
> 
> I am envisioning a situation like this:
> 
> The initial full archive run will be split into dar slices, and will
> cover many discs.
> 
> Future archive runs will likely be a differential against the full
> archive.  (Maybe some incrementals against a differential as well.)
> 
> In any case, that much is clear enough.
> 
> Now, the question of restoration arises.  dar_manager can ingest these
> archives (or isolated catalogs).  Is dar_manager slice-aware?  

No, dar_manager does not need to be slice-aware. Furthermore, 
dar/dar_manager do support re-slicing a backup/archive (using dar_xform) 
and the re-sliced backup/archive can be used in place of the original in 
total transparency with the same isolated catalogue.

> In other
> words, can I get it to tell me, for a given restoration, "I will be
> needing slices 5, 3, and 10 of backup A and slice 2 of backup B"?

This is dar's task (not dar_manager's) to do that, see -Tslice option:

  dar -l <backup/archive> -Tslice
  dar -l <backup/archive> -Tslice -g some/file

This also works using an isolated catalogue, but the slicing used will 
be the one of the backup/archive it has been isolated from. If dar_xform 
is used to re-slice the source backup, the isolated catalogue stay 
usable to rescue the internal catalogue of the re-sliced archive/backup 
but the slicing information it will give will still be the one of the 
original backup/archive.

Note that location of data and metadata stored in a catalogue correspond 
to the amount of byte from the beginning of the backup as if there was 
only a single slice. If the backup is sliced this is the duty of the 
underlying "sar" object to abstract this slicing and convert these 
absolutes offsets to slice number + relative offset, knowing the slicing 
information of the backup (= initial slice size and other slices size).

> 
> One other question... when creating a differential backup, can dar
> efficiently store a rename (noticing the inode number is the same but
> now has a different name or is in a different directory), or will all
> renames cause files to be stored anew?

As of today the answer is no: dar does not detect renamed or moved file.
using the inode number information to do that is interesting but does 
not address all use cases (like a copied files then deleted source file, 
or restored file from a previous backup) ; and consideration should be 
made about the amount of RAM necessary to hold a dictionary to 
efficiently search a whole backup for an inode number and the additional 
CPU cycles to search this dictionary each time a new or deleted file is 
about to be saved/recorded. There would also be some complexity to keep 
the trace of file movement in each backup, for it can be usable at 
restoration time.

> 
> Thanks!
> 
> - John
> 

Cheers,
Denis