Re: dar_manager questions

On 12/06/2023 03:30, John Goerzen wrote:
> Hello,

Hello,

> 
> As I'm trying to sort out my archiving scheme, I've got some questions
> about dar_manager.
> 
> First, let's start with a simple scenario.  Say I have a set of USB
> backup drives.  I start writing backup data to drive 1.  It gets a full
> backup.  Then, I'm writing incrementals/differentials.  I save off
> isolated catalogs from each backup run and ingest them into dar_manager.
> 
> Eventually I run out of drives, and at that point I delete the contents
> of drive 1 and start over there, deleting the contents of each
> previously-used drive before reusing it.
> 
> At this point, I would probably use dar_manager --delete to remove the
> full backup as I'm about to overwrite it.  That API is a bit
> inconvenient (it would be more convenient to be able to pass it a
> filename) but it'll do.  

Please, feel free to open a feature request for that, this appears to be 
simple to implement, though, several entries in the database could have 
the same name (nothing forbids that today). It should thus be defined 
which one(s) to remove (the first added, all of them under that name)... 
and also maybe forbidding adding a new backup of the same basename as 
one already existing in the database.

https://sourceforge.net/p/dar/feature-requests/

> Now what happens to all the
> incrementals/differentials that were based on the full backup?  And on
> subsequent ones?  Do I just need to start a new dar_manager database at > this point?

when you will ask to restore a particular file, if the latest modified 
state is in a incremental backup, this file will be restored, if else it 
was in the backup you just destroy, dar_manager will be able to restore 
this file (obviously).

Things could get worse if delta binary was used, as removing the full 
backup will not allow to restore the file because binary delta you could 
still have in more recent backup would not have something to base the 
patch on, unless by chance the status of the file in filesystem is the 
one the file had in the backup you removed... (If that's the case, take 
a chance playing on loto too!)

> 
> Now for the second scenario, let's say I am backing up the same data to
> three different backup media sets.  At any given moment, at least one
> will be off-site.  It is easy enough to base an incremental on the
> most recent backup in a set.
> 
> But now, what of dar_manager?  Can I safely import these three different
> disjoint scenarios into dar_manager?  

yes, there is no restriction (except decremental backups that must not 
be added to a dar_manager database, as documented)

you could make only full backups and have tens or hundred of these in 
dar_manager. But you can also have several full+incremental, 
full+differential, 
full+differential+incremental+differential+incremental backups recorded 
in the same database. The important point is to have feed them in 
chronological order.

at restoration time dar_manager will look for each file's status stack 
(see dar_manager's -f option) and add this file to the list of file to 
restore from one or more backups:
- possibly only a full backup
- possibly a full backup, and the "inode only change"
- possibly a fill backup and the latest incremental backup
- possibly a full backup and many incremental backup when using binary 
delta.

Once all requested files under restoration have been evaluated, 
dar_manager invokes dar for each requested backup applied to the list of 
files to restore for that particular backup.

>My guess here is that I'd be
> better off having three distinct dar_manager databases.  That would make
> answering the question "exactly what drive has the most recent archive
> of a given file" a little more annoying (but still doable).  Or is there
> a way to maintain these separate chains in a single database?

Personally what I use is an single dar_manager database with many full 
backup followed each by many incremental ones. When it's time to recycle 
a 'set', the oldest full and its following associated incremental 
backups are dropped from storage and their references removed from the 
dar_manager database. Usually before removing I test that the backup 
were still usable (dar -t / par2verify). You should probably do the same 
especially when the storage time is long and/or the storage medium is 
weak (I assume USB key are, compared to tape or optical disks).

I had no issues with that since almost 20 years ;)

Note that if dar_manager is very convenient to restore a given file/set 
of file/directory that has been removed by mistake several months or 
year ago (and thus not having to seek several hundred backups), for 
restoration after a disk crash, it is more efficient to directly use dar 
restoring the backups from the latest full backup up to the most recent one.

> 
> Thanks!
> 
> - John
> 

My 2 cents,
Denis