Re: Restoring everything with dar_manager

On 16/06/2023 23:17, John Goerzen wrote:>
 > On Fri, Jun 16 2023, Denis Corbin wrote:
 >
 >> [[PGP Signed Part:Undecided]]
 >> On 16/06/2023 21:13, John Goerzen wrote:
 >>> Hi,
 >>> In looking through things a bit, I see that dar_manager does nothing if
 >>> no files are passed to -r.  I tried both / and . as possibilities, and
 >>> neither did anything.  I want to restore an entire archive using
 >>> dar_manager; is there a way to do that that I'm missing?  (Similar to
 >>> just dar -x without any filters.)
 >>
 >> first, its much more efficient to restore with dar directly when it 
comes to
 >> restore an entire archive. If this just a full backup, just one line 
or command
 >> is needed. If full and differential and/or incremental, simply run 
dar with in
 >> turn the full and diff(s) and incremental(s) in the order the have 
been created
 >> (eventually adding -w option not to be annoyed when restoring a 
modified file,
 >> else dar would ask for confirmation for each file).
 >
 > Hi Denis,
Hi John,

 >
 > So here are the scenarios I had in mind, but from an efficiency and an
 > ease of use perspective.
 >
 > Let's say you had these backups:
 >
 > #1 - full
 > #2 - incremental based on #1
 > #3 - incremental based on #2
 >
 > Now say that in #2, you deleted 50GB of data that was in #1.  If you
 > just play back the whole thing, that's 50GB of unnecessary restores (10
 > DVD+R slices if you wrote it to DVD+R).  I believe dar_manager can
 > optimize this.
This is an interesting scenario. Yes, dar_manager has the means to 
address this, avoiding restoring the 50G in #1 to have them being 
removed afterward in #2. But the current -r option implementation is not 
adapted to that. In the beginning it was only expected to receive files 
(and not directories), so the user would not provide several tens of 
thousand entries. Directories has been added later and just expand the 
list of file from the content of each directory, recursively. Now 
dar_manager may ask dar to restore much more file, but as the mechanism 
to transfer a list of files between dar_manager and dar has not changed, 
it should be reviewed to reduce the memory footprint (for example 
relying on a file listing would reduce a lot the memory footprint). 
Actually the list of file is stored in memory and passed to dar using an 
anonymous pipe. The database searching algorithm could also be 
simplified (as all entries are now concerned, no need complex 
consideration of what entry should be considered or not for restoration).

Feel free to add a feature request about that, but be aware this is not 
a simple work behind the scene, so it will take time to be implemented. 
But I agree that such feature request is valuable.

Today, if you know the path of the 50 GB that has been removed in #2, 
you can exclude it while restorating #1 and #2. (-P option or equivalent 
for example).

An alternative to avoid restoring the 50 GB of data would be to build 
decremental backup, rather than incremental ones. So you then only 
restore the last backup (which is a full backup). The drawback is that 
decremental are not compatible with dar_manager... and transforming the 
former latest full backup into a decremental takes some time as it 
implies reading the latest full backup and the former latest one.

Last, another possibility would be to build a full backup from 
differential/incremental ones, and then restore this built full backup. 
See 'full-from-diff' target in /etc/darrc. The full backup made from #2 
will imply reading the whole increment #2 and part of the #1 that have 
not changed between #1 and #2. So you will not copy nor read the 50 GB 
of data. Once done you can either build a full-from-diff of #3 and 
restore only #3-full built backup, or depending on you context, restore 
the #2-full flavor one, then restore the original #3 incremental and 
following ones.

 >
 > Secondly, what if your backup scheme is more complicated?  Say you also
 > have:
 >
 > #4 - differential based on #1
 >
you are absolutely right, having dar_manager handling that really makes 
sense. But as mentioned above, this requires some reconsideration of the 
current implementation. Thus time.

 > Further let's imagine that before running backup #3, you created 10GB of
 > data, which was deleted before running backup #4.  Backup #4 isn't going
 > to record that deletion, since that data wasn't present in #1.  What
 > will the result be?
for each particular entry, dar_manager has a list of status per backup 
stored in the database. Adding a new backup which does not contain a 
given entry leads to the same result in the database as adding a 
differential backup which records this file as removed. As such, these 
10G of data would not be restored.

Note the behavior is the the same if excluding data from backup in a 
recent backup.

 >  Probably more data than really was on the system at
 > #4.  I can imagine other scenarios like this as well.
just test and experiment, with a few files or a single file and, please, 
report any issue you could find ;) This will also reassure you and 
possibly improve dar/dar_manager.

 >
 > dar_manager would presumably know there's no need to even touch #2 or #3
 > to get you to the place of #4.
yes

 >
 > Thanks,
 >
 > John
 >
Cheers,
Denis