Re: Restoring everything with dar_manager

Hi Denis,

Thanks for all the conversation!  As an aside, I have just uploaded the
first libthreadar-enabled dar to Debian, and am also putting finishing
touches on another couple of blog posts about it.  I'll share more about
these in a few days.

On to the replies:

On Sat, Jun 17 2023, Denis Corbin wrote:

>> Let's say you had these backups:
>>
>> #1 - full
>> #2 - incremental based on #1
>> #3 - incremental based on #2
>>
>> Now say that in #2, you deleted 50GB of data that was in #1.  If you
>> just play back the whole thing, that's 50GB of unnecessary restores (10
>> DVD+R slices if you wrote it to DVD+R).  I believe dar_manager can
>> optimize this.

..snip..

> transfer a list of files between dar_manager and dar has not changed, it should
> be reviewed to reduce the memory footprint (for example relying on a file
> listing would reduce a lot the memory footprint). Actually the list of file is
> stored in memory and passed to dar using an anonymous pipe. The database
> searching algorithm could also be simplified (as all entries are now concerned,
> no need complex consideration of what entry should be considered or not for
> restoration).
>
> Feel free to add a feature request about that, but be aware this is not a simple
> work behind the scene, so it will take time to be implemented. But I agree that
> such feature request is valuable.

Completely understood!  https://github.com/Edrusb/DAR/issues/51 submitted.

> Today, if you know the path of the 50 GB that has been removed in #2, you can
> exclude it while restorating #1 and #2. (-P option or equivalent for example).

True.


> An alternative to avoid restoring the 50 GB of data would be to build
> decremental backup, rather than incremental ones. So you then only restore the
> last backup (which is a full backup). The drawback is that decremental are not
> compatible with dar_manager... and transforming the former latest full backup
> into a decremental takes some time as it implies reading the latest full backup
> and the former latest one.

Understood, but decrementals aren't practical for this particular use
case.

I guess the question I have left is: is there presently a way for a user
to do consistent restore from a backup set consisting of both
incremental and differentials, without having the manually track the
order to do so outside of dar?

>> Further let's imagine that before running backup #3, you created 10GB of
>> data, which was deleted before running backup #4.  Backup #4 isn't going
>> to record that deletion, since that data wasn't present in #1.  What
>> will the result be?
> for each particular entry, dar_manager has a list of status per backup stored in
> the database. Adding a new backup which does not contain a given entry leads to
> the same result in the database as adding a differential backup which records
> this file as removed. As such, these 10G of data would not be restored.

Right, but let's say we can't use dar_manager - perhaps because we want
to restore it all.  Here's a test I did:

jgoerzen@hephaestus:/tmp/dartest$ mkdir data
jgoerzen@hephaestus:/tmp/dartest$ echo one > data/file1
jgoerzen@hephaestus:/tmp/dartest$ dar -c bak1 -R data
jgoerzen@hephaestus:/tmp/dartest$ dar -c bak2 --ref bak1 -R data
jgoerzen@hephaestus:/tmp/dartest$ rm data/file2
jgoerzen@hephaestus:/tmp/dartest$ dar -c bak3 --ref bak1 -R data

jgoerzen@hephaestus:/tmp/dartest$ dar -x bak1 -R restore
jgoerzen@hephaestus:/tmp/dartest$ dar -x bak2 -R restore
jgoerzen@hephaestus:/tmp/dartest$ dar -x bak3 -R restore
jgoerzen@hephaestus:/tmp/dartest$ ls -l data restore
data:
total 1
-rw-r--r-- 1 jgoerzen jgoerzen 4 Jun 17 23:36 file1

restore:
total 1
-rw-r--r-- 1 jgoerzen jgoerzen 4 Jun 17 23:36 file1
-rw-r--r-- 1 jgoerzen jgoerzen 4 Jun 17 23:37 file2

As I thought.  I wonder if there's a way for dar to clean up files on
the filesystem that weren't present at backup time?  I'm not immediately
seeing one.

Thanks!

- John