Re: Multi-level incremental back-ups (was: Decremental backup: hard-disk space consideration)

On 15/02/2021 16:30, Stefan Müller wrote:
> Am 03.02.21 um 19:07 schrieb Denis Corbin:
[...]
>>
>> Thus, retention is thus very flexible in the strategy you expect to use.
> 
> Thanks for taking the time for your elaborate answer! My question had
> been almost as thoroughly worked out -- up to the title, that is. Sorry
> for any confusion this may have created. I had originally been thinking
> about using decremental backuups but their incompatibility with delta
> signatures made all related questions become irrelevant. Nevertheless,
> my main concern is hard-disk space, indeed.
> 
> The pointer to dar_manager is very much appreciated as I have not
> studied it yet but definitively will.
> 
> If I understand your description correctly, then the level-1 back-ups
> would contain multiple identical copies of files because they only
> depend on the level-0 back-up but not upon each other.

no, a given backup may it be full or differential does usually only
contain a single copy of a given file. The level-1 backup is a
differential one, it contains only data of file that change since the
level-0 used as reference was done.

> A file that was
> created immediately after the level-0-back-up would be contained in each
> one of them, wouldn't it?

It would be contained in level-0 backup (because it is a full backup),
as it has changed right after this backup it will be stored in the
level-1 backup too, either as a full copy (eventually compressed) or as
a binary delta based on the version stored in the level-0 backup

If it does not change any further, level-2 backup will just contain its
metadata (permission, date, ownership...)

But if it changes after a level-1 backup was made, it will be stored a
second time in the first level-2 backup that follows the change as well
as the following ones,

the next-month, the level-1 backup will also have a copy of it but
unless it changes again during that month, the level-2 backup will only
hold its metadata

> 
> Maybe I got the terminology wrong. I thought of incremental back-ups to
> be moving one level up at each stage as opposed to differential ones
> that always depend on a full back-up.

Well, you are right, the term differential/incremental are used
differently depending on people. One can define differential when it is
based on a full backup and incremental else. I usually do that too.

But here, when we consider the level-2 backups of a given month, if they
are all based on the level-1 backup made the first day of that month,
one could say they are "differential" backups, by opposition to the
situation where each level-2 backup takes as reference the backup of the
day before, in which case each level-2 can be seen as an incremental.

In the first case you restore level-0, level-1 and the level-2 of the
day you want.

In the second, you restore level-0, level-1, and all the level-2 from
the 2nd of the month to the day you want, in that order.

> 
> Using your terminology, the monthly back-ups should be level-0 for the
> original one, level-1 for the first incremental one, level-2 for the
> second etc. Within one year there would be up to level-12, within five
> years up to level-60.

difficult to understand which reference you take for each backup, seen
the numbering you use, I assume the backup of each month uses the backup
of the previous month as reference (except the first month which is a
full backup).

> 
> The daily back-ups would be like branches to these monthly ones. Thus
> there would be level-0.1 up to level-0.31 for the first month, each
> incremental back-ups, depending on each other. For the second month
> there would be level-1.1 up to level-1.31 incremental back-ups and so on.


OK I see,

> 
> The first monthly level-1 incremental back-up would basically wrap the
> preceding 31 daily ones (level-0.1  up level-0.31) into one single
> incremental back-up. This way, a single file change would be included
> twice into the pool, once in a daily and once in a monthly incremental
> back-up.

correct,

> 
> Since I envision almost "perpetual" or at least very long-term retention
> of the back-ups, I could delete the daily back-ups after a while and
> retain only the monthly ones.

Yes this is possible,

> Thereby, file changes would not balloon
> the pool as much anymore. After some epoch, the pool can be set aside
> and a new one can be started with the same back-up strategy.


That makes sense, yes


> 
> It must be obvious that I do not have any hands-on experience so I am
> very thankful for any comments!

dar_manager will be of greate help here to retrieve a deleted files
easily even long after it has been removed (by accident), so there is no
drawback here, human operation is a single command, though the
rebuilding of a file my imply a lot of backup if binary delta has been
used and the file changed a lot over time (thus has a binary delta in
each backup)

The issue is more annoying when it would come to restore the whole
system, in that case you would not use dar_manager, but would be more
efficient restoring the backups in order from level-0 to level-1-N then
from level-1-N to level-2-M (where N is the month and M the day of the
month you want ot restore the system state to).

This is a choice between the space requirement for backup and the time
to restore the backup, nothing is perfect. Added to this the probability
of a system global crash and you should take the best decision that
matches your context :^)

> 
> Cheers!
> Stefan Müller
> 
> 

Cheers,
Denis

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support