Re: Decremental backup: hard-disk space consideration

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

let me reformulate your backup strategy to be sure to understand it
properly:

- - you start by a full backup let's call it the level-0 backup
- - each first day of the month you create an incremental backup based
on the first level-0 backup. Let's call these level-1 backups
- - each day of the month except the first, you create an incremental
backup based on the level-1 backup of the first day of the month,
let's call these level-2 backups

is it correct?

You don't mention the retention period of backups, but as the backup of
each month (level-1) do only depend on the full backup (level-0), you
can keep all of them down to only the latest level-1 backup (and still
the level-0 backup of course).
As level-2 backups depend on the level-1 done the first of the month,
you can keep all of them in a month, down the lastest level-2, plus
the latest level-1 and the latest level-0

Thus, retention is thus very flexible in the strategy you expect to use.

Assuming you keep the same level-0 backup forever, years after that,
restoring a file in the state it has the 25th of December will require
either:
- - only the level-2 of the 25th (if the file was too small or
explicitly excluded from binary delta)
- - the level-1 of December alone (if it has not changed since that time
and was not concerned by binary delta)
- - the level-0 alone (if it has not changed since day one)
- - the level-0 and the level-1 (if it has not changed since december
1st) and binary delta has been used
- - the level-0, level-1 of December and level-2 of the 25th when binary
delta has been used twice

To ease restoration, you can feed all backups (level-0, level-1 and
level-2) to a dar_manager database. Dar_manager will manage to fetch
the only necessary backups to restore a particular file or set of files.

So after 10 years you would need at most 3 backups.

now depending on the size of the level-1 backups, you might one day
consider doing a new level-0 backup, and define the retention policy
for the number of level-0 sets you want to keep, the number of level-1
you want to keep in each level-0 set and the number of level-2 backup
you want to keep as well.

To my point of view, thanks to the systematic use of par2, the delta
binary is not an issue. Using binary delta Without par2, the most
sensible piece of data is the level-0 backup: if a corruption takes
places in it, it will affect one file (or more depending on the amount
of corrupted bytes) and this file will not be restorable from this
level-0 backup, nor from any level-1 and level-2 backups, even if those
other backups are sane.

In brief, to me this is a good strategy and you can simplify the
restoration process using dar_manager

Regards,
Denis


On 02/02/2021 20:42, Stefan Müller wrote:
> Hello list members
>
> Dar has been on my mind literally for years and now the time has
> finally come to plunge into this project. During that long period
> the amount of data that I want to back-up has grown to about one
> terabyte. It consists of pretty static data storage of local
> machines.
>
> I already have a working backup mechanism using Borgbackup which
> does data deduplication and can be pretty much left alone. I want
> to add Dar as independent solution for a long-term mirror of the
> data storage. My strategy is to gradually fill up the hard-disk as
> long as there is space. Full hard-disks are archived and replaced
> by new bigger ones.
>
> After reading through most of the documentation my current idea of
> backup strategy is the following. I wonder if it is at all feasible
> and would welcome experienced opinions.
>
> = Back-up strategy =
>
> In general, I want to use delta signatures in combination with
> Par2.
>
> 1.) Create monthly incremental back-ups at the first of every
> month. If the back-up plan had started on January 1st, 2020, the
> monthly Dar pool e.g. in December 2020 would consist of a full
> back-up 202001 with incremental back-ups 202001, 202002, ...,
> 202012
>
> 2.) Create daily incremental back-ups based on the latest
> (incremental) monthly back-up. So the daily Dar pool would consist,
> e.g. for December 2020, of incremental back-ups 20201202, 20201203,
> ..., 20201231 all based on 202012.
>
> For the previous months there would be incremental back-ups
> 20201102, 20201103, ..., 20201130 all based on 202011 and so on.
>
> = Restoration =
>
> In order to restore the state of a file from December 25th, 2020,
> I would have to first re-build the incremental back-up of the
> specific month by going back from the initial full back-up. Then
> the process would consist of re-building the incremental daily
> back-ups from the monthly one.
>
> I suspect that these two restoration chains can be combined into a
> single one. The selection of archives can easily be coded.
>
> = Rationale =
>
> Using all incremental backups with delta signatures keeps
> hard-disk space requirements reasonably low.
>
> Restoring a file after a five year period would need the
> combination of at most about 90 archives, about 150 over a ten year
> period. These numbers would be about 20 times higher if using daily
> back-ups only.
>
> = Questions =
>
> Obviously, I have not tested that idea and cannot tell if it is
> feasible and efficient. Any comments?
>
> Thanks in advance!
>
> Sincerely, St. Müller
>
>
> _______________________________________________ Dar-support mailing
> list [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-support
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAmAa5moACgkQCDGwvQPY
sYKYHw//V+kwaDSwEyhu3c4LKAlmIqJvdHNFTdMQDbEfnqwdRdNaQpwUMRuho0Th
Rpb48DCGLGTcPQrALNeLvSB4lLidFXPQ6oCqF8VexzOp0ws1bmoi935tqObftLzy
dod4GsVX4LJZvV3F+7asdfXKWjnb3UUHeZm7FgPDYuojexlygmoaAPezy4zSKWAb
7scfU/OUnK5KVOA+a28QvWzE1oUssie5dfBnmCMXNKQcj2282VeAamJhfpJRTVx/
Z5jUHo1wAl4BM618lQ7SjDFroohTUr+OkCJnHXJbxIL2bI3l//vLWc3PPl02bV6S
n1kx4u8AXD4ji9TwMzEl8Zn/hYKYiwIO4bI0fIf6fhS/ah45JpE+Ml3aK6ipqZvS
j1j9WawcJJIBSPNk1K14XlO693/iytKNZfI7VCjKHodAJZtGTZb2Q0Y011WM+gTe
DevgEk4bgjDnUfef8sYfTk9xpl9OvPCwhNdne1CpqOfQgreVm3+bsNqy7hmTl8NN
nVk4Tv4QUzh5XAChIcgp5vcRoBctF6t9aDLN20J9EkzhN+7qJb7O/9kmisLTNMCt
iW3ubNwGBsw5JG5T6u+B/J1FR+Cbr1FjFbhIdYPjzTFTDf9/XB9gphtxOLR2cJiX
I7NI0ghE4EJvv8htnBFOTqnFWHyMzxKxcXsdODWrpFie6ULtliE=
=ZFAq
-----END PGP SIGNATURE-----


_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support