Re: convert old full + differentials into new full

On 16/04/2021 20:57, imschmeg wrote:
> On Fri, 16 Apr 2021 20:15:28 +0200
> Denis Corbin <[email protected]> wrote:
> 
>> On 16/04/2021 19:36, imschmeg wrote:
>>> Is there a way to convert an old full backup (at time t0) plus a
>>> set of progressing differential backups off of that full backup (at
>>> times t1...tn) into a new full backup (at time tn) by only working
>>> with the backups themselves (not the original sources) and without
>>> restoring anything?  
>>
>> yes, this is possible, see FAQ for details:
>> http://dar.linux.free.fr/doc/FAQ.html#full-from-diff
> 
> Thanks for that link, Denis!
> 
> However, the link suggests that I need to make the new full backup at
> t1 by using the old one at t0 plus a differential at t1, and then
> proceed with that a single incremental time step at a time.  In other
> words, I can't move in one single operation from a full backup at time
> t0 plus as sequence of incrementals to a full backup at time tn.  Is
> that correct?


Correct, dar can read at most two backups (merging operation) to produce
a third one. With a full + 10 incremental backups you may have the
latest version of each file spread in all full and all incremental ones.
Making a full backup of the latest incremental one in one step would
require ingesting all 11 backups at once...

Though you can script that easily and run the script once ;)

> 
>>
>>
>>>
>>> I'd like to do this as a background task ocassionally, so that
>>> newer differential backups, after this background task has
>>> completed, can use the newer full backup as reference.   
>>
>> if you just want to base a "differential" backup on the previous
>> "differential" backup (which method is also known as incremental
>> backup) no need to rebuild the full backup, just give to -A option
>> the previous differential/incremental backup.
> 
> That's what I am already doing.  Perhaps I am confusing the terminology
> here, and should really be talking about incremental backups instead of
> differential backups. What I have is a full backup at t0, then an
> incremental at t1 using t0 as reference, then another incremental at t2
> using the incremental at t1 as reference, etc..
> 
> At some point, though, I have many incrementals and am in danger of
> needing to process them all in order to resore the most recent image of
> the sources.  This is why I want to migrate the full backup forward.

yes that's the drawback of endless incremental backups...

> 
> Although, perhaps I should abandon that scheme for one with
> differentials at t1..tn that are all based on the same full at t0, as
> then I can use the full-from-diff mechanism in a single step to make a
> full backup at tn.  Or mix them, so that t1..tn-1 are incremental, but
> tn is differential (directly referencing the full at t0), so that I can
> create a new full backup using full-from-diff in a single step at tn but
> not t2..tn-1.

Or maybe making a mix, having several raw of incremental (I) backups
based on the same full (F) backup, (D standing for an incremental based
on the full thus a "Differential" backup):

F+D-I-I-I-I
 |
 +D-I-I-I-I
 |
 +D-I-I-I-I

F+...

There is no best solution for all cases, it always depends on the
context/environment/"data life".

> 
>>
>> hint: have a look at dar_manager to ease restoration of a particular
>> file from a possibly huge set of full, differential and incremental
>> backups
>>
>>> This way I can
>>> avoid having to do a new full backup from the sources,  
>>
>> Doing that will probably be slower than doing a new full backup, as
>> you will have to uncompress, eventually uncipher the old full backup
>> the old differential backup and manipulate than just reading the data
>> from filesystem... then in both cases dar will compress and
>> eventually cipher the resulting full backup.... I let you test and
>> report the result of your experimentation here, if you like ;^)
> 
> I am not using dar's compression or encryption.  Or, rather, the sources
> are encrypted on a per-file basis prior to dar seeing them, as the file
> system I'm backing up provides this per-file encryption.  Hence, it is
> useless for dar to attempt compression or encryption.

of course, no need to waste CPU cycle

> 
> Also, I am willing to spend more time on this conversion as it can be
> done without interfering with the user of the original sources,
> provided it doesn't consume too much additional storage on its own.
> 

Yes, if this operation is not done on the same host, this avoids
consuming I/O on the production host a full backup would require.

Dar does not use intermediate storage, and the resulting merged backup
should not use more space than a normal full backup done with the same
compression/encryption/... options.



>