Re: [Patch] 3.1.0 partial and yearly backups

Jeffrey J. Kosowsky wrote:

> Holger Parplies wrote at about 05:24:15 +0100 on Monday, March 14, 2011:
>  > Hi all,
>  
>  > I see why this would be desirable for the initial backup, but ...
> pp >
>  > 1.) Doesn't BackupPC already do this for the initial backup? The
>  > initial
>  >     backup will always be a full, and failing full backups are stored
>  >     as partials and reused as reference - albeit not indefinitely -,
>  >     aren't they?
>  >     If there is a bug in this implementation (is there?), shouldn't it
>  >     be fixed rather than worked around?
> 
> I may be wrong... but I thought the problem is with incrementals and
> not with fulls since as your rightly point out, fulls already take
> advantage of partial backups. I think the problem is that incrementals
> may have even a small number of large files that have changed since
> the last full. Since they are discarded when an incremental fails, you
> need to start over again. If you have a few huge files, this can be
> wasteful. I think the intent of the code (which I have not examined)
> is to do for incrementals what already happens to fulls.
> 
There is no bug. BackupPC works as designed and described.
But BackupPC only overwrite a previous partial backup if the new backup 
contains more files as the previous partial.
It doesn't store an interrupted backup with less filecount but more storage 
capacity. So if you have a partial and you get a new big file in the source 
you have to transfer this big file again and again until the whole backup 
will run fine.
If you have a slow internet connection and/or an internet provider which 
does periodically reset the connection you doesn't be able to gain a 
complete backup.
And this disadvantage regards both, incremental as well as full backups.
> 
>  > 2.) Wouldn't your patch introduce a potential dramatic slowdown for
>  > later
>  >     backups?
>  >     Imagine your data set has 1000000 files and 300 GB of data. An
>  >     arbitrary backup fails on file number 3 after transferring 1 MB of
>  >     data because of a client reboot, network failure, whatever. Using
>  >     the partial backup as reference would mean re-transferring almost
>  >     all of the data, rather than just discarding the 1 MB that were
>  >     transfered and doing a normal rsync delta, would it not?
> 
> I'm not sure I understand the concern. I assumed (perhaps incorrectly)
> that the patched code saved newly transferred files from failed
> incrementals so that when the incremental restarts it doesn't need to
> retransfer that data.
Yes Jeffrey, you are right.
Because if the patch partials will be covered by 
BackupPC_dump/RsyncFileIO.pm like incrementals.
So I would expect the only performance impact is during the begin of a 
backup where RsyncFileIO.pm is looking for the most recent version of each 
already stored file.
But I dont't expect that there is a performance difference between backups 
stored as incremental or stored as partial.
> 
> 
>  >     To fix this correctly, you would need to merge the partial with its
>  >     reference backup and use that as a new reference, presuming that is
>  >     at all possible (which I doubt).
Yes, thats a possible alternative. I first tried to patch BackupPC_link. But 
I'm not happy with the solution.
a) I didn't implement it without errors? At least my tests failed partly.
b) It contains more changes against the original  as the elected solution.
>
> Yes - I assumed though that this is what the code would and should do
> -- either that or do some type of manual check to see if the file
> exists in a stored partial before transferring.
> 
>  > 
>  > 3.) You need to be aware that the partial backups do *not* represent a
>  >     snapshot of your system state that existed at any point in time.
>  >     They are broken by definition and only useful - in my opinion -
>  >     internally to BackupPC for speeding up the next backup. Once that
>  >     has completed, they are obsolete. Why would you want to keep them?
>  
> Agreed. Again I assumed that they would only be kept to speed up the
> next incremental.
>
Yes, you are absolutly right. Therefore all collected partials will be 
removed as soon as a backup successfull finished.
Because of reducement of changes I decide to use the original cleanup 
routine from BackupPC_dump (BackupExpire). Therefore the partials will be 
removed as recently as the next backup begins.
> 
>  > 4.) Failed backups are retried at the next wakeup after failing, aren't
>  > they?
>  >     Wouldn't that mean that you *could* end up storing many failed
>  >     partials (e.g. 24 per day) for one host, for which you have
>  >     scheduled much larger backup intervals (e.g. daily)?
>  > 
Yes, that's rigth too. In my point of view I decided to get a daily 
"complete" backup. Maybee íncremental maybee full. But at least complete.
Therefore in my opinion it is the right way to retry a failed backup as soon 
as possible.
>  > 
>  > My knowledge of BackupPC is getting a little rusty, so I might be
>  > mistaken on any of these points, but I'm sure Jeffrey will correct me
>  > if so ;-).
> 
> I'm no expert here ;) But I do understand the motivation for this
> patch in that even if an incremental fails on the very last file after
> transferring perhaps thousands of changed files, one needs to start
> again from scratch. And that wouldn't it be nice if at a minimum any
> previously transferred files could be saved to speed up the next
> incremental (of course some type of merging would be necessary to have
> the right baseline).
> 
> This might prove very helpful on things like laptops where sometimes
> they don't remain on long enough for an incremental to complete so it
> would be nice if the incremental could be restarted taking advantage
> of previously transferred files (I say restarted and not resumed
> because resuming would leave an even less point-in-time approximation
> of a backup since there might be a long time gap between one attempt
> and the next).
Yes - but the same arguments are true for full backups because the 
implemented algorithm only depends of the file count and do fully ignore the 
amount of transfered data.

-- 
Don't Panic


------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
BackupPC-devel mailing list
[email protected]
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/