Re: [Patch] 3.1.0 partial and yearly backups

Holger Parplies wrote at about 05:24:15 +0100 on Monday, March 14, 2011:
 > Hi all,
 
 > I see why this would be desirable for the initial backup, but ...
pp > 
 > 1.) Doesn't BackupPC already do this for the initial backup? The initial
 >     backup will always be a full, and failing full backups are stored as
 >     partials and reused as reference - albeit not indefinitely -,
 >     aren't they?
 >     If there is a bug in this implementation (is there?), shouldn't it be
 >     fixed rather than worked around?

I may be wrong... but I thought the problem is with incrementals and
not with fulls since as your rightly point out, fulls already take
advantage of partial backups. I think the problem is that incrementals
may have even a small number of large files that have changed since
the last full. Since they are discarded when an incremental fails, you
need to start over again. If you have a few huge files, this can be
wasteful. I think the intent of the code (which I have not examined)
is to do for incrementals what already happens to fulls.


 > 2.) Wouldn't your patch introduce a potential dramatic slowdown for later
 >     backups?
 >     Imagine your data set has 1000000 files and 300 GB of data. An arbitrary
 >     backup fails on file number 3 after transferring 1 MB of data because of
 >     a client reboot, network failure, whatever. Using the partial backup as
 >     reference would mean re-transferring almost all of the data, rather than
 >     just discarding the 1 MB that were transfered and doing a normal rsync
 >     delta, would it not?

I'm not sure I understand the concern. I assumed (perhaps incorrectly)
that the patched code saved newly transferred files from failed
incrementals so that when the incremental restarts it doesn't need to
retransfer that data.

 >     To fix this correctly, you would need to merge the partial with its
 >     reference backup and use that as a new reference, presuming that is at
 >     all possible (which I doubt).

Yes - I assumed though that this is what the code would and should do
-- either that or do some type of manual check to see if the file
exists in a stored partial before transferring.

 > 
 > 3.) You need to be aware that the partial backups do *not* represent a
 >     snapshot of your system state that existed at any point in time. They
 >     are broken by definition and only useful - in my opinion - internally
 >     to BackupPC for speeding up the next backup. Once that has completed,
 >     they are obsolete. Why would you want to keep them?
 
Agreed. Again I assumed that they would only be kept to speed up the
next incremental.

 > 4.) Failed backups are retried at the next wakeup after failing, aren't they?
 >     Wouldn't that mean that you *could* end up storing many failed partials
 >     (e.g. 24 per day) for one host, for which you have scheduled much larger
 >     backup intervals (e.g. daily)?
 > 
 > My knowledge of BackupPC is getting a little rusty, so I might be mistaken on
 > any of these points, but I'm sure Jeffrey will correct me if so ;-).

I'm no expert here ;) But I do understand the motivation for this
patch in that even if an incremental fails on the very last file after
transferring perhaps thousands of changed files, one needs to start
again from scratch. And that wouldn't it be nice if at a minimum any
previously transferred files could be saved to speed up the next
incremental (of course some type of merging would be necessary to have
the right baseline).

This might prove very helpful on things like laptops where sometimes
they don't remain on long enough for an incremental to complete so it
would be nice if the incremental could be restarted taking advantage
of previously transferred files (I say restarted and not resumed
because resuming would leave an even less point-in-time approximation
of a backup since there might be a long time gap between one attempt
and the next).

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
BackupPC-devel mailing list
[email protected]
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/