Re: [Patch] 3.1.0 partial and yearly backups

Matthias Meyer wrote:

> Jeffrey J. Kosowsky wrote:
> 
>> Holger Parplies wrote at about 05:24:15 +0100 on Monday, March 14, 2011:
>>  > Hi all,
>>  
>>  > I see why this would be desirable for the initial backup, but ...
>> pp >
>>  > 1.) Doesn't BackupPC already do this for the initial backup? The
>>  > initial
>>  >     backup will always be a full, and failing full backups are stored
>>  >     as partials and reused as reference - albeit not indefinitely -,
>>  >     aren't they?
>>  >     If there is a bug in this implementation (is there?), shouldn't it
>>  >     be fixed rather than worked around?
>> 
>> I may be wrong... but I thought the problem is with incrementals and
>> not with fulls since as your rightly point out, fulls already take
>> advantage of partial backups. I think the problem is that incrementals
>> may have even a small number of large files that have changed since
>> the last full. Since they are discarded when an incremental fails, you
>> need to start over again. If you have a few huge files, this can be
>> wasteful. I think the intent of the code (which I have not examined)
>> is to do for incrementals what already happens to fulls.
>> 
> There is no bug. BackupPC works as designed and described.
> But BackupPC only overwrite a previous partial backup if the new backup
> contains more files as the previous partial.
> It doesn't store an interrupted backup with less filecount but more
> storage capacity. So if you have a partial and you get a new big file in
> the source you have to transfer this big file again and again until the
> whole backup will run fine.
> If you have a slow internet connection and/or an internet provider which
> does periodically reset the connection you doesn't be able to gain a
> complete backup.
> And this disadvantage regards both, incremental as well as full backups.
>> 
>>  > 2.) Wouldn't your patch introduce a potential dramatic slowdown for
>>  > later
>>  >     backups?
>>  >     Imagine your data set has 1000000 files and 300 GB of data. An
>>  >     arbitrary backup fails on file number 3 after transferring 1 MB of
>>  >     data because of a client reboot, network failure, whatever. Using
>>  >     the partial backup as reference would mean re-transferring almost
>>  >     all of the data, rather than just discarding the 1 MB that were
>>  >     transfered and doing a normal rsync delta, would it not?
>> 
>> I'm not sure I understand the concern. I assumed (perhaps incorrectly)
>> that the patched code saved newly transferred files from failed
>> incrementals so that when the incremental restarts it doesn't need to
>> retransfer that data.
> Yes Jeffrey, you are right.
> Because if the patch partials will be covered by
Because of the patch - partials...
> BackupPC_dump/RsyncFileIO.pm like incrementals.
> So I would expect the only performance impact is during the begin of a
> backup where RsyncFileIO.pm is looking for the most recent version of each
> already stored file.
> But I dont't expect that there is a performance difference between backups
> stored as incremental or stored as partial.
>> 
>> 
>>  >     To fix this correctly, you would need to merge the partial with
>>  >     its reference backup and use that as a new reference, presuming
>>  >     that is at all possible (which I doubt).
> Yes, thats a possible alternative. I first tried to patch BackupPC_link.
> But I'm not happy with the solution.
> a) I didn't implement it without errors? At least my tests failed partly.
> b) It contains more changes against the original  as the elected solution.
>>
>> Yes - I assumed though that this is what the code would and should do
>> -- either that or do some type of manual check to see if the file
>> exists in a stored partial before transferring.
>> 
>>  > 
>>  > 3.) You need to be aware that the partial backups do *not* represent a
>>  >     snapshot of your system state that existed at any point in time.
>>  >     They are broken by definition and only useful - in my opinion -
>>  >     internally to BackupPC for speeding up the next backup. Once that
>>  >     has completed, they are obsolete. Why would you want to keep them?
>>  
>> Agreed. Again I assumed that they would only be kept to speed up the
>> next incremental.
>>
> Yes, you are absolutly right. Therefore all collected partials will be
> removed as soon as a backup successfull finished.
> Because of reducement of changes I decide to use the original cleanup
> routine from BackupPC_dump (BackupExpire). Therefore the partials will be
> removed as recently as the next backup begins.
>> 
>>  > 4.) Failed backups are retried at the next wakeup after failing,
>>  > aren't they?
>>  >     Wouldn't that mean that you *could* end up storing many failed
>>  >     partials (e.g. 24 per day) for one host, for which you have
>>  >     scheduled much larger backup intervals (e.g. daily)?
>>  > 
> Yes, that's rigth too. In my point of view I decided to get a daily
> "complete" backup. Maybee íncremental maybee full. But at least complete.
> Therefore in my opinion it is the right way to retry a failed backup as
> soon as possible.
>>  > 
>>  > My knowledge of BackupPC is getting a little rusty, so I might be
>>  > mistaken on any of these points, but I'm sure Jeffrey will correct me
>>  > if so ;-).
>> 
>> I'm no expert here ;) But I do understand the motivation for this
>> patch in that even if an incremental fails on the very last file after
>> transferring perhaps thousands of changed files, one needs to start
>> again from scratch. And that wouldn't it be nice if at a minimum any
>> previously transferred files could be saved to speed up the next
>> incremental (of course some type of merging would be necessary to have
>> the right baseline).
>> 
>> This might prove very helpful on things like laptops where sometimes
>> they don't remain on long enough for an incremental to complete so it
>> would be nice if the incremental could be restarted taking advantage
>> of previously transferred files (I say restarted and not resumed
>> because resuming would leave an even less point-in-time approximation
>> of a backup since there might be a long time gap between one attempt
>> and the next).
> Yes - but the same arguments are true for full backups because the
> implemented algorithm only depends of the file count and do fully ignore
> the amount of transfered data.
> 

-- 
Don't Panic


------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
BackupPC-devel mailing list
[email protected]
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/