Re: Recovering from an interruption during backup

Hi Jakob,

There are some countermeasures in duplicity to help at least with some 
issues. Tuning  "--backend-retry-delay" "--num-retries" "--concurrency" 
to recover from failures and speed up.
But I think in your case you should review your setup.
Duplicity is a file backup, but not a file sync solution. Even if 
duplicity is able to recover from all interruptions while running one 
backup over 3 weeks, it could cause inconsistencies, because duplicity 
is not doing a snapshot when it starts but reading each file at the time 
when it is placed in a volume. Means if there are file changes while the 
backup is running and files depend on each other, they may not align to 
each other.

You should consider a different setup like:
Do a backup to a local drive. Could be something like an USB3 disk, 
which is fast enough to complete timely. In a 2nd step sync the content 
of the USB disk to s3. There are tools out for this. So each volume can 
be transferred/retried separately, which should be a good granularity. 
Even if USB disks are know to be unreliable, having a local copy helps 
to speed up restore if the disk survived the disaster.

An other option is to split backups into parts. E.g. I have a huge photo 
collection and create a dedicated backup per year, as photos of each 
year are stored in a separate folder. This allows to restart those 
backups more granularly. I also use the option `--skip-if-no-change` as 
most of the folders doesn't change or very rarely. As there is not 
advantage to do another full backup if nothing has changed, is saves a 
lot of unnecessary s3 transfers/costs.
I have created a wrapper for duplicity 
https://github.com/poggenpower/duplicity-backup/ to support this use 
case if separate backups per folder. It will also trigger full backups 
based on the amount of incremental backup made and not by time (it is 
still on my task list to implement this option in duplicity itself).

  Sorry no easy button ‾\_(ツ)_/‾

Bye

   Thomas


On 15 August 2024 07:02:10 CEST, Jakob Bohm via Duplicity-talk 
<[email protected]> wrote:

    Dear Duplicity community. As I am backing up lots of data with
    Duplicity, I find that the biggest backup volume takes over 3 weeks
    to do a full backup (starting a new chain), seemingly bottlenecked
    by the processing speed of AWS storage or the speed of the local
    router. This creates a special problem: If anything interrupts this
    3 week long upload process (Such as: Router failure, AWS outage,
    process interrupt or local machine crash), there is no clear way to
    continue the backup without starting over the entire backup .  It
    would be really nice if there was a clear way to make the backup
    continue where it left off, as long as at least some of the local
    Duplicity state directories survive the interruption. Enjoy Jakob

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk