Re: Recovering from an interruption during backup

hey Jakob,

completely agree with Thomas here. you should either
1. split your backup in smaller backup sets. not sure that's helpful assuming the "speed" stays the same splitting it evenly would need 21 sets just for runs around one day length.
2. just create your backups to a local file:// backend destination and sync independently from that, like e.g. with rclone.

just out of interest.
a. how much data do you backup that it takes that long?
b. what kind? small, huge files?
c. does the content change completely or stay overall the same?
d. did you change duplicity volume size?

sunny regards ..ede

On 15.08.2024 10:12, Thomas Laubrock via Duplicity-talk wrote:
> Hi Jakob,
>
> There are some countermeasures in duplicity to help at least with some issues. Tuning  "--backend-retry-delay" "--num-retries" "--concurrency" to recover from failures and speed up.
> But I think in your case you should review your setup.
> Duplicity is a file backup, but not a file sync solution. Even if duplicity is able to recover from all interruptions while running one backup over 3 weeks, it could cause inconsistencies, because duplicity is not doing a snapshot when it starts but reading each file at the time when it is placed in a volume. Means if there are file changes while the backup is running and files depend on each other, they may not align to each other.
>
> You should consider a different setup like:
> Do a backup to a local drive. Could be something like an USB3 disk, which is fast enough to complete timely. In a 2nd step sync the content of the USB disk to s3. There are tools out for this. So each volume can be transferred/retried separately, which should be a good granularity. Even if USB disks are know to be unreliable, having a local copy helps to speed up restore if the disk survived the disaster.
>
> An other option is to split backups into parts. E.g. I have a huge photo collection and create a dedicated backup per year, as photos of each year are stored in a separate folder. This allows to restart those backups more granularly. I also use the option `--skip-if-no-change` as most of the folders doesn't change or very rarely. As there is not advantage to do another full backup if nothing has changed, is saves a lot of unnecessary s3 transfers/costs.
> I have created a wrapper for duplicity https://github.com/poggenpower/duplicity-backup/ to support this use case if separate backups per folder. It will also trigger full backups based on the amount of incremental backup made and not by time (it is still on my task list to implement this option in duplicity itself).
>
>   Sorry no easy button ‾\_(ツ)_/‾
>
> Bye
>
>    Thomas
>
>
> On 15 August 2024 07:02:10 CEST, Jakob Bohm via Duplicity-talk <[email protected]> wrote:
>
>     Dear Duplicity community. As I am backing up lots of data with Duplicity, I find that the biggest backup volume takes over 3 weeks to do a full backup (starting a new chain), seemingly bottlenecked by the processing speed of AWS storage or the speed of the local router. This creates a special problem: If anything interrupts this 3 week long upload process (Such as: Router failure, AWS outage, process interrupt or local machine crash), there is no clear way to continue the backup without starting over the entire backup .  It would be really nice if there was a clear way to make the backup continue where it left off, as long as at least some of the local Duplicity state directories survive the interruption. Enjoy Jakob
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk