Re: Performance issues w/ S3

Hi,

Duplicity has no code logic for throttle bandwidth.

Volume creation and transfer via backend are pretty well isolated.

I would do two tests to get a better picture.
Check speed of duplicity running the same backup to a local filesystem using file://. (You can cancel the backup if you have an idea about the speed before running out of disk space)
And run a boto3 code snippet and upload a 200MB file to your storage independently from duplicity.

Then you should see what's causing the bottleneck.

Bye
 Thomas


On 11 July 2024 23:56:47 CEST, tink via Duplicity-talk <[email protected]> wrote:
>Hi,
>
>I've been banging my head against this for a while. (duplicity 2.2.3)
>I'm trying to backup a sizeable chunk of data to Wasabi S3 storage, and
>using boto3+s3 with
>   --volsize 200 \
>   --s3-multipart-chunk-size 20 \
>   --s3-multipart-max-procs 6 \
>am seeing horribly slow upload speeds (shuffling 400GB in 24 h, so roughly
>5 MB/s).
>
>Using aws cli tools on the same server to the same bucket I get 80MB/s.
>Looking at a tcpdump during both attempts the latter looks fine and swift,
>the duplicity one is seeing the far end throttling me badly, every other
>second it tells me to back off for 1.5 seconds (setting a win-size of 0 for
>that period of time).
>
>
>What am I doing wrong here?
>
>Cheers,
>Tink
>
>--
>A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>A: Top-posting.
>Q: What is the most annoying thing in e-mail?

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk