Re: Performance issues w/ S3

Thanks Thomas,

I'll give the push to a local filesystem a shot on Monday when I'm back at
work, and may even try to wrangle up a small Python wrapper and push a file
up using just boto3.

Cheers,
Tink

On Sat, 13 Jul 2024 at 05:57, Thomas Laubrock via Duplicity-talk <
[email protected]> wrote:

> Hi,
>
> Duplicity has no code logic for throttle bandwidth.
>
> Volume creation and transfer via backend are pretty well isolated.
>
> I would do two tests to get a better picture.
> Check speed of duplicity running the same backup to a local filesystem
> using file://. (You can cancel the backup if you have an idea about the
> speed before running out of disk space)
> And run a boto3 code snippet and upload a 200MB file to your storage
> independently from duplicity.
>
> Then you should see what's causing the bottleneck.
>
> Bye
> Thomas
>
>
> On 11 July 2024 23:56:47 CEST, tink via Duplicity-talk <
> [email protected]> wrote:
> >Hi,
> >
> >I've been banging my head against this for a while. (duplicity 2.2.3)
> >I'm trying to backup a sizeable chunk of data to Wasabi S3 storage, and
> >using boto3+s3 with
> >   --volsize 200 \
> >   --s3-multipart-chunk-size 20 \
> >   --s3-multipart-max-procs 6 \
> >am seeing horribly slow upload speeds (shuffling 400GB in 24 h, so roughly
> >5 MB/s).
> >
> >Using aws cli tools on the same server to the same bucket I get 80MB/s.
> >Looking at a tcpdump during both attempts the latter looks fine and swift,
> >the duplicity one is seeing the far end throttling me badly, every other
> >second it tells me to back off for 1.5 seconds (setting a win-size of 0
> for
> >that period of time).
> >
> >
> >What am I doing wrong here?
> >
> >Cheers,
> >Tink
> >
> >--
> >A: Because it messes up the order in which people normally read text.
> >Q: Why is top-posting such a bad thing?
> >A: Top-posting.
> >Q: What is the most annoying thing in e-mail?
> _______________________________________________
> Duplicity-talk mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


-- 

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk