Re: Performance issues w/ S3

Hi Thomas, All,

Thank you so much for all the effort and sorry for my prolonged silence.

I have now tested w/ aws cli, Thoma's python snippet and duplicity
(versions duplicity 0.8.22 on Debian 12, duplicity 0.8.12 on Ubuntu 20.04
and duplicity 2.2.3 on alpine) ... in all cases aws cli is fastest, the
python code somewhat
slower (adding some 35% transfer time) and duplicity crawling & stalling
.... a 10 G transfer that took 2 minutes to wasabi using aws cli taking
over 15+/-5 minutes in each instance.
Less stalling against aws than against wasabi, but still ...




On Mon, 15 Jul 2024 at 01:41, Thomas Laubrock via Duplicity-talk <
[email protected]> wrote:

> Hi Ede
>
> Am 13.07.24 um 18:11 schrieb edgar.soldin--- via Duplicity-talk:
> > maybe it's the multi_chunking? duplicity's s3 backend uses some
> > default config vars
> >
> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252
>
>
> As written the uploads saturating my upload limits, therefore I don't
> think that this will have heavy impact.
>
> Netto upload times from pur bot3 and duplicity are the same.
>
> >
> >
> > to set up a custom TransferConfig and finally use upload_file()
> >
> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186
> >
> >
> > afaiu the chunking only happens if the file is bigger than the given
> > chunk size. so maybe this should be lowered or the test file expanded 🙂
>
>
> Me and Tink using the default Volume Size of 200 MB, chunks are 20MB by
> default
>
>
> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py#L243-252
>
>
> Thoses setting sound pretty reasonable to me. I don't think that
> chunking will have heavy impact here.
>
> The boto3 default chunk size is a 8MB
>
>
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html
>
>
> Bye
>
>   Thomas
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


-- 

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk