Re: Performance issues w/ S3

Hi Tink,

Then I think duplicity is spending the time on the volume generation side.

Did you make a test for the local file system?

Run duplicity in debug mode `-v d` and with `--log-timestamp` could help to see where duplicity is spending its time. 

Further you can add --no-compression and --no-encryption to reduce extra cycle on the volume creation side. 

Can you describe what your backup data looks like. Many small files, one big file, ...

Bye
  Thomas

On 19 July 2024 02:13:07 CEST, tink via Duplicity-talk <[email protected]> wrote:
>Hi Thomas, All,
>
>Thank you so much for all the effort and sorry for my prolonged silence.
>
>I have now tested w/ aws cli, Thoma's python snippet and duplicity
>(versions duplicity 0.8.22 on Debian 12, duplicity 0.8.12 on Ubuntu 20.04
>and duplicity 2.2.3 on alpine) ... in all cases aws cli is fastest, the
>python code somewhat
>slower (adding some 35% transfer time) and duplicity crawling & stalling
>.... a 10 G transfer that took 2 minutes to wasabi using aws cli taking
>over 15+/-5 minutes in each instance.
>Less stalling against aws than against wasabi, but still ...
>
>
>
>
>On Mon, 15 Jul 2024 at 01:41, Thomas Laubrock via Duplicity-talk <
>[email protected]> wrote:
>
>> Hi Ede
>>
>> Am 13.07.24 um 18:11 schrieb edgar.soldin--- via Duplicity-talk:
>> > maybe it's the multi_chunking? duplicity's s3 backend uses some
>> > default config vars
>> >
>> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252
>>
>>
>> As written the uploads saturating my upload limits, therefore I don't
>> think that this will have heavy impact.
>>
>> Netto upload times from pur bot3 and duplicity are the same.
>>
>> >
>> >
>> > to set up a custom TransferConfig and finally use upload_file()
>> >
>> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186
>> >
>> >
>> > afaiu the chunking only happens if the file is bigger than the given
>> > chunk size. so maybe this should be lowered or the test file expanded 🙂
>>
>>
>> Me and Tink using the default Volume Size of 200 MB, chunks are 20MB by
>> default
>>
>>
>> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py#L243-252
>>
>>
>> Thoses setting sound pretty reasonable to me. I don't think that
>> chunking will have heavy impact here.
>>
>> The boto3 default chunk size is a 8MB
>>
>>
>> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html
>>
>>
>> Bye
>>
>>   Thomas
>>
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> [email protected]
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
>
>-- 
>
>A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>A: Top-posting.
>Q: What is the most annoying thing in e-mail?

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk