Re: Performance issues w/ S3

On Tue, 23 Jul 2024 at 20:53, Thomas Laubrock via Duplicity-talk <
[email protected]> wrote:

> Hi Tink,
> Am 23.07.24 um 04:13 schrieb tink via Duplicity-talk:
>
> On Mon, 22 Jul 2024 at 21:57, Thomas Laubrock via Duplicity-talk <
> [email protected]> wrote:
>
>> Am 21.07.24 um 23:21 schrieb tink via Duplicity-talk:
>>
>> And without compression and encryption the local version takes 55 seconds.
>>
>> The compression and encryption is done by feeding the data into GnuPG
>> binary installed on your system.
>>
>> I suggest following tests:
>>
>> - use `gpg` and encrypt a file manually and check the timing.
>>
> Encrypting a 10GB file (using compression) with GPG took  6m34s (again
> reading from rust, writing to SSD).
> gpg with no compression takes 37s
>
> OK, there is not much duplicity can do here, as it rely on gpg. Maybe you
> have any strange gpg setting in your system. Working with gpg-key in
> opposite with symmetric passphrase may also have an advantage.
>
> IMHO gpg itself has no option to run on multiple cores.
> Because of the architecture of duplicity volumes must be created in
> sequence.
>
> Long story short, if you are able to speed up gpg, you duplicity will get
> the same advantage.
>
> Upload to AWS seems not to be your bottleneck, but if try
> `--concurrency=X`. In my env volume creation worked fast enough to feed 2-3
> uploads. Keep in mind that parallel upload will pretty fast saturate other
> limits like network bandwidth or server limits.
>

Hi Thomas,

I can't quite do `--concurrency` on the server proper yet, haven't managed
to get 3.0.0 to behave.
Will keep trying to experiment.

As an aside: using 3.0.0 on a debian 12 VM and testing a 10G transfer I get
twice the speed to AWS S3 that I get to Wasabi, which is somewhat worrying.

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk