Re: Performance issues w/ S3

Hi Thomas,


On Mon, 22 Jul 2024 at 21:57, Thomas Laubrock via Duplicity-talk <
[email protected]> wrote:

> Hi Tink,
> Am 21.07.24 um 23:21 schrieb tink via Duplicity-talk:
>
> And without compression and encryption the local version takes 55 seconds.
>
> The compression and encryption is done by feeding the data into GnuPG
> binary installed on your system.
>
> I suggest following tests:
>
> - use `gpg` and encrypt a file manually and check the timing.
>
Encrypting a 10GB file (using compression) with GPG took  6m34s (again
reading from rust, writing to SSD).
gpg with no compression takes 37s


> While running following commands my help to find the bottleneck:
>
> `vmstat -w 5`
>
> or
>
> `iostat`
>
> to check I/O wait
>
> `watch ps -p <put pid here> -L -o pid,tid,psr,pcpu,docker,args`
>
> allow to see how the load is spread between threads and which CPU they
> running on. (keep in mind that 100% means full usage of all cores in your
> system. In your case 6.25% means the process is occupying a full core )
>
There's no I/O wait the whole time, the one CPU sits at 96-100% and the
overall average 5 minute load is ~ 1.3


> Btw. are you running on bare metall or any virtualization?
>
The backed up server is bare metal


>
> Bye
>
>  Thomas
>

> Keep in mind, that python by default is not able to take advantage from
> multiple cores. Duplicity has some functionality to share load across
> processes to bypass this limitation, but this still have some boundaries.
>
Yeah  -  I was hoping to be able to get at least a few parallel encryptions
going and ideally
also a few parallel network upload streams (as aws cli does).

Cheers,
Tink

_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk