Re: Performance issues w/ S3
"edgar.soldin--- via Duplicity-talk" <[email protected]>
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
On 23.07.2024 04:13, tink via Duplicity-talk wrote: > Encrypting a 10GB file (using compression) with GPG took 6m34s (again reading from rust, writing to SSD). > gpg with no compression takes 37s out of interest just played a bit on a local debian box with gpg options, which could be given as `--gpg-options` to duplicity --compress-algo name https://www.gnupg.org/documentation/manuals/gnupg/GPG-Esoteric-Options.html#index-compress_002dalgo --compress-level n https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration-Options.html#index-compress_002dlevel tl;dr gpg 2.2.40 only utilized one core zip and zlib (gpg-default) are 6times slower than uncompressed bzip2 is way way slower tried zlib compression levels but couldn't determine a notable difference in conclusion. so smaller volumes with duplicity 3.x `--concurrency` may be a way to saturate a broad upload bandwidth. but overall it still does not account for a 5MB/s throughout. even your 10GB/6m34s would be ruffly 25MB/s. sunny regards ..ede -- # create test file :/data/tmp$ dd if=/dev/random of=1gb bs=$((1024*1024)) count=$((1*1024)) 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.4176 s, 65.4 MB/s # test is a passphraseless test key `gpg --batch --passphrase '' --quick-gen-key test default default` # gpg tests sorted by runtime descending :/data/tmp$ time gpg --encrypt -r test --compress-algo bzip2 --output 1gb.asym.bzip2 1gb real 2m44.136s user 2m41.990s sys 0m1.599s :/data/tmp$ time gpg --encrypt -r test --output 1gb.asym.def 1gb real 0m33.228s user 0m31.631s sys 0m1.590s :/data/tmp$ time gpg --encrypt -r test --compress-algo zlib --output 1gb.asym.zlib 1gb real 0m32.356s user 0m30.700s sys 0m1.636s :/data/tmp$ time gpg --encrypt -r test --compress-algo zip --output 1gb.asym.zip 1gb real 0m29.357s user 0m27.759s sys 0m1.595s :/data/tmp$ time gpg --encrypt -r test --compress-algo none --output 1gb.asym.none 1gb real 0m5.296s user 0m3.935s sys 0m1.361s # compression level tests, 6 is the default value according to man page :/data/tmp$ time gpg --encrypt -r test --compress-algo zlib --compress-level 9 --output 1gb.asym.zlib.9 1gb real 0m32.433s user 0m30.714s sys 0m1.694s :/data/tmp$ time gpg --encrypt -r test --compress-algo zlib --compress-level 4 --output 1gb.asym.zlib.4 1gb real 0m32.298s user 0m30.626s sys 0m1.654s :/data/tmp$ time gpg --encrypt -r test --compress-algo zlib --compress-level 2 --output 1gb.asym.zlib.2 1gb real 0m30.869s user 0m29.165s sys 0m1.696s :/data/tmp$ time gpg --encrypt -r test --compress-algo zlib --compress-level 1 --output 1gb.asym.zlib.1 1gb real 0m31.110s user 0m29.425s sys 0m1.663s _______________________________________________ Duplicity-talk mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/duplicity-talk