Re: Performance issues w/ S3
"edgar.soldin--- via Duplicity-talk" <[email protected]>
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
On 13.07.2024 16:44, Thomas via Duplicity-talk wrote: > Hi Tink, > > > Thanks to chatGPT, it took a few minutes to create a test script: > > https://gist.github.com/poggenpower/7f629a2c1d80e22e01caaa25c9567ab9 > > I am ending with a transfer speed around 16 MB/s (~12 sec / 200MB) regardless if I use boto3 1.34.144 or 1.34.108 and regardless if I test and upload to AWS or Wasabi. I think I am limited by my upload bandwidth (150MBit/s) here. > > A full duplicity backups of 200MB data take ~21 sec regardless if destination is AWS or Wasabi. If you use `--log-timestamp -v i` then you can check to upload time per volume. > > 2024-07-13 14:22:13,178 INFO Uploading s3://test-py//duplicity-full.20240713T142201Z.vol1.difftar.gpg to STANDARD Storage > 2024-07-13 14:22:24,846 INFO Processed volume 1 > > I am ending up with ~12 sec per volume, which is the same as using pur boto3. maybe it's the multi_chunking? duplicity's s3 backend uses some default config vars https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252 to set up a custom TransferConfig and finally use upload_file() https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186 afaiu the chunking only happens if the file is bigger than the given chunk size. so maybe this should be lowered or the test file expanded :) sunny regards ..ede _______________________________________________ Duplicity-talk mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/duplicity-talk