Re: Performance issues w/ S3
Thomas Laubrock via Duplicity-talk <[email protected]>
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
Hi Ede Am 13.07.24 um 18:11 schrieb edgar.soldin--- via Duplicity-talk: > maybe it's the multi_chunking? duplicity's s3 backend uses some > default config vars > https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252 As written the uploads saturating my upload limits, therefore I don't think that this will have heavy impact. Netto upload times from pur bot3 and duplicity are the same. > > > to set up a custom TransferConfig and finally use upload_file() > https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186 > > > afaiu the chunking only happens if the file is bigger than the given > chunk size. so maybe this should be lowered or the test file expanded 🙂 Me and Tink using the default Volume Size of 200 MB, chunks are 20MB by default https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py#L243-252 Thoses setting sound pretty reasonable to me. I don't think that chunking will have heavy impact here. The boto3 default chunk size is a 8MB https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html Bye Thomas _______________________________________________ Duplicity-talk mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/duplicity-talk