Re: Performance issues w/ S3
"edgar.soldin--- via Duplicity-talk" <[email protected]>
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
On 14.07.2024 15:40, Thomas Laubrock via Duplicity-talk wrote: > Hi Ede > > Am 13.07.24 um 18:11 schrieb edgar.soldin--- via Duplicity-talk: >> maybe it's the multi_chunking? duplicity's s3 backend uses some default config vars >> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252 > > > As written the uploads saturating my upload limits, therefore I don't think that this will have heavy impact. > > Netto upload times from pur bot3 and duplicity are the same. was just speculating :) >> >> >> to set up a custom TransferConfig and finally use upload_file() >> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186 >> >> afaiu the chunking only happens if the file is bigger than the given chunk size. so maybe this should be lowered or the test file expanded 🙂 > > > Me and Tink using the default Volume Size of 200 MB, chunks are 20MB by default > > https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py#L243-252 yeah misread it for 200, my bad > Thoses setting sound pretty reasonable to me. I don't think that chunking will have heavy impact here. > > The boto3 default chunk size is a 8MB > > https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html right. sorry for the noise then. sunny regards ..ede _______________________________________________ Duplicity-talk mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/duplicity-talk