Re: High S3 503 SlowDo wn throttling during pea k writes with deduplica tion.enable=true (v3.9.0)
Benoit TELLIER <[email protected]>
| Newsgroups | gmane.comp.jakarta.james.user |
|---|---|
| Message-ID | <[email protected]> |
Hello Nice feedback, thanks a lot for sharing it ! The difference might be that if writing always same bodies it actually write to the same objects, concurrently. I would encourage to run the S3 test suite against this environment ( ) to check for overall compliance. Just hard code endpoint + keys in the test configuration. Feel free to share test results here. > We suspect this may be caused by single-partition prefix hot-spotting in S3 when generating content-addressed SHA-256 BlobIds You can triage MinIO compatibility mode that inserts some / in the blobIds emulating a folder structure. CF https://james.staged.apache.org/james-project/3.10.0/servers/distributed/configure/blobstore.html#_improve_listing_support_for_minio > RecommendationsDefault works well on OVH S3 to be honnest. Pooling set up: setting it higher (5000) is generally beneficial. Recent improvements: not anything that significantly changes the game but small enhancements are possible. The most significant being: - JAMES-4182 Implement ZstdBlobStoreDAO, allowing blob compression (#3016) - JAMES-4131 Configure a fallback bucket for S3 (#2719) -- Best regards, Benoit TELLIER General manager of Linagora VIETNAM. Product owner for Twake-Mail product. Chairman of the Apache James project. Mail: [email protected] Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) On Aug 18, 2026 10:56 AM, from James A Via Server-user <[email protected]>Hi James Support Team, We are running a distributed Apache James deployment (v3.9.0) backed by Cassandra 4.1.x, RabbitMQ 3.x, and AWS S3 as our BlobStore. We are currently load testing. During our load testing, we observed a distinct issue with S3 throttling: With deduplication.enable=true: We experience recurring 503 SlowDown error spikes from AWS S3 under high write/spooling throughput. With deduplication.enable=false: The 503 SlowDown errors disappear entirely under the exact same traffic load. Because deduplication provides S3 cost and capacity savings, we want to keep it enabled. We suspect this may be caused by single-partition prefix hot-spotting in S3 when generating content-addressed SHA-256 BlobIds, or synchronous hash calculation bottlenecks under heavy concurrency. Could you please advise on: 1. Recommended s3.properties / blob.properties tuning for content-addressed deduplicated BlobStores at scale (e.g., prefix hashing / sub-bucket routing configurations to distribute S3 partition keys). 2. Recommended AWS SDK / connection pool settings (such as objectstorage.s3.http.concurrency or retry/backoff policies) inside James to gracefully handle S3 rate limits. 3. Any known fixes or improvements in more recent James releases regarding BlobStore deduplication and S3 partitioning. Thanks in advance for your guidance!