Warning about certain AWS S3 storage classes
Mirko Vogt via Duplicity-talk <[email protected]>
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
Hello, I'd like to share a first-hand nightmare story with duplicity and Amazon's AWS S3 as storage backend. This is not about a failed backup/recovery in the way of data loss, but more like money loss. Lot's of tutorials suggest using duplicity's `--s3-use-ia` option when backing up to AWS S3 buckets. It changes the AWS S3 storage class of stored objects in respective bucket to a more cost efficient one, if requests are low and objects are stored >=30 days. From the option's description in duplicity: "This storage class has a lower storage cost but a higher per-request cost, and the storage cost is calculated against a 30-day storage minimum. According to Amazon, this storage is ideal for long-term file storage, backups, and disaster recovery.". What it does *not* mention, though, is: there's a penalty if objects are stored <30 days. And I'm not talking about "they're billed for 30 days regardless of if they get deleted before", but an actual penalty. On the invoice it's stated as: "$0.0135 per GB-Month prorated for objects deleted or overwritten before 30 days in Standard-Infrequent Access 3,113.091 GB-Mo" One might be inclined to think, that this does not usually apply, as backups are usually stored >30 days - and that was indeed the case for me the past months or even years. What most likely happened in my very case now, though, is, that my duplicity script /attempted/ to do a (full) backup, but failed - after already having written hundreds of GB to AWS. Next time the script was called (usually once a day by cron) it cleaned up the incomplete data from the previously aborted session. Hence, hundreds of GBs of objects on IA-storage did only stay there around 1 day. My bill for August increased by literally 822%. Long story short: While this is definitely not duplicity's fault, I'd suggest adding this as a warning to the docs / man-page for the `s3-use-ia`-option. There are also penalties for other storage classes supported by duplicity, namely the various glaciers / deep-archiving ones. Those might also deserve a warning. Best mirko _______________________________________________ Duplicity-talk mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/duplicity-talk