Re: Warning about certain AWS S3 storage classes

Mirko,

I understand your nightmare.  But I am not quite sure of the remedy.  Trying to adapt duplicity documentation to specific billing structure by AWS seems to be a different nightmare.  Maybe a generic warning for all backends, that cleaning up after an aborted backup can result in a lot of short term files being deleted?  Whether that has billing effects is up to the user and Amazon to disclose.

On a related note, you might want to look at Wasabi as an S3 storage platform.  They have no transfer fees and a simple billing structure. (No direct involvement with them, just a long time customer).

-Scott



> On Sep 8, 2024, at 23:22, Mirko Vogt via Duplicity-talk <[email protected]> wrote:
> 
> Hello,
> 
> I'd like to share a first-hand nightmare story with duplicity and Amazon's AWS S3 as storage backend.
> This is not about a failed backup/recovery in the way of data loss, but more like money loss.
> 
> Lot's of tutorials suggest using duplicity's `--s3-use-ia` option when backing up to AWS S3 buckets.
> 
> It changes the AWS S3 storage class of stored objects in respective bucket to a more cost efficient one, if requests are low and objects are stored >=30 days. From the option's description in duplicity:
> "This storage class  has  a  lower storage cost but a higher per-request cost, and the storage cost is calculated against a 30-day storage minimum.
> According to Amazon, this storage is ideal for long-term file storage, backups, and disaster recovery.".
> 
> What it does *not* mention, though, is: there's a penalty if objects are stored <30 days. And I'm not talking about "they're billed for 30 days regardless of if they get deleted before", but an actual penalty.
> 
> On the invoice it's stated as:
> 
> "$0.0135 per GB-Month prorated for objects deleted or overwritten before 30 days in Standard-Infrequent Access 3,113.091 GB-Mo"
> 
> One might be inclined to think, that this does not usually apply, as backups are usually stored >30 days - and that was indeed the case for me the past months or even years.
> 
> What most likely happened in my very case now, though, is, that my duplicity script /attempted/ to do a (full) backup, but failed - after already having written hundreds of GB to AWS.
> 
> Next time the script was called (usually once a day by cron) it cleaned up the incomplete data from the previously aborted session. Hence, hundreds of GBs of objects on IA-storage did only stay there around 1 day.
> 
> My bill for August increased by literally 822%.
> 
> Long story short: While this is definitely not duplicity's fault, I'd suggest adding this as a warning to the docs / man-page for the `s3-use-ia`-option.
> 
> There are also penalties for other storage classes supported by duplicity, namely the various glaciers / deep-archiving ones. Those might also deserve a warning.
> 
> Best
> 
>  mirko
> 
> _______________________________________________
> Duplicity-talk mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk