Day 6 - S3 Object Expiry

AWS Cost Optimisation Advent Calendar 2024

4 min read time

#aws #s3

Welcome to day 6 of the unofficial AWS Cost Optimisation Advent Calendar 2024, where every day we will be sharing new tips or tricks to help you optimise your cloud costs before Christmas 2024.

No-one likes the idea of deleting data. Setting an expiry on your S3 objects or at the bucket level can help you manage your costs and ensure you are not storing data you will likely never need again.

As a best practice, you should periodically review your S3 buckets to see if any can benefit from S3 object expiry.

Some examples of buckets which typically can benefit are:

1. Temporary files: If you are using S3 to store temporary files for processing then you can set an expiry on these files to ensure they are deleted after a certain period of time. This can be particularly useful if you are using S3 as a temporary store for files which are being processed by a Lambda function.
2. Logs: If you are storing logs in S3 then you can set an expiry on these logs to ensure they are deleted after a certain period of time. This can be particularly useful if you are storing logs for compliance reasons but do not need to keep them for more than a certain period of time.
3. Backups: If you are storing backups in S3 then you can set an expiry on these backups to ensure they are deleted after a certain period of time. This can be particularly useful if you are storing backups for disaster recovery purposes but do not need to keep them for more than a certain period of time.
4. Old versions: If you are using versioning in S3 then you can set an expiry on old versions to ensure they are deleted after a certain period of time. This can be particularly useful if you are using versioning for compliance reasons but do not need to keep old versions for more than a certain period of time.

To set an expiry on your S3 objects you can use lifecycle policies which allow you to define rules for objects in your S3 bucket to automatically expire them after a certain period of time.

It also supports filters so you can apply different rules to different objects.

In addition, you can also expire objects based on size so if you have a bucket with variable size non-critical data you may decide to expire only excessively large objects.

Important Notes

Versioned Objects

If you have versioning enabled on your buckets then enabling expiration works slightly differently. All your old versions will be deleted however the current version will just have a delete marker making it non-current but still in the bucket. This can be useful if you need to recover the data but do not need it to be the current version. This does however also mean you are still paying for the object and this could be unexpected if you have set expiry as a cost optimisation.

Permanently Deleted

Objects which are expired are permanently deleted and cannot be recovered. This is particularly important to note if you are using S3 as a backup store and have set an expiry on your backups. This does mean that special attention should be given when setting expiry on objects.

Storage Lens

If you are managing a large amount of buckets and different object types you may benefit from using Storage Lens. This can help you monitor and report on your storage usage and costs. It is very useful for understanding your storage usage and can help you identify where you can save costs.

Conclusion

Just a quick one today but setting expiry on your S3 objects can be a great way to ensure you are not paying indefinitely for logs and other temporary data. It is therefore worth checking if any S3 buckets can benefit at least annually.

To be one of the first to know when the next edition is published please follow us on LinkedIn, X or subscribe to the RSS feed.