Data protection is top of mind of our customers, and having a data back up strategy is critical to ensure compliance, disaster recovery readiness, and business continuity. For years, some customers are able to absorb the costs of keeping data in an active storage tier. As they experience exponential growth in business and the volume of data, it becomes very costly for them to retain all of that data, especially infrequently accessed data. In addition, some customers risk non-compliance by deleting such data. One cost-savings solution is moving seldom-accessed data into a cold-storage tier, but in doing so it is important to track and monitor the amount of archived data to stay under your archival limits while generating the most cost savings possible.
Amazon Elastic Block Store (EBS) customers use EBS Snapshots to capture a point-in-time copy of their EBS resources for backup compliance and disaster recovery. With the introduction of EBS Snapshot Archive, a new storage tier for archival, customers can now optimize storage costs and save up to 75% with this low-cost, long-term storage for rarely-accessed snapshots that do not need frequent or fast retrieval. In a recent announcement of the increase in archival limit, customers can now archive 25 snapshots concurrently to the Snapshots Archive tier by default. This default limit increase makes it easier to move snapshots to the Snapshot Archive tier at scale. However, for customers trying to further reduce costs by taking the full advantage of the default limit increase, they will need to be cognizant in selecting and submitting the request, keeping track of the archival limits, and monitoring the progress of the archival process.
In this blog post, we will create a custom AWS CloudWatch dashboard using an AWS CloudFormation template to help you better monitor and keep track of snapshot archival. The proposed solution will automatically select eligible snapshots for archival based on the tags and their age. It will also control the rate of requests to avoid throttling. Finally, it will provide a graphical overview of archived snapshots, current progress, and historical savings using AWS CloudWatch Metrics.
This solution consists of creating an AWS CloudWatch dashboard with an AWS CloudFormation template. Once the solution is successfully implemented, you can access a Snapshot Archive Dashboard inside the CloudWatch console that shows the total number of snapshots in the account from the selected region, number of Archives in progress, and cost savings. You will gain the ability to track snapshot archival progress, monitor concurrent in-progress snapshot archive limit per account, and estimate cost savings in real-time.
The solution involves a serverless approach using AWS Lambda, Amazon EventBridge, Amazon Simple Queue Service, and Amazon CloudWatch. The following image maps out the high-level diagram of components that will be deployed by the AWS CloudFormation template provided in this blog post:
In order to have all the components in the above diagram working together and create the custom dashboard, complete the following steps:
- Log in to the AWS Management Console and go to CloudFormation service.
- Create a new stack by launching CloudFormation template.
- Specify stack parameters.
- Add tags resources to track in Cost Explorer.
- Acknowledge AWS CloudFormation and select Create stack.
As mentioned in the previous section, the solution is built on a serverless approach using AWS Lambda, Amazon EventBridge, Amazon Simple Queue Service, and Amazon CloudWatch. Therefore, you will need to set up an AWS account with sufficient permissions to create resources in the aforementioned services.
Step 1: Log in to the AWS Management Console and go to CloudFormation service
- Log in to AWS Management Console and select the adequate region.
- Prepare to deploy a new stack to create resources using with the following link: Creating a stack on the AWS CloudFormation console.
Step 2: Create a new stack by launching CloudFormation template
- Launch the following CloudFormation template directly from the CloudFormation console by specifying this URL as Amazon S3 URL.
Step 3: Specify stack parameters
Specify the following parameters to determine the snapshot eligibility for archival and the operational options of the tool:
- How old snapshots must be before archiving: The minimum number of days the snapshot must exist before being archived.
- Tags: Only archive snapshots with specific tag Key and Values. The criteria for selection of snapshots eligible for archival is based on the snapshot age, and optionally, tags. If no tags are provided, the archival solution will try to archive every snapshot older than the age in days defined while deploying the CloudFormation template. The default value is 90 days. Tags consist of Key/Value pairs and the basic syntax to specify a tag is:
tag: mandatory prefix (including colon character)
tag_name: the “Key” for the tag
tag_value: the value expected for the respective key
As an example, the following resource:
This would be represented by the following configuration:
It is possible to extend the selection to resources with the same tag Key and different values. Using the example above, if you want to archive snapshots from Staging and Production environments you would use the following configuration:
- ExecTime: The time in UTC when the tool will scan for new snapshots eligible for archival. As an example, the tool will scan the existent standard snapshot at 04:00AM UTC by selecting “4” in this option.
- LogLevel: the level of details provided in the logs.
- LogRetention: How many days the logs resulting of the tool execution will be kept.
- DryRun: If enabled (true), the tool will only simulate the archival request, without actually changing the snapshot tier. Useful for tests or troubleshooting.
Step 4: Add tags resources to track in Cost Explorer
- It is possible to add tags to the resources used by the tool, allowing to keep track of the costs with Cost Explorer. Note that the tags defined under Configure stack options step have no effect in the tool functionality:
Step 5: Acknowledge AWS CloudFormation and select Create stack
- Acknowledge AWS CloudFormation to create IAM resources and Auto Expand:
2. Once the stack creation is completed, you can navigate to the Amazon CloudWatch console and select Dashboard, then select the created “SnapshotArchiveDashboard.”
You can see the following details in your dashboard:
- Period: time period in which data points will be selected and consolidated on the dashboard widgets. Affects the details in the following widgets:
- Monthly costs
- Savings in the period
- Log Messages
The remaining widgets reflect the current state and won’t change if a different period is selected.
- Monthly costs: the amount spent with each EBS snapshot tier per month.
- Snapshots distribution: Graphical representation of EBS snapshot usage in number of resources per tier.
- Total snapshots: numeric representation of current usage of standard snapshots and the total size allocated for them.
- Savings in the period: Amount saved by archived EBS snapshots if compared with the standard tier costs.Standard snapshots created after archives are full snapshots. The “gross savings” value doesn’t consider costs of subsequent full standard snapshots while “relative savings” subtracts costs of the first full standard snapshots created after archives.
- Archives in progress: Number of EBS snapshots being archived at the moment.
- Queue size: Number of EBS snapshots waiting to be archived.
To avoid incurring future charges, delete the stack.
In this blog post, we presented a solution for implementing a custom CloudWatch dashboard to analyze EBS Snapshots for archival and cost saving purposes. With the help of this live dashboard, you can track snapshot archives in the Region per account and look at the cost savings in real time. This is important because you can now better plan your backup strategy by having the ability to track archival progress, monitor concurrent in-progress snapshot archive limit per account, and estimate cost savings. With real-time monitoring and increased visibility, you can reduce the risk of non-compliance by avoiding the deletion of snapshots for exceeding archival limit.
Thanks for reading this blog on live monitoring and cost savings on AWS. If you have any comments or questions, leave them in the comments section.