top of page

Getting Started with Amazon CUR and Athena - How much am I spending on a bucket?

Ever realize how hard it is to find out exactly what something costs you in AWS? It's meant to be that way, let's learn how to expose these costs. This series is dedicated to helping you set up the right tooling to understand your AWS Costs. By the end of this article you'll be able to calculate cost of your S3 buckets, and a bunch of other things.... it's only going to take eight hours, why's it so slow? No one knows (Someone probably does)

Resources reated in this article have associated usage costs


WHAT DO I NEED TO START?

  • An AWS Account with active resources

  • Access to an IAM User or Role on AWS with at least Power User level permissions.

  • If you're using the CLI commands you'll need AWS CLI installed and configured


CONFIGURING THE BUCKET

If you're using the GUI, go ahead on to the next step - the bucket will be created during the process.

Execute the below command

aws s3 mb s3://$bucket/

We'll need to modify the bucket policy, create a file with the below content - replace $bucket with your bucket name, then execute the command below

{
	"Version": "2008-10-17",
	"Id": "Policy1335892530063",
	"Statement": [
		{
			"Sid": "Stmt1335892150622",
			"Effect": "Allow",
			"Principal": {
				"Service": "billingreports.amazonaws.com"
			},
			"Action": [
				"s3:GetBucketAcl",
				"s3:GetBucketPolicy"
			],
			"Resource": "arn:aws:s3:::$bucket"
		},
		{
			"Sid": "Stmt1335892526596",
			"Effect": "Allow",
			"Principal": {
				"Service": "billingreports.amazonaws.com"
			},
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::$bucket/*"
		}
	]
}
aws s3api put-bucket-policy --bucket $bucket --policy file://$filename

CREATE THE REPORT

Let's create the Cost and Utilization Report, this will gather the data from your utilization and dump it into an S3 bucket. From there, we'll be able to query it.

CLI

Create a file with the following content, replacing the values with your pre-created bucket - The region must be the same as the bucket!


{
		"ReportName": "CATESTCUR",
		"TimeUnit": "HOURLY",
		"Format": "Parquet",
		"Compression": "Parquet",
		"AdditionalSchemaElements": [
			"RESOURCES"
		],
		"S3Bucket": "$bucket",
		"S3Prefix": "cr",
		"S3Region": "$region",
		"AdditionalArtifacts": [
			"ATHENA"
		],
		"RefreshClosedReports": true,
		"ReportVersioning": "OVERWRITE_REPORT"
}

Execute the following command, this will create a CUR report definition

aws cur put-report-definition --report-definition file://$file


GUI

To get started go to the following billing dashboard once you're logged in on your account. Here

Enter your report name and select "include resource IDs", then select Next


Select "Configure Bucket" then "Next", then validate the policy and select "Save"


Select the delivery options

  • Create the path prefix

  • Set the granularity to Hourly

  • Enable Athena Integration

Select next and confirm on the next screen


WAIT 8 HOURS

Wait, what? The initial delivery of the report (And cloudformation template!) we need takes 8 hours


SET UP THE ATHENA INTEGRATION

It's time, we can finally set up the integration. We'll be creating a cloudformation stack that's going to create the below resources

  • AWS Glue Objects

  • Lambda Functions

  • SNS Topics

  • Athena Workgrounds

  • IAM Roles - To allow the writing of Cloudwatch Logs

CLI

Execute the following command, replacing the path with your bucket name and path prefix

aws cloudformation create-stack --stack-name curcfn --capabilities CAPABILITY_IAM --template-url https://$bucket.$region.amazonaws.com/$prefix/$reportname/crawler-cfn.yml

Wait for the stack to finish creating

GUI

Go to the following URL, Here, and select the correct region. Enter the the template URL for the generated object

https://$bucket.$region.amazonaws.com/$prefix/$report/crawler-cfn.yml


Enter in a stack name you'll remember, then select next, leave everything default on the next page and click next


Review the stack settings, scroll down to the bottom of the page and make sure you select "I acknowledge that AWS Cloudformation might create IAM Resources"


Wait for the stack to say "Complete"

FINALLY, ATHENA!

Amazon Athena lets you perform SQL Queries on a number of objects, such as ones stored in S3. We've set up a workspace and Glue catalog with the Cloudformation template, now we can query!

Go to the Athena Console here, Here, and select the correct region

Warning: You may receive a note that you need to create a results bucket before you can query, follow the instructions provided through the console

It's time to execute your first query, check the status of the CUR Database before querying anything else! Make sure the correct database is selected and enter the below query

select status from cost_and_usage_data_status

If it says ready, you're good to go!

HOW DO I FIND OUT HOW MUCH A BUCKET COST ME?

It's difficult to figure out in the console, but with CUR and Athena you'll be able to easily find out what costs have accrued after you've set up the reports!

The below query will gather the information about all products related to S3, group them in by bucket name and aggregate the costs together.

SELECT line_item_resource_id,
sum(line_item_blended_cost) AS cost, month
from CA 
WHERE line_item_product_code = 'AmazonS3'
GROUP BY line_item_resource_id, month
ORDER BY line_item_resource_id;

Enter the above query into the Athena GUI


We can see here the bucket "ws" is costing us $0.0087/Month including API calls - This hosts a version of our website for testing, wonder how it costs us so little? There'll be a blog article on that soon as well!

WANT TO LEARN MORE?

Over the next few weeks we'll be releasing more in-depth Amazon CUR and Athena queries and use-cases. In the meantime the best resource to help you write additional queries is understanding the line items in the report, AWS has a glossary of these on their website, here, https://docs.aws.amazon.com/cur/latest/userguide/Lineitem-columns.html

FEELING THE BURN FROM YOUR AWS BILL AND NEED IMMEDIATE RELIEF?

Reach out to our team on CloudAccountant.io, schedule a risk free 20 minute session with one of our experts to give you a demo of what we can really do. On average we've saved our clients 72% on their AWS bill, without any risk on their end

bottom of page