Amazon have an API for that
It was words to that affect I heard, I wish it was what I wanted though. After watching a colleague get metrics for the billing it was clear after a short look they were very good for estimating your total expense but they were never going to be an accurate figure.
So in short if you want to be disappointed by Amazon guessing your monthly cost you can find out some information Here.
If you wish to learn more, read on…
What do you want?
We were looking at the metrics because we wanted to report on our running costs on a granular basis so we could see the hourly cost of our account as and when new services were turned on or off or if we added / subtracted a node from an existing system. In addition to that snapshot we wanted to be able to compare one week to the next and with some other operational metrics such as users on line.
So after discussing for a a little while it was clear the Amazon metrics are good for predicting, not good for historical and ultimately not very accurate as it was always a potential and never an actual. I made the decision that we were better off getting the information our selves, which at first sounded crazy, and the more I think about it the more I agree, really amazon should offer this in a better way through their current billing.
Knowing what we wanted meant I could not bother with tracking the things we don’t care about. What is really important to us, is it the disk space being used? is it the bandwidth? cost of ELB’s? Nope, we just really care about how much does it cost for the instances we are running.
In the end that’s all that mattered, are we costing more money or less money and with this money are we providing more or less value. Ideally we will reduce cost and increase performance but until we start tracking the figures we have no idea what is actually going on unless we spend hours looking at a summarised cost and guessing backwards… well until now anyway.
Over the last 5 days I’ve spent some time knocking together some ruby scripts that will poll amazon and get back the cost of an account based on the current running instances across all regions. For us that is good enough, but I decided to add extra functionality by getting all fo the nodes as well, it sort of acts like an audit trail and will allow us to do more in depth monitoring if we so wish, for example… if we switch instance type does that save us more money and give more performance? Well we wouldn’t know either, especially if we didn’t track what was running.
We also wanted to track the number of objects in a bucket within S3, now our S3 buckets have millions objects in each of them, if you use the aws-sdk to get this it will take forever +1 day, if you use the right_aws it will still take a long time but not as much (over 30 mins for us). This isn’t acceptable so we’re looking at alternative ways to generate this number quicker, but the short answer is it’ll be slow, if I come up with a fancy s3 object count alternative i’ll let you know, but for now I have had to abandon that. Unless Amazon want to add two simpel options like s3.totalObjects and s3.totalSize…
It’s just data
So, we gathered our Amazon data, we gathered some data from our product, this was about a day into the project, all of this was done currently on the fly and it use to take 20 seconds or so to get the information. We had a quick review and it was decided we should track now vs the last week, this made a slight difference as it meant we now needed to store the data.
This is a good thing, by storing the data we care about we no longer have to make lots of long winded calls that hang for 20 seconds, it’s all local, speed++
Needless to say the joys of storing the data and searching back through it is all interesting, but I’m not going to go into them.
To take the data and turn it into something useful requires reports to be written, all the time it’s raw data no one will particularly care, once a figure is associated with a cost per user or a this account costs X per hour people care more. One of the decisions made was to take the data and do some abstraction of it separately to the formatting of the output, mainly as there’s multiple places to send the data, we might want to graph some in graphite, email some of the other data and squirrel away the rest in a CSV output somewhere.
An advantage of this is now that the data has been generated there’s one file to modify to chose how and what data should get returned which gives us the ability to essentially write bespoke reports for what ever is floating our boat this week.
A Freebie
I thought this might be useful, It’s the class we are using to get our instance data from amazon, it’s missing a lot error checking but so far it is functional, and as everyone knows before it can be useful it must first work.
#
# Get instance size cost
#
require 'net/http'
require 'rubygems'
require 'json'
require 'aws-sdk'
class AWSInstance
def initialize (access_key, secret_key)
@access_key_id=access_key
@secret_access_key=secret_key
end
def cost
instances = get_running_instances
cost = 0.00
#Calc Cost
instances.each_pair do |region,value|
value.each_pair do |instance_type, value|
cost += (value.size.to_f * price(instance_type,region).to_f)
end
end
return cost
end
def get_instances
#Return all running instaces as that has a cost in its hash
return get_running_instances
end
private
def price (api_size, region)
rhel_url = "/rhel/pricing-on-demand-instances.json"
url = "/ec2/pricing/pricing-on-demand-instances.json"
price = 0
size=""
instance_type=""
response = Net::HTTP.get_response("aws.amazon.com",rhel_url)
#puts response.body
# Convert to hash
json_hash = JSON.parse(response.body)
# api_size i.e. m1.xlarge
# Get some info to help looking up the json
case api_size
when "m1.small"
size = "sm"
instance_type = "stdODI"
when "m1.medium"
size = "med"
instance_type = "stdODI"
when "m1.large"
size = "lg"
instance_type = "stdODI"
when "m1.xlarge"
size = "xl"
instance_type = "stdODI"
when "t1.micro"
size = "u"
instance_type = "uODI"
when "m2.xlarge"
size = "xl"
instance_type = "hiMemODI"
when "m2.2xlarge"
size = "xxl"
instance_type = "hiMemODI"
when "m2.4xlarge"
size = "xxxxl"
instance_type = "hiMemODI"
when "c1.medium"
size = "med"
instance_type = "hiCPUODI"
when "c1.xlarge"
size = "xl"
instance_type = "hiCPUODI"
when "cc1.4xlarge"
size = "xxxxl"
instance_type = "clusterComputeI"
when "cc2.8xlarge"
size = "xxxxxxxxl"
instance_type = "clusterComputeI"
when "cg1.4xlarge"
size = "xxxxl"
instance_type = "clusterGPUI"
when "hi1.4xlarge"
size = "xxxxl"
instance_type = "hiIoODI"
end
# json_hash["config"]["regions"][0]["instanceTypes"][0]["sizes"][3]["valueColumns"][0]["prices"]["USD"]
json_hash["config"]["regions"].each do |r|
if (r["region"] == region.sub(/-1$/,''))
r["instanceTypes"].each do |it|
if (it["type"] == instance_type)
it["sizes"].each do |sz|
if (sz["size"] == size)
price=sz["valueColumns"][0]["prices"]["USD"]
end
end
end
end
end
end
return price
end
def get_running_instances
#Set up EC2 connection
ec2 = AWS::EC2.new(:access_key_id => @access_key_id, :secret_access_key=> @secret_access_key)
instance_hash = Hash.new
#Get a list of instances
#Memorize cuts down on chatter
AWS.memoize do
ec2.regions.each do |region|
instance_hash.merge!({region.name => {}})
region.instances.each do |instance|
if (instance.status == :running)
#Need to create a blank hash of instance_type else merge fails
if (!instance_hash[region.name].has_key?(instance.instance_type) )
instance_hash[region.name].merge!({instance.instance_type => {}})
end
#For all running instances
tmp_hash={instance.id => {:env => instance.tags.Env, :role => instance.tags.Role, :name => instance.tags.Name, :cost => price(instance.instance_type,region.name) }}
instance_hash[region.name][instance.instance_type].merge!(tmp_hash)
end
end
end
end
return instance_hash
end
end #End class
soimasysadmin 