Now for something a little petty

Text is nice, graphs are pretty

Over the last few weeks I have been writing more and more metric gathering tasks to identify how the systems we use are being used and what is valuable about them. All of this wonderful metric information at the moment is text based and mailed out, but text representation works fine for things that are tangible, such as the cost or number of users, but what if you wan to know how many users are online at 8 am or 10 am or 1 pm? Well this is where something pretty comes into the mix.

As part of the metrics gathering I have been looking at various graph drawing tools and there are quite a few out there, some although technically brilliant are ugly, some or pretty but limited. Over a longer term it will probably make sense to use some javascript library to draw the graphs, but I wanted something now and we had a graphite server which was being used for some more generic stuff but I hadn’t done anything with it.

Get some data in

Graphite is pretty cool, you just send some very basic information to it and it tracks it, it can then take care of the display of the information and certain functions like the average or max of a metric. All of the graphs are drawn on the flu so you can change the time frame, add extra plots and all this good stuff. Initially I was put off by Graphite because it looks messy but I decided that now was a good time to learn.

The first challenge I had was putting some data into it, because it just takes a text string you can update it using net cat if you really wanted, but I decided to g for a pure ruby implementation.

#
# Graphite DAO
#

require 'socket'

class GraphiteDAO

  def initialize (server, port)
    @server = server
    @port = port
  end

  def put_graphite(metric_name, metric_value, date = Time.now.to_i)
    string = metric_name.to_s + " " + metric_value.to_s + " " + date.to_s
    #puts "string = #{string}"
    write(string)
  end


  private
  def write(data)
    socket = TCPSocket.open(@server, @port)
      socket.puts "#{data}"
    socket.close
  end
end

This is very functional and there is no error checking but its enough to get it working, as you can see I just take the metric name i.e. the path in graphite and the value, you can optionally pass in the date if you wish, but it made more sense to just use “now” as the date in most cases.

The metric name is a path in graphie, so the metric name I pass in may be “prod.application.concurrent_users” this allows for the data to be structured logically within graphite and is easier to recall later.

Getting something out

By far the most useful thing I learnt about graphite is changing the line mode to be connected, it turns the tiny line spec that I can’t see into a connected line… so now graphs are visible. In short graphite lets you select any number of metrics to graph onto one graph, and the default view is sort of a graph builder. This typically works fine if you only have 1 metric or you are comparing the same metric from three sources, however if the range between the lowest metric and the highest is too far, you just end up with a straight line for each, this is not a graphite issue but a general issue with graphs.

Now for the really useful stuff, you can use graphite to render specific graphs for you by calling a specific url to render your graphs, so once you know what the path is to the metric you can do all sorts of stuff.

for example…

http://my-graphite-server/render?target=prod.application.concurrent_users&width=800&height=600&lineMode=connected&title=Concurrent%20Users

Because it is just a matter of adding options of multiple targets it is easy to use, and the documentation is okay (and Here)

The other element that is really useful is drawing graphs that don’t exist for example

http://my-graphite-server/render?graphType=pie&target=alpha:111&target=beta:222&target=gamma:333

The rendering engine is good and I like it, so considering this whole thing only took 25 mins to work out it was well worth the effort in doing so, a quick return for not much effort

Sysadmins in a Developers world

It’s all back to front

Well it was about 9 months back when I was touching on Developers in a sysadmin world and my initial thoughts were along the lines of we are better at different tasks, and after spending a week doing only development I am of the same opinion still.

Over the last 6 months we have had our solitary developer, coding away making great things happen, predominately developing a portal that allows us to deploy environments in 15 mins vs the 2 days it took before and the whole things is very pretty, it even has its own Favicon.ico which we are all pleased about. In addition to just deploying, it also allows us to scale up and down the environments it creates and despite constant interruptions it is coming along really well and in the next month we will be providing it as a service to the engineering teams to self serve.

As more and more of our tools are developing we are also in-housing more and more of our tools. As the regular readers know I do dable with the odd slightly more complex program than the average sysadmin might tackle. When we are faced with a situation such as monitoring the operations, by this I mean, the number of user growth week on week and the cost of running the environment(s) it just made more sense to do it our selves. There are tools out there that provide various dashboards like Geckoboard which can all do approximately 80% of the job, but it’s that last 20% that adds the usefulness, as such we are trying to develop tolls that are pluggable and extensible and support multiple outputs. For example the Metrics report we have will also support Geckoboard, Graphite, Email and probably have it’s own web interface.

For us it is becoming more about having the flexibility to add and remove components and keeping the flexibility around it, this introduces challenges with what ever being written needing to be pluggable and easy to maintain, which often make sit complicated.

I used classes, as a necessity

Typically when I program there is not much need for classes or even objects for that matter, a simple var and some nice loops and conditional statements would be plenty. Well not so much anymore, The last project was metrics and as with other projects I got it working within a day or two, and I hated it, it took over 30 seconds for it to run and generate the report I needed but not in the right format and then the level of detail in the metrics was not high enough, it could manage weekly but it was not good enough.

I decided that I’d have a chat with a few developers to help with the structure of the application, at first I was dubious, but it turned out well. The key step which I wouldn’t have made until it was a real problem was to separate out the the tasks that gathers the raw data, the tasks that manipulates the data into useful numbers, the bit that stores the data, the bit that manipulates the data into useful numbers and then finally the bit that outputs the pretty data.

This was an evolutionary step, I would have got to the point of understanding the need to separate each step out but not until it had become a real big pain many months later. Another advantage of splitting it out was how much simpler each step was, there were classes defining methods for getting data that were being used in classes to format the data that were being used… you get the idea. Rather than being one class to connect to amazon, manipulate the data and return an object that could be used to generate the metrics everything was done on much smaller steps. As a result it was a lot easier to write small chunks of code “that just worked” and it made debugging a lot easier, and I feel like I progressed my understanding, and this is always a good thing.

Who should do what

I touched on this in my other post, but I want to amend it based on a better understanding. To summarise I pretty much said as it is, Developers develop, sysadmins admin. They do, and certainly that should be their focus, but I think there is a lot to be gained from both points of view when pushed to work in the others world.

Before our developer joined the focus was on making the build, test and release process better, after forcing the developer to do sysadmin work for a month or so while the team was trying to grow and cope with the loss of a team member, it became clear that the time wasted for us all was not getting a build though but by us not being able to paralise the testing or being agile enough to re-deploy an environment if it was not quite right. These steps and understandings would not have happened if we didn’t encroach on each others work and gain the understanding from the other persons perspective.

Summary

This is what DevOps is really about, forget sysadmins doing code, forget about developers doing sysadmin work, it is about us meeting in the middle and understanding the issues we each face and working together to solve bigger problems.

Simple facts with Puppet

It’s not that hard

When I first started looking at facter it was magic, things just happened and when I entered facter a list of variables appeared and all of these variables are available to use within puppet modules / manifests to help make life easier. After approximately 2 years of thinking how good they were and how nice it would be to have my own I finally took the time to look at it and try to work it out….

For those of you that don’t know, facter is a framework for providing facts about a host system that puppet can use to make intelligent decisions about what to do and can be used to determine the operating system, release of it, local IPs etc etc. This gives you flexibility in puppet to do things like choose what packages to install based on Linux distribution or insert the local IP address into a template.

Writing Facts

So, Lets look at a standard fact that comes with it so you can see the complexity involved an understand why after glancing at it I never went much further.

# Fact: interfaces
#
# Purpose:
#
# Resolution:
#
# Caveats:
#

# interfaces.rb
# Try to get additional Facts about the machine's network interfaces
#
# Original concept Copyright (C) 2007 psychedelys <psychedelys@gmail.com>
# Update and *BSD support (C) 2007 James Turnbull <james@lovedthanlost.net>
#

require 'facter/util/ip'

# Note that most of this only works on a fixed list of platforms; notably, Darwin
# is missing.

Facter.add(:interfaces) do
  confine :kernel => Facter::Util::IP.supported_platforms
  setcode do
    Facter::Util::IP.get_interfaces.collect { |iface| Facter::Util::IP.alphafy(iface) }.join(",")
  end
end

Facter::Util::IP.get_interfaces.each do |interface|

  # Make a fact for each detail of each interface.  Yay.
  #   There's no point in confining these facts, since we wouldn't be able to create
  # them if we weren't running on a supported platform.
  %w{ipaddress ipaddress6 macaddress netmask}.each do |label|
    Facter.add(label + "_" + Facter::Util::IP.alphafy(interface)) do
      setcode do
        Facter::Util::IP.get_interface_value(interface, label)
      end
    end
  end
end

This is stolen, and all it does is provide a comma separated list of interfaces as follows: eth0, eth1 etc etc

Now, when I started looking at facter I knew no ruby and it was a bit daunting, but alas I learnt some and never bothered looking at facter again until my boss managed to simplify one down to it’s bear essentials, which is the one line…

Facter.add("bob") { setcode { "bob" } }

At this point onwards all you need to do is learn some ruby to make sure you can populate that appropriately or, use bash to get the details and populate the fact, in the next example I just grab the pid of apache from ps

apachepid=`ps -fu apache | grep apache | awk '{ print $2}'`

Facter.add(:apachepid) {
	setcode { apachepid }
}

So if you know bash, and you can copy and paste you can do something like the above, now this is ruby, so you can do a lot more complex things but that’s not for today

Okay, so now something more complex is needed…. What if you’re in Amazon and use the Tags on your EC2 instances and you want to use them in puppet ? well you can just query amazon and get a result and use that, although that will take forever and 1 day to run as AWS is not the quickest. This is an issue we had to over come, so we decided to run a script that would query amazon in it’s own time and populate the tags onto the file system, at which point we can read them quickly with facter.

So first, a shell script.

#!/bin/bash
source /path/to/aws/config.inc

# Grab all tags
IFS=$'\n'
for i in $($EC2_HOME/bin/ec2-describe-tags --filter "resource-type=instance" --filter "resource-id=`facter ec2_instance_id`" | grep -v cloudformation | cut -f 4-)
do
        key=$(echo $i | cut -f1)
        value=$(echo $i | cut -f2-)

        if [ ! -d "/opt/facts/tags/" ]
        then
                mkdir -p /opt/facts/tags
        fi
        if [ -n $value ]
        then
                echo $value > /opt/facts/tags/$key
		/usr/bin/logger set fact $key to $value
        fi
done

So this isn’t the best script in the world, but it is simple, it pulls a set of tags out of amazon and basically stores them in a directory where the file name is the tag name and the content is the tag value.
So now we have the facts locally with bash, something we’re all a bit more familiar with we can then take something like facter which is alien ruby and force some bash inside it but still generate facts that provide value

tags=`ls /opt/facts/tags/`

tags.each do |keys|
        value = `cat /opt/facts/tags/#{keys}`
        fact = "ec2_#{keys.chomp}"
        Facter.add(fact) { setcode { value.chomp } }
end

The first thing we do is produce a list of tags (directory list) and then we use some ruby to loop through it and yet more bash to get the values.
None of this is complicated, and hopefully these few examples are enough to encourage people to start writing facts even if they are an abomination to the ruby language but at least you have value without needing to spend time understanding or learning ruby.

Summary

Facts aren’t that hard to write, and thanks to being ruby you can make them as complicated as you like or as simple as you like, and you can even break into bash as needed. So now a caveat, although you can write facts quickly with this half bash/ruby mix by far, just learning ruby will make life easier in the long run, you can then start to incorporate some more complex logic into the facts to provide more value within puppet.

A useful link for facter incase you feel like reading more

Ruby, Pass by value or pass by reference?

I wish I knew for sure…

The only reason I’m writing this is I got bitten while writing some code for Sentinel I was expecting a value based on the fact I thought I was passing by value so the original object would remain unaffected.

Needless to say it was actually passing by reference, although if you read some articles it suggests that it actually only passes by value and those values just so happen to be object ID’s which point to objects… It get’s very complicated.

In an earlier post I talked about my programming ability, and I have written not to long ago about starting to code in Ruby So to be clear, I am not a programmer, I never will be, this stuff that i’m writing about now is what I’d class as ground breaking for me, it’s probably a really simple concept, but for some reason it has been really difficult to get my head around I struggled in C with it, more so in Java, and now Ruby; the difference? this time I actually went off did some reading and got some friends involved in a rant (both programmers who understood it all instantly even though they’d never seen ruby before, Annoying)

My quest started as most do, on Google where I came across this I’d highly recommend reading this page and the following two links that appear on that page One, two a combination of those and more reading and more talking I think I finally got my head around it.

Where did it go wrong?

Laziness is probably the answer here, I didn’t bother to read anything about it I just found a tutorial and went with it, got my head around the basics and started to dive in; which is fine as long as you don’t mind spending your evenings like I did, reading lots and coding.

I started out on an assumption that unless I specified otherwise it would be pass by value, mistake 1. In fairness it wasn’t a big mistake but annoying none the less. The other mistake I made was just not doing any research into how the langage worked, I could have spent a little time looking. So just now I did that a 2 min google and i came across this which would have saved me a little time, Granted I should have followed this straight after following Steve Litt’s it would have saved me a lot of hassle.

There is but one more thing I would have done differently, I would have read about the class I was about to use… Instead I was following examples and basically shooting in the dark and hoping it would get better, I wouldn’t do it with Sysadmin, so why do it with programming?

Just those few things would have made a big difference, not only to my understanding of Ruby but also would have ultimately saved me coding a lot of return’s that were not needed, so now I have to re-write some Sentinel code and make better use of methods I didn’t know existed. Well, at least it keeps me busy and I’m still learning new things.

Summary

I realise in the middle of this I digressed a little, but The links above are all very useful, especial those for Sentinel! If you are looking to get into Ruby I’d suggest the tips and links in “Where did it go wrong?” It would have saved me time and effor so no doubt it will save you time and effort; likewise if you were looking for the really techy explanation of Ruby pass by value or reference re-read “I wish I knew for sure…” all the information is there. I could try my best to explain it but that would be morally wrong, who knows what tripe could be produced based on a beginners understanding of a core piece of functionality.