DevOps team DNA

Hi, this is my first post on Matt’s blog. I’ve been an avid supporter of his blogging for a while and today got an invite to contribute. So here’s my post (created very quickly before he changes his mind).

My job has always been within an operations department of software product companies. I started at a small company as ‘everything’ support and slowly drifting towards a specialisation in the more recently branded DevOp’sy areas as I made my way through various acquisitions and mergers. Over the past couple of years I’ve found myself building DevOps teams. During that time I’ve discovered some of the things that work and almost everything that doesn’t work (or it feels like that :) ).

Some of the things that have worked..   (for me anyway)

Obviously these are going to be quite subjective and I doubt they will work for everyone. I’ll focus mostly on what I think are the key ingredients of a successful team. Maybe some people will find it interesting. Bare in mind that this only really applies to an operations team that supports a Cloud service.

I’m not a big football fan but I can draw some parallels between football managers and DevOps teams. You don’t see Arsenal winning and losing games based on their process redesigns. I may be simplifying, and I’m sure tactics plays a large part, but I believe you get quite a bit more out of a team when you have excellent players. Players who excel in different areas. My teams tend to be 5 – 7 players nowadays and between all of us we need to cover a few areas.

The first is product knowledge.. If you have a product guru in your team then you’ve got a productivity catalyst. So many aspects of our work involves investigating whether issues are product vs config and whether we can improve things from an operational perspective that will result in the product running better. The most recent team has a Product Architect and he’s awesome. He’s on the cutting edge of ideas for the product, for Amazon AWS and for all of the supporting technologies. Having a dedicated resource to do all of this in the background is great – it means that when we automate his prototypes and release them we get the maximum benefit. Recent examples include our Public API work and the work being done on our Amazon architecture to improve speed (CDN’s etc).

The second role I’ve always tried to fill is an engineer (at least one person, preferably two). Get the most senior developer(s) that you can, who knows the language of the product and build system of the product that you are supporting. You can now write the high level instrumentation that every DevOps teams need – as is true with any automation project. There is only ever so far you can go with Bash (I tend to take things beyond where they are supposed to be with Bash as it is). Ultimately having a senior developer or two buys you a massive amount of flexibility. Need a web service for something like externalised Puppet variables?.. you can write your own. Backup scripts not fast enough?.. a senior developer will make those scripts look very feeble in comparison when rewritten in their preferred language and multithreaded. I’m careful about not reinventing the wheel and will usually go off and clone something from Github before starting from scratch myself. But having some people who can write stuff from scratch is a major advantage. One caveat I would say for this role – hire from outside. Developers usually end up getting pulled back to work on stuff they did at the company at some stage. If you can, hire a new person and liven things up. Obviously tell the engineering teams that the hire(s) are for instrumentation in case they get worried that you want to start adding buttons to the product :)

Lastly, the sysadmins. I’d actually consider myself one of these at heart. Getting a good sysadmin can be tricky. It’s not uncommon to read 100 CV’s before finding someone even remotely eligible.  For a DevOps team you need a reasonably rare mix of skills.. people who know linux inside out, who can script and get excited by the latest batch of tools, and nowadays you need to throw Puppet / Chef into the mix. I have a couple of these currently and consider myself extremely blessed. Everything that we do is checked into source control (we use AWS as our data center) and this buys us a lot of things.. like the ability to automate everything, reduce costs by deleting and recreating at whim and disaster recovery. However, you pay for those things buy hiring really good people.. which is a cost saving in the long run once the cost saving benefits of the team start to show.

Now if you add in all of those types of role.. what I’ve found works quite nicely is running the team without being too focussed on the separation of responsibility. Everyone is on call 24/7. Everyone is expect to know the product inside out (although nobody will get near the level of expertise of the Product Architect), everyone scripts (even me) and ultimately everyone will end up doing some programming tasks. You can probably see from Matt’s previous blog posts about the Metrics project he got the chance to learn some Ruby. I think it’s important that everyone knows a bit of everyone else’s job.. although when under pressure everyone naturally drops back into doing what they are good at to speed things along.

This probably looks a little odd from the outside. But it makes things fun, everyone stays engaged and ultimately we all share the same goal: scale to 1 million users :D

Hiring in IT

The background

So for those of you that do not know me, I work at a Open Source company called Alfresco and we are hiring a lot of people this year. As a result we are suffering like all other IT companies trying to hire in the UK at the moment, there’s a skills shortage in IT and it is not something that we as an individual company can solve. If you are a person currently looking for work and struggling in IT I suggest you stop reading now.

What’s the real issue

It’s not a lack of candidates, we get a regular stream of good looking CV’s, but we’re not in the habit of looking for just anyone, we want people that are passionate about open source, people that want to make a difference. I suppose that by it’s self is not difficult, but when you combine that with a relatively decent job spec, and then the personal fit of the team etc it becomes a bit more challenging. We quite regularly get either very personable people that are lacking the technical skills to do the job or some very technical people that are missing the personal skills, needless to say it’s about compromise.

The above is based on a good candidate that just isn’t right for us, however in most cases (80% ish) the candidates are, how do we say this, overly optimistic with their CV’s? there’s been a number of people we’ve interviewed off of the strength of their CV and then they honestly expect us to not ask questions about their past experiences. Quite a number of CV’s have items on them where the person was part of a team doing the task and they claim (on the CV) to have been more or less the sole member, it all becomes very painful in the interview when you start asking questions on it. Do people really think we won’t ask?

So with that in mind, be weary of CV’s that have a lot of jargon on them, if possible aim for CV’s that have less on them, especially if that’s backed up with years of experience. it is more likely that that candidate is only listing things they are proficient in rather than things they have touched, or watched someone else touch.

The right compromise

If you think you won’t have to compromise on some element of the job spec / candidate persona then you should also probably stop reading this and go back to that wonderful place you come from where pigs fly and monkeys act as butlers.

It’s easier to say this than to live with the consequences, needless to say I have been unfortunate enough to live with some compromises that just weren’t quite right. Lets look at it in little chunks.

Technical ability is not everything, if you do have to compromise, compromise on this, you shouldn’t shoot your self in the foot, but choose a candidate that can do the job at hand and has room to grow. Hopefully the candidate will want to learn new things and develop their careers so it wouldn’t be the end of the world as long as they show signs of wanting to progress and develop.

Personality is key, there’s no point hiring someone that will come in and act as a rouge agent or isolated because they can’t gel with the team. I’d say the personality of the person is an area where you really don’t want to compromise on, if they only seem to slightly not fit with the team don’t worry about it, the team dynamic will change slightly but it should settle, you would need to identify which elements of the team’s “quirks’ you want to preserve and which ones would be benefited by having a slightly different personality in the team.

A little digression but probably valuable at this point, in the past I’ve had a sysadmin working along side me who seemed personable and would fit in personality wise (and they did) they seemed a little weak technically but we hoped that would develop; it turned out when they started they thought they were more technical than they actually were. They were the sort of person that you could give a task to and eventually they’d have something that worked, but they may not necessarily know how they got it to work. They also had issues arguing technical points, sticking hard and fast to their guns while not having a good reason to at the time and requiring a few days of googling to come back with a link that summarised that you could do it their way or the other way, so it also didn’t back up their ideals.

To be honest it was a real struggle for me personally as what I needed in the team was someone that was technical enough to take some of the workload off of me, but despite spending time training and explaining why things were done that way, the knowledge didn’t sink in. I’d feel bad that I couldn’t get the information in, but other people tried and failed as well, so the common problem is probably the cause. The biggest issue was that they were not willing to progress, maybe because they thought they knew it all, or because they just weren’t able to grasp the technical detail.

So if you have to compromise on technology skills make sure they are personable and willing to learn and progress, people can develop technical know how if they have the personality that allows them to, but changing someones personality is much much harder.

If you’re interested in working at Alfresco you should keep a regular eye on our Careers page, it doesn’t have all of our jobs on it so feel free to email your CV through just on the off chance, we’re always looking for good people.