Cut down, deliver early and often

Deliver all the things

There’s idealistic people in the world and that’s fine (thanks for reading by the way :) ), and there’s pragmatic people, I want to go through how you can provide solutions that give a pragmatic approach to delivering value and doing it in a way that gets it done on time but still helps you get to your idealistic goals.

Often when sitting down and planning with senior management bods a feature list as long as your arm comes out, the reality is even if this is thought to be the bear minimum list, in reality it probably isn’t and is instead a bloated minimum, there’s always room to cut out features, so agree some prioritisation on the features and operate a bucket approach, one in one out.

After the features are agreed you can now start about delivering them, cutting where necessary.

Deadlines for Deadlines sake

When it comes to deadlines people take them a bit like marmite, you either love it or you hate it. Some people feel that having a deadline is a sure fire way of creating a bad product as corners are cut, others think that if you have a deadline you can at least work towards something with an end in sight rather than running off into the wild forever and ever.

To deliver the features needed it’s better to have a deadline, and one that stretches you and forces you to make some cuts, it’s not about not delivering or delivering badly it’s just about delivering what is needed, worse case scenario you have to deliver everything, but this way at least you do it in stages.

Certainly with a deadline you can help focus people on delivering what is important, I think sometimes people get caught up in trying to deliver everything perfectly for the deadline rather than delivering the value they already have. Some of my colleagues and mysef are currently working on a monitoring and metrics platform that integrates fully with nagios style checks but also allows you to write them in the web browser and test them on a server of your choice before distributing. The idea being that you can take the monitoring up to a real time level while reporting back business level reporting and everything in between so you have one place to go to to find out why something isn’t working and how well it has been doing, how many people have signed up; it’s a devops dashboard really.

Anyway, for a couple of months we have been identifying the core technologies and implementing various key functionality to the product but with at no point was any of it “working” some bits sort of worked but not quite, some bits just weren’t there. There was no real end date to this project as it’s something that will keep involving until it works and is useful however we need something to work towards and after a couple of months of sorting out the technology a deadline was set to do a demo and within a week we had the product up and working with the pages we needed with the correct functionality and everything working fine. Writing a nagios check on the fly, pushing it to the server distributing it to all the others and then reporting all in less than 30 seconds, wonderful.

I’m not saying what we did in that week was “production ready” but if our livelihood depended on it, it was good enough, and thats what being agile and lean is about. What is the least amount of work I can do to get me to the minimum product I need in the least amount of effort. The key is to obviously not get stuck delivering bear minimum all the time, with every sprint you need to improve upon what was there as well as add the new stuff; I think it is necessary to always fix something up when adding new features to get the product better and it certainly works for us.

Alternatively, of course, we could have not had a deadline and kept drifting aimlessly into the distance ensuring that the technology was “just right” all the time but the reality is we have to deliver something somewhere.

Iterate

Anyone that is familiar with Agile, Scrum, Extreme programming etc knows it’s better to deliver in small bite size pieces than in large chunks, you can provide value back to the business quicker and you focus on doing the task rather than doing it well. Not all tasks can be done by cutting a few corners but there’s normally a quick way, a good way and the right way fo doing it, so choose one and go for it, if it is a bit of angular that pulls down a list of plugins, go the quick way, if it’s a graphing engine that needs to draw lots of graphs and is used everywhere do it the right way; you’re sensible people, find a balance.

I’ve been talking all about software development which is where most of these methodologies come from, but they can be applied to systems administration as well, I think the same goes for sysadmins as it does programmers, they tend to get stuck in doing the best solution rather than the solution the business needs. Just to dispel any hopes and dreams, maybe save some time by realising that the business cares it works and is stable not how elegant or easy to maintain it is. So when coming up with a load balancing solution, maybe version 1 is haproxy with basic config and version 2 is a bit more in depth, version 3 is F5 & haproxy, version 4 is F5, haproxy and caching…. By all means have the hopes and dreams of the gold solution, but deliver the bronze one okay. If people really use the system and it provides more value iteratively make it better, maybe a bronze + a bit of silver, litle chunks, often.

Summary

Don’t get stuck in the end goal, think about what does the client really need or the business really need, bear minimum; deliver that, measure usage, iterate and improve.

Valuable vs Rapid change

Change, Change, Change, All Change!

For those of you in IT where a change seems a rare and wonderful thing, this post is probably not for you, if you are implementing less than a change a week, this post is not for you…

So where I am, we make in some cases several changes a day through our various systems, all in a bid to keep away the Big Bang of changes (also known as the Service disrupter), quite simply, our ethos is many small releases so the change pattern looks like a saw tooth not a set of steps.

Just like this:

Courtesy of Its Tech up north

see those tiny little releases, that’s the aim, the right hand side of the picture is when it all goes wrong.

So, back to the topic, lots of changes all the time, And at this point you may think I’m going to deviate into discussing the merits of quick releases and longer releases, as per the graphic. Well you’d be wrong, you should all be doing the left side of the graphic.

What’s a Valuable change?

It is all too easy to fall into the trap of Agile = Quick therefore make many changes and use all those rapid changes later to fix things up again. This is not right. This is not to say that you shouldn’t make a change rapidly if it fixes a service affecting issue… i.e. reconfiguring apache; however this is to say that you shouldn’t release half arsed.

So a valuable change is a well thought out change that adds benefit, it is in line with the road map and well tested. There is no time limit on a valuable change, you could spend months working on it, you could spend minuets. The key is as above, it’s on the road map, it’s well tested and it adds benefit, if any of these three things are not in alignment you probably have a rapid change; sorry for your service loss.

A Rapid change!

They are cool, they are quick, everyone thinks they are amazing because of how quickly they made it into the product / the service and they are by far the most pointless changes you’ll ever have to make.

Sometimes you will need to make rapid changes, i.e. one of your nodes in the cluster isn’t working so you need to make a change to remove it. This is not in line with the road map, it is not even a sensible change in the bigger picture, but it is absolutely necessary in the short term.

So rapid changes are fine to fix service affecting issues, you have to keep the service running, no excuses.

Rapid changes are ugly, often poorly documented, thought out and designed. So they should not be a method for implementing changes for the long term, i.e. a new way of logging in, a new graphic, a new way of balancing traffic etc etc

Rapid changes always produce Technical Debt where as valuable changes do not. So the sure fire way to identify a rapid change is by the fact it has technical debt.

Slightly off topic, I wonder if it’s worth graphing the tickets that produce technical debt and link those back to the change that produced it, you would then be able to show how much effort was created by a rapid change and categorically prove how bad it was to do and therefore discourage then in the future…

Summary

You will always have some element of rapid change, the aim is to make the number of rapid changes as small as possible, as stated they aways produce technical debt and that is really bad for you, really bad. Where possible even in a crises you should try to come up with solutions that are valuable and do not produce technical debt.

This should always be your aim, for the sake of another couple of mins of thinking about a solution, do it, it may save you a lot of technical debt and reduce the pain of supporting a system that is not as ideal as it should be.