Valuable vs Rapid change

Change, Change, Change, All Change!

For those of you in IT where a change seems a rare and wonderful thing, this post is probably not for you, if you are implementing less than a change a week, this post is not for you…

So where I am, we make in some cases several changes a day through our various systems, all in a bid to keep away the Big Bang of changes (also known as the Service disrupter), quite simply, our ethos is many small releases so the change pattern looks like a saw tooth not a set of steps.

Just like this:

Courtesy of Its Tech up north

see those tiny little releases, that’s the aim, the right hand side of the picture is when it all goes wrong.

So, back to the topic, lots of changes all the time, And at this point you may think I’m going to deviate into discussing the merits of quick releases and longer releases, as per the graphic. Well you’d be wrong, you should all be doing the left side of the graphic.

What’s a Valuable change?

It is all too easy to fall into the trap of Agile = Quick therefore make many changes and use all those rapid changes later to fix things up again. This is not right. This is not to say that you shouldn’t make a change rapidly if it fixes a service affecting issue… i.e. reconfiguring apache; however this is to say that you shouldn’t release half arsed.

So a valuable change is a well thought out change that adds benefit, it is in line with the road map and well tested. There is no time limit on a valuable change, you could spend months working on it, you could spend minuets. The key is as above, it’s on the road map, it’s well tested and it adds benefit, if any of these three things are not in alignment you probably have a rapid change; sorry for your service loss.

A Rapid change!

They are cool, they are quick, everyone thinks they are amazing because of how quickly they made it into the product / the service and they are by far the most pointless changes you’ll ever have to make.

Sometimes you will need to make rapid changes, i.e. one of your nodes in the cluster isn’t working so you need to make a change to remove it. This is not in line with the road map, it is not even a sensible change in the bigger picture, but it is absolutely necessary in the short term.

So rapid changes are fine to fix service affecting issues, you have to keep the service running, no excuses.

Rapid changes are ugly, often poorly documented, thought out and designed. So they should not be a method for implementing changes for the long term, i.e. a new way of logging in, a new graphic, a new way of balancing traffic etc etc

Rapid changes always produce Technical Debt where as valuable changes do not. So the sure fire way to identify a rapid change is by the fact it has technical debt.

Slightly off topic, I wonder if it’s worth graphing the tickets that produce technical debt and link those back to the change that produced it, you would then be able to show how much effort was created by a rapid change and categorically prove how bad it was to do and therefore discourage then in the future…


You will always have some element of rapid change, the aim is to make the number of rapid changes as small as possible, as stated they aways produce technical debt and that is really bad for you, really bad. Where possible even in a crises you should try to come up with solutions that are valuable and do not produce technical debt.

This should always be your aim, for the sake of another couple of mins of thinking about a solution, do it, it may save you a lot of technical debt and reduce the pain of supporting a system that is not as ideal as it should be.

Dealing with rapid change

It’s never going to be perfect

I know a number of sysadmins, myself included; who always are striving to get to the “perfect” system. It’s a wonderful goal to have and it should definitely be what you’re aiming to do, but you shouldn’t hold onto the perfect vision despite everything else, at the end of the day you are at work to provide value to the business not to hinder it. This is not to say that you shouldn’t have morals, and that there are not things that you need to stick to your guns on, as long as there’s a good business reason for it you can pretty much do that, but again, at the end of the day you are there for the businesses benefit.

The first part of dealing with rapid change is to realise that it will never be perfect, any designs you draw up now will be totally inaccurate a few months down the line or the business would have changed significantly where what you were aiming for has now moved. Goal post syndrome. So you have to be adaptable, your plan can not be strict in execution but it can be a guideline to what you are trying to achieve. Once you have accepted that it will never be perfect it makes life a lot easier when compromising on the little decisions and means you can focus on getting towards the goal rather than achieving the goal, as long as each change is a progression towards the end goal there is really no need to argue the point, you may want to go “above and beyond” to deliver a little more benefit but that should be off your own back.

The second part is all about knowing what is important to hold onto and what is not. It’s very easy to treat every decision as being vital to achieving the goal but normally they are not so important. As I mentioned before, it’s more important that you are moving towards the goal than achieving it, so any decision that moves towards the goal should not be contested, by all means try and add extra work in to get to the goal quicker but do not contest the task.

You should never really carry out tasks that make things worse, and I’d suggest it is these that you hold onto a little bit (not a lot…), make sure you point out why the task makes things worse, if needed make sure you get it in writing, but ultimately you need to get the task done if the business needs it, regardless of it making the overall system worse.

Making sure it doesn’t get un-manageable

So assuming that you have had to make some changes that are not in-line with the end goal you need a way of dealing with it, you have to track the fact that because of one change you have extra work to do. This is called “Technical Debt”, it’s what you accrue when you make a change that is not overall beneficial to the management of the environment or has a negative impact on the business in the long run. You’re always going to have some technical debt because you will never reach your goal, and if you do you will be dreaming up new ways to create technical debt.

These types of changes that produce technical debt can be summarised as “Tactical” typically because the business needs it now to remain competitive. The trick here is to not be objectionable to doing the tactical change, at the end of the day you have to deliver X by Y, so the scope of changing how you do that is finite, from the sysadmin point of view you can do an elaborate deployment if you wish, as long as you meet the time based deadline. This gives you the ability to try and carry out the task in a better way and hopefully reducing the technical debt.

This post was all about how to deal with rapid change, so far I’ve only really taked about making the decisions around what is important to hold on to and not arguing about every detail. This is important as it will make you less objectionable and hopefully minimise the technical debt, but you still need to deal with the technical debt.

Every time you have to make a tactical change that produces technical debt, spend a couple of mins logging tickets or making note of all of the individual actions that need to happen to get back out of the situation that has now been created. This is vital as the next time it comes to prioritising a project you have a shed load of evidence that says what you need to already fix to get back to a positive position, to make sure that is easy to plan in when you log the tickets make sure you capture the amount of time it will take to do the task.

Hopefully your boss will like the fact that the work has been captured and you can discuss how to reduce the technical debt to get “back on track” if you’re really unfortunate and your boss still has desires for tactical changes that eventually lead to fire fighting you have a nice list of tass that you have raised to say that you should be doing those to get back to a point of not fire fighting.


Don’t be too stuck on arguing the point of a task, eventually you’ll lose so just focus on minimising the technical debt that it causes and where possible only make changes that progress towards the end goal. I’ll write something else next week about dealing with technical debt, mainly because this was getting too long. Just take away from this that you don’t have to hold on to you technical bone to get your way, you just need to make sure that if there is a negative change the remediation work is tracked.