Dealing with Technical Debt

A brief history

Last week I touched on Dealing with rapid change and I kinda got carried away and exceeded my self imposed 800 word limit. So the point of today’s blog is about capturing the technical debt and then how best to deal with it so you don’t end up in a situation where the service starts suffering because of tactical decisions made without suitable foresight.

So to summerise what technical debt is for those that did not read last weeks post (shame on you), Technical Debt is what you accrue when you have to make a change that does not head towards the overal goal but instead causes additional headaches for managing the environment or some how limits you in achieving the overal goal of a perfect system.

Capturing technical debt

My personal preference to tracking technical debt is a ticketing system, just create a way within that to easily identify what is technical debt and what is a normal task. You don’t have to use a ticketing system, you could just write it on paper or in a spreadsheet or what ever, the important thing is that you capture it with a reasonable amount of detail.

Avoid the temptation of filling the ticket with a lot of information, just put enough in it that explains what the problem is and why it needs to be fixed, if you have some suggestions on how it could be fixed add them to the ticket but don’t worry about saying “this is the solution” that can be done when the ticket is actioned.

Try and make sure you capture the following as a rule of thumb:

  • What is the issue?
  • Why is it an issue?
  • Any future tasks that are dependant on it
  • How long the task will take
  • A rough priority

These things will help prioritise the task when it comes to the planning the next set of tasks / projects that need to be actioned, but it doesn’t really help it get prioritised. Why? because it will never be the focus of the business to do the boring work that makes it stable unless there are issues or it has to be done as dependancy of another task.

Actioning technical debt

As I pointed out before, the business will never prioritise the technical debt unless there is a real issue for it to do so, service stability or dependancy on another task. This is a fact of life, and if you’ve found your self in this situation getting frustrated about all the hundreds of days of backlog tasks of technical debt that is accruing, panic no more.

As I pointed out above, the business will never want to do the technical debt unless there is good cause to do so, so the point of capturing the tickets with the right information is that the dependancies are clear, the outcomes of not doing it are clear, this makes it easier to discuss it as a business requirement. That is not enough though, you will get some tasks done but you will not be decreasing the technical debt it will continue to increase.

You need to create an environment in your immediate team and the extended team that understand why the technical debt needs to be actioned. This will be easier to do than convincing the whole business as to why it is important. Once you have the buy-in of the teams everyone will hopefully understand why it is important to keep the technical debt to a minimum. This will help it take a higher priority when it comes to scheduling tasks and help reduce the technical debt in the long run, it would also be a good idea as a group to identify an amount of technical debt that as a group you are happy to live with, this can be calculated on the amount of effort days required to deliver each task. This is a sure fire way of getting technical debt actioned and will help ensure that it remains at a level that is sustainable.

There’s always going to be a risk that you’ve tried all of the above and you’ve still not gotten anywhere, the technical debt keeps rising and the environment continues to get more and more complicated to work within. Simple, do not worry about it, you did what you were meant to, you raised the issues, you pointed it out to your boss, you can kick back and relax, maybe even take deep joy in the moment when it all fails and someone asks why and you just point out the years worth of technical debt that has ground your system to a halt. In short, it’s not your problem, it’s your bosses, you just need to make sure you capture it and raise it.

Summary

Technical debt is bad, if you’re not aware of it you need to be, you need to have a mechanism for dealing with it and if you don’t your systems will grind to a halt and you will probably be one of those companies that re-builds the entire infrastructure every 3-5 years because the technical debt was so large that the environment was unmanageable.

Dealing with rapid change

It’s never going to be perfect

I know a number of sysadmins, myself included; who always are striving to get to the “perfect” system. It’s a wonderful goal to have and it should definitely be what you’re aiming to do, but you shouldn’t hold onto the perfect vision despite everything else, at the end of the day you are at work to provide value to the business not to hinder it. This is not to say that you shouldn’t have morals, and that there are not things that you need to stick to your guns on, as long as there’s a good business reason for it you can pretty much do that, but again, at the end of the day you are there for the businesses benefit.

The first part of dealing with rapid change is to realise that it will never be perfect, any designs you draw up now will be totally inaccurate a few months down the line or the business would have changed significantly where what you were aiming for has now moved. Goal post syndrome. So you have to be adaptable, your plan can not be strict in execution but it can be a guideline to what you are trying to achieve. Once you have accepted that it will never be perfect it makes life a lot easier when compromising on the little decisions and means you can focus on getting towards the goal rather than achieving the goal, as long as each change is a progression towards the end goal there is really no need to argue the point, you may want to go “above and beyond” to deliver a little more benefit but that should be off your own back.

The second part is all about knowing what is important to hold onto and what is not. It’s very easy to treat every decision as being vital to achieving the goal but normally they are not so important. As I mentioned before, it’s more important that you are moving towards the goal than achieving it, so any decision that moves towards the goal should not be contested, by all means try and add extra work in to get to the goal quicker but do not contest the task.

You should never really carry out tasks that make things worse, and I’d suggest it is these that you hold onto a little bit (not a lot…), make sure you point out why the task makes things worse, if needed make sure you get it in writing, but ultimately you need to get the task done if the business needs it, regardless of it making the overall system worse.

Making sure it doesn’t get un-manageable

So assuming that you have had to make some changes that are not in-line with the end goal you need a way of dealing with it, you have to track the fact that because of one change you have extra work to do. This is called “Technical Debt”, it’s what you accrue when you make a change that is not overall beneficial to the management of the environment or has a negative impact on the business in the long run. You’re always going to have some technical debt because you will never reach your goal, and if you do you will be dreaming up new ways to create technical debt.

These types of changes that produce technical debt can be summarised as “Tactical” typically because the business needs it now to remain competitive. The trick here is to not be objectionable to doing the tactical change, at the end of the day you have to deliver X by Y, so the scope of changing how you do that is finite, from the sysadmin point of view you can do an elaborate deployment if you wish, as long as you meet the time based deadline. This gives you the ability to try and carry out the task in a better way and hopefully reducing the technical debt.

This post was all about how to deal with rapid change, so far I’ve only really taked about making the decisions around what is important to hold on to and not arguing about every detail. This is important as it will make you less objectionable and hopefully minimise the technical debt, but you still need to deal with the technical debt.

Every time you have to make a tactical change that produces technical debt, spend a couple of mins logging tickets or making note of all of the individual actions that need to happen to get back out of the situation that has now been created. This is vital as the next time it comes to prioritising a project you have a shed load of evidence that says what you need to already fix to get back to a positive position, to make sure that is easy to plan in when you log the tickets make sure you capture the amount of time it will take to do the task.

Hopefully your boss will like the fact that the work has been captured and you can discuss how to reduce the technical debt to get “back on track” if you’re really unfortunate and your boss still has desires for tactical changes that eventually lead to fire fighting you have a nice list of tass that you have raised to say that you should be doing those to get back to a point of not fire fighting.

Summary

Don’t be too stuck on arguing the point of a task, eventually you’ll lose so just focus on minimising the technical debt that it causes and where possible only make changes that progress towards the end goal. I’ll write something else next week about dealing with technical debt, mainly because this was getting too long. Just take away from this that you don’t have to hold on to you technical bone to get your way, you just need to make sure that if there is a negative change the remediation work is tracked.