Fire!

There comes a time in every Sysadmin’s life where some management bod comes rushing over red faced and out of breath, From my personal experience I highly recommend not replying with “It is working okay for me” That does not get brownie points.

Having learnt from that experience I do highly recommend showing a bit of understanding instead, It is far far better to reply with “Let me look at this right away”. That doesn’t necessarily mean you do need to look at it right away, but if it’s someone in your direct reporting path I’d highly recommend following up on the lip service.

Whether you like it or not, sysadmins are on the front line, it doesn’t matter if you are labelled as “3rd line” or “4th line” you are still on the front line. The buck stops with you, sorry. This also means your customer service skills need to be exceptional, again, sorry. I realise that most sysadmins are technology focused, the latest and the best technology geek-o-rama is all important etc etc yadda yadda *Yawn* For reference that is what the Sales / Marketing guy is doing while you’re explaining the problem that he really doesn’t care about.

At this point you’re wondering what this has to do with fire fighting, absolutely nothing, Wrong!

Welcome to the world of expectation management

So you’ve had the angry management bod, You’ve correctly not told them “it’s all okay” and you didn’t chew their ear off about all that wonderful technology stuff that may be causing the potential issue. At this point there’s a few acceptable ways to mitigate another bigger (and redder) management bod turning up.

“Let me look at that right now for you”

“Let me have a quick look and see what the issue is and I’ll give you an update in 5 mins”

“Thanks for letting us know about this issue, we’ll send out a note to everyone with an update shortly”

Here we have our 3 easy answers to all issues:

1, You know what it is and you know it’s an easy fix, You could ask them to log a ticket (this will annoy them for what it’s worth..) or you can just do it quickly. Even if you have a policy where tickets must be logged, just do the work and log the ticket for them. Once you’ve fixed it you can let them know that the easiest way to get this change done is through the ticketing system.

2, You know what the issue is, It may be 5 mins, it may be 25 mins. At least this way you can get rid of the person standing over your shoulder and focus on fixing the issue. Best case is you fix it in 5 mins walk over (yes, walk over and say you fixed it!) and you have a happy person. Worse case scenario is you spend 5 mins working out what the issue is and walk over (yes, again with the walking over) and you update the person with the ETA, at which point you rush back to your desk and try to fix the issue, shortly followed by… you guessed it, A walk ver to their desk / office to inform them it is now fixed, “Would you mind having a quick test?” Why would you risk it not working! or something else going wrong… Customer service, remember you’re on the front line, if it’s not fixed you’ll just have the same manager or a bigger manager standing at your desk, this, in case you’re struggling to make the connection is “pro-active” shock!, horror! it’s sort of like a management buzz word but it just means you do a good job and get confirmation from the affected person or parties that you did do a good job.

3, Quite frankly, if you’re using this response you’ve got bigger issues with environment. Something catastrophic has gone wrong, emphasise if needed. Your aim here is to re-assure the person raising the issue “Oh, that is not good, I can see how this is affecting your ability to do your job” (I wouldn’t use those words, but you get the idea) Make sure that you escalate this to your boss so they can take care of the management with the business freeing you up to do what you do best, Grab coffee. I was half serious this time, You’re going to be at your desk a while, take 2 mins to sort your self out so you can focus on the issue. Make sure that someone (ideally your boss) is sending out regular updates, in the updates tell people when you will next update them and stick to these time frames; and remember when you start talking about the DNS cache being corrupted on one of the slave caches in the backup data centre affecting the transfer to the internet they are ….*yawning* Do 3 things:

Summarise in the subject / heading (yes this will not be a direct contact event…)

Give a one or two sentences at the top of the email with a tiny bit more detail

Knock your self out and write all that pent-up technological mumbo jumbo you really feel the urge to get on paper for the techno-geeks in the company to read, but please, do it at the bottom so as people become less interested in the technology they can stop reading higher up

Why was that so important?

Well, in short people like being informed, they like being given relevent information and they like it even more when you do what you say you are going to do.
If you can stick to doing those things and ensuring that you try to understand the real issue to the people affected you will be a better sysadmin for it. I realise it’s easy to build up the walls and start blaming other people or to become complacent around outages. If you were thinking, even for one moment “well it was their fault” … Bad sysadmin! really if it is any ones fault it is your fault. Why didn’t your monitoring pick it up? Why were you not able to send out an announcement before you had someone standing at your desk all red faced.

If you ever get anyone at your desk for something more than “How do I do…” something is not right. Try and be self critical and remember that you can not change other people (well you can but it’s much harder than you might think.)

The whole point about managing expectations is that people will start to have trust in you, the department and when you say 5 mins, they know it means 5 mins not 20. If you’re unfortunate enough to be in a position where your IT department already is the lowest of the low and people are always fuming about what you do and that you never deliver, well expectation management will help you out, in a time of crises people typically accept the authoritarian figure’s word so you can use this implied authority as a foot hold into fixing the bigger issue which is trust.

So who did burn down that building?

Well the most important thing here is that it wasn’t you, find some other sucker to pin the blame on, Why build up your reputation to knock it back down again! … Seriously?

Okay the most important thing is to actually not be part of any blame culture, it’s so counter-productive it’s not funny, you end up in a position where people will actively not do work because they won’t take the risk / blame if something goes wrong. If you start hearing someone blaming another department or you hear your self doing it. Stop them, interject, put the record straight. There’s a big difference between saying “Because Bob didn’t ask for the backups to be done we lost the data” and “when we were setting up the database our process wasn’t good enough to catch the fact we had missed the backups” “we missed the backups”, “Our process wasn’t..” key terms, after all even in a team of one it is a team effort between you and the rest of the business.

Obviously if you’re in the meeting and saying that the best continuation of that sentence is “But Ted has now checked the other database servers and we have also updated our process to catch this if it was to happen again”. Hang on, The problem was fixed and you didn’t take any glory, even though you spent an hour explaining to Ted how to even check if there was a back up and you re-wrote the entire process! Correct! If you want to be taken seriously and not destroy trust you’re busy building up in the rest of the business, don’t ever take the glory. People are smart enough to know that you were involved and doing a good job, and by thanking Ted publicly you’ve motivated a colleague and more importantly made your self not look self-centered. Which for those of you struggling… this means you gain more trust from the business as they believe you have brought into the bigger picture, what the company is trying to achieve.

In some case you would have used your best political speech to explain what the problem was and how it was fixed, for some reason people are still not happy. Ideally in this position your manager will speak up and take the blame, if you’re unfortunate to not be in this position, or you are the one that made the mistake, or maybe your colleague did but he is not taking the blame; Don’t be afraid to take some of the blame, “Sorry, I can see now that I should have double checked the backups”

Summary

• Be nice, have empathy with the users issues and really try to understand how this affects them and their day to day business.
• Don’t be afraid of talking face to face with people, if you can you should, people like talking to people, and they love talking to people that think they’re special and important.
• Do what you say you are going to do, no excuses, if you’re struggling to do things in the time frame you set out, check back in and expand the update, explain why it hasn’t happened yet
• Don’t blame others, Thank those that helped, Take no glory, And if necessary Take the blame

With thanks to: Oldonliner For the fancy image.