4 months in and still blogging!

4 months!

So I started this on a whim, and the whim included writing two blogs a week and this whim has seemingly continued week on week until we are hear today, 4 months on from that fateful first post

I’m probably Just as surprised as as those that know me that I have been able to keep this going. I had a number of people tell me I was a bit crazy and wasting my time and that writing 2 articles a week would be difficult, yet my stubbornness has paid off and here we are.

So what’s happened in the last 4 months

I’ve written a number of articles so here’s some stats:

top 5 most popular

  1. RHN Satellite vs Puppet, A clear victory? – 200+ views
  2. Home page / Archives – 160+
  3. Puppet inheritance, revisited – 40+
  4. Ruby, Pass by value or pass by reference? – 25
  5. DNS results in AWS aren’t always right – 23

I guess the Home page one is because most people read the articles after it has been published and before it gets archived, it also goes to show that I have written only one article so far that people care about or at least will google for and find my article as a result.

top 5 country views

Average views per month/ week / day

At the time of writing they are as follows:

Month: 220 (last month was 284 views, this month is 350+)
Week: 55 (Weekly views range from 27 to 111)
Day: 7.8 (Lowest is 1 view (on a sunday) highest thus far is 34 which was this month)

I expect all of these to go up, last month I doubled my views over the previous two months and in June at the time of writing I am on target to do the same again.

Interestingly enough this is the first time I’ve really looked at what is / is not popular and interestingly enough it is the technical posts, which I find the easiest to write due to the job, the less technical posts (Fridays’) are not so popular but still get good views.

What’s coming up

Well to be honest much of the same, I have some articles to write about chef comparisons and puppet designed for large scale, externalised configuration, Mcollective, role based puppet in multiple environments all of which i’m sure will be interesting and popular.

As for the less technical side I don’t know yet, these are the hardest to find a subject that is worth writing something about without repeating something that has already been said.

With that in mind I need to think about how often I will continue to write certain posts but other than that it’s business as usual for the near term at least, of course if anyone else would like to write articles that would save me doing it :D

Amazon one simple outage, one day of hell

ARRRGHHH!!!

What a horrible day for me, for those of you that didn’t notice… Amazon had another outage in the East coast see Here for details. Well I can honestly say it was not a pleasurable day for myself, I had just got into the office about to get my first coffee of the day at 8:15 BST when we had a system down alert from our cloud product, Arse. For those that read my other Posts you’ll know I recommend coffee for any problem solving. However first things first, stabilise the situation, annoyingly I was so concerned about the box which I couldn’t become root on I didn’t do the simplest step and it took my boss to remind me I could just remove the affected server from the Apache load balancer, there for the service was affected for 14 mins, probably would have been 10 if I’d have had my coffee!.

Just as I stabilised the service, a colleague turned up for the day and mentioned that Amazon had some issues, turns out he found out by following other cloud based companies such as Heroku, Rightscale and Netflix, a good strategy to employ later. I of course started to prioritise my workload as any good sysadmin does, grab coffee, fix problem, rinse & repeat until desired results are obtained.

EBS Volumes missing

We have a number of EBS volumes, luckily only 1 of 130 was affected, but oh boy didn’t strage things happen. So for starters to gain root access to our box we had to restart it. When it came back up one of the disks wasn’t attached which was rather confusing trying to start tomcat to find that it was no longer there (we mount /var on an EBS volume).

We identified that we had to re-enable IO and as suggested fsck the disks, so this is probably where things went horribly wrong.

I tried using tune2fs to change the disk mount count above that which fsck kicks in to force a fsck on reboot, it turns out this wasn’t going to work, I’m not sure why but it didn’t seem to kick in. In the end we took the 3 volumes on this server and attached them to another server, from here we were able to check the disks and only 1 had some minor issues. we re-attached them back to the original node and hoped for the best…

So obviously it got worse

Now, those that have used AWS know that attaching disks can be challenging when in the API / console you tell a device to mount as /dev/sde to find it attaches as /dev/xvdi, as a result we are meticulous with our disk mount points and I ensured I mounted them in the same order on the same box. So surely turning it on would be okay. Not so much. Our box came up, it passed all of the Amazon checks, it is network available.

For some reason we were getting connection refused from ssh, this is odd. I logged tickets with Amazon’s support, this was pretty much a waste of time and effort. We spent a couple of hours discussing how it wouldn’t have been possible for an OS to change its own firewall rules until I resorted to mounting the root EBS volume for the server fsck’ing it and re-configuring iptables to allow any TCP connection. Guess what, it still didn’t respond.

Somewhere along the line this box has gone from unresponsive to working but missing a drive to totally buggered, not a good day. Luckily we are able to re-build a node and hook it back into to the clusters / load balancers, but why should we need to do that, in a traditional data centre I would have logged onto the console and just simple fixed the issue.

Summary

I have never liked the idea of cloud providers for running core services and have always felt they’d be better in a traditional hosted environment, it is definitely a good place to float an idea or to try something out but longer term DIY. I’m not saying AWS isn’t good, it has a purpose, bursting web traffic or bursting for stateless servers is one. Amazon do now have a console feature that if you had attached a key to the server you could use it to give you that local data centre type console. Unfortunately for us we do not deploy our Amazon boxes with any keys apart form user ones, maybe this is something we will review, more likely we will increase the automation of our environment so a server is just a thing and not a name.
We will be doing this and over the next few months I will start posting more details as we have it going, needless to say it is slightly bleeding edge and as a result is constantly in development.

Anyway, rant over, horrible day.

Loving what you do

A while back…

I touched on that you should love what you do Here. I also found myself over the last few months wondering if I was still loving what I do, the short answer is no, the longer answer is yes. As those of you that read the other post will know, you should always be doing what you love else what’s the point. I identified a long while back, maybe 6-9 months that there are certain areas of my role that I prefer over others and a few years ago there’s certain things I really enjoy doing. So before going to much further I do enjoy my work, it is challenging and rewarding and that should be enough, but there was a little niggle that was telling me that something wasn’t right with what I was doing.

I had a sit down and a think and I realised I enjoy doing the technical hands on elements to a point, after that point the enjoyment drops off rapidly. For me that point is when the system starts to bed in and becomes more routine, it’s not an element I enjoy of the role but it is something that needs to be done, it is very important in fact. I’ve been really lucky that I don’t seem to be around long enough with any projects to see them through to completion, or when they are completed I’m then working on another large project and tend to skip the day to day work around that, so although there are elements I find dull, I don’t mind as they don’t normally affect me for too long.

There is also something else I know I like doing and I’d like to do more of. At a previous employer I was given the opportunity to do some mentoring of a graduate, and I found I really enjoyed that, I found it much more rewarding than the day job, I felt like I was actually helping someone achieve more, which is very rewarding.

Bring what you love to work

With all of this in mind I started to think about what else I could do within my current job role to give me that feeling again. Well the obvious answer for me was to get some graduates to help out and give me some people to mentor which I enjoy doing. Unfortunately this time that plan did not work out, but it is getting closer and one day will work out.

So this time that plan didn’t work out, so I’m now in the position of finding another way to make that happen, maybe graduates was a bridge to far. Maybe there is something else that could be done to give the same feedback, I will of course need to think about it a bit more.

The main point is that if there is a small element missing from your job, in my case mentoring, find a way to bring it into your job and don’t get disheartened if one method doesn’t work. Keep trying. It is important to keep trying especially if there is only one element that makes it not quite the right role for you.

Don’t run away

Other than trying to bring the elements you do want into your role you could try and change role, this isn’t a bad idea but think about it. I recently started considering other roles that had more of a leadership part to it that would allow me to mentor people and help develop them but as with most people the roles I was looking for didn’t exist in my current company.

The challenge than changes from trying to bring one element into a all round good role to ensuring that all elements of a potential new role meet or exceed your current position. For me the exercise was relatively pointless as all the other elements of, my role are perfect for what I want to do, but if you are not in the same position then moving on may be the right thing for you, but don’t do it to try and make one element better as more than likely the other good areas of the role will probably suffer, it’s all about swings and roundabouts.

Summary

Don’t try to change your job if every element is good about your current role except one element, instead try and find ways to fix that element in your current role instead. There is always a way.

Apache URL enoding

This was a little annoying…

I came across an interesting Apache quirk the week before last, it totally make sense why it happens and I was at first a little surprised, one because no one had noticed previously and secondly because it was happening at all.

We noticed that if a url like http://bob/file.php?id=$frank went to an apache then the dollar symbol got encoded, which is perfectly normal behaviour, it sees a special character it deals with it. In our case this was being trigged by a URL redirect from http to https. Something I thought was odd which I never got to the bottom of is why did it do the re-write at all? If the http to https rewrite rule was not there it just passes it through so it is a by product of the rewrite.

This in its self is fine, other than manipulation of the url should probably be an option to turn on rather than off but I guess that depends on how popular it is. Either way this can be stopped by simply telling it to not encode the URL with the [NE] flag on the end of the rule.

The annoying element of all this is no one noticed an issue, the application is able to un-encode a URL and to work with the non encoded URL and yet still things were not quite right.

It turned out with a bit of digging that if you sent in a URL of http://bob/file.php?id=%24frank apache ended up encoding the encoded URL resulting in a URL that looked like this – http://bob/file.php?id=%2524frank

I can understand that Apache doesn’t know it’s encoded already, but considering we only send out URL’s with $ in what on earth was causing it to go horribly wrong?

A bit of digging

It turned out that some web-based email service thought the best thing they could do to all URL’s is re-endcode them for you.

For example, Hotmail:

Gmail:

Can you see the difference? try clicking on the image for a more human readable one.

Not sure why our good friends at Microsoft decided it was a good idea to change peoples URL’s, there probably is one, but I’d like to think that Gmail is as complicated as Hotmail and they seem to have found a solution.

Much time of many people was spent working out how this issue occurred, but none the less it is resolved, I do feel a bit silly for not spotting the double encoding myself but at least now I know and you know that Hotmail does URL encoding and Gmail does not.

Stress

We’ve all been there at some point

Stress isn’t fun, there’s a point before the un-fun stage of stress that seems to help make us a little more productive. Everyone needs a mechanism for dealing with it else they will just get overwhelmed at some point and then enters the not so fun side of stress.

It was a few years back when I had my first stress related incident, working on a project trying to get a delivery done by a certain time, working some 15 hour days to help get it done. What did it take for all the stress to build up and make me break as a person? Just one project manager being an arse, insisting that more is done and we should work longer hours; he was so out of touch with the amount of effort that was already going in. I had to walk out of that meeting I was in tears within 30 paces, I punched one of those cubical divide things as I was walking past (the cap on the top flew off and hit the ceiling which drew a little more attention than needed!) and I ended in the toilets sobbing like a child. Welcome to stress.

Identifying stress

I had no idea I was getting stressed, it was sort of like a ninja attack one minuet fine, the next blubbering wreck. HowStuffWorks is pretty good for this. In short, everything stresses us out, everything. Not being able to go to toilet, not being able to meet up with a friend, having coffee, not having coffee and so on. So identifying what causes stress is quite straight forward… Everything. As the article say’s, you can’t always remove the stress but you can reduce them.

After my “episode” I spent some time thinking about what really tipped the balance for me, I looked at what was going on at home and work, what was different about the work I was doing now to 6 months ago and so on. For me it seems what triggers the most stress is not being able to do what people expect of me, the normal day to day work is fine and project work is fine. Probably worth mentioning that before my “episode” there was two of us on the project doing the same sort of things, unfortunately my colleague had just been signed off with stress as well, so my work load doubled and it was already pretty high. This was another contributing factor, not because I had to pick up my colleagues work but because my colleague wasn’t there to share the mumbling about the project which helps reduce stress.

So if you find your self stressing try and identified what’s changed, it may not be one thing it could be a combination of things causing it, but as with most things until you know what is causing it you can’t do much to fix it.

Dealing with stress

I use to deal with stress by putting a very clear boundary between work and home, I go into work typically between 7:30 and 8am and leave normally around 6pm with the occasional desertion from work from 3pm onwards if I can. when things get too busy and I start noticing I’m beginning to get stressed I tend to start brining the hours in, I start making sure I do my contractual hours more so than the hours I typically do. Likewise I don’t take work home with me, I very rarely do work in the evening or at weekends unless it’s something I want to do like a special project I kicked off.

So the clear boundaries help separate the stressy part of the day from the non stressy, reducing the stress caused by work.

The other, and the most important thing is to stop caring about the work, yes it’s important, yes it needs to be done, but it’s not all on you, you are not the only person involved in this, it’s a team game. This helps reduce the worrying element of trying to get it out and takes some of the self imposed pressure off of you, you do need to care about the quality of the work you do, but not so much what that work is.

The other most useful step is asking for help, I use to be very bad at asking for help; even at school I was being told to ask for more help and the same goes for work. Last time I had a look I wasn’t wearing my underpants on top of my trousers, or had a big ‘S’ on my chest within a shield, or was a fictional comic book hero, I’m assuming you aren’t either. By asking for help you may get some, you may not; but you stand a better chance than not asking at all.

Securing your password post LinkedIn

Big news

Anyone that’s in IT and hopefully the wider community would have seen the news that LinkedIn have had the password hashes of 6.5 million users stolen (read more and the number keeps increasing…).

So for people that don’t know here’s a quick summary of what LinkedIn say has been stolen… Your password hash is a one-way encryption of your password, typically this is with something like MD5 or SHA1, they look like this:

Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "password" | md5
286755fad04869ca523320acce0dc6a

A simple hash of the word “password”, the important thing to realise is that when you login in it is the hash that is checked not your password; what does this mean? it means I don’t need to know your password, just a string of characters that generate the same hash. There’s Hash table sand Rainbow tables on the internet that are basically just random characters hashed and stored in a file or DB, it is then a matter of doing a simple lookup of the hash they have with the one in the table.

What LinkedIn have done now is mitigate that lookup by adding a salt to the password, basically a random string is prepended to the string to change your password before it is hashed, example below:

Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "`cat /dev/urandom | head -c12`password" | md5
a3aadeacee5d742980cf3cff67ca0cb0
Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "`cat /dev/urandom | head -c12`password" | md5
84957ff57478e1a82432dafb6f39bab8
Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "`cat /dev/urandom | head -c12`password" | md5
a6936fdb9af34da281758c8649829651
Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "`cat /dev/urandom | head -c12`password" | md5
3775e0dd675473e57326612304058317
Matthew-Smiths-MacBook-Pro-2:~ soimafreak$ echo "`cat /dev/urandom | head -c12`password" | md5
9ff93bc19c28c54aa2249e90b73e9f11

As you can see each time the password is generated it s different, this presents 1 problem, your password will be different every time it is generated so typically the salt (random string) is stored with your password. This only really protects against your password being easily guessed if someone has already generated a hash of the password (as even two people with the same password have different hashes). So they have to store your salt with your password, which is fine as it’s sole purpose it to make your your “password” is not the same as my “password”, for more information I would read: This

Secure your passwords

I could go through all the basic stuff of choose passwords of 6-8 characters, at least one upper case and a number; bog standard password stuff and it is completely useless. Back in the day that was considered good, but the modern convention is you are better off with a passphrase rather than a password.

For example: “SgsFd76!£” – Good use of the english £ symbol as it’s not on all keyboards! However this is still easy to crak for a modern graphics card, See this

A much better password would be: “thisismysecurepassphraseforexample” more characters = more combinations for the computer to check through, this is much better than the first example, even better than that though: “AJSACBVCjndsf76287*798sdjb7afbafuxvi679(UoahsfkjbauytSDgsbf ” – This is probably the best you can hope for but if you’re like me and don’t have an eidetic memory this will be impossible to remember and therefore pointless to you.

So what would you say if I told you all of my passwords are in that odd format that I can’t remember, and better yet I have a different one for every website / computer and I have no idea what any of them are… Crazy? Probably, but it’s true.

I use KeePassX to store all of my passwords in which means I have to remember one odd passphrase and that’s it. You can even secure it with a key pair if you so wish, I considered this overkill but it’s still nice to have the option.

So for every website there is a different random 20-26 character passphrase that I don’t know, it’s annoying how many websites don’t support more than 6-12 characters or enforce you to use poor passwords such as Virgin Media’s email service, Ticketmaster and the Student Loans company; on a side not the SLC was by far the hardest service for me to get access too asking questions I didn’t know all the answers to and then made me put in a password online that was pathetic…

So the big issue you have now is that you don’t know any of your passwords, this could be an issue. I personally use Dropbox on all my computers and mobile devices and as such it keeps up to date. The only risky thing I do is store my Dropbox password in KeePassX which Means I could lose everything, this is why I make a regular local back-up on multiple computers; if you are almost certain you’ll lose your backup you could just set a passphrase for Dropbox that you can remember.

Summary

Use passphrases rather than passwords and make sure that you use different phrases on different sites, because this is a mission use something like KeePassX to help you out and just make sure you back that up appropriately. I’ve been using it for over a year now with no issues, the only thing that is annoying more than anything, when setting up mobile devices I would have to type in a 26 character password, this is a pain; one I live with.

Getting motivated

Are you motivated?

I remember reading an article a few months back from HBR and it was very insightful, and I highly recommend reading it before continuing… It certainly struck an accord with me, as I think I’m very motivated. However I hate finishing things off, the closer I get to completing the less interest I typically have. This isn’t a bad, but it is something I need to be aware of, I certainly wouldn’t be any good at following a set process day in and day out or finishing things off perfectly, but I am very good at getting 90% of the way there and getting it up and working in a sensible way.

If you’ve heard of Belbin he came up with some tests that identified the roles within a team and predicted that teams made up with a mix of roles functioned better than those with predominant roles. If you imagine a team where everyone wants to be in charge you can already work out they’ll spend more time bickering than actually getting the task done. I imagine that based on my results (Co-ordinate, with secondaries of plant & teamwork) is why I prefer to be making the decisions rather than finishing off the work, which back to the motivation element is probably why I also don’t like completing things, there’s a lack of decisions to be made towards the end.

So I am motivated, but I find it a real struggle to finish things off, sometimes it just has to be done and I will typically work from home or put my headphones in and just get on with it. That is how I deal with motivating myself to finish tasks, but what do you do if you aren’t motivated to start?

Getting motivated

I believe that to be motivated about something you need to have a few things, you need to be part of the process, making some of the decisions or feeling that your input is valued. You also need to want it to succeed, this is the difficult one to achieve, if you’re part of a team and you don’t think it’s the right thing to do and you don’t really want it to succeed then that will bring down the rest of the team, as well as frustrating you and making you more unmotivated.

You have to take some responsibility for changing your perspective on a project or a task, sure your manager could identify that you are not too keen on it and they could throw some rewards your way to help with the motivation, but ultimately you need to want to make yourself motivated and communicate what is stopping you be motivated.

When I am in a situation where I’m struggling to get motivated about it I tend to look at why we are doing it, in most cases it is to get us to our end goal on the road map at which point, even if I disagree with the task I remember it’s an iterative process and we are always iterating towards the goal, that is all it normally takes for me to get back to being motivated. I am not suggesting that will work for you, but you should have a look and a think and see why you are not being motivated and then come to terms with it. For me it is always abut getting to the end goal, as long as we’re doing that I’m happy, I use to struggle more with it in the past, always wanting the perfect solution, but after time I realised that is not really feasible so I came to terms with accepting little victories that are towards the goal rather than the end goal in one step.

Summary

As nice as it is to blame everyone else for you not being motivated to do something, you have to take some responsibility to get motivated. There’s only so much your boss will be able to do, they may throw money at you which might well help in the short term but it won’t in the long term. So look at yourself and why you are not motivated and try to work out what you need to do to get yourself motivated about the task in hand.