Structure is good

Like everyone that has used puppet and is using puppet; the puppet structure we use to use to manage our nodes was relatively straight forward. But before getting into that lets go into the two or three ways you would know of already.

1. Node inheritance, Typically you define a number of nodes that define physical and logical structures that suite your business and inherit from each of these to end up with a host node that has the correct details relavent to the data centre it is in and its logical assignment within it.
2. Class inheritance, Similar to node inheritance, you create your modules which are typically agnostic and more than likely take a number of parameters; then create a module that contains a set number of node types, for example, “web” nodes. So the web node would include the apache module and would do all of the configuration you expect of all apache nodes, this can be done to the point of each node manifest simple including a web, or wiki type class.
3, By far I imagine the most common, is a mixture of them both.

Either of these methods is fine, however what will essentially happen is duplication and you’ll be faced with situations where you want 90% of a web node but not that last 10%; you then create a one of module or node and before you know it you have several web nodes or several roles defining the same task.

It’s fair to say we do not do any one of those, node inheritance a little to save duplication of code but that’s about it, I’ll touch on this more next week, but this week is about foundations, and in puppet that means modules.

Parameterise everything

Parameterised classes help with this level of flexibility but if your module isn’t fully parameterised it’s a bit of a pain, so a slight detour based on experience.

I imagine like everyone out there, I write puppet modules that are half arsed on most days. I parameterise the variables I need for my templates and stick to a simple module layout to stop my classes getting to busy. This isn’t a bad way of doing things, you get quite a lot of benefit with out a lot of investment in time, and I for one will continue to write modules that are half arsed, but with one difference, more parameters.

Occasionally I have to extend my module, some times I don’t have time to re-factor and make it perfect so I work around my module, if you’re looking to build technical debt, this is a good way of going. and if you continue down this route you will end up with modules that are complicated and hard to follow.
Initially when introducing module layouts to people they assume they will make life more complicated because you have split a module into classes, but the reality is rather than having 1 file with 100 lines you have 10 files with 10 lines, and of those 10 files you may only be using 2 files in your configuration which can make life a lot simpler than sifting through code.

A typical module layout I use will consist of the following

[root@rincewind modules]# ls example/
files  manifests  README  templates
[root@rincewind modules]# ls example/manifests/
install.pp  config.pp  init.pp  params.pp

For those not familiar with such naming schemes… init is loaded by default and will typically call include the install class and the config class. So if I want to install / configure an entire module that has had some reasonable params set I just “include modulename”. You can read more about modules and how I typically use them Here It was rather experimental at the time but with only a few tweaks it is now in a production environment and is by far the easiest module we have to maintain.

The main reason this structure works for us is that we are able to set a number of sensible defaults in params which reduces the number of params we need to pass in while allowing us to still have a lot of parameters on each class. typically it means our actual use of the classes even though they may have 40+ params won’t normally go past 10.

The big thing we did different with this module is to parameterise about 90% of all the configuration options, now you may think that’s fine for small apps but what about large ones, well we did it with our own product Alfresco, which is quite large and has a lot of options. Granted we can’t account for every possible configuration but we gave it a shot. Consequently we now have a puppet module that out of the box should work with multiple application containers (tomcat, jboss etc ) but also allows for the application to be run in different containers on the same machine.

The advantage of this up front effort on a module which is core to what we are doing is that we are now able to just change options with out having to do a major re-factor of code, adding more complexity in terms of configuration is simpler as it has been split into multiple classes that are very specific to their tasks. If we want to change most of the configuration it is simply a matter of changing that parameter.

Be specific
So you have a module that is nicely parameterise and that’s it, well not quite, you have to make sure your module is specific to it’s task, and this is one that was a little odd to get over. Puppet is great at configuration management, it’s not so good at managing dependancies and order, so we made the decision to simplify our installation process so rather than having the puppet module try and role out the application and multiple versions of it, with different configurations for different versions, it simply set’s up the environment and sets the configuration with no application deployed.

So yes, you can install within the module if you wish, and we use to do that and some complicated expanding / re-zipping of the application to include plugins, but we also have a yum repo, or git, or a ftp server where we can just write a much more specific script to deal with the installation.

By separating out the installation we have been able to cut a lot of code from our module that was trying to do far more than was sensible.

Be sensible
When you look through configuration there are sometimes options that will only ever be set if X or Y is set, in this case write more complicated modules to cope with that logic and don’t just add everything as a parameter. When you look through your module and you start seeing lots of logic, you’ve gone wrong.

Typically now when writing modules the only logic that ends up in the pp file is is feature enabled or disabled, if you try to make a manifest in a module all things to all people you end up with a lot of logic and complications that are not needed. To tackle this, after writing a base for the module I then cloned the params file and the application manifest which does most of the config and made it more specific to the needs of the main task that I’m trying to achieve. It is 90% the same as the normal application one but with a few extra params and allows me to add specific files that are 100% relavent to what we do in the cloud with out ruining the main application one incase we decide to change direction or if we need a more traditional configuration.

The point is more that you should avoid over complicating each manifest in the module, create more simple manifests that are sensible for the task in hand.

This module layout will give you something to work with rather than against, we spent a lot of time making our whole puppet infrastructure as simple as possible to explain, this doesn’t mean it’s simple it just means everyone understands what it does, me, my boss, my bosses boss and anyone else that asks “How does it work?”