Magnificent manifests

Back in the day, we use to have maybe 7 node manifests per environment and 6 or 7 environments, which is a lot of individual nodes to manage. As a result not everything was always the same.

We got to the point where each environment, although using the same modules was not always quite right, we used variables to set each environment up, but each variable was in each environment and different, yes you can keep it in check with diff and good process, but that’s time consuming. Why not just have one manifest for each tier of the application? that’s what we have now every tier is identical in terms of its use of the manifest, but obviously every environment has its own settings. So how do you have 1 manifest 100s of environments and guarantee that everything is the same across each environment except a small number of differences which can be easily seen in one file.

Now you could try and manage the different sets of vars through node inheritance but you then end up with a quite complex set of inheritance which can be a pain to track through. At first I was a little confused at how we would reduce our manifests down and still be able to set variables in the right places. Before going to far, lets look at a typical node inheritance set up.

[12:56]msmith@pinf01:/etc/puppet/manifests$ tree
.
|-- company
|   `-- init.pp
|-- dev
|   `-- init.pp
|-- site1
|   |-- init.pp
|   `-- nodes.pp
|-- site2
|   |-- dev_nodes.pp
|   |-- init.pp
|   |-- nodes.pp
|   `-- test_nodes.pp
|-- site3
|   |-- init.pp
|   `-- nodes.pp
|-- prod
|   `-- init.pp
|-- site.pp
`-- test
    `-- init.pp

In this set up, The company init.pp sets up company wide details, domain names, IP addresses for firewall rules or anything that is applicable to multiple sites. The site init.pp inherits the company one and sets up any variables needed for that specific site like DNS servers. In some of the sites we have additional configuration which inherits that site’s init.pp and sets more specific information for say the test or dev environment and finally there is an environment init.pp which sets variables for that type of environment such as environment prefix or other settings that are specific to that environment.

With this set up it gives you plenty of room to grow but can be harder to work out where the variables come from as you will have to track them back through many levels of inheritance; this can make it hard for new people to the team to identify where the variables should be set which is made harder with parameterised classes and defaults.

Now with classes you can create a class/subclass that is used to set a majority of your defaults and this can be useful if you have multiple companies or distinct groups which always insist on having different settings, the advantage of this is that your node declarations then remain minimal. However, this can also make things complicated as you have meta classes that set some of the params needed for a class but not all, so you’ll still end up passing in params within your nodes to the meta class which in turns passes them into your modules and so on. This is typically more inheritance and paramaters than most sysadmins can handle.

So we had the same issue, it was never a 5 min talk to explain what is set where and when because it could be set in 3 or 4 nodes, passed as a param and this was the same for each of the 6 environments. After much talking and disbelief we eventually started on a journey to reduce 7 environments and what was 50+ node files (each controlling multiple nodes via regexp) down to a handful.

[msmith@puppet manifests]$ tree
.
├── account.pp
├── roles
│   ├── app1.pp
│   ├── app2.pp
│   ├── default.pp
│   ├── app3.pp
│   ├── app4.pp
│   ├── app5.pp
│   └── app6.pp
└── site.pp

So to start with, the modules are still parameterised as before, heavily so to make it easy to turn features on and off and where possible everything is set as a default within the module within a params.pp file, so there is one place within a module to change the defaults for all subclasses. Each appX.pp will load which ever sets of classes it needs and where possible it will set the params to make use of various internal variables that are needed.

Each appX.pp is inherited from the default which sets most of the classes we want on each app such as ssh, mcollecitve, aws tools. This inherits the account.pp which sets variables that are relavent to each account we use, or classes that are relavent to each account, the account in this case is an amazon account. We set different yum servers, infrastructure servers and so on.

Now we still have environments, and we choose to load the variables for this via hiera, but we have gone to great lengths to restrict the variables in the environment down to those that are absolutely necessary as such we have not yet needed to make sue of hiera’s hierarchy but we understand the necessity of keeping that option open to us.

The new structure means we have at the moment no more than 5 places to edit vars for every app / env / node. This particular structure works for us because we run a small set of node types but in multiple environments. As time goes on the roles manifests directory will grow to incorporate new servers and if we need to deal with nodes of a similar role type apart from option X/Y/Z we will make use of hiera to add another layer to the system.

Next week I’ll go through the node manifests in some detail which should hopefully cement some of this blurb.