Introduction
Puppet is a systems administration automation tool for managing one or more servers (where "more" can scale to hundreds or thousands of servers).
The puppet master accepts connections from puppet nodes running on remote servers.
The puppet nodes connects to a puppetmaster, and asks for a configuration and a list of resources. If the configuration or any resources have changed, it performs the changes according to instructions in the configuration.
As your puppet installation grows, you'll feel the need for a bit of further control. You can easily shoot yourself in the foot with puppet if you add the wrong bit of configuration, and since it scales quite well, any mistake can be propagated quite far.
To avoid distributed disasters, we can add a staging environment to the puppet master.
Staging
A staging environment is a mirror of the production environment that can be used to test and review new versions of your modules, manifests, facts and files.
In this environment, errors and incompatibilities can be spotted and corrected in order to prevent them from appearing in the production environment
Environments
Puppet and puppetmaster can use a different named environment in order to select a different set of configuration for the nodes. We do not have to install a different puppetmaster, or run puppetmasterd on a different port to do this.
The default environment of puppet is "production". In previous versions of puppet the default environment was "development". Regardless of default environment, we'll refer to this as the "production" environment. If you tell puppet to use an environment unknown on the puppetmaster, the master will select the default environment.
By adding another environment to your puppet master, you can offer this set to your nodes so they can test against other versions of your manifests and data.
The different environments will share most metadata, but differ on the set of manifests, files, facts and modules that are presented to the puppet node.
While the environment can separate most things, like SSL certificate handling, we'll use a common SSL CA for both testing and production. If the system is certified by puppetca on production, it'll work on testing as well.
Version control
Using a Version Control System for holding your manifests and data is important. Handling multiple environments makes it even more important.
Workflow
The workflow becomes like this:
- Make changes to manifests or data
- Check in the changes.
- Check out the changes in the testing environment
Test the changes on the pupppet node.
"puppetd -t --noop --environment=testing"If you need to make more changes, go back to 1.
- Check out the changes in the production environment
- Tell the running puppetd to look for changes, or run puppetd manually
Invoking puppetd
Invoke puppetd with "-t --noop --environment=testing" to print changes from the testing environment only, as compared to the running production environment.
Configuring puppetmaster
The environment argument passed to puppetd makes it across to the puppetmasterd when puppet connects. Puppetmasterd looks for configuration for this environment in /etc/puppet/puppetd.conf.
Add a new section to /etc/puppet/puppet.conf, and give it the same name as the environment we're going to use for testing. I call it "testing", but there's nothing wrong with "frobnication", or any other thing.
/etc/puppet/puppetd.conf:
[testing]
<statements are added here>
Directory layout
If we're going to have multiple environments, it makes sense to have them as similar as possible, under a shared directory root.
Use this root for the data that is going to differ between environments: manifests, templates, and files.
I'm using "/srv/puppet/env" for this in this document
Under this directory root, I create one directory per environment, with the same name as the environment used.
- /srv/puppet/env/production
- /srv/puppet/env/testing
Under each of these directories, I have the following content:
- fileserver.conf (setting: fileserverconfig)
- templates/ (setting: templatedir)
- manifests/ (setting: manifestdir)
- modules/ and clients/ (setting: modulepath)
Configuration
Add the following to /etc/puppet/puppetd.conf:
[testing]
manifestdir=/srv/puppet/env/testing/manifests
templatedir=/srv/puppet/env/testing/templates
fileserverconfig=/srv/puppet/env/testing/fileserver.conf
modulepath=/srv/puppet/env/testing/modules:/srv/puppet/env/testing/clients
Stored configuration
Depending on how you use the stored configuration, you have a few choices:
If you're not using collections across nodes, you can turn storeconfigs off. This will increase the speed of your testing.
If your testing may interfere with your production configuration, you should use another database for testing.
Scripting
To reduce the risk of errors, I recommend you implement a few scripts. puppet-test
Used on the nodes to test against the "testing" environment. This script is really simple, and is included here:
#!/bin/sh
/usr/bin/puppetd -t --noop --environment=testing $@
Usage:
puppet-test [puppetd options]
Examples:
puppet-test --tags httpd Check for changes for the httpd tag in the testing environment
puppet-update
Used on the master to update pieces of the testing and production environment from the repository.
The implementation of the script is dependent on the version control method used for the manifests, templates and files.
This is the usage and examples of a script I use.
Based on your preferences, environment, VCS, work habits, grade of paranoia and ITILization, you should roll your own here.
Usage:
Usage: puppet-update [options]
-m : Update a module used by many clients. The default is to update a client module in the clients/ folder, prefixed with "c_", as well as a file called .pp in the manifests/nodes directory.
-n : Do not update, dry-run and show changes only.
-p : Update the production environment. The default is to update the testing environment.a
Examples:
puppet-update acme
In the testing environment, update the "clients/c_acme" module directory, and the manifests/nodes/acme.pp file from the repository.
puppet-update -m openssh
In the testing environment, update the "modules/openssh" module directory from the repository.
puppet-update -p -n -m httpd
In the production environment, show a diff of changes that would be made to the "modules/httpd" module directory from the repository.
To tag or not to tag
If you're creating tags in your VCS, you gain further control of your production environment. It makes it easier to select which commits make it to production, and to compare and revert if needed. It will also add another item to your workflow.
If you're in a high-SLA environment, adding tags is worth considering.
I would add the following to the puppet-update script:
-t <tag> - select tag to use for this module or client.