This tutorial will explain how to create a new Puppet environment using best practices such as version control, a site-local module, and roles. It assumes the reader will be using Ubuntu. The instructions were verified with Ubuntu 12.04.
This tutorial was inspired by Chris Hoge's excellent openstack-deploy tutorial.
- Setting Up Version Control
- Configuring the Puppet Server with Puppet
- A Note About the Module Subcommand
- Keeping Track of Modules
- Configuring NTP
- Class or Include?
- Roles and Profiles
- The First Puppet Run
- Configuring the Firewall
- Hiera and the Firewall
- Configuring Puppet as a Puppet Master
- Creating the Puppet Agent Role
- Committing Everything
Install the PuppetLabs apt repo:
The version of Puppet that will be installed will be the latest version from PuppetLabs.
git will be used for source control and
vim is installed as an editor.
Setting Up Version Control
It's a best practice to keep all the configuration management information in a version control provider such as
git. You may host the repository in a public area such as Github, but since the repository will probably contain sensitive information, it's recommended to use an internal
git server. Managing one is very easy by using gitolite.
Some people keep the entire
/etc/puppet directory in the repository. There's nothing wrong with this and if this is what you'd like to do, the following should work:
Another way of keeping all Puppet information in the repository is to create a module that will only be used for the particular Puppet environment being created. This method will be used for this tutorial. Begin by creating a module:
ext directory will be used for supplemental and extra files in the environment. In this case, it's currently holding the
site.pp manifest has an
import statement to the
nodes.pp manifest – this is a common pattern in Puppet environments.
site.pp will also host any global Puppet settings, if any.
nodes.pp manifest will contain the
Configuring the Puppet Server with Puppet
At this point,
puppet.example.com is a pretty bare server with only a few packages installed. It has Puppet installed, but it's not really being used for anything. Now we'll begin using Puppet to configure this server.
Things begin to break when two servers with skewed times try communicating with each other. To ensure this doesn't happen, NTP will be installed and configured. First, install the
puppetlabs/ntp Puppet module:
A Note About the Module Subcommand
puppet command has a built-in subcommand to install modules. It's able to find the module by looking it up at the Forge.
There are pros and cons to this. On one hand, it provides an easy way to install a module plus any other modules it depends on. On the other hand, if you install a module that has a conflicting dependency with another module, the command will break. Additionally, sometimes the version of the module hosted at the Forge is outdated. When this happens, you need to manually download the module from its designated home – usually github.
I usually clone the modules directly from Github. The
puppet module command was included in this tutorial as an example.
Keeping Track of Modules
Several modules will be installed in this environment. To keep track of what modules have been installed, I usually add them to a bash script:
This way, if
puppet.example.com ever needs rebuild, this script can be run to install all required modules.
Note that there is a great project called Librarian-Puppet that can better handle module management. It will not be used in this tutorial, but I highly recommend looking into it.
Now that the
puppetlabs/ntp module is installed, it can be used to install and configure NTP on any server under Puppet control. The
puppetlabs/ntp module is a pretty simple module and rarely needs any parameters.
nodes.pp file, add the following:
Class or Include?
In Puppet, if the module being applied does not take any parameters, it can be applied using an
include statement. However, if parameters are required, then
class is required. While I like the succinct form of
include, I find having two forms of applying a module confusing and therefore stick to
Roles and Profiles
There's an issue with this, though. This class will need applied to every server:
There's a lot of repeated configuration here and it'll only get worse as more modules are applied to many of the same servers. A better way to apply modules to nodes is to use the Roles and Profiles pattern. This pattern is described here:
- Designing Puppet - Roles and Profiles
- Puppet Roles and Profiles with a Simple Module Structure (part 2)
A good role to start with is the "base" role. This role will be applied to all servers, so it's important that this role contains very generic and global settings. To start, create a new manifest called
/etc/puppet/modules/site/manifests/roles/base.pp with the following contents:
anchor resource is a common pattern used in Puppet to contain declared classes in a parent class. Intuitively, one would expect classes defined in a parent class to be applied when the parent class is applied. However, due to a bug in Puppet, this does not happen. Instead, classes are applied outside of their parent class which can cause major issues with ordering. By configuring all classes in the parent class to require the
anchor, they are now "tied down" to the parent class.
Note that this only applies to the
class resource. All other resources are not affected by this bug.
site::roles::base role can be applied to all servers:
With only the
ntp module being used in
site::roles::base so far, this looks the exact same as before. To better show the usefulness of roles, create a new manifest called
/etc/puppet/modules/site/manifests/base/packages.pp with the following contents:
Now add the following to the
site::roles::base role, before
All nodes with the
site::roles::base role will now have a standard set of packages installed.
The First Puppet Run
At this point, Puppet can be run for the first time. If all goes well, NTP will be installed and running when Puppet has finished:
Configuring the Firewall
puppetlabs/firewall module will be used to build the basis of a deny-by-default firewall.
puppetlabs/firewall module by cloning it from github:
Also add it to the
Next, create a directory called
/etc/puppet/modules/site/manifests/firewall. In this directory, two manifests callsed
post.pp will be created:
Next, add the following to the
site::roles::base role, before the base packages are applied:
Now all servers will have a deny-by-default firewall applied. I wouldn't recommend applying this configuration yet because you'll be locked out if you're working on this server remotely.
Hiera and the Firewall
This is a good place to introduce Hiera - a tool to store structured configuration data outside of Puppet manifests. Hiera is installed by default with the Puppet package, so installing a
hiera package is not needed. However, to utilize Hiera's merging feature, the
deep_merge gem needs to be installed:
To configure Hiera, create
/etc/puppet/modules/site/ext/hiera.yaml with the following contents:
Next, link this configuration file to two locations:
At the moment, the only hierarchy configured is
common. This means that Hiera will only read data from a single file:
Add the following to that file:
Add any other networks or hosts (
/32) to this list that you need.
Next, add the following to the bottom of
This block of code will pull the list of trusted networks from Hiera, loop through them, and create a firewall rule allowing all traffic to the servers from these networks. In order to use the iteration feature (
each), add the following to
/etc/puppet/puppet.conf under the
Now the next time you apply the Puppet configuration, a deny-by-default firewall will be enabled with explicit allow rules for each trusted network you specified in Hiera.
Configuring Puppet as a Puppet Master
Up until now, the Puppet configuration has been applied by using the standalone
apply command. Stanalone mode is fine and has many uses, but requires the entire Puppet configuration (
/etc/puppet) to be available on every server running the
apply command. This section will configure
puppet.example.com as a Puppet Master which allow other servers to use
puppet.example.com as a central Puppet repository.
The Puppet Master service will be available on port 8140. The
puppet command is able to launch as a daemon and bind to port 8140, however, this is only recommended for testing purposes. Instead, Apache and Passenger will be used to host the Puppet daemon.
In addition, PuppetDB will also be installed and configured. It will use a PostgreSQL backend.
In order to set this up, a few new modules are needed. The first is the puppetlabs/apache module:
The next module is the puppetlabs/passenger module:
The puppetlabs-opertions/puppet module will configure Puppet:
The puppetlabs/puppetdb module will install and configure PuppetDB:
The puppetlabs/postgresql module will install and configure PostgreSQL:
The puppetlabs/ruby module is needed by the
The puppetlabs/concat module is needed by the
And finally, the puppetlabs/inifile module will assist with configuration by providing an easy way to manipulate INI files:
Creating the Puppet Master Role
Similar to the
site::roles::base role, a
site::roles::puppet::master role will be created which will abstract the Puppet Master configuration into a single manifest.
/etc/puppet/modules/site/manifests/roles/puppet/master.pp with the following contents:
A lot of the parameters in the
::puppet::server class can be ignored – they're used for configuring very specific environments.
Note the use of Hiera in the
::puppet::agent class. In
common.yaml, add the following:
puppetdb classes require no parameters – the
puppetlabs/puppetdb module does a very good job at setting up PuppetDB with sane defaults.
puppet::master role is a much better example of roles than the original
site::roles::base role. Note the use of multiple modules as well as local site-specific configurations from the
site module. This is all rolled into one manifest that can easily be applied to a server.
Applying the Puppet Master Role
puppet.example.com will be a Puppet Master, add it to the
puppet.example.com node definition:
With all of this in place, run Puppet:
You might have to run this twice. This is normal for large configuration runs or bootstrap scenarios such as this.
When everything has finished, you should now be able to switch to
puppet agent mode:
Note that you will probably see Puppet trying to start or stop the built-in
puppet daemon service. This is a bug with the
puppetlabs-operations/puppet module and will probably be fixed soon.
Creating the Puppet Agent Role
With the Puppet Master role configured, a role for the Puppet Agent should be created. Create
/etc/puppet/modules/site/manifests/roles/puppet/agent.pp with the following contents:
Applying the Puppet Agent Role
Due to how the
puppetlabs-operations/puppet module is designed, the
::puppet::agent classes cannot be declared separately. This is why
::puppet::agent is declared in the
site::roles::puppet::master role. Because of this, the
site::roles::puppet::agent role can't be applied to the
Instead, it will have to be applied to individual nodes:
A lot of work has been done here. To see all of the changes that were made, do the following:
All of this should be commited into
This tutorial went through the initial steps needed to create a brand new Puppet environment. It described how to lay the foundation of a scalable Puppet environment by using version control, a site-local module, and roles.