terrarum

Introduction

This article is the first in a series about testing Puppet manifests. It will cover the basics such as puppet-lint and Smoke Testing and detailed testing with rspec.

• Introduction
• Why Test?
• Testing Puppet
  • Smoke Testing
  • RSpec
    • Setting Up
    • Example
      • Inventory
      • Fixtures
      • Basic Structure
      • Default Parameters
      • First Context
      • The Second Context
      • A Note About Facts
      • Final Result
  • Puppet Lint
• Conclusion

Why Test?

For systems administrators, testing usually involves configuring something on a test server before it's done on a production server. It's understandable, then, why the development world of testing (test driven development, unit tests, behavior driven development, etc) can seem very foreign.

Take, for example, applying the "sysadmin" method of testing to Puppet manifests: one would write a set of Puppet manifests, provision a test server, then apply the manifest and watch for errors. If there were any, the manifests would be fixed and the test deployment would be re-run.

This kind of testing can be very time consuming. Further, it is only verifying that the manifests are valid for one type of server.

By utilizing methods from the development world, not only can the testing cycle be simplified, but also automated and categorized. Multiple tests can even be created to validate the same set of manifests against different types of servers.

Yes, these methods might seem foreign at first, but if you give them a chance, you'll find that your configuration management system is a lot more maintainable.

Testing Puppet

There are two main ways to test Puppet manifests for errors.

Smoke Testing

The easiest and most basic way is known as Smoke Testing. I'm sure lots of people have used this method without even knowing it was an official way of testing.

Smoke Testing involves writing a basic manifest, then applying that manifest with the --noop option. That's it.

As an example, let's say you were writing a manifest to configure Apache on a server. It might look something like this:

include apache

apache::vhost { 'example.com':
  docroot     => '/var/www/example.com',
  serveralias => ['www.example.com']
}

Now instead of applying this manifest as-is, try it with --noop:

$ puppet apply --verbose --noop example.pp
...
Error: Invalid parameter serveralias at /root/example.pp:6 on node example.com

Notice the caught error.

serveralias is not a correct parameter, serveraliases is.

Smoke Testing is a very low-cost form of testing and I highly recommend utilizing it.

RSpec

RSpec is a more advanced form of testing. It provides (yet another) Ruby DSL that describes how code should behave. RSpec takes a lot longer to set up and learn than simple Smoke Testing, but if you're managing a large and complex module, it's very useful and worth the effort.

The rspec-puppet project is an extension of RSpec that makes testing Puppet manifests with RSpec easier.

Setting Up

The easiest way to get started with rspec-puppet is to follow this blog entry. It covers how to install and configure rspec-puppet by using the puppetlabs_spec_helper gem. It also walks through a basic manifest and how to write tests for that manifest.

The rspec-puppet project page also has a helpful tutorial.

Example

I wanted to include an example because when I started learning RSpec (and I'm still very much a beginner), any example I could find was helpful. Just trying to contribute back.

This example uses my puppet-reprepro module.

For this example, tests will be created for an existing manifest. While testing methodologies state that you should write the tests first, in my opinion, this rarely happens. Keeping up with that tradition, this is how I would write an initial set of tests for a manifest. Once the initial set is written, more tests can continue to be added to account for future features.

I'll use the distribution.pp manifest. It's a small manifest that can easily highlight most features of rspec-puppet.

Inventory

First, inventory the resources in the manifest. In distribution.pp, there are the following resources:

• concat::fragment
• exec
• file
• cron
• concat

Next, check for conditionals. In distribution.pp, there are two.

When testing conditionals, make sure you test all logical branches.

Next, check for any classes that are included in this manifest. There are two in distribution.pp.

Finally, review the parameters, their defaults, and any facts that might need to be set (for example, the Operating System or system architecture).

Fixtures

With everything inventoried, make note of any modules that the manifest uses. These modules need to be added to the .fixtures.yml file.

In this case, the concat module is used, so the following is added to .fixtures.yml:

fixtures:
  repositories:
    concat: https://github.com/puppetlabs/puppetlabs-concat

Basic Structure

Once the fixtures are in place, create the actual RSpec test file. This file will called modules/reprepro/spec/defines/distribution_spec.rb:

require 'spec_helper'

describe 'reprepro::distribution' do

end

Default Parameters

Next, I create a hash for the default parameters:

describe 'reprepro::distribution' do
  let :default_params do
    {
      :repository     => 'localpkgs',
      :origin         => 'Foobar',
      :label          => 'Foobar',
      :suite          => 'precise',
      :architectures  => 'amd64 i386',
      :components     => 'main contrib non-free',
      :description    => 'Package repository for local site maintenance',
      :sign_with      => 'F4D5DAA8',
      :not_automatic  => 'No',
      :install_cron   => true,
    }
  end
end

The hash is first built by using the default values of the Class Parameters. Next, Required Parameters are added with values that can be globally used for all tests.

There's one caveat to this, though. Note line 53. $basedir is supposed to pull its default value from the reprepro::params class. To my knowledge, rspec-puppet does not support this, so instead, just hard-code the value in the default params:

describe 'reprepro::distribution' do
  let :default_params do
    {
      :repository     => 'localpkgs',
      :origin         => 'Foobar',
      :label          => 'Foobar',
      :suite          => 'precise',
      :architectures  => 'amd64 i386',
      :components     => 'main contrib non-free',
      :description    => 'Package repository for local site maintenance',
      :sign_with      => 'F4D5DAA8',
      :basedir        => '/var/packages',
      :not_automatic  => 'No',
      :install_cron   => true,
    }
  end
end

First Context

A context is a group of related tests. For example, tests related to a specific Operating System. In this example, the first context will be tests involving valid results for when all default values are used.

All I usually do is check and see if all resources that were inventoried earlier are successfully applied:

  context "With default parameters" do
    let(:title) { 'precise' }
    let :params do
      default_params.merge({
        :name     => 'precise',
        :codename => 'precise'
      })
    end

    it { should contain_class('reprepro::params') }
    it { should contain_class('concat::setup') }

    it do
      should contain_concat__fragment('distribution-precise').with({
        :target => '/var/packages/localpkgs/conf/distributions'
      })
    end

    it do
      should contain_exec('export distribution precise').with({
        :command => "su -c 'reprepro -b /var/packages/localpkgs export precise' reprepro"
      })
    end

    it { should contain_file('/var/packages/localpkgs/tmp/precise') }
    it { should contain_cron('precise cron') }

    it { should_not contain_concat('/var/packages/localpkgs/conf/updates') }
  end

A few things to note about this. The first is

    let :params do
      default_params.merge({
        :name     => 'precise',
        :codename => 'precise'
      })
    end

Merging new parameters into the Default Parameters is an easy way for multiple tests to re-use a standard set of parameters and modify them as needed.

In this case, :name and :codename are being set. I did this here because these parameters would not be known until the title of the reprepro::distribution is known:

    let(:title) { 'precise' }

:name, :codename, and :title could have all been set in the default parameters.

The next area to note is

    it { should_not contain_concat('/var/packages/localpkgs/conf/updates') }

Remember I mentioned that we'll be testing the different branches of each conditional. By default, the $update parameter is '', which should translate to false and so the resulting catalog will not contain the concat resource.

A default value of '' is probably not the best choice. Setting $update to false would have probably been better. This brings up an interesting point about writing tests for an existing manifest: you're able to find areas in the code that can be done better. This isn't the best time to change this value, though. It'd be better to finish the tests and then go back fix both the manifest and test later.

The Second Context

For the second context, I check to see what happens when $update is set to true:

 context "With update set" do
   let(:title) { 'precise' }
   let :params do
     default_params.merge({
       :name     => 'precise',
       :codename => 'precise',
       :update   => true
     })
   end

   it { should contain_concat('/var/packages/localpkgs/conf/updates') }
 end

This context only checks for the one resource. This is because everything else in the first context still applies. No sense in repeating code.

A Note About Facts

If you save this file and run:

$ rake spec

You'll see an error about concat_basedir not being set. This is because the manifest is using the concat module.

Setting $concat_basedir in the default_params hash raises another error because $concat_basedir does not exist as a parameter in distribution.pp. To get around this error, I did the following at the top of the rspec file:

  let :facts do
    {
      :concat_basedir => '/foo'
    }
  end

Maybe there's a better way to do this.

Final Result

With all of this in place, you should have a distribution_spec.rb file that looks like this.

You can run the tests by doing:

$ rake spec

Puppet Lint

Running puppet-lint against your Puppet manifests is a great way to make sure you're writing Puppet with best practices in mind.

To install puppet-lint, do:

$ sudo gem install puppet-lint

Next, add the following to your Rakefile:

require 'puppet-lint/tasks/puppet-lint'

Now you can run puppet-lint by doing:

$ rake lint

Conclusion

This concludes Part 1 of Puppet Testing. This part covered the benefits of testing, basic Puppet testing with Smoke Tests, advanced testing with rspec-puppet, and finally, best practice checking with puppet-lint.

I should note that I'm still very new to testing. If there are better ways to do any of the above, I would love to hear them.