terrarum

Introduction

Waffles is a simple configuration management system written in Bash. It's small, simple, and has allowed me to easily sketch out various environments that I want to experiment with. If you want to jump right in, head over to the homepage. The rest of this article will cover the history and some personal thoughts on why I created it.

Update: I apologize if Jekyll flags this as a new post in the RSS feed. After originally publishing this article, I thought of a few other items I wanted to mention. In addition, I noticed Waffles was posted to Reddit, so I wanted to address a few of the comments made.

• Introduction
• Defining the Problem
• Attempted Solutions
  • First Version
  • Second Version
  • Roadblock
  • Third Version
  • Fourth and Final Version
• Current Status
• Is Waffles Competing?
  • Is Waffles Better Than X?
• Conclusion

Defining the Problem

The last article I wrote, Puppet Infrastructure 2015, was quite a beast. In a way, it was a cathartic exercise to write down all of my practices to sanely work with Puppet every day. After I published the article, I couldn't get over the fact that all of those practices are just for Puppet – not any of the services Puppet sets up and configures. It didn't sit right with me, so I began to look into why I found configuration management systems so complex.

"Complex" and "Simple" are subjective terms. One core focus was the fact that Omnibus and all-in-one packages are becoming a popular way to ease the installation and configuration of the configuration management system. In my opinion, when configuring your configuration management system becomes so involved that it's easier to just have an all-in-one installation, that's "complex". I wanted to see if it was possible to escape that. I wanted to see if it was possible to create a configuration management system that was able to work out of the box on modern Linux systems and able to be installed by simply cloning a git repository.

Secondly, I wanted to see if it was possible to strip "resource models" down to more simple components. Some systems have modeled resources in a specific language, like Python, while others have chosen to use a subprocess to interact with the best native command available. Both methods have their merits, though I favor the latter more. I wanted to take that method further: why not just use a Unix shell and related tools? After all, for decades, the standard way of interacting with a Unix-based system has been through the command-line.

That's not to say that resource models in configuration management systems are the way they are for no reason. Take Puppet's Types and Providers system, for example. Decoupling the "type" and "provider" has enabled Puppet to provide a common resource interface for a variety of platforms and backends. In addition, the Types and Providers system provides the user with a way to easily manage all resources of a given type on a system, provide input validation, and a lot of other features.

I have nothing but respect for the Types and Providers system. But I still wanted to see if it was possible to create a tool that provided the same core idea (abstract a resource into a simple model that allowed for easy management) in a more simple way. It's an experiment to see what happens when a robust catalog system, input validation system, etc were removed for something more bare. Would chaos ensue?

Third, I wanted to break out of the "compiled manifest" and "workflow" views of configuration management systems. Every configuration management system has some sort of DSL, and for a good reason. You can read about the decision to use a DSL with Puppet here. I have nothing against DSLs (you can see how plain Bash evolved into the Bash-based DSL in Waffles here), but I wanted to see if it was possible to not use a compiled manifest or workflow, and instead use a plain old top-down execution sequence.

With these thoughts in mind, I set off to see what I could build.

As a real-world test to validate my solution, I created a detailed list of steps that are required to build a simple, yet robust, Memcached service (this is the reason why there are so many references to Memcached in the Waffles documentation).

You can easily get away with installing Memcached by doing:

$ sudo apt-get install -y memcached

But what if you also need to:

• Edit /etc/memcached.conf
• Have other common packages installed (logwatch, Postfix, fail2ban, Sensu, Nagios, etc)
• Have some standard users or SSH keys installed
• Have some standard firewall rules installed

You can see how the idea of a "simple" Memcached service quickly becomes a first-class service in your environment. But does that mean the configuration management system must be complex to satisfy this?

Attempted Solutions

First Version

My initial solution was written in Golang. This was because I was doing a lot of work with OpenStack and Terraform and it minimized context switching to stay with Go. I ended up with a working system that compiled shell scripts together into a master shell script "role" (my term for a fully-defined node) and referenced data stored in a hierarchical YAML system (a very watered down Hiera).

It looked like this:

roles:
  memcached:
    server_types: [lxc]
    environments: [production, testing]

The program would then search a structured directory for shell scripts that contained memcached, lxc, production, and testing and create two master scripts:

• memcached_lxc_production.sh
• memcached_lxc_testing.sh

I didn't like it at all. The first thing I removed was the YAML data system. I felt that it carried too much overhead. Then I decided to get rid of Go altogether. The Go component was only being used to organize shell scripts. Why not just make everything in shell?

Second Version

The next major iteration was completely written in Bash. It used Bash variables to store data and Bash arrays to store a list of scripts that should be executed. Here's what a "role" looked like:

ATTRIBUTES=(
  [environment]="production"
  [location]="honolulu"
)

RUN=(
  "site/acng.sh"
  "site/packages.sh"
  "rabbitmq"
  "rabbitmq/repo.sh"
  "rabbitmq/server.sh"
  "openstack/rabbitmq.sh"
)

The ATTRIBUTES hash stored key/value pairs that described unique features of the role. The RUN array stored a list of scripts that would be executed in a top-down order. This was working very well and it enabled me to easily deploy all sorts of environments. I wasn't totally happy with the design, but I couldn't figure out what exactly I didn't like.

Roadblock

That got put on hold when I ran into a major roadblock: I was deploying a Consul cluster when I ran into the need to use a JSON-based configuration file. What would be the best way to handle it?

• Static files meant data embedded inside the file and outside of the data system
• Templates meant another layer of programming logic
• External languages broke the Bash-only feature

I was stumped for a few days until I realized: Augeas! I've always had a lot of respect for the Augeas project but never had a reason to use it – until now. Even better, Augeas was able to cleanly parse and write JSON files. So I made Augeas an optional, but encouraged, component of Waffles.

Third Version

Now back to figuring out why I didn't like the current iteration. I realized I didn't like the format of the "role". I wanted the "role" to look more like a shell script and not like a metadata file that describes a system.

So I made some changes and was happier. The above RabbitMQ role now looked like:

stdlib.data common
stdlib.data openstack/rabbitmq

stdlib.profile site/acng
stdlib.profile site/packages
stdlib.module rabbitmq
stdlib.module rabbitmq/repo
stdlib.module rabbitmq/server
stdlib.module openstack/rabbitmq

Fourth and Final Version

There was still one part that bugged me: modules. In the above example, rabbitmq and openstack were both modules. I didn't like how I had profiles and modules mixed in the role. I refactored the above so that profiles were just an abstraction of modules, like in Puppet, but I didn't like that either. Finally, I decided to get rid of modules altogether. You can read more about this decision here.

So modules went away and the above turned into:

stdlib.data common
stdlib.data openstack/rabbitmq

stdlib.profile common/acng
stdlib.profile common/packages
stdlib.profile rabbitmq/repo
stdlib.profile rabbitmq/server
stdlib.profile openstack/rabbitmq

At that point I was very satisfied with how it looked. I used it for a week or two and was still happy and decided to finally release it publicly.

Current Status

Now I have a tool that I find fits my definition of "simple" and "intuitive".

But Waffles is nowhere near complete. The only milestone that has been reached is that I'm happy with its core design. Waffles certainly isn't at feature parity with other configuration management systems. For some features, this is only a matter of time. For other features, they just aren't applicable to Waffles's core ideas. For example, I want to have some kind of pull-based mechanism so clients can contact a central Waffles server. At the same time, I don't feel a PuppetDB or Etcd database belongs in Waffles.

Is Waffles Competing?

Waffles is just another tool. I have no intention to market Waffles as a competitor to other configuration management systems. The only reason Puppet has been called out throughout this article is because it's the system that I'm most knowledgeable with.

Is Waffles Better Than X?

Again, Waffles is just another tool that does a certain action in a certain way. The only thing that matters is what tool you find most useful. I have nothing but respect for every other configuration management system available today. If it wasn't for those projects, I wouldn't have been able to learn what I like and don't like in order to create my own.

As mentioned above, Puppet is called out a lot because it's the system I'm most knowledgeable with. Puppet is a great tool and I still use it everyday. I've also spent time with Chef, Ansible, Juju, and Salt and find them to be great tools, too.

I will make one comment about Ansible, though, because comparisons will be made (I'm assuming because Ansible is known to be a small and simple configuration management system): I feel Waffles and Ansible are two separate tools that do things very differently. I feel the biggest difference is that Ansible is a workflow-based system while Waffles is just a bunch of shell functions that execute from top to bottom.

I take no offence if you don't like Waffles. You can see from a lot of the other tools and code I've created that I do things very differently sometimes – that's just me.

Conclusion

This article gave the history and thought process behind a new project of mine: Waffles. Try it out and tell your friends. Feedback is welcome!