This article will explain various methods of controlling
apt (the Debian and Ubuntu package manager) across several servers with Puppet and the benefits of doing so.
- Caching apt Packages
- Mirroring apt Repositories
- Creating Your Own apt Repository
- Use-case: Puppet 2.7.x
Caching apt Packages
apt downloads a package, it is locally cached on the server until you explicitly clear the cache. This gives you the ability to reinstall the package without having
apt download it again.
On individual servers, this works fine. But what about groups of several servers? If, for example, you ran the following on 15 different servers to install
tmux would have been downloaded from a public server 15 times. This is very inefficient – especially if the package was significantly large.
A solution to this is to designate a server to be an apt proxy server. All other servers will go through this server to download any packages. If the package does not exist on the apt proxy server, it will be downloaded from a public server. Now when
tmux is installed on 15 different servers, it is only downloaded from a public server once.
The best application for apt proxying is apt-cacher-ng.
To set up
acng by using Puppet, either use the
puppet module tool or download the module directly from github:
Next, apply the following class to the "apt server" (the "apt server" is the server that will provide all
apt services throughout this article):
acng is installed, it will automatically use the contents of your
sources.list file as the upstream
apt repository. This will work for 90% of use-cases.
On every other server that will use the apt server as a proxy, apply the following manifest:
Mirroring apt Repositories
acng is useful if your servers use the same subset of packages. If you find that your servers are downloading a wide variety of packages – maybe even most of the available packages in a repository – it would make more sense to just mirror the entire repository instead of letting
acng inevitably do it.
apt-mirror is an application that provides an easy way to mirror one or more repositories. To configure with Puppet, first install the
Next, apply the base manifest:
apt_mirror module comes with a defined type to add individual mirrors. For example, to mirror
contrib repositories for Ubuntu Precise, use the following:
apt_mirror installs a cron entry that will run once a day. It creates new mirrors that do not yet exist or updates existing mirrors with any changes.
In order to fully utilize the mirror, you will need to make it accessible to all of your servers. I recommend following this article for instructions on how to do this manually. I'll also describe how to do this with Puppet later in this article.
Creating Your Own apt Repository
There are several reasons to create your own
- To distribute your own packages
- To limit the packages available to servers
- To add packages not available in the standard repositories
apt repository is not the most straightforward task. The following page contains a very long list of tools to assist. The most popular tool seems to be
reprepro which is what I have chosen to use.
For a great overview on how to use
reprepro, please see this article.
reprepro with Puppet, first install the module:
The following manifest will install
reprepro and set up the
reprepro is installed, GPG will need configured. Modern
apt versions use GPG to sign both packages and files that maintain the
apt repository. In my opinion, it is difficult to use Puppet to maintain GPG keys, so I recommend doing this part outside of Puppet.
First, change to the
Next, generate a GPG key:
Follow and answer the prompts. I recommend not using a passphrase. Although it's insecure to do such a thing, GPG is just being used to sign arbitrary
apt repository files – you can always delete the repository and re-create it if you feel your key has become compromised.
Once the key has been created, export it and store it in the
reprepro Puppet module:
Now to continue with Puppet. The
reprepro Puppet module has the ability to crate and maintain several
apt repositories. Each repository contains one or more distributions. For most cases, and for the purpose of this document, only one of each will be created.
First, the repository:
And next the distribution:
sign_with value can be obtained by doing
Once these manifests have been applied to the apt server, you can begin adding packages to your custom repository. The
reprepro Puppet module installs a cron entry that will monitor the
$basedir/$repository/tmp/$distribution directory for any
*.deb files. If it finds any, it will add the files to the repository and clean the
tmp dir out.
Making Your Repository Available
Now that you have an
apt repository with packages, you will want all of your servers to be able to download those packages. You'll need to make your repository publicly accessible in order to do this. The easiest way is by installing a web server like Apache:
Next, create a virtual host that uses your repository as its
Also make sure your servers can access your GPG key that you generated:
Finally, tell your servers about your repository. You can use the Puppet
apt module to do this:
And use the following manifest on each server:
Now you should be able to perform an
apt-cache search on each server for packages in your repository.
Use-case: Puppet 2.7.x
One creative way of using both
reprepro is to ensure your servers are running the latest version of Puppet, but keep them at the
Puppetlabs maintains their own
apt repository at http://apt.puppetlabs.com. It's a great repository and it's always up to date. However, it makes no distinction between the 2.7 and 3.0 branches of Puppet. This means that unless you specifically specify a 2.7.x version of Puppet, you will get a 3.0 version.
Personally, I'm not ready to work with 3.0, but at the same time, I want my servers to have the latest 2.7 version.
The first step in easily doing this is to mirror the Puppetlabs
apt server has mirrored the repository, navigate to the mirror's location (most likely
Inside this directory, navigate further to
This directory will contain several directories with a title of a single letter. These directories will contain subdirectories of various Puppet-related packages. For example:
Sort through these directories and find the latest 2.7.x versions of each. Copy them to your
cron will pick them up and add them to your personal repository.
Now all servers will have access to the latest 2.7.x version of Puppet without having to specify a specific version. Whenever a new 2.7.x release comes out,
apt-mirror will retrieve it within 24 hours. Just copy the new version to
/var/lib/apt/repo/$repository/tmp/$distribution and it will then be added to your personal repository.
This article described several different ways of configuring
apt to help control how groups of servers access packages. Implementing these methods yields benefits of faster downloads as well as limiting or widening access to packages when needed. By using Puppet, you can easily configure a central server to host these roles or configure groups of servers to access the designated apt server.