tl,dr

You should use configuration management tools to deploy your application automatically. Vagrant, chef, puppet and littlechef are extraordenary tools!

Intro

You are a new developer in a project which has been on hold for a couple of month now. You'd like to set it up, but nothing is working. You can't even get the tests to pass.

Wouldn't it be nice if you could just automatically setup a VM or any other machine for that matter? Ideally, you could event setup your work environment according to the projects requirements.

This is where Configuration Management tools like puppet and chef come into play. Github also recently released boxen, which can be used to keep developers in sync and track project dependencies. And there is vagrant, a tool which sets up a VM with your chef & puppet recipies.

This post is about how to setup your project so that it can be automatically deployed via vagrant, chef or puppet. I am asuming a Ruby on Rails project, but the method described here should work with any kind of project. We will start out by deploying our rails application to a VM with vagrant. After that, we will deploy it to a remote EC2 instance.

Deployment with Vagrant

Directory Structure

I usually group all deployment and infrastructure specific files in a directory called deploy/. Here is what's in it:

$ ll
-rw-r--r--   1 robin  staff   673 Mar 14 16:43 Cheffile
-rw-r--r--   1 robin  staff  1166 Mar 14 16:43 Cheffile.lock
-rw-r--r--   1 robin  staff  4042 Mar 14 16:08 Vagrantfile
drwxr-xr-x  13 robin  staff   442 Mar 14 17:16 cookbooks/
drwxr-xr-x   3 robin  staff   102 Mar 15 18:46 data_bags/
drwxr-xr-x   3 robin  staff   102 Mar 14 13:13 tmp/
  • Cheffile: A librarian file and specifies which cookbooks and versions to use.
  • Cheffile.lock: Once you run librarian-chef install, this file is created. As with the Gemfile.lock in Ruby/Bundler, this file keeps track of which versions are installed. This is useful when you did not specify the very concrete version of the cookbook you want to use. Then the used version is kept in this file and when another developer runs librarian-chef install, he will install the very same version, even if in the meantime a new version did come out.
  • Vagrantfile: This file is created by vagrant init. It contains the configuration of your virtual machine. We'll take a look at it later.
  • cookbooks/: This directory is filled with the cookbooks specified in the Cheffile by running librarian-chef install.
  • data_bags/: This directory contains chef databags. We need this directory because the rails_app cookbook requires configuration of the apps via databags. More later.
  • tmp/: As usually, for temporary files.

Cheffile

So here is what the Cheffile looks like for a simple rails appplication:

site 'http://community.opscode.com/api/v1'

cookbook 'rvm', :git => 'https://github.com/fnichol/chef-rvm', :ref => 'master'
cookbook 'rails_app', :path => '/Users/robin/Code/arc/rails_app'
cookbook 'postfix', :git => 'https://github.com/cookbooks/postfix.git'
cookbook 'postgresql', :git => 'https://github.com/phlipper/chef-postgresql', :ref => 'ff499fcff50c1113f74cc32352ebfbede93a06cc'
cookbook 'nginx', :git => 'https://github.com/cookbooks/nginx.git'

We use my rails_app cookbook to deploy our application. Where I use a relative path, you'd have to use the git url.

Run librarian-chef install will install the cookbooks with the revisions specified in the Cheffile into the cookboos/ directory. But beware, the cookbooks directory is overwritten every time you runy librarian!

After you've run librarian-chef install once, the Cheffile.lock is created.

Vagrantfile

The Vagrantfile is a file created when running vagrant init. It defines, how your virtual machine should be setup. Let's have a look:

Vagrant::Config.run do |config|
  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "myUbuntu1204chef"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  config.vm.box_url = "http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-vagrant-amd64-disk1.box"

  # Forward a port from the guest to the host, which allows for outside
  # computers to access the VM, whereas host only networking does not.
  config.vm.forward_port 80, 8080

  config.vm.provision :chef_solo do |chef|
     chef.cookbooks_path = "./cookbooks"
     chef.data_bags_path = "./data_bags"
     chef.add_recipe "rails_app"
     chef.json = { :rails_app => {bag_items: 'app'} }
  end
end

This is all the configuration you need. However, the original file created by vagrant init is far more verbose. So when in doubt, check it (the vagrant docs) out.

The rails_app cookbook requires configuration of the applications that should be deployed via databags. You can configure which items to use with the rails_app attribute. In the example above, we specify that the file data_bags/rails_apps/vagrant.json should be used.[^1]

[^1]: So why do we have in { :rails_app => {bag_items: 'vagrant'} } rails_app as singular but the data_bags/rails_apps directory is plural? Because :rails_app => basically says: here follows the configuration for the rails_app cookbook. Whereas in data_bags/rails_apps multiple app configurations can live. This which databag and configuration option is used is specified by the cookbook.

data_bags/rails_apps/app.json

{
    "id": "app",
    "repository": "git@bitbucket.org:rweng/testapp.git",
    "precompile": true,
    "seed": true,
    "user": "app",
    "deploy_to": "/app",
    "rails_env": "production",
    "databases": {
        "production": {
            "adapter": "postgresql",
            "username": "vagrant",
            "password": "vagrant",
            "database": "app_prod"
        }
    },
    "deploy_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEoQIBAAKCAQEAq...
    Gt2ef\nLugomnGLa3AsizfMYQwyqmn4WWpl02fxaCYo3Giw/st23rVJEg==\n-----END RSA PRIVATE KEY-----",
    "authorized_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqbeyjdt4t5...
    4eKsD1JCHQ== rweng@MacBookProRobin.local"
}

Now that we have all files, let's start the VM.

vagrant up

Running vagrant up should start the VM and deploy your application. You can ssh into the VM with vagrant ssh. In case errors occur (which is, to be frank, quite often), you can update or cookbooks or configuration and run the following command to rerun chef-solo:

$ sudo chef-solo -j /tmp/vagrant-chef-1/dna.json -c /tmp/vagrant-chef-1/solo.rb

If everything worked fine, you application should be available at http://localhost.

Deploy to EC2

For deployment to remote machines, I like to use a tool called Littlechef. It basically wraps around chef-solo and makes it easy to transfer you cookbooks to the remote location and run them.

Go ahead and install Littlechef as descibed on their Readme. Now, littlechef need a couple more directories and files to function:

$ mkdir nodes roles site-cookbooks
$ echo "[userinfo]
ssh-config = ~/.ssh/config" > auth.cfg

In the auth.cfg file you specify how to login to your remote machines. I like to let this kind of configuration live in ~/.ssh/config which then looks like that:

Host myhost
HostName myhost.com
User root

I usually give myself passwordless access with:

$ cat ~/.ssh/id_rsa.pub | ssh user@node.network "cat >> .ssh/authorized_keys"

Now when everything is setup, you can let littlechef install chef on the remote machine:

$ fix node:myhost deploy_chef:gems=yes

If this ran successfully, it should have created the file nodes/myhost.json. If it didn't, run the rails_app recipe. Without configuration, it will not do anything:

$ fix node:myhost recipe:rails_app

Now when the file nodes/myhost.json exists, append the following configuration. In the end, the file should look like this

{
    "ipaddress": "....",
    "name": "myhost",
    "run_list": [
        "recipe[rails_app]"
    ],
    "rails_app": {
        "bag_items": "app"
    }
}

Now when you run fix node:myhost it should automatically deploy your application.

Ending Notes / Automating Dev Environment

The existing tools are great for automating deployment. However, they are not ideal for autmating development environment. There are a couple of problems:

  • If you let the code in the VM, how to connect your IDE to it?
  • How to synchronize the VM code with your IDE without overwriting stuff (through shared folders)?
  • Does your application need extra development dependencies?

One way to tackle these problems could be to let your IDE run inside the VM. Then have three recipes: common, development, production. However all solutions rise new problems, e.g. how to synchronize custom settings if your IDE.

If we take a step back, we need to decide if it is necessary to run the app in the VM, or if we could just setup our machine with the newest version of everything. On the one hand, if we continue development we can also also upgrade the dependencies. On the other hand, if the application was on hold for some time you might not even get it to start with newer dependencies.

Letting your IDE run in the VM is no option for me. Right now, it seems as if we have to live with the idea that we have to setup our development environment manually.

Troubleshooting

The default ubuntu 12.04 box has no chef installed by default. So if you run accross the following error, thy sshing into the VM and install chef manually:

[default] Running provisioner: Vagrant::Provisioners::ChefSolo...
The chef binary (either `chef-solo` or `chef-client`) was not found on
the VM and is required for chef provisioning. Please verify that chef
is installed and that the binary is available on the PATH.
FAILED WITH CODE: 231
$ vagrant ssh
$ sudo gem install chef --no-rdoc --no-ri

I like to package the VM after that so I don't have to do this again. How this is done is decribed here.