Make All Environments Consistent
Consistent environments are important on software projects. They increase the team’s productivity, make it easier to diagnose, replicate and resolve bugs, make it easier to tune performance and scale the application, and reduce the cost of administration and maintenance. Building consistent environments is a great candidate for automation; in fact, tools like Chef and Puppet were written for exactly this purpose. But Chef and Puppet are complicated, heavyweight solutions. They also require a considerable amount of infrastructure themselves (git, Ruby, etc.)!
For lighter, simpler configuration needs I prefer Makefiles. Makefiles are great for this sort of thing because versions of Make exist for every operating system, and most UNIX systems come with some version already installed. I use Make to install my personal configuration settings for common utilities on each machine I use (see http://github.com/sl4mmy/dotfiles), for example.
Makefile syntax is familiar to system administrators and easy to learn, and because Makefiles are shell scripts (with a few format constraints) they map directly to the sequence of steps you would perform manually to setup your environments. But arguably the biggest advantage of Makefiles is they won’t do anything when the file about to be installed or modified exists. This is critical in shared production environments where services might already be installed and configured for other applications, and you don’t want to risk accidentally overwriting something important.
Consider a project dependent on Java and JRuby, and a convention that third-party applications should be installed under /opt/apps/<name>/<name>-<version>. The following Makefile installs and configures this project’s environment accordingly:
# Makefile
JAVA_VERSION=1.6.0_18
JRUBY_VERSION=1.4.0
# TARGET 0:
all: java jruby
# TARGET 1:
java: /opt/apps/java/java-${JAVA_VERSION}/
# TARGET 2:
jruby: /opt/apps/jruby/jruby-${JRUBY_VERSION}/
# TARGET 3:
/opt/apps/java/java-${JAVA_VERSION}/: /opt/apps/java/ /tmp/java-${JAVA_VERSION}.tar.gz
(cd /opt/apps/java; tar czf /tmp/java-${JAVA_VERSION}.tar.gz)
# TARGET 4:
/opt/apps/jruby/jruby-${JRUBY_VERSION}/: /opt/apps/jruby/ /tmp/jruby-${JRUBY_VERSION}.tar.gz
(cd /opt/apps/jruby; tar czf /tmp/jruby-${JRUBY_VERSION}.tar.gz)
# TARGET 5:
/opt/apps/java/: /opt/apps/
mkdir /opt/apps/java
# TARGET 6:
/opt/apps/jruby/: /opt/apps/
mkdir /opt/apps/jruby
# TARGET 7:
/tmp/java-${JAVA_VERSION}.tar.gz:
# Shell commands to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz
# TARGET 8:
/tmp/jruby-${JRUBY_VERSION}.tar.gz:
# Shell commands to download an archive of JRuby and save it as /tmp/jruby-${JRUBY_VERSION}.tar.gz
# TARGET 9:
/opt/apps/:
mkdir -p /opt/apps
Starting from the top:
# Makefile
JAVA_VERSION=1.6.0_18
JRUBY_VERSION=1.4.0
You should always name your Makefiles as Makefile because Make will look for that file in the current directory by default when it runs. You are free to name your Makefiles anything you wish, but if they are not named Makefile then Make will require additional configuration telling it which file to load.
Makefile comments start with #, just like shell scripts. The # Makefile comment in this example is not necessary, it is only for illustrative purposes.
The first two non-comment lines define variables that will be used throughout this Makefile. You may safely assume the syntax for defining and referencing variables in Makefiles is identical to shell scripting.
# TARGET 0:
all: java jruby
Target 0, named all here but could be named anything, is the default target because it is the first target declared in this Makefile. Running make in the directory containing this Makefile is equivalent to running make all.
Make is a file-centric build tool; it assumes the purpose of each target is to produce (or “make,” natch) a single file. By convention, the name of a target is the name of the output file produced by invoking that target. In our example, Make assumes that the all target will produce a file in the current directory named all. Make uses this convention to determine whether or not it actually needs to execute that target: if a file named all already exists in the current directory, Make won’t execute the all target.
Of course, targets are not required to actually produce a file with the same name as the target, but if they don’t Make will never think such targets are up-to-date and will execute them every time. In this case, we want Make to do exactly that; our goal is not to produce a file named all in the current directory, our goal is to set up a fully working environment on any machine by simply copying this Makefile and running make. We are using all as a convenience target so that we can make both java and jruby by default. If there were more dependencies, we could create separate targets for each of them and add them as additional prerequisites of all.
Our all target is declared with two prerequisites: java and jruby. Make executes all prerequisites before executing the target, so our java and jruby targets will execute before all.
# TARGET 1:
java: /opt/apps/java/java-${JAVA_VERSION}/
# TARGET 2:
jruby: /opt/apps/jruby/jruby-${JRUBY_VERSION}/
The java and jruby targets are declared in Target 1 and Target 2, and each have a single prerequisite of the Java and JRuby installation directories defined in our convention. Absolute paths are valid target names in Makefiles, and since they are used as prerequisites here, Make will search for targets with those names before it executes the java or jruby targets.
# TARGET 3:
/opt/apps/java/java-${JAVA_VERSION}/: /opt/apps/java/ /tmp/java-${JAVA_VERSION}.tar.gz
(cd /opt/apps/java; tar czf /tmp/java-${JAVA_VERSION}.tar.gz)
# TARGET 4:
/opt/apps/jruby/jruby-${JRUBY_VERSION}/: /opt/apps/jruby/ /tmp/jruby-${JRUBY_VERSION}.tar.gz
(cd /opt/apps/jruby; tar czf /tmp/jruby-${JRUBY_VERSION}.tar.gz)
The /opt/apps/java/java-${JAVA_VERSION}/ and /opt/apps/jruby/jruby-${JRUBY_VERSION}/ targets are declared in Target 3 and Target 4. They both have two prerequisites, and one step. Notice that the trailing slash in each target’s name makes it explicit to Make that these targets produce directories.
Steps are shell commands Make runs when it executes a target, and virtually any valid shell script is also a valid step. Steps are associated with the target immediately above them in the Makefile, and they must begin with a TAB (no spaces!). Multiple steps can be associated with a target and will run in order, but each line must begin with a TAB (no spaces!). Makefiles are very touchy about whitespace: every line beginning with a TAB below a target definition is a step for that target, and the first line that does not begin with a TAB is the separator between targets.
When Target 3 is executed, Make will spawn a temporary subshell, change the current working directory to /opt/apps/java and expand /tmp/java-${JAVA_VERSION}.tar.gz there. The parenthesis around the step definition are not required by Make, they are used here as in a regular shell script: the enclosed commands will run in a subshell, returning to the parent shell when finished. I did this so that changing directories in this step won’t affect the rest of the Makefile. Target 4 does the same for JRuby.
# TARGET 5:
/opt/apps/java/: /opt/apps/
mkdir /opt/apps/java
# TARGET 6:
/opt/apps/jruby/: /opt/apps/
mkdir /opt/apps/jruby
...
# TARGET 9:
/opt/apps/:
mkdir -p /opt/apps
Target 3 (/opt/apps/java/java-${JAVA_VERSION}/) depends on /opt/apps/java/, declared as Target 5, and Target 4 (/opt/apps/jruby/jruby-${JRUBY_VERSION}/) depends on /opt/apps/jruby/, declared as Target 6. Both targets depend on /opt/apps/, declared as Target 9. These three targets ensure that the correct directory structure exists according to our convention, creating missing directories as necessary.
# TARGET 7:
/tmp/java-${JAVA_VERSION}.tar.gz:
# Shell commands to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz
# TARGET 8:
/tmp/jruby-${JRUBY_VERSION}.tar.gz:
# Shell commands to download an archive of JRuby and save it as /tmp/jruby-${JRUBY_VERSION}.tar.gz
Targets 7 and 8 are placeholders showing how to download the necessary files before building an environment.
So, what happens when you run make on a clean environment without any of the directory structure or tarballs necessary for this project? Follow along by tracing the prerequisites in the Makefile.
Since all is the default target, Make starts by looking for a file named all in the current directory; unable to find that file, Make tries to execute the prerequisites of all, java and jruby. Since no file named java exists in the current directory either, Make looks up the target named java and then tries to execute its prerequisite, /opt/apps/java/java-${JAVA_VERSION}/. Since that directory does not exist, Make looks up the target named /opt/apps/java/java-${JAVA_VERSION}/ and tries to execute its prerequisites, /opt/apps/java/ and /tmp/java-${JAVA_VERSION}.tar.gz. Since the directory /opt/apps/java/ does not exist, Make looks up the target named /opt/apps/java/ and tries to execute its prerequisite, /opt/apps/. Since that directory does not exist, Make looks up the target named /opt/apps/, sees that it has no prerequisites, and executes the steps associated with that target (mkdir -p /opt/apps) which produces the directory /opt/apps/.
Make then backtracks to the /opt/apps/java/ target, determines that its prerequisites were previously executed, and executes its steps (mkdir /opt/apps/java) which produces the directory /opt/apps/java/. Make backtracks again to the /opt/apps/java/java-${JAVA_VERSION}/ target, sees that it still has one unsatisfied prerequisite, and looks up the target named /tmp/java-${JAVA_VERSION}.tar.gz. Make sees that /tmp/java-${JAVA_VERSION}.tar.gz has no prerequisites, and executes its steps to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz. Make backtracks to the /opt/apps/java/java-${JAVA_VERSION}/ target again, sees that all of its prerequisites are now satisfied, and executes its steps which produce the directory /opt/apps/java/java-${JAVA_VERSION}/.
Make then backtracks all the way back to the all target, and does the same for its other prerequisite, jruby.
Now, consider what happens when you re-run make in the directory containing this Makefile (or run it for the first time in an environment which was previously setup manually). Make sees that all is the default target, but cannot find a file named all in the current directory, so it tries to execute its prerequisites, java and jruby. There is no file named java in the current directory, either, but the java target’s only prerequisite is /opt/apps/java/java-${JAVA_VERSION}/, and that directory does exist, so Make skips the prerequisite’s steps and goes straight to executing the steps associated with the java target. Since there are none, nothing happens, and Make backtracks to the all target and tries to execute its other prerequisite, jruby. Similarly, there is no file named jruby, but the directory corresponding to the jruby target’s only prerequisite does exist, so Make skips executing the prerequisite target, going straight to the steps associated with jruby. Again, there are none so nothing happens, Make backtracks back to the all target, determines its prerequisites were previously executed and so tries to execute the steps associated with all. Since there are no steps associated with all, nothing happens, and Make finishes successfully without having done anything. Perfect for production environments!
What happens if you delete /tmp/java-${JAVA_VERSION}.tar.gz and re-run make in the directory containing the Makefile? Well, nothing. Make never sees that file is missing since the directory /opt/apps/java-${JAVA_VERSION} exists. But if you delete the directory /opt/apps/java-${JAVA_VERSION}/ and re-run make, then /tmp/java-${JAVA_VERSION}.tar.gz will be downloaded again and /opt/apps/java-${JAVA_VERSION}/ will be re-created. The point is: you must carefully ensure all of your application’s dependencies are properly exposed as prerequisites in your Makefile, not hidden as nested prerequisites of prerequisites.
I love Make as a low-touch solution for setting up consistent environments! It’s simple to learn and easy to use, and it is low-risk for shared production environments because it reduces the likelihood of modifying or deleting files used by other applications. And when you outgrow Make’s capabilities and need a more powerful tool, Makefiles are the perfect way to consistently install and configure Chef or Puppet across multiple machines.