The beauty of automated builds

Just about any developer knows that if you’ve got even a moderately complicated project, you have to have automated builds. This helps to ensure not only that the builds you provide to the public are consistent, but also that you can regenerate a past build from any commit in your tree. Especially when dealing with binary-only platforms such as MacOS and Windows, automated builds are also the ticket to getting new code out to users frequently.

I’ve always had automated build scripts for my major projects (CHIRP and D-RATS), mostly because of the amount of work involved in building a complex PyGTK project on Windows. The scripts would copy the code up to a VM running Windows via scp, and then ssh into a cygwin environment to actually generate the build, create a .zip distribution and then run the scriptable NSIS installer builder. I would do this every time I needed to publish a build to my users, which was usually every couple of months.

Lately, I’ve moved all of that to a Jenkins system, which automatically generates builds for both projects on all three platforms every night where there was a change. It publishes these to an externally-visible server where users can fetch them with a web browser. In addition, it runs the automated tests and generates a model support matrix, pushes those out to the server as well, and then emails the users mailing list to let them know that a build is available.

This has been really beneficial for getting changes tested, because anytime I fix a bug, the reporting user needs only wait until the following day to fetch the next daily build, test the fix, and report back to me. It’s a thing of beauty and it actually saves me a lot of time.

The one thing I had to figure out in all of this, however, was how to make it easy to generate builds rapidly during development. On Windows, I have a drive letter mapped to my main development directory, and thus I can go run python manually from the command line against my working tree. However, there are plenty of issues which crop up only in the frozen environment that py2exe creates and pushing small changes to the external repository just for testing is not feasible nor desirable. Thus, I needed a way to tell Jenkins to build what I’m working on right now on a given platform. Since Jenkins is really designed around the principle of building from a repository, I could have just copied each of the jobs and made them pull from a temporary repository that I junk up with small changes. However, that means I’ve got two copies of the complicated build job to maintain, plus I have to commit or refresh and push each time I need to test. That’s ugly.

What I ended up doing was writing a small script to use Jenkins’ CLI tool and I’m quite happy with the result. I have ssh access to the build machine, so I have the script generate a diff of my current tree against what’s currently pushed to the public repository. It then copies that patch file up to the build machine into /tmp. Next, the script requests a build of the correct job using the CLI tool and passes the path to the temporary patch file. The job is configured to take an optional PATCH parameter, and if present, it applies it to the working directory before building. With this, as I’m working, I can just run something like this:

$ ./ win32
Executing chirp-win32 build...SUCCESS

The amount of legwork happening in the background truly makes this a major convenience. My script looks like this:

proj=$(basename `pwd`)

do_cli() {
    java -jar jenkins-cli.jar -s $url $*

if [ -z "$arch" ]; then
    echo "Specify arch (sdist, macos, win32)"
    exit 1;

hg qdiff > build.patch
scp -q build.patch

echo -n "Executing $proj-$arch build..."
do_cli build $proj-${arch}    \
    -p PATCH=/tmp/build.patch \
    -p BUILD=test -s > .build_status
if [ $? -eq 0 ]; then
    echo "SUCCESS"
    echo "FAILED"

if [ "$status" = "Succeeded" ]; then
    bno=$(cat .build_status | cut -d ' ' -f 3 | sed 's/#//')
    do_cli set-build-display-name ${proj}-${arch} $bno DevTest

notify-send "Build $status" "Build of $arch $status"

Notice the call to notify-send at the end. When the build is done (the win32 job takes several minutes), I get a nice desktop notification, which means I can switch away to something else while waiting for the build to complete.

Category(s): Codemonkeying
Tags: , , , , ,

Comments are closed.