At the day job I’ve been tasked with setting up an automated build system. This is a Good Thing™ and I actually volunteered to do this task. I’m interested in doing what can help us make our code better, deliver our products faster, and make our lives easier in terms of automating things we don’t necessarily need to be involved in. So a Mac Mini was obtained to be a build machine, and I’ve been working to set things up.

I should note that I’m not a dedicated buildmeister. I know at larger companies they can have a person (or persons) whose full-time job is dealing with builds, but I’ve never been one of those people. I’ve done some build hacking in the past, but it was always homebrewed scripts and systems. This time around, let’s use an established system instead of homebrew.

It seems the modern hotness is Jenkins. Yes there are other options, but all signs via Google searching point to it. As well, it seemed (note the verb tense) like the more Mac-friendly solution. If nothing else, it had a nice Mac OS X installer.

I’m still far from having our complete build system, and I reckon as I learn more things on this list will change. But I’ve already learned a few useful things and I felt like sharing and adding to the greater knowledge-base out there that, through Google-Fu, helped me get this far. Maybe with my additions, someone else can be helped and maybe with a little less frustration and time sink.

The Project

A little context about the project being built.

Working on Mac OS X, developing iOS apps. Thus we’re using Mac OS X 10.7 Lion as our dev runtime environment. We’re using Xcode 4.3.3. Jenkins is version 1.474. We use git and a mix of private and public repositories on github.com.

First Time Problems

Started with a brand new, fresh from the box Mac Mini. Of course, before attempting any of this CI-specific work, the box was brought up to date with OS updates, Mac App Store updates, and so on. Note! Jenkins is a Java app and Java is NOT installed by default. So after you run the Jenkins installer and it tries to start Jenkins, things will probably fail. The OS will prompt you to install Java, so you’ll have to do that, but then Jenkins should end up running. Not a big hurdle, but it’s there.

Make sure to launch Xcode.app (the GUI app) and get it properly happy. This is mostly ensuring the various downloadable components get downloaded and installed, like command line tools and such.

You will be using command line tools, thus you will have to run xcode-select. But being as this is Xcode 4, things are different.

$ sudo /usr/bin/xcode-select -switch /Applications/Xcode.app/Contents/Developer

This will ensure xcodebuild can be found. And note, the first time you run xcodebuild, it won’t run! While you agreed to the license agreement when you first launched the GUI Xcode.app, that’s not good enough for xcodebuild; you have to agree to it all over again via the command line. Just be aware. It’s not a big deal, it’s a first-time-only occurrence, but it’s there to deal with.

The User (login) Matters

This was the source of much fun and frustration for me.

The first time I installed Jenkins I had no idea what to expect. That it had a proper Mac OS X Installer .pkg was cool, but it was also hiding a secret. Any time there’s a “Customize” button I like to click it and see what options there are. I noticed it provided two options:

• Start at boot as “daemon”
• Start at boot as “jenkins”

I had no idea what the relevance of these two options were. The default was “daemon” and “jenkins” was unchecked. I just figured to trust the default installer settings. Ha ha ha…  This actually caused me the most trouble and pain. I won’t recount the many hours spent dealing with this, but I will explain the issues.

I opted to go with the installer’s default of “daemon” but that creates a problem because “daemon” is a special user. When Jenkins needs to do things, it’s going to look in non-typical places, like /var/root for ssh keys or the like. Basically, it’s going to cause you a lot of headache.

When you search around for information about this, everyone starts to talk about using the “dscl” command line tool to create a hidden secret “jenkins” user and run Jenkins that way. This makes sense because it creates a user of restricted ability so it helps to keep the system secure and minimize chance of damage should someone gain access to the system via the Jenkins system or login. But in practice, this turned out to be a big pain in the ass because of what we’re doing. I’m sure there are some projects where this won’t be that problematic. But writing iOS apps brings issues. Apple makes great products, but you find they are great as long as you color within the lines; the lines might be really wide and vast, but still you must color within them. Trying to deal with this secret “jenkins” user created various issues.

For Xcode (or xcodebuild) to do code signing, it needs access to private keys and certificates and other such things. Thus the Keychain is involved, but this hidden “jenkins” user doesn’t have one. Again, more searching will turn up possible solutions, but they are not ideal solutions. In fact, one solution of putting these into the “system” keychain really defeats the purpose doesn’t it? Then there are the .mobileprovision files needed during the signing process, and those must exist somewhere in the “jenkins” user structure. In my case, the provisioning files may be updated fairly often due to the addition of device UDID’s, and setting up the initial .mobileprovision files by downloading to the logged in user then copying it all into the hidden jenkins user locations… it was just turning into a massive pain in the ass. And you’ll have to do this at least once a year, when you renew your iOS account with Apple.

So I fumbled with various permutations of these user setups and it was all just frustrating to me.

In the end, I backed everything out and started over (for the umpteenth time). I created a “jenkins” user via the System Preferences. That is a full-on login-able user named “jenkins”, admin user. I then logged in as the “jenkins” user. I ran the Jenkins installer, customize, and selected the “start at boot as ‘jenkins’” option (and deselected the “start at boot as ‘daemon’” option). I ran the installer. That did not immediately succeed, but the resolution was simple. Stop the daemon (using launchctl). Edit the .plist for the Jenkins launchd job and changed the location of JENKINS_HOME to /Users/Jenkins. Then restarted the machine. Life was good.

I think today I’m going to try doing it “yet again” tho… deleting the Jenkins user and creating it again, then installing again, and this time making a subfolder, like /Users/Jenkins/Jenkins_CI or something like that. That’ll contain all the stuff in one folder and not litter the actual home directory.

Is this the right thing to do? Well, from a pure security standpoint, no. It’s now an admin user that can log in and have all sorts of fun. But my feeling is this machine will be behind our firewall. If this box gets compromised, we have bigger problems anyways.  Could it mean bigger problems for me down the line with Jenkins-CI itself? Maybe. We’ll see. I’m not necessarily advocating taking this approach, but it’s just the one I’ve presently taken. It appears that it will create less long-term hell for me, especially in dealing with Xcode and the OS.

Authentication

Another source of pain was dealing with authentication issues.

Since we use private repositories on github, we need to authenticate. This proved to be more difficult than it should have been.

When I first tried a basic build of our project, I installed the various git plugins within Jenkins, created the new job, set it to obtain the source via the http protocol, and off we go. Well, that failed. The job just hung forever, I reckon waiting for authentication. So how can we authenticate? I saw no configuration options for it. So I started playing with ssh.

ssh would actually be a good thing in many ways, but with all the aforementioned user/login issues, it was becoming a massive pain. The way Mac OS X seems to handle ssh is by launching the ssh-agent on demand. Well… it can’t do that for the secret jenkins user. I tried all sorts of things, but I just could NOT get it working. *sigh*   Another vote for making a fully-realized jenkins users instead of this secret hacked one.

The one thing I had to then do was edit my ssh keys to not have a passphrase, and that would work (no need for ssh-agent, no need for interactive issues). But lacking a passphrase ain’t so hot. But it seemed to work… well, at first.

The next problem? Our git project has submodules. In the .gitmodules list, the “url” for each submodule used http as the protocol. And so, when the Jenkins-CI git plugin tried to obtain the submodules, we were back to the original problem. FML.

I tried fiddling with the repository URL as specified in the “git” section of the Jenkins-CI job to be something like:

https://:@github.com/path/to/repo.git

and while that worked for the top-level repository, the submodules still tripped me up. Plus, I did not like the idea of storing a password in plain text and sight like that.

I was able to solve it tho… through the magic of the .netrc file.

I created the following file in /Users/jenkins/.netrc

machine github.com
login <github username>
password <github password>

Saved it. Did a “chmod 600″ on it. And lo…. everything worked. Huzzah!

I think the git plugin should have a way of doing authentication, but I’m still too much of a Jenkins n00b to know really where the fault lies.

More Ahead

At this point, I’m able to clone our git repository and build the app. It’s very basic at this point, with much configuration and fiddling yet to come. But I spent a good deal of time on these matters this past week and tried to sift through a bunch of search engine results to wind up here. So much of the information out there is Linux or Windows based, which is fine for the general but some of the little details aren’t there. And I can’t imagine we’re the first people to need to build in such a way, but I just couldn’t find anything specific to the particulars of our setup. So here’s hoping this helps contribute back.

As well, if you the reader have any information, insight, comment, etc. please share. I’m still learning, and constructive contribution is appreciated.