Dr Nic

Trialing RedCar instead of TextMate – replacing my aliases

I’m interested to trial RedCar in my life, instead of TextMate. So I need RedCar
to appear on my screen whenever I think “give me an editor”.

I’ve had m bound to TextMate for years. I also have it set to my $EDITOR variable, so it is launched by git commands, etc.

To change the default to RedCar, and to allow me to toggle between RedCar and TextMate, AND to allow me to use Edge RedCar (from source instead from a RubyGem).

This file comes from my editor.sh file which is loaded into all shell terminals.

I can change to TextMate with:

use_textmate

Back to RedCar (via gem) or RedCar (from source):

use_redcar_gem
use_redcar_dev

Making CI easier to do than not to with Hudson CI and Vagrant

It irked me a little that I could develop on one stack (OS X, Rubinius, Sqlite3), run continuous integration (CI) on another stack (Ubuntu, Ruby 1.8.7, Postgresql), and deploy into another stack (Gentoo, Ruby 1.9.2, MySQL). I think what irks and worries me is that there are three sets of differences to be aware of. A bug in production? Was it a missing test scenario or one of the many differences between production and CI environments?

So I think I have two solutions.

First, use a VM that matches the production environment. Each different production environment would mean another VM. If you are managing your own production environment, then all you need is the tools (described in this article) to recreate your production environment in a VM.

Second, use a clone of your production environment. That is, if you deploy to Engine Yard AppCloud then run CI in Engine Yard AppCloud; if you deploy to a single Ubuntu instance on Slicehost, the have another matching Ubuntu instance on Slicehost.

I’ll write about the first solution – using VMs – here, and I’ll write on the Engine Yard Blog about the solution for Engine Yard AppCloud customers. For AppCloud users life will be even easier because there are zero setup steps to ensure you have consistent environments. It’s been one of my favorite projects in the two months since I arrived at Engine Yard.

I’ll also introduce a CLI for talking to Hudson – one that assumes you are working on Ruby/Rails projects – and makes it really easy to get up and running with a server, your Rails/Ruby projects, and any VMs you need (optional for Hudson, but they are the point of my line of thinking).

And I’ll introduce Vagrant, a CLI for creating/managing/destroying VMs.

Oh, and I’ll introduce Hudson CI.

Ok, let’s fix all our CI problems in one go…

Hudson CI

I needed a CI tool to allow tests to run inside VMs or on remote servers/VMs. Back in May I found and fell in love with Hudson CI. Fortunately it had support for “slaves”, and a way for CI jobs (your applications or rubygems) to select which slaves can be used. Hudson is also great because it is easy to try out (one click to install and launch), easy to configure, and has 350+ plugins.

What is Hudson?

CLI for Hudson

Charles Lowell’s hudson.rb project is a CLI for launching Hudson (it’s bundled inside the gem), and a set of CLI tasks to add/remove projects (jobs), add slaves, trigger builds and more.

gem install hudson

You can launch Hudson via:

hudson server  # Default: --port 3001
open http://localhost:3001

It currently assumes you are working on Ruby projects, using bundler, and attempts to create a useful set of default steps for your CI jobs.

cd /some/rails3/app
hudson create . --host localhost --port 3001 --template rails3

Hudson CI will automatically start running steps to install your application’s gems, database, and run your tests. When it does it on your local machine it’s not that impressive. When it happens on a fresh VM or slave node and you didn’t have to do anything to set it up, it’s awesome.

V is for Vagrant, VirtualBox and all things VM

If you can script it, then you can automate it. Fortunately, as a virtual machine VirtualBox is both scriptable and FREE! Secondly, if you have a VM, you’ll need to script its setup/provisioning; so fortunately there exists chef and puppet, amongst others.

Thirdly, if you are really really really lazy, like me, you will want Vagrant. Created by Mitchell Hashimoto and John Bender, Vagrant is a tool for building and distributing virtualized development environments. Everyone in your team have their own OS/configuration? Got Windows users on your team? Use Vagrant.

Once you have VirtualBox installed, getting any of your projects live within a VM is trival:

gem install vagrant
vagrant box add base http://files.vagrantup.com/lucid32.box
vagrant init
vagrant up
vagrant ssh
$ cd /vagrant/
$ ls -al

You will now see the contents of your project folder, but from within the VM instance! They are linked – a change to either is automatically reflected inside and outside of the VM.

In this example, the Guest OS being downloaded and installed into VirtualBox (note: without any GUI) is Ubuntu Lucid, but you could use any VirtualBox packaged unix system.

For wonderful guided tour of Vagrant see the Getting Started video.

Vagrant – Getting Started from Mitchell Hashimoto on Vimeo.

Hudson CI + Vagrant = Perfect CI

Whether you choose to use VMs for a production-like development environment (a good idea for anyone, a wonderful idea for Windows developers who are very far removed from their production experience), it is a very good idea to have a CI environment as similar to your production environment as possible. Here, we want a VM instance with all the components/utilities/rubies set up that you have in production.

I have created an example Rails application that is setup to use Vagrant for a CI slave VM.

See the README for complete instructions. See the Vagrantfile and the cookbooks folder for the configuration and provisioning recipes.

Once you have the VM instantiated, you add it to your Hudson CI master (either the localhost one above or your remotely hosted server) with the CLI:

hudson add_node localhost --name "VM" --label railsapp-vagrant ...

(See the example Rails application for the other flags I used to get this working). It is now available to all Hudson jobs; but has a label “railsapp-vagrant” to allow jobs (Hudson’s name for a project) to specify that slave node specifically.

To add the Rails application to Hudson CI, and force it to run the tests in this VM:

hudson create . --template rails3 --assigned-node railsapp-vagrant

If you visit the Hudson master (at http://localhost:3010 in the example) you will see the job automatically running (“building”) within your VM. It will use bundler to install the gems, and run all the tests.

That’s it?

I know, CI is historically a pain in the arse. When CruiseControl was the only CI kid-on-the-block, it was standard for people to respond “4 days” to the question “How long does it take to get set up?”

It can now be really easy.

More importantly than being easy, you are running an application’s tests within an isolated VM that you can design to match your production environment.

I think there can be a good future for CI and Rails applications. Thoughts on this solution?

Thanks

Some of the technology in this article is old, other bits are very new, but for all of it I thank all the creators and contributors. Hudson CI was created by Kohsuke Kawaguchi during his days at Sun. He is now offering Hudson support services via his company InfraDNA.

The Hudson.rb project was created by Charles Lowell to bundle the Hudson CI and some common useful plugins for Ruby/Rails projects. He came all the way to Gothenburg, Sweden for NordicRuby conference (great conference by the way!) I fell in love with Hudson and started helping Charles on Hudson.rb.

Thanks also to both Kohsuke and Charles for starting work on a JRuby plugin for Hudson CI, and writing up a progress report.

Thanks to everyone who agrees that Hudson CI is awesome.

Thanks to Mitchell Hashimoto and John Bender for creating Vagrant. It is an incredible tool for developing within a VM (on OS X or Windows).

Finally, thanks to Bo Jeanes who helped on the Engine Yard AppCloud version of this project. Coming soon!

RubyConf & RubyBayou

If you’re in New Orleans this week for RubyConf, I’ll be at the local Ruby group RubyBayou talking about Hudson CI, Vagrant, and AppCloud on Thursday night. There’s a happy hour from 5pm till the 7pm start.

Location: LaunchPad NOLA, 643 Magazine St, New Orleans, LA 70130

Come and let me convince you about the wonders of Hudson CI and having a CI test environment that matches your production env. You know you want to be convinced. It’s good for you, like fruit.

Don’t forget Chuck Norris

Always remember to install the Chuck Norris plugin for Hudson, and enable it for each job. Don’t forget Chuck Norris.

Coming to America

Have you ever looked at your biological watch and thought, “it looks like it’s time to change the world?”

My biological time piece has a full set of inscriptions: go to university, chase girls, get a post-graduate qualification, catch a girl, get a professional job, marry the girl, change into contracting, change into consulting (and ponder what the difference is), work overseas (and marvel at the differences), make a baby, change into training, move back to Australia, buy a home, start a consultancy (Mocra), make another baby, grow the consultancy, and …

Really, I’m not sure what comes next in the script of normal life. Retire in 30 years? Create more little open source projects? Just keep growing the consultancy? Until what?

A few weeks ago, John Dillon, the CEO of Engine Yard, drew some pictures on a whiteboard for me and asked, “Do you want to help change the world?” Sure, a classic Steve Jobs one-liner from the history books of Apple. I said, “Yes”.

On the 1st of September I will start work at 500 3rd Street, San Francisco as Engine Yard
’s VP, Technology. Job description: help web application developers win.

22000km to work

The daily commute

Google Maps suggests my daily commute to work will be 22,300km. Some of that will be heavy morning traffic, so at say 30km/hr, that is almost 750 hours. Each way.

So instead, my family will pack its bags and move overseas again. Though for the first time, we’re coming to America!

How to move your family to America

It hasn’t been all smooth sailing on the home front since I broached the idea with my wife. Personally, have you ever turned to your wife and said, “Honey, I think we are going to move to America!”

Try it at home for a laugh. Perhaps your wife will say something witty or clever like mine. Something like, “No.”

“No, baby, really. I have something important to do for a few years. It’s in America. San Francisco. Apparently it’s lovely!”

“Is the weather nice?” she might ask looking for any reason to want to leave beautiful, sunny Brisbane.

“No. I think it’s kind of cold there. And foggy. There was even a taxi with ‘Fog City Taxi’ on its side. But there are wonderful schools and the people are fabulous and are from all over the world!”

“So our young kids won’t get an American accent?” she might ask pleadingly.

“Well, they won’t get all of the accents, no.”

“…”

“I love you!”

And so eventually your wife will say yes.

How do we get to win as web developers?

Choose to use Rails. Improve Rails. Improve the ecosystem above/below/around Rails. Improve our daily lives. Expand the sweet spot of Rails to solve problems.

In the future, can we look back at the work we did, the fun we had, the problems we solved, the time we spent working, and feel that it was worth the effort? Will we remember when we first discovered Ruby and Rails and remember the joy?

This is a world-changing set of issues to care about. I’ve always cared about them before now, but never had resources or reach to work on them beyond my own little projects or cunningly worded, suggestive emails/tweets/chat with other open source developers.

Though from now onwards I will get to work with one of the greatest concentrations of Rails talent and one of the largest open source programs in the our community. I’m more than just a little bit excited. Imagine working with full-time contributors to Rails, JRuby and Rubinius as well as the dozens of developers, devops and more.

As Engine Yard continues to grow and win then its capacity to fund and resource more contributions will grow. And then we all win faster.

Can you imagine what else should be funded, promoted or prioritised? I have a dozen solid ideas but I would love suggestions.

Mocra

I will miss my guys at Mocra, a wonderful Rails consultancy with clients around the world whilst based out of Brisbane, Australia. I founded and ran Mocra for the last two wonderful years. It will be slightly annoying – I won’t get to take any credit for all the success the Mocra team will have in the years to come.

Definitely, if you need an awesome Rails team for your project, contact Mocra. If you need a personal introduction, let me know. I know some people there.

San Francisco

I am very excited to move to SF and to hang out on a regular basis all the people I have only been able to see at conferences.

I’d also love suggestions of where a family of four (including a 4yo and 2yo; oh, plus another one due in February) could live. I’ve briefly seen parts of SF and many of the surrounding towns in the bay area. Thanks to Randall, Tammer and Marcy for their turns as tour guides during my visit a few weeks ago.

Packaging TextMate bundles in OS X DMGs

Last week Engine Yard released a CLI for their Engine Yard AppCloud. Delights such as:

ey deploy
ey rebuild
ey logs
ey ssh

Engine Yard.tmbundle

They simultaneously released a TextMate bundle to deploy, rebuild, view logs, etc using Ctrl+Alt+Cmd+E. Like all TextMate bundles, you can install it in one of two ways: via git (see the README), or via a beautiful DMG. Download it here!

Yes indeed, TextMate bundles can now be packaged up and distributed via DMGs using ChocTop!

Ruby on Rails.tmbundle

For example, the Ruby on Rails.tmbundle using a simple purple theme.

First, the Engine Yard tmbundle

To use the Engine Yard tmbundle, you first need to install and use the CLI once. Instructions at the bottom of the information page.

How to package a TextMate bundle into a DMG

ChocTop is a packaging and distribution tool originally designed only for Cocoa applications, but can now package any assets, URL links, or even the whole project folder itself. This makes it ideal for packaging TextMate bundles which have no compiled/built output to distribute (like a Cocoa application), rather the project folder itself is the distributed item (the Engine Yard.tmbundle folder in this case).

Getting started

Everything is added into your TextMate bundle project. For example, with the EngineYard bundle:

gem install choctop
cd Library/Application\ Support/TextMate/Bundles/Engine\ Yard.tmbundle
install_choctop . --tmbundle

If your tmbundle already has a Rakefile, then don’t overwrite it. Instead, inside the Rakefile, add the ChocTop configuration:

require "choctop"

ChocTop::Configuration.new do |s|
  s.add_root :position => [290, 200], :exclude => %w[appcast build .bundle .git]
  s.add_link 'http://github.com/engineyard/engineyard.tmbundle', 'GitHub', :position => [520, 200]
  s.defaults :textmate
end

For TextMate bundles the DMG magic is from the s.add_root line. The resulting DMG will include the entire project as a folder/bundle. For example, you’ll want to exclude appcast, build, .bundle (if you’re using Bundler), and .git folders.

The s.defaults :textmate provides a generic background and volume icon for a TextMate bundle DMG. See below for customising the background and volume icons. The :position coordinates above are for the generic background.

Building your DMG

To build your DMG and then view it in Finder:

rake dmg
open appcast/build/*.dmg
# or together
rake dmg[automount]

You can now share the DMG file. See below for how to upload it to a server.

Versioning

In future, it would be great to use Sparkle’s auto-update mechanism (as seen in nearly every Cocoa application). ChocTop will automatically generate the required XML feed; TextMate nor the bundle has a way to ask Sparkle to poll for it nor update itself, yet.

But, you can start versioning your DMGs today:

$ rake version:current
0.0.0
$ rake version:bump:major
$ rake version:current
1.0.0
$ rake dmg

The DMG will now have a version number.

Uploading new DMG versions

The original ChocTop was designed for Cocoa applications and included Sparkle support so your Cocoa applications automatically updated themselves when you built and uploaded a new version. I haven’t got a solution for this for TextMate bundles yet; but it seems like a good idea for the future.

Nonetheless, ChocTop still includes a rake upload task to ship new versions of your DMG to a server somewhere.

In your Rakefile, add the following config lines to the ChocTop block:

s.base_url   = 'http://some.host.com/upload/folder'
s.remote_dir = '/path/to/upload/folder'
s.host       = 'some.host.com'
s.user       = 'remote-user'

The s.base_url is the URL from where the DMG file will be found by users. Later, when I figure out how to do auto-updating of the TextMate bundles, it will also use this URL. This URL is also used to determine the host for uploading the file.

The s.remote_dir is a path on the target server that maps to the base_url. This folder must already exist; rsync cannot create it as far as I can tell. So ssh into the machine and mkdir -p /path/to/upload/folder

The latter two are optional: s.host is derived from s.base_url and s.user defaults to your current local user.

To upload the latest DMG, run:

rake upload

You can now share the URL http://some.host.com/upload/folder for people to download the DMG. A small PHP script redirects from the folder path to the DMG filename.

Customising

ChocTop allows you to customise nearly everything.

The s.defaults :textmate line is similar to the following configuration:

s.background_file = "...choctop/assets/textmate_background.jpg"
s.volume_icon     = "...choctop/assets/textmate_volume.icns"
s.icon_size       = 104
s.icon_text_size  = 12

For TextMate bundles, perhaps put customised assets into a Support/dmg folder.

A background image should include blank space for the large YourBundle.tmbundle icon and webloc URL file to your GitHub project (or other target URLs). There are no size constraints on the background image. Design something beautiful.

The volume icon is an icns file. You create this using OS X’s Icon Composer application. Start with a transparent png file and drop it into the box with the corresponding size.

For the Engine Yard tmbundle, the following configuration is used:

s.background_file = 'Support/dmg/engineyard.tmbundle.dmg.png'
s.volume_icon     = 'Support/dmg/engineyard.dmg.icns'

Additional files

If there are other files you explicitly want bundled in the DMG, say a pretty README.html or a folder of documentation, then you can specify them:

s.add_file 'README.html', :position => [50, 100]
s.add_file 'docs', :position => [100, 100], :name => 'Documentation'

Summary

ChocTop is pretty cool for bundling any set of files into a custom DMG, especially Cocoa applications and now TextMate bundles.

Hopefully one day we can have Sparkle auto-updates for TextMate bundles too.