Ever see example shell commands like this and wish you could paste them in?
Fear no more. A pastie is at hand.
Here’s how to install it:
In your `.bash_profile` add the following:
@dwaite and others for `$@` instead of `$0 $1 …`.
It irked me a little that I could develop on one stack (OS X, Rubinius, Sqlite3), run continuous integration (CI) on another stack (Ubuntu, Ruby 1.8.7, Postgresql), and deploy into another stack (Gentoo, Ruby 1.9.2, MySQL). I think what irks and worries me is that there are three sets of differences to be aware of. A bug in production? Was it a missing test scenario or one of the many differences between production and CI environments?
So I think I have two solutions.
First, use a VM that matches the production environment. Each different production environment would mean another VM. If you are managing your own production environment, then all you need is the tools (described in this article) to recreate your production environment in a VM.
Second, use a clone of your production environment. That is, if you deploy to Engine Yard AppCloud then run CI in Engine Yard AppCloud; if you deploy to a single Ubuntu instance on Slicehost, the have another matching Ubuntu instance on Slicehost.
I’ll write about the first solution – using VMs – here, and I’ll write on the Engine Yard Blog about the solution for Engine Yard AppCloud customers. For AppCloud users life will be even easier because there are zero setup steps to ensure you have consistent environments. It’s been one of my favorite projects in the two months since I arrived at Engine Yard.
I’ll also introduce a CLI for talking to Hudson – one that assumes you are working on Ruby/Rails projects – and makes it really easy to get up and running with a server, your Rails/Ruby projects, and any VMs you need (optional for Hudson, but they are the point of my line of thinking).
And I’ll introduce Vagrant, a CLI for creating/managing/destroying VMs.
Oh, and I’ll introduce Hudson CI.
Ok, let’s fix all our CI problems in one go…
I needed a CI tool to allow tests to run inside VMs or on remote servers/VMs. Back in May I found and fell in love with Hudson CI. Fortunately it had support for “slaves”, and a way for CI jobs (your applications or rubygems) to select which slaves can be used. Hudson is also great because it is easy to try out (one click to install and launch), easy to configure, and has 350+ plugins.
Charles Lowell’s hudson.rb project is a CLI for launching Hudson (it’s bundled inside the gem), and a set of CLI tasks to add/remove projects (jobs), add slaves, trigger builds and more.
gem install hudson
You can launch Hudson via:
hudson server # Default: --port 3001
open http://localhost:3001
It currently assumes you are working on Ruby projects, using bundler, and attempts to create a useful set of default steps for your CI jobs.
cd /some/rails3/app
hudson create . --host localhost --port 3001 --template rails3
Hudson CI will automatically start running steps to install your application’s gems, database, and run your tests. When it does it on your local machine it’s not that impressive. When it happens on a fresh VM or slave node and you didn’t have to do anything to set it up, it’s awesome.
If you can script it, then you can automate it. Fortunately, as a virtual machine VirtualBox is both scriptable and FREE! Secondly, if you have a VM, you’ll need to script its setup/provisioning; so fortunately there exists chef and puppet, amongst others.
Thirdly, if you are really really really lazy, like me, you will want Vagrant. Created by Mitchell Hashimoto and John Bender, Vagrant is a tool for building and distributing virtualized development environments. Everyone in your team have their own OS/configuration? Got Windows users on your team? Use Vagrant.
Once you have VirtualBox installed, getting any of your projects live within a VM is trival:
gem install vagrant
vagrant box add base http://files.vagrantup.com/lucid32.box
vagrant init
vagrant up
vagrant ssh
$ cd /vagrant/
$ ls -al
You will now see the contents of your project folder, but from within the VM instance! They are linked – a change to either is automatically reflected inside and outside of the VM.
In this example, the Guest OS being downloaded and installed into VirtualBox (note: without any GUI) is Ubuntu Lucid, but you could use any VirtualBox packaged unix system.
For wonderful guided tour of Vagrant see the Getting Started video.
Vagrant – Getting Started from Mitchell Hashimoto on Vimeo.
Whether you choose to use VMs for a production-like development environment (a good idea for anyone, a wonderful idea for Windows developers who are very far removed from their production experience), it is a very good idea to have a CI environment as similar to your production environment as possible. Here, we want a VM instance with all the components/utilities/rubies set up that you have in production.
I have created an example Rails application that is setup to use Vagrant for a CI slave VM.
See the README for complete instructions. See the Vagrantfile and the cookbooks folder for the configuration and provisioning recipes.
Once you have the VM instantiated, you add it to your Hudson CI master (either the localhost one above or your remotely hosted server) with the CLI:
hudson add_node localhost --name "VM" --label railsapp-vagrant ...
(See the example Rails application for the other flags I used to get this working). It is now available to all Hudson jobs; but has a label “railsapp-vagrant” to allow jobs (Hudson’s name for a project) to specify that slave node specifically.
To add the Rails application to Hudson CI, and force it to run the tests in this VM:
hudson create . --template rails3 --assigned-node railsapp-vagrant
If you visit the Hudson master (at http://localhost:3010 in the example) you will see the job automatically running (“building”) within your VM. It will use bundler to install the gems, and run all the tests.
I know, CI is historically a pain in the arse. When CruiseControl was the only CI kid-on-the-block, it was standard for people to respond “4 days” to the question “How long does it take to get set up?”
It can now be really easy.
More importantly than being easy, you are running an application’s tests within an isolated VM that you can design to match your production environment.
I think there can be a good future for CI and Rails applications. Thoughts on this solution?
Some of the technology in this article is old, other bits are very new, but for all of it I thank all the creators and contributors. Hudson CI was created by Kohsuke Kawaguchi during his days at Sun. He is now offering Hudson support services via his company InfraDNA.
The Hudson.rb project was created by Charles Lowell to bundle the Hudson CI and some common useful plugins for Ruby/Rails projects. He came all the way to Gothenburg, Sweden for NordicRuby conference (great conference by the way!) I fell in love with Hudson and started helping Charles on Hudson.rb.
Thanks also to both Kohsuke and Charles for starting work on a JRuby plugin for Hudson CI, and writing up a progress report.
Thanks to everyone who agrees that Hudson CI is awesome.
Thanks to Mitchell Hashimoto and John Bender for creating Vagrant. It is an incredible tool for developing within a VM (on OS X or Windows).
Finally, thanks to Bo Jeanes who helped on the Engine Yard AppCloud version of this project. Coming soon!
If you’re in New Orleans this week for RubyConf, I’ll be at the local Ruby group RubyBayou talking about Hudson CI, Vagrant, and AppCloud on Thursday night. There’s a happy hour from 5pm till the 7pm start.
Location: LaunchPad NOLA, 643 Magazine St, New Orleans, LA 70130
Come and let me convince you about the wonders of Hudson CI and having a CI test environment that matches your production env. You know you want to be convinced. It’s good for you, like fruit.
Always remember to install the Chuck Norris plugin for Hudson, and enable it for each job. Don’t forget Chuck Norris.
In some TextMate bundles, if you save a file it will also validate the file and show any syntax errors in a tooltip. This is awesome. (e.g. JavaScript and CoffeeScript)
So I added the same thing to my Ruby.tmbundle. Install this, save a dodgy Ruby file and you’ll now see something like:
Do you think the following syntax error tooltip is more useful?
Yes it lovely, and the new Ruby.tmbundle will automatically do this if it can find rbx in your TextMate’s $PATH. Yeah yeah.
If you have Homebrew installed:
brew install rubinius
Then in TextMate, add your homebrew bin folder to the $PATH.
PATH variable, and add “:/path/to/homebrew/bin”For example, if you have homebrew installed in ~/.homebrew then you might add :/Users/drnic/.homebrew/bin
. My complete $PATH in TextMate is:
/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/usr/local/bin/:/Users/drnic/.homebrew/bin
Save a dodgy Ruby file and see the beautifully helpful syntax message.
To install via Git:
mkdir -p ~/Library/Application\ Support/TextMate/Bundles
cd ~/Library/Application\ Support/TextMate/Bundles
git clone git://github.com/drnic/ruby-tmbundle.git "Ruby.tmbundle"
osascript -e 'tell app "TextMate" to reload bundles'
I’m pretty excited about CoffeeScript as a clean-syntax replacement for pure JavaScript.
Imagine all the syntactical delights of Ruby and Haml for your JavaScript. You write in a nice language, but get normal JavaScript at runtime. All whilst having full access to 3rd-party JavaScript libraries (jQuery, PrototypeJS), debugging support (it becomes pure, readable JavaScript), existing support from test suites (it’s normal JavaScript) and growing support from various text editors (TextMate, Vim, Emacs).
What simple delights?
No trailing semi-colons. No { some_code() } function/closure brackets. String interpolation. Multi-line strings. Explicit class syntax. Array slicing. An existential ? operator.
Scroll down the home page for awesome example after example.
These aren’t library extensions. This is clean, purposeful syntax.
You can play with the joyful syntax of CoffeeScript on the website. After reading the basic examples on the CoffeeScript home page, press “TRY COFFEESCRIPT” in the header menu.
As you play with the syntax, the equivalent JavaScript is printed on the right hand side (see image above).
How nice is that syntax? Very.
For NodeJS (get latest release URL; using 0.1.31 as 0.1.32 doesn’t unpack for me):
cd /usr/local/src
wget http://nodejs.org/dist/node-v0.1.31.tar.gz
tar xfv node-v0.1.31.tar.gz
cd node-v0.1.31
./configure
make
sudo make install
For CoffeeScript (get latest release URL):
cd /usr/local/src
wget http://github.com/jashkenas/coffee-script/tarball/0.5.5
tar xfv jashkenas-coffee-script-bcf7b3f.tar.gz
cd jashkenas-coffee-script-bcf7b3f
sudo bin/cake install
Now test that everything is in place:
$ coffee --version
CoffeeScript version 0.5.5
$ coffee -e "sys: require 'sys'; sys.puts 'hello world\n'"
hello world
Phew!
Note, in the command-line/on the server, you are using the NodeJS JavaScript environment. It supports the CommonJS API for loading modules (normal JavaScript: var sys = require('sys')).
Your application source code will have *.coffee files containing your sexy, short CoffeeScript. But at runtime, the browser needs the generated JavaScript.
I’ve been using the Jonas Nicklas’ bistro_car gem:
gem install bistro_car
mkdir -p app/scripts
In your Rails config/environment.rb file, add:
config.gem 'bistro_car'
And in your layouts, such as app/views/layouts/application.html.erb add to the <head> or the bottom:
<%= coffee_script_bundle %>
Now you’re good to go. Add your CoffeeScript files in app/scripts/*.coffee and they will be automatically available as JavaScript.
Check that this hasn’t happened:
$ coffee --version
CoffeeScript version 0.3.2
$ which coffee
/usr/bin/coffee
Arrgh, we should be using /usr/local/bin/coffee. bistro_car currently installs the old rubygem-based version of coffee-script; and you might be unlucky to have your $PATH find the wrong one first.
Either delete it (sudo rm /usr/bin/coffee and restart your shell) or make sure /usr/local/bin is earlier in your $PATH than /usr/bin, where RubyGems installed the old, unnecessary version of coffee command.
Create a file app/scripts/application.coffee with contents:
powers: [1,2,3,4].map (i) -> i * i
alert powers
Load up a view in a browser and see [1,4,9,16]. You win! Throw in some jQuery/PrototypeJS/whatever. Beautiful.
View the source of the page, navigate to public/javascripts/bundle/default.js and you’ll see the generated source:
(function(){
var powers;
powers = [1, 2, 3, 4].map(function(i) {
return i * i;
});
alert(powers);
})();
Heroku is a great place to host apps. Though it doesn’t have CoffeeScript installed so it cannot dynamically convert the *.coffee files into JavaScript.
If you want to use Heroku I guess we need to perform the conversion locally and deploy it.
But. In development and integration testing I want bistro_car’s dynamically generated default.js. In production, I need a cached version.
In application.html.haml I use (I can’t keep pretending I use erb):
- if Rails.env.production?
= javascript_include_tag "coffeescripts"
- else
= coffee_script_bundle
Now we’re just left with the hassle of automatically generating public/javascripts/coffeescripts.js.
First, a rake task. Second, a git pre-commit hook.
Create lib/tasks/bistro_car.rake:
desc "Generate the cached bundle/default.js file from app/scripts/*.coffee files"
task :bistro_car => :environment do
path = "public/javascripts/coffeescripts.js"
puts "Building *.coffee -> #{path}"
File.open(path, "w") { |file| file << BistroCar::Bundle.new('default').to_javascript }
end
file "public/javascripts/coffeescripts.js" => Dir[File.join(Rails.root, 'app/scripts/*.coffee')] do |t|
Rake::Task["bistro_car"].invoke
end
Now you can create coffeescripts.js and add it to the repo with:
rake public/javascripts/coffeescripts.js
git add public/javascripts/coffeescripts.js
git commit -m "Initial bundled coffeescripts file"
Now create .git/hooks/pre-commit:
#!/bin/sh
exec rake public/javascripts/coffeescripts.js
And make it executable (and git commit will invoke it automatically):
chmod +x .git/hooks/pre-commit
Phew.
Now, whenever you change a *.coffee script and you are about to commit it, the cached-production-only coffeescripts.js is automatically updated and included in the same commit.
Seems like a clean hack.
Why not make a library to do this? Well I’m hoping there is a better, cleaner way. Perhaps bistro_car can include a rails generator to package these bits and pieces itself, if my approach happens to be the best way.
Nonetheless, let history record that CoffeeScript is very cool though in the world of Heroku living with it is non-trivial at the moment.
Ever used a rubygem, found a bug, and just wanted to quickly bust out the big guns and fix it quickly?
The gem command doesn’t come packed with a way to find the original source repository for a gem. At best, most gems at least come bundled with the complete source, tests and documentation. Some gems don’t. Fair enough, since having access to the complete source via the gem still doesn’t allow you to fix a bug and share it with the world.
For that you access to the repo, a quick way to fork it, and a post-github way to share a gem version from yours truly.
The github gem and gemcutter are the modern day tools of master hackermanship.
Let’s say you find a bug in a gem, say rails, and you want to go to town on its source.
You know the gem is called rails but you’ve no idea what the github repo is called. Never fear.
$ gem sources -a http://gemcutter.org
$ sudo gem install github
$ gh clone --search rails
Select a repository to clone:
1. rails/rails # Ruby on Rails
2. technoweenie/restful-authentication # Generates common user ...
3. justinfrench/formtastic # A Rails form builder plugin ...
?
Press 1 and you’ll get a clone of rails/rails.
Alternately, if you want a fork or you know the exact user/repo already:
$ gh clone rails/rails
Now, fork your own version:
$ cd rails
$ gh fork
You now have your own fork. The origin remote also now points to your fork rather than the rails/rails repository:
$ git remote show origin
* remote origin
Fetch URL: git@github.com:drnic/rails.git
Push URL: git@github.com:drnic/rails.git
So, make your changes, push them. Send a pull request or github issue or lighthouse ticket or what have you.
Want to get to the github project home page for your fork?
$ gh home
Let’s say you patched the rails gem itself but you want to share your changes via your own gem.
In the olden days, github did this for you. Now you use gemcutter, and a little manual effort to do your own renaming.
First, install the gems locally, use them, and make sure all is good.
For rails, you install the edge gems (3.0.pre) with:
$ rake install
You can’t see ‘rake install’ in the rake -T list (hence my patch), but I think the following expression displays all tasks regardless if they have a description or not:
$ rake -P | grep "^r"
Rails is composed of several gems, unlike most projects that are distributed as a single gem. Here we want to share our commit within a new drnic-rails gem, but not touch the others.
Edit the railties/rails.gemspec file from:
Gem::Specification.new do |s|
s.platform = Gem::Platform::RUBY
s.name = 'rails'
s.version = '3.0.pre'
...
and give your personal gem a new name:
Gem::Specification.new do |s|
s.platform = Gem::Platform::RUBY
s.name = 'drnic-rails'
s.version = '3.0.pre'
To build and distribute the new gem:
$ gem build railties/rails.gemspec
$ sudo gem install gemcutter
$ gem push drnic-rails-3.0.pre.gem
Pushing gem to Gemcutter...
Successfully registered gem: drnic-rails (3.0.pre)
Follow any first-time gemcutter instructions and SUCCESS! Now I have my own drnic-rails gem.
To find, clone, and fork any rubygem that is hosted on github:
$ sudo gem install drnic-github
$ gh clone --search rails
$ gh fork
To personalise the gem and share it on gemcutter:
> edit the project.gemspec to have a unique name, e.g. yourname-project
$ gem build project.gemspec
$ sudo gem install gemcutter
$ gem push yourname-project-1.0.0.gem
I think this makes it much easier, faster and more fun to hack other people’s stuff.