Dr Nic

Already know C extensions in RubyGems? Cool - then just run the following cmds and see what can be generated for you; plus check out the TextMate bundle at the bottom.

sudo gem install newgem
newgem pickaxe
cd pickaxe
script/generate extconf my_test
rake test

For everyone else…

Its 15000km from Brisbane AU to Prague CZ where Euruko2008 - the European Ruby Conf - was held. I came ready to talk, to met lots of cool multi-lingual Rubyists, and to learn. Ooh, I learnt something alright.

Tim Becker was introducing Native C Extensions for Ruby, and fortunately he said “now, everyone, follow along with this example”. I’d never done native C extensions, but I’d received lots of requests from RubyGem developers on how to do it. I had no idea.

So I was typing in everything Tim told me to type in. When Tim changed slides too quickly, I may have yelled at him to slow down. Perhaps I was the only one doing his tutorial out of 300 people, but I didn’t care. This was gold.

After he finished his session, I dragged him off into the corridor with Jonas Pfenniger (zimbatm), and the three of us mapped out a generic layout for how native C extensions work within RubyGems. I didn’t know any of this, but Tim and Jonas did, and we probably looked silly sitting at a small table in the middle of the narrow corridor.

But at the end, we had a working RubyGem with native C extensions that were built: when the tests were executed via rake, and when the gem was installed. The next day I figured out how to get the C extension built via autotest.

Thanks to Tim and Jonas I was able to then write a extconf generator for RubyGems so that its super-super easy to get started writing native C extensions within RubyGems.

Tutorial

This tutorial is for *nix, as I’m still investigating win32 extensions, and jruby + .net/ironruby extensions. So when I figure that out - hopefully with the help of other people currently at RubyFools conferences, I’ll get back to you.

The code comes from the Pickaxe book - p264, and we’ll insert it into a new RubyGem using newgem (version 0.20.1+):

sudo gem install newgem
newgem pickaxe
cd pickaxe
script/generate extconf my_test

Create a test for a class MyTest that doesn’t exist yet:

# test/test_my_test_extn.rb
require "test/unit"
require 'pickaxe'

class TestMyTestExtn < Test::Unit::TestCase
  def test_working
    t = MyTest.new
    assert_equal(Object, MyTest.superclass)
    assert_equal(MyTest, t.class)
    t.add(1)
    t.add(2)
    assert_equal([1,2], t.instance_eval(”@arr”))
  end
end

Run rake to build the C extension and run the tests. You can also run autottest and it will automatically build the C extension before running the tests.

To create the MyTest class, using the code from p262 of the Pickaxe book:

#include "ruby.h"
static int id_push;

static VALUE t_init(VALUE self)
{
  VALUE arr;
  arr = rb_ary_new();
  rb_iv_set(self, "@arr", arr);
  return self;
}

static VALUE t_add(VALUE self, VALUE obj)
{
  VALUE arr;
  arr = rb_iv_get(self, "@arr");
  rb_funcall(arr, id_push, 1, obj);
  return arr;
}

VALUE cTest;
void Init_my_test() {
  cTest = rb_define_class("MyTest", rb_cObject);
  rb_define_method(cTest, "initialize", t_init, 0);
  rb_define_method(cTest, "add", t_add, 1);
  id_push = rb_intern("push");
}

To lib/pickaxe.rb:

require "my_test.so"
# or require 'my_test' if its unique

The last line will import the generated shared library. If the RubyGem is tested or installed on Windows, then the .dll file will be automatically loaded instead. The “.so” notation is merely a placeholder to explicitly specify the shared C-extension, rather than any Ruby library of the same name.

Now run tests (rake), the C extension will be rebuilt and the tests will pass.

Build and install RubyGem

rake manifest:refresh
rake install_gem
irb -rubygems -rpickaxe
> a = MyTest.new
> a.add 3

You have successfully created a C-extension within RubyGems, using TDD.

TextMate bundle for Ruby C extensions

I’ve started a TextMate bundle to give syntax highlighting + some simple snippets for developing the C files for Ruby extensions.

To install:

cd ~/Library/Application Support/TextMate/Bundles
git clone git://github.com/drnic/ruby-c-extensions-tmbundle.git "Ruby C Extensions.tmbundle"

or 

wget http://github.com/drnic/ruby-c-extensions-tmbundle/tarball/master
tar xfv drnic-ruby-c-extensions-tmbundle-master.tar.gz
mv drnic-ruby-c-extensions-tmbundle-master "Ruby C Extensions.tmbundle"

Then restart TextMate or “Reload Bundles”.

You can clone/fork the source via http://github.com/drnic/ruby-c-extensions-tmbundle/tree/master

Go straight to video below

I took a video recorder to the RejectConf, like I did in Portland. Unfortunately, there were two reasons I didn’t record any of them.

Firstly, there didn’t seem to be any obvious place to position the camera.

Secondly, it was deemed critical that everyone does their talks in the dark. The conference isn’t run in the dark, local Ruby groups don’t run meetings in the dark but consecutive RejectConfs have been run by adminstrators with a dark fetish. Great for drinking beer and heckling presenters. Bad for video recording.

The makers of the JVC HDD camcorder - a nifty device with a 20G HDD in it - don’t make it possible to record in the dark. Not because it doesn’t have a “night time mode” - apparently it does - but when you are already in the dark, you can’t figure out how to turn it on.

Ok, fine, if I’d made an effort I could have figured it out. So, let’s use excuse #1 as the reason for not recording the presentations.

Its a valid excuse, the Pirate Cove was chock-a-block full of people. The local Berlin Ruby Group did an awesome job of finding a great “underground”-esque venue.

Fortunately a fellow Australian - Marcus Crafter - had a front row position, and a MacBook Pro. With said device, he captured my talk on RubiGen (and John Barton’s).

RejectConf video of RubiGen

In 5 minutes, I make a Merb generator, using RubiGen and NewGem. Nifty stuff indeed.

Recorded and published by Marcus Crafter

There were lots of other awesome presentations (that is perhaps a dubious inference that mine was awesome), but it was dark, I had beer in both hands, and I was too busy yelling “AUSTRALIA!!!” to write notes.

What a great night

The New Gem Generator (0.13.0)’s newgem command now behaves like the rails command thanks to RubiGen - a new project that is an extraction of the rails_generator.

Developing a RubyGem? You can now have a script/generate command, just like Rails. Other projects, say like Merb, can do this too.

Awesomeness with a Gold Star. (disclosure: Dr Nic awarded this Gold Star to himself)

Warning

This article can get confusing.

Especially around the heading “Generating Generators”.

Plus, this article talks about two projects - an update to New Gem Generator (0.13.0) and the new RubiGen project.

So, give yourself time to digest it all, and then perhaps come back and read it again.

And then perhaps re-write the article for me.

“A generator that can provide generators to your projects, so that you can write generators for your other projects.”

Re-write it - I dare you.

Background

One of the killer features of Rails is the Rails Generator.

It does three things:

• generates an entire scaffold for your application, thus sharing with you its conventions (over configuration) methodology; and
• it then allows you to generate more stuff, like models, controllers, plugins etc. Things that are relevant to a Rails app. Finally,
• it allows you to write your own generators for Rails apps.

Now you can do all this with New Gem.

Killer Feature #1 - upgrade path

Never before have you been able to run ‘newgem’ on top of an existing RubyGem. Why? It used to blow away the entire folder, and then start writing new files… not very friendly.

Now, just like the rails command, you can go into your RubyGem development directory, and run newgem . and you will be asked which files to override.

NOTE: Copy your Rakefile to Rakefile.old, and after running newgem . copy the configuration information into the new file config/hoe.rb.

If you’re using newgem already, I dare you to upgrade as above. (Ok, perhaps make a backup copy of your work first…)

Killer Feature #2 - script/generate

Once you’ve upgraded, or create a new RubyGem (using newgem gemname), you’ll now have two scripts: script/generate and script/destroy, just like Rails has. (and matching generate.cmd and destroy.cmd for Windows users)

Try them out:

script/generate
...
Installed Generators
  Rubygems: application_generator, component_generator, executable, install_jruby, install_rspec, install_website
  Builtin: test_unit

Oh yeah, I’ve had some fun extracting things into generators.

• Install the dubious-looking NewGem website - script/generate install_website.
• Install RSpec support - script/generate install_rspec.
• Make the RubyGem a JRuby gem - script/generate install_jruby (the generated gem will have -jruby.gem in its name)
• Create an executable Ruby app - script/generate executable appname

These generators are also reused via the newgem commands various options (Run newgem to see them.)

Killer Feature #3 - generate generators

I’ll post more about RubiGen later, but you can create a new rails-like command-line app that generates a whole stack load of directories and files using script/generate application_generator appname.

There is a large USAGE rundown if you run script/generate application_generator.

Want to create your own generators for developing RubyGems? (similar to creating a generator for a Rails app, but for RubyGems)

script/generate component_generator foobar rubygems
    create  rubygems_generators/foobar/templates
    exists  test
    create  rubygems_generators/foobar/foobar_generator.rb
    create  test/test_foobar_generator.rb
    create  test/test_generator_helper.rb
    create  rubygems_generators/foobar/USAGE
    readme  readme

Firstly, note test/test_generatorname_generator.rb - that’s right, you get a test stub for your new generator. Start there, write tests, then write your generator. There’s inline help for useful assertions.

Secondly, note rubygems_generators folder. This folder is the “scope” of the generator. As it starts with “rubygems” the generator will only be available when you are developing rubygems. It will not show up in Rails, nor Merb or Camping or any other place that may support RubiGen one day.

Similarly, if you want to write a Rails generator using the component_generator then specify the scope as rails.

script/generate component_generator booya rails
     create  rails_generators/booya/templates
     exists  test
     create  rails_generators/booya/booya_generator.rb
     create  test/test_booya_generator.rb
  identical  test/test_generator_helper.rb
     create  rails_generators/booya/USAGE
     readme  readme

In Edge Rails (and any Rails version after 1.2.3), the script/generate command in Rails will search all RubyGems for /rails_generator/* folders in addition to the existing search paths (ticket).

Thanks goes to Rails Generator

I’ve long been in love with the Rails Generator for what it does, the beautiful syntax for specifying a generator, etc. I don’t know who wrote what bit, e.g. if DHH wrote all of it, or others wrote nice chunks, but its awesome. Thanks DHH and co. Thanks to Jeremy Kemper

RubiGen is 95% Rails Generator code, with extensions to support scoping. Setting Rails Generator free of its Rails constraints is a tribute to it.

Now RubyGems can use generators, and any other frameworks can integrate generators and their developers can write and distribute additional generators for those frameworks.

I award myself 2 gold stars, and a scratch-n-sniff

Summary: click to select

Keynote

DIY syntax

Dr Nic Academy

I also took the opportunity to introduce Dr Nic Academy - training courses in Ruby on Rails by me. The first course will be 7th and 8th of July in Amsterdam. If you are thinking of attending, hold off buying tickets!! The new website and a special bonus discount price will be coming soon. I’d hate you to miss out.

Before or after you’re sharing your own code as open-source, you might find yourself itching to fix someone else’s code.

Twelve months ago I made my first contribution to someone else’s project – I added some auto-generated tests to the AjaxScaffold project (now ActiveScaffold and no longer a rails generator). It was the first RubyGem I really investigated. It was the first Rails Generator I investigated. It was the first OSS project I think I’ve ever submitted a patch to. It took days.

Its always the same process

Yesterday I created a patch for Merb: an update to their README file so it converted to RDoc nicely. The whole patching process: spotting the ugly README, checking out merb trunk, changing its README, testing, creating a patch, and submitting it to Merb’s trac, took about 30 minutes whilst also doing other things.

The process was the same, I’m just better at it.

Step-by-step

So here’s how to fix other people’s code. It has Ruby-specific examples, but I’m sure the process is the same in all OSS communities. Its also worth checking out the slides from Josh Susser’s Railsconf talk on contributing to Rails Core.

1. Get annoyed by a defect or missing feature

Open source software isn’t written by an army of coders, with a back up army of testers, documenters, and “Dummies Guide” book authors. Rather, its code that fixed the author’s problem; and they were nice enough to flesh it out generically so you could use it. Or they were seeking fame and glory.

So you find problems. The project owner and all its users will be super grateful if you fix it for them.

2. Finding the source

The modern Ruby community uses subversion/svn for their source repositories. So now is your excuse to install it and start to learn how to use it.

The tricky part is “where is the repository?”

If the project has a RubyForge project go to (if its URL is http://XYZ.rubyforge.org, then its project page is http://rubyforge.org/projects/XYZ) then try the “SCM” tab on the project page. If the project isn’t hosting on RubyForge SVN, hopefully they have disabled this menu tab. If not, you’ll get version 0 when you do the checkout to tell you the project’s not hosted here.

Alternately, the project’s homepage/README file should tell you where the repository is.

Finally, email the project owner. And tell them to get their act together and publicise their SVN repos.

3. Checkout the source

Hopefully you’re told the complete URL to the trunk of the project. For anonymous access to the newgem RubyForge project the URL is svn://rubyforge.org/var/svn/newgem/trunk

A project SVN repository might contain multiple projects, and may or may not use the common trunk/branches/tags root folders. So you may need to navigate around the repository and til you find the trunk of the project’s code.

So for a generic RubyForge project, the root URL for anonymous access is svn://rubyforge.org/var/svn/XYZ.

$ svn list svn://rubyforge.org/var/svn/newgem
branches/
tags/
trunk/
$ svn list svn://rubyforge.org/var/svn/newgem/trunk
History.txt
…

[Thx to Seth - originally I suggested doing a checkout; but list is cleaner.]

If there is a trunk root folder, then do the following:

rm -rf XYZ
svn co svn://rubyforge.org/var/svn/XYZ/trunk XYZ
cd XYZ

And now you’ve checked out the trunk or “edge” of the project. The latest and greatest.

4. Snoop around

Load up the project into your editor and poke around.

If it has test cases (!) try running them. You may need to create some databases and/or modify the Rakefile/connection details to work with your database. Hopefully the project comes with rake tasks to create/drop the databases. rake -T should find them.

If its a RubyGem, try installing the gem locally (on top of the existing defective gem that you’re about to fix). Again, run rake -T and hopefully there is a task install_gem or similar to build and install the gem from source. If not, send hate mail to the author. And then fix the Rakefile to use the Hoe gem. And then send a patch to the author with the fixed Rakefile.

rake install_gem

5. Make changes

Finally, you get to change their code.

If the project has a suite of test cases, please make an effort to add “failing test cases”. That is, add test cases that demonstrate the defect by failing (unless the project includes your fix).

Run the tests to ensure they are working.

6. Refresh the code from the repository (svn up)

The original repository may have changed since you did the checkout, so update your version.

svn up

If there are any conflicts, go fix them and delete any helper files created by SVN.

7. Create the patch

First, you need to tell subversion about any folders/files you’ve added (svn add) or deleted (svn remove).

Then, from the root folder of the trunk:

svn diff > i_fixed_a_bug.patch

Give the patch file a useful, readable name. Like readme_updated_for_rdoc_syntax.patch.

8a. Submit patch to trac

[link to 8a]

Each project may have a different preferred way to submit patches.

If the project uses trac then:

1. Register an account (normally a link at the top right of the page) if available. Some trac installations require an account (e.g. Rails) and others allow anonymous tickets.
2. Click “New Ticket”
3. Fill in details (note, there is nowhere initially to add your patch). Prefix the subject with [PATCH]
4. Save the ticket (check the box “Attach files”)
5. Attach the patch file and save

8b. Submit patch to Google Groups

[link to 8b]

Projects hosted on rubyforge SVN cannot use trac. I personally use Google Groups for my projects – the project communities are all small enough such that its manageable.

1. Join the Group (using Google Account)
2. Add a picture to your Google Groups account; I cannot communicate how relatively unnecessary this step truly is
3. Go to Files section (link on right hand side)
4. Attach your patch file to the group
5. Get the URL for the uploaded file
6. Post a note/write an email to the group describing your patch, and include the URL for the patch file

8c. Submit patch to RubyForge Tracker

[link to 8c] [Thx Chris Martin]

RubyForge offers a bug/patch/feature request tracker for each Project. It is enabled by default for new projects:

1. Login or Create new Rubyforge account
2. Go to project page and click “Tracker“ from sub-menu
3. Click “Submit New”
4. Fill in details (each RubyForge Project can contain many sub-projects/gems - pick from Category drop-down)
5. Check the box “Check to Upload & Attach File”
6. Browse and select your patch file; put file name in File Description field as you’ve already made it human readable, because you’re a champion.
7. Save the ticket

8d. Submit patch to LaunchPad Tracker

[link to 8d]

“I have to say I quite like Launchpad, now that I’ve got used to it. If you’re an Ubuntu guy like me then it is perfect.” - from Neil Wilson

1. Login to/Register for your Launchpad account
2. From The Launchpad bug tracker click the Report a bug button.
3. Enter the name of the project, or select it from the list.
4. The project in question will almost certainly provide its own bug reporting link that takes you directly to the Report a New bug page.
5. Enter a one line summary and click Continue
6. Fill in the comment box and click Submit Bug Report
7. Then click the link “Add a comment/attachment”, and Browse for the patch file. Check “This attachment is a patch” and add a Description if required.
8. Click Save Changes

8z. Email the project owner

[link to 8z]

See title: Email the project owner. Prefix the subject with [PATCH].

Why 8z? I figured readers might want to share other ways they coordinate within a project, like Google Code, etc. I haven’t used it. So, (c) - (y) are placeholders

Corollary: Tell users how to fix problems

If you administer some OSS projects, you are doing yourself a disservice not telling your users how to quickly, easily submit patches.

Either copy and paste the relevant sections above, and/or link them to this “8 steps” page.

Make it really clear on your project website that patches are welcome and will be rewarded with free t-shirts and other complimentary items of stationary, and watch those patches roll in.

For example, on the New Gem Generator site (and in all default project websites), there is a link to this page and a reference to section on Google Groups as the default.

How to submit patches

Read the 8 steps for fixing other people’s code and for section 8b, use the Google Group above..

The trunk repository is svn://rubyforge.org/var/svn/newgem/trunk for anonymous access.