Dr Nic

How to make a good home-made Open Source

Want to be the funniest person at the next hacker’s picnic? Point at a bottle of red ketchup with its lid next to it on the table and pronounce “Hey look, Open Source.”

Be ready with follow-ups like “Can you pass me the Haml?”

If you’ve used Ruby on Rails, Apache, Emacs, or Linux then you would have been impressed by the awesome quality of these free bits of software which are so important to us. They are free, they are important, and they are awesome.

Paying money for poor commercial software makes awesome, important free software appear even more awesome and important.

The facts seem gloomy. You are a humble developer. Awesome, important free software is a Herculean achievement.

Conclusion? You implicitly believe you will never write awesome, important free software.

But “never” is an awfully long time. And is the only goal “awesome, important free software”?

Reasons to write?

Personally, I don’t think I’ve ever created an open source project that is either important or awesome. I think my motivations for open source — my own projects or stuff added to other’s projects — is either:

“Wouldn’t it be cool if you could do XYZ?” or “Seriously. Why can’t I do XYZ?”

I was either amused or annoyed. Dr Nic’s Magic Models was a joke. ChocTop was vented frustration.

Perhaps there are different reasons. I find the following examples inspiring.

I think the late Why the Lucky Stiff created entertaining free software.

Tim Lucas created artistic free software (‘View Source’ to see the header comment)

Christian Neukirchen created liberating free software.

Ryan Davis created free tools.

Chris Wanstrath ports free software.

Is there a muse that you can choose?

What other reasons are there for writing examples? Perhaps leave comments below and I’ll add them to the list above.

Who? Me?

And “awesome” sounds awfully challenging to aim for. Surely, “Awesome” is just one end of a scale with “Worthless” at the other end. “Moderately Good”, “Average”, “Below Average”, and “Where are the test cases?!” are in the middle.

Have you ever visited a friend who you find putting on the finishing touches to a 6’ by 4’ canvas painting of their entire family from their last Christmas dinner together, and they say “want to help?” Unlikely. Fortunately open source software “paintings” are a free-for-all.

You can write Libraries, Adaptors, Applications, Frameworks, Tools, Extensions and Services.

You don’t even need to create new free software. Fix something that someone else broke. Add a feature that was missing. Write documentation after you eventually figured out what to do.

Mid-Year’s Resolution

It’s now April. If you’re still looking for a 2010 New Year’s Resolution, borrow this one: “Write some open source software.”

If you’re going to RailsConf, perhaps come along to my tutorial The 8 Steps to Contributing to OSS or let’s catch up in the corridors. It’s going to be a great RailsConf!

rails2010_header_bg

Installing Mingle with Capistrano/Deprec

The other day I visited the ThoughtWorks office in Melbourne (they hosted the Melb Ruby meeting). The most notable feature of their office versus any office I’ve ever worked in before were the 5×3 cards (and post-it notes) … all over the walls.

From my understanding – which came after asking lots of questions – is that the cards help with:

  • The team (developers, clients, etc) deciding what parts of the application are most important to be built. The cards are isolated stories/use cases and should implicitly/explicitly state the business value. At each stage of the project – “iterations” which are 1 or 2 week long – the team decides which cards/stories should be implemented.
  • The cards for an iteration are placed on the wall. Thus it is transparent to all the team what work is being performed.
  • From left to right, the wall is divided into columns. Each card can be placed in one column. Columns are typically labelled with stage names like “Under development”, “Completed”, “QA/Testing”, “Accepted”. This gives a visualisation of what work/card is where in the development/release process. It can also indicate if there is a bottleneck – a lot of cards in the “QA/Testing” column might mean you need more testers.

The first point – using stories on cards to decide what should be built now – seemed like a winning idea. A new client of mine has trouble translating a gazillion feature requests into a sequence of “this part should be built now” stages. Everything is uber cricital, all the time. And often the feature requests don’t seem related to something a user might actually want to use.

Whilst cards on a wall would work if the client and developers were co-located (aka “sitting together”), that’s not the situation here. I’m in Brisbane, and they are in Sydney (1000km).

With colourful 5×3 cards and the promise of fruitful client relationships burned into my retina I wondered what web tools might help with emulating this process. The cards. The wall. The transparency. The understanding that “software costs money to build so what is most important?”

And I remembered Mingle – a tool built by ThoughtWorks for agile project development, but not (to my knowledge) used by any ThoughtWorks teams themselves. UPDATE: Jay Fields says several teams are currently using it.

But I can understand why they don’t use it. They have their own 5×3 cards with their “ThoughtWorks” logo on it. Well, they had some. I stole a bunch. They also have walls and co-location.

Nonetheless, I gave it a burl. I runs nicely when the server is installed on my CoreDuo 2gig MacBook. It didn’t run nicely at all on my 256Mb VPS that was already busy running a bunch of other things (mongrels, mysql, etc).

Mingle requires 2 gig of RAM. Even on the ever affordable Slicehost, that’s $140 USD a month. Mingle might be free for < 5 users (and OSS projects) but there is a notable hardware requirement, thanks I guess to the Java/JRuby/Rails stack.

But I've done something useful - I've written capistrano/deprec recipies to install Mingle on a remote machine with (minimal) manual intervention. You just need to choose a machine with 2G RAM spare. You also need Java and MySQL installed, but there are capistrano tasks to help with that too.

Ahh, the world of remote automation.

The recipe below is mostly *nix agnostic, except the java_install task which uses ubuntu/debian's apt-get system. Actually, it tells you what to manually run yourself, as Sun are tools and require you to "click" the word "Yes" twice. I don't know how to automate this with Capistrano.

The mingle_install task downloads and installs Mingle, and includes an JVM optimisation option that I found on the forums (the MingleServer.vmoptions file created).

So, from scratch, if you want to install Mingle on a remote server (independently of any Rails app etc), do the following:

sudo gem install deprec -v 1.9.1
alias cap1='cap _1.4.1_'
mkdir -p mingle_installer/config
cd !$
vim deploy.rb

And insert the following into the deploy.rb script.

Modify YOURDOMAIN to an IP that you have root access to. The script will create two folders - /usr/local/src/mingle and /var/www/apps/mingle. Its pretty harmless.

To install, run the capistrano tasks:

cd ..
cap1 java_install
cap1 mingle_install
cap1 mingle_start

BUT, mingle_start doesn't work. The server just doesn't start and I don't know why.

So, ssh into your machine (you had to do this for the java install step too BTW), and run:

/usr/local/src/mingle/MingleServer start

If we can fix this then that'd be uber cool.

Going offline without your favourite Subversion repository?

UPDATE: useful gitify command below.

All my client projects are hosted on Subversion repos. All my OSS projects are hosted on Rubyforge Subversion repos.

And tomorrow I head off for the 2nd “no internet” RailsCamp megafest, being held outside of Melbourne. From what I heard about the 1st one, it involved a lot of Guitar Hero II and beer. This weekend I’m promised that someone is bringing a Wii – dear God I do love the tennis game.

Like MacDonalds burgers and Veganism, if there are two things that don’t go together its Subversion and “No Internet”.

Fortunately, for all RailsCampers, there is a solution to your fears that you won’t be able to hack on your favourite project, and will be forced into playing Guitar Hero II/III and/or the Wii for the whole weekend.

Git.

Specifically, the git-svn command.

More specifically, follow the following steps:

  1. git-svn clone <your svn url> projectname.git
  2. Go camping for the weekend.
  3. Commit changes with git commit -a -m "hahaha I'm camping and working - doh!"
  4. Come home from camping.
  5. git-svn rebase to re-import any SVN changes that might have been committed whilst you were away. If there are conflicts, just follow the instructions.
  6. git-svn dcommit and your svn repository will be updated with all your fancy changes; no one will ever know you used Git.

Installing Git

I remember having problems getting git-svn to run because the default path to the perl cmd is crap, and you need to specify PERL_PATH='/usr/bin/env perl', I think, before installation.

For MacPorts etc, its sudo port install git-core. I think the reason its not just called ‘git’ is to remind you that you know nothing about git. Which is fair. You don’t if you don’t know how to install it, I guess.

From source, get the tarball from the home page.

For Windows, use cygwin.

Gitify command

Want a utility cmd to create a Git repo in the parent folder of a Subversion project? That is, go to your Rails app that is checked out from a Subversion repo, and run gitify, and you will have a foldername.git folder in the parent folder.

Stick this code into ~/bin/gitify and add ~/bin to your path:

Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”

At this point in time there’s no facility in Rails to talk to more than one database at a time.

Alex Payne

I possibly have such a facility. Perhaps it will help, and I will get some DHH-love and perhaps a free Twitter account for my troubles. Or perhaps a t-shirt.

As a bonus, the solution even includes decent Ruby-fu syntax. So, if you’re just here for the view:

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

That code just there solves all our problems. It will invoke Person.find(:all) on a random database connection to (assumably) a clone database. Awesomeness I think. I hope it helps Twitter and all the Twit-sers (or whatever you call a user of Twitter).

This solution comes from the magic_multi_connections gem.

What is going on here?

I think a tutorial is the best way to demonstrate what is happening here. So, let's create a rails app and mix in the magic_multi_connections gem.

First, get the gem. Second, create a rails app:

$ sudo gem install magic_multi_connections
$ rails multi -d sqlite3

Now edit the config/database.yml file to create some more databases:

development:
  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000

development_clone1:
  adapter: sqlite3
  database: db/development_clone1.sqlite3
  timeout: 5000

development_clone2:
  adapter: sqlite3
  database: db/development_clone2.sqlite3
  timeout: 5000

But please pretend these are uber-MySQL clusters or whatever.

Think of :development as the read-write connection, and the :development_cloneN connections are for read-only access.

At the bottom of your environment.rb file, add the following:

require 'magic_multi_connections'
connection_names = ActiveRecord::Base.configurations.keys.select do |name|
  name =~ /^#{ENV['RAILS_ENV']}_clone/
end
@@connection_pool = connection_names.map do |connection_name|
  Object.class_eval <<-EOS
    module #{connection_name.camelize}
      establish_connection :#{connection_name}
    end
  EOS
  connection_name.camelize.constantize
end

Let's test what this gives us in the console:

$ ruby script/console
>> @@connection_pool
=> [DevelopmentClone1, DevelopmentClone2]
>> DevelopmentClone1.class
=> Module
>> DevelopmentClone1.connection_spec
=> :development_clone1

Our new modules will act as connections. One module per connection. The code above gives them names to match the connection names, but its really irrelevant what they are called, thanks to the mysterious conn method.

So, go create some models and some data. I'll use Person as the class here.

To setup the schemas in our clone databases, we'll use rake db:migrate. To do this:

$ cp config/environments/development.rb config/environments/development_clone1.rb
$ cp config/environments/development.rb config/environments/development_clone2.rb
$ rake db:migrate RAILS_ENV=development
$ rake db:migrate RAILS_ENV=development_clone1
$ rake db:migrate RAILS_ENV=development_clone2

To differentiate the databases in our example, assume there are two Person records in the :development database, and none in the two clones. Of course, in real-life, they are clones. You'd have a replicate mechanism in there somewhere.

Now, we can access our normal Rails modules through our connection modules. Magically of course.

>> ActiveRecord::Base.active_connections.keys
=> []
>> Person.count
=> 2
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base"]
>> DevelopmentClone1::Person.count
=> 0
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base", "DevelopmentClone1::Person"]

Wowzers. Person and DevelopmentClone1::Person classes?

But note - Person.count => 2 and DevelopmentClone1::Person.count => 0 - they are accessing different databases. The same class definition Person is being used for multiple database connections. We never defined more Person classes. Just the standard default one in app/models/person.rb.

The active_connections result shows that DevelopmentClone1::Person has its own connection. Yet you never had to manually call DevelopmentClone1::Person.establish_connection :development_clone1 - it was called automatically when the class is created.

Of course, DevelopmentClone2::Person is automatically connected to :development_clone2, and so on.

Behind the scenes

Let's look at our generated classes:

$ ruby script/console
>> DevelopmentClone1::Person
=> DevelopmentClone1::Person
>> Person
=> Person
>> DevelopmentClone1::Person.superclass
=> Person

That is, there is a DevelopmentClone1::Person class, automagically generated for you, which is a subclass of Person, so it has all its behaviour etc.

Dynamic connection pools within Rails controllers

The magic of the conn method will now be revealed:

$ ruby script/console
>> def conn
>>   @@connection_pool[rand(@@connection_pool.size)]
>> end
>> conn::Person.name
=> "DevelopmentClone2::Person"
>> conn::Person.name
=> "DevelopmentClone1::Person"

The conn method randomly returns one of the connection modules. Subsequently, conn::Person returns a Person class that is connected to a random clone database. Booya. Free Twitter swag coming my way.

Place the conn method in the ApplicationController class, and you can get dynamic connection pooling within Rails actions as needed, as in the following example (from the top of the article):

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

Decent Ruby-fu, I think. Certainly better than manually calling establish_connection on model classes before each call (or in a before_filter call, I guess).

This is just a concept

I know this tutorial above works. But that might be the extent of what I know. Let me know if this has any quirks (especially if you solve them), or if this is a stupid idea for implementing connection pooling with nice Ruby syntax.

Hope it helps.

Post-Halloween RadRails trick – all TextMate snippets available

Textmate Snippets to Radrails

HTML snippets
48
Ruby snippets
199

I tease myself with things I can’t have – like watching TextMate demo videos. One wonderful video shows a test-driven design demonstration by James Edward Gray II where he used Textmate snippets to generate the various assert statements. For example:

ase => assert_equal(expected, actual)
asn => assert_nil(instance)
asm => assert_match(/expected_pattern/, actual_string)

I’d been using Corban Brook’s templates but they didn’t include any assert or migration templates. And I wanted them without having to spend $3049 on TextMate [1]

This made me cranky

Solution 1: Learn Emacs. Setting up Emacs with all the modules you’ll need is a non-trival exercise. That is, put aside a day or so and don’t expect to be finished when time runs out. Plus, a version of GNU Emacs for Windows hasn’t been released since 2005. Not a lot of OSS love there.

Solution 2: Offer to build Textmate for Windows. I’m sure I read once that the Textmate author was open to porting of Textmate to Windows. That must be the past, because their website claims very strongly that there will never be a port:

countless requests for both a Windows and Linux port, but TextMate remains exclusive for the Mac, and that is how we like it!

That’s just lazy if you ask me.

Solution 3: Port Textmate’s snippet libraries (bundles) over to RadRails. Textmate has a more powerful syntax for its snippets than RadRails has for its templates (a limitation inherited from Eclipse), but SUCCESS! On the 31st of October – the day when Textmate was lauded around the blogosphere for its Halloween theme, I ported all their snippets into RadRails. OoooOOOOHHH spooooky.

You can now access the latest and greatest in templates/snippets for Radrails, for Ruby/Rails and RHTML (or click on the yellow boxes at the top).

Installation

  1. Download the two template XML files.
  2. Open RadRails
  3. Go to Windows > Preferences > Ruby > Editor > Templates, to see the list of included templates
  4. Click on any template, and press Ctrl-A to select them all.
  5. Click on Remove, to delete them all
  6. Click Import…, select the file ruby-rr-templates.xml
  7. Change to Rails > Editiors > RHTML Editor > RHTML Templates
  8. Remove all the current templates
  9. Click Import…, select the file rhtml-rr-templates.xml
  10. Press Ok to close the dialog and you’re done

Too difficult? The next release of RadRails will include these new templates.

How to use templates/snippets?

You can peruse a textmate-snippets-cheat-sheet, but really, spend 20 minutes looking through the template lists in RadRails. The name of the template is the text you type into the editor to activate the template.

So, the name of assert_equals is ase. So, type ase and press CTRL-SPACE, and a list of options pops up. Press return on the selected option.

Want to see all available assert_* templates? Type as and CTRL-SPACE and there they are. Nifty.

Patching/Updating

The RadRails guys are going to host these templates to allow anonymous access to make submitting patches relatively easy (if you know how to wield SVN). In future perhaps there will be a nice website for collaboratively sharing and downloading updates to templates.

Templates for JavaScript and CSS

I’ve put in a request for adding template support for JS and CSS as I have ported the JavaScript+CSS bundles as well. We just need a UI to import them into RadRails.

[1] TextMate is 39 euros = $49 at the moment. Plus $3000 for a new Mac.