Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”

At this point in time there’s no facility in Rails to talk to more than one database at a time.

Alex Payne

I possibly have such a facility. Perhaps it will help, and I will get some DHH-love and perhaps a free Twitter account for my troubles. Or perhaps a t-shirt.

As a bonus, the solution even includes decent Ruby-fu syntax. So, if you’re just here for the view:

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

That code just there solves all our problems. It will invoke Person.find(:all) on a random database connection to (assumably) a clone database. Awesomeness I think. I hope it helps Twitter and all the Twit-sers (or whatever you call a user of Twitter).

This solution comes from the magic_multi_connections gem.

What is going on here?

I think a tutorial is the best way to demonstrate what is happening here. So, let's create a rails app and mix in the magic_multi_connections gem.

First, get the gem. Second, create a rails app:

$ sudo gem install magic_multi_connections
$ rails multi -d sqlite3

Now edit the config/database.yml file to create some more databases:

development:
  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000

development_clone1:
  adapter: sqlite3
  database: db/development_clone1.sqlite3
  timeout: 5000

development_clone2:
  adapter: sqlite3
  database: db/development_clone2.sqlite3
  timeout: 5000

But please pretend these are uber-MySQL clusters or whatever.

Think of :development as the read-write connection, and the :development_cloneN connections are for read-only access.

At the bottom of your environment.rb file, add the following:

require 'magic_multi_connections'
connection_names = ActiveRecord::Base.configurations.keys.select do |name|
  name =~ /^#{ENV['RAILS_ENV']}_clone/
end
@@connection_pool = connection_names.map do |connection_name|
  Object.class_eval <<-EOS
    module #{connection_name.camelize}
      establish_connection :#{connection_name}
    end
  EOS
  connection_name.camelize.constantize
end

Let's test what this gives us in the console:

$ ruby script/console
>> @@connection_pool
=> [DevelopmentClone1, DevelopmentClone2]
>> DevelopmentClone1.class
=> Module
>> DevelopmentClone1.connection_spec
=> :development_clone1

Our new modules will act as connections. One module per connection. The code above gives them names to match the connection names, but its really irrelevant what they are called, thanks to the mysterious conn method.

So, go create some models and some data. I'll use Person as the class here.

To setup the schemas in our clone databases, we'll use rake db:migrate. To do this:

$ cp config/environments/development.rb config/environments/development_clone1.rb
$ cp config/environments/development.rb config/environments/development_clone2.rb
$ rake db:migrate RAILS_ENV=development
$ rake db:migrate RAILS_ENV=development_clone1
$ rake db:migrate RAILS_ENV=development_clone2

To differentiate the databases in our example, assume there are two Person records in the :development database, and none in the two clones. Of course, in real-life, they are clones. You'd have a replicate mechanism in there somewhere.

Now, we can access our normal Rails modules through our connection modules. Magically of course.

>> ActiveRecord::Base.active_connections.keys
=> []
>> Person.count
=> 2
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base"]
>> DevelopmentClone1::Person.count
=> 0
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base", "DevelopmentClone1::Person"]

Wowzers. Person and DevelopmentClone1::Person classes?

But note - Person.count => 2 and DevelopmentClone1::Person.count => 0 - they are accessing different databases. The same class definition Person is being used for multiple database connections. We never defined more Person classes. Just the standard default one in app/models/person.rb.

The active_connections result shows that DevelopmentClone1::Person has its own connection. Yet you never had to manually call DevelopmentClone1::Person.establish_connection :development_clone1 - it was called automatically when the class is created.

Of course, DevelopmentClone2::Person is automatically connected to :development_clone2, and so on.

Behind the scenes

Let's look at our generated classes:

$ ruby script/console
>> DevelopmentClone1::Person
=> DevelopmentClone1::Person
>> Person
=> Person
>> DevelopmentClone1::Person.superclass
=> Person

That is, there is a DevelopmentClone1::Person class, automagically generated for you, which is a subclass of Person, so it has all its behaviour etc.

Dynamic connection pools within Rails controllers

The magic of the conn method will now be revealed:

$ ruby script/console
>> def conn
>>   @@connection_pool[rand(@@connection_pool.size)]
>> end
>> conn::Person.name
=> "DevelopmentClone2::Person"
>> conn::Person.name
=> "DevelopmentClone1::Person"

The conn method randomly returns one of the connection modules. Subsequently, conn::Person returns a Person class that is connected to a random clone database. Booya. Free Twitter swag coming my way.

Place the conn method in the ApplicationController class, and you can get dynamic connection pooling within Rails actions as needed, as in the following example (from the top of the article):

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

Decent Ruby-fu, I think. Certainly better than manually calling establish_connection on model classes before each call (or in a before_filter call, I guess).

This is just a concept

I know this tutorial above works. But that might be the extent of what I know. Let me know if this has any quirks (especially if you solve them), or if this is a stupid idea for implementing connection pooling with nice Ruby syntax.

Hope it helps.

Related posts:

1. Using CoffeeScript in Rails and even on Heroku I’m pretty excited about CoffeeScript as a clean-syntax replacement for...
2. First look at rails 3.0.pre This article is out of date in some aspects....
3. Rails themes can remember things I was getting annoyed at having to remember all the...
4. Install any HTML theme/template into your Rails app Have you ever even bothered to Google for “rails...
5. Closing in on The Dream: “one-click-to-deploy Rails apps” Got a simple app you want to build? Allocate...