Dr Nic

Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”

At this point in time there’s no facility in Rails to talk to more than one database at a time.

Alex Payne

I possibly have such a facility. Perhaps it will help, and I will get some DHH-love and perhaps a free Twitter account for my troubles. Or perhaps a t-shirt.

As a bonus, the solution even includes decent Ruby-fu syntax. So, if you’re just here for the view:

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)

That code just there solves all our problems. It will invoke Person.find(:all) on a random database connection to (assumably) a clone database. Awesomeness I think. I hope it helps Twitter and all the Twit-sers (or whatever you call a user of Twitter).

This solution comes from the magic_multi_connections gem.

What is going on here?

I think a tutorial is the best way to demonstrate what is happening here. So, let's create a rails app and mix in the magic_multi_connections gem.

First, get the gem. Second, create a rails app:

$ sudo gem install magic_multi_connections
$ rails multi -d sqlite3

Now edit the config/database.yml file to create some more databases:

  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000

  adapter: sqlite3
  database: db/development_clone1.sqlite3
  timeout: 5000

  adapter: sqlite3
  database: db/development_clone2.sqlite3
  timeout: 5000

But please pretend these are uber-MySQL clusters or whatever.

Think of :development as the read-write connection, and the :development_cloneN connections are for read-only access.

At the bottom of your environment.rb file, add the following:

require 'magic_multi_connections'
connection_names = ActiveRecord::Base.configurations.keys.select do |name|
  name =~ /^#{ENV['RAILS_ENV']}_clone/
@@connection_pool = connection_names.map do |connection_name|
  Object.class_eval <<-EOS
    module #{connection_name.camelize}
      establish_connection :#{connection_name}

Let's test what this gives us in the console:

$ ruby script/console
>> @@connection_pool
=> [DevelopmentClone1, DevelopmentClone2]
>> DevelopmentClone1.class
=> Module
>> DevelopmentClone1.connection_spec
=> :development_clone1

Our new modules will act as connections. One module per connection. The code above gives them names to match the connection names, but its really irrelevant what they are called, thanks to the mysterious conn method.

So, go create some models and some data. I'll use Person as the class here.

To setup the schemas in our clone databases, we'll use rake db:migrate. To do this:

$ cp config/environments/development.rb config/environments/development_clone1.rb
$ cp config/environments/development.rb config/environments/development_clone2.rb
$ rake db:migrate RAILS_ENV=development
$ rake db:migrate RAILS_ENV=development_clone1
$ rake db:migrate RAILS_ENV=development_clone2

To differentiate the databases in our example, assume there are two Person records in the :development database, and none in the two clones. Of course, in real-life, they are clones. You'd have a replicate mechanism in there somewhere.

Now, we can access our normal Rails modules through our connection modules. Magically of course.

>> ActiveRecord::Base.active_connections.keys
=> []
>> Person.count
=> 2
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base"]
>> DevelopmentClone1::Person.count
=> 0
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base", "DevelopmentClone1::Person"]

Wowzers. Person and DevelopmentClone1::Person classes?

But note - Person.count => 2 and DevelopmentClone1::Person.count => 0 - they are accessing different databases. The same class definition Person is being used for multiple database connections. We never defined more Person classes. Just the standard default one in app/models/person.rb.

The active_connections result shows that DevelopmentClone1::Person has its own connection. Yet you never had to manually call DevelopmentClone1::Person.establish_connection :development_clone1 - it was called automatically when the class is created.

Of course, DevelopmentClone2::Person is automatically connected to :development_clone2, and so on.

Behind the scenes

Let's look at our generated classes:

$ ruby script/console
>> DevelopmentClone1::Person
=> DevelopmentClone1::Person
>> Person
=> Person
>> DevelopmentClone1::Person.superclass
=> Person

That is, there is a DevelopmentClone1::Person class, automagically generated for you, which is a subclass of Person, so it has all its behaviour etc.

Dynamic connection pools within Rails controllers

The magic of the conn method will now be revealed:

$ ruby script/console
>> def conn
>>   @@connection_pool[rand(@@connection_pool.size)]
>> end
>> conn::Person.name
=> "DevelopmentClone2::Person"
>> conn::Person.name
=> "DevelopmentClone1::Person"

The conn method randomly returns one of the connection modules. Subsequently, conn::Person returns a Person class that is connected to a random clone database. Booya. Free Twitter swag coming my way.

Place the conn method in the ApplicationController class, and you can get dynamic connection pooling within Rails actions as needed, as in the following example (from the top of the article):

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)

Decent Ruby-fu, I think. Certainly better than manually calling establish_connection on model classes before each call (or in a before_filter call, I guess).

This is just a concept

I know this tutorial above works. But that might be the extent of what I know. Let me know if this has any quirks (especially if you solve them), or if this is a stupid idea for implementing connection pooling with nice Ruby syntax.

Hope it helps.

Spring Collection – the Modular Magic Models

Dr Nic’s Magic Models are like cheating on your taxes but without jail time. Its like filling out your tax return, and the tax office saying, “No, no, you keep your money – this year’s on us.”

Its like coming home and dinner is already cooked for you.

Its like being good looking and funny.

But this is Ruby, so we call it Magic!

With the release of 0.9.1, you can configure how much cheating, lying and thievery – err, magic – you want.

Already got model classes, but you haven’t gotten around to writing validations for some of them?

class Person < ActiveRecord::Base

As always, if you don't have a Person class, you'll get it for free plus validations.

As always, with automatically created classes AND pre-existing classes, if your schema supports an association and you ask for it it will be there. If you don't want a certain association to exist in your application... don't ask for it. The Magic Models are no place for enforcing law and order.

The other publicly announced, above board, for-sale to the general public feature you might like to know about is Magic Modules.

You can now use modules to specify common features of magic models. Currently supported is table name prefixes. If you need anything else, ask.

module Admin
  magic_module :prefix_table_name => 'admin_'

Admin::Permission.table_name # => 'admin_permissions'

The upcoming Magic Multi-Connections use this same concept to allow modules to specify different database connections instead of using superclasses. All this and more in the Magic Models: Spring collection series.

The end.

Ok, there's more. Secret stuff. Backdoor hooks into a Harry Potter world of mystery. Ways to execute code, import modules, and what-not upon newly created magic models. Awesome magicalness.

But you'll need to read the code to find them, and have some imagination as to what you'd want to dynamically do to your generated classes.

But as an example to prod your brain. Our schema at work supports date-ranging on many tables. The table will have a primary id key, plus another primary key - effective_start_date. All these tables end with the suffix '_history'. So when a new model class is created, I use the hooks to test the table name suffix, and if table_name =~ /_history$/ then I include a module that adds more methods etc to these date-ranged classes.

Magic Models: the Spring collection

Over the last few months, we at Dr Nic Magic Models (Sweden) Inc. have been gradually selling down all the company shares to pension funds at over-inflated prices, whilst at the same time refusing to release new patches for old projects that sorely need them – Composite Primary Keys and Dr Nic’s Magic Models.

Magic Models 0.8.0 exhibited range of wacky behaviour and I’ve been suggesting people 0.7.2 for a long time, instead of releasing 0.9.0, which I mostly wrote last year.

I received a huge patch for Composite Primary Keys in my inbox in February and despite the emails of “when will this patch be released” by the punters, I sat on it.

I’ve also mentioned in emails several new projects that I’ve either never released or they are lurking on RubyForge without any celebratory fanfare. For example, the Magic Model Generator – got 200 tables in a legacy schema? Generate 200 model files in 20 seconds.

The share price has now plummeted, and over Easter I bought back all the shares at half price. All that’s left to do now is unveil a barrage of new projects and releases and watch the share price soar. Booya.

So, over the next week or so, you will be treated to:

Magic Models by Dr Nic - Spring

So call the babysitter, grab yourself a champagne, and find a seat in front of the catwalk: its show time.

Meta-Magic in Ruby: Dr Nic Unplugged in Stockholm

Last nights’ Ruby meeting in Stockholm had a great turn out and starred Ola Bini sharing the latest and greatest about JRuby, and myself giving an overview on the wonders of Meta-Magic in Ruby.

I’ll write a separate post on Ola’s presentation shortly. It was awesome and I videoed it. Hehehe.

But first and foremost, lets talk about me. Or rather, let’s talk about my talk, which was also videoed.

Meta-magic in a programming language is as important to programmers as changeable ring tones are to teenagers. Authors of programming languages cannot provide every feature to everyone, so it is so wonderful to be able to add new language features and extensions that you want. Everyone knows you can add Jessica Simpson as your mobile ring tone, but not all programmers know that you can add new features to their programming world.

So here is an overview to a new world of happiness. It also overviews how the Magic Models work, and introduces a new gem I’m working on – the Magic Wiggly Lines – described as “genius or insane

Sentences with links

I have suggested I’d release 0.9 many times on Google Groups.

You see sentences like this in many many blogs. I have empathy for the authors who diligently markup each word with a different URL, but it is a wonderful way to describe and hyperlink in a meaningful way.

In the above fabricated, never-before-published quote, all the links are from a Google Groups search. So I knew I’d want to auto-generate it, because the mere thought of manual labour drives me to automation.

This resulted in the new sentence_with_links gem (I like bundling small chunks of code in new gems instead of stand alone library files so that the README file and test cases are encapsulated with the library code).


> gem install sentence_with_links


That is, String now has a with_links method, that generates the HTML as above. What if there are more links than words in the sentence? See the “Jessica Simpson” image search example below.

Mashup with Google Groups

For the above demonstration, let’s use our handy Hpricot parser to scrape the GoogleGroups search results, and .

Mashup with Google Images using JSON API

Demo: There are many Jessica Simpson images on Google
(also here, here, here, here, here, here, here, here, here, here, here, and here).

Note how the with_links function handles more links than sentence words? It appends a multitude of ‘here’ links. If you can think of more pleasurable alternatives, let me know. Either way, we don’t want to miss out on any important Jessica Simpson images. They are all there.

Instead of HTML, let’s parse some JSON returned by Google’s alter-ego search interface SearchMash, which provides a JSON API.

First, get the JSON Ruby parser:

gem install json

Then slap it all together…

And there you have it, sentences with links, the lazy Ruby way.