Dr Nic

Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”

At this point in time there’s no facility in Rails to talk to more than one database at a time.

Alex Payne

I possibly have such a facility. Perhaps it will help, and I will get some DHH-love and perhaps a free Twitter account for my troubles. Or perhaps a t-shirt.

As a bonus, the solution even includes decent Ruby-fu syntax. So, if you’re just here for the view:

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)

That code just there solves all our problems. It will invoke Person.find(:all) on a random database connection to (assumably) a clone database. Awesomeness I think. I hope it helps Twitter and all the Twit-sers (or whatever you call a user of Twitter).

This solution comes from the magic_multi_connections gem.

What is going on here?

I think a tutorial is the best way to demonstrate what is happening here. So, let's create a rails app and mix in the magic_multi_connections gem.

First, get the gem. Second, create a rails app:

$ sudo gem install magic_multi_connections
$ rails multi -d sqlite3

Now edit the config/database.yml file to create some more databases:

  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000

  adapter: sqlite3
  database: db/development_clone1.sqlite3
  timeout: 5000

  adapter: sqlite3
  database: db/development_clone2.sqlite3
  timeout: 5000

But please pretend these are uber-MySQL clusters or whatever.

Think of :development as the read-write connection, and the :development_cloneN connections are for read-only access.

At the bottom of your environment.rb file, add the following:

require 'magic_multi_connections'
connection_names = ActiveRecord::Base.configurations.keys.select do |name|
  name =~ /^#{ENV['RAILS_ENV']}_clone/
@@connection_pool = connection_names.map do |connection_name|
  Object.class_eval <<-EOS
    module #{connection_name.camelize}
      establish_connection :#{connection_name}

Let's test what this gives us in the console:

$ ruby script/console
>> @@connection_pool
=> [DevelopmentClone1, DevelopmentClone2]
>> DevelopmentClone1.class
=> Module
>> DevelopmentClone1.connection_spec
=> :development_clone1

Our new modules will act as connections. One module per connection. The code above gives them names to match the connection names, but its really irrelevant what they are called, thanks to the mysterious conn method.

So, go create some models and some data. I'll use Person as the class here.

To setup the schemas in our clone databases, we'll use rake db:migrate. To do this:

$ cp config/environments/development.rb config/environments/development_clone1.rb
$ cp config/environments/development.rb config/environments/development_clone2.rb
$ rake db:migrate RAILS_ENV=development
$ rake db:migrate RAILS_ENV=development_clone1
$ rake db:migrate RAILS_ENV=development_clone2

To differentiate the databases in our example, assume there are two Person records in the :development database, and none in the two clones. Of course, in real-life, they are clones. You'd have a replicate mechanism in there somewhere.

Now, we can access our normal Rails modules through our connection modules. Magically of course.

>> ActiveRecord::Base.active_connections.keys
=> []
>> Person.count
=> 2
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base"]
>> DevelopmentClone1::Person.count
=> 0
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base", "DevelopmentClone1::Person"]

Wowzers. Person and DevelopmentClone1::Person classes?

But note - Person.count => 2 and DevelopmentClone1::Person.count => 0 - they are accessing different databases. The same class definition Person is being used for multiple database connections. We never defined more Person classes. Just the standard default one in app/models/person.rb.

The active_connections result shows that DevelopmentClone1::Person has its own connection. Yet you never had to manually call DevelopmentClone1::Person.establish_connection :development_clone1 - it was called automatically when the class is created.

Of course, DevelopmentClone2::Person is automatically connected to :development_clone2, and so on.

Behind the scenes

Let's look at our generated classes:

$ ruby script/console
>> DevelopmentClone1::Person
=> DevelopmentClone1::Person
>> Person
=> Person
>> DevelopmentClone1::Person.superclass
=> Person

That is, there is a DevelopmentClone1::Person class, automagically generated for you, which is a subclass of Person, so it has all its behaviour etc.

Dynamic connection pools within Rails controllers

The magic of the conn method will now be revealed:

$ ruby script/console
>> def conn
>>   @@connection_pool[rand(@@connection_pool.size)]
>> end
>> conn::Person.name
=> "DevelopmentClone2::Person"
>> conn::Person.name
=> "DevelopmentClone1::Person"

The conn method randomly returns one of the connection modules. Subsequently, conn::Person returns a Person class that is connected to a random clone database. Booya. Free Twitter swag coming my way.

Place the conn method in the ApplicationController class, and you can get dynamic connection pooling within Rails actions as needed, as in the following example (from the top of the article):

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)

Decent Ruby-fu, I think. Certainly better than manually calling establish_connection on model classes before each call (or in a before_filter call, I guess).

This is just a concept

I know this tutorial above works. But that might be the extent of what I know. Let me know if this has any quirks (especially if you solve them), or if this is a stupid idea for implementing connection pooling with nice Ruby syntax.

Hope it helps.

Spring Collection – the Modular Magic Models

Dr Nic’s Magic Models are like cheating on your taxes but without jail time. Its like filling out your tax return, and the tax office saying, “No, no, you keep your money – this year’s on us.”

Its like coming home and dinner is already cooked for you.

Its like being good looking and funny.

But this is Ruby, so we call it Magic!

With the release of 0.9.1, you can configure how much cheating, lying and thievery – err, magic – you want.

Already got model classes, but you haven’t gotten around to writing validations for some of them?

class Person < ActiveRecord::Base

As always, if you don't have a Person class, you'll get it for free plus validations.

As always, with automatically created classes AND pre-existing classes, if your schema supports an association and you ask for it it will be there. If you don't want a certain association to exist in your application... don't ask for it. The Magic Models are no place for enforcing law and order.

The other publicly announced, above board, for-sale to the general public feature you might like to know about is Magic Modules.

You can now use modules to specify common features of magic models. Currently supported is table name prefixes. If you need anything else, ask.

module Admin
  magic_module :prefix_table_name => 'admin_'

Admin::Permission.table_name # => 'admin_permissions'

The upcoming Magic Multi-Connections use this same concept to allow modules to specify different database connections instead of using superclasses. All this and more in the Magic Models: Spring collection series.

The end.

Ok, there's more. Secret stuff. Backdoor hooks into a Harry Potter world of mystery. Ways to execute code, import modules, and what-not upon newly created magic models. Awesome magicalness.

But you'll need to read the code to find them, and have some imagination as to what you'd want to dynamically do to your generated classes.

But as an example to prod your brain. Our schema at work supports date-ranging on many tables. The table will have a primary id key, plus another primary key - effective_start_date. All these tables end with the suffix '_history'. So when a new model class is created, I use the hooks to test the table name suffix, and if table_name =~ /_history$/ then I include a module that adds more methods etc to these date-ranged classes.

Meta-Magic in Ruby: Dr Nic Unplugged in Stockholm

Last nights’ Ruby meeting in Stockholm had a great turn out and starred Ola Bini sharing the latest and greatest about JRuby, and myself giving an overview on the wonders of Meta-Magic in Ruby.

I’ll write a separate post on Ola’s presentation shortly. It was awesome and I videoed it. Hehehe.

But first and foremost, lets talk about me. Or rather, let’s talk about my talk, which was also videoed.

Meta-magic in a programming language is as important to programmers as changeable ring tones are to teenagers. Authors of programming languages cannot provide every feature to everyone, so it is so wonderful to be able to add new language features and extensions that you want. Everyone knows you can add Jessica Simpson as your mobile ring tone, but not all programmers know that you can add new features to their programming world.

So here is an overview to a new world of happiness. It also overviews how the Magic Models work, and introduces a new gem I’m working on – the Magic Wiggly Lines – described as “genius or insane

[ANN] Dr Nic’s Magic Models 0.8 – Validate Anything, Anytime, Anywhere

Ladies and Gentlemen, welcome one and all to the greatest Magic Show in Programming. If you bought free tickets to the last show, then you’ll be prepared to pay double for this show because its twice as good. Magic Models provide virtual validations on all database constraints!

In the first run of the sell-out Magic Models – “I can’t believe its an ActiveRecord” – Super Show – you were amazed as we stripped away the necessary essentials of an ActiveRecord – right in front of your eyes – the necessary became unnecessary.

We removed the validates_presence_of validations… and the Magic Model still validated.

We removed the has_many, belongs_to, and has_many :through associations… and the Magic Model still associated.

We removed the actual class definition itself – the very thing that provides the ActiveRecord CRUD layer… and the Magic Model still CRUDded.

You applauded loudly, and then walked out the front door surruptitiously ignoring the donation jar.

Today. Ladies and Gentlemen, I have removed the donation jar entirely so as to not distract your singular focus on the All New Magic Show!

Drums rolling in the background…

Let us start with a humble, unassuming database schema. Today’s schema syntax of choice? SQL.

CREATE TABLE `groups` (
  `id` int(11) NOT NULL auto_increment,
  `name` varchar(50) NOT NULL UNIQUE,
  `description` varchar(50) default NULL,
  `some_int` integer default NULL,
  `some_float` float default NULL,
  PRIMARY KEY  (`id`)
) TYPE=InnoDB;

The first change to note, is that all Magic Models and their associations are now being generated at startup time, and not dynamically.

>> ActiveRecord::Base.send :subclasses
=> [CGI::Session::ActiveRecordStore::Session, Group]

This is the current solution to supporting the :include option on find calls. Debate the pros/cons of this on the forum. It may be helpful for the various individuals trying to get Magic Models working with Streamlined.

Now… the magic!!! Watch carefully…

>> g = Group.new
>> g.name = "x" * 51
=> "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> g.valid?
=> false
>> g.errors.full_messages
=> ["Name is too long (maximum is 50 characters)"]

Crikey! That’s a beauty! That name is just too big for the database and boy does the validation automatically tell you that. Fantastic isn’t it!?

Ok, that’s enough Steve Irwin impressions.

>> g = Group.new
>> g.name = "x" * 50
=> "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> g.valid?
=> true


Ok, more tricks… more drumrolls…

>> g.some_int = 99.9
=> 99.9
>> g.some_float = "string"
=> "string"
>> g.valid?
=> false
>> g.errors.full_messages
=> ["Some float is not a number", "Some int is not a number"]


And now for the final Virtual Validation magic trick… I draw your attention to the SQL: the name field is UNIQUE. Could the magic models… could they… no… maybe? Oh maybe they could support validation over unique fields?!?!?!

100 dancing girls and guys in silver spandex appear on the stage. Of course, more drums are rolling…

>> g = Group.new
=> #<Group:0x3884eb8 @attributes={"name"=>"", ... >
>> g.name = "test"
=> "test"
>> g.save
=> true
>> g2 = Group.new
=> #<Group:0x3884eb8 @attributes={"name"=>"", ... >
>> g2.name = "test"
=> "test"
>> g2.save
=> false
>> g2.errors.full_messages
=> ["Name has already been taken"]

Dr Nic takes a bow and is escorted off by his 100 dancers and 3 large elephants that turned up late.


gem install dr_nic_magic_models

The at the top of your Ruby script or at the bottom of your Rails’ environment.rb file, place the line:

require 'dr_nic_magic_models'


Matthew Painter made this possible. He is the author, producer, and cheer leader for this release. Whilst I was fart-arsing around writing Dr Nic’s GrandCivs, and he was doing something more important – making the Magic Models the ultimate magic show in town. And he has a funky website with a draggable background.

The original ideas for the extended Virtual Validations came from Red Hill Consulting: though again it was Matthew who did the work to integrate it into the Magic Models.

The whole of Magic Models architecture has been refactored. Matthew did that.

My 2+ gig of Gmail account filled with diff files. Matthew did that. You get the picture. Thanks Matt!

Anything, Anytime, Anywhere

“Anything, Anytime, Anywhere” was the slogan of the Goodies – a British TV show that was rerun over and over again after school in Australia. Their fame amongst school kids in the 70s and 80s, gives them Rock God status amongst midaged Australians today. They toured Australia in 2005 with 13 sell-out shows. I didn’t get tickets. If you did get tickets, I don’t want to know about it because I won’t talk to you.

[Trick] Natural language DSL in Ruby

Ever tried to explain DSLs (domain specific languages) to someone, so that they know how wonderful Ruby is, and just run out of useful examples? Try this piece of Street Magic on them…

The effect

See if your unsuspecting friend can figure out what this code will do. If they can, then say, “That’s because you are a domain expert in English. Good for you. Now use Ruby you Java boy.”

>> I.say "I love the Ruby language"
You said, 'I love the Ruby language'

Cool, yes? Yes.

How this trick works

To set this up for your unsuspecting Ruby noob, type the following into your console/irb:

>> class I; def self.say(text); puts "You said, '#{text}'"; end; end
=> nil

Now you have a constant I upon which you can call class methods.

In long hand, this is:

class I
  def self.say(text)
    puts "You said, '#{text}'"

Create other pronoun classes, such as You, Everyone, etc and give them methods representing verbs, such as say, said, want_to.

For example,

class You
  def self.cannot(action, target)
    puts "Didn't your mother tell you never say never? Of course you can #{action} #{target}"

Which delights us with:

>> You.cannot :learn, "Ruby"
Didn't your mother tell you never say never? Of course you can learn Ruby

“I still don’t know what a DSL is”

That’s unfortunate for you.