Dr Nic

MagicCGI shows OpenID user count

OpenID count

In the last 20 days, 43 people have used OpenID to leave comments. That’s very cool.

Corollary: add OpenID login to your blog.

Even cooler – Dynamic counter

The screen shot comes from the Comments form.

If you look at the comments form in a week, month, year, the counter above have be changed from its original value 43.

No fancy WordPress plugins (I don’t do PHP)

No Apache tricks. (I don’t know any Apache tricks)

Instead with Javascript/HTML attached to JSON attached to a RubyCGI script attached to my WordPress database via ActiveRecords and some magic.

I call it MagicCGI. I also call it Frigging Scary.

Try the following:

$ curl -v "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_openid_identities&action=count&format=json"
< Content-Type: txt/json
<
43

Or some XML?

$ curl -v "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_openid_identities&action=count&format=xml"
< Content-Type: txt/xml
<
<?xml version="1.0" encoding="UTF-8"?>
<result>
  <count type="integer">43</count>
</result>

Raw data?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_posts&field=post_title&limit=5&order=post_title"
[{"attributes": {"post_title": "OpenID count"}},
 {"attributes": {"post_title": "MagicCGI shows OpenID user count"}},
 {"attributes": {"post_title": "map_by_method now works with ActiveRecord associations"}},
 {"attributes": {"post_title": "Feedburner"}},
 {"attributes": {"post_title": "One year on the InterTubes"}}
]

Getting kinda scary now, I think.

Ooh, just how much magic?

Want a list of available tables to play with?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?meta=tables"
[{table_name: 'wp_users'},...]
  • add &format=xml to XML output; JSON is default
  • add &meta=columns to include the column schema definitions
  • add &table_name=wp_posts for each table you want (instead of all the tables)

E.g. to see the columns for wp_posts and no other table, in XML:

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?meta=tables&meta=columns&table_name=wp_posts&format=xml"

A list of all urls and internal user_ids for users/OpenID users/registered commenters?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_users&field=user_url&field=id"
[{"attributes": {"id": "1", "user_url": "http://drnicwilliams.com"}}, ...

So, now we know Dr Nic = user id 1.

Oh oh oh, how about a list of comments for a specific user?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&user_id=1"
...comments by Dr Nic...
or
$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&user_id=1&action=count"
232

Dr Nic’s commented in his own blog 232 times? Out of how many comments?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&action=count"
963

Where are the user emails? Where are their passwords?

Hidden! See below.

Callbacks?

JSON URLs support callback=someCallbackMethod and/or variable=someLocalVariable.

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&action=count&callback=someMethod"
someMethod(963);

What does wp_drnicwilliams.cgi look like?

Something a little like this…

#!/usr/local/bin/ruby
require 'magic_cgi'         # loads the render magic, model magic, and meta-model magic
include Render

require 'magic_cgi/config/wordpress'  # connect to DB using WordPress installation (wp-config.php)
MagicCGI::Config::WordPress.establish_connection "/path/to/drnicwilliams/web/public"

# The following is defaulted for WordPress connections:
MagicCGI::Config.hidden_tables |= %w[wp_openid_nonces wp_openid_associations wp_usermeta wp_tla_rss_map wp_tla_data]
MagicCGI::Config.hidden_columns['wp_users'] = %w[user_email user_pass user_activation_key]
MagicCGI::Config.hidden_columns['wp_openid_identities'] = %w[hash]
MagicCGI::Config.hidden_columns['wp_comments'] = %w[comment_author_email comment_author_IP]
MagicCGI::Config.hidden_columns['wp_posts'] = %w[post_password]

render do |params|
  data = DbTable.from_params(params)
  data ||= begin
    table_name = params['table'].first || 'wp_users'
    model_name = ActiveRecord::Base.class_name(table_name)
    klass = MagicCGI::MagicModel.create_class(model_name, ActiveRecord::Base)
    klass.find_or_count_by_params(params)
  end
  data
end

Can I write my own CGI scripts for my own DBs?

Sure. Its a library called MagicCGI.

What is MagicCGI?

Coming soon.

map_by_method now works with ActiveRecord associations

I was always annoyed that map_by_method was broken for ActiveRecord has_many associations. 6 mths later I finally fixed it.

That’s the magic of Open Source Software. [/end sarcasm]

So now, the following example works like it should:

$ gem install map_by_method
$ console
> require 'map_by_method'  # stick this in your environment.rb for Rails
> user = User.find_by_name "Dr Nic"
> user.companies.map_by_name
=> ['Dr Nic Academy', 'Dr Nic Institute of Being Silly']
> user.companies.map_by_id_and_name
=> [[1, 'Dr Nic Academy'], [9, 'Dr Nic Institute of Being Silly']]

Recap: why use map_by_method?

Try the following example:

> user.companies.map_by_employees.flatten
=> list of all employees of user

Versus:

> user.companies.map { |company| company.employees}.flatten
or
> user.companies.map(&:employees).flatten

Or compare:

> user.companies.map_by_id_and_name
=> [[1, 'Dr Nic Academy'], [9, 'Dr Nic Institute of Being Silly']]

Versus:

> user.companies.map { |company| [company.id, company.name]}

That is, it looks and feels just like ActiveRecord’s #find method, with its find_by_first_name_and_last_name magic.

Summary

No {, }, |, &, or : required. Just clean method names.

Bonus other gem

In the spirit of ActiveRecord hacks, there is to_activerecord:

$ gem install to_activerecord
$ console
> require 'to_activerecord'  # stick this in your environment.rb for Rails
> [1,2,3].to_user
=> [list of User with id's 1,2,3]

To me, this suffix operator reads cleaner than the traditional:

> User.find([1,2,3])

For example, if you want to perform an operation on the list of Users:

> ids = [1,2,3]
> ids.to_user.map_by_name
=> ['Dr Nic', 'Banjo', 'Nancy']

Versus:

> User.find(ids).map_by_name

Magic Wiggly Lines => GuessMethod, by Chris Shea

If you ever make time to code just for pleasure, then method_missing and const_missing are just begging for abuse.

Chris Shea has come up with GuessMethod – a very cool hack that now deprecates my concept of Magic Wiggly Linesa spell-checker for runtime code.

What’s it do? Cop a squiz at this genius…

$ gem install guessmethod -y
$ irb
> require 'rubygems'
> require 'guessmethod'
> class Object; include GuessMethod; end  # though this could go in the guessmethod.rb file in the gem
> class Product; def name; "Some product"; end; end
> Prodct.nw.nae
attention: replacing non-existant constant Prodct with Product for Object
attention: sending new instead of nw to Product:Class
attention: sending name instead of nae to #<Product:0x144ff10>:Product
=> "Some product"

That’s going straight into my .irbrc file. My bad spelling, coupled with my British/Australian English, will never slow me down again!

UPDATE: actually, it doesn’t like being in the .irbrc file for Rails console; so in the config/environments/development.rb files will have to do for the moment.

Auto-completer for my blog comments

It took 4 hours to return from the town of Strängnäs to Stockholm via train. Normally, its 40 minutes. The train never turned up. 4 * 60 – 40 = 200 minutes of non-travel.

Stuck in a train station for 200 minutes I tackled a problem that has irked me for some time.

In my blog comments, when I reply to someone else’s comment I’ll normally use the syntax: “@chris – thanks for the kind feedback, I’ll send the money via paypal“.

And then Chris says “@Dr Nic – yeah, send it to the usual account; the tax man will never find it“.

And then I say “@chris – no problem“.

Its not that “@chris” is difficult to spell, but I’ve used IRC and Gmail long enough that I like auto-completion. The absence of auto-completion in Skype irks me too.

What I really want is:

“@c” + TAB expands to “@chris – “

So that’s what I now have. And its neat.

Demo of auto-completer

Unfortunately, the first person to leave a comment below won’t have anyone to reply to, and so theoretically wouldn’t get to experience the joys of the feature, that is so awesome and sexy that my very own wife called:

“That’s nice dear, now help with dinner”.

So, by default “Nic”, “Dr Nic”, “drnic” are available. The 2nd commenter will get these + the name of the 1st commenter. The 3rd commenter will get… hehe, you thought I was going to type that out.

Here’s what you’ll see when you save your comments:

Testing Autocompleter

Public release

Its currently implemented for WordPress, using Prototype/Scriptaculous. Well it works on my WordPress theme. It figures out the auto-completion list from the HTML, not from the blog/forum server.

I’m not sure how different everyone’s HTML is for their comment blocks. That makes it tricky to release the conde at the moment for general consumption. You might be an HTML guru and make it work, but your sister isn’t.

So, can you please comment below (wink wink) with the name of your blog software, and include your blog url in the URL field. I might be asking for grief here, but I’ll scope out everyone’s blog comment HTML and see how different/similar they are.

BUT!…

If someone in the comments has already mentioned your blog software (WordPress, Mephisto, etc) then you MUST reply to their comment. If you are first to comment with your blog software, then use “@Dr Nic” instead.

This is life-or-death critical as it gives you an excuse to play with the auto-completer thingy. Lives are at stake here people!

Don’t have a blog, but want to comment anyway to play with it in all its snazziness? Go for gold.

Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”

At this point in time there’s no facility in Rails to talk to more than one database at a time.

Alex Payne

I possibly have such a facility. Perhaps it will help, and I will get some DHH-love and perhaps a free Twitter account for my troubles. Or perhaps a t-shirt.

As a bonus, the solution even includes decent Ruby-fu syntax. So, if you’re just here for the view:

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

That code just there solves all our problems. It will invoke Person.find(:all) on a random database connection to (assumably) a clone database. Awesomeness I think. I hope it helps Twitter and all the Twit-sers (or whatever you call a user of Twitter).

This solution comes from the magic_multi_connections gem.

What is going on here?

I think a tutorial is the best way to demonstrate what is happening here. So, let's create a rails app and mix in the magic_multi_connections gem.

First, get the gem. Second, create a rails app:

$ sudo gem install magic_multi_connections
$ rails multi -d sqlite3

Now edit the config/database.yml file to create some more databases:

development:
  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000

development_clone1:
  adapter: sqlite3
  database: db/development_clone1.sqlite3
  timeout: 5000

development_clone2:
  adapter: sqlite3
  database: db/development_clone2.sqlite3
  timeout: 5000

But please pretend these are uber-MySQL clusters or whatever.

Think of :development as the read-write connection, and the :development_cloneN connections are for read-only access.

At the bottom of your environment.rb file, add the following:

require 'magic_multi_connections'
connection_names = ActiveRecord::Base.configurations.keys.select do |name|
  name =~ /^#{ENV['RAILS_ENV']}_clone/
end
@@connection_pool = connection_names.map do |connection_name|
  Object.class_eval <<-EOS
    module #{connection_name.camelize}
      establish_connection :#{connection_name}
    end
  EOS
  connection_name.camelize.constantize
end

Let's test what this gives us in the console:

$ ruby script/console
>> @@connection_pool
=> [DevelopmentClone1, DevelopmentClone2]
>> DevelopmentClone1.class
=> Module
>> DevelopmentClone1.connection_spec
=> :development_clone1

Our new modules will act as connections. One module per connection. The code above gives them names to match the connection names, but its really irrelevant what they are called, thanks to the mysterious conn method.

So, go create some models and some data. I'll use Person as the class here.

To setup the schemas in our clone databases, we'll use rake db:migrate. To do this:

$ cp config/environments/development.rb config/environments/development_clone1.rb
$ cp config/environments/development.rb config/environments/development_clone2.rb
$ rake db:migrate RAILS_ENV=development
$ rake db:migrate RAILS_ENV=development_clone1
$ rake db:migrate RAILS_ENV=development_clone2

To differentiate the databases in our example, assume there are two Person records in the :development database, and none in the two clones. Of course, in real-life, they are clones. You'd have a replicate mechanism in there somewhere.

Now, we can access our normal Rails modules through our connection modules. Magically of course.

>> ActiveRecord::Base.active_connections.keys
=> []
>> Person.count
=> 2
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base"]
>> DevelopmentClone1::Person.count
=> 0
>> ActiveRecord::Base.active_connections.keys
=> ["ActiveRecord::Base", "DevelopmentClone1::Person"]

Wowzers. Person and DevelopmentClone1::Person classes?

But note - Person.count => 2 and DevelopmentClone1::Person.count => 0 - they are accessing different databases. The same class definition Person is being used for multiple database connections. We never defined more Person classes. Just the standard default one in app/models/person.rb.

The active_connections result shows that DevelopmentClone1::Person has its own connection. Yet you never had to manually call DevelopmentClone1::Person.establish_connection :development_clone1 - it was called automatically when the class is created.

Of course, DevelopmentClone2::Person is automatically connected to :development_clone2, and so on.

Behind the scenes

Let's look at our generated classes:

$ ruby script/console
>> DevelopmentClone1::Person
=> DevelopmentClone1::Person
>> Person
=> Person
>> DevelopmentClone1::Person.superclass
=> Person

That is, there is a DevelopmentClone1::Person class, automagically generated for you, which is a subclass of Person, so it has all its behaviour etc.

Dynamic connection pools within Rails controllers

The magic of the conn method will now be revealed:

$ ruby script/console
>> def conn
>>   @@connection_pool[rand(@@connection_pool.size)]
>> end
>> conn::Person.name
=> "DevelopmentClone2::Person"
>> conn::Person.name
=> "DevelopmentClone1::Person"

The conn method randomly returns one of the connection modules. Subsequently, conn::Person returns a Person class that is connected to a random clone database. Booya. Free Twitter swag coming my way.

Place the conn method in the ApplicationController class, and you can get dynamic connection pooling within Rails actions as needed, as in the following example (from the top of the article):

class PeopleController < ApplicationController
  def index
    @people = conn::Person.find(:all)
  end
end

Decent Ruby-fu, I think. Certainly better than manually calling establish_connection on model classes before each call (or in a before_filter call, I guess).

This is just a concept

I know this tutorial above works. But that might be the extent of what I know. Let me know if this has any quirks (especially if you solve them), or if this is a stupid idea for implementing connection pooling with nice Ruby syntax.

Hope it helps.