Dr Nic

Previously, I mentioned a multi-step/multi-project solution to doing JavaScript Unit Testing for Rails, including an autotest script to make TDD life easier for the autotest-addicted.

It was too many steps, too many different projects, and too much dependence on prototypejs. So let’s fix this via the spanky new newjs (introduced here):

Instead of several steps, there is now two steps:

cd myrailsapp                # rails 2.0 app
sudo gem install newjs       # v1.3.1+
script/generate javascript_test application
    create  vendor/plugins/javascript_unittest/lib
    create  vendor/plugins/javascript_unittest/tasks
    create  test/javascript/assets/jsunittest.js
    create  test/javascript/assets/unittest.css
    create  config/javascript_test_autotest.yml.sample
    create  vendor/plugins/javascript_unittest/lib/jstest.rb
    create  vendor/plugins/javascript_unittest/tasks/runner.rake
    create  vendor/plugins/javascript_unittest/tasks/autotest.rake
    create  vendor/plugins/javascript_unittest/README
    create  test/javascript/application_test.html
    create  script/rstakeout
    create  script/js_autotest

It installs all assets, gives you a rake test:javascripts task, gives you script/js_autotest, and finally (and most importantly), creates the application_test.html test stub.

As always, you can also run a single test file by loading it into a browser.

Prototype independence

Previously, the test suite – unittest.js – had a dependency on prototypejs. This was wonderful if you’re developing in prototypejs, but could cause some grief if you weren’t.

newjs now comes packed with an identical test suite – jsunittest – but it is built without a dependency on prototypejs.

Merb et al support

I wrote a rails generator because I used Rails. If a merb/other framework (Ruby or non-Ruby) wants a similar generator, just ask. Its pretty simple – I copy over the rails_generators folder into a merb_generators folder in newjs and change some things based on your frameworks’ structure.

Alternate test suite support

If you like another test suite, and want it included in the generators, this might require some additional effort on your part. It probably has different mechanisms for launching runners etc; whereas I was able to steal the rake tests + support libraries from prototypejs. But, it can be done.

Peepcode

These tools will be discussed in animated detail in the forth coming Peepcode “JavaScript Unit Testing”.

Want to start a new JavaScript project for a library or widget/badge for your website?

You know you should do TDD but wouldn’t know how to get started, what support libraries you need and where to get them from?

You need tools to deploy your library, your website etc?

You’d like a consistent structure to your project so that IDEs could provide support (toggle btw src/person.js and test/unit/person_test.html) [I haven’t done this bit yet, I’m sure I’ll add it to the JavaScipt Unit testing textmate bundle one day soon].

I couldn’t find anything helpful like this, so I created newjs – the JavaScript Project Generator.

Why?

But first, an brief history of everything…

I fell instantly in love with Rails for a couple reasons:

• rails & script/generate commands – they teach you what files you need
  and where they should go
• Ajax support – the marketing phrase “its as easy to use Ajax as not to” took me
  away from ASP.NET where the new Ajax.NET library was non-trivial to use; RJS
  didn’t exist at this stage, but Rails’ JavaScript support was still awesome
• TextMate – though I didn’t get to use it for 18 mths til I bought a Mac

Later I fell in love with Ruby, for its meta-programming, syntactical possibilities
and free TDD support within Rails.

Then I fell out of love with JavaScript. Partly because Rails started generating JavaScript for me, and ultimately because I didn’t have test support. Whether you use Rails JavaScript helpers, or write your own unobtrusive JavaScript libraries you’re still living in an unhappy world without tests; let alone without TDD.

I still wrote JavaScript because it still needed writing. But a lot of my JavaScript
became “write-once, modify-never” affairs. Not just because I had no tests,
but each project had a different structure, different deployment processes, etc.

Finally, this year I figured out “TDD for JavaScript”. For every 1000 blog articles about Rails or Ruby, there is 0-1 article on unit testing JavaScript. Or perhaps I just don’t know how to use Google. Soon, the “JavaScript Unit Testing” PeepCode will be finished, so hopefully it will add to this lacking body of knowledge.

What I needed now was one-line starter-kit for new JavaScript projects that included:

• A standard structure for JavaScript libraries/projects
• In-built testing support, with rake tasks and script/generate unit_test
  generators
• Tasks for deploying distribution files (via version numbers)
• Tasks for managing the project website

I don’t think there is one already, thus my hand was forced: newjs.

Installation & Getting Started

Installation and maintenance of generated JavaScript projects requires the installation of Ruby and RubyGems.

The command-line application newjs is installed as below, for any operating system (except the ‘sudo’ part – use as necessary):

sudo gem install newjs

You’ll note it uses RubiGen because I like to cheat when it comes to generating things.

To kick-off your new project/library, run the command-line app newjs:

$ newjs mylib -a "Dr Nic" -e "drnicwilliams@gmail.com" -u "http://mylib.rubyforge.org"
      create  config
      create  lib
      create  src
      create  script
      create  tasks
      create  test/assets
      create  test/assets/unittest.css
      create  test/assets/unittest.js
      create  test/assets/prototype.js
      create  tasks/javascript_test_autotest_tasks.rake
      create  tasks/environment.rake
      create  tasks/deploy.rake
      create  config/javascript_test_autotest.yml.sample
      create  lib/protodoc.rb
      create  lib/jstest.rb
      create  Rakefile
      create  README.txt
      create  History.txt
      create  License.txt
      create  src/HEADER
      create  src/mylib.js
      create  script/rstakeout
      create  script/js_autotest
  dependency  install_rubigen_scripts
      exists    script
      create    script/generate
      create    script/destroy

And you thought you wanted to create all that by hand…

TDD for JavaScript

Personally, I can never remember what basic HTML + JavaScript goes in the test HTML files. I quite like the javascript_test plugin for Rails, which allows you to generate a test HTML stub. So I’ve included a version of it
here.

Two actually: one for unit tests and one for functional tests.

$ script/generate unit_test some_lib
  create  test/unit
  create  test/unit/some_lib_test.html

For functional tests, use functional_test generator. See the newjs site for more details.

Lifecycle of JavaScript files

You create JavaScript files in src/ and then when you run rake dist
they can compressed into dist/project_name.js.

To make this work, add new src/ files into src/project_name.js via
<%= include "file_name.js" %>.

This is the build mechanism used by prototypejs and I stole it. Actually, I stole nearly everything from the prototypejs source structure.

Other stuff

Like newgem you can create a simple website (via textile files):

script/generate install_website

BONUS: instead of the crap purple default theme for newgem, newjs websites have a nice yellow theme – see example.

Source for newjs

Get it via git: master branch

Credits

The generated structure, the unittest.js test suite, and its included help files mostly come from prototypejs and newgem.

The generator mechanism is from RubiGem which comes from the Rails Generator created by Jeremy “I made someone leave Rails for PHP” Kemper.

Actually, I stole nearly everything from the prototypejs source structure. They are much cleverer than I am.

I used to love Javascript so much that it would scare me. Why? Because I used to never write any tests. No unit tests. No integration tests (e.g. Selenium). Why? I didn’t know how. Not properly. I didn’t know how to write Javascript unit tests with an autotesting tool, like ZenTest’s autotest command.

And now…

I am now free to write as much Javascript as I like knowing forever I’ll write Javascript tests first, and Javascript in libraries second. TDD for Javascript beckons!

Below is a “getting started” tutorial, a helpful autotesting plugin, and hints about a TextMate bundle for Javascript Unit Testing and a future Peepcode on Javascript Unit Testing.

Installation

Fundamentally, the ideas here are app server-neutral. We’re using the unittest.js library from scriptaculous. But here I’ll show/demo the Rails plugins that make this all uber easy to get started.

Firstly, create a blank rails app, or use an existing app. The tutorial should be non-invasive – just a few “demo” files to delete when you’re done. This tutorial is independent of your app.

ruby script/plugin install javascript_test
mkdir test/javascript
ln -s ../../vendor/plugins/javascript_test/assets/ test/javascript/assets

For Windows people copy vendor/plugins/javascript_test/assets into test/javascript/assets, as you don’t have symbolic links, which is sad.

At this stage, you can add and commit these files to your project. The rest of this article is destructible demo files and tests.

Creating javascript tests

Let’s say

$ ruby script/generate javascript_test maths
    exists  test/javascript
    exists  public/javascripts
    create  test/javascript/maths_test.html
    create  public/javascripts/maths.js

maths.js is just a blank javascript file. The important file is maths_test.html. Its packed with lots of “getting started” goodies.

Open test/javascript/maths_test.html and find the testTruth method.

testTruth: function() { with(this) {
  assert(true);
}}

This is an example of a javascript test method, with a sample assert call. It has the same api as the Ruby test::unit assert call. Thomas Fuchs’ presentation Adventures in JavaScript testing gives a good rundown of the available assert methods (and the BDD style syntax that is available). I’m also preparing a TextMate bundle to generate the assert calls, with the same “as”, “ase”, “asnn” tab completions as the Ruby bundle. It makes me happy when using it.

Running unit tests

The most holistic way to run your javascript unittests, is against all the browsers you have on your local machine. The javascript_test plugin comes with a rake task for this: test:javascripts

$ rake test:javascripts
(in /Users/nicwilliams/Documents/rails_apps/imindi_theme)
/test/javascript/maths_test.html on Safari: SUCCESS
/test/javascript/maths_test.html of Firefox: SUCCESS
Skipping Internet Explorer, not supported on this OS
Skipping Konqueror, not supported on this OS

In the browsers you’ll see:

It runs all your tests on all browsers. That’s great for a Continuous Integration process, but for me, coming from the ZenTest world of autotest, I only want to run those tests for which something has changed.

I want continual feedback. Small change, run small test. autotest gives me that for Ruby, but rake test:javascript doesn’t.

Autotesting javascript tests

On my Mac, I’ve set up something as a starting point for autotesting javascript tests, based on the “small change, run small test” principle.

$ ruby script/plugin install http://drnicutilities.rubyforge.org/svn/plugins/javascript_test_autotest/
.
.
Edit config/javascript_test_autotest.yml for the browser(s) to use for autotesting.

Open config/javascript_test_autotest.yml and uncomment the browser(s) you want to autotest with. I autotest on Safari on the Mac, even though I develop/debug on Firefox, because it reloads each page in the same tab, which I like, and I’m having problems getting this to work into Firefox at all.

My yml file looks like:

browsers:
  safari: '/Applications/Safari.app/Contents/MacOS/Safari'

So, the current version is known to work for Mac OS X and Safari. If you get this plugin working on different O/S and browsers, let me know below or please submit a patch to http://groups.google.com/group/drnicutilities

From a command line:

$ script/js_autotest
Watching public/javascripts/controls.js, public/javascripts/application.js, test/javascript/maths_test.html, public/javascripts/effects.js, public/javascripts/dragdrop.js, public/javascripts/prototype.js

Files: 6

Let’s TDD with js_autotest

In test/javascript/maths_test.html, replace testTruth, with the following and save:

// Maths class should exist
testMathsClassShouldExist: function() { with(this) {
  assert(Maths, 'Where is Maths?');
}},

js_autotest will automatically load maths_test.html into your browser, and its tests executed.

Now, let’s fix the problem by creating the Maths javascript class. In public/javascripts/maths.js, add:

var Maths = Class.create();
Maths.prototype = {
  initialize: function() {

  }
}

js_autotest will again automatically reload maths_test.html into your browser, and its tests executed.

Yay for TDD and Yay for “small change, run small test”.

Peepcode

I love Geoffrey’s Peepcodes. Geoffrey has all my money in Peepcode credits. More importantly, Geoffrey’s due to give birth this year to #1 child. Geoffrey, who works from home, is blissfully unaware that soon he’ll be a full-time father and part-time Peepcode author. This could mean fewer peepcodes on important peepcode-worthy topics.

I always wished there was a Peepcode on Javascript Unit Testing. It would take many blog posts to cover as much as can be covered in a single 60 minutes peepcode. So, we’re writing/videoing one.

As a Javascript developer [has anyone ever heard of a Javascript Users Group?] let me know what specifically you’d like covered in this video, so I don’t accidently miss something.

[optional] Install some patches

There was a section here about some patches that could be applied. These have now been applied, and javascript_test plugin has been updated with prototype 1.6.0.1. Life is good.

I’m still trying to justify my effort writing the MagicCGI code. It let you get an XML or JSON feed for any database, with some basic conditionals, limits etc. The existing demo is for my blog database.

I think this one is kinda cool – showing off all your blog posts/articles on a timeline:

To get the timeline working with the schema output from the Magic CGI, I needed to write my own Timeline EventSource. Currently I don’t do anything fancy with the generated bubbles – I just use the defaults.

The MagicCGI query gets all wp_posts (Wordpress schema) rows, where “post_status=publish” and only returns fields that are relevant (notably ignores the large post_content field holding the blog content):

http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?
format=json&table=wp_posts&post_status=publish&
field=ID&field=post_title&field=post_date_gmt&
field=post_modified_gmt&field=guid&field=comment_count

The other cool thing I did here was to deploy it all with Capistrano (an html page + javascript libs). It even deploys/manages a copy of the Timeline trunk onto the server. This is the first time I’ve deployed a non-Ruby/Rails app using Capistrano, and once I got it set up it becomes much easier to manage than using an FTP app, etc.

How easy? cap1 update (note the dubious use of capistrano 1… I still… haven’t… converted… to 2.0…)

I’m not confident enough that my solution is sexy enough to outline in detail, so if you’re interested in deploying Javascript apps etc with Capistrano, just checkout the code (below) and look at the config/deploy.rb script. (note that I’ve disabled the deprec require statement as it assumes I’m wanted to run some mongrels etc, but deprec is very handy for setting up ssh and other fun stuff at the start).

So, no details here, just a fun example.

If the Timeline tickles your fancy, their website has lots of tutorials, and/or check out my html/javascript code.

svn co http://drnicwilliams.com/svn/blog_timeline/trunk blog_timeline

If the MagicCGI tickles your fancy, its also only on svn at the moment, though its docco should be pretty good. I think.

svn co http://rubyforge.org/var/svn/magicmodels/magic_cgi/trunk magic_cgi

In the last 20 days, 43 people have used OpenID to leave comments. That’s very cool.

Corollary: add OpenID login to your blog.

Even cooler – Dynamic counter

The screen shot comes from the Comments form.

If you look at the comments form in a week, month, year, the counter above have be changed from its original value 43.

No fancy Wordpress plugins (I don’t do PHP)

No Apache tricks. (I don’t know any Apache tricks)

Instead with Javascript/HTML attached to JSON attached to a RubyCGI script attached to my Wordpress database via ActiveRecords and some magic.

I call it MagicCGI. I also call it Frigging Scary.

Try the following:

$ curl -v "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_openid_identities&action=count&format=json"
< Content-Type: txt/json
<
43

Or some XML?

$ curl -v "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_openid_identities&action=count&format=xml"
< Content-Type: txt/xml
<
<?xml version="1.0" encoding="UTF-8"?>
<result>
  <count type="integer">43</count>
</result>

Raw data?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_posts&field=post_title&limit=5&order=post_title"
[{"attributes": {"post_title": "OpenID count"}},
 {"attributes": {"post_title": "MagicCGI shows OpenID user count"}},
 {"attributes": {"post_title": "map_by_method now works with ActiveRecord associations"}},
 {"attributes": {"post_title": "Feedburner"}},
 {"attributes": {"post_title": "One year on the InterTubes"}}
]

Getting kinda scary now, I think.

Ooh, just how much magic?

Want a list of available tables to play with?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?meta=tables"
[{table_name: 'wp_users'},...]

• add &format=xml to XML output; JSON is default
• add &meta=columns to include the column schema definitions
• add &table_name=wp_posts for each table you want (instead of all the tables)

E.g. to see the columns for wp_posts and no other table, in XML:

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?meta=tables&meta=columns&table_name=wp_posts&format=xml"

A list of all urls and internal user_ids for users/OpenID users/registered commenters?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_users&field=user_url&field=id"
[{"attributes": {"id": "1", "user_url": "http://drnicwilliams.com"}}, ...

So, now we know Dr Nic = user id 1.

Oh oh oh, how about a list of comments for a specific user?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&user_id=1"
...comments by Dr Nic...
or
$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&user_id=1&action=count"
232

Dr Nic’s commented in his own blog 232 times? Out of how many comments?

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&action=count"
963

Where are the user emails? Where are their passwords?

Hidden! See below.

Callbacks?

JSON URLs support callback=someCallbackMethod and/or variable=someLocalVariable.

$ curl "http://drnicwilliams.com/cgi-bin/wp_drnicwilliams.cgi?table=wp_comments&action=count&callback=someMethod"
someMethod(963);

What does wp_drnicwilliams.cgi look like?

Something a little like this…

#!/usr/local/bin/ruby
require 'magic_cgi'         # loads the render magic, model magic, and meta-model magic
include Render

require 'magic_cgi/config/wordpress'  # connect to DB using Wordpress installation (wp-config.php)
MagicCGI::Config::Wordpress.establish_connection "/path/to/drnicwilliams/web/public"

# The following is defaulted for Wordpress connections:
MagicCGI::Config.hidden_tables |= %w[wp_openid_nonces wp_openid_associations wp_usermeta wp_tla_rss_map wp_tla_data]
MagicCGI::Config.hidden_columns['wp_users'] = %w[user_email user_pass user_activation_key]
MagicCGI::Config.hidden_columns['wp_openid_identities'] = %w[hash]
MagicCGI::Config.hidden_columns['wp_comments'] = %w[comment_author_email comment_author_IP]
MagicCGI::Config.hidden_columns['wp_posts'] = %w[post_password]

render do |params|
  data = DbTable.from_params(params)
  data ||= begin
    table_name = params['table'].first || 'wp_users'
    model_name = ActiveRecord::Base.class_name(table_name)
    klass = MagicCGI::MagicModel.create_class(model_name, ActiveRecord::Base)
    klass.find_or_count_by_params(params)
  end
  data
end

Can I write my own CGI scripts for my own DBs?

Sure. Its a library called MagicCGI.

What is MagicCGI?

Coming soon.