Dr Nic

Hacking someone’s gem with github and gemcutter

gemcutter

Ever used a rubygem, found a bug, and just wanted to quickly bust out the big guns and fix it quickly?

The gem command doesn’t come packed with a way to find the original source repository for a gem. At best, most gems at least come bundled with the complete source, tests and documentation. Some gems don’t. Fair enough, since having access to the complete source via the gem still doesn’t allow you to fix a bug and share it with the world.

For that you access to the repo, a quick way to fork it, and a post-github way to share a gem version from yours truly.

The github gem and gemcutter are the modern day tools of master hackermanship.

Instant forking fun

Let’s say you find a bug in a gem, say rails, and you want to go to town on its source.

You know the gem is called rails but you’ve no idea what the github repo is called. Never fear.

$ gem sources -a http://gemcutter.org
$ sudo gem install github
$ gh clone --search rails
Select a repository to clone:
1.  rails/rails                         # Ruby on Rails
2.  technoweenie/restful-authentication # Generates common user ...
3.  justinfrench/formtastic             # A Rails form builder plugin ...
?

Press 1 and you’ll get a clone of rails/rails.

Alternately, if you want a fork or you know the exact user/repo already:

$ gh clone rails/rails

Now, fork your own version:

$ cd rails
$ gh fork

You now have your own fork. The origin remote also now points to your fork rather than the rails/rails repository:

$ git remote show origin
* remote origin
  Fetch URL: git@github.com:drnic/rails.git
  Push  URL: git@github.com:drnic/rails.git

So, make your changes, push them. Send a pull request or github issue or lighthouse ticket or what have you.

Want to get to the github project home page for your fork?

$ gh home

Instant gem sharing

Let’s say you patched the rails gem itself but you want to share your changes via your own gem.

In the olden days, github did this for you. Now you use gemcutter, and a little manual effort to do your own renaming.

First, install the gems locally, use them, and make sure all is good.

For rails, you install the edge gems (3.0.pre) with:

$ rake install

You can’t see ‘rake install’ in the rake -T list (hence my patch), but I think the following expression displays all tasks regardless if they have a description or not:

$ rake -P | grep "^r"

Rails is composed of several gems, unlike most projects that are distributed as a single gem. Here we want to share our commit within a new drnic-rails gem, but not touch the others.

Edit the railties/rails.gemspec file from:

Gem::Specification.new do |s|
  s.platform = Gem::Platform::RUBY
  s.name = 'rails'
  s.version = '3.0.pre'
...

and give your personal gem a new name:

Gem::Specification.new do |s|
  s.platform = Gem::Platform::RUBY
  s.name = 'drnic-rails'
  s.version = '3.0.pre'

To build and distribute the new gem:

$ gem build railties/rails.gemspec
$ sudo gem install gemcutter
$ gem push drnic-rails-3.0.pre.gem
  Pushing gem to Gemcutter...
  Successfully registered gem: drnic-rails (3.0.pre)

Follow any first-time gemcutter instructions and SUCCESS! Now I have my own drnic-rails gem.

Summary

To find, clone, and fork any rubygem that is hosted on github:

$ sudo gem install drnic-github
$ gh clone --search rails
$ gh fork

To personalise the gem and share it on gemcutter:

> edit the project.gemspec to have a unique name, e.g. yourname-project
$ gem build project.gemspec
$ sudo gem install gemcutter
$ gem push yourname-project-1.0.0.gem

I think this makes it much easier, faster and more fun to hack other people’s stuff.

Related posts:

  1. Migrating project websites to github pages with sake tasks, new websites with jekyll_generator Its almost Christmas time and that means presents. It...
  2. newgem 1.0.0 all thanks to Cucumber The New Gem Generator (newgem) was exciting, moderately revolutionary, and...
  3. My attempt at sake task management I’ve used sake intermittently in my workflow. It competes...
  4. [ANN] Generating new gems for graceful goodliness I don’t like you [1]. You don’t share code....

21 Responses to “Hacking someone’s gem with github and gemcutter”

  1. [...] Hacking someone’s gem with github and gemcutter Tags: debugging, github, rails, ruby, scalability, webdevelopment Comment (RSS)  |  Trackback [...]

  2. Luis Lavena says:

    I shared my concerns about the usage of Gemcutter in the GitHub way:

    http://groups.google.com/group/gemcutter/browse_thread/thread/80fd383032105250

    I still think that publishing forked gems and making other gems depend on those specific forked ones (instead of the official) is flawed.

    At least in RubyForge you had the approval process to avoid duplication, now with Gemcutter you can’t.

  3. I have to agree with Luis.

    I hope that people don’t think gems with names matching /^\w+-.*$/ are forks. Or that my fork of “github-gem” that I pushed called “drnic-github-gem” is actually from drnic.

    I’m starting to be convinced by Yusuke’s wheat and chaff arguments. :-(

  4. For another bit in the gem hacking cycle, if you just want to peek into a gem I whipped up open_gem that lets you just do ‘gem open rails’.

    I find it handy for that phase where you’re not sure if the bug is in the library or your understanding of the library ;)

  5. [...] This post was mentioned on Twitter by Ruby on rails Guru, Dr Nic, Ozéias Sant'ana, eladmeidar, Sandip Ghosh and others. Sandip Ghosh said: RT @drnic: Hacking someone's gem with github and gemcutter http://url.mocra.com/4king [...]

  6. Dr Nic says:

    @Luis + Aaron, I deep down agree with you. Ultimately github’s mechanism encouraged the path of username-prefix forking of gems for the last year or so.

    I don’t believe rubyforge stopped anyone from pushing out a “drnic-rails” gem – you just need a project of any name and you can push dozens of gems through it (the seattlerb project hosts all their gems, instead of one project per gem).

    Gemcutter makes it even easier.

    But both allowed someone to push out a “drnic-rails” without the permission of “drnic” nor “rails” maintainers.

  7. Dr Nic says:

    @adam – I use gem open activerecord all day long. It’s great.

  8. @drnic But is the canonical gem source really the place to publish forks? Why not run a gem server instead? Or if you don’t want to do that, wait for the subdomain support from gemcutter.

    Pushing forks to the canonical source seems like a Big Deal®, and not the best thing for having people try out some cool new feature you added to xyz gem. IMO if you’re pushing a fork to a canonical server you are saying “I want to replace you”.

    /2cents

  9. Dr Nic says:

    @aaron – I’m slow to these sorts of parties. I only started using github’s gem server just before they turned it off. I don’t know right from wrong anymore. I am confused.

  10. @drnic Search your heart, I’m sure you’ll find the right path. ;-)

  11. Richie Vos says:

    I feel like the argument against what gemcutter’s doing is pretty similar to the argument against git. Why would you want people to be able to easily fork a project, if you let them do that then how is anyone ever going to know where the real true source lives?

    Lots of logic to that, but people are smarter than that. 50 blah-rails gems out there on gemcutter? I bet they’re all scratching some itch, and I bet it’d serve the rails guys to take a look and see what’s going on to get an idea of what people are thinking. Also, as a user if I see 50 people forked that, I’m going to know that there’s some value in the whole X-rails gems, since that many people have found it useful enough to customize.

    And towards the canonical’ness, if I make a gem and somebody forks it, and then a bunch of other people use the fork, and google ranks theirs higher, and gemcutter shows theirs has more downloads, why is mine better than that one? Let the best gem win.

    The reason I’m totally sold on this is I’ve been reaping the benefits of this recently. Example:

    1. my team finds a bug in the thinking-sphinx gem
    2. we do what nic’s talking about and fork the source on github, fix it, push it, and send the primary maintainer a pull request (which is a big step I think a lot of people skip)
    3. we want to use that code now, so using jewler + gemcutter we create a new gem named moneypools-thinking-sphinx out on the interweb with our fix (hadn’t seen nic’s helpers before)
    4. on all our boxes (including production) we can just install the new gem using our normal gem tools (gem, rake gems:install, bundler) right away — no jumping through hoops.

    That’s a really awesome experience, and incredibly convenient.

    The final follow-up is the maintainer pulls in our changes and pushes a new version of his gem out there. Ours is now outdated. This is the tricky part where I’m not sure what should happen. Deleting it means anyone using it would be SOL, keeping it out there though might not provide much value. Gemcutter takes the ‘leave it there’ approach.

    So my overall summary is I think this’s one of those cases where the hypothetical bad-cases are outweighed by the real-world convenience and experiences.

    Sorry for the long-commont/post-hijacking/rant(?)

    P.S. I do think the more features github/gemcutter/others add to help ferret out what are the most active/popular versions the better.

  12. Luis Lavena says:

    @Richie

    While your logic sounds reasonable, and under certain circumstances valid, it just turn into a complicated environment for new comers.

    For example, in your case, let’s say your Rails/Merb applications depends on this moneypools-thinking-sphinx.

    Because is a cool project, or someone want to put it in their system, they clone it and start dealing with the dependencies.

    Either using Rails gem dependencies or Bundler, you would get moneypools-thinking-sphinx, which is cool.

    That is because both Bundler and Rails-gem mechanism uses gem 'foo-bar' prior the require of the library.

    Now, things turn hairy for people that do not pollute their scripts with gem statements.

    As my example in the group, I found issues with some users that had installed relevance-rcov and others spiceycode-rcov, which both offer “rcov.rb” to the require method.

    Of course, relevance was loaded first, but could be “abc-rcov”, with lower version number than relevance which could be loaded due this simple require.

    So, now you want to help out someone that is doing require ‘rcov’ and you ask “please tell me the version of rcov gem you have installed”

    gem list rcov will throw no results.

    Instead, the need to do gem search rcov, which will also bring a lot of noise to the results.

    Now try to explain to that user “ok, you got the wrong rcov” and the answer will be “I haven’t installed it, was installed by XYZ, but MN uses this other version”

    All the above, is not hypothetical, I’m giving a simple example of something that already happened to me and some users contacting me about RubyInstaller.

    So, after you burn with this stuff a couple of times… instead of catching it late, why not fix the root of the issue?

    In my company we built our own gem server and published there even gems for our private projects, not just open-source forks, that worked for us, and could work for others.

    Apologizes for the long answer and my english, which I believe sometimes is not good enough to explain myself.

    Cheers.

  13. adevadeh says:

    seems to be a slight problem in the gemspec:

    /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load — highline (LoadError)

    need to add highline as a dependency.

    Thanks for this nice helper gem, i’m trying it out right now.

  14. Dr Nic says:

    @adevadeh oops, thanks. I’ve added the highline dependency and I’ve released 0.3.10.

  15. Richie Vos says:

    Maybe I should move to the mailing-list, but for now:

    @Luis great points. I’m going to have to think about that some more. One thing that comes to mind though is that’s not a problem unique to namespaced gems, the same problem can occur with gem A depending on gem C version 1.0 and gem B depending on gem C version 1.5. I do readily admit that upgrading from gem A 1.0 to gem A 1.5 is probably much more likely to work than from moneypools-A 1.0 to gem A-1.0.

    The other thing is a library’s probably requiring a non-default version of a gem for a reason. If they weren’t using a non-default version they’d probably do something like monkey-patching or something else under-the-covers to patch in the functionality they need. I’d much rather be debugging gem dependency issues than trying to debug a chain of patches applied to a gem.

    But again, I do think most of the time people are going to go with a gem that is (one of or multiple of):
    1. canonical
    2. with a special feature they require
    3. popular

    1 should be the default (and will be with a good maintainer)
    2 isn’t avoidable, if you need the feature and it’s not getting pulled in, what’cha gonna do
    3′s not a good one to use unless ‘popular’ means maintained and the real one isn’t

  16. @Richie

    WRT #2: Maybe the maintainer hasn’t pulled in “cool new feature x” for some *very important* reason. Or maybe the maintainer doesn’t know that there is demand. Is it the canonical author’s responsibility to keep an eye on all of the forks? Why not talk to the author about “cool new feature x” rather than publish a fork? If your fork is experimental, why are you littering the canonical gem repository with it?

    WRT #3: In that case you need to take over maintenance of the canonical gem or change your gem name from “user-xx” to something else and make it a real replacement with different top-level .rb files. Then people can use the non-forked version without entering “what gem did I require this from” hell.

  17. [...] wanted to quickly bust out the big guns and fix it quickly? Surely, we all have.. so he's written Hacking someone's gem with github and gemcutter to show us how to easily fork an existing gem, make our changes, and get it deployed on Gemcutter [...]

  18. aalex says:

    Hi Dr Nic! this post is great -> fixing someone’s code in less than a minute! Super! I think this is so cool that I produced a demo of this for my RubyPulse screencast series…

    -aaalex

  19. [...] Dr Nic ’s Hacking someone’s gem with github and gemcutter [...]

  20. Hey, Dr. Nic, I don’t know why but installing gems from github no longer works for all the gems, so I wrote this post > http://raflabs.com/blogs/silence-is-foo/2010/07/19/installing-a-gem-fork-from-github-source/