Tom Ward

Geohash toy: code released

2011-09-24T00:00:00+01:00

A couple of weeks ago I wrote about a small toy app I'd written to explore geohashes. Now I've cleaned the code up a little, upgraded it to rails 3.1 and released it here on github. Enjoy.

A small toy to explore geohashes

2011-09-15T00:00:00+01:00

For an app I've been building, I've been looking into geohashes. For those who don't know, the geohash format is a simple way to encode latitude and longitude into a single string. As an example, Nelson's Column in London (51.507794, -0.127952) has the geohash gcpvj0dyds.

Geohashes have a couple of interesting features. First, as you remove characters, you lose precision. gcpvj0dyds fairly accurately points to Nelson's Column; gcpvj0d represents the South-West of Trafalgar Square and some of the Mall; and gcpvj covers most of Central London, as well as Islington and King's Cross. A geohash doesn't really represent a point, but rather a bounding area within which a point may lie. The longer the geohash, the smaller that bounding area.

The other interesting property geohashes have is that nearby locations usually (but not always) share similar prefixes. So much of North London is in gcpv, while much of South London is in gcpu. However, due to the Prime Meridian passing through Greenwhich, South East London has the geohash u10h - wildly different than the other two.

This probably sounds a bit complicated. I was having trouble getting my head around the concept, so to try and get to grips with geohashes I've written a toy app that draws them on a map. To try it out, go to http://geohash.gofreerange.com, click the map, zoom and play. If you find it useful, let me know.

Update: This code is now available [on github](https://github.com/tomafro/geohash-explorer).

Tip: Automatic bundle exec for rake and other gems

2011-09-01T00:00:00+01:00

It's irritating to run gem commands like rake, cap, rspec and others, only to find they needed to be executed via bundle exec. As a simple solution, I use a simple zsh function, combined with aliases for commonly used commands.

Here's the function (which I've named be):

if [[ -a Gemfile ]]; then
  bundle exec $*
else
  command $*
fi

It's very simple. If there's a Gemfile in the pwd, it runs commands through bundle exec. Otherwise it just runs them.

I've combined this with some aliases for much less pain and less frustration:

alias rake='be rake'
alias cap='be cap'
alias rspec='be rspec'

Presenting the #blue api

2011-08-02T00:00:00+01:00

In building #blue (sign up now!), one of the problems we faced was how to build json data in response to requests to our API. The typical rails solution would be to override #as_json in a model class, then write a controller like this:

    class ContactsController < ApiController
      responds_to :json

      def show
        respond_with Contact.find(params[:id])
      end
    end

I always prefer to keep my controllers as skinny as possible, so this looks like a great solution. The respond_with call takes care of converting the message to json and responding with the right Content-Type, all in a simple call. However it has a number of problems and disadvantages.

The biggest issue for our API is that rather than expose the id of each model, we've tried to encourage the use of the uri instead. So the json returned for a single contact (for example) looks like this:

    {
      "contact": {
        "uri": "https://api.example.com/contacts/ccpwjc",
        "name": "George",
        "email": "george@handmade.org",
        "msisdn": "447897897899",
        "phone_number": "07897897899",
        "messages": "https://api.example.com/contacts/ccpwjc/messages"
      }
    }

It doesn't just have a uri for the actual contact, but also for the messages belonging to that contact (and yes, I regret not calling that attribute messages_uri). Models can't generate uris, and shouldn't really be aware of them, so overriding #as_json doesn't work. In any case, the json structure is really presentation logic, not business logic. It doesn't belong in the model.

Presenting a single model

The solution we've used is to build a presenter for each model, solely responsible for building the json. Here's an example for a contact:

  class ContactPresenter
    include Rails.application.routes.url_helpers

    attr_accessor :controller, :subject
    delegate :params, :url_options, :to => :controller
    delegate :errors, :to => :subject

    def initialize(controller, subject)
      @controller = controller
      @subject = subject
    end

    def as_json(options = {})
      {:contact => {
        :uri => uri,
        :email => subject.email,
        :name => subject.name,
        :msisdn => subject.msisdn,
        :phone_number => subject.phone_number,
        :messages => api_contact_messages_url(:contact_id => subject.id)
      }
    end

    def uri
      api_contact_url(:id => subject.id)
    end
  end

It's now simple to rewrite our controller to use the new presenter:

    class ContactsController < ApiController
      responds_to :json

      def show
        respond_with ContactPresenter.new(self, Contact.find(params[:id]))
      end
    end

Presenting pages of models

The presenter above works well for a single model, but many of our API calls return a page of results. The /contacts for example returns all the contacts belonging to a user (of which there may be hundreds). Luckily it's simple to adapt this pattern to present pages like this. First, we change our original #as_json method slightly:

  def as_json(options = {})
    if options[:partial]
      {
        :uri => uri,
        :email => subject.email,
        :name => subject.name,
        :msisdn => subject.msisdn,
        :phone_number => subject.phone_number,
        :messages => api_contact_messages_url(:contact_id => subject.id)
      }
    else
      {:contact => as_json(:partial => true)}
    end
  end

This change allows us to call as_json with the options :partial. With the option, a hash of data is returned. Without, the same hash is returned, wrapped in another hash.

Next, add a page presenter:

class ContactPagePresenter
  include Rails.application.routes.url_helpers

  attr_accessor :controller, :subject
  delegate :params, :url_options, :to => :controller
  delegate :errors, :to => :subject

  def initialize(controller, subject)
    @controller = controller
    @subject = subject
  end

  def as_json(options = {})
    contacts = subject.map {|o| ContactPresenter.new(controller, o).as_json(:partial => true) }

    {:contacts => contacts}.tap do |result|
      if subject.previous_page
        result[:previous_page_uri] = contacts_url(subject.previous_page)
      end

      if subject.next_page
        result[:next_page_uri] = contacts_url(subject.next_page)
      end
    end
  end
end

Finally, we can add an index action using this presenter:

class ContactsController < ApiController
  responds_to :json

  def index
    respond_with ContactPagePresenter.new(self, Contact.paginate(:page => params[:page], :per_page => 50))
  end
end

Refactoring common logic

The code above is a very much simplified version of what we do in #blue. We have many controllers, and several different models, so in our actual code we've abstracted out as much common logic as possible. In reality, our contacts controller looks more like this:

class ContactsController < ApiController
  before_filter :find_contact, :only => [:show, :update]

  def show
    present @contact
  end

  def index
    present_page_of current_account.contacts
  end

  def create
    @contact = current_account.contacts.build(attributes)
    @contact.save
    present @contact
  end

  def update
    @contact.update_attributes(attributes)
    present @contact
  end

  private

  def find_contact
    @contact = current_account.contacts.where(:_id => params[:id]).first
    head :status => :not_found unless @contact
  end
end

I think the code looks pretty clean. The clever stuff happens in the #present and #present_page_of methods, defined in the superclass:

class ApiController < ApplicationController::Base
  protected

  def present(instance, options = {})
    presenter = presenter_class.new(self, instance)
    options[:location] ||= presenter.uri if request.post? && subject.errors.empty?
    respond_with presenter, options
  end

  def present_page_of(collection, options = {})
    presenter = page_presenter_class.new(self, page_of(collection))
    respond_with presenter, options
  end

  def page_of(collection)
    collection.paginate(:page => params[:page], :per_page => 50)
  end

  def presenter_class
    (self.class.name.gsub!("Controller", "").singularize + "Presenter").constantize
  end

  def page_presenter_class
    (self.class.name.gsub!("Controller", "").singularize + "PagePresenter").constantize
  end
end

The #present and #present_page_of methods handle determining the correct presenter to us, as well as paginating the collection where required. They still use rails build in #respond_with method, which helps provide the correct response headers for each request. As the ContactPresenter delegates #errors to its subject, if there are validation errors, #respond_with correctly returns a 422.

One further motivation for this pattern (other than moving presentation logic out of the model) is that should we want to release a new version of our API, we'll be able to get a lot of the way there simply by swapping which presenter is used. We started using this code about 8 months ago, and I'm still pretty happy with it. I hope you find something useful in it too.

Any comments or suggestions, please get in touch with me on twitter.

#blue opens for business

2011-03-03T00:00:00+00:00

If you're an O2 customer based in the UK, you might be interested that #blue, a project we've built, has recently (re)opened for business. It's a soft launch, but feel free to tell your friends.

#blue gets copies of every SMS message you send or receive, and makes them available to you on the web. You can search your messages, read whole conversations, and even reply, all from the comfort of your browser.

As well as a nice-looking web UI, there's also an API that allows other apps to read and manipulate your messages and contacts (once you give them permission). Think automatic posting to twitter, or showing new messages as alerts on your desktop. The possibilities are endless.

Oh, and it's all free. All you need is an O2 phone.

If you think it sounds interesting, ask for a beta request. You should get an invitation very quickly. Once on board, please send me any suggestions and feedback. It's going to be interesting to see where we can take this project.

Mongo instrumentation released as a gem

2011-02-19T00:00:00+00:00

Enough people seemed to comment and like the mongo instrumentation code I wrote about yesterday that I've packaged it up and released it as a gem.

The mongo-rails-instrumentation gem is available on rubygems, and the code is up on github.

Adding it to a project is simple, just put the following in your Gemfile, run bundle install and restart your app.

    gem 'mongo-rails-instrumentation', '~>0.1'

Please add any suggestions, improvements and comments to the code in github. I hope people find it useful.

Experimental Mongo instrumentation (for Rails 3)

2011-02-18T00:00:00+00:00

Update: Changed to instrument methods on the Mongo::Connection

One of our latest rails projects uses Mongo as a backend. We're just starting to get some traffic, and as we do, we're monitoring the logs for slow requests. When using ActiveRecord, rails splits out the recorded request time like so:

    Completed 200 OK in 6ms (Views 370.5ms | ActiveRecord: 2.3ms)

We wanted to do the same for our Mongo accesses, just to give a rough idea as to what our requests were doing. Luckily Rails 3 makes this relatively straightforward, providing hooks to instrument methods, subscribe to log messages and add information to the request log. Here, then, is my first stab (mainly harvested from ActiveRecord):

module Mongo
  module Instrumentation
    def self.instrument(clazz, *methods)
      clazz.module_eval do
        methods.each do |m|
          class_eval %{def #{m}_with_instrumentation(*args, &block)
            ActiveSupport::Notifications.instrumenter.instrument "mongo.mongo", :name => "#{m}" do
              #{m}_without_instrumentation(*args, &block)
            end
          end
          }

          alias_method_chain m, :instrumentation
        end
      end
    end

    class Railtie < Rails::Railtie
      initializer "mongo.instrumentation" do |app|
        Mongo::Instrumentation.instrument Mongo::Connection, :send_message, :send_message_with_safe_check, :receive_message

        ActiveSupport.on_load(:action_controller) do
          include Mongo::Instrumentation::ControllerRuntime
        end

        Mongo::Instrumentation::LogSubscriber.attach_to :mongo
      end
    end

    module ControllerRuntime
      extend ActiveSupport::Concern

      protected

      attr_internal :mongo_runtime

      def cleanup_view_runtime
        mongo_rt_before_render = Mongo::Instrumentation::LogSubscriber.reset_runtime
        runtime = super
        mongo_rt_after_render = Mongo::Instrumentation::LogSubscriber.reset_runtime
        self.mongo_runtime = mongo_rt_before_render + mongo_rt_after_render
        runtime - mongo_rt_after_render
      end

      def append_info_to_payload(payload)
        super
        payload[:mongo_runtime] = mongo_runtime
      end

      module ClassMethods
        def log_process_action(payload)
          messages, mongo_runtime = super, payload[:mongo_runtime]
          messages << ("Mongo: %.1fms" % mongo_runtime.to_f) if mongo_runtime
          messages
        end
      end
    end

    class LogSubscriber < ActiveSupport::LogSubscriber
      def self.runtime=(value)
        Thread.current["mongo_mongo_runtime"] = value
      end

      def self.runtime
        Thread.current["mongo_mongo_runtime"] ||= 0
      end

      def self.reset_runtime
        rt, self.runtime = runtime, 0
        rt
      end

      def mongo(event)
        self.class.runtime += event.duration
      end
    end
  end
end

It looks complicated, but it's actually pretty simple. Data access methods in ~~Mongo::DB and Mongo::Collection~~ Mongo::Connection are hijacked and surrounded by an ActiveSupport::Notifications.instrumenter.instrument block. This triggers events which are listened to by the LogSubscriber, summing the total time spent in Mongo. The ControllerRuntime then collects this count to be displayed, and resets the sum to zero ready for the next request. The output looks like this:

    Completed 200 OK in 838ms (Views: 370.5ms | ActiveRecord: 2.3ms | Mongo: 441.5ms)

It's just a first stab, so any comments and improvements are more then welcome. It's here on gist so please fork away.

A home for my Active Record column reader

2011-02-08T00:00:00+00:00

@kraykray was nice enough to let me know that Rails 3.03 had broken my column-reader code.

I've always seen it as a toy rather than a serious project, but getting a bug report made me re-evaluate. If people are using it, even a tiny piece of code like this deserves a proper home.

So here it is, as both a github repository and a gem. Enjoy!

An updated rails template for gem bundler

2010-02-28T00:00:00+00:00

Update 8th February 2011:

Bundler has changed a lot since I wrote these instructions. Use them at your own risk!

A few months ago I wrote a rails template for gem bundler. Since then, bundler has changed a lot, and my template no longer works. Here then is an updated version, based on this gist from Andre Arko. Using it, you should be able to get a rails 2.3.5 project working with bundler in less than 5 minutes.

The first step is to install the latest bundler. At the time of writing, this was 0.9.9.

gem install bundler

Now you should be able to run the template, either on a new project, or on an existing rails 2.3.5 project.

rails -m http://github.com/tomafro/dotfiles/raw/master/resources/rails/bundler.rb <project>

On a fresh project, that should be all you need to do. On an existing that used an older version of bundler, you'll need to remove the old hooks in config/preinitializer.rb and config/environment.rb, and the gems folder.

Explaining the template, step by step

The first step creates the project Gemfile, with rails available in all environments, and ruby-debug included in development. If the project has other gems, they should be added here, rather than using rails own config.gem mechanism.

file 'Gemfile', %{
source 'http://rubygems.org'

gem 'rails', '#{Rails::VERSION::STRING}'

group :development do
  gem 'ruby-debug'
end
}.strip

The next step is get bundler to load correctly. This is done in two stages. First, in config\preinitializer.rb bundler needs to be setup. This adds all the bundled gems to the ruby load path, but doesn't initialise them.

  
append_file '/config/preinitializer.rb', %{
begin
  # Require the preresolved locked set of gems.
  require File.expand_path('../../.bundle/environment', __FILE__)
rescue LoadError
  # Fallback on doing the resolve at runtime.
  require "rubygems"
  require "bundler"
  if Bundler::VERSION <= "0.9.5"
    raise RuntimeError, "Bundler incompatible.\n" +
      "Your bundler version is incompatible with Rails 2.3 and an unlocked bundle.\n" +
      "Run `gem install bundler` to upgrade or `bundle lock` to lock."
  else
    Bundler.setup
  end
end
}.strip

Second, the rails boot process is modified to start the bundler environment. This 'requires' all gems in the bundle, letting them run initialisation code.

gsub_file 'config/boot.rb', "Rails.boot!", %{
  
class Rails::Boot
 def run
   load_initializer
   extend_environment
   Rails::Initializer.run(:set_load_path)
 end

 def extend_environment
   Rails::Initializer.class_eval do
     old_load = instance_method(:load_environment)
     define_method(:load_environment) do
       Bundler.require :default, Rails.env
       old_load.bind(self).call
     end
   end
 end
end

Rails.boot!
}

All that's left now is a little cleaning up. The .bundle folder should never be checked into the code repository as it holds machine-local configuration, so it's added to .gitignore. Finally, bundle install is run to fetch the bundled gems.

append_file '/.gitignore', %{
/.bundle
}

run 'bundle install'

And that's it. I hope you find it useful.

Rails 3 direct column reader

2010-02-11T00:00:00+00:00

Whilst trying to get my head around arel and it's relationship to ActiveRecord in rails 3, I've updated the simple ColumnReader class I introduced last year. It lets you read the (correctly cast) column values for an ActiveRecord class, without the overhead of instantiating each object.

Here's the updated code:

module ColumnReader
  def column_reader(column_name, options = {})
    name = options.delete(:as) || column_name.to_s.pluralize
    column = columns_hash[column_name.to_s]
  
    self.module_eval %{
      def self.#{name}
        query = scoped.arel.project(arel_table[:#{column_name}])
        connection.select_all(query.to_sql).collect do |value| 
          v = value.values.first
          #{column.type_cast_code('v')}
        end
      end
    }
  end

  ActiveRecord::Base.extend(self)
end

The code isn't that different, though using scoped over construct_finder_sql feels a lot nicer. If you've got suggestions for improvement gist away.

Usage is similar to before, only using the new rails 3 syntax:

class Animal < ActiveRecord::Base
  column_reader 'id'
  column_reader 'name'  
 
  named_scope :dangerous, :conditions => {:carnivorous => true} 
end

Animal.names 
#=> ['Lion', 'Tiger', 'Zebra', 'Gazelle']
 
Animal.limit(1).names 
#=> ['Lion'] (Normal finder options supported)
 
Animal.dangerous.names 
#=> ['Lion', 'Tiger'] (Scoping respected)
 
Animal.ids
#=> [1, 2, 3] (Values cast correctly)

I'm still not entirely convinced of the value of this helper, so if you find a good use tweet me. Enjoy!