Tom Ward

Infinite sequences in ruby

2012-12-31T00:00:00+00:00

Infinite sequences in ruby

One feature of harmonia is tasks that recur on a schedule, e.g. every Thursday, or on the 30th day of each month. For these tasks we need to know not just when they’ll next occur, but also things like the next 4 occurrences, or all occurrences this month.

To do this we’ve used a technique more common in clojure: using an infinite sequence.

Defining simple infinite sequences

Ruby 1.9 and above let us define infinite sequences using the Enumerator class. A simple example is the sequence of integers:

integers = Enumerator.new do |yielder|
  n = 0
  loop do
    yielder.yield n
    n = n + 1
  end
end

>> integers.take(5)
=> [0, 1, 2, 3, 4]
>> integers.take(10)
=> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Here’s how this works. The block passed to Enumerator.new defines our sequence. It takes a yielder argument with a special method #yield, used to return elements in the sequence. Whenever #yield is called, execution of the block stops. Execution only restarts if more elements are needed, making the sequence lazy. The Enumerator class handles this stopping and starting execution — we only need to worry about how to generate each element.

Most of the code above is concerned with looping and yielding values, not generating them. We can factor this out, giving us a method that makes it trivial to define new sequences:

def sequence(&generator)
  Enumerator.new do |yielder|
    n = 0
    loop do
      yielder.yield generator.call(n)
      n = n + 1
    end
  end
end

>> integers = sequence {|n| n}
>> integers.take(10)
=> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

>> squares = sequence {|n| n * n}
>> squares.take(10)
=> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

When using infinite sequences laziness is extremely important, as it’s impossible to generate all members of an infinite list in anything less than infinite time. A sequence that outputs whenever a new value is calculated shows this laziness in action:

integers = sequence {|n| puts "Calculating result #{n}"; n}

>> integers.take(3)
Calculating result 0
Calculating result 1
Calculating result 2
=> [1, 2, 3]

Now we can define sequences, how can we use them? We’ve already seen that we can #take any number of elements from our sequence. We can also use #take_while to take elements until a condition is met, such as finding all square numbers under 250:

>> squares.take_while {|n| n < 250}
=> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225]

In fact all Enumerable methods are available, but we have to take care in using them. As our sequences are infinite, any method that iterates over all members has the potential to take an infinite amount of time. For example calling #any? will either return true (if a matching element exists) or never return.

Another big drawback is that when we call Enumerable methods, laziness isn’t preserved. Suppose we want the first 5 odd square numbers. We might try the following:

>> squares.select {|n| n % 2 == 1}.take(5)

Unfortunately this will never return. Even though we only want a finite set of results, the call to #select operates on the full infinite sequence before it returns. #take(5) is never called. The same problem exists with #map, #drop, #reject and more.

Preserving laziness across derived sequences

Without laziness preservation, our sequences seem of limited use. In ruby 2.0 we can use #lazy to make our sequences lazier, but in 1.9 this isn’t available to us. Thankfully we can get around this by generating lazy versions of Enumerable methods ourselves. Let’s take the previous example, finding the first 5 odd square numbers. We hit a roadblock because #select never returned. If instead of using #select we use a new Enumerator to do our selecting, we can work around this:

odd_squares = Enumerator.new do |yielder|
  squares.each do |square|
    yielder.yield square if (square % 2 == 1)
  end
end

>> odd_squares.take(5)
=> [1, 9, 25, 49, 81]
>> odd_squares.take(10)
=> [1, 9, 25, 49, 81, 121, 169, 225, 289, 361]

Our new Enumerator iterates lazily through our original sequence, yielding only odd values. We’ve chained two enumerators together to preserve laziness.

This is all a bit cumbersome as is, but we can turn this into a ‘#select’ method on a new LazyEnumerator class:

class LazyEnumerator < Enumerator
  def select(&block)
    self.class.new do |yielder|
      each do |value|
        yielder.yield value if block.call(value)
      end
    end
  end
end

def lazy_sequence(&generator)
  LazyEnumerator.new do |yielder|
    n = 0
    loop do
      yielder.yield generator.call(n)
      n = n + 1
    end
  end
end

>> lazy_squares = lazy_sequence {|n| n * n}
>> lazy_squares.select {|n| n % 2 == 1}.take(5)
=> [1, 9, 25, 49, 81]

#reject and #map can be chained in a similar way to #select:

class LazyEnumerator < Enumerator
  def reject(&block)
    self.class.new do |yielder|
      each do |value|
        yielder.yield value unless block.call(value)
      end
    end
  end

  def map(&block)
    self.class.new do |yielder|
      each do |value|
        yielder.yield block.call(value)
      end
    end
  end
end

#drop and #drop_while are slightly more complicated, but follow a similar pattern. The main difference being that they need to keep trap of how much to drop:

class LazyEnumerator < Enumerator
  def drop(n)
    self.class.new do |yielder|
      dropped_enough = false
      dropped = 0
      each do |element, index|
        dropped_enough ||= dropped >= n
        yielder.yield element if dropped_enough
        dropped = dropped + 1
      end
    end
  end

  def drop_while(&block)
    self.class.new do |yielder|
      match_found = false
      each do |element|
        match_found ||= !block.call(element)
        yielder.yield element if match_found
      end
    end
  end
end

Together, these methods give us a LazyEnumerator that can be chained in a great number of ways, giving our sequences a lot of power. #take and #drop let us select which members of a sequence we’re interested, while #select, #reject and #map allow us to build new sequences from existing ones:

>> integers = lazy_sequence {|n| n}
>> squares = integers.map {|n| n * n }
>> odd_squares = squares.select {|n| n % 2 == 1}
>> odd_squares.drop(10).take(10)
=> [441, 529, 625, 729, 841, 961, 1089, 1225, 1369, 1521]

Using only a few simple methods, we’ve been able to answer a complicated (though contrived) question, what are the second ten odd square numbers? This particular answer may not be interesting, but the technique of defining and deriving infinite sequences is much more general and useful. This is only a small sample of what you can do with them.

Deploying a rails app from scratch using recap

2012-12-27T00:00:00+00:00

Deploying a rails app from scratch using recap

If you follow our company blog you’ll know that we’re working on Harmonia, our virtual office manager. I thought I’d explain how we use recap to deploy harmonia, to show how easy and fast recap makes application deployment.

Harmonia is a fairly standard rails application. As well as a web front-end, it has two other processes. A queue worker is used to send outgoing emails, whilst the core of the application is the ticker; a process which ‘ticks’ every minute, assigning tasks to team members. We use foreman to declare these processes in the following Procfile:

web: bundle exec unicorn -p $PORT -c unicorn.conf.rb
ticker: bundle exec rails runner script/ticker.rb
worker: bundle exec rake environment resque:work QUEUE=assignments VVERBOSE=1

All of these processes touch application code, so whenever we deploy a new version of the app (which we do frequently) they need to be restarted. Our app also has a database with associated migrations, uses environment variables like DATABASE_URL for configuration, and has a number of gem dependencies managed by bundler.

This is all handled by recap.

Getting started – adding recap to the project

Using recap with a rails project is simple. First add gem 'recap' to the Gemfile and run bundle install. Next run bundle exec recap setup, which will generate a Capfile, guessing values for the git repository and app name. You should check these values and change the server to point to your app server. As an example, the complete Capfile for harmonia is shown below:

require 'recap/recipes/rails'

set :application, 'harmonia'
set :repository, 'git@github.com:freerange/harmonia.git'

server 'bison.harmonia.io', :app

Applications deployed with recap need their own user, owning all files and processes. Assuming we can ssh into our server and are listed as a sudoer, we can create this user automatically running cap bootstrap. This will also add our own ssh user to the application group, allowing it to deploy the application.

Next we can set any environment variables we need for configuration. These are loaded in the application user’s .profile, so are available to all processes started by recap. In harmonia we set our smtp credentials, the server port, some api keys and more, using commands like cap env:set PORT=7000 and cap env:set SMTP_PASSWORD=secret.

The app is now almost ready to deploy. We can prepare it for deployment with cap deploy:setup, which clones the code repository, installs our gem bundle, sets up the database and precompiles our assets.

Finally, running cap deploy will start the app for the first time, launching each process defined in the Procfile with the environment variables we previously set.

Really fast deployments

While recap makes it very easy to get apps up and running the first time, it comes into its doing subsequent deployments. At Go Free Range we like to deploy apps we’re working on very frequently. One thing that helps ensure we do this is making each deployment as fast as it can be.

Using git as recap does is already a very quick way to get code changes onto servers, but recap takes things a step further. By testing to see which files have changed it knows which tasks can be skipped. For example, database migrations won’t be run if db/schema.rb has not changed; the gem bundle won’t be re-installed unless Gemfile.lock has been updated, and foreman process scripts won’t be exported if the Procfile is unchanged. In fact, if these files don’t exist, these tasks will never run at all.

The future

Using recap with Harmonia has made our deployment process very fast and simple. When the main harmonia server became over-burdened and we decided to commission a new machine dedicated to harmonia, recap made that process quick and painless. As well as harmonia, recap is also used to deploy the Go Free Range website, this blog, and a number of other small sites and projects where it has proven itself well. For larger projects, there are some features (such as more control as to what processes run where) that are missing, but I plan to add these in the next release. For all other sites, recap has proven itself a lightweight and capable alternative to the standard Capistrano deployment recipes.

Past, Present and Future

2012-11-16T00:00:00+00:00

Past, Present and Future

For me and the rest of Go Free Range, the big event this week was the public launch of Inside Government, the government focussed part of GOV.UK. Inside Government aims to replace the majority of government department and agency websites with a single place for all news, policies, publications and consultations. On Wednesday, the first two department websites (for the Department for Transport and the Department for Communities and Local Government) were moved across.

We’ve been working on this project for 14 months, from the initial commit, through the beta launch right up until today. I’ve enjoyed it, but it has been extremely hard work. And although there are hundreds of things I’d still like to change, there are hundreds more I’m proud of.

Harmonia

As well as Inside Government, we’ve also been slowly inviting people to Harmonia, our ‘virtual office manager’. In case you haven’t read the Go Free Range blog, Harmonia is a tool we use to assign all the tasks needed to keep our company running. Rather than use a schedule, each piece of work is assigned in a way designed to be both random and fair.

One variation or another of Harmonia has been helping us for well over a year, but now we’ve built a web app for everyone to use. It’s still a bit rough around the edges, but we’d love it if people signed up and gave us feedback. We’re building this because we want to learn how to build better products, for the future of Go Free Range.

Tip: Bundler with --binstubs

2012-06-21T00:00:00+01:00

Tip: Bundler with --binstubs

In a previous blog, I wrote how I’d aliased commands such as rake, cap and rspec to run either with or without bundle exec, based on the presence of a Gemfile. I gave up on that a while ago. Instead, I’ve started installing all my bundles like this:

bundle install --path .bundle/gems --binstubs .bundle/bin

I often use features like bundle open <gem> to debug and edit failing gems, so I like to keep each application’s gems isolated. The --path .bundle/gems installs them within an application’s .bundle directory. As well as isolating my gems, it has the added benefit that I can blow away the gemset with rm -rf .bundle

The --binstubs .bundle/bin option installs bundle-aware scripts for each command provided by a bundled gem. For example, a bundle including rake will generate a .bundle/bin/rake script. By adding ./.bundle/bin to the front of my environment PATH, the bundled version of rake will run when I’m in the application folder. I never have to type bundle exec!

Obviously typing that long bundle install command each time is tedious, so I’ve aliased it to bi:

alias bi='bundle install --path .bundle/gems --binstubs .bundle/bin'

I’ve been using these options for a few months, and so far I’m very happy with them.

Working inside government

2012-02-29T00:00:00+00:00

Working inside government

Since September, I and the other Go Free Range guys have been working for the government. We’ve been helping the able and growing Government Digital Service to build whitehall, our code name for the Inside Government section of gov.uk. Yesterday we removed the password and opened our doors to the public.

For those who don’t know, gov.uk is the UK government’s attempt to not only consolidate government websites to a single domain, but also build these sites in the right way. That means public, open source, continual delivery and more.

While the first part of gov.uk focussed on helping citizens, Inside Government is about the business of government. What policies the government has, how it’s implementing them, not to mention the news and speeches, publications, consultations and other things the government does each day.

It’s only the first release of many, a small glimpse into what will one day be. How it ends up will be shaped by feedback from the people who use it.

Geohash toy: code released

2011-09-24T00:00:00+01:00

Geohash toy: code released

A couple of weeks ago I wrote about a small toy app I’d written to explore geohashes. Now I’ve cleaned the code up a little, upgraded it to rails 3.1 and released it here on github. Enjoy.

A small toy to explore geohashes

2011-09-15T00:00:00+01:00

A small toy to explore geohashes

For an app I’ve been building, I’ve been looking into geohashes. For those who don’t know, the geohash format is a simple way to encode latitude and longitude into a single string. As an example, Nelson’s Column in London (51.507794, -0.127952) has the geohash gcpvj0dyds.

Geohashes have a couple of interesting features. First, as you remove characters, you lose precision. gcpvj0dyds fairly accurately points to Nelson’s Column; gcpvj0d represents the South-West of Trafalgar Square and some of the Mall; and gcpvj covers most of Central London, as well as Islington and King’s Cross. A geohash doesn’t really represent a point, but rather a bounding area within which a point may lie. The longer the geohash, the smaller that bounding area.

The other interesting property geohashes have is that nearby locations usually (but not always) share similar prefixes. So much of North London is in gcpv, while much of South London is in gcpu. However, due to the Prime Meridian passing through Greenwhich, South East London has the geohash u10h – wildly different than the other two.

This probably sounds a bit complicated. I was having trouble getting my head around the concept, so to try and get to grips with geohashes I’ve written a toy app that draws them on a map. To try it out, go to http://geohash.gofreerange.com, click the map, zoom and play. If you find it useful, let me know.

Update: This code is now available [on github](https://github.com/tomafro/geohash-explorer).

Tip: Automatic bundle exec for rake and other gems

2011-09-01T00:00:00+01:00

Tip: Automatic bundle exec for rake and other gems

It’s irritating to run gem commands like rake, cap, rspec and others, only to find they needed to be executed via bundle exec. As a simple solution, I use a simple zsh function, combined with aliases for commonly used commands.

Here’s the function (which I’ve named be):

if [[ -a Gemfile ]]; then
  bundle exec $*
else
  command $*
fi

It’s very simple. If there’s a Gemfile in the pwd, it runs commands through bundle exec. Otherwise it just runs them.

I’ve combined this with some aliases for much less pain and less frustration:

alias rake='be rake'
alias cap='be cap'
alias rspec='be rspec'

Presenting the #blue api

2011-08-02T00:00:00+01:00

Presenting the #blue api

In building #blue (sign up now!), one of the problems we faced was how to build json data in response to requests to our API. The typical rails solution would be to override #as_json in a model class, then write a controller like this:

class ContactsController < ApiController
  responds_to :json

  def show
    respond_with Contact.find(params[:id])
  end
end

I always prefer to keep my controllers as skinny as possible, so this looks like a great solution. The respond_with call takes care of converting the message to json and responding with the right Content-Type, all in a simple call. However it has a number of problems and disadvantages.

The biggest issue for our API is that rather than expose the id of each model, we’ve tried to encourage the use of the uri instead. So the json returned for a single contact (for example) looks like this:

{
  "contact": {
    "uri": "https://api.example.com/contacts/ccpwjc",
    "name": "George",
    "email": "george@handmade.org",
    "msisdn": "447897897899",
    "phone_number": "07897897899",
    "messages": "https://api.example.com/contacts/ccpwjc/messages"
  }
}

It doesn’t just have a uri for the actual contact, but also for the messages belonging to that contact (and yes, I regret not calling that attribute messages_uri). Models can’t generate uris, and shouldn’t really be aware of them, so overriding #as_json doesn’t work. In any case, the json structure is really presentation logic, not business logic. It doesn’t belong in the model.

Presenting a single model

The solution we’ve used is to build a presenter for each model, solely responsible for building the json. Here’s an example for a contact:

class ContactPresenter
  include Rails.application.routes.url_helpers

  attr_accessor :controller, :subject
  delegate :params, :url_options, :to => :controller
  delegate :errors, :to => :subject

  def initialize(controller, subject)
    @controller = controller
    @subject = subject
  end

  def as_json(options = {})
    {:contact => {
      :uri => uri,
      :email => subject.email,
      :name => subject.name,
      :msisdn => subject.msisdn,
      :phone_number => subject.phone_number,
      :messages => api_contact_messages_url(:contact_id => subject.id)
    }
  end

  def uri
    api_contact_url(:id => subject.id)
  end
end

It’s now simple to rewrite our controller to use the new presenter:

class ContactsController < ApiController
  responds_to :json

  def show
    respond_with ContactPresenter.new(self, Contact.find(params[:id]))
  end
end

Presenting pages of models

The presenter above works well for a single model, but many of our API calls return a page of results. The /contacts for example returns all the contacts belonging to a user (of which there may be hundreds). Luckily it’s simple to adapt this pattern to present pages like this. First, we change our original #as_json method slightly:

def as_json(options = {})
  if options[:partial]
    {
      :uri => uri,
      :email => subject.email,
      :name => subject.name,
      :msisdn => subject.msisdn,
      :phone_number => subject.phone_number,
      :messages => api_contact_messages_url(:contact_id => subject.id)
    }
  else
    {:contact => as_json(:partial => true)}
  end
end

This change allows us to call as_json with the options :partial. With the option, a hash of data is returned. Without, the same hash is returned, wrapped in another hash.

Next, add a page presenter:

class ContactPagePresenter
  include Rails.application.routes.url_helpers

  attr_accessor :controller, :subject
  delegate :params, :url_options, :to => :controller
  delegate :errors, :to => :subject

  def initialize(controller, subject)
    @controller = controller
    @subject = subject
  end

  def as_json(options = {})
    contacts = subject.map {|o| ContactPresenter.new(controller, o).as_json(:partial => true) }

    {:contacts => contacts}.tap do |result|
      if subject.previous_page
        result[:previous_page_uri] = contacts_url(subject.previous_page)
      end

      if subject.next_page
        result[:next_page_uri] = contacts_url(subject.next_page)
      end
    end
  end
end

Finally, we can add an index action using this presenter:

class ContactsController < ApiController
  responds_to :json

  def index
    respond_with ContactPagePresenter.new(self, Contact.paginate(
      :page => params[:page], 
      :per_page => 50)
    )
  end
end

Refactoring common logic

The code above is a very much simplified version of what we do in #blue. We have many controllers, and several different models, so in our actual code we’ve abstracted out as much common logic as possible. In reality, our contacts controller looks more like this:

class ContactsController < ApiController
  before_filter :find_contact, :only => [:show, :update]

  def show
    present @contact
  end

  def index
    present_page_of current_account.contacts
  end

  def create
    @contact = current_account.contacts.build(attributes)
    @contact.save
    present @contact
  end

  def update
    @contact.update_attributes(attributes)
    present @contact
  end

  private

  def find_contact
    @contact = current_account.contacts.where(:_id => params[:id]).first
    head :status => :not_found unless @contact
  end
end

I think the code looks pretty clean. The clever stuff happens in the #present and #present_page_of methods, defined in the superclass:

class ApiController < ApplicationController::Base
  protected

  def present(instance, options = {})
    presenter = presenter_class.new(self, instance)
    options[:location] ||= presenter.uri if request.post? && subject.errors.empty?
    respond_with presenter, options
  end

  def present_page_of(collection, options = {})
    presenter = page_presenter_class.new(self, page_of(collection))
    respond_with presenter, options
  end

  def page_of(collection)
    collection.paginate(:page => params[:page], :per_page => 50)
  end

  def presenter_class
    (self.class.name.gsub!("Controller", "").singularize + "Presenter").constantize
  end

  def page_presenter_class
    (self.class.name.gsub!("Controller", "").singularize + "PagePresenter").constantize
  end
end

The #present and #present_page_of methods handle determining the correct presenter to us, as well as paginating the collection where required. They still use rails build in #respond_with method, which helps provide the correct response headers for each request. As the ContactPresenter delegates #errors to its subject, if there are validation errors, #respond_with correctly returns a 422.

One further motivation for this pattern (other than moving presentation logic out of the model) is that should we want to release a new version of our API, we’ll be able to get a lot of the way there simply by swapping which presenter is used. We started using this code about 8 months ago, and I’m still pretty happy with it. I hope you find something useful in it too.

Any comments or suggestions, please get in touch with me on twitter.

#blue opens for business

2011-03-03T00:00:00+00:00

#blue opens for business

If you’re an O2 customer based in the UK, you might be interested that #blue, a project we’ve built, has recently (re)opened for business. It’s a soft launch, but feel free to tell your friends.

#blue gets copies of every SMS message you send or receive, and makes them available to you on the web. You can search your messages, read whole conversations, and even reply, all from the comfort of your browser.

As well as a nice-looking web UI, there’s also an API that allows other apps to read and manipulate your messages and contacts (once you give them permission). Think automatic posting to twitter, or showing new messages as alerts on your desktop. The possibilities are endless.

Oh, and it’s all free. All you need is an O2 phone.

If you think it sounds interesting, ask for a beta request. You should get an invitation very quickly. Once on board, please send me any suggestions and feedback. It’s going to be interesting to see where we can take this project.