We have some quite large tables where we need to iterate over each element on a daily basis. Using find(:all) for this purpose can be really memory consuming, since every element must be fetched from the database and must be kept in RAM until the last iteration is over. Since we recently touched the limits of our available memory in production, we had to come up with a different way to iterate over all records. The obvious solution is to fetch, say, 1000 records at a time, like so:

count = User.count
steps = count / 1000
steps.times do |i|
  User.find(:all, :limit => 1000,  ffset => i * 1000).each do |user|
    # do your stuff here
  end
end

This works pretty well, but since there are so many places where we iterate over large tables, we want to keep it DRY from the beginning. So here is an ActiveRecord extension that does exactly that:

class ActiveRecord::Base
  def self.for_each(conditions = nil, step = 1000, &block)
    c = count :conditions => conditions
    (c / step).times do |i|
      find(:all, :conditions => conditions,
           :limit => step,  ffset => step * i).each do |model|
        yield model
      end
    end
  end
end

Usage of this code snippet is straightforward. To iterate over all users, you’d write:

User.for_each do |user|
  # do stuff here
end

Conditions are also supported, so if you want to iterate over all users who joined today, you’d write:

User.for_each ['created_at > ?', Date.today.to_time] do |user|
  # do stuff
end

Some things to note:

• The order clause is not supported. It would not make sense anyways, since you are iterating over every record.
• Don’t modify the table you are iterating over in ways that influence the search condition. Otherwise it could happen that records are skipped.
• When using find(:all), the results are a snapshot of the database from the time of the query. Subsequent modifications are not taken into account, which is usually desirable. With the above approach, you must be aware that the table is queried several times, possibly with modifications from other sources (this is related to “don’t modify the table”, above)

Popularity: 1% [?]

RailsConf Europe 2008

Session – Design for Usability

presented by Christian Lupp

In his talk at RubyConf Europe 2008 Christian Lupp showed a designers approach to solving problems. It my seem strange to have a designer give a talk in front of a developer crowd, but I think you will find, that we can learn a few things by observing other professions’ methodologies.

He told the audience that many great ideas have evolved on paper. You start out with rough sketches and prototypes. In this stage you basically create a landscape of your application and it should help you to understand what the problems are you want to solve. Once you’ve dont that, you redefine these prototypes iteratetively. Be confident to throw stuff away that isn’t working out and explore alternative solutions. Fail fast and early.

Hmm, this all sounds an aweful lot much like agile software development, doesn’t it?

He also mentioned that you should always take the context into account when constructing your application. Without thinking about the user’s perspective and his needs, may it be a human or a machine, it’s likely that you develop in the wrong direction.

If you have two equal solutions to a problem, take the simple solution. This is called the Occham’s razor Behold! Knowledge from the 14th century folks, but still true. In order to find the simplest solution, you reduce functionality until your application is missing something important, then you go one step back.

Then the talk got technical after all.

Christian asked why we as web developers should use techniques like unobstrusive javascript at all and listed some valid points. First, people using screenreaders will love you. Second, the marketing and SEO guys will love you. You improve the overall performance of your website and thus speed up the overall user experience on your website. Users don’t like to wait. If a page is snappy and responsive, they are more likely to come back.

Christian also shared some basic design tipps with us.

Repeat with DRY. Don’t show different elements on every page. Instead develop an overall graphical and UX concept. It doesn’t only make you site look more professional but will also help users to find their way around on your site. Usability equals recognizable patterns, so repeat yourself and use layout grids etc. that will help your users to understand the content of your site faster.

Another good tipp is to always validate user input immediately by javascript for example. Give your user feedback all the time and instantly, eventough the action he started may take longer. Be creative and entertain you user while he is waiting for search results. Instead of letting the user wait for the search to finish, you can show him an immediary screen which informs him that his search is beeing performed and could take a few moments to finish. You informed your user and in the backgorund your server is already working on finding relevant results for him. This pattern is already used by most forum software.

That was “Design for Usability”, a very interesting talk.

Popularity: 1% [?]