imedo Development Blog

Webrat supports file upload in web forms with the method attach_file. Due to a lack of multipart form support for integration tests in Action Pack 2.0.2, this does not work with Rails 2.0.2.

It has been added in version 2.1., but for those who are still using 2.0.2, with the following code you can upgrade the particular part of Action Pack to 2.1. Just add it to your tests so that it overwrites the loaded Action Pack code.

AKPC_IDS += "51702,";

Popularity: 1% [?]

To use rails with rack support you can either use the Ezras Rails fork (or edge Rails) or the simpler alternative to get started is just use the rack adapter included in “thin” or “fuzed” (which is basically the same thing – or each of them is based on the other one… or something. Quite self-referential)

To start a rails app with rack just fire up an plain irb and type this:

require 'rack'
require "~/tmp/fuzed/rlibs/rails_adapter.rb"
#=> true
app = Rack::Adapter::Rails.new(:root => "/Users/hvolkmer/imedo/code/imedo")

Now create a request and let your rails app handle it:

req = {"METHOD" => "GET",
 "HTTP_VERSION" => [1, 1],
 "PATH_INFO" => "/",
 "QUERY_STRING" => "",
 "SERVERNAME" => "testing:8002",
 "HEADERS" => {"connection" => "keep-alive",
              "accept" => "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
              "host" => "localhost:8002",
              "referer" => "http://localhost:8002/main/ready",
              "user_agent" => "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3",
              "keep_alive" => "300",
              "content_length" => "7",
              "content_type" => "application/x-www-form-urlencoded",
              "Cache-Control" => "max-age=0",
              "Accept-Charset" => "ISO-8859-1,utf-8;q=0.7,*;q=0.7",
              "Accept-Encoding" => "gzip,deflate",
              "Accept-Language" => "en-us,en;q=0.5"},
 "HTTP_COOKIE" => "_some_session_id=d3eae987aab3230377abc433b7a8d7c1",
 "postdata" => "val=foo"}

 response = app.call(req)

Now why is this useful?

Well, you can test your Rails app without a webserver and still utilize the full real application stack (unlike in test environment). We currently use this to get some information about memory usage of certain parts of the application. With rack there is very little overhead (rack itself) compared to say mongrel or thin. Mongrel and thin have their own behaviour when it comes to high loads due to the request queue. You can easily script a load test using rack with just some ruby scripts. No need for httperf or siege . These tools are really useful to test real life loads but when you just want to see how a certain part of your app behaves when you put x requests at it, rack can do this pretty well.

And it’s always a good idea to look at things in isolation. Rack does a pretty good job for that. Also who needs a web browser when you can look at your rails app in irb

Popularity: 1% [?]

Recently, I ported our environment.rb to the new Rails Initializers. Over time, we added a lot of initialization code in environment.rb. In particular, there are a lot of small monkey-patches that we apply to rails, or 3rd-party plugins. So for that, the Rails Initializers are a very good fit.

Still, after I cleaned up that part, there were still things like this in environment.rb:

google_maps_api_key = case RAILS_ENV
when 'test' then "the-test-key"
when 'development' then "the-development-key"
when 'production' then "the-production-key"
end

GeoKit::Geocoders.google = google_maps_api_key

While this could arguably be handled by a YAML configuration file, unfortunately GeoKit (at least the version we use right now) does not support configuration via YAML.

Another thing we do is disable Globalize in the development environment to speed up page rendering on our local machines. So the corresponding part of the environment.rb file looked something like

if RAILS_ENV == 'production'
  include Globalize
else
  # In reality, this is slightly bigger, to simulate more methods
  # of the Globalize API
  class String; def t; self; end; end
end

Hardcoding environment conditionals is somewhat like hardcoding admin privileges to the user with ID 1. Rather, an environment should be seen as a generic container for a bunch of configuration options. So I came up with a better way of using Rails Initializers like follows:

By default, Rails loads every Ruby file in config/initializers and its subdirectories after the plugins are fully loaded. This is handlet by the Rails::Initializer#load_application_initializers method. I wrote a small monkey patch to change this behavior such that only the files in config/initializers (without subdirectories) are loaded, followed by the files in config/initializers/RAILS_ENV.

class Rails::Initializer
  def load_application_initializers
    ["#{configuration.root_path}/config/initializers/*.rb",
     "#{configuration.root_path}/config/initializers/#{RAILS_ENV}/**/*.rb"].each do |path|
      Dir[path].sort.each do |initializer|
        load(initializer)
      end
    end
  end
end

With this code, our Globalize initialization is distributed across two files:

config/initializers/globalize.rb

class String
  def t
    self
  end
end

config/initializers/production/globalize.rb

include Globalize

Note: In recent Rails versions, the original load_application_initializers method is slightly more complex, such that the following patch would be used instead (warning: I have not tried that one just yet, since we’re still porting to Rails 2.1)

class Rails::Initializer
  def load_application_initializers
    return unless gems_dependencies_loaded

    ["#{configuration.root_path}/config/initializers/*.rb",
     "#{configuration.root_path}/config/initializers/#{RAILS_ENV}/**/*.rb"].each do |path|
      Dir[path].sort.each do |initializer|
        load(initializer)
      end
    end
  end
end

Note that this monkey patch can not be applied as a Rails Initializer, since it patches the way Rails Initializers are loaded (though it would be awesome, if something like this would be possible; effectively patching a method while it is executed ). This means that you need to require the file containing the monkey patch from environment.rb, before you run the Rails Initializer.

Enjoy your cleaned up environment.rb and initializers.

Popularity: 6% [?]

As requested here is the presentation for our awesome email plugin:

Slides

Popularity: 2% [?]

I don’t know why we didn’t try that earlier, but haproxy is a far better proxy solution then all the others we tried. Ok, we didn’t try that many but the most common ones on the Rails deployment landscape:

• Apache 2.2 mod_proxy_balancer
• nginx standard proxy module
• nginx with fair proxy module
• haproxy

So why is haproxy so much better? It is better because it actually can limit the requests per mongrel to one at a time. This important because otherwise you get behavour like this: One mongrel has a long running request and through round-robin, gets another request while other mongrels are idleing. Also mongrels with a request queue bigger then one start to eat memory like hell.

The same thing is possible with Apache’s mod_proxy_balancer. We tried and failed to get it working. And it seems as if we are not alone with that problem

The plain nginx balancer has the same problem. This is where the fair proxy module comes in. It’s supposed to send requests only to idleing mongrels. But we had the same “mongrels with many requests while others are ideling” problem again.

We finally tried haproxy which is in use for Rails deployments for quite some time but got a lot of buzz recently. Ilya Grigorik wrote a nice article about load balancing QoS with haproxy and Alexander Staubo posted a performance comparison of nginx and haproxy which got the attention of William Tarreau (the haproxy author). They found some haproxy bugs which got fixed and resulted in an even better performance. Details can be found in the second comparison of haproxy and nginx

So we are quite happy with haproxy at the moment and hope it stays this way.

Popularity: 1% [?]

If you ever wrote something similar like this (or more complex constructs):

<%= @jadda.name rescue @jadda.nickname rescue 'unknown' %>

and then wished you had a construct like try(*these) in prototype? Surrender not, for there’s light at the end of the tunnel!

I present you try these which I found on Chu Yeow’s Blog today.

Here’s the mixins code:

1
2
3
4
5
6
7
8

module Kernel
  def try(*these)
    raise ArgumentError, 'try requires at least 2 arguments' if these.size <= 1
    fallback = these.pop unless these.last.respond_to?(:call)
    these.each { |candidate| begin return candidate.call rescue next end }
    fallback || raise(RuntimeError, 'None of the given procs succeeded')
  end
end

That’s all. Use it like this:

<%= try(lambda{ @jadda.name }, lambda{ @jadda }, lambda{ 'unknown' }) %>

and it will call all these blocks until one will yield no exception or the end of the array is reached. If one yields a result it will end the loop and return the value yielded by that block.

Hope this is useful for someone.

Popularity: 1% [?]

I want to say a few things about a I read lately, called Design Patterns in Ruby.

In this book, Russ Olsen discusses 14 out of the original 23 design patterns described by the GoF. In addition he also talks about 3 not unique but very common patterns in Ruby. Alongside he gives a very nice overview of Ruby for the Ruby beginner and shows some sweet little tricks.

I find the book very comprehensive with the special paragraphs of when to not use a certain pattern, which is really helpful to not get overly enthusiastic with a pattern and try to use it everywhere. Design Patterns are after all solutions that you should use when you found a fitting problem and not when you made a problem fit.

Eventually I created a presentation with a short (not so short, actually) overview of the patterns for my colleagues which I want to share with all of you here.

The presentation on video:

Online Videos by Veoh.com

The slides:

Special Thanks

to Jonathan Weiss, who’s introduction and review pointed me to this book.

Popularity: 1% [?]

For the impatient

• Project Page
• Documentation
• Download
• Github repository

What is it?

Imagine you have some fancy acts_as_something plugin and, of course, you want to test it. How do you do it? In the Rails app you are developing it for? This could lead to longer environment load times and this is hardly testing in isolation. Creating a rails app just for testing the plugin? That’s better but creating a rails app for every new plugin – that’s so un-DRY! And you cannot test the plugin on its own. You could ship the bare rails app with your plugin but again: not DRY and lots of code you don’t really need for your plugin.

dry_plugin_test_helper to the rescue!

What it does

The gem sets up the test environment which means you have a stub rails app with your plugin and the following models:

• Article: belongs_to :author, has_many :comments
• Author: has_many :articles
• Comment: belongs_to :articles, belongs_to :user
• User: has_many :comments

The models will be added to a sqlite in memory database for fast testing.

To use the gem enter the following lines in your test_helper.rb or abstract_unit.rb (or however you may call it):

1
2
3
4

require 'rubygems'
require 'dry_plugin_test_helper'

PluginTestEnvironment.initialize_environment(File.dirname(__FILE__))

You can add your own test models using a migration in your test directory like
this:

1
2
3
4
5
6
7
8

 PluginTestEnvironment::Migration.setup do 

   create_table "animals", :force => true do |t|
     t.column "name", :string
     t.column "age",  :integer
   end

 end

If you don’t want the standard models you can initialize the test environment
like this:

PluginTestEnvironment.initialize_environment(File.dirname(__FILE__), :use_standard_migration => false)

How it works

The gem contains a stripped down rails directory structure and boots up this rails environment and then adds the plugin under test to this environment. It uses the sqlite3 in memory db for faster testing.

Caveats

You can really only test the plugin that you are testing. If your plugin has some dependencies to other plugins (why should it?) you cannot test it using this gem out of he box. You’d have to mess with the load path yourself (I haven’t tried it)

Todo

• support for RSpec and other testing frameworks (might already work – haven’t tried it)
• Automatic Rails version discovery
• Clean up / strip down rails env to bare minimum (It’s not completly clean yet)

Dependencies

The dry_plugin_test_helper gem depends on Rails and sqlite3-ruby

Getting it, License and Patches

Get the complete source code through Github. License is MIT. That means that you can do whatever you want with the software, as long as the copyright statement stays intact. Please be a kind open source citizen, and give back your patches and extensions. Just fork the code on Github, and after you’re done, send us a pull request. Thanks for your help!

• Project Page
• Documentation
• Download
• Github repository

Popularity: 1% [?]

Update

There is a new version available. More details are available here.

For the impatient

• Project Page
• Documentation
• Download
• Github repository

Introduction

Every Rails developer knows this: Your application is fast and responsive, up until a point where the data set handled in one request gets so large that request times become unacceptable. The obvious solution is to identify code that does not have to run immediately, but can be delegated to another process. There are several ready-made solutions for delegating execution of code to a background process, like ActiveMessaging using ActiveMQ, BackgrounDRb using a DRb server, or databased-driven Job queues.

The problems with each of the above mentioned approaches are:

• Responsibilities get cut out of objects (most of the time, the code running in the background process actually belongs to an object residing in the foreground process). This can be solved by delegating the time-consuming task to the background process, which clones the object, and delegates the task back to it, only in another process.
• Not DRY. Background processes tend to repeat code, like the above mentioned back-delegation. This can be reduced to a certain extent, but there will always be some overhead.
• Not Failsafe. Almost always, the task at hand is delegated over a socket of some sort. If the other process is busy, hangs or is down, there can be timeouts which result in ugly exceptions, and in the end, discard the task. To make background processing failsafe, you need to write a lot more code, which is repetitive as well.

We want to present our solution to all of the above problems, that also is very elegant. Running a task in the background is as easy as

background do
  # run your code
end

The communication with the background process is configurable, as is the error handling. For example, you could use ActiveMQ for queueing your background tasks. If the connection to ActiveMQ fails for some reason, the task could be executed in-process. If this fails (e.g. because of an error or timeout), the task would be dumped to disk for a later replay when the problem is fixed.

How does it work?

Actually, the solution is pretty easy. The block’s code and local variables needed by the block are serialized and sent to the handler, which then evaluates the block in the context of the local variables. The attentive reader might notice that it is impossible to serialize code blocks, let alone know all the block’s local variables in advance. Well, that is almost true.

There is a genius piece of code, called proc_source, that allows you to serialize code blocks. It works by parsing the source file that contains the block. This is possible, because code blocks know where they are in the source code.

It turns out that the local variables can’t be accessed with plain Ruby. But that is actually a good thing, because we might want to control which objects are sent over the wire, and only choose the ones that are actually needed, in order to save computation time and bandwidth.

So, to attach local variables to the code block, you’d use the following method call:

background :locals => { :user => current_user } do
  user.do_some_time_consuming_operation
end

Failsafe background processing

To make sure that your task is executed even when your background process of choice is not available, you can specify a handler, and a couple of fallback handlers. The handler and fallback handlers are tried in order, until the first one succeeds.

background :locals => { :user => current_user },
                  :handler => :active_messaging,
                  :fallback => [:in_process, :disk, :forget] do
  user.do_some_time_consuming_operation
end

In this example, the :active_messaging handler is tried first. If it fails, the code is executed in-process. If it still fails, the code is dumped into a file, and if even this fails, the task is discarded.

The self object

One of the amazing things is that you can use the self keyword inside blocks. This works because the object, in which the code is executed, is serialized as well.

class User
  def some_operation
    variable = some_evaluation
    background :locals => { :variable => variable } do
      self.do_something_with(variable)
    end
  end
end

Error reporting

As a developer, you might want to be informed when something goes wrong, in order to fix it. But since every project uses a different error reporting system, the error reporting is configurable. Also, some code is rather important, while other code is optional, so that you might not want to be informed about every error. To specify the error reporting, use the optional :reporter parameter:

background :reporter => :exception_notification do
  # background code
end

Note that errors are only reported if an exception occurs while talking to the background process; if you want to be informed when an error occurs while the block is executed in the other process, you need to implement your own reporting for the backgrond process.

Decorating existing methods

Most of the time, you want a whole method to be executed in another process. To make this pattern DRY, the background method can be used as a method decorator, when it is called in class-level scope:

class User
  def do_something_complicated(parameter, argument)
    # complicated things
  end

  # execute all calls to do_something_complicated in the background
  background :do_something_complicated, :params => ['parameter', 'argument']
end

Note that you have to specify all of the methods parameter names as they are written in the original method’s definiton. This is not 100% DRY, but neccessary to correctly send all the parameters to the background process. If you have a suggestion on how to avoid this, please let us know.

Default configuration

You can configure the default background handler, a default fallback chain as well as a default error reporter. The configuration lies in the Background::Config class.

Security Issues

As with any background processing, you need to be careful about the requests that are processed. Since the background plugin executes arbitrary Ruby code, you need to take special care that no unfiltered user input is injected. Make sure that your firewall does not allow connections from the outside, and that the code that connects from the inside is controlled by you. We don’t take any responsibility for any damage caused by the operation of the background plugin.

Limitations

Since singleton objects can not be serialized, all of the singleton methods are stripped away before objects are sent to the background process. Be aware of this fact if you rely on these methods. Most of the time, it should be easy to extend the objects again inside the code block.

Dependencies

The background plugin depends only on ActiveSupport, which is part of Rails.

Getting it, License and Patches

Get the complete source code through Github. License is MIT. That means that you can do whatever you want with the software, as long as the copyright statement stays intact. Please be a kind open source citizen, and give back your patches and extensions. Just fork the code on Github, and after you’re done, send us a pull request. Thanks for your help!

• Project Page
• Documentation
• Download
• Github repository

Popularity: 30% [?]

We recently stumbled upon a strange behaviour of ActionMailer when we tried to deliver multipart e-mails with an UTF-8 subject.

One would expect a correctly quoted subject along the lines of:

Subject: =?utf-8?Q?=5bBenachrichtigung_von_imedo=2ede=5d_xyz_hat_Sie_gedr=c3=bcckt?=

but instead we got an encoded string without the charset like this:

Subject: =??Q?=5bBenachrichtigung_von_imedo=2ede=5d_xyz_hat_Sie_gedr=c3=bcckt?=

Our investigations showed, that only multipart mails seem to show this behaviour and setting the charset explicitly in the deliver-method of the mailer doesn’t help.

So to get the task done we are currently using quoted_printable directly to encode the subject line, as all the other parts work just fine.

subject  quoted_printable(("[Benachrichtigung von imedo] XYZ hat sie gedrückt", 'utf-8')

We will look into submitting a patch for Rails, but for the time beeing this works just fine and we hope that it will help others that encounter the same problem.

Popularity: 1% [?]