imedo Development Blog

Recently, I held a presentation about the background plugin, at the Ruby User Group Berlin. The presentation was very well received, and the effect was that our watchlist on github got a boost.

Anyways, here is the presentation. Enjoy.

Popularity: 1% [?]

Recently, we released the background plugin, that allows you to run any Ruby code block in the background. We primarily use it in combination with ActiveMessaging, but any background processing framework like message queues, job queues or background tasks can be used with it.

Some people have asked us if we could add support to configuring the backend, such that it is possible to choose the queue in ActiveMessaging over which the block is sent. To keep it flexible for future releases, it is possible to configure any backend with an options hash like this:

background :handler => [ { :active_messaging => { :queue => :my_queue } },
                         { :disk => { :directory => '/tmp/my_queue' } },
                           :forget ] do
  # do your task
end

Granted that the example above is a little verbose, but usually you wouldn’t have to configure that much. However, if you do, you could wrap the example above like this:

def my_queue(&block)
  background :handler => [ { :active_messaging => { :queue => :my_queue } },
                           { :disk => { :directory => '/tmp/my_queue' } },
                             :forget ] do
    yield
  end
end

Note that the :fallback option of the last version is no longer available; just give an array to the :handler option.

Getting it, License and Patches

Get the complete source code through Github. License is MIT. That means that you can do whatever you want with the software, as long as the copyright statement stays intact. Please be a kind open source citizen, and give back your patches and extensions. Just fork the code on Github, and after you’re done, send us a pull request. Thanks for your help!

• Project Page
• Documentation
• Download
• Github repository

Popularity: 1% [?]

Update

There is a new version available. More details are available here.

For the impatient

• Project Page
• Documentation
• Download
• Github repository

Introduction

Every Rails developer knows this: Your application is fast and responsive, up until a point where the data set handled in one request gets so large that request times become unacceptable. The obvious solution is to identify code that does not have to run immediately, but can be delegated to another process. There are several ready-made solutions for delegating execution of code to a background process, like ActiveMessaging using ActiveMQ, BackgrounDRb using a DRb server, or databased-driven Job queues.

The problems with each of the above mentioned approaches are:

• Responsibilities get cut out of objects (most of the time, the code running in the background process actually belongs to an object residing in the foreground process). This can be solved by delegating the time-consuming task to the background process, which clones the object, and delegates the task back to it, only in another process.
• Not DRY. Background processes tend to repeat code, like the above mentioned back-delegation. This can be reduced to a certain extent, but there will always be some overhead.
• Not Failsafe. Almost always, the task at hand is delegated over a socket of some sort. If the other process is busy, hangs or is down, there can be timeouts which result in ugly exceptions, and in the end, discard the task. To make background processing failsafe, you need to write a lot more code, which is repetitive as well.

We want to present our solution to all of the above problems, that also is very elegant. Running a task in the background is as easy as

background do
  # run your code
end

The communication with the background process is configurable, as is the error handling. For example, you could use ActiveMQ for queueing your background tasks. If the connection to ActiveMQ fails for some reason, the task could be executed in-process. If this fails (e.g. because of an error or timeout), the task would be dumped to disk for a later replay when the problem is fixed.

How does it work?

Actually, the solution is pretty easy. The block’s code and local variables needed by the block are serialized and sent to the handler, which then evaluates the block in the context of the local variables. The attentive reader might notice that it is impossible to serialize code blocks, let alone know all the block’s local variables in advance. Well, that is almost true.

There is a genius piece of code, called proc_source, that allows you to serialize code blocks. It works by parsing the source file that contains the block. This is possible, because code blocks know where they are in the source code.

It turns out that the local variables can’t be accessed with plain Ruby. But that is actually a good thing, because we might want to control which objects are sent over the wire, and only choose the ones that are actually needed, in order to save computation time and bandwidth.

So, to attach local variables to the code block, you’d use the following method call:

background :locals => { :user => current_user } do
  user.do_some_time_consuming_operation
end

Failsafe background processing

To make sure that your task is executed even when your background process of choice is not available, you can specify a handler, and a couple of fallback handlers. The handler and fallback handlers are tried in order, until the first one succeeds.

background :locals => { :user => current_user },
                  :handler => :active_messaging,
                  :fallback => [:in_process, :disk, :forget] do
  user.do_some_time_consuming_operation
end

In this example, the :active_messaging handler is tried first. If it fails, the code is executed in-process. If it still fails, the code is dumped into a file, and if even this fails, the task is discarded.

The self object

One of the amazing things is that you can use the self keyword inside blocks. This works because the object, in which the code is executed, is serialized as well.

class User
  def some_operation
    variable = some_evaluation
    background :locals => { :variable => variable } do
      self.do_something_with(variable)
    end
  end
end

Error reporting

As a developer, you might want to be informed when something goes wrong, in order to fix it. But since every project uses a different error reporting system, the error reporting is configurable. Also, some code is rather important, while other code is optional, so that you might not want to be informed about every error. To specify the error reporting, use the optional :reporter parameter:

background :reporter => :exception_notification do
  # background code
end

Note that errors are only reported if an exception occurs while talking to the background process; if you want to be informed when an error occurs while the block is executed in the other process, you need to implement your own reporting for the backgrond process.

Decorating existing methods

Most of the time, you want a whole method to be executed in another process. To make this pattern DRY, the background method can be used as a method decorator, when it is called in class-level scope:

class User
  def do_something_complicated(parameter, argument)
    # complicated things
  end

  # execute all calls to do_something_complicated in the background
  background :do_something_complicated, :params => ['parameter', 'argument']
end

Note that you have to specify all of the methods parameter names as they are written in the original method’s definiton. This is not 100% DRY, but neccessary to correctly send all the parameters to the background process. If you have a suggestion on how to avoid this, please let us know.

Default configuration

You can configure the default background handler, a default fallback chain as well as a default error reporter. The configuration lies in the Background::Config class.

Security Issues

As with any background processing, you need to be careful about the requests that are processed. Since the background plugin executes arbitrary Ruby code, you need to take special care that no unfiltered user input is injected. Make sure that your firewall does not allow connections from the outside, and that the code that connects from the inside is controlled by you. We don’t take any responsibility for any damage caused by the operation of the background plugin.

Limitations

Since singleton objects can not be serialized, all of the singleton methods are stripped away before objects are sent to the background process. Be aware of this fact if you rely on these methods. Most of the time, it should be easy to extend the objects again inside the code block.

Dependencies

The background plugin depends only on ActiveSupport, which is part of Rails.

Getting it, License and Patches

Get the complete source code through Github. License is MIT. That means that you can do whatever you want with the software, as long as the copyright statement stays intact. Please be a kind open source citizen, and give back your patches and extensions. Just fork the code on Github, and after you’re done, send us a pull request. Thanks for your help!

• Project Page
• Documentation
• Download
• Github repository

Popularity: 1% [?]

Welcome to the first part of our blog series about optimizing rails views. This part is about remove white space.

Introduction

In order to save bandwidth, imedo.de packages it’s rails views on deploy. This is accomplished with a simple rake task, that parses every erb file and removes unneccessary white space. But what is unneccessary white space? First of all, unless we’re in a pre tag, subsequent white space characters are displayed as one character in HTML (for the sake of simplicity, we will ignore pre tags for now). As it turns out, unneccessary single white space characters are hard to distinguish from neccessary single white space characters. Consider the following example (for clarity, the white space character is displayed as an under score, and the new line character is displayed as a question mark):

<a_href="/some/url">Some_url</a>_|?
<a_href="/another/url">Another_url</a>

Rendered in HTML, this code looks like the folloing

Some url |
Another url

Packaging this code simply by removing newlines, for instance, breaks the layout:

<a_href="/some/url">Some_url</a>_|<a_href="/another/url">Another_url</a>

Some url |Another url

Note that the pipe symbol is now attached to the second link, which is not the intended result.

An obvious solution to this problem is to convert every white space character to a regular space character, and then remove duplicates. But this does not yield optimal results, as can be seen in this example:

<ul>?
__<li>?
____<a_href="/some/url">Some_url</a>?
__</li>?
__<li>?
____<a_href="/another/url">Another_url</a>?
__</li>?
</ul>?

Which, compressed by the above algorithm, leads to the following markup:

<ul>_<li>_<a_href="/some/url">Some_url</a>_</li>_<li>_<a_href="/another/url">Another_url</a>_</li>_</ul>_

whereas the optimal solution in this particular example would be to remove any white space characters between tags:

<ul><li><a_href="/some/url">Some_url</a></li><li><a_href="/another/url">Another_url</a></li></ul>

Note that it is not sufficient to remove all white-space-only text nodes in the DOM tree.

Aggressive Packaging

As we said above, it’s hard to detect neccessary white space. neccessary white space depends on so many factors, that not even all browsers get it right. That’s why we don’t even bother trying. So the simple rule is:

Don’t depend on new lines and indentation showing up as space.

(See the section Conservative Packaging of this article, if you can’t follow this rule). This is a bad practice, anyway. A visible space between words should be treated as a first-class HTML citizen, and should not be the result of some random code beautification things, like line breaks or indentation. Actually, as I packaged all of our view files (about 2000 or so) for the first time, there were only a handful of places where we depended on indentation or new lines showing up as spaces. To get rid of these implicit whitespace dependencies is easy, particularly if you use Globalize (an internationalization plugin, that forces you to write any text as an erb tag) for every user-visible hardcoded string. Just make sure, that you don’t have a word as the last token in a line, with the next word as the first token in the following line. So, this is bad practice:

<p>This is
bad practice.</p>
<p>There is the
<em>next</em> paragraph.</p>

while this is good practice:

<p>This is good practice</p>
<p>There is the <em>next</em> paragraph.</p>

With this complication eliminated, it turned out to be very simple to compress the partials. ERB comes packaged with a lexer class called ERB::Compiler::TrimScanner that splits an erb view into small pieces, in a way that is perfect for this task. One small thing to note is that we don’t want to remove any white space characters inside erb tags (<%= ... %>), since white space, especially the newline character, sometimes matters in Ruby.

So, the code to compress a view file is as short as this:

# Weird TrimScanner bug: Lines consisting only of %> (without whitespace) are lexed as >
content = File.read(file).split("\n").collect { |line| line == '%>' ? ' %>' : line }.join("\n")
scanner = ERB::Compiler::TrimScanner.new(content, '>', true)
array = []
scanner.scan { |x| array << x }

inside = false # true, if we are inside an erb tag
# remove whitespace
content = array.collect do |e|
  if e =~ /<%/
    inside = true
  elsif e =~ /%>/
    inside = false
  end
  if e.is_a?(Symbol) || e.is_a?(ERB::Compiler::PercentLine)
    nil
  elsif inside
    e
  else
    e.strip.gsub(/\s+/, ' ')
  end
end.join

File.open(file, 'w') {|file| file.print content}

This very aggressive white space removal works for us, because we have

• no inline Javascript, that might depend on consecutive spaces.
• no actual text written in the views, but strings that are translated with Globalize on the fly (so there is no new line in the middle of a text that might glue words together).
• every user-visible text embedded in a paragraph or span tag.
• the closing tag of a DOM element containing text (i.e. where whitespace matters) on the same line as the opening text.

That being said, we still have to disable white space removal in certain instances, like text/plain emails.

Conservative Packaging

For those of you who do depend on newlines as white space, just remove the strip call in line ? of the code snippet above. The compression will not be optimal, but better than no compression at all.

Disclaimer

Important: If you try any of the view optimizing techniques, please make sure that you have a quick and painless way to revert the changes, because files are rewritten with optimized content. We are not responsible for any damage that any of the code may have caused.

Download and installation

• Download the view transformer plugin here.
• Install it by copying the plugin in your vendor/plugins folder.

Configuration

To configure the view translator, modify the file view_transformer.yml.

Popularity: 1% [?]