imedo Development Blog

I don’t know why we didn’t try that earlier, but haproxy is a far better proxy solution then all the others we tried. Ok, we didn’t try that many but the most common ones on the Rails deployment landscape:

• Apache 2.2 mod_proxy_balancer
• nginx standard proxy module
• nginx with fair proxy module
• haproxy

So why is haproxy so much better? It is better because it actually can limit the requests per mongrel to one at a time. This important because otherwise you get behavour like this: One mongrel has a long running request and through round-robin, gets another request while other mongrels are idleing. Also mongrels with a request queue bigger then one start to eat memory like hell.

The same thing is possible with Apache’s mod_proxy_balancer. We tried and failed to get it working. And it seems as if we are not alone with that problem

The plain nginx balancer has the same problem. This is where the fair proxy module comes in. It’s supposed to send requests only to idleing mongrels. But we had the same “mongrels with many requests while others are ideling” problem again.

We finally tried haproxy which is in use for Rails deployments for quite some time but got a lot of buzz recently. Ilya Grigorik wrote a nice article about load balancing QoS with haproxy and Alexander Staubo posted a performance comparison of nginx and haproxy which got the attention of William Tarreau (the haproxy author). They found some haproxy bugs which got fixed and resulted in an even better performance. Details can be found in the second comparison of haproxy and nginx

So we are quite happy with haproxy at the moment and hope it stays this way.

Popularity: 1% [?]

Ilya Grigorik wrote about mongrel proctitle plugin which is indeed very useful. We discovered the mongrel plugin about two months ago and use it in production without problems. It has helped us to debug several problems.

We run more than one Rails app per server and so it hurts to install the plugin for every app again (oh that painful DRY nerve )

The solution of course is to just create a gem and install it. Luckily there is already such a thing: rtomayko’s mongrel proctitle github repository

There’s also a proctitle plugin (actually a rackup file) for thin: thin proctitle git repository – no gem yet, though.

Popularity: 1% [?]

So you wake up, brew a coffee check the system status and realize that some mongrels aren’t running despite the fact, that monit should take care of them. So just start them manually and see what happens, right?

$ mongrel_rails cluster::restart -C /path_to/mongrel_cluster.yml --clean --only 8101
** Ruby version is not up-to-date; loading cgi_multipart_eof_fix
invalid option: --clean for command 'cluster::restart'

What? Why?

That’s exactly the command that monit has been using for almost a year in our production system…

So what changed? Apparently not much… just another gem was installed in preparation for the next deploy. But mongrel and mongrel_rails are still the same version. That’s strange.

I haven’t really found out what was wrong with the gem, but I’ve found a solution which is also mentioned in Paul Goscicki article :


gem cleanup mongrel_cluster


Just wipe all old mongrel_cluster gems and you’re good to go. YMMV. Friday saved.

Popularity: 1% [?]