Saturday, January 31, 2009

Benchmarking JRuby on Rails

Last night, while working on a project I found a really neat use of Rails Components, but I also noticed that this part of Rails is deprecated, among other reasons because it's slow.

Well, how slow? During my quest to find out, I collected some interesting data, and even more importantly put JRuby and MRI Ruby face to face.

Disclaimer: the benchmarks were not done on a well isolated and specially configured test harness, but I did my best to gather data with informational value. All the components were used with OOB settings.

Setup

  • ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] + Mongrel Web Server 1.1.4
  • jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [x86_64-java] + GlassFish gem version: 0.9.2
  • common backend: mysql5 5.0.75 Source distribution (InnoDB table engine, Rails pool set to 30)


Benchmarks

I used an excellent high quality benchmarking framework Faban for my tests. I was lazy, so I only used fhb (very similar to ab, but without its flaws) to invoke simple benchmarks:
  • simple request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/buckets/1
  • component request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/bucket1/object1
Both tests were run with JRuby as well as with RMI Ruby and in addition to that I ran the tests with Rails in single-threaded as well as multi-threaded modes. I didn't use mongler clusters or glassfish pooled instances - there was always only one Ruby instance serving all the requests.

Results

ruby 1.8.6 + mongrel
---------------------------------
simple action + single-threaded:
ops/sec: 210.900
% errors: 0.0
avg. time: 0.047
max time: 0.382
90th %: 0.095

simple action + multi-threaded:
ops/sec: 226.483
% errors: 0.0
avg. time: 0.044
max time: 0.180
90th %: 0.095

component action + single-threaded:
ops/sec: 132.950
% errors: 0.0
avg. time: 0.075
max time: 0.214
90th %: 0.130

component action + multi-threaded:
ops/sec: 131.775
% errors: 0.0
avg. time: 0.076
max time: 0.279
90th %: 0.125

jruby 1.2.6 + glassfish gem 0.9.2
----------------------------------
simple action + single-threaded:
ops/sec: 141.417
% errors: 0.0
avg. time: 0.070
max time: 0.259
90th %: 0.115

simple action + multi-threaded:
ops/sec: 247.333
% errors: 0.0
avg. time: 0.040
max time: 0.318
90th %: 0.065

component action + single-threaded:
ops/sec: 107.858
% errors: 0.0
avg. time: 0.092
max time: 0.595
90th %: 0.145

component action + multi-threaded:
ops/sec: 179.042
% errors: 0.0
avg. time: 0.055
max time: 0.357
90th %: 0.085


Platform/ActionSimple+/-Component+/-
Ruby ST 210ops0%132ops0%
Ruby MT226ops7.62%131ops-0.76%
JRuby ST141ops-32.86%107ops-18.94%
JRuby MT247ops17.62%179ops35.61%
(ST - single-threaded; MT - multi-threaded)

Conclusion

From my tests it appears that MRI is faster in single threaded mode, but JRuby makes up for the loss big time in the multi-threaded tests. It's also interesting to see that the multi-threaded mode gives MRI(green threads) a performance boost, but it's nowhere close to the boost that JRuby(native threads) can squeeze out from using multiple threads.

During the tests I noticed that rails was reporting more times spent in the db when using JRuby (2-80ms) compared to MRI (1-3ms). I don't know how reliable this data is but I wonder if this is the bottleneck that is holding JRuby back in the single threaded mode.

2 comments:

Anonymous said...

I benchmarked jruby-1.1.6, glassfish3 and rails-2.2 running on threadsafe mode.

The page I tested had several db lookups (perhaps 20), so it was rather rough on the implementation. I noticed that jruby+glassfish with several threads (i.e. stressing the CPU at 400%) was as fast as ruby+mongrel in single thread (i.e. stressing the CPU at 100%). So, jruby+glassfish was 4 times slower than ruby+mongrel. The DB used was mysql, and on jruby, I used the jdbcmysql adapter.

Igor Minar said...

Was the number of requests per second the same in both cases? The difference in cpu usage only shows that jruby can use all four cpu cores you have, while c-ruby can use only one.