Sunday, January 27, 2008

JRuby on Rails Rewrite of mediacast.sun.com Launched

A few days ago, we finally released Mediacast 2.0 - a complete rewrite of the old Mediacast application.



The original application was based on a few servlets, filters, and a lot of JSPs. It was all put together in hurry a few of years ago, when one of those fire-drill requests came to provide Sun employees with a site where they could publish files. The old code was hard to maintain and extend, and that's why we decided to EOL the old code base and write something better from scratch.

I'm a fan of Rails and for the past year I've been amazed by the great progress of the JRuby project. This was one of the reasons why I suggested that we could try to rewrite the app in Rails and deploy it in a regular Java web container, thanks to JRuby and Goldspike. It took some time for Rama to give us a go-ahead, but finally in late September, two other colleagues (with no Rails or JRuby experience) and I started to work on the rewrite alongside our other projects (forums.sun.com and wikis.sun.com).

Many people asked us why JRuby on Rails was picked for this project. Here is an incomplete list of reasons:
  • we were starting from scratch, so we were not tied to any legacy code, and could pick any web framework that runs on Java
  • proof of concept project to evaluate the technology for other uses across our organization
  • verify that Rails really delivers the rapid development promise
  • something new/fun for the team to help balance out the not-so-fun stuff :)

Here is my experience:

Rails and JRuby Learning Curve


I worked on my first Rails project in the summer of 2006 and since then I worked on a few other (internal) Rails projects. But having to teach others and deploy an application externally using JRuby was a new experience for me.

As far as teaching goes, I can't critique myself, but what I can say is that I had a lot of good help thanks to some Rails, Ruby and JRuby books, and many websites and blogs that have sprung up on the Internet in the last couple of years.

Learning JRuby (with previous Ruby and Java experience) is easy, because most of the time "it just works" and very rarely is the developer aware that the C-based MRI Ruby interpreter is not being used. The ability to access and seamlessly integrate with existing or new Java code is a huge plus without which we would not have been able to launch this project successfully (more on this later).

The deployment part was a different story. The Ruby on Rails application can be deployed in many different ways and JRuby offers a new alternative approach to all of them. Instead of e.g. having Apache webserver reverse-proxy requests to an army of Mongrels, JRuby and Goldspike make it possible to deploy Rails applications in a JavaEE web container. IMO this is much cleaner than anything else that is available in the non-JRuby Rails world. The downside is that this deployment method is quite new and there isn't a lot of documentation and community knowledge about it.

Application Architecture


Our app is a simple database-driven Rails application consisting of a handful of models, 4 controllers, and a bunch of view templates. The only piece of data that is not stored in the database are the actual files, which are stored on the file system.

To be on the safe side when it comes to performance, we employed quite a lot of fragment caching, which sped up our application quite a bit.

Development Environment


Our development environment is based on a self-contained1 JRuby (1.0.3) on Rails (1.2.6) application stored in a Mercurial repository. The IDE we use is NetBeans 6 with Ruby support and Mercurial plugin. The DB of our choice is MySQL and servers we use during development are WEBrick and Glassfish v2ur1.

NetBeans makes it super easy to write the Ruby code and offers a lot of neat features for Rails application development. I have to say that Tor and the gang did a great job. In fact I switched from Eclipse to NetBeans thanks to its great Ruby support.

Because the application runs in the JVM we can use JConsole to monitor the app while load testing which comes in really handy!

So all is good here, except for one thing. I got used to using the amazing ruby-debug debugger for debugging my Ruby on Rails applications. This debugger is not yet available for JRuby on Rails application (unless you hack your way through). I read somewhere that jruby-debug should be available with NetBeans 6.1. Once that is done, the JRoR dev environment will be on par with RoR (in fact thanks to tools like JConsole it will be superior).

Production Environment


We use a pair of load-balanced T2000 with Solaris 10, JDK6 and SJSAS 9.1u1. These two servers share a nfs NAS drive used for file storage and fragment cache storage. The DB backend is a MySQL database server, which we access through a connection pool set up in the app server.

All of this was fairly easy to set up. As you can see there were no special requirements when comparing this environment to a usual JavaEE production environment.

The Good


Once my colleagues grasped some RoR basics, we got the core of the application up and running fairly fast. Sometimes it still amazes even me how much one can do with Rails in a short amount of time. Thanks to a small amount of code one needs to write, the code review process is fast as well, and fixing a bug often means changing only a few lines of code.

The Bad


Since most of the C-based gems are not compatible out-of-box with JRuby (at the moment), the jruby-extras project aims to deliver JRuby compatible versions of these libraries. One of these libraries - JRuby-OpenSSL - was needed for me to integrate our app with our authentication webservice. I soon found out that JRuby-OpenSSL was not completely implemented yet and parts of functionality that I needed were missing. I'm sure that it is just a matter of time when problems like this will go away (if it hasn't happened already) as these libraries will mature.

The Ugly


The Goldspike project provides a bridge between the JavaEE and RoR world. It does this by dispatching the incoming HTTP requests into Rails running in a JRuby runtime.

This works great for all requests that take little time to process, but if you have long-running requests like large file uploads or downloads, these requests will occupy your JRuby runtimes and you soon realize that you are running out of runtimes in your small pool, at which point your application becomes unresponsive for any new requests.

This was the biggest problem we hit, but thanks to the possibility to seamlessly integrate Java code into our JRoR app and a great idea that my colleague Peter had, we solved this issue quite elegantly.

JRuby and Java Come to the Rescue


The two main issues we faced and that consumed most of my time on this project were related to the immaturity of the libraries around JRuby and application characteristics specific to JRoR deployment. Both of these issues can be resolved thanks to years of long Java and JavaEE history, and their libraries.

The problem with reliable connection to our SOAP-over-HTTPS based authentication webservice was resolved by generating a client Java webservice stub and using that instead of SOAP4R which didn't work properly because of the problems with JRuby-OpenSSL.

The second problem with the long-running processes occupying our precious JRuby runtimes, we solved by using a servlet filter and a fake HttpServletResponse (that we sent to Goldspike instead of the original one) and streaming the data from the filter instead from the runtime. I'll write a separate blog entry on this.

Overall Impression


This project was/is fun to work on. We experimented with quite a few new (for us) technologies and learned a lot along the way. To be honest, I expected to deploy the app much sooner but at that time I was not aware of the two above mentioned issues which consumed a lot of my time.

I was a bit worried about the performance of the application, but that turned out to be a non issue once we had the download servlet filter in place, and with the performance improvements in JRuby 1.1 things will be even better in the future.

Overall I'm happy with the outcome of our project and I look forward to adding more functionality to the application.

Our Future Plans


From the infrastructure perspective:
  • upgrade to JRuby 1.1 and Rails 2.0
  • start using Warbler, which looks to be superior to Goldspike's Rails plugin, for building the war file
  • experiment with in-memory session state replication in Glassfish

From the feature perspective:
  • better categorization of media items
  • search functionality via integration with search.sun.com
  • previews of media items
  • new UI design
  • and the toughest one - audio and video streaming


JRuby/Goldspike/Glassfish Things I Would Like to See Improved


  • I'm not sure if it is us and our (mis)configuration, Goldspike, JRuby or Glassfish, but right now we need quite a big JVM heap to keep things running. With 8 JRuby instances in the pool and Http thread count set to 512, we need -Xmx set to 2-2.5GB. This is a bit too much I think. We'll have to look into this and find the culprit.
  • The application startup is quite slow, especially on a machine like T2000, which doesn't perform well for heavy-weighted single-threaded operations. I don't see a reason, why Goldspike couldn't initialize the JRuby runtime pool concurrently cutting down the startup time significantly.
  • Offload JRuby runtimes as much as possible. Currently operations like file upload or file download (via Rails' send_file) are handled by JRuby runtimes. I think that it should be possible to take care of these operations outside of the runtime, allowing the runtime to process other rails requests in the meantime.



1 - Rails and all the other gems and jars are frozen into the project

20 comments:

Unknown said...

Cool stuff! :)

rama said...

my only mention is as the reluctant manager? pbbt! ;) most managers wouldn't even assume the risk, let alone the initial time investment to ramp up. ymmv.

Igor Minar said...

lol, I can attach your picture under the entry if that makes you happy :-).

As far as the risk goes, you are right. Nobody expected this to be a project without risk, but experimenting with new technologies, goes hand in hand with taking a risk. The future development of this project will show if it was worth it.

/i

liederrat said...

Nice project, congratulations! Wich setup do you use for load balancing? Session aware HTTP inspection with (which) external devices?

Anonymous said...

Whoa, excellent post! There were quite a lot of additional work on JRuby 1.1 to fight memory leaks and reduce the memory pressure, not to mention aggressive performance optimizations. So, hopefully, transition to JRuby 1.1 might bring some additional benefits. :)

Igor Minar said...

@liederrat the session store is shared between the nodes so we don't need session affinity.

@vladimir cool :)

Anonymous said...

So which MySQL storage engine(s) you're using?

AkitaOnRails said...

Awesome post and project. It feels great to see this kind of thing happening under Sun's umbrella which makes me respect even more this company. Keep going with more feedback on Jruby projects.

Anonymous said...

Igor, thanks for taking the risk. I am trying to find a nice project to use jRuby with at work. We are working with the iSeries, and because we are relegated to either Java or RPG (yeah...I had to google it) I would like to try jRuby.

Rich said...

Glad to see Sun deploying on JRoR! I this is the year for JRoR. We deployed a JRoR app at Oracle back in November. You can read (and compare notes) here.

Anonymous said...

What do you think of deploying a (J)RoR application into Glassfish, with mySQL, on a Dedibox (Linux Box, 2GHz, 1GB RAM; http://dedibox.fr)
Our project is a game with some real time features : actually all is done by polling to refresh the game arena. I think of combining the RoR app in Glassfish with Comet. But in case of success (many concurrent users) i think our Dedibox won't support it, because of its small capacities... perhaps should we think about Cloud Computing with Amazon for scaling, without paying too much (Sun Servers are too expensive for us ;-) )...

Igor Minar said...

@anonymous: InnoDB

@AkitaOnRails: There are many other cool things happening under "Sun's umbrella"

@Dave Thompson: The only reason why we were able to use Rails on our project is that with JRuby, the application looks like Java, breathes like Java and most importantly can be deployed and supported as (or almost as) a true Java app.

@Rich: I'm aware of your project. I think that you are doing great with your oracle mix app. Congratz!

@jurassicGeek: You don't need to worry much about the CPU speed, depending on your application specifics and if you really want this to be your production server, you should consider upgrading RAM. I'm not a sales person, but from what I hear Sun's HW is not considered expensive any more. In addition to that there are some great programs through which you can get some awesome deals. Check out Sun's Startup Essentials and Try and Buy programs.

Anonymous said...

Hi Igor,

What kind of performance are you getting on mediacast using glassfish and goldspike?

Anonymous said...

I'd love to hear more about the servlet filter w/ fake HttpServletResponse... is that there to read the full content from the HttpServletRequest before passing it onto Goldspike (minimizing blocking jruby instance time), or just on the sending side (so jruby can basically just write to memory and then be reclaimed while the filter pumps the in-memory buffer to the blocking socket)?

Or something else?

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

We started using JRuby on Rails too. Where the MRI garbage collector just crashes and burns, the JVM GC really shines. We had one government application which was rather leaky and JRuby just cracked it. Pack it up with warbler and GlassFish and you've got one amazing platform.

Anonymous said...

Thanks for the great post.
About the servlet filter w/ fake HttpServletResponse. How do you implement it? Is it written entirely in java, or can you call ruby code from there. It looks like an alternative to BackgroundRB living inside the same server, which is very good.

Igor Minar said...

Jose,

The implementation of the filter is pure java code, but there is a callback api that can call the rails call when a download finishes.

After some time I started to like the servlet filter solution less an less, because of several constraints that it put on my app.

I reimplemented the entire piece that sends files from scratch and made it mediacast.sun.com agnostic. Besides code cleanup, another advantage is that the preliminary benchmarks tell me that the performance and scalability will be quite something.

The project will be available soon (2-3 weeks) for download and will be open source.

Stay tuned...

nabeelz6 said...

Good stuff! Is the project available for download yet?

Igor Minar said...

It took a while, but I finally managed to publicly announce grizzly-sendfile - the solution for the file streaming problem.