How to Install a Glassfish Patch

Recently I've been working quite extensively with glassfish and had a need to apply a patch for some of the core code.

To my big surprise I was not able to find the official way to apply patches to a glassfish installation. I found several patches posted on the issue tracker and paying customers have access to tested and supported patch bundles, which are released outside of the regular release cycle. But even some extensive googling didn't easily reveal how to apply them.

Then luckily I found this discussion on the glassfish mailing list, which describes the process.

To make life easier for others and google to index this information, here is a brief recap:

1. Create a directory <glassfishroot>/lib/patches
2. Copy the jar with your patch into this directory
3. Edit your <glassfishroot>/domain/<domainname>/config/domain.xml and add attribute classpath-prefix="${com.sun.aas.installRoot}/lib/patches/<patchname.jar>" to the <java-config ...> node
4. Restart your domain

I tested this with Glassfish v2, I'll have to check if it works with the upcoming v3 as well.

My Confluence 3.0 Wishlist

Confluence 2.9 was released last month and I've seen references to 2.10 in the Confluence issue tracker, so I expect to see it out in 1-2 months. That makes me think about what's next.

As a part of my adventures of working on Sun's external wiki wikis.sun.com, I've been working on Confluence plugins and even the Confluence core code for a year and a half now, adding new features, enhancing the existing features and very often fixing bugs. Sometimes it was trivial to enhance the code or fix a bug, other times it was not, but what I want to write about today are things that were not possible at all without irreversibly forking the code.

Confluence 3.0 should be a version that really deserves to have the first digit incremented. Not because marketing said it's time for that, but because the changes in the application are so significant.

I'm sure that Atlassian has lots of ideas about what Confluence 3.0 should look like, but Atlassian guys, in case you start to run out of ideas, here is my wish list:

Fix the Database Schema

Confluence has been in development for years and the database schema definitely shows that. Since the database is the heart of the application, I think it deserves a lot of attention and major performance boost could be gained by doing a clean up.

Specific improvements:

• Establish and in the future enforce naming conventions
• Replace all the natural foreign keys with surrogate keys, e.g. user name, spacekey, group name should be replaced with ids in all the referencing tables (this would finally allow CONF-4063 to be implemented)
• Add caches for the lower function (patch) and maybe counter caches

Rework the Clustering

Clustering is usually supposed to fulfill two functions: scalability and robustness. In the case of Confluence mainly the second attribute is missing. In fact, I'd go as far as saying that a Confluence cluster is less robust than a single instance of Confluence. Why? Because the way it is implemented makes the entire cluster vulnerable when one node has problems.

I personally experienced several cluster lock-ups or crashes, usually initiated by a separate Confluence bug, in which the effect was multiplied by the clustering code. One of such of these bugs: CONF-12319

Mike's presentation covers quite a few design goals behind the implementation in Confluence. Clustering can really get ugly and complicated and Mike covered it pretty well. Unfortunately the distributed share part of the clustering makes Confluence prone to problems.

One of the clustering goals that Mike emphasizes in his presentation is that clustering should be "admin-friendly" (low admin overhead and easy setup). While I agree with the low overhead part, the easiness of setup should not compromise the goals which clustering is trying to fulfill in the first place. Clustering is for people who are serious about running Confluence, and as such should be expected to be qualified for the job.

Specific improvements:

• Either reevaluate the distributed share clustering so that it is super robust, or consider implementing clustering via a centralized share
• Avoid shutting down the entire cluster when "cluster panic" is detected. A better solution, which avoids unnecessary downtime, would be to shut down all the nodes, except for the nodes properly clustered with the oldest node.

Clean Up the HTML and CSS Code

The html code that comes out of Confluence is horrendous. While the rendered output looks pretty pleasant, looking under the hood (browsing the source code in a browser) is not recommended for pregnant women, men with ED, high cholesterol, and generally not recommended for people over 50.

Some improvements were done in this area in the recent releases, but all of them were just minor cosmetic surgeries. Confluence really needs major surgery that will bring the html code up to current standards. The benefit of this will be much faster page loads and code that is easier to maintain and enhance.

Specific improvements:

• Rewrite most of the templates and macros to make them XHTML 1.0 compliant
• Minify and combine javascript and css files (CONF-8622)
• Use image sprites to even further speed up page loads (especially in the rich text editor)

Redo the URI Namespace

Human friendly URIs and URLs are becoming more and more important on today's Internet. Confluence is not doing well in this area.

Specific improvements:

• /display/MySpace/My+Page - is the /display part really necessary? Can't we do /MySpace/My+Page
• /pages/diffpages.action?pageId=2490471&originalId=45714293 - What is this? I don't know. How about: /MyWiki/My+Page/diff/23:22. I think that actually means something. There might be a better format, this is just a thought.
• I think in general redoing the URI name space using REST conventions would be interesting.

Improve Atlassian-Renderer

When I was creating come patches for the atlassian-renderer I was surprised to find that atlassian-render, the module responsible for rendering wiki markup into html is full of hardcoded html snippets. The main reason why this surprised me is that most of the Confluence code is pluggable, which allows for parts of the code to be replaced with a better version without a lot of problem. This is not the case with the render. And this presents two problems: it's not possible get Confluence to directly render anything else than html (pdf and doc are only derived from the html), and it's not possible to use anything else than Confluence markup as the input for the renderer.

The first problem makes me unable to render custom output like docbook or to improve the PDF output, which is pretty poor.

The second issue means that all the customers that use Confluence are locked-in because all the content created via Confluence is Confluence-specific and can't be easily moved to a different wiki engine when needed.

In my opinion the sooner all major wiki engine developers settle on one wiki markup standard the sooner we will all be better off. This might be especially difficult for Atlassian to swallow and implement, because they standardized on their own markup that they also use in their other products.

An interesting initiative that is gaining a lot of traction is Creole, a standardized wiki markup. Confluence is one of the few major wiki players that doesn't support this initiative.

Specific improvement:

• Split the current renderer into two pluggable parts: parser and renderer
• Implement Creole support (CONF-12077)

Improve Developer Documentation

spent countless hours, especially in the beginnings trying to figure out how Confluence works and how Confluence plugins should be written. I learned some new tricks and that's the good part, the bad thing is that the experience could have been much better if the the code contained more javadocs comments and if the plugin interfaces and mainly the configuration file format was better documented.

Specific improvements:

• Add JavaDoc comments where missing
• Finally provide complete specification and documentation for the plugin config file (JRA-12183)

Thanks! :-)

That's about as much as I can think of for now. There are probably other things that I missed and then there enhancements around security, which I know are already on the roadmap.

I understand that most of the changes above will create incompatibilities with many existing themes and plugins, but hey, Confluence 3 will happen only once EVAR and releases like this are expected to bring major incompatibilities. Data can always be migrated automatically and existing plugins and themes will be migrated when there are people interested in using them and proper migration instructions are provided.

I hope that Confluence 3 will not be a "marketing" release, but instead something really cool that all users and developers will enjoy working with.

Unintrusive but secure passwordless ssh authentication

On a daily basis I need to log in to many remote servers inside or outside of Sun via SSH, often dozens of times per day. This can get pretty tiresome if you need to type in your password with every log in.

Some suggest setting up so-called "passwordless" authentication by generating ssh keys and specifying empty passphrase for the private key. This will result in passwordless authentication, but will also decreased security. Should anyone get hold of your private key, (s)he'll get access to all of your remote systems.

ssh-agent can help a lot in keeping the security level high and minimizing the number of times you need to type in the password. However, if you use a terminal with tabs or use both local and remote terminals on your workstation, you'll end up running many ssh-agent processes and having to authenticate every time you start such a process, which diminishes most of the conveniences of using ssh-agent.

Frustrated with this situation and with a bit of help from Martin, I created a shell script, which I added to my .bash_profile startup script. All I have to do now is to authenticate when my first terminal session starts and I'm good until the next time I restart my OS. sweeeet...

Here is how you could set it up on a workstation and a remote-server:

First, if you haven't generated your private/public ssh key pair, do that now:

workstation $ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/user/.ssh/id_rsa): **********************
Enter same passphrase again: **********************
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
01:a6:95:23:1c:74:53:c7:f4:87:07:a2:50:ef:99:16 user@Computer.local

Now make sure that the file system permissions are set up correctly:

workstation $ cd ~/.ssh/
workstation $ ls -l
total 56
-rw-------  1 user  staff  1743 Aug 31 00:13 id_rsa
-rw-r--r--  1 user  staff   398 Aug 31 00:13 id_rsa.pub

The id_rsa file must be readable only by owner, if this is not true, the key will be ignored.

On the remote server you need to authorize your newly generated key pair by appending it's public key to the ~/.ssh/authorized_keys file under your remote home directory:

workstation $ cat ~/.ssh/id_rsa.pub |ssh user@remote-server 'sh -c "cat - >>~/.ssh/authorized_keys"'

Now you can try to log in:

workstation $ ssh user@remote-server
Enter passphrase for /Users/user/.ssh/id_rsa: **********************
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
Last login: Thu Sep 11 20:19:19 2008
remote-server $

If you open a new tab in your terminal and try to log in again, you'll be asked to enter the passphrase yet again. This is where my script becomes useful. First download the script from Mediacast: ssh-agent-init.sh and store it somewhere in your home directory

workstation $ mkdir ~/bin
workstation $ cd ~/bin
workstation $ wget http://mediacast.sun.com/users/IgorMinar/media/ssh-agent-init.sh
workstation $ chmod o+x ~/bin/ssh-agent-init.sh

The next (last) step is optional if you want to start the script manually you can skip it.

I wanted to have this script automatically invoked when I start my terminal for the first time in the interactive mode. All I needed to modify were my .bash_profile (used for interactive sessions) and .bashrc (used for non-interactive sessions) startup scripts for my bash shell (modifications are in italics):

workstation $ cat ~/.bash_profile
...
...

. ~/bin/ssh-agent-init.sh

workstation $ cat ~/.bashrc
export NONINTERACTIVE=1
. ~/.bash_profile

(Note: I source the ~/.bash_profile script from the ~/.bashrc script to enable code reuse between the two scripts.)

That's it! If you now try to open a terminal tab for the first time, you'll be asked for passphrase. Once that is done, any new tab or any other shell session created under the same account will reuse the same ssh-agent process.

I've been using this script on my MacOS X laptop as well as OpenSolaris workstation for a few weeks now and it's been working like charm.

Enough with the Flash Already

It often bothers me when Flash is being misused for small visual effects on home pages of websites. Not just that it slows things down, but it also usually breaks these websites terribly when I want to copy&paste stuff or when one doesn't have the Flash plugin installed.

Margarete, has recently bugged me quite a bit about making the front page of her website a bit more "visual". She started with an idea of adding some flash to her website, which I dismissed quickly.

Fortunately I was able to come up with a neat little CSS code, which is way better than Flash for what she was trying to achieve. Don't be mistaken, I didn't even need JavaScript to do all the subtle movements and eager image loading, it's all pure CSS.

The front page:


A small "visual" effect when one mouses over the "Blog" element:


I still plan to enhance the code a bit more to make it faster to load by decreasing the number of HTTP requests needed to load the asset files as well as size of the images, but I'm quite happy with what it looks and feels like already. Check out the live version at www.fgslovakia.com.

A good technical article describing a technique very similar to the one that I used can be found the A List Apart website.

BTrace == DTrace for Java

Last week, I was trying to nail down a bug in SunWikis that was triggered by some kind of race condition. These kinds of issues are pretty nasty, especially if the application in question is a pretty complex beast, something that Confluence definitely qualifies as.

The debugger was no help because the timing had to be very precise for the issue to occur, and logging was totally useless because of zillions lines of logs that would be difficult to safely filter out. In many cases the information I needed was not present in logs anyway and since my guess was that the bug was coming from 3rd party code, logged data couldn't be easily expanded. DTrace, which I blogged about in my previous post, could have revealed some information, but I think that it would be very difficult to write a good D script that could give me some clues.

While waiting for my dev server to restart, I came across an interesting blog post that caught my eye. It mentioned this thing called BTrace, that I hadn't heard about before. It promised it to be a DTrace aimed at Java apps. With a fair amount of skepticism, I navigated to the BTrace website and started reading. Four things about BTrace earned my interest:

• It's based on same or very similar principals as DTrace, but specialized for Java apps
• No restarts necessary, you can observe a running app by connecting to a JVM identified by a PID
• The BTrace programs are written in a language that is a subset of Java, and heavily based on annotations. This was a bit of a turnoff for me at first, until I found out that the programs don't need to be compiled manually but btrace takes care of that for me. Bravo!
• I can call DTrace from my BTrace programs if I need to instrument native code or the OS

Armed with all this info and my eyes sparking, I wrote my first BTrace program and it worked as advertised! No restarts, no compilation, no hassle.

A few hours later, I had a program that was observing exactly those parts of Confluence that I needed. All of this while the application was running uninterrupted. I was able to snoop on how my code was interacting with Confluence and Confluence was in turn interacting with Hibernate in a way that I would have never dreamed of. All of this while concurrent events aimed at triggering the bug were happening.

Running BTrace programs is as easy as this:

$ ./bin/btrace \
  -classpath hibernate-2.1.8-atlassian.jar:spring-hibernate2-2.0.6.jar \
  6266 HibernateSessions.java

Where 6266 is the PID of my app/web server and HibernateSessions.java is my BTrace program.

Parts of the program looks like this (check out the javadoc comments for explanation of the code):

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import net.sf.hibernate.HibernateException;
import net.sf.hibernate.collection.PersistentCollection;
import net.sf.hibernate.engine.SessionImplementor;
import org.springframework.orm.hibernate.support.OpenSessionInViewFilter;

@BTrace
public class HibernateSessions {
  /**
   * A thread local variable used to filter out all the events that we
   * are not interested it.
   */
  @TLS private static boolean viaSso = false;
  private static boolean printStack = false;

  /** print a message when a thread enters doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter"
  )
  public static void enableProbing() {
      println(str(currentThread(), " - Entered SSO Filter"));
      viaSso = true;
  }

  /** print a message when exiting the doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter",
      location=@Location(Kind.RETURN)
  ) 
  public static void disableProbing() {
      println(str(currentThread(),  " - Exited SSO Filter"));
      viaSso = false;
  }

  /**
   * print an message with a HibernateException is thrown, with detailed 
   * info about the current context
   */
  @OnMethod(
      clazz="net.sf.hibernate.HibernateException",
      method=""
  )
  public static void onthrowreturn(HibernateException self, String s)  {
      println(str(currentThread(), 
              strcat(" - HibernateException was thrown: ",
              strcat(s, strcat(" | collection: ", str(collection))))));
  }

  /**
   * Print a message when Hibernate is attaching a collection to a session.
   * This was crucial info for me to get in order to resolve the issue
   */
  @OnMethod(
      clazz="net.sf.hibernate.collection.PersistentCollection",
      method="setCurrentSession",
      location = @Location(Kind.RETURN)
  )
  public static void setSessionForCollection(boolean returnValue)  {
      if (returnValue == true) {
        println(str(currentThread(),
          strcat(" - a collection was attached to a session. Collection:",
          str(collection))));
  }
}

The result of this was a workaround for my code as well as a pretty detailed bug report that Atlassian confirmed to be a very similar to a confirmed bug that was reported recently. I could hardly achieve this without BTrace or without an in-depth knowledge of how all the Confluence code works.

As great as BTrace is, it is still a beta. If you decide to use it in production environment you should keep that in mind. During the several hours I spent working with BTrace, I experienced one JVM crash/lockup caused by BTrace. I bet that this kind of issues will be ironed out soon, and often this kind of risk is worth undertaking in order to resolve otherwise untraceable bugs.

Since last week, BTrace is part of my toolkit and I already have other mysteries on my ToDo list that I want to shine some light on with BTrace. I applaud the BTrace team for giving us this amazing tool!

DTrace and Java - Observations and Docs

For a long time I've been hearing about DTrace and how cool it was. I read a lot about it and saw some presentation on observing Java apps with DTrace, but only recently have I found enough free time to extensively play with it. And oh boy, it really is cool. But even with all this coolness, dtracing Java apps feels a bit awkward.

I think there are two reasons for that. The D language, which looks like C, but has a few fundamental differences, is pretty far from what Java developers are used to. And the second reason is the fact that DTrace is in reality instrumenting JVM and not the application running in the JVM. This means that dtracing Java is limited by abilities of the Java DTrace provider and also that the transition between the different layers of the stack can add some awkwardness.

Having said that, DTrace is really awesome and can be enormously helpful in many cases when developing or troubleshooting (Java) apps.

One thing I really miss in DTrace is regex support in any part of D scripts. There is a limited globbing support and some workarounds, but they ain't pretty.

Here are some DTrace resources that I found useful:

• DTrace Wiki Docs - the complete documentation (I love to see cool docs like this at our SunWikis)
• a DTrace HowTo
• DTrace for Java at Solaris Internals
• DTrace Toolkit - a compilation of DTrace scripts written by DTrace pros

Catching StackOverflowError and a Bug in Regex Implementation in Java

Shortly after we launched SunWikis almost a year ago, we started having an issue with StackOverflowError, which was occurring when the content of certain wiki pages was being parsed and URLs where being extracted. I documented the issue in CONF-9392.

The problem is not really a bug in Confluence (even though using an overly-complex regular expressions doesn't help the case), but it's a problem in JDK. After a brief search at bugs.sun.com I found the root cause of our StackOverflowError documented as bug 6337993.

Originally I tried to mitigate the situation by increasing the memory reserved for the stack data via the -Xss JVM parameter. This helped a little bit, but wasn't good enough in most cases.

Last week I decided to go against everything I've been taught, and wrote a patch for Confluence that wraps the part of code that results in the StackOverflowError into a try/catch block. I know that any throwables that extend from Error should not be caught by a client code because they usually indicate a failure that only JVM should try to recover from, but IMO in the case of StackOverflowError, the situation is a bit different. That is mainly because before throwing the StackOverflowError, JVM pops the stack, so by the time the code execution gets to the catch block, JVM has already recovered from the error.

I don't claim this to be a solution to the problem, it's just a workaround that works better than increasing the stack size in this particular case. The fact that Confluence doesn't find all the URLs in wiki pages (used mainly to list outgoing links in the page info view) is just a small sacrifice, compared to inability to save or copy the page.

As for the solution, it seems that reimplementing Java's regular expressions library would be the most suitable one. I tried to run a code that fails in Java in JRuby, which uses a port of Oniguruma regex engine for Java and things worked flawlessly and as I read it also gives JRuby a performance boost over java.util.regex.

Mediacast Deployment Diagram

During our last meeting, Arun suggested that I publish the high-level deployment diagram of mediacast.sun.com. So here it is:



I described everything in textual form in my previous blog entry about mediacast.

JRuby on Rails Rewrite of mediacast.sun.com Launched

A few days ago, we finally released Mediacast 2.0 - a complete rewrite of the old Mediacast application.



The original application was based on a few servlets, filters, and a lot of JSPs. It was all put together in hurry a few of years ago, when one of those fire-drill requests came to provide Sun employees with a site where they could publish files. The old code was hard to maintain and extend, and that's why we decided to EOL the old code base and write something better from scratch.

I'm a fan of Rails and for the past year I've been amazed by the great progress of the JRuby project. This was one of the reasons why I suggested that we could try to rewrite the app in Rails and deploy it in a regular Java web container, thanks to JRuby and Goldspike. It took some time for Rama to give us a go-ahead, but finally in late September, two other colleagues (with no Rails or JRuby experience) and I started to work on the rewrite alongside our other projects (forums.sun.com and wikis.sun.com).

Many people asked us why JRuby on Rails was picked for this project. Here is an incomplete list of reasons:

• we were starting from scratch, so we were not tied to any legacy code, and could pick any web framework that runs on Java
• proof of concept project to evaluate the technology for other uses across our organization
• verify that Rails really delivers the rapid development promise
• something new/fun for the team to help balance out the not-so-fun stuff :)

Here is my experience:

Rails and JRuby Learning Curve

I worked on my first Rails project in the summer of 2006 and since then I worked on a few other (internal) Rails projects. But having to teach others and deploy an application externally using JRuby was a new experience for me.

As far as teaching goes, I can't critique myself, but what I can say is that I had a lot of good help thanks to some Rails, Ruby and JRuby books, and many websites and blogs that have sprung up on the Internet in the last couple of years.

Learning JRuby (with previous Ruby and Java experience) is easy, because most of the time "it just works" and very rarely is the developer aware that the C-based MRI Ruby interpreter is not being used. The ability to access and seamlessly integrate with existing or new Java code is a huge plus without which we would not have been able to launch this project successfully (more on this later).

The deployment part was a different story. The Ruby on Rails application can be deployed in many different ways and JRuby offers a new alternative approach to all of them. Instead of e.g. having Apache webserver reverse-proxy requests to an army of Mongrels, JRuby and Goldspike make it possible to deploy Rails applications in a JavaEE web container. IMO this is much cleaner than anything else that is available in the non-JRuby Rails world. The downside is that this deployment method is quite new and there isn't a lot of documentation and community knowledge about it.

Application Architecture

Our app is a simple database-driven Rails application consisting of a handful of models, 4 controllers, and a bunch of view templates. The only piece of data that is not stored in the database are the actual files, which are stored on the file system.

To be on the safe side when it comes to performance, we employed quite a lot of fragment caching, which sped up our application quite a bit.

Development Environment

Our development environment is based on a self-contained¹ JRuby (1.0.3) on Rails (1.2.6) application stored in a Mercurial repository. The IDE we use is NetBeans 6 with Ruby support and Mercurial plugin. The DB of our choice is MySQL and servers we use during development are WEBrick and Glassfish v2ur1.

NetBeans makes it super easy to write the Ruby code and offers a lot of neat features for Rails application development. I have to say that Tor and the gang did a great job. In fact I switched from Eclipse to NetBeans thanks to its great Ruby support.

Because the application runs in the JVM we can use JConsole to monitor the app while load testing which comes in really handy!

So all is good here, except for one thing. I got used to using the amazing ruby-debug debugger for debugging my Ruby on Rails applications. This debugger is not yet available for JRuby on Rails application (unless you hack your way through). I read somewhere that jruby-debug should be available with NetBeans 6.1. Once that is done, the JRoR dev environment will be on par with RoR (in fact thanks to tools like JConsole it will be superior).

Production Environment

We use a pair of load-balanced T2000 with Solaris 10, JDK6 and SJSAS 9.1u1. These two servers share a nfs NAS drive used for file storage and fragment cache storage. The DB backend is a MySQL database server, which we access through a connection pool set up in the app server.

All of this was fairly easy to set up. As you can see there were no special requirements when comparing this environment to a usual JavaEE production environment.

The Good

Once my colleagues grasped some RoR basics, we got the core of the application up and running fairly fast. Sometimes it still amazes even me how much one can do with Rails in a short amount of time. Thanks to a small amount of code one needs to write, the code review process is fast as well, and fixing a bug often means changing only a few lines of code.

The Bad

Since most of the C-based gems are not compatible out-of-box with JRuby (at the moment), the jruby-extras project aims to deliver JRuby compatible versions of these libraries. One of these libraries - JRuby-OpenSSL - was needed for me to integrate our app with our authentication webservice. I soon found out that JRuby-OpenSSL was not completely implemented yet and parts of functionality that I needed were missing. I'm sure that it is just a matter of time when problems like this will go away (if it hasn't happened already) as these libraries will mature.

The Ugly

The Goldspike project provides a bridge between the JavaEE and RoR world. It does this by dispatching the incoming HTTP requests into Rails running in a JRuby runtime.

This works great for all requests that take little time to process, but if you have long-running requests like large file uploads or downloads, these requests will occupy your JRuby runtimes and you soon realize that you are running out of runtimes in your small pool, at which point your application becomes unresponsive for any new requests.

This was the biggest problem we hit, but thanks to the possibility to seamlessly integrate Java code into our JRoR app and a great idea that my colleague Peter had, we solved this issue quite elegantly.

JRuby and Java Come to the Rescue

The two main issues we faced and that consumed most of my time on this project were related to the immaturity of the libraries around JRuby and application characteristics specific to JRoR deployment. Both of these issues can be resolved thanks to years of long Java and JavaEE history, and their libraries.

The problem with reliable connection to our SOAP-over-HTTPS based authentication webservice was resolved by generating a client Java webservice stub and using that instead of SOAP4R which didn't work properly because of the problems with JRuby-OpenSSL.

The second problem with the long-running processes occupying our precious JRuby runtimes, we solved by using a servlet filter and a fake HttpServletResponse (that we sent to Goldspike instead of the original one) and streaming the data from the filter instead from the runtime. I'll write a separate blog entry on this.

Overall Impression

This project was/is fun to work on. We experimented with quite a few new (for us) technologies and learned a lot along the way. To be honest, I expected to deploy the app much sooner but at that time I was not aware of the two above mentioned issues which consumed a lot of my time.

I was a bit worried about the performance of the application, but that turned out to be a non issue once we had the download servlet filter in place, and with the performance improvements in JRuby 1.1 things will be even better in the future.

Overall I'm happy with the outcome of our project and I look forward to adding more functionality to the application.

Our Future Plans

From the infrastructure perspective:

• upgrade to JRuby 1.1 and Rails 2.0
• start using Warbler, which looks to be superior to Goldspike's Rails plugin, for building the war file
• experiment with in-memory session state replication in Glassfish

From the feature perspective:

• better categorization of media items
• search functionality via integration with search.sun.com
• previews of media items
• new UI design
• and the toughest one - audio and video streaming

JRuby/Goldspike/Glassfish Things I Would Like to See Improved

• I'm not sure if it is us and our (mis)configuration, Goldspike, JRuby or Glassfish, but right now we need quite a big JVM heap to keep things running. With 8 JRuby instances in the pool and Http thread count set to 512, we need -Xmx set to 2-2.5GB. This is a bit too much I think. We'll have to look into this and find the culprit.
• The application startup is quite slow, especially on a machine like T2000, which doesn't perform well for heavy-weighted single-threaded operations. I don't see a reason, why Goldspike couldn't initialize the JRuby runtime pool concurrently cutting down the startup time significantly.
• Offload JRuby runtimes as much as possible. Currently operations like file upload or file download (via Rails' send_file) are handled by JRuby runtimes. I think that it should be possible to take care of these operations outside of the runtime, allowing the runtime to process other rails requests in the meantime.

¹ - Rails and all the other gems and jars are frozen into the project

On the Move (Away from Blogger)

Today was a big day for my blog. I started a "big" move that was hanging over my mind for a few months.

I got my own domain igorminar.com and changed the blogger.com settings so that the domain name blog.igorminar.com is being used instead of net3x.blogspot.com.

Except for a small hick-up with DNS cache and a weird 404 Server not Found error (described in this blog post) everything went smoothly. My plan is to keep on using Blogger's custom domain feature for a few weeks and then migrate all the content over to my own blogging engine. I'll most likely end up using Mephisto, JRuby, Glassfish and MySQL, but things can still change.

The reason why I'm moving in two phases is to keep all the referring links that point to the old urls as well as most of the search engine ranking. The Blogger's custom domain feature now redirects everything to the new urls via 301 Permanent Redirect, so hopefully the transition will be close to painless. I'm not SEO expert, but this for sure is an interesting experiment.

And for those SEO "experts" that were claiming that getting Blogger to do proper redirects after a blog move was impossible here is a little proof:

$ wget --spider net3x.blogspot.com
--16:34:30--  http://net3x.blogspot.com/
      => `index.html.2'
Resolving net3x.blogspot.com... 72.14.207.191
Connecting to net3x.blogspot.com|72.14.207.191|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://blog.igorminar.com/ [following]
--16:34:30--  http://blog.igorminar.com/
      => `index.html.2'
Resolving blog.igorminar.com... 66.249.81.121
Connecting to blog.igorminar.com|66.249.81.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
200 OK

My Custom Prompt for Leopard's Terminal

As I mentioned in my previous post, even though it is not perfect, I settled on using Terminal as my terminal application.

The default prompt that comes with MacOS is not very useful, so I configured my own by adding this line to my .bashrc file:

export PS1="\[\033]0;\u@\h:\w\007\]\u@\h:\W\\$ "

This results in a prompt that looks like this:

This prompt displays the prompt as username@hostname:currentdir$ and in addition to that it sets the window title to username@hostname:pwd.

Leopard's Terminal Tabs

As I shared yesterday, I love tabs. One of the few good things that MacOS 10.5 Leopard brought, was enhanced Terminal application. A basic feature - Tab support - that was missing in Terminal for way too long, was finally added, and Terminal app became finally usable for anyone who spends a great time of the day working on the command line.

Before Leopard, I used iTerm, but there was just something about it that I didn't like, but the availability of tabs made me get over it. Now with Leopard there is no need for iTerm any more.

Just to point out that Apple seems to be unable to get anything 100% right these days, there are two flaws in the Terminal's tab implementation.

The first flaw is that you can't easily switch between tabs using keyboard shortcut. All you can do is to use cmd+{ and cmd+} to switch to previous or next tab respectively. What a "wonderful" choice of keys that require use of right hand or both hands. Most importantly, one can't use the handy cmd+1, cmd+2, ..., cmd+9 shortcuts (or anything similar) to switch to one of the first nine tabs. Maybe Apple doesn't want to overload us with new features and is planning to add this to Terminal in MacOS 10.6. :-) Fortunately, the cmd+{ and cmd+} shortcuts can be remapped via System Preferences.

The second flaw is that Terminal doesn't let you name the tabs or configure how the tabs should be named. This is a useful feature that many other terminal applications have, but Apple decided that using the name of the program of the currently running process is good enough. Well, this is what it looks like:

People are doing all kinds of crazy things to work around this issue. Including using hardlinks to change names of programs or creating dummy processes that carry the tab name.

I would be very happy if Apple did something similar to what xterm does - using an environmental variable to set name of the tab dynamically. But I have a feeling that we'll see MacOSX Trash go 3D sooner than properly working Terminal tabs. :(

Mac Applications That I Use

I've been meaning to write this entry on applications installed on my computer for a few months now, but never found time to do it. Hopefully none of those new Mac users that I told to check out my blog "in a few days", several months ago, got angry at me. If nothing else, my promise helped me to get more regular readers :).

Anyway, here is the list of apps installed on my Mac (all these apps work with Leopard, unless I noted otherwise):

Essentials

A Mac without these apps would never be the same:

• Quicksilver - an app that helps you find things on your computer and execute actions on these things. I use it to launch apps, pause iTunes, find contacts in the AddressBook, and many many many other things. Most of the cool functionality is added via plugins, so don't forget to checkout the plugin section of preferences. This app is a must have. Here are Some docs and a good video.
• Adium - a really handy IM client that can handle all the popular IM protocols. Definitely a must have. (it's much better than iChat when it comes to IM-ing).
• Growl - Growl is a notification system for Mac OS X: it allows applications that support Growl to send you notifications. Notifications include Adium or Skype messages, network up/down events, volume mount/unmount events, currently played iTunes song and many others. More info - One extension that I found to be incompatible with Leopard is GrowlMail for Mail.app notifications, when I received many new messages at once this extension would cause the Mail.app to crash.
• MacPorts - This is a geeky one: The MacPorts Project is an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system.
• Firefox - no description needed :)

Nice to Have

• iStat menus - a nifty app that displays stats about cpu load, cpu temperature, memory and network utilization, and a lot of other interesting data about your Mac. I use it as a replacement of MenuMeters which I couldn't get to run reliably on Leopard.
• Perian - Perian is a free, open source QuickTime component that adds native support for many popular video formats.
• Camino - yeah, as if the fact that I'm obsessed with tabs was not enough, I usually run 2 or 3 browsers at the same time to be able to take advantage of their unique features, or more often, while developing webapps, I need to be able to have more than one session open.
• Chicken of the VNC - A handy VNC client
• MacFUSE + sshfs - MacOS X implementation of FUSE, which makes it possible to painlessly mount all kinds of different filesystems e.g. sshfs.
• MacFusion - GUI frontend for MacFUSE
• VLC - alternative to QuickTime video player with build-in support for all kinds of video and audio codes, subtitle support and many other features
• Vine Server (OSXvnc) - a good VNC server in case you need a bit more control than the Leopard's built-in server offers
• Skype - I have a love-hate relationship with Skype. As soon as I find an alternative crossplatform communication tool with audio and video support and cheap calls to Slovakia, I'll switch. Maybe Gizmo will become that tool one day
• NeoOffice - don't forget to install the latest Patch. I have big hopes for OpenOffice Aqua to be released soon. My initial experiments revealed some very good performance results.
• iShowU - a cool app for creating screencasts. This is the only paid-for app on my list. But it's well worth it.
• Caffeine - Caffeine is a tiny program that prevents your Mac from automatically going to sleep, dimming the screen or starting screen savers. Very useful during video calls, while reading PDFs or watching movies.
• Twitterrific - a Twitter client
• Azureus - Java BitTorrent client
• SkeyCalc - An OTP (S/Key) calculator
• Cisco VPN Client

Development

• NetBeans - there is a lot of controversy in the community when it comes to the Eclipse vs NetBeans question. I used Eclipse for a really long time, but ever since I tried NB6 Milestone 8 or so, I started liking NB more than Eclipse. I'll give Eclipse another shot in a few weeks, but for now NetBeans is my default IDE. And as far as Ruby/JRuby development is concerned, I don't think that this is going to change for a long time.
• JRuby - a pure Java-based Ruby interpreter
• GlassFish - JavaEE 5.0 compliant application server

Other Stuff

• Drivers for my Vydeo eSATA 34: Dual Port 34mm Express Card - More info in my older blog entry

Other Lists

Rama and Martin composed similar lists some time ago, so you might want to check those out as well.

I'm Obsessed with Tabs

This is a screenshot I took during a regular development/research day:



What is your average number of browser tabs opened at once?

MultiSudokuSolver - A fun project I worked on during the Xmas break

While I was visiting my family in Slovakia during the winter 2005/2006 I found a sudoku magazine in my dad's apartment. I like this kind of stuff, so I mediately started solving one puzzle after another, until I got to a "Speciality MultiSudoku" puzzle:



This puzzle is composed of five regular 9x9 puzzles that are interconnected and you can't solve either of them alone, meaning that you need to be solving all five of them at the same time and use intermediate results from one puzzle to get an intermediate result for another puzzle that this puzzle is connected with. It sounded like a good challenge so I started working on it. After two or three nights I realized that I made a mistake!! Ughhh. I erased everything I had solved and started from scratch. After another few nights I found another problem, got turned off by this puzzle and put it away.

I came across the puzzle once again when my wife and I returned to Slovakia this winter. I found the puzzle in the apartment and, being a person who likes challenges and doesn't get turned off my failures for too long, I decided to erase everything and this time be very careful and solve the puzzle once and for all.

After about two evenings I found something I didn't want to see. An error!!! It was hard to believe it, but there it was. I tried to fix it, but if you don't spot an error in a sudoku puzzle early enough, you'll spend more time fixing it than if you started from scratch.

Do you think I felt like starting from scratch? No! But I couldn't let this be. So since I had 10 or so days off during the Christmas break (thanks Sun!!!), I decided to use my brain in a more productive way and to write a small program that would solve the puzzle for me.

Given my interest in Ruby and JRuby and the type of problem I was about to solve, the language choice was an easy one to make.

A few days later I had a script that solved the puzzle for me within 1.07 seconds. Ya! You heard right! Solved not in a few evenings but in just a little more than one second.

Top Left:
718|956|324
452|387|619
936|142|587
-----------
341|528|796
695|713|842
287|694|153
-----------
523|461|978
869|275|431
174|839|265

Top Right:
635|428|971
214|679|853
897|531|426
-----------
976|382|514
152|764|389
348|915|267
-----------
463|857|192
529|146|738
781|293|645

Bottom Left:
728|154|396
194|836|752
653|279|184
-----------
519|368|427
346|725|918
872|491|563
-----------
231|987|645
467|512|839
985|643|271

Bottom Right:
215|849|637
348|627|591
976|153|824
-----------
869|431|752
154|972|368
732|568|149
-----------
521|796|483
483|215|976
697|384|215

Center:
978|215|463
431|678|529
265|439|781
-----------
813|754|692
649|382|157
527|961|834
-----------
396|847|215
752|196|348
184|523|976

Lessons learned: "automate automate automate!" and "Don't work hard, but work smart!" :)

The algorithm is very simple. The model is based on the simplest unit - a cell which is part of a row, column and a square. Each puzzle consists of 9 columns, 9 rows and 9 squares. Cells are flexible enough to be part of more than one square, row or a column at a time, so I can have cells that are present in more than one puzzle at a time. With a flexible model like this, all that the program needs to do is to use some basic rules to eliminate candidates and determine cell values. This type of solver is often referred to as human-style solver, because it uses the same techniques used by humans.

If anyone is interested in having a look at the source code it can be downloaded from here: multi_sudoku_solver.rb. A word of warning - the code is not cleaned up nor documented. This was just a fun project that I worked on, and the fact that I achieved what I set out to achieve was a good enough milestone to call this project done.

After this experience I don't feel like solving sudoku puzzles any more. They are just a repetitious problems that are well suited for automated solution. Most importantly - I actually had more fun writing the program than I had with solving the puzzle :).