How to Install a Glassfish Patch

Recently I've been working quite extensively with glassfish and had a need to apply a patch for some of the core code.

To my big surprise I was not able to find the official way to apply patches to a glassfish installation. I found several patches posted on the issue tracker and paying customers have access to tested and supported patch bundles, which are released outside of the regular release cycle. But even some extensive googling didn't easily reveal how to apply them.

Then luckily I found this discussion on the glassfish mailing list, which describes the process.

To make life easier for others and google to index this information, here is a brief recap:

1. Create a directory <glassfishroot>/lib/patches
2. Copy the jar with your patch into this directory
3. Edit your <glassfishroot>/domain/<domainname>/config/domain.xml and add attribute classpath-prefix="${com.sun.aas.installRoot}/lib/patches/<patchname.jar>" to the <java-config ...> node
4. Restart your domain

I tested this with Glassfish v2, I'll have to check if it works with the upcoming v3 as well.

My Confluence 3.0 Wishlist

Confluence 2.9 was released last month and I've seen references to 2.10 in the Confluence issue tracker, so I expect to see it out in 1-2 months. That makes me think about what's next.

As a part of my adventures of working on Sun's external wiki wikis.sun.com, I've been working on Confluence plugins and even the Confluence core code for a year and a half now, adding new features, enhancing the existing features and very often fixing bugs. Sometimes it was trivial to enhance the code or fix a bug, other times it was not, but what I want to write about today are things that were not possible at all without irreversibly forking the code.

Confluence 3.0 should be a version that really deserves to have the first digit incremented. Not because marketing said it's time for that, but because the changes in the application are so significant.

I'm sure that Atlassian has lots of ideas about what Confluence 3.0 should look like, but Atlassian guys, in case you start to run out of ideas, here is my wish list:

Fix the Database Schema

Confluence has been in development for years and the database schema definitely shows that. Since the database is the heart of the application, I think it deserves a lot of attention and major performance boost could be gained by doing a clean up.

Specific improvements:

• Establish and in the future enforce naming conventions
• Replace all the natural foreign keys with surrogate keys, e.g. user name, spacekey, group name should be replaced with ids in all the referencing tables (this would finally allow CONF-4063 to be implemented)
• Add caches for the lower function (patch) and maybe counter caches

Rework the Clustering

Clustering is usually supposed to fulfill two functions: scalability and robustness. In the case of Confluence mainly the second attribute is missing. In fact, I'd go as far as saying that a Confluence cluster is less robust than a single instance of Confluence. Why? Because the way it is implemented makes the entire cluster vulnerable when one node has problems.

I personally experienced several cluster lock-ups or crashes, usually initiated by a separate Confluence bug, in which the effect was multiplied by the clustering code. One of such of these bugs: CONF-12319

Mike's presentation covers quite a few design goals behind the implementation in Confluence. Clustering can really get ugly and complicated and Mike covered it pretty well. Unfortunately the distributed share part of the clustering makes Confluence prone to problems.

One of the clustering goals that Mike emphasizes in his presentation is that clustering should be "admin-friendly" (low admin overhead and easy setup). While I agree with the low overhead part, the easiness of setup should not compromise the goals which clustering is trying to fulfill in the first place. Clustering is for people who are serious about running Confluence, and as such should be expected to be qualified for the job.

Specific improvements:

• Either reevaluate the distributed share clustering so that it is super robust, or consider implementing clustering via a centralized share
• Avoid shutting down the entire cluster when "cluster panic" is detected. A better solution, which avoids unnecessary downtime, would be to shut down all the nodes, except for the nodes properly clustered with the oldest node.

Clean Up the HTML and CSS Code

The html code that comes out of Confluence is horrendous. While the rendered output looks pretty pleasant, looking under the hood (browsing the source code in a browser) is not recommended for pregnant women, men with ED, high cholesterol, and generally not recommended for people over 50.

Some improvements were done in this area in the recent releases, but all of them were just minor cosmetic surgeries. Confluence really needs major surgery that will bring the html code up to current standards. The benefit of this will be much faster page loads and code that is easier to maintain and enhance.

Specific improvements:

• Rewrite most of the templates and macros to make them XHTML 1.0 compliant
• Minify and combine javascript and css files (CONF-8622)
• Use image sprites to even further speed up page loads (especially in the rich text editor)

Redo the URI Namespace

Human friendly URIs and URLs are becoming more and more important on today's Internet. Confluence is not doing well in this area.

Specific improvements:

• /display/MySpace/My+Page - is the /display part really necessary? Can't we do /MySpace/My+Page
• /pages/diffpages.action?pageId=2490471&originalId=45714293 - What is this? I don't know. How about: /MyWiki/My+Page/diff/23:22. I think that actually means something. There might be a better format, this is just a thought.
• I think in general redoing the URI name space using REST conventions would be interesting.

Improve Atlassian-Renderer

When I was creating come patches for the atlassian-renderer I was surprised to find that atlassian-render, the module responsible for rendering wiki markup into html is full of hardcoded html snippets. The main reason why this surprised me is that most of the Confluence code is pluggable, which allows for parts of the code to be replaced with a better version without a lot of problem. This is not the case with the render. And this presents two problems: it's not possible get Confluence to directly render anything else than html (pdf and doc are only derived from the html), and it's not possible to use anything else than Confluence markup as the input for the renderer.

The first problem makes me unable to render custom output like docbook or to improve the PDF output, which is pretty poor.

The second issue means that all the customers that use Confluence are locked-in because all the content created via Confluence is Confluence-specific and can't be easily moved to a different wiki engine when needed.

In my opinion the sooner all major wiki engine developers settle on one wiki markup standard the sooner we will all be better off. This might be especially difficult for Atlassian to swallow and implement, because they standardized on their own markup that they also use in their other products.

An interesting initiative that is gaining a lot of traction is Creole, a standardized wiki markup. Confluence is one of the few major wiki players that doesn't support this initiative.

Specific improvement:

• Split the current renderer into two pluggable parts: parser and renderer
• Implement Creole support (CONF-12077)

Improve Developer Documentation

spent countless hours, especially in the beginnings trying to figure out how Confluence works and how Confluence plugins should be written. I learned some new tricks and that's the good part, the bad thing is that the experience could have been much better if the the code contained more javadocs comments and if the plugin interfaces and mainly the configuration file format was better documented.

Specific improvements:

• Add JavaDoc comments where missing
• Finally provide complete specification and documentation for the plugin config file (JRA-12183)

Thanks! :-)

That's about as much as I can think of for now. There are probably other things that I missed and then there enhancements around security, which I know are already on the roadmap.

I understand that most of the changes above will create incompatibilities with many existing themes and plugins, but hey, Confluence 3 will happen only once EVAR and releases like this are expected to bring major incompatibilities. Data can always be migrated automatically and existing plugins and themes will be migrated when there are people interested in using them and proper migration instructions are provided.

I hope that Confluence 3 will not be a "marketing" release, but instead something really cool that all users and developers will enjoy working with.

Unintrusive but secure passwordless ssh authentication

On a daily basis I need to log in to many remote servers inside or outside of Sun via SSH, often dozens of times per day. This can get pretty tiresome if you need to type in your password with every log in.

Some suggest setting up so-called "passwordless" authentication by generating ssh keys and specifying empty passphrase for the private key. This will result in passwordless authentication, but will also decreased security. Should anyone get hold of your private key, (s)he'll get access to all of your remote systems.

ssh-agent can help a lot in keeping the security level high and minimizing the number of times you need to type in the password. However, if you use a terminal with tabs or use both local and remote terminals on your workstation, you'll end up running many ssh-agent processes and having to authenticate every time you start such a process, which diminishes most of the conveniences of using ssh-agent.

Frustrated with this situation and with a bit of help from Martin, I created a shell script, which I added to my .bash_profile startup script. All I have to do now is to authenticate when my first terminal session starts and I'm good until the next time I restart my OS. sweeeet...

Here is how you could set it up on a workstation and a remote-server:

First, if you haven't generated your private/public ssh key pair, do that now:

workstation $ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/user/.ssh/id_rsa): **********************
Enter same passphrase again: **********************
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
01:a6:95:23:1c:74:53:c7:f4:87:07:a2:50:ef:99:16 user@Computer.local

Now make sure that the file system permissions are set up correctly:

workstation $ cd ~/.ssh/
workstation $ ls -l
total 56
-rw-------  1 user  staff  1743 Aug 31 00:13 id_rsa
-rw-r--r--  1 user  staff   398 Aug 31 00:13 id_rsa.pub

The id_rsa file must be readable only by owner, if this is not true, the key will be ignored.

On the remote server you need to authorize your newly generated key pair by appending it's public key to the ~/.ssh/authorized_keys file under your remote home directory:

workstation $ cat ~/.ssh/id_rsa.pub |ssh user@remote-server 'sh -c "cat - >>~/.ssh/authorized_keys"'

Now you can try to log in:

workstation $ ssh user@remote-server
Enter passphrase for /Users/user/.ssh/id_rsa: **********************
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
Last login: Thu Sep 11 20:19:19 2008
remote-server $

If you open a new tab in your terminal and try to log in again, you'll be asked to enter the passphrase yet again. This is where my script becomes useful. First download the script from Mediacast: ssh-agent-init.sh and store it somewhere in your home directory

workstation $ mkdir ~/bin
workstation $ cd ~/bin
workstation $ wget http://mediacast.sun.com/users/IgorMinar/media/ssh-agent-init.sh
workstation $ chmod o+x ~/bin/ssh-agent-init.sh

The next (last) step is optional if you want to start the script manually you can skip it.

I wanted to have this script automatically invoked when I start my terminal for the first time in the interactive mode. All I needed to modify were my .bash_profile (used for interactive sessions) and .bashrc (used for non-interactive sessions) startup scripts for my bash shell (modifications are in italics):

workstation $ cat ~/.bash_profile
...
...

. ~/bin/ssh-agent-init.sh

workstation $ cat ~/.bashrc
export NONINTERACTIVE=1
. ~/.bash_profile

(Note: I source the ~/.bash_profile script from the ~/.bashrc script to enable code reuse between the two scripts.)

That's it! If you now try to open a terminal tab for the first time, you'll be asked for passphrase. Once that is done, any new tab or any other shell session created under the same account will reuse the same ssh-agent process.

I've been using this script on my MacOS X laptop as well as OpenSolaris workstation for a few weeks now and it's been working like charm.

Enough with the Flash Already

It often bothers me when Flash is being misused for small visual effects on home pages of websites. Not just that it slows things down, but it also usually breaks these websites terribly when I want to copy&paste stuff or when one doesn't have the Flash plugin installed.

Margarete, has recently bugged me quite a bit about making the front page of her website a bit more "visual". She started with an idea of adding some flash to her website, which I dismissed quickly.

Fortunately I was able to come up with a neat little CSS code, which is way better than Flash for what she was trying to achieve. Don't be mistaken, I didn't even need JavaScript to do all the subtle movements and eager image loading, it's all pure CSS.

The front page:


A small "visual" effect when one mouses over the "Blog" element:


I still plan to enhance the code a bit more to make it faster to load by decreasing the number of HTTP requests needed to load the asset files as well as size of the images, but I'm quite happy with what it looks and feels like already. Check out the live version at www.fgslovakia.com.

A good technical article describing a technique very similar to the one that I used can be found the A List Apart website.

BTrace == DTrace for Java

Last week, I was trying to nail down a bug in SunWikis that was triggered by some kind of race condition. These kinds of issues are pretty nasty, especially if the application in question is a pretty complex beast, something that Confluence definitely qualifies as.

The debugger was no help because the timing had to be very precise for the issue to occur, and logging was totally useless because of zillions lines of logs that would be difficult to safely filter out. In many cases the information I needed was not present in logs anyway and since my guess was that the bug was coming from 3rd party code, logged data couldn't be easily expanded. DTrace, which I blogged about in my previous post, could have revealed some information, but I think that it would be very difficult to write a good D script that could give me some clues.

While waiting for my dev server to restart, I came across an interesting blog post that caught my eye. It mentioned this thing called BTrace, that I hadn't heard about before. It promised it to be a DTrace aimed at Java apps. With a fair amount of skepticism, I navigated to the BTrace website and started reading. Four things about BTrace earned my interest:

• It's based on same or very similar principals as DTrace, but specialized for Java apps
• No restarts necessary, you can observe a running app by connecting to a JVM identified by a PID
• The BTrace programs are written in a language that is a subset of Java, and heavily based on annotations. This was a bit of a turnoff for me at first, until I found out that the programs don't need to be compiled manually but btrace takes care of that for me. Bravo!
• I can call DTrace from my BTrace programs if I need to instrument native code or the OS

Armed with all this info and my eyes sparking, I wrote my first BTrace program and it worked as advertised! No restarts, no compilation, no hassle.

A few hours later, I had a program that was observing exactly those parts of Confluence that I needed. All of this while the application was running uninterrupted. I was able to snoop on how my code was interacting with Confluence and Confluence was in turn interacting with Hibernate in a way that I would have never dreamed of. All of this while concurrent events aimed at triggering the bug were happening.

Running BTrace programs is as easy as this:

$ ./bin/btrace \
  -classpath hibernate-2.1.8-atlassian.jar:spring-hibernate2-2.0.6.jar \
  6266 HibernateSessions.java

Where 6266 is the PID of my app/web server and HibernateSessions.java is my BTrace program.

Parts of the program looks like this (check out the javadoc comments for explanation of the code):

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import net.sf.hibernate.HibernateException;
import net.sf.hibernate.collection.PersistentCollection;
import net.sf.hibernate.engine.SessionImplementor;
import org.springframework.orm.hibernate.support.OpenSessionInViewFilter;

@BTrace
public class HibernateSessions {
  /**
   * A thread local variable used to filter out all the events that we
   * are not interested it.
   */
  @TLS private static boolean viaSso = false;
  private static boolean printStack = false;

  /** print a message when a thread enters doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter"
  )
  public static void enableProbing() {
      println(str(currentThread(), " - Entered SSO Filter"));
      viaSso = true;
  }

  /** print a message when exiting the doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter",
      location=@Location(Kind.RETURN)
  ) 
  public static void disableProbing() {
      println(str(currentThread(),  " - Exited SSO Filter"));
      viaSso = false;
  }

  /**
   * print an message with a HibernateException is thrown, with detailed 
   * info about the current context
   */
  @OnMethod(
      clazz="net.sf.hibernate.HibernateException",
      method=""
  )
  public static void onthrowreturn(HibernateException self, String s)  {
      println(str(currentThread(), 
              strcat(" - HibernateException was thrown: ",
              strcat(s, strcat(" | collection: ", str(collection))))));
  }

  /**
   * Print a message when Hibernate is attaching a collection to a session.
   * This was crucial info for me to get in order to resolve the issue
   */
  @OnMethod(
      clazz="net.sf.hibernate.collection.PersistentCollection",
      method="setCurrentSession",
      location = @Location(Kind.RETURN)
  )
  public static void setSessionForCollection(boolean returnValue)  {
      if (returnValue == true) {
        println(str(currentThread(),
          strcat(" - a collection was attached to a session. Collection:",
          str(collection))));
  }
}

The result of this was a workaround for my code as well as a pretty detailed bug report that Atlassian confirmed to be a very similar to a confirmed bug that was reported recently. I could hardly achieve this without BTrace or without an in-depth knowledge of how all the Confluence code works.

As great as BTrace is, it is still a beta. If you decide to use it in production environment you should keep that in mind. During the several hours I spent working with BTrace, I experienced one JVM crash/lockup caused by BTrace. I bet that this kind of issues will be ironed out soon, and often this kind of risk is worth undertaking in order to resolve otherwise untraceable bugs.

Since last week, BTrace is part of my toolkit and I already have other mysteries on my ToDo list that I want to shine some light on with BTrace. I applaud the BTrace team for giving us this amazing tool!

DTrace and Java - Observations and Docs

For a long time I've been hearing about DTrace and how cool it was. I read a lot about it and saw some presentation on observing Java apps with DTrace, but only recently have I found enough free time to extensively play with it. And oh boy, it really is cool. But even with all this coolness, dtracing Java apps feels a bit awkward.

I think there are two reasons for that. The D language, which looks like C, but has a few fundamental differences, is pretty far from what Java developers are used to. And the second reason is the fact that DTrace is in reality instrumenting JVM and not the application running in the JVM. This means that dtracing Java is limited by abilities of the Java DTrace provider and also that the transition between the different layers of the stack can add some awkwardness.

Having said that, DTrace is really awesome and can be enormously helpful in many cases when developing or troubleshooting (Java) apps.

One thing I really miss in DTrace is regex support in any part of D scripts. There is a limited globbing support and some workarounds, but they ain't pretty.

Here are some DTrace resources that I found useful:

• DTrace Wiki Docs - the complete documentation (I love to see cool docs like this at our SunWikis)
• a DTrace HowTo
• DTrace for Java at Solaris Internals
• DTrace Toolkit - a compilation of DTrace scripts written by DTrace pros

Catching StackOverflowError and a Bug in Regex Implementation in Java

Shortly after we launched SunWikis almost a year ago, we started having an issue with StackOverflowError, which was occurring when the content of certain wiki pages was being parsed and URLs where being extracted. I documented the issue in CONF-9392.

The problem is not really a bug in Confluence (even though using an overly-complex regular expressions doesn't help the case), but it's a problem in JDK. After a brief search at bugs.sun.com I found the root cause of our StackOverflowError documented as bug 6337993.

Originally I tried to mitigate the situation by increasing the memory reserved for the stack data via the -Xss JVM parameter. This helped a little bit, but wasn't good enough in most cases.

Last week I decided to go against everything I've been taught, and wrote a patch for Confluence that wraps the part of code that results in the StackOverflowError into a try/catch block. I know that any throwables that extend from Error should not be caught by a client code because they usually indicate a failure that only JVM should try to recover from, but IMO in the case of StackOverflowError, the situation is a bit different. That is mainly because before throwing the StackOverflowError, JVM pops the stack, so by the time the code execution gets to the catch block, JVM has already recovered from the error.

I don't claim this to be a solution to the problem, it's just a workaround that works better than increasing the stack size in this particular case. The fact that Confluence doesn't find all the URLs in wiki pages (used mainly to list outgoing links in the page info view) is just a small sacrifice, compared to inability to save or copy the page.

As for the solution, it seems that reimplementing Java's regular expressions library would be the most suitable one. I tried to run a code that fails in Java in JRuby, which uses a port of Oniguruma regex engine for Java and things worked flawlessly and as I read it also gives JRuby a performance boost over java.util.regex.