simplericity

Exploring the Glassfish commit history with SVNSearch

I recently added the Glassfish Subversion repository to SVNSearch.org. Paul Sterk, the community manager for Glassfish, asked if I could provide some background on SVNSearch.org and the benefits of having Glassfish indexed there. So, here are some ways to use SVNSearch:

Who changed what, when?

SVNSearch helps you keep track of, understand and learn from changes in version control systems. The basic interface shows you the most recent commits. Here's a screenshot from the Glassfish project:

But SVNSearch can do better than that. You can limit your search by log message, path, file name, file type, date, author or a referenced issue number. The following screenshot shows all commits done by kohsuke last year in the /gfsvn directory having the word "fixed" in the log message:

You might have noticed the charts on the left side of the screen. These statistics show commit distribution by path, time and author and they are generated for each search you make.

What changes were made to file X in revision Y?

Once you've find the commit you're looking for you might be interested in see what changes were actually made in the files affected by the commit. By clicking on the a path SVNSearch shows you a side-by-side colored diff of the change, including inline changes:

Which developers work on the same part of the project?

For each file in the repository, SVNSearch knows who made any change to that file. It also knows who else have touched that file and uses this information to provide a reasonably sophisticated collaboration graph. This graph will have one node for each developer in the project and there will be an edge between any two developers that have worked on the same code. Developers tend to be placed near closely if they've worked on lots of common files.

Developers who have committed recently are red while people who haven't committed in a while gets a more blue color. The Glassfish shows a group of developers in the upper left part of the graph, so let's zoom in on them and show their names:

The collaboration graph can be also useful for identifying parts of the code base that have been abandoned or for learning about the knowledge flow in the project. Sometimes two groups of developers are connected by a single developer, that means this developer is probably important for the flow of knowledge in the project since he's the only one who's worked on both parts of the code base.

When do people commit?

Say you want to get an overview of who is or has been involved in a project. You don't care about the details what actually changed so a detailed change list doesn't actually help you. For these cases, SVNSearch provides a timeline plot of all changes in the project, grouped by author.

Every red dot in this plot is a commit. The X axis is the timeline of the project and the Y axis actually show time of day of the commit. The list of authors can be ordered by name, first commit, last commit, total lifespan or number of commits which is the ordering used in the above example from Glassfish.

Developer turnover

Long lived software projects always experience a certain degree of developer turnover. People leave the project and new developers join. SVNSearch includes a developer turnover chart that is based on the work of Gregorio Robles and Jesus M. Gonzalez-Barahona of Universidad Rey Juan Carlos, Spain.

The idea is to track the 20% most active active developers in each one of ten intervals of the total timeline of the project and then track how active these developers are in the rest of the time intervals of the project.

While the theory is interesting, you don't really have to understand it all to make use of it. A rule of thumb is that many horizontal lines indicates a stable core group, while diagonal lines signals that the core group is being replaced.

The Glassfish project has a quite stable core group according to SVNSearch:

"Let me know when it's committed"

Dependencies is something we have a lot of, not only in our software but also in software development. Have you ever had to wait for someone else to make a change in "their" code before you can continue with "your" coding?

Any search in SVNSearch has an RSS feed, so if you're waiting for a specific change, why not just search for that and then subscribe to the feed? Here's an example show recent changes in Glasshfish using the RSS Screen Saver in OS X:

Background and future of the SVNSearch project

I started working on SVNSearch about four years back. The company I worked for at the time used CVS and suggested we started using Subversion instead. There tool support for Subversion wasn't all that good at the time but I was able to find alternatives for most of our tool chain. The CVS tool that I couldn't find a replacement for was the query interface in ViewCVS. So that's how I started working on SVNSearch.

Over the years I've added the features you've seen here and more, but I've spent most of the time making sure SVNSearch is really, really fast and making it scale to millions of revisions. SVNSearch.org today indexes over 3.3 million revisions with the largest repository being KDE with over 840.000 commits alone.

SVNSearch.org is a free service I provide to the open source community. My plan is to offer SVNSearch as a product on a commercial basis and I have a few customers in a private beta round now. I can't promise any public release date yet, but if you're interested in joining the beta, let me know.

I've recently added enterprise features such as LDAP integration, Kerberos / Active Directory Single Sign-On, and role based security. But those features are more interesting when using SVNSearch inside a company firewall and probably not that interesting for an open source project like Glassfish :-)

Oh, just one more thing: I always enjoy feedback!

Whether you like SVNSearch or not, I'd love to hear your thoughts. Perhaps you've found a bug, have an idea for a new feature or just want to let me know you're using it. In any case don't hesitate to use the feedback form on SVNSearch.org or just send me an email. (feedback at svnsearch dot com)

Yes, cross-platform Single Sign-On for Java Webapps is possible.

A customer participating in the SVNSearch beta round asked me about Single Sign-On solutions for SVNSearch. SSO has become something of holy grail in the Enterprise. Everyone wants it but few seem to really achieve it. The Wikipedia article for Single Single-On even cites a Gartner report stating that "no one can achieve it without a homogeneous IT infrastructure".

What could be a better motivation than Gartner saying it's impossible? I decided I'd prove them wrong and took on the task to create a cross platform, Single Sign-On solution for SVNSearch.

The NTLM headache

As it turns out, there are only a couple of options available for Web SSO . The Java CIFS Client Library (JCIFS) implements Microsoft's NTLM technology. NTLM is a challenge-response authentication mechanism, and being a product of Microsoft it's not much of a surprise that it has a history of security weaknesses.

NTLM has caused me lots of headache on a past project because it doesn't play nice with the basic rules of HTTP. NTLM is a connection-oriented protocol, so both the server and the client needs to keep the same TCP connection open for at least the three request it takes to do the authentication. This is just fine when Internet Explorer is talking to Microsoft's IIS server, but when you introduce a web server like Tomcat, weird things start happening. In our case Tomcat would start sending Connection: close headers under heavy load. This is a perfectly sensible thing to do – if you're running low on available threads, turing off keep-alive is better than refusing requests. But in the NTLM case it made Internet Explorer fail with a non-decipherable error page.

Finally, NTLM isn't really cross platform. Both Firefox and Safari have adopted NTLM, but they'll pop up a username/password dialog on non Windows platforms.

Enter Kerberos

So you might not be surprised to hear I wanted something better than NTLM. I'd been reading about Kerberos for years, but my impression was that it required an army of sysadmins to set up and a Ph.D in cryptography to create software for it. Then I realized Microsoft actually use Kerberos for their domain controllers so I already had the infrastructure set up. I started looking for open source projects implementing Kerberos authentication, but I couldn't really find anything. There were a lot of pointers to a an old patch providing Kerberos support for JCIFS, but it doesn't seem like it ever got applied.

So it seemed like I had no option but to roll my own. In doing so, I had to spend some time trying to understand how Kerberos works. And it's not really that hard. Every time you log on to your Windows Domain Controller, Windows acquires a Kerberos token from the Domain Controller. This token can be used to fetch another type of token that lets you log into network services like my Java based web server. The service also has to authenticate itself and that is taken care of by issuing KeyTab files containing secret keys for the service.

The implementation

Kerberos is implemented in Java through the GSS API. GSS is a just a wrapper API around token based security mechanisms like Kerberos and doesn't provide any security itself. The authentication of the service is done through a JAAS LoginModule, the Krb5LoginModule. The module needs to be configured to use a KeyTab file and to store the credentials in the Subject. This way the credentials can be used in the next step when authenticating users. You only need to do the service login once, after that you can keep the Subject around as long as the service is running.

It's important to name the service principal correctly. The principal name needs to match the host name of the service and the host name needs to have a DNS A record who's IP address resolves back to the host name. User authentication is done through the GSSAPI. I'm using the GSSManager to create a GSSContext. This context is then fed the Kerberos token in the acceptSecContext() method and if isEstablished() returns true we can fetch the identity of the authenticated user with getSrcName().

Web-based Kerberos

Kerberos was invented before the web and doesn't specify how tokens should be transported in a web context. Microsoft implemented SPNEGO in Internet Explorer 5.01 as an HTTP authentication mechanism. When the server requires user authentication it sends a HTTP 401 status back to the user with a «WWW-Authenticate: Negotiate» header. The browser then resubmits the request, now with a «Authorization» header containing a security token. This ping-pong game continues until the underlying security mechanism is negotiated and the user is authenticated. SPNEGO is a pseudo mechanism used to negotiate the use of a real mechanism in the case that the server and client don't agree on a mechanism upfront. SPNEGO is implemented in Java 1.6 as a part of the Java GSS API. However, the SPNEGO token is DER encoded so it's pretty straight forward to extract the real Kerberos token from a SPNEGO token if you're running on Java 5 or 1.4.

Cross-platform support

OS X supports Kerberos. You can set up OS X to acquire Kerberos tickets when logging in or you can use kinit command line tool or the graphical utility Kerberos.app for ticket administration. Most Linux distributions also have Kerberos support. Kerberos is supported in both Safari and Firefox, although Firefox doesn't send SPNEGO tokens but pure Kerberos tokens instead. So my code needed to take that into account.

The result

Although it did require a fair amount of research and development I must say I'm very happy with the end result. I now have a 100% pure Java Single Sing-On toolkit. Having complete control of the authentication process also gives some nice extras. I can present the user with a fallback login page if automatic login fails. I can also let the administrator test the Kerberos configuration and provide some useful feedback if something doesn't work.

If there's any interest in this code please let me know. I might consider making a small open source project out of it or contribute it to an existing project. And to the Gartner dudes: I now have Kerberos based SSO solution that works great in a cross-platform environment, so go eat your socks!

Implementing decorator templates with Apache Velocity

A couple of years ago I started replacing JSPs with Apache Velocity for generating HTML in my web applications. The simplicity and speed of Velocity was very appealing to me compared to the complex beast that JSP is.

Compiling JSP to Java, then to byte code and then loading and running it as class obviously must have seemed like a good idea to those who came up with it. For me, it's always been crazy talk.

When designing HTML templates reusability is always an important goal. You want to define your header, menu and footer once and then reuse them for both your front page, article page or about page.

The no-brainer solution to this is to use tags or directives to include common snippets of HTML into the final page. JSP supports this with the @include syntax or the <jsp:include/> tag:

Like any no-brainer solution this one is not very smart. You end up with brittle and inflexible templates where you need to touch all your templates to do any kind of structural changes to your main design.

A much better solution is to apply the decorator pattern. Instead of pulling in common pieces of HTML everywhere you need them you just specify what's special with this particular page and then apply a decorator template to it. This template will take care of creating the actual design.

There's no built in support for creating decorator templates in JSP, so what used to do was to create a small taglib that would let me define a snippet of HTML as a section and then output that snippet in the design template. I know a lot of people use Sitemesh for this, but I found the taglib solution worked very well for my purpose.

Like JSP, Velocity has no built in support for creating decorator templates. At first I just accepted this situation as a trade-off. With Velocity being so simple, I'd just have to learn to live with all the #incude directives right?

That was until I discovered that Velocity actually has support for custom directives. So now all I had to do was to create a directive that rendered it's body into a string and stored that in the context for later use. This turned out to be very easy:

public class Section extends Directive {
     public String getName() {
         return "section";
     }
 
     public int getType() {
         return BLOCK;
     }
 
     public boolean render(InternalContextAdapter context, Writer writer, Node node) throws IOException, ResourceNotFoundException, ParseErrorException, MethodInvocationException {
         ASTReference nameNode = (ASTReference) node.jjtGetChild(0);
         String name = nameNode.getRootString();
         StringWriter sw = new StringWriter();
         node.jjtGetChild(1).render(context, sw);
         context.put(name, sw.toString());
         return true;
     }
 }

Now I can create templates like this:

#section($content)
<h1>Content here</h1>
<p>Hello World</p>
#end

#parse("design.vm")

Registering my #section directive with Velocity was easy. Adding a userDirective property with the name of the directive class did the trick.

Seeing how easy this is and how much better it made my template system I'm amazed that Velocity doesn't already have something like this in its core.