simplericity - glassfish tag

Slf4j on Glassfish: When smart ideas collide

Fri, 02 Jan 2009 11:02:56 GMT

Have you ever noticed how two ideas can seem really smart one by one, but not all that great when combined?

I recently tried to install Nexus, the excellent Maven repository manager from Sonatype. (And by the way: Every organization using Maven needs a repository manager and Nexus is the best one out there!)

A secondary goal of the project I was working on was to get some exerience using Glassfish. But sadly, Nexus failed to deploy on Glassfish:

All I got was this error message in the admin console. A closer look in the Glassfish logs didn't reveal any stacktrace or relevant messages. Since my project was a pretty busy one, I soon gave up and found another way to run Nexus.

A couple of days ago however, I decided to find out what caused the StackOverflowError. I restarted Glassfish with debugging enabled and attached IntelliJ IDEA's debugger to it. After setting an exception breakpoint it was easy to see what was really happening under the hood.

I redeployed and got a stacktrace starting with this:

httpWorkerThread-4848-1@1033 daemon, priority=10, in group 'Grizzly', status: 'RUNNING'
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:491)
at java.lang.StringBuffer.append(StringBuffer.java:302)
at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:132)
at java.text.DateFormat.format(DateFormat.java:314)
at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:444)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:64)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:503)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:301)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:597)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:169)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:242)
at java.util.logging.Logger.log(Logger.java:452)
at java.util.logging.Logger.doLog(Logger.java:474)
at java.util.logging.Logger.logp(Logger.java:590)
at com.sun.common.util.logging.LoggingOutputStream.flush(LoggingOutputStream.java:90)
at java.io.PrintStream.write(PrintStream.java:414)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
at sun.nio.cs.StreamEncoder$CharsetSE.implWrite(StreamEncoder.java:395)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:136)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:146)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:204)
at java.io.Writer.write(Writer.java:126)
at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:47)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:301)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:597)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:169)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:242)
at java.util.logging.Logger.log(Logger.java:452)
at java.util.logging.Logger.doLog(Logger.java:474)
at java.util.logging.Logger.logp(Logger.java:590)
at com.sun.common.util.logging.LoggingOutputStream.flush(LoggingOutputStream.java:90)

and then about 1000 more lines of the same. There was clearly some kind of loop in the logging system that eventually caused the StackOverflowError. How could this happen?

I knew Nexus was using SLF4J, the Simple Logging Facade for Java. SLF4J was set up to log to Log4J. And Log4J was set up with a ConsoleAppender. From the stracktrace I could tell that Log4J wasn't printing its output to the console, but to a LoggingOutputStream. Some googling revealed that LoggingOutputStream is set up by Glassfish to catch any output to System.out and redirect it to the Java Logging API (JUL).

So Log4J was obviously feeding its log lines into what it thought was the console while it was actually fed into the Java Logging API instead. But why did this cause a loop?

The answer is found a few lines up the stacktrace in a class called SLF4JBridgeHandler. This class installs a bridge that will redirect all logs from the Java Logging API back to SLF4J. Which is pretty smart thing to do if you have code logging to JUL, but you want all code to log to be handled throgh SLF4J instead. Combined with Glassfish's LoggingOutputStream however, it caused an infinite loop:

The Nexus application code logs to SLF4J
SLF4J logs to Log4J
Log4J writes its log lines to System.out through the configured ConsoleAppender
The log lines are processed by Glassfish's LoggingOutputStream and sent to Java Logging
Java Logging sends its logs to SLF4J because of the SLF4JBridgeHandler.
SLF4J logs to.. Wait.. Houston, we have a problem!

So the next time you come up an idea you think is really smart, take a minute or two to consider the possibility that right now, somewhere else in the world, someone else might be thinking about an equally smart idea that when combined with yours will make the world explode.

This issue is reported in the Nexus issue tracker as NEXUS-1307.

Exploring the Glassfish commit history with SVNSearch

Sat, 16 Aug 2008 14:12:00 GMT

I recently added the Glassfish Subversion repository to SVNSearch.org. Paul Sterk, the community manager for Glassfish, asked if I could provide some background on SVNSearch.org and the benefits of having Glassfish indexed there. So, here are some ways to use SVNSearch:

Who changed what, when?

SVNSearch helps you keep track of, understand and learn from changes in version control systems. The basic interface shows you the most recent commits. Here's a screenshot from the Glassfish project:

But SVNSearch can do better than that. You can limit your search by log message, path, file name, file type, date, author or a referenced issue number. The following screenshot shows all commits done by kohsuke last year in the /gfsvn directory having the word "fixed" in the log message:

You might have noticed the charts on the left side of the screen. These statistics show commit distribution by path, time and author and they are generated for each search you make.

What changes were made to file X in revision Y?

Once you've find the commit you're looking for you might be interested in see what changes were actually made in the files affected by the commit. By clicking on the a path SVNSearch shows you a side-by-side colored diff of the change, including inline changes:

Which developers work on the same part of the project?

For each file in the repository, SVNSearch knows who made any change to that file. It also knows who else have touched that file and uses this information to provide a reasonably sophisticated collaboration graph. This graph will have one node for each developer in the project and there will be an edge between any two developers that have worked on the same code. Developers tend to be placed near closely if they've worked on lots of common files.

Developers who have committed recently are red while people who haven't committed in a while gets a more blue color. The Glassfish shows a group of developers in the upper left part of the graph, so let's zoom in on them and show their names:

The collaboration graph can be also useful for identifying parts of the code base that have been abandoned or for learning about the knowledge flow in the project. Sometimes two groups of developers are connected by a single developer, that means this developer is probably important for the flow of knowledge in the project since he's the only one who's worked on both parts of the code base.

When do people commit?

Say you want to get an overview of who is or has been involved in a project. You don't care about the details what actually changed so a detailed change list doesn't actually help you. For these cases, SVNSearch provides a timeline plot of all changes in the project, grouped by author.

Every red dot in this plot is a commit. The X axis is the timeline of the project and the Y axis actually show time of day of the commit. The list of authors can be ordered by name, first commit, last commit, total lifespan or number of commits which is the ordering used in the above example from Glassfish.

Developer turnover

Long lived software projects always experience a certain degree of developer turnover. People leave the project and new developers join. SVNSearch includes a developer turnover chart that is based on the work of Gregorio Robles and Jesus M. Gonzalez-Barahona of Universidad Rey Juan Carlos, Spain.

The idea is to track the 20% most active active developers in each one of ten intervals of the total timeline of the project and then track how active these developers are in the rest of the time intervals of the project.

While the theory is interesting, you don't really have to understand it all to make use of it. A rule of thumb is that many horizontal lines indicates a stable core group, while diagonal lines signals that the core group is being replaced.

The Glassfish project has a quite stable core group according to SVNSearch:

"Let me know when it's committed"

Dependencies is something we have a lot of, not only in our software but also in software development. Have you ever had to wait for someone else to make a change in "their" code before you can continue with "your" coding?

Any search in SVNSearch has an RSS feed, so if you're waiting for a specific change, why not just search for that and then subscribe to the feed? Here's an example show recent changes in Glasshfish using the RSS Screen Saver in OS X:

Background and future of the SVNSearch project

I started working on SVNSearch about four years back. The company I worked for at the time used CVS and suggested we started using Subversion instead. There tool support for Subversion wasn't all that good at the time but I was able to find alternatives for most of our tool chain. The CVS tool that I couldn't find a replacement for was the query interface in ViewCVS. So that's how I started working on SVNSearch.

Over the years I've added the features you've seen here and more, but I've spent most of the time making sure SVNSearch is really, really fast and making it scale to millions of revisions. SVNSearch.org today indexes over 3.3 million revisions with the largest repository being KDE with over 840.000 commits alone.

SVNSearch.org is a free service I provide to the open source community. My plan is to offer SVNSearch as a product on a commercial basis and I have a few customers in a private beta round now. I can't promise any public release date yet, but if you're interested in joining the beta, let me know.

I've recently added enterprise features such as LDAP integration, Kerberos / Active Directory Single Sign-On, and role based security. But those features are more interesting when using SVNSearch inside a company firewall and probably not that interesting for an open source project like Glassfish :-)

Oh, just one more thing: I always enjoy feedback!

Whether you like SVNSearch or not, I'd love to hear your thoughts. Perhaps you've found a bug, have an idea for a new feature or just want to let me know you're using it. In any case don't hesitate to use the feedback form on SVNSearch.org or just send me an email. (feedback at svnsearch dot com)