Chip's Instant Managed Beans

Logging informational and error messages is a common approach to debugging in the lab and troubleshooting in the field. So common, that there are two widely used Java frameworks for managing message logging: the logging package that now comes with Java, and the more enterprise-oriented Apache log4j. And every C/C++ project I've worked on in the past decade or so had some kind of log message management framework, even if it was roll-your-own.

But message logging sometimes isn't all its cracked up to be. Frequently it isn't even possible. Most embedded projects I've worked on in the past three decades had no persistent store of any kind. The smaller projects were lucky to have a console serial port buried on the board somewhere, and the more complex ones might have had an Ethernet port. But even if there was a consenting syslog server to which to log messages, logging wasn't always practical or possible on real-time systems because of the processing overhead and bandwidth required to log the potential firehose of output when things really got sideways. More than once, unfortunately sometimes while troubleshooting a customer system, I have turned up the log level only to discover that the CPU cost was so high the hardware missed its watchdog poke. Wackiness ensued.

Even under the best of circumstances, a huge message log is frequently just more information than you need. Nothing like having your trusty field support person FTP a twenty megabyte log file to your desktop and tell you the customer wants to know what happened by close of business. (Fortunately, a helpful manager will call you every twenty minutes to "assist with your analysis".)

My embedded mentors Tamarra Noirot and Randy Billinger patiently taught me another approach that is an alternative or sometimes a supplement to logging: collections of counters managed as objects and exposed upon demand to the outside world. Sometimes you would give an eye tooth to know if at any time in the past had the board ever encountered a hard error on an I/O port, had the signaling stack ever thrown an exception, had a DSP ever been rebooted, had an incoming packet ever arrived with an invalid checksum. If you (or your trusty field support person) could just query the system to see if any of those error counters had ever incremented, you could, perhaps, at least come up with a plausible story of what probably happened. And maybe even get a clue that would help you fix the problem. Or at least shift the blame. Incrementing a counter is a whole lot cheaper than logging a message, and frequently carries just as much information.

There are several mechanisms to expose counters to the outside world. Sometimes it's a simple command line interpreter and a serial port. On systems which have an internet prototocol stack, it's telnet or secure shell. On yet more complex systems, it's an SNMP MIB.

If you are developing in Java, you're in luck: the latest release comes with a remote management capability: managed beans. A bean is a common Java design pattern. It is a class which has a no-argument constructor, its properties are accessible and modifable via public gettor and settor methods named according to a convention, and it is serializable. Hence, Java objects that are beans can be created, accessed, modified and even persisted, using Java's reflection mechanism, without any prior knowledge of the implementation. A managed bean, or mbean, can be registered with an mbean server so that it can be accessed and modified through the server, even remotely across a network. Java 1.5 has a built-in platform mbean server, and also includes jconsole, a GUI tool with which to browse registered mbeans.

This stuff makes embedded developers' mouths water. It you're not into Java, imagine having a standard mechanism that allowed you to export C++ objects to an SNMP MIB, than having a MIB browser that let you examine the contents of those objects, change them, and even call methods inside of them. It would give you the capability to monitor and control applications from the laptop in your office, even if those applications are deeply embedded in your remote servers without any external user interface.

As much as this sounds like rocket science, implementing a standard mbean in Java is actually pretty simple. But standard mbeans, whose capabilities and interface are fixed at compile time, aren't that flexible. Dynamic mbeans, on the other hand, are incredibly flexible, but not that simple to implement.

What we need is a serving of Chip's Instant Managed Beans. Instant mbeans are general-purpose dynamic mbeans that allow you to instrument your application with almost no effort at all. Instant mbeans come in two delicious flavors: Counters and Parameters.

The Counters instant mbean lets you expose an array of long integer counters inside your application to a management tool like jconsole. You can watch activity counters inside your application increment, giving you that warm fuzzy feeling that your software is actually doing something. You can reset error counters and see if they change. You can monitor high water marks. You can even alter tuning parameters in real-time.

And instant mbeans are so easy to use. Here is a code snippet that creates an instant mbean for some error counters in an application.

enum Counter {
RECEIVED_INVALID_MESSAGE,
ILLEGAL_STATE_ENCOUNTERED,
CAUGHT_IO_EXCEPTION,
SHOULD_BE_IMPOSSIBLE
}
 

Counters counters = new Counters(Counter.class);

Next we register the instant mbean with an mbean server under a Java object name. Sure, we could use any mbean server and object name we wanted to, but if we don't do anything more than the code snippet below, the Counters mbean will be registered with the default platform mbean server under a perfectly usable default object name.

counters.start();

Now if our application encounters an illegal state in its state machine, it executes the following code snippet to increment the appropriate counter. It can even log it, as shown below.

counters.inc(Counter.ILLEGAL_STATE_ENCOUNTERED);
 

logger.warning("illegal state encountered: count="
+ counters.get
(Counter.ILLEGAL_STATE_ENCOUNTERED));

If we remembered to run our application with the JVM flag

-Dcom.sun.management.jmxremote

so that the platform mbean server allows connections from the outside world, then all we need to do is run jconsole on the server itself, or remotely using the appropriate URL, and the values of all our error counters will be visible through the GUI to watch and even to modify.

Here is a screen snapshot of jconsole managing an application using exactly the code from above.



Chip's Instant Managed Beans are just that simple. But wait, there's more!

If your application is configured using, for example, a Java properties file, you can use the Parameters instant mbean to expose the Properties object. You can observe through jconsole what the configuration parameters are, and even alter them in real-time. And no inefficient polling here! The Parameters instant mbean lets you register a callback object so that your application will be notified when a parameter has been changed through jconsole.

Given the properties file

ConfigurationFile.properties

with the following keyword=value pairs

AcceptTimeoutPeriodMs=2000
MaximumQueueSize=200
Temporary=/var/tmp
RootPath=./Buckaroo

here is a code snippet to create a Parameter instant mbean from a Properties object.

Properties properties = new Properties();
InputStream stream =
new FileInputStream
("ConfigurationFile.properties");
properties.load(stream);
 

Parameters parameters = new Parameters(properties);
parameters.start();

Now we can use jconsole to see the how our application was configured, and even to change the configuration as it runs.

Here is another screen snapshot of jconsole managing an application using exactly the code from above.



You can start out using instant mbeans to quickly get your application instrumented. Then, as the mood strikes you, use the instant mbeans as examples for developing your own more complicated managed beans.

The Counters and Parameters instant mbeans are part of the Buckaroo open source Java library available from Digital Aggregates under the Apache License. Chip's Instant Managed Beans makes remote application management easy and fun for the whole family!

Sources

Buckaroo, Digital Aggregates Corp., 2006