Thursday, August 16, 2007

What's Old Is New Again

Back in January I wrote an article on the Java Business Integration (JBI) specification, and my experience with a particular implementation of it, Apache's ServiceMix Enterprise Service Bus (ESB). One aspect of JBI really threw me for a loop back when I was in the trenches using it in a product development effort: the behavior of send versus sendSync.

JBI (and ServiceMix) offers two different mechanisms for sending a message across the ESB: send and sendSync. The send call is asynchronous, meaning it's fire and forget. Once the method returns successfully to your application, the only thing you really know is that your message exchange object has entered your delivery channel. This is true even if the message requires a response; the response will arrive later asynchronously on the same delivery channel.

It's like dropping a letter in the mailbox. Although it's likely it will be delivered to the destination, there is no guarantee. Furthermore, there is no guarantee that the order in which you dropped many letters addressed to the same recipient in the same mailbox will be the order in which they will arrive and be read by the recipient. (This is the part that really threw me.)

If all you are doing is sending in product registration cards for the new desktop PC system you just bought, it's no big deal. But if you are mailing deposits and money transfers to your bank, order is suddenly important: if all the transfers arrive and are processed before the deposits, some of the withdrawals associated with the transfers may bounce. That's because the former example is stateless, while the latter example is stateful.

JBI provides a sendSync call which is synchronous, meaning your application blocks on the sendSync call until it is acknowledged by the recipient. The kind of acknowledgement depends on the message exchange pattern (MEP) you are using. For example, if you are sending an InOnly MEP, then all you get back is a indication of Done: you are notified that the recipient received your message and that it actively marked the message exchange as completed. If you are sending an InOut MEP, then you get back a message exchange containing a response from the recipient, and it is you who must then mark the message exchange as completed by sending a Done back to the recipient. You have guaranteed delivery, or at least you know if it didn't work before proceeding, and (more subtly) order is guaranteed to be preserved among successive sendSyncs that you perform to the same recipient.

Synchronous

This sounds simple, but in a system of any complexity at all, it may not be pretty.

The most obvious design issue is that the original sender (the service consumer in JBI parlance) is blocked waiting for the recipient (the service provider) to process the message exchange. On a loaded system, the recipient may be busy processing a long queue of pending requests from many senders. This causes the sender to block for potentially a long period of time. Since the sender may itself be acting as a service provider to other service consumers (that is, they are trying to send it requests and get a response), pending requests can back up in many components throughout the system. Wackiness may ensue. Even if the system isn't busy, handling some requests may require long latency operations like persistence handling, database queries, or remote network accesses.

The other obvious design issue is that if there are circumstances in which the recipient may itself act as a service consumer to the original sender's service provider, that is, the recipient may as part of its processing make a request of the original sender, and both components use sendSync, the system deadlocks. The original sender is waiting on its sendSync as a consumer to the recipient as a provider, and the recipient as a consumer is waiting on yet another sendSync to the original sender as a provider. Neither sendSync will ever complete, or they will timeout if they were coded with a timeout parameter.

This is not a new issue. Developers (like me) old enough to be taking their meals through a straw will recognize a similar issue in the remote procedure call (RPC) paradigm that was fashionable in the 1990s. In RPC, distributed components communicated with one another through function calls that were mapped by frameworks like CORBA, OSF DCE, or SunRPC (I've used 'em all) into network messages. Developers (like me) old enough to have one foot in the grave will remember dealing with this issue when writing bidirectional messaging frameworks using Berkeley sockets and TCP in the 1980s. I dimly recall similar issues arising when writing communications layers for PDP-11s using RS232 serial connections in the 1970s.

Geoff Towell, a colleague of mine in the JBI adventure, remarked that in his experience "systems using synchronous message passing aren't scalable." He also noted that "systems using asynchronous message passing can be difficult to write." In my experience, he was correct on both counts. The fix is the same whether you are using JBI, RPCs, sockets, or serial ports.

To insure guaranteed delivery and preserve order, you use synchronous message passing: sendSync, an RPC call with a returned value, a TCP socket, or a serial protocol that requires a reply. But the response you get back merely indicates that the recipient received your request and queued it for later processing. It says nothing about when the recipient will actually get around to processing your request. When it does, the relative roles of the components will reverse: the original recipient will act as a consumer and perform a sendSync with a new message exchange to the original sender, who is now acting as a provider and will appropriately complete the new message exchange. Hence, message passing is synchronous, but message processing is asynchronous.

Asynchronous

We did the same thing with RPCs: the successful return of the remote procedure call merely meant that the called component received the parameters, not that it actually did anything with them. When the called component completed processing the request, it would invoke a callback function in the original calling component, doing another RPC in the opposite direction to deliver the response.

This is why the design of the sender and recipient gets ugly: they may both have to handle multiple requests concurrently. They typically do this by implementing multiple concurrent state machines, each machine implemented as an object. For each request, whether originated by the sender or received by the recipient, a new state machine object is created. Many of these objects may exist simultaneously in both the sender and the recipient as many requests are asynchronously processed. Each state machine is maintained in the recipient until a response for the request that particular state machine represents can be sent, and the recipient's state machine transitions to its final state. The original sender maintains its own state machine for each request until the corresponding response is received and processed, then that state machine also transitions to its final state.

(If you are into automata theory or formal languages, all of this will sound very familiar. The message exchange between the two components describes a protocol. Finite state machines and other automata are typically used to implement parsers for formal languages. Formal languages are described by formal grammars. The fact that you frequently use a state machine to implement a protocol is why protocols are often described in terms of a formal grammar. Such grammars are remarkably useful and should be in every developer's toolbox. But that is a topic for another article.)

There are a number of ways you might implement multiple concurrent state machines. The simplest is to have a separate thread for each request in both the sender and the recipient. This works well in systems in which the cost of context switching and maintaining a thread is zero. The fact that there are no such systems means this approach is seldom used.

(It can in fact get bad a lot faster than you might expect, since on some systems I have seen the cost of context switching increase proportional to the square of the number of active threads. I wrote a marvelous white paper on this topic, that this blog article is too small to contain.)

You may have a fixed size pool of threads in the recipient that service a common queue of incoming requests. I have seen this work well in both Java and C++ implementations in which the number of possible concurrent outstanding requests is small, the lifespan of each request is short, and concurrent pending requests are mostly independent. There are Java and C++ frameworks that provide this capability.

When I've run up against systems that have to handle 38,000 concurrent long-duration requests (and no, I didn't pick that number out of thin air), neither approach scales, and I resort to designing and coding an application-specific concurrent state machine implementation that runs inside a small number (like one) of threads. This is not as hard as it sounds.

(Dan Kegel wrote a really great article on this scalability issue in the context of server-side socket applications in systems of UNIX-like sensibilities; see The C10K Problem.)

My web services friends will no doubt be up in arms over this article, either because I'm suggesting using synchronous message passing, or because I'm suggesting using asynchronous message processing. Probably both. But my background for the past thirty years has been in building robust, scalable server-side real-time systems, and what I have described here is a design pattern I have found to work.

Update (2008-07-07)

I've recently been reading about continuations, which are a mechanism to, very roughly speaking, pick up a prior computation where it left off. It's funny: being the wizened old man that I am, I always thought of continuations as a form of checkpointing, from my time in the deep past with IBM mainframes and Cray supercomputers. It wasn't until recently that I realized that continuations serve essentially the same purpose for web servers as the state machine architecture I describe here and have implemented on several real-time systems over the years. For that matter, checkpoints served a similar purpose on mainframes and supercomputers.

I suspect that the motivation for all of these mechanisms was slightly different. Checkpoints existed because hardware was slow and expensive, it wasn't uncommon for the system to crash, and when it did you wanted to pick up work were it left off. The state machine architecture I describe here was mostly done for scalability, handling tens of thousands of simultaneous connections with just a few threads. Continuations seem to be mostly motivated not just by robustness and scalability, but the state-less nature of RESTful HTTP operations.

Maybe as I read more it will become more clear to me that continuations are completely different. In the spirit of The Sapir-Whorf Hypothesis, I should probably learn a programming language or framework that supports continuations natively.

Tuesday, August 14, 2007

Tool Economics

"If all you have is a hammer, everything looks like a nail." - Benard Baruch

I was teaching a class in embedded development at a client's site the other week. We were discussing approaches to fixing an actual bug in their firmware code base. This client uses the IAR Embedded Workbench, an IDE that provides all the usual tools: an editor, a build environment supporting the ARM-based tool chain, a JTAG interface, a down loader, a debugger, etc. When I've done product development for this client, I've used this tool, and while its debugger is the best thing since sliced bread, I found its editor to be a little weak. I preferred instead to do my editing using Eclipse with the CDT and Subversion plug-ins. The IAR EWB and Eclipse played well together, in my experience, allowing me to leverage the best of both worlds.

We discussed for a few minutes the merits of using the debugger to test our hypothesis of what the bug was, where to put in breakpoints, what variables we needed to look at. "Or," someone said, "we could just change the code and see what happens."

I had one of those brief moments of cognitive dissonance where I wondered "Who the heck said that?" and then realized "Oh, it was me!" We were all victims of tool economics.

Back in the 1970s, when I got paid to write code in FORTRAN, COBOL and assembly language, we didn't so much have a tool chain as a tool string or maybe a tool thread. We typed our code onto punch cards using a keypunch, which we frequently had to stand in line for. We submitted them to the operator at the I/O window of the computer room. Our stack of punch cards were queued up with a lot of other similar stacks by the card reader. The operator fed them in, where our job was queued on disk by the mainframe operating system. (This was OS/MFT running HASP on a IBM 360/65, for those that remember when glaciers covered the land and we communicated mostly with grunts.) Eventually the OS would run our jobs and, if all went well, our printout would come banging out on the line printer. The operator would separate our printouts and place them with our original card deck back in the I/O window. On a really good day, you might get two or three runs in, providing you got to the computer center early and left late.

The latency involved in a single iteration of the edit-compile-test-debug cycle bred a very strong offline approach to code development: you looked at your code until your eyes bled, trying as hard as you could to fix as many bugs as possible, running what-if simulations in your head because that was a lot faster that actually running your program. You saved old punch cards, just in case you needed to put those lines back in (maybe even in a different location), and your printouts were heavily annotated with quill pen marks (okay, I exaggerate, they were felt-tip pens). Changes were meticulously planned in advance and discussed with as many other developers as you could lay your hands on. You think you multi-task now? The only way to be productive in those days was to work on many completely different software applications simultaneously, so you could pipeline all of those edit-compile-test-debug cycles. Every new printout arriving at the I/O window was an opportunity to context switch in an effort to remember what the heck you were thinking several hours ago.

Sitting here, writing this while running Eclipse on my laptop and accessing my code base wirelessly from a remote Subversion server, I have zero desire to return to those bad old days. But it does illustrate how our tool chain affects how we approach a problem. Because the developers at my client were focused on using the EWB, their problem solving approach necessarily revolved around debugging, because the debugger was the EWB's greatest strength. My problem solving approach revolved around editing and refactoring with Eclipse, trying an experiment to see what happened, then maybe using Subversion to back my changes out. I resorted to using the debugger only when I had a unit test really go sideways and trap to the RTOS, or when I just couldn't see what was going on without single stepping through the code and watching variables change. Using Eclipse as part of the tool chain made it economical to try new things and verify proposed solutions quickly.

Tool economics affects all phases of software development. It limits us in the solutions we are willing to consider, and forces us to reject some potential solutions out of hand. It colors the processes we use, and the quality we may be able to achieve within the constraints of schedule and budget.

Some of the developers at my client preferred to use the debugger to unit test their code. It was easy to love the debugger in this role, but it was a very labor intensive process. This was okay economically as long as you only had to do this once, when you originally wrote the code.

But no successful code base is static. At least two-thirds of the cost of software development is code maintenance, changing the code base after its initial release. Refactoring, something tools like Eclipse vastly simplify, becomes a way of life. Going through the debugger every time you change a particular function is expensive. I chose instead to write unit tests embedded in the production code base, typically conditionally compiled in so that the test code was present in the development environment but not shipped as part of the product. If I changed a particular function, I just reran the unit test. If the test failed and I was absolutely clueless as to why, then I turned to the debugger. I paid the price of writing the unit test once, and then used it for free henceforth. For sure there are occasionally some Heisenbergian issues with the unit test affecting the functionality of the code base, typically due to memory utilization or real-time, but those were rare.

Embedding unit tests in the code base is just a more economical approach to software development from a developer utilization viewpoint. But the real reason I tend to use that approach is that it's a rare embedded development project I work on in which I have access to a debugger as good as the one provided by the IAR EWB. I'm generally forced into more formal unit testing approach because I really don't have a choice. The fact that it's really cheaper in the long run is just gravy. This is yet another way in which tool availability forces a particular approach.

Tool economics effects how we design our applications too. I remember a job in which I found myself working in an eight million line code base whose oldest elements dated back at least to the early 1980s. Unit testing was extremely expensive. There were no formal unit tests. You tested your change by reserving a lab machine for sometime in the future, loading the application, and spending anywhere from five minutes to hours testing on real hardware and real devices, sometimes in concert with logic and protocol analyzers. I admit that many times I chose a particular design, or even an architectural direction, because it minimized the number of subsystems I would have to touch, and hence re-test. Choosing the "best" solution had nothing to do with it. Choosing a solution that met requirements while at the same time could be completed within the limitations of the schedule had everything to do with it.

You think I was wrong? The managers to which I reported would disagree with you. Admittedly this was sometimes a case of time to market (TTM) taking precedence over cost of goods sold (COGS) (see The Irresistible Force and the Unmovable Object), but that was the trade-off sanctioned by that particular development organization.

The expense of testing also effected the quality of the code base in perhaps unexpected ways. Many times I recall discussing with my fellow developers over beer or lattes that, in the adventure game that was navigating this huge code base, in the course of our work we occasionally encountered code that we knew deep in our gut simply could not work. It could never have worked. What to do? We could have fixed it, but then we would have had to test it, even if it were a trivial change, placing us behind schedule on our actual assignments, and making it more likely we would be in the next wave of layoffs. We could have written and filed a bug report (and too our credit, we typically did just that), but bug reports have a habit of ending up back on the plate of the original submitter. So when we encountered such code, and if time was short, we sometimes just backed away slowly, and pretended never to have seen it. The high cost of testing drove that dysfunctional behavior.

If you are a developer, you deserve the best tools you organization can afford. No matter whether you are in the open source world, the Microsoft camp, or the embedded domain, there are very good tools to be had. The availability of good tools, and your familiarity with them, will affect not only how you approach your job, but the quality of the product that you produce.

If you are a manager, you need to make it a priority to get your developers the best tools you can afford. Your schedule and the quality of the resulting product depends on it. Just meeting requirements is not sufficient, because a requirement must be testable. Having a requirement that says "this software must be maintainable in the face of even unanticipated changes over its entire lifespan" is not testable. Whether you appreciate it or not, your developers are making economic decisions minute by minute that affect both the TTM and the COGS of your product. And, hence, the financials of your company.

Tuesday, July 31, 2007

The Irresistible Force and the Unmovable Object

Decades ago I remember seeing a print cartoon in which a distinguished scientist of Einsteinian sensabilities was deriving a hugely complex equation on a blackboard. From top to bottom of the blackboard the equation got simpler and simpler until it was finally reduced to Time = Money.

Einstein discovered that the two most basic things in the Universe, matter and energy, were equivalent. But in all human endeavors, it doesn't take much experience to develop the gut feeling that time and money are the only things that really matter, and that they are in some way also equivalent.

It is no different in the realm of software development, where time is the prime force and money is the fundamental particle from which all other things, good and bad, flow. Most managers realize this, although they (and I) prefer to think in terms of not time, but time to market (TTM), and not money, but cost of goods sold (COGS). Software developers should think in these terms too.

TTM is the time it takes a product to go from inception to its first (typically beta) customer. All other things being equal (hint: they never are), shorter is better. COGS is the sum of all the costs that can be directly attributed to the production of a product, including both the materials and the labor that go into its manufacture. Again, all other things being equal, smaller is better.

There are lots of good reasons why making TTM as short as possible is important. Having a shorter TTM than your competitors lets you introduce products leveraging new technology before they do. This may give you an advantage in market share. It may let you establish de facto standards in the market which can force your competitors into playing catchup. None of the rationales for shortening TTM are a sure thing, but they are typical.

But there is a more subtle reason for a short TTM. In Lies, Damn Lies, and Net Present Value, I talked about how when you are seeking funding for the development of a new product from your funding agency, whether that agency is upper management, venture capitalists, or the Department of Defense, you are competing with all the other uses to which that funding may be applied. Unfortunately, you are not just competing with other proposed product developments, but with compound interest.

If your funding source merely places your proposed funding in some investment vehicle for the same duration as your proposed TTM, their money grows much faster than linearly. The longer it stays invested, the faster it grows. And all the really big growth in terms of absolute value is biased towards the end of the investment period. What this means is that extending your development period (and hence, your TTM) by even a little bit can have a huge impact on the potential value of your product in the marketplace when compared to just having put the same money in the stock market. This is referred to as the cost of money (COM).

Doubling your development time doesn't double the cost of your project, at least not in the eyes of your investors. It's more like an order of magnitude because of the COM. TTM is an strong force in all product development. But because of the uncertainties that frequently occur when developing bleeding edge products using new and perhaps unproven technologies, it is an irresistible force in that arena.

If TTM is the irresistible force, COGS is the unmovable object. If your product does not cost less to produce than you can make from selling it, then you are obviously not going to make any money selling that product. You've heard the old joke: "How do we do it? Volume." COGS is what this joke is talking about.

COGS is not nearly that simple. A strategy used by many manufacturers is to sell a product as a loss leader, that is, for less than its cost to manufacture, in order to acquire market share, with the intent to raise the price after a bunch of customers are locked in. If you are selling a commodity, that is, a product which can be easily replaced with an equivalent competing product, this obviously won't work, because there is no lock in. But in high-tech, it is not unusual for products to be sold at a steep discount, or, as in the open source world, even given away for free, where money can be made through collateral products, such as service contracts.

(We may pause here to note that in some markets or some countries, there are actually laws against selling products for less than their cost to manufacture. So your mileage may vary.)

COGS has its own special subtlety when it comes to software development. Many managers (and developers) see COGS are purely the up-front cost of developing a software product. The cost of physically manufacturing a software product once it has been developed is nearly zero. It is the cost of replicating and shipping CDROMs, or of running a web server from which the software is downloaded.

This leads TTM and COGS to interact is a manner that may seem fairly obvious. Adding more developers to a project would appear to increase its COGS in more or less in a linear fashion, but reducing TTM can decrease its COGS in a much greater than linear fashion, thanks to the COM. So adding developers to a project seems like a good deal.

Most developers instinctively know that there is a flaw in that argument, just as we all know that three women cannot produce a baby in three months. In the classic software engineering text The Mythical Man Month, author Fred Brooks notes that adding people to a late project makes it later. There are a couple of reasons for this.

First, adding people to a project does not achieve a linear improvement in your schedule. Each added person will cost some time from the existing staff as they have to answer questions and train the new person. Most companies I've worked for have some rule of thumb for the net cost of bringing a new person up to speed. I've seen values ranging from two months to two years used, the latter being for working in an eight-million line legacy code base. The real cost depends greatly on both the developer and the project. My experience is that developers which have a lot of experience recognizing common design and architectural patterns come up to speed a lot faster since they tend to effectively and automatically abstract away a lot of the unnecessary details, while inexperienced developers worry about every gory little thing.

Second, there is a communications overhead as all members of the development group have to coordinate with each other. Software development is, contrary to its mythology, an extremely collaborative activity. This communications cost increases non-linearly. In the worst case, where all developers have to coordinate with one another, as a rule of thumb the interpersonal communications overhead is proportional to the square of the number of the developers.

So merely adding developers to a project to decrease TTM isn't generally the answer.

There is another, largely ignored, aspect to COGS. In The Total Cost of Code Ownership, I talked about how at least 67% of the total life-cycle cost of a successful software product is spent on maintenance. Maintenance is changing existing code: to fix bugs, to add new features or otherwise improve the product, or to adopt to changes outside of the software product itself. The original development of the product accounts for at most a third of its total life-cycle cost. (Other studies place the maintenance portion of the budget at 70% or even 90%. I suspect this depends on the longevity of the software product.)

This means if you somehow magically reduce TTM to zero, you have at best only reduced your total cost of code ownership, which for a software product is effectively your COGS, by a third.

Why is this fact unappreciated? Lots of reasons, mostly tied to (big surprise) money. Market or product managers get compensated for bringing a successful new product to market. Developers are compensated for developing exciting new products. Neither are given raises, promotions, or bonuses for keeping a legacy product working. Maintenance of legacy products is frequently an activity done in the margins. And even if it isn't, that activity is often done by a group separate from the original developers, with a name like sustaining development or current engineering, whose funding comes from a different business unit, silo, or cost center.

This is all a little ironic since for many companies, it is the successful legacy product that is generating most of the revenue that funds the development of new products. I recall once being forbidden by a department head from consulting on a complex legacy product for which I was one of the original developers, despite the fact that the issue with the product was impacting high profile customers. Not his silo, hence not his problem.

It gets uglier. Every developer has worked on projects in which in an effort to reduce TTM, they were encouraged to cut corners. I remember a systems engineer once saying in a meeting "I don't think we have time in the schedule for doing unit tests." (To our credit, the development team ignored him.) But cutting corners leads to what embedded pundit Jack Ganssle refers to as technical debt: it's a debt you take on to reduce TTM, but one which you must eventually pay back. Why? Because technical debt increases the cost of maintaining your product, and hence increases your COGS. Put enough quick and dirty hacks in your software product, and eventually your COGS rises enough that you can no longer sell your product at a profit.

The fact that you may not be keeping a good accounting of the COGS of your product, because its maintenance was moved into a different silo, may lead this issue to bite you unexpectedly. Moving a cost into an area into which it cannot be accounted for does not mean the cost goes away. (I have seen this principle at work when companies do away with desktop PC support, placing the burden of keeping the infrastructure running on the shoulders of their developers. The cost didn't go away. It just got moved into an area from which it could not be accounted for.)

Are there ways to reduce both TTM and COGS? That would be the holy grail. Anything which allows you to generate maintainable code faster would do so. Using languages and design patterns that admit easy refactoring, and using IDEs and other tools that support the same, are a step in the right direction. Software reuse has been seen largely as a failed policy, mostly I think for political reasons. Yet, the C Standard Library is probably the most reused piece of software in the world. As I like to say: "Nobody rewrites printf." The wealth of frameworks and libraries available for Java is a hopeful sign too, but there are so many of them, and they are so complex, that they can bring their own TTM and COGS problems to the table. Witness Fun Facts to Known and Tell: Java Business Integration.

Still, anything that allows you to work at a higher level of abstraction has the potential to reduce both TTM and COGS. Higher-level languages: moving from assembler to C to to C++ to Java to domain specific languages like Ruby. Objected-oriented design: moving from C to C++ or Java, or at least using OO designs in C code. Operating systems with more capabilities: moving from your favorite tiny RTOS to Linux. Reusable libraries or frameworks: C standard library, C++ Standard Template Library (STL), any of the many invaluable frameworks in Java. Using tools that automate or at least simplify the development process: Eclipse for C, C++ or Java, or any of the many great Java tools like Cobertura and JUnit.

When I plan, architect, design, and implement software products, or when I am evaluating new tools, lowering both TTM and COGS is always on the forefront of my mind. In If Java is the new COBOL, is C++ the new assembly? I argued on both a TTM and COGS basis for broader use of both Java and C++ in embedded applications.

In my three decade plus experience, I have seen a lot of effort is spent on reducing TTM. But when it comes to reducing COGS, only the hardware guys get any of the glory. Software COGS is largely ignored. To effectively reduce software COGS, you have to make software easy to modify and maintain.