The Irresistible Force and the Unmovable Object

Decades ago I remember seeing a print cartoon in which a distinguished scientist of Einsteinian sensabilities was deriving a hugely complex equation on a blackboard. From top to bottom of the blackboard the equation got simpler and simpler until it was finally reduced to Time = Money.

Einstein discovered that the two most basic things in the Universe, matter and energy, were equivalent. But in all human endeavors, it doesn't take much experience to develop the gut feeling that time and money are the only things that really matter, and that they are in some way also equivalent.

It is no different in the realm of software development, where time is the prime force and money is the fundamental particle from which all other things, good and bad, flow. Most managers realize this, although they (and I) prefer to think in terms of not time, but time to market (TTM), and not money, but cost of goods sold (COGS). Software developers should think in these terms too.

TTM is the time it takes a product to go from inception to its first (typically beta) customer. All other things being equal (hint: they never are), shorter is better. COGS is the sum of all the costs that can be directly attributed to the production of a product, including both the materials and the labor that go into its manufacture. Again, all other things being equal, smaller is better.

There are lots of good reasons why making TTM as short as possible is important. Having a shorter TTM than your competitors lets you introduce products leveraging new technology before they do. This may give you an advantage in market share. It may let you establish de facto standards in the market which can force your competitors into playing catchup. None of the rationales for shortening TTM are a sure thing, but they are typical.

But there is a more subtle reason for a short TTM. In Lies, Damn Lies, and Net Present Value, I talked about how when you are seeking funding for the development of a new product from your funding agency, whether that agency is upper management, venture capitalists, or the Department of Defense, you are competing with all the other uses to which that funding may be applied. Unfortunately, you are not just competing with other proposed product developments, but with compound interest.

If your funding source merely places your proposed funding in some investment vehicle for the same duration as your proposed TTM, their money grows much faster than linearly. The longer it stays invested, the faster it grows. And all the really big growth in terms of absolute value is biased towards the end of the investment period. What this means is that extending your development period (and hence, your TTM) by even a little bit can have a huge impact on the potential value of your product in the marketplace when compared to just having put the same money in the stock market. This is referred to as the cost of money (COM).

Doubling your development time doesn't double the cost of your project, at least not in the eyes of your investors. It's more like an order of magnitude because of the COM. TTM is an strong force in all product development. But because of the uncertainties that frequently occur when developing bleeding edge products using new and perhaps unproven technologies, it is an irresistible force in that arena.

If TTM is the irresistible force, COGS is the unmovable object. If your product does not cost less to produce than you can make from selling it, then you are obviously not going to make any money selling that product. You've heard the old joke: "How do we do it? Volume." COGS is what this joke is talking about.

COGS is not nearly that simple. A strategy used by many manufacturers is to sell a product as a loss leader, that is, for less than its cost to manufacture, in order to acquire market share, with the intent to raise the price after a bunch of customers are locked in. If you are selling a commodity, that is, a product which can be easily replaced with an equivalent competing product, this obviously won't work, because there is no lock in. But in high-tech, it is not unusual for products to be sold at a steep discount, or, as in the open source world, even given away for free, where money can be made through collateral products, such as service contracts.

(We may pause here to note that in some markets or some countries, there are actually laws against selling products for less than their cost to manufacture. So your mileage may vary.)

COGS has its own special subtlety when it comes to software development. Many managers (and developers) see COGS are purely the up-front cost of developing a software product. The cost of physically manufacturing a software product once it has been developed is nearly zero. It is the cost of replicating and shipping CDROMs, or of running a web server from which the software is downloaded.

This leads TTM and COGS to interact is a manner that may seem fairly obvious. Adding more developers to a project would appear to increase its COGS in more or less in a linear fashion, but reducing TTM can decrease its COGS in a much greater than linear fashion, thanks to the COM. So adding developers to a project seems like a good deal.

Most developers instinctively know that there is a flaw in that argument, just as we all know that three women cannot produce a baby in three months. In the classic software engineering text The Mythical Man Month, author Fred Brooks notes that adding people to a late project makes it later. There are a couple of reasons for this.

First, adding people to a project does not achieve a linear improvement in your schedule. Each added person will cost some time from the existing staff as they have to answer questions and train the new person. Most companies I've worked for have some rule of thumb for the net cost of bringing a new person up to speed. I've seen values ranging from two months to two years used, the latter being for working in an eight-million line legacy code base. The real cost depends greatly on both the developer and the project. My experience is that developers which have a lot of experience recognizing common design and architectural patterns come up to speed a lot faster since they tend to effectively and automatically abstract away a lot of the unnecessary details, while inexperienced developers worry about every gory little thing.

Second, there is a communications overhead as all members of the development group have to coordinate with each other. Software development is, contrary to its mythology, an extremely collaborative activity. This communications cost increases non-linearly. In the worst case, where all developers have to coordinate with one another, as a rule of thumb the interpersonal communications overhead is proportional to the square of the number of the developers.

So merely adding developers to a project to decrease TTM isn't generally the answer.

There is another, largely ignored, aspect to COGS. In The Total Cost of Code Ownership, I talked about how at least 67% of the total life-cycle cost of a successful software product is spent on maintenance. Maintenance is changing existing code: to fix bugs, to add new features or otherwise improve the product, or to adopt to changes outside of the software product itself. The original development of the product accounts for at most a third of its total life-cycle cost. (Other studies place the maintenance portion of the budget at 70% or even 90%. I suspect this depends on the longevity of the software product.)

This means if you somehow magically reduce TTM to zero, you have at best only reduced your total cost of code ownership, which for a software product is effectively your COGS, by a third.

Why is this fact unappreciated? Lots of reasons, mostly tied to (big surprise) money. Market or product managers get compensated for bringing a successful new product to market. Developers are compensated for developing exciting new products. Neither are given raises, promotions, or bonuses for keeping a legacy product working. Maintenance of legacy products is frequently an activity done in the margins. And even if it isn't, that activity is often done by a group separate from the original developers, with a name like sustaining development or current engineering, whose funding comes from a different business unit, silo, or cost center.

This is all a little ironic since for many companies, it is the successful legacy product that is generating most of the revenue that funds the development of new products. I recall once being forbidden by a department head from consulting on a complex legacy product for which I was one of the original developers, despite the fact that the issue with the product was impacting high profile customers. Not his silo, hence not his problem.

It gets uglier. Every developer has worked on projects in which in an effort to reduce TTM, they were encouraged to cut corners. I remember a systems engineer once saying in a meeting "I don't think we have time in the schedule for doing unit tests." (To our credit, the development team ignored him.) But cutting corners leads to what embedded pundit Jack Ganssle refers to as technical debt: it's a debt you take on to reduce TTM, but one which you must eventually pay back. Why? Because technical debt increases the cost of maintaining your product, and hence increases your COGS. Put enough quick and dirty hacks in your software product, and eventually your COGS rises enough that you can no longer sell your product at a profit.

The fact that you may not be keeping a good accounting of the COGS of your product, because its maintenance was moved into a different silo, may lead this issue to bite you unexpectedly. Moving a cost into an area into which it cannot be accounted for does not mean the cost goes away. (I have seen this principle at work when companies do away with desktop PC support, placing the burden of keeping the infrastructure running on the shoulders of their developers. The cost didn't go away. It just got moved into an area from which it could not be accounted for.)

Are there ways to reduce both TTM and COGS? That would be the holy grail. Anything which allows you to generate maintainable code faster would do so. Using languages and design patterns that admit easy refactoring, and using IDEs and other tools that support the same, are a step in the right direction. Software reuse has been seen largely as a failed policy, mostly I think for political reasons. Yet, the C Standard Library is probably the most reused piece of software in the world. As I like to say: "Nobody rewrites printf." The wealth of frameworks and libraries available for Java is a hopeful sign too, but there are so many of them, and they are so complex, that they can bring their own TTM and COGS problems to the table. Witness Fun Facts to Known and Tell: Java Business Integration.

Still, anything that allows you to work at a higher level of abstraction has the potential to reduce both TTM and COGS. Higher-level languages: moving from assembler to C to to C++ to Java to domain specific languages like Ruby. Objected-oriented design: moving from C to C++ or Java, or at least using OO designs in C code. Operating systems with more capabilities: moving from your favorite tiny RTOS to Linux. Reusable libraries or frameworks: C standard library, C++ Standard Template Library (STL), any of the many invaluable frameworks in Java. Using tools that automate or at least simplify the development process: Eclipse for C, C++ or Java, or any of the many great Java tools like Cobertura and JUnit.

When I plan, architect, design, and implement software products, or when I am evaluating new tools, lowering both TTM and COGS is always on the forefront of my mind. In If Java is the new COBOL, is C++ the new assembly? I argued on both a TTM and COGS basis for broader use of both Java and C++ in embedded applications.

In my three decade plus experience, I have seen a lot of effort is spent on reducing TTM. But when it comes to reducing COGS, only the hardware guys get any of the glory. Software COGS is largely ignored. To effectively reduce software COGS, you have to make software easy to modify and maintain.

Asterisk, WiFi, and a Nokia N800 Internet Tablet

In A Future Without Keyboards I mentioned that I had just started to toy around with a Nokia N800 Internet Tablet. This is a pocket-sized Linux system that is less a product than it is an open source platform, which means it's a developer's fantasy.

The device includes all sorts of cool hardware, including a high resolution color touch-sensitive LCD display, WiFi and BlueTooth connectivity, USB and microphone/stereo headphone jacks, a built-in microphone and stereo speakers, an aimable CCD camera, an FM tuner, and two SD-compatible flash card slots. The base software includes a very usable web browser, an email client, a camera, an FM radio, and an Internet radio. I recently added a BlueTooth keyboard. Other accessories I may try in the future include a navigation kit that gives the unit GPS capabilities.



In Asterisk, WiFi, HomePlug, and an Avaya SIP Phone, I talked about the Asterisk PBX in the secure underground Digital Aggregates Corporation computer center (it's a Dell PC in the basement). I've successfully used a number of SIP endpoints with the PBX, including an Avaya 4610SW SIP hardphone and several PC-based SIP softphones. I've wanted a mobile WiFi SIP phone to use around the house and office but I wasn't terribly impressed with either the price or the quality of any of commercial options.

I recently installed the third-party (and quite experimental) Internet Communications Software Update for the N800 from Maemo which includes a SIP stack. It extends the existing internet capabilities of the N800 to include a SIP softphone. So far, initial testing has achieved an occasional two-way talk path. Debugging continues, and it's entirely possible that this is pilot error on my part. Still, I am much encouraged by the prospect of a pocket sized universal internet appliance, something I've sought ever since I used my first Palm PDA years ago.

In If Java is the new Cobol, is C++ the new assembly? I talked about how I thought Java was really underutilized in the U.S. as compared to Europe and Japan in using it as a platform for embedded applications. I made an economic argument based on cost of goods sold, not because Java is the hip thing to be using. (I'm told by those in the know that Ruby On Rails is the hip thing to be using.)

I'm using this little project as an excuse to play with Java Micro Edition (ME) in the Connected Device Configuration (CDC) by installing the C Virtual Machine (CVM) package for the N800 from the open source PhoneMe project. About all I've done so far is get the provided test suite to run successfully, and learn that you can't run applications on the N800 from the SD flash card. Still, it's a step in the right direction.

Hand-held networked universal appliances have been a staple in science fiction. I think they're near to becoming a commercial reality today. Mobile phones are close. Devices like the N800 are closer.

All Things Are Temporary

As if I needed any more evidence that life is too short, I was provided with some on Sunday, July 22nd, that I would have much preferred to do without. My old friend, occasional colleague, and frequent shooting buddy Paul Hyder died that morning after suffering the second of two strokes in the span of a little over a week. I had just sent him a get-well card, and was idly wondering when I could get him back out on the shooting range, and what help I might be able to give him, when his wife Marti notified many of us of his passing. Paul was barely 56.

Paul and I had worked together as engineers as the National Center for Atmospheric Research in Boulder Colorado over ten years ago and discovered that we had many shared interests: technology, science fiction, firearms, and a warped sense of humor. Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I were happy to be frequent recipients of he and his wife's gracious hospitality at their mountain home in Evergreen, where we enjoyed the company of their eclectic group of friends. We liked going out to dinner with Paul and Marti, and I for one always took advantage of his interest in gourmet food and wine. Paul was my go-to guy for tips on great new restaurants.

Paul and I were both members of a private outdoor shooting range in the foothills, where several times a year we would meet for a morning of shooting or to take a tactical class together. We did all the usual guy things: blowed stuff up real good, did tactical drills together, discussed who had the cooler firearm (I did), who was the better shot (he was), and just generally enjoyed each other's company while being outdoors in the great state of Colorado. Our interest in this past-time was such that I vividly remember us going one day in the dead of winter when it was so cold, we could only shoot a few rounds before retiring to his truck to defrost. Common sense finally prevailed and we beat a tactical retreat to the nearest good restaurant.

It will take some time for me to get used to the idea of life without Paul Hyder: his quirky personality, his mountain-man appearance, his sense of humor, his deep technical knowledge of networking, and his Glock 10mm. When my mom died in 2006, I am a little ashamed to admit that in the weeks and months afterwards, I found myself thinking about her more than I did when she was alive. When my friend and colleague Mark Uris died just a few months ago (also at age 56), I found it to be true again. I'm sure it will be the same for Paul.

We should never take our family and friends for granted. All things are temporary.