Saturday, December 08, 2007

Can't win. Can't break even. Can't quit the game.

From BusinessWeek, December 17, 2007, p. 68, "A Battery That Can Power A Whole Town":

"Hyperion Power Generation (HPG) is developing a nuclear battery capable of powering a town. The size of a hot tub, it can put out more than 25 megawatts for five years, enough to run 25,000 homes."

Here's that non-free lunch you ordered.

Sunday, November 25, 2007

The End of Civilization as You Know It

It has come to my attention that there are those of you out there that do not believe that Western Civilization will eventually collapse. How you justify this hubris is a mystery to me.

Henry Kissinger once said "Every civilization that has ever existed has ultimately collapsed." This seems so obvious that I'm always surprised that others found Dr. Kissinger's quote to be controversial. But for me, to assume otherwise would be to assume that anything wrought by man could be immortal. This is contrary to both physics and history. Thanks to the laws of thermodynamics, nothing is immortal.

Even corporations have a natural life span. Sure, General Electric has been around for as long as anyone can remember. But I bet at one time everyone thought the Dutch East India Company would be a permanent fixture, too (and would always pay an 18% dividend). To its credit, it lasted nearly two hundred years, before finally going bankrupt.

If nothing else, you know that in a few billion years or so our sun will run out of fuel. That's just part of the cosmological circle of life. Unless Western Civilization has escaped the grip of the Earth and our Solar System by then, it (and every other civilization on it at the time) is toast. It's not a matter of if, but when.

So it seems a safe bet that Western Civilization will eventually collapse, like every other civilization before it, permanently and irrevocably. It might collapse sooner than that, depending on various other catastrophic scenarios that are fun to ponder, like major asteroid or comet impacts, pandemics, or the super volcano under Yellowstone National Park erupting. On a purely statistical basis, it seems likely that one of those will happen (maybe more than once) long before our sun goes nova.

I think what people are actually saying, though, when they disagree with me, is that Western Civilization will not permanently collapse in their lifetime. This, I might agree with. But I would counter with that it seems almost a sure thing that your little corner of Western Civilization can collapse temporarily, given the right circumstances. This is the lesson of Hurricane Katrina. It was a hard lesson to all involved, but entropy means it is a lot easier to move from order to disorder (like, say, in Baghdad) than from disorder to order (as in New Orleans).

I used to spend my summers as a kid living in a farm house without running water, telephone, or the Internet, only a fireplace for heat, one iffy channel of broadcast television on a good day, and electricity most of the time. (I read a lot, drew water out of a well, and routinely walked around with a firearm.) As a professional, I've stayed in hotels many time zones away, where the tap water was not potable, and where I endured regularly scheduled rolling blackouts that put an end temporarily to elevators and air conditioning. (I also ate a lot of food which I had no idea what it really was, but I sort of liked that part.) These experiences really drove home the fact that the conveniences of modern day life are not evenly distributed, even in these United States.

I'm a fan of author Steven Pressfield. He's probably best known for his golf book The Legend of Bagger Vance because Robert Redford made it into a movie starring Will Smith, Matt Damon, and Charlize Theron. (Yes, I know it's not really about golf.) But if you've read more than one book by Pressfield, you probably already know that he is better known among his fans as an author of well researched historical fiction taking place in ancient Greece.

His book Last of the Amazons is about the siege of the city of Athens by the Amazons and their allies circa fifth century B.C. Greek scholars are still uncertain whether the Amazons, a tribe dominated by fierce woman warriors centered around what is now Eastern Europe, were real or myth. But if they were myth, the ancient Greeks devoted a great deal of time and effort to making bas-reliefs depicting the mythical war between them.

In Pressfield's book, we see Athens right at the early stages of its experimentation with democracy. The fictional Greeks in his book were aware of how fragile this beginning of Western Civilization was, and how easily it could all come tumbling down. Losing the war meant more than just death or enslavement. It meant an end to democracy, the end of the city state, and a return to being a nomadic tribe of hunter-gatherers living in animal skin huts.

I thought about this a lot while reading Pressfield's book.

When I say that Western Civilization is a house of cards, what I'm saying, in part, is that we shouldn't take civilization, Western or otherwise, for granted. It is fragile and all too easily lost. And once lost, difficult to regain.

Sunday, November 18, 2007

Peak Oil and Finding Good Help

I had lunch the other day with Ken, my local Ph.D. in theoretical physics who somehow ended up with a career in telecommunications. He remarked that ethanol fuel wasn't cost effective because without the government subsidies it cost more energy to make than it yielded. This reminded me of the fact that we don't have to actually run out of oil; we just have to run out of oil that takes less energy to extract than it yields. There may be lots of oil left in the ground, waiting for some more cost effective method of extraction.

I'm a fan of the blogs of several economists. One of them wrote the other day that the best way to break the ethanol myth was quit having early caucuses in Iowa. The candidates all go to Iowa and come away with the notion that corn is really important. If we had early caucuses in, I dunno, Colorado, maybe they'd think oil was important. Or toxic waste. Or snow.

Corn isn't even nutritionally that great. Mostly it's sugar and fiber. And we lack the crucial enzymes naturally to break down what amino acids it does have. Which is why you can starve to death eating corn unless it's treated with lye. I think a really interesting thriller would be to have the bulk of the U.S. population develop a food allergy to high fructose corn syrup, rendering all processed food inedible. Not as crazy as it sounds. Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) tells me that the development of food allergies is not well understood, happens suddenly, and is at least anecdotally linked with high exposure.

In some ways I find this thread connected with colleagues' complaints on their difficulty finding competent technical people. Welcome to our (near) future. I worked at a university for years, and we followed the "engineer production" curve closely. It's been cyclic for many many decades. A guy I used to work with when I was at a national lab always said that the big U.S. Department of Energy labs (with whom we worked closely) were a welfare program for physicists, because it was likely we'd need physicists for reasons of national security, and if we needed them, we couldn't wait a generation for the system to create them. I'm wondering if something similar is going to happen with the Information Economy. We're going to find out that we need to artificially stimulate the production of engineers because we can't tolerate the latency in the manufacturing process. This is a case where free markets don't really work. (Adam Smith didn't believe free markets totally worked either.)

Another friend of mine passed along an article in the WSJ Online where Dow Chemical got into trouble with its biggest customers by daring to suggest that maybe we should be saving oil to make important stuff like plastic, instead of burning it up. They had to backpedal when auto makers said "increase mileage -- are you crazy?!?"

To to link the threads even more, the friend who passed along that article was a technologist with multiple college degrees who left a high paying job with a large telecommunications equipment manufacturer to become a mail clerk in a civil service job for his city government. He was disgusted and just wanted to have an 8-5 job where he could turn his brain off. Another old friend of mine who is my age left the same company a few years ago to enter the police academy, a decision that while I have no desire to emulate, I never the less greatly admire. Now he's a sargeant in the police department, training other officers. These guys aren't reflected in the unemployment statistics, because they aren't unemployed; they just bailed out of the technology domain out of disgust.

I admit that seeing simularities between finding competant software engineers and peak oil may seem kind of twisted. But oil and software engineers both are resources that are not easily found or produced. In some dark part of my mind, I'm kind of looking forward to the collapse of Western Civilization. I'm pretty convinced it's all built on a house of cards, requiring cheap energy and lots of technologists to keep it running, both of which we're exhausting (in all senses). I am confident I won't survive the collapse, but what the hell, it'll at least be interesting.

One thing for sure -- upper management will find a way to blame the engineers for all of it.


There Ain't No Such Thing As A Free Lunch

Some friends and I have been talking about peak oil and the implications of actually finding a cheap, plentiful alternative energy source.

I'm not a global warming expert (although I have stayed at a Holiday Inn Express), but the role greenhouse gasses play is that they increase the retention of the ambient heat that we generate. Instead of letting it be emitted as infrared into space, they reflect it back or store it as heating and emit it back. The CO2 itself doesn't generate any heat itself, it just traps the heat we create (hence the name "greenhouse" gas). So global warming is an issue of ambient heat generation AND our inability to dissipate it because of greenhouse gasses.

The role that energy production and its use plays in heat generation is covered in the second law of thermodynamics which deals with entropy. All energy is, over time, broken down into unusable heat, because the entropy (disorder) of all physical systems increase over time. Meaning: every single bit of energy we generate and use eventually turns into heat. It may be a multi-step process, but as you drive your car down the road, your engine (electric or gasoline) generates heat; your tires generate heat from friction with the road; your car generates heat from friction with the air; the refinery that made the gasoline generated heat in its production. It all becomes heat. You can't get around this: it's the way the Flying Spaghetti Monster built the universe in his/her/its/their infinite wisdom. "Heat death" is pretty much the ultimate end of everything. When we're dead and buried, as we rot we generate heat, as our bodies release their stored chemical energy.

If we have some form of really cheap, really easily had energy, such that our energy production and use really increases, our heat generation rises by exactly that same amount. Greenhouse gasses in the atmosphere means that heat gets trapped instead of being radiated away as IR. But even if we didn't have any greenhouse gasses in the atmosphere, there is a physical limit to how much heat can be generated away per unit time. Basically, cheap available energy means we eventually cook ourselves to death. Literally.

Are there ways around this? It's been seriously suggested we build giant lasers that would somehow (this part is really unclear to me) radiate waste heat into space. I can imagine an SF story where we fry some passing alien craft and start an interplanetary war.

This has been proposed even for space craft with large power plants, since the only way to get rid of heat in space is by radiation; neither convection nor conduction works because there's no material to which to convect or conduct, like air or water. This is one of the reasons the Space Shuttle has big honking radiators built into the cargo bay doors and why they have to open them up when in orbit, even if they're not using the cargo bay. Their waste heat has no where else to go. This also places a physical limit to how much energy the Shuttle can expend, and how big a Shuttle we can build.

Quoting Robert Heinlein: TANSTAAFL. Cheap energy means increased heat production. In a way it just moves the problem somewhere else.

If you think about it, what is oil or coal? It's stored chemical energy. Where did the energy come from? From plants, and maybe some dinosaurs, that lived, ate, and grew millions of years ago. Because no system is 100% efficient, those plants (and dinosaurs) consumed a LOT more stored chemical energy, in the form of plants, dinosaurs, and soil and chemicals that once was plants and dinosaurs and rocks, than they created. Today we're taking advantage of a process that began millions of years ago, and that we cannot replicate. We can't make more oil because, besides the fact that we're impatient and can't wait millions of years (hence we invade small middle eastern dictatorships), the natural resources that went into making the oil, the incredibly rich biosphere, doesn't exist anymore. It went into making the oil that we DO have right now. Pretty much a one-way process.

Just like entropy and heat death.

Saturday, November 10, 2007


We all have guilty pleasures. I CAN HAS CHEEZBURGER is one of mine. Now the LOLinator lets you can enjoy any web site through an LOLcats filter, including this one.

Sunday, October 28, 2007

Do you know where your pension is?

"Expecting a Lump-Sum Pension? Read This" in the November 5, 2007 issue of BusinessWeek contains this little bit of news:

Starting next year, companies won't be able to make lump-sum distributions if their pension plans are more than 40% underfunded.

Congress is understandably concerned that the growing wave of baby boomers nearing retirement coupled with underfunded pension plans will put the plans underwater if many retirees take a lump-sum distribution (as most do). It would be like a run on the bank. Prohibiting lump-sum distributions from underfunded plans gives the company time to fix the pension fund.

This is probably only of interest to folks young enough to still be working, but old enough to have a pension plan and thinking of taking the money and running. Many high-tech companies either never had a pension plan, or quietly froze their plan some years ago.

Still, when I cashed out of a company that I had been with for a decade, the lump-sum distribution I got from their frozen pension plan was nothing to sneeze at. It is now earning interest in an Individual Retirement Account that I control.

If you're thinking of retiring, or just thinking of leaving and cashing out like I did, the article suggests ways you might be able to determine the health of your company's pension fund.

Wednesday, September 26, 2007

The Return on Investment in People

Starting in 1974, I spent about fifteen years at a university that was next to a major military base. To no one's surprise, many of my classmates, students, and co-workers were at that time currently or formerly in the military. While I was working at the campus computer center, I had three such co-workers who had two things in common: they were all former nuclear weapons technicians, and they were all going to college on the GI Bill.

This morning, National Public Radio's Morning Edition featured a story by John McChesney titled GI Bill's Impact Slipping in Recent Years. It's worth a listen. Some interesting facts from the story:

At the close of World War II, only 5% of Americans had college degrees.

Only 40% of the veterans of WW II had high school diplomas.

The GI Bill paid for full tuition and books and provided a modest living stipend.

Ten million WWII vets went to college on the GI Bill.

That number included fourteen who would go on to win the Nobel Prize.

Twenty-four who would go on to win the Pulitzer Prize.

The GI Bill would educate 91,000 scientists.

And 67,000 physicians.

As well as several presidents, senators, and Supreme Court justices.

How much did it cost? Educating the WWII veterans cost about $50 billion in 2007 adjusted dollars, and some estimates place the return to the U. S. economy at a equally adjusted $350 billion, a seven-fold ROI. The increase in tax dollars alone just from the boost in earning power for the newly minted college graduates more than paid for the GI Bill.

The story is part of a continuing series on the reduced effectiveness of the GI Bill. Given the statistics cited above, it would be hard not to be in favor cranking up the investment in education for current veterans.

But surely there's a more general lesson to be learned here regarding the ROI on investing in education for your employees. I've personally witnessed a large technology company reduce its educational subsidy from "your entire degree" to "just the classes we think are pertinent to your current project", and downsizing its employee training program from a rich set of intense forty-hour in-house classes to generic introductory web-based presentations. As both an employee, and as a former educator, it broke my heart to see how little interest the company had in investing in its people, beyond reminding them at every six-month review that "continuous learning" was critical to their continued employment. How they accomplished that continuous learning was left as an exercise for the employee.

Do we really think the ROI for those WW II vets is that substantially greater from the ROI we get from investing in the education of our employees? And as an employee, doesn't it suggest to you that education is virtually free, given the likely increase in your earning power?

Employees should aggressively seek training and education, whether that means choosing to work for companies that provide and subsidize it, or paying for it themselves.

I'd be interested in comments from any veterans on their experiences, good and bad, with the GI Bill, and its impact on their lives. And comments and suggestions from readers in general on their experience with training and education provided or subsidized by their employer.

Thursday, September 20, 2007

Pocket Web Service

In A Future Without Keyboards and Asterisk, WiFi, and a Nokia N800 Internet Tablet, I mentioned how I was playing with the Nokia N800 Internet Tablet, a pocket-sized, Linux-based, WiFi-capable, internet appliance. I also mentoned that I had installed a C Virtual Machine (CVM) on it, a version of the JVM configured for the Java Connected Device Configuration (CDC).

After just a few hours of futzing around, I can now serve the entire Digital Aggregates Corporation web site off the N800 using some Java code I slung together based on the prior work of Jon Berg of Sure, it's not the fastest or even the smallest web server on the planet. I'm not about to expose it outside the company firewall. And of course I'm sure I'm not the first to do this. But I can sit at my laptop and browse pages on a web server that I'm carrying around in my shirt pocket.

Here is the source zip and the jar file (served off the production web server) if you want to have this kind of joy yourself. I dropped the jar file into the Foundation directory that comes with the CVM distribution, placed a copy of the entire web site in a www directory, then fired the server up with this shell script.

cd Foundation*
./bin/cvm -cp chapparal-0.3.jar \
com.diag.chapparal.http.Server \
/home/user/www 300000 8080

I then pointed my laptop's web browser at and watched wackiness ensue! (The IP address is DHCP served from behind my NAT firewall, so your mileage will vary.) The three arguments to the Server main are the root path prefixed to every file request, the number of milliseconds the server stays up, and the HTTP listen port to use.

I developed the whole thing on my laptop using Eclipse, tested it using junit, built it for the target with ant using the 1.6 compiler in 1.4 source and target mode, and then downloaded the jar to the N800 via its own web browser.

Wicked mad fun!

Update (2008-01-03)

Sun Microsystems offers a Java SE 5.0 implementation for Linux on an embedded PowerPC that you can use for a ninety-day free trial. (If you want to ship it as a product, you have to pay a royalty.) As an experiment I recently installed this JVM on such a system, and as a demo ran my Java-based pocket web server on it. Took almost no effort, worked like a charm. It makes a nice proof of concept.

Wednesday, September 19, 2007

The Sapir-Whorf Hypothesis

"When all you have is a hammer, everything looks like a nail." -- Bernard Baruch

In Just In Time Learning, I argued that we should concentrate on learning the abstract rather than the concrete, for example new design patterns instead of new programming languages, because of their broader applicability. Now it may appear that I am going to argue for just the opposite.

One of the best ways I've found to learn new design patterns is to learn new programming languages that depend upon those patterns. One of the problems I have with using C++ to teach object oriented programming is that you don't have to use OOP to use C++. This was by design. It allows C++ to more easily and incrementally penetrate into legacy C code bases. You can use C++ as a "better" C, and ignore all the OO stuff. But this allows you to cheat if you are using C++ to learn OOP. Better to use a language like Java, which forces you to use OOP. Well, I suppose you can cheat in Java too, if you care to make everything static, but it's a heckuva lot more work than to just do it the right way straight off. Learning Java forces you into an OO way of thinking.

This is a variation of the Sapir-Whorf Hypothesis. No, Sapir-Whorf doesn't have anything to do with Klingons, although it may have something to do with the Klingon language. In the early to mid-twentieth century, linguist and anthropologist Edward Sapir and his student Benjamin Whorf formed a hypothesis that suggested that any natural language conditions the thoughts of the people that use it. In other words, not only does human culture influence the language used by that culture, but that the language in turn influences the thoughts of its speakers. If your natural language required that every sentence started with "Please", would children raised in such a society inevitably be more polite?

The hypothesis is essentially untestable, and has hence been controversial. Separating people from language for the purposes of a double blind study is impossible. But as a computer scientist and software developer, I find it hard not to be compelled by it. Because not only to do the design patterns you use influence what programming language you may choose to develop in, the programming language you develop in may influence what design patterns you choose.

In undergraduate and graduate school (this was back in the Dark Ages, you understand), I was lucky enough to have been exposed to a wide range of programming languages. C, much less Java, hadn't even been invented by the time I was an undergraduate. There was a rich culture of experimentation and innovation regarding programming languages, because no one language had really taken root as the "default" language to use for software development the way C and Java have today.

For sure, FORTRAN was was the language of choice (and still is) in scientific computing, and COBOL was the lingua franca for business applications. But every developer back then knew that neither FORTRAN nor COBOL was perfect for the burgeoning backlog of desired software applications, particularly for systems programming. That's why I ended up writing thousands of lines of assembler code (frequently using complex macro packages that made it look like a higher-level language) in the 1970s, and why C was invented at Bell Labs as a kind of structured assembler language. But it also lead to dozens, maybe hundreds, of other programming languages being tossed out into the software development community to see if any of them would stick. Let a thousand flowers bloom; let a thousand schools of thought contend. Except unlike Mao, the computer science community meant it.

Some of these languages were domain specific. Some were designed around novel hardware or operating system architectures. Some tossed the whole idea of procedural programming out the window, leading to whole new techniques for designing algorithms. Learning and programming in these languages lead to a many new paradigms and patterns being imprinted on your brain.

But see, here's the thing. Many of these new schools of thought can be adopted to languages like C, C++, and Java. They can lead to new and better ways to solve problems. Ways that were more efficient, more easily understood, and even more economically maintainable.

Some of the languages I learned in those Dark Ages persist today. Some do not. Some only existed for a short while in the feverish brains of my colleagues. But from each of them I learned something that I might not have learned had I just learned C or Java. Here are some examples.

Prolog. Prolog is a logic programming language. You do not write procedures or methods. You make logical statements that are themselves either true or false, or which can be evaluated as true or false with the insertion of boolean values into variables. A Prolog program might consist of hundreds or even thousands of logical statements, some standalone, some referring to one another. When you run the program, you set the initial conditions of any input variables, and the Prolog interpreter searches the entire solution space described by the Prolog program for a path through the search tree that could succeed. (There may be more than one.) If it exhaustively goes through all possible paths without finding a path that succeeds, the program fails. Of course, there might be a print statement or other side effect along the way.

Here is a Prolog program that was once part of my USENET posting signature. It's not too hard to guess what it means.


Yep, it's a disclaimer: these opinions belong to me; they do not belong to anyone else. Hacker humor. The bang prevents the interpreter from back tracking back through the solution tree to try another path, and the fail causes the program to fail.

Prolog blew my mind because it was such a different approach to writing algorithms. It's syntax was very grammar like (I was really into formal languages and parsing at the time), and I immediately set to work using it to implement a parser for another programming language. When I submitted a sample program in that language to my Prolog program, if the sample program was syntactically correct, the Prolog program succeeded, otherwise it failed. Side effects managed the symbol table and emitted pseudo-machine code.

Lisp. I was never a Lisp hacker, but the List Programming language was my introduction to functional programming style and the idea of representing complex structures as trees and performing transforms on them. Both of these ideas, functional programming style and structure representation and transformation, became part of my every day software design tool kit to show up again and again in the systems I develop. I think if I were to encounter Lisp now, I would appreciate it so much more than I did as an undergraduate, having spent the past thirty years applying its lessons in countless systems and applications.

SNOBOL. SNOBOL is a string processing language with regular expressions. Sure, that seems mundane now, but this predates UNIX and awk. Snobol taught me how to think of strings not as collections of bytes but as malleable forms that could be described and manipulated using complex regular expressions. Years later when I starting using UNIX, writing parsers and data tranformers using awk came naturally, partly because of my prior SNOBOL experience.

Back in the day it was a little game to see how many lines of code it took to write an 80-80 lister. An 80-80 lister was a program that read in punch cards and printed their content to the line printer. Not that you really needed to write an 80-80 lister, the IBM utility IEBGENER worked just fine. But it was just a quick way to evaluate the power of a programming language. An 80-80 lister in SNOBOL looked like this.


It was almost cheating, really.

Simscript. Simscript was an invention of the RAND Corporation, that wacky place that also brought us the Delphi technique of estimation and most of the ideas behind Dr. Strangelove. Simscript is a procedural programming language with support for simulation in the same way that Java supports multithreading: as part of the language itself. I only used Simscript as a student (it is a closed proprietary language, hence hard to get to), but all during my career I found myself wishing I had it around. For some reason I keep encountering the need to write simulations with event streams whose emission rates meet some statistical distribution. What is a few thousand lines of code in C become a few hundred lines of code in Simscript. I learned the basic design patterns of simulation in Simscript as an undergraduate, and I keep applying them in my professional career over and over in other languages.

Forth. Forth is a stack-oriented threaded interpreted language that is still in use today. Here, "threaded" has nothing to do with multithreading, but indicates how the interpretor stores the intermediate form of the program. A Forth program consists of a free form stream of words and numbers. New Forth words can be trivially defined. There is no compiler. Words are parsed and either executed directly or converted into some intermediate form (typically an address) into a dictionary as you type them in. Here is the definition of a simple Forth program that squares its input.


This is a new word called SQUARE. It duplicates the value on the top of the stack so that there are two copies on the stack, then it multiplies those two values together while popping them off the stack, and pushes the result on the stack. The following Forth code squares the value of 12 and prints it out.


A useful Forth interpreter can be implemented in just a few kilobytes, and as such it remains popular in embedded circles. There is an IEEE standard for portable boot firmware that is Forth-based, and to this day you can walk up to many Solaris-based Sun workstations, hit a control sequence on the keyboard, and break into a Forth prompt from an interpreter in the boot ROM. I've embedded a Forth interpreter as a shell inside a C++ application, and have also used Forth's threaded interpretive approach in C and C++ code as a kind of self-modifying code model.

SAS. SAS is another closed, proprietary language that I've been fortunate enough to have access to several times in my career. It's a statistical analysis system, not a language really, but more like a large collection of feature-rich statistical programs with a consistent, high level interface.

I love SAS. Because simulation keeps rearing its ugly head in my work, I keep needing a way to crunch a lot of numbers and produce beautiful charts to summarize it. Spreadsheets don't cut it when you're dealing with gigabytes of data. For SAS, that's all in a days work.

SAS taught me the value of a consistent UI among a toolchain, and how that UI could look like a higher level language. It also taught me how to manage large datasets, something else it necessarily does well.

FP. FP is a pure functional programming language, pure in the sense that it has no concept of variables. It was first described by IBM fellow John Backus, the inventor of the FORTRAN programming language and Backus-Naur syntax, in his Turing Award lecture. It combines some of the ideas behind Kenneth Iverson's APL (another interesting language) with a functional programming style in an effort to liberate software design from the constraints of the traditional von Neumann computer architecture. I never actually used a working FP compiler, but the language turned out to be the partial inspiration of much research during my graduate student years.

Functional programming has informed my software design for the past thirty years, breaking me free early on from the traditional thinking of "compute a value, store the result" and the need to store partial results in variables. The functional programming paradigm maps to Java particularly well, since exceptions can be used to handle errors when using a functional design model.

BAFL. BAFL was another functional language based partly on FP. It existed only in the research group from which my thesis project emerged. What made BAFL really different from other languages, besides being functional instead of procedural, is that it had lazy evaluation. It never did a computation until it was convinced you were actually going to use the result. Lazy evaluation is the interpretive equivalent of the common compiler optimization of not generating code for parts of the program which have no effect.

I remember once we wrote this huge BAFL program to compute some mathematical function or other, turned it loose, and... nothing. We made a couple of corrections, tried it again, and... still nothing. We were looking for the bug when we realized we never told it to do anything with the result, so it never computed it. We added a print statement, and then the interpreter finally executed the huge evaluation tree that it had built.

BADJR. An implementation of BADJR, a language which only existed in the mind of my thesis advisor and mentor Bob Dixon, was my thesis project. In many ways it was the most bizarre programming language I have ever used. It had arbitrary precision fixed point arithmetic, implemented in two's complement base 100 binary coded decimal. It had no language constructs for iteration, instead opting to use efficient tail recursion in which the stack doesn't grow. And it had single-assignment variables: you could assign a variable a value once and only once, after which it was immutable.

When I describe this to people, particularly the single assignment part, a common initial reaction is "this can't work". The idea of single assignment seems more foreign than having no variables at all. But it is simply a very different programming style. The value of a particular variable in a particular stack frame can never be changed, but you can recurse and create a new copy of the same variable in a new stack frame and change that version.

BADJR was sufficiently powerful that another graduate student used it to implement a Prolog interpreter for his thesis project (and he went onto Bell Labs a decade ahead of me -- I always was the slow one). The entire research program from which my thesis emerged was inspired by Japan's Fifth Generation Computer Systems Project, which was to use AI (okay, so now you know it failed), massively parallel computers, and dataflow architectures, to take over the world. BADJR's single assignment architecture mapped well to massively parallel dataflow architectures, and the 5G project had already expressed an intent to leverage Prolog in its AI work. So we thought this was pretty cool stuff.

And it was, in its own way. BADJR pounded recursion as a school of thought into me like no other effort had. It's arbitrary precision arithmetic taught me how multiplication and division actually worked (and led to Mrs. Overclock giving me a copy of Knuth, including the ever-so-critical Seminumerical Algorithms, as a gift). And the two garbage collectors I had to implement for it (reference counting, and mark-and-sweep) helped me appreciate the issues underlying GC in the JVM decades later.

Assembler. Sure, laugh. I learned how and why programming languages work under the hood by writing tens of thousands of lines of IBM BAL and PDP-11 PAL assembler code (and made me a real C zealot as a result). That experience has served me well in my embedded work, when figuring out some of the more esoteric Linux kernel macros, and in fixing the occasional errant DSP application.

But more than that: my early assembler work circa 1976 was my introduction to object oriented design. No, really. The OS/360 I/O subsystem's assembler code introduced me to polymorphism, inheritance, modularity, and encapsulation, and to such OO mechanisms as dynamic binding and virtual tables. When C came along to be the portable assembler language I had always wanted, I used some of these same techniques, admittedly primitively implemented. When C++ finally came along, I breathed a sigh of relief: finally, a language that does all this for me.

Okay, maybe I really wasn't that smart. I'm not sure I really appreciated all those design patterns in the IOS. I just knew a good idea when I saw it, and the first rule of engineering is: steal all the good ideas.

Today I might recommend languages like Ruby and Haskell, although Prolog, Lisp, Simscript, Forth and SAS are still alive and kicking, and still very much worth learning just for the purposes of bending your head around their very different ideas of how to code. And assembler, maybe particularly Intel P4 assembler, still has its place too.

Learning new, and more importantly, radically different, programming languages will make you a better developer even if in your day job all you use is C or Java.

What new and different programming languages would you recommend?

Monday, September 17, 2007

Hail Ming! says I'm a Nerd God.  What are you?  Click here!

Domo Arigato, Mr. Roboto

"Travel is fatal to prejudice, bigotry, and narrow-mindedness, and many of our people need it sorely on these accounts." -- Mark Twain

Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I just returned from nearly three weeks in Japan. Five days of it were spent attending Nippon 2007, the 65th World Science Fiction Convention, in Yokohama. Nine days were spent with a group of over two dozen fans from the convention, travelling around Japan seeing the sights, having a grand time, and just generally making a nuisance of ourselves being henna gaijin. The remaining time Mrs. Overclock and I spent making a careful survey of the extensive and wonderful Tokyo subway systems, gawking like the small town rubes we really are.

First, the convention. The location of the World Science Fiction Convention is chosen by voting members of the WorldCon two years in advance from among the bidding locations. The WorldCon is completely fan-run. The most senior committee members may receive some compensation since for them it is very much a full time job, but for every one else it is a labor of love. For a convention that pulls anywhere from 2000 to 8000 attendees, the fact that it is a volunteer effort alone makes it a remarkable undertaking.

Nippon 2007 had about 2200 attendees, of which about 800 were from outside of Japan. While many WorldCons are held outside of the U.S., this was the first to be held in Japan, and hence the first to be run by Japanese fans. If my own experience is any indication, it was a complete success. The Japanese fans deserve a big domo arigato goziamas from this henna gaijin. Well done.

For many of us from the west, being in Japan is the closest we are likely to come to making first contact on another planet. For science fiction fans, this is a very resonant thing indeed. There were times I felt like I was living in The Mote in God's Eye by Niven and Pournelle, along with all that implies. (I might add that Mote is the only novel of any kind that I reread every decade or so.)

It is as difficult to quantify or qualify science fiction fandom as it is to precisely define science fiction. One of our tour group remarked that it is less than a family, but more than an organization. Mrs. Overclock characterizes it as a tribe, as in "Nothing good can come from dating outside your tribe." I like this a lot. Merriam-Webster says a tribe is "a social group comprising numerous families, clans, or generations together with slaves, dependents, or adopted strangers" or "a group of persons having a common character, occupation, or interest". Either of these serves as a workable definition of science fiction fandom.

I always know that I'll share some common interest with any fan. Perhaps I'm not into filk, maybe you aren't into anime, but we can both agree the latest William Gibson novel is just frackin' great. That's why travelling with fen (the plural of fan) almost always works well: you know the dinner conversation is going to be interesting, and almost all fen have a nearly insatiable thirst for the new and different, usually for the downright weird. They also seem to almost all have a wicked good sense of humor.

On our way to distant WorldCons, Mrs. Overclock and I play spot-the-fan in airports. It is cheating if they pull out the latest Gregory Benford novel. It is interesting to note that Mrs. Overclock and I had no problems identifying Japanese members of our tribe, regardless of the fact that we shared no spoken or written language with them. Apparently the fan gene transcends ethnic boundaries. Truly, this warms even my neutronium heart.

If science fiction is the religion of fandom, then it is a religion which has a huge and varied pantheon of gods. This is not unlike Shintoism, or for that matter Greek and Hindu mythology. And like those other pantheons, our gods walk among us. I had casual conversations at this particular convention with Gregory Benford, Larry Niven, and Joe Haldeman, and once I observed Mrs. Overclock chatting with Charles Stross. (For that matter, at least two members of our tour group were published authors, and Mrs. Overclock herself appears in a DVD and on two music CDs of science fictional sensability that you can purchase on

Lest you think it's all about elves, Star Wars, and Spock ears, one of the panels I attended was a slide slow and Q&A session by a forensic anthropologist who worked on a U.N. war crimes investigation team in Kosovo, Serbia. Holy crap, that was an eye opener; how far the Austro-Hungarian empire has crumbled. Other panels I attended included topics like The Sapir-Whorf Hypothesis (with Geoffrey Landis) -- no, that has nothing to do with Klingons, it's from the field of linguistics, the growth in public surveillance (with David Brin), the current thinking in life extension (with Greg Benford and Joe Haldeman), the Singularity (with Benford and Charles Stross), and legal aspects of current directions in intellectual property law (with Cory Doctrow). Only at a science fiction convention, folks. Maybe they should call it an "awesome mind blowing stuff convention".

"The future is already here. It's just not evenly distributed." -- William Gibson

This quote came to me while I found myself travelling with a group of more than two dozen fen at nearly 200 miles an hour across Japan in a superexpress or "bullet" train. It may be that every age thinks of itself as an age of miracles, but this is the age of miracles in which I live: that I can be sitting in first class comfort reading a book while traveling at the speed of Formula 1 race car. I can do so in the company of a group from five different nations (U.S.A., Canada, Great Britain, Australia, and Sweden) yet all in the same tribe. And I can order an iced coffee from the lady with the food cart. Sometimes you just gotta step back and live in the moment.

Fandom is populated by an inordinate number of technologists, and perhaps rightly so, since it is they that must work in the future professionally. But our little group of tourists included two physicians, two lawyers, a nurse, a librarian, a retired school teacher, an airline pilot, someone running for public office, and a megalomaniacal supervillain. It was about evenly distributed in gender (provided we agree that there are two genders), and probably statistically typical for sexual orientation and physical abilities. We were probably not typical for number of college degrees per capita.

What the demographics won't tell you is that it was a damned interesting group of people with which to travel. It was with this group that we made our whirlwind tour of Tokyo, Mount Fuji, Hakone, Hiroshima, Miyajima, Kyoto, and Osaka, all the while dealing with the effects of Typhoon #9, a.ka. "Fitow", as it hit the coast of Japan. It was sobering to stand near ground zero in Hiroshima and realize that with the exception of a single structure preserved for posterity, everything around us dated from one fateful millisecond in 1945. While we did not see every single Shinto shrine and Buddhist temple in Japan (seems like there is one just around every corner), I'd like to think we hit all the big names. Despite the fact that I purified myself at every one, I still managed to uphold my tradition when travelling abroad and bring home a new strain of viral crud. When the epidemic hits North America, you can blame me.

Our tour would not have been nearly so successful (in fact, it could have been total disaster -- getting fen to break their target lock with shiny pretty things is like herding cats) without the hard work of a lot of folks, and they also deserve a big domo: fearless leader and fan wrangler Ken Smookler, North American travel agent Alice Colody, our extraordinarily patient Japanese guides Kaori-san, Nobuyo-san, Yasuko-san, Yuki-san, and the many other expediters and drivers whose names I didn't catch. You brought 'em back alive!

"I don't think we're in Kansas anymore, Toto." -- Dorothy in The Wizard of Oz

Mrs. Overclock and I completed our trip with a few days spent exploring Tokyo. We stayed at The B Akasaka, a small boutique hotel that I would recommend. It is five minutes walk from the Akasaka subway station on the Chiyoda line that is right next to (there is a God) a Starbucks. Every morning we could be found there with our lattes and curry pies, pouring over maps planning that day's campaign, while avoiding the commuter rush hour.

Tokyo is unbelievably huge, to us anyway (maybe less so if you live in New York City or London). It is divided up into twenty-three wards, any one of which is the size of downtown Denver Colorado. Tokyo and Yokohama are the first and second largest cities in Japan, and the Tokyo-Yokohama corridor is the most densely populated region on the planet. We took an hour-long bus ride from Yokohama to Tokyo and never once left the urban cityscape. I've read that it was this corridor that was the inspiration for William Gibson's U.S. eastern seaboard "Sprawl" in his novel Neuromancer.

But we mastered the London tube, so we were equally determined to tackle the Tokyo subway system. We managed to navigate our way not only across different subway lines, but even across different competing subway systems, the Tokyo Metro and the Toei. We found our objectives in Akihabara ("Electric City"), Asakusa, Shibuya, and Shinjuku. We had a good time. We drank a lot of beer, hydration under such circumstances being crucial to clear thinking. That, and we never had to drive.

Several times while in Japan, and never more so than while using the subway, it occurred to us that there are just so many successful design patterns for doing anything. And so it turns out that the Tokyo subway works much like the London, Chicago, and Washington D.C. subways. This also applies to getting a cab, ordering a meal, and paying for a scale model Tachikoma in a Akahabara robot store.

We did encounter some new patterns.

The shinkansen or "superexpress" trains (only called "bullet trains" in the West) have almost no luggage space. It's a good thing too, since a superexpress stops for only about one minute at each station (this is not an exaggeration). You have to be at the door with your luggage in hand ready to get off or on. The Japanese solve this problem by shipping luggage seperately via motor freight. We used this successfully twice during our trip, packing for a few days in backbacks until we could rendezvous with our luggage. We also did laundry twice during the trip, having taken only a week's worth of clothes. Partly this was due to the desire to travel light, but also due to weight restrictions on domestic flights.

We successfully deciphered the bizarre Japanese system of street addresses that confounds even its cab drivers to find Mrs. Overclock's Japanese bead factory outlet. We are both pretty amazed that this worked. The Japanese system does not make use of street names. In fact, with the exception of very major thoroughfares, streets are not named. Although some Japanese cities are based on grid system adopted from the Chinese, Tokyo, especially the older parts, is a maze of twisty passages which all look alike. Apparently this was a deliberate attempt by paranoid shogun to make invasion difficult.

We ate at a typical Japanese lunch counter, where you begin by inserting coins into a vending machine. As you insert coins, possible meal selections are illuminated on the menu on the front. When you finally make a selection, you get a ticket. You give the ticket to the person behind the counter, and after a few minutes of hash slinging your meal appears. The staff never handles money. Mrs. Overclock and I discovered that the chicken katsu don was just like what we eat at our favorite Japanese restaurant near our home.

The Japanese have a very civilized approach to religion: you can have none, or you can have several simultaneously, and only practice when it suits you. I found this practical approach so compelling that I found myself praying at every Shinto shrine and Buddhist temple we visited, when I haven't been to a Western church in years except for weddings or funerals.

My month spent in the People's Republic of China in 1995 was a life changing experience for me, and it prepared me for three weeks spent being deaf, dumb, and illiterate in Japan. It is humbling to not be able to read signs or even ask simple questions. But I never really felt handicapped. Japan can be easily navigated, enjoyed, and savoured, without speaking Japanese.

And it should be. While the U.S. has a culture of individualism that (I would like to think) promotes innovation, Japan has a culture of group harmony that promotes efficient production. As peoples we are stronger together than we are separately, because our strengths are complementary. Both cultures have much to learn from one another.

Plus, they have lots of really cool stuff.

Wednesday, August 22, 2007


A note from Housekeeping: Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I will be leaving shortly for a three week trip to Japan, where internet access (for us, anyway) will be limited. The fannish among you will correctly guess that we will be attending Nippon 2007, the 65th World Science Fiction Convention, held this year in Yokohama. We will also be spending time in Tokyo, Hiroshima, Kyoto and Osaka. Perhaps we will see you there!

Just In Time Learning

It has been said that there are only seven basic plots from which all stories derive. Textbooks define seven simple machines on which all mechanical devices depend. These numbers are in dispute, of course, just as the number of fundamental particles has grown from the neutron, electron, and proton that you may have been taught in grade school. When I sat around the campfire with the tribe as a child, it was pretty much just Earth, Air, Fire and Water.

You may be surprised to find that in 1936 mathematician Alan Turing proved that all computing devices can be defined in terms of a few simple operations which he defined to be a Universal Turing Machine. For sure, the UTM is an abstract concept; programming in assembler (any assembler) would be a tremendous step up. But the basic idea is that all computing is at its heart built on a foundation of just a few very simple operations, which are combined macro-like to define more complex operations, which are in turn combined to define yet more complex operations, pretty much ad infinitum. It's macros all the way up until you finally arrive at languages like Ruby, Prolog, and Haskell.

Barry Karafin, computer scientist, professor, one-time CEO, and former head of Bell Laboratories, once said: "The half-life of any high technology is around five years". That's not true for all technologies, of course. As much as we hate to admit it, FORTRAN and COBOL have actually done pretty well for themselves and are still in use today. C has done remarkably better, serving as a portable assembler language in most of its applications. The jury is still out on C++ and Java, although I have more faith in the latter than in the former. It's not clear when or even if the Internet Protocol will be ever be replaced, but probably not in my lifetime.

While it may be fun to learn the latest hot new language (or framework, or any other technology), and hence feel hip for at least a brief while, to me that hot new language just doesn't really feel all that new. It seems more like one of those movies that try to recapture the feeling of a television show from thirty or so years ago, and frequently doing a terrible job of it. Even if a new technology is successful, you usually get the feeling that it won't be successful for all that long before it's replaced by the next big thing. The success of C, Java, and TCP/IP is that they've stuck around long enough to make becoming an expert in them worth your time.

I don't know how many basic algorithms or design patterns there are, but it's a lot more than seven. But I'm pretty sure it's not infinite either. And I know that those algorithms and patterns aren't language-specific. Sure, a pattern might be easier to implement in one language than in another, speaking as someone that has done object-oriented design by implementing the C++ virtual table in assembler. But as long as your language is Turing-equivalent, I'm positive you can implement any algorithm or design pattern in it.

That's why I find it a lot more satisfying to learn algorithms, design patterns, and basic techniques, than I do to learn a new programming language. And that's why for many high technologies, I practice Just In Time Learning. I deliberately delay in learning most new technologies until it becomes apparent that I'm actually going to need to know it. Then I ramp up quickly.

For sure, there are lots of things I've learn just for the sheer joy of it. Digital Aggregates has an Asterisk server because I got interested in the idea of an open source PBX. I have a Nokia N800 on my desk because I'm intrigued with the idea of an wireless internet appliance. I installed the CVM on it because I'm interested in using embedded Java in such devices. But I'm very careful where I spend my time in learning new things. This is partly because I am old and don't have much time left (see One Hundred Books), and partly because the value equation inside my head tells me it's just not worth the time I do have when compared to other things I could be spending my time learning. I prefer to learn high-leverage stuff.

Does this mean that I am arguing against the life-long learning that most career advisers say is absolutely necessary? Just the opposite. To practice JITL, you have to be really good at coming up to speed quickly. To do that, you must learn how to learn. And the only way to do that is by practicing learning new things. But you have to be picky about where you spend your time, because even though you are probably a lot younger than me, it is likely that your time is limited as well. (Whether the time of your children is similarly limited is a discussion we'll leave to the Singularity folks.)

Does it mean that I refuse to learn niche technologies that I need to do my current project? No, indeed. But it does mean that learning those niche technologies is made easier by that fact that I can see in them patterns and techniques in common with stuff I already know.

Fortunately, my career has been one big rummage sale of projects whose learning requirements were all over the map, ranging from writing cryptic assembler code for resource constrained digital signal processing chips to writing Java code for business process automation. Yet the same design patterns keep showing up, time and time again, regardless of the project or the language in which we implemented it. And in those rare moments in which I thought to myself "Hey... I've never see this technique before!", I had the wonderful feeling that I was adding yet another useful tool to my toolbox.

The older I get, the more interested I become in some of the really hard problems, like process issues (people are always more challenging to deal with than technology), or in the big picture areas like finance and economics. Some of the perhaps more esoteric stuff on my list to learn include rate monotonic analysis, game theory, and queueing theory. These aren't technologies, but techniques that can be applied to many areas, and for the distant foreseeable future. Many companies require their employees to take so many days of training per year, then offer the most introductory of classes in technologies that are likely to have disappeared in just a couple of years. Unless they are teaching a technology that is going to be imminently used by the employee, their training dollars would be better spent in teaching more fundamental skills and more broadly applicable tools.

I'd better get a move on. Time is short.

Thursday, August 16, 2007

What's Old Is New Again

Back in January I wrote an article on the Java Business Integration (JBI) specification, and my experience with a particular implementation of it, Apache's ServiceMix Enterprise Service Bus (ESB). One aspect of JBI really threw me for a loop back when I was in the trenches using it in a product development effort: the behavior of send versus sendSync.

JBI (and ServiceMix) offers two different mechanisms for sending a message across the ESB: send and sendSync. The send call is asynchronous, meaning it's fire and forget. Once the method returns successfully to your application, the only thing you really know is that your message exchange object has entered your delivery channel. This is true even if the message requires a response; the response will arrive later asynchronously on the same delivery channel.

It's like dropping a letter in the mailbox. Although it's likely it will be delivered to the destination, there is no guarantee. Furthermore, there is no guarantee that the order in which you dropped many letters addressed to the same recipient in the same mailbox will be the order in which they will arrive and be read by the recipient. (This is the part that really threw me.)

If all you are doing is sending in product registration cards for the new desktop PC system you just bought, it's no big deal. But if you are mailing deposits and money transfers to your bank, order is suddenly important: if all the transfers arrive and are processed before the deposits, some of the withdrawals associated with the transfers may bounce. That's because the former example is stateless, while the latter example is stateful.

JBI provides a sendSync call which is synchronous, meaning your application blocks on the sendSync call until it is acknowledged by the recipient. The kind of acknowledgement depends on the message exchange pattern (MEP) you are using. For example, if you are sending an InOnly MEP, then all you get back is a indication of Done: you are notified that the recipient received your message and that it actively marked the message exchange as completed. If you are sending an InOut MEP, then you get back a message exchange containing a response from the recipient, and it is you who must then mark the message exchange as completed by sending a Done back to the recipient. You have guaranteed delivery, or at least you know if it didn't work before proceeding, and (more subtly) order is guaranteed to be preserved among successive sendSyncs that you perform to the same recipient.


This sounds simple, but in a system of any complexity at all, it may not be pretty.

The most obvious design issue is that the original sender (the service consumer in JBI parlance) is blocked waiting for the recipient (the service provider) to process the message exchange. On a loaded system, the recipient may be busy processing a long queue of pending requests from many senders. This causes the sender to block for potentially a long period of time. Since the sender may itself be acting as a service provider to other service consumers (that is, they are trying to send it requests and get a response), pending requests can back up in many components throughout the system. Wackiness may ensue. Even if the system isn't busy, handling some requests may require long latency operations like persistence handling, database queries, or remote network accesses.

The other obvious design issue is that if there are circumstances in which the recipient may itself act as a service consumer to the original sender's service provider, that is, the recipient may as part of its processing make a request of the original sender, and both components use sendSync, the system deadlocks. The original sender is waiting on its sendSync as a consumer to the recipient as a provider, and the recipient as a consumer is waiting on yet another sendSync to the original sender as a provider. Neither sendSync will ever complete, or they will timeout if they were coded with a timeout parameter.

This is not a new issue. Developers (like me) old enough to be taking their meals through a straw will recognize a similar issue in the remote procedure call (RPC) paradigm that was fashionable in the 1990s. In RPC, distributed components communicated with one another through function calls that were mapped by frameworks like CORBA, OSF DCE, or SunRPC (I've used 'em all) into network messages. Developers (like me) old enough to have one foot in the grave will remember dealing with this issue when writing bidirectional messaging frameworks using Berkeley sockets and TCP in the 1980s. I dimly recall similar issues arising when writing communications layers for PDP-11s using RS232 serial connections in the 1970s.

Geoff Towell, a colleague of mine in the JBI adventure, remarked that in his experience "systems using synchronous message passing aren't scalable." He also noted that "systems using asynchronous message passing can be difficult to write." In my experience, he was correct on both counts. The fix is the same whether you are using JBI, RPCs, sockets, or serial ports.

To insure guaranteed delivery and preserve order, you use synchronous message passing: sendSync, an RPC call with a returned value, a TCP socket, or a serial protocol that requires a reply. But the response you get back merely indicates that the recipient received your request and queued it for later processing. It says nothing about when the recipient will actually get around to processing your request. When it does, the relative roles of the components will reverse: the original recipient will act as a consumer and perform a sendSync with a new message exchange to the original sender, who is now acting as a provider and will appropriately complete the new message exchange. Hence, message passing is synchronous, but message processing is asynchronous.


We did the same thing with RPCs: the successful return of the remote procedure call merely meant that the called component received the parameters, not that it actually did anything with them. When the called component completed processing the request, it would invoke a callback function in the original calling component, doing another RPC in the opposite direction to deliver the response.

This is why the design of the sender and recipient gets ugly: they may both have to handle multiple requests concurrently. They typically do this by implementing multiple concurrent state machines, each machine implemented as an object. For each request, whether originated by the sender or received by the recipient, a new state machine object is created. Many of these objects may exist simultaneously in both the sender and the recipient as many requests are asynchronously processed. Each state machine is maintained in the recipient until a response for the request that particular state machine represents can be sent, and the recipient's state machine transitions to its final state. The original sender maintains its own state machine for each request until the corresponding response is received and processed, then that state machine also transitions to its final state.

(If you are into automata theory or formal languages, all of this will sound very familiar. The message exchange between the two components describes a protocol. Finite state machines and other automata are typically used to implement parsers for formal languages. Formal languages are described by formal grammars. The fact that you frequently use a state machine to implement a protocol is why protocols are often described in terms of a formal grammar. Such grammars are remarkably useful and should be in every developer's toolbox. But that is a topic for another article.)

There are a number of ways you might implement multiple concurrent state machines. The simplest is to have a separate thread for each request in both the sender and the recipient. This works well in systems in which the cost of context switching and maintaining a thread is zero. The fact that there are no such systems means this approach is seldom used.

(It can in fact get bad a lot faster than you might expect, since on some systems I have seen the cost of context switching increase proportional to the square of the number of active threads. I wrote a marvelous white paper on this topic, that this blog article is too small to contain.)

You may have a fixed size pool of threads in the recipient that service a common queue of incoming requests. I have seen this work well in both Java and C++ implementations in which the number of possible concurrent outstanding requests is small, the lifespan of each request is short, and concurrent pending requests are mostly independent. There are Java and C++ frameworks that provide this capability.

When I've run up against systems that have to handle 38,000 concurrent long-duration requests (and no, I didn't pick that number out of thin air), neither approach scales, and I resort to designing and coding an application-specific concurrent state machine implementation that runs inside a small number (like one) of threads. This is not as hard as it sounds.

(Dan Kegel wrote a really great article on this scalability issue in the context of server-side socket applications in systems of UNIX-like sensibilities; see The C10K Problem.)

My web services friends will no doubt be up in arms over this article, either because I'm suggesting using synchronous message passing, or because I'm suggesting using asynchronous message processing. Probably both. But my background for the past thirty years has been in building robust, scalable server-side real-time systems, and what I have described here is a design pattern I have found to work.

Update (2008-07-07)

I've recently been reading about continuations, which are a mechanism to, very roughly speaking, pick up a prior computation where it left off. It's funny: being the wizened old man that I am, I always thought of continuations as a form of checkpointing, from my time in the deep past with IBM mainframes and Cray supercomputers. It wasn't until recently that I realized that continuations serve essentially the same purpose for web servers as the state machine architecture I describe here and have implemented on several real-time systems over the years. For that matter, checkpoints served a similar purpose on mainframes and supercomputers.

I suspect that the motivation for all of these mechanisms was slightly different. Checkpoints existed because hardware was slow and expensive, it wasn't uncommon for the system to crash, and when it did you wanted to pick up work were it left off. The state machine architecture I describe here was mostly done for scalability, handling tens of thousands of simultaneous connections with just a few threads. Continuations seem to be mostly motivated not just by robustness and scalability, but the state-less nature of RESTful HTTP operations.

Maybe as I read more it will become more clear to me that continuations are completely different. In the spirit of The Sapir-Whorf Hypothesis, I should probably learn a programming language or framework that supports continuations natively.

Tuesday, August 14, 2007

Tool Economics

"If all you have is a hammer, everything looks like a nail." - Benard Baruch

I was teaching a class in embedded development at a client's site the other week. We were discussing approaches to fixing an actual bug in their firmware code base. This client uses the IAR Embedded Workbench, an IDE that provides all the usual tools: an editor, a build environment supporting the ARM-based tool chain, a JTAG interface, a down loader, a debugger, etc. When I've done product development for this client, I've used this tool, and while its debugger is the best thing since sliced bread, I found its editor to be a little weak. I preferred instead to do my editing using Eclipse with the CDT and Subversion plug-ins. The IAR EWB and Eclipse played well together, in my experience, allowing me to leverage the best of both worlds.

We discussed for a few minutes the merits of using the debugger to test our hypothesis of what the bug was, where to put in breakpoints, what variables we needed to look at. "Or," someone said, "we could just change the code and see what happens."

I had one of those brief moments of cognitive dissonance where I wondered "Who the heck said that?" and then realized "Oh, it was me!" We were all victims of tool economics.

Back in the 1970s, when I got paid to write code in FORTRAN, COBOL and assembly language, we didn't so much have a tool chain as a tool string or maybe a tool thread. We typed our code onto punch cards using a keypunch, which we frequently had to stand in line for. We submitted them to the operator at the I/O window of the computer room. Our stack of punch cards were queued up with a lot of other similar stacks by the card reader. The operator fed them in, where our job was queued on disk by the mainframe operating system. (This was OS/MFT running HASP on a IBM 360/65, for those that remember when glaciers covered the land and we communicated mostly with grunts.) Eventually the OS would run our jobs and, if all went well, our printout would come banging out on the line printer. The operator would separate our printouts and place them with our original card deck back in the I/O window. On a really good day, you might get two or three runs in, providing you got to the computer center early and left late.

The latency involved in a single iteration of the edit-compile-test-debug cycle bred a very strong offline approach to code development: you looked at your code until your eyes bled, trying as hard as you could to fix as many bugs as possible, running what-if simulations in your head because that was a lot faster that actually running your program. You saved old punch cards, just in case you needed to put those lines back in (maybe even in a different location), and your printouts were heavily annotated with quill pen marks (okay, I exaggerate, they were felt-tip pens). Changes were meticulously planned in advance and discussed with as many other developers as you could lay your hands on. You think you multi-task now? The only way to be productive in those days was to work on many completely different software applications simultaneously, so you could pipeline all of those edit-compile-test-debug cycles. Every new printout arriving at the I/O window was an opportunity to context switch in an effort to remember what the heck you were thinking several hours ago.

Sitting here, writing this while running Eclipse on my laptop and accessing my code base wirelessly from a remote Subversion server, I have zero desire to return to those bad old days. But it does illustrate how our tool chain affects how we approach a problem. Because the developers at my client were focused on using the EWB, their problem solving approach necessarily revolved around debugging, because the debugger was the EWB's greatest strength. My problem solving approach revolved around editing and refactoring with Eclipse, trying an experiment to see what happened, then maybe using Subversion to back my changes out. I resorted to using the debugger only when I had a unit test really go sideways and trap to the RTOS, or when I just couldn't see what was going on without single stepping through the code and watching variables change. Using Eclipse as part of the tool chain made it economical to try new things and verify proposed solutions quickly.

Tool economics affects all phases of software development. It limits us in the solutions we are willing to consider, and forces us to reject some potential solutions out of hand. It colors the processes we use, and the quality we may be able to achieve within the constraints of schedule and budget.

Some of the developers at my client preferred to use the debugger to unit test their code. It was easy to love the debugger in this role, but it was a very labor intensive process. This was okay economically as long as you only had to do this once, when you originally wrote the code.

But no successful code base is static. At least two-thirds of the cost of software development is code maintenance, changing the code base after its initial release. Refactoring, something tools like Eclipse vastly simplify, becomes a way of life. Going through the debugger every time you change a particular function is expensive. I chose instead to write unit tests embedded in the production code base, typically conditionally compiled in so that the test code was present in the development environment but not shipped as part of the product. If I changed a particular function, I just reran the unit test. If the test failed and I was absolutely clueless as to why, then I turned to the debugger. I paid the price of writing the unit test once, and then used it for free henceforth. For sure there are occasionally some Heisenbergian issues with the unit test affecting the functionality of the code base, typically due to memory utilization or real-time, but those were rare.

Embedding unit tests in the code base is just a more economical approach to software development from a developer utilization viewpoint. But the real reason I tend to use that approach is that it's a rare embedded development project I work on in which I have access to a debugger as good as the one provided by the IAR EWB. I'm generally forced into more formal unit testing approach because I really don't have a choice. The fact that it's really cheaper in the long run is just gravy. This is yet another way in which tool availability forces a particular approach.

Tool economics effects how we design our applications too. I remember a job in which I found myself working in an eight million line code base whose oldest elements dated back at least to the early 1980s. Unit testing was extremely expensive. There were no formal unit tests. You tested your change by reserving a lab machine for sometime in the future, loading the application, and spending anywhere from five minutes to hours testing on real hardware and real devices, sometimes in concert with logic and protocol analyzers. I admit that many times I chose a particular design, or even an architectural direction, because it minimized the number of subsystems I would have to touch, and hence re-test. Choosing the "best" solution had nothing to do with it. Choosing a solution that met requirements while at the same time could be completed within the limitations of the schedule had everything to do with it.

You think I was wrong? The managers to which I reported would disagree with you. Admittedly this was sometimes a case of time to market (TTM) taking precedence over cost of goods sold (COGS) (see The Irresistible Force and the Unmovable Object), but that was the trade-off sanctioned by that particular development organization.

The expense of testing also effected the quality of the code base in perhaps unexpected ways. Many times I recall discussing with my fellow developers over beer or lattes that, in the adventure game that was navigating this huge code base, in the course of our work we occasionally encountered code that we knew deep in our gut simply could not work. It could never have worked. What to do? We could have fixed it, but then we would have had to test it, even if it were a trivial change, placing us behind schedule on our actual assignments, and making it more likely we would be in the next wave of layoffs. We could have written and filed a bug report (and too our credit, we typically did just that), but bug reports have a habit of ending up back on the plate of the original submitter. So when we encountered such code, and if time was short, we sometimes just backed away slowly, and pretended never to have seen it. The high cost of testing drove that dysfunctional behavior.

If you are a developer, you deserve the best tools you organization can afford. No matter whether you are in the open source world, the Microsoft camp, or the embedded domain, there are very good tools to be had. The availability of good tools, and your familiarity with them, will affect not only how you approach your job, but the quality of the product that you produce.

If you are a manager, you need to make it a priority to get your developers the best tools you can afford. Your schedule and the quality of the resulting product depends on it. Just meeting requirements is not sufficient, because a requirement must be testable. Having a requirement that says "this software must be maintainable in the face of even unanticipated changes over its entire lifespan" is not testable. Whether you appreciate it or not, your developers are making economic decisions minute by minute that affect both the TTM and the COGS of your product. And, hence, the financials of your company.

Tuesday, July 31, 2007

The Irresistible Force and the Unmovable Object

Decades ago I remember seeing a print cartoon in which a distinguished scientist of Einsteinian sensabilities was deriving a hugely complex equation on a blackboard. From top to bottom of the blackboard the equation got simpler and simpler until it was finally reduced to Time = Money.

Einstein discovered that the two most basic things in the Universe, matter and energy, were equivalent. But in all human endeavors, it doesn't take much experience to develop the gut feeling that time and money are the only things that really matter, and that they are in some way also equivalent.

It is no different in the realm of software development, where time is the prime force and money is the fundamental particle from which all other things, good and bad, flow. Most managers realize this, although they (and I) prefer to think in terms of not time, but time to market (TTM), and not money, but cost of goods sold (COGS). Software developers should think in these terms too.

TTM is the time it takes a product to go from inception to its first (typically beta) customer. All other things being equal (hint: they never are), shorter is better. COGS is the sum of all the costs that can be directly attributed to the production of a product, including both the materials and the labor that go into its manufacture. Again, all other things being equal, smaller is better.

There are lots of good reasons why making TTM as short as possible is important. Having a shorter TTM than your competitors lets you introduce products leveraging new technology before they do. This may give you an advantage in market share. It may let you establish de facto standards in the market which can force your competitors into playing catchup. None of the rationales for shortening TTM are a sure thing, but they are typical.

But there is a more subtle reason for a short TTM. In Lies, Damn Lies, and Net Present Value, I talked about how when you are seeking funding for the development of a new product from your funding agency, whether that agency is upper management, venture capitalists, or the Department of Defense, you are competing with all the other uses to which that funding may be applied. Unfortunately, you are not just competing with other proposed product developments, but with compound interest.

If your funding source merely places your proposed funding in some investment vehicle for the same duration as your proposed TTM, their money grows much faster than linearly. The longer it stays invested, the faster it grows. And all the really big growth in terms of absolute value is biased towards the end of the investment period. What this means is that extending your development period (and hence, your TTM) by even a little bit can have a huge impact on the potential value of your product in the marketplace when compared to just having put the same money in the stock market. This is referred to as the cost of money (COM).

Doubling your development time doesn't double the cost of your project, at least not in the eyes of your investors. It's more like an order of magnitude because of the COM. TTM is an strong force in all product development. But because of the uncertainties that frequently occur when developing bleeding edge products using new and perhaps unproven technologies, it is an irresistible force in that arena.

If TTM is the irresistible force, COGS is the unmovable object. If your product does not cost less to produce than you can make from selling it, then you are obviously not going to make any money selling that product. You've heard the old joke: "How do we do it? Volume." COGS is what this joke is talking about.

COGS is not nearly that simple. A strategy used by many manufacturers is to sell a product as a loss leader, that is, for less than its cost to manufacture, in order to acquire market share, with the intent to raise the price after a bunch of customers are locked in. If you are selling a commodity, that is, a product which can be easily replaced with an equivalent competing product, this obviously won't work, because there is no lock in. But in high-tech, it is not unusual for products to be sold at a steep discount, or, as in the open source world, even given away for free, where money can be made through collateral products, such as service contracts.

(We may pause here to note that in some markets or some countries, there are actually laws against selling products for less than their cost to manufacture. So your mileage may vary.)

COGS has its own special subtlety when it comes to software development. Many managers (and developers) see COGS are purely the up-front cost of developing a software product. The cost of physically manufacturing a software product once it has been developed is nearly zero. It is the cost of replicating and shipping CDROMs, or of running a web server from which the software is downloaded.

This leads TTM and COGS to interact is a manner that may seem fairly obvious. Adding more developers to a project would appear to increase its COGS in more or less in a linear fashion, but reducing TTM can decrease its COGS in a much greater than linear fashion, thanks to the COM. So adding developers to a project seems like a good deal.

Most developers instinctively know that there is a flaw in that argument, just as we all know that three women cannot produce a baby in three months. In the classic software engineering text The Mythical Man Month, author Fred Brooks notes that adding people to a late project makes it later. There are a couple of reasons for this.

First, adding people to a project does not achieve a linear improvement in your schedule. Each added person will cost some time from the existing staff as they have to answer questions and train the new person. Most companies I've worked for have some rule of thumb for the net cost of bringing a new person up to speed. I've seen values ranging from two months to two years used, the latter being for working in an eight-million line legacy code base. The real cost depends greatly on both the developer and the project. My experience is that developers which have a lot of experience recognizing common design and architectural patterns come up to speed a lot faster since they tend to effectively and automatically abstract away a lot of the unnecessary details, while inexperienced developers worry about every gory little thing.

Second, there is a communications overhead as all members of the development group have to coordinate with each other. Software development is, contrary to its mythology, an extremely collaborative activity. This communications cost increases non-linearly. In the worst case, where all developers have to coordinate with one another, as a rule of thumb the interpersonal communications overhead is proportional to the square of the number of the developers.

So merely adding developers to a project to decrease TTM isn't generally the answer.

There is another, largely ignored, aspect to COGS. In The Total Cost of Code Ownership, I talked about how at least 67% of the total life-cycle cost of a successful software product is spent on maintenance. Maintenance is changing existing code: to fix bugs, to add new features or otherwise improve the product, or to adopt to changes outside of the software product itself. The original development of the product accounts for at most a third of its total life-cycle cost. (Other studies place the maintenance portion of the budget at 70% or even 90%. I suspect this depends on the longevity of the software product.)

This means if you somehow magically reduce TTM to zero, you have at best only reduced your total cost of code ownership, which for a software product is effectively your COGS, by a third.

Why is this fact unappreciated? Lots of reasons, mostly tied to (big surprise) money. Market or product managers get compensated for bringing a successful new product to market. Developers are compensated for developing exciting new products. Neither are given raises, promotions, or bonuses for keeping a legacy product working. Maintenance of legacy products is frequently an activity done in the margins. And even if it isn't, that activity is often done by a group separate from the original developers, with a name like sustaining development or current engineering, whose funding comes from a different business unit, silo, or cost center.

This is all a little ironic since for many companies, it is the successful legacy product that is generating most of the revenue that funds the development of new products. I recall once being forbidden by a department head from consulting on a complex legacy product for which I was one of the original developers, despite the fact that the issue with the product was impacting high profile customers. Not his silo, hence not his problem.

It gets uglier. Every developer has worked on projects in which in an effort to reduce TTM, they were encouraged to cut corners. I remember a systems engineer once saying in a meeting "I don't think we have time in the schedule for doing unit tests." (To our credit, the development team ignored him.) But cutting corners leads to what embedded pundit Jack Ganssle refers to as technical debt: it's a debt you take on to reduce TTM, but one which you must eventually pay back. Why? Because technical debt increases the cost of maintaining your product, and hence increases your COGS. Put enough quick and dirty hacks in your software product, and eventually your COGS rises enough that you can no longer sell your product at a profit.

The fact that you may not be keeping a good accounting of the COGS of your product, because its maintenance was moved into a different silo, may lead this issue to bite you unexpectedly. Moving a cost into an area into which it cannot be accounted for does not mean the cost goes away. (I have seen this principle at work when companies do away with desktop PC support, placing the burden of keeping the infrastructure running on the shoulders of their developers. The cost didn't go away. It just got moved into an area from which it could not be accounted for.)

Are there ways to reduce both TTM and COGS? That would be the holy grail. Anything which allows you to generate maintainable code faster would do so. Using languages and design patterns that admit easy refactoring, and using IDEs and other tools that support the same, are a step in the right direction. Software reuse has been seen largely as a failed policy, mostly I think for political reasons. Yet, the C Standard Library is probably the most reused piece of software in the world. As I like to say: "Nobody rewrites printf." The wealth of frameworks and libraries available for Java is a hopeful sign too, but there are so many of them, and they are so complex, that they can bring their own TTM and COGS problems to the table. Witness Fun Facts to Known and Tell: Java Business Integration.

Still, anything that allows you to work at a higher level of abstraction has the potential to reduce both TTM and COGS. Higher-level languages: moving from assembler to C to to C++ to Java to domain specific languages like Ruby. Objected-oriented design: moving from C to C++ or Java, or at least using OO designs in C code. Operating systems with more capabilities: moving from your favorite tiny RTOS to Linux. Reusable libraries or frameworks: C standard library, C++ Standard Template Library (STL), any of the many invaluable frameworks in Java. Using tools that automate or at least simplify the development process: Eclipse for C, C++ or Java, or any of the many great Java tools like Cobertura and JUnit.

When I plan, architect, design, and implement software products, or when I am evaluating new tools, lowering both TTM and COGS is always on the forefront of my mind. In If Java is the new COBOL, is C++ the new assembly? I argued on both a TTM and COGS basis for broader use of both Java and C++ in embedded applications.

In my three decade plus experience, I have seen a lot of effort is spent on reducing TTM. But when it comes to reducing COGS, only the hardware guys get any of the glory. Software COGS is largely ignored. To effectively reduce software COGS, you have to make software easy to modify and maintain.