Wednesday, December 23, 2009

Product Development as Warfighting

Pop quiz. Closed book. Multiple choice. Complete the following.

John Boyd (1927-1997) was:

A. in the U.S. Army Air Corps and later U.S. Air Force and served in three wars.

B. the chief instructor at the Fighter Weapons School at Nellis Air Force Base in Nevada. He was known as Forty-Second Boyd for his boast that he could win any dogfight in forty-seconds. He was undefeated.

C. the author of Aerial Attack Study, the first text book on air combat. The book, in either its original or pirated form, became the principle text book on air combat for every air force in the world.

D. the inventor of Energy-Maneuverability Theory, which applied thermodynamics to aircraft performance. E-M Theory became an invaluable evidence-based tool in the design, specification, evaluation, and comparison of aircraft.

E. a key player in the design of the F-15, F-16, and A-1 aircraft, and was an fierce opponent of the initial flawed designs of the M1-A1 Abrams tank and the Bradley Fighting Vehicle.

F. the thought leader of the Defense Reform Movement at the Pentagon, applying an evidence-based approach to military procurement. He required for the first time in decades that competing defense contractors build working prototypes of weapons systems so that they could be directly compared.

G. a retired Air Force officer who synthesized centuries of strategic thinking to codify Maneuver Warfare. He lectured at Quantico and his work was eventually adopted by the U.S. Marine Corps as official doctrine. A statue of Boyd, in his Air Force flight suit, can be found at the Marine Corps Research Center at Quantico.

Of course it's a trick question. Would you expect less from me? If you or I had achieved any one of these accomplishments, we would have considered it the capstone of a brilliant career. Boyd achieved all of them.

That's not all. He desegregated the Las Vegas casinos in the late 1950s by insisting that his black officers be allowed to dine with the rest of his men in the restaurants. And for a time he commanded what was probably the most secret base in South East Asia, responsible for monitoring the Ho Chi Minh trail.

But his achievements did not come without a price. He barely spoke to his wife, although he apparently spent enough time with her to have five children, whom he also ignored. His career stalled at the rank of full colonel because he had become a thorn in the side of too many senior officers in all branches of service. In retirement, he refused to double dip, and his pension was just enough to house his large family in a tiny basement apartment in a bad part of town. If you saw him on the street, you might conclude from the way he was dressed and acted that he was among the ranks of the homeless instead of one of the greatest strategic thinkers of all time.

And although nothing I have read has ever suggested it, all of the descriptions of Boyd's behavior and mannerisms point to the symptoms of Asperger Syndrome. It's like checking off bullet items on a checklist. Economist Tyler Cowen has written that for some professions, Asperger's or related traits on the autism spectrum are not necessarily dysfunctions but can actually be a competitive advantage. Those with Asperger's frequently are able to bring an extreme mental focus, an obsessive attention to detail, and other cognitive abilities to bear on highly complex problems. What they may lack in the social skills considered to be naturally occurring by others can be learned.

One thing for sure, if Boyd did have Aspergers, he was probably in good company. It has been speculated that luminaries ranging from Thomas Jefferson to Albert Einstein to Alan Turing also lived on the autism spectrum.

There are several recurring themes in Boyd's work, briefings, and teachings.

People. Ideas. Things.

It took me decades to figure this out on my own. In any endeavour, people are more important than ideas. Ideas are more important than things. Things (or as Boyd put it, hardware) are the least important of all.

Boyd, who felt that the Air Force was too much a technocracy enamored with the latest new hardware, preached this time and time again to anyone that would listen. It is people who make the difference. It is the idea that can be more broadly applied. The hardware is just a passing fad.

This is the mistake that many technologists make in their personal lives. When given the choice between going out with your friends and sitting at home alone cruising the web, generally the best long term choice is to go with your friends. If trying to decide between spending money on travel with your spouse or buying a new home computer, take the trip. Learning a new concept is more important than learning how to use the latest gadget.

It's also the mistake made by many hiring managers, architects, and technical leads. A really good developer can more than make up for marginal or legacy technology. A developer who is good at synthesizing and applying new ideas is better than a developer that has the latest new technology on their resume.

This is also why design patterns are more important than specific tools, because the pattern is more broadly applicable. A developer who can intuitively apply design patterns is even better.

To be or to do.

Boyd saw that a lot of the officers at the Pentagon were careerists, more interested in getting promoted than doing the right thing. He also realized that in a bureaucracy it was frequently impossible to do both. Advancement in a bureaucracy typically depends on defending the status quo, not on shaking things up. Boyd lectured his followers on the need to decide whether they were going to be something or to do something.

George Benard Shaw once said "The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man." The fact that Boyd and many of his followers were unreasonable men is why most of them saw their careers stall in mid-flight: they choose to accomplish something great rather than to become something great.

I should be the last one to give you career advice. But I will tell you that if you have any ambition at all, you will face this choice, probably many times, during your career in high-technology. I personally have always chosen to do rather than to be. But then again, I am living proof that you can spend a decade in an organization without getting promoted or accomplishing anything significant. If you find yourself in that situation, then both of your engines have flamed out and your plane is on fire. Consider bailing out.

Fast Transients.

It may seem strange, particularly to those who have served in the military, that a self-described "dumb fighter jock" would have revolutionized both air combat and ground warfare. But Boyd found many similarities. All of his work revealed the importance of fast transients: the ability, and the willingness, to change course quickly. It was this critical aspect of his concept of air combat that directly led to the design of the small, nimble, and very successful, F-16 fighter, when all Air Force doctrine at the time called for higher, further, and faster in a straight line. It also was the main idea behind maneuver warfare: move quickly, probe for weaknesses, don't get bogged down, don't worry about your flanks, let the enemy worry about his flanks. This can be applied to business, to development processes, and to architecture and design.

If your adversary is on your tail, not changing anything will not improve your situation. Changing too slowly will just give him time to adapt. You have to commit to change, to changing quickly, and to do so in a way that your adversary cannot anticipate. You must exploit what Sun Tzu referred to as cheng and ch'i, the expected and the unexpected. This is the core of what is innovation, doing the unexpected (and is what Apple in my opinion is very good at).

Be prepared to toss out a business plan or a technical design when new evidence comes to light that makes the current path unworkable. For sure, fix it if it can be fixed. But if it can't, don't spent endless meetings debating or fretting about it or moaning about the compromise of your beautiful aesthetic design. You're a craftsman, not an artist. Take the control stick in your hands and change course.

If during development you become blocked on a problem, don't sit around waiting for things to change. Find an area to work where you are not blocked. Make progress where you can. You will accomplish something while waiting for the blockage to resolve itself, and you may find that the progress that you made actually led to its resolution. This is the essence of blitzkrieg.

When designing complex real-time systems, expect emergent behavior. Be prepared to deal with it. Expect failures to cascade quickly through the layers of your architecture. Don't get so overwhelmed with a fire hose of log messages that you can no longer tell what is going on. More is not necessarily better. Eliminate detail. Reduce the logging level until the chaff and smoke and clutter is reduced so that you can clearly see your target. Achieve fingerspitzengefuhl, or finger-tip feel, an intuitive mental model of how your systems works and reacts.

Keep the end goal in sight. It is easy for technologists to become side tracked with the cool details of the latest new thing, to become tied up in dealing with a stack of non-critical bug reports, or to continue to refine and refactor that working code until it meets some arbitrary personal design aesthetic. Don't get bogged down in the thousands of tiny technical details so that you can no longer see the big picture. Make all decisions based on what will help you achieve your end goal, which is always shipping a product. Organizational guru Steven Covey says "begin with the end in mind", but among warfighters this is what is known as schwerpunkt: maintaining a focus on the objective.

Observe, Orient, Decide, Act.

Boyd's most famous idea (and probably the most misunderstood, including by me) is the OODA Loop. Boyd described how people process information to deal with the outside world. In his briefings he applied it to conflict, but it's much more broadly applicable than that. You observe what is happening. You orient, by which he meant you synthesize and interpret what is happening based on your own experience and cognitive biases. You decide on a course of action. You act on that decision. Because your actions change the circumstances in which you find yourself, you iterate and do it all over again.

When in a conflict with an adversary, both parties are iterating in their own OODA Loops. The key to success is to be able to iterate faster than your adversary. Boyd called this getting inside your adversary's decision loop. When applied on the battlefield, Marine Corps commanders remarked that it was like you were running both sides of the battle. The enemy spends all their cognitive bandwidth trying to figure out what the heck you are doing and reacting to old news.

Of all of Boyd's ideas, this is the one that has gotten the most press, mostly in the business world, for obvious reasons. It is the idea of fast transients writ large, taken from the realm of air combat and applied to conflict in general. Success in competition relies on not just making the correct decision based on the evidence at hand, but making that decision faster than the other guy.

What makes Boyd's OODA Loops go beyond just fast transients is the fact that you can negatively impact your adversary by controlling how and what he observes, and by exploiting his cognitive biases in how he interprets it. This is the role of most security and intelligence organizations (including business security and intelligence), but is crucial even among field commanders. You can force your adversary into making decisions based on the wrong information. Because you can do this to your adversary, you must assume that he is doing it to you.

Sometimes your adversary is an architecture, a design, or a legacy code base. You have to be aware of what your cognitive biases are, and whether they are blinding you to what can be done, or what needs to be done. This is one of the reasons I believe developers should learn new programming languages, particularly languages that are very different from what they use everyday, even if they never intend to use those different languages in production.

For example, if you spend most of your time using procedural languages like C, you should learn object oriented languages like C++. If you use an unmanaged language like C++, you should learn Java. If you are a C, C++ and Java expert, you should learn Prolog, Snobol, Lisp, Haskell, Smalltalk, or any number of other non-traditional languages, just to learn a radically different mindset and approach to solving problems. You may find idioms and patterns in those languages that you can productively apply to your everyday work.

Since ideas are more important than things, learning concepts like finite state machines, push down automata, and formal grammars, which can be applied in any language, are really useful too. I had several courses in graduate school on automata and formal languages that were ostensibly theory. Yet they completely changed, for the better, the way I view many programming issues I encounter in my daily work.

A point about OODA Loops that is frequently ignored, but is clear from Boyd's briefing slides, is that this iterative process is fractal: there are loops within loops at all different levels of scale. This fractally iterative process describes how I see the development process in general: you constantly iterate at all levels of scale: implementation, design, architecture, and requirements, with feedback from all levels constantly changing other levels as you decide what can and cannot be done, what must be done and what is optional, given the technology, resources, and schedule. This is really what Agile Development at its core is all about: fast yet effective iteration.

John Boyd is my hero.

I find the story of John Boyd to not only be inspirational, but also a cautionary tale regarding work-life balance and the fine line between genius and dysfunction. Because Boyd chose to deliver his message primarily through classified briefings, and seldom if ever published any of his ideas, he is largely unknown today. When his work is used, he typically goes uncredited. This is even though his ideas revolutionized much thinking in the military, in business, and for me, in high-technology development.

John Boyd died of cancer in 1997. He is buried in Arlington National Cemetery.

Sources

Robert Coram, Boyd: The Fighter Pilot Who Changed the Art of War, Little, Brown and Company, 2002 (This nearly 500 page book is surely the definitive biography of John Boyd. It is also one of the best non-fiction books I have ever read. Coram makes Boyd's story inspirational, compelling, and cautionary.)

James Fallows, National Defense, Random House, 1981 (Fallows frequently interviewed and wrote about Boyd and national security issues for the Atlantic while Boyd was at the Pentagon. This National Book Award winner covers a lot more territory than just John Boyd. Fallows' coverage leans neither left nor right, and is still worth a read today, although his remarks on nuclear policy makes me nostalgic for the Cold War when nuclear weapons were in the hands of professionals instead of radicals, thugs, and lunatics.)

Grant T. Hammond, The Mind of War: John Boyd and National Security, Smithsonian Institution Press, 2001 (This book was researched and largely written while Boyd was still alive. Coram states that Boyd prohibited Hammond from including much in the way of personal detail. It is an excellent introduction to Boyd's ideas, but Coram's book gives a picture of Boyd the man.)


U. S. Marine Corps Staff, Warfighting, MCDP 1, 1997

Saturday, December 12, 2009

In Praise of do while (false)

By now I would have thought that everyone knew the joys of the C, C++, and Java language construct do while (false). You can find articles written about it on the web from as far back as 1994, which might as well be Neolithic cave drawings. Yet I continue to have questions about do while (false) (or do while (0) in C if you haven't defined false) from code inspectors who should know better. (You know who you are.)

There is nothing magic about do while (false). It does exactly what you think it does, which is to say, very little. In fact, it does so little, your typical optimizing compiler won’t generate any code for it. Yet, it is a really handy tool to keep in your toolbox, right next to your duct tape and vice grips.

Common Exit Flow of Control

All C++ functions have a common entry point. It is frequently desirable for functions to have a common exit point. There are all sorts of reasons for this. The most pragmatic reason is having a common entry and exit point makes it easy to add debugging statements that log the arguments being passed into the function, and the results generated by the function. If the flow of control needs to return prematurely, it can do so while not avoiding the logging statement at the common exit, just by doing a break.

bool function1(int argument1) {
int rc = 0;
printf(“%s[%d]: function1(%d)\n”,
__FILE__, __LINE__, argument1);
do {
// Some really complicated code.
if (bogus) {
rc = -1;
break;
}
// Some more really complicated code.
if (giveup) {
rc = -2;
break;
}
// Yet more really complicated code.
if (error) {
rc = -3;
break;
}
} while (false);
printf(“%s[%d]: function1=%d\n”,
__FILE__, __LINE__, rc);
return rc;
}

The inner logic uses the break statement to drop out the bottom of the do while (false). No need for labels and goto statements. No need for maintaining and checking flags. And if the inner logic completes and the flow of control finds itself at the while (false), it simply drops through. No harm, no foul, no iteration.

You can imagine replacing the logging statement with a close() system call, a free() call, a delete operator, or anything else you need to make sure you do to clean up after yourself. I routinely use this construct to eliminate any possibility of resource leaks in code that uses temporarily allocated resources like file descriptors, sockets, and dynamically acquired memory. I also routinely use it to fix resource leaks in legacy code, which seem to be surprisingly common in my experience, although using wonderful tools like valgrind during unit testing have helped a lot in that regard.

Adherents of other languages both more modern and more ancient will recognize this control structure as something you might have known as a do-end. It would be great if C++, C, and Java had something similar, perhaps a way to use break to exit out the bottom of any compound statement (that is, a block of statements enclosed in { curly braces } ). Alas, a break can only occur in the context of a switch or loop construct. So in order to use break, we must provide the compiler with a loop construct, albeit one that never actually loops.

This pattern is applicable to any block of logic, not just functions. I frequently use it when I am writing a long sequence of data transformations or functions calls, all of which must succeed for the result to be useful. If any step in the sequence fails, it does a break to the end of the block. Refactoring fans will be pleased that the pattern can be used to refactor spaghetti code into something more readable. The Design By Contract crowd will like the fact that code written with a common exit can establish preconditions above the do and postconditions below the while (false). Formal Verification folks will like the idea of establishing invariants (assertions that remain true during execution) before and after the do while (false). And although I find the idea of proving any non-trivial piece of code correct pretty laughable, I do find the concept of invariants to be very powerful when reasoning about program correctness, and the use of do while (false) helps me greatly in that regard.

The use of the break statement is obviously not universally applicable. If you are using it from inside another looping control structure, including other do while (false) constructs, or from inside a switch statement, then it is not going to drop to the bottom of the outer do while (false).

Instead, if you are using an ancient language like C, which I have a lot of affection for, the same way I might have had for Latin, had I studied Latin in high school instead of goofing off in the computer lab, you could have accomplished the same thing using a goto. In fact, this application is one of the few in which I find the use of goto acceptable. The maintainers of the Linux kernel use this pattern routinely, and so do I as my clients frequently call upon me to hack the 2.4 and 2.6 kernels to support their newest bleeding edge hardware platform.

But if you are using Java or any other C-like language that doesn’t have a goto, or if like me you find the use of goto a slippery slope, or even perhaps it is too reminiscent of those thousands of lines of FORTRAN IV that you wrote decades ago, the memory of which you are desperately trying to suppress, then this is a useful technique.

The use of do while (false) to implement a common exit flow of control is merely good practice. There is another context in which it is absolutely necessary.

Compound Statements and Preprocessor Macros

My name is Chip, and I use the C preprocessor when writing in C++. As much as the C++ purists like inline functions (and truth be told so do I), there are situations in which they just don’t cut it. I have used preprocessor macros to do fun things like computing the largest signed two’s complement binary number of any basic data type. I’ve tried to write an inline function to do that, and I would be pleased to see the results of anyone who did so successfully without using the preprocessor. (I’ve done it with a templated function, but then it could not be used in C.) The C preprocessor is a powerful form of code reuse known as code generation, and like all powers, with it comes responsibility. It must be used only for good and never for evil.

So given that I’m going to use the C preprocessor whether the C++ crowd likes it or not, consider the following code snippet.

#define TRANSFORM(_A_, _B_) \
function1(_A_); \
function2(_B_)

Now consider its use in this context.

if (transformable)
TRANSFORM(x, y);

It’s not going to do the right thing, is it? The preprocessor will expand it thusly.

if (transformable)
function1(x);
function2(y);

This is clearly not what the user of the macro intended. You might be able to make up a lot of excuses for writing macros like the one above, but regardless, you have done something to surprise anyone that uses it. You have designed an abstraction that does not conform to the behavior any competent programmer would expect. You can argue that your coding standard requires { curly braces } around even single statements in if else blocks. This is not going to be helpful to your fellow developer who has to port ten thousand lines of third-party code, code which follows its own coding standard, and wants to use your macro to make their job easier.

The logical thing is to place the function calls in a compound block instead.

#define TRANSFORM(_A_, _B_) \
{ \
function1(_A_); \
function2(_B_); \
}

Then our code snippet will expand into something like this.

if (transformable)
{
function1(x);
function2(y);
};

Looks better at first glance, doesn’t it? Now both functions are part of the conditional. Yes, there is a dangling semicolon at the end which is functionally a null statement. But so far, so good.

So try this.

if (transformable)
TRANSFORM(x, y);
else
TRANSFORM(q, r);

Now our snippet expands to something like this.

if (transformable)
{
function1(x);
function2(y);
};
else
{
function1(q);
function2(r);
};

This will not compile. The semicolon trailing the first invocation of the TRANSFORM macro is actually a null statement, separate from the compound block preceding it. It becomes a statement in-between the if clause and the else clause. Using a semicolon following the macro invocation in the expected way leaves the else clause dangling.

The fact that this code does not compile is the good news. The programmer using your macro will merely think that you are incompetent, and will never use your macro, nor probably any code that you write, ever again.

A much worse case would be if the resulting code compiled, but did the wrong thing. I have tried very hard to find a code snippet which compiles but does the wrong thing. I have been unsuccessful. I’m not saying that such a code snippet does not exist, merely that I am not smart enough to find it. If such a snippet exists, then the programmer using your macro will think that you are incompetent while they sit with a baseball bat in the bushes next your house waiting for you to come home. If we were truly judged by a jury of our peers, it would be completely justifiable homicide.

A common approach to fixing this is to use the macro without a semi-colon at the end.

if (transformable)
TRANSFORM(x, y)
else
TRANSFORM(q, r)

This is an unsatisfying solution. You are requiring the user to write code in an unexpected and surprising way. Worse, the requirement to omit the semi-colon is merely an artifact of having to use a compound statement. If a thousand years from now the definition of your macro changes so that it is not a compound statement, then you must churn every single application of it to add the semi-colon. Or you have to put the semi-colon in the macro definition itself, which may cause all sorts of wackiness to ensue. Wouldn’t it be better to just make the macro work like any other C++ statement?

Like Lassie, do while (false) comes to our rescue. What is it, girl? The barn is on fire? Timmy fell into the well? We write our macro thusly.

#define TRANSFORM(_A_, _B_) \
do { \
function1(_A_); \
function2(_B_); \
} while (false)

The preprocessor now expands our macro into a single C++ statement that must be properly terminated by a semicolon. Hence

if (transformable)
TRANSFORM(x, y);
else
TRANSFORM(q, r);

becomes

if (transformable)
do {
function1(x);
function2(y);
} while (false);
else
do {
function1(q);
function2(r);
} while (false);

The semicolon, added by the user of the macro, is now a required part of the syntax, not a dangling null statement.

All of the snippets I have shown not only compile, but do the expected thing when executed. The do while (false) control structure serves as a compound statement that is both syntactically and semantically well behaved.

One context in which do while (false) does not work is when you are using the preprocessor to generate code that declares variables.

#define ALLOCATE(_A_, _B_) \
do { \
int _A_; \
int _B_; \
} while (false)

The variables will be allocated on the stack then immediately deallocated when the do while (false) construct terminates. This is fine if the scope of the variables is limited to the code inside the do while (false). It is not so useful if they are being declared for use outside of that scope. The simple compound statement has the same flaw.

The Little Control Structure That Could

I hope I have given you a new appreciation of do while (false), the control structure that does so much, while generating so little.