Diminuto and the War Against Abstraction

I don't want to know how it works under the hood.
-- Former coworker and Java developer

Decades ago, back when software was programmed instead of developed, practitioners of the art were classified into two broad categories: application programmers and systems programmers. The former wrote programs that used databases and produced reports. The latter understood how everything actually worked under the hood and was able to fix it when it broke.

Today we just call all of them software developers. Most of them develop software that uses databases and produces reports (typically in HTML). And systems programmers, in the classical sense, appear to me to be few and far between.

Several years ago, in his blog Joel On Software, Joel Spolsky wrote about The Perils of JavaSchools (which I just re-read in his latest book More Joel on Software). Joel bemoaned the transition of many computer science curricula to exclusively teaching Java. CS programs want a simple programming language that allows them to teach important concepts without unnecessary pain. Joel's argument was that a little pain is a good thing: until you have to debug something where the only diagnostic you get is "Segmentation Fault", you aren't really going to understand what's going on under the hood.

Now don't get me wrong: I've made good money developing in Java, enjoyed it, and was glad to get the work. I helped develop a successful commercial telecommunications product whose firmware I wrote partly in Java. I paid my dues in the Enterprise Java wars by working on a team that developed a business process automation product based on the Java Business Integration specification. I've been known to sling some open source Java stuff out there too.

But even though I wrote a good portion of that firmware in Java, I wrote the underlying device drivers in C. And when that enterprise product got sideways, I had to dive in and figure out how the heck threads where being managed in the framework we were using. Even the relatively simple open source stuff required that I figure out how the JMX protocol interacted with internet firewalls.

For my typical daily work, much of which involves writing firmware for custom hardware that I've never seen before, I'd be glad to see "Segmentation Fault" on a display when my code fails. Hell, I'd be glad to see an LED change state. Sometimes all I notice is that things have gotten really quiet as the switching power supply gears down.

The software industry has seen immense growth since I started programming in 1970. The typical hardware platform for both the desktop and servers has undergone a brutal natural selection to where it seems like everything is a Pentium processor running either Linux or Windows, depending on your religion, nearly all user interfaces are now browser based, and even those that aren't are still graphical. Everything else was just too expensive, or too hard to use, and were winnowed out in a process that Darwin would have appreciated without understanding any of the technology. Even Apple was subject to this effect, moving to a version of UNIX and eventually into the house of Intel.

This evolution has served us well. It's why I can afford to have a quad-core server in the basement. And why Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) loves our DVR.

But, see, here's the thing: it only seems that way. In fact, the number of embedded processors which are not Pentium chips, frequently running an operating system other than Linux or Windows, has exploded as consumer devices like cell phones, internet tablets, PDAs, GPS receivers, MP3 players, and other hand-held devices have taken over our purses, briefcases, and messenger bags. Even more so is the introduction of microprocessors into our everyday lives in places where we aren't really aware of them, from our kitchen appliances to our vehicles to our office equipment.

Someone has to know how to develop software for these kinds of devices: weird-ass hardware that no one has seen before, stuff that has severe resource constraints in terms of CPU speed and memory footprint, complex interfaces to physical hardware, and form factors so small your hardware guy has problems getting a logic probe on to a pin to tell you what his hardware thinks it's saying to your software. You can't just grab a Java framework that you found via Google to solve these kinds of problems.

And with the advent of multi-core servers, and the breakage of simple consistent memory models on such systems, developers using C, C++, and even Java and C#, on white box Pentium servers now have to have a deeper understanding of threading models and how the hardware handles data coherency between processor cores. Simple errors like "Segmentation Fault" will be a fond memory. Failures will instead be of the nature of "it works most of the time" and "we can't reproduce it".

Yeah, I know I sound like a cranky old man: I remember when I had to walk three miles to the computer center, uphill both ways, through a blinding snow storm, in order to use a keypunch to write software in IBM 360 assembler. But to be a systems programmer, you must understand, to significant depth, what is going on under the hood, no matter whether you are writing in Java, C#, C++, C, or assembly language. You have to be both willing and able to tunnel through all the layers of code to figure out how something works. You must be fearless. You must be a foot soldier in the war against abstraction.

Alas, it will be an uphill battle. Because abstraction isn't just a good thing, it is a necessary thing. All human knowledge builds on what came before it. Engineering is the invention of abstractions and the layering of them on top of one another in the quest for conceptual simplicity. If, in order to have a new car, we each had to start by mining ore with our hands, we'd all still be walking.

But systems programmers must be able to operate on many levels of abstraction simultaneously. They must be able to appreciate that a Java framework supports inversion of control to connect several web services components, while at the same time understanding when the volatile keyword is necessary for the JVM to insert the necessary memory barrier machine instructions.

Believe it or not, universities used to teach systems programming. They didn't call it that, but that was the mindset that they were trying to instill. For me, it was CS431 (later CEG431), "Real-Time Software Design", which I took from Professor Robert Dixon, way back in 1976. In this course, students developed a tiny real-time operating system that supported semaphores and message passing, and then used it to develop a multi-threaded application with interrupt-driven memory-mapped device drivers. It was all written in assembler language. It had to run on an actual PDP-11 minicomputer with real devices. It had to run all of the devices at their rated speed. And to pass the course, you had to take a core dump of your running system and do a complete dump analysis to the satisfaction of the instructor.

CS431 was required for undergraduate computer science and computer engineering majors to graduate, and for graduate students in those majors to take upper division courses. It had a 50% attrition rate. I passed it. Barely. And, remarkably, when I went on to graduate school, I ended up teaching it. (And Bob Dixon became my thesis advisor and mentor.)

It was the best of courses. It was the worst of courses. For some students, it prevented them from ever completing their degree. For others, it was the first time in their academic careers they had ever been challenged, and they loved it. I was somewhere in the middle. It was a bitch of a course to pass, but since at the time I was working full-time in a job where my title actually was "systems programmer", I was compelled to understand it.

Looking back, I find that CS431 has informed a successful career on my part for the past 30+ years. No matter what I do for a living, from writing board support packages for custom hardware, to developing protocol stacks from reams of specifications, to designing network traffic management firmware, to writing Java frameworks, there's a little bit of CS431 in all of it.

If you were to teach such a course today, what would you use? The PDP-11 evolved into VAX which evolved into the Alpha, all of which died when the Digital Equipment Corporation merged with Compaq who then merged with Hewlett-Packard. What processor architecture would you choose? What hardware? What kind of software would you write for it? What language would you use? What kinds of projects could you assign?

Diminuto is my small, personal attempt to answer those questions. It is one megalomaniacal supervillain's vision for teaching a new generation of systems programmers. I'll be writing about it in the articles to come.