Widows and Orphans and Working from Home

Terrific - and terrifying - article from The Atlantic's "Work In Progress" blog by Dror Poleg, author of the book Rethinking Real Estate: the next crisis will start with empty office buildings.

The commercial real estate market - once so stable it was considered a widows and orphans investment - is changing radically. 25% of commercial real estate in large cities is empty, and that only counts the space whose leases have expired; it doesn't count leased space that isn't occupied, and is unlikely to be occupied again when the lease expires. Many real estate firms are "handing the keys to the bank" by defaulting on their loans.

https://www.theatlantic.com/ideas/archive/2023/06/commercial-real-estate-crisis-empty-offices/674310/

I've been thinking about this ever since the Spousal Unit and I attended the World Science Fiction Convention in Chicago in 2022, post pandemic. It was held downtown at the Hyatt Regency on Wacker Driver right next to the huge Illinois Center complex.

We've attended conventions in this very same venue many times. I was shocked to see how the pandemic and the work at home movement had changed it. Illinois Center is an office building complex that sits atop a vast underground environment linking many such complexes. When we've been there in the past, on working days, this environment was full of retail, food, and service shops, and people bustling through it. This last time, it was almost empty, with a lot of empty storefronts.

Standing in our hotel room and peering at the adjacent office building, on a working day, the I could see the window office space on several floors were empty; I saw one single office worker, looking at what appeared to be large blueprints or schematics.

The commercial real estate market underpins a lot of city tax revenue and investments including pension plans. The clock is ticking: according to Poleg, a third of all office leases expire by 2026.

Informed Delivery

 The U.S. Postal Service (USPS) has a feature they offer called Informed Delivery (ID). It's free to residential customers. I signed up for it. Most every day in which mail is delivered - around 7AM local time for me - you receive an email containing black & white scanned images of the paper mail you can expect to receive at the specified street address in the next few days. The USPS has to scan all paper mail. That's the only scalable way to sort and route it. They have to scan it whether I sign up for ID or not. Considering the volume of mail, the variations in address formats, and the support for even hand written envelopes, it's a remarkable technological achievement.

But now I wonder: who else can get this ID email for my address? Can law enforcement request it? Does it require a subpoena or a search warrant? Or is it considered public information, like the stuff in your trash bin waiting for pick up at your curb? Do the laws restricting domestic surveillance prevent the CIA or NSA from receiving it? What about the FBI or the DHS? Who else might have access to it? Can it be used to construct a vast network of implied communication, much as intelligence organizations do today with social media accounts?

Maybe this is how conspiracy theories get started.

Unintended Consequences of the Information Economy IV

Daniel Kim, CTO of Geosite, a company that provides geospatial tools, wrote an eye-opening essay in the national security blog War on the Rocks. In "Startups and the Defense Department's Compliance Labyrinth", he describes what companies have to go through to comply with the enormous, complex, and often redundant, conflicting, and changing requirements to deal with the U.S. federal government, and especially with the its Department of Defense. Total initial cost for Geosite: US$300,000. Compare this with the typical size of contract that start-ups in the U.S. government's Small Business Innovation Research (SBIR) program receive: about US$1,000,000; the cost of compliance could be more than a quarter of the entire budget.

Much of the overhead is in the realm of cybersecurity. No one can fault the DoD for requiring stringent security mechanisms. But it does place contracting with the DoD out of the scope of many small- to medium-sized companies. And even for large companies, it is an incentive for the business to seek revenue elsewhere where it is more easily made in the commercial sector.

Furthermore, the technical work necessary for compliance either takes time away from the core technical team in smaller organizations, or requires hiring (and paying) additional staff with the most hard to come by (and expensive) skill sets. As I am constantly reminded when I chat with a friend of mine who makes her living as a cybersecurity engineer, there is not a lot of overlap between the skill sets of folks that do the kinds of work I do and the folks that do the kinds of work she does. Kim cites a slew of standards, many from the National Institute of Standards and Technology, that document the processes and infrastructure necessary for compliance. Just being familiar with these tomes would be a significant effort.

I've mentioned before that my tiny one-man corporation has done its share of work over the years in the defense domain, but always as a sub-contractor to a far larger organization that provided all the infrastructure and process that was required to comply with the customers' requirements. Kim also mentions the U.S. Air Force's Platform One and its concept of a "Software Factory": a kind of pre-laid infrastructure surround that temporarily assimilates a start-up and provides it with a much simpler set of requirements. (If you have the kind of LinkedIn network that I have, you have already heard a lot about this.)

Alas, without the kind of support provided by these kinds of organizations, the DoD is not able to innovate in the same way, and at the same speed, as the commercial sector. Nor even easily take operational advantage of new and shiny technology that comes out of successful commercial start-ups. Which means, for the most part, it's another windfall for the handful of existing huge defense prime contractors.

Unintended Consequences of the Information Economy III

In "Unintended Consequences of the Information Economy II" (2021), I mentioned an Executive Order issued in May of this year (2021) by the Biden White House: "Executive Order on Improving the Nation's Cybersecurity". This morning over a leisurely Christmas Eve breakfast I read the fifty-four page EO from start to finish. (It's not that onerous; the way it is formatted on the White House web site, with very short lines, it was expeditious to print it two web pages per physical page.)

I'm tempted to say that I've never read an EO before, so I didn't know what to expect. I've read so much stuff over the past decades,  I can't say for certain. (I once found myself having to read U.S Department of State International Traffic in Arms Regulations in regards to a commercial product I was helping develop for the business aviation space.) But for sure, I didn't expect the EO to be as interesting (or as readable) as it was. Here are some of the things I consider highlights:

• There was a a lot of verbiage about the need for federal agencies to share information with one another on malicious cyber campaigns. This may seem obvious, but in many organizations - government and commercial - this might be a hard sell. There is little incentive for organizations to admit they've been hacked, for lots of reasons. So an EO which requires federal agencies to do so is probably a good thing.

• There was also a lot of talk about the need for vendors who sell into the federal government space, especially cloud service providers (CSP), to share information on malicious cyber campaigns. Good idea, especially since they may detect such campaigns in their commercial environments as well. Same disincentives apply as above.

• The EO requires federal agencies using cloud services to adopt Zero Trust Architecture "as practicable". Also a good idea, and (if I am to be honest) something I haven't been that great about in some of my own work. I need to do better, and so does the U.S. Government.

• There is a section on the need for vendors that sell into the federal space to use a secure software development environment. It recommended such actions as administratively separate build environments, and automated tools to demonstrate conformance to secure development processes. There was a lot of text about the need for products to provide a verifiable software bill of materials (SBOM). Keep in mind this was written back in May of this year, months before the Log4J vulnerability came to light. If software products in use today that depend on Log4J had such an SBOM, a lot of IT folks might be having a much more relaxed Christmas holiday.

• Really an eye opener for me: there is a section proposing a cybersecurity labeling standard for consumer internet-connected (i.e. Internet of Things) devices. I very much look forward to seeing what happens with this. As a cybersecurity engineer friend of mine quips: "the 'S' in 'IoT' stands for 'Security'". I confess to still be grappling with security concerns - and the tradeoff between security and real-time behavior - in my own IoT projects.

• Most unexpected: a long section explaining the use and value of system log files. Preaching to the choir, President Biden. It requires federal agencies and their IT service providers to save such logs in a secure manner, and to periodically verify the logs against hashes to insure they haven't been modified. Such logs are crucial for post-hoc analysis of lots of stuff, not just malicious cyber incidents.

• The EO contains the usual caveats and exceptions for the Department of Defense and the Intelligence Community, but at the same time requires cooperation and leadership across the DoD and IC for this effort.

• Several requirements mention the use of automated software tools to audit, verify, and analyze the security of government IT systems. This is going to attract a lot of tool vendor attention. It will be interesting to see how those vendors address this, since the tools themselves - which are likely to be large and complex - will also be recursively subject to these same requirements. (I find myself reminiscing about the Gödel Incompleteness Theorems from graduate school.)

I still think the major impact of this will be more revenue for the small handful of ginormous prime contractors in the defense space. But I'd like to believe it's a step in the right direction.

Unintended Consequences of the Information Economy II

In "Unintended Consequences of the Information Economy" (2014) I cited an article in Foreign Affairs (2014), the journal of the Council on Foreign Relations, by former Deputy Secretary of Defense William Lynn. He talked in part about how companies in the technology sector typically invest far more of their revenue in research and development than do the handful of prime defense contractors in the United States.

What I didn't mention is how we arrived at the situation we currently find ourselves in: with just handful of big prime defense contractors.

In 1993, during the Clinton administration, then-Deputy Secretary of Defense William Perry convened defense industry executives into a meeting that came to be called "The Last Supper". He informed them that due to a huge reduction in the defense budget, there would have to be a consolidation of the defense industry.

John Mintz wrote about this and its consequences in "How a Dinner Led To A Feeding Frenzy" in the Washington Post (1997).

Perry's warnings helped set off one of the fastest transformations of any modern U.S. industry, as about a dozen leading American military contractors folded into only four. And soon it's likely only three will remain, with Lockheed Martin Corp.'s announcement yesterday that it plans to buy Northrop Grumman Corp. for $11.6 billion.

The unintended side-effect of the consolidation of the defense industry into just a handful of prime contractors was that there is now far less competition in the defense sector. If the Pentagon wants to buy a new major weapons system, there may only be a single contractor capable of delivering it.

Since this happened, the U.S. Departments of Defense and Justice and the Federal Trade Commission have tried to reverse this process by opposing further mergers in the defense industry.

John Deutch, also a former Deputy Secretary of Defense as well as Director of Central Intelligence, argued, in "Consolidation of the U.S. Defense Industrial Base", published by the Defense Acquisition University's Acquisition Review Quarterly (2001), that the consolidation also led to far less stability in the companies that did survive this process

In the 1993–1998 period of euphoria, defense companies experienced significant increases in equity prices based on the expectation of revenue growth and margin improvement from cost savings. In 1998, the outlook for the industry began to darken for several reasons. First, DoD reversed the consolidation policy. Second, expected cost savings were not shared with the companies, and hence margins were squeezed, especially from increasing interest payments on debt required to fund acquisitions. Third, defense companies making acquisitions were overly optimistic about the expected growth in top-line revenues from DoD, foreign military sales, and commercial spin-offs of defense technology. The anticipated increase in defense outlays had not materialized.

Finally, some key companies found it difficult to manage their expanded enterprises effectively in all respects and to meet their optimistic financial targets. The capital markets quickly shifted to more glamorous (at that time) dot.com and high-tech stocks not associated with defense.

My tiny one-man corporation has done its share of government contracting over the years, but always as a subcontractor to a far larger organization that had all the infrastructure, people, and processes in place to deal with the federal bureaucracy. The overhead involved is a significant barrier to entry for smaller organizations. And, remarkably, to larger organizations.

In my original article cited above, I related the story of Boston Dynamics, the spin-off of MIT that designs the human- and dog-shaped robots we all watch on YouTube. After the Defense Advanced Research Projects Agency (DARPA) poured a bunch of funding into the company, it was bought by Google in 2013, who basically said “thanks, but no thanks” to any further DoD involvement. Google went on to sell Boston Dynamics to a Japanese company, which in turn sold it to a South Korean company. All that government funding resulted in intellectual property that didn’t even stay in the United States, much less in the DoD. Many large tech firms have big revenue streams that for the most part don’t rely on the U.S. government; there are easier ways to make (lots more) money in the commercial sector.

This leaves much of the technology needs of the U.S. government in the hands of just a few big prime defense contractors.

In a recent edition of his newsletter "The Embedded Muse", embedded software and hardware technology pundit Jack Ganssle mentions that the Biden White House has issued an "Executive Order on Improving the Nation's Cybersecurity". Jack writes:

Uh oh. Do you sell to the US government? Since they buy pretty much everything, pretty much everyone does. A new executive order re security will make our lives much, much harder. Though the details are still being fleshed out, a pretty good overview here will raise your blood pressure. 

Jack references an article by a vendor who is, of course, trying to sell you something, but is none the less a pretty good overview of the EO. From that sales pitch:

This EO directs these agencies to develop new security requirements for software vendors selling into the U.S. government. These requirements will be incorporated into federal contracts for commercial software and hardware with the intent of imposing “more rigorous and predictable mechanisms for ensuring that products function securely, and as intended.”  This is a monumental shift that will have an immediate impact on global software development processes and lifecycles.

In addition to  a host of new information and operational security measures that government agencies need to implement, the new order establishes a robust approach to supply chain security. The new requirements will include security testing throughout the development process as well as a Software Bill of Materials (SBOM) to address security issues in open source components.

I expect this EO to be a huge boon for the few existing big prime defense contractors, while preventing the small-to-medium, and even large, tech companies from participating in providing innovative technology solutions to the federal government.

As both a software engineer and a taxpayer with many decades of experience doing both, I have very mixed feelings about this. I can appreciate the need to make sure that our tax dollars aren’t wasted, that expenditures are all accounted for, and that the products our government purchases are reliable and secure. But I feel pretty confident in predicting that it will mostly mean that giants like Raytheon and Lockheed-Martin will do well.

Small Talk

Despite the fact that I rate off the scale on introversion on every personality test I've ever taken, I am skeptical about the the Work From Home and Remote Learning trends. As necessary as they are during these Plague Times, I don't think we have any idea of the long term effects on our corporate culture, our productivity, our innovation, or on our and our children's emotional maturity and communications skills. We are a species that evolved to live and work in tribes.

This morning I read two articles that I liked on this topic.

In "What a Year of WFH Has Done to Our Relationships at Work" (Baym et al., HBR, 2021-03-22), three Microsoft researchers analyze ginormous datasets to determine how communication between team members have changed. Hint: they've become much more siloed.

In "I've lost my conversational mojo - can I relearn the art of small talk?" (Samadder, The Guardian, 2021-04-17), actor and columnist Rhik Samadder talks - socially distantly - to experts in the art of small talk, including to the neuroscientist for which the "Dunbar Number" is named. Upshot: the experts think we'll adjust back to being social again more readily than we might think.

I am concerned about limiting our interpersonal communications to those channels moderated by technology. It's not just about the loss of body language and other social cues. Or the lack of the productive backchannel that is the spontaneous hallway conversation. Or the magic that happens when you get a few smart people in the same room in front of a whiteboard.

It's also about the way in which I turned from being a socially awkward introvert to being - I am told, anyway - skilled at collaborating, teaching, public speaking, and giving executive briefings (while still being a little socially awkward). Being a strong introvert - some might even say a nerd - those skills didn't come naturally to me; they were learned through years of hard-won, and sometimes even painful, experience. Experience that we are eliminating in the WFH and RL world.

I hear a lot of folks talk about how much they prefer WFH and RL. And a lot of those preferences are totally legitimate. But for some of those folks, the strong introverts, I fear that they prefer it because it gives them an excuse not to grow and learn and exercise important social skills.

I also fear this is just another facet of late-stage capitalism, in the form of another cost - the cost of providing adequate infrastructure - that corporations can push off onto their employees, while the resulting hidden expense to the organization remains largely unaccounted for.

I don't think we appreciate what we're giving up. And we have little understanding for the possible long term consequences.

Update (2021-04-22)

Laurie Kenley, a cloud security wrangler, made these insightful remarks in another forum, and kindly gave me permission to reproduce them here verbatim.

I am off the scales in extroversion and I have been working from home for 2 1/2 years. I had to re-create my life around the fact that I would not be getting social interaction in my job well before the pandemic. I had it all figured out. And when the plague hit, all of that collapsed. It took a lot of figuring out how to be OK. Not great but OK.

One thing I will say that has been “better“ during the pandemic as a remote worker, is that everyone is remote now and it’s a level playing field. I have been able to be more effective and more plugged in to my coworkers than ever before in this remote job. To use a Hamilton reference, "the room where it happens" is now a virtual room. It has democratized decision making in a way I don't necessarily expect to last in the coming months.

Anticipation

You may have read by now that the online vaccine management system that the Trump administration paid US$44M for from the consulting firm Deloitte in a no-bid contract is a disaster. Riddled with bugs like making and then randomly cancelling appointments. States are giving up and cobbling together their own solutions.

Cat Ferguson, "What went wrong with America’s $44 million vaccine data system?", MIT Technology Review, 2021-01-30

About half my career has been spent doing work funded by the U.S. government, either in Big Science or in the Military and Intelligence sector. What I found the most interesting about this mess is one of the reasons Deloitte got the non-competitive contract in the first place. Did they have any experience in large scalable distributed medical computer systems? No. Infectious diseases? Nope. Vaccination programs? Not even. What they had experience in is government contracting. And the procurement systems implemented by the feds make it difficult - not to mention less profitable, compared to other work in the commercial sector - for anyone who isn't already doing government contracting to do government contracting.

I'm reminded of when Google bought Boston Dynamics. The robot company had been mostly funded by DARPA, the U.S. DoD's advanced research arm, the folks that have a track record of bringing us things like the Internet. Once Google bought Boston Dynamics, they declined to renew the DARPA contract. The DoD-related industry was surprised; I read articles in the DoD trade and national security press about this at the time. But I doubt many in the high tech industry even batted an eye. Selling to the Feds used to be a big thing. Now, not so much, compared to the global market. (Google eventually sold the company to Hyundai. Yet another bleeding edge albeit immature technology transferred to a foreign company.)

Another issue that occurs to me is how long it takes to design, develop, test, and deploy a product as big and complex as the proposed vaccine management system. Contrary, I suspect, to expectations in the White House at the time, a commercial effort of this type would take a year or two, all told. Regardless of the sales pitch from any consulting company, there is no fast path. That's why you have to anticipate this kind of need, and address it well in advance of it becoming a necessity. It's like insurance: you have to buy it before you need it, with the understanding - and the hope - that you never actually have to use it.

I don't know how, but we need to do better.

Layers

 "Either you design a layered architecture, or you'll wish you had."
-- Ken Howard, Ph.D., Bell Labs

When Colorado got hit recently with an early winter storm bringing several inches of snow and 10ºF temperatures before Halloween had even arrived, I found myself wearing four layers of clothing while shoveling my driveway. It gave me time to think about the software layers that developed organically in Diminuto,  my open source C library and framework that has been the basis of almost all of my personal projects, and some or all of which has found its way into several shipping products. It will become obvious I'm using the term layers here a little differently than you might have seen before when referring to software architecture or network protocol stacks.

History

My work on Diminuto ("tiny") started in earnest in early 2008, with portions of it borrowed from earlier work I did in C++ in 2005, and based on even older C code I wrote in 1994. I began with a physically large and relatively expensive Atmel development board with an ARMv4 processor, a chip that seems laughably underpowered compared to the current ARMv7 and ARMv8 offerings that are in your mobile phone. This board predated the Raspberry Pi (2012), and even the BeagleBoard (2008). Having spent years writing C and C++ code for one real-time operating system (RTOS) or another like VxWorks, I was experimenting with building a minimalist Linux-based system that could run comfortably on the (then) memory-limited ARM processor. My testbed consisted of:

• a custom Linux kernel configured with only those features I really needed;
• uClibc, a stripped down C library;
• BusyBox, that combines a bunch of commonly used Linux/GNU utilities into a single binary executable; and
• my C-based application.

(You can click on any image to see a larger version.)

Even though I was shooting for a tidy KonMari-style system that would make Marie Kondo proud, I still found it useful to organize some of my code into a library and set of header files. Even then it had occurred to me that I might reuse some of this stuff in another context one day.

Diminuto, and my work that used it, eventually migrated to more capable platforms, like the BeagleBoard.

And then to the first Raspberry Pi.

Eventually I found myself using later, even more powerful Raspberry Pis, and that Single Board Computer (SBC), with Diminuto, running on its Debian-based Linux/GNU distro, became my standard deployment platform for a lot of later projects, like Roundhouse, my IPv6 testbed.

I occasionally found myself porting Diminuto to more powerful platforms, like the Nvidia Jetson TK1 development board.

Diminuto played a key role in even more exotic projects, like building Astrolabe, my own stratum-0 NTP server using a chip-scale cesium atomic clock (CSAC).

Over the span of twelve years, Diminuto continued to grow organically in a kind of positive feedback loop, as the C library and framework became more useful than my quest for a tiny embedded Linux system. I would work a gig to develop some commercial product, and encounter a software feature in either the Linux kernel or the GNU library that I thought was useful - like the run-time loadable user-space modules that are used by the Asterisk open source PBX - and I would fold support for that feature, along with unit and functional tests and sometimes command line tools, into Diminuto. Or I would use some or all of Diminuto in some commercial development, and find some things in my code that didn't work so well, so I would fix it. And I would continue to leverage that growing base of reusable code in my own personal work.

Today Diminuto amounts to over 56,000 lines of C and bash code distributed amongst almost 350 files, implementing 84 different features and 38 different command line utilities, with 97 unit test programs and 28 functional test programs. It forms the underpinnings for almost all of my Linux/GNU-based personal projects in C or C++, like Hazer, my project that explores technologies like Differential Global Navigation Satellite Systems (DGNSS) and Inertial Measurement Units (IMU).

(To give you another way to think about this evolution, decades ago I started out using the Source Code Control System (SCCS) to manage my source code changes to the precursors to Diminuto. For Diminuto, I started out using the Revision Control System (RCS). Then I migrated to Concurrent Versions System (CVS). Then to Subversion (SVN). And finally Git, with the repository hosted on GitHub. There are pieces of Diminuto that came from that earliest work that have been managed by five different code management systems.)

Different people learn in different ways. I envy my friends and colleagues who can pick up a book on some new technology, read it, and claim to have learned something. (Liars.) I can only learn by doing. I'm a very hands-on student. (Which is why I also have git repositories of C++, Java, Python, JavaScript, Go, and Rust code.)

Working on Diminuto has been fundamental to my ability to learn not just new (to me) and evolving C, Linux, GNU, and POSIX features, but also to experiment with Application Program Interface (API) design. I've learned a lot about porting Diminuto between processor architectures (I use it almost daily on ARMv7, ARMv8, and Intel x86_64 machines), and between different versions of Linux and GNU, and even different operating systems and C libraries; in the past I've ported all or parts of it to uClibc, Cygwin, Android/Bionic, and MacOS/Darwin. It has given me a way to write useful code once, test it throughly, and then reuse it, either in my own projects or in products I've helped my clients develop. As much as I like being paid by the hour, there are more fulfilling things to do than to write the same underlying code over and over again.

Even though I have at times ported Diminuto to other platforms, it is not portable, nor is it intended to be. It is specific to Linux and GNU, and to the kinds of work that I am routinely called upon to do. I regression test changes only against relatively recent versions of the kernel and the C library.

As Diminuto evolved incrementally in its virtuous cycle, without really much of a long-term plan on my part, I none the less noticed that it had developed a layered architecture, driven by the needs of whatever I was using it for at the time. Maybe this was an accident, or maybe it's my forty-plus years of experience talking. But in any case, as my old friend and colleague at Bell Labs, Dr. Ken Howard, would surely remark, a layered architecture is a good thing.

Layers

Layer 0: Documentation

Like most software developers, I'm more interested in writing code than I am in writing documentation. But like most software developers of a certain level of experience, I have come to realize that I have to document my work, if not for others, then at least for myself.  Plus, documentation is its own reward: while writing comments I sometimes think: "Hey, wait a minute. That doesn't make any sense at all." It's like a one-person code review. Also, like other developers of a certain age, I've looked at code I wrote months or years ago and wondered: "What the heck does this even do? And WHY?"

Documentation for Diminuto (and for most of my other projects) comes in three flavors:

• blog articles like the one you are reading now;
• Wiki-style documentation like that in the Diminuto README; and
• Doxygen comments embedded directly in the code.

Doxygen is a software tool that takes your source code containing comments written in Doxygen's markup language, and produces (with a little help from other tools) documentation in the form of HTML pages, traditional UNIX manual pages, and PDF files.

Doxygen supports comments for files, functions, types, structures, variables, constants, preprocessor symbols, pretty much anything you can do in C. (Doxygen supports other languages as well.) The PDF reference manual for Diminuto currently runs to 784 pages! Of course, the resulting documentation is only as good as the Doxygen comments from which it is generated.

My approach is to document each Diminuto feature with Doxygen comments in its .h header file - the part of Diminuto that is publicly accessible even in a binary installation - which defines the API for that feature. Doxygen comments may also appear in .c translation units to describe functions that are not part of the public API.

Even if I never refer to the generated documentation, Doxygen provides a canonical comment format for documenting my code. I routinely use the Doxygen comments to remind myself how my own code works.

Layer 1: Types

As much as possible I use the standard pre-defined data types provided by GNU and Linux, like the following:

• size_t and ssize_t defined in <stddef.h>;
• fixed-width integer types like int32_t and uint64_t defined in <stdint.h>;
• bool defined in <stdbool.h>; and
• pid_t in <sys/types.h>.

But it quickly became clear that it would be a good thing to have some standard Diminuto types that could be used broadly across all feature implementations. The header file

"com/diag/diminuto/diminuto_types.h"

not only automatically includes the standard header files cited above, but also defines some types that are used in Diminuto, like:

• diminuto_ticks_t that holds the common Diminuto unit of time;
• diminuto_ipv4_t that holds an IP version 4 address;
• diminuto_ipv6_t that holds an IP version 6 address; and
• diminuto_port_t that holds an IP port number.

All the higher layers are based on these types.

(Why the long #include path above for the Diminuto header file? It's to make integration into other code bases easier. See Some Stuff That Has Worked For Me In C and C++.)

Layer 2: Logging

I wanted to write reusable C code that could be part of a command line tool, part of a system daemon, or even part of a loadable module that runs in kernel space. To do that, I had to abstract away the specific mechanism used to display error messages. The Diminuto logging API defined in the header file

"com/diag/diminuto/diminuto_log.h"

does that. It borrows its design from

• the standard syslog(3) interface;
• from the Linux kernel's printk interface;
• from Android/Bionic's logging mechanism; and
• even from non-C logging facilities like those I've used in Java.

Diminuto defines eight log levels, from highest to lowest severity:

• emergency;
• alert;
• critical;
• error;
• warning;
• notice;
• information; and
• debug.

Whether log messages are displayed or not is determined by an eight-bit log mask. The default value of the log mask is 0xfc, which enables all log messages except information and debug. The log mask can be set programmatically or from an environmental variable. As a special case, if the environmental variable, which is called COM_DIAG_DIMINUTO_LOG_MASK, has the value "~0", all log messages are enabled.

So far, this sounds pretty mundane. Here's the interesting part.

• If the log function call is compiled as part of a kernel module, the Diminuto log functions automatically revert to using the kernel's printk logging mechanism; each severity levels is translated to the appropriate printk level; and their emission is controlled as usual by the value of /proc/sys/kernel/printk.
• If the calling process is one whose parent is the init process a.k.a. process id 1 (this typically means the child process is a daemon that has been inherited by init), or if that process is the session leader (which typically means it has no controlling terminal on which to display a log message), the log function routes the formatted message to the standard syslog(3) API; its severity level is translated to the appropriate syslog(3) severity; and its emission can be controlled via the standard setlogmask(3). For most systems, this means the message will be passed to the syslog daemon (however that may be implemented) and written to a file such as /var/log/syslog.
• In all other cases,  the formatted message is written to standard error.

This vastly simplified the handling of log messages. It made Diminuto code much more easily reusable in a broad variety of contexts. The same function, using Diminuto logging, can be used in a command line utility or in a daemon; either way, output messages (in particular, error messages) will be handled appropriately. Some functions can even be used in the context of a kernel-space device driver. But more importantly, it helps insure that error messages - and, as we will see below, a lot of useful context for debugging - are not lost. Furthermore, the emission of each Diminuto log message is serialized under the hood by a POSIX mutex, insuring that messages from different threads in the same process don't step on each other in the error stream.

Layer 3: Errors

The standard function perror(3), defined in <stdio.h>, prints an caller-defined error message on standard error along with a system-defined error message associated with  the global variable errno, which is typically set by failing system calls and C library functions. For example, the code snippet

errno = EINVAL;

perror("framistat");

where EINVAL is an integer constant defined in <errno.h>, displays the message

framistat: Invalid argument

on standard error.

The code snippet

errno = EINVAL;

diminuto_perror("framistat");

on the other hand displays

2020-10-27T19:28:02.889470Z "nickel" <EROR> [25416] {7f981766e740} foo.c@3: framistat: "Invalid argument" (22)

on standard error, or logs it to the system log, as appropriate. (It is displayed as one line; the rendering here folds it up.)

Because diminuto_perror goes through the normal Diminuto logging mechanism, the message includes the date and time in UTC with microsecond resolution, the host name of the computer, the severity level, the process identifier of the caller, the thread identifier of the caller, the name of the translation unit, the source line within the translation unit, and even the numeric value of the error. 

The diminuto_perror API is defined in the header file

"com/diag/diminuto/diminuto_log.h"

and is part of the Diminuto log feature.

Layer 4: Assertions

The standard assert(3) function, defined in <assert.h>, evaluates a caller-specified scaler expression, and if the result of that expression is false, it displays an error message on standard error and ends the execution of the caller using abort(3) defined in <stdlib.h>.

foo: foo.c:2: main: Assertion `1 == 2' failed.

Aborted (core dumped)

The diminuto_assert API that is defined in the header file

"com/diag/diminuto/diminuto_assert.h"

does all that too, but it emits the error message using the Diminuto log feature, so all that logging goodness applies here too.

2020-10-27T19:48:15.562081Z "nickel" <EROR> [25564] {7f5ed5040740} foo.c@2: diminuto_assert(1 == 2) FAILED! 0=""

Aborted (core dumped)

(It also prints the value of errno, just in case it's useful to the developer debugging this failure.)

This is a lot more useful than it might first appear. Truth is, most serious errors - like error returns from systems calls - hardly ever happen, and if they do, they aren't typically realistically recoverable. A bunch of "error recovery" code often just serves to clutter up the source code, complicate testing and debugging, and add to the cognitive load of the developer. Most of the time, the best thing to do is to dump core and exit, and leave the recovery to higher minds - like the user sitting at the terminal, or maybe systemd. Using diminuto_assert can really clean up your code by reducing both error handling and error messaging to one function call.

Here's a tiny code snippet from the gpstool utility in my Hazer GPS project, which is built on top of Diminuto.

If the standard malloc(3) function fails to allocate memory, the entire program just goes toes up. Not a lot of drama, just an error message and a core dump, without a lot of other error handling code cluttering up the translation unit.

Layer 5: Unit Tests

I've used a lot of different unit test frameworks over the years for C, C++, Java, and Python development. The Diminuto unit tests didn't need a lot of complex support for mocking and what not. I wanted something dirt simple but functional. Most of all, I didn't want users of Diminuto to have to install an entirely different package to support the unit tests. So an almost trivial unit test frame became part of Diminuto itself. (And before you ask: yes, there is a unit test in Diminuto for the Diminuto unit test framework.)

Here is a code snippet that is one simple unit test from the suite that exercises the Diminuto InterProcess Communication (IPC) feature.

The unit test framework is nothing more than a series of C preprocessor macros. Here you can see that a new test is declared, a mixture of assertions (where failures are fatal) and expectations (where failures are logged and counted) are made, and the outcome of the test is displayed. If the unit test succeeds, the following is displayed.

2020-10-27T20:27:08.354153Z "nickel" <NOTE> [25741] {7fbabe66a000} tst/unittest-ipc.c@237: TEST: test=8

2020-10-27T20:27:08.354193Z "nickel" <NOTE> [25741] {7fbabe66a000} tst/unittest-ipc.c@246: STATUS: test=8 errors=0 total=0 SUCCESS.

The unit test framework API is defined in the header file

"com/diag/diminuto/diminuto_unittest.h"

and makes use of the Diminuto log feature.

Layer 6: Wrappers

Most of the features in Diminuto have fairly complex implementations. Some don't. Some are not much more than wrappers around system calls and C library calls. But for something that is used often, a little wrapping can lead to fewer surprises (contrary to the usual case with wrapping).

Consider the code snippet below, from the Hazer gpstool utility. 

You can see if the standard fopen(3) call fails, the error leg calls both diminuto_perror, which prints the name of the file that failed to open and also the error string corresponding to the errno set by fopen(3), and diminuto_assert, which core dumps the application.

Compare that to this code snippet, also from the Hazer gpstool utility,

in which only diminuto_assert is called.

Why the difference? Because the failing function in the second example is itself part of Diminuto. It already called diminuto_perror before it returned to the application. Even just having simple wrappers around frequently used system calls and library calls that automatically print an error message - using the smart Diminuto log-based features - reduces the amount of clutter in the source code of the application.

Of course, there maybe circumstances which the application should to print its own error message with more specific contextual information. But most of the time, the error emitted by the Diminuto code is sufficient.

Layer 7: Models

Why do some Diminuto features have complex implementations? Because they implement a specific model or design pattern of using a particular system call or C library facility. That model may have complex behavior. Typically the model is based on forty-plus years of hard won experience on my part on doing exactly this kind of thing. Or sometimes just on my opinion of the right way to do things. Or on horrifically tragic mistakes I've made in the past.

Here's an example that I already talked about in Timer Threads. Diminuto has a timer feature whose API is defined in

"com/diag/diminuto/diminuto_timers.h"

and which is based on POSIX timers. There is a potential race condition in POSIX timers between the application that starts (arms in POSIX-speak) and stops (disarms) a timer and the callback function that is invoked by the timer. POSIX timer callbacks run in a thread-like context - which is just a man page way of saying it is in fact a POSIX thread, separate from the application thread. When the application stops the timer, the callback may already be running, especially with a periodic timer, and on a multi-core (which these days is virtually any) processor. If the application releases resources - for example,  frees memory, or closes files - that it shares with the timer callback function, that function can have those resources pulled out from under it. Wackiness ensues.

Diminuto incorporates a POSIX mutex and condition in every Diminuto timer. When the application calls the Diminuto function to stop the timer, the function calls the POSIX function to stop the timer (not shown below), enters a critical section, indicates to the timer callback that it is placing it in a disarmed state, then waits for the timer callback to acknowledge it.

When the Diminuto timer callback (a private proxy function that invokes the user-specified callback) has finished its current business, it enters a critical section, checks to see if it has been disarmed, and if so, acknowledges that fact and signals the waiting application.

This prevents the race condition. This is some fairly complicated stuff, and it's all transparent to the application.

What if you don't want to use that model? Fine. Don't use Diminuto in that particular instance. There's nothing keeping you from coding to the native Linux or GNU API. Hey, knock yourself out. You can still make use of the Diminuto log facility. You can make your own horrifically tragic mistakes. In these here parts, we call that a learning experience.

Layer 8: Functional Tests

Besides unit tests, Diminuto also includes a host of functional tests. These form the last line of defense against some boneheaded blunder on my part. They also provide, along with Diminuto's unit tests and command line tools, some really useful examples (I use them myself) of how to use Diminuto in the real world.

Some of the functional tests use special test fixtures I fabricated just for this purpose, to test hardware-centric features like General Purpose Input/Output (GPIO) pin input and output, serial port reading and writing, and pulse width modulation (PWM).

When I make a change to any of this hardware-related code, or its dependencies, or just when I'm feeling lonely, I get the text fixture out, hook everything up, and run some functional tests that do things like light up LEDs in response to button presses, or adjust the illumination of an LED to achieve some specific value of lux on a light sensor.

The functional tests and command line tools form the outermost layer of the Diminuto framework.

End Matter

Diminuto has been enormously useful to me (and, sometimes, to my clients) for a wide variety of reasons, not the least of which are the lessons I carried away from the layered architecture which emerged organically from its development.

If you want to see non-trivial examples of Diminuto in action, checkout the gpstool and rtktool applications that are part of my Hazer GPS/GNSS project. They each make good use of a variety of Diminuto features. gpstool, rtktool, and their Diminuto underpinnings, are in constant use, 24x7, as part of my Differential GNSS work.

Supporting Material

Chip Overclock, "Being Evidence-Based Using the sizeof Operator", 2015-03, <https://coverclock.blogspot.com/2015/03/being-evidence-based-using-sizeof.html>

Chip Overclock, "Buried Treasure", 2017-01, <https://coverclock.blogspot.com/2017/01/buried-treasure.html>

Chip Overclock, "Some Stuff That Has Worked For Me In C", 2017-04, <https://coverclock.blogspot.com/2017/04/some-stuff-that-has-worked-for-me-in-c.html>

Chip Overclock, "renameat2(2)", 2018-04, <https://coverclock.blogspot.com/2018/04/renameat22.html>

Chip Overclock, "When The Silicon Meets The Road", 2018-07, <https://coverclock.blogspot.com/2018/07/when-silicon-meets-road.html>

Chip Overclock, "When Learning By Doing Goes To Eleven", 2020-03, <https://coverclock.blogspot.com/2020/03/when-learning-by-doing-goes-to-eleven.html>

Chip Overclock, "Headless", 2020-06, <https://coverclock.blogspot.com/2020/06/headless.html>

Chip Overclock, "Clock Time", 2020-10, <https://coverclock.blogspot.com/2020/10/clock-time.html>

Chip Overclock, "Timer Threads", 2020-10, <https://coverclock.blogspot.com/2020/10/timer-threads.html>

Afterword (updated 2020-11-19)

This afterword got so long I promoted it to its own blog article: Is the Virtuous Cycle really that virtuous?

Process

I've been around long enough that I've worked under a lot of development processes: Waterfall, Agile, Kanban, etc. Over the decades I have made the following observations.

1. No one process is going to ensure success.
2. Some processes will work better than others for some kinds of work.
3. What process you use isn't nearly as important as the fact that you have a process at all, everyone knows what it is, and everyone follows it.
4. All development is fractally iterative, whether you want it to be or not.
5. No matter what process you use, you have to have the right people; no process will make the wrong people successful.
6. Engineers need structure, and successful ones know it; if you have the right people, but no process, those people will spontaneously adopt a process - an existing one if they're experienced - that works for them.

Habits

You’ve heard the maxim that it’s far far more expensive to acquire a new customer than keep an existing one? Part of this is habit. Humans are for the most part creatures of habit. To keep an existing customer, you just need to keep feeding their habit (e.g. a latte every morning at the local Starbucks drive through), and not give them a reason to change it (e.g. keep botching their order). But to get a new customer, you have to get them to change their habits. You have to convince them to do your thing in place of whatever thing they've been doing. That’s a big deal.

I wonder how many of us are going to have our habits rewired due to all of this Coronavirus craziness. Sure, I used to go to Starbucks around 0600 every morning and read for an hour or two. Maybe when all this is over, I realize I can save a lot of calories and money by just making a cup of coffee at home - like I’ve been doing for the past several weeks... or months.

Yeah, I used to go to the gym six times a week. But maybe I can save a lot of time and effort by staying with doing those core floor exercises on my mat in the living room three times a week, and taking a long walk outside for cardio the other three times.

Maybe I learn an important lesson: I don’t really need Starbucks four dollar lattes, or an expensive gym membership.

Could be a lot of places are going to find out a lot of their regulars aren’t coming back.

Other ideas:

Businesses are slow to rehire, acting conservatively in the face of an uncertain future (H/T to Demian Neidetcher).

Businesses find their old employees have made other plans, so they are forced to hire new people who must be retrained, preventing them from achieving their prior level of service, at least anytime soon.

Businesses don't survive this, forcing us to make other accommodations anyway (H/T to Kelly Dixon).

I think it's going to be a substantially different world at the far end of all of this; not just politically and medically, but professionally and personally.

Considerations for Ethical Decision Making

JJ Snow, U. S. Air Force CTO, passed this along: considerations for ethical decision making for the U.S. Special Operations Forces from the Joint Special Operations University. I think it is a lot more broadly applicable than just to the SOF community: business, politics, governance, academia, science, etc. I'd argue that these are important guidelines for anyone who manages people in today's organizational environments. Substitute manager for SOF operator and management for SOF and see what you think. I found it made for thought provoking reading. (Cut and pasted from LinkedIn.)

She writes:

What should guide us in tough times? Critical thinking, ethics, compassion and morality.  

Joint Special Operations University proposed six SOF ethical decision-making truths: 

1. Individual moral character is neither inherent nor fixed. Ethical decision-making requires continuing education for even the most experienced SOF operators.   

2. SOF operators will be morally challenged when they are least prepared to deal with it. Ethical problem-solving skills must be developed and strengthened.  

3. SOF ethical decision-making must be developed with honest and frank consideration for the harsh realities of SOF environments and operational requirements. SOF units must see the world for the way it is, not for how they might want it to be.  

4. Binary ethical codes do not provide sufficient guidance in SOF environments. In fact, strict adherence to binary ethical codes can be harmful in some SOF environments.  

5. SOF leaders should not be naïve or insensitive to human behavior and must recognize that people are not as ethical as they think they are. SOF operators need training to close the gap between the expectation and reality of what they must do.  

6. SOF culture must become an environment where conversations about ethical decisions, good and bad, are a natural occurrence.

One Prototype Is Worth A Dozen Meetings

When I founded my company in 1995, things were different. The legal and tax classification of limited liability company was brand new, and there was so little legal precedent concerning LLCs that the best legal and tax advice I could get was to stick with a subchapter-S corporation or S-corp. The World Wide Web existed, but barely, so none of the incorporation process was online. I had to register actual physical paperwork, my articles of incorporation, with the state of Colorado. It was the same when I applied to the Internal Revenue Service for an Employer Identification Number, which is like a Social Security Number except for companies instead of people. The process was complicated enough that I used a firm that specialized in dealing with all that bureaucracy to guide me through it.

And, most shockingly, Google didn't exist. And wouldn't exist for years. I outsourced my email handling to a local service provider to which I connected (and still do today) via arcane protocols like SMTP,  POP and IMAP. My desktop was an Apple Macintosh Classic II, and the Linux distro I ran on my Dell server was Yggdrasil.

The entire process, both the business side and the technical side, took weeks to get set up and running, culminating in the creation of the Digital Aggregates Corporation nineteen years ago this month.

Just the other day I had suggested to one of my long-time clients that it might be a good idea to create a different company under which to productize and market some of the awesome intellectual property they had created as collateral from one of their projects. The more I thought about it, the more I realized that I really had no idea the magnitude of what I was suggesting. But I was pretty sure that it would be a whole lot easier today than it was in 1995.

There was only one way to find out.

Here's the thing: I'm a very hands-on learner. For me to understand something, I have to actually do it, touch it, play with it. For sure, I want to understand the theory behind it, and build my own abstract mental model of it, but I'm not going to be able to internalize those things and make them a part of me until I have my hands on it.

And that includes code. Until I suck the code base into Eclipse, peruse it, build it, install it, run it, until I look at your schematics, and then lay my hands the actual hardware and apply power to it and see some LEDs blink, it's all too theoretical for me. By which I mean, I can't claim to have any expertise in it or knowledge about it.

I have thirteen projects hosted on GitHub -- and a few more that are not -- all of which are the result of me deciding something like "Well, I guess I'd better learn Python." Thousands of lines of working code later, I feel that I can claim with a straight face to know a little Python. For me, just reading a book is not going to cut it.

It is the same for me in business. That's why I created Digital Aggregates all those years ago. I had some vague notion that maybe it would be a way to earn a living someday, but mostly I did it just to see how it was done.

So I decided to create another company.

Disclaimer: I'm not a lawyer, nor a tax accountant.  But I know people who are, from whom I get good advice. Before setting off on a new business venture, do what I did when I first incorporated in 1995, and talk to some experts.

First, I got out a few of my favorite coffee table reference books, ones with beautiful color pictures of largely historical artifacts, and started picking out potential domain names. All the good ones are already taken, of course. I didn't want some high falutin' name like "Framistat Systems" or "Extremely Awesome Products". I wanted something short, one word. It would be nice if it were pronounceable, but it wasn't a deal breaker. And in order to have some business credibility, it had to be in the .com top level domain. I made a list.

Next, I went to my domain name registrar, Network Solutions, and starting searching. To no one's surprise, the very first one I tried, gladius.com (gladius being the Latin name for the short sword used to such good effect by the Roman soldiers of ancient times), was taken. By LucasFilm. Bet there's a story there. I only went through a handful before I found one on my list that was available. I immediately registered cranequin.com, plus some additional services, for an annual cost of around US$45.00.

(credit: Wikimedia)

Cranequin, if you mangle French like I do, is pronounced something like "krwa-ne-ke(n)" where that final nasal "n" is not even really said aloud. A cranequin is a mechanical winding apparatus used to draw back the bow on the kinds of crossbows you might use to take down large game. Or a man in armor.

(credit: Wikipedia)

The limited liability company now has a long history of success as the preferred organization for small businesses. And for a long time now, the State of Colorado allows you to file pretty much all your business paperwork online, something I have to do once a year or so, so I knew exactly where to go. It has become obvious to me over the years that both the State of Colorado and the U.S. government really want you to start a business, and they make it remarkably easy to do so. I established that Cranequin LLC was available as a business name in Colorado, so I submitted the electronic form to create it. After just a few minutes and US$50.00, I had my Colorado business identification number. In the eyes of the law,  Cranequin LLC is a wholly owned subsidiary of Digital Aggregates.

Then I went to the Internal Revenue Service web site and with some judicious searching figured out that I probably didn't really need a separate EIN, but it would probably make my tax accountant happy. So with the information from the State of Colorado web site in hand, I registered Cranequin LLC with the IRS and had a federal tax identification number for it. No charge.

Next stop was to Google to set up all the technical infrastructure like electronic mail in their cloud, for the cost of US$5.00 per user per month. There's an annual plan too, which saves you US$10.00 per user per year, but I decided to stick with the pay as you go plan. I was already familiar with the Google's Apps for Work from using the service at one of my clients. It seemed like the obvious choice for a tiny startup (and maybe now for large organizations as well). By far the hardest part of this step was having to laboriously edit the MX DNS records via the Network Solutions website to point electronic mail addressed to cranequin.com to Google, and then learning patience because it takes a while for everything to propagate across the interwebs.

I had already purchased web forwarding from Network Solutions. I hacked at the Digital Aggregates web site to create a really basic home page, nothing more really than a place holder. Then it was back to the Network Solutions website to set up the web forwarding, and www.cranequin.com was born.



All told it took about six hours total, spread across three days. I registered the domain name on Sunday, did the stuff with the State of Colorado, the Internal Revenue Service, and Google on Monday, and the home page and web forwarding on Tuesday. Everything was done online over a web interface. No paper, no signatures.

The up front costs were US$50.00, and the annual costs look to be about US$160.00 for domain services and (so far) two email users. If you do your own tax returns, that's probably it. I piggy backed Cranequin LLC on the same mailbox I rent for a mailing and shipping address for Digital Aggregates Corporation.

Will I keep my little prototype limited liability company? For a while at least. After all, when I created Digital Aggregates Corporation as an experiment, I really had no idea that years later it would become the principle mechanism though which I made a very good living. Maybe I'll have similar success with Cranequin LLC.

Unintended Consequences of the Information Economy

William Lynn, former Deputy Secretary of Defense in the Obama administration, writes in the journal Foreign Affairs on how the transition from a manufacturing economy to an information economy has affected the U.S. Department of Defense in "The End of the Military-Industrial Complex".

For more than a decade, U.S. defense companies have been lagging further and further behind large commercial companies in technology investment. Although the Pentagon historically exported many technologies to the commercial sector, it is now a net importer. Indeed, next-generation commercial technology has leapt far ahead of what the defense industry can produce in areas spanning 3-D printing, cloud computing, cybersecurity, nanotechnology, robotics, and more. In addition, commercial information technology dominates national security today as much as it does the private sector. Soldiers now use smartphones to gather real-time surveillance from drones and send messages to fellow soldiers. 

Keeping up with commercial innovations will be difficult, if not impossible. The combined R & D budgets of five of the largest U.S. defense contractors (about $4 billion, according to the research firm Capital Alpha Partners) amount to less than half of what companies such as Microsoft or Toyota spend on R & D in a single year. Taken together, these five U.S. defense titans do not even rank among the top 20 individual industrial investors worldwide. Instead of funding R & D, defense companies have been returning the overwhelming majority of their available cash to shareholders in the form of dividends and stock buybacks. As a result, from 2000 to 2012, company-funded R & D spending at the top U.S. defense firms dropped from 3.5 percent to roughly two percent of sales, according to Capital Alpha Partners. The leading commercial companies, by contrast, invest an average of eight percent of their revenue in R & D.

Lynn opens with the example of Google's purchase of Boston Dynamics, the robotics firm that designed, among other devices, the BigDog, the four-legged load carrying robot. BigDog was originally funded by the U. S. DoD. Google announced that while they would honor Boston Dynamics existing military commitments, they would not be seeking further work from the DoD. Google basically reached into their deep pockets and pulled the advanced robotic technology rug right out from under the Defense Department.

Lynn identifies several trends that may be ending the Military-Industrial Complex as we have come to know it. High tech companies are reticent to reveal what may be valuable intellectual property to the U. S. government. They don't see a reason to have to deal with the vast government procurement and contracting bureaucracy when more money can be made more easily in the commercial space. Commercial companies increasingly exploit globalization, manufacturing goods or building research facilities overseas where it makes economic sense, something the U.S. military is understandably reluctant to do for both national security and political reasons. Lynn talks about how the Defense Department is going to have to come to terms with these trends unless it wants to lose its technological advantage.

What Lynn doesn’t talk about (but probably knows): in the 21st century information economy, the key component to growth isn’t enormous capital investment in manufacturing capacity, something the military establishment used to good effect during World War II, but instead enormous people investment in innovation capacity.

Just yesterday National Public Radio ran a story on this very topic: "Future U. S. Manufacturing Jobs Will Require More Brain Than Brawn". Planet Money's Adam Davidson remarks on how this is affecting the world of work.

If you want to succeed for the coming decades, you don't just need to be trained and then a few years later retrained. You need a continuous improvement in your education. The main skill you need is the skill to learn more skills. The one certainty we have is manufacturing is going to look more and more like computer programming and engineering. It's going to involve a lot more brain work and a lot less brawn work. And that means probably a smaller number of people can benefit, but those who can benefit will probably benefit quite a bit.

The DoD can’t just dial up more innovation capacity by throwing money at the problem, like they did in WWII. Nor, in a free country, can the U. S. government just mandate for whom companies choose to work. Innovation capacity requires not only brilliant engineers, who are hard enough to come by, and who cannot be easily identified in the job market, but also a willingness to accept a lot of risk: to try and perhaps to fail, over and over. To old school 20th century managers, this looks a lot like waste, but in fact it’s a necessary part of the innovation process. The economics of conflict is changing just like the economics of everything else is changing. It’s as if one smart guy with a laptop, some open source software stacks, and a 3-D printer, can now manufacture hydrogen bombs.

The DoD has to start thinking more about leveraging not just globalization (like, as Lynn suggests, by buying German-made artillery), but also consumer technologies where there are enormous economies of scale (so they’re relatively cheap compared to specialized albeit low volume goods), not to mention more profitable than the DoD could make it for the contractor. The days of the DoD calling the shots in high-tech are over. Global market forces are going to make the decisions about who makes what and for whom.

My career, so far spanning four decades, has been distributed among academia and big science, defense contracting, and commercial high-tech product development. While the transition to the information economy has been very very good to me, I've been thinking about Lynn's article a lot lately, and what it means for my life, my colleagues, my clients, and my country.

The Very Big and the Very Small

In

K. Asanovic at al., The Landscape of Parallel Computing Research: A View from Berkeley, EECS Department, U. C. Berkeley, UCB/EECS 2006-183, December 2006

(a paper I've cited here before) its authors, which include David Patterson (as in Patterson and Hennesy), remark

Note that there is a tension between embedded and high performance computing, which surfaced in many of our discussions. We argue that these two ends of the computing spectrum have more in common looking forward than they did in the past. [Page 4]

That's been my experience too, although perhaps for different reasons than the authors cite. I’ve made an excellent living flipping back and forth between the high performance and embedded domains. It turns out the skill sets are mostly the same. In particular, developers in both domains are constantly concerned about very low level details in the realm where software runs close to bare metal, and are always dealing with issues of real-time, asynchronicity, parallelism, and concurrency. These are relatively rare skills that are hard to come by for both the employer and the employee.

I was reminded of this as my tiny one-man company, Digital Aggregates Corporation, buys its fourth Android tablet to use as a development system. These tablets contain powerful multi-core ARM-based processors as well as other embedded microcontrollers and devices. And increasingly I am seeing the embedded and mobile device domains adopt technologies originally developed for large-scale systems, like Security Enhanced Linux (SELinux) and OS-level containerization like Linux Containers (LXC).

I’ve seen large development organizations axe their firmware developers as the company decided to get out of the hardware business to focus on large multi-core server-side software applications. What a remarkable lack of insight into the nature of the technologies on which their businesses depend.

Observations on Product Development: Part 5

1. The Kübler-Ross Model (a.k.a. "The Five Stages of Grief": Denial, Anger, Bargaining, Depression, Acceptance) is also applicable to product development projects that are in trouble.
2. Build servers must have monotonically increasing clocks; otherwise builds using tools like make are not reliably reproducible.
3. Consider completely divorcing your product build system from whatever damn fool build system that generates your root file system; the latter is seldom optimized for the same things as the former should be.
4. If you want to eventually generate revenue, you must first optimize for developer productivity; everything else is negotiable.
5. If you don't believe #4, read your company's financial report and see where most of its money is being spent.
6. Don't build the Taj Mahal on a foundation of sand and loose gravel, no matter what your precious agile methodology says about customer value.
7. It almost never needs to be gold plated.

Richard Cook: How Complex Systems Fail

I reread Richard Cook's paper on How Complex Systems Fail this morning for maybe the third time. My only complaint is that the paper is a summary of a ton of safety research over the past several decades without a lot of references being cited. But it's a terrific executive summary, and it mentions the constant tension between safety and production pressure that is discussed at length in the work by safety researchers Jens Rasmussen and Sidney Dekker that I've cited.

http://www.ctlab.org/documents/How%20Complex%20Systems%20Fail.pdf

Cook's paper is just a few pages long and is worth your time. Cook is a medical doctor whose interest is specifically in patient safety, a topic that Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I frequently discuss.

Here's a video of Dr. Cook giving a talk on this topic. (If your device doesn't support Flash you can find the video on YouTube.)

Maybe the System Isn't the Solution - Maybe It's the Problem

I read a lot. It takes discipline; I set aside at least an hour every single day for reading. I read both fiction and non-fiction, but as I've gotten older I've definitely trended more towards non-fiction. Even the most speculative of fiction authors often have a hard time matching the amazing stuff I find in the news every day. I think it's one of the reasons that one of my favorite fiction authors, William Gibson, has transitioned from writing about the near future to writing about the recent past. You can't make this stuff up.

Much my non-fiction choices have been in the realms of economics, and how and why complex systems fail. My initial motivation for both of these topics was in fact professional interest. I'm a little surprised to discover that my recent choices in reading material tend to combine these two topics, a relation that has been slowly dawning on me over the past decade or so.

Charles Perrow, Normal Accidents: Living with High Risk Technologies, Princeton University Press, 1999

On June 1, 1974 in Flixborough England, a chemical plant that manufactured a component of nylon exploded. The blast was estimated to be equivalent to as much as 45 tons of TNT. The plant was destroyed. All of the maintenance records that might have provided more details as to the exact cause of the disaster were destroyed. Buildings a thousand feet away from the plant were damaged. Windows a mile and three quarters away were broken. Twenty-eight people were killed, and at least eighty-nine were injured. It is believed that a cloud of cyclohexane vapor, a byproduct of production, escaped and ignited, generated the immense fuel-air explosion.

This is one of the many catastrophic accidents that Perrow describes in detail in his book. I would be lying if I didn't admit to a certain amount of fascinated schadenfreude for some of the apocalyptic events he documents. Two common themes that emerge is the presence of non-linear dynamics and tight-coupling.

Perrow writes that that the exact functioning of many of the reactions used for production in large commercial chemical plants are not well understood theoretically, even by experts in the field. In an effort to make such plants more efficient, their production systems are full of positive and negative feedback loops which can (and do) react in non-linear ways, and which are only stable within a narrow range of operating conditions. Even the physics may not be well understood; Perrow talks about how neutron bombardment embrittlement of nuclear reactor vessels came as a surprise.

It's no coincidence that the phrase "going non-linear" is a euphemism for a situation going very wrong very quickly. In The Logic of Failure: Recognizing and Avoiding Error in Complex Situations (which I wrote about in Imperfect People Build Imperfect Systems) [1], psychologist Dietrich Dörner writes about how humans really suck at conceptualizing and understanding complex non-linear systems. This means when complex systems like chemical plants enter an unexpected region of their operational parameters, typically due to one or more failures or mistakes lining up in time, it is difficult for human operators to understand what is happening, or to even recognize that something is happening at all. My own experience suggests it is difficult for designers of digital control systems to anticipate such failures, particularly when the systems under their control suddenly and unexpectedly depart from their normal operational envelope. Perrow captures the essence of non-linear behavior when he writes that it is a sensitivity to "trivial events in nontrivial systems" (p. 43).

Perrow echoes Dörner when he writes about how the cognitive limitations of humans limit their response to failures in complex systems.

We construct an expected world because we can't handle the complexity of the present one, and then process the information that fits the expected world, and find reason to exclude the information that might contradict it. (p. 214)

Confronted with ambiguous signals, the safest reality was constructed. (p. 217)

Those old enough to remember The Church of the Subgenius from the 1970s may or may not be surprised to find that that UFO religion parody had it right: "slack" is vitally important. Perrow writes about how tight-coupling in complex systems means that there is little margin for error. When things go bad in a system with little or no slack, the badness cascades through the system very quickly. Slack, which is just another word for decoupling, gives more opportunities to respond, including improvisationally, to failure in both space and time. This will seem familiar to designers of real-time software: the insertion of loose-coupling, for example in the form of message queues, between producer and consumer processes may increase stability at the expense of real-time response. Chemical and manufacturing plant designers face the same trade-offs.

Perrow makes the distinction between processes - chemical, nuclear, and otherwise - that are "transformative" versus "additive". The former can have much more complex interactions between its components, and may feature non-linear behavior which may not be understood by the designer of the process. He likens the additive production process to making a sandwich where as the transformative process is making a soufflé: the end result is drastically greater than the sum of its component parts.

Perrow writes about how safety systems frequently have unintended consequences that nullify their effects. The introduction of shipboard radar did not yield the expected reduction in maritime accidents because ships used it as an excuse to go faster. This is true in chemical and manufacturing processes as well; safety features allow the plants to be run at higher levels of production, resulting in greater revenue at the same accident rate.

The title of Perrow's book stems from his assertion that given complex systems with tight-coupling and non-linear behavior, in which safety subsystems are either subverted or exploited, accidents are inevitable, or in some sense, normal, to be expected.

My biggest criticism of Normal Accidents is that Perrow allows his own politics to influence his more analytical writing to the point that sometimes he sounds like a conspiracy theorist. He frequently talks about how the "elite" (by which he means executive management) may sacrifice safety for personal gain. In fact, people act in the way that they are incentivized to act (that's a good basic definition of the field of economics right there). Perrow recognizes this at some level: he uses the term "practical drift" to describe an economically-motivated departure of formal processes, in the form of a local adaptation, from what was required by reality, for example reducing the frequency of routine maintenance to reduce down-time and save money. That's the principle theme of Sidney Dekker's book featured in the next section.

(Edit 2020-01-28: Subsequent reading on my part suggests that the term "elites" seems to be widely used amongst sociologists, at least those I read that are writing in the field of failure analysis. I find the term value laden and disconnected with my own experience.)

Perrow was a sociology professor at Yale when he originally wrote Normal Accidents in 1984, inspired by the events at Three Mile Island in 1979. This book, updated in 1999, has been widely cited by accident researchers since it's publication. His book originally came under fire with his prediction of the likelihood of more nuclear accidents, but events have Fukushima may have caused people to reconsider that criticism. The enormous chemical plant explosion in West, Texas in 2013 makes you suspect that not much has changed since Perrow wrote his chapter on chemical plant catastrophes. His book predates the events of 9/11, but in his chapter on aircraft safety he spookily mentions that although aircraft could hit buildings, they somehow remarkably do not (although he wasn't considering deliberate acts of terrorism).

Sidney Dekker, Drift Into Failure: From Hunting Broken Components to Understanding Complex Systems, Ashgate Publishing Company,  2011

[Bhopal, Flixborough, and Chernobyl] were the effects of a systematic migration of organizational behavior under the influence of pressure toward cost-effectiveness in an aggressive, competitive environment. 

- Jens Rasmussen and Inge Svedung, Proactive Risk Management in a Dynamic Society, Raddningsverket, Swedish Rescue Services Agency, 2000

When I read that quote on page one of Drift Into Failure, I knew I was hooked. Dekker is the first researcher I have read on this topic that has expressed what has been percolating in my brain for several years now: market forces play a significant role, perhaps the most important role, in why complex systems fail. (He also made me a fan of Danish safety researcher Jens Rasmussen, who has apparently been writing about this very thing for many years.)

The gist of Dekker's assertion is that economic forces frequently cause an incremental departure from normal operating procedure, creating circumstances in which systems gradually over time move outside of their envelope of safe operation, and finally, sometimes catastrophically, beyond their safety margin. This incremental departure - for example, gradually increasing maintenance intervals to reduce downtime and save money, a change which is eventually accepted as the norm because nothing bad happened (yet) - is termed normalization of deviance (quoting the research of Diane Vaughan [2], of whom I am also now a fan). The perhaps unwitting acceptance of a greater level of risk - during which the system has not yet failed even though it is operating beyond its design parameters - is risk homeostasis. This gradual slide into the danger zone is what Dekker refers to as a drift into failure. Software engineers will recognize that Dekker is describing an effect in our field known as technical debt: the continual and gradual cutting of corners in the design and implementation of a software system that eventually results in the system becoming overwhelmingly unmaintainable.

Dekker describes a diagram from another paper by Rasmussen [3] that succinctly sums this up.



Market forces ("Management Pressure toward Efficiency") and labor forces ("Gradient toward Least Effort") tend to cause the operational point of a complex system to migrate into its error margin and beyond. Once the operational point passes the "Boundary of functionally acceptable performance", the system may fail, suddenly and catastrophically.

I glommed onto this so strongly because I've been thinking about this for years. It has surely colored my choices in reading material as I've read more and more popular books on economics, ranging from traditional micro- and macro-economics to the more esoteric game theory, and most recently a book on organization economics. I like reading about economics for the same reason I enjoy reading about physics: economics describes reality in a way that I find useful. Dekker's book was the first I've read to make explicit the connection I'd been suspecting between economics and system failure. The near inevitability of economic forces pushing systems into failure harkens back to Perrow's "normal accidents".

Drift into failure isn't merely a hypothesis. Dekker, himself both a safety researcher and a commercial airline pilot, writes at length about Alaska Airlines flight 261, a McDonnell Douglas MD-80 that on January 31, 2000 took off from Puerto Vallarta, Mexico destined for Seattle, Washington. In Dekker's harrowing transcription of the flight deck voice recorder, the pilots vainly try to regain control of the aircraft after the sudden failure of its trim jackscrew-nut assembly, a mechanical component that controlled its horizontal stabilizer. The rear stabilizer eventually slid to its far end of travel and stuck, creating enormous downward pitch forces and rendering the aircraft uncontrollable. For a brief period the pilots resorted to flying the huge commercial airliner inverted, a tactic that almost worked. But shortly afterwards the aircraft hit the Pacific Ocean near Port Hueneme, California, breaking apart, killing all five crew and eighty-three passengers.

The story of how the jackscrew-nut on flight 261 failed became a classic case study in normalization of deviance. When the MD-80 was launched in the mid-1960s, McDonnell Douglas recommended that the trim jackscrew assembly be lubricated every 300 to 350 flight hours, the so-called "B" maintenance interval. That means grounding the plane every few weeks. The access hatch to reach the assembly was small and awkwardly located, making the job difficult and time consuming. So in 1985, the airline reduced the maintenance interval to every 700 hours, or every other "B" maintenance interval. In 1987, the frequency of the "B" maintenance itself was moved to every 500 flight hours, making the  interval between lubrication 1000 flight hours. In 1988, the "B" maintenance was eliminated, and that work redistributed in the "A" and "C" maintenance. The lubrication was scheduled for every eighth "A" maintenance, done every 125 hours, so the jackscrew was still lubricated every 1000 hours. In 1991, the "A" maintenance was extended 150 flight hours, leaving the lubrication to be done every 1200 hours. In 1994, the "A" maintenance was extended to 200 hours; lubrication every 1600 flight hours. In 1996, the airline moved the lubrication task from the "A" maintenance task list to a task list that was done every eight calendar months, regardless of flight time. For flight 261 that might have translated to the jackscrew assembly being lubricated every 2550 hours. The jackscrew assembly recovered from the ocean floor did not show any evidence of ever having been lubricated.

Each individual change in the maintenance schedule for the jackscrew assembly was by itself a minor alteration. Doubling the interval from 350 hours to 700 hours still placed the maintenance well within the error margin for that component. Each change was come by honestly to save money and reduce effort, with no conscious intent to impact safety. But over time, each incremental change, done without consideration of any of the prior changes, gradually pushed the system into a maintenance interval that was nearly ten times the duration recommended but the manufacturer.

If, like me, you have ever worked in a code base of millions of lines of code that were the result of efforts of hundreds of developers over the span of decades, this may all sound familiar. As I myself once said: "How does a switch statement get to be four thousand lines long? One line at a time, baby, one line at a time."

John Gall, Systemantics: How Systems Work and Especially How They Fail, Pocket Books, 1975

It's hard to believe from the perspective of 2014, but way back in the 1970s it was acceptable, and even not that uncommon, to write a business best seller that was short, funny, and wise. Books like The Peter Principle and Up the Organization and Systemantics. How I long for those days.

It was in the 1970s, when I was a systems programmer in an IBM mainframe shop, that I read Systemantics because it was recommended to me by my colleague (and now old friend) Mike Manuel. I returned to it and re-read it recently because I saw it cited in a contemporary article on software engineering.

Gall's book holds up. Gall, who was, of all things, a pediatrician, spent much of his life studying and writing about why systems fail. He was studying more or less human systems -- organizations -- but what he wrote about applies to technological systems as well. He summarizes his research in a few dozen short dictums. Here are a few that are among my favorites.

The real world is whatever is reported to the system. 

The bigger the system, the narrower and more specialized the interface with individuals. 

A simple system, designed from scratch, sometimes works. 

A complex system that works is invariably found to have evolved from a simple system that works. 

A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system. 

Complex systems usually operate in failure mode. 

When a fail-safe system fails, it fails by failing to fail safe. 

Loose systems last longer and work better.

To no one's surprise, Gall is frequently quoted in the software engineering literature. Many of Gall's principles described in this short (158 page) and very entertaining book are expounded upon in much longer and far less readable tomes. As a litmus test, if you didn't find yourself nodding at many - perhaps all - of the dictums quoted above, then I question whether or not you are an experienced software engineer. Heck, the real world is whatever is reported to the system and complex systems usually operate in failure mode alone are worth the price of admission. I had to learn those the hard way while developing error recovery algorithms for a large telecommunications system.

Gall coins the term anergy as the opposite of energy, equivalent in a biological system to torpor: "A coiled spring is full of energy. When fully uncoiled, it is full of anergy." (p. 149) Anergy is measured in units of effort required to bring about the desired change. Hence:

The Law of Conservation of Anergy: the total amount of anergy in the universe is constant.

It may not seem obvious at first glance, but this simple conservation law explains unintended consequences, incentive distortion, measurement dysfunction, perverse incentives, the tragedy of the commons, really a whole class of problems discussed in business and economics. If you fix one problem, the total number of problems in the universe is not reduced. Because anergy must be conserved, other problems must be spontaneously created as a result of your actions. There is no end to it.

Galls book was so successful that he published a whole raft of follow-on books. But those are mostly just expansions of the basic ideas he laid out so succinctly in his original paperback. In a way, they just represent one more large complex system that evolved from a small simple system that worked.

Footnotes

[1] Dietrich Dörner, The Logic of Failure, Basic Books, 1997

[2] Diane Vaughan, The Challenger Launch Decision, University of Chicago Press, 1997

[3] Jens Rasmussen, "Risk Management In A Dynamic Society: A Modelling Problem", Safety Science, 27.2/3, Elsevier Science Ltd., 1997, pp. 183-213

(Footnotes added 2020-01-23)