The Kernel

(photo credit: Dale Courte)

The Kernel - which is the only name it ever had, long predating the use of the term in the context of Linux - was a tiny little real-time operating system kernel written in PDP-11 assembler language circa 1980 by Dayton Clark when he was a graduate student in computer science at Wright State University. The Kernel was used in the infamous CS431 Advanced Programming course, which later became CEG431 Real-Time Software Design when the degree program split into seperate Computer Science and Computer Engineering majors. CS431 was developed by Wright State professors Bob Dixon and Joe Kohler. Over the span of the ten week quarter, students had to complete a complex multi-threaded application which used the Kernel to synchronize and communicate among several threads, each of which controlled interrupt driven I/O devices, some of which supported DMA. It was all written in PDP-11 assembler and ran in a total of eight kilobytes of core memory on a PDP-11/05. The course culminated in an oral exam featuring a dump analysis. You had to pass the oral exam to pass the course.

CS431 was required for undergraduate computer science/computer engineering majors to graduate, and for graduate students to take upper division graduate courses. Students either loved it or hated it. Area employers, who for the most part were heavily into defense-related embedded development in the neighborhood of the ginormous Wright-Patterson Air Force Base, loved it. And perhaps for that reason, some students came to love it in retrospect.

Around 1982 through 1985, the code was substantially cleaned up, refactored, and documented by John Sloan and David Hemmendinger, both of whom, along with Dale Courte and others, all graduate students, went on to teach the course. The kernel was split into two source files, the pure code System part, and the impure data Control Block part, to expedite burning the pure portion into EEPROM. It supported process creation and destruction, counting semaphore wait and signal, and asynchronous message passing. It did so in only a handful of PDP-11 machine instructions.

Besides being used pedagogically, the Kernel made its way into a number of other research projects, including SLICK (which supported message passing among a network of LSI-11 microprocessors), TASK4TH (a FORTH-based LSI-11 workstation for real-time data acquisition and robotic control), and FPS (a functional programming system). The Kernel was ported to a number of other microprocessor targets, re-written in C and C++, and, rumor has it, used by former students in their own production systems in the commercial and defense sectors. My alter ego John Sloan wrote a technical report describing the Kernel in detail, as well as suggesting techniques for debugging embedded applications that use it, which was always a challenge.

It is tempting to say that the Kernel is of historical interest only. But tiny little microkernels taking only a few machine instructions have come into vogue as minuscule microcontrollers become embedded in everything. The Kernel was more than a tool used to teach a class. It has informed my career for the past thirty-five years.

(2017-11-08: To facilitate the efforts of far future - and, it must be said, extremely hypothetical - cyberarcheologists, the source code for the Kernel, the technical report, and other related material, have been stored in a repository on GitHub: https://github.com/coverclock/edu-wright-cs431 .)

xkcd: Digital Resource Lifespan

Today's excellent (as usual) xkcd cartoon by Randall Munroe reminds us of the ephemerality of digital information.

Once again, this being a topic I have raised over and over in this blog, I'd like to mention that the definitive work on this topic is

Jeff Rothenberg, "Ensuring the Longevity of Digital Documents", Scientific American, vol. 272, no. 1, January 1995 .

A digital copy of Rothenberg's article can, at least at the moment, be found at

https://www.clir.org/pubs/archives/ensuring.pdf .

It's well worth reading. And appropriately terrifying, given the season.

It's not just the digital media itself that has a (sometimes remarkably short) lifespan, but also the technology - both hardware and software - to read the media, and even the specifications that describe the format and encoding of the media. I've seen a vase from ancient Greece inscribed with art and symbols that are perfectly legible millennia later. I've been up close to a first edition of a book written by Galileo in the early 18th century. We can still read the letters John Adams and his wife Abagail wrote to one another during the time of the American war of independence. I have CD-ROMs that are just a few years old which are no longer usable. And trying to read the eight-inch floppy disks for the CP/M file system that have the text of my master's thesis in WordStar format? Good luck. (The thirty-four year old hardbound version on acid-free paper is completely legible.)

The only way to insure the longevity of digital information is to continuously refresh it onto new media using current technology. This becomes less and less practical as the amount of digital information continues to grow an a rate that can only be described as alarming.

You Cannot Cheat What You Do Not Measure

The international edition of the German magazine Spiegel has a two part article about the three graduate students who uncovered the huge emissions fraud by Volkswagen. This is of interest to motorheads and techies alike. The case is another great example of measurement dysfunction (and its kin incentive distortion, moral hazard, perverse incentives, unintended consequences, etc.) albeit on a enormous scale.

http://www.spiegel.de/international/business/the-three-students-who-discovered-dieselgate-a-1173686.html

http://www.spiegel.de/international/business/the-three-students-who-discovered-dieselgate-a-1173686-2.html

Thought Experiment

A long long time ago, in a university far far away, I was thinking about how the classic ring-shaped rotating Von Braun space station, like the one with the Hilton hotel in the movie 2001: A Space Odyssey, uses centrifugal forces to simulate gravity. I began to wonder: if you took that space station way way out into deep space, presumably its occupants would still feel pushed towards the floor that was the outer portion of the ring. But if they looked out the window, there is no way they could tell relative to what they were rotating. In fact, other than the fact they could still sip coffee out of open-topped mugs, they couldn't tell they were rotating at all, visually.

(image credit: Wikimedia)

When faced with such conundrums, I did what I always did at the time, I asked my friend and colleague David Hemmendinger. David and I were both graduate students in computer science at the time. But while I had a freshly minted bachelors degree in CS, David already had a Ph.D. in philosophy. At the time he was the smartest guy I knew. (He may still be the smartest guy I know. But in the more than three decades hence, I've continued my habit of working with people a lot smarter than I am, so it's not a given.)

Dave explained that this was a question that troubled lots of people. He told me about Mach's Principle - the same Mach as in Mach Numbers, for describing supersonic speeds. Mach proposed that the rotation that results in centripetal force is relative to absolute space (take that, Relativity!). It's the same principle that causes gimbaled gyroscopes to try to maintain their orientation as they spin no matter how you move and rotate them; they are trying to remain in the same position relative to the entire mass of the universe.

Seriously?

That sounded pretty darn fishy to me. I'd had several physics courses as an undergraduate and knew the math behind Relativity well enough. I understood that rotation was a form of acceleration, and that accelerated inertial reference frames were different from unaccelerated inertial frames of reference.

But how exactly does the distant mass in the universe affect the gyroscope on my desk? Apparently instantaneously? I didn't buy it. I could tell from Dave's clouded expression and half smile that he didn't quite buy it either. He made it clear that he was just reporting what he had read.

This question stayed with me for the next thirty-plus years. I just figured I was kinda stupid, and that someday I would come across the answer.

So this morning I read Tony Rothman's article in American Scientist, "The Forgotten Mystery of Inertia".

https://www.americanscientist.org/article/the-forgotten-mystery-of-inertia

It's not long. And it's an easy read. Here's is what I learned.

No one understands this.

This same basic thought experiment troubled Newton, Einstein, Mach, Gödel, and lots of physicists and philosophers right up into the present day.

Everyone knows the what of inertia; the math we use to describe it is completely accessible. But no one can explain the how.

How do rotating space stations simulate gravity? How do gyroscopes strive maintain their original orientation - RELATIVE TO WHAT, FOR CROM'S SAKE?!?!?!? - once they start spinning, and do so with such reliability that we use them inside gyrocompasses as part of inertial navigation systems in everything from nuclear submarines to intercontinental ballistic missiles?

I'm not stupid. (Well, not because of this, anyway.)

The world we live in is. Just. That. Weird.

"If you don't find reality fascinating, you just aren't paying attention." - Me, often.

My WWVB Radio Clock

Obelisk, a.k.a. O-3, is the third (and probably final) digital clock I've designed and built that is ultimately synchronized to the atomic clocks that serve as our standard for time in the U.S. It synchronizes to the sixty kilohertz transmission from radio station WWVB in Fort Collins Colorado. WWVB is operated by the National Institute for Standards and Technology (NIST). NIST has one of their big facilities in Boulder Colorado, where the F2 master atomic clock, the source of ultimate time and frequency reference for the United States, is located. Fort Collins is just a hour or so drive north of where I am located near Denver Colorado. But the WWVB time signal reaches most of the continental U.S. and beyond, typically at night when the sun doesn't interfere with the long-wave transmission.



The hardware of Obelisk is very similar to my prior two clocks. It is based on a Raspberry Pi 3 Model B running Raspbian, a Linux distro based on Debian. It uses an Adafruit LCD display. And it includes a real-time clock board with a battery backup, based on the DS1307 chip, to maintain the time when the system isn't running.

What sets this clock apart is that it uses a tiny SYM-RFT-60 AM radio receiver configured out-of-the-box to pick up the WWVB sixty kilohertz transmission. The receiver provides just six pins: one for power, one for ground, two for the antenna, one for reset, and the last for the digitized and inverted data derived from the modulated radio signal. The reset and data pins are connected to GPIO pins on the Pi.

The WWVB radio signal is amplitude modulated: the reference state is 0 dBr and the asserted state is -17 dBr.  The receiver inverts this so that the reference state is read by the GPIO as a zero, and the asserted state is read as a one. The transmitter emits one pulse per second, with the leading edge of each pulse aligned to the beginning of a second as determined by an atomic clock located at the transmitter. The duration of the pulse is used to encode three symbols: 200 milliseconds is a binary zero, 500 milliseconds is a binary one, and 800 milliseconds is a special marker used to identify the boundaries in the data frame.

A frame consists of exactly sixty pulses, so there is always one frame per minute. The leading edge of the first pulse in the frame indicates the beginning of the first second, :00, in the current minute, and the leading edge of the last pulse of the frame indicates last second, :59.

Values in fields are encoded as binary coded decimal (BCD) digits. So, for example, the minute of the hour is encoded as two BCD fields, the tens portion, and the ones portion. The UTC minute and hour, the day of the year (Julian day), and the last two digits of the year, are encoded in fields in the frame, along with other useful stuff like whether the current year is a leap year, whether a leap second is going to be added at the end of the current month, whether Daylight Saving Time is in effect, and the difference between the UTC time and UT1 time. (UTC is kept within +/- 0.9 seconds of UT1, a time reference based on celestial observations that is relative to the Earth's rotation, through the introduction of leap seconds.)

Obelisk merely samples the GPIO pin in user-space using a periodic timer, and measures the pulse width to determine the three symbols. Most everything else is just the bookkeeping involved with decoding the fields in the frame and turning them into a useful timestamp. Not that that was completely trivial.

The hardest part of this process was designing the state machine to parse the language whose grammar is implied by the layout of the symbols in the frame. While the AM radio signal can be received throughout all or most of the continental United States, it is extremely susceptible to interference. (This is what Steely Dan was alluding to when they sang "no static at all" in their 1978 song FM.) The WWVB signal can be corrupted by the placement of the radio antenna, by other electronic devices (particularly LCD displays), and even by strong sunlight. So the Obelisk parser tends towards paranoia, with a lot of stuff for error detection and recovery.

One technique I have found useful to measure the quality of my clocks is to run the Network Time Protocol daemon on them and use my implementation as an NTP reference clock. Obelisk emulates a GPS time source by emitting standard National Marine Electronics Association (NMEA) RMC sentences containing a UTC time stamp every second once it is disciplined to WWVB. It also generates a Pulse Per Second (PPS) on another GPIO pin; the PPS is synchronized, subject to some software delay and jitter, to the leading edge of the WWVB pulses that are timed to begin at the start of every second.

NTP continuously compares the reference clock with clocks from other NTP servers on the internet. Querying the NTP daemon on Obelisk for its statistics often reveals that my implementation is pretty usable. It will never be as good as a GPS-disciplined NTP server; the WWVB signal is just not that reliable. But as a desk clock, it's completely serviceable. Below you can see the output from the ntpq command that includes my radio clock masquerading as PPS and GPS reference clocks. This snapshot was taken while NTP was pretty happy with the output of my software.



WWVB radio clocks are inexpensive and common; there are several such off-the-shelf devices scattered around the Palatial Overclock Estate. And WWVB radio clocks can be quite small. One of the wristwatches in my collection is a Casio Wave Ceptor with a built in WWVB radio receiver which it uses to set itself. (It's also solar powered.)



My first clock, Hourglass (O-1), synchronizes to the signals from the satellites in the Global Positioning System. Like other radio navigation systems I have studied - for example, LORAN, for Long Range Navigation - GPS is based on time: the accuracy of the GPS position fix depends on the precision of the timestamps generated by the GPS system. Each GPS satellite has multiple atomic clocks, each of which is synchronized to the U.S. Air Force master atomic clock at the GPS ground station near Colorado Springs Colorado. I described Hourglass in My Stratum-1 Desk Clock. It was a relatively straightforward project, using off the shelf hardware and open source software. If you want an inexpensive stratum-1 NTP server, this is the approach I would recommend; reasonably priced commercial products are also available and which I describe in this same article.



My second clock, Astrolabe (O-2), similarly synchronized to GPS, but uses a miniature cesium atomic clock to maintain time if and when the GPS signal was lost or compromised. I described Astrolabe in My Stratum-0 Atomic Clock. The software for Astrolabe was almost identical to that of Hourglass, although I found the hardware design much more challenging; there was a fair amount of hardware hacking (for a software guy like me anyway) to integrate the board containing the chip-scale atomic clock with the rest of the system. The biggest problem was that the Raspberry Pi is a 3.3 volt device and the CSAC board is a 5 volt device with standard RS-232 12 volt serial output. The design was complex enough that I felt the need to design in test points through which I could examine both the cesium atomic clock and its uBlox GPS receiver independently of the rest of the system.



Obelisk differs from these two clocks in that I had to write a substantial amount of software to decode the WWVB signal and put it in a form usable by the open source software in the clock. But that's what learning experiences are all about. Obelisk provides a user-space application I call wwvbtool. The lower level details of decoding the WWVB transmission is handled by functions in the Obelisk library, which can be used for other WWVB-based applications. wwvbtool also makes use of my Hazer (GPS sentence generation and parsing) and Diminuto (Linux systems programming) libraries. wwvbtool, Obelisk, Hazer, and Diminuto are all written in C. All of the code is open source - licensed under either the GPL or the LGPL - and can be found on GitHub.

https://github.com/coverclock/com-diag-obelisk 

https://github.com/coverclock/com-diag-hazer  

https://github.com/coverclock/com-diag-diminuto 

Although wwvbtool can be used from the command line, Obelisk runs it as a background daemon where it communicates with the GPS daemon gpsd, which in turn communicates with the NTP daemon ntpd. Obelisk also uses the same hourglass.py Python script I wrote for Hourglass and Astrolabe to maintain the LCD time display.

When Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I first moved to Colorado in 1989, I didn't appreciate that I was going to be living at a nexus of super precise time keeping, what with NIST's F2 atomic clock in Boulder, the WWVB NIST transmitter near Fort Collins, and the GPS ground station near Colorado Springs, all a relatively short drive west, north, and south respectively.

But precision timing and frequency sources keep cropping up again and again in my professional life, from the supercomputers and their associated peripherals and networks that I worked with at the National Center for Atmospheric Research in Boulder, to the traffic shaping and traffic policing firmware I wrote for ATM OC-3 optical networking products at Bell Labs north of Denver, to the geolocation and navigation systems in products I helped develop for the business aviation marketplace for a client near Denver.

The other day I was talking about Hourglass, Astrolabe, and Obelisk with a friend and colleague of mine. He repeated that aphorism "A man with more than one clock never knows what time it is." I had to remind him that my three clocks all display exactly the same time.

Time and Space

Once again my friends and colleagues at Gogo Business Aviation in Broomfield Colorado indulged me, in the form of my alter-ego John Sloan, by letting me come and give a talk entitled Time & Space on the historical connection between precision timekeeping and navigation. They taped it and put it on the 'tube. I had a great time, and hopefully I wasn't the only one. Among lots of other things, I describe how both GPS and atomic clocks work.

I did, in my enthusiasm, misspeak a couple of times - that's the hazard of giving an hour long talk without slides or notes - and have made some corrections below.

A big Thank You to the folks at Gogo for sponsoring this.

Corrigendum

1. Despite my saying that the escapement wheel is not a gear (and it's not), I incorrectly referred to it as a gear once or twice.
2. When I gave the example about longitude, it would be 5 hours and 11 minutes before GMT, not after.
3. One of my Ph.D. physicist colleagues remarked that it's not really a different electron orbital shell to which the electron transitions but instead a hyperfine transition. (I'll have to look that up.)
4. In my excitement, I said neutrons have a positive charge when of course I meant it is protons which have a positive charge. (A particularly egregious mistake since I've know that since grade school.)
5. The GPS block IIF satellite has three atomic clocks. not four: two rubidium and one cesium. An earlier model GPS satellite I've studied has two of each.
6. There are a variety of algorithms used by GPS to determine your position; I described one class of algorithms. There are simpler closed-form solutions that work for time measurements from four satellites. And there are even simpler algorithms that work for three satellites, but which don't compute altitude. The iterative solution I describe is the most general one, solving for all three spatial coordinates, for time, and for as many satellites as can be observed.
7. I said the volume of the final GPS solution was fifteen meters across. It's actually about five meters across, or about fifteen feet. That jives with a timestamp resolution of about fifteen nanoseconds (light travels about a foot in a nanosecond), and not the 1.5 nanoseconds I quote in the talk. (Maybe I should break down and use notes or slides, huh?)

Mo My MIMO

I have a tough time thinking of the Internet of Things as a new technology arena, considering I've worked on all kinds of embedded systems that were connected wirelessly, directly or indirectly, to the Internet. These ranged from eight-bit micro controllers to in-flight entertainment systems for business jets.

Recently I've been futzing around with a couple of IoT devices:



the Amazon Web Services IoT Button and



the TP-Link Smart Plug Mini. I was inspired by the clever work of my friend, former colleague, and one time college flat mate Paul Moorman who used the IoT Button with a Smart Plug to implement a wireless light switch. Based on prior work by other folks who reverse engineered the Smart Plug (and which he referenced in his article), Paul controlled the Smart Plug by punching a hole in his firewall for the port on the Smart Plug from which it receives commands, 9999, and then writing an AWS Lambda script in Python to talk to it. Pressing the IoT button stimulates the Lambda script in the AWS cloud to send the appropriate message to the Smart Plug to turn the power to any connected device on or off. Pretty cool.

I'd be lying if I didn't admit I was a little envious of Paul's ingenuity.

Ultimately I came up with my own practical application - significantly, as it turned out, one that required no development or firewall changes on my part - of just the Smart Plug: to turn our "cat cam" - an existing web cam in our kitchen near the cats' food bowls - on and off from my iPhone. It was pretty trivial to do. On a recent trip from Denver to Phoenix I was able to turn the cat cam on remotely using the TP-Link Kasa app and then use the webcam's app to peek at our beloved feline overlords (long may they rule).

Here's the thing: I had to punch a hole in my firewall for a port forwarding rule to connect to the cat cam from the webcam app on my phone, resolving the address using dynamic DNS. But I didn't have to punch a hole for the Smart Plug, or use dynamic or any other kind of DNS identifying my home network. What had to be happening is that the Kasa app was connecting to a server in the inter webs, and the Smart Plug was making its own outgoing bidirectional TCP connection through my firewall to a server (perhaps the same one) also on the inter webs.



My first stab at reverse engineering this was to dig out my AirPcap NX tool, a USB 2.0 device from Riverbed Technologies that allows you to look at WiFi traffic and, providing you have the WPA pass phrase and catch the wireless end point when it first connects to the WiFI network, decrypt and decode IP packets as they fly through the air. The AirPcap has drivers for Windows and it is recognized by Wireshark.

This pretty much went according to plan until I realized that I was only seeing the traffic from the server on the inter webs to the Smart Plug, not in the other direction. Peering closely at the TCP connection setup responses from the server, I could tell definitively that the Smart Plug was the connection instigator. The Smart Plug was establishing a TCP stream socket to port 50443 on server dev.tplinkcloud.com. That domain name resolved to a server that was hosted in the AWS cloud (and unrelated to the AWS IoT button implementation). I could see packets from the server to the Smart Plug, but not in the other direction.

This put me in a paroxysm of web searching.

What I think was happening (see disclaimer above) is that my home WiFi access point supports 3 x 3 MIMO (multiple input multiple output) WiFi channels: it exploits spatial multiplexing to divide the packet stream into three transmit and three receive WiFi streams, all on the same radio channel, and which are all in the air simultaneously, each being handled by a separate antenna. My AirPcap NX supports at most 2 x 2 MIMO; it simply could not see all the traffic in the air.

I was using the AirPcap on my company laptop, a Lenovo ThinkPad running Windows 7. As part of my research, almost by accident, I came across an article describing the WiFi radios used by Apple, and discovered that my personal MacBook Pro came equipped with a radio capable of 3 x 3 MIMO with the same WiFi channels and technologies as my access point. Even better, the MacOS WiFi drivers natively support "monitor" mode, equivalent to promiscuous mode for Ethernet wired interfaces. And Wireshark for MacOS understood all of this already.

It didn't take very long for me to verify all of this using my MacBook Pro. I had Wireshark on my personal laptop peeking at all of the WiFi traffic between the Smart Plug and the TP-Link server in the Amazon cloud.



I was pretty proud of myself until Paul asked me why I didn't just insert a SharkTap into the Ethernet connection between my cable modem and my access point and Wireshark the traffic from there. A little embaressed, I did that too.



For sure that was the simpler solution, but it had the side effect of NATting all the IP addresses on my home network to the one address assigned via DHCP to my access point by my internet service provider. It made the Wireshark traces a little harder to grok, but really no big deal.

Never the less, I'm glad I wasn't as smart of Paul and so didn't think of that solution right off the bat. I now know I have another useful tool in my troubleshooting arsenal, one that came ready made right out of the box.

Practical Matters

I'm frequently working on prototype hardware whose design bears only a superficial resemblance to what we are eventually going to ship. Sometimes there is a couple generations of prototypes in very different form factors, with the final version arriving not much before our ship date. Prototypes give the software team a head start (the header, the better) while the hardware team (with a little help from guys like me) work out the bugs in the hardware design. I may find myself rolling an engineering model running on batteries around in a parking lot to test that the cellular modem transitions seamlessly between cell towers. Or crawling around in a twin-engine executive business jet trying to figure out why the satellite communications unit isn't coming up. Or sitting on the floor in a drafty and noisy and sometimes dark server or telecom room between equipment racks, tangled in a maze of cables, with a laptop in my lap.

I love doing this kind of work. But it does require a fair amount of flexibility and no small amount of improvisation on my part. Here - in no particular order - are some of the tools I've found useful over the past few years. Maybe you will too.

Cheater Safety Glasses
Disposable Ear Plugs



The LASIK surgery I got eons ago was a miracle; I still have 20/20 distance vision. But the older I get, the worse my close up vision becomes. So besides protecting my eyes, these safety glasses, that have reading glasses molded into them at the bottom of each lens, let me see what the display on that SATCOM box in the equipment bay is actually telling me. I keep a pair of these in just about every tool kit I have, in my daily briefcase, and in the garage at home. I buy 150s (1.5 diopter) currently, but alas soon to be 175s, at my local hardware store.

My hearing, which has never been very good (a youth spent target shooting, riding motorcycles, and working in white-noisy computer and equipment rooms), hasn't gotten any better. I try to protect what little I have with disposable foam ear plugs. I have several different kinds laying around, but these E.A.R. brand Taperfit with the lanyard work best for me for day to day work. I use them in the lab, in the equipment and server rooms, and on site like when the pilot fires up the auxiliary power unit. Sometimes I use them at the coffee shop. You can buy these at your local hardware store too, but I order the big industrial sized box of individually wrapped ones from Amazon and hand them out (some might say forcibly) to co-workers.

Roll Your Own Custom Outlet Strip



I buy these little outlet expanders and grounded switches and when circumstances require I fabricate my own outlet strip on the spot. The outlet expanders play well with power bricks, and have a little light to indicate that they are electrified. You can place the switches wherever you need to control power to an individual device or set of devices. Add a heavy duty extension cord, and you have a really useful and flexible piece of electrical power distribution kit.

Folding Stool



My knees are getting too old to squat on the floor as I peer at some piece of gear. So I buy these folding stools. Sometimes I sit on them, sometimes I sit on the floor and use them as a laptop stand. They are sturdy, can handle my manly weight, are easy to schlep around, and when folded don't take up much room. I order them from Amazon, but I've seen something similar at my local hardware store.

Aluminum Project Clipboard





When I head to the hardware store to buy something to solve some problem in the lab, I don't want to find out I should have brought a tape measure. Or reading glasses. Or writing material. Or a compass (seriously). I find these aluminum project clipboards, beloved by state highway patrol officers everywhere, really useful. They are pretty indestructible, and hold a bunch of useful stuff inside. Also useful in the field, for all the same reasons. I got this one at my local hardware store.

Velcro Cable Ties



I don't know how I lived without these. When I get a new cable - ethernet, USB, video,  electrical, doesn't matter - the first thing I do now is toss the useless wire cable tie and replace it with one of these velcro cable ties. They come on continuous perforated spools of 25, so you just unwrap one and tear it off. They have a loop hole on one end so you can lasso them to one end of the cable, or connect several together when you need a longer cable tie. They can be easily trimmed with scissors to make them shorter and easier to deal with. My only complaint is they only come in black or gray; I'd like something like safety orange or high-visibility yellow or even shocking pink. I buy them at my local office supply store.

Zip-lock Bags



Embedded development means having  a lot of small related parts and other crap you have to keep track of. I like these Hefty-brand bags with the sliding zip-lock for that. They come in lots of sizes; I have at least one box of each. They have a label area that takes well to writing on with a Sharpie. They are cheap and sturdy. I keep everything from connectors to a Raspberry Pi with its power supply in them. I buy these at my local grocery.

Quirky Cordies Cable Organizer



I have a dozen of these Quirky Cordies cable organizers laying around or permanently assigned to specific tasks. You place cables in between the upright loopy fingers. The Cordies are weighted in the base so they tend not to get pulled off lab benches or window sills (antennas, you know) by the weight of the cables they are holding. They come in a variety of colors (although again not safety orange or high-visibility yellow). I order these off Amazon. I can see seven of them just as I sit here writing this at my lab bench in my home office.

Improvised Ground
Anti-Static Strap



Ideally when you use an anti-static strap - you are using an anti-static strap, aren't you? - you want to ground yourself to the chassis or grounding point of whatever you are working on. Easy in the lab, sometimes not to simple in the field. Although it's not the perfect solution, I came up with these improvised grounds by buying three-prong plugs, and grounding wires (package of two), at my local hardware store. Five minutes spent putting them together makes for an improvised ground in any grounded outlet I plug it into. You do have to be aware of the possibility of ground loops if the stuff you are working on isn't plugged into the same electrical circuit, so some caution is called for.

Although any anti-static strap is better than no anti-static strap, I like these with the Twist-O-Flex bands. I buy them off Amazon. The alligator clip pulls off to reveal a male connector compatible with the female plugs on the anti-static mats I like to use.

Thermal Label Maker



Folks that work with me will tell you I am an obsessive-compulsive labeler. That's why when the quality process inspector comes by to do an audit, it's my lab bench they take them to. I also add my email address and phone number to most of the equipment my company owns in addition to an official property tag. Lots of folks make these thermal label makers, all of which use proprietary cassettes containing a spool of thermal label tape, and which typically come in a variety of foreground and background colors. I think Brother may have discontinued this particular model, which I like because it is capable of printing two lines of text. But once you get bit by the labelling bug, you may start labelling stuff around the house. I bought this one from my local office supply store.

Carlisle Folding Cart





The company that makes this folding cart, Carlisle, sells a variety of sizes and shapes, mostly I suspect for the food service and catering industry. Turns out it's also really useful for field testing. I like this medium-sized but heavy-duty one because it's big enough to carry some serious systems under test, but folds up small enough to fit in the back of my car. It has locking wheels, so you don't see your twenty thousand dollar prototype roll away across the parking lot.

USB LTE Cellular Modem



Admittedly, this is an expensive piece of kit. But when I've needed it, I've really needed it. I've used this Verizon MiFi U620L LTE modem to get an internet connection on a Windows laptop when working in an airplane hanger with no WiFi. Or on a Mac laptop while parked in my car. Or on a Raspberry Pi running on a battery pack and sending telemetry to the mothership. I got this one from Verizon and added it to my cellular phone plan. The incremental cost to my plan was relatively tiny, although the device itself was pretty pricey. But sometimes it's actually faster than the local WiFi, and likely more secure.

Raspberry Pi
Raspberry Pi TTL Serial to USB Adapter

The Raspberry Pi single board computer (SBC) is frequently useful for early prototyping of the more powerful ARM-based embedded systems. I keep one attached to my network in my home office just so I can do quick unit tests on a 64-bit ARM target.

The FTDI TTL-232R-RPI debug cable is an off the shelf serial console to USB adapter that uses my favorite USB-to-serial solution, the FTDI chip. You can order these off Amazon. I added the "BYO 68A" label, which reminds me that the black wire goes to pin 6, the yellow wire to pin 8, and the orange wire to pin 10, on the Pi.

Ginormous Lithium Uninterruptible Power Supply





Another expensive piece of kit, this InteliCharge X1300 UPS has a big lithium battery pack that provides up to 1150Wh to 110V or 220V mains outlets, plus two USB outlets, and a 12V cigarette lighter outlet, for your portable power needs. I've run a large embedded system intended for aircraft power for hours on this unit. I bought mine from Amazon, although I notice they no longer list it; a few moments of web searching suggests it's no longer available. I have a smaller 400Wh unit from Anker that I have also used and which would be my second choice. I've used the InteliCharge to test equipment in a Faraday cage, to roll around in a parking lot to test cellular connectivity, or to move a piece of equipment around in a building without having to power it down.



This photograph shows the UPS and at least four other items from this article in use at a client's site. The UPS is powering the system under test, a laptop, and a VOIP phone. That is indeed Gigi the cat from the anime Kiki's Delivery Service on that bag on the bottom shelf in the photograph above.

USB Charger/Battery



On a much smaller scale, devices like this Anker PowerCore Fusion 5000, which combines a wall wart USB charger with a 5000mAh USB battery, are just the thing for powering the Raspberry Pi and similar USB-powered devices off the grid.



Here is the PowerCore plus at least five other items from this article. The battery will power the Raspberry Pi so that it can send geolocation telemetry via the LTE modem back to the mothership.

Important safety tip: the PowerCore Fusion 5000 is not a uninterruptible USB power supply. Disconnect the PowerCore from the mains electrical outlet and anything connected to it will notice a power disruption while the device switches to its internal battery.

Also: while I pack the Fusion in my travel kit, I learned the hard way to also bring along a convention USB wall charger - a small two-port Anker with a foldable plug - to charge my smartphone at night as I sleep. Because the Fusion switches back and forth between charging the smartphone and charging itself, the first time I used the Fusion, my iPhone kept chiming all through the night to indicate it was now connected to a charger.

NRT ViperBoard



The embedded systems I help my clients develop always have scads of general purpose input/output (GPIO) pins used to communicate with or control sensors or devices. But often, most of the application software the team develops could be tested on a desktop personal computer or even a laptop long before a laboratory prototype is available. But such mainstream systems typically have no GPIO pins. The ViperBoard from Nano River Technologies (NRT) solves that problem. Attach a ViperBoard via USB to your PC and suddenly you have a bunch of GPIO pins, as well as I2C and SPI serial busses. This is also useful for interfacing your PC to lab test equipment. The ViperBoard is supported right out of the box by the more recent Linux kernels. (I added the plastic standoff legs and the "2" label.)

GlobalSat USB GPS Receiver



Similarly, for products which will eventually feature geolocation, a lot of the application software could be tested on a desktop or laptop if only GPS were available. There are lots of USB GPS receivers (and I've tried most of them), but my go-to unit is the GlobalSat BU-353S4. Plug it into a USB port and it enumerates as a serial port. Read from the serial port and you get a stream of standard NMEA sentences providing latitude, longitude, time, and date.

(Update 2018-12-24: I like the later GlobalSat BU-353W10, based on the Ublox 8 chipset, even better.)

USB to Serial to USB Adapter



I've written before about my deep love for the CableMax USB 2.0 to RS-232 DB-9 Adapter, which contains an FTDI chip, my favorite USB to serial converter. Here's an application of it that has turned out to be remarkably useful: I attached two adapters back to back with a null modem converter in between, making a device that can connect two computers via a serial connection, all via USB. This may sound mundane, but lots of embedded application software still make use of serial ports to talk to devices. This little piece of kit lets you test such software on your desktop or laptop without a serial port in sight.

SharkTap (updated 2017-06-28) (updated 2017-07-11)



I've mentioned this device before, but it bears repeating. The SharkTap is an Ethernet tap designed for running Wireshark traces on a LAN. It supports ten megabit, one hundred megabit, and gigabit Ethernet speeds. It passes through Power over Ethernet (PoE) so it can be used on the growing number of devices (like, for me, VoIP phones) that get their power over their Ethernet connection. It is itself powered via a USB cable.



The SharkTap is conceptually dirt simple: it contains a three-port (at least) Broadcom Ethernet switch configured to pass through traffic in either direction on the (as shown here) blue and green Ethernet cables, while mirroring everything in both direction onto the red Ethernet cable.



Here's a photograph I took just moments ago of me using the SharkTap to peek at the traffic between an IoT device - a TP-Link Mini Smart Plug - with its server - in the Amazon Web Services (AWS) cloud as it turns out - by tapping into the WAN traffic between a cable modem and a router/access point.



The SharkTap folks have introduced another model, the SharkTapUSB, that works identically to the one I describe above, but in addition has a built-in USB 3.0 to Gigabit Ethernet interface. This greatly simplifies using laptops which lack a wired Ethernet interface (i.e. pretty much all of them), while using the same USB 3.0 connection to power the SharkTap. It worked great with my MacBook Pro, eliminating the need for me to use my Apple Thunderbolt Gigabit Ethernet adaptor.

USB Power Supply (updated 2017-07-11)



If you're like me, you have a pile of USB power supplies laying around and you have discovered that not all USB power supplies are created equal. I like this one in particular, the Anker PowerPort 2. It provides a maximum of 2.4A to each of two USB ports (4.8A total). It's small and its electrical plug folds away, making it easier to cram into your field kit.

Illuminated Magnifying Glass (updated 2018-05-21)



If you don't need one of these now, you eventually will. Invaluable for reading the fine print on chips, and a million other uses.  I probably own close to a dozen of these, scattered around in briefcases, tool kits, work bench drawers, etc. I buy them at my local office supply store. The battery-powered LED turns off when you slide the cover closed. The specific brand or model matters less than its features and availability.

Analog Phone SIP Gateway (updated 2018-12-24)



This little device allows you to connect analog phones or even FAX machines to a VoIP service provider that supports SIP via a standard Subscriber Line Interface Circuit (SLIC). There are lots of different models and brands of these devices. I own a couple of this particular model, a Cisco SPA112 Small Business Analog Telephone Adapter (ATA) with Router, that supports two analog lines. It costs under US$100.

As retro as it sounds, lots of the embedded telecommunication systems I work on still have analog ports. These are used to support existing legacy devices that specify a SLIC has their hardware interface. Sometimes these really are just analog phones. Sometimes they are cabin communication systems for business aircraft. Sometimes they are satellite communication systems.

Whatever they are, I need a way to test that interface in the product I'm developing months before I have access to an actual example of the legacy device (if that indeed ever happens before we ship an early production model cross country for integration testing). This little box lets me do that with some level of credibility and confidence. Because both the SLIC and the SIP stack are implemented by a third party, I can at least rest assured that my product successfully talks to some other device that I or a colleague didn't write ourselves.

LED Work Light (updated 2021-07-16)

The older I get, the more I appreciate having adequate light. These do the job: a battery-powered multiple-super-bright LED work light. I bought a single one, then a pack of four, then another pack of four. They're not much larger than a pocket penlight, with a pocket clip which has a strong magnet embedded in it. I put one in my car, in Mrs. Overclock's car, my motorcycle touring tank bag, the garage, the basement, my home office, my work briefcase, etc.  No clue how long the batteries last, since I have yet to replace the three AAAs in any of them yet. 

Nail Polish (updated 2022-03-29)

Nail polish turns out to be remarkably useful for marking things like correct settings on potentiometers, or the position of fasteners that should not unfasten. I just used a tiny bit this morning to mark one side of the fuel petcock knob on my Triumph Bonneville so that I can see the damn thing without my reading glasses so that the bike doesn't mysteriously stop running sixty seconds after leaving my driveway.

You may already have this in your maker kit, for those special occasions down at the club. But if not, you can find small bottles - one of which may last you your lifetime - in a staggering selection of colors. (I like bright red, but you do you.) Much like automotive touch-up paint, the bottle cap comes with its own little applicator.

Lab Assistant

It's useful to have a lab assistant to help with all the chores that come with product development and to bounce ideas off of. I've had a lot of good ones over the years. Just make sure you teach them good electro-static discharge (ESD) practices. Here you can see one of my assistants using an anti-static mat.

Stranger Android Things

As far back as 2011 I was talking up using Android as a Linux-based embedded operating system. In those days I was running it on a BeagleBoard. I liked that idea then and I still like it now.



Android has a rich set of features, many of which would be really useful in an embedded product at the higher end. It is architected to make use of multiple network devices - for example, WiFi and a cellular modem - which is becoming common in the kinds of things I work on. Android has a mature tool chain and development infrastructure that includes stuff like a graphical IDE and a debugger. It is a well understood platform on which to deploy third-party applications, which at least one of my clients is eyeing as a business model for one of their products. And best of all, Android has enormous market forces incentivizing other folks to do a lot of the work for you.

I never got any traction on that idea. But maybe now that Google is promoting Android Things - as in Android for the Internet of Things - that will change. Android Things (formerly the platform known as "brillo") is a stripped down version of Android that can be run headless. It has the usual Android Run Time (ART) Java Virtual Machine (JVM) plus support for native C and C++ code, so you can port your legacy drivers and libraries, but do your new application development in an object oriented language that has first-class built-in support for threading, synchronization, and lots of other useful stuff.

Or will. Currently Android Things is just a pre-built developer preview that you can load on a selection of inexpensive development boards like the Raspberry Pi 3. It includes drivers and Java APIs for common embedded hardware interfaces like an Inter-Integrated Circuit (I²C) bus, a Serial Peripheral Interface (SPI) bus, Pulse-Width Modulation (PWM) output, and General Purpose Input/Output (GPIO) pins.

Third-party developers have in turn written higher level drivers and Java APIs on top of these for a selection of peripherals on commercially available expansion boards. For the Pi, that would be the Pimoroni Rainbow Hat. The Rainbow Hat has an APA102 LED strip (SPI), a BMX280 temperature and pressure sensor (I²C), two two-character HT16K33 alphanumeric segments (also I²C), a piezoelectric buzzer (PWM), and some discrete LEDs and buttons (GPIO). Below is a photograph of a Rainbow Hat on a Pi running my single threaded RainbowHatDemo application.



The source for Android Things hasn't been released yet, so porting any of my C or C++ libraries will have to wait. But the developer preview, with a Raspberry Pi and the Rainbow Hat, is a great way to see where this is all going. I wrote some toy applications using Android Studio just to try to remember how to use Java, which I haven't done anything serious with in a decade. You can find the repository, code-named Deringer, with all my source code as Android Studio projects, on GitHub. Below is a little code snippet from my multi-threaded RainbowHatThing application that reads the temperature and pressure sensor and logs the results.

    public void run() {
        Log.i(getClass().getSimpleName(), "begin");
        while (!lifecycleDone()) {
            try {
                float[] readings = sensor.readTemperatureAndPressure();
                float centigrade = readings[0];
                float fahrenheit = (centigrade * 9.0f / 5.0f) + 32.0f;
                float hectopascals = readings[1];
                float inches = hectopascals * 0.02953f;
                Log.i(getClass().getSimpleName(), centigrade + "C " + fahrenheit + "F " + hectopascals + "hPa " + inches + "in");
                pause(1000);
            } catch (IOException e) {
                // Do nothing.
            }
        }
        Log.i(getClass().getSimpleName(), "end");
    }

Here is a little video (with audio) of the RainbowHatThing application running many threads concurrently on my tiny test system.



Will Android Things take off as a platform for the more capable IoT targets? No clue. But for sure there are many economic reasons to take this very seriously.

The Cost of Doing Business

My software development career over the past forty-plus years has been all over the map: supercomputers, eight-bit micro controllers, vast distributed telecommunications systems, multi-core microprocessors. The one thing they all have in common - at least at the level I work at near bare metal - is that the skill sets are all basically the same. But one thing that isn't the same is the cost of the tools to debug, functionally test, and validate my work.

Perhaps twenty years ago, I was writing firmware in C and C++ for optical networking products based on ATM and SONET. One night I found myself working late in a lab trying to debug some recalcitrant real-time code. I was surrounded by an H-P broadband analyzer, two H-P ATM protocol analyzers, and a tangle of multimode optical fiber. I realized I had maybe US$300,000 worth of test equipment dedicated to this one task, never mind my high-end workstation and the room-filling racks of telecommunications equipment. I paused for a moment to realize that an organization smaller than Bell Labs might have had a tough time tackling this product development effort, the investment in infrastructure was so significant.



Moore's Law hasn't really changed things that much, because as the tools got cheaper, the systems we develop and debug have gotten larger and more powerful right along with them. Just the other day I was working on my stratum-0 atomic clock project and I realized that my little hobbyist oscilloscope - one of the most useful tools I have ever purchased - was completely inadequate for doing any kind of jitter analysis on the output of the device. After some reflection, I decided I probably didn't know anyone that had an oscilloscope that could measure the jitter of my chip-scale atomic clock. Any jitter that might be detected would surely be in the oscillator used by the measurement instrument itself.



See, our measurement of time is a funny thing. It is a completely synthetic artifact, a singularly human invention. Thanks to natural variation in the Earth's rotation and in its orbit, time isn't even tied to the movement of the Sun any longer. It's defined, in the International System of Units, to the beat of an oscillator based on the cesium-133 time standard. And the thing I was trying to measure was itself an oscillator based on the cesium-133 time standard. The only way we can measure the precision and accuracy of a clock is by using a more precise and more accurate clock. I'm not likely to ever have one of those.

The larger lesson here is this: before you decide to tackle a project, make sure you can get the tools you need for the entire life-cycle of the product. You may find out that the economics of developing a product are substantially different from the economics of debugging, testing, validating, and supporting that product.