The Enrollment Cliff

In 2007-2008 the sub-prime mortgage debacle led to the Great Recession, which in turn led to a drop in the birth rate. Fourteen years later, the Denver Colorado public school board is debating whether to close ten schools in the city/county due to low enrollment (and reduced tax income). Jefferson County Colorado (where I live) is having similar discussions. As you might guess, this is controversial with parents.

A week ago I read this in Vox

https://www.vox.com/the-highlight/23428166/college-enrollment-population-education-crash

which I'm now seeing referenced other places: colleges and universities are facing a similar "enrollment cliff", not just due to the birth rate, but from high school students selecting career paths other than college.

Pursuing a career path that does not include a college degree is a decision which I do not disagree - not every career requires a college degree, contrary to popular opinion (mostly from college marketing departments). And some extraordinary individuals can get all the training and knowledge they need through rigorous self-directed learning, even in technology fields like mine. (It's been my privilege to work with a few terrific folks like this.) But they are the exception, not the rule. Not every person has that skill set.

(I am deeply cynical however of major tech giants telling high school students that they do not need a college degree to work in the tech field, merely a certificate from specialized training. This is in not untrue. But they omit the words "perhaps for a while, and maybe at a lower salary". I wonder if the long term effect of this will be to encourage unionization in the tech field. [I don't object to this, either.])

Why do I care? For years I've been on an advisory board for the computer science department at my alma mater, so am aware of the impacts of not just enrollment but - to be honest - fads in the technology fields have on academia. And I'm watching this traveling wave of change move through the elementary and high schools, to the colleges and universities, and finally to employers.

I put my money where my mouth is on both sides of this: some years ago I endowed a modest scholarship at my alma mater in honor of my mentor and thesis advisor, the late great Bob Dixon, and I annually donate to support a lab at my alma mater that provides facilities and tools for students to do their own projects for self-directed learning (basically a makerspace supported by the computer science department).

I've also made it pretty obvious where I stand on continuous self-directed learning, by constantly doing technology projects of my own, having over thirty open-source software repositories on GitHub, and writing about all of it in this blog since 2006.

An old friend, university classmate, former fellow university employee, and one-time flat-mate of mine remarked recently that colleges are part of a free-market and are going to have to learn cost control. I agree. Although I'm concerned as much with the second order effects of this wave on our society as I am with the effects on, for example, my alma mater.

The Time Police Play Hardball

Alain Aspect, John Clauser and Anton Zeilinger won the Nobel Prize for Physics this year for their groundbreaking experiments in the field of quantum physics that verified Bell's Theorem.

One of the more unsettling discoveries in the past half century is that the universe is not locally real. “Real,” meaning that objects have definite properties independent of observation—an apple can be red even when no one is looking; “local” means objects can only be influenced by their surroundings, and that any influence cannot travel faster than light. Investigations at the frontiers of quantum physics have found that these things cannot both be true. Instead, the evidence shows objects are not influenced solely by their surroundings and they may also lack definite properties prior to measurement. As Albert Einstein famously bemoaned to a friend, “Do you really believe the moon is not there when you are not looking at it?”

...

Blame for this achievement has now been laid squarely on the shoulders of three physicists: John Clauser, Alain Aspect and Anton Zeilinger. They equally split the 2022 Nobel Prize in Physics “for experiments with entangled photons, establishing the violation of Bell inequalities and pioneering quantum information science.” (“Bell inequalities” refers to the pioneering work of the Northern Irish physicist John Stewart Bell, who laid the foundations for this year’s Physics Nobel in the early 1960s.) 

[D. Garisto, "The Universe Is Not Locally Real, and the Physics Nobel Prize Winners Proved It", Scientific American, 2022-10-06]

https://www.scientificamerican.com/article/the-universe-is-not-locally-real-and-the-physics-nobel-prize-winners-proved-it

It might occur to you to watch a 'Tube video or two, or read a popsci article on this work (like the one from Scientific American above). I've done this, with a dilettante's interest in the matter. I urge you to be cautious, and show restraint. I'm not entirely joking.

I've been reading and watching material on this very topic for years now. And I find it deeply troubling. As did Albert Einstein and his colleagues Boris Podolsky and Nathan Rosen (who were far far smarter than I've ever been) when they wrote their now-famous paper on the "EPR Paradox". They basically argued "the universe cannot work this way".

But alas, the results of actual experiments performed by these Nobel laureates (and which, remarkably, you can duplicate some of which just using three linear polarizing camera lens filters, which I've done) prove that indeed the universe is not "locally real". As John Stewart Bell (who died before winning a well deserved Nobel himself) showed, quantum physics implies that one of two things must be true: either things can interact in a non-local manner (exchanging information faster than the speed of light), or objects do not have intrinsic physical qualities (even basic stuff, like color or mass) until we measure them.

Regarding non-local behavior: I remind my fellow science fiction fans that non-local behavior does not imply that we can use it for faster than light (FTL) communication. There is no way to modulate the effect to carry data. And you really do not want FTL, since its implications for our perception of reality - regarding our ability to distinguish between cause and effect - are even more dire, an issue which becomes obvious after a little reading about Special Relativity.

Regarding the measurement issue: the idea that "objects do not have intrinsic physical qualities until we measure them" sounds suspiciously like "as an optimization, reality isn't rendered until it comes into the viewpoint of a character in the game".

This is a lot more disturbing than it might first appear. One of the possible explanations (not the only one, but perhaps the simplest one) is "superdeterminism", in which the future is every bit as fixed as the past. I've heard several physicists - both quantum physicists and cosmologists, for different reasons - say "there's no such thing as free will; get over it".

My favorite quantum physicist and science explainer, Dr. Sabine Hossenfelder, brought this topic up in her blog/vlog just recently.

Many headlines promptly claimed that this means spooky action is real. But this is not correct. Rather, the experiments showed that you either have to stick with quantum mechanics and accept spooky action, or you reject spooky action and accept superdeterminism. Which is why I keep saying we need experiments to test superdeterminism.

[S. Hossenfelder, "Science News Oct 12", Backreaction, 2022-10-12]

http://backreaction.blogspot.com/2022/10/science-news-oct-12.html

The phrase "spooky action" here refers to Einstein's description of non-local behavior - specifically the collapse of the wave function upon observation, not quantum entanglement - as "spooky action at a distance", an effect that directly contradicts his own Special Theory of Relativity (which has been experimentally verified countless times, in fact, every time you use a GPS receiver). I look forward to reading about the design of an experiment for superdeterminism.

But such an experiment might incur sanctions from the Time Variance Authority (TVA). Last night Mrs. Overclock and I started watching the MCU series Loki, streaming on Disney+. It's fun, and funny, and of course Tom Hiddleston (as the Norse God of Mischief) and Owen Wilson (as a TVA agent) are terrific. But the best part of it is that it basically posits superdeterminism in the form of a "Sacred Time Line". It features the TVA, a time-police bureaucracy, that insures that everyone conforms to it. TVA operatives can "reset" (their term) the time line back to before it was disrupted, and "prune" (ditto) alternate history branches off of the time line. These actions typically cause time line violators to cease to exist. Or, indeed, to not ever have existed. (Or do they?)

The funny thing about "no free will" is that I'm still happier if I live my life - trying to be a good a person as I can be - as if I had free will. It's a strategy I suggest for everyone.

(Slightly revised on 2022-10-24 for clarity.)

Playing with a RISC-V SBC running Linux

Recently I began playing with a VisionFive RISC-V-based Single Board Computer (SBC). It was designed by StarFive. I purchased it from ameriDroid.

RISC-V ("risk five") is a relatively new Instruction Set Architecture (ISA), developed by a team at UC Berkeley, that is entirely open source, unlike Intel and ARM processor ISAs. This is the first RISC-V board I know of that is designed to run Linux (in this case, based on the Fedora distribution). It has two 64-bit 1GHz SiFive U74 CPU cores and eight gigabytes of RAM.

"boron" is my code name for this project. (Every project gets a code name.) I added the nylon feet to hold the board up off the anti-static mat on my workbench. I started out with a keyboard, mouse, and HDMI display attached to the board, but since it supported SSH right out of the box, I quickly reverted to using the command line to build and test my SW, my preferred approach. Also, it reduces the cable clutter on my workbench, which is supporting another project at the moment.

The two projects of mine that I use the most, Diminuto (a C systems-programming library) and Hazer (a GNSS toolkit), were unusually easy to clone, build, and run; unlike most other Linux distros, I didn't have to install any optional packages (much to my surprise). I have successfully completed running all the unit tests for both projects, and am now moving on to functional testing and practical use.

Since my work tends not to be very CPU intensive, I haven't run any performance tests. I'll probably leave that to someone else. I was more worried about SW functionality. But note that this board is more expensive, and and its processor slower, than the ARM-based Raspberry Pi 4 SBCs I use these days.

But the ARM processor design, and the BroadCom implementation of it, on the Pi is closed, and I appreciated the chance to try my stuff out on the open RISC-V, after having read so much about it over the past few years.

Update: 2022-08-20

Just to provide some evidence that I have non-trivial stuff running on the VisionFive, here's a screen snapshot of gpstool running on the board.

gpstool is the Swiss Army knife of the Hazer toolkit, which is built on top of Diminuto. Here, it's processing input in real-time from a GlobalSat BU-353W10 GNSS receiver. The receiver, a USB dongle, is based on the widely used U-blox UBX-M8030 module.

Update (2023-03-06)

I participated in crowd funding the StarFive VisionFive 2 SBC, which has four 1.5GHz RISC-V processor cores and 8GB of memory.

I got the board a few weeks ago and just recently got around to booting up the Debian-based Linux image provided by StarFive. Next step is to get my own software working on it.

It's About Time

Disclaimer: I am but a humble software/firmware developer. I can barely do basic soldering. But my projects often require that I step out of my comfort zone and come to at least a dilettante's understanding of what is going in the bare metal of the embedded and real-time systems I help develop. In this article, I have left my comfort zone so far behind that I can only see it dimly in the distance, as it waves to me from the horizon.

How do you test the accuracy (correctness) and precision (repeatability) of a clock? By comparing it to another clock. Presumably a more accurate and precise clock. How do you know your reference clock is more accurate and precise? 

Studying this problem is how I learned the term clock trip. This is when you, for example, run a cesium atomic clock on batteries, synchronize (match the clock time) and syntonize (match the frequency) it with a reference clock, take the portable atomic clock on a long airplane flight, and then compare it with the clock under test.

Or, at least, that is how it used to be done. These days, everyone compares their clocks with GPS time, which anyone with a GPS receiver can access. When you synchronize and syntonize a clock with a GPS receiver, it is said to be GPS-disciplined. Then, how good your clock is depends in part on how good your GPS receiver is, how good its own internal clock is, and how good the atomic clocks are in the GPS satellite constellation. But at least you save on plane fare.

I'm not new to the idea of characterizing one clock with another clock.

Timegrapher 

(Click on any image to see a larger version.)

Here is a tool I have used to test mechanical watches: the Timegrapher. It has a sound transducer (a fancy name for a type of microphone) that listens to the ticking of a mechanical watch placed in its test fixture, compares it to its internal quartz oscillator, and displays the results.

I don't find any information about the quartz oscillator used in the my Timegrapher in its documentation. This is a common refrain when testing clocks. But any quartz oscillator - including the one in your quartz wristwatch, if you care to wear one - is likely to be far far better than the spring-driven mechanical movement in the Rolex GMT Master II shown above, and at a tiny fraction of the cost. The same cannot be said, alas, when comparing a quartz oscillator to a clock that is disciplined - that is, continually steered or adjusted - to a GPS reference.

Network Time Protocol

I currently have six small Network Time Protocol (NTP) servers on the local area network at the Palatial Overclock Estate. Two are commercial devices, and the rest are ones I designed and built myself. All but one are GPS-disciplined; the odd one out has an AM radio receiver it uses to pickup and decode the NIST WWVB time transmission. One of the GPS-disciplined servers I built has a cesium chip-scale atomic clock (CSAC) it uses as a frequency source.

Exactly how does one assess the accuracy and precision of a GPS-disciplined clock - especially one that incorporates a cesium atomic clock, a technology from which is currently derived the literal definition of the second - without having a clock that is even more accurate and precise?

That's the problem I continually face: any test instrument - like a digital oscilloscope - that I am able to afford, even one with an oven-controlled quartz oscillator (OCXO) or a temperature-compensated quartz oscillator (TCXO), either of which is far better than the quartz oscillator in my Timegrapher, is likely to be not nearly as good as the GPS-disciplined clock I built. If I try to measure the jitter (short term variation) or drift (long term variation) in my GPS-disciplined clock, am I measuring the error in the clock, or the error in the instrument?

I addressed this problem with my NTP servers by building a little test system, Cesium, using open source NTP software. Cesium runs 24x7, and periodically requests the time from all of my NTP servers plus some public NTP servers on the Internet. The statistical tests in the NTP software score each server. I wrote about this in Monitoring GPS and NTP: Yet Another Raspberry Pi Project

What clock is Cesium using? It's Raspberry Pi has a 19.2MHz crystal oscillator (later Pi models use a 54MHz crystal oscillator). The Raspberry Pi oscillator is not temperature compensated, nor is it GPS-disciplined. And its frequency isn't the final word in how precise or accurate it is. This little tool is useful at a gross level, but it is in no way adequate for more rigorous testing.

UBX-ZED-F9T

This problem came to light again recently when I purchased a Sparkfun board with a U-blox ZED-F9T Global Navigation Satellite System (GNSS) receiver. When its antenna remains in a fixed position over a long period of time (like the window of my home office), so that the long term weighted average of its navigation solution has a low error, the F9T can act as a very accurate and precise time and frequency source. It bases its solution on an ensemble of satellites that are part of the U.S. GPS, Russian GLONASS, Chinese BeiDou, and European Galileo GNSS constellations.

The ZED-F9T is another example of the ninth generation of U-blox GNSS receivers, like the ones I have used in my Differential GNSS (DGNSS) projects that I wrote about in Practical Differential Geolocation. This receiver, though, is especially designed to be a time and frequency source. The SparkFun board is equipped with three SMA connectors: the center one is the input for the receiver's active multiband GNSS antenna, and the other two are outputs for software-configurable digital timing pulses.

Who doesn't need a GPS-disciplined time and frequency source? I decided to set up test bed using the ZED-F9T - ignoring the fact that I don't have a high-quality laboratory-calibrated oscilloscope (the cost of which could easily run into five or even six figures) - and see what I could see.

The documentation for the F9T say it can emit an output pulse with a frequency up to 25MHz. Although the internals of the F9T are proprietary, this is likely to be a performance limit of its baseband processor, which is probably has some kind of ARM core. The F9T (and in fact all GNSS receivers) has a broadband processor where the digital signal processing of the gigahertz GNSS signals is done, and a more conventional baseband processor where solution computations and input and output messaging is performed. F9T docs say that the device's real time clock (RTC) is driven by a 32KHz crystal oscillator, and suggest that it is not temperature compensated. The F9T's datasheet claims an output time pulse jitter of ±4 nanoseconds. The device is marketed as being suitable for stringent 5G cellular network timing.

 I wrote a script, using my Hazer open source GNSS toolkit, to configure the F9T's first time pulse output to emit a 1Hz pulse, and its second time pulse output to emit a 10MHz pulse. 1Hz, a.k.a. one pulse per second (1PPS), is a commonly used mechanism to syntonize a time clock to GNSS time, while 10MHz is (as we will soon see) a commonly used frequency reference for test and measurement equipment.

Arrayed next to the F9T on my workbench is Rhodium, a Raspberry Pi, and my modest test equipment (described below). The Pi is where my software runs. The Pi has a Universal Serial Bus (USB) connection to the F9T to provide power and a serial connection for configuration and monitoring.

TAPR TICC

The TAPR TICC is a time interval counter with claimed 60 picosecond resolution. It is produced by the Tucson Amateur Packet Radio (TAPR) organization. It consists of a specialized board mated to an Arduino Mega 2560. It has three SMA connectors: one for its reference clock input (more on that below), and two channels of input that it can timestamp, either in absolute or differential terms. It uses the Arduino USB connection for power, and as a serial connection for configuration and monitoring.

    

The TICC is a relatively low frequency (albeit high resolution) device, because it outputs a line of ASCII text for every pulse it detects. This makes it suitable for characterizing the 1PPS output of the F9T.

The TAPR TICC documentation has this to say about its timing capabilities.

The TAPR TICC is a two-channel timestamping counter with better than 60 picosecond resolution and less than 100 picosecond typical jitter. It has an Allan Deviation noise floor below 1e-10 for a one second measurement.

The TICC requires an external 10 MHz reference clock at nominally +3 dBm.

The 10MHz reference clock required by the TICC was provided by a LeoNTP, one of my commercial GPS-disciplined NTP servers, manufactured by Leo Bodnar Electronics. I've had the LeoNTP on my home network for several years. It consistently scores well with my Cesium NTP monitoring tool. I also find its front-panel user interface easy to use. That's two of the reasons I chose it as my reference clock. Another is that it has a BNC connector with a 10MHz output for just this purpose.

The docs for the LeoNTP has this to say about its timing accuracy and precision of its BNC time pulse output.

3.3V into High Impedance, 1.5V into 50Ohm PPS/1Mhz/10Mhz Accuracy 30ns RMS, 60ns 99%

While this isn't bad, you'll notice that it's not nearly as good as the claimed performance of the F9T. One of the applications I hope to use the F9T for is as a GPS-disciplined 10MHz reference clock for the TICC and other such devices.

This is a screen snapshot of the output of the TICC as it timestamps the 1PPS time pulses from the Z9T. You can see the difference between the time pulses are in generally the range of single digit nanoseconds. Given that its reference clock has a claimed precision that is about an order of magnitude worse than that of the Z9T, this is pretty good, although it's not clear whether I'm really measuring the Z9T, or the LeoNTP.

Digilent Analog Discovery 2

The Diligent Analog Discovery 2 is a recent generation USB oscilloscope. The device claims it can sample analog inputs 100 million times a second (100MS/s). Since the 10MHz (100ns wide) time pulse output of the Z9T generates 10MS/s, I had some hope that this handy tool would serve. (Long time readers will know that this is not my first USB oscilloscope. But it is the first one with this high a sampling rate.)

All the sampling and measurement hardware is in the little palm-sized pod, which is USB-connected to a host computer (a Lenovo laptop running Windows 10 in my case) for power and to provide the graphical display. The laptop runs Digilent's WaveForms software.

One great feature of the AD2 that sets it apart from other USB scopes is that it has an optional adaptor board for BNC connectors. This allows you to use conventional oscilloscope probes with it. Or, in my case, BNC-to-SMA cables.

Hint: if you buy an AD2, regardless of what you use it for, you really want the adaptor board. Depending on your application, you may also want to switch the blue jumper blocks, visible here on the adapter, from the AC pins - as shipped - to the DC pins. Failure to do so causes waveforms like the 1PPS time pulse to look really funky, as the underlying circuitry tries to remove the underlying DC component from the AC signal.

The AD2 documentation has some useful information about the device's clock generator, much of which is over my head.

A precision oscillator (IC31) generates a low jitter, 20 MHz clock.
The ADF4360-9 Clock Generator PLL with Integrated VCO is configured for generating a 200 MHz differential clock for the ADC and a 100 MHz single-ended clock for the DAC.

Analog Devices ADIsimPLL software was used for designing the clock generator. The PLL filter is optimized for constant frequency (low Loop Bandwidth = 50 kHz and Phase Margin = 60°). 

The Phase jitter using a brick wall filter (10.0 kHz to 100 kHz) is 0.04° rms.

The AD2 is capable of being both an oscilloscope and a wave form generator. I'm guessing the former uses the Analog-to-Digital Convertor (ADC), and the latter uses the Digital-to-Analog Convertor (DAC), mentioned above. (The AD2 can also be used as a digital logic analyzer, suitable for commonly used external interfaces on microcontrollers and embedded processors, like SPI, I2C, and UART.)

Low Frequency Events

Pointing the AD2 at the 1PPS time pulse output of the F9T yielded the following wave form.

Once I collected some 1PPS output samples in the AD2, I used the software's built in capability to measure the period between two successive pulses.

The AD2's measurement of the pulses being 1.002 seconds apart (a period equivalent to a frequency of about 9.98MHz) is not great, and not typical of the timing reported by the TICC. Once again, it's not clear I'm measuring the F9T, or the goodness of the AD2. But given the AD2's sampling rate of 100MHz, I would expect a measurement error of as much as ±10ns, once again not as good as the F9T's claimed ±4ns. (You're probably detecting a theme here.)

Cranking the display resolution up on the stored samples, in effect zooming in on the graph, the WaveForms tool shows what the docs say is a "noise band indicating glitch or higher frequency components than the sampling frequency". (It's not clear to me how the device knows anything about what's going on above its sampling frequency.)

Cranking the resolution a little higher shows a graph that indicates the actual points at which the 1PPS wave form was sampled. Now we can see some ground truth (which was probably already obvious to the hardware folks): the display consists of straight lines between successive sampling events. The rise of the 1PPS output line from ground to VDC (the 3.3V logic level of the F9T) cannot be instantaneous, and in fact is captured by the 100MHz sampling rate of the AD2. This explains the slight jog in the leading edge of some of the 1PPS pulses that is visible in the lower resolution displays. (This insight will be even more useful shortly.)

High(er) Frequency Events

Pointing the AD2 at the F9T's 10MHz time pulse output, we would expect see even more measurement artifacts, since the AD2's sampling rate is merely ten times the frequency of the signal we are measuring.

The 10MHz time pulse is a digital signal, but not a sine wave. But its visualization by the AD2, and the fact the output signal cannot change instantaneously, makes it look like one.

Measuring the period of the time pulse becomes even more challenging. At what point do we compare successive pulses? The TAPR TICC, for example, counts anything that rises above a threshold of 1.7V as a pulse. Doing so here gives us a frequency of about 9.83MHz.

It occurred to me that measuring at the point in the pulse where it starts to rise might be a better indication of the goodness of the F9T. Doing so gives us a frequency just barely short of 10MHz, off by a claimed 0.2ns, probably within the measurement error of the AD2, and certainly within the claimed precision of the F9T. (0.2ns is smaller than the period of the sampling rate, 10ns, so once again it's not clear to me how the instrument knows this.)

Cranking the display resolution a little higher shows us the AD2's sampling points on the 10MHz wave form. (It also shows us why the wave form is a little funky looking, as the straight lines between samples become more obvious.) We can more carefully choose the data points that we used to measure the interval between time pulses.

It occurs to me (again, probably obvious to the hardware folks) that how good the 10MHz output of the F9T appears will depend upon both the input impedance - effectively the speed at which the analog signal can change, due to the electrical characteristics of the circuit - and the measurement threshold (e.g. 1.7V?) of the device that is making use of it.

Conclusions

Neither TAPR TICC nor the Digilent Analog Discovery 2 are good (read: expensive) enough to really judge the quality of the output of the U-Blox ZED-F9T. I'll leave that to the folks with the $25,000 (or more) oscilloscopes with the oven-controlled quartz crystals. However, they were absolutely indispensable at debugging my configuration script for the F9T.  Without them, I would have had no idea if I were even in the ball park with the fairly complex messaging my software was sending to the F9T. For no other reason than this, I recommend cost effective instruments like these for any embedded software developer's toolkit.

As for my own applications of the F9T, my opinion is: if it's good enough for 5G, it's probably good enough for me.

Pocketwatch

Here, in reverse chronological order, are photographs of my latest clock, codenamed Pocketwatch, that joins its kin at the Palatial Overclock Estate. It is based on the Application Development Kit (ADK) for the EverSet ES100-MOD receiver module. 

(Since first publishing this post, I modified the firmware that comes with the ADK. See the update at the bottom.)

The top line on the LCD display is always the time in UTC (or "Zulu" in military nomenclature, hence the "Z"), while the lower three lines scroll with various status messages.

This clock differs from my other disciplined (that is: derives its time and frequency from an external source) clocks. This was was a kit, from Windsor Ontario-based Universal Solder (yes, that's a pun), not my own design. It is not a Network Time Protocol (NTP) server and it is not connected to my home network. It is based on an Arduino-type microcontroller (firmware included), instead of a Raspberry Pi running Linux. And while it is disciplined to the NIST radio time signal, unlike my own "radio clock" (my preferred term to the incorrect "atomic clock" as you often see these devices referred to), this one receives and decodes the newer (and better) Binary Phase-Shift Keying (BPSK) modulated signal from WWVB near Fort Collins Colorado.

Another improvised mounting fixture from one of my favorite maker suppliers, Office Depot. They're used to me now: "What can I help you find?" "Well... I'm looking for something on which I can mount a printed circuit board. I'm not sure what I need, but I'll know it when I see it." "Oh... okay, I'll just let you be, then."

This was a pretty ambitious project, in terms of assembly, for an old software guy like me. Once again I am reminded of the joy of using good tools, like my Weller thermostatically controlled soldering iron, or my PanaVise vise. This build was complicated enough that I was a little surprised (and relieved) that it worked the first time.

Eventually there will be four different printed circuit boards (PCBs): this motherboard, a carrier board for the tiny radio receiver that is on its own additional little board, and the LCD board.

This kit was remarkably inexpensive for what it was. I did have to add an 5VDC power brick to it. I've purchased individual parts from Universal Solder in the past, including the WWVB AM radio module that I used in my own radio clock.

Update (2022-05-06)

Here is my modified ADK displaying the local time for my time zone (MST) taking DST into account (MDT). It should automatically adjust the local time as DST goes into and out of effect.

I modified the firmware that Universal Solder provided pre-burned into the Arduino-like AVR microcontroller. You can now use the S1 and S3 push buttons provided by the ADK to move the time zone ahead or behind UTC by one hour per press. The S2 button can be used to enable or disable Daylight Saving Time (because not all U.S. states in each of its four time zones use DST). You can find my modifications in the Pocketwatch repository on GitHub.

https://github.com/coverclock/com-cranequin-pocketwatch

The Virtual Body Politic

Recently I had the opportunity to educate U.S. legislators on two topics that I'm concerned about: the vulnerability of our encrypted data to - eventually - being cracked by quantum algorithms and why that poses a risk today, and the lack of resiliency in the U.S. Global Positioning System on which so much of our modern infrastructure now depends.

(I continue to correct and amend this article as new information comes to light.)

IEEE-USA Congressional Visits Day

The past few days I've been doing a lot of talking to people - for just a few minutes at a time - about two topics of personal concern: post-quantum cryptography, and GPS resiliency. This was part of IEEE-USA's 2022 Congressional Visits Day. IEEE is the Institute for Electrical and Electronics Engineers, an international professional society to which I belong.

For the CVD, the U.S. arm of IEEE recruits and trains volunteers from its members in congressional districts in every state to meet with legislative aids/assistants (LAs) and other staff of their elected officials in the U.S. Senate and House of Representatives. This year, I was one of these volunteers. At one time this was actually done in person; these days it's done with Zoom - apparently the preferred videoconferencing solution of the U.S. Congress - which is a lot more cost effective considering these meetings last at most half an hour. Be prepared, be brief, be polite, be respectful, be clear, talk fast, or don't talk at all.

Team Colorado (my term) consisted of myself and three other IEEE members from Congressional districts covering municipalities like Boulder, Aurora, Broomfield, Westminster, Arvada, all part of the Denver/Boulder metro area. (IEEE has over 4000 members in Colorado alone, and over 500 in the Congressional district in which I live, CO-7; I live in Arvada, and have worked in Boulder, Westminster, and Broomfield.) All of the teams took more than three hours of Zoom-based training and practice from IEEE-USA's Director of Government Relations, with the assistance of a consulting firm that specializes in helping to plan, schedule, and manage activities just like this.

To be clear, this isn't lobbying. On each team were voting constituents of senators and representatives with whose staff we were meeting. For sure, IEEE-USA hopes at least some of us choose to promote the bills and agendas which which the organization is concerned, like the CHIPS Act, which would promote the rehoming of chip manufacturing back to the United States, and they were careful to let us know what those bills and agendas were. But we're encouraged to talk about whatever we are concerned about (and, hopefully, something our elected officials can do something about), as long as we can do so in just a few short minutes.

After soliciting suggestions from several of my clients and former colleagues in the Boulder, Westminster, and Broomfield areas, post-quantum cryptography and GPS resiliency are the two topics on which I chose to spend my speaking time. Since the President's budget is just now being sent to Congress for debate and compromise, this is an excellent time for such discussions to take place (which I am sure was no accident of scheduling).

Much of my concern has to do with the pressure the currently high rate of inflation has on the actual spending power of those budget dollars. Budget numbers may seem like increases when compared to last year, but are in fact decreases when the effect of inflation is taken into account. This will create incentives to delay or even cancel projects and activities that may seem to be addressing very long term concerns (long term being anything longer than of a term of office), or even that are just hard to understand. Fortunately, explaining stuff is part of my job.

Ultimately Team Colorado met for a half hour each - an extraordinarily generous amount of time, given these people's schedules - with aids and staffers from the offices of

• Senator John Hickenlooper (CO-S),
• Senator Michael Bennet (CO-S),
• Representative Edwin Perlmutter (CO-07, which includes Arvada and Westminster),
• Representative Jason Crow (CO-06, which includes Aurora), and
• Representative Joe Neguse (CO-02, which includes Boulder and Broomfield).

(Rep. Perlmutter is my representative, and I was delighted to meet him in person one day a few years ago as I was doing chores in my front yard and he was canvassing my semi-rural neighborhood.)

Here, in a more verbose form, is what I talked about.

Post-Quantum Cryptography

For sure, this sounds like an extraordinarily esoteric topic. It doesn't help that much of what we hear and read about quantum computing is just hype. Unless there are some unexpected breakthroughs (which is of course possible), we are a decade or maybe two from a quantum computer that isn't at best a research topic.

But once we do, it will be capable of breaking most of the encryption algorithms in common use today. This is because there is already a quantum-based algorithm - Shor's Algorithm - to factor very large numbers. Large number factorization is the basis for most modern encryption algorithms that use public and private keys. There is also a known quantum-based search algorithm - Grover's Algorithm - that can be used to break shared encryption keys.

Why might this be a near-term problem, if we're twenty years away from such a capability? It is believed that hackers are already breaking into commercial and government systems and stealing (I'm told the term of art is exfiltrating) sensitive encrypted data with long-term value, with the expectation that they will eventually be able to decrypt it. They are most likely state sponsored, for example by China, North Korea, Iran, and Russia, and are known as advanced persistent threats. Their sponsors are playing the long game.

The National Institute of Standards and Technology (NIST, part of the Department of Commerce) has already solicited candidates for quantum-resistant encryption algorithms, and the process of evaluating and winnowing down the candidates into just a few is now being done. The next step will be to have working reference implementations that can be tested, so that a handful can be blessed for use. Then to define these algorithms precisely into public standards. Then to implement these new standards into existing cryptographic software products. And finally - the big task - will be to convince everyone to migrate over to them, not just for data "in flight" or "on the wire" but for data "at rest" that has long term value. This could take years, since we're talking about huge existing repositories of stored data that could be compromised. (And it's already too late for the data that has already been exfiltrated, which I am told is the technical term of art).

New cryptographic algorithms are only as good as it takes for someone to find a way to crack them. This is a kind of arms race fought by mathematicians. Adopting quantum-resisted algorithms and migrating our data to them is a big job. The sooner we get started the better off we'll all be.

NIST has requested FY2023 budget to increase their investment in fundamental measurement, quantum science, and measurement dissemination portfolio by $15 million. Providing this budget item remains intact, I'm fairly confident that progress in this area will continue to be made.

Like any threat that is not well understood, this one has brought out a lot of folks claiming to have solved this problem. Some - maybe most - are charlatans, merely taking advantage of FUD: fear, uncertainty, and doubt. Not too long ago, one of my clients asked for my help in evaluating the claims of a start-up pushing its own proprietary cryptographic technology that it claimed to be  quantum-resistant. I'll tell you the same thing I told them: don't buy into it. This area is the realm of Ph.D. mathematicians who specialized in it. If the vendor's claims are legitimate, they will submit their algorithm to peer review through the NIST process. Wait for an officially approved suite of rigorously evaluated and tested post-quantum cryptographic algorithms that have been put into a recognized standard and implemented in the widely used software packages. If you're going to have to go through this costly and time consuming encryption migration process, you want to make sure you are better off when you finish than when you started.

GPS Resiliency

I spent some of my brief allotted time talking about the fragility of the U.S. Global Positioning System (GPS) and how dependent the U.S. was on it for Positioning, Navigation, and Timing (PNT). Virtually every telecommunications technology we care about, from our mobile phones to the internet connections to our homes and workplaces, relies, directly or indirectly, on GPS as a precise timing reference. And precision agriculture depends on GPS, with differential corrections, as a geolocation reference with centimeter precision.

Short term outages - hours, days, maybe even weeks - which can be due to ground or space system failures, deliberate jamming, or sunspots (I personally have witnessed the effects of all of these on my own equipment), aren't a problem, as long as telecommunications providers continue to furnish their systems with highly precise holdover clocks - typically cesium or rubidium atomic clocks. Short term outages do, however, disrupt geolocation applications.

It's the long term stuff - over thirty days maybe - that I worry about. Russia and China have each demonstrated anti-satellite weapons. And it only takes one Mad Russian to decide to burn the world. A study sponsored by NIST estimates that after thirty days of GPS outage, the U.S. would be losing about US$1B a day in lost productivity and capability; far more if it occurred during planting season, such is the impact of geolocation technologies on precision agriculture.

I should also mention that the GPS constellation serves a dual purpose: the satellites also carry the space-based portion of the U.S. nuclear detonation detection system.

In 2017, Senator Ted Cruz (R-TX) sponsored Senate bill S.2220, "National Timing Resilience and Security Act of 2017", co-sponsored by Senator Edward J. Markey (D-MA).

"This bill requires the Department of Transportation to provide for the establishment, sustainment, and operation of a complement to and backup for the timing component of the Global Positioning System (GPS). (GPS satellites contain atomic clocks that provide precise time data and allow GPS receivers to synchronize to those clocks.) The system must: (1) reduce critical dependencies on the GPS network; (2) ensure the availability of uncorrupted and non-degraded timing signals for military and civilian users if GPS timing signals are corrupted or otherwise unavailable; and (3) be land-based, operational in 2 years, and capable of operation for 20 years."

President Donald Trump signed this act with its bi-partisan support into law in 2018. The DoT has provided a roadmap for this effort. The DoT FY 2023 "Budget Highlights" doesn't mention it as far as I can tell. Certainly it was not implemented during the two year deadline. (BTW, this is not the "Wide Area Augmentation System for GPS", which is the WAAS system that virtually all of our GPS systems already use, whether we know it or not.)

The RAND Corporation produced a pretty interesting book on the topic, Analyzing a More Resilient National Positioning, Navigation, and Timing Capability, under contract to the Department of Homeland Security; I bought a nice bound copy from Amazon, but the PDF is available online as well. I have come to agree with the authors of the book that GPS backup is a remarkably difficult problem to solve - well, not technically, but economically and politically. It's hard to come up with a satisfactory solution that does not 

1. put the U.S. government in direct competition with the private sector;
2. provide perverse incentives for the private sector to do something you don't want them to do (like eliminate their expensive holdover clocks from their telecommunications systems); or
3. micromanage the precise solution in a kinda of classically inefficient Soviet way.

Some of GPS's strengths are also weaknesses: it's globally ubiquitous, it's wireless, it generally works well, and it's virtually free: there are no subscription fees, and the receivers are relatively inexpensive and simple to use. This creates a certain amount of dysfunction in the market for geolocation and precision timing services when you find yourself shopping for a commercial alternative. We are, in some respects, victims of the success of GPS.

I don't find much evidence of progress on this topic, other than the RAND study I cited. Bill S.2220, both explicitly and tangentially, refers to the old LORAN (LOng RAnge Navigation) system - a radio geolocation technology based on hyperbolic navigation that was developed around the close of WWII - by virtue of the fact that it talks about the U.S. Coast Guard, which is responsible for the old LORAN system, turning equipment over to the DoT. The RAND study finds enhanced LORAN (eLORAN) adequate for precision timing, but not for all of the precision geolocation applications for which GPS is used. For example, tests in the U.K. demonstrated that, with differential corrections, eLORAN was capable of a timing precision of 50ns, and a positioning precision of less than 10m. (Differential GPS is capable of positioning precision of a centimeter or less, making it useful for precision agriculture.)

(I have read suggestions of using the emerging 5G cellular system as a PNT system. But IMO that begs the question of how the 5G system derives its own timing. If the 5G base stations are disciplined to GPS, that isn't really helpful.)

Funding for the ground-based GPS resiliency effort described in S.2220 have not materialized. I suspect it's another example of an unfunded mandate. In a discussion on this topic in an internet forum, someone remarked that little progress on this is likely to be made unless a crisis occurs. I tend to agree. I am reduced to saying "I am really concerned about this."

It occurred to me that an interesting fictional - I hope - techno-thriller would feature Russia having clandestinely equipped all of their GLONASS satellites - a Global Navigation Satellite System (GNSS) similar in function to the U.S. GPS - with GPS jamming transmitters. Because of its low power (a mere 44.8 watts at an altitude of 20,200 kilometers/12,500 miles), and its use of code division multiplexing (a technology that makes extremely efficient use of precious radio frequency spectrum), GPS is trivially easy to jam. So equipped, Vladimir Putin could flip a switch to turn the unencrypted GLONASS signals off (while leaving its encrypted signals intact) and deny the use of GPS with jamming. (Commercial geolocation devices these days commonly use both GPS and GLONASS; the more expensive and later ones may also use the E.U. Galileo and PRC BeiDou GNSS satellite constellations as well.) This could deny the use of both GPS and GLONASS to commercial geolocation equipment.

Update 2022-04-16

This isn't really my area, but from a practical perspective - and given the lack of competency and relatively primitive equipment displayed by Russian forces in Ukraine - I can't see Russia destroying all thirty-one satellites in the GPS space segment using anti-satellite weapons, no matter what Vladimir Putin threatens. Russia could perhaps destroy enough to make GPS coverage intermittent, despite the U.S. having spare GPS satellites already in orbit. This would have little or no effect on the use of GPS as a precision timing and frequency reference, thanks to Stratum 0 holdover clocks used by telecom providers. It would however have a serious and long term impact on the use of GPS for geolocation: continuous positioning and navigation would no longer be possible.

The worse case scenario is the destruction of one or more GPS space vehicles in Medium Earth Orbit triggering the Kessler Syndrome: debris from the GPS satellite destroying other space assets in MEO, the debris from which causes yet more destruction of other assets also in MEO, in a cascading failure - including, possibly, destroying Russia's own GLONASS satellites. This might be enough of a deterrent to prevent Russia from using such a strategy.

I would think for sure using anti-satellite weapons against GPS space vehicles would be considered an act of war against the U.S. But GPS satellites may be disabled in other ways, some of which I've already described, such as being jammed from orbit. That would not be as escalating as destroying them, and could be maintained for a long enough time to impact their use as timing and frequency references.

Epilogue

I found the IEEE-USA CVD effort time consuming but enlightening. I fell down a rabbit hole or two while researching these topics to provide follow-up information to questions asked by the legislative aids, which included a legal counsel (i.e. lawyer), and a Brookings Institution fellow who specialized (coincidentally) in GPS and space policy. I would do it again. Thanks so much to the hard-working folks in Washington D.C. for their time.

Sources

116th Congress, "CHIPS for America Act", H.R. 7178, 2020-06-11, https://www.congress.gov/bill/116th-congress/house-bill/7178

The White House, "Biden-Harris Administration Bringing Semiconductor Manufacturing Back to America", fact sheet, 2022-01-21, https://www.whitehouse.gov/briefing-room/statements-releases/2022/01/21/fact-sheet-biden-harris-administration-bringing-semiconductor-manufacturing-back-to-america-2/

OMB, Budget of the U.S. Government Fiscal Year 2023, Office of Management and Budget, ISBN 978-0-16-095232-6, 2022, https://www.whitehouse.gov/wp-content/uploads/2022/03/budget_fy2023.pdf

Post-Quantum Cryptography

Patrick Howell O'Neill, "The quest for quantum-proof encryption just made a leap forward", MIT Technology Review, 2020-08-03, https://www.technologyreview.com/2020/08/03/1005891/search-for-quantum-proof-encryption-computing-nist/

Eric Hysen, "Preparing for Post-Quantum Cryptography", PD 140-15, Department of Homeland Security, 2021-09-17, https://www.dhs.gov/sites/default/files/publications/usm_quantum_memo_0.pdf

DHS, "Preparing for Post-Quantum Cryptography", infographic, Department of Homeland Security, 2021-10, https://www.dhs.gov/sites/default/files/publications/post-quantum_cryptography_infographic_october_2021_508.pdf

The President, "Improving the Nation's Cybersecurity", EO 14028, 2021-05-12, Federal Register 86.93, 2021-05-17, https://www.govinfo.gov/content/pkg/FR-2021-05-17/pdf/2021-10460.pdf

DHS/CISA, DoC/NIST, "Post-Quantum Cryptography Frequently Asked Questions", FAQ, Department of Homeland Security/Cybersecurity and Infrastructure Security Agency, Department of Commerce/National Institute of Standards and Technology, 2021-10, https://www.dhs.gov/sites/default/files/publications/post_quantum_cryptography_faq_3_seals_october_2021_508.pdf

Patrick Howell O'Neill, "The US is worried that hackers are stealing data today so quantum computers can crack it in a decade", MIT Technology Review, 2021-11-03, https://www.technologyreview.com/2021/11/03/1039171/hackers-quantum-computers-us-homeland-security-cryptography/

DHS, "Post-Quantum Cryptography", Department of Homeland Security, 2021-12-14, https://www.dhs.gov/quantum

Sankar Das Sarma, "Quantum computing has a hype problem", MIT Technology Review, 2022-03-28, https://www.technologyreview.com/2022/03/28/1048355/quantum-computing-has-a-hype-problem/

DoC/NIST/CSRC, "Post-Quantum Cryptography", Department of Commerce/National Institute of Standards and Technology/Computer Security Resource Center, 2022-03-10, https://csrc.nist.gov/Projects/post-quantum-cryptography

DoC/NIST, "Post-Quantum Cryptography Standardization", Department of Commerce/National Institute of Standards and Technology, 2022-03-10, https://csrc.nist.gov/projects/post-quantum-cryptography/post-quantum-cryptography-standardization

Doc/NIST, "FY 2023: Presidential Budget Request Summary", Department of Commerce/National Institute of Standards and Technology, 2022-03-31, https://www.nist.gov/director/congressional-and-legislative-affairs/nist-appropriations-summary-1/fy-2023-presidential

DoC/NIST, "Fiscal Year 2023 Budget Submission to Congress", National Technical Information Service, 2022-03, https://www.commerce.gov/sites/default/files/2022-03/FY2023-NIST-NTIS-Congressional-Budget-Submission.pdf

NCCoE, "Migration to Post-Quantum Cryptography", Department of Commerce/National Institute of Standards and Technology/National Cybersecurity Center of Excellence, https://www.nccoe.nist.gov/crypto-agility-considerations-migrating-post-quantum-cryptographic-algorithms

Wikipedia, "Shor's algorithm", 2022-04-09, https://en.wikipedia.org/wiki/Shor's_algorithm

Wikipedia, "Grover's algorithm", 2022-04-01, https://en.wikipedia.org/wiki/Grover's_algorithm

GPS Resiliency

115th Congress, "National Timing Resilience and Security Act of 2017", S.2220, https://www.congress.gov/bill/115th-congress/senate-bill/2220

Office of Senator Ted Cruz, "President Signs Sens. Cruz, Markey's Bipartisan National Timing Resilience and Security Act Into Law", PR, 2018-12-04, https://www.cruz.senate.gov/newsroom/press-releases/president-signs-sens-cruz-markey-and-rsquos-bipartisan-national-timing-resilience-and-security-act-into-law

Alan C. O'Conner et al., "Economic Benefits of the Global Positioning System (GPS)", RTI International, 0215471, 2019-06, https://www.nist.gov/system/files/documents/2020/02/06/gps_finalreport618.pdf

Michael P. Gallaher, "Economic Benefits of the Global Positioning System (GPS), RTI International, 2019-11-20, https://www.gps.gov/governance/advisory/meetings/2019-11/gallaher.pdf

The President, "Strengthening National Resilience Through Responsible Use of Positioning, Navigation, and Timing Services", EO 13905, 2020-02-12, Federal Register, 85.32, 2020-02-18, https://www.govinfo.gov/content/pkg/FR-2020-02-18/pdf/2020-03337.pdf

Dana A. Goward, "How America is losing the GPS war - and risking everything", Association of Old Crows webinar, Resilient Navigation and Timing Foundation, 2022-03-26, https://rntfnd.org/wp-content/uploads/AOC-Losing-GPS-War.pdf

Dana A. Goward, "How America is losing the GPS war - and risking everything", transcript, Resilient Navigation and Timing Foundation, 2022-03-26, https://rntfnd.org/wp-content/uploads/Goward-AOC-Text-March-2020.pdf

Editor, "Great Questions! Q&A from 'How America is losing the GPS war' (Part 1)", Resilient Navigation and Timing Foundation, 2022-03-29, https://rntfnd.org/2020/03/29/great-questions-q-a-from-how-america-is-losing-the-gps-war-part-1/

Editor, "Great Questions! Q&A from 'How America is losing the GPS war' (Part 2)", Resilient Navigation and Timing Foundation, 2022-03-31, https://rntfnd.org/2020/03/31/great-questions-q-a-from-how-america-is-losing-the-gps-war-part-2/

DHS, "Report on Positioning, Navigation, and Timing (PNT) Backup and Complementary Capabilities to the Global Positioning System (GPS)", Department of Homeland Security, 2020-04-08, https://www.cisa.gov/sites/default/files/publications/report-on-pnt-backup-complementary-capabilities-to-gps_508.pdf

DOT, "National Timing Resilience and Security Act: Roadmap to Implementation", Report to Congress, Department of Transportation, 2021-01, https://www.transportation.gov/sites/dot.gov/files/2021-01/NTRSA%20Report%20to%20Congress_Final_January%202021.pdf

Stefano Ruffini et al., "5G synchronization requirements and solutions", Ericsson Technology Review, #01-2021, pp. 2-13, 2021-01-13, https://www.ericsson.com/48e592/assets/local/reports-papers/ericsson-technology-review/docs/2021/5g-synchronization-requirements-and-solutions.pdf

Matteo Luccio, "Opposite and Complementary - eLORAN is Part of the Solution to GPS Vulnerability", GPS World, 32.11, pp. 24-27, 2021-11, https://cdn.coverstand.com/59713/726592/fb31a974139579735778a7f202a3dbfe67d57f63.1.pdf 

Richard Mason et al., "Analyzing a More Resilient National Positioning, Navigation, and Timing Capability", Research Summary, Homeland Security Operational Analysis Center, RAND Corporation, RR2970, 2021, https://www.rand.org/content/dam/rand/pubs/research_reports/RR2900/RR2970/RAND_RR2970.summary.pdf

Richard Mason et al., "Analyzing a More Resilient National Positioning, Navigation, and Timing Capability", National Defense Authorization Act Fiscal Year 2017 Report to Congress: PNT Requirements, and Analysis of Alternatives, Homeland Security Operational Analysis Center, RAND Corporation, RR2970, 2021, https://www.rand.org/content/dam/rand/pubs/research_reports/RR2900/RR2970/RAND_RR2970.pdf

Richard Mason et al., Analyzing a More Resilient National Positioning, Navigation, and Timing Capability, RAND Corporation, 2021, https://amazon.com/dp/197740362X

Matteo Luccio, "10 Questions on eLoran", GPS World, 33.1, pp. 6-8, 2022-01, https://cdn.coverstand.com/59713/734537/8a8ce04baadba3d98c80aceae2e0c5011c4dc55c.2.pdf

Wikipedia, "Anti-satellite weapon", 2022-03-13, https://en.wikipedia.org/wiki/Anti-satellite_weapon

DoT, "U.S. Transportation Secretary Pete Buttigieg Announces the President’s Fiscal Year 2023 Budget for the U.S. Department of Transportation", Department of Transportation, 2022-03-28, https://www.transportation.gov/briefing-room/us-transportation-secretary-pete-buttigieg-announces-presidents-fiscal-year-2023

DoT, "Budget Highlights 2023", Department of Transportation, 2022, https://www.transportation.gov/sites/dot.gov/files/2022-03/Budget_Highlights_FY2023.pdf

Dana A. Goward, Rep. John Garamendi, "Putin is holding GPS hostage - Here's how to get it back", opinion, C4ISRNET, 2022-04-12, https://www.c4isrnet.com/opinion/2022/04/12/putin-is-holding-gps-hostage-heres-how-to-get-it-back/

Matteo Luccio, "Russia's Attack Raises Vulnerability Concerns", GPS World, 33.4, p. 6, 2022-04, https://cdn.coverstand.com/59713/744206/617d13bd681cf9f34129871033fa60662b9121e5.2.pdf

Alan Grant, Dana Goward, "Ten Answers About eLoran", GPS World, 33.4, pp. 28-33, 2022-04, https://cdn.coverstand.com/59713/744206/617d13bd681cf9f34129871033fa60662b9121e5.2.pdf

Supply and Demand

I've been using the Raspberry Pi single board computers (SBC) ever since the Model 1 (see below) came out in 2012. At about US$40 for a board, it is often easier just to buy a new board for a new project, rather than taking apart an existing project and sacrificing its SBC. Today, I'm buying Raspberry Pi 4B boards, still around US$40 each - more if you want more memory, and who doesn't?

For a long time now I've been having an intermittent problem with one of my projects, Obelisk, my WWVB Radio Clock, a desk clock/NTP server I built using a Raspberry Pi 3B and tiny AM receiver board that picks up the NIST WWVB digital time signals from the transmitter near Fort Collins Colorado. Obelisk drops off my home WiFi network, after which it becomes impossible to diagnose since I can't log into it. I power cycle the unit, and it works fine for another month or so until it fails again. Since the clock and display keep running (meaning my software on the 3B is still running), this looked to be a problem with the WiFi chip on the 3B. I have about a dozen Raspberry Pis around the Palatial Overclock Estate that run 24x7, and this is the only one that seems to have this problem.

I finally decided to replace the Raspberry Pi in Obelisk. I figured I would just order another 3B board, only to be taught an important lesson about supply and demand: the cheapest 3B I could find online was US$175, and that was used; a new one was over US$250!

That isn't as crazy as it sounds. Aside from the current semiconductor chip shortage, one of the design changes made in the Raspberry Pi 4B was to reverse the placement of the Ethernet jack with the dual USB A jacks at one end of the board. This not only made all of the older Raspberry Pi cases useless with the new 4B, but made use of the 4B problematic to retrofit into a lot of applications that were originally made with older versions of the SBC. (The 4B also added a second micro HDMI port, so those old cases would have been obsolete anyway... which may explain why the Pi architects swapped the Ethernet and USB A jacks.)

Fortunately I have four plastic storage bins of Raspberry Pi paraphernalia in the basement, ranging from my original Raspberry Pi model 1 to, fortunately, several Raspberry Pi 3Bs. They were all in cases, labeled, and ready to be reused as part of whatever project for which they were originally purchased. But in this case, I figured the only reasonable thing to do was to open up one of the cases and extract the 3B. (So long to Tin, a Raspberry Pi 3B that took part in my Roundhouse IPv6 project.)

It only took me a few minutes, and accidentally breaking a nylon spacer, to swap the 3B in Obelisk with this other one. Will that fix the problem? Only time will tell. But for sure it had never occurred to me that the Little SBC That Could would become a collector's item.

Solar Power

I spent part of the late morning - U.S. Mountain time zone plus Daylight Saving Time or MDT (this will become important shortly) - yesterday diagnosing a major failure in my Differential GNSS test bed.

Differential GNSS (or Differential GPS if it just uses the U.S. satellite constellation) is a technique that uses a fixed GNSS base station at a known location to broadcast corrections to mobile, or rover, GNSS receivers. Since the base station is fixed, it knows that any jitter in its position calculation must be due to environmental factors, like weather in the ionosphere, that produce variations in the travel time of the signals from the satellites. Rovers in the same geographic vicinity that are computing their positions using signals from the same satellites as the base station (and so are likely to be subject to the same environmental conditions), can apply the corrections transmitted by the base station to improve their own calculations, even though they are moving and each of their position calculations are constantly changing. Centimeter position precision is possible, versus the precision of many meters typical of uncorrected GNSS. This is why autonomous vehicles cannot just use un-augmented GNSS for navigation - they can't reliably tell what highway lane they are in. Such precision is however perfectly adequate for cruise missiles.

My test rover had completely lost the ability to get a GNSS fix; it could not see any of the satellites in the GPS (U.S.), GLONASS (Russian), Galileo (E.U.), or BeiDou (Chinese) constellations. Meanwhile, the base station was working just fine. Both units run 24x7 at the Palatial Overclock Estate. The rover antenna sits in a south-side (i.e. sun-facing) window of my home office, while the base antenna sits in a skylight above my kitchen, with an excellent view of the sky above but partially shielded by the peak of the roof. 

My initial thought was that either the amplified antenna or the RF section of the GNSS receiver on the rover had failed. I stopped working on it to go out to lunch with the Spousal Unit. I checked on the rover later when we got back, and it was working fine.

I had just about decided that maybe this was an intermittent failure, when I checked my laptop and saw this email from the Space Weather mailing list of the U.S. National Oceanic and Atmospheric Administration (NOAA) that had arrived while I was at lunch:

Space Weather Message Code: SUMX01

Serial Number: 120

Issue Time: 2022 Mar 30 1824 UTC

SUMMARY: X-ray Event exceeded X1

Begin Time: 2022 Mar 30 1721 UTC

Maximum Time: 2022 Mar 30 1737 UTC

End Time: 2022 Mar 30 1746 UTC

X-ray Class: X1.3

Location: N16W41

NOAA Scale: R3 - Strong

NOAA Space Weather Scale descriptions can be found at

www.swpc.noaa.gov/noaa-scales-explanation

Potential Impacts: Area of impact consists of large portions of the sunlit side of Earth, strongest at the sub-solar point.

Radio - Wide area blackout of HF (high frequency) radio communication for about an hour.

17:37 UTC would have been 11:37 MDT. My tax dollars at work, and money well spent, if I may say.

This is not the first time I've detected interesting stuff with my GNSS experiments in Denver. Once, I'm pretty sure I detected pre-announced GPS jamming tests at White Sands test range in New Mexico. Another time, I believe I picked up the testing of a next-generation GPS satellite on a then-unused channel formerly assigned to a decommissioned GPS satellite. 

The U.S. Global Positioning System uses code-division multiplexing. CDM makes extremely efficient use of the radio frequency spectrum, but it is highly susceptible to being trivially jammed. With GPS, this can be done, either deliberately or accidentally, just by RF white noise of the correct gigahertz frequencies. A colleague of mine observed that the effects of this naturally occurring solar weather could easily be misconstrued as GPS jamming. Given the current state of international tensions, such a mistake could lead to unwarranted escalation.

Everyone talks about the space weather, but at least NOAA does something about it.

Layers II

In "Layers", I wrote about one way to think about the architecture of Diminuto ("tiny"), my open-source library of C functions that I find useful for the kinds of systems programming tasks I am frequently called upon to perform, either by my clients or for my own projects. That article considered the layers of Diminuto in a conceptual fashion: documentation, types, logging, errors, assertions, unit tests, wrappers, models, and functional tests. In this article, I'll be taking a more functional approach, writing about the library in terms of projects (repositories), themes (broad capabilities within a project), and features (individual APIs that support those capabilities). The themes, and the features within each theme, are roughly in order from lowest layer to highest (higher layers depend on lower layers).

Disclaimer: The purpose of both this article and the earlier "Layers" is mostly to document the architecture of the library, largely doing so for my own use. Whether you find it useful or even interesting may depend on whether you do the kinds of low-level things I get paid to do, in C or similar systems programming languages, especially in the Linux/GNU/POSIX milieu. If your daily-driver programming language is something like JavaScript or Python (both of which I've also used), most of this kind of stuff will already have been provided in some language-specific package.

In all cases, the features I name correspond to C header files of the same name; for example, the feature diminuto_time corresponds to the header file diminuto_time.h in a standard directory in which the public Diminuto APIs are defined. (There are also private Diminuto APIs that are shared amongst the features themselves.)

All of the features described here have unit tests, some have functional tests (some of which require purpose-built hardware test fixtures), and many are used in command line utilities, that not only test the features, but serve as demonstrations on how the features may be used. Other projects of mine, some of which are mentioned below, are built on top of Diminuto, and are also good examples of how to leverage the library.

These are just a subset of the Diminuto features. I've omitted some of the more obscure ones.

Theme: Types

Most of the Diminuto header files define their own types that are used by that particular feature. But some types are so ubiquitous that their definitions are unified into a single header file. This is especially true for types that may be used to pass function arguments between features, or for types that would otherwise result in circular feature dependencies.

Features:

diminuto_types includes the C library and system header files that define frequently used types like int64_t and size_t. It also defines widely-used Diminuto-specific types like diminuto_ticks_t (more on ticks later) and diminuto_ipv4_t.

Theme: Macros

This theme includes features that define macros that do useful things in the spirit of the standard C sizeof operator. These get used a lot.

Features:

diminuto_countof defines a macro that returns the dimension of a one-dimensional array (not its size).

diminuto_widthof defines a trivial macro the returns the bit-width of a type.

diminuto_typeof defines a common macro to infer the type of a variable. It should work with either GNU or ISO compliant C compilers.

diminuto_offsetof defines a macro that computes the byte-offset of a specified field from the beginning of the structure in which it resides.

diminuto_containerof defines a macro that computes the memory address of a container given the structure describing the container and a pointer to a field inside the structure.

diminuto_memberof defines a macro that creates a reference to a specified field in a structure without having a pointer to a structure of that type. It is typically used in conjunction with the sizeof operator.

diminuto_minmaxof defines macros that compute the minimum or maximum possible values of a type, automatically taking into account whether or not the type is signed, for integer types that use two-complement encoding.

diminuto_cxxcapi, diminuto_begin, and diminuto_end define macros that simplify the integration of C and C++ code bases. ("cxxcapi" may be pronounced "sexy API".)

diminuto_new and diminuto_delete (added 2023-03) dynamically allocate and free objects of a caller-specified type, and invoke caller-defined constructor and destructor functions.

Theme: Data Structures

Features:

diminuto_list implements a circular doubly-linked list object. I have found this data structure to be remarkable flexible and useful.

diminuto_tree implements a self-balancing red-black tree. I based this implementation on code in the U-Boot boot loader, which was authored by the same person who wrote the red-black tree implementation used internally by the Linux kernel. (I don't know which one came first, the U-Boot or the Linux kernel implementation. My code looks nothing like either of those, and all bugs are strictly my own).

diminuto_store implements a keyword-value store using a diminuto_tree.

Theme: Time

One of my complaints about the various Linux/GNU and POSIX library function calls that deal with time is that the units are all different: seconds, microseconds, nanoseconds, etc. And all of the functions use structures, which makes it challenging to use them in computation. Diminuto instead defines a standard unit called a tick that is a sixty-four bit wide integer. There is an unsigned version, diminuto_tick_t, and a signed version, diminuto_stick_t (which is totally okay to pronounce as "stick"). All of the time-related Diminuto features use one of these two types in which to store durations, timeouts, and the time of day. The current resolution of the Diminuto tick is one nanosecond.

Features:

diminuto_frequency provides functions that return the resolution of the Diminuto tick in units of Hertz. All of the other Diminuto features in this theme also define functions that return the maximum resolution of their underlying Linux or POSIX implementation in units of Hertz. This provides applications guidance as to what time-related values are useful.

diminuto_time provides functions to return the time of day in ticks since the POSIX Epoch, the elapsed time of a monotonically increasing clock that is not subject to NTP or CLI adjustments, the CPU time for the calling process, the CPU time for the calling thread, and the value of a logical clock that simply returns a monotonically increasing counter value. With the exception of the logical clock, all functions return values in units of ticks.

diminuto_delay provides a function to delay the calling thread until at least the specified number of ticks have elapsed. There is both an interruptible (by a POSIX signal) and an uninterruptible version. It also defines a function that forces the calling thread to context switch, and another to block the thread until any unblocked signal arrives. (There is also a signal theme that we will discuss shortly.)

diminuto_timer provides functions that use real-time POSIX timers to create and manage timers of both the one-shot and the periodic variety. It also defines functions on top of these that mimic the old setitimer(2) semantics, but with monotonic POSIX timers that (unlike the old mechanism) are not subject to variation due to NTP or CLI time adjustments.

Theme: Utilities

This theme includes features that don't quite fit in anywhere else, but these features are used in many of the higher layers.

Features:

diminuto_log defines macros and provides functions that implement a multi-level logging mechanism that automatically determines whether to write messages to the system log (syslog(3)) if the caller is a daemon, or to standard error (stderr(3)) if it is not. It also includes a way to import what log levels are enabled and disabled from an environmental variable. (As you might expect, this gets used everywhere.)

diminuto_unittest defines macros for a simple unit-test framework. (This gets used in almost all of the Diminuto unit tests, except for a few which predate it.)

diminuto_dump provides functions to write a formatted multi-line hexadecimal dump of application data to a FILE stream like stderr(3).

diminuto_phex provides functions to write application data to a FILE stream like stderr(3), while turning unprintable characters into standard C-style escape sequences. (The name is short for "print hex" and may be pronounced "fex".)

diminuto_escape provides functions to expand single unprintable characters in a buffer into C-style escape sequences, and to collapse C-style escape sequences in a buffer into single characters.

diminuto_fibonacci provides functions to generate Fibonacci numbers. (Yeah, I know this sounds crazy; but Fibonacci numbers lend themselves well to problems in backoff-and-retry error recovery, and progressively larger dynamic resource allocation.)

Theme: State Machines

There is really only one complicated state machine in Diminuto, but it's a useful one.

Features:

diminuto_framer is a state machine that implements a framing mechanism for arbitrary binary octets over sequential data streams. Such streams can be over serial ports (actual serial hardware or USB serial adaptors), TCP stream sockets, or what have you. A frame contains a header with a sixteen-bit length (65535 maximum payload size) and sixteen-bit sequence number (65536 window size), a Fletcher-16 checksum that protects the header, and a Kermit-16 (a.k.a. CRC-CCITT) cyclic redundancy check that protects the payload. The Framer uses a byte-stuffing mechanism borrowed from the High-level Data Link Control (HDLC) protocol. The Framer feature offers a multilevel API that gives the application a lot of discretion about how it wants to approach input and output of its serial data. (Added 2023-05-01.)

Theme: Signals

Signals are a kind of software interrupt widely used in UNIX/Linux/POSIX systems software. There are all kinds of signals, but only a few of the most commonly used are directly supported by features in Diminuto. The API for handling each signal is mostly the same for each feature, except where noted. (I tend to add support for additional signals in Diminuto organically, i.e. when I need them.)

Features:

diminuto_alarm supports the SIGALRM signal: a timer has elapsed.

diminuto_hangup supports the SIGHUP signal: the controlling terminal has disconnected.

diminuto_interrupter supports the SIGINT signal: the user has typed control-C on the controlling terminal.

diminuto_reaper supports the SIGCHLD signal: a child of the receiving process has terminated. Left to its own devices, this feature will also reap the child: collect its final exit state so that the kernel can reclaim its resources.

diminuto_terminator supports the SIGTERM signal: the receiving process has encountered a condition that would otherwise terminate it.

diminuto_uninterruptiblesection defines a set of bracketing C macros - typically a begin macro and an end macro - that can be used to temporarily block or delay the reception of a specified signal. They are used to protect a kind of critical section. (We'll see this same pattern later in other themes and features.)

Theme: Interprocess Communication

Features:

diminuto_ipc provides socket-related functions that are common across all addressing and protocol flavors. Many of the functions in this feature are applicable whether the application knows what kind of socket it is applying them to.

diminuto_ipc4, diminuto_ipc6, and diminuto_ipcl provide functions that simplify establishing socket connections with remote end points, using IPv4, IPv6, or Local (a.k.a. UNIX domain) addressing, respectively. The orientation of the connections may be streams (TCP), datagrams (UDP), or - in the case of Local sockets - sequenced packets (SCTP). This last orientation, sequenced packets, can be used to transfer open file descriptors across process boundaries, and there is a unit test that does exactly that. (Sadly, SCTP, a protocol developed to connect SS7 telephone switches in the PSTN, which combines the reliability of streams with the fixed message boundaries of datagrams, and which is directly supported by the Linux/GNU OS, is not supported by most internet routing hardware, which is why it is only supported in Diminuto for Local sockets.)

diminuto_mux and diminuto_poll provide mechanisms to multiplex file descriptor (including socket) I/O. Both provide the capability to either block indefinitely or to timeout; in the latter case, the timeout duration is specified in (you guessed it) ticks.

diminuto_scattergather provides an (admittedly overly complicated) interface to handle vector I/O.

Theme: Processes

Features:

diminuto_core provides a function that enables an application to produce a core dump if it terminates unexpectedly, and also a function to cause it to terminate expectedly with a core dump if it is enabled.

diminuto_daemon provides functions for the caller to turn itself into a daemon (which does not inherit its parent's file descriptors), or a service (which does inherit its parent's file descriptors), or to run a shell command as a separate process, in the background, detached from the caller.

diminuto_module provides functions to implement a simplified interface to the system dynamic linker. This allows an application to dynamically load and invoke separately compiled functions at run-time. (I totally swiped this idea from the Asterisk open-source PBX.)

Theme: Threads

These Diminuto features are wrappers around the use of pthreads(7) functions. They aren't just conveniences; they implement a specific model of using POSIX threads that I've found broadly useful in other projects.

Features:

diminuto_mutex provides functions to support the use of mutexes for mutual exclusion.

diminuto_condition provides functions to support the use of condition variables.

diminuto_thread provides functions that support the creation, management, and termination of POSIX threads.

diminuto_criticalsection defines a set of bracketing C macros that use a caller-provided mutex to protect a critical section.

diminuto_readerwriter implements a first-in first-out readers-writers synchronization lock.

Theme: File System

Features:

diminuto_fs provides functions to interrogate an object in the file system for its type; to canonicalize a file system path name (e.g. resolve all soft links); to progressively make all the sub-components in a directory path; and to recursively walk the file system, applying a caller-provided function to each object encountered.

diminuto_lock provides a kind of atomic locking capability using a file in the file system. This technique is widely used by daemons.

diminuto_observation provides a mechanism to atomically update a file in the file system such that other processes will never see an intermediate state in the contents of the file. I have used this in other projects to support headless operation of long-running processes.

Theme: Real-Time

This theme addresses issues in the real-time behavior of a system, including traffic shaping (controlling the rate at which events are generated) and traffic policing (controlling the rate at which events are accepted).

Features:

diminuto_throttle implements a mechanism that shapes or polices real-time events using a traffic contract enforced using the Generic Cell Rate Algorithm (GCRA). The traffic contract is specified using an interarrival increment (in the literature, referred to as i), a total threshold or limit (l), an expected interarrival time (x), and an actual interarrival time (x₁). All the parameters are specified in ticks.

diminuto_shaper implements a more complex traffic contract enforced using two diminuto_throttle objects, one to control the peak rate with a specified jitter tolerance, and one to control the sustained rate with a specified maximum burst size.

Theme: Close to Bare Metal, Safeties Off

This theme collects the features that interact more directly with hardware, or with the kernel in ways that may require that the calling application have root privileges.

Features:

diminuto_serial provides a simple set of functions to configure a serial port.

diminuto_cue provides a simple function to debounce a digital signal. (I didn't invent this algorithm, and I admire the person that did.)

diminuto_pin provides functions to control and interrogate General Purpose I/O (GPIO) pins through the standard /sys kernel interface. GPIO pins are manipulated via FILE streams, and hence ultimately use file descriptors, which allows them to be multiplexed, and read via interrupts on the leading or trailing edge of a digital signal.

diminuto_i2c provides convenience functions for communicating with other devices over an I²C bus. (This is one of several features that I test using a breadboard with a purpose-built test fixture.)

diminuto_coherentsection defines a set of bracketing C macros. The begin macro implements a read memory barrier with acquire semantics, and the end macro implements a write memory barrier with release semantics. (It bothers me that I have no real way to test this.)

diminuto_memory provides functions to interrogate the kernel for details about the memory sub-system, like the virtual page size, and the Level 1 cache line size.

diminuto_map provides functions for privileged applications to map physical memory addresses (for example, of memory-mapped hardware devices) to virtual memory addressed, and to release such mappings. (I confess I don't test this very often.)

diminuto_modulator implements a Pulse Width Modulation (PWM) generator using diminuto_timer and diminuto_pin. No guarantees as to its precision and accuracy, since it is running in user space. (I've used this successfully for LED brightness control.)

diminuto_controller implements a Proportional/Integral/Differential (PID) controller, although again no guarantees as to its precision and accuracy, since it is running in user space. (I have used this successfully to implement a feed back loop between a light sensor and an LED, and between an A/D voltage sensor and a D/A voltage output.)

Theme: Command Line Tools

Here is a partial list of Diminuto command line utilities, built using the library, that can be run interactively or from a shell script. (These are indispensable tools in my day-to-day development, testing, and reverse engineering toolbox; they alone might justify use of the library.)

Features:

coreable enables core dumps for the specified command.

dump displays its input in a formatted hexadecimal dump.

endpoint converts an endpoint name consisting of a printable IPv4 or IPv6 address, host name, domain name, or local path name, and a printable port number or service name, into a binary IPv4 or IPv6 address or canonical path name (with all the soft links resolved), and binary port number.

internettool verifies internet connectivity.

log logs messages taken from the command line and/or standard input.

observe watches for an observation file and indicates when it arrives.

phex displays its input in a printable form, changing unprintable characters into C-style escape sequences.

pinchange executes a command when a GPIO pin changes state.

pintool manipulates GPIO pins.

pps (pulse per second) uses pintool to multiplex on a 1PPS GPIO pin.

renametool atomically renames files or swaps files in the same file system.

serialtool manipulates serial ports.

shaper shapes traffic in a shell pipeline.

Other Projects

Briefly I'll mention some of my other projects that make use of Diminuto. This is only a partial list. I'll abbreviate the names of the features just for readability.

Hazer is my project for all things GPS, or more broadly GNSS, and even for some IMU work. Hazer uses Critical Section, IPC4, IPC6, List, Dump, Escape, Lock, Observation, Thread, Tree, many of the Signal handlers, Log, and others. The applications gpstool and rtktool in Hazer especially depend on Diminuto.

Assay is my project that uses the GNU Bison and Flex tools (formerly YACC and Lex in other versions of UNIX) to implement a parser for parameter files, or what are generically referred to by Microsoft and others as parmfiles, containing keyword=value pairs. Assay uses Container Of, Dump, Escape, Tree, and other features not mentioned here. (Be forewarned that everyone has their own idea of what the syntax of a parameter file should be. Assay is no different.)

Codex was my excuse to learn how to use OpenSSL. Codex uses Critical Section, Tree, Delay, IPC4, IPC6, and Log.

Placer was my way of relearning how to use SQLite after being away from it for years. Placer uses Log, Phex, and Fibonacci.

In addition, Diminuto, or portions of it, has shipped in several products I helped my clients develop.