Stranger Android Things

As far back as 2011 I was talking up using Android as a Linux-based embedded operating system. In those days I was running it on a BeagleBoard. I liked that idea then and I still like it now.



Android has a rich set of features, many of which would be really useful in an embedded product at the higher end. It is architected to make use of multiple network devices - for example, WiFi and a cellular modem - which is becoming common in the kinds of things I work on. Android has a mature tool chain and development infrastructure that includes stuff like a graphical IDE and a debugger. It is a well understood platform on which to deploy third-party applications, which at least one of my clients is eyeing as a business model for one of their products. And best of all, Android has enormous market forces incentivizing other folks to do a lot of the work for you.

I never got any traction on that idea. But maybe now that Google is promoting Android Things - as in Android for the Internet of Things - that will change. Android Things (formerly the platform known as "brillo") is a stripped down version of Android that can be run headless. It has the usual Android Run Time (ART) Java Virtual Machine (JVM) plus support for native C and C++ code, so you can port your legacy drivers and libraries, but do your new application development in an object oriented language that has first-class built-in support for threading, synchronization, and lots of other useful stuff.

Or will. Currently Android Things is just a pre-built developer preview that you can load on a selection of inexpensive development boards like the Raspberry Pi 3. It includes drivers and Java APIs for common embedded hardware interfaces like an Inter-Integrated Circuit (I²C) bus, a Serial Peripheral Interface (SPI) bus, Pulse-Width Modulation (PWM) output, and General Purpose Input/Output (GPIO) pins.

Third-party developers have in turn written higher level drivers and Java APIs on top of these for a selection of peripherals on commercially available expansion boards. For the Pi, that would be the Pimoroni Rainbow Hat. The Rainbow Hat has an APA102 LED strip (SPI), a BMX280 temperature and pressure sensor (I²C), two two-character HT16K33 alphanumeric segments (also I²C), a piezoelectric buzzer (PWM), and some discrete LEDs and buttons (GPIO). Below is a photograph of a Rainbow Hat on a Pi running my single threaded RainbowHatDemo application.



The source for Android Things hasn't been released yet, so porting any of my C or C++ libraries will have to wait. But the developer preview, with a Raspberry Pi and the Rainbow Hat, is a great way to see where this is all going. I wrote some toy applications using Android Studio just to try to remember how to use Java, which I haven't done anything serious with in a decade. You can find the repository, code-named Deringer, with all my source code as Android Studio projects, on GitHub. Below is a little code snippet from my multi-threaded RainbowHatThing application that reads the temperature and pressure sensor and logs the results.

    public void run() {
        Log.i(getClass().getSimpleName(), "begin");
        while (!lifecycleDone()) {
            try {
                float[] readings = sensor.readTemperatureAndPressure();
                float centigrade = readings[0];
                float fahrenheit = (centigrade * 9.0f / 5.0f) + 32.0f;
                float hectopascals = readings[1];
                float inches = hectopascals * 0.02953f;
                Log.i(getClass().getSimpleName(), centigrade + "C " + fahrenheit + "F " + hectopascals + "hPa " + inches + "in");
                pause(1000);
            } catch (IOException e) {
                // Do nothing.
            }
        }
        Log.i(getClass().getSimpleName(), "end");
    }

Here is a little video (with audio) of the RainbowHatThing application running many threads concurrently on my tiny test system.



Will Android Things take off as a platform for the more capable IoT targets? No clue. But for sure there are many economic reasons to take this very seriously.

The Cost of Doing Business

My software development career over the past forty-plus years has been all over the map: supercomputers, eight-bit micro controllers, vast distributed telecommunications systems, multi-core microprocessors. The one thing they all have in common - at least at the level I work at near bare metal - is that the skill sets are all basically the same. But one thing that isn't the same is the cost of the tools to debug, functionally test, and validate my work.

Perhaps twenty years ago, I was writing firmware in C and C++ for optical networking products based on ATM and SONET. One night I found myself working late in a lab trying to debug some recalcitrant real-time code. I was surrounded by an H-P broadband analyzer, two H-P ATM protocol analyzers, and a tangle of multimode optical fiber. I realized I had maybe US$300,000 worth of test equipment dedicated to this one task, never mind my high-end workstation and the room-filling racks of telecommunications equipment. I paused for a moment to realize that an organization smaller than Bell Labs might have had a tough time tackling this product development effort, the investment in infrastructure was so significant.



Moore's Law hasn't really changed things that much, because as the tools got cheaper, the systems we develop and debug have gotten larger and more powerful right along with them. Just the other day I was working on my stratum-0 atomic clock project and I realized that my little hobbyist oscilloscope - one of the most useful tools I have ever purchased - was completely inadequate for doing any kind of jitter analysis on the output of the device. After some reflection, I decided I probably didn't know anyone that had an oscilloscope that could measure the jitter of my chip-scale atomic clock. Any jitter that might be detected would surely be in the oscillator used by the measurement instrument itself.



See, our measurement of time is a funny thing. It is a completely synthetic artifact, a singularly human invention. Thanks to natural variation in the Earth's rotation and in its orbit, time isn't even tied to the movement of the Sun any longer. It's defined, in the International System of Units, to the beat of an oscillator based on the cesium-133 time standard. And the thing I was trying to measure was itself an oscillator based on the cesium-133 time standard. The only way we can measure the precision and accuracy of a clock is by using a more precise and more accurate clock. I'm not likely to ever have one of those.

The larger lesson here is this: before you decide to tackle a project, make sure you can get the tools you need for the entire life-cycle of the product. You may find out that the economics of developing a product are substantially different from the economics of debugging, testing, validating, and supporting that product.

Engines of Time

In My Stratum-1 Desk Clock and My Stratum-0 Atomic Clock I described the two clocks that I built: O-1 a.k.a. "Hourglass", and O-2 a.k.a. "Astrolabe". Both act as stratum-1 NTP servers on my home network, along with two commercial units, "Waterclock" and "Sundial", that I have previously described. I mounted both clocks on inexpensive acrylic fixtures from office supply stores that just happened to be about the right size and shape to serve as stands to display my handiwork.

Hourglass is my desk clock sitting on a shelf in my home office. It is a Raspberry Pi 3 using Raspbian and runs the GPS daemon and the NTP daemon. It connects to my home network wirelessly, although the wirefull port works as well. It uses the NMEA output and 1PPS strobe from a Uputronics GPS board to discipline its system clock. It has a real-time clock board that with the push of a button can save the time to a battery-backed clock to preserve its state should it need to be powered off. Thanks to the standard Linux kernel and GNU software, it automatically adjusts to Daylight Saving Time, and GPS itself automatically adjusts to the occasional leap second. As long as it can receive GPS signals from an amplified antenna sitting in my office window, it will be within milliseconds (at least) of Universal Coordinated Time.

Astrolabe sits on a small table in the living room. It is virtually identical to Hourglass in both hardware and software, but replaces the Uputronics GPS board with a Jackson Labs Technologies CSAC GPSDO, a board that has a Microsemi Chip-Scale Atomic Clock that it phase locks to GPS time. Astrolabe receives NMEA output from the GPS chip on the GPSDO board, but gets its 1PPS strobe from the CSAC cesium-133 oscillator. Once the CSAC is initially disciplined to the GPS 1PPS, and the Pi's system clock is disciplined to both GPS and 1PPS, even if it subsequently cannot receive GPS signals from the matchbook-sized amplified antenna sitting in the living room window, it will still be within milliseconds or even microseconds of UTC. Astrolabe also includes a couple of test points: a BNC connector to the 10 MHz output of the CSAC, and an RS-232 serial port to the command interface of the GPSDO.

This is the most accurate and precise timepiece I will ever own, perhaps could ever own, my own stratum-0 atomic clock. Far far better clocks exist, but I am unlikely to ever own them, much less have them in my living room. Its time is traceable to the atomic clocks in the U.S. NAVSTAR GPS satellite constellation in orbit, which are traceable to the U.S. Air Force Master Control Station clock in Colorado Springs Colorado, which is traceable to the Washington Master Clock at the U.S. Naval Observatory in Washington D.C., which is traceable to the NIST-F2 clock at NIST in Boulder Colorado, which is synchronized with atomic clocks all over the world coordinated by the International Bureau of Weights and Measures in Paris.

I have been fascinated by time, time measurement, and time keeping, for decades. Having spent the past forty-plus years working on real-time systems close to bare metal, this was perhaps inevitable. The ability for information to cross the boundary between the real world and the digital world fundamentally depends on the quality of our frequency sources. And our interpretation of reality rests on the accuracy and precision of our clocks.

My Stratum-0 Atomic Clock

In My Stratum-1 Desk Clock I wrote about my quest for accurate and precise timekeeping that lead me to hack together a desk clock that disciplined its system clock against time taken from the U.S. global positioning system (GPS). I have come to think of this first effort as "O-1", inspired by John Harrison, the eighteenth-century carpenter who invented the marine chronometer and many of the fundamental innovations that remain in quality mechanical clocks today. As detailed in Dava Sobel's excellent book Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time, Harrison designed and built a series of chronometers which he named H-1 (for Harrison-1) through H-5, in competition for the prize offered by England's Board of Longitude, an organization that in many ways was the DARPA and NSF of its day. Some other horologists of his time adopted his nomenclature for their own entries into that competition.

But O-1 has a flaw: it is only as good as its signal from the GPS satellites. Should the GPS transmissions, from which the one Hertz 1PPS pulse was synthesized, be unavailable, due to weather or what not, O-1 would be only as good as the standard quartz oscillator used by its Raspberry Pi 3. In time, O-1 would drift away from UTC. What good is that? What I needed was a better, more stable oscillator.

What I needed for O-2 was an atomic clock.

Atomic clocks are not clocks in the sense of telling the time of day or measuring time duration. What they are are extremely stable frequency sources, a mechanism to generate very regular ticks, tied to a fundamental physical property of the universe. Modern atomic clocks have the usual trade off of cost versus accuracy and precision, so that they can range from the big refrigerator-sized units fielded by the U.S. National Institute of Standards and Technology (NIST) costing hundreds of thousands of dollars, to the suitcase-sized commercial units found in telecommunications systems and which cost tens of thousands of dollars (and which from time to time you can find used on eBay for around U.S. twelve to fifteen grand).

All of those were outside my price range.

In 2001, and again in 2008, emboldened by developments in practical manufacturing of micro-electro-mechanical systems (MEMS), and by research by organizations like NIST into exploiting quantum effects to eliminate some of the bulkier portions of classic atomic clock designs, the U.S. Defense Advanced Projects Agency (DARPA) notified the R&D community, in BAA-01-32 and BAA-08-32 respectively, that it was interested in funding research in, and development of, a chip-scale atomic clock (CSAC). A CSAC would be an atomic clock - specifically, a cesium-133 reference oscillator - packaged in a surface-mount device.

The implications of such a device are wide ranging and profound. Precision frequency sources are ubiquitous components in both military and civilian applications in the realm of telecommunications, sensors and instrumentation, navigation, and even munitions. A relatively inexpensive, easily portable, and - it must be said - economically destructible cesium reference oscillator would open up vast possibilities of new mobile devices requiring precision timing and frequency standards.

Your tax dollars at work: San Jose-based Symmetricom, who had already purchased the frequency and time standard business unit of Agilent, formerly Hewlett-Packard, the folks who made the suitcase-sized cesium atomic clocks, developed the SA.45s CSAC. (Symmetricom has since been acquired by its California neighbor Microsemi.) Symmetricom with Las Vegas-based Jackson Labs Technologies, a developer of GPS-disciplined oscillators (GPSDO), integrated the CSAC "physics package" (as it is euphemistically called) with a board that incorporates a u-blox GPS chip and a 32-bit ARM-based microcontroller, to create a CSAC GPSDO. This provides the time of day via the output of the GPS chip, and phase locks the cesium oscillator with the 1PPS derived from the GPS system.



Once the output of the CSAC is initially disciplined against GPS, the board can maintain better than 0.3 parts per billion stability in the absence of a GPS signal. An O-2 built with the JLT CSAC GPSDO could be within a few microseconds of UTC per day, even without GPS.

It sure wasn't cheap, but it was doable.



O-2 runs virtually the same software as O-1, with only minor tweaks. The function of the O-1 GPS board has been replaced by the JLT GPSDO, but the interface to the Raspberry Pi 3 is the same: NMEA 0183 sentences arrive via the serial port, and the 1PPS via a GPIO pin. The interface between the two boards was only complicated by the need for level translation: from RS-232 on the GPSDO to TTL on the O-2 for NMEA, and from 5V logic on the GPSDO to 3.3V logic on the O-2 for 1PPS.

The ARM microcontroller on the GPSDO has its own LCD display (on the right) in addition to the O-2 LCD display (on the left). Here I show it - somewhat redundantly - displaying the time, but a button on the GPSDO can be used to cycle through a number of status displays.

The GPSDO has two serial ports, one read-only providing the raw NMEA stream from the u-blox chip which I use to drive the GPS daemon on the Raspberry Pi, and one read-write serial port used for ASCII commands to the ARM microcontroller on the GPSDO to which I attached a FTDI serial-to-USB convertor connected to my desktop. The GPSDO exports a second serial control interface via its own FTDI chip over the USB connection to the Raspberry Pi that is also used to power the device.

Like O-1 (a.k.a. "hourglass"), O-2 (a.k.a. "astrolabe") doubles as a stratum-1 NTP server on my local area network, accessible via either WiFi or by a CAT5 cable.



(I now have four stratum-1 NTP servers on my LAN, all requiring GPS antennas. I dunno, is that too many?)



O-1 got mounted on a fixture to become a desk clock in my home office, costing hundreds of dollars in materials. I anticipate O-2 similarly becoming a mantle or wall clock, costing thousands of dollars. But both projects were sure awesome learning experiences.

Acknowledgements

Thanks to reader Fazal Majid who shamed me into developing the O-2. Thanks also to Said Jackson and the folks at Jackson Labs Technologies who graciously put up with some questions.

Repositories

https://github.com/coverclock/com-diag-astrolabe

https://www.ntpsec.org/white-papers/stratum-1-microserver-howto/clockmaker

git://git.savannah.nongnu.org/gpsd.git

https://gitlab.com/NTPsec/ntpsec.git

Sources

R. L. Beard, J. D. White, J. A. Murray, "Military Applications of Time and Frequency", Proc. of the 28th Annual Precise Time and Time Interval Systems and Applications Meeting, Reston, Virginia, 1996

Jackson Labs, CSAC GPSDO User Manual, ver. 1.6, 80200506, 2014-03-16

Jackson Labs, CSAC GPSDO Release Notes, rel. 1.2, 80200505, 2012-01-30

Jackson Labs, CSAC GPSDO Operating Recommendations, rev. 1.1, 2016-04-13

J. Jespersen, J. Fitz-Randolph, From Sundials to Atomic Clocks: Understanding Time and Frequency, 2nd ed., Dover, 1999

A. Lal, "Integrated Micro Primary Atomic Clock Technology (IMPACT)", DARPA BAA-08-32, 2008-04-21

Microsemi, SA.45s Chip-Scale Atomic Clock User Guide, rev. C, 098-00055-000, 2016-07

G. Miller, E. Raymond, GPSD Time Service HOWTO, ver. 2.10, 2016-09

E. Raymond et. al, Building GPSD from source, 2016-04-10

E. Raymond et. al, Hacker's Guide to GPSD, 2016-09-02

K. Shenoi, "Clock, Oscillators, and PLLs: An introduction to synchronization and timing in telecommunications", Workshop on Synchronization in Telecommunication Systems, San Jose, CA, 2013-04

D. Sobel, Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time, Bloomsbury USA, 1995

Symmetricom, "Introduction to Symmetricom's QUANTUM(tm) Chip Scale Atomic Clock SA.45s CSAC", 2012 International Technical Meeting of The Institute of Navigation, Newport Beach, CA, 2012-01

Symmetricom, The SA.45S Chip-Scale Atomic Clock, 2011 Sanford PNT Symposium, Menlo Park, CA, 2011-11-18

W. Tang, "A Chip-Scale Atomic Clock", DARPA BAA-01-32, 2001-07-06