Friday, March 31, 2017

My Stratum-1 Desk Clock

Long time readers who remember my article Does anybody really know what time it is? from way back in 2006 will appreciate that I have a deep and abiding interest in timekeeping. So it might come as a surprise to those folks that it has taken me this long to build a stratum-1 desk clock. This is a desk clock so accurate and precise that it is within a few milliseconds of atomic clocks, which aren't really clocks in the usual sense but oscillators that make super precise frequency standards, the best sources of clock ticks that humankind knows how to make.

My Stratum-1 Desk Clock

I cobbled together this little project from
All of the heavy lifting is done by open source software written by other folks. I followed Eric S. Raymond's Stratum-1-Microserver HOWTO and used his clockmaker Python script to do most of the software build and install. As a result, the Pi runs the gpsd Global Positioning System (GPS) daemon and the ntpd Network Time Protocol (NTP) daemon. I administered the WiFi radio on the desk clock so that it can be queried by NTP daemons running on other systems on my local area network. As far as I know, this is the only desk clock I've ever owned that has virtual memory and supports ssh.

The GPS daemon reads the National Marine Electronics Association (NMEA) sentences output from the GPS board via the Pi's serial port. These sentences contain the computed time and date based on timestamps received from both the U. S. NAVSTAR GPS and the Russian GLONASS satellites in mid-earth orbit. The daemon also synchronizes with the GPS board's One Pulse Per Second (1PPS) output via a General Purpose Input/Output (GPIO) pin. This 1Hz heartbeat is derived from the GNSS signals, and is the high technology digital equivalent to the "when you hear the tone, the time will be..." from the old time-of-day telephone service.

The Real-Time Clock (RTC) saves the current time in non-volatile memory when the Pi is shutdown, and keeps ticking off the seconds even when the Pi is powered off, thanks to a battery backup. The time is restored from the RTC when the Pi boots up. This allows the desk clock to know the time and date (within a fraction of a second anyway) before the GPS board can achieve a lock on enough satellites (four at least) to compute the time.

The LCD display is driven by a modest little Python script that I wrote that just queries the Linux system clock five times a second. The NTP daemon keeps the system clock synchronized to GPS Time (GPST) plus the appropriate number of leap seconds offset from Coordinated Universal Time (UTC) as received from the satellites. The GNU library handles the conversion from UTC to local time, depending on the administered time zone, as well as any arcane conversions necessary for Daylight Saving Time (DST).

This is remarkably complex, but all handled by the open source software. The Python script that I wrote is perhaps a dozen lines of code at most. By virtue of the fact that the time maintained by the GNSS satellites is stratum-0 - we'll get to what that means in a moment - and the desk clock is constantly being corrected to follow that clock source, my desk clock is effectively a stratum-1 clock.


There was a time when owning a stratum-1 clock would have been a major financial investment. Today, it's a hobby.

You can find the additional files I modified or added on the Raspberry Pi on GitHub: .

Too Much Information

A little background might be called for to really appreciate my desk clock. I haven't discovered the optimal order to present these factlets, so you will have to forgive a few forward references.

GNSS. There are two fully operational Global Navigation Satellite Systems in medium Earth Orbit (about 12,000 miles altitude). The Navigation Satellite Timing and Ranging (NAVSTAR) Global Positioning System (GPS) constellation was the first one, launched and supported by the U. S. Department of Defense. The Globalnaya Navigazionnaya Sputnikovaya Sistema (GLONASS) is the Russian constellation. BeiDou-2 is the constellation that China is building. Galileo is the constellation the European Union is building. Some GPS receivers only receive NAVSTAR signals; some (like the one in my desk clock) receive both NAVSTAR and GLONASS signals. (Henceforth I shall use the initialism GPS as a generic term.)

Accuracy versus precision. Accuracy is a measure of systematic error. Precision is a measure of random error. The classic example is if you fire arrows at a target and the arrows scatter in and around the bullseye, you are accurate but not precise; if the arrows all hit dead on the same spot that is not the bullseye, you are precise but not accurate. Clocks can be described the same way.

(Updated 2018-04-09) You can see the difference between accuracy and precision illustrated in the diagram regarding offset, jitter, and drift below.

Clocks that are synchronized to GPS are accurate because of the timestamps transmitted by the GPS satellites, which can be read from the NMEA data stream presented by the GPS receiver. They are precise because of the 1Hz 1PPS pulse derived from the GPS transmissions that indicates exactly when the timestamp is correct. Not all GPS receivers emit the 1PPS pulse.

Clock strata. Clocks in information and telecommunication systems are conventionally classified as to their precision, accuracy, and stability into strata. (This classification predates GPS and its use as a clock source.) Stratum-0 clocks are the most accurate and precise clocks we know how to construct: "atomic" clocks that discipline the duration of their ticks by measuring the physical properties of streams of cesium or rubidium atoms. Stratum-1 clocks are those that derive their timing from stratum-0 clocks, and so forth.

(Updated 2017-04-11) Although I don't see it referred to much, ANSI T1.101 (6.1, p. 24, 1987) specified the minimum accuracy for clocks of each strata in dimensionless units, omitting stratum-0; this was in the days of copper (e.g. DS-1) and occasionally optical (e.g. OC-3) telecommunications channels.

  • Stratum-1: 1 x 10-11
  • Stratum-2: 1.6 x 10-8
  • Stratum-3: 4.6 x 10-6
  • Stratum-4: 3.2 x 10-5

Clocks of each stratum are kept accurate and precise by a combination of continual adjustment to their higher-quality lower-stratum clock source, and by their own physical properties. For example, digital electronic clocks discipline the period of their ticks on the vibration of a crystal, which is more consistent if it is in a temperature controlled environment. I have worked on embedded systems which had surface mount clock crystals contained in a tiny oven - a thermostatically controlled chamber that kept the crystal at a constant operating temperature to insure consistent behavior - called an Oven Controlled Crystal Oscillator (OCXO). The inherent quality of a clock is a measure of its ability to holdover - to maintain the correct time during any period when it temporily loses contact with its lower-stratum clock source.

Time of day, duration, and frequency. When we talk about clocks we are really talking about three fundamentally different kinds of things: an instrument which measures the time of day; one which measures time duration; and one that serves as a frequency standard. Sometimes the same device can be used for all three, but in digital information and telecommunications systems, typically not.

When we think of a clock, we probably first think of one that tells us the time of day. In fact, some computer systems refer to such a device as the Time Of Day (TOD) clock. (POSIX likes to refer to this as the real-time clock, but this time is no more, and sometimes a lot less, real than other ways of measuring time.) A TOD clock is useful for knowing when to meet your friends for lunch. But it's usefulness as a measurement of time duration breaks down when it is

  • sprung forward or fallen backward for Daylight Saving Time, or
  • the NTP daemon adjusts the clock to synchronize it with a remote time source, or
  • a leap second is inserted (or subtracted) to re-align the TOD clock with the solar day.

Time Of Day clocks are useful for time stamping events in log files, especially if the clocks on different systems are more or less synchronized, and are even more useful if such timestamps are kept in a common time zone, like Coordinated Universal Time, so that related events in different systems can be easily correlated. But such time stamps are at best only approximately useful for measuring duration, since the TOD clock may be routinely adjusted forwards or backwards while the system is running. The POSIX standard defines the gettimeofday(2) system call to acquire the time of day, but this call is not appropriate for measuring duration for just the reasons cited.

When we think of measuring time duration, we might think of a stopwatch. The POSIX standard defines just such a capability in the clock_gettime(2) system call when it is invoked using CLOCK_MONOTONIC_RAW argument. This provides access to a monotonically increasing hardware clock with which duration can be measured by subtracting successive values from prior values. It is useless for telling time of day, but it is the right way to measure a time duration since it does not change as the TOD clock is adjusted.

All conventional digital electronic systems are synchronous: actions in integrated circuits are triggered by digital pulses generated, directly or indirectly, by the tick of a common frequency standard. Typically this frequency standard is a quartz crystal oscillator - basically the same hardware as in your quartz wristwatch. (It is this oscillator function - the thing that provides the tick of the clock - that the atomic clock provides.) Sometimes it is a periodic digital pulse arriving from an external source, like a communication channel. The entire global telecommunication infrastructure - landlines, internet connections, cellular systems - are in effect all synchronized using a shared frequency standard. A Bell Labs colleague once described this as "getting everyone in the world to jump up and down at the same time". Before GPS was available to provide such a shared standard, using a common constellation of satellites containing atomic clocks, AT&T owned several ground-based atomic clocks, which were used as high quality frequency standards for their network and others. The POSIX standard defines the setitimer(2) system call that can be used to access an interval timer that fires with a caller-defined period.

Offset, jitter, and drift. These are terms used to describe variation between two clocks, or between successive ticks of the same clock. Offset is easy: it is the absolute difference between two clocks. The hard part is that the offset is likely to change as time goes by; that's where jitter and drift come in. The exact definitions of jitter and drift are somewhat open to debate, but here is how I have come to think about them: jitter is a high-frequency, short-term, statistical variation in a clock; drift is a low-frequency, long-term, systematic bias in a clock. Or: jitter is about precision, drift is about accuracy.

Jitter and Drift (JPG)

Jitter in real-time data streams like audio or voice is a small variation in the expected inter-arrival time of the packets, either a little too early or a little too late. With jitter, there is no correlation between successive packets; the next packet is just as likely to be early as late, so that the mean arrival time is correct. As long as jitter is small, packets arriving early can be accommodated by using jitter buffers; the receiver just holds the data until its expected arrival time occurs, then plays it out. However, if a packet of data arrives too late, it may have to be discarded because its "best if used before" time has past and the next packet in the stream is right on its tail, and the user will notice a audible or visible artifact in the play back of the stream.

Drift is most typically unidirectional, although it can also be bidirectional. Let's say you have an interval timer that is supposed to fire once per second - 1Hz. So you expect a timer event to occur at 1 second, 2 seconds, 3 seconds, etc. on the timeline. But the clock that is driving the interval timer is running at a frequency that is 10% too slow, so its period is 10% too long; the first timer event arrives at 1.1 second, the second at 2.2 seconds, the third at 3.3 seconds. The timeline of the events is shifting in the increasing direction. Something similar happens if the clock is 10% too fast, except the events on the time line shift in the decreasing direction. 10% is a big error; if the error in the clock is really tiny, you might not notice the shift in the timeline until a lot of them go by. This kind of thing can happen with the old school RS-232 serial connections because the transmitter and the receiver had no common reference clock. (My digital design friends would say that each end of the serial connection is in a different clock domain.) The received data would seem just fine - because the receiver was still sampling the incoming pulses within the window of correctness - until its clock drifted its sampling into the next character frame, at which point that character would be read incorrectly. That's why RS-232 has start and stop bits at the beginning and end of each character frame, whose purpose is to identify such a framing error so that the bad character can be discarded.

Navigation and time. Global navigation satellite systems know nothing about position. What they know about is time. Every GPS satellite has multiple redundant atomic clocks - cesium and/or rubidium - which are the most precise timepieces humankind knows how to construct. Every GPS satellite is accurate because the onboard atomic clocks are synchronized to within a few nanoseconds of each other by commands from ground stations.

Each GPS satellite continually transmits its unique identity number, a timestamp, and its ephemeris (a description of its orbit). By receiving this information from four different satellites, the receiver can solve four equations in four unknowns - its x, y and z position in space, and its t position in time -  because it can compute its exact distance from each of the four satellites. The receiver then uses an abstract model of the earth - the World Geodetic System - to convert the x, y and z spatial coordinates into something humans find more useful: latitude, longitude, and altitude above mean sea level.

Each satellite also transmits the offset between GPS time (which does not recognize leap seconds) and UTC (which does). This offset changes each time the International Earth Rotation Service (IERS) adds a leap second to UTC. This is done to keep our wall clocks in sync with the rotation of the Earth as it gradually slows down due to the drag of the tidal forces of the Sun and the Moon.

The use of time for navigation in this way is a very old idea. The technology of mechanical clocks themselves was driven by the need for an accurate and precise portable time standard with which to compute longitude when using celestial navigation during the age of sail. Later, in the age of steam, ships captains noted when they heard foghorns, which ran off mechanical clockwork automation, and could use the offset from their ship's clock plus knowledge of the speed of sound to compute their distance from shore. In WWII, Long Range Navigation (LORAN) used radio beacons to similar effect, although before digital computers it required the ship carry an oscilloscope - which in those days was the size of a dorm refrigerator and took three men and a little boy to carry - tied to the LORAN receiver; the operator would measure distances between pulses from LORAN transmitters at known locations to triangulate the ship's position.

Distributed systems and time. Whether you are running a vast distributed network of devices, a few computers in an organization, or just a laptop, there are a lot of advantages to synchronizing the Time Of Day clock on a computer with the Internet at large. That, and a lot of other network protocols depend on it. That's what the Network Time Protocol (NTP), and (on Linux/GNU systems anyway) the NTP daemon (ntpd) does. NTP on the client periodically exchanges time stamps with a remote server, and from those time stamps it computes a clock offset from the server and a round trip time between it and the server. This allows NTP to gradually synchronizes the clock on the local client with that on the remote server, often to within just a few milliseconds, by inserting or removing ticks of the TOD clock. It can do this with a number of remote servers and, by measuring the quality of the received timestamps in terms of offset, jitter, and drift, decide which remote server has the most accurate and precise clock and the best network connection.

While the statistical filters and algorithms used by NTP are beyond my ken, the protocol is not in principle complicated. Measurements are based on four time stamps:

  • T1 is when the client transmitted the request packet,
  • T2 is when the server received the request packet,
  • T3 is when the server transmitted the response packet, and
  • T4 is when the client received the response packet.

The time offset (which can be positive or negative) between the client and the server clocks is
Toffset = ((T2 - T1) + (T3 - T4)) / 2
(the trick to this algebra is the assumption that the travel time for the packet is the same in either direction) and the round trip time between the client and server is
Trtt = (T4 - T1) - (T3 - T2) .
Buying Instead of Building

It took a bit of effort to put this desk clock together. Besides the basic hardware assembly and minor software hacking, the Adafruit LCD board was a kit, so I had to break out the Weller soldering station.

Some Assembly Is Required

But the principle of operation of my desk clock is so simple, you could be forgiven for wondering "Hey, why can't I just buy one of these devices?" Indeed, you can. And for a lot less money than my desk clock cost me to build, taking my hourly rate into account. Cheap enough that you might be tempted, as I was, to actually purchase a couple of inexpensive GPS-based NTP servers and try them out on your home network, comparing them to your home brew device and to each other.

Communication Systems Solutions TM 1000A. The CSS TM 1000A (the TM stands for Time Machines) is US$300 GPS-disclipined NTP server in (literally) a black box.

CSS TM 1000A Time Machine NTP Server

Its front panel has three LEDs indicating power, satellite lock, and the 1PPS heartbeat.

CSS TM 1000A Time Machine NTP Server (back)

The rear panel has a round jack for its dedicated power brick, an RJ45 Ethernet jack, a DB9S serial port for debugging, and an SMA connector for the GPS antenna.

The SMA connector provides a voltage bias that is used to power the amplifier in the remote GPS antenna.

Screen Shot 2017-03-29 at 10.20.09 AM

The TM 1000A is (mostly) easily administered using a password protected web page at its initial address of I say "mostly" because it took about an hour of fiddling on my part to understand that the device's embedded web server didn't work with the Safari browser on my desktop Mac. It wasn't until I plugged a USB-to-serial adaptor into the TM 1000A's debug serial port and watched the diagnostic output that I switched to using Firefox and everything worked just fine. Had I started with Firefox, device setup would have been a matter of five minutes or so.

One peculiarity of the web page is that it reports the signal strength of three satellites it uses to compute its time and space solution. My understanding is that four satellites are necessary for a solution since there are four unknowns: x, y, z and t. Not a big deal, but it does call into question my own understanding of the theory behind all of this.

The TM 1000A responds just fine to requests from NTP daemons on other computers on my LAN. It does not however seem to respond to queries from ntpq, the NTP administration tool. That's not a deal breaker, but it can make debugging NTP problems more difficult. I fired off a query to the CSS tech support email address, but have not heard back yet.
Update 2017-04-03: The CSS folks have verified that the Safari browser is not supported, and neither is ntpq.
I haven't tried this, but the TM 1000A can be configured to emit its 1PPS digital pulse on one of the pins on the serial connector. This means the device can in principle be use to discipline digital clocks on other devices using (for example) a phase-locked loop (PLL).

The best part about the TM 1000A is that you can order it from and have it at your front door in a couple of days.

Leo Bodnar Electronics LeoNTP. The Leo Bodnar LeoNTP is a US$325 GPS-disclipined NTP server that would fit in your shirt pocket.

Leo Bodnar LeoNTP NTP Server

It's front panel has an LCD display and a simple turn/push control whose use is reminiscent of the scroll wheel on the original Apple iPod.

Leo Bodnar LeoNTP NTP Server (back)

The rear panel has a BNC connector, an RJ45 Ethernet jack, a female USB type C connector for power, and an SMA connector for the GPS antenna.

I didn't test it, but the BNC connector can be administered to output the 1PPS digital pulse, allowing it in principle to be used to discipline other digital clocks.

I didn't test it, but the Ethernet jack supports Power over Ethernet (PoE), a common feature of Ethernet switches used in many telecommunications applications, which eliminates the need for a power brick (lots of VoIP phones for example use PoE, requiring only the Ethernet cable be run to the device from a PoE-capable switch).

The SMA connector provides a voltage bias that is used to power the amplifier in the remote GPS antenna.

Leo Bodnar LeoNTP Stratum-1 NTP Server (operational)

The LCD display and the scroll button are the sole means of administering the device. It is so simple to use that it makes you wonder what all the fuss about web pages is about. Administering the device's static IP address was a matter of just a few seconds work. The downside is anyone walking by the device can administer it; there is no security. That's not a problem for me (unless the cats figure it out), but it might be for you. The color of the display, white or orange, indicates whether the device has a satellite lock (white is good). A green dot in the upper left corner of the display blinks at 1Hz to indicate the 1PPS signal. A bar graph indicates the network load on the device as it processes NTP requests.

In theory, the front panel can be administered to display the local time by setting a time zone. However, not my time zone. Which is just as well, since to actually be correct, the device would have to understand Daylight Saving Time in my neck of the woods, which the powers that be are apt to change, requiring a firmware upgrade of the device. I'm happy with it displaying UTC.


The LeoNTP slowly responds to ntpq queries, which is useful for debugging. The LeoNTP reports that it contains a 32-bit ARMv7E-M microcontroller. This suggests that the LeoNTP runs some flavor of real-time operating system (RTOS), or possibly even just a task loop, instead of Linux, which (as we shall see) is the likely reason for its higher precision and lower jitter when reporting the time via NTP.

I ordered my LeoNTP from Uputronics in England (and had it at my front door near Denver Colorado via UPS in just a few days - yet more evidence that I am living in the future), but it's sold in the U.S. and U.K. by other dealers as well.


I now have both off-the-shelf NTP servers wired up to the Ethernet switch in my home office to which everything but my Mac laptop are connected. Both of these servers are plugged into a small UPS so that they can continue to keep time in the event of a brief power outage. The home brew desk clock communicates over WiFi and has a real-time clock with a battery backup.


The TM 1000A (left) is named "waterclock". The LeoNTP (right) is named "sundial". My home brew desk clock is named "hourglass".


I ran coaxial cable behind the bookcases and whatnot in my home office so that I can place three GPS antennas - two patch antennas and an outdoor marine antenna - in my second story office window where they can get a clear view of the sky. (This stage of the project may require an understanding spousal unit.) The transmissions from the GPS and GLONASS satellite constellations are remarkably low power - I'm told they have about the same received power as a night light - and their transmissions can be obstructed by as little as heavy tree cover. I was concerned about the glass window (and now I'm concerned about replacing this window with a more energy efficient model), but this setup seems to work fine: all three NTP servers quickly achieve a satellite lock. Early testing with the GPS board on the Raspberry Pi with a small patch antenna revealed it routinely can see as many as twelve satellites amongst the GPS and GLONASS constellations. The LeoNTP with the big waterproof marine antenna reports it can see fifteen satellites.

I have the NTP daemon on my Ubuntu development system "mercury" configured to query all three NTP servers, plus some other publicly accessible NTP servers on the Internet.


Here is the output of an ntpq query on mercury. Using its own algorithms to measure the quality of each NTP server as a timing source, the NTP daemon on mercury has chosen the LeoNTP as its primary time reference, with the TM 1000A and the Pi as its backups. You can see the wired LeoNTP and TM 1000A have substantially lower round trip delay than the wireless Pi, and all three are a lot lower than any of the Internet servers. The LeoNTP wins in terms of measured jitter; that's why I suspect that the LeoNTP is running an RTOS with a lot less software latency and scheduling variation than you see with a typical Linux system. The use of stratum-1 time sources on the local network causes the NTP daemon on mercury to declare itself a stratum-2 time source.

All three NTP servers have been running for days now with no issues.


I'd be happy with either the TM 1000A or the LeoNTP as a time server on my home network. But the LeoNTP won me over with its clever user interface design, it's pretty front panel LCD display, its support of ntpqand its marginally better performance. It's support of a BNC connector could be a plus in the future; BNC and coax is a common mechanism used for exporting precision timing pulses to other devices. It's lack of security wasn't a concern for me.


There can be no doubt that it is a lot cheaper to go with either of the off-the-shelf NTP servers than building your own. But maybe not as fun. Plus, now, when asked "Does anybody really know what time it is?" I can definitively say "I do!"


2017-04-10: added the jitter and drift diagram.


ANSI, Telecommunications - Synchronization Interface Standards for Digital Networks, T1.101, 1987

E. Kaplan (ed.), Understanding GPS Principles and Applications, Artech House Publishers, 1996

G. Miller, E. Raymond, "GPSD Time Service HOWTO",

D. Mills, Computer Network Time Synchronization (2nd ed.), CRC Press, 2011

D. Mills, J. Martin, J. Burbank, W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithm Specification", RFC 5905, 2010-06

J. Mogul, D. Mills, J. Brittenson, J. Stone, U. Windl, "Pulse-Per-Second API for UNIX-like Operating Systems" (Version 1.0), RFC 2783, 2000-03

C. Overclock, "Does anybody really know what time it is?",

Rakon Limited, "Timekeeping with Quartz Crystals", 2009

Raltron Electronics Corporation, "Stratum Levels Defined",

E. Raymond, "Stratum-1-Microserver HOWTO",

D. Sobel, W. Andrewes, The Illustrated Longitude, Walker & Company, 2003

Wikipedia, "Accuracy and precision",

Wikipedia, "Atomic clock",

Wikipedia, "Error analysis for the Global Positioning System",

Wikipedia, "Foghorn",

Wikipedia, "Global Positioning System",

Wikipedia, "GPS signals",

Wikipedia, "History of longitude",

Wikipedia, "John Harrison",

Wikipedia, "LORAN",

Wikipedia, "Marine chronometer",

Wikipedia, "Network Time Protocol",

Wednesday, March 29, 2017

John Sloan and Internet Protocol version 6

Once again my good friends at Gogo Business Aviation in Broomfield Colorado have been following my work, this time on the Internet Protocol version 6 (IPv6) testbed that I wrote about in Buried Treasure. They invited my alter ego John Sloan to give a talk about it. The fact that they fed us all pizza may have had more to do with the packed house than the topic at hand. But Gogo has a vast system of cell towers - whose antennas point up - and an associated ground network to talk to aircraft that use their Air To Ground (ATG) network connectivity product. So they know a thing or two about networking in the cloud.

The folks at Gogo recorded the talk, made it available on YouTube, and generously allowed me to share it. You can find the slides for this talk on GitHub.

Thanks once again to my colleagues at Gogo Business Aviation for sponsoring this talk. It was fun!