When Learning By Doing Goes To Eleven

There are different modes of learning, and each modality works best for different people. I am always envious of my friends and colleagues who can learn by reading a book. That never worked for me. I can only learn by doing, albeit sometimes with a book or web page propped up in front of me. This explains why I have over thirty repositories on GitHub. And why I have a GPS-disclipined stratum-0 NTP server with a cesium chip-scale atomic clock in my living room.



Some of my repos on GitHub are standalone projects, like the stratum-0 server or my learning experiences with Go and Rust. But some of them follow a pattern: my project implements a Linux/GNU-based framework implemented in C as a library and a set of header files, then some unit tests, then some functional tests, then maybe some tools I found useful. Why C? Because for the past few decades, most of my income has come from working on embedded and real-time systems, close to bare metal, in C and sometimes C++. C and C++ have been very very good to me. Although I do admit to some hacking now and then in Java, Python, and occasionally even in JavaScript (although I couldn't write a line of JavaScript from scratch even if my life depended on it).

While more or less wrapping up my most recent C-based learning project, it occurred to me that maybe there was an overall pattern to these C frameworks. I had incorporated decades of experience working in ginormous C and C++ code bases into their architecture and the design of their APIs, being careful to make sure they could be used together. I started to wonder if I was subconsciously working towards some greater goal. So I decided to list these frameworks here, all in one place, for the first time, mostly for my own benefit, in the hope the bigger picture would reveal itself.

So here they are, in no particular order. Some of these projects have been around long enough that I have ported them through four different source code control systems. All of them have been cloned, built, tested, and used on

• x86_64 i7-7567U,
• x86_64 i7-5557U, and
• ARMv7 BCM2835

targets and with

• Ubuntu 18.04 "bionic",
• Ubuntu 19.04 "disco", and
• Raspbian 10 "buster"

platforms, among others (but be sure to read the README files for each repo to see what issues there may be with specific platforms). All of them are licensed by default under version 2.1 of the Lesser GNU Public License (LGPL). Most have associated with them a list of articles I have written about it here in this same blog, and all have - seriously - a playlist of music videos.

Hazer

Description: Parse NMEA and other typical output from GNSS devices.
Inception: 2017
Repository: https://github.com/coverclock/com-diag-hazer
Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA9FMHIG_VmuBCVTQUzB-BZF

Hazer started out as a way for me to learn the National Marine Equipment Association (NMEA) standard used to describe the output of virtually all GPS, and later GNSS, devices. I had encountered NMEA with GPS in many embedded projects over the years, but had never dealt with it from scratch. Hazer was so useful, it evolved into a tool to test GPS/GNSS devices, then to implement other more specific projects like a moving map display and vehicle tracker when integrated with Google Earth, GPS-disciplined NTP servers, and most recently a differential GNSS system with a fixed base station and a mobile rover.

Articles:

• Better Never Than Late, https://coverclock.blogspot.com/2017/02/better-never-than-late.html
• My WWVB Radio Clock, https://coverclock.blogspot.com/2017/10/my-wwvb-radio-clock.html
• A Menagerie of GPS Devices, https://coverclock.blogspot.com/2018/04/a-menagerie-of-gps-devices-with-usb.html
• Practical Geolocation, https://coverclock.blogspot.com/2018/08/practical-geolocation.html
• Practical Geolocation II, https://coverclock.blogspot.com/2018/08/practical-geolocation-ii.html
• GPS Satellite PRN 4, https://coverclock.blogspot.com/2018/11/gps-satellite-prn-4.html
• Monitoring GPS and NTP, https://coverclock.blogspot.com/2018/12/monitoring-gps-and-ntp-yet-another.html

Assay

Description: Parse INI-format configuration files using bison and flex. 

Inception: 2015

Repository: https://github.com/coverclock/com-diag-assay

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA-YIyldvOB56QS-HZed_4g1

Having written parsers and lexical scanners when I was in graduate school (one in Prolog, if you can believe it), long before I ever had access to a UNIX system, I was naturally interested in learning how to use yacc and lex. In some of my commercial work I routinely encountered INI-format configuration files. Those seemed to be generally useful things, with a format that was easier for humans to grok than the XML or JSON files I routinely dealt with too. So I implemented an INI file parser using the GNU counterparts bison and flex, using a syntax that was more or less based on the countless INI implementation I found in the field. The parser includes the ability to run a process and collect its output as the value of the property, which makes for some entertaining capabilities.

Articles:

• Configuration Files Are Just Another Form Of Message Passing, https://coverclock.blogspot.com/2015/03/configuration-files-are-just-another.html

Codex

Description: Provide a slightly simpler C API to OpenSSL and BoringSSL.

Inception: 2018

Repository: https://github.com/coverclock/com-diag-codex

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA8yVIm-Pw5yfTcHNTbhNSux

I had never written C code to directly use the various libraries that each implement the Secure Socket Layer (SSL). And I knew I was in for a lengthy learning experience, as I learned about all the stuff that came along with it: keys and their generation, certificate verification and revocation, encryption algorithms and methods, and so forth. I was fortunate to have a native guide: my old office mate from my Bell Labs days was deeply into all of this, and was generous to a fault with his time and expertise. In the end I not only learned the basics of effectively using OpenSSL and its variants, but also came up with a slightly simplified API that I was confident that I could use in other applications.

Articles:

• Using The Open Secure Socket Layer In C, https://coverclock.blogspot.com/2018/04/using-open-secure-socket-layer-in-c.html

Placer

Description: Automate C-based schema generation for the SQLite 3 RDBMS.

Inception: 2020

Repository: https://github.com/coverclock/com-diag-placer

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA8enrRacXyINR3oUbKBWDSH

My most recent project entailed my getting back into SQLite, the embeddable server-less SQL relational database management system. SQLite is used in every iOS device, every Android device, and (rumor has it) Windows 10. It is said to be the most widely deployed DBMS in the world, and it seems likely that this is true. Many Linux-based embedded products I've worked on in the past used SQLite to manage persistent data. But I'd never coded up an SQLite application myself from scratch. So: yet another learning experience. I always try to accomplish more than one goal with every project; in this one, it was to leverage x-macros - perhaps the most twisted use of the C preprocessor outside of the International Obfuscated C Code Contest - to automate the generation of C structures, functions, and variables to implement user-defined schemas for database tables. I still have a lot to learn about SQLite, but there is a lot of useful knowledge encapsulated in the library, unit tests, and functional tests for this project.

Articles: (none yet)

Diminuto

Description: Implement commonly useful C systems programming capabilities.

Inception: 2008

Repository: https://github.com/coverclock/com-diag-diminuto

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA--8T8OyYiDnLAMWGIirosP

All of the other projects in this list rely on Diminuto to provide common underlying facilities. Diminuto started out as a project to build a minimal Linux 2.4 system on top of an ARMv4 processor. Over time, the framework and library became far more important than the original goal. Portions of Diminuto are now shipping in at least four different products manufactured by various clients (whether they realize it or not). When I implement a feature in any C-based project, I pause and consider whether it might be more generally useful. If I decide that it is, I implement it in Diminuto instead of the root project. Diminuto has thus grown organically to include a bunch of useful stuff, only a small portion of which is listed here.

• a simple mechanism to debounce digital I/O pins;
• a demonization function;
• an API to handle time and data stamps, time and duration measurement, and time delays;
• functions to expand and collapse C-style escape sequences in strings;
• a socket API that supports outgoing and incoming IPv4 and IPv6 connections;
• a logging API that writes to the system log if the caller is a daemon, or to standard error if not;
• a simplified API for socket and file descriptor multiplexing;
• an API to configure serial ports;
• a traffic shaping API based on the virtual scheduling algorithm;
• a red-black tree implementation;
• a simple unit test framework;
• a file system walker.

Articles:

• Some Stuff That Has Worked For Me In C, https://coverclock.blogspot.com/2017/04/some-stuff-that-has-worked-for-me-in-c.html
• When The Silicon Meets The Road, https://coverclock.blogspot.com/2018/07/when-silicon-meets-road.html

Update (2020-03-11)

If you install the doxygen, TeX, and LaTeX packages documented in the Makefile comments for the documentation make target for each of these projects, you can generate HTML and PDF documentation for each library via make documentation readme manuals. The build artifacts will be in subdirectories under out/host/doc in each build directory (with host being replaced with whatever you used for TARGET if you changed it).