Fun Facts to Know and Tell: RPi on an SSD, Microsoft Linux

As an aside from my main projects - mostly my GPS and LTE-M efforts - I've been working on a couple of things that I am a little surprised to have worked as well as they have (even though they may not be perfect).

Raspberry Pi Running From An SSD

The latest Raspberry Pi board - the 4B - introduced a firmware-based boot loader than can be easily configured to boot from something other than the resident micro SD card. It was pretty easy to configure it to boot from a USB-attached SSD.

Here's an image of one of my Raspberry Pi 4B SBCs sitting on top of a Samsung T5 that it booted from and is running off of. (The T5s have since been replaced by the newer Samsung T7.)

I also found some instructions to enable the SATA TRIM command on the SSD so that the OS could get sectors on the SSD erased as a background activity.

It's not perfect: the SSD pulls enough power that I had to add an externally powered USB hub to run other stuff like GNSS receivers and LTE-M radios. But I'm running a full blown Ubuntu OS instead of Raspberry Pi OS (a.k.a. Raspbian). I run Ubuntu elsewhere on a Raspberry Pi using a micro SD card, but this is a lot more spacious, not to mention a lot faster.

Microsoft's Linux Distribution

Remarkably, Microsoft has created its own Linux distro for the x86_64 target: the Fedora-ish CBL-Mariner (for Common Base Linux). It's intended as an internal Linux distro for Microsoft's Azure cloud infrastructure. I got it running as a virtual machine under VMware Fusion on my old desktop Mac. It works pretty well; I got my Diminuto C library and my Hazer GPS project running on it with very little effort.

It's not perfect: weirdly, it can't display the Unicode degree symbol, which is used in a couple of places in my Hazer code (gotta believe this is just a Unicode font issue). And the IPv6 unit tests in Diminuto don't pass 100% (current thinking this has nothing to do with CBL-Meridian, but with a change I made in my company web site). But otherwise I'm processing input as usual from a GNSS USB dongle.

Update 2022-01-25: The degree symbol display was a day-one bug in my Hazer code, related to setting the locale, that didn't manifest under Debian-based OSes. (If I had followed my own documentation, this would not have occurred.) The IPv6 issues were breakage caused by my own changes in the Digital Aggregates web site to support HTTPS and were unrelated to either CBL-Meridian or support in Diminuto for IPv6.

Human-Machine Teaming and Autonomous Lethal Weapons Systems

I've been doing a lot of reading - and thinking - lately about autonomous lethal weapons systems. I've never helped develop one, but certainly the skills required to do so are in my wheelhouse. I'm philosophically opposed to them; I'm a fan of Isaac Asimov's Three Laws of Robotics. Yet I also believe that autonomous weapons systems are inevitable, and probably necessary. Such systems - e.g. armed autonomous flying drones used in land or sea battles - can go-where and do-what humans cannot. Physics is ruthless.

If our adversaries use them, I don’t see that we will have any choice but to do so as well in order to remain competitive on the battlefield. There’s a strong economic (and possibly even humanitarian, if they can reduce human error and danger to civilian populations) incentive to use autonomous lethal weapons. They will be particularly attractive to smaller first-world states, or any organization exploiting asymmetric warfare. Such systems may be fully autonomous, partially autonomous, or optionally autonomous. Combining a human operator with automation (a term I prefer to "artificial intelligence") is a form of human-machine teaming (HMT).

Lots of people in other domains are thinking about this. In its SAE J3016 standard, the Society of Automotive Engineers defines six levels of driving automation for vehicles, ranging from 0 (fully manual) to 5 (fully autonomous).

(Click on the image to see a larger version.)

It occurs to me that this might be applied to weapons systems as well. Here are some ways J3016 might be adapted to apply to weapons systems.

• 0 - the human operator has full control over the weapon system at all times.
• 1 - automation may assist the human operator with targeting, stabilization, etc.
• 2 - the human operator may relinquish control to the automation but can override its decisions.
• 3 - the automation may take control if it detects the human operator is impaired.
• 4 - the automation operates the weapon but a human operator is still required to approve a kill decision.
• 5 - the automation makes the kill decision without any human guidance or approval.

(H/T to John Stuckey for the inspiration for this.)

Test What You Detonate, Detonate What You Test

Another great article from the national security blog War On The Rocks: "When Software Bugs Go Nuclear: Testing A Digital Arsenal" by Laura Epifanovskaya, a researcher formerly in the U.S. Department of Energy's nuclear weapons program.

Motivated by the fact that the U.S. nuclear weapons stockpile is transitioning from analog to digital control systems - so as to interface with the digital systems in the latest generation of strategic weapons delivery platforms - she writes about the need for formal methods, Design For Test (a design methodology that applies to software as well as hardware), and NASA's motto "Test what you fly, fly what you test", in critical reliability systems.

There is a lot here that is applicable, not just to nuclear weapons, but to any complex high-technology system that absolutely, positively has to work right the first time - and never work when it's not supposed to. Her article also has some interesting tidbits into how nuclear weapons are tested without actually setting off a fusion reaction. Fascinating stuff.

Ghidra 9.2.2

 Ghidra is an open source graphical tool produced by the U.S. National Security Agency (NSA) to analyze binary (compiled) software images for a wide variety of hardware targets and compiler platforms. Amongst its many capabilities are generating a disassembly listing and (remarkably) generating a decompiled listing back into C. This latter case requires that the tool recognize idiomatic instruction sequences produced by common compilers like GNU C, as well as deal with some special cases in which assembler code is inserted inline by the C developer, e.g. for memory barriers, low level synchronization primitives, and accessing special hardware registers.

The NSA released version 9.0 of Ghidra to the public just about two years ago. I wrote a couple of articles (see references below) here about using it to reverse engineer compiled code from my Diminuto, Hazer, and Scattergun projects. I did find a couple of bugs - multiple symbol tables in shared objects confusing the disassembler, and inserted memory barrier instructions confusing the decompiler - reported them, and got useful feedback from one of the developers.

Yesterday I installed the latest public release of Ghidra, 9.2.2, and tried two of the specific cases that were problematic that I wrote about in my two blog articles. Casual perusal of the Ghidra 9.2.2 disassembled and decompiled output for ARM7 binaries suggests both of these issues have been resolved. I updated both articles with remarks to this effect, and included screenshots of the updated output.

Reverse engineering has a lot more applications than just cybersecurity. Analyzing deliberate malware isn't typically my area; but debugging proprietary embedded software, and integrating my and my clients' new code with legacy systems, is an integral part of what I do for a living. Tools like this can be crucial for dealing with undocumented or under documented legacy systems. Reverse engineering tools like Ghidra should be in the toolkits of embedded and other low level systems developers.

References

C. Overclock, "Ghidra", 2019-03-25

C. Overclock, "Decoding Special Machine Instruction Sequences with Ghidra", 2019-03-26

Where the RF Meets the Road

My Hazer project (com-diag-hazer) - the basis for all my GPS/GNSS work that I've written about - depends mightily on my Diminuto project (com-diag-diminuto) - my C systems programming library and framework - for its infrastructure. Testing some of the Hazer features finally got complicated enough that I built a dedicated test fixture for that project, just as I had to build a little test fixture for  Diminuto that I described in Where the Silicon Means the Road.

While Hazer's gpstool utility supports a wide variety of GPS/GNSS receivers, the one I chose for the test fixture is a SparkFun board equipped with a u-blox NEO M9N device. Here is some gpstool output generated while exercising the M9N.

I chose the M9N because it can receive and process transmissions from the satellite constellations of the U.S. GPS (a.k.a. NAVSTAR) system, the Russian GLONASS system, the European Galileo system, and the Chinese BeiDou 2 (a.k.a. COMPASS) system simultaneously. This tests more code paths in the underlying Hazer library and in gpstool.

The M9N is powered by a USB C connection to a dedicated Raspberry Pi 4B running Raspberry Pi OS, the Debian-based version of GNU/Linux formerly known as Raspbian. The USB connection is also the serial communications channel between the Pi and the M9N. The Pi runs headless; when necessary, I ssh into it from my desktop Mac and use the command line interface (although in a pinch, I have connected up a display, keyboard, and mouse to the Pi, or even attached a USB-to-serial adapter to its console port pins).

You can see the SMA Radio Frequency (RF) connector emerging from the container, where it connects to the coaxial cable of the GNSS antenna used by the M9N. The small multi-band active patch antenna sits on the top shelf of my workbench within view of the window.

You can also see a green wire and a blue wire that run from the M9N container to the Raspberry Pi.

The green wire connects the 1PPS ("one pulse per second") signal from the M9N to a General Purpose Input/Output (GPIO) pin on the Raspberry Pi. 1PPS is a standard precision timing signal derived from the GNSS solution by the receiver (although not all of them export it, and when they do, the mechanism varies). I make use of 1PPS in all of my GNSS-disciplined Network Time Protocol (NTP) micro servers that I've written about previously, including the one that incorporates a chip-scale cesium atomic clock. When gpstool is configured to monitor the 1PPS signal, it uses Diminuto's GPIO feature to interrogate the input GPIO pin using the select(2) system call inside a dedicated POSIX thread (which is also a Diminuto feature).

The blue wire connects an second, output, GPIO pin on the Raspberry Pi to an LED in the little container where the M9N is mounted. When gpstool monitors 1PPS, it can be configured for the thread to strobe the second GPIO pin to follow the 1PPS signal. This is not just a test of Hazer and gpstool, but a test of Diminuto's GPIO feature as well.

My hardware engineer colleagues will confirm that I'm not a hardware guy by any stretch of the imagination. But hacking together these little hardware test fixtures gives me peace of mind. As long as I see that LED blinking at 1Hz, I know that, while my code is not perfect, it is at least sane.

Anticipation

You may have read by now that the online vaccine management system that the Trump administration paid US$44M for from the consulting firm Deloitte in a no-bid contract is a disaster. Riddled with bugs like making and then randomly cancelling appointments. States are giving up and cobbling together their own solutions.

Cat Ferguson, "What went wrong with America’s $44 million vaccine data system?", MIT Technology Review, 2021-01-30

About half my career has been spent doing work funded by the U.S. government, either in Big Science or in the Military and Intelligence sector. What I found the most interesting about this mess is one of the reasons Deloitte got the non-competitive contract in the first place. Did they have any experience in large scalable distributed medical computer systems? No. Infectious diseases? Nope. Vaccination programs? Not even. What they had experience in is government contracting. And the procurement systems implemented by the feds make it difficult - not to mention less profitable, compared to other work in the commercial sector - for anyone who isn't already doing government contracting to do government contracting.

I'm reminded of when Google bought Boston Dynamics. The robot company had been mostly funded by DARPA, the U.S. DoD's advanced research arm, the folks that have a track record of bringing us things like the Internet. Once Google bought Boston Dynamics, they declined to renew the DARPA contract. The DoD-related industry was surprised; I read articles in the DoD trade and national security press about this at the time. But I doubt many in the high tech industry even batted an eye. Selling to the Feds used to be a big thing. Now, not so much, compared to the global market. (Google eventually sold the company to Hyundai. Yet another bleeding edge albeit immature technology transferred to a foreign company.)

Another issue that occurs to me is how long it takes to design, develop, test, and deploy a product as big and complex as the proposed vaccine management system. Contrary, I suspect, to expectations in the White House at the time, a commercial effort of this type would take a year or two, all told. Regardless of the sales pitch from any consulting company, there is no fast path. That's why you have to anticipate this kind of need, and address it well in advance of it becoming a necessity. It's like insurance: you have to buy it before you need it, with the understanding - and the hope - that you never actually have to use it.

I don't know how, but we need to do better.

Process

I've been around long enough that I've worked under a lot of development processes: Waterfall, Agile, Kanban, etc. Over the decades I have made the following observations.

1. No one process is going to ensure success.
2. Some processes will work better than others for some kinds of work.
3. What process you use isn't nearly as important as the fact that you have a process at all, everyone knows what it is, and everyone follows it.
4. All development is fractally iterative, whether you want it to be or not.
5. No matter what process you use, you have to have the right people; no process will make the wrong people successful.
6. Engineers need structure, and successful ones know it; if you have the right people, but no process, those people will spontaneously adopt a process - an existing one if they're experienced - that works for them.

When Learning By Doing Goes To Eleven

There are different modes of learning, and each modality works best for different people. I am always envious of my friends and colleagues who can learn by reading a book. That never worked for me. I can only learn by doing, albeit sometimes with a book or web page propped up in front of me. This explains why I have over thirty repositories on GitHub. And why I have a GPS-disclipined stratum-0 NTP server with a cesium chip-scale atomic clock in my living room.



Some of my repos on GitHub are standalone projects, like the stratum-0 server or my learning experiences with Go and Rust. But some of them follow a pattern: my project implements a Linux/GNU-based framework implemented in C as a library and a set of header files, then some unit tests, then some functional tests, then maybe some tools I found useful. Why C? Because for the past few decades, most of my income has come from working on embedded and real-time systems, close to bare metal, in C and sometimes C++. C and C++ have been very very good to me. Although I do admit to some hacking now and then in Java, Python, and occasionally even in JavaScript (although I couldn't write a line of JavaScript from scratch even if my life depended on it).

While more or less wrapping up my most recent C-based learning project, it occurred to me that maybe there was an overall pattern to these C frameworks. I had incorporated decades of experience working in ginormous C and C++ code bases into their architecture and the design of their APIs, being careful to make sure they could be used together. I started to wonder if I was subconsciously working towards some greater goal. So I decided to list these frameworks here, all in one place, for the first time, mostly for my own benefit, in the hope the bigger picture would reveal itself.

So here they are, in no particular order. Some of these projects have been around long enough that I have ported them through four different source code control systems. All of them have been cloned, built, tested, and used on

• x86_64 i7-7567U,
• x86_64 i7-5557U, and
• ARMv7 BCM2835

targets and with

• Ubuntu 18.04 "bionic",
• Ubuntu 19.04 "disco", and
• Raspbian 10 "buster"

platforms, among others (but be sure to read the README files for each repo to see what issues there may be with specific platforms). All of them are licensed by default under version 2.1 of the Lesser GNU Public License (LGPL). Most have associated with them a list of articles I have written about it here in this same blog, and all have - seriously - a playlist of music videos.

Hazer

Description: Parse NMEA and other typical output from GNSS devices.
Inception: 2017
Repository: https://github.com/coverclock/com-diag-hazer
Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA9FMHIG_VmuBCVTQUzB-BZF

Hazer started out as a way for me to learn the National Marine Equipment Association (NMEA) standard used to describe the output of virtually all GPS, and later GNSS, devices. I had encountered NMEA with GPS in many embedded projects over the years, but had never dealt with it from scratch. Hazer was so useful, it evolved into a tool to test GPS/GNSS devices, then to implement other more specific projects like a moving map display and vehicle tracker when integrated with Google Earth, GPS-disciplined NTP servers, and most recently a differential GNSS system with a fixed base station and a mobile rover.

Articles:

• Better Never Than Late, https://coverclock.blogspot.com/2017/02/better-never-than-late.html
• My WWVB Radio Clock, https://coverclock.blogspot.com/2017/10/my-wwvb-radio-clock.html
• A Menagerie of GPS Devices, https://coverclock.blogspot.com/2018/04/a-menagerie-of-gps-devices-with-usb.html
• Practical Geolocation, https://coverclock.blogspot.com/2018/08/practical-geolocation.html
• Practical Geolocation II, https://coverclock.blogspot.com/2018/08/practical-geolocation-ii.html
• GPS Satellite PRN 4, https://coverclock.blogspot.com/2018/11/gps-satellite-prn-4.html
• Monitoring GPS and NTP, https://coverclock.blogspot.com/2018/12/monitoring-gps-and-ntp-yet-another.html

Assay

Description: Parse INI-format configuration files using bison and flex. 

Inception: 2015

Repository: https://github.com/coverclock/com-diag-assay

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA-YIyldvOB56QS-HZed_4g1

Having written parsers and lexical scanners when I was in graduate school (one in Prolog, if you can believe it), long before I ever had access to a UNIX system, I was naturally interested in learning how to use yacc and lex. In some of my commercial work I routinely encountered INI-format configuration files. Those seemed to be generally useful things, with a format that was easier for humans to grok than the XML or JSON files I routinely dealt with too. So I implemented an INI file parser using the GNU counterparts bison and flex, using a syntax that was more or less based on the countless INI implementation I found in the field. The parser includes the ability to run a process and collect its output as the value of the property, which makes for some entertaining capabilities.

Articles:

• Configuration Files Are Just Another Form Of Message Passing, https://coverclock.blogspot.com/2015/03/configuration-files-are-just-another.html

Codex

Description: Provide a slightly simpler C API to OpenSSL and BoringSSL.

Inception: 2018

Repository: https://github.com/coverclock/com-diag-codex

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA8yVIm-Pw5yfTcHNTbhNSux

I had never written C code to directly use the various libraries that each implement the Secure Socket Layer (SSL). And I knew I was in for a lengthy learning experience, as I learned about all the stuff that came along with it: keys and their generation, certificate verification and revocation, encryption algorithms and methods, and so forth. I was fortunate to have a native guide: my old office mate from my Bell Labs days was deeply into all of this, and was generous to a fault with his time and expertise. In the end I not only learned the basics of effectively using OpenSSL and its variants, but also came up with a slightly simplified API that I was confident that I could use in other applications.

Articles:

• Using The Open Secure Socket Layer In C, https://coverclock.blogspot.com/2018/04/using-open-secure-socket-layer-in-c.html

Placer

Description: Automate C-based schema generation for the SQLite 3 RDBMS.

Inception: 2020

Repository: https://github.com/coverclock/com-diag-placer

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA8enrRacXyINR3oUbKBWDSH

My most recent project entailed my getting back into SQLite, the embeddable server-less SQL relational database management system. SQLite is used in every iOS device, every Android device, and (rumor has it) Windows 10. It is said to be the most widely deployed DBMS in the world, and it seems likely that this is true. Many Linux-based embedded products I've worked on in the past used SQLite to manage persistent data. But I'd never coded up an SQLite application myself from scratch. So: yet another learning experience. I always try to accomplish more than one goal with every project; in this one, it was to leverage x-macros - perhaps the most twisted use of the C preprocessor outside of the International Obfuscated C Code Contest - to automate the generation of C structures, functions, and variables to implement user-defined schemas for database tables. I still have a lot to learn about SQLite, but there is a lot of useful knowledge encapsulated in the library, unit tests, and functional tests for this project.

Articles: (none yet)

Diminuto

Description: Implement commonly useful C systems programming capabilities.

Inception: 2008

Repository: https://github.com/coverclock/com-diag-diminuto

Playlist: https://www.youtube.com/playlist?list=PLd7Yo1333iA--8T8OyYiDnLAMWGIirosP

All of the other projects in this list rely on Diminuto to provide common underlying facilities. Diminuto started out as a project to build a minimal Linux 2.4 system on top of an ARMv4 processor. Over time, the framework and library became far more important than the original goal. Portions of Diminuto are now shipping in at least four different products manufactured by various clients (whether they realize it or not). When I implement a feature in any C-based project, I pause and consider whether it might be more generally useful. If I decide that it is, I implement it in Diminuto instead of the root project. Diminuto has thus grown organically to include a bunch of useful stuff, only a small portion of which is listed here.

• a simple mechanism to debounce digital I/O pins;
• a demonization function;
• an API to handle time and data stamps, time and duration measurement, and time delays;
• functions to expand and collapse C-style escape sequences in strings;
• a socket API that supports outgoing and incoming IPv4 and IPv6 connections;
• a logging API that writes to the system log if the caller is a daemon, or to standard error if not;
• a simplified API for socket and file descriptor multiplexing;
• an API to configure serial ports;
• a traffic shaping API based on the virtual scheduling algorithm;
• a red-black tree implementation;
• a simple unit test framework;
• a file system walker.

Articles:

• Some Stuff That Has Worked For Me In C, https://coverclock.blogspot.com/2017/04/some-stuff-that-has-worked-for-me-in-c.html
• When The Silicon Meets The Road, https://coverclock.blogspot.com/2018/07/when-silicon-meets-road.html

Update (2020-03-11)

If you install the doxygen, TeX, and LaTeX packages documented in the Makefile comments for the documentation make target for each of these projects, you can generate HTML and PDF documentation for each library via make documentation readme manuals. The build artifacts will be in subdirectories under out/host/doc in each build directory (with host being replaced with whatever you used for TARGET if you changed it).

This Is What You Have To Deal With

I'm passing this anecdote along not because I'm asking for help (although any insight would be welcome), but as an example of what a low-level real-time development looks like.

I spent several days troubleshooting what I still assume to be a bug in my software that deals with output from Global Navigation Satellite System (GNSS) - of which the U.S. Global Positioning System (GPS) is one example - receivers, regarding the synchronization of the NMEA, UBX, and RTCM state machines with the input stream from the U-blox UBX-ZED-F9P receiver (a U-Blox 9 device) on my development system named Nickel (Intel NUC5i7RYH, Ubuntu 18.04.1).

NMEA is the typical output of GNSS receivers, consisting of "sentences" of printable ASCII characters. UBX is a proprietary binary format used for messaging to and from GNSS receivers manufactured by U-blox. RTCM is a standard binary format used for messaging to and from GNSS receivers that support differential GNSS (DGNSS). When a receiver like the F9P generates all three types of output, your software has to be able to grok all three formats and switch between them seamlessly.

I tested the same code on Cadmium (NUC7i7BNH, Ubuntu 16.04.5) and saw the same symptom: every tens of seconds or so my software would loose sync with the input stream than regain it shortly afterwards.

I could not reproduce the problem on Gold, an ARM-based Raspberry Pi (Broadcom BCM2837B0 SoC, Raspbian 9.4) using exactly the same application software and the same receiver hardware, nor on Bodega or Mochila, also ARM-based Pis (also Broadcom BCM2837B0, Raspbian 9.8).

So I ran the standard Linux utility socat on Nickel to save about a minute's worth of NMEA data (UBX and RTCM weren't enabled on the F9P for this test). I ran my software against the file (so: no real-time constraints) and recorded the Sync Lost messages that included the offset into the data where the errors occurred. At this point I was pretty sure I'd found a bug in the state machines in my software.

Then I did a hex dump of the original file generated by socat, checked at those offsets, and lo and behold the NMEA stream was corrupted: periodically a spurious handful of bytes, looking suspiciously like a fragment of an NMEA sentence, was inserted at the end of a valid NMEA sentence.

So this corruption occurs without my software being involved at all.

I ran my software on Nickel using the GlobalSat BU353W10 receiver (a U-Blox 8 device), and I saw similar occasional loss of sync due to corruption of the NMEA stream. As before, the same software and GPS hardware on a Pi worked without problems.

I updated Cadmium to Ubuntu 19.04 and observed the same misbehavior with the BU353W10. I updated the BIOS firmware on Cadmium to the latest version and it still lost sync every tens of seconds.

I built the same software and used the BU353W10 receiver on Mercury (Dell OptiPlex 7040, Ubuntu 14.04.4) and saw sync lost every tens of seconds.

The evidence points an issue either with the USB hardware or firmware on all of my Intel boxes, or with the USB software stacks in many versions of Ubuntu (even though both Ubuntu and Raspbian are based on Debian, making it likely that their USB stacks share a common provenance).

I'm not buying it. Either seem unlikely. What is the likelihood that I'm the only developer in existence to have run into this? Gotta be pretty much zero. And my search-fu has turned up nothing.

I'd like this to be a bug in my software - and I am still assuming it is, perhaps a flaw shared with socat - because then I could fix it. But its root cause remains a mystery.

I use the Intel-based NUCs as my development systems (the one Intel-based Dell box is a legacy development system), and the Pis as test systems, typically deploying to ARM-based systems in the field.

I have no clue at this point what this is. But when I figure it out - and I hope that I do - I will have learned something useful.

Update (2019-06-19)

Just by way of providing actual evidence, here is a snippet extracted from about one minute of data collected via the command

cat /dev/ttyACM0 > FILE

running on Nickel, the Intel NUC5i7RYH running Ubuntu 18.04.1, from a BU353W10 (U-blox 8) USB-dongle GPS receiver:

$GNGSA,M,3,07,30,08,28,09,11,27,18,51,48,13,01,1.04,0.55,0.88*1D\r\n
$GNGSA,M,3,78,87,88,77,68,69,79,67,81,,,,1.04,0.55,0.88*12\r\n
7A\r\n
$GNGGA,141257.00,3947.65236,N,10509.20014,W,2,12,0.57,1715.2,M,-21.5,M,,0000*4A\r\n
$GNGSA,M,3,07,30,08,28,09,11,27,51,48,13,01,05,1.05,0.57,0.88*12\r\n

The 7A\r\n ('7', 'A', carriage return, line feed) inserted between two sentences is obvious. It has the appearance of a checksum and standard end matter of another NMEA sentence (and in fact there are several sentences following this corruption that happen have 0x7A as their checksum).

Keep in mind that these corruptions don't appear when I run the same test on an ARM-based Raspberry Pi (Update 2019-11-11: specifically a Raspberry Pi 3B+). This makes me wonder if there is another process that runs on the NUC but not the Pi that is consuming data from the same port. This isn't as crazy as it sounds; I have inadvertently run redundant instances of my gpstool software or the gpsd GPS daemon and run into similar issues (yes, I already checked for that). However, running the commands

fuser /dev/ttyACM0

and

lsof | grep ttyACM0

(which displays what process has a particular file open, and all open files and the processes using them, respectively) only show the cat command when I run this test.

An interesting difference between the Ubuntu NUC and the Raspbian Pi is that on the former it isn't necessary to configure /dev/ttyACM0 as a serial port (baud rate, raw mode, etc.) in my application; on the latter, it is (although it doesn't seem to matter to what I set the baud rate).

Update (2019-11-11)

I reproduced this issue on a Raspberry Pi 4B running Raspbian 10 ("buster") using a ZED-F9P (U-Blox 9) receiver chip. It occurs on a regular basis, perhaps once or twice a minute.

It also occurs extremely rarely on a Raspberry Pi 3B+ running exactly the same Raspbian 10 image (but probably with different modules dynamically loaded at boot time) and with the same GPS receiver; on a long term survey that has been running just short of sixteen days, my software has detected two such errors

I reproduced the issue while running the usbmon USB debugger. usbmon allows you to capture the raw USB packets using tcpdump (even though it has nothing to do with the TCP protocol), and then interpret the results using wireshark or its command line equivalent tshark.

The command line sequence I used looked something like this.

sudo modprobe usbmon 

ls /dev/usbmon* 

sudo tcpdump -i usbmon3 -w trace.pcap 

tshark -r trace.pcap -V -x

Here is the snippet from the log file from my gpstool showing a corrupted data fragment in between two valid NMEA sentences. My software merely notes that it was unable to detect the beginning of a valid data frame, in NMEA, UBX, or RTCM format, and had to resync to find the beginning of the next valid frame.

INP [72] $GAGSV,2,2,08,13,23,304,27,15,32,246,37,24,30,058,26,25,40,123,41,7*71\r\n
2019-11-09T21:17:06.058145Z [26254] {b6f79530} Sync Lost 0x0000000000051c5d 0x38 '8'
2019-11-09T21:17:06.058213Z [26254] {b6f79530} Sync NMEA 0x0000000000051cac
INP [72] $GAGSV,2,2,08,13,23,304,23,15,32,246,36,24,30,058,22,25,40,123,41,2*75\r\n

Here is a (very very long) snippet of the usbmon output from tshark for this same section of the data stream. There are a lot more USB frames than there are NMEA sentences; the trick is to find those frames for which the Data length field is non-zero; those will be the frames actually carrying data payloads that gpstool will see. (I have emboldened those fields so you can find them more easily. I have also emboldened the actual payload in the hexadecimal dump.)

Frame 13557: 136 bytes on wire (1088 bits), 136 bytes captured (1088 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044367000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044367000 seconds
    [Time delta from previous captured frame: 0.000064000 seconds]
    [Time delta from previous displayed frame: 0.000064000 seconds]
    [Time since reference or first frame: 180.668568000 seconds]
    Frame Number: 13557
    Frame Length: 136 bytes (1088 bits)
    Capture Length: 136 bytes (1088 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.9.2]
    [Destination: host]
    URB id: 0x00000000ef306840
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1573334226
    URB usec: 44367
    URB status: Success (0)
    URB length [bytes]: 72
    Data length [bytes]: 72
    [Request in: 13526]
    [Time from request: 0.001832000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0
Leftover Capture Data: 2447414753562c322c322c30382c31332c32332c3330342c...

0000  40 68 30 ef 00 00 00 00 43 03 82 09 01 00 2d 00   @h0.....C.....-.
0010  d2 2c c7 5d 00 00 00 00 4f ad 00 00 00 00 00 00   .,.]....O.......
0020  48 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00   H...H...........
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................
0040  24 47 41 47 53 56 2c 32 2c 32 2c 30 38 2c 31 33   $GAGSV,2,2,08,13
0050  2c 32 33 2c 33 30 34 2c 32 37 2c 31 35 2c 33 32   ,23,304,27,15,32
0060  2c 32 34 36 2c 33 37 2c 32 34 2c 33 30 2c 30 35   ,246,37,24,30,05
0070  38 2c 32 36 2c 32 35 2c 34 30 2c 31 32 33 2c 34   8,26,25,40,123,4
0080  31 2c 37 2a 37 31 0d 0a                           1,7*71..

Frame 13558: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044378000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044378000 seconds
    [Time delta from previous captured frame: 0.000011000 seconds]
    [Time delta from previous displayed frame: 0.000011000 seconds]
    [Time since reference or first frame: 180.668579000 seconds]
    Frame Number: 13558
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: host]
    [Destination: 1.9.2]
    URB id: 0x00000000ef306840
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1573334226
    URB usec: 44378
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 128
    Data length [bytes]: 0
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  40 68 30 ef 00 00 00 00 53 03 82 09 01 00 2d 3c   @h0.....S.....-<
0010  d2 2c c7 5d 00 00 00 00 5a ad 00 00 8d ff ff ff   .,.]....Z.......
0020  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13559: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044479000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044479000 seconds
    [Time delta from previous captured frame: 0.000101000 seconds]
    [Time delta from previous displayed frame: 0.000101000 seconds]
    [Time since reference or first frame: 180.668680000 seconds]
    Frame Number: 13559
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.9.2]
    [Destination: host]
    URB id: 0x00000000ef306780
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1573334226
    URB usec: 44479
    URB status: Success (0)
    URB length [bytes]: 0
    Data length [bytes]: 0
    [Request in: 13528]
    [Time from request: 0.001918000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  80 67 30 ef 00 00 00 00 43 03 82 09 01 00 2d 00   .g0.....C.....-.
0010  d2 2c c7 5d 00 00 00 00 bf ad 00 00 00 00 00 00   .,.]............
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13560: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044490000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044490000 seconds
    [Time delta from previous captured frame: 0.000011000 seconds]
    [Time delta from previous displayed frame: 0.000011000 seconds]
    [Time since reference or first frame: 180.668691000 seconds]
    Frame Number: 13560
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: host]
    [Destination: 1.9.2]
    URB id: 0x00000000ef306780
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1573334226
    URB usec: 44490
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 128
    Data length [bytes]: 0
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  80 67 30 ef 00 00 00 00 53 03 82 09 01 00 2d 3c   .g0.....S.....-<
0010  d2 2c c7 5d 00 00 00 00 ca ad 00 00 8d ff ff ff   .,.]............
0020  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13561: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044502000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044502000 seconds
    [Time delta from previous captured frame: 0.000012000 seconds]
    [Time delta from previous displayed frame: 0.000012000 seconds]
    [Time since reference or first frame: 180.668703000 seconds]
    Frame Number: 13561
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.9.2]
    [Destination: host]
    URB id: 0x00000000ef3069c0
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1573334226
    URB usec: 44502
    URB status: Success (0)
    URB length [bytes]: 0
    Data length [bytes]: 0
    [Request in: 13530]
    [Time from request: 0.001579000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  c0 69 30 ef 00 00 00 00 43 03 82 09 01 00 2d 00   .i0.....C.....-.
0010  d2 2c c7 5d 00 00 00 00 d6 ad 00 00 00 00 00 00   .,.]............
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13562: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044508000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044508000 seconds
    [Time delta from previous captured frame: 0.000006000 seconds]
    [Time delta from previous displayed frame: 0.000006000 seconds]
    [Time since reference or first frame: 180.668709000 seconds]
    Frame Number: 13562
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: host]
    [Destination: 1.9.2]
    URB id: 0x00000000ef3069c0
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1573334226
    URB usec: 44508
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 128
    Data length [bytes]: 0
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  c0 69 30 ef 00 00 00 00 53 03 82 09 01 00 2d 3c   .i0.....S.....-<
0010  d2 2c c7 5d 00 00 00 00 dc ad 00 00 8d ff ff ff   .,.]............
0020  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13563: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044519000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044519000 seconds
    [Time delta from previous captured frame: 0.000011000 seconds]
    [Time delta from previous displayed frame: 0.000011000 seconds]
    [Time since reference or first frame: 180.668720000 seconds]
    Frame Number: 13563
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.9.2]
    [Destination: host]
    URB id: 0x00000000efa68780
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1573334226
    URB usec: 44519
    URB status: Success (0)
    URB length [bytes]: 8
    Data length [bytes]: 8
    [Request in: 13532]
    [Time from request: 0.001553000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0
Leftover Capture Data: 382c322a37300d0a

0000  80 87 a6 ef 00 00 00 00 43 03 82 09 01 00 2d 00   ........C.....-.
0010  d2 2c c7 5d 00 00 00 00 e7 ad 00 00 00 00 00 00   .,.]............
0020  08 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................
0040  38 2c 32 2a 37 30 0d 0a                           8,2*70..

Frame 13564: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.044527000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.044527000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000008000 seconds]
    [Time since reference or first frame: 180.668728000 seconds]
    Frame Number: 13564
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: host]
    [Destination: 1.9.2]
    URB id: 0x00000000efa68780
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1573334226
    URB usec: 44527
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 128
    Data length [bytes]: 0
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0

0000  80 87 a6 ef 00 00 00 00 53 03 82 09 01 00 2d 3c   ........S.....-<
0010  d2 2c c7 5d 00 00 00 00 ef ad 00 00 8d ff ff ff   .,.]............
0020  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................

Frame 13565: 136 bytes on wire (1088 bits), 136 bytes captured (1088 bits)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Nov  9, 2019 14:17:06.045116000 MST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1573334226.045116000 seconds
    [Time delta from previous captured frame: 0.000589000 seconds]
    [Time delta from previous displayed frame: 0.000589000 seconds]
    [Time since reference or first frame: 180.669317000 seconds]
    Frame Number: 13565
    Frame Length: 136 bytes (1088 bits)
    Capture Length: 136 bytes (1088 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.9.2]
    [Destination: host]
    URB id: 0x00000000dbf6b180
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x82, Direction: IN
        1... .... = Direction: IN (1)
        .... 0010 = Endpoint number: 2
    Device: 9
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1573334226
    URB usec: 45116
    URB status: Success (0)
    URB length [bytes]: 72
    Data length [bytes]: 72
    [Request in: 13534]
    [Time from request: 0.002069000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000204
    Number of ISO descriptors: 0
Leftover Capture Data: 2447414753562c322c322c30382c31332c32332c3330342c...

0000  80 b1 f6 db 00 00 00 00 43 03 82 09 01 00 2d 00   ........C.....-.
0010  d2 2c c7 5d 00 00 00 00 3c b0 00 00 00 00 00 00   .,.]....<.......
0020  48 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00   H...H...........
0030  00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 00   ................
0040  24 47 41 47 53 56 2c 32 2c 32 2c 30 38 2c 31 33   $GAGSV,2,2,08,13
0050  2c 32 33 2c 33 30 34 2c 32 33 2c 31 35 2c 33 32   ,23,304,23,15,32
0060  2c 32 34 36 2c 33 36 2c 32 34 2c 33 30 2c 30 35   ,246,36,24,30,05
0070  38 2c 32 32 2c 32 35 2c 34 30 2c 31 32 33 2c 34   8,22,25,40,123,4
0080  31 2c 32 2a 37 35 0d 0a                           1,2*75..

You can see, as far as usbmon is concerned anyway, that in between the two valid NMEA sentences, the USB stack is processing a payload consisting of 8,2*70\r\n. This seems pretty obviously to be the last eight bytes of an NMEA sentence, including the checksum, carriage return, and line feed.

Whether this is a hardware or firmware issue with the USB devices on either the U-blox chip or the Raspberry Pi, or some incompatibility between the two, or a software issue in the Linux USB stack, is hard to say. Perhaps only a USB hardware analyzer could say for sure, by examining the USB packets passing between the two devices on the wire. But if this were an issue commonly occurring on devices that express the USB ACM interface as these GPS receivers do, you would think it would be common knowledge.

One colleague suggested I enable modem control (RTS/CTS handshake) in the Linux serial port driver, since a modem is what the USB Abstract Control Model interface looks like to the application. That was easily done since that capability is already supported in my application. It didn't make any difference.

Update (2019-11-11 again)

For purposes of my own documentation, here is the output of the lsusb command on the same Raspberry Pi 4B.

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 009: ID 1546:01a9 U-Blox AG
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And then from a subsequent lsusb -s 1:9 -v command.

Bus 001 Device 009: ID 1546:01a9 U-Blox AG
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            2 Communications
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x1546 U-Blox AG
  idProduct          0x01a9
  bcdDevice            1.00
  iManufacturer           1
  iProduct                2
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x003e
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xc0
      Self Powered
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              0
      CDC Header:
        bcdCDC               1.10
      CDC ACM:
        bmCapabilities       0x02
          line coding and serial state
      CDC Call Management:
        bmCapabilities       0x03
          call management
          use DataInterface
        bDataInterface          1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval             255
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0
      bInterfaceProtocol    255 Vendor specific
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0

Update (2019-11-19)

I went full in and purchased something I should have had a long time ago: a USB analyzer, specifically a Total Phase Beagle USB 12 protocol analyzer. It took about ten minutes of learning how to use it while reproducing this problem to verify that what my software sees, and what usbmon reports, is what is indeed coming over the wire: many valid NMEA sentences followed by what appears to be a partial NMEA sentence consisting of its last few bytes, in this example 8,2*7E.




This is distinct from the case in which an NMEA sentence is split into two successive USB frames, which also occurs and seems to work fine.

I also see an example of a botched NMEA sentence. The sentence starts out correctly, $GLGSV,3, then the rest of the data payload is non-printable characters which seem to have some pattern (but that might be my imagination). The analyzer flags the frame as invalid with an error identified as P7C (no clue, yet).



Then there is a frame with another partial sentence ending after that, and then a complete NMEA sentence split across two frames.

Testing with three ZED-F9P receivers, all on SparkFun GPS-RTK2 boards, I see this problem with corrupted output over the USB interface more frequently the faster processor I use; on an Intel Core i7-5557U @ 3.10GHz x 2 x 2 ("Nickel"), it happens every few seconds; on a Raspberry Pi 4B ARM Cortex-A72 @ 1.5GHz x 4 ("Rhodium"), it happens a few times a minute; on a Raspberry Pi 3B+ ARM Cortex-A53 @ 1.4GHz x 4 ("Gold"), it happens rarely, like once in many hours.

Investigation continues.

Update (2019-11-22)

I see this data corruption about once every five minutes when running on Ubuntu 19.10 inside VMware Workstation 15 Pro on an Lenovo ThinkPad T430s (Intel Core i7-3520M @ 2.90GHz) running Windows 10. Hard to say how the frequency jives with prior results given that it is running inside a virtual machine.