Christmas Pi

I have come to the conclusion that you could replace "climate change" with "IPv6" and get rhetoric on both sides of the debate that would sound plausible. And maybe vice versa. As some of my friends and colleagues have remarked

Bah! I've been hearing the IP address doomsdayers complain for years but people are still hooking up their thermostats to the Internet. IPv6 is a lie. I'll only believe it exists when I see polar bear that can support a 128bit address.

and

Please supply the names of all staff involved in engineering IPv6 readiness.

and even

I for one do not believe that IPv6 is caused by humans. 

See what I mean? In any case, I decided it was time for me to see what IPv6 had to offer.

(The following was updated on 2017-01-19.)



Roundhouse is my IPv6 router than runs the open source OpenWrt on a Raspberry Pi 2, and manages an IPv6-through-IPv4 WAN tunnel via Hurricane Electric, an (among other things) IPv6 tunnel provider. The router exposes the IPv6 WAN over an Ethernet switch on the upper right.

You can find the Roundhouse project, which is mostly just a makefile and some configuration, here:

http://github.com/coverclock/com-diag-roundhouse

The other Raspberry Pi 2 and 3 systems shown here, scrounged from other projects, form the bulk of my IPv6 testbed.

Tin has as its only network connection an LTE cellular modem through which it uses IPv6. I use it for penetration testing of my IPv6 firewall and to demonstrate the global routing of IPv6 global unicast addresses.

Zinc is normally used for testing software that handles GPIO and serial ports, but here it accesses systems on the IPv6 testbed from my production IPv4 network.

Bronze is normally my 32-bit ARM reference platform, but here it demonstrates a dual stack capability by using IPv4 on the IPv6 LAN.

Copper accesses IPv6 devices using a variety of mechanisms including IPv6 global unicast addresses, v4-mapped IPv6 addresses, and IPv6 local unique addresses.

Lead acts as a server, accepting datagrams over UDP or connections over TCP, and echoing any data it receives back to the sender.

I've been expanding Diminuto, my open source C/Linux/GNU-based systems programming library, to support its socket API on IPv6. It wasn't difficult. This library includes the software I've been running on on the Raspberry Pis. Portions of Diminuto have found their way into several commercial products. You can find it here:

http://github.com/coverclock/com-diag-diminuto

Finally I'm still collecting and testing hardware entropy generators, a project called Scattergun that my alter ego recently gave a talk about. I hope to one day write more about that.



You can find Scattergun here:

http://github.com/coverclock/com-diag-scattergun

I hope this convinces you that I have not been slacking off during the holidays.

Best wishes to you and yours. Season's greetings!

(And an especially jolly Ho Ho Ho and Thank You to Alan, Doug, and Dale for their IPv6 comments.)

Evolution

Like most technologists, my home is an informal museum of the history of computing. For many years I had been running Asterisk, an open source IP PBX, on an ancient Dell Dimension 3000 personal computer. Here is the PC after being decommissioned, along with its VGA monitor and its PS2 keyboard and mouse.



In an effort to clear out some of the older computer gear, I replaced the Dell with an Asterisk appliance: a Grandstream UCM6102, which is a purpose-built off-the-shelf IP PBX based around an embedded processor with an ARM core. It's about the size of a paperback book... or a Kindle, if you prefer.



The new device supports my analog business line that I still insist on maintaining, and even an old Radio Shack analog phone that is an antique itself and which I use just for testing. It works just fine with the SIP phone in my home office. My dial plan changed a bit, as I accommodated the new system's way of doing things. Since I'm the only user, this was a minor issue.

I took the old PC to be recycled. Just before it was carted off, I used my iPhone to take a photograph of its company asset tag, and of the Dell service tag on the back. It was only when I got home that I realized it was Digital Aggregates Corporation asset #1 (although that probably has more to do with when I started using asset tags to keep track of company property than anything else.)



I looked its service tag up on the Dell web site and was told that my system shipped in August 2005. This old Dell PC was at that time the fastest computer in the Palatial Overclock Estate. I was more than a little surprised to discover that, eleven years later, it was still the fastest computer in the house, on a per processor core basis.

Dell lists it as a singe-core Pentium 4 processor at 3 GHz. My second oldest Linux system is a four-core Q6600 processor at 2.4 GHz. My newest Linux system is an eight-core i7 processor at 2.8 GHz. My MacBook Pro runs at 2.8 GHz, and my MacMini desktop at 2.6 GHz.

I have written before at length about the transition from higher clock speeds to multiple cores, and the problems this entails with taking advantage in software of this perhaps hypothetical increase in performance. But I didn't expect to be faced with this evolution in such practical terms.

I'm having some second thoughts.

Separate But Equal III

I'm making it a hobby to notice when data communications technologies move to separate control and data into independent channels, a common architecture pattern I've written about here more than once. A little more than a year ago I was writing software to control a Sierra Wireless MC7354 cellular modem in an Internet of Things project for which, somewhat remarkably, the Thing was going to be an aircraft on the ground.

After the usual noodling around and web searches I discovered that to use the modem in LTE mode, the usual serial port and PPP wasn't going to hack it: too slow. Instead I entered the (for me anyway) brave new world of QMI (Qualcomm MSM Interface) and RMNET, proprietary Qualcomm modem interfaces supported by both a software stack from Qualcomm for their "Gobi" family of devices, and an open source software stack consisting of the libqmi library, the qmicli command line tool, and the qmi_wwan driver, for your favorite Linux distro.

In PPP mode, the control of the modem is done using variants of the Hayes modem AT commands from the 1970s that we old folks all know and love. Once the data call was set up, a data channel using Point to Point Protocol was established over the same serial port, and IP packets were tunneled through it to the far end. Lots of layers of bits on top of bits there, all limited by the baud rate of the serial port.

But when using QMI and RMNET, the device exposes a control channel using Qualcomm's QMI messaging protocol, and a data channel using Qualcomm's RMNET virtual USB Ethernet framework. Once the appropriate QMI messaging is exchanged with the modem, the RMNET interface appears as just another Ethernet dongle. Except this Ethernet cable terminates wirelessly somewhere else in the world.

Again: separation of control and data. The old serial port interface is exposed via USB as well, and in fact has its own uses which can be exploited even while a data call is up over the QMI and RMNET channels. But it was another great example of optimizing a control channel for small packets with low latency, and a data channel for big packets with high bandwidth.

A Rambling Rant by an Old Man

I have had a good deal of career success. I attribute that success to a lot of things. One of those things is being evidence-based. Don't guess. Test. Measure. Look and see. Ask. If you can avoid guessing, do so. Sure, there are times you have to make a decision with incomplete information. But a lot of the time you can either defer that decision until more information is available, or improve the information on which you base your decision.

Not all opinions have the same value. For example, I have worked with people who have shared in a Nobel prize for their work in climate physics. Their opinions on climate change carry a lot more weight than yours. When you offer an opinion, on climate change or any other topic, I am going to assign an information content score to what you say. The definition of information is data that reduces uncertainty. The score I assign to your opinion might be zero.

The older I get, the more I seek out the original research and data on which people base their opinions. Often they either heard something second hand, third hand, or N-hand -- this is the "Chinese Whispers" problem -- or maybe the original study was flawed or biased, or maybe they are just full of crap because they believe what they want to believe. Most of the postings on Facebook fall into this latter category.

I like to read stuff that reinforces my opinions. I have read articles suggesting that this is common. I also like to read stuff that challenges my assumptions. I have read articles suggesting that this is rare. But it's the stuff that challenges my assumptions -- like education is a universal good, or that better communication unambiguously improves life -- that I find the most useful, and, really, the most interesting.

I like to read fiction, and have done so for decades. But increasingly I find myself more drawn to reading non-fiction. I find reality to be a lot more interesting, and stranger, and more compelling, than most of the fiction I read. It's as if reality exists just to entertain me. If you don't think reality is a damned interesting thing, then you aren't paying attention.

I'm old. Really old. I'm at an age where I have to read the obituaries, not just to make sure I'm not listed, but because I've lost many close friends and valued colleagues to accidents, strokes, cancer, heart disease, suicide, and even murder. I have a strong sense of my own mortality, and that time is short. Here's the thing: it's short for all of us, regardless of age. So you'll have to excuse me if I just don't have time for your bullshit.

I like people that agree with me. I also like people that don't agree with me. And I really like people for whom, when they don't agree with me, there is at least an even chance that they are right and I am wrong.

I find that, generally speaking, people act in the way in which they are incentivized to act. So if people aren't acting the way you want them to act, you should examine their incentives. Often, you will be the one (mis-)incentivizing them.

It's becoming increasingly difficult to find a really good soft pretzel.

How do you get to Carnegie Hall?

So I told him the same thing I tell all of them.

It's taken me forty years to get where I am. Two college degrees. Long periods of not making much money but getting a lot of great experience. Mentoring by many generous people smarter than I am. A lot of hard work. Many late nights and more than a few weekends. Even a skipped vacation or two. Occasional good judgement. Some luck. It helps a lot that I love what I do so much that I do it even when I'm not getting paid for it. There could be a faster way to get here, but you're asking the wrong guy what it is.

This is never a popular answer.

(Thanks to my friend and colleague Doug Young for suggesting the title.)

Hard Power Off, Solid State Disks, and Flash Memory: The Story Continues

Two more articles have come to my attention on the reliability of flash-based mass storage devices, from two organizations that should have some experience with such devices under their collective belts: Facebook and Google.

From Carnegie Mellon University and Facebook, Inc.:

J. Meza, Q. Wu, S. Kumar, O. Mutlu, A Large-Scale Study of Flash Memory Failures in the Field, ACM SIGMETRICS '15, June 15-19, 2015, Portland, OR, USA

Snippet:

Sparse data layout across an SSD's physical address space (e.g., non-contiguously allocated data) leads to high SSD failure rates;  dense data layout (e.g.,  contiguous data) can also negatively impact reliability under certain conditions, likely due to adversarial access patterns.

From University of Toronto and Google, Inc:

B. Schroeder, R. Lagisetty, A. Merchant, Flash Reliability in Production: The Expected and Unexepected, USENIX FAST '16, February 22-25, 2016, Santa Clara, CA, USA

Snippet:

In summary, we find that the flash drives in our study experience significantly lower replacement rates (within their rated lifetime) than hard disk drives. On the downside, they experience significantly higher rates of uncorrectable errors than hard disk drives.

Nearly five years after writing the first of the articles cited below regarding my own experience with incorporating SSDs and other flash-based mass storage in products, I continue to find challenges with their application. Not that I don't use them myself; semiconductor-based mass storage has replaced magnetic media in virtually all of the more recent systems that I deal with on a regular basis. But I find that the design practices of both hardware and software architects still haven't caught up with the implications of their use.

References

C. Overclock, Data Remanence and Solid State Drives, 2011-06-20

C. Overclock, The Death of Hard Power Off, 2012-07-17

C. Overclock, Hard Power Off Is Dead But Not Buried, 2013-03-04