Tuesday, June 13, 2017

Practical Matters

I'm frequently working on prototype hardware whose design bears only a superficial resemblance to what we are eventually going to ship. Sometimes there is a couple generations of prototypes in very different form factors, with the final version arriving not much before our ship date. Prototypes give the software team a head start (the header, the better) while the hardware team (with a little help from guys like me) work out the bugs in the hardware design. I may find myself rolling an engineering model running on batteries around in a parking lot to test that the cellular modem transitions seamlessly between cell towers. Or crawling around in a twin-engine executive business jet trying to figure out why the satellite communications unit isn't coming up. Or sitting on the floor in a drafty and noisy and sometimes dark server or telecom room between equipment racks, tangled in a maze of cables, with a laptop in my lap.

I love doing this kind of work. But it does require a fair amount of flexibility and no small amount of improvisation on my part. Here - in no particular order - are some of the tools I've found useful over the past few years. Maybe you will too.

Cheater Safety Glasses
Disposable Ear Plugs


The LASIK surgery I got eons ago was a miracle; I still have 20/20 distance vision. But the older I get, the worse my close up vision becomes. So besides protecting my eyes, these safety glasses, that have reading glasses molded into them at the bottom of each lens, let me see what the display on that SATCOM box in the equipment bay is actually telling me. I keep a pair of these in just about every tool kit I have, in my daily briefcase, and in the garage at home. I buy 150s (1.5 diopter) currently, but alas soon to be 175s, at my local hardware store.

My hearing, which has never been very good (a youth spent target shooting, riding motorcycles, and working in white-noisy computer and equipment rooms), hasn't gotten any better. I try to protect what little I have with disposable foam ear plugs. I have several different kinds laying around, but these E.A.R. brand Taperfit with the lanyard work best for me for day to day work. I use them in the lab, in the equipment and server rooms, and on site like when the pilot fires up the auxiliary power unit. Sometimes I use them at the coffee shop. You can buy these at your local hardware store too, but I order the big industrial sized box of individually wrapped ones from Amazon and hand them out (some might say forcibly) to co-workers.

Roll Your Own Custom Outlet Strip


I buy these little outlet expanders and grounded switches and when circumstances require I fabricate my own outlet strip on the spot. The outlet expanders play well with power bricks, and have a little light to indicate that they are electrified. You can place the switches wherever you need to control power to an individual device or set of devices. Add a heavy duty extension cord, and you have a really useful and flexible piece of electrical power distribution kit.

Folding Stool


My knees are getting too old to squat on the floor as I peer at some piece of gear. So I buy these folding stools. Sometimes I sit on them, sometimes I sit on the floor and use them as a laptop stand. They are sturdy, can handle my manly weight, are easy to schlep around, and when folded don't take up much room. I order them from Amazon, but I've seen something similar at my local hardware store.

Aluminum Project Clipboard



When I head to the hardware store to buy something to solve some problem in the lab, I don't want to find out I should have brought a tape measure. Or reading glasses. Or writing material. Or a compass (seriously). I find these aluminum project clipboards, beloved by state highway patrol officers everywhere, really useful. They are pretty indestructible, and hold a bunch of useful stuff inside. Also useful in the field, for all the same reasons. I got this one at my local hardware store.

Velcro Cable Ties


I don't know how I lived without these. When I get a new cable - ethernet, USB, video,  electrical, doesn't matter - the first thing I do now is toss the useless wire cable tie and replace it with one of these velcro cable ties. They come on continuous perforated spools of 25, so you just unwrap one and tear it off. They have a loop hole on one end so you can lasso them to one end of the cable, or connect several together when you need a longer cable tie. They can be easily trimmed with scissors to make them shorter and easier to deal with. My only complaint is they only come in black or gray; I'd like something like safety orange or high-visibility yellow or even shocking pink. I buy them at my local office supply store.

Zip-lock Bags


Embedded development means having  a lot of small related parts and other crap you have to keep track of. I like these Hefty-brand bags with the sliding zip-lock for that. They come in lots of sizes; I have at least one box of each. They have a label area that takes well to writing on with a Sharpie. They are cheap and sturdy. I keep everything from connectors to a Raspberry Pi with its power supply in them. I buy these at my local grocery.

Quirky Cordies Cable Organizer


I have a dozen of these Quirky Cordies cable organizers laying around or permanently assigned to specific tasks. You place cables in between the upright loopy fingers. The Cordies are weighted in the base so they tend not to get pulled off lab benches or window sills (antennas, you know) by the weight of the cables they are holding. They come in a variety of colors (although again not safety orange or high-visibility yellow). I order these off Amazon. I can see seven of them just as I sit here writing this at my lab bench in my home office.

Improvised Ground
Anti-Static Strap


Ideally when you use an anti-static strap - you are using an anti-static strap, aren't you? - you want to ground yourself to the chassis or grounding point of whatever you are working on. Easy in the lab, sometimes not to simple in the field. Although it's not the perfect solution, I came up with these improvised grounds by buying three-prong plugs, and grounding wires (package of two), at my local hardware store. Five minutes spent putting them together makes for an improvised ground in any grounded outlet I plug it into. You do have to be aware of the possibility of ground loops if the stuff you are working on isn't plugged into the same electrical circuit, so some caution is called for.

Although any anti-static strap is better than no anti-static strap, I like these with the Twist-O-Flex bands. I buy them off Amazon. The alligator clip pulls off to reveal a male connector compatible with the female plugs on the anti-static mats I like to use.

Thermal Label Maker


Folks that work with me will tell you I am an obsessive-compulsive labeler. That's why when the quality process inspector comes by to do an audit, it's my lab bench they take them to. I also add my email address and phone number to most of the equipment my company owns in addition to an official property tag. Lots of folks make these thermal label makers, all of which use proprietary cassettes containing a spool of thermal label tape, and which typically come in a variety of foreground and background colors. I think Brother may have discontinued this particular model, which I like because it is capable of printing two lines of text. But once you get bit by the labelling bug, you may start labelling stuff around the house. I bought this one from my local office supply store.

Carlisle Folding Cart



The company that makes this folding cart, Carlisle, sells a variety of sizes and shapes, mostly I suspect for the food service and catering industry. Turns out it's also really useful for field testing. I like this medium-sized but heavy-duty one because it's big enough to carry some serious systems under test, but folds up small enough to fit in the back of my car. It has locking wheels, so you don't see your twenty thousand dollar prototype roll away across the parking lot.

USB LTE Cellular Modem


Admittedly, this is an expensive piece of kit. But when I've needed it, I've really needed it. I've used this Verizon MiFi U620L LTE modem to get an internet connection on a Windows laptop when working in an airplane hanger with no WiFi. Or on a Mac laptop while parked in my car. Or on a Raspberry Pi running on a battery pack and sending telemetry to the mothership. I got this one from Verizon and added it to my cellular phone plan. The incremental cost to my plan was relatively tiny, although the device itself was pretty pricey. But sometimes it's actually faster than the local WiFi, and likely more secure.

Raspberry Pi
Raspberry Pi TTL Serial to USB Adapter

Raspberry Pi 3 "tin"


The Raspberry Pi single board computer (SBC) is frequently useful for early prototyping of the more powerful ARM-based embedded systems. I keep one attached to my network in my home office just so I can do quick unit tests on a 64-bit ARM target.

The FTDI TTL-232R-RPI debug cable is an off the shelf serial console to USB adapter that uses my favorite USB-to-serial solution, the FTDI chip. You can order these off Amazon. I added the "BYO 68A" label, which reminds me that the black wire goes to pin 6, the yellow wire to pin 8, and the orange wire to pin 10, on the Pi.

Ginormous Lithium Uninterruptible Power Supply



Another expensive piece of kit, this InteliCharge X1300 UPS has a big lithium battery pack that provides up to 1150Wh to 110V or 220V mains outlets, plus two USB outlets, and a 12V cigarette lighter outlet, for your portable power needs. I've run a large embedded system intended for aircraft power for hours on this unit. I bought mine from Amazon, although I notice they no longer list it; a few moments of web searching suggests it's no longer available. I have a smaller 400Wh unit from Anker that I have also used and which would be my second choice. I've used the InteliCharge to test equipment in a Faraday cage, to roll around in a parking lot to test cellular connectivity, or to move a piece of equipment around in a building without having to power it down.


This photograph shows the UPS and at least four other items from this article in use at a client's site. The UPS is powering the system under test, a laptop, and a VOIP phone. That is indeed Gigi the cat from the anime Kiki's Delivery Service on that bag on the bottom shelf in the photograph above.

USB Charger/Battery

Anker PowerCore Fusion 5000 Charger/Battery

On a much smaller scale, devices like this Anker PowerCore Fusion 5000, which combines a wall wart USB charger with a 5000mAh USB battery, are just the thing for powering the Raspberry Pi and similar USB-powered devices off the grid.

Hazer sender "tin"

Here is the PowerCore plus at least five other items from this article. The battery will power the Raspberry Pi so that it can send geolocation telemetry via the LTE modem back to the mothership.

Important safety tip: The PowerCore Fusion 5000 is not a uninterruptible USB power supply. Disconnect the PowerCore from the mains electrical outlet and anything connected to it will notice a power disruption while the device switches to its internal battery.

NRT ViperBoard


The embedded systems I help my clients develop always have scads of general purpose input/output (GPIO) pins used to communicate with or control sensors or devices. But often, most of the application software the team develops could be tested on a desktop personal computer or even a laptop long before a laboratory prototype is available. But such mainstream systems typically have no GPIO pins. The ViperBoard from Nano River Technologies (NRT) solves that problem. Attach a ViperBoard via USB to your PC and suddenly you have a bunch of GPIO pins, as well as I2C and SPI serial busses. This is also useful for interfacing your PC to lab test equipment. The ViperBoard is supported right out of the box by the more recent Linux kernels. (I added the plastic standoff legs and the "2" label.)

GlobalSat USB GPS Receiver


Similarly, for products which will eventually feature geolocation, a lot of the application software could be tested on a desktop or laptop if only GPS were available. There are lots of USB GPS receivers (and I've tried most of them), but my go-to unit is the GlobalSat BU-353S4. Plug it into a USB port and it enumerates as a serial port. Read from the serial port and you get a stream of standard NMEA sentences providing latitude, longitude, time, and date.

USB to Serial to USB Adapter


I've written before about my deep love for the CableMax USB 2.0 to RS-232 DB-9 Adapter, which contains an FTDI chip, my favorite USB to serial converter. Here's an application of it that has turned out to be remarkably useful: I attached two adapters back to back with a null modem converter in between, making a device that can connect two computers via a serial connection, all via USB. This may sound mundane, but lots of embedded application software still make use of serial ports to talk to devices. This little piece of kit lets you test such software on your desktop or laptop without a serial port in sight.

Lab Assistant


It's useful to have a lab assistant to help with all the chores that come with product development and to bounce ideas off of. I've had a lot of good ones over the years. Just make sure you teach them good electro-static discharge (ESD) practices. Here you can see one of my assistants using an anti-static mat.

Tuesday, May 30, 2017

Stranger Android Things

As far back as 2011 I was talking up using Android as a Linux-based embedded operating system. In those days I was running it on a BeagleBoard. I liked that idea then and I still like it now.

BeagleBoard xM Rev C

Android has a rich set of features, many of which would be really useful in an embedded product at the higher end. It is architected to make use of multiple network devices - for example, WiFi and a cellular modem - which is becoming common in the kinds of things I work on. Android has a mature tool chain and development infrastructure that includes stuff like a graphical IDE and a debugger. It is a well understood platform on which to deploy third-party applications, which at least one of my clients is eyeing as a business model for one of their products. And best of all, Android has enormous market forces incentivizing other folks to do a lot of the work for you.

I never got any traction on that idea. But maybe now that Google is promoting Android Things - as in Android for the Internet of Things - that will change. Android Things (formerly the platform known as "brillo") is a stripped down version of Android that can be run headless. It has the usual Android Run Time (ART) Java Virtual Machine (JVM) plus support for native C and C++ code, so you can port your legacy drivers and libraries, but do your new application development in an object oriented language that has first-class built-in support for threading, synchronization, and lots of other useful stuff.

Or will. Currently Android Things is just a pre-built developer preview that you can load on a selection of inexpensive development boards like the Raspberry Pi 3. It includes drivers and Java APIs for common embedded hardware interfaces like an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, Pulse-Width Modulation (PWM) output, and General Purpose Input/Output (GPIO) pins.

Third-party developers have in turn written higher level drivers and Java APIs on top of these for a selection of peripherals on commercially available expansion boards. For the Pi, that would be the Pimoroni Rainbow Hat. The Rainbow Hat has an APA102 LED strip (SPI), a BMX280 temperature and pressure sensor (I2C), two two-character HT16K33 alphanumeric segments (also I2C), a piezoelectric buzzer (PWM), and some discrete LEDs and buttons (GPIO). Below is a photograph of a Rainbow Hat on a Pi running my single threaded RainbowHatDemo application.


The source for Android Things hasn't been released yet, so porting any of my C or C++ libraries will have to wait. But the developer preview, with a Raspberry Pi and the Rainbow Hat, is a great way to see where this is all going. I wrote some toy applications using Android Studio just to try to remember how to use Java, which I haven't done anything serious with in a decade. You can find the repository, code-named Deringer, with all my source code as Android Studio projects, on GitHub. Below is a little code snippet from my multi-threaded RainbowHatThing application that reads the temperature and pressure sensor and logs the results.

    public void run() {
        Log.i(getClass().getSimpleName(), "begin");
        while (!lifecycleDone()) {
            try {
                float[] readings = sensor.readTemperatureAndPressure();
                float centigrade = readings[0];
                float fahrenheit = (centigrade * 9.0f / 5.0f) + 32.0f;
                float hectopascals = readings[1];
                float inches = hectopascals * 0.02953f;
                Log.i(getClass().getSimpleName(), centigrade + "C " + fahrenheit + "F " + hectopascals + "hPa " + inches + "in");
            } catch (IOException e) {
                // Do nothing.
        Log.i(getClass().getSimpleName(), "end");

Here is a little video (with audio) of the RainbowHatThing application running many threads concurrently on my tiny test system.

Will Android Things take off as a platform for the more capable IoT targets? No clue. But for sure there are many economic reasons to take this very seriously.

Monday, May 22, 2017

The Cost of Doing Business

My software development career over the past forty-plus years has been all over the map: supercomputers, eight-bit micro controllers, vast distributed telecommunications systems, multi-core microprocessors. The one thing they all have in common - at least at the level I work at near bare metal - is that the skill sets are all basically the same. But one thing that isn't the same is the cost of the tools to debug, functionally test, and validate my work.

Perhaps twenty years ago, I was writing firmware in C and C++ for optical networking products based on ATM and SONET. One night I found myself working late in a lab trying to debug some recalcitrant real-time code. I was surrounded by an H-P broadband analyzer, two H-P ATM protocol analyzers, and a tangle of multimode optical fiber. I realized I had maybe US$300,000 worth of test equipment dedicated to this one task, never mind my high-end workstation and the room-filling racks of telecommunications equipment. I paused for a moment to realize that an organization smaller than Bell Labs might have had a tough time tackling this product development effort, the investment in infrastructure was so significant.


Moore's Law hasn't really changed things that much, because as the tools got cheaper, the systems we develop and debug have gotten larger and more powerful right along with them. Just the other day I was working on my stratum-0 atomic clock project and I realized that my little hobbyist oscilloscope - one of the most useful tools I have ever purchased - was completely inadequate for doing any kind of jitter analysis on the output of the device. After some reflection, I decided I probably didn't know anyone that had an oscilloscope that could measure the jitter of my chip-scale atomic clock. Any jitter that might be detected would surely be in the oscillator used by the measurement instrument itself.


See, our measurement of time is a funny thing. It is a completely synthetic artifact, a singularly human invention. Thanks to natural variation in the Earth's rotation and in its orbit, time isn't even tied to the movement of the Sun any longer. It's defined, in the International System of Units, to the beat of an oscillator based on the cesium-133 time standard. And the thing I was trying to measure was itself an oscillator based on the cesium-133 time standard. The only way we can measure the precision and accuracy of a clock is by using a more precise and more accurate clock. I'm not likely to ever have one of those.

The larger lesson here is this: before you decide to tackle a project, make sure you can get the tools you need for the entire life-cycle of the product. You may find out that the economics of developing a product are substantially different from the economics of debugging, testing, validating, and supporting that product.