I screwed up.
I'm passing this along as a cautionary tale of working with tool chains in the embedded realm. For those of you who write a lot of C and C++ code for the Linux platform and compile on the same architecture on which your code will run, for most of you that would be the Intel Pentium architecture, you may not know how good you've got it. When doing embedded development, it's quite common -- in fact typical -- to compile on one processor but run the resulting binaries on a completely different processor.
In my case, I build on a Linux-based Intel Pentium server, but run the resulting binaries (including a Linux kernel) on an ARM9, ARM11, or PowerPC processor, depending on the project. So my GNU C compiler has been compiled and configured to execute on an Intel Pentium processor, which is my host or build architecture, but it generates instructions for, say, an ARM9 processor, which is my target architecture. This is called cross compilation.
Not only does my gcc/g++ compiler generate completely different machine instructions in its .o and a.out files, but uses a completely different set of GNU header files than it would if I were compiling for my Pentium server, header files that are specific to my target architecture. When I'm compiling very low level code, the header files may be specific not only to the target architecture, but to the Linux kernel version running on the target.
This is where I tripped up. Back when I originally worked on Diminuto and Arroyo on the AT91RM9200-EK board, I just happened to be using the same Linux kernel version on my build server as I was on my ARM9 target processor. Everything worked just fine, but it worked coincidentally. I recently upgraded my build server to a later version of Linux, and also, in an unrelated effort, upgraded my compiler suite to the latest version of the excellent GNU-based CodeSourcery ARM tool chain that supports the ARM11 instruction set. Just as a test I tried building Diminuto and Arroyo, and they wouldn't build. The first thing I suspected was the new tool chain. So I fell back to the old tool chain I had used to originally build these old projects. And they still wouldn't build.
It didn't take that much sleuthing to realize that the problem was in the kernel header files. The wrong kernel header files. I needed to use the kernel header files for the kernel I had built myself, not the ones that came with the tool chain, nor the ones on my build server.
(The fact that the build worked at all is still kind of remarkable considering even using the same kernel, my build server had the header files for the Pentium x86 architecture while Arroyo needed the header files for the arm architecture. If you peruse the source distribution for the Linux kernel you will see that it can be built for many different architectures with many different architecture-dependent header files. Apparently for the header files that Arroyo needed they weren't that different.)
Fixing this was a bit of an effort. I had to churn the Diminuto and Arroyo make-based build systems, but also, as a precaution, the Desperado build system just to make sure I hadn't messed up my C++ build process.
And of course, that wasn't enough: I had to do a soup-to-nuts Arroyo kernel and root file system build using the revised build process and new CodeSourcery tool chain, then go through a complete installation on to the AT91RM9200-EK board, and go through the update process just to make sure I hadn't screwed something else up. This all occurred at the same time I happened to be replacing my old IBM ThinkPad T30 running Windows XP with a Apple Mac Mini with 27" Cinema Display as my desktop development environment. When something failed, I had lots of stuff to look at.
As you may guess, I eventually got it all to work. But it wasn't easy. It brings to mind how unexpectedly difficult (not to mention time consuming) embedded development can be, even (or, frankly, especially) when using open source tools and source code.
Below is a picture of the whole mess. The huge Cinema display dominates the picture. The compact Mac Mini sits on a shelf above it. You can see the green AT91RM9200-EK board to the lower right of the photograph. The temporarily abandoned tiny red BeagleBoard is looking a little lonely just below the center of the photograph.
But wait, what's this: a Windows 7 notebook in the bottom center of the photograph? That's right. When you do embedded development, you often live and die by the serial console terminal. And despite having tried several USB-to-serial converters and two different serial console software tools, I have discovered that there is nothing so far in the Mac environment equivalent to my beloved Putty utility. I'd be glad to be proven wrong.
Update (2011-01-03)
As I mentioned in the comments, I'm pretty happy with MacWise, a commercial terminal emulator for the Mac. I'm using it with a TrendNet USB-to-Serial adaptor/cable. To get MacWise to work reliably with the TrendNet device I had to download a new driver for the USB-to-Serial chipset from the Apple support web site. But it's been smooth sailing since then.
