Chip Overclock

Cache Coherency and Memory Models

2009-06-14T16:39:00.022-06:00

Sometime around 2002 I was working on an object-oriented multi-threading framework in C++ for an embedded project and the most remarkable thing happened. I had an object that represented a thread. My application created the object via the usual new operator and started the thread. The first thing the thread did was access its own thread object. It core dumped. Core dump analysis using gdb revealed that the virtual pointer in the object was null.

I know what you're thinking.

That's impossible.

That's what I thought too. That event led to a personal research project that consumed many of hours of my free time and led to my putting together a presentation on the Implications of Memory Models (or Lack of Them) for Software Developers. In it, I describe the research of others that reveals that modern processor memory subsystems have broken the memory models implicit in the software.

This includes software written in higher level languages like C, C++, and Java (prior to Version 6 anyway) and as well as that written in assembler. It includes multi-processors, multi-core uniprocessors, and even those single core uniprocessors that are hyper-threaded. The upshot is that, thanks to various processor, memory, and compiler optimizations, memory may not be read or written when you think it is from looking at your code. Or even the compiled code.

Java addressed this issue in Version 6, providing you use volatile or the language's built-in synchronization mechanisms. But C and C++ developers dealing with variables shared among threads must still resort to the appropriate use of the volatile and explicit memory barrier instructions, which may be provided by your threading library's synchronization functions, providing you use them.

Fortunately, at least gcc offers the generic __sync_synchronize() built-in that performs a platform-specific memory barrier with full fence semantics. Unfortunately, in their paper Volatiles are Miscompiled, and What to Do about It, Eric Eide and John Regehr at University at Utah reveal that code containing the volatile keyword is frequently miscompiled. The life of a systems programmer is seldom simple.

Recently I discovered Ulrich Drepper's lengthy white paper on What Every Programmer Should Know About Memory. He gives a detailed explanation of how the hardware architecture of cache and memory subsystems can affect performance and correctness of software. I recommend it to every one that does embedded or systems development.

(Actually, I recommend it to all developers. But my Java buddies have pointed out on more than one occasion that they don't really care to know how things work under the hood. That's an attitude that I can't relate to. But I was one of those kids who took clocks apart to see what made them tick.)

It took me a couple of weeks to read Drepper's 114 page paper cover to cover. The entire time I was reading his description of the MESI cache coherency protocol, I was trying, without much success, to reconcile it with my prior reading on processor memory models.

Then finally I got to section 6.4.2, "Atomicity Operations", page 68:

If multiple threads modify the same memory location concurrently, processors do not guarantee any speciﬁc result. This is a deliberate decision made to avoid costs which are unnecessary in 99.999% of all cases. For instance, if a memory location is in the ‘S’ state and two threads concurrently have to increment its value, the execution pipeline does not have to wait for the cache line to be available in the ‘E’ state before reading the old value from the cache to perform the addition. Instead it reads the value currently in the cache and, once the cache line is available in state ‘E’, the new value is written back. The result is not as expected if the two cache reads in the two threads happen simultaneously; one addition will be lost.

If this doesn't run chills down your spine, you're not thinking clearly. Drepper goes on to discuss some of the same atomicity operations I talk about in my presentation. His paper also details the write reordering which I suspect was the cause of my null C++ virtual pointer.

Why is all this stuff important?

In the early 1980s I was a computer science graduate student working on my thesis under the direction of my mentor Bob Dixon, in a research group that was looking at programming language and operating system architectures for massively parallel processors. This was when Japan's Fifth Generation project, dataflow architectures, and the massively parallel Connection Machine were big. At the time, it all seemed bleeding edge and, given our budget, largely hypothetical.

But just a few years later I found myself working at a national lab, the National Center for Atmospheric Research, which not only had a Connection Machine, but other exotic highly parallel systems including several Cray Research supercomputers with multiple processors and vector hardware.

A few years later still and I am more than a little surprised to find that I have a four-core Dell server running Linux in my basement. The future has a way of happening.

In The Landscape of Parallel Computing Research: A View from Berkeley, researchers David Patterson et al. describe the future that they see: not just multi-core, but many-core, the equivalent of a massively parallel processor running on your desktop, with the dozens or hundreds of cores sharing memory. Suddenly, understanding concurrency, synchronization, and memory models (or the eventual higher level languages that implement it all without the developer having to worry about it) seems pretty necessary.

For me, it's 1983 all over again.

The Vulnerability of Volatiles

2009-02-17T12:38:00.005-07:00

As if having to deal with broken memory models isn't bad enough. In his latest column in Embedded Systems Design magazine, Jack Ganssle points us to research that shows that virtually all C compilers mis-compile code that uses volatiles. Here's the original article and presentation, both by Eric Eide and John Regehr at the University of Utah.

House M.D. and Evidence-Based Troubleshooting

2009-01-05T07:32:00.007-07:00

Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I got sucked into the television series House M.D. over the holidays. The USA cable network was running a marathon of House reruns. We haven't watched the first-run episodes (which run on the Fox cable network) except mostly by accident, and the last thing we need is to watch more television. But Mrs. Overclock is understandably interested in medical dramas (and, unfettered by the need to fill an hour of air time, will often beat House and his diagnostic team to the solution). I was a little surprised to find that House tickles certain centers of my brain, too.

If you're not familiar with the show, Gregory House (played by British comic actor Hugh Laurie in an impressive dramatic turn) leads a diagnostic group of physicians at a hospital in New Jersey. Each week they to try to cure some critically ill patient with a mysterious illness. They don't always succeed. It's a tribute to evidence based medicine. They have to choose tests that won't kill the patient, without really knowing what's wrong with the patient. They keep eliminating possibilities, and racking their brains to think up new ones. Frequently there is more than one problem, and they interact in strange ways. And always, the clock is ticking.

It's exactly like troubleshooting large, complex, distributed, real-time, high-availability, production systems, like a PBX.

I've done my share of field support of such systems. I've lived in a small room with several other developers, clustered around a workstation looking at a remote customer system. And I've gotten on a plane with a laptop and a protocol analyzer in my checked luggage. It's just like House. You keep brainstorming ideas of what could be going wrong. You try to think of tests to isolate the problem, to indict a particular hardware or software component, without crashing the production system. Your pour through log files, frequently inventing filtering tools on the fly, to make sense of the fire hose of information, to eliminate possibilities. You have to always keep track of what you know you know, of what you know you don't know, always being prepared to discard a much loved hypothesis in the face of new evidence. And always, the clock is ticking.

Frequently when I do this kind of work, and particularly when I am doing development for these kinds of systems, I have to keep reminding myself that not only are careers at stake, but lives. Customers may depend on the my software to dial 911 when someone has chest pain. Even in a non-safety critical situation, rebooting to see if it fixes the problem isn't an option if it means dropping hundreds of in progress calls. House's team can't afford to take a cavalier attitude, and neither could I.

I found a lot to relate to, watching House M.D. If you want to know what developing for high-availability systems is like, you would be well advised to check it out.

Abstraction in C using Sources and Sinks

2009-01-03T14:32:00.023-07:00

The problem with telling people that I do embedded software development is that no one really knows what that means. Including me. I've worked on embedded systems that were so resource constrained that they were written completely in assembly code, had no operating system, and in order to fix a bug we had to mine the code base to find instructions we could eliminate in order to make room. I've worked on embedded systems that had a full blown multi-user Linux system as their operating system, and the code base was hundreds of thousands of lines of bleeding-edge C++ that looked like the poster child for the use of templates and dynamic memory allocation.

Somewhere in between there is a sweet spot in which the application is written in C, but could still benefit from an object oriented design and implementation. OO scares a lot of the old school embedded developers (you know who you are). Yet my introduction to the use of those classics of OO - abstraction, encapsulation, and inheritance - was in 1974 when I was programming in assembly and using IBM's OS/360 Input/Ouput Subsystem, which was written in the late 1960s. Many years later when I was studying how C++ worked under the hood, the idea of the virtual pointer and the virtual table seemed quite familiar. I'd seen it all before in the OS/360 IOS. You get as old as I am, you begin to wonder if you're ever going to see anything that's truly innovative.

One of the most useful applications of OO in embedded systems, in my not-so-humble opinion, is that of abstracting out I/O: the ability to write applications that are agnostic as to from where their input comes and to where their output goes. This is exactly what the IOS and its myriad of pointer-filled control blocks (what C programmers would call structures) accomplished.

I used this idea for one of my clients when I was hired to implement a wide assortment of bitmap-graphic processing functions for an existing C-based product whose digital hardware could fit in your shirt pocket. I quickly realized that a lot of the code I had to write - like a deflate-based decompression algorithm, or a decoder for Portable Network Graphics (PNG) files - would be used in many places in the system. I also realized that the data for these algorithms could be buffered in memory, could come streaming in from a serial, Ethernet, or USB ports, or could be stored in a flash-based file system. I badly needed some abstraction for sources and sinks that I could implement on this tiny, non-POSIX platform.

As I implemented it, a Source is an abstract interface that requires the implementation to provide a read byte and a push byte method. The read produces a byte if it is available, End Of File (EOF) if the Source is empty and no byte would ever be available, and End of Data (EOD) if the byte isn't available now but might be in the future. The push pushes a single byte back into the Source where it is guaranteed to be the next byte produced by a subsequent read. (This vastly simplifies any algorithm that can be described by a grammar with one-character look-ahead, for example LL(1) or LALR(1). Such grammars are extremely useful ways of describing any algorithm that does parsing or data transformation or indeed any state machine. Both the deflate and PNG applications mentioned above can be described as grammars, and from there the implementation in code is almost a mechanical process. But that is an article for another day.)

int readSource(Source * that);
int pushSource(Source * that, char data);

A Sink is an abstract interface that requires the implementation to provide a write byte method. The write consumes the byte provided by the caller if it can, returns EOF if the Sink is full and no byte will ever be consumed, and EOD if the byte cannot be consumed now but might be in the future.

int writeSink(Sink * that, char data);

There are also a close method for both Sources and Sinks, which may or may not do anything, depending on the underlying implementation. No open method? Nope, because as we shall soon see the open is part of the implementation, not the abstraction.

As you can see, these simple byte operations expect a pointer to a Source or a Sink structure (what C++ programmers would call an object). What these pointers actually point to depends on the implementation. But the application doesn't need to know this. It just takes whatever Source or Sink pointer you give it and implements the application (what my younger colleagues would call the business logic) in an I/O agnostic manner.

What might an application look like that uses Sources and Sinks?

Here is a trivial one that just copies all of the data from a Source to a Sink until either the Source is empty or the Sink is full

Source * source;
Sink * sink;
int data;

while (1) {
if ((data = readSource(source)) < 0) { break; }
if (writeSink(sink, data) < 0) { break; }
}

What kind of Sources and Sinks might one implement?

A BufferSource implements a Source that produces its data from a fixed size memory buffer.

Source * openBufferSource(BufferSource * that, void * buffer, size_t size);

A BufferSink implements a Sink that consumes its data into a similar memory buffer.

Sink * openBufferSink(BufferSink * that, void * buffer, size_t size);

A FileSource produces data from a file in the file system (in Linux this could be just about anything, thanks to the /proc and /sys file systems).

Source * openFileSource(FileSource * that, const char * path);

A DescriptorSink consumes data to any file descriptor (like a TCP/IP socket).

Sink * openDescriptorSink(DescriptorSink * that, int fd);

A NullSink consumes all data without out any complaint and tosses it into the bit bucket.

Sink * openNullSink(NullSink * that);

A CompositeSource concatenates two Sources into a single Source.

Source * openCompositeSource(CompositeSource * that, Source * primary, Source * secondary);

An ExpanderSink writes each data byte to two different Sinks. (Yes, of course you can use another ExpanderSink as one or even both of the Sinks.)

Sink * openExpanderSink(ExpanderSink * that, Sink * primary, Sink * secondary);

A Fletcher8Sink computes a Fletcher 8-bit checksum as the Sink is written and automatically appends the checksum to the Sink when it is closed. (And before you ask, of course there is a Fletcher8Source that automatically verifies the data as it is read from the Source.)

Sink * openFletcher8Sink(Fletcher8Sink * that, Sink * primary);

A RingBuffer implements a circular buffer that exposes both a Source and a Sink interface.

RingBuffer * openRingBuffer(RingBuffer * that, void * buffer, size_t size);
Source * sourceRingBuffer(RingBuffer * that);
Sink * sinkRingBuffer(RingBuffer * that);

How are these Sources and Sinks implemented?

Easily. Most are a few lines of code, a few maybe a page. I'll be writing more about the implementation. But you can find the source code for these and other Sources and Sinks (and their unit tests) now in the Digital Aggregates Concha distribution. Concha is a clean-room open-source implementation of the Source and Sink design pattern (which my younger colleagues will recognize as a form of dependency injection). It is licensed under the Desperado modified LGPL which allows static linking without any viral licensing implications.

Practical Concurrency II

2008-11-10T20:48:00.002-07:00

Another great source of information on practical, real-world concurrency is the "Effective Concurrency" column that Herb Sutter is writing for Dr. Dobb's Journal. I've been reading them in the print issues, but you can find them by searching for "Herb Sutter" on the web site. The latest issue which just arrived today at the Palatial Overclock Estate (a.k.a. the Heavily-Armed Overclock Compound) is on "Understanding Parallel Performance".

Practical Concurrency

2008-11-09T14:09:00.002-07:00

Bryan Cantrill and Jeff Bonwick, both of Sun Microsystems, have written "Real-world Concurrency", published online in the September edition (6.5) of ACM Queue. It's a great set of tips, tricks, and lessons learned for those of us that have to deal with multi-threaded code. I've already passed it along to several of my colleagues working in C, C++ or Java on embedded or multi-core platforms. Queue is the Association for Computing Machinery's online journal targeted at near-term real-world computing issues.

Negative Results Are Still Results

2008-11-08T14:25:00.007-07:00

That's a remark made by my mentor Bob Dixon when he was my thesis advisor circa 1984. It's one of those pieces of wisdom that we accumulate and carry around in our brains for the rest of our lives. It had reason to percolate to the surface as I read Why Current Publication Practices May Distort Science by N. Young et al., published in the Medicine blog of the Public Library of Science. Young and his coauthors discuss the effects of "The Winner's Curse" on how scientific results are (or are not) published.

When an item is auctioned, potential buyers make offers, or bids, on the item. Although the dollar amounts of the bids may not be normally distributed, for sure they are spread out. Some bidders bid low, some bid high, depending on how each perceives the value of the item being auctioned. What is the actual market value of the item? One definition might be the average of all the bids. But who actually gets the item being auctioned? The highest bidder. That means, by this definition of market value, the winner of the auction always pays above market value for the item. That's "The Winner's Curse".

Of course, it's not that simple. The winner may be bidding higher because they believe they have special knowledge or capabilities that will let them leverage the item to greater value than the other bidders. Or they may be desperate. Or just foolish. (I personally have fallen into that category at one art auction, and ended up $250 poorer because of it.) But in general, auctions are good for the seller, maybe not so good for the buyer.

Young et al. apply this to scientific publications. Research with the most spectacular results tend to be what gets published. And, broadly speaking, research with positive results has a much better chance of getting published than research with negative results. Researchers whose projects yield positive results win the auction for space in refereed journals.

This gives a false perception of the state of a particular area of research, since projects that yield negative results are not published and hence are not part of the "average" perceived by those who read the scientific journals. In a "publish or perish" academic climate, there is a strong motivation to produce only positive results, to quickly terminate research that yields negative results, or even to phrase results to make them appear positive. Surely negative results are just as valuable, since they can prune the tree of possible avenues of inquiry for other researchers. But such research is seldom published.

Long time readers are just waiting for me to say this: this is yet another example of measurement dysfunction.

I've been accused by my friend and occasional colleague Rodney Black of having an academic bent. (He sincerely meant it as a compliment, and I don't deny it.) So I appreciate this issue that is perhaps of interest mostly to those who spend most of their time on the R end of the R&D spectrum. (Although since this affects much medical research, including the clinical trials done by pharmaceutical companies, it probably should be of great concern to all of us.)

But there is an equally strong bias at the D end of the spectrum too. Technology efforts that crash and burn seldom get discussed, seldom get written up, are usually quietly buried and the participants sent to the career equivalent of Guantanamo Bay, or maybe Gulag Archipelago. But like those research projects yielding negative results, these failed efforts would be valuable as object lessons, cautionary tales, and as port mortems on what to do differently next time.

This too leads to a false sense of the "average" of the state of the art. It causes artificially inflated expectations, frequently on the part of upper management. All they see in the in-flight magazine are the success stories. "History is written by the victors", as Winston Churchill once said.

I'm as guilty of this as any one else. I've had a few spectacular failures in my career. But you'd have to ply me with more than a few gin and tonics to get me to talk about them. (I encourage you to try.) That may be human nature, but it's wrong.

Two-time Nobel Prize winner Linus Pauling once said "The best way to have a good idea is to have lots of ideas". Meaning, most of your ideas aren't going to work out. If you have "a track record of success", it's only because you've done a very good job at hiding your failures.

Not only is failure an option, it's downright necessary.

(Update 2009-02-18)

I was having breakfast the other day with two of my oldest friends, Brian and Paul, who are the closest thing to homies I could have when I was in college in the 1970s. We were discussing our career ups and downs. I was a little surprised to hear someone say that the few real disasters I had had in my career were in hindsight the best things that had ever happened to me. I was even more surprised to realize it was me saying it. But it's true. Funny how life works out.

Updating Arroyo

2008-11-02T15:26:00.020-07:00

In From Diminuto to Arroyo I described Arroyo, another version of my Linux-based project to teach embedded software design and assembly language programming on a commercially available evaluation board, the ARM-based AT91RM9200-EK. My earlier project, Diminuto, was completely memory resident. It is a good example of smaller "diskless" embedded Linux systems. Arroyo uses an Secure Digital (SD) card as its root file system "disk", and can similarly use USB drives too. It is a good example of larger "diskful" embedded Linux systems.

Understand that Diminuto thinks it has a disk, when all it has is RAM. Arroyo thinks it has a disk, but is actually using flash-based devices that conspire with their device drivers to emulate IDE and SCSI disks. There's not a traditional spinning disk (or any other moving part) in either system. Diminuto still has all the drivers and utilities needed to mount and use SD cards and USB drives, it just doesn't need them to run.

Arroyo supports a variety of file systems like EXT2 and EXT3 (also popular in PC-based Linux systems) and VFAT (or FAT-32, popular in Windows systems). Most SD and USB storage devices come right out of the package pre-formatted for VFAT. Although VFAT doesn't support some traditional UNIX features like soft links, they still can be used directly without reformatting or reinitialization on Diminuto or Arroyo. I like the EXT3 file system in particular, because like EXT2, EXT3 supports all the standard UNIX features. But unlike EXT2, it is a journaled file system. This makes is extremely robust, especially in the face of the kind of treatment that consumer devices have to expect, like being powered off in the middle of operations.

The latest version of the Arroyo root file system image includes a full set of EXT2 and EXT3 file system maintenance utilities that allows you to fix and update your "disk-based" Arroyo file systems without resorting to using the same tools off-line on a server (which is how I originally built the first Arroyo root file system on an SD card). It also includes a tool that allows you to examine, initialize, modify partitions on your SD and USB storage devices. In this article I'll show you how to download a new Arroyo file system image from a server and install it using nothing but tools on Arroyo itself.

We will begin by using an SD card as our original root file system,
create an EXT3 file system on a USB drive,
download the new root file system image (which is just a file) from a server to that USB drive,
install the root file system image on a second USB drive,
reboot the system and temporarily use the USB drive as our new root file system,
install the same root file system image on our original SD card, and
reboot the system to use the new root file system image back on our original SD card.

Notice at no point do I actually boot the Linux operating system off an SD card or USB drive. Both Arroyo and Diminuto still depend on booting the Linux kernel from a TFTP server, and Diminuto's root file system image is embedded inside its kernel image. We use a TFTP server because the version of the U-Boot boot loader on the AT91RM9200-EK board doesn't have the capability of booting from a storage device. Later versions of U-Boot have this capability, and I have used it to good effect on other projects. That would eliminate the need for a TFTP server for Arroyo. But updating to a new version of U-Boot requires some expensive hardware, like a flash programmer, and is a story for another time.

(Blogger wraps or even truncates long lines of preformatted text. If you want to see a complete verbatim log of this process, including all of my typos and other mistakes, you can find one here.)

Begin by using an SD card as our original root file system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial
Environment size: 459/8188 bytes
Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
################################################################# ################################################################# #######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start
## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:01:08 UTC (883612868)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
kjournald starting. Commit interval 5 seconds
EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ifup: interface lo already configured
ifup: interface eth0 already configured

www.diag.com Arroyo 0.0

arroyo login: root
#

Create an EXT3 file system on a USB drive.

Insert a USB drive into one of the two host USB ports on the EK board. Note that it the kernel hotplug facility detects it automatically after a few seconds and gives it the name "/dev/sda1". That's short for "SCSI Disk #A Partition #1", because the USB storage device driver in Linux emulates a SCSI device.

usb 1-2: new full speed USB device using at91_ohci and address 2
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 0:0:0:0: [sda] Attached SCSI removable disk

Make an EXT2 file system on the USB drive.

# mke2fs /dev/sda1
mke2fs 1.41.2 (02-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
121920 inodes, 487184 blocks
24359 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=499122176
15 block groups
32768 blocks per group, 32768 fragments per group
8128 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Create an EXT3 journal file on the USB drive. This turns it into an EXT3 file system. Other than the journal file (which is a invisible file by virtue of having no directory entry), EXT2 and EXT3 file systems are bit for bit compatible. You can mount an EXT3 file system as an EXT2 file system (but then the journal won't be used).

# tune2fs -j /dev/sda1
tune2fs 1.41.2 (02-Oct-2008)
Creating journal inode: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Mount the newly minted EXT3 file system so that we can use it.

# mount /dev/sda1 /mnt
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Download the new root file system image from a server to the first USB drive.

# cd /mnt
# ftpget -u coverclock -p password 192.168.1.222 arroyo-root.ext2 /var/tmp/arroyo-root.ext2

Install the root file system image on a second USB drive.

Insert a second USB drive. This one gets called "/dev/sdb1" when hotplug detects it.

usb 1-1: new full speed USB device using at91_ohci and address 3
usb 1-1: configuration #1 chosen from 1 choice
scsi1 : SCSI emulation for USB Mass Storage devices
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 1:0:0:0: [sdb] Attached SCSI removable disk

Copy the root file system image from the file we just downloaded onto the first USB drive to the second USB drive, using the second drive as a raw block device instead of as a file system.

# dd if=arroyo-root.ext2 of=/dev/sdb1 bs=1M
127+1 records in
127+1 records out

Run a file system check on the second USB drive, which now appears to have an EXT2 file system on it. It will be a small file system, about over a hundred megabytes, because that is the size of the root file system image that we downloaded.

# e2fsck -f /dev/sdb1
e2fsck 1.41.2 (02-Oct-2008)
Filesystem did not have a UUID; generating one.

Superblock last write time is in the future. Fix? no

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 4438/7168 files (0.2% non-contiguous), 111875/130700 blocks

Resize the file system on the second USB drive so that it consumes the entire USB drive which is about two gigabytes.

# resize2fs /dev/sdb1
resize2fs 1.41.2 (02-Oct-2008)
Resizing the filesystem on /dev/sdb1 to 1948736 (1k) blocks.
The filesystem on /dev/sdb1 is now 1948736 blocks long.

Reboot the system and temporarily use the USB drive as our new root file system.

This is an important step, and it doesn't show up in this log: remove the first USB drive, the one to which we FTPed the root file system image. Leave the second USB drive containing the new root file system. It doesn't matter which USB host port it's in. I'll explain why below.

# sync
# sync
# sync
# reboot
The system is going down NOW!
Sending SIGTERM to all processes
Sending SIGKILL to all processes
Requesting system reboot
Restarting system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv

baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial

Environment size: 459/8188 bytes

Change the bootargs variable in the U-Boot bootloader so that it boots from "/dev/sda1". Why "/dev/sda1" and not "/dev/sdb1"? The naming of the USB storage devices by the hotplug process depends on the order in which the devices are detected, not which USB host port in which they are inserted. If you boot the system with two USB devices inserted, the order in which they will be detected is non-deterministic, depending on the timing of the USB hardware inside the drives enumerating to the USB controller on the EK board. We only leave the USB drive that we want to use as our root file system inserted, so we can be sure to know what name to provision here. We also set the root delay to 5 seconds. This causes the boot process to pause long enough for the hotplug facility in the kernel to detect the USB drive before we try to mount it as our root file system.

Uboot> setenv bootargs 'console=ttyS0,115200 mem=32M rootdelay=5 root=/dev/sda1'

Download the Arroyo Linux kernel image from the TFTP server and start it.

Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start

## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=5 root=/dev/sda1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usb 1-2: new full speed USB device using at91_ohci and address 2
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:01:21 UTC (883612881)
Waiting 5sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 0:0:0:0: [sda] Attached SCSI removable disk
VFS: Mounted root (ext2 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex

www.diag.com Arroyo 0.0

arroyo login: root
#

We are running Linux using the USB drive as our root file system.

Install the same root file system image on our original SD card.

Reinsert the first USB drive, the one containing the root file system image file that we downloaded, and mount it.

usb 1-1: new full speed USB device using at91_ohci and address 3
usb 1-1: configuration #1 chosen from 1 choice
scsi1 : SCSI emulation for USB Mass Storage devices
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 1:0:0:0: [sdb] Attached SCSI removable disk

# mount /dev/sdb1 /mnt
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
# cd /mnt
# ls
arroyo-root.ext2 lost+found

Just for fun, run the fdisk partitioning utility on the SD card, "/dev/mmcblk0p1".

# fdisk /dev/mmcblk0p1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/mmcblk0p1

Command (m for help): m
Command action
a toggle bootable flag
b edit bsd disklabel
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): p

Disk /dev/mmcblk0p1: 2 GB, 2023418880 bytes
255 heads, 63 sectors/track, 246 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/mmcblk0p1p1 1 247 1983996 83 Linux

Warning: Partition 1 does not end on cylinder boundary.

Command (m for help): q

Install the file system image on the SD card much like we did on the USB drive. Run a file system check on it, and resize it.

# dd if=arroyo-root.ext2 of=/dev/mmcblk0p1 bs=1M
usb 1-1: reset full speed USB device using at91_ohci and address 3
usb 1-1: reset full speed USB device using at91_ohci and address 3
127+1 records in
127+1 records out
# e2fsck -f /dev/mmcblk0p1
e2fsck 1.41.2 (02-Oct-2008)
Filesystem did not have a UUID; generating one.

Superblock last write time is in the future. Fix? yes

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mmcblk0p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mmcblk0p1: 4438/7168 files (0.2% non-contiguous), 111875/130700 blocks
#
# resize2fs /dev/mmcblk0p1
resize2fs 1.41.2 (02-Oct-2008)
Resizing the filesystem on /dev/mmcblk0p1 to 1983616 (1k) blocks.
The filesystem on /dev/mmcblk0p1 is now 1983616 blocks long.

Reboot the system to use the new root file system image on our original SD card.

We just use the standard "arroyo" and "start" scripts that we used when we first brought the system up. But this time, the root file system will be the one we just installed.

# sync
# sync
# sync
# reboot
The system is going down NOW!
Sending SIGTERM to all processes
Sending SIGKILL to all processes
Requesting system reboot
Restarting system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial

Environment size: 459/8188 bytes
Uboot> run arroyo

TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
################################################################# #######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start

## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usb 1-1: new full speed USB device using at91_ohci and address 2
usb 1-1: configuration #1 chosen from 1 choice
usb 1-2: new full speed USB device using at91_ohci and address 3
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
scsi1 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:39 UTC (883612839)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
VFS: Mounted root (ext2 filesystem).
Freeing init memory: 96K
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sda] Write Protect is off
sd 1:0:0:0: [sda] Assuming drive cache: write through
sd 1:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sda] Write Protect is off
sd 1:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 1:0:0:0: [sda] Attached SCSI removable disk
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sdb] Write Protect is off
sd 0:0:0:0: [sdb] Assuming drive cache: write through
sd 0:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sdb] Write Protect is off
sd 0:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 0:0:0:0: [sdb] Attached SCSI removable disk
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex

www.diag.com Arroyo 0.0

arroyo login: root

Just for fun, run the partitioning utility to look at both of the USB drives.

# fdisk /dev/sda1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/sda1

Command (m for help): p

Disk /dev/sda1: 1 GB, 1990517760 bytes
255 heads, 63 sectors/track, 242 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1p1 1 243 1951866 83 Linux

Warning: Partition 1 does not end on cylinder boundary.

Command (m for help): q
# fdisk /dev/sdb1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/sdb1

Command (m for help): p

Disk /dev/sdb1: 1 GB, 1990517760 bytes
255 heads, 63 sectors/track, 242 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1p1 1 243 1951866 83 Linux
Warning: Partition 1 does not end on cylinder boundary.
Command (m for help): q
# mount
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw)
#

Part of the beauty of using Linux as an embedded operating system is that the tools and techniques you use to work on it are the same as those you use on your server. In fact, the commands I used to build my first root file system on an SD card for Arroyo on my Ubuntu-based Dell server were almost identical to those I used here on Arroyo itself, differing on only slightly in the device names because of differences in hardware between the two systems.

When Code Coverage Is Necessary But Not Sufficient

2008-10-19T16:37:00.005-06:00

In Can unit testing be a waste? pbielicki argues that it is far more efficient to test at the highest possible level of abstraction and rely on your code coverage tools to insure that all code paths have been tested. That might work in his world, but just a couple of days after having read that article I ran into a situation where developers had been doing just that and missed a bug completely.

The C++ code was implementing an algorithm in an open specification for the binary encoding of an identity string into a binary format for transmission to a remote piece of equipment. The identity string is a globally unique fifteen digit decimal number. The string is broken up in to several separate fields and each field is encoded from decimal into binary before being stuffed into a protocol packet destined for the remote equipment. There are several encoding algorithms depending on the length of the specific field, which range from one to three decimal digits in length.

Looking at the couple of dozen lines of C++ code in the base, I thought "It would be reasonable to assume that if any valid identity string were to be encoded into binary, then decoded back into a decimal string, the result would match exactly the original fifteen digit string." So knowing exactly nothing yet about the specific algorithms from the spec, I wrote a little unit test to do precisely that.

How many of the possible values did I test for each encoding algorithm? All of them, of course. On a 2.8 gigahertz Pentium 4, cycles are easy to come by.

What did I find? Twenty percent of the possible values failed this simple unit test.

I am pleased to report, however, that code coverage was one hundred percent. In fact, testing any single input value for any of the algorithms would have yielded code coverage of one hundred percent. Given that eighty percent of the possible values passed unit testing, it would be easy, in fact, likely, that you would not catch this bug just testing a few selected values. Conceivably you could test a lot of values, and still not have any failures, if you just happened to stay within the eighty percent that worked.

It wasn't a matter of code coverage. It was a matter of input range coverage.

I have written unit tests that ran for many minutes, testing a huge range of input values. For an algorithm that did time and date calculations, I had a unit test that ran for days. Fortunately, I didn't have to run it very often. But when I did run it, and it completed successfully, I was pretty darn sure that the underlying code worked.

Cycles are cheap. When dealing with what are fundamentally mathematical algorithms, there is no reason not to test a lot of values. When in doubt, test all of them.

Accessibility

2008-09-26T09:42:00.014-06:00

I recently took a nine day motorcycle trip from Denver Colorado to Dayton Ohio to visit family and friends. I rode my BMW R1100RT about 2500 miles (maybe 4000 kilometers) round trip. When I'm riding solo eight to ten hours a day without benefit of distractions like music or conversation, I spend a lot of time paying attention to what goes by and listening to what Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) calls my ThinkMan™. Here is some of my play list for that trip.

Route US-36 splits off from Interstate 70 just East of Denver. US-36 has a completely different character depending on what state you're traveling through. In Eastern Colorado it's kind of spooky, passing through ghost towns of shuttered buildings. In Kansas it's a two lane road with lots of local traffic and a little town every fifty miles or so. In Missouri, it's a four-lane divided controlled-access highway. In Illinois, it is either nothing but two-lane asphalt going through endless corn fields, or it merges with I-72. US-36 ends in Indianapolis Indiana, where I hopped onto I-70 for the final run to Dayton. Where as I-70 is just a vast ribbon of asphalt from Denver to Dayton, US-36 has enough variety to keep your attention.

I like staying in the little Mom and Pop motels along the way, where you ring the bell at the counter and someone leaves the television in their living room to check you in. I was traveling with my Nokia N810, so tiny it takes up almost no room in my motorcycle luggage. The internet tablet and the availability of WiFi made it easy to keep up with my email or do a little web cruising. I was amazed at how many of these little motels had WiFi. For example, both motels in Smith Center Kansas (near the geographic center of the contiguous United States) had hand-written signs "WiFi Internet" in their office windows.

I realized it was cable television that made the WiFi easy. When cable television became a "must have" for every little motel, it forced them to put in the broadband infrastructure. Once that was done, getting internet access over the cable and installing a LinkSys WAP was a piece of cake. And nearly all of them were LinkSys: seldom did I see the WAP SSID changed from its out-of-the-box default.

At first I thought that seeing URLs on billboards advertising farm equipment in the middle of Kansas was a sign that the web really had gone mainstream. But I suspect that farmers in the middle of Kansas may have realized the usefulness of the web and the internet long before many city dwellers did. When you're relatively isolated, being able to mouse-up Amazon.com and order just about anything from a Long Tail selection larger than any urban store might seem especially important. And since the internet was originally designed so that military communication would survive a nuclear war, being able to stay in touch in the aftermath of tornadoes and blizzards is pretty darned useful too.

My experience is that farmers and ranchers are pragmatic adopters and exploiters of not just bio-tech and agri-tech, but just about any useful-tech. A friend of mine once worked for a company that developed a local area network technology designed to transmit across barbed wire. Another worked with a rancher who tagged his cattle with RFID-like devices to automatically track how long each animal stood at the feeding trough. This is pretty cool stuff.

The ghost towns along US-36 got me to thinking, is there some critical mass that is necessary to keep a town alive? Did the conversion of the more southern Route US-40 to I-70 kill the towns in Colorado, but not those in Kansas and Missouri?

When I was but a lad growing up in Ohio, shortly after the glaciers receded, I had reason to spend some part of most summers in rural Eastern Kentucky. No running water, one broadcast television channel, electricity most of the time. What time wasn't spent on chores was spent reading, shooting, crashing a dirt bike, or just goofing around. A twenty minute drive in either direction on the road would get you, depending on which direction you turned out of the gravel driveway, to Grayson or Sandy Hook. Both towns had main streets maybe a block or two long with no building taller than two stories. Sandy Hook had a bit of an edge, since it was the county seat, and had the high school and the funeral home.

Decades later Mrs. Overclock and I flew into Lexington Kentucky and rented a car to drive to Sandy Hook for a funeral, while staying at a motel in Grayson. I was startled by the contrast between the two towns. Grayson was huge and sprawling, and many choices of chain hotels, restaurants, and retail. Sandy Hook looked like a ghost town, with nearly all the buildings shuttered and boarded up. Only the high school, the funeral home, and the county buildings remained.

It was the interstate highway of course. In the intervening decades, I-65 had been built, and it had a Grayson exit. Also, the U. S. Army Corps of Engineers undertook a huge flood control project between the two towns (incidentally permanently flooding the property where my mom's family originally had their farm). The resulting dam, reservoir and lake, and no doubt the beautiful Appalachian scenery, motivated Kentucky to develop a state park and recreation area. Regardless of the fact that the lake was pretty much in between both towns, it was named "Grayson Lake State Park". Tourism joined tobacco and coal mining as a major industry in the area. Grayson prospered. Sandy Hook mostly disappeared.

During the intervening decades between wandering in the woods with a firearm and flying around the country with Mrs. Overclock, I was a student of Computer Science at Wright State University. The WSU main campus is in Fairborn Ohio on what was once farm land. (Indeed, my thesis advisor and mentor Bob Dixon was the first faculty member hired by the University, and his initial office was in a farm house standing on the property.) And when I was there, it still mostly looked like farm land: woods and rolling hills. Unfortunately, that was about all there was around campus. If you wanted to dine off campus, you had to get in a car and drive at least fifteen minutes even to get fast food.

Then Interstate 675, a bypass around Dayton Ohio connecting I-75 and I-70, was built, with several exits for campus and the nearby Wright-Patterson Air Force Base. You can guess how this story ends. I barely recognize the area now. Enormous development, condos, retail, dining. And no doubt at least in part due to the greater accessibility that I-675 provided the University, the campus must be about triple its size than when I was a student, with about 16,000 students (the majority of which, interestingly enough, are women).

It probably goes without saying the accessibility is a key factor in the ability to develop and grow. And of course, it's not just physical accessibility, but virtual accessibility; maybe even more so. Many years ago, my old comrade Doug Supp (who is still at Wright State, but is surely thinking about retirement by now) and I wrote a proposal to bring the internet to the University. It seems laughable now, but way back then, back in the 1980s, we actually had to sell it. Not everyone was convinced that this new fangled internet thing was worth the money, or that it would amount to anything. After all, this predated the World Wide Web, so we were really talking about technologies like electronic mail, telnet, USENET, and file transfer being the killer applications.

Doug and I used the growth we saw that was so evident outside of our office windows as a result of I-675 as a rationale for the project, which we called TURNPIKE, trying to draw the analogy between physical accessibility and virtual accessibility as important for the growth of the University. Fortunately for all involved, the University bought into it. It was by no means a sure thing. By the way, TURNPIKE stood for The University Resource Network for the Pursuit of Information, Knowledge and Education. Yeah, we had to rack our brains for that one.

(Wright State University is also known for having a handicap-accessible campus right from its very inception in the late 1960s. Another victory for accessibility.)

Tooling along US-36 cross country with the hum of the 1100cc opposed twin engine in my ears, I had a lot of time to think about how important accessibility is.

And gas stations. Always on the lookout for gas stations.

From Diminuto to Arroyo

2008-09-25T14:48:00.015-06:00

If you've lasted this long reading my articles on Diminuto, you know it's my attempt to create an environment for teaching real-time software design, embedded development, and assembly language programming, using commercially available hardware and open source software. Diminuto uses the Atmel AT91RM9200-EK evaluation kit (EK), a single board computer (SBC) that has an ARM9 processor with memory management and a host of peripherals, including Ethernet, USB ports, and a Secure Digital (SD) card slot. You may also recall that Diminuto was built using uClibc, a reduced memory footprint C library, and has a root file system that is memory resident.

(Not that everyone doesn't know what these look like, but below is a photograph of an SD card, which is a little larger than the size of my thumbnail, and a USB storage drive, which is about the size of my thumb. Both of these particular examples hold two gigabytes, a fact that an old guy like me finds astounding.)

Now it's time for Diminuto Phase II, which I call Arroyo. Arroyo has a vastly larger root file system that runs "disk" resident on an SD card. I'm using an EXT3 journaled file system on a 2GB SD card; the root file system utilizes less than 10% of the card. Arroyo is still based on BusyBox (1.11.2), but it uses the full Standard C Library and includes a complete Bash (3.2) shell. Because Arroyo doesn't use a RAM-resident root file system, it has a memory footprint not much larger than Diminuto. Like Diminuto, Arroyo supports USB storage drives as well, making it easy for students to take their projects with them.

(Below is a photograph of the EK SBC with the SD card in its card slot in the left rear and two USB drives in the host USB ports in the center front. Both Diminuto and Arroyo can access these devices just like disk drives. Very slow disk drives. Arroyo uses the SD card for its root file system, although in practice, as with any other Linux system, just about everything commonly used ends up cached in memory.)

Arroyo was not built using Buildroot, although it does make use of the genext2fs host utility created by the same folks to build the EXT2 file system image. Arroyo was built using individual open source components including the Linux 2.6.26.3 kernel. The only patch required to any of the open source software was to the kernel to address the wrong board type code reported by U-Boot that I've discussed before.

Arroyo was created using a pre-built tool chain, Sourcery G++ Lite, provided by ARM, Ltd. and the folks at CodeSourcery. The tool chain can be downloaded for free, and provides full support for the later ARM Embedded Application Binary Interface (EABI).

I've ported the Desperado embedded C++ library to Arroyo, including John Sadler's Ficl embeddable Forth interpreter, and ran most of the unit tests.

Here is an example of booting Arroyo on the EK SBC. Note that I overrode the U-Boot bootargs environmental variable to point the kernel at the SD card for its root file system. Since the SD card device driver is hot plugged into the system when you insert the SD card, I told the kernel to delay for a couple of seconds to allow the device to come online. (You can also see this log file here.)


boot 1.0 (Aug  8 2003 - 12:29:00)

Uncompressing image...



U-Boot 1.1.1 (Oct  2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0  BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash:  8 MB
In:    serial
Out:   serial
Err:   serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
bootargs=console=ttyS0,115200 mem=32M
filesize=1ab33c
fileaddr=21000000
ipaddr=192.168.1.223
serverip=192.168.1.222
start=bootm 21000000
netmask=255.255.255.0
gatewayip=192.168.1.1
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
stdin=serial
stdout=serial
stderr=serial

Environment size: 427/8188 bytes
Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> setenv bootargs 'console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1'
Uboot> run start
## Booting image at 21000000 ...
Image Name:   Linux-2.6.26.3
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    1279628 Bytes =  1.2 MB
Load Address: 20008000
Entry Point:  20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #35 Thu Sep 25 14:05:44 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:59 UTC (883612859)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex



www.diag.com Arroyo 0.0
arroyo login: root
# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:00:00:00:00:00
    inet addr:192.168.1.223  Bcast:192.168.1.255  Mask:255.255.255.0
    UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    RX packets:1 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:195 (195.0 B)  TX bytes:0 (0.0 B)
    Interrupt:24 Base address:0xc000

lo        Link encap:Local Loopback
    inet addr:127.0.0.1  Mask:255.0.0.0
    UP LOOPBACK RUNNING  MTU:16436  Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

# df
Filesystem                Size      Used Available Use% Mounted on
rootfs                    1.9G    133.2M      1.7G   7% /
/dev/root                 1.9G    133.2M      1.7G   7% /
tmpfs                    14.6M         0     14.6M   0% /tmp
# bash
[root@arroyo ~]# cat /proc/meminfo
MemTotal:        29824 kB
MemFree:         19696 kB
Buffers:           888 kB
Cached:           6552 kB
SwapCached:          0 kB
Active:           3600 kB
Inactive:         4588 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               4 kB
Writeback:           0 kB
AnonPages:         756 kB
Mapped:           1480 kB
Slab:             1332 kB
SReclaimable:      232 kB
SUnreclaim:       1100 kB
PageTables:         92 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:     14912 kB
Committed_AS:     1912 kB
VmallocTotal:   989184 kB
VmallocUsed:      1088 kB
VmallocChunk:   986108 kB
[root@arroyo ~]#

Diminuto and Arroyo are both excellent teaching examples of embedded Linux systems at opposite ends of the resource spectrum, and both run on the identical EK hardware and firmware. I'll be writing more about both Diminuto and Arroyo in articles to come.

Diminuto Nuts and Bolts

2008-09-23T08:14:00.022-06:00

Diminuto is my attempt to put together a platform on which to teach real-time software design, systems programming, embedded software development and assembly language programming, all using real commercial hardware and open source software. In Choosing Software for Diminuto, I talked about the software choices I used for Diminuto and my strategies for configuring them for a small memory footprint. In this this article, I'll describe in more detail how I built Diminuto and show an example of the system booting on the Atmel AT91RM9200-EK evaluation kit (EK).

Diminuto consists of the Linux 2.6.25.10 kernel, the uClibc 0.9.29 C library and its associated tool chain, and BusyBox 1.11.1. But it isn't enough to just download all this stuff off the web and build it. For one thing, the use of the reduced memory footprint uClibc library requires some patches be applied here and there in both the bootable system and its tool chain. For another, you are still faced with the need to build a root file system that can be incorporated as an initial RAM file system (initramfs) within the kernel image.

Because this work is pretty common to anyone building an embedded Linux system using the software I chose, the uClibc folks provide another tool, Buildroot, to assist with it. Buildroot is a collection of configuration utilities, scripts, and makefiles that automate much of the process of downloading, patching, and building both the tool chain that runs on the host (for example, your desktop Linux PC) and the software that boots on the target (in our case, the EK board), and creating the root file system. Some of what Buildroot does is necessary because I chose to use the uClibc C library. Some of what it does would be necessary no matter what tool chain or C library I chose to use.

As you might expect, there were some issues I had to resolve to get Buildroot and the artifacts that it creates to work on the EK board.

Buildroot is a moving target. Even more so than most other open source software, Buildroot is a moving target. It does not appear to have any release management. When you download Buildroot, you do so directly from its Subversion repository. While you may easily choose any revision you want, there is no guarantee that any particular revision will work, or even build, for the particular choice of hardware, software, and options that you want to use.

There are a lot of folks contributing to Buildroot, so the repository is under constant churn. I went through several iterations trying to get various revisions of Buildroot to work with Diminuto before settling on Buildroot 22987. The most current revision as I write this is 23454; that means there have been 467 changes since I got my copy of 22987. Your mileage will almost certainly if you choose to try using a different version of Buildroot than I did.

U-Boot on the EK board isn't quite right. U-Boot is a bootloader like GRUB, and LILO. It lives on the hardware in some persistent memory like ROM or flash. It does some basic hardware initialization necessary for booting the Linux kernel. It has a console command interface and some basic scripting capability. It can boot a Linux kernel image that resides in ROM or flash. It can also get a bootable Linux image from a server across the network using the Trivial File Transfer Protocol (TFTP). Later versions of U-Boot that I have used even understand how to read EXT2 file systems on IDE devices so that Linux can be booted from a "disk" (which as I mentioned in the prior article, may actually be a flash device). I've built embedded Linux systems to boot from U-Boot using all three of these methods: from flash, using TFTP (which is what Diminuto uses), and from a solid-state IDE "disk".

U-Boot is very popular in the embedded community, while GRUB and LILO are mostly used on PCs. The EK board comes with U-Boot 1.1.1 pre-installed in flash. This places the EK user way ahead of just having "bare metal".

However, the U-Boot that comes on the EK isn't perfect. One of the things that U-Boot does upon transferring control to the Linux kernel is pass a board type code (really just a magic integer) to the kernel, telling the kernel what kind of board it is running on. This admits the possibility of building a single kernel to run on several variations of hardware, or for the kernel to check that it is running on the right board. Unfortunately, the U-Boot that comes on the EK board identifies the board as the earlier, more expensive, and no longer available Development Kit (DK) board, which varies slightly in its hardware. Wackiness ensues. The Linux kernel, built for the EK board, panics.

Later versions of U-Boot allow you to override the board type using a U-Boot environmental variable; U-Boot 1.1.1 lacks this feature. Much later versions of U-Boot support an even sexier feature called the flat device tree, where an entire hierarchical database of hardware information is passed to the kernel. This is supported in the latest versions of the Linux kernel for the PowerPC architecture, but is not applicable to Diminuto's ARM architecture.

The right way to fix this would be to install a correctly configured version of U-Boot, or a later version that has the override feature, in the flash on the EK board. But this would require every user of Diminuto have access to a tool like a Abatron BDI 3000 JTAG debugger to reflash U-Boot. A JTAG debugger is a piece of hardware that attaches to a target processor and allows you to read and write its registers and memory, set breakpoints, and write to system devices like flash memory. The best ones, like the BDI, have gdbserver interfaces that allow them to be used with the GNU gdb debugger.

These tools are great to have around, and are nearly indispensable for production embedded system development. Which is why I have one sitting four feet away from me. But unfortunately, they also cost thousands of dollars and would place Diminuto out of the price range of many potential users. (I do recommend that if you are outfitting a lab with several Diminuto stations for teaching a course, having a single JTAG debugger to share among all the lab users is a very good investment indeed.)

So instead I patched in the Linux kernel used by Diminuto to change the board code reported by U-Boot to that of the EK board. This patch was just a few lines of ARM assembly code in the portion of the Linux kernel executed very early (like after just a handful of machine instructions) in the boot process. This is a hack for sure, but one which allows the Diminuto kernel to be used on any EK board right out of the box without requiring modifications to the board or its firmware.

(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)

There are some issues with the choice of ABI. An Application Binary Interface (ABI) establishes a standard about how an application calls into the Linux kernel, how data structures are packed in memory, how subroutines are called and receive parameters, how stack frames are organized, generally how all that stuff that application programmers take for granted but systems programmers have to worry about works.

Alas, there are two different (and incompatible) ARM ABIs, the Old (really, original) ABI (OABI) and the new, improved Embedded ABI (EABI). The more recent EABI has many advantages, particularly in its emulation of floating point, and Buildroot supports it as an option. But I was unable to get it work work reliably. (Googling suggests I am in good company.) Specifically, there appears to be an issue in the kernel signal handling code for the ARM architecture when using the EABI: like, it doesn't work in 2.6.25.10. Signaling handling in the kernel is more bizarre than I ever expected: it involves pushing machine code on to the stack of the running application, where it is later executed and popped off. This means its implementation is architecture and ABI specific.

In the end I had to build everything using the OABI and put off using the EABI for another day.

Linux wasn't happy about the relatively large initial RAM file system. When I tried to build Linux 2.6.25.10 using the relatively large initramfs, the linker was unable to resolve some of the external symbols in the kernel because they would result in branches being made to displacements farther away (around the initramfs) than was allowed by the ARM branch instruction. I rearranged slightly the memory order of the kernel components at link time to eliminate this issue.

I'm not convinced that this approach doesn't leave some RAM unavailable for use following the loading of the initial RAM file system, but the footprint of the Diminuto software is so small compared to the EK board's available memory, that I don't consider this a critical issue. I also noted in subsequent work using another tool chain, a slightly later version of the Linux kernel, and the EABI, that this issues appears to have been fixed.

(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)

Buildroot is good at building, not so good at cleaning. Buildroot can build the GNU tool chain, the Linux kernel, Busybox, and the root file system, with not much more than a make at its top level. However, doing a make clean at its top level seldom achieves the desired effect.

When working on various Linux kernel or Busybox configurations, I found it best to first do a top level make in the Buildroot directory, then iterate by descending in to the specific directories for the kernel or Busybox and doing the appropriate make there. This allows Buildroot to do all of its source code patches and and attend to all of its configuration needs, and then allowed me to tweak things as necessary with finer granularity.

(A makefile which automates much of this process is provided in the Diminuto distribution on the Diminuto web page.)

So finally we come down to booting Diminuto on an EK board right out of the box. When you attach a serial console (for me, that was an old ThinkPad laptop running putty) to the EK board and power it up, you see U-Boot come up and issue a prompt

just like you did in the article Diminuto Right Out of the Box. If you already tried booting the 2.4 kernel and RAM disk that the EK board comes with, you have already administered the U-Boot environmental variables necessary to give the EK board an IP address and point it at your TFTP server. Here are mine; your mileage will of course vary.


setenv ipaddr 192.168.1.223
setenv serverip 192.168.1.222
setenv netmask 255.255.255.0
setenv gatewayip 192.168.1.1

To make life easier, I defined two new U-Boot environmental variables, download and start, which are macros that download the 2.6 kernel and initramfs image from the TFTP server and start the boot process. I also had to change the variable bootargs, which is the boot command line argument list passed by U-Boot to the kernel.


setenv start 'bootm 21000000'
setenv download 'tftp 21000000 diminuto-linux-2.6.25.10'
setenv bootargs 'console=ttyS0,115200 mem=32M'

I saved these U-Boot environmental variables to flash where they would persist across resets and power cycles.


saveenv

Finally, I use the macros I just defined to boot Diminuto.


run download
run start

That's all there is to it. Here's a complete log where I reset the board, ask U-Boot to display all of its environmental variables, boot up Diminuto, login into the system, and display its memory usage. (You can also find this in a separate page here.)


boot 1.0 (Aug  8 2003 - 12:29:00)



Uncompressing image...



U-Boot 1.1.1 (Oct  2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0  BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash:  8 MB
In:    serial
Out:   serial
Err:   serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
bootargs=console=ttyS0,115200 mem=32M
filesize=1ab33c
fileaddr=21000000
ipaddr=192.168.1.223
serverip=192.168.1.222
start=bootm 21000000
netmask=255.255.255.0
download=tftp 21000000 diminuto-linux-2.6.25.10
gatewayip=192.168.1.1
stdin=serial
stdout=serial
stderr=serial

Environment size: 384/8188 bytes
Uboot> run download
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'diminuto-linux-2.6.25.10'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
######################################
done
Bytes transferred = 2519904 (267360 hex)
Uboot> run start
## Booting image at 21000000 ...
Image Name:   Linux-2.6.25.10
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    2519840 Bytes =  2.4 MB
Load Address: 20008000
Entry Point:  20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux.......................................................................................................................... done, booting the kernel.
Linux version 2.6.25.10 (jsloan@silver) (gcc version 4.2.4) #49 Tue Sep 2 14:50:42 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 28472KB available (2399K code, 274K data, 108K init)
SLUB: Genslabs=12, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 152 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NetWinder Floating Point Emulator V0.97 (extended precision)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
at91_spi: Baud rate set to 5990400
AT91 SPI driver loaded
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:43 UTC (883612843)
Freeing init memory: 108K
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex




www.diag.com Diminuto 0.0

diminuto login: root
# cat /proc/meminfo
MemTotal:        28580 kB
MemFree:         22400 kB
Buffers:             0 kB
Cached:           3956 kB
SwapCached:          0 kB
Active:            976 kB
Inactive:         3320 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:         356 kB
Mapped:            388 kB
Slab:             1324 kB
SReclaimable:      296 kB
SUnreclaim:       1028 kB
PageTables:         72 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:     14288 kB
Committed_AS:      868 kB
VmallocTotal:   989184 kB
VmallocUsed:      1088 kB
VmallocChunk:   986108 kB
#

You can see that of the 32MB of RAM total on the EK board, nearly 28MB is available for use, and of that, about 22MB is available for use by applications you may choose to develop for Diminuto. The system supports persistent storage in the form of both SD cards and USB drives (I've used both). The associated tool chain has the complete C++ Standard Library which includes the Standard Template Library (STL), and the complete POSIX Threads (pthreads) library.

I have ported and successfully unit tested almost all of the Desperado embedded C++ library to Diminuto, including John Sadler's Ficl embedded Forth interpreter. I did run into some weirdness (I'm tempted to say bugs, but I'm sure it will turn out to be pilot error) with code generated by the C++ compiler that I worked around, and discuss on the Diminuto web page.

I was motivated to build Diminuto by my curiosity to see if I could create a cost effective environment that could be used to teach some of the same stuff I learned decades ago using assembler code and PDP-11s, but with actual, modern, commercially available hardware, and open source software. For sure, there is a lot more than could be done to expand this project (and I'll do that as time permits), but I'm really pleased at what I've accomplished. I hope others find some inspiration in this.

In a future article, I'll show the classic "Hello, World" example, where we compile a simple program on our host PC, FTP it to the target EK board, and execute it.

Why Free Markets Don't Work: Measurement Dysfunction as Management Strategy

2008-09-04T08:27:00.008-06:00

Reading Robert Austin's book Measuring and Managing Performance in Organizations (Dorset House, 1996), was such an eye-opening experience for me that it has merited mention in this blog several times before this, most prominently in In Defense of Misbehavior. Once you understand measurement dysfunction, in which the performance measurement of individuals almost inevitably drives dysfunction into an organization, you will never again see the world in quite the same way.

The gist of it is this: an old axiom of business is "You cannot manage what you do not measure." But in his Ph.D. thesis, Austin, formerly an executive with Ford Motor Company and now on the faculty of the Harvard Business School, applies agency theory, a form of game theory, to show that in all but the most trivial of situations, it is impossible to measure all metrics applicable to an individual's performance. There will always be aspects of performance that cannot be measured. When rewarded (or punished) based on the metrics you do measure, the person being measured will be highly incentivized to game the system by slacking off on the those unmeasurable metrics. A classic example are call center agents hanging up on customers in the middle of calls to meet their quotas.

Austin's work has been cited by Tom DeMarco ("Mad About Measurement") in Why Does Software Cost So Much? (Dorset House, 1995), and by Joel Spolsky ("Measurement") in Joel on Software (Apress, 2004). It compelled me to get a copy of his dissertation (Robert Daniel Austin, Theories of measurement and dysfunction in organizations, Carnegie Mellon University, University Microfilm, #9522945, 1995), and even put a talk together on the topic which I've given several times. Given the recent horrifying cover story in BusinessWeek, "Managing by the Numbers: How IBM improves quality by tracking employees' every move" (September 8, 2008) , this topic is bound to become popular blog fodder once again.

But that's not what I'm here to talk about. I'm here to talk about why free markets don't work.

A few years ago, a large telecommunications equipment manufacturer decided to save money by eliminating all desktop PC support and having their product developers support their own predominantly-Windows desktop computers. This was done, of course, as a money savings move.

Did they save money? On paper sure, because they eliminated a bunch of IT staff from the bottom line. In reality, who the hell knows? All they really did was move their desktop IT support from where its cost could be measured to where it could not be measured. Most of the product line of this manufacturer was Linux-based, so the developers, who were earning hefty salaries for their Linux and telecommunications expertise, were not Windows experts. Most of their Windows support experience came from buying a home PC from BestBuy.

I suspect those developers were the most expensive IT support staff on the planet. But its all good because those costs weren't measured. Nor was the loss in productivity, the reduction in efficiency and quality in the work environment, and the moral hit to both the developers and the IT staff that came with the assumption that those two skill sets were interchangeable. I mean, really, it's just all computers, right? How hard can it be?

Sure, you could see this as a form of measurement dysfunction: some VP got a bonus by gaming the system. This is how people most often think of measurement dysfunction, as a more general term for incentive distortion. But this can also be seen in another way, not inadvertent dysfunction caused by the measurement of performance of individuals, but in a deliberate application so as to move costs to where they could not be measured. It's measurement dysfunction as management strategy.

Pollution is another form of measurement dysfunction. Companies realize cost savings by not dealing with the environmental costs of manufacturing. That's because, without government regulation, environmental costs are not borne by the polluter. The costs are pushed off onto the downstream neighbors of the manufacturer, or into the surrounding community, even onto future generations. As long as you just think locally and take a short term view, there is a strong incentive to move environmental costs to where they cannot be measured. Offshoring is just another form of this: moving environmental, or even societal, costs to where they are not measured.

In his book, Best Business Crime Writing of the Year (Random House, 2002), James Surowiecki collects magazine and newspaper articles on the spate of corporate crime that lead to the demise of Enron, WorldCom, Tyco, and other best-in-class companies. The actions of the executives described in these articles illustrate another form measurement dysfunction. Federal, state and local laws are a form of measurement. There is a strong (given the huge dollar amounts involved, I'm tempted to say irresistible) incentive to game the system by testing the limits and loopholes of those laws in an effort to derive revenue from areas for which the results of actions are not measured. I suspect that many of these executives didn't even realize they had committed a crime until the men with badges, guns, and handcuffs showed up with warrants, subpoenas, and writs. The Sarbanes-Oxley Act is an attempt to fix this dysfunction by measuring a metric that had not been measured before.

Measurement dysfunction is much more broadly applicable than just as incentive distortion. That's why it's known by other names in other contexts: unintended consequences, perverse incentives, and moral hazard are some I know about. With all due respect to my libertarian friends (and I have a lot of them), I argue that measurement dysfunction is why free markets don't work.

There will always be an economic incentive for participants in a truly free market to dump costs into or derive revenue from any gray area in which there is no measurement. And in an information economy, there are a lot of gray areas. That's because the measurement of all dimensions of market costs is impossible. Adam Smith described an "invisible hand" that guides markets to the most efficient allocation of resources. But for many metrics of market cost, there is no hand at all, invisible or otherwise. I'm not arguing for controlled markets. But I am arguing for regulated markets: that a significant (and perhaps unfortunate) amount of government regulation is necessary to act as a visible hand, forcing market participants to bear the full costs of their transactions.

In the realm of performance measurement of individuals, Austin argues against extrinsic motivation, for example bonuses tied to specific goals, because of the measurement dysfunction this produces. Instead, intrinsic motivation, for example corporate culture, causes individuals to do the right thing independent of incentives. If intrinsic motivators worked at the corporate level, free markets could work as well. But it's been argued that corporations are psychopathic because, devoid of any other legal restraint, their sole purpose is to maximize shareholder value.

I'm not saying that corporations are evil. I'm the founder and CEO (and janitor) of a corporation myself. But I am saying that it would be foolish not to acknowledge the measurement dysfunction that occurs in a truly free market.

Choosing Software for Diminuto

2008-09-02T16:25:00.027-06:00

Diminuto is my attempt to put together a platform on which to teach real-time software design, systems programming, embedded software development and assembly language programming, all using real commercial hardware and open source software. In Diminuto Right Out of the Box, I described how to get the Atmel AT91RM9200-EK evaluation kit (EK) running with the software it ships with, which includes a Linux version 2.4 kernel and a tool chain based on the GNU Compiler Collection version 2.95.3. In this this article, I'll describe the software I used to build a reduced-memory footprint system using the Linux version 2.6 kernel and a tool chain based on GCC 4.2.4.

My career as a technologist has been pretty much all over the map. I've always thought of that as a good thing, although I've been accused by my betters of being everything from "a renaissance man" to "unfocused". My career seems to shuttle back and forth between developing for high-end server-side systems and doing real-time and embedded development. Consequently, the systems for which I've developed have ranged from processors with eight kilobytes of memory to distributed networks of servers to supercomputers.

Even among embedded systems, the range has been kind of startling. In the 1970s I wrote standalone assembler code for PDP-11s with 8KB of core memory. Core memory was kind of neat because it maintained its state across power cycles. We used to patch our code by toggling in new machine instructions from the switch register on the front panel. More recently, and more typically, I've found myself writing mostly in C, C++ and occasionally even Java, for PowerPC, i960, or ARM microprocessors running some commercial real-time operating system (RTOS) like VxWorks, pSOS, C-Executive, or RTX which run completely memory resident. Meanwhile, the server-side portion of my career evolved from using just about every UNIX variant known to man to being 100% Linux based. Probably no surprise there.

With the availability of cheaper random access memory (RAM), and relatively inexpensive microprocessors that have a memory management unit (MMU), perhaps it was inevitable that these two career paths merge with the advent of embedded Linux. But even among embedded Linux systems there is a broad range of configurations, ranging from systems that still run completely memory resident and which are reminiscent of those VxWorks systems, to those that have persistent storage in the form of solid-state "disks" such that they seem more like a Linux server.

Whether you are accustomed to developing for Linux on a PC, or you are an embedded developer used to VxWorks, embedded Linux will seem like a weird combination of the familiar and the bizarre, not quite fitting in with either world. The kernel seems familiar to the PC developer, but there may be no persistent storage and hence no paging of virtual memory, and you may do everything logged in as root while the system runs in single-user mode. The lack of persistent storage seems natural to the embedded developer, but the system has an MMU which maps between physical and virtual memory addresses, the processor executes code in privileged and non-privileged mode, it insists on having a root file system even though it may be completely memory resident in something called a RAM disk, and there is an unusual wealth of sophisticated applications and tools that can run on the system. The need to compile and link on a host system (like a conventional Linux PC with an Intel processor and a cross-compilation tool chain) but run on a completely different target system (like the AT91RM9200-EK board with an ARM processor and no tool chain at all) freaks out the PC developer but is strictly in the comfort zone of the embedded developer.

Many embedded systems have pretty severe resource constraints when compared to your typical PC. The EK board has 32 megabytes of RAM and a processor delivering a claimed 200 million instructions per second (MIPS). The Dell 530 quad-core server in the basement (a.k.a. the vast Digital Aggregates corporate data center) on which I run the Diminuto tool chain has three gigabytes of RAM and its speed is several billion instructions per second (GIPS). My Nokia N810 internet tablet, more typical of hand-held consumer devices, has 128 megabytes of RAM, just four times that of the EK, with maybe a 400 MIPS processor. All of these are Linux systems, but clearly they are not quite the same Linux system.

Living within your means, RAM-wise, isn't just an issue for the kernel and any programs that are running at any one time. Although Linux requires a root file system, I've mentioned that many embedded devices have no persistent storage device. The kernel, any running programs, and the root file system in some kind of RAM disk, all share the available RAM. The kernel and the RAM disk are loaded into RAM at boot time, either from some read-only memory (ROM) or across a network using something like the Trivial File Transfer Protocol (TFTP). Reducing the memory footprint of the RAM disk turns out to be important too.

Or maybe not. Ridiculously large surface mounted flash memory devices with integrated controllers that emulate IDE disk drives are becoming more available, driven in part by Windows CE hand-held devices. The multi-gigabyte single chip disk drives that I have used come from the factory pre-formatted for the Windows FAT file system. Linux itself includes drivers for devices like Secure Digital (SD) flash cards and USB flash drives which emulate SCSI disk drives.

Persistent storage devices are becoming more common in all but the most cost sensitive or tiniest embedded systems. However, such solid state persistent storage devices tend to be very slow when compared to disk drives. Even when they are available, there is a tendency to keep as much as possible of the working set of "disk" blocks from these devices cached in memory.

It's easy to argue that the trend will be towards more and more RAM in embedded devices, just as there has been for servers, desktops, and laptops. But the trend may be in just the opposite direction for many embedded applications. Microcontrollers (which is what microprocessors are called when you don't know that they're there) are being used in physically smaller, and more price sensitive, devices. There is a strong competitive pressure to shave the manufacturing cost of a device that will be sold in the millions. Sometimes this means less RAM and eliminating a flash drive. Pennies count in such applications.

I've built both types of embedded Linux systems: those that had a lot of RAM and a large flash-based disk drive, and those that booted from ROM and ran completely in RAM. The former wasn't substantially different from the user's point of view from a PC-based system: multi-user, ran the SSH daemon, even ran an Apache web server and a Java SE JVM, and your files were still there in your home directory when the system was rebooted. The latter: single user mode, could only be accessed from a serial port, and every press of the reset button was a new day RAM disk-wise. It's easy to get spoiled with the former. I learned more from the latter. Diminuto marches right down the middle: the basic system runs completely resident in RAM, but supports the use of persistent file systems on SD cards and USB drives.

Here are some approaches to reducing the resource footprint of a Linux system.

Reduce the Size of the Kernel

You download the latest Linux kernel from www.kernel.org: all eight million lines of C code in twenty thousand source files. In the case of Diminuto, which uses an ARM-based processor, you run the command

make menuconfig ARCH=arm

and are delighted to discover that you can select the options AT91 and AT91RM9200-EK to get support for the devices in your system-on-a-chip. Then you build the kernel and discover to your horror that the resulting bootable image is enormous. True, the Linux kernel has direct support for the AT91RM9200-EK board. But if you just select that option, you will find that by default the configuration process has selected tons of other options and drivers you could care less about, including a lot of stuff you don't even recognize, and also left out a bunch of stuff you know you're going to need.

To generate exactly the kernel you need without blowing your memory budget, you must laboriously go through all the dozens of nested configuration menus, figure out whether you really need each item, and then either enable or disable it. This includes the seemingly hundreds of device drivers, including the ones that can be built as loadable modules (more on that later). This may take several iterations, some research, and some hard decisions ("Hmmmm, do I really need IPV6?"), but it's worth it. In the end you'll get what you need with a memory footprint you can live with. If you forget something, you can always generate another kernel and try it.

During this process you will be tempted to build every device driver as a loadable module. You will do this because this is exactly what you do for your PC-based Linux system. Loadable kernel modules and device drivers are not linked directly into the kernel image, but instead are loaded dynamically on demand from the file system. Building loadable modules for your embedded system doesn't actually hurt anything, as long as you realize it's mostly a waste of time. Devices other than what are on your board aren't likely to magically appear; the EK doesn't have a back plane into which you can easily plug in other boards. And you have pretty much a 100% chance of using the devices that are implemented on your board. Furthermore, unused loadable modules in a RAM disk file system take up RAM whether you are using them or not. My advice is to build all the kernel options and device drivers you know you are going to use directly into the kernel, and explicitly omit everything else.

There is an important exception to this rule: if you are writing your own device drivers for your board, you may want to use loadable modules for reasons of intellectual property and licensing. Conventional wisdom is that loadable modules are not contaminated by the GNU Public License (GPL), whereas modules linked directly into the kernel are. I'm not a lawyer, and I'm not aware of any legal precedent having been set for this, but when I've written device drivers for my clients for closed, proprietary, custom hardware, I've built them as loadable modules.

In Diminuto, I built version 2.6.25.10 of the Linux kernel using the approach I just described, including all supported devices, directly into the kernel, and omitting everything else.

There are also alternatives to the conventional Linux kernel. uCLinux is a version of Linux that does not require an MMU. I've never used it. But it does admit the possibility of running Linux on microcontrollers that are in the traditional sphere of RTOSes like VxWorks. uCLinux has been ported to many of the embedded microprocessors on which I've worked over the years. I think it's worth examining if that's the world in which you are working.

Reduce the Size of Binary Executables

A statically linked binary executable is a program which incorporates all of its code in a single monolithic image. If several programs are statically linked against the same library, they each contain unique copies of the same functions and subroutines that they reference in that library. Everyone gets their own copy of printf. There are sometimes reasons to do this (it may facilitate debugging), but it makes every binary executable a lot larger than it otherwise needs to be.

A shared library in Linux is like a Dynamic Link Library (DLL) in Windows (or so I'm told). It is a way for binary executables to share common code. One copy of the shared library is loaded into memory. When a binary executable which was originally dynamically linked against that shared library is run, the program loader fills in the references to the shared functions and subroutines inside the executable so that they point into the shared library. The MMU is used to create the illusion that everyone is using their own copy of the library, even though everyone is sharing the same code.

This is a good thing, generally, whether you have an embedded system with 32MB of RAM or a PC with 3GB of RAM. But it does take some care to manage the installation and maintenance of the shared library. Woe be to the person who runs a binary executable that was dynamically linked on the host system against a different version of the shared library from that which is resident on the target system.

It also requires some care in the construction of the shared library itself. A shared library that incorporates a bunch of functions and subroutines that aren't used by any binary executable is still wasting RAM. Shared libraries are only a win if the code in them is actually shared. This is why I chose to build Diminuto using uClibc, a version of the Standard C library with a reduced memory footprint. uClibc gets its reduction through a variety of means: by omitting some features not needed in most embedded applications, by reducing the capacity of some retained features, and by some code re-factoring. I used version 0.9.29 of uClibc for Diminuto.

I have also built embedded Linux systems using the full blown GNU standard C library, so your mileage may vary. If you are running completely memory resident, I recommend you at least look at uClibc to see if it meets your needs. If you have the extravagant pleasure of a "disk" on your target, you may consider using the GNU library. On my To Do list for Diminuto is to build a root file system using the full GNU library that boots from an EXT3 file system on the Secure Digital (SD) flash card supported by the EK. I built support for EXT3 and the SD driver into the kernel, so that same kernel should work for either system. I'd like to eventually run the ARM-version of Sun's Standard Edition (SE) of their Java Virtual Machine (JVM) using this platform. (I've done this in the past on embedded PowerPC platforms with zero issues.) I routinely use EXT3 file systems on USB drives for persistent storage with the current RAM-resident Diminuto system.

One thing I did not scrimp on was the C++ standard library and the POSIX Thread library. I knew I was going to port my embedded, real-time C++ toolkit, Desperado, to Diminuto, and I wanted unfettered access to the complete Standard Template Library (STL) and to pthreads. Recently, however, a reduced footprint version of the C++ standard library, uClibc++, has been developed that is worth a look. I haven't used it.

I've had no library-compatibility issues with porting either Desperado or my C-based Diminuto library to the AT91RM9200-EK and running their unit tests. I have had some weirdness with the GNU arm-linux-g++ compiler itself which I discuss on the Diminuto web page.

Reduce the Size of the Root File System

Much of the Linux root file system is taken up with scripts and utilities to manage its multi-user environment and to provide a rich set of tools for those multi-users. The entire huge System V init script infrastructure exists mainly to take a multi-user system up and down in an orderly manner. The typical embedded system, however, runs a very limited set of applications, may have no traditional user interface at all except for during initial development, all processes may run as root, and it is brought up and down strictly with the power switch. This requires a much simpler infrastructure that eliminates a lot of the stuff usually found in the root file system.

This is also one of the reasons I recommend using an EXT3 journaled file system if you have persistent storage. EXT3 is bit-compatible with EXT2, a staple of Linux systems for a long time, with the addition of a journal file. In fact, an EXT3 file system can be mounted as an EXT2 file system, in which case the system ignores the journal file. EXT3 is a robust approach for an embedded system where someone may hit the reset button at any time. I built Diminuto with support for EXT2, EXT3, and VFAT (a.k.a. Windows FAT-32, especially useful for USB drives that are traded between Windows and Linux systems).

PC developers will be horrified to know that most work is done on embedded Linux systems while the developer is logged in as root, but this will seem natural to the embedded developer. This is because nearly all of the work that needs to be done on the target system has to be done as root, while all the development work that needs to be done on the host system is done logged in as a normal unprivileged user. Also, a scrogged target system can typically be recovered by just hitting the reset button, with the resulting loss of everything you may have had on the RAM disk. Linux offers the ability to segregate processes and files on the target system by user ID, which does provide a great deal more protection. I recommend thinking about this if non-developers have access to the innards of your embedded Linux system, even if they don't have login access. For example, you will surely want to run any web server under a non-root account.

BusyBox is a very clever way of reducing both the RAM and disk footprints of a Linux system without much impact on functionality. BusyBox is a single monolithic program that implements the functionality of dozens of commonly used Linux commands. It achieves this significant resource reduction, even when using shared libraries, by yet more sharing of common code, excluding seldom used options, and by being less scalable than the equivalent full featured GNU utilities. BusyBox is a multi-call binary: if it is invoked through a soft link called ls, it behaves like the ls command.

I built Diminuto using version 1.11.1 of BusyBox. A quick tour through all of the bin directories on my PATH reveals that my root file system on Diminuto has all of five binary executables:

/sbin/ldconfig
/usr/bin/ldd
/usr/bin/gdbserver
/usr/local/bin/getubenv, and
/bin/busybox.

The dozens of other commands are just soft links to /bin/busybox. (And the first three of those binaries are just to expedite debugging.)

The use of BusyBox is a huge win, even if you can't use all of its capabilities. I am currently using the Bourne-like shell provided by BusyBox. If your embedded system depends heavily on complex shell scripts, and if resources permit, consider loading your shell of choice (e.g. the Bourne Shell) on your embedded system. Scripting is a huge win productivity-wise over implementing the same functionality in C code, and the additional resources required by a full blown shell interpreter can be well worth it.

In future articles I'll discuss how I built the root file system for Diminuto, describe the adventure that was getting the 2.6 Linux kernel running on the AT91RM9200-EK board, and give you a tour of the running system.

Diminuto Right Out of the Box

2008-08-28T10:36:00.028-06:00

Choosing Hardware for Diminuto

2008-08-27T10:50:00.034-06:00

Diminuto is my attempt to put together a platform on which to teach real-time software design, systems programming, embedded software development and assembly language programming, all using real commercial hardware and open source software. Sure, it may be a stupid idea. But damn it, it's my stupid idea.

When I first started thinking about this, I scribbled down some basic goals.

The cost of entry has to be reasonable.

A university had to be able to put together a teaching laboratory with several stations at a reasonable cost. And although a university CS, CEG or EE department was the customer I had in mind, the cost shouldn't rule out the occasional individual from trying their hand at learning this stuff. This ruled out a lot of otherwise interesting evaluation boards and development kits that cost thousands of dollars or required specialized equipment to program. It also required the use of open source tools to keep the cost of the tool chain down. Using open source also admitted the possibility of modifying existing code, like the kernel or device drivers, as part of a class assignment.

(Disclaimer: if you are a professional embedded developer, or if you manage such, I would be remiss if I didn't tell you that paying for commercial tools for embedded development is, IMHO, well worth it. I've used tools like IAR Embedded Workbench and MontaVista DevRocket to extremely good effect, and would not hesitate to use them again. While embedded development is completely doable with open source tools, you will be trading time for money, and an essential element of happiness is knowing what your time is worth. If you are a university with a big endowment, you are certainly encouraged to use these tools also. Your students will no doubt encounter them in the real world of embedded product development.)

The hardware had to be commercially available.

No special developments or custom boards, nor processors that were far off the mainstream, nor stuff that was hard to find. Partly this was a cost issue, but also I wanted it to be likely that students using this platform would encounter something very similar in the work place.

The instruction set of the processor had to be amenable to learning assembly language programming.

The advent of reduced instruction set computer (RISC) architectures made this challenging. The PDP-11 had the simplest instruction set of any processor I've ever examined, but many current RISC machines make even the old IBM 370 instruction set seem simple in comparison. The platform had to have an instruction set that was rich enough to not be too painful to use and to illustrate all the necessary concepts, yet simple enough to be taught in sufficient depth in perhaps a ten week college course.

The processor had to be widely used in actual devices, especially consumer hand-held devices.

I wanted students to use processor architectures that they would be likely to encounter in the real world. I wanted them to look at their PDA, cell phone, MP3 player, or other hand-held, and be able say to themselves "I'm writing code that could run on this device".

The platform had to be sufficient to run Linux, but had to be resource constrained enough that developing for it wasn't like developing for a Pentium-based Linux desktop.

I've used a lot of real-time operating systems (RTOS) during my years in embedded development: roll-your-own, pSOS, VxWorks, PSX/C-Executive, and RTX are the ones that come immediately to mind. But Linux is becoming more and more common in embedded devices, as the devices become more powerful, and tools become available that allow Linux to run in a smaller memory footprint. I had already gotten Linux running on several embedded platforms, ranging from one that booted from ROM and ran only single-user with a RAM disk, to one that booted and ran multi-user from a persistent disk-like file system.

Still, embedded systems have resource constraints that make developing for them unlike developing for your typical white box PC. In fact, the resource constraints often get tighter and tighter, because companies continue to shoehorn microcontrollers into smaller and cheaper devices. The ability to run Linux on a resource constrained platform was the best of both worlds: familiar, relatively inexpensive, and challenging.

Realistically, this meant a 32-bit processor with a memory management unit (MMU).

The platform had to be capable of running a lighter weight real-time operating system without undue pain.

It couldn't just run Linux, because Linux isn't always the answer, particularly for hard-real-time or highly memory constrained applications. There had to be at least the possibility of running some other open source or at least inexpensive RTOS without a huge amount of pain writing a board support package to initialize the processor, the MMU, etc. The fact that the platform could run Linux meant that much of this code could be cribbed from the Linux boot code for the platform and used in other, perhaps standalone, applications.

The platform had to have a rich set of contemporary, useful devices, including Ethernet and removable storage media.

Students had to have access to devices both for accessing and running the system and for programming as part of their class assignments. Removable media would allow them to take their projects with them when they left their station in the laboratory.

For cost reasons, this almost certainly required a system-on-a-chip (SoC), in which I/O controllers are embedded in the processor chip itself. This has become common in real-life embedded development, with ARM-based and PowerPC-based processors having integrated controllers for technologies like serial, Ethernet, and even USB. It is also a good example of specialized multi-core architecture.

The platform must include a boot-loader in ROM.

I didn't care so much what the boot-loader was, although something mainstream like U-Boot, widely used in the embedded community, was very desirable. The boot-loader was necessary so that each station didn't require an exotic (read: expensive) tool, like as a JTAG hardware debugger, to load software. The ability to boot from removable media or the network was very highly desirable. Having a boot-loader whose source code was open source also meant that code to initialize the processor could be cribbed from this and used elsewhere.

The tool chain must provide C and C++ compilers and an assembler, and be expandable to include other languages.

C is the lingua franca of embedded development, and is an absolute requirement if you are going to go about hacking the Linux kernel. C++ is, in my personal experience, a lot more common in embedded application development than embedded development pundits would have you believe. If you have C and C++, chances are you are getting an assembler as part of the deal too, but in any case it's required if you are going to learn even a little bit about assembly language.

But I wanted students to be able to explore other options as well, like Java, or Digital Mars' D, or even scripting languages. A tool chain based on the GNU Compiler Collection (GCC) seemed like a pretty open-ended choice.

The platform must expose its devices physically and programmatically.

By this I mean I wanted the chips physically accessible, and didn't want the lower level device details hidden by a BIOS in ROM. I wanted the possibility of students getting experience using logic analyzers, oscilloscopes, or other diagnostic equipment, because that's part of the real-life embedded experience. Not having a ROM meant that existing Linux device drivers for the platform could be studied in all their glorious awful detail regarding the hardware interface and perhaps even modified as part of a class assignment.

The platform should be an unlikely theft target.

When I started contemplating this project, my first step was to Google around to see what others had done before me. (I also interviewed some of my embedded mentors from Bell Labs, Tam, Doug and Randy, quizzed my former colleagues from graduate school who were still in academia, Dale and David, and even pinged my former professors, most of whom had long since retired.)

I was particularly impressed with the work of Arnold Berger (University of Washington) and Dennis Brylow (Marquette University), both of whom tackled the problem of teaching embedded software development. (In fact, I will excuse you if you simply ignore everything else I have to say on this topic and go off and base a course around their work, so delighted I will be that you are doing anything at all.) Berger took a approach very similar to mine. Brylow did what I thought was pretty damn clever and used existing consumer devices like LinkSys routers that could be pried open and programmed and modified. Before embarking on this project, I also experimented with my own Nokia N800 hand-held to see if it could be used as platform.

But Berger did his work in 2000, so his choices for hardware were different from mine. Things change a lot in just a few years. And while consumer devices like the LinkSys and N800 were more cost effective, I didn't think they exposed the hardware sufficiently, and more importantly, I thought they would be attractive theft targets, having had some experience with that during my time in academia.

The platform should have a form factor that is typical of prototypes in the real world of embedded software development.

I just liked the idea of students laboring over a circuit board. It was nostalgic, particularly of the prototype phase of a project before the physical cabinet or chassis has been completed.

My next step was to choose some candidates for processors. I admit I had my own prejudices.

The Motorola (now Freescale Semiconductor) PowerPC was the original processor architecture of the IBM RS/6000, with which I've also had much experience during the supercomputing phase of my career. It was also the architecture of choice for almost every embedded telecommunications product I've ever worked on. I wasn't that crazy about its instruction set from a teaching point of view, having had to learn enough of it to debug the occasional ugly problem. Development kits were easy to come by, but a little more expensive than I was happy with. SoC implementations were common. I was very familiar with running Linux and other RTOSes on such processors, and was confident that the project was completely doable on the PowerPC.

The ARM series of processor architectures can be found in, I dunno, thousands of consumer devices. ARM doesn't actually produce hardware, they produce IP (intellectual property) processor cores that can be dropped into various logic devices. Which means everybody and his brother (247 licensees to be exact, including Freescale Semiconductor) makes processor chips that run the ARM instruction set. I've worked on a couple of projects that used ARM cores, one of which ran Linux. I probably own a dozen devices that contain ARM-based processors, several within arm's reach as I write this. The ARM instruction set is pretty nice, a little reminiscent of the PDP-11, particularly in its 16-bit Thumb mode. ARM-based development kits are widely available and reasonably priced, and SoC implementations are easy to come by.

The next step was crucial to making life easier for all of us: I downloaded the latest Linux 2.6 kernel, ran make menuconfig for ARCH=arm and ARCH=powerpc, and explored the kernel configuration menu for what development kits were already supported with board support packages by the kernel and its build environment. I limited my Google search for development kits to these choices.

No use in making life harder than it needs to be.

Eventually I settled on the Atmel AT91RM9200-EK evalutation kit. This board has an AT91RM9200 SoC based on the ARM9 core, has the U-Boot bootloader in flash, an MMU, 32 megabytes of RAM, 10/100 Base-T Ethernet, several serial ports, supports removable media in the form of both SD cards and USB drives, and has a JTAG debugging port.

In upcoming articles I'll describe the AT91RM9200-EK in detail, and discuss how to get Linux running on it.

Diminuto and the War Against Abstraction

2008-08-26T11:01:00.013-06:00

I don't want to know how it works under the hood.
-- Former coworker and Java developer

Decades ago, back when software was programmed instead of developed, practitioners of the art were classified into two broad categories: application programmers and systems programmers. The former wrote programs that used databases and produced reports. The latter understood how everything actually worked under the hood and was able to fix it when it broke.

Today we just call all of them software developers. Most of them develop software that uses databases and produces reports (typically in HTML). And systems programmers, in the classical sense, appear to me to be few and far between.

Several years ago, in his blog Joel On Software, Joel Spolsky wrote about The Perils of JavaSchools (which I just re-read in his latest book More Joel on Software). Joel bemoaned the transition of many computer science curricula to exclusively teaching Java. CS programs want a simple programming language that allows them to teach important concepts without unnecessary pain. Joel's argument was that a little pain is a good thing: until you have to debug something where the only diagnostic you get is "Segmentation Fault", you aren't really going to understand what's going on under the hood.

Now don't get me wrong: I've made good money developing in Java, enjoyed it, and was glad to get the work. I helped develop a successful commercial telecommunications product whose firmware I wrote partly in Java. I paid my dues in the Enterprise Java wars by working on a team that developed a business process automation product based on the Java Business Integration specification. I've been known to sling some open source Java stuff out there too.

But even though I wrote a good portion of that firmware in Java, I wrote the underlying device drivers in C. And when that enterprise product got sideways, I had to dive in and figure out how the heck threads where being managed in the framework we were using. Even the relatively simple open source stuff required that I figure out how the JMX protocol interacted with internet firewalls.

For my typical daily work, much of which involves writing firmware for custom hardware that I've never seen before, I'd be glad to see "Segmentation Fault" on a display when my code fails. Hell, I'd be glad to see an LED change state. Sometimes all I notice is that things have gotten really quiet as the switching power supply gears down.

The software industry has seen immense growth since I started programming in 1970. The typical hardware platform for both the desktop and servers has undergone a brutal natural selection to where it seems like everything is a Pentium processor running either Linux or Windows, depending on your religion, nearly all user interfaces are now browser based, and even those that aren't are still graphical. Everything else was just too expensive, or too hard to use, and were winnowed out in a process that Darwin would have appreciated without understanding any of the technology. Even Apple was subject to this effect, moving to a version of UNIX and eventually into the house of Intel.

This evolution has served us well. It's why I can afford to have a quad-core server in the basement. And why Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) loves our DVR.

But, see, here's the thing: it only seems that way. In fact, the number of embedded processors which are not Pentium chips, frequently running an operating system other than Linux or Windows, has exploded as consumer devices like cell phones, internet tablets, PDAs, GPS receivers, MP3 players, and other hand-held devices have taken over our purses, briefcases, and messenger bags. Even more so is the introduction of microprocessors into our everyday lives in places where we aren't really aware of them, from our kitchen appliances to our vehicles to our office equipment.

Someone has to know how to develop software for these kinds of devices: weird-ass hardware that no one has seen before, stuff that has severe resource constraints in terms of CPU speed and memory footprint, complex interfaces to physical hardware, and form factors so small your hardware guy has problems getting a logic probe on to a pin to tell you what his hardware thinks it's saying to your software. You can't just grab a Java framework that you found via Google to solve these kinds of problems.

And with the advent of multi-core servers, and the breakage of simple consistent memory models on such systems, developers using C, C++, and even Java and C#, on white box Pentium servers now have to have a deeper understanding of threading models and how the hardware handles data coherency between processor cores. Simple errors like "Segmentation Fault" will be a fond memory. Failures will instead be of the nature of "it works most of the time" and "we can't reproduce it".

Yeah, I know I sound like a cranky old man: I remember when I had to walk three miles to the computer center, uphill both ways, through a blinding snow storm, in order to use a keypunch to write software in IBM 360 assembler. But to be a systems programmer, you must understand, to significant depth, what is going on under the hood, no matter whether you are writing in Java, C#, C++, C, or assembly language. You have to be both willing and able to tunnel through all the layers of code to figure out how something works. You must be fearless. You must be a foot soldier in the war against abstraction.

Alas, it will be an uphill battle. Because abstraction isn't just a good thing, it is a necessary thing. All human knowledge builds on what came before it. Engineering is the invention of abstractions and the layering of them on top of one another in the quest for conceptual simplicity. If, in order to have a new car, we each had to start by mining ore with our hands, we'd all still be walking.

But systems programmers must be able to operate on many levels of abstraction simultaneously. They must be able to appreciate that a Java framework supports inversion of control to connect several web services components, while at the same time understanding when the volatile keyword is necessary for the JVM to insert the necessary memory barrier machine instructions.

Believe it or not, universities used to teach systems programming. They didn't call it that, but that was the mindset that they were trying to instill. For me, it was CS431 (later CEG431), "Real-Time Software Design", which I took from Professor Robert Dixon, way back in 1976. In this course, students developed a tiny real-time operating system that supported semaphores and message passing, and then used it to develop a multi-threaded application with interrupt-driven memory-mapped device drivers. It was all written in assembler language. It had to run on an actual PDP-11 minicomputer with real devices. It had to run all of the devices at their rated speed. And to pass the course, you had to take a core dump of your running system and do a complete dump analysis to the satisfaction of the instructor.

CS431 was required for undergraduate computer science and computer engineering majors to graduate, and for graduate students in those majors to take upper division courses. It had a 50% attrition rate. I passed it. Barely. And, remarkably, when I went on to graduate school, I ended up teaching it. (And Bob Dixon became my thesis advisor and mentor.)

It was the best of courses. It was the worst of courses. For some students, it prevented them from ever completing their degree. For others, it was the first time in their academic careers they had ever been challenged, and they loved it. I was somewhere in the middle. It was a bitch of a course to pass, but since at the time I was working full-time in a job where my title actually was "systems programmer", I was compelled to understand it.

Looking back, I find that CS431 has informed a successful career on my part for the past 30+ years. No matter what I do for a living, from writing board support packages for custom hardware, to developing protocol stacks from reams of specifications, to designing network traffic management firmware, to writing Java frameworks, there's a little bit of CS431 in all of it.

If you were to teach such a course today, what would you use? The PDP-11 evolved into VAX which evolved into the Alpha, all of which died when the Digital Equipment Corporation merged with Compaq who then merged with Hewlett-Packard. What processor architecture would you choose? What hardware? What kind of software would you write for it? What language would you use? What kinds of projects could you assign?

Diminuto is my small, personal attempt to answer those questions. It is one megalomaniacal supervillain's vision for teaching a new generation of systems programmers. I'll be writing about it in the articles to come.

Large Hadron Rap

2008-08-01T17:31:00.003-06:00

I've been amusing myself by telling my friends that the solution to the Fermi Paradox is that civilizations exist just long enough to build a Large Hadron Supercollider. (And check out these incredible photographs.) Even so, the destruction of Earth may just have been worth it for this, the Large Hadron Rap:

Malcolm Gladwell on the Challenge of Hiring in the Modern World

2008-07-09T09:51:00.003-06:00

I'd be the first to admit that I'm a Malcolm Gladwell fanboy. Gladwell is a writer for the New Yorker and the author of The Tipping Point, Blink, and the upcoming Outliers. I don't always agree with what he says, or how he says it, but that makes him even more valuable to my mind.

In this twenty-minute or so video from the 2008 New Yorker Conference, Gladwell talks about one of my favorite topics, measurement dysfunction (although he never uses that term) , as applied to choosing who to hire. Like everything he says or writes, it's fascinating. He talks about the the total lack of correlation between the pre-hiring evaluation and post-hiring performance of professional hockey and football players, school teachers, and lawyers. He has lots more examples that didn't fit in the time allotted for his talk.

I'm sure that my various co-workers are tired of hearing me talk about how freelancing and contracting is the inevitable future employment model, and that the only thing that is holding it back now is job lock because of the lack of universal health care in the United States. There are two reasons why this is our future.

Competitive pressures are increasingly making it difficult for organizations to keep talented (and hence expensive) employees on the payroll without keeping them working on projects that directly lead to revenue generation in the short term. This is not good in the long term for either the organization or the employee in an information economy. It's a form of eating your seed corn and it can only be successful in the long term if your employees are an easily replaceable fungible commodity. (Hint: they're not. Regardless of the state of the economy, there is always a competitive marketplace for the really good people.)
It's impossible to know who to hire. There is no test that tells you who the great employees will be. Past performance is no predictor of future performance. There is no way to tell ahead of time which job candidate will be a good match for your organizational culture and for the task at hand. The only way to know for sure is to hire them and try them out. This is the crux of Gladwell's talk. That's why I think the wave of the future is hiring contractors, and then keep hiring the good ones. And not hiring them as regular employees (see point #1). (Yes, I realize there's an IRS issue here. That needs to be addressed along with the health care issue.)

Gladwell's talk is worth your time if you have any input into hiring in your organization. And even if you don't.

Thinking About Software with Gödel and Cantor

2008-07-08T11:18:00.020-06:00

Most people that work with me are surprised to learn that I test at the far end of the introversion scale on the Myers-Briggs Type Indicator personality test (I'm an INTJ). Since I have a reputation as a very hands-on engineer, they might also be surprised by the fact that I enjoyed taking a lot of computer science theory classes when I was in graduate school: formal languages, computability, algorithms, etc. I insist on finding practical applications for this stuff all the time. And occasionally I entertain myself by thinking about things like Gödel Numbers.

In 1931, philosopher, logician, and mathematician Kurt Gödel showed how you could take any formal language and represent it as a mathematical function by encoding the symbols of the language numerically and representing the productions of the language as a function that operates on the resulting numbers. Whenever you have formal systems A and B, showing that one can be converted into the other is always a win, because it's a form of code reuse: proving something about A now also applies to B. It didn't take long to show that a Gödel number could be represented as a string that could be operated on by a Turing Machine, adding yet another formal system to the party.

You can trivially apply this to any piece of software as well. Just take a core dump of your program's initial state in memory. You can think of this result as a hexadecimal representation of its Gödel number. Now take another dump of your program after every instruction executes. You end up with a series of Gödel numbers. Now write a single mathematical formula that generates this series of numbers. You have just converted your program into a abstract function amenable to all sorts of formal analysis. True, it's so large and ugly that such analysis may be intractable, but it's at least theoretically possible.

Remember those intelligence tests you took in grade school where you had to choose the next number in a sequence? I am sorry to inform you that any number can be the correct answer. It is trivially easy to write a mathematical function to generate any arbitrary sequence of numbers. As Mrs. Overclock (a.k.a Dr. Overclock, Medicine Woman) frequently points out, the fact that she and I both did well in school has more to do with the fact that we are good at taking standardized tests, not that we're necessarily smart. Of course, knowing what answer is expected is itself a useful life skill. But I digress...

Theoretical computer scientists are fond of talking about Turing Machines with infinitely long tapes. They do this mostly so that the are not constrained by physical limitations. What they really mean is "the tape is just as long as it needs to be". The fact that there are algorithms that ultimately try to march along to the end of an infinite tape is part of the fun. This is a form of the Halting Problem.

We can do this with our abstract mathematical function too. We just assume we always have enough bits to represent any possible Gödel number we want to compute. An infinite number of bits should be about right. But what kind of infinity?

It turns out that there are different orders of infinity. Around 1874, set theorist Georg Cantor showed that indeed while the number of integer numbers (or the cardinality of the set of integer numbers to be more precise) was infinite, and the numbers of real numbers (ditto) was infinite, there had to be more real numbers than integer numbers. That's because we could form a one-to-one correspondence between the set of integer numbers

(... -1, 0, 1, 2, 3, ...)

and the set of real-numbers

(... -1.0, 0.0, 1.0, 2.0, 3.0, ...)

but in between each of those real numbers was another infinite series of entries in the set of real numbers

(... -1.0, ... 0.0, 0.001, ... 0.01, ... 0.1, ... 0.2, ... 1.0, ... 2.0, ... 3.0, ...) .

Cantor called the transfinite number that was the cardinality of the set of all integers ℵ₀ ("aleph null") and the transfinite (but larger) number that was the cardinality of the set of all real numbers ℵ₁ ("aleph one").

What is the relationship between ℵ₀ and ℵ₁? You might expect that since there is an infinite number of entries in the set of real numbers in between each entry that corresponds to an entry in the infinite set of integer numbers, that ℵ₁ would equal ℵ₀², or maybe ℵ₀^ℵ₀. Not so fast...

Since we can make a one-to-one correspondence, or enumerate, each bit in our hexadecimal Gödel number with an integer number, the number of bits in our infinitely large Gödel number is ℵ₀. So how many unique Gödel numbers can we form from ℵ₀ bits? That's easy for any software developer: 2^ℵ₀.

Here's where it gets interesting (no, really). What order of infinity is 2^ℵ₀? It turns out, according to Cantor's continuum hypothesis, that this value is ℵ₁, the cardinality of the set of all real numbers.

Coincidence? You decide.

Daily Organization

2008-07-08T09:04:00.018-06:00

The nature of the work I do frequently involves custom hardware and a lot of expensive diagnostic instruments. That typically precludes working from home. (Besides, when you're as lucky as I have been to work with such great people, why would you want to?) In Red, Yellow, Green Blue I described my dirt simple way of keeping stuff organized in the briefcase I schlepp around to customer sites. Now I'll describe how I keep organized while I'm working at the customer site.

I spent years in academia, more years at a national laboratory rooted in academia, and years again at Bell Labs and two of its spin-offs with a long tradition in research and development. The lab notebook was a fixture at all of these places. Yet its utility wasn't apparent to me until I began freelancing. Go figure. Now I wonder how I ever lived without it. It's a tool for organization, an indispensable part of my billing process, a cache memory, and an archival storage mechanism, all rolled into a handy item that fits in a briefcase or a file cabinet.

The lab notebook is traditionally a hardbound notebook with blank ruled pages, and there are still plenty of folks that use those, especially if the company is paying for them. But it can also be nothing more than the spiral bound notebook like you may have used in college. I use something in between. John Dickinson is a British company that makes a series of spiral-bound Black n' Red notebooks with plastic covers. I use them exclusively as my lab notebooks. They are sturdy and handle the occasional beverage spill. The fact that they are in my two favorite colors is just icing on the cake. Each client has their own notebook, and I label the front with the client's name and a sequential number. I typically go through a notebook every two months, more or less, on a project. Although they come in ISO standard sizes (i.e. not 8.5" x 11") they still fit in a standard file cabinet. The elastic strap riveted to the back cover keeps all my ancillary crap from falling out.

I'm always collecting odd scraps of paper, notes, and odds and ends that fit nicely in the pocket at the front of the notebook. I use TUL-brand gel pens almost exclusively, which I buy in boxes of a dozen at OfficeMax. Their ink can't be easily erased, they are reliable, and (most importantly) they fit inside the spiral binding of the notebook.

I make a note of the date at the start of every working day, and make a note of at least my start and stop times. I can usually reconstruct what I worked on that day, and approximately for how long, just by perusing my notes. (You may notice that the dates on these pages are over four months apart. Some projects work that way.) Symbols in the margins give me further guidance: stars are actions items and must be checked off; asterisks mark information that I am likely to need later; question marks are things I need to find out; arrows point me forwards or backwards to related information.

Note that my lab notebooks are the intellectual property of my company, Digital Aggregates, not of the customer. I have never had a customer question this, but sometimes it pays to be clear. If a customer calls or e-mails me with a question, the notebook comes out of my briefcase or file cabinet. Given the occasional senior moment, I've been known to refer to my archived notebooks to remember how to do things like a Linux kernel build.

I keep my own weekly time cards that I use for invoicing. Data from this paper form gets entered into a QuickBooks time card, which I use to generate monthly invoices. This paper time card also gets covered with notes like customer project numbers and billing codes. I keep a PDF version of the time card form on the web so I can always print a blank one at a customer site. I've never had a client that required me to get a signature on my time cards, but several of them do require me to use their own web-based time card system, which is very reasonable. I make sure that the customer time card and my time card agree. Accountants Payable departments don't like surprises, even if it means being invoiced for less than amount indicated in their own system. (I learned this the hard way. Never round down. In fact, never round at all. A fifteen minute discrepancy in the client's favor nearly meant a month delay in getting paid.)

In A Future Without Keyboards I mentioned that I also use a Palm T|X PDA to help stay organized. It keeps my calendar, my contact database, and a lot of notes, and I keep it synced to my ThinkPad X61 laptop. When I travel light, I carry a Nokia N810 internet tablet to keep up with email and web cruising.

But I have also learned not to depend too much on anything that requires electricity. (For example, I wear a self-winding mechanical wristwatch.) Hence, the lab notebook and the paper time card, both of which have served me well.

Geek Test

2008-05-27T18:19:00.001-06:00

48.71795% - Super Geek

I don't know whether to be happy or sad about this score.

But I do know I wanted to do better.

Generic Embedded Programming Using the C Preprocessor

2008-05-03T14:18:00.040-06:00

In the article Generic Embedded Programming Using Templates, I described how C++ templates could be used for generic programming in embedded applications. Generic programming represents a higher level of abstraction in which you express an algorithm without regard to the data types on which it operates. Experienced C developers may find that templates remind them of the C preprocessor. This is apropos, because in this article I will describe one way to use the C preprocessor to do a simple but useful form of generic programming in C.

Like C++ templates, the C preprocessor provides a powerful form of software reuse known as code generation. The symbolic substitution and macro capability provided by the C preprocessor, along with the native capabilities of the C compiler itself, lend themselves to a simple form of generic programming that can ease the burden of software maintenance. One of headaches of using the standard header files available in C and POSIX is the wealth of integer type definitions that hide what the actual data type is. For example size_t is returned by the standard I/O functions fread and fwrite. What is the range of size_t? Is is signed or unsigned? What about ssize_t which is returned by the recv socket system call? How about pid_t that is returned by the getpid system call?

Obviously, a few minutes perusing the hundreds of files under /usr/include (made simpler using a tool like cscope) will eventually lead you to discover that size_t is unsigned, and its counterpart ssize_t is signed. But on some systems size_t is thirty-two bits wide, others sixty-four bits. And on one system to which I've ported C code, it was a sixty-bit wide floating point data type, because that was the only data type that architecture had.

So what happens if you want to write portable code that assigns the minimum or maximum possible values to a variable declared as either of these two data types? You may be tempted to use the symbols MININT and MAXINT that are defined in /usr/include/values.h for the ssize_t variable, but is this safe? What if the code is moved to another machines where ssize_t isn't the same width as an int? What about all those integer data types defined in the third-party library you are porting? What if the next upgrade changes one of those type definitions you are using in your application? It's enough to make you want to turn to Java, which has a fixed set of integer data types which are the same on all platforms

But there's a simple way to write type-independent code where you generate the correct values for the integer data type automatically, providing the architecture implements true two's complement binary integers.

For size_t, or any other unsigned type, the minimum possible value is, of course, 0. So the minimum of an unsigned integer is easily defined. Let's suppose we define a simple macro to generate that value.

#define minuint(TYPE) ((TYPE)0)

This macro just casts the value zero to the appropriate type. It can be used just like this.

size_t min_size_t = minuint(size_t);

The maximum possible value of any unsigned type has all of its bit set. This is just the value ~0. So this is our next macro, and an example of its use.

#define maxuint(TYPE) ((TYPE)(~minuint(TYPE)))

size_t max_size_t = maxuint(size_t);

By now you may suspect that we can do the same for signed types. And so we shall. The minimum value of any signed number is the largest (numerically smallest) negative number. In two-complement arithmetic, this number has its most significant bit set and all of its other bits are zero. Using the C compile-time operator sizeof which returns the number of bytes in a variable or a type, this is easily computed.

#define minsint(TYPE) ((TYPE)(((TYPE)1)<<((sizeof(TYPE)*8)-1)))

ssize_t min_size_t = minsint(ssize_t);

Get it? The sizeof gives us the size of the argument TYPE. Multiplying that value by eight gives us the number of bits. Subtracting one gives us the bit position (relative to zero) of the top bit, and shifting the 1 (cast to the correct type) gives us the largest (numerically smallest) negative number.

It may come as no surprise by now that the largest signed number is again the bit-wise inversion of the smallest signed number, just as it was for unsigned numbers.

#define maxsint(TYPE) ((TYPE)(~minsint(TYPE)))

ssize_t max_size_t = maxsint(ssize_t);

So there you have it.

What's that? Yes, that's right, with these macros you have to know whether the data type is signed or unsigned. What? Really, must I do everything myself? Oh, very well; how about these macros, which will work with either signed or unsigned integer types?

#define minint(TYPE) ((maxuint(TYPE)>0)?minuint(TYPE):minsint(TYPE))

#define maxint(TYPE) ((maxuint(TYPE)>0)?maxuint(TYPE):maxsint(TYPE))

See the trick? If the integer type is signed, the value generated by the maxuint macro is interpreted as a -1. It the integer type is unsigned, the value is interpreted as the largest possible number for that data type which is always positive. The conditional expression automatically chooses the appropriate form of the minimum or maximum macros.

And here's an even better trick. All of this is done at compile time. The preprocessor expands the macros to their equivalent C expressions and submits the generated code to the C compiler. Since none of the values in the resulting expressions are run-time variables, the C compiler reduces the initializers to their equivalent integer numbers. If someone changes the underlying integer type of some type definition, the C compiler will automatically generate the correct minimum and maximum values the next time your code is built. Expanding on the trick above, you can even check to make sure that the data type is signed or unsigned.

Here's a code snippet that demonstrates the macros using a variety of integer data types.


size_t minsize = minint(size_t);
size_t maxsize = maxint(size_t);
ssize_t minssize = minint(ssize_t);
ssize_t maxssize = maxint(ssize_t);
pid_t minpid = minint(pid_t);
pid_t maxpid = maxint(pid_t);
uint64_t minuint64 = minint(uint64_t);
uint64_t maxuint64 = maxint(uint64_t);
int64_t minint64 = minint(int64_t);
int64_t maxint64 = maxint(int64_t);
short minshort = minint(short);
short maxshort = maxint(short);
unsigned short minushort = minint(unsigned short);
unsigned short maxushort = maxint(unsigned short);

printf("minsize 0x%08x %u\n", minsize, minsize);
printf("maxsize 0x%08x %u\n", maxsize, maxsize);
printf("minssize 0x%08x %d\n", minssize, minssize);
printf("maxssize 0x%08x %d\n", maxssize, maxssize);
printf("minpid 0x%08x %d\n", minpid, minpid);
printf("maxpid 0x%08x %d\n", maxpid, maxpid);
printf("minuint64 0x%016llx %llu\n", minuint64, minuint64);
printf("maxuint64 0x%016llx %llu\n", maxuint64, maxuint64);
printf("minint64 0x%016llx %lld\n", minint64, minint64);
printf("maxint64 0x%016llx %lld\n", maxint64, maxint64);
printf("minshort 0x%04x %d\n", minshort, minshort);
printf("maxshort 0x%04x %d\n", maxshort, maxshort);
printf("minushort 0x%04x %d\n", minushort, minushort);
printf("maxushort 0x%04x %d\n", maxushort, maxushort);

Here are the results when you run it. (This code was tested on a P4 Linux system and a P4 Cygwin system.)

minsize 0x00000000 0
maxsize 0xffffffff 4294967295
minssize 0x80000000 -2147483648
maxssize 0x7fffffff 2147483647
minpid 0x80000000 -2147483648
maxpid 0x7fffffff 2147483647
minuint64 0x0000000000000000 0
maxuint64 0xffffffffffffffff 18446744073709551615
minint64 0x8000000000000000 -9223372036854775808
maxint64 0x7fffffffffffffff 9223372036854775807
minshort 0xffff8000 -32768
maxshort 0x7fff 32767
minushort 0x0000 0
maxushort 0xffff 65535

This technique, as simple as it is, can make your C code more portable, more readable by eliminating some magic numbers, and more easily maintainable.

Update (2008-05-04)

When I write production code I have a habit of cranking the warning level of the GNU C compiler up to the max. This causes the compiler to object to checking for unsigned values being negative. To circumvent this in the versions of these macros I wrote for the Desperado C++ library, I reversed the check in the equivalent of the maxint and minint macros. I made this change in the macros above from their form in my original article.

Update (2008-12-03)

Jeremy Overesch identified a bug in the code above which I have corrected in this version of the article. The far more complicated Desperado C++ code, from which these macros were derived, had it right. In my effort to make the code readable, I broke it. And since I didn't port the full unit test from Desperado, I missed the bug. Shame on me, and a big Thank You to Jeremy.

Update (2009-03-16)

The Blogger editor seems intent on hosing up the source code in this article, regardless of how it looks in the preview. I apologize in advance for any formatting weirdness that I miss.

Globalization and Amazon.com

2008-04-27T06:53:00.007-06:00

I had a The World Is Flat moment the other day. Some random synapse in my brain decided that I should read the old Alistair MacLean cold-war thriller Ice Station Zebra. I've always been a fan of the movie (me and Howard Hughes) and the soundtrack by Michel Legrand, but had never read anything by MacLean. So I hied me over to Amazon.com to order the book.

I was a little surprised to find it was out of print. But thanks to Amazon's Marketplace service, several used and even new copies were available from any number of sources. I'm no stranger to this service, in which independent booksellers use Amazon as a storefront. In Outsourcing Technical Services for Small Businesses, I wrote about how this service was a great way to get barely used copies of expensive or out of print technical books (the only thing that depreciates as quickly as high tech is fresh fruit). Nearly instant gratification was only a few clicks away.

It was only after I got the verification e-mail that I realized I had ordered the book from a bookstore in the U.K. Well, okay, I'm no stranger to that either. It seems that many of my favorite authors these days are Scottish: Iain Banks, Ian McDonald, Ken MacLeod, Charles Stross. I discovered that I can order one of their books a year in advance of it appearing in the U.S. by getting it through Amazon.co.uk, which is no more click-distant for me than its U.S. counterpart. (Note to U.S. publishers: the cover art on the U.K. editions is far superior too.)

It wasn't until the MacLean book arrived at the Palatial Overclock Estate (a.k.a. the Heavily Armed Overclock Compound) in an envelope covered in Deutsche Post stickers that I discovered that it came from a warehouse in Germany.

Man, you just gotta love the web. It isn't just that these things are possible. It's that they're so easy that you don't even realize that they're happening.

But it isn't just the web. It's the huge number of independent vendors who use the infrastructures provided by Amazon and eBay to reach out and e-touch someone. It's the amazing international shipping services that somehow transport tangible goods almost as fast as the Internet transports intangible bits. And, in the case of my book, they can do it all for under ten bucks.

I'm amazed. But apparently, some bean counters are disturbed.

The April 28th issue of BusinessWeek had an article on "The World's 50 Most Innovative Companies". Jeff Bezos was on the cover, Amazon being #11. Web services developers are already quite aware of the infrastructure Amazon can provide, quite cost effectively, to e-commerce web sites.

A week later, in the feedback section of the May 5th issue, a reader identified by the screen name "Beejat" writes:

If e-retailers like Amazon are offering computing on tap, doesn't it indicate that they have overinvested in technology? ... I am convinced Amazon has too much firepower - more than it needs to run its business. Its emphasis on its corporate computing market looks like a positive spin on a rather poor capital investment strategy.

Beejat, I know where you're coming from. You're coming from a bottom line, quarterly results, improving shareholder value, reducing costs point of view. The point of view that makes investors money in the short term, while eliminating any vision of the long term.

But Beejat, here's where you're confused. Selling books isn't Amazon's business. Creating the future of e-commerce is their business. Selling me stuff is just the way they unit test it.

The Future Has a Way of Happening

2008-04-20T17:29:00.013-06:00

In his blog Coding Horror, Jeff Atwood answers the question: Should All Developers Have Manycore CPUs? Probably to no one's surprise, the answer is "yeah, duh". Jeff concentrates mostly on the performance and productivity issues. For me, it's all about economics too, but not necessarily the economics Jeff is talking about.

Let's suppose a company has a product that generates, directly or indirectly (as, through service contracts) the bulk of its revenue. I'm talking hundreds of millions of dollars a year. It's the kind of revenue stream most of us poor sods working for tiny start-ups just dream about. This single product is without question the crown jewel of their product line. The installed user base is huge, global, and growing. It is considered by its customers to be a critical and strategic piece of their enterprise infrastructure, and has been for decades.

But, see, here's the thing: this product evolved from code written as far back as the late 1970s. By some estimates, the code base is eight million lines of code, mostly C. Much of it was designed and implemented long before multi-processor servers were cost effective, existing only here or there in various corporate and government labs. Although the application is multi-threaded, most of it was written before there was any way to run it on a platform where the race conditions that might occur in a truly concurrent environment could be exercised.

It wasn't (necessarily) that the developers -- and there would be hundreds of them over the years -- knowingly wrote code that wouldn't work on a multi-core platform. It's just that, well, it wasn't an issue. These developers were more worried about being attacked by a saber-toothed tiger or trampled by a herd of mastadon on the long commute home across the ice pack, then some hypothetical science-fictional multi-core future.

And, of course, even if the occasional forward thinking developer was worried, there was no way to test it.

Fast forward thirty years later. Multi-core servers are now cheap. So cheap, server manufacturers will soon see no point in making a server that isn't multi-core. Even if they wanted to, the day is coming when the microprocessor chip manufacturers will no longer make single-core microprocessors except perhaps for specialized, embedded applications. Chip manufacturers have been pushed into the multi-core future by having hit the wall for CPU clock rates, due mainly to issues in power dissipation. And they're dragging the rest of us in to the multi-core future, some of us kicking and screaming.

This hypothetical company's product, the thing that pays all the bills, won't run reliably on the current generation of servers. And they can't make it work. Finding all the race conditions that show up in a thousand lines of code is a challenge. Doing so in an eight million line code base is for all practical purposes impossible. They've also hit the CPU clock speed wall, and are reduced to limiting their application to run on a single core, while the other cores run (occasionally) an Apache web server for the system administration interface, or maybe blinks some LEDs.

In other words, they are screwed.

I don't want to make it sound like anyone is at fault here. For sure, my crystal ball hasn't been any better than theirs was over the past thirty years. At one time, I actually worked at a national lab which had bunch of those multi-processor, shared-memory systems. We called them supercomputers. But it never occurred to me that I'd have one of those systems in my home. In my basement. Under a table. And that it would have cost only about a grand, U.S. That's just crazy talk.

But for this company, the future caught up with them. When it came to designing for future multi-processor architectures, they effectively, and correctly, practiced an agile development methodology. Unfortunately, the future has a way of happening, and the agile principle of "you ain't gonna need it" eventually turned into "you need it right now, and there ain't no way to get it".

(There is some irony here. This same hypothetical company let go -- or otherwise drove away or simply ignored -- nearly all of their firmware developers. These were developers who were quite experienced in designing code for multi-processor shared-memory architectures, albeit for embedded applications. These guys and gals dreamed concurrently in their sleep, and could have been a valuable development resource for the server-side application architecture. But that amazing short-sightedness is a topic for another article.)

So with all due respect to Jeff Atwood, here's why I think all of your developers and testers should have a multi-core platform on which to write and test their applications: if they don't, and if your product has any success at all in the marketplace, sooner or later you will be screwed too. It's simply a matter of developing and testing on the platform you expect to deploy to the field. Or, as the rocket scientists say, "test what you ship, ship what you test".

I've written about some of the issues in the multi-core future in Small Is Beautiful, But Many Is Scary and its follow-up, and my alter-ego has given a talk on problems software developers are running into in writing multi-threaded code for multi-core (and, in the case of hyper-threading, sometimes even single-core) platforms. The company that issues me my paycheck, Digital Aggregates, put its money where my mouth is: we just added a Dell 530 with an Intel 6600 2.4 GHz quad-core processor to the server farm.