Friday, September 26, 2008

Accessibility

I recently took a nine day motorcycle trip from Denver Colorado to Dayton Ohio to visit family and friends. I rode my BMW R1100RT about 2500 miles (maybe 4000 kilometers) round trip. When I'm riding solo eight to ten hours a day without benefit of distractions like music or conversation, I spend a lot of time paying attention to what goes by and listening to what Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) calls my ThinkMan™. Here is some of my play list for that trip.

Route US-36 splits off from Interstate 70 just East of Denver. US-36 has a completely different character depending on what state you're traveling through. In Eastern Colorado it's kind of spooky, passing through ghost towns of shuttered buildings. In Kansas it's a two lane road with lots of local traffic and a little town every fifty miles or so. In Missouri, it's a four-lane divided controlled-access highway. In Illinois, it is either nothing but two-lane asphalt going through endless corn fields, or it merges with I-72. My travels on US-36 end in Indianapolis Indiana, where I hopped onto I-70 for the final run to Dayton. Where as I-70 is just a vast ribbon of asphalt from Denver to Dayton, US-36 has enough variety to keep your attention.

I like staying in the little Mom and Pop motels along the way, where you ring the bell at the counter and someone leaves the television in their living room to check you in. I was traveling with my Nokia N810, so tiny it takes up almost no room in my motorcycle luggage. The internet tablet and the availability of WiFi made it easy to keep up with my email or do a little web cruising. I was amazed at how many of these little motels had WiFi. For example, both motels in Smith Center Kansas (near the geographic center of the contiguous United States) had hand-written signs "WiFi Internet" in their office windows.

I realized it was cable television that made the WiFi easy. When cable television became a "must have" for every little motel, it forced them to put in the broadband infrastructure. Once that was done, getting internet access over the cable and installing a LinkSys WAP was a piece of cake. And nearly all of them were LinkSys: seldom did I see the WAP SSID changed from its out-of-the-box default.

At first I thought that seeing URLs on billboards advertising farm equipment in the middle of Kansas was a sign that the web really had gone mainstream. But I suspect that farmers in the middle of Kansas may have realized the usefulness of the web and the internet long before many city dwellers did. When you're relatively isolated, being able to mouse-up Amazon.com and order just about anything from a Long Tail selection larger than any urban store might seem especially important. And since the internet was originally designed so that military communication would survive a nuclear war, being able to stay in touch in the aftermath of tornadoes and blizzards is pretty darned useful too.

My experience is that farmers and ranchers are pragmatic adopters and exploiters of not just bio-tech and agri-tech, but just about any useful-tech. A friend of mine once worked for a company that developed a local area network technology designed to transmit across barbed wire. Another worked with a rancher who tagged his cattle with RFID-like devices to automatically track how long each animal stood at the feeding trough. This is pretty cool stuff.

The ghost towns along US-36 got me to thinking, is there some critical mass that is necessary to keep a town alive? Did the conversion of the more southern Route US-40 to I-70 kill the towns in Colorado, but not those in Kansas and Missouri?

When I was but a lad growing up in Ohio, shortly after the glaciers receded, I had reason to spend some part of most summers in rural Eastern Kentucky. No running water, one broadcast television channel, electricity most of the time. What time wasn't spent on chores was spent reading, shooting, crashing a dirt bike, or just goofing around. A twenty minute drive in either direction on the road would get you, depending on which direction you turned out of the gravel driveway, to Grayson or Sandy Hook. Both towns had main streets maybe a block or two long with no building taller than two stories. Sandy Hook had a bit of an edge for us, since it is the seat of Elliott county where my family owned land, and had the high school and the funeral home. Grayson is the county seat of the adjacent Carter county.

Decades later Mrs. Overclock and I flew into Lexington Kentucky and rented a car to drive to Sandy Hook for a funeral, while staying at a motel in Grayson. I was startled by the contrast between the two towns. Grayson was huge and sprawling, and many choices of chain hotels, restaurants, and retail. Sandy Hook looked like a ghost town, with nearly all the buildings shuttered and boarded up. Only the high school, the funeral home, and the county buildings remained.

It was the interstate highway of course. In the intervening decades, Interstate 64 had been built, and it had a Grayson exit. Also, the U. S. Army Corps of Engineers undertook a huge flood control project between the two towns (incidentally permanently flooding the property where my mom's family originally had their farm). The resulting dam, reservoir and lake, and no doubt the beautiful Appalachian scenery, motivated Kentucky to develop a state park and recreation area. Regardless of the fact that the lake was pretty much in between both towns, it was named "Grayson Lake State Park". Tourism joined tobacco and coal mining as a major industry in the area. Grayson prospered. Sandy Hook mostly disappeared.

During the intervening decades between wandering in the woods with a firearm and flying around the country with Mrs. Overclock, I was a student of Computer Science at Wright State University. The WSU main campus is in Fairborn Ohio on what was once farm land. (Indeed, my thesis advisor and mentor Bob Dixon was the first faculty member hired by the University, and his initial office was in a farm house standing on the property.) And when I was there, it still mostly looked like farm land: woods and rolling hills. Unfortunately, that was about all there was around campus. If you wanted to dine off campus, you had to get in a car and drive at least fifteen minutes even to get fast food.

Then Interstate 675, a bypass around Dayton Ohio connecting I-75 and I-70, was built, with several exits for campus and the nearby Wright-Patterson Air Force Base. You can guess how this story ends. I barely recognize the area now. Enormous development, condos, retail, dining. And no doubt at least in part due to the greater accessibility that I-675 provided the University, the campus must be about triple its size than when I was a student, with about 16,000 students (the majority of which, interestingly enough, are women).

It probably goes without saying the accessibility is a key factor in the ability to develop and grow. And of course, it's not just physical accessibility, but virtual accessibility; maybe even more so. Many years ago, my old comrade Doug Supp (who is still at Wright State, but is surely thinking about retirement by now) and I wrote a proposal to bring the internet to the University. It seems laughable now, but way back then, back in the 1980s, we actually had to sell it. Not everyone was convinced that this new fangled internet thing was worth the money, or that it would amount to anything. After all, this predated the World Wide Web, so we were really talking about technologies like electronic mail, telnet, USENET, and file transfer being the killer applications.

Doug and I used the growth we saw that was so evident outside of our office windows as a result of I-675 as a rationale for the project, which we called TURNPIKE, trying to draw the analogy between physical accessibility and virtual accessibility as important for the growth of the University. Fortunately for all involved, the University bought into it. It was by no means a sure thing. By the way, TURNPIKE stood for The University Resource Network for the Pursuit of Information, Knowledge and Education. Yeah, we had to rack our brains for that one.

(Wright State University is also known for having a handicap-accessible campus right from its very inception in the late 1960s. Another victory for accessibility.)

Tooling along US-36 cross country with the hum of the 1100cc opposed twin engine in my ears, I had a lot of time to think about how important accessibility is.

And gas stations. Always on the lookout for gas stations.

(Update 2022-07-31: minor corrections.)

Thursday, September 25, 2008

From Diminuto to Arroyo

If you've lasted this long reading my articles on Diminuto, you know it's my attempt to create an environment for teaching real-time software design, embedded development, and assembly language programming, using commercially available hardware and open source software. Diminuto uses the Atmel AT91RM9200-EK evaluation kit (EK), a single board computer (SBC) that has an ARM9 processor with memory management and a host of peripherals, including Ethernet, USB ports, and a Secure Digital (SD) card slot. You may also recall that Diminuto was built using uClibc, a reduced memory footprint C library, and has a root file system that is memory resident.

(Not that everyone doesn't know what these look like, but below is a photograph of an SD card, which is a little larger than the size of my thumbnail, and a USB storage drive, which is about the size of my thumb. Both of these particular examples hold two gigabytes, a fact that an old guy like me finds astounding.)

Removable "Disk" Drives

Now it's time for Diminuto Phase II, which I call Arroyo. Arroyo has a vastly larger root file system that runs "disk" resident on an SD card. I'm using an EXT3 journaled file system on a 2GB SD card; the root file system utilizes less than 10% of the card. Arroyo is still based on BusyBox (1.11.2), but it uses the full Standard C Library and includes a complete Bash (3.2) shell. Because Arroyo doesn't use a RAM-resident root file system, it has a memory footprint not much larger than Diminuto. Like Diminuto, Arroyo supports USB storage drives as well, making it easy for students to take their projects with them.

(Below is a photograph of the EK SBC with the SD card in its card slot in the left rear and two USB drives in the host USB ports in the center front. Both Diminuto and Arroyo can access these devices just like disk drives. Very slow disk drives. Arroyo uses the SD card for its root file system, although in practice, as with any other Linux system, just about everything commonly used ends up cached in memory.)

Arroyo/Diminuto with Three "Disk" Drives

Arroyo was not built using Buildroot, although it does make use of the genext2fs host utility created by the same folks to build the EXT2 file system image. Arroyo was built using individual open source components including the Linux 2.6.26.3 kernel. The only patch required to any of the open source software was to the kernel to address the wrong board type code reported by U-Boot that I've discussed before.

Arroyo was created using a pre-built tool chain, Sourcery G++ Lite, provided by ARM, Ltd. and the folks at CodeSourcery. The tool chain can be downloaded for free, and provides full support for the later ARM Embedded Application Binary Interface (EABI).

I've ported the Desperado embedded C++ library to Arroyo, including John Sadler's Ficl embeddable Forth interpreter, and ran most of the unit tests.

Here is an example of booting Arroyo on the EK SBC. Note that I overrode the U-Boot bootargs environmental variable to point the kernel at the SD card for its root file system. Since the SD card device driver is hot plugged into the system when you insert the SD card, I told the kernel to delay for a couple of seconds to allow the device to come online. (You can also see this log file here.)


boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...



U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
bootargs=console=ttyS0,115200 mem=32M
filesize=1ab33c
fileaddr=21000000
ipaddr=192.168.1.223
serverip=192.168.1.222
start=bootm 21000000
netmask=255.255.255.0
gatewayip=192.168.1.1
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
stdin=serial
stdout=serial
stderr=serial

Environment size: 427/8188 bytes
Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> setenv bootargs 'console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1'
Uboot> run start
## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #35 Thu Sep 25 14:05:44 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:59 UTC (883612859)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
kjournald starting. Commit interval 5 seconds
EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex



www.diag.com Arroyo 0.0
arroyo login: root
# ifconfig
eth0 Link encap:Ethernet HWaddr 02:00:00:00:00:00
inet addr:192.168.1.223 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:195 (195.0 B) TX bytes:0 (0.0 B)
Interrupt:24 Base address:0xc000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

# df
Filesystem Size Used Available Use% Mounted on
rootfs 1.9G 133.2M 1.7G 7% /
/dev/root 1.9G 133.2M 1.7G 7% /
tmpfs 14.6M 0 14.6M 0% /tmp
# bash
[root@arroyo ~]# cat /proc/meminfo
MemTotal: 29824 kB
MemFree: 19696 kB
Buffers: 888 kB
Cached: 6552 kB
SwapCached: 0 kB
Active: 3600 kB
Inactive: 4588 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 756 kB
Mapped: 1480 kB
Slab: 1332 kB
SReclaimable: 232 kB
SUnreclaim: 1100 kB
PageTables: 92 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14912 kB
Committed_AS: 1912 kB
VmallocTotal: 989184 kB
VmallocUsed: 1088 kB
VmallocChunk: 986108 kB
[root@arroyo ~]#


Diminuto and Arroyo are both excellent teaching examples of embedded Linux systems at opposite ends of the resource spectrum, and both run on the identical EK hardware and firmware. I'll be writing more about both Diminuto and Arroyo in articles to come.

Tuesday, September 23, 2008

Diminuto Nuts and Bolts

Diminuto is my attempt to put together a platform on which to teach real-time software design, systems programming, embedded software development and assembly language programming, all using real commercial hardware and open source software. In Choosing Software for Diminuto, I talked about the software choices I used for Diminuto and my strategies for configuring them for a small memory footprint. In this this article, I'll describe in more detail how I built Diminuto and show an example of the system booting on the Atmel AT91RM9200-EK evaluation kit (EK).

Diminuto consists of the Linux 2.6.25.10 kernel, the uClibc 0.9.29 C library and its associated tool chain, and BusyBox 1.11.1. But it isn't enough to just download all this stuff off the web and build it. For one thing, the use of the reduced memory footprint uClibc library requires some patches be applied here and there in both the bootable system and its tool chain. For another, you are still faced with the need to build a root file system that can be incorporated as an initial RAM file system (initramfs) within the kernel image.

Because this work is pretty common to anyone building an embedded Linux system using the software I chose, the uClibc folks provide another tool, Buildroot, to assist with it. Buildroot is a collection of configuration utilities, scripts, and makefiles that automate much of the process of downloading, patching, and building both the tool chain that runs on the host (for example, your desktop Linux PC) and the software that boots on the target (in our case, the EK board), and creating the root file system. Some of what Buildroot does is necessary because I chose to use the uClibc C library. Some of what it does would be necessary no matter what tool chain or C library I chose to use.

As you might expect, there were some issues I had to resolve to get Buildroot and the artifacts that it creates to work on the EK board.

Buildroot is a moving target. Even more so than most other open source software, Buildroot is a moving target. It does not appear to have any release management. When you download Buildroot, you do so directly from its Subversion repository. While you may easily choose any revision you want, there is no guarantee that any particular revision will work, or even build, for the particular choice of hardware, software, and options that you want to use.

There are a lot of folks contributing to Buildroot, so the repository is under constant churn. I went through several iterations trying to get various revisions of Buildroot to work with Diminuto before settling on Buildroot 22987. The most current revision as I write this is 23454; that means there have been 467 changes since I got my copy of 22987. Your mileage will almost certainly if you choose to try using a different version of Buildroot than I did.

U-Boot on the EK board isn't quite right. U-Boot is a bootloader like GRUB, and LILO. It lives on the hardware in some persistent memory like ROM or flash. It does some basic hardware initialization necessary for booting the Linux kernel. It has a console command interface and some basic scripting capability. It can boot a Linux kernel image that resides in ROM or flash. It can also get a bootable Linux image from a server across the network using the Trivial File Transfer Protocol (TFTP). Later versions of U-Boot that I have used even understand how to read EXT2 file systems on IDE devices so that Linux can be booted from a "disk" (which as I mentioned in the prior article, may actually be a flash device). I've built embedded Linux systems to boot from U-Boot using all three of these methods: from flash, using TFTP (which is what Diminuto uses), and from a solid-state IDE "disk".

U-Boot is very popular in the embedded community, while GRUB and LILO are mostly used on PCs. The EK board comes with U-Boot 1.1.1 pre-installed in flash. This places the EK user way ahead of just having "bare metal".

However, the U-Boot that comes on the EK isn't perfect. One of the things that U-Boot does upon transferring control to the Linux kernel is pass a board type code (really just a magic integer) to the kernel, telling the kernel what kind of board it is running on. This admits the possibility of building a single kernel to run on several variations of hardware, or for the kernel to check that it is running on the right board. Unfortunately, the U-Boot that comes on the EK board identifies the board as the earlier, more expensive, and no longer available Development Kit (DK) board, which varies slightly in its hardware. Wackiness ensues. The Linux kernel, built for the EK board, panics.

Later versions of U-Boot allow you to override the board type using a U-Boot environmental variable; U-Boot 1.1.1 lacks this feature. Much later versions of U-Boot support an even sexier feature called the flat device tree, where an entire hierarchical database of hardware information is passed to the kernel. This is supported in the latest versions of the Linux kernel for the PowerPC architecture, but is not applicable to Diminuto's ARM architecture.

The right way to fix this would be to install a correctly configured version of U-Boot, or a later version that has the override feature, in the flash on the EK board. But this would require every user of Diminuto have access to a tool like a Abatron BDI 3000 JTAG debugger to reflash U-Boot. A JTAG debugger is a piece of hardware that attaches to a target processor and allows you to read and write its registers and memory, set breakpoints, and write to system devices like flash memory. The best ones, like the BDI, have gdbserver interfaces that allow them to be used with the GNU gdb debugger.

These tools are great to have around, and are nearly indispensable for production embedded system development. Which is why I have one sitting four feet away from me. But unfortunately, they also cost thousands of dollars and would place Diminuto out of the price range of many potential users. (I do recommend that if you are outfitting a lab with several Diminuto stations for teaching a course, having a single JTAG debugger to share among all the lab users is a very good investment indeed.)

So instead I patched in the Linux kernel used by Diminuto to change the board code reported by U-Boot to that of the EK board. This patch was just a few lines of ARM assembly code in the portion of the Linux kernel executed very early (like after just a handful of machine instructions) in the boot process. This is a hack for sure, but one which allows the Diminuto kernel to be used on any EK board right out of the box without requiring modifications to the board or its firmware.

(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)

There are some issues with the choice of ABI. An Application Binary Interface (ABI) establishes a standard about how an application calls into the Linux kernel, how data structures are packed in memory, how subroutines are called and receive parameters, how stack frames are organized, generally how all that stuff that application programmers take for granted but systems programmers have to worry about works.

Alas, there are two different (and incompatible) ARM ABIs, the Old (really, original) ABI (OABI) and the new, improved Embedded ABI (EABI). The more recent EABI has many advantages, particularly in its emulation of floating point, and Buildroot supports it as an option. But I was unable to get it work work reliably. (Googling suggests I am in good company.) Specifically, there appears to be an issue in the kernel signal handling code for the ARM architecture when using the EABI: like, it doesn't work in 2.6.25.10. Signaling handling in the kernel is more bizarre than I ever expected: it involves pushing machine code on to the stack of the running application, where it is later executed and popped off. This means its implementation is architecture and ABI specific.

In the end I had to build everything using the OABI and put off using the EABI for another day.

Linux wasn't happy about the relatively large initial RAM file system. When I tried to build Linux 2.6.25.10 using the relatively large initramfs, the linker was unable to resolve some of the external symbols in the kernel because they would result in branches being made to displacements farther away (around the initramfs) than was allowed by the ARM branch instruction. I rearranged slightly the memory order of the kernel components at link time to eliminate this issue.

I'm not convinced that this approach doesn't leave some RAM unavailable for use following the loading of the initial RAM file system, but the footprint of the Diminuto software is so small compared to the EK board's available memory, that I don't consider this a critical issue. I also noted in subsequent work using another tool chain, a slightly later version of the Linux kernel, and the EABI, that this issues appears to have been fixed.

(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)

Buildroot is good at building, not so good at cleaning. Buildroot can build the GNU tool chain, the Linux kernel, Busybox, and the root file system, with not much more than a make at its top level. However, doing a make clean at its top level seldom achieves the desired effect.

When working on various Linux kernel or Busybox configurations, I found it best to first do a top level make in the Buildroot directory, then iterate by descending in to the specific directories for the kernel or Busybox and doing the appropriate make there. This allows Buildroot to do all of its source code patches and and attend to all of its configuration needs, and then allowed me to tweak things as necessary with finer granularity.

(A makefile which automates much of this process is provided in the Diminuto distribution on the Diminuto web page.)

So finally we come down to booting Diminuto on an EK board right out of the box. When you attach a serial console (for me, that was an old ThinkPad laptop running putty) to the EK board and power it up, you see U-Boot come up and issue a prompt

U-Boot Initial Dialog Upon Power-Up/Reset

just like you did in the article Diminuto Right Out of the Box. If you already tried booting the 2.4 kernel and RAM disk that the EK board comes with, you have already administered the U-Boot environmental variables necessary to give the EK board an IP address and point it at your TFTP server. Here are mine; your mileage will of course vary.


setenv ipaddr 192.168.1.223
setenv serverip 192.168.1.222
setenv netmask 255.255.255.0
setenv gatewayip 192.168.1.1


To make life easier, I defined two new U-Boot environmental variables, download and start, which are macros that download the 2.6 kernel and initramfs image from the TFTP server and start the boot process. I also had to change the variable bootargs, which is the boot command line argument list passed by U-Boot to the kernel.


setenv start 'bootm 21000000'
setenv download 'tftp 21000000 diminuto-linux-2.6.25.10'
setenv bootargs 'console=ttyS0,115200 mem=32M'


I saved these U-Boot environmental variables to flash where they would persist across resets and power cycles.


saveenv


Finally, I use the macros I just defined to boot Diminuto.


run download
run start


That's all there is to it. Here's a complete log where I reset the board, ask U-Boot to display all of its environmental variables, boot up Diminuto, login into the system, and display its memory usage. (You can also find this in a separate page here.)


boot 1.0 (Aug 8 2003 - 12:29:00)



Uncompressing image...



U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
bootargs=console=ttyS0,115200 mem=32M
filesize=1ab33c
fileaddr=21000000
ipaddr=192.168.1.223
serverip=192.168.1.222
start=bootm 21000000
netmask=255.255.255.0
download=tftp 21000000 diminuto-linux-2.6.25.10
gatewayip=192.168.1.1
stdin=serial
stdout=serial
stderr=serial

Environment size: 384/8188 bytes
Uboot> run download
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'diminuto-linux-2.6.25.10'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
######################################
done
Bytes transferred = 2519904 (267360 hex)
Uboot> run start
## Booting image at 21000000 ...
Image Name: Linux-2.6.25.10
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2519840 Bytes = 2.4 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux.......................................................................................................................... done, booting the kernel.
Linux version 2.6.25.10 (jsloan@silver) (gcc version 4.2.4) #49 Tue Sep 2 14:50:42 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 28472KB available (2399K code, 274K data, 108K init)
SLUB: Genslabs=12, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 152 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NetWinder Floating Point Emulator V0.97 (extended precision)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
at91_spi: Baud rate set to 5990400
AT91 SPI driver loaded
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:43 UTC (883612843)
Freeing init memory: 108K
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex




www.diag.com Diminuto 0.0

diminuto login: root
# cat /proc/meminfo
MemTotal: 28580 kB
MemFree: 22400 kB
Buffers: 0 kB
Cached: 3956 kB
SwapCached: 0 kB
Active: 976 kB
Inactive: 3320 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 356 kB
Mapped: 388 kB
Slab: 1324 kB
SReclaimable: 296 kB
SUnreclaim: 1028 kB
PageTables: 72 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 14288 kB
Committed_AS: 868 kB
VmallocTotal: 989184 kB
VmallocUsed: 1088 kB
VmallocChunk: 986108 kB
#


You can see that of the 32MB of RAM total on the EK board, nearly 28MB is available for use, and of that, about 22MB is available for use by applications you may choose to develop for Diminuto. The system supports persistent storage in the form of both SD cards and USB drives (I've used both). The associated tool chain has the complete C++ Standard Library which includes the Standard Template Library (STL), and the complete POSIX Threads (pthreads) library.

I have ported and successfully unit tested almost all of the Desperado embedded C++ library to Diminuto, including John Sadler's Ficl embedded Forth interpreter. I did run into some weirdness (I'm tempted to say bugs, but I'm sure it will turn out to be pilot error) with code generated by the C++ compiler that I worked around, and discuss on the Diminuto web page.

I was motivated to build Diminuto by my curiosity to see if I could create a cost effective environment that could be used to teach some of the same stuff I learned decades ago using assembler code and PDP-11s, but with actual, modern, commercially available hardware, and open source software. For sure, there is a lot more than could be done to expand this project (and I'll do that as time permits), but I'm really pleased at what I've accomplished. I hope others find some inspiration in this.

In a future article, I'll show the classic "Hello, World" example, where we compile a simple program on our host PC, FTP it to the target EK board, and execute it.

Thursday, September 04, 2008

Why Free Markets Don't Work: Measurement Dysfunction as Management Strategy

Reading Robert Austin's book Measuring and Managing Performance in Organizations (Dorset House, 1996), was such an eye-opening experience for me that it has merited mention in this blog several times before this, most prominently in In Defense of Misbehavior. Once you understand measurement dysfunction, in which the performance measurement of individuals almost inevitably drives dysfunction into an organization, you will never again see the world in quite the same way.

The gist of it is this: an old axiom of business is "You cannot manage what you do not measure." But in his Ph.D. thesis, Austin, formerly an executive with Ford Motor Company and now on the faculty of the Harvard Business School, applies agency theory, a form of game theory, to show that in all but the most trivial of situations, it is impossible to measure all metrics applicable to an individual's performance. There will always be aspects of performance that cannot be measured. When rewarded (or punished) based on the metrics you do measure, the person being measured will be highly incentivized to game the system by slacking off on the those unmeasurable metrics. A classic example are call center agents hanging up on customers in the middle of calls to meet their quotas.

Austin's work has been cited by Tom DeMarco ("Mad About Measurement") in Why Does Software Cost So Much? (Dorset House, 1995), and by Joel Spolsky ("Measurement") in Joel on Software (Apress, 2004). It compelled me to get a copy of his dissertation (Robert Daniel Austin, Theories of measurement and dysfunction in organizations, Carnegie Mellon University, University Microfilm, #9522945, 1995), and even put a talk together on the topic which I've given several times. Given the recent horrifying cover story in BusinessWeek, "Managing by the Numbers: How IBM improves quality by tracking employees' every move" (September 8, 2008) , this topic is bound to become popular blog fodder once again.

But that's not what I'm here to talk about. I'm here to talk about why free markets don't work.

A few years ago, a large telecommunications equipment manufacturer decided to save money by eliminating all desktop PC support and having their product developers support their own predominantly-Windows desktop computers. This was done, of course, as a money savings move.

Did they save money? On paper sure, because they eliminated a bunch of IT staff from the bottom line. In reality, who the hell knows? All they really did was move their desktop IT support from where its cost could be measured to where it could not be measured. Most of the product line of this manufacturer was Linux-based, so the developers, who were earning hefty salaries for their Linux and telecommunications expertise, were not Windows experts. Most of their Windows support experience came from buying a home PC from BestBuy.

I suspect those developers were the most expensive IT support staff on the planet. But its all good because those costs weren't measured. Nor was the loss in productivity, the reduction in efficiency and quality in the work environment, and the moral hit to both the developers and the IT staff that came with the assumption that those two skill sets were interchangeable. I mean, really, it's just all computers, right? How hard can it be?

Sure, you could see this as a form of measurement dysfunction: some VP got a bonus by gaming the system. This is how people most often think of measurement dysfunction, as a more general term for incentive distortion. But this can also be seen in another way, not inadvertent dysfunction caused by the measurement of performance of individuals, but in a deliberate application so as to move costs to where they could not be measured. It's measurement dysfunction as management strategy.

Pollution is another form of measurement dysfunction. Companies realize cost savings by not dealing with the environmental costs of manufacturing. That's because, without government regulation, environmental costs are not borne by the polluter. The costs are pushed off onto the downstream neighbors of the manufacturer, or into the surrounding community, even onto future generations. As long as you just think locally and take a short term view, there is a strong incentive to move environmental costs to where they cannot be measured. Offshoring is just another form of this: moving environmental, or even societal, costs to where they are not measured.

In his book, Best Business Crime Writing of the Year (Random House, 2002), James Surowiecki collects magazine and newspaper articles on the spate of corporate crime that lead to the demise of Enron, WorldCom, Tyco, and other best-in-class companies. The actions of the executives described in these articles illustrate another form measurement dysfunction. Federal, state and local laws are a form of measurement. There is a strong (given the huge dollar amounts involved, I'm tempted to say irresistible) incentive to game the system by testing the limits and loopholes of those laws in an effort to derive revenue from areas for which the results of actions are not measured. I suspect that many of these executives didn't even realize they had committed a crime until the men with badges, guns, and handcuffs showed up with warrants, subpoenas, and writs. The Sarbanes-Oxley Act is an attempt to fix this dysfunction by measuring a metric that had not been measured before.

Measurement dysfunction is much more broadly applicable than just as incentive distortion. That's why it's known by other names in other contexts: unintended consequences, perverse incentives, and moral hazard are some I know about. With all due respect to my libertarian friends (and I have a lot of them), I argue that measurement dysfunction is why free markets don't work.

There will always be an economic incentive for participants in a truly free market to dump costs into or derive revenue from any gray area in which there is no measurement. And in an information economy, there are a lot of gray areas. That's because the measurement of all dimensions of market costs is impossible. Adam Smith described an "invisible hand" that guides markets to the most efficient allocation of resources. But for many metrics of market cost, there is no hand at all, invisible or otherwise. I'm not arguing for controlled markets. But I am arguing for regulated markets: that a significant (and perhaps unfortunate) amount of government regulation is necessary to act as a visible hand, forcing market participants to bear the full costs of their transactions.

In the realm of performance measurement of individuals, Austin argues against extrinsic motivation, for example bonuses tied to specific goals, because of the measurement dysfunction this produces. Instead, intrinsic motivation, for example corporate culture, causes individuals to do the right thing independent of incentives. If intrinsic motivators worked at the corporate level, free markets could work as well. But it's been argued that corporations are psychopathic because, devoid of any other legal restraint, their sole purpose is to maximize shareholder value.

I'm not saying that corporations are evil. I'm the founder and CEO (and janitor) of a corporation myself. But I am saying that it would be foolish not to acknowledge the measurement dysfunction that occurs in a truly free market.

Tuesday, September 02, 2008

Choosing Software for Diminuto

Diminuto is my attempt to put together a platform on which to teach real-time software design, systems programming, embedded software development and assembly language programming, all using real commercial hardware and open source software. In Diminuto Right Out of the Box, I described how to get the Atmel AT91RM9200-EK evaluation kit (EK) running with the software it ships with, which includes a Linux version 2.4 kernel and a tool chain based on the GNU Compiler Collection version 2.95.3. In this this article, I'll describe the software I used to build a reduced-memory footprint system using the Linux version 2.6 kernel and a tool chain based on GCC 4.2.4.

My career as a technologist has been pretty much all over the map. I've always thought of that as a good thing, although I've been accused by my betters of being everything from "a renaissance man" to "unfocused". My career seems to shuttle back and forth between developing for high-end server-side systems and doing real-time and embedded development. Consequently, the systems for which I've developed have ranged from processors with eight kilobytes of memory to distributed networks of servers to supercomputers.

Even among embedded systems, the range has been kind of startling. In the 1970s I wrote standalone assembler code for PDP-11s with 8KB of core memory. Core memory was kind of neat because it maintained its state across power cycles. We used to patch our code by toggling in new machine instructions from the switch register on the front panel. More recently, and more typically, I've found myself writing mostly in C, C++ and occasionally even Java, for PowerPC, i960, or ARM microprocessors running some commercial real-time operating system (RTOS) like VxWorks, pSOS, C-Executive, or RTX which run completely memory resident. Meanwhile, the server-side portion of my career evolved from using just about every UNIX variant known to man to being 100% Linux based. Probably no surprise there.

With the availability of cheaper random access memory (RAM), and relatively inexpensive microprocessors that have a memory management unit (MMU), perhaps it was inevitable that these two career paths merge with the advent of embedded Linux. But even among embedded Linux systems there is a broad range of configurations, ranging from systems that still run completely memory resident and which are reminiscent of those VxWorks systems, to those that have persistent storage in the form of solid-state "disks" such that they seem more like a Linux server.

Whether you are accustomed to developing for Linux on a PC, or you are an embedded developer used to VxWorks, embedded Linux will seem like a weird combination of the familiar and the bizarre, not quite fitting in with either world. The kernel seems familiar to the PC developer, but there may be no persistent storage and hence no paging of virtual memory, and you may do everything logged in as root while the system runs in single-user mode. The lack of persistent storage seems natural to the embedded developer, but the system has an MMU which maps between physical and virtual memory addresses, the processor executes code in privileged and non-privileged mode, it insists on having a root file system even though it may be completely memory resident in something called a RAM disk, and there is an unusual wealth of sophisticated applications and tools that can run on the system. The need to compile and link on a host system (like a conventional Linux PC with an Intel processor and a cross-compilation tool chain) but run on a completely different target system (like the AT91RM9200-EK board with an ARM processor and no tool chain at all) freaks out the PC developer but is strictly in the comfort zone of the embedded developer.

Many embedded systems have pretty severe resource constraints when compared to your typical PC. The EK board has 32 megabytes of RAM and a processor delivering a claimed 200 million instructions per second (MIPS). The Dell 530 quad-core server in the basement (a.k.a. the vast Digital Aggregates corporate data center) on which I run the Diminuto tool chain has three gigabytes of RAM and its speed is several billion instructions per second (GIPS). My Nokia N810 internet tablet, more typical of hand-held consumer devices, has 128 megabytes of RAM, just four times that of the EK, with maybe a 400 MIPS processor. All of these are Linux systems, but clearly they are not quite the same Linux system.

Living within your means, RAM-wise, isn't just an issue for the kernel and any programs that are running at any one time. Although Linux requires a root file system, I've mentioned that many embedded devices have no persistent storage device. The kernel, any running programs, and the root file system in some kind of RAM disk, all share the available RAM. The kernel and the RAM disk are loaded into RAM at boot time, either from some read-only memory (ROM) or across a network using something like the Trivial File Transfer Protocol (TFTP). Reducing the memory footprint of the RAM disk turns out to be important too.

Or maybe not. Ridiculously large surface mounted flash memory devices with integrated controllers that emulate IDE disk drives are becoming more available, driven in part by Windows CE hand-held devices. The multi-gigabyte single chip disk drives that I have used come from the factory pre-formatted for the Windows FAT file system. Linux itself includes drivers for devices like Secure Digital (SD) flash cards and USB flash drives which emulate SCSI disk drives.

Persistent storage devices are becoming more common in all but the most cost sensitive or tiniest embedded systems. However, such solid state persistent storage devices tend to be very slow when compared to disk drives. Even when they are available, there is a tendency to keep as much as possible of the working set of "disk" blocks from these devices cached in memory.

It's easy to argue that the trend will be towards more and more RAM in embedded devices, just as there has been for servers, desktops, and laptops. But the trend may be in just the opposite direction for many embedded applications. Microcontrollers (which is what microprocessors are called when you don't know that they're there) are being used in physically smaller, and more price sensitive, devices. There is a strong competitive pressure to shave the manufacturing cost of a device that will be sold in the millions. Sometimes this means less RAM and eliminating a flash drive. Pennies count in such applications.

I've built both types of embedded Linux systems: those that had a lot of RAM and a large flash-based disk drive, and those that booted from ROM and ran completely in RAM. The former wasn't substantially different from the user's point of view from a PC-based system: multi-user, ran the SSH daemon, even ran an Apache web server and a Java SE JVM, and your files were still there in your home directory when the system was rebooted. The latter: single user mode, could only be accessed from a serial port, and every press of the reset button was a new day RAM disk-wise. It's easy to get spoiled with the former. I learned more from the latter. Diminuto marches right down the middle: the basic system runs completely resident in RAM, but supports the use of persistent file systems on SD cards and USB drives.

Here are some approaches to reducing the resource footprint of a Linux system.

Reduce the Size of the Kernel

You download the latest Linux kernel from www.kernel.org: all eight million lines of C code in twenty thousand source files. In the case of Diminuto, which uses an ARM-based processor, you run the command

make menuconfig ARCH=arm

and are delighted to discover that you can select the options AT91 and AT91RM9200-EK to get support for the devices in your system-on-a-chip. Then you build the kernel and discover to your horror that the resulting bootable image is enormous. True, the Linux kernel has direct support for the AT91RM9200-EK board. But if you just select that option, you will find that by default the configuration process has selected tons of other options and drivers you could care less about, including a lot of stuff you don't even recognize, and also left out a bunch of stuff you know you're going to need.

To generate exactly the kernel you need without blowing your memory budget, you must laboriously go through all the dozens of nested configuration menus, figure out whether you really need each item, and then either enable or disable it. This includes the seemingly hundreds of device drivers, including the ones that can be built as loadable modules (more on that later). This may take several iterations, some research, and some hard decisions ("Hmmmm, do I really need IPV6?"), but it's worth it. In the end you'll get what you need with a memory footprint you can live with. If you forget something, you can always generate another kernel and try it.

During this process you will be tempted to build every device driver as a loadable module. You will do this because this is exactly what you do for your PC-based Linux system. Loadable kernel modules and device drivers are not linked directly into the kernel image, but instead are loaded dynamically on demand from the file system. Building loadable modules for your embedded system doesn't actually hurt anything, as long as you realize it's mostly a waste of time. Devices other than what are on your board aren't likely to magically appear; the EK doesn't have a back plane into which you can easily plug in other boards. And you have pretty much a 100% chance of using the devices that are implemented on your board. Furthermore, unused loadable modules in a RAM disk file system take up RAM whether you are using them or not. My advice is to build all the kernel options and device drivers you know you are going to use directly into the kernel, and explicitly omit everything else.

There is an important exception to this rule: if you are writing your own device drivers for your board, you may want to use loadable modules for reasons of intellectual property and licensing. Conventional wisdom is that loadable modules are not contaminated by the GNU Public License (GPL), whereas modules linked directly into the kernel are. I'm not a lawyer, and I'm not aware of any legal precedent having been set for this, but when I've written device drivers for my clients for closed, proprietary, custom hardware, I've built them as loadable modules.

In Diminuto, I built version 2.6.25.10 of the Linux kernel using the approach I just described, including all supported devices, directly into the kernel, and omitting everything else.

There are also alternatives to the conventional Linux kernel. uCLinux is a version of Linux that does not require an MMU. I've never used it. But it does admit the possibility of running Linux on microcontrollers that are in the traditional sphere of RTOSes like VxWorks. uCLinux has been ported to many of the embedded microprocessors on which I've worked over the years. I think it's worth examining if that's the world in which you are working.

Reduce the Size of Binary Executables

A statically linked binary executable is a program which incorporates all of its code in a single monolithic image. If several programs are statically linked against the same library, they each contain unique copies of the same functions and subroutines that they reference in that library. Everyone gets their own copy of printf. There are sometimes reasons to do this (it may facilitate debugging), but it makes every binary executable a lot larger than it otherwise needs to be.

A shared library in Linux is like a Dynamic Link Library (DLL) in Windows (or so I'm told). It is a way for binary executables to share common code. One copy of the shared library is loaded into memory. When a binary executable which was originally dynamically linked against that shared library is run, the program loader fills in the references to the shared functions and subroutines inside the executable so that they point into the shared library. The MMU is used to create the illusion that everyone is using their own copy of the library, even though everyone is sharing the same code.

This is a good thing, generally, whether you have an embedded system with 32MB of RAM or a PC with 3GB of RAM. But it does take some care to manage the installation and maintenance of the shared library. Woe be to the person who runs a binary executable that was dynamically linked on the host system against a different version of the shared library from that which is resident on the target system.

It also requires some care in the construction of the shared library itself. A shared library that incorporates a bunch of functions and subroutines that aren't used by any binary executable is still wasting RAM. Shared libraries are only a win if the code in them is actually shared. This is why I chose to build Diminuto using uClibc, a version of the Standard C library with a reduced memory footprint. uClibc gets its reduction through a variety of means: by omitting some features not needed in most embedded applications, by reducing the capacity of some retained features, and by some code re-factoring. I used version 0.9.29 of uClibc for Diminuto.

I have also built embedded Linux systems using the full blown GNU standard C library, so your mileage may vary. If you are running completely memory resident, I recommend you at least look at uClibc to see if it meets your needs. If you have the extravagant pleasure of a "disk" on your target, you may consider using the GNU library. On my To Do list for Diminuto is to build a root file system using the full GNU library that boots from an EXT3 file system on the Secure Digital (SD) flash card supported by the EK. I built support for EXT3 and the SD driver into the kernel, so that same kernel should work for either system. I'd like to eventually run the ARM-version of Sun's Standard Edition (SE) of their Java Virtual Machine (JVM) using this platform. (I've done this in the past on embedded PowerPC platforms with zero issues.) I routinely use EXT3 file systems on USB drives for persistent storage with the current RAM-resident Diminuto system.

One thing I did not scrimp on was the C++ standard library and the POSIX Thread library. I knew I was going to port my embedded, real-time C++ toolkit, Desperado, to Diminuto, and I wanted unfettered access to the complete Standard Template Library (STL) and to pthreads. Recently, however, a reduced footprint version of the C++ standard library, uClibc++, has been developed that is worth a look. I haven't used it.

I've had no library-compatibility issues with porting either Desperado or my C-based Diminuto library to the AT91RM9200-EK and running their unit tests. I have had some weirdness with the GNU arm-linux-g++ compiler itself which I discuss on the Diminuto web page.

Reduce the Size of the Root File System

Much of the Linux root file system is taken up with scripts and utilities to manage its multi-user environment and to provide a rich set of tools for those multi-users. The entire huge System V init script infrastructure exists mainly to take a multi-user system up and down in an orderly manner. The typical embedded system, however, runs a very limited set of applications, may have no traditional user interface at all except for during initial development, all processes may run as root, and it is brought up and down strictly with the power switch. This requires a much simpler infrastructure that eliminates a lot of the stuff usually found in the root file system.

This is also one of the reasons I recommend using an EXT3 journaled file system if you have persistent storage. EXT3 is bit-compatible with EXT2, a staple of Linux systems for a long time, with the addition of a journal file. In fact, an EXT3 file system can be mounted as an EXT2 file system, in which case the system ignores the journal file. EXT3 is a robust approach for an embedded system where someone may hit the reset button at any time. I built Diminuto with support for EXT2, EXT3, and VFAT (a.k.a. Windows FAT-32, especially useful for USB drives that are traded between Windows and Linux systems).

PC developers will be horrified to know that most work is done on embedded Linux systems while the developer is logged in as root, but this will seem natural to the embedded developer. This is because nearly all of the work that needs to be done on the target system has to be done as root, while all the development work that needs to be done on the host system is done logged in as a normal unprivileged user. Also, a scrogged target system can typically be recovered by just hitting the reset button, with the resulting loss of everything you may have had on the RAM disk. Linux offers the ability to segregate processes and files on the target system by user ID, which does provide a great deal more protection. I recommend thinking about this if non-developers have access to the innards of your embedded Linux system, even if they don't have login access. For example, you will surely want to run any web server under a non-root account.

BusyBox is a very clever way of reducing both the RAM and disk footprints of a Linux system without much impact on functionality. BusyBox is a single monolithic program that implements the functionality of dozens of commonly used Linux commands. It achieves this significant resource reduction, even when using shared libraries, by yet more sharing of common code, excluding seldom used options, and by being less scalable than the equivalent full featured GNU utilities. BusyBox is a multi-call binary: if it is invoked through a soft link called ls, it behaves like the ls command.

I built Diminuto using version 1.11.1 of BusyBox. A quick tour through all of the bin directories on my PATH reveals that my root file system on Diminuto has all of five binary executables:

/sbin/ldconfig
/usr/bin/ldd
/usr/bin/gdbserver
/usr/local/bin/getubenv, and
/bin/busybox.

The dozens of other commands are just soft links to /bin/busybox. (And the first three of those binaries are just to expedite debugging.)

The use of BusyBox is a huge win, even if you can't use all of its capabilities. I am currently using the Bourne-like shell provided by BusyBox. If your embedded system depends heavily on complex shell scripts, and if resources permit, consider loading your shell of choice (e.g. the Bourne Shell) on your embedded system. Scripting is a huge win productivity-wise over implementing the same functionality in C code, and the additional resources required by a full blown shell interpreter can be well worth it.

In future articles I'll discuss how I built the root file system for Diminuto, describe the adventure that was getting the 2.6 Linux kernel running on the AT91RM9200-EK board, and give you a tour of the running system.