Diminuto consists of the Linux 2.6.25.10 kernel, the uClibc 0.9.29 C library and its associated tool chain, and BusyBox 1.11.1. But it isn't enough to just download all this stuff off the web and build it. For one thing, the use of the reduced memory footprint uClibc library requires some patches be applied here and there in both the bootable system and its tool chain. For another, you are still faced with the need to build a root file system that can be incorporated as an initial RAM file system (initramfs) within the kernel image.
Because this work is pretty common to anyone building an embedded Linux system using the software I chose, the uClibc folks provide another tool, Buildroot, to assist with it. Buildroot is a collection of configuration utilities, scripts, and makefiles that automate much of the process of downloading, patching, and building both the tool chain that runs on the host (for example, your desktop Linux PC) and the software that boots on the target (in our case, the EK board), and creating the root file system. Some of what Buildroot does is necessary because I chose to use the uClibc C library. Some of what it does would be necessary no matter what tool chain or C library I chose to use.
As you might expect, there were some issues I had to resolve to get Buildroot and the artifacts that it creates to work on the EK board.
Buildroot is a moving target. Even more so than most other open source software, Buildroot is a moving target. It does not appear to have any release management. When you download Buildroot, you do so directly from its Subversion repository. While you may easily choose any revision you want, there is no guarantee that any particular revision will work, or even build, for the particular choice of hardware, software, and options that you want to use.
There are a lot of folks contributing to Buildroot, so the repository is under constant churn. I went through several iterations trying to get various revisions of Buildroot to work with Diminuto before settling on Buildroot 22987. The most current revision as I write this is 23454; that means there have been 467 changes since I got my copy of 22987. Your mileage will almost certainly if you choose to try using a different version of Buildroot than I did.
U-Boot on the EK board isn't quite right. U-Boot is a bootloader like GRUB, and LILO. It lives on the hardware in some persistent memory like ROM or flash. It does some basic hardware initialization necessary for booting the Linux kernel. It has a console command interface and some basic scripting capability. It can boot a Linux kernel image that resides in ROM or flash. It can also get a bootable Linux image from a server across the network using the Trivial File Transfer Protocol (TFTP). Later versions of U-Boot that I have used even understand how to read EXT2 file systems on IDE devices so that Linux can be booted from a "disk" (which as I mentioned in the prior article, may actually be a flash device). I've built embedded Linux systems to boot from U-Boot using all three of these methods: from flash, using TFTP (which is what Diminuto uses), and from a solid-state IDE "disk".
U-Boot is very popular in the embedded community, while GRUB and LILO are mostly used on PCs. The EK board comes with U-Boot 1.1.1 pre-installed in flash. This places the EK user way ahead of just having "bare metal".
However, the U-Boot that comes on the EK isn't perfect. One of the things that U-Boot does upon transferring control to the Linux kernel is pass a board type code (really just a magic integer) to the kernel, telling the kernel what kind of board it is running on. This admits the possibility of building a single kernel to run on several variations of hardware, or for the kernel to check that it is running on the right board. Unfortunately, the U-Boot that comes on the EK board identifies the board as the earlier, more expensive, and no longer available Development Kit (DK) board, which varies slightly in its hardware. Wackiness ensues. The Linux kernel, built for the EK board, panics.
Later versions of U-Boot allow you to override the board type using a U-Boot environmental variable; U-Boot 1.1.1 lacks this feature. Much later versions of U-Boot support an even sexier feature called the flat device tree, where an entire hierarchical database of hardware information is passed to the kernel. This is supported in the latest versions of the Linux kernel for the PowerPC architecture, but is not applicable to Diminuto's ARM architecture.
The right way to fix this would be to install a correctly configured version of U-Boot, or a later version that has the override feature, in the flash on the EK board. But this would require every user of Diminuto have access to a tool like a Abatron BDI 3000 JTAG debugger to reflash U-Boot. A JTAG debugger is a piece of hardware that attaches to a target processor and allows you to read and write its registers and memory, set breakpoints, and write to system devices like flash memory. The best ones, like the BDI, have gdbserver interfaces that allow them to be used with the GNU gdb debugger.
These tools are great to have around, and are nearly indispensable for production embedded system development. Which is why I have one sitting four feet away from me. But unfortunately, they also cost thousands of dollars and would place Diminuto out of the price range of many potential users. (I do recommend that if you are outfitting a lab with several Diminuto stations for teaching a course, having a single JTAG debugger to share among all the lab users is a very good investment indeed.)
So instead I patched in the Linux kernel used by Diminuto to change the board code reported by U-Boot to that of the EK board. This patch was just a few lines of ARM assembly code in the portion of the Linux kernel executed very early (like after just a handful of machine instructions) in the boot process. This is a hack for sure, but one which allows the Diminuto kernel to be used on any EK board right out of the box without requiring modifications to the board or its firmware.
(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)
There are some issues with the choice of ABI. An Application Binary Interface (ABI) establishes a standard about how an application calls into the Linux kernel, how data structures are packed in memory, how subroutines are called and receive parameters, how stack frames are organized, generally how all that stuff that application programmers take for granted but systems programmers have to worry about works.
Alas, there are two different (and incompatible) ARM ABIs, the Old (really, original) ABI (OABI) and the new, improved Embedded ABI (EABI). The more recent EABI has many advantages, particularly in its emulation of floating point, and Buildroot supports it as an option. But I was unable to get it work work reliably. (Googling suggests I am in good company.) Specifically, there appears to be an issue in the kernel signal handling code for the ARM architecture when using the EABI: like, it doesn't work in 2.6.25.10. Signaling handling in the kernel is more bizarre than I ever expected: it involves pushing machine code on to the stack of the running application, where it is later executed and popped off. This means its implementation is architecture and ABI specific.
In the end I had to build everything using the OABI and put off using the EABI for another day.
Linux wasn't happy about the relatively large initial RAM file system. When I tried to build Linux 2.6.25.10 using the relatively large initramfs, the linker was unable to resolve some of the external symbols in the kernel because they would result in branches being made to displacements farther away (around the initramfs) than was allowed by the ARM branch instruction. I rearranged slightly the memory order of the kernel components at link time to eliminate this issue.
I'm not convinced that this approach doesn't leave some RAM unavailable for use following the loading of the initial RAM file system, but the footprint of the Diminuto software is so small compared to the EK board's available memory, that I don't consider this a critical issue. I also noted in subsequent work using another tool chain, a slightly later version of the Linux kernel, and the EABI, that this issues appears to have been fixed.
(This patch to Linux 2.6.25.10 is provided in the Diminuto distribution on the Diminuto web page.)
Buildroot is good at building, not so good at cleaning. Buildroot can build the GNU tool chain, the Linux kernel, Busybox, and the root file system, with not much more than a make at its top level. However, doing a make clean at its top level seldom achieves the desired effect.
When working on various Linux kernel or Busybox configurations, I found it best to first do a top level make in the Buildroot directory, then iterate by descending in to the specific directories for the kernel or Busybox and doing the appropriate make there. This allows Buildroot to do all of its source code patches and and attend to all of its configuration needs, and then allowed me to tweak things as necessary with finer granularity.
(A makefile which automates much of this process is provided in the Diminuto distribution on the Diminuto web page.)
So finally we come down to booting Diminuto on an EK board right out of the box. When you attach a serial console (for me, that was an old ThinkPad laptop running putty) to the EK board and power it up, you see U-Boot come up and issue a prompt
just like you did in the article Diminuto Right Out of the Box. If you already tried booting the 2.4 kernel and RAM disk that the EK board comes with, you have already administered the U-Boot environmental variables necessary to give the EK board an IP address and point it at your TFTP server. Here are mine; your mileage will of course vary.
setenv ipaddr 192.168.1.223
setenv serverip 192.168.1.222
setenv netmask 255.255.255.0
setenv gatewayip 192.168.1.1
To make life easier, I defined two new U-Boot environmental variables, download and start, which are macros that download the 2.6 kernel and initramfs image from the TFTP server and start the boot process. I also had to change the variable bootargs, which is the boot command line argument list passed by U-Boot to the kernel.
setenv start 'bootm 21000000'
setenv download 'tftp 21000000 diminuto-linux-2.6.25.10'
setenv bootargs 'console=ttyS0,115200 mem=32M'
I saved these U-Boot environmental variables to flash where they would persist across resets and power cycles.
saveenv
Finally, I use the macros I just defined to boot Diminuto.
run download
run start
That's all there is to it. Here's a complete log where I reset the board, ask U-Boot to display all of its environmental variables, boot up Diminuto, login into the system, and display its memory usage. (You can also find this in a separate page here.)
boot 1.0 (Aug 8 2003 - 12:29:00)
Uncompressing image...
U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)
U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
bootargs=console=ttyS0,115200 mem=32M
filesize=1ab33c
fileaddr=21000000
ipaddr=192.168.1.223
serverip=192.168.1.222
start=bootm 21000000
netmask=255.255.255.0
download=tftp 21000000 diminuto-linux-2.6.25.10
gatewayip=192.168.1.1
stdin=serial
stdout=serial
stderr=serial
Environment size: 384/8188 bytes
Uboot> run download
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'diminuto-linux-2.6.25.10'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
######################################
done
Bytes transferred = 2519904 (267360 hex)
Uboot> run start
## Booting image at 21000000 ...
Image Name: Linux-2.6.25.10
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2519840 Bytes = 2.4 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK
Starting kernel ...
Uncompressing Linux.......................................................................................................................... done, booting the kernel.
Linux version 2.6.25.10 (jsloan@silver) (gcc version 4.2.4) #49 Tue Sep 2 14:50:42 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 28472KB available (2399K code, 274K data, 108K init)
SLUB: Genslabs=12, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 152 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NetWinder Floating Point Emulator V0.97 (extended precision)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
at91_spi: Baud rate set to 5990400
AT91 SPI driver loaded
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:43 UTC (883612843)
Freeing init memory: 108K
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex
www.diag.com Diminuto 0.0
diminuto login: root
# cat /proc/meminfo
MemTotal: 28580 kB
MemFree: 22400 kB
Buffers: 0 kB
Cached: 3956 kB
SwapCached: 0 kB
Active: 976 kB
Inactive: 3320 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 356 kB
Mapped: 388 kB
Slab: 1324 kB
SReclaimable: 296 kB
SUnreclaim: 1028 kB
PageTables: 72 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 14288 kB
Committed_AS: 868 kB
VmallocTotal: 989184 kB
VmallocUsed: 1088 kB
VmallocChunk: 986108 kB
#
You can see that of the 32MB of RAM total on the EK board, nearly 28MB is available for use, and of that, about 22MB is available for use by applications you may choose to develop for Diminuto. The system supports persistent storage in the form of both SD cards and USB drives (I've used both). The associated tool chain has the complete C++ Standard Library which includes the Standard Template Library (STL), and the complete POSIX Threads (pthreads) library.
I have ported and successfully unit tested almost all of the Desperado embedded C++ library to Diminuto, including John Sadler's Ficl embedded Forth interpreter. I did run into some weirdness (I'm tempted to say bugs, but I'm sure it will turn out to be pilot error) with code generated by the C++ compiler that I worked around, and discuss on the Diminuto web page.
I was motivated to build Diminuto by my curiosity to see if I could create a cost effective environment that could be used to teach some of the same stuff I learned decades ago using assembler code and PDP-11s, but with actual, modern, commercially available hardware, and open source software. For sure, there is a lot more than could be done to expand this project (and I'll do that as time permits), but I'm really pleased at what I've accomplished. I hope others find some inspiration in this.
In a future article, I'll show the classic "Hello, World" example, where we compile a simple program on our host PC, FTP it to the target EK board, and execute it.
No comments:
Post a Comment