Monday, November 10, 2008

Practical Concurrency II

Another great source of information on practical, real-world concurrency is the "Effective Concurrency" column that Herb Sutter is writing for Dr. Dobb's Journal. I've been reading them in the print issues, but you can find them by searching for "Herb Sutter" on the web site. The latest issue which just arrived today at the Palatial Overclock Estate (a.k.a. the Heavily-Armed Overclock Compound) is on "Understanding Parallel Performance".

Sunday, November 09, 2008

Practical Concurrency

Bryan Cantrill and Jeff Bonwick, both of Sun Microsystems, have written "Real-world Concurrency", published online in the September edition (6.5) of ACM Queue. It's a great set of tips, tricks, and lessons learned for those of us that have to deal with multi-threaded code. I've already passed it along to several of my colleagues working in C, C++ or Java on embedded or multi-core platforms. Queue is the Association for Computing Machinery's online journal targeted at near-term real-world computing issues.

Saturday, November 08, 2008

Negative Results Are Still Results

That's a remark made by my mentor Bob Dixon when he was my thesis advisor circa 1984. It's one of those pieces of wisdom that we accumulate and carry around in our brains for the rest of our lives. It had reason to percolate to the surface as I read Why Current Publication Practices May Distort Science by N. Young et al., published in the Medicine blog of the Public Library of Science. Young and his coauthors discuss the effects of "The Winner's Curse" on how scientific results are (or are not) published.

When an item is auctioned, potential buyers make offers, or bids, on the item. Although the dollar amounts of the bids may not be normally distributed, for sure they are spread out. Some bidders bid low, some bid high, depending on how each perceives the value of the item being auctioned. What is the actual market value of the item? One definition might be the average of all the bids. But who actually gets the item being auctioned? The highest bidder. That means, by this definition of market value, the winner of the auction always pays above market value for the item. That's "The Winner's Curse".

Of course, it's not that simple. The winner may be bidding higher because they believe they have special knowledge or capabilities that will let them leverage the item to greater value than the other bidders. Or they may be desperate. Or just foolish. (I personally have fallen into that category at one art auction, and ended up $250 poorer because of it.) But in general, auctions are good for the seller, maybe not so good for the buyer.

Young et al. apply this to scientific publications. Research with the most spectacular results tend to be what gets published. And, broadly speaking, research with positive results has a much better chance of getting published than research with negative results. Researchers whose projects yield positive results win the auction for space in refereed journals.

This gives a false perception of the state of a particular area of research, since projects that yield negative results are not published and hence are not part of the "average" perceived by those who read the scientific journals. In a "publish or perish" academic climate, there is a strong motivation to produce only positive results, to quickly terminate research that yields negative results, or even to phrase results to make them appear positive. Surely negative results are just as valuable, since they can prune the tree of possible avenues of inquiry for other researchers. But such research is seldom published.

Long time readers are just waiting for me to say this: this is yet another example of measurement dysfunction.

I've been accused by my friend and occasional colleague Rodney Black of having an academic bent. (He sincerely meant it as a compliment, and I don't deny it.) So I appreciate this issue that is perhaps of interest mostly to those who spend most of their time on the R end of the R&D spectrum. (Although since this affects much medical research, including the clinical trials done by pharmaceutical companies, it probably should be of great concern to all of us.)

But there is an equally strong bias at the D end of the spectrum too. Technology efforts that crash and burn seldom get discussed, seldom get written up, are usually quietly buried and the participants sent to the career equivalent of Guantanamo Bay, or maybe Gulag Archipelago. But like those research projects yielding negative results, these failed efforts would be valuable as object lessons, cautionary tales, and as port mortems on what to do differently next time.

This too leads to a false sense of the "average" of the state of the art. It causes artificially inflated expectations, frequently on the part of upper management. All they see in the in-flight magazine are the success stories. "History is written by the victors", as Winston Churchill once said.

I'm as guilty of this as any one else. I've had a few spectacular failures in my career. But you'd have to ply me with more than a few gin and tonics to get me to talk about them. (I encourage you to try.) That may be human nature, but it's wrong.

Two-time Nobel Prize winner Linus Pauling once said "The best way to have a good idea is to have lots of ideas". Meaning, most of your ideas aren't going to work out. If you have "a track record of success", it's only because you've done a very good job at hiding your failures.

Not only is failure an option, it's downright necessary.


(Update 2009-02-18)

I was having breakfast the other day with two of my oldest friends, Brian and Paul, who are the closest thing to homies I could have when I was in college in the 1970s. We were discussing our career ups and downs. I was a little surprised to hear someone say that the few real disasters I had had in my career were in hindsight the best things that had ever happened to me. I was even more surprised to realize it was me saying it. But it's true. Funny how life works out.

(Update 2012-12-21)

I eventually did write about the most spectacular failure of my career, and how it was simultaneously the my biggest stroke of luck, in Dead Man Walking.

Sunday, November 02, 2008

Updating Arroyo

In From Diminuto to Arroyo I described Arroyo, another version of my Linux-based project to teach embedded software design and assembly language programming on a commercially available evaluation board, the ARM-based AT91RM9200-EK. My earlier project, Diminuto, was completely memory resident. It is a good example of smaller "diskless" embedded Linux systems. Arroyo uses an Secure Digital (SD) card as its root file system "disk", and can similarly use USB drives too. It is a good example of larger "diskful" embedded Linux systems.

Understand that Diminuto thinks it has a disk, when all it has is RAM. Arroyo thinks it has a disk, but is actually using flash-based devices that conspire with their device drivers to emulate IDE and SCSI disks. There's not a traditional spinning disk (or any other moving part) in either system. Diminuto still has all the drivers and utilities needed to mount and use SD cards and USB drives, it just doesn't need them to run.

Arroyo supports a variety of file systems like EXT2 and EXT3 (also popular in PC-based Linux systems) and VFAT (or FAT-32, popular in Windows systems). Most SD and USB storage devices come right out of the package pre-formatted for VFAT. Although VFAT doesn't support some traditional UNIX features like soft links, they still can be used directly without reformatting or reinitialization on Diminuto or Arroyo. I like the EXT3 file system in particular, because like EXT2, EXT3 supports all the standard UNIX features. But unlike EXT2, it is a journaled file system. This makes is extremely robust, especially in the face of the kind of treatment that consumer devices have to expect, like being powered off in the middle of operations.

The latest version of the Arroyo root file system image includes a full set of EXT2 and EXT3 file system maintenance utilities that allows you to fix and update your "disk-based" Arroyo file systems without resorting to using the same tools off-line on a server (which is how I originally built the first Arroyo root file system on an SD card). It also includes a tool that allows you to examine, initialize, modify partitions on your SD and USB storage devices. In this article I'll show you how to download a new Arroyo file system image from a server and install it using nothing but tools on Arroyo itself.

  • We will begin by using an SD card as our original root file system,
  • create an EXT3 file system on a USB drive,
  • download the new root file system image (which is just a file) from a server to that USB drive,
  • install the root file system image on a second USB drive,
  • reboot the system and temporarily use the USB drive as our new root file system,
  • install the same root file system image on our original SD card, and
  • reboot the system to use the new root file system image back on our original SD card.

Notice at no point do I actually boot the Linux operating system off an SD card or USB drive. Both Arroyo and Diminuto still depend on booting the Linux kernel from a TFTP server, and Diminuto's root file system image is embedded inside its kernel image. We use a TFTP server because the version of the U-Boot boot loader on the AT91RM9200-EK board doesn't have the capability of booting from a storage device. Later versions of U-Boot have this capability, and I have used it to good effect on other projects. That would eliminate the need for a TFTP server for Arroyo. But updating to a new version of U-Boot requires some expensive hardware, like a flash programmer, and is a story for another time.

(Blogger wraps or even truncates long lines of preformatted text. If you want to see a complete verbatim log of this process, including all of my typos and other mistakes, you can find one here.)

Begin by using an SD card as our original root file system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial
Environment size: 459/8188 bytes
Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
################################################################# ################################################################# #######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start
## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:01:08 UTC (883612868)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
kjournald starting. Commit interval 5 seconds
EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ifup: interface lo already configured
ifup: interface eth0 already configured

www.diag.com Arroyo 0.0

arroyo login: root
#

Create an EXT3 file system on a USB drive.

Insert a USB drive into one of the two host USB ports on the EK board. Note that it the kernel hotplug facility detects it automatically after a few seconds and gives it the name "/dev/sda1". That's short for "SCSI Disk #A Partition #1", because the USB storage device driver in Linux emulates a SCSI device.

usb 1-2: new full speed USB device using at91_ohci and address 2
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 0:0:0:0: [sda] Attached SCSI removable disk

Make an EXT2 file system on the USB drive.

# mke2fs /dev/sda1
mke2fs 1.41.2 (02-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
121920 inodes, 487184 blocks
24359 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=499122176
15 block groups
32768 blocks per group, 32768 fragments per group
8128 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Create an EXT3 journal file on the USB drive. This turns it into an EXT3 file system. Other than the journal file (which is a invisible file by virtue of having no directory entry), EXT2 and EXT3 file systems are bit for bit compatible. You can mount an EXT3 file system as an EXT2 file system (but then the journal won't be used).

# tune2fs -j /dev/sda1
tune2fs 1.41.2 (02-Oct-2008)
Creating journal inode: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Mount the newly minted EXT3 file system so that we can use it.

# mount /dev/sda1 /mnt
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Download the new root file system image from a server to the first USB drive.

# cd /mnt
# ftpget -u coverclock -p password 192.168.1.222 arroyo-root.ext2 /var/tmp/arroyo-root.ext2

Install the root file system image on a second USB drive.

Insert a second USB drive. This one gets called "/dev/sdb1" when hotplug detects it.

usb 1-1: new full speed USB device using at91_ohci and address 3
usb 1-1: configuration #1 chosen from 1 choice
scsi1 : SCSI emulation for USB Mass Storage devices
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 1:0:0:0: [sdb] Attached SCSI removable disk

Copy the root file system image from the file we just downloaded onto the first USB drive to the second USB drive, using the second drive as a raw block device instead of as a file system.

# dd if=arroyo-root.ext2 of=/dev/sdb1 bs=1M
127+1 records in
127+1 records out

Run a file system check on the second USB drive, which now appears to have an EXT2 file system on it. It will be a small file system, about over a hundred megabytes, because that is the size of the root file system image that we downloaded.

# e2fsck -f /dev/sdb1
e2fsck 1.41.2 (02-Oct-2008)
Filesystem did not have a UUID; generating one.

Superblock last write time is in the future. Fix? no

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 4438/7168 files (0.2% non-contiguous), 111875/130700 blocks

Resize the file system on the second USB drive so that it consumes the entire USB drive which is about two gigabytes.

# resize2fs /dev/sdb1
resize2fs 1.41.2 (02-Oct-2008)
Resizing the filesystem on /dev/sdb1 to 1948736 (1k) blocks.
The filesystem on /dev/sdb1 is now 1948736 blocks long.

Reboot the system and temporarily use the USB drive as our new root file system.

This is an important step, and it doesn't show up in this log: remove the first USB drive, the one to which we FTPed the root file system image. Leave the second USB drive containing the new root file system. It doesn't matter which USB host port it's in. I'll explain why below.

# sync
# sync
# sync
# reboot
The system is going down NOW!
Sending SIGTERM to all processes
Sending SIGKILL to all processes
Requesting system reboot
Restarting system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)

U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv

baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial

Environment size: 459/8188 bytes

Change the bootargs variable in the U-Boot bootloader so that it boots from "/dev/sda1". Why "/dev/sda1" and not "/dev/sdb1"? The naming of the USB storage devices by the hotplug process depends on the order in which the devices are detected, not which USB host port in which they are inserted. If you boot the system with two USB devices inserted, the order in which they will be detected is non-deterministic, depending on the timing of the USB hardware inside the drives enumerating to the USB controller on the EK board. We only leave the USB drive that we want to use as our root file system inserted, so we can be sure to know what name to provision here. We also set the root delay to 5 seconds. This causes the boot process to pause long enough for the hotplug facility in the kernel to detect the USB drive before we try to mount it as our root file system.

Uboot> setenv bootargs 'console=ttyS0,115200 mem=32M rootdelay=5 root=/dev/sda1'

Download the Arroyo Linux kernel image from the TFTP server and start it.

Uboot> run arroyo
TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
#################################################################
#######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start

## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=5 root=/dev/sda1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usb 1-2: new full speed USB device using at91_ohci and address 2
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:01:21 UTC (883612881)
Waiting 5sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 0:0:0:0: [sda] Attached SCSI removable disk
VFS: Mounted root (ext2 filesystem).
Freeing init memory: 96K
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex

www.diag.com Arroyo 0.0

arroyo login: root
#

We are running Linux using the USB drive as our root file system.

Install the same root file system image on our original SD card.

Reinsert the first USB drive, the one containing the root file system image file that we downloaded, and mount it.

usb 1-1: new full speed USB device using at91_ohci and address 3
usb 1-1: configuration #1 chosen from 1 choice
scsi1 : SCSI emulation for USB Mass Storage devices
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 1:0:0:0: [sdb] Attached SCSI removable disk

# mount /dev/sdb1 /mnt
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
# cd /mnt
# ls
arroyo-root.ext2 lost+found

Just for fun, run the fdisk partitioning utility on the SD card, "/dev/mmcblk0p1".

# fdisk /dev/mmcblk0p1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/mmcblk0p1

Command (m for help): m
Command action
a toggle bootable flag
b edit bsd disklabel
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): p

Disk /dev/mmcblk0p1: 2 GB, 2023418880 bytes
255 heads, 63 sectors/track, 246 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/mmcblk0p1p1 1 247 1983996 83 Linux

Warning: Partition 1 does not end on cylinder boundary.

Command (m for help): q

Install the file system image on the SD card much like we did on the USB drive. Run a file system check on it, and resize it.

# dd if=arroyo-root.ext2 of=/dev/mmcblk0p1 bs=1M
usb 1-1: reset full speed USB device using at91_ohci and address 3
usb 1-1: reset full speed USB device using at91_ohci and address 3
127+1 records in
127+1 records out
# e2fsck -f /dev/mmcblk0p1
e2fsck 1.41.2 (02-Oct-2008)
Filesystem did not have a UUID; generating one.

Superblock last write time is in the future. Fix? yes

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mmcblk0p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mmcblk0p1: 4438/7168 files (0.2% non-contiguous), 111875/130700 blocks
#
# resize2fs /dev/mmcblk0p1
resize2fs 1.41.2 (02-Oct-2008)
Resizing the filesystem on /dev/mmcblk0p1 to 1983616 (1k) blocks.
The filesystem on /dev/mmcblk0p1 is now 1983616 blocks long.

Reboot the system to use the new root file system image on our original SD card.

We just use the standard "arroyo" and "start" scripts that we used when we first brought the system up. But this time, the root file system will be the one we just installed.

# sync
# sync
# sync
# reboot
The system is going down NOW!
Sending SIGTERM to all processes
Sending SIGKILL to all processes
Requesting system reboot
Restarting system.

boot 1.0 (Aug 8 2003 - 12:29:00)

Uncompressing image...

U-Boot 1.1.1 (Oct 2 2004 - 19:04:01)


U-Boot code: 21F00000 -> 21F16DF0 BSS: -> 21F1B4AC
RAM Configuration:
Bank #0: 20000000 32 MB
Atmel: AT49BV6416 (64Mbit)
Flash: 8 MB
In: serial
Out: serial
Err: serial
Uboot> printenv
baudrate=115200
ethaddr=02:00:00:00:00:00
bootdelay=5
kernel=tftp 21000000 linux-ek
ramdisk=tftp 21100000 ramdisk
start=bootm 21000000
diminuto=tftp 21000000 diminuto-linux-2.6.25.10
arroyo=tftp 21000000 arroyo-linux-2.6.26.3
filesize=1386cc
fileaddr=21000000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.223
serverip=192.168.1.222
bootargs=console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
stdin=serial
stdout=serial
stderr=serial

Environment size: 459/8188 bytes
Uboot> run arroyo

TFTP from server 192.168.1.222; our IP address is 192.168.1.223
Filename 'arroyo-linux-2.6.26.3'.
Load address: 0x21000000
Loading: #################################################################
#################################################################
################################################################# #######################################################
done
Bytes transferred = 1279692 (1386cc hex)
Uboot> run start

## Booting image at 21000000 ...
Image Name: Linux-2.6.26.3
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1279628 Bytes = 1.2 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux................................................................................... done, booting the kernel.
Linux version 2.6.26.3 (jsloan@silver) (gcc version 4.2.3 (Sourcery G++ Lite 2008q1-126)) #38 Sun Oct 12 15:10:54 MDT 2008
CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
Machine: Atmel AT91RM9200-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz
CPU0: D VIVT write-back cache
CPU0: I cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
CPU0: D cache: 16384 bytes, associativity 64, 32 byte lines, 8 sets
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: console=ttyS0,115200 mem=32M rootdelay=2 root=/dev/mmcblk0p1
AT91: 128 gpio irqs in 4 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 29728KB available (2344K code, 274K data, 96K init)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 192 bytes
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 58
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Non-volatile memory driver v1.2
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffc4000 (irq = 7) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (02:00:00:00:00:00)
eth0: Davicom 9161 PHY (Copper)
Driver 'sd' needs updating - please use bus_type methods
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffe0000 (irq 13)
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 23, io mem 0x00300000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usb 1-1: new full speed USB device using at91_ohci and address 2
usb 1-1: configuration #1 chosen from 1 choice
usb 1-2: new full speed USB device using at91_ohci and address 3
usb 1-2: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
scsi1 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
at91_rtc at91_rtc: rtc core: registered at91_rtc as rtc0
AT91 Real Time Clock driver.
i2c /dev entries driver
i2c-gpio i2c-gpio: using pins 57 (SDA) and 58 (SCL)
AT91 Watchdog Timer enabled (5 seconds)
at91_mci at91_mci: 4 wire bus mode not supported - using 1 wire
Registered led device: green
Registered led device: yellow
Registered led device: red
TCP cubic registered
NET: Registered protocol family 17
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:39 UTC (883612839)
Waiting 2sec before mounting root device...
mmc0: card is read-write
mmc0: new SD card at address e624
mmcblk0: mmc0:e624 SD02G 1985024KiB
mmcblk0: p1
VFS: Mounted root (ext2 filesystem).
Freeing init memory: 96K
scsi 1:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 1:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sda] Write Protect is off
sd 1:0:0:0: [sda] Assuming drive cache: write through
sd 1:0:0:0: [sda] 3905536 512-byte hardware sectors (2000 MB)
sd 1:0:0:0: [sda] Write Protect is off
sd 1:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 1:0:0:0: [sda] Attached SCSI removable disk
scsi 0:0:0:0: Direct-Access TOSHIBA TransMemory PMAP PQ: 0 ANSI: 0 CCS
sd 0:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sdb] Write Protect is off
sd 0:0:0:0: [sdb] Assuming drive cache: write through
sd 0:0:0:0: [sdb] 3905536 512-byte hardware sectors (2000 MB)
sd 0:0:0:0: [sdb] Write Protect is off
sd 0:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 0:0:0:0: [sdb] Attached SCSI removable disk
Initializing random number generator... done.
Starting network...
ip: RTNETLINK answers: File exists
eth0: Link now 100-FullDuplex

www.diag.com Arroyo 0.0

arroyo login: root

Just for fun, run the partitioning utility to look at both of the USB drives.

# fdisk /dev/sda1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/sda1

Command (m for help): p

Disk /dev/sda1: 1 GB, 1990517760 bytes
255 heads, 63 sectors/track, 242 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1p1 1 243 1951866 83 Linux

Warning: Partition 1 does not end on cylinder boundary.

Command (m for help): q
# fdisk /dev/sdb1
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Using /dev/sdb1

Command (m for help): p

Disk /dev/sdb1: 1 GB, 1990517760 bytes
255 heads, 63 sectors/track, 242 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1p1 1 243 1951866 83 Linux
Warning: Partition 1 does not end on cylinder boundary.
Command (m for help): q
# mount
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw)
#

Part of the beauty of using Linux as an embedded operating system is that the tools and techniques you use to work on it are the same as those you use on your server. In fact, the commands I used to build my first root file system on an SD card for Arroyo on my Ubuntu-based Dell server were almost identical to those I used here on Arroyo itself, differing on only slightly in the device names because of differences in hardware between the two systems.