Memory Mapped Devices on the Beagle Board with Android

If you're an embedded software developer, you already know that memory is not always as it appears. Sometimes it is really an interface into a hardware device merely pretending to be memory but which may exhibit behavior that is very non-memory like. (And thanks to memory models crippled by compiler optimizations and high-performance memory subsystems, this is unfortunately sometimes true of real memory as well.)

This is not that different conceptually from the Linux /proc and /sys directories, which contain objects that pretend to be files in a file system, but are in fact interfaces into various subsystems and device drivers in kernel-space. While some hardware architectures have specialized machine instructions to interface with I/O devices, most of the ones I use (going all the way back to the PDP-11 in the 1970s) expect I/O devices to expose control and data registers that can be read and written by software in a memory-like way. The mapping of hardware interfaces into the physical memory space is called memory mapped I/O. This is typically how the Linux kernel and device drivers interface with I/O devices. It might surprise you to learn that Linux provides system calls such that user-space applications can do this too.

I recently got around to porting Diminuto, my C-based systems programming library and toolkit, to Contraption, my project to experiment with the ARM-based Beagle Board running the Android software stack. This included the mmdriver device driver and the utilities memtool (no relation to any other memtool) and mmdrivertool.

memtool is a user-space application that uses the Linux system calls mmap and munmap to map an arbitrary region of physical memory space specified in a command line argument into virtual memory space. It then allows you to manipulate that physical memory space with operations like read, write, set, and clear. If that physical memory space happens to be an I/O device, you find yourself controlling actual hardware. Since such activities are fraught with peril (you can crash your system, as I have done many times, or even damage it) you have to be root to run memtool.

Here is the help menu that memtool displays. (This was scraped off the console of my Beagle Board running the FroYo release of Android Rowboat. Apologies in advance for any violence done to it by the Blogger editor that I miss.)

bash-3.2# memtool -?
usage: memtool [ -d ] [ -o ] [ -a ADDDRESS ] [ -l BYTES ] [ -[1|2|4|8] ADDRESS ]
[ -r | -[s|S|c|C|w] NUMBER ] [ -u USECONDS ] [ -t | -f ] [ ... ]
-1 ADDRESS    Use byte at ADDRESS
-2 ADDRESS    Use halfword at ADDRESS
-4 ADDRESS    Use word at ADDRESS
-8 ADDRESS    Use doubleword at ADDRESS
-C NUMBER     Clear 1<<NUMBER mask at ADDRESS
-S NUMBER     Set 1<<NUMBER mask at ADDRESS
-a ADDRESS    Optionally map region at ADDRESS
-c NUMBER     Clear NUMBER mask at ADDRESS
-d            Enable debug mode
-f            Proceed if the last result was 0
-l BYTES      Optionally map BYTES in length
-o            Enable core dumps
-r            Read ADDRESS
-s NUMBER     Set NUMBER mask at ADDRESS
-t            Proceed if the last result was !0
-u USECONDS   Sleep for USECONDS microseconds
-w NUMBER     Write NUMBER to ADDRESS
-?            Print menu

The Beagle Board has a couple of tiny intermittent-contact buttons on it. One of them is the RESET button. Play with the Beagle Board enough like I do and you'll learn where the RESET button is. Right next to it is a button labelled USER which is connected to pad AE21 on the Beagle Board's Texas Instruments OMAP3530DCBB72 processor. You can read the state of this button by inspecting bit seven of the GPIO_DATAIN register for the GPIO1 module. GPIO1 is one of the many I/O controllers built into the OMAP System On a Chip (SoC).

(In this photograph you can see the USER button, and just above it, the RESET button, down at the lower left.)

It will probably not surprise you to know that the GPIO_DATAIN for GPIO1 is mapped into physical memory space, at location 0x48310038 on this OMAP processor, and that I can use memtool to read it. Bit seven would be represented by the mask (1<<7) or 0x80. When the USER button is pressed, this bit reads as a one. (If you're not used to this type of thing, it's kind of magical.)

Here I am using memtool to read location 0x48310038 while pressing and releasing the USER button. memtool prints the value of the memory location to standard output in decimal so it can be more easily used in shell scripts. The little Diminuto utility hex prints it in hexadecimal to make our lives easier.

bash-3.2# memtool -4 0x48310038 -r | hex

0x3c

bash-3.2# memtool -4 0x48310038 -r | hex

0xbc

bash-3.2# memtool -4 0x48310038 -r | hex

0x3c

Field Programmable Gate Arrays (FPGAs) are programmable digital logic devices. They have a broad range of capabilities and applications, depending on the brand and model, from glue logic to connect discrete hardware devices together to implementing entire microprocessors and I/O cores. FPGAs have been an intrinsic part of every embedded system I've worked on in the past fifteen years.

A common approach to interfacing an FPGA with a processor is, you guessed it, memory mapped I/O. The FPGA exposes a set of registers, anywhere from a few to dozens, into the physical memory space of the processor. You can use memtool to communicate with an FPGA in such an architecture. I have routinely done this during board bring up, testing, and troubleshooting.

Another approach is to use the Diminuto generic memory mapped device driver mmdriver and its associated utility mmdrivertool. The mmdriver device driver is a loadable module that lives and works in kernel-space. It uses the kernel functions ioremap and iounmap to map a physical memory space into virtual memory space just once when it is installed. It then takes commands via the ioctl Linux system call to read, write, set, and clear portions of that physical memory space. The mmdrivertool is a user-space utility that allows you to send commands to mmdriver via command line arguments in a fashion similar to how I used memtool.

Here is the help menu that mmdrivertool displays.

bash-3.2# mmdrivertool -?
usage: mmdrivertool [ -d ] [ -U DEVICE ] [ -[1|2|4|8] OFFSET ] [ -r | -[s|S|c|C|w] NUMBER ] [ -u USECONDS ] [ -t | -f ] [ ... ]
-1 OFFSET   Use byte at OFFSET
-2 OFFSET   Use halfword at OFFSET
-4 OFFSET   Use word at OFFSET
-8 OFFSET   Use doubleword at OFFSET
-C NUMBER   Clear 1<<NUMBER mask at OFFSET
-S NUMBER   Set 1<<NUMBER mask at OFFSET
-U DEVICE   Use DEVICE instead of /dev/mmdriver
-c NUMBER   Clear NUMBER mask at OFFSET
-d          Enable debug mode
-f          Proceed if the last result was 0
-r          Read OFFSET
-s NUMBER   Set NUMBER mask at OFFSET
-t          Proceed if the last result was !0
-u USECONDS Sleep for USECONDS microseconds
-w NUMBER   Write NUMBER to OFFSET
-?          Print menu

Because mmdriver maps a fixed region of physical memory space when it is installed, areas within that space are specified on the mmdrivertool command line as offsets from the beginning of that space, instead of as physical memory addresses as with memtool. Otherwise, the two commands look very similar.

Here I am installing the mmdriver device driver module and specifying what region of physical memory space it is to map. In this example, I just map the GPIO_DATAIN register of the GPIO1 module. But you could just as easily map an entire FPGA register set into virtual memory. Because mmdriver is built on top of two other Diminuto kernel modules, I install those first.

bash-3.2# lsmod

omaplfb 8882 0 - Live 0xbf032000

pvrsrvkm 137146 31 omaplfb, Live 0xbf000000

bash-3.2# insmod diminuto_kernel_map.ko

bash-3.2# insmod diminuto_kernel_datum.ko

bash-3.2# insmod diminuto_mmdriver.ko "begin=0x48310038 end=0x4831003c"

bash-3.2# lsmod

diminuto_mmdriver 3633 0 - Live 0xbf03d000

diminuto_kernel_datum 883 0 - Live 0xbf037000

diminuto_kernel_map 1057 1 diminuto_mmdriver, Live 0xbf02d000

omaplfb 8882 0 - Live 0xbf032000

pvrsrvkm 137146 31 omaplfb, Live 0xbf000000

By default, mmdriver registers itself as a miscellaneous device and allows the Linux miscellaneous device driver misc to dynamically assign its minor device number. I can use the /proc file system to see what the major (10) and minor (51) device numbers for mmdriver are. (You can also specify your own major device number for mmdriver at install time.)

bash-3.2# cat /proc/devices

Character devices:

1 mem

4 /dev/vc/0

4 tty

4 ttyS

5 /dev/tty

5 /dev/console

5 /dev/ptmx

7 vcs

10 misc

13 input

29 fb

81 video4linux

89 i2c

90 mtd

116 alsa

128 ptm

136 pts

153 spi

180 usb

189 usb_device

250 pvrsrvkm

251 omap-resizer

252 omap-previewer

253 usbmon

254 rtc

bash-3.2# cat /proc/misc

51 mmdriver

52 network_throughput

53 network_latency

54 cpu_dma_latency

55 log_system

56 log_radio

57 log_events

58 log_main

59 binder

130 watchdog

60 alarm

223 uinput

1 psaux

61 android_adb_enable

62 android_adb

63 ashmem

Now I create a character device node in the file system for mmdriver using its major and minor device number. Because Android doesn't provide a mknod command, I'll cheat and use busybox, which I described in a prior article.

bash-3.2# busybox mknod /dev/mmdriver c 10 51

bash-3.2# ls -l /dev/mmdriver

crw-rw-rw- root root 10, 51 2011-01-24 22:17 mmdriver

This may seem like a lot of work, but the point is you only have to install the modules and make the device node once at boot time, and it can be automated in a script.

Now I can use mmdrivertool to do exactly the same thing as I did with memtool while pressing and releasing the USER button on the Beagle Board.

bash-3.2# mmdrivertool -4 0 -r | hex

0x3c

bash-3.2# mmdrivertool -4 0 -r | hex

0xbc

bash-3.2# mmdrivertool -4 0 -r | hex

0x3c

All of the source code for what I have described here is available in the latest Diminuto release. It is all licensed under the GPL or the LGPL.

Not to make things more complicated than they need to be, but I've used the term memory mapping in two different senses in this article. I/O devices map their control and data registers into a physical memory address space such that they appear to be at a physical memory location. But in most Linux systems, real physical random-access memory is itself mapped. Both I/O registers and physical memory are mapped into a virtual memory space using a hardware memory management unit (MMU) that is part of the OMAP processor. Software, whether it be running in kernel-space or user-space, accesses I/O registers and real memory using virtual memory addresses, not the actual physical memory addresses.

The value 0x48310038 cited in the examples above is a physical address. The register it points to is mapped by both memtool (using user-space system calls) and mmdriver (using kernel-space functions) into a virtual address space, and it is the virtual address of this space that this these tools actually use, merely accessing it as a pointer in C. The tools themselves handle this and hide the details from the user, but you can see it in the source code.

Remember earlier when I said memory mapped devices may exhibit non-memory-like behavior? This is a lot less common than it used to be, but three thousand page processor reference manuals still bear close scrutiny lest wackiness ensue.

Inter-Integrated Circuit (I²C, pronounced I-squared-C) is a two-wire serial bus standard commonly used to interrogate simple off-chip devices like temperature or power sensors. An I²C implementation in a sensor is not much more complicated than a simple state machine and a shift register. Here is a sentence from Freescale Semiconductor's MPC8349E PowerQUICC II Pro Integrated Host Processor Family Reference Manual on a memory mapped hardware data register for its I²C controller.

In master receive mode, reading the data register allows the read to occur, but also allows the I²C module to receive the next byte of data on the I²C interface. [Rev. 1, page 17-9]

What this is saying is that reading this memory mapped register on the PowerPC not only returns the most recent value read from the I²C bus, but clocks in the next data byte from the bus. By reading a location that looks like memory, and which is accessed in C just as if you were reading a variable, not only is the value of that location changed by your read, but it changes the state of a slave device that is completely outside of the PowerPC chip. If you don't have a lot of experience in embedded development, this could be a really surprising behavior.

I have written much already in this blog about memory models on modern microprocessor architectures (for example, here and here). I'll write more in the future about how to tell when memory, or something that looks like memory, is actually being accessed in your C or C++ program. When you're dealing with memory mapped hardware devices, it pays to know these things.

Update (2011-01-27)

Just days after writing this article I was trying to set some bits in an FPGA register and my software was getting a bus error, which on this platform meant I was trying to access a physical address that didn't exist. It turns out the firmware developer had made that register mapped into memory by the FPGA write-only by virtue of not implementing a read interface. (And they had documented it. I was just too lazy to look closely at the comments in the header file while writing my software. My bad.) That means I could not set a bit by doing

fpga->field |= (1<<21);

but instead had to

fpga->field = (1<<21);

The difference is subtle enough you may have to be an embedded wonk to know the difference: the former does a read-modify-write memory operation, while the latter just does a write.

Business as usual in the field of embedded development.