Monday, January 24, 2011

Memory Mapped Devices on the Beagle Board with Android

If you're an embedded software developer, you already know that memory is not always as it appears. Sometimes it is really an interface into a hardware device merely pretending to be memory but which may exhibit behavior that is very non-memory like. (And thanks to memory models crippled by compiler optimizations and high-performance memory subsystems, this is unfortunately sometimes true of real memory as well.)

This is not that different conceptually from the Linux /proc and /sys directories, which contain objects that pretend to be files in a file system, but are in fact interfaces into various subsystems and device drivers in kernel-space. While some hardware architectures have specialized machine instructions to interface with I/O devices, most of the ones I use (going all the way back to the PDP-11 in the 1970s) expect I/O devices to expose control and data registers that can be read and written by software in a memory-like way. The mapping of hardware interfaces into the physical memory space is called memory mapped I/O. This is typically how the Linux kernel and device drivers interface with I/O devices. It might surprise you to learn that Linux provides system calls such that user-space applications can do this too.

I recently got around to porting Diminuto, my C-based systems programming library and toolkit, to Contraption, my project to experiment with the ARM-based Beagle Board running the Android software stack. This included the mmdriver device driver and the utilities memtool (no relation to any other memtool) and mmdrivertool.

memtool is a user-space application that uses the Linux system calls mmap and munmap to map an arbitrary region of physical memory space specified in a command line argument into virtual memory space. It then allows you to manipulate that physical memory space with operations like read, write, set, and clear. If that physical memory space happens to be an I/O device, you find yourself controlling actual hardware. Since such activities are fraught with peril (you can crash your system, as I have done many times, or even damage it) you have to be root to run memtool.

Here is the help menu that memtool displays. (This was scraped off the console of my Beagle Board running the FroYo release of Android Rowboat. Apologies in advance for any violence done to it by the Blogger editor that I miss.)

bash-3.2# memtool -?
usage: memtool [ -d ] [ -o ] [ -a ADDDRESS ] [ -l BYTES ] [ -[1|2|4|8] ADDRESS ]
[ -r | -[s|S|c|C|w] NUMBER ] [ -u USECONDS ] [ -t | -f ] [ ... ]
-1 ADDRESS Use byte at ADDRESS
-2 ADDRESS Use halfword at ADDRESS
-4 ADDRESS Use word at ADDRESS
-8 ADDRESS Use doubleword at ADDRESS
-a ADDRESS Optionally map region at ADDRESS
-d Enable debug mode
-f Proceed if the last result was 0
-l BYTES Optionally map BYTES in length
-o Enable core dumps
-t Proceed if the last result was !0
-u USECONDS Sleep for USECONDS microseconds
-? Print menu

The Beagle Board has a couple of tiny intermittent-contact buttons on it. One of them is the RESET button. Play with the Beagle Board enough like I do and you'll learn where the RESET button is. Right next to it is a button labelled USER which is connected to pad AE21 on the Beagle Board's Texas Instruments OMAP3530DCBB72 processor. You can read the state of this button by inspecting bit seven of the GPIO_DATAIN register for the GPIO1 module. GPIO1 is one of the many I/O controllers built into the OMAP System On a Chip (SoC).

(In this photograph you can see the USER button, and just above it, the RESET button, down at the lower left.)


It will probably not surprise you to know that the GPIO_DATAIN for GPIO1 is mapped into physical memory space, at location 0x48310038 on this OMAP processor, and that I can use memtool to read it. Bit seven would be represented by the mask (1<<7) or 0x80. When the USER button is pressed, this bit reads as a one. (If you're not used to this type of thing, it's kind of magical.)

Here I am using memtool to read location 0x48310038 while pressing and releasing the USER button. memtool prints the value of the memory location to standard output in decimal so it can be more easily used in shell scripts. The little Diminuto utility hex prints it in hexadecimal to make our lives easier.

bash-3.2# memtool -4 0x48310038 -r | hex
bash-3.2# memtool -4 0x48310038 -r | hex
bash-3.2# memtool -4 0x48310038 -r | hex

Field Programmable Gate Arrays (FPGAs) are programmable digital logic devices. They have a broad range of capabilities and applications, depending on the brand and model, from glue logic to connect discrete hardware devices together to implementing entire microprocessors and I/O cores. FPGAs have been an intrinsic part of every embedded system I've worked on in the past fifteen years.

A common approach to interfacing an FPGA with a processor is, you guessed it, memory mapped I/O. The FPGA exposes a set of registers, anywhere from a few to dozens, into the physical memory space of the processor. You can use memtool to communicate with an FPGA in such an architecture. I have routinely done this during board bring up, testing, and troubleshooting.

Another approach is to use the Diminuto generic memory mapped device driver mmdriver and its associated utility mmdrivertool. The mmdriver device driver is a loadable module that lives and works in kernel-space. It uses the kernel functions ioremap and iounmap to map a physical memory space into virtual memory space just once when it is installed. It then takes commands via the ioctl Linux system call to read, write, set, and clear portions of that physical memory space. The mmdrivertool is a user-space utility that allows you to send commands to mmdriver via command line arguments in a fashion similar to how I used memtool.

Here is the help menu that mmdrivertool displays.

bash-3.2# mmdrivertool -?
usage: mmdrivertool [ -d ] [ -U DEVICE ] [ -[1|2|4|8] OFFSET ] [ -r | -[s|S|c|C|w] NUMBER ] [ -u USECONDS ] [ -t | -f ] [ ... ]
-1 OFFSET Use byte at OFFSET
-2 OFFSET Use halfword at OFFSET
-4 OFFSET Use word at OFFSET
-8 OFFSET Use doubleword at OFFSET
-C NUMBER Clear 1<<NUMBER mask at OFFSET
-U DEVICE Use DEVICE instead of /dev/mmdriver
-d Enable debug mode
-f Proceed if the last result was 0
-r Read OFFSET
-t Proceed if the last result was !0
-u USECONDS Sleep for USECONDS microseconds
-? Print menu

Because mmdriver maps a fixed region of physical memory space when it is installed, areas within that space are specified on the mmdrivertool command line as offsets from the beginning of that space, instead of as physical memory addresses as with memtool. Otherwise, the two commands look very similar.

Here I am installing the mmdriver device driver module and specifying what region of physical memory space it is to map. In this example, I just map the GPIO_DATAIN register of the GPIO1 module. But you could just as easily map an entire FPGA register set into virtual memory. Because mmdriver is built on top of two other Diminuto kernel modules, I install those first.

bash-3.2# lsmod
omaplfb 8882 0 - Live 0xbf032000
pvrsrvkm 137146 31 omaplfb, Live 0xbf000000
bash-3.2# insmod diminuto_kernel_map.ko
bash-3.2# insmod diminuto_kernel_datum.ko
bash-3.2# insmod diminuto_mmdriver.ko "begin=0x48310038 end=0x4831003c"
bash-3.2# lsmod
diminuto_mmdriver 3633 0 - Live 0xbf03d000
diminuto_kernel_datum 883 0 - Live 0xbf037000
diminuto_kernel_map 1057 1 diminuto_mmdriver, Live 0xbf02d000
omaplfb 8882 0 - Live 0xbf032000
pvrsrvkm 137146 31 omaplfb, Live 0xbf000000

By default, mmdriver registers itself as a miscellaneous device and allows the Linux miscellaneous device driver misc to dynamically assign its minor device number. I can use the /proc file system to see what the major (10) and minor (51) device numbers for mmdriver are. (You can also specify your own major device number for mmdriver at install time.)

bash-3.2# cat /proc/devices
Character devices:
1 mem
4 /dev/vc/0
4 tty
4 ttyS
5 /dev/tty
5 /dev/console
5 /dev/ptmx
7 vcs
10 misc
13 input
29 fb
81 video4linux
89 i2c
90 mtd
116 alsa
128 ptm
136 pts
153 spi
180 usb
189 usb_device
250 pvrsrvkm
251 omap-resizer
252 omap-previewer
253 usbmon
254 rtc

bash-3.2# cat /proc/misc
51 mmdriver
52 network_throughput
53 network_latency
54 cpu_dma_latency
55 log_system
56 log_radio
57 log_events
58 log_main
59 binder
130 watchdog
60 alarm
223 uinput
1 psaux
61 android_adb_enable
62 android_adb
63 ashmem

Now I create a character device node in the file system for mmdriver using its major and minor device number. Because Android doesn't provide a mknod command, I'll cheat and use busybox, which I described in a prior article.

bash-3.2# busybox mknod /dev/mmdriver c 10 51
bash-3.2# ls -l /dev/mmdriver
crw-rw-rw- root root 10, 51 2011-01-24 22:17 mmdriver

This may seem like a lot of work, but the point is you only have to install the modules and make the device node once at boot time, and it can be automated in a script.

Now I can use mmdrivertool to do exactly the same thing as I did with memtool while pressing and releasing the USER button on the Beagle Board.

bash-3.2# mmdrivertool -4 0 -r | hex
bash-3.2# mmdrivertool -4 0 -r | hex
bash-3.2# mmdrivertool -4 0 -r | hex

All of the source code for what I have described here is available in the latest Diminuto release. It is all licensed under the GPL or the LGPL.

Not to make things more complicated than they need to be, but I've used the term memory mapping in two different senses in this article. I/O devices map their control and data registers into a physical memory address space such that they appear to be at a physical memory location. But in most Linux systems, real physical random-access memory is itself mapped. Both I/O registers and physical memory are mapped into a virtual memory space using a hardware memory management unit (MMU) that is part of the OMAP processor. Software, whether it be running in kernel-space or user-space, accesses I/O registers and real memory using virtual memory addresses, not the actual physical memory addresses.

The value 0x48310038 cited in the examples above is a physical address. The register it points to is mapped by both memtool (using user-space system calls) and mmdriver (using kernel-space functions) into a virtual address space, and it is the virtual address of this space that this these tools actually use, merely accessing it as a pointer in C. The tools themselves handle this and hide the details from the user, but you can see it in the source code.

Remember earlier when I said memory mapped devices may exhibit non-memory-like behavior? This is a lot less common than it used to be, but three thousand page processor reference manuals still bear close scrutiny lest wackiness ensue.

Inter-Integrated Circuit (I2C, pronounced I-squared-C) is a two-wire serial bus standard commonly used to interrogate simple off-chip devices like temperature or power sensors. An I2C implementation in a sensor is not much more complicated than a simple state machine and a shift register. Here is a sentence from Freescale Semiconductor's MPC8349E PowerQUICC II Pro Integrated Host Processor Family Reference Manual on a memory mapped hardware data register for its I2C controller.

In master receive mode, reading the data register allows the read to occur, but also allows the I2C module to receive the next byte of data on the I2C interface. [Rev. 1, page 17-9]

What this is saying is that reading this memory mapped register on the PowerPC not only returns the most recent value read from the I2C bus, but clocks in the next data byte from the bus. By reading a location that looks like memory, and which is accessed in C just as if you were reading a variable, not only is the value of that location changed by your read, but it changes the state of a slave device that is completely outside of the PowerPC chip. If you don't have a lot of experience in embedded development, this could be a really surprising behavior.

I have written much already in this blog about memory models on modern microprocessor architectures (for example, here and here). I'll write more in the future about how to tell when memory, or something that looks like memory, is actually being accessed in your C or C++ program. When you're dealing with memory mapped hardware devices, it pays to know these things.

Update (2011-01-27)

Just days after writing this article I was trying to set some bits in an FPGA register and my software was getting a bus error, which on this platform meant I was trying to access a physical address that didn't exist. It turns out the firmware developer had made that register mapped into memory by the FPGA write-only by virtue of not implementing a read interface. (And they had documented it. I was just too lazy to look closely at the comments in the header file while writing my software. My bad.) That means I could not set a bit by doing

fpga->field |= (1<<21);

but instead had to

fpga->field = (1<<21);

The difference is subtle enough you may have to be an embedded wonk to know the difference: the former does a read-modify-write memory operation, while the latter just does a write.

Business as usual in the field of embedded development.


Kaiwan N Billimoria said...

Hi, excellent article! Thanks.

I have a qs: i've been searching like heck for a single reference manual / doc which clearly shows the memory map (registers, MMIO, etc) on the BEagleBoard OMAP3 (i have a Rev C board).

Can you point out a good url/whatever ?


Chip Overclock said...

I use

TI, OMAP35x Applications Processor Technical Reference Manual, SPRUF98G, May 2010

Yes, it's long, at 3423 pages. And it's not as clearly written as similar reference manuals I've used from Freescale or Atmel. But it seems pretty complete, and I've been able to tease the information I need from it.

Of course, it's not specific to the BeagleBoard. So to understand how the BB uses all those GPIO pins, you have to read the appropriate BeagleBoard System Reference Manual. It takes some effort to correlate the two sometimes.

Best of luck, Kaiwan, and thanks for the comment!

Chip Overclock said...

It just occurred to me that you might have been referring to the newer BeagleBoard xM Revision C (which I just started using) and not the Revision C of the original BeagleBoard (which is what I was using when I wrote this article). The xM uses a different processor, the DM3730, instead of the OMAP3530, but it's still an ARM Cortex-A8 core and is compatible with the same OMAP3 family. But it has a different reference manual. It's even longer.

TI, AM/DM37x Multimedia Device Technical Reference Manual, SPRUGN4L, June 2011

Kaiwan N Billimoria said...

No, am using the older / original Rev C board...thanks anyway.

I have seen the manual you mentioned; was just hoping there's something easier to use :)

Another unrelated thing:
What would you recommend as a good ethernet over usb solution for the beagle board (rev C)?

Chip Overclock said...

Alas, I can't speak to that. I used the Zippy2 expansion board, which among other things provided an Ethernet port whose interface was via SPI. The Zippy2 is no longer in production. The xM board has it's own Ethernet port connected through the integrated USB hub.

jithu said...

I am wondering how exactly do u implement the memtool without any kernel components just using mmap ... I fully understood the second approach using the kernel module which does ioremap ; hence can u throw some light into the parameters for mmap for the first approach ... I see only an fd and don't see how u can specify a physical address to mmap.

Chip Overclock said...

If you look at the source code in diminuto_map.c in the Diminuto tarball available at

and read the man page for mmap(2), you'll see that the mmap() system call maps an arbitrary section of physical memory of a given size (the second parameter) at a specific physical address (the sixth parameter) to a virtual memory location. It does this by manipulating the tables used by the processor's memory management unit (MMU) and the Linux kernel to map physical addresses to virtual addresses. The fifth parameter is the file descriptor created by opening the /dev/mem device.

The only snag is that you have to have root permissions to open /dev/mem.