Friday, October 14, 2011

Automating Maintenance on Embedded Systems with Biscuits

If you make a living, as I have from time to time, developing embedded systems, it's not unusual to find your work day interrupted on a regular basis by application developers that want their own development systems updated to the latest platform release, or firmware installed to support a new version of an FPGA, or some other routine system maintenance chore. Even embedded systems based on Linux and GNU are frequently too small to implement the tools used by big iron, graphical interfaces like the Synaptic Package Manager for Ubuntu, or even command line tools like yum, apt, or dpkg. Sometimes the chore is as mundane as creating a tar ball of the system logs to ship back to you so you can debug a field problem. But you still don't want to burden a non-geek user with all the skills, and maybe even the root password, necessary to do that. What would be great is if you could automate that system maintenance chore in a secure fashion and get it done by just handing the user a USB thumb drive and telling them where to stick it. Biscuits do that.

A biscuit is a cpio archive that has been compressed with bzip2 and then encrypted using gpg (GNU Privacy Guard a.k.a. GNUPG) into a binary file. The biscuit command decrypts and decompresses the file, by default named biscuit.bin in the current working directory, and extracts its contents into a temporary directory. It then looks for an unencrypted executable file named biscuit, script or binary, in that directory. If it finds it, it modifies the PATH and LD_LIBRARY_PATH environmental variables to include the temporary directory, changes its current working directory to the original delivery media (for example, a USB drive), and then runs the executable. What happens next is up to the executable. When the executable exits, the temporary directory is cleaned up and and deleted. While it runs, the executable has access to both the temporary directory, where any other collateral material from inside the biscuit can be found, and the working directory, where results may be placed.

Usage: biscuit [ -C working_directory ] [ -f biscuit_file ]

The encryption defaults to EIGamal (ELG-E), an asymmetric key encryption algorithm based on Diffie-Hellman key exchange, using 1024 bit keys. EIGamal is not patent encumbered, and although longer keys are possible, 1024 bits strikes a good balance between speed and security for slower embedded processors. The server where biscuits are packaged maintains a key ring containing the public (encrypting) keys for all the embedded systems on which biscuits may be deployed. Each embedded system has a key ring containing its own secret (decrypting) keys.

Different unique public/secret key pairs can be created, allowing you to distinguish between product lines, architectures, even individual systems, whatever makes sense for your business. This allows you to create biscuits for product flavor A that cannot be executed on product flavor B: the wrong biscuits simply won't successfully decrypt. The encryption serves as both an authentication mechanism (the biscuit comes from a reputable source) and an authorization mechanism (the biscuit can be executed using root privileges). Since users can't crack open biscuits, they not only can't create their own biscuits, they can't even see what your biscuits do. Biscuits are opaque binary files. They cannot be easily reverse engineered without cracking the encryption. They can be delivered on removable media such as USB thumb drives, CD-ROMs, or DVDs, or they can be downloaded across a network via ftp, tftp,or scp.

I've been using this or similar approaches to automating system maintenance of embedded systems for several years now. A link to a clean room implementation of biscuits can be found on the Biscuit project web page. It's mostly just a Makefile to build GNUPG for multiple architectures, generate and manage the keys for both the build server and the embedded hosts, create the biscuit command, and package up biscuit binary files, plus a couple of example biscuit scripts. But the distribution tar ball also contains some useful stuff for automating the use of biscuits in your embedded system, depending on your needs and comfort level.

Users can just run the biscuit command manually and point it at a biscuit.bin binary file. You can set your embedded host up so that only root can do so, so that users must use sudo to run biscuit.

If your embedded Linux system uses udev, like big iron distributions like Ubuntu, you can install the 99-com-diag-biscuit.rules file into /etc/udev/rules.d/99-com-diag-biscuit.rules . That will cause your system to temporarily mount any block device that is inserted, look for a biscuit.bin file, run the biscuit command against it, and unmount the device when it completes. The udev rules I wrote for my use only apply to removable storage devices by excluding the device names used for permanently attached block storage. You should customize these rules to fit your own platform.

If your system uses a simpler version of udev, like embedded distributions like Angström, you can install the mount.sh script into /etc/udev/scripts/mount.sh to do the same thing. Or just look at the one line I inserted into Angström's mount.sh that runs biscuit against the auto-mounted media, and do something similar on your system.

If you have an older embedded system that doesn't use udev but instead uses hotplug, you can install the biscuit.hotplug.sh script into /etc/hotplug.d/block/biscuit.hotplug to accomplish the same thing.

If your kernel was configured to support hotplug, but you don't have any of the infrastructure, you can still do the above, plus you can install hotplug.sh into /sbin/hotplug. Look for /proc/sys/kernel/hotplug on your embedded host. If it exists, your kernel is built to support it. If its contents are blank, then you need to tell the kernel to invoke /sbin/hotplug by echo-ing that string into /proc/sys/kernel/hotplug every time you boot your system, or rebuild your kernel configured to do this automatically.

There are some caveats you would be wise to heed.

Once biscuits are released into the wild, you must assume that they will go feral. People will dig up a USB drive you gave them years before, and insert it into a system. They may do so out of desperation, remembering that this did something useful long ago. Or they may simply have forgotten where the drive came from and don't even know that there is a biscuit on it. This is the dark side of biscuits: they are effectively viruses.

Biscuit scripts deal with the former by being paranoid: for example, before loading new firmware on a system, the script makes sure that the system isn't already running an even newer version of the firmware. I have also written scripts that deleted the original encrypted file from the USB drive, making it a use once biscuit. Or the script dropped an identifying file on the USB drive (I like using a MAC address from the host embedded system as a file name), and checked for the presence of this breadcrumb first whenever it runs to check whether it has passed this way before. This approach works well because the user can delete the breadcrumb manually to force the biscuit to rerun.

If you include collateral material on the USB drive, for example scripts, shared libraries, binaries, data, try hard to put it inside the biscuit. I have seen scripts that invoked another script in cleartext on the USB drive outside of the encrypted file. This allows a villain to simply rewrite the cleartext script to do whatever they want, like starting a root shell on the console, subverting the authentication and authorization provided by the encryption. Sometimes the collateral is simply too large to put inside the biscuit because of limits in available temporary storage, which on embedded systems is often RAM disk, into which the biscuit binary file is unpacked. If that's the case, put checksums or hashes (I like MD5 or SHA-1) of the collateral inside the biscuit and have the script verify the likely integrity of the collateral before using it.

Similarly, use some care when the biscuit script executes commands that are on the embedded host. This is of course typically done, for example running fundamental stuff like bash. But don't run scripts or binaries that can be altered by unprivileged users. Your security is only as good as that of your root password.

It is important that you build both the build server gpg and the host embedded system gpg from the same GNUPG source tree. You must not assume that different versions of GNUPG are interoperable. That may be the case, but if it isn't you'll end up creating biscuits on your build server that your embedded hosts can't use. If you do upgrade to a new version of GNUPG on your build server while you have deployed embedded hosts using the older version, be very thorough in your interoperability testing.

Once you build keys and have installed them on embedded hosts, check the keys into source code control or otherwise archive them. You will not get the same keys if you generate them again, even if you use exactly the same parameters. Again, you can render biscuits unusable if you are not careful. However, with some not too mad skills, you can recover lost keys from your deployed embedded systems themselves should a crisis occur.

Be paranoid about your keys, not just on the build server, but on your embedded hosts too. On the embedded hosts, which may be deployed into the field and hence presumably beyond your physical control, the keys and the directory that holds them should be accessible only by root. Unprivileged users should use sudo to run the biscuit command manually, if they can do so at all.

But with some care, biscuits are a means of safely automating a variety of system maintenance functions without giving away the keys to the kingdom or forcing end users or even application developers to become embedded systems programmers. For complex tasks like software update, they are a way of implementing complex logic and making it a part of the software distribution package, not part of a command in the embedded host itself.

No comments: