1.5.6 Optimize Operating System
The Eris prototype robot was initially developed using Ubuntu Linux
10.10 for an operating system (OS). Ubuntu was chosen because it is very easy to install and it provided support for all of the robot’s hardware. The software was installed to an 8GB USB flash disk and connected to the robot with a breakout/dongle cable. While this system was easy to implement, it has many drawbacks:
 The robot is very slow to start up
 The graphical desktop wastes system resources
 Heavy write activity to the file system will degrade the flash memory o System log files o Memory swap space
 Unexpected power loss may corrupt the file system
 Flash disk not securely attached to the robot’s chassis
The OS has some other less obvious drawbacks as well. Ubuntu is a modern operating system with easy to use software installation and update tools. It may be “too easy” to install new software that could reduce the robot’s performance or cause compatibility issues. Finally, any modifications made to system were undocumented.
OS Functional Requirements
 The robot will start and be ready for use in 30 seconds or less
 The robot will boot from a compact flash card attached to the
Pluto board
 The OS will support all existing hardware and software systems
 A backup of the OS will be stored on another computer in Coover
3050
OS Non-Functional Requirements
 The setup and installation procedure will be documented
 The setup and installation will be as automated as possible
 The OS will be designed with future expansion in mind
Operating System Research
There are hundreds of Linux distributions available on the Internet that could be used to replace Ubuntu, choosing one to use wasn’t trivial.
The previous Korebot robots used Familiar Linux, a community developed
OS for Compaq PDAs. The Familiar distribution isn’t compatible with x86 processors, therefore it cannot be used for our project.
The Eris platform looks like a traditional embedded system, but the hardware has more in common with traditional desktop computers. Eris has

a normal BIOS, it boots from USB, IDE/ATA and S-ATA – not an embedded
ROM device. The PC/104+ system is electrically equivalent to normal PCI and ISA buses. Since we need to support “desktop-like” hardware, we looked at “desktop-like” systems that had embedded qualities.
Debian Linux - http://www.debian.org/
Debian was a logical consideration after seeing Ubuntu on the robot.
(Ubuntu is based on Debian Linux) It is possible to install Debian without a graphical desktop and it has much of the same hardware support. The combined support of the Debian and Ubuntu web forums provides an excellent support system.
Emdebian Linux - http://www.emdebian.org/
Emdebian Linux is derived from the last few versions of the regular
Debian Linux distribution. The goal of the project is to reduce the size of the applications and packages that are installed by removing support for extra languages and documentation. It was originally developed for
ARM processors but has started to support x86 and PowerPC with the newest version. They offer two versions for our platform: Emdebian Grip and Emdebian Baked. The “Grip” version is based on Debian Linux 6 and tries to maintain compatibility instead of optimizing for size. The
“Baked” version is more of a concept – take a normal Debian system and remove anything that isn’t necessary for operation. Once the system has been “baked” into your hardware it cannot be modified in place, the whole system must be re-installed.
RTLinux Free - http://www.rtlinuxfree.com/
RTLinux Free isn’t an operating system in and of itself, it’s a set of patches for a Linux 2.4 kernel designed to improve the performance of embedded systems. RTLinux Free is produced by WindRiver Linux.
TinyCore and MicroCore Linux http://distro.ibiblio.org/tinycorelinux/
The focus of TinyCore Linux is to provide a graphical desktop and package manager using a minimal amount of resources. MicroCore Linux is similar, but it doesn’t have a graphical interface. Both versions of the operating system boot very quickly, usually 3 or 4 seconds. These systems are unique in that they use a RAM based file system for storage.
Once the software is running, the boot image is no longer used. This ensures that the file system won’t be corrupted if power is lost, but makes permanent changes to the robot difficult.

Linux From Scratch - http://www.linuxfromscratch.org/
Linux from scratch is a set of instructions and guidelines for building a Linux based operating system from the ground up. The 323 page manual covers everything between compiling your compiler and installing the boot loader after building your Linux system. This could result in a very well optimized OS for the robot, but it would take a very long time to set up. The manual and provided software are an excellent resource for optimization and configuration of Linux systems.
OS Research Results
The robot will run a combination of the proposed operating systems. We needed to get the robot running as soon as possible to test systems not directly related to the OS. (IMU requirements, wheel design, etc) To meet this need, we chose the standard Debian Linux distribution as a base OS. A customized kernel was built from the 2.6.35.4 kernel sources.
The new kernel was combined with a copy of the Debian network installer on a USB flash disk and an OS was installed to an external 2.5” ATA hard disk. The robot is currently able to connect to the Korebot Server and load AI programs, but the motor controller is producing erratic results.
1 Once the motors are stable, the OS can be written to a USB flash drive for testing the entire system.
The current 2 “development” installation takes up 430MB of disk space.
This is small for a typical Debian system, it is still much larger than other embedded systems. Using the Linux from Scratch manual and the
Emdebian Baked documentation, this system will be trimmed and reduced to a minimum level. Finally, an image of the disk will be created for use as a ramdisk file system similar to MicroCore Linux.
Operating System Test Plan
Robot Start-up Time
Testing the start time is very simple – if the robot is ready for use in
30 seconds or less, it passes the test. A significant portion of the boot time is consumed by the BIOS self tests.
1
This was likely due to the poor tolerances of the old wheel design and the low resolution motor encoders.
2 From April 2011. In December 2011 the system is 1.2GB, largely due to the Debian
Live packages on the disk.

Kernel Performance Tuning
The Linux kernel has many configuration and tuning parameters than can improve (or decrease) overall system performance. Adjustments to the kernel scheduler, system timer frequency and any relevant complier flags should be tested. The effects of these parameters could be tested with a special AI program that reads the input of the motor encoders and duplicates the data on several GPIO pins. The copied data could be compared to the encoder signal lines using an oscilloscope.
File System Reliability
Start an AI program on the robot, then disconnect the power source. The
OS should not become corrupted after any number of cycles.
Kernel Development
Building the kernel was fairly straightforward, most of the work was completed during the first semester of the project. The first two kernels were developed with a “traditional” embedded OS in mind. The third kernel, eris-00.03, took a more traditional desktop OS approach.
It was merged into a Debian installation image and used to install the
Development OS on the robot. The eris-00.04 kernel is very similar to its predecessor, but additional file systems were included to make it compatible with Debian Live systems. All of the kernels were compiled on a Debian 6 workstation in Coover 3050. Each kernel 3 and its menuconfig settings file were committed to the SVN repositiory along with a document describing the changes and rationale behind each version.
Current Operating Systems
Development OS
This is the most complete system available to run on the robot. It has the newest drivers and kernels plus a wide variety of tools to test the robot's performance. It is the primary target for new robot features.
The development OS can also used to build Debian Live systems. It is installed on a 2.5” ATA hard disk, but will soon move to a 3.5” SATA
3 The eris-00.02 kernel was not saved, I don't know what happened to it. The menuconfig file is present.

disk to make connecting the drive to an assembled robot easier.
Good
Boots the newest kernel and initial ram disk
Connects to the WiFi network automatically
Support for the PCM-MIO-G
Bad
Cannot be used when the robot is in motion, the magnetic disk may be damaged
The file system can be corrupted if system power is lost
“Veisha OS”
In order to demonstrate the robot during the Veisha 2011 festival, a portion of the [then current] development OS was copied to a 512MB USB flash disk. Once transferred, the system was modified to connect with the WiFi network automatically and a shell script was written to initialize the FPGA and connect to a Korebot control server. Since the system is considerably smaller than the Development OS, it was used as a starting point for experiments with compressed/read-only file systems.
During the second semester the Veisha OS was written to a 512MB compact flash card to reduce the amount of wiring and connectors on the robot.
Good
First new OS that could be used to run the robot on the play field
Total size of 206MB
Automatic boot process was very similar to the older Korebots
Bad
Slow boot time 4
GRUB crashes after the initial transfer and is difficult to repair 5
Does not support the PCM-MIO-G
The file system can be corrupted if system power is lost
System log files may degrade the flash memory over time
Debian Live
The attempts to convert the Development and Veisha systems to use a read-only file system were not progressing well, an alternative was
4 When connected through a USB interface
5 See the Operating System section of the Appendix for more information

needed. Further research produced the Debian Live tools which can be used to make Linux “Live CD” images. The tools assemble a generic kernel, initial ramdisk and root filesystem (compressed with SquashFS) into a standard ISO disk image that can be copied to any storage device with the “dd” command. The images can be customized with software packages and configuration files. The current Debian Live system can boot the robot, but it does not yet support any of the robot's features.
It should be possible to merge the eris-00.04 kernel and initrd image with this system.
Good
Easy to create additional copies of the OS
Maintenance of Debian Live images is already documented
Read-only file system cannot be corrupted
Bad
Designed to produce “generic” systems, not custom ones
Not much more than a proof-of-concept test at this stage
First attempt to modify the ISO image didn't work
Boot performance tests
The system POST function takes approximately 18 seconds to complete.
This leaves only 12 seconds to finish booting the robot. Currently, none of the systems meet this requirement.
Device OS Name Kernel / Size Initrd /
Size
Root FS Boot Time*
Sandisk
Cruzer 8GB
Misc.
512MB
Debian
Live vmlinuz / 2.08
MiB
Veisha OS eris-00.03 /
2.65 MiB initrd.img
/ 9.89 MiB
Squash FS 46 s initrderis-
00.03.img.g
z / 8.90
MiB ext2 56 s
Edge 512MB
CF
Veisha OS eris-00.03 /
2.65 MiB initrderis-
00.03.img.g
z / 8.90 ext2 20 s

MiB
Toshiba
MK8025GAS
Developmen t eris-00.04 kernel / 2.85
MiB initrd.img-
2.6.35.4omn
ibot-00.04-
20111013 /
3.22 MiB ext2 20 s
*This does not include the system POST
Since the two disks with the Veisha OS had such different boot times, the performance of each disk was analyzed and compared to other devices in the lab. The disks were tested on the robot using the “hdparm” utility. The robot was running the Development OS with the eris-00.04 kernel.
Command : hdparm -Tt [device] example: hdparm -Tt /dev/sda
Device Capacity Interface Cached Reads Buffered
Reads
Current Use
Toshiba
MK8025GAS
80GB ATA / UDMA-33* 471.70 MB/s 26.89 MB/s Development
OS
Edge 512 CF 512MB ATA / UDMA-33* 476.09 MB/s 16.49 MB/s Veisha OS
Edge SDCFB-128 128MB ATA / PIO-4 312.73 MB/s 2.73 MB/s
SanDisk Cruzer 8GB
Kensington DT
Mini Fun G2
PNY Attache
8GB
8GB
USB 2.0
USB 2.0
USB 2.0
Misc. 512MB 512MB USB 2.0
471.82 MB/s 15.68 MB/s
480.13 MB/s 19.53 MB/s
482.90 MB/s 25.05 MB/s
469.22 MB/s 15.42 MB/s
Debian Live
Veisha OS
Lexar
Jumpdrive
Elite
128MB USB 2.0 437.93 MB/s 8.32 MB/s
*The ETX-N270+Pluto supports ATA-6/133 and the Toshiba disk supports
ATA-5/100 but the bus is limited because a 40 wire cable is being used.
The 512MB CF card supports ATA-5/100, but the CF socket on the motherboard has a 40 wire interface as well.
To verify these results, a second set of tests were performed on a Dell
Latitude D430 running Debian 6 Linux. The included “Disk Utility”

program was used to benchmark the USB disks' performance. The Toshiba hard disk was tested in an external 2.5” ATA/USB 2.0 enclosure. The CF cards were not included in this test because the card reader in the lab is a USB 1.1 device and would not provide adequate performance.
Device Average Read Performance Average Latency
Misc 512MB USB 16.8 MB/s 0.6ms
Sandisk Cruzer 8GB 17.7 MB/s
Kingston DT Mini Fun G2 21.0 MB/s
0.8ms
0.7ms
PNY Attache 8GB 34.4 MB/s 0.6ms
Toshiba MK8025GAS 24.8 MB/s 20.5ms
Given that the two 512MB disks have similar throughput and latency, their interfaces seemed to be responsible for the performance difference. It is worth noting that the disk connected to a USB port takes several times longer to load the kernel and initrd image than the compact flash card. Connecting the Toshiba hard disk to the robot through USB produces a boot time of 48 seconds. The Debian Live system utilizes the Syslinux boot loader whereas the Development OS and Veisha
OS use GRUB. The Syslinux loader seems to have higher throughput
“early” in the boot process.
Building the functionality of the OS was the primary objective, followed by efforts to make the system more robust. Copying the 4 th generation kernel and initrd image to the 512MB CF card will reduce the amount of data that must be loaded into memory. The robot's current operating system was installed with a minimal feature set, but it is still loading unnecessary components at boot time that could be disabled. Purchasing a high performance compact flash card may improve the boot time as well.
When the Development OS was created, the ext2 file system was selected for simplicity. Given the robot's performance capabilities, a more robust journaling file system (such as ext3 or ext4) could be used to improve the robot's resilience to unexpected shutdown conditions without impacting performance. Further development with Debian Live systems will be pursued as well.
For more detailed information about the development of the robot's operating system, see the Appendix or reference the following documents

on the SVN server:
Debian Live
First run with Debian Live
Integrating robot-specific items into Debian Live live-manual.portrait-letter.en (third party)
Configuring / fixing Eris' boot loader(s) boot problems.txt emergency console 2- CF WARZ grub emergency console
Size reduction / “embedding”
Creating a Squashed FS on the CF Card - Oct 5 2011 squashconsole
Trimming Eris OS for Veisha ibm_ramdisk_script
Building / modifying initrd images
Making initrd.img and configuring grub console output for initrd writeup making initrd images newc (third party) ramfs-rootfs-initramfs (third party, very good)
Kernel Information eris-00.01-20110305 notes eris-00.02-20110318 notes eris-00.03-20110324 notes eris-00.04-20111013 notes
Installing Modules Examle (kernel 00.04)
Installing Modules on Eris and starting up Wifi kernel build errors
Hardware Information:
Chips on Pluto Expansion eris network info
I-O Ports, IRQs and Memory development notes (from TS-ADC16 tests)