Selection of Real-time Linux Versions, Hardware, and Test Software
Deliverable for Center Initiative “Analysis and Test of Real-Time Linux Operating Systems '02”
Kalynnda Berens, Principal Investigator
Terminology:
Linux is an open source operating system kernel designed for a multi-user platform. It is the “free”
equivalent of the UNIX operating system (OS). The term “Linux” is commonly used to encompass both
the kernel and the set of software applications produced by the Gnu project (Free Software Foundation).
This operating environment should be referred to as Gnu/Linux.
Standard Linux is the product of a development team lead by Linus Torvalds. Because Linux is open
source (i.e. the source code is freely available), many variations exist. These versions attempt to meet a
specific need in a particular environment or provide enhancements.
Distributions are particular combinations of Gnu/Linux software, perhaps with custom software added to
the mix. Red Hat, SuSE, and other “major players” produce distributions for desktop and server systems.
Distributions exist for embedded systems as well.
The term embedded refers to systems that are (usually) not user-interactive. Embedded systems usually
control hardware, are command-driven, and have limited (or no) user peripherals, such as a keyboard,
mouse, or monitor. Embedded systems range from small and simple to large and complex, and run on 8,
16, 32, and even 64 bit processors.
Real-time systems may or may not be embedded systems. Real-time adds the requirement of timeliness
to the system. If an event, calculation, or other action is late, it is wrong. The timing does not have to be
fast – a real-time system could have the requirement to execute a task exactly at midnight each day.
Hard real-time systems have absolute deadlines that must be met. Soft real-time systems have a time
tolerance within which the event can occur.
Overview:
In recent years, the Gnu/Linux operating system has been extended into the embedded and real-time
environments. For embedded systems, the Gnu/Linux OS has been reduced in size. The exact
configuration differs among distributions. The versions of real-time Linux vary according to the
philosophy chosen. Real-time is created by:
1. A separate real-time kernel, with Linux run as a low-priority task.
2. Modifications to the kernel, such as the scheduler and timing
3. A combination of the above.
The purpose of this Center Initiative is to characterize several versions of real-time and embedded Linux,
for possible use in future NASA projects. A secondary purpose is to provide viable alternatives to
expensive real-time operating systems, such as VxWorks. This deliverable details what versions of Linux
will be tested, and the hardware they will run on.
Over 30 real-time or embedded versions of Linux were found. Some were complete systems (i.e.
Gnu/Linux), others were just versions of the Linux kernel. Real-time and embedded Linux versions range
from single-developer versions to expensive commercial ones, with many in between. The table at the
end of this document lists all the versions of real-time or embedded Linux that were considered.
The figure below is taken from an article by Kevin Dankwardt, of K Computing, “Comparing real-time
Linux alternatives”. It covers most of the major real-time Linux versions, but not all.
The selection criteria for real-time/embedded Linux are:

Project maturity – Is the version far enough along to be useful? Is the project maintained by a
team or commercial company?

Applicability – Is the version specialized or general?
processors?

“Market Share” - Is the version being used currently in projects? Is it well known enough that
projects have heard of it and are considering using it?

Cost – What is the price for the Linux version?

Extras – Does the version have any useful extras? Does it provide any additional safety,
reliability, or other factors?
Does it run on multiple embedded
There is a large price gap between “free” real-time Linux versions and most commercial versions. Most of
the low-cost or free versions came from University or government research. Many of the commercial
versions start with a free version and then add proprietary software. Others produce a proprietary kernel,
and then run Linux over it. Because the research funds to purchase hardware and software are limited,
several commercial vendors cannot be considered at this time. However, free version that use the same
or similar methods to achieve real-time performance are available.
Below is a list of the real-time Linux versions under consideration, along with the criteria information. The
“Market Share” and “Project Maturity” criteria have already been applied, to produce this list. All of these
versions of real-time Linux are well established, well known, and well supported. An additional selection
factor is to choose the versions such that a sampling of the various methods of achieving real-time is
included.






RT-Linux, from FSM Labs. This variant includes a separate real-time, minimalist kernel. Linux is
run as a low-priority task. Real-time programs cannot use Linux device drivers. Claims hard realtime performance. One problem with this approach is that real-time tasks run in kernel space and
are not protected from malfunctioning applications. Linux virtual memory processes protect user
applications by limiting the ability to write outside of an application’s memory space.
o
Applicability:
This is one of the best-known versions of real-time Linux. Runs on
PowerPC, x86, and Alpha processor families.
o
Cost:
Nominal
RTAI (with LXRT), from DAIPM in Italy. Like RT Linux, this variant uses a separate real-time
kernel. RTAI includes more functionality on the real-time side than RT-Linux. It also includes
LXRT, which allows some real-time functions to operate in user mode (and therefore be protected
from other applications).
o
Applicability:
PowerPC.
This is another well-known version of real-time Linux. Runs on x86 and
o
Cost:
Free
KURT Linux, from University of Kansas. Modified Linux kernel for real-time scheduling and
microsecond timing. This is a “firm” real-time system (better than soft, but not able to meet all the
demands of a hard real-time system).
o
Applicability:
While university developed, this variant has been around a while and has
been used in several projects. It’s method of achieving real-time differs from RT-Linux
and RTAI.
o
Cost:
Free
Rapid Response Linux, from Munich University. This variant is fairly new. It uses the same
timing improvements as KURT, and adds a low-latency patch. Preliminary tests show good realtime performance.
o
Applicability:
The method this version uses to achieve real-time performance differs
from RT-Linux and RTAI. The one concern is that it is very new and mostly untested in
real applications.
o
Cost:
Free
RED Linux, from UC Irvine. This version modifies Linux scheduling. It adds modifications for
short kernel preemption and a high resolution timer. REDICE Linux is a commercial product
based on this version of Linux.
o
Applicability:
This university-created real-time variant modifies the kernel. It differs
from other approaches by adding the ability to preempt kernel processes to improve
timing.
o
Cost:
Free
HardHat Linux, from MontaVista. This version includes a preemptible kernel and other
modifications to achieve real-time performance. It uses the RT-Linux approach, along with kernel
modifications. Like RED Linux, it includes kernel preemption.
o
Applicability:
This is a commercial, well known product that has been used for several
years. It draws on the open source products, and adds unique features.
o
Cost:
$25000 for the professional version. The Journeyman license provides
the software for free, for evaluation purposes, and with no support.




TimeSys Linux, from TimeSys. This commercial variant uses RTAI for real-time, and includes
other enhancements.
o
Applicability:
This commercial product has been used for several years. It draws on
open source and also adds new features.
o
Cost:
version only.
$25000 for full version, $15000 for real-time only, free for embedded
Embedix, from Lineo. This commercial version uses RTAI as it’s real-time core. In addition,
Embedix provides many features useful for embedded development.
o
Applicability:
This is a well-known commercial product. It supports multiple processor
families. It has been used in many projects in various industries.
o
Cost:
$3000 development license + $1500 for board support package.
REDICE Linux, from REDsonic. This commercial variant is based on RED Linux. It continues the
development of that version, and adds a predictable system scheduler.
o
Applicability:
This is a commercial product that has been used in various projects. It
supports multiple processors.
o
Cost:
$5000 for a developer kit that includes hardware (Version 2). Version 1
is essentially RED Linux and is free (no support).
OnCore OS + Linux, from OnCore. This commercial version provides a separate real-time OS,
upon which Linux can run. The OS runs in a virtual mode, where all tasks are protected from
other malfunctioning tasks. RT-Linux and RTAI real-time kernels are not protected. This OS uses
a different protection mechanism from VxWorks AE.
o
Applicability:
This OS may provide outstanding protection from malfunctioning
processes. While it is “one step removed” from Linux, the ability to protect kernel
processes, much like the new VxWorks AE, is an important ability for software safety.
o
Cost:
$9500 development license + $2000/year support. OnCore may waive
the license cost for the purposes of this research.
Selected Real-time Linux Variants
Variant
Rationale
RT-Linux
This is one of the most common versions of real-time Linux in
use. It uses a separate real-time kernel, with Linux as the
lowest-priority task. RT-Linux runs on several processor
families.
RTAI
Like RT-Linux, uses a separate real-time kernel. Adds ability to
do some real-time tasks within user space. More functionality
within the real-time kernel than RT-Linux.
KURT or
Rapid Reaction Linux may be too new to be useful. KURT will
be used if that proves to be the case. Unlike RT-Linux or RTAI,
these variants modify the Linux kernel to provide real-time
capabilities. Rapid Reaction includes the preemptible kernel
patch.
Rapid Reaction Linux
OnCore OS + Linux
Contingent on OS provided for
evaluation purposes at reduced cost.
While the “Linux” part of this OS is limited, the basic OnCore
OS promises to provide superior protection capabilities for
software applications, perhaps even better than VxWorks AE
(and at 1/3 the cost).
Variant
Rationale
HardHat Linux
HardHat combines multiple philosophies, such as scheduler
and timing modifications to the Linux kernel, plus making the
kernel preemptible.
Journeyman version (fully functional,
free for evaluation) will be used unless
the Professional version is provided for
evaluation at reduced cost.
HardHat includes many of the same modifications as RED
Linux and REDICE Linux.
Hardware Selection
The embedded hardware used to test the selected real-time Linux variants must be able to run all or most
of the variants. Two processor families will be used to prevent hardware factors from skewing any results.
The processor families most commonly supported are Intel (80386 and above) and PowerPC. Within the
Intel family, some of the variants support 80386 and 80486 processors, but a few only support Pentiums
and above. Therefore, the Intel embedded board will have a Pentium or better processor.
Within the PowerPC family, the OnCore OS had the greatest limitation. Some PowerPC processors had
Memory Management Units (MMUs) that were not adequate for the OS. The PowerPC 405 processor is
supported by all the variants that support PowerPCs (OnCore OS, HardHat, RTAI, and RT-Linux).
Selection of the hardware is based on meeting the above processor criteria and on cost. The two boards
selected are:

EP405 from Embedded Planet, with PowerPC 405GP processor on a PC/104 form factor

SBC2590 from Micro/Sys, with Pentium processor and PC/104-Plus form factor.
In addition to the embedded hardware, a development system is required. This system will be a nonODIN machine running a standard Linux distribution. Many options exist, both in desktop systems and
notebooks. The final selection will be made closer to the time of purchase, as many models are
discontinued and computer prices usually keep falling with time .
Test Applications
Preliminary research has uncovered some useful programs and tools for testing real-time Linux. Most are
free. The actual test harness that will be created as part of this research will draw on these tools as much
as possible.
Testing-related tools for real-time Linux include:

Dynamic Probes from IBM, for kernel debugging (free)

Linux Tract Toolkit provides information on events within the kernel (free)

strace is a system call tracer (free)

System Call Tracker traces system calls as well (free)

Linux Standard Base Test Suite tests conformance to various standards (free)

The Linux Test Project provides many tests for Linux (free)

StethoScope is real-time data monitoring program ($2500)
Budget
$12000 was budgeted for purchase of the required software and hardware. The selected hardware and
software falls well within that budget. The breakdown of the selected software and hardware is as follows:



Software
$3000 maximum
o
RT-Linux $150
o
RTAI
o
KURT or Rapid Reaction Linux (free)
o
OnCore OS $2000 for support, software at no charge (regularly $9500)
o
HardHat (free) for Journeyman, possible nominal charge for Professional
(free)
Embedded Hardware
$3000 maximum
o
EP405 $1000
o
SBC2590 $1400
o
Miscellaneous cables, power supplies, and other hardware will be necessary.
Development System
$3000 maximum
Real-time and Embedded Linux Variants
Version
Source
Embedded Kernel
?
Only?
n/a
Yes
Description
Cost
ETL/Japan
Realtime?
Yes
ART Linux
Free, in beta testing
LynuxWorks
No
Yes
No
Patch to Linux kernel. Inspired by RT-Linux, adds:
source level compatibility of device drivers, no priority
inversion, binary compatibility of user programs with
regular Linux, interrupts as periodical events, real-time
task execution at user privilege level to protect tasks.
Multiple processors, tool set.
BlueCat
Coollinux
Coollogic
DWCS Linux GA Tech
No
Yes
Yes
No
No
Yes
ELKS
Open Source
No
Yes
Yes
Embedded
Debian
Embedix
Open source
No
Yes
No
Lineo
Yes
Yes
No
Embenix
Dasan
No
Yes
?
Emblin
EmLinux
ETLinux
Open source
Open source
Prosa SRL
No
No
No
Yes
Yes
Yes
Yes?
No
Yes?
Hard Hat
MontaVista
Yes
Yes
No
High
Availability
Linux
Jailbait
Motorola
No
?
?
Not free, did not get
quote.
Designed for Internet Appliances
Free?
Dynamic Window-constrained Scheduling. Scheduler for Free
Linux that provides real-timer performance. i386+ only.
Embeddable Linux Kernel Subset, a project to build a
Free
small kernel subset of Linux that can run on machines
with limited processor and memory resources. For nonMMU machines.
Project to create an embedded version of the Debian
Free
distribution. Very early stage (no software yet).
Uses RTAI. Multiple processor families.
$3000 development
license. $1500 per
BSP
Designed for specific processor boards, but applicable to Free
PowerPC, ARM, and StrongARM processors
Very small version, i386+ only. 1 developer.
Free
Very low memory footprint. i386+ only.
Free
Etlinux is a complete Linux-based system designed to
Free
run on very small industrial computers.
Runs on x86, PowerPC, StrongARM, MIPS, SH, ARM,
$25000 for
and Xscale microprocessors.
Professional.
Journeyman (no
support, limited
processors) is free
Designed for the telecommunications industry. Tied to
Free?
specialized hot-swappable hardware.
Open source
No
somewhat
No
Medium-sized distribution designed for Internet
Appliances.
Free
Version
Source
Realtime?
Yes
Embedded Kernel
?
Only?
Yes
Yes
KURT Linux
University of
Kansas
Lem
Open Source
No
Yes
No
Linux/RK
Carnegie Mellon
University
Ucambridge,
AT&T
Transmeta
Open source
Neoware Systems
Yes
No
Yes
Yes (soft) na
Yes
No
No
No
Yes
Yes
Yes
No
No
No
Yes
Yes
No
PeeWee
Open Source
No
Yes
Linux
Pocket Linux Open source
No
Yes
Q Linux
AT&T, UT, UMass Yes (soft) na
No
QS Linux
QuickStack
Yes
No
Rapid
Reaction
Linux
Red Blue
Linux
RED Linux
Munich University Yes
na
Yes
Esfia
No
Yes
No
UC Irvine
Yes
Yes
Yes
REDICE
Linux
REDsonic
Linux-SRT
Midori Linux
muLinux
NeoLinux
OnCore OS + OnCore
Linux
No
No
Yes
Description
Cost
Modified Linux kernel: Scheduler and timing (UTIME
patch). Firm real-time system. i386+ only – Pentium or
higher
A small version of Gnu/Linux. Includes networking and
graphics server.
Uses a “resource kernel” approach. Fairly theoretical at
this point. i386+ only
Assigns percentage of CPU to each task (rate-based
scheduling). Also has patch to Xfree86 for graphics
Small Gnu/Linux distribution. i386+ only.
Small version of Linux (< 2 MB), i386+ only.
Gnu/Linux version targeted at “appliance” applications
(internet appliance, etc.) – dedicated machines.
Separate OS that uses virtual memory to protect
processes from each other. Linux runs on top of the
realtime OS.
Small Gnu/Linux. Project focuses on improving
configuration and installation on embedded devices.
Designed for Web/Internet access mostly.
Designed for multimedia. i386+ only. Updated scheduling
algorithm.
Designed for QuickStack hardware, may run on other
systems.
Modified Linux kernel: low latency patch plus UTIME
patch for timing. Test results show combination gives
very good real-time behavior. i386+, Pentium+ only.
Designed for wireless “internet appliances”.
Free
Free
Free
Free
Free
Free
Free
$9500 + $2000/year
support
Free
Free
Free
Free
Free
Free
Modifies Linux scheduling, adds short kernel preemption, Free
high-resolution timer.
Based on RED Linux. It has a high precision timer, short about $5000 for
preemption kernel, and predictable system schedulers.
developer kits,
version 2. Version 1
is free.
Version
Source
FSM Labs
Realtime?
Yes
Embedded Kernel
?
Only?
Yes
No
RT-Linux
RTAI
DIAPM
Yes
Yes
Yes
Super Linux
2000
Thin Linux
e-CAT
No
Yes
No
Open source
No
Yes
No
TimeSys
Linux
TimeSys
Yes
Yes
No
uClinux
Description
Cost
Separate real-time kernel. Real-time programs use the
kernel, not Linux (including device drivers). Claims hard
real-time performance. Kernel is flat memory model (no
protection).
Like RT-Linux, separate real-time kernel. More
functionality in kernel. Also, with LXRT, adds real-time
capability in user (protected) space.
Network-oriented embedded distribution.
$150 for open source
Minimalist Gnu/Linux system, with capability of selecting
“packages” to include.
Uses RTAI for real-time. Added “Reservation Module”
that keeps CPU “space” open for an event to use (such
as unexpected processing required). Other modifications
as well.
Designed for non-MMU processors (microcontrollers or
small microprocessors).
Free
Not Free. Did not get
quote.
Free
Free for embedded
only, no support.
$15000 for real-time
version. $25000 for
version with
Reservation Module.
Free