Cleanroom Software
Engineering
By Derek B. Larson
Cleanroom Software Engineering
► What
is Cleanroom Software Engineering?
► Cleanroom Process
► Waterfall Model into a Cleanroom
► Case Studies
Introduction
► Developed
IBM
► The stated goal of Cleanroom is software
that exhibit a zero defect rate.
 mathematical and statistical methods
► IBM
developed a device controller product
using Cleanroom Software Engineering that
had zero failures in three years used at
300+ locations.
[1]
Intro Cont.
Hardware cleanrooms
keep problems out by
keeping potential
contaminating factors
from reaching the
product.
Why Cleanroom SE
► The
reason to use Cleanroom Software
Engineering is simple: quality
► Quality improvements of 10 to 20 times
have been reported when the Cleanroom
process was demonstrated in industry
► If defects can cause loss of life or critical
financial loss
► Increases in productivity
Why Cont.
► Increases
in productivity?
 Cleanroom is a uniquely defined process that
decreases required testing, error correction, and
maintenance.
 These savings offset any additional overhead
needed for the quality control
 “Errorless Software Development”. If no errors
enter development, no errors need to be tested
or debugged. Elimination of testing and
debugging means faster product development.
Reqs for Cleanroom SE
►
►
►
Most spend a lot of time and money at the
start of the project for preventing defects.
Most use statistical methods to ensure
quality.
Most formally state and “prove”
requirement needs. (YEAH ‘Z’)
[2]
Cleanroom Functions
► Cleanroom
uses three types of functions
► All code that is developed will follow the
basic structure of these functions.
► Because the functions are sound, the code
will likewise be sound.
► These functions are called “Boxes”
[1]
The Three Boxes
► Black
Box
► State Box
► Clear Box
Black Box
►Black
box is a view of an object that
hides the implementation process and
data .
►By modeling code as a series of black
boxes, we can ensure its quality verses
our specification by ensuring that the
actual black box performs according to
the black box definition.
[1]
Black Box Cont.
►It
will describe how that system will
respond to stimuli; this is done usually
in a formal specification language.
 Z (Zed)
Picture from http://uebb.cs.tu-berlin.de/zeta/zetadistdoc/Z-img13.gif
State Box
► State
box is where the view of an object
shows the data implementation, but does
not show the implementation process
► It describes how state information is being
transformed
► In essence, the “history” of the black box is
replaced by an existing “state”.
[1]
State Box Cont.
► This
is an abstraction of the history that
allows us to take a higher-level view of the
system
► Must ensure that there is no “history” case
that is unaccounted for.
[1]
Clear Box
► Clear
box shows both the data and process
implementation
► The goal is to stepwise refine functions and
prove them as being correct.
► Clear boxes show what is actually necessary
to go between the old state and the new
state.
[1]
Clear Box Cont.
► Sometimes
there are multiple paths or
multiple states that can result from a state
box – the clear box lets us examine and
design these transitions with flow control.
► In the clear box, the procedures required to
implement the state box transition function
are defined explicitly.
[1]
Cleanroom Approach
► Requirements
Analysis
 Producing and reviewing “informal
specifications.”
► High-level
Design
 Converting the requirements into state
machines and functions
► Detailed
Design
 Further refinement of functions
[3]
Cleanroom Approach Cont.
► Coding
by increment
 Developing code and verifying it using formal methods.
 Compiling code or unit testing is prohibited
► Pretest
by increment
 Generation of test cases
► Statistical
Testing by Increment
 The code is compiled, linked, and tested. Then the
results are validated.
[3]
Cleanroom Approach Cont.
► Cleanroom
Software Engineering prohibits
unit testing
► In a Cleanroom development, correctness
verification replaces unit testing and
debugging
► After coding is complete, the software
immediately enters system test with no
debugging.
[3]
Cleanroom Approach Cont.
► All
test errors are accounted for from the first
execution of the program with no private testing
allowed
► The role of system testing in the Cleanroom
process is to certify the quality of the software
with the systems specifications in mind.
► Not doing unit testing can only be done if the
above requirements are followed, that way many
of the defects are already found and fixed, so
when the system is done coding it should be close
to no defects.
[3]
Correctness Verification
► Once
a piece of code is developed it goes through
the Correctness Verification process.
► correctness verification phase takes the developed
code and compares it against the specification to
see if it really does what is outlined in the
specification.
► The specifications define the conditions that code
must meet in order to fulfill the function for which
it was developed.
[1]
Correctness Verification Cont.
► Correctness
verification uses functiontheoretic static code analysis to do just that.
► The term ‘function theoretic’ implies that
there is a one-to-one mapping between the
code and the specifications.
Correctness Verification Example
function isNumeric(char c)
{
if ((c >= ‘0’) and (c <= ‘9’))
return true;
else
return false;
}
Example Explanation
► If
the character passed in is in the set
[0,1,2,3,4,5,6,7,8,9] we expect the function to
return true.
► Based on what we know about character sets and
the language used to develop the code if the
character is in the set then the logic ((c >= ‘0’)
and (c <= ‘9’)) will return true.
► When the character is not in the set then the logic
will return false.
► Both cases result in the expected behavior for the
entire method, and so the code passes the
correctness verification. [1]
Statistical Usage Testing
► In
conjunction with the box structure
specifications in the pre-development phase,
usage specifications are created for the statistical
testing phase.
► Usage specifications are simply descriptions of
how the system will be eventually used.
► Usage models need to be defined for all possible
scenarios for a given piece of code along with the
probabilities of each scenario occurring. [1]
Statistical Usage Testing Cont.
► Use
Markov Chains
► Markov chains are essentially directed
graphs with nodes as states of use and arcs
as the stimuli that cause state transitions.
► This is the most efficient way to test
software, since the most destructive
problems will be eliminated first, and money
will not be spent on potentially harmless
problems if it is not available. [1]
Waterfall Model into a Cleanroom
► Waterfall
model…we all know what that is
► We want to take the Cleanroom model and
add some milestones to the model.
Waterfall Model into a Cleanroom
Milestones
► Software
Specification Review
 (Historical) Addresses requirements and
external interfaces
 (Cleanroom) Remains the same with increment
plan-mapping requirements to increment.
► Preliminary
Design Review
 (H) Top-level architecture
 (Cr) Top-level architecture updated as needed in
later increments
Waterfall Model into a Cleanroom
Milestones Cont.
► Critical
Design Review
 (H) Detailed Design
 (Cr) Detailed Design of functionality for
particular increment
► Qualification
Test
 (H) Verify requirements
 (Cr) Verify requirements. Performed on final
increment. Earlier increments have informal QT
Case Studies
► STARS
► Tank-automotive
and Armaments Command
STARS
► Developed
by the Department of Defense
► STARS receives radar data and flight plan
information and presents the information to
air traffic controllers on high resolution, 20"
x 20" color displays allowing the controller
to monitor, control, and accept hand-off of
air traffic [4]
STARS Cont.
STARS is capable of tracking
up to 1350 airborne aircraft
simultaneously within a
terminal area. The system
interfaces with multiple radars
(up to 16 short and long
range), 128 controller
positions, 20 remote towers,
and a 400 by 400 mile area of
coverage. [4]
STARS Cont.
► The
“STARS” program emphasized on three main
points
 Process-driven
 Re-use based
 Integrated software engineering environment
► STARS
evaluated current "traditional" processes.
Then determined that a quality management
philosophy (putting decision making in the hands
of workers, focusing on processes, quantitative
measurements) is critical and that Cleanroom
Software Engineering follows this philosophy [4]
STARS Cont.
► Cleanroom
was combined with the TRW
(spiral) Ada Process Model
► Produced software at a rate of $30-40 per
line of code versus industry averages of
$130 per line for software of similar
complexity and development timeframe (the
size of the application is greater than 300
KLOC) [4]
STARS Savings
► $130
- $30 = $100 per line of code
► The project is around 300K lines of code
► So, $100 * (around)300K = (around)
$30,000,000
► Could buy 30, million dollar
houses
► Or about one month of Alex
Rodriguez playing baseball
[4]
Tank-automotive and Armaments
Command
► TACOM
generates, provides, and sustains
mobility, lethality, and survivability for
soldiers, other U.S. Armed Services, and our
allies - all to ensure Army readiness today,
tomorrow, and beyond
Tank-automotive and Armaments
Command Cont.
► After
seven project increments
(approximately 90K lines of code)
► 4.2:1 productivity increase
► 20:1 return on investment has been
documented
► Projects experienced an overall testing error
rate (represents all errors found in all
testing) of 0.5 errors/KLOC [4]
Summary
► Cleanroom
software development may be a
wonderful advance in the process of software
development or may just be a downright weird
approach, most likely a little of both.
► Looking at Cleanroom from a theorists’ point of
view Cleanroom provides a theoretical foundation
to software development in its use of
mathematically based software development and
statistical quality control.
Summary
► By
not introducing errors into the
development phase there should be no
testing requirement in the process.
► Statistical testing provides the benchmark
as to performance and failure rate and helps
verify the inputs to the development
process by checking its outputs.
On the other Hand
► Many
people sees Cleanroom as too radical a
departure from conventional software
development.
► The level of experience and training required to
have a functional team of Cleanroom developers
may not cost effective
► Developers develop code, and Cleanroom is more
about specifications and statistical models than
coding.
In the End
► One
has to understand the appropriate use of
Cleanroom software development
► Cleanroom is suitable for very particular types of
software where the human and financial risks of
having errors are too great to be left to chance
► This generally does not fit the mold of mainstream
software development, in which the concentration
is often on getting the best price in the best time
period.
► From what has been shown, Cleanroom is
anything but a mainstream development process.
References
1.
2.
3.
4.
Cleanroom Software Engineering.
http://www.criticaljunction.com/werbicki/SENG623/Group
/SENG623W03_Cleanroom.pdf
Uta.edu, Cleanroom Software Engineering. Found at:
www.uta.edu/cse/levine/fall99/cse5324/cr/clean/page.ht
ml
DACS, The Data and Analysis Center for Software.
Found at:
www.dacs.dtic.mil/databases/url/key.php?keycode=64
Carnegie Mellon, Software Engineering Institute. Found
at: www.sei.cmu.edu/str/descriptions/cleanroom.html