Advanced Topics
Cleanroom Software Engineering
Quality Assurance
CSE 495
Maj Brian Hermann
Cleanroom Overview
What is it?
Motivations
Activities
Phased Implementation
Concerns
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 2
Cleanroom - What is it?
A method for developing software with known and
predictable reliability.
Combines rigorous methods of software
specification, design, correctness verification, and
statistical quality certification in a new life cycle
based on incremental development.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 3
Cleanroom--What is it?
(cont)
A software development philosophy which is based
on static verification techniques. The objective of
this approach to software development is zerodefect software.
Based on the notion that defects in software should
be avoided rather than detected and repaired. It
relies on a strict, formalized inspection process to
discover faults before the software is tested.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 4
Cleanroom Motivations
Software with known Mean Time to Failure
Software with near zero defects
Quality software which is cheaper to produce
Dyer 92
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 5
Cleanroom Activities
Formal Specification
Increment Planning
Formal Design
Correctness Verification
Statistical Testing
Software Reliability Measurement
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 6
Formal Specification
Formal methods (box structuring techniques, Z,
VDM, etc.) force a more exacting analysis of the
requirements and tend to minimize ambiguity,
inconsistency, and incompleteness in the resultant
software specification.
Two specifications:
– Functional
– Usage
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 7
Increment Planning
The software is partitioned into increments which are
developed separately using the cleanroom process.
An increment is a useful, albeit limited, system in its
own right.
Users can feed back reports of the system and
propose changes that are required.
Important because it minimizes the disruption
caused to the development process by customerrequested requirements changes.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 8
Formal Design
A rigorous and formal design method is a necessary
element for generating software whose correctness
can be verified.
A structured programming process, using only a
limited number of control and data abstraction
constructs, is recommended.
The program development process is a process of
stepwise refinement of the specification.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 9
Correctness Verification
Correctness is defined as the equivalence between a
requirement and the design, documented with the design
primitives, which supposedly implements the requirement.
Designs are verified first by the designer when constructing
a design and subsequently by independent inspectors when
reviewing the design.
With some algebraic manipulation, the question of
correctness for a total software product can be reduced to
the summation of the correctness proofs for the component
parts.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 10
Statistical Testing
Statistical testing is a software testing process in which the
objective is to measure the reliability of the software rather
than to discover software faults.
– Determine the operational profile
– Select or generate a set of test data corresponding to the
operational profile
– Apply these test cases to the program, recording the amount of
execution time between each observed system failure
– After a statistically significant number of failures have been
observed, the software reliability can be computed.
Testing the software the way users intend to use it.
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 11
Statistical Testing Difficulties
Operational profile uncertainty
– Difficult to anticipate usage for new or innovative systems
– System usage can change over time
– Many systems have many improbable, but not impossible,
conditions
High costs of operational profile generation
Statistical uncertainty when high reliability is specified
– It may be hard to generate (especially automatically) test data to
create rare, but valid, inputs
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 12
Software Reliability
Measurement
Software quality is often represented in errors per
thousand LOC (KSLOC)
– This is useful to the developer, but of limited value to
the user
Software reliability, in terms of mean time to failure
(MTTF), is a more meaningful measure for the user,
which gives a positive quality indicator (longer MTTF
is better) and which can be estimated prior to
software delivery.
Dyer 92
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 13
Phased Cleanroom Implementation
1.
2.
Configuration
Managed
Software
No Developer
Testing
3.
Correctness
Verification
Baseline
Process
Statistical
Testing
Reliability
Measurement
Formal
Specifications
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Statistical
Process
Control
Dyer 92
Slide 14
Cleanroom Concerns
Too mathematical and not usable in an environment
of changing requirements
Developer unit testing is an essential step in
software development and can not be eliminated
Randomized testing was neither a cost effective
method nor could it provide adequate requirements
coverage
Distrust in the concept of a software MTTF and in its
role in the software development process
Dyer 92
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 15
Cleanroom - Reasons for
Slow Growth
A belief that correctness verification is too theoretical
to be usable in real software development
It advocates no developer testing and replaces it
with correctness verification and statistical quality
control -- concepts opposite of how most software is
developed today
Maturity of the software development industry
Henderson 95
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 16
Summary
What is it?
Motivations
Activities
Phased Implementation
Concerns
CSE 495
Lesson 14– Tuesday (4 Dec 2001)
Slide 17