Progress in Testing
Component-Based Software
Craig H. Wittenberg
Component Applications Group
Microsoft Research
Talk Outline
Our project
Demos
Component and composition
Progress in testing components
Areas for further research
Aug. 23, 2000
ISSTA 2000
2
High Level Goals
A system of flexible components for a
family of products
Faster and cheaper application
assembly and evolution
Reliable, efficient, and consistent
applications working together
Aug. 23, 2000
ISSTA 2000
3
Octarine Demo
Prototype written in C
Looks familiar
Adding Bullets, Music
Followed by architectural drawings
Aug. 23, 2000
ISSTA 2000
4
Octarine Text Architecture
ISequence
Story
ISequence
Text
ISequence
Music
Table
ISequence
Text
Aug. 23, 2000
ISSTA 2000
5
Adding Bullets/Numbering
ISequence
Story
ISequence
Text
ISequence
Music
ISequence
ISequence
Bullets/
Numbering
Text
Table
ISequence
Text
Aug. 23, 2000
ISSTA 2000
6
Landscape Demo
More recent
Many small components
Static and dynamic connections
Multiple uses of:

Text, Layout, Grid, Data pipes
Architecture for many Uis
Small devices
 Non-visual interfaces

Aug. 23, 2000
ISSTA 2000
7
Our Approach
Factoring and composition
Precise interface specs
Significant design and prototype step
Huge emphasis on testing
Control inter-component dependencies
Tools support (design, build, and run time)
Critical mass of designs, components,
frameworks, large demos
Aug. 23, 2000
ISSTA 2000
8
Component
“A software component is a unit of
composition with contractually specified
interfaces and explicit context dependencies
only. A software component can be
deployed independently and is subject to
composition by third parties.”
Szyperski, Component Software, pg. 34.
Key points: Contracts, Composition,
Dependencies, Deployment
No observable global state
Aug. 23, 2000
ISSTA 2000
9
Component DLL
Binary package plus docs
No top-level APIs
Contains one or more classes





Implementation of contracts only
No implementation inheritance
Realize behavior of component
Instances of classes are connected
Dependencies upon


Aug. 23, 2000
other classes
services from other instances
ISSTA 2000
10
Precise Interface Specs
a.k.a. Software Contracts
An interface has:




Name (compile time and runtime)
Methods with pre/post conditions
Abstract model with invariants
State table
Specified separately from implementation
Implementation does the obvious and:

maps model variables to internal state
Never changed once published
Aug. 23, 2000
ISSTA 2000
11
ILight Methods and Model
ILight, {2461A1A0-639E-11d2-874D-00AA0060FCA9}
TurnOn();
 TurnOff();
 SetIntensity(int desired);

Model
boolean on;
 int intensity; // range is 0..100.

Aug. 23, 2000
ISSTA 2000
12
ILight Pre/Post Conditions
TurnOn():

Post: on == true.
TurnOff():

Post: on == false.
SetIntensity(desired):
Pre: desired >=0 && desired <= 100.
 Post: intensity == desired.

Aug. 23, 2000
ISSTA 2000
13
A Light
The Light class implements ILight
Contract-specific and implementationspecific model mapping
ILight
Light
Map
ILight
model to
Impl.
Aug. 23, 2000
ISSTA 2000
14
Record Filter
Filter processes one set of records into
another
IRecordSet
IRecordSet
Record
Filter
IString
Map
IRecordSet
model to
Impl.
Aug. 23, 2000
Plants
ISSTA 2000
Filter
Spec
15
Light Switch
Converts calls to push() into on() and
off()
IPushIt
ILight
Light
Switch
A light
Map IPushIt
model to
Impl.
Aug. 23, 2000
ISSTA 2000
16
Closure
Executes closure over parameters

data or connected objects
IUnknown
IDoIt
…
Closure
Parameters
…
Aug. 23, 2000
ISSTA 2000
…
Map IDoIt
model to
Impl.
17
Composition
Composites looks like components


Can test small pieces
Can also use large ones
Separate structure from algorithms


Definition/analysis of relationships
Eliminates explicit dependencies
No glue as usually meant


connection glue is factored out
algorithmic glue is encapsulated
Aug. 23, 2000
ISSTA 2000
18
Composition time
Structure may be decided at

Design time



Fully static composition
Parameterized composition
Runtime


Dynamic composition described as changes to existing
structure
May be downloaded as add-on software package
Haven’t fully explored tradeoffs; tending
towards more runtime composition
Aug. 23, 2000
ISSTA 2000
19
Example of Composition
Button to turn on light
Each piece has one+ contracts
IPushIt
Button
IButton
Events
Buttonto-Light
Adapter
ILight
Light
Desc.: Button, Adapter+stuff, Light, B->A, A->L
Aug. 23, 2000
ISSTA 2000
20
Encapsulation of the Switch
Edge Connector for lookup/connection
IPushIt
(role)
Edge Conn.
ILight
Button
IButton
Events
Buttonto-Light
Adapter
Light
Desc.: Button, Adapter+stuff, Light, B->A, A->L, EC(out B, in A)
Aug. 23, 2000
ISSTA 2000
21
Granularity of classes
Leaf classes

Up to 1500 lines of C


Probably less in Java or C#
1-30 classes per component DLL
Static composites
Normal range: 5-10 instances
 Packaged like a leaf class
 Connection description very cheap

Aug. 23, 2000
ISSTA 2000
22
Testing Open Systems
Late composition
Usually after component shipped
 Robustness of composite related to pieces
 System testing may be done by the user
 Need to support in-the-field debugging

Unit testing is critical
Must strive for more than the 80% case
 Finding evidence of correctness

Aug. 23, 2000
ISSTA 2000
23
Testing Components
Verify contract as implemented



Check pre/post conditions
Expose contract model variables
Composites have contracts too
Verify contracts consumed


Addresses some integration issues
Fault injection
Test and measure the binary form
Allow feedback to improve contract, impl., etc.
Aug. 23, 2000
ISSTA 2000
24
What is a robust component?
Beizer on statement and branch coverage:
“This is the weakest criterion in the family:
testing less than this for new software is
unconscionable and should be criminalized.”
For us:


Contract coverage
100% coverage at the object level



Coverage of all branches in sub-expressions
Ad hoc approach to selecting paths
Code review
Aug. 23, 2000
ISSTA 2000
25
Testing Framework
Test
Sequencer
Contract
Scenarios
State
Mapping
Objects
Pre/Post
Checks
(delegator)
Test
Subject
Dependencies
Aug. 23, 2000
ISSTA 2000
26
Testing a Light
Test
Sequencer
ILight
Scenarios
Map
ILight
model to
Impl.
Aug. 23, 2000
ISSTA 2000
ILight
Pre/Post
Checks
Light
27
Testing a Switch
Test
Sequencer
Switch
Scenarios
Map
model to
Impl.
Switch
Pre/Post
Checks
Switch
ILight
Checks
Aug. 23, 2000
ISSTA 2000
Light
28
Production Process
One small team per component
Deliverables include
Code for one component DLL
 Documentation
 Any new or updated tests

Different levels of hardness
Factored, versioned build system
Audit at end
Aug. 23, 2000
ISSTA 2000
29
Revisions
Bugs still happen
Usually modify and release new
component
In rare cases, release in place
Aug. 23, 2000
ISSTA 2000
30
Cost of Development
Composition RT
Connection Helper
Collections
Test framework
Deployment RT
#People #Months #Classes
2
5
5
1
2
1
3
6
6
1
24
3.5
26
31
Totals
145
43
KLOC
Ship Test
4
9
1
6
6
45
8
42
63
53
131
KLOC / year “shipped” = 4.4
KLOC / class = 1.2
Aug. 23, 2000
ISSTA 2000
31
Aug. 23, 2000
ISSTA 2000
2/11/2000
2/3/2000
1/28/2000
1/21/2000
1/13/2000
1/7/2000
12/20/1999
12/14/1999
12/9/1999
11/30/1999
10/28/1999
10/21/1999
10/15/1999
9/21/1999
The Road to 100% Coverage
110.00%
100.00%
90.00%
80.00%
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
32
Interesting Bugs
Code and test made same bug
Coverage of last branch yielded bugs,
twice
Post-condition in test was incomplete
Several difficult MT bugs
Shutdown of complex structures
Aug. 23, 2000
ISSTA 2000
33
Future Work
Reducing Cost
Testing Composites
Increasing Coverage
Contract Coverage
Contract Specification
Aug. 23, 2000
ISSTA 2000
34
Reducing Cost
Generate pre and post condition
delegators

Involves specification in contract and in
implementation
Apply consistency checks to contracts
Select test cases based on coverage
goals
Testing generics
Aug. 23, 2000
ISSTA 2000
35
Reducing Cost, cont.
Test specification language
Multi-threaded components

Track low level state and locks held for
detecting race conditions
Compiler-generated code

Reducing noise in output
Use a GC runtime
Aug. 23, 2000
ISSTA 2000
36
Testing Compositions
Static ones are like a component
Use composition description to help
Look for new paths not executed by unit test
Domain-specific



GUI (composition of bits on screen)
Network-based (different failure modes)
Hardware/software composites
Define contract for composite and try to show
correspondence with structure
Aug. 23, 2000
ISSTA 2000
37
Achieving More Coverage
Loops
More paths through cleanup code
Choosing other paths to measure

E.g., D-U paths in code or in state diagram
Aug. 23, 2000
ISSTA 2000
38
Contract Coverage
Sequencing tests
State walking; choosing paths is key
 Genetic algorithm

Test case correlation

Track variations generated from specs with
actual tests
Aug. 23, 2000
ISSTA 2000
39
Contract Specification Issues
Reentrancy, events
Need a description of the state of an
instance when an event handler is called
 E.g., when button fires events to adapter

State mutation described in spec

Can use that to help direct MT testing
Aug. 23, 2000
ISSTA 2000
40
Related Work
Brooks, Wirth, Parnas: various
Gall: Systemantics
Petroski: Designs and Errors
Williams: Inheritance + QI
Meyer: Design by Contract
Kiczales: Open Implementation
Harel, Wegner: Turing revisited
Aug. 23, 2000
ISSTA 2000
41
Related Work, cont.
CMU, Nierstrasz: Composition
Gamma, et. al., Coplien: Patterns
D’Souza & Wills: Catalysis
Beizer, RST Corp: Testing
Beck and others: Extreme Prog.
Holmes, et. all: Synchronization
Szyperski: Beyond OOP
Aug. 23, 2000
ISSTA 2000
42
Summary
Complex, robust, evolving software
systems
Component and composite approach
Contracts and testing
Need better tools!
http://research.microsoft.com/comapps
Aug. 23, 2000
ISSTA 2000
43