Black-Box Test Reduction Using
Input-Output Analysis
ISSTA ‘00
Patrick J. Schroeder, Bogdan Korel
Department of Computer Science
Illinois Institute of Technology
Chicago, IL USA
PJS
ISSTA '00
1
Presentation Overview
1) Motivation
2) Black-Box Testing
3) Input-Output Analysis
4) Determining Input-Output Relationships
5) Conclusion
PJS
ISSTA '00
2
Motivation for Research

Experience in Industry as a black-box tester of large,
complex, telecommunications systems
 Root cause analysis of customer-found faults often
discovered ineffective or redundant black-box tests

PJS
Additional information from the software implementation
often improves black-box testing (after the initial release!)
ISSTA '00
3
Black-Box Testing

Black-box testing is testing from a functional or behavioral
perspective to ensure a program meets its specification
 Testing usually conducted without knowledge of software
implementation - system treated as a “black box”

Black-box test design techniques include: equivalence
partitioning, boundary value analysis, cause-effect graphing,
random testing
PJS
ISSTA '00
4
Observations on Black-Box Testing


Testing from the black-box perspective generates a large
number of tests (e.g. AT&T Definity PBX > 3 million tests)
All testers are faced with resource limitations; the number of
tests must be reduced
 Test reduction done randomly or based on “engineering
judgement” may lead to ineffective or redundant tests
PJS
ISSTA '00
5
Combinatorial Testing



Combinatorial testing: For programs with multiple inputs, one
must decide how to test different combinations of selected test
data values
The most thorough approach is to test all combinations - this is
quite often impossible due to the large number of tests
However, not to consider testing different input combinations
could lead to an unacceptable number of undetected software
faults
PJS
ISSTA '00
6
Combinatorial Explosion
X1
X2
X3
. . .
Xn
Program P
If n = 10 and 10 test data values are selected for each
variable, the total number of combinatorial test cases
is 1010
PJS
ISSTA '00
7
Current Approaches

Testers may try to identify “important” combinations to test
using specifications, historical fault data, and engineering
judgement
 Orthogonal arrays, or other combinatorial design techniques,
may be used to tests pair-wise or n-way combinations of
inputs

PJS
Difficult to determine impact on fault-detection capability of
the test set using these techniques
ISSTA '00
8
Definition of the Problem


In combinatorial testing, black-box testers face the situation
where the number of possible test cases far exceeds the time
and resources available to execute them
How can we reduce the number of combinatorial tests while
maintaining the fault-detection capability of combinatorial
testing?
PJS
ISSTA '00
9
Proposed Technique



Observations:
Current techniques do not consider program output variables
For a class of programs, not all program inputs affect all program outputs
We propose the use of Input-Output (IO) Analysis to determine
which program inputs affect program outputs
This information can be used to reduce the number of
combinatorial tests while “maintaining” the fault-detection
capability of the test set
PJS
ISSTA '00
10
Simple Example using IO Analysis
Input data selected using equivalence partitioning:
A={1, 3}
B={N,S,E,W}
C={TDC, BDM}
W
Z
Total Number of Combinatorial Tests: 2 * 4 * 2 = 16
PJS
ISSTA '00
11
IO Analysis Applied
A={1, 3}
B={N,S,E,W}
W
PJS
C={TDC, BDM}
Z
ISSTA '00
12
IO Analysis Applied
A={1, 3}
PJS
B={N,S,E,W}
C={TDC, BDM}
W
Z
T(W)
(1, N, TDC)
(1, N, BDM)
(3, N, TDC)
(3, N, BDM)
T(Z)
(1, N, TDC)
(1, S, TDC)
(1, E, TDC)
(1, W, TDC)
ISSTA '00
T(W)  T(Z)
(1, N, TDC)
(1, N, BDM)
(3, N, TDC)
(3, N, BDM)
(1, S, TDC)
(1, E, TDC)
(1, W, TDC)
13
Optimization Problem
• Create the smallest possible test set which covers all data
combinations in the individual output test sets
• Test set optimized using “Don’t Care” (DC) conditions
T(W)
(1, DC, TDC)
(1, DC, BDM)
(3, DC, TDC)
(3, DC, BDM)
PJS
T(Z)
(DC, N, DC)
(DC, S, DC)
(DC, E, DC)
(DC, W, DC)
ISSTA '00
Toptimized
(1, N, TDC)
(1, S, BDM)
(3, E, TDC)
(3, W, BDM)
14
Determining Input-Output Relationships



Information may be gathered in a variety of ways (e.g. from
specifications, code)
Manual collection of information difficult and time consuming
Automated collection possible through:
Static Analysis
Dynamic Analysis
Execution-oriented Analysis
PJS
ISSTA '00
15
Automated Analysis Techniques

Static techniques analyze a program’s source code - static
program dependencies used to determine IO relationships

Dynamic techniques analyze a program’s execution trace
across a set of test data - dynamic dependencies used to
determine IO relationships

Execution-oriented techniques manipulate individual inputs
across multiple program executions in a test harness to
determine IO relationships
PJS
ISSTA '00
16
Technique Advantages/Disadvantages

Static techniques reliably determine the absence of relationships
between inputs and outputs; however, they may identify
relationships that are not actually present
 Dynamic analysis and execution-oriented analysis reliably
determine the presence of input-output relationships; however,
they cannot guarantee that all relationships will be detected

Dynamic analysis and execution-oriented analysis may be used
when static analysis, perhaps in over-estimation, indicates that
all program inputs affect all program outputs
PJS
ISSTA '00
17
Experimental Study Results
1) Warehouse Management System
Combinatorial Tests:
Using IO Analysis
3,125
40
2) DACS Configuration System
Combinatorial Tests
Using IO Analysis
22,500
158
3) Liquidity Ratio Spreadsheet
Combinatorial Tests
Using IO Analysis
PJS
1.95 Million
625
ISSTA '00
18
Conclusion


For the class of program and type of problem defined, inputoutput analysis can reduce the number of combinatorial tests
while “maintaining” the fault-detection capability of the test
set
Research continues on identifying program types and testing
situations where our approach is applicable
 Additional experimentation with execution-oriented analysis
to provide program dependencies, and other information
useful in black-box testing, is underway
PJS
ISSTA '00
19