Covering Arrays –
Mathematical, Engineering,
and Scientific Perspectives
Alan Hartman
May 2006
Fields Institute Workshop on Covering Arrays – Carleton University, Ottawa
Copyright 2006 © IBM Corporation
Outline
• Mathematics
• Engineering
• Science
Mathematics
• Definition
• My favorite constructions
Covering Array - Definition
• A covering array of strength t for k parameters over
domains of size n with R test cases is:
• An Rxk matrix with entries in {0,1,…n-1}
• Every Rxt submatrix contains every ordered t-tuple over
{0,1,…n-1} in at least one of its rows
•
CA(t, k , n)
is the minimum R such that there exists a
covering suite of R test cases with parameters t, k, n
• Equivalent to finding the maximum k such that there exists
a covering suite of R test cases with parameters t, k, n
Combinatorial Constructions
CA(2, k ,2)  R
Let R be the smallest integer such that
 R 1 

k  
 R / 2
K
2-3 4 5-10 11-15 16-35 36-56 57-126
R
4
5 6
7
8
9
10
First row all 0
Subsequent columns are incidence vectors of
R

1
subsets
of
an
set
R
/
2
 
Combinatorial Constructions
CA(2, k ,2)  R
 R 1 

k  
 R / 2
First row all 0
Subsequent columns are incidence vectors of
R / 2 subsets of an R  1 set
00 in first row
11 since any pair of R / 2 subsets intersect
01 and 10 since the subsets are the same size
Best possible by Sperner’s Lemma and
Erdös-Ko-Rado Theorem
Recursion for t>2
Let T (t , n) - the Turan number - be the maximum
number of edges over all graphs with t vertices
having no clique of size n  1
CA(t, k 2 , n)  (1  T (t, n))CA(t, k , n)
Uses perfect covering suites (OAs) of strength 2
to square the number of columns
Example: t=3, n=2
The maximum number of edges over all graphs
with 3 vertices having no clique of size 3 is 2.
CA(3, k 2 ,2)  3CA(3, k ,2)
A
A
kA
A
1
A
kA
A
2
A
kA
...
3
A
2
A
... kA
k
... k-1A
The general construction
CA(t, k 2 , n)  (1  T (t, n))CA(t, k , n)
T(t,n)+1
k
1
2
3
k
B[i,j]
N A A A … A
k
2
k
T(t,n)+1
C[i , j]=A
2
B[j,i]
Engineering Aspects
• Motivation
• Applications
• Input Coverage
• FSM Reduction
• Military
• Ease of Use
• Effectiveness
Applications
•Software input coverage test suites
•Software and hardware FSM model
reduction
•Blind disoriented robots on a line
Testing - Motivation
• Testing is important
• Testing is expensive
• Poor testing is even more expensive
Downtime Hourly Cost
•
Brokerage operations
$6,450,000
•
Credit card authorization
$2,600,000
•
Ebay (1 outage 22 hours)
$225,000
•
Amazon.com
$180,000
•
Package shipping services
$150,000
•
Home shopping channel
$113,000
•
Catalog sales center
$90,000
•
Airline reservation center
$89,000
•
Cellular service activation
$41,000
•
On-line network fees
$25,000
•
ATM service fees
$14,000
A Testing Problem
• Test an internet site:
– Operating system: Win, Linux, Solaris
– Browser: Explorer, Netscape
– Printer: Epsom, HP, IBM
– Protocol: Token Ring, Ethernet
36 combinations (OS, Br, Prt, Pcl)
1 tester
3 days
A Covering Array Solution
OS
Windows
Windows
Windows
Linux
Linux
Linux
Solaris
Solaris
Solaris
Browser
Explorer
Netscape
Explorer
Netscape
Explorer
Netscape
Explorer
Netscape
Explorer
Printer
Epsom
HP
IBM
Epsom
HP
IBM
Epsom
HP
IBM
Protocol
TR
Enet
Enet
TR
Enet
TR
Enet
TR
Enet
Model Based Testing with FSMs
• An abstract model of SW
• Nodes are labeled by tuples of state variables
• Arcs are labeled by possible inputs to the SW
• Test cases are generated by judicious choices
of paths through the state space
• Problem: State and Arc explosion
• Solution: Covering Arrays to sample the arcs at
each state
Blind disoriented synchronized robots
•k robots start at points 1,2,…k on a line
•They cannot see each other
•They have no common sense of direction
•They all move at speed 1
•They have to rendezvous in a minimum
number of steps
Blind disoriented synchronized robots
•Stage 1: Identify the extreme robots
•Stage 2: Extreme robots proceed towards
the centre.
•Optimize stage 1 by giving each robot a
column in a binary covering suite with t=2
•Interpret 0 as a move ½ left then ½ right
•Interpret 1 as a move ½ right then ½ left
•After CA(2, k ,2) steps the extreme robots
are known
Usability and Extensibility
• The Pairwise Website (www.pairwise.org)
lists 20 tools for constructing covering
arrays
•Key factors for usability:
•NO MATH
•Constraints! – but how
•Distinction between legal and illegal
inputs
•
1. CATS (Constrained Array Test System) *) [Sherwood] Bell Labs.
•
2. OATS (Orthogonal Array Test System) *) [Phadke] AT&T
•
3. AETG Telecordia Web-based, commercial
•
4. IPO (PairTest) *) [Tai/Lei]
•
5. TConfig [Williams] Java-applet
•
6. TCG (Test Case Generator) *) NASA
•
7. AllPairs Satisfice Perl script, free, GPL
•
8. Pro-Test SigmaZone GUI, commercial
•
9. CTS (Combinatorial Test Services) IBM Free for non-commercial use
•
10. Jenny [Jenkins] Command-line, free, public-domain
•
11. ReduceArray2 STSC, U.S. Air Force Spreadsheet-based, free
•
12. TestCover Testcover.com Web-based, commercial
•
13. DDA *) [Colburn/Cohen/Turban]
•
14. Test Vector Generator GUI, free
•
15. OA1 k sharp technology
•
16. CTE-XL Daimler Chrystler GUI, free
•
17. AllPairs [McDowell] Command-line, free
•
18. Intelligent Test Case Handler (replaces CTS) IBM Free for non-commercial
use
•
19. CaseMaker Díaz & Hilterscheid GUI, commercial
•
20. PICT Microsoft Command-line, free
WHITCH
•WHITCH is open and extensible
• 62 downloads – since January
• You get a free Covering Array GUI
• Why not use it to liven up your next grant
application?
•www.alphaworks.ibm.com/tech/whitch
WHITCH – Interface
• GUI panel for defining Types
• GUI panel for defining Columns
• GUI panels for choosing algorithms, variations etc.
• Java interfaces for plugging in new ideas, algorithms, panels
Science
• Empirical Software Engineering
• What are the issues
But is it all worth the trouble?
• Schroeder, Bolaki, Gopu, 2004 pointed
out that random test suites of the same
size are equally as effective as covering
arrays in detecting injected SW defects
• James Bach has been trumpeting the
news in all the engineering forums
• Do we have an answer?
SBG Arguments
• Previous papers compared pairwise to
“conventional testing”
• Correlated code coverage with fault
detection
• Small applications (6-8 KLOC)
• Artificially seeded bugs
Empirical SW Testing
• Need further experiments to determine if
and when covering arrays do help:
• Real software
• Real faults
• Real software testers