2013 REU Program at ECU
Software Testing - Foundations,
Tools, and Applications
Lecture 2
May 21, 2013
Software Testing Research
Dr. Sergiy Vilkomir
Exhaustive testing is impossible
A restricted number of test cases should be selected
There are different ways
(testing methods/strategies/criteria/approaches)
to select test cases
There are no method that guarantee
that software is 100% correct
Testing can show that defects are present,
but cannot prove that there are no defects.
2/38
Pair-Wise
Pair-Wise: A value of each level for each factor must be
combined with a value from every levels for each other
factor.
In other words, we must cover all possible pairs of values.
Pair-wise testing provides a small set of test cases. Why?
One test case covers several pairs
A
B
C
Test case
green
red
blue
red
covers
green
green
pairs:
red
blue
blue
It is a practical alternative to testing all combinations.
Pair-Wise
Pair-Wise: A value of each level for each factor must be
combined with a value from every levels for each other
factor.
A
B
C
red
red
red
red
green
blue
green
red
blue
blue
red
blue
blue
blue
green
green
green
green
blue
red
green
blue
green
red
red
blue
green
Exercise: Choose one pair and find a test case which covers this pair
Pairwise designs: Example
Suppose that a program to be tested requires 3 inputs, one
corresponding to each input variable. Each variable can take only
one of two distinct values.
Considering each input variable as a factor, the total number of
factor combinations is 23. Let X, Y, and Z denote the three input
variables and {X1, X2}, {Y1, Y2}, {Z1, Z2} their respective sets of
values. All possible combinations of these three factors follow.
Pairwise designs: Reducing the combinations
Now suppose we want to generate tests such that each pair appears
in at least one test. There are 12 such pairs: (X1, Y1), (X1, Y2),
(X1, Z1), (X1, Z2), (X2, Y1), (X2, Y2), (X2, Z1), (X2, Z2), (Y1,
Z1), (Y1, Z2), (Y2, Z1), and (Y2, Z2). The following four
combinations cover all pairs:
The above design is also known as a pairwise design. It is a
balanced design because each value occurs exactly the same
number of times. There are several sets of four combinations that
cover all 12 pairs.
Pairwise combinatorial testing
•
Pairwise combination (instead of exhaustive)
– Generate combinations that efficiently cover all pairs
– Rationale: most failures are triggered by single values
or combinations of a few values. Covering pairs
reduces the number of test cases, but reveals most
faults
Example: Display Control
The total number of combinations –
432 (3x4x3x4x3) test cases
Display Mode
Language
Fonts
Color
full-graphics
English
Minimal
Monochrome Hand-held
text-only
French
Standard
Color-map
Laptop
limitedbandwidth
Spanish
Documentloaded
16-bit
Full-size
Portuguese
True-color
Screen size
Pairwise combinations: 17 test cases
Language
Color
Display Mode
Fonts
Screen Size
English
Monochrome
Full-graphics
Minimal
Hand-held
English
Color-map
Text-only
Standard
Full-size
English
16-bit
Limited-bandwidth
-
Full-size
English
True-color
Text-only
Document-loaded
Laptop
French
Monochrome
Limited-bandwidth
Standard
Laptop
French
Color-map
Full-graphics
Document-loaded
Full-size
French
16-bit
Text-only
Minimal
-
French
True-color
-
-
Hand-held
Spanish
Monochrome
-
Document-loaded
Full-size
Spanish
Color-map
Limited-bandwidth
Minimal
Hand-held
Spanish
16-bit
Full-graphics
Standard
Laptop
Spanish
True-color
Text-only
-
Hand-held
Portuguese
-
-
Monochrome
Text-only
Portuguese
Color-map
-
Minimal
Laptop
Portuguese
16-bit
Limited-bandwidth
Document-loaded
Hand-held
Portuguese
True-color
Full-graphics
Minimal
Full-size
Portuguese
True-color
Limited-bandwidth
Standard
Hand-held
Exercise: Choose one pair and find a test case which covers this pair
Number of test cases
Example 1
Example 2
Example3
Number of factors
2
3
5
Number of all combinations
8
27
432
Number of pairwise test cases
4
9
17
Number of factors
Number of all combinations
Number of pairwise test cases
System 1
System 2
61
75
29 – 2 levels
17 – 3 levels
15 – 4 levels
35 – 2 levels
39 – 3 levels
1 – 4 levels
7.4 x 1025
5.5 x 1029
41
28
In-class exercise
Consider software with input parameters:
–Dest: London, Paris, Chicago, NY
–M: 1000, 2000, 3000
–Direct: Y, N
• Create test cases according to pair-wise approach.
• Number of all possible combinations?
4 x 3 x 2 = 24
• Number of pairs?
• Minimal number of test cases?
Start with factors with
max numbers of levels:
Dest - M:
4 x 3 = 12
We need min 12 test cases
Dest - M:
4 x 3 = 12
Dest – Direct: 4 x 2 = 8
M – Direct: 3 x 2 = 6
Total:
26
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
t1
L
t2
L
t3
L
t4
P
t5
P
t6
P
t7
C
t8
C
t9
C
t10
N
t11
N
t12
N
M
Direct
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
M
t1
L
1
t2
L
2
t3
L
3
t4
P
1
t5
P
2
t6
P
3
t7
C
1
t8
C
2
t9
C
3
t10
N
1
t11
N
2
t12
N
3
Direct
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
M
Direct
t1
L
1
Y
t2
L
2
N
t3
L
3
-
t4
P
1
-
t5
P
2
Y
t6
P
3
N
t7
C
1
N
t8
C
2
-
t9
C
3
Y
t10
N
1
Y
t11
N
2
N
t12
N
3
-
t-Wise
t-Wise: A value of each level for each group of t
factors must be combined.
• Sometimes: t-way (3-way, 4-way, etc)
• A natural extension is to require combinations of t values
instead of 2
• t-wise is expensive and benefits are not clear
• Example:
• 4-way
• 30 parameters
• 5 values each
• 3,800 test cases (could be too many)
Combinatorial Methods in Software Testing
•
•
National Institute of Standards and
Technology (NIST)
http://csrc.nist.gov/groups/SNS/acts/index.html
Presentation by Rick
Kuhn, (NIST) at ECU,
March 22, 2012
http://core.ecu.edu/ST
RG/seminars.html
16/18
Software Testing Study
Software Testing is a part of ECU MSc SE program
•
SENG 6265 Foundations of Software Testing
•
SENG 6270 Software Verification and Validation
Software Testing Research
Research - ?
New
Scientific approach: investigation, analysis, comparison,
justification, etc.
Publications
http://core.ecu.edu/STRG/
Project: Testing of Mobile Applications
•
•
•
•
•
•
•
•
Android Development Tools
Monkeyrunner tools from Android SDK
Case study (testing)
Robotium
Calabash-Android
Comparison, analysis
Pair-wise approach for mobile testing
Cloud Testing of Mobile Systems (CTOMS) framework