2014 REU Program at ECU
Software Testing - Foundations,
Tools, and Applications
Lecture 2
May 27, 2014
Software Testing Research
Dr. Sergiy Vilkomir
Exhaustive testing is impossible
A restricted number of test cases should be selected
There are different ways
(testing methods/strategies/criteria/approaches)
to select test cases
There are no method that guarantee
that software is 100% correct
Testing can show that defects are present,
but cannot prove that there are no defects.
2/38
All Combinations
All Combinations: All combinations of levels from all factors
must be used.
•
•
•
•
Example:
3 factors: A, B, C
3 levels for each factor: red, blue, green
Total number of all combinations: 33 = 27
Pair-Wise
Pair-Wise: A value of each level for each factor must be
combined with a value from every levels for each other
factor.
In other words, we must cover all possible pairs of values.
Pair-wise testing provides a small set of test cases. Why?
One test case covers several pairs
A
B
C
Test case
green
red
blue
red
covers
green
green
pairs:
red
blue
blue
It is a practical alternative to testing all combinations.
Pair-Wise
Pair-Wise: A value of each level for each factor must be
combined with a value from every levels for each other
factor.
A
B
C
red
red
red
red
green
blue
green
red
blue
blue
red
blue
blue
blue
green
green
green
green
blue
red
green
blue
green
red
red
blue
green
Exercise: Choose one pair and find a test case which covers this pair
Pairwise combinatorial testing
•
Pairwise combination (instead of exhaustive)
– Generate combinations that efficiently cover all pairs
– Rationale: most failures are triggered by single values
or combinations of a few values. Covering pairs
reduces the number of test cases, but reveals most
faults
Example: Display Control
The total number of combinations –
432 (3x4x3x4x3) test cases
Display Mode
Language
Fonts
Color
full-graphics
English
Minimal
Monochrome Hand-held
text-only
French
Standard
Color-map
Laptop
limitedbandwidth
Spanish
Documentloaded
16-bit
Full-size
Portuguese
True-color
Screen size
Pairwise combinations: 17 test cases
Language
Color
Display Mode
Fonts
Screen Size
English
Monochrome
Full-graphics
Minimal
Hand-held
English
Color-map
Text-only
Standard
Full-size
English
16-bit
Limited-bandwidth
-
Full-size
English
True-color
Text-only
Document-loaded
Laptop
French
Monochrome
Limited-bandwidth
Standard
Laptop
French
Color-map
Full-graphics
Document-loaded
Full-size
French
16-bit
Text-only
Minimal
-
French
True-color
-
-
Hand-held
Spanish
Monochrome
-
Document-loaded
Full-size
Spanish
Color-map
Limited-bandwidth
Minimal
Hand-held
Spanish
16-bit
Full-graphics
Standard
Laptop
Spanish
True-color
Text-only
-
Hand-held
Portuguese
-
-
Monochrome
Text-only
Portuguese
Color-map
-
Minimal
Laptop
Portuguese
16-bit
Limited-bandwidth
Document-loaded
Hand-held
Portuguese
True-color
Full-graphics
Minimal
Full-size
Portuguese
True-color
Limited-bandwidth
Standard
Hand-held
Exercise: Choose one pair and find a test case which covers this pair
Algorithms
•
•
•
•
•
There are several algorithms for test case generation
according to pairwise approach
Several techniques are used:
– Latin Squares
– Orthogonal arrays, etc
One of the well known algorithms is IPO (in-parameter-order),
1998
It is hard to generate test manually for a large number of
factors and/or levels
There are many tools for pairwise test case generation
Pairwise tools
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
1. CATS (Constrained Array Test System) [Sherwood] Bell Labs.
2. OATS (Orthogonal Array Test System) [Phadke] AT&T
.3 AETG Telecordia Web-based, commercial
4. IPO (PairTest) [Tai/Lei]
5. TConfig [Williams] Java-applet
6. TCG (Test Case Generator) NASA
7. AllPairs Satisfice Perl script, free, GPL
8. Pro-Test SigmaZone GUI, commercial
9. CTS (Combinatorial Test Services) IBM Free for non-commercial use
10. Jenny [Jenkins] Command-line, free, public-domain
11. ReduceArray2 STSC, U.S. Air Force Spreadsheet-based, free
12. TestCover Testcover.com Web-based, commercial
13. DDA [Colburn/Cohen/Turban]
14. Test Vector Generator GUI, free
15. OA1 k sharp technology, etc
Number of test cases
Example 1
Example 2
Example3
Number of factors
2
3
5
Number of all combinations
8
27
432
Number of pairwise test cases
4
9
17
Number of factors
Number of all combinations
Number of pairwise test cases
System 1
System 2
61
75
29 – 2 levels
17 – 3 levels
15 – 4 levels
35 – 2 levels
39 – 3 levels
1 – 4 levels
7.4 x 1025
5.5 x 1029
41
28
In-class exercise
Consider software with input parameters:
–Dest: London, Paris, Chicago, NY
–M: 1000, 2000, 3000
–Direct: Y, N
• Create test cases according to pair-wise approach.
• Number of all possible combinations?
4 x 3 x 2 = 24
• Number of pairs?
• Minimal number of test cases?
Start with factors with
max numbers of levels:
Dest - M:
4 x 3 = 12
We need min 12 test cases
Dest - M:
4 x 3 = 12
Dest – Direct: 4 x 2 = 8
M – Direct: 3 x 2 = 6
Total:
26
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
t1
L
t2
L
t3
L
t4
P
t5
P
t6
P
t7
C
t8
C
t9
C
t10
N
t11
N
t12
N
M
Direct
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
M
t1
L
1
t2
L
2
t3
L
3
t4
P
1
t5
P
2
t6
P
3
t7
C
1
t8
C
2
t9
C
3
t10
N
1
t11
N
2
t12
N
3
Direct
In-class exercise
– Dest: London, Paris, Chicago, NY
– M: 1000, 2000, 3000
– Direct: Y, N
Dest
M
Direct
t1
L
1
Y
t2
L
2
N
t3
L
3
-
t4
P
1
-
t5
P
2
Y
t6
P
3
N
t7
C
1
N
t8
C
2
-
t9
C
3
Y
t10
N
1
Y
t11
N
2
N
t12
N
3
-
t-Wise
t-Wise: A value of each level for each group of t
factors must be combined.
• Sometimes: t-way (3-way, 4-way, etc)
• A natural extension is to require combinations of t values
instead of 2
• t-wise is expensive and benefits are not clear
• Example:
• 4-way
• 30 parameters
• 5 values each
• 3,800 test cases (could be too many)
Combinatorial Methods in Software Testing
•
•
National Institute of Standards and
Technology (NIST)
http://csrc.nist.gov/groups/SNS/acts/index.html
Presentation by Rick
Kuhn, (NIST) at ECU,
March 22, 2012
http://core.ecu.edu/ST
RG/seminars.html
17/18
Software Testing Study
Software Testing is a part of ECU MSc SE program
•
SENG 6265 Foundations of Software Testing
•
SENG 6270 Software Verification and Validation
Software Testing Research
Research - ?
New
Scientific approach: investigation, analysis, comparison,
justification, etc.
Publications
Long period of time – BS, MS, PhD ….
http://core.ecu.edu/STRG/
Project 1
Mobile testing to detect
device specific faults​
21/
Device-specific failures
•
•
•
Device-specific failures are very common for mobile software
applications
An application works reliably on many smartphones and
tablets, but does not work properly (i.e., fails) on some
specific devices
Examples:
• App normally works under some specific operating system but fails
under the latest or older OS
• Graphics created for high-resolution screens are not shown
properly on the mobile devices with extra-high or low screen
resolutions
•
Factors: OS, screen resolution, screen size, device type
(smartphone or tablet), handset manufacturer, RAM, etc.
22/
Device-specific failures
How many?
•
There are many different mobile devices
•
Sufficient testing is required on different mobile devices
•
Such testing is expensive and time-consuming
23
Android Fragmentation Visualized (July 2013)
From OpenSignal, http://opensignal.com/reports/fragmentation-2013/
From OpenSignal: We have seen 11,868 distinct devices
download our app in the past few months. In our report last
year we saw 3,997
24
Research task
Optimal selection of mobile devices for testing
S. Vilkomir and B. Amstutz, "Using Combinatorial Approaches
for Testing Mobile Applications", Proceedings of the IEEE
Seventh International Conference on Software Testing,
Verification and Validation Workshops (ICSTW 2014), March
31 - April 4, 2014, Cleveland, Ohio, USA, pp. 78-83.
http://core.ecu.edu/STRG/publications/Vilkomir-IWCT-2014Proceedings.pdf
•
•
Chauncey R. Perry, Testing Mobile Applications to Detect
Device Specific Faults, MS project, ECU, May 2014​
25
Device-specific failures
26/
Device-specific failures
27/
Device-specific failures
28/
Device-specific failures
29/
Project 2
Effective Test Generation using
Combinatorial Design and ​MCDC
30/
Terminology
•
•
Logical expression (the program points at which the control
flow can divide into various paths) = Decision
Atomic predicates (elementary Boolean expressions),
which form component parts of decisions = Conditions
Example:
d = ACBD
d – decision
Values of decisions and conditions are
TRUE ( 1 ) or FALSE ( 0 )
A, B, C, D – conditions
Conditions depend on program data, for example:
A ↔ x<7
B ↔ y=3
31
Condition/decision coverage
Condition/decision coverage =
= decision coverage + condition coverage
Condition/Decision (D/CC): every decision in the program has taken all
possible outcomes at least once, and every condition in each decision has
taken all possible outcomes at least once
32
32
Condition/decision coverage
Example: d = (A  B)  (C  D)
Do these test cases satisfy C/DC ?
Two test
cases
A
B C D d
A
B C D d
1
1
0
0
1
1
1
1
0
1
1
0
1
0
0
0
0
0
1
0
The same outcomes - NO
all possible outcomes - YES
33
MC/DC
•
“Independent affect” is the main idea of Modified
Condition/Decision Coverage
(MC/DC)
•
Suggested in 1992-1994 for avionic software
•
DO-178B. Software Considerations in Airborne
Systems and Equipment Certification. USA, 1992.
Chilenski, J. and Miller, S. Software Engineering
Journal, 1994
•
Test cases according to MC/DC
d = (A  B)  C
1
2
3
4
5
6
7
8
A
1
1
1
0
1
0
0
0
B
1
1
0
1
0
1
0
0
C
1
0
1
1
0
0
1
0
d
1
0
1
1
0
0
0
0
A
B
*
*
*
*
C
MC/DC
*
*
*
*
*
*
Project 2
•
•
•
•
•
•
Measure combinatorial coverage of test sets that were
developed using non-combinatorial methods including
MCDC.
Measure MCDC coverage of CT test sets.
Propose methods for enhancing CT approaches to achieve
MCDC and other source code coverage criteria.
Empirically investigate effectiveness of the proposed
methods when substantial information of code is available.
To the greatest extent possible, real-world software should
be used, i.e., code should not be small seeded-error
programs.
Decrease cost of achieving coverage criteria as compared
with other approaches.
Develop a tool for test case generation according to the
proposed methods