2013 REU Program at ECU Software Testing - Foundations, Tools, and Applications Lecture 2 May 21, 2013 Software Testing Research Dr. Sergiy Vilkomir Exhaustive testing is impossible A restricted number of test cases should be selected There are different ways (testing methods/strategies/criteria/approaches) to select test cases There are no method that guarantee that software is 100% correct Testing can show that defects are present, but cannot prove that there are no defects. 2/38 Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. In other words, we must cover all possible pairs of values. Pair-wise testing provides a small set of test cases. Why? One test case covers several pairs A B C Test case green red blue red covers green green pairs: red blue blue It is a practical alternative to testing all combinations. Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. A B C red red red red green blue green red blue blue red blue blue blue green green green green blue red green blue green red red blue green Exercise: Choose one pair and find a test case which covers this pair Pairwise designs: Example Suppose that a program to be tested requires 3 inputs, one corresponding to each input variable. Each variable can take only one of two distinct values. Considering each input variable as a factor, the total number of factor combinations is 23. Let X, Y, and Z denote the three input variables and {X1, X2}, {Y1, Y2}, {Z1, Z2} their respective sets of values. All possible combinations of these three factors follow. Pairwise designs: Reducing the combinations Now suppose we want to generate tests such that each pair appears in at least one test. There are 12 such pairs: (X1, Y1), (X1, Y2), (X1, Z1), (X1, Z2), (X2, Y1), (X2, Y2), (X2, Z1), (X2, Z2), (Y1, Z1), (Y1, Z2), (Y2, Z1), and (Y2, Z2). The following four combinations cover all pairs: The above design is also known as a pairwise design. It is a balanced design because each value occurs exactly the same number of times. There are several sets of four combinations that cover all 12 pairs. Pairwise combinatorial testing • Pairwise combination (instead of exhaustive) – Generate combinations that efficiently cover all pairs – Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs reduces the number of test cases, but reveals most faults Example: Display Control The total number of combinations – 432 (3x4x3x4x3) test cases Display Mode Language Fonts Color full-graphics English Minimal Monochrome Hand-held text-only French Standard Color-map Laptop limitedbandwidth Spanish Documentloaded 16-bit Full-size Portuguese True-color Screen size Pairwise combinations: 17 test cases Language Color Display Mode Fonts Screen Size English Monochrome Full-graphics Minimal Hand-held English Color-map Text-only Standard Full-size English 16-bit Limited-bandwidth - Full-size English True-color Text-only Document-loaded Laptop French Monochrome Limited-bandwidth Standard Laptop French Color-map Full-graphics Document-loaded Full-size French 16-bit Text-only Minimal - French True-color - - Hand-held Spanish Monochrome - Document-loaded Full-size Spanish Color-map Limited-bandwidth Minimal Hand-held Spanish 16-bit Full-graphics Standard Laptop Spanish True-color Text-only - Hand-held Portuguese - - Monochrome Text-only Portuguese Color-map - Minimal Laptop Portuguese 16-bit Limited-bandwidth Document-loaded Hand-held Portuguese True-color Full-graphics Minimal Full-size Portuguese True-color Limited-bandwidth Standard Hand-held Exercise: Choose one pair and find a test case which covers this pair Number of test cases Example 1 Example 2 Example3 Number of factors 2 3 5 Number of all combinations 8 27 432 Number of pairwise test cases 4 9 17 Number of factors Number of all combinations Number of pairwise test cases System 1 System 2 61 75 29 – 2 levels 17 – 3 levels 15 – 4 levels 35 – 2 levels 39 – 3 levels 1 – 4 levels 7.4 x 1025 5.5 x 1029 41 28 In-class exercise Consider software with input parameters: –Dest: London, Paris, Chicago, NY –M: 1000, 2000, 3000 –Direct: Y, N • Create test cases according to pair-wise approach. • Number of all possible combinations? 4 x 3 x 2 = 24 • Number of pairs? • Minimal number of test cases? Start with factors with max numbers of levels: Dest - M: 4 x 3 = 12 We need min 12 test cases Dest - M: 4 x 3 = 12 Dest – Direct: 4 x 2 = 8 M – Direct: 3 x 2 = 6 Total: 26 In-class exercise – Dest: London, Paris, Chicago, NY – M: 1000, 2000, 3000 – Direct: Y, N Dest t1 L t2 L t3 L t4 P t5 P t6 P t7 C t8 C t9 C t10 N t11 N t12 N M Direct In-class exercise – Dest: London, Paris, Chicago, NY – M: 1000, 2000, 3000 – Direct: Y, N Dest M t1 L 1 t2 L 2 t3 L 3 t4 P 1 t5 P 2 t6 P 3 t7 C 1 t8 C 2 t9 C 3 t10 N 1 t11 N 2 t12 N 3 Direct In-class exercise – Dest: London, Paris, Chicago, NY – M: 1000, 2000, 3000 – Direct: Y, N Dest M Direct t1 L 1 Y t2 L 2 N t3 L 3 - t4 P 1 - t5 P 2 Y t6 P 3 N t7 C 1 N t8 C 2 - t9 C 3 Y t10 N 1 Y t11 N 2 N t12 N 3 - t-Wise t-Wise: A value of each level for each group of t factors must be combined. • Sometimes: t-way (3-way, 4-way, etc) • A natural extension is to require combinations of t values instead of 2 • t-wise is expensive and benefits are not clear • Example: • 4-way • 30 parameters • 5 values each • 3,800 test cases (could be too many) Combinatorial Methods in Software Testing • • National Institute of Standards and Technology (NIST) http://csrc.nist.gov/groups/SNS/acts/index.html Presentation by Rick Kuhn, (NIST) at ECU, March 22, 2012 http://core.ecu.edu/ST RG/seminars.html 16/18 Software Testing Study Software Testing is a part of ECU MSc SE program • SENG 6265 Foundations of Software Testing • SENG 6270 Software Verification and Validation Software Testing Research Research - ? New Scientific approach: investigation, analysis, comparison, justification, etc. Publications http://core.ecu.edu/STRG/ Project: Testing of Mobile Applications • • • • • • • • Android Development Tools Monkeyrunner tools from Android SDK Case study (testing) Robotium Calabash-Android Comparison, analysis Pair-wise approach for mobile testing Cloud Testing of Mobile Systems (CTOMS) framework