A General Discussion on Functional
(Black-box) Testing
•
What are some of the concerns of testers ?
– Have we got enough time to test (effort & schedule)?
– Have we tested all the areas (coverage & efficiency)?
– Are we discovering the defects (effectiveness)?
Effort & (Schedule)
• Effort ≡ amount of time required to i) develop, ii)
execute, and iii) analyze the result of each test case:
– Decision table (highest)
• May be less number of test cases
• But requires deeper analysis of conditions (which come from
looking at the various combinations and dependencies)
– Completeness
– Avoiding redundancy
– Partitioned Equivalence class (middle)
• Picking a representative from each of the partitioned class
– Avoiding redundancy
• Difficulty is in coming up with the equivalence relation
– Complete and disjoint
– Boundary value (lowest)
• Easy to identify the boundaries
• Generating test cases are easier
• May expand into many test cases (especially for robust worst-case)
Testing Efficiency
• Efficiency ≡ amount of “coverage” provided
by each test case.
– Coverage of what?
• Possible combinations of inputs and expected outputs
• Possible logical conditions
– What about redundancy?
• With efficiency, we are striving to i) maximize
coverage (completeness) and ii) reduce
redundancy
Efficiency
– Decision table (high)
• Start with listing all n conditions and 2n combinations
– Completeness
• Analyze the irrelevant and reduce the redundant ones
– Avoiding redundancy
– Partitioned Equivalence class (high to medium)
• Define an equivalence relationship
– Completeness
• Pick a representative test case from each disjoint class
– Minimizing redundancy
• “Strong Robust” Equivalence (e.g. illegal cases) can
generate lots of test cases
– Boundary value (medium to low)
• Define the boundaries of valid and invalid inputs (a special
case of equivalence relation?)
– Completeness
• Pick data from within , on , and outside of the boundaries
– Reducing redundancy as much as possible
Effectiveness
• Effectiveness ≡ number of defects
discovered per test case executed.
• It is difficult to gauge this except from past
experience:
– People tend to make mistakes with boundaries
• Boundary value testing
• Equivalence class testing
– People tend to make mistakes with combinations
of inputs (logic) where there are dependencies
• Decision table testing
Testing (Effort)
= # of
Test
cases
= Time to
develop each
test case
sophistication
Boundary
Value
Equivalence
Class
Decision
Table
Need for Metrics and History
• In order for us to better understand and
manage testing, we need to better define the
attributes:
– Effort
– Efficiency
– Effectiveness
• We need to keep records (time; people-time;
defects discovery - type & rate) of test cases:
– Development of Test Cases
– Execution of Test Cases
– Test Result Analysis
Insurance Premium Problem
(~ page 123 in your text – 3rd edition)
• Premium = Base Rate * age multiplier – SafeDriver reduction
– where Base Rate = $500 and premium is computed according to table
below, assuming the driver has less than 12 points.
Age range
16 ≤ age < 25
25 ≤ age < 35
35 ≤ age < 45
45 ≤ age < 60
60 ≤ age < 100
Age multiplier
Point cutoff
SafeDriver
Red.
2.8
≤1
50
1.8
≤ 3
50
1.0
≤ 5
100
0.8
≤ 7
150
1.5
≤ 5
200
Inputs to this system : 1) age and 2) points
Let’s consider 3 types of Functional Testing (Black
Box) Approaches – varies a bit from p 123 of text
1. A easy way is to start with boundary value, but dependent:
– age : 16 – 99 and points 0 – 12.
•
•
•
{16, 17, 45, 98, 99}
(0, 1, 5, 11, 12}
This will create 5 x 5 = 25 test cases ---- but, perhaps, not very satisfactory!
2. Continuing with boundary value, but looking at each range of ages
and each of the potential points.
– There are 5 age ranges and 5 boundary test cases for each range ->25 ages
– There are 0 – 12 or 13 possible penalty points (if we use all 13 points).
– Considering these are dependent or “worst case” gives us 25 x 13 = 325 cases
3. If we partition by age, then there are 5 partitions given, and partition
the points by {0,1}; {2,3}; {4,5}; {6,7}; (8,9,10,11,12}.
– We have 5 x 5 = 25 equivalent partitions --- potentially 25 test cases
4. We can also look to decision table approach:
– Consider criterion: {16-24}; {25-34}; {35-44}; {45-59}; {60-99}
– For each of the above criterion consider pts criterion (0-1); (2-3);(4-5);(6-7); (8-12)
– Consider the action of rules of multiplying by age multiplier and the redcution for
each of the possible criterion. ------ the possible criteria values are 210 ---- huge! ------- but many are “impossible” cases ---- so we can reduce it down.
Iterate through the Test Techniques
• Page 123 of text on insurance premium
– Start with a very simple boundary value testing;
• End points of ages [16, 100)
• End points of violation points [0, 12)
Note the boundaries
– For more completeness of coverage, consider the intervals
of
• Age
• Points
– Think of intervals as equivalence classes and reduce test
cases
– Because there are conditions that involve dependencies of
inputs, use decision tables and further reduce excessive test
cases.
• (e.g. combined conditions of penalty points beyond 1 and age
interval of 16 to 25 do not need to be included since that
combination is not allowed - - - - a saving of four “irrelevant”
test cases – see page 130 of text.)
1. Iterate from the simplest to the more difficult test case development method.
2. Review all the developed test cases and choose the ones you want to keep