A General Discussion on Functional
(Black-box) Testing
•
What are some of the concerns of testers ?
– Have we got enough time to test (effort & schedule)?
– Have we tested all the areas (coverage & efficiency)?
– Are we discovering the defects (effectiveness)?
Effort & (Schedule)
• Effort ≡ amount of time required to i) develop, ii)
execute, and iii) analyze the result of each test case:
– Decision table (highest)
• May be less number of test cases
• But requires deeper analysis of conditions (which come from
looking at the various combinations and dependencies)
– Completeness
– Avoiding redundancy
– Partitioned Equivalence class (middle)
• Picking a representative from each of the partitioned class
– Avoiding redundancy
• Difficulty is in coming up with the equivalence relation
– Complete and disjoint
– Boundary value (lowest)
• Easy to identify the boundaries
• Generating test cases are easier
• May expand into many test cases (especially for robust worst-case)
Testing Efficiency
• Efficiency ≡ amount of “coverage” provided
by each test case.
– Coverage of what?
• Possible combinations of inputs and expected outputs
• Possible logical conditions
– What about redundancy?
• With efficiency, we are striving to i) maximize
coverage (completeness) and ii) reduce
redundancy
Efficiency
– Decision table (high)
• Start with listing all n conditions and 2n combinations
– Completeness
• Analyze the irrelevant and reduce the redundant ones
– Avoiding redundancy
– Partitioned Equivalence class (high to medium)
• Define an equivalence relationship
– Completeness
• Pick a representative test case from each disjoint class
– Minimizing redundancy
• “Strong Robust” Equivalence (e.g. illegal cases) can
generate lots of test cases
– Boundary value (medium to low)
• Define the boundaries of valid and invalid inputs (a special
case of equivalence relation?)
– Completeness
• Pick data from within , on , and outside of the boundaries
– Reducing redundancy as much as possible
Effectiveness
• Effectiveness ≡ number of defects
discovered per test case executed.
• It is difficult to gauge this except from past
experience:
– People tend to make mistakes with boundaries
• Boundary value testing
• Equivalence class testing
– People tend to make mistakes with combinations
of inputs (logic) where there are dependencies
• Decision table testing
Testing (Effort)
= # of
Test
cases
= Time to
develop each
test case
sophistication
Boundary
Value
Equivalence
Class
Decision
Table
Need for Metrics and History
• In order for us to better understand and
manage testing, we need to better define the
attributes:
– Effort
– Efficiency
– Effectiveness
• We need to keep records (time; people-time;
defects discovery - type & rate) of test cases:
– Development of Test Cases
– Execution of Test Cases
– Test Result Analysis