White Box and Black Box
Testing
Tor Stålhane
What is White Box testing
White box testing is testing where we use the
info available from the code of the component
to generate tests.
This info is usually used to achieve coverage in
one way or another – e.g.
• Code coverage
• Path coverage
• Decision coverage
Debugging will always be white-box testing
Coverage report. Example – 1
Coverage report. Example – 2
McCabe’s cyclomatic complexity
Mathematically, the cyclomatic complexity of a structured
program is defined with reference to a directed graph
containing the basic blocks of the program, with an edge
between two basic blocks if control may pass from the first
to the second (the control flow graph of the program). The
complexity is then defined as:
v(G) = E − N + 2P
v(G) = cyclomatic complexity
E = the number of edges of the graph
N = the number of nodes of the graph
P = the number of connected components
Graph example
We have eight nodes – N = 8 –
nine edges – E = 9 – and we have
only one component – P = 1.
Thus, we have
v(G) = 9 – 8 + 2 = 3.
Simple case - 1
S1;
IF P1 THEN S2 ELSE S3
S4;
S1
P1
S2
S3
S4
One predicate – P1. v(G) = 2
Two test cases can cover all code
Simple case – 2
S1;
IF P1 THEN X := a/c ELSE S3;
S4;
S1
P1
S3
a/c
S4
One predicate – P1. v(G) = 2
Two test cases will cover all paths
but not all cases. What about the
case c = 0?
Statement coverage – 1
IF in_data > 10
{out_data = 4;}
ELSE
{out_data = 5;}
IF out_data == 8
{update_panel();}
P1
S2
S1
P2
S3
empty
How can we obtain full statement coverage?
Statement coverage – 2
out_data = 0
IF in_data > 10
{out_data = 4;}
update_panel();
If we set in_data to 12 we will have full
statement coverage. What is the problem?
Decision coverage
IF (in_data > 10 OR sub_mode ==3)
{out_data = 4;}
ELSE
{…..}
P1
P1-1
P1-2
P1 is really two decisions
P1-1: in_data > 10
P1-2: sub_mode == 3
We need to cover both decisions
empty
empty
S1
Using v(G)
The minimum number of paths through the
code is v(G).
As long as the code graph is a DAG – Directed
Acyclic Graph – the maximum number of
paths is 2**|{predicates}|
Thus, we have that
V(G) < number of paths < 2**|{predicates}|
Problem – the loop
S1
P1
S2
S3
S1;
DO
IF P1 THEN S2 ELSE S3;
S4
OD UNTIL P2
S5;
S4
P2
S5
No DAG. v(G) = 3 and Max
is 4 but there is an “infinite”
number of paths.
Nested decisions
S1
P1
S3
S2
S4
P2
S1;
IF P1 THEN S2 ELSE
S3;
IF P2 THEN S4 ELSE S5
FI
S6;
S5
S6
v(G) = 3, while Max = 4.
Three test case will cover all
paths.
Using a decision table – 1
A decision table is a general technique used to
achieve full path coverage. It will, however,
in many cases, lead to over-testing.
The idea is simple.
1. Make a table of all predicates.
2. Insert all combinations of True / False – 1 / 0
– for each predicate
3. Construct a test for each combination.
Using a decision table – 2
P1
P2
P3
0
0
0
0
0
1
0
1
0
0
1
1
1
0
0
1
0
1
1
1
0
1
1
1
Test description or reference
Using a decision table – 3
Three things to remember: The approach as it is
presented here is only practical for
• Situations where we have binary decisions.
• Small chunks of code – e.g. class methods and
small components. It will be too laborious for large
chunks of code.
Note that code that is difficult to reach – difficult to
construct the necessary predicates – may not be
needed as part of the system.
Decision table example – binary
P1
P2
0
0
Test description or
reference
S1, S3, S5, S6
0
1
S1, S3, S4, S6
1
0
S1, S2, S6
1
1
S1, S2, S6
S1
P1
S3
S2
S4
P2
S5
S6
The last test is not necessary
Decision table example – tertiary
S1
S2
P3
S4 S5
S9
S3
P1
P2
S6
S7
S10
S8
P1
P2
P3
Test description or
reference
0
0
0
0
0
0
1
1
1
1
1
1
0
0
0
1
1
1
0
0
0
1
1
1
0
1
2
0
1
2
0
1
2
0
1
2
S1, S3, S9
S1, S4, S9
S1, S5, S9
S1, S3, S9
S1, S4, S9
S1, S5, S9
S2, S6, S8
S2, S6, S8
S2, S6, S8
S2, S7, S8
S2, S7, S8
S2, S7, S8
What about loops
Loops are the great problem in white box
testing. It is common practice to test the
system going through each loop
• 0 times – loop code never executed
• 1 time – loop code executed once
• 5 times – loop code executed several times
• 20 times – loop code executed “many” times
Loop diagram
for (i; P1; i++)
{S1}
The empty branch is
taken when P1 is
False
P1
<<empty>>
S1
Error messages
Since we have access to the code we should
1. Identify all error conditions
2. Provoke each identified error condition
3. Check if the error is treated in a satisfactory
manner – e.g. that the error message is
clear, to the point and helpful for the
intended users.
What is Black Box testing
Black box testing is also called functional testing.
The main ideas are simple:
1. Define initial component state, input and
expected output for the test.
2. Set the component in the required state.
3. Give the defined input
4. Observe the output and compare to the
expected output.
Info for Black Box testing
That we do not have access to the code does not
mean that one test is just as good as the other
one. We should consider the following info:
• Algorithm understanding
• Parts of the solutions that are difficult to
implement
• Special – often seldom occurring – cases.
Clues from the algorithm
We should consider two pieces of info:
• Difficult parts of the algorithm used
• Borders between different types of solution –
e.g. if P1 then use S1 else use S2. Here we
need to consider if the predicate is
– Correct, i.e. contain the right variables
– Complete, i.e. contains all necessary conditions
Black Box vs. White Box testing
We can contrast the two methods as follows:
• White Box testing
– Understanding the implemented code.
– Checking the implementation
– Debugging
• Black Box testing
– Understanding the algorithm used.
– Checking the solution – functional testing
Testing real time systems
W-T. Tsai et al. have suggested a pattern based
way of testing real time / embedded systems.
They have introduced eight patterns. Using
these they have shown through experiments
that, using these eight patterns, they
identified on the average 95% of all defects.
We will have a look at three of the patterns.
Together, these three patterns discovered 60%
of all defects found
Patterns and coverage (from Tsai)
Basic scenario pattern - BSP
PreCondition == true / {Set activation time}
Check for
precondition
IsTimeout == true / [report fail]
Check
post-condition
PostCondition == true / [report success]
BSP – example
Requirement to be tested:
If the alarm is disarmed using the remote
controller, then the driver and passenger
doors are unlocked.
• Precondition: the alarm is disarmed using the
remote controller
• Post-condition: the driver and passenger
doors are unlocked
BSP test
1. Generate input to make precondition
a. False => nothing happens
b. True => input activation time
2. Check system response
a. Timeout = true => fails
b. Timeout = false and post condition OK => OK
c. Timeout false and post condition not OK => fails
Key-event service pattern - KSP
KeyEventOccurred / [SetActivationTime]
Check
precondition
Check for key
event
IsTimeout == true / [report fail]
PreCondition == true
Check
post-condition
PostCondition == true / [report success]
KSP- example
Requirement to be tested:
When either of the doors are opened, if the
ignition is turned on by car key, then the alarm
horn beeps three times
• Precondition: either of the doors are opened
• Key-event: the ignition is turned on by car key
• Post-condition: the alarm horn beeps three
times
KSP test
1. Generate key event
a. Not right event => wait
b. Right event => set activation time
2. Check precondition
a. Not OK => wait
b. OK => check post condition
3. Check post condition
a. Timeout = true => fails
b. Timeout = false and post condition OK => OK
c. Timeout false and post condition not OK => fails
Timed key-event service pattern - TKSP
KeyEventOccurred / [SetActivationTime]
Check
precondition
PreCondition == true
DurationExpired /
[report not exercised]
Check for key
event
IsTimeout == true / [report fail]
Check
post-condition
PostCondition == true / [report success]
TKSP – example (1)
Requirement to be tested:
When driver and passenger doors remain
unlocked, if within 0.5 seconds after the lock
command is issued by remote controller or car
key, then the alarm horn will beep once
TKSP – example (2)
• Precondition: driver and passenger doors
remain unlocked
• Key-event: lock command is issued by remote
controller or car key
• Duration: 0.5 seconds
• Post-condition: the alarm horn will beep once
TKSP test
1. Generate key event
a. Not right event => wait for new event or duration expired
b. Right event => set activation time
2. Check precondition
a. Not OK => wait
b. OK => check post condition
3. Check post condition
a. Timeout = true => fails
b. Timeout = false and post condition OK => OK
c. Timeout false and post condition not OK => fails
Test automation – 1
Test automation – 2
1.
2.
3.
4.
4
Generate stimuli
Get necessary data
Collect events
Check events
2
3
1
Test automation example – BSP
1. Illegal command
a. Generate something
b. Nothing happens => test OK
Post condition = “doors unlocked” => test fails
2. Legal command
a. Generate “alarm disabled”
b. Activation time = T
c. Timeout = True => test fails
Timeout = false
post condition = “doors unlocked” => test OK
Needs, models and methods – 1
If we say to the developers that they need to do
e.g. unit testing this is just a statement of
need, it is not a statement of how – the
method to use.
If we say that they should use white box testing,
this helps a little, but there is still a lot of
freedom when it comes to how.
Example – white box testing
White box testing =>
• Static white box testing =>
– Code inspection
– Code walkthrough
• Dynamic white box testing =>
–
–
–
–
Statement coverage
Path coverage
Decision coverage
All define – use coverage
Needs, models and methods – 2
We can use the following structure to organize
the decisions:
Strategy or need => Technique => Method
This will be applicable on all levels, e.g.
acceptance test, system test, integration tests
and unit tests.
For unit test we can for instance use:
Unit test => White box test => Path coverage