Grey Box testing
Tor Stålhane
What is Grey Box testing
Grey Box testing is testing done with limited
knowledge of the internal of the system.
Grey Box testers have access to detailed design
documents with information beyond
requirements documents.
Grey Box tests are generated based on
information such as state-based models or
architecture diagrams of the target system.
State based testing
The tests are derived from a state model of the
system. We can derive the state model in several
way, e.g. from
• Expected system behavior
• State part of a UML design or requirements
specification.
• Other state diagrams
Most system will, however, have a large number of
states
Binder’s state control faults – 1
Binder has make a list of common state –related
problems in software systems. This list may be
used as an input to
• State based testing
• State machine or code inspection
Binder’s state control faults – 2
• Missing or incorrect
– Transitions – new state is legal but incorrect
– Events – valid message ignored
• Extra, missing or corrupt state – unpredictable
behavior
• Sneak path – message accepted when it
should not be accepted
• Illegal message failure – unexpected message
causes a failure
• Trap door – system accepts undefined
message.
State test criteria
We can choose one or more of the following test
selection criteria:
• All states – testing passes through all states
• All events – testing forces all events to occur
at least once
• All actions – testing forces all actions to be
produced at least once
State test strategies
All round-trip paths
• All transition sequences beginning and ending
in the same state
• All simple paths from initial to final state
This strategy will help you to find
• All invalid or missing states
• Some extra states
• All event an action faults
Round-trip path tree – 1
A round-trip path tree
• Is built from a state transition diagram
• Includes all round-trip paths
– Transition sequences beginning and ending in the
same state
– Simple paths for initial to final state. If a loop is
present, we use only one iteration
• Is used to
– Check conformance to explicit behavioral models
– Find sneak paths
Round-trip path tree – 2
A test strategy based on round-trip path trees
will reveal:
• All state control faults
• All sneak paths – messages are accepted when
they should not
• Many corrupt states - unpredictable behavior
Round-trip path tree – 3
It is important to remember that a test based on
the round-trip path tree is model-based.
Full coverage from the round-trip path tree
implies full model coverage.
Full path coverage – which we will look at later –
implies full code coverage.
Challenge for round-trip path testing
In order to test a system based on state
transitions via triggers, predicates (guards)
and activities, we need to be able to observe
and register these entities.
Thus, we may need to include “points of
observations” in the code that gives us access
to the necessary information.
Round-trip tree – small example
a
a
A
a[p1] / w
B
b[p2] / u
a[p1] / w
A
C
B
C
A
b[p2] / u
W
A
W
B
A
W
Transitions
Each transition in a state diagram has the form
trigger-signature [guard] / activity. All parts are
optional
• trigger-signature: usually a single event that
triggers a potential change of state.
• guard: a Boolean condition that must be true
for the transition to take place.
• activity: an action that is performed during
the transition.
Test description – 1
Each test completes one branch of the roundtrip tree – from a to W. The necessary
transitions describes the test case.
a
The table on the next slide
shows the test case for
a -> A -> C -> A
A
B
C
A
W
B
A
A
W
Test description – 2
ID
Start
state
a
A
C
1
2
3
Event
Condition
Reaction
constructor
a
b
p1
p2
w
u
New
state
A
C
A
a
a
a[p1] / w
A
B
b[p2] / u
a[p1] / w
A
b[p2] / u
C
W
B
C
A
W
B
A
A
W
Sneak path test cases
A sneak path – message accepted when it
should not be accepted – can occur if
• There is an unspecified transition
• The transition occur even if the guard
predicate is false
Sneak path test description
Event
Condition
Reaction
1
Start
state
a
constructor
-
2
A
c
p1
3
A
a
p1 - false
Error
message
Error
message
a
a[p1] / w
A
A
a[p1] / w
A
a[p1] / w
B
b[p2] / u
C
New
state
A
a
b[p2] / u
ID
W
A
B
C
A
b[p2] / u
A
W
B
A
W
State diagram for a sensor - 1
w
C
E
D
B
a
A
State diagram for a sensor - 2
w
E
a
A
D
C
B
Sensor round-trip path tree
a
A
[sensor alarm] / sound alarm
[no sensor alarm] / test
E
B
[false alarm] / test
[test fails] / replace
D
[alarm OK / request reset]
[test OK]
w
[test fails] / replace
E
C
[ACK] / reset
/ test
E
w
Acknowledgement
Most of the previous presentation is based on a
slide set from the University of Ottawa,
Canada
Mutation testing
Tor Stålhane
Type 1 mutation testing – 1
Type 1 mutation testing is done as follows:
1. Write a chunk of code
2. Write a set of tests
3. Test and correct until the test suite runs
without errors
4. Change a random part of the code – e.g. a
“+” to a “-”. This is called a code mutant. We
will only consider mutants that compiles
without error messages
5. Run the test suite again
Type 1 mutation testing – 2
6. If the tests suite
– runs without errors, then we need to extend the
test suite until we discover the defect.
– diagnoses the defect and got back to step 4 to
create a new mutant.
The test process stops when all of X new
mutants are discovered by the current test
suite.
Type 2 mutation testing
Type 2 mutation testing – also called “fuzzing” –
has many ideas in common with random
testing. The main difference is that:
• Random testing requires us to generate
random tests from scratch.
• Type 2 mutation testing starts with an input
that works OK and then change part of it in a
random way.
Software functions are not continuous
When we discuss mutation testing, it is
important to remember that a function
implemented in software is not continuous.
E.g. x = 2.00 and x = 1.99 can give dramatically
different results.
A small changes in input can have a large effect
on the output.
Type 2 mutation testing example – 1
SUT – a system for computing F(x) – takes an
input consisting of
• F – a three character string identifying a
probability distribution function.
• A real number x. The allowed value range will
depend on F, e.g. if F = “ exp”, then x must be
a positive number, while if F = “nor”, then x
may be any number.
Type 2 mutation testing example – 2
We can perform type 2 mutation testing as follows:
1. Run a test with input <“exp”, 3>
2. Check that the result is correct
3. Make a mutant by drawing a random integer value
1 (F) or 2 (x).
–
–
If we draw a 1, generate a random integer n from 0 to 10 – string size – and
generate a random string of length n
If we draw a 2, generate a random real value x
4. Compute F(x)
5. Check the result – especially any error messages
6. If we are satisfied then stop, otherwise repeat from
step 3
Mutant testing strategies – 1
The number of possible mutants is large. In
order to have a reasonably sized test set there
are several strategies for reducing the number
of mutated systems of components.
The following slides will give a short description
of these strategies. Those who will use the
strategies should consult the paper by
Papdakis and Malevris on mutation testing.
Mutant testing strategies – 2
Mutation testing strategies are either of the first
order or the second order.
• First order strategy – perform a random
selection of a portion of the generated
mutants – e.g. 10% or 20%.
• Second order strategy – combine two mutants
to get one component to test. The strategy
will differ depending on how we choose the
mutants.
Assumptions
The following strategies assume that
• Different mutants of the same code can be
combined – i.e. they do not concern the same
operator, ID etc.
• Mutants that do not introduce errors – e.g.
mutations of comments – are removed from
the mutant set.
Second order mutation testing – 1
• Random Mix: combine two mutants selected
at random from the mutant set. Delete after
selection to avoid choosing the same mutant
twice.
• First2Last: sort all mutants according to code
line number. Combine the first with the last,
the second with the second-to-last and so on.
• Same Node: same as Random Mix but
selected from the same code block
Second order mutation testing – 2
• Same Unit: same as Random Mix but selected
from the same unit – e.g. class or method
• DiffOp: same as Random Mix but we select
mutants where different operators are
mutated – e.g. one mutation of a “+” and one
mutation of a “>”.
The strategies described above can also be
combined – e.g. SU_DiffOp consists of using
the DiffOp strategy but only on two mutation
from the same unit
Effectiveness measures
We will use two measures for mutant test
effectiveness – test effectiveness and cost
effectiveness.
Note that low values implies good results –
number of exposed faults is large.
Test Effectiveness 
No. of testcases
No. of Exposed Faults
Cost Effectiveness 
No. of testcases No. of EquivalentMutants
No. of Exposed Faults
Comments to First Order mutants
Selecting 10% of all generated mutants is best
both with regard to cost effectiveness and test
effectiveness.
Strong Mutation – using all generated mutants –
is the worst.
Comments to Second Order mutants
• SU_F2Last – Same Unit and First combined
with Last – scores highest on test effectiveness
• Random Mix – score highest on cost
effectiveness
• No second order strategy is more effective
than the Rand(10%) strategy. Here we have
that FD = No. of test cases / 1.34