FSM Based Conformance and
Incremental Testing Methods
Khaled El-Fakih
American University of Sharjah, UAE
1
Outline

Part-1:



FSM Based Conformance Testing Methods
FSM Based Incremental Testing Methods
Part-2
2
Outline of Part-1







Introduction
FSM Conformance Testing
FSM Testing Methods
Experiments with FSM Testing Methods
Incremental Testing Methods
Experiments with Incremental Methods
Some Related References
3
Testing Methods: Functional (Black-Box)
Methods
Given Requirements of a System
 Generate Test Cases from Requirements
to check correctness of their
implementation(s)
=> Tests are derived from requirements

4
Testing Methods: Structural (White-Box
Methods


Tests are Derived for a given Program
using a testing criterion
Examples of Testing Criteria:


All uses of variables, All paths in program,
All conditions in program, etc..
5
Conformance Testing


A Black-box testing approach
Given:



A Formal Specification Spec
(requirements) of the System
A black-box Implementation Imp of Spec
Derive Tests from Spec to Determine if
Imp conforms to Spec

Conforms: has many definitions depending on
the Formal Specification
6
What is a Test/Test Suite

A Test of a given (FSM) Spec is a finite
sequence of input/output symbols


Eg: i1/o1 i2/o1
A Test Suite is a set of of Many Tests
7
How to Determine if Imp conforms to
Spec ?



Derive (using a Testing Method) from Spec a
TestSuite
Apply a Test from TestSuite to Imp
If for every Test of TesSuite:
Observed Behavior of Imp to Test =
Expected behavior specified in Test. Then,
Imp conforms to Spec.
Else, Imp does not conform to Spec (i.e.
Imp is a faulty implementation)
8
Type of Faults/Fault Model

Depends on Formal Specification model,
for example in FSM Based Testing we
usually consider output and transfer
faults (more on this later)
9
Conformance Relation


Thus in Conformance Testing, we derive
tests that check a conformance
relationship between a given
specification and its implementations.
This is done based on an assumed fault
model (Fault Types)
Examples of Conformance Relations:

equivalence, reduction relations,
etc..(more on this later)
10
Conformance Testing Revisited
Spec
FSM
LTS
Test Generator
Relation
Apply to
Test Case
Imp
Observed
Output
FAIL
W
Wp
HIS
UIOv
H
No
Expected
Output
Expected=Observed
Yes
Pass
11
FSM-Based Conformance
Testing


The Specification Spec is written as a
Finite State Machine (FSM)
What is a Finite State Machine



Set of states
Sets of Input and Output Symbols
Transitions between states each labeled
with an input/output symbols.
12
Example Deterministic FSM
y/0
s1
x/1
s2
x/1
y/0
x/0
y/1
y/0
s3
x/1
s4
Initial State: S1
Inputs : x, y
Outputs : 0, 1
States : S1, S2, S3, S4
13
Deterministic FSM Formal
Definition
An FSM is a 6-tuple:
M = (S, I , O,  , , s1) where
S: States
I: Input alphabet
O: Output alphabet
: Transition function S I S
: Output function S I O
s1 : Initial state
14
Partial FSMs

An FSM is partial if for some state s and
input I ;
Transition Function is not defined, i.e:
:
S I  Ø (Undefined)
and also Output Function is not defined,
i.e.:
: S I Ø (Undefined)
15
FSM-Testing Methods

Derive Tests in two phases:


Phase-1: State Identification Phase
Phase-2: Transition Testing Phase
Assumptions:


Fault Model/Types of Imp
Reset capability (written r ) in Imp
16
Fault Models

Types of Faults in Imp of Spec.

Output Fault(s)

Transfer Fault(s)

Multiple Faults
17
Output Fault
Spec ( S, I, O, S , S , s0 ) &
Imp ( S, I, O, I , I , s0 )
Transition (s,i) of Imp has an output fault if
S (s,i) = I (s,i) while S (s,i)  I (s,i)
1
i1 / O1
i1 / O2
2
Spec
Imp
18
Transition Fault
Spec ( S, I, O, S , S , s0 ) &
Imp ( S, I, O, I , I , s0 )
Transition (s,i) of Imp has a transfer fault if
S (s,i)  I(s,i)
1
i1 / O1
i1 / O1
Spec
2
#
Imp
19
Multiple Faults
Several transitions have multiple output
and/or transfer faults
20
Reset/No-reset Methods

Reset Methods: Assumes that Spec
(Imp) FSMs has a reset transition from
each state to the starting states of
Spec (Imp).


Examples: W, Wp, UIOv, UIO, HIS, H,
methods
No Reset Methods: No reset is assumed

Examples: DS and UIO methods
21
Equivalence Conformance Relation



Two states of an FSM are Equivalent if
they produce the same output for every
possible input
Two FSMs are Equivalent (=) if their
initial (starting) states are equivalent
We say that:
Imp conforms to Spec if Imp =Spec
22
State Identification Facilities

Reason: Since while testing we would
like to make sure that for every state in
Imp there is a corresponding state in
Spec, we would like to have a
sequence or set of sequences that
distinguish all states of Spec (Imp).
23
Characterization Set: W
For any two different states si and sj of
an FSM M,
 sequence b  W such that:
(si, b) ≠ (sj, b).
That is, for every two states of M, the W
set includes at least one sequence that
distinguishes these states.

24
Characterization Set (Example)
y/0
s1
y/0
x/0
y
S1
1
0
S2
1
1
S3
0
0
S4
0
1
x/1
s2
x/1
x
y/1
y/1
s3
x/0
s4
W set = { x, y }
25
FSM Based Testing Methods





W method
Wp method
HIS method
UIOv method
H method
26
W-method
Given a complete deterministic FSM
specification Spec, a W set for Spec.
Step1: Derive sequences to reach from
the initial state all states of Spec.
The set of these sequences is called State
Cover Set for Spec and denoted
byCoverSet
27
Example of CoverSet
y/0
s1
x/1
s2
x/1
y/0
x/0
y/1
y/1
s3
x/0
s4
CoverSet = {=empty word ,
S1
S2
y,
S3
x,
yy }
S4
28
W Method: Phase-1: Deriving State
Identification Sequences
Objective: For each state of Imp check that
there exists a corresponding state in Spec
Tests of Phase-1 = r.CoverSet.W
Formally: Check if there exists a mapping
h: T (states of Imp)  S (states of Spec) such
that:
h(t) = sj  t  W sj
29
Example:Phase-1 (W Method)
y/0
s1
x/1
s2
x/1
y/0
x/0
CoverSet ={, y, x, yy}
y/1
y/0
s3
x/1
s4
Inputs, X ={x, y}
W ={x, y}
r.CoverSet.W = r.{, y, x, yy}{x, y}=
rx, ry, ryx, ryy, rxx, rxy, ryyx,
ryyy
30
W-Method: Phase-2: Deriving
Transition Testing sequences
Objective: For each transition of Imp
check that there exists a corresponding
transition in Spec.
Tests of Phase-2: r.CoverSet.X .W
31
Phase-2 Formally
I(t, x)
t
=
S(h(t), x)
h(t)
x/Y
=
x/Y
MS
MI
h(I(t, x) )
=
S(h(t), x )
t
x
MI
x
MS
32
Example: Phase-2 (W method)
y/0
s1
x/1
s2
x/1
y/0
x/0
Inputs ={x, y}
CoverSet ={, y, x, yy}
y/1
y/0
s3
x/1
s4
W ={x, y}
r. CoverSet. Inputs. W =
r. {, y, x, yy}. {x, y}. {x, y}
33
Example of W Method (revisited)
y/0
s1
x/1
s2
x/1
y/0
x/0
y/1
y/0
s3
x/1
s4
34
Other Methods
Derivatives of the W method.
 Wp
 UIOv
 HIS
 DS
 H
35
Wp Method
As the W Method, However in Phase-2 a
subset of W (an appropriate set of state
identifiers) is selected to check the
ending state of a transition.
As the W method, not applicable when
Spec is partial and does not have a W
set.
36
Wp Method (continued)
Phase-1 : As Phase-1 of W method
Phase-2: for every sequence j of CoverSet
set and input x, we form sequences
r.j.x.Wk
Where Wk is a subset of W that distinguishes sk
(reached after r.j.x) of Spec from all other
states
 Wp generates shorter tests than W method
37
HIS Method
Uses a Separating Family of State
Identifiers instead of a W set in Phases
1 and 2.
A separating family for Spec is a
collection of harmonized state
identifiers.
38
A Family of Harmonized State
Identifiers
39
HIS Method: Phase-1
Given a Separating Family {H1,, H2} of states
of Spec.
Phase-1: for every sequence j of CoverSet
set, we form sequences
r.j.Hj
Where Hj is a state identifier of state Sj reached
after applying the input r.j
40
HIS Method: Phase-2
For every sequence j of CoverSet set
and input x, we form sequences
r. j. x. Hk
Where Hk is a state identifier of state Sk
reached after applying the input r. j . x
41
UIOv Method
As the Wp method. However, uses a
characterization a set of Unique Input
Output Sequences (UIO), where for
each state of Spec, a single (UIO)
sequence can be used for distinguishing
the state from all other states.
A UIO may not exist for some states of a
given Spec.

42
Distinguishing Sequence (DS) Method
A Distinguishing Sequence (DS) is ONE
sequence that distinguishes all states of
a given FSM Spec
DS-method: As W method, however the
W set now has one sequence, the DS
An FSM Spec may not have a DS
43
H Method
Alteration of the HIS method where
appropriate state identifiers are selected for
testing transitions. For checking the ending
state s of different incoming transitions to s,
different state identifiers can be used.
Promises: Applicable to any (reduced) partial
FSM Spec as the HIS method. Generates
shorter tests than the HIS and other methods

44
Experimental Evaluation of Testing
Methods
Objectives of Experiments:
 Compare the length of Test Suites
 Compare length of test suites with
worst-case theoretic bound
 Determine How often DS method a not
applicable
45
Experimental Results
Average Length of Test Suites
14500
12500
10500
8500
W
Wp
HIS
UIOv
H
DS
6500
4500
2500
500
46
Ratio : Length of Test Suites
Experimental Results (continued)
1.0
0.8
(Wp/HIS)/W
UIOv/W
H/W
0.6
0.4
0.2
47
Experimental Results


Ratios of the Wp/W, HIS/W, H/W are
almost independent of the size of Spec.
H method generates shorter tests than
all other methods.
48
Experimental Results


The DS method is applicable only to
19% of all conducted experiments
The order of all derived test suites is
O(5n2) which is lower than the
theoretical worst-case order O(kn3), n
is number of states and k is number of
inputs of Spec
49
How to Deal with Partial Specifications

Approach 1: A Partial FSM is converted
to a completely specified FSM, where
for every state s and every undefined
input x:
(s, x) = s
(s, x) = NULL
50
How to Deal with Partial Specifications

Approach-2: A Partial FSM is
converted to a completely specified
FSM, where for every state s and every
undefined input x:
(s, x) = ErrorState
(s, x) = ErrorOuput
51
How to Deal with Partial Specifications
Approach-3: For every state s and
every undefined input x :
(s, x) can take any state in S
(s, x) can take any output in Y
Now, the completed Spec is nondeterministic, where we still assume the
implementation is deterministic. Thus,
Spec has more information than Imp.

52
Incremental Testing Methods
Objective:
Minimize the efforts (length of tests)
required for testing modified or
incrementally implemented
specification.
53
Incremental Testing




Given a specification Spec and its
Conforming Implementation Imp
Modify Spec and obtain Spec ’
Implement the modified parts of Spec
in Imp and Obtain Imp’
Generate Incremental Tests to Check
that the modified parts of Spec’ are
correctly implemented in Imp’
54
Types of Modifications to Spec




Modify outputs of transition(s)
Modify ending states of transition(s)
Add/Delete Transitions
Add/Delete States
55
Incremental Methods
Also have two phases as non-incremental
methods
 Phase-1: Identify Some States of Spec’
in Imp’
 Phase-2: Test Appropriate Transitions of
Spec’
56
Case-1: Incremental W method



Applies: When all states of the modified
specification have state identifiers in the
unmodified part (i.e. do not traverse
modified transitions)
No State Identification is Needed
Phase-2: Only Test Modified
Transitions in Transition Testing Phase
57
Case-2: Incremental W method



Applies when each state of Spec’ has an
identifier in the unmodified part and not all
states are reachable through unmodified
transitions
Phase-1: Identify States that have state
identifiers in the unmodified parts. However,
apply only these identifiers that pass through
modified transitions
Phase-2: Test Only Modified Transitions
58
Case-3: Incremental W method


Applies when some states of Spec’ are only
reachable through modified transitions
Phase-1:



Identify each state that has state identifiers
passing through modified transitions
Identify each state that is only reachable
through unmodified transitions
Phase-2:


Test modified transitions
Test all outgoing transitions of states only
reachable through modified transitions
59
Experiments with Incremental v.s.
Non-Incremental Methods
Percentage of
Modifications
of Specification
5%
10 %
15 %
20 %
Incremental Tests
v.s. non-incremental tests
36
11
6
4
times shorter
times shorter
times shorter
times shorter
60
Testing when Imp has more states
than Spec
Assume that Spec has n states and the upper
bound on the number of states of Imp is m,
where m > n
A cover set for any Imp (with up to m states) of
Spec is
r.CoverSet.Xm-n,
where Xm-n is the set of all input sequences
over the alphabet X of length m-n.
61
W method when Imp has more states
than Spec
Given: W set, CoverSet , and Input
Alphabets X of Spec

Phase-1: State Identification Phase

When m = n, r.CoverSet.W

When m > n, r.CoverSet.Xm-n.W
62
W method when Imp has more states
than Spec

Phase-2: Transition Testing Phase

When m = n, r.CoverSet.X.W

When m > n, r.CoverSet.X.Xm-n.W
63
References: FSM Based Testing: Surveys, Experiments, Tools
G. v. Bochmann, A. Petrenko, “Protocol testing: review of methods and relevance for software testing,” Proc.
International Symposium on Software Testing and Analysis, Seattle, 1994, pp. 109-123.
D. Lee and M. Yannakakis, “Principles and methods of testing finite state machines-a survey”, Proceedings of
the IEEE, vol. 84, no. 8, 1996, pp. 1090-1123.
D. P. Sidhu, and T. K. Leung, “Formal methods for protocol testing: a detailed study,” IEEE Trans. SE, vol. 15,
no. 4, 1989, pp. 413-426.
M. Yannakakis and D. Lee, “Testing finite state machines: fault detection”, Journal of Computer and System
Sciences, 50, 1995, pp. 209-227.
Lai, “A survey of communication protocol testing, The Journal of Systems and Software, 62, 2002, pp. 21-46.
D. P. Sidhu and T. K. Leung, “Experience with test generation for real protocols”, in Proc. SIGCOMM ’88 Symp.:
Communication Architectures and Protocols, 1988, pp. 257-261.
S. T. Chanson, A.A.F., Loureiro, S.T. Vuong, “On tools supporting the use of formal description techniques in
protocol development,” Computer Networks and ISDN Systems, 25, 1993.
A. Dahbura and K. Sabnani, “Experience in estimating fault coverage of a protocol test”, in Proc. IEEE
INFOCOM’88, 1988, pp. 71-79.
Dorofeeva, M., El-Fakih, K., Maag S., A. R.Cavalli, & Yevtushenko, N. (2005). Experimental evaluation of FSMbased testing methods, In Proc. IEEE International Conference on Software Engineering and Formal
Methods (SEFM05), Germany, Sept. ’05 (to appear).
64
References (FSM Methods with resets)
[W method]
M. P. Vasilevskii, “Failure diagnosis of automata,” translated from Kibernetika, No.4, 1973, pp. 98-108.
T. S. Chow, “Test design modeled by finite-state machines,” IEEE Trans. SE, vol. 4, no.3, 1978, pp. 178-187.
[Wp method]
S. Fujiwara, G. v. Bochmann, F. Khendek, M. Amalou, and A. Ghedamsi, “Test selection based on finite state
models,” IEEE Trans. SE, vol. 17, no. 6, 1991, pp. 591-603.
[UIOv method]
S. T. Vuong, W.W.L. Chan, and M.R. Ito, “The UIOv-method for protocol test sequence generation,” Proc. of the
IFIP TC6 2nd IWPTS, North-Holland, 1989, pp. 161-175.
[HIS method]
Petrenko, N. Yevtushenko, A. Lebedev, and A. Das, “Nondeterministic state machines in protocol conformance
testing,” Proc. of the IFIP 6th IWPTS, France, 1993, pp. 363-378.
Yevtushenko and A. Petrenko, Test derivation method for an arbitrary deterministic automaton, Automatic Control
and Computer Sciences, Allerton Press Inc., USA, #5, 1990.
[H method]
Koufareva, M. Dorofeeva. “A novel modification of W-method”. Joint Bulletin of the Novosibirsk computing center
and A.P. Ershov institute of informatics systems. Series: Computing science, issue: 18, 2002, NCC Publisher,
Novosibirsk. - PP. 69-81.
Dorofeeva, M., El-Fakih, K., & Yevtushenko, N. (2005). An improved conformance testing method. In Proc. of the
IFIP 25th International Conference on Formal Techniques for Networked and Distributed Systems (FORTE 2005),
Taiwan.
65
References: FSM Incremental Testing
Methods
Petrenko, and N. Yevtushenko, “Test suite generation for a fsm with a given type of
implementation errors,” Proc. 12th Int. Workshop on Protocol Specification, Testing and
Verification, 1992, pp. 229-243.
Koufareva, A. Petrenko, and N. Yevtushenko, “Test generation driven by user-defined fault models”, in Proc. of
the IFIP TC6 12th International Workshop on Testing of Communicating Systems, Hungary, 1999, pp. 215-233.
K. El-Fakih, N. Yevtushenko, and G. v. Bochmann, “FSM-based incremental conformance testing methods”,
IEEE Transactions on Software Engineering, 30(7), 425-436.
K. El-Fakih, N. Yevtushenko, and G. v. Bochmann, “FSM based re-testing Methods,” Proc. of the IFIP TC6 14th
International Conference on Testing of Communicating Systems (TestCom), Germany, 2002, pp. 373-389.
66