Verification and Validation in an SDL/MSC setting
Øystein Haugen
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /1
Formal Methods — The Buzzword of the Nineties?
The Techniques:
The Problems:
The successes
Advanced (formal)
Critical (small&complicated&important)
SDL/MSC
The theorists
Advanced (big&complex)
The practitioners
Trivial (informal)
Verification and Validation /2
Trivial (small&simple)
•
Scaling
•
Tutoring
•
Transparency
•
Automating
Øystein Haugen, Ericsson NorARC, Ifi Uio
The need for more formal techniques
Formal
Informal
Complicated
Simple
Cryptic
Inflexible
Machinelike
Comprehensible
Flexible
Human
Natural
Mathematical
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /3
More advantages of the formal:
Formal
Informal
Complicated
Simple
Cryptic
Inflexible
Machinelike
Mathematical
Conceivable
Flexible
Human
Natural
Unambiguous
Controlled redundancy
Analyzable (auotmatically)
Transformable (to machine code)
Often ambiguous
Haphazard redundancy
Unclear semantics
Transformable only by humans
Verification and Validation /4
Øystein Haugen, Ericsson NorARC, Ifi Uio
”Formal description" – a definition
The meaning of a “formal description”relies exclusively on the
defined semantics of the language in which the description is made.
The meaning of an “informal description”depends on the
interpretation of natural language terms used in the description.
Verification and Validation /5
Øystein Haugen, Ericsson NorARC, Ifi Uio
Two examples
main ()
{ int fn = 1;
int fnm1 =0;
cout << "-1: " << fnm1 << "\n" ;
cout << "0: " << fn << "\n" ;
for (int n=1; n<10; n++)
{ int fnp1 = fn + fnm1;
fnm1 = fn ;
fn = fnp1;
The first 10 Fibonacci
numbers
cout << n <<": " << fn << "\n";
}
}
Verification and Validation /6
Øystein Haugen, Ericsson NorARC, Ifi Uio
Mind blow 1
Some years ago there were rumors of a navy maneuvre south of
Italy that had fired a ground-to-air missile and a passenger
airplane having been hit.
This was perhaps not true, but through simulation testing of an
already delivered missile system, researcher spent only 3 hours
to find a sequence of events (which the designers of the system
had not considered) which could fire a missile [Harel 92]
Verification and Validation /7
Øystein Haugen, Ericsson NorARC, Ifi Uio
Mind Blow 2
Tharac 25 was a radiation machine for cancer
therapy which caused a series of deaths in USA
1995-87.
Tharac 25 used software from Tharac 20 which
had been in use for a number of years without any
accidents of excessive radiation. Tharac 25 had
substituted some hardware by new and better
ones.
What nobody thought about was that hte old
hardware had functioned as a “firewall”(literally
speaking) against high levels of radiation. The new
platform had no such implicit mechanisms.
The fire wall function of the old machine was not
intended. Everybody thought that the software had
been thoroughly tested during the years in Tharac
20. [de Roever 92]
Verification and Validation /8
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification or Validation
lVerification: to establish the truth of correspondence between
a software product and its specification (from the Latin veritas,
“truth”).
lValidation: to establish the fitness or worth of a software
product for its operational mission (from the Latin valere, “to
be worth”).
lBarry Boehm rephrased this by:
– Verification: are we building the product right?
– Validation: are we building the right product?”
Verification and Validation /9
Øystein Haugen, Ericsson NorARC, Ifi Uio
Levels of system validation
1. Test-oriented
(Testing is the most important means to establish the correctness of the
system. Testing is on the implementation of the system)
2. Inspection-oriented
(Inspections involve human readers who control the quality of the
descriptions)
3. Animation-oriented
(Animation and simulation are executions of the system based on
descriptions on higher abstraction levels than implementation)
4. Analysis-oriented
(Formal analysis is used in order to prove statements about the system, or
to disclose hidden aspects with a system)
5. Synthesis-oriented
(When the implementation can be synthesized from a description of the
requirements)
Verification and Validation /10
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation in TIMe
domain
Verification
domain
family
specification
Verification
Validation
design
Verification
Validation
Validation
Validation
Market
needs
implementation
instance
needs
needs
Verificainstance
configuration
needs
sy stem
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /11
Legend
Meta- and General properties
Prescriptive properties
Complementary object
model
Verification and Validation /12
Descriptive properties
Main Object Model
Øystein Haugen, Ericsson NorARC, Ifi Uio
Testing
Quality assessment
Prescriptive properties
as test case purposes
(MSC)
Complementary object
model
Descriptive properties (provided)
Main Object Model
(SDL)
produce
Test suite
(TTCN)
input
compare
Executing implementation
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /13
Inspection (Verification)
Inspection
Prescriptive properties
Is the correspondence correct?
Complementary object
model
Verification and Validation /14
Descriptive properties
Is the transformation correct?
Main Object Model
Øystein Haugen, Ericsson NorARC, Ifi Uio
Inspection (Validation)
Quality assessment
Prescriptive properties
Descriptive properties
What is the quality and complex ity?
Complementary object
model
Main Object Model
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /15
Simulation
human evaluation
General properties
Required properties
Provided properties
input from simulation pilot
Prototype shortcuts
Prototype Object Model (SDL)
Automatic
generation
Executing Simulation Model
Verification and Validation /16
Øystein Haugen, Ericsson NorARC, Ifi Uio
Formal Analysis (Verification)
General properties
Functional requirement
specification (MSC)
Non-functional requirements
Are the object model
consistent with the requirement?
Implementation design
(object model)
Design Object Model (SDL)
Is the implementationoriented object model
a refinement?
More implementation oriented Object Model
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /17
Formal Analysis + Simulation
Design Object Model (SDL)
Executing Design Object Model
Simulation
Provided Requirements (MSC)
Model Checking
Implementation Object Model (SDL)
Verification and Validation /18
Øystein Haugen, Ericsson NorARC, Ifi Uio
Formal Analysis (Validation, General Quality)
General properties
Prescriptive properties
Descriptive properties
Do the general properties hold?
Complementary object
model
Main Object Model
labsence of deadlock;
labsence of livelock;
lall control states in a process should be reachable;
lthe input port of a process should not increase beyond any
bound.
Verification and Validation /19
Øystein Haugen, Ericsson NorARC, Ifi Uio
Model Checking vs. Proofs (from Shrivas)
lVerification using Theorem Proving
– Given program M and property P, proves it is true that M implies P
– works for finite or infinte-state systems
– Drawbacks:
lworks best on data-intensive rather than control-intensive programs
lusually requires human input and guidance
lVerification using Model Checking
– Given program M and property P, automatically show that P is valid in M
– works best on systems given as finite automaton
– Drawbacks:
lworks best on control-intensive programs with a small state space
lmay require human intervention for minimizing space/time
Verification and Validation /20
Øystein Haugen, Ericsson NorARC, Ifi Uio
Finite transition system?
lSDL does not describe finite transition systems
lWe must introduce limitations to make the total execution
graph finite
– limit on signal buffer size (length of input port)
– finite valuespace for variables
Verification and Validation /21
Øystein Haugen, Ericsson NorARC, Ifi Uio
Basic model
lconcurrent components (B1,B2,B3)
– finite state machines
lchannels (c1,c2,c3,c4,c5)
– incl. signalroutes
lexternal input (e1,e2)
– may come at any time
linternal signals (i1,i2)
lexternal output (e3)
latomic transitions
linterleaving semantics
Verification and Validation /22
Øystein Haugen, Ericsson NorARC, Ifi Uio
Atomic transitions
lAn SDL transition cannot be interrupted
– this means that no other transition from the same process can be
started while it is running
– transitions from other processes may run freely in parallel
lWhat if a process has multiple signalroutes?
– then signals from different signalroutes may merge in any order
just maintaining the order on each signalroute
– therefore to keep the idea of atomic transitions we need to have
multiple input buffers for each component!
process B2
process B1
i2
i1
i1
Øystein Haugen, Ericsson NorARC, Ifi Uio
Verification and Validation /23
Exhaustive Simulation, the simple example
system DiningPhilosophers
[Right,AskLeft]
[Left]
A
B
[Left,AskRight]
[Right]
process A
process B
Idle
Idle
Left
AskRight
Left
WaitR
_
AskLeft
Right
Right
Right
Left
Right
Error
WaitL
_
Error
Left
Left
AskRight
Right
Eat
Right
Left
WaitR
AskRight
AskLeft
Eat
Idle
Verification and Validation /24
AskLeft
Idle
WaitL
Øystein Haugen, Ericsson NorARC, Ifi Uio
Exhaustive Simulation, the execution tree
(Idle[], Idle[])
Right
Left
(WaitR[], Idle[AskRight])
“same”
AskRight
(WaitR[Right], Idle[])
Right
(Idle[], Idle[Right])
(Idle[AskLeft], WaitL[])
(symmetrical to left side)
Right
Right
(WaitR[AskLeft], WaitL[AskRight])
AskLeft
AskRight
(WaitR[], WaitL[AskRight])
(WaitR[AskLeft], WaitL[])
AskRight
AskLeft
(WaitR[], WaitL[])
Verification and Validation /25
Øystein Haugen, Ericsson NorARC, Ifi Uio
Random Exploration
lTo explore some of the total state space, but not all
lGuided exploration
– traversal order
ldepth first: always explore new signals from a node
lbreadth first: explore new signals of the parent node
– other criteria
lpure probability
lSupertrace
Verification and Validation /26
Øystein Haugen, Ericsson NorARC, Ifi Uio
Supertrace in a (very small) nutshell (1/2)
lBiggest practical problem with state exploration is the number
of states that must be recognized
lSecond problem is the speed of the symbolic execution
l“Supertrace” is an algorithm invented by Gerard Holzmann to
overcome both these major problems
Verification and Validation /27
Øystein Haugen, Ericsson NorARC, Ifi Uio
Supertrace (2/2)
lProblem: given a system state S during an execution, to know
quickly whether the execution has covered this state before
– Hash state S onto a numerical range implemented as an address
space A. The result can be called a.
– Look at the bit corresponding to the address a. If the bit in a is on,
this means that the state has been explored before!
lSolution “deficiency”
– The bit(a) is true even when the state has not been explored
– the probability for deficiency is bigger the smaller the A
lSolution improval
– Use another independent hash space and require both bits true
lSolution Coverage of significant state space
– can reach very close to 100% with only 1% of memory space
needed for traditional exhaustive search
Verification and Validation /28
Øystein Haugen, Ericsson NorARC, Ifi Uio