Experiences in using formal techniques and
technologies in real-life applications
Alexander Petrenko,
Institute for System Programming of Russian
Academy of Sciences (ISPRAS)
January, 2001
Topics
Case study of industrial application of
formal specification for testing software
New opportunities of the approach
RedVerst background (1994-1999)
RedVerst - R&D in Verification, Specification and Testing
1994. Nortel’s tender and ISPRAS background
Nortel’s request: Formal specification of OS kernel API and
automated regression testing based on the specification hidden
state
 ISPRAS background: Soviet space programs, real-time
language, compilers, formal techniques applied to specification
of networks and programming languages

1996. First results
OS kernel API was fully specified
 Technology for test generation
 The errors was detected during formal specification and testing

1999. Feasibility proved
over 6 subsystems have been verified (total size over 400Kline)
 6 patent applications have been filed (property of Nortel Networks)

RedVerst background (2000)
KVEST*) is introduced in Nortel Networks stream of regression
testing and delivery
Protocol specification and verification techniques are developed
and introduced
Parser and compiler specification and verification techniques
designed
Microsoft Research IPv6 verification project started
VppTest tool for co-verification process prototyped
UniTesK concept is developed
J@T project started (under contract with ATS APS)
___________________
*) KVEST technology is property of Nortel Networks
Agenda
KVEST: verification of Nortel Networks software
Lessons learned
Co-verification process and VppTest tool
UniTesK concept and Java extension for
specification and verification
Selection range
Testing
Approach
White box
Specification
Languages
VDM-SL, Z
SDL, LOTOS
RAISE/RSL
In-house PL extension
Explicit models
Pre- and post-conditions
Algebraic specification
Input test vectors
Test sequences
Test drivers&Test oracles
Test coverage metrics
Stubs
Specification
Style
Test
components
generated
automatically
Black box
Selection results
Testing
Approach
Black box
White box
Specification
Languages
VDM-SL, Z
Specification
Style
Explicit models
SDL, LOTOS
RAISE/RSL
In-house PL extension
Pre- and post-conditions
Algebraic specification
Test
components
generated
automatically
Input test vectors
Test sequences
Test oracles
Test coverage metrics
Stubs
KVEST technology stream
Documentation
Source code
Interface
definition
Software contract
contents
Interface A1
…………………….
Interface A2
…………………….
Specification
Test drivers
and test vectors
Test suite
production
Detected errors &
test coverage
reports
Legend
- source data
Test plans
Test execution
analysis
- results
- phases /
works
Test execution scheme
Test plans
UNIX workstation
Test suite composer
Test suite
Test vectors and sequences
Test drivers
Program
behavior model
Analysis
Verdict and
trace
SUT
Target platform
Repository
Test execution management
Unix workstation
Navigator:
- test suite generation
- test plan run
- test report browser
Target platform
Test suite
Script driver
Manually
developed
Generated
Repository
Test bed:
- process control
- communication
- basic data conversion
Test execution management
Unix workstation
Navigator
- test suite generation
test plan
-- test
plan run
- test report browser
Target platform
Test suite
Script
driver
Script
driver
Data iterators,
converters,…
Basic driver
SUT
Repository
Test bed
- processcontrol
control &
- process
communication
- communication
- data conversion
- basic data conversion
Test drivers and test plans
Test plan: loads the script drivers, calls them with
definite options, and monitors their execution
Script driver: generates sets of input parameters, calls
basic drivers, evaluates results of the test sequences as a
whole
Basic test driver: tests single procedure by receiving
input, calling the procedure, recording output, assigning a
verdict
Test generation scheme
Manually
Developed
Components
RAISE specifications
Basic driver
generator
Tools
(under UNIX/Windows)
Script driver
skeletons
Script driver
generator
Test vector generator
RAISE -> target language compiler
Object files for
target platform
Basic drivers
Test suites
Test vectors
Script drivers
Test generation, details
Script driver
skeletons
Manually
Developed
Components
Data converters
Iterators
RAISE specifications
State observers
Basic driver
generator
Test vector
generator
Tools
Script driver
generator
RAISE -> target language compiler
Object files for target
platform
Basic
drivers
Test suites
Test
vectors
Script drivers
Filters
Iterators
Data converters
KVEST facilities
Facilities for test generation
Fully automated
Data iterations
For integer and boolean
Basic drivers (test oracles)
For any specs
Filtering
For any specs
Test sequence generation
For any specs and well-factorized
FSM
Accurate domain partition
Based on pre- and post-conditions
FSM extraction from
specification
Automatic FSM traversing
FSM factorization
Manual techniques
In other cases
Formalized technique
Note. 6 patent applications have been filed for KVEST techniques
KVEST statistics (KIL project).
Ratio: Handmade vs. Generated components
Script
drivers
250
200
150
Basic
drivers
KLOC
Hand made
100
Generated
50
0
Scenarios
Specifications
Agenda
KVEST: verification of Nortel Networks software
Lessons learned
Co-verification process and VppTest tool
UniTesK concept and Java extension for
specification and verification
Lessons learned
Feasibility proven:
applicable for real-life software including case of



hidden state
non-determinism
asynchronous processes
Key engineering issues:



Incremental testing and incremental specification
Re-generation automation
Specification re-use vs. test suite re-use
Software engineers’ resistance reasons



Non traditional (specification) language
Non-open test suite architecture
No patterns for specification
Agenda
KVEST: verification of Nortel Networks software
Lessons learned
Co-verification process and VppTest tool
UniTesK concept and Java extension for
specification and verification
Goals of VppTest project
Improved software development process
integrated design, specification, and
verification in concurrent fashion:
co-verification process
Prototype tools for supporting the process
software development process
Gap
Functionality definition
Design
Implementation
System testing
Integration testing
Unit testing
Co-verification process
Draft specification
Functionality definition
Test suite
Prototype
Decomposition
specification
Design
Test suite
Prototypes of
subsystems
Unit specification
Implementation
Test suite
Unit implementation
VppTest & Environment
Rational Rose
IFAD VDMTool box
VppTest
What is VDM & VDMTools®?
VDMTools® is integrated environment for
development and execution of VDM-SL and
VDM++ models
VDM-SL is one of most used specification
languages in the world.
There is an ISO standard for VDM-SL
VDM++ is OO extension of VDM-SL
VDMTools® is ideal in modeling, prototyping, and
implementation of safety-critical software
Code generation directly to target systems in
C++/Java
Vide: http://www.ifad.dk/
VDM++ specification example*):
Digital Multiplexed Radio Telephone System
Remote
Station (RS)
Remote
Station (RS)
Subscriber
Subscriber
Subscriber
Subscriber
Remote
Station (RS)
Subscriber
Subscriber
To Public Switching Telephone Network (PSTN)
Remote
Station (RS)
Central Station (CS)
Subscriber
Subscriber
*) - http://www.iist.unu.edu/newrh/III/1/page.html
Roderick Durmiendo and Chris George. Formal Development of a Digital Mutiplexed Radio-Telephone System.
Research Report 67, UNU/IIST, P.O.Box 3058, Macau, Feb 1996
Specification and tests
architecture for prototype level
<<env>>
Phone
Constraints
and executable
specification
+network
SimpleNetwork
-phoneMap
SCRD
<<generated>>
SCRDEngine
+scrd
+tracer
SCRD_SimpleNetwork_generic
Tracer
<<generated>>
Legend:
-tracer
manually
developed classes
pre-built classes
OracleSimpleNetwork
+object
SCRD_SimpleNetwork
generated classes
Software to be developed
Remote Station Software
•POTS (Plain-OldTelephone-Service) for
subscribers
Central Station Software
•Handling request for
trunks from RSs
•Request for trunk to CS
•Switching trunks
•Linking phone to trunk
•Connecting RSs
•Sending/receiving data
to/from CS and phone
•Sending/receiving
data to/from RS
Specification and tests for prototype level






constraint specification and executable one SimpleNetwork.vpp
generic oracle – SimpleNetworkOracle.vpp (to regenerate it in
tool, select “simple_network.vdm” in “Specifications” tab, rightclick and select “Generate oracle” in menu)
pre-built files for script drivers: scrd.vpp, scrd_engine.vpp,
tracer.vpp
generic files for script driver: SimpleNetworkScrdGen.vpp,
SimpleNetworkScrdTemplate.vpp (to regenerate it, go to “Scrd
bases” tab, select “SimpleNetwork” in combo box, right-click
near it and select “Generate”)
manually developed script driver scrdSimpleNetwork.vpp
manually developed classes emulating system environment:
Phone.vpp
VDMToolbox-Rational Rose-VppTest
demo bundle (1)
VDMToolbox and VppTest conduct
specification analysis and test suite generation
VDMToolbox-Rational Rose-VppTest
demo bundle (2, 3)
VDMToolbox provides VDM++ mapping to UML
Rational Rose provides browsing VDM++ classes and
supports UML class diagram viewing
VDMToolbox-Rational Rose-VppTest
demo bundle (4)
VppTest runs test of SimpleNetwork
VDMToolbox-Rational Rose-VppTest
demo bundle (5)
VDMToolbox visualizes trace of test execution
in dynamics
The test has finished
Final verdict is OK
Agenda
KVEST: verification of Nortel Networks software
Lessons learned
Co-verification process and VppTest tool
UniTesK concept and Java extension for
specification and verification
UniTesK
UniTesK – Unified Testing & specification tool Kit
Predecessors:



ADL – Sun Microsystems,
Eiffel,
Barbara Liskov’s works
Key principles:





Joint consideration of specification and test artifacts: verification suite
Extension of usual programming languages
No extra features
Open test suite architecture
Could be integrated with software development and modeling environment
J@va: an extension of Java for specification and test design
J@K: Toolkit for J@va*)
___________________________
*) - The toolkit is property of ATS APS
Artifact space triplet. Details
Functional Specification
Test
Specification
Executable Test Suite
System
under test
J@va: projection of UniTesK on Java.
Specification example
/**
Object deq()
returns it,
- removes the head object from the queue and
updates the state of the queue
*/
specification synchronized Object deq()
updates items.?
{
pre { return !items.isEmpty(); }
post
{
if(items.size() == 1)
branch "Single element in the queue";
else
branch "Several elements in the queue";
QueueSpecifications old_state = @ clone();
old_state.items.insertElementAt(deq, 0);
}
}
}
return @ clone().equals(old_state);
Java is extended with
Java features
extended
•
•
•
Specification
classes
Exception
handling
Inheritance
Additional
facilities
Pre- and postconditions
Invariants
Variable access
description
Optional
features
Test scenarios
Axioms and
equations
Test oracle environment
Test oracle / target class link
Test execution
Inheritance in specifications and test
oracles
Specification suite and scenario
Example: PriorityQueue
Element
Method PQueue
Interface
Method isEmpty
boolean isEmpty()
Method enq
void enq(Object obj, int
prty)
Adds to the queue element obj
with priority prty. The added
element is inserted in the queue as
last element with the priority
specified.
Method enq without priority
Void enq(Object obj)
Adds to the queue the element
obj with minimal priority.
Method deq
Object deq()
Removes the first element with
maximal priority from the queue
and returns this element.
Constant MIN_Priority
final static
int MIN_Priority
The minimum value of priority
represented as integer.
Constant MAX_Priority
final static
int MAX_Priority
The maximum value of priority
represented as integer.
PQueue()
Semantics
Constructor making an empty
queue.
Returns true if the queue is empty,
returns false in other case.
List of J@va files
Text of QueueSpecification class can be found in the file QueueSpecification.jvs
Text of PQueueSpecification class can be found in the file PQueueSpecification.jvs
Text of Priority class can be found in the file Priority.jvs
Text of PQueueMediator class can be found in the file PQueueMediator.jvs
Text of PriorityMediator class can be found in the file PriorityMediator.jvs
Text of PQueueTestScenario class can be found in the file PQueueTestScenario.jvs
Text of PQueueGenState class can be found in the file PQueueGenState.java
Auxiliary classes ObjectIterator and PriorityIterator are contained in ObjectIterator.java and
PriorityIterator.java correspondingly.
J@T demo environment (1).
Specification space
Specifications of enq and deq methods
are lpresented in right window
J@T demo environment (2).
Specification space
Size of Queue specification is 3305 bytes
J@T demo environment (3).
Specification space
The test oracles and model of implementation state
are generated from the specification
Size of these generated test suite components
is 7 times bigger
JavaTesK demo environment (4).
Test scenario space
Test scenario, size is 2588 bytes
JavaTesK demo environment (5).
Test scenario space
The generated script driver
is 4 times bigger
Further plans
Specification Based Testing
Current State of the Art:
•Assertions
•Coverage of assertions
Critical mass
Users’ requirements:
•Infrastructure for specification
and test generation
•Tight integration with software
development environment
•Patterns for specification and
test scenarios
•Language and instrumental
support for re-use
References
1. I.Bourdonov, A.Kossatchev, A.Petrenko, and D.Galter. KVEST:
Automated Generation of Test Suites from Formal Specifications.
Proceedings of World Congress of Formal Methods, Toulouse, France,
LNCS, No. 1708, 1999, pp. 608-621.
2. I.B.Bourdonov, A.S.Kossatchev, V.V.Kuliamin. Using Finite State
Machines in Program Testing. "Programming and Computer
Software", 2000, №2.
3. A.Petrenko, A.Vorobiev. Industrial Experience in Using Formal Methods
for Software Development in Nortel Networks.- Proc. Of the TCS2000
Conf., Washington.,DC, June, 2000.
4. A.K.Petrenko, I.B.Bourdonov, A.S.Kossatchev, V.V.Kuliamin.
Experiences in using testing tools and technology in real-life
applications.- Proceedings of SETT’01, India, Pune, 2001.
Contacts
Alexander K.Petrenko
109004, B.Kommunisticheskaya, 25,
Moscow, Russia.
E-mail: petrenko@ispras.ru
Web: http://ispras.ru/~RedVerst
Phone: 007 (095) 912-5317
Fax: 007 (095) 912-1524