Client and Server Verification for Web
Services Using Interface Grammars
Graham Huges, Tevfik Bultan, Muath Alkhalaf
Department of Computer Science
University of California, Santa Barbara
Outline
•
•
•
•
•
•
Motivation
Interface Grammars
Interface Compiler
Interface Grammars for Web Services
A Case Study
Conclusions
Model Checking Software
• Model checking
– An automated software verification technique
– Exhaustive exploration of the state space of a program
to find bugs
• Systematically explore all possible behaviors of a program
– look for violations of the properties of interest
• assertion violations, deadlock
• Software model checkers: Verisoft, Java PathFinder (JPF),
SLAM, BLAST, CBMC
Two Challenges in Model Checking
• State space explosion
– Exponential increase in the state space with increasing
number of variables and threads
• State space includes everything: threads, variables,
control stack, heap
• Environment generation
– Finding models for parts of software that are
• either not available for analysis, or
• are outside the scope of the model checker
Modular Verification
• Modularity is key to scalability of any verification
technique
– Moreover, it can help isolating the behavior you wish
to focus on, removing the parts that are beyond the
scope of your verification technique
• Modularity is also a key concept for successful software
design
– The question is finding effective ways of exploiting the
modularity in software during verification
Interfaces for Modularity
• How do we do modular verification?
– Divide the software to a set of modules
– Check each module in isolation
• How do we isolate a module during verification/testing?
– Provide stubs representing other modules (environment)
• How do we get the stubs representing other modules?
– Write interfaces
• Interfaces specify the behavior of a module from the
viewpoint of other modules
• Generate stubs from the interfaces
Our Approach: Interface Grammars
Here is the basic use case for our approach:
1. User writes an interface grammar specifying the interface
of a component
2. Our interface compiler automatically generates a stub
from the interface grammar
3. Automatically generated stub provides the environment
during modular verification
Interface
Grammar
Interface
Grammar
Interface Grammars
Interface
Compiler
Component B
Component B
Stub
Component A
Model
Checker
Component A
An Example
• An interface grammar for transactions
– Specifies the appropriate ordering for calls to a
transaction manager
Start →
Base →
|
Tail →
|
Base
begin Tail Base
ε
commit
rollback
– Calls are the terminal symbols of the interface
grammar
An Example
• Consider the call sequence
begin rollback begin commit
Start →
Base →
|
Tail →
|
• Here is a derivation:
Start
 Base
 begin Tail Base
 begin rollback Base
 begin rollback begin Tail Base
 begin rollback begin commit Base
 begin rollback begin commit
Base
begin Tail Base
ε
commit
rollback
Another Example
• This example can also be specified as a Finite State
Machine (FSM)
begin
commit
rollback
• However, the following grammar which specifies nested
transactions cannot be specified as a FSM
Start →
Base →
|
Tail →
|
Base
begin Base Tail Base
ε
commit
rollback
Yet Another Example
• Let’s add another operation called setrollbackonly
which forces all the pending transactions to finish with
rollback instead of commit
• We achieve this by extending the interface grammars with
semantic predicates and semantic actions
Start →
Base →
Tail
|
|
→
|
«r:=false; l:=0» Base
begin «l:=l+1» Base Tail
«l:=l-1; if l=0 then r:=false» Base
setrollbackonly «r:=true» Base
ε
«r=false» commit
rollback
Interface Grammar Translation
• Our interface compiler translates interface grammars to
executable code:
– the generated code is the stub for the component
• The generated code is a parser that
– parses the incoming calls
– while making sure that the incoming calls conform to the
interface grammar specification
Interface
Grammar
Verification with Interface Grammars
parser stack
Component Stub
Interface
Compiler
parse
table
Top-down
parser
semantic
predicates
and
semantic
actions
Program
invocation
(lookahead)
Model
Checker
Interface Grammars: Client vs. Server
• Interface grammars can be used for
– Client verification: Generate a stub for the server
– Server verification: Generate a driver for the server
Interface
Parser
Stub
Client
Driver
Server
Interface
Compiler
Sentence
Generator
Interface Grammars and Data
• A crucial part of the interface specification is specifying the
allowable values for the arguments and generating
allowable return values
• Approach 1: These can be specified in the semantic
actions and semantic predicates of the interface grammars
• Approach 2: Can we specify the constraints about the
arguments and return values using the grammar rules?
– Yes, grammar productions can be used to specify the
structure of most recursive data structures.
Modular Verification of Web Services
• We applied our modular verification approach based on
interface grammars to client and server side verification of
Web services
Interface Grammars for Web Services
Our approach:
1. A WSDL-to-interface grammar translator automatically
generates grammar productions that generate and/or
validate XML arguments and return values
2. User adds control flow constraints by modifying the
grammar
3. Interface compiler automatically generates a stub for client
side verification and a driver for server-side verification
Interface Grammars for Web Services
A Case Study: AWS-ECS
• We performed both client and server side verification for
the Amazon E-Commerce Web Service (AWS-ECS) using
our approach
• AWS-ECS WSDL specification lists 40 operations
– that provide differing ways of searching Amazon’s
product database
• We focused on the following core operations:
– ItemSearch, CartCreate, CartAdd, CartModify,
CartGet, CartClear
Client Verification
• For client verification we used a demonstration client
provided by Amazon: AWS-ECS Java Sample
• This client does not check any constraints such as
– You should not try to insert an item to a shopping cart
before creating a shopping cart
• When such requests are sent to AWS-ECS they would
return an error message
• Using our approach we can easily check if the client allows
such erroneous requests
• We used the Java PathFinder (JPF) model checker
– Falsification time changes with the type of faults we are
looking for (data or control errors), changes from 10 to
60 seconds
Client Verification Setup
Test
Driver
Hand written
Driver
Web Service
Client
Server Stub
AWS-ECS
Java Sample
Automatically
Generated Stub/Parser
• The AWS-ECS Client uses Apache Axis SOAP library
• Given a WSDL specification, Apache Axis automatically
generates Java classes as wrappers for SOAP calls
• We replaced the Java classes generated by Apache Axis
and forwarded them to our automatically generated
parser/server stub
• We wrote a driver which non-deterministically generates
input events for the Client, simulating user behavior
• JPF model checker exhaustively explores all nondeterministic choices (both in the driver and the stub)
Input Validation Check
• We checked if the client performs input validation upon
receiving an input from the user before passing the
corresponding request to the server
• Possible erroneous inputs by the user
– Type errors
• For example: Entering a string for a numeric field
– Nonsensical data
• For example: Adding a nonexistent item to a
nonexisting shopping cart
– Uncorrelated data
• For example: Search for an item but then try to insert
another nonexisting one
Input Validation Experiments with JPF
Type
Time (sec.)
Memory (MB)
Type error
12.5
25
Nonsensical data
11.1
25
Uncorrelated data 20.8
43
Control-Flow Validation Check
• Call sequences that can lead to errors
– Such as modifying the contents of a cart after clearing it
• We wrote a driver that:
– First initializes the cart and
– Then generates a sequence of user events nondeterministically
• We experimented for different event sequence sizes
Control-Flow Validation Experiments
Type
Depth
Time
(sec.)
Memory
(MB)
Errors
First error
2
31.8
43
0
First error
3
64.2
62
1
First error
4
49.6
73
1
First error
5
57.3
73
1
All errors
2
31.8
43
0
All errors
3
77.3
62
2
All errors
4
266.8
112
15
All errors
5
862.6
230
68
AWS-ECS: Server Verification
• Our interface compiler automatically generates a driver that
sends sequences of requests to AWS-ECS server and
checks that the return values conform to the interface
specification
• The driver is a sentence generator
– It generates sequences of SOAP requests based on the
interface specification
• We used two algorithms for sentence generation:
– A random sentence generation algorithm
– Purdom’s algorithm: A directed sentence generation
algorithm that guarantees production coverage
Directed Sentence Generation
• Number of sentences generated: 5
• Average derivation length: 24
• Average number of SOAP requests/responses: 3.8
• Verification time: 20 seconds
Random Sentence Algorithm
• Number of sentences generated: 100
• Average derivation length: 17.5
• Average number of SOAP requests/responses: 3.2
350
100%
300
80%
250
200
60%
time in
seconds
150
Percentage of
covered productions
Percentage of
covered nonterminals
40%
100
20%
50
0%
0
0
20
40
60
80 100 120
Number of executions
0 5 10 15 20 25 30 35 40 45 50
Number of exceutions
Server verification
• We found two errors during server side verification
– Errors were discovered within 10 seconds
• These errors indicate a mismatch between the interface
specification and the server implementation
• It may mean that we misunderstood the description of the
Web service interface
• It may also mean that there is an error in the service
implementation
Conclusions
• Modular verification is a necessity
• Interfaces are crucial for modular verification
• Interface grammars provide a new specification
mechanism for interfaces
• Interface grammars can be used for automated stub
generation leading to modular verification for both client
and server for verification of Web Services
THE END
Related Work: Interfaces
•
•
•
•
•
L. de Alfaro and T. A. Henzinger. Interface automata.
O. Tkachuk, M. B. Dwyer, and C. Pasareanu. Automated
environment generation for software model checking.
A. Betin-Can and T. Bultan. Verifiable concurrent
programming using concurrency controllers.
T. Ball and S. K. Rajamani. SLAM interface specification
language.
G. T. Leavens et al.: JML
Related: Grammar-based Testing
• A. G. Duncan, J. S. Hurchinson: Using attributed grammars
to test designs and implementations
• P. M. Maurer: Generating test data with enhanced context
free grammars
• P. M. Maurer: The design and implementation of a
grammar-based data generator
• E. G. Sirer and B. N. Bershad: Using production grammars
in software testing