Unbounded Data Model
Verification Using SMT Solvers
Jaideep Nijjar
Tevfik Bultan
University of California, Santa Barbara
ASE 2012
Web Software Everywhere
• Commerce, entertainment, social interaction
• We will rely on web apps more in the future
• Web apps + cloud will make desktop apps obsolete
Acknowledgement: NSF Support
HTTP Status 500 type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: flp.fl_appl_stts not found. Specify owner.objectname or use sp_help to check whether the object e
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:830)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:763)
org.apache.jsp.fastlane_jsp._jspService(fastlane_jsp.java:242)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:675)
org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:642)
org.apache.jsp.index_jsp._jspService(index_jsp.java:44)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
root cause
java.sql.SQLException: flp.fl_appl_stts not found. Specify owner.objectname or use sp_help to check whether the object exists (sp
gov.nsf.fastlane.util.ApplicationStatus.<init>(ApplicationStatus.java:95)
org.apache.jsp.fastlane_jsp._jspService(fastlane_jsp.java:79)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
It is not just NSF
Web Application Dependability
Web Application Dependability
Web Application Dependability is a
Problem
• Web applications are hard to program
• Distributed behavior, interaction among many components and
many languages
• Web applications are hard to test
• Highly dynamic behavior and concurrency
• Web applications are easy targets for hackers
• They are notorious for security vulnerabilities and unreliable
behavior
My research group’s goal:
• Improving dependability of web applications using automated
verification techniques!
We have a hammer
Automated Verification
Unfortunately, life is complicated
Web application
dependability
problems
Automated verification
techniques
Making it work
Separation of concerns
+ modularity
+ abstraction/extraction
Data model
problems
Formal data
model
+
Properties
SMT-based
verification
Three-Tier Architecture
Browser
Web Server
Backend
Database
Three-Tier Arch. + MVC Pattern
Browser
Web
Server
Controller
Model
Backend
Database
• MVC pattern has become the
standard way to structure web
applications:
Views
•
•
•
•
Ruby on Rails
Zend for PHP
CakePHP
Struts for
Java
• Django for
Python
• …
Our Approach
Web
Application
MVC
Design
Pattern
• Ruby on Rails
Data Model
• ActiveRecords
Formal
Model
Automatic
Extraction
Automatic
Translation
+
Automatic
Projection
+
Properties
• Sets and
Relations
Unbounded
Verification
• SMT
Solver
Outline
•
•
•
•
•
•
•
Motivation
Overview of Our Approach
Rails Semantics
Translation to SMT-LIB
Experiments
Related Work
Conclusions
A Rails Data Model Example
Role
class User < ActiveRecord::Base
has_and_belongs_to_many :roles
has_one :profile, :dependent => :destroy
has_many :photos, :through => :profile
end
class Role < ActiveRecord::Base
has_and_belongs_to_many :users
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos, :dependent => :destroy
has_many :videos, :dependent => :destroy,
:conditions => "format='mp4'"
end
class Tag < ActiveRecord::Base
belongs_to :taggable, :polymorphic => true
end
class Video < ActiveRecord::Base
belongs_to :profile
has_many :tags, :as => :taggable
end
class Photo < ActiveRecord::Base
...
*
*
1
User
1
*
0..1
Photo
1
Taggable
*
Tag
*
1
Profile
1
format=.‘mp4’
1
*
Video
Rails Data Models
• Data model verification: Analyzing the relationships
between data objects
• Specified in Rails using association declarations inside
the ActiveRecord files
• The basic relationships
• One-to-one
• One-to-many
• Many-to-many
• Extensions to the basic relationships using Options
• :through, :conditions, :polymorphic, :dependent
The Three Basic Relationships in Rails
• One-to-One (One-to-ZeroOrOne)
class User < ActiveRecord::Base
has_one :profile
end
User
1
.
class Profile < ActiveRecord::Base
belongs_to :user
end
0..1
Profile
.
• One-to-Many
class Profile < ActiveRecord::Base
has_many :videos
end
Profile
1
.
class Video < ActiveRecord::Base
belongs_to :profile
end
*
Video
The Three Basic Relationships in Rails
• Many-to-Many
class User < ActiveRecord::Base
has_and_belongs_to_many :users
end
class Role < ActiveRecord::Base
has_and_belongs_to_many :roles
end
User
*
*
Role
Options to Extend the Basic Relationships
• :through Option
• To express transitive relations
• :conditions Option
• To relate a subset of objects to another class
• :polymorphic Option
• To express polymorphic relationships
• :dependent Option
• On delete, this option expresses whether to delete the associated
objects or not
The :through Option
class User < ActiveRecord::Base
has_one :profile
has_many :photos, :through => :profile
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos
end
class Photo < ActiveRecord::Base
belongs_to :profile
Profile
end
0..1
1
*
1
User
1
*
Photo
The :dependent Option
class User < ActiveRecord::Base
has_one :profile, :dependent => :destroy
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos, :dependent => :destroy
end
User
1
0..1
Profile
1
*
Photo
• :delete directly delete the associated objects without looking
at its dependencies
• :destroy first checks whether the associated objects
themselves have associations with the :dependent option set
Formalizing Rails Semantics
Formal data model: M = <S, C, D>
• S: The sets and relations of the data model (data model schema)
• e.g. { Photo, Profile, Role, Tag, Video, User} and the relations
between them
• C: Constraints on the relations
• Cardinality constraints, transitive relations, conditional relations,
polymorphic relations
• D: Dependency constraints
• Express conditions on two consecutive instances of a relation such
that deletion of an object from the first instance leads to the other
instance
Formalizing Rails Semantics
• I = <O,R> is an instance of the data model M = <S,C,D>,
denoted by I |= M, iff
1. the sets in O and the relations in R follow the schema S, and
2. R |= C
• Given a pair of data model instances I = <O,R> and I’ = <O’,R’>
(I, I’) is a behavior of the data model M = <S,C,D>,
denoted by (I, I’) |= M, iff
1. O and R and O’ and R’ follow the schema S
2. R |= C and R’ |= C, and
3. (R,R’) |= D
Data Model Properties
Given a data model M = <S,C,D>, we define four types of properties:
1. state assertions (AS): properties that we expect to hold for each
instance of the data model
2. behavior assertions (AB): properties that we expect to hold for
each pair of instances that form a behavior of the data model
3. state predicates (PS): properties we expect to hold in some
instance of the data model
4. behavior predicates (PB): properties we expect to hold in some
pair of instances that form a behavior of the data model
Outline
•
•
•
•
•
•
•
Motivation
Overview of Our Approach
Rails Semantics
Translation to SMT-LIB
Experiments
Related Work
Conclusions
Translation to SMT-LIB
• Given a data model M = <S, C, D>
we translate the constraints C and D to formulas in the theory of
uninterpreted functions
• We use the SMT-LIB format
• We need quantification for some constraints
Translation to SMT-LIB
• One-to-Many Relation
RAILS:
SMT-LIB:
class Profile
has_many :videos
end
class Video
belongs_to :profile
end
(declare-sort Profile 0)
(declare-sort Video 0)
(declare-fun my_relation (Video) Profile).
Translation to SMT-LIB
• One-to-One Relation
RAILS:
SMT-LIB:
class User
has_one :profile
end
class Profile
belongs_to :user
end
(declare-sort User 0)
(declare-sort Profile 0)
(declare-fun my_relation (Profile) User).
(assert (forall ((x1 Profile)(x2 Profile))
(=> (not (= x1 x2)) (not (= (my_relation x1) (my_relation x2) ))
) ))
Translation to SMT-LIB
Many-to-Many Relation
RAILS:
SMT-LIB:
class User
has_and_belongs_to_many :roles
end
class Role
has_and_belongs_to_many :users
end
(declare-sort Role 0)
(declare-sort User 0)
(declare-fun my_relation (Role User) Bool)
Translating the :through Option
class Profile <
ActiveRecord::Base
belongs_to :user
has_many :photos
end
class Photo <
ActiveRecord::Base
belongs_to :profile
End
class User <
ActiveRecord::Base
has_one :profile
has_many :photos,
:through => :profile
end
Profile
0..1
1
1
User
1
*
*
Photo
(declare-sort Profile 0)
(declare-sort Photo 0)
(declare-sort User 0)
(declare-fun profile_photo (Photo)
Profile)
(declare-fun user_profile (Profile) User)
(declare-fun user_photo (Photo) User)
(assert (forall ((u User)(ph Photo))
(iff (= u (user_photo ph))
(exists ((p Profile))
(and (= u (user_profile p))
(= p (profile_photo ph)) ))
))
)
Translating the :dependent Option
• The :dependent option specifies what behavior to take on
deletion of an object with regards to its associated objects
• To incorporate this dynamism, the model must allow analysis of how
sets of objects and their relations change from one state to the next
class User <
ActiveRecord::Base
has_one :account,
:dependent => :destroy
end
(declare-sort Profile 0)
(declare-sort User 0)
(declare-fun Post_User (User) Bool)
(declare-fun Post_Profile (Profile) Bool)
.
class Profile <
ActiveRecord::Base
belongs_to :user
end
(declare-fun user_profile (Profile) User)
(declare-fun Post_user_profile
(Profile User) Bool)
Translating the :dependent Option
(assert (not (forall ((x User)) (=> (and
(forall ((a User)) (ite (= a x) (not (Post_User a)) (Post_User a)))
(forall ((b Profile)) (ite (= x (user_profile b))
(not (Post_Profile b)) (Post_Profile b) ))
(forall ((a Profile) (b User)) (ite (and (= b (user_profile a))
(Post_Profile a)) (Post_user_profile a b)
(not (Post_user_profile a b)) ))
) ;Remaining property-specific constraints go here
)))
• Update sets relations of its associated object(s) based on the
use of the :dependent option
• A relation is only updated if it is a :belongs_to or
:has_and_belongs_to_many relationship
• In the database, the foreign key is stored with the object that
has the :belongs_to relationship
Verification
• Once the data model is translated to SMT-LIB format we can
state properties about the data model again in SMT-LIB and
then use an SMT-Solver to check if the property holds in the
data model
• However, when we do that, for some large models, SMTSolver times out!
• Can we improve the efficiency of the verification process?
Property-Based Data Model Projection
• Basic idea: Given a property to verify, reduce the size of the
generated SMT-LIB specification by removing declarations
and constraints that do not depend on the property
• Formally, given a data model M = <S, C, D> and a property p,
(M, p) = MP
where MP = ⟨S, CP, DP⟩ is the projected data model such that
CP ⊆ C and DP ⊆ D
• Key Property: For any property p, M |= p ⇔ (M, p) |= p
• Implemented as part of our tool
• Algorithm Input: Active Record files, property p
• Output: The projected SMT-LIB specification
• Removes constraints on those classes and relations that are not
explicitly mentioned in the property nor related to them based
on transitive relations, dependency constraints or polymorphic
relations
Data Model Projection: Example
Role
Data
Model, M:
*
Property, p:
A User’s Photos are the same
as the User’s Profile’s Photos.
*
1
User
1
*
0..1
Photo
*
1
*
Tag
(M, p) =
1
User
1
1
1
Taggable
Profile
1
*
Video
*
Photo
0..1
*
1
Profile
Verification Overview
Active
Records
Data Model
Properties
Formal Data
Model
Projection
Translator
SMT-LIB
Specification
Counter-example
Data Model
Instance
SMT Solver
(Z3)
Unknown
Verified
Outline
•
•
•
•
•
•
•
Motivation
Overview of Our Approach
Rails Semantics
Translation to SMT-LIB
Experiments
Related Work
Conclusion
Experiments
• We used five open-source Rails apps in our experiments:
•
•
•
•
•
LovdByLess: Social networking site
Tracks: An application to manage things-to-do lists
OpenSourceRails(OSR): Social project gallery application
Fat FreeCRM: Customer relations management software
Substruct: An e-commerce application
LovdBy
Less
LOC 3787
Data Model Classes 13
Tracks
OSR
Fat Free
Substruct
CRM
6062
4295
12069
15639
13
15
20
17
• We wrote 10 properties for each application
Types of Properties Checked
• Relationship Cardinality
Note
• Is an Opportunity always
assigned to some Campaign?
• Transitive Relations
• Is a Note’s User the same as the
Note’s Project’s User?
User
• Deletion Does Not Cause Dangling References
• Are there any dangling Todos after a User is deleted?
• Deletion Propagates to Associated Objects
• Does the User related to a Lead still exist
after the Lead has been deleted?
Project
Experimental Results
• 50 properties checked, 16 failed, 11 were data model errors
• For example in Tracks, a Note’s User can be different than Note’s
Project’s User
• Currently being enforced by the controller
• Since this could have been enforced using the :through option, we
consider this a data-modeling error
• From OpenSourceRails: User deletion fails to propagate to
associated Bookmarks
User
1
*
Bookmark
• Leaves orphaned bookmarks in database
• Could have been enforced in the data model by setting the
:dependent option on the relation between User and Bookmark
Performance
• To measure performance, we recorded
• The amount of time it took for Z3 to run and check the properties
• The number of variables produced in the SMT specfication
• The time and number of variables are averaged over the properties
for each application
• To compare with bounded verification, we repeated these
experiments using the tool from our previous work and Alloy
Analyzer
• The amount of time it took for Alloy to run
• The number of variables generated in the boolean formula generated for
the SAT solver
• Taken over an increasing bound, from at most 10 objects for each class
to at most 35 objects for each class
8
6
2.5
Tracks
2
4
2
0
8
6
25
OSR
20
1.5
15
1
10
0.5
5
0
0
10 15 20 25 30 35
Verification Time (s)
Verification Time (s)
Performance: Verification Time
Substruct
10 15 20 25 30 35
2.5
2
10 15 20 25 30 35
LovdByLess
Alloy
1.5
4
Z3
1
2
0.5
0
Z3+proj
0
10 15 20 25 30 35
FatFreeCRM
10 15 20 25 30 35
Scope
Performance: Formula Size (Variables)
Z3
Alloy
No. Variables (thousands)
No. Variables
200
150
100
50
0
800
600
400
200
0
10 15 20 25 30 35
Scope
non-proj
proj
LovdByLess
Tracks
OSR
Substruct
FatFreeCRM
Unbounded vs Bounded Performance
• Why does unbounded verification out-perform bounded so
drastically?
Possible reasons:
• SMT solvers operate at a higher level of abstraction than SAT
solvers
• Z3 uses many heuristics to eliminate quantifiers in formulas
• Implementation languages are different
• Z3 implemented in C++
• Alloy (as well as the SAT Solver it uses) is implemented in Java
Outline
•
•
•
•
•
•
•
Motivation
Overview of Our Approach
Rails Semantics
Translation to SMT-LIB
Experiments
Related Work
Conclusions
Related Work
• Previous work on Data Model Verification using Alloy
• [Cunha and Pacheco, SEFM 2009] maps relational database schemas
to Alloy; not automated
• [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy,
and uses the Analyzer to produces instances of the data model to
show consistency
• These approaches bounded, not unbounded technique like ours
• [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser
and business logic interactions
• A different class of bugs than the data model related bugs we focus on
• Unbounded verification of Alloy specifications using SMT Solvers
• [Ghazi et al, FM 2011], approach not implemented
• More challenging domain since Alloy language contains constructs
such as transitive closures which do not appoar in the data models
we extract
Related Work
• Specification and Analysis of Conceptual Data Models
• [McGill et al ISSTA 2011, Smaragdakis et al ASE 2009, Halpin et al
IMRD 2009]
• Model-driven (forward engineering) approaches, whereas we
performed model extraction (reverse engineering)
• Formal modeling of Web Applications
• [Book et al ASE 2004, Hallé et al ASE 2010, Han et al MoDELS
2007]
• Focus on navigation aspects as opposed to data model
• SMT Solvers More Efficient than SAT Solvers
• Observed in other verification domains [Cordeiro et al, ASE 2009]
• The data model verification problem we investigate is different
from the problems studied in earlier works
Conclusions
• Goal: To automatically discover data model errors in Ruby on
Rails web applications
• Approach: Automatically extract a formal data model,
translate it to the theory of uninterpreted functions, and
verify using an SMT-solver
• Use property-based data model projection for efficiency
• Implementation: An automatic translator from Rails
ActiveRecords to SMT-LIB
• Handles three basic relationships and several options (:through,
:conditions, :polymorphic, :dependent)
• Experiments: Found multiple data model errors on five open
source applications
• Unbounded verification of data models is feasible and more efficient
than bounded verification!
Future Work
• Analyzing dynamic behavior
• Model object creation in addition object deletion
• Fuse the data model with the navigation model in order
to analyze dynamic data model behavior
• Check temporal properties
• Automatic Property Inference
• Current work requires manual property writing
• Use the inherent graph structure in the the data model to
automatically infer properties about the data model
• Automatic Repair
• When verifier concludes that a data model is violated,
automatically generate a repair that establishes the violated
property
Questions?