An Integrated Data Model Verifier
with Property Templates
Jaideep Nijjar
Ivan Bocic
Tevfik Bultan
{jaideepnijjar, bo, bultan}@cs.ucsb.edu
University of California, Santa Barbara
Department of Computer Science
Verification Lab
FormaliSE 2013
Web Software Everywhere
• Commerce, entertainment, social interaction
• We will rely on web apps more in the future
• Web apps + cloud will make desktop apps obsolete
Model-View-Controller Pattern
DB
Model
• MVC pattern has
become the standard
way to structure web
applications
•
•
•
•
•
•
Ruby on Rails
Zend for PHP
CakePHP
Struts for Java
Django for Python
…
View
• Benefits of the
MVC pattern:
Controller
• Separation of
concerns
• Modularity
• Abstraction
A Data Model Verification Approach
MVC
• Ruby on Rails
Application
Data
Model
MVC
Design
Principles
Automatic
Extraction
• ActiveRecords
Formal
Model
Add data
model properties
• Sets and
Relations
Verification
• Bounded
and
Unbounded
iDaVer: Integrated Data Model Verifier
Active Records
Properties
(expressed using
Templates)
Choice of
technique
Model Extraction
formal data model +
property + technique
Verification
Bounded
(Alloy)
Property
Verified
Unbounded
(FOL Thm Prover)
Property Failed +
Counterexample
Unbounded
(SMT Solver)
Unknown
Outline
• Motivation
• Overview of Approach
• Rails Data Models
• Basic Relations
• Options to Extend Relations
• Formalization of Semantics
•
•
•
•
Property Templates
Verification Techniques
Case Study
Conclusions and Future Work
A Rails Data Model Example
class User < ActiveRecord::Base
has_and_belongs_to_many :roles
has_one :profile, :dependent => :destroy
has_many :photos, :through => :profile
end
class Role < ActiveRecord::Base
has_and_belongs_to_many :users
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos, :dependent => :destroy
has_many :videos, :dependent => :destroy,
:conditions => "format='mp4'"
end
class Tag < ActiveRecord::Base
belongs_to :taggable, :polymorphic => true
end
class Video < ActiveRecord::Base
belongs_to :profile
has_many :tags, :as => :taggable
end
class Photo < ActiveRecord::Base
...
Role
*
*
1
User
1
*
0..1
Photo
1
Taggable
*
Tag
*
1
Profile
1
format=.‘mp4’
1
*
Video
Rails Data Models
• Data model verification: Analyzing the relationships
between data objects
• Specified in Rails using association declarations inside
the ActiveRecord files
• The basic relationships
• One-to-one
• One-to-many
• Many-to-many
• Extensions to the basic relationships using Options
• :through, :conditions, :polymorphic, :dependent
The Three Basic Relationships in Rails
• One-to-One
class User < ActiveRecord::Base
has_one :profile
end
User
1
.
class Profile < ActiveRecord::Base
belongs_to :user
end
1
Profile
.
• One-to-Many
class Profile < ActiveRecord::Base
has_many :videos
end
Profile
1
.
class Video < ActiveRecord::Base
belongs_to :profile
end
*
Video
The Three Basic Relationships in Rails
• Many-to-Many
class User < ActiveRecord::Base
has_and_belongs_to_many :users
end
class Role < ActiveRecord::Base
has_and_belongs_to_many :roles
end
User
*
*
Role
Options to Extend the Basic Relationships
• :through Option
• To express transitive relations
• :conditions Option
• To relate a subset of objects to another class
• :polymorphic Option
• To express polymorphic relationships
• :dependent Option
• On delete, this option expresses whether to delete the associated
objects or not
The :through Option
class User < ActiveRecord::Base
has_one :profile
has_many :photos, :through => :profile
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos
end
class Photo < ActiveRecord::Base
belongs_to :profile
end
Profile
0..1
1
*
1
User
1
*
Photo
The :dependent Option
class User < ActiveRecord::Base
has_one :profile, :dependent => :destroy
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos, :dependent => :destroy
end
User
1
0..1
Profile
1
*
Photo
• :delete directly delete the associated objects without looking
at its dependencies
• :destroy first checks whether the associated objects
themselves have associations with the :dependent option set
Formalizing Rails Semantics
Formal data model: M = <S, C, D>
• S: The sets and relations of the data model (data model schema)
• e.g. { Photo, Profile, Role, Tag, Video, User} and the relations
between them
• C: Constraints on the relations
• Cardinality constraints, transitive relations, conditional relations,
polymorphic relations
• D: Dependency constraints
• Express conditions on two consecutive instances of a relation such
that deletion of an object from the first instance leads to the other
instance
Outline
•
•
•
•
•
•
•
Motivation
Overview of Approach
Rails Data Models
Property Templates
Verification Techniques
Case Study
Conclusions and Future Work
Property Templates
• User-friendly and makes it easy to express properties
• Manually writing properties is a time-consuming and error-prone
process
• Requires familiarity with input modeling language of solver
• Templates are language-neutral
• Do not require familiarity with SMT-LIB, Spass and Alloy
languages, and understanding of output specifications
• Make it easy to rerun the tool and switch the verification
technique, by not requiring the user to rewrite the property
• Eight property templates available for the most common data
model properties
Property Templates
• alwaysRelated[classA, classB]
• To check that objects from classA are always related to objects of
classB
• someMultipleRelated[classA, classB]
• To check that it is possible for objects of classA to be related to
more than one object of classB
• someUnrelated[classA, classB]
• To check that it is possible for an object of classA to not be
related to any objects from classB
• transitive[classA, classB, classC]
• To check that the relation between classA and classC is the
composition of the relations between classA and classB, and
classB and classC
Property Templates
• noDangling[classA, classB]
• To check that when an object of classA is deleted, there are no
objects from classB that are left with a dangling reference to the
deleted object
• deletePropagates[classA, classB]
• To check that when an object of classA is deleted, related objects
in classB are also deleted
• noDeletePropagation[classA, classB]
• To check that when an object of classA is deleted, related objects
in classB are not deleted
• noOrphans[classA, classB]
• To check that deleting an object from classA does not cause
related objects in classB to be orphaned
• An orphan is an object that is related to no other object
Outline
•
•
•
•
•
Motivation
Overview of Approach
Rails Data Models
Property Templates
Verification Techniques
• Bounded Verification with Alloy
• Unbounded Verification with SMT Solver
• Unbounded Verification with FOL Theorem Prover
• Case Study
• Conclusions and Future Work
Active Records
Choice of
technique
Properties
iDaVer
UNBOUNDED VERIFICATION
Model Extraction
SMT-LIB
Encoder
formal data model + property
formula
SMT
Solver
BOUNDED VERIFICATION
Alloy
Encoder
Results
Interpreter
formula
Alloy
Analyzer
Results
Interpreter
Property
Verified
FOL
Encoder
instance
or unsat
Property Failed +
Counterexample
instance
or unsat
or unknown
formula
Theorem
Prover
Unknown
Results
Interpreter
proof found or
completion found
or timeout
Active Records
Choice of
technique
Properties
Bounded
Verification
UNBOUNDED VERIFICATION
Model Extraction
SMT-LIB
Encoder
formal data model + property
formula
SMT
Solver
BOUNDED VERIFICATION
Alloy
Encoder
Results
Interpreter
formula
Alloy
Analyzer
Results
Interpreter
FOL
Encoder
instance
or unsat
Property Failed +
Counterexample
formula
Theorem
Prover
Results
Property
Verified
instance
or unsat
or unknown
Unknown
Interpreter
proof found or
completion found
or timeout
Alloy Language
• A declarative specification language for object modeling
• Based on first-order logic
• Set-based representation of objects
• Defines sets of objects using signatures (sigs)
• Defines relations using fields inside the signatures
• Add additional constraints about the model as facts
• Well-suited for formally specifying data models
• Can add assertions to specify properties about the
specification
Alloy Analyzer
• Automated verification of Alloy specifications is undecidable if
the domains are not bounded
• To ensure decidability, Alloy Analyzer restricts the domains to
a finite scope
• User-specified
• A finite bound on the sizes of the domains
• SAT-based bounded verification
• Alloy Analyzer translates the Alloy verification query to a Boolean
logic formula satisfiability query
• Then invokes an off-the-shelf SAT-solver
Sample Translation to Alloy
class User < ActiveRecord::Base
has_one :profile
end
class Profile < ActiveRecord::Base
belongs_to :user
end
sig Profile {}
sig User {}
one sig State {
profiles: set Profile,
users: set User,
relation: Profile lone -> one User
}
Bounded Verification of Data Models
• Automatically translate the
formal data model extracted
from the Active Records and
the property to Alloy
• User may specify a bound or
use the default
• Use Alloy Analyzer to perform
bounded verification
• Possible outputs:
• Assertion holds within the
given bound
• A counterexample proving
falsified assertions
Formal data model
+ Property
+ Bound
BOUNDED VERIFICATION
formula
Alloy
Encoder
Alloy
Analyzer
instance
or unsat
Property Verifies
(within bound)
Results
Interpreter
Property Fails +
Counterexample
Active Records
Choice of
technique
Properties
Unbounded
with SMT Solver
UNBOUNDED VERIFICATION
Model Extraction
SMT-LIB
Encoder
formal data model + property
formula
SMT
Solver
BOUNDED VERIFICATION
Alloy
Encoder
Results
Interpreter
formula
Alloy
Analyzer
Results
Interpreter
FOL
Encoder
instance
or unsat
Property Failed +
Counterexample
formula
Theorem
Prover
Results
Property
Verified
instance
or unsat
or unknown
Unknown
Interpreter
proof found or
completion found
or timeout
Satisfiability Modulo Theories (SMT)
• SMT-solvers automatically check the satisfiability of first-order
formulas with respect to a set of theories
• Typical theories include: Linear Arithmetic, Arrays, Bit Vectors,
Equality with Uninterpreted Functions
• Only implicit universal quantification
• Theory of Equality with Uninterpreted Functions.
• Language: Variables, Constants, Uninterpreted function symbols,
Predicate ‘=‘, and Boolean connectives
• Uninterpreted functions have no properties except their
signature and functional consistency: a = b => F(a) = F(b)
• Example formula: F(x) = F(G(y)) ˅ H(x,y) = 1
SMT Solvers
• There are many SMT solvers out there that support popular
theories
• However many of them are not suitable for us
• we need support for quantified formulas to handle the property
templates
• Microsoft’s Z3 supports quantified expressions in the theory
of uninterpreted functions
• Uses heuristics for eliminating quantifiers in formulas
• May return ‘unknown’ during satisfiability check
SMT-LIB
•
•
•
•
Defines a standard input language for SMT Solvers
Defines theories and logics in which formula can be written
Lisp-like format: Specifications are a series of s-expressions
Declare types using declare-sort command
• (declare-sort User)
• Declare uninterpreted functions using declare-fun
command
• (declare-fun foo (Domain) Range)
• Quantifier commands: (forall )and (exists )
• Add constraints and properties using assert command
• Check satisfiability by using (check-sat) command
Sample Translation to SMT-LIB
class User < ActiveRecord::Base
has_one :profile
end
class Profile < ActiveRecord::Base
belongs_to :user
end
(declare-sort User)
(declare-sort Profile)
(declare-fun relation (Profile) User)
(assert (forall ((p1 Profile)(p2 Profile))
(=> (not (= p1 p2))
(not (= (relation p1) (relation p2) ))
)
))
Unbounded Verification using SMT Solvers
• Automatically translate formal
data model and property into
the theory of uninterpreted
functions with quantification
(SMT-LIB)
• Use the SMT solver Z3 to
perform satisfiability check
• For failing assertion properties,
our tool interprets outputs and
forms a counterexample
• Unknowns (and timeouts)
possible since the theory is
undecidable
Formal data model
+ Property
UNBOUNDED VERIFICATION
Property
Verified
SMT-LIB
Encoder
formula
SMT
Solver
instance
or unsat
or unknown
Results
Interpreter
Property Failed +
Counterexample
Unknown
Active Records
Choice of
technique
Properties
Unbounded
Verification
UNBOUNDED VERIFICATION
Model Extraction
SMT-LIB
Encoder
formal data model + property
formula
SMT
Solver
BOUNDED VERIFICATION
Alloy
Encoder
Results
Interpreter
formula
Alloy
Analyzer
Results
Interpreter
Property
Verified
FOL
Encoder
instance
or unsat
Property Failed +
Counterexample
instance
or unsat
or unknown
formula
Theorem
Prover
Unknown
Results
Interpreter
proof found or
completion found
or timeout
FOL Theorem Provers
• Rails data models and properties are expressible in first-order
logic (FOL) with equality and quantifiers
• Note that this is an undecidable theory
• There are automated theorem provers for first-order logic
• They use search strategies to find proofs
• However due to undecidability of the FOL they cannot always give
a definite answer
• We use the FOL theorem prover, Spass
Modeling Active Records using FOL
• Model object types by declaring a unary predicate
• Returns true if an object is a member of that class
• Model relations between data objects using binary predicates
• Returns true if the two objects are related
• Axioms are used to express constraints on the data model, e.g.
• Specifying cardinality, dependency, and transitivity constraints on
relations
• Specifying that predicates denoting classes not related by inheritance
are mutually exclusive
• Conjectures are used to model the property to be checked
• To verify the property holds on the data model, send the following
formula to the theorem prover:
axioms => conjecture
Sample Translation to Spass
list_of_symbols.
sorts[Profile, User].
predicates[(relation, 2)].
end_of_list.
list_of_formulae(axioms).
formula(forall([Profile(a)], not(User(a)))).
formula(forall([User(a)], not(Profile(a)))).
formula(forall([a, b], implies(
relation(a, b), and(Profile(a), User(b))))).
formula(forall([a, b1, b2], implies(
and(relation(b1,a), relation(b2,a)), equal(b1,b2)))).
formula(forall([a, b1, b2], implies(
and(relation(a,b1), relation(a,b2)), equal(b1,b2)))).
formula(forall( [Profile(a)],
exists([b], relation(a, b)) )).
end_of_list.
Unbounded Verification of Data Models
using Theorem Provers
• Automatically translate formal
data model and property into
Spass’s input language (FOL)
• Use the theorem prover Spass
to see if formula provable
• Interpret results to determine
whether property holds
• Does not produce
counterexamples since
theorem provers are not
designed to do so
• If Spass times out (due to
undecidability of theory),
iDaVer returns ‘Unknown’
Formal data model
+ Property
UNBOUNDED VERIFICATION
formula
FOL
Encoder
Theorem
Prover
proof found or
completion found
or timeout
Property
Verified
Results
Interpreter
Property Failed
Unknown
Outline
•
•
•
•
•
•
•
Motivation
Overview of Approach
Rails Data Models
Property Templates
Verification Techniques
Case Study
Conclusions and Future Work
Case Study
• LovdByLess, a social networking application
• LOC: 3787
• Number Active Record files: 13
• iDaver input:
• Path of the directory containing the Active Record files
• Name of the file containing the properties to check
(Expressed using property templates!)
• Verification technique
Case Study
• Check: alwaysRelated[Photo, Profile]
• Solver: Spass
+ Unbounded verification
− No sample instance
− May report unknown or timeout
Case Study
• Check: someUnrelated[ForumTopic, ForumPost]
• Solver: Z3
+ Unbounded verification + Sample instance
− May report unknown or timeout
Case Study
• Check: deletePropagates[Profile, Photo]
• Solver: Alloy
Case Study
• Check: deletePropagates[Profile, Photo]
• Solver: Alloy
+ Counterexample data model instance
+ Always returns a result (for small domains)
− Bounded
− Slower
Summary of Technique Pros and Cons
Alloy
SMT Solvers
Theorem
Provers
Unbounded
Verification
No
Yes
Yes
Produces
Counterexample
Yes
Yes
No
Generates
Unknown
No
Yes
Yes
Slow
Fast
Fast*
Speed
*In particular, Spass was slightly faster than Z3 in our experiments
and timed out less frequently
Outline
•
•
•
•
•
•
•
Motivation
Overview of Approach
Rails Data Models
Property Templates
Verification Techniques
Case Study
Conclusions and Future Work
Conclusions and Future Work
• It is possible to extract formal specifications from MVC-based
data models and analyze them
• We were able to find data model errors in real-world
applications using some of these techniques (ISSTA’11, ASE’12)
• Integration of multiple automated verification tools makes
overall approach more flexible
• Property templates simplify property specification
• We have some recent work on
• automated property inference (ISSTA’13)
• analyzing actions that update the data store (submitted)
• Main goal: Verifiable data model specification!
Questions?
Related Work
• Verification of Web Applications
• [Krishnamurti et al, Springer 2006 ] focuses on correct handling of
the control flow given the unique characteristics of web apps
• Works such as [Hallé et al, ASE 2010] and [Han et al, MoDELS
2007] use state machines to formally model navigation behavior
• In contrast to these works, we focus on analyzing the data model
• Formal Modeling of Web Applications
• WebAlloy [Chang, 2009]: user specifies the data model and
access control policies; implementation automatically synthesized
• WebML [Ceri et al, Computer Networks 2000]: a modeling
language developed specifically for modeling web applications;
no verification
• In contrast, we perform model extraction (reverse engineering)
Related Work
• Verification of Ruby on Rails applications
• Rubicon [Near et al, FSE 2012] verifies the Controller whereas we
verify the Data Model
• Requires manual specification of application behavior, whereas
we verify manually written properties
• Limited to bounded verification
• Data Model Verification using Alloy
• [Cunha and Pacheco, SEFM 2009] maps relational database
schemas to Alloy; not automated
• [Wang et al, ASWEC 2006] translates ORA-SS specifications to
Alloy, and uses the Analyzer to produces instances of the data
model to show consistency
• [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser
and business logic interactions
Related Work
• Unbounded Verification of Alloy Specifications using SMT
Solvers
• [Ghazi et al, FM 2011], approach not implemented
• More challenging domain since Alloy language contains
constructs such as transitive closures
• Specification and Analysis of Conceptual Data Models
• [Smaragdakis et al ASE 2009, McGill et al ISSTA 2011] use
Object Role Modeling to express data model and constraints
• Focus is on checking consistency and producing test cases
efficiently
• Using Patterns to Facilitate Formal Property Specification
• First proposed for temporal logic properties [Dwyer et al ICSE 1999]
• The templates we present are not temporal and are specific to
data model analysis