Data Model Analysis and
Verification
Tevfik Bultan
University of California Santa Barbara
Joint work with
Jaideep Nijjar and Ivan Bocic
1
Acknowledgement: NSF Support
HTTP Status 500 type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: flp.fl_appl_stts not found. Specify owner.objectname or use sp_help to check whether the object e
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:830)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:763)
org.apache.jsp.fastlane_jsp._jspService(fastlane_jsp.java:242)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:675)
org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:642)
org.apache.jsp.index_jsp._jspService(index_jsp.java:44)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
root cause
2
java.sql.SQLException: flp.fl_appl_stts not found. Specify owner.objectname or use sp_help to check whether the object exists (sp
gov.nsf.fastlane.util.ApplicationStatus.<init>(ApplicationStatus.java:95)
org.apache.jsp.fastlane_jsp._jspService(fastlane_jsp.java:79)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:105)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:336)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:302)
Web Application Dependability
3
Web Application Dependability
4
Web Application Dependability
President Obama: “I want to go in and
fix myself, but I don't write code"
5
Web Application Dependability
TRACKS: A todo list application http://getontracks.org/
Context
Recurring Todo
Feed the Dog
EDIT
6
Web Application Dependability
TRACKS
• Todos may be interdependent
(if a Todo must be completed before another)
• Todo cycles.. ?
7
Web Application Architecture
Data
Model
OOP
ORM
Rel
Db
RESTful
Controller
View
• Model View Controller (MVC) pattern:
Ruby on Rails, Zend for PHP, CakePHP, Struts for Java,
Django for Python, …
• Object Relational Mapping (ORM)
ActiveRecord, Hibernate, …
8
An Example Rails Data Model
Static Data Model
class User < ActiveRecord::Base
has_many :todos
Data Model Updates: Actions
has_many :projects
end
class Project < ActiveRecord::Base
class ProjectsController < ApplicationController
def destroy
belongs_to :user
@project = Project.find(params[:project_id])
has_many :todos
@project.notes.each do |note|
has_many :notes
note.delete
end
end
class Todo < ActiveRecord::Base
@project.delete
belongs_to :user
belongs_to :project
end
respond_to(...)
end
end
class Note < ActiveRecord::Base
belongs_to :project
end
9
Static Data Model
• ActiveRecord class declarations
• sets of objects
• ActiveRecord association declarations
• has_one, has_many, belongs_to, has_and_belongs_to_many
• Association declarations can be used to declare the three
basic types of relations between classes
• one-to-one
• one-to-many
• many-to-many
10
Extensions to Static Data Model
• :through Option
• To express relations which are composition of other relations
• :conditions Option
• To relate a subset of objects to another class
• :polymorphic Option
• To express polymorphic relationships
• :dependent Option
• On delete, this option expresses whether to delete the associated
objects or not
11
The :through Option
class User < ActiveRecord::Base
has_one :profile
has_many :photos, :through => :profile
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos
end
class Photo < ActiveRecord::Base
belongs_to :profile
Profile
end
1
1
*
1
User
1
*
Photo
12
The :dependent Option
class User < ActiveRecord::Base
has_one :profile, :dependent => :destroy
end
class Profile < ActiveRecord::Base
belongs_to :user
has_many :photos, :dependent => :destroy
end
User
1
1
Profile
1
*
Photo
• :delete directly delete the associated objects without looking
at its dependencies
• :destroy first checks whether the associated objects
themselves have associations with the :dependent option set
13
Data Model Verification
• Formalize the static data model as
• A set of classes
• A set of relations between those classes
• A set of constraints on the relations that are imposed by the
association declarations
• Given a formal data model we can automatically check if a
given property holds for the data model
• Automated verification determines: Do the constraints of the
data model imply the property?
14
iDaVer
(Integrated Data Model Verifier)
ActiveRecord
Bound
Model
Extraction
n
bound
Property
formal data model
+ property
BOUNDED VERIFICATION
formula
UNBOUNDED VERIFICATION
Alloy
Encoder
SMT-LIB
Encoder
formula
Alloy
Analyzer
SMT
Solver
instance
or unsat
Results
Results
Interpreter
Interpreter
instance
or unsat
or unknown
15
Property
Verified
Property Failed +
Counterexample
Unknown
How Automated is
Automated Verification?
• All except one step: Property specification
• Example: It is possible to have a User who does not have any
Photos.
• In Alloy:
pred prop { all s: PreState | some u: User | all p: Photo |
(p not in (s.photo_user).u) }
• In SMT-LIB:
(assert (exists ((a PolymorphicClass)) (forall ((p Photo))
(and (isUser a) (not (= p (a user_photo p)))) )))
• Can we make it easier?
16
Property Templates
• Property templates for property specification
• Language-neutral
• Do not require familiarity with SMT-LIB and Alloy
• Example property template:
• noOrphans[classA, classB]
• To check that deleting an object from classA does not cause related
objects in classB to be orphaned
• Easily rerun tool and switch the verification technique,
without having to rewrite the property
• We developed seven property templates for the most
common data model properties
17
Case Study with iDaVer
• LovdByLess, a social networking application
• LOC: 3787
• Number Active Record files: 13
• iDaver input:
• Path of the directory containing the Active Record files
• Name of the file containing the properties to check
(Expressed using property templates!)
• Verification technique
18
Case Study
• Check: someUnrelated[ForumTopic, ForumPost]
• Solver: Z3
19
+ Unbounded verification + Sample instance
− May report unknown or timeout
Case Study
• Check: deletePropagates[Profile, Photo]
• Solver: Alloy
20
Case Study
• Check: deletePropagates[Profile, Photo]
• Solver: Alloy
+ Counterexample data model instance
+ Always returns a result (for small domains)
− Bounded
− Slower
21
Can We Do More?
22
Automatic Property Inference
• Automatically infer properties based on data model schema
• Data model schema: A directed, annotated graph that represents
the relations
• Look for patterns in the data model schema and infer a
property if a pattern that corresponds to a property appears
...
• For example, orphan prevention
0
1
...
n
23
Automatic Data Model Repair
...
• noOrphans(X, Y) property failing means deleting an object
from class X creates an orphan chain that starts with
associated object in class Y
• Repair: Set :dependent option to :destroy on association
declaration in class X and on remaining relations in the chain
that starts with class Y
X
Y
...
Set :dependent => :destroy
on all relations in chain
N
24
Summary
Formal Data
Model
Formal Data Model
+ Properties
Verification
Results
Active
Records
Model
Extraction
Property
Inference
Verification
for failing
properties
Data Model
Repair
25
Experimental Results
Application
LovdByLess
Substruct
Tracks
FatFreeCRM
OSR
TOTAL
Property Type
# Inferred
# Timeout
# Failed
deletePropagates
13
0
10
noOrphans
0
0
0
transitive
1
0
1
deletePropagates
27
0
16
noOrphans
2
0
1
transitive
4
0
4
deletePropagates
15
0
6
noOrphans
1
0
1
transitive
12
0
12
deletePropagates
32
1
19
noOrphans
5
0
0
transitive
6
2
6
deletePropagates
19
0
12
noOrphans
1
0
1
transitive
7
0
7
145
3
96
26
# Data Model
& Application
Errors
# Data Model
Errors
# Failures Due
to Rails
Limitations
# False
Positives
deletePropagates
1
9
0
0
noOrphans
0
0
0
0
transitive
0
0
0
1
deletePropagates
1
3
5
7
noOrphans
0
1
0
0
transitive
0
1
0
3
deletePropagates
1
1
3
1
noOrphans
0
0
0
1
transitive
0
7
0
5
deletePropagates
0
18
1
0
noOrphans
0
0
0
0
transitive
0
0
0
6
deletePropagates
0
12
0
0
noOrphans
0
1
0
0
transitive
0
7
0
0
TOTAL
3
60
9
28
Property Type
27
What About Data Model Actions?
Static Data Model
class User < ActiveRecord::Base
has_many :todos
Data Model Updates: Actions
has_many :projects
end
class Project < ActiveRecord::Base
class ProjectsController < ApplicationController
def destroy
belongs_to :user
@project = Project.find(params[:project_id])
has_many :todos
@project.notes.each do |note|
has_many :notes
note.delete
end
end
class Todo < ActiveRecord::Base
@project.delete
belongs_to :user
belongs_to :project
end
respond_to(...)
end
end
class Note < ActiveRecord::Base
belongs_to :project
end
28
Verification of Data Model Actions
29
29
Abstract Data Stores
Rails
Abstract Data Store
class User
has_many :todos
has_many :projects
end
class User {
0+ Todo todos inverseof user
0+ Project projects inverseof
user
}
class Project
belongs_to :user
has_many :todos
has_many :notes
end
class Project {
0..1 User user
0+ Todo todos inverseof project
0+ Note notes inverseof project
}
class Todo
belongs_to :user
belongs_to :project
end
class Todo {
0..1 User user
0..1 Project project
}
class Note
belongs_to :project
end
class Note {
0..1 Project project
}
30
Abstract Data Stores
def project_destroy
@project = Project.find(
params[:project_id])
@project.notes.each do |note|
note.delete
end
@project.delete
respond_to(...)
end
action project_destroy() {
at_project =
oneof(allof(Project))
foreach note: at_project.notes {
delete note
}
delete at_project
invariant(forall{ |project|
!project.user.empty?
})
invariant(forall{ |user|
forall(Project project:
not empty(project.user)
)
forall(User user:
user in
user.projects.todos.users
)
user.projects.todos.include?(user
)
})
}
Our library allows developers to specify invariants in native Ruby
31
Extraction
Extraction is hard for actions
• Dynamic type system
• Metaprogramming
• Eval
• Ghost Methods such as: User.find_by_name(‘Rob’)
Observations
• The schema is static
• Action declarations are static
• ORM classes and methods do not change their semantic
during execution
• even if the implementation code is generated dynamically
32
Extraction via Instrumented Execution
• Boot-up the Rails runtime in a simulated environment
• Without opening sockets or connecting to the database
• Prepare action methods for extraction
• ORM operations will record their invocation instead of
communicating with the database
• Method calls propagate instrumentation just before
execution
• Extraction is path insensitive, executing both branches
subsequently
• Trigger an HTTP request that triggers an action
33
Verification via Translation to FOL
• A predicate is generated for each class and association
User(o) means that o is an instance of User
Project_user(t) means that t represents an association
between a Project object and User object
• Type system constraints become axioms
∀u: User(u) → ¬(Project(u) ∨ Todo(u) ...)
• Cardinality of associations is expressed through axioms
eg. 0..1:
∀t1, t2:
(Project_user(t1) ∧ Project_user(t2) ∧
Project_user_lhs(t1) = Project_user_lhs(t2))
→ Project_user_rhs(t1) = Project_user_rhs(t2)
34
Translation of Statements to FOL
• An action is a sequential composition of statements.
• Statements
• A state is represented with a predicate denoting all entities
that exist in a state
• A statement is a migration between states
e.g., a create Note statement:
¬pre_state(newly_created())
¬∃t: post_state(t) ∧ Note_project_lhs(t) = newly_created() ∀o:
(post_state(o) ↔ (pre_state(o) ∨ o = newly_created()))
35
Inductive Verification
• Inv(s) is a formula denoting that all invariants hold in state s
• Action(s, s’) is a formula denoting that the action may
transition from state s to state s’
Check if:
∀s, s’: Inv(s) ∧ Action(s, s’) → Inv(s’)
36
Translation of Loops to FOL
• We only support ForEach loops (for now)
• They correspond to universal quantification
• Statements can execute multiple times in loops
• Contexts to differentiate iterations
• Ordering of iterations
• Iteration interdependence
37
Example - Loop Execution
pre-state
class PostsController
def destroy_tags
...
posts = Post.where(id: params[:post_ids])
...
posts.each do |p|
p.tags.destroy_all!
end
...
end
end
post-state
38
Loop Verification Is Difficult
●
●
Iteration interdependency
Unknown bound on the number of iterations
Standard techniques for loop verification
●
Bounded unrolling
●
Loop invariants
39
Coexecution
●
●
●
Capture the effects (the delta) of each iteration’s execution
directly from the pre-state
Combine these deltas using the delta union operation
Execute the union from the pre-state to reach the post-state
40
Example - Coexecution
prestate
post-state
41
Coexecutability Condition
When are sequential execution and coexecution equivalent?
class PostsController
def destroy_tags
...
posts = Post.where(id: params[:post_ids])
...
posts.each do |p|
p.tags.destroy_all!
end
...
end
end
?
=
42
Coexecutability Condition
●
Infer the create, delete and read sets of iterations
○
○
●
Create and delete sets are easily inferred
Read sets, not as easy
Check if create, delete and read operations may conflict
across iterations on the same object
○
○
○
○
read/delete is a conflict
read/read is not a conflict
delete/delete is not a conflict
...
43
Experiments
Experimented on 3 open source Rails applications
• FatFreeCRM, Tracks, Kandan
• 272 actions, 23 invariants
Identified 4 bugs
• Reported to original developers
• All immediately confirmed and, since, fixed
• Missed by previous verification efforts on these applications
44
Experiments
45
45
Experiments on Coexecutability
● All these loops are coexecutable
46
Experiments on Coexecutability
Performance:
● At least an order of magnitude 45% of
the time
● Four orders of magnitude in 13% of
cases
47
Publications
•Jaideep Nijjar and Tevfik Bultan. Bounded Verification of Ruby on
Rails Data Models. In Proc. International Symposium on Software
Testing and Analysis (ISSTA), pages 67-77, 2011.
•Jaideep Nijjar and Tevfik Bultan. Unbounded Data Model Verification
Using SMT Solvers. In Proc. 27th IEEE/ACM Int. Conf. Automated
Software Engineering (ASE), pages 210-219, 2012.
•Jaideep Nijjar, Ivan Bocic and Tevfik Bultan. An Integrated Data Model
Verifier with Property Templates. In Proc. 1st FME Workshop on
Formal Methods in Software Engineering (FormaliSE 2013).
•Jaideep Nijjar and Tevfik Bultan. Data Model Property Inference and
Repair. In Proc. International Symposium on Software Testing and
Analysis (ISSTA), pages 202-212, 2013.
•Ivan Bocic, and Tevfik Bultan. Inductive Verification of Data Model
Invariants for Web Applications. In Proc. International Conference on
Software Engineering (ICSE), pages 620-631, 2014.
48