Improving Software Security with Precise
Static and Runtime Analysis
Benjamin Livshits
SUIF Compiler Group
Computer Systems Lab
Stanford University
1
http://suif.stanford.edu/~livshits/work/griffin/
Security Vulnerabilities: Last 10 Years
6,000
5,000
4,000
3,000
2,000
1,000
0
1995
2
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
source: NIST/DHS Vulnerability DB
“Today over 70% of attacks
against a company's Web site
or Web application come at the
'Application
Layer'
not
the
Network or System layer.”
– Gartner Group
3
Which Security Vulnerabilities are Most Prevalent?
•
Analyzed 500 vulnerability reports, one week in November 2005
294 vuln.
or 58%
Input Validation
Denial of Service
Other
File Include
Authentication Bypass
Tem p. File M anipulation
M em ory Corruption
Unauthorized Access
Privilege Escalation
Heap Overflow
0
4
50
100
150
200
250
300
350
source: securityfocus.com
Focusing on Input Validation Issues
SQL Injection
Cross-site Scripting
Buffer Overrun
Information Disclosure
Code Execution
Other
Path Traversal
Web application
vulnerabilities
Format String
Integer Overflow
HTTP Response Splitting
0
5
20
40
60
80
100
source: securityfocus.com
SQL Injection Example
•
Web form allows user to look up
account details
• Underneath: Java J2EE Web
application serving requests
String username = req.getParameter(“user”);
String password = req.getParameter(“pwd”);
String query = “SELECT * FROM Users WHERE
username =“ + user +
“ AND Password =“ + password;
con.executeQuery(query);
6
Injecting Malicious Data (1)
submit
...
...
String query = “SELECT * FROM Users WHERE
username = 'bob'
AND password = ‘********‘”;
...
7
Injecting Malicious Data (2)
submit
...
...
String query = “SELECT * FROM Users WHERE
username = 'bob‘-‘AND password = ‘ ‘”;
...
8
Injecting Malicious Data (3)
submit
...
...
String query = “SELECT * FROM Users WHERE
username = 'bob‘; DROP Users-‘AND password = ‘‘”;
...
9
Web Application Attack Techniques
Command injection
SQL injections
HTTP request smuggling
Cross-site tracing
Path traversal
Parameter manipulation
Cross-site scripting
Hidden field manipulation
HTTP request splitting
Cookie poisoning
Header manipulation
Second-level injection
10
Web Application Attack Techniques
1. Sources (inject)
2. Sinks (exploit)
•
Parameter manipulation
• SQL injections
•
Hidden field manipulation
• Cross-site scripting
•
Header manipulation
• HTTP request splitting
•
Cookie poisoning
• Path traversal
•
Second-level injection
• Command injection
1. Parameter manipulation + 2. SQL injection = vulnerability
11
Goals of the Griffin Software Security Project
Cross Site Scripting
Cross Site Tracing
HTTP Response
SQL injection
Splitting
HTTP Request
Path traversal
Smuggling
DOM-Based XSS
Cookie Poisoning
Domain Contamination
2000
•
12
2001
2002
2003
2004
2005
2006
•
Yahoo! mail
May 2006
•
sourceforge.net
Apr. 2006
•
Myspace.com
Apr. 2006
•
ebay.com
Apr. 2006
•
google.com
Apr. 2006
•
hotmail.com
Feb. 2006
•
ri.gov
Jan. 2006
•
…
Financial impact (Gartner group report)
– Cost per incident:
$300,000+
– Total cost of online fraud:
$400B/year
Existing Application Security Solutions
Client-side input validation
Application firewalls
13
Penetration testing
Code reviews
Code
Client-side
Penetration
Application
input validation
testing
firewalls
reviews
(manual
and
automatic)
Overview
Static
Extensions
Overview of the Griffin Project
Dynamic
Experiments
Conclusions
Future
14
Griffin Project: Framework Architecture
Vulnerability
specification
Static analysis
[Livshits and Lam,
Usenix Security ’05]
Vulnerability
warnings
15
Pros:
ü Find vulnerabilities early
ü Explores all program executions
ü Sound, finds all vuln. of particular kind
Cons:
û May still have imprecision
Application
bytecode
Provided
by user
Dynamic analysis
[Martin, Livshits, and Lam,
OOPSLA ’05]
Instrumented
application
Pros:
ü Keeps vulnerabilities from doing harm
ü Can recover from exploits
ü No false positives
Cons:
û Incurs an overhead
Static Analysis for Bug Finding: Why Soundness?
•
•
Recent bug finding techniques
–
Instrinsa Prefix [SoftPractExp 2000]
– FindBugs [Hovemeyer, OOPSLA ‘04]
–
Metal [Engler, et.al. PLDI ‘02]
– Saturn [Xie, et.al. FSE ‘05]
These techniques lack soundness
→
best-effort attempts
NULL
Buffer
dereferences
overruns
Format
Performance
string violations
errors
Find
Need
& fix
to common
find & fixbugs
all
SQL
Memory
injections
leaks
Cross-site
Datascripting
races
Locking
errors
TOCTOU
16
Following Unsafe Information Flow / Taint Flow
“…” + “…”
Servlet.getParameter(“user”)
(source)
sanitizer
How do we know
what these are?
sink
Statement.executeQuery(...)
(sink)
17
Vulnerability Specification
• User needs to specify
–
–
–
–
Source methods
Sink methods
Derivation methods
Sanitization methods
• PQL: Program Query Language
[Martin, Livshits, and Lam
OOPSLA’05]
query simpleSQLInjection
returns
object String param, derived;
uses
object HttpServletRequest req;
object Connection
con;
object StringBuffer
temp;
matches {
param = req.getParameter(_);
– General language for describing
events on objects
• Real queries are longer
– 100+ lines of PQL
– Captures all vulnerabilities
– Suitable for all J2EE applications
18
temp.append(param);
derived = temp.toString();
}
con.executeQuery(derived);
Overview
Static
Extensions
Static Analysis
Dynamic
Experiments
Conclusions
Future
19
Motivation: Why Pointer Analysis?
String username = req.getParameter(“user");
list1.addFirst(username);
...
String str = (String) list2.getFirst();
con.executeQuery(str);
• What objects do username and str point to?
• Question answered by pointer analysis
– A classic compiler problem for 20 years+
– Rely on context-sensitive inclusion-based pointer
analysis [Whaley and Lam PLDI’04]
20
Pointer Analysis Precision
Runtime
heap
Static
representation
Static approximation
o1
•
•
21
o2
o3
Imprecision of pointer analysis → false positives
Precision-enhancing pointer analysis features
h
Towards Greater Pointer Analysis Precision
• Precision of pointer analysis greatly affects end results
• Variety of pointer analysis techniques suggested
map sensitive
object sensitive
context sensitive
[PLDI’04]
base line
22
Importance of Context Sensitivity
imprecision → excessive tainting → false positives
tainted
c1
c1
String id(String str) {
return str;
}
untainted
tainted
c2
c2
points-to(vc
: VarContext,
Var, h : Heap)
points-to(v
: Var, hv :: Heap)
Context
Context insensitive
sensitivity
23
Handling Containers: Object Sensitivity
1. String s1 = new String(); // h1
2. String s2 = new String(); // h2
3.
4. Map map1 = new HashMap();
5. Map map2 = new HashMap();
6.
7. map1.put(key, s1);
8. map2.put(key, s2);
9.
10. String s = (String) map2.get(key);
points-to(vc
points-to(vc
points-to(vo
: VarContext,
: VarContext,
Heap,
vovo
Var,
: Var,
vo
h :2vh
Heap)
: :Heap,
: Var,
Heap)v : Var,
points-to(vo
Heap,
1:vHeap,
1::Heap,
2: :v
ho
ho ::Heap,
Heap, hh ::Heap)
Heap)
1-level
24
object
1-level
Context
Object
sensitivity
object
sensitivity
sensitivity
sensitivity
+ context sensitivity
Inlining: Poor Man’s Object Sensitivity
•
Call graph inlining is a practical alternative
–
–
•
•
25
Inline selected allocations sites
•
Containers: HashMap, Vector, LinkedList,…
•
String factories: String.toLowerCase(), StringBuffer.toString(), ...
Generally, gives precise object sensitive results
Need to know what to inline: determining that is hard
–
Inlining too little → false positives
–
Inlining too much → doesn’t scale
–
Iterative process
Can’t always do inlining
–
Recursion
–
Virtual methods with >1 target
Map Sensitivity
1. ...
2. String username = request.getParameter(“user”)
3. map.put(“USER_NAME”, username);
...
“USER_NAME” ≠ “SEARCH_QUERY”
4. String query = (String) map.get(“SEARCH_QUERY”);
5. stmt.executeQuery(query);
6. ...
• Maps with constant string keys are common in Java
• Map sensitivity: augment pointer analysis:
– Model HashMap.put/get operations specially
26
Pointer Analysis Hierarchy
Context
sensitivity
None
Object
sensitivity
None
Map
sensitivity
None
1-OS
k-CFA
k-OS
Flow
sensitivity
None
Local flow
Constant
keys
Predicatesensitive
Symbolic analysis
of keys
Interprocedural
predicate-sensitive
Constant
string keys
Local flow
∞-CFA
CFL reachability
∞-CFA
27
∞-OS
Partial 1-OS
PQL into Datalog Translation
PQL Query
Datalog
Query
actual(i
temp.append(param);
3, v4, 0), ret(i3, v5),
call(c
derived
= temp.toString();
3, i3, "StringBuffer.toString"),
pointsto(c3, v4, htemp),
pointsto(c
con.executeQuery(derived);
3, v5, hderived),
}
actual(i4, v6, 0), actual(i4, v7, 1),
call(c4, i4, "Connection.execute"),
pointsto(c4, v6, hcon),
pointsto(c4, v7, hderived).
28
[Whaley, Avots,
Carbin, Lam,
APLAS ’05]
Relevant
instrumentation
points
Datalog solver
simpleSQLInjection(h
query simpleSQLInjection
param, hderived ) :–
ret(i
returns
1, v1),
call(c
object
"ServletRequest.getParameter"),
param, derived;
1, i2, String
pointsto(c
uses
1, v1, hparam),
object ServletRequest req;
actual(i
object
, 0), actual(i2,con;
v3, 1),
2, v2Connection
call(c
object
"StringBuffer.append"),
temp;
2, i2, StringBuffer
pointsto(c
matches {2, v2, htemp),
pointsto(c
param2,= vreq.getParameter(_);
3, hparam),
Vulnerability
warnings
Eclipse Interface to Analysis Results
• Vulnerability traces are exported into Eclipse for review
– source → o1 → o2 → … → on → sink
29
Importance of a Sound Solution
• Soundness:
– only way to provide guarantees on application’s security posture
– allows us to remove instrumentation points for runtime analysis
• Soundness claim
Our analysis finds all vulnerabilities
in statically analyzed code that are
captured by the specification
30
Overview
Static
Extensions
Static Analysis Extensions
Dynamic
Experiments
Conclusions
Future
31
Towards Completeness
• Completeness goal:
specify roots
call graph discovery
– analyze all code that
may be executed at
runtime
32
Generating a Static Analysis Harness
public
class Harness {
<servlet>
<servlet-name>blojsomcommentapi</servlet-name>
public
static void main(String[] args){
<servlet-class>
org.blojsom.extension.comment.CommentAPIServlet
processServlets();
</servlet-class>
processActions();
<init-param>
<param-name>blojsom-configuration</param-name>
processTags();
</init-param>
<init-param>
processFilters();
<param-name>smtp-server</param-name>
}
<param-value>localhost</param-value>
...</init-param>
<load-on-startup>3</load-on-startup>
}</servlet>
App
App
App
web.xml
web.xml
web.xml
Application Server
(JBoss)
33
500,000+ lines of code
1,500,000+ lines of code
2M+ lines of code
Reflection in Java Applications
• Reflection is very common in large apps
• Often used to load plug-ins, avoid incompatibilities, etc.
...
try {
Class macOS = Class.forName("gruntspud.standalone.os.MacOSX");
Class argC[] = {ViewManager.class};
Object arg[] = {context.getViewManager()};
Method init = macOS.getMethod("init", argC);
Object obj
= macOS.newInstance();
init.invoke(obj, arg);
} catch (Throwable t) {
// not on macos
}
...
34
Reflection Resolution
Constants
1.
2.
3.
4.
Specification points
String className = ...;
Class c = Class.forName(className);
Object o = c.newInstance();
T t
= (T) o;
1. String className = ...;
2. Class c = Class.forName(className);
Object o = new T1();
Object o = new T2();
Object o = new T3();
4. T t
= (T) o;
35
Q: what
object does
this create?
[Livshits,
Whaley, and
Lam, APLAS’05]
Reflection Resolution Results
• Applied to 6 large Java apps, 190,000 lines combined
Methods
18,000
Call graph sizes compared
16,000
14,000
12,000
10,000
8,000
6,000
4,000
2,000
0
36
jgap
f reet t s
grunt spud
jedit
columba
jf reechart
Overview
Static
Extensions
Dynamic Analysis
Dynamic
Experiments
Conclusions
Future
37
Runtime Vulnerability Prevention
App
[Martin, Livshits, and
Lam, OOPSLA’05]
App
1. Detect and stop
2. Detect and recover
Vulnerability
specification
App
Application Server
(JBoss)
38
Runtime Instrumentation Engine
• PQL spec → into state machines
– Run alongside program, keep track of partial matches
– Run recovery code before match
{x=o3}
y := x
{x=y=o3}
ε
ε
{x=o3}
ε
{x=o3}
t=x.toString()
{x=o3}
t.append(x)
ε
y := derived(t)
{x=y=o3}
ε
STOP
ε
y := derived(t)
sanitizer
39
Reducing Instrumentation Overhead
query simpleSQLInjection
returns
object String param, derived;
uses
object ServletRequest req;
object Connection
con;
object StringBuffer
temp;
matches {
param = req.getParameter(_);
1.
2.
3.
4.
5.
6.
7.
String name = req.getParameter(“name”);
StringBuffer temp.append(param);
buf1 = new StringBuffer();
derived = temp.toString();
StringBuffer buf2 = new StringBuffer(“def”);
con.executeQuery(derived);
buf2.append(“abc”);
}
buf1.append(name);
con.executeQuery(buf1.toString());
con.executeQuery(buf2.toString());
• Instrument events on objects that may be part of a match
• Soundness allows to remove instrumentation points
40
Overview
Static
Extensions
Experimental Results
Dynamic
Experiments
Conclusions
Future
41
Experimental Evaluation
Benchmarks Total Lines
SecuriBench
Macro
11
3,203,698
SecuriBench
Micro
102
5,588
Griffin
• Comprehensive evaluation:
– SecuriBench Macro [SoftSecTools ’05]
– SecuriBench Micro
– Google: SecuriBench
• Compare Griffin to a commercially available tool
– Griffin vs. Secure Software CodeAssure
– CodeAssure: March 2006 version
42
CodeAssure
17,542
95,845
90
Lines of Code
Benchmark Statistics
17,678
117,207
73
296
27
1.1
72,795
170,893
537
575
63
1.2.6
5,635
226,931
38
754300,00065
2.30
53,309
332,902
245
395200,00041
1.1
2,601
352,737
21
796100,00034
1.0-BETA-1
48,220
445,461
446
859
1.6-beta1
42,920
449,395
333
945
47
454,735
107
920
35
557,592
276
972
133
327,284 3,203,698 2,166
6,823
542
roller
Total
43
0.9.9
14,495
52,089
Classes 600,000
Jars
311500,00029
400,000
0
roller
0.0.22
Files
jorganizer
jorganizer
jgossip
jboard
pebble
webgoat
snipsnap
personalblog
road2hibernate
pebble
blojsom
snipsnap
personalblog
Exp. LOC
road2hibernate
jgossip
LOC
blojsom
Benchmark
Version
LOC
jboard
0.3
Expanded LOC
webgoat
0.9
68
SecuriBench Macro: Static Summary
Benchmark
Sinks
Vulnerabilities
False positives
1
8
2
0
blueblog
11
32
2
0
webgoat
13
47
5
0
personalblog
45
22
40
0
blojsom
34
29
6
0
138
88
16
0
2
13
13
0
pebble
123
39
0
0
roller
41
66
5
147
116
20
7
0
22
29
0
0
Totals
98
147
jboard
snipsnap
road2hibernate
jorganizer
jgossip
44
Sources
Vulnerability Classification
Sinks
Sources
SQL
injections
HTTP splitting
Cross-site
scripting
Path
traversal
Totals
Header
manipulation
0
1
0
0
1
Parameter
manipulation
55
11
2
10
78
Cookie
poisoning
0
1
0
0
1
Non-Web
inputs
15
0
0
3
18
Totals
70
13
2
13
98
• Reported issues back to program maintainers
• Most of them responded, most confirmed as exploitable
• Vulnerability advisories issued
45
SecuriBench Micro: Static Summary
Category
False Positives
Vulnerabilities
basic
41
59
3
collections
14
13
3
interprocedural
0
0
0
arrays
9
8
4
predicates
9
4
4
sanitizers
6
4
2
aliasing
6
11
1
data structures
6
5
1
strong updates
5
1
2
factories
0
0
0
session
3
3
1
102
111
21
Total
46
Tests
A Study of False Positives in blojsom
Base
With context
sensitivity
Lines of Code
50 K
Expanded LOC
333 K
Classes
395
Sources
40
Sinks
29
Vulnerabilities
6
Q: How important are analysis features
114
for avoidingWithfalse
positives?
object
sensitivity
84
43
47
With map
sensitivity
With sanitizers
added
5
0
Griffin vs. CodeAssure
Griffin
CodeAssure
160
140
120
SecuriBench
Macro
100
80
80+
60
40
20
Q: What is the
relationship
between
false
Vulnerabilities
False positives
False negatives
0
positives and false negatives?
120
100
80
60
SecuriBench
Micro
40
40+
20
0
Vulnerabilities
48
False positives
False negatives
Deep vs. Shallow Vulnerability Traces
80
70
Number of Traces
60
Q: How complex are the
50
vulnerabilities we find?
40
30
20
10
0
1
3
4
5
6
Length of Vulnerability Trace
49
7
8
9
10
11
12
13
14
16
Analyzing personalblog
Hibernate library code
Application code
226,931
Analyzed lines of code
45
Total sources
sinks
137
Total sinks
Used sources
5
Used sinks
8
1,806
Reachable objects
Paths through the graph
100
sf.hibernate.Session.find(…)
Q: What is the connectivity between
sources and sinks?
40
Source-sink invocation site pairs
sources
objects
roller
1 falsely tainted object → 100+ false positives
50
Runtime Analysis Results
• Experimental confirmation
– Blocked exploits at runtime in our experiments
• Naïve implementation
– Instrument every string operation → high overhead
• Optimized implementation
– 82-99% of all instrumentation points are removed
140%
Unoptim ized
Overhead (% )
120%
Optim ized
100%
80%
< 1%
60%
40%
20%
0%
webgoat
51
personalblog
road2hibernate
snipsnap
roller
Overview
Static
Extensions
Dynamic
Experiments
Conclusions
Future
52
Related Work &
Conclusions
Summary of Griffin Project Contributions
53
Effective solution
to Web App Sec
problems
•
Effective solution addressing a large range of real
life problem in domain of Web Application Security
Static
analysis
•
Pushed state of the art in sound pointer analysis;
precisely handle large modern applications
Runtime
analysis
•
Design of an efficient dynamic techniques that
recover from security exploits at runtime
Experimental
validation
• Comprehensive, large scale evaluation of problem
complexity and analysis features; discovered many
previously unknown vulnerabilities
Related Work
•
Web application security work
– Penetration testing tools (black box testing)
– Application firewalls (listen on the wire and find patterns)
•
Practical automatic static error detection tools
– WebSSARI (static and dynamic analysis of PHP) [Huang,... ’04]
– JDBC checker (analysis of Java strings) [Wasserman, Su ’04]
– SQLRand (SQL keyword randomization) [Boyd and Keromytis ’04]
Web Application Security Papers
20
15
[Usenix ’05]
10
5
0
2004
2005
2006
[OOSPLA ’05]
54
*Source: http://suif.stanford.edu/~livshits/work/griffin/lit.html
Future Work
Applying Model-checking to Web applications
Learning Specification from Runtime Histories
Partitioned BDDs to Scale bddbddb Better
Analyze Sources of Imprecision in Datalog
Analyzing Sanitization Routines
Attack Vectors in Library Code
Type Qualifiers in Java
Using Model Checking to Break Sanitizers
55
The End.
56
Griffin Security Project
http://suif.stanford.edu/~livshits/work/griffin/
Stanford SecuriBench
http://suif.stanford.edu/~livshits/securibench/
Stanford SecuriBench Micro
http://suif.stanford.edu/~livshits/work/securibench-micro/
PQL language
http://pql.sourceforge.net/
1.
Finding Security Vulnerabilities in Java Applications with Static Analysis, Livshits and Lam, 2005.
2.
Finding Application Errors and Security Flaws Using PQL, Martin, Livshits, and Lam, 2005.
3.
Defining a Set of Common Benchmarks for Web Application Security, Livshits, 2005.
4.
Reflection Analysis for Java, Livshits, Whaley and Lam, 2005.
5.
DynaMine: Finding Common Error Patterns by Mining Software Revision Histories, Livshits and Zimmermann, 2005.
6.
Locating Matching Method Calls by Mining Revision History Data, Livshits and Zimmermann, 2005.
7.
Turning Eclipse Against Itself: Finding Bugs in Eclipse Code Using Lightweight Static Analysis, 2005.
8.
Context-Sensitive Program Analysis as Database Queries, Lam, Whaley, Livshits, Martin, Avots, Carbin, Unkel, 2005.
9.
Improving Software Security with a C Pointer Analysis, Avots, Dalton, Livshits, M.S. Lam, 2005.
10.
Findings Security Errors in Java Applications Using Lightweight Static Analysis, Livshits, 2004.
11.
Tracking Pointers with Path and Context Sensitivity for Bug Detection in C Programs, Livshits and Lam, 2003.
12.
Mining Additions of Method Calls in ArgoUML, Zimmerman, Breu, Lindig, and Livshits, 2006.