Software Security Monitors:
Theory & Practice
David Walker
Princeton University
(joint work with Lujo Bauer and Jay Ligatti)
General-purpose Security Monitors
• A security monitor (program monitor) is a process
that runs in parallel with an untrusted application
– monitors examine application actions
• decide to allow/disallow application actions
• may terminate an application, log application
actions, etc.
– monitors detect, prevent, and recover from
erroneous or malicious behavior at run time
– monitors generalize specific enforcement
mechanisms such as access control lists, etc.
July 2003
Software Security Monitors
David Walker
What is a security monitor?
Monitors analyze & transform untrusted
application actions:
Input Stream
…
a3
a2
Monitor
a2
a1
Output Stream
…
a4
a2
a2
a1
Application generates actions to be input into monitor
Machine executes actions output by monitor
July 2003
Software Security Monitors
David Walker
Possible Monitor Actions
• Accept the action
• Halt the application
• Suppress (skip) the operation
• Insert some computation
• Also: replace results; raise
exceptions
July 2003
Software Security Monitors
David Walker
Formalizing security monitors
• Security monitors => formal automata that
transform a stream of program actions
• Given: a set of possible program actions A
• Monitors are deterministic state machines:
(Q, q0, T) where
Q = state set
q0 = start state
T = transition function
July 2003
Software Security Monitors
David Walker
Operational Semantics
 Single step (determined by T):
So
(Sin, q)  (Sin’, q’)
 Multi-step (reflexive, transitive closure of T):
So
 (Sin, q)  (Sin’, q)
 Output sequence is observable
Input sequences are not observable
July 2003
Software Security Monitors
David Walker
A Hierarchy of Security Monitors
We classify monitors based on their transformational
abilities (ie: based on T).
Insert
Suppress
Truncation
Suppression

Insertion

Edit

July 2003

Software Security Monitors
OK
Halt








David Walker
An Example: E-Banana.com
• Set of application actions:
A={
}
take(n),
pay(n),
browse,
receipt
//
//
//
//
take n bananas
pay for n bananas
browse for bananas
commit
• Edit Automaton:
pay(n)
browse
browse
start
init
take(n)
pn
tn
take(n)
pay(n)
tpn
receipt
pay(n);take(n);receipt
July 2003
Software Security Monitors
David Walker
Edit Automata
 Definition: (Q,q0,T)
– where T = (t,e,i)
– State transition function t
• t : action x state  state
– Emission function e
• e : action x state  {+,-}
– Insertion function i
• i : action x state  action sequence x state
July 2003
Software Security Monitors
David Walker
Edit Automata
• Operational Semantics
a
– (S, q) 
(S’, q’)
if S=a;S’ and t(a,q)=q’ and e(a,q)= +
– (S, q)  (S’, q’)
if S=a;S’ and t(a,q)=q’ and e(a,q)= Sins
– (S, q)  (S, q’)
if S=a;S’ and i(a,q)=(Sins, q’)
– (S, q)  (empty, q)
otherwise
July 2003
Software Security Monitors
(E-Accept)
(E-Suppress)
(E-Insert)
(E-Halt)
David Walker
Security Policies
• A program execution is a sequence of actions
• A Security Property is a predicate over
executions.
• Example Properties:
– P(S) iff bananas taken equal bananas paid for in S
– Access control, resource bounds policies are policies
• Non-properties:
– Relations between different executions of a program
– Information-flow policies
July 2003
Software Security Monitors
David Walker
What does it mean to enforce a policy?
• Principle of Soundness
All observable outputs obey the policy
 sequences Sin .  state q’ .  sequence So
So
1. (Sin, q0)  (empty, q’)
2. P(So)
• Principle of Transparency
Semantics of executions that already obey policy
must be preserved
3. P(Sin) (Sin So)
July 2003
Software Security Monitors
David Walker
Some Useful Equivalences
 Remove/Insert unnecessary actions
– fclose(f);fclose(f) fclose(f)
• Replace a sequence with equivalent actions
– socket(S);send(S,m) socketSend(S,m)
• Permute independent actions
– fopen(f);fopen(g) fopen(g);fopen(f)
• Necessary properties:
– reflexive, symmetic & transitive
– S S’  P(S) P(S’)
July 2003
Software Security Monitors
David Walker
E-Banana.com
• Equivalence Rules:
1) (browse; S)  S
2) (S1; take(n); pay(n); S2) 
(S1; pay(n); take(n); S2)
July 2003
Software Security Monitors
David Walker
Conservative Enforcement
 Enforcer satisfies Soundness but not
necessarily Transparency

 properties P .
( sequence S . P(S)) 
P can be conservatively enforced
Conservative
July 2003
Software Security Monitors
David Walker
Effective Enforcement
 Enforcer satisfies Soundness and
Transparency
 provides some flexibility for the enforcer to edit
the execution sequence
 guarantees the final results of running the
application with the monitor are semantically
equivalent to running the application without
the monitor
Effective
Conservative
July 2003
Software Security Monitors
David Walker
Precise Enforcement
 Definition
 Enforcer satisfies Soundness and Transparency
 Enforcer must output actions in lock-step with
application
 Motivation
 In some scenarios, operations cannot be delayed
without disrupting application semantics
Precise
Effective
Conservative
July 2003
Software Security Monitors
David Walker
What properties can be enforced?
• The enforceable properties depend upon
– the definition of enforcement (conservative, effective,
precise)
– the class of automaton (truncation, suppression,
insertion, edit)
– the space of possible input programs
• if the monitor can assume certain “bad” executions do not
occur, it can enforce more properties
• static program analysis (type systems; proof-carrying code)
can constrain program execution in ways useful to run-time
monitors
July 2003
Software Security Monitors
David Walker
Effective Enforcement
• An E-Banana.com policy:
browse*; ((take(n);pay(n) | pay(n);take(n)) ; receipt)*
– Our edit automaton is an effective enforcer:
• It satisfies Soundness
• It satisfies Transparency
• Proofs are by induction over the possible inputs
– Less powerful automata (truncation, suppression and
insertion) cannot enforce the E-Banana property
• Proof by contradiction shows either Soundness or
Transparency will be violated
July 2003
Software Security Monitors
David Walker
A Simple Theorem
• Theorem: Any decideable predicate P on
executions is a property that can be effectively
enforced by some edit automaton
– Proof: construct a transactional edit automaton that
suppresses and logs program actions when ¬P(S)
and commits (outputs) when P(S), for every initial
sequence of actions S in a program execution
July 2003
Software Security Monitors
David Walker
Effectively Enforceable Properties
Editing Properties
Insertion
Properties
July 2003
Trunc.
Prop.
Software Security Monitors
Suppression
Properties
David Walker
Summary of theoretical results
•
We have developed the following
rigorous methodology for reasoning
about run-time security:
1. Define the computational framework
using formal operational semantics
2. Define what it means to enforce a policy
3. Prove results about enforceable policies &
mechanisms from definitions 1 & 2
July 2003
Software Security Monitors
David Walker
Future Work/Research Ideas
• Proper definitions of enforcement for infinite
execution sequences
– Understanding edit automata on infinite sequences
• Understand transactional policies & develop
“transaction automata”
– what can they enforce?
• Incorporate more practical elements into the model
– security environment; cryptographic secrets
– replacement of results, exceptions and program state
July 2003
Software Security Monitors
David Walker
Polymer, the Language
• Polymer
– A domain-specific language for programming
security monitors (ie: edit automata)
– Java + a couple of simple extensions:
• atomic policy definitions encapsulating
– a set of security-relevant actions
– security state
– decision procedure that produces security “suggestions”
(halt, suppress action, insert action, etc)
• compositional policy definitions involving
– higher-order policy combinators
July 2003
Software Security Monitors
David Walker
Securing Untrusted Applications
untrusted code
describes securityrelevant program
points
Java
application
policy
interface
instrumented
application
contains hooks
to call monitor
July 2003
separately compiled
from policy
Software Security Monitors
David Walker
Securing Untrusted Applications
Java
application
policy
interface
policy
implementation
instrumented
application
combines
application
and policy
July 2003
implements
dynamic
security policy
secure application
Software Security Monitors
David Walker
Atomic Polymer Policy
class limitFiles extends Policy {
private int openFiles = 0;
private int maxOpen = 0;
limitFiles(int max) {
maxOpen = max;
}
new policy
definition
extends
policy class
private policy state
....
}
July 2003
policy constructor
Software Security Monitors
David Walker
Atomic Polymer Policy Continued
class limitFiles extends Policy {
private int openFiles = ...
private int maxOpen = ...
public ActionPattern[] actions =
new ActionPattern[] {
<File fileOpen(String)>, <void fileClose(File)>
};
....
set of policyrelevant methods
}
July 2003
Software Security Monitors
David Walker
Atomic Polymer Policy Continued
class limitFiles extends Policy {
private int openFiles = ...
private int maxOpen = ...
public ActionPattern[] actions = ...
Suggestion before(Action a) {
aswitch (a) {
case fileOpen(String s) :
if (++openFiles <= maxOpen)
return Suggestion.OK();
else
return Suggestion.Halt();
case fileClose(File f) : ...
July 2003
Software Security Monitors
policy
behavior
David Walker
Atomic Polymer Policy Continued
class limitFiles extends Policy {
private int openFiles = ...
private int maxOpen = ...
public ActionPattern[] actions = ...
Suggestion before(Action a) {
aswitch (a) {
case fileOpen(String s) :
if (++openFiles <= maxOpen)
return Suggestion.OK();
else
return Suggestion.Halt();
case fileClose(File f) : ...
July 2003
Software Security Monitors
David Walker
Atomic Polymer Policy Continued
class limitFiles extends Policy {
public ActionPattern[] actions = ...
private int openFiles = ...
private int maxOpen = ...
Suggestion before(Action a) {
aswitch (a) {
case fileOpen(String s) :
if (++openFiles <= maxOpen)
return Suggestion.OK();
else
return Suggestion.Halt();
case fileClose(File f) : ...
July 2003
Software Security Monitors
David Walker
Complex Monitors
• Combine atomic policies defined over a
variety of different resources
– eg: sample applet policy
• file system access control
• number of files opened
• restricted network access
– no network access after local file is read
– communication with applet source only
July 2003
Software Security Monitors
David Walker
Policy Combinators
• Programmers may write parameterized
policy combinators:
– And, Or, Forall, Exists, Chinese wall,...
AndPolicy:
P1
s1
P2

s2
s
July 2003
Software Security Monitors
David Walker
Policy Combinators
class AndPolicy extends Policy {
private Policy p1;
private Policy p2;
AndPolicy(Policy pol1, Policy pol2) {
p1 = pol1;
p2 = pol2;
...
first-class
}
policies
}
July 2003
Software Security Monitors
David Walker
Policy Combinators
class AndPolicy extends Policy {
...
Suggestion before(Action a) {
}
}
July 2003
Suggestion s1 = p1.before(a);
Suggestion s2 = p2.before(a);
if (s1.isOK() && s2.isOK())
return Suggestion.OK();
else ...
using
suggestions
In reality, writing combinators
is very tricky
Software Security Monitors
David Walker
Summary of Language Design
•
Polymer facilitates the implementation
of program monitors by
1. encapsulating all elements (relevant
actions, state, decision procedure) of
atomic policies in a single place
2. providing mechanisms to compose
policies in a well-defined manner
3. coming equipped with a formal semantics
– we’re working on it
July 2003
Software Security Monitors
David Walker
Conclusions
• Technology for securing extensible systems is in
high demand
– Software security monitors are one part of the solution
• For more information, see
– Edit Automata: Enforcement Mechanisms for Run-time Security
Policies. IJIS 2003.
– Types and effects for non-interfering program monitors. ISSS
2002 & LNCS 2609.
– More Enforceable Security Policies. FCS 2002.
– www.cs.princeton.edu/sip/projects/polymer/
July 2003
Software Security Monitors
David Walker
End
July 2003
Software Security Monitors
David Walker
Realistic Monitors
• Protect complex system interfaces
– interfaces replicate functionality in many
different places
– method parameters communicate
information in different forms
– eg: Java file system interface
• 9 different methods to open files
• 4 different methods to close files
• filename strings, file objects, self used to
identify files
July 2003
Software Security Monitors
David Walker
Abstract Action Definitions
java.lang.io
FileReader(String fileName);
FileReader(File file);
RandomAccessFile(...);
...
FileReader.close();
RandomAccessFile.close();
...
July 2003
Software Security Monitors
fileOpen(String n);
fileClose();
David Walker
Abstract Action Definitions
class fileOpen extends ActionSig {
boolean canMatch(Action a) {
aswitch (a) {
case FileReader(_) : return true;
case RandomAccessFile () : return true;
...
}
}
String parameter1(Action a) {
....
}
July 2003
Software Security Monitors
David Walker
Abstract Action Pattern Matching
class limitFiles extends Policy {
...
fileOpen.parameter1(a)
Suggestion step(Action a) {
aswitch (a) {
case fileOpen(String s) :
...
case fileClose() :
}
July 2003
}
...
fileOpen.canMatch(a)
Software Security Monitors
David Walker
Taxonomy of Precisely Enforceable
Properties
July 2003
Software Security Monitors
David Walker
Secure Application
Untrusted application
Host
System
(Java)
Program Monitor Definition
Polymer language extensions
Java core
July 2003
Software Security Monitors
David Walker
Policy Architecture: Simple Policies
system
interface
Host
System
(Java)
Simple
Policy
Def.
Polymer language extensions
Java core
July 2003
Software Security Monitors
David Walker
Policy Architecture: Abstract Actions
abstract system
interface
Host
System
(Java)
concrete system
interface
July 2003
Abstract
Action
Def.
Simple
Policy
Def.
Polymer language extensions
Java core
Software Security Monitors
David Walker
Policy Architecture:
Complex Policies
Complex,
System-specific
Policy
abstract system
interface
Host
System
(Java)
concrete system
interface
July 2003
Abstract
Action
Def.
Simple
Policy
Def.
Policy
Comb.
Def.
Polymer language extensions
Java core
Software Security Monitors
David Walker
Securing Extensible Systems
• Many questions:
– Our application requires property X. Can
we enforce it precisely or will we have to
get by with an approximation?
– How do we write down our policy
succinctly and unambiguously?
– What specific mechanism will we need to
enforce our policy?
– How do we implement the mechanism?
July 2003
Software Security Monitors
David Walker
Summary
• A general framework for formal reasoning
about security monitors
– defined a hierarchy of security monitors
– gave meaning to the word “enforceable”
– developed rigorous proofs concerning enforceable
properties
• Polymer: A programming language for
composing security monitors
– techniques for modular monitor design &
composition
– formal semantics as an extension of
FeatherWeight Java
July 2003
Software Security Monitors
David Walker