A Binary Rewriting Defense
Against Stack-based Buffer
Overflow Attacks
Manish Prasad, Tzi-cker Chiueh
SUNY Stony Brook
Experimental Computer Systems Lab
Roadmap





Binary Translation
Buffer Overflow and Return Address
Defense (RAD)
Static BT and RAD
Experimental Results
Conclusion
Experimental Computer Systems Lab
Binary Translation

Motivation


Traditional Applications



To do things without access to source
Legacy Code Migration (HP Aries, UQBT)
Program Optimization (Etch, IBM BOA)
Approaches (and their limitations)


Static – Lacks transparency, accuracy
Dynamic – Less efficient
Experimental Computer Systems Lab
Binary Translation and
Software Security

Legacy applications from outside vendor
Experimental Computer Systems Lab
Binary Translation and
Software Security
NO SOURCE CODE
CERT Advisory
Application A
Application
Vendor X A
Vendor X
VULNERABILITY !!!
Experimental Computer Systems Lab
Binary Translation and
Software Security
CERT Advisory
Application A
Vendor X
VULNERABILITY !!!
Experimental Computer Systems Lab
Has it been done yet ??

Dynamic Translation



DynamoRIO [MIT]
LibVerify [Bell-Labs]
Hardware Support [UIUC]
Experimental Computer Systems Lab
Why Static Binary Translation
(SBT) ?

Inferring legacy program behavior


Application-specific security policies
Use Static Analysis if you can

Butler Lampson, Hints for Computer
System Design
Experimental Computer Systems Lab
Goals


How far can we go with pure static BT ?
Buffer Overflow protection



Widespread
Simple yet covers most SBT issues
Foundation for general Win32/PE
instrumentation framework
Experimental Computer Systems Lab
Contributions

Comprehensive Treatment of Static BT




Why and where it fails ?
Prototype implementation incorporating
static BT state-of-the-art
High disassembly precision
Exhaustive experiments with several
commercial grade Windows applications
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Stack
Growth
4G
Stack Evolution on a Function Call
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Stack
Growth
Function Arguments
caller
4G
Stack Evolution on a Function Call
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Stack
Growth
Return Address
CALL
Function Arguments
4G
Stack Evolution on a Function Call
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Other Local Variables
Local Buffer
callee
Local Variables
Stack
Growth
Old Frame Pointer
Return Address
Function Arguments
4G
Stack Evolution on a Function Call
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Other Local Variables
Local Buffer
target
Local Variables
Stack
Growth
Old Frame Pointer
Return Address
Function Arguments
4G
Stack Evolution on a Function Call
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Stack
Growth
Other Local Variables
Attack
Buffer
Local Buffer
A
Local Variables
A
Old Frame Pointer
A
Return Address
A
Function Arguments
Exp
A = addr of
exploit
Exp = Exploit
Code
4G
Unbounded Buffer Copy
Experimental Computer Systems Lab
Buffer Overflow and RAD
Buffer Overflow Attack
0
Other Local Variables
Address of Exploit
Address of Exploit
Stack
Growth
Address of Exploit
return
Address of Exploit
Exploit Code
4G
Overflow Aftermath
Experimental Computer Systems Lab
Buffer Overflow and RAD
Return Address Defense (RAD)
Stack
Return Address Repository
(RAR)
Prologue
Return Address Save
CALL
caller
Return Addr
Arguments
Ret Addr Copy
Experimental Computer Systems Lab
Buffer Overflow and RAD
Return Address Defense (RAD)
Stack
callee
Return Address Repository
Local
Variable
Space
Return Addr
Arguments
Ret Addr Copy
Experimental Computer Systems Lab
Buffer Overflow and RAD
Return Address Defense (RAD)
Stack
Unsafe
Buffer
Copy
Return Address Repository
Address
Of
Exploit
Code
Ret addr
corrupted
Ret Addr Copy
Experimental Computer Systems Lab
Buffer Overflow and RAD
Return Address Defense (RAD)
Stack
Return Address Repository
Epilogue
Return Address Check
Ret addr
corrupted
Ret Addr Copy
Experimental Computer Systems Lab
Buffer Overflow and RAD
Return Address Defense (RAD)
Stack
Ret addr
corrupted
Return Address Repository
Attack !!
Ret Addr Copy
Experimental Computer Systems Lab
Static BT and RAD
Static BT & RAD – Central
Issues


Disassembly
Code Instrumentation
Binary File
Disassembler
Instrumentation
Experimental Computer Systems Lab
Static BT and RAD
Disassembly


Core component for static analysis of
binaries
Principal Approaches


Linear Sweep
Recursive Traversal
Experimental Computer Systems Lab
Static BT and RAD
Disassembly


Core component for static analysis of
binaries
First Byte
valid inst
Principal Approaches
valid inst


Linear Sweep
Recursive Traversal
valid inst
valid inst
invalid inst
Experimental Computer Systems Lab
Static BT and RAD
Disassembly


Core component for static analysis of
binaries
Entry Point
Principal Approaches


Linear Sweep
Recursive Traversal
CALL fn
fn
RET
Experimental Computer Systems Lab
Static BT and RAD
Disassembly


Core component for static analysis of
binaries
Entry Point
Principal Approaches


Linear Sweep
Recursive Traversal
CALL fn
RET
fn
RET
Experimental Computer Systems Lab
Static BT and RAD
Disassembly


Core component for static analysis of
binaries
Entry Point
Principal Approaches


Recursive Traversal
Linear Sweep
CALL fn
RET
fn
RET
Experimental Computer Systems Lab
Static BT and RAD
Disassembly – Impediments





Code/Data distinction
Variable x86 instruction size
Indirect Branches
Functions without explicit CALL
PIC
Experimental Computer Systems Lab
Static BT and RAD
0x0F0x85
0x850xC0
0xC00x0F
0x0F0x85
0x85 …..
….
0x0F
JNE offset
Disassembly – Impediments
0x0F // data
TEST eax, eax
JNE offset





Code/Data distinction
Variable x86 instruction size
Indirect Branches
Functions without explicit CALL
PIC
Experimental Computer Systems Lab
Static BT and RAD
0x0F0x85
0x850xC0
0xC00x0F
0x0F0x85
0x85 …..
….
0x0F
JNE offset
Disassembly – Impediments
0x0F // data
TEST eax, eax
JNE offset





Code/Data distinction
Variable x86 instruction size
Indirect Branches
Functions without explicit CALL
PIC
Experimental Computer Systems Lab
Static BT and RAD
Disassembly – Impediments





Code/Data distinction
Variable x86 instruction size
Indirect Branches
Functions without explicit CALL
PIC
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Approach

Recursive Traversal


Linear Sweep


Accurately code/data identification
Reach areas not covered by recursive
traversal
Compiler-independent heuristics

Recover from errors
Experimental Computer Systems Lab
Static BT and RAD
Disassembly Accuracy
Experimental Computer Systems Lab
Static BT and RAD
Code Instrumentation


Add desired functionality
Preserve original program semantics
Experimental Computer Systems Lab
Static BT and RAD
Code Instrumentation – RAD


Add buffer overflow protection
Preserve original program semantics
Experimental Computer Systems Lab
Static BT and RAD
What to instrument ?

Function boundary identification



Prologue and epilogue must be both
instrumented/uninstrumented
False alarms
Pattern matching (for ‘interesting’
functions)

Stack frame allocation and deallocation
Experimental Computer Systems Lab
Prototype
What to instrument ?
Binary File
Disassembler
Core Binary
Rewrite
Engine
Each Instruction
RAD
If ‘interesting’
function
Experimental Computer Systems Lab
Static BT and RAD
JMP (save
Prologue
ret addr)
Inserting Checking Code
JMP (check
Epilogue
ret addr)





JMP at prologue and epilogue to RAD code
Replace 5 byte worth instructions
Should not disturb branch targets
Stack frame allocation (at prologue) >= 5
bytes
Stack frame deallocation possible in 2 – 4
bytes

Return address check in INT 3 handler
Experimental Computer Systems Lab
Static BT and RAD
Inserting CheckingADDCode
ESP, x // 3-6 bytes





PUSH
EBP////11byte
byte
POP EBP
JMP at prologue and epilogueRET
to RAD
code
// 1 byte
MOV EBP, ESP // 2 byte
Replace 5 byte worth instructions
MOV ESP, EBP // 2 bytes
POPESP,
EBPx //
// 13-6byte
bytes
Should not disturb branch SUB
targets
RET // 1 byte
Stack frame allocation (at prologue) >= 5
INT 3 //
LEAVE
// 11 byte
byte
bytes
RET // 1 byte
Stack frame deallocation possible in 2 – 4
bytes

Return address check in INT 3 handler
Experimental Computer Systems Lab
Static BT and RAD
INT 3 Statistics
Experimental Computer Systems Lab
Experimental Evaluation
Experimental Evaluation

Goals of experiments



Effect on program correctness
Resilience to buffer overflow attacks
Performance and space overhead
Experimental Computer Systems Lab
Experimental Evaluation
Space Overhead
Micro-Benchmark
Macro-Benchmark
Experimental Computer Systems Lab
Experimental Evaluation
Execution Time Overhead
Overhead = Execution Time with RAD – Execution Time without RAD
Execution Time without RAD
Experimental Computer Systems Lab
Experimental Evaluation
Resilience to Buffer Overflow
Attack




Windows Help (Winhlp32.exe)
Windows NT 4.0 with Service Pack 4.0
Content file (.CNT) with long heading
string
Published exploit code
Resists Attack !!
Experimental Computer Systems Lab
Known Limitations





Disassembly Limitations
Hand-crafted Assembly
RAD Limitations
Multi-Threaded Applications
Self-Modifying Code
Experimental Computer Systems Lab
Known Limitations

Disassembly Limitations





Indirect branches
Hand-crafted Assembly
RAD Limitations
Multi-Threaded Applications
Self-Modifying Code
Experimental Computer Systems Lab
Indirect Branches and Control
Flow Analysis
Experimental Computer Systems Lab
Known Limitations


Disassembly Limitations
Hand-crafted Assembly




Inter-procedural jumps, multiple function
entry points
RAD Limitations
Multi-Threaded Applications
Self-Modifying Code
Experimental Computer Systems Lab
Known Limitations



Disassembly Limitations
Hand-crafted Assembly
RAD Limitations



Memory Pointer corruption
Multi-Threaded Applications
Self-Modifying Code
Experimental Computer Systems Lab
Known Limitations




Disassembly Limitations
Hand-crafted Assembly
RAD Limitations
Multi-Threaded Applications


Per-thread RAR needed
Self-Modifying Code
Experimental Computer Systems Lab
Known Limitations





Disassembly Limitations
Hand-crafted Assembly
RAD Limitations
Multi-Threaded Applications
Self-Modifying Code

Beyond static analysis
Experimental Computer Systems Lab
Conclusions





First study to implement Static BT stateof-the-art into a working system
High disassembly accuracy
Demonstrated effectiveness on
commercial grade Windows applications
Qualified success as a security tool
Basis for a general Win32/PE
Instrumentation framework
Experimental Computer Systems Lab
Looking Ahead




Support for DLLs
Dynamic translation to ameliorate static
BT deficiencies
Automatic application-specific security
policy generation
Copy and Tamper resistant software
Experimental Computer Systems Lab
Project Page


http://www.ecsl.cs.sunysb.edu/brew.htm
Disassembler Sources

http://www.ecsl.cs.sunysb.edu/disassembler.tgz
Experimental Computer Systems Lab
Questions ??
Experimental Computer Systems Lab
… and I’m looking for work !!!
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
0x400100
DATA
:
:
CALL 0x40011C
JMP 0x400110
:
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
0x40012E
RET
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Experimental Computer Systems Lab
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
0x400100
DATA
Entry Point
:
:
CALL 0x40011C
JMP 0x400110
:
Inter-Procedural
Call Graph
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
0x40012E
RET
Pass 1
Recursive
Traversal
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
0x400100
DATA
Entry Point
:
:
Function Control
Flow Graph
CALL 0x40011C
JMP 0x400110
:
Inter-Procedural
Call Graph
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
Backward
Edge
0x40012E
RET
Pass 1
Recursive
Traversal
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Function
Body
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
0x400100
DATA
Entry Point
:
:
CALL 0x40011C
JMP 0x400110
:
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
0x40012E
RET
Pass 1
Recursive
Traversal
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Function
Body
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
Pass 2
Linear Sweep
0x400100
DATA
:
:
CALL 0x40011C
JMP 0x400110
:
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
0x40012E
RET
Legal inst.
Bytes as code
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Function
Body
Experimental Computer Systems Lab
Static BT and RAD
Disassembler – Passes
Pass 3
Error Recovery
Heuristics
0x400100
DATA
:
:
CALL 0x40011C
JMP 0x400110
:
0x400110
RET
MOV EAX, 0x40012E; PUSH 0x400100
CALL EAX
RET
0x40011C
0x40012E
RET
Code sequence
not ending with
JMP/RET
=> not code
MOV EAX, ECX
CALL 0X400140
DATA
DATA
Function
Body
Experimental Computer Systems Lab
Experimental Evaluation
Execution Time
Micro-benchmark
Penalty = Additional RAD Overhead
Original Run Time
Experimental Computer Systems Lab
WattProbe


Linux-based experimental platform for lowpower computing
Power vs. Performance measurements




Fine Grained
High Resolution
Across Multiple Resources
Power characterization of interactive
applications

Application-specific low power modes
Experimental Computer Systems Lab
Caching Infrastructure for Stackable
File Systems (FAST ’03 WIP)

Cache coherence support


Minimal changes to the page cache
code


Transparent to native and stackable file
systems
Hooks into a cache manager
Prototype Linux implementation
Experimental Computer Systems Lab
Communication and Membership
Management for Linux Clusters

Low-latency link-level semi-reliable
transport layer (Springer LNCS, HiPC
’02)


No hardware support
Membership Management


Cluster-wide consistent membership view
Scalable failure detection
Experimental Computer Systems Lab
Thanks
Experimental Computer Systems Lab
Experimental Computer Systems Lab
Experimental Computer Systems Lab
Prototype
Disassembly Limitations


False Negatives (Functions Missed)
False Positives (Falsely Identified
Functions)
Experimental Computer Systems Lab
Prototype
Disassembly Limitations

False Negatives (Functions Missed)



Partly/fully misidentified as data
Identified fully as code
False Positives (Falsely Identified
Functions
Instrumented
Prologue
Data
Function
Body
RAR
Security
Overflow
Hole
Identified
as Data
Uninstrumented
Epilogue
Experimental Computer Systems Lab
Prototype
Disassembly Limitations

False Negatives (Functions Missed)



Partly/fully misidentified as data
Identified fully as code
False Positives (Falsely Identified
Functions RAR overflow Fn1 Instrumented
Prologue
Identified as Data
Security Hole
Fn2
Uninstrumented
Epilogue
Experimental Computer Systems Lab
Prototype
Disassembly Limitations

False Negatives (Functions Missed)



Partly/fully misidentified as data (pop up)
Identified fully as code
False Positives (Falsely Identified
Functions)
Identified
Data
as Code
Security Hole
Identified as Code
Interesting Prolog
Function
Body
Experimental Computer Systems Lab
Prototype
Disassembly Limitations


False Negatives (Functions Missed)
False Positives (Falsely Identified
Functions)



Code following data/unconditional
branch
Indirect Jump
Missed attacks
Data
RAR overflow
Function
Jump Target
Entry Pt
Experimental Computer Systems Lab
Prototype
Indirect Branches and Control
Flow Analysis
Experimental Computer Systems Lab
Prototype
RAD Limitations

Attacks resisted



Return Address
Frame Pointer
Does not protect

Memory pointer attacks


Function pointer corruption
Import table overwrites
Experimental Computer Systems Lab
Prototype
Multi-Threaded Applications


RAR shared between threads
Possible Solution




Thread Information Block (TIB) access
using FS register
Thread Local Storage (TLS) slots in TIB
Separate RAR for each thread
Thread-specific RAR addresses in TLS slots
Experimental Computer Systems Lab
Prototype
Hand-Crafted Assembly Quirks
Fn1 entered
Return address not saved
No interesting prolog
JNE label
Fn 1
No interesting epilog
False Alarm
Interesting prolog
label
Exit from Fn2
Return address check made
Fn 2
Interesting epilog
Inter-procedural jump in MSAccess
Experimental Computer Systems Lab
Prototype
Software Architecture & Flow
of Control
Binary File
Disassembler
Initialization
Core Binary
Rewrite
RAD
Engine
Instrumentation
Set up
RAD mine zones
PE-specific
component
New PE
Section
Experimental Computer Systems Lab