Behavior-Based Malware Detection
Somesh Jha
University of Wisconsin,
Madison
The Malware Problem
Host-based malicious-code detection:
• New program arrives an end-host system.
• Need to identify whether the program is
malicious or not.
Viruses, trojans, backdoors, bots, adware,
spyware, ...
June 2011
Somesh Jha: Behavior-Based Malware Detection
2
Malware: A Threat Assessment
Win32 viruses and other malware
Source: Symantec Research
12,000
10,866
Total viruses and worms
Total families
9,000
Total number
7,360
6,000
4,496
3,000
1,702
994
445
687
Jan.-June
2002
July-Dec.
2002
0
June 2011
Jan.-June
2003
July-Dec.
2003
Jan.-June
2004
Somesh Jha: Behavior-Based Malware Detection
July-Dec.
2004
Jan.-June
2005
3
Malware: A Threat Assessment
New Win32 virus and worm variants 2002-2005
Source: Symantec Research
12,000
10,866
Total viruses and worms
Total families
9,000
Total number
7,360
6,000
4,496
3,000
1,702
445
994
687
N/A
N/A
141
184
164
171
170
Jan.-June
2002
July-Dec.
2002
Jan.-June
2003
July-Dec.
2003
Jan.-June
2004
July-Dec.
2004
Jan.-June
2005
0
Period
June 2011
Somesh Jha: Behavior-Based Malware Detection
4
Symantec Threat Report 2010
• Highlights from the report
• See
– http://www.symantec.com/en/uk/business/
theme.jsp?themeid=threatreport
June 2011
Somesh Jha: Behavior-Based Malware Detection
5
Demographics
• Where do attacks emerge?
• US is still top on the list
– 19% in 2009 (23% in 2008)
• Emergence of other countries in the top
10 list
– Brazil and India
– Emergence of these new countries related to
increased internet connectivity in these
countries
Attack Targets
• Who are the attackers targeting?
• Old news
– Spam, identity theft, …
– Still important factors
• New Trend
– It looks like hackers are now targeting
enterprises and government organizations
– The goal seems to theft of sensitive data or
espionage
– Stuxnet is most sophisticated example of this
attack
Vulnerabilities Exploited
• What vulnerabilities are attackers
exploiting?
• It seems like web-based attacks are the
most popular
– Mozilla Firefox seems to be the most
vulnerable
• The most common Web-based attack in
2009 was related to malicious PDF
activity
– Exploits vulnerabilities in “plug ins” that
read the attached PDF file
Malware Trends
• What types of malware were most
prevalent?
• Trojans rule!
– Out of 10 malware families detected 6 were
Trojans (2 worms, 1 back door, and 1 virus)
• Tool kits for creating malware and
variants have matured
– Popular kits: SpyEye, Fragus, Zues, …
– In 2009 Symantec encountered 90,000
variants of malware variants created by the
Zues toolkit
Take Aways
• Demographics of attack origins is
expanding
• Web is the major vector for attack
• Trojans are the most prevalent form of
malware
• Creating malware variants is easy
because the toolkits have matured
• Enterprises and organizations are going to
be increasingly targeted
Market Trends
• Security market will have a rapid growth
in other countries (e.g., Brazil and India)
– Reason: Demographics of attack origin
• Enterprise market will expand
– Reason: Enterprises are being targeted by
the attackers
• Other technologies for detection and
remediation will become important
Defenses
• Simple measures
– Having policies in an enterprise can go a long
way
– For example, don’t open a PDF attachment if
you don’t recognize the sender
• Signature-based detection is not enough
– In 2009 Symantec created 2,895,000
signatures
– In 2008 they created 1,691,323 signatures
– These detectors need to be complemented
with other types of detection
Defenses
• Complementing technologies
– Behavior-based and reputation-based detection
can complement signature-based detection
– These complementing defenses can keep the
number of signatures in check
– These two technologies are mentioned
throughout the report
• Data breaches
– Keep confidential data secure even if an
enterprise gets compromised
– There are several solutions in the market
– Remediation solutions will also gain traction
Key Definitions
Variants : New strains of viruses that
borrow code, to varying degrees, directly
from other known viruses.
Source: Symantec Security Response Glossary
Family: a set of variants with a common
code base.
Beagle family has 197 variants (as of Nov. 30).
Warezov family has 218 variants (as on Nov. 27).
The Malware Problem
• Malware writers use any and all
techniques to evade detection.
– Obfuscation / packing / encryption
– Remote code updates
– Rootkit-based hiding
• Detectors use technology from 15 years
ago: signature-based detection.
Signature-Based Detection
lea
push
push
call
pop
lea
pop
push
push
lea
push
call
eax, [ebp+Data]
offset aServices_exe
eax
_strcat
ecx
eax, [ebp+Data]
ecx
edi
eax
eax, [ebp+ExistingFileName]
eax
ds:CopyFileA
8D
68
50
E8
59
8D
59
57
50
8D
50
FF
85 D8 FE FF FF
78 8E 40 00
69 06 00 00
85 D8 FE FF FF
85 D4 FD FF FF
15 C0 60 40 00
Signature
• Signatures (aka scan-strings) are the most
common malware detection mechanism.
Signature Detection Does Not Scale
One signature for one malware instance.
June 2011
Somesh Jha: Behavior-Based Malware Detection
17
Current Signature Management
McAfee: release daily updates
– Trying to move to hourly “beta” updates
DAT
File #
Date
Threats
Detected
New Threats
Added
Threats
Updated
4578
Sep. 09
147,382
22
188
4579
Sep. 12
147,828
27
231
4580
Sep. 13
148,000
11
236
4581
Sep. 14
148,368
42
140
4582
Sep. 15
148,721
16
203
4583
Sep. 16
149,050
18
117
Source: McAfee DAT Readme
June 2011
Somesh Jha: Behavior-Based Malware Detection
18
Huge Signature Databases
• Recently, McAfee announced the addition
of the 200,000th signature.
– More signatures than files on a standard
Windows machine (approx. 100k).
• McAfee notes that:
“Good family detection becomes crucial
for a less worrisome experience on the
Internet.”
Source: McAfee Avert Labs
Roadmap to Better Detection
• Make the malware writer’s job as hard as
possible.
• Detect malware families,
not individual malware instances.
• Catch behavior,
not syntactic artifacts.
June 2011
Somesh Jha: Behavior-Based Malware Detection
20
Outline
•
•
•
•
•
Introduction
Threat Model
Evaluation of Current Detectors
Behavior-Based Detection
Future Directions
June 2011
Somesh Jha: Behavior-Based Malware Detection
21
Threat Model
• Malware writers craft their programs so
to avoid detection.
Two common evasion techniques:
– Program Obfuscation
(Preserves malicious behavior)
– Program Evolution
(Enhances malicious behavior)
June 2011
Somesh Jha: Behavior-Based Malware Detection
22
Obfuscations for Evasion
Nop insertion
Register renaming
Junk insertion
Instruction reordering
Encryption
Compression
Reversing of branch conditions
Equivalent instruction substitution
Basic block reordering
...
June 2011
Somesh Jha: Behavior-Based Malware Detection
23
Evasion Through Junk Insertion
lea
nop
push
push
nop
call
nop
pop
push
lea
call
pop
nop
push
nop
push
nop
lea
pop
push
lea
call
pop
push
push
nop
lea
push
call
June 2011
eax, [ebp+Data]
offset aServices_exe
offset
eax
aServices_exe
_strcat
ecx
eax
eax, [ebp+Data]
_strcat
ecx
edi
eax
eax, [ebp+ExistingFileName]
ecx
eax
eax,
ds:CopyFileA
[ebp+Data]
ecx
edi
eax
8D
68
50
E8
59
8D
59
57
50
8D
50
FF
85 D8 FE FF FF
78 8E 40 00
69 06 00 00
85 D8 FE FF FF
85 D4 FD FF FF
15 C0 60 40 00
Signature
eax, [ebp+ExistingFileName]
eax
ds:CopyFileA
Somesh Jha: Behavior-Based Malware Detection
24
Evasion Through Reordering
lea
lea
eax, [ebp+Data]
eax, [ebp+Data]
nop
jmp label_one
push
offset aServices_exe
nop
label_two:
nop
lea
eax, [ebp+Data]
push
...
eax
call
push
_strcateax
nop
call
ds:CopyFileA
nop
jmp label_three
nop
pop
label_one:
ecx
lea
...
eax, [ebp+Data]
pop
call
ecx
_strcat
push
...
edi
push
jmp
eaxlabel_two
nop
lea
label_three:
eax, ...
[ebp+ExistingFileName]
push
eax
call
ds:CopyFileA
June 2011
Somesh Jha: Behavior-Based Malware Detection
8D 85
90*
68 78
90*
50
90*
E8 69
90*
59
90*
.
.
.
90*
50
90*
FF 15
D8 FE FF FF
8E 40 00
06 00 00
C0 60 40 00
Regex Signature
25
Evasion Through Encryption
lea
eax, data_area
esi,
[ebp+Data]
jmp label_one
mov
ecx, 37
again:
label_two:
xor byte ptr [esi+ecx], 0x01
lea
loop
eax,
again[ebp+Data]
...
jmp
data_area
push
.
eax
call
.
ds:CopyFileA
jmp
.
label_three
data_area:
label_one:
db
8C 84 D9 FF ...
...
.
call
.
_strcat
...
.
jmp
db label_two
FE 14 C1 61 ...
label_three: ...
8D 85
90*
68 78
90*
50
90*
E8 69
90*
59
90*
.
.
.
90*
50
90*
FF 15
D8 FE FF FF
8E 40 00
06 00 00
C0 60 40 00
Regex Signature
June 2011
Somesh Jha: Behavior-Based Malware Detection
26
Evasion Through Evolution
• Malware writers are good at software
engineering:
– Modular designs
– High-level languages
– Sharing of exploits, payloads, and evasion
techniques
Example:
Beagle e-mail virus gained additional
functionality with each version.
June 2011
Somesh Jha: Behavior-Based Malware Detection
27
Beagle Evolution
Source: J. Gordon, infectionvectors.com
• More than 100 variants, not counting
associated components.
Formglieder
Mitglieder
Tarno
Bank Info Theft
Spam relay
Password Theft
Beagle
Tooso
Weakens security
June 2011
LDPinch
Mass mailer
Password Theft
Lodear
Monikey
Update Engine
Propagation Mgr
Somesh Jha: Behavior-Based Malware Detection
28
Outline
•
•
•
•
Introduction
Threat Model
Behavior-Based Detection
Mining Malicious Behaviors
June 2011
Somesh Jha: Behavior-Based Malware Detection
29
Empirical Study
[Christodorescu & Jha, ISSTA 2004]
• Start with a set of known viruses.
• Create obfuscated versions:
– Reordering
– Register/variable renaming
– Encryption
• Measure resilience to obfuscation
(detection rate of obfuscated versions)
June 2011
Somesh Jha: Behavior-Based Malware Detection
30
Evaluation Goal: Resilience
Question 1:
• How resistant is a virus scanner to
obfuscations or variants of known worms?
Question 2:
• Using the limitations of a virus scanner,
can a blackhat determine its detection
algorithm?
June 2011
Somesh Jha: Behavior-Based Malware Detection
31
Outline
•
•
•
•
•
Introduction
Threat Model
Evaluation of Current Detectors
Behavior-Based Detection
Future Directions
June 2011
Somesh Jha: Behavior-Based Malware Detection
32
Describing Malicious Behavior
[Christodorescu et al., Oakland 2005]
• Informal description:
“Mass-mailing virus”
• A more precision description:
“A program that:
sends messages containing copies of
itself,
using the SMTP protocol,
in a large number over a short period
of time.”
June 2011
Somesh Jha: Behavior-Based Malware Detection
33
Malspec
• A specification of behavior.
push
push
push
call
...
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
push
push
push
call
connect(Y);
=
Y
+
send(Z,T);
“HELO”
Z
Syntactic info
T
Semantic info
Malware Instance
(Netsky.B)
June 2011
Malspec
Somesh Jha: Behavior-Based Malware Detection
34
Obfuscation Preserves Behavior
push
push
push
call
...
push
push
push
call
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
push
nop
push
xor
xor
push
call
...
push
push
pop
push
push
call
10h
eax
eax, ebx
eax, ebx
edi
connect
; compose SMTP
; command "HELO ..."
eax
eax
eax
ecx
edi
send
• Junk insertion + code
reordering.
June 2011
Somesh Jha: Behavior-Based Malware Detection
35
Obfuscation Preserves Behavior
push
push
push
call
...
push
push
push
call
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
• Junk insertion + code
reordering.
June 2011
push
nop
push
jmp
L4: push
push
jmp
L2: xor
push
call
...
push
push
jmp
L1: xor
jmp
L3: pop
jmp
L5: call
Somesh Jha: Behavior-Based Malware Detection
10h
eax
L1
ecx
edi
L5
eax, ebx
edi
connect
; compose SMTP
; command "HELO ..."
eax
eax
L3
eax, ebx
L2
eax
L4
send
36
Obfuscation Preserves Behavior
push
push
push
call
...
push
push
push
call
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
• Junk insertion + code
reordering.
June 2011
push
nop
push
jmp
L4: push
push
jmp
L2: xor
push
call
...
push
push
jmp
L1: xor
jmp
L3: pop
jmp
L5: call
Somesh Jha: Behavior-Based Malware Detection
10h
eax
L1
ecx
edi
L5
eax, ebx
edi
connect
; compose SMTP
; command "HELO ..."
eax
eax
L3
eax, ebx
L2
eax
L4
send
37
Evolution Preserves Behavior
push
push
push
call
...
push
push
push
call
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
push
push
push
call
...
jnz
...
10h
eax
edi
connect
; check return code
error_handler
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
; check return code
error_handler
push
push
push
call
...
jnz
...
error_handler:
...
• Add error handling.
June 2011
Somesh Jha: Behavior-Based Malware Detection
38
Evolution Preserves Behavior
push
push
push
call
...
push
push
push
call
10h
eax
edi
connect
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
push
push
push
call
...
jnz
...
10h
eax
edi
connect
; check return code
error_handler
; compose SMTP
; command "HELO ..."
eax
ecx
edi
send
; check return code
error_handler
push
push
push
call
...
jnz
...
error_handler:
...
• Add error handling.
June 2011
Somesh Jha: Behavior-Based Malware Detection
39
Detection Using Malspecs
Malspec
Static detection:
Given an executable
binary, check whether it
satisfies the malspec.
φ
Just like model checking, but...
• Malicious code allows no
assumptions to be made
• Real-time constraints
June 2011
Somesh Jha: Behavior-Based Malware Detection
40
A Behavior-Based Detector
• Match the syntactic constructs, then
check the semantic information.
connect(Y);
Y
“HELO”
send(Z,T);
Z
Syntactic info
T
Semantic info
Malspec
June 2011
Somesh Jha: Behavior-Based Malware Detection
41
Check the Semantic Info
Program (Netsky.O):
push
push
push
call
...
push
lea
push
call
10h
eax
[ebp+s]
connect
ebx
eax, [ebp+s]
eax
send_email
connect(Y);
“HELO”
send(Z,T);
T
Z
send_email()
...
lea
push
lea
push
call
; compose SMTP
; command “HELO ..."
eax, [ebp+arg1]
eax
eax, [ebp+buffer]
eax
SMTP_send_and_rcv
Syntactic info
Somesh Jha: Behavior-Based Malware Detection
Semantic info
Malspec
SMTP_send_and_rcv()
push
push
mov
push
call
June 2011
Y
eax
[ebp+arg1]
eax, [ebp+arg2]
[eax]
send
42
Check with the Oracle
• Assume we have an oracle that can
validate value predicates.
Does
eax before == ebx after
for the code sequence:
push eax
call foo
mov ebx, [ebp+4]
?
Yes.
June 2011
Somesh Jha: Behavior-Based Malware Detection
43
Check the Semantic Info
Program (Netsky.O):
A:
push
push
push
call
...
push
lea
push
call
10h
eax
[ebp+s]
connect
ebx
eax, [ebp+s]
eax
send_email
connect(Y);
“HELO”
send(Z,T);
T
Z
send_email()
...
lea
push
lea
push
call
Syntactic info
; compose SMTP
; command “HELO ..."
eax, [ebp+arg1]
eax
eax, [ebp+buffer]
eax
SMTP_send_and_rcv
B:
June 2011
Y
Somesh Jha: Behavior-Based Malware Detection
Semantic info
Malspec
SMTP_send_and_rcv()
push
push
mov
push
call
eax
[ebp+arg1]
eax, [ebp+arg2]
[eax]
send
44
Query the Oracle
Program (Netsky.O):
A:
push
push
push
call
...
push
lea
push
call
10h
eax
[ebp+s]
connect
ebx
eax, [ebp+s]
eax
send_email
connect(Y);
“HELO”
send(Z,T);
T
Z
Does
send_email()
Syntactic info
memory[ebp@A+4] ==
...
; compose SMTP
memory[ebp@B+4]
hold
; command “HELO ..."
for the code
sequence
lea
eax, [ebp+arg1]
between
push
eaxA and B?
lea
push
call
eax, [ebp+buffer]
eax
SMTP_send_and_rcv
Yes.
B:
June 2011
Y
Somesh Jha: Behavior-Based Malware Detection
Semantic info
Malspec
SMTP_send_and_rcv()
push
push
mov
push
call
eax
[ebp+arg1]
eax, [ebp+arg2]
[eax]
send
45
A Recipe for an Oracle
• Instance of program verification problem:
Does program P respect property φ ?
Code
Fragment P
Expressions
e1, …, ek
June 2011
More powerful, higher cost
Pattern
Matching
Random
Execution
Yes
No
Simplify
UCLID
Theorem Prover Model Checker
Somesh Jha: Behavior-Based Malware Detection
Yes
Yes
46
A Behavior-Based Prototype
• Developed malspecs for several families
of worms.
• No false positives.
• Improved resilience to common
obfuscations.
June 2011
Somesh Jha: Behavior-Based Malware Detection
47
Evaluation of Malspecs
Decryption sig
Prototype
detector
Netsky.B
Mass-mailing sig
Netsky.C

Netsky.D

Netsky.O

Netsky.P

Netsky.T

Netsky.W

McAfee uses individual signatures for each worm.
Malspecs provide forward detection.
June 2011
Somesh Jha: Behavior-Based Malware Detection
48
Performance
• Prototype is slower than commercial antivirus tools.
Malware Family
Netsky
Beagle
Running Time
Average
Std. Deviation
99.57 s
56.41 s
41.01 s
40.72 s
• Plenty of room for improvement.
e.g. disassembler: 25% of time.
June 2011
Somesh Jha: Behavior-Based Malware Detection
49
Evaluation: False Positive Rate
• Tested the malspecs on 2,000 benign
Windows binaries.
• False positive rate: 0%
Disassembly rate
100%
80%
60%
40%
20%
0%
0B
35,840 B
71,680 B
107,520 B
143,360 B
Program size (grouped in 5 kB increments)
June 2011
Somesh Jha: Behavior-Based Malware Detection
50
Evaluation: Obfuscation Resilience
• Different types garbage insertion applied
to Beagle.Y to obtain more variants.
Obfuscation Type
Behavior-Based Detection
McAfee
Average Time
Detection Rate
Nop insertion
74.81 s
100%
75%
Stack op. insertion
159.10 s
100%
25%
Math op. insertion
186.50 s
95%
5%
June 2011
Somesh Jha: Behavior-Based Malware Detection
51
Formally Assessing Resilience
[POPL 2007]
• Soundness (no false positives)
• Completeness (no false negatives)
Y
Program
Detector
Malspec
“HELO”
Z
Obfuscation
agmoPrr
June 2011
T
?
Somesh Jha: Behavior-Based Malware Detection
52
Approach to Assessing Resilience
• Detector “filters out” irrelevant aspects
of the program (described in terms of
Program
trace semantics).
Abstraction
Y
Program
Detector
Program
=
Malspec
“HELO”
Z
agmoPrr
June 2011
T
?
Somesh Jha: Behavior-Based Malware Detection
53
Dynamic
Behavior-Based Detection
• Threatfire
• Sana Security
• Novashield
June 2011
Somesh Jha: Behavior-Based Malware Detection
54
NovaShield Behavior Engine
Architecture
User
User
User
Process
Process
Process
Security
Policies
OS Kernel
File
Monitor
Registry
Monitor
Process
Monitor
Network
Monitor
Behavior
Engine
Additional Information
• Papers
– M. Christodorescu and S. Jha, Testing Malware
Detectors, International Sympoisum on Testing and
Analysis (ISSTA), 2004
– M. Christodorescu, S. Seshia, S. Jha, D. Song, and R.
Bryant, Semantics-Aware Malware Detection, IEEE
Symposium on Security and Privacy (Oakland), 2005.
– M. Dalla Preda, M. Christodorescu, S. Debray and S.
Jha, A Semantics-Based Approach to Malware
Detection, Symposium on Principles of Programming
Languages (POPL), January 2007.
• Website
– http://www.cs.wisc.edu/~jha/
June 2011
Somesh Jha: Behavior-Based Malware Detection
56
Behavior-Based Detection
The old way – match syntactic signatures:
One-toone
< 50%
detection
The new way – examine underlying behavior:
One-tomany
Specifying Behaviors
NtOpenKey
“…\CurrentVersion\Run
”
NtDeleteValueKey
“McAfee Firewall”
Specifying Behaviors
Behavior-graph representation
– Nodes epresent events & arguments
• System calls, library calls, high-level events
– Edges represent data dependencies
• Data substring equality, resource generation/use
– Argument values are crucial!
June 2011
Somesh Jha: Behavior-Based Malware Detection
59
Finding the Needle in the
Haystack
NtOpenKey
“…\InternetSettings\...
”
NtOpenKey
“…\CurrentVersion\Run
”
NtSetValueKey
“ProxyBypass”
NtDeleteValueKey
“McAfee Firewall”
Large, Complex Problem
• Behavior graphs are large
– Between tens of thousands to millions of
nodes
• New malware is ever-present
– Lower bound of 7,933 samples/day in 2009
• Large, diverse benign application pool
– Windows 7 is backwards compatible to NT/95
• Manual analysis, brute force not feasible
Large, Complex Problem
• Behavior graphs are large
– Between tens of thousands to millions of
nodes
• New malware is ever-present
– Lower bound of 7,933 samples/day in 2009
• Large, diverse benign application pool
– Windows 7 is backwards compatible to NT/95
• Manual analysis, brute force not feasible
62
Synthesizing Optimal Malware Specifications
June 2011
Our Contributions
• New specification-synthesis algorithm
– Perform efficient, large-scale data mining
first to uncover suspicious behaviors
– Probabilistically refines and optimizes
specifications
• Key algorithms scale to real problem size
– Reduces the window of vulnerability
• Tunable true positive/false positive rate
– 86% TP for low FP, 100% TP for higher FP
63
Synthesizing Optimal Malware Specifications
June 2011
Holmes: Our Approach to
Specification Synthesis
•Roadmap:
– Workflow
1.Mine significant behaviors
2.Synthesize specification
– Results
– Conclusion
NtOpenKey
“…\CurrentVersion\Run
”
NtDeleteValueKey
“McAfee Firewall”
Significant Behaviors
NtOpenKey
“…\CurrentVersion\Run
”
NtDeleteValueKey
“McAfee Firewall”
• Significant behaviors discriminate between
labeled malicious and benign sets
• Measured statistically via frequency
counting of subgraphs
– Can use information gain, cross entropy, G-test,
…
Key Requirement
• Significant behavior appears in many
malware graphs, few benign graphs
Leap Mining: Extracting
Significant Behaviors
• Want to find subgraph that optimizes
significance measure
• Problem: Number of candidate subgraphs
is factorial in # Nodes + # Edges
Leap Mining (Contd)
• Insight: Correlation between structural
similarity, significance score similarity to
guide search [Yan et al., SIGMOD ‘08]
– “Leap” over branches in search tree with
similar structure
• Future: Probabilistically compress source
graphs to mine behaviors more efficiently
[Chen et al, VLDB ‘09]
June 2011
Somesh Jha: Behavior-Based Malware Detection
68
Leap Mining: Example
Significance
score
similar to
parent!
Most
significant
pattern!
This means
we can prune
siblings
Significance
0.1
0.2
0.8
0`
Holmes: Our Approach to
Specification Synthesis
•Roadmap:
– Workflow
1. Mine significant behaviors
2. Synthesize specification
– Results
– Conclusion
NtOpenKey
“…\CurrentVersion\Run
”
NtDeleteValueKey
“McAfee Firewall”
Naïve Synthesis: Just Significant
Behaviors
• Use all significant behaviors exhibited by
a specific sample
• Pros:
– Not path-dependent
– Significance metric likely to select behaviors
that give low false positives
• Cons:
– Some significant behaviors may be variantspecific  false negatives!
– Some samples may not exhibit many mined
suspicious behaviors  false positives!
Searching for the Optimal
Specification
• Insight: significant behaviors are
suspicious behaviors
• A good spec. is the right combination of
suspicious behaviors
• Given a malware set, search using
concept analysis
– Concept is a pair: ({malware samples},
{suspicious behaviors})
– Find set of concepts with optimal true/false
positive characteristics
Simulated Annealing
• Concept space is enormous: factorial in number of suspicious behaviors
• Simulated annealing: probabilistic search over localized portions of
solution space
– Derive new solutions greedily most of the time
– With certain probability, move to sub-optimal solutions in the search 
avoid local minima
– Known sampling methods, cooling schedules to guarantee optimal
convergence
Simulated Annealing: Example
Probabilistically
take sub-optimal
solution!
Detection Rate
11
8`
7
6
False Positives
1
0
5
Workflow
Behavior Mining
Known Malware
Specification Synthesis
Significant Behaviors
Benign Apps
Recent Malware
Benign Apps
75
Discriminative
Specification
Holmes: Our Approach to
Specification Synthesis
• Roadmap:
– Workflow
1.Mine significant behaviors
2.Synthesize specification
– Results
– Conclusion
NtOpenKey
“…\CurrentVersion\Run
”
NtDeleteValueKey
“McAfee Firewall”
Evaluation Workflow
Behavior Mining
Known Malware
Behavior-Based
Malware Detection
Specification Synthesis
(with 10-fold cross-validation)
Significant Behaviors
Discriminative
Specification
492 samples
Benign Apps
166 behaviors
Recent Malware
11 apps
1 specification
New Malware
378 samples
Benign Apps
42 samples
Benign Apps
28 apps
28 apps
Detection
Results
Corpus Details
• 912 malware samples
– 18 AV-labeled families
• Spyware, worms, bots, filesystem viruses, …
– 492 samples in 6 families for mining
– 420 samples in 12 families for synthesis &
evaluation
• 49 benign applications
– Behaviorally-diverse set: browsers, system
administration, media…
78
Synthesizing Optimal Malware Specifications
June 2011
Corpus Details (Contd)
• Trace collection accounts for a single
path
– 120 seconds for malware
– Typical usage patterns for benign
applications
Behavior Mining Results
• Mined 109 unique behaviors
– 18.1 per family, on average
– 77 manually deemed malicious
• Non-malicious behaviors due to sample size
• Most behaviors correspond to those in AV
databases
– Mined some unreported by AV, e.g. code
injection & browser reconfiguration in worms
and viruses
– Some behaviors missing (likely) due to singlepath collection
80
Synthesizing Optimal Malware Specifications
June 2011
Specification Synthesis Results
• 0 FP on test corpus for 86.5% detection rate
• TP/FP tradeoff configurable
• Better than commercial AV on our corpus: Sana (42.61%), Threatfire
(61.70%)
81
Synthesizing Optimal Malware Specifications
June 2011
Specification Synthesis Results
• 0 FP on test corpus for 86.5% detection rate
• TP/FP tradeoff configurable
• Better than commercial AV on our corpus: Sana (42.61%), Threatfire
(61.70%)
82
Synthesizing Optimal Malware Specifications
June 2011
Performance and Scalability
• Behavior mining runtime varies between
families
– Worst-case exponential; can tweak tradeoff
in accuracy
– Similarity between malicious/benign graphs
affects runtime
– Can easily parallelize for linear speedup
• Specification synthesis works quickly
– Most specifications found in under one
minute (near-optimal solutions)
– Optimal solution can be found in exponential
time using same algorithm
Synthesizing Optimal Malware Specifications
June 2011
Conclusions
• Synthesizing specifications is hard!
NtOpenKey
• Holmes utilizes large-scale data
mining to
“…\CurrentVersion\Run
”
extract suspicious behaviors
• Holmes probabilistically searches for
NtDeleteValueKey
near-optimal specifications using
“McAfee Firewall”
suspicious behaviors
• Detection results beat industry results
• Algorithms scale to real problem size
84
Synthesizing Optimal Malware Specifications
June 2011
Additional Information
• Matt Fredrikson, Somesh Jha, Mihai
Christodorescu, Reiner Sailer, Xifeng Yan
– Synthesizing Near-Optimal Malware
Specifications from Suspicious Behaviors.
– IEEE Symposium on Security and Privacy,
2010.
June 2011
Somesh Jha: Behavior-Based Malware Detection
85
Outline
•
•
•
•
•
Introduction
Threat Model
Evaluation of Current Detectors
Behavior-Based Detection
Future Directions
June 2011
Somesh Jha: Behavior-Based Malware Detection
86
Take aways
• Malware detection is $5-6 billion dollar
industry
• No well defined threat model
• Need to formally defined a threat model
and design detection techniques based on
it
• Behavior-based malware detection is a
move towards that vision
June 2011
Somesh Jha: Behavior-Based Malware Detection
87
On the theoretical side
• Can we prove oracle completeness
results?
– For example, if the oracle can give me a
perfect control-flow graph, I can handle
reordering heuristics perfectly
• How about bounding the adversary?
– Computational power (like in cryptography)
– Limit the class of obfuscations
June 2011
Somesh Jha: Behavior-Based Malware Detection
88
Questions?
Naïve Synthesis: Full
Specification
• Use entire behavior graph for malware
sample
• Pros:
– Fits malware very tightly
– Low false positives
• Cons:
– Path-specific: e.g. some looping/branching
behavior, non-determinism not critical for
specification
– Impossible to build full graph – behaviors not
in training run are not accounted for
90
Synthesizing Optimal Malware Specifications
June 2011
Specifying Behaviors
• Behavior graph representation
– Nodes represent events & arguments
• System calls, library calls, high-level events
– Edges represent data dependencies
• Data substring equality, resource generation/use
– Argument values are crucial!
NtOpenKey
DefUse(1, 1)
NtDeleteValueKey
91
Too
specific
general
!
NtOpenKey
(501, ACC_WRITE, “Run”,
)
Just
Right
DefUse(1, 1)
NtDeleteValueKey
(501, “… Firewall”, )
Synthesizing Optimal Malware Specifications
NtOpenKey
“…\CurrentVersion\Run”
DefUse(1, 1)
NtDeleteValueKey
“McAfee Firewall”
June 2011
Multi-Faceted Problem
• Detailed behavior information makes
large, data-rich raw source
• Difficult to extract complete behavior
information
– See multi-path problem [Cadar et al., CCS
‘06], [Moser et al., Oakland ‘07]
• Malicious and benign behaviors look
similar
– Benign application update vs. malicious
dropping
– Benign network activity vs. malicious C&C
92
Synthesizing Optimal Malware Specifications
June 2011
Start up
• There is a startup which is
commercializing some of the ideas
presented in this talk
• Securitas Technologies Inc.
– See www.securitastech.com
June 2011
Somesh Jha: Behavior-Based Malware Detection
93
Here be Dragons!
Disclaimer
Virus detection is undecidable.
[Cohen 1984]
Best approximation up to now:
byte signatures.
June 2011
Somesh Jha: Behavior-Based Malware Detection
95
My Proposal for a Solution
• Make the malware writer’s job as hard as
possible.
• Stop malware based on behavior:
– Employ semantics of instructions
– Use enforceable interfaces
– Combine static and dynamic techniques
June 2011
Somesh Jha: Behavior-Based Malware Detection
96
Current AV Detection Methods
• Scan strings
(byte sequences from a malicious executable)
– Enhanced using regular expressions
• Heuristics
– Binary file structure
– APIs used
– Byte (n-gram) distribution
June 2011
Somesh Jha: Behavior-Based Malware Detection
97
Previous Research
• Different structures over bytes
N-gram distributions
[Li, Wang, & Stolfo, SMC 2005]
Neural networks, Bayes
[Arnold & Tesauro, VB2000]
Additional features: DLL imports, syscalls
[Schultz, Eskin, Zadok, & Stolfo, Oakland 2001]
• Different information about the program
Slices from syscalls
[Lo, Levitt, & Ollson, 1995]
Recovery of high-level constructs
[Bergeron, Debbabi, Erhioui, & Ktari, SREIS 2001]
Model checking
[Kinder, Katzenbeisser, Schallhart, & Veith, DIMVA 2005]
June 2011
Somesh Jha: Behavior-Based Malware Detection
98
Key Observations
Variants : New strains of viruses that
borrow code, to varying degrees, directly
from other known viruses.
Source: Symantec Security Response Glossary
• Syntactic signatures cannot capture
variants.
• Syntactic signature methods do not scale.
Need to focus on behavior.
June 2011
Somesh Jha: Behavior-Based Malware Detection
99
Behavior-Based Detection
• How to describe malicious behavior?
• How to identify malicious behavior?
My
Previous
Research
– Static Techniques
– Static + Dynamic Techniques
• How to automatically learn
malicious behavior?
Proposed
Research
• How effective are these techniques?
June 2011
Somesh Jha: Behavior-Based Malware Detection
100
Previous
Research
A Language to Describe
Malicious Behaviors
Establishing a Threat Model
A threat model has three components:
• Attack Model
: Malicious Behavior
How is the attack performed?
• Defensive Goal
: Trusted Computing Base
What is the system designed to protect?
• Time
: Forever?
How long is the protection operational?
June 2011
Somesh Jha: Behavior-Based Malware Detection
102
Choosing a TCB
►
►
►
Program
Libraries
OS Kernel
Processor
API calls
System calls
Instructions
TCB:
Libraries/Interpreter
TCB:
+
OS
OS
TCB:
+ Processor
Processor
• Interface to TCB has to be enforceable.
For this talk: TCB = OS + Processor.
June 2011
Somesh Jha: Behavior-Based Malware Detection
103
Formal Definition of Malspec
Σ = { σk }k≥1 is the set of system calls
V = { vi }i≥1 is the set of uninterpreted vars
A is a logic of formulas over V
G = (N,E) is a graph:
Vertices are labeled with system calls from Σ
instantiated with variables from V.
Edges are labeled with predicates in A.
June 2011
Somesh Jha: Behavior-Based Malware Detection
104
Malspec Benefits
• Representation-independent
–
–
–
–
Depends only on the interface to the TCB
Ignores functions boundaries
Ignores specific data structures
Ignores process boundaries
• Order-independent
– Allows any order of operations, as long as the
dependence predicates are satisfied.
June 2011
Somesh Jha: Behavior-Based Malware Detection
105
Previous
Research
Static Detection of
Malicious Behavior
Step 1: Matching Nodes
Straightforward…
… except for encrypted code!
• Encryption & compression effectively
hide the system calls (i.e., the TCB
operations).
• Solution: Malware normalization
June 2011
Somesh Jha: Behavior-Based Malware Detection
107
A Malware Normalizer
• Dynamic analysis technique:
– Run program in a contained environment
– Stop as soon as control flow reaches a
previously written address
– Reconstruct program with current memory
snapshot
Packed
Executable
Normalizer
Unpacked
Executable
Qemu
(system
emulator)
June 2011
Somesh Jha: Behavior-Based Malware Detection
108
Detector Characteristics
• Intraprocedural:
– Flow sensitive
Handles many
syntactic
obfuscations
• Interprocedural:
– Context sensitive
OR
– Context insensitive
June 2011
Handles changes
through
evolution
Somesh Jha: Behavior-Based Malware Detection
109
Step 2: Predicate Verification
Check whether a program path satisfies the
corresponding malspec predicate.
Requirements for the predicate logic:
• Addition, comparison, multiplication
• Bit-vector arithmetic
• Arrays
• On 32-bit values (and soon 64-bit values)
June 2011
Somesh Jha: Behavior-Based Malware Detection
110
A Simple Verifier
Malspec
For predicates that express
preservation of values.
φ
φ(A): A1 = A2
• Syntactic check:
Compare code sequence with a
known set of obfuscations
– Nops, pushes & pops
– Operations on non-live variables
June 2011
Somesh Jha: Behavior-Based Malware Detection
111
Preliminary Results
[Christodorescu & Jha, USENIX Security 2003]
Detection succeeds in the presence of:
– Code reordering
– Simple junk insertion
– Register renaming
Zero missed detections
(compared to very high missed detection
rates for commercial virus scanner)
June 2011
Somesh Jha: Behavior-Based Malware Detection
112
A Value-Preservation Verifier
Malspec
Express program path as a
state transformer.
φ
– Use instruction semantics
∂
∂φ?
Use decision procedures.
June 2011
Somesh Jha: Behavior-Based Malware Detection
113
Verification Tools
• Instance of program verification problem:
Does program P respect property φ ?
Code
Fragment
Predicate
φ
More powerful, higher cost
Pattern
Matching
Random
Execution
Yes
No
Simplify
UCLID
Theorem Prover Model Checker
Yes
Yes
Random
Abstract
Interpretation
June 2011
Somesh Jha: Behavior-Based Malware Detection
114
Evaluation of Value-Preservation
[Christodorescu & Jha, Oakland 2005]
Decryption malspec
Netsky.B
Prototype
detector
Mass-mailing malspec
Netsky.C

Netsky.D

Netsky.O

Netsky.P

Netsky.T

Netsky.W

McAfee uses individual signatures for each worm.
Semantic malspecs provide forward detection.
June 2011
Somesh Jha: Behavior-Based Malware Detection
115
Architecture (up to now)
Executable
Malware
Normalizer
Normalized
Executable
Semantics-Aware
Malware Detector
Malspec
Library
Decision Procedures
Semantic Query Engine
Static Analyses
Instruction/Syscall Semantics
June 2011
Somesh Jha: Behavior-Based Malware Detection
116
Proposed
Research
Hybrid Detection of
Malicious Behavior
Static Analysis is Not Perfect
• Safety at the cost of precision
– Good for strict security, bad for usable
security.
φ
Perl interpreter
June 2011
Somesh Jha: Behavior-Based Malware Detection
118
Imprecision of Static Analysis
• Many sources of imprecision :
–
–
–
–
Disassembly
Control flow reconstruction
Loops, recursion
Malspec predicate verification (decision
procedures)
• Leads to false positives
June 2011
Somesh Jha: Behavior-Based Malware Detection
119
Dynamic Analysis
• As precise as possible for a particular
execution
– Can retrieve any part of program state
– Adds time dimension
• But... adds runtime overhead
– Emulators are orders of magnitude slower
June 2011
Somesh Jha: Behavior-Based Malware Detection
120
A Hybrid Malware Detector
Combine static + dynamic
– Identify where static analysis loses precision
– Have the dynamic analyzer check those
locations
Detection goal:
Check only whether malicious behavior
appears in the current execution.
Small (<10%) runtime overhead needed.
June 2011
Somesh Jha: Behavior-Based Malware Detection
121
Example
Static Stage
Dynamic Stage
φ
Runtime
monitoring

Perl interpreter
Perl interpreter
Runtime monitor determines whether
portion of trace satisfies predicate.
June 2011
Somesh Jha: Behavior-Based Malware Detection
122
Hybrid Detector Operation
1. Determine path validity
Static analysis identifies a certain path as
possibly malicious.
Dynamic analysis confirms that the current
execution trace follows that path.
2. Check that trace satisfies predicate
At the end of the trace segment that matches
the path, verify the malspec predicate.
June 2011
Somesh Jha: Behavior-Based Malware Detection
123
Architecture [hybrid]
Executable
Malware
Normalizer
Normalized
Executable
Semantics-Aware
Malware Detector
Malspec
Library
Decision Procedures
Semantic Query Engine
StaticStatic
+ Dynamic
Analyses
Analyses
Instruction/Syscall Semantics
June 2011
Somesh Jha: Behavior-Based Malware Detection
124
Proposed
Research
Automatic Extraction of
Malicious Behavior
Deriving Malspecs
Goal:
Extract a malspec from a sample program
labeled as malicious.
• Requirements
– Capture behavior, not implementation
– Low to no false positives
multiple samples
Two options
one sample
June 2011
Somesh Jha: Behavior-Based Malware Detection
126
Malspec from Multiple Samples
Learning a malspec from multiple samples:
1. Identify common sequences of system
calls.
–
Subgraph isomorphism
2. For each pair of system calls, construct
a predicate describing the actual code
paths.
June 2011
Symbolic execution, human expert
Somesh Jha: Behavior-Based Malware Detection
127
Example
Beagle.B
Beagle.C
foo( A, B )
X =socket()
X =socket()
write( A, B )
connect( Y )
connect( Y )
write( Z, “EHLO ...” )
foo( Z, “EHLO ...” )
write( A, “TO ” + address )
foo( A, “TO ” + address )
write( B, “DATA” )
foo( B, “DATA” )
write( C, body )
foo( C, body )
close( D )
close( D )
June 2011
read( C )
Somesh Jha: Behavior-Based Malware Detection
128
Malspec from One Sample
Additional semantic information needed
• System call API usage rules
– Provides sequencing information and some
data flow information
• Network protocol semantics
– Provides sequencing information and
additional data flow information
June 2011
Somesh Jha: Behavior-Based Malware Detection
129
Example: Beagle.B
System call rules:
socket
X =socket()

connect
(write|read)*

 close
connect( Y )
write( Z, “EHLO ...” )
SMTP protocol:
write( A, “TO ” + address )
write( B, “DATA” )
write( C, body )
write(“EHLO”)
write(“MAILTO”+addr)
write(“DATA”)
write(body)
close( D )
June 2011
Somesh Jha: Behavior-Based Malware Detection
130
Complete Architecture
Executable
Malware
Normalizer
Normalized
Executable
Malspec
Generator
Semantics-Aware
Malware Detector
Malspec
Library
Decision Procedures
Semantic Query Engine
Static + Dynamic Analyses
Instruction/Syscall Semantics
June 2011
Somesh Jha: Behavior-Based Malware Detection
131
Proposed
Research
Theoretical Limits of
Behavior-Based Detection
What Does This Buy Us?
• How strong (theoretically) is this system?
OR
How much harder does the malware
writer have to work to evade my system?
Goal:
“Design” a computationally-bounded
adversary. Assess the behavior-based
detector against this adversary.
June 2011
Somesh Jha: Behavior-Based Malware Detection
133
Timeline
2005
June
2006
Malspec extraction
from many samples
Malspec extraction
from one sample
June 2011
Hybrid detection
• runtime monitor
• path checking
• predicate checking
Somesh Jha: Behavior-Based Malware Detection
2007
June
Theoretical work
Thesis writing
Interview season
134
Behavior-Based Malware Detection
Somesh Jha
Joint work with Mihai Christodorescu
June 2011
Somesh Jha: Behavior-Based Malware Detection
136
Step 2: Unification
• One-way unification to associate program
expressions with the uninterpreted
variables in the malspec.
• Result: one binding map for each
matched pair (malspec node, program
location).
June 2011
Somesh Jha: Behavior-Based Malware Detection
137
Evaluation: Obfuscation Resilience
• Different types junk insertion applied to
Beagle.Y to obtain more variants.
Obfuscation Type
Semantics-Aware Detection
McAfee
Average Time
Detection Rate
Nop insertion
74.81 s
100%
75%
Stack op. insertion
159.10 s
100%
25%
Math op. insertion
186.50 s
95%
5%
June 2011
Somesh Jha: Behavior-Based Malware Detection
138
Problems with Dynamic Analysis
• Execution may have affected the host
machine in a malicious way.
Goal:
Stop execution as soon as it
enters a path that is
certainly malicious.
Perl interpreter
• Static analysis can help identify these
points of no return.
June 2011
Somesh Jha: Behavior-Based Malware Detection
139