NetShield: Massive Semantics-Based
Vulnerability Signature Matching
for High-Speed Networks
Zhichun Li, Gao Xia, Hongyu Gao, Yi Tang, Yan
Chen, Bin Liu, Junchen Jiang, and Yuezhou Lv
NEC Laboratories America, Inc.
Northwestern University
Tsinghua University
1
To keep network safe is a grand
challenge
Worms and Botnets are still popular
 e.g. Conficker worm outbreak in
2008 and infected 9~15 million hosts.
2
NIDS/NIPS Overview
NIDS/NIPS (Network Intrusion
Detection/Prevention System)
Signature
DB
Packets
NIDS/NIPS
`
`
`
Security • Accuracy
alerts
• Speed
3
State Of The Art
Regular expression (regex) based approaches
Used by: Cisco IPS, Juniper IPS, open source Bro
Example: .*Abc.*\x90+de[^\r\n]{30}
Pros
Cons
• Can efficiently match
multiple sigs
simultaneously,
through DFA
• Can describe the
syntactic context
• Limited expressive
power
• Cannot describe the
semantic context
• Inaccurate
4
State Of The Art
Vulnerability Signature [Wang et al. 04]
Blaster Worm (WINRPC) Example:
Vulnerability: design flaws enable the bad
BIND:
inputs lead&&
therpc_vers_minor==1
program to a bad&&
state
rpc_vers==5
packed_drep==\x10\x00\x00\x00
Good
&& context[0].abstract_syntax.uuid=UUID_RemoteActivation state
BIND-ACK:
Bad input
rpc_vers==5
&& rpc_vers_minor==1
CALL:
rpc_vers==5 && rpc_vers_minors==1 && packed_drep==\x10\x00\x00\x00
Bad
Vulnerability
&& opnum==0x00 && stub.RemoteActivationBody.actual_length>=40
state
Signature
&& matchRE(stub.buffer, /^\x5c\x00\x5c\x00/)
Pros
• Directly describe
semantic context
• Very expressive, can
express the vulnerability
condition exactly
• Accurate
Cons
• Slow!
• Existing approaches all
use sequential matching
• Require protocol parsing
5
Regex vs. Vulnerabilty Sigs
Vulnerability Signature matching
Parsing
Matching
Combining
Regex cannot substitute parsing
Theoretical prospective
Practical prospective
Protocol Context
Context
Regex grammar
Sensitive
Free
• HTTP chunk encoding
• DNS label pointers
6
Regex V.S. Vulnerabilty Sigs
Regex + Parsing cannot solve the problem
• Regex assumes a single input
• Regex cannot help with combining phase
Cannot simply extend regex approaches
for vulnerability signatures
7
Speed
High
Motivation of NetShield
State of the
art regex Sig
IDSes
NetShield
Theoretical accuracy
limitation of regex
Low
Existing
Vulnerability
Sig IDS
Low
Accuracy
High
8
Research Challenges and Solutions
• Challenges
– Matching thousands of vulnerability
signatures simultaneously
• Sequential matching match multiple sigs.
simultaneously
– High speed protocol parsing
• Solutions (achieving 10s Gps throughput)
– An efficient algorithm which matches multiple
sigs simultaneously
– A tailored parsing design for high-speed
signature matching
9
– Code & ruleset release at www.nshield.org
Outline
•
•
•
•
•
Motivation
High Speed Matching for Large Rulesets
High Speed Parsing
Evaluation
Research Contributions
10
Background
• Vulnerability signature basic
– Use protocol semantics to express vulnerabilities
– Defined on a sequence of PDUs & one predicate for
PDU
Blastereach
Worm
(WINRPC) Example:
BIND:
– Example: ver==1 && method==“put” && len(buf)>300
rpc_vers==5 && rpc_vers_minor==1 && packed_drep==\x10\x00\x00\x00
context[0].abstract_syntax.uuid=UUID_RemoteActivation
•&&Data
representations
BIND-ACK:
– The &&
basic
data types used in predicates: numbers and
rpc_vers==5
rpc_vers_minor==1
CALL:strings
rpc_vers==5
&& rpc_vers_minors==1
– number
operators: ==, >,&&
<,packed_drep==\x10\x00\x00\x00
>=, <=
&& opnum==0x00 && stub.RemoteActivationBody.actual_length>=40
– String operators:
==, match_re(.,.), len(.).
&& matchRE(stub.buffer,
/^\x5c\x00\x5c\x00/)
11
Matching Problem Formulation
• Suppose we have n signatures, defined on k
matching dimensions (matchers)
– A matcher is a two-tuple (field, operation) or a fourtuple for the associative array elements
– Translate the n signatures to a n by k table
– This translation unlocks the potential of matching
multiple signatures simultaneously
Rule 4: URI.Filename=“fp40reg.dll” && len(Headers[“host”])>300
RuleID Method == Filename == Header == LEN
1
DELETE
*
*
2
POST
Header.php
*
3
*
awstats.pl
*
4
*
fp40reg.dll
name==“host”; len(value)>300
5
*
*
name==“User-Agent”; len(value)>544
12
Signature Matching
• Basic scheme for single PDU case
• Refinement
– Allow negative conditions
– Handle array cases
– Handle associative array cases
– Handle mutual exclusive cases
• Extend to Multiple PDU Matching (MPM)
– Allow checkpoints.
13
Difficulty of the Single PDU matching
Bad News
– A well-known computational geometric problem
can be reduced to this problem.
– And that problem has bad worst case bound
O((log N)K-1) time or O(NK) space (worst case
ruleset)
Good News
– Measurement study on Snort and Cisco ruleset
– The real-world rulesets are good: the
matchers are selective.
– With our design O(K)
14
Matching Algorithms
Candidate Selection Algorithm
1.Pre-computation: Decides the rule order
and matcher order
2.Runtime: Decomposition. Match each
matcher separately and iteratively combine
the results efficiently
15
Step 2: Iterative Matching
PDU={Method=POST, Filename=fp40reg.dll,
Header: name=“host”, len(value)=450}
S1={2} Candidates after match Column 1 (method==)
S2=S1 A2+B2 ={2} {}+{4}={}+{4}={4}
S3=S2 A3+B3={4} {4}+{}={4}+{}={4}
Si  Ai 1
Don’t care
RuleID Method == Filename
== Header == LEN
R1
R2
R3
1
2
DELETE
SiPOST
* matcher i+1 *
Header.php
*
*
3
*
awstats.pl
4
*
fp40reg.dll
5
*
*
Si  Ai 1
require
In Ai+1 len(value)>300
name==“host”;
matcher i+1
name==“User-Agent”; len(value)>544
16
Complexity Analysis
Three HTTP traces:
avg(|Si|)<0.04
• Merging complexity
Two WINRPC
– Need k-1 merging iterations
traces: avg(|Si|)<1.5
– For each iteration
• Merge complexity O(n) the worst case, since Si can
have O(n) candidates in the worst case rulesets
• For real-world rulesets, # of candidates is a small
constant. Therefore, O(1)
– For real-world rulesets: O(k) which is the
optimal we can get
17
Outline
•
•
•
•
•
Motivation
High Speed Matching for Large Rulesets.
High Speed Parsing
Evaluation
Research Contribution
18
High Speed Parsing
Tree-based vs. Stream Parsers
Keep the whole parse
Parsing and matching
VS.
tree in memory
on the fly
Parse all the nodes
in the tree
Only signature related
VS.
fields (leaf nodes)
• Design a parsing state machine
19
High Speed Parsing
• Build an automated parser generator,
UltraPAC
Protocol
Spec.
Signature
Set
Parsing State
Machine
Protocol
Parser
field_1:
length = 5;
goto field_5;
field_2:
length = 10;
goto field_6;
…
20
Outline
•
•
•
•
•
Motivation
High Speed Matching for Large Rulesets.
High Speed Parsing
Evaluation
Research Contributions
21
Evaluation Methodology
Fully implemented prototype
10,000 lines of C++ and
3,000 lines of Python
Deployed at a DC in Tsinghua
Univ. with up to 106Mbps
• 26GB+ Traces from Tsinghua Univ. (TH), Northwestern
(NU) and DARPA
• Run on a P4 3.8Ghz single core PC w/ 4GB memory
• After TCP reassembly and preload the PDUs in memory
• For HTTP we have 794 vulnerability signatures which
cover 973 Snort rules.
• For WINRPC we have 45 vulnerability signatures which
22
cover 3,519 Snort rules
Parsing Results
Trace
TH
DNS
TH
NU
TH
WINRPC WINRPC HTTP
Avg flow len (B)
77
879
596
6.6K 55K 2.1K
Throughput
(Gbps)
Binpac
Our parser
0.31
3.43
1.41
16.2
1.11
12.9
2.10 14.2 1.69
7.46 44.4 6.67
11.2
Max. memory per 16
11.5
15
11.6
15
3.6
14
Speed up ratio
NU
HTTP
3.1
14
DARPA
HTTP
3.9
14
connection
(bytes)
23
Parsing+Matching Results
8-core 11.0
Trace
TH
NU
TH
WINRPC WINRPC HTTP
NU
HTTP
DARPA
HTTP
Avg flow length (B)
879
596
6.6K
55K
2.1K
10.68
14.37
4
9.23
10.61
1.8
0.34
2.63
11.3
2.37 0.28
17.63 1.85
11.7 8.8
1.48
32
0.033 0.038 0.0023
28
28
28
Throughput (Gbps)
Sequential
CS Matching
Matching only time
speedup ratio
Avg # of Candidates 1.16
Avg. memory per
connection (bytes)
32
24
Scalability Results
Throughput (Gbps)
0
1
2
3
4
Performance
decrease
gracefully
0
200
400
600
# of rules used
800
25
Research Contribution
Make vulnerability signature a practical solution
for NIDS/NIPS
Regular Expression Exists Vul. IDS
NetShield
Accuracy
Poor
Good
Good
Speed
Good
Poor
Good
Memory
Good
??
Good
• Multiple sig. matching  candidate
selection algorithm
• Parsing  parsing state machine
Tools at www.nshield.org
26
Q&A
Q&A
27