Detecting Abnormal Traffic
Jennifer Rexford
Fall 2010 (TTh 1:30-2:50 in COS 302)
COS 561: Advanced Computer Networks
http://www.cs.princeton.edu/courses/archive/fall10/cos561/
Intrusions
• Intrusions
– Actions that attempt to bypass security mechanisms
– E.g., unauthorized access, inflicting harm, etc.
• Example intrusions
– Denial-of-service attacks
– Scans
– Worms and viruses
– Host compromises
• Intrusion detection
– Monitoring and analyzing traffic
– Identifying abnormal activities
– Assessing severity and raising alarms
2
Where to Detect Intrusions?
• End host: OS or application
– Includes logins, file I/O, program executions, etc.
– Can work with encrypted traffic and at lower speeds
– Avoid extra packet reassembly and ambiguity
• Network: at enterprise edge
– Single location for detecting and blocking attacks
– Avoid reliance on the end host, OS, user, ..
– Reduce overhead on the end host and network
• Network: in the backbone
– Multiple locations for detecting correlations in traffic
– Too expensive to inspect the contents of every packet
3
What to Do After Detection?
• Passively log
– Collect, analyze, and record data
– Report results to system/network administrator
– Allow human to drive any response
– Slower response, but better handling of false positives
• Actively defend
– Detect problems in real time
– Automatically generate a response
– E.g., drop the traffic, engage the adversary, fight back
– Faster response, but worse handling of false positives
4
How to Detect Intrusions?
• The challenge
– We don’t know all the bad things that could happen
– And telling good from bad is fundamentally hard
• Anomaly detection
– What is usual, is known
– What is unusual, is bad
• Signature detection
– What is bad, is know
– What is not bad, is good
5
How to Detect Intrusions?
Anomaly Detection
Patterns Train to create a
baseline of normal
network traffic
Detection Detect statistically
significant deviations
from normal
Pros
Can detect novel
(“zero day”) attacks
Signature Detection
Codify patterns of
known vulnerabilities
or attacks
Detect matches to
the patterns in the
signatures
Builds on past
experiences
Cons
Misses novel attacks;
requires continuous
updates to signatures
May miss low-rate
attacks; high rate of
false alarms
6
Anomaly Detection
• Traffic volume
– Detect deviations in bytes/sec or packets/sec over time
– Not effective for detection low-volume attacks
• Traffic features
– Detect changes in distributions of traffic characteristics
– E.g., traffic distribution by IP address, port number,
packet size, TCP flags, etc.
– Aids in classifying the anomaly (e.g., DoS vs. port scan)
• Detection techniques
– Statistical techniques
– Machine learning
– ...
7
Signature Detection
• Examples
– Excessive login attempts
– TCP packet with both SYN and RST set
– HTTP with “GET /cgi-bin/phf?”
alert tcp $EXTERNAL any -> $HTTP_SERVERS $HTTP_PORTS
(msg: “WEB-CGI phf access”; flow:to_server, established;
uricontent: “\phf”; nocase; reference: bugtraq, 629;
reference: arachnids, 128; reference: cve, CVE-1999-0067;
classtype: web-application-activity; sid: 886; rev: 8;)
• Packet processing
– Deep-packet inspection
– Regular expression matching
8
Hybrid Solution: Arbor Example
anomaly
detection
scrubber
http://www.arbornetworks.com/en/stopping-ddos-attacks.html
9
Challenges
• Accuracy
– False positives
– False negatives
• High volume of data
– Measuring the traffic at line rate
– Real-time analysis of packet streams
– Deep-packet inspection
– Connection reassembly
• Attackers gaming the monitoring system
– Splitting offending content across multiple packets
– Overloading the monitor
– Exploiting ambiguity in which packet the receiver uses
10
Discussion
11
Network Security
• Denial-of-service mitigation
– Pushback and source quenching
– Secure Overlay Services
• Spam mitigation
– Network-based detection of spam
• Protecting against bogus DNS and certificates
– Comparing results across multiple clients
– DNS-SEC
• End-to-end encryption: SSL and IPSEC
• Clean-slate approaches
– Self-certifying addresses
– “Off by default”
– Path verification
12
Traffic Measurement
Why is Traffic Measurement Important?
• Billing the customer
– Measure usage on links to/from customers
– Applying billing model to generate a bill
• Traffic engineering and capacity planning
– Measure the traffic matrix (i.e., offered load)
– Tune routing protocol or add new capacity
• Denial-of-service attack detection
– Identify anomalies in the traffic
– Configure routers to block the offending traffic
• Analyze application-level issues
– Evaluate benefits of deploying a Web caching proxy
– Quantify fraction of traffic that is P2P file sharing
Collecting Traffic Data: SNMP
• Simple Network Management Protocol
– Standard Management Information Base (MIB)
– Protocol for querying the MIBs
• Advantage: ubiquitous
– Supported on all networking equipment
– Multiple products for polling and analyzing data
• Disadvantages: dumb
– Coarse granularity of the measurement data
 E.g., number of byte/packet per interface per 5 minutes
– Cannot express complex queries on the data
– Unreliable delivery of the data using UDP
Collecting Traffic Data: Packet Monitoring
• Packet monitoring
– Passively collecting IP packets on a link
– Recording IP, TCP/UDP, or application-layer traces
• Advantages: details
– Fine-grain timing information
 E.g., can analyze the burstiness of the traffic
– Fine-grain packet contents
 Addresses, port numbers, TCP flags, URLs, etc.
• Disadvantages: overhead
– Hard to keep up with high-speed links
– Often requires a separate monitoring device
Collecting Traffic Data: Flow Statistics
• Flow monitoring (e.g., Cisco Netflow)
– Statistics about groups of related packets (e.g., same
IP/TCP headers and close in time)
– Recording header information, counts, and time
• Advantages: detail with less overhead
– Almost as good as packet monitoring, except no finegrain timing information or packet contents
– Often implemented directly on the interface card
• Disadvantages: trade-off detail and overhead
– Less detail than packet monitoring
– Less ubiquitous than SNMP statistics
Using the Traffic Data in Network Operations
• SNMP byte/packet counts: everywhere
– Tracking link utilizations and detecting anomalies
– Generating bills for traffic on customer links
– Inference of the offered load (i.e., traffic matrix)
• Packet monitoring: selected locations
– Analyzing the small time-scale behavior of traffic
– Troubleshooting specific problems on demand
• Flow monitoring: selective, e.g,. network edge
– Tracking the application mix
– Direct computation of the traffic matrix
– Input to denial-of-service attack detection
Flow Measurement
Flow Measurement: Outline
• Definition
– Passively collecting statistics about groups of packets
– Group packets based on headers and spacing in time
– Essentially a way to aggregate packet measurement data
• Scope
– Medium-grain information about user behavior
– Passively monitoring the link or the interface/router
– Helpful in characterizing, detecting, diagnosing, and fixing
• Outline
–
–
–
–
Definition of an IP “flow” (sequence of related packets)
Flow measurement data and its applications
Mechanics of collecting flow-level measurements
Reducing the overheads of flow-level measurement
IP Flows
flow 1
flow 2
flow 3
flow 4
• Set of packets that “belong together”
– Source/destination IP addresses and port numbers
– Same protocol, ToS bits, …
– Same input/output interfaces at a router (if known)
• Packets that are “close” together in time
– Maximum spacing between packets (e.g., 30 sec)
– Example: flows 2 and 4 are different flows due to time
Flow Abstraction
• A flow is not exactly the same as a “session”
– Sequence of related packets may be multiple flows
(due to the “close together in time” requirement)
– Sequence of related packets may not follow the same
links (due to changes in IP routing)
• Motivation for this abstraction
– As close to a “session” as possible from inside the
network
– Flow switching paradigm from IP-over-ATM technology
– Router optimization for forwarding/ACL decisions
(cache the result after the first packet in a flow)
– … might as well throw in a few counters
Recording Traffic Statistics (e.g., Netflow)
• Packet header information (same for every packet)
– Source and destination IP addresses
– Source and destination TCP/UDP port numbers
– Other IP/TCP/UDP header fields (protocol, ToS bits, etc.)
• Aggregate traffic information (summary of traffic)
– Start/finish time of the flow (time of first & last packet)
– Total number of bytes and number of packets in the flow
– TCP flags (e.g., logical OR over sequence of packets)
SYN
start
ACK
ACK
FIN
finish
4 packets
1436 bytes
SYN, ACK, & FIN
Recording Routing Info (e.g., Netflow)
• Input and output interfaces
– Input interface is where the packets entered the router
– Output interface is the “next hop” in the forwarding table
• Source and destination IP prefix (mask length)
– Longest prefix match on the src and dest IP addresses
• Source and destination ASnumbers
– Origin AS for src/dest prefix in the BGP routing table
forwarding table
Processor
Line card
Line card
Line card
BGP table
Line card
Switching
Fabric
Line card
Line card
Measuring Traffic as it Flows By
source
dest
input
output
source
prefix
source AS
dest
prefix
intermediate AS
dest AS
Source and destination: IP header
Source and dest prefix: forwarding table or BGP table
Source and destination AS: BGP table
Packet vs. Flow Measurement
• Basic statistics (available from both techniques)
– Traffic mix by IP addresses, port numbers, and protocol
– Average packet size
• Traffic over time
– Both: traffic volumes on a medium-to-large time scale
– Packet: burstiness of the traffic on a small time scale
• Statistics per TCP connection
– Both: number of packets & bytes transferred over the link
– Packet: frequency of lost or out-of-order packets, and the number of
application-level bytes delivered
• Per-packet info (available only from packet traces)
– TCP seq/ack #s, receiver window, per-packet flags, …
– Probability distribution of packet sizes
– Application-level header and body (full packet contents)
Collecting Flow Measurements
Route CPU that generates flow records
…may degrade forwarding performance
CPU
Router A
Line card that generates flow records
Router A
…more efficient to support
measurement in each line card
Packet monitor that generates flow records
Router A
Router B
Monitor
…third party
Router Collecting Flow Measurement
• Advantage
– No need for separate measurement device(s)
– Monitor traffic over all links in/out of router (parallelism)
– Ease of providing routing information for each flow
• Disadvantage
– Requirement for support in the router product(s)
– Danger of competing with other 1st-order router features
– Possible degradation of the throughput of the router
– Difficulty of online analysis/aggregation of data on router
• Practical application
– View from multiple vantage points (e.g., all edge links)
Packet Monitor Collecting Flow Records
• Advantages
– No performance impact on packet forwarding
– No dependence on support by router vendor
– Possibility of customizing the thinning of the data
• Disadvantages
– Overhead/cost of tapping a link & reconstructing packets
– Cost of buying, deploying, and managing extra equipment
– No access to routing info (input/output link, IP prefix, etc.)
• Practical application
– Selective monitoring of a small number of links
– Deployment in front of particular services or sites
• Packet monitor vendors support flow-level output
Mechanics: Flow Cache
• Maintain a cache of active flows
– Storage of byte/packet counts, timestamps, etc.
• Compute a key per incoming packet
– Concatenation of source, destination, port #s, etc.
• Index into the flow cache based on the key
– Creation or updating of an entry in the flow cache
key #bytes, #packets, start, finish
heade
r
packet
key
key #bytes, #packets, start, finish
Mechanics: Evicting Cache Entries
• Flow timeout
– Remove flows that have not received a packet recently
– Periodic sequencing through the cache to time out flows
– New packet triggers the creation of a new flow
• Cache replacement
– Remove flow(s) when the flow cache is full
– Evict existing flow(s) upon creating a new cache entry
– Apply eviction policy (LRU, random flow, etc.)
• Long-lived flows
– Remove flow(s) that persist for a long time (e.g., 30 min)
– … otherwise flow statistics don’t become available
– … and the byte and packet counters might overflow
Sampling: Packet Sampling
• Packet sampling before flow creation (Sampled Netflow)
– 1-out-of-m sampling of individual packets (e.g., m=100)
– Create of flow records over the sampled packets
• Reducing overhead
– Avoid per-packet overhead on (m-1)/m packets
– Avoid creating records for a large number of small flows
• Increasing overhead (in some cases)
– May split some long transfers into multiple flow records
– … due to larger time gaps between successive packets
time
not sampled
timeout
two flows
Conclusions
• Flow measurement
– Medium-grain view of traffic on one or more links
• Advantages
– Lower measurement volume than full packet traces
– Available on high-end line cards (Cisco Netflow)
– Control over overhead via aggregation and sampling
• Disadvantages
– Computation and memory requirements for flow cache
– Loss of fine-grain timing and per-packet information
– Not uniformly supported by router vendors