Defeating Large Scale Attacks:
Technology and Strategies for
Global Network Monitoring
The NetViewer Experiment
PAVG in collaboration with
Networking Systems
R. Kamath, E. Jang, D. Luckham
Project Goals
Detect system misuse on a global
level
 User re-configurable and flexible
 Hierarchical organization of monitors
 Correlation of distributed monitors
 Monitor activity from diverse sources
 Monitor at multiple levels of
abstraction

2
Stanford NetViewer Experiment

Uses Stanford Rapide Toolset
 Uses Complex Event Processing
technology
 Uses Talarian’s SmartSocketsTM
middleware for distributed processing
FOR MORE INFO...
Http://pavg.stanford.edu/rapide
Http://pavg.stanford.edu/cep
3
NetViewer Experiment setup
NetFlow FlowCollector
Intrusion
Monitor
Map
CEP
Logger
Flow
Efficiency
Monitor
Filter
Map
PassThrough
Monitor
Log Files
Cisco
NetFlow
FlowCollector
Complex Event
Processing
Monitor Views
4
SUNet Campus Network
Internet
Core
Gateway
Undergrad
Education
Redundancy
Gateway
Grad.
Education
Admin
Host 1
Business
School
Redundancy
Gateway
Stanford
Hospital
Computer
Center 1
Redundancy
Gateway
Admin
Host 2
To
FlowCollector
Redundancy
Gateway
Computer
Center 2
Core
Gateway
Internet
5
Complex Event Processing

Accept network ‘events’ from any source
– CISCO NetFlow FlowCollector, tcpdump

Correlates events based on content and
temporal relationship between events
 Event Processing Agents (EPAs)
connected in an Event Processing Network
(EPNs)
 Both post-mortem and real-time
processing
6
Event Processing Agents (EPAs)
-- Loggers and Filters

Loggers
– Convert external data into events
– E.g. CISCO FlowCollector logs to events

Filters
– Select a subset of events based on pattern
– E.g. Only connections from Stanford hosts
7
EPAs-- Maps and Viewers

Maps
– Search for patterns in input events
– Generate appropriate output events
– E.g. look for IP scans and generate alarms

Viewers
– Graphical display of data in events
– Tables, Bar Graphs
8
RapNet User interface

RapNet
–
–
–
–
–
Graphical Interface to NetViewer tool
Easy access to EPA and EPN library
Easy re-configuration of EPAs
Easy modification of EPNs
Construct new EPNs using EPAs
9
NetViewer running under RapNet
10
Hierarchical monitoring

Two types of hierarchy
– Abstraction hierarchy
• NetViewer monitors data at different abstraction
levels
– Topological hierarchy
• NetViewers at different locations

NetViewers at different levels
communicate using SmartSockets
middleware
 General case: arbitrary network of
monitors
11
Network Abstraction Hierarchy

Application layer
– Host-based monitoring
– Data exchanged by SMTP, TELNET, FTP, HTTP
protocols

Transport layer
– Data exchanged by TCP/IP suite of protocols

Network layer
– Router-based monitoring
– IP and UDP packets
12
Topological Hierarchy -- multiple
gateways example

Distributed processing of data
 Each NetViewer at level 1 monitors data
from a different gateway
 Results (e.g. top 10 IPs) from level 1
NetViewers sent to level2 NetViewers
 Level 2 NetViewers correlate the results of
level 1 NetViewers
– E.g. compute top 10 IPs over all gateways
13
Distributed monitoring on SUNet
Receiver running
NetViewer 3
Sender running
NetViewer 1
Core gateway
Admin host
Admin host
SmartSockets over SUNet
Sender running
NetViewer 2
Admin host
Press gateway
14
Current Status -- EPAs

Library of Event Processing Agents (EPAs)
– Traffic categories
• Web, Mail, DNS, ftp …
– Scan Detectors
• IP scan, Port scan
– Policy violation detectors
• Access to restricted hosts
• Access to restricted ports on hosts
– Traffic event filters
• Web, Mail, Hosts, Networks
15
Current Status -- EPNs

Library of Viewers
– Tables
– Bar graphs
– Pie charts

Library of Event Processing Networks
(EPNs)
– Network of EPAs
– Graphical viewers to display results
16
Research Directions

Hierarchical monitoring
– Data sources from different layers
– Correlation of results from multiple NetViewers

Accept more input formats
 Distributed processing
– Assign individual EPAs within a NetViewer to
run on different machines

Expand EPA library
– Work on mail spam detection
17
Experiment results on SUNet

NetViewer used to process router logs
– Real-time performance of about 1000 log
records/sec

Generated traffic statistics
– Top IPs by packets or bytes
– Classification of traffic into categories such as
internal/external, web/mail/DNS etc.

Intrusion detection
– Detected IP and port scans
– Well-known attack signature e.g. finger attack
18
Related projects -- CIDF

Correlates information from multiple
intrusion detectors
– Reduces false alarms
– Prioritizes network warnings

Part of the DARPA Common Intrusion
Detection Framework (CIDF)
– Multiple intrusion detectors in cyber battlefield
FOR MORE INFO...
Http://seclab.cs.ucdavis.edu/cidf
19
Overview of the CIDF project

Goal
Experiment with semantic interoperability of different
components in CIDF

Groups Involved
Group A: produces GIDOs, questions, detailed English
description of the events, and the answers to the
questions.
Group B: gets 10 scenarios and produces 10 GIDOs
describing the scenarios.
Group C: gets the questions and high level scenarios from
B and builds the code. Then, gets 10 GIDOs and
produces text answers to the questions - Stanford
belongs to group C.
20
Processing GIDOs with CEP agents
Question
Target ID
Description
Search Pattern
Input
GIDO




CIDFLogger
CMEvent
Builds events
Points to C++ GIDO tree
Question Agent
Process the C++ GIDO
tree with Question
Answer
to user
Make each GIDO an event
Use (and fix) our existing cidfLogger
Separate event processing agent called “Qagent”
Provides flexible way of handling GIDOs
21
Qagent



Finds an answer from a given GIDO and a query
pattern.
Qagent traverses the tree to find all the possible
paths that can lead to the answer.
The question is fed to the program as a text file
with two sections:
– The input file may contain a text description
– Patterns to be searched from the tree.
The pattern lines are preceded with “@question:”

Implemented in C++ (I.e. not map language)
– Easier tree traversal
– File input
22
Pattern Language

Lists of SID separated by comma. Answer is the
subtree after the last SID
Attack,AttackSpecifics,IPV4Address

“#true” or “#false” to get the sibling SID rather
than child SID of the last SID for the answer.
ByMeansOf,Attack#true

‘^’ to indicate that the SID is one of the base SID
that applies to all other parts of the pattern
^And,^Copy,Outcome,ReturnCode?success=FileSource,File
Name
23
Examples
Event1
Brief description:
This is an attack that began on
Monday, May 24, at 12:44.
What is the certainty of this
attack?
@question:
Attack,Certainty
( Attack
( Initiator
( IPV4Address 134.52.160.76 )
)
( Target
( IPV4Address 134.52.160.114 )
)
( AttackSpecifics
( Certainty 100 )
( Severity 50 )
( AttackID 000000020000000f )
)
( When
( BeginTime Mon May 24 12:44:17
1999 PDT )
( EndTime Mon May 24 12:44:18 1999
PDT )
)
)
24
Team Members





Rajesh Kamath (rkamath@pavg)
David Luckham (dcl@pavg)
Eunhei Jang (ejang@pavg)
John Kenney (jjk@pavg)
James Vera (vera@pavg)
25