The Open Source Bro IDS
Overview and Recent Developments
Robin Sommer
International Computer Science Institute, &
Lawrence Berkeley National Laboratory
robin@icsi.berkeley.edu
http://www.icir.org/robin
2011 CACR Higher Education Cybersecurity Summit
Indiana University
Outline
CACR Higher Education Cybersecurity Summit
2
Outline
Philosophy and Architecture
A framework for network traffic analysis.
CACR Higher Education Cybersecurity Summit
2
Outline
Philosophy and Architecture
A framework for network traffic analysis.
Usage and Deployment
Logs, scripts, communication, cluster.
CACR Higher Education Cybersecurity Summit
2
Outline
Philosophy and Architecture
A framework for network traffic analysis.
Usage and Deployment
Logs, scripts, communication, cluster.
Roadmap
The Bro world is moving quite rapidly.
CACR Higher Education Cybersecurity Summit
2
What is Bro?
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
“Domain-specific Python”
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
CACR Higher Education Cybersecurity Summit
3
What is Bro?
Packet Capture
Sum is m
ore
Traffic Inspection
than the
pieces
Attack Detection
“Domain-specific Python”
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
CACR Higher Education Cybersecurity Summit
3
Philosophy
CACR Higher Education Cybersecurity Summit
4
Philosophy
Real-time network analysis framework
Primarily an IDS, but many use it for general traffic analysis.
CACR Higher Education Cybersecurity Summit
4
Philosophy
Real-time network analysis framework
Primarily an IDS, but many use it for general traffic analysis.
Highly stateful
Tracks extensive application-layer network state.
CACR Higher Education Cybersecurity Summit
4
Philosophy
Real-time network analysis framework
Primarily an IDS, but many use it for general traffic analysis.
Highly stateful
Tracks extensive application-layer network state.
Policy-neutral at the core
Can accommodate a range of detection approaches.
CACR Higher Education Cybersecurity Summit
4
Philosophy
Real-time network analysis framework
Primarily an IDS, but many use it for general traffic analysis.
Highly stateful
Tracks extensive application-layer network state.
Policy-neutral at the core
Can accommodate a range of detection approaches.
Supports forensics
Extensively logs what it sees.
CACR Higher Education Cybersecurity Summit
4
Target Audience
CACR Higher Education Cybersecurity Summit
5
Target Audience
Scientific environments
Effective with liberal security policies
CACR Higher Education Cybersecurity Summit
5
Target Audience
Scientific environments
Effective with liberal security policies
Network-savvy users
Requires understanding of your network
CACR Higher Education Cybersecurity Summit
5
Target Audience
Scientific environments
Effective with liberal security policies
Network-savvy users
Requires understanding of your network
Unixy mindset
Command-line based, fully customizable
CACR Higher Education Cybersecurity Summit
5
Deployment
Internal
Network
Internet
CACR Higher Education Cybersecurity Summit
6
Deployment
Tap
Internet
Internal
Network
Bro
CACR Higher Education Cybersecurity Summit
6
Deployment
Tap
Internet
Internal
Network
Bro
Commodity platforms
Standard PC & NICs, FreeBSD/Linux
CACR Higher Education Cybersecurity Summit
6
Activity Logs: TCP
CACR Higher Education Cybersecurity Summit
7
Activity Logs: TCP
One-line summaries of all connections
Most basic, yet often extremely helpful output.
CACR Higher Education Cybersecurity Summit
7
Activity Logs: TCP
One-line summaries of all connections
Most basic, yet often extremely helpful output.
> bro -i en0 tcp
[ ... wait ...]
> cat conn.log
CACR Higher Education Cybersecurity Summit
7
Activity Logs: TCP
One-line summaries of all connections
Most basic, yet often extremely helpful output.
> bro -i en0 tcp
[ ... wait ...]
> cat conn.log
Time
Duration
Source
Destination
1144876596.658302 1.206521 192.150.186.169 62.26.220.2 \
http 53052
80
tcp
874
1841
SF
X
Service SPort
DPort
Proto
SrcBytes
DstBytes
CACR Higher Education Cybersecurity Summit
TCPState Local?
7
Activity Logs: TCP
One-line summaries of all connections
Most basic, yet often extremely helpful output.
> bro -i en0 tcp
[ ... wait ...]
> cat conn.log
Time
Duration
Source
Destination
1144876596.658302 1.206521 192.150.186.169 62.26.220.2 \
http 53052
80
tcp
874
1841
SF
X
Service SPort
DPort
Proto
SrcBytes
DstBytes
TCPState Local?
LBNL has connection logs for every connection attempt since June 94!
CACR Higher Education Cybersecurity Summit
7
Activity Logs: HTTP
> cat http.log
[...]
1144876588.30 %2
1144876588.30 %2
1144876588.30 %2
1144876588.30 %2
1144876588.30 %2
1144876588.30 %2
[...]
1144876588.77 %2
1144876588.77 %2
1144876588.77 %2
[...]
1144876588.77 %2
1144876588.78 %2
1144876588.78 %2
1144876588.94 %2
[...]
start 192.150.186.169:53041 > 195.71.11.67:80
GET /index.html (200 "OK" [57634] www.spiegel.de)
> HOST: www.spiegel.de
> USER-AGENT: Mozilla/5.0 (Macintosh; PPC Mac OS ...
> ACCEPT: text/xml,application/xml,application/xhtml ...
> ACCEPT-LANGUAGE: en-us,en;q=0.7,de;q=0.3
< SERVER: Apache/1.3.26 (Unix) mod_fastcgi/2.2.12
< CACHE-CONTROL: max-age=120
< EXPIRES: Wed, 12 Apr 2006 21:18:28 GMT
<=
<=
<=
<=
1500
1500
1500
1500
bytes:
bytes:
bytes:
bytes:
"<!-- Vignette StoryServer 5.0 Wed Apr..."
"r "http://spiegel.ivwbox.de" r..."
"icon.ico" type="image/ico">^M^J ..."
"erver 5.0 Mon Mar 27 15:56:55 ..."
CACR Higher Education Cybersecurity Summit
8
Activity Logs: HTTP
> cat http.log
[...]
1144876588.30 %2 start 192.150.186.169:53041 > 195.71.11.67:80
1144876588.30 %2 GET /index.html (200 "OK" [57634] www.spiegel.de)
1144876588.30 %2 > HOST: www.spiegel.de
1144876588.30 %2 > USER-AGENT: Mozilla/5.0 (Macintosh; PPC Mac OS ...
1144876588.30 %2 > ACCEPT: text/xml,application/xml,application/xhtml ...
1144876588.30 %2 > ACCEPT-LANGUAGE: en-us,en;q=0.7,de;q=0.3
[...]
1144876588.77 %2 < SERVER: Apache/1.3.26 (Unix) mod_fastcgi/2.2.12
1144876588.77 %2 < CACHE-CONTROL: max-age=120
1144876588.77 %2 < EXPIRES: Wed, 12 Apr 2006 21:18:28 GMT
[...]
1144876588.77 %2 <= 1500 bytes: "<!-- Vignette StoryServer 5.0 Wed Apr..."
1144876588.78 %2 <= 1500 bytes: "r "http://spiegel.ivwbox.de" r..."
1144876588.78 %2 <= 1500 bytes: "icon.ico" type="image/ico">^M^J ..."
1144876588.94 %2 <= 1500 bytes: "erver 5.0 Mon Mar 27 15:56:55 ..."
[...]
Supported Protocols
ARP, UDP, TCP, ICMP
BitTorrent, DNS, FTP, Gnutella, HTTP, IRC, NFS, POP3, RPC, SMB, SMTP, SSH, SSL
... and more
CACR Higher Education Cybersecurity Summit
8
Architecture
Packets
Network
CACR Higher Education Cybersecurity Summit
9
Architecture
Events
Protocol Decoding
Event Engine
Packets
Network
CACR Higher Education Cybersecurity Summit
9
Architecture
Logs
Analysis Logic
Notification
Policy Script Interpreter
Events
Protocol Decoding
Event Engine
Packets
Network
CACR Higher Education Cybersecurity Summit
9
Architecture
Logs
Notification
“User Interface”
Analysis Logic
Policy Script Interpreter
Events
Protocol Decoding
Event Engine
Packets
Network
CACR Higher Education Cybersecurity Summit
9
Event Model
Web
Client
1.2.3.4/4321
Request for /index.html
Status OK plus data
Web
Server
5.6.7.8/80
CACR Higher Education Cybersecurity Summit
10
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
ACK
...
Request for /index.html
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
CACR Higher Education Cybersecurity Summit
10
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Request for /index.html
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
connection_established(1.2.3.4/4321 5.6.7.8/80)
CACR Higher Education Cybersecurity Summit
10
Event Model
Request for /index.html
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
connection_established(1.2.3.4/4321 5.6.7.8/80)
TCP stream reassembly for originator
Event
http_request(1.2.3.4/4321 5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
10
Event Model
Request for /index.html
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
connection_established(1.2.3.4/4321 5.6.7.8/80)
TCP stream reassembly for originator
Event
http_request(1.2.3.4/4321 5.6.7.8/80, “GET”, “/index.html”)
TCP stream reassembly for responder
Event
http_reply(1.2.3.4/4321 5.6.7.8/80, 200, “OK”, data)
CACR Higher Education Cybersecurity Summit
10
Event Model
Request for /index.html
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
connection_established(1.2.3.4/4321 5.6.7.8/80)
TCP stream reassembly for originator
Event
http_request(1.2.3.4/4321 5.6.7.8/80, “GET”, “/index.html”)
TCP stream reassembly for responder
Event
http_reply(1.2.3.4/4321 5.6.7.8/80, 200, “OK”, data)
Event
connection_finished(1.2.3.4/4321, 5.6.7.8/80)
CACR Higher Education Cybersecurity Summit
10
Script Example: Matching URLs
Task: Report all Web requests for files called “passwd”.
Example: http_request(1.2.3.4/4321
5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
11
Script Example: Matching URLs
Task: Report all Web requests for files called “passwd”.
event http_request(c: connection, method: string, path: string)
{
if ( method == “GET” && path == /.*passwd/ )
NOTICE(SensitiveURL, c, path); # Alarm.
}
(Syntax simplified.)
Example: http_request(1.2.3.4/4321
5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
11
Script Example: Matching URLs
Task: Task:
Report
Report
all successful
all Web Web
requests
requests
for files
for called
files called
“passwd”.
“passwd”.
event http_request(c: connection, method: string, path: string)
{
if ( method == “GET” && path == /.*passwd/ )
NOTICE(SensitiveURL, c, path); # Alarm.
}
(Syntax simplified.)
Example: http_request(1.2.3.4/4321
5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
11
Script Example: Matching URLs
Task: Task:
Report
Report
all successful
all Web Web
requests
requests
for files
for called
files called
“passwd”.
“passwd”.
global potentially_sensitive: table[connection] of string;
event http_request(c: connection, method: string, path: string)
{
if ( method == “GET” && path == /.*passwd/ )
NOTICE(SensitiveURL, c, path);
potentially_sensitive[c]
= path;
# #
Alarm.
Add to table.
}
(Syntax simplified.)
Example: http_request(1.2.3.4/4321
5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
11
Script Example: Matching URLs
Task: Task:
Report
Report
all successful
all Web Web
requests
requests
for files
for called
files called
“passwd”.
“passwd”.
global potentially_sensitive: table[connection] of string;
event http_request(c: connection, method: string, path: string)
{
if ( method == “GET” && path == /.*passwd/ )
NOTICE(SensitiveURL, c, path);
potentially_sensitive[c]
= path;
# #
Alarm.
Add to table.
}
event http_reply(c: connection, response: int, reason: string)
{
if ( response == OK && c in potentially_sensitive )
NOTICE(SensitiveURL, c, potentially_sensitive[c]);
}
(Syntax simplified.)
Example: http_request(1.2.3.4/4321
5.6.7.8/80, “GET”, “/index.html”)
CACR Higher Education Cybersecurity Summit
11
Script Example: Scan Detector
Task: Count failed connection attempts per source address.
CACR Higher Education Cybersecurity Summit
12
Script Example: Scan Detector
Task: Count failed connection attempts per source address.
global attempts: table[addr] of int &default=0;
event connection_rejected(c: connection)
{
local source = c$orig_h; !
# Get source address.
local n = ++attempts[source];! # Increase counter.
if ( n == SOME_THRESHOLD )
! # Check for threshold.
NOTICE(Scanner, source); ! # If so, report.
}
(Syntax simplified.)
CACR Higher Education Cybersecurity Summit
12
Script Example: Tracking Services
Task: Report hosts accepting an SSH connection for the first time
CACR Higher Education Cybersecurity Summit
13
Script Example: Tracking Services
Task: Report hosts accepting an SSH connection for the first time
global ssh_hosts: set[addr];
event connection_established(c: connection)
{
local responder = c$id$resp_h; # Responder’s address
local service = c$id$resp_p;
# Responder’s port
if ( service != 22/tcp )
return; # Not SSH.
if ( responder in ssh_hosts )
return; # We already know this one.
NOTICE(SSHHostFound, responder); # Alarm
add ssh_hosts[responder]; # Found a new host
}
(Syntax simplified.)
CACR Higher Education Cybersecurity Summit
13
Distributed Scripts
CACR Higher Education Cybersecurity Summit
14
Distributed Scripts
Bro comes with >20,000 lines of script
Prewritten functionality that can just be loaded
CACR Higher Education Cybersecurity Summit
14
Distributed Scripts
Bro comes with >20,000 lines of script
Prewritten functionality that can just be loaded
Scripts generate logs and alarms
Amendable to customization and extension
CACR Higher Education Cybersecurity Summit
14
Communication Architecture
CACR Higher Education Cybersecurity Summit
15
Communication Architecture
Bro A
Logs
Notifications
Policy Script
Interpreter
Events
Event Engine
Packets
Network
CACR Higher Education Cybersecurity Summit
15
Communication Architecture
Bro A
Logs
Notifications
Policy Script
Interpreter
Bro B
Logs
Notification
Policy Script
Interpreter
Events
Events
Event Engine
Event Engine
Packets
Packets
Network
Network
CACR Higher Education Cybersecurity Summit
15
Communication Architecture
Bro A
Logs
Bro B
Logs
Notifications
Policy Script
Interpreter
State Operations
Notification
Policy Script
Interpreter
Events
Events
Event Engine
Event Engine
Packets
Packets
Network
Network
CACR Higher Education Cybersecurity Summit
15
Communication Architecture
Bro A
Logs
Bro B
Logs
Notifications
Policy Script
Interpreter
Events
State Operations
Event Stream
Event Engine
Notification
Policy Script
Interpreter
Events
Event Engine
Packets
Packets
Network
Network
CACR Higher Education Cybersecurity Summit
15
Communication Architecture
Bro A
Logs
Notifications
Policy Script
Interpreter
Events
State Operations
Event Stream
Event Engine
Packets
Network
Broccoli
Logs
Bro B
External
Applications
Notification
Policy Script
Interpreter
Events
Event Engine
Packets
Network
CACR Higher Education Cybersecurity Summit
15
Load-Balancer Approach
Bro
10Gbps
Event Engine
Packet Analysis
Script
Interpreter
Detection Logic
CACR Higher Education Cybersecurity Summit
16
Load-Balancer Approach
Flow
Packets
Bro 1
10Gbps
External Packet Demultiplexer
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
Bro 2
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
Bro 3
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
CACR Higher Education Cybersecurity Summit
16
Load-Balancer Approach
Flow
Packets
Bro 1
10Gbps
External Packet Demultiplexer
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
Communication
Bro 2
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
Communication
Bro 3
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
CACR Higher Education Cybersecurity Summit
16
Load-Balancer Approach
Flow
Packets
Bro 1
External Packet Demultiplexer
Event Engine
10Gbps
“Bro Cluster
”
Protocol Decoding
Script
Interpreter
Analysis Logic
Communication
Bro 2
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
Communication
Bro 3
Event Engine
Protocol Decoding
Script
Interpreter
Analysis Logic
CACR Higher Education Cybersecurity Summit
16
Cluster Challenges
CACR Higher Education Cybersecurity Summit
17
Cluster Challenges
Communication capability required
Bro has communication primitives built-in
CACR Higher Education Cybersecurity Summit
17
Cluster Challenges
Communication capability required
Bro has communication primitives built-in
Management of multi-machine setup is tedious
Management interface transparently hides complexity (BroControl)
CACR Higher Education Cybersecurity Summit
17
Cluster Challenges
Communication capability required
Bro has communication primitives built-in
Management of multi-machine setup is tedious
Management interface transparently hides complexity (BroControl)
Demultiplexer needs to operate at line-rate
A number of solutions, including a commercially available appliance
CACR Higher Education Cybersecurity Summit
17
UC Berkeley Research Cluster
CACR Higher Education Cybersecurity Summit
18
UC Berkeley Research Cluster
CACR Higher Education Cybersecurity Summit
18
UC Berkeley Research Cluster
CACR Higher Education Cybersecurity Summit
18
Roadmap
CACR Higher Education Cybersecurity Summit
19
Research Heritage
CACR Higher Education Cybersecurity Summit
20
Research Heritage
Bro development has focused on research
We were lacking resources for documentation, polishing, functionality
CACR Higher Education Cybersecurity Summit
20
Research Heritage
Bro development has focused on research
We were lacking resources for documentation, polishing, functionality
NSF is now funding Bro development
Full-time engineers working on user experience and performance
Office of Cyberinfrastructure
CACR Higher Education Cybersecurity Summit
20
Target: Blue Waters @ NCSA
CACR Higher Education Cybersecurity Summit
21
Target: Blue Waters @ NCSA
10 PF/s peak performance
s
n
io
at
lic
p
ap
n
o
d
e
n
ai
st
su
/s
F
>1 P
>300,000 cores
>1 Petabyte memory
>10 Petabyte disk storage
>0.5 Exabyte archival storage
t facility
o
o
-f
e
ar
u
q
-s
0
0
,0
8
8
in
d
e
st
o
H
CACR Higher Education Cybersecurity Summit
21
Next Release
CACR Higher Education Cybersecurity Summit
22
Next Release
Bro 1.6
•
•
•
•
•
•
•
•
•
•
New build, installation, and test framework.
New quick-start guide.
Overhauled default policy scripts.
Comprehensive script documentation.
New interface for extending scripts
New logging framework.
Code cleanup.
Lots of little things fixed.
Git repositories.
New web server.
CACR Higher Education Cybersecurity Summit
22
Roadmap
CACR Higher Education Cybersecurity Summit
23
Roadmap
Medium-term
•
•
•
•
•
•
•
•
•
Comprehensive user and reference manuals.
High-performance binary logging.
Integration of external “intelligence sources”.
Improved IPv6 support.
New protocols.
Database interface.
Community script repository.
More code cleanup.
More things fixed.
CACR Higher Education Cybersecurity Summit
23
Roadmap
Medium-term
•
•
•
•
•
•
•
•
•
Comprehensive user and reference manuals.
High-performance binary logging.
Integration of external “intelligence sources”.
Improved IPv6 support.
New protocols.
Database interface.
Community script repository.
More code cleanup.
More things fixed.
Long-Term
•
•
•
Script compilation.
Multi-threading.
GUI.
CACR Higher Education Cybersecurity Summit
23
Community
CACR Higher Education Cybersecurity Summit
24
Community
We aim to involve the community more
Feedback and contributions welcome!
CACR Higher Education Cybersecurity Summit
24
Community
We aim to involve the community more
Feedback and contributions welcome!
We can help you getting started
Get in touch and ask us!
CACR Higher Education Cybersecurity Summit
24
Community
We aim to involve the community more
Feedback and contributions welcome!
We can help you getting started
Get in touch and ask us!
Let us know if you’re using Bro
Operationally, experimentally, research?
CACR Higher Education Cybersecurity Summit
24
Summary
CACR Higher Education Cybersecurity Summit
25
Summary
Philosophy and Architecture
“A Python for Network Traffic Analysis”
CACR Higher Education Cybersecurity Summit
25
Summary
Philosophy and Architecture
“A Python for Network Traffic Analysis”
Usage and Deployment
Activity logs, scripting examples, Bro Cluster
CACR Higher Education Cybersecurity Summit
25
Summary
Philosophy and Architecture
“A Python for Network Traffic Analysis”
Usage and Deployment
Activity logs, scripting examples, Bro Cluster
Roadmap
Milestones and community
CACR Higher Education Cybersecurity Summit
25
Get in Touch
Homepage
www.bro-ids.org
info@bro-ids.org
EMail
mailman.icsi.berkeley.edu/
mailman/listinfo/bro
git.bro-ids.org
Development
mailman.icsi.berkeley.edu/
mailman/listinfo/bro-dev
CACR Higher Education Cybersecurity Summit
26
Get in Touch
Homepage
www.bro-ids.org
info@bro-ids.org
EMail
mailman.icsi.berkeley.edu/
mailman/listinfo/bro
git.bro-ids.org
Development
mailman.icsi.berkeley.edu/
mailman/listinfo/bro-dev
Thanks for your attention!
CACR Higher Education Cybersecurity Summit
26