The challenges of using an
intrusion detection system:
is it worth the effort?
Rodrigo Werlinger, Kirstie Hawkey, Kasia Muldner,
Pooya Jaferian, Konstantin Beznosov
University of British Columbia, Canada
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Werlinger, R., Muldner, K., Jaferian, P., Hawkey, K., Beznosov, K.
2
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Motivation
Literature
 “This task was based upon
the monitoring and
analysis phase of ID, the
most time-consuming and
cognitively challenging
subtask in ID [9, 10, 23]”.

3
“Command Line or Pretty
Lines? Comparing Textual and
Visual Interfaces for Intrusion
Detection” Thompson et al.,
CHI 2007
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Motivation
Literature:
Pre-processing phase of ID is
relatively easy
“This task was based upon
the monitoring and analysis
phase of ID, the most timeconsuming and cognitively
challenging subtask in ID [9,
10, 23]”.
Command Line or Pretty Lines?
Comparing Textual and Visual
Interfaces for Intrusion Detection”
Thompson et al., CHI 2007
4
Our Perception:
IDS configuration is *hard*
 Rodrigo’s current experience
deploying an IDS
 His prior experiences in a
telecommunications co.
 Collective recollections of 1+
interview participants
describing IDS configuration
as a major hurdle
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Intrusion Detection Systems (IDSs)
 Intrusion detection phases: deployment,
monitoring, analysis, response
 Still need human intervention
 ID requires high level of security expertise,
organizational knowledge & collaboration
 Most current research focuses on
supporting monitoring + analysis phases
(e.g., visualization, better detection algs)
5
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Research questions
 What do security practitioners expect from
an IDS?
 What are the difficulties they face when
installing and configuring an IDS?
 How can the usability of an IDS be
improved?
6
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Approach
 Semi-structured interviews
• 9/34 discussed IDS
• 6 Academic, 1 Financial Services, 1 Scientific
Services, 1 Consultant
• 1 Security Manager, 1 IT Manager, 5 security,
2 general IT w/ security duties
 Participatory observation
• ~15 hours on IDS (~90 total)
• Working with 2 senior Academic SPs
7
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Results from Interviews
[an IDS is] “one of the most controversial
[tools] – some really love it, but some
really hate it”
8
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
IDS Expectations: Advantages
 Problem identification
 Activities inside/outside firewall
 Reduction of uncertainty
 Could provide assurance of effectiveness of
security measures
 Monitoring with privacy
 Decreased time pressure for maintenance
 If using an Intrusion Prevention system
9
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
IDS Expectations: Disadvantages
 Financial expense
 Work and time required
• Tuning the system
 Unreliability
• Buggy, dropped packets
 Lack of clear utility
• Hard to see an improvement, often sit idle
10
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Results from Participatory Observation
 History
• IDS installed 2 years prior in one network
domain
• Crashed, memory space issues
• Unclear whether problem was with setup or newly
added wireless
• No time to confirm exact cause
• Decided to re-install from scratch on a
different network
• This delayed for several months
• High workload, competing priorities
11
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Issues deploying an IDS (1/5)
Deciding on the purpose of the IDS
1. Improve efficiency of monitoring
2. But also:
•
Statistics on network security
•
Support for increasing security budget
 Ultimately, (2) proved too complicated…
12
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Issues deploying an IDS (2/5)
Integrating the IDS in the network
 To connect the IDS, 2 ports were needed
 Wanted to use port-mirroring feature to
select traffic wanted to monitor
These requirements could not be realized
IDS installed in a less critical network
13
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Issues deploying an IDS (3/5)
Configuration via IDS GUI
 Quick tune option
 But inadequate for complex task:
• Can’t specify hard disk partitions
• No support for configuring IDS security
settings (server firewall rules)
14
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Issues deploying an IDS (4/5)
Distributed Environment
 Extra overhead
• Involvement of various organizational members
without security as a priority
 Multiple stakeholders need to configure IDS
• But IDS did not support fine-grained access
control
 Compromise: less critical network, but
autonomy
15
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Issues deploying an IDS (5/5)
Usability / Utility Tradeoffs
 Ideally IDS would have been deployed in
critical network (utility high, usability low)
 Hard to assess IDS utility without full
deployment
• Unclear if large network domain more demanding
 False positives vs. false negatives tradeoff
• Can’t tune until running
16
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Challenges throughout IDS deployment
Considerations
Before Deploying
• Show
economic
benefit to get buy-in
• Minimize overhead
costs (stakeholders)
• Broad knowledge of
organization &
systems
17
Configuration &
Validation
Ongoing Use
• Distributed
• Collaboration
environment
• Initial configuration
hurdle
• Determine
appropriate test bed
features
• “A bit of smarts”
• Reports for different
stakeholders
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Planning
 IDSs not yet de facto tools
IDS utility must be clear, but
until deployed and configured…..
 IDS deployment impacts many
stakeholders
Formalize via dedicated project
Involve stakeholders
18
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Configuration and validation
 Configuration hurdle (rule customization)
Quick tuning
 Distributed environment
Flexible reporting
 How to test the IDS (“all or nothing” tool)
Support for finding testbed
19
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
On-going usage
 Detection of trends
• “A bit of smarts”
Artificial intelligence
 IDS usage via various stakeholders
Collaboration features
Flexible reports
20
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Summary
 Many factors will determine whether
deploying an IDS is worth the effort
 Challenges are present in all stages and not
limited to technology
 Tool support needed to help meet the
challenges
 More study needed to determine
generalizability of our participants’
experiences
21
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Thank you
hawkey@ece.ubc.ca
22
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Challenges and recommendations
 Why an IDS?

Broad
and
deep
 Perceptions of IDSs
knowledge
 Intensive
 Planning and installation
collaboration
 Representative
Testebed
23
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Technical and organizational challenges
 Broad and deep knowledge
 Intensive collaboration
 Representative Testbed
 Meaningful reports
24
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
 Original slides that came right after the
results
25
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Stages to deploy an IDS
Planning
• Show economic
benefit
• Minimize costs
• Detection efficient
26
Configuration &
Validation
• Distributed
environment
• Initial configuration
hurdle
• Determine
appropriate testbed
Ongoing Use
• Collaboration
features
•“A bit of smarts”
• Reports for different
stakeholders
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Planning
IDS not only to detect attacks
 Management buy-in
 Compare different points in the network
Dedicated project
Show economic
benefit
• Involve other stakeholders
• Competing priorities
Minimize costs,
Detection efficient
27
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
Configuration and validation
Configuration hurdle
 Customization of the rules
Quick tuning
Distributed environment:
 How to distribute alarms
Flexible criteria
How to test the IDS
 “All or nothing” tool
Find test-bed
28
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
On-going usage
Detection of trends
 “A bit of smarts”
Artificial intelligence
Collaboration features
 Incorporate changes in the systems
Collaboration features
Better reports
 Meaningful reports
29
Flexible reports
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)
30
Laboratory for Education and Research in Secure Systems Engineering
(lersse.ece.ubc.ca)