Computer Security:
Principles and Practice
Chapter 6 – Intrusion Detection
by William Stallings and Lawrie Brown
Lecture slides: Some by Lawrie Brown,
some by Susan Lincke
Objectives
The student should be able to:
 Define how a signature-based, anomaly-based, and rule-based IDS works.
 Define false positives, false negatives, and how both affect the sensitivity of an
IDS.
 Describe the difference between an IDS and IPS and the
advantages/disadvantages of each.
 Describe when you would use a host IDS/IPS and/or a network IDS/IPS and
some advantages of each.
 Describe functions of different Host IDS systems: System integrity verifiers,
statistics monitors, deception systems, and configuration auditors.
 Draw the internal configuration of a tap, and describe how a switch SPAN port
works.
 Describe the three responses that CISCO IDS’s can support in response to an
attack.
 Describe the capabilities of Snort, including its features, cost, programmability,
configurability, and its directory structure.
Examples of Intrusion









remote root compromise
web server defacement
guessing / cracking passwords
copying / viewing sensitive data / databases
running a packet sniffer
distributing pirated software
using an unsecured modem to access net
impersonating a user to reset password
using an unattended workstation
Security Intrusion & Detection
Security Intrusion
A security event, or combination of multiple security events, that
constitutes a security incident in which an intruder gains, or
attempts to gain, access to a system (or system resource)
without having authorization to do so.
Intrusion Detection
A security service that monitors and analyzes system events for the
purpose of finding, and providing real-time or near real-time
warning of attempts to access system resources in an
unauthorized manner.
Intrusion Prevention
An intrusion detection system that proactively acts to counteract a
threat directly.
Insider Attacks

among most difficult to detect and prevent
 employees have access & systems knowledge
 may be motivated by revenge / entitlement



when employment terminated
taking customer data when move to competitor
IDS / IPS may help but also need:

least privilege, monitor logs, strong authentication,
termination process to block access & mirror data
Insider Behavior Example
1.
2.
3.
4.
5.
6.
7.
create network accounts for themselves and
their friends
access accounts and applications they wouldn't
normally use for their daily jobs
e-mail former and prospective employers
conduct furtive instant-messaging chats
visit web sites that cater to disgruntled
employees, such as f'dcompany.com
perform large downloads and file copying
access the network during off hours.
Intrusion Techniques
 objective
to gain access or increase
privileges
 initial attacks often exploit system or
software vulnerabilities to execute code to
get backdoor

 or

e.g. buffer overflow
to gain protected information
e.g. password guessing or acquisition
Intrusion Detection Systems

classify intrusion detection/prevention systems
(IDS / IPS) as:



Host-based IDS/IPS: monitor single host activity
Network-based IDS/IPS: monitor network traffic
logical components:



sensors - collect data
analyzers - determine if intrusion has occurred
user interface - manage / direct / view IDS/IPS
IDS/IPS Principles
 assume
intruder behavior differs from
legitimate users



expect overlap as shown
observe deviations
from past history
problems of:
• false positives
• false negatives
• must compromise
IDS/IPS Principles
Where to set limit (draw arrow)?

false positives: normal behavior labeled attack
 false negatives: attack labeled normal behavior
 must compromise
IDS Requirements









run continually
be fault tolerant
resist subversion
impose a minimal overhead on system (if hids)
configured according to system security policies
adapt to changes in systems and users
scale to monitor large numbers of systems
provide graceful degradation of service
allow dynamic reconfiguration
Host-Based IDS

specialized software to monitor system activity to
detect suspicious behavior



primary purpose is to detect intrusions, log suspicious
events, and send alerts
can detect both external and internal intrusions
two approaches, often used in combination:

anomaly detection - defines normal/expected
behavior
• threshold detection
• profile based

signature detection - defines proper behavior
Audit Records
a
fundamental tool for intrusion detection
 two variants:

native audit records - provided by O/S
• always available but may not be optimum

detection-specific audit records - IDS specific
• additional overhead but specific to IDS task
• often log individual elementary actions
Audit Record Contents






Subject: initiator of action
Action: Operation performed: login, read, perform I/O,
execute
Object: Receptor of action: file, program message,
printer, …
Exception-condition: Type of exception
Resource-usage: Amount of resources used
Time-stamp: When it occurred
Smith
Execute
<library>
Copy.exe
0
CPU=0002 11058721678
Smith
Write
<library>
mainDB
Writeviol
Records =
0
11058721678
Anomaly Detection

threshold detection




checks excessive event occurrences over time
alone a crude and ineffective intruder detector
must determine both thresholds and time intervals
profile based



characterize past behavior of users / groups
then detect significant deviations
based on analysis of audit records
• gather metrics: counter, gauge, interval timer, resource utilization
• analyze: mean and standard deviation, multivariate, Markov
process, time series, operational model
Login & Session Activity
Measure
Model
Type of Intrusion Detected
Login frequency by date
and time
Mean and standard Intruders likely to login after
deviation
normal hours
Frequency of login at
different locations
Mean and standard Login from place rarely at
deviation
Time since last login
Operational
Break-in at dead account
Elapsed time per session Mean and standard Significant deviations =
deviation
masquerader?
Quantity of output to
location
Mean and standard Excessive data transmitted
deviation
could be leakage of sensitive
data
Session resource
utilization
Mean and standard Unusual processor or I/O
deviation
levels = intruder?
Password failures at
login
Operational
Attempted break-in by
guessing
Command or Program
Execution Activity
Measure
Model
Type of Intrusion Detection
Execution
frequency
Mean and Standard
Deviation
Different set of commands used, or
privileged commands => break-in
Program
resource
utilization
Mean and Standard
Deviation
High values may result from virus,
Trojan horse, worm
Execution
denials
Operations model
Penetration attempt by user to seeks
higher privileges
File Access Activity
Measure
Model
Type of Intrusion Detected
Read, write, create,
delete frequency
Mean and
Abnormalities for access may
Standard deviation signify masquerading or browsing
Records read, written
Mean and
Attempt to obtain sensitive data
standard deviation by inference and aggregation
Failure count for read,
write, create, delete
Operational
May detect users who
persistently attempt to access
unauthorized files
Signature Detection
 observe
events on system and applying a
set of rules to decide if intruder
 approaches:

rule-based anomaly detection
• analyze historical audit records for expected
behavior, then match with current behavior

rule-based penetration identification
• rules identify known penetrations / weaknesses
• often by analyzing attack scripts from Internet
• supplemented with rules from security experts
Rule based penetration
examples






Users should not read files in other users’ personal
directories
Users must not write to others’ files
Users who log in after hours often access the same files
they used earlier
Users do not generally open disk devices directly but rely
on higher-level operating system utilities
Users should not be logged in more than once to the
same system
Users do not make copies of system programs
Signature- versus AnomalyBased IDS
Signature-based: Looks for attack signatures in packets or logs
 Retains signatures in a signature database or rule set(s).
 Can create custom rules – sometimes with wildcards
Benefits & Limitations
Benefit: Can name specific attacks, allowing for appropriate reaction
Limitations:
 More signatures translates into lower transaction rates
 Slight deviations from the signature won’t be caught: e.g., blank vs.
%20
 New attacks cannot be caught
Anomaly-based or Heuristic:
Looks for unexpected behavior
Baseline-based Intrusion Detection: ‘Expected’ performance is
known
 Thresholds are established differentiating normal vs. abnormal
behavior
 E.g., Rate of SYN or Ping packets change
 E.g., Monitoring processor usage at night
 E.g., Packet is not formatted as expected
Rule-based Intrusion Detection: Certain actions are not allowed
 E.g., Log accesses to password file
 Benefits & Limitations
Benefit: Quick at recognizing new large-scale worm attacks
Limitation: Cannot name the attack; Cannot detect attacks near norm
System Sensitivity
False positive: Innocent action logged as
an attack
False negative: Attack not recognized
Sensitivity of the system: The degree of
False positives to False negatives
 Administrator must achieve the right
balance of sensitivity
Types of NIDS, NIPS
Passive mode or IDS: Monitors network traffic only
 Does not affect performance of network traffic
 Can be incapable of sending on network
Active mode or IPS: Performs inline processing of
packets
 Causes penalty on performance – problematic
for very busy networks
 Must be capable of sending on network
IDS versus IPS
Intrusion Detection System (IDS): Sniffs and reports
possible violations
 Difference between Firewall/IDS: Can name attack
Intrusion Prevention System (IPS): Reports violations
and prevents attacks from occurring
 Does inline processing, similar to a Firewall: drop
packets, reset connections, route suspicious traffic for
analysis
 Problems: Delays in processing; bottleneck
 Since IDS/IPS have high rate of False Positives, they
require extensive optimization
What an IDS Cannot Detect
 Passwords
not changed from default
 File transfer of confidential files (unless
specifically programmed)
 Social engineering techniques
 Decipher encrypted messages on a
network
Bus or other broadcast configuration
Star Configuration
Attacks can be categorized as:
NIDS/NIPS Detects
Single Packet Attacks Multiple Packet Attacks
Context
(Header)
Attacks
Ping of Death: Packet
> buffer size 65535
Land.c Attack: Source
& Destination IP
Address is same
Port Sweep
TCP Hijack
SYN Flood Attack
Content
(Data)
Attacks
DNS Attack: Incorrect
data
Character Mode Attacks:
Bypass ASCII signatures
to send Unicode or
hexadecimal signatures
Comparison of Host versus
Network-Based IDS/IPS
Quality
Cost of
Ownership
Network-Based IDS
One strategically-placed IDS serves
community of computers. A separate
NIDS computer must be purchased.
Strengths
Extensive protocol-based attack
patterns are supported.
Cannot check all encrypted packets.
Evidence
Removal
Real-Time
Detection &
Response
Malicious
Intent
Detection
Complement
& Verification
Operating
System
Host-Based IDS
Each host requires its own IDS software
package
Monitors important system components such as
key DLL’s and the NT Registry.
Monitors user & file access activity.
Some protocol attack patterns are supported for
stack-based HIDS.
Operates after decryption process – can check
all encrypted packets.
Attackers cannot remove evidence once Attackers can change logs.
captured
Alarms occur as traffic is captured – real Alarm occurs as log is monitored in near-real
time
time.
Protocol alarms occur in real time.
Placing an NIDS outside the firewall can
inform you of attacks that did not
succeed
Can be used to verify proper functioning Can compliment other systems.
of firewall.
False positives are less likely to occur.
False positives will occur
Operating System independent
Specific to one O.S. and requires O.S. to be
functioning properly & not compromised.
Switch A
Router B
Tap
NIDS
More Detail Below…
Switch
A
Tap Internal Configuration
Top Layer
Switch
Router
B
NIDS
The switch can buffer overlapping traffic to a degree.
However be careful not to overload the switch.
How to Attach an IDS
Switch: Central router routes traffic only

to destination node.
High throughput since the
simultaneous transmissions can occur
between different pairs.
Hub: repeats traffic to all nodes


Switch Port Analyzer (SPAN) allows


a network sniffer to monitor
TX/RX/both transmissions between 2
(or sometimes more) nodes
(commonly switch & router)
Disadvantage: Switch only has one
SPAN port, switch performance
degradation
Advantage: No extra equipment, easy
to install
Disadvantage: Throughput limitations
since all nodes share same physical
link – cannot implement duplex
transmission between switch/router
Advantage: Easy to install and
configure
Tap: A ‘T’ or listening device forwards


traffic to the NIDS
Disadvantage: Usually monitors in
one direction only, dictates stealth
configuration
Advantage: Fault tolerant on power
failure, no throughput degradation,
protects IDS from attacks
Distributed Host-Based IDS
Distributed Host-Based IDS
Network-Based IDS
 network-based



monitor traffic at selected points on a network
in (near) real time to detect intrusion patterns
may examine network, transport and/or
application level protocol activity directed
toward systems
 comprises


IDS (NIDS)
a number of sensors
inline (possibly as part of other net device)
passive (monitors copy of traffic)
NIDS Sensor Deployment
Intrusion Detection Techniques
 signature

at application, transport, network layers;
unexpected application services, policy violations
 anomaly

detection
detection
of denial of service attacks, scanning, worms
 when
potential violation detected sensor
sends an alert and logs information


used by analysis module to refine intrusion
detection parameters and algorithms
by security admin to improve protection
Distributed Adaptive Intrusion
Detection
Intrusion
Detection
Exchange
Format
Honeypots
 are




decoy systems
filled with fabricated info
instrumented with monitors / event loggers
divert and hold attacker to collect activity info
without exposing production systems
 initially
were single systems
 more recently are/emulate entire networks
Honeypot: Looks Real

System with NO OTHER USERS or USED
APPLICATIONS – LOG all access attempts
 Honeypots are high maintenance, high risk
 Honeypots are not legally a form of entrapment
 Types may include:




Port Monitor: Sockets-based program that listens for
connections.
Deception System: Pretends it is a real application by sending
valid replies (e.g. mail)
Multi-protocol Deception System: Pretends to support
multiple applications
Full network system plus IDS
Honeypots
Advantages:
 Watch and learn from
attackers to
strengthen defense
 Lure an attacker to a
safe place to identify
and stop the attacker
 Keep attackers busy
in a safe environment
for hours
Disadvantages:
 A hacked honeypot
can serve as
launching pad into
rest of network
 Honeypots must be
maintained and
monitored
Honeypot
Deployment
SNORT
 lightweight


IDS
real-time packet capture and rule analysis
passive or inline (IPS)
SNORT Rules





use a simple, flexible rule definition language
with fixed header and zero or more options
header includes: action, protocol, source IP, source
port, direction, dest IP, dest port
many options
example rule to detect TCP SYN-FIN attack:
Alert tcp $EXTERNAL_NET any -> $HOME_NET any \
(msg: "SCAN SYN FIN"; flags: SF, 12; \
reference: arachnids, 198; classtype: attempted-recon;)
SNORT NIDS->NIPS
Snort Format:
{cmd} {protocol}
{sourceIP}
{sourcePort}
{direction} {destIP}
{destPort}
(<keyword>:<value>;
<keyword>:<value>)

Cmd=alert pass log
activate dynamic



log=packet text only,
alert writes to alert file
Protocol=ip udp icmp
tcp arp, igrp, gre,
ospf, rip, …
 Port= :1024 or
1024:6000
 Direction= -> or <>
Snort
Command
Example
Snort Format:

{cmd} {protocol} {sourceIP} {sourcePort} {direction} {destIP} {destPort}
(<keyword>:<value>; <keyword>:<value>)
Example:
 var HTTP_SERVERS [192.168.1.50/32]
 var HOME_NET [192.168.1.0/24]
 var EXTERNAL_NET !HOME_NET
 alert tcp $EXTERNAL_NET any ->
$HTTP_SERVERS 80 (msg:”WEB-IIS
cmd.exe access”; flags: A+;
content:”cmd.exe”; nocase;
classtype:web_application-attack;)
Snort Keywords















Keywords can include:
dsize: maximum packet size; larger sizes indicate problems.
ttl: IP time to live value.
fragbits: R=Reserved, D=Don’t Fragment, M=More Fragment.
ipopts: IP options: lsrr: loose source routing; ssrr=strict source routing.
flags: S=Syn, A=Ack, F=Fin, R=Reset, +=and/or more;
itype: ICMP packet type
content: <text or hexadecimal data to search for>
uricontent: Content of the URL (e.g., ”/bin/ps”)
offset: the position in the packet payload to begin searching for a match.
nocase: Deactivates case-sensitivity
sid: signature ID; describes more about the signature
ip_proto: protocol after IP header (e.g., DNS=53)
rev: rule revision number
logto: file to write log to.
Snort IPS Additional
Commands
New commands used for inline
configurations:
 drop: Alert and drop the packet
 sdrop: Drop the packet but don’t trigger
the alert
 E.g.: sdrop udp $EXTERNAL_NET any …
Snort IPS Added Keywords
resp:<resp_keyword>[,resp_keyword]
 <resp_keyword> = rst_snd, rst_rcv, rst_all, icmp_net, icmp_host,
icmp_port, icmp_all
 Sends RST to packet sender/recipient/both; Sends
host/port/network Unreachable
react:<react_keyword>[,react_keyword]
 <react_keyword> = block, warn, msg, proxy
 Used with HTTP-based attacks.
 E.g.: alert tcp any any <> $HOME_NET 80 (content:
“naughtyContent”; msg: “Not allowed!”; react: block,msg;)
replace: “text to replace content with”
 Allows replacement of potentially dangerous text with safe text:
“cmd.exe”->”nocmd.exe”
CISCO Secure Intrusion
Detection
Management console: Must be in a secure
location
 May alarm, log, page, and/or email
administrator
 Allows configuration of necessary
signatures
CISCO Secure Intrusion
Detection
Sensor Response to Attack: One or more of:
 Shun: Dynamically rewrites the access lists to
disallow attackers access to internal networks


Can change access lists on firewall or router.
Log: Save alarm information (at sensor and/or
management console)
 TCP Reset: Send a TCP reset to terminate the
connection, after initial attack packets have
reached victim.
Summary

introduced intruders & intrusion detection


intrusion detection approaches





hackers, criminals, insiders
host-based (single and distributed)
network
distributed adaptive
exchange format
honeypots
 SNORT example