Introduction to Honeypot, Denial-ofService, and Rootkit
Cliff C. Zou
CAP6135
Spring, 2010
1
What Is a Honeypot?


Abstract definition:
“A honeypot is an information
system resource whose value lies
in unauthorized or illicit use of
that resource.” (Lance Spitzner)
Concrete definition:
“A honeypot is a faked
vulnerable system used for the
purpose of being attacked,
probed, exploited and
compromised.”
2
Example of a Simple Honeypot

Install vulnerable OS and software on a
machine

Install monitor or IDS software

Connect to the Internet (with global IP)


Wait & monitor being scanned, attacked,
compromised
Finish analysis, clean the machine
3
Benefit of Deploying Honeypots

Risk mitigation:


Lure an attacker away from the real production
systems (“easy target“).
IDS-like functionality:

Since no legitimate traffic should take place to or
from the honeypot, any traffic appearing is evil
and can initiate further actions.
4
Benefit of Deploying Honeypots

Attack analysis:
Find out reasons, and strategies why and how
you are attacked.
 Binary and behavior analysis of capture
malicious code


Evidence:


Once the attacker is identified, all data captured
may be used in a legal procedure.
Increased knowledge
5
Honeypot Classification

High-interaction honeypots


A full and working OS is provided for being attacked
VMware virtual environment


Low-interaction honeypots


Only emulate specific network services
No real interaction or OS


Several VMware virtual hosts in one physical machine
Honeyd
Honeynet/honeyfarm

A network of honeypots
6
Low-Interaction Honeypots

Pros:




Easy to install (simple program)
No risk (no vulnerable software to be attacked)
One machine supports hundreds of honeypots, covers
hundreds of IP addresses
Cons:

No real interaction to be captured



Limited logging/monitor function
Hard to detect unknown attacks; hard to generate filters
Easily detectable by attackers
7
High-Interaction Honeypots

Pros:
Real OS, capture all attack traffic/actions
 Can discover unknown attacks/vulnerabilites
 Can capture and anlayze code behavior


Cons:
Time-consuming to build/maintain
 Time-consuming to analysis attack
 Risk of being used as stepping stone
 High computer resource requirement

8
Honeynet


A network of honeypots
High-interaction honeynet


Low-interaction honeynet




A distributed network composing many honeypots
Emulate a virtual network in one physical machine
Example: honeyd
Mixed honeynet
 “Scalability, Fidelity and Containment in the
Potemkin Virtual Honeyfarm”, presented next
week
Reference: http://www.ccc.de/congress/2004/fahrplan/files/135honeypot-forensics-slides.ppt
9
Honeypot-Aware Botnet [Zou’07]

Honeypot is widely used by defenders
Ability to detect unknown attacks
 Ability to monitor attacker actions (e.g., botnet
C&C)


Botnet attackers will adapt to honeypot
defense
When they feel the real threat from honeypot
 We need to think one step ahead

10
Honeypot Detection Principles

Hardware/software specific honeypot detection

Detect virtual environment via specific code




E.g., time response, memory address
Detect faculty honeypot program
Case by case detection
Detection based on fundamental difference

Honeypot defenders are liable for attacks sending out



Liability law will become mature
It’s a moral issue as well
Real attackers bear no liability
11

Check whether a bot can send out malicious traffic or not
Detection of Honeypot Bot

bot

1 malicious traffic


Sensor (secret)
C&C
Infection traffic




Real liability to defenders
No exposure issue: a bot needs to do this regardless
Other honeypot detection traffic

Port scanning, email spam, web request (DoS?)
12
Two-stage Reconnaissance to Detect
Honeypot in Constructing P2P Botnets

Host A



3


spearhead

Host B



2
spearhead
request
main-force
Fully distributed


No central sensor is used
Could be fooled by double-honeypot


1
Counterattack is presented in our paper
Lightweighted spearhead code


13
Infect + honeypot detection
Speedup UDP-based infection
Host C

Defense against
Honeypot-Aware Attacks

Permit dedicated honeypot detection systems to
send out malicious traffic


Redirect outgoing traffic to a second honeypot


Not effective for sensor-based honeypot detection
Figure out what outgoing traffic is for honeypot
detection, and then allow it


Need law and strict policy
It could be very hard
Neverthless, honeypot is still a valuable monitoring
and
detection/defense tool
14
Distributed Denial of Service
(DDoS) Attack


Send large amount of traffic to a server so that the
server has no resource to serve normal users
Attacking format:

Consume target memory/CPU resource



SYN flood (backscatter paper presented before)
Database query…
Congest target Internet connection


Many sources attack traffic overwhelm target link
Very hard to defend
15
Why hard to defined DDoS attack?

Internet IP protocol has no built-in security

No authentication of source IP




SYN flood with faked source IP
However, IP is true after connection is setup
Servers are supposed to accept unsolicited service
requests
Lack of collaboration ways among Internet
community

How can you ask an ISP in another country to block
certain traffic for you?
16
DDoS Defenses

Increase servers capacity


Use Internet web caching service


Cluster of machine, Multi-CPUs, larger Internet
access
E.g., Akamai
Defense Methods (many in research stage)
SYN cookies (http://en.wikipedia.org/wiki/SYN_cookies)
 SOS
 IP traceback

17
SYN Cookies

SYN flood attack
Fill up server’s SYN queue
 Property: attacker does not respond to SYN/ACK
from victim.


Defense
Fact: normal client responds to SYN/ACK
 Remove initial SYN queue
 Server encode info in TCP seq. number


Use it to reconstruct the initial SYN
18
DoS spoofed attack defense: IP
traceback

Suppose a victim can call ISPs upstream to
block certain traffic

SYN flood: which traffic to block?

IP traceback:
Find out the real attacking host for SYN flood
 Based on large amount of attacking packets
 Need a little help from routers (packet marking)

19
SOS: Secure Overlay Service

Central Idea:

Use many TCP connection respondent machines
Only setup connections relay to server
 Identity of server is secrete

20
The Evolution of Malware


Malware, including spyware, adware and viruses
want to be hard to detect and/or hard to remove
Rootkits are a fast evolving technology to achieve
these goals




Cloaking technology applied to malware
Not malware by itself
Example rootkit-based viruses: W32.Maslan.A@mm,
W32.Opasa@mm
Rootkit history

Appeared as stealth viruses


One of the first known PC viruses, Brain, was stealth
First “rootkit” appeared on SunOS in 1994

Replacement of core system utilities (ls, ps, etc.) to hide malware
processes
Cloaking

Modern rootkits can cloak:







Several major rootkit technologies





Processes
Services
TCP/IP ports
Files
Registry keys
User accounts
User-mode API filtering
Kernel-mode API filtering
Kernel-mode data structure manipulation
Process hijacking
Visit www.rootkit.com for tools and information
User-Mode API Filtering

Attack user-mode system query APIs
Taskmgr.exe

Explorer.exe,
Winlogon.exe

Ntdll.dll


Rootkit
user mode

kernel mode

Explorer.exe, Malware.exe, Winlogon.e


Con: can be bypassed by going directly to kernelmode APIs

Pro: can infect unprivileged user accounts

Examples: HackerDefender, Afx
Kernel-Mode API Filtering
Attack kernel-mode system query APIs

Taskmgr.exe

Explorer.exe,
Winlogon.exe

Ntdll.dll

user mode

kernel mode
Explorer.exe,
Winlogon.exe

Explorer.exe, Malware.exe,
Winlogon.exe


Cons:


Requires admin privilege to install
Difficult to write

Pro: very thorough cloak

Example: NT Rootkit


Rootkit
Kernel-Mode Data Structure
Manipulation


Also called Direct Kernel Object Manipulation
Attacks active process data structure


Query API doesn’t see the process
Kernel still schedules process’ threads
Active Explorer.exe
Processes


Cons:





Malware.exe

Requires admin privilege to install
Can cause crashes
Detection already developed
Pro: more advanced variations possible
Example: FU
Winlogon.exe

Process Hijacking

Hide inside a legitimate process
Explorer.exe
Malware


Con: doesn’t survive reboot

Pro: extremely hard to detect

Example: Code Red
Detecting Rootkits

All cloaks have holes
Leave some APIs unfiltered
 Have detectable side effects
 Can’t cloak when OS is offline


Rootkit detection attacks holes
Cat-and-mouse game
 Several examples





Microsoft Research Strider/Ghostbuster
RKDetect
Sysinternals RootkitRevealer
F-Secure BlackLight
Simple Rootkit Detection

Perform a directory listing online and
compare with secure alternate OS boot
(see http://research.microsoft.com/rootkit/ )
Offline OS is Windows PE, ERD
Commander, BartPE
dir /s /ah * > dirscan.txt
windiff dirscanon.txt
dirscanoff.txt


This won’t detect non-persistent rootkits
that save to disk during shutdown
RootkitRevealer

RootkitRevealer (RKR) runs online

RKR tries to bypass rootkit to uncover cloaked objects



All detectors listed do the same
RKR scans HKLM\Software, HKLM\System and the file
system
Performs Windows API scan and compares with raw data
structure scan
RootkitRevealer

Filtered Windows API

Rootkit

omits malware files and keys

Malware files and keys
are visible in raw scan

Windows API

Raw file system,
Raw Registry hive

Demo

HackerDefender


HackerDefender before and after view of file system
Detecting HackerDefender with RootkitRevealer
RootkitRevealer Limitations

Rootkits have already attacked RKR
directly by not cloaking when scanned
RKR is given true system view
 Windows API scan looks like raw scan


SysInternals have modified RKR to be a
harder to detect by rootkits
RKR is adopting rootkit techniques itself
 Rootkit authors will continue to find ways
around RKR’s cloak
 It’s a game nobody can win

Dealing with Rootkits

Unless you have specific uninstall
instructions from an authoritative source:
Reformat the system and reinstall Windows!


Don’t rely on “rename” functionality
offered by some rootkit detectors
It might not have detected all a rootkit’s
components
 The rename might not be effective
