Introduction to Honeypot, Denial-ofService, and Rootkit
Cliff C. Zou
CAP6135
Spring, 2011
1
Acknowledgement

Some contents on honeypot are from


http://staff.washington.edu/dittrich/talks/arohoneynets.ppt
Some figures on DDoS are from

http://www.cisco.com/web/IT/events/pdf/iin2005/dist
ributed_denial.pdf
2
What Is a Honeypot?


Abstract definition:
“A honeypot is an information
system resource whose value lies
in unauthorized or illicit use of
that resource.” (Lance Spitzner)
Concrete definition:
“A honeypot is a faked
vulnerable system used for the
purpose of being attacked,
probed, exploited and
compromised.”
3
Example of a Simple Honeypot

Install vulnerable OS and software on a
machine

Install monitor or IDS software

Connect to the Internet (with global IP)


Wait & monitor being scanned, attacked,
compromised
Finish analysis, clean the machine
4
Benefit of Deploying Honeypots

Risk mitigation:


Lure an attacker away from the real production
systems (“easy target“).
IDS-like functionality:

Since no legitimate traffic should take place to or
from the honeypot, any traffic appearing is evil
and can initiate further actions.
5
Benefit of Deploying Honeypots

Attack analysis:
Find out reasons, and strategies why and how
you are attacked.
 Binary and behavior analysis of capture
malicious code


Evidence:


Once the attacker is identified, all data captured
may be used in a legal procedure.
Increased knowledge
6
Honeypot Classification

High-interaction honeypots


A full and working OS is provided for being attacked
VMware virtual environment


Low-interaction honeypots


Only emulate specific network services
No real interaction or OS


Several VMware virtual hosts in one physical machine
Honeyd
Honeynet/honeyfarm

A network of honeypots
7
Low-Interaction Honeypots

Pros:





Easy to install (simple program)
No risk (no vulnerable software to be attacked)
One machine supports hundreds of honeypots, covers
hundreds of IP addresses
Can distinguish most attacks on the same port
Cons:

No real interaction to be captured



Limited logging/monitor function
Hard to detect unknown attacks; hard to generate filters
Easily detectable by attackers
8
Emulation of Services
QUIT* )
echo -e "221 Goodbye.\r"
exit 0;;
SYST* )
echo -e "215 UNIX Type: L8\r"
;;
HELP* )
echo -e "214-The following commands are recognized (* =>'s unimplemented).\r"
echo -e "
USER
PORT
STOR
MSAM*
RNTO
NLST
MKD
CDUP\r"
echo -e "
PASS
PASV
APPE
MRSQ*
ABOR
SITE
XMKD
XCUP\r"
echo -e "
ACCT*
TYPE
MLFL*
MRCP*
DELE
SYST
RMD
STOU\r"
echo -e "
SMNT*
STRU
MAIL*
ALLO
CWD
STAT
XRMD
SIZE\r"
echo -e "
REIN*
MODE
MSND*
REST
XCWD
HELP
PWD
MDTM\r"
echo -e "
QUIT
RETR
MSOM*
RNFR
LIST
NOOP
XPWD\r"
echo -e "214 Direct comments to ftp@$domain.\r"
;;
USER* )
9
High-Interaction Honeypots

Pros:
Real OS, capture all attack traffic/actions
 Can discover unknown attacks/vulnerabilites
 Can capture and anlayze code behavior


Cons:
Time-consuming to build/maintain
 Time-consuming to analysis attack
 Risk of being used as stepping stone
 High computer resource requirement

10
Honeynet


A network of honeypots
High-interaction honeynet


A distributed network composing many honeypots
Low-interaction honeynet


Emulate a virtual network in one physical machine
Example: honeyd
11
Gen II Honeynet
12
Data Control

Prevent a honeypot being used by attackers
to attack others (legal/ethnical issues)
13
Honeypot-Aware Botnet [Zou’07]

Honeypot is widely used by defenders
Ability to detect unknown attacks
 Ability to monitor attacker actions (e.g., botnet
C&C)


Botnet attackers will adapt to honeypot
defense
When they feel the real threat from honeypot
 We need to think one step ahead

14
Honeypot Detection Principles

Hardware/software specific honeypot detection

Detect virtual environment via specific code




E.g., time response, memory address
Detect faculty honeypot program
Case by case detection
Detection based on fundamental difference

Honeypot defenders are liable for attacks sending out



Liability law will become mature
It’s a moral issue as well
Real attackers bear no liability
15

Check whether a bot can send out malicious traffic or not
Detection of Honeypot Bot
bot
1 malicious traffic
Sensor (secret)
C&C

Infection traffic



Real liability to defenders
No exposure issue: a bot needs to do this regardless
Other honeypot detection traffic

Port scanning, email spam, web request (DoS?)
16
Two-stage Reconnaissance to Detect
Honeypot in Constructing P2P Botnets
1
Host A
spearhead
3
Host B
2
spearhead
request
main-force

Fully distributed


No central sensor is used
Could be fooled by double-honeypot


Counterattack is presented in our paper
Lightweighted spearhead code


17
Infect + honeypot detection
Speedup UDP-based infection
Host C
Defense against
Honeypot-Aware Attacks

Permit dedicated honeypot detection systems to
send out malicious traffic


Redirect outgoing traffic to a second honeypot


Not effective for sensor-based honeypot detection
Figure out what outgoing traffic is for honeypot
detection, and then allow it


Need law and strict policy
It could be very hard
Neverthless, honeypot is still a valuable monitoring
and
detection/defense tool
18
Distributed Denial of Service
(DDoS) Attack


Send large amount of traffic to a server so that the
server has no resource to serve normal users
Attacking format:

Consume target memory/CPU resource



SYN flood (backscatter paper presented before)
Database query…
Congest target Internet connection


Many sources attack traffic overwhelm target link
Very hard to defend
19
Why hard to defined DDoS attack?

Internet IP protocol has no built-in security

No authentication of source IP




SYN flood with faked source IP
However, IP is true after connection is setup
Servers are supposed to accept unsolicited service
requests
Lack of collaboration ways among Internet
community

How can you ask an ISP in another country to block
certain traffic for you?
20
DDoS Defenses

Increase servers capacity


Use Internet web caching service


Cluster of machine, Multi-CPUs, larger Internet
access
E.g., Akamai
Defense Methods (many in research stage)
SYN cookies (http://en.wikipedia.org/wiki/SYN_cookies)
 SOS
 IP traceback

21
22
23
SYN Cookies

SYN flood attack
Fill up server’s SYN queue
 Property: attacker does not respond to SYN/ACK
from victim.


Defense
Fact: normal client responds to SYN/ACK
 Remove initial SYN queue
 Server encode info in TCP seq. number


Use it to reconstruct the initial SYN
24
DoS spoofed attack defense: IP
traceback

Suppose a victim can call ISPs upstream to
block certain traffic

SYN flood: which traffic to block?

IP traceback:
Find out the real attacking host for SYN flood
 Based on large amount of attacking packets
 Need a little help from routers (packet marking)

25
SOS: Secure Overlay Service

Central Idea:

Use many TCP connection respondent machines
Only setup connections relay to server
 Identity of server is secrete

26
The Evolution of Malware


Malware, including spyware, adware and viruses
want to be hard to detect and/or hard to remove
Rootkits are a fast evolving technology to achieve
these goals




Cloaking technology applied to malware
Not malware by itself
Example rootkit-based viruses: W32.Maslan.A@mm,
W32.Opasa@mm
Rootkit history

Appeared as stealth viruses


One of the first known PC viruses, Brain, was stealth
First “rootkit” appeared on SunOS in 1994

Replacement of core system utilities (ls, ps, etc.) to hide malware
processes
Cloaking

Modern rootkits can cloak:







Several major rootkit technologies





Processes
Services
TCP/IP ports
Files
Registry keys
User accounts
User-mode API filtering
Kernel-mode API filtering
Kernel-mode data structure manipulation
Process hijacking
Visit www.rootkit.com for tools and information
User-Mode API Filtering

Attack user-mode system query APIs
Taskmgr.exe
Explorer.exe,
Winlogon.exe
Ntdll.dll
Rootkit
user mode
kernel mode

Explorer.exe, Malware.exe, Winlogon.exe
Con: can be bypassed by going directly to kernelmode APIs

Pro: can infect unprivileged user accounts

Examples: HackerDefender, Afx
Kernel-Mode API Filtering
Attack kernel-mode system query APIs

Taskmgr.exe
Explorer.exe,
Winlogon.exe
Ntdll.dll
user mode

kernel mode
Explorer.exe,
Winlogon.exe

Explorer.exe, Malware.exe,
Winlogon.exe

Cons:


Requires admin privilege to install
Difficult to write

Pro: very thorough cloak

Example: NT Rootkit
Rootkit
Kernel-Mode Data Structure
Manipulation


Also called Direct Kernel Object Manipulation (DKOM)
Attacks active process data structure


Query API doesn’t see the process
Kernel still schedules process’ threads
Active
Processes




Malware.exe
Cons:


Explorer.exe
Requires admin privilege to install
Can cause crashes
Detection already developed
Pro: more advanced variations possible
Example: FU
Winlogon.exe
Process Hijacking

Hide inside a legitimate process
Explorer.exe
Malware

Con: doesn’t survive reboot

Pro: extremely hard to detect

Example: Code Red
Detecting Rootkits

All cloaks have holes
Leave some APIs unfiltered
 Have detectable side effects
 Can’t cloak when OS is offline


Rootkit detection attacks holes
Cat-and-mouse game
 Several examples





Microsoft Research Strider/Ghostbuster
RKDetect
Sysinternals RootkitRevealer
F-Secure BlackLight
Simple Rootkit Detection

Perform a directory listing online and
compare with secure alternate OS boot
(see http://research.microsoft.com/rootkit/ )
Offline OS is Windows PE, ERD
Commander, BartPE
dir /s /ah * > dirscan.txt
windiff dirscanon.txt
dirscanoff.txt


This won’t detect non-persistent rootkits
that save to disk during shutdown
RootkitRevealer

RootkitRevealer (RKR) runs online

RKR tries to bypass rootkit to uncover cloaked objects



All detectors listed do the same
RKR scans HKLM\Software, HKLM\System and the file
system
Performs Windows API scan and compares with raw data
structure scan
RootkitRevealer
Filtered Windows API
omits malware files and keys
Rootkit
Windows API
Raw file system,
Raw Registry hive
Malware files and keys
are visible in raw scan
Demo

HackerDefender


HackerDefender before and after view of file system
Detecting HackerDefender with RootkitRevealer
Dealing with Rootkits

Unless you have specific uninstall
instructions from an authoritative source:
Reformat the system and reinstall Windows!

Don’t rely on “rename” functionality
offered by some rootkit detectors
It might not have detected all a rootkit’s
components
 The rename might not be effective
