Log Analysis and Intrusion
Detection
By
Srikrishna Gudavalli
Venkata Naga Vamsi Krishna
Ravi Kiran Yellepeddy
Log Analysis (Windows And linux)
What is log analysis?
Describes an event (or) process
activity in detail on the system.
Examples :
• user authentication event log
• ftp authentication .
Setup for LogAnalysis
• Application Log
Specific to particular application.
eg:MS word,Windows Media Player
• Security Log
Specifically logs all the security features.
• System Log
Logs all the system related activities.
Log Files
• Lokasi bergantung, umumnya pada
/var/log
• Contoh
/var/log/mail.log
/var/log/messages
/var/log/daemon.log
/var/log/apache/access-log
/var/log/apache/error-log
/var/adm/utmp
/var/adm/wtmp
Log Analysis and Correlation
• Syslogs, messages logs, other Unix
host logs
Security/Auth Log
Mar 9 13:07:49
Mar 9 13:09:24
Mar 9 13:09:27
Mar 9 13:09:28
Mar 9 13:09:28
Mar 9 13:09:31
Mar 9 13:12:13
Mar 9 13:12:13
Mar 9 13:12:14
Mar 9 13:12:16
Mar 9 13:12:18
Mar 9 13:15:23
Mar 9 13:15:24
Mar 9 13:15:27
Mar 9 13:15:28
Mar 9 13:15:29
Mar 9 13:26:43
Mar 9 13:37:15
Mar 9 13:37:44
Mar 9 17:17:19
Mar 9 17:17:26
Mar 9 17:50:13
Mar 10 11:12:02
Mar 10 11:13:07
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
nile
in.telnetd[1315]: connect from 68.62.72.193
in.rlogind[1321]: connect from 68.62.72.193
in.ftpd[1326]: connect from 68.62.72.193
in.rshd[1329]: connect from 68.62.72.193
in.telnetd[1333]: connect from 68.62.72.193
in.fingerd[1334]: connect from 68.62.72.193
in.fingerd[1352]: connect from 68.62.72.193
in.rlogind[1357]: connect from 68.62.72.193
in.rshd[1360]: connect from 68.62.72.193
in.telnetd[1365]: connect from 68.62.72.193
in.ftpd[1368]: connect from 68.62.72.193
in.ftpd[1382]: connect from 68.62.72.193
in.telnetd[1384]: connect from 68.62.72.193
in.rshd[1396]: connect from 68.62.72.193
in.rlogind[1398]: connect from 68.62.72.193
in.fingerd[1400]: connect from 68.62.72.193
login: ROOT LOGIN ON tty1
in.ftpd[1447]: connect from 68.62.72.193
in.fingerd[1448]: connect from 68.62.72.193
in.telnetd[1521]: connect from 12.87.62.43
login: LOGIN ON 0 BY pstephen FROM 43.detroit-16-17rs.mi.dial-access.att.net
in.ftpd[1556]: connect from 216.205.122.231
in.ftpd[8929]: connect from 200.68.32.185
in.ftpd[8965]: connect from 68.62.72.193
5
Log Analysis and Correlation
• TCPDump logs
11:30:27.181108 eth0 < pcp01103425pcs.aubrnh01.mi.comcast.net.17697 > nile.ftp: . 1:1(0) ack 1 win 4288 (DF)
11:30:27.190617 eth0 > arp who-has ubr01-a-rtr.aubrnh01.mi.comcast.net tell nile (0:0:86:54:50:5b)
11:30:27.198369 eth0 < arp reply ubr01-a-rtr.aubrnh01.mi.comcast.net is-at 0:5:5f:e9:10:54 (0:0:86:54:50:5b)
11:30:27.207662 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20012 1/2/2 PTR
pcp01103425pcs.aubrnh01.mi.comcast.net. (174) (DF)
11:30:27.218149 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20013 1/2/2 A
pcp01103425pcs.aubrnh01.mi.comcast.net (151) (DF)
11:30:27.230334 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20014 1/2/2 PTR
pcp01103425pcs.aubrnh01.mi.comcast.net. (174) (DF)
11:30:27.231013 eth0 > nile.ftp > pcp01103425pcs.aubrnh01.mi.comcast.net.17697: P 1:80(79) ack 1 win 32120 (DF) [tos
0x10]
11:30:27.253084 eth0 < pcp01103425pcs.aubrnh01.mi.comcast.net.17697 > nile.ftp: P 1:16(15) ack 80 win 4209 (DF)
11:30:27.253122 eth0 > nile.ftp > pcp01103425pcs.aubrnh01.mi.comcast.net.17697: . 80:80(0) ack 16 win 32120 (DF) [tos
0x10]
6
Log Analysis and Correlation
• Intrusion Detection Log (RealSecure)
Event Date
9/10/2001 11:27
9/10/2001 11:27
9/10/2001 11:27
9/10/2001 11:27
9/10/2001 11:27
9/10/2001 11:27
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
9/10/2001 11:44
Event Name
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
SNMP_Activity
Protocol Sourcce
Src Port
ID
Port Dest Port Name
17
17
17
17
17
17
17
17
17
17
17
17
17
17
17
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
162
162
162
162
162
162
162
162
162
162
162
162
162
162
162
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
1030
Dest Port
Name
Src Address
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
SNMPTRAP
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
192.168.10.199
Dest Address
10.1.230.102
10.1.230.102
10.4.18.245
10.4.18.245
10.4.18.245
10.4.18.245
192.168.6.75
192.168.6.75
192.168.6.75
192.168.6.75
10.1.151.231
10.1.151.231
10.1.151.231
10.1.151.231
10.1.151.246
Engine IP
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
192.168.9.243
7
Log Analysis and Correlation
• Intrusion Detection Log (SNORT
Summary)
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
16
16
16
16
16
17
17
17
19
20
20
20
20
20
20
20
20
21
21
22
22
02:45:37
07:17:06
14:54:20
14:54:20
14:54:20
06:02:32
06:02:32
09:45:28
08:00:19
01:26:00
03:49:38
03:49:39
21:48:55
21:48:55
22:48:13
22:48:13
23:00:33
11:01:27
11:01:28
08:36:29
08:36:29
Courtesy of The Honeynet Project
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
lisa
snort[7483]: IDS13/portmap-request-mountd: 200.190.13.181:1372 -> 172.16.1.107:111
snort[7483]: IDS128/web-cgi-phf: 200.190.8.220:55220 -> 172.16.1.107:80
snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.101
snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.105
snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.107
snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.101:111
snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.107:111
snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.105:111
snort[3515]: IDS/DNS-version-query: 212.25.75.196:1723 -> 172.16.1.101:53
snort[3515]: IDS212/dns-zone-transfer: 24.234.45.60:4075 -> 172.16.1.107:53
snort[3515]: IDS/DNS-version-query: 216.123.23.5:4349 -> 172.16.1.101:53
snort[3515]: IDS/DNS-version-query: 216.123.23.5:4350 -> 172.16.1.107:53
snort[12353]: IDS246/large-icmp: 129.142.224.3 -> 172.16.1.107
snort[12353]: IDS246/large-icmp: 129.142.224.3 -> 172.16.1.107
snort[12632]: IDS159/Ping Microsoft Windows: 216.228.4.204 -> 172.16.1.101
snort[12632]: IDS159/Ping Microsoft Windows: 216.228.4.204 -> 172.16.1.101
snort[12657]: IDS171/Ping zeros: 216.228.4.133 -> 172.16.1.101
snort[12777]: IDS/DNS-version-query: 207.236.55.76:4039 -> 172.16.1.101:53
snort[12777]: IDS/DNS-version-query: 207.236.55.76:4044 -> 172.16.1.107:53
snort[743]: IDS/DNS-version-query: 212.244.222.100:1368 -> 172.16.1.101:53
snort[743]: IDS/DNS-version-query: 212.244.222.100:1328 -> 172.16.1.107:53
8
Log Analysis and Correlation
• Intrusion
Detection
Log
(SNORT
Raw Log)
[**] WEB-MISC 403 Forbidden [**]
07/29-23:59:17.752579 0:0:C5:75:67:2C -> 0:AA:0:B7:E9:56 type:0x800 len:0x246
209.235.0.178:80 -> 63.222.202.8:1550 TCP TTL:43 TOS:0x0 ID:22555 IpLen:20 DgmLen:568 DF
***AP*** Seq: 0x85B19798 Ack: 0x4E439F5C Win: 0x7D78 TcpLen: 20
48 54 54 50 2F 31 2E 31 20 34 30 33 20 46 6F 72 HTTP/1.1 403 For
62 69 64 64 65 6E 0D 0A 44 61 74 65 3A 20 4D 6F bidden..Date: Mo
6E 2C 20 33 30 20 4A 75 6C 20 32 30 30 31 20 30 n, 30 Jul 2001 0
33 3A 35 38 3A 35 38 20 47 4D 54 0D 0A 53 65 72 3:58:58 GMT..Ser
76 65 72 3A 20 41 70 61 63 68 65 2F 31 2E 33 2E ver: Apache/1.3.
31 39 20 28 55 6E 69 78 29 20 6D 6F 64 5F 73 73 19 (Unix) mod_ss
6C 2F 32 2E 38 2E 33 20 4F 70 65 6E 53 53 4C 2F l/2.8.3 OpenSSL/
30 2E 39 2E 36 61 20 6D 6F 64 5F 70 65 72 6C 2F 0.9.6a mod_perl/
31 2E 32 35 20 6D 6F 64 5F 67 7A 69 70 2F 31 2E 1.25 mod_gzip/1.
33 2E 31 39 2E 31 61 20 50 48 50 2F 34 2E 30 2E 3.19.1a PHP/4.0.
36 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 63 6..Connection: c
6C 6F 73 65 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 lose..Content-Ty
70 65 3A 20 74 65 78 74 2F 68 74 6D 6C 3B 20 63 pe: text/html; c
68 61 72 73 65 74 3D 69 73 6F 2D 38 38 35 39 2D harset=iso-885931 0D 0A 0D 0A 3C 21 44 4F 43 54 59 50 45 20 48 1....<!DOCTYPE H
54 4D 4C 20 50 55 42 4C 49 43 20 22 2D 2F 2F 49 TML PUBLIC "-//I
45 54 46 2F 2F 44 54 44 20 48 54 4D 4C 20 32 2E ETF//DTD HTML 2.
30 2F 2F 45 4E 22 3E 0A 3C 48 54 4D 4C 3E 3C 48 0//EN">.<HTML><H
45 41 44 3E 0A 3C 54 49 54 4C 45 3E 34 30 33 20 EAD>.<TITLE>403
46 6F 72 62 69 64 64 65 6E 3C 2F 54 49 54 4C 45 Forbidden</TITLE
3E 0A 3C 2F 48 45 41 44 3E 3C 42 4F 44 59 3E 0A >.</HEAD><BODY>.
3C 48 31 3E 46 6F 72 62 69 64 64 65 6E 3C 2F 48 <H1>Forbidden</H
31 3E 0A 59 6F 75 20 64 6F 6E 27 74 20 68 61 76 1>.You don't hav
65 20 70 65 72 6D 69 73 73 69 6F 6E 20 74 6F 20 e permission to
61 63 63 65 73 73 20 2F 63 67 69 2D 62 69 6E 2F access /cgi-bin/
61 64 63 79 63 6C 65 2F 61 64 63 79 63 6C 65 2E adcycle/adcycle.
63 67 69 0A 6F 6E 20 74 68 69 73 20 73 65 72 76 cgi.on this serv
65 72 2E 3C 50 3E 0A 3C 48 52 3E 0A 3C 41 44 44 er.<P>.<HR>.<ADD
52 45 53 53 3E 41 70 61 63 68 65 2F 31 2E 33 2E RESS>Apache/1.3.
31 39 20 53 65 72 76 65 72 20 61 74 20 74 68 65 19 Server at the
62 61 62 79 63 6F 72 6E 65 72 2E 63 6F 6D 20 50 babycorner.com P
6F 72 74 20 38 30 3C 2F 41 44 44 52 45 53 53 3E ort 80</ADDRESS>
9
0A 3C 2F 42 4F 44 59 3E 3C 2F 48 54 4D 4C 3E 0A .</BODY></HTML>.
Log Analysis and Correlation
• Correlating data from multiple sources
– Normalizing
• Same events may have different names depending
upon the source
– Translating IDS codes
» Cisco NetRanger: 4052
» ISS RealSecure: Chargen_Denial_of_Service
• Use to build a chain of evidence
10
Log Analysis and Correlation
• Correlating data from multiple sources
– Deconfliction
• Same event shows up multiple times with same names
– Certain types of denial of service attacks
– Some penetration attacks
» Use care not to remove individual steps in an attack
scenario
• Same event repeated so rapidly that the logging device
reports a large number of the same event in a very short
(sometimes sub-second) period of time
• Multiple rapid events that make an attack scenario such as
a port scan
• Deconflicted events are used with normalized data to
create an event timeline
11
Log Analysis and Correlation
• Correlating data from multiple sources
– Creating chain of evidence and event timelines
• Using deconflicted and normalized events on
multiple data sources, chart the chain of events
into an event timeline
– Carefully note the timebase of various data sources
and correct to a common timebase
– Note events and attack scenarios – correlate
connected events into scenarios
• Document every assumption with evidence and,
if possible, corroboration using both forensic
and traditional investigation
12
Log Analysis and Correlation
• Forensic handling of deleted or
modified logs
– Useful only in certain types of systems
• Recovering deleted logs
– System must support recovery of ambient data
• Recovering altered logs
– Logging source must delete old log and create a
new one when the log is altered
– System must support recovery of ambient data
13
Web Server Log Analysis
Different types of log files
•
•
•
•
Access
Error
Referral
Other
SD'98 (c) David Strom, Inc.
15
Access logs
•
•
•
•
•
Domain name
Date, time
Server command processed and result
URL of visitor
Bytes transmitted
SD'98 (c) David Strom, Inc.
16
Sample access log data
•
•
•
•
•
•
rm258.fav.usu.edu [31/May/1995:09:03:23 +0600] "GET /NEI.html
HTTP/1.0" 302 396
rm258.fav.usu.edu [31/May/1995:09:03:28 +0600] "GET
/xculture/nei/nei.html HTTP/1.0" 200 2114
rm258.fav.usu.edu [31/May/1995:09:03:30 +0600] "GET
/gifs/sedlbutton.gif HTTP/1.0" 200 1336
129.71.83.161 [31/May/1995:09:20:32 +0600] "GET /RELs.html
HTTP/1.0" 304 0
Leslie-Francis.tenet.edu [31/May/1995:09:36:06 +0600] "GET /
HTTP/1.0" 200 1867
ls973.ulib.albany.edu [31/May/1995:09:40:52 +0600] "GET
/viii1.html HTTP/1.0" 404 244
SD'98 (c) David Strom, Inc.
17
Errors reported in your logs
• Clients that time out (or leave in
frustration!)
• Scripts that don’t produce any output
• Server bugs
• User authentication or configuration
problems
SD'98 (c) David Strom, Inc.
18
Sample error log data
•
•
•
•
•
•
[Thu May 30 07:25:32 1996] send timed out for bamberg.sedl.org
[Thu May 30 07:57:41 1996] send timed out for kenya.sedl.org
[Thu May 30 08:23:11 1996] send timed out for ppp092.kyotoinet.or.jp
[Thu May 30 09:15:52 1996] access to
/usr/local/www/htdocs/scimath/compass/vol03 failed for
170.211.67.51, reason: File does not exist
[Thu May 30 09:57:56 1996] send timed out for dd10048.compuserve.com
[Thu May 30 10:47:25 1996] read timed out for ncia110b.ncia.net
SD'98 (c) David Strom, Inc.
19
Referral logs
• Who links to your site?
• Who downloads your pages?
SD'98 (c) David Strom, Inc.
20
Sample referral log data
•
•
•
•
•
•
•
•
http://www.isisnet.com/ ->/change/welcome.html
http://www.ipl.org/ref/RR/EDU/Research-rr.html ->/welcome.html
http://www.tenet.edu/snp/main.html ->/policy/networks/toc.html
http://www.tenet.edu/new/main.html ->/policy/networks/toc.html
http://guide-p.infoseek.com/NS/Titles?qt=teacher+training >/resources/SCIMAST/announcement.html
http://www.tenet.edu/new/main.html ->/policy/networks/toc.html
http://www.tenet.edu/new/main.html ->/policy/networks/toc.html
http://www.nwrel.org/national/regional-labs.html >/welcome.html
SD'98 (c) David Strom, Inc.
21
Common log format
• Output by most standard servers
• Needed by most third-party log analyzers
• hoohoo.ncsa.uiuc.edu/docs/setup/httpd/Overview.html
SD'98 (c) David Strom, Inc.
22
Extended/custom log formats
• Log whatever you wish in whatever order
you wish
• Useful if you will read them regularly!
• But can’t work with the analyzers
• Now in IIS v4, NSCP v3, others.
SD'98 (c) David Strom, Inc.
23
What you can learn from your
log files
• Hits per day
• Domain origins
• The path people take in and around your
web
• Problem areas
SD'98 (c) David Strom, Inc.
24
HITS
• (How Idiots Track Success)
• Nobody uses this word anymore
• Doesn’t really measure individual users,
just access
• Catching servers and proxies mess up
these statistics
SD'98 (c) David Strom, Inc.
25
Domain origins
• Where users are coming from -sometimes
• Just because they are from ibm.net
doesn’t mean they work at IBM!
• Forgotten accounts, friends and family
using the account
• Hacked user names
• Proxies don’t help here either
SD'98 (c) David Strom, Inc.
26
The path people take in and
around your web
• Search engines help sometimes
• Which search site was the most popular
front door
• Who links to you and why
• Is there a pattern or a random walk?
SD'98 (c) David Strom, Inc.
27
Problem areas to deal with
• Broken links (locally)
• Broken outbound links
• Time outs (sunspots?)
SD'98 (c) David Strom, Inc.
28
What you can’t learn from your
logs
• Who are these people, anyway?
– No specific user names
– Is it a bot or a real human?
• How long did they view a page?
– Most people don’t spend much time on your
web
– Where did they go visit next?
SD'98 (c) David Strom, Inc.
29
What technologies are
available?
•
•
•
•
•
Built-in analyzer tools
Sites that capture user info
Secure sites with registration
Build your own from perl
Third-party tools
SD'98 (c) David Strom, Inc.
30
Built-in tools
•
•
•
•
WebSite, website.ora.com
IIS with Site Server, www.microsoft.com/iis
Netscape servers, www.netscape.com
Easy to use but limited
SD'98 (c) David Strom, Inc.
31
WebSite Professional v2
• Win NT, 95
• Best web server for learning about logs,
best docs
• QuickStats module for instant analysis:
– single report but nice set of information
– shows today, last two days requests and
unique hosts
– IP addresses of visitors, average
requests/hour
SD'98 (c) David Strom, Inc.
32
IIS Site Server
• NT Server v4 w/SP3 only
• Lots of preconfigured reports
• Two versions, Express and Full
(customized reports)
• backoffice.microsoft.com/products/siteserv
er/express/
SD'98 (c) David Strom, Inc.
33
Netscape v3 web servers
• Various NT, Unix versions
• Reports for a few variables but nothing too
extensive
• Best to use a third-party tool here
SD'98 (c) David Strom, Inc.
34
Sites that capture user info
• WebCounter, www.digits.com -- third-party
hit counter
• Someone else does the programming and
debugging
• But beyond your control
SD'98 (c) David Strom, Inc.
35
Secure sites with registration
• You know your users
• But many won’t register, or forget their
passwords
• Requires scripting, database integration,
more maintenance
SD'98 (c) David Strom, Inc.
36
Build your own from perl
• Needs some in-house support
• Works best with Unix-based webs
• Examples:
– refstats,
members.aol.com/htmlguru/refstats.html
– surfreport, bienlogic.com/SurfReport/
SD'98 (c) David Strom, Inc.
37
Third-party tools
•
•
•
•
•
WebTracker, www.CQMInc.com/webtrack
WebTrends, www.webtrends.com
net.Genesis, www.netgen.com
MarketWave, www.marketwave.com
IIS Assistant, www.go-iis.com
SD'98 (c) David Strom, Inc.
38
Third-party tools (con’t)
• Can make very pretty reports
• Customizable
• Make sure they support your particular log
format
• Not that expensive, mostly run on
Windows
SD'98 (c) David Strom, Inc.
39