SiLK Installation
A Walk-through
Joe McManus
joe@cert.org
© 2007 Carnegie Mellon University
Topics
Topics Covered in this Training
• What SiLK is and is not
• SiLK on a Box
• SiLK with remote flow collection
• Building SiLK RPMs
• Monitoring SiLK Processes
• Basic SiLK Queries
—
Useful queries for an administrator
© 2007 Carnegie Mellon University
2
Introduction to SiLK
System for internet Level Knowledge
Data
collection
Data
analysis
NOT:
Intrusion
Audit
detection/prevention system
tools for your network
Automated
report generator
© 2007 Carnegie Mellon University
3
Collection & Analysis Infrastructure
SiLK stores the data and allows access to the data
for analysis.
YAF turns packets into flows for SiLK over IPFIX.
SiLK can take input from IPFIX or Netflow.
Netflow is convenient as it implemented on many
routers and requires no additional hardware.
© 2007 Carnegie Mellon University
4
Collection & Analysis Infrastructure
© 2007 Carnegie Mellon University
5
SiLK on a Box
Standalone flow collection
© 2007 Carnegie Mellon University
6
Standalone Collection and Analysis
We will now go over installing SiLK and YAF to
create a stand alone flow collection and analysis box.
We will assume you are on a linux server with root
access.
© 2007 Carnegie Mellon University
7
Downloading SiLK
SiLK can be downloaded from:
http://tools.netsa.cert.org/silk/
YAF is available from:
http://tools.netsa.cert.org/yaf/
Fixbuf is available from:
http://tools.netsa.cert.org/fixbuf/
© 2007 Carnegie Mellon University
8
Downloading (continued…)
Log in to your Linux server and use wget to download
the software.
[root@silk tmp]#cd /tmp
[root@silk tmp]# wget \
http://tools.netsa.cert.org/releases/silk-1.1.3.tar.gz
[root@silk tmp]# wget \
http://tools.netsa.cert.org/releases/libfixbuf-0.8.0.tar.gz
[root@silk tmp]# wget \
http://tools.netsa.cert.org/releases/yaf-1.0.0.tar.gz
© 2007 Carnegie Mellon University
9
Build fixbuf
Fixbuf is required to have YAF talk to SiLK over
IPFIX.
[root@silk tmp]# tar -zxvf libfixbuf-0.8.0.tar.gz
[root@silk tmp]# cd libfixbuf-0.8.0
[root@silk libfixbuf-0.8.0]# ./configure && make
&& make install
© 2007 Carnegie Mellon University
10
Errors
checking for cc... no
checking for cl... no
configure: error: no acceptable C compiler found
in $PATH
We need to make sure the system has the required
packages to build the suite.
© 2007 Carnegie Mellon University
11
YUM
We can use YUM to install the missing C compiler.
[root@silk libfixbuf-0.8.0]# yum install gcc
Try building Fixbuf again.
Which brings us to the next error:
checking whether we are using the GNU C++
compiler... no
checking whether g++ accepts -g... no
checking dependency style of g++... none
© 2007 Carnegie Mellon University
12
YUM
If you do not know which package you are looking for
YUM can help with this.
[root@silk libfixbuf-0.8.0]#
yum search c++
[root@silk libfixbuf-0.8.0]# yum install gcc-c++
© 2007 Carnegie Mellon University
13
RPM Confusion
[root@silk libfixbuf-0.8.0]# ./configure
checking for GLIB - version >= 2.4.7... no
*** Could not run GLIB test program, checking why...
*** The test program failed to compile or link. See the file
config.log for the
*** exact error that occured. This usually means GLIB is
incorrectly installed.
configure: error: Cannot find a suitable glib2 (>= 2.4.7)
This error is misleading.
© 2007 Carnegie Mellon University
14
RPM Confusion
[root@silk libfixbuf-0.8.0]# yum install glib2
Loaded plugins: refresh-packagekit
Setting up Install Process
Parsing package install arguments
Package glib2-2.16.6-2.fc9.i386 already installed and
latest version
Nothing to do
[root@silk ~]# rpm -qa | grep
glib2-2.16.6-2.fc9.i386
glib2
Wuh?!?
© 2007 Carnegie Mellon University
15
RPM Confusion
What is actually missing is the glib2-devel rpm.
[root@silk ~]# yum install glib2-devel
You may have to install the libpcap-devel rpm as
well. This is dependent on which install of linux you
have done.
© 2007 Carnegie Mellon University
16
Fixbuf install
[root@silk libfixbuf-0.8.0]# ./configure
[root@silk libfixbuf-0.8.0]# make
[root@silk libfixbuf-0.8.0]# make install
[root@silk libfixbuf-0.8.0]# ls /usr/local/lib
libfixbuf-0.8.0.so.8
libfixbuf.a
libfixbuf.so
libfixbuf-0.8.0.so.8.0.0
libfixbuf.la
pkgconfig
© 2007 Carnegie Mellon University
17
YAF Install
YAF is our tool for flow collection, there are others.
[root@silk libfixbuf-0.8.0]# cd ..
[root@silk tmp]# tar -zxvf yaf-1.0.0.tar.gz
[root@silk tmp]# cd yaf-1.0.0
checking for libfixbuf >= 0.7.2... configure:
error: Cannot find a suitable libfixbuf (>=
0.7.2): Package libfixbuf was not found in the
pkg-config search path.
Perhaps you should add the directory containing
`libfixbuf.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libfixbuf' found
© 2007 Carnegie Mellon University
18
YAF Install
Didn’t we just install fixbuf?
-Yes, but pkgconfig does not know to look in
/usr/local.
[root@silk yaf-1.0.0]# export \
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
[root@silk yaf-1.0.0]# ./configure
[root@silk yaf-1.0.0]# make
[root@silk yaf-1.0.0]# make install
© 2007 Carnegie Mellon University
19
SiLK Install
Uncompress the SiLK tarball.
[root@silk yaf-1.0.0]# cd ..
[root@silk tmp]# tar -zxvf silk-1.1.3.tar.gz
[root@silk tmp]# cd silk-1.1.3
We will build in support for PySiLK.
[root@silk silk-1.1.3]# yum install python-devel
© 2007 Carnegie Mellon University
20
SiLK Install (cont…)
[root@silk silk-1.1.3]# ./configure
\
--withlibfixbuf=/usr/local/lib/pkgconfig/ \
--with-python
SiLK creates a great summary.
© 2007 Carnegie Mellon University
21
Summary
* Configured package:
* Host type:
SiLK 1.1.3
i686-pc-linux-gnu
* Source files ($top_srcdir): .
* Install directory:
/usr/local
* Root of packed data tree: /data
* Packing logic:
via run-time plugin
* Timezone support:
UTC
* Default compression method: SK_COMPMETHOD_NONE
* IPv6 support:
NO
* IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lgthread-2.0-lrt -lglib-2.0)
* Transport encryption support: NO (gnutls not found)
* IPA support:
NO
* LIBPCAP support:
YES (-lpcap)
* Python support:
YES (-L/usr/kerberos/lib -Xlinker -export-dynamic -ld -lutil -lm -L/usr/lib -lpython2.5 -pthread)
*Python package destination: /usr/lib/python2.5/site-packages
* Build analysis tools:
YES
* Build packing tools:
YES
* Compiler (CC):
gcc
* Compiler flags (CFLAGS):
-I$(top_srcdir)/src/include -DNDEBUG -D_GNU_SOURCE=1
-D_FILE_OFFSET_BITS=64 -O3 -fno-strict-aliasing -Wall -W -Wmissing-prototypes -Wformat=2
-Wdeclaration-after-statement
* Linker flags (LDFLAGS):
* Libraries (LIBS):
-ldl -lm
© 2007 Carnegie Mellon University
22
SiLK Install (cont…)
[root@silk silk-1.1.3]# make
[root@silk silk-1.1.3]# make install
[root@silk silk-1.1.3]# rwfilter --version
rwfilter: part of SiLK 1.1.3;
configuration settings:
*
*
*
*
*
*
*
*
*
Root of packed data tree:
Packing logic:
Timezone support:
Available compression methods:
IPv6 support:
IPFIX collection support:
Transport encryption:
PySiLK support:
Enable assert():
/data
Run-time plug-in
UTC
none [default]
no
yes
no
/usr/lib/python2.5/site-packages
no
© 2007 Carnegie Mellon University
23
SiLK Configuration
We need to create a place to store the flow data.
The default for is /data.
[root@silk ~]# mkdir /data
We will need to create two configuration files for
SiLK, sensor.conf and silk.conf
© 2007 Carnegie Mellon University
24
silk.conf
A few sample silk.conf files are included with the
distribution. They are installed under
/usr/local/share/silk.
We would want to use twoway-silk.conf
[root@silk /data]# cp \
/usr/local/share/silk/twoway -silk.conf .
© 2007 Carnegie Mellon University
25
silk.conf
[root@silk data]# vi /data/silk.conf
#Define Sensors
sensor 0 localhost
class all
sensors localhost
end class
version 1
© 2007 Carnegie Mellon University
26
SiLK Configuration
class all
type 0 in
in
type 1 out out
type 2 inweb iw
type 3 outweb ow
type 4 innull innull
type 5 outnull outnull
type 6 int2int int2int
type 7 ext2ext ext2ext
type 8 inicmp inicmp
type 9 outicmp outicmp
type 10 other other
default-types in inweb inicmp
end class
© 2007 Carnegie Mellon University
27
silk.conf storage heirarchy
© 2007 Carnegie Mellon University
28
silk.conf storage hierarchy.
I find it good practice to have silk store its data under
/data/SENSOR-NAME/type.
The benefit of storing data this way is that you can
use unix groups to control access to flow data on
each sensor.
i.e.
/data/engineering
/data/sales
/data/logistics
© 2007 Carnegie Mellon University
29
silk.conf
# The default path format from SILK_DATA_ROOTDIR
path-format "%N/%T/%Y/%m/%d/%x“
%N= Sensor Name
%T=Type In/Out/int2int…
%Y=Year
%m=month
%d=data
%x=flowtype-sensor_YearMonthDay.Hour
© 2007 Carnegie Mellon University
30
silk.conf
# The plug-in to load to get the packing logic to use in rwflowpack.
# The --packing-logic switch to rwflowpack will override this value.
# If SiLK was configured with hard-coded packing logic, this value is
# ignored.
# The plug-in to load to get the packing logic to use in rwflowpack.
# The --packing-logic switch to rwflowpack will override this value.
# If SiLK was configured with hard-coded packing logic, this value is
# ignored.
packing-logic "packlogic-twoway.so“
© 2007 Carnegie Mellon University
31
sensor.conf
We will now have to edit the sensor.conf file.
[root@silk data]# vi /data/sensor.conf
probe localhost ipfix
listen-on-port 18001
protocol tcp
accept-from-host 127.0.0.1
end probe
© 2007 Carnegie Mellon University
32
sensor.conf
sensor localhost
ipfix-probes localhost
internal-ipblock 192.168.1.0/24
external-ipblock remainder
end sensor
© 2007 Carnegie Mellon University
33
rwflowpack
rwflowpack is the daemon that will listen for traffic
from YAF.
A sample file is included, which we will customize.
[root@silk]# cp \
/usr/local/share/silk/etc/rwflowpack.conf \
/usr/local/etc/.
[root@silk data]# vi /usr/local/etc/rwflowpack.conf
© 2007 Carnegie Mellon University
34
rwflowpack.conf
Change the following values:
ENABLED=yes
SENSOR_CONFIG=/data/sensor.conf
SITE_CONFIG=/data/silk.conf
LOG_TYPE=legacy
LOG_DIR=/var/log
CREATE_DIRECTORIES=yes
COMPRESSION_TYPE=best
© 2007 Carnegie Mellon University
35
rwflowpack.conf
To make rwflowpack start on boot we can add it
using chkconfig. SiLK includes sample startup scripts
with the distribution.
[root@silk data]# cp
/usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d
[root@silk data]# chkconfig rwflowpack on
[root@silk data]# chkconfig --list rwflowpack
rwflowpack
0:off
1:off
2:on
3:on
4:on
5:on
6:off
© 2007 Carnegie Mellon University
36
Test YAF
Test that YAF is able to listen on the interface.
[root@silk data]# yaf --live=pcap --in=eth0 --out=- -v
| yafscii
[2009-01-08 17:06:10] yaf starting
[2009-01-08 17:06:10] running as root in --live mode,
but not dropping privilege
© 2007 Carnegie Mellon University
37
Test YAF
Let that run for a minute. Then enter CTRL-C.
C2009-01-08 17:07:19.900 tcp 10.0.0.2:1005 => 10.0.0.3:2049
90b24967:0b0cbdd9 AF/A:AF/0 (2/104 <-> 1/52) rtt 0 ms
[2009-01-08 17:07:42] Processed 57 packets into 9 flows:
[2009-01-08 17:07:42]
Mean flow rate 0.18/s.
[2009-01-08 17:07:42]
[2009-01-08 17:07:42]
Mean packet rate 1.11/s.
Virtual bandwidth 0.0008 Mbps.
[2009-01-08 17:07:42]
[2009-01-08 17:07:42]
Maximum flow table size 8.
6 flush events.
[2009-01-08 17:07:42] Rejected 1 out-of-sequence packets.
[2009-01-08 17:07:42] Assembled 0 fragments into 0 packets:
[2009-01-08 17:07:42]
[2009-01-08 17:07:42]
Expired 0 incomplete fragmented packets.
Maximum fragment table size 0.
[2009-01-08 17:07:42] Rejected 44 packets during decode:
[2009-01-08 17:07:42]
44 due to unsupported/rejected packet type:
[2009-01-08 17:07:42]
44 unsupported/rejected Layer 3 headers.
[2009-01-08 17:07:42] yaf terminating
© 2007 Carnegie Mellon University
38
IPTables
You must open the firewall for YAF to connect to
rwflowpack.
[root@silk data]# iptables -I INPUT \
-s 127.0.0.1 -p tcp -m tcp \
--dport 18001 -j ACCEPT
[root@silk data]# service iptables save
iptables: Saving firewall rules to
/etc/sysconfig/iptables:[ OK ]
© 2007 Carnegie Mellon University
39
IPTables
We can check to see if the rule has been added
correctly.
[root@silk data]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source
destination
1 ACCEPT tcp -- 127.0.0.1
0.0.0.0/0
tcp dpt:18001
….
© 2007 Carnegie Mellon University
40
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
© 2007 Carnegie Mellon University
41
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
The silk option tells YAF to format the output for
silk.
© 2007 Carnegie Mellon University
42
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
The ipfix=tcp option tells YAF use ipfix over tcp.
The ipfix-port=18001 option specifies which
port.
© 2007 Carnegie Mellon University
43
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
The live=pcap option tells YAF the capture device
uses pcap.
The only other option is dag for use with Endace Dag
cards.
© 2007 Carnegie Mellon University
44
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
The in=eth0 option tells YAF which input device to
use.
If you are using a dag card, the option would be
dag0.
© 2007 Carnegie Mellon University
45
Starting YAF
[root@silk data]# yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 --ipfixport=18001 &
The out=127.0.0.1 option tells YAF where to send
its output.
© 2007 Carnegie Mellon University
46
Checking on YAF
You can check the status of YAF with the following
command:
[root@silk data]# kill -SIGUSR1 `pgrep yaf`
[2009-01-08 21:53:29] Rejected 1 out-of-sequence
packets.
© 2007 Carnegie Mellon University
47
Checking on The Collection Process
When you first start yaf one of the first places to look
is in the log file under /var/log.
[root@silk data]# tail /var/log/rwflowpack20090108.log
Jan 8 16:53:40 silk rwflowpack[10605]:
Flushing files after 120 seconds.
Jan 8 16:54:20 silk rwflowpack[10605]:
Opening new output file
/data/localhost/ext2ext/2009/01/08/ext2extlocalhost_20090108.21
© 2007 Carnegie Mellon University
48
Checking on The Collection Process
Did yaf start correctly?
[root@silk data]# ps -ef | grep yaf
root
10630 10475 0 16:52 pts/1
00:00:00 yaf --silk --ipfix=tcp -live=pcap --in=eth0 --out=127.0.0.1 -ipfix-port=18001
© 2007 Carnegie Mellon University
49
Checking on The Collection Process
If not can yaf connect to rwflowpack?
[root@silk data]# telnet localhost
18001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Is rwflowpack running?
[root@silk data]# ps -ef | grep yaf
© 2007 Carnegie Mellon University
50
Try it out
Lets try out a simple rwfilter command to see if it is
working:
[root@silk data]# rwfilter --type=all --proto=0- --pass=stdout | rwcut | head
sIP|
dIP|sPort|dPort|pro|
packets|
10.0.0.25|
10.0.0.6|43393|
22| 6|
localhost|
10.0.0.6|
10.0.0.25|
22|43393| 6|
0.023|2009/01/08T21:53:25.812|localhost|
10.0.0.6|
10.0.0.25|
22|43393| 6|
0.000|2009/01/08T21:53:25.812|localhost|
10.0.0.25|
10.0.0.6|43393|
22| 6|
0.000|2009/01/08T21:53:25.812|localhost|
bytes|
6|
flags|
332|FSRPA
sTime|
dur|
eTime|
sensor|
|2009/01/08T21:53:25.789|
0.023|2009/01/08T21:53:25.812
4|
972| S PA
|2009/01/08T21:53:25.789|
1|
52|F
1|
40|
A
R
|2009/01/08T21:53:25.812|
|2009/01/08T21:53:25.812|
© 2007 Carnegie Mellon University
51
Routers
Using routers as a data feed
© 2007 Carnegie Mellon University
52
Receiving data from a router
Another common way of receiving data is from a
router on a mirror port or span port.
In Cisco IOS this is called a span port and if very
easy to configure.
© 2007 Carnegie Mellon University
53
Span Port
IOS Example:
interface FastEthernet0/24
port monitor FastEthernet0/1
port monitor FastEthernet0/2
…..
port monitor FastEthernet0/23
port monitor VLAN1
© 2007 Carnegie Mellon University
54
YAF & Span port
You would set up sensor.conf with the same options
as in the previous example. Although you would
want to have a second interface on the box listening
for it set up in promiscuous mode with arp disabled
as well.
Yaf would also start with the same options as before,
but pointing to your second nic.
© 2007 Carnegie Mellon University
55
SiLK & Netflow
Silk can take netflow directly from a netflow
compatible router.
You would have to open your firewall to accept UDP
on the port you decide to use and make changes to
sensor.conf as show in the next slide.
© 2007 Carnegie Mellon University
56
SiLK & Netflow
probe S2 netflow-v5
listen-on-port 9901
protocol udp accept-from-host
172.16.22.22
end probe
sensor S2
netflow-v5-probes S2
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
© 2007 Carnegie Mellon University
57
YAF & DAG Cards
YAF is capable of including support for Endace DAG
Cards.
It is it as simple as adding -with-dag option to
./configure.
When YAF is started you would use the following
options:
[root@silk data]# yaf--silk --ipfix=tcp -live=dag --in=dag0 --out=127.0.0.1 -ipfix-port=18001
© 2007 Carnegie Mellon University
58
Working With Taps
Tunnelling
© 2007 Carnegie Mellon University
59
Working With Taps
© 2007 Carnegie Mellon University
60
Tunneling
YAF can only listen on one interface at a time. If we
are using YAF with a tap one interface will receive
inbound and one would receive outbound traffic. For
some programs like snort this is useful.
But with SiLK it classifies traffic by CIDR block in
sensor.conf.
We can use port bonding on Linux to solve this issue.
© 2007 Carnegie Mellon University
61
Tunneling
Fedora 9 works fine with the Trendnet usb ethernet
adapter available at Radio Shack.
http://www.radioshack.com/product/index.jsp?product
Id=2806154
© 2007 Carnegie Mellon University
62
Tunneling
Plug in both adapters and run dmesg looking for the
mac address to use in the ifcfg-ethX file.
[root@silk ~]# dmesg | tail -20
usb 1-8: new high speed USB device using ehci_hcd and address 5
usb 1-8: configuration #1 chosen from 1 choice
usb 1-8: New USB device found, idVendor=0b95, idProduct=7720
usb 1-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-8: Product: AX88772
eth1: register 'asix' at usb-0000:00:1d.7-8, ASIX AX88772 USB 2.0 Ethernet,
00:50:b6:04:47:11
usbcore: registered new interface driver asix
usb 1-7: new high speed USB device using ehci_hcd and address 6
usb 1-7: configuration #1 chosen from 1 choice
eth2: register 'asix' at usb-0000:00:1d.7-7, ASIX AX88772 USB 2.0 Ethernet,
00:50:b6:04:46:fc
usb 1-7: New USB device found, idVendor=0b95, idProduct=7720
usb 1-7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-7: Product: AX88772
© 2007 Carnegie Mellon University
63
Tunneling
[root@silk ~]# cd /etc/sysconfig/network-scripts/
[root@silk network-scripts]# vi ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
ONBOOT=yes
PROMISC=yes
USERCTL=no
PEERDNS=no
© 2007 Carnegie Mellon University
64
Tunneling
[root@silk network-scripts]# vi ifcfg-eth1
DEVICE=eth1
HWADDR=00:50:b6:04:67:11
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
© 2007 Carnegie Mellon University
65
Tunneling
[root@silk network-scripts]# vi ifcfg-eth2
DEVICE=eth2
HWADDR=00:50:b6:04:67:12
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
© 2007 Carnegie Mellon University
66
Tunneling
You need to load the bonding module:
[root@silk ~]# vi /etc/modprobe.conf
alias bond0 bonding
Next you can start the bonded interface:
[root@silk network-scripts]# ifup bond0
© 2007 Carnegie Mellon University
67
Tunneling
On some systems the PROMISC=yes option in ifcfgbond0 is ignored. If it is then editing /etc/rc.d/rc.local
and adding it there can help:
[root@silk data]# vi /etc/rc.d/rc.local
/sbin/ifconfig bond0 promisc –arp up
© 2007 Carnegie Mellon University
68
RPMs
Standardizing installs
© 2007 Carnegie Mellon University
69
RPMs
RPMs make management of software easier.
In some situations you may have remote sensors and
may not want to include compilers on them, RPMs
make installing software easier.
Our software generates .spec files for RPMs during
the ./configure process.
© 2007 Carnegie Mellon University
70
RPM
You first need to create a .rpmmacros file.
[root@silk libfixbuf-0.8.0]# vi ~/.rpmmacros
%packager
%vendor
%_topdir
joe mcmanus <joe@cert.org>
cert.org
/tmp/rpms
© 2007 Carnegie Mellon University
71
RPM
Create the build heirarchy.
[root@silk
[root@silk
[root@silk
[root@silk
[root@silk
[root@silk
tmp]#
tmp]#
tmp]#
tmp]#
tmp]#
tmp]#
mkdir
mkdir
mkdir
mkdir
mkdir
mkdir
/tmp/rpms
/tmp/rpms/SPECS
/tmp/rpms/SOURCES
/tmp/rpms/SRPMS
/tmp/rpms/RPMS
/tmp/rpms/BUILD
© 2007 Carnegie Mellon University
72
RPM
Install the rpms to build rpms.
[root@silk tmp]# yum install rpm-build
Clean up our previous installs
[root@silk
[root@silk
uninstall
[root@silk
[root@silk
uninstall;
[root@silk
uninstall;
tmp]# cd libfixbuf-0.8.0
libfixbuf-0.8.0]# make
libfixbuf-0.8.0]# cd ..
tmp]# cd yaf-1.0.0; make
cd ..
tmp]# cd silk-1.1.3; make
cd ..
© 2007 Carnegie Mellon University
73
RPM-fixbuf
Prepare the fixbuf rpm
[root@silk tmp]# cd libfixbuf-0.8.0
[root@silk libfixbuf-0.8.0]# make clean
[root@silk libfixbuf-0.8.0]# ./configure
[root@silk libfixbuf-0.8.0]# cp libfixbuf.spec
/tmp/rpms/SPECS/
[root@silk libfixbuf-0.8.0]# cd /tmp/rpms/SPECS/
[root@silk SPECS]# cp /tmp/libfixbuf-0.8.0.tar.gz
/tmp/rpms/SOURCES/
[root@silk SPECS]# rpmbuild -ba libfixbuf.spec
© 2007 Carnegie Mellon University
74
RPM-fixbuf
Now that you have built the rpms, install them.
[root@silk SPECS]# rpm -iv
../RPMS/i386/libfixbuf-*.rpm
Preparing packages for installation...
libfixbuf-0.8.0-1
libfixbuf-devel-0.8.0
We have to install the devel package to build the yaf
package.
You would not need devel in production.
© 2007 Carnegie Mellon University
75
RPM -YAF
[root@silk SPECS]# cp /tmp/yaf-1.0.0.tar.gz
/tmp/rpms/SOURCES/
[root@silk SPECS]# cp /tmp/yaf-1.0.0/yaf.spec
[root@silk SPECS]# rpmbuild -ba yaf.spec
© 2007 Carnegie Mellon University
76
RPM-YAF
Install it.
[root@silk SPECS]# rpm -ivh ../RPMS/i386/yaf-1.0.01.i386.rpm
Preparing...
########################################### [100%]
file /usr/lib/libltdl.so.3 from install of yaf-1.0.01.i386 conflicts with file from package libtool-ltdl1.5.24-6.fc9.i386
YAF uses autoconf to build and it can’t find the libtool
headers so it tries to install its own version of libtool.
© 2007 Carnegie Mellon University
77
RPM-YAF
What we need to install is libtool-ltdl-devel and edit
yaf.spec.
[root@silk SPECS]# vi yaf.spec
<go to the %build section append this to the end>
--enable-ltdl-install=no
[root@silk SPECS]# rpmbuild -ba yaf.spec
[root@silk SPECS]#
1.0.0-1.i386.rpm
rpm -iv ../RPMS/i386/yaf-
© 2007 Carnegie Mellon University
78
RPM-SiLK
We now need to build the SiLK rpms. To do this we
must first install flex and bison.
root@silk SPECS]# yum install -y flex bison
[root@silk SPECS]# cp /tmp/silk-1.1.3/silk.spec .
[root@silk SPECS]# cp /tmp/silk-1.1.3.tar.gz
../SOURCES/
[root@silk SPECS]# rpmbuild -ba silk.spec
© 2007 Carnegie Mellon University
79
RPM-SiLK
Silk builds many rpms. This gives you the option to
just install parts of SiLK on one host, i.e. just the
collector, and the full analysis suite on another.
[root@silk i386]# rpm -ivh silk-common-1.1.3-1.i386.rpm \
silk-analysis-1.1.3-1.i386.rpm \
silk-rwflowpack-1.1.3-1.i386.rpm
© 2007 Carnegie Mellon University
80
RPM-SiLK
There are a few differences with the SiLK rpm install.
Installs
rwflowpack.conf /etc/sysconfig/
 I create a link /etc/rwflowpack.conf
 [root@silk i386]# ln -s /etc/sysconfig/rwflowpack.conf
/etc/rwflowpack.conf
It
copies rwflowpack to init.d for you.
© 2007 Carnegie Mellon University
81
RPM-Yaf
YAF comes with a sample startup script called
Airdaemon. I use my own.
[root@silk ~]# vi /etc/init.d/yaf
<See sample on wiki>
[root@silk ~]# chkconfig --add yaf
[root@silk ~]# service yaf start
© 2007 Carnegie Mellon University
82
Monitoring
© 2007 Carnegie Mellon University
83
rwfilter
Top twenty talkers.
[root@silk ~]$rwfilter --proto=0- --type=all -pass=stdout | rwstats --top --flows --count 10 --sip
INPUT SIZE: 2062 records for 66 unique keys
SOURCE IP Key: Top 10 flow counts
sIP|
Records|%_of_total|
cumul_%|
192.168.1.10|
1038| 50.339476| 50.339476|
192.168.1.223|
624| 30.261882| 80.601358|
192.168.1.108|
209| 10.135790| 90.737148|
68.180.131.16|
19|
0.921435| 91.658584|
208.44.108.137|
16|
0.775946| 92.434530|
204.74.66.247|
11|
0.533463| 92.967992|
12.183.125.5|
9|
0.436469| 93.404462|
69.63.176.9|
9|
0.436469| 93.840931|
63.247.72.26|
7|
0.339476| 94.180407|
207.242.93.120|
7|
0.339476| 94.519884|
© 2007 Carnegie Mellon University
84
rwfilter
Top 20 Machines by bytes and source ip.
[root@silk ~]$ rwfilter --sensor=localhost --type=all -proto=0- --pass=stdout | rwstats --top --bytes -count=20 --sip
sIP|
192.168.1.15|
Bytes|%_of_total|
cumul_%|
136107978| 92.600924| 92.400924|
192.168.1.115|
5556727|
3.772368| 96.173272|
192.168.1.158|
6428084|
3.006136| 99.179408|
192.168.1.113|
682127|
0.327306| 99.506714|
192.168.1.115|
600827|
0.272113| 99.778827|
© 2007 Carnegie Mellon University
85
rwfilter
Top 20 machines by destination/incoming.
[joe@silk ~]$ rwfilter --sensor=localhost --type=all -proto=0- --pass=stdout | rwstats --top --bytes -count=20 --dip
INPUT SIZE: 9195 records for 255 unique keys
DESTINATION IP Key: Top 20 byte counts
dIP|
Bytes|%_of_total|
192.168.1.158|
80622155| 54.402049|
192.168.1.115|
56285305| 36.721620|
192.168.1.15|
11288271| 7.636019|
192.168.1.113|
1513238| 0.685611|
192.168.1.115|
611659| 0.278469|
192.168.1.255|
163238| 0.096894|
192.168.1.223|
116826| 0.077675|
226.0.0.251|
35627| 0.024150|
cumul_%|
54.402049|
91.123669|
98.759687|
99.445098|
99.723568|
99.820462|
99.898137|
99.922237|
© 2007 Carnegie Mellon University
86
rwfilter
[joe@silk ~]$ rwfilter --sensor=localhost --type=all -proto=0- --pass=stdout --daddress=192.168.1.108 | rwsort -fields=bytes | rwsort --fields=bytes| rwcut -fields=sip,dip,dport,sport,dur| tail
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.10|
192.168.1.108|62168|
192.168.1.108|62418|
192.168.1.108|62419|
192.168.1.108|62420|
192.168.1.108|62404|
192.168.1.108|62403|
192.168.1.108|62402|
192.168.1.108|56022|
192.168.1.108|62086|
192.168.1.108|62086|
139|
0.278|
139|
0.321|
139|
0.304|
139|
0.295|
139|
0.467|
139|
0.472|
139|
0.520|
53| 513.045|
139| 1755.572|
139| 483.900|
© 2007 Carnegie Mellon University
87
rwfilter
As an administrator you may want to check for
scanners or set a baseline for expected amount of
traffic for each day and check to see if your total
traffic is % above on a day and look for the root
cause –
youtube
netflix streaming
“slashdotting”
© 2007 Carnegie Mellon University
88
Monitoring
A simple ps type script can monitor yaf.
#!/bin/bash
mailto=joe@cert.org
proccount=“1”
proc=“yaf”
running=`ps –ef | grep yaf | grep –c –v grep
>/dev/null`
If [ “$running” –lt “$proccount” ]
then
mail –s”$proc not running `date`” $mailto <<EOF
Less than $proccount $proc running on $HOSTNAME at
‘date`
Message created by $0
EOF
fi
© 2007 Carnegie Mellon University
89
Monitoring
The same script can be used for rwflowpack.
But what about if it is running and a link in the chain
has broken and no data is being stored?
We can use rwfilter to look for flows created in the
last 30 minute period, we could tail the
rwflowpack.log and look for new files being written.
© 2007 Carnegie Mellon University
90
Monitoring
This example will use rwfilter to check for flows
created in the last hour.
now=`date –utc +%Y/%m/%d:%H`
for loop in `ls -F /data/ | grep \/ | sed ‘s/\///’`
do
flowcount=` rwfilter –sensor=$loop \
–start-date=$now \
--type=all –proto=0- \
| rwcut | wc –l `
If [ “$flowcount “ –eq “0” ]
then
mail –s”Error: $loop has no records” foo@bar.com
fi
© 2007 Carnegie Mellon University
91
Wrap up
© 2007 Carnegie Mellon University
92
Going Forward
This system we set up works nicely. But it can be
improved. We can run yaf, rwflowcap and rwsender
on the sensor and then rwreceiver and rwflowpack on
the packing machine.
Now if yaf cannot connect to rwreciever it stops, data
would be lost.
© 2007 Carnegie Mellon University
93
Questions?
Any questions?
Email joe@cert.org with any questions you have.
© 2007 Carnegie Mellon University
94