Building Collector Plugins
Admin Guide
Copyright © Alienvault 2010
All rights reserved. No part of this work may be reproduced or transmitted in any form or by
any means, electronic or mechanical, including photocopying, recording, or by any
information storage or retrieval system, without the prior written permission of the
copyright owner and publisher.
Any trademarks referenced herein are the property of their respective holders.
Building Collector Plugins - Admin Guide
Table of Content
1
Overview..................................................................................................................................................... 4
1.1
1.1.1
Event Collection .......................................................................................................................... 4
1.1.2
Event Normalization ................................................................................................................... 4
1.2
OSSIM Server Role .............................................................................................................................. 6
1.2.1
Event Enrichment ....................................................................................................................... 6
1.2.2
Policies and Actions .................................................................................................................... 7
1.3
2
OSSIM Agent Role ............................................................................................................................... 4
The Configuration Workflow .............................................................................................................. 8
Configuring Detector Plugins ....................................................................................................................10
2.1
Rsyslog ..............................................................................................................................................10
2.1.1
Configuration File .....................................................................................................................10
2.1.2
Listener Configuration ..............................................................................................................10
2.1.3
Filters ........................................................................................................................................10
2.2
OSSIM Agent Configuration ..............................................................................................................11
2.2.1
Configuration File .....................................................................................................................11
2.2.2
Parameters ...............................................................................................................................11
2.3
Detector Plugin Configuration ..........................................................................................................13
2.3.1
Configuration Files ....................................................................................................................13
2.3.2
Common Event Types ...............................................................................................................13
2.3.3
Parameters ...............................................................................................................................13
2.3.4
Using Local (Plugin) Variables...................................................................................................15
2.3.5
Using Global (Agent) Variables .................................................................................................15
2.4
Aliases ...............................................................................................................................................16
2.4.1
Path...........................................................................................................................................16
2.4.2
Predefined Regular Expressions ...............................................................................................16
2.5
Functions .......................................................................................................................................... 16
2.5.1
Path...........................................................................................................................................16
2.5.2
Conversions ..............................................................................................................................16
2.5.3
Application Specific Translations ..............................................................................................17
2.5.4
User Defined Translations ........................................................................................................17
2.6
Event Fields....................................................................................................................................... 18
2.7
Rules .................................................................................................................................................19
2.7.1
Page 2
Evaluation Order.......................................................................................................................19
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.7.2
2.8
Structure ...................................................................................................................................19
Loading Plugins .................................................................................................................................21
2.8.1
Priority and Reliability values ...................................................................................................21
2.8.2
SQL Statement ..........................................................................................................................21
2.9
Plugin Activation ...............................................................................................................................22
2.9.1
Activate the Plugin on the Server Side .....................................................................................22
2.9.2
Activate the Plugin on the Agent Side ......................................................................................22
3
Log files .....................................................................................................................................................22
4
Debugging .................................................................................................................................................22
5
Appendix...................................................................................................................................................23
5.1
Regular Expressions ..........................................................................................................................23
5.2
Configuration Example .....................................................................................................................25
5.2.1
Scenario ....................................................................................................................................25
5.2.2
Write a script to monitor the “last” status ...............................................................................25
5.2.3
Log sample ................................................................................................................................25
5.2.4
Collect the logs in a new log file ...............................................................................................25
5.2.5
Restart “rsyslog” .......................................................................................................................26
5.2.6
Check whether the new entries are written in the new log file...............................................26
5.2.7
Create a plugin file....................................................................................................................26
5.2.8
Register the Plugin with the OSSIM Agent ...............................................................................29
5.2.9
Register the Plugin with the OSSIM Server ..............................................................................30
5.2.10
Check whether the plugin was successfully registered ............................................................31
5.2.11
Restart the OSSIM Server .........................................................................................................31
5.2.12
Restart the OSSIM Agent ..........................................................................................................31
5.2.13
Check whether Events and Alarms are received ......................................................................32
Page 3
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
1 Overview
1.1 OSSIM Agent Role
1.1.1 Event Collection
The collection process involves extracting the data logs from the source systems (Security, OS,
RDBMS, etc.) and allows first steps for event log filtering. At this stage can be decided what is going
to be read by the OSSIM Agent and what is going to be discarded before having an impact on the
system performance.
Before starting to write a plugin some actions to reduce the amount of events could be considered:
o
-
Manage the log level settings at the application and managed device level
-
Fix the problem that is generating events
-
Use Pcap filters to ignore certain hosts or networks (Snort, Tcpdump...)
-
In deployments with a big amount of analysed data, filtering at the application level
should be done whenever possible
Log Files
Good practice is to use one log file per plugin in order to increase performance. Having just
one generic log file, all the plugins would have to read the same extensive content in order
to catch the few relevant entries.
Using rsyslog it is possible to filter the collected logs based on the syslog tags.
1.1.2 Event Normalization
In the normalization stage a series of rules or functions applies to the data extracted from the source
system in order to transform it in a common OSSIM format.
o
Raw Event
The raw event might be a generic syslog message, an application log, an SNMP trap, the
result of an SNMP or SQL Query or some other kind of information in a more or less
structured form that is appended to a log file.
Example:
dmz01:/var/log/auth.log:
May 30 13:15:52 dmz01 sshd[12980]: Accepted password for root from
192.168.178.20 port 4445 ssh2
Page 4
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
o
Normalized Event
There is a certain set of fields which are required in order to ensure a consistent evaluation
and correlation of the events by the OSSIM server. These fields can be populated with
information from the log message or statically through the plug-in.
Example:
ossim-sensor:/var/log/ossim/agent.log:
2010-05-30 13:15:49,441 Output [INFO]: event type="detector" date="1275239752"
sensor="192.168.178.201" interface="eth0" plugin_id="4003" plugin_sid="7"
src_ip="192.168.178.20" src_port="4445" dst_ip="192.168.178.200" dst_port="22"
username="root" log="May 30 13:15:52 dmz01 sshd[12980]: Accepted password for
root from 192.168.178.20 port 4445 ssh2" fdate="2010-05-30 13:15:52" tzone="0"
Page 5
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
1.2 OSSIM Server Role
1.2.1 Event Enrichment
The OSSIM server enriches the received normalized event with the metadata stored in the OSSIM
Database.
o
Enriched Event
The OSSIM Server enriches the event with the Priority and Reliability values, which are
specific to the event type (plugin_id) and subtype (plugin_sid), as well as with the Asset
Value which is specific to the Source (asset_src) and the Destination (asset_dst) hosts.
Example:
ossim:/var/log/ossim/server.log:
2010-05-30 06:48:41 OSSIM-Message: Event received: event id="0" alarm="0"
type="detector" fdate="2010-05-30 13:15:52" date="1275239752" tzone="0"
plugin_id="4003" plugin_sid="7" src_ip="192.168.178.20" src_port="4445"
dst_ip="192.168.178.200" dst_port="22" sensor="192.168.178.201" interface="eth0"
protocol="TCP" asset_src="2" asset_dst="2" log="May 30 13:15:52 dmz01
sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2"
username="root"
o
Priority
The priority is related to threats and it reflects the importance of a specific attack, having
nothing to do with a specific host or environment. It only measures the relative importance
of the attack itself.
Range: 0 - 5
Default value: 1
Example:
A Unix server running Samba gets attacked by the Sasser worm .
Apart from the fact that the attack won’t have an impact on the given environment, it
has the potential to exploit a big security hole and for that reason the priority is
considered as being high.
o
Reliability
Classical risk-assessment would refer it as "probability ". Since it's quite difficult to
determine how probable it is for a network to be exposed to certain vulnerabilities, the IDS
related “reliability” approach was considered more appropriate.
Range: 0 - 10
Default value : 1.
Example:
If a host connects to 5 different hosts in the same subnet using port 445, could be a
normal behavior, unreliable for IDS purposes. If connecting to 15 hosts would be
Page 6
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
suspicious, with 500 connections to different hosts in less than an hour the attack
would get more and more reliable.
o
Asset Value
It is assigned to both the Source and the Destination Hosts and represents the importance
the host has to the enterprise.
Range: 0 - 5
Default value: 1 (also used for hosts not being defined in the asset database)
Example:
A database server can have an asset value of 5, a development test server an asset
value of 2 and an unknown host in the Internet causing a portscan event would just
have an asset value of 1.
o
Alarm
Based on the Event Priority (0-5), Event Reliability (0-10) and the Asset Value (0-5), a Risk
Value (0-10) is calculated and for values equal or greater than 1 Alerts are generated.
The Risk is calculated based on the following formula:
Risk = (Priority * Reliability * Asset) / 25
1.2.2 Policies and Actions
Policies are defined in order to define what has to be done with the events as they reach the OSSIM
Server:
Correlation (i.e. checked against the correlation directives)
Forwarding (i.e. one copy is sent to the forensic storage)
Actions (i.e. send an email)
Discard - the last filter possibility before saving the event in the database, although it is
recommended to filter the events as close to the source as possible.
Policies can make decisions on which events are going to be filtered based on:
Source and Destination Assets (Hosts, Networks, ANY...)
Ports
Plugin Group
Time Range
Page 7
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
1.3 The Configuration Workflow
o
Collect a Log Sample
First thing to start with is checking which log messages the application generates and eventually
identify sets of logs having a similar structure. Those logs having a similar structure will be where
possible covered by a single collector rule.
o
Create a Plugin File
Best is to copy one existing file and modify its content to match the new application. Should a plugin
exist for a similar application, it is recommended to copy such a file, as there is a good chance that
rules have a similar content and are grouped in a similar way - a generic HTTP-Proxy log will always
contain a URL, a generic Firewall log will contain a Source IP Address and Source Port as well as a
Destination IP Address and Destination Port. Some user defined fields might be defined for a specific
application and the correlation at the server level can be simplified if similar applications use the
same user defined fields.
o
Define a Generic Rule
This is the last Rule to evaluate, which catches all the events that cannot be grouped under specific
rules.
o
Define Specific Rules
The Specific rules are defined for specific error conditions or categories of events. There might also
be that one single rule is used to generate different types or subtypes of events.
o
Discard Noise
Events that are considered noise can be discarded by OSSIM by excluding certain event subtypes
(Plugin_SIDs) in the plugin file, by the way the regular expressions are defined or by using policies.
However, the best way to discard events is by filtering them on the monitored device or at syslog
level on the host running the OSSIM Agent.
o
Review the Evaluation Order
The rules are evaluated alphabetically, which means that all it counts is the name of a rule and not
the position in the plug-in file. The Generic Rule might even be on the first position if the name is
properly chosen. Having rules alphabetically placed after the Generic Rule will have as effect that
the corresponding logs will be evaluated as generic events instead of having the proper event type
and subtype assigned.
o
Register the Plugin with the OSSIM Agent
In order to have a Plugin activated and sending events to the OSSIM server, the path to the plugin file
has to be specified in the Agent configuration file.
Page 8
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
o
Register the Plugin with the OSSIM Server
This is required in order to let the server know which events should be expected and which priority
and reliability values the events should get assigned.
o
Activate the Plugin on the Server Side
Restart the OSSIM Server process.
o
Activate the Plugin on the Agent Side
Restart the OSSIM Agent process.
o
Testing
Using the logger command sample logs can be replayed in order to test the operation of the OSSIM
Agent or Server.
Page 9
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2 Configuring Detector Plugins
2.1 Rsyslog
Rsyslog is the Syslog implementation shipped with OSSIM and allows configuring filtering and
forwarding in a really easy way compared to the classical syslog daemon. Syslog is also the common
method to send and receive logs.
Before starting with the plugin configuration it is recommended to check whether the subset of logs
the plugin will normalize are saved in an individual file and whether noise can be filtered before
reaching the plugin rules.
2.1.1 Configuration File
/etc/rsyslog.conf
2.1.2 Listener Configuration
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
2.1.3 Filters
Forward certain events to a local file
if $msg contains 'error' then /var/log/error
if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and ($msg contains 'error1' or
$msg contains 'error0') then /var/log/somelog
Stop processing some events
if $msg contains 'error' then ~
Regex in Rsyslog
http://www.rsyslog.com/user-regex.php
Page 10
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.2 OSSIM Agent Configuration
2.2.1 Configuration File
/etc/ossim/agent/config.cfg
2.2.2 Parameters
[daemon]
daemon:
Daemon mode (True or False)
pid:
Path to the PID file (Process identifier)
[event-consolidation]
Enables event consolidation at agent level. It is recommended to use polices instead of this
feature as consolidation at the agent level affects the correlation process.
by_plugin:
List of plugins that will be consolidated
enable:
Enable or disable (True or False)
time:
Wait n seconds to consolidate the events before sending them
Example:
[event-consolidation]
by_plugin=1001-1150,1501-1550,4001-4010
enable=False
time=10
[log]
Configures the verbose level and the path to the different log files
error:
File in which the error events will be stored
file:
File in which all the agent logs will be stored
stats:
File in which the agent stats will be stored (Every 5 minutes)
verbose:
Configures the verbose level (Debug, Info, Warning, Error or
Critical)
[output-plain]
Writes in a log file what is being sent to the OSSIM Server (Useful for debugging and
developing purposes)
enable:
Enable or disable (True or False)
file:
File in which the output-plain will be stored
[output-server]
Configures the server to which events are sent
Page 11
enable:
Enable or disable sending events to the server (True or False)
ip:
IP address of the OSSIM Server
port:
Listening port of the OSSIM Server
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
[plugin-defaults]
In this category variables can be defined to be used in the plugins configuration.
Example:
[plugin-defaults]
date_format=%Y-%m-%d %H:%M:%S
interface=eth0
sensor=192.168.178.201
[plugins]
Defines which plugins (detectors and monitors) are enabled
name_of_the_plugin=path_to_the_plugin_config_file
Example:
[plugins]
postfix=/etc/ossim/agent/plugins/postfix.cfg
ssh=/etc/ossim/agent/plugins/ssh.cfg
[watchdog]
Monitor the process associated to each plugin (In case it is running in the same machine)
Page 12
enable:
Enable or disable (True or False)
interval:
Wait X seconds between checks
restart_interval:
Restart the process every X seconds (This has to be enabled in each
plugin)
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.3 Detector Plugin Configuration
2.3.1 Configuration Files
/etc/ossim/agent/plugins/*.cfg
2.3.2 Common Event Types
Copy and modify the existing plugin files to create plugins of the following types.
a.
Log
- Reading from files
Plugin statement: source=log
b.
Database
- Reading from databases
Plugin statement: source=database
−
mssql
- Microsoft SQL
Plugin statement: source_type=mssql
−
mysql
- MySQL
Plugin statement: source_type=mysql
c.
SDEE
- Cisco device logs
Plugin statement: source=sdee
d.
SnortLog
- Snort logs
Plugin statement: source=snortlog
e.
WMI
- Windows Management Instrumentation
Plugin statement: source=wmi
2.3.3 Parameters
[DEFAULT]
Any variable defined inside this category will be sent to the OSSIM Server if not modified by
a plugin rule. User reserved range is between 9000 and 10000.
plugin_id:
Numerical identifier of the plugin within the OSSIM system
Example:
plugin_id=4003
Page 13
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
[config]
type:
detector
enable:
Enable or Disable the plugin (It must be enabled in config.cfg)
source:
Source of the events (log, mssql, mysql, wmi)
location:
The file(s) where the logs can be found - can contain multiple
comma-separated files
create_file:
Create the log file in case it does not exist
process:
Name of the process generating logs (If on the same system)
start:
Start the process when the agent starts (yes/no)
stop:
Stop the process when the agent stops (yes/no)
startup:
Command that starts the process
shutdown:
Command that stops the process
exclude_sids=SID List
Use this option to exclude SIDs
Example (hp-eva):
process=snmptrapd
start=yes
stop=yes
startup=/etc/init.d/snmpd start
shutdown=/etc/init.d/snmpd stop
exclude_sids=404,200,403
[translation]
string=value
Used to map strings to their corresponding values
Example (Postfix):
[translation]
sent=10
bounced=11
[Rule IDs – Specific Rules]
Here are the events collected and normalized.
event_type=event
regexp=Regular Expression
plugin_sid=Plugin SID
Event_Field=Value
Example(ssh):
[01 - Failed password]
event_type=event
regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for inval
user (?P<user>\S+)\s+from\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)"
plugin_sid=1
date={normalize_date($1)}
Page 14
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
src_ip={$src}
dst_ip={resolv($sensor)}
src_port={$sport}
username={$user} [Rule IDs – Specific Rules]
[Rule ID – Generic Rule]
Example (ssh):
[99 - Generic rule]
# Nov 15 11:55:35 11.1.4.9 sshd[1769702]: **********
event_type=event
regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*"
plugin_sid=99
date={normalize_date($1)}
dst_ip={resolv($sensor)}
Note: As rules are ordered alphabetically the Generic Rule has to have the highest Rule ID .
2.3.4 Using Local (Plugin) Variables
The different configuration variables defined in the plugin configuration file can be used with the
following syntax:
%()s
Example:
process=pads
shutdown=killall -9 %(process)s
2.3.5 Using Global (Agent) Variables
\_CFG()
Example:
In the agent configuration file (/etc/ossim/agent/config.cfg):
[watchdog]
restart_interval=3600 ; seconds between plugin process restart
In the plugin configuration file(/etc/ossim/agent/plugins/*.cfg):
restart_interval=\_CFG(watchdog,restart_interval)
Page 15
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.4 Aliases
2.4.1 Path
/etc/ossim/agent/aliases.cfg
2.4.2 Predefined Regular Expressions
The predefined regular expressions can be used when creating new plugins.
IPV4= \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
MAC= \w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}
PORT= \d{1,5}
TIME=
\d\d:\d\d:\d\d
SYSLOG_DATE= \w{3}\s+\d{1,2}\s\d\d:\d\d:\d\d
SYSLOG_WY_DATE= \w+\s+\d{1,2}\s\d{4}\s\d\d:\d\d:\d\d
To use an Alias in the regular expression use the \IPV4, \MAC, \SYSLOG_DATE, etc.
2.5 Functions
2.5.1 Path
/usr/share/ossim-agent/ossim_agent/ParserUtil.py
2.5.2 Conversions
Page 16
resolv(host):
translates a host name to an IPv4 address
resolv_ip(addr):
translates an IPv4 address to a host name
resolv_port(port):
translate a port name into its number
normalize_date(date):
convert date strings to isoformat (must tag the regular expressions
with the following: <year>, <month>, <minute>, <hour>, <minute>,
<second> or <timestamp> for timestamps. To define new date
formats add a new regexp to the DATE_REGEXPS array.
normalize_protocol(proto):
translates the protocols to the protocol numbers, based on the
PROTO_TABLE
md5sum(datastring):
calculates the md5 checksum
upper(string):
all upper case
hextoint(string):
get the integer value of a hexadecimal number
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.5.3 Application Specific Translations
snort_id(id):
adds 1000 to the Snort ID
intrushield_sid(sid,name):
all McAfee Intrushield IDs are divisible by 256, and this length
doesn't fit in the OSSIM table ( mcafee_sid =
hextoint(mcafee_sid)/256)
netscreen_idp_sid(msg):
translates the Netscreen messages based on the
NETSCREEN_IDP_SID_TRANSLATION_TABLE translation table
(defined in ParserUtil.py)
iss_siteprotector_sid(msg):
translates the ISS_SiteProtector messages based on the
ISS_SITEPROTECTOR_SID_TRANSLATION_MAP translation table
(defined in ParserUtil.py)
resolv_iface(iface):
normalize interface name to either “ext” or “int”
2.5.4 User Defined Translations
translate(string):
translates strings based on the entries defined in the [translation]
section of the plugin.
Example (from the iptables plugin):
# The translation section in the plugin configuration file
[translation]
ACCEPT=1
REJECT=2
DROP=3
DENY=3
Inbound=4
Outbound=5
# Rule ID
[0 - iptables]
# Log sample
# Oct 31 08:59:25 M2600001 kernel: RULE 0 -- ACCEPT IN= OUT=lo SRC=127.0.0.1
DST=127.0.0.1 LEN=60
# TOS=0x00 PREC=0x00 TTL=64 ID=8437 DF PROTO=TCP SPT=57275 DPT=836
SEQ=2806649400
# ACK=0 WINDOW=32767 RES=0x00 SYN URGP=0
# Log Parsing
regexp=(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(\S*) (\S*):.*?(\S+)\s+IN=(\S*) OUT=(\S*) SRC=(\S+)
DST=(\S+) LEN=(\d+) \S+ \S+ TTL=(\d+) .*? PROTO=(\S*) SPT=(\d*) DPT=(\d*)
………
# plugin_sid is set to 1, the translated value for ACCEPT
plugin_sid={translate($4)}
Page 17
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.6 Event Fields
Mandatory – no default values, have always to be set when creating a new plugin
plugin_id
Event Type
plugin_sid
Event Subtype
Mandatory – default values are assigned by the OSSIM Agent
date
The time the event has been collected from the device
sensor
The IP Address of the sensor collecting the event
interface
The interface where the event has been collected
protocol
IP Protocol (see /etc/protocols)
src_ip
The Source IP Address
src_port
The Source Port
dst_ip
The Destination IP Address
dst_port
The Destination Port
username
The User referred in the event
password
The Password referred in the event
filename
The Filename referred in the event
Optional
userdata1 – userdata9 User defined fields that could be used in custom reports,
correlation directives, etc.
Special types of events and the list of fields that can be used in each event type:
Host-os-event
Host-mac-event
Host-service-event
host
host
host
os
mac
sensor
sensor
vendor
interface
interface
sensor
port
date
interface
protocol
date
service
application
date
Page 18
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.7 Rules
The Rules define the format of each event and how they are normalized. It is composed by a regular
expression and the list of fields that the event will include once it is sent to the OSSIM Server.
In some cases only one regular expression will collect every event coming from one application, in
some other cases more than one rule will be required.
2.7.1 Evaluation Order
Rules are loading in alphabetical order based on the name given to each rule (Rule ID).
Once the log matches the regex of one rule the ossim agent stops processing the event, therefore
generic rules must be the last to be evaluated.
2.7.2 Structure
o
Name / Rule ID
The name of the rule is mandatory
o
Regular Expression
The regexp field contains the regular expression that defines the format of the events, and
extracts the information to normalize the event.
The regular expression has to be written following Python regular expression syntax:
http://docs.python.org/library/re.html
The information extracted by the regular expression from the log can be accessed by:
Position: (\d\d):(\d\d):(\d\d)
hour={$1}
minutes ={$2}
seconds={$3}
Tags: (?P<hour>\d\d):(?P<minutes>\d\d)(?P<seconds>\d\d)
hour={$hour}
minutes ={$minutes}
seconds={$seconds}
o
Normalized Fields
As the server must receive normalized events, where IP addresses for instance are using the
IPV4 format and the date uses the format YYYY-MM-DD HH:MM:SS (2010-12-31 22:57:00)
To simplify the process of normalizing events functions are defined (more details on
functions can be found in the “Functions” section of this document):
resolv()
Translates hostnames into IPV4 addresses (DNS queries)
Page 19
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
normalize_date()
The normalize_date function translates many date formats into the format
accepted by the OSSIM Server.
o
Translations
Used for instance when the Event ID is not numeric, but plugin_sid has to be numeric.
Translations have to be defined inside the [translation] section. The actual translation is
triggered by using the translate() function.
o
Exclusions
Some events can be filtered during the collection process editing the configuration file for
each plugin:
Page 20
-
Using the option exclude_sids
-
Modifying the regular expressions to avoid matching certain events
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.8 Loading Plugins
2.8.1 Priority and Reliability values
For each Plugin_ID/Plugin_SID pair the Priority and Reliability values will have to be defined while
registering the plugin with the OSSIM Server.
2.8.2 SQL Statement
Similar to copying an existing plugin file and customize it in order to create a new plugin file, an SQL
script can be copied and customized in order to insert the new Plugin information in the database.
The sample SQL script can be found under:
/usr/share/doc/ossim-mysql/contrib/plugins/*.sql
Other than with the Plugin configuration file, the SQL script should be created and executed on the
OSSIM Server and not where the OSSIM Agent runs.
The following is performed by the SQL script:
-
Remove the Plugin ID from the ”plugin” table, should such an entry already exist
-
Remove the Plugin SIDs from the ”plugin_sid” table, should already exist
-
Insert the new Plugin ID information into the “plugin” table
-
Insert the new Plugin SIDs into the “plugin_sid” table
To run the script use the following command (please double-check the content of the SQL scripts
and the command line syntax before applying the changes to the database):
ossim-server:/usr/share/doc/ossim-mysql/contrib/plugins# ossim-db < ssh.sql
Example (/usr/share/doc/ossim-mysql/contrib/plugins/ssh.sql):
-- SSHd
-- plugin_id: 4003
DELETE FROM plugin WHERE id = "4003";
DELETE FROM plugin_sid where plugin_id = "4003";
INSERT INTO plugin (id, type, name, description) VALUES (4003, 1, 'sshd', 'SSHd: Secure Shell
daemon');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES
(4003, 1, NULL, NULL, 'SSHd: Failed password', 3, 2);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES
(4003, 2, NULL, NULL, 'SSHd: Failed publickey', 2, 2);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority,reliability) VALUES
(4003, 99, NULL, NULL, 'SSHd: Generic SSH Event', 1, 1);
Page 21
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
2.9 Plugin Activation
2.9.1 Activate the Plugin on the Server Side
Restart the OSSIM Server process:
ossim-server:~#/etc/init.d/ossim-server restart
2.9.2 Activate the Plugin on the Agent Side
Restart the OSSIM Agent process:
ossim-sensor:~#/etc/init.d/ossim-agent restart
3 Log files
Generic Syslog
/var/log/syslog (Unix)
/var/adm/messages (Solaris)
To identify where the logs for specific applications or certain logging levels are saved, check the
/etc/syslog.conf or /etc/rsyslog.conf files.
OSSIM Agent
/var/log/ossim/agent.log
OSSIM Server
/var/log/ossim/server.log
4 Debugging
Note: Do never leave an application running in Debug mode in a production environment
OSSIM Agent
ossim-agent –vv
OSSIM Server
ossim-server –D6
Page 22
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5 Appendix
5.1 Regular Expressions
Operator
Meaning
c
A non-special character matches with itself
\c
Removes the special meaning of the character c; The RE \$ matches with $
^
Indicates the beginning of the line
$
Indicates the end of the line
.
Any individual character
[…]
One or any of the characters …; accepts intervals of the type a-z, 0-9, A-Z
[^…]
A char different from … ; Accepts intervals of the type a-z, 0-9, A-Z
Regular Expression
Matches with
a.b
axb aab abb aSb a#b ...
a..b
axxb aaab abbb a4$b ...
[abc]
a b c (one character srtings)
[aA]
a A (one character srtings)
[aA][bB]
ab Ab aB AB (two character srtings)
[0123456789]
0123456789
[0-9]
0123456789
[A-Za-z]
A B C ... Z a b c ... Z
[0-9][0-9][0-9]
000 001 .. 009 010 .. 019 100 .. 999
[0-9]*
empty_chain 0 1 9 00 99 123 456 999 9999 ...
[0-9][0-9]*
0 1 9 00 99 123 456 999 9999 99999 99999999 ...
^.*$
A full line
Page 23
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
Operator
Meaning
r*
0 or more occurrences of the RE r
r+
1 or more occurrences of the RE r
r?
0 or an occurrence of the RE r, and no more
r{n}
n occurrences of the RE r
r{,m}
0 or at most m occurrences of the RE r
r{n,m}
N or more occurrences of the RE r, but at most m
r1|r2
The RE r1 or the RE r2
Regular expression
Matches with
[0-9]+
0 1 9 00 99 123 456 999 9999 99999 99999999 ..
[0-9]?
empty_string 0 1 2 .. 9
(ab)*
empty_string ab ababab abababababab
([0-9]+ab)*
empty_string 1234ab 9ab9ab9ab 9876543210ab 99ab99ab ...
Regular expression
Matches with
Equals
\d
Any decimal character
[0-9]
\D
Any non decimal character
[^0-9]
\s
Any space character
[ \t\n\r\f\v]
\S
Any non space character
[^ \t\n\r\f\v]
\w
Any alphanumeric character
[a-zA-Z0-9_]
and “_”
\W
Any non alphanumeric character
\Z
End of line
Page 24
[^a-zA-Z0-9_]
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5.2 Configuration Example
5.2.1 Scenario
In order to detect user logons on a Unix system, the “last” command output will be used. The “last”
command displays the content of the /var/log/wtmp file, where closed and opened terminal sessions
as well as system restarts are logged.
To just create events on status updates, the last output will be collected periodically and compared
to the similar information saved with the previous loop.
The status updates will be sent by syslog, with the help of the “logger” command.
5.2.2 Write a script to monitor the “last” status
#!/bin/sh
# create the file if does not exist
touch /var/log/last.prev
while true
do
# get last entries
last > /var/log/last.new
# send new entries to syslog
diff /var/log/last.prev /var/log/last.new | grep '^>' | logger -t LOGON_EXAMPLE -p local2.info
# move .new to .prev
mv /var/log/last.new /var/log/last.prev
sleep 5
done
5.2.3 Log sample
dmz01:~# tail -f /var/log/messages
Jul 14 19:21:32 dmz01 LOGON_EXAMPLE: > root pts/3
Jul 14 19:23:28 dmz01 LOGON_EXAMPLE: > dbadmin pts/3
Jul 14 19:23:59 dmz01 LOGON_EXAMPLE: > root pts/4
Jul 14 19:24:09 dmz01 LOGON_EXAMPLE: > root pts/4
Jul 14 19:24:09 dmz01 LOGON_EXAMPLE: > dbadmin pts/3
Jul 14 19:24:09 dmz01 LOGON_EXAMPLE: > root pts/2
Jul 14 19:24:54 dmz01 LOGON_EXAMPLE: > root pts/2
Jul 14 19:26:15 dmz01 LOGON_EXAMPLE: > root pts/2
Jul 14 19:26:20 dmz01 LOGON_EXAMPLE: > ossim pts/2
Jul 14 19:26:25 dmz01 LOGON_EXAMPLE: > ossim pts/2
localhost
Wed Jul 14 18:49 - 19:21 (00:31)
localhost
Wed Jul 14 19:23 still logged in
localhost
Wed Jul 14 19:23 still logged in
localhost
Wed Jul 14 19:23 - 19:24 (00:00)
localhost
Wed Jul 14 19:23 - 19:24 (00:00)
172.22.22.10 Wed Jul 14 18:38 - 19:24 (00:45)
172.22.22.10 Wed Jul 14 19:24 still logged in
172.22.22.10 Wed Jul 14 19:24 - 19:26 (00:01)
172.22.22.10 Wed Jul 14 19:26 still logged in
172.22.22.10 Wed Jul 14 19:26 - 19:26 (00:00)
5.2.4 Collect the logs in a new log file
Add the following to the rsyslog.conf on the system running the OSSIM Agent:
#
# LOGON_EXAMPLE
#
local2.info
/var/log/last_logon.log
Page 25
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5.2.5 Restart “rsyslog”
opensourcesim:~# /etc/init.d/rsyslogd restart
5.2.6 Check whether the new entries are written in the new log file
opensourcesim:/etc/ossim/agent/plugins# tail -f /var/log/last_logon.log
Jul 14 19:38:49 dmz01 LOGON_EXAMPLE: > root pts/2
localhost
Wed Jul 14 19:38 still logged in
Jul 14 19:38:54 dmz01 LOGON_EXAMPLE: > root pts/2
localhost
Wed Jul 14 19:38 - 19:38 (00:00)
Jul 14 19:38:59 dmz01 LOGON_EXAMPLE: > ossim pts/2
localhost
Wed Jul 14 19:38 still logged in
Jul 14 19:40:51 dmz01 LOGON_EXAMPLE: > ossim pts/2
localhost
Wed Jul 14 19:38 - 19:40 (00:01)
Jul 14 20:15:09 dmz01 LOGON_EXAMPLE: > reboot system boot 2.6.31.6
Wed Jul 14 17:39 - 20:15 (02:35)
5.2.7 Create a plugin file
Copy an existing plugin to build the new one on the existing structure
opensourcesim:/etc/ossim/agent/plugins# cp syslog.cfg example.cfg
Set the new plugin specific parameters
;; Building Plugins Example
;; plugin_id: 9001
;; type: detector
[DEFAULT]
plugin_id=9001
[config]
type=detector
enable=yes
source=log
# Enable syslog to log everything to one file. Add it to log rotation also.
# echo "*.* /var/log/all.log" >> /etc/syslog.conf; killall -HUP syslogd
#location=/var/log/all.log
location=/var/log/last_logon.log
# create log file if it does not exists,
# otherwise stop processing this plugin
create_file=true
process=
start=no
stop=no
startup=
shutdown=
## rules
[Rule 01 - Console Session Open]
# Jul 14 20:36:47 dmz01 LOGON_EXAMPLE: > root tty1
Wed Jul 14 20:36 still logged in
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>tty\d+)\s+(?P<logged_event>.*still logged in.*))$"
sensor=\_CFG(plugin-defaults,sensor)
Page 26
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
date={normalize_date($1)}
plugin_sid=1
username={$username}
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
[Rule 02 - Console Session Closed]
# Jul 14 20:35:46 dmz01 LOGON_EXAMPLE: > root tty1
Wed Jul 14 20:18 - 20:35 (00:17)
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>tty\d+)\s+(?P<logged_event>.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=2
username={$username}
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
[Rule 03 - New User Session - IP]
# Jul 14 20:21:49 dmz01 LOGON_EXAMPLE: > root pts/1
172.22.22.10 Wed Jul 14 20:21 still logged in
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>[^\s]+)\s+(?P<source>\IPV4)\s+(?P<logged_event>.*still logged in.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=3
username={$username}
src_ip={$source}
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
[Rule 04 - New User Session - hostname]
# Jul 14 19:23:28 dmz01 LOGON_EXAMPLE: > dbadmin pts/3
localhost
Wed Jul 14 19:23 still logged in
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>[^\s]+)\s+(?P<source>localhost)\s+(?P<logged_event>.*still logged in.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=3
username={$username}
src_ip=127.0.0.1
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
Page 27
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
[Rule 05 - User Session Closed - IP]
# Jul 14 19:26:25 dmz01 LOGON_EXAMPLE: > ossim pts/2
172.22.22.10 Wed Jul 14 19:26 - 19:26 (00:00)
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>[^\s]+)\s+(?P<source>\IPV4)\s+(?P<logged_event>.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=4
username={$username}
src_ip={$source}
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
[Rule 06 - User Session Closed - hostname]
# Jul 14 19:33:56 dmz01 LOGON_EXAMPLE: > root pts/2
localhost
Wed Jul 14 19:33 - 19:33 (00:00)
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:
>\s+(?P<username>[^\s]+)\s+(?P<tty>[^\s]+)\s+(?P<source>localhost)\s+(?P<logged_event>.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=4
username={$username}
src_ip=127.0.0.1
dst_ip={resolv($host)}
userdata1={$tty}
userdata2={md5sum($logline)}
userdata3={$logline}
userdata4={$logged_event}
[Rule 07 - Reboot Detected]
# Jul 14 20:15:09 dmz01 LOGON_EXAMPLE: > reboot system boot 2.6.31.6
Mon May 24 13:51 - 20:15 (51+06:23)
event_type=event
regexp="^(?P<logline>(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE: >reboot.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($1)}
plugin_sid=5
userdata1={md5sum($logline)}
userdata2={$logline}
userdata3={$generator}
userdata4={$logged_event}
[Rule 99 - Catch all]
# Whatever doesn't match the above rules
event_type=event
regexp="^(?P<logline>(?P<date>\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P<host>[^\s]+)\s+LOGON_EXAMPLE:.*))$"
sensor=\_CFG(plugin-defaults,sensor)
date={normalize_date($date)}
plugin_sid=99
userdata1={md5sum($logline)}
userdata2={$logline}
userdata3={$logged_event}
Page 28
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5.2.8 Register the Plugin with the OSSIM Agent
Add the example.cfg plugin to the agent configuration file
opensourcesim:~# more /etc/ossim/agent/config.cfg
[plugins]
syslog=/etc/ossim/agent/plugins/syslog.cfg
example=/etc/ossim/agent/plugins/example.cfg
wmi-system-logger=/etc/ossim/agent/plugins/wmi-system-logger.cfg
Alternatively the plugin can be activated with the ossim-setup utility:
opensourcesim:~# ossim-setup
1)
Change Sensor Settings
2)
Select Detector Plugins
3)
Select the “example” Plugin
Page 29
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
4)
Save & Exit
5.2.9 Register the Plugin with the OSSIM Server
Copy an existing SQL script to build the new one on the existing structure
opensourcesim:/usr/share/doc/ossim-mysql/contrib/plugins# cp syslog.sql example.sql
Get a list of the rules defined in the Plugin configuration file.
opensourcesim:~# grep '^\[' /etc/ossim/agent/plugins/example.cfg
[Rule 01 - Console Session Open]
[Rule 02 - Console Session Closed]
[Rule 03 - New User Session - IP]
[Rule 04 - New User Session - hostname]
[Rule 05 - User Session Closed - IP]
[Rule 06 - User Session Closed - hostname]
[Rule 07 - Reboot Detected]
[Rule 99 - Catch all]
Rules having the same plugin_sid will only require one SQL statement and plugin_sid defined on the
OSSIM server. Different rules where used just because both IP addresses and hostnames are
returned as sources by the “last” command.
-- plugin_id: 9001
DELETE FROM plugin WHERE id = "9001";
DELETE FROM plugin_sid where plugin_id = "9001";
INSERT INTO plugin (id, type, name, description) VALUES (9001, 1, 'Example', 'User logons based on the last output');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 1, NULL, NULL, 'Login: System
console' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 2, NULL, NULL, 'Logout: System
console' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 3, NULL, NULL, 'Login: Pseudo
terminal' , 3, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 4, NULL, NULL, 'Logout: Pseudo
terminal' , 3, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 5, NULL, NULL, 'System reboot:
Restarted' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 99, NULL, NULL, 'Last: Generic
messages' , 1, 1);
After changing the script to reflect the Plugin IDs and SIDs, load the changes with the command:
opensourcesim:/usr/share/doc/ossim-mysql/contrib/plugins# cat example.sql | ossim-db
Page 30
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5.2.10
Check whether the plugin was successfully registered
Plugin ID
Plugin SIDs
5.2.11
Restart the OSSIM Server
opensourcesim:~# /etc/init.d/ossim-server restart
5.2.12
Restart the OSSIM Agent
opensourcesim:~# /etc/init.d/ossim-agent restart
Page 31
Copyright © Alienvault 2010
Building Collector Plugins - Admin Guide
5.2.13
Check whether Events and Alarms are received
Events
Alarms
Page 32
Copyright © Alienvault 2010