Normalizing Data for use in Forensics
Analysis
Major Activities
 XML Parser for the Application Log data
provider
 Metadata map
 Forensics template
 Modified Forensics.xml file to make custom
forensics template available in the Control
Center Console
 Rule to archive data from the application log
Custom XML Parser
Metadata

Used to tag events with information to store the data in an
IDMEF compliant manner in the database
−
IDMEF – Intrusion Detection Message Exchange Format
http://www.rfc-editor.org/rfc/rfc4765.txt

Used to correlate different event properties (i.e. Source IP
from two different events)

Maps are built in an XML file and imported into the
database

Mapping is stored in the OnePoint database
− MetadataFieldDefinition
− MetadataLocalization
− MetadataMap
− MetadataMapMain
Field Map

Used to uniquely identify fields to be used in mapping

Each field that can be used in a metadata map is defined here

Beginning in SM 6, the field map can no longer be modified
−
Any prior modifications will be removed.

Datatypes are restricted in the field map

Example mapping of standard IDMEF field:
<field id="501" IDMEF="analyzer.model" datatype="string">
<Locale id="1033">
<DisplayName>Device Model</DisplayName>
</Locale>
</field>
−
Example mapping for an additional data field:
<field id="915" IDMEF="Signature Name" datatype="string">
<Locale id="1033">
<DisplayName>Signature Name</DisplayName>
</Locale>
</field>
The Hierarchy of Maps
 Metadata Indexes must
be unique at each level
 As an event flows
down the system,
additional properties
are mapped
 The most specific
mapping definition is
the one that will be
applied.
Metadata Hierarchy (Cont…)

Catchall – used only by NetIQ to ensure that
specific fields are mapped so that they are
available to the system for reporting purposes

Event Source - must be unique throughout the
entire system

Event ID - Metadata Index should start at 1 and
must not be repeated per Event Source

Dll Specific - last level and it follows the same
rules as Event ID
Constructing the Map – EventIDMap
<EventIDMap name=“TippingPoint” id=“0”
version=“5.50.0.158”>
</EventIDMap>
 Name is an arbitrary name for the map (and cannot
be renamed)
 Can leave the id = 0 since SM will automatically
assign this value when you import the map
 Version is used in the import process
− When incrementing the version number do not make major
leaps in the versioning. Use the lowest level version segment
to increment it.
Constructing the Map – Source
<Source id=“TippingPoint” MetadataIndex=“1002”>
<Params>
Define Parameters here that are consistent throughout the event source
</Params>
</Source>
Source id MUST match the value you put for the event
source in the XML parser in order for SM to determine that
this is the appropriate map for that event
 MetadataIndex number must be unique (for
custom maps, use ID numbers 1000-9999).
Don’t use ID 1001!
Constructing the Map – Event
<Event id=“200” MetadataIndex=“1”>
<Params>
Define Parameters specifically for this event id.
</Params>
</Event>
Event IDs should align to what has been defined in the
parser.
 MetadataIndex always start at “1” at the Event
level and increase for each new event defined
Constructing the Map – DLL Specific
<DLLSpecific dllname="MSAuditE.dll" start="5.0.2149.1"
end="6.0.0.0" MetadataIndex="1">
<Params>
Define Parameters here that are consistent throughout the particular message
dll.
</Params>
</DLLSpecific>
DllSpecific is used for events coming from a Windows NT
event log. Requires the raw name of the msg dll.
MSAuditE.dll is the Security Log. SM_Msgs.dll is the Security
Manager dll. The start and stop are the version ranges of the
dll that will have this map applied.
 MetadataIndex always start at “1” at the Event level
and increase for each new event defined
Constructing the Map – Defining Parameters
 The Params section is the where the mapping to
IDMEF actually occurs.
<Params>
<param field=“source.node.address.address” source=“6”/>
<param field=“target.node.address.address” source=“8”/>
<param field="Logon Type" source="4" additionaldata="true" />
</Params>
The numbers refer to parameters - “6” = “Parameter 6”
Parameters that do not have a specific IDMEF Field have to be
identified as being additional data by setting additionaldata =
“true”.
Supported Lookups & Operators

Lookups
− Lookup=“SID” - resolves the SID contained in the specified
parameter <param field="target.user.userid.name" source="1"
lookup="SID"/>
−
Lookup=“IP” - reverse lookups the host name in the specified
parameter <param field="source.node.Address.address" source="6"
lookup="IP"/>
NOTE: Using IP Address lookups can be expensive. If the name
resolution fails, it has to time out before the map can be
complete

Operators
− string literals:
− concatenate:
<param field=“X" source=“’foo’"/>
<param field="source.user.userid.name" source="2 +
'\' + 1"/>
−
conditional:<param
Then 5 else 3"/>
field="classification.name" source="if(6 = ‘foo')
Dynamic Event Properties
 Dynamic event properties are pieces of each event that are typically
defined in the workflow.
 $GUID
$ProviderInstance
 $EventNumber
$EventType
 $EventTime
$Category
 $ProviderName
$Message
 $UserName
$UserDomainName
 $Domain
$Computer
 $AgentComputer
$AgentDomain
 $SourceName
$MessageDLL
 $DllVersion
$Description
 $EventImportance
$AnalyzerModel
Example of a Completed Map
<EventIDMap name="Security Event Log" id="0" version="5.10.0.154">
<Source id="Security" MetadataIndex="0">
<Params>
<param field="message" source="$Message" additionaldata="true" />
<param field="analyzer.model" source="'Windows'" />
</Params>
(...)
<Event id="642" MetadataIndex="43">
<Params>
<param field="CLS" source="'PENTASAFE:3.2.2.55'" />
<param field="source.node.name" source="$Computer" />
<param field="source.node.Address.address" source="$Computer" lookup="IP" />
</Params>
<DLLSpecific dllname="MSAuditE.dll" start="5.0.2149.1" end="6.0.0.0" MetadataIndex="1">
<Params> <!-- Windows 2000 and later -->
<param field="target.user.userid.name" source="2" />
<param field="source.user.userid.name" source="5" />
<param field="Account Change" source="1" additionaldata="true" />
<param field="target_account_id" source="4" lookup="SID" additionaldata="true" />
<param field="target_object_domain" source="3" lookup="SID" additionaldata="true" />
</Params>
</DLLSpecific>
<DLLSpecific dllname="MSAuditE.dll" start="4.0.1371.1" end="5.0.0.0" MetadataIndex="2">
<Params> <!-- Windows NT 4.0 -->
<param field="target.user.userid.name" source="1" />
<param field="target_account_id" source="3" lookup="SID" additionaldata="true" />
<param field="source.user.userid.name" source="4" lookup="SID" />
</Params>
</DLLSpecific>
</Event>
(...)
</Source>
</EventIDMap>
Parameters of Interest
 Analyzer.model
− This is an arbitrary name. Typically this will be the data provider name.
 Analyzer.node.name
− This should be set to equal the parameter value indicating the device
that generated the event.
 Classification.name
− This should be set to equal the parameter value of the Event ID
number.
 Any additional IDMEF field that is a part of the Reporting Cube
MetadataImporter.exe
This tool is contained in
C:\Program Files\NetIQ
Security Manager\Onepoint.
Be careful….this tool has
limited error checking
capabilities – just because it
says an import completed
successfully does not
necessarily mean that it did!
Forensics Template
 Allows you to generate platform specific forensic
reports
 Think about what fields you want to include in the
report and associate them with the appropriate
IDMEF data.
 Must be copied to all UI machines in the following
directory:
− <InstallLocation>\NetIQ Security
Manager\Onepoint\VSOC\Config directory
Forensics Template
Notice the syntax for “nonstandard” IDMEF metadata fields. Just like in the
metadata map, you need to have a special syntax associated:
The syntax is:
<Field name=“Forensic label” idmef=“ADDITIONALDATA.DATA”
add_meaning=“nonstandardIDEMEFfieldname” />
Forensic label = whatever you wanted the display name to be in the template
nonstandardIDMEFfieldname = whatever you called the nonstandard field in the
metadata map.
Forensics.xml
 Stored in the C:\Program Files\NetIQ Security
Manager\Onepoint\VSOC\Config directory
 Makes the custom forensics template you already
created available in the dropdown menu
 Add a row to the file Containing the name of the
report type, the PRG GUID, and the name of the
template XML file.
 Note for 5.6 – the network node MUST be a member of an SM Computer
Group in order for report to populate properly.
Forensics.xml
Row Name = Arbitrary display name for the forensic template
PRGID = GUID of the PRG containing archive rules for the provider. Note: SM will
list the computer groups associated with this GUID when defining a forensic query.
The analyzer nodes that are returned by the query MUST be contained in one of
these computer groups, or the data will not display in SM 5.6 (this is less of an issue
in 5.0-5.5 since there is no security filtering on archival).
AnalyzerModel = Whatever was defined as the analyzer model in the metadata map
FILENAME = name of the XML file used for the Forensic template
Adding Custom Computer Groups to the
Cross Platform Template
 The Cross Platform forensic template has a PRGID of “00000000-
0000-0000-0000-000000000000” allowing it to return all of the
computer groups associated with a “NetIQ” module.
−
The forensics wizard queries the Modules tables and uses the idProcessRule to in turn look up all of the computer
groups tied to a module.
 The correct way to address this is to insert a record into the Module
table that builds the association to your custom PRG. Here is a SQL
query that will handle this insertion for you:
DECLARE @RuleGUID uniqueidentifier
DECLARE @ModuleName nvarchar
DECLARE @Version nvarchar
SET @RuleGUID = 'F1D5A4A5-C05D-406E-BC61-7BB4658FAFAA' --Insert the idProcessRule of your PRG here.
SET @ModuleName = <Some Name>' --Insert a name for the module here.
SET @Version = '5.60.001' --Insert a version number here.
------------Do not edit anything below this line-----------Insert INTO Module (idModule, Name, Description, idProcessRule, FeatureCode, Reserve, Lastmodified, Version, idNQMModule)
VALUES (NEWID(), @ModuleName, '', @RuleGUID, '', '', GETUTCDATE(), @Version, NULL)
 Make sure you have a description in the Computer Group or you will
create an unhandled exception in the Forensic Wizard.
Adding endpoints to CG outside of Config
Wizard
 To force inclusion in the computer group outside of
a Configuration Wizard, run this stored procedure
against the Onepoint database via SQL Query
Analyzer:
exec dbo.ComputerInsertNetworkNode 'domainname',
'computername', 'computergroupname'
Or (if endpoint does not have a domain membership)
exec dbo.ComputerInsertNetworkNode NULL, 'computername',
'computergroupname‘
 For example, to add the device “10.1.1.14” to the
Snort Intrusion Detection computer group, you
would run:
exec dbo.ComputerInsertNetworkNode NULL, ’10.1.1.14', 'Snort
Intrusion Detection'
Best Practices
 Before you even start on the XML parser, try to think
about what IDMEF fields you want to report on.
Keep notes on what IDMEF parameters these IDMEF
fields will map to when constructing your parser.
*Remember the dimensions used in the SMReporting OLAP Cube.
 When making your parser, try to make sure that a
given metadata value will always map to the same
event parameter. This makes the metadata map a
lot simpler.
 Note that importing metadata maps will NOT
overwrite what is already in the Onepoint database
unless the version is a later version
Best Practices - cont
 You can test any text based log with a syslog
simulator
 NEVER modify the field map
 Datatypes are restricted in the field map
 Watch out if you want to use lookups in the metadata
map
 Be constantly aware of the dimensions available in
the cube. These are the only fields that will be
summarized.
 And did I mention “Be constantly aware of the
dimensions available in the cube.”
Report Dimensions
Network Nodes
UTC Date
Local Date
Has a user hierarchy for Model and Name. It also has Domain and the GUID as attributes of it.
Has a user hierarchy with 4 levels for Year, Quarter, Month and Day
This is computed using the UTC date and the offset in minutes. Has a user hierarchy with 4 levels for Year, Quarter,
Month and Day
Source Address
Target Address
Source User
Target User
Severity
Will not be allowed to be part of any aggregate.
Will not be allowed to be part of any aggregate.
Event Classification
Event Source
User hierarchy of 3 columns namely Event Category, Classification and the ID itself.
If a given model can have multiple event source, this dimension will capture it. E.g., for windows module, the values
could be security/application/system.
Source Port
Target Port
Source Protocol
Target Protocol
Source Interface
Target Service Name
Source Node Name
Target Node Name
Action
2 different severities (from a field map point of view) are combined in to one dimension. The field map fields for this
one are ‘Severity’ and ‘Syslog severity’.
Consolidated dimension that will get values from multiple module specific fields. E.g. firewall actions, operation,
command, etc.
Rule
Every firewall has its own way of tagging the rule. So this is a combined dimension which contains different column
(from a field map point of view) that represent the same value. E.g rule id, rule name, rule number.
Timing Type
Currently supports 4 possible values are Business hours, Non Business hours, Lunch Hours and Week ends. The
timings are driven by the Trend analysis time properties in Global settings\Log Manager\Adanced Tab.
Object
This dimension will contain File/registry/AD account names. The type will define what object it is.