Uploaded by Yuqing Wang

Administering Splunk Enterprise Security 5.3

advertisement
Administering Splunk Enterprise Security
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
1
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Document Usage Guidelines
• Should be used only for enrolled students
• Not meant to be a self-paced document
• Do not distribute
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
2
May 16, 2012
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Course Prerequisites
• Prerequisites:
– Splunk Fundamentals 1 & 2
– Splunk Enterprise System Administration
– Splunk Enterprise Data Administration
• Recommended:
– Advanced Searching and Reporting
– Splunk Enterprise Cluster Administration
– Architecting Splunk Deployments
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
3
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Course Goals
• Understand basics of ES end-user features
• Plan an ES deployment
• Perform initial ES installation and configuration
• Manage data intake and normalization in ES
• Create correlation searches
• Configure ES lookups
• Configure the ES threat intelligence framework
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
4
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Course Outline
1.
Introduction to ES
8.
Validating ES Data
2.
Security Monitoring
9.
Custom Add-ons
3.
Incident Investigation
10. Tuning Correlation Searches
4.
Analyst Tools & Data Sources
11. Creating Correlation Searches
5.
ES Deployment
12. Lookups & Identity Management
6.
Installation
13. Threat Intelligence Framework
7.
Initial Configuration
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
5
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 1:
Introduction to ES
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
6
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Describe the features and capabilities of Splunk Enterprise
Security (ES)
• Explain how ES helps security practitioners detect and respond to
threats
• Describe correlation searches and notable events
• Describe ES user roles
• Log on to ES
• Discuss the Use Case Library
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
7
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Course Overview
• This course is for consultants and administrators who will be
planning, installing, or configuring ES
• The first part of the course is a high-level overview of the features
used by security professionals
– Focus on how the product works for end-users
• The second part of the course covers the details of deployment,
installation, and configuration
– Focus on how to get the system up and running, and fine-tune it for
site requirements
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
8
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Overview of Splunk Enterprise Security
• Built on the Splunk Operational Intelligence platform
– ES is a Splunk app, installed on a Splunk server
• Leverages Splunk's powerful search capabilities
• Provides tools for security practitioners to detect and respond to
security threats and incidents
• Efficiently manage, analyze and mitigate security breaches
• Highly customizable for your specific enterprise requirements
• Real-time, scalable, context-aware, focused on content
• Makes all data — not just your “security data”— relevant to
your security effort
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
9
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Users
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
10
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
How ES Works
• Security-related data is acquired by add-ons
in your enterprise from servers, routers, etc.
– This data is forwarded to Splunk indexers and
stored as events
• ES runs real-time searches, looking for
indicators of threats, vulnerabilities, or attacks
– If a search discovers something that needs
attention, ES displays it on one or more of its
dashboards
– You can then investigate the issue, track it,
analyze it, and take the appropriate action
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
11
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Data Flow
Firewalls/Proxies
• cisco-pix
• pa-networks
• juniper-networks
• bluecoat
Vulnerability Scanners
(port scanning, testing
vulnerabilities)
• nessus
• mcafee
Intrusion Detection System
(packet sniffing)
• snort
• dragon-ids
• mcafee
Production Servers
(any operating system)
• microsoft-av
• linux-secure
• windows:*
• access-combined
Network Capture (Stream)
• stream:tcp
• stream:udp
• stream:http
Splunk ES
(events, data models)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
12
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Correlation Searches
• Correlation searches run in the background to detect evidence of
attacks, known threats, or vulnerabilities
– These searches run either in real-time or on a schedule
• ES ships with many correlation searches, which can be modified
or extended as needed
• Each correlation search looks for one specific type of threat,
vulnerability, or sign of malicious attack
• If a correlation search finds something that requires attention, an
alert is triggered which creates a notable event
– Other triggers include: send email, run script, update risk score
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
13
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Notable Events
• Correlation searches create notable events in the notable index
• Notable events are created with fields, event types, and tags that
provide information necessary for incident investigation and a link
to the original source event(s)
• Search for the notable events in the notable index
– In ES, select Search > Search to run a manual search
– Run a search like index=notable for a given time period to see the
notable events
– Event Source fields show the correlation search that created the
notable event
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
14
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Beyond Notable Events
• ES provides many advanced tools you can use to examine
security data in detail, such as:
– Risk and threat analysis
– Threat activity detection
– Protocol (stream) intelligence
– Adaptive response
• Security practitioners use these tools:
– Forensic investigation of existing breaches
– Analyze your environment for new threats
– Examine the history of old breaches to understand how they
happened Generated
and prevent
them in the future
for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
15
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Roles
ES Roles (required for ES login)
ES User
ES Analyst
ES Admin
Runs real-time searches
and views all ES
dashboards
Owns notable events
and performs notable
event status changes
Configures ES systemwide, including adding
ES users, managing
correlation searches, and
adding new data sources
User
Power
Admin
Standard Splunk Roles
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
16
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Accessing ES
• ES typically runs on a secure (HTTPS) port, with a
URL for the Splunk server similar to:
https://eshostname:8000
• Users must have an assigned role on the ES server
• Once logged on, ES displays in the list of apps on the
Splunk home page
• Users can configure ES to be the default app to open
under their Preferences
– Click the user name on the top menu bar and select
Preferences
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
17
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
The ES Home Page
ES Menus
Search events, and work issues
Monitor status
ES Documentation site
Configuration tools
Product tour tutorial
Community support
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
18
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Content Management
Configure > Content > Content Management
Filter list by Type
Filter list by App
Create new content
Text filter
Enable, disable, or export
the selected content
Enable or disable
Click a title to edit
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
19
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Active Correlation Searches
• Select Configure > Content > Content Management
• Select the Type as Correlation Search, and the Status as Enabled
– Note which searches are enabled
• By default, only ES Admins can enable, disable, modify, or add new
correlation searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
20
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Events and Data Models in ES
• As raw data is input into ES, it is processed as follows:
– The raw data is converted into events and stored in indexes
– Events are normalized into the Splunk Common Information Model
– Events are added to accelerated data models
• All ES correlation searches, dashboards, and reports use these
accelerated data models in their searches
• You can create your own custom searches based on the events in
main or the data models as needed
– Use Search > Search to create a new search
– Use Search > Datasets to view or create datasets using data models
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
21
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Search Example
• Use | tstats to create reports based on accelerated data models
– Use | tstats summariesonly=t to restrict results to accelerated data
for performance improvement
• Use Search > Datasets to build datasets using ES data models
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
22
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Notable Events Example
1
3 Run a search in the notable index
In the menu bar,
click Search
Under Selected Fields, click source
4
2 In the drop-down,
click Search
5
Examine the list of sources found over the
last 60 minutes, indicating which correlation
searches have generated notable events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
23
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Use Case Library
• The Use Case Library contains analytic stories which are ready-touse examples of how to use ES to quickly identify the scope of
attacks, determine mitigation options, and take remedial action
• Analytic stories:
– Contain the searches needed to implement the story in your own ES
environment
– Provide an explanation of what the searches achieve and how to
convert a search into adaptive response actions, where appropriate
• The Splunk Enterprise Security Content Update (ESCU) add-on
delivers analytic stories to customers as part of a content
subscription service and is updated often with new stories
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
24
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Use Case Library (cont.)
Configure > All Configurations > Content > Use Case Library
Bookmark stories
specific to your duties
Choose a topic to focus
on related use cases
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
25
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Use Case Specifics
Expand an Analytic Story
Sourcetypes use by
the detection
searches for this
analytic story
Detection Searches are
correlation searches that
populate the story results
Recommended Data Sources
that are likely to provide
valuable data
Lookups used by the
detection searches for this
analytic story
Data Models used
by the detection
searches for this
analytic story
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
26
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 1 Lab: Overview of Splunk ES
Time: 10 minutes
Tasks:
• Log on to your lab Splunk server and navigate to the ES home
page
• Examine the source events ES is using to monitor the security
environment and notable events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
27
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 2:
Security Monitoring
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
28
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Use the Security Posture dashboard to monitor the status of ES
• Use the Incident Review dashboard to investigate notable events
• Take ownership of an incident and move it through the
investigation workflow
• Create notable events
• Suppress notable events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
29
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Monitoring and Response
1. ES continually runs correlation searches for known types of threats
and vulnerabilities
– There are 60 built-in correlation searches, and more in the Use Case
Library
– Plus you can create your own
2. When a correlation search detects any Indicators of Compromise
(IOC), ES creates an alert called a notable event or incident
– IOC is an industry term, while notable event & incident are ES terms
3. ES enables you to track, update, and resolve incidents
– Security Posture dashboard provides a cross-domain SOC overview
– Incident Review dashboard to inspect and manage incidents
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
30
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
The Security Posture Dashboard
• An overview of the Enterprise Security condition
• Key Indicators (KI)
at the top provide
an at-a-glance view of
notable event status
over the last 24 hours
• The four panels
provide additional summary information categorized by urgency, time,
and most common notable event types and sources
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
31
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Key Indicators
Only ES Admins can edit Key Indicators
Large number = total number of notable events in that category
Trend of events indicator: red for
increase green for decrease
Black = no threshold
Red = over threshold
Green = under threshold
Total increase or decrease of the
most recent 24 hours compared to the
preceding 24 hour period
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
32
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
KI Drill Down to Incident Review
1
2
From the Security
Posture dashboard,
click a Key Indicator
total value
The information
for the KI opens in
Incident Review
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
33
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Security Posture Panels
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
34
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Notable Event Urgency
• Each notable event has an urgency field, ranging from
informational to critical
• Urgency is a combination of two factors:
– Severity
ê Based on the raw event(s) found by the
correlation search
– Priority
ê Assigned to the associated assets or identities—i.e., the server or user
ê If more than one asset or identity is involved in a single notable event,
the one with the highest priority determines the urgency
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
35
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Urgency Table
Shown is a partial view of the Urgency table (lookup)
Asset/Identity Priority
Event Severity
Info
Low
M e d iu m
Hig h
Critical
Low
Info
Low
Low
Medium
High
M e d iu m
Info
Low
Medium
High
Critical
Hig h
Info
Medium
Medium
High
Critical
Critical
Info
Medium
High
Critical
Critical
Asset/Identity Priority + Event Severity = Urgency
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
36
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Drill Down Support
1
Hover over an item to
preview details about the
underlying notable events
2
Click an item in the Security
Posture dashboard to open the
related notable events in the
Incident Review dashboard
From the Incident Review dashboard:
a. Drill down into the notable details
b. Take ownership
c. Work the issue
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
37
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review Dashboard
Filter options
Timeline & job controls
Urgency
Expand for
details
Add event(s) to an investigation
Actions
menu
Sortable column headers
Notable
Events
Investigation bar
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
38
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review Filter Fields
• Status: New, In Progress, Pending, Resolved, Closed
– Along with Owner, is used to track the status of an incident
• Urgency: Info, Low, Medium, High, Critical
• Security Domain: Access, Endpoint, Identity, Network, Threat, Audit
• Owner: The user assigned to investigate and resolve an incident
• Correlation Search Name: The title of a correlation search —wildcards
(*) are supported
• Sequenced Event: show only events from sequence templates
• Search: Splunk search language expressions
• Tag: A list of tag names
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
39
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Using the Incident Review Dashboard
• Select one or more values per field
– More than one value per field are ORed together
• Urgency values can be toggled on and off
– Grey values are “off” and will not be displayed
• If values are set for more than one field, the fields are
ANDed together
• Status, Owner, Security Domain and Tag support
multiple OR values
– The default All is ignored if other values are selected
• Name supports wildcards, Search supports full SPL
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
40
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Notable Event Details
Notable event
details drop-down
Notable event
Actions menu
Risk score
Field Action
menu
All fields for the notable event, with Action menus for each field
Note
You cannot expand an event until the
search is complete. Not all incidents
have all the same detail items.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
41
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Create a Short ID from Event Details
1
Click Create a Short ID for ES to
automatically generate a Short ID that
makes it easier to find and share
2
The Short ID
replaces the Create
Short ID link
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
42
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Create a Short ID: Notable Event Actions
1
2
From the notable
event Actions
drop-down,
creating a Short
ID is possible
using Share
Notable Event
In addition to creating a Short ID, this enables sharing the event via a link:
• Click the Bookmark button to copy the link for sharing
or
• Click and drag the Bookmark button to your bookmarks bar to save the link
3
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
43
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Search for a Short ID or Investigation
1
3
2
Click
Associations
Select Short ID or
Investigation from
the drop-down
In the Select… field, enter all or part of the Short ID
or Investigation name
(drop-down appears and filters as you type)
Or
Click or scroll to the Short ID or Investigation name
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
44
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Action Menus
• Notable events in Incident Review have two Action menus
– Each event has an Actions menu
– The fields have an Action menu
– And, the fields in the event
details have Action menus
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
45
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Notable Event Actions Menu
• Each notable event has an Actions menu with options related to
the event, such as:
– Adding the event to an investigation
– Suppressing the notable event
– Sharing the notable event with others
– Initiating adaptive response actions
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
46
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Field Actions Menus
• Each notable event field has an
Action menu allowing you to:
– investigate the asset, set tags,
access other ES dashboards,
analyze the data in the field,
and more
• Risk scores for hosts or users
are displayed next to fields
– Click a risk score to open the
Risk Analysis dashboard for that
asset or identity
Note
Scroll the menu to make sure you
see all the available field actions.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
47
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Workflow: Concepts
1. Assign an owner
2. Investigate
3. Implement corrective measures
Investigators are responsible for
changing workflow status values
as they work incidents
ES Admins can define and add new status values, and assign
values to different roles. Statuses in your environment may differ!
New - not yet being worked
In progress - investigation underway
Pending - various: work in progress, awaiting action, etc.
Resolved - fixed, awaiting verification
Closed - fix verified
Note
When a notable is assigned an owner it is
tracked as an incident in the KV Store.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
48
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Workflow: Procedures
As needed, add selected event(s) to an
investigation. It will appear under Related
Investigations in the event details
1
2
Select one or
more events
Click Edit Selected
3
Set Status,
Urgency, Owner,
and Comment
4
Click Save
changes
As needed, click an icon on the
Investigation Bar to view an investigation,
add a new one, or
perform a quick search
Investigation Bar
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
49
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review History
1
Select View all review activity for
this Notable Event to open a
new search showing all review
events for the current issue
2
The `incident_review` macro can be used in custom
searches and reports for incident status tracking by
directly accessing the KV Store
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
50
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adaptive Responses
• A notable event may contain adaptive
responses the analyst can initiate
1
– Actions menu: select other adaptive
2
responses to execute
– Adaptive Responses: displays a list of
previously executed responses
– Next Steps: click a suggested response
• Depending on the type of notable event,
different adaptive responses are available
– Examples: ping host, change risk, run a
script, nslookup, send to UBA, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
51
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Creating and Suppressing Notable Events
• By default, ES Analysts do not have permission to perform these
actions
– An ES Admin must enable these capabilities for ES Analysts
• Manual creation: useful when you have source event data that has
not (yet) been identified by ES as suspicious, and you want to
create a notable event that will identify the issue and allow you to
track it
• Suppression: useful if you are getting false positives from a host
or a user, and you want to exclude future notable events from that
host or user
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
52
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Creating Notable Events
• Create an ad hoc notable event
– For instance, you find an event in Splunk
that has not triggered a correlation search,
but you feel it should be investigated
• Steps:
Run a search on the source events
2. Expand an event and select Event Actions
3. Select Create notable event
4. Enter the desired data for the notable event
5. Click Save
1.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
53
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Suppressing Notable Events
You may suppress notable events
that are false positives, like a server
temporarily misconfigured
• From Incident Review:
Expand the notable event's
Actions menu
2. Select Suppress Notable
Events
Note
end date is optional. If left
3. Set description and The
blank, all future notable events
from the dest field AND signature
dates
are suppressed.
4. Click Save
1.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
54
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Managing Notable Event Suppressions
• After you create a new notable event suppression, you will see the
list of suppressions as a confirmation
– You can access this list via Configure > Incident Management >
Notable Event Suppressions
• Only ES Admins can edit these suppressions by default
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
55
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 2 Lab: Monitoring with ES
Time: 40 minutes
Scenario: An expired user account has been detected attempting
to log on to high priority resources
Tasks:
• Use the Security Posture dashboard
• Research unauthorized network access
• Begin working the issue
• Resolve the issue
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
56
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 3:
Incident Investigation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
57
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Review the Investigation dashboard
• Start an investigation
• Review the Investigation Workbench
• Add artifacts to an investigation
• Assign collaborators and update the status of an investigation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
58
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigations
• An investigation is a collection of activities and notes related to
work done on a specific issue, such as a breach or other incident
• Someone often starts an investigation when they receive notice of
a notable event, alert action, email, help desk ticket, or phone call
• Investigations are organized chronologically into timelines
• Investigations can be managed by one or more analysts
• Use investigations to:
– Visualize progress
– Document work
– Share information
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
59
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add Entries to an Investigation
It is important to add items to investigations to document the
purpose of the steps you have taken to research the issue and to
provide any details that may be useful to your team’s future
investigation work. You can add several types of entries:
• Notable Event
• Action History
• Splunk Event
• Notable Event Suppression Update
• Search String
• Panel Filter
• Note
• Attachments
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
60
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Basic Workflow
1. Open an investigation and as you progress, continue to
add collaborators
– keep the investigation status updated
–
2. Explore any Investigation Workbench artifacts that were
automatically populated from notable events
3. Add more artifacts, events, actions, searches, notes and other
documentation that may be related
4. Filter on specific elements and time ranges across the timeline
5. Close the investigation after completing it
6. Review and share the Investigation Summary as needed
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
61
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigations Dashboard
Lists all investigations
Add investigations
Filter investigations
Click an investigation to view its related
entries (default view is Workbench)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
62
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Start an Investigation
• By default, only ess_admin and ess_analyst have permission to
start investigations. They can start investigations in several ways:
– from the Incident Review dashboard’s Actions menu
– on the Investigations dashboard
– from any ES dashboard using the Investigation Bar at the bottom of
the ES window
– when searching raw events, from the Event Actions menu
• Each investigation has one owner and can have any number of
additional collaborators
– Only owners and collaborators can work and modify the investigation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
63
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add Collaborators to an Investigation
Hover over a collaborator to
view the name, or click to edit
Click to add a
collaborator
Select a user to change
write permissions or
remove as a collaborator
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
64
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Update Investigation Status
• When you open an
investigation, its
status is New
1
• Investigations can
only be deleted by
admins
• Analysts can delete
investigation entries
Edit the Title, Status, and
Description of the investigation
2
3
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
65
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Explore The Investigation Workbench
1. From Incident Review, select suspicious incidents
2. Add the events to a new or existing investigation and open it
3. The Investigation Workbench opens. In the left panel under Artifacts,
select the assets and identities you find interesting and click Explore
(the panels at the right will populate with related data)
4. Drill down into the data panels to examine context, endpoint data,
network data, risk scores, alerts, and more
5. Add more data and documentation in the form of artifacts, searches,
and notes
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
66
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Workbench Panel Data
Context Panel
E nd p oi nt D at a P anel
Network Data Panel
• Risk Scores
• File System Changes
• Web Activity
• IDS Alerts
• Registry Activity
• Email Data
• Notable Events
• Process Activity
• Network Traffic Data
• System Vulnerabilities
• Service Activity
• DNS Data
• Latest OS Updates
• User Account Changes
• Certificate Activity
• Computer Inventory
• Port Activity
• Network Session Data
• Authentication Data
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
67
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Workbench Panels
Change time range
Change panel
Toggle panel
description
1
Select
Artifact(s)
Expand panel view
2
Add selected to the
workbench
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
68
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Artifacts
• Artifacts are assets or identities you may add to an investigation to
determine whether they are involved in the overall incident
• There are several ways to add an artifact to an investigation
– From a notable event (set up by an admin)
ê Actions > Add Event to Investigation
– Manually
ê Add Artifact button
ê Investigation bar’s Add Artifact icon
– From a workbench panel (select any item)
– From an investigation event (Timeline View > Details > click a value)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
69
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Artifacts (cont.)
Artifacts
search filter
2
3
1
Enter artifact information
and select Type
Click a value to open the
Add Artifacts window and
add it as an artifact
Add this artifact to
workbench
Add other
artifacts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
70
Also opens the Add
Artifacts window
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Steps to Add Artifacts Manually
1. Click Add Artifact button or click
2. Select Add artifact or Add multiple artifacts and enter the artifact(s)
you want to add (all the artifacts you add on this window must be the
same type: either assets or identities)
3. Select either Asset or Identity artifact
4. For multiple artifacts, for separator click New Line or Comma
5. Optionally, add a Description and Label(s) (separate labels with
<Enter> or <,>)
6. Optionally, Expand artifact (seeks correlated items from lookups)
7. Click Add to Scope
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
71
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Summary View
Expand for details
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
72
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Timeline View
Change
view
Scroll left
(newer)
Filter by type
Add collaborator
Filter by
search
Edit, delete, or open in incident review
Scroll right
(older)
Click an item to see
details in upper panel
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
73
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Timeline Details View to Add Artifacts
3
Add Artifacts view opens
and auto-populates
3
1
Click Details to view
all fields and values
2
4
Click an
item to
add it as
an artifact
4
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
74
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Edit the Title of an Investigation Entry
1
2
From the Action menu
choose Edit Entry
Change the Title and click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
75
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Bar
The Investigation Bar displays at the bottom of many ES windows
and dashboards including Security Posture and Incident Review
Enable
Livefeed
Add
Artifact
Quick
Search
Add
Notes
Action
History
Select or add an
Investigation
Investigation Bar
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
76
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Bar and Inline Timeline View
From the Investigation Bar, toggle to the Timeline view
Inline Investigation Timeline
Timeline
Zoom
1
Investigation
Entries
Jump to
start
Select an
Investigation
2 Toggle to Timeline view
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
77
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Enable Notable Event Livefeed
• Get a visual notification when any related notable events occur for the
investigation
• Select an investigation, click the bell icon, and enable notification
• Bell icon turns orange within five minutes of the occurrence
• Acknowledge events and add them to the investigation
Enable
Livefeed
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
78
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Quick Search
Click and drag to resize
the search window.
Double click to toggle
full screen to minimized
2
4
3
Enter search criteria
Determine
whether the
results are
useful to the
investigation
5
1
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
79
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add Events
Add notable events from
Incident Review
Or
Add source events from a
search results window
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
80
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Timeline: List View
From Timeline, change
view to List View
Use the Action menu to
delete selected entries
Search which entries to view
View
details
Filter entries by Notable Event, Adaptive
Response Action, etc.
Edit or delete entries or open in
Incident Review
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
81
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Investigation Notes
• Use notes to add textual information and attachments relevant to
the investigation
– Why you ran each search and what the results mean for your
investigation
– Add screenshots from outside sources
as attachments
• There are two types of notes,
standard and timeline
– Timeline notes show up in the
Timeline Slide View, while standard
notes do not
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
82
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add a Note
1
2
Enter a title
2
Click to add a note
3
Modify time as
needed
default = now
Enter comments
4
1
Click to view notes
Add attachments (text or
binary format). 4MB max per
file and are stored in KV Store.
5
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
83
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add a Note (cont.)
• Standard and Timeline notes use the same Create Note window
– To create a timeline note, and have it visible on the Timeline Slide View,
select Add new Timeline Note, or the Show on timeline box
– A standard note will not display in the Slide View, but will display in the List
View prepended with “Draft:”
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
84
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Action History Items
• Action History is a reverse chronological list of all of your activities
in ES, such as searches run and dashboards used
– Add pertinent entries from your Action History to to an investigation
and supplement them with notes
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
85
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add Action History Items
1
2
Select type
Modify time
as needed
4
5
3
Filter search as needed
Select items
6
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
86
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Customizing Investigation Workbench
• ES admins can customize the Investigation Workbench by:
– Adding new types of panels and tabs
– Creating investigation profiles that correspond to specialized
investigation types
– Applying profiles to notable events from correlation searches
• Example:
– Create a profile called Ransomware Attack, that is applied to notable
events created by a malware correlation search that detects
ransomware, and enables an extra tab named Ransomware that
displays detailed searches focused on known ransomware threats
docs.splunk.com/Documentation/ES/latest/Admin/Customizeinvestigations
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
87
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 3 Lab: Investigating an Issue with ES
Time: 40 minutes
Scenario: Remove false positive events for test servers
Tasks:
1. Test workstation status
2. Remove the false positives from the list of incidents
3. Suppress notable events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
88
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 4:
Analyst Tools & Data Sources
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
89
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Identify ES security analyst tools
• Map security tools to data sources
• Examine management tools for analyst dashboards
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
90
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Security Intelligence
• Besides Security Posture and Incident Review, ES provides many
dashboards and tools for security practitioners
– Risk analysis
– Protocol analysis
– Threat intelligence
– User intelligence
– Web intelligence
– Asset and identity viewers
– Glass tables
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
91
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Dashboard Data Dependencies
• Each dashboard panel’s search pulls events from a data model
• If a panel is missing information, examine the panel’s search to
see which data model is used; this can help you understand why
the data is missing
• Causes:
– The data is not in Splunk: install add-ons to input the data
– The data is present in Splunk but is not normalized correctly: modify
normalization settings
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
92
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Risk Analysis
• Correlation searches can add a numeric risk value to objects
(systems or users)
• Risk can be increased by any event that occurs to an object
• The amount of risk assigned can be configured per-object and
per-event
• This is different than priority, severity, or urgency
– Allows you to see cumulative risk caused by multiple events over time
– Allows you to fine-tune the way you interpret threats or vulnerabilities
to your enterprise
• Admins configure risk values to correlation searches and objects
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
93
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Risk Analysis Dashboard
Security Intelligence > Risk Analysis
Filters
Manually
add risk
Key
Indicators
Timeline shows most active risk-increasing events
Events
causing the
most risk
Objects with
most risk
Events affecting risk scores
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
94
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Risk Data Sources
• Note that the Risk data model is the data source
for the panels on the Risk Analysis dashboard
• Each panel has its own search
• Use a search like
|datamodel Risk All_Risk search
to see the sources, sourcetypes and indexes
that are being used in this data model
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
95
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Protocol Intelligence
• Protocol Intelligence in ES provides a set of tools to
analyze network stream data
• Access the protocol intelligence dashboards using
Security Intelligence > Protocol Intelligence
• Protocol Center: overview of network activity
• Traffic Size Analysis: Overall network traffic activity
trends
• Three main subject groups, each with an activity
overview and deep search capability, for DNS, SSL,
and email
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
96
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Stream Events
• Stream events are generated from the Stream app
– Optional install
• Stream events are stored with stream:xxxx source types
– Examples: tcp,
udp, dns, smtp, http
• Standard field extractions:
– Capture time, type, size, source/destination info
• Depending on specific source type, additional fields are extracted
– HTTP: cookies, request parameters, etc.
– SMTP: sender, receiver, subject, summary of body
– DNS: DNS query, query type, DNS host, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
97
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Protocol Intelligence: Protocol Center
Security Intelligence > Protocol Center
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
98
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
User Intelligence: Assets & Identities
• Assets are devices, such as routers, servers, etc.
– Assets are identified by IP address, MAC address, or host name
• Identities are people
– Identities are identified by user name, email address, etc.
• Both assets and identities are managed in ES with lookup tables
– ES can show a meaningful name instead of an IP address or user ID
– You can define watchlists for both assets and identities
• Asset and identity lookups are customized for your environment
by an ES Admin
– Discussed in detail later in the course
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
99
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Access User Activity from Action Menus
• After running a search,
you can open the Action
menu for the user field
and select User Activity
– Opens the
User Activity dashboard
displaying account activity
for the specific user
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
100
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
User Intelligence > User Activity
Security Intelligence > User Intelligence > User Activity
Users accessing
external sites that have
been added to a
watchlist
Risk assigned by various correlation
searches on user activity
External email
Users accessing watchlisted sites
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
101
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Access Anomalies
Security Intelligence > User Intelligence > Access Anomalies
This dashboard is dependent on the gia_summary index, which is filled
by the Access - Geographically Improbable Access - Summary Gen
scheduled search hourly. This search is disabled by default; enable it to
use this dashboard.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
102
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Using the Access Anomalies Dashboard
• Searches for user access during the requested time period,
defaults to 60 minutes
• Displays user access events with locations more than 500 miles
from their previous access location
• The distance (miles) and speed (miles per hour) between
locations yields an indicator of improbability for a user to actually
log in from both locations
• Many access events spanning a short time from many
geographically remote locations is suspicious
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
103
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Web Intelligence
The Web Intelligence menu contains analysis dashboards that are
useful for inspecting various aspects of your website network activity
HTTP Category
HTTP User Agent
New Domain
URL Length
Explore the types of websites being accessed
through your network
Examine the web user agents being used on
your network
See what external domains are being
accessed
Examine request URLs for unusual contents
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
104
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Web Intelligence: HTTP Category Analysis
• Overview of websites used in your organization by category
• Categories are defined by Websense
www.websense.com/content/support/library/web/v76/siem/siem.pdf
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
105
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Per-panel Filtering
• Analysis dashboards provide highlighting or filtering of items on
dashboard views
– After you have determined that an event is not a threat, you can add it
to your whitelist to remove it from the dashboard view
– If an event is determined to be a threat, use the Advanced Filter editor
to add the item to your blacklist of known threats
• This feature is unavailable by default for ES Analysts
– Can be enabled by an ES Admin
• For instance, for HTTP Category Analysis you may want to filter
out expected categories and highlight unwanted categories
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
106
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Creating Per-panel Filters
3
2
Choose to either
filter or highlight
4
Click Per-panel Filter
1
Select one or more events
in the dashboard
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
107
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Filtered vs. Highlighted Events
• Filtered events are no longer displayed
• Highlighted events are marked yellow in the Per-panel Filter
column and are displayed at the top of the list by default
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
108
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Managing Per-panel Filtering Lookups
• Edit filters in the corresponding lookup table
• Access the lookup table by
– clicking View/edit existing filters in the Per-
panel Filter window
Or
– by selecting the lookup table under
Configure > Content > Content
M a n a ge m e n t
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
109
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Per-panel Filter Audit
Audit > Per-panel Filter Audit
• Display data on per-panel filter usage
• See who creates per-panel filters, and what data is being filtered
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
110
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Security Domain Dashboards
• The Security Domains
menu provides access to
analytical dashboards
organized by security domain
• Each set of dashboards contains tools to
search source events related to a
domain, such as Network or Endpoint
• These dashboards display results from original source events
accessed via accelerated data models
• Use the dashboard requirements matrix or examine panel
searches to Generated
determine
specific data model dependencies
for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
111
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Access Domain: Access Center
Security Domains > Access > Access Center
Use filters to focus on specific types of events
Key Indicators show
overview of notable
events and trends
over previous 24
hours compared to
24-48 hours before
Panels show summaries of Access notable
events over time by action, app, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
112
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Access Domain: Access Search
Security Domains > Access > Access Search
Search events specific to Access domain
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
113
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Glass Tables
• Glass tables are custom views that can display security indicators
as well as symbols, icons, and graphics
• Glass tables are visualizations that can be used for status
displays or to enhance understanding of security status
• A glass table can display current or past information
• Glass tables are stored in the KV Store
• Access can be controlled by roles
• Accessed on the Glass Tables menu
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
114
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Glass Table: Standard View
Custom
icons
Gauge indicators
Select
time
Toggle edit
mode
Contextual
graphics
Text
Metrics with
threshold
colors and
trend metrics
Timelines
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
115
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Glass Table: Edit Mode
Controls
Tools
Settings for
selected
widget
Metrics
Work area
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
116
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Glass Table Data Requirements
• Glass table metric values are dependent on ad-hoc searches, or
Key Indicators
– Ad-hoc searches: SPL that results in a single value to display
– Key Indicators are the values displayed at the top of many
dashboards, such as Security Posture
• Glass tables are maintained in the KV Store:
– Application: SplunkEnterpriseSecuritySuite
– Collection: SplunkEnterpriseSecuritySuite_glasstables
• Glass tables can be exported and imported in Configure >
Content > Content Management
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
117
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 4 Lab: Dashboard Data Sources
Time: 15 minutes
Tasks:
• Examine the dashboard panel data sources
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
118
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 5:
ES Deployment
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
119
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Identify deployment topologies
• Examine the deployment checklist
• Understand pre-deployment requirements
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
120
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Deployment Checklist
1. Determine size and scope of installation
2. Configure additional servers if needed
3. Obtain the ES software
4. Determine software installation requirements for search heads,
indexers, and forwarders
5. Install all ES apps on search head(s)
6. Deploy indexer configurations
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
121
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Impact on Resources
• ES generally requires a new, dedicated search head or search
head cluster
– ES is only compatible with other CIM-compatible apps
– ES adds a large number of searches and search results
• Hardware must meet or exceed Splunk minimum requirements:
docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware
• ES increases some hardware requirements:
docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Splunk
_Enterprise_system_requirements
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
122
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Supported Architectures
• Single server (proof of concept, testing, dev)
• Distributed search (single search head, multiple indexers)
• Search head clustering
docs.splunk.com/Documentation/ES/latest/Install/InstallEnterpriseSecuritySHC
• Indexer clustering (including multi-site)
docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
123
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding ES to an Existing Site
Before ES
After ES
After ES install, ES increases
search requirements, adds an extra
search head and 2 more indexers
Pre-ES site with a single search head
and 3 indexers supporting
~500GB/day of indexed data
Log on here
for ES
Log on here for
Splunk search
ES
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
124
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Search Head Requirements
• A dedicated server or cluster for the ES search head(s) with only
CIM-compliant apps installed
• 64 bit OS, minimum 32 GB and 16 processor cores
– Additional memory and CPU capacity may be needed depending on
number of concurrent users, searches, etc.
• Configure search head forwarding:
docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata
• If enabling Monitoring Console, do not use distributed mode
docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Monitoring_Console
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
125
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Indexer Requirements
• Increased search load in ES typically requires more indexers
– Assume at most 100GB/day per indexer
– Hardware minimum: 16 CPU cores, 32 GB RAM
• The exact number of indexers required depends on:
– Types and amounts of data being used by ES
– Number of active correlation searches
– Number of real-time correlation searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
126
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Indexer Cluster Requirements
• You can only enable ES on one search head or search head
cluster for each indexer cluster
• On a multisite indexer cluster:
– Enable summary replication to improve performance
docs.splunk.com/Documentation/Splunk/latest/Indexer/Clustersandsummaryreplication
– Disable search affinity
docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity
• Make sure you use the indexer cluster master to deploy any ES
add-ons to the indexer tier
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
127
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Accelerated Data Model Storage
• In addition to index storage requirements, ES requires space for
accelerated data models
• Acceleration requires approximately 3.4 x (daily input volume) of
additional space per year, or more if replicated in an indexer cluster
• Example: input volume of 500 GB per day with one year retention
– 500 GB * 3.4 = 1700 GB additional space for accelerated data model storage
• Space is added across all indexers
– Example: if there are 5 indexers, 1700 GB / 5 = ~ 340GB per indexer
additional space is required
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
128
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
More About Accelerated Data Models
• Most ES searches are executed on accelerated data models
• The storage volumes allocated for acceleration should be tuned
for best performance and replicated if in a cluster
• By default, acceleration storage is allocated in the same location
as the index containing the raw events being accelerated
• Use the tstatsHomePath setting in indexes.conf if needed
to specify alternate locations for your accelerated storage
docs.splunk.com/Documentation/ES/latest/Install/Datamodels#Configuring_storage_volumes
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
129
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Indexed Real Time Search
• ES automatically configures Splunk to use indexed real time
searching
docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches
#Indexed_real-time_search
• Improves concurrent real time search performance at the cost of a
small delay in delivering real time results from searches
• Leave turned on in ES for best performance
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
130
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Forwarder Requirements
• In general, forwarders are unaffected by ES installation
• However, some add-ons that ES depends on must be deployed to
forwarders to collect data
• Examples:
– Windows add-on
– *NIX add-on
– Splunk Stream add-on
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
131
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
App/Add-on Deployment Options
• Depending on your requirements, you may need to distribute addons to other Splunk instances like search heads, indexers, and
heavy forwarders
• Use the appropriate app and add-on deployment methodology:
– Forwarders and non-clustered Indexers: use Forwarder Management
(Deployment Server)
– Indexer clusters: use the master node to deploy apps to peer nodes
– Search head clusters: use the deployer to deploy apps to cluster
members
docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
132
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder
• splunkbase.splunk.com/app/2962/
• Builds add-ons for custom ES data
• Normalizes custom data into the
Common Information Model
• Built-in validation
• Should not be used on
production servers
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
133
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Important Resources
• Splunk Data Administration, Splunk System Administration, Splunk
Cluster Administration, and Architecting and Deploying Splunk
courses from Splunk Education
• Distributed Splunk overview:
docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverview
• Capacity planning:
docs.splunk.com/Documentation/Splunk/latest/Capacity/Accommodatemanysim
ultaneoussearches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
134
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 5 Lab: Plan a Deployment
Time: 20 minutes
Scenario:
You are working with a client that has Splunk installed on a distributed site.
There is one search head and 4 indexers indexing 800 GB/day, with a
retention period of 1 year. The customer is marginally happy with current
performance. All servers are at basic minimum Splunk hardware levels. No
new inputs are planned after installing ES.
What alterations to their configuration do you suggest before installing ES?
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
135
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 6:
Installation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
136
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• List ES pre-installation requirements
• Identify steps for downloading and installing ES
• Test a new install
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
137
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installation Checklist
• Assuming prep work from previous module is completed, these are the
steps for a single server or distributed (non-clustered) site:
1.
2.
3.
4.
5.
Confirm the environment meets the minimum system requirements for
Splunk Enterprise and ES
Install ES app on search head
Disable un-needed add-ons
Create Splunk_TA_ForIndexers and deploy to indexers
Deploy input-time add-ons to forwarders
• If using deployment server to deploy ES-installed apps and add-ons,
disable it before the installation, and re-enable after installation
https://docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
138
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Apps and Add-ons
Domain Add-ons (DA)
(views, UI components)
Tech Add-ons (TA)
(input, normalization)
Supporting Add-ons (SA)
(searches, macros,
data models, utilities)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
139
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Installation: ES Framework
Main ES Application
S u p p o r t i n g a d d -o n s
• SplunkEnterpriseSecuritySuite
• SA-AccessProtection
• SA-AuditAndDataProtection
• SA-EndpointProtection
• SA-IdentityManagement
• SA-NetworkProtection
• SA-ThreatIntelligence
• SA-UEBA
• SA-Utils
• Splunk_SA_CIM
• Splunk_SA_ExtremeSearch
Domain add-ons
• DA-ESS-AccessProtection
• DA-ESS-EndpointProtection
• DA-ESS-IdentityManagement
• DA-ESS-NetworkProtection
• DA-ESS-ThreatIntelligence
These add-ons are all distributed with the ES installer,
and are only required to be on Splunk search heads.
Generally you will not need to edit any of their
configuration files directly; most settings are available
via the admin user interface.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
140
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Installation: Technology Add-ons
Stand-alone TAs
• Splunk_TA_bluecoat-proxysg
• Splunk_TA_bro
• Splunk_TA_flowfix
• Splunk_TA_juniper
• Splunk_TA_mcafee
• Splunk_TA_nessus
• Splunk_TA_nix
• Splunk_TA_oracle
• Splunk_TA_ossec
• Splunk_TA_rsa-securid
• Splunk_TA_sophos
• Splunk_TA_sourcefire
• Splunk_TA_symantec-ep
• Splunk_TA_ueba
• Splunk_TA_websense-cg
• Splunk_TA_windows
ES TAs
• TA-airdefense
• TA-alcatel
• TA-cef
• TA-fortinet
• TA-ftp
• TA-nmap
• TA-tippingpoint
• TA-trendmicro
These are only distributed as part of ES
Technology add-ons (TA’s) can configure inputs on
forwarders, parsing on indexers, and normalizing on
search heads. See next slide for deployment. More TA’s
are available from Splunkbase for other technologies.
These can also be downloaded from Splunkbase
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
141
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
What Gets Installed Where?
• Install the full ES app on the search head
– Installs all DAs, SAs, and TAs
• Create and install Splunk_TA_ForIndexers on indexers and heavy
forwarders
Note
– Includes all configurations from
PS recommends not using the indexes.conf
included with Splunk_TA_ForIndexers. Instead,
use it to build a new indexes.conf that fine
tunes the index properties of the app.
all enabled TAs, as well as
indexes.conf settings
• Install TAs on forwarders if they do input phase actions
– See TA readme files and their inputs.conf and props.conf files
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
142
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Typical Server Architecture
Forwarders
input-time TAs
Indexers & heavy forwarders
ES index configurations
And index-time TA configurations
(via Splunk_TA_ForIndexers)
Universal
Forwarders gather
operational and
security data and
send to indexers
or heavy
forwarders
Search Head(s)
ES app + all DAs, SAs
and TAs
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
143
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installing on a Single Search Head
• Start with a clean basic Splunk installation
• Do not uninstall any of the default apps which are part
of the basic Splunk package, as they are required by ES
• ES functions best without the installation of additional apps on top of
the basic Splunk package
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
144
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Uploading the ES App
1. Obtain the ES App from SplunkBase/sales rep
2. Upload ES App on the designated ES search head
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
145
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Starting the Installation
• After re-starting Splunk, navigate to the ES app
• You will be prompted to set up the app
– Click Continue to app setup page to begin the installation process
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
146
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installation: App Management
• The first step of the installer allows you to select apps to either
exclude or disable
• Select any apps to disable or exclude, then click Start
Configuration Process to continue
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
147
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installation: Complete
When the installation process is complete, you’ll be prompted to restart the server again
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
148
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Splunk Web Now on HTTPS
• ES converts Splunk Web to HTTPS
– Port is not changed
– You can change the server back to HTTP
in web.conf if desired
• The pre-loaded SSL certificates are self-signed
– This causes a browser warning, but they are completely secure
– You can install your own externally validated certificates
docs.splunk.com/Documentation/Splunk/latest/Security/Howtogetthird-partycertificates
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
149
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Is Installed on the Search Head!
• ES also installs:
– Extreme Search
ES
– *Nix or Windows add-on
Extreme Search
• The Stream app, if installed, can be
integrated with ES
Add-on Builder
• If you want additional add-ons or
apps like the Splunk Add-on Builder
or ES Content Updates, you will
need to install them yourself
*Nix Add-on
Stream App
ES Content Update
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
150
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Standard ES Add-Ons
• ES ships with several add-ons for common security data sources
– For a complete list:
docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons
• Each add-on is related to a specific vendor product or technology
• Each has a specific add-on name and one or more event
sourcetypes
• Some, like the FTP, *NIX, and Windows add-ons, are designed to
input OS data and will require configuration before use
• See the README file in each add-on to for configuration steps
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
151
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Disable Unused ES Add-ons
• Tech add-ons are intended for use with specific technologies
– For example, Splunk Add-on for Websense, Splunk Add-on for Trend
Micro, etc.
• If not already done at install, disable add-ons for products you are
not using on search heads, indexers, and forwarders
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
152
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installing ES on a Search Head Cluster
• The installer will dynamically detect if you are installing in a single
search head environment or search head cluster environment
• Install ES on the Deployer
On the Splunk toolbar, select Apps > Manage Apps and click Install
App from File
2. Click Choose File and select the Splunk Enterprise Security file
3. Click Upload to begin the installation
4. Click Continue to app setup page
5. Click Start Configuration Process, and wait for it to complete
6. Use the Deployer to deploy ES to the cluster members. From the
deployer run: splunk apply shcluster-bundle
1.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
153
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Configuration Page
Navigate to ES > Configure > All Configurations
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
154
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Distributed Configuration Management
• ES > Configure > General > Distributed Configuration Management
– Download Splunk_TA_ForIndexers
ê Creates the Splunk_TA_ForIndexers.spl add-on
ê Collects index-time configurations
and basic index definitions into one
package to simplify the deployment
of add-on configurations to
on-premises indexers
– Download Splunk_TA_AROnPrem
ê Creates the Splunk_TA_AROnPrem.spl add-on that is used when setting up
an adaptive response relay from an ES Cloud search head to an onpremises heavy forwarder
https://docs.splunk.com/Documentation/ES/latest/Admin/Adaptiveresponserelay
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
155
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Deploy Indexer Configurations
• To created the Splunk_TA_ForIndexers.spl, click Download
Splunk_TA_ForIndexers
• Set auto deployment to No, to
manually send the .spl
to the indexers
• Select at least one option and
click Download the Package
props.conf and
transforms.conf
indexes.conf
– Include index time properties:
includes the props.conf and
transforms.conf files in the package
– Include index definitions: includes the indexes.conf file in the package
• Copy the downloaded
.spl to your indexers
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
156
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Integrity Control
• You can optionally enable data integrity control to ensure that the
data ES relies on in indexes is not tampered with
• Data integrity applies hashes on all indexed data
• Configure in indexes.conf or on the index properties in settings
– Set enableDataIntegrityControl to true and re-start server
– Only new inputs will be hashed
– Can be set per-index or globally
• Test integrity from the command line or script:
bin/splunk check-integrity –index <indexname>
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
157
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Protection Audit
Audit > Data Protection
• Displays status of data protection settings per index
• Also displays status for sensitive data if the Personally Identifiable
Information Detected correlation search is enabled
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
158
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Splunk Stream and ES
• ES can use wire data captures from the Splunk Stream app
– Supports Protocol Intelligence
• Install the Splunk Stream app on the ES server
• Install the Stream add-on (Splunk_TA_stream) on machines
where you want to capture data
• Details on installing and configuring Stream:
docs.splunk.com/Documentation/StreamApp
• Details on integrating Stream with ES:
docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
159
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Stream Data Flow
Splunk ES
With Stream app
Execute and
display search
results
Production Servers
with forwarders and
Stream add-on
Capture network data and
forward to indexers
Indexers
Store captured
stream data
Captured data does not
include message content
unless specifically configured
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
160
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 6 Lab: Post-installation Tasks
Time: 15 minutes
Tasks:
• Disable un-needed add-ons
• Create an app package for your indexer(s) (Splunk_TA_ForIndexers)
• Enable special role capabilities
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
161
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 7:
Initial Configuration
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
162
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Configure user roles and capabilities
• Set general configuration options
• Add external integrations
• Configure local domain information
• Customize Key Indicators
• Customize navigation and view permissions
• Customize incident review settings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
163
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Initial Configuration
• There are several general configuration tasks to perform before
you begin to use ES:
– User roles and capabilities
– General configuration options
– External integrations, such as UBA or domain lookup
– Configure local domain information
– Configure Key Indicators
– Customize incident review settings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
164
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
General ES Configurations
ES > Configure > General > General Settings
• Set or modify various ES
parameters
Filter by app and/or text
• Example: indexed real time
on or off, and changing the
indexed real time delay
Modify values
https://docs.splunk.com/Documentation/ES/latest/Admin/Generalsettings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
165
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Roles
• You can put users in the ES User or ES Analyst role as needed
• Users should not be added to the ES Admin role
• Add the ES Admin role as an inherited role to a regular Splunk
role—such as Admin
– All users in the Splunk Admin role also inherit ES Admin abilities
docs.splunk.com/Documentation/ES/latest/Install/ConfigureUsersRoles
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
166
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Enabling Role Capabilities
ES > Configure > General > Permissions
• Enable or disable capabilities for the ess_analyst or ess_user role
• Example: many sites want to allow analysts to manually create or
suppress notable events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
167
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuration Check
Configure > General > Configuration Checker
• A list of automatic checks on ES configurations
• Generally leave alone, unless directed by Splunk Support
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
168
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Local and Cloud Domains
• Some correlation searches need to differentiate between your
local domain vs. external domains
– For instance, if you work at Acme Corp, you may have local domains
ending in acme.com, acmecorp.com, etc.
• Also, there are external cloud domains you may use frequently
that are not suspicious
– External vendors for accounting, expenses, document sharing, etc.
• Your email system may use different email domains from your
standard corporate domain
– Due to acquisitions, mergers, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
169
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing Domain Tables
• Select Configure > Content >
Content Management and select
Type: Lookup
• Click to edit any of the following lookups:
– Corporate Web Domains: domains in
your enterprise
– Corporate Email Domains: email domains
– Cloud Domains: external vendor sites
• Right-click a row, select insert row below and add your domains
– Right-click and delete any sample rows
• Click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
170
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Domain Analysis
• The New Domain Analysis dashboard relies on domain name
lookup information retrieved via a modular input from
domaintools.com
Add your domaintools.com credentials in the Credentials Manager
2. Configure the settings for the Network Query input
3. Enable whois checking
4. Check for events in the whois index
1.
docs.splunk.com/Documentation/ES/latest/User/ThreatListActivitydashboard#Configure_the_ext
ernal_API_for_WHOIS_data
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
171
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Credentials
• In ES, navigate to Configure > General > Credential Management and
click New Credential
• Enter the domaintools.com credentials,
an app, and click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
172
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring the whois_domaintools Input
• Select Configure > Data Enrichment >
Whois Management
• Edit the whois_domaintools entry:
– API Host: URI to your account’s server
– API User: your domaintools.com user name
(password will be retrieved from credential
manager automatically)
– App: the app you stored the credentials in
– Leave other fields with default values unless
you have a proxy or want to alter defaults for queue interval, etc
• Click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
173
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Enabling the Domain Analysis Setting
• Modify the domain analysis setting
– Navigate to Configure > General > General Settings
– Change the Domain Analysis setting to Enabled
• The whois system is now enabled
– Domain name lookup happens when events with IP addresses are
indexed
– Domain info is stored in the whois index and used by the New Domain
Analysis dashboard
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
174
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES and User Behavior Analytics (UBA)
• Splunk User Behavior Analytics (UBA) is a separate solution that
extends your ability to detect insider threats
• UBA can forward insider threat intelligence to ES
docs.splunk.com/Documentation/TA-ueba/latest/User/UsetoIntegrate
• ES can forward notable events to UBA for insider threat analysis
docs.splunk.com/Documentation/ES/latest/User/SendUBASearchResults
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
175
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Integrating Splunk UBA
• Sending ES notable events to UBA
– Configure > UBA Setup
– Enter UBA server host and port and select protocol
• Send UBA insider threat intel to ES
– This integration is on the UBA side
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
176
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Key Indicators
• Key Indicators (KIs)
appear in many ES views
• By default, KIs do not have
a threshold set, so the current count is displayed in black
• You can configure thresholds for each KI
– If the count is above the threshold, the value is shown in red
– Green indicates a value below the threshold
• You can also re-order, delete, or add KIs
• Click Edit to display the edit tools
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
177
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing Key Indicator Order
Save changes
Cancel
Drag and drop to re-arrange
Remove KI
from display
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
178
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add a New Key Indicator
Save
Cancel
Add a new Key
Indicator
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
179
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Changing KI Thresholds
• You may want to use different threshold values
– For instance, if you have a very large
organization, you may expect a few minor
security threats per day, and therefore
would want to increase some of the
thresholds above their defaults
• Edit the Key Indicator panel
• Enter a value in the Threshold field
• Save the new panel settings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
180
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing Key Indicator Searches
• Configure > Content > Content Management and select
Type : Key Indicator
• Select a search name to edit indicator search definition
– Click Edit Acceleration to configure an acceleration search schedule
• To make a new Key Indicator search, click
Create New Content > Key Indicator Search
Click to edit
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
181
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding a Key Indicator Search: 1
• Enter Name, App, Title and Subtitle
Add a search that generates a current
and delta value
• Drilldown URL can be a search,
dashboard or view to open on click
• Add optional acceleration settings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
182
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Example Key Indicator Search
• Example Key Indicator search:
– The `get_delta` macro looks for fields current_count and
historical_count and outputs delta
– The two counts should be based on the two previous 24 hour periods
– Use tstats `summariesonly` if possible for performance
| tstats `summariesonly` count as current_count
from datamodel=Risk.All_Risk
where All_Risk.risk_object_type="user" All_Risk.risk_score>60
earliest=-24h@h latest=+0s
| appendcols [|tstats `summariesonly` count as historical_count
from datamodel=Risk.All_Risk
where All_Risk.risk_object_type="user" All_Risk.risk_score>60
earliest=-48h@h latest=-24h@h ]
| `get_delta`
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
183
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding a Key Indicator Search: 2
• Value contains the current value for the
previous 24 hour period
• Delta contains the difference between
the value for the previous 24 hour
period and the preceding 24 hour
period
• Rendering Options for threshold
coloring, suffix notation, and inversion
• Click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
184
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review KV Store
• All incident review status changes and comments are stored in
the incident_review KV Store collection
• Use the `incident_review` macro to retrieve information from
this lookup
• Example: you are working on a new incident that is similar to one
you worked on before and you want to search for comments
related to the incident
|`incident_review` | search comment = "*...text...*"
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
185
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review KV Store Maintenance
• Periodically clear data from the incident review KV Store:
| inputlookup incident_review_lookup
| eval age = (now()-time)/86400 | search age < 30
| fields - age
| outputlookup incident_review_lookup append=f
• Use the splunk clean command to completely clear out the
incident review collection:
splunk clean kvstore -app SA-ThreatIntelligence
-collection incident_review
• Splunk must be running to use splunk clean kvstore
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
186
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Incident Review Status and Settings
• Configure > Incident Management > Incident Review Settings
– Allows analysts to change notable urgency (default = yes)
– Requires comments when changing status (default = no)
– Sets minimum comment length
– Customizes field display
• Configure > Incident Management > Notable Event Statuses
– Changes the names of default statuses or adds new ones
– Controls the permissions for statuses by role—for instance, restrict
who can transition an incident to closed or resolved
– Applies to status values for incidents and investigations
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
187
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Modifying Urgency Calculation
• You can change the matrix that determines how correlation
severity and asset/identity priority combine to set urgency
– Select Configure > Content > Content Management and edit the
Urgency Levels lookup
– Each row is one combination of priority and severity, with the resulting
urgency shown in the right-most column
– Modify as needed, and save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
188
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Customizing Incident Review
• Add or remove fields to an incident’s table or event attributes
– Table = collapsed, one line per incident (default)
– Event = expanded details
• Example: display the src (IP) field at the right side of the table
attributes
– Navigate to Configure > Incident Management > Incident Review
Settings
– Under Table Attributes, click Insert below on the last row
– Enter src for the field to be displayed, and Source for the label
– Click Done
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
189
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Example: Adding a Field as a Column
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
190
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Untriaged Incident Alert
• The Untriaged Notable Events correlation search can be
configured and customized for your site as needed
• By default, it prepares a list of all notable events in new status or
unassigned owner over the last 48 hours
• Configure its adaptive response actions to send email to a group,
run a script, or create a new notable event with a specific owner
responsible to assign incidents to analysts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
191
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Configuration Health Audit
Audit > ES Configuration Health
• Checks ES
configurations for
settings that may
conflict with ES
defaults
• Useful to check ES
status after initial
configuration or
upgrade
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
192
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Controlling and Customizing Views
• Set permissions on dashboards and
reports to control access
– Only views the current user has
access to are displayed
in navigation
• Clone views and edit to create
custom alternatives
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
193
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Customizing ES Menus
• The ES application enables you to customize the menu system to
add, remove, or move menu items
• Navigate to Configure > General > Navigation to edit the
navigation menus
– Use drag and drop to move menus and menu items
– The checkmark icon in an item’s top-left corner makes it the default
– Use the X icon in an item’s top-right corner to delete it
– Click Save when finished
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
194
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Navigation
i.e. from a new app or add-on
Add a new menu
Undo
Make default
Add menu
items
Save
Edit
Delete
Drag and drop items to
re-sequence menus
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
195
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding a Menu Item
Note
Add a New View adds a top-level menu item; to
add an item to a menu, use the
icon.
Note
Adding links to filtered incident review results:
The view list scrolls—there are many to pick from.
docs.splunk.com/Documentation/ES/latest/Admin/Customizemenubar#Add_a_link_to_a_filtered_view_of_Incident_Review
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
196
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 7 Lab: Initial Configuration
Time: 20 minutes
Tasks:
• Configure Key Indicators
• Modify dashboard permissions:
– Remove access to the Predictive Analytics dashboard for analysts
• Customize navigation:
– Make Security Posture the default view
– Make a menu item more accessible by moving it to the top row
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
197
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 8:
Validating ES Data
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
198
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Verify data is correctly configured for use in ES
• Validate normalization configurations
• Install additional add-ons
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
199
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Data Flow
• ES uses Splunk events for all correlation and analytical searches
using the following process:
Data is input from its source, indexed into events and a sourcetype
is applied
2. Tech add-ons apply normalization configurations based on the
sourcetypes that assign the events to a data model
3. The data model events are accelerated and placed into accelerated
storage, with retention periods up to 1 year
4. All ES correlation searches and dashboard searches are based on
accelerated data model events
1.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
200
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
From Input to Dashboard
Normalization
HPAS
Storage
Unaccelerated DM
Inputs
DA+SA+ ES app
DM
Acceleration
Data Models
Technology
Add-on
Index
TA apps
Inputs
Data Acceleration
Parsing/Indexing
TA_ForIndexers
Forwarders
Search Time
Dashboards
_raw searches
Notable events & summary indexes
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
201
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Data Models
• ES uses data models in the Common Information Model (CIM)
docs.splunk.com/Documentation/CIM/latest/User/Howtousethesereferencetables
• Each data model defines a standard set of field names for events
that share a logical context, such as:
– Malware: anti-virus logs
– Performance: OS metrics like CPU and memory usage
– Authentication: log-on and authorization events
– Network Traffic: network activity
• Data models are conceptual maps, not containers
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
202
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Normalization
• Normalization converts non-standard field names and values into a
uniform set of standardized fields within a data model
• Report designers can build report searches based on these standard
terms without knowing where the data originally came from
• Example: one sourcetype has events with an ACCESS field, containing
numeric codes like 0 (access allowed) and 1 (access denied).
Another sourcetype has an Action field, with values “allowed” and
“denied”. After normalization, both sourcetypes will have the Action
field and use the same values, making it easier to build reports
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
203
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Normalization Process
• Normalization is a search-time process based on event
sourcetypes and includes steps such as:
– Adding tags, which control which events are displayed by which data
models
– Changing field names and values to conform to data model
specifications
• Add-ons automatically normalize most common sourcetypes
• You may have to adjust normalization rules, or create new
normalization add-ons for custom data
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
204
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
CIM Setup
ES > Configure > CIM Setup
• Use the CIM add-on to change data
model settings like acceleration, index
whitelist, and tag whitelist
– Select a data model on the left
• Enable acceleration for the data model
to return results faster for searches,
reports, and dashboard panels that
reference the data model
• For more information:
https://docs.splunk.com/Documentation/CIM/latest/User/Setup#Accelerating_CIM_data_models
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
205
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
CIM Setup (cont.)
ES > Configure > CIM Setup
• Use Indexes whitelist to improve performance by constraining the
indexes that each data model searches (by default, a data model
searches all indexes)
• Use Tags whitelist to restrict the tag attribute of a data model to
specific tag values to improve performance
– By default, whitelists configured for a data model are used as the tags
for the child datasets
https://docs.splunk.com/Documentation/CIM/latest/User/Setup#Accelerate_CIM_data_models
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
206
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Data Input Troubleshooting
• Ideally, after installing ES you will find that all the searches and
dashboards work automatically
• However, if any events have non-standard sourcetypes, the
normalization configurations in the tech add-ons won’t work
– Example: an admin created a sourcetype and the name is incorrect
– Fix: specify the correct sourcetype name in your configuration files
• If you have incoming data from a technology that requires a tech
add-on that does not ship with ES, you’ll have to install it
• If you have custom data to use in ES, you might have to create
your own TA
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
207
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Confirming Normalization
• Match your enabled TAs to CIM data models and verify the events
are being added to the correct data models
– Use the dashboard requirements matrix to determine which data
models support each dashboard
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix
– Also useful: blogs.splunk.com/2015/05/01/relating-add-ons-to-cim
• If a sourcetype is not showing up in a data model:
– Check the sourcetype
– Make sure the TA is installed
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
208
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Steps for Initial Data Verification
1. Make a list of all sourcetypes required by ES
–
This will be dependent on the exact set of technologies and security
products in use at your site
2. Map the sourcetypes to the TA that normalizes it
3. Confirm that the correct sourcetype name is being used
–
Verify against the TA documentation
4. Install additional TAs if needed
5. Verify that normalization is happening
–
Make sure the sourcetype is appearing in the correct data model
and that all searches are executing as expected
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
209
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Map Sourcetypes to Tech Add-ons
1. Match each sourcetype to the tech add-on that will normalize it
–
Use add-on documentation to determine which sourcetypes are
supported
docs.splunk.com/Documentation/AddOns
2. Make sure the correct sourcetype name is being set
Change the sourcetype setting to the correct one, or
– Edit the TA to use the local sourcetype name variant if necessary
–
3. Install (or create) any missing tech add-ons
4. Disable un-needed ES tech add-ons
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
210
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Finding More Add-ons
• Splunkbase has additional add-ons available for ES
https://splunkbase.splunk.com/
• Add-ons must be CIM-compliant to be compatible with ES
• Search Splunkbase and/or the add-on documentation for the
vendor or technology names related to the sourcetype you are
trying to normalize
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
211
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Examining Data Model Contents
• Use the datamodel command to examine the sourcetypes
contained in the data model
|datamodel Network_Traffic All_Traffic search | stats
count by sourcetype
• If the sourcetype is present, the events are correctly tagged and
fields can be checked for normalization
• If the sourcetype or fields are missing:
– Locate an add-on in Splunkbase that corresponds to the vendor or
technology for the sourcetype, or
– Build your own
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
212
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Problem: Missing Cisco ASA Events
1. As you audit the data in Splunk, you find that you want to use
events from the Cisco router logs with sourcetypes cisco:asa
and cisco:fwsm
2. Confirm that this data is present in Splunk indexes, but ES is not
displaying it in any dashboards
3. The Network Traffic data model does not contain these
sourcetypes
– This is because the events are not being tagged with the network and
communicate tags, and also the fields are not being aliased to the
proper names required in the data model
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
213
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Solution: the Cisco Add-on
• In Splunkbase, the Splunk Add-on for Cisco ASA is:
– CIM-compliant
– Designed for use with ES
• Source types:
– cisco:asa: Authentication, Change Analysis, Network Sessions,
Network Traffic, Malware
– cisco:fwsm: Authentication, Network Sessions, Network Traffic
– cisco:pix: Authentication, Network Sessions, Network
http://docs.splunk.com/Documentation/CIM/latest/User/Authentication
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
214
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Installing Add-ons on the ES Search Head
• Not all add-ons and apps require you to restart Splunk. Check the
add-on documentation on Splunkbase for individual instructions
• Knowledge objects in add-ons and apps that are installed on the
same search head as ES, and are exported to other apps or
exported system-wide (export = system) are automatically
visible in ES
• Check the TA Readme file for specific add-on information
– If it indicates it does index-time actions, re-generate and re-deploy
the Splunk_TA_ForIndexers add-on
– Carry out any additional TA setup in the Readme
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
215
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Content Profile Audit
Audit > Content Profile
• Maps data models to searches and dashboard panels
• Quick indication of missing
data
• Usually means either a
data source has not been
configured or normalization
is not complete
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
216
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Model Audit
Audit > Data Model Audit
• Determine which data
models are using the most
storage or processor time
• Note that you can easily
see each data model’s size,
retention settings, and
current refresh status
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
217
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Forwarder Audit
Audit > Forwarder Audit
• Ensures hosts are properly
forwarding data to Splunk
• Detects forwarders that
have failed
• Can be set to monitor all
hosts, or only hosts
configured as
is_expected in the ES
Assets lookup table
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
218
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Indexing Audit
Audit > Indexing Audit
• Summary of events indexed
per day (EPD)
• Time series shows trends
• Also summarized by index
(main, threat_activity, etc.)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
219
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 8 Lab: Validate ES Data
Time: 25 minutes
Tasks:
– Plan and verify inputs
– Examine data model activity
– Install a new Splunk technology add-on to automatically normalize
Cisco ASA events
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
220
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 9:
Custom Add-ons
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
221
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Use custom data in ES
• Create an add-on for a custom sourcetype
• Describe add-on troubleshooting
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
222
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Custom Data Input
• If you have custom data sources you want ES to recognize, create
an add-on to make your custom events CIM-compliant
• Your add-on should contain:
–
–
–
–
–
Data inputs (if required)
Field extractions (if required)
A tagged event type that maps your sourcetype to the appropriate
CIM data model
Field aliases to map non-standard field names to CIM field names
Eval statements (calculated fields) or lookups to map non-standard
field values to CIM field values
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
223
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Models and the CIM
• Your custom events are referenced by CIM data models
– See the CIM documentation for a list of all the data models and their
contents (docs.splunk.com/Documentation/CIM)
• Once you determine which data model should reference your
events, plan which CIM fields relate to your custom fields
• Example:
– You want the Network Traffic data model to return your events
– At docs.splunk.com/Documentation/CIM/latest/User/NetworkTraffic,
you see the list of required and optional fields for this data model
– You make a mapping of your fields to CIM fields
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
224
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Normalization Strategy
• Not all of the source fields will match CIM fields
– You can ignore the extra source fields, or omit them, as appropriate
• Not all of the CIM fields will be present in the source events
– Use eval statements or regex-based field extractions to generate
these fields with valid values if possible, or with placeholder values if
no valid values can be determined
• Should you populate every CIM field in the target data model?
– You need to at least populate the fields used by ES dashboards and
correlation searches
– Mapping as many of the data model fields as possible will make your
events more robust for future use in new views, searches or reports
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
225
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Planning Normalization Requirements
• Determine the dashboards that will display your events
• Use the dashboard requirements matrix to determine the data
model(s) and field names the dashboard(s) require:
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
226
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Model Definitions
• The data model names in the dashboard requirements matrix are
linked to the data model’s CIM documentation
• Use this documentation to determine the tags, field names and
field values your events must use to be CIM-compliant
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
227
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Mapping Original Fields to CIM Fields
• Plan your normalization settings
using a table
• List the required CIM-compliant
field names
• Match them to corresponding
original source fields
• Determine if normalization
is required for each
field’s name and value
Original
CIM
Procedure
sender
src
alias
receiver
dest
alias
method
app
alias
user
user
none
account
unused
ignore
missing
signature
Use eval to create default value
SSID
unused
regex to mask all but last 4 digits
status
action
Use eval to translate source
numeric codes to CIM terms
...
...
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
228
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Splunk Add-on Builder
• Very fast way to build out the initial TA
• Use it to create sourcetypes,
extractions, and data model mapping
• TAs can:
– Automatically input data into Splunk
– Extract fields and map fields to the CIM
– Create alert actions
https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
229
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Getting Started
• Install the Add-on Builder from Splunkbase
• Navigate to the Add-on Builder home page
• Click New Add-on
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
230
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Create Add-on
• Enter a name for the add-on
– This field becomes the name of the new app
– The builder adds a TA- prefix
• Add other optional project items
• Click Create
– This creates a new add-on app on the local
Splunk server
• Your add-on home page is displayed
• You may see a system message to restart Splunk—you can defer
this until done with the new add-on
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
231
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder Home Page
1
Manage Source
Types
2
Map to Data
Models
3
Validate and
Package
Note
The Add-on Builder can do a lot of
things, but for CIM normalization you
only need to add sample data and the
CIM mapping function.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
232
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Sample Data
• Select Manage Source Types
– May need to reboot first if add-on is newly created
• If your sample data is already in Splunk,
use Add > Import from Splunk
1
– Select from a sourcetype list and click Save
– You also specify event breaks, time-stamping and other settings
• You can add multiple sourcetypes if desired
3
2
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
233
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Data Model Mapping
• Select Map to Data Models
• Click New Data Model Mapping
• Enter a name for the new event type
• Select the source type you are mapping
• Click Save
• The Data Model
Mapping Details
view opens
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
234
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Event Types
• Before you can add data model mappings, you must identify your
sourcetype(s) with an event type
– This is used to generate the correct tags for your events to match the
CIM target data model’s constraints
• On the Data Model Mapping Details page, each sourcetype you
added in the sample data must map to one event type
– More than one sourcetype can map to the same event type
• You can also add search criteria to filter out unwanted events from
your data model mapping
– This excludes the events from the data model acceleration
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
235
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
CIM Mapping
Select FIELDALIAS or EVAL
2
from the New Knowledge
Object drop-down
1
Select one or
more target
data models
3
Select a
source field
Click OK
6
Click Done
4
5
Select a
target field
Note
The source event type or expression field can be
an eval statement (to transform the source value
to the CIM required format).
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
236
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Validate and Package
• You can validate your add-on for best practices, CIM mapping,
and field extractions
– Any errors indicate a problem that should be corrected
– Warnings are non-fatal but might need attention
– If you select App pre-certification, a Splunkbase login is required
• Use Download Package to create an SPL package you can
deploy to your production environment
– The add-on is already active on the local system
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
237
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Add-on Builder: Validate and Package (cont.)
Click to
download
1
2
Select validations to apply
3
Click to start
validation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
238
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 9 Lab: Building a Custom Add-on
Time: 45 minutes
Tasks:
• Plan a new add-on for custom data
• Create the add-on with the Splunk Add-on Builder
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
239
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 10:
Tuning Correlation Searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
240
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Describe correlation search operation
• Customize correlation searches
• Describe numeric vs. conceptual thresholds
• Discuss the Event Sequencing Engine
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
241
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Plan, Install, Evaluate, Refine
• Start with a base level of enabled correlation searches
– Security events in the enterprise
– Anomalous audit trails
• Adjust correlation search sensitivity
– False positives: returning results when none are actually there
– False negatives: returning no results when something is expected
• Revisit and adjust thresholds as needed
– New security data is added to your ES install
– The size of what is monitored shrinks or grows
– Decreased number of open issues (i.e. ES is working!!)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
242
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Managed Content
• Correlation searches are one type of ES content
– Correlation searches are stored as saved searches
– Content in ES is any search or view that can be shared and used
between multiple ES sites
• Examples:
– Correlation and Key Indicator searches
– Entity (asset or identity) swim lane searches
– Lookups
– Views (dashboards and panels)
– Saved searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
243
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Content Management Functions
Configure > Content > Content Management
Enable, disable, or export
Filter list by Type
Filter list by
App
Create new content
Text filter
Enable or disable
Click a title to edit
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
244
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Content Management Functions (cont.)
Configure > Content > Content Management
Expand the Information (i)
column to verify dependency
and usage information
The details for each type of
content, and each individual
knowledge object vary
http://docs.splunk.com/Documentation/ES/latest/Admin/Expandcontentmanagementsearches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
245
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Enabling Correlation Searches
• All correlation searches are disabled by default
• Enable the correlation searches that make sense for your
environment
• Consider:
– Types of vulnerabilities or threats you have determined might exist
– Type of security operations you are focused on, i.e., malware,
intrusion detection, audit, change monitoring, etc.
– You may need to increase hardware specs if you have many
correlation searches running
– You can improve overall performance by making less critical
correlation searches scheduled instead of real-time
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
246
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Scheduling a Correlation Search
• By default, all correlation searches
run in indexed real-time mode
• If changed to scheduled, it will
execute every 5 minutes by default
• When editing the scheduled
search, you can change the
time range settings Start time,
End time, and Cron Schedule
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
247
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Tuning Correlation Searches
• Threshold: the criteria that causes a correlation search to trigger
• Scheduling and throttling: how often to run the search and how
often to generate notable events for the same type of incidents
• Adaptive Responses: list of actions to take, including possibly
creating a notable event or setting risk
– Notable event settings: severity, default owner, default status, etc.
– Risk: assigning, increasing, or decreasing the risk score for a given
type of threat or incident
– Other adaptive responses include sending email, running scripts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
248
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Correlation Thresholds
• Some correlation searches may generate more (or fewer) notable
events than you want
• Examine the search string and look for comparison terms in
search or where/xswhere functions and modify as appropriate
for your environment
• Two types of thresholds:
– Numeric
– Conceptual
docs.splunk.com/Documentation/ES/latest/User/ConfigureCorrelationSearches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
249
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Numeric Thresholds
• Simple numeric comparisons
• Example: Excessive DNS Failures
• Note where command with numeric comparison
• Change the numeric value if you need to alter how
frequently notable events are generated in your environment
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
250
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Conceptual Thresholds
• Use Extreme Search functions
• Example: Brute Force Access Behavior Detected
• Note the xswhere command using “medium” as a threshold
– Change as appropriate
docs.splunk.com/Documentation/ES/latest/Admin/Extremesearchexample
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
251
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Choosing Conceptual Thresholds
• In the previous example, the
brute force correlation search
generates a notable event if
there are more than “medium”
failures in an hour
• Use the xslistconcepts
command to determine other
conceptual terms for this
threshold
• Extreme Search conceptual terms map to dynamic ranges of
values that are automatically calculated and updated
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
252
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Determining Conceptual Ranges
Use xsdisplaycontext to display a graph of the values used in
the concepts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
253
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Correlation Search Throttling
• Once a correlation search has been triggered, you probably don’t
want it to immediately re-trigger again for the same issue
• Most OOTB correlation searches throttle alerts to once a day
• If you want to modify this, change the Window duration
• In most cases, leave the Fields to group by alone
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
254
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adaptive Response Actions
• When a correlation search detects an issue, it can initiate one or
more adaptive response actions
• The most common response is to create a notable event
• Many also add risk to the objects associated with the issue
• Other responses can include sending email, running a script,
stream capture, and sending data to UBA
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
255
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Customizing Notable Event Default Values
• Expand the notable adaptive response
• You can modify all the properties of the
notable event that is created by a triggered
correlation search—typically:
– Severity
– Default Owner
– Default Status
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
256
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Event Sequencing Engine
• The Event Sequencing Engine groups correlation searches into batches
of events, in a specific sequence, by specific attributes, or both
• Event sequencing is configured in Sequence Templates
• Sequence Templates:
– Define which Start, Transition, and End
correlation searches need to occur, and the
match conditions
– Define if the transitional searches have to occur
in a given order, or if they can occur in any order
• Templates run as a real-time searches and listen for incoming notable
events and risk modifiers that are triggered by the correlation searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
257
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Scenario
1
Create a template to detect high
priority hosts with multiple
malware infections, excluding test
host ACME-004. Then, detect if
the host has an abnormally high
number of HTTP method events,
excluding any “unknown” methods
Set the starting correlation search to
Endpoint – High Or Critical Priority Host
With Malware. Set the expression to detect
all destinations (dest) except ACME-004.
Give the template a
name and description,
and select the ES app.
2
https://docs.splunk.com/Documentation/ES/latest/Admin/Sequencecorrelationsearches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
258
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Scenario (cont.)
3
4 Set the ending correlation search to Web – Abnormally
High Number of HTTP Method Events By Src, and the
expression to detect all methods except unknown. Also,
set the time limit for the template to run to 60 days.
Add the transitional correlation search Endpoint –
Host With Multiple Infections with the expression to
detect all destinations (dest) except ACME-004.
Note
5 Add a title, urgency, and
security domain for the notable
events that are created when
the template is triggered.
In this template, the Enforce Ordering
box has been unchecked. Therefore, the
transitional searches do not have to
happen in order, they just have to exist
for the template to trigger.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
259
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Results
The results of the Sequence
Templates are Sequenced Events,
which are viewed in the
Incident Review dashboard
Transitions display the
correlation searches
matched in the template.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
260
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 10 Lab: Tuning Correlation Searches
Time: 15 minutes
Tasks:
• Identify numeric thresholds in a correlation search
• Identify conceptual thresholds in a correlation search
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
261
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 11:
Creating Correlation Searches
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
262
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Create a custom correlation search
• Manage adaptive responses
• Manage content import/export
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
263
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Creating a New Correlation Search
1. Determine a pattern of events that indicates an issue you want
to respond to with a notable event or other action
2. Create a new correlation search in the UI using
Configure > Content > Content Management and select Create
New Content > Correlation Search
–
Use Guided Mode if desired
3. Configure scheduling and throttling
4. Configure the alert responses (notable event, etc.)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
264
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Correlation Search Example: Risk
• This example creates a new
correlation search that generates
a notable event once a day for any
server with a risk score over 100
• On the Content Management
page, select Create New Content >
Correlation Search
• Enter the search name, App,
UI Dispatch Context, and Description
• Select Guided Mode to create the actual search
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
265
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Correlation Search Example: Risk (cont.)
• From the Guided Search Editor,
select the Risk Analysis data model
and the All_Risk dataset
• Set Summaries only to Yes
– The correlation search will only search in
accelerated data
– This is faster, but un-accelerated data is
ignored
• Select the time range for the search
• Click Next
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
266
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding a Where Filter
• Next, add filter expressions to limit the source events the correlation
search retrieves
– This could be used to focus
on high priority assets or
specific business units
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
267
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Aggregate and Split-by Functions
• Next, add aggregate functions to
perform operations like count, sum,
or average on fields in the data model
• Optionally, add split-by conditions
to aggregate values categorically
– The example takes the sum of all
risk per host
• Click Next
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
268
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Filters
• Define the logic to determine
what condition will trigger a
new notable event
• In this case, a notable event
is generated if the risk score
for any one host is greater than 100
• Click Next
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
269
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Parsing the Search
• Finally, the search is parsed
and displayed
• After verifying the test, select
Done to save the correlation
search criteria and continue
configuring the rest of the
correlation search fields
• If you edit the search string manually later, you will not be able to
use guided mode to modify the search string
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
270
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Setting the Time Range
• Configure time range options
– Earliest Time and Latest Time are
relative to the scheduled start time
– Cron schedule is how often to run
the search
ê The default is ‘*/5 * * * *’ which is
every five minutes
ê This is over ruled if the correlation
search is set to real time
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
271
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Setting a Schedule
• Scheduling: real-time or continuous
– Manages real-time scheduling
– Typically, leave the default of real-time
• Schedule Window: seconds (or “auto”)
– Allow some flexibility in scheduling to
improve scheduling efficiency
• Scheduling Priority: higher-priority
searches will be selected first by scheduler if a conflict occurs
docs.splunk.com/Documentation/Splunk/latest/Admin/Savedsearchesconf
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
272
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Setting Trigger Conditions
• Normally, a correlation search will trigger its actions (notable,
etc.) if any results are found by the search
• You can use the Trigger Conditions to alter this default
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
273
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Setting Throttling
• Throttling: You should throttle based on a field’s value
– Example: no more than one notable event per host per day (86,400
seconds)
• More than one field can be selected
– Throttling is based on all the field values ANDed together
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
274
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding Response Actions
1
Expand the list of
adaptive responses
2
Select the Notable
response
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
275
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Notable Event Fields
• Configure notable event field values
– Title, description, security domain, severity
– Default owner and status
– Drill-down settings
• Embed field values in title, description, and
drill-down fields using $fieldname$ format
• Description fields support URLs to external
locations
– Useful for best practices documents,
investigation procedures, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
276
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Notable Event Fields (cont.)
• Control actions taken when a
notable is added to an investigation
– Select an investigation profile to
apply to the investigation
– Automatically extract assets and
identities that will be added to
the investigation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
277
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Notable Event Fields (cont.)
• You can control the “next steps”
and “recommended actions” adaptive
responses that appear in Incident
Review
– Next steps appear as links in the
notable event details
– Recommended Actions appear in
the notable event’s Actions menu
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
278
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Saving the Correlation Search
• Click Save to create the new
correlation search
• Click Close and navigate back
to the Content Management page
• Your new search will now display in the list of Correlation
Searches for the ES app
• You can enable, disable, and change to scheduled or real-time as
desired
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
279
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adaptive Response Actions
• Besides (or instead of) creating notable events, adaptive
response actions can automate other critical tasks
• One or more adaptive response actions can be added to each
correlation search
– The action will be executed if the correlation search finds any
matches
• ES ships with a set of default adaptive responses
• You can also install additional adaptive responses, and control
who can access each adaptive response
docs.splunk.com/Documentation/ES/latest/Admin/Setupadaptiveresponse
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
280
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Default Adaptive Response Actions
Notable, Risk
Create a notable event or add to an object’s risk score
Send email, create Splunk
message
Run script
Send email to one or more people, or add a system message in the
Splunk web interface
Execute an automated script. Example: when a correlation search
indicates a host is infected with malware, run a script to quarantine the
target server
Stream capture
Automatically begin collecting detailed network information
Nbstat, nslookup, ping
Execute diagnostic command and attach output to the notable event to
assist in analysis
If User Behavior Analytics is installed and integrated, send the notable
event to UBA for analysis/send to Splunk telemetry
Create a threat intel artifact. Example: a new type of infection is
discovered; add the characteristics of the infection (file name, source IP,
code hash, etc.) to the threat intel database so that future similar attacks
will be immediately alerted
Send to UBA/output to
telemetry endpoint
Add Threat Intelligence
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
281
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Managing Adaptive Responses
Select Settings > Alert actions to enable/disable, change
permissions, or add new adaptive responses
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
282
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adaptive Response Action Center
Audit > Adaptive Response Action Center
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
283
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Content Import/Export
• You can export any of the content types on the Content
Management page by selecting them in the custom search list
and choosing Export
• Enter an app name, prefix, label, version and build number, and
click Export
– The content will be downloaded to your workstation as an .spl file
– It can then be installed as a new app into another ES search head
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
284
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Example: Content Export
Note
DA-ESS is a recommended
prefix for content add-ons, but
is not required.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
285
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Content Export Best Practices
• The app name for the content export is uploaded to the
etc/apps directory of the receiving server
• Be careful when exporting updates to your content
– Example: you export correlation1, naming it correlations.spl,
and upload it to another ES server. Later you export correlation2,
again using correlations.spl as the export name. When you
upload correlations.spl to the second server, it overwrites the old
version of correlations.spl, deleting correlation1
• Either use new app names each time (which could be difficult to
manage) or make sure you always include all content (old and
new) each time you export
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
286
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 11 Lab: New Correlation Searches
Time: 20 minutes
Tasks:
• Create a custom correlation search
– SSH logins are prohibited in your environment. Create a custom
correlation search that detects successful SSH logins and generates
a notable event to alert analysts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
287
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 12:
Lookups and Identity Management
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
288
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Identify ES-specific lookups
• Describe the interaction of lookups with correlation searches and
other ES functions
• Configure asset and identity lookups
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
289
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Data Enrichment
• ES uses many lookup tables to store extended data for use on
dashboards and by searches
– Examples: assets, identities, incident review, threat intelligence, and
categories
• Some lookups are managed by the KV store
– Examples: incident review, threat intel collections
• Others are stored in CSV files in the lookup directories of several
supporting add-ons
• Use Configure > Content > Content Management to manage ES
lookups, filter Type to Lookup
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
290
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
ES Lookup Management
Configure > Content > Content Management
Add a new lookup
Filter for Lookups
Edit lookup
settings
Add a new lookup table
Click a lookup
to edit
Remove from ES
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
291
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing and Viewing Lookup Files
Right-click to add or remove
columns or rows
Click a cell to edit
contents
Click Save when done
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
292
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Adding New Lookup Configurations
• Select Create New Content >
Managed Lookup
• Add a new lookup file or select pre-existing
• Select a containing app
• Set lookup file and definition name, label and
description
• Select Manually Edited for a simple lookup or
Per panel filtering for a panel filter
• Click Save
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
293
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing a Lookup Configuration
• You can modify the lookup type,
label and description
• If you disable Editing, the
lookup can not be updated
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
294
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Lookup Types: 1
Assets
Information about the devices in your environment
Identities
Threat intel
Information about the people in your environment
Locally produced and managed collections of threat intelligence
Interesting ports,
processes, services
Domains
Descriptions of ports, processes, and services, including
prohibited flags
Configure the local corporate domain; correlation searches watch
for non-corporate email and web access
ES Instrumentation * Several lookups used in conjunction with Splunk introspection and
usage data telemetry
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
295
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Lookup Types: 2
Categories
Category definitions for assets and identities
Action history whitelist
Risk Object Types
Mask searches from the action history for investigation journals
Extend risk object type definitions
Security Domains
Edit or extend the list of security domains—access, network, etc
Urgency levels
Edit or extend the urgency level titles
Expected Views
Enable tracking of view use in ES—shows up in View audit
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
296
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Lookups and Correlation Searches
Lookup
Supported Correlation Search
Interesting Processes
Prohibited Process Detected
Interesting Services
Prohibited Service Detected
Prohibited Traffic
Prohibited Port Activity Detected
Local * Intel
Threat List Activity
Asset
Expected Host Not Reporting. Should Timesync Host Not Syncing
Identity
Watchlisted Event Observed
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
297
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Managed Lookups Audit
Audit > Managed Lookups Audit
• Reports on managed
lookups and
collections (i.e. data,
services, transforms,
KV Store, CSV)
• Shows the growth of
lookups over time and
the markers for
anomalous growth
• Can be used to determine if any managed lookups are growing too large
and need to be pruned
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
298
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Asset and Identity Investigators
• Both investigator dashboards allow you to enter an asset or
identity name and a search range
• Both return a time-sequenced set of swim lanes showing activity
for that asset or identity over time, comparing activity between:
– Threats
– IDS attacks
– Authentication activity
– Malware attacks
– Notable events
– Changes (such as firmware or software upgrades, etc.)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
299
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Accessing the Investigators
• Both the asset and identity investigators can be accessed on:
– The User Intelligence menu, or
– From field action menus in the Incident Review dashboard
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
300
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
User Intelligence: Asset Investigator
Search by asset name
Asset information
Swim lanes
showing activity
across areas
Details about the
selected events
in the swim lane
Selecting a bar (set of events)
shows details at right
Area graph shows activity over time period
Choose time
span for search
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
301
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
User Intelligence: Identity Investigator
Search by identity name
Same tools and
functionality as the
Asset Investigator
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
302
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
How to Interpret Investigators
• The swim lanes visually show activity in various areas in time
sequence, making it easy to see incidents that are simultaneous
or sequential
• Activities that coincide in time may have a cause-effect
relationship
• For example:
– A server shows a burst of authentications at 1:15 am
– At 1:17 am, a malware attack notable event is triggered for that server
– The asset investigator makes it apparent that there is a possible
cause-effect relationship spanning across two (or more) swim lanes
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
303
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Pan and Zoom
End
Start
Dragging the pan/zoom controls changes the time frame for the
search and re-executes the search, showing only the activity in the
selected range
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
304
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Swimlanes
• Click Edit and select a collection
of swimlanes
• Use the Custom collection to
select specific swimlanes
Drag swimlanes up
and down into the
order you prefer
• Customize swimlane colors
• All changes are saved as
preferences for the current user
• ES Admins can add new swimlanes and can set
overall defaults and permissions per role as
needed
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
305
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Special Lookups: Assets and Identities
• Assets and identities are managed by the Identity Management
modular input and Identity – XXX – Lookup Gen searches
• All assets and identities are checked for changes automatically
every 300 seconds
• The searches create multiple expanded versions of the lookup
tables for use during searches
• Identity management lookups are stored in
SA-IdentityManagement/lookups
docs.splunk.com/Documentation/ES/latest/User/Identitymanagement
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
306
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Managing Identity and Asset Lists
• Configure > Data Enrichment > Identity Management
• static_assets and static_identities are the normal lookups
– demo_ lookups can be enabled for testing
• administrative_identities documents privileged accounts like root
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
307
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Uploading Assets and Identities
• Initially, load corporate asset and identity data using Splunk addons such as LDAP search or DB Connect
• Periodically re-run to keep assets and identities in ES updated
• You don’t need to include every piece of hardware or every
person – focus on the ones with the most significance
• You don’t need to populate every column
• Asset and identity lookups scale up into the 10k-100k range
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
308
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Example: LDAP Search Identity Upload
|ldapsearch domain=<domain_name> search="(&(objectclass=user)(!(objectClass=computer)))"
|makemv userAccountControl
|search userAccountControl="NORMAL_ACCOUNT"
|eval suffix=""
|eval priority="medium"
|eval category="normal"
|eval watchlist="false"
|eval endDate=""
|table
sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,
manager,priority,department,category,watchlist,whenCreated,endDate
|rename sAMAccountName as identity, personalTitle as prefix, displayName as nick,
givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as
phone2, manager as managedBy, department as bunit, whenCreated as startDate
|outputlookup my_identity_lookup
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
309
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Troubleshooting Assets and Identities
• Examine CSV files in
$SPLUNK_HOME/etc/apps/SA-IdentityManagement/lookups
– Verify that all CSV files are properly formatted
– Verify that expanded versions have been created
• Check the log files
index=_internal sourcetype=python_modular_input
category=asset OR category=identity
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
310
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
More Troubleshooting
• To test an asset match:
| makeresults | eval src="1.2.3.4" | `get_asset(src)`
• To test an identity match:
| makeresults | eval user="hax0r" |
`get_identity4events(user)`
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
311
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Asset Matching
• ES takes the value from an event’s src, dvc, or dest field
and tries to match it to these columns in the asset lookup:
Order
1
Column
ip
Description
match the IP address or address range
2
mac
match on a Media Access Control address
3
dns
match on DNS name
4
nt_host
match on Windows Machine Name (a.k.a. NetBIOS name)
• ES uses the above order to make its first match, then checks
CIDR-based matches for IP addresses
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
312
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
CIDR Asset Matching
• For ip and mac field ranges, if more than one range matches, ES
matches on the smallest range
For example, host=1.2.3.4 matches both
the first and second IP ranges; however,
it only matches on the second one since
that’s the smaller range
• Asset matching allows you to create large, catch-all categories on
MAC or IP ranges, yet still single out smaller groups or individual
IPs within the larger group
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
313
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Other Interesting Asset Columns
Field
Value
Description
bunit
Text field
Arbitrary “business unit” for the asset; useful for filtering
certain views.
category
Text field
User-defined category for asset. The list of options for
this field is retrieved from a separate lookup list.
is_expected
True/False
If true, ES expects this asset to always be running and
sending data to Splunk. If it stops, a notable event is
created. Defaults to False.
should_timesync, True/False
should_update
Works the same as is_expected, except it alerts on
failure to time sync or failure to update.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
314
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Identity Matching
• ES takes a value from an event’s user, src_user, email,
src_email field and tries to match it in the identities lookup:
Order Column
Description
identity
1
Exact match on any one of a list of pipe-separated list of user names in
identity column
Email
2
Exact match
Email
3
First part of email, ie “htrapper” of “htrapper@acmetech.com”
Any
4
Disabled by default—see “conventions” in identityLookup.conf.spec
• There is also a configuration UI to specify which of these to use
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
315
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Identity Lookup Configuration
Configure > Data Enrichment > Identity Lookup Configuration
docs.splunk.com/Documentation/ES/latest/User/Identitymanagement
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
316
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Other Interesting Identity Fields
Field
Identity (key)
Description
Pipe delimited list of usernames representing the identity
Prefix
Nick
First
Middle
Last
Suffix
Email
Phone
watchlist
Prefix of the identity (for example, Dr.)
Nickname of the identity
First name of the identity
Middle name or initial of the identity
Last name of the identity
Suffix of the identity (for example, Jr.)
Email address of the identity
Phone number of the identity
True/false
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
317
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Watchlisting Assets and Identities
• You can add identities and assets to a watchlist, which then
highlights them in various dashboards and searches
– Example: watchlisted users are shown on the User Activity dashboard
• Watchlist users by setting the watchlist to true in
static_identities.csv
• Add assets to watchlists by:
– Configure > General > General Settings
– Edit Website Watchlist Search and add asset IP or DNS
• Watchlisted assets or identities also trigger the Watchlisted Event
Observed correlation search, if enabled
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
318
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Asset Center
Security Domains > Identity > Asset Center
• Overview of assets
• Visualizations by
priority, business
unit, and category
• Table at bottom
shows all asset
lookup columns
Distribution of assets by
priority
Distribution of assets by
business unit
Distribution of assets by
category
All assets with dns, nt_host, ip, mac
address, owner, priority, location,
category, and PCI domain
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
319
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Identity Center
Security Domains Identity > Identity Center
• Overview of identities
• Bottom table shows all
identity lookup columns
Identities by priority
Identities by business unit
Identities by category
Identity information is shown with name, contact
info, priority, business unit, watchlist (boolean:
true or false), and start and end dates.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
320
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Asset/Identity Correlation Performance
Configure > Data Enrichment >
Identity Correlation
• Control asset/identity correlation
by sourcetype
• This can be useful for performance
improvement
• If a given sourcetype does not contain
“interesting” assets or identities, there
is no need for asset/identity correlation
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
321
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 12 Lab: Adjusting Asset Priority
Time: 25 minutes
Tasks:
• Modify asset priority for PROD-MFS servers
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
322
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 13:
Threat Intelligence Framework
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
323
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Objectives
• Describe threat lists and threat list administration tools
• Configure a new threat list
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
324
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
The Threat Intelligence Framework
• The Threat Activity Detected correlation search creates a notable
event if it detects an indicator of compromise (IOC) contained in a
threat intelligence collection
• The threat intel collections are populated automatically by
downloads from external threat libraries
• Threats are categorized by:
– Group: the source or entity originating the threat
– Category: the type of threat, like backdoor, APT, financial, etc.
– Collection: organized by threat method or routing, such as email, file,
process, user, etc.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
325
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Included Generic Intel Sources
Data List
Data
Provider
Data Provider Website
Cisco Umbrella 1 Million Sites
Cisco
https://umbrella.cisco.com/blog/2016/12/
14/cisco-umbrella-1-million/
Alexa Top 1 Million Sites (deprecated)
Alexa
Internet
http://www.alexa.com/topsites
ICANN Top-level Domains List
IANA
http://www.iana.org/domains/root/db
MaxMind GeoIP ASN IPv4 database
MaxMind https://dev.maxmind.com/geoip/geoip2/g
eoip2-anonymous-ip-csv-database/
MaxMind GeoIP ASN IPv6 database
MaxMind https://dev.maxmind.com/geoip/geoip2/g
eoip2-anonymous-ip-csv-database/
Mozilla Public Suffix List
Mozilla
https://publicsuffix.org
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
326
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Threat Activity Dashboard
Security Intelligence > Threat Intelligence > Threat Activity
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
327
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Using Threat Activity
• Displays events related to known threat sites over
the desired time period
• Panels
– Threat activity over time by threat collection
– Most active threat collections and sources
– Threat activity detail
• Filters
– Threat group: a known threat source—i.e., “who”
– Threat category: threat type, such as APT, backdoor, etc.
– Threat Match Value: Choose a filter from a list of fields
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
328
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Threat Activity Details
• Use threat details to examine the most recent threat events,
including source, destination, sourcetype (i.e., how was it
detected), threat collection, group, and category
• You can also filter or highlight as per the other Advanced Threat
dashboards
– Select one or more rows, then click Advanced Filter
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
329
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Threat Artifacts
Security Intelligence > Threat Intelligence > Threat Artifacts
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
330
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Using Threat Artifacts
• Threat Artifacts displays the current content of the threat
intelligence data that ES has downloaded
• You can use the filters at the top to select a threat artifact type,
and then filter by fields relevant to the selected artifact type
• The threat overview panel displays the items that have been
downloaded from threat lists or STIX/TAXII sources
• The sub-panels display statistics on the threat intelligence data by
endpoint, email, network and certificate
• The tabs allow you to drill down into these categories and gain
additional details for each type of threat
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
331
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Configuring Threat Intelligence
• ES can download the following threat intelligence types:
– Threat lists: IP addresses of known malicious sites
– STIX/TAXII: detailed information about known threats, including threat
type, source, etc.
– OpenIOC: Additional detailed information about known threats
• You can also configure local threat lists
• Many intel sources require regular refresh from external sources
• This information is used by the Threat Activity Detected correlation
search
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
332
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
The Threat Intelligence Framework
• Threat intel is downloaded regularly from external and internal
sources by the Threat Download Manager modular input
– This data is parsed into KV store collections with “_intel” suffixes
– These are used as lookups during threat generation searches
• Threat gen searches run periodically (by default every 5 minutes)
and scan for threat activity related to any of the threat collections
– When threat matches are found, events are generated in the
threat_activity index and appear in the Threat Intelligence data model
• This data model is scanned by the Threat Activity Detected
correlation search and new notables for threat activity are created
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
333
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Types of Intelligence Downloads
• Threat intelligence
– Contains information needed to support the Threat Activity correlation
search
– Will be parsed into threat collections by the threat intelligence
framework
• Generic intelligence
– Non-threat descriptive information, such as lookups for top-level
domains, used to add additional details in ES views and reports
docs.splunk.com/Documentation/ES/latest/Admin/Includedthreatintelsources
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
334
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Intelligence Download Management
Configure > Data Enrichment > Intelligence Downloads
Note
Intelligence downloads cannot
have spaces in the names.
Use underscores or dashes
instead.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
335
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Editing Threat Downloads
Note
The is Threat Intelligence
checkbox determines if this is
threat or generic intel.
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
336
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
About Threat Downloads
Field
Description
Is Threat Intel
If true, processed by threat intel framework
Sinkhole
If true, deletes downloaded file after processing
Type
For TAXII feed must be “taxii”, otherwise not used
URL, Interval, User agent
Path to the download, frequency and user agent settings
Weight
An integer used to calculate risk for this threat type
Max age
How long to retain threat intel (default -30d)
POST arguments
Passed to source server if needed; i.e.; credentials, etc.
Parsing options
Extraction rules for formatted feed
Download options
Interval, timeout, etc.
Proxy options
Configure proxy firewall settings
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
337
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Manual Intelligence File Upload
• To manually upload one threat intelligence file:
Navigate to Configure > Data Enrichment >
Threat Intelligence Uploads
2. Add a file name for an OpenIOC, STIX or
CSV file
3. Configure options for weight, category, etc.
4. Click Save
1.
docs.splunk.com/Documentation/ES/latest/Admin/
Uploadthreatfile
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
338
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
OpenIOC Batch File Upload
• OpenIOC threat collections are not handled via the threat
intelligence downloader
• To add OpenIOC files to your ES threat intelligence framework:
Copy the OpenIOC files to etc/apps/
DA-ESS-ThreatIntelligence/local/data/threat_intel
2. Check var/log/splunk/threat_intelligence_manager.log
for the progress of the download
3. Examine the contents of the target threat intel KV store collections for
threat artifacts from the source OpenIOCs
1.
• This can be done on an automated, scheduled basis
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
339
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Threat Intel Collections
• After download, threat intel data is stored in KV store collections
with an “_intel” suffix, such as file_intel, ip_intel,
email_intel, etc.
– Use Settings > Lookups to see them all
– Use |inputlookup to examine their contents
• Use the Threat Artifacts dashboard to examine the overall
contents of the entire threat intelligence framework
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
340
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Threat Intelligence Audit
Audit > Threat Intelligence Audit
• displays status and time
for all downloads as well as
recent audit events
• Filter by enabled/disabled
state, local/remote, source, etc.
• Failed downloads are reported
in the system message list at the top of the SplunkWeb page
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
341
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Module 13 Lab: Threat Intel Framework
Time: 20 minutes
Tasks:
• Add a new threat list download
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
342
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
What’s Next?
OPTION 1
Splunk Enterprise
System Administration
Become a Splunk Enterprise Security
Certified Admin!
Splunk Enterprise
Data Administration
OPTION 2
Splunk Enterprise
Security Certified Admin
Exam
Administering Splunk
Enterprise Security
Splunk Cloud
Administration
Splunk Education Content. See splunk.com/education for access, registration dates and
cost (if applicable). Questions can be addressed to education_amer@splunk.com
Splunk Certification Exams. See splunk.com/education for access, registration dates and
cost (if applicable). Questions can be addressed to certification@splunk.com
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
343
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Support Programs
• Community
– The Splunk Answers Site: answers.splunk.com
Post specific questions and get them answered by Splunk community experts
– Splunk Docs: docs.splunk.com
These are constantly updated. Be sure to select the version of Splunk you are using
– Wiki: wiki.splunk.com
A community space where you can share what you know with other Splunk users
– IRC Channel: #splunk on the EFNet IRC server Many well-informed Splunk users “hang out” here
• Global Support
Support for critical issues, a dedicated resource to manage your account – 24 x 7 x 365
– Web: http://www.splunk.com/index.php/submit_issue
• Enterprise Support
Access you customer support team by phone and manage your cases online 24 x 7
(depending on support contract)
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
344
Administering Splunk Enterprise Security
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
.conf19
Splunk University
October 19-21, 2019
The Venetian Sands Expo
4 Days of Innovation
350 Education Sessions
20 Hours of Networking
October 21-24, 2019
Las Vegas, NV
“Hands down the most beneficial and attendee focused conference
I have attended!”
– Michael Mills, Senior Consultant, Booz Allen Hamilton
Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution
Administering Splunk Enterprise Security
sign
up
for
notifications
@
conf.splunk.com
345
Copyright © 2019 Splunk, Inc. All rights reserved
|
1 July 2019
Download