Administering Splunk Enterprise Security 5.3

Administering Splunk Enterprise Security Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 1 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Document Usage Guidelines • Should be used only for enrolled students • Not meant to be a self-paced document • Do not distribute Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 2 May 16, 2012 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Course Prerequisites • Prerequisites: – Splunk Fundamentals 1 & 2 – Splunk Enterprise System Administration – Splunk Enterprise Data Administration • Recommended: – Advanced Searching and Reporting – Splunk Enterprise Cluster Administration – Architecting Splunk Deployments Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 3 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Course Goals • Understand basics of ES end-user features • Plan an ES deployment • Perform initial ES installation and configuration • Manage data intake and normalization in ES • Create correlation searches • Configure ES lookups • Configure the ES threat intelligence framework Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 4 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Course Outline 1. Introduction to ES 8. Validating ES Data 2. Security Monitoring 9. Custom Add-ons 3. Incident Investigation 10. Tuning Correlation Searches 4. Analyst Tools & Data Sources 11. Creating Correlation Searches 5. ES Deployment 12. Lookups & Identity Management 6. Installation 13. Threat Intelligence Framework 7. Initial Configuration Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 5 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 1: Introduction to ES Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 6 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Describe the features and capabilities of Splunk Enterprise Security (ES) • Explain how ES helps security practitioners detect and respond to threats • Describe correlation searches and notable events • Describe ES user roles • Log on to ES • Discuss the Use Case Library Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 7 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Course Overview • This course is for consultants and administrators who will be planning, installing, or configuring ES • The first part of the course is a high-level overview of the features used by security professionals – Focus on how the product works for end-users • The second part of the course covers the details of deployment, installation, and configuration – Focus on how to get the system up and running, and fine-tune it for site requirements Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 8 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Overview of Splunk Enterprise Security • Built on the Splunk Operational Intelligence platform – ES is a Splunk app, installed on a Splunk server • Leverages Splunk's powerful search capabilities • Provides tools for security practitioners to detect and respond to security threats and incidents • Efficiently manage, analyze and mitigate security breaches • Highly customizable for your specific enterprise requirements • Real-time, scalable, context-aware, focused on content • Makes all data — not just your “security data”— relevant to your security effort Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 9 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Users Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 10 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 How ES Works • Security-related data is acquired by add-ons in your enterprise from servers, routers, etc. – This data is forwarded to Splunk indexers and stored as events • ES runs real-time searches, looking for indicators of threats, vulnerabilities, or attacks – If a search discovers something that needs attention, ES displays it on one or more of its dashboards – You can then investigate the issue, track it, analyze it, and take the appropriate action Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 11 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Data Flow Firewalls/Proxies • cisco-pix • pa-networks • juniper-networks • bluecoat Vulnerability Scanners (port scanning, testing vulnerabilities) • nessus • mcafee Intrusion Detection System (packet sniffing) • snort • dragon-ids • mcafee Production Servers (any operating system) • microsoft-av • linux-secure • windows:* • access-combined Network Capture (Stream) • stream:tcp • stream:udp • stream:http Splunk ES (events, data models) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 12 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Correlation Searches • Correlation searches run in the background to detect evidence of attacks, known threats, or vulnerabilities – These searches run either in real-time or on a schedule • ES ships with many correlation searches, which can be modified or extended as needed • Each correlation search looks for one specific type of threat, vulnerability, or sign of malicious attack • If a correlation search finds something that requires attention, an alert is triggered which creates a notable event – Other triggers include: send email, run script, update risk score Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 13 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Notable Events • Correlation searches create notable events in the notable index • Notable events are created with fields, event types, and tags that provide information necessary for incident investigation and a link to the original source event(s) • Search for the notable events in the notable index – In ES, select Search > Search to run a manual search – Run a search like index=notable for a given time period to see the notable events – Event Source fields show the correlation search that created the notable event Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 14 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Beyond Notable Events • ES provides many advanced tools you can use to examine security data in detail, such as: – Risk and threat analysis – Threat activity detection – Protocol (stream) intelligence – Adaptive response • Security practitioners use these tools: – Forensic investigation of existing breaches – Analyze your environment for new threats – Examine the history of old breaches to understand how they happened Generated and prevent them in the future for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 15 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Roles ES Roles (required for ES login) ES User ES Analyst ES Admin Runs real-time searches and views all ES dashboards Owns notable events and performs notable event status changes Configures ES systemwide, including adding ES users, managing correlation searches, and adding new data sources User Power Admin Standard Splunk Roles Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 16 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Accessing ES • ES typically runs on a secure (HTTPS) port, with a URL for the Splunk server similar to: https://eshostname:8000 • Users must have an assigned role on the ES server • Once logged on, ES displays in the list of apps on the Splunk home page • Users can configure ES to be the default app to open under their Preferences – Click the user name on the top menu bar and select Preferences Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 17 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 The ES Home Page ES Menus Search events, and work issues Monitor status ES Documentation site Configuration tools Product tour tutorial Community support Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 18 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Content Management Configure > Content > Content Management Filter list by Type Filter list by App Create new content Text filter Enable, disable, or export the selected content Enable or disable Click a title to edit Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 19 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Active Correlation Searches • Select Configure > Content > Content Management • Select the Type as Correlation Search, and the Status as Enabled – Note which searches are enabled • By default, only ES Admins can enable, disable, modify, or add new correlation searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 20 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Events and Data Models in ES • As raw data is input into ES, it is processed as follows: – The raw data is converted into events and stored in indexes – Events are normalized into the Splunk Common Information Model – Events are added to accelerated data models • All ES correlation searches, dashboards, and reports use these accelerated data models in their searches • You can create your own custom searches based on the events in main or the data models as needed – Use Search > Search to create a new search – Use Search > Datasets to view or create datasets using data models Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 21 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Search Example • Use | tstats to create reports based on accelerated data models – Use | tstats summariesonly=t to restrict results to accelerated data for performance improvement • Use Search > Datasets to build datasets using ES data models Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 22 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Notable Events Example 1 3 Run a search in the notable index In the menu bar, click Search Under Selected Fields, click source 4 2 In the drop-down, click Search 5 Examine the list of sources found over the last 60 minutes, indicating which correlation searches have generated notable events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 23 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Use Case Library • The Use Case Library contains analytic stories which are ready-touse examples of how to use ES to quickly identify the scope of attacks, determine mitigation options, and take remedial action • Analytic stories: – Contain the searches needed to implement the story in your own ES environment – Provide an explanation of what the searches achieve and how to convert a search into adaptive response actions, where appropriate • The Splunk Enterprise Security Content Update (ESCU) add-on delivers analytic stories to customers as part of a content subscription service and is updated often with new stories Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 24 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Use Case Library (cont.) Configure > All Configurations > Content > Use Case Library Bookmark stories specific to your duties Choose a topic to focus on related use cases Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 25 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Use Case Specifics Expand an Analytic Story Sourcetypes use by the detection searches for this analytic story Detection Searches are correlation searches that populate the story results Recommended Data Sources that are likely to provide valuable data Lookups used by the detection searches for this analytic story Data Models used by the detection searches for this analytic story Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 26 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 1 Lab: Overview of Splunk ES Time: 10 minutes Tasks: • Log on to your lab Splunk server and navigate to the ES home page • Examine the source events ES is using to monitor the security environment and notable events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 27 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 2: Security Monitoring Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 28 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Use the Security Posture dashboard to monitor the status of ES • Use the Incident Review dashboard to investigate notable events • Take ownership of an incident and move it through the investigation workflow • Create notable events • Suppress notable events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 29 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Monitoring and Response 1. ES continually runs correlation searches for known types of threats and vulnerabilities – There are 60 built-in correlation searches, and more in the Use Case Library – Plus you can create your own 2. When a correlation search detects any Indicators of Compromise (IOC), ES creates an alert called a notable event or incident – IOC is an industry term, while notable event & incident are ES terms 3. ES enables you to track, update, and resolve incidents – Security Posture dashboard provides a cross-domain SOC overview – Incident Review dashboard to inspect and manage incidents Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 30 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 The Security Posture Dashboard • An overview of the Enterprise Security condition • Key Indicators (KI) at the top provide an at-a-glance view of notable event status over the last 24 hours • The four panels provide additional summary information categorized by urgency, time, and most common notable event types and sources Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 31 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Key Indicators Only ES Admins can edit Key Indicators Large number = total number of notable events in that category Trend of events indicator: red for increase green for decrease Black = no threshold Red = over threshold Green = under threshold Total increase or decrease of the most recent 24 hours compared to the preceding 24 hour period Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 32 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 KI Drill Down to Incident Review 1 2 From the Security Posture dashboard, click a Key Indicator total value The information for the KI opens in Incident Review Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 33 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Security Posture Panels Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 34 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Notable Event Urgency • Each notable event has an urgency field, ranging from informational to critical • Urgency is a combination of two factors: – Severity ê Based on the raw event(s) found by the correlation search – Priority ê Assigned to the associated assets or identities—i.e., the server or user ê If more than one asset or identity is involved in a single notable event, the one with the highest priority determines the urgency Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 35 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Urgency Table Shown is a partial view of the Urgency table (lookup) Asset/Identity Priority Event Severity Info Low M e d iu m Hig h Critical Low Info Low Low Medium High M e d iu m Info Low Medium High Critical Hig h Info Medium Medium High Critical Critical Info Medium High Critical Critical Asset/Identity Priority + Event Severity = Urgency Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 36 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Drill Down Support 1 Hover over an item to preview details about the underlying notable events 2 Click an item in the Security Posture dashboard to open the related notable events in the Incident Review dashboard From the Incident Review dashboard: a. Drill down into the notable details b. Take ownership c. Work the issue Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 37 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review Dashboard Filter options Timeline & job controls Urgency Expand for details Add event(s) to an investigation Actions menu Sortable column headers Notable Events Investigation bar Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 38 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review Filter Fields • Status: New, In Progress, Pending, Resolved, Closed – Along with Owner, is used to track the status of an incident • Urgency: Info, Low, Medium, High, Critical • Security Domain: Access, Endpoint, Identity, Network, Threat, Audit • Owner: The user assigned to investigate and resolve an incident • Correlation Search Name: The title of a correlation search —wildcards (*) are supported • Sequenced Event: show only events from sequence templates • Search: Splunk search language expressions • Tag: A list of tag names Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 39 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Using the Incident Review Dashboard • Select one or more values per field – More than one value per field are ORed together • Urgency values can be toggled on and off – Grey values are “off” and will not be displayed • If values are set for more than one field, the fields are ANDed together • Status, Owner, Security Domain and Tag support multiple OR values – The default All is ignored if other values are selected • Name supports wildcards, Search supports full SPL Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 40 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Notable Event Details Notable event details drop-down Notable event Actions menu Risk score Field Action menu All fields for the notable event, with Action menus for each field Note You cannot expand an event until the search is complete. Not all incidents have all the same detail items. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 41 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Create a Short ID from Event Details 1 Click Create a Short ID for ES to automatically generate a Short ID that makes it easier to find and share 2 The Short ID replaces the Create Short ID link Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 42 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Create a Short ID: Notable Event Actions 1 2 From the notable event Actions drop-down, creating a Short ID is possible using Share Notable Event In addition to creating a Short ID, this enables sharing the event via a link: • Click the Bookmark button to copy the link for sharing or • Click and drag the Bookmark button to your bookmarks bar to save the link 3 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 43 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Search for a Short ID or Investigation 1 3 2 Click Associations Select Short ID or Investigation from the drop-down In the Select… field, enter all or part of the Short ID or Investigation name (drop-down appears and filters as you type) Or Click or scroll to the Short ID or Investigation name Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 44 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Action Menus • Notable events in Incident Review have two Action menus – Each event has an Actions menu – The fields have an Action menu – And, the fields in the event details have Action menus Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 45 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Notable Event Actions Menu • Each notable event has an Actions menu with options related to the event, such as: – Adding the event to an investigation – Suppressing the notable event – Sharing the notable event with others – Initiating adaptive response actions Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 46 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Field Actions Menus • Each notable event field has an Action menu allowing you to: – investigate the asset, set tags, access other ES dashboards, analyze the data in the field, and more • Risk scores for hosts or users are displayed next to fields – Click a risk score to open the Risk Analysis dashboard for that asset or identity Note Scroll the menu to make sure you see all the available field actions. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 47 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Workflow: Concepts 1. Assign an owner 2. Investigate 3. Implement corrective measures Investigators are responsible for changing workflow status values as they work incidents ES Admins can define and add new status values, and assign values to different roles. Statuses in your environment may differ! New - not yet being worked In progress - investigation underway Pending - various: work in progress, awaiting action, etc. Resolved - fixed, awaiting verification Closed - fix verified Note When a notable is assigned an owner it is tracked as an incident in the KV Store. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 48 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Workflow: Procedures As needed, add selected event(s) to an investigation. It will appear under Related Investigations in the event details 1 2 Select one or more events Click Edit Selected 3 Set Status, Urgency, Owner, and Comment 4 Click Save changes As needed, click an icon on the Investigation Bar to view an investigation, add a new one, or perform a quick search Investigation Bar Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 49 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review History 1 Select View all review activity for this Notable Event to open a new search showing all review events for the current issue 2 The ìncident_review` macro can be used in custom searches and reports for incident status tracking by directly accessing the KV Store Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 50 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adaptive Responses • A notable event may contain adaptive responses the analyst can initiate 1 – Actions menu: select other adaptive 2 responses to execute – Adaptive Responses: displays a list of previously executed responses – Next Steps: click a suggested response • Depending on the type of notable event, different adaptive responses are available – Examples: ping host, change risk, run a script, nslookup, send to UBA, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 51 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Creating and Suppressing Notable Events • By default, ES Analysts do not have permission to perform these actions – An ES Admin must enable these capabilities for ES Analysts • Manual creation: useful when you have source event data that has not (yet) been identified by ES as suspicious, and you want to create a notable event that will identify the issue and allow you to track it • Suppression: useful if you are getting false positives from a host or a user, and you want to exclude future notable events from that host or user Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 52 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Creating Notable Events • Create an ad hoc notable event – For instance, you find an event in Splunk that has not triggered a correlation search, but you feel it should be investigated • Steps: Run a search on the source events 2. Expand an event and select Event Actions 3. Select Create notable event 4. Enter the desired data for the notable event 5. Click Save 1. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 53 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Suppressing Notable Events You may suppress notable events that are false positives, like a server temporarily misconfigured • From Incident Review: Expand the notable event's Actions menu 2. Select Suppress Notable Events Note end date is optional. If left 3. Set description and The blank, all future notable events from the dest field AND signature dates are suppressed. 4. Click Save 1. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 54 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Managing Notable Event Suppressions • After you create a new notable event suppression, you will see the list of suppressions as a confirmation – You can access this list via Configure > Incident Management > Notable Event Suppressions • Only ES Admins can edit these suppressions by default Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 55 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 2 Lab: Monitoring with ES Time: 40 minutes Scenario: An expired user account has been detected attempting to log on to high priority resources Tasks: • Use the Security Posture dashboard • Research unauthorized network access • Begin working the issue • Resolve the issue Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 56 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 3: Incident Investigation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 57 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Review the Investigation dashboard • Start an investigation • Review the Investigation Workbench • Add artifacts to an investigation • Assign collaborators and update the status of an investigation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 58 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigations • An investigation is a collection of activities and notes related to work done on a specific issue, such as a breach or other incident • Someone often starts an investigation when they receive notice of a notable event, alert action, email, help desk ticket, or phone call • Investigations are organized chronologically into timelines • Investigations can be managed by one or more analysts • Use investigations to: – Visualize progress – Document work – Share information Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 59 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add Entries to an Investigation It is important to add items to investigations to document the purpose of the steps you have taken to research the issue and to provide any details that may be useful to your team’s future investigation work. You can add several types of entries: • Notable Event • Action History • Splunk Event • Notable Event Suppression Update • Search String • Panel Filter • Note • Attachments Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 60 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Basic Workflow 1. Open an investigation and as you progress, continue to add collaborators – keep the investigation status updated – 2. Explore any Investigation Workbench artifacts that were automatically populated from notable events 3. Add more artifacts, events, actions, searches, notes and other documentation that may be related 4. Filter on specific elements and time ranges across the timeline 5. Close the investigation after completing it 6. Review and share the Investigation Summary as needed Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 61 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigations Dashboard Lists all investigations Add investigations Filter investigations Click an investigation to view its related entries (default view is Workbench) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 62 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Start an Investigation • By default, only ess_admin and ess_analyst have permission to start investigations. They can start investigations in several ways: – from the Incident Review dashboard’s Actions menu – on the Investigations dashboard – from any ES dashboard using the Investigation Bar at the bottom of the ES window – when searching raw events, from the Event Actions menu • Each investigation has one owner and can have any number of additional collaborators – Only owners and collaborators can work and modify the investigation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 63 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add Collaborators to an Investigation Hover over a collaborator to view the name, or click to edit Click to add a collaborator Select a user to change write permissions or remove as a collaborator Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 64 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Update Investigation Status • When you open an investigation, its status is New 1 • Investigations can only be deleted by admins • Analysts can delete investigation entries Edit the Title, Status, and Description of the investigation 2 3 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 65 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Explore The Investigation Workbench 1. From Incident Review, select suspicious incidents 2. Add the events to a new or existing investigation and open it 3. The Investigation Workbench opens. In the left panel under Artifacts, select the assets and identities you find interesting and click Explore (the panels at the right will populate with related data) 4. Drill down into the data panels to examine context, endpoint data, network data, risk scores, alerts, and more 5. Add more data and documentation in the form of artifacts, searches, and notes Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 66 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Workbench Panel Data Context Panel E nd p oi nt D at a P anel Network Data Panel • Risk Scores • File System Changes • Web Activity • IDS Alerts • Registry Activity • Email Data • Notable Events • Process Activity • Network Traffic Data • System Vulnerabilities • Service Activity • DNS Data • Latest OS Updates • User Account Changes • Certificate Activity • Computer Inventory • Port Activity • Network Session Data • Authentication Data Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 67 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Workbench Panels Change time range Change panel Toggle panel description 1 Select Artifact(s) Expand panel view 2 Add selected to the workbench Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 68 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Artifacts • Artifacts are assets or identities you may add to an investigation to determine whether they are involved in the overall incident • There are several ways to add an artifact to an investigation – From a notable event (set up by an admin) ê Actions > Add Event to Investigation – Manually ê Add Artifact button ê Investigation bar’s Add Artifact icon – From a workbench panel (select any item) – From an investigation event (Timeline View > Details > click a value) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 69 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Artifacts (cont.) Artifacts search filter 2 3 1 Enter artifact information and select Type Click a value to open the Add Artifacts window and add it as an artifact Add this artifact to workbench Add other artifacts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 70 Also opens the Add Artifacts window Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Steps to Add Artifacts Manually 1. Click Add Artifact button or click 2. Select Add artifact or Add multiple artifacts and enter the artifact(s) you want to add (all the artifacts you add on this window must be the same type: either assets or identities) 3. Select either Asset or Identity artifact 4. For multiple artifacts, for separator click New Line or Comma 5. Optionally, add a Description and Label(s) (separate labels with <Enter> or <,>) 6. Optionally, Expand artifact (seeks correlated items from lookups) 7. Click Add to Scope Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 71 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Summary View Expand for details Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 72 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Timeline View Change view Scroll left (newer) Filter by type Add collaborator Filter by search Edit, delete, or open in incident review Scroll right (older) Click an item to see details in upper panel Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 73 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Timeline Details View to Add Artifacts 3 Add Artifacts view opens and auto-populates 3 1 Click Details to view all fields and values 2 4 Click an item to add it as an artifact 4 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 74 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Edit the Title of an Investigation Entry 1 2 From the Action menu choose Edit Entry Change the Title and click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 75 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Bar The Investigation Bar displays at the bottom of many ES windows and dashboards including Security Posture and Incident Review Enable Livefeed Add Artifact Quick Search Add Notes Action History Select or add an Investigation Investigation Bar Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 76 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Bar and Inline Timeline View From the Investigation Bar, toggle to the Timeline view Inline Investigation Timeline Timeline Zoom 1 Investigation Entries Jump to start Select an Investigation 2 Toggle to Timeline view Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 77 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Enable Notable Event Livefeed • Get a visual notification when any related notable events occur for the investigation • Select an investigation, click the bell icon, and enable notification • Bell icon turns orange within five minutes of the occurrence • Acknowledge events and add them to the investigation Enable Livefeed Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 78 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Quick Search Click and drag to resize the search window. Double click to toggle full screen to minimized 2 4 3 Enter search criteria Determine whether the results are useful to the investigation 5 1 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 79 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add Events Add notable events from Incident Review Or Add source events from a search results window Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 80 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Timeline: List View From Timeline, change view to List View Use the Action menu to delete selected entries Search which entries to view View details Filter entries by Notable Event, Adaptive Response Action, etc. Edit or delete entries or open in Incident Review Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 81 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Investigation Notes • Use notes to add textual information and attachments relevant to the investigation – Why you ran each search and what the results mean for your investigation – Add screenshots from outside sources as attachments • There are two types of notes, standard and timeline – Timeline notes show up in the Timeline Slide View, while standard notes do not Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 82 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add a Note 1 2 Enter a title 2 Click to add a note 3 Modify time as needed default = now Enter comments 4 1 Click to view notes Add attachments (text or binary format). 4MB max per file and are stored in KV Store. 5 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 83 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add a Note (cont.) • Standard and Timeline notes use the same Create Note window – To create a timeline note, and have it visible on the Timeline Slide View, select Add new Timeline Note, or the Show on timeline box – A standard note will not display in the Slide View, but will display in the List View prepended with “Draft:” Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 84 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Action History Items • Action History is a reverse chronological list of all of your activities in ES, such as searches run and dashboards used – Add pertinent entries from your Action History to to an investigation and supplement them with notes Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 85 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add Action History Items 1 2 Select type Modify time as needed 4 5 3 Filter search as needed Select items 6 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 86 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Customizing Investigation Workbench • ES admins can customize the Investigation Workbench by: – Adding new types of panels and tabs – Creating investigation profiles that correspond to specialized investigation types – Applying profiles to notable events from correlation searches • Example: – Create a profile called Ransomware Attack, that is applied to notable events created by a malware correlation search that detects ransomware, and enables an extra tab named Ransomware that displays detailed searches focused on known ransomware threats docs.splunk.com/Documentation/ES/latest/Admin/Customizeinvestigations Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 87 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 3 Lab: Investigating an Issue with ES Time: 40 minutes Scenario: Remove false positive events for test servers Tasks: 1. Test workstation status 2. Remove the false positives from the list of incidents 3. Suppress notable events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 88 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 4: Analyst Tools & Data Sources Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 89 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Identify ES security analyst tools • Map security tools to data sources • Examine management tools for analyst dashboards Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 90 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Security Intelligence • Besides Security Posture and Incident Review, ES provides many dashboards and tools for security practitioners – Risk analysis – Protocol analysis – Threat intelligence – User intelligence – Web intelligence – Asset and identity viewers – Glass tables Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 91 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Dashboard Data Dependencies • Each dashboard panel’s search pulls events from a data model • If a panel is missing information, examine the panel’s search to see which data model is used; this can help you understand why the data is missing • Causes: – The data is not in Splunk: install add-ons to input the data – The data is present in Splunk but is not normalized correctly: modify normalization settings docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 92 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Risk Analysis • Correlation searches can add a numeric risk value to objects (systems or users) • Risk can be increased by any event that occurs to an object • The amount of risk assigned can be configured per-object and per-event • This is different than priority, severity, or urgency – Allows you to see cumulative risk caused by multiple events over time – Allows you to fine-tune the way you interpret threats or vulnerabilities to your enterprise • Admins configure risk values to correlation searches and objects Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 93 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Risk Analysis Dashboard Security Intelligence > Risk Analysis Filters Manually add risk Key Indicators Timeline shows most active risk-increasing events Events causing the most risk Objects with most risk Events affecting risk scores Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 94 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Risk Data Sources • Note that the Risk data model is the data source for the panels on the Risk Analysis dashboard • Each panel has its own search • Use a search like |datamodel Risk All_Risk search to see the sources, sourcetypes and indexes that are being used in this data model Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 95 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Protocol Intelligence • Protocol Intelligence in ES provides a set of tools to analyze network stream data • Access the protocol intelligence dashboards using Security Intelligence > Protocol Intelligence • Protocol Center: overview of network activity • Traffic Size Analysis: Overall network traffic activity trends • Three main subject groups, each with an activity overview and deep search capability, for DNS, SSL, and email Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 96 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Stream Events • Stream events are generated from the Stream app – Optional install • Stream events are stored with stream:xxxx source types – Examples: tcp, udp, dns, smtp, http • Standard field extractions: – Capture time, type, size, source/destination info • Depending on specific source type, additional fields are extracted – HTTP: cookies, request parameters, etc. – SMTP: sender, receiver, subject, summary of body – DNS: DNS query, query type, DNS host, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 97 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Protocol Intelligence: Protocol Center Security Intelligence > Protocol Center Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 98 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 User Intelligence: Assets & Identities • Assets are devices, such as routers, servers, etc. – Assets are identified by IP address, MAC address, or host name • Identities are people – Identities are identified by user name, email address, etc. • Both assets and identities are managed in ES with lookup tables – ES can show a meaningful name instead of an IP address or user ID – You can define watchlists for both assets and identities • Asset and identity lookups are customized for your environment by an ES Admin – Discussed in detail later in the course Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 99 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Access User Activity from Action Menus • After running a search, you can open the Action menu for the user field and select User Activity – Opens the User Activity dashboard displaying account activity for the specific user Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 100 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 User Intelligence > User Activity Security Intelligence > User Intelligence > User Activity Users accessing external sites that have been added to a watchlist Risk assigned by various correlation searches on user activity External email Users accessing watchlisted sites Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 101 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Access Anomalies Security Intelligence > User Intelligence > Access Anomalies This dashboard is dependent on the gia_summary index, which is filled by the Access - Geographically Improbable Access - Summary Gen scheduled search hourly. This search is disabled by default; enable it to use this dashboard. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 102 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Using the Access Anomalies Dashboard • Searches for user access during the requested time period, defaults to 60 minutes • Displays user access events with locations more than 500 miles from their previous access location • The distance (miles) and speed (miles per hour) between locations yields an indicator of improbability for a user to actually log in from both locations • Many access events spanning a short time from many geographically remote locations is suspicious Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 103 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Web Intelligence The Web Intelligence menu contains analysis dashboards that are useful for inspecting various aspects of your website network activity HTTP Category HTTP User Agent New Domain URL Length Explore the types of websites being accessed through your network Examine the web user agents being used on your network See what external domains are being accessed Examine request URLs for unusual contents Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 104 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Web Intelligence: HTTP Category Analysis • Overview of websites used in your organization by category • Categories are defined by Websense www.websense.com/content/support/library/web/v76/siem/siem.pdf Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 105 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Per-panel Filtering • Analysis dashboards provide highlighting or filtering of items on dashboard views – After you have determined that an event is not a threat, you can add it to your whitelist to remove it from the dashboard view – If an event is determined to be a threat, use the Advanced Filter editor to add the item to your blacklist of known threats • This feature is unavailable by default for ES Analysts – Can be enabled by an ES Admin • For instance, for HTTP Category Analysis you may want to filter out expected categories and highlight unwanted categories Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 106 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Creating Per-panel Filters 3 2 Choose to either filter or highlight 4 Click Per-panel Filter 1 Select one or more events in the dashboard Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 107 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Filtered vs. Highlighted Events • Filtered events are no longer displayed • Highlighted events are marked yellow in the Per-panel Filter column and are displayed at the top of the list by default Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 108 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Managing Per-panel Filtering Lookups • Edit filters in the corresponding lookup table • Access the lookup table by – clicking View/edit existing filters in the Per- panel Filter window Or – by selecting the lookup table under Configure > Content > Content M a n a ge m e n t Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 109 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Per-panel Filter Audit Audit > Per-panel Filter Audit • Display data on per-panel filter usage • See who creates per-panel filters, and what data is being filtered Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 110 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Security Domain Dashboards • The Security Domains menu provides access to analytical dashboards organized by security domain • Each set of dashboards contains tools to search source events related to a domain, such as Network or Endpoint • These dashboards display results from original source events accessed via accelerated data models • Use the dashboard requirements matrix or examine panel searches to Generated determine specific data model dependencies for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 111 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Access Domain: Access Center Security Domains > Access > Access Center Use filters to focus on specific types of events Key Indicators show overview of notable events and trends over previous 24 hours compared to 24-48 hours before Panels show summaries of Access notable events over time by action, app, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 112 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Access Domain: Access Search Security Domains > Access > Access Search Search events specific to Access domain Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 113 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Glass Tables • Glass tables are custom views that can display security indicators as well as symbols, icons, and graphics • Glass tables are visualizations that can be used for status displays or to enhance understanding of security status • A glass table can display current or past information • Glass tables are stored in the KV Store • Access can be controlled by roles • Accessed on the Glass Tables menu Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 114 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Glass Table: Standard View Custom icons Gauge indicators Select time Toggle edit mode Contextual graphics Text Metrics with threshold colors and trend metrics Timelines Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 115 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Glass Table: Edit Mode Controls Tools Settings for selected widget Metrics Work area Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 116 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Glass Table Data Requirements • Glass table metric values are dependent on ad-hoc searches, or Key Indicators – Ad-hoc searches: SPL that results in a single value to display – Key Indicators are the values displayed at the top of many dashboards, such as Security Posture • Glass tables are maintained in the KV Store: – Application: SplunkEnterpriseSecuritySuite – Collection: SplunkEnterpriseSecuritySuite_glasstables • Glass tables can be exported and imported in Configure > Content > Content Management Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 117 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 4 Lab: Dashboard Data Sources Time: 15 minutes Tasks: • Examine the dashboard panel data sources Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 118 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 5: ES Deployment Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 119 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Identify deployment topologies • Examine the deployment checklist • Understand pre-deployment requirements Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 120 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Deployment Checklist 1. Determine size and scope of installation 2. Configure additional servers if needed 3. Obtain the ES software 4. Determine software installation requirements for search heads, indexers, and forwarders 5. Install all ES apps on search head(s) 6. Deploy indexer configurations Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 121 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Impact on Resources • ES generally requires a new, dedicated search head or search head cluster – ES is only compatible with other CIM-compatible apps – ES adds a large number of searches and search results • Hardware must meet or exceed Splunk minimum requirements: docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware • ES increases some hardware requirements: docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Splunk _Enterprise_system_requirements Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 122 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Supported Architectures • Single server (proof of concept, testing, dev) • Distributed search (single search head, multiple indexers) • Search head clustering docs.splunk.com/Documentation/ES/latest/Install/InstallEnterpriseSecuritySHC • Indexer clustering (including multi-site) docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 123 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding ES to an Existing Site Before ES After ES After ES install, ES increases search requirements, adds an extra search head and 2 more indexers Pre-ES site with a single search head and 3 indexers supporting ~500GB/day of indexed data Log on here for ES Log on here for Splunk search ES Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 124 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Search Head Requirements • A dedicated server or cluster for the ES search head(s) with only CIM-compliant apps installed • 64 bit OS, minimum 32 GB and 16 processor cores – Additional memory and CPU capacity may be needed depending on number of concurrent users, searches, etc. • Configure search head forwarding: docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata • If enabling Monitoring Console, do not use distributed mode docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Monitoring_Console Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 125 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Indexer Requirements • Increased search load in ES typically requires more indexers – Assume at most 100GB/day per indexer – Hardware minimum: 16 CPU cores, 32 GB RAM • The exact number of indexers required depends on: – Types and amounts of data being used by ES – Number of active correlation searches – Number of real-time correlation searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 126 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Indexer Cluster Requirements • You can only enable ES on one search head or search head cluster for each indexer cluster • On a multisite indexer cluster: – Enable summary replication to improve performance docs.splunk.com/Documentation/Splunk/latest/Indexer/Clustersandsummaryreplication – Disable search affinity docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity • Make sure you use the indexer cluster master to deploy any ES add-ons to the indexer tier Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 127 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Accelerated Data Model Storage • In addition to index storage requirements, ES requires space for accelerated data models • Acceleration requires approximately 3.4 x (daily input volume) of additional space per year, or more if replicated in an indexer cluster • Example: input volume of 500 GB per day with one year retention – 500 GB * 3.4 = 1700 GB additional space for accelerated data model storage • Space is added across all indexers – Example: if there are 5 indexers, 1700 GB / 5 = ~ 340GB per indexer additional space is required Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 128 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 More About Accelerated Data Models • Most ES searches are executed on accelerated data models • The storage volumes allocated for acceleration should be tuned for best performance and replicated if in a cluster • By default, acceleration storage is allocated in the same location as the index containing the raw events being accelerated • Use the tstatsHomePath setting in indexes.conf if needed to specify alternate locations for your accelerated storage docs.splunk.com/Documentation/ES/latest/Install/Datamodels#Configuring_storage_volumes Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 129 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Indexed Real Time Search • ES automatically configures Splunk to use indexed real time searching docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches #Indexed_real-time_search • Improves concurrent real time search performance at the cost of a small delay in delivering real time results from searches • Leave turned on in ES for best performance Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 130 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Forwarder Requirements • In general, forwarders are unaffected by ES installation • However, some add-ons that ES depends on must be deployed to forwarders to collect data • Examples: – Windows add-on – *NIX add-on – Splunk Stream add-on Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 131 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 App/Add-on Deployment Options • Depending on your requirements, you may need to distribute addons to other Splunk instances like search heads, indexers, and heavy forwarders • Use the appropriate app and add-on deployment methodology: – Forwarders and non-clustered Indexers: use Forwarder Management (Deployment Server) – Indexer clusters: use the master node to deploy apps to peer nodes – Search head clusters: use the deployer to deploy apps to cluster members docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 132 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder • splunkbase.splunk.com/app/2962/ • Builds add-ons for custom ES data • Normalizes custom data into the Common Information Model • Built-in validation • Should not be used on production servers Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 133 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Important Resources • Splunk Data Administration, Splunk System Administration, Splunk Cluster Administration, and Architecting and Deploying Splunk courses from Splunk Education • Distributed Splunk overview: docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverview • Capacity planning: docs.splunk.com/Documentation/Splunk/latest/Capacity/Accommodatemanysim ultaneoussearches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 134 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 5 Lab: Plan a Deployment Time: 20 minutes Scenario: You are working with a client that has Splunk installed on a distributed site. There is one search head and 4 indexers indexing 800 GB/day, with a retention period of 1 year. The customer is marginally happy with current performance. All servers are at basic minimum Splunk hardware levels. No new inputs are planned after installing ES. What alterations to their configuration do you suggest before installing ES? Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 135 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 6: Installation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 136 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • List ES pre-installation requirements • Identify steps for downloading and installing ES • Test a new install Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 137 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installation Checklist • Assuming prep work from previous module is completed, these are the steps for a single server or distributed (non-clustered) site: 1. 2. 3. 4. 5. Confirm the environment meets the minimum system requirements for Splunk Enterprise and ES Install ES app on search head Disable un-needed add-ons Create Splunk_TA_ForIndexers and deploy to indexers Deploy input-time add-ons to forwarders • If using deployment server to deploy ES-installed apps and add-ons, disable it before the installation, and re-enable after installation https://docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 138 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Apps and Add-ons Domain Add-ons (DA) (views, UI components) Tech Add-ons (TA) (input, normalization) Supporting Add-ons (SA) (searches, macros, data models, utilities) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 139 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Installation: ES Framework Main ES Application S u p p o r t i n g a d d -o n s • SplunkEnterpriseSecuritySuite • SA-AccessProtection • SA-AuditAndDataProtection • SA-EndpointProtection • SA-IdentityManagement • SA-NetworkProtection • SA-ThreatIntelligence • SA-UEBA • SA-Utils • Splunk_SA_CIM • Splunk_SA_ExtremeSearch Domain add-ons • DA-ESS-AccessProtection • DA-ESS-EndpointProtection • DA-ESS-IdentityManagement • DA-ESS-NetworkProtection • DA-ESS-ThreatIntelligence These add-ons are all distributed with the ES installer, and are only required to be on Splunk search heads. Generally you will not need to edit any of their configuration files directly; most settings are available via the admin user interface. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 140 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Installation: Technology Add-ons Stand-alone TAs • Splunk_TA_bluecoat-proxysg • Splunk_TA_bro • Splunk_TA_flowfix • Splunk_TA_juniper • Splunk_TA_mcafee • Splunk_TA_nessus • Splunk_TA_nix • Splunk_TA_oracle • Splunk_TA_ossec • Splunk_TA_rsa-securid • Splunk_TA_sophos • Splunk_TA_sourcefire • Splunk_TA_symantec-ep • Splunk_TA_ueba • Splunk_TA_websense-cg • Splunk_TA_windows ES TAs • TA-airdefense • TA-alcatel • TA-cef • TA-fortinet • TA-ftp • TA-nmap • TA-tippingpoint • TA-trendmicro These are only distributed as part of ES Technology add-ons (TA’s) can configure inputs on forwarders, parsing on indexers, and normalizing on search heads. See next slide for deployment. More TA’s are available from Splunkbase for other technologies. These can also be downloaded from Splunkbase Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 141 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 What Gets Installed Where? • Install the full ES app on the search head – Installs all DAs, SAs, and TAs • Create and install Splunk_TA_ForIndexers on indexers and heavy forwarders Note – Includes all configurations from PS recommends not using the indexes.conf included with Splunk_TA_ForIndexers. Instead, use it to build a new indexes.conf that fine tunes the index properties of the app. all enabled TAs, as well as indexes.conf settings • Install TAs on forwarders if they do input phase actions – See TA readme files and their inputs.conf and props.conf files Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 142 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Typical Server Architecture Forwarders input-time TAs Indexers & heavy forwarders ES index configurations And index-time TA configurations (via Splunk_TA_ForIndexers) Universal Forwarders gather operational and security data and send to indexers or heavy forwarders Search Head(s) ES app + all DAs, SAs and TAs Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 143 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installing on a Single Search Head • Start with a clean basic Splunk installation • Do not uninstall any of the default apps which are part of the basic Splunk package, as they are required by ES • ES functions best without the installation of additional apps on top of the basic Splunk package Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 144 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Uploading the ES App 1. Obtain the ES App from SplunkBase/sales rep 2. Upload ES App on the designated ES search head Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 145 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Starting the Installation • After re-starting Splunk, navigate to the ES app • You will be prompted to set up the app – Click Continue to app setup page to begin the installation process Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 146 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installation: App Management • The first step of the installer allows you to select apps to either exclude or disable • Select any apps to disable or exclude, then click Start Configuration Process to continue Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 147 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installation: Complete When the installation process is complete, you’ll be prompted to restart the server again Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 148 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Splunk Web Now on HTTPS • ES converts Splunk Web to HTTPS – Port is not changed – You can change the server back to HTTP in web.conf if desired • The pre-loaded SSL certificates are self-signed – This causes a browser warning, but they are completely secure – You can install your own externally validated certificates docs.splunk.com/Documentation/Splunk/latest/Security/Howtogetthird-partycertificates Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 149 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Is Installed on the Search Head! • ES also installs: – Extreme Search ES – *Nix or Windows add-on Extreme Search • The Stream app, if installed, can be integrated with ES Add-on Builder • If you want additional add-ons or apps like the Splunk Add-on Builder or ES Content Updates, you will need to install them yourself *Nix Add-on Stream App ES Content Update Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 150 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Standard ES Add-Ons • ES ships with several add-ons for common security data sources – For a complete list: docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons • Each add-on is related to a specific vendor product or technology • Each has a specific add-on name and one or more event sourcetypes • Some, like the FTP, *NIX, and Windows add-ons, are designed to input OS data and will require configuration before use • See the README file in each add-on to for configuration steps Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 151 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Disable Unused ES Add-ons • Tech add-ons are intended for use with specific technologies – For example, Splunk Add-on for Websense, Splunk Add-on for Trend Micro, etc. • If not already done at install, disable add-ons for products you are not using on search heads, indexers, and forwarders Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 152 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installing ES on a Search Head Cluster • The installer will dynamically detect if you are installing in a single search head environment or search head cluster environment • Install ES on the Deployer On the Splunk toolbar, select Apps > Manage Apps and click Install App from File 2. Click Choose File and select the Splunk Enterprise Security file 3. Click Upload to begin the installation 4. Click Continue to app setup page 5. Click Start Configuration Process, and wait for it to complete 6. Use the Deployer to deploy ES to the cluster members. From the deployer run: splunk apply shcluster-bundle 1. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 153 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Configuration Page Navigate to ES > Configure > All Configurations Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 154 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Distributed Configuration Management • ES > Configure > General > Distributed Configuration Management – Download Splunk_TA_ForIndexers ê Creates the Splunk_TA_ForIndexers.spl add-on ê Collects index-time configurations and basic index definitions into one package to simplify the deployment of add-on configurations to on-premises indexers – Download Splunk_TA_AROnPrem ê Creates the Splunk_TA_AROnPrem.spl add-on that is used when setting up an adaptive response relay from an ES Cloud search head to an onpremises heavy forwarder https://docs.splunk.com/Documentation/ES/latest/Admin/Adaptiveresponserelay Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 155 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Deploy Indexer Configurations • To created the Splunk_TA_ForIndexers.spl, click Download Splunk_TA_ForIndexers • Set auto deployment to No, to manually send the .spl to the indexers • Select at least one option and click Download the Package props.conf and transforms.conf indexes.conf – Include index time properties: includes the props.conf and transforms.conf files in the package – Include index definitions: includes the indexes.conf file in the package • Copy the downloaded .spl to your indexers Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 156 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Integrity Control • You can optionally enable data integrity control to ensure that the data ES relies on in indexes is not tampered with • Data integrity applies hashes on all indexed data • Configure in indexes.conf or on the index properties in settings – Set enableDataIntegrityControl to true and re-start server – Only new inputs will be hashed – Can be set per-index or globally • Test integrity from the command line or script: bin/splunk check-integrity –index <indexname> Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 157 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Protection Audit Audit > Data Protection • Displays status of data protection settings per index • Also displays status for sensitive data if the Personally Identifiable Information Detected correlation search is enabled Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 158 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Splunk Stream and ES • ES can use wire data captures from the Splunk Stream app – Supports Protocol Intelligence • Install the Splunk Stream app on the ES server • Install the Stream add-on (Splunk_TA_stream) on machines where you want to capture data • Details on installing and configuring Stream: docs.splunk.com/Documentation/StreamApp • Details on integrating Stream with ES: docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 159 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Stream Data Flow Splunk ES With Stream app Execute and display search results Production Servers with forwarders and Stream add-on Capture network data and forward to indexers Indexers Store captured stream data Captured data does not include message content unless specifically configured Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 160 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 6 Lab: Post-installation Tasks Time: 15 minutes Tasks: • Disable un-needed add-ons • Create an app package for your indexer(s) (Splunk_TA_ForIndexers) • Enable special role capabilities Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 161 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 7: Initial Configuration Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 162 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Configure user roles and capabilities • Set general configuration options • Add external integrations • Configure local domain information • Customize Key Indicators • Customize navigation and view permissions • Customize incident review settings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 163 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Initial Configuration • There are several general configuration tasks to perform before you begin to use ES: – User roles and capabilities – General configuration options – External integrations, such as UBA or domain lookup – Configure local domain information – Configure Key Indicators – Customize incident review settings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 164 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 General ES Configurations ES > Configure > General > General Settings • Set or modify various ES parameters Filter by app and/or text • Example: indexed real time on or off, and changing the indexed real time delay Modify values https://docs.splunk.com/Documentation/ES/latest/Admin/Generalsettings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 165 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Roles • You can put users in the ES User or ES Analyst role as needed • Users should not be added to the ES Admin role • Add the ES Admin role as an inherited role to a regular Splunk role—such as Admin – All users in the Splunk Admin role also inherit ES Admin abilities docs.splunk.com/Documentation/ES/latest/Install/ConfigureUsersRoles Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 166 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Enabling Role Capabilities ES > Configure > General > Permissions • Enable or disable capabilities for the ess_analyst or ess_user role • Example: many sites want to allow analysts to manually create or suppress notable events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 167 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuration Check Configure > General > Configuration Checker • A list of automatic checks on ES configurations • Generally leave alone, unless directed by Splunk Support Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 168 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Local and Cloud Domains • Some correlation searches need to differentiate between your local domain vs. external domains – For instance, if you work at Acme Corp, you may have local domains ending in acme.com, acmecorp.com, etc. • Also, there are external cloud domains you may use frequently that are not suspicious – External vendors for accounting, expenses, document sharing, etc. • Your email system may use different email domains from your standard corporate domain – Due to acquisitions, mergers, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 169 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing Domain Tables • Select Configure > Content > Content Management and select Type: Lookup • Click to edit any of the following lookups: – Corporate Web Domains: domains in your enterprise – Corporate Email Domains: email domains – Cloud Domains: external vendor sites • Right-click a row, select insert row below and add your domains – Right-click and delete any sample rows • Click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 170 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Domain Analysis • The New Domain Analysis dashboard relies on domain name lookup information retrieved via a modular input from domaintools.com Add your domaintools.com credentials in the Credentials Manager 2. Configure the settings for the Network Query input 3. Enable whois checking 4. Check for events in the whois index 1. docs.splunk.com/Documentation/ES/latest/User/ThreatListActivitydashboard#Configure_the_ext ernal_API_for_WHOIS_data Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 171 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Credentials • In ES, navigate to Configure > General > Credential Management and click New Credential • Enter the domaintools.com credentials, an app, and click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 172 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring the whois_domaintools Input • Select Configure > Data Enrichment > Whois Management • Edit the whois_domaintools entry: – API Host: URI to your account’s server – API User: your domaintools.com user name (password will be retrieved from credential manager automatically) – App: the app you stored the credentials in – Leave other fields with default values unless you have a proxy or want to alter defaults for queue interval, etc • Click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 173 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Enabling the Domain Analysis Setting • Modify the domain analysis setting – Navigate to Configure > General > General Settings – Change the Domain Analysis setting to Enabled • The whois system is now enabled – Domain name lookup happens when events with IP addresses are indexed – Domain info is stored in the whois index and used by the New Domain Analysis dashboard Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 174 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES and User Behavior Analytics (UBA) • Splunk User Behavior Analytics (UBA) is a separate solution that extends your ability to detect insider threats • UBA can forward insider threat intelligence to ES docs.splunk.com/Documentation/TA-ueba/latest/User/UsetoIntegrate • ES can forward notable events to UBA for insider threat analysis docs.splunk.com/Documentation/ES/latest/User/SendUBASearchResults Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 175 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Integrating Splunk UBA • Sending ES notable events to UBA – Configure > UBA Setup – Enter UBA server host and port and select protocol • Send UBA insider threat intel to ES – This integration is on the UBA side Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 176 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Key Indicators • Key Indicators (KIs) appear in many ES views • By default, KIs do not have a threshold set, so the current count is displayed in black • You can configure thresholds for each KI – If the count is above the threshold, the value is shown in red – Green indicates a value below the threshold • You can also re-order, delete, or add KIs • Click Edit to display the edit tools Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 177 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing Key Indicator Order Save changes Cancel Drag and drop to re-arrange Remove KI from display Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 178 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add a New Key Indicator Save Cancel Add a new Key Indicator Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 179 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Changing KI Thresholds • You may want to use different threshold values – For instance, if you have a very large organization, you may expect a few minor security threats per day, and therefore would want to increase some of the thresholds above their defaults • Edit the Key Indicator panel • Enter a value in the Threshold field • Save the new panel settings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 180 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing Key Indicator Searches • Configure > Content > Content Management and select Type : Key Indicator • Select a search name to edit indicator search definition – Click Edit Acceleration to configure an acceleration search schedule • To make a new Key Indicator search, click Create New Content > Key Indicator Search Click to edit Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 181 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding a Key Indicator Search: 1 • Enter Name, App, Title and Subtitle Add a search that generates a current and delta value • Drilldown URL can be a search, dashboard or view to open on click • Add optional acceleration settings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 182 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Example Key Indicator Search • Example Key Indicator search: – The `get_delta` macro looks for fields current_count and historical_count and outputs delta – The two counts should be based on the two previous 24 hour periods – Use tstats `summariesonly` if possible for performance | tstats `summariesonly` count as current_count from datamodel=Risk.All_Risk where All_Risk.risk_object_type="user" All_Risk.risk_score>60 earliest=-24h@h latest=+0s | appendcols [|tstats `summariesonly` count as historical_count from datamodel=Risk.All_Risk where All_Risk.risk_object_type="user" All_Risk.risk_score>60 earliest=-48h@h latest=-24h@h ] | `get_delta` Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 183 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding a Key Indicator Search: 2 • Value contains the current value for the previous 24 hour period • Delta contains the difference between the value for the previous 24 hour period and the preceding 24 hour period • Rendering Options for threshold coloring, suffix notation, and inversion • Click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 184 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review KV Store • All incident review status changes and comments are stored in the incident_review KV Store collection • Use the ìncident_review` macro to retrieve information from this lookup • Example: you are working on a new incident that is similar to one you worked on before and you want to search for comments related to the incident |ìncident_review` | search comment = "*...text...*" Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 185 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review KV Store Maintenance • Periodically clear data from the incident review KV Store: | inputlookup incident_review_lookup | eval age = (now()-time)/86400 | search age < 30 | fields - age | outputlookup incident_review_lookup append=f • Use the splunk clean command to completely clear out the incident review collection: splunk clean kvstore -app SA-ThreatIntelligence -collection incident_review • Splunk must be running to use splunk clean kvstore Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 186 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Incident Review Status and Settings • Configure > Incident Management > Incident Review Settings – Allows analysts to change notable urgency (default = yes) – Requires comments when changing status (default = no) – Sets minimum comment length – Customizes field display • Configure > Incident Management > Notable Event Statuses – Changes the names of default statuses or adds new ones – Controls the permissions for statuses by role—for instance, restrict who can transition an incident to closed or resolved – Applies to status values for incidents and investigations Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 187 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Modifying Urgency Calculation • You can change the matrix that determines how correlation severity and asset/identity priority combine to set urgency – Select Configure > Content > Content Management and edit the Urgency Levels lookup – Each row is one combination of priority and severity, with the resulting urgency shown in the right-most column – Modify as needed, and save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 188 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Customizing Incident Review • Add or remove fields to an incident’s table or event attributes – Table = collapsed, one line per incident (default) – Event = expanded details • Example: display the src (IP) field at the right side of the table attributes – Navigate to Configure > Incident Management > Incident Review Settings – Under Table Attributes, click Insert below on the last row – Enter src for the field to be displayed, and Source for the label – Click Done Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 189 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Example: Adding a Field as a Column Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 190 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Untriaged Incident Alert • The Untriaged Notable Events correlation search can be configured and customized for your site as needed • By default, it prepares a list of all notable events in new status or unassigned owner over the last 48 hours • Configure its adaptive response actions to send email to a group, run a script, or create a new notable event with a specific owner responsible to assign incidents to analysts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 191 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Configuration Health Audit Audit > ES Configuration Health • Checks ES configurations for settings that may conflict with ES defaults • Useful to check ES status after initial configuration or upgrade Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 192 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Controlling and Customizing Views • Set permissions on dashboards and reports to control access – Only views the current user has access to are displayed in navigation • Clone views and edit to create custom alternatives Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 193 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Customizing ES Menus • The ES application enables you to customize the menu system to add, remove, or move menu items • Navigate to Configure > General > Navigation to edit the navigation menus – Use drag and drop to move menus and menu items – The checkmark icon in an item’s top-left corner makes it the default – Use the X icon in an item’s top-right corner to delete it – Click Save when finished Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 194 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Navigation i.e. from a new app or add-on Add a new menu Undo Make default Add menu items Save Edit Delete Drag and drop items to re-sequence menus Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 195 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding a Menu Item Note Add a New View adds a top-level menu item; to add an item to a menu, use the icon. Note Adding links to filtered incident review results: The view list scrolls—there are many to pick from. docs.splunk.com/Documentation/ES/latest/Admin/Customizemenubar#Add_a_link_to_a_filtered_view_of_Incident_Review Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 196 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 7 Lab: Initial Configuration Time: 20 minutes Tasks: • Configure Key Indicators • Modify dashboard permissions: – Remove access to the Predictive Analytics dashboard for analysts • Customize navigation: – Make Security Posture the default view – Make a menu item more accessible by moving it to the top row Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 197 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 8: Validating ES Data Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 198 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Verify data is correctly configured for use in ES • Validate normalization configurations • Install additional add-ons Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 199 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Data Flow • ES uses Splunk events for all correlation and analytical searches using the following process: Data is input from its source, indexed into events and a sourcetype is applied 2. Tech add-ons apply normalization configurations based on the sourcetypes that assign the events to a data model 3. The data model events are accelerated and placed into accelerated storage, with retention periods up to 1 year 4. All ES correlation searches and dashboard searches are based on accelerated data model events 1. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 200 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 From Input to Dashboard Normalization HPAS Storage Unaccelerated DM Inputs DA+SA+ ES app DM Acceleration Data Models Technology Add-on Index TA apps Inputs Data Acceleration Parsing/Indexing TA_ForIndexers Forwarders Search Time Dashboards _raw searches Notable events & summary indexes Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 201 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Data Models • ES uses data models in the Common Information Model (CIM) docs.splunk.com/Documentation/CIM/latest/User/Howtousethesereferencetables • Each data model defines a standard set of field names for events that share a logical context, such as: – Malware: anti-virus logs – Performance: OS metrics like CPU and memory usage – Authentication: log-on and authorization events – Network Traffic: network activity • Data models are conceptual maps, not containers Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 202 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Normalization • Normalization converts non-standard field names and values into a uniform set of standardized fields within a data model • Report designers can build report searches based on these standard terms without knowing where the data originally came from • Example: one sourcetype has events with an ACCESS field, containing numeric codes like 0 (access allowed) and 1 (access denied). Another sourcetype has an Action field, with values “allowed” and “denied”. After normalization, both sourcetypes will have the Action field and use the same values, making it easier to build reports Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 203 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Normalization Process • Normalization is a search-time process based on event sourcetypes and includes steps such as: – Adding tags, which control which events are displayed by which data models – Changing field names and values to conform to data model specifications • Add-ons automatically normalize most common sourcetypes • You may have to adjust normalization rules, or create new normalization add-ons for custom data Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 204 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 CIM Setup ES > Configure > CIM Setup • Use the CIM add-on to change data model settings like acceleration, index whitelist, and tag whitelist – Select a data model on the left • Enable acceleration for the data model to return results faster for searches, reports, and dashboard panels that reference the data model • For more information: https://docs.splunk.com/Documentation/CIM/latest/User/Setup#Accelerating_CIM_data_models Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 205 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 CIM Setup (cont.) ES > Configure > CIM Setup • Use Indexes whitelist to improve performance by constraining the indexes that each data model searches (by default, a data model searches all indexes) • Use Tags whitelist to restrict the tag attribute of a data model to specific tag values to improve performance – By default, whitelists configured for a data model are used as the tags for the child datasets https://docs.splunk.com/Documentation/CIM/latest/User/Setup#Accelerate_CIM_data_models Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 206 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Data Input Troubleshooting • Ideally, after installing ES you will find that all the searches and dashboards work automatically • However, if any events have non-standard sourcetypes, the normalization configurations in the tech add-ons won’t work – Example: an admin created a sourcetype and the name is incorrect – Fix: specify the correct sourcetype name in your configuration files • If you have incoming data from a technology that requires a tech add-on that does not ship with ES, you’ll have to install it • If you have custom data to use in ES, you might have to create your own TA Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 207 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Confirming Normalization • Match your enabled TAs to CIM data models and verify the events are being added to the correct data models – Use the dashboard requirements matrix to determine which data models support each dashboard docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix – Also useful: blogs.splunk.com/2015/05/01/relating-add-ons-to-cim • If a sourcetype is not showing up in a data model: – Check the sourcetype – Make sure the TA is installed Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 208 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Steps for Initial Data Verification 1. Make a list of all sourcetypes required by ES – This will be dependent on the exact set of technologies and security products in use at your site 2. Map the sourcetypes to the TA that normalizes it 3. Confirm that the correct sourcetype name is being used – Verify against the TA documentation 4. Install additional TAs if needed 5. Verify that normalization is happening – Make sure the sourcetype is appearing in the correct data model and that all searches are executing as expected Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 209 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Map Sourcetypes to Tech Add-ons 1. Match each sourcetype to the tech add-on that will normalize it – Use add-on documentation to determine which sourcetypes are supported docs.splunk.com/Documentation/AddOns 2. Make sure the correct sourcetype name is being set Change the sourcetype setting to the correct one, or – Edit the TA to use the local sourcetype name variant if necessary – 3. Install (or create) any missing tech add-ons 4. Disable un-needed ES tech add-ons Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 210 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Finding More Add-ons • Splunkbase has additional add-ons available for ES https://splunkbase.splunk.com/ • Add-ons must be CIM-compliant to be compatible with ES • Search Splunkbase and/or the add-on documentation for the vendor or technology names related to the sourcetype you are trying to normalize Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 211 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Examining Data Model Contents • Use the datamodel command to examine the sourcetypes contained in the data model |datamodel Network_Traffic All_Traffic search | stats count by sourcetype • If the sourcetype is present, the events are correctly tagged and fields can be checked for normalization • If the sourcetype or fields are missing: – Locate an add-on in Splunkbase that corresponds to the vendor or technology for the sourcetype, or – Build your own Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 212 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Problem: Missing Cisco ASA Events 1. As you audit the data in Splunk, you find that you want to use events from the Cisco router logs with sourcetypes cisco:asa and cisco:fwsm 2. Confirm that this data is present in Splunk indexes, but ES is not displaying it in any dashboards 3. The Network Traffic data model does not contain these sourcetypes – This is because the events are not being tagged with the network and communicate tags, and also the fields are not being aliased to the proper names required in the data model Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 213 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Solution: the Cisco Add-on • In Splunkbase, the Splunk Add-on for Cisco ASA is: – CIM-compliant – Designed for use with ES • Source types: – cisco:asa: Authentication, Change Analysis, Network Sessions, Network Traffic, Malware – cisco:fwsm: Authentication, Network Sessions, Network Traffic – cisco:pix: Authentication, Network Sessions, Network http://docs.splunk.com/Documentation/CIM/latest/User/Authentication Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 214 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Installing Add-ons on the ES Search Head • Not all add-ons and apps require you to restart Splunk. Check the add-on documentation on Splunkbase for individual instructions • Knowledge objects in add-ons and apps that are installed on the same search head as ES, and are exported to other apps or exported system-wide (export = system) are automatically visible in ES • Check the TA Readme file for specific add-on information – If it indicates it does index-time actions, re-generate and re-deploy the Splunk_TA_ForIndexers add-on – Carry out any additional TA setup in the Readme Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 215 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Content Profile Audit Audit > Content Profile • Maps data models to searches and dashboard panels • Quick indication of missing data • Usually means either a data source has not been configured or normalization is not complete Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 216 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Model Audit Audit > Data Model Audit • Determine which data models are using the most storage or processor time • Note that you can easily see each data model’s size, retention settings, and current refresh status Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 217 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Forwarder Audit Audit > Forwarder Audit • Ensures hosts are properly forwarding data to Splunk • Detects forwarders that have failed • Can be set to monitor all hosts, or only hosts configured as is_expected in the ES Assets lookup table Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 218 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Indexing Audit Audit > Indexing Audit • Summary of events indexed per day (EPD) • Time series shows trends • Also summarized by index (main, threat_activity, etc.) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 219 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 8 Lab: Validate ES Data Time: 25 minutes Tasks: – Plan and verify inputs – Examine data model activity – Install a new Splunk technology add-on to automatically normalize Cisco ASA events Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 220 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 9: Custom Add-ons Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 221 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Use custom data in ES • Create an add-on for a custom sourcetype • Describe add-on troubleshooting Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 222 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Custom Data Input • If you have custom data sources you want ES to recognize, create an add-on to make your custom events CIM-compliant • Your add-on should contain: – – – – – Data inputs (if required) Field extractions (if required) A tagged event type that maps your sourcetype to the appropriate CIM data model Field aliases to map non-standard field names to CIM field names Eval statements (calculated fields) or lookups to map non-standard field values to CIM field values Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 223 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Models and the CIM • Your custom events are referenced by CIM data models – See the CIM documentation for a list of all the data models and their contents (docs.splunk.com/Documentation/CIM) • Once you determine which data model should reference your events, plan which CIM fields relate to your custom fields • Example: – You want the Network Traffic data model to return your events – At docs.splunk.com/Documentation/CIM/latest/User/NetworkTraffic, you see the list of required and optional fields for this data model – You make a mapping of your fields to CIM fields Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 224 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Normalization Strategy • Not all of the source fields will match CIM fields – You can ignore the extra source fields, or omit them, as appropriate • Not all of the CIM fields will be present in the source events – Use eval statements or regex-based field extractions to generate these fields with valid values if possible, or with placeholder values if no valid values can be determined • Should you populate every CIM field in the target data model? – You need to at least populate the fields used by ES dashboards and correlation searches – Mapping as many of the data model fields as possible will make your events more robust for future use in new views, searches or reports Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 225 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Planning Normalization Requirements • Determine the dashboards that will display your events • Use the dashboard requirements matrix to determine the data model(s) and field names the dashboard(s) require: docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 226 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Model Definitions • The data model names in the dashboard requirements matrix are linked to the data model’s CIM documentation • Use this documentation to determine the tags, field names and field values your events must use to be CIM-compliant Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 227 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Mapping Original Fields to CIM Fields • Plan your normalization settings using a table • List the required CIM-compliant field names • Match them to corresponding original source fields • Determine if normalization is required for each field’s name and value Original CIM Procedure sender src alias receiver dest alias method app alias user user none account unused ignore missing signature Use eval to create default value SSID unused regex to mask all but last 4 digits status action Use eval to translate source numeric codes to CIM terms ... ... Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 228 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Splunk Add-on Builder • Very fast way to build out the initial TA • Use it to create sourcetypes, extractions, and data model mapping • TAs can: – Automatically input data into Splunk – Extract fields and map fields to the CIM – Create alert actions https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 229 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Getting Started • Install the Add-on Builder from Splunkbase • Navigate to the Add-on Builder home page • Click New Add-on Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 230 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Create Add-on • Enter a name for the add-on – This field becomes the name of the new app – The builder adds a TA- prefix • Add other optional project items • Click Create – This creates a new add-on app on the local Splunk server • Your add-on home page is displayed • You may see a system message to restart Splunk—you can defer this until done with the new add-on Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 231 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder Home Page 1 Manage Source Types 2 Map to Data Models 3 Validate and Package Note The Add-on Builder can do a lot of things, but for CIM normalization you only need to add sample data and the CIM mapping function. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 232 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Sample Data • Select Manage Source Types – May need to reboot first if add-on is newly created • If your sample data is already in Splunk, use Add > Import from Splunk 1 – Select from a sourcetype list and click Save – You also specify event breaks, time-stamping and other settings • You can add multiple sourcetypes if desired 3 2 Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 233 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Data Model Mapping • Select Map to Data Models • Click New Data Model Mapping • Enter a name for the new event type • Select the source type you are mapping • Click Save • The Data Model Mapping Details view opens Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 234 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Event Types • Before you can add data model mappings, you must identify your sourcetype(s) with an event type – This is used to generate the correct tags for your events to match the CIM target data model’s constraints • On the Data Model Mapping Details page, each sourcetype you added in the sample data must map to one event type – More than one sourcetype can map to the same event type • You can also add search criteria to filter out unwanted events from your data model mapping – This excludes the events from the data model acceleration Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 235 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 CIM Mapping Select FIELDALIAS or EVAL 2 from the New Knowledge Object drop-down 1 Select one or more target data models 3 Select a source field Click OK 6 Click Done 4 5 Select a target field Note The source event type or expression field can be an eval statement (to transform the source value to the CIM required format). Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 236 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Validate and Package • You can validate your add-on for best practices, CIM mapping, and field extractions – Any errors indicate a problem that should be corrected – Warnings are non-fatal but might need attention – If you select App pre-certification, a Splunkbase login is required • Use Download Package to create an SPL package you can deploy to your production environment – The add-on is already active on the local system Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 237 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Add-on Builder: Validate and Package (cont.) Click to download 1 2 Select validations to apply 3 Click to start validation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 238 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 9 Lab: Building a Custom Add-on Time: 45 minutes Tasks: • Plan a new add-on for custom data • Create the add-on with the Splunk Add-on Builder Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 239 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 10: Tuning Correlation Searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 240 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Describe correlation search operation • Customize correlation searches • Describe numeric vs. conceptual thresholds • Discuss the Event Sequencing Engine Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 241 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Plan, Install, Evaluate, Refine • Start with a base level of enabled correlation searches – Security events in the enterprise – Anomalous audit trails • Adjust correlation search sensitivity – False positives: returning results when none are actually there – False negatives: returning no results when something is expected • Revisit and adjust thresholds as needed – New security data is added to your ES install – The size of what is monitored shrinks or grows – Decreased number of open issues (i.e. ES is working!!) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 242 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Managed Content • Correlation searches are one type of ES content – Correlation searches are stored as saved searches – Content in ES is any search or view that can be shared and used between multiple ES sites • Examples: – Correlation and Key Indicator searches – Entity (asset or identity) swim lane searches – Lookups – Views (dashboards and panels) – Saved searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 243 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Content Management Functions Configure > Content > Content Management Enable, disable, or export Filter list by Type Filter list by App Create new content Text filter Enable or disable Click a title to edit Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 244 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Content Management Functions (cont.) Configure > Content > Content Management Expand the Information (i) column to verify dependency and usage information The details for each type of content, and each individual knowledge object vary http://docs.splunk.com/Documentation/ES/latest/Admin/Expandcontentmanagementsearches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 245 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Enabling Correlation Searches • All correlation searches are disabled by default • Enable the correlation searches that make sense for your environment • Consider: – Types of vulnerabilities or threats you have determined might exist – Type of security operations you are focused on, i.e., malware, intrusion detection, audit, change monitoring, etc. – You may need to increase hardware specs if you have many correlation searches running – You can improve overall performance by making less critical correlation searches scheduled instead of real-time Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 246 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Scheduling a Correlation Search • By default, all correlation searches run in indexed real-time mode • If changed to scheduled, it will execute every 5 minutes by default • When editing the scheduled search, you can change the time range settings Start time, End time, and Cron Schedule Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 247 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Tuning Correlation Searches • Threshold: the criteria that causes a correlation search to trigger • Scheduling and throttling: how often to run the search and how often to generate notable events for the same type of incidents • Adaptive Responses: list of actions to take, including possibly creating a notable event or setting risk – Notable event settings: severity, default owner, default status, etc. – Risk: assigning, increasing, or decreasing the risk score for a given type of threat or incident – Other adaptive responses include sending email, running scripts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 248 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Correlation Thresholds • Some correlation searches may generate more (or fewer) notable events than you want • Examine the search string and look for comparison terms in search or where/xswhere functions and modify as appropriate for your environment • Two types of thresholds: – Numeric – Conceptual docs.splunk.com/Documentation/ES/latest/User/ConfigureCorrelationSearches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 249 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Numeric Thresholds • Simple numeric comparisons • Example: Excessive DNS Failures • Note where command with numeric comparison • Change the numeric value if you need to alter how frequently notable events are generated in your environment Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 250 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Conceptual Thresholds • Use Extreme Search functions • Example: Brute Force Access Behavior Detected • Note the xswhere command using “medium” as a threshold – Change as appropriate docs.splunk.com/Documentation/ES/latest/Admin/Extremesearchexample Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 251 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Choosing Conceptual Thresholds • In the previous example, the brute force correlation search generates a notable event if there are more than “medium” failures in an hour • Use the xslistconcepts command to determine other conceptual terms for this threshold • Extreme Search conceptual terms map to dynamic ranges of values that are automatically calculated and updated Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 252 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Determining Conceptual Ranges Use xsdisplaycontext to display a graph of the values used in the concepts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 253 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Correlation Search Throttling • Once a correlation search has been triggered, you probably don’t want it to immediately re-trigger again for the same issue • Most OOTB correlation searches throttle alerts to once a day • If you want to modify this, change the Window duration • In most cases, leave the Fields to group by alone Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 254 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adaptive Response Actions • When a correlation search detects an issue, it can initiate one or more adaptive response actions • The most common response is to create a notable event • Many also add risk to the objects associated with the issue • Other responses can include sending email, running a script, stream capture, and sending data to UBA Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 255 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Customizing Notable Event Default Values • Expand the notable adaptive response • You can modify all the properties of the notable event that is created by a triggered correlation search—typically: – Severity – Default Owner – Default Status Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 256 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Event Sequencing Engine • The Event Sequencing Engine groups correlation searches into batches of events, in a specific sequence, by specific attributes, or both • Event sequencing is configured in Sequence Templates • Sequence Templates: – Define which Start, Transition, and End correlation searches need to occur, and the match conditions – Define if the transitional searches have to occur in a given order, or if they can occur in any order • Templates run as a real-time searches and listen for incoming notable events and risk modifiers that are triggered by the correlation searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 257 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Scenario 1 Create a template to detect high priority hosts with multiple malware infections, excluding test host ACME-004. Then, detect if the host has an abnormally high number of HTTP method events, excluding any “unknown” methods Set the starting correlation search to Endpoint – High Or Critical Priority Host With Malware. Set the expression to detect all destinations (dest) except ACME-004. Give the template a name and description, and select the ES app. 2 https://docs.splunk.com/Documentation/ES/latest/Admin/Sequencecorrelationsearches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 258 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Scenario (cont.) 3 4 Set the ending correlation search to Web – Abnormally High Number of HTTP Method Events By Src, and the expression to detect all methods except unknown. Also, set the time limit for the template to run to 60 days. Add the transitional correlation search Endpoint – Host With Multiple Infections with the expression to detect all destinations (dest) except ACME-004. Note 5 Add a title, urgency, and security domain for the notable events that are created when the template is triggered. In this template, the Enforce Ordering box has been unchecked. Therefore, the transitional searches do not have to happen in order, they just have to exist for the template to trigger. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 259 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Results The results of the Sequence Templates are Sequenced Events, which are viewed in the Incident Review dashboard Transitions display the correlation searches matched in the template. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 260 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 10 Lab: Tuning Correlation Searches Time: 15 minutes Tasks: • Identify numeric thresholds in a correlation search • Identify conceptual thresholds in a correlation search Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 261 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 11: Creating Correlation Searches Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 262 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Create a custom correlation search • Manage adaptive responses • Manage content import/export Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 263 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Creating a New Correlation Search 1. Determine a pattern of events that indicates an issue you want to respond to with a notable event or other action 2. Create a new correlation search in the UI using Configure > Content > Content Management and select Create New Content > Correlation Search – Use Guided Mode if desired 3. Configure scheduling and throttling 4. Configure the alert responses (notable event, etc.) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 264 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Correlation Search Example: Risk • This example creates a new correlation search that generates a notable event once a day for any server with a risk score over 100 • On the Content Management page, select Create New Content > Correlation Search • Enter the search name, App, UI Dispatch Context, and Description • Select Guided Mode to create the actual search Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 265 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Correlation Search Example: Risk (cont.) • From the Guided Search Editor, select the Risk Analysis data model and the All_Risk dataset • Set Summaries only to Yes – The correlation search will only search in accelerated data – This is faster, but un-accelerated data is ignored • Select the time range for the search • Click Next Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 266 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding a Where Filter • Next, add filter expressions to limit the source events the correlation search retrieves – This could be used to focus on high priority assets or specific business units Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 267 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Aggregate and Split-by Functions • Next, add aggregate functions to perform operations like count, sum, or average on fields in the data model • Optionally, add split-by conditions to aggregate values categorically – The example takes the sum of all risk per host • Click Next Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 268 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Filters • Define the logic to determine what condition will trigger a new notable event • In this case, a notable event is generated if the risk score for any one host is greater than 100 • Click Next Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 269 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Parsing the Search • Finally, the search is parsed and displayed • After verifying the test, select Done to save the correlation search criteria and continue configuring the rest of the correlation search fields • If you edit the search string manually later, you will not be able to use guided mode to modify the search string Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 270 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Setting the Time Range • Configure time range options – Earliest Time and Latest Time are relative to the scheduled start time – Cron schedule is how often to run the search ê The default is ‘*/5 * * * *’ which is every five minutes ê This is over ruled if the correlation search is set to real time Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 271 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Setting a Schedule • Scheduling: real-time or continuous – Manages real-time scheduling – Typically, leave the default of real-time • Schedule Window: seconds (or “auto”) – Allow some flexibility in scheduling to improve scheduling efficiency • Scheduling Priority: higher-priority searches will be selected first by scheduler if a conflict occurs docs.splunk.com/Documentation/Splunk/latest/Admin/Savedsearchesconf Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 272 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Setting Trigger Conditions • Normally, a correlation search will trigger its actions (notable, etc.) if any results are found by the search • You can use the Trigger Conditions to alter this default Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 273 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Setting Throttling • Throttling: You should throttle based on a field’s value – Example: no more than one notable event per host per day (86,400 seconds) • More than one field can be selected – Throttling is based on all the field values ANDed together Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 274 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding Response Actions 1 Expand the list of adaptive responses 2 Select the Notable response Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 275 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Notable Event Fields • Configure notable event field values – Title, description, security domain, severity – Default owner and status – Drill-down settings • Embed field values in title, description, and drill-down fields using $fieldname$ format • Description fields support URLs to external locations – Useful for best practices documents, investigation procedures, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 276 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Notable Event Fields (cont.) • Control actions taken when a notable is added to an investigation – Select an investigation profile to apply to the investigation – Automatically extract assets and identities that will be added to the investigation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 277 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Notable Event Fields (cont.) • You can control the “next steps” and “recommended actions” adaptive responses that appear in Incident Review – Next steps appear as links in the notable event details – Recommended Actions appear in the notable event’s Actions menu Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 278 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Saving the Correlation Search • Click Save to create the new correlation search • Click Close and navigate back to the Content Management page • Your new search will now display in the list of Correlation Searches for the ES app • You can enable, disable, and change to scheduled or real-time as desired Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 279 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adaptive Response Actions • Besides (or instead of) creating notable events, adaptive response actions can automate other critical tasks • One or more adaptive response actions can be added to each correlation search – The action will be executed if the correlation search finds any matches • ES ships with a set of default adaptive responses • You can also install additional adaptive responses, and control who can access each adaptive response docs.splunk.com/Documentation/ES/latest/Admin/Setupadaptiveresponse Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 280 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Default Adaptive Response Actions Notable, Risk Create a notable event or add to an object’s risk score Send email, create Splunk message Run script Send email to one or more people, or add a system message in the Splunk web interface Execute an automated script. Example: when a correlation search indicates a host is infected with malware, run a script to quarantine the target server Stream capture Automatically begin collecting detailed network information Nbstat, nslookup, ping Execute diagnostic command and attach output to the notable event to assist in analysis If User Behavior Analytics is installed and integrated, send the notable event to UBA for analysis/send to Splunk telemetry Create a threat intel artifact. Example: a new type of infection is discovered; add the characteristics of the infection (file name, source IP, code hash, etc.) to the threat intel database so that future similar attacks will be immediately alerted Send to UBA/output to telemetry endpoint Add Threat Intelligence Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 281 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Managing Adaptive Responses Select Settings > Alert actions to enable/disable, change permissions, or add new adaptive responses Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 282 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adaptive Response Action Center Audit > Adaptive Response Action Center Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 283 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Content Import/Export • You can export any of the content types on the Content Management page by selecting them in the custom search list and choosing Export • Enter an app name, prefix, label, version and build number, and click Export – The content will be downloaded to your workstation as an .spl file – It can then be installed as a new app into another ES search head Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 284 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Example: Content Export Note DA-ESS is a recommended prefix for content add-ons, but is not required. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 285 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Content Export Best Practices • The app name for the content export is uploaded to the etc/apps directory of the receiving server • Be careful when exporting updates to your content – Example: you export correlation1, naming it correlations.spl, and upload it to another ES server. Later you export correlation2, again using correlations.spl as the export name. When you upload correlations.spl to the second server, it overwrites the old version of correlations.spl, deleting correlation1 • Either use new app names each time (which could be difficult to manage) or make sure you always include all content (old and new) each time you export Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 286 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 11 Lab: New Correlation Searches Time: 20 minutes Tasks: • Create a custom correlation search – SSH logins are prohibited in your environment. Create a custom correlation search that detects successful SSH logins and generates a notable event to alert analysts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 287 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 12: Lookups and Identity Management Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 288 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Identify ES-specific lookups • Describe the interaction of lookups with correlation searches and other ES functions • Configure asset and identity lookups Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 289 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Data Enrichment • ES uses many lookup tables to store extended data for use on dashboards and by searches – Examples: assets, identities, incident review, threat intelligence, and categories • Some lookups are managed by the KV store – Examples: incident review, threat intel collections • Others are stored in CSV files in the lookup directories of several supporting add-ons • Use Configure > Content > Content Management to manage ES lookups, filter Type to Lookup Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 290 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 ES Lookup Management Configure > Content > Content Management Add a new lookup Filter for Lookups Edit lookup settings Add a new lookup table Click a lookup to edit Remove from ES Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 291 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing and Viewing Lookup Files Right-click to add or remove columns or rows Click a cell to edit contents Click Save when done Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 292 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Adding New Lookup Configurations • Select Create New Content > Managed Lookup • Add a new lookup file or select pre-existing • Select a containing app • Set lookup file and definition name, label and description • Select Manually Edited for a simple lookup or Per panel filtering for a panel filter • Click Save Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 293 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing a Lookup Configuration • You can modify the lookup type, label and description • If you disable Editing, the lookup can not be updated Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 294 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Lookup Types: 1 Assets Information about the devices in your environment Identities Threat intel Information about the people in your environment Locally produced and managed collections of threat intelligence Interesting ports, processes, services Domains Descriptions of ports, processes, and services, including prohibited flags Configure the local corporate domain; correlation searches watch for non-corporate email and web access ES Instrumentation * Several lookups used in conjunction with Splunk introspection and usage data telemetry Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 295 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Lookup Types: 2 Categories Category definitions for assets and identities Action history whitelist Risk Object Types Mask searches from the action history for investigation journals Extend risk object type definitions Security Domains Edit or extend the list of security domains—access, network, etc Urgency levels Edit or extend the urgency level titles Expected Views Enable tracking of view use in ES—shows up in View audit Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 296 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Lookups and Correlation Searches Lookup Supported Correlation Search Interesting Processes Prohibited Process Detected Interesting Services Prohibited Service Detected Prohibited Traffic Prohibited Port Activity Detected Local * Intel Threat List Activity Asset Expected Host Not Reporting. Should Timesync Host Not Syncing Identity Watchlisted Event Observed Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 297 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Managed Lookups Audit Audit > Managed Lookups Audit • Reports on managed lookups and collections (i.e. data, services, transforms, KV Store, CSV) • Shows the growth of lookups over time and the markers for anomalous growth • Can be used to determine if any managed lookups are growing too large and need to be pruned Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 298 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Asset and Identity Investigators • Both investigator dashboards allow you to enter an asset or identity name and a search range • Both return a time-sequenced set of swim lanes showing activity for that asset or identity over time, comparing activity between: – Threats – IDS attacks – Authentication activity – Malware attacks – Notable events – Changes (such as firmware or software upgrades, etc.) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 299 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Accessing the Investigators • Both the asset and identity investigators can be accessed on: – The User Intelligence menu, or – From field action menus in the Incident Review dashboard Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 300 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 User Intelligence: Asset Investigator Search by asset name Asset information Swim lanes showing activity across areas Details about the selected events in the swim lane Selecting a bar (set of events) shows details at right Area graph shows activity over time period Choose time span for search Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 301 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 User Intelligence: Identity Investigator Search by identity name Same tools and functionality as the Asset Investigator Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 302 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 How to Interpret Investigators • The swim lanes visually show activity in various areas in time sequence, making it easy to see incidents that are simultaneous or sequential • Activities that coincide in time may have a cause-effect relationship • For example: – A server shows a burst of authentications at 1:15 am – At 1:17 am, a malware attack notable event is triggered for that server – The asset investigator makes it apparent that there is a possible cause-effect relationship spanning across two (or more) swim lanes Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 303 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Pan and Zoom End Start Dragging the pan/zoom controls changes the time frame for the search and re-executes the search, showing only the activity in the selected range Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 304 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Swimlanes • Click Edit and select a collection of swimlanes • Use the Custom collection to select specific swimlanes Drag swimlanes up and down into the order you prefer • Customize swimlane colors • All changes are saved as preferences for the current user • ES Admins can add new swimlanes and can set overall defaults and permissions per role as needed Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 305 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Special Lookups: Assets and Identities • Assets and identities are managed by the Identity Management modular input and Identity – XXX – Lookup Gen searches • All assets and identities are checked for changes automatically every 300 seconds • The searches create multiple expanded versions of the lookup tables for use during searches • Identity management lookups are stored in SA-IdentityManagement/lookups docs.splunk.com/Documentation/ES/latest/User/Identitymanagement Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 306 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Managing Identity and Asset Lists • Configure > Data Enrichment > Identity Management • static_assets and static_identities are the normal lookups – demo_ lookups can be enabled for testing • administrative_identities documents privileged accounts like root Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 307 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Uploading Assets and Identities • Initially, load corporate asset and identity data using Splunk addons such as LDAP search or DB Connect • Periodically re-run to keep assets and identities in ES updated • You don’t need to include every piece of hardware or every person – focus on the ones with the most significance • You don’t need to populate every column • Asset and identity lookups scale up into the 10k-100k range Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 308 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Example: LDAP Search Identity Upload |ldapsearch domain=<domain_name> search="(&(objectclass=user)(!(objectClass=computer)))" |makemv userAccountControl |search userAccountControl="NORMAL_ACCOUNT" |eval suffix="" |eval priority="medium" |eval category="normal" |eval watchlist="false" |eval endDate="" |table sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile, manager,priority,department,category,watchlist,whenCreated,endDate |rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate |outputlookup my_identity_lookup Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 309 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Troubleshooting Assets and Identities • Examine CSV files in $SPLUNK_HOME/etc/apps/SA-IdentityManagement/lookups – Verify that all CSV files are properly formatted – Verify that expanded versions have been created • Check the log files index=_internal sourcetype=python_modular_input category=asset OR category=identity Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 310 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 More Troubleshooting • To test an asset match: | makeresults | eval src="1.2.3.4" | `get_asset(src)` • To test an identity match: | makeresults | eval user="hax0r" | `get_identity4events(user)` Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 311 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Asset Matching • ES takes the value from an event’s src, dvc, or dest field and tries to match it to these columns in the asset lookup: Order 1 Column ip Description match the IP address or address range 2 mac match on a Media Access Control address 3 dns match on DNS name 4 nt_host match on Windows Machine Name (a.k.a. NetBIOS name) • ES uses the above order to make its first match, then checks CIDR-based matches for IP addresses Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 312 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 CIDR Asset Matching • For ip and mac field ranges, if more than one range matches, ES matches on the smallest range For example, host=1.2.3.4 matches both the first and second IP ranges; however, it only matches on the second one since that’s the smaller range • Asset matching allows you to create large, catch-all categories on MAC or IP ranges, yet still single out smaller groups or individual IPs within the larger group Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 313 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Other Interesting Asset Columns Field Value Description bunit Text field Arbitrary “business unit” for the asset; useful for filtering certain views. category Text field User-defined category for asset. The list of options for this field is retrieved from a separate lookup list. is_expected True/False If true, ES expects this asset to always be running and sending data to Splunk. If it stops, a notable event is created. Defaults to False. should_timesync, True/False should_update Works the same as is_expected, except it alerts on failure to time sync or failure to update. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 314 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Identity Matching • ES takes a value from an event’s user, src_user, email, src_email field and tries to match it in the identities lookup: Order Column Description identity 1 Exact match on any one of a list of pipe-separated list of user names in identity column Email 2 Exact match Email 3 First part of email, ie “htrapper” of “htrapper@acmetech.com” Any 4 Disabled by default—see “conventions” in identityLookup.conf.spec • There is also a configuration UI to specify which of these to use Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 315 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Identity Lookup Configuration Configure > Data Enrichment > Identity Lookup Configuration docs.splunk.com/Documentation/ES/latest/User/Identitymanagement Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 316 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Other Interesting Identity Fields Field Identity (key) Description Pipe delimited list of usernames representing the identity Prefix Nick First Middle Last Suffix Email Phone watchlist Prefix of the identity (for example, Dr.) Nickname of the identity First name of the identity Middle name or initial of the identity Last name of the identity Suffix of the identity (for example, Jr.) Email address of the identity Phone number of the identity True/false Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 317 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Watchlisting Assets and Identities • You can add identities and assets to a watchlist, which then highlights them in various dashboards and searches – Example: watchlisted users are shown on the User Activity dashboard • Watchlist users by setting the watchlist to true in static_identities.csv • Add assets to watchlists by: – Configure > General > General Settings – Edit Website Watchlist Search and add asset IP or DNS • Watchlisted assets or identities also trigger the Watchlisted Event Observed correlation search, if enabled Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 318 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Asset Center Security Domains > Identity > Asset Center • Overview of assets • Visualizations by priority, business unit, and category • Table at bottom shows all asset lookup columns Distribution of assets by priority Distribution of assets by business unit Distribution of assets by category All assets with dns, nt_host, ip, mac address, owner, priority, location, category, and PCI domain Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 319 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Identity Center Security Domains Identity > Identity Center • Overview of identities • Bottom table shows all identity lookup columns Identities by priority Identities by business unit Identities by category Identity information is shown with name, contact info, priority, business unit, watchlist (boolean: true or false), and start and end dates. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 320 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Asset/Identity Correlation Performance Configure > Data Enrichment > Identity Correlation • Control asset/identity correlation by sourcetype • This can be useful for performance improvement • If a given sourcetype does not contain “interesting” assets or identities, there is no need for asset/identity correlation Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 321 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 12 Lab: Adjusting Asset Priority Time: 25 minutes Tasks: • Modify asset priority for PROD-MFS servers Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 322 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 13: Threat Intelligence Framework Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 323 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Objectives • Describe threat lists and threat list administration tools • Configure a new threat list Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 324 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 The Threat Intelligence Framework • The Threat Activity Detected correlation search creates a notable event if it detects an indicator of compromise (IOC) contained in a threat intelligence collection • The threat intel collections are populated automatically by downloads from external threat libraries • Threats are categorized by: – Group: the source or entity originating the threat – Category: the type of threat, like backdoor, APT, financial, etc. – Collection: organized by threat method or routing, such as email, file, process, user, etc. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 325 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Included Generic Intel Sources Data List Data Provider Data Provider Website Cisco Umbrella 1 Million Sites Cisco https://umbrella.cisco.com/blog/2016/12/ 14/cisco-umbrella-1-million/ Alexa Top 1 Million Sites (deprecated) Alexa Internet http://www.alexa.com/topsites ICANN Top-level Domains List IANA http://www.iana.org/domains/root/db MaxMind GeoIP ASN IPv4 database MaxMind https://dev.maxmind.com/geoip/geoip2/g eoip2-anonymous-ip-csv-database/ MaxMind GeoIP ASN IPv6 database MaxMind https://dev.maxmind.com/geoip/geoip2/g eoip2-anonymous-ip-csv-database/ Mozilla Public Suffix List Mozilla https://publicsuffix.org Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 326 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Threat Activity Dashboard Security Intelligence > Threat Intelligence > Threat Activity Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 327 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Using Threat Activity • Displays events related to known threat sites over the desired time period • Panels – Threat activity over time by threat collection – Most active threat collections and sources – Threat activity detail • Filters – Threat group: a known threat source—i.e., “who” – Threat category: threat type, such as APT, backdoor, etc. – Threat Match Value: Choose a filter from a list of fields Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 328 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Threat Activity Details • Use threat details to examine the most recent threat events, including source, destination, sourcetype (i.e., how was it detected), threat collection, group, and category • You can also filter or highlight as per the other Advanced Threat dashboards – Select one or more rows, then click Advanced Filter Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 329 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Threat Artifacts Security Intelligence > Threat Intelligence > Threat Artifacts Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 330 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Using Threat Artifacts • Threat Artifacts displays the current content of the threat intelligence data that ES has downloaded • You can use the filters at the top to select a threat artifact type, and then filter by fields relevant to the selected artifact type • The threat overview panel displays the items that have been downloaded from threat lists or STIX/TAXII sources • The sub-panels display statistics on the threat intelligence data by endpoint, email, network and certificate • The tabs allow you to drill down into these categories and gain additional details for each type of threat Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 331 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Configuring Threat Intelligence • ES can download the following threat intelligence types: – Threat lists: IP addresses of known malicious sites – STIX/TAXII: detailed information about known threats, including threat type, source, etc. – OpenIOC: Additional detailed information about known threats • You can also configure local threat lists • Many intel sources require regular refresh from external sources • This information is used by the Threat Activity Detected correlation search Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 332 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 The Threat Intelligence Framework • Threat intel is downloaded regularly from external and internal sources by the Threat Download Manager modular input – This data is parsed into KV store collections with “_intel” suffixes – These are used as lookups during threat generation searches • Threat gen searches run periodically (by default every 5 minutes) and scan for threat activity related to any of the threat collections – When threat matches are found, events are generated in the threat_activity index and appear in the Threat Intelligence data model • This data model is scanned by the Threat Activity Detected correlation search and new notables for threat activity are created Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 333 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Types of Intelligence Downloads • Threat intelligence – Contains information needed to support the Threat Activity correlation search – Will be parsed into threat collections by the threat intelligence framework • Generic intelligence – Non-threat descriptive information, such as lookups for top-level domains, used to add additional details in ES views and reports docs.splunk.com/Documentation/ES/latest/Admin/Includedthreatintelsources Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 334 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Intelligence Download Management Configure > Data Enrichment > Intelligence Downloads Note Intelligence downloads cannot have spaces in the names. Use underscores or dashes instead. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 335 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Editing Threat Downloads Note The is Threat Intelligence checkbox determines if this is threat or generic intel. Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 336 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 About Threat Downloads Field Description Is Threat Intel If true, processed by threat intel framework Sinkhole If true, deletes downloaded file after processing Type For TAXII feed must be “taxii”, otherwise not used URL, Interval, User agent Path to the download, frequency and user agent settings Weight An integer used to calculate risk for this threat type Max age How long to retain threat intel (default -30d) POST arguments Passed to source server if needed; i.e.; credentials, etc. Parsing options Extraction rules for formatted feed Download options Interval, timeout, etc. Proxy options Configure proxy firewall settings Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 337 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Manual Intelligence File Upload • To manually upload one threat intelligence file: Navigate to Configure > Data Enrichment > Threat Intelligence Uploads 2. Add a file name for an OpenIOC, STIX or CSV file 3. Configure options for weight, category, etc. 4. Click Save 1. docs.splunk.com/Documentation/ES/latest/Admin/ Uploadthreatfile Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 338 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 OpenIOC Batch File Upload • OpenIOC threat collections are not handled via the threat intelligence downloader • To add OpenIOC files to your ES threat intelligence framework: Copy the OpenIOC files to etc/apps/ DA-ESS-ThreatIntelligence/local/data/threat_intel 2. Check var/log/splunk/threat_intelligence_manager.log for the progress of the download 3. Examine the contents of the target threat intel KV store collections for threat artifacts from the source OpenIOCs 1. • This can be done on an automated, scheduled basis Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 339 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Threat Intel Collections • After download, threat intel data is stored in KV store collections with an “_intel” suffix, such as file_intel, ip_intel, email_intel, etc. – Use Settings > Lookups to see them all – Use |inputlookup to examine their contents • Use the Threat Artifacts dashboard to examine the overall contents of the entire threat intelligence framework Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 340 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Threat Intelligence Audit Audit > Threat Intelligence Audit • displays status and time for all downloads as well as recent audit events • Filter by enabled/disabled state, local/remote, source, etc. • Failed downloads are reported in the system message list at the top of the SplunkWeb page Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 341 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Module 13 Lab: Threat Intel Framework Time: 20 minutes Tasks: • Add a new threat list download Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 342 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 What’s Next? OPTION 1 Splunk Enterprise System Administration Become a Splunk Enterprise Security Certified Admin! Splunk Enterprise Data Administration OPTION 2 Splunk Enterprise Security Certified Admin Exam Administering Splunk Enterprise Security Splunk Cloud Administration Splunk Education Content. See splunk.com/education for access, registration dates and cost (if applicable). Questions can be addressed to education_amer@splunk.com Splunk Certification Exams. See splunk.com/education for access, registration dates and cost (if applicable). Questions can be addressed to certification@splunk.com Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 343 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 Support Programs • Community – The Splunk Answers Site: answers.splunk.com Post specific questions and get them answered by Splunk community experts – Splunk Docs: docs.splunk.com These are constantly updated. Be sure to select the version of Splunk you are using – Wiki: wiki.splunk.com A community space where you can share what you know with other Splunk users – IRC Channel: #splunk on the EFNet IRC server Many well-informed Splunk users “hang out” here • Global Support Support for critical issues, a dedicated resource to manage your account – 24 x 7 x 365 – Web: http://www.splunk.com/index.php/submit_issue • Enterprise Support Access you customer support team by phone and manage your cases online 24 x 7 (depending on support contract) Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution 344 Administering Splunk Enterprise Security Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019 .conf19 Splunk University October 19-21, 2019 The Venetian Sands Expo 4 Days of Innovation 350 Education Sessions 20 Hours of Networking October 21-24, 2019 Las Vegas, NV “Hands down the most beneficial and attendee focused conference I have attended!” – Michael Mills, Senior Consultant, Booz Allen Hamilton Generated for Katie Brown (katieb@splunk.com) (C) Splunk Inc, not for distribution Administering Splunk Enterprise Security sign up for notifications @ conf.splunk.com 345 Copyright © 2019 Splunk, Inc. All rights reserved | 1 July 2019

Administering Splunk Enterprise Security 5.3

Related documents

Products

Support

Administering Splunk Enterprise Security 5.3

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib