Interested in learning
more about security?
SANS Institute
InfoSec Reading Room
This paper is from the SANS Institute Reading Room site. Reposting is not permitted without express written permission.
The Race to Detection: A Look at Rapidly Changing
IR Practices
With the rapidly changing risk environment, those assigned to protect their organizations must be agile in
adapting technology to meet the challenges presented to them. Read this paper to learn what leading incident
response practices are doing, and what they plan for the future.
Copyright SANS Institute
Author Retains Full Rights
The Race to Detection:
A Look at Rapidly Changing IR Practices
A SANS Survey
Written by Alissa Torres
August 2015
Sponsored by
Bit9 + Carbon Black
©2015 SANS™ Institute
Executive Summary
The incident responder works under the constant pressure of an impossible mission:
to detect and remediate critical incidents before an organization suffers significant
loss. Even as attacks become more frequent and sophisticated, incident response (IR)
Key Takeaways
Open APIs, the ability to integrate with other vendors, the ability
to host remediation, and being lightweight are paramount
technology features influencing leading IR firms’ and practitioners’
future technology buy decisions.
practitioners find themselves becoming less reactive and
increasingly more proactive.
And, given the real-time environment and increasing visibility
of breaches, IR is evolving rapidly. Many organizations do not
have the resources to maintain an effective IR team and turn
T he biggest gaps in current IR technologies are a lack of
compatibility with other products, ease of implementation,
memory analysis, lack of baseline creation functionality and lack
of automation, all keys to reducing false positives and streamlining
the IR process.
to professional practices that specialize in it.
T he greatest challenges faced by IR firms in a typical IR
engagement are lack of knowledge of the client network
environment and customer endpoint inventories/asset
management practices.
IR processes, technologies and service trends, as well as gain
T he most common triggers for IR service requests are conditiontriggered alerts from security information and event management
technology, followed by third-party notification and anomalous
network traffic, with antivirus scans reported to be one of the least
effective detection triggers of advanced attacks.
current challenges experienced in the information security
To fully understand the dynamics of the rapid evolution of
incident response, SANS interviewed leading IR consultants
and surveyed a targeted group of practitioners to pinpoint
some insight into the future of IR practices.
Interviews and online survey input highlight many of the
field at large: imperfect utilization of technology, significant
shortages in skilled staff and budgetary challenges. When
an organization’s IR capabilities are degraded, it experiences
an increased risk for undetected intrusions, greater attacker
dwell times and increased exposure for actual breaches. With the rapidly changing risk
environment, those assigned to protect their organizations must be agile in adapting
technology to meet the challenges presented to them. These and other issues are
discussed in the following pages.
SANS ANALYST PROGRAM
1
The Race to Detection: A Look at Rapidly Changing IR Practices
Methodology and Participants
SANS identified leaders from in-house and external IR services and performed individual
interviews with a subset of that sample using a guided survey instrument. The results
of these interviews were then used to shape an online survey that an additional set
of individuals in the IR consulting field was invited to take during May and June 2015.
Participants reflected both senior leadership and a cross-section of IR roles from leading
IR consulting practices: 32% IR team members, 29% team leaders, 28% consultants and
6% practice leads (see Figure 1).
What is your primary role in incident response (IR) within your firm?
Team member
Team leader
Consultant
Practice leader
Other
Figure 1. Respondent Roles
Most (53%) had five years of experience or more, with 19% having worked in IR for five
years. Table 1 shows IR roles by average years of experience.
Table 1.
IR Roles and Average Years of Experience
Role
Average Years of
Experience
Team Member
3.4
Consultant
4.3
Team Lead
6.1
Practice Lead
7.0
Other
4.9
Overall
4.7
Not surprisingly, 91% of the overall group credit practical experience with preparing
them for their duties, while 51% have received vendor training and 38% have earned a
certification in IR. Regardless of the role respondents filled, the same order of training
methodologies is evident.
SANS ANALYST PROGRAM
2
The Race to Detection: A Look at Rapidly Changing IR Practices
Methodology and Participants
(CONTINUED)
Figure 2 illustrates the training received by members fulfilling each role.
How have you received your training in IR?
Team member
Consultant
Team leader
Practice leader
0%
10%
20%
30%
40%
Other
Vendor-provided training
Certification program in forensics
Certification programs in IR, such as SANS
Practical experience
Figure 2. IR Training by Role
This sample of experienced responders provides a unique view of the state of IR today
and where those in the trenches think the field is going.
SANS ANALYST PROGRAM
3
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Today and in the Future
In the first part of 2015, the number of attacks against U.S. retailers dropped by 50%; yet
the number of records stolen remains at near record highs.1 While the number of attacks
has dropped, the breaches are both wider (involving multiple victims) and deeper
(involving more data).
At the same time, the emphasis for culpability and ownership is affecting the
executive chain of command. Designated as the data owners, high-ranking officials
can no longer deny responsibility for a security compromise. According to one senior
interviewee, executives are beginning to see security and IR as one component of
Breaches are wider
now and affect more
businesses. That
impacts management
and the chain of
command more than
ever before. Executives
are now seeing
protecting their jobs.
The threat landscape is changing rapidly. Over the past 18 months, most respondents
cited the increasing sophistication and efficiency of cyber attackers, an increasing
prevalence of sophisticated malware and increased targeted attacks specific to their
customers’ environments as the external factors affecting the future of needed IR
capabilities.
Over the next 18 months, most feel these trends will continue. Many recent attacks by
crime organizations mimic techniques utilized by nation-state actors and described
in detailed, publicly available reports. As PCWorld reported, cybercriminals targeting
security, in general,
point-of-sale devices are increasingly adopting techniques previously seen only in
and incident response,
advanced persistent threat-style (APT) attacks.2 One interviewee provided a clear
in particular, as keys to
protecting their jobs.
—Ricardo Bruno,
warning based on trends he is seeing toward more destructive attacks as attackers
“who don’t care about the environment” penetrate the defenses.
There is some good news for the future. Based on interviews with IR leaders, the industry
is already responding in the following ways:
CTO and Cofounder,
• More sharing of threat intelligence, cooperation and openness among would-be
Activesec, Inc.
competitors in the IR field
• Evolving tools and techniques to support the need for quicker analysis as opposed
to deep-dive forensics (memory dumps, hard drive imaging)
• Shorter discovery times due to maturation of techniques such as traffic
containment and segmentation
• Increased customer awareness of the need to call in the professional A-team for IR
reaction and remediation
• Decreased attacker time in the network, which requires swift response to incidents
SANS ANALYST PROGRAM
1
w
ww.net-security.org/secworld.php?id=17784
2
w
ww.pcworld.com/article/2918732/cybercriminals-borrow-from-apt-playbook-in-attack-against-pos-vendors.html
4
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Today and in the Future
(CONTINUED)
Looking ahead, SANS believes attacks will increase in frequency and sophistication
with more nation-state involvement. Various regulations already require third-party
assessments. Increased media coverage and public outrage will undoubtedly foster
more regulatory requirements around IR, with which organizations must comply.
The security issues highlighted in the recent past will fuel several market trends
highlighted in our preliminary interviews:
• Tool Maturation and Consolidation. More mature products, many offered
by emerging companies, are entering the endpoint-threat-detection-response
(ETDR) space as defined by Gartner.3
• Community of Sharing. There appears to be a move toward a more open
community for sharing threat intelligence across organizations/service providers,
a marked shift from the past culture that barred communication due to fear of
TAKEAWAY:
competition.
One area in which
• Continued Growth in IR Services. One interviewee described possible growth as
organizations can decrease
“exponential.” IR is dynamic. This emphasis on growth is backed up by 54% of our
IR costs is by developing
respondents who felt that demand for third-party IR services would increase and
educated in-house “tier 1”
another 29% who believe growth will continue but that IR services will be brought
capabilities: responders who
in-house.
can mitigate lower-level
To be effective, techniques and tools have to be cutting edge. Experts in the field
incidents, avoiding costly
need to have relevant, current experience. Most smaller/medium businesses
third-party service rates and
(and even larger ones) will have difficulty bringing IR capabilities in-house due
to a lack of qualified IR experts and will definitely have trouble sustaining the
delays in response.
specialized capability, both because of salary (~$200K/employee/year) and cost of
maintaining current expertise. Expertise offered by a specialized third party may
be more cost effective.
• Proactive Customer Involvement. As breaches become more commonplace and
the availability of IR service providers grows, so does an organization’s demand
for IR support, especially with proactive services. One of our survey respondents
provided a unique perspective on this as he predicted an “increase in outsourcing
initially; however, this is likely to collapse as organizations realize the outsource
model isn’t the best way to deal with incidents because that approach needs
corporate ownership to really drive remediation.” While this may be the case for
large organizations, which can stand up internal security operations centers, small
to midsize companies rarely achieve such in-house capabilities despite prohibitive
third-party services costs.
3
SANS ANALYST PROGRAM
w
ww.gartner.com/doc/2596321/endpoint-threat-detection-response-tools
5
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Today and in the Future
(CONTINUED)
In addition, many of our respondents reported breach preparation and prevention
as among the most commonly requested services from their customers. These
proactive service offerings allow organizations to prepare for critical incidents
by first identifying vulnerabilities in their enterprise, devising road maps to risk
mitigation, and implementing efficient and automated response technologies and
processes to reduce the time required for investigation and remediation. Other
Over the next 18
such services in increased demand include mock IR drills, in which a company
months, we believe
tests its IR team, processes, procedures and technologies in anticipation of a
critical event to identify gaps or weaknesses. Significant return on investment may
we will see more
accrue to companies that take these steps, should they ultimately become victims
customers requesting
of a data breach. Companies should include the availability of proactive service
mock incident
response, acting out an
actual breach.
—Matt Dowling,
STIGroup, Ltd.
offerings as selection criteria for external IR service providers.
One challenge, however, remains. Increased visibility and focus on IR with increased
executive recognition does not necessarily translate into immediate action. According
to the Ponemon Institute’s 2015 Global Study on IT Security Spending and Investments,
46% of organizations are increasing their IT security budget4 as a result of this
increased focus. However, the increased focus doesn’t necessarily result in immediate
improvement in IR processes. Maturation of IR capabilities is a slow process, and
increased effectiveness will not be realized overnight. Even as a deliberate effort
is made toward standing up internal capabilities, organizations will still encounter
incidents that reach beyond their team’s skill set, requiring third-party services to fill in
the technical gaps.
4
SANS ANALYST PROGRAM
w
ww.ponemon.org/blog/around-the-world-it-security-practitioners-face-a-common-problem-a-budget-that-is-inadequate-to-deal-with-cyber-threats
6
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Services
IR practices provide a wide range of services to their clients. The majority of respondents
(77%) offer breach discovery and detection services, as well as post-breach services,
including forensic analysis (75%) and breach response, containment, and remediation
(71%). See Figure 3.
What services does your IR practice provide to its clients?
Select all that apply.
80%
70%
60%
50%
40%
30%
20%
10%
Other
Red team
Reverse engineering
Pre-breach testing
Breach prevention
Pre-breach training and
preparation
Penetration testing
SOC assistance
Security architecture assistance
Breach response/
Containment/Remediation
Forensic analysis
Breach discovery or detection
0%
Figure 3. IR Services Offered
To provide these services, the number of IR professionals dedicated to work an
engagement varied widely across respondents. Most engagements start with a small
team (one to five people) for triage/assessment. Team members have specific skills
(team lead, forensics experts for network and/or endpoint, security analytics) to guide
their roles in IR. Once the nature of the incident and work required is assessed, team
leaders can tailor the response to the customer, incident type and scope—assigning
the complete team as needed. Customer resources may also be used, ranging anywhere
from a handful of responders to hundreds of staff members, depending on the
magnitude of the incident.
SANS ANALYST PROGRAM
7
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Services
(CONTINUED)
Ten years ago, IR was mostly driven by alert notification. Today, clients of IR consultancies
are becoming more proactive, applying IR response to their risk assessments. IR practices
offer more pre-breach services such as security architecture assistance consulting (67%),
security operations center (SOC) assistance (62%), penetration testing (59%), prebreach training and preparation (53%), and pre-breach testing (43%). Proactive services
mentioned by interviewees included:
• P
roviding organizational support, open source intelligence monitoring and breach
analysis for anomalous activities in conjunction with security architecture assessment
• D
eveloping a pre-breach preparation package for a client, setting a baseline
to provide visibility into what may happen, and identifying and implementing
corrective controls to improve security posture
From my experiences,
• C
onducting mock breach exercises using actual malware to provide a realistic test
of the organization’s detection, containment and remediation capabilities
clients have wanted
to leverage in-house
resources and use
outside firms for
knowledge transfer
and coaching over the
last 18 months. Clients
are very sensitive to
costs and, as a result,
forensics demand has
dropped dramatically.
—Anonymous
When asked which services listed in Figure 3 their clients most frequently requested,
46% reported proactive services such as SOC assistance, penetration testing, assistance
with the security architecture, breach prevention, and training and preparation as their
top requests. Although IR practices offer more proactive options, 45% indicated their
clients still rely heavily on them for reactive services, especially breach discovery and
response. The remaining respondents provided no response.
For organizations that have conducted pre-breach preparations, augmented IR support
provided by external services may be more efficient and more easily integrated. When
organizations have deployed enterprise response and monitoring tools and archived
historic endpoint and network data, IR scoping and investigation can begin immediately,
decreasing the overall cost of the breach. It is notable, however, that more than one of
the professionals we interviewed expressed concerns about using the existing response
technology present in a compromised customer environment. They prefer to put in
place their own endpoint and network monitoring tools, assuming that those currently
deployed are also compromised.
According to our respondents, demand for digital forensics services as part of an
intrusion investigation is decreasing. Organizations are beginning to favor triage and
live-system analysis over deep-dive forensics of compromised systems for both the cost
and time savings realized. The 2014 Ponemon Cost of Data Breach report estimates the
cost of an average intrusion at about $3.5 million. Reducing detailed forensic analysis of
compromised systems can help reduce that expense.5 Another reason for the decreasing
demand may be the increased visibility into endpoints and network traffic provided by
today’s enterprise remote triage tools. These capabilities were not available in tools until
recently, and without them, deep-dive forensic analysis was usually required to retrace
attacker activity or isolate malicious code.
5
SANS ANALYST PROGRAM
www.ponemon.org/blog/ponemon-institute-releases-2014-cost-of-data-breach-global-analysis
8
The Race to Detection: A Look at Rapidly Changing IR Practices
IR Engagement Challenge: The Client
The greatest challenge faced by most practices in a typical engagement is the client.
Respondents faced unanticipated challenges in transitioning a compromised organization
to containment and remediation due to poor IT practices on the part of the client.
Determining what normal looks like is not easy, even if you are working in the same
TAKEAWAY:
The greatest challenges
faced by IR firms in a
typical IR engagement
are lack of knowledge
environment every day. But for a consultant, baselining a foreign environment presents
a significant hurdle. A leading concern, cited by 32% of respondents, is the absence
of a system inventory. This can lead customers to wildly underestimate the number of
servers, endpoints, subnets and network devices in their environment.
Figure 4 shows those client practices that impede a fast ramp-up for an IR engagement.
What is the greatest challenge you face in a typical IR engagement?
of the client network
environment and customer
endpoint inventories/asset
C ustomer system inventory/
Asset management
management practices.
K nowledge of client network
environment
Availability of adequate client tools
Documentation of client network
TAKEAWAY:
Other
Work with clients to record
a baseline of network and
Figure 4. Greatest Challenge in IR Engagements
endpoint activity prior to an
incident. Simple baselining of
operations performed in an
analyst’s downtime can save
thousands of dollars during a
typical incident response.
Best practices call for creating an inventory of authorized and unauthorized devices.
Cited in the first Critical Security Control,6 implementation of device inventories will help
organizations improve their risk posture and make it easier for incident responders to do
their jobs.
Closely related, and also cited by 32%, is lack of knowledge of the client’s network
environment. When a customer does not have a proper baseline for net flow, network
traffic, network device configuration, endpoint builds or user activity, the responder’s
familiarization period draws out and delays progress. Add in complex organizational
structures, such as newly acquired companies with diverse polices and security
implementations across suborganizations, and the challenge increases exponentially.
6
SANS ANALYST PROGRAM
www.counciloncybersecurity.org/critical-controls
9
The Race to Detection: A Look at Rapidly Changing IR Practices
IR Engagement Challenge: The Client
(CONTINUED)
TAKEAWAY:
Availability of adequate customer tools, cited by 19%, represents a significant challenge
Organizations can expedite
to third-party incident responders. Without tools in place to both detect and remediate,
investigations and decrease
the time and money spent
on breach mitigation/
responders face an uphill battle to regain control of systems. Lack of documentation
of the client network environment and its complex organizational structures, such as
a newly acquired company with diverse polices and security implementations across
suborganizations, is cited as another significant barrier to delivery of effective IR
remediation through the
services by 12%. Customers rarely have up-to-date network configuration diagrams or
use of in-house or third-
regimented change-management programs in place.
party services (on retainer
Two interviewees noted that they are on retainer for some of their clients. Such a
or pay-as-you go contracts)
relationship can be valuable. Consultants set prepaid hours to prepare a client to detect
by investing in preemptive
an incident and to provide support should the client encounter a critical incident. An IR
network documentation and
asset inventory.
SANS ANALYST PROGRAM
professional can provide maximum value by preparing the customer environment for
the inevitable, paving the way with baseline information for those on the customer site
working an IR event.
10
The Race to Detection: A Look at Rapidly Changing IR Practices
Measuring Effectiveness
Actionable metrics are key to evaluating success in security processes, including IR.
Formal, proprietary metrics that are considered to be part of the practice’s intellectual
Proprietary metrics are
part of our intellectual
property that we
deliver to clients as
property are used by 14% of respondents.
Another 8% have published formal metrics. The majority (34%), however, mention
establishing metrics based on the customer’s needs—for example, a baseline that
shows the number of infections resulting from spearphishing or unauthorized access
detection—to establish value for that engagement. Sadly, 33% have no established
a service. We use
metrics, speaking to the immaturity of the IR process in these organizations. See Figure 5.
these to determine
Has your practice established any metrics to gauge the effectiveness
of its IR processes?
the effectiveness
of implemented
Y es, we have formal metrics as part
of our intellectual property.
processes and
Y es, we have published formal
metrics.
solutions over the long
term through follow-
N o, we have not established formal
metrics.
up and health checks.
N o, we use metrics, but they vary
according to the engagement.
—Marc Bleicher,
Fidelis Security
Unknown/Unsure
Figure 5. Establishment of IR Metrics
SANS ANALYST PROGRAM
11
The Race to Detection: A Look at Rapidly Changing IR Practices
Measuring Effectiveness
(CONTINUED)
Industry standards do play a role in measuring, monitoring and benchmarking IR
processes. The standards most commonly used by our respondents were PCI DSS (66%),
ISO 27001 (59%), Critical Security Controls (45%) and HIPAA (36%), as illustrated in
Figure 6.
Does your practice measure, monitor and benchmark against any industry standards?
Select all that apply.
70%
60%
50%
40%
30%
20%
10%
Statement on Auditing
Standards, No. 70
Consensus Audit Guidelines
(CAG)
FedRAMP security controls
Other
ISACA COBIT Framework
HIPAA
Critical Security Controls
ISO 27001
PCI DSS
0%
Figure 6. Industry Standards Used in IR
It is not surprising that PCI was the most commonly mentioned standard because it is a
very widely adopted standard, and virtually every business has data that falls under its
security requirements. The Critical Security Controls (CSCs),7 while not as widely adopted
as ISO 27001 standards, offer a prioritized approach to improving network security. This
prioritization is particularly important for organizations that have a limited budget and
simply can’t afford to “do it all” right now.
7
SANS ANALYST PROGRAM
www.counciloncybersecurity.org/critical-controls
12
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
Respondents make use of a variety of technologies in their own IR practices. The five
most commonly used tools/technologies include forensic analysis (96%), endpoint
threat detection (ETD) with scanning capability (88%), antivirus endpoint management
(82%), ETD as an always-on sensor (77%) and log management appliances (71%). The
majority of respondents also use security information and event management (SIEM;
71%), cloud-based ETD (70%), network packet capture (63%) and host-based monitoring
(62%). Use of these tools, illustrated in Figure 7, encompasses the areas identified as best
practices in security monitoring and maintaining visibility into network and host activity.
What tools and technologies does your practice use for IR?
100%
90%
80%
70%
Endpoint Threat
Detection (ETD) Tools:
60%
A category of tools used
40%
to detect and investigate
30%
Other
Risk management platform
Network security
management systems
Configuration management
Netflow monitoring
File integrity monitoring software
Asset inventory/
Asset management
Host-based monitoring
Network capture
ETD—Cloud
SIEM
Log management appliances
ETD—Always-on sensor
0%
Antivirus endpoint management
endpoints
10%
ETD—Scan
of such activity on hosts and
20%
Forensic
suspicious activity or traces
50%
Figure 7. Implemented IR Tools and Technologies
As more organizations incorporate ETD with always-on sensor capability, as
implemented in 77% of our respondents’ organizations, we would expect earlier
detection and greater insight into real-time system changes. In the near future, it is likely
that ETD will play a vital role as a trigger for present-day IR incidents and mature as an
effective investigative technology made possible by the associated real-time collection
of system state that is immediately available for analysis.
SANS ANALYST PROGRAM
13
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
These tools also correlate well with the most common triggers for IR incidents: 49%
of respondents ranked condition-triggered alerts from a SIEM as a primary trigger for
incidents to which their practice responds. Properly implemented log management
appliances and well-tuned SIEM functions result in reduced false positives and alerts that
are actionable. Other common triggers include third-party notifications, chosen by 46%,
and anomalous network traffic, selected by 39% (see Figure 8).
What triggers most of the IR incidents to which your practice responds?
Select the top three.
The most common triggers
50%
from SIEM technology,
40%
followed by third-party
30%
Figure 8. IR Triggers
Figure 8. IR Triggers
SANS ANALYST PROGRAM
Targeted attacks
services respond.
0%
Alerts from antivirus scans
detection triggers to which IR
10%
Anomalies found with
proactive discovery
are one of the less effective
User reporting
network traffic. Antivirus scans
20%
Condition-triggered alerts
from SIEM
notification and anomalous
Alerts from host-based IDS
(HIDS)
condition-triggered alerts
Anomalous network traffic
for IR service requests are
Third-party notifications
TAKEAWAY:
14
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
Procurement Decisions
In today’s environment, IR practices have essentially three approaches to acquiring
tools: developing their own, selecting tools from product vendors, or using open
source tools. With a heavy reliance on forensics and ETD (as illustrated previously in
Figure 7), most seem to continue to utilize commercial off-the-shelf tools (COTS), as
illustrated in Figure 9.
Does your practice develop its own tools (custom built), utilize open source,
or procure competitively (COTS) for forensic and endpoint threat detection (ETD)?
40%
30%
20%
10%
0%
ETD—Always-on
sensor
ETD—Cloud
COTS
ETD—Scan
Open Source
Forensic
Other
Custom Built
Figure 9. Approaches to Tool Selection
SANS ANALYST PROGRAM
15
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
So what factors do organizations consider when deciding whether to buy existing
tools or develop their own? Ease of overall use was deemed most important by 61%
of respondents, followed by life-cycle costs related to purchase, implementation and
maintenance (59%). Because no one tool fits all the needs of the IR team—there is no
single multipurpose IR tool—50% of practices also pay close attention to elements
related to interoperability across complementary products. Figure 10 shows the top
factors that respondents use to select products and complete a “make/buy” decision.
What factors do you use to determine whether your practice develops its own tools or
uses available products for IR? Select all that apply.
60%
50%
40%
30%
20%
10%
Other
Line of business in developing tools
for sale outside of the practice
Licensing for customer use or resale
Owned or requested by client
Return on investment for client
Ease of deployment into
malicious environment
Fill a gap
Interoperability with other products
used for IR
Life-cycle costs (Purchasing/
Implementation/Maintenance)
Ease of overall use, including
deployment
0%
Figure 10. Buy or Build?
SANS ANALYST PROGRAM
16
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
Technical Needs
But what are they seeking in their tools? The most important things to consider are the
technical features. Scanning, blocking of execution, and SIEM integration, selected by
65%, 53% and 53%, respectively, represent the features respondents indicated affected
their current product selections and make/buy decisions.
However, fewer than 10% of respondents identified scanning as an additional need
moving forward. This dramatic decrease could reflect the morphing of the term scanning
into a more universal term for remote system survey of any sort. Obviously, signaturebased scans have their limitations, but system audit collection and continuous endpoint
monitoring also fit the evolving definition of today’s IR tools’ “scanning” capabilities and
eliminate many of the gaps in which the sweeping tools lack visibility.
The next two criteria that carried the greatest weight in the selection of current tools
were blocking of execution and SIEM integration. Users are going to click links, visit
compromised websites and/or launch malicious attachments. Having a security tool
that prevents the execution of malicious code, based either on whitelisting or on static/
behavioral analysis, will prevent an unsuspecting user’s action from resulting in an
infection that could grow into a full-blown critical breach.
SANS ANALYST PROGRAM
17
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
Other important aspects our respondents considered in selecting their current tools and
technologies were full packet capture (52%) and memory analysis (50%). See Figure 11.
What do you consider the most important technical features for IR products
that affected either the make/buy decision or the product selection?
What additional features do you need?
70%
60%
50%
40%
making about
effective IR services
and programs.
SANS ANALYST PROGRAM
Open API
Ability to integrate with
other vendors
Common standards-based
IOC format (e.g., STIX)
Lightweight
Ability for host remediation
Retention
Memory injection detection
Sandbox analysis of binaries
Current
respondents as a
Memory analysis
their future decision
fewer than 10% of
Scanning
feature that will guide
was identified by
Collection of binaries
0%
Isolation of hosts
based technologies,
10%
Full packet capture
similar signature-
20%
SIEM integration
as antivirus and
30%
Blocking of execution
Scanning, such
Need
Figure 11. Key Technical Features
What is interesting from Figure 11 is that many of the less important features that
influenced current product selection or make/buy decisions appear to be growing in
importance as a future need. Scanning, such as antivirus and similar signature-based
technologies, was identified by fewer than 10% of respondents as a feature that will
guide their future decision making about effective IR services and programs. The
technical features that were key factors in acquiring respondents’ current tools will
continue to be necessary in effective investigations, but they have become expected
in IR technologies today. Figure 12 illustrates the key features respondents will be
concerned with in the future.
18
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
Features Still Needed in IR Practices
40%
35%
30%
25%
20%
15%
10%
5%
Scanning
Retention
Blocking of execution
Isolation of hosts
Collection of binaries
SIEM integration
Common standards-based IOC
format (e.g., STIX)
Full packet capture
Ability for host remediation
Memory injection detection
Sandbox analysis of binaries
Lightweight
Memory analysis
Ability to integrate with other vendors
Open API
0%
Figure 12. Key Technical Features Needed
The pace at which incident response teams must evolve to stay one step ahead of
adversaries (or even just one step behind with rapid detection) will be achieved only
through the streamlining of the IR process. Products that meet the feature requests of IR
teams, such as open APIs, the ability to integrate with other vendors, allowing for host
remediation, being lightweight and enabling memory analysis, will drive this progress.
SANS ANALYST PROGRAM
19
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
What Are Critical Gaps in Technologies and Tools?
The identified needs in future IR tools and technologies are expounded upon in the
responses from our survey pool concerning critical gaps. The most commonly cited
critical gap in current IR tool functionality is compatibility between IR tools, as illustrated
in Figure 13.
Critical Gaps in IR Tools
22%
TAKEAWAY:
The biggest gaps in current
IR technologies are a lack
of compatibility with
22% Interoperability
among tools
11%
11% E ase of tool
implementation
6%
6%
6%
6% Memory analysis
other products, ease of
6% Baselining capability
implementation, memory
6% Automation
analysis, lack of baseline
creation functionality and lack
of automation. Closing these
gaps is key to reducing false
positives and streamlining the
IR process.
Figure 13. Top Five Critical Gaps in IR Tools
Tools that have robust APIs allowing for compatibility between applications provide
for better data correlation and faster containment and remediation. And tools that
support the incorporation of collected data from various technologies reduce the time
required for the analyst to obtain the evidence necessary to begin an investigation. The
compatibility of applications, then, can streamline the IR process, resulting in time and
cost savings.
SANS ANALYST PROGRAM
20
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
Also mentioned by 11% of our respondents as a critical deficit in today’s tools is ease
of implementation and utilization. Respondents stated that their current tools and
technologies were fine, but lack of staff trained to effectively operate them proved to
be an obstacle. This may be indicative of tools lacking intuitive user interfaces and data
analysis processes being too complex.
Interviewees were not
convinced that onesize-fits-all integrated
security software
systems are always
the right choice. Some
prefer to pick best-ofbreed tools in each
category instead of
a multipurpose tool
that provides many
IR functions but may
not be able to provide
the visibility needed
for proper intrusion
analysis.
Memory analysis and correlating process injection detection capabilities also ranked
among the top gaps in IR technologies. The demand for these feature requests is
growing, because it is increasingly more common to see malware running on the system
solely in physical memory and leaving little or no trace on the file system for forensic
analysis. Visibility into real-time system state is a feature offered by endpoint threat
detection tools, monitoring for suspicious process creation or signs of code injection.
Collection and archiving of this continuous stream of system state data allow IR teams to
play back how and why malware dropped on a system, achieving in-depth visibility into
endpoints and speeding the overall investigative process.
Often limitations exist in how malware discovered in an organization’s environment can
be analyzed. Making use of web-based collective antivirus-scanning websites may not
be an option due to operational security concerns. Merely uploading a sample divulges
its identification to the attacker, who may be monitoring the site for uploads, and
enables the attacker to take evasive action, increasing the overall cost of the IR. In such
instances, some respondents have developed custom malware analysis sandboxes as
well as tools that integrate threat intelligence in their IR services.
Interviewees were not convinced that one-size-fits-all integrated security software
systems are the right choice for every environment. Some prefer to pick best-of-breed
tools in each category instead of choosing a multipurpose tool that provides many
IR functions but may not be able to provide the level of visibility required for proper
intrusion analysis.
For some small and midsize businesses, specialized commercial tools are out of reach
because neither the budget required for acquisition nor the funds to pay the analyst
costs to use them properly are available. As one survey respondent noted, “The tools
and technology are fine; the problem is lack of funding to hire skilled and experienced
operators.”
SANS ANALYST PROGRAM
21
The Race to Detection: A Look at Rapidly Changing IR Practices
Incident Response Technologies and Tools
(CONTINUED)
The Intangibles
Tools are necessary,
but they are not a
magic bullet. You need
people—key people—
to make the right
calls as to what tools
to use and how to
apply them.
—Ricardo Bruno,
CTO and Co-founder,
Activesec, Inc.
Factors cited in our interviews that are often overlooked when assessing and selecting
tools for IR are ease of installation (“Does it meet the 12-hour window to get up to
speed?”), baselining, past experience with the vendor, compatibility of tools, and
maintenance.
Expense is often an overriding factor. Many IR managers obtain an initial quote for a
tool they may have taken the time to evaluate in their environment, but they often fail
to assess the total cost of ownership of its implementation. Added costs also include
training staff to make use of security products; without the knowledge of how to use the
tool effectively, it is impossible to expect to improve security and reduce risk.
In many environments, the purchase and installation are approved, but allocations for
both training and training time for employees are overlooked or explicitly denied. For
59% of respondents, life-cycle costs are a factor in the decision to develop a custom
in-house tool or deploy a commercial tool. In a recent Ponemon report, higher than
expected installation costs were cited by 32% of respondents as the reason they regret
some of their technology investments.8
Other characteristics used in tool selection include whether a tool vendor offers 24x7
support for its product. Full features advertised by tool vendors are often not put to the
test until a critical incident strikes. Both in-house teams and consultants benefit from
having accessible and responsive technical support.
A common problem is that everybody is looking for the silver bullet
that doesn’t exist. We need to focus on the human element.
One of the most critical factors is the corporate users.
—Matt Dowling, STIGroup, Ltd.
8
SANS ANALYST PROGRAM
www.ponemon.org/blog/around-the-world-it-security-practitioners-face-a-common-problem-a-budget-that-is-inadequate-to-deal-with-cyber-threats
22
The Race to Detection: A Look at Rapidly Changing IR Practices
Recommendations for Building the Practice
The IR services arena is increasingly competitive, with more practices entering
the market. Yet robust IR practices do not mature overnight. An estimated 19% of
respondents have been building their IR practice for five years or more. At the other end
of the scale, more than 55% of respondents have been building their practice for only
two years or less.
What ranks as most important in building a robust and successful practice? Skilled
personnel rank at the top of the list according to 67% of our respondents, followed by
protecting the reputation and integrity of the practice (50%) and the ability to meet
customer IR needs (47%)—needs that can change rapidly. Figure 14 shows the relative
importance of the key elements.
Please rate the importance of each of these elements in building a robust IR practice.
(1 is the least important, 5 is most important.)
Skilled personnel
Reputation and integrity of your
practice
Ability to meet IR needs
Availability to provide services
when requested
Ability to interact with customer
Up-to-date technology and tools
Other
0%
5 (Most important)
10%
4
20%
3 (Important)
30%
2
40%
50%
60%
70%
1 (Least important)
Figure 14. Elements of a Robust IR Practice
Ability to meet customer IR needs is very closely tied to having skilled personnel. Robust
IR programs must be able to handle diverse and challenging scenarios with rapid
response time—expertise that only highly skilled technical professionals possess. Having
an experienced team in place can provide a more effective response and meet customer
demands quickly and efficiently.
SANS ANALYST PROGRAM
23
The Race to Detection: A Look at Rapidly Changing IR Practices
Recommendations for Building the Practice
(CONTINUED)
Meeting Customer Demands
As competition in the IR services space increases, it is imperative that IR service
providers streamline their processes, validate and extend their technological offerings,
and maintain skilled personnel to deal with customer engagements. These objectives
become increasingly more important as companies seek to distinguish themselves in the
marketplace. Though it is tempting to search for the most robust, feature-rich security
product offerings, place priority on the people using the technology rather than on the
technology itself.
Acquiring the tools
Respondents ranked employing skilled personnel as the most important element for
that best match the
sustaining their practice, followed by equal levels of importance for increased training
team’s skill sets will
and technology improvements. Building the right team with complementary skills is
always maximize an
organization’s return
on investment.
a more effective strategy than acquiring multiple tools. IR staff are often seen as the
solution to making security tools work effectively. But this is a false assumption. In
reality, acquiring the tools that best match the team’s skill sets will always maximize an
organization’s return on investment.
Budgeting Priorities
Only 68% of respondents anticipated receiving additional funding/budget to acquire
more skilled staff, the same percentage who reported a projection of additional budget
for future technology improvements. Increased training finished third, with 64%
anticipating the need for more funding in this area. It appears that the IR organizations
represented in this survey are anticipating spending their money to develop their most
important asset—the people who respond to incidents.
SANS ANALYST PROGRAM
24
The Race to Detection: A Look at Rapidly Changing IR Practices
Recommendations for Building the Practice
(CONTINUED)
Keeping Skills Alive
The reason for lower budgets for training may be that IR skills are not solely acquired
through formal training. Rather, they are learned in a master-apprentice relationship,
with a junior team member shadowing a senior member. It is in this manner that
institutional “tribal” knowledge can be passed on, including “lessons learned” from
past customer engagements and past incidents. We often think training programs and
education are the key sources of institutional knowledge about IR. However, 91% cited
practical experience as their source of training (see Figure 15).
An important element
How have you received your training in IR?
Select the top three.
when building a
robust IR practice is
100%
understanding that
80%
attack techniques
60%
to synthesize the
lessons learned in
multiple customer
Certification program
in forensics
and to be able
Other
0%
Certification program
in IR, such as SANS
continuous training
20%
Vendor-provided
training
it essential to have
40%
Practical experience
are evolving, making
Figure 15. Training in IR
environments.
Although varied options for technical education exist, many of the responding
—Blair Gillam,
organizations are critically understaffed, making the prospect of sending someone out
Breach Intelligence
of the shop for a week at a time for skills training daunting. Yet, continued training is
imperative because the attack and movement techniques employed by attackers morph
and have become more sophisticated.
IR practices need to seek ways to balance the demands on time and maintaining
practical knowledge. Training needs to be continuous, allowing staff to “play” with new
techniques and tools as part of their daily routine, giving them enough time to become
acquainted with how the threats and the market are evolving. Organizations must
make an investment in developing the expertise if a practice wants to remain a leader
in this space.
SANS ANALYST PROGRAM
25
The Race to Detection: A Look at Rapidly Changing IR Practices
Recommendations for Building the Practice
(CONTINUED)
The Bottom Line
Based on the results of our interviews and survey of IR professionals, we have gained
insight into the current state of IR, from the perspective of both an in-house team and an
IR service provider. Both subsets of the respondent pool are embattled by the shortage
of skilled personnel, difficulty of data correlation among tools, and increasingly more
sophisticated malware and attacker behaviors. Future developments in IR, based on our
polled respondent pool, will likely include escalating demands for outsourced as well as
in-house services, tempered by increased automation and improvement in technologies.
SANS ANALYST PROGRAM
26
The Race to Detection: A Look at Rapidly Changing IR Practices
About the Author
Alissa Torres is a SANS analyst and certified SANS instructor specializing in advanced computer
forensics and incident response (IR). She has extensive experience in information security in the
government, academic and corporate environments. Alissa has served as an incident handler and
as a digital forensic investigator on an internal security team. She has taught at the Defense Cyber
Investigations Training Academy (DCITA), delivering IR and network basics to security professionals
entering the forensics community. A GIAC Certified Forensic Analyst (GCFA), Alissa holds the GCFE,
GPEN, CISSP, EnCE, CFCE, MCT and CTT+ certifications.
Sponsor
SANS would like to thank this paper’s sponsor:
SANS ANALYST PROGRAM
27
The Race to Detection: A Look at Rapidly Changing IR Practices
Last Updated: September 30th, 2016
Upcoming SANS Training
Click Here for a full list of all Upcoming SANS Events by Location
SANS Seattle 2016
Seattle, WAUS
Oct 03, 2016 - Oct 08, 2016
Live Event
SANS Oslo 2016
Oslo, NO
Oct 03, 2016 - Oct 08, 2016
Live Event
SANS Baltimore 2016
Baltimore, MDUS
Oct 10, 2016 - Oct 15, 2016
Live Event
SANS Tokyo Autumn 2016
Tokyo, JP
Oct 17, 2016 - Oct 29, 2016
Live Event
SANS Tysons Corner 2016
Tysons Corner, VAUS
Oct 22, 2016 - Oct 29, 2016
Live Event
SANS San Diego 2016
San Diego, CAUS
Oct 23, 2016 - Oct 28, 2016
Live Event
SOS SANS October Singapore 2016
Singapore, SG
Oct 24, 2016 - Nov 06, 2016
Live Event
SANS FOR508 Hamburg in German
Hamburg, DE
Oct 24, 2016 - Oct 29, 2016
Live Event
SANS Munich Autumn 2016
Munich, DE
Oct 24, 2016 - Oct 29, 2016
Live Event
Pen Test HackFest Summit & Training
Crystal City, VAUS
Nov 02, 2016 - Nov 09, 2016
Live Event
SANS Sydney 2016
Sydney, AU
Nov 03, 2016 - Nov 19, 2016
Live Event
SANS Gulf Region 2016
Dubai, AE
Nov 05, 2016 - Nov 17, 2016
Live Event
DEV534: Secure DevOps
Nashville, TNUS
Nov 07, 2016 - Nov 08, 2016
Live Event
SANS Miami 2016
Miami, FLUS
Nov 07, 2016 - Nov 12, 2016
Live Event
European Security Awareness Summit
London, GB
Nov 09, 2016 - Nov 11, 2016
Live Event
DEV531: Defending Mobile Apps
Nashville, TNUS
Nov 09, 2016 - Nov 10, 2016
Live Event
SANS London 2016
London, GB
Nov 12, 2016 - Nov 21, 2016
Live Event
Healthcare CyberSecurity Summit & Training
Houston, TXUS
Nov 14, 2016 - Nov 21, 2016
Live Event
SANS San Francisco 2016
San Francisco, CAUS
Nov 27, 2016 - Dec 02, 2016
Live Event
SANS Hyderabad 2016
Hyderabad, IN
Nov 28, 2016 - Dec 10, 2016
Live Event
MGT517 - Managing Security Ops
Washington, DCUS
Nov 28, 2016 - Dec 02, 2016
Live Event
ICS410@Delhi
New Delhi, IN
Dec 05, 2016 - Dec 09, 2016
Live Event
SANS Cologne
Cologne, DE
Dec 05, 2016 - Dec 10, 2016
Live Event
SEC 560@ SANS Seoul 2016
Seoul, KR
Dec 05, 2016 - Dec 10, 2016
Live Event
SANS Dublin
Dublin, IE
Dec 05, 2016 - Dec 10, 2016
Live Event
SANS Cyber Defense Initiative 2016
Washington, DCUS
Dec 10, 2016 - Dec 17, 2016
Live Event
SANS Amsterdam 2016
Amsterdam, NL
Dec 12, 2016 - Dec 17, 2016
Live Event
SANS Frankfurt 2016
Frankfurt, DE
Dec 12, 2016 - Dec 17, 2016
Live Event
SANS DFIR Prague 2016
OnlineCZ
Oct 03, 2016 - Oct 15, 2016
Live Event
SANS OnDemand
Books & MP3s OnlyUS
Anytime
Self Paced