Security Metrics
Special Interest Group
Key Points Presentation
WARNING
This presentation is confidential and purely
for the attention of and use by organisations
that are Members of the Information Security
Forum (ISF). If you are not a Member of the
ISF or have received this document in error,
please destroy it or contact the ISF on
isfinfo@securityforum.org or on
+44 (0)20 7213 1745.
Any storage or use of this document by
organisations which are not Members of the
ISF is not permitted and strictly prohibited.
This document has been produced with care and to the best of our
ability. However, both the Information Security Forum and
Information Security Forum Ltd accept no responsibility for any
problems or incidents arising from its use.
Key findings (1 of 2)
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
3
Key findings (2 of 2)
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
4
About this presentation
The presentation summarises the research and conclusions
from the ISF Special Interest Group (SIG) on Security
Metrics.
The presentation can be used by Members to:
• understand the topic, without reading the associated
report
• gain an overview of the key issues and findings of the
project
• provide material for their own presentations on this topic.
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
5
The SIG project approach
The approach taken included:
1. Holding nine Member Work Group meetings
• Over 120 attendees
• Average attendee evaluation 4.3 out of 5
2. Analysing 56 Member-completed questionnaires
3. Interviewing 12 Members
• Covered most sectors and geographical locations
4. Researching published material on security metrics
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
6
Project Deliverables
Report
SIG meeting minutes
Key point presentation
These deliverables are also available on MX2
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
7
Outline of presentation
A.
Defining security metrics
B.
Member usage of security metrics
C.
Main issues
D.
Key actions
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
8
A. Defining security metrics
What are security metrics?
“ Objective, quantifiable measures against
specific targets that enable an organisation
to judge the effectiveness of information
security in that organisation.
“
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
10
What are security metrics?
“
Security metrics should be:
•
The information provided should:
-
Quantifiable
-
Allow effective analysis
-
Consistently measured
-
Enable reporting
-
Repeatable
-
Enhance understanding
-
Assist in managing information
security
-
Demonstrate the value of
information security to the
business
Metrics should be: timely;
reliable; trustable; accurate;
simple (at a certain level);
provable; meaningful and
easily understandable;
repeatable; verifiable; and
scaleable.
“
•
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
11
Characteristics of security metrics
Characteristic
Comments
Title
Purpose
Cost
Type
A meaningful title (or name) to describe the security metric
What the security metric is designed to do
An estimate or actual cost of collecting the security metric
What the security metric is, for example: technical, managerial, leading or
lagging; numerical or textual
 where the data for the security metric can be collected
 where previous data used in the security metric is located
 where previous instances of the security metric can be found
How often:
 the data needs to be collected
 the security metric needs to be presented
The category a security metric should be placed in:
 number
 frequency
 duration
 cost
Criteria for starting and stopping the:
 collection of data for the security metric
 use and presentation of the security metric
An estimate of, or actual, time period in which data will be collected
An estimate of, or actual, time period in which the security metric will be used
Location
Frequency
Category
Start / stop criteria
Duration of collection
Duration of use
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
12
Examples of security metrics by category
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
13
B: Member usage of security
metrics
A model for understanding security metrics
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
15
Common reasons for using security metrics
•
Managing information security
•
•
Providing information for
management reporting
Supporting risk-based approach
to information security
•
Supplying information for risk
management
•
Providing information about
information security risks
•
•
Indicating compliance to
legislation, regulation and
standards
Showing efficiency, effectiveness •
and performance against
objectives
•
Demonstrating the value of
information security
Highlighting information security
strengths and weaknesses
Benchmarking information
security arrangements
“
We need to continuously
improve and justify what we do
to management.
Copyright © 2006 Information Security Forum Limited
“
•
•
Security metrics SIG
16
What security metrics are currently used?
Risk management
-
Number of incidents
-
-
Number of business-critical
incidents
number of information risk
analyses performed
-
number of high/critical information
security risks identified
-
number of high/critical information
security risks mitigated
-
•
•
Incidents
Cost of individual incidents
Virus protection
-
frequency of virus incidents in a
specific period
-
frequency of virus incidents
compared to previous periods
-
number of viruses blocked at
gateway/perimeter defences
“
We only use the data we can get
our hands on easily. That may
not be the right thing to do.
“
•
•
Patch management
-
number of vulnerabilities
recorded/patches issued (per
period)
-
time to patch (eg estate or critical
systems/applications)
-
percentage of systems patched,
against Service Level
Agreement/policy
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
17
What security metrics are currently used?
•
•
Compliance
-
number of staff attending
awareness training
-
number of inappropriate internet
sites accessed Virus protection
Audit findings
•
Cost
-
total financial losses (eg lost
sales, orders or production)
caused by information security
incidents
-
total financial value of regulatory
or other fines imposed after
information security incidents
-
number of internal audit findings
-
number of external audit findings
(eg failure to comply with
regulation)
-
total financial losses due to fraud
(including legal and recovery
costs)
-
percentage of major information
security-related findings left
unresolved over a stated period
of time
-
total cost of security (cost of
controls + cost of incidents)
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
18
Audiences for security metrics
•
“
“
Metrics are a way of
communicating with the
board to gain backing for
your projects.
Most common audiences:
-
CISO
-
IT function
-
Senior Management
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
19
Examples of presentation methods
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
20
C: Main issues
Main issues with security metrics
Components of the model for
understanding security metrics
Member usage of security
metrics (from Part 3)
Issues identified
Why security metrics are used
 managing information security in an
organisation
 providing information for management
reporting
 indicating compliance to legislation,
regulation and standards
 supporting risk management activities
 no clear purpose
 difficult to relate security metrics to
the business
 incompatibility of security metrics
with business metrics
What is currently used and collected
 incidents
 virus protection
 risk management
 patch management
 compliance to internal policies
 audit findings
 cost
 presented to a range of audiences
 presented using a variety of different
formats
 difficult to select security metrics
 few, high-level, business-oriented
security metrics
 lack of a clear, enterprise-wide, view
of information security
How security metrics are used
 difficult to identify the correct
audience
 difficult to select and match the
presentation format to the audience
 inaccurate portrayal of information
security
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
22
Addressing the issues
•
Members agreed that the concepts of measuring security and security
metrics have considerable merit.
•
The management saying “you can’t manage what you can’t measure”
still holds true and many attendees agreed with this statement.
•
The issues identified here are not about security metrics in themselves
but about using the right security metrics for an organisation
•
Using the right security metrics delivers benefit and improves
communication with non-information security professionals
(eg business people, accountants, executives and managers).
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
23
D: Key actions
Key actions
A.
Define requirements
B.
Identify relevant security metrics
C.
Collect data required
D.
Produce security metrics
E.
Prepare presentations
F.
Use dashboards and scorecards
G.
Review the use of security metrics
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
25
Key Actions
A. Define requirements
B. Identify relevant security metrics
• Define and understand audience • Decide which security metrics to
requirements
use
“
Obtain funding
You have to understand the
requirements and have
objectives before you start
to collect metrics. You don’t
want to spend man-hours
collecting useless
information.
•
Review against objectives
•
Review the chosen security
metrics for ‘balance’
“
Metrics round off the picture
– but don’t forget the
intangibles!
“
•
Seek input from managers and
staff
“
•
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
26
Key Actions
A. Collect data required
• Define data required for use in
security metrics
D. Produce security metric
• Perform analysis and/or
aggregation of data
•
•
Analyse metrics
•
Test for correlation in dataset
•
Collect context data
•
Normalise and store the data
Metrics must have a context
– otherwise they may not
be understandable.
“
“
“
Business isn’t always
interested in numbers;
trends matter too.
Copyright © 2006 Information Security Forum Limited
•
“
Collect data for use in security
metrics
Security metrics SIG
27
Key Actions
E. Prepare presentations
• Match the presentation to the
audience
F. Use dashboards and/or
scorecards
• Dashboards
•
•
“
“
Fewer reports are required
if you have a security
dashboard – you can field
many enquiries with a
general response.
Balanced scorecards
The idea of using a
balanced scorecard
elegantly links information
security and business.
“
“
Select presentation formats
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
28
Key Actions
G. Review the use of security
metrics
• Review security metrics used
•
Review presentation format
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
29
Mapping the key actions with the model
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
30
Possible future development
Dashboard based on ISF products
(Survey, Healthcheck, Meta Standard)
Balanced scorecard based on
the Meta Standard
Copyright © 2006 Information Security Forum Limited
•
Security metrics SIG
31
Project contacts
Adrian Davis
Project Programme Manager:
Tel: +44 (0)207 213 3372
Email: adrian.davis@securityforum.org
Christopher Petch
Project Associate
Tel: +44 (0)207 212 3012
Email: christopher.m.petch@securityforum.org