Security Metrics
Spring 2005
CS996
Information Security Management
Polytechnic University
Michael Aiello
Overview
•
•
•
•
•
•
•
•
•
•
Why do we care?
What is a metric?
How do we decide which metrics to collect?
How are they collected?
Effective risk analysis through security metrics
How do security metrics make a corporation money
(operational risk)?
“Compliance competition” and security ROI
POSA Example
Problems Experienced
Homework
Why do we care?
• How do we know how “secure” an
organization is?
– Metrics help define “secure”
– Metrics let us benchmark our security
investments against other organizations
– Compliance
– The metrics “gathering” process often leads to
identification of security inconsistencies or
holes
Why do we care: Example
• Manager asks, “Are we secure?”
• Without metrics:
“Well that depends on how you look at it.”
• With metrics:
“No doubt about it. Look at our risk score
before we implemented that firewall
project. It’s down 10 points. We are
definitely more secure today than we were
before.”
Why do we care: Example
• Manager Asks: “Have the changes that we
implemented improved our security posture?”
• Without metrics:
“Sure. They must have, right?”
• With metrics:
“Absolutely. Look at our risk score before we
made the recommended changes, and now it’s
down 25 points. No question, the changes
reduced our security risk.”
Motorola CISO on Metrics
• “Security experts can't measure their
success without security metrics, and what
can't be measured can't be effectively
managed.” (William Boni, President
CISO, Motorola Inc. www.secmet.org)
What is a metric?
• The National Institute of Standards and Technology
(NIST) define metrics as tools designed to facilitate
decision-making and improve performance and
accountability through collection, analysis, and reporting
of relevant performance-related data. Metrics are simply
a standard or system of measurement. In this case, it is
a standard for measuring security, specifically measuring
an organization’s security posture. Although there are
some published standards for measuring security, ideally
security metrics should be adjusted and tuned to fit a
specific organization or situation.
Examples of metrics
• Total number of remote connections over a one
month period (VPN, ISDN, dial-up, remote
desktop)
• Maximum number of concurrent remote by user
• The percentage of total applications that have a
contingency plan by application criticality.
• Time to analyze and recommend action on a
security event
• Number of Linux servers at least 90% compliant
with the Linux platform security standard
Security Metric Categories
• Platform
– Number of Linux servers that are compliant with EFS policy
• Network
– DMZ port scans
• Incident
– Number of hosts infected with worm XYZ
• Vendor
– Average security rating for vendors that touch active customer files
• People
– Number of terminated employees with administrator access
• Industry
– Number of public security incidents in sector ABC with severity score Z
• Political
– Hacktivism scores, amount of sites listing sector/company ABC as
potential target
Security Metric Types
• Real Time
– Number of concurrent connections to VPN
– Usually from incident response systems
• Polled
– Number of password reset requests (monthly),
– Usually from SA’s or SME’s
• Incident based
– Number of machines infected with worm XYZ
– Number of vendors suffering from infections of worm
XYZ
– Usually from industry intelligence/incident
response/SA’s and SME’s
How do we decide which one’s to
collect?
•
•
•
•
•
Policy Mining / Easy to Spot Anomalies
Risk Scoring
ROI / Vendor Evaluations
Regulatory / Cover the industry standards
“Tips” / Visionaries
Policy Mining Example
•
•
•
•
•
•
•
•
•
•
Policy Statement: All users who connect remotely must be uniquely
authenticated.
Enforcement Mechanism: Users are required to authenticate with a
username, password and securID token to gain access to the internal
network.
Network Policy (VPN): A user Kerberos account must authenticate with both
the Radius and securID privilege granting servers before VPN connectivity
is established
Question: How do we tell if a user is uniquely authenticated?
Metric: Maximum number of remote connections by user in a month.
Metric: Maximum number of concurrent connections for a single user
Metric: Total time connected in a single month
Metric: Number of users granted remote VPN privileges
Metric: Number of securID reset requests in a given month
Metric/Alert: user connecting to VPN from different countries simultaneously
Risk Scoring
• Metric: Maximum number of remote
connections by user in a month.
– Impact: 6/10 (we care about this 6/10 relative
to other metrics in this policy’s risk, this score
may come from SME’s/upper
management/industry direction)
– Risk: 20% + (10% * last month’s count)
– this is where the “soft” analysis takes place
ROI / Vendor Evaluations
• “We spent $XXXXX on 4 new application
penetration testers, are our applications more
secure now? Should we hire another one?”
– Metrics specific to applications not pen-tested, and
those that are
• “We are spending $YYYY on product XYZ, is it
worth it to renew the contract or should we start
looking for a new solution?”
– Metrics specifically surrounding product XYZ and the
problem it is solving
• Number of successful social engineering attacks and their
impact before and after the online training seminar
Regulatory
(doesn’t yet exist for most industries)
• Baseline metrics (from Spire Security)
– Number of patched machines / total
– Number services running on external facing
machines
– Port Scanning
– Incidents
Standards (from Spire Security)
Finances
Market Cap
Overall Revenue/Funding level
Overall Expenses
Workforce
Number of Employees
Number of contractors/temps
Number of locations with dedicated IT
IT Spending
Budgets for Operations, Maintenance, Capital
Employees
Equipment
Count of Servers, appliances, databases, client PCs, Laptops,
PDAs
Network Traffic
Count of flows, possible flows, actual flows, blocked flows, sessions,
commands, transactions
Security Spending
Operations, Maintenance, Capital Expenditures, Number of Security
FTEs
Standards contd.
Identity Management
Management budget
Management FTEs
Total User Repositories
Total User Accounts
Count of user accounts created, accounts modified, password
resets, accounts disabled/deleted, accounts evaluated
Authentication Events
Number of failed authentications
Vulnerability Management Spending
Number of servers/applications/PCs scanned
Number of Vulnerability Management FTEs
Count of open ports, known vulnerabilities, patches, configuration
changes
Trust Management spending
Count of Trust Management FTEs, policies written, certificates
issued, signed documents, encrypted documents
Threat Management Spending
Count of Threat Management FTEs, alerts, compromised systems
“Tips” / Visionaries
• Investigations/Government/Regulator may
ask information security to “monitor”
specific activities
• A “visionary” (author/upper
management/consultant) will come up with
a new/derived metric to collect in order to
report on a new phenomenon
How are metrics collected?
• Categorize and define the metric and its owner
• Determine and document metric source
– Automated
• database connection
• Script file output
– Manual
•
•
•
•
Email polling
Form entry
Manual file updating
Report analysis/research
How are metrics collected?
• Define/document collection process for each metric
– A pull replication query mirrors the critical IDS alerts from server
ABC database BCA to the metrics collection server DEF
database BCA. DEF then sums and categorizes the alerts. The
final counts are archived in table QRS in database BCA on
server DEF.
– Joe runs a stored procedure on server XYS database YZD which
he manually correlates with Radius logs aging over the past 3
months. The report is then stored on share ABCD and Joe sends
an email to Sally indicating the metric is updated. Sally then
enters the metric information into the metric collection database
using the form at URLQYZX
Effective risk analysis through
security metrics
• How do we make decisions based on the
metrics now that we have them?
• Metrics which are collected should match
high impact risk items. (only spend money
collecting those with high risk scores)
Risk Breakdown Example
• Risk Measurement: Federation information security risk
score (akin to homeland security colors extremely vague,
policy generally shouldn’t be created based on such high
indicators)
• Risk Components: Network, Incident, Vendor, People,
Industry, Political
• Subcomponent: Federation-Global-Network-TradingFTSE
• Metric inventory for subcomponent:
ID4786(A),ID2235(B),ID8674(C)
• Subcomponent risk score calculation:
• 50%(A*(∆last 4 months(B))) + (50% * C’s rolling average)
• Security risk analysts and SMEs create score weightings
This is complicated and expensive,
why do we do it this way?
• As people, we are generally bad at
concentrating on more than 7
factors/metrics/indicators at a time
• Risk scoring lets us define and objectively
monitor the “big picture” information
security view
• Correlations
• Alerting / “Smarter” automated responses
How do security metrics make a
corporation money (operational risk)?
• Legislation (Basel II) says “you have to withhold
15% of last year’s revenue unless you can prove
that you have mitigated your risk”
• Metrics are your proof, risk scores are your slice
description of the “state of the union”
• In general, the less money you have to
withhold/spend on insurance, the more money
you make
“Compliance competition” and
security ROI
• The faster/better you can prove compliance the
higher the bar is for your competition.
– Reports and systems that are easy for auditors to
use.
– Meaningful and provable risk scores with metrics
– Higher bar for competition means they spend more
time/money/effort complying
• Metrics give us a way of measuring security ROI
– Pretend we are auditors evaluating our own
organization: now we can measure our security
posture before and after implementing a solution
POSA Example
Polytechnic University
CS996 Information Security Management
(example POSA system here)
• Quick reminder of system
4 Sale & user information
8 Complete transaction
CFAC
5 Y/N
1 Sale information
7 Complete Trans.
Register
6 Y/N
POSA
2 Display
Sale Info
3 User CC
information
USER
PSOA Metrics
• After doing thorough risk analysis and identifying
Assets/Threats/Vulnerabilities assume the
following combinations are deemed the most
important:
Asset
Threat
Vuln
Credit Card
Database on CFAC
Insider Employee
Mal-intent System
Administrators
Credit Card
Database on CFAC
External “hackers”
Network hopping/
Sniffing
Payment Processing
Availability
Competition/Hactivists Denial of service
Natural Disasters
Physical equipment
destruction
Ask the question, do we have policy and
guidelines to mitigate/monitor these
combinations?
• If not, create them
• In our case, assume we do
• Mine these policies for potential metrics
Asset
Threat
Vuln
Credit Card Database on
CFAC
Insider Employee
Mal-intent System
Administrators
Credit Card Database on
CFAC
External “hackers”
Network hopping/ Sniffing
Payment Processing
Availability
Competition/Hactivists
Natural Disasters
Denial of service
Physical equipment destruction
Risk Measurement: Customer Credit Card
Privacy Score
• Risk Components: Network, Incident, Vendor, People, Industry
• Subcomponent: Firmwide-Platform-Database-Internal-Access
Control
• Metric inventory for subcomponent:
(A)(Real-Time)Number of databases with more than 100 credit cards which do not store
credit card numbers in an encrypted manner:
(B)(Real-Time)Number of system administrators with view level access to non-encrypted
databases with more than 100 credit cards:
(C)(Real-Time)Number of system administrators who have criminal history with view level
access to non-encrypted databases with more than 100 credit cards
(D)(Real-Time)Number of system administrators who have criminal history with view level
access to encrypted databases with more than 100 credit cards
(E)(Monthly) Number of employees leaving the firm which have had access to nonencrypted databases with more than 100 credit cards:
(F)(Real-Time)Number of databases with encrypted credit cards
(G)(Real-Time)Number of administrators with ability to decrypt encrypted credit cards
(H)(Real-Time)Age of keys used to encrypt credit cards (days)
(I)(Manual)Industry time to break one credit card encryption (days)
(J)(Manual)% of administrators who have attended social engineering seminars
Subcomponent risk score calculation:
(0.75 * A * ((B * .1) + (C * .3))) * (0.25 * F * ( ....) )
How do we do the risk score
calculation?
• Risk = Threat * Vulnerability * Expected Loss
• ALE (Average Loss Expectancy) = probability of loss *
total loss potential
• Asset Valuing
–
–
–
–
Productivity Value
Revenue Value
Liquid Financial Assets Value
Intellectual Property Value
• Potential Loss
–
–
–
–
–
Confidentiality
Integrity
Availability
Productivity
Liability
How do we do the risk score
calculation?
• Measuring Risk
– Manifest risk = ratio of malicious events to total
events
• (sessions, commands transactions)
– Inherent risk = likelihood that a configuration will
contribute to a compromise
• Open ports and services running compared to historical
vulnerabilities on those ports
– Contributory risk = measure of process errors during
normal course of operations that contribute to a
compromise
• User Account Management procedures
How do we do the risk score
calculation?
• Correlation
– Expert systems, obvious correlations
• Historical testing
– Look at data leading up to an incident, see what changed
• SME predictions/insight
– Sometimes they just know
• Industry trends
– We don’t really care about 1024 bit key breaking in 2005, will we
in 2012?
• Firm specific
– Explosion in certain incident types may necessitate a change in
the equation
Optimization
• After doing risk calculation, don’t spend
time/money to collect metrics that don’t change
the final score that much
• Analyze risk score equation against reality, are
we really reporting the proper state of the union?
– Add “reliability” fields to metrics and weigh according
to them as well.
• Correlation/expert system analysis
• Prioritize future metrics to be collected (find the
“value” of a metric (risk of not collecting the
metrics))
Problems Experienced
• Systems unable to report highly critical metrics
• System integration
• SAs not willing/able to invest time setting up processes to collect
metrics
• Ad-Hoc requests/reports and their implication on the overall view
– We didn’t do a penetration test on most of our servers, how
can call our network secure?
• Vague risk descriptions, Vague Metric Requests
• Impossible metrics (usually external)
– Number of credit card accounts compromised globally across
any firm
• Missing/incomplete historical data
• Mistrust/inaction/devaluing because of qualitative components
• Complete trust
References
• http://www.secmet.org
• http://csrc.nist.gov/publications/nistpubs/8
00-55/sp800-55.pdf
• http://www1.netsec.net/content/securitybri
ef/archive/2004-09_Metrics.pdf
• http://www.cert.org/octave/
Homework
•
•
•
•
•
Pretend you are a security analyst at Polytechnic. You have just had the
following (highly simplified and fictitious) conversation with a senior
manager.
Manager: “Another engineering school was just sued because student’s
transcripts could be accessed by anyone online. How secure is our new
grade transaction server?”
You: “We believe the new design is secure, however, haven’t allocated time,
money and effort to post-implementation security evaluation.
Manager: “I need to show the board that we are not prone to this type of
humiliation, what can you pull together for me in the next 3 months?”
Your assignment is to describe how you would structure your new metrics
proposal which includes the following sections.
– Description of which metrics you will be collecting (Based on risk analysis.
Remember, this should be a minimal set, you only have 3 months to set this up)
– A metric collection process example for one of the metrics
– Suggested, simple weighting of metrics to calculate the overall risk score of the
system.