Defining Security
Metrics
Security Planning
Susan Lincke
Security Planning: An Applied Approach | 3/23/2016 | 2
SABSA High-Level Framework
Security Planning: An Applied Approach | 3/23/2016 | 3
Gap Analysis
The difference between
where you are and where you
want to be: (For example:
# malware infections/month
Rate of finding illegal
software, hardware
Security awareness training
averages
Security Planning: An Applied Approach | 3/23/2016 | 4
SEI/COBIT Level 4 Monitoring:
Includes Metrics
Metrics inform management (and independent auditors) of the effectiveness of the
security program
Monitoring achievement of control objective may be more important than perfecting
security procedures
Security Planning: An Applied Approach | 3/23/2016 | 5
Which metrics to use?
Business-Driven
Addresses specific business risks
• Inherent industry risks
Tailored to organization
Measures adherence to control
objectives
Technology-Driven
Addresses recent threats
observed by CERT
• CERT: Computer Emergency
Readiness Team
Addresses recent forensic data
Security Planning: An Applied Approach | 3/23/2016 | 6
Monitoring Function:
Business-Driven Metrics
Executive mgmt is interested in
Strategic risk, budget, policy.
Metrics Review every 6 months-1 year
Metrics
Tactical
Metrics
Determine effectiveness of
security program: risk changes,
compliance, incident response tests.
Review quarterly to half-year
Operational
Metrics Technical details:
E.g., firewall, logs, IPS,
vulnerability tests.
Review weekly.
Automate statistics.
Security Planning: An Applied Approach | 3/23/2016 | 7
Monitoring Function:
Business-Driven Metrics
Project Plan or Budget Metrics
Strategic Risk performance
Metrics Disaster Recovery Test results
Audit results
Regulatory compliance results
Metrics
Tactical
Metrics
Policy compliance metrics
Exceptions to policy/standards
Changes in process or system
affecting risk
Incident management effectiveness
Operational Vulnerability Scan results
Metrics Server config. standards
compliance
IDS monitoring results
Firewall log analysis
Patch mgmt status
Security Planning: An Applied Approach | 3/23/2016 | 8
Which metrics?
Step 1: What are the most important security areas … threats …. regulation …
to monitor in your organization?
Step 2: Which metrics make the most sense to collect. Can they be
automated?
Step 3: Consider the 3 perspectives: strategic, tactical, operational metrics,
relative to 3 audiences.
Security Planning: An Applied Approach | 3/23/2016 | 9
Monitoring Function: Metrics
Risk:
The aggregate ALE
% of risk eliminated, mitigated,
transferred
# of open risks due to inaction
Cost Effectiveness:
What is:
Cost of workstation security per user
Cost of email spam and virus protection
per mailbox
Operational Performance
Time to detect and contain incidents
% packages installed without problem
% of systems audited in last quarter
Organizational Awareness:
% of employees passing quiz, after
training vs. 3 months later
% of employees taking training
Technical Security Architecture
# of malware identified and neutralized
Types of compromises, by severity &
attack type
Attack attempts repelled by control
devices
Volume of messages, KB processed by
communications control devices
Security Process Monitoring:
Last date and type of BCP, DRP, IRP testing
Last date asset inventories were reviewed
& updated
Frequency of executive mgmt review
activities compared to planned
Security Planning: An Applied Approach | 3/23/2016 | 10
Monitoring Function: Metrics cont’d
Security Management Framework:
Completeness and clarity of security
documentation
Inclusion of security in each project plan
Rate of issue recurrence
Compliance:
Rate of compliance with regulation or
policy
Rate of automation of compliance tests
Frequency of compliance testing
Secure Software Development:
Rate of projects passing compliance audits
Percent of development staff certified in
security
Rate of teams reporting code reviews on
high-risk code in past 6 months
Incident Response Metrics
# of Reported Incidents
# of Detected Incidents
Average time to respond to incident
Average time to resolve an incident
Total number of incidents successfully
resolved
Total damage from reported or detected
incidents
Total damage if incidents had not been
contained in a timely manner
Security Planning: An Applied Approach | 3/23/2016 | 11
Workbook: Metrics
Metrics Selected
What are the most important areas to monitor in your organization?
Lunatic gunman
FERPA Violation
Category
Major Risks:
Metric
Operational
Web Availability
Calculation & Collection
Method
Period of
Reporting
Information Tech. Group
1 year
Cost of incidents
Incident Response totals
6 months
% employees passing FERPA
quiz
Annual email requesting
testing
1 year
% employees completing
FERPA training
Two annual trainings with
1 year
sign-in. Performance review
# Hours Web unavailable
Incident Response form
6 months
# brute force attacks
Incident Response form
1 month
# malware infections
Incident Response form
1 month
Strategic Cost of security/terminal
Tactical
Cracking Attempt
Security Planning: An Applied Approach | 3/23/2016 | 12
SANS-Recommended
Critical Controls for Effective Cyber Defense
TECHNOLOGY-DRIVEN METRICS
Security Planning: An Applied Approach | 3/23/2016 | 13
Creating a baseline configuration of
network
Security Planning: An Applied Approach | 3/23/2016 | 14
Noticing inappropriate ‘additions’ to the
network
New PC
New AP
New
wireless
Security Planning: An Applied Approach | 3/23/2016 | 15
Checking the security configuration of
network
Patched? Legal software?
Firewall on & security configured?
Antivirus on and patched?
Limit USB access?
WPA2
AES,
EAP/
TLS?
Monitor
Network?
Withstands attacks?
SQL, buffer overflow,
cross-site scripting,
clickjacking, …
Security Planning: An Applied Approach | 3/23/2016 | 16
Noticing inappropriate actions
New sys
admin or
user acct
Transfer of
confidential
data or illegal
packets
Detect
new
network
service
Security Planning: An Applied Approach | 3/23/2016 | 17
SANS: Critical Controls for Effective Cyber
Defense
Typical SANS Metric:
Temporarily install unauthorized software, hardware or configuration on a
device. It should be:
• found within 24 hours (or best: 2 minutes)
• isolated within one hour confirmed by alert/email
• reported every 24 hours until issue is resolved.
Security Planning: An Applied Approach | 3/23/2016 | 18
SANS Critical Control 1:
Inventory of Authorized Devices
Ensure all devices (with IP address) on network are known,
configured properly, and patched.
Scan network daily or use DHCP reports or passive monitoring.
Compare results with baseline configuration.
Metric: Temporarily install unauthorized device.
Security Planning: An Applied Approach | 3/23/2016 | 19
SANS: Critical Control 2:
Inventory of Authorized Software
Ensure all software is approved and recently patched
• Whitelist defines the permitted list of software.
• Blacklist defines illegal software (e.g., IT tools).
• Endpoint Security Suites (ESS) contain antivirus,
antispyware, firewall, IDS/IPS, s/w white/blacklisting.
Metric: Temporarily install unauthorized software on a
device.
Security Planning: An Applied Approach | 3/23/2016 | 20
SANS Critical Control 3:
Secure Configurations for Hardware &
Software
All devices are hardened using recommended security
configurations
• Illegal software list exists, includes Telnet, VNC, RDP
• New software is quarantined and monitored.
• Imaged software is maintained in an updated state.
Build secure images, and use configuration checking tools daily.
Metric: Temporarily attempt to change a set of random
configurations.
Security Planning: An Applied Approach | 3/23/2016 | 21
SANS Critical Control 4:
Continuous Vulnerability Assessment
Run vulnerability scans on all systems at least weekly, preferably
daily. Problem fixes are verified through additional scans.
• Vulnerability scanning tools (updated) for: wireless, server,
endpoint, etc.
• Automated patch management tools notify via email when all
systems have been patched.
Metric: If the scan does not complete in 24 hours, an email
notification occurs.
Security Planning: An Applied Approach | 3/23/2016 | 22
SANS Critical Control 5:
Malware Defense
Antivirus/antispyware is always updated
• Run against all data: shared files, server data, mobile data.
Additional controls: blocking social media, limiting external
devices (USB), using web proxy gateways, network monitoring.
• Endpoint security suites report tool is updated and active on all systems
Metric: For install of benign malware (e.g., security/hacking tool),
antivirus prevents installation or execution or quarantines
software
• Sends an alert/email within one hour indicating specific device and owner
Security Planning: An Applied Approach | 3/23/2016 | 23
SANS Critical Control 6:
Application S/W Security
New application software is tested for security vulnerabilities:
• Web vulnerabilities: buffer overflow, SQL injection, cross-site scripting,
cross-site request forgery, clickjacking of code, and performance during
DDOS attacks.
• Input validated for size, type
• No system error messages reported directly to user
Automated testing includes static code analyzers and automated web
scanning.
Configurations include application firewalls and hardened databases.
Metric: An attack on the software generates a log/email within 24 hours (or
less).
Automated web scanning occurs weekly or daily
Security Planning: An Applied Approach | 3/23/2016 | 24
SANS Critical Control 7:
Wireless Device Control
Wireless access points are securely configured with WPA2 protocol and AES
encryption.
• Extensible Authentication Protocol-Transport Layer Security (EAP/TLS)
provides mutual authentication.
• Only registered, security-approved devices are able to connect
Wireless networks are configured for the minimum required radio footprint.
Metrics: Wireless intrusion detection systems detect available wireless access
points and deactivate rogue access points within 1 hour
Vulnerability scanners can detect unauthorized wireless access points
connected to the Internet.
Security Planning: An Applied Approach | 3/23/2016 | 25
SANS Critical Control 8:
Data Recovery Capability
Backups are maintained at least weekly and more often for
critical data.
Backups are encrypted and securely stored.
Multiple staff can perform backup/recovery.
Metric: Test backups quarterly for a random sample of systems.
This includes operating system, software, and data restoration.
Security Planning: An Applied Approach | 3/23/2016 | 26
SANS Critical Control 9:
Security Skills Assessment
Security awareness training: required for end users, system
owners
Security training: necessary for programmers, system, security
and network administrators
Metric: Test security awareness understanding
• Periodically test social engineering tests via phishing emails and
phone call
• Employees who fail a test must attend a class
Security Planning: An Applied Approach | 3/23/2016 | 27
SANS Critical Control 10:
Secure Network Configurations
A configuration DB tracks approved configurations in config.
mgmt. for network devices: firewalls, routers, switches.
Tools perform rule set sanity checking for Access Control Lists.
Two-factor identification is used for network devices.
Metric: Any change to the configuration of a network device is
reported within 24 hours
Security Planning: An Applied Approach | 3/23/2016 | 28
SANS Critical Controls
11. Control of Network
Ports, Protocols and
Services:
Default Deny packets.
Periodically review for
restriction
Metric: Measure time to
recognize added network
service
12. Controlled
Administrative Privilege:
Minimal elevated privileges
Passwords are complex, changed
periodically, 2-factor
Metric: Measure time to
recognize new sys admin
Security Planning: An Applied Approach | 3/23/2016 | 29
SANS Critical Controls
13. Boundary Defense:
Use firewall zones to filter
incoming and outgoing traffic.
Blacklist & whitelist network
addresses
14. Analysis of Security Audit
Logs:
Server logs are write-only and
archived for months.
Firewalls log all allowed and
blocked traffic.
Unauthorized access attempts are
Metric: Measure time to
recognize unauthorized packets logged
Metric: Measure time to
recognize no log space
Security Planning: An Applied Approach | 3/23/2016 | 30
SANS Critical Controls
15. Need to Know Access:
Prevent exfiltration of data (e.g., to
competitors)
Classify data
Use restrictive firewall configurations
Log access to confidential data
Metric: Measure time to recognize
unauthorized access
16. Account Monitoring and
Control:
Terminated accounts -> removed
Expired password/ disabled/ locked out
accounts, -> investigated
Failed logins -> lockouts
Inactivity -> locked sessions
Unusual time access -> alert
Data exfiltration recognized by keywords.
Metric: Measure time to recognize new/
changed user accounts
Security Planning: An Applied Approach | 3/23/2016 | 31
SANS Critical Controls
17. Data Loss Prevention:
18. Incident Response:
Prevent exfiltration of
proprietary or confidential info
• Encrypt mobile and USB
devices
• Disable USB
Incident Response Plan defines
who does what for various
conditions
IRP includes contact information
for third party contractors
Metric: Measure time to
recognize transfer of
confidential data file
Security Planning: An Applied Approach | 3/23/2016 | 32
SANS Critical Controls
19. Secure Network
Engineering:
Separate zones exist: DMZ,
middleware, private network
• DMZ accessed through proxy
firewall
• DMZ DNS is in DMZ; internal DNS
is in internal zone, …
Emergency config. for restricted
network is ready for quick
deployment.
20. Penetration Tests:
Penetration tests =
vulnerability tests + attacker
tests
Red Team exercises test
incident response team
reactions
Metric: Measure false
positive, false negative, true
positive rate
Security Planning: An Applied Approach | 3/23/2016 | 33
Question
The difference between where an organization performs and
where they intend to perform is known as:
1. Gap analysis
2. Quality Control
3. Performance Measurement
4. Benchmarking
Security Planning: An Applied Approach | 3/23/2016 | 34
Question
The MOST important metrics when measuring compliance
include:
1. Metrics most easily automated
2. Metrics related to intrusion detection
3. Those recommended by best practices
4. Metrics measuring conformance to policy
Security Planning: An Applied Approach | 3/23/2016 | 35
Question
SANS recommends that an initial maximum allowable time to
detect a problem in a network or server configuration is:
1. Two minutes
2. One hour
3. One day
4. One week