Everything You Always Wanted to Know About Maturity Models
Dr. Nader Mehravari
Research Scientist, CERT® Division
Dr. Nader Mehravari is with the CERT® Program at the Software Engineering
Institute (SEI), a unit of Carnegie Mellon University in Pittsburgh, PA. His current
areas of interest and research include operational resilience, protection and
sustainment of critical infrastructure, preparedness planning, and associated risk
management principles and practices.
Nader was with Lockheed Martin from 1992 through 2011. In his most recent
assignment, he was the Director for Business Resiliency. In this capacity, he led and
oversaw all preparedness planning and associated governance and compliance
activities. He was responsible for building and leading Lockheed Martin's resiliency
program where he successfully implemented a modern, integrated, risk management
based approach to disaster recovery, business continuity, pandemic planning, crisis
management, emergency management, and workforce continuity for all of Lockheed
Martin.
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Notices
Copyright 2014 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No.
FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a
federally funded research and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the United States Department of Defense.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO
WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING,
BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY,
EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON
UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM
PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or
electronic form without requesting formal permission. Permission is required for any other use. Requests for
permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.
Carnegie Mellon® and CERT® are registered marks of Carnegie Mellon University.
DM-0000899
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Outline
Setting the Stage
—
The need for “measuring” operational activities & their effectiveness
—
Are we doing the right things?
—
Are we using the right tools to measure?
—
Are we measuring the right things?
ABCs of Maturity Models
—
What are Maturity Models?
—
Types of Maturity Models
—
Examples of Maturity Models
Closing Thoughts
—
A few cautions
—
Determining when and which type to use
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Setting the Stage
•  The need for “measuring” operational activities & their effectiveness
•  Are we doing the right things?
•  Are we using the right tools to measure?
•  Are we measuring the right things?
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Today’s Operating Environment
Rapid changes in
technology and its
application in a wide range
of industries.
Introduction of many new
systems, business
processes, markets, risks,
and enterprise approaches.
Many immature products
and services being
consumed by enterprises
that themselves are in a
state of change.
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Challenges at Hand
How can you tell if you are doing a good job of managing these changes?
How best to monitor your progress on an ongoing basis?
How do you manage the interactions of systems
and processes that are continually changing?
How do poor processes impact
interoperability, safety, reliability,
efficiency, and effectiveness?
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Which tool should I use?
Your organization wants to know SOMETHING about your
mission operation:
•  How EFFECTIVE are we?
•  Do we have the right SKILLS and CAPABILITIES?
•  Do we have the right TECHNOLOGIES?
OR
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Observation
The development and use
of maturity models in security,
continuity, IT operations, &
resilience space is increasing
dramatically.
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Do maturity models measure the right thing?
v  May not measure what you think it measures
Ø  Practice maturity vs. organizational maturity?
v  May give you inaccurate data on which to
base decisions
Ø  Process performance vs. product performance?
v  Can increase cost but reduce benefit
Ø  An improved process may not result in compliance
v  May provide a false sense of confidence
Ø  A robust process may not stop all malware
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
ABCs of Maturity Models
•
What are Maturity Models?
•
Types of Maturity Models
•
Examples of Maturity Models
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Maturity Model Defined
An organized way to convey a path of experience, wisdom,
perfection, or acculturation.
Depicts an evolutionary progression of an attribute,
characteristic, pattern, or practice.
The subject of a maturity model can be objects or things,
ways of doing something, characteristics of something,
practices, controls, or processes.
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Maturity Models Provide…
Means for assessing and benchmarking performance
Ability to assess how a set of characteristics have evolved
Expression of body knowledge of best practices
Identification of gaps and improvement plans
Roadmap for model-based improvement
Demonstrated results of improvement efforts
Common language or taxonomy
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Key Components of a Maturity Model
Levels
•
The measurement scale
•
The transitional states
Domains
•
Logical groupings of like attributes into areas of importance to the subject matter
and intent of the model
•
Logical groupings of like practices, processes, or good things to do
Attributes
•
Core content of the model arranged by domains and levels
•
Typically based on observed practices, standards, or expert knowledge
Diagnostic Methods
•
For assessment, measurement, gap identification, benchmarking
Improvement Roadmaps
•
To guide improvement efforts (e.g., Plan-Do-Check-Act)
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Types of Maturity Models
There are three types of maturity models
•  Progression Maturity Models
•  Capability Maturity Models (CMM)
Information Technology/Risk Management
Part Two focuses on using CERT-RMM to establish a foundation for sustaining operational resilience
management processes in complex environments where risks rapidly emerge and change.
Part Three details all 26 CERT-RMM process areas, from asset definition through vulnerability
resolution. For each, complete descriptions of goals and practices are presented, with realistic examples.
Part Four contains appendices, including Targeted Improvement Roadmaps, a glossary, and other
reference materials.
This book will be valuable to anyone seeking to improve the mission assurance of high-value services, including
leaders of large enterprise or organizational units, security or business continuity specialists, managers of large
IT operations, and those using methodologies such as ISO 27000, COBIT, ITIL, or CMMI.
informit.com/seiseries
cert.org/resilience
Cover illustration by nikamata/iStock.com
Text printed on recycled paper
ISBN-13: 978-0-321-71243-1
ISBN-10:
0-321-71243-9
9
5 7 9 9 9
Caralli
Allen
White
The authors are senior technical staff members within the CERT Program of the Software Engineering
Institute (SEI). Richard A. Caralli, Resilient Enterprise Management technical manager, develops and
delivers methods, tools, and techniques for enterprise security and resilience management. He led the
development of CERT-RMM. Julia H. Allen develops and transitions executive outreach programs in
enterprise security and governance and researches software security and assurance. She served as the SEI’s
Acting Director and Deputy Director/COO and authored The CERT® Guide to System and Network Security
Practices (Addison-Wesley, 2001). David W. White, a core member of the CERT-RMM development team,
develops CERT-RMM and related products and helps organizations apply them.
®
A Maturity
Model for
Managing
Operational
Resilience
SEI SERIES t A CERT ® BOOK
Part One summarizes the value of a process improvement approach to managing resilience, explains
CERT-RMM’s conventions and core principles, describes the model architecturally, and shows how it
supports relationships tightly linked to your objectives.
CERT Resilience
Management Model
®
This book both introduces CERT-RMM and presents the model in its entirety. It begins with essential background
for all professionals, whether they have previously used process improvement models or not. Next, it explains
CERT-RMM’s Generic Goals and Practices and discusses various approaches for using the model. Short essays
by a number of contributors illustrate how CERT-RMM can be applied for different purposes or can be used
to improve an existing program. Finally, the book provides a complete baseline understanding of all 26 process
areas included in CERT-RMM
One or more may be appropriate
for your particular needs
CERT -RMM, Version 1.1
®
CERT® Resilience Management Model (CERT-RMM) is an innovative and transformative way to
manage operational resilience in complex, risk-evolving environments. CERT-RMM distills years of research
into best practices for managing the security and survivability of people, information, technology, and
facilities. It integrates these best practices into a unified, capability-focused maturity model that encompasses
security, business continuity, and IT operations. By using CERT-RMM, organizations can escape silo-driven
approaches to managing operational risk and align to achieve strategic resilience management goals.
CERT Resilience
Management Model
•  Hybrid Maturity Models
Richard A. Caralli
Julia H. Allen
David W. White
780321 712431
$79.99 US | 95.99 CANADA
Not all maturity models are CMMs
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Maturity Models
Simple progression or scaling of an attribute,
characteristic, pattern, or practice
Levels describe higher states of achievement,
advancement, completeness, or evolution
Levels can be arbitrary as agreed upon by
users, industry, etc.
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Maturity Models - Example
A Maturity Progression for
Toy Building Bricks
Lego Mindstorms
Lego Architecture
Lego Technic
Lego City
Lego Duplo
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Maturity Models - Example
A Maturity
Progression for
Human Mobility
A Maturity Progression for
Authentication
Three-factor authentication
Two-factor authentication
Fly
Addition of changing every 60 days
Sprint
Use of strong passwords
Run
Use of simple passwords
Jog
Walk
Crawl
Progress does not necessarily equate to maturity
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Maturity Models - Example
Higher levels may be characterized as “tool-­‐enabled” A Maturity
Progression for
Counting
Computer
Calculator
These characteriza,ons are typically arbitrary Lower levels may be characterized as “primi6ve” Adding machine
Slide rule
Abacus
Pencil and paper
Sticks/Stones
Fingers
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Model Example: SGMM
Smart Grid Maturity Model
5 4 175 Characteris6cs: Features you 3 would expect to see at each stage of the smart grid journey 2 1 0 SMR
Strategy,
Management, &
Regulatory
OS
Organization &
Structure
GO
Grid Operations
WAM
Work & Asset
Management
TECH
Technology
CUST
Customer
VCI
Value Chain
Integration
SE
Societal &
Environmental
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Benefits and Limitations of Progression Models
Benefits
Limitations
v  Provides a transformative
roadmap
v  Levels are arbitrarily defined
and may be meaningless
v  Simple to understand and
adopt; low adoption cost
v  Achieving higher levels does
not necessarily translate into
“maturity”
v  Easy to recalibrate as
technologies and practices
advance
v  Often confused with CMMs—
thus users inaccurately project
traits of CMMs on progression
models
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Capability Maturity Models (CMM)
A more complex instrument
Characterizes
•  the maturity of processes
•  the degree to which processes are institutionalized
•  the degree to which the organization demonstrates process maturity
to be retained during times of stress.
®
Part One summarizes the value of a process improvement approach to managing resilience, explains
CERT-RMM’s conventions and core principles, describes the model architecturally, and shows how it
supports relationships tightly linked to your objectives.
Part Two focuses on using CERT-RMM to establish a foundation for sustaining operational resilience
management processes in complex environments where risks rapidly emerge and change.
Part Three details all 26 CERT-RMM process areas, from asset definition through vulnerability
resolution. For each, complete descriptions of goals and practices are presented, with realistic examples.
Part Four contains appendices, including Targeted Improvement Roadmaps, a glossary, and other
reference materials.
This book will be valuable to anyone seeking to improve the mission assurance of high-value services, including
leaders of large enterprise or organizational units, security or business continuity specialists, managers of large
IT operations, and those using methodologies such as ISO 27000, COBIT, ITIL, or CMMI.
The authors are senior technical staff members within the CERT Program of the Software Engineering
Institute (SEI). Richard A. Caralli, Resilient Enterprise Management technical manager, develops and
delivers methods, tools, and techniques for enterprise security and resilience management. He led the
development of CERT-RMM. Julia H. Allen develops and transitions executive outreach programs in
enterprise security and governance and researches software security and assurance. She served as the SEI’s
Acting Director and Deputy Director/COO and authored The CERT® Guide to System and Network Security
Practices (Addison-Wesley, 2001). David W. White, a core member of the CERT-RMM development team,
develops CERT-RMM and related products and helps organizations apply them.
informit.com/seiseries
cert.org/resilience
Cover illustration by nikamata/iStock.com
Text printed on recycled paper
ISBN-13: 978-0-321-71243-1
ISBN-10:
0-321-71243-9
9
5 7 9 9 9
CERT Resilience
Management Model
®
A Maturity
Model for
Managing
Operational
Resilience
SEI SERIES t A CERT ® BOOK
This book both introduces CERT-RMM and presents the model in its entirety. It begins with essential background
for all professionals, whether they have previously used process improvement models or not. Next, it explains
CERT-RMM’s Generic Goals and Practices and discusses various approaches for using the model. Short essays
by a number of contributors illustrate how CERT-RMM can be applied for different purposes or can be used
to improve an existing program. Finally, the book provides a complete baseline understanding of all 26 process
areas included in CERT-RMM
Caralli
Allen
White
•  Institutionalized processes are more likely
CERT -RMM, Version 1.1
®
Levels reflect the degree to which a particular
set of practices have been institutionalized
Information Technology/Risk Management
CERT® Resilience Management Model (CERT-RMM) is an innovative and transformative way to
manage operational resilience in complex, risk-evolving environments. CERT-RMM distills years of research
into best practices for managing the security and survivability of people, information, technology, and
facilities. It integrates these best practices into a unified, capability-focused maturity model that encompasses
security, business continuity, and IT operations. By using CERT-RMM, organizations can escape silo-driven
approaches to managing operational risk and align to achieve strategic resilience management goals.
CERT Resilience
Management Model
•  the maturity of the culture of the organization
Richard A. Caralli
Julia H. Allen
David W. White
780321 712431
$79.99 US | 95.99 CANADA
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
What do these organizations have in common?
Chain of Command
Unit Cohesion
Regulations
Customer Happiness
Strong
Culture
Customer Service
Tradition
Protection
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
CMM Levels – An Example
Processes are acculturated, defined, measured, and governed Prac,ces are performed Prac,ces are incomplete Level 3
Level 2
Higher degrees of
institutionalization
translate to more stable
processes that
•  Managed •  are repeatable
Level 1
results over time
•  Defined •  Performed •  produce consistent
•  are retained during
times of stress
Level 0
•  Incomplete CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
CMM Levels – Another Example
Example 3
Optimized
Continuous and effective,
integrated, proactive, usually
automated
Measured
Well-managed, formal, often
automated, evaluated frequently
Defined
Intuitive, not documented, occurs
only when necessary
Repeatable
Intuitive, not documented, occurs
only when necessary
Ad hoc
Occasional, not consistent, not
planned, disorganized
Nonexistent
Not understood, not formalized,
need is not recognized
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Examples of CMM Levels
Example 1
Example 3
Optimized
Shared
Quantitatively Managed
Defined
Defined
Measured
Managed
Managed
Ad hoc
Planned
Example 2
Externally integrated
Performed but ad hoc
Incomplete
Internally integrated
Managed
Performed
Initiated
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Capability Maturity Model Example: CERT-RMM
Framework for managing and improving opera6onal resilience http://www.cert.org/resilience/
“…an extensive super-set of
the things an organization
could do to be more resilient.”
- CERT-RMM adopter
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
CMM Example: CERT-RMM
CERT-RMM Process Areas (Domains)
Access Management
Measurement and Analysis
Asset Definition and Mgmt.
Monitoring
Communications
Organizational Process Focus
Compliance
Organizational Process Definition
Controls Management
Organizational Training & Awareness
Enterprise Focus
People Management
Environmental Control
Resilience Requirements Development
External Dependencies
Resilience Requirements Mgmt.
Financial Resource Mgmt.
Resilient Technical Solution Engr.
Human Resource Management
Risk Management
Identity Management
Service Continuity
Incident Management & Control
Technology Management
Knowledge & Information Mgmt.
Vulnerability Analysis & Resolution
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
CMM Example: CERT-RMM
Consider the Incident Management and Control (IMC)
domain from CERT-RMM:
•  Goal 1: Establish the IMC process
•  Goal 2: Detect events
•  Goal 3: Declare incidents
•  Goal 4: Respond to and recover from incidents
•  Goal 5: Establish incident learning
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
CMM Example: CERT-RMM
Level 3 Level 0 Level 1 Level 2 Incomplete Performed Managed Defined “We don’t do (all of) the prac6ces.” “We do the prac6ces.” “We do the prac6ces AND we plan and govern the process, resource it, train people to do it, monitor it, etc…” We do everything in level 2 AND we have a defined process and collect improvement informa6on.” Ins,tu,onaliza,on is cumula,ve . . . . . CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Benefits and Limitations of CMMs
Benefits
v  Provides for measurement
of core competencies
v  Provides for rigorous
measurement of capability
—the ability to retain core
competencies under times
of stress
v  Can provide a path to
quantitative measurement
Limitations
v  Sometimes difficult to
understand and apply; high
adoption cost
v  “Maturity” may not translate into
actual results
v  Potential false sense of
achievement: achieving high
maturity in security practices
may not mean the organization
is “secure”
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
•  Run •  Defined Level 2 Level 2 •  Jog Level 1 •  Walk Level 0 •  Crawl Progression Model Distribu,on of core prac,ces across levels Level 3 Core prac6ces Level 3 •  Managed Level 1 •  Performed Level 0 •  Incomplete Distribu,on of ins,tu,onalizing features Compare: Progression vs CMM
Core prac6ces Capability Model CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Hybrid Maturity Model
Combines the best features of progression and capability maturity models
•  Allows for measurement of evolution or achievement as in progression models
•  Adds the ability to measure capability or institutionalization with the rigor
of a CMM
Levels reflect both achievement and capability
Transitions between levels:
•  Similar to a capability model (i.e., describe capability maturity)
•  Architecturally use the characteristics, indicators,
attributes, or patterns of a progression model
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Hybrid Maturity Models
Capability or “maturity” levels Domain 1
Domain 2
Domains: Specific categories of aZributes, characteris6cs, paZerns, or prac6ces that form the content of the model Domain 3
Domain 4
Domain n
Level 4
Defined
Level 3
Model content: Specific aZributes, characteris6cs, paZerns, or prac6ces that represent progression and capability Measured
Level 2
Managed
Level 1
Planned
Level 0
Incomplete
Maturity levels: Defined sets of characteris6cs and outcomes, plus capability considera5ons CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Hybrid Model Example: ES-C2M2
Electricity Subsector Cybersecurity Capability Maturity Model (ES-­‐C2M2) CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Benefits and Limitations of Hybrid Models
Benefits
v  Provides for easy
measurement of core
competencies as well as
approximation of capability
v  Can adapt easily to
evolution of technologies
and practices without
sacrificing capability
measurement
Limitations
v  “Maturity” concept is
approximated; not as rigorous
as CMM
v  Combination of attributes with
institutionalizing features at
each level can be arbitrary
v  Low adoption cost
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Closing Thoughts
•
A few cautions
•
Determining when and which type to use
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
First and Foremost
Have a clear understanding of your business objectives for using any type of
improvement model
•  How the model will meet these objectives
Understand how this initiative fits with others that are mainstream for the
organization (not a new add-on)
Have visible sponsorship of executives and senior leaders who are essential for
success
Have well-defined outcome measures that are regularly reported and reviewed
Have a plan and committed resources
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
A Few Cautions
Progression models may be easier to adopt but may not be sustainable
(aka sticky)
Definitions of levels can be arbitrary
Measuring process performance and maturity is useful but may not be
sufficient
Exercise care when using maturity models for specific purposes
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Progression Models May Not Be Sustainable
A progression model provides a roadmap or scale of a particular
characteristic, indicator, attribute, pattern, or practice
•  Focuses on practices or controls and their progression from least mature to
most mature
•  Cannot be used to measure the extent to which an organization is capable
of sustaining the practice in times of disruption and stress (the practice has not
become part of the DNA)
A hybrid or capability maturity model adds the dimension of
organizational capability to practice progression
•  Thus able to measure an organization’s “resilience” in the presence of
disruption and stress
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Definitions of Levels Can Be Arbitrary
Often defined by consensus of subject matter experts
Can simply reflect a plateau or a place in a progression or scale
Often have not been validated or are difficult to validate based on
experience and measurement
May neglect to represent the capability and capacity of an organization
to sustain operations in the presence of disruption and stress
40
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Measuring Process Performance May Not Be
Sufficient
Experience demonstrates that the quality of the process directly affects
the quality of the product
•  However, process performance and maturity are only one aspect
Also need to consider the performance and maturity of
•  The product and its outcomes
•  The supporting technologies
•  The environment within which the product operates
•  Knowledge, skills, and abilities of people with respect to all of these
•  Which of these dimensions to emphasize given product objectives
41
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
When Does It Make Sense to Use Maturity Models?
Requirement for a structured approach
Demonstrated, measurable results based on an established body of
knowledge
A defined roadmap from a current state to a desired state
An ability to monitor and measure progress, particularly in the
presence of change
•  Response to a strategic improvement or new product/new market objective
42
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
When Does It Make Sense to Use Maturity Models?
(cont.)
Desire to answer these questions in a repeatable, predictable manner:
•  How do I compare with my peers? (ability to benchmark)
•  How can I determine how secure I am and if I am secure enough?
•  How do I measure my current state? Characterize my desired state?
•  What concrete actions do I need to take to improve? And in what
order?
•  How do I measure progress toward my desired state?
•  How do I adapt to change?
43
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
Exercise Care When Using Maturity Models
If the immediate need is to respond to an in-progress disruptive event
—  Robust
processes are not yet in place
—  Current
protection and defensive mechanisms are failing
—  Need
to stop the bleeding, stabilize operations, rely on experts
In response to current and new compliance requirements
—  In
a highly regulated industry
—  Must
demonstrate compliance with specific laws, regulations and standard(s)
—  Standard,
defined processes and mapping new compliance requirements to
these can be quite effective
44
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
References
Crosby, P. B. (1979). Quality is Free. New York: New American Library. ISBN 0-451-62247-2.
Nolan, R. L. (July 1973). "Managing the computer resource: A stage hypothesis". Comm. ACM 16 (7): 399–405. doi:10.1145/362280.362284
Humphrey, W. S. (March 1988). "Characterizing the software process: A maturity framework". IEEE Software 5 (2): 73–79. doi:10.119/52.2014
Humphrey, W. S. (1989). Managing the Software Process. SEI series in software engineering. Reading, Mass.: Addison-Wesley. ISBN
0-201-18095-2
Paulk, Mark C.; Weber, Charles V.; & Curtis, Bill. The Capability Maturity Model: Guidelines for Improving the Software Process. AddisonWesley Professional, 1994.
CMMI Overview. http://www.sei.cmu.edu/cmmi/ (2012).
Chris McClean and Khalid Kark, “Introducing the Forrester Information Security Maturity Model,” Forrester Research , July 27, 2010.
Caralli, Richard A.; Allen, Julia H.; & White, David W. CERT® Resilience Management Model: A Maturity Model for Managing Operational
Resilience. Addison-Wesley, 2011.
Caralli, Richard A. Discerning the Intent of Maturity Models from Characterizations of Security Posture. Software Engineering Institute,
Carnegie Mellon University, 2012. http://resources.sei.cmu.edu/library/asset-view.cfm?assetID=58922
Electricity Subsector Cybersecurity Maturity Model. Carnegie Mellon University, 2012. http://energy.gov/sites/prod/files/Electricity
%20Subsector%20Cybersecurity%20Capabilities%20Maturity%20Model%20%28ES-C2M2%29%20-%20May%202012.pdf
Smart Grid: Tools and Methods. http://www.sei.cmu.edu/smartgrid/tools/ (2012).
Resilience Management. http://www.cert.org/resilience (2012).
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University
As projects continue to grow in scale and complexity, effective collaboration across geographical, cultural, and technical boundaries is increasingly
prevalent and essential to system success. SATURN 2012 will explore the theme of “Architecture: Catalyst for Collaboration.”
Introduction to the CERT Resilience Management Model
February 18 - 20, 2014 (SEI, Arlington, VA)
June 17 - 19, 2014 (SEI, Pittsburgh, PA)
See Materials Widget for course document
CERT® Operational Resilience:
Manage, Protect, and Sustain
Twitter #CERTopRES
© 2013 Carnegie Mellon University