New SEI Course!
Practical Risk Management:
Framework and Methods
September 23-24, 2009
Arlington, VA
Register at:
www.sei.cmu.edu/products/courses/p78.html
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
1
13th International Software Product Line
Conference 2009 (SPLC)
http://www.sei.cmu.edu/splc2009/index.html
Organizations Need
Software Product Lines
Now More Than Ever!
Effectively using software product lines
improves time to market, cost,
productivity, and quality. They also
enable rapid market entry and flexible
response. And, using software product
lines simplifies software maintenance
and enhancement.
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
2
Research, Technology, and System Solutions Program:
Working with the SEI
If you need to improve …
The SEI can…
the structure and behavior of your
software-reliant systems (regardless of scale)
your ability to predict that behavior
harness the appropriate technology to
help you solve specific problems
help you launch initiatives
help you improve your capabilities
conduct applied research that meets
your needs
partner with you to create leading
edge techniques, methods, and tools
For more information contact info@sei.cmu.edu
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
3
CERT's Podcast Series:
Security for Business Leaders.
http://www.cert.org/podcast/
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
4
SEPG Conference Series
SEPG is the premier, global conference
series on software and systems process
management
http://www.sei.cmu.edu/sepg/index.html
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
5
Get Certified!
SEI Certifications:
Proof of your skill from a world leader in
software engineering.
http://www.sei.cmu.edu/certification/
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
6
Want a Closer Connection to the SEI?
Become an SEI Member!
http://www.sei.cmu.edu/membership/
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
7
Do you have the knowledge you need?
SEI Education & Training
http://www.sei.cmu.edu/products/courses/
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
8
A Practical Approach for
Managing Risk
Christopher Alberts
Audrey Dorofee
June 18, 2009
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
© 2009 Carnegie Mellon University
Biography: Christopher Alberts
Christopher Alberts is a senior member of the technical
staff at the Software Engineering Institute. He is currently
developing methods for managing systemic risk during
the development and operation of software-intensive
systems and systems of systems. Prior to his work in this
area, he co-developed the OCTAVE® approach for
managing information security risks and the Continuous
Risk Management methodology for managing software
development project risks. He has also co-authored two
books, “Managing Information Security Risks: The
OCTAVESM Approach” (Addison-Wesley 2002) and the
“Continuous Risk Management Guidebook” (Software
Engineering Institute 1996).
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
10
Biography: Audrey Dorofee
Audrey Dorofee is a senior member of the technical
staff at the Software Engineering Institute. She is
currently focused on the development and transition of
advanced methods, tools and techniques for
managing risk and opportunity in complex
environments. She has co-authored two books,
Managing Information Security Risks: The OCTAVESM
Approach (Addison-Wesley 2002) and the Continuous
Risk Management Guidebook (Software Engineering
Institute 1996).
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
11
Polling Question #1
Are you experienced in managing risk?
Answers:
• Yes – experienced in managing risks
• No – new to risk management
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
12
Mission Success in Complex Environments
(MSCE) Project
Part of the SEI Acquisition Support Program (ASP), the MSCE Project
develops methods, tools, and techniques for
• Advancing the state-of-the-practice for risk management
• Assuring success in complex, uncertain environments
The project builds on more than 17 years of SEI research and
development in risk management.
• Continuous Risk Management for software-development projects
• Operationally Critical Threat, Asset, and Vulnerability Evaluation
(OCTAVE®) for organizational security
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
13
Topics
Mosaic Approach
Driver Analysis
Standard Set of Program Drivers
Risk Management Framework
Implementing Mosaic
Summary
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
14
Mosaic Approach
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
15
Widespread Use of Risk Management
Most programs and organizations implement some type of risk
management approach when developing and operating softwareintensive systems.
• Risk management plan
• Processes
• Tools
However, preventable failures continue to occur.
• Uneven and inconsistent application of risk-management practice
• Significant gaps in risk-management practice
• Ineffective integration of risk-management practice
• Increasingly complex management environment
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
16
Rethinking Risk Management: A New Paradigm
Traditional Paradigm
New Paradigm
Managing potential hazards
Achieving success
Tactical approach
Systemic approach
Point solutions
Integrated, holistic solutions
• Single type of risk (e.g., program,
security, architecture)
• Single life-cycle phase
• Single entity (e.g., program, process,
organization, system)
• Multiple types of risk
• Applicable across the life cycle
• Scalable to multi-enterprise, multisystem environments
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
17
Tactical and Systemic Approaches
Potential
Event
Consequence
Condition
Condition
Potential
Event
Consequence
Condition
Potential
Event
Condition
Potential
Event
Impact on
Objectives
Consequence
Condition
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
18
Mosaic
What
An approach for managing risk and opportunity across the life cycle and
supply chain
Core Technologies
Assessment Methods
Risk Management Framework
Products and Services
Courses
Workshops
Course and Workshop Combinations
Evaluations
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
19
Mosaic: Focus on Assessment
Every organization has
preferred management practices
Mosaic also provides guidance
for leveraging existing management
practices to develop, implement,
and track risk mitigation plans
Organizational
Management
Practices
Do
The foundation of the Mosaic
approach is a suite of methods
for assessing risk continuously
Act
Plan
Check
Mosaic Management
Guidance
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
20
Other Types of Analysis
Risk Simulation Models
Mission Assurance
Analysis
Integrated Risk and
Opportunity Analysis
Mission Success
Analysis
Intermediate Risk Analysis
Driver identification and analysis
provide a common front end
for multiple back-end analyses
Basic Risk Analysis
Mosaic assessments are
modular in design
Gap Analysis
Mosaic Assessments
Driver Analysis
Driver Identification
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
21
Mosaic: A Range of Analysis Options
Basic Analysis
Gap Basic Risk Intermediate
Analysis Analysis Risk Analysis
Advanced Analysis
Mission Success
Analysis
Integrated Risk and
Opportunity Analysis
Mission
Assurance
Analysis
Risk
Simulation
Models
Mosaic analysis methods range from basic to advanced.
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
22
Driver Analysis
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
23
Mosaic: Driver-Based Assessment
A driver is a factor that has a strong influence on the eventual
outcome or result.
Key Objectives
Driver 1
Driver 2
Positive Conditions and
Potential Events
Driver 3
…
Driver N
Negative Conditions and
Potential Events
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
24
Driver Framework
Driver Categories
Objectives
Preparation
Execution
Environment
Resilience
Result
The driver framework is a common structure for classifying a
set of drivers.
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
25
Drivers: Success and Failure States
The process being used
to develop (and deploy)
the system is sufficient.
Success State
The process being used
to develop (and deploy)
the system is insufficient.
Failure State
Process
A driver can guide the outcome toward key objectives
(success state) or away from them (failure state).
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
26
Mosaic: Integrating Multiple Types of Risk
Driver Categories
Objectives
Preparation
Execution
Environment
Resilience
Result
Process risk
IT risk
Product risk
Programmatic interoperability risk
Security risk
Requirements risk
Operational risk
Architecture risk
Mosaic provides an integrated view of
the overall risk to key objectives.
System integration risk
System survivability risk
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
27
Basic Set of Drivers for Software Programs
1.
Program Objectives
11. Compliance
2.
Plan
12. Event Management
3.
Process
13. Requirements
4.
Task Execution
14. Design and Architecture
5.
Coordination
15. System Capability
6.
External Interfaces
16. System Integration
7.
Information Management
17. Operational Support
8.
Technology
18. Adoption Barriers
9.
Facilities and Equipment
19. Operational Preparedness
10. Organizational Conditions
20. Certification and Accreditation
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
28
Driver Analysis
Question
3. Is the process being used to develop and
deploy the system sufficient?
Answer
No
Likely
no
Equally
likely
Likely
yes
Yes
Don’t
Know


X




Consider: Process design; measurements and
controls; process efficiency and effectiveness;
acquisition and development life cycles; training
Driver questions are phrased from the success perspective.
Probability is incorporated into the range of answers for each driver.
The rationale for selecting an answer is recorded.
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
29
© 2009 Carnegie Mellon University
20. Certification & Accreditation
No
19. Operational Preparedness
Likely No
18. Adoption Barriers
Equally
Likely
17. Operational Support
Likely Yes
16. System Integration
Likely Yes
15. System Capability
Programmatic Drivers
14. Design & Architecture
Yes
Driver Value
Yes
13. Requirements
12. Event Management
11. Compliance
10. Organizational Conditions
9. Facilities & Equipment
8. Technology
7. Information Management
6. External Interfaces
5. Coordination
4. Task Execution
3. Process
2. Plan
1. Program Objectives
Driver Value
Driver Profile
Product Drivers
Equally
Likely
Likely No
No
A simple analysis provides insight into current conditions.
A Practical Approach for Managing Risk
30
Basic Risk Analysis: Mission Risk
Mission Risk
Probability
Impact
Risk
Exposure
3. The process being used to develop
and deploy the system is insufficient.
High
Severe
High
Determined using results of
driver analysis
Determined using standard
risk analysis methods
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
31
Risk Profile
Risk Profile
Objectives
High
1.
Execution
Program Objectives
Medium 4.
Low
5.
Resilience
Task Execution
2.
Plan
High
3.
Process
Event Management
Coordination
Minimal 6.
External Interfaces
Minimal 7.
Information Management
Minimal 8.
Technology
Minimal 9.
Facilities and Equipment
Preparation
Medium
Medium 12.
Product
Environment
High
10.
Organizational Conditions
Minimal
11.
Compliance
Low
13.
Medium 14.
Requirements
Design and Architecture
Low
15.
System Capability
High
16.
System Integration
Medium 17.
Operational Support
Medium 18.
Adoption Barriers
Medium 19.
Operational Preparedness
Medium 20.
Certification and Accreditation
Approach for Managing Risk
A risk profile can be presented in relation toA Practical
a framework
or
taxonomy.
© 2009 Carnegie Mellon University
32
Standard Set of Program
Drivers
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
33
Driver Questions: Objectives
1. Program Objectives
• Are program objectives (product, cost, schedule) realistic and
achievable?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
34
Driver Questions: Preparation
2. Plan
• Is the plan for developing (and deploying) the system sufficient?
3. Process
• Is the process being used to develop (and deploy) the system sufficient?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
35
Driver Questions: Execution -1
4. Task Execution
• Are tasks and activities performed effectively and efficiently?
5. Coordination
• Are activities within each team and across teams coordinated
appropriately?
6. External Interfaces
• Will work products from suppliers, partners, or collaborators meet the
program’s quality and timeliness requirements?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
36
Driver Questions: Execution -2
7. Information Management
• Is the program’s information managed appropriately?
8. Technology
• Does the program team have the tools and technologies it needs to
develop the system and transition it to operations?
9. Facilities and Equipment
• Are facilities and equipment sufficient to support the program?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
37
Driver Questions: Environment
10. Organizational Conditions
• Are enterprise, organizational, and political conditions facilitating
completion of program activities?
11. Compliance
• Does the program comply with all relevant policies, laws, and
regulations?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
38
Driver Questions: Resilience
12. Event Management
• Does the program have sufficient capacity and capability to identify and
manage potential events and changing circumstances?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
39
Driver Questions: Result -1
13. Requirements
• Are system requirements well understood?
14. Design and Architecture
• Are the design and architecture sufficient to meet system requirements
and provide the desired operational capability?
15. System Capability
• Will the system satisfactorily meet its requirements?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
40
Driver Questions: Result -2
16. System Integration
• Will the system sufficiently integrate and interoperate with other systems
when deployed?
17. Operational Support
• Will the system effectively support operations?
18. Adoption Barriers
• Have barriers to customer/user adoption of the system been managed
appropriately?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
41
Driver Questions: Result -3
19. Operational Preparedness
• Will people be prepared to operate, use, and maintain the system?
20. Certification and Accreditation
• Will the system be appropriately certified and accredited for operational
use?
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
42
Polling Question #2
Do you use a risk management method that addresses all 20 driver
questions?
Answers:
• Yes
• No
• Don’t know
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
43
Risk Management
Framework
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
44
Mosaic: Enabling Best Practice
Mosaic also provides guidance for determining if an existing risk
management practice is effective.
• The Risk Management Framework defines best practice for risk
management.
• Mosaic provides approaches for evaluating a program’s risk management
practice.
– Consistency Evaluation – establishes whether key framework
requirements are satisfied by a risk management practice
– Effectiveness Evaluation – establishes the likelihood that a risk
management practice will produce intended results (i.e., keep risk
within an acceptable tolerance)
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
45
Risk Management Framework -1
Phase 2
Perform Risk
Management Activities
Phase 3
Sustain and Improve
Risk Management
Activities
Assess
a te
tig
Mi
Pla
n
Phase 1
Prepare for
Risk Management
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
46
Risk Management Framework -2
The Risk Management Framework is implementation independent.
• Defines risk management activities
• Does not specify how to perform those activities
The framework provides a
• Foundation for a comprehensive risk management methodology
• Basis for improving a risk management practice
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
47
Polling Question #3
Is your current risk management practice effective?
Answers:
• Effective – all critical risks are being identified and mitigated; no
unexpected, critical problems
• Needs improvement – some critical problems are showing up that should
have been caught as risks
• Not very helpful – information not used by managers making decisions
• Just a check-the-box process because we have to do it
• Don’t know
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
48
Implementing Mosaic
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
49
Ways of Implementing Mosaic
Improve an existing risk management practice using the Risk
Management Framework
Adopt one of Mosaic’s assessment methods
• Select the appropriate assessment “platform” (basic to advanced)
• Tailor drivers and artifacts based on mission and objectives
Use Mosaic to integrate risk information in a multi-enterprise
environment
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
50
Mosaic: An Integrated Decision-Making Approach
Decision-Making Data
Back-End
Analysis
Systemic
View
Tactical
View
Driver
Analysis
Positive
Conditions
Strengths
Negative
Conditions
Weaknesses/
Issues
Potential Events
with Positive
Consequences
Tactical
Opportunities
Potential Events
with Negative
Consequences
Tactical
Risks
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
51
Extending Driver Analysis
Driver analysis provide a foundation for program decision making.
• Mission assurance analysis (Mission
Assurance Analysis Protocol – MAAP)
• Risk simulation models
Other Types of Analysis
Mission Assurance
Analysis
Risk Simulation Models
• Integrated risk and opportunity analysis
Integrated Risk and
Opportunity Analysis
• Mission success analysis
Mission Success
Analysis
• Intermediate risk analysis
Basic Risk Analysis
• Basic risk analysis (Risk Diagnostic)
Gap Analysis
• Gap analysis (Mission Diagnostic)
Intermediate Risk Analysis
Mosaic also includes a variety of back-end analyses for more in-depth
evaluation of drivers.
Driver Analysis
Driver Identification
• Others
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
52
Mosaic in Multi-Enterprise Environments
Programs that cross multiple organizational boundaries require a
systemic viewpoint when managing risk.
• Acquire and maintain abroad view of the risk to program objectives
• Avoid local optimization of risk
• Keep volume of risk data to a manageable level
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
53
Integrated View of Risk in Multi-Enterprise
Environments
SEI Mosaic
SEI Continuous Risk Management
SEI Mosaic
Proprietary Risk
Management
Proprietary Risk
Management
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
54
Summary
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
55
Mosaic Assessments: Key Characteristics
Straightforward and easy to apply
Comprehensive, holistic view of a program’s risk drivers
Fully scalable to multi-system and multi-enterprise environments
Easily integrated with existing management practices
Success oriented
Systemic, top-down analysis
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
56
Mosaic Assessments: Application in Multiple
Domains
Program risk management
Mission and software assurance
Information technology (IT) management
Data management
Cyber-security management
Business process management
Critical infrastructure protection
Others
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
57
Potential Areas of Future Research
Metrics
Risk-based improvement
Modeling and simulation
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
58
Mosaic Resources
SEI web pages
http://www.sei.cmu.edu/risk/
• Twenty Questions for Program Managers
• Presentations
• Technical Reports
– A Framework for Categorizing Key Drivers of Risk
– Mission Diagnostic Protocol, Version 1.0: A Risk-Based Approach for
Assessing the Potential for Success
– Preview of the Mission Assurance Analysis Protocol (MAAP):
Assessing Risk and Opportunity in Complex Environments
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
59
Mosaic: Portfolio -1
Courses
• Risk Management Framework: Best Practices in Risk Management
• Introduction to Practical Risk Management
• Practical Risk Management: Framework and Methods
Workshops
• Risk Management Tailoring and
Improvement Workshops
Course and Workshop Combinations
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
60
Mosaic: Portfolio -2
Evaluations
• Systemic Risk Evaluation
• Mission Success Evaluation
• Risk Management Framework Evaluation
• Custom Evaluation
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
61
Focus of Mosaic Products and Services
Basic Analysis
Advanced Analysis
Gap Basic Risk Intermediate
Analysis Analysis Risk Analysis
Mission Success
Analysis
Integrated Risk and
Opportunity Analysis
Courses
and
Workshops
Evaluations
Mission
Assurance
Analysis
Risk
Simulation
Models
Research and
Development
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
62
Public Training in September 2009
Practical Risk Management: Framework and Methods
• September 23-24, 2009
• SEI office in Arlington, VA
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
63
For Additional Information
Christopher Alberts
Email: cja@sei.cmu.edu
Phone: 412-268-3045
Fax: 412-268-5758
Audrey Dorofee
Email: ajd@sei.cmu.edu
Phone: 412-268-6396
Fax: 412-268-5758
WWW
http://www.sei.cmu.edu/risk/
U.S. mail
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213-3890
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
64
A Practical Approach for Managing Risk
© 2009 Carnegie Mellon University
65