COCOTS Risk Analyzer and
Process Usage
Ye Yang, Barry Boehm
Center for Software Engineering
University of Southern California
Annual Research Review
Mar. 14th, 2006
3/14/2006
USC-CSE
1
Outline
•
•
•
•
•
•
Motivation
COCOTS Model
COCOTS Risk Analyzer
Evaluation
Process Usage: Risk-Based Prioritization
Conclusions
3/14/2006
USC-CSE
2
Motivation
• Enable COTS integration risk analysis with
COCOTS cost estimation inputs
• Identify relative risk levels of COTS-based
development (CBD)
• Provide recommendations to improve risk
management practices
3/14/2006
USC-CSE
3
COCOTS Model
- Calibrated to 20 industry projects
3/14/2006
USC-CSE
4
COCOTS Glue Code Sub-model
Cost
Name
Definition
Factors
Size Driver Glue Code Size The total amount of COTS glue code developed for the
system.
AAREN
Application Architectural Engineering
Scale
Factor
ACIEP
COTS Integrator Experience with Product
ACIPC
COTS Integrator Personnel Capability
Effort
Integrator Experience with COTS Integration Processes
Multiplier AXCIP
APCON
Integrator Personnel Continuity
ACPMT
COTS Product Maturity
ACSEW
COTS Supplier Product Extension Willingness
APCPX
COTS Product Interface Complexity
ACPPS
COTS Supplier Product Support
ACPTD
COTS Supplier Provided Training and Documentation
ACREL
Constraints on Application System/Subsystem Reliability
AACPX
Application Interface Complexity
ACPER
Constraints on COTS Technical Performance
ASPRT
Application System Portability
3/14/2006
USC-CSE
5
COCOTS Risk Analyzer
Input (Cost
Factor Ratings)
1. Identify risks
of rating
combinations
User
Output
(Risk Summary)
Mitigation
Strategy
6. Provide Risk
5.
Mitigation
Advices
Risk Rules
Knowledge
Knowledge
Base
Risk Rules
Base
5. Assess
4.
Overall Risk
Risk Level
Scheme
2. Evaluate Risk
Probability
3. Analyze Risk
Severity
3/14/2006
USC-CSE
6
Knowledge Base
• Contents
– Risk Rules (RR)
– Risk level scheme
– Common risk mitigation strategy
• Constructing approach
– Expert Delphi Survey
– Empirical study results
– Literature review
3/14/2006
USC-CSE
7
Risk Rule
• A CBD risk situation
– a combination of two cost attributes at their
extreme ratings
• Risk Rule (RR)
– An identified risk situation is formulated as a risk
rule. E.g. one example RR:
IF ((COTS Product Complexity > Nominal)
AND (Integrator’s Experience on COTS Product < Nominal))
THEN there is a project risk.
3/14/2006
USC-CSE
8
SIZE
AAREN
ACIEP
ACIPC
AXCIP
APCON
ACPMT
ACSEW
APCPX
ACPPS
ACPTD
ACREL
AACPX
ACPER
ASPRT
Risk Situation Identification
SIZE
AAREN
ACIEP
ACIPC
AXCIP
APCON
ACPMT
ACSEW
APCPX
ACPPS
ACPTD
ACREL
AACPX
ACPER
ASPRT
USC-CSE
# of
responses
% of
responses
# of risk
situations
>=3
>50%
24
2
40%
26
1
20%
28
24 Risk Rules
formulated in the
knowledge base
SIZE
AAREN
ACIEP
ACIPC
AXCIP
APCON
ACPMT
ACSEW
APCPX
ACPPS
ACPTD
ACREL
AACPX
ACPER
Total # of Delphi responses: 5
ASPRT
3/14/2006
>=50%
40%
20%
(Percentage of responses over total)
9
Risk Potential Rating for Cost Factors
Mapping between cost factor’s rating to
its risk potential rating:
Cost Factors
AAREN, ACIEP,
ACIPC, AXCIP,
APCON, ACPMT,
ACSEW, ACPPS,
ACPTD
APCPX, ACREL,
AACPX, ACPER,
ASPRT
3/14/2006
Cost Factor Rating
Very Low
Low
Nominal
High
Very High
Very Low
Low
Nominal
High
Very High
USC-CSE
Risk Probability Rating
Worst Case
Risk Prone
Moderate
OK
OK
OK
OK
Moderate
Risk Prone
Worst Case
10
Risk Level Scheme
Assignment of risk probability levels:
Attribute 1
Worst Case Risk Prone Moderate OK
Worst Case Severe
Significant General
Attribute 2 Risk Prone Significant General
Moderate
General
OK
Quantitative weighting scheme:
3/14/2006
Risk level
Quantifier
Severe
0.4
Significant
0.2
General
0.1
USC-CSE
11
Productivity Range
• Reflects the cost consequence
of risk occurring
• Combines both expert judgment
and industry data calibration
2.58
ACIPC
2.51
Cost Factor
APCON
ACPMT
2.10
AAREN
2.09
APCPX
1.80
ACIEP
1.79
1.69
AACPX
ACPPS
1.48
ACREL
1.48
ACPTD
1.43
AXCIP
1.42
ACPER
1.22
ACSEW
1.22
ASPRT
0.00
1.14
0.50
1.00
1.50
2.00
2.50
3.00
Productivity Range
3/14/2006
USC-CSE
12
Project Risk Quantification
• Project Overall Risk:
–
–
Riskprobij corresponds to the nonlinear relative probability of the risk occurring
The product of PRi and PR j represents the cost consequence of the risk occurring
• Risk interpretation:
– Normalized scale: 0 ~ 100
– 100 represents the situation where each cost factor is
rated at its most expensive extremity
– 0 ~ 5: low risk; 5 ~ 15: medium risk; 15 ~ 50: high risk; 50
~ 100: very high risk
3/14/2006
USC-CSE
13
Risk Mitigation Recommendations
•
Knowledge base built on previous empirical
study results, e.g.:
3/14/2006
Risk Rule
Risk Situation
APCPX_ACIPC
(High, Very Low)
Complex integration with inexperienced
personnel
Consider more compatible
COTS; re-staffing; training;
consultant mentoring
ACREL_ACPMT
(High, Low)
High-reliability application dependent on
immature COTS
Consider more mature
COTS; reliability-enhancing
COTS wrappers; risk-based
testing
ACPER_AAREN
(High, Very Low)
Unvalidated architecture with COTS
performance shortfalls
Benchmark current and
alternative COTS choices;
reassess performance
requirements vs.
achievables
USC-CSE
Mitigation Advice
14
Evaluation Results
50
45
45
40
y = 0.6749x - 2.3975
R2 = 0.8948
40
y = 45.75x + 0.6143
R2 = 0.6283
35
30
Analyzed Risk
Analyzed Risks
35
25
20
15
30
25
20
15
10
10
5
5
0
0
0
10
20
30
40
50
60
0
0.1
0.2
0.3
Reported Risks
Data: 9 USC e-services projects
0.5
0.6
0.7
0.8
0.9
Data: 7 COCOTS calibration projects
USC e-services
Web-based campus-wide
e-services applications
Domain such as library services
# COTS 1 ~ 6
Duration 24 weeks
6 person by 24 weeks
Effort
0.2 ~ 10 KSLOC
Size
3/14/2006
0.4
Reported Prob.(Risk)
Industry
Generally large scale
comminication, control
systems
1 ~ 53
1 ~ 56 months
1 ~ 1411 person-month
0.1 ~ 390 KSLOC
USC-CSE
15
Process Usage – An Example
• COTS A and B are our strongest COTS
choices
– But there is some chance that they have
incompatible HCI’s
– Probability of loss P(L)
• COTS C is almost as good as B, and it is
compatible with A
3/14/2006
USC-CSE
16
Risk-Driven CBD Process Framework
Start
C
P1: Identify Objective,
Constraints and
Priorities (OC&Ps)
P7: Custom Development
Yes
No
Process
Area
P6: Can adjust
OC&Ps?
P2: Do Relevant COTS
Products Exist?
No
Yes or Unsure
No acceptable or risky
COTS-Based Solution
P3: Assess COTS
Candidates
Partial COTS solution best
A
Decision
/Review
P5: Multiple COTS
cover all OC&Ps?
No, Custom code
Required to satisfy
all OC&Ps
Single Full-COTS solution
satisfies all OC&Ps
P8: Coordinate
custom code and glue
code development
P4: Tailoring Required?
Deploy
A
Assessment
T
Tailoring
G
GlueCode
C
Custom
code
Yes
G
C
Yes
No
No
P9: Develop Custom
Code
P10: Develop
Glue Code
T
P11: Tailor COTS
3/14/2006
P12: Productize,
Test and Transition
USC-CSE
Deploy
17
Different Risk Strategy Resulting in
Different Process
(a) Risk Avoidance:
COTS C adequate
Choose
COTS C
(b) Risk Transfer:
COTS C not
adequate
Choose
COTS B
Integrate
COTS A, C
Develop
Application,
Integrate A & B
Problem
(c) Risk Reduction:
Custom $, IP
(d) Risk Acceptance:
Developer $, IP
3/14/2006
Develop
Application
Deliver
OK
Develop
Application
Deliver
Use risk reserve
to fix problem
Develop rest
of application
Choose
COTS B
Develop parts of
application, use
wrappers to
integrate A and B
USC-CSE
Deliver
Package
wrappers for
future use
18
Conclusions
• CBD brings a host of unique risk items
• Many risk techniques/tools require intensive user
inputs
• COCOTS Risk Analyzer provides a handy way to
automate the CBD risk analysis by leveraging on
existing knowledge and expertise in both cost
estimation and risk mgmt.
• Case study shows how it supports process decisions
following the risk based prioritization strategy
3/14/2006
USC-CSE
19
Backup Slides
3/14/2006
USC-CSE
20
Risk Potential Rating
• Captures the underlying relation between
cost attributes and the impact of their specific
ratings on project risk
– 4 Levels
• OK, Moderate, Risk Prone, and Worst Case
• Two types of treatments
– Transforming continuous Size representation into
discrete risk potential ratings
– Mapping cost driver ratings into risk potential
ratings
3/14/2006
USC-CSE
21
Risk Potential Rating for Size
Delphi Responses for Size Rating (Size in KSLOC):
Rating
Response 1
Response 2
Response 3
Response 4
Response 5
Median
Stdev
3/14/2006
OK
Moderate Risk Prone Worse Case
1
2
10
50
2
5
10
25
1
3
10
10
1
2
10
50
1
2
10
50
1
2
10
50
0.447214 1.30384
0
18.5741756
USC-CSE
22
Risk Based Prioritization Strategy
Risk
Spiral
CBD process
Description
Strategy Quadrants
Decision
Step
Framework Step
S1
Q1
P1, P2
Identify OC&Ps, COTS/other alternatives
S2
Q2a
P3
Evaluate COTS vs. OC&Ps (incl.
COCOTS)
S3
Q2a
P3
Identify risks, incl. COCOTS risk analysis
S4
Q2b
P3
Assess risks, resolution alternatives; If
risks manageable, go to S7
S5
Q2b, Q1
P6
Negotiate OC&P adjustments; If none
acceptable, drop COTS options (P7)
S6
Q2a
P3
If OC&P adjustments successful, go to
S7; If not, go to S5
S7
Q3
P4 or P5
Execute acceptable solution
3/14/2006
USC-CSE
23