Expert COSYSMO Update
Raymond Madachy
rjmadach@nps.edu
USC-CSSE Annual Research Review
March 17, 2009
Expert COSYSMO Introduction
• An expert system tool for systems engineering risk management
based on the Constructive Systems Engineering Cost Model
(COSYSMO) [Valerdi 2005]
– Automatically identifies project risks in conjunction with cost estimation similar to Expert
COCOMO [Madachy 1997] and provides related advice
– Supports project planning by identifying, categorizing and quantifying system-level risks
– Supports project execution with automated risk mitigation advice for management
consideration
• Risk situations are characterized by combinations of cost driver
values indicating increased effort with a potential for more problems
–
Includes 98 risk conditions
• Simultaneously calculates cost to enable tradeoffs with risk
https://diana.nps.edu/MSAcq/tools/ExpertCOSYSMO.php
http://csse.usc.edu/tools/ExpertCOSYSMO.php
2
Method
• Analyzes patterns of cost driver ratings
submitted for a COSYSMO cost estimate
against pre-determined risk rules
– Identifies individual risks that an experienced
systems engineering manager might recognize
but often fails to take into account
– Helps users determine and rank sources of
project risk. With these risks, mitigation plans
are created based on the relative risk severities
and provided advice
3
Method (cont.)
• COSYSMO cost factor combinations used as abstractions for
formulating risk heuristics
– E.g. if Architecture Understanding = Very Low and Level of Service
Requirements = Very High, then there is a high risk
• Since systems with high service requirements are more difficult to
implement especially when the architecture is not well understood
• Elicitation of knowledge from systems engineering domain
experts in CSSE-sponsored workshops
– Survey used to identify and quantify risks
• Devised knowledge representation scheme and risk
quantification algorithm
• All risk rules are fired when the effort multipliers of both cost
factors involved are > 1.0
– Can be more granular for gradations of risk
• Recent extension for risk mitigation advice
4
ARCH
LSVC
MIGR
TRSK
DOCU
INST
RECU
TEAM
PCAP
PEXP
PROC
SITE
TOOL
SIZE (REQ + INTF + ALG + OPSC)
Requirements Understanding
Architecture Understanding
Level of Service Requirements (the ilities)
Migration Complexity (legacy system considerations)
Technology Risk (maturity of technology)
Documentation match to life cycle needs
Number and Diversity of Installations or Platforms
Number of Recursive Levels in the Design
Stakeholder Team Cohesion
Personnel/team capability
Personnel Experience and Continuity
Process Capability
Multisite Coordination
Tool Support
RQMT
SIZE
Risk Conditions
21
21
17
9
9
9
12
7
10
5
5
8
12
7
8
4
3
3
4
1
2
7
5
7
5
10
8
2
10
9
11
3
1
6
3
4
8
5
6
6
4
4
4
3
4
9
10
11
4
7
9
4
5
8
7
11
8
11
4
7
5
2
6
7
9
12
7
5
5
2
3
3
6
4
7
3
9
10
6
4
6
3
5
3
2
8
2
8
8
8
5
7
1
4
2
4
5
3
5
5
3
5
3
8
8
high risk
small x = 0.5; big X = 1
medium risk
n = 19
low risk
5
Risk Taxonomy and Weighting
Project Risk
Product risk
Process risk
Personnel risk
Platform risk
#categories #category risks
Project Risk=
risk leveli , j * effort multiplier p roducti , j
j= 1
i= 1

where risk level =
1
2
4

moderate
high
very high
effort multiplier product=
(driver #1 effort multiplier) * (driver #2 effort multiplier) ... * (driver #n effort multiplier).
6
Expert COSYSMO Inputs
7
Expert COSYSMO Outputs
8
New Risk Mitigation Advice
• All risks triggered when both related effort multipliers are > 1.0
• See handout for review
9
Size Risk Elaboration
• Need to establish size range thresholds for risk rules
• COSYSMO size
distribution:
• Min = 82, Max = 17,763 equivalent requirements
• Proposed ranges
– Small:
– Medium:
– Large:
< 5,000 equivalent requirements
between 5,000 and 15,000 equivalent requirements
> 15,000 equivalent requirements
• See handout on size related risks
10
Finer Assignment of Risk Levels
• Different severity risks would entail contours of different strength
• Some risk combinations may require re-interpretation of the nominal boundaries
Current and Future Work
•
•
•
•
•
Scaling the risk summary outputs for each category and defining ranges for
low, medium and high risks
Create more granular risk quantification rules
Consider 3-way risk interactions
Add rules to detect COSYSMO input anomalies
Systems engineering risk data from industrial projects will be analyzed to
enhance and refine the technique
– Perform statistical testing
•
Domain experts from industry and government will continue to provide
feedback and clarification
– Supporting surveys and workshops will be continued
•
Integrating alternate risk uncertainty approaches into a more complete risk
management framework
– Recently added Monte-Carlo analysis
12
Tasks
So Far
• COSYSMO workshop experts have identified
and prioritized risks, and provided advice
• Researchers devised the risk taxonomy and its
weighting scheme
Today
• We invite your comments on all
• Help us review and complete the advice
13
Assignments
•
•
•
•
Dan/Beth pp. 1-2
Jared/Stan p. 3
Mauricio/Garry p. 4
Miles/Ricardo p. 5 and size risks
14
References
• R. Madachy, Heuristic Risk Assessment Using
Cost Factors, IEEE Software, May 1997
• Valerdi R., The Constructive Systems
Engineering Cost Model (COSYSMO), PhD
Dissertation, University of Southern California,
Los Angeles, CA, May 2005
• http://csse.usc.edu/tools/ExpertCOSYSMO.php
15