University of Southern California
Center for Systems and Software Engineering
Using the Incremental Commitment
Model (ICM) to Help Execute
Competitive Prototyping (CP)
—Charts with Notes—
Barry Boehm and Jo Ann Lane
University of Southern California
Center for Systems and Software Engineering
http://csse.usc.edu
University of Southern California
Center for Systems and Software Engineering
Outline
• Motivation and Context
• Nature of the ICM
• Applying ICM Principles to CP
• Conclusions, References, Acronyms
– Copy of Young Memo
July 2008
©USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Motivation and Context
• DoD is emphasizing CP for system acquisition
– Young memo, September 2007
• CP can produce significant benefits, but also
has risks
– Benefits related to incremental commitment
– Examples of risks from experiences, workshops
• The risk-driven ICM can help address the risks
– Primarily through its underlying principles
July 2008
©USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
Young Memo: Prototyping and Competition
• Discover issues before costly SDD phase
– Producing detailed designs in SDD
– Not solving myriad technical issues
• Services and Agencies to produce competitive
prototypes through Milestone B
– Reduce technical risk, validate designs and cost estimates,
evaluate manufacturing processes, refine requirements
• Will reduce time to fielding
– And enhance govt.-industry teambuilding, SysE skills,
attractiveness to next generation of technologists
• Applies to all programs requiring USD(AT&L) approval
– Should be extended to appropriate programs below ACAT I
03/19/2008
©USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
Incremental Commitment in
Gambling
• Total Commitment: Roulette
– Put your chips on a number
• E.g., a value of a key performance parameter
– Wait and see if you win or lose
• Incremental Commitment: Poker, Blackjack
– Put some chips in
– See your cards, some of others’ cards
– Decide whether, how much to commit to
proceed
03/19/2008
©USC-CSSE
5
University of Southern California
Center for Systems and Software Engineering
Scalable remotely controlled
operations
03/19/2008
©USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Total vs. Incremental Commitment – 4:1 RPV
• Total Commitment
–
–
–
–
–
Agent technology demo and PR: Can do 4:1 for $1B
Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months
PDR: many outstanding risks, undefined interfaces
$800M, 40 months: “halfway” through integration and test
1:1 IOC after $3B, 80 months
• CP-based Incremental Commitment [number of competing teams]
–
–
–
–
–
$25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1
$75M, 8 mo. to ACR [3]: agent technology may do 1:1; some risks
$225M, 10 mo. to DCR [2]: validated architecture, high-risk elements
$675M, 18 mo. to IOC [1]: viable 1:1 capability
1:1 IOC after $1B, 42 months
03/19/2008
©USC-CSSE
7
University of Southern California
Center for Systems and Software Engineering
Example Risks Involved in CP
Based on TRW, DARPA, SAIC experiences; workshop
• Seductiveness of sunny-day demos
– Lack of coverage of rainy-day off-nominal scenarios
– Lack of off-ramps for infeasible outcomes
• Underemphasis on quality factor tradeoffs
– Scalability, performance, safety, security, adaptability
• Discontinuous support of developers, evaluators
– Loss of key team members
– Inadequate evaluation of competitors
• Underestimation of productization costs
– Brooks factor of 9 for software
– May be higher for hardware
• Underemphasis on non-prototype factors
July 2008
©USC-CSSE
8
University of Southern California
Center for Systems and Software Engineering
Milestone B Focus on Technology Maturity
Misses Many OSD/AT&L Systemic Root Causes
1 Technical process (35 instances)
6 Lack of appropriate staff (23)
- V&V, integration, modeling&sim.
2 Management process (31)
7 Ineffective organization (22)
3 Acquisition practices (26)
8 Ineffective communication (21)
4 Requirements process (25)
9 Program realism (21)
5 Competing priorities (23)
10 Contract structure (20)
•Some of these are root causes of technology immaturity
•Can address these via evidence-based Milestone B exit criteria
•Technology Development Strategy
•Capability Development Document
•Evidence of affordability, KPP satisfaction, program achievability
03/19/2008
©USC-CSSE
9
University of Southern California
Center for Systems and Software Engineering
Outline
• Motivation and Context
• Nature of the ICM
• Applying ICM Principles to CP
• Conclusions, References, Acronyms
– Copy of Young Memo
July 2008
©USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
What is the ICM?
• Risk-driven framework for tailoring system lifecycle processes
• Integrates the strengths of phased and risk-driven
spiral process models
• Synthesizes together principles critical to
successful system development
– Commitment and accountability of system sponsors
– Success-critical stakeholder satisficing
– Incremental growth of system definition and stakeholder
Principles
commitment
trump
diagrams…
– Concurrent engineering
– Iterative development cycles
– Risk-based activity levels and evidence-based milestones
Principles Used by 60-80% of CrossTalk Top-5 projects, 2002-2005
July 2008
©USC-CSSE
11
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Life Cycle Process: Overview
Stage I: Definition
Stage II: Development and Operations
Anchor Point
Milestones
Synchronize, stabilize concurrency via FEDs
Risk patterns
determine life cycle
process
03/19/2008
©USC-CSSE
12
University of Southern California
Center for Systems and Software Engineering
ICM HSI Levels of Activity for Complex Systems
03/19/2008
©USC-CSSE
13
University of Southern California
Center for Systems and Software Engineering
Anchor Point Feasibility Evidence Description
• Evidence provided by developer and validated by
independent experts that:
If the system is built to the specified architecture, it will
– Satisfy the requirements: capability, interfaces, level of service,
and evolution
– Support the operational concept
– Be buildable within the budgets and schedules in the plan
– Generate a viable return on investment
– Generate satisfactory outcomes for all of the success-critical
stakeholders
• All major risks resolved or covered by risk management
plans
• Serves as basis for stakeholders’ commitment to proceed
Can be used to strengthen current schedule- or event-based reviews
03/19/2008
©USC-CSSE
14
University of Southern California
Center for Systems and Software Engineering
ICM Nature and Origins
• Integrates hardware, software, and human factors
elements of systems engineering
– Concurrent exploration of needs and opportunities
– Concurrent engineering of hardware, software, human aspects
– Concurrency stabilized via anchor point milestones
• Developed in response to DoD-related issues
– Clarify “spiral development” usage in DoD Instruction 5000.2
• Initial phased version (2005)
– Explain Future Combat System of systems spiral usage to GAO
• Underlying process principles (2006)
– Provide framework for human-systems integration
• National Research Council report (2007)
• Integrates strengths of current process models
– But not their weaknesses
03/19/2008
©USC-CSSE
15
University of Southern California
Center for Systems and Software Engineering
ICM Integrates Strengths of Current Process Models
But not their weaknesses
• V-Model: Emphasis on early verification and validation
– But not ease of sequential, single-increment interpretation
• Spiral Model: Risk-driven activity prioritization
– But not lack of well-defined in-process milestones
• RUP and MBASE: Concurrent engineering stabilized by
anchor point milestones
– But not software orientation
• Lean Development: Emphasis on value-adding activities
– But not repeatable manufacturing orientation
• Agile Methods: Adaptability to unexpected change
– But not software orientation, lack of scalability
03/19/2008
©USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
Outline
• Motivation and Context
• Nature of the ICM
• Applying ICM Principles to CP
• Conclusions, References, Acronyms
– Copy of Young Memo
July 2008
©USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
Applying ICM Principles and Practices to CP
• When, what, and how much to prototype?
– Risk management principle: buying information to
reduce risk
• Whom to involve in CP?
– Satisficing principle: all success-critical stakeholders
• How to sequence CP?
– Incremental growth, iteration principles
• How to plan for CP?
– Concurrent engineering principle: more parallel effort
• What is needed at Milestone B besides
prototypes?
– Risk management principle: systemic analysis insights
July 2008
©USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
When, What, and How Much to Prototype?
 Buying information to reduce risk
• When and what: Expected value of perfect
information
• How much is enough: Simple statistical
decision theory
July 2008
©USC-CSSE
19
University of Southern California
Center for Systems and Software Engineering
When and What to Prototype: Early RPV Example
• Bold approach
0.5 probability of success: Value VBS = $100M
0.5 probability of failure: Value VBF = - $20M
• Conservative approach
Value VC = $20M
• Expected value with no information
EVNI = max(EVB, EVC) = max(.5($100M)+.5(-$20M), $20M)
= max($50M-$10M,$20M) = $40M
• Expected value with perfect information
EVPI = 0.5[max(VBS,VC)] + 0.5[max(VBF,VC)]
= 0.5 * max($100M,$20M) + 0.5 * max(-$20M,$20M)
= 0.5 * $100M + 0.5 * $20M = $60M
• Expected value of perfect information
EVPI = EVPI – EVNI = $20M
• Can spend up to $20M buying information to reduce risk
July 2008
©USC-CSSE
20
University of Southern California
Center for Systems and Software Engineering
If Risk Exposure is Low, CP Has Less Value
• Risk Exposure RE = Prob(Loss) * Size(Loss)
• Value of CP (EVPI) would be very small if the
Bold approach is less risky
– Prob(Loss) = Prob (VBF) is near zero rather than 0.5
– Size(Loss) = VBF is near $20M rather than -$20M
July 2008
©USC-CSSE
21
University of Southern California
Center for Systems and Software Engineering
How Much Prototyping is Enough?
 Value of imperfect information
•
Larger CP investments reduce the probability of
– False Negatives (FN): prototype fails, but approach would succeed
– False Positives (FP): prototype succeeds, but approach would fail
Can calculate EV(Prototype) from previous data plus P(FN), P(FP)
EV(CP)
EV(Info)
Net EV(CP)
P(FN)
P(FP)
CP Cost
0
$40M
0
0
8
6
Net EV (CP) ($M)
•
$2M
0.3
0.2
$46M
$6M
$4M
$5M
0.2
0.1
$52M
$12M
$7M
$10M
0.15
0.075
$54M
$14M
$4M
$15M
0.1
0.05
$56M
$16M
$1M
$22M
0.0
0.0
$60M
$20M
-$2M
•
4
2
0
-2
$0
$2
$5
$10
$15
$22
-4
-6
Cost (CP) in $M
Added CP decision criterion
– The prototype can cost-effectively reduce the uncertainty
July 2008
©USC-CSSE
22
University of Southern California
Center for Systems and Software Engineering
Summary: CP Pays Off When
•
The basic CP value propositions are satisfied
1. There is significant risk exposure in making the wrong
decision
2. The prototype can cost-effectively reduce the risk
exposure
•
There are net positive side effects
3. The CP process does not consume too much calendar
time
4. The prototypes have added value for teambuilding or
training
5. The prototypes can be used as part of the product
July 2008
©USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Applying ICM Principles and Practices to CP
• When, what, and how much to prototype?
– Risk management principle: buying information to
reduce risk
• Whom to involve in CP?
– Satisficing principle: all success-critical stakeholders
• How to sequence CP?
– Incremental growth, iteration principles
• How to plan for CP?
– Concurrent engineering principle: more parallel effort
• What is needed at Milestone B besides
prototypes?
– Risk management principle: systemic analysis insights
July 2008
©USC-CSSE
24
University of Southern California
Center for Systems and Software Engineering
Whom to Involve in CP?
 Satisficing principle: All success-critical stakeholders
• Success-critical: high risk of neglecting their
interests
–
–
–
–
Acquirers
Developers
Users
Testers
 Operators
 Maintainers
 Interoperators
 Others
• Risk-driven level of involvement
– Interoperators: initially high-level; increasing detail
• Need to have CRACK stakeholder participants
– Committed, Representative, Authorized, Collaborative,
Knowledgeable
July 2008
©USC-CSSE
25
University of Southern California
Center for Systems and Software Engineering
How to Sequence CP?
 Iterative cycles; incremental commitment principles
100%
Traditional Degree
Of Commitment
Incremental CP
Commitments
Traditional
Degree Of
Understanding
Blanchard-Fabrycky, 1998
July 2008
©USC-CSSE
26
University of Southern California
Center for Systems and Software Engineering
Actual CP Situation: Need to Conserve Momentum
• Need time to evaluate and rebaseline
• Eliminated competitors’ experience lost
100%
Degree of
Commitment
Need to keep
competitors
productive,
compensated
Degree of
Understanding
Need to capitalize
on lost experience
Proto-1
July 2008
Eval-1
Proto-2
Eval-2
©USC-CSSE
Proto-3 Eval-3
27
University of Southern California
Center for Systems and Software Engineering
Keeping Competitors Productive and Supported
During Evaluations
 Concurrent engineering principle
• Provide support for a core group within each
competitor organization
– Focused on supporting evaluation activities
– Avoiding loss of tacit knowledge and momentum
• Key evaluation support activities might include
– Supporting prototype exercises
– Answering questions about critical success factors
• Important to keep evaluation and selection period
as short as possible
– Through extensive preparation activities (see next chart)
July 2008
©USC-CSSE
28
University of Southern California
Center for Systems and Software Engineering
Keeping Acquirers Productive and
Supported During Prototyping
• Adjusting plans based on new information
• Preparing evaluation tools and testbeds
– Criteria, scenarios, experts, stakeholders, detailed
procedures
• Possibly assimilating downselected competitors
– IV&V contracts as consolation prizes
• Identifying, involving success-critical stakeholders
• Reviewing interim progress
• Pursuing complementary acquisition initiatives
– Operational concept definition, life cycle planning, external
interface negotiation, mission cost-effectiveness analysis
July 2008
©USC-CSSE
29
University of Southern California
Center for Systems and Software Engineering
Applying ICM Principles and Practices to CP
• When, what, and how much to prototype?
– Risk management principle: buying information to
reduce risk
• Whom to involve in CP?
– Satisficing principle: all success-critical stakeholders
• How to sequence CP?
– Incremental growth, iteration principles
• How to plan for CP?
– Concurrent engineering principle: more parallel effort
• What is needed at Milestone B besides
prototypes?
– Risk management principle: systemic analysis insights
July 2008
©USC-CSSE
30
University of Southern California
Center for Systems and Software Engineering
Later CP Rounds Need Increasing
Focus on Complementary Practices
 By all success critical stakeholders
• Stakeholder roles, responsibilities, authority,
accountability
• Capability priorities and sequencing of
development increments
• Concurrent engineering of requirements,
architecture, feasibility evidence
• Early preparation of development infrastructure
(i.e., key parts of the architecture)
• Acquisition planning, contracting, management,
staffing, test and evaluation
July 2008
©USC-CSSE
31
University of Southern California
Center for Systems and Software Engineering
When to Stop CP
 Commitment and accountability principle: Off-ramps
• Inadequate technology base
– Lack of evidence of scalability, security, accuracy,
robustness, airworthiness, useful lifetime, …
– Better to pursue as research, exploratory development
• Better alternative solutions emerge
– Commercial, other government
• Key success-critical stakeholders decommit
– Infrastructure providers, strategic partners, changed
leadership
Important to emphasize possibility of off-ramps….
July 2008
©USC-CSSE
32
University of Southern California
Center for Systems and Software Engineering
Acquiring Organization’s ICM-Based CP Plan
• Addresses issues discussed above
– Risk-driven prototyping rounds, concurrent definition and
development, continuity of support, stakeholder involvement,
off-ramps
• Organized around key management questions
– Objectives (why?): concept feasibility, best system solution
– Milestones and Schedules (what? when?): Number and timing of
competitive rounds; entry and exit criteria, including off-ramps
– Responsibilities (who? where?): Success-critical stakeholder
roles and responsibilities for activities and artifacts
– Approach (how?): Management approach or evaluation
guidelines, technical approach or evaluation methods, facilities,
tools, and concurrent engineering
– Resources (how much?): Necessary resources for acquirers,
competitors, evaluators, other stakeholders across full range of
prototyping and evaluation rounds
– Assumptions (whereas?): Conditions for exercise of off-ramps,
rebaselining of priorities and criteria
• Provides a stable framework for pursuing CP
July 2008
©USC-CSSE
33
University of Southern California
Center for Systems and Software Engineering
Outline
• Motivation and Context
• Nature of the ICM
• Applying ICM Principles to CP
• Conclusions, References, Acronyms
– Copy of Young Memo
July 2008
©USC-CSSE
34
University of Southern California
Center for Systems and Software Engineering
CP Conclusions
• CP most effective in reducing technical risk
– If project is low-risk, may not need CP
• May be worth it for teambuilding
• Other significant risks need resolution by Milestone B
– Systemic Analysis DataBase (SADB) sources: management,
acquisition, requirements, staffing, organizing, contracting
• CP requires significant, continuing preparation
– Prototypes are just tip of iceberg
– Need evaluation criteria, tools, testbeds, scenarios, staffing,
procedures
• Need to sustain CP momentum across evaluation breaks
– Useful competitor tasks to do; need funding support
• ICM provides effective framework for CP plan, execution
– CP value propositions, milestone criteria, guiding principles
• CP will involve changes in cultures and institutions
– Need continuous corporate assessment and improvement of
CP-related principles, processes, and practices
July 2008
©USC-CSSE
35
University of Southern California
Center for Systems and Software Engineering
References
•
•
•
•
•
•
•
•
•
•
Blanchard, B., and Fabrycky, W., Systems Engineering and Analysis,
Prentice Hall, 1998 (3rd ed.)
Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century
Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9,
2006.
Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition,
Systems Engineering, and Software Engineering,” CrossTalk, October 2007,
pp. 4-9.
Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems
and Software Engineering,” Proceedings, CSER 2008, April 2008.
Boehm, B., “Dealing with Uncertainties, Risk, and the Value of Information,”
Chapters 19-20, Software Engineering Economics, Prentice Hall, 1981.
Brooks, F. The Mythical Man-Month, Addison Wesley, 1995 (2nd ed.).
Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6,
http://akss.dau.mil/dag/, 2006.
Department of Defense (DoD), Instruction 5000.2, Operation of the Defense
Acquisition System, May 2003.
Department of Defense (DoD), Systems Engineering Plan Preparation Guide,
USD(AT&L), 2004.
Pew, R. W., and Mavor, A. S., Human-System Integration in the System
Development Process: A New Look, National Academy Press, 2007.
July 2008
©USC-CSSE
36
University of Southern California
Center for Systems and Software Engineering
List of Acronyms
CD
CP
DCR
DoD
ECR
EV
EVNI
EVPI
FCR
FED
GAO
July 2008
Concept Development
Competitive Prototyping
Development Commitment
Review
Department of Defense
Exploration Commitment
Review
Expected Value
Expected Value, No Information
Expected Value, Perfect
Information
Foundations Commitment
Review
Feasibility Evidence Description
Government Accounting Office
ICM
KPP
MBASE
OCR
P(FN)
P(FP)
RE
RUP
V&V
VB
VBS
VBF
VC
VCR
©USC-CSSE
Incremental Commitment Model
Key Performance Parameter
Model-Based Architecting and
Software Engineering
Operations Commitment
Review
Probability of False Negatives
Probability of False Positives
Risk Exposure
Rational Unified Process
Verification and Validation
Value of Bold approach
VB for success
VB for failure
Value of Conservative approach
Valuation Commitment Review
37
University of Southern California
Center for Systems and Software Engineering
Competitive Prototyping Policy: John Young Memo
July 2008
©USC-CSSE
38