University of Southern California
Center for Systems and Software Engineering
Life Cycle Cost Modeling and Risk
Assessment for 21st Century Enterprises
Barry Boehm, Jo Ann Lane, Supannika
Koolmanojwong, Richard Turner
USC-CSSE ARR, April 29, 2014
boehm@usc.edu,
http://csse.usc.edu
University of Southern California
Center for Systems and Software Engineering
Outline
• Challenges for 21st Century Enterprises
• Incremental Commitment Spiral Model (ICSM)
Framework for Addressing Challenges
• Proposed Approach for Cost Modeling
• Updated Survey of Top-10 Risks
1/13/2014
© USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Challenges for 21st Century Enterprises
• Need to deal continually with the 3 D’s
– Dependability: 24/7 assurance of capabilities
• Requires high assurance of robust systems, SoSs
– Dynamism: coping with unforeseen change
• Requires versatility, modifiability, adaptability
• Dependability + Dynamism = Resilience
– Diversity: of D + D needs across enterprise
• No one-size-fits-all solutions
• Need to identify enterprise frameworks for generating
system and SoS solutions
– ICSM provides process framework for doing this
• Need corresponding capabilities for cost, schedule,
and tradespace modeling
1/13/2014
© USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
ICSM Framework for Addressing 3D Challenges
• Identify success-critical stakeholders (SCSHs) and
their 3D value propositions
– At enterprise-portfolio level and portfolio-project level
• Explore alternative enterprise and portfolio
architectures and their ability to support value
propositions at all three levels
– Develop, evaluate evidence of feasibility for each
– Baseline most satisfactory combination
• Pilot, evaluate, evolve baseline combination
– Identify risks of further use for each
– If risks unacceptable to SCSHs, descope or discontinue
• And explore alternative approaches to satisfying needs
• Avoids Procrustean Bed of one-size-fits-all solutions
1/13/2014
© USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
The Procrustean Bed
• Procrustes: Greek Mythology
– Rogue smith and bandit
– Hostel with one-size-fits-all bed
– Guests too small: stretch them
to fit
– Guests too large: lop off the
offending parts
1/13/2014
© USC-CSSE
5
University of Southern California
Center for Systems and Software Engineering
Build Your Own Procrustean Bed
• Pure Waterfall, Vee: Fixed Price and Spec Contract
– Lop off needed changes as requirements creep
• Pure Agile: Easiest First; Dedicated On-Site Customer
– Later scalability and assurance problems; single-failure point
• Voice of the Customer: Accept All “Requirements”
– Gold-plating; neglect voices of acquirer, developer, owner
• Piling on Incompatible Constraints: No Way Out
– Project Example: Waterfall, COTS, Ada, GOTS Reuse
• Inflexible Standards: No Choice But Tailoring Down
– MIL-STD-498: choice of 23, 6, or 1 DID denied
• Overconstrained Maturity Models: Excluding Expertise
– Software CMM: Exclude software group from system rqts.
1/13/2014
© USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Current System Acquisition Methods
Too easy to misinterpret as one-size-fits-all
• V-Model1
• Spiral Model2
High level guidance assumes that acquirers have extensive acquisition experience...
Without experience, too easy to misinterpret and auger in with disastrous results...
1
2
http://en.wikipedia.org/wiki/V-Model
1/13/2014
© USC-CSSE
http://en.wikipedia.org/wiki/Spiral_model
7
University of Southern California
Center for Systems and Software Engineering
Procrustean Example: DoD Acquisition Process
University of Southern California
Center for Systems and Software Engineering
Progress: Draft DoDI 5000.02 2013-03-12
• MODEL 1: HARDWARE INTENSIVE PROGRAM
• MODEL 2: DEFENSE UNIQUE SOFTWARE
INTENSIVE PROGRAM
• MODEL 3: INCREMENTALLY FIELDED SOFTWARE
INTENSIVE PROGRAM
• MODEL 4: ACCELERATED ACQUISITION PROGRAM
• Hybrid Program A (Hardware Dominant).
• Hybrid Program B (Software Dominant).
1/13/2014
© USC-CSSE
9
University of Southern California
Center for Systems and Software Engineering
What is the ICSM?
• Risk-driven framework for determining and
evolving best-fit system life-cycle process
• Integrates the strengths of phased and riskdriven spiral process models
• Synthesizes together principles critical to
successful system development
– Stakeholder value-based guidance
– Incremental commitment and accountability
– Concurrent multidiscipline engineering
– Evidence and risk-driven decisions
Principles
trump
diagrams…
Principles used by 60-80% of CrossTalk Top-5 projects, 2002-2005
1/13/2014
© USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Spiral Model
Cumulative Level of Understanding, Product and Process
Detail (Risk-Driven)
Concurrent
Engineering of
Products and
Processes
OPERATION2
DEVELOPMENT3
FOUNDATIONS4
OPERATION1
DEVELOPMENT2
FOUNDATIONS3
DEVELOPMENT1
FOUNDATIONS2
FOUNDATIONS
RISK-BASED
STAKEHOLDER
COMMITMENT
REVIEW
POINTS:
VALUATION
EXPLORATION
6
5
4
3
2
1
Opportunities to
proceed, skip
phases
backtrack, or
terminate
Risk-Based Decisions
Evidence-Based Review Content
- A first-class deliverable
- Independent expert review
- Shortfalls are uncertainties and risks
Acceptable
Negligible
Risk
Too High,
Unaddressable
High, but
Addressable
1/13/2014
© USC-CSSE
11
1
Exploration Commitment Review
2
Valuation Commitment Review
3
Foundations Commitment Review
4
Development Commitment Review
5
Operations1 and Development2
Commitment Review
6
Operations2 and Development3
Commitment Review
University of Southern California
Center for Systems and Software Engineering
ICSM Nature and Origins
• Integrates hardware, software, and human factors
elements of systems life cycle
– Concurrent exploration of needs and opportunities
– Concurrent engineering of hardware, software, human aspects
– Concurrency stabilized via anchor point milestones
• Developed in response to a variety of issues
– Clarify “spiral development” usage
• Initial phased version (2005)
– Provide framework for human-systems integration
• National Research Council report (2007)
• Integrates strengths of current process models
– But not their weaknesses
• Facilitates transition from existing practices
– Electronic Process Guide (2009)
1/13/2014
Copyright
© USC-CSSE
© USC-CSSE
12
University of Southern California
Center for Systems and Software Engineering
Incremental Commitment in
Gambling
• Total Commitment: Roulette
– Put your chips on a number
• E.g., a value of a key performance parameter
– Wait and see if you win or lose
• Incremental Commitment: Poker, Blackjack
– Put some chips in
– See your cards, some of others’ cards
– Decide whether, how much to commit to
proceed
1/13/2014
Copyright
© USC-CSSE
© USC-CSSE
13
13
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Spiral Process: Phased View
Anchor Point
Milestones
Synchronize, stabilize concurrency via FEDs
Risk patterns
determine life
cycle process
1/13/2014
© USC-CSSE
14
University of Southern California
Center for Systems and Software Engineering
ICSM Activity
Levels for
Complex
Systems
1/13/2014
© USC-CSSE
15
University of Southern California
Center for Systems and Software Engineering
Anchor Point Feasibility Evidence Descriptions
• Evidence provided by developer and validated by independent
experts that:
If the system is built to the specified architecture, it will
– Satisfy the requirements: capability, interfaces, level of service, and
evolution
– Support the operational concept
– Be buildable within the budgets and schedules in the plan
– Generate a viable return on investment
– Generate satisfactory outcomes for all of the success-critical
stakeholders
• All major risks resolved or covered by risk management plans
• Serves as basis for stakeholders’ commitment to proceed
• Synchronizes and stabilizes concurrent activities
Can be used to strengthen current schedule- or event-based reviews
1/13/2014
© USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
The ICSM as Risk-Driven Process
Generator
• Stage I of the ICSM has 3 decision nodes with 4 options/node
– Culminating with incremental development in Stage II
– Some options involve go-backs
– Results in many possible process paths
• Can use ICSM risk patterns to generate frequently-used
processes
– With confidence that they fit the situation
• Can generally determine this in the Exploration phase
– Develop as proposed plan with risk-based evidence at VCR
milestone
– Adjustable in later phases
1/13/2014
© USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
Different Risk Patterns Yield Different Processes
1/13/2014
Copyright
© USC-CSSE
© USC-CSSE
18
18
University of Southern California
Center for Systems and Software Engineering
The ICSM Process Decision Table:
Key Decision Inputs
•
•
•
•
Product and project size and complexity
Requirements volatility
Mission criticality
Nature of Non-Developmental/COTS/Services
support
– Commercial, open-source, reused components
– Cloud services
• Organizational and Personnel Capability
1/13/2014
© USC-CSSE
19
University of Southern California
Center for Systems and Software Engineering
ICSM Common Case Decision Table
University of Southern California
Center for Systems and Software Engineering
Case 2: Architected Agile
•
Exploration phase determines
– Need to accommodate fairly rapid change, emergent requirements, early user
capability
– Low risk of scalability up to 100 people
– NDI support of growth envelope
– Nucleus of highly agile-capable personnel
– Moderate to high loss due to increment defects
•
•
•
•
•
•
•
•
•
Example: Business data processing
Size/complexity: Medium
Anticipated change rate (% per month): 1-10%
Criticality: Medium to high
NDI support: Good, most in place
Organizational and personnel capability: Agile-ready, med-high capability
Key Stage I activities: Combined Valuation and Architecting phase,
complete NDI preparation
Key Stage II activities: Architecture-based scrum of scrums
Time/build: 2-4 weeks
Time/increment: 2-6 months
1/13/2014
© USC-CSSE
21
University of Southern California
Center for Systems and Software Engineering
USA Medical Case Study
• 1400 software people; 7M SLOC; 7 sites
– 4 in Europe, 2 in India
• 500 medical applications; 500 financial; others
• Survivability-critical software problems
– Reliability, productivity, performance, interoperability
– Sarbanes-Oxley requirements
– Management receptive to radical change
• Some limited experimental use of agile methods
– Led by top software technologist/manager
• Committed to total change around Scrum and XP
1/13/2014
© USC-CSSE
22
University of Southern California
Center for Systems and Software Engineering
USA Medical Adoption Profile
100
90
80
70
60
50
40
30
20
10
0
• July 2004 - July 2005
Scrum
Teams
2004
Jul
1/13/2014
2006
Jul
– Recruit top people from all
sites into core team(s)
– Get external expert help
– Develop architecture
– Early Scrum successes with
infrastructure
– Revise policies and practices
– Train, reculture everyone
– Manage expectations
• July 2005 – July 2006
– Begin full-scale development
– Core teams as mentors
© USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Architected Agile Approach
• Uses Scrum of Scrums approach
– Up to 10 Scrum teams of 10 people each
– Has worked for distributed international teams
– Going to three levels generally infeasible
• General approach shown below
– Often tailored to special circumstances
1/13/2014
© USC-CSSE
24
University of Southern California
Center for Systems and Software Engineering
Architected Agile – USA Medical
• Include customers and marketers
– New roles; do’s/don’ts/opportunities; CRACK personnel; full
collaboration and teamwork; expectations management
• Scrum; most XP practices; added company practices
– 6-12 person teams with team rooms, dedicated servers
– Hourly smoke test; nightly build and regression test
– Just-in-time analysis; story-point estimates; fail fast; detailed short-term
plans; company architecture compliance
– Embrace change in applications and practices
– Global teams: wikis, daily virtual meetings, act as if next-door
• Release management
– 2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint;
1-6 month beta test
– Next Sprint Zero concurrent with Release Sprint
• Initiative manager and team
– Define practices; evolve infrastructure; provide training; guide
implementation; evaluate compliance/usage; continuous improvement
1/13/2014
© USC-CSSE
25
University of Southern California
Center for Systems and Software Engineering
Approach Used to Evolve ICSM
• FCR success condition
– Describes at least one feasible architecture
– Satisfying requirements within cost/schedule/resource
constraints
– Viable cost-effective business case
– Stakeholder concurrence on key system parameters
• Projects That Failed FCR Criteria
- 1996: 4 out of 16 (25%)
- 1997: 4 out of 15 (27%)
why?
3/22/01
©USC-CSE
26
University of Southern California
Center for Systems and Software Engineering
The Two Cultures: What’s Easy and Hard?
•Easy/hard things for software people
“If you can do queries with all those ands, ors, synonyms, data
ranges, etc., it should be easy to do natural language queries.”
“If you can scan the document and digitize the text, it should be
easy to digitize the figures too.”
•Easy/hard things for librarians
“It was nice that you could add this access feature, but it overly
(centralizes, decentralizes) control of our intellectual property
rights.”
“It was nice that you could extend the system to serve the medical
people, but they haven’t agreed to live with our usage guidelines.”
3/22/01
©USC-CSE
27
University of Southern California
Center for Systems and Software Engineering
1998 Simplifier/Complicator Experiment
• Identify application simplifiers and complicators
– For each digital library sub-domain
– For both developers and clients
• Provide with explanations to developers and clients
– Highlight relation to risk management
• Homework exercise to analyze simplifiers and
complicators
– For two of upcoming digital library projects
• Evaluate effect on LCO review failure rate
3/22/01
©USC-CSE
28
University of Southern California
Center for Systems and Software Engineering
Example S&C’s for Developers
Type of
Application
Simple Block Diagram
query
Catalog MM asset
info
Multimedia
Archive
update
notification
query
update
MM
Archive
MM asset
Examples
1, 3, 4, 5,
6, 7, 8, 9,
10, 11,
12, 13,
14, 15,
20, 31,
32, 35,
36, 37, 39
Simplifiers
·
·
·
Use standard
query languages
Use standard or
COTS search
engine
Uniform media
formats
Complicators
·
·
·
·
·
·
·
·
3/22/01
©USC-CSE
Natural language
processing
Automated
cataloging or
indexing
Digitizing large
archives
Digitizing
complex or
fragile artifacts
Rapid access to
large Archives
Access to
heterogeneous
media collections
Automated
annotation/descrip
tion/ or meanings
to digital assets
Integration of
legacy systems
29
University of Southern California
Center for Systems and Software Engineering
The Results
• Projects That Failed LCO Criteria
- 1996: 4 out of 16 (25%)
- 1997: 4 out of 15 (27%)
- 1998: 1 out of 19 (5%)
- 1999: 1 out of 22 (5%)
- 2000: 2 out of 20 (10%)
• Post-1997 failures due to non-S&C causes
– Team cohesion, client outages
3/22/01
©USC-CSE
30
University of Southern California
Center for Systems and Software Engineering
Outline
• Challenges for 21st Century Enterprises
• Incremental Commitment Spiral Model (ICSM)
Framework for Addressing Challenges
• Proposed Approach for Cost Modeling
• Updated Survey of Top-10 Risks
1/13/2014
© USC-CSSE
31
University of Southern California
Center for Systems and Software Engineering
Example: COCOMO II Cost Model
• 1995: one-size-fits-all model for 21st century software
• 1999: poor fit for schedule-optimized projects; CORADMO
• 2000: poor fit for COTS-intensive projects: COCOTS
• 2003: need model for product line investment: COPLIMO
• 2003: poor fit for agile projects: Agile COCOMO II (partial)
• 2012: poor fit for incremental development: COINCOMO
1/13/2014
© USC-CSSE
32
COCOMO Family of Cost Models
University of Southern California
Center for Systems and Software Engineering
Software Cost Models
COCOMO 81
1981
DBA COCOMO
2004
COCOMO II
2000
iDAVE
2004
COQUALMO
1998
AGILE C II
2003
COCOTS
2000
COINCOMO
2004,2012
COPLIMO
2003
COTIPMO
2011
Other Independent
Estimation Models
COSYSMO
2005
COSoSIMO
2007
COPSEMO
1998
COPROMO
1998
COSECMO
2004
CORADMO
1999,2012
Software Extensions
Legend:
Model has been calibrated with historical project data and expert (Delphi) data
Model is derived from COCOMO II
Model has been calibrated with expert (Delphi) data
2/7/2014
Dates indicate the time thatUSC-CSSE/SERC
the first paper was published for the model
33
University of Southern California
Center for Systems and Software Engineering
COCOMO II Data by 5-Year Periods
1/13/2014
© USC-CSSE
34
University of Southern California
Center for Systems and Software Engineering
COCOMO II Data: Productivity Trends
1/13/2014
© USC-CSSE
35
University of Southern California
Center for Systems and Software Engineering
COCOMO II Data: Process Maturity Trends
1/13/2014
© USC-CSSE
36
University of Southern California
Center for Systems and Software Engineering
SRDR Data: Productivity vs. Size, CMM Level
400
350
300
250
0-1 EKSLOC
1-10 EKSLOC
200
10-50 EKSLOC
50-100 EKSLOC
150
>100 EKSLOC
100
50
0
CMM 2
1/13/2014
CMM 3
CMM 4
CMM 5
CMMI 3
© USC-CSSE
CMMI 4
CMMI 5
37
University of Southern California
Center for Systems and Software Engineering
Trends Confounded by Missing Variables
Incremental Development Productivity Decline
QMP
12
10
8
Productivity
6
Linear (Productivity)
Log. (Productivity)
4
y = -0.7989x + 8.5493
R² = 0.3693
2
y = -2.708ln(x) + 8.7233
R² = 0.5326
0
1
12/03/2013
2
3
4
5
Copyright © USC-CSSE
6
38
University of Southern California
Center for Systems and Software Engineering
COCOMO II Status and COCOMO III Plans
• Baseline COCOMO II calibrated to 161 project data points
– Pred (20%; 30%) = (63%; 70%) general; (75%;80%) local
• Added 149 data points 2000-2009
– Pred (30%) < 40% general; some sources ~70-80% local
– Some improvement with added variables (year; domain; agility)
• But some data-source mismatches unexplainable
• Vu Nguyen analysis of full dataset suggests further adjusting
– Rating scales for experience, tools, reliability
• Proposed approach for COCOMO III
– Explore models for unexplained existing sources or drop
– Try added variables for mostly-general fit to existing data
– Obtain more data to validate results
1/13/2014
© USC-CSSE
39
University of Southern California
Center for Systems and Software Engineering
COSYSMO 2.0 Status and COSYSMO 3.0 Plans
• Current COSYSMO 2.0 covers SysE with reuse (SEWR)
– But not SysE for reuse (SEFR), requirements volatility (SERV)
Also have a COSYSMO covering SERV but not SEWR, SEFR
• Based on Boeing data vs. BAE Systems data for COSYSMO 2.0
• Proposed approach for COSYSMO 3.0
– Converge rating scales for SEWR, SEFR, SERV
– Apply to existing BAE, Boeing data points
– Obtain additional data points including SEWR, SEFR, SERV
• Including original COSYSMO 1.0 data points where possible
– Derive best-fit COCOMO 3.0 for overall dataset
• Perhaps involving added variables, domain subsetting
– Implies continuation of NDAs for full dataset
• Also explore COSYSMO as an improved framework for
prediction of system development costs
1/13/2014
© USC-CSSE
40
University of Southern California
Center for Systems and Software Engineering
Outline
• Challenges for 21st Century Enterprises
• Incremental Commitment Spiral Model (ICSM)
Framework for Addressing Challenges
• Proposed Approach for Cost Modeling
• Updated Survey of Top-10 Risks
1/13/2014
© USC-CSSE
41
University of Southern California
Center for Systems and Software Engineering
Updated Survey of Top-10 Risks
• ICSM book Risk chapter has list of top-10 risks
– Based on Thammanoon survey of several top-10 risk lists
1. Inflated expectations
2. Success-critical stakeholder lack of involvement or value
conflicts
3. Underdefined plans and requirements
4. Architecture/reuse/non-developmental item (NDI) conflicts
5. Personnel shortfalls
6. Immature or obsolete processes; unbridled requirements
volatility
7. Human–system integration shortfalls
8. Legacy asset incompatibilities
9. Unbalanced nonfunctional requirements or -ilities
10. Immature technology
1/13/2014
© USC-CSSE
42
University of Southern California
Center for Systems and Software Engineering
Candidates for Top-10 risks
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
_______
Bad architectural decisions
Bad NDI (COTS, services, open source, reuse) decisions
Immature technology
Inadequate or misunderstood requirements
Lack of senior management commitment
Misaligned stakeholders
Neglected stakeholders
Obsolete processes, technology
Personnel shortfalls
Uncontrollable external interfaces
Unrealistic budgets and/or schedules (Inflated Expectations)
Unresolved quality-attribute conflicts
Unsatisfactory human-system interface
Weak/no analysis of alternatives
Weak business case, competition assessment
_______
________________________________________________________
1/13/2014
© USC-CSSE
43