USC
C S E
University of Southern California
Center for Software Engineering
Value-Based Software Engineering:
Case Study and Value-Based Control
Barry Boehm, USC
CS 510 Lecture
Fall 2012
USC
C S E
University of Southern California
Center for Software Engineering
Outline
• Value-based software engineering (VBSE)
process framework
• Application to case study; key practices
• Conclusions and references
©USC-CSSE
2
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1
• Engine: Theory W (stakeholder win-win): What values are
important?
– Enterprise Success Theorem
– Theory of Justice
– Win-Win Equilibrium and Negotiation
• Four Supporting Theories
– Dependency Theory: How do dependencies affect value
realization?
– Results chains; value chains; cost/schedule/performance
tradeoffs
– Utility Theory: How important are the values?
– Multi-attribute utility; Maslow need hierarchy
– Decision Theory: How do values determine decisions?
– Investment theory; game theory; statistical decision theory
– Control Theory: How to monitor and control value realization
– Feedback control; adaptive control; spiral risk control
©USC-CSSE
3
USC
C S E
University of Southern California
Center for Software Engineering
VBSE Theory 4+1 Structure
Dependency
Theory
How do dependencies
affect value realization?
Utility Theory
What values are important?
How is success assured?
How important are the
values?
Theory W:
SCS Win-Win
How to adapt to change and
control value realization?
How do values determine
decision choices?
Control Theory
Decision Theory
©USC-CSSE
4
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1 Process
– With a great deal of concurrency and backtracking
5a, 7b. Option, solution
development & analysis
Dependency
Theory
Utility Theory
2a. Results Chains
2. Identify SCSs
3b, 5a, 7b. Cost/schedule/
performance tradeoffs
3b, 7a. Solution Analysis
3. SCS Value
Propositions
(Win conditions)
4. SCS expectations
management
Theory W:
SCS Win-Win
6, 7c. Refine, Execute,
Monitor & Control Plans
Control Theory
5a, 7b. Prototyping
5. SCS Win-Win
Negotiation
1. Protagonist goals
3a. Solution exploration
7. Risk, opportunity, change
Decision Theory
management
6a, 7c. State measurement,
prediction, correction;
Milestone synchronization
5a. Investment analysis,
Risk analysis
SCS: Success-Critical Stakeholder
©USC-CSSE
5
USC
C S E
University of Southern California
Center for Software Engineering
Need Incremental Approach to VBSE Process
• Cannot make all decisions in a single pass
–
–
–
–
–
Emergent technology: social networking
Emergent requirements: outdo competitors
TBD standards: information exchange; infrastructure
Evolving platforms: mobile devices
Emerging stakeholders: strategic partners, suppliers
• Best to incrementally define scope, features, details
– But do best-effort architecting for the future
• Avoid easiest-first increments on large, critical projects
• Concurrently determine requirements and solutions
• Best to incrementally develop product features
– Avoid obsolete deliveries
– Early focus on critical infrastructure, high-value features
©USC-CSSE
6
USC
C S E
University of Southern California
Center for Software Engineering
Outline
• Value-based software engineering (VBSE)
process framework
• Application to case study; key practices
• Conclusions and references
©USC-CSSE
7
USC
C S E
University of Southern California
Center for Software Engineering
Example Project: Sierra Mountainbikes
– Based on what would have worked on a
similar project
• Quality leader in specialty area
• Competitively priced
• Major problems with order processing
– Delivery delays and mistakes
– Poor synchronization of order entry,
confirmation, fulfillment
– Disorganized responses to problem
situations
– Excess costs; low distributor satisfaction
©USC-CSSE
8
USC
C S E
University of Southern California
Center for Software Engineering
Order Processing Project Goals
Goals: Improve profits, market share,
customer satisfaction via improved order
processing
Questions: Current state? Root causes of
problems? Keys to improvement?
Metrics: Balanced Scorecard of benefits
realized, proxies
– Customer satisfaction ratings; key elements
(ITV: in-transit visibility)
– Overhead cost reduction
– Actual vs. expected benefit and cost flows, ROI
©USC-CSSE
9
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1 Process, Steps 1 and 2
– With a great deal of concurrency and backtracking
Dependency
Theory
2a. Results Chains
Utility Theory
2. Identify SCSs
Theory W:
SCS Win-Win
1. Protagonist goals
Control Theory
Decision Theory
SCS: Success-Critical Stakeholder
©USC-CSSE
10
USC
C S E
University of Southern California
Center for Software Engineering
Frequent Protagonist Classes
Without a pro-active protagonist, a project won’t start
Protagonist Class
Goals
Authority
Ideas
Resources
Leader with Goals, Baseline Agenda
X
X
X
X
Leader with Goals, Open Agenda
X
X
Entrepreneur with Goals, Baseline
Agenda
X
Entrepreneur with Goals, Open Agenda
X
Inventor with Goals, Ideas
X
Consortium with Shared Goals
X
X
X
X
X
X
(X)
(X)
•Sierra Moutainbikes: Susan Swanson, new CEO
– Bicycle champion, MBA, 15 years’ experience
– Leads with goals, open agenda
©USC-CSSE
11
USC
C S E
University of Southern California
Center for Software Engineering
DMR/BRA* Results Chain
Order to delivery time is
an important buying criterion
INITIATIVE
Contribution
Implement a new order
entry system
ASSUMPTION
OUTCOME
Contribution
OUTCOME
Reduced order processing cycle
(intermediate outcome)
Increased sales
Reduce time to process
order
Reduce time to deliver product
*DMR Consulting Group’s Benefits Realization Approach
©USC-CSSE
12
USC
C S E
University of Southern California
Center for Software Engineering
Expanded Order Processing System Benefits Chain
Distributors, retailers,
customers
Assumptions
- Increasing market size
- Continuing consumer satisfaction with product
- Relatively stable e-commerce infrastructure
- Continued high staff performance
New order-entry
system
Developers
Less time,
fewer
errors per
order
entry
system
New order fulfillment
processes,
outreach, training
New order fulfillment
system
Safety, fairness
inputs
Less time, fewer
errors in order
processing
Faster,
better
order
entry
system
Interoperability
inputs
Increased
customer
satisfaction,
decreased
operations costs
Faster order-entry steps, errors
On-time assembly
New order-entry
processes,
outreach, training
Improved supplier
coordination
Sales personnel,
distributors
©USC-CSSE
Increased
sales,
profitability,
customer
satisfaction
Increased profits,
growth
Suppliers
13
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1 Process, Steps 3 and 4
– With a great deal of concurrency and backtracking
Dependency
Theory
2a. Results Chains
3b. Cost/schedule/
performance tradeoffs
Utility Theory
2. Identify SCSs
3b. Solution Analysis
3. SCS Value
Propositions
(Win conditions)
4. SCS expectations
management
Theory W:
SCS Win-Win
1. Protagonist goals
Control Theory
Decision Theory
SCS: Success-Critical Stakeholder
©USC-CSSE
14
USC
C S E
University of Southern California
Center for Software Engineering
The Model-Clash Spider Web: Master Net
- Stakeholder value propositions (win conditions)
©USC-CSSE
15
USC
C S E
University of Southern California
Center for Software Engineering
There is No Universal Quality-Value Metric
• Different stakeholders rely on different value attributes
–
–
–
–
–
Protection: safety, security, privacy
Robustness: reliability, availability, survivability
Quality of Service: performance, accuracy, ease of use
Adaptability: evolvability, interoperability
Affordability: cost, schedule, reusability
• Value attributes continue to tier down
– Performance: response time, resource consumption (CPU,
memory, comm.)
• Value attributes are scenario-dependent
– 5 seconds normal response time; 2 seconds in crisis
• Value attributes often conflict
– Most often with performance and affordability
©USC-CSSE
16
USC
C S E
University of Southern California
Center for Software Engineering
Major Information System Quality Stakeholders
•Dependents
- passengers, patients
•Information Suppliers
-citizens, companies
•Information Brokers
-financial services,
news media
Information System •Information Consumers
-decisions, education,
entertainment
•Mission Controllers
-pilots, distribution
controllers
Developers, Acquirers, Administrators
©USC-CSSE
17
USC
C S E
University of Southern California
Center for Software Engineering
Overview of Stakeholder/Value
Dependencies
•
Strength of direct dependency on value attribute
**- Critical ;
*-Significant; blank-insignificant or indirect
Stakeholders
Info. Suppliers, Dependents
Info. Brokers
Info. Consumers
Mission Controllers, Administrators
Developers, Acquirers
©USC-CSSE
**
**
*
*
*
**
**
**
*
*
**
*
**
*
**
*
*
**
**
**
18
USC
C S E
University of Southern California
Center for Software Engineering
Elaboration of Stakeholder/Value Dependencies
Acquirers
Administrators
Info
Consumers
Cost
Schedule
Reusability
Info Brokers
Protection
Safety
Security
Privacy
Robustross
Reliability
Availability
Survivability
Quality of Service
Performance
Accuracy, Consistency
Accessibility, ease of use;
difficulty of misuse
Evolvability
Interoperability
Dependents
Info Suppliers
Q-Attributes
Developers, Maintenance
Mission Controllers
Stakeholders
Mission crit uncrit
*
**
**
**
*
**
**
**
**
*
**
**
**
**
**
**
**
**
**
**
**
**
**
*
**
**
**
**
**
**
**
*
*
*
©USC-CSSE
**
**
**
**
**
**
*-**
*
**
**
*
**
**
**
**
**
**
*
**
**
*
19
USC
University of Southern California
C S E
Center for Software Engineering
Cost, Schedule, Quality: Pick any Two?
C
S
Q
©USC-CSSE
20
USC
University of Southern California
C S E
Center for Software Engineering
Tradeoffs Among Cost, Schedule, and
Reliability: COCOMO II
9
(RELY, MTBF (hours))
8
(VL, 1)
Cost ($M)
7
(L, 10)
6
5
(N, 300)
4
(H, 10K)
3
(VH, 300K)
•For 100-KSLOC set of features
•Can “pick all three” with 77-KSLOC set of features
2
1
-- Cost/Schedule/RELY:
“pick any two” points
0
0
10
20
30
40
50
Development Time (Months)
©USC-CSSE
21
USC
C S E
University of Southern California
Center for Software Engineering
Cost, Schedule, Quality: Pick any Two?
• Consider C, S, Q as Independent Variable
– Feature Set as Dependent Variable
C
S
C
S
Q
©USC-CSSE
Q
22
USC
C S E
University of Southern California
Center for Software Engineering
C, S, Q as Independent Variable
• Determine Desired Delivered Defect Density (D4)
– Or a value-based equivalent
• Prioritize desired features
– Via QFD, IPT, stakeholder win-win
• Determine Core Capability
– 90% confidence of D4 within cost and schedule
– Balance parametric models and expert judgment
• Architect for ease of adding next-priority features
– Hide sources of change within modules (Parnas)
• Develop core capability to D4 quality level
– Usually in less than available cost and schedule
• Add next priority features as resources permit
• Versions used successfully on 32 of 34 USC digital library
projects
©USC-CSSE
23
USC
C S E
University of Southern California
Center for Software Engineering
Implications for Quality Engineering
• There is no universal quality metric to optimize
• Need to identify system’s success-critical stakeholders
– And their quality priorities
• Need to balance satisfaction of stakeholder dependencies
– Stakeholder win-win negotiation
– Quality attribute tradeoff analysis
• Need value-of-quality models, methods, and tools
– COCOMO example: cost-schedule-quality- size tradeoffs
– Stakeholder win-win negotiation tools
©USC-CSSE
24
USC
C S E
University of Southern California
Center for Software Engineering
Win-lose Generally Becomes Lose-lose
Proposed Solution
“Winner”
Loser
Quick, Cheap,
Sloppy Product
Developer &
Customer
User
Lots of
Developer & User
“bells and whistles”
Customer
Driving too hard a
Customer & User
bargain
Developer
Actually, nobody wins in these situations
©USC-CSSE
25
USC
C S E
University of Southern California
Center for Software Engineering
Key Concepts
• Win Condition: Objective which makes a
stakeholder feel like a winner
• Issue: Conflict or constraint on a win
condition
• Option: A way of overcoming an issue
• Agreement: Mutual commitment to an
option or win condition
©USC-CSSE
26
USC
C S E
University of Southern California
Center for Software Engineering
WinWin Negotiation Model
Win Condition
Issue
involves
covers
Agreement
addresses
adopts
Option
WinWin Equilibrium State
- All Win Conditions covered by Agreements
- No outstanding Issues
©USC-CSSE
27
USC
C S E
University of Southern California
Center for Software Engineering
EasyWinWin OnLine Negotiation Steps
©USC-CSSE
28
USC
C S E
University of Southern California
Center for Software Engineering
Red cells indicate lack of
consensus.
Oral discussion of cell
graph reveals unshared
information, unnoticed
assumptions, hidden
issues, constraints, etc.
©USC-CSSE
29
USC
C S E
University of Southern California
Center for Software Engineering
WikiWinWin: Identify and Resolve Issues
©USC-CSSE
30
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1 Process, Step 5
– With a great deal of concurrency and backtracking
5a. Option, solution
development & analysis
Dependency
Theory
2a. Results Chains
3b, 5a. Cost/schedule/
performance tradeoffs
Utility Theory
2. Identify SCSs
3b. Solution Analysis
3. SCS Value
Propositions
(Win conditions)
4. SCS expectations
management
Theory W:
SCS Win-Win
5a. Prototyping
5. SCS Win-Win
Negotiation
1. Protagonist goals
3a. Solution exploration
Control Theory
Decision Theory
5a. Investment analysis,
Risk analysis
SCS: Success-Critical Stakeholder
©USC-CSSE
31
USC
C S E
University of Southern California
Center for Software Engineering
Project Strategy and Partnerships
• Partner with eServices, Inc. for order processing and
fulfillment system
– Profit sharing using jointly-developed business case
• Partner with key distributors to provide user feedback
– Evaluate prototypes, beta-test early versions, provide
satisfaction ratings
• Incremental development using MBASE/RUP anchor points
– Life Cycle Objectives; Architecture (LCO; LCA)
– Core Capability Drivethrough (CCD)
– Initial; Full Operational Capability (IOC; FOC)
• Architect for later supply chain extensions
©USC-CSSE
32
USC
C S E
University of Southern California
Center for Software Engineering
Business Case Analysis
• Estimate costs and schedules
– COCOMO II and/or alternative for software
– PRICE H or alternative for hardware
– COSYSMO for systems engineering
• Estimate financial benefits
– Increased profits
– Reduced operating costs
• Compute Return on Investment
– ROI = (Benefits – Costs) / Costs
– Normalized to present value
• Identify quantitative metrics for other goals
– Customer satisfaction ratings
• Ease of use; In-transit visibility; overall
– Late delivery percentage
©USC-CSSE
33
USC
C S E
University of Southern California
Center for Software Engineering
Order Processing System Schedules and Budgets
Milestone
Due Date
Budget ($K)
Cumulative Budget ($K)
Inception Readiness
1/1/2004
0
0
Life Cycle Objectives
1/31/2004
120
120
Life Cycle Architecture
3/31/2004
280
400
Core Capability Drivethrough
7/31/2004
650
1050
Initial Oper. Capability: SW
9/30/2004
350
1400
Initial Oper. Capability: HW
9/30/2004
2100
3500
Developed IOC
12/31/2004
500
4000
Responsive IOC
3/31/2005
500
4500
Full Oper. Cap’y CCD
7/31/2005
700
5200
FOC Beta
9/30/2005
400
5600
FOC Deployed
12/31/2005
400
6000
Annual Oper. & Maintenance
3800
Annual O&M; Old System
7600
©USC-CSSE
34
USC
C S E
University of Southern California
Center for Software Engineering
Order Processing System: Expected Benefits
and Business Case
New System
Current System
Market Share %
Sales
Profits
Market Share %
Sales
Profits
Cost Savings
Change in Profits
Cum. Change in Profits
Cum. Cost
ROI
Late Delivery %
Customer Satisfaction (0-5)
In-Transit Visibility (0-5)
Ease of Use (0-5)
Customers
Market Size ($M)
Financial
12/31/03
12/31/04
360
400
20
20
72
80
7
8
20
20
72
80
7
8
0
0
0
0
0
0
0
4
0
-1
12.4
11.4
1.7
3.0
1.0
2.5
1.8
3.0
12/31/05
12/31/06
440
480
20
20
88
96
9
10
22
25
97
120
10
13
2.2
3.2
3.2
6.2
3.2
9.4
6
6.5
-.47
.45
7.0
4.0
4.0
4.3
3.5
4.0
4.0
4.3
12/31/07
520
20
104
11
28
146
16
4.0
9.0
18.4
7
1.63
3.0
4.5
4.3
4.5
12/31/08
560
20
112
12
30
168
19
4.4
11.4 29.8
7.5
2.97
2.5
4.6
4.6
4.6
Date
©USC-CSSE
35
USC
C S E
University of Southern California
Center for Software Engineering
Initial VBSE Theory: 4+1 Process, Steps 6 and 7
– With a great deal of concurrency and backtracking
5a, 7b. Option, solution
development & analysis
Dependency
Theory
Utility Theory
2a. Results Chains
3b, 5a, 7b. Cost/schedule/
2. Identify SCSs
performance tradeoffs
3b, 7a. Solution Analysis
3. SCS Value
Propositions
(Win conditions)
Theory W:
SCS Win-Win
6, 7c. Refine, Execute,
Monitor & Control Plans
Control Theory
4. SCS expectations
management
5a, 7b. Prototyping
5. SCS Win-Win
Negotiation
1. Protagonist goals
3a. Solution exploration
7. Risk, opportunity, change
Decision Theory
management
6a, 7c. State measurement,
prediction, correction;
Milestone synchronization
5a. Investment analysis,
Risk analysis
SCS: Success-Critical Stakeholder
©USC-CSSE
36
USC
C S E
University of Southern California
Center for Software Engineering
A Real Earned Value System
• Current “earned value” systems monitor cost and
schedule, not business value
– Budgeted cost of work performed (“earned”)
– Budgeted cost of work scheduled (“yearned”)
– Actual costs vs. schedule (“burned”)
• A real earned value system monitors benefits
realized
– Financial benefits realized vs. cost (ROI)
– Benefits realized vs. schedule
- Including non-financial metrics
– Actual costs vs. schedule
©USC-CSSE
37
USC
C S E
University of Southern California
Center for Software Engineering
Customer Satisfaction
ITV
Ease of Use
72
7.0
12.4
1.7
1.0
1.8
3/31/04
427
20
72
7.0
12.4
1.7
1.0
1.8
7/31/04
1050
7/20/04
1096
9/30/04
1400
9/30/04
1532
2.4* 1.0* 2.7*
9/30/04
3500
3432
12/31/04
4000
20
80
8.0
0.0
-1.0 11.4
3.0
2.5
3.0
12/20/04
4041
22
88
8.6
0.6
-.85 10.8
2.8
1.6
3.2
3/31/05
4500
300
9.0
3.5
3.0
3.5
3/30/05
4604
324
7.4
3.3
1.6
3.8
Full Op’l Capability
CCD
7/31/05
5200
1000
7/28/05
5328
946
Full Op’l Capability
Beta
9/30/05
5600
1700
9/30/05
5689
1851
12/31/05
6000
2200
22
106 12.2
3.2
-.47
7.0
4.0
3.5
4.0
12/20/05
5977
2483
24
115 13.5
5.1
-.15
4.8
4.1
3.3
4.2
6/30/06
6250
Deployed IOC
Responsive IOC
Full Op’l Capability
Deployed Release 2.1
Increased COTS ITV risk,
fallback identified.
Using COTS ITV fallback; new
HW competitor; renegotiating
HW.
2.7* 1.4* 2.8*
10/11/04
Hardware IOC
Risks/Opportunities
Late Delivery %
20
ROI
400
Cum. Profits
3/31/04
Op’l Cost Savings
Annual Profits ($M)
Software Initial
Op’l Capability (IOC)
Annual Sales ($M)
Core Capability
Demo (CCD)
Market Share %
Life Cycle Architecture
Cost ($K)
Milestone
Schedule
Value-Based Expected/Actual Outcome Tracking Capability
$200K savings from
renegotiated HW.
3.5* 2.5* 3.8*
New COTS ITV source
identified, being prototyped.
New COTS ITV source initially
integrated.
3.8* 3.1* 4.1*
©USC-CSSE
38
USC
C S E
University of Southern California
Center for Software Engineering
Conclusions
• Current SE methods are basically valueneutral
• Value-neutral SE methods are increasingly
risky
• VBSE agenda making progress, but major
challenges remain
– Evolving VBSE theory
– Creating VB counterparts for value-neutral SE
methods
©USC-CSSE
39
USC
C S E
University of Southern California
Center for Software Engineering
References - I
C. Baldwin & K. Clark, Design Rules: The Power of Modularity, MIT Press, 1999.
S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, and P. Gruenbacher (eds.), Value-Based
Software Engineering, Springer, 2005.
B. Boehm, “Value-Based Software Engineering,” ACM Software Engineering Notes,
March 2003.
B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D.
Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, 2000.
B. Boehm and L. Huang, “Value-Based Software Engineering: A Case Study, Computer,
March 2003, pp. 33-41.
B. Boehm & K. Sullivan, “Software Economics: A Roadmap,” The Future of Software
Economics, A. Finkelstein (ed.), ACM Press, 2000.
B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed,
Addison Wesley, 2003 (to appear).
B. Boehm, L. Huang, A. Jain. R. Madachy, “ The ROI of Software Dependability: The
iDAVE Model”, IEEE Software Special Issue on Return on Investment, May/June 2004.
M. Denne and J. Cleland-Huang, Software by Numbers, Prentice Hall, 2004.
©USC-CSSE
40
USC
C S E
University of Southern California
Center for Software Engineering
References - II
S. Faulk, D. Harmon, and D. Raffo, “Value-Based Software Engineering (VBSE): A
Value-Driven Approach to Product-Line Engineering,” Proceedings, First Intl.
Conf. On SW Product Line Engineering, August 2000.
R. Kaplan & D. Norton, The Balanced Scorecard: Translating Strategy into Action,
Harvard Business School Press, 1996.
D. Reifer, Making the Software Business Case, Addison Wesley, 2002.
K. Sullivan, Y. Cai, B. Hallen, and W. Griswold, “The Structure and Value of
Modularity in Software Design,” Proceedings, ESEC/FSE, 2001, ACM Press, pp.
99-108.
J. Thorp and DMR, The Information Paradox, McGraw Hill, 1998.
S. Tockey, Return on Software, Addison Wesley, 2004.
Economics-Driven Software Engineering Research (EDSER) web site:
www.edser.org
MBASE web site : sunset.usc.edu/research/MBASE
©USC-CSSE
41