Chapters 19 – 20:
Dealing With Uncertainties
•
•
•
•
•
•
TPS Example: Uncertain outcome
Decision rules for complete uncertainty
Utility functions
Expected value of perfect information
Statistical decision theory
The value-of-information procedure
Fixed typo on slide 5. 10/12/2015.
TPS Decision Problem 7
OS Development Options:
• Option B-Conservative (BC)
– Sure to work (cost=$600K)
– Performance = 4000 tr/sec
• Option B-Bold (BB)
– If successful, Cost=$600K
Perf.= 4667 tr/sec
– If not,
Cost=$100K
Perf.= 4000 tr/sec
Which option should we choose?
Operating System
Development Options
Option BB (Bold)
Successful
Not
Successful
Option BC
(Conservative)
Option A
Performance (tr/sec)
4667
4000
4000
2400
Value ($0.3K per tr/sec)
1400
1200
1200
720
Basic cost
600
600
600
170
Total cost
600
700
600
170
Net value NV
800
500
600
550
250
-50
50
0
NV relative to Option A
Payoff Matrix for OS Options
BB, BC
State of Nature
Alternative
Favorable
Unfavorable
BB (Bold)
250
-50
BC (Conservative)
50
50
Decision Rules for Complete
Uncertainty
• Maximin Rule. Determine minimum payoff for
each option. Choose option maximizing the
minimum payoff.
• Maximax Rule. Determine max-payoff for each
option. Choose option maximizing max-payoff.
• Laplace Rule. Assume all states of nature equally
likely. Choose option with maximum expected
value.
BB = 0.5(250) + 0.5(-50) = 100
BC = 0.5(50) + 0.5(50) = 50
Difficulty with Laplace Rule
Favorable
U1
U2
BB
250
-50
-50
Expected
Value
50
BC
50
50
50
50
U1 : performance failure
U2 : reliability failure
Breakeven Analysis
• Treat uncertainty as a parameter
P= Prob (Nature is favorable)
• Compute expected values as f(P)
EV (BB) = P(250) + (1-P)(-50) = -50 + 300P
EV (BC) = P(50) + (1-P)(50) = 50
• Determine breakeven point P such that
EV (BB) = EV (BC): -50 + 300P = 50, P=0.333
If we feel that actually P>0.333, Choose BB
P<0.333, Choose BC
Utility Functions
Is a Guaranteed $50K
The same as
A 1/3 chance for $250K
and A 2/3 chance for -$50K?
Utility Function for Two Groups of Managers
Utility
20
10
-3
-2
-1
1
2
3
4
5
Payoff ($M)
-10
Group 1
-20
Group 2
Possible TPS Utility Function
20
10
(170,15)
(50,5)
(100,10)
100
-100
-10
(-50,-20)
-20
(-80,-25)
(250,20)
200
300
Utility Functions in S/W
Engineering
• Managers prefer loss-aversion
– EV-approach unrealistic with losses
• Managers’ U.F.’S linear for positive payoffs
– Can use EV-approach then
• People’s U.F.’S aren’t
– Identical
– Easy to predict
– Constant
TPS Decision Problem 7
State of Nature
Alternative
Favorable
Unfavorable
BB (Bold)
250
-50
BC (Conservative)
50
50
•Available decision rules inadequate
•Need better information
Info. has economic value
Expected Value of Perfect Information
(EVPI)
Build a Prototype for $10K
– If prototype succeeds, choose BB
Payoff: $250K – 10K = $240K
– If prototype fails, choose BC
Payoff: $50K – 10K = $40K
If equally likely,
Ev = 0.5 ($240K) + 0.5 ($40K) = $140K
Could invest up to $50K and do better than before
– thus, EVPI = $50K
However, Prototype Will Give
Imperfect Information
That is,
P(IB|SF) = 0.0
Investigation (Prototype) says choose bold
state of nature: Bold will fail
P(IB|SS) = 1.0
Investigation (prototype) says choose bold
state of nature: bold will succeed
Suppose we assess the prototype’s imperfections as
P(IB|SF) = 0.20,
P(IB|SS) = 0.90
And Suppose the states of nature are equally likely
P(SF) = 0.50
P(SS) = 0.50
We would like to compute the expected value of using
the prototype
EV(IB,IC) = P(IB) (Payoff if use bold)
+P(IC) (Payoff if use conservative)
= P(IB) [ P(SS|IB) ($250K) + P(SE|IB) (-$50K) ]
+ P(IC) ($50K)
But these aren’t the probabilities we know
How to get the probabilities we need
P(IB) = P(IB|SS) P(SS) + P(IB|SF) P(SF)
P(IC) = 1 – P(IB)
P(SS|IB) = P(IB|SS) P(SS)
P(IB)
(Bayes’ formula)
P(SF|IB) = 1 – P (SS|IB)
P(SS|IB) = Prob (we will choose Bold in a state of nature where it will succeed)
Prob (we will choose Bold)
Net Expected Value of Prototype
PROTO
COST, $K
P (PS|SF)
P (PS|SS)
0
EV, $K
NET EV,
$K
60
0
0.30
0.80
69.3
4.3
10
0.20
0.90
78.2
8.2
20
0.10
0.95
86.8
6.8
30
0.00
1.00
90
0
NET EV, $K
5
8
4
0
10
20
PROTO COST, $K
30
Conditions for Successful
Prototyping (or Other Info-Buying)
1.
2.
3.
4.
5.
There exist alternatives whose payoffs vary
greatly depending on some states of nature.
The critical states of nature have an appreciable
probability of occurring.
The prototype has a high probability of
accurately identifying the critical states of nature.
The required cost and schedule of the prototype
does not overly curtail its net value.
There exist significant side benefits derived from
building the prototype.
Pitfalls Associated by
Success Conditions
1. Always build a prototype or simulation
–
May not satisfy conditions 3,4
2. Always build the software twice
–
May not satisfy conditions 1,2
3. Build the software purely top-down
–
May not satisfy conditions 1,2
4. Prove every piece of code correct
–
May not satisfy conditions 1,2,4
5. Nominal-case testing is sufficient
–
May need off-nominal testing to satisfy
conditions 1,2,3
Statistical Decision Theory: Other
S/W Engineering Applications
How much should we invest in:
– User information gathering
– Make-or-buy information
– Simulation
– Testing
– Program verification
How much should our customers invest in:
– MIS, query systems, traffic models, CAD,
automatic test equipment, …
The Software
Engineering Field
Exists Because
Processed
Information Has Value