EMSE 269 - Elements of Problem Solving and Decision Making
CHAPTER 8. SUBJECTIVE PROBABILITY
Frequentist interpretation of probability:
Probability = Relative frequency of occurrence of an event
Frequentist Definition requires one to specify a repeatable experiment.
Example: Throwing a fair coin, Pr(Heads)=0.5
What about following events?
•
Core Meltdown of Nuclear Reactor
•
You being alive at the age of 50?
•
Unsure of the final outcome, though event has occurred.
•
One basketball team beating the other in next day's match
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 202
EMSE 269 - Elements of Problem Solving and Decision Making
Without doubt the above events are uncertain and we talk about the probability of the above events. These probabilities HOWEVER are
NOT Relative Frequencies of Occurrences. What are they?
Subjectivist Interpretation of Probability:
Probability = Degree of belief in the occurrence of an event
By assessing a probability for the events above, one expresses one's DEGREE
OF BELIEF . High probabilities coincide with high degree of belief. Low probabilities coincide with low degree of belief.
Why do we need it?
•
Frequentist interpretation not always applicable.
•
Allows to model and structure individualistic uncertainty through probability.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 203
EMSE 269 - Elements of Problem Solving and Decision Making
ASSESSING SUBJECTIVE DISCRETE PROBABILITIES:
•
Direct Methods: Directly ask for probability assessments.
DO NOT WORK WELL IF EXPERTS ARE NOT
KNOWLEDGABLE ABOUT PROBABILITIES
DO NOT WORK WELL IF PROBABILITIES IN
QUESTIONS ARE VERY SMALL
(SUCH AS FOR EXAMPLE IN RISK ANALYSES)
•
Indirect Methods: Formulate questions in expert's domain of expertise and extract probability assessment through probability modeling. Examples:
Betting Strategies, Reference Lotteries, Paired Comparison Method for
Relative Probabilities.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 204
EMSE 269 - Elements of Problem Solving and Decision Making
1. Assessing Subjective Discrete Probabilities: Betting Strategies
Event: Lakers winning the NBA title this season
STEP 1: Offer a person to choose between following the following bets, where
X=100, Y=0.
Lakers Win
Max Profit
X
Bet for Lakers
Lakers Loose
Lakers Win
-Y
-X
Bet against Lakers
Lakers Loose
Y
Chapter 8 - Page 205 Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 2: Offer a person to choose between following the following bets, where
X=0, Y=100. (Consistency Check)
Max Profit
Lakers Win
X
Bet for Lakers
Lakers Loose
Lakers Win
-Y
-X
Bet against Lakers
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Lakers Loose Y
Chapter 8 - Page 206
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 3: Offer a person to choose between following the following bets, where
X=100, Y=100.
Lakers Win
Max Profit
X
Bet for Lakers
Lakers Loose
Lakers Win
-Y
-X
Bet against Lakers
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Lakers Loose
Y
Chapter 8 - Page 207
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 4: Offer a person to choose between following the following bets, where
X=50, Y=100.
Max Profit
Lakers Win
X
Bet for Lakers
Lakers Loose
Lakers Win
-Y
-X
Bet against Lakers
Lakers Loose Y
CONTINUE UNTIL POINT OF INDIFFERENCE HAS BEEN REACHED.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 208
EMSE 269 - Elements of Problem Solving and Decision Making
Assumption:
When a person is indifferent between bets the expected payoffs from the bets must be the same.
Thus:
X
∗
Pr(LW) - Y
∗
Pr(LL)= -X
∗
Pr(LW) + Y
∗
Pr(LL)
2
∗
X
∗
Pr(LW) - 2
∗
Y
∗
(1- Pr(LW) )=0
Pr(LW) = X
Y
+
Y .
Example: X=50, Y=100
⇒
Pr(LW)=
2
3
≈
66.66%
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 209
EMSE 269 - Elements of Problem Solving and Decision Making
2. Assessing Subjective Discrete Probabilities: Reference Loteries
Event: Lakers winning the NBA title this season
Choose two prices A and B, such that A>>B.
Lakers Win
Hawaiian Trip
Lottery 1
Lakers Loose
(p)
Beer
Hawaiian Trip
Lottery 2
(1-p)
Beer
Chapter 8 - Page 210 Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
EMSE 269 - Elements of Problem Solving and Decision Making
Lottery 2 is the REFERENCE LOTTERY and a probability mechanism is specified for lottery 2.
Examples of Probability Mechanisms:
•
Throwing a fair coin
•
Ball in an urn
•
Throwing a die,
•
Wheel of fortune
Strategy:
1. Specify p
1
.
Ask which one do you prefer?
2. If Lottery 1 is preferred offer change p i
to p
1+i
>
p i .
3. If Lottery 2 is preferred offer change p i to p
1+i
<
p i .
4. When indifference point is reached STOP, else goto 2.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 211
EMSE 269 - Elements of Problem Solving and Decision Making
Assumption:
When Indifference Point has been reached
Pr(LW) = p
Consistency Checking:
Subjective Probabilities must follow the laws of probability
Example:
If expert specifies Pr(A), Pr(B|A) and Pr(A
∩
B
) then
Pr(B|A)
∗
Pr(A) = Pr(A
∩
B
)
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 212
EMSE 269 - Elements of Problem Solving and Decision Making
3. Pairwise Comparisons of Situations (Not on Final Exam)
Chapter 8 - Page 213 Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
EMSE 269 - Elements of Problem Solving and Decision Making
Question: 1
Situation 1
Issaquah
SEA-BRE(A)
Navy
Crossing
0.5 – 5 miles
No Vessel
No Vessel
No Vessel
> 0.5 Miles
Along Ferry
0
Situation 1 is worse
89
Attribute
Ferry Class
Ferry Route
1st Interacting Vessel
Traffic Scenario 1 st
Vessel
Traffic Proximity 1 st
Vessel
2nd Interacting Vessel
Traffic Scenario 2 nd
Vessel
Traffic Proximity 2 nd
Vessel
Visibility
Wind Direction
Wind Speed
Likelihood of Collision
9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9
Situation 2
-
-
Product Tanker
-
-
-
-
-
-
-
-
<===================X==================>> Situation 2 is worse
9: NINE TIMES MORE LIKELY to result in a collision.
7: SEVEN TIMES MORE LIKELY to result in a collision.
5: FIVE TIMES LIKELY to result in a collision.
3: THREE TIMES MORE LIKELY to result in a collision.
1: EQUALY LIKELY to result in a collision.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 214
EMSE 269 - Elements of Problem Solving and Decision Making
UNDERLYING PROBABILITY MODEL FOR PAIRWISE COMPARISON
QUESTIONNAIRE
1
=
1
=
2
=
2
1
=
P
0 e
B
T
Y ( X
1
)
3
Pr( Accident | Prop . Failure, X
1
)
Pr( Accident | Prop.
Failure, X
2
)
P
0 e
β T
Y
P
0 e
β T
( X
1
)
Y ( X 2 )
e
β T
(
Y ( X
1
)
−
Y ( X
2
)
)
4 LN
Pr( Accident
Pr( Accident
|
|
Prop . Failure, X
1
)
Prop.
Failure, X
2
)
T
(
Y ( X
1
)
Y ( X
2
)
)
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 215
EMSE 269 - Elements of Problem Solving and Decision Making
SUBJECTIVE PROBABILITY: ASSESSING CONTINUOUS CDF’S.
Method 1: Ask for distribution parameters e.g.
Method 2: Ask for distribution quantities and solve for parameters (Chapter 10)
Method 3: Ask for shape of CDF e.g. by Assessing a Number of Quantiles
Assessing Quantiles
Definition: x p is the p -th quantile of random variable X
⇔
F(x p
)=Pr(X
≤ x p
)
= p
Terminology: quantile, fractile, percentile, quartile
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 216
EMSE 269 - Elements of Problem Solving and Decision Making
Assessing CDF is often conducted by assessing a number of quantiles.
Method 3A: Use quantile estimates to solve distribution parameters.
Method 3B: Connect multiple quantile estimates by straight lines to approximate the CDF
Example:
Uncertain Event: Current Age of a Movie Actress (e.g. ….)
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 217
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 1: You know age of actress is between 30 and 65
STEP 2: Consider Reference Lottery
Age ≤ 46
Hawaiian Trip
Lottery 1
Age > 46
Beer
(p)
Hawaiian Trip
Lottery 2
(1-p)
Beer
You decide you are indifferent for p=0.5
⇒
Pr(Age
≤
46)=0.5
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 218
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 3: Consider Reference Lottery
Age ≤ 50
Hawaiian Trip
Lottery 1
Age > 50
Beer
(p)
Hawaiian Trip
Lottery 2
(1-p)
Beer
You decide you are indifferent for p=0.8
⇒
Pr(Age
≤
50)=0.8
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 219
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 4: Consider Reference Lottery
Age ≤ 40
Hawaiian Trip
Lottery 1
Age > 40
Beer
(p)
Hawaiian Trip
Lottery 2
(1-p)
Beer
You decide you are indifferent for p=0.05
⇒
Pr(Age
≤
40)=0.05
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 220
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 5: Approximate Cumulative Distribution Function
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0 10 20 30
Years
40 50 60 70
Chapter 8 - Page 221 Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
EMSE 269 - Elements of Problem Solving and Decision Making
Use of REFERENCE LOTTERIES to assess quantiles:
1. Fix Horizontal Axes:
Age ≤ 46
Hawaiian Trip
Lottery 1
Age > 46
Beer
(p)
Hawaiian Trip
Lottery 2
(1-p)
Beer
Strategy: Adjust Probability p until indifference point has been reached by using a probability mechanism.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 222
EMSE 269 - Elements of Problem Solving and Decision Making
2. Fix Vertical Axes:
Age ≤ x
Hawaiian Trip
Lottery 1
Age > x
(0.35)
Beer
Hawaiian Trip
Lottery 2
(0.65)
Beer
Strategy: Adjust the Age x until indifference point has been reached.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 223
EMSE 269 - Elements of Problem Solving and Decision Making
OVERALL STRATEGY TO ASSESS CONTINUOUS CUMULATIVE
DISTRIBUTION FUNCTION:
STEP 1: Ask for the Median (50% Quantile).
STEP 2: Ask for Extreme Values (0% quantile, 100% quantile).
STEP 3: Ask for High/Low Values (5% Quantile, 95% Quantile).
STEP 4: Ask for 1 st
and 3 rd
Quartile (25% Quantile, 75% Quantile).
STEP 5 A: Approximate CDF through straight line technique
STEP 5 B: Model CDF between assessed points
STEP 5 C: Calculate “Best Fit” in a family of CDF’s.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 224
EMSE 269 - Elements of Problem Solving and Decision Making
USING CONTINUOUS CDF’S IN DECISION TREES
•
Advanced Approach: Monte Carlo Simulation (Chapter 11)
•
Simple Approach: Use Discrete Approximation that well approximates the
Expected value of the underlying continuous distributions
1. Extended Pearson Tukey-Method:
•
A Continuous Fan Node is replaced by a Three Branch Uncertainy Node
•
Extended Pearson Tukey-Method specifies what Three Outcomes to choose and which Three Probabilities to assign to these outcomes.
• Works well for Symmetric Continuous Distribution Functions
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 225
EMSE 269 - Elements of Problem Solving and Decision Making
95% Quantile
Median
5% Quantile
0.50
0.40
0.30
0.20
0.10
0.00
1.00
0.90
0.80
0.70
0.60
0
Age=30
Age = 65
10
20 30
Years
40
40 46
50 60
61
70
Age= 40 (0.185)
Age = 46 (0.630)
Age = 61 (0.185)
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 226
EMSE 269 - Elements of Problem Solving and Decision Making
Next, calculate The Expected Value of The Discrete Approximation :
40 0.185
7.4
46
61
0.63
0.185
E[Age]
29.0
11.3
47.7
2. Four Point Bracket Median Method:
A Continuous Fan Node is replaced by a Four Branch Uncertainy node
STEP 1: Divide total range in four equally likely intervals
STEP 2: Determine bracket median in each interval
STEP 3: Assing equal probabilitities in to all bracket medians (0.25 in this case)
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 227
EMSE 269 - Elements of Problem Solving and Decision Making
0.25
0.25
0.25
0.25
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0
0.125
10 20 30 40 50
Years
41 44 48 53
60
Age=30
1 2 3 4
Age= 41 (0.25)
Age= 44 (0.25)
70
Age= 48 (0.25)
Age = 65
Age = 53 (0.25)
Chapter 8 - Page 228 Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
EMSE 269 - Elements of Problem Solving and Decision Making
Next, calculate The Expected Value of The Discrete Approximation :
41
44
48
53
0.25
0.25
0.25
0.25
E[Age]
10.3
11.0
12.0
13.3
46.5
Accuracy of Bracket Median method can be improved by using a five point approximation , a six point approximation , etc. … until the approximated expected value does not change any more (beyond a specified accuracy level).
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 229
EMSE 269 - Elements of Problem Solving and Decision Making
PITFALLS: HEURISTICS AND BIASES
Thinking probabilistically is not easy!!!!
When eliciting expert judgment, expert use primitive cognitive techniques to make their assessments. These techniques are in general simple and intuitively appealing, however they may result in a number of biases.
•
Representative Bias:
Probability estimate is made on the basis of Similarities within a Group . One tend to ignore relevant information such as incidence/base rate.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 230
EMSE 269 - Elements of Problem Solving and Decision Making
Example:
•
X is the event that “a person is sloppy dressed”
•
In your judgement Managers (M) are well dressed: Pr(X|M)=0.1.
•
In your judgement Computer Scientist are badly dressed. Pr(X|C)=0.8.
At a conference with 90% attendence of managers and 10% attendance of computer scientist you Observe a Person and notices that he dresses
(particularly) sloppy.
•
What do think is more likely?:
"The person is a computer scientist" or
"The persion is a manager"
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 231
EMSE 269 - Elements of Problem Solving and Decision Making
WHAT THE ANSWER SHOULD BE?
Pr( X | C ) Pr( C )
Pr( C
Pr( M
| X )
| X )
Pr( X
Pr( X )
| M ) Pr( M )
Pr( X )
Pr( X
Pr( X
| C ) Pr( C )
| M ) Pr( M )
0 .
8
0 .
1
1
0 .
1 * 0 .
9
IN OTHER WORDS: It is more likely that this person is a manager than a computer scientist.
•
Availability Bias:
Probability estimate is made according to the ease with which one can retrieve similar events.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 232
EMSE 269 - Elements of Problem Solving and Decision Making
•
Anchoring Bias:
One makes first assessment (anchor) and make subsequent assessments relative to this anchor.
•
Motivational Bias:
Incentives are always present, such that people do not really say what they believe.
DECOMPOSITION AND PROBABILITY ASSESSMENTS
•
Break down problem into finer detail using probability laws until you have reached a point at which experts are comfortable in making the assessment in a meaning full manner.
Next, aggregate the detail assessment using probability laws to obtain probability estimates at a lower level of detail.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 233
EMSE 269 - Elements of Problem Solving and Decision Making
•
Stock Market Example
Stock
Price
Market
Stock
Price
Stock Price Up
Market Up
Stock Price Up
Stock Price Down
Stock Price Up
Stock Price Down
Market Down
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Stock Price Down
Chapter 8 - Page 234
EMSE 269 - Elements of Problem Solving and Decision Making
EXPERT JUDGEMENT ELICITATION PROCEDURE
STRUCTURED APPROACH TO CAPTURING AN
EXPERTS KNOWLEDGE BASE
KNOWLEDGE BASE INTO
MODELERS SKILLED IN
DECOMPOSITION
AND AGGREGATION OF
ASSESSMENTS
NORMATIVE
EXPERTS
SUBSTANTIVE
KNOWLEDGABLE ABOUT
THE SUBJECT MATTER
AND EXTENSIVE EXPERIENCE
AND CONVERT HIS\HER
QUANTITATIVE ASSESSMENTS .
ELICITATION PROCESS =
MULTIPLE CYCLES (AT LEAST 2)
1. DECOMPOSITION OF
EVENT OF INTEREST TO
A MEANINGFULL LEVEL FOR
SUBSTANTIVE EXPERT
2. ELICITATION OF JUDGMENT OF
SUBSTANTIVE EXPERT FACILI-
TATED BY NORMATIVE EXPERT
3. AGGREGATION OF
JUDGEMENTS
BY NORMATIVE EXPERT
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 235
EMSE 269 - Elements of Problem Solving and Decision Making
•
Nuclear Regulatory Example
Control
System Failure?
Electrical Power
Failure?
Cooling
System Failure?
Accident?
Which probability estimates do we need to calculate:
The Probability of An Accident?
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 236
EMSE 269 - Elements of Problem Solving and Decision Making
A = Accident, L = Cooling System Failure,
N = Control System Failure, E = Electrical System Failure.
STAGE 1: WORK BACKWORDS
STEP 1: Assess the four conditional probabilities:
Pr( A | L , N ), Pr( A | L , N ), Pr( A | L , N ), Pr( A | L , N )
STEP 2: Assess the two conditional probabilities:
Pr( L | E ), Pr( L | E )
STEP 3: Assess the two conditional probabilities
Pr( N | E ), Pr( N | E )
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 237
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 4: Assess the probability:
Pr( E )
STAGE 2: AGGREGATE DETAILED PROBABILITY ESTIMATES
TO ASSESS THE PROBABILITY OF AN ACCIDENT
STEP 5: Apply Law of Total Probability
Pr( A )
Pr( A | L , N )
Pr( L , N )
Pr( A | L , N )
Pr( L , N )
Pr( A | L , N )
Pr( L , N )
Pr( A | L , N )
Pr( L , N )
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 238
EMSE 269 - Elements of Problem Solving and Decision Making
STEP 6: Apply Law of Total Probability
Pr( L , N )
Pr( L , N | E )
Pr( E )
Pr( L , N | E )
Pr( E )
Do the same with:
Pr( L , N ), Pr( L , N ), Pr( L , N )
STEP 7: Apply Conditional Independence assumption
Pr( L , N | E )
Pr( L | E )
Pr( N | E )
STEP 8: Go Back to STEP 5 and substitute the appropriate value to calculate the probability of an accident.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 239
EMSE 269 - Elements of Problem Solving and Decision Making
EXPERT JUDGMENT ELICITATION PRINCIPLES
(Source: “Experts in Uncertainty” ISBN: 0-019-506465-8 by Roger M. Cooke)
1. Reproducibility:
It must be possible for Scientific peers to review and if necessary reproduce all calculations. This entails that the calculational model must be fully specified and the ingredient data must be made available.
2. Accountability:
The source of Expert Judgment must be identified (who do they work for and what is their level of expertise).
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 240
EMSE 269 - Elements of Problem Solving and Decision Making
3. Empirical Control:
Expert probability assessment must in principle be susceptible to empirical control.
4. Neutrality:
The method for combining/evaluating expert judgements should encourage experts to state true opinions.
5. Fairness:
All Experts are treated equally, prior to processing the results of observation
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 241
EMSE 269 - Elements of Problem Solving and Decision Making
PRACTICAL EXPERT JUDGMENT ELICITATION GUIDELINES
1. The questions must be clear.
Prepare an attractive format for the questions and graphic format for the answers.
2. Perform a dry run. Be prepared to change questionnaire format.
3. An Analyst must be present during the elicitation.
4. Prepare a brief explanation of the elicitation format and of the model for processing the responses.
5. Avoid Coaching. (You are not the Expert)
6. The elicitation session should not exceed 1 hour.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 242
EMSE 269 - Elements of Problem Solving and Decision Making
COHERENCE AND THE DUTCH BOOK
•
Subjective probabilities must follow The Laws of Probability.
•
If they do not, the person assessing the probabilities is incoherent.
Incoherence
Possibility of a Dutch Book
Dutch Book:
A series of bets in which the opponent is guaranteed to loose and you win.
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 243
EMSE 269 - Elements of Problem Solving and Decision Making
Example:
There will be a Basketball Game tonight between Lakers and Celtics and your friend says that Pr(Lakers Win)=40% and Pr(Celtic Win)=50%.
You note that the probabilities do not add up to 1, but your friend stubbornly refuses to change his initial estimates.
You think, "GREAT!" let's set up a series of bets!
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 244
EMSE 269 - Elements of Problem Solving and Decision Making
BET 1
Lakers Win (0.4)
Max Profit
$60
BET 2
Celtics Loose (0.5)
Max Profit
-$50
Lakers Loose (0.6) Celtics Win (0.5)
$50
-$40
Note that, EMV of both bets equal $0 according to his probability assessments and can thus be considered fair and he should be willing to engage in both.
Lakers Win : Bet 1 - You win $60, Bet 2: You Loose $50, Net Profit: $10
Lakers Loose: Bet 1 - You loose $40, Bet 2: You win $50, Net Profit: $10
Instructor: Dr. J. Rene van Dorp
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
Chapter 8 - Page 245