Evidence-Based Decision Making
If doctors can do it...
administrators can do it?
NVZD Voorjaarscongres – 4 Juni 2015 - Nyenrode
Exercise
Think about a decision you have been involved in
making. This decision should be one which:
Was reasonably important for your organization
Involved spending significant resources
Involved several or more people
Was made over a period of time (ie. weeks or months)
Did not have an easy ‘answer’
Exercise
Discuss with your neighbor (1 min)
What exactly was the problem (or opportunity)?
How many alternative decision options were considered?
How much evidence was used, and from which sources
(scientific, organizational, experience, crystal ball?)
Was any attempt made to explicitly evaluate its quality or
trustworthiness?
Evidence based decision-making:
What is it?
Evidence-based decision making
Central Premise:
Decisions should be based on a
combination of critical thinking and
the ‘best available evidence‘.
Evidence?
outcome of scientific research,
organizational facts & figures,
benchmarking, best practices,
personal experience
All managers and leaders
base their decisions on
‘evidence’
But…many managers and
leaders pay little or no
attention to the quality of the
evidence they base their
decisions on
Trust me, 20 years of
management experience
SO ...
Teach managers/leaders
how to critically evaluate the
validity, and generalizability of
the evidence and help them
find ‘the best available’
evidence
Evidence based decision
Scientific
research
outcomes
diagnosis
Organizational data,
facts and figures
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
intervention
Stakeholders’ values
and concerns
Evidence based practice:
Where does it come from?
Medicine: Founding fathers
David Sackett
Gordon Guyatt
McMaster University Medical School, Canada
How it all started
5 steps of EBmed
1. Ask: translate a practical issue into an answerable question
2. Acquire: systematically search for and retrieve the evidence
3. Appraise: critically judge the trustworthiness of the evidence
4. Apply: incorporate the evidence into the decision-making process
5. Assess: evaluate the outcome of the decision taken
Evidence-Based Practice
1991
Medicine
1998
Education
2000
Social care, public policy
Nursing, Criminal justice,
Policing, Architecture, Conservation
2010
Management
Evidence-Based Practice
Evidence-Based Practice
Evidence-Based Practice
Evidence-Based Practice
Evidence-based decision-making
=
the use of evidence from multiple
sources to increase the likelihood of a
favourable outcome
Focus on the decision making process
Think in terms of probability
Evidence-Based Decision-Making
Why do we need it?
True or false?
1. Incompetent people benefit more from feedback than
highly competent people.
2. Task conflict improves work group performance while
relational conflict harms it.
3. Encouraging employees to participate in decision
making is more effective for improving organizational
performance than setting performance goals.
How evidence-based is your HR director?
 959 (US) + 626 (Dutch) HR professionals
 35 statements, based on an extensive body of evidence
 true / false / uncertain
HR Professionals' beliefs about effective human resource practices: correspondence
between research and practice, (Rynes et al, 2002, Sanders et al 2008)
Outcome: not better than random chance
Evidence-based decision making
Scientific
research
outcomes
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
Organizational data,
facts and figures
Stakeholders’ values
and concerns
Thinking critical about
professional experience and judgment
Discuss with your neighbor (1 min)
Why is a physician’s clinical experience,
as a rule, more trustworthy than
a manager’s professional experience?
Developing expertise
1. A sufficiently regular, predictable environment
2. Opportunities to learn regularities through prolonged
practice and feedback
The management domain is not highly
favorable to expertise!
Bounded rationality
Bounded rationality / prospect theory
System 1
 Fast
 Intuitive, associative
 heuristics & biases
System 2




Slow (lazy)
Deliberate,
Reasoning
Rational
System 1: short cuts
System 1: short cuts
Shepard’s tables
System 1: necessary to survive
95%
System 1: cognitive errors








Seeing order in randomness
Overconfidence bias
Halo effect
False consensus effect
Group think
Self serving attribution bias
Sunk cost fallacy
Cognitive dissonance reduction







Confirmation bias
Authority bias
Small numbers fallacy
In-group bias
Recall bias
Anchoring bias
Availability bias
Cognitive errors
1. Pattern recognition
2. Confirmation-bias
3. Groupthink
Error 1: pattern recognition
We are predisposed to see order, pattern and causal
relations in the world.
Patternicity: The tendency to find meaningful patterns in
both meaningful and meaningless noise.
Bias 1: pattern recognition
We are pattern seeking primates: association learning
Points of impact of V-1 bombs in London
Points of impact of V-1 bombs in London
Error 1: pattern recognition
 A Type I error or a false positive, is
believing a pattern is real when it is not
(finding a non existent pattern)
 A Type II error or a false negative, is
not believing a pattern is real when it is
(not recognizing a real pattern)
Dr. Michael Shermer
(Director of the Skeptics Society)
Error 1: pattern recognition
 A Type I error or a false positive: believe that the
rustle in the grass is a dangerous predator when it is
just the wind (low cost)
Error 1: pattern recognition
 A Type II error or a false negative: believe that the
rustle in the grass is just the wind when it is a
dangerous predator (high cost)
Error 1: pattern recognition
 A Type I error or a false positive: believe that the
rustle in the grass is a dangerous predator when it is
just the wind (low cost)
 A Type II error or a false negative: believe that the
rustle in the grass is just the wind when it is a
dangerous predator (high cost)
Error 1: pattern recognition
superstitious
rituals
more stress = more prone to type 1 errors
superstitious
rituals
Error 1: pattern recognition
Cognitive errors
1. Pattern recognition
2. Confirmation-bias
3. Groupthink
2. Confirmation bias
We are predisposed to selectively
search for or interpret information in
ways that confirms our existing beliefs,
expectations and assumptions, and
ignore information to the contrary.
In other words, we “see what we want to
see”
2. Confirmation bias
Example
You may believe that astrology actually
works. As a result of confirmation bias
you’ll remember only those instances
when when the prediction in the astrology
column came true and forget the majority
of the cases when the prediction was very
wrong. As a result you will continue to
believe astrology has some base in reality
Error 2: confirmation bias
Confirmation bias
Pattern recognition
McKinsey (1997 case study / 2001 book)
McKinsey: case study
War on Talent
Errors
1. Pattern recognition
2. Confirmation-bias
3. Groupthink
Error 3: Groupthink
Groupthink:
Groupthink is a psychological
phenomenon that occurs within a
group of people, in which the desire
for harmony or conformity in the
group results in an incorrect or
irrational decision
Bias 3: Group think
Error 3: Groupthink
Group think?
Lean Management / Lean Six Sigma
Self steering / autonomous teams
Agile working / New World of Working
Value based management / health care
Talent management
Employee engagement
Bounded rationality
“I’ve been studying judgment for 45 years, and I’m no better
than when I started. I make extreme predictions. I’m overconfident. I fall for every one of the biases.”
Evidence based decision
Scientific
research
outcomes
diagnosis
Organizational data,
facts and figures
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
intervention
Stakeholders’ values
and concerns
Evidence-based decision making
Scientific
research
outcomes
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
Organizational data,
facts and figures
Stakeholders’ values
and concerns
Organizational data
Laszlo Bock (CHRO Google)
People operate with beliefs
& biases. To the extent you
can reduce both and replace
them with data, you gain a
clear competitive advantage
Types organizational evidence
1. financial data (cash flow, solvability)
2. business outcomes (ROI, market share)
3. customer/client impact (customer satisfaction)
4. performance indicators (occupancy rate, failure
frequency)
5. HR metrics (absenteeism, employee engagement)
6. marketing intelligence (brand awareness, customer
feedback)
7. ‘soft’ data (organizational culture, trust in senior
management, leadership style, commitment)
8. data from benchmarking
Organizational facts and figures
Examples
Can your organization correlate/regress
productivity
customer satisfaction
level of education
years of experience
+
failure frequency
employee satisfaction
employee turnover
absenteeism
Trends
“Where the evidence is strong, we should act on it.
Where the evidence is suggestive, we should consider it.
Where the evidence is weak, we should build the
knowledge to support better decisions in the future.”
Jeffrey Zients, acting director of the
Office of Management and Budget and
President Obama’s Economic Advisor
In the next weeks,
before you make a decision, ask yourself:
 What exactly is the problem?
 What is the evidence available?
 Was any attempt made to explicitly
evaluate its trustworthiness?