Evidence-Base Practice
Evidence-based practice?
Decisions should be based on a
combination of critical thinking
and the ‘best available evidence‘.
Evidence =
Various types of information
outcome of scientific research,
organizational facts & data,
benchmarking, best practices,
collective experience, personal
experience, intuition
All managers base their
decisions on ‘evidence’
However ...
Many managers pay little
or no attention to the quality
of the evidence they base
their decisions on
Trust me, 20 years of
management experience
Teach managers how to
critically evaluate the validity,
and generalizability of the
evidence and help them find
‘the best available’ evidence
Best available evidence
 Experiential evidence: the professional insight, understanding, and
expertise of practitioners
 Organizational evidence; data, facts and figures, business
intelligence, and benchmarks
 Scientific evidence: outcome from scientific research published in
peer reviewed journals
 Organizational values and stakeholders’ concerns
Evidence based decision
Best available
scientific evidence
Best available
experiential evidence
Decision
making process
Best available
organizational
evidence
Organizational values
and stakeholders’
concerns
Evidence based decision
Best available
scientific evidence
diagnosis
Best available
organizational
evidence
Best available
experiential evidence
Decision
making process
intervention
Organizational values
and stakeholders’
concerns
Evidence based decision
It’s about probabilities
(not golden bullets)
Evidence-Based Practice
1991
Medicine
1998
Education
1999
Social care, public policy
2000
Nursing
2000
Criminal justice
????
Management?
Evidence-Based Practice
Evidence-Based Practice
Evidence-Based Practice
Evidence-Based Practice
Evidence-Based Practice
What’s the evidence for
evidence-based practice?
Got evidence?
 Forecasts or risk assessments based on the
aggregated experience of multiple persons are more
accurate than forecasts based on the experience of one
person (provided that the forecasts are made
independently before being averaged together)
-
Choudhry, N.K., et al. Systematic review: the relationship between clinical experience and quality of
health care. Ann Intern Med. 2005; 142 (4)
Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London,
2012; p 286 and p 690
Bauer A., et al. Forecast Evaluation with Cross Sectional Data: The Blue Chip Surveys. Economic
Review, Federal Reserva bank of Atlanta, 2003.
Servan-Schreiber, E., et al. Prediction Markets: Does Money Matter? Electronic Markets, 2004: 14 (31).
Scott Armstrong, J. Combining Forecasts, in Principles of Forecasting: A handbook for Researchers and
Practitioners, Kluwer Academic Publishers, New York, 2001
Got evidence?
 Professional judgments based on hard data or
statistical models are more accurate than judgments
solely based on experiential evidence
-
Yaniv, I., & Choshen-Hillel, S. (2011). Exploiting the Wisdom of Others to Make Better Decisions:
Suspending Judgment Reduces Egocentrism and Increases Accuracy, Journal of Behavioral Decision
Making, 2012; 25 (5) p 427–434
Lewis, M. Moneyball: The Art of Winning an Unfair Game. Barnes and Noble, 2003
Grove, W.M. Clinical Versus Statistical Prediction. Journal of Clinical Psychology, 2005; 61(10), p 1233–
1243
Got evidence?
 A decision based on the combination of different types
of evidence (experiential, organizational, scientific)
leads to better outcomes than a decision based on just
one source of evidence
-
-
Antman, E.M. et al, A comparison of results of meta-analyses of randomized control trials and
recommendations of clinical experts, JAMA, 1992: 268 (2) p 240 – 248
McNees, S.K. The Role of Judgment in Macroeconomic Forecasting Accuracy, International Journal of
Forecasting, 1990; 6 (3), p 28-299
Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London,
2012; p 286 and p 690
Tetlock, P. E. Expert Political Judgement, Princeton, NJ: Princeton University Press, 2006
Evidence-Base Practice
Post mortem analysis
Evidence-based perspective
NOT: Did they make the right decision?
BUT: Assessment of the decision making process
 What kind of evidence was there?
 Was this the best available evidence?
 Is there evidence from scientific research to
support (or call into question) the approach
taken?
Decision making process
Hospital, The Netherlands
Organization
 550 beds
 3300 employees
 210 medical specialists
 225,000 admissions
 Top Clinical & Teaching hospital
 Structure: Business Units
Decision: Leadership training
All managers:
 Board of directors
 Division managers
 Unit managers
 Head nurses
Cause I: reorganization
From:
20 business units - 20 senior managers - 40 supervisors
To:
20 business units – 8 senior managers - 40 supervisors
Cause I: reorganization
 Ambition
 Experience
 Education (MBA)
 Assessment
 Selection
Gap
Cause I: reorganization
larger span of control, more responsibilities = extra skills
They need
extra skills
We need
extra skills
Cause II: leadership climate
Employee / Job satisfaction: leadership
Cause II: leadership climate
Board of directors
Cause III: management development
Project team
 10 members
 Expertise (HR)
 Representation (nurses, doctors, managers)
 Support base
Process: months!
 Sessions with all stakeholders (medical staff, nurses,
managers, staff council)
What is leadership?
What kind of leadership does the OLVG need?
OLVG leadership vision
OLVG Leadership vision
 Passion and business
 Inspire and connect
 Leadership concerns us all
Selection procedure
Longlist: 30
 References (other hospitals, network), reputation
Shortlist: 5
 Academic, Educational, Training, HR Consulting, Wild card
First round: 3
 Paper pitch: proposal based on documents
Final round: 1
 Carousel: board, medical staff, head nurses, managers
Decision making process
Decision making process
1. What was the problem / issue
2. What kind of evidence was there?
3. Was this the best available evidence?
3 steps
1. Problem identification
2. Surfacing assumptions
3. Logic model
Step 1: What is the problem?
For which problem is ….. the solution?
- For who(m)
- Why?
- How big?
- How do we know (what is the evidence?)
Decision making process
1. What was the problem / issue
2. What kind of evidence was there?
3. Was this the best available evidence?
Step 2: What are the assumptions?
Assumptions are often hidden
3 steps
1. Problem identification
2. Surfacing assumptions
3. Logic model
Logic model
Decision making process
1. What was the problem / issue?
2. What kind of evidence was there?
3. Was this the best available evidence?
Best available evidence?
Best available
scientific evidence
Best available
experiential evidence
Decision
making process
Best available
organizational
evidence
Organizational values
and stakeholders’
concerns
Cause I: reorganization
larger span of control, more responsibilities = extra skills
They need
extra skills
We need
extra skills
Cause I: reorganization
larger span of control, more responsibilities = extra skills
 Finance?
 Risk management?
 Problem solving?
 Planning?
 Performance management?
 Change management?
 Business plan?
We need
extra skills
Cause II: leadership climate
Employee / Job satisfaction: leadership
Survey: Leadership
 I can always count on my manager when having job
related problems.
 I can always ask my manager for support
 The relationship with my manager is good.
 I have conflicts with my manager
 There is a good atmosphere between my manager
and me
always – often – sometimes - never
Survey: Leadership
 My manager inspires me to do the best in my job
 My manager sets a good example
 My manager makes me feel that my work is
meaningful
 My manager takes interest in my career
development.
always – often – sometimes - never
Survey: outcome ‘leadership’
 Favourable versus unfavourable
 Comparisson with 2011
 Benchmark with other divisions / units
 No baseline
 No confidence intervals
 No focus groups
 No correlations / regressions
 How did new managers score?
Cause II: leadership climate
Board of directors: strategy
Decision making process
1. Dus wat is het probleem?
2. Hoe zeker kunnen we er van zijn
dat leiderschapstraining de
oplossing voor ons probleem is?
3. Zijn er andere opties mogelijk?
Selection procedure
Longlist: 30
 References (other hospitals, network), reputation
Shortlist: 5
 Academic, Educational, Training, HR Consulting, Wild card
First round: 3
 Paper pitch: proposal based on documents
Final round: 1
 Carousel: board, medical staff, nurses, managers
leadership training: dm process
 No problem definition & not objectified
 No experiential evidence consulted
 No organizational evidence consulted
 Focus on shareholders’ concerns
 Selection of training companies based on
experience, recommendation or reputation
 No explicit selection criteria / procedure
Evidence based decision
Postgraduate School
?
Best available
scientific evidence
Best available
experiential evidence
✗
Decision
making process
✗
Best available
organizational
evidence
Organizational values
and stakeholders’
concerns
✓
Critically Appraised Topic
Question:
To what extent will leadership training improve the
effectiveness of the leaders (e.g. managers, head
nurses, physician executives) of the Amsterdam
Medical Centre? In addition, what are the
characteristics of effective leadership training
programs?
leadership training: scientific evidence
 15 meta analyses, 5 relevant
 37 (‘systematic’) reviews, 2 relevant
 Lots of relevant primary studies (by-catch)
leadership training: scientific evidence
 Overall (50 years): small (.2) to moderate (.4) effect sizes
 Little information on ROI (sometimes negative)
 Tailor made programs designed on the basis of an analysis
of tasks / skills are more effective and have a positive
impact on motivation
leadership training: scientific evidence
 Leadership trainings that focus on general management
skills and / or interpersonal / social skills show higher
effect sizes than those based on a specific leadership
‘model’
 The trainings should be of reasonable length (>3 days)
and repeated periodically to be effective
leadership training: scientific evidence
 The training should include opportunities to practice (in
vivo)
 A working climate that supports the ‘transfer’ to the
workplace (organization, supervisor, peers) motivates
employees to apply what they have learned.
Reactions
Denial
Anger
Bargaining
Acceptance
Who
knew?
Evidence based decision
Best available
scientific evidence
Best available
experiential evidence
Decision
making process
Best available
organizational
evidence
Organizational values and
stakeholders’ concerns
Think about it:
In the next weeks, before you make a
decision, ask yourself
 Did you ask the right questions?
 What is the evidence available?
 What is the quality of the evidence?
 Is it the best available evidence?