Evidence-Based
Practice
A New Approach of Teaching
The Practice of Management
Eric Barends, CEBMa – Denise Rousseau, CMU
May 6th, 2014
Who are we and what is our mission (today)?
Denise Rousseau
Eric Barends
•
•
•
•
•
•
•
•
researcher
teacher
professor
CMU / Heinz / Tepper
manager
teacher
director
CEBMa
Mission Today
• Teaching Evidence-based practice (in 1 hour)
• Disappointment
• Inspiration
Postgraduate Course
Postgraduate Course
Postgraduate Course
Current developments
Postgraduate Course
 Education
 Access to research databases
 Rapid Evidence Assessments
 Building a community
Future developments: practice
Postgraduate Course
 CEBMa Database of Evidence Summaries
 Online learning modules
 Accreditational bodies
Today
1. Some background
2. Teaching: learning principles
3. Teaching: curriculum
4. Teaching: examples
5. Your questions
1. Some background
Evidence based management:
What is it?
Evidence-based practice
Central Premise:
Decisions should be based on a
combination of critical thinking
and the ‘best available evidence‘.
Evidence?
findings from scientific research,
organizational facts & figures,
benchmarking, best practices,
professional experience
16
All managers base their
decisions on ‘evidence’
17
But…many managers pay
little or no attention to the
quality of the evidence they
base their decisions on
18
Trust me, 20 years of
management experience
19
SO ...
20
Teach managers how to
critically evaluate the
trustworthiness of evidence
from multiple sources
and help them find ‘the best available’ evidence
An example
An example
An example
An example
Maslow, A.H. "A Theory of Human Motivation”
An example
An example
Scientific
research
findings
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
Organizational data,
facts and figures
Stakeholders’ values
and concerns
Professional experience and judgment
Scientific research findings
Maslow, A.H. (1943). "A Theory of Human Motivation," Psychological Review 50(4)
Wahba, M. A., & Bridwell, L. G. (1976). Maslow reconsidered: A review of
research on the need hierarchy theory. Organizational Behavior and Human
Performance, 15(2)
How evidence-based are we (managers)?
“I’ve never thought I need more
evidence before making a decision;
I know what needs to be done, we
get on with it and we get results.”
True (likely) or false (not likely)?
1. Incompetent people benefit more from feedback than
highly competent people.
2. Task conflict improves work group performance while
relational conflict harms it.
3. Encouraging employees to participate in decision
making is more effective for improving organizational
performance than setting performance goals.
How evidence-based are we?
HR Professionals' beliefs about effective human resource practices: correspondence
between research and practice, (Rynes et al, 2002, Sanders et al 2008)
 959 (US) + 626 (Dutch) HR professionals
 35 statements, based on an extensive body of evidence
 true / false / uncertain
Outcome: not better than random chance
EBP:
Teach managers how to
critically evaluate the
trustworthiness of evidence
from multiple sources
and help them find ‘the best available’ evidence
2. Teaching EBP:
Learning principles
Discuss with your neighbours:
When it comes to teaching,
what are important learning principles?
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
56
57
58
59
60
0
1
2
3
4
5
6
7
8
9
Learning principles
1. The brain needs time to get used to new things.
Longer periods in between practice sessions lead to
a longer overall retention.
2. Training program should include opportunities for
practice, linked to real world situations .
3. The human brain seeks comfort in what it knows
and is familiar with; addressing / confronting prior
knowledge increases understanding (start from
where the students are)
Learning principles
4. Higher order thinking only happens when people
work on questions / problems / issues themselves.
5. Learning from failure is important for learning.
Speculating and predicting before finding the
correct answers helps people become adaptive
learners / experts.
Teaching EBP =
 Small groups
 Problem based
 Real life cases
EBP starts with a practical question,
not with an academic answer
3. Teaching EBP:
Curriculum
EBP: Curriculum
 Limitations of human judgment & common forms of cognitive bias
 Retrieving & critically assessing experiential evidence
 Retrieving & critically assessing organizational evidence (qualitative
& quantitative)
 Searching in research databases
 Efficiently reading research articles
 Critically appraising evidence from research
 Weighing and aggregating evidence from multiple sources
 Incorporating evidence into the decision making process
 Different types of decisions and decision-making processes
 Assessing the outcome of decisions made
Scientific
research
findings
Professional
experience and
judgment
Ask
Acquire
Appraise
Aggregate
Apply
Assess
Organizational data,
facts and figures
Stakeholders’ values
and concerns
EBP: outcome
 A critical and reflective attitude
 Skills to distinguish trustworthy from less
trustworthy evidence.
 Thinking in terms of probabilities
CAT: Critically Appraised Topic
CAT: Critically Appraised Topic
A critically appraised topic (CAT) is a
structured, short (2 – 5 pages max) summary
of evidence on a topic of interest, focused
around a practical problem or question..
CAT: structure
1) Background / context
2) Question (PICOC)
3) Search strategy
4) Results / evidence summary
5) Findings
6) Limitations
7) Recommendation
CAT: 3 hits
1. Small group (2-3), scientific evidence, list
of topics
2. Individual, scientific evidence & theory,
own topic
3. Individual, evidence from multiple
sources, real life question / issue
CAT: examples
 Is there a valid and reliable way to measure the productivity of
knowledge workers?
 To what extent will leadership training improve the
effectiveness of leaders (e.g. managers, executives), what are
the characteristics of effective leadership training programs?
 What research evaluating the effects of 360-degree has been
published in peer-reviewed scholarly journals in the past 10
years? Which of the variables that are expected to have an
impact on the effectiveness of 360-degree feedback are most
widely studied and what is known of their effect?
CAT-walk
4. Teaching EBP:
Examples
4. Teaching EBP: examples
 Critical attitude
 ASK
 Critically appraisal: research findings
 Organizational evidence
4. Teaching EBP: examples
 Critical attitude
 ASK
 Critically appraisal: research findings
 Organizational evidence
Discuss with your neighbours:
What are the most common cognitive
biases in management?
(give an example)
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
56
57
58
59
60
0
1
2
3
4
5
6
7
8
9
Confirmation bias
We are predisposed to selectively
search for or interpret information in
ways that confirms our existing beliefs,
expectations and assumptions, and
ignore information to the contrary.
In other words, we “see what we want to
see”
Increase of information
The amount of
information increases
faster than our ability to
process it.
McKinsey (1997 / 2001)
Case study /
best practice
War on Talent
Biases
1. Pattern recognition
2. Confirmation-bias
3. Small numbers fallacy
4. Outcome bias
5. Halo effect
6. Authority-bias
7. Groupthink
8. Availability bias
Assignment
One of your best friends is trader on the stock exchange. He
enthusiastically tells you he has analyzed a large number of
financial and economic data and that he has discovered an
interesting phenomenon: "The position of the Dow Jones
index multiplied by the price of oil is two days ahead of the
gold price!" In other words, if both the Dow Jones and the oil
price go up, the price of gold will rise within a day.
 Bias?
 Critical questions?
Bias: pattern recognition
We are predisposed to see order, pattern and causal
relations in the world.
Patternicity: The tendency to find meaningful patterns in
both meaningful and meaningless noise.
Assignment
doctor A
Last operation:
patient died on the
operating table
Most people will prefer doctor B
 Why? Bias?
 Critical questions?
doctor B
Last operation:
patient recovered
faster than expected
Bias: Outcome bias
Outcome bias
We are inclined to evaluate the
quality of a decision (intervention,
method) on the basis of its
outcome.
The Asch Experiment
Nasa, Challenger
Bias: Groupthink
Groupthink:
Groupthink is a psychological
phenomenon that occurs within a
group of people, in which the desire
for harmony or conformity in the
group results in an incorrect or
irrational decision
Bounded rationality
System 1
 Fast
 Intuitive, associative
 heuristics & biases
System 2
 Slow (lazy)
 Deliberate, ‘reasoning’
 Rational
Bounded rationality
limbic system and
brainstem
(system 1)
neo cortex
(system 2)
Teaching EBP: examples
 Critical attitude
 ASK
 Critically appraisal: research findings
 Organizational evidence
5 steps
Postgraduate Course
1. Problem identification
2. Surfacing assumptions
3. Logic model
4. Formulating a searchable question
5. Formulate sub-questions
Step 1: What is the problem?
Postgraduate Course
For which problem is ….. the solution?
 For who(m)
 Why
 How big ?
 How do we know (what is the evidence?)
The problem
Postgraduate Course
Our hospital
“Door to needle time”, 48 UK hospitals in Westmidland
Postgraduate Course
Discuss with your neighbours
possible causes for these
differences that could not be
solved by the implementation of
Lean / Six Sigma
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
56
57
58
59
60
0
1
2
3
4
5
6
7
8
9
5 steps
Postgraduate Course
1. Problem identification
2. Surfacing assumptions
3. Logic model
4. Formulating a searchable question
5. Formulate sub-questions
Logic model
Postgraduate Course
Logic model
Postgraduate Course
A logic model spells out the process by which a
problem or intervention is expected to produce
certain outcomes.
In making expectations (its logic) explicit, a logic
model helps identify the kind of evidence
needed.
4. Teaching EBP: examples
 Critical attitude
 ASK
 Critically appraisal: research findings
 Organizational evidence
Critical appraisal
Always start with a practical question:
- does it work?
- how many employees …?
- how do employees feel about …?
How could we find out?
87
Critical appraisal
Amanda Burls:
“I never tell them anything about randomization or blinding, I
just ask: “How are you going to know? How would you test
this if I would give you a half million dollars to test it?” And
when they come up with a suggestion I say, “Alright, can you
think of any reason you got the results showing it works,
while in fact it doesn’t.” And they say, “Well, it could be this, it
could be that” And then I say, “Ok, then redesign your study
so it can’t be this,” and what they come up with are precisely
those things: randomization and blinding.”
Critical appraisal
How trustworthy is this study?
 Two studies, different designs
 Studies with methodological flaws
 Best available evidence (so what now?)
 Online course!
4. Teaching EBP: examples
 Critical attitude
 ASK
 Critically appraisal: research findings
 Organizational evidence
91
Organizational facts and figures
Organizational facts and figures
Organizational facts and figures
95
Organizational facts and figures
Organizational facts and figures
Examples
Organizational evidence
1. Added value
1. Types of organizational evidence
1. Starting point: logic model
2. Statistical aspects to consider
3. Barriers to overcome
5. Barriers to overcome
1. Small numbers problem
2. Measurement errors
3. Context
4. Tainted data
5. Politics