PDW, Annual AOM 2012, Boston
Postgraduate Course
Evidence-Based Management:
Three New Approaches to Teaching the
Practice of Management
Denise
Rousseau
Blake
Jelley
Wendy
Carroll
Eric
Barends
PDW, Annual AOM 2012, Boston
Postgraduate Course
1. Denise Rousseau: Introduction
2. Wendy Carroll & Blake Jelley: Push Approach
Subgroups
3. Eric Barends: Pull approach
4. Video (9 min)
Subgroups
5. Denise Rousseau: Process Approach
Definition
Postgraduate Course
Evidence-based management means making decisions
about the management of employees, teams or
organizations through the conscientious, explicit and
judicious use of four sources of information:
1.
The best available scientific evidence
2.
Organizational facts, metrics and characteristics
3.
Stakeholders’ values and concerns
4.
Practitioner expertise and judgment
Four sources
Postgraduate Course
Barends, Rousseau, Carroll, & Jelley
2012 Academy of Management PDW
Evidence-Based Management: Three New Approaches to Teaching the Practice of Management
Teaching EBMgt:
The “Push” Approach
R. Blake Jelley & Wendy R. Carroll
Overview
• Our Perspectives and Context
• Importance of the “Push” Approach
• Principles and Resources
Our Perspectives and Context
• Our backgrounds
– Education
– Applied experiences
• Teaching in the UPEI School of Business
– Undergraduate
– EMBA (launched 2008)
• Oxford Handbook of EBMgt chapter
– Jelley, Carroll, & Rousseau (2012). Reflections on teaching evidencebased management.
• Less about the “push” approach
Importance of the “Push” Approach
• Bounded Rationality, Heuristics, Biases
• Kahneman (2011). Thinking, Fast and Slow
– “System 1” (fast; automatic)
– “System 2” (slow; deliberate)
• See also Kahneman & Klein (2009).
Conditions for Intuitive Expertise: A Failure to Disagree
(2009, American Psychologist)
Importance of the “Push” Approach
• A path toward the development of expertise in management?
• Developing expert skill and intuition (see Kahneman & Klein, 2009;
Kahneman, 2011)
– A sufficiently regular, predictable environment
– Opportunities to learn regularities through prolonged
practice and feedback
• The management domain is not highly favourable to skilled
intuition
– Intuition is an important consideration, not the final word
– Managers need to avoid overconfidence in intuition
Importance of the “Push” Approach
• System 1 will engage!
• Expert intuition is not magic…
– “You can feel [Herbert] Simon’s impatience with the
mythologizing of expert intuition when he writes: ‘The
situation has provided a cue; this cue has given the expert
access to information stored in memory, and the information
provides the answer. Intuition is nothing more and nothing
less than recognition’” (Kahneman, 2011, p. 11).
Importance of the “Push” Approach
• Making intuition more friendly to EBMgt.
– “You do as much homework as possible beforehand so that the
intuition is as informed as it can be” (Kahneman, In Kahneman &
Klein, 2010, McKinsey Quarterly).
– “It is easier to make good decisions quickly if managers are
educated and evidence savvy” (John Zanardelli, 2012, p. 196;
President & CEO, Ashbury Heights).
– Program System 1 with evidence-based principles.
• Think fast, well, and set triggers for System 2.
• Bolster, not replace, more deliberate processing.
Importance of the “Push” Approach
• Practitioners are not well-informed about managementrelated knowledge
– E.g., Senior SHRM members = 57% correct (Rynes et al., 2002)
• Are educators much better?
• Various ways to “push” EB knowledge.
– Management education as a key.
• Also, ME can integrate push, pull, and process approaches
Principles and Resources
• Use of Diagnostic Quizzes
– Examples…
• HRM
(Rynes et al., 2002)
• “100 things… & 50 more things you need to know” books
• Advertising (Armstrong & Green’s adprin.com)
– Discussion of dissemination vs. exposing students as
uninformed
– Links to critical thinking and the “pull” approach
Principles and Resources
• Concerns about what and how we teach… and who does the
teaching
– Our body of knowledge
•
•
•
•
Benefits of systematic research
Volume of research
Focus on novelty over integration, etc.
Pluralism
– Textbooks
– Instructors
– Teaching methods
Principles and Resources
Other References:
•
Existing research syntheses;
•
Individual synthesis and translation articles
–
•
•
E.g., Allen, Bryant, & Vardaman (2010). Retaining Talent:
Replacing Misconceptions with Evidence-based Strategies.
AOM Perspectives [Best Paper]
SHRM Effective Practice Guidelines;
SHRM-SIOP’s new collaborative series.
Principles and Resources
• Identify and teach the “core” management body of
knowledge; less content, more practice
• Focus on topics, theories, and principles that:
– (a) Have a solid evidence-base
– (b) Are practical to apply
• Are role-relevant
• Have implications for practice; address important practice issues
• Involve procedural as well as declarative knowledge
– (c) Are durable
• Over time
• Applicable in various situations
(Miner, 2003; Rousseau & McCarthy, 2007)
Since we can’t teach everything,
what are the most important evidencebased things we need to program into
our students?
References
Allen, D. G., Bryant, P. C., & Vardaman, J. M. (2010). Retaining talent: Replacing misconceptions with evidence-based strategies.
Academy of Management Perspectives, 24(2), 48-64.
Armstrong, J. S., & Green, K. C. (2012). Advertising principles: Evidence-based knowledge on persuasion through advertising.
Retrieved from http://advertisingprinciples.com/ [see http://advertisingprinciples.com/en/try/test-your-advertising-iq]
Charlier, S. D., Brown, K. G., & Rynes, S. L. (2011). Teaching evidence-based management in MBA programs: What evidence is
there? Academy of Management Learning & Education, 10(2), 222-236.
Eichinger, R. W., Lombardo, M. M., & Ulrich, D. (2004). 100 things you need to know: Best people practices for managers & HR
(Vol. 1). Minneapolis, MN: Lominger.
Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar, Strauss, Giroux.
Kahneman, D. & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515-526.
Kahneman, D. & Klein, G. (2010). When can you trust your gut? McKinsey Quarterly, Issue 2, 58-67.
Jelley, R. B., Carroll, W. R., & Rousseau, D. M. (2012). Reflections on teaching evidence-based management. In D. M. Rousseau
(Ed.), The Oxford Handbook of Evidence-Based Management (pp. 337-355). New York: Oxford University Press.
Latham, G. P. (2009). Becoming the evidence-based manager: Making the science of management work for you. Boston: DaviesBlack.
Locke, E. A. (2009). Handbook of principles of organizational behavior: Indispensable knowledge for evidence-based management
(2nd ed.). Chichester, UK: Wiley.
Miner, J. B. (2003). The rated importance, scientific validity, and practical usefulness of organizational behavior theories: A
quantitative review. Academy of Management Learning & Education, 2(3), 250-268.
Pearce, J. L. (2009). Organizational behavior: Real research for real managers. Irvine, CA: Melvin & Leigh.
References
Pearce, J. L. (2012). Creating evidence-based management textbooks. In D. M. Rousseau (Ed.), The Oxford Handbook of EvidenceBased Management (pp. 377-386). New York: Oxford University Press.
Pfeffer, J., & Sutton, R. I. (2006). Hard facts, dangerous half-truths, and total nonsense. Boston, MA: Harvard Business School
Press.
Rousseau, D. M. (Ed.) (2012). The Oxford Handbook of Evidence-Based Management. New York: Oxford University Press.
Rousseau, D. M. (2012). Designing a better business school: Channelling Herbert Simon, addressing the critics, and developing
actionable knowledge for professionalizing managers. Journal of Management Studies, 49(3), 600-618.
Rousseau, D. M., & McCarthy, S. (2007). Educating managers from an evidence-based perspective. Academy of Management
Learning & Education, 6, 84–101.
Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals’ beliefs about effective human resource practices:
Correspondence between research and practice. Human Resource Management, 41(2), 149–174.
Society for Human Resource Management Foundation (2012). Effective practice guidelines series.
http://www.shrm.org/about/foundation/products/pages/default.aspx
Society for Industrial and Organizational Psychology & Society for Human Resource Management (2012). Publication and
dissemination of science to practice: A research collaboration between the Society for Human Resource Management (SHRM)
and the Society for Industrial and Organizational Psychology (SIOP). http://www.siop.org/SIOP-SHRM%5Cdefault.aspx
Ulrich, D., Eichinger, R., Kulas, J., & De Meuse, K. (2007). 50 more things you need to know: The science behind best people
practices for managers & HR professionals (Vol. 2). Minneapolis, MN: Lominger.
Zanardelli, J. (2012). At the intersection of the academy and practice at Ashbury Heights. In D. M. Rousseau (Ed.), The Oxford
Handbook of Evidence-Based Management (pp. 191-197). New York: Oxford University Press.
PDW, Annual AOM 2012, Boston
Postgraduate Course
Evidence-Based Management:
Three New Approaches to Teaching the Practice of Management
Part 2:
The 5-step pull approach
Eric Barends
Postgraduate Course
Trust me, I’m a manager.
Push vs Pull
Postgraduate Course
Push: teaching management principles
based upon a convergent body of
research and telling students what to do.
Pull: teaching students how to find,
appraise and apply the outcome of
research (evidence) by themselves
Postgraduate Course
Why do we need a
pull approach?
Problem I: too much ‘evidence’
Postgraduate Course
 HRM: 1,400 articles in 2011 (ABI/INFORM). For an HR
manager to keep up this means reading 3 to 4 articles
every day (for a ‘general’ manager more than 50!)
Problem II: false information
Postgraduate Course
 Half of what you learn will be shown to be either dead
wrong or out-of-date within 7 years of your graduation;
the trouble is that nobody can tell you which half
Problem III: half time value
Postgraduate Course
5 years? 7 years? 10 years?
Pull
Postgraduate Course
Pull: teaching students how to find, appraise
and apply evidence by themselves
Starting point
Postgraduate Course
Start with a practical question,
(not with an academic answer)
 Problem based
 Real live case
 Just in time
The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate a focused question (Ask)
2. Search for the best available evidence (Acquire)
3. Critically appraise the evidence (Appraise)
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
(Apply)
5. Monitor the outcome (Assess)
Postgraduate Course
1. Formulate a focused question
Asking the right question?
Postgraduate Course
 Does team-building work?
 Does the introduction of self-steering teams work?
 Does management development improve the
performance of managers?
 Does employee participation prevent resistance to
change?
 Is 360 degree feedback effective?
Focused question?
Postgraduate Course
 Does team-building work?
 What is a ‘team’?
 What kind of team?
 In what contexts/ settings?
 What counts as ‘team-building’?
 What does ‘work’ mean?
 What outcomes are relevant?
 Over what time periods?
Answerable question: PICOC
Postgraduate Course
P = Population
I = Intervention or success factor
C = Comparison
O = Outcome
C = Context
Focused question: PICOC
Postgraduate Course
P = Population
I = Intervention or successfactor
C = Comparison
O = Outcome
C = Context
 Employee productivity?
 Job satisfaction?
 Return on investment?
 Market share?
 Organizational commitment?
Postgraduate Course
2. Finding the best available evidence
Searching evidence
Postgraduate Course
Where do we search?
Databases
Postgraduate Course
 ABI/INFORM
 Business Source Elite
 PsycINFO
 Web of Knowledge
 ERIC
 Google Scholar
Searching evidence
Postgraduate Course
How do we search?
Search Strategy
Search strategy
Postgraduate Course
Two types of search strategies
Snowball method
Building blocks method
Hands on instruction
Postgraduate Course
Postgraduate Course
3. Critical appraisal of studies
Making sense of evidence
Postgraduate Course
The best available evidence =
Studies with the highest internal validity
(does it work?)
Studies with the highest external validity
(does it work for my employees / my organization?)
Research designs
Postgraduate Course
Which study for which question?
The “best” evidence depends on the question type !
Which design for which
question?
Postgraduate Course
Explanation
Postgraduate Course
Best research design?
Critical appraisal
Postgraduate Course
Critical appraisal
Postgraduate Course
1. Is the study design appropriate to the stated aims?
2. Was a control group used?
3. Was a pretest used?
4. Are the measurements likely to be valid and reliable?
5. Could bias or confounding have occurred?
6. How large was the effect size?
Postgraduate Course
Step 4: Turning evidence into
practice
Organization concerns
Postgraduate Course
Always ask yourself to what extent the evidence
is applicable in your situation:
1. Is your organization / population so different from those in the
study that its results are difficult to apply?
2. How relevant is the study (or outcome) to what you are
seeking to understand or decide?
3. What are your organization’s potential benefits and harms from
the intervention?
4. Is the intervention feasible in your setting?
Four sources
Postgraduate Course
Feasible?
Postgraduate Course
 organizational facts and characteristics
 cultural aspects
 stakeholders’ values and concerns
 political aspects
 financial aspects /cost-effectiveness / ROI
 priorities
 change readiness / resistance to change
 implementation capacity
 timing
Exercises / Assignments
Postgraduate Course
 Popular management book / guided field trip
 Surfacing assumptions
 Needle & haystack assignment
 Myth busters, snake oil symposium
 Find the Flaws
 Persuasive paper / presentation
 CAT
CAT: Critically Appraised Topic
Postgraduate Course
CAT: Critically Appraised Topic
Postgraduate Course
A critically appraised topic (CAT) is a
structured, short (2 pages max) summary
of evidence on a topic of interest, usually
focused around a practical problem or
question..
CAT: structure
Postgraduate Course
1) Background / context
2) Question (PICOC)
3) Search strategy
4) Results / evidence summary
5) Findings
6) Limitations
7) Recommendation
CAT-walk
Postgraduate Course
CAT: example
Postgraduate Course
See: www.cebma.org/presentations
(CAT Organizational Trust and Job Satisfaction)
Postgraduate Course
What are the skills that are manifest in
this video that are relevant to what you
are trying to teach at your classes?
Denise M. Rousseau
Barends, Rousseau, Carroll, & Jelley
2012 Academy of Management PDW
Evidence-Based Management: Three New Approaches to Teaching the Practice of Management



Bounded Rationality
The Small Numbers
Problem of Individual
Experience
Prone to See Patterns
Even in Random Data
The “Human” Problem

Critical Thinking

Decision Supports

Research
• Large Ns > individual
experience
• Controls reduce bias
Evidence-Based Practice



Bounded Rationality
The Small Numbers
Problem of Individual
Experience
Prone to See Patterns
Even in Random Data
The “Human” Problem

Critical Thinking

Decision Supports

Research
• Large Ns > individual
experience
• Controls reduce bias
Evidence-Based Practice
Get
critical “evidence” in advance
• Prime your KSAs
Make
the right decision as fast as needed
• Not necessarily “as fast as possible”
best (evidence-based) strategies for
different decision types
Learn
• Identify the type of decision you face
• Then, engage the right decision strategy

Routine decisions for which there is a “best evidencebased way” to do things
• Hiring call center workers, management trainees
• Giving periodic performance feedback
• Running a geographically distributed meeting





Acquire science-based evidence and org facts to identify
effective practices
Develop standard operating procedures with users
Gather org-evidence to evaluate SOP effects
Modify as needed
Put in user-friendly form (Checklist, Diagram)

Non-routine Decisions (# Stakeholders & Goals)
• New facility start up
• Solving space problem in existing building
• Developing a company-wide performance management system

Evidence-Based Pull Approach

Yates’s Cardinal Rules
Note: What is non-routine to one organization may be
routine in another (e.g., new store start ups are routine
in McDonald’s )
Beneficiary Interests/Values
A Big Picture
Decision
Other
Contributors
Decision Processes: Cardinal Issue Resolution
Preliminaries
1—Need
2—Mode
3—Investment
Core
4—Options
5—Possibilities
6—Judgment
7—Value
8—Tradeoffs
Aftermath
9—Acceptability
10—Implementation
Resolution Contributors: E.g., Resources, Tools, Biology,
Habits, “Natural” Experiences, Training, History, Culture
 Hypercomplex Decision with High Risk and Many
Unknowns (i.e. Black Swans)
 Use Sensemaking
 Weick and Sutcliffe’s Resilience Process
•
•
•
•
•
Gather information and check assumptions
Run experiments (in parallel if several alternatives are identified)
Multiple trials to learn by doing
Build on small wins
Continue to question assumptions

Routine Decisions
• Atul Gawande

Novel Decisions (due # Stakeholders & Goals)
• Frank Yates

Hypercomplex with Many Unknowns/Risky Decisions
• Karl Weick & Kathleen Sutcliffe

What type of decision situation do you face?

DIAGNOSIS: Appropriate decision strategy?

Product of this is critical thinking that
overtime helps you become more aware of
assumptions and gaps in logic

Heath, C., Larrick, R. P., & Klayman, J. (1998) Cognitive repairs: How organizational practices can
compensate for individual shortcomings. Review of Organizational Behavior, 20, 1–38.

Gawande, A. (2009). Checklist manifesto: How to get things right. New York: Henry Holt.

Larrick, R.K. (2009/) Broaden the decision frame to make effective decisions. In E.A. Locke (ed.),
Handbook of principles of organizational behavior: Indispensable knowledge for evidence-based
management. New York: Wiley (pp. 461-515).

Taleb, N. N. (2010). The black swan: The impact of the highly improbable. (2nd ed.) New York:: Penguin.

Weick, K.E, & Sutcliffe, K. (2007). Managing the Unexpected: Resilient Performance in an Age of
Uncertainty. New York: Wiley.

Yates, J. F. (2003). Decision management. San Francisco: Jossey-Bass.

Yates, J.F. & Potwoworski, G. (2012). Evidence-based decision management. In D,M, Rousseau (ed):
Handbook of Evidence-Based Management: New York: Oxford University Press, this volume.