Pit stop EBMgt
Evidence-Based Management
(Searching & Finding Scientific Evidence)
Pitstop Evidence-Based HR, VOV lerend netwerk,
Antwerpen, 11 januari 2013
Pit stop EBMgt
1. Evidence-Based Management
A (very) Short Introduction
Medicine: Founding fathers
Pit stop EBMgt
David Sackett
Gordon Guyatt
McMaster University Medical School, Canada
How it all started
Pit stop EBMgt
Problem I: too much information
Pit stop EBMgt
 More than 1 million articles in 40,000 medical journals per
year (= 1995; now probably more than 2 million). For a
specialist to keep up this means reading 25 articles every
day (for a GP more than 100!)
 Most of the new insights and treatment
methods don’t reach the target group
Problem I: too much information
Pit stop EBMgt
 HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR
manager to keep up this means reading 3 to 4 articles
every day (for a ‘general’ manager more than 50!)
Problem II: persistent convictions
Pit stop EBMgt
if you’re
hyperventilating
breathe into a bag
Problem III: jumping to conclusions
Pit stop EBMgt
elderly people who have
an irregular heartbeat are
much more likely to die of
coronary disease
give them a drug that
reduces the
number of
irregular beats
How 40,000 cardiologists can be wrong
Pit stop EBMgt
In the early 1980s newly introduced antiarrhythmic drugs were found to be highly
successful at suppressing arrhythmias.
Not until a RCT was performed was it realized
that, although these drugs suppressed
arrhythmias, they actually increased mortality.
The CAST trial revealed Excess mortality of
56/1000.
By the time the results of this trial were
published, at least 100,000 such patients had
been taking these drugs.
Pit stop EBMgt
David Sackett
 Half of what you learn in medical school will be
shown to be either dead wrong or out-of-date
within 5 years of your graduation; the trouble is that
nobody can tell you which half.
 The most important thing to learn is how to learn
on your own.
 (Remember that your teachers are as full of bullshit
as your parents)
Evidence-Based Practice
Pit stop EBMgt
1991
Medicine
1998
Education
1999
Social care
2000
Nursing
2000
Criminal justice
????
Management?
Pit stop EBMgt
2. Evidence based management:
What is it?
Definition
Pit stop EBMgt
Evidence based practice:
Improve information to support decision making
Definition
Pit stop EBMgt
Evidence-based management means making decisions
about the management of employees, teams or
organizations through the conscientious, explicit and
judicious use of four sources of information:
1.
The best available scientific evidence
2.
Organizational facts, metrics and characteristics
3.
Stakeholders’ values and concerns
4.
Practitioner expertise and judgment
Four sources
Pit stop EBMgt
Pit stop EBMgt
Evidence-based practice:
 Focuses on the decision making process
 Thinks in terms of probability (instead of
golden bullets).
Pit stop EBMgt
3. Evidence-based management:
Why do we need it?
Four sources
Pit stop EBMgt
Pit stop EBMgt
Trust me: 20 years of management experience!
Bounded rationality
Pit stop EBMgt
Bounded rationality
Pit stop EBMgt
System 1
 Fast
 Intuitive, associative
 heuristics & biases
System 2
 Slow (lazy)
 Deliberate, ‘reasoning’
 Rational
System 1: prone to biases
Pit stop EBMgt









Seeing order in randomness
Mental corner cutting
Misinterpretation of incomplete data
Halo effect
False consensus effect
Group think

Self serving bias

Sunk cost fallacy

Cognitive dissonance reduction 




Confirmation bias
Authority bias
Small numbers fallacy
In-group bias
Recall bias
Anchoring bias
Inaccurate covariation detection
Distortions due to plausibility
Pit stop EBMgt
2. Searching & Finding
The Best Available Scientific Evidence
Pit stop EBMgt
Step 1: Formulate a focused question
5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critical appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results
Asking the right question?
Pit stop EBMgt
Effect
vs
Non-effect
Types of questions: effect
Pit stop EBMgt
Does it work?
Does it work better than ....?
Does it have an effect on ....?
Effect
What is the success factor for ....?
What is required to make it work ...?
Will it do more good than harm?
Types of questions: non-effect
Pit stop EBMgt
Needs:
What do people want or need?
Attitude:
What do people think or feel?
Experience:
What are peoples’ experiences?
Prevalence:
How many / often do people / organizations ...?
Procedure:
How can we implement ...?
Process:
How does it work?
Explanation:
Why / how does it work?
Economics:
How much does it cost?
Focused question?
Pit stop EBMgt
 Does team-building work?
 What are the costs and benefits of self-steering teams?
 What are the success factors for culture change?
 Does management development improve the
performance of managers?
 Does employee participation prevent resistance to
change?
 How do employees feel about 360 degree feedback?
Foreground question?
Pit stop EBMgt
 Does team-building work?
 What is a ‘team’?
 What kind of teams?
 In what contexts/settings?
 What counts as ‘team-building’?
 What does ‘work’ mean?
 What outcomes are relevant?
 Over what time periods?
Answerable question: PICOC
Pit stop EBMgt
P = Population
I = Intervention (or success factor)
C = Comparison
O = Outcome
C = Context
Example: merger
Pit stop EBMgt
Answerable question: PICOC
Pit stop EBMgt
Imagine you are a consultant, your client is the board of
directors of a large Canadian health-care organization.
The board of directors has plans for a merger with a
smaller healthcare organization in a nearby town.
However, it’s been said that the organizational culture
differs widely between the two organizations. The board
of directors asks you if this culture-difference can impede
a successful outcome of a merger. Most of them
intuitively sense that cultural differences matter, but they
want an evidence-based advice.
Answerable question: PICOC
Pit stop EBMgt
What else would you like to know?
Answerable question: PICOC
Pit stop EBMgt
P: What kind of Population are we talking about? Middle managers,
back-office employees, medical staff, clerical staff?
O: What kind of Outcome are we aiming for? Employee productivity,
return on investment, profit margin, competitive position, innovation
power, market share, customer satisfaction?
P/C: And how is the assumed cultural difference assessed? Is it the
personal view of some managers or is it measured by a validated
instrument?
Answerable question: PICOC
Pit stop EBMgt
According to the board the objective of the merger is to
integrate the back-office of the two organizations (ICT,
finance, purchasing, facilities, personnel administration,
etc.) in order to create economy of scale. The front
offices and primary process of the two organizations will
remain separate.
The cultural difference is not objectively assessed (it is
the perception of the senior managers of both
organizations).
Answerable question: PICOC
Pit stop EBMgt
P = back office employees in a healthcare organisation
I = merger, integration back office
C = status quo
O = economy of scale
C = different organizational culture, unequal
Exercise
Pit stop EBMgt
1. Read the following five scenario’s
2. Formulate on the basis of each scenario a focused
question (use the PICOC format).
Scenario 2
Pit stop EBMgt
“Ik werk als manager bij een grote financiële dienstverlener. Het
concern is een holdingmaatschappij met daaronder een aantal divisies
met een eigen P&L verantwoordelijkheid. Het accountmanagement van
de divisies wordt aangestuurd op basis van een eigen set kritische
prestatie indicatoren (kpi’s) die niet onderling is afgestemd.
Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen
rijden bij gezamenlijke klanten en de omzet wordt ‘weggekaapt’ bij de
ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de
Executive Board van de holding een beroep gedaan op de
divisiedirecties om meer met elkaar te gaan samenwerken.
Mijn vraag is: staat het feit dat de divisies geen gedeelde kpi’s hebben
een goede interne samenwerking in de weg of hoeft dit geen probleem
te zijn?”
Scenario 3
Pit stop EBMgt
“I am a Dutch quality manager at a large international brewery. In
the past 4 years I facilitated a lean management program that is
running for more than 10 years in this organization. Since a
couple of years, however, we see a decline in results, such as:
 The number of problem analysis and improvement teams has
been increased, but the amount of breakdowns and short stops
in the production lines has not decreased.
 There is slow progress for the total program for the
Netherlands at this moment.
According to the middle managers the improvement program
does not address really important topics. I would therefore like to
know how our employees feel about lean management.”
Pit stop EBMgt
1. Neem een actueel probleem / vraag / issue dat speelt in de
organisatie waar je stage hebt gelopen (of een vraagstuk
dat je persoonlijke interesse heeft).
2. Formuleer op basis daarvan een onderzoeksvraag
3. Geef aan wat voor type vraag het is (effect of non-effect)
4. Benoem de PICOC elementen.
Pit stop EBMgt
Step 2: Search for the best available
evidence
5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1.
Formulate an answerable question (PICOC)
2.
Search for the best available evidence
3.
Critical appraise the quality of the found evidence
4.
Integrate the evidence with managerial expertise and
organizational concerns and apply
5.
Monitor and evaluate the results
Searching evidence
Pit stop EBMgt
What do we search?
Information sources
Pit stop EBMgt
Type of Information
Source
Current Information
Wall street Journal, Financial Times,
Business week, Financieel Dagblad
Overview of a subject
Textbooks and popular books
General background
Encyclopedias, yearbooks & book reviews
Academic Information
ABI/INFORM, Business Source Premier,
Emerald, PsychInfo, Science Direct
Statistical Information
CBS Statline, Eurostat
Theories about a subject
Textbooks and encyclopedias
Company information
Company Annual Reports, Datastream,
Factiva.com, Amadeus
Articles in peer reviewed journals
Pit stop EBMgt
Searching for evidence
Postgraduate Course
Databases
Postgraduate Course
 ABI/INFORM
 Business Source Elite
 PsycINFO
 Web of Knowledge
 ERIC
 Google Scholar
Searching for evidence
Postgraduate Course
Searching evidence
Postgraduate Course
How do we search?
Search Strategy
Search strategy
Pit stop EBMgt
Two types of search strategies
Snowball method
Building blocks method
Snowball method
Pit stop EBMgt
Starting from one book or article, you search
for other literature on the same topic.
 Snowballing to older publications by finding out which
publications were used by the author (see bibliography of
book or article).
 Snowballing to more recent publications by finding out
how often that book or article has been
cited by other authors (see Web of
Knowledge or Google Scholar).
Snowball method
Pit stop EBMgt
Building blocks method
Pit stop EBMgt
Keyword 1
Keyword 2
Keyword 3
Keyword 4
Synonyms or
Synonyms or
Synonyms or
Synonyms or
related terms
related terms
related terms
related terms
• ….
• ….
• ….
• ….
• ….
OR • ….
OR • ….
OR • ….
• ….
• ….
• ….
• ….
• ….
• ….
• ….
• ….
AND
AND
AND
Answerable question: PICOC
Pit stop EBMgt
P = back office employees, health care
I = merger, integration, back office
C = status quo
O = economy of scale
C = different organizational culture, unequal
1. Underline the keywords
2. Number the order of importance from 1-4
Answerable question: PICOC
Pit stop EBMgt
P = back office employees 5. healthcare
I = 1. merger, 3. integration, back office
C = status quo
O = 4. economy of scale
C = different 2. organizational culture, unequal
1. Underline the keywords
2. Number the order of importance
Search terms
Pit stop EBMgt
Operationalise your Pico elements!
O = long term profitability?
Construct validity
Pit stop EBMgt
The Ambidextrous Organization (??)
Select keywords
Pit stop EBMgt
The keywords of your PICOC may be enough.
If not, select more words by using:
 synonyms
 alternate spelling, translations
 related terms / words / subjects
 narrower or broader terms
corporate culture: organizational behavior/character, corporate identity
merger: acquisition, take-over, fusion, combination, unification
profitability: profit, advantage, return on investment, shareholder value
Search in the following order
Pit stop EBMgt
1. Thesaurus (SU)
2. Title (TI)
3. Abstract (AB)
4. Alternate spellings, synonyms
Search Query: an example
Pit stop EBMgt
I
I
O
C
1. Merger
3. Integration
4. Health care
organization
2. Corporate
culture
• Merger
• Healthcare
organization
• Integration
• Fusion
• Non profit
• Combination
• Take over
OR
OR
• Not for profit
• Corporate culture
• Organizational
behavior
OR • Organizational
• Acquisition
character
• Unification
• Corporate
•…
identity
AND
AND • Core beliefs
• Shared values
Boolean operators
Pit stop EBMgt
apples AND oranges
apples
oranges
apples OR oranges
apples
oranges
Start up
Pit stop EBMgt
Always search for indexed topics
or thesaurus terms
Pit stop EBMgt
thesaurus terms
Pit stop EBMgt
Combining results
Pit stop EBMgt
Use the history function to combine results
Combining results
Pit stop EBMgt
Use the history function to combine results
Narrow your results
Pit stop EBMgt
Narrow your results
Pit stop EBMgt
Save your results
Pit stop EBMgt
If an article is not available in ABI/ Inform or
BSE: try Google (advanced search)
Pit stop EBMgt
Summary
Pit stop EBMgt
1.
Formulate a focused question and break it down into keywords (PICOC),
2.
Underline the 2 to 4 most relevant keywords,
3.
Start with the most relevant keyword: is it listed in the thesaurus? If not: search in
the title or abstract.
4.
Repeat your search with alternative terms, synonyms or alternate spellings.
5.
Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative
terms) with OR (use the history function!)
6.
Repeat above steps for keyword 2
7.
Combine the outcome of your search with keyword 1 and 2 with AND (use the
history function!)
8.
Filter the articles with “studies” (subject > studies)
9.
Read the titles and abstracts of your results, select the relevant articles and safe
them.
10. No full text available in ABI/Inform or BS Elite? Try Google Advanced.
Pit stop EBMgt
Learning through play !
 Try all buttons
 Make lots of mistakes
 Have fun !
Assignment: search strategy
Pit stop EBMgt
Search for research articles on teambuilding
Exercise: Search for evidence
Pit stop EBMgt
2. Search in ABI/Inform:
1. How many articles has Stephen Covey published in peer reviewed journals?
2. How many of these articles are based on scientific research?
3. Are there articles (by other authors) that are critical of Covey’s 7 Habits?
4. How many of these critical articles are based on scientific research?
Pit stop EBMgt
Evidence-based management
Step 3: Critical appraisal of studies
5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critically appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results
Pit stop EBMgt
Effect
vs
Non-effect
Pit stop EBMgt
Internal validity
Internal validity
Pit stop EBMgt
internal validity = indicates to what extent the
results of the research may be biased and is thus
a comment on the degree to which alternative
explanations for the outcome found are possible.
Levels of internal validity
Pit stop EBMgt
Which design for which
question?
Pit stop EBMgt
Explanation
Pit stop EBMgt
Best research design?
Critical appraisal
Pit stop EBMgt
Critical appraisal
Pit stop EBMgt
Postgraduate
Course
1. Is the study design appropriate to the stated aims?
2. Was a control group used?
3. Was a pretest used?
4. Are the measurements likely to be valid and reliable?
5. Could bias or confounding have occurred?
6. How large was the effect size?
Levels of internal validity
Pit stop EBMgt
1. Were there enough subjects in the study?
2. Was a control group used?
3. Were the subjects randomly assigned?
4. Was a pretest used?
5. Was the study started prior to the intervention or event?
6. Was the outcome measured in an objective and reliable way?
6x yes
= very high (A)
5x yes
= high (A)
4-3x yes
= limited (B)
2x yes
= low (C)
1-0x yes
= very low (D)