Postgraduate Course
6. Evidence based management:
What is the best available evidence?
5-step approach
Postgraduate Course
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critically appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results
Intermezzo
Postgraduate Course
Why are disciplines as such as
psychoanalysis, astrology and
parapsychology widely regarded as
pseudo-science?
Falsifiability
Postgraduate Course
“It is easy to obtain evidence in favor of virtually any theory,
but such ‘corroboration’ should count scientifically only if it
is the positive result of a genuinely ‘risky’ prediction, which
might conceivably have been false.
… A theory is scientific only if it is refutable
by a conceivable event. Every genuine test
of a scientific theory, then, is logically an
attempt to refute or to falsify it.”
Carl Popper
Falsifiability
Postgraduate Course
“Inspect every piece of pseudoscience and you will find a
security blanket, a thumb to suck, a skirt to hold.
What have we to offer in exchange? Uncertainty!
Insecurity!”
Isaac Asimov
Research designs
Postgraduate Course
What is the BEST car?
Research designs
Postgraduate Course
Which design for which question?
Which design for which question?
Postgraduate Course
Explanation
Postgraduate Course
Best research design?
Best available?
Postgraduate Course
Postgraduate Course
The best available evidence =
Studies with the highest internal validity
Studies with the highest external validity
Postgraduate Course
1. Best available evidence:
internal validity
Internal validity
Postgraduate Course
internal validity = indicates to what extent the
results of the research may be biased and is thus
a comment on the degree to which alternative
explanations for the outcome found are possible
(confounding).
Causality
Postgraduate Course
When do we know there is causal relation?
Three criteria:
1. the "cause" and the "effect" are related
2. the "cause" precedes the "effect" in time
3. there are no plausible alternative explanations for the
observed effect
Causality
Postgraduate Course
Considerations for research:
1. Are the "cause" and the "effect” related: effect size
INTERNAL VALIDITY
1. Does the "cause" precedes the "effect" in time:
before and after measurement
2. Are there no plausible alternative explanations for
the observed effect: randomization, control group
Internal validity
Postgraduate Course
internal validity = indicates to what extent the
results of the research may be biased and is
thus a comment on the degree to which
alternative explanations for the outcome found
are possible (confounding).
Methodological pitfalls
Postgraduate Course
 Bias
 Confounding
 Reverse Causation
Bias
Postgraduate Course
Bias: distortion of the outcome due to
systematic errors caused by the way the
study is designed or conducted.
NB: If bias is not taken into account then any
conclusions drawn may be wrong!
Forms of bias
Postgraduate Course
1. Selection bias
2. Information (detection) bias
3. Performance bias
4. Exclusion (attrition) bias
5. Publication bias
…
…
30. …..
Selection bias
Postgraduate Course
Error in the way participants in a study were selected.
Because of this comparison groups differ in measured or
unmeasured baseline characteristics.
Information bias
Postgraduate Course
Distortion of the outcome due to misinterpretation of information
or systematic errors in the the measurement of research
variables which leads to misclassification.
Information bias can be prevented by the use of standardized
measurement instruments, hard outcome measures, validated
questionnaires and objective, independent and blinded
assessors.
Types of information bias:
 Reporting bias (recall bias)
 Observer bias (interviewer bias, halo-effect)
Confounding
Postgraduate Course
Confounding is the idea that a 3rd variable can distort or
confuse (or confound..) a relationship between two other
variables. For instance, when factor X causes disease Y,
that relationship could be confounded by factor C that is
associated with both factor X and disease Y. C would be
an alternative explanation for the relationship observed
between X and Y.
What are the confounders?
Postgraduate Course
1.
Shoe size & quality of handwriting
2.
Body length & body weight
3.
Number of storks & birth rate
4.
Smoking youngsters & better lung function
Confounding
Postgraduate Course
Postgraduate Course
Correlation does not equal causation!
Postgraduate Course
http://kill-or-cure.heroku.com/
Reverse causation
Postgraduate Course
Reverse causation
Postgraduate Course
?
Charismatic leaders
Successful companies
Internal validity
Postgraduate Course
Cause and effect can be established
only through the proper research
design: no amount of statistical hand
waving can turn correlations into
conclusions about causation !!!
Levels of internal validity
Postgraduate Course
Levels of internal validity
Postgraduate Course
It is shown that …
It is likely that …
There are signs
that …
Experts are of the
opinion that …
Keep in mind!
Postgraduate Course
The levels of internal validity can only be used to determine
which type of research is the best method to assess the
validity of the cause-and-effect relationship that might exist
between an intervention (or moderator) and its outcomes.
In this respect, cross-sectional studies and case-studies have
the ‘weakest’ design. This of course doesn’t mean that
cross-sectional studies and case-studies have a weak design
overall. After all, different types of research questions
require different types of research designs. A case study for
instance is clearly a strong design for assessing why or in
which way an effect has occurred, but obviously not the
most suitable design for assessing the strength of a possible
cause-and-effect relationship.
Internal validity
Postgraduate Course
But … sometimes observational
studies are as good as RCT’s
When the size of effect is very large (swamps
the bias)
Internal validity
Postgraduate Course
These treatments have not been tested in RCTs:
are they supported by poor evidence?
Heimlich manoeuvre
Dehydration: drinking water
Cardiac arrest: AED
Postgraduate Course
2. Best available evidence:
external validity
External validity: generalizability
Postgraduate Course
Always ask yourself to what extent the evidence
is generalizable to your situation:
Ecological validity: Is your organization so different from
those in the study that its results may be difficult to apply?
Population validity: Is your population so different from
those in the study that its results may be difficult to apply?
Generalizability
Postgraduate Course
Same Population?
Same Intervention?
Same Comparison?
Same Outcome?
Same Context?
Generalizability
Postgraduate Course
Keep in mind:
What works in one narrowly defined setting
might not work in another,
but some psychological principles
are generalizable to all human beings.
Internal vs external validity
Postgraduate Course
All research designs are flawed – though each is flawed
differently. For instance, research designs with a high internal
validity, such as controlled studies, may be less suited to
generalization, which restricts their practical usability. Sample
surveys and field research, on the other hand, have lower
internal validity, but can sometimes be more useful for
management practice. So there is always a trade off between
internal validity (precision in control and measurements of
variables) and external validity (generalizability with respect
to populations, setting and context).
Best available evidence?
Postgraduate Course
internal validity
external validity
often high
often low
external validity
sometimes high
internal validity
external validity
often low
often …?