CAUSAL INFERENCE FROM
MULTIPLE STUDIES,
CRITICAL ASSESSMENT OF
ASSUMPTIONS
Presented by:
Bethany Harmon, Jenny Jackson, and Jill Pawlowski
H 615
November 22, 2013
Overview
● Generalized Causal Inference: Methods for Multiple
Studies (Ch. 13)
○
Multistudy programs of research
Summaries of multiple studies
○
○
○
Narrative reviews
Quantitative reviews
● Critical Assessment of Assumptions (Ch. 14)
● Reflection Activity (10 minutes)
Choosing a psychotherapist
• Imagine for a moment you are experiencing a difficult emotional time in
your life…
• Because you are an informed, intelligent person, you do some reading
on psychotherapy and discover that many different approaches are
available.
• These assorted styles of psychotherapy, although they stem from
different theories and employ different techniques, all share the same
basic goal:
to help you change your life in ways that make you a happier,
more productive, and effective person
http://www.edmondschools.net/Portals/3/docs/Terri_McGill/READ-TherapistChoice.pdf
Choosing a psychotherapist
• Which type of therapy should you choose?
• What you would really like to know is:
• Does psychotherapy really work?
• If it does, which method works best?
• While many comparison studies have been done, most of them:
• support the method used by the psychologists conducting the study
• were rather small
• are spread over a wide range of books and journals
• This makes a fully informed judgment extremely difficult.
What should you do?
You have 2 options
Generalized causal inference:
methods for multiple studies
• Single studies typically lack large,
heterogeneous samples across persons,
settings, times, treatments, and outcome
measures; rarely use diverse methodologies
• Multiple studies usually have greater
variation in all of the above = allows for
better tests of generalized causal inference
http://library.downstate.edu/EBM2/2100.htm
Generalizing from multiple studies
• How is this accomplished?
• Multistudy programs of research
• Summaries of multiple studies
Multistudy programs of research
• Phased models
• Same topic, same researcher/lab
• Research proceeds in phases, from basic research discovery to
application in real world settings
• Directed programs of experiments
• One researcher/one lab over time, or multiple researchers/multiple
labs at same time
• Aim is to systematically investigate over many experiments the
explanatory variables (moderators, mediators, constructs) that may
account for an effect, gradually refining generalizations
Multistudy programs of research
• Phased models
• Example: phases of cancer research
• Pre-phase: basic research on the effects of a treatment on cancer cells
•
•
•
•
•
in test tubes and on animals
Phase I: review existing basic and applied research to determine
testable hypothesis about new cancer agent
Phase II: method development to ensure accurate and valid research
procedures and technology are available to study the agent
Phase III: intervention efficacy trials
Phase IV: intervention effectiveness studies
Phase V: field studies aimed at entire communities to determine public
health impact
Multistudy programs of research
• Phased models
• Excel at exploring generalized causal inference
• Rely on purposive sampling of persons, settings, times, outcomes,
and treatments
• Address the 5 principles of causal explanation:
• Surface similarity: using human cancer cell lines to create comparable
•
•
•
•
tumors on animals
Ruling out irrelevancies: deliberately making patient characteristics
diverse
Making discriminations: identifying which kind of cancers are most/least
affected
Interpolation/extrapolation: varying drug doses to see how toxicity and
clinical response may vary
Causal explanation: developing models of how a drug acts on human
cancer cells so that the potency of that action can be increased
Multistudy programs of research
Advantages
• Control over aspects of
generalized causal inference
from study to study
• Pursuit of precisely the
questions that need answering
at any given time
Disadvantages
Summaries of multiple studies
• Same topic, different researchers
• Allows more precise estimates of effects than could be had from
single studies
• Better exploration of how causal relationships might change over
variation in study features
• Helps clarify nature of relationship, boundaries, behavior within those
boundaries, and explanations
• Wider knowledge base often results in more
credibility than claims based on single studies
Narrative reviews of experiments
• Describe existing literature using narrative descriptions without
attempting quantitative syntheses of study results
• Reviewer relies on statistical significance levels to draw conclusions
about intervention effectiveness
• Summary of votes suggests whether treatment works
• Also allows examination of potential moderators of the generalizability
of intervention effects
Narrative reviews combining experimental and
nonexperimental research
• Field experiments plus evidence from surveys, animal studies, and
basic laboratory studies
• Purpose is to match evidence across studies to a pattern that would
be consistent with and explain an effect and clarify its generalizability
• Useful when direct experimental manipulation of the treatment would be
unethical or impossible with humans
• Results can be controversial
Narrative reviews
Advantages
• Hypothesis generation
Disadvantages
• Difficult to keep straight all of the
• Thick description of a literature
• Theory development with
•
qualitative categories and
relationships among variables
•
•
•
relationships between effects and
potential moderators
Traditionally rely on box score
summaries of results from significance
tests of outcomes
Imprecise descriptions of study results
Even more complex when trying to
examine relationships among
outcomes and potential moderators
Overwhelming!
Quantitative reviews of existing research
• Glass (1976) coined the term meta-analysis
• AKA “Mega-Silliness”, “Blossoming Nose Blemish”, and “Classic of Social Science”
• http://garfield.library.upenn.edu/classics1983/A1983QF87200001.pdf
• Meta-analysis converts each study’s outcomes to a common
effect size metric, such that different outcomes have the same
means and standard deviations.
• The common effect size metric can then be more readily averaged
across studies
Essential steps of meta-analysis
• Problem identification and literature review
• Aim: develop clear research question and framework of
inclusion criteria for studies in the meta-analysis
• Studies included in review must address the question of interest and be
relevant to the treatments, units, settings, measures, and times outlined in
the guiding problem formation
• May also have methodological criteria
Essential steps of meta-analysis
• Coding of studies
• Meta-analyses use a common coding scheme to quantify study
characteristics and results
• Codes should reflect the researcher’s hypotheses
• For example, in a review of interventions for childhood obesity, codes might be
developed based on whether the intervention included…
• Behavioral, educational, environmental, diet, and/or physical activity features
• Parental or family involvement
• Outcome measures (e.g. BMI, change in diet or physical activity behaviors)
• Codes often include characteristics of the study report, participants,
intervention, intervention process, and methodology
Essential steps of meta-analysis
• Computing effect sizes
• Various study outcomes must be converted to a common metric before
outcomes can be meaningfully compared over studies
• Two most appropriate effect size measures for meta-analyses of experiments:
• Standardized mean difference statistic (d) – for continuous outcome measures
di = X t i – X c i
si
where Xti = mean of treatment group in ith study
Xci = mean of comparison group in ith study
si = pooled standard deviation of the two groups
• Odds ratio (o) – for dichotomous outcome measures
oi = AD
BC
Essential steps of meta-analysis
• Analyzing meta-analytic data
• Theoretically, meta-analytic effect sizes are analyzed just like any other
social or behavioral data, using both descriptive and inferential statistics
with univariate and multivariate techniques
• Unusual statistical features
• Desirability of weighting effect size estimates by a function of study sample size
• Use of tests for homogeneity of effect sizes
• Hierarchical nature of meta-analytic data
• Dependency of effect sizes within studies
• Presence of publication bias
Essential steps of meta-analysis
• Interpreting and presenting results
• Generally, interpreting and presenting meta-analytic results poses few
special problems
• Keep in mind: most meta-analytic data are correlational
• We never randomly assign studies to the categories analyzed
• Quasi-experimental design features to rule out threats to validity are rarely used
in meta-analysis
• Meta-analysts record data observationally
• Exception is for reviews of randomized experiments of the same intervention
Meta-analysis and the 5 principles of
generalized causal inference
• Surface similarity: assess the match between
research operations and the prototypical features
of the targets of generalization
• Multiple studies typically represent many more
constructs than any one study
• Greater chances of finding studies with operations
that reflect constructs that may be of particular policy
or research interest
• Reviews are superior to single studies in their
potential to better represent prototypical attributes
Meta-analysis and the 5 principles of
generalized causal inference
• Ruling out irrelevancies: consider deviations
from the prototype that might not change the
causal conclusion
• Must show that a given causal relationship holds
over irrelevant features present in multiple studies
• Some meta-analyses aim to decrease, not increase
heterogeneity of irrelevancies
• Others welcome heterogeneity to the extent that it
resembles that found in the settings of desired
application
Meta-analysis and the 5 principles of
generalized causal inference
• Making discriminations: demonstrate that an
inference holds only for the construct as specified, not
for some alternative or respecified construct
• Aim is to make discriminations about the parts of the
target persons, settings, measures, treatments, and time
for which a cause and effect relationship will not hold
• Meta-analysis helps to clarify discourse about important
constructs that have and have not been studied, and to
clarify the range and boundaries of causal relationships
Relevance and irrelevance are always subject to
revision over time
Meta-analysis and the 5 principles of
generalized causal inference
• Interpolation and extrapolation: specify the range of persons,
settings, treatments, outcomes, and times over which the cause
and effect relationship holds more or less strongly or not at all
• Requires exploration of the existing range of instances to discover how
effect sizes vary with position along the range
• The extremes are likely to be further apart and the intermediate values
more numerous than in a single study
• Response surface modeling works better with meta-analysis than with
single studies
Meta-analysis and the 5 principles of
generalized causal inference
• Causal explanation: developing and testing
explanatory theories about the target of
generalization
1.
Easier to break down persons, settings, treatments,
and outcomes into their component parts in order to
identify casually relevant components
2.
Multiple regression can be used to identify
redundancy among variables that moderate
outcomes
3.
Full explanation requires analysis of the
micromediating causal processes that take place
after a cause has been varied and before and effect
has occurred
Conclusion
• “Our enthusiasm for the methodology must be tempered by realistic
understandings of its limitations.” (Shadish et al, 2002; pg. 446)
• Flaws of meta-analysis are less problematic than those of narrative reviews
• Be no less or more critical of meta-analysis than of any other scientific method
Quasi-experimentation: Ruling out
Threats
• Relay heavily on researcher judgments especially on
plausibility
• Improve inferences by adding design elements
Quasi-experiments: Pattern matching
• Description
• Intended to rule out specific identified alternative causal pathways
• Problems
• Specificity of theory
• Chance
• More likely with more complex predictions
• Statistical tests for fit over mean differences
• not as well developed
Excuses Against RCT
• Problem: Quasi designs undermine likelihood of
conducting RCT
• RCT: Must be convinced of inferential-benefits over quasi
before incurring costs
• Role of disciplinary culture
• “Each study calls for the strongest possible design, not
the design of least resistance.” (pg. 487)
RCT Objection: Successful
Implementation
• Comparison not between well don RCT and Quasi but
between imperfect RCT and well done Quasi
• Counter:
• RCT’s may have better counterfactual even with treatment
degradation
• Even with attrition, RCT give better effect size estimates
• ** “Careful study and judgment to decide” (pg. 489)
RCT Objection: Strong Theory and
Standardized Treatment Implementation
• Intervention-as-implemented
• Loss of statistical power
• Large samples
• Reduce influence of extraneous variables
• Examine implementation quality
• Understanding of effectiveness in real world setttings
RCT Objection: Unfavorable Trade Offs?
• Critiques
• Conservative Standards
• Descriptive over explanatory
• Belated program evaluation
• Counters
• Based on artificial dichotomies, correctible problems, and
oversimplifications
RCT Objection: Invalid Model of Research
Utilization
• Critique
• Use of naïve rational choice model
• Counters
• Causal relationships explained with information from multiple
sources
• Contested rather than unanimous outcomes
• Short-term instrumental use with minor variants on existing
practices
• Research for conceptual changes
RCT Objection: Experiment and Policy
Conditions Differ
• Experiment outcomes affected by:
• Enthusiasm
• Implementation differs form real-world setting
• Random-assignment
• Counters
• Lack of generalizability is an empirical question
• Representation of real world conditions is a tradeoff
• Planning can improve similarities in populations
• True of any methodology
• Disruptions in implementation is true in locally invented novel programs
• Major Problem: Effects of policy can only be seen by studding the
entire event
RCT Objection: Imposing Treatment Vs.
Local Solutions
• Critique:
• Locally generated solutions produce better results
• Counters:
• Experiments were needed to find out that imposed treatment did
not work
• Null effects could be due to methodological weaknesses
• Distinguish between political-economic currency and intervention
effects
Causal Generation Theory
• Randomization allows for causation with few assumptions
• Follow 5-principles of generalized causal inferences when
randomization is not an option
• Researchers rely to heavily on weak purposive alternatives
• Causal generalization is more complicated than
establishing causal relationships
Non-experimental Alternatives
• Intensive causal studies
• Theory-based Evaluations
• Weaker quasi-experiments
• Statistical controls
Review of Key Terms
• How do we know what is true?
• Correspondence theory
• Coherence theory
• Pragmatism
Review of Key Terms
• Quine-Duhem thesis as a reaction to logical positivism
• An epistemology is a philosophy of the justifications for
knowledge claims.
• Naïve realism
• Epistemological realism
• Epistemological relativism
Shadish, Cook & Campbell on “Facts”
• Critical multiplism
• “Stubbornly replicable” observations by independent critics
• Building multiple theories into observations
• Dialectic process
• Example of psychotherapy
• Use and misuse of causal knowledge
Reflection
• What is the role of theory in the construction of
•
•
•
•
knowledge?
How will you choose your research question?
What do you do when you end up with non-significant
results?
If you have significant results, how can you be confident
in your findings?
How will you approach issues around use and misuse of
your findings?