Executive Summary
Causality Beyond Simple RCTs
Health services research (HSR) addresses many causal questions, including comparative
effectiveness research and analyses of policy changes. HSR offers an important
opportunity to bridge existing disciplinary divides and facilitate discussion of valid causal
inference. In the past, clinical research has made extensive use of the randomized control
trial (RCT), still often viewed as the gold, if not only, standard for credible inference. But
practitioners have learned that RCTs have technical, financial, and ethical limitations,
especially for many questions addressed in epidemiology, economics and health services
research. The good news is that relatively recent research has shown that when properly
conducted, analyses of observational (i.e., non-randomized trial) data can lead to valid
causal conclusions too. Further, observational data have two great advantages over RCTs:
they are more plentiful and almost always cheaper to collect.
Given the ubiquity of observational data, health services researchers need to have the full
complement of analytic tools for drawing causal inferences. Yet it is difficult to find
helpful summaries. Instead of discussing models or research questions, this paper
approaches the issue by first considering available data. We suggest a limited
classification system for observational data, and briefly review some of the analytic
approaches that are appropriate for each type of data.
(a) In the first case, data are available only from the post-treatment (or "exposed") time
period on subjects who were not randomized to the treatment and control groups (e.g.,
cross-sectional survey). Useful approaches include instrumental variables (IV)
estimation, sample selection models, and regression discontinuity designs.
(b) In the second case, data are available on the same non-random treatment and control
group subjects, but from both the pre- and post-treatment time periods. Multiple
observations might be available on subjects in the pre- and post-treatment time periods
(e.g., pre- post-treatment designs and true panel datasets). In addition to the methods
applicable to the first case, analysts often turn to difference in difference models and
panel data methods with endogenous choice of the treatment.
(c) Finally, we might have data available on different subjects in the pre- and posttreatment time periods, non-randomly assigned to the treatment and control groups.
These are not true panel data, and we must be concerned about in- and out-migration into
the treatment and control groups. An example would states that enacted some type of
health care reform versus those that did not, with randomly drawn samples of survey
respondents in both sets of states in the periods prior to and after the implementation of
the reform in the “treatment” state. The presence of the treatment might draw subjects
from the control states into the treatment states in the post-treatment period..
What should a policymaker make of inferences drawn from observational research?
There is no theoretical reason to believe only results from RCTs. If we imposed that
restriction with any enthusiasm, human learning would come to a virtual halt.
Policymakers and their staffs need to begin thinking “causally” about the findings from
both RCTs and observational research studies and ask simple but penetrating questions
about those research designs and the findings.