Introduction to Selection Bias in
Observational
Obse
vat o a Stud
Studies
es
Steven D. Pizer
June 29, 2010
Outline
• Overview of selection bias: how it arises in
study designs and problems it causes.
• Choice of Methods
–
–
–
–
Propensity scores methods (Paul).
Propensity
i scores application
li i (Matt).
(
)
Instrumental variables (Steve).
A combined approach (Paul).
• Questions and Answers
Answers.
Overview
• Selection bias is well known.
– Randomized controlled trials eliminate it
• Why conduct observational studies?
–
–
–
–
Cost of data collection.
Ethical considerations.
Faster results.
“Real world” settings.
• But selection into treatment often correlated with
outcome.
t
– For example . . .
Study Suggests TV-watching Lowers Physical Activity
27 Aug 2006
A study of low-income housing residents has
documented that the more television p
people
p say
y they
y
watched, the less active they were, researchers from
Dana-Farber Cancer Institute and colleagues report.
The findings of television's effects on physical activity
are the first to be based on objective measurements
using pedometers
pedometers, rather than the study subjects'
subjects
memories of their physical activity, say the
researchers. The study will be published online by the
A
American
i
JJournall off P
Public
bli H
Health
lth on July
J l 27 and
d
later in the journal's September 2006 issue.
Overview:
Source of Bias in RCTs
Treatment
group
Flip of a Coin
Outcome
Sorting
Comparison
group
• In RCTs, randomization ensures that
– Observed (and unobserved) covariates are balanced
between treatment and control groups
– Only
O l difference
diff
is
i treatment
t t
t assignment
i
t
– Thus, only cause of outcome difference is treatment
• No bias b/c coin flip is only
onl driver
dri er of sorting and
coin flip has no impact on outcomes
Overview: Source of Bias in
Observational Studies
Patient
characteristics
Observed: health,
income, ed, dist.
Unobserved: health,
skills attitudes
skills,
Sorting
Outcome
Comparison
group
Provider
characteristics
Observed: staff,
costs, congestion,
Unobserved:
culture, attitudes,
leadership
Treatment
group
Institutional
f t
factors
laws, programs
• In non-randomized studies, things get messy b/c there are
many drivers of sorting that also affect outcomes.
Observational Study Scenarios
• Scenario A: New clinical intervention. Self-care
training
i i for
f CHF.
CHF S
Self-reported
lf
d hhealth
lh&
satisfaction.
• Scenario B: Network-level study of guideline
adherence. Annual eye and foot exams for
diabetics. Amputations and retinopathy.
• Scenario C: National studyy of Medicaid HCBS.
NH admission, mortality.
Scenario A: CHF Self-Care Study
Patient
characteristics
Observed: health,
income, ed, dist.
skills, attitudes
Provider
characteristics
Sorting
Training
group
Outcome
health,
satisfaction
Comparison
group
Observed: staff,,
costs, congestion,
culture, attitudes,
leadership
 Small study with primary data collection.
 All important factors are observed.
Scenario B: Guidelines Study
Patient
characteristics
Observed: health,
income, ed, dist.
Unobserved: health,
skills attitudes
skills,
Sorting
Outcome
amputations
retinopathy
Non-adherent
group
Provider
characteristics
Observed: staff,
costs, congestion,
Unobserved:
culture, attitudes,
leadership
Adherent
group
Institutional
f t
factors
standards
 Unobserved factors are important.
 Institutional factors are related to outcome.
Scenario C: HCBS Study
Patient
characteristics
Observed: health,
income, ed, dist.
Unobserved: health,
skills attitudes
skills,
Sorting
Outcome
NH admits,
mortality
Comparison
group
Provider
characteristics
Observed: staff,
costs, congestion,
Unobserved:
culture, attitudes,
leadership
HCBS
group
Institutional
f t
factors
program location
 Unobserved factors are important.
 Institutional factors drive sorting w/o affecting outcome.
Scenarios: Lessons
• A: Small study w/o unobservables.
P
Propensity
it scores.
g study
y w/ unobservables & w/o
• B: Larger
sorting variable uncorrelated w/ outcome.
y flawed.
Fatally
• C: Large study w/ unobservables &
uncorrelated sorting variable.
variable Instrumental
Variables.
Translating Diagram Into
Equations
Patient
characteristics
Provider
characteristics
h
Sorting
Institutional
factors
T t
Treatment
t
Outcome
Comparison
Eq 1: Outcome = Treatment + Xpatient + Xprovider + u1
Eq 2: Treatment = Xpatient + Xprovider + Xinstitutions + u2
Selection bias occurs in Eq 1 when u1 is correlated with u2,
and therefore with Treatment.
Intermission
This presentation adapted from:
Pizer, SD, “An Intuitive Review of Methods for Observational Studies
Of Comparative Effectiveness,” Health Services and Outcomes Research
Methodology, 9(1) (March 2009): 54-68.
Shameless Plug
Shameless Plugg
Instrumental Variables
• Overview
• Practice patterns as instruments
Instrumental Variables: Overview
• Selection bias means naively estimated
effect of Treatment on Outcome includes
influence of unobservables correlated with
Treatment variable.
• So wee have
ha e to either remove
remo e effect of
correlated unobservable or control for it.
IV Overview (2)
• Instrumental variables (IV) uses variables
th t affect
that
ff t sorting
ti but
b t are nott related
l t d to
t
patient or provider unobservables.
• Think of coin flip that influences treatment.
• These are often institutional factors.
– Example: Residence in county with HCBS
waiver program.
program HCBS recipients vs.
vs others
=> bias. Waiver county residents vs. others =>
no bias.
IV: A General Approach
Eq 1: Outcome = O(Treat, u2hat, Xpatient, Xprovider) + u1
Eq 2: Treatment = T(Xpatient, Xprovider, Xinstitutions) + u2
• Applicable to linear or nonlinear models with
additive errors.
• Estimate Eq 2 (like propensity score estimation).
• Construct ppredicted value of u2 ((u2hat).
)
• Add u2hat to outcome equation to control for
correlated unobservables.
IV: Issues
Eq 1: Outcome = O(Treat, u2hat, Xpatient, Xprovider) + u1
Eq 2: Treatment = T(Xpatient, Xprovider, Xinstitutions) + u2
• Must have identifying instrument(s): Xinstitutions in
this case.
• Identifying
Id if i instrument(s)
i
( ) must be
b strongly
l
correlated with Treatment and excludable from
Outcome equation
equation.
• Estimate only applies to those affected by
()
instrument(s).
IV Issues: LATE
• LATE: Local Average Treatment Effects.
• Instrument may not affect everyone equally.
• IV estimate only applies to segment of
population “in play”.
• Example: HCBS demo only applies to
patients who might avoid nursing homes.
Practice Patterns as Instruments
Source: Dartmouth Institute for Health Policy and Practice
Practice Pattern Variations
• The intensity of medical practice patterns
varies geographically (Dartmouth Atlas).
• Reasons: physician training and culture;
financial incentives; marketing campaigns;
diff sion of technology;
diffusion
technolog ; quality
q alit
improvement programs.
• Uncertainty about best care.
Practice Patterns As Instruments
• Instrument: strongly correlated with
treatment & excludable from outcome.
• Uncertainty sounds like randomness,
randomness but it
isn’t always the same. Excludable?
• Physician
h i i culture
l
and
d marketing
k i (yes).
( )
• Technology diffusion and quality (no).
• Patients select provider for treatment (no).
Culture & Marketing
• Prostate: Sommers et al. (2008) find type of
specialist determines treatment choice
((surgery,
g y, radiation,, medical management).
g
)
• Atypical antipsychotics: Leslie, Mohamed
and Rosenheck (2009) find 60% off-label
off label
use.
Quality & Tech Diffusion
• Diabetes care: Impact of strict standard
(HbA1c <=7) not known, but adherence to
standard correlated with other quality.
q
y
• Rheumatoid arthritis: Cost effectiveness of
ne biologics
new
biologics. Use not yet
et widespread
idespread so
correlated with being a “early adopter” of
other technologies.
Patient Selection
• Patients might seek physicians or hospitals
known to provide particular treatments
((Brookhart and Schneeweiss,, 2007).
)
• Are patients aware of practice differences?
• Is provider
id choice
h i related
l d to severity
i off
condition?
Useful Control Variables
• Severity of condition
• Quality of care
• Provider type or length of clinical
relationship
Operational Questions
• Is there widespread uncertainty about what
is best?
• Are options related to severity?
• Is the practice pattern under study related to
other
h quality
li patterns?
• Do patients select providers to get desired
treatment?
Prominent Examples
• Cardiac cath: Stukel et al. (2007) use
regional cath rates.
• Atypical antipsychotics: Wang et al.
al (2005)
use physician prescribing rates.
• Cox-2 inhibitors:
i hibi
Schneeweiss
h
i et al.
l (2006)
(
)
use physician’s most recent NSAID
prescription.
Certainty Can Become
Uncertainty
• PSA screening
• HRT for menopause
• Cox-2
Cox 2 inhibitors
Summary
• Practice patterns can make good
instruments if:
– Widespread variation
– Uncertainty about what is best
– Quality/technology/severity
Q lit /t h l /
it either
ith uncorrelated
l t d
or controlled
– Patients don’t choose provider to get treatment