Addressing Selection Bias
Addressing
Selection Bias
In Observational Studies
In Observational Studies
AcademyHealth Methods Workshop
J
June 29, 2009
29 2009
Outline of Workshop
Outline of Workshop
• Overview
Overview of how causation can be attributed of how causation can be attributed
in observational studies
• How selection bias can arise
• Review of best practices for propensity score Review of best practices for propensity score
modeling
• Illustration of PS model building and testing
• Conceptual review of instrumental variables Conceptual review of instrumental variables
(IV) analysis
• Application of IV and PS
l
f
d
Speakers
• Paul Hebert, PhD
Paul Hebert PhD
– Seattle VA HSR&D and University of Washington
• Matt Maciejewski, PhD
Matt Maciejewski PhD
– Durham VA HSR&D and Duke University
• Steve Pizer, PhD
Steve Pizer, PhD
– Boston VA HSR&D and Boston University
Applications of Applications
of
Propensity Score Matching
Propensity Score Matching
Matthew L. Maciejewski, PhD
Matthew
L Maciejewski PhD
Durham VA HSR&D and Duke University
Outline of Workshop
Outline of Workshop
• Overview
Overview of how causation can be attributed of how causation can be attributed
in observational studies
• How selection bias can arise
• Review of best practices for propensity score Review of best practices for propensity score
modeling
• Illustration of PS model building and testing
• Conceptual review of instrumental variables Conceptual review of instrumental variables
(IV) analysis
• Application of IV and PS
l
f
d
Core Challenge of Quasi‐Experimental Studies
• Identifying causal effect of treatment in absence of randomization to ensure covariate balance
of randomization to ensure covariate balance
• How do we know that outcome difference between treatment & control is due to
• Effect of treatment on outcome
• And not
And not effect of selection on treatment?
effect of selection on treatment?
Treatment Effect vs. Selection Effect
• How to differentiate the treatment effect from the selection effect?
from the selection effect?
Treatment Effect is…
Harmful
Null
Protective
Treatment > < Treatment Treatment >
Favorable
Control (depends) > Control
Control
selection
Treatment <
Treatment Treatment > Treatment >
No selection Treatment < No selection
Adverse Adverse
selection
Control
= Control
Control
Treatment < Treatment
<
Control
Treatment Treatment
< Control
Treatment > <
Treatment
><
Control (depends)
Which Cell Relates to Your Finding?
• Suppose your analysis finds outcomes are better for treatment group
for treatment group
Treatment Effect is…
Harmful
Null
Protective
Treatment > < Treatment Treatment >
Favorable
Control (depends) > Control
Control
selection
Treatment >
Treatment > No selection
No selection
Control
Adverse Adverse
selection
Treatment > <
Treatment
><
Control (depends)
Best Case Scenario?
• Estimate predictors of treatment (for PS & IV)
• If no selection on observables, 1
If no selection on observables, 1st result is right
result is right
Treatment Effect is…
Harmful
Null
Protective
Treatment > < Treatment Treatment >
Favorable
Control (depends) > Control
Control
selection
Treatment >
Treatment > No selection
No selection
Control
Adverse Adverse
selection
Treatment > <
Treatment
><
Control (depends)
More Complicated & Common Case
• If favorable selection exists but unaccounted for
If favorable selection exists but unaccounted for
• Benefit of treatment overstated, if Tx protective
• Benefit of treatment misrepresented, if Tx null
f f
d f
ll
• Harm of treatment understated, if Tx harmful
,
Treatment Effect is…
Harmful
Null
Protective
Favorable
selection
Treatment > < Treatment Control (depends) > Control
Treatment >
Control
• Need to account for selection bias somehow
Differentiating Treatment & Selection Effects with Propensity Scores
• Coefficients in treatment equation may indicate selection
– Even though parameter estimates in propensity scores are not of primary interest
scores are not of primary interest
• Effective matching or weighting with propensity scores can reduce imbalance in observed covariates
– May reduce bias due to selection on observables
May reduce bias due to selection on observables
– Does nothing for selection on unobservables
Steps in Propensity Score Model Building
• Step 1: Model Pr(Tx group)
– Pr(Tx group) = β
( g p) β0+β
β1·Xi+εit
• Step 2: Identify matches • Step 3: Assess sample loss & balance in covariates between groups
– If balanced, then done; otherwise, back to Step 1
• Step 4: Outcomes analysis on matched subset
Step 4: Outcomes analysis on matched subset
SD formulas: Austin Nov 2009 Stat Med
Mechanics of Propensity Score Matching
• Step 1: Model Pr(Tx group) on final model
– Generate predicted value (aka propensity score)
p
(
p p
y
)
• Step 2: Identify matches (see Paul’s slides)
– Examine common support to identify overall E
i
id if
ll
balance & proportion of original sample excluded
• Step 3: Assess sample loss & covariate balance
– Assess covariate balance for equivalence between Assess covariate balance for equivalence between
subgroups in this match
Step 1: What X’ss do you use?
Step 1: What X
do you use?
Risk Factor
Instrument
Treatment
Outcome
C f
Confounders
d
Step 2: Examine the Common Support and Overall Balance
Distribution of Propensity Score Before Matching
Distribution of Propensity Score After Matching
20.0
15.0
17.5
12.5
10.0
Pe rce n t
12.5
0
0
Pe rc e n t
15 0
15.0
10.0
7.5
7.5
50
5.0
5.0
2.5
2.5
0
0
20.0
15.0
12.5
12.5
10.0
10.0
7.5
Pe rce n t
15.0
1
1
Pe rc e n t
17.5
75
7.5
5.0
5.0
2.5
2.5
0
0.21 0.25 0.29 0.33 0.37 0.41 0.45 0.49 0.53 0.57 0.61 0.65 0.69 0.73 0.77 0.81
Estimated Probability
0
0.3525 0.3825 0.4125 0.4425 0.4725 0.5025 0.5325 0.5625 0.5925 0.6225 0.6525 0.6825 0.7125 0.7425 0.7725 0.8025
prop
'
Step 3: Treatment Estimation &
Specification Iteration for Balance
• Example: Impact of Rx copayment increase on adherence to anti‐HTNs
adherence to anti
HTNs
Number of….
Covariates Imbalanced….
Iteration Main Effects Interactions Quadratics Overall Q1 Q2 Q3 Q4 Q5
1
11
0
0
7
3
1
1
4
7
2
13
0
0
8
2
3
1
4
6
3
11
8
0
10
3
0
1
3
7
4
14
0
0
8
3
1
2
4
6
5
14
14
0
15
1
0
1
4
6
10
10
2
11
1
0
3
2
7
…
112
Maciejewski, et al. 2010 AJMC
Step 3: Improvement in Covariate Balance After Final Matching
Standardized Differences
Standardized
Differences
Unmatched Sample
Matched Sample
Age in years
Age in years
‐4
4.4%
4%
0 5%
0.5%
Male
16.2%
1.3%
Comorbidity (ETG)
0.1%
1.1%
Medication Count
15.6%
‐1.6%
Ave Generic Copay ($)
6.5%
3.9%
1+ Ninety Day Fills
‐20.6%
‐0.6%
Prescriptions filled
that were generic
3.0%
‐5.2%
Number in Treatment
Number
in Treatment
Number in Control
15,417
15
417
8,823
8,298
8
298
8,298
Maciejewski, et al. under review
Take Away Points
Take‐Away Points
• Identifying the causal effect is challenging in observational studies due to selection bias
– Assessment of selection bias is first step
• Propensity
Propensity modeling can inform selection bias modeling can inform selection bias
question and may improve identification of causal effect of treatment
– Must satisfy assumption of unmeasured confounders
y
p
• PS model building can be time consuming
• Assess sample loss & covariate balance
l l
b l
Luo, Gardiner & Bradley 2010 MCRR
Questions?