MBSW 2012
Assessing the Impact of Unmeasured
Confounding in Comparative
Observational Research
Doug Faries
Sr Research Scientist, Eli Lilly & Company
Based on Work Done by: Karen Price,
Xiaomei Peng, Manjiri Pawaskar,
James Stamey, John Seaman
Outline
Background
• Quality Implementation of Observational Research
• Motivation of Unmeasured Confounding Issue
Unmeasured Confounding Methods
• Focus on: Rule Out / Bayesian Modeling / Multiple Imputation
Example Cost Analysis
Summary
The Observational Research Problem
(or Challenge)
Selection
Bias
Confounders
• Physicians/patients did not select
treatment ‘at random’ but based on a
variety of factors – so Groups A and B
differ in some aspects other than
treatment
• A variable is a Confounder if it is
associated with both treatment selection
and outcome
The Observational Research Challenge
Selection
Selection
Bias
Bias
Confounders
• Physicians/patients did not select treatment ‘at random’ but
based on a variety of factors – so Groups A and B differ in
some aspects other than treatment
• A variable is a Confounder if it is associated with both
treatment selection and outcome
Measured: Information is collected within the study
and statistical adjustment is possible
Unmeasured: Information on the confounder is not
available from the study
Can We Get Causal Inference From
Observational (non-randomized) Data?
YES – IF 3 key assumptions hold …..
1. “No Unmeasured Confounders”
– An ASSUMPTION! Can not be
definitively verified.
2. “No Perfect Confounding”
3. Correct Models are used
Hierarchy of Evidence
Vandenbrouke (2008), Concato (2000), ….
1
• Systematic Reviews of Randomized
Controlled Trials
2
• Randomized Controlled Trials
3
• Prospective Follow-up Studies
4
• Retrospective Follow-up Studies
5
• Case Control Studies
6
• Anecdotal Case Report and Series
Increasing the Quality of Observational
Research (Rubin 2007, 2008)
Keep core statistical (RCT) design principles in mind that
are sometimes overlooked in observational research …..
Prospective Specification
Multiplicity
Replication
Sensitivity Analyses
Current State of the Union Regarding
Unmeasured Confounding
What should I do about
unmeasured confounding?
Current State of the Union Regarding
Unmeasured Confounding
What should I do about
unmeasured confounding?
Just mention it as a
limitation in the
Discussion Section
and move on!
There are new methods in the literature!
“Best Practices” include sensitivity analyses
EXPERT
Figure 1: Unmeasured Confounding Options
Unmeasured
Confounding
Information
Available
Method
None
1) Rule Out
2) IV
External
1) Bayesian
2) Algebraic
Internal
1) Bayesian
2) Multiple Imputation
3) Propensity Calibration
Example for Today
Pawaskar M, Zagar A, Sugihara T, Shi L (2011). Healthcare
resource utilization and costs assessment of type 2 diabetes
…... J Med Econ. 2011;14(1):16-27.
No direct measure of glycemic control was available in the
original claims database. However, after linking with a
laboratory file, A1C values were obtained in a subset (about
20%) of the sample;
– A1C was a significant predictor of treatment selection (p<.001)
but only modestly related to outcome (costs)
Our Work: Sensitivity Analysis using this Internal Information
Information Available: None
Rule Out
CONCEPT: Quantify how strong and imbalanced a
confounder would need to be in order to explain (‘rule
out’) the observed treatment difference
Instrumental Variables
CONCEPT: Use of an Instrument (variable associated
with treatment selection but not with outcome) allows one
to mimic randomization
Rule-out Method
Concept: Quantify how strong and imbalanced a
confounder would need to be in order to explain
(“rule out”) the observed result.
This approach attempts to find all combinations of
1) the confounder-outcome relationship and
2) the confounder-treatment relationship,necessary to move the observed point estimate
to zero.
Rule Out – Simple Model
Basic (additive) Model:
AMD = TTD + Bias
•
AMD is the apparent (observed) mean treatment difference
•
TTD is the true (fully adjusted) mean treatment difference
•
Bias is a function of:
– The imbalance of the unmeasured confounding factor between
treatment groups
– The strength of the association between the unmeasured confounder
and the outcome
Rule Out – Simple Spreadsheet
Bias
Pc0 – Pc1
AMD
(Estimate)
CE
AMD (Lower
95% CL)
CE
.95
$ 2,597
$ 2,734
$ 690
$ 726
.90
$ 2,597
$ 2,885
$ 690
$ 767
$ 2,597
$ 51,940
$ 690
$ 13,800
…..
.05
Fixed Value 1
Fixed Value 2
So, a confounder occurring in 20%
more patients in Cohort A
(compared to Cohort B) which
results in $15,000 higher cost per
patient would eliminate the
observed difference
Trt A is
Not Less
Costly
Trt A
remains
Less
Costly
Confounder - Outcome
Association
Company Confidential
File name/location
Copyright © 2000 Eli Lilly and Company
Confounder – Cohort
Association
Rule-out Method – Example
Rule Out Example
100
90
80
AMD (best estimate)
AMD (Lower 95% CI)
70
Confounder /
Treatment
Relationship
(PC1-PC0)
60
50
40
30
20
10
0
0.00
1000.00 2000.00 3000.00 4000.00 5000.00 6000.00 7000.00 8000.00 9000.00 10000.00
CE (Confounder Effect on Outcome %)
Information Available: External
Concept: Use information external to the study (e.g.
data from literature or other databases) to estimate
parameters regarding unmeasured confounding
(e.g. strength of association with outcome and
treatment).
Bayesian Models:
Incorporate the external information through a prior
distribution and account for the uncertainty surrounding
the external estimates
External Adjustment (ctd)
Some Issues:
1) Transportability
2) Correlation of Unmeasured Confounder with
variables already accounted for in the analysis
model
Algebraic External Adjustment Examples:
- Schneeweiss et al JAGS 2005
- Schneeweiss et al CNS Drugs 2009
Information Available: Internal
Concept: Use information from the patients in the
study (e.g. subsample of chart review data for a
retrospective claims database study) to estimate
parameters regarding unmeasured confounding
With Internal data can avoid transportability assumption and
can account for correlation between unmeasured confounder
and measured confounders
Information Available: Internal
Methods
1) Propensity Score Calibration
• Sturmer et al, Am J Epi 2005
2) Bayesian Modeling
• McCandless Stat Med 2007
3) Multiple Imputation
• Faries (submitted)
Propensity Score Calibration
Concept: Utilize additional data - variables not in full
sample but available for a subset of patients - to modify
the propensity score adjustment
Two propensity scores (PS) are calculated for the
validation data,
- “Error Prone” PS: utilizes only covariates available
for the full sample of patients
- “Gold Standard” PS: calculated using both the
covariates in the main study along with the
additional confounding covariates.
Regression calibration (measurement error modeling) is
then applied to adjust the regression coefficients and thus
compensate for the unmeasured confounding.
Propensity Score Calibration (ctd)
Error Prone Propensity Score Model (PSEP)
PSEP  Pr(X  1 | z1, z 2,..., zn)
Gold Standard Propensity Score Model (PSGS)
PSGS  Pr(X  1 | z1, z 2,...,zn,  )
Calibration Model:
E[ PSGS ]   0 1 X  2 PSEP
File name/location
Company Confidential
Copyright © 2000 Eli Lilly and Company
Propensity Score Calibration
Validity relies on surrogacy of the error prone
propensity for the gold standard propensity.
•
“error prone PS” must be independent of the outcome given “gold
standard PS” and treatment.
For our example – surrogacy assumption not clearly satisfied
– Correlations of A1C & Outcome was negative
– Correlations of Other Covariates & Outcome was positive
Bayesian Twin Regression Models
Concept: Bayesian models naturally incorporate
additional sources of information – such as internal
subset data or external information from other
studies - through prior distributions
Outcome  0   1 * T reatment  2 * UnmConf  3 * MeasConf
Logit P(UnmConf  1)      Treatment    MeasConf
Internal data serves in essence as informative prior
information for parameters relating to unmeasured confounder
Implementation: WinBUGS (SAS 9.3 code upcoming)
Bayesian Twin Regression Models
Outcome  0   1 * T reatment  * UnmConf  * MeasConf 
logit P(UnmConf  1)      TRT     MeasConf
Priors:
Uninformative:
 , ,
Informative:
,  0 ,  1,  2
Keys to Bayesian Approach
• Incorporation of best available information through
Informative Priors
• Best available data – whether internal (via subset data) or External data (e.g.
literature)
• Informative Priors – not just a way to add uncertainty (McCandless 2007)
• Yields a posterior distribution (point and interval estimates) for
the treatment effect adjusted for the unmeasured confounder U.
• Fixed Modeling (Schneeweiss 2006) failed to incorporate variability
• Flexible data driven model
• No restrictions on relationships on associations between variables as in
measurement error approaches (Sturmer 2007).
Missing Data Multiple Imputation
(for internal data)
Concept: This is a missing data problem – use a
well accepted method -- Multiple Imputation!
Imputation Model: Treatment, Measured Covariates, and Outcome
(in order to account for he association between confounder/outcome
and confounder / treatment)
Used > 5 replications due to amount of missing data
Implementation: PROC MI in SAS
Example: Summary of Sensitivity
Analyses with Internal A1C Data
Conclusions
Comparative effectiveness research should include some level of
‘Unmeasured Confounding’ assessment to help consumers of the
data understand the robustness of the findings.
Bayesian and MI methods are promising approaches
- naturally incorporate additional information (internal or external)
- can use internal data to avoid development of prior.
Lots of Remaining Questions
• When is one method preferred to another?
• How much ‘internal data’ is needed for each method?
• When is it cost effective to obtain the internal information as opposed to more
easily available external data?
Other References
Goodman M, Barraj LM, Mink PJ, Britton NL, Yager JW, Flanders D, Kelsh MA (2007). Estimating uncertainty in observational
studies of associations between continuous variables: example of methylmercury and neuropsychological testing in children
Epi Perspec Innovat 4:9.
Gustafson P, McCandless LC (2010). Probabilistic Approaches to Better Quantifying the Results of Epidemiologic Studies. Int. J
Environ Res Pub Hlth 7:1520-1539.
McCandless, L.C. Gustafson, P. Levy, A.R. Bayesian sensitivity analysis for unmeasured confounding in observational studies.
Statist. Med. 2007, 26, 2331-2347.
McCandless, L.C. Gustafson, P. Levy, A.R. A sensitivity analysis using information about measured confounders yielded improved
assessments of uncertainty from unmeasured confounding. J. Clin. Epidemiol. 2008, 61, 247–255.
Pawaskar M, Zagar A, Sugihara T, Shi L (2011). Healthcare resource utilization and costs assessment of type 2 diabetes patients
initiating exenatide BID or glargine: a retrospective database analysis. J Med Econ. 2011;14(1):16-27. Epub 2010 Dec 15.
Schneeweiss S, Setoguchi S, Brookhard MA, Kaci L, Wang PS (2009). Assessing Residual Confouding of the Association
between Antipsychotic Medications and Risk of Death Using Survey Data. CNS Drugs, 23(2):171-180.
Schneeweiss S, Wang PS (2005). Claims Data Studies of Sedative-Hypnotics and Hip Fractures in Older People: Exploring
Residual Confounding Using Survey Information. JAGS 53:948-955.
Schneeweiss S, Wang PS (2005). Claims Data Studies of Sedative-Hypnotics and Hip Fractures in Older People: Exploring
Residual Confounding Using Survey Information. JAGS 53:948-955.
Sturmer T, Schneeweiss S, Rothman KJ, Avorn J, Glynn RJ (2007). Performance of Propensity Score Calibation – A Simulation
Study. Am J Epi 165(10):1110-1118.
Sturmer T, Schneeweiss S, Avorn J, Glynn RJ (2005). Adjusting Effect Estimates for Unmeasured Confounding with Validation
Data using Propensity Score Calibration. Am J Epi 162(3):279-289.
Weiner MG, Xie D, Tannen RL (2008). Replication of the Scandinavian Simvastatin Survival Study using a primary care medical
record database prompted exploration of a new method to address unmeasured confounding. Pharmacoepi Drug Safety
DOI: 10.1002/pds
Xue F, Strombom I, Turnbull B, Zhu S, Seeger JD (2011). Duloxetine for Depression and the Incidence of Hepatic Events in
Adults. J Clin Psych 31:517-522/