Can administrative data increase
the practicality of clinical trials?
An example from the Women’s Health Initiative
Garnet Anderson
Fred Hutchinson Cancer Research Center
Pragmatic trial (coined by Schwartz and
Lellouch in 1967)
• A randomized controlled trial designed to inform decisions about
practice
• Used to describe a trial designed to test the effectiveness of the
intervention in a broad routine clinical practice setting (as opposed to
testing the efficacy in an ideal setting)
Pragmatic methods and motivations
• Broaden eligibility to improve generalizability, increase recruitment
yield and perhaps reduce costs
• Test interventions/delivery mechanisms that better emulate clinical
practice and thus improve estimates of population level impact and
reduce costs
• Limit data collection to minimize participant burden and reduce costs
Was the original Women’s Health Initiative a
pragmatic trial?
DM
48,835
HT
27,347
CaD 36,282
os
CT=68,132
93,676
Total WHI =161,809
WHI transitions in post-intervention phase
• Protocol streamlined to annual mail follow-up
• In 2010, documentation and central adjudication of outcomes limited
to African American, Hispanic and former HT participants (Medical
Records Cohort, n~22,000)
• Outcomes for remaining participants (Self-Report Cohort, N~71,000)
limited to self-report or passive follow-up sources (NDI, Medicare)
Women’s Health Initiative Strong & Healthy Trial
Marcia L. Stefanick, Ph.D.
Charles L. Kooperberg, Ph.D.
Andrea Z. LaCroix, Ph.D.
A Pragmatic Trial : Physical Activity to Improve CV Health in Women
1 U01 HL122280-01
Primary Hypothesis
To assess whether aerobic physical activity combined with
muscle strengthening, balance and flexibility exercises, and
reduced sedentary behavior, will reduce major CV events (MI,
stroke, CV death) in older women, compared to “Usual Activity”
(Control) over 4-5 years of follow-up
Based on: Report of the Physical Activity Guidelines Advisory Committee, 2008. (Chapter 5:
Active Older Adults) http://www.health.gov/paguidelines/guidelines/chapter5.aspx
Implemented through: National Institutes on Aging (NIA) http://go4life.nia.nih.gov/
Eligibility
• WHI participant, alive and in active follow-up
• In Medical Records Cohort or in Self Report cohort and enrolled in
Medicare Part A/B
• Exclusions
• Inability to walk
• Dementia
• Residing in nursing home
Study Design: Zelen’s randomized consent design
Eligible
based on
existing
data
no
Follow, per WHI protocol
yes
Control
(n ~26,000)
Randomize
Follow, per WHI protocol
Opt Out: no
Intervention
(n ~ 26,000)
Follow, per WHI protocol
Consent
yes
WHISH PA (Go4Life®) Intervention
deliver mail-based [+ website, etc.]
± IVR** (phone) + live advisor, PRN
** Interactive Voice Response System (Consent)
Zelen, M. The New England Journal of Medicine 1979; 300: 1242–1245.
Sources of outcomes data
• Self-reported health events annually, for all WHI participants
• Fully documented, adjudicated outcomes for the Medical Records
Cohort
• Outcomes derived from Medicare claims and NDI among women
from the Self-Report Cohort
Considerations in using Medicare claims data
for outcomes assessment
• Coverage
• Types of data available for outcomes assessments
• Quality of inference
• Logistics
Medicare data routinely available
• Denominator
• Inpatient (MedPar)
• Outpatient
• Home Health
• Skilled Nursing
• Hospice
• Durable Medical Equipment
• Carrier
• Part D—Prescription Drug
WHI Medicare linkage
• Submitted 151,116 names, all with valid SSN
•
•
•
•
140,471 (93%) were age eligible as of 12/31/07 or had already died at age>65
142,195 names returned by Center for Medicare/Medicaid Services (CMS)
132,109 perfect matches on 5 identifiers (94% of submitted, age-eligible)
3,203 “fuzzy” matches (2% of submitted age eligible)
• Small discrepancies in one identifier
• 90% linkage of submitted participants
• 96% linkage of age eligible participants
Note: All WHI participants are now over age 65
Linkage to Medicare does not mean claims
data are available
• Medicare claims not available for Managed Care (MC) members
• Included in denominator file but because reimbursement is capitated, MC
organizations don’t submit claims
• MC penetrance varies over time and geographic region
• Individuals may change their coverage type over time
WHI participants’ Medicare enrollment status
by calendar year
140000
120000
WHI Participants
100000
80000
Any CMS
FFS A+B
60000
MC
40000
20000
0
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
Calendar Year
2004
2005
2006
2007
2008
2009
2010
2011
2012
Individuals can change their type of coverage
• ~98% kept the same type of coverage over a one year period
(1/1/2011-1/1/2012)
• 64% had Fee-For-Service plans (Part A or A+B)
• 34% were in Managed Care
• Of the 2.12% that change types of coverage,
• 0.55% went from Managed Care to Fee-For-Service
• 1.57% went from FFS to MC
Determining “events” in claims data
• Claims include up to 10 diagnosis codes based on ICD-9-CM
• One “primary” diagnosis
• Up to 9 secondary diagnosis
• “Primary” designation and number of secondary diagnoses included
is determined by the submitting institution
Assessing agreement rates between WHI
adjudicated events and Medicare claims-based
diagnoses
• Eligibility:
• Age ≥ 65 at WHI enrollment
• Linked to Medicare and continuously enrolled in fee-for-service Medicare Part
A (in-patient)
• WHI events
• Documented, centrally adjudicated (hospitalized) cardiovascular outcomes
• Only first post-enrollment event of each type considered
Chronic Disease Warehouse provides a
catalog of algorithms for diagnoses
• Myocardial infarction: hospital discharge codes (410.x0, 410.x1) as
either primary or one of 9 secondary discharge diagnosis codes.
• Coronary revascularization: ICD-9-CM procedures codes for CABG
(36.1x, 36.2) or PCTA (00.66, 36.0, 36.00, 36.01, 36.02, 36.05, 36.97)
• Stroke: ICD-9-CM codes (430.xx, 431.xx, 433.x1, 434.x1, 436.xx, 437.1x, and
437.9x) in any diagnostic position
• Abdominal aortic aneurysm ICD-9-CM diagnosis (441.3-441.5, 441.9) or
procedure (38.34,38.44,39.25,39.52,39.71) or CPT codes (35081, 35082,
35102, 35103, 35091, 0001T, 0002T, 35800, 34802-34805, 34830-34832)
Research Data Assistance Center (ResDAC.org) is an excellent resource for working with CMS data
Agreement rates for acute MI using principle
diagnosis code
Medicare
Yes
No
Total
WHI
Yes
No Total
914
281 1195
431 35771 36202
1345 36052
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Sens
68%
Spec
99.2%
PPV
76%
NPV K (95%98.8%
CI)
0.71 (0.69-0.73)
Overall
98.1%
accuracy
Sensitivity improves but PPV declines using
principle or secondary diagnosis codes for aMI
Medicare
Yes
No
Total
WHI
Yes
No Total
1062
439 1501
283 35613 35896
1345 36052
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Sens
79%
Spec
98.8%
PPV
71%
NPV K (95%99%
CI)
0.74 (0.72-0.75)
Overall
98.0%
accuracy
Agreement rates for procedures are higher:
Coronary bypass graft surgery
Medicare
Yes
No
Total
WHI
Yes
No Total
795
96
891
53 36549 36506
848 36549
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Sens
94%
Spec
99.7%
PPV
89%
NPV K (95%99.9%
CI)
0.91 (0.90-0.93)
Overall
99.6%
accuracy
Agreement rates: Stroke events
Medicare
Yes
No
Total
Lakshminarayan et al., Stroke (2014)
WHI
Yes
No Total
478
318
796
104 30817 30603
582 30817
Sens
82%
Spec
99.0%
PPV
60.1%
NPV K (95%99.7%
CI)
0.91 (0.90-0.93)
Overall
98.7%
accuracy
Medicare up-coding or missing WHI events
All
No self-reported, hospitalized
event
No adjudication
Adjudication did not identify
stroke
Lakshminarayan et al., Stroke (2014)
WHI No/
Medicare Yes
318
182 (57%)
PPV after
Adjustment
60.1%
77.9%
57 (18%)
79 (25%)
85.8%
Sensitivity and PPV improve when looking at
agreement rates at the person level: Stroke
Medicare
Yes
No
Total
Lakshminarayan et al., Stroke (2014)
WHI
Yes
No Total
505
240
745
77 30817 30603
582 30817
Sens
86.8%
Spec
99.0%
PPV
67.8%
NPV K (95%99.7%
CI)
0.91 (0.90-0.93)
Overall
98.7%
accuracy
Agreement rates for cancer incidence
Site
WHI-Yes
CMS-Yes
WHI-No
CMS-Yes
WHI-No
CMS-Yes
WHI-No
CMS-No
Sens
(%)
Spec
(%)
PPV
(%)
NPV
(%)
Overall
Breast
3451
1562
56
98302
98.4
98.4
68.8
99.94
98.4
Colorectal
1145
613
39
105947
96.7
99.4
65.1
99.96
99.4
Endometrial
621
146
17
61642
97.3
99.8
81.0
99.97
99.7
Lung
1236
634
63
105468
95.2
99.4
66.1
99.99
99.4
Melanoma
357
785
22
105355
94.2
99.3
31.3
99.98
99.2
Ovarian
353
372
19
86168
94.9
99.6
48.7
99.98
99.6
Medicare derived cancers based on presence of relevant ICD-9 cods in MedPar (in patient data), any position, or the first occurring
combination of 2 outpatient or carrier claims containing these codes that are 1-365 days apart.
Analyzing outcomes found in claims but not in
WHI
Breast
(n=1562)
Colorectal
(N=613)
Endometrial
(N=146)
Ovarian
(N=372)
N(%)
N(%)
N(%)
N(%)
Related diagnosis
888 (56.9)
210 (34.3)
62 (42.5)
199 (53.5)
Other cancer or
hysterectomy reported
292(18.7)
130 (21.2)
47 (32.2)
91 (24.5)
Other non-cancer
hospitalization reported
199 (12.7)
148 (24.1)
11 (7.5)
41 (11.0)
Cancer self-reported &
denied, or not adjudicated
95 (6.1)
85 (13.9)
19 (13.0)
24 (6.5)
No information
88 (5.6)
40 (6.5)
7 (4.8)
17 (4.6)
Correspondence of event dates
Exact same date
Within ± 1 day
Within ± 7 days
Within ± 30 days
Exactly ± 365 days
aMI
82%
94%
Stroke
83%
88%
95%
96%
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Lakshminarayan et al., Stroke (2014)
Mell et al., J of Vascular Surgery (2014)
AAA
89%
89%
89%
92%
LE PAD
75%
79%
82%
85%
0.7%
CAS
90%
93%
96%
96%
0.4%
Summary of comparisons between claims-based
and protocol-defined events in sample with
continuous FFS Medicare enrollment
• More events counted in Medicare than in WHI across all conditions
• Agreement rates ranged from good to excellent for clinical diagnoses
• Excellent agreement rates for procedures
• Accuracy of dates is acceptable for most failure time analyses
• Errors in both sources contribute to the disagreements
Is outcome misclassification in an
RCT benign?
Existing methods for mismeasured outcomes
focus on discrete proportional hazards
• Halloran ME and Longini IM. Using validation sets for outcomes and exposure to
infection in vaccine field studies. Am J of Epimiol 2001
• Meier AS, Richardson BA, and Hughes JP. Discrete proportional hazards models
for mismeasured outcomes. Biometrics 2003
• Magaret AS. Incorporating validation subsets into discrete proportional hazards
models for mismeasured outcomes. Statist in Med 2008
Comparing RCT results using adjudicated and
claims based outcomes
• Sample: Women > 65 years of age at randomization in either the HT
trial component
• ITT analyses comparing HT to placebo
• WHI adjudicated outcomes, with censoring at last-follow-up date or death
from other causes
• Claims-based outcomes, with censoring at end of enrollment in FFS Medicare,
death from other causes, or 12/31/2007 (last available claims data)
Effect of misclassification
of failure time events on
inference in a RCT
setting: WHI HT trial
Medicare found
fewer clinical diagnoses,
more procedures, similar
hazard ratios and overall
inference
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Assumptions:
Independent measurement error:
• Errors in outcomes data are comparable across treatment arms
• May not be defensible if treatment leads to symptoms the result in greater
physician contact/diagnostic procedures, etc.
Outcomes information from secondary source (claims) does not affect
the failure risk, given information on true failure time and treatment
assignment
Some impracticalities of using
claims data for outcomes
surveillance
Events data collection is not defined by
protocol
• Study outcomes primarily limited to ICD-9 diagnoses or procedures
• Diagnoses represent the community standard(s)
• Changes in diagnostic procedures/codes occur outside of the
protocol, may be driven by economic factors
• Data availability is at the mercy of another agency, its policies and
practices and those of the institutions who submit data to it
Trial monitoring complexities
• Will data be available soon enough for monitoring purposes?
• Annual installments, based on calendar year
• Timeline for release is not guaranteed
• Changes in data file structure over time
• Requires additional processing time
• Difficulty in locking the code for definitions
Outcomes collection and analysis plan
• For participant in the Medical Records Cohort, document and
adjudicate all key outcomes
• For the remaining participants in the Self Report cohort, use Medicare
Part A/B
• Documenting/adjudicating events for those who switch to managed care
• Analyses based on a stratified Cox Proportional Hazards model with
source of data as one of the stratification factors
Monitoring plan considerations
• Intervention is currently available to all (including the comparison
group)
• Ethical requirements to stop for efficacy are reduced
• Need a full-scale evaluation of this public health program
• Safety concerns are dominant
• Medicare may be too tardy, insensitive to monitor safety
• Options: Supplement Medicare data with self-report, particularly in the early
phases
Conclusions
• Claims data provide a key source of selected outcomes data that are
standardized and available across the nation for a large segment of
the population
• The correspondence between claims derived outcomes and
traditionally documented and adjudicated outcomes are good to
excellent but vary by type of outcome
• Randomized trials using claims data for outcomes may derive valid
inference if measurement error assumptions are met
• Timeliness of obtaining claims data may not be adequate for trial
monitoring
Dale Burwen
Ross Prentice
Mary Pettinger
Roberta Ray
Joseph Larson
Charles Kooperberg
Andrea LaCroix
Marcia Stefanick
Acknowledgement of colleagues and collaborators