The Impact of Survey Design Modifications on Health Care Utilization Estimates in a
National Longitudinal Health Care Survey
Steven B. Cohen, Ph.D.
Trena Ezzati-Rice, M.S.
Marc Zodet, M.S.








Need for essential data on health care utilization to inform health care policy and practice
Description of the Medical Expenditure Panel Survey
(MEPS): purpose, design and analytical capacity
Nonresponse and post-stratification adjustments
Recent survey design modifications:
(1) CAPI upgrade; (2) Sample Redesign
Evaluation of impact of design modifications on health care utilization estimates
Impact of design modifications on model-based analyses of health care use
Discussion

Annual Survey of 14,000 households: provides national estimates of health care use, expenditures, insurance coverage, sources of payment, access to care and health care quality
Permits studies of:
 Distribution of expenditures and sources of payment
 Role of demographics, family structure, insurance
 Expenditures for specific conditions
 Trends over time





Survey of U.S. civilian noninstitutionalized population
Sub-sample of respondents to the National Health
Interview Survey (NHIS)
Oversample of minorities and other target groups
Panel Survey – new panel introduced each year
– Continuous data collection over 2 ½ year period
– 5 in-person interviews (CAPI)
– Data from 1st year of new panel combined with data from 2nd year of previous panel

MEPS Household
Component
MEPS Panel 13 2008-
2009
NHIS
2007
1/1/2008
Round 1
1/1/2009
Round 2 Round 3 Round 4 Round 5
NHIS
2008
Round 1 Round 2 Round 3 Round 4 Round 5
MEPS Panel 14
2009-2010







Tool Chest of Methods to Maximize
Survey Response









Recruitment of experienced interviews and bilingual
10+ days training (including procedures for obtaining signed consents)
Uses of MEPS data as reference materials for interviewers
Periodic retraining and special trainings (e.g. methods to improve response rates)
Respondent remuneration
Advance mailings from co-sponsors of survey
Monthly planning calendar and MEPS DVD
Daily emails to interviewers regarding interviewing progress
Multiple contacts for refusal conversions

 Multiplicative response rates (RR): product of


NHIS RR and
MEPS RR (multiplicative function of round specific
RR):


MEPS rounds 1-3 of new panel (YR1 estimates)
MEPS rounds 3-5 of old panel (YR2 estimates)





Overall annual RR (~65%)
Highest RR 1 st year, new panel (~66-71%)
Lowest RR 2 nd year, old panel (~63-65%)
Post-survey nonresponse adjustments
–
–
Dwelling unit level
Person level survey attrition

NHIS variables used as potential covariates in forming DU
NR adjustment cells
Demographic
Age
Household
Characteristic
DU size
Socio-
Economic
Status
Geographic Health
Poverty status Census region Health status
Race/ethnicity Has phone Education
Marital status Working/reason not work (e.g., attending school, retired, etc,)
Gender Type of PSU
Income
Employment status
MSA size
MSA/nonMSA
Need help
Urban/Rural
Any Asian
Any Black

NEW NHIS variables added as potential covariates in forming DU NR adjustment cells
Demographic
Household
Characteristic
Socio-
Economic
Status
Interview language
Type of home – house, Apt., etc.
Category of medical expense
U.S. Citizenship Time no phone Home ownership
Health
Number of nights in hospital
Healthcare coverage
Born in US


Within each adjustment cell:
A(c) =
 i
 c
 i
 c
E ( i ) W 1 ( i )
R ( i ) W 1 ( i ) ratio of the sum of weights of all eligible (E) units in the cell to the sum of weights of only the respondents (R) in the cell

Person Level Adjustments:
Annual Estimates




Person Level (survey attrition)
Nonresponse Adjustment Covariates
 Factors associated with survey attrition (after R1)
– Indicator for initial refusal to R1interview
–
–
–
–
–
–
–
–
–
–
–
Family size
Age
MSA, census region
Marital status (family reference person)
Race/ethnicity
Education of reference person
Employment status
Health insurance status
Total expenditures (in yr 1 for yr 2 adj.)
# doctor visits (in yr 1)
Self reported health status

2009
Round 1 Round 2
2010
Round 3 Round 4 Round 5
Individuals in the
2009 sample with positive weights that left the civilian population prior to
2010, with no return
&
2009 sample also responding in
2010 with complete information for both 2009 and
2010





Re-engineered CAPI Interview: Windows-based
Platform replaces DOS-based system for Panel 12
New NHIS Sample Design Introduced in 2006: MEPS
Panel 12 selected from redesigned NHIS sample
Year 2 of MEPS Panel 11 based on original MEPS survey design
The overlapping panel structure in MEPS allows for a comparison of survey estimates across the alternative designed for the same time period

Evaluation of Concordance of Healthcare
Utilization Estimates: Comparison of results from new and original designs




MEPS has overlapping panel design: 1st year of new panel combined with data from 2nd year of previous year’s panel to yield annual data
Multiplicative response rates: product of NHIS RR and
MEPS RR (multiplicative function of round specific RR:
3 rounds for new panel/5 rounds for old panel)
Detailed adjustments for survey nonresponse and poststratification:
Compare 2007 health care utilization estimates based on new design (MEPS Panel 12 – Year 1) with original design (MEPS Panel 11-Year 2)

Testing for Survey Redesign Effects
Comparisons of panel specific national health care utilization estimates derived from the MEPS for the following health care



 services:
 ambulatory visits (office- based visits and outpatient facility visits) in-patient stays
ER visits dental visits prescribed medicine purchases
For the overall population, and further subset by age classification (0-17, 18-64, 65+)
 Model-based tests for survey redesign effects

Capacity of MEPS to Produce
Comparable NHIS Estimates of
Health Care Utilization
The following NHIS measures of health care utilization were selected in support of these analyses:
 Have you been hospitalized OVERNIGHT in the past 12 months?
(yes; no; refused/not ascertained/DK)
 How many different times did you stay in any hospital overnight or longer DURING THE PAST 12 MONTHS? (#; refused/not ascertained/DK)
 Altogether how many nights were you in the hospital DURING THE
PAST 12 MONTHS? (#; refused/not ascertained/DK)
 During the past 12 MONTHS did you receive care from doctors or other health care professionals 10 or more times? Do not include telephone calls. (yes; no; refused/not ascertained/DK)
 DURING THE PAST 12 MONTHS, have you delayed seeking medical care because of worry about the cost? (yes; no; refused/not ascertained/DK)
 DURING THE PAST 12 MONTHS, was there any time when you needed medical care, but did not get it because you/the family couldn't afford it? (yes; no; refused/not ascertained/DK)

Options for aligning redesign-based estimates with the original design
Option
Period of
Applicability
Restrict time trend analyses to sample with old design
Implementation of measure specific adjustments to a set of estimation weights
Implementation of
“bridging” adjustments to the primary survey estimation weight
No additional adjustments for redesign
Year(s) with overlap between redesign and prior design
When the survey redesign is implemented and subsequent years
When the survey redesign is implemented and subsequent years
When the survey redesign is implemented and subsequent years
Constraints
Loss in precision
Introduction of greater variability in resultant survey estimates ; complicates model based analyses
Dependence on within survey adjustments or availability of comparable external data source for national control totals
Possibility of differences detected in trends partially attributable to redesign







Need for accurate and reliable national data on health care utilization to inform policy and practice
MEPS design features and analytical capacity
Statistical, methodological and operational design features to adjust for nonresponse and attrition
Evaluation of impact MEPS redesign on health care utilization estimates
Impact on model based studies
Some evidence of redesign effect




MEPS includes a linked survey of medical providers for expenditures: use of medical event information to evaluate household reports of health care use
MEPS data periodically linked to Medicare claims data for evaluations: permits examination of accuracy of household reported data
Implement additional improvements to the CAPI interview and enhanced post-survey adjustment strategies