15
3. Evaluation Methods and Data Sources
This chapter describes the methods we used to evaluate TSSD, along with
evaluation components we had included in our original evaluation plan but
dropped by agreement with the study’s sponsor.
Overview of Evaluation Activities
At the beginning of the evaluation, RAND met with TMA project staff to develop
and finalize an evaluation plan in order to meet the evaluation goals specified in
the 1999 NDAA (and listed in Chapter 2). In practice, two factors required us to
revise our initial evaluation plan. The first, discussed in greater detail later in this
chapter, was the low rate of enrollment in the demonstration (total confirmed
enrollment was 344 in September 2000 and 355 in November 2000, out of an
eligible sample of approximately 11,000). The second factor was the passage of
TFL, which substantially changed the context in which TSSD was being
conducted. This section outlines our evaluation activities, along with activities
that had been planned initially.
Briefings with Program Staff
RAND’s evaluation began in May 2000. We began by meeting with TMA staff
including Duaine Goodno, the TMA staff person responsible for overseeing the
demonstration. TMA staff provided background materials on TSSD, including
printed materials that had been distributed to eligible beneficiaries. Because
RAND’s evaluation began after the start of the demonstration, and particularly
after the majority of TMA’s efforts to publicize TSSD to eligible beneficiaries, we
relied on TMA staff to describe the dissemination efforts.
TMA’s publicity efforts were concentrated in the months preceding the initial
enrollment period. The Iowa Foundation for Medical Care (the DoD contractor
responsible for administering the demonstration) developed a database, drawn
from the Defense Enrollment Eligibility Reporting System (DEERS), of all eligible
beneficiaries in the demonstration areas. The Iowa Foundation mailed
informational materials about TSSD to all eligible beneficiaries, and beneficiaries
could obtain additional information by telephone. Mr. Goodno and
representatives from the Iowa Foundation also visited the two demonstration
16
areas and conducted “town meetings” at various locations to inform eligible
beneficiaries about TSSD.
TMA informed us that they stopped marketing the demonstration to
beneficiaries after the demonstration began because of administrative difficulties
in administering the benefit and processing the claims (Goodno, 2000).
Automated Data Collection
TMA received monthly reports regarding TSSD enrollment and disenrollment
from the Iowa Foundation for Medical Care. TMA forwarded these reports to us
on a monthly basis. In addition, in preparation for direct data collection activities
with beneficiaries, we requested and obtained data on all enrolled beneficiaries
and on all eligible beneficiaries from the Iowa Foundation. These data are
described in greater detail later in this chapter.
Our original evaluation design included a plan to obtain medical and pharmacy
claims data on TSSD enrollees and on eligible beneficiaries from the Iowa
Foundation, the DoD, and Medicare. However, the low enrollment in TSSD
inhibited meaningful quantitative comparison between TSSD enrollees and
nonenrolled eligibles (in particular due to the very skewed nature of health care
expenditures, which makes a small sample very susceptible to outliers).1 In
addition, following the introduction of TFL, RAND and TMA agreed to restrict
the scope of the evaluation and shorten its duration. Therefore, with the
agreement of TMA, we eliminated the claims analysis.
Primary Data Collection
We conducted focus groups with TSSD enrollees and nonenrolled eligible
beneficiaries to collect information about their attitudes toward the
demonstration, their reasons for enrolling or remaining unenrolled, and
information on other factors related to their enrollment. In addition, we
conducted a mail survey of TSSD enrollees and nonenrolled eligible beneficiaries.
Focus group and survey activities are described in greater detail in the remainder
of this chapter.
We had originally anticipated sampling beneficiaries who had enrolled and then
disenrolled from TSSD (for reasons other than death or relocation). However, the
number of disenrollees was very small and did not support a separate analysis.
________________
1See, for example, Sturm, Unutzer, and Katon (1999).
17
In the remainder of this chapter, we describe the design of the focus groups and
the beneficiary survey.
Focus Group Design
There were two main goals for conducting focus groups as part of the TSSD
evaluation: (1) to obtain qualitative information about the reasons why eligible
beneficiaries enrolled or did not enroll in the demonstration program and to
obtain opinions of and experiences with the demonstration in particular and the
military health system in general and (2) to pilot test a survey questionnaire prior
to administration to a larger sample of eligible beneficiaries.
Site Selection
Focus groups were conducted in each of the two demonstration areas: Santa
Clara County, California, and Cherokee County, Texas. Selection of focus group
sites in Santa Clara, California, and Longview, Texas, was based on the density of
recently enrolled beneficiaries within a 15-mile radius, the availability of
adequate facilities for conducting the focus groups, and the ease of access for
participants.
Recruitment
Separate focus groups were used for TSSD enrollees and nonenrolled but eligible
beneficiaries. The separate groups were used for two reasons: (1) because of the
different relationships to the demonstration program of the two groups and (2) to
avoid the discussion from turning into an informational meeting for
nonenrollees.
For both sites, 30 enrollees and 30 nonenrollees were identified whose primary
residence was within 15 miles of the focus group location. Data about
beneficiaries were provided by the eligibility and enrollment files maintained by
the Iowa Foundation for Medical Care. In several cases, more than 30
beneficiaries met these criteria, in which case 30 were selected randomly. For
each focus group, sponsors2 were twice as likely to be sampled as spouses were
to ensure that no more than nine spouses would be included in any one focus
group (to meet Office of Management and Budget [OMB] requirements).
_________________
2 “Sponsors” in this context refers to persons whose military career qualifies them and their
eligible dependents for health benefits.
18
Potential focus group participants received a recruitment letter from RAND
accompanied by an endorsement letter from the study sponsor (see Appendixes
A and B for sample letters). The endorsement letter was then followed by a
phone call from RAND to confirm participation.
The goal was to confirm 12 participants for each focus group, with the
expectation that 8 to 10 of them would actually attend. Recruitment and
participation goals were met or surpassed for all focus groups (see Table 3.1).
Discussion Structure and Content
The focus group was divided into two parts: (1) the pilot test of the survey
instrument and (2) a discussion guided by a set of interview questions developed
in advance of the sessions (see Appendix C). Each focus group session began
with introductions and a description of the purpose of the pilot test before the
discussion commenced. Confidentiality issues were discussed and participants
were reminded of the voluntary nature of their participation. RAND’s Human
Subjects Protection Committee approved the focus group protocol (see
Appendix C).
Mail Survey Design
The main goal for conducting a beneficiary survey as part of the TSSD evaluation
was to understand enrollment patterns by identifying characteristics that
distinguished enrollees from nonenrollees. We were also interested in assessing
beneficiaries’ stated reasons for enrollment or disenrollment, beneficiaries’
experiences with military-sponsored health care, and enrollees’ experiences with
TSSD.
In practice, our evaluation began after nearly all of the beneficiaries who would
eventually enroll in TSSD had enrolled, and our survey was conducted after most
enrollees had been participating in the program for more than six months.
Because we were concerned about the accuracy of retrospective reporting of
outcomes such as health care use, we were unable to collect data on health care
use preceding the availability of, or enrollment in, TSSD.
19
Table 3.1
Focus Group Participants
TSSD Site
Santa Clara County, California
Cherokee County, Texas
Focus Group
Date (2001)
January 23
January 25
Number of Participants
Enrollee
Nonenrollee
Focus Groups Focus Groups
16
8
15
12
We initially proposed reinterviewing survey respondents in the second year of
the demonstration to assess their experience with TSSD. However, this option
was dropped due to the low enrollment in TSSD and due to the introduction
of TFL.
Questionnaire Development
Development of the mail survey began in September 2000 with the identification
of domains that would be examined in the survey. A draft instrument was
developed using several sources for questions including the Medicare Current
Beneficiary Survey and the Health Care Survey for Medicare-Eligible Military
Retirees and Dependents. In particular, health status and service utilization items
from these surveys were adapted to the TSSD questionnaire.
The survey was designed to elicit information on the respondent and his or her
spouse’s use of health care services, their stated preferences for health plan
features, their current health insurance coverage, their knowledge of and
experience with TRICARE, their attitudes toward military health care, and their
health status (see Appendix E for a sample of the survey). Demographic
information (e.g., income, education, and age) was also collected. The
questionnaire was divided into seven sections as described in Table 3.2. In
addition, the final page of the survey invited respondents to share any other
comments they might have.
With the exception of the questions regarding health insurance coverage, the
questionnaires for the two survey sites were identical. For California, the section
of the questionnaire regarding the respondent’s health insurance coverage asked
specifically about Medicare HMO’s, a health insurance option not available to the
Texas study population.
20
Table 3.2
Topics Covered by TSSD Mail Survey
Section
A
Topic
Use of health care services by respondent
B
Opinion regarding health plan features and benefits
C
Current health insurance coverage of respondent
D
Respondent’s knowledge of and participation in the TSSD program
E
Health status of respondent
F
Demographic information
G
Information regarding spouse’s use of health care services, health plan
coverage, participation in the TSSD program, and health status
Pilot Testing
To evaluate the questionnaire, a pilot version was administered to four focus
groups that included 51 TSSD enrollees and eligibles. At the beginning of each
focus group, participants were asked to complete the pilot questionnaire. They
were instructed to flag any unclear or hard-to-answer questions with the page
marker stickers that were supplied and to write any comments or questions on
the margin next to the specific item. Participants were asked to spend up to 30
minutes on the pilot questionnaire. The participants were also told that they
would have an opportunity to discuss with us at the end of the focus group
session any questions or concerns they had regarding the questionnaire. The
guided discussions that followed the pilot test focused on the factors that
influenced participants’ choice of health plans, their reasons for enrolling or not
enrolling in TSSD, and their level of satisfaction with TSSD.
All participants in each of the four focus group sessions completed the survey.
Participants completed the survey in 30 to 35 minutes, leaving roughly 40
minutes for discussion. Completion time did not appear related to enrollment
status or demonstration site. Participants did not voice objections to the survey
content or specific questions during or after the general discussion. None of the
participants chose to stay after the end of the discussion to discuss the
questionnaire in more detail.
Participants in each of the four groups marked questions requiring clarification.
For example, some were confused as to whether MTF visits for medical treatment
included pharmacy visits. Several areas in which participants had trouble
following skip patterns were noted in reviewing completed pilot-tested
questionnaires. The final version of the instrument was revised and simplified in
light of these findings. The project team noticed that some of the respondents had
21
difficulty turning the pages of the questionnaire. On the basis of this observation,
the final version of the survey instrument was bound to ease page turning.
Other refinements to the instrument’s language and skip patterns, as well as
refinements to the order of sections and items within sections, were made prior
to the main data collection.
Sampling
Our sample was drawn from two separate lists. The Iowa Foundation for
Medical Care supplied the project team with data on individuals enrolled in the
TSSD and on all beneficiaries eligible to enroll in TSSD. Core data elements came
from DEERS, with enrollment status and updated contact information added by
the Iowa Foundation. The files were current as of September 6, 2000, the date of
the extracts.
Given the relatively small number of TSSD enrollees per site, all households with
enrolled sponsors were sampled. Households with enrolled spouses but without
enrolled sponsors were sampled subject to OMB regulations; these rules required
us to contact no more than nine enrolled spouses directly without prior OMB
approval of the data collection instrument and survey activities. Therefore, for
the 15 spouses enrolled in TSSD and living in households with eligible but
unenrolled sponsors, we surveyed the sponsor (the data collection instrument
included a section on spouse’s characteristics).3 The enrollee sample included
focus group participants because of the small total number of enrollees.
For the TSSD eligible group, only households with at least one eligible sponsor
were sampled. To comply with OMB regulations, households with eligible
spouses but no eligible sponsors were excluded from the study. Only one eligible
sponsor per household was included in the sampling frame. Focus group
participants and eligibles with enrolled spouses were excluded from the
sampling frame.
Table 3.3 describes the sampling frame from which the study sample was
selected. All 203 enrolled households (including 15 with enrolled spouses and
nonenrolled sponsors) with adequate sponsor information—meaning that the
_________________
3A small number of TSSD enrollees consisted of spouses with no (eligible) sponsor in the
household, either because the sponsor was not Medicare-eligible or because the sponsor was
divorced or deceased. We did not survey these enrollees.
22
Table 3.3
Size and Composition of TSSD Sampling Frame and Sample
Total
Households
California
Texas
Total
California
Texas
Total
75
174
249
5,866
2,379
8,245
Spouse
Excluded:
Only
No Sponsor
Information
Enrolleda
Enrolleesc
18
7
28
8
46
15
Eligiblesd
1,840
N/A
584
N/A
2,424
N/A
Households
in Sampling
Frameb
Households
Sampled
57
146
203
57
146
203
4,026
1,795
5,821
825
825
1,650
aHouseholds in which the spouse has the same address as the sponsor.
bEqual to “total families” minus “families with no sponsor information” in the data files.
c Households in which either the sponsor or the spouse is enrolled in the TSSD program. Only the
sponsor was eligible to receive the survey questionnaire. Only one sponsor per household was
selected. Households that participated in the focus groups were not excluded.
dHouseholds in which neither the sponsor nor the spouse is enrolled in the TSSD program. Only
the sponsor was eligible to receive the survey questionnaire. Only one sponsor per household was
selected. Households that participated in the focus groups were excluded.
data contained at least one name and a complete mailing address, and in the case
of households with enrolled spouses, the spouse’s address had to match the
sponsor’s address—were included in the sample. A random sample of 1,650
eligible households was selected from the 5,821 households in the sampling
frame.
Data Collection
Data collection from the beneficiary survey began in March 2001 and was
completed in June 2001. The study packet was sent to the sponsor of the selected
household asking him/her to complete the questionnaire. The study packet
included an advance letter, hard-copy questionnaire, and postage-paid return
envelope. The packet also included an endorsement letter from the study
sponsor. The enrollee sample was also divided into groups depending on
whether the household had been invited to participate in the focus groups or not.
A different advance letter was used for these two groups. (See Appendix A for
samples of the recruitment letters.)
Table 3.4 lists the mailings to potential respondents, the dates of the mailings,
and the response rates. Cases returned as undeliverable by the U.S. Postal Service
(USPS) were not tracked any further. Respondents were deemed ineligible if they
23
Table 3.4
Response Rates by Fielding Task
Fielding Task
First mailing
Reminder letter
Second mailing
Third mailing
Sample Size
1,853
1,331
981
356
1,853a
Total
Dates (2001)
3/1–3/12
3/15–3/26
3/27–4/5
6/20–6/21
Response
210 (11%)
479 (36%)
454 (46%)
79 (38%)
3/1–6/21
1,222 (66%)
aAs we describe later in this chapter, some beneficiaries were dropped from our sample frame
due to death or relocation.
were deceased or moved out of the demonstration area (based on the list of Zip
codes in the TSSD catchment area).
Phone prompts to nonrespondents (approximately 400 cases) were conducted
from mid-May through mid-June. Nonrespondents received an average of two
calls during this time period. Nonrespondents without phone numbers were
tracked through directory assistance or a Web locator search engine. A number of
survey packets were remailed as a result of the phone prompts. Completed
surveys believed to have been returned in response to the third mailing pushed
the overall response rate from 68 percent to 73 percent. Most of the converted
nonresponses were from respondents in the TSSD eligible study group from
both sites.
Table 3.5 provides a breakdown of survey participation. The response rate
among TSSD enrollees was 95 percent (187 out of 197), whereas among TSSD
eligibles it was 70 percent (1,035 out of 1,488). The undeliverable rate was
Table 3.5
Response Rates by Demonstration Site and Sample Type
Sample
California
Enrolled
Eligible
Texas
Enrolled
Eligible
Total
N
882
57
825
971
146
825
1,853
Complete
552
52
500
670
135
535
1,222
Deceased
57
1
56
47
3
44
104
Out
of
Areaa
31
0
31
33
2
31
64
Refusal
32
1
31
39
0
39
71
Other
Nonresponseb
210
3
207
182
6
176
392
Response
Ratec
70%
93%
68%
75%
96%
71%
73%
aCases no longer living in the program catchment area as determined by the zip codes of their
new addresses provided by the USPS.
bIncludes cases whose study packet was returned undelivered by the USPS because of a wrong
address and for which no new address was provided; (n=140).
c Based on a study-eligible sample (excluding “Deceased” and “Out of Area”); (n=1,685).
24
8 percent of the sample eligible to respond to the survey. The out-of-area cases
and deceased cases represented 3 percent and 6 percent of the entire sample,
respectively. Overall, these cases (undeliverable, out of area, and deceased) were
distributed fairly equally among the two sites but were more likely to occur
among the TSSD eligible group than among the TSSD enrollee group because the
contact information in the enrollee group was more likely to be current. Four
percent of the sample who were eligible to respond to the survey actively refused
to participate; about half of those cases were too old or too sick to complete the
survey, while the remainder said they were not interested. Twelve cases returned
separate written comments on various issues related to their health care
coverage. An additional six completed surveys were received from TSSD-eligible
respondents in California after the data collection period had ended.
We note that 58 percent (109 out of 187) of TSSD enrollees and 40 percent (405
out of 1,035) of TSSD eligibles provided some kind of write-in comments on the
final page of their survey instrument. Anonymous survey comments on DoD
health care policy are listed in Appendix F.
Data Quality
In this section, we discuss three issues related to the quality of the survey data.
Accuracy of Self-Reported Data. One issue related to the survey’s data quality
concerns the accuracy of self-reported TSSD enrollment. Of the 1,045 respondents
we identified as not being enrolled in TSSD based on DEERS data, 33 reported on
the survey that they were enrolled in TSSD (and, in general, answered the TSSDspecific questions in the survey). Similarly, of the 177 respondents we identified
as enrolled in TSSD based on DEERS data, 9 reported on the survey that they
were not enrolled in TSSD. These discrepancies may reflect a change in TSSD
enrollment status between the time of the data extract and our field period
and/or confusion between TSSD and TFL, which was generating substantial
publicity especially in the second half of our field period. Finally, a small number
of respondents identified themselves as uncertain as to whether they were
enrolled in TSSD. In practice, however, the few who actually were enrolled,
according to DEERS, completed the survey using the instructions for enrollees.
Table 3.6 provides more detail regarding DEERS versus self-reported data on
TSSD enrollment. In practice, our analyses of survey data and health insurance
3
Spouse’s enrollment uncertain
31
1,048
0
3
82
5
4
Enrolled
123
18
540
27
19
494
Texas
Nonenrolled
25
NOTE: Shaded cells indicate a conflict between sponsor reports and DEERS/Iowa Foundation data.
28
9
140
33
1,042
107
Spouse enrolled
Spouse not enrolled (or no spouse)
33
994
26
985
6
Total
195
9
Total
Nonenrolled
33
Sponsor’s enrollment uncertain
Enrolled
162
Sponsor not enrolled
Sponsor’s Reports Regarding
TSSD Enrollment Status
Sponsor enrolled
18
543
109
24
498
Total
148
Enrollment Patterns Among TSSD Survey Respondents (N=1,222)
Table 3.6
3
6
25
1
5
Enrolled
39
10
502
6
8
491
California
Nonenrolled
8
13
505
31
9
496
Total
47
25
26
choice (described in Chapter 5) excluded beneficiaries whose TSSD enrollment
status was inconsistent with DEERS data.4
Survey Nonresponse. A second issue related to the quality of the survey data is
the rate of nonresponse. The overall rate of response of 73 percent seems
acceptable, particularly for an elderly population. However, the response rate
was significantly higher for households that included one or more TSSD
enrollees than it was for those that did not. To examine these issues, we
estimated a logistic regression using all sampled beneficiaries (excluding those
we could identify as being deceased or having relocated out of the demonstration
areas). The dependent variable was TSSD survey response. Explanatory variables
included demographic variables available from the administrative data—i.e., age,
race, gender, marital status, and state of residence—and TSSD enrollment status.
Table 3.7 provides predicted nonresponse rates associated with particular
beneficiary characteristics based on the parameter estimates from this logistic
regression. Each prediction is standardized for the values of all other covariates
in the model.
Table 3.7 indicates that response rates rose significantly with age, that men were
substantially more likely to complete the survey than women (although p > 0.05),
and that unmarried respondents were significantly more likely to return the
survey than those who were married. In addition, beneficiaries in Texas were
significantly more likely to return the survey than those in California. Finally, the
response rate was significantly higher for sponsors who were enrolled in TSSD
relative to unenrolled sponsors with or without an enrolled spouse.
Differences in response rates, along with other unobservable differences between
eligible beneficiaries who did or did not enroll in TSSD, respectively, will
complicate any comparisons of insurance-related outcomes between enrollees
and eligibles. Results of such quantitative analyses should therefore be
interpreted cautiously.
Item Nonresponse. A third issue regarding the quality of the survey data is item
nonresponse. Nonresponse varied by item but in general was low. For
demographic characteristics and health characteristics, item nonresponse
occurred with less than 5 percent of the respondents, and even for a relatively
sensitive question such as one regarding household income, only 8 percent of
respondents failed to give an answer.
________________
4A small number of sponsors also reported that they were not eligible for Medicare, even
though they were military retirees and age-eligible for Medicare. We also omitted these beneficiaries
from most analyses.
27
Table 3.7
Predictors of Survey Nonresponse
Fraction Not
Responding
0.30
0.28
P-value of
Contrast
0.527
White
Other
0.29
0.31
0.481
Age 65–74
Age 75–84
Age 85 and older
0.32
0.28
0.24
0.000
Female
Male
0.48
0.29
0.098
Unmarried
Married
0.13
0.32
0.000
California
Texas
0.33
0.25
0.002
Only sponsor enrolled
Only spouse enrolled
Sponsor and spouse enrolled
Neither enrolled
0.09
0.33
0.13
0.30
0.000
Sponsor Characteristic
Nonofficer
Officer
NOTE: This table includes the predicted fraction of nonrespondents
standardized for the covariates in the table. The sample includes all
beneficiaries to whom we mailed a survey, excluding those who we
subsequently learned were deceased or had moved out of the TSSD
catchment area; (n=1,681).
Rather than delete a large number of cases with missing data, we imputed values
for the missing items. For data elements, particularly demographics, that were
available from DEERS, we imputed missing data using that information. To
preserve sample size, particularly among TSSD enrollees, we imputed other key
measures for survey respondents with item-nonresponse data by using the
“impute” command in the Stata 7.0 statistical analysis package.5
_________________
5 For each variable to be imputed, we specified a list of predictor variables, including a core set
of demographic characteristics available on all beneficiaries from DEERS. Stata used these measures
as explanatory variables in ordinary least squares regression models, with the measure to be imputed
as the dependent variable. The imputed value was the prediction from the regression model for
beneficiaries with a missing value of the dependent variable and nonmissing values of the predictors.