4
------------------------------------------------ ----------------------------------------------
Research Questions, Hypothesis & Analysis Plan
CAROLYN LICHTENSTEIN, L. JOSEPH SONNEFELD,
STEVE BANKS, BRIAN YATES
Outcome Study
The primary goal of the COSP MRI was to establish the extent to which the addition of
consumer-operated services (COSP) to traditional mental health services (TMHS) changes
selected consumer and cost outcomes for mental health consumers, that is, to answer the
following research question: To what extent does participation in a COSP program affect
selected consumer outcomes for consumers who use traditional service programs? The COSP
MRI GFA mandated that five outcomes (empowerment, employment, housing, social inclusion
and satisfaction with services) be examined. During the first year of the initiative, this list was
elaborated further by the Steering Committee, the Research Subcommittee, and several
Working Groups. Outcomes were specified as being “intermediate” or “long term;” all
intermediate outcomes are also considered as long-term outcomes, though not all long-term
outcomes are included as intermediate outcomes. In addition, discussions by the COSP MRI
Steering Committee led to the hypothesis that the greatest effect would be seen in a more
general construct, “well-being.” New consumer outcomes, subsumed under the broad heading
of “well-being,” were added to the multi-site study. These outcomes include hope, self-esteem,
meaning of life, social inclusion, subjective quality of life, social acceptance, reduced distress
with symptoms, and status of personal recovery. The following section briefly describes the
measures used for the GFA-mandated outcomes and the “well-being” measure as defined within
COSP MRI.
Outcome Measures
Empowerment
Three sub-domains of empowerment were specified:
1. Making Decisions (Rogers et al., 1997) - subsumes self-efficacy, self-esteem,
achievement, and control in life.
81
2. Personal Empowerment (Segal, Silverman, and Temkin, 1995) - includes the
extent to which respondents perceived choice in living situation, how they spent
their money, their leisure activities, and their help-seeking behavior, as well as their
view of the likelihood that they would have sufficient resources to meet basic needs
and wants.
3. Organizationally Mediated Empowerment (Segal, Silverman, and Temkin, 1995)
- subsuming leadership roles in, and influence over, the political processes of
organizations.
Housing
Because the GFA did not require the use of specific housing outcomes, several housing
measures were examined, including (a) stability of housing, (b) type of housing, and (c) type of
cohabitants.
Employment
Employment measures used in COSP MRI include holding a paying job at the time of
the participant interview and working for pay or on a volunteer basis at the time of the
interview.
Social Inclusion
Two scales were used to assess subjective and objective social inclusion, defined as active
participation in the life of one’s social network and community:
1. Objective Social Inclusion (Lehman, 1988) - measuring the extent to which the
respondents socialized by phone, letter or in person with people other than those in
which they may live.
82
2. Subjective Social Inclusion (Lehman, 1988) - measuring how respondents feel
about the amount of social contact in their lives.
The addition of this outcome to the study reflected the notion that consumer-operated services
might be particularly successful in increasing participants’ abilities to integrate themselves into
their chosen communities, in part, through reducing stigma.
Satisfaction with Services
The GFA-specified domain of consumer satisfaction with services was expanded to include
consumer-reported outcomes of services. Both consumer-rated satisfaction and outcomes were
rated somewhat differently for traditional and consumer-operated programs.
1. Satisfaction with Services (MHSIP, 1996) - assesses participants’ satisfaction with
both TMHS and COSP services by examining various aspects of the program as a
whole, staff members, and treatment approach.
2. Outcomes of Services (MHSIP, 1996) - measures the extent to which participants’
involvement in TMHS or COSP had a significant impact on improvements in the areas
of housing, family relations, social situations, physical health care, etc.
Well-being
The broad construct of well-being was theoretically justified by a review of the general
literature on well-being and the research on peer-support programs and consumer/survivor
recovery. A composite well-being measure was therefore constructed by combining several
outcome measures included in the common protocol: (a) two measures of empowerment
(personal and for making decisions); (b) hope, (c) social inclusion (subjective), (d) quality of life,
(e) meaning of life, (f) recovery, (g) social acceptance, and (h) symptoms. To construct this
measure, the baseline data (collected prior to randomization) were explored. The distributional
properties of potential variables (including ceiling and floor effects) were examined, as well as
83
their interrelationships (through a factor analysis) and correlations with self-reported level of
prior engagement in consumer-operated programs. The development of this measure is
described in more detail in Chapter 7 of this report.
Hypotheses
The randomized design employed by all sites allows for a traditional randomized
clinical trial (RCT) approach to data analysis, so the COSP MRI Steering Committee decided
that the first analysis would be an intent-to-treat approach that tests the primary hypothesis
using data from all sites:
Primary Hypothesis: Participants in the group offered both traditional and consumeroperated services will show greater improvement in well-being over the course of the study than
will participants offered only traditional mental health services.
The decision to focus on this primary hypothesis was based on many discussions about all of
the outcomes specified in the GFA. Steering Committee members felt that participants’
subjective feelings about themselves and their lives, as reflected in a well-being measure
incorporating two of the GFA measures plus related constructs, would be the most likely to
change in the short-term timeframe of the COSP MRI.
Because participation in the COSP programs was voluntary and the COSP programs
were also available to study participants who had been randomly assigned to the control
(TMHS-only) condition, random assignment to the experimental (TMHS+COSP) condition
was expected to have a less-than-perfect correlation with participants’ actual engagement in the
consumer-operated services. To address this likelihood, an “as-engaged” analytical approach
augmented with a propensity score approach was also taken in order to produce estimates of
the effects of actual participation in COSP programs.
84
Secondary Hypotheses
Several secondary hypotheses were developed to be examined subsequent to the primary
hypothesis test:
1. Improvement in well-being is related to particular aspects of the interventions
2. Participants who actually attended the COSP intervention will show greater
improvement in well-being than participants who never attended.
Cross-Site Analysis Plan
The COSP MRI has three distinctive features that were addressed in the multi-site
evaluation plan:
1. Participants were randomly assigned to two experimental groups, or conditions.
2. The eight participating COSP programs implemented very different interventions of
at least three distinct types.
3. Participation in COSP programs was entirely voluntary, and COSP programs were
not closed to participants who were randomly assigned to the TMHS-only
condition.
As is the case for any analytic plan, attempting to reflect these features can lead to alternative
formulations of both the values that guide the analyses and of the statistical techniques to be
employed in the analyses. In many of the alternatives, similar scientific conclusions would be
reached, though the strength (observed p-values) might vary. In some cases, however, the
results could be scientifically different, indicating that they are dependent on the underlying
assumptions or the statistical model used. To guard against this, multiple techniques and
influence diagnostics were employed in the cross-site analyses.
In addition to the longitudinal analyses described below, a set of analyses were carried
out to examine the baseline data. These analyses focused on describing the characteristics of the
85
COSP MRI participants, using both univariate examinations of the Common Protocol (CP)
items and scales, and bivariate examinations of the interrelationships among these items. The
baseline data were also analyzed as if they represented a pilot test of the longitudinal data. In
other words, the baseline data were used to develop the well-being measure and to examine
relationships of this measure with other participant characteristics that might be considered
covariates in longitudinal analyses.
Intent-to-Treat (ITT) Analysis
In the ITT analyses, those assigned to the TMHS+COSP condition were compared to
those assigned to the TMHS-only condition to estimate treatment effects, regardless of their
actual engagement in the program. These analyses assessed the primary hypothesis and the
first secondary hypothesis. In keeping with the concepts associated with a prospective RCT,
this analysis adopted a p-value of .05 for the test of the primary hypothesis. To accomplish this,
the well-being composite measure was selected as the primary endpoint in the intent-to-treat
analysis, rather than testing each of several separate measures. A similar composite-measure
approach has been used in a number of RCT studies (e.g., Quezado et al., 1998) and is common
in human service evaluations.
Multilevel, or random regression, models (Gibbons et al., 1993; Hedeker, Gibbons, &
Davis, 1991; Raudenbush & Bryk, 2002) were fit to the longitudinal data. These models allow
for the nesting of observations within individuals and individuals within sites, inclusion of
observations with some missing outcomes, fixed and/or random coefficients, modeling a
complex variance-covariance structure and the incorporation of time-dependent covariates. The
major advantages of random regression techniques are the regression-style format, the ability
to test the sensitivity of the results for specific variance-covariance structure, and the inclusion
of individuals who have some missing data in the analyses. To examine the underlying
86
assumptions associated with the statistical model, the residuals were tested for normality using
a Shapiro-Wilk (1965) test in the early stages of the modeling. Once the residual distribution
was determined to be normal, this test was not repeated for subsequent models.
The intent-to-treat model estimated was:
Well-being = Time + Group + Site + Participant + Group*Time + Site*Time
+ Site*Group + Site*Group*Time.
This is a three-level model, in which time is nested within participant, and participants are
nested within site. When testing the primary hypothesis, the interest is in the group-time
interaction. Time was modeled by not only a linear component but polynomial terms as well.
The random regression approach also allows for the examination of the characteristics of the
interventions at the different sites and their relationship with outcome variables.
It is necessary in interpreting pooled analyses to be careful not to draw spurious
conclusions; a particular danger given the many known sources of cross-site heterogeneity and
substantial differences in the number of participants across sites. The biases can be as extreme
as those seen in Simpson’s paradox (Hand, 1979) in which an overall effect is reversed in each
site. Even if the differences between pooled and site effects are not as extreme as Simpson’s
paradox, important modifications of effects can be missed or misreported if the heterogeneity
among sites is ignored (FDA 1998; Lechner, 2002).
Although higher-order interaction terms involving cluster and site were included in
each analytic model, the low power of tests for these terms (due to the small number of sites)
greatly reduces confidence in them as decision-making tools. Simple plots of the site effects
within each cluster were examined to add evidence to the specific statistical tests of these
interaction terms. Moreover, judgments were made about noteworthy modifications of effect,
relying on considerations such as whether there are patterns of effects that are of a magnitude
87
that would be judged practically important if statistically significant. In particular, any
heterogeneity of effects across sites or clusters in ways that would change conclusions in pooled
analyses were noted. If, for example, the two sites in a cluster had effect sizes greater than .2,
but in opposite directions, on a particular outcome, any possible explanations of the difference
(in terms of factors such as programs and populations) were explored.
Randomization may fail to balance the experimental groups on important baseline
variables, and differential attrition or loss to follow-up may occur across the two groups. Thus,
in the analytic dataset consisting of individuals who have any follow-up data, the groups may
differ on important baseline variables. The differential (TMHS-only compared to
TMHS+COSP) attrition rates were examined by such covariates as gender, racial/ethnic
background, diagnosis, educational attainment, current employment status, housing, whether
ever homeless, and current living situation. These attrition rates were computed for all
participants pooled together, within clusters, and within sites.
The dataset of COSP MRI participants with follow-ups was also examined for evidence
of “imbalance” between the treatment and control groups. That is, the equivalence of the
TMHS-only and TMHS+COSP participants was assessed, comparing the same set of
covariates examined in the attrition analysis. The attrition and baseline equivalence analysis
results were used to aid in the interpretation of the modeling results.
The final set of ITT analyses focused on the relationship of the comparative TMHSonly/TMHS+COSP change over time in well-being with characteristics of the TMHS and
COSP programs. For these analyses, the slopes from the multilevel modeling were correlated
with the FACIT scores assigned to the programs (see Chapter 5 for a description of the
FACIT). These analyses were intended to examine whether particular aspects of the programs
included in the COSP MRI were more closely related to changes in well-being over the course
88
of the study than other aspects, in an effort to better understand what might be driving any
observed improvement in well-being.
As-Engaged Analysis
As noted earlier, two factors might have affected the attendance at COSP programs of
participants assigned to either experimental group: the voluntary nature of the COSP
programs, and the availability of the COSP programs to individuals who were randomized to
the TMHS-only condition. An intent-to-treat analysis may produce overly conservative
estimates of treatment effect when little of the innovative service (i.e., COSP) is actually
received by participants assigned to the experimental group, or when members of the control
group attend COSP. For this reason, as-engaged analyses that reflected the values and
voluntary nature of the COSP MRI programs and might also be expected to produce a less
biased estimate of the treatment effect were implemented. The as-engaged analyses examined
the second of the secondary hypotheses.
Since as-engaged analyses compare those individuals who actually receive the
intervention (however that may be defined) to those who do not receive the intervention, they
can overestimate the true treatment effect because participants who are more likely to choose to
participate in the intervention are also more likely to benefit from the intervention. The
analyses therefore accounted for this self-selection as much as possible, using the best statistical
adjustment methods available.
The first as-engaged analysis was to classify participants as “engaged” or “not engaged”
and construct an “engagement” variable so that the analytic strategies used in the intent-totreat analysis could be repeated. The proportion of those assigned to the TMHS+COSP
experimental group who were actually engaged in those programs was also estimated. Two
sources of data on participants’ engagement or level of participation in the COSP programs
89
were collected:
1. Logs of visits to the program maintained by program staff for the cost study
2. Participant self-reports at follow-up, including both (a) a single item indicating
whether a participant had visited the COSP program being studied at least once in
the previous 4 months, and (b) his or her level of participation in 17 categories of
program activities at that COSP program.
In some sites (especially drop-in centers), the visits data were known to under-report actual
visits. Other sites believed that there was substantial over-reporting by participants of
involvement in program activities at the COSP program, due not only to the usual telescoping
problems with retrospective self-reports, but because of misidentification of the COSP or
participants’ inability to distinguish between those activities that were components of the
COSP program under study and incidental contacts with the sponsoring agency.
The two data sources measuring engagement were cleaned, merged and analyzed to
determine the most appropriate measures of engagement. Individual-level comparisons of visits
and program-activity data were conducted, including correlations and examination of
distributions overall, by cluster, and by site. The analyses implemented to determine and
compute the measure of engagement used in the as-engaged analyses are described in detail in
Chapter 8. In addition to lack of participation by those assigned to the TMHS+COSP
experimental group, participants assigned to the TMHS-only experimental group were free to
attend the COSP program. These “cross-over” study participants were identified and their
presence considered in developing the engagement measure (see Chapter 8).
The as-engaged groups defined by “participated in intervention” vs. “did not participate”
were examined in much the same way as were the designed intent-to-treat groups. The
differential engagement rates were examined by participant characteristics to better understand
90
whether engagement occurred at higher rates among particular subgroups of participants. The
characteristics examined included gender, racial/ethnic background, diagnosis, educational
attainment, current employment status, housing, whether ever homeless, and current living
situation. The differential attrition rates for the two new groups were also examined.
To account for participant self-selection into the COSP programs, a method known as
“propensity scoring” was used. Rubin introduced propensity scoring in 1973 as a method for
eliminating overt biases that exist in observational studies (Rosenbaum & Rubin, 1983; Rubin,
1997; Little and Rubin, 2000). This method has also been employed to re-balance randomized
clinical trials after unsuccessful randomization or differential attrition. Propensity scoring is an
alternative to standard covariance adjustments for observed differences between groups on pretreatment variables. This procedure is particularly useful when the number of pre-treatment
variables to be adjusted for is large or their distributions vary substantially between treatment
groups—(i.e., circumstances in which conventional covariance adjustments may perform
poorly). The propensity score is just the pre-treatment probability of receiving the treatment,
given an individual’s pre-treatment values of the selected variables. This conditional probability
must be estimated from the data in the study. For the COSP MRI analysis, a logistic regression
was used to predict whether a study participant attended the COSP program. Baseline
variables, including demographic, clinical, and treatment history, will serve as the independent
variables in the regression.
A probability of engagement group membership was estimated for each individual from
the logistic regression. This probability represents the “propensity” of each participant in the
final dataset to have attended the COSP intervention at his/her site, based on all the observed
pre-treatment characteristics included in the logistic regression. Following the procedure
recommended by Rubin and Rosenbaum, these scores were then divided into 5 categories (each
91
containing approximately 20% of the study participants) and the balance of the covariates
examined within each stratum. The sample size for the COSP MRI was too small, however, to
support 5 strata with propensity scores that overlapped sufficiently for the engaged/not
engaged groups. The 5 strata were therefore collapsed into the three terciles of the propensity
score distribution. The terciles were defined for each site’s distribution separately. The effect of
the intervention was examined only for those individuals who had similar propensity scores,
that is, for each tercile separately.
Finally, in an effort to examine whether the “dose” of COSP attendance affected
participants’ improvement in well-being, the engaged group of participants were divided into
two groups representing high vs. low levels of engagement. The comparison of the three
groups consisting of (a) no attendance at the intervention COSP, (b) a small amount of
attendance at the intervention COSP, and (c) a larger amount of attendance at the intervention,
was examined to determine whether increasing involvement in COSP activities is related to
increasing improvement in well-being.
COST STUDY
The primary objective of the COSP MRI Cost Study was to measure, describe, and
compare monetary value of resources, monetary outcomes, cost-effectiveness, and cost-benefit
for Consumer-Operated Services (COSs) and Traditional Mental Health Services (TMHSs).
Additional objectives of the COSP MRI Cost Study were to (a) measure the extent of
participation in COSP and TMHS programs and in specific COSP and TMHS activities, and (b)
analyze relationships of costs to program operating characteristics. A description of the
approaches taken to achieve these analytic objectives follows.
Examination of Costs, Cost-Effectiveness and Cost-Benefits
The following major steps were taken to accomplish the first objective:
92
1. Measure and describe the monetary value (i.e., costs) of resources consumed during
the participation by specific consumers in COSP and TMHS programs participating
in the COSP MRI
2. Compare costs for consumers participating in different COSP programs and
different types of COSP interventions (i.e., Drop-In, Education/Advocacy, and Peer
Support);
3. Compare costs for consumers assigned to the TMHS-only condition and costs for
consumers participating in the COSP condition;
4. Measure monetary and monetizable outcomes of participation in TMHS and COSP
programs, in terms of increased income due to employment, as well as, reduction in
utilization of health and other services (e.g., inpatient hospitalization, emergency
room utilization, and crisis intervention), plus reduced costs for housing, criminal
justice, vocational rehabilitation, and income support;
5. Compare the monetary value of these outcomes for the two experimental groups;
6. Analyze the relationships between costs, participation in specific TMHS and COSP
activities, and monetary outcomes (cost-benefit analysis); and
7. Analyze the relationships between costs, participation in specific TMHS and COSP
activities, and nonmonetary outcomes (cost-effectiveness analysis) (Yates, 1977a, 1980,
1996, 1999, 2000; Yates, Delany, & Lockwood, 2001; Yates, Haven, & Thoresen,
1979; Yates & Newman, 1980a, 1980b).
Aggregation of Data
Because these measures of resources’ costs and monetary outcomes were collected at the
level of individual consumers, statistical descriptions of costs and monetary outcomes, along
with statistical comparisons, were performed for a variety of potentially illuminating levels and
93
types of aggregation. These levels of aggregation include; (a) by site, aggregated over
consumers and time; and (b) by type of COSP intervention (Drop-In, Education/Advocacy,
Peer Support), aggregated over consumers and time. These statistical aggregations
(descriptions and comparisons) were broken down by individual quarters of the COSP MRI’s
timeframe, to examine temporal variability within a given site on measures such as cost per
visit (cf. Yates, 1977, 1996). Other temporal distinctions examined include (a) regular versus
extended recruitment periods, and (b) periods when the program clearly was operating under
capacity versus at capacity (and over capacity, if that occurred).
Statistical Analyses
Additional statistical analyses, including whichever forms of regression modeling were
justified by the statistical properties of cost and monetary outcome measures (e.g., logistic or
multiple regression) were performed to examine the statistical significance, direction, and
strength of relationships between resources’ costs, specific program activities, and both
monetary and nonmonetary outcomes (Besteman, Greenfield, De Smet, Yates, & Filipczak,
1995). Monetary and nonmonetary outcomes were examined at both specific levels (e.g., health
care cost savings, empowerment enhancement) and as a composite index (e.g., total cost
savings, overall impact) (Yates, 1996, 1999).
Other analyses were conducted to examine the significance, direction, and strength of
relationships between the amounts of specific resources consumed, activities participated in,
psychological and social processes occurring within consumers, and monetary as well as
nonmonetary outcomes achieved. These cost → activity → process → outcome analyses
included path analyses and structural equation modeling as well as linear programming (Yates,
1980, 1996; Yates, Delany, & Lockwood, 2001; Yates, Besteman, Filipczak, Greenfield, & De
Smet, 1995; see also Cohen & Cohen, 1993; Jöreskog & Sörbom, 1989).
94
Relationships of Costs to Program Characteristics
The Cost Study was designed to examine the relationships between a variety of cost and
program operation measures, such as how the average cost per visit and consumer may vary as
a function of (a) the total budget, (b) the total number of consumers in the program, (c) the type
of COSP and TMHS programs, and (d) other explanatory variables (e.g., total program budget,
variations between sites in cost of living, method by which consumer-operated services were
delivered). In addition, analyses related differences in frequency, intensity, and range of
participation in COSP program activities to differences in the average costs of different types of
COSP (e.g., to examine whether more program activities were participated in by consumers
attending COSs with higher costs per visit).
Further Analyses
The analyses that were implemented, and whose findings are described in later chapters,
focused on testing the primary hypothesis. Many additional analyses are possible, however, that
explore the interrelationships among the many outcomes beyond well-being, levels of
engagement in the COSP intervention, and subgroups of participants with different
characteristics.
Examination of Covariates
Both participant- and site-level factors may affect the relationship between group
assignment (experimental or as-engaged) and outcomes. Characteristics thought to be related
to well-being, based on relevant research literature, could be investigated and included in the
multilevel models.
One particular analysis of this type would be to examine further the relationship
between FACIT scores and the domains of interest. This might be accomplished in one of two
95
ways. The first would be to construct a three-level model, with site (or condition within site) as
the third level, and incorporate the FACIT scores into the original random regression models.
Though conceptually possible, there are occasions when these models have difficulty estimating
parameters as the number of levels increase. An alternative method, which addresses the
central question of the relationship between FACIT scores and various outcomes, might be to
estimate the effect size within each site (or program within a site) and then perform a summary
analysis that examines the relationship between FACIT scores and the effect sizes. This
analytic technique is known as meta-regression and has been used in a number of settings (for
example, see Freeman, Gerstenberger, Banks, & Natanson, 2002). One example would be to
investigate whether there was a greater effect of participating in a COSP program for
subgroups of individuals. This could be accomplished by including interaction terms between
participants with particular characteristics and the group assignment variable. As noted by
Natanson, Esposito, and Banks (1998), there are statistical concerns in using an uncontrolled pvalue to identify a subgroup that experiences a treatment effect, while the overall study
population shows no effect. Consequently, appropriate methods for controlling p-values should
be used.
Growth curve modeling might be used to further examine differences across subgroups
of participants. Such modeling estimates each individual’s “growth curve” or change in outcome
over the course of the COSP MRI, and then identifies various subgroups of participants
exhibiting different patterns of change over time (e.g., classes of individuals who have positive
versus negative changes during the course of the study).
96
Examination of Outcome Measures
Outcomes Other Than Well-being
The well-being measure is a composite of several measures, each of which might exhibit
a different pattern over time, and have a slightly different relationship with the experimental or
as-engaged groups. It is possible that the outcomes most affected by actual participation in the
intervention (the as-engaged model) will not be the same as those most affected by being
invited to participate (the ITT model). For example, well-being might be considered more
appropriate for the ITT analysis than employment, which might be more suitable for the asengaged analysis.
In addition, the outcome measures other than well-being might be examined more
closely than they were for this report. Summary measures might be developed for housing and
employment that could then be examined over time, and within subgroups.
Further Exploration and Modification of Well-being Measure
Further research suggests several other ways to define a well-being measure using the
COSP CP outcome scales. First, Ryff and Singer (1996) describe six key dimensions of
“psychological well-being.” Re-considering the relationship of the COSP outcome scales to
these dimensions may lead to some modification to the composite well-being measure. Second,
Masse et al. (1998) suggest subjective well-being and psychological distress are not merely the
opposite poles of the same axis of mental health, but that these are separate constructs that
must be measured independently. This means that an increase in measures constituting the
positive aspect of well-being, such as hope or self-esteem, will not necessarily coincide with a
decrease in measures constituting psychological distress, such as recovery assessment, and that
two separate composite measures might be more appropriate. Third, the construct driving the
improvement of well-being observed in the COSP MRI is hope. In addition, the psychometric
97
analyses of the well-being measure indicated that hope was one of the most important
components and exhibited a strong trend over time. The Herth Hope Index and its subscales
might be examined as outcomes on their own.
Analyses that were implemented as part of the effort to develop the well-being measure
suggested that different outcome scales were affected by involvement in different types of
consumer-operated programs. More broadly, the psychometric characteristics of the current
synthetic measure leave open the possibility that it is not coherent enough to avoid the problem
of muddling some effects that might be apparent in some of its more sharply delineated, more
theoretically supported constructs. Thus, re-defining the well-being measure would be useful to
develop measures best-suited to each program type (drop-in, education/advocacy, or peersupport), while using theory and literature as a framework.
Further Engagement Analyses
Rather than constructing a single engagement variable in a manner similar to what one
would do in an intent-to-treat model, “treatment” can be conceptualized as a time-dependent
covariate. In this model, individuals may move in to and out of treatment throughout the
course of the study. Such an analysis would have to address the difficulty that the CP interview
was completed on one schedule (every 4 months based on each individual’s entry into the study)
and the visits data were submitted every 3 months based on the calendar years of the study
period. However, this difficulty can be at least partially addressed by comparing the date of
each CP interview with the dates of the calendar quarters.
Since random regression modeling allows for the inclusion of time-dependent treatment
covariates, such a “dose” variable could easily replace the engagement group covariate in the
models described earlier. This covariate could take the form of the engaged categorization or a
continuous measure of program involvement and would assess a participant’s outcomes in
98
relation to his or her level of engagement at the time of, or prior, to the outcome assessment.
Further Cost Analyses
Cost-effectiveness and cost-benefit analyses would be the continuation of the cost
analyses undertaken for this report. Costs of different types of COSP are related to nonmonetary
outcomes in cost-effectiveness analysis, and to monetary outcomes in cost-benefit analysis. Such
analyses would begin by quantifying the monetary outcomes (i.e., the benefits) of COSP,
including both possible savings in health, mental health, criminal justice, housing, income
support, and other human services, as well as, increments in consumers’ income from part- and
full-time employment. The analyses would proceed to examine how these and other possible
monetary outcomes are related to changes in the biopsychosocial processes, such as
psychological well-being, hypothesized to be improved by consumer-operated services. The
degree to which these processes are improved by consumer-operated services would be the
focus of additional analyses.
Relationships between COSP costs and changes in biopsychosocial processes would be
examined in cost-effectiveness analyses. These analyses would examine how relationships between
costs and effectiveness were affected by the type of, and site for, COSP program
implementation, as well as, by the degree of engagement of the consumer in the specific COSP
(i.e., number of visits). The impact of participation in different specific program activities, as
reported by the consumer, also would be examined. Cost-benefit analyses would duplicate these
cost-effectiveness analyses, but would use monetary outcomes in lieu of nonmonetary measures
of effectiveness. If sufficient data were available, a final set of path analyses might examine the
significance, direction, and magnitude of relationships between the costs, procedures, processes,
and outcomes described above, integrating the cost-effectiveness and cost-benefit analyses.
99
References
Besteman, K. J., Greenfield, L., De Smet, A., Yates, B. T., & Filipczak, J. (1995). Retention in
methadone maintenance by duration in treatment and reason for discharge [abstract].
In L. S. Harris (Ed.), Problems of drug dependence, 1994: Proceedings of the 56th Annual
Scientific Meeting, The College on Problems of Drug Dependence, Inc. (NIDA Research
Monograph 153, p. 469). Rockville, MD: National Institutes of Health.
Cohen, J., & Cohen, P. (1993). Applied multiple regression/correlation analysis for the behavioral
sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Freeman, B. D., Gerstenberger, E. P., Banks, S., Natanson, C. (2002). Using secondary data in
statistical analysis. In John I. Gallin (Ed.), Principles and Practice of Clinical Research (pp.
251-267). San Diego: Academic Press.
Gibbons, R. D., Hedecker, D., Elkin, I., Waternaux, C., Kraemer, H. C., Greenhouse, J. B., Shea,
M. T., Imber, S. D., Sotsky, S. M., Watkins, J. T. (1993). Some conceptual and statistical
issues in analysis of longitudinal psychiatric data. Archives of General Psychiatry, 50, 739750.
Hand, D. J. (1979). Psychiatric examples of Simpson's Paradox. British Journal of Psychiatry, 135,
90-96.
Hedeker, D., Gibbons, R. D. & Davis J. M. (1991). Random regression models for multicenter
clinical trials data. Psychopharmacological Bulletin, 27, 73–77.
Jöreskog, K. G., & Sörbom, D. (1989). LISREL 7: A guide to the program and applications (2nd ed.).
Chicago: SPSS.
Lechner, M. (2002). Program heterogeneity and propensity score matching: An application to
the evaluation of active labor market policies. The Review of Economics and Statistics,
84(2), 205–220.
100
Lehman, A. F. (1988). A quality of life interview for the chronically mentally ill. Evaluation and
Program Planning, 11, 51-62.
Little, R. J., & Rubin, D. B. (2000). Causal effects in clinical and epidemiological studies via
potential outcomes: concepts and analytical approaches. Annual Review of Public Health,
21, 121-145.
Masse, R., Poulin, C., Dassa, C., Lambert, J., Belair, S., Battaglini, A. (1998). The structure of
mental health: Higher order confirmatory factor analyses of psychological distress and
well-being measures. Social Indicators Research, 45, 475-505.
Mental Health Statistics Improvement Program (MHSIP) Task Force on a Consumer-Oriented
Mental Health Report Card (1996). The MHSIP Consumer-Oriented Mental Health Report
Card, Center for Mental Health Services. Rockville, MD.
Natanson, C., Esposito C. J., & Banks S. M. (1998). The sirens' songs of confirmatory sepsis trials:
selection bias and sampling error. Critical Care Medicine, 26 (12), 1927-1931.
Quezado, Z. M., Natanson, C., Karzai, W., Danner, R. L., Koev, C. A., Fitz, Y., Dolan, D. P.,
Richmond, S., Banks, S. M., Wilson, L., & Eichacker, P. Q. (1998). Cardiopulmonary
effects of inhaled nitric oxide in normal dogs and during E. coli pneumonia and sepsis.
Journal of Applied Physiology, 84, 107-115.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis
methods (2nd Ed.). Newbury Park, CA: Sage.
Rogers, E. S., Chamberlin, J., Ellison, M. L., & Crean, T. (1997). Consumer-constructed scale
to measure empowerment among users of mental health services. Psychiatric Services,
48(8), 1042-1047.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in
observational studies for causal effects. Biometrika ,70, 41-55.
101
Rubin, D. B. (1997). Estimating causal effects from large data sets using propensity scores.
Annals of Internal Medicine, 127, 757-763.
Ryff, C. & Singer, B. (1996). Psychological well-being: Meaning, measurement, and implications
for psychotherapy research. Psychotherapy and Psychosomatics, 65, 14-23.
Segal, S. P., Silverman, C. S., & Temkin, T. (1995). Measuring empowerment in client-run selfhelp agencies. Community Mental Health Journal, 31(3), 215-227.
Shapiro, S. S. & Wilk, M. B. (1965). An analysis of variance test for normality (complete
samples). Biometrika, 52, 591-611.
Substance Abuse and Mental Health Services Administration, Center for Mental Health
Services (1998). Cooperative agreements to evaluate consumer-operated human service
programs for persons with serious mental illness. Guidance for Applicants (GFA) No. SM
98-004.
U.S. Food and Drug Administration. (September 16, 1998). Guidance on statistical principles
for clinical trials - International Conference on Harmonisation. Federal Register, 63
(179), 49583-49598.
Yates, B. T. (1977a). Cost-effectiveness analysis: Using it for our own good. State Psycho-logical
Association Newsletter, 8, 6.
Yates, B. T. (1977b). A cost-effectiveness analysis of a residential treatment program for
behaviorally disturbed children. In P. Mittler (Ed.), Research to practice in mental
retardation: Vol. 1. (pp. 435-445). Baltimore: University Park.
Yates, B. T. (1980). Improving effectiveness and reducing costs in mental health. Springfield, IL:
Thomas.
Yates, B. T. (1996). Analyzing costs, procedures, processes, and outcomes in human services: An
introduction. Thousand Oaks, CA: Sage Publications.
102
Yates, B. T. (1999). Measuring and improving cost, cost-effectiveness, and cost-benefit for substance
abuse treatment programs. Rockville, MD: National Institute on Drug Abuse, NIH
Publication No. 99-4518.
Yates, B. T. (2000). Cost-benefit analysis and cost-effectiveness analysis. In A. Kazdin (Ed.),
Encyclopedia of Psychology. Washington, DC: American Psychological Association.
Yates, B. T., Besteman, K. J., Filipczak, J., Greenfield, L., & De Smet, A. (1995, abstract).
Resource → procedure → process → outcome analysis (RPPOA): Preliminary findings
of cost-effectiveness analysis of a methadone maintenance program. In L. S. Harris
(Ed.), Problems of drug dependence, 1994: Proceedings of the 56th Annual Scientific Meeting,
The College on Problems of Drug Dependence, Inc. NIDA Research Monograph 153, p. 156.
Rockville, MD: National Institutes of Health.
Yates, B. T., Delany, P. J., & Lockwood Dillard, D. (2001). Using cost → procedure → process
→ outcome analysis to improve social work practice. In B. A. Thyer (Ed.), Handbook of
social work research (pp. 207-238). Thousand Oaks, CA: Sage.
Yates, B. T., Haven, W. G., & Thoresen, C. E. (1979). Cost-effectiveness analysis at Learning
House: How much change for how much money? In J. S. Stumphauzer (Ed.), Progress in
behavior therapy with delinquents (pp. 186-222). Springfield, IL: Charles C. Thomas.
Yates, B. T., & Newman, F. L. (1980a). Findings of cost-effectiveness and cost-benefit analyses
of psychotherapy. In G. VandenBos (Ed.), Psychotherapy: Practice, research, policy. Beverly
Hills, CA: Sage.
Yates, B. T., & Newman, F. L. (1980b). Approaches to cost-effectiveness analysis and costbenefit analysis of psychotherapy. In G. VandenBos (Ed.), Psychotherapy: Practice, research, policy (pp. 103-162). Beverly Hills, CA: Sage.
103