1
Do all schools allocate bonuses equally?: A latent profile analysis of schools receiving
cooperative performance incentives
Meghan V. Hauptli, Brooke Soden-Hensler, and Laura B. Lang
Florida State University
--- DRAFT: WORKING PAPER ----- DO NOT DISTRIBUTE OR CITE WITHOUT PERMISSION ---
Contact:
Meghan V. Hauptli
2010 Levy Ave., Suite 100
Tallahassee, FL 32310
mvh07@fsu.edu
_____________________________
This research was conducted while the first and second authors were supported by an FCRR Predoctoral
Interdisciplinary Research Fellowship, funded by the Institute of Education Sciences, US Department of
Education (R305B04074). Views expressed herein are those of the authors and have neither been
reviewed nor approved by the granting agency.
2
Do all schools allocate bonuses equally?: A latent profile analysis of schools receiving
cooperative performance incentives
Introduction
Federal policy-makers have encouraged the use of performance pay by requiring states to
apply student achievement data to teacher compensation decisions through the federal Race to
the Top (2009) grant competition. Similarly, federal commitment to performance pay can be
seen in U.S. DOE’s more than four-fold funding increase to the Teacher Incentive Fund (TIF)
last year (Education Commission of the States, 2010). Florida has experimented with
performance pay initiatives over the past three decades and has been viewed on the national
scene as a performance pay pioneer.
Descriptions in the research literature of performance pay in Florida have focused on
Florida’s Merit Award Program (MAP; e.g., Podgursky & Springer, 2007b) and have largely
ignored the substantially larger program analyzed in this study, the Florida School Recognition
Program (FSRP). For example, in 2005-2006, only $22,657,830 in MAP awards were distributed
state-wide compared to $157,587,811 in state-wide FSRP awards. A similar pattern was seen in
the state’s largest school district, Miami-Dade County; in the same 2005-2006 school year,
$4,520,496 in MAP awards were distributed in Miami-Dade compared to $21,476,233 in FSRP
awards (FL DOE, 2007; Wellman, 2007). The MAP awards salary supplements of at least five
percent of an individual’s base salary to teachers and administrators identified as improving
student achievement, whereas the Florida School Recognition Program’s cooperative
performance incentives (CPI) are awarded to schools that have significantly raised or sustained
improved student performance as measured by school grades. Additionally, distribution of MAP
funds to individuals is determined at the state-level whereas with the FSRP, funds are awarded to
3
schools and school-level employees decide how to allocate the cooperative performance
incentives.
This introduction begins with a brief discussion of performance pay, followed by an
overview of the body of evidence related to cooperative performance incentives (CPI)
specifically, and concludes with a description of one such CPI program, the Florida School
Recognition Program, and the aims of the current study.
Performance Pay
Performance pay, or merit pay, is a financial incentive strategy used to incent teachers to
improve instructional practice and student achievement outcomes. The rationale for performance
pay is often presented in contrast to the traditional single salary schedule; a key difference
between the single salary schedule and performance pay lies in the traditional model’s reliance
on “inputs” (e.g., years of experience and advanced degrees), whereas performance pay is more
closely tied to teacher quality “outputs” such as student achievement and teacher evaluations
(Hanushek, 2003). Policy experimentation with performance pay gained traction in the mid1980s after the release of the A Nation at Risk (1986) report. More recently, with an era of
accountability created by No Child Left Behind (NCLB), performance pay policies endeavor to
reward teachers whose students perform well, through traditional performance models, or
demonstrate growth, through value-added models, on state assessments. Although related
literature suggests that performance pay plans should be structured to include a spectrum of
performance standards (Koppich, 2005; National Institute for Excellence in Teaching, 2007),
plans that rely primarily on student achievement data from state accountability tests are the
norm.
4
Many policy-makers view performance pay as a means to bridge the disconnect between
monetary inputs and the often murky and difficult to quantify teacher quality outputs.
Performance pay advocates argue for incentivizing teaching because they claim it reduces the
free-rider problem of unmotivated teachers coasting through undistinguished careers, and will
enable districts to recruit, retain, and reward high quality teachers. Longstanding concerns that
“the nature of teachers’ work” may be incompatible with performance pay schemes (Murnane &
Cohen, 1986), concerns that public sector conditions make performance pay difficult to award
(Ballou, 2001), and recent findings from the National Center on Performance Incentive’s POINT
experiment which indicated that bonuses alone do not improve student achievement outcomes
(Springer et al., 2010) have done little to slow a growing national interest in reforming teacher
compensation by moving to value-added measures.
Empirical questions regarding the impact of performance pay on a variety of outcomes,
the optimal size of bonuses, and whether or not performance pay is more efficiently awarded at
the individual, teacher team, or whole school level remain largely unanswered. Podgursky and
Springer (2007a) noted that while the research literature is thin, it is promising, and they argued
for further policy experimentation.
Cooperative Performance Incentives
Cooperative performance incentives (CPIs), a form of performance pay awarded at the
school level, are intended to improve teaching and learning in subsequent years. A review of the
literature, using the broadest of search parameters, yielded a very small number of articles related
solely to the Florida School Recognition Program (FSRP). Gayles (2007) used linear regression
to examine the relation between free and reduced-price lunch and minority status on FSRP
awards, and found that poverty and race negatively influence receipt of FSRP awards. Thus, he
5
argued that the relationship between these demographic variables and receipt of FSRP
cooperative performance incentives reproduces existing social stratification. In a similarly
designed but unpublished study of the FSRP conducted by the non-profit Okaloosa Citizens
Alliance, Morris (2008) examined the relation between receipt of FSRP awards and free and
reduced-price lunch status. Morris found a strong, negative relationship between the two
variables (r = -0.677). He used this correlation to argue that socio-economic status, not teacher
quality, is predictive of receipt of FSRP awards and thus the FSRP “is in need of fundamental
revision” (p.4). Results of both studies are threatened by a methodological unit of analysis
problem as the researchers analyzed the data at the district-level rather than the school-level, the
level at which the awards are distributed. The approach is problematic because the FSRP is a
policy that bypasses school district offices and awards cooperative performance incentives
directly to individual school sites. Furthermore, neither study examined how resources are
allocated at the school-level.
While Gayles (2007) and Morris (2008) both pointed to limitations of the FSRP,
Peterson, Hanushek, and West (2009) recommended that the program be retained. In an external
evaluation of Florida’s K-12 education system prepared for the State Board of Education titled
Sustaining Progress in Times of Fiscal Crisis, the report’s authors stated:
Even if the fiscal situation deteriorates further, Florida should protect the main features of
its existing accountability system. The costs of testing students and calculating results are
very small relative to the benefits of accountability. (p. 16)
They also went on to specify that the FSRP should be preserved even in the context of the statewide budget crisis. The report did not include empirical analyses to substantiate the
recommendation.
6
Cooperative performance incentive plans are not unique to Florida. Indeed, experimental
and quasi-experimental studies have been conducted overseas in countries such as India and
Israel (see Taylor & Springer, 2010, for a comprehensive review which includes these
international studies). The extant review of the literature, however, is largely limited to the North
American context, where much of the early research was theoretical or descriptive. Raham
(2000) conducted a research synthesis of CPI programs functioning in the 1990s in Kentucky,
North Carolina, Texas, Britain, and Alberta. Based on the author’s review of these programs, she
outlined several conditions for policy-makers to consider for viable CPI implementation: “open
and public access to performance information, site-based decision making, a support system to
help schools disaggregate and interpret data, persistence over years, and a significant
commitment of resources” (p. 154). In the same issue of the Peabody Journal of Education,
Kelley, Conley, and Kimball (2000) examined the rewards and sanctions programs in Maryland
and Kentucky, and the effects of each state’s group-based performance award program. Only
Kentucky’s group-based performance award program allowed for salary bonuses to employees.
Notably, Kentucky’s program was very similar to the FSRP in that it left the decision-making to
school-site employees, “certified staff at the school decided collectively how to use the money.
In practice, most of the awards were used for salary bonuses for teachers and other staff” (p.
163). The authors combined surveys of school principals with interviews from a variety of
stakeholders to draw comparisons between the two states’ programs. The analysis was targeted at
understanding the accountability context rather than evaluating CPIs.
A critical challenge faced by performance pay policies relates to garnering buy-in from
the very stakeholders who stand to benefit— the teachers. Kelley, Heneman, Milanowski (2002),
using a mixed methods approach, analyzed interviews and survey data to examine the
7
motivational effects of school-based performance award programs on teachers’ and principals’
attitudes. According to the authors:
A striking finding in both the qualitative and quantitative data from both program sites
was the low perceived probability that the bonus would actually be paid when school
goals were met. Moreover, teacher belief that earned bonuses would be paid increased
only slightly for those who had earned the bonuses in the past award cycle. These data
suggest teacher skepticism regarding the bonus portion of the program. (p. 386)
Gratz (2005) called into question the idea that attempts to impact teacher motivation would lead
to improved student achievement: “In order to believe that pay for performance will produce
greater results through motivation alone, one must believe that a substantial number of teachers
simply aren’t trying hard enough.” This is an important reminder of an assumption which
underlies the rationale for any type of performance pay scheme.
Figlio and Kenny (2007) created a teacher incentive survey to complement data collected
by the National Education Longitudinal Survey (NELS) and found:
evidence that the use of teacher salary incentives is associated with higher levels of
student performance, all else equal. Regardless of the measure of teacher financial
incentives (i.e., whether the school offers relatively high levels or relatively low levels of
incentives, as well as the ways in which the incentives are cumulated) the incentive
coefficients are positive and at least marginally significant. (p. 910)
Furthermore, the authors found “that while selectively-administered merit pay programs are
associated with increased student test scores, those that award bonuses to very large fractions of
teachers are apparently not associated with student outcomes” (p. 910). This finding that student
test scores were higher in schools that offered individual performance pay rather than
8
performance pay to the whole school raised questions about the rationale for cooperative
performance incentive programs.
In an evaluation of the Governor’s Educator Excellence Grants program (GEEG), a CPI
pilot program in Texas, Taylor and Springer (2010) examined CPI plans from the employer’s
perspective and from the perspective of the teachers who designed the plans. By examining the
dispersion of awards within schools, the researchers found that teachers design relatively weak,
group-oriented incentive pay plans. Moreover, the egalitarian approach favored by teachers was
not in keeping with the Texas Education Agency’s intention of bonuses ranging from $3,000 to
$10,000 per teacher. The study also included analyses investigating the impact of schools’ CPI
plans on both teacher turnover and student achievement outcomes. While Taylor and Springer
found evidence to suggest that teachers who received the CPIs were less likely to turn over, they
did not find evidence of GEEG program impacts on student performance. Unlike the GEEG
program under which only teachers were eligible for bonuses, all school employees are eligible
for FSRP bonuses.
Goodman & Turner (2010) article here-- followed by a summary statement underscoring
the need for this study.
The Florida School Recognition Program
Often referred to as “A+ funds” by school-level actors, the FSRP cooperative
performance incentives are a component of Florida’s A++ Accountability program (§1003.428)
instituted by former Governor Jeb Bush. The cooperative performance incentives are awarded to
schools that have significantly raised or sustained improved student performance as measured by
school grades. Schools that improve by at least one letter grade (e.g., from an F to a D, or from a
D to a B, or maintain an A grade) earn the CPI. School grades were based exclusively on student
9
performance on the high-stakes state accountability assessment, the Florida Comprehensive
Assessment Test (FCAT). As specified in Florida Statute §1008.36, schools must use their FSRP
awards for one or any combination of the following: 1.) nonrecurring faculty and staff bonuses;
2.) nonrecurring expenditures for educational equipment and materials; or 3.) temporary
personnel to assist in maintaining or improving student performance. Beyond this guidance,
faculty and the school advisory committee, who are assigned the task of collectively deciding
how to allocate the CPI at each school site, have considerable discretion with regard to how they
distribute the award.
The FSRP awards in 2005-2006 and 2006-2007 were based on a formula that provided
$100 for each full-time equivalent (FTE) student. The per-pupil funding for this program
decreased to $85 FTE in 2007-2008 and subsequently to $75 FTE in 2008-2009 and remains at
that level currently as a result of the economic down-turn and a state-wide budget crisis. School
grades are generally released at the end of the school year. Decision-making regarding the
distribution of the CPI at each school-site occurs during the first quarter of the subsequent school
year with a deadline for allocation plans due to the district office by November 1. In the event
that stakeholders cannot agree on a distribution plan, Florida Statute indicates that the
cooperative performance incentive would be equally distributed to all classroom teachers
teaching in the school.
Our prior work on this project indicated that nearly all of the FSRP CPIs were allocated
to employee bonuses (approximately 89%) and, thus, these bonuses are the focus of the current
project. Moreover, we found that teachers of FCAT tested grades and subjects were awarded
larger mean bonuses than teachers of non-FCAT grades and subject (e.g., grades K-2 and
subjects such as social studies, physical education, art, and music), and that there were
10
differences in mean employee bonus award amounts across job classifications (i.e., teachers,
administrators, non-instructional certified staff, paraprofessionals, support staff, and clerical) and
school levels (i.e., elementary, K-8, middle, and high) (Hauptli, Soden-Hensler, & Lang, 2010a,
2010b, 2010c).
Given that the decision-making process for the allocation of the FSRP funds at the
school-level is not specified and the legislation grants the decision-making authority to the
school staff and school advisory committee, there is considerable opportunity for variability in
distribution patterns across schools. The FSRP presented a unique opportunity to examine how
school site employees, as opposed to state or district-level policy-makers, allocated cooperative
performance incentives.
In the current study, we examined personnel and salary records to identify latent profiles
(groups) of schools based on their school-level allocation profiles using latent profile analysis
(LPA). LPA is a quantitative technique used to identify latent groups (in this study, each school’s
FSRP allocation model determined its school “type,” or group) based on the observed values of
continuous variables (here, school-level mean dollar amounts of FSRP funds allocated to
employees). Specifically we asked,
1.) Based on school-level allocation models, what latent groups of schools emerged?
2.) How much of the variance in continued exceptional performance (i.e., continued
receipt of FSRP awards by maintaining “A” grades or continuing to improve) is
explained by group membership?
We hypothesized that latent groups of schools would emerge based on factors including: (1) how
bonuses were allocated across instructional and non-instructional employee job classifications,
and (2) how bonuses were allocated to teachers of FCAT tested grades and subjects versus non-
11
FCAT teachers. We further hypothesized that if a latent profile that rewarded teachers
substantially higher than other employee classifications were to emerge, then this profile might
predict future receipt of the CPI more strongly than egalitarian profiles.
Method
Sample
Personnel and salary records for all employees (N=27,385, N=17,251, and N=30,030) in
non-charter schools (N=203, N=133, and N=210) that received FSRP cooperative performance
incentives in the Miami-Dade County Public School district for the 2005-2006, 2006-2007, and
2007-2008 school years, respectively, were utilized. The 203 schools receiving FSRP awards in
2005-2006 represent 66.34% of all non-charter schools in the district. The sample decreased in
2006-2007 to 133 schools which represent 43.32% of all non-charter schools in the district. This
dip in school grade performance was observed state-wide in 2006-2007 and reflects a shift in
Florida Department of Education’s accountability standards. The 210 schools receiving FSRP
awards in 2007-2008 represent 69.08% of all non-charter schools in the district. More than half
(54%) of schools from the 2005-2006 sample (n = 110) are represented in the 2006-2007 sample.
An even higher percentage (80%) of schools from the 2006-2007 sample (n = 107) are
represented in the 2007-2008 sample. Schools that did not have any have employees in one of
the job classifications were dropped from the analysis. As a result, our analytic sample was
slightly smaller for each of the three years due to missing information for one or more job
classification (N=199, N=130, N=205).
Across schools, thirty-eight district-provided general employee job descriptions were
collapsed into 6 employee job classifications: teachers, administrators, non-instructional certified
staff (NICs), paraprofessionals, clerical, and support staff. Within the teacher job classification,
12
137 teacher job descriptions were coded dichotomously to distinguish between teachers of FCAT
tested grades and subjects and teachers of non-FCAT tested grades and subjects. Science
teachers were not coded as teachers of FCAT tested grades and subjects in the first two years of
the data, but were counted as teachers of FCAT tested grades and subjects in the 2007-2008 year
once FCAT science was included in the calculation of school grades.
Latent Profile Analysis
Latent profile analysis (LPA) is a type of latent class analysis (LCA). Whereas traditional
LCA uses categorical observed variables as indicator variables, latent profile analysis (LPA) uses
continuous observed variables. These techniques are used to classify cases into groups based
upon scores on a number of variables. Both are similar to cluster analysis conceptually, though
LCA and LPA are multivariate approaches that assume an underlying latent variable that
determines an individual’s class membership rather than relying on researcher imposed cut-off
points to determine groupings. Parallels can also be drawn between latent profile analysis and the
widely used factor analysis technique. Factor analysis groups similar items and variables into
continuous latent factors, whereas LPA analyses provide a means of grouping individuals into
latent categories on the basis of shared characteristics that distinguish members of one group
from members of another group. If these shared characteristics represent a range of well-defined
characteristics (such as bonus allocation patterns), the resulting groups could characterize cases
(schools) by the nature of their bonus distributions to employees, the indicators of interest in the
current study. Indeed, one goal in the current study was to determine if profiles (groups) of
schools emerged based upon the manner in which they allocated bonus awards to individual
employees based upon their job classification.
13
All analyses were conducted using MPlus 4.1 (Muthén & Muthén, 1998). In LPA, there
are multiple statistical indictors of model fit and one uses a combination of statistical
considerations to decide on the best fitting model. When there is substantive theory to guide
selection of groups (profiles), this should also be considered (e.g., in medical scenarios where
diagnostic criteria exist). However, given the uniqueness of school-determined bonus allocation
there was no substantive theory on which to rely and, thus, only statistical indicators were used.
In our analyses, we considered two widely used information criteria, or “predictive” fit
indices: Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Both
indices take into account model fit and complexity and lower values indicating more
parsimonious model fit. Conceptually, they provide an indication of how likely the model would
fit the data well with a replication sample (Kline, 2005). The Bayesian information Criterion
(BIC) was given more weight in our considerations because recent simulation studies suggest
that the BIC provides the most reliable indicators of true model fit (Nylund, Asparouhov, &
Muthén, 2007). BIC values can further aid in identifying appropriate models when the values of
various models are plotted. Plotted BIC values can be interpreted as a scree plot such that when
the slope of the plotted curve begins to level, very little information is being gained relative to
the number of degrees of freedom used in the model to identify additional profiles (groups).
Additionally, we included entropy, a measure of the “degree of fuzziness” in profile
(group) membership which assesses the extent to which the groups are differentiated from one
another (Ramaswamy, DeSarbo, Reibstein, & Robinson, 1993). Entropy values closer to or
exactly 1 indicate better classification and values greater than .80 demonstrate good separation of
identified groups (profiles; Ramaswamy et al., 1993). Finally, we compared two tests reported in
the MPlus program (Muthén & Muthén, 1998), the Lo-Mendell-Rubin Likelihood Ratio Test
14
(TECH11; Lo, Mendell, & Rubin, 2001), and a parametric bootstrapped likelihood ratio test
(TECH14) which has shown to be a good indicator of model fit (Nylund, Asparouhov, &
Muthén, 2007). Both TECH11 and TECH14 assess the fit between the model being fit and a
model with one fewer groups (profiles) and provide a p-value that indicates which model fits
best. As an example, for a four-group model a nonsignificant p-value (p > .05) indicates that the
three-group model fits better than does the four-group model.
A series of models was fitted to determine the number of profiles that provide the best fit
to the schools’ employee bonus allocation patterns. Each model was evaluated using these
statistics, and once the best fit was determined, each school's predicted profile membership was
extracted and was used as a predictor in models of future receipt of FSRP cooperative
performance incentives.
Results
Latent Profile Analysis
The indicators of school cooperative performance incentive (CPI) distribution patterns
(mean bonus amounts awarded to employees in each of the seven job classifications within each
school) were used as manifest variables in an LPA of 534 schools, all of the schools that received
FSRP awards for each of the three academic years investigated (2005-2006, 2006-2007, and
2007-2008).
A total of six LPA models were tested and evaluated using the criteria described above to
determine the number of profiles (groups) that exist in the data. The results of the indices used to
evaluate model fit are presented in Table 2. As the number of profiles (groups) increased, the
AIC and BIC decreased. As described above, the BIC values are more stable (Nylund,
Asparouhov, & Muthén, 2007) and were plotted (see Figure 1) to determine the point at which
15
the addition of profiles was no longer adding substantially to the model in order to select a
parsimonious solution. However, the plotted BICs resulted in a rather smooth curve and there
was no obvious “elbow” at which to make a selection, thus, this fit indicator was not informative
in the current study. All entropy values were greater than .80 and therefore considered at an
acceptable level. The TECH11 results were significant for both the two- and three-profile
models. Taken together, these show that the three-profile model provides significantly better fit
to the data than fewer profiles. A four-profile model resulted in a nonsignificant value indicating
that the four-profile model was not providing a significantly better fit to the data. Despite the fact
that the six-profile model also provide a statistically significant model fit, the three-profile model
was selected since other studies have found that the first instance where the p-value is
nonsignificant is a good indication to stop increasing the number of profiles (Nylund,
Asparouhov, & Muthén, 2007). The TECH14 indices remained significant for all models and
therefore were not an informative index for these models.
In addition to determining that the fit indices were indicating selection of a three-profile
model, inspection of the probabilities of a school falling into a particular profile (group) should
be acceptably high and the three profiles should be sensible. For the three-profile solution, a
school has a 33% chance of membership in any of the three classes and although higher
probabilities of profile membership may be statistically significant, they might not be
substantially high to the level of being particularly meaningful from a research standpoint.
Posterior probabilities for each of the three profiles show that the mean probability that the
school was placed in the group for which membership was most likely ranged from 91% to 94%.
Thus, on average schools were assigned to profiles that they were very highly likely to be
members.
16
In order to determine that the three profiles produced by the model were sensible and
sufficiently distinct, mean bonus amounts for each of the seven employee job classifications (i.e,.
FCAT teachers, non-FCAT teachers, administrators, non-instructional certified staff,
paraprofessionals, support staff, and clerical) for schools in each of the profiles were plotted for
each academic year (see Figures 2-4 for 2005-2006, 2006-2007, and 2007-2008, respectively).
Additionally, numbers of schools falling into each profile, means, and standard deviations are
presented in Tables 3-5 (for 2005-2006, 2006-2007, and 2007-2008, respectively). In all profiles,
FCAT teachers, non-FCAT teachers, and administrators received the highest bonuses and NICs
tended to receive just less than the two teacher groups whereas the remaining job classifications’
bonuses were substantially lower, a finding that is consistent with previous research using this
data (Hauptli, Soden-Hensler, & Lang, 2010a, Hauptli, Soden-Hensler, & Lang, 2010c).
The first profile that was extracted had relatively low bonuses across each of the seven
job classifications compared to the entire sample and will hereafter be referred to as the “Low
Bonus” profile. Total FSRP CPIs to schools in this Low Bonus Profile ranged from $11,001 to
$196,351 across the three years. The second extracted profile gave higher bonuses
(approximately $250 more) to teachers, administrators, and NICs, but awarded lower bonus
amounts to paraprofessionals, support staff, and clerical employees comparable to those in the
“Low Bonus” profile. Whereas schools in this profile were similar to the “Low bonus” schools
for non-professional employees, it more closely resembled the “High Bonus” (though not quite
as high) for the teachers, administrators, and NICs. This profile is referred to as the “Medium
Bonus” profile since it falls between the “Low Bonus” and “High Bonus” profiles with regard to
allocation of bonus amounts to teachers, administrators, and NICs. Total FSRP CPIs to schools
in this Medium Bonus Profile ranged from $31,301 to $318,465 across the three years. The third
17
and final profile, the “High Bonus” profile, awarded the highest bonuses in each of the seven job
classifications. The pattern of relative bonus distribution to employees was similar to that of the
“Low Bonus” profile except that the bonuses were higher overall. More specifically, mean dollar
amount increases to professionals (teachers, administrators, and NICs) were higher than for other
employees, particularly in the first two years of our sample when CPIs were at the higher rate per
FTE. Total FSRP CPIs to schools in this High Bonus Profile ranged from $84,457 to $352,928
across the three years.
Given how school FSRP awards are determined (i.e., based on student enrollment) and
the simplified naming of the latent profiles, it is tempting to simply attribute bonus allocations to
school size, but inspection of the distributions of bonuses awarded by employee job
classifications (most easily seen in Figures 2-4) illustrates that schools chose to distribute funds
differently both as a function of the FSRP CPI amount and employee job classification. Patterns
were not the same across the various profiles identified. Schools chose to allocate bonuses
differently based on employee job classifications, and, it seems, greater bonuses were awarded to
those who had the most direct impacts on student achievement and accordingly the greatest
potential to affect FSRP receipt. Furthermore, though it was typically the case that employees in
all job classifications were awarded bonuses, the dollar amounts awarded differed as a function
of the size of the CPI schools received; when there was more CPI money to distribute,
professionals tended to get greater increases than did other employees. See Table 6 to compare
the frequency of schools from each school level across the three profiles.
Logistic regression
Logistic regression was performed to assess the impact of schools’ profiles of bonus
distributions to employees on the likelihood that they would receive a FSRP cooperative
18
performance incentive (CPI) the following academic year. The full model (including profile of
bonus distribution and the constant) was statistically significant for each of the three academic
years investigated (see Table 7), indicating that the models distinguished between schools that
received and did not receive a CPI the following year. Each of the models explained
approximately 5.8%, 13.1%, and 6.7% (Nagelkerke R-squared; 2005-2006, 2006-2007, and
2007-2008, respectively) of the variance in subsequent year bonus status. Since the profile of
school patterns of bonus distribution used to predict future FSRP cooperative performance
incentive receipt is categorical, all impacts are characterized in reference to a comparison profile,
here, the comparison group is the “Low Bonus” profile.
In 2005-2006, for schools in the "Medium Bonus" profile, the odds of receiving the
cooperative performance incentive the next academic year were 2.41 times as great or 141%
greater than for those in the "Low Bonus" profile. Put another way, schools in the "Medium
Bonus" profile had a probability of receiving a CPI the next academic year that was 0.21 higher
than those in the "Low Bonus" profile, a significant difference (β = .88, p = .016; see Table 8).
In contrast, for schools in the "High Bonus" profile, the odds of receiving a CPI the next
academic year were nearly identical (1.02 times as high or .02% higher than for those in the
"Low Bonus" profile). Accordingly, schools in the "High Bonus" profile had a nearly identical
probability of receiving a CPI the next academic year (0.005 higher than those in the "Low
Bonus" profile; β = .02, p = .967).
The pattern of results was somewhat different for schools earning CPIs for performance
in the 2006-2007 academic year. Compared to the “Low Bonus” profile, schools in the "Medium
Bonus" profile had odds of receiving a CPI the next academic year that were 4.31 times as great
or 331% greater than for those in the "Low Bonus" profile. In terms of probabilities, the
19
"Medium Bonus" profile schools had a probability of receiving a CPI the next academic year that
was 0.26 higher than those in the "Low Bonus" profile, a significant difference (β = 1.46, p =
.004). Similarly, for schools in the "High Bonus" profile, the odds of receiving a CPI the next
academic year were 6.88 times as high or 588% higher than for those in the "Low Bonus"
profile. That translates into a probability that was 0.30 higher for the "High Bonus" profile
schools compared to the "Low Bonus" profile schools, though this difference did not reach
significance (β = 1.93, p = .078) likely due to the very small number of schools (N = 12) in the
"High Bonus" profile.
For the final year analyzed (2007-2008), though a similar pattern emerged for the
“Medium Bonus” profile, a different patterned was seen for the “High Bonus” profile. As in
2005-2006 and 2006-2007, for schools in the "Medium Bonus" profile, the odds of receiving a
CPI the next academic year were greater (1.91 times as great or 91% greater) than for those in
the "Low Bonus" profile. However, the magnitude of the increased probability of this difference
was somewhat lower (0.11 higher for “Medium Bonus”) this year and the difference did not
reach significance (β = 0.65, p = .105). In contrast to the other years investigated, for schools in
the "High Bonus" profile, the odds of receiving a CPI the next academic year were 0.33 times as
low or 67% lower than for those in the "Low Bonus" profile. That is, schools in the "High
Bonus" profile had a probability of receiving a CPI the next academic year that is 0.26 lower
than those in the "Low Bonus" profile, a significant difference (β = -1.10, p = .025).
Discussion
Cooperative performance incentives (CPIs), a form of performance pay awarded at the
school level, are intended to improve teaching and learning outcomes in subsequent years. Using
employee-level data, we examined school-level allocation patterns through Latent Profile
20
Analysis (LPA). Across the three years of this study’s sample, three profiles, or groups, of
schools emerged: Low Bonus, Medium Bonus, and High Bonus. The dollar amounts awarded to
the three latent groups differed as a function of the size of the CPI schools received and across
employee job classifications. The results seem to support our hypothesis that the manner in
which employee bonuses were allocated across instructional and non-instructional job
classifications would be an important factor in identifying the latent groups. When there were
larger pots of CPI money to distribute, instructional employees tended to receive greater
increases than non-instructional employees. Schools chose to allocate bonuses differently based
on employee job classifications, and, it seems, greater bonuses were awarded to those who had
the most direct impacts on student performance and accordingly the greatest potential to affect
FSRP receipt. Additionally, we found preliminary evidence to suggest that certain profiles of
CPI distribution to employees were predictive of FSRP CPI receipt the following school year.
However, the pattern of results across years of the sample varied both in terms of which profile
significantly predicted the probability of receiving a CPI the subsequent school year and in terms
of the magnitude of the probabilities, making it difficult to draw broader policy implications
from the results.
Because of the correlational nature of the data, it is entirely possible that explanations
other than the identified bonus profiles might drive future receipt of FSRP cooperative
performance incentives. For example, the accountability context, curricular changes, or many
other plausible explanations could potentially overshadow the influence of bonus profiles in
explaining why a school received the CPI in a subsequent year. Future analyses will control for
demographic variables and school characteristics (possibly to include SES, LEP, ESE, and
school size).
21
Given national interest in applying student achievement data to teacher compensation
decisions and recently released findings from the National Center on Performance Incentives’
POINT experiment which indicate that bonuses alone do not improve student achievement
outcomes (Springer et al., 2010), it is important to understand how established performance pay
programs such as the Florida School Recognition Program (FSRP) are functioning. This study
represents a first step in identifying allocation models of FSRP cooperative performance
incentives and examining the relation between these models and receipt of FSRP CPI the
subsequent school year.
22
References
Ballou, D. (2001). Pay for performance in public and private schools. Economics of Education
Review, 20(1), 51-61.
Education Commission of the States. (2010). Teacher merit pay: What do we know? The
Progress of Education Reform, 11(3), 1-4. Denver, CO: Education Commission of the
States.
Figlio, D. N. & Kenny, L. W. (2007). Individual teacher incentives and student
performance. Journal of Public Economics, 91, 901-914.
Florida Department of Education. (2007). Florida School Recognition Program. Tallahassee, FL:
Author. Retrieved from http://www.fldoe.org/evaluation/schrmain.asp
Florida Statute. Title XLVIII Chapter 1008.36. Florida School Recognition Program.
Retrieved from
http://www.leg.state.fl.us/statutes/index.cfm?mode=View%20Statutes&SubMenu=1&Ap
p_mode=Display_Statute&Search_String=1008.36&URL=CH1008/Sec36.HTM
Gayles, J. (2007). Race, reward, and reform: An implicative examination of the Florida
School Recognition Program. Educational Policy, 21(3), 439-456.
Goodman, S. F., & Turner, L. J. (2010). Teacher incentive pay and educational outcomes:
Evidence from the New York City Bonus Program.
Gratz, D. G. (2005). Lessons from Denver: The pay for performance pilot. Phi Delta
Kappan, 86(8).
Hanushek, E. A. (2003). The failure of input-based schooling policies. Economic Journal,
113(485), F64-F98.
Hauptli, M. V., Soden Hensler, B., & Lang, L. B. (2010a). Cooperative performance pay
incentives: Investigating allocation patterns over two years. Poster presented at the 5th
Annual Institute for Education Science Research conference, National Harbor, MD.
Hauptli, M. V., Soden Hensler, B., & Lang, L. B. (2010b). Investigating allocation patterns in
teacher bonuses: Florida’s School Recognition Program. Poster presented at the Florida
Education Research Association, Orlando, FL.
Hauptli, M. V., Soden Hensler, B., & Lang, L. B. (2010c). Investigating cooperative
performance pay incentives: Allocation patterns in Florida’s School Recognition Program.
Poster presented at the American Education Finance Association conference in Richmond,
VA.
23
Kelley, C., Conley, S., & Kimball, S. (2000). Payment for results: Effects of the Kentucky and
Maryland group-based performance award programs. Peabody Journal of Education, 75(4),
159-199.
Kelley, C., Heneman, H., & Milanowski, A. (2002). Teacher motivation and school-based
performance awards. Educational Administration Quarterly, 38(3), 372-401.
Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New
York: Guildford Press.
Koppich, J. E. (2005). All teachers are not the same: A multiple approach to teacher
compensation. Education Next, 5(1), 13-15.
Ladd, H. (1999). The Dallas school accountability and incentive program: An evaluation of its
impacts on student outcomes. Economics of Education Review, 18(1), 1.
Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal
mixture. Biometrika, 88, 767−778.
Morris, C. J. (2008, March). The Florida School Recognition Program: Is it time for a
second look? Unpublished manuscript. Retrieved from
http://www.oca1787.org/documents/flrecognition.pdf
Murnane, R. J., & Cohen, D. K. (1986). Merit pay and the evaluation problem: Why most merit
pay plans fail and a few survive. Harvard Educational Review, 56(1), 1-17.
Muthén, L. K., & Muthén, B. O. (1998–2007). MPlus user’s guide, Fourth Edition. Los Angeles,
CA: Author.
Nylund, K., Asparouhov, T. & Muthen, B. (2007). Deciding on the number of classes in latent
class analysis: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535569.
Peterson, P. E., Hanushek, E. A., & West, M. R. (2009, March 16). Sustaining progress in
times of fiscal crisis. Report prearded for the Florida State Board of Education.
Retrieved from http://www.hks.harvard.edu/pepg/PDF/FLReport09.pdf
Podgursky, M., & Springer, M. G. (2007a). Credentials versus performance: Review of the
teacher performance pay research. Peabody Journal of Education, 82(4), 551-573.
Podgursky, M. J., & Springer, M. G. (2007b). Teacher performance pay: A review. Journal of
Policy Analysis & Management, 26(4), 909-950.
Raham, H. (2000). Cooperative performance incentive plans. Peabody Journal of Education,
75(4), 142-158.
24
Ramaswamy, V., DeSarbo, W. S., Reibstein, D. J., & Robinson, W. T. (1993). An empirical
pooling approach for estimating marketing mix elasticities with PIMS data. Marketing
Science, 12(1), 103−124.
Springer, M.G., Ballou, D., Hamilton, L., Le, V., Lockwood, J.R., McCaffrey, D., Pepper, M.,
and Stecher, B. (2010). Teacher Pay for Performance: Experimental Evidence from the
Project on Incentives in Teaching. Nashville, TN: National Center on Performance
Incentives at Vanderbilt University.
Taylor, L. L., & Springer, M. G. (2010). Designing incentives for public sector workers:
Evidence from the Governor’s Educator Excellence Grant Program in Texas. Nashville,
TN: National Center on Performance Incentives at Vanderbilt University.
Wellman, M. (2007). Restrictive district requirements limited participating in performance pay
systems (OPPAGA Report No. 07-01). Tallahassee, FL: Office of Program Policy
Analysis and Governmental Accountability. Retrieved electronically from
http://www.oppaga.state.fl.us/MonitorDocs/Reports/pdf/0701rpt.pdf
25
Table 1. Review of research related to cooperative performance incentives
Author(s), pub.
Sample:
Methods
Findings*/Recommendations
year
location, years
National
Student test scores higher in schools that offer
Figlio &
Quantitative,
representative
individual performance pay than performance pay to
Kenny (2007)*
Survey data
sample, 2000
whole school.
Quantitative,
Gayles
Florida, 1998-1999
Poverty and race influence receipt of FSRP awards
Linear
(2007)*
to 2002-2003
and reproduce existing social stratification.
regression
Goodman &
Turner (2010)
Kelley,
Conley, &
Kimball
(2000)*
Maryland and
Kentucky,
late 1990s
Qualitative &
quantitative
approaches
Principals’ perceptions of monetary bonuses as
motivating for teachers is significantly higher in state
that awarded bonuses to schools (Maryland) rather
than to individual teachers (Kentucky).
Kelley,
Heneman,
Milanowski
(2002)*
Kentucky and
CharlotteQualitative &
Mecklenburg (North quantitative
Carolina), late
approaches
1990s
Survey data revealed teachers skeptical that reaching
school goals would deliver promised bonus awards.
Ladd (1999)*
Dallas Independent
School District,
1991-1992 to 19941995
Findings suggest mixed results for school-based
performance awards on student performance. Positive
and relatively large gains for Hispanic and white
seventh graders in Dallas relative to other cities, but
not for black students.
Quantitative,
Panel analysis
26
Morris (2008)
Florida, 2006-2007
Quantitative,
Linear
regression
Peterson,
Hanushek, &
West (2009)
Florida, 2009
Evaluation
Report
Recommended that FSRP be retained.
Multiple
Research
synthesis
Article reviewed the research literature and outlined
several conditions policy-makers should consider for
CPI implementation to be viable.
Raham
(2000)*
Taylor &
Springer
Texas
(2010)
*Indicates articles that appeared in peer-reviewed journals.
Socio-economic status, not teacher quality, is
predictive of FCAT performance and thus the FSRP
“is in need of fundamental revision” (p. 4).
27
Table 2. Fit Indices for LPA Model Testing
Profiles
2
3
4
5
6
Entropy
AIC
BIC
TECH11
TECH14
0.922
0.824
0.861
0.887
0.889
48976
48500
48207
47976
47765
49071
48629
48370
48173
47996
0.011
0.032
0.125
0.606
0.049
0.000
0.000
0.000
0.000
0.000
NOTE: AIC = Akaike Information Criteria, BIC = Bayesian Information
Criterion.
28
Table 3. 2005-2006 Mean Bonus Amounts Awarded to Employees by Job Classification for
Schools within Latent Bonus Profiles
N
"Low Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
43
"Medium Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
121
"High Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
35
M
SD
730.50
642.00
755.20
130.93
137.82
178.67
577.63
275.12
199.06
203.35
125.85
105.16
99.00
85.31
1007.63
934.96
1009.47
130.37
127.05
238.98
854.75
257.83
149.66
168.55
161.16
122.57
73.95
88.48
1153.44
1074.38
1182.23
195.22
198.59
258.90
1085.87
522.97
363.50
485.08
201.75
192.19
170.66
192.50
NOTE: N refers to number of schools within the latent
profile.
29
Table 4. 2006-2007 Mean Bonus Amounts Awarded to Employees by Job Classification for
Schools within Latent Bonus Profiles
N
"Low Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
39
"Medium Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
79
"High Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
12
M
SD
736.06
648.71
712.53
131.84
141.92
232.07
561.06
262.53
187.61
187.46
163.31
97.70
106.90
97.76
957.88
899.00
978.67
96.21
82.53
188.25
788.96
244.06
134.21
130.72
145.96
179.92
70.56
68.94
1150.13
1146.64
1156.54
193.21
213.15
245.04
1037.84
559.09
355.80
432.98
227.97
256.86
103.50
142.69
NOTE: N refers to number of schools within the latent
profile.
30
Table 5. 2007-2008 Mean Bonus Amounts Awarded to Employees by Job Classification for
Schools within Latent Bonus Profiles
N
"Low Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
128
"Medium Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
57
"High Bonus"
FCAT Teacher
Non-FCAT Teacher
Administrator
Non-Instruct.
Certified
Paraprofessional
Support Staff
Clerical
20
M
SD
695.58
634.13
664.93
147.26
139.08
187.35
555.51
244.10
158.43
151.87
153.04
122.85
92.13
88.86
933.02
873.27
922.20
110.96
100.03
208.58
796.78
265.38
171.65
188.79
122.66
131.63
88.70
110.84
998.20
911.33
1046.85
135.09
151.67
249.59
954.94
589.51
435.58
479.80
118.94
213.22
202.62
202.68
NOTE: N refers to number of schools within the latent
profile.
31
Table 6. Frequency of schools from each school level by Profile
2005-2006
Elem.
K-8
Middle
High
Low Profile
35
2
5
1
Medium Profile 89
15
14
3
High Profile
3
1
26
5
18
45
9
Total 127
6-12
0
0
0
0
K-12
0
0
0
0
199
2006-2007
Elem.
Low Profile
33
Medium Profile 64
High Profile
2
Total 99
K-8
2
11
2
15
Middle
1
3
8
12
High
3
1
0
4
6-12
0
0
0
0
K-12
0
0
0
0
130
K-8
10
9
0
19
Middle
15
16
8
39
High
5
8
12
25
6-12
1
0
0
1
K-12
1
0
0
1
205
2007-2008
Elem.
Low Profile
96
Medium Profile 24
High Profile
0
Total 120
32
Table 7
Logistic Regression Predicting Likelihood of FSRP Cooperative Performance Incentive Receipt the Following Academic Year
Variable
2005-2006
Bonus Profile
"Medium
Bonus"
"High Bonus"
Constant
2006-2007
Bonus Profile
"Medium
Bonus"
"High Bonus"
Constant
2007-2008
Bonus Profile
"Medium
Bonus"
"High Bonus"
Constant
β
0.88
0.02
-0.42
1.46
1.93
0.47
0.65
-1.10
0.90
S.E.
0.36
0.47
0.31
0.47
1.10
0.33
0.40
0.49
0.19
Wald
df
Sig.
8.57
2.00
0.01
5.85
0.00
1.86
1.00
1.00
1.00
0.02
0.97
0.17
10.88
2.00
0.00
9.59
3.10
2.04
1.00
1.00
1.00
0.00
0.08
0.15
9.45
2.00
0.01
2.63
5.05
21.30
1.00
1.00
1.00
0.10
0.02
0.00
Exp(β)
95% CI for Exp(β)
Lower
Upper
2.41
1.02
0.65
1.18
0.41
4.91
2.54
4.31
6.87
1.60
1.71
0.80
10.88
58.81
1.91
0.33
2.46
0.87
0.13
4.18
0.87
NOTE: 2005-2006 Model χ2 (2, N = 199) = 8.78, p < .05, R2 = .058 (Nagelkerke). 2006-2007 Model χ2
(2, N = 130) = 11.24, p < .01, R2 = .131 (Nagelkerke). 2007-2008 Model χ2 (2, N = 205) = 9.832, p <
.01, R2 = .067 (Nagelkerke). "Low Bonus" was the comparison group for all models.
33
Table 8
Probabilities of FSRP Cooperative Performance Incentive Receipt the Following Academic Year
Probability
2005-2006
"Low Bonus"
"Medium Bonus"
"High Bonus"
0.40
0.61
0.40
2006-2007
"Low Bonus"
"Medium Bonus"
"High Bonus"
0.62
0.87
0.92
2007-2008
"Low Bonus"
"Medium Bonus"
"High Bonus"
0.71
0.82
0.45
34
Figure 1.
Bayesian Information Criterion from Models with Different numbers of Latent Profiles
49200
49000
48800
BIC
48600
48400
BIC
48200
48000
47800
47600
0
2
4
6
Profiles
8
10
35
Figure 2.
Latent Profiles of Schools Based on Allocation Patterns of Bonuses to Employees in the 20052006 School Year.
Latent Profiles of Schools based on Bonus
Distribution
$1,400.00
Mean Bonus Amount
$1,200.00
$1,000.00
$800.00
$600.00
$400.00
$200.00
$0.00
FCAT
Teacher
Non-FCAT
Teacher
Admin.
NonInstruct.
Certified
Paras
Support
Staff
Clerical
"Low"
730.50
642.00
755.20
577.63
275.12
199.06
203.35
"Medium"
1007.63
934.96
1009.47
854.75
257.83
149.66
168.55
"High"
1153.44
1074.38
1182.23
1085.87
522.97
363.50
485.08
36
Figure 3.
Latent Profiles of Schools Based on Allocation Patterns of Bonuses to Employees in the 20062007 School Year.
Latent Profiles of Schools based on Bonus
Distribution
$1,400.00
Mean Bonus Amount
$1,200.00
$1,000.00
$800.00
$600.00
$400.00
$200.00
$0.00
FCAT
Teacher
Non-FCAT
Teacher
Admin.
NonInstruct.
Certified
Paras
Support
Staff
Clerical
"Low"
736.06
648.71
712.53
561.06
262.53
187.61
187.46
"Medium"
957.88
899.00
978.67
788.96
244.06
134.21
130.72
"High"
1150.13
1146.64
1156.54
1037.84
559.09
355.80
432.98
37
Figure 4.
Latent Profiles of Schools Based on Allocation Patterns of Bonuses to Employees in the 20072008 School Year.
Latent Profiles of Schools based on Bonus
Distribution
$1,400.00
Mean Bonus Amount
$1,200.00
$1,000.00
$800.00
$600.00
$400.00
$200.00
$0.00
FCAT
Teacher
Non-FCAT
Teacher
Admin.
NonInstruct.
Certified
Paras
Support
Staff
Clerical
"Low"
695.58
634.13
664.93
555.51
244.10
158.43
151.87
"Medium"
933.02
873.27
922.20
796.78
265.38
171.65
188.79
"High"
998.20
911.33
1046.85
954.94
589.51
435.58
479.80