The Performance of Albany High School on the California
Academic Performance Index, 2002-2009
By Ira Sharenow
November 1, 2011
Executive Summary...…………………………………………………………………..2
Introduction and Background Information…………………………………………...3
Performance of Albany High School on the API……………………………………...4
Statistical Analysis of the Data…………………………………………………………6
Conclusion……………………………………………………………………………….9
Summary Tables……………………………………………………………………….10
Bibliography……………………………………………………………………………11
Appendix A: List of Analyses Performed…………………………………………….12
1
Executive Summary
This report demonstrates that Albany High School (AHS) is underperforming, given the
high quality of its students. Moreover, the performance of the high school relative to the
rest of the state and relative to peer schools declined during the period of study, 20022009. Scores on the base Academic Performance Index (API) as computed by the
California Department of Education (CDE) are used as the measure of performance. The
quality of students is measured using demographic data. In particular, parental education
is the primary predictor.
For more technical readers, I performed an exploratory data analysis, including creating
multiple graphs, computing means, medians, Pearson correlation coefficients, and
Spearman correlation coefficients. Then I performed various linear regressions using the
R statistical programming language. Standard diagnostic tests were performed; outliers
and influential points were analyzed; Box-Cox transformations were performed; and
alternative models were considered. Mixed effects models were also analyzed. Then a
peer subgroup was selected and the analyses were performed anew. The results were
consistent across all models.
2
Introduction and Background Information
Each year California public school students take a battery of tests. The California
Department of Education (CDE), then weights the test scores, producing a composite
score for each school called the Academic Performance Index (API). This study uses the
base API scores as the measure of school performance. There are also “growth” scores,
but there is little difference between the two.
CDE converts the API scores using two distinct decile rankings. One is an overall
ranking. The second, and the one which is more relevant to this study, is a peer ranking
based on demographic data. In 2002, Albany High School (AHS) ranked in the top 10%
overall (decile 10) and in the top 30% (decile 8) in the CDE peer group. By 2009, AHS
fell to the top 20% overall (decile 9) and to the bottom 10% in its peer group (decile 1).
See Table 3 below.
This report takes an in depth look at the declining AHS performance, both with respect to
a subset of all schools and to a peer group based on schools which had similar
demographics in 2002.
In 2002, Albany voters elected several new school board members and they along with
the superintendent embarked on a policy of admitting large numbers of non-residents
from lower performing districts such as the West Contra Costa County Unified School
District.
The Albany Unified School District (AUSD) administration declined to provide separate
data for residents and non-residents, so I can only note that the decline in performance
occurred at the same time as non-district enrollment increased. Further research based on
more detailed data will be needed in order to determine if there is a cause and effect
relationship.
The next section describes the declining performance of Albany High School and uses
some graphs to clearly present the data’s message. Then a statistical analysis section
includes a more technical analysis and that section is followed by some example R code.
3
Performance of Albany High School on the API
There are several ways to see that Albany High School (AHS) has had a declining
performance and that its recent scores are well below the level they should be.
From 2002 to 2009, overall state scores increased 96 points, from 645 to 741, and peer
group scores increased 71 points, from 790 to 861. In contrast Albany High School’s
scores decreased 14 points, going from 820 to 806. See Chart 1.
A second way to look at the data is via ranks. Out of 677 schools statewide, in 2002 AHS
ranked 21 on the API but in 2009 AHS fell to a rank of 135. In the peer group of 50
schools, AHS fell from 12th place to 47th place. See Chart 2.
A third approach, and the main emphasis of the statistical analysis section, compared the
actual AHS score to its predicted score. In the first few years, AHS scored better on the
API than it was predicted to do and did so by as much as 30 points. Unfortunately by
2009, it scored 62 points below its predicted score, scoring 806, even though its predicted
score was 868 when parental education was the predictor and no adjustments were made
to the model. See Chart 3.
4
A more sophisticated technique, mixed effects for longitudinal data, provided further
evidence of the decline.
In conclusion, the analyses produced consistent results; Albany High School scored
below its potential and there has been a decline in recent years. The decline is associated
with a steep increase in transfers into the district from poorly performing districts such as
the West Contra Costa Unified School District. The Albany Unified School District needs
to produce data that will enable researchers to determine if the transfer policy is the cause
of the decline in Albany’s scores.
5
Statistical Analysis of the Data
1. Overview
Based on informal observations of the performance of Albany High School (AHS) on
California’s Academic Performance Index (API) combined with the knowledge that after
the November 2002 elections the Albany School Board embarked on an aggressive
campaign to admit large numbers of students from very poorly performing districts, I
hypothesized that there was a statistically significant and meaningful decline in the AHS
performance. Albany is a highly educated community that has a strong focus on student
education, so it is not surprising that AHS students continue to score well above the state
average on the API. However, as demonstrated by comparisons with peer groups,
conditional on the AHS demographics, in recent years AHS scored well below
expectations. Comparing the trend of AHS scores to the state’s and to the peer group’s
shows that the AHS trend is also negative. See Charts 1, 2, and 3 above.
The intent of this paper is to provide information that is useful to the general public and
which is then supported by statistical analyses. Consequently, most results use average
parental education (education level of the student’s most educated parent) as the single
predictor. By state law, California uses numerous predictors which are then summarized
by a School Characteristics Index (SCI). The state then uses the SCI to select 100 similar
schools and uses SCI as its main predictor.
Since the SCI is not an intuitive explanatory variable, CDE has performed some analyses
based on the three main predictors – average parental education (AVGED), percentage of
students on a government funded reduced price meals program (MEALS) and the
percentage of students who are English Learners (EL).
I did analyses using just AVGED and also using the three main predictors that CDE used.
While in some years the standard ANOVA for comparing a model and a subset of that
model produced a statistically significant difference in the models, for the purpose of
presenting findings to the public and for public policy purposes, the predicted scores for
AHS differed by an inconsequential amount. Another analysis was performed using SCI
as the predictor.
Moreover, Albany is atypical in that many English learners are children of very highly
educated people, including graduate students. Furthermore some recent studies question
the accuracy of the English Learners classification methods. A further examination of
Albany’s demographics is beyond the scope of this paper, but it does seem that based on
these subject matter considerations just using average education is a reasonable approach.
2. Methodology overview
For each year, 2002-2009, the standard normal regression modeling techniques were
used. The goal was to find a simple model that was capable of giving reasonably accurate
6
predictions for schools in the upper quarter. Various diagnostic tests were performed and
the Box-Cox transformation was utilized.
Finally, mixed effects models for longitudinal data were analyzed.
A subset of 50 peer schools was chosen based on schools with approximately the same
AVGED as AHS had in 2002. Then the analyses were repeated.
The results were consistent across all analyses – AHS performed poorly given its
demographics and its performance declined in recent years. Please see the table below.
3. The data
The data was downloaded from the CDE API database web site. The documents were
read into R and the data was merged into one large data frame. Only the 677 schools that
had at least 300 scores on the 2002 API (AHS had 576) and were not flagged for an
irregularity and which had an AVGED greater than 1.3 for all years, 2002-2009, were
included. AHS AVGED ~ 4.1.
AVGED is computed as follows. Each student’s most highly educated parent’s education
is scored as: Less than HS graduate = 1; HS graduate = 2; some college = 3; college
degree = 4; graduate degree = 5. Then the average is taken for the school.
My selection process eliminated many of the lowest performing schools but AHS is in the
top quarter of schools, so the selection process had little impact on the analysis. See
Table 3 for a comparison of the report’s descriptive data for the full dataset.
On numerous occasions in 2010, including API demographic data, the Albany Unified
School District submitted inaccurate data to CDE, the details of which are beyond the
scope of this report. Consequently 2010 API results were not included in this report.
No student level data was available to me. School level data was used.
4. The main normal model analyses
My starting point was the standard one predictor linear model with API the response
variable and average parental education (AVGED) the predictor. Separate analyses were
done for each year, 2002-2009. Scatter plots and other diagnostic tools produced
similarly shaped distributions of data. Therefore, year 2009 data will be the main
example data.
The scatter plot indicated that a straight line model was a reasonable starting point.
However, plots of residuals versus predicted values and qqplots indicated that there were
some very large residuals. I used several strategies to transform the data, including
7
omitting the most influential points or omitting the largest outliers. Additionally a BoxCox transformation was performed.
I also analyzed the three predictor model with average parental education (AVGED),
percentage of students on the government subsidized meals program (MEALS), and the
percent of English Learners (EL) as the three predictors.
Finally, I used the School Characteristics Index (SCI) as my explanatory variable.
The AHS predicted scores were similar across all methodologies and are summarized
below. See Table 1.
5. A closer look at the 2009 data
See Tables 1, 2, and 3 for summaries of the results.
The standard linear models and summary functions were run. There were several outliers
and they were sequentially removed. However, the AHS predictions for 2002 and 2009
did not change.
Then the analysis was redone with leverage as the statistic of interest. Cook’s distances
were computed. In this case I eliminated the points with the 14 largest distances, but the
2002 and 2009 AHS predicted scores only decreased by five and two points, respectively.
The Box-Cox transformations reduced the 2002 and 2009 AHS predicted scores by zero
and six points, respectively.
8
The three predictor model was also analyzed. Predictions were similar and the standard
F-test to compare a model to a sub-model was not significant at the 5% level for 2002
and was not significant at the 1% level for 2009.
Finally mixed effects models were analyzed and similar results obtained. The selected
model had AVGED as a fixed effects predictor and an interaction term, AVGED*Years.
Then I selected a peer group of 50 schools, which had similar AVGED scores in 2002. I
redid the analysis. The results were quite similar to the results for the full data set.
Conclusion
Multiple statistical methods demonstrate that Albany High School is underperforming on
the Academic Performance Index and in recent years its performance relative to all state
schools and to a peer group has declined.
9
Summary Tables
Table 1
All Data: AHS score versus predicted scores
Year
AHS
API
Predicted
(AVGED)
Predicted
(AVGED +
MEALS +
EL)
2002
817
800
796
779
795
800
800
801
2009
806
868
860
859
866
868
862
869
Predicted
(SCI)
Predicted
w/o
influential
Predicted w/o
outliers
Predicted BoxCox
Predicted
Mixed
Effects
Table 2
Peer Data: AHS score versus
predicted scores
Year
2002
2009
AHS
API
817
806
Predicted
(AVGED
Predicted + MEALS
(AVGED)
+ EL)
806
802
865
850
Table 3
AHS Deciles: For this report and as reported by CDE
2002
2009
CDE Report
Decile Decile
10
10
9
9
CDE
Peer
Decile
8
1
Report
Peer
Decile
8
1
10
CDE
Median
API Score
585
676
Report
Median API
Score
645
741
Bibliography
Bates, Douglas. (2010). lme4: Mixed-effects Modeling with R. http://lme4.r-forge.rproject.org/book/. Madison, Wisconsin.
Bates, Douglas. (2011). Short Course: Mixed models in R using the lme4 package.
http://lme4.r-forge.r-project.org/slides/2011-01-11-Madison/. Madison, Wisconsin.
California Department of Education API site. http://www.cde.ca.gov/ta/ac/ap/
Conover, W.J. (1999). Practical Nonparametric Statistics, Third Edition. John Wiley &
Sons. Hoboken, NJ.
Dalgaard, Peter. (2008). Introductory Statistics with R. Springer. New York, NY.
Faraway, Julian. (2005). Linear Models with R. Chapman & Hall, Boca Raton, FL.
Guo, S, Liu, R, Bernstein, R. (2009). Comparing Similar Schools Rank Changes:
Evaluating a Three Individual Variable Method. California Educational Research
Association Conference Presentation, San Francisco, CA.
Iman, Ronald and Conover, W.J. (1979). The Use of the Rank Transform in Regression.
Technometrics, vol 21, no 4.
Kutner, M, Nachtsheim, C, Neter, J, Li, W. (2004). Applied Linear Statistical Models,
Fifth Edition. McGraw-Hill Irwin, New York, NY.
Spector, Phil. (2008). Data Manipulation with R. Springer. New York, NY.
11
Appendix A: List of Analyses Performed
All Data
lm(API ~ AVGED)
lm(API ~ AVGED + MEALS + EL)
lm(API~ SCI)
Mixed Effects library(lme4):
lmer(API ~ 1 + AVGED + AVGED*Years + (1 | Years))
Peer Data
lm(API ~ AVGED)
lm(API ~ AVGED + MEALS + EL)
lm(API~ SCI)
Diagnostic and remedial techniques
Plots of residuals versus predicted
Plots of abs(residuals) versus predicted
qqplots
Analysis of outliers
Analysis of leverage points
Tests for normality of residuals
Tests for constant variance
F tests for model reduction
Box-Cox transformations
12