Running head: ARTICLE CRTIQUE
Article Critique
Joshua E. Crespi
The Florida State University
1
2
ARTICLE CRITIQUE
Introduction/Overview
Carnoy and Loeb adequately and extensively explain the research topic area of
accountability and student testing clearly stated in the opening paragraph. The first paragraph of
the article states its objective to be to "examine factors associated with stronger accountability
and ask whether stronger statewide approaches to accountability achieve their declared goal of
improving student outcomes" (Carnoy & Loeb, 2002). The authors then go on to discuss the
measurement tools to consist of the performance students show on tests, however it is seen that
students' "improvements" on state tests might not always be the most accurate level of
measurement (Carnoy & Loeb, 2002). In this way, the authors acknowledge other means of
measurement included in their research to include retention rates, math test scores, and high
school survival rates.
Carnoy and Loeb begin their article with a short defense to the educational importance
behind the research and the reason/problem area for which the research has been conducted. It
has been seen that policy and state control are affected by strong accountability of standardized
testing. However, the authors do not pose a strong identified problem area. It is proposed that the
main research question seeks to identify if stronger statewide approaches can better assist in the
achievement of improving students' results, but the authors do not state the need or desired
amount of improvement for student’s outcomes. In this way, more clarity from the authors of the
article to identify the need/problem area for which the research was conducted would be
beneficial to the overall theme of their thesis.
After clearly identifying the purpose and modes of measurement of the research for the
article through their clear thesis statement, Carnoy and Loeb pose five interest questions. The
theme of the research questions are specific and particular in which the authors ask for particular
3
ARTICLE CRITIQUE
examples, instances, and clarification in their research. For example, in the first research
question Carony and Loeb clarify the first question by posing another question asking for a more
particular example. Along these same lines, the more clarified question is then followed up by
discussing the importance of this analysis to better show the relationship between accountability,
student performance, and the underlying factors that are at the heart of this study. In the posed
research questions numbers three and four, hypotheses are clearly stated after the relative
research questions with a scholarly source cited at the end of that research. It is seen that the
following hypotheses are included in the research questions: "Once these systems are
implemented, they are supposed to improve the likelihood that students finish high school. But
according to some analysts, the pressure on schools to do well on high school minimum
competency tests pushes administrators and teachers to increase retention rates in the first year of
high school" (Carnoy & Loeb, 2002) and " if accountability raises student performance in the
earlier grades, this should ultimately raise the proportion completing high school; if, however,
strong accountability increases retention rates in earlier grades, it could well reduce survival
rates" (Carnoy & Loeb, 2002).
Literature Review
The research questions presented at the beginning of the article were broad. The main
goal of this article was to find the relationship between accountability and educational success.
However, additional questions are posed at the beginning to find various other relationships
between accountability and the school systems. This article also strived to examine the
characteristics between varying states' test results, retention rates, progression rates, and effect on
ethnic population. With these many goals in mind, it is necessary for the article to have broad
sources such as "Negro Education in Alabama: A Study in Cotton and Steel" (Carnoy & Loeb,
4
ARTICLE CRITIQUE
2002). The author is able to surprisingly bring in these various sources without ruining the flow
of the article.
The literature used to support the authors case comes from a variety of sources. The main
type of sources used were peer-reviewed journals such as "Economics of Education Review"
(Carnoy & Loeb, 2002). However, many other types of sources are evident such as published
books. Additionally, data collected from other studies was used to eliminate bias when possible.
The study takes quantitative data from various other studies and pulls them together to form new
data. In this way, information from various others states are taken to form a more complete study
of the country. The study attempts to set controls to make the data from the different studies
come together to form more correct data. For example, when comparing 8th graders and 4th
graders to determine if they are performing at or better than skill level, the researchers "...control
for the 1996 test score to test whether lower scoring states in 1996 had a significantly higher gain
in the next four years independent of the accountability index. We include as controls those
variables that were significant predictors of accountability strength in Equation 1. We also
include population growth and growth in the percent of Black or Hispanic students" (Carnoy &
Loeb, 2002).
Methods (Design, Data Sources, Sample, Measures & Analysis)
The research design used in this article is infernal statistics with a multiple regression
analysis. There are various independent variables used in this study such as average scale score
of 4th grade students in state I (t), proportion of African American and Hispanic students in state
I (r), the state population (p), proportion of state funding (s), and dollars per pupil (d). In that
example, the dependent variable is the strength of accountability (a). This equation attempts to
estimate accountability implementation as a function. The study then uses another equation to
5
ARTICLE CRITIQUE
determine student achievement as a function of accountability. In this equation, the 1996 test
scores and variables that were significant predicting accountability are used as controls as
previously describe. A final equation is used to check the coefficient and to check for possible
bias. There are various charts throughout the study that clearly label the independent variables,
the dependent variable, and their scores in each area.
This study has been conducted using a quantitative approach. Even with the various
controls in the study, I feel that this is a threat to the internal validity of the study. The age
groups studied range from the young (4th graders) to the older (high school students). Even when
accountability is held at a state level, students as young as 4th graders do not always comprehend
the importance of their school work. This means they may not always perform to their best
ability on tests. No level of accountability at any level would be able to make these students
understand the importance of their performance. This is something the students must figure out
on their own. It is seen that these threats described are briefly mentioned in the study.
Results
As previously described, the authors chose to include various hypotheses in their five
research questions. During the results portion of their article, the answers to these research
questions and results of their hypotheses were clearly identified. Firstly, Carony and Loeb
hypothesized “that states with lower student test scores earlier in the 1990s would be more likely
to implement strong systems" (Carnoy & Loeb, 2002). In the results, the study was shown to
support this hypothesis. However, this hypothesis was only supported as seen in White students
test scores. It was found that relationship did not exist "between early Black student test scores
and the strength of accountability" (Carnoy & Loeb, 2002). Secondly, in regards to Carony and
6
ARTICLE CRITIQUE
Loeb's centralization hypothesis, the authors included public school's revenues. In this way, it
was shown that many schools in the south carried a centralized funding system.
Furthermore, Carony and Loeb clearly present and discuss the results. With every result
that was given, the authors made sure to give a short discussion on the meaning on the findings.
For example, as previously seen, their first hypothesis was support, but only for the White
student population. Carony and Loeb then took the time to discuss what these finding meant for
the research of the tests and scores and strength of accountability.
Discussion
In sum, the results support the conclusions drawn from this study. As previously
explained, the results correlated with the proposed hypotheses in the beginning of the article.
However, the hypotheses held true for only certain populations, as it is described that "the results
suggest that states with lower achieving White students are more likely to implement strong
systems" (Carnoy & Loeb, 2002). Carony and Loeb offer alternative explanations that are
presented in description of none evident "positive affect of accountability on student progression
through high school" (Carnoy & Loeb, 2002). Carony and Loeb at this point go through three
possible explanations as to why the results show no evidence of this positive affect. For example,
Carony and Loeb clearly state that “an alternative explanation is that despite the positive effects
of high-stakes accountability on math test scores, it may simply be too early to assess the longterm implications of this relatively new policy initiative on attainment outcomes" (Carnoy &
Loeb, 2002). Also concerning the conclusion portion of this article/study, the authors did not
discuss the findings in comparison to prior research. Furthermore, no further research or research
directions are identified. The article concludes with the affirmation of a positive relationship
existing "for Black, White, and Hispanic students" (Carnoy & Loeb, 2002).
7
ARTICLE CRITIQUE
References
Carnoy, M., & Loeb, S. (2002). Does external accountability affect student outcomes? A crossstate analysis. Educational Evaluation and Policy Analysis, 305-331.
Johnson, B., & Chrstensen, L. (2008). Educational research: Quantitative, qualitiative, and
mixed approaches. Los Angeles: Sage Publications, Inc.