Jonathan Wiens
Assessment Specialist
Oregon Department of Education
What is Growth?
By growth we mean a measure of the
change in student test scores from year to
year.
The evaluation of growth can be:
 Criterion-based: Growth is measured
relative to a particular goal.
 Normative: Growth is measured relative
to other schools or students.
2
Growth Model
Considerations
3
Types of Growth Models
Growth models can measure different effects, such as:
 “Improvement” – the change in performance at a
particular grade or school from year to year. Compares
different sets of students from year to year. (AYP Safe
Harbor)
 Cohort – the change in performance of a particular
group of students over time. (Probability Curve)
 Individual Student – the change in an individual
student’s score over time. (Report Card Growth
Model)
4
Growth Model Measures
Growth models can measure/predict:
 Changes in percent met
 Changes in mean (average) test scores
 Individual student gains
These gains can be analyzed to measure:
 Rates of student learning
 Long term trends
 Effect Sizes
 Changes in achievement gaps
5
Statistical Considerations
Various statistical models can be used to evaluate or
predict student growth. These can vary from
 Simple Gain Scores (change in test score or performance
level from year to year); to
 Complex multivariable regressions (use student test
histories, together with other factors to predict or
measure growth) using proprietary software.
Policy decisions weigh the benefits of a simple to explain
system versus a complex system that may be more
statistically reliable.*
* - These are not necessarily mutually exclusive.
6
Why Multiple
Measures
of Growth
Are Important
7
Case Study: A Tale of Four Schools
Consider four schools and their AYP performance:
School A
School B
School C
School D
% Met
AYP Status
63%
60%
60%
55%
Met
Met
Met
Not Met
The fourth school receives a poor rating, while the
other three are all given the same “Met” rating. Let’s
look at some of these schools in multiple ways.
8
Longitudinal School Performance
By looking at school performance longitudinally (i.e. year to
year) we begin to be able to differentiate the
performance of Schools A, B and C.*
Year 1
Year 2
Year 3
School A
69%
66%
60%
School B
60%
60%
60%
School C
51%
57%
63%
School D
48%
53%
57%
Basis for AYP
We also see that School D is showing improvement, but this is
not rewarded, while school C’s improvement is recognized by
AYP.
* Longitudinal school data is used for determining Safe Harbor and school
improvement status.
9
Longitudinal Cohort Performance
By following cohorts of students, as indicate by the arrows,
we can see another view of school effectiveness.
School B
Grade
Yr 1
Yr 2
Yr 3
6
58%
52%
55%%
7
61%
63%
57%
8
61%
65%
68%
Average
60%
60%
60%
Even though the school wide percentage of students
meeting standard is the same each year, this school is
moving students toward standard.
10
Individual Student Growth
The most detailed view of growth occurs at
the student level.
Reading
Cut
Grade Score Growth Score “Gap”*
3
190
204
14
4
206
16
211
5
5
211
5
218
7
6
219
8
222
3
Which years are a “success” for the student?
* - The “Gap” is the difference between the student score and the cut score.
11
Why Measure Growth?
Low
Achievement
High
Achievement
By looking at growth and status together, we can form a more complete picture
of school effectiveness. Growth models may be able to differentiate schools of
equal performance.
Coasting
Leading
Losing
Ground
Learning
Low Growth
High Growth
12
Accountability
and Reporting
13
AYP Growth Models
The Growth Model Pilot Project allows states to apply
growth in AYP determinations. The pilot project
placed restrictions on growth models. They are
generally criterion-based. Requirements include:
 Expect students to be at standard within three or four
years.
 Allow students who meet growth expectations to be
counted positively in AYP determinations.
 May not use margin of error.
 Are technically adequate measures of growth.
14
Growth Models for AYP
Models in use include:
 Growth Targets – Students are given individuals target
scores for each year. Students who meet their growth
targets are counted as “met” for AYP. (AR, AZ, FL, etc)
 Performance Level – Schools are given credit (using an
index system) when students move up in performance
levels. (IA, DE)
 Projection Models – Each student’s test history is used to
project future performance. Schools are given credit for
students projected to be proficient in three or four years.
(TN, OH, CO*)
* - Colorado’s model is a combination of projection and target setting.
15
New Federal Requirements
The State Fiscal Stabilization Fund requires ODE report:
 The “average statewide school gains” in reading and in
math for each AYP subgroup.
 The number and percentage of Title I and charter
schools that have made progress in reading and math.
 Making progress is a normative measure.
ODE has committed to produce a school gains measure
and to publish the results by September 2011.*
* - The Federal School Improvement Grant can also potentially use growth in its
determination of the low performing schools.
16
Stakeholder Requests
ODE has also received a number of requests to
implement a growth model that includes all students.
 The report card growth model applies only to students
who did not meet in the prior year.
 Stakeholders have asked that our current model be
extended to students who meet and to students who
exceed.
 So …
17
Over the next year…
 We will be working with stakeholders to
extend our growth model to all students.
 ODE will be working with stakeholders to
determine a school gains model.
 We are currently exploring several potential
models.
 Two of the models that use individual student
growth are reviewed in this presentation.
 Next, an overview of these models …
18
Individual Student
Growth Models
-- Two Examples
19
Student Level Growth
How do we evaluate growth at the student level?
 One size fits all?
 Varied growth expectations?
How do we evaluate schools?
 Percent meeting growth expectations?
 Average Growth?
The next slide shows why this is a thorny issue:
20
Median Reading Growth – Actual Data
Growth From Grade 5
25
20
15
10
5
0
200
-5
210
220
230
240
250
-10
Growth to Grade 6
Growth to Grade 7
Growth to Grade 8
21
Oregon’s Model -Growth Targets
The growth model Oregon has implemented on the
school report cards is criterion-based.
 Students below standard in the previous year are
given a target score for the current year.*
 Students whose score is at or above their target “meet
growth.”
 Targets are based on the student’s prior year test score
and the growth in cut scores.
* - does not apply to 10th grade.
22
Growth
Targets are
Provided
for each
student
who did
not meet
Available on the
Growth Model web
page
23
Growth Target Calculator
Available on the Growth Model web page
24
Progress Targets
We need to develop “progress targets” for students that
meet or exceed. Possible features of these targets
include:
 Progress targets must align with growth targets.
 Growth should be expected for all students.
 Higher performing students may have lower growth
expectations.
The following slides show some preliminary data …
25
2009-10 Growth Data
 The following charts show the percentage of students
meeting either their growth target or their progress
target in 2009-10, by prior year test score.
 One possible choice for progress targets is shown.
 Notice that the growth and progress targets are
approximately equally rigorous across achievement
levels at most grades.
 There are some remaining issues, however.
26
Growth in Grade 4
100.0
Percent Meeting Growth
80.0
60.0
40.0
20.0
0.0
180
185
190
195
200
205
210
215
220
225
230
235
240
245
250
Score in Grade 3
Math
Reading
27
Growth in Grade 5
100.0
Percent Meeting Growth
80.0
60.0
40.0
20.0
0.0
185
190
195
200
205
210
215
220
225
230
235
240
245
250
255
Score in Grade 4
Math
Reading
28
Growth in Grade 8
100.0
Percent Meeting Growth
80.0
60.0
40.0
20.0
0.0
200
205
210
215
220
225
230
235
240
245
250
255
260
265
270
275
Score in Grade 7
Math
Reading
29
Growth Percentiles
– The Colorado Model
Colorado has implemented a growth model that is
normative.
 A student’s growth is compared to other students with
the same prior achievement.
 School growth is reported relative to state averages.
30
Growth Percentiles
Student growth is measured in percentiles.
 Among students with the same score last year, how
does a particular student’s growth compare to others
 Students are given a “growth percentile.”
 A growth percentile of 60 would mean that student
showed as much or more growth than 60 percent of
the students who started with the same score.
 The average growth percentile is calculated for each
school.
 Results are displayed using the quadrant model.
31
Growth Percentiles
 This model should, by definition, be independent of
prior status.
 While normative, it can be applied to determine
progress toward meeting standard.
 The background model is statistically complex, but the
data is easy to understand.
 ODE will be running this model on 2009-10
assessment data.
32
During the next year …
 ODE and stakeholder advisory groups will examine
these two models.
 Each may inform the other.
 We hope to have a growth model for meets and
exceeds students by September 2011.
33
Resources
 Longitudinal Student Growth Model web page is at:
http://www.ode.state.or.us/search/page/?id=2495
 This page contains links to:
– Growth Model Targets spreadsheet
– Growth Target Calculator
– 2008-09 Probability Curve Spreadsheet
 e-mail: jon.wiens@state.or.us
34