Using Data Warehouse Historical
Data as Baselines for Student Growth
DATAG Presentation
Randy Simmons
December 7, 2012
Eastern Suffolk BOCES Research
Objective – To Document the Reliable Correlation
of Leveled Scores on Related NYS Assessments
Taken in Sequence
Purpose - to Provide Data Warehouse Districts in
Suffolk Country Historical Data to Use for SLO
Growth Predictions
Problem – Current rush to develop parallel local
assessments that provide valid measures of
student achievement at the beginning of the year.
Typical District High School Local Assessment Plan






Use prior Regents questions for course that have not taken
as yet.
This type of assessment assures radically low baseline
scores because it violates the basic assessment principle of
the student having an “Opportunity to Learn”.
September 12, 2012
Bridges Data and Professional
Development
2
Research Longitudinal Data Files




First File – 2006 to 2011 student
performance from ELA and Math grade 7 to
all Regents Scores in following four years
Second File – 2007 to 2012 student
performance from ELA and Math grade 7 to
all Regents Scores in following four years
First file has 27,028 student cases who
took multiple assessments in three or more
years.
Second file has 27,079 student cases who
took multiple assessments in three or more
years.
September 12, 2012
Bridges Data and Professional
Development
3
The Key Issues for the Problem of
Documenting Growth


Can we use leveled performance on NYS
Assessments to set a baseline from which to
measure growth?
Key Issues




Is student performance stable across NYS tests in the same
department or content area?
Can we adjust historical student performance data to align
with the higher performance level expectations that began
in 2010?
How can we convert unequal scale scores to score scales
with comparable performance levels?
What method can we use to set growth or achievement
expectations?
December 7, 2012
Bridges Data and Professional
Development
4
ELA Performance from Grade 7 to 11
Comparison of Suffolk County ELA Success Rates on the ELA-7, ELA-8
and English Regents in 2006 - 2010
100.0%
93.2%
90.0%
80.0%
70.0%
67.5%
70.5%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
2006 ELA-7
September 12, 2012
2007 ELA-8
Bridges Data and Professional
Development
2010 English Regents
5
ELA Performance of Same Students
from Grade 7 to 11
Suffolk County Student ELA Growth of Proficiency Rates for
Matching Students Going From 2006 ELA-7 to the Grade 11
English Regents
96.4%
100.0%
90.0%
80.0%
72.0%
76.4%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
2006 ELA-7
September 12, 2012
2007 ELA-8
Bridges Data and Professional
Development
2010 English Regents
6
Using the NYS Reset ELA Levels to Create
Adjusted Equated Levels and Scores

Apply the 2010 scale score cut points to earlier
historical 2006-2009 data

NYS equated the ELA-8 level 3 cut point with 75 on
the English Regents

Measuring growth requires scales that are equivalent

Convert ELA3-8 and English Regents to a Scale with equally
distributed leveled scores

Level defined percentile scores with 25 P-Scores in each level
Bridges Data and Professional
Development
7
ELA Adjusted Performance of Same
Students from Grade 7 to 11
Suffolk County Student ELA Growth of Adjusted Proficiency
Rates for Matching Students Going From 2006 to 2010
100.0%
90.0%
82.5%
80.0%
65.9%
70.0%
55.3%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
2006 ELA-7
September 12, 2012
2007 ELA-8
Bridges Data and Professional
Development
2010 English Regents
8
Correlations of Adjusted ELA
Performance from 2006 to 2010
* All correlations are significant at the .01 level of significance. There were
14, 614 students who took all three tests.
Bridges Data and Professional
Development
9
Correlations of Adjusted ELA
Performance from 2007 to 2011
Correlation Matrix
New ELA707
Level P-Score
New ELA808
Level P-Score
ELA11 Level
P-Score
Correlation
New ELA707 Level
1.000
.939
.832
P-Score
New ELA808 Level
.939
1.000
.838
P-Score
ELA11 Level P-Score
.832
.838
1.000
Sig. (1-tailed) New ELA707 Level
.000
.000
P-Score
New ELA808 Level
.000
.000
P-Score
ELA11 Level P-Score
.000
.000
* All correlations are significant at the .01 level of significance. There were
15, 177 students who took all three tests.
Bridges Data and Professional
Development
10
Sub-Group ELA to English Regents
Correlations
Display of Correlations among ELA-8 to English Regents Adjusted Scores by
Poverty, Disabilty and LEP Status
Correlation Coefficients
0.70
Not Poverty
0.67
0.66
0.65
Poverty
0.63
0.63
0.61
0.60
0.56
0.55
Disability
Not
Disabled
LEP
Not LEP
0.50
Bridges Data and Professional
Development
11
Indicators of Stable Performance

ELA-7 to ELA-8 year to year performance is
strongly correlated (.71 and .91 in 2007-08).

Three year correlations from ELA-8 to the Grade
11 English Regents remains strong (.67 and .84
in 2008-2011).

The decline in this correlation among disabled,
ELL and low income is minimal.

There is evidence of year by year increases in
adjusted leveled performance.
Bridges Data and Professional
Development
12
A Model for Projecting Future
Student Performance


Identify the mean
average historical
performance of
students in short
score ranges in levels
2 and 3
Smaller numbers of
students in levels 1
and 4 require larger
score range groups for
projections
December 7, 2012


Level 1
Levels 2 and 3





Bottom Quartile
2nd Quartile
3rd Quartile
Top Quartile
Level 4

Low and High
Bridges Data and Professional
Development
13
A Model for Projecting Future
Student Performance
December 7, 2012
Bridges Data and Professional
Development
14
Essential Questions Break

In Table Groups Answer the Following –

What are your districts doing to determine
baselines for local growth?

What tests do you believe can be reasonably
linked in a sequence to predict growth?

How would you like your Data Warehouse to
provide you historical data for local
measurement of growth?
December 7, 2012
Bridges Data and Professional
Development
15
Math Performance of Same Students
from Grade 7 to 11
Comparison of Suffolk Country Success Rates in Math-8,
Integrated Algebra and Geometry from 2007 to 2009
85.5%
90.0%
80.0%
70.0%
80.6%
73.1%
67.4%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
2006 Math-7
September 12, 2012
2007 Math-8
2008 Algebra
Bridges Data and Professional
Development
Suffolk County
2009 Geometry
16
Math Adjusted Performance of Same
Students from Grade 7 to 11
Suffolk County Student Math Growth by Performance Levels
for Matching Students Going From 2007 Math-8 to 2009
Algebra to 2010 Geometry
100.0%
90.0%
82.2%
80.0%
70.0%
60.0%
50.0%
40.0%
43.5%
44.5%
42.8%
30.0%
20.0%
10.0%
0.0%
2006 Math-7
September 12, 2012
2007
Math-8
2008 Algebra
Bridges Data and Professional
Development
2009 Geometry17
Correlations of Adjusted Math-8 to IA-9
Performance
Correlation Matrix
New Math807
Level P-Score
Integrated
Algebra908
Level New
P-Score
1.000
.528
.635
.528
1.000
.546
.635
.546
1.000
.000
.000
New Math706
Level P-Score
Correlation
New Math706 Level
P-Score
New Math807 Level
P-Score
Integrated Algebra908
Level New P-Score
Sig. (1-tailed) New Math706 Level
P-Score
* All correlations
are significant at the
New Math807 Level
number of students
P-Score taking the Math 8
9 was 8,880. Integrated Algebra908
Level New P-Score
December 7, 2012
.01 level of significance. The
.000
.000
followed
by the IA Regents
in grades
.000
Bridges Data and Professional
Development
.000
18
Sub-Group Math-8 to Regents
Correlations
Display of Correlations among Math-8 to Integrated Algebra and Geometry
Regents Adjusted Scores by Poverty, Disabilty and LEP Status
Correlation Coefficients
0.50
0.44
0.40
Not Poverty
0.47
0.43
0.41 0.42
0.36
Poverty
0.38
0.34
0.36 0.36 0.36
0.31
0.30
Not
Disabled
LEP
0.20
0.10
Not LEP
0.00
Math-8 to IA
December 7, 2012
Disability
Math-8 to Geometry
Bridges Data and Professional
Development
19
Indicators of Stable Performance

Math-7 and Math-8 year to year performance is well
correlated (.53), but not as strongly correlated as the ELA8 and following tests

Year to year correlations from Math-8 to Integrated Algebra
and Geometry is similar, probably because different parts of
the cohort take these courses in different years

The decline in this correlation among disabled, ELL and low
income is minimal

Stable adjusted levels from Math-8 to Algebra indicate that
the NYS projected outcome of 80 on Integrated Algebra for
students reaching the bottom of level 3 is correct.
December 7, 2012
Bridges Data and Professional
Development
20
A Model for Projecting Future
Student Performance
December 7, 2012
Bridges Data and Professional
Development
21
Correlations of Leveled P-Scores in
Science – Three or More Paths
Correlation Matrix
Science807
Level P-Score
Living
Environmen
t908 Level
P-Score
EARSCI1009
PScore
Correlation
Science807 Level
1.000
.678
.613
P-Score
Living Environment908
.678
1.000
.894
Level P-Score
EARSCI1009PScore
.613
.894
1.000
Sig. (1-tailed) Science807 Level
.161
.193
P-Score
Living Environment908
.161
.053
Level P-Score
EARSCI1009PScore
.053
* All correlations
are significant at the .01 .193
level of significance.
There were
4,158 students who took Living Environment
9. There were
5,787
.151in grade .402
.272
who took Earth Science in grade
in grade 10.
Bridges Data and Professional
Development
22
Correlations of Leveled P-Scores in
Science – Three or More Paths
* All correlations are significant at the .01 level of significance. There were
2, 019 students who took Earth Science in grade 9 followed by Living
Environment in grade 10.
Bridges Data and Professional
23
Development
Correlations of Leveled P-Scores in
Science – The Advanced Path
* All correlations are significant at the .01 level of significance. There were
1, 117 students who took Living Environment in grade 8 and Earth Science
in grade 9.
Bridges Data and Professional
24
Development
Indicators of Stable Performance

Science-8 year to year performance is strongly
correlated (.70 and .68) with following Earth
Science and Living Environment performance

Year to year correlations from Living Environment
to Earth Science is even stronger (.86)

Earth Science performance is strongly correlated
with Living Environment when it is taken first (.71)

There are three or more paths through the Science
Regents course based upon district policy and
when highly performing students take their first
Regents course.
Bridges Data and Professional
Development
25
Correlations Starting From Old Social
Studies- Through U.S. History
Correlation Matrix
SS807 Level
P-Score
Global
His tory1009
Level P-Score
US
His tory1110
Level P-Score
Correlation
SS807 Level
1.000
.744
.704
P-Score
Global His tory1009
.744
1.000
.764
Level P-Score
US History1110
.704
.764
1.000
Level P-Score
Sig. (1-tailed) SS807 Level
.000
.000
P-Score
* All correlations are significant at the .01 level of significance. There were
Global His tory1009
.000
.000
16,132 students
who took the Social Studies-8
and Global History in
grade
Level P-Score
10. There were 15,538 that took the Social Studies-8 and U.S. History in
US History1110
grade 11.
.000
.000
Level P-Score
Bridges Data and Professional
Development
26
Next Steps
Use the SPSS Modeler to calculate prediction success rates
which could be applied to achievement targets.
Specify projections for NCLB sub-groups which will improve
prediction estimate.
When projections are based upon 2 or 3 year gap of time,
create a trajectory of growth ending in the end point
prediction.
Generate Model for DW Reports Re-Rostered that provides
school districts baselines and optional achievement growth
projections with a known margin of error
Example –









ELA-8 Students in the top quartile of level 2 are projected to
achieve an average 81 (within a 3 point margin of error) on the
grade 11 English Regents
Projected prediction rate of success – 80%
In grade 10 this student is projected to achieve an average of 78.
In grade 9, this student is projected to achieve an average of 75.
December 7, 2012
Bridges Data and Professional
Development
27