Jason Brinkley – ECU BIOS
October 2013

• Data was provided from Greene County schools in the form of EVOS data for every student in grades 4-8 with a focus on reading and math scores. While there was some variation in what information was pulled, in general the combined data had the following:
• Data collection school year: 2009-2010, 2010-2011,
2011-2012
Pseudo Student ID: For tracking across three years of data collection
Teacher Code: For tracking teachers across three years of data collection
Demographics: Gender, Grade, Ethnicity, LEP

• The current year’s test score
• A standardized ‘C Score’
• A state assigned ‘predicted score’ and ‘predicted C Score’
• A growth variable that measures the difference between the students achieved ‘C score’ and the state ‘Predicted C score’.
• For teacher achievement purposes, growth is measured by how well students perform versus predictions.
• Test scores are also given ‘level’ which are ordinal categories that represent achievement cut-offs (1,2,3,4) and correspond more to the traditional idea of end of grade testing.
• Past Year Scores: Same information as the test scores listed above, but for the previous school year.

Grade
Ethnicity sex
6
7
8
4
5
H
I
A
B
M
P
W
F
M
2009-
2010
2010-
2011
2011-
2012
20.57% 19.83% 18.07%
19.94% 20.37% 20.02%
20.97% 20.68% 20.27%
18.75% 20.68% 20.61%
19.78% 18.44% 21.03%
0.16% 0.15% 0.00%
41.54% 41.28% 40.41%
24.07% 24.92% 25.61%
0.00% 0.15% 0.08%
2.62%
0.00%
1.47%
0.08%
1.44%
0.08%
31.61% 31.94% 32.38%
50.28% 49.69% 50.51%
49.72% 50.31% 49.49%

Score
Reading
N
Mean
Std Dev
Score
Math
N
Mean
Std Dev
2009-2010
1267
352
9
2009-2010
1259
348
10
Year
2010-2011
1244
354
9
Year
2010-2011
1212
349
10
2011-2012
1195
355
9
2011-2012
1184
350
10

• ANOVA testing reveals (p < 0.0001) that there are significant differences between years for both math and reading raw scores.
• Further examination shows that each year saw a significant increase in math scores while there is a significant difference between 09/10 and 11 only (no difference between 09 and 10 cohorts).

• The last three slides illustrate that Greene
County has made some improvements over the selected time frame.
• We also see that many students are deemed
‘proficient’ (i.e. they reach a level greater than
1), especially in math. Therefore looking at only a variable like ‘level’ as a student achievement outcome may not be as informative.

• C Scores and Growth Scores are a different type of measure on student impact.
• The end goal of using such measures is to provide a more standardized way of measuring student achievement and teacher impact.
• However, as we will see, these measures may prove to be difficult to use for research purposes.

2010-2011 2009-2010 2011-2012
Here we see the distribution of actual C Scores and state predicted C Scores for math tests for all three years of interest. Scores typically range from -2.5 to 2 with Greene
County schools averaging slightly less than 0 year. The correlation between predicted and actual scores hovers around 0.80, strong but not perfect.

C Score
Math Year
N
Mean
Std Dev
2009-2010
1223
-0.27
0.8
2010-2011
1211
-0.03
0.9
2011-2012
1163
0.05
0.9
Reading
C Score N
Mean
Std Dev
2009-2010
1259
-0.44
0.9
Year
2010-2011
1212
-0.38
0.9
2011-2012
1184
-0.24
0.9

Math Growth Averages:
2009-2010: -0.3
2010-2011: 0.19
2011-2012: 0.17
Percent with Positive
Growth:
2009-2010: 49%
2010-2011: 58%
2011-2012: 61%

Reading Growth
Averages:
2009-2010: -0.1
2010-2011: -0.03
2011-2012: 0.10
Percent with Positive
Growth:
2009-2010: 42%
2010-2011: 43%
2011-2012: 58%

• There were 27 teachers in each area in Greene County from the 2009-2010 data. Of the math teachers, on
40% had a mean growth value greater than zero. Only
11% of reading teachers showed positive growth.
• In 2010-2011 there were 32 teachers in each area and
75% had a mean growth greater than zero. 35% of reading teachers showed positive growth.
• In 2011-2012 there were 32 teachers in each area and
63% of math teachers had a mean growth greater than zero. 63% of reading teachers showed positive growth.

• The data shows that Greene County is moving in a positive direction. While we see only mild increases in raw end of grade scores, when those scores are centered toward statewide averages we see that the increases are even greater.
• Likewise we see on a one on one basis that many teachers are experiencing higher growth rates, there are more students who perform better than what the state is predicting.

The downside of just focusing on growth. Aggregated across all years.

Here we see that Growth skews when broken down by teachers with students predicted to get only 3 and 4 levels on reading EOG tests. We see that as Greene
County makes continued improvements that those students experience lower growth rates and those teachers have lower mean growth scores. Likely because those students start to hit some sort of glass ceiling in scores.

E
F
G
J
K
A
B
C
D
P
Q
R
Teacher
Code
Number of
Student s
26
40
26
112
26
111
21
0
0
40
0
26
Grade
4
8
4
5
4
7
4
.
.
5
.
4
2009-2010
Predicte d C
Score
Mean
Actual C
Score
Mean
-0.43
-0.49
.
.
0.01
-0.67
-0.43
-0.48
0.25
-0.36
.
-0.55
-0.36
-0.82
.
.
-0.07
-0.86
-0.96
-0.54
0.28
-0.47
.
-0.62
0.03
-0.40
.
.
-0.10
-0.18
-0.49
-0.08
0.02
-0.11
.
-0.09
Mean
Growth
Number of
Student s
24
22
0
0
0
99
22
18
26
17
0
25
Grade
.
.
4
5
.
7
4
5
4
5
.
4
2010-2011
Predicte d C
Score
Mean
Actual C
Score
Mean
-0.57
-0.42
-0.46
-0.40
-0.08
-0.55
.
.
.
-0.24
.
-0.63
-0.41
-0.86
-0.50
-0.40
-0.23
-0.88
.
.
.
-0.44
.
-0.67
0.16
-0.38
-0.03
0.00
-0.15
-0.39
.
.
.
-0.20
.
-0.05
Mean
Growth
Number of
Student s
11
22
0
0
0
63
26
21
0
22
20
27
Grade
.
.
4
5
.
7
4
5
.
5
4
4
2011-2012
Predicte d C
Score
Mean
Actual C
Score
Mean
-0.56
-0.54
-0.68
.
0.99
-0.71
.
.
.
-0.43
0.01
-0.36
-0.22
-0.57
-0.69
.
1.28
-0.71
.
.
.
-0.48
-0.02
-0.10
Mean
Growth
0.33
-0.06
-0.01
.
0.28
0.02
.
.
.
-0.05
-0.03
0.26
Reading EOG Scores and Growth

In general we see that the predicted C Score of ECU interns is worse than those classrooms without an intern. Likewise we see a higher rate of students predicted to have a 2 on the EOG Reading test.

• Another way to look at this data would be to pull off the values that have adequate (randomizable) controls and then do a quasi-randomized casecontrol analysis.
• Pulling off teachers with adequate controls from years 2009-2010 and 2010-2011 we can use each teachers first ECU intern experience (No TQP) and match with a similar grade and class size for that same year of a teacher with no ECU intern experience:

2009-2010
2009-2010
2009-2010
2010-2011
2010-2011
2010-2011
2010-2011
2010-2011
2009-2010
2009-2010
Year
Teacher
Code
C
F
P
A
J
B
G
K
D
E
Number of
Students
21
26
26
26
24
18
22
17
111
112
Grade
5
5
5
7
8
4
4
4
4
4
Predicted
C Score
Mean
-0.49
-0.43
0.25
-0.4
-0.08
-0.46
-0.55
-0.24
-0.43
-0.48
Actual C
Score
Mean
-0.82
-0.96
0.28
-0.4
-0.23
-0.5
-0.88
-0.44
-0.36
-0.54
Mean
Growth
Teacher
Code
-0.4 II
-0.49 X
0.02 KKK
0 MMM
-0.15 NNN
-0.03 BB
-0.39 EEE
-0.2 H
0.03 BBB
-0.08 WW
Number of
Students
23
25
25
26
22
18
25
24
105
102
Grade
5
5
5
7
8
4
4
4
4
4
Predicted
C Score
Mean
0.21
0.08
-0.33
-0.55
1.11
-0.28
-0.6
-0.67
-0.52
-0.61
Actual C
Score
Mean
0.16
-0.25
-0.6
-0.53
1.29
-0.66
-0.87
-0.9
-0.51
-0.8
Mean
Growth
-0.03
-0.31
-0.28
0.04
0.18
-0.45
-0.32
-0.24
0.05
-0.16
In 40% of matches, classes with ECU interns show higher growth.
Likewise 40% of pairs had classrooms with lower predicted C Score means.
Given that the mean predicted C Score in 90% of these instances are negative, this analysis is likely more appropriate for studying ECU intern impact.

• For research purposes, it seems appropriate to study not only growth but either raw or C Scores as well.
• The level may also provide useful ways to separate out important analysis.
• High growth is much more prevalent in the lower end of the spectrum with students who are predicted to score a 4 less likely to achieve the same amount of growth.
• ECU interns (pre-TQP) were seemingly sent to classrooms with lower predicted scores and as such any study that measures their impact has to be mindful to make adequate comparisons with ‘control’ groups.