2010 Outcome Evaluation Report:
Virginia Reading First
November 30, 2010
2010 Outcome Evaluation Report:
Virginia Reading First
Submitted to:
Virginia Department of Education
James Monroe Building
101 N. 14th Street
Richmond, VA 23219
804.786.3925
Submitted by:
Sonia Jurich, M.D., Ed.D., Project Director
Michael Frye, M.S., Research Associate
RMC Research Corporation
1501 Wilson Boulevard, Suite 1250
Arlington, VA 22209
Phone: 703.558.4000
Fax: 703.558.4823
TABLE OF CONTENTS
List of Tables .................................................................................................................................. ii
List of Figures ................................................................................................................................ iii
Executive Summary ....................................................................................................................... iv
Introduction ..................................................................................................................................... 1
2010 Assessment Results ................................................................................................................ 3
Stanford Reading First .................................................................................................................4
Aggregated Results (All Students) ...........................................................................................5
Disaggregated Results ..............................................................................................................9
Virginia Standards of Learning ..................................................................................................16
Analysis Over Time ...................................................................................................................... 19
Stanford Reading First ...............................................................................................................19
Virginia Standards of Learning ..................................................................................................23
Conclusions ................................................................................................................................... 28
Addressing the Research Questions ...........................................................................................28
Next Steps ..................................................................................................................................33
Virginia Reading First Outcome Evaluation – i
LIST OF TABLES
Table 1: Number of students participating in the 2010 Stanford Reading First by grade ........ 4
Table 2: Demographics of students participating in the 2010 Stanford Reading First ............. 5
Table 3: Percentage of students at grade level on the 2010 Stanford Reading First
disaggregated by grade and gender ......................................................................................... 10
Table 4: Percentage of students at grade level on the 2010 Stanford Reading First
disaggregated by grade and socioeconomic status.................................................................. 11
Table 5: Percentage of students at grade level in the 2010 Stanford Reading First
disaggregated by grade and race/ethnicity .............................................................................. 13
Table 6: Percentage of students at grade level in the 2010 Stanford Reading First by grade
and participation in special education ..................................................................................... 14
Table 7: Percentage of students at grade level in the 2010 Stanford Reading First by grade
and LEP classification............................................................................................................. 15
Table 8: Demographics of the Virginia Reading First students taking the SOL English/
reading assessment for grade three ......................................................................................... 17
Table 9: Results of the 2010 SOL English/reading assessment for grade three disaggregated
by student subgroups............................................................................................................... 18
Table 10: Effect Size for change in SOL English/reading assessment between 2004 and 2009
................................................................................................................................................. 25
Virginia Reading First Outcome Evaluation – ii
LIST OF FIGURES
Figure 1: Results of the 2010 Stanford Reading First – multiple choice subtest ...................... 6
Figure 2: Results of the 2010 Stanford Reading First – kindergarten ...................................... 6
Figure 3: Results of the 2010 Stanford Reading First – first grade .......................................... 7
Figure 4: Results of the 2010 Stanford Reading First – second grade...................................... 8
Figure 5: Percentage of students at grade level in the 2010 Stanford Reading First ................ 8
Figure 6: Percentile ranks on the 2010 Stanford Reading First – multiple choice subtest ....... 9
Figure 7: Percentile ranks for the multiple choice subtest of the 2010 Stanford Reading First
Assessment by grade and student subgroups .......................................................................... 16
Figure 8: Percentage of kindergarten students at grade level on the Stanford Reading First
(from 2004 to 2010) ................................................................................................................ 20
Figure 9: Percentage of first-grade students at grade level on the Stanford Reading First
(from 2004 to 2010) ................................................................................................................ 20
Figure 10: Percentage of second-grade students at grade level in the Stanford Reading First
(from 2004 to 2010) ................................................................................................................ 21
Figure 11: Stanford Reading First average NCE Scores for students in Cohort 1 schools
(between 2004 and 2009) ........................................................................................................ 22
Figure 12: Stanford Reading First average NCE Scores for students in Cohort 2 schools
(between 2007 and 2010) ........................................................................................................ 22
Figure 13: Percentage of third-grade students passing the SOL English/Reading ................. 23
Figure 14: Percentage of third-grade students passing the SOL English/Reading - Cohort 2
schools..................................................................................................................................... 24
Figure 15: Average scaled scores for the 2010 SOL English/reading assessment third grade
by years of attendance in Virginia Reading First Schools ...................................................... 26
Figure 16: Proficiency levels on the SOL English/reading assessment for grade three by years
in schools ................................................................................................................................ 27
Virginia Reading First Outcome Evaluation – iii
EXECUTIVE SUMMARY
Reading First is a program from the U.S. Department of Education, authorized under
the No Child Left Behind Act of 2001. The program provides grants to states to improve
reading instruction for students from kindergarten to grade three through the use of
scientifically-based reading programs. The Virginia Department of Education (VDOE)
received a Reading First grant to begin implementing the program with the 2003-2004
academic year. Grant funds were used to hire a reading coach at each participant school,
provide intensive professional development for the coaches and teachers, and purchase
scientifically-based reading programs and interventions. VDOE Reading First reading
specialists provided additional technical assistance, professional development, and
monitoring. In 2008, RMC Research Corporation competed for, and was awarded a contract
to conduct the evaluation of the program’s impact on student performance as measured by
standardized reading assessments – the Stanford Reading First (SRF) for students from
kindergarten to grade two, and the Virginia Standards of Learning (SOL), for students in
grade three.
This report reflects the analyses of student performance on the tests for the 2009-2010
academic year, the final year of the program. It also provides a longitudinal perspective of the
impact of Reading First on student academic performance from the 2003-2004 to 2009-2010
academic years. The report complements the two previous evaluation reports that RMC
provided to VDOE in October 2008 and November 2009.
Accomplishments of the Virginia Reading First program during the 2009-2010
academic year included:

A total of 21 schools in 17 school divisions served 5,339 students from kindergarten to
grade three. Of these students, 60 percent were classified as economically
disadvantaged (SIED), 47 percent were minorities, seven percent were students with
limited English proficiency (LEP), and nine percent were students with disabilities
(SWD).
Virginia Reading First Outcome Evaluation –iv

Students scoring at grade level in the Stanford Reading First (SRF), that is conducted
with students from kindergarten to grade two, included 73 percent of kindergarteners,
66 percent of first-grade students, and 65 percent of second-grade students.

73 percent of the students in grade three in the Virginia Reading First schools passed
the SOL English/Reading, with 44 percent scoring at the proficient level and 31
percent attaining advanced proficiency.

The percentage of students passing the SOL English/reading assessment disaggregated
by student subgroup was as follows:

Girls –78 percent

White – 81 percent

Boys – 71 percent

LEP – 56 percent

African Americans – 67 percent

SWD – 50 percent

Hispanics – 64 percent

SIED – 68 percent.
In addition, a longitudinal perspective across the seven years of the program highlights
areas of accomplishments and areas that require ongoing attention.
Celebrating success

Between spring 2004 and spring 2010, kindergarten results from the different
components of the SRF kindergarten assessment increased between 13 and 29 percent.
In the last two test years (2009 and 2010), Virginia Reading First kindergarten
students scored on average at the 70th percentile on the SRF.

Results on the SOL English/reading for students in grade three at Virginia Reading
First schools also showed improvements. In 2004, 58 percent of third graders in
Virginia Reading First schools passed the assessment, compared to 72 percent
statewide, for a difference of 14 percentage points. In 2010, 75 percent of the third
graders in Reading First schools passed the SOL English/reading assessment,
compared to 83 percent statewide, an eight percentage point difference.

Between spring 2004 and spring 2010, the percentage of students in third grade
passing the SOL English/reading assessment grew by 29 percent in Reading First
schools, compared to 15 percent statewide.

Improved academic achievement was observed for all student subgroups, and the
academic gap between minority and non-minority students declined, as gains in test
results for minority students outpaced the gains for non-minority students.
Virginia Reading First Outcome Evaluation –v

Considering the schools that benefitted from six years in the program (cohort 1
schools), the effect size of Reading First on results for the SOL English/reading
assessment was above 0.5 - a strong result for education programs that are not
implemented under controlled conditions.

The longer the students stayed in a Virginia Reading First school, the greater their
likelihood to score at the advanced level on the grade 3 SOL English/reading.
Looking forward

More attention should be given to the teaching of vocabulary development and reading
comprehension, the two components of the SRF that showed the weakest results
across all three grade levels and all student subgroups.

Achievement gaps on the grade 3 SOL English/reading decreased by 20 to 30 percent
for students from minority and SIED backgrounds between 2004 and 2009. Although
students from traditionally under-performing subgroups are showing improvement,
they still lag behind their peers and the effort to help these students should not decline.

Time enrolled in a Reading First school proved to have a direct positive relationship
on student performance in the SOL. However, one third of students in third grade in a
given year are new to the schools. The impact of mobility on student performance
poses a significant challenge for schools in high poverty areas in Virginia as well as
nationwide. Finding ways to minimize this effect is essential to sustain and expand
the progress made by Virginia Reading First on the reading skills of Virginia students.
In summary, Virginia Reading First schools showed strong improvements throughout
the grant years. For these schools and school divisions, the challenge will be to maintain and
expand these gains as grant funds expire. For the remaining schools, the challenge is to attain
the success of the Reading First schools. At this time, the Virginia Department of Education,
as many other state agencies across the nation, is actively involved in the development of a
comprehensive literacy plan from birth to grade twelve. This is an important time to revisit
the findings from the Reading First program, and re-examine these findings from the
perspective of new advances in reading research. This merging of evidence derived from
practice and research findings creates the perfect foundation for a plan that will ensure that all
students graduate from Virginia schools with strong literacy skills.
Virginia Reading First Outcome Evaluation –vi
INTRODUCTION
Reading First is a program from the U.S. Department of Education (USED) authorized
by Title I, Part B, Subpart 1 of the Elementary and Secondary Education Act, as amended by
the No Child Left Behind Act of 2001, Public Law 107-110. The objective of Reading First is
to promote the use of scientifically-based reading programs and practices, instructional tools
and assessments that are supported by research, with the ultimate goal of ensuring “that all
children learn to read well by the end of third grade”.1
The Virginia Department of Education (VDOE) received a Reading First grant that
spanned seven years, from 2003-2004 to 2009-2010 academic year. During this period, the
Virginia Reading First program served more than 90 schools in 43 school divisions statewide.
A total of 132,038 students participated in the program. Seventy percent of the Virginia
Reading First schools were located in small towns or rural areas, 20 percent were located in
middle-sized towns (population less than 250,000), and 10 percent in suburban (urban fringe)
areas.2
The evaluation of the Virginia Reading First program was initially conducted by the
University of Virginia. In fall 2007, VDOE issued a Request for Proposals for the evaluation
of the program’s impact on student academic achievement. Two assessments were used to
measure student performance in reading for students from kindergarten to second grade: the
Stanford Reading First (SRF), and the Virginia Standards of Learning (SOL) English/reading
assessment for students in grade three. RMC Research Corporation competed for and was
awarded the contract.
This is the last of three reports that RMC prepared for VDOE on the evaluation of
Virginia Reading First. The first report presented findings from the analysis of the reading
assessments conducted during spring 2008. The second report summarized findings from the
2009 assessment and discussed changes in assessment results from spring 2004 to spring
2009. This report includes an analysis of the reading assessments conducted in spring 2010
1
U.S. Department of Education, Office of Communications and Outreach, Guide to U.S. Department of Education
Programs, Washington, D.C., 2009, p. 171.
2
Data retrieved on September 29, 2010, from http://readingfirstdataonline.org/awards.aspx
Virginia Reading First Outcome Evaluation – 1
and revisits the longitudinal analysis presented in the 2009 report to integrate the most recent
findings. The report comprises three sections:

Section 1, 2010 Assessments, presents the results from the assessments conducted in
spring 2010 with students from kindergarten through third grade.

Section 2, Longitudinal Perspective, provides a descriptive analysis of changes in
academic performance for Virginia Reading First students across the grant years.

Section 3, Conclusions, addresses the evaluation questions and discusses next steps.
The evaluators’ objective was to write a report that provides state, school division, and
school staff with useful evidence regarding the impact of the Virginia Reading First program
on student performance on standardized reading assessments. Graphics are used frequently to
facilitate the explanation of findings. Results from the statistical tests are mentioned in the
body of the report and included only when essential to explain a statement. Footnotes are
used for further clarification or information on data sources without interrupting the flow of
the narrative. The evaluators hope that the information in this report will be helpful to inform
decisions on how to sustain the program gains and improve on its shortcomings as the
Reading First students move to upper grades and new students enter Virginia schools eager to
learn and succeed.
Virginia Reading First Outcome Evaluation – 2
2010 ASSESSMENT RESULTS
In the 2009-2010 academic year, 21 schools participated in the Virginia Reading First
Program. This section provides information about results for the reading assessments that
these schools conducted in spring 2010. Virginia Reading First schools used two assessments
to test the reading skills of its students: the Stanford Reading First (SRF) and the Virginia SOL
English/reading assessment.
Stanford Reading First: The SRF is a norm-referenced test published by Pearson.
The Virginia Reading First program used the test for students in kindergarten through second
grade. The test comprises two subtests: multiple choice and oral fluency. The multiple choice
subtest includes five components: phonemic awareness, phonics, vocabulary development,
reading fluency, and reading comprehension. The oral fluency subtest comprises two
components: speaking vocabulary and oral reading fluency. In addition to the multiple choice
subtest, the students also responded to the speaking vocabulary component of the oral fluency
subtest. The oral reading fluency was voluntary in Virginia and most schools did not use the
test. Results for the multiple choice subtest are provided in scaled scores, normal curve
equivalent, stanines, and percentiles. Based on the percentiles, three proficiency levels are
determined:

At Grade Level: students who score at or above the 40th percentile,

Needs Additional Intervention: students who score between the 20th and the 40th
percentiles, and

Needs Substantial Intervention: students who score below the 20th percentile.
Standards of Learning: The SOL English/reading assessment is a criterionreferenced achievement test that is conducted with students from grade three to high school.
Results are provided in scaled scores and proficiency levels determined by the number of
correct items. For the third-grade SOL English/reading assessment, the proficiency levels are
determined as follows:

Fail/Basic: 13 correct items out of 35 items;

Pass/Proficient: 23 correct items out of 35 items; and

Pass/Advanced: 31 correct items out of 35 items.
Virginia Reading First Outcome Evaluation – 3
This section is divided into two subsections. The first subsection discusses the results
for the SRF tests, and the second subsection presents the SOL English/reading assessment
results.
Stanford Reading First
Description of the population
The 21 schools receiving Reading First grants in the 2009-2010 academic year served
a total of 4,059 students in kindergarten through grade two. Of these 4,059 students, 4,007
(98.7 percent) took the Stanford Reading First (SRF) multiple choice test and the speaking
vocabulary part of the oral fluency test. Because the percentage of students who did not take
the SRF was about one percent of student enrollment, an analysis of absentees’ demographics
was not needed. Table 1 displays the number of students who took the SRF assessment in
spring 2010 disaggregated by grade. Table 2 disaggregates the group of test-takers by grade
and subgroups.
As seen in the tables, the students taking the SRF in spring of 2010 were equally
distributed by grade and had similar gender representation (about half males and half
females). Regarding race/ethnicity, the majority of test-takers were white (52 percent),
followed by students from African American (35 percent) and Hispanic (10 percent) origins.
Students classified as Asian and “other” comprised about 3 percent of the test-taking
population. The majority (59 percent) of the students were identified as economically
disadvantaged (SIED); 7 percent were classified as having Limited English Proficiency
(LEP), and 8 percent were students with disabilities (SWD).
Table 1: Number of students participating in the 2010 Stanford Reading First by grade
Grades (N)
Total
Categories
K
First
Second
Number
Percent
Total students per grade
1,330
1,369
1,360
4,059
100.0
Students taking the 2010 SRF
1,310
1,362
1,335
4,007
98.7
Virginia Reading First Outcome Evaluation – 4
Table 2: Demographics of students participating in the 2010 Stanford Reading First
Grades (N)
Total
Subgroups
K
First
Second
Number
Percent
Female
664
677
656
1,997
49.8
Male
646
685
679
2,010
50.2
Asian
10
13
8
31
0.8
African American
433
462
499
1,394
34.8
Hispanic
123
130
140
393
9.8
White
705
727
663
2,095
52.3
Other
39
30
25
94
2.3
762
795
823
2,380
59.4
Students with disabilities (SWD)
76
118
140
334
8.3
Students with Limited English
Proficiency (LEP)
69
102
109
280
7.0
1,310
1,362
1,335
4,007
Gender
Race/
Ethnicity
Students identified as economically
disadvantaged (SIED)
Total test-takers
100.0
Aggregated Results (All Students)
Figure 1 displays the percentages of students who scored at the different performance
levels in the multiple choice part of the SRF. The multiple choice subtest comprises five
components: phonemic awareness, phonics, vocabulary development, reading fluency, and
reading comprehension. As seen in the chart, the majority of students scored at grade level
for all three grades tested with SRF. The percentage in need of intervention (additional or
substantial) was greater for students in grades one and two when compared to kindergarten
students.
Virginia Reading First Outcome Evaluation – 5
Figure 1: Results of the 2010 Stanford Reading First – multiple choice subtest
100%
8%
15%
20%
21%
66%
65%
Grade 1 (N=1,362)
Grade 2 (N=1,335)
19%
80%
Students (%)
14%
60%
40%
73%
20%
0%
Kindergarten (N=1,310)
At Grade Level
Needs Additional Intervention
Needs Substantial Intervention
Figure 2 breaks down the components of the multiple choice test for kindergarten
students, and includes results for the speaking vocabulary test, which is part of the oral
fluency component. As shown in the graphic, the majority of students were at grade level for
phonemic awareness (88 percent), phonics (76 percent), reading fluency (67 percent), and
speaking vocabulary (64 percent). About half of the kindergarteners were at grade level for
reading comprehension, while fewer than half (46 percent) were at grade level for vocabulary
development. It must be noted that more than 20 percent of the kindergarten students are in
need of substantial intervention for reading fluency and 40 percent are in need of additional
intervention for vocabulary development.
Figure 2: Results of the 2010 Stanford Reading First – kindergarten
100%
8%
14%
40%
11%
40%
60%
88%
13%
23%
14%
80%
Students (%)
10%
76%
37%
14%
64%
67%
51%
46%
20%
18%
0%
Phonemic
Awareness
At Grade Level
Phonics
Vocabulary
Development
Reading Fluency
Needs Additional Intervention
Reading
Comprehension
Speaking
Vocabulary
Needs Substantial Intervention
Virginia Reading First Outcome Evaluation – 6
Figure 3 displays assessment results for students in first grade. As explained in the
2009 evaluation report, the weak results in the phonics test have been attributed to the lack of
alignment between the content of the test and the Virginia English Standards of Learning,
rather than a lack of phonics instruction within the schools. For the other components of the
SRF, 60 percent or more of the first-grade students are scoring at grade level. It must be
noted the percentages of first-grade students in need of substantial intervention in vocabulary
development (21 percent) and reading fluency (19 percent).
Figure 3: Results of the 2010 Stanford Reading First – first grade
Students (%)
100%
6%
11%
25%
19%
17%
20%
61%
61%
Vocabulary
Development
Reading Fluency
15%
80%
60%
40%
9%
12%
21%
29%
38%
82%
20%
73%
62%
37%
0%
Phonemic
Awareness
Phonics
At Grade Level
Needs Additional Intervention
Reading
Comprehension
Speaking
Vocabulary
Needs Substantial Intervention
Figure 4 displays results for students in second grade. Except for phonemic
awareness, where almost all students were at grade level, results for the other components
were not as strong as for students in kindergarten and grade one. A little more than half of the
students were at grade level in vocabulary development (54 percent), reading fluency (55
percent) and speaking vocabulary (55 percent). Additionally, the percentage of students in
need of substantial intervention is high, and varies from 21 percent, in the speaking
vocabulary component, to 46 percent in phonics.
Virginia Reading First Outcome Evaluation – 7
Figure 4: Results of the 2010 Stanford Reading First – second grade
Students (%)
100%
80%
28%
26%
18%
19%
54%
55%
Vocabulary
Development
Reading Fluency
46%
60%
21%
31%
24%
23%
11%
96%
40%
43%
20%
55%
46%
0%
Phonemic
Awareness
Phonics
At Grade Level
Needs Additional Intervention
Reading
Comprehension
Speaking
Vocabulary
Needs Substantial Intervention
Figure 5 compares results across grades, focusing on the percentage of students who
were classified at grade level in each component of the test.
Figure 5: Percentage of students at grade level in the 2010 Stanford Reading First
64%
62%
Speaking Vocabulary
55%
51%
Reading Comprehension
73%
46%
67%
Reading Fluency
61%
55%
46%
Vocabulary Development
61%
54%
76%
Phonics
37%
43%
88%
Phonemic Awareness
82%
96%
0%
20%
Kindergarten
40%
Grade 1
60%
80%
100%
Grade 2
As suggested by the graphic, 82 percent or more of Virginia Reading First students,
regardless of grade, were at grade level on the phonemic awareness component of the test.
Virginia Reading First Outcome Evaluation – 8
Results are particularly strong for students in second grade. First-grade students had strong
results in reading comprehension (73 percent, compared to 51 percent for kindergarten and 46
percent for second-grade students) and vocabulary development (61 percent, compared to 54
percent for kindergarteners and 46 percent for second-grade students). Students at grade level
in phonics comprised the majority (76 percent) of kindergarteners, but fewer than half of the
students in first and second grade.
The mean scaled scores in the multiple choice section of the SRF were averaged and
converted into percentiles to facilitate a comparison between results for Virginia Reading
First students, and those who took the test nationwide. As shown in Figure 6, kindergarten
students in Virginia Reading First schools scored, on average, at or above 72 percent of testtakers nationwide at the same grade level. Virginia Reading First first-grade students, on
average, scored at or above 58 percent of their peers nationwide, while second-grade students
scored about midway on the national sample (53th percentile). As a reminder, the 50th
percentile is the expected average if the results follow a normal distribution. Therefore, the
results in the multiple choice sub-test of the SRF suggest that Virginia Reading First
kindergarteners are doing, on average, better in the test than peers nationwide, while students
in first and second grades are doing about the same, on average, as their peers across the
nation. These percentile ranks were similar to those found in the 2009 tests (73rd, 58th, and
54th percentiles, respectively).
Figure 6: Percentile ranks on the 2010 Stanford Reading First – multiple choice subtest
72nd
58th
53th
Kindergarten
Grade 1
Grade 2
Disaggregated Results
Analyses of the 2010 SRF results were conducted with three student subgroups that
have greater representation among Virginia Reading First students: gender, economically
disadvantaged students, and minority students. The numbers of students with disabilities and
those classified as having Limited English Proficiency were too small to ensure reliable
results.
Virginia Reading First Outcome Evaluation – 9
Gender: A one-way analysis of variance (ANOVA) was conducted to examine the
mean scores for boys and girls in the components of the SRF.3 The results of the test indicate
that in kindergarten, girls and boys perform at similar levels in the SRF assessments for all
components, except reading comprehension. For this component, girls outperformed boys.
At first grade, girls outperformed boys in four of the six components, including phonemic
awareness, reading fluency, reading comprehension, and speaking vocabulary. At second
grade, girls outperformed boys in all components, except vocabulary development. Results in
the vocabulary development component of the SRF were similar for boys and girls in all three
grade levels. Table 3 displays the percentage of girls and boys who scored at grade level in
the different components of the SRF by grades. A brief examination of the table confirms the
results of the comparison of means test. In general, more girls than boys scored at grade
levels in all the components of the test and at all grade levels.
Table 3: Percentage of students at grade level on the 2010 Stanford Reading First
disaggregated by grade and gender
52.4
65.8
664
Male
88.4
74.1
44.4
66.7
48.5
62.4
646
Total
(N)
Speaking
Vocabulary
66.3
Reading
Fluency
48.8
Vocabulary
Development
77.6
Phonics
87.8
Phonemic
Awareness
Female
Subgroup
Reading
Comprehension
Percentage At Grade Level
Kindergarten
First Grade
Female
83.9
37.5
61.6
63.8
77.0
62.2
677
Male
80.9
36.5
61.2
57.7
69.3
58.5
685
Second Grade
Female
97.1
47.9
55.6
58.1
48.5
60.4
656
Male
95.3
39.0
52.3
52.7
44.2
49.8
679
3
All comparison of means tests for results on the six components of SRF used raw scores, as scale scores are
provided only for the multiple choice subtest. Statistical significance was established at α= 0.05. The term
“outperforms” indicates differences that are statistically significant between the groups being compared.
Virginia Reading First Outcome Evaluation – 10
Students identified as economically disadvantaged: The comparison of means
analysis found that the students who were not economically disadvantaged (Not-SIED)
students outperformed their SIED peers in all components of the test and at all grade levels.
Results were statistically significant even with α = 0.01.4 Table 4 presents the percentage of
students who scored at grade level disaggregated by socioeconomic status. Two groups were
included: students identified as economically disadvantaged (SIED) and students who were
not economically disadvantaged (Not SIED).
Table 4: Percentage of students at grade level on the 2010 Stanford Reading First
disaggregated by grade and socioeconomic status
Speaking
Vocabulary
36.0
58.5
42.5
58.4
762
94.0
83.9
60.6
77.6
61.5
72.1
548
Vocabulary
Development
70.1
Phonics
83.9
Phonemic
Awareness
Reading
Comprehension
Reading Fluency
Percentage At Grade Level
Total
(N)
Kindergarten
SIED
Not
SIED
First Grade
SIED
Not
SIED
77.5
30.1
52.8
54.1
67.4
57.2
795
89.2
46.7
73.4
70.0
81.1
69.5
567
Second Grade
SIED
Not
SIED
94.8
36.8
46.3
48.2
38.2
50.1
823
98.4
53.9
66.2
66.8
59.4
62.9
512
Race/ethnicity: A comparison of means analysis was conducted using an
independent sample t-test to examine the differences in mean scores across the race/ethnic
subgroup. Only three categories had sufficient student numbers for a reliable comparison:
4
A lower alpha indicates a greater probability that the same results will be obtained no matter how many times
the test is repeated; in non-statistical terms, the lower alpha indicates that the gap in performance between the
two groups is real and requires attention.
Virginia Reading First Outcome Evaluation – 11
whites, African Americans, and Hispanics. Results of the test indicated that mean scores for
students identified as white were significantly higher than mean scores for African American
students across all components and grades, except for phonemic awareness and reading
fluency at kindergarten, and speaking vocabulary at first grade. In these three cases, mean
scores for the African American students were similar to mean scores for white students. The
comparison between scores for Hispanics and white students showed that mean scores for
white students tended to be significantly higher than mean scores for Hispanic students,
except for phonemic awareness in first and second grades. For these two components, the
average scores for Hispanic students were similar to the average scores for white students.
Table 5 displays the percentages of students scoring at grade level disaggregated by
grade and race/ethnicity. The percentages in the table indicate how many students from a
specific race/ethnicity scored at grade level compared to the total enrollment in grade for that
same race/ethnicity. For instance, 86 percent of the African American students in
kindergarten scored at grade level in phonemic awareness; 85 percent of all Hispanic students
in first grade scored at grade level in phonemic awareness, and so on.
Virginia Reading First Outcome Evaluation – 12
Table 5: Percentage of students at grade level in the 2010 Stanford Reading First
disaggregated by grade and race/ethnicity
Speaking
Vocabulary
90.0
60.6
100.0
80.8
70.0
10
86.1
71.6
39.0
64.4
47.3
60.7
433
78.0
72.4
24.4
58.5
29.3
50.4
123
White
91.9
79.3
55.2
69.5
56.7
69.2
705
Other
74.4
69.2
30.8
51.3
30.8
51.3
39
Reading
Fluency
Vocabulary
Development
80.0
Phonics
Asian
African
American
Hispanic
Phonemic
Awareness
Reading
Comprehension
Percentage At Grade Level
Total
(N)
Kindergarten
First Grade
Asian
African
American
Hispanic
92.3
69.2
76.9
69.2
84.6
61.5
13
76.4
27.7
49.4
53.7
67.6
59.7
462
85.4
33.1
46.2
56.2
63.1
53.8
130
White
85.4
43.3
71.3
65.7
77.6
65.3
727
Other
83.3
33.3
66.7
63.3
86.7
66.7
30
Second Grade
Asian
African
American
Hispanic
100.0
50.0
75.0
62.5
62.5
100.0
8
95.4
33.5
46.5
44.3
33.9
47.3
499
95.7
39.3
21.4
44.3
37.9
50.7
140
White
97.3
51.3
66.1
66.1
57.2
62.0
663
Other
84.0
52.0
56.0
52.0
48.0
32.0
25
Students with disabilities: Table 6 displays the percentage of students with
disabilities (SWD) and students who do not receive special education services (Not SWD).
As seen in the table, SWD students are less likely to score at grade level, when compared to
Not SWD students. As a group, the SWD students seem to struggle particularly in reading
comprehension and speaking vocabulary beginning in kindergarten, and their difficulties in
these two areas appear to remain as they move to upper grades.
Virginia Reading First Outcome Evaluation – 13
Table 6: Percentage of students at grade level in the 2010 Stanford Reading First by grade
and participation in special education
Speaking
Vocabulary
64.5
32.9
61.8
28.9
48.7
76
Not SWD
88.8
76.6
47.1
66.8
51.8
65.1
1,234
Vocabulary
Development
76.3
Phonics
SWD
Phonemic
Awareness
Reading
Comprehension
Reading Fluency
Percentage At Grade Level
Total
(N)
Kindergarten
First Grade
SWD
58.5
20.3
35.6
39.0
55.9
39.0
118
Not SWD
84.6
38.6
63.8
62.8
74.8
64.5
1,244
Second Grade
SWD
88.6
23.6
45.7
42.1
30.0
43.6
140
Not SWD
97.1
45.7
54.9
56.9
48.2
56.3
1,195
When analyzing results for students with disabilities, one must take into account the
fact that these students differ considerably in the type and intensity of their disabilities, and
how the disability affects their cognitive abilities. Comparisons between such a diverse group
and another group that is more homogenous are not reliable or helpful. For this reason, a
comparison of means test was not conducted.
Limited English Proficient Students: Table 7 displays the percentages of students
who scored at grade level in the SRF test disaggregated in two subgroups: students with
limited English proficiency (LEP) and students proficient in English (Not LEP). As the table
shows, LEP and Not LEP did well in the phonemic awareness test at all three grade levels. In
all other components, LEP students were less likely to score at grade level when compared to
their Not LEP peers. Vocabulary development was the component of the SRF where LEP
students had the greatest difficulty in all three grade levels, but particularly in the second
grade. Caution must be exercised when interpreting results for LEP students. First, the small
number of these students does not allow for reliable comparisons with results from Not LEP
Virginia Reading First Outcome Evaluation – 14
students. Second, students under this classification represent a range of levels of proficiency
in the English language.
Table 7: Percentage of students at grade level in the 2010 Stanford Reading First by grade
and LEP classification
Speaking
Vocabulary
23.2
58.0
29.0
37.7
69
Not LEP
89.3
76.5
47.7
67.1
51.6
65.9
1,241
Reading
Fluency
66.7
Vocabulary
Development
75.4
Phonics
LEP
Phonemic
Awareness
Reading
Comprehension
Percentage At Grade Level
Total (N)
Kindergarten
First Grade
LEP
82.4
32.4
40.2
52.9
60.8
48.0
102
Not LEP
82.4
37.7
63.7
61.6
74.2
63.9
1,260
Second Grade
LEP
94.5
31.2
18.3
34.9
29.4
48.6
109
Not LEP
96.3
44.8
57.7
57.6
48.2
55.9
1,226
Summary: Figure 7 displays the results of percentile ranks [converted from the mean
scaled score] for the multiple choice test by grade and student subgroups. As seen in the
table, kindergarten students scored above the 50th percentile on the multiple choice subtest for
all student subgroups. For first-grade students, only students receiving special education
services (SWD) scored below the 50th percentile. For second-grade students, three subgroups
scored at or below the 40th percentile: Hispanics (40th percentile), LEP (33rd percentile), and
SWD (37th percentile).
Virginia Reading First Outcome Evaluation – 15
Figure 7: Percentile ranks for the multiple choice subtest of the 2010 Stanford Reading
First Assessment by grade and student subgroups
100
90
80
Percentile
70
60
50
40
30
20
10
0
Kindergarten
Grade 1
Grade 2
Note: The dotted line indicates the 40th percentile – the cut score for “at grade level.”
Virginia Standards of Learning
In the 2009-2010 academic year, Virginia Reading First schools served 1,346 thirdgrade students. Of these, 1,332 (99 percent) took the Standards of Learning (SOL)
English/reading assessment in spring 2010. Table 8 displays the demographics of the students
who participated in the test in spring 2010.
Virginia Reading First Outcome Evaluation – 16
Table 8: Demographics of the Virginia Reading First students taking the SOL English/
reading assessment for grade three
Subgroups
Number
Percentage
Female
680
51.1
Male
652
48.9
African American
462
34.7
13
1.0
Hispanic
114
8.6
White
726
54.5
Other
17
1.3
LEP
105
7.9
SWD
140
10.5
SIED
807
60.6
1,332
100.0
Gender
Asian
Race/
ethnicity
Total (all test takers)
Of the 1,332 third-grade students who took the 2010 Standards of Learning (SOL)
English/reading assessment, 992 (75 percent) passed the test, with 44 percent demonstrating
proficiency and 31 percent demonstrating advanced proficiency. Statewide, 83 percent of the
students in third grade passed the Standards of Learning (SOL) English/reading assessment,
with 43 percent demonstrating proficiency, and 41 percent demonstrating advanced
proficiency.
Table 9 displays the assessment results disaggregated by student demographics. As
seen in the table, half or more of the students in all subgroups passed the test. The majority of
students who failed the test scored at the basic level, with fewer than 10 percent scoring below
basic, except for students with disabilities (13 percent below basic). It is of note that more
than one third of the girls and white students attained advanced proficiency. Students
identified as economically disadvantaged did equally well, with close to 70 percent passing
the test.
Virginia Reading First Outcome Evaluation – 17
Table 9: Results of the 2010 SOL English/reading assessment for grade three disaggregated
by student subgroups
Percentages1
Pass
Subgroups
Gender
Race/
Ethnicity2
LEP
SWD
SIED
Female
Male
Black
Hispanic
White
Adv.
Prof.
Total
Basic
34.7
27.0
19.7
21.9
39.4
19.0
14.3
23.3
42.8
44.3
47.0
42.1
41.6
37.1
35.7
45.0
77.5
71.3
66.7
64.0
81.0
56.1
50.0
68.3
20.3
22.9
27.9
27.2
16.9
34.3
37.1
26.5
Fail
Below
Basic
2.2
5.8
5.4
8.8
2.1
9.5
12.9
5.2
Total
Total
(N)
22.5
28.7
33.3
36.0
19.0
32.8
50.0
31.7
680
652
462
114
726
105
140
807
1 Percentages
calculated as the number of students within subgroups at a specific performance level relative to the total
number of students in that same subgroup (e.g., number of girls scoring advanced divided by total number of girls).
2 Information is provided for groups with 30 or more students.
Virginia Reading First Outcome Evaluation – 18
ANALYSIS OVER TIME
This section is divided into two subsections. The first focuses on the impact of
Reading First on students in grades K-2, as measured by the Stanford Reading First (SRF)
assessments. The second focuses on the impact of the program on students in grade three, as
measured by the Virginia Standards of Learning English/Reading assessment. An explanation
about the types of analyses used and their limitations is included in the 2009 evaluation
report. The 2009 report also included an analysis of the association between scores on SRF
and SOL. That analysis found a weak relationship between the two tests. A new analysis was
done this year with similar results. Since no new information can be presented, the analysis
was omitted in this report.
Stanford Reading First
Figures 8 to 10 display the percentages of Virginia Reading First students at grade
level in the different components of the SRF across the grant years. The numbers in
parentheses at the horizontal axis represent the number of students who took the SRF, while
the vertical axis reflects the percentage of students who scored at grade level in the different
components of the test. Only the five components that comprise the multiple choice subtest
are included.
Figure 8 focuses on students in kindergarten. Kindergarten students showed steady
progress in all components of the test across the years, with the greatest increase occurring in
the first year. Between 2004 and 2010, the percentage of kindergarten students at grade level
in the SRF assessment increased by 13 percent in phonemic awareness, 22 percent in phonics,
24 percent in vocabulary development, 19 percent in reading fluency, and 29 percent in
reading comprehension. Vocabulary development and reading comprehension, which were
the components with the lowest percentages of students at grade level in all of the years,
showed the greatest improvements. The phonemic awareness component had the strongest
results, with 80 percent or more of the students scoring at grade level across the years.
Virginia Reading First Outcome Evaluation – 19
Figure 8: Percentage of kindergarten students at grade level on the Stanford Reading First
(from 2004 to 2010)
100%
80%
60%
40%
20%
0%
2004 (4,398) 2005 (3,471) 2006 (4,436) 2007 (5,704) 2008 (5,727) 2009 (5,200) 2010 (1,310)
Phonemic awareness
Reading fluency
Phonics
Reading comprehension
Vocabulary development
Figure 9 displays the percentage of students in first grade who scored at grade level on
the SRF between 2004 and 2010. Growth in test performance for first-grade students was
negligible, except for phonics. Between 2004 and 2010, the percentage of first-grade students
who scored at grade level on the SRF test increased by two percent in vocabulary
development, three percent in phonemic awareness, four percent in reading comprehension,
six percent in reading fluency, but 25 percent in phonics.
Figure 9: Percentage of first-grade students at grade level on the Stanford Reading First
(from 2004 to 2010)
100%
80%
60%
40%
20%
0%
2004 (4,150) 2005 (3,852) 2006 (4,562) 2007 (5,723) 2008 (5,719) 2009 (5,303) 2010 (1,362)
Phonemic awareness
Reading fluency
Phonics
Reading comprehension
Vocabulary development
Virginia Reading First Outcome Evaluation – 20
Figure 10 displays the percentage of second-grade students who scored at grade level
on the SRF. It is of note the difference in scores for the phonemic awareness component,
when compared to the other components. While 90 percent or more of the students scored at
grade level in the phonemics awareness component across the years, about 60 percent or
fewer scored at grade level on the other components. Between 2004 and 2010, growth was
nonexistent or minimal for reading fluency (no growth), phonemic awareness (three percent),
and vocabulary development (six percent). At reading comprehension, the scores showed a
growth of nine percent between the beginning and end of the Virginia Reading First program.
Although phonics was the component with the weakest performance, it showed the greatest
growth (24 percent).
Figure 10: Percentage of second-grade students at grade level in the Stanford Reading
First (from 2004 to 2010)
100%
80%
60%
40%
20%
0%
2004 (4,294) 2005 (3,214) 2006 (4,411) 2007 (5,552) 2008 (5,558) 2009 (5,375) 2010 (1,335)
Phonemic awareness
Reading fluency
Phonics
Reading comprehension
Vocabulary development
Figure 11 displays the average normal curve equivalent (NCE) scores for students in
cohort 1 schools. Cohort 1 comprised 63 schools that implemented Reading First from 2004
to 2009 (their last year in the program). The analysis used the average NCE scores for the
multiple choice section of the SRF. The NCE is an equal-interval scale with scores dispersed
on a normal distribution curve (mean=50; standard deviation = 21.06). Students who score 50
NCEs are considered at grade level. A change of seven NCEs corresponds approximately to
one grade level. Scores for the 2004 SRF multiple choice subtest did not include NCE and
therefore, the analysis starts with the 2005 results. As seen in the graphic, increases in scores
were small. Between 2005 and 2009, mean scores increased from 56 to 60 NCEs for students
Virginia Reading First Outcome Evaluation – 21
in kindergarten; from 52 to 54 NCEs for students in first grade; and no increase for students in
grade two.
Figure 11: Stanford Reading First average NCE Scores for students in cohort 1 schools
(between 2004 and 2009)
80
70
NCE
60
50
40
30
20
2005
2006
2007
Kindergarten
2008
Grade 1
2009
Grade 2
A similar analysis was conducted with cohort 2 schools. Cohort 2 includes 19 schools
that implemented the program from 2007 to 2010. (Two additional schools entered the
program in 2008-2009 and are not included in the analysis depicted in Figure 12, below.) As
displayed in Figure 12, between the beginning (2007) and end (2010) of the Reading First
program for cohort 2 schools, NCE scores increased five points on average for students in
kindergarten, from 55 to 60 NCEs, from 51 to 53 NCEs for students in first grade, and no
change for students in second grade.
Figure 12: Stanford Reading First average NCE Scores for students in cohort 2 schools
(between 2007 and 2010)
80
70
NCE
60
50
40
30
20
2007
2008
Kindergarten
2009
Grade 1
2010
Grade 2
Virginia Reading First Outcome Evaluation – 22
Virginia Standards of Learning
Gains across years: Figure 13 shows results for the third-grade Standards of Learning
(SOL) English/reading assessment for cohort 1 of Virginia Reading First schools and
statewide schools.5 Increases in the percentage of third-grade students who passed this
assessment were particularly large between the first and second grant years. In 2004, 58
percent of the students in the cohort 1 schools passed the test, compared to 70 percent in 2005.
Between 2004 and 2009, the last grant year for cohort 1 schools, the percentage of third-grade
students who passed the SOL reading assessment increased by 43 percent in Reading First
schools, compared to 19 percent statewide.
Figure 13: Percentage of third-grade students passing the SOL English/Reading
assessment
Passing+Advanced
100%
80%
60%
77% 78%
72%
71% 70%
80%
76%
84% 81%
86% 83%
2008
2009
58%
40%
20%
0%
2004
2005
State
2006
2007
Cohort 1-Reading First
Figure 14 displays the same information for cohort 2 schools. Cohort 2 includes the
19 schools that received Reading First grants in 2007 and remained in the program until 2010.
(Two additional schools entered the program in 2008-2009 and are not included in the
analysis depicted in Figure 14 below.) Gains for cohort 2 schools were small and comparable
to statewide gains. It is important to observe that the cohort 2 schools had no more than four
5
Statewide data retrieved October 1, 2010 from http://www.doe.virginia.gov/statistics_reports/
school_report_card/index.shtml
Virginia Reading First Outcome Evaluation – 23
years in the program. The different results for the two cohorts highlight the influence of time
in program on student academic performance.
Figure 14: Percentage of third-grade students passing the SOL English/Reading
assessment - Cohort 2 schools
100%
Passing+Advanced
80%
80%
84%
75%
81%
86%
85%
83%
77%
60%
40%
20%
0%
2007
2008
State
2009
2010
VRF
Effect size: Although the Virginia Reading First schools fell short of the level of
statewide performance, the data suggest strong improvements in the schools that received full
program implementation (cohort 1 schools). To assess the magnitude of this improvement
over the years, an effect size was calculated. Effect size is a statistical concept that measures
the strength of the relationship between two variables during the period of treatment. In this
case, the variables are student performance on the Standards of Learning (SOL) English/
reading assessment in the treatment (Virginia Reading First schools) and comparison
(statewide) groups. To measure the effect size, a Cohen’s h using transformed population
proportions was calculated as follows:
ES = 2*arcsin(√𝑝1) - 2*arcsin(√𝑝2 )
where 𝑝1is the proportion of students proficient on the Standards of Learning (SOL) English/
reading assessment in the last grant year (2009), and 𝑝2 is the proportion of students in first
year of Reading First program (2004)]. Results are displayed in Table 10.
Virginia Reading First Outcome Evaluation – 24
Table 10: Effect Size for change in SOL English/reading assessment between 2004 and
2009
Schools
Proportion
Difference
Effect Size
Effect
Virginia Reading First – Cohort 1
43%
0.561
Medium
State
19%
0.349
Small
The effect size for change on the third grade SOL reading assessment is higher for the
Virginia Reading First schools (ES = 0.561) than schools statewide (ES = 0.349). The effect
size of change on the SOL in the Virginia Reading First schools is interpreted as medium,
while change at the state level is interpreted as small. It is of note that effect sizes in
situations where an effective treatment is not administered in a controlled setting tend to be
small. Therefore, the schools that participated for six years in the Virginia Reading First
program demonstrated considerable improvement in SOL results.
Time in program (students): Students who took the Standards of Learning (SOL)
English/reading assessment as third graders in 2010 were tracked back to their first year in
Virginia Reading First schools. Each third-grade student who took the 2010 test fit in one of
four categories: (1) Students who were in the school for four years and took the SRF in spring
2007 as kindergarteners (591 students); (2) Students who were in the school for three years
and took the SRF for the first time as first graders in spring 2008 (143 students); (3) Students
who were in the school for two years and took the SRF for the first time as second graders in
spring 2009 (198 students); and (4) Students who entered the school in grade three in 2010
(400 students). Students who were retained in a grade or those who exited the school and
might have returned in a later year were not included in the analysis.
Figure 15 displays the average scaled score for the 2010 SOL English/reading
assessment for each of the four groups. As seen in the graphic, the mean scaled score
increased as the number of years in a Virginia Reading First school increased. The average
SOL scaled score for students who attended a Virginia Reading First school for four years
was significantly higher than students who had attended for three or fewer years.
Virginia Reading First Outcome Evaluation – 25
Figure 15: Average scaled scores for the 2010 SOL English/reading assessment third grade
by years of attendance in Virginia Reading First Schools
474.6
Scaled scores
480
470
452.0
460
450
446.8
438.9
440
430
420
410
400
1 year
2 years
3 years
4 years*
*Tukey’s HSD post-hoc analysis; significance at α = 0.05
The analysis was then repeated using proficiency levels. Figure 16 displays the
percentage of students who scored at the proficient level and those who scored at the
advanced level. As suggested by the graphic, the longer the students remained in a Virginia
Reading First school, the greater their likelihood of passing the Standards of Learning (SOL)
English/reading assessment. Specifically, these students were more likely to score at the
advanced level than students who had been in Virginia Reading First schools two or fewer
years. The percentage of four-year students scoring advanced on the SOL is statistically
significantly higher than students who attended Virginia Reading First schools for two years
or less.6
Statistical significance was calculated at α = 0.05. The alpha was adjusted for multiple comparisons using four
groups and calculated to be α* = 0.017. Logistic regression analysis controlled for differences of race/ethnicity,
SIED status and SWD status.
6
Virginia Reading First Outcome Evaluation – 26
Figure 16: Proficiency levels on the SOL English/reading assessment for grade three by
years in schools
4 years
42.8%
3 years
38.6%*
39.2%
2 years
30.8%
49.0%
1 year
23.2%
43.5%
0%
10%
20%
23.5%
30%
Pass
40%
50%
60%
70%
80%
90%
Advanced
NOTE: (*) indicates that the difference between this and other proportions is statistically significant at α = 0.05
Virginia Reading First Outcome Evaluation – 27
CONCLUSIONS
In 2010, the last year of the Virginia Reading First program, 21 schools in 17 school
divisions served 5,339 students from kindergarten to grade three. Of these, 60 percent were
classified as economically disadvantaged, 47 percent were minorities, seven percent were
students with limited English proficiency, and nine percent were students with disabilities.
Results on the Stanford Reading First (SRF) showed that students scoring at grade
level comprised 73 percent of kindergarteners, 66 percent of first-grade students, and 65
percent of students in second grade. For the three grades, the two weakest components in the
SRF were vocabulary development and reading comprehension. Alternatively, phonemic
awareness was the strongest component with 80 to 96 percent of the students scoring at grade
level. Scores on the phonics component should be taken with caution as this component of
the SRF is not aligned with the Virginia Standards of Learning, particularly for grade one.
When scores are disaggregated by subgroups, scores for boys and girls were similar at
kindergarten, but differences start to appear as they move to upper grades. In grade two, girls
outperformed boys in almost all components of the SRF. Although the academic gap between
white students and minority students (African Americans and Hispanics) still remained, a
decrease in the gap was noted. Vocabulary and reading comprehension were the two
components that require the most attention for SIED, Hispanics, African Americans, and LEP
students. Reading fluency is another area where students with disabilities need further help.
Seventy-three percent of the students in grade three in the Virginia Reading First
schools passed the Standards of Learning (SOL) English/reading assessment. Of these, 44
percent scored at the proficient level and 31 percent attained the advanced level. The majority
(60 percent or more) of the students passed the test in almost all subgroups, except for
students with disabilities and students with limited English proficiency. For these two groups,
the passing rate was 50 percent and 56 percent, respectively.
Addressing the Research Questions
1. What are the demographic characteristics of students in Virginia Reading First
schools?
Virginia Reading First Outcome Evaluation – 28
Virginia Reading First schools served a large number of students that are traditionally
less likely to obtain academic success. In the last year of the grant (2009-2010 academic
year), the Virginia Reading First program involved 21 schools with 5,339 students from
kindergarten to grade three. Of these, 60 percent were classified as economically
disadvantaged, and 47 percent were students from minority backgrounds. Across the grant
period, from September 2003 through June 2010, Virginia Reading First schools served a total
of 132,038 students; 56 percent were classified as economically disadvantaged, 9.8 percent
were students identified as having disabilities, and four percent were identified as having
limited English proficiency. Minority students comprised about half of the total student
population.
2. What are the student characteristics in Virginia Reading First schools compared to
Virginia’s overall population of students in kindergarten to grade three?
Compared to Virginia’s overall student population for the same age group, Virginia
Reading First schools were more likely to serve minority students and students classified as
economically disadvantaged. In 2010, 53 percent of the students from kindergarten to grade
three in Virginia Reading First schools were white, while 47 percent were students from
minority background, particularly African Americans (35 percent), and Hispanics (six
percent). Statewide, for the same age group, the percentages were: 55 percent white, 24
percent African American, and 11 percent Hispanics. Regarding SIED, in 2004, 57 percent of
third graders in Virginia Reading First schools were classified as economically disadvantaged,
compared to 62 percent in 2010.
3. What percentage of students in kindergarten to grade two in schools receiving Reading
First subgrants achieved proficiency on the grade-level outcome measure in each year?
Between 2004 and 2010, the percentage of students at grade level in the kindergarten
SRF assessment increased by 13 percent in phonemic awareness, 22 percent in phonics, 24
percent in vocabulary development, 19 percent in reading fluency, and 29 percent in reading
comprehension. By spring 2010, 88 percent of kindergarteners were scoring at grade level in
phonemic awareness, 76 percent were at grade level in phonics, 67 percent in reading fluency,
while about half were at grade level in vocabulary development and reading comprehension.
For first graders, the percentage of students at grade level during the same period
increased by 25 percent in phonics, six percent in reading fluency, four percent in reading
Virginia Reading First Outcome Evaluation – 29
comprehension, three percent in phonemic awareness, and two percent in vocabulary
development. By spring 2010, 82 percent of first-grade students were scoring at grade level
in phonemic awareness, 73 percent were at grade level in reading comprehension, 61 percent
were at grade level in reading fluency, and 61 percent were at grade level in vocabulary
development. Results in phonics (37 percent at grade level) are not reliable, as the phonics
component of SRF does not align with the Virginia Standards of Learning for grade one.
At second grade, between 2004 and 2010, the percentage of students at grade level
increased by 24 percent for phonics, nine percent in reading comprehension, six percent in
vocabulary development, three percent in phonemic awareness, but no change in average for
reading fluency. By spring 2010, 96 percent of the students in grade two were scoring at
grade level in phonemic awareness, 55 percent in reading fluency, 54 percent in vocabulary
development, 46 percent in reading comprehension and 43 percent in phonics. Despite the
low score in phonics, this was the SRF component with the greatest improvement across the
years.
4. How did student achievement in grade three compare between schools receiving
Reading First subgrants and the state average?
Third-grade students made strong improvements on the Standards of Learning (SOL)
English/reading assessment. In 2004, 58 percent of third graders in Virginia Reading First
schools passed this assessment, compared to 72 percent statewide. In 2010, 75 percent of the
third graders in Reading First schools passed the Standards of Learning (SOL)
English/reading assessment, compared to 83 percent statewide, an eight percentage point
difference. The increase in percentage of students passing the assessment between 2004 and
2010 was 29 percent for Virginia Reading First Schools, compared to a 15 percent increase
statewide. A calculation of effect size on the SOL results across the time of the program
showed that improvements in reading for students in Virginia Reading First schools were
stronger than those for students statewide.
5. How did reading achievement on the SRF assessment compare to national norms
provided by Harcourt?
In 2010, students in kindergarten scored, on average, at or above 73 percent of
kindergarten students nationwide in the multiple choice component of the SRF; first-grade
students scored at or above 58 percent of first graders taking the test nationwide, and second
Virginia Reading First Outcome Evaluation – 30
graders scored at or above 53 percent of the national sample of second graders. These results
were very similar to results for the 2009 SRF assessment. Across seven years of the Virginia
Reading First program, kindergarten students scored above the national norm, while students
in first and second grades scored at the norm provided by Harcourt.
6. How did student performance in Reading First schools vary by subgroup?
Regarding gender, kindergarten girls and boys performed similarly on the SRF, except
for the reading comprehension component, where girls outperformed boys.7 In grade one,
girls outperformed boys in all but two components of the test (vocabulary development and
phonics). At grade two, girls outperformed boys in all components but one (vocabulary
development). At grade three, girls outperformed boys in the Standards of Learning (SOL)
English/reading assessment; mean scaled score for girls was 465 and for boys the mean scaled
score was 449.
Results for students identified as economically disadvantaged (SIED) were compared
to the test results for students who were not identified as economically disadvantaged (Not
SIED). Not SIED outperformed SIED in all grade levels for both the SRF and SOL
assessments. The SIED mean scale in the grade 3 SOL was 442 and the Not SIED was 482.
An analysis of data from Cohort 1 schools (the schools that were in the program for six
years) showed a decrease in the achievement gap between SIED and Not SIED students. On
the 2006 SOL English/reading assessment,8 68.9 percent of Not SIED passed the assessment,
compared to 51.7 percent for SIED. On the 2009 SOL, the last year for Cohort 1 schools, 89
percent of Not SIED and 76.4 percent of SIED passed the SOL English/reading assessment.
Therefore, not only did both subgroups improve their level of performance, but the
achievement gap decreased by 30.2 percent.
Students with disabilities (SWD) had their assessment results compared to students
who did not receive special education services (Not SWD). Not SWD outperformed their
SWD peers in all grades and assessments; their mean scaled score was 463 for the SOL,
compared to a 404 mean scaled score for the SWD.
The word “outperformed” is used here to indicate statistical significant differences in test results for the
subgroups, with a 95 percent confidence level.
8
SOL data from years prior to 2006 were not disaggregated for SIED and Not SIED students.
7
Virginia Reading First Outcome Evaluation – 31
Regarding race/ethnicity, the analysis focused on the three largest groups within this
subgroup: African Americans, Hispanics, and whites. White students tended to outperform
African American students in most components of the SRF, except phonemic awareness and
reading fluency at kindergarten, and speaking vocabulary at first grade. In these three cases,
mean scores for African American and white students were similar. White students also
outperformed Hispanic students in nearly all components, except in the phonemic awareness
test for first and second grades. In these two tests, Hispanic and white students had similar
scores.
An analysis of data from Cohort 1 schools found a decrease in the achievement gap
between white students and their African American and Hispanic peers for the grade 3 SOL
English/reading. In the 2004 SOL, 66.8 percent of the white students passed the test,
compared to 51.4 percent of African Americans and 52.4 percent of Hispanics. In the 2009
SOL, 87.5 percent of the white students passed the test, compared to 75.5 percent of African
Americans and 77.8 percent of Hispanics. Therefore, the achievement gap on the SOL
between African American and white students decreased by 22.1 percent, while the gap
between Hispanic and white students decreased by 32.6 percent.
7. Has student achievement in all subgroups shown consistent improvement over time?
Focusing solely on the statewide assessment, the percentage of students with
disabilities in Virginia Reading First schools who passed the SOL English/reading assessment
increased by 24 percent between 2004 and 2010. In 2010, 73 percent of the students with
disabilities in Virginia Reading First schools had passed the Standards of Learning (SOL)
English/reading assessment, compared to 53 percent in 2005. A decrease in the academic gap
was also observed for students from minority backgrounds, as described above. Time spent in
a Virginia Reading First school had a positive impact on student outcome, as students who
had been in the schools for four years showed stronger improvements on the test compared to
students who had been in the schools for two or fewer years. The longer the students stayed
in the program, the greater their likelihood to score at the advanced level. Mobility is a reality
for these schools, though. For instance, in 2010, one third of the third-grade classes in
Virginia Reading First schools were comprised of students who had enrolled in the school for
the first time that year.
Virginia Reading First Outcome Evaluation – 32
Next Steps
The analyses of reading scores for Virginia Reading First schools provided evidence
of the impact of the program on the reading skills of all participant students, but particularly
those students who are most at risk for academic failure. Evidence of this success is the
improvement in scores, particularly for the Virginia Standards of Learning (SOL) English/
reading assessment. The VDOE, school divisions, and schools should celebrate these results,
while ensuring that the instructional strategies learned during the Reading First program are
not forgotten as the grant funds expire.
While celebrating success is important, it is equally important to revisit the results of
the evaluation reports to understand where further efforts are required or where changing of
strategies may be needed. One area that deserves attention is the discrepancy between results
on the phonemic awareness component of the SRF and the vocabulary development and
reading comprehension components. While it is elating to see that 96 percent of students in
second grade are at grade level in phonemic awareness, it is of concern that fewer than half of
the same students are at grade level in reading comprehension. It is true that phonemic
awareness is a foundational skill for reading, but without a strong vocabulary and
comprehension of reading passages, students will have difficulty reading in the upper grades,
where more complex skills are required. Likewise, it is invigorating to see that academic gaps
between the different student subgroups are being decreased. However, students that are
economically disadvantaged still lag behind their peers from upper socioeconomic strata, at
least regarding their achievement in standardized assessments.
At this time, the Virginia Department of Education, like many other state agencies
across the nation, is actively involved in the development of a comprehensive literacy plan
birth to grade twelve. This is an important time to revisit the findings from the Reading First
program, and re-examine these findings from the perspective of new advances in reading
research. This merging of evidence derived from practice and research findings can help
create strong foundations for a plan that will ensure that all students graduate from Virginia
schools with strong literacy skills.
Virginia Reading First Outcome Evaluation – 33