Evaluating the NYC Core Knowledge Language Arts
Pilot: Summary of Kindergarten and Grade 1 Results
HIGHLIGHTS
Research and Policy
Support Group
1
Summary of Kindergarten Results
2008-2009 School Year
Research and Policy
Support Group
2
Focus of the Evaluation
Hypothesis: Kindergarteners taught with the Core Knowledge Language Arts
(CKLA) Program will gain reading competencies at a faster rate than their
peers.
Methodology: A multi-method, longitudinal research design
YEAR 1
Literacy Assessments (at 10 CKLA schools & 10 comparison schools, 4
of which are Reading First schools):
• Pre- and post-test of literacy skills
• Additional tests of literacy skills at end of each year
• Periodic assessments throughout the year (DIBELS)
Teacher and Administrator Surveys (at 10 CKLA schools):
• Assesses satisfaction with and impact of CKR
Case studies (at 3 CKLA schools):
• Classroom observations, administrator & teacher
interviews
3
Similar Demographics at CKLA and Comparison Schools
Percent of Students
(Number of Students for School Size)
CKLA Schools (N = 584)
Comparison Schools (N = 307)*
Note: These and other data were used to select comparison schools (data as of 2007-08 school year).
4
Different Literacy Domains: Greater Gains & Higher Spring
Scores for CKLA Students than Comparison Students in All
Literacy Domains
W-J Brief Reading
Basic
Reading
Skills
Oral
Reading
Comprehension
Decoding
Written
Spelling
Oral Reading
Comprehension,
Vocabulary,
Basic
Reading,
Decoding
W-J Letter Word
Identification
W-J Passage
Comprehension
W-J Word Attack
W-J Spelling of
Sounds
Terra Nova
Reading





CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig. Higher
Spring Scores
CKLA Sig. Higher
Spring Scores
CKLA Sig. Higher
Spring Scores





CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig. Higher
Spring Scores
CKLA Sig. Higher
Spring Scores
CKLA Sig. Higher
Spring Scores
CKLA
Schools
Compared
to…
All
Comparison
Schools
Reading First
Comparison
Schools
5
5X Greater Literacy Gains for CKLA Students than Students
at Demographically Similar Comparison Schools
Average Fall-Spring Gain
in Scale Score Points
Woodcock-Johnson (Brief Reading Test)
CKLA Students
Significantly Higher
p < .001
CKLA
Comparison
6
6X Greater Literacy Gains for CKLA Students than Students
at Reading First Comparison Schools
Average Fall-Spring Gain
in Scale Score Points
Woodcock-Johnson (Brief Reading Test)
CKR Students
Significantly Higher
p < .001
(N = 314)*
(N = 103)*
* Data presented represent the four comparison schools that are Reading First schools and the four CKR schools who
were statistically comparable matches.
7
At All Achievement Levels, Greater Literacy Gains for CKLA
Students than Students at Comparison & Reading First
Schools
Average Fall-Spring Gain
in Scale Score Points
Woodcock-Johnson (Brief Reading Test)
CKR & All Comparison Schools
CKR & Reading First Schools
CKR Schools
CKR Schools
Comparison Schools
Reading First Schools
8
Summary of Findings
 By nearly all measures, Core Knowledge Language Arts (CKLA)
students made significantly greater gains in early literacy than peer
students.
 Compared to peers, kindergarteners taught with the CKLA
program made more progress in all areas of reading tested:
spelling, phonemic awareness, decoding, and comprehension.
 These findings hold for two types of peer comparisons: 1)
demographically similar schools and 2) demographically
similar schools using Reading First (another intensive,
focused literacy program).
 Surveys and case studies indicate overall high levels of
administrator and teacher satisfaction with the CKLA Pilot, while
also offering guidance for year 2 implementation and evaluation
work.
 Administrators would recommend program to others; teachers
rate CKLA more favorably than other programs.
 Administrators report change in teacher practice: more datadriven instruction & teacher collaboration.
 Teachers and administrators feel more positively about the
Skills Strand than the Listening and Learning Strand,
particularly regarding student engagement.
 Teachers may need more support addressing needs of
struggling readers with CKLA & managing time to complete
lessons.
9
Summary of Grade 1 Results
2009-2010 School Year
Research and Policy
Support Group
10
Focus of the Evaluation
Hypothesis: First Graders taught with the Core Knowledge Language Arts (CKLA)
Program will gain reading competencies and content knowledge (science and
social studies skills) at a faster rate than their peers.
Methodology: A multi-method, longitudinal research design
YEAR 2
Assessments (at 10 CKLA schools & 10 comparison schools, 3* of which are
Reading First schools):
• Pre- and post-test of literacy skills
• Additional tests of literacy skills at end of each year
• Tests of science and social studies skills at end of each year
Teacher and Administrator Surveys (at 10 CKLA schools):
• Assesses satisfaction with and impact of CKLA
Case studies (at 5 CKLA schools):
• Classroom observations, administrator & teacher interviews
* Last year there were 4 Reading First comparison schools. This year there were 3 Reading First comparison
schools: one comparison school used Reading First in Year 1 but not in Year 2.
11
Overall, CKLA and comparison schools had similar demographic profiles.
However, the differences between groups, especially % of Black/ Hispanic
students, was significantly larger than last year.
As a result, all analyses control for student demographics.
Percent of Students
(Number of Students for School Size)
CKLA Students (N = 781)
Comparison Students (N = 343)*
•N = the number of
students for whom both
fall and spring data were
available.
A random sample of half
of the students in the
comparison schools were
selected for testing. All
students in CKR schools
were tested.
Note: Data from 08-09 was used to select comparison schools (data presented here are from the 2009-10
school year).
12
CKLA students displayed greater gains & higher Spring scores in nearly
all literacy and content knowledge domains relative to comparison
students
Woodcock Johnson (Fall and Spring)
Test
Terra Nova (Spring Only)
W-J Brief Reading
W-J Letter
Word
Identification
(Basic Reading
Skills)
Terra Nova
Reading
W-J Passage
Comprehension
W-J Word
Attack
W-J Spelling of
Sounds
(Oral Reading
Comprehension)
(Decoding)
Terra Nova
Social Studies
Terra Nova
Science
(Written Spelling)
(Oral Reading
Comprehension,
Vocabulary,
Basic Reading,
Decoding)
CKLA Schools Compared to…
All
Comparison
Schools
Reading First
Comparison
Schools







CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater
Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Spring
Scores
CKLA Sig.
Greater Spring
Scores
CKLA Sig.
Greater Spring
Scores






X
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater
Gains/
Spring Scores
CKLA Sig.
Greater Gains/
Spring Scores
CKLA Sig.
Greater Spring
Scores
CKLA Sig.
Greater Spring
Scores
No Significant
Difference
Note: All analyses control for student demographic characteristics.
13
CKLA students displayed nearly 2x greater literacy gains relative to
students at demographically similar comparison schools
The magnitude of the group difference was smaller than last year
(in Year 1, CKLA students’ gains
were 5X greater than their peers).
Average Fall-Spring Gain
in Scale Score Points
Woodcock-Johnson (Brief Reading Test)
CKLA Students
Significantly Higher
p < .0001
(N = 758)
(N = 342)
14
CKLA students displayed 6x greater literacy gains relative to students at
Reading First comparison schools
The magnitude of the group difference was smaller than last year
(in Year 1, CKLA students’ gained
14.3 pts while Reading First schools declined 4.1 pts).
Average Fall-Spring Gain
in Scale Score Points
Woodcock-Johnson (Brief Reading Test)
CKLA Students
Significantly Higher
p < .0001
(N = 233)*
(N = 100)*
* Data presented represent the three comparison schools that are Reading First schools and the three CKR schools
who were statistically comparable matches.
15
Measuring fidelity of implementation to the CKLA curriculum
• Examining implementation fidelity allows us to better determine:
• 1) whether achievement gains can be attributable to the CKR
program and
• 2) which components of the CKR program teachers are
struggling to implement and those they are implementing
successfully
• We took several different approaches to measuring fidelity in the
CKR Pilot schools:
• Site visits with classroom observations
• Site visits were conducted at 5 randomly selected
Pilot schools
• 15 classrooms were observed
• Interviews with administrators and teachers
• Interviews were conducted with 13 teachers and 7
administrators
16
Example of a school with high fidelity
Anecdotes from Classroom Observation
•
The first classroom had a vibrant atmosphere. Approximately 30 students, all of whom were
seated on the carpet, appeared completely absorbed, watching their teacher and the images
about the U.S. space program that were projected on the screen beside her.
•
The classroom was covered with student work, Core Knowledge posters and chaining charts, and
a hanging model of our solar system.
•
The students were eager to share their answers, some of them using challenging vocabulary
words that they had learned from earlier lessons like “perseverance.”
•
After a writing/drawing exercise, in the final minutes of the lesson the teacher played a song about
the sun that she had found on the internet. Some students pretended to be conductors, which
they had learned about in the Mozart unit they had recently completed.
•
This school had a teacher-coordinator who had participated as a kindergarten teacher the year
before and was providing specific support for both the Core Knowledge pilot in first grade and the
kindergarten classrooms, which had continued to use the curriculum.
•
The school is also using the Core Knowledge curriculum for all grades.
17
Example of a school with high fidelity
Administrators’ Views
Teachers’ Views
•
•
•
Both teachers were very positive about the
program’s approach to developing early childhood
literacy in their 1st graders.
They liked using the program, its structure, content,
and activities, and they also could see the students
benefiting from it.
Their criticisms of the program were of more
“mechanical” elements (e..g, lack of page numbers
in the readers, would have liked captions to
accompany the pictures and make them more
interactive, they felt rushed to get through all the
material for any give day)
•
Administrators were very supportive of the program
•
They perceived large benefit from the program’s
“multi-intelligence” approach and have been excited
to watch the students make connections between
lessons and to their lives outside the classroom as
well.
•
They perceived more parent involvement—children
dragging their parents to the library, parents
commenting that they are learning from their
children. Administrators’ Views
•
They felt the program worked well largely because
their teachers are well supported and have fully
embraced the program.
•
Criticisms were that the program needs to include
more “higher-order thinking” questions and provide
material tailored for special populations (e.g., ELLs).
18
Example of a school with lower fidelity
Anecdotes from Classroom Observation
• The structure of the classrooms themselves created a challenge for the three first
grade teachers—they only have three walls, and are open onto a hallway on the
fourth side. Not only did the teachers have to contend with noise from the other
classes, but also with the distractions of passersby using the hallway.
• In two of the three classes, students were having a difficult time responding
appropriately to their teacher’s directions, questions, and assignments.
• The students in one of these classrooms also had a difficult time listening
respectfully to each other during the discussion time at the end of a Listening and
Learning lesson. This was the only example we saw of this kind of behavior across
all 15 classrooms.
• One of the classrooms did not have the Core Knowledge materials posted and was
missing the chaining and tricky words.
• Toward the end of the year, the CKR literacy coordinators worked with the teachers
to take a different approach to differentiation. They divided the students into three
homogenous skill groups and then each teacher worked with one group three days
a week for a 45 min. session.
19
Example of a school with lower fidelity
Teachers’ Views
•
Did not feel supported by the school’s
administrators. This made it that much more
difficult to be successful in their implementation
of the program because the administrators were
out of touch with what the teachers needed in
order to be fully successful in their
implementation.
•
Sometimes found it difficult to fit in all the
content for both the Listening and Learning and
the Skills Strand, and would resort to picking out
the highlights for a reading or selecting the most
feasible activities in order to optimize their time.
•
For all the challenges, the teachers were
positive about the program itself—they saw their
students engaging with it and connecting it to
their lives outside the classroom.
Administrator Views
•
The administrator we interviewed was very
enthusiastic about the program.
•
Confirmed that the teachers like the program,
but needed a little extra support.
•
Researchers and implementation managers
also had difficulty communicating with
administrator (e.g., email inbox was always so
full that any emails we sent were returned).
20
Fidelity of implementation and teacher effectiveness made a difference:
Year 2 School Gains on WJIII
Brief Reading Assessment
Significantly
Positive Gain
Non-significant
Gain
Significantly
Negative Gain
Pilot/Comparison
School
Average Fall to
Spring Gain,
Year 2
Pilot
19.1
Pilot
10.2
Pilot*
13.1
Pilot*
3.5
Pilot
2.6
Pilot*
3.7
Comparison
2.6
Comparison
2.1
Comparison
Pilot
2.2
1.2
Comparison
2.4
Comparison
1.3
Comparison
1.1
Comparison
1.1
Comparison
Pilot*
1.3
0.5
Comparison
Pilot
0.3
-1.5
Comparison
Pilot*
-6.0
-13.2
Three Site Visit Schools Did Not Significantly Gain or
Declined in Performance from Fall to Spring:
• One of these schools was the low fidelity school described
previously.
• One of these schools had a teacher new to teaching first
grade who had been in and out of the classroom, and the
relationship between the principal and this teacher was
strained.
• All schools spent less than the required 60 minutes on the
Listening and Learning Strand and the Skills Strand.
• One of these schools also had a very strong focus on
writing, which would occasionally cut into their other strand
time.
21
Executive Summary – Achievement Gains
 By nearly all measures, Core Knowledge Language Arts (CKLA)
students made significantly greater gains in Year 2 than their peers
 Overall Achievement
 When compared to peers (both peers at demographically similar
schools and peers at Reading First schools), 1st graders taught with
the CKLA program made more progress in reading and social
studies.

Spring science scores were significantly greater for students in pilot schools than comparisons schools, but
not significantly different from students in Reading First comparison schools.
 High vs. Low Performers
 High and low performing CK students made larger gains than
comparison students
 Year 2 vs. Year 1 of the Program.
 Both students with only one year of CKLA instruction and those with
two years of CKLA instruction made greater gains than their peers
at comparison schools.
22
Executive Summary – Surveys & Case Studies
 Surveys and case studies indicate overall high levels of
administrator and teacher satisfaction with the CKLA Program and
high levels of fidelity in program implementation.
 While survey respondents had favorable reactions to CKR, they
indicated that they felt CKLA rated similarly to other programs they
had used.
 Surveys and case studies revealed that teachers are struggling with
how to differentiate instruction with CKR, in particular for special
populations.
 Case study schools’ fidelity to the program aligned with their
achievement gains: schools with low fidelity demonstrated less
gains
23