Word Knowledge and Comprehension Outcomes for the Second Year of
Implementation of an Academic word Vocabulary Intervention
Margaret McKeown, Amy Crosson, Isabel Beck, Cheryl Sandora, Nancy Artz,
and Debra Moore
University of Pittsburgh
Purpose
A preponderance of evidence suggests that too few students are reading at levels
needed to learn from required school texts (American Institutes for Research, 2006;
Heller & Greenleaf, 2007; NAEP, 2011). The most recent NAEP results show that only
34% of 8th graders read at or above proficient levels. Vocabulary demands of texts that
students are assigned in middle school are cited as a major source of reading difficulty.
Results from the NAEP vocabulary assessment (NCES, 2012) seem to confirm that
conclusion, as vocabulary performance—assessed within the context of passage reading-closely aligned with comprehension scores. The purpose of the work to be reported was
to enhance vocabulary knowledge and comprehension through an intensive, interactive
vocabulary intervention for middle school students.
Perspective
The perspective underlying the research to be reported is that vocabulary
knowledge is a network of semantic connections built from multiple encounters with
words in meaningful contexts (Perfetti, 2007; Reichle & Perfetti, 2003). The greater the
variety and meaningfulness of encounters the more likely that learners will develop
semantic representations that foster successful, efficient comprehension of text (Perfetti
& Hart, 2002; Nagy & Scott, 2000). These findings based on behavioral and neuroscience
data align with consensus reached over decades of instructional research, that
comprehension can be affected by vocabulary instruction if the instruction provides
experiences with words in multiple contexts and promotes active processing--interacting
with and integrating various uses of words and building connections to prior knowledge
and related concepts (Baumann, Kame’enui, & Ash, 2003; Bolger, Balass, Landen, &
Perfetti, 2008; Nagy & Scott, 2000; National Reading Panel, 2000; Perfetti & Hart, 2002;
Perfetti &, Stafura 2014; Stahl & Fairbanks, 1986). Context integration, the ability to
accommodate word meaning within surrounding context in order to make sense of the
context, has been identified as a key component that links vocabulary knowledge with
successful comprehension (Jenkins, Pany, & Schreck, 1978; Kame’enui, Carnine, &
Freschi, 1982; Perfetti, 2007).
Method
Intervention Design
The vehicle for our work to enhance comprehension through vocabulary was an
instructional intervention designed to teach academic words to middle school students,
Robust Academic Vocabulary Encounters (RAVE). Target words were selected from the
Academic Word List (AWL) (Coxhead, 2000), which consists of words that commonly
1
appear across academic texts from a range of domains. The principles of our
instructional design were presentation of definitional and contextual information,
encounters in multiple contexts, and active processing. A particular goal for providing
contextual information was to expose students to various uses and senses of words. An
inherent characteristic of academic words, because they are found across domains, is that
the words carry various senses in different contexts. We identified and adapted contexts
from a variety of print and internet sources to represent authentic uses of target words and
to engage students in discussion of different senses.
Students were introduced to each word through a pair of contexts and a definition
developed to present a core meaning for each word, that is, one that would apply to
multiple senses. The contexts and definitions served as grist for promoting integration of
the word within the contexts. For example, students read a context describing a program
intended to expose students to different kinds of music, and read the definition “If you
expose something, you let it show or make it known.” They were then asked how the
meaning of the word fit the context. A typical response was: “The program let them
know about new kinds of music.”
Active processing is embodied in RAVE through activities that prompt students to
evaluate, elaborate, and generate various uses of the words. These activities were
presented over several days following initial instruction. For example, students are asked
to respond to scenarios around the words (“What would be relevant to deciding which
movie to go to?”) and to differentiate how words influence a context (explain how to
convince your boss that you’re a dynamic worker vs. a worker with integrity).
Materials were developed for two years of instruction, for sixth and seventh
grades, and implemented over two years on one school. This paper reports on the second
year of implementation. Results from the sixth-grade implementation were presented in
McKeown, Crosson, Beck, Sandora, Artz, Nelson, and Zhang (2013).
Study Design
Participants. The impact of RAVE on student learning of academic vocabulary
and comprehension was investigated using a quasi-experimental design with seventh
graders during 2012-13, the second full year of implementation of RAVE. The
participants were one seventh-grade reading teacher and 108 students (RAVE n=64;
control n=44) from all five seventh-grade classes in a public school within a working
class community in the northeastern United States. Eighty-nine of these students had
participated in the project as sixth-graders. Due to the school’s structuring of the seventh
grade classes, 44 control students and 43 of the RAVE students were in heterogeneously
grouped classrooms and 21 RAVE students were in an honors class. Results will be
reported separately for the honors class, referred to hereafter as RAVE-Honors.
Students in the RAVE conditions received instruction on 96 academic words over
about 22 weeks’ time, while control students received an approximately equal dosage
from a commercial vocabulary program (Bacon, 2005).
Measures. The selection of measures was driven by what we wanted to know
about students’ learning. Specifically, our goals included measuring depth of knowledge
of word meaning, especially for words with multiple senses; access to the words for
fluent processing, and the ability to use the words to further comprehension of text. We
will present results on four measures: researcher-designed tests of word knowledge,
2
lexical decision, and passage comprehension, and a standardized comprehension
assessment.
First, the word knowledge measure, the Cloze Evaluation of Depth of Word
Knowledge (Cloe), tested all 96 instructed words and was administered as a pre and posttest. This task comprised four cloze sentences per target word and required students to
determine whether the target word made sense in each of the sentences. The measure
tested students’ knowledge of multiple senses of academic words (e.g., physical and
mental sense of confine). Foils were constructed to include an easier, syntactic foil
(requiring incorrect part of speech) and a more difficult semantic foil with a prototypical
association to the target word (e.g., using an association with prison for the target word
confine, in the foil “Prisoners often _____ letters to send news to their families.”) to test
different levels of word knowledge. Cloe has had acceptable internal consistency
(Cronbach’s = .91).
In addition to all words taught in seventh grade intervention, 12 words that were
taught only in sixth grade were also included on the test. These words had the greatest
learning gains from pre to post in sixth grade. The goal was to evaluate retention of
learning.
Second, a lexical decision task was administered following the intervention to
assess fluency of access to target word meanings. Fluent semantic access is an aspect of
word knowledge that is essential for comprehension (Beck, Perfetti, & McKeown, 1982;
Richter, Isberner, Naumann & Neeb, 2013). In this computer-based reaction-time
measure, students made decisions as to whether strings of letters shown on the screen
were a word. Three sets of words/nonwords were shown at random (using E-Prime
software). The first set comprised 20 RAVE words that had received an average score of
3 or higher on the Cloe pre-test for both control and RAVE students, or that were
prevalent in the school environment. The second set comprised 20 comparison words
equated for frequency and orthographic complexity. The third set comprised nonwords
equated for orthographic complexity. Our hypothesis was that more efficient decisionmaking about target words that were already somewhat familiar to most students in the
RAVE and control conditions would indicate that participation in the RAVE intervention
may support development of more fluent semantic access to the words in the intervention.
Third, a passage comprehension task was administered to all students as a posttest.
The task comprised four short passages containing six target words each followed by six
multiple choice questions that asked about the part of the context pertaining to a target
word. For example, one of the texts described Harry Houdini as “the world’s most
reliable escape artist.” The question about that context was: “What made Houdini so
famous?” and the correct response was: “He escaped successfully every time.” Thus the
questions tapped students’ ability to integrate the meaning of the target word within the
context.
This measure was based on an existing comprehension assessment, TOIW-C,
used with fourth graders in an efficacy study of vocabulary instruction (Apthorp et al.,
2012). It was administered to 1,450 fourth-grade students, and demonstrated a significant
difference between instructed and control students, with an effect size of 0.41, using
Glass’s d. Reliability coefficient (Cronbach’s ) was .75.
Fourth, a standardized reading comprehension assessment, the Gates-MacGinitie
Reading Tests (MacGinitie, MacGinitie, Maria, Dreyer & Hughes, 2007) was
3
administered pre and post to assess potential influence of student learning on general
comprehension. Standardized reading comprehension tests are blunt instruments, and it
is still an open question as to why and when growth on such measures should be expected
in relation to specific interventions. We hypothesized that such growth might be evident
for students exposed to RAVE instruction for two years. As this measure was also
administered as a pre and post test in sixth grade, student performance was examined to
investigate impact of the intervention on comprehension over a two-year period.
Results
Word knowledge. A one-way between subjects analysis of variance was
performed on the gain score as a function of treatment. The Gates seventh grade pretest
score was not a significant predictor and therefore not included as a covariate. There was
a significant difference on Cloe gain score between control, RAVE, and RAVE-Honors
groups, F(2, 103) = 16.93, p < .000, η2 = .247, thus post hoc pairwise comparisons using
a Bonferroni adjustment were performed. The RAVE-Honors group had significantly
higher scores than the control (mean diff = 41.52, p < .000, Cohen’s d = 1.95) and RAVE
groups (mean diff = 24.31, p = .003, Cohen’s d = 0.91). The RAVE group had
significantly higher scores than the control group, mean diff = 17.21 p = .012, Cohen’s d
= 0.60. See Table 1 for descriptive statistics.
For the 12 sixth grade words included on the seventh grade Cloe, a mixed analysis
of variance was performed on the scores as a function of time and condition. The withinsubjects independent variable was time with four levels; 6th grade pretest, 6th grade
posttest, 7th grade pretest, 7th grade posttest. The between-subjects variable was 6th grade
condition (RAVE or control). The finding of interest is that, while there were no
significant differences between RAVE and control students at sixth grade pretest, at all
subsequent time points students in the RAVE condition had significantly higher scores
than control students (Figure 1). This finding suggests that learning in sixth grade was
retained (to some degree) for a full year after the intervention. Further, results of the Cloe
supported our hypothesized continuum of difficulty, with syntactic foils the easiest to
reject and semantic foils, which presented a typical association, the most difficult
(Crosson, McKeown, Beck, & Ward, 2012).
Lexical decision. RAVE students made more efficient and accurate decisions
about target words (mean=887.77 milliseconds) than words that were matched for
frequency and orthographic complexity (mean=987.64 milliseconds), whereas for
comparison students there was no difference in the accuracy and efficiency with which
they read the words (target word mean=949.62 milliseconds; matched word
mean=982.36 milliseconds). (Results displayed in Figure 1.). A repeated measures
analysis of variance was performed on the reaction times with word type (RAVE or
matched) as within subject variable and treatment condition as between subject variable.
There was a significant interaction, showing that the RAVE students had faster reaction
time to the RAVE words than to the matched control words, F = 4.671; p < .05.
Passage comprehension. A one-way between-subjects analysis of covariance was
performed on passage comprehension scores as a function of treatment after adjusting for
7th grade Gates total pretest score. There was a significant difference in Passage
Comprehension score among control, RAVE and RAVE-Honors groups; F(2,104) = 8.83,
p < .000, partial η2 = .145, thus post hoc pairwise comparisons using a Bonferroni
4
adjustment were performed. The RAVE-Honors group had significantly higher passage
comprehension than the control (mean diff = 4.86, p = .001, Cohen’s d =1.20), and RAVE
group (mean diff = 3.37, p= .017, Cohen’s d = 0.82). The RAVE group had marginally
significantly higher posttest scores than control (mean diff = 1.49, p = .059, Cohen’s d =
0.43). See Table 1 for descriptive statistics.
Standardized reading comprehension. The effect of RAVE on the standardized
reading assessment was tested by analyzing performance from the outset (6th grade
pretest) to the end (7th grade posttest) of the intervention (Table 2). A 2 x 4 mixed
analysis of variance was performed on Gates scaled scores as a function of time and
grouping. The RAVE-Honors group was not included in the analysis. The withinsubjects independent variable was time with four levels; pretest in 6th grade, posttest in
6th grade, pretest in 7th grade, and posttest in 7th grade. The between-subjects variable was
the pattern of intervention with two levels, students who were in the control group both
years and students who experienced RAVE both years.
The analysis of Gates scale scores among time points averaged across groups
revealed a significant difference, F (3, 108) = 3.92, p = .011, partial η2 = .098. The 6th
grade pretest was significantly lower than the 7th grade posttest (mean diff = -10.72, p
= .047). Examining the difference across time points for each group revealed no
significant difference for students who experienced the Control conditions in both years.
However, for students who received RAVE for two years, the difference between the 6th
grade pretest and the 7th grade posttest approached significance (p = .066).
Significance
The significance of the work to be reported lies in the fact that the RAVE
program promoted knowledge of academic words and text comprehension. However, a
caveat is that high-achieving students selected by the school for the honors program
demonstrated larger gains on both the word knowledge and passage comprehension tasks.
This finding lends further confirmation to decades of research demonstrating “Matthew
effects” in reading. However, while smaller in magnitude, treatment effects for the (nonhonors) RAVE group were significant for the test of word knowledge and marginally
significant for the passage comprehension task. Effects on a standardized comprehension
measure for the (non-honors) RAVE group also were marginally significant and
promising. The findings for the seventh grade implementation of RAVE are consistent
with those from sixth grade in word knowledge gains, yet the sixth grade results did not
show gains for passage comprehension or the standardized reading measure (McKeown
et al., 2013). This may suggest that comprehension gains from vocabulary instruction
take time to accumulate before affecting comprehension.
Moreover, qualitative data from the project indicate that students noticed and used
target words outside of school (McKeown, Crosson, Artz, Sandora, & Beck, 2013) and
that the lessons included elaborated discussion of word meaning and use (all lessons were
recorded and transcribed). Discussion has been suggested as key to supporting students
to move vocabulary knowledge to comprehension (Elleman, Lindo, Morphy, & Compton,
2009; Snow, 2013). Excerpts from RAVE lessons shed light on how that process might
unfold.
The following excerpt from a lesson transcript illustrates how discussion may
have supported students to develop understanding of target words as used in different
5
contexts. This lesson introduced the word potential with two contexts. The first was
about an artist who walked around the city looking for scenes “with the potential to be
captured in a painting and become a great work of art.” The second was about people
trying to sell their home and seeking “potential buyers.”
Following discussion of each context and the word’s definition, students
responded to the question of what was learned about the word potential from the two
contexts. The discussion shows that students struggled at first; the teacher provided a
review of the essential ideas from the contexts, and then a student was able to provide a
strong overview of the word:
Anetta:
You could…oh, you can make, like… Okay, they like their own, certain
kind of thing and they like…they have their own type of feelings and stuff.
Like, I know what it means but the two different passages.
Teacher:
Okay, but in those two passages, potential means the same thing even
though it’s used in two different contexts. Madison?
Madison:
It’s, like, saying how they like some of these things but they also didn’t
like some of them. It’s, like, they’re likes and dislikes.
Teacher:
Okay, let’s think about this. In context one, everything in his environment
had the potential of being in his painting. In context two, they were
hoping the person who looked at the house, was a potential buyer. So,
what do those two things have in common, Michael?
Michael:
Because they’re…he’s, like, saying the painting is able to become a great
work of art and then when they said… in the second text, it’s saying the
buyer is …the three buyers are able to buy the house.
Implications of these findings for instruction and assessment will be discussed.
6
Table 1.
Descriptive statistics for Cloe test of word knowledge and Passage Comprehension for
all groups.
n
Cloe Gain Score Passage Comprehension
Mean (SD)
Mean (SD)
RAVE-Honors 21 31.48 (18.95)
16.19 (.883)
RAVE
43
7.17 (33.09)
13.11 (.520)
Control
44 -10.05 (23.37)
11.62 (.528)
Table 2.
Descriptive statistics for Gates-MacGinitie Extended Scale Score at four time points for
students assigned to the control condition (n=14) and students assigned to the RAVE
condition (n=24) for two consecutive years.
Condition
Control
Time*
Mean
Std. Error
1
501.14
5.37
2
508.71
5.10
3
513.50
5.66
4
509.14
7.12
RAVE
1
506.67
4.10
2
513.04
3.90
3
515.75
4.32
4
519.08
5.44
th
th
* Time 1= 6 grade pretest’ Time 2= 6 grade posttest; Time 3= 7th grade pretest; Time
4= 7th grade posttest
7
Estimated Marginal Means for 12 Words
37
35
33
Control
31
RAVE
29
27
25
1
2
3
4
Time
Figure 1. Estimated marginal means for scores on the 12 sixth grade words tested at four
time points (pre and post 6th, pre and post 7th) as a function of time and treatment for
students who were in the RAVE group in sixth grade (n=37) and the control group in
sixth grade (n=52) regardless of condition in seventh grade.
8
1000
980
960
940
920
900
880
860
840
820
Control group
RAVE group
Figure 2. Reaction times on average for control and RAVE students on RAVE and
matched words.
9
525
520
Gates ESS Adjusted Means
515
510
505
Control
500
RAVE
495
490
485
480
475
1
2
3
4
Time
Figure 3. Estimated marginal means for Gates extended scales scores as a function of
time and treatment for students who were in RAVE (n=24) or control conditions (n=14)
for two consecutive years.
10
References
American Institutes for Research. (2006). Lessons and recommendations from the
Alabama Reading Initiative: Sustaining focus on secondary reading.
Washington, DC.
Apthorp, H.S., Randel, B., Cherasaro, T., Clark, T., McKeown, M.G., & Beck, I.L.
(2012). Effects of a supplemental vocabulary program on word knowledge and
passage comprehension. Journal of Research on Educational Effectiveness, 5(2),
160-188).
Bacon, D.L. (2005). Word Master: Seeing and Using Words. Level 7. Oakdale, NY:
EDCON Publishing Group.
Baumann, J.F., Kame’enui, E.J., & Ash, G.E. (2003). Research on vocabulary
instruction: Voltaire redux. In J. Flood, D. Lapp, J.R. Squire, & J.M. Jensen
(Eds.), Handbook of research on teaching the English language arts (pp. 752785). Mahwah NJ: Erlbaum & Associates.
Beck, I.L., Perfetti, C.A., & McKeown, M.G. (1982). Effects of long-term vocabulary
instruction on lexical access and reading comprehension. Journal of Educational
Psychology, 74(4), 506-521.
Bolger, D.J., Balass, M., Landen, E., & Perfetti, C.A. (2008). Contextual variation and
definitions in learning the meaning of words. Discourse Processes, 45(2), 122159.
Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34(2), 213-238.
Crosson, A.C., McKeown, M.G., Beck, I.L., & Ward, A.K. (July 2012). Cloe Assessment
of Partial Word Knowledge: Development of an Innovative Assessment of
Vocabulary Depth. Poster presented at the Nineteenth Annual Conference of
Society for Scientific Studies of Reading, Montreal.
Elleman, A.M., Lindo, E.J., Morphy, P., & Compton, D.L. (2009). The Impact of
Vocabulary Instruction on Passage-Level Comprehension of School-Age
Children: A Meta-Analysis'. Journal of Research on Educational Effectiveness,
2,(1), 1- 44.
Heller, R., & Greenleaf, C.L. (2007). Literacy instruction in the content areas: Getting
to the core of middle and high school improvement. Washington, DC: Alliance
for Excellent Education.
Jenkins, J.R., Pany, D., & Schreck, J. (1978). Vocabulary and reading comprehension:
Instructional effects (Tech. Rep. No. 100). Urbana: University of Illinois, Center
for the Study of Reading. (ERIC Document Reproduction Service No.
ED160999).
11
Kame’enui, E.J., Carnine, D.W., & Freschi, R. (1982). Effects of text construction and
instructional procedures for teaching word meanings on comprehension and recall.
Reading Research Quarterly, 17(3), 367-388.
MacGinitie, W.H., MacGinitie, R.K., Maria, K., Dreyer, L.G., & Hughes, K.E. (2007).
Gates-MacGinitie Reading Tests (4th Edition). Rolling Meadows, IL: Riverside
Publishing.
McKeown, M. G., Crosson, A. C., Artz, N. J., Sandora, C., & Beck, I. L. (2013). In the
Media: Expanding Students’ Experience with Academic Vocabulary. The
Reading Teacher, 67(1), 45-53.
McKeown, M.G., Crosson, A.C., Beck, I.L., Sandora, C., Artz, N.J., Nelson, J., & Zhang,
C. (April 2013). An Academic Vocabulary Intervention to Enhance Word
Knowledge and Comprehension for Middle School Students. Paper presented at
the American Educational Research Association Annual Meeting, San Francisco.
Nagy, W.E. & Scott, J.A. (2000). Vocabulary processes. In M.L. Kamil, P.B. Mosenthal,
P. David Pearson, & R. Barr (Eds.), Handbook of reading research, Volume III,
pp. 69-284. Mahwah, NJ: Erlbaum.
National Center for Education Statistics, (2011). National Assessment of Educational
Progress (NAEP), (NCES 2012-457). Institute of Education Sciences, U.S.
Department of Education, Washington, D.C.
National Center for Education Statistics, (2012). Vocabulary Results from the 2009 and
2011 NAEP Reading Assessments. (NCES 2013-452). Institute of Education
Sciences, U.S. Department of Education, Washington, D.C.
National Reading Panel. (2000). Teaching children to read: An evidence-based
assessment of the scientific literature on reading and its implications for reading
instruction. Washington, D.C.: National Institutes of Health. (NIH Pub. No. 004754).
Perfetti, C.A. (2007). Reading ability: Lexical quality to comprehension. Scientific
Studies of Reading, 11(4), 357-383.
Perfetti, C.A., & Hart, L. (2002). The lexical quality hypothesis. In L. Verhoeven, C.
Elbro, & P. Reitsma (Eds.), Precursors of functional literacy. (Published as Vol.
11 of the series Studies in Written Language and Literacy). Philadelphia, PA:
John Benjamins Publishing Company.
Perfetti, C., & Stafura, J. (2014). Word knowledge in a theory of reading comprehension.
Scientific Studies of Reading, 18(1), 22-37.
Reichle, E.D., & Perfetti, C.A. (2003). Morphology in word identification: A wordexperience model that accounts for morpheme frequency effects. Scientific
Studies of Reading, 7(1), 219-238.
12
Richter, T., Isberner, M., Naumann, J., & Neeb, Y. (2013). Lexical quality and reading
comprehension in primary school children. Scientific Studies of Reading, 17(6),
415-434.
Snow, C. (2013). Discussion to support vocabulary and reading comprehension in ELL
classrooms.
wg.serpmedia.org/download_files_misc/NABE%20Orlando%20Feb%202013.pdf
Stahl, S.A., & Fairbanks, M.M. (1986). The effects of vocabulary instruction: A modelbased meta-analysis. Review of Educational Research, 56, 72–110.
13