Using the Observation Survey
as Your Assessment Tool:
A Rationale
Francisco X. Gómez-Bellengé
The Ohio State University
Assessment & RTI
• In a recent article, Richard Allington (2006) said
he was skeptical of RTI if RTI meant merely the
juxtaposition of commercial programs across the
three tiers;
• This could lead to “curricular incoherence”
• In an article titled “The Congruence of
Classroom and Remedial Reading Instruction”
(Elementary School Journal, 1985), Richard
Allington, Peter Johnston & Peter Afflerbach
argued that any intervention had to build
primarily around students’ instructional needs.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Allington cont’d
• For instance, if students struggled
because they did not use effective
comprehension strategies, then expanding
comprehension strategy instruction in the
intervention lessons would seem a good
idea.
• If students struggled because they did not
self-monitor, then that would become one
focus of the intervention lessons.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Allington 3
• Designing effective intervention requires
that a student’s specific instructional
needs be identified.
• Because curricular coherence is needed
across the three tiers, meeting students’
specific instructional needs can be
achieved in part with a common
assessment framework.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Assessment in K-2
 In K-2, this assessment framework could
be the Observation Survey
• It is a not for profit assessment
• Developed by Marie Clay, the founder of
Reading Recovery
• It is comprised of six tasks that are real
world tasks
• Administered individually by teachers
Using OS for AssessmentFrancisco X. Gomez Bellenge
Observation Survey
• Requires teacher training and an
interpretation of scores
• Recent U.S. grade 1 norms available (fall,
mid-year, year-end)
• Recent New Zealand age norms available
• The Observation Survey is a criterion
referenced assessment
• Measures are not normally distributed
Using OS for AssessmentFrancisco X. Gomez Bellenge
The Six Measures of An
Observation Survey of Early
Literacy Achievement
Text Reading Level
Scoring: text levels 00-02 = readiness; 3-8 = pre primer; 912 = primer; 14-16 = end of grade 1; 18-20 = grade 2;
22-24 = grade 3; 26-30 = grades 4-6.
Purpose: to determine an appropriate level of text difficulty
and to record, using a running record, what the child
does when reading continuous text
Task: to read texts representing a gradient of difficulty until
the highest text level with 90% accuracy or better is
determined with teacher recording text reading
behaviors during the oral reading task; texts were
drawn from established basal systems and have, over
the years, proved to be a stable measure of reading
performance.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Letter Identification
• Scoring: maximum score = 54
• Purpose: to find out what letters the child
knows and the preferred mode of
identification
• Task: to identify upper and lower case
letters and conventional print forms of ‘a’
and ‘g’
Using OS for AssessmentFrancisco X. Gomez Bellenge
Ohio Word Test
• Scoring: maximum score = 20
• Purpose: to find out whether the child is
building up a personal resource of reading
vocabulary
• Task: to read a list of 20 high-frequency
words
Using OS for AssessmentFrancisco X. Gomez Bellenge
Concepts About Print
• Scoring: maximum score = 24
• Purpose: to find out what the child has
learned about the way spoken language is
put into print
• Task: to perform a variety of tasks during
book reading by the teacher
Using OS for AssessmentFrancisco X. Gomez Bellenge
Writing Vocabulary
• Scoring: count of words in a 10 minute
time limit
• Purpose: to find out whether the child is
building a personal resource of words that
are known and that can be written in every
detail
• Task: to write all known words in 10
minutes
Using OS for AssessmentFrancisco X. Gomez Bellenge
Hearing and Recording Sounds
in Words
• Scoring: maximum score = 37
• Purpose: to assess phonemic awareness
by determining how well the child
represents the sounds of letters and
clusters of letters in graphic form
• Task: to write a dictated sentence, with
credit for every sound correctly
represented
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & CBM
• The Observation Survey is similar to
Curriculum-Based Measurement:
– Well-designed standardized assessments
– Established reliability and validity
– Designed to inform instruction
– Not for profit
– Not normally distributed
– Grounded in substantial scientific peer-review
research
Using OS for AssessmentFrancisco X. Gomez Bellenge
DIBELS
• Developed by a group at the University of
Oregon
• Also teacher administered individually
• Also meant to inform instructional
decision-making but not inform instruction
itself, e.g.,
• nonsense word and speeded tasks are not
related to real-world classroom tasks
Using OS for AssessmentFrancisco X. Gomez Bellenge
DIBELS Measures
Using OS for AssessmentFrancisco X. Gomez Bellenge
DIBELS 2
• Minimum training involved
• No interpretation of scores-decision
criteria set in advance
• Developers of DIBELS + writers of
Reading First legislation + advisers to
writers of state grants + grant reviewers=
Conflict of interest, Inspector General Report
Using OS for AssessmentFrancisco X. Gomez Bellenge
Decision criteria
Using OS for AssessmentFrancisco X. Gomez Bellenge
Assessment and Identification of
First-Grade Students at Risk:
Correlating the Dynamic
Indicators of Basic Early Literacy
Skills and An Observation Survey
of Early Literacy Achievement
Correlating the OS & DIBELS
• The Iowa Test of Basic Skills is one of the
most respected standardized assessments
• DIBELS creators demonstrated worth of
their assessment by comparing to ITBS
• CBM developers compare their
assessments to ITBS
• OS researchers have also looked at ITBS
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & ITBS
Study found that OS & ITBS
• Both identify same students as low
readers
• Measures correlate moderately to highly
• Progress for Reading Recovery students
evident on both the Observation Survey &
the Iowa Test of Basic Skills
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & DIBELS: Theory of Learning
• DIBELS and OS assessment are based on
alternate theoretical constructs and ways
of defining the processes of early literacy
acquisition
• With DIBELS, fluency and accuracy are
proxies for degree of learning
• Underlying theory is that skills acquisition
equates with emergence of reading
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & RR Theory of Learning
• The OS was designed to assess a range of
items and behaviors accounting for the
complexity of learning how to read in first grade
• During the earliest stages of literacy acquisition
in particular, speed of response may mask
important developmental markers of students’
progress.
• Emphasis is placed on strategic behavior used
by beginning literacy learners
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & DIBELS
• The measures correlate reasonably well,
especially when you would expect them to:
OS Measure
DIBELS Measure
Letter Identification
Letter Naming Fluency
Text Reading Level
Oral Reading Fluency
Hearing & Recording
Sounds in Words
Nonsense Word
Fluency & Phoneme
Segmentation Fluency
Using OS for AssessmentFrancisco X. Gomez Bellenge
OS & DIBELS/2
• For Reading Recovery students, both the
OS & DIBELS show progress during the
intervention
• Big difference: Only ½ of students
identified as at-risk by OS & Reading
Recovery are identified as at-risk using
DIBELS criteria for “needs substantial
intervention” classification.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Why the discrepancy
• Students who do well on speeded tasks of
specific skills may not have necessarily
developed the ability to process text and
extract meaning from it.
• These students will do well on DIBELS &
on some tasks of the OS
• If not identified as at-risk early, problems
may develop later (e.g., grade 3)
Using OS for AssessmentFrancisco X. Gomez Bellenge
Scott Paris, RRQ, 2005
Theories about reading have neglected basic
differences in the developmental trajectories of
skills related to reading. … that some reading
skills, such as learning the letters of the
alphabet, are constrained to small sets of
knowledge that are mastered in relatively brief
periods of development. In contrast, other skills,
such as vocabulary, are unconstrained by the
knowledge to be acquired or the duration of
learning.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Early Literacy Development
• Scott Paris argues that when looking at a
specific skill, (e.g. letter identification) early
in a child’s life there will be no progress on
that measure, followed by a period of rapid
progress, after which no progress occurs
because the skills has been acquired.
Using OS for AssessmentFrancisco X. Gomez Bellenge
100%
0%
Birth
age 5
Using OS for AssessmentFrancisco X. Gomez Bellenge
age 7
Early Literacy Development/2
• Before and after the skill of task is being
learned, students either know it or don’t
know it; there is little variation and
therefore, the distribution of scores would
be non-normal; this is due to the nature of
the task, not a flaw in its assessment.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Non-normal distributions
Fall Text Reading Level
Spring Letter Identification
Using OS for AssessmentFrancisco X. Gomez Bellenge
DIBELS & Normal Distribution
• DIBELS forces a normal distribution of
scores even when it should no be. This is
done by either speeding the task or
creating a task that is not the actual
reading task (e.g., nonsense word fluency)
• Normal distribution of scores is sometimes
confused with being standardized, reliable,
or valid.
Using OS for AssessmentFrancisco X. Gomez Bellenge
Normal Distribution
• Normal distribution is just that- it does not
mean a task is reliable, valid or
standardized
• Normal distribution is required under
conventional statistics to measure effect of
a treatment-more modern techniques
obviate this need
Using OS for AssessmentFrancisco X. Gomez Bellenge
Letter Identification Task for
Adults:
What’s a reasonable time per
letter?
L
J
I
T
Using OS for AssessmentFrancisco X. Gomez Bellenge
L J I T
Using OS for AssessmentFrancisco X. Gomez Bellenge
And another thing …
• Scott Paris’s article, while brilliant, misses
an important point: While the acquisition of
specific skills does not occur evenly over
time, the emergence of literacy, of learning
to extract meaning from text, is largely
unobserved and not directly depended on
discrete skills acquisition
Using OS for AssessmentFrancisco X. Gomez Bellenge
Assessment
• Any assessment system must
accommodate the complexity of the
process of literacy acquisition and the
change that occurs over time.
Using OS for AssessmentFrancisco X. Gomez Bellenge
RTI & Reading Recovery
Statistics &
Conceptual Framework
Using OS for AssessmentFrancisco X. Gomez Bellenge
Using OS for AssessmentFrancisco X. Gomez Bellenge
Reading Recovery National 05-06
Using OS for AssessmentFrancisco X. Gomez Bellenge
Parting Thoughts
• The Observation Survey is a valid and
reliable assessment tool
• It is well suited to be used within an RTI
framework
• DIBELS is not mandated; selection has
been severely criticized in studies
• You can’t buy a coherent RTI off the shelf
• Literacy & Sped folks need to talk to each
other
Using OS for AssessmentFrancisco X. Gomez Bellenge