The Problem of Comprehension:
A Potential Barrier to Reading CurriculumBased Measurement as a Progress
Monitoring and Screening Instrument
Mark R. Shinn, Ph.D.
Professor and School Psychology Program
National Louis University, Skokie, IL
markshinn@icloud.com
http://markshinn.org
February 21st, 2013
My Area of Expertise
Editor and
Contributor to
2 Major Texts
on CBM
1 of 6 members of Technical
Review Panel, National Center
for Student Progress
Monitoring, USDE/OSEP
2003-2007
Author of More than 75
Refereed Journal Articles and
Book Chapters on the Topic of
CBM, Progress Monitoring,
and Screening
Disclosure
Disclosure
Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for
their AIMSweb product that provides CBM assessment materials and organizes
and report the information from 3 tiers, including RTI. He provides technical
support and training.
Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for
their Vmath product, a remedial mathematics intervention but has no financial
interests. He helped them develop their progress monitoring system.
Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their
Jamestown Reading Navigator (JRN) product and receives royalties.He helped
them develop their progress monitoring system.
Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the
CORE (Consortium on Reaching Excellence) and receives a stipend for participation.
He provides training and product development advice.
Learning to Read is Critical for School (and
Life) Success
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based
Measurement (R-CBM) is Among our Most Powerful Tools!
And the Number 1 Most
Powerful TEACHING Variable
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
BUT...
What about comprehension?
What about “word callers?”
I know this kid who...reads “fluently” but
doesn’t understand a THING they
read...
Presentation is Based on the Following
White Paper in PRODUCTION
Will Be
Available in
•pdf format
•iBook format
Shinn, M.R. (2013). The problem of comprehension: A
potential barrier to Reading Curriculum-Based
Measurement (R-CBM) as a progress monitoring
and screening instrument. Minneapolis, MN:
Pearson Assessment.
A “glossy” and official
Pearson version will be
finished soon and sent to
you.
References on CBM Criterion Related Validity
Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling
reading disability. Topics in Learning and Learning Disability, 2, 53-59.
Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading
competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239-256.
Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures.
Remedial and Special Education, 9, 20-28.
Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers: An investigation of the accuracy of
teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32,
228-240.
Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on
curriculum-based measurement in reading. The Journal of Special Education, 41, 85-120.
Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A
confirmatory analysis of its relation to reading. School Psychology Review, 21, 458-478.
Accessing Reading Materials
markshinn.org
1. Click on the Downloads for
Professionals Icon
2. Click on the Presentations and
Handouts Folder
3.Click on AIMSweb Webinar R-CBM
and the Problem of Comprehension
(Sponsored by Pearson) 2013 Folder
10
Big Ideas
1.
When students read passages aloud for 1 minute, we are obtaining a holistic assessment of
general reading skills.
2.
A strong empirical relation has been demonstrated between R-CBM and measures of general
reading and reading comprehension (RC).
3.
Persuasive empirical data are not always persuasive of teacher opinion.
4.
General reading skills are necessary, but not sufficient for comprehension.
5.
RC is not a single “thing." Judgments are based on:
(a) What a student is expected to read.
(b) How the student will be assessed.
6.
RC is influenced by a number of important variables, including
(a) General reading ability
(b) Language, especially vocabulary and familiarity with cultural idioms
(c) Knowledge, especially content knowledge and knowledge of the world
(d) Metacognitive strategies, including motivation and interest
2 Other Big Ideas We Can’t Discuss Today
1.
If educators are concerned about specific student’s RC, then we must have a systematic
assessment process to address these concerns, starting with ruling out general reading
skill deficits, but also including interviews, observations, and diagnostic assessment.
See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In
A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda,
MD: National Association of School Psychologists.
•
If students have low general reading skills, RC strategy intervention should be time
limited and the use of such strategies should be actively taught, expected, supported,
and rewarded in Tier 1, general education and/or content area classes.
See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension
strategies. American Educator, Winter, 39-50.
2 Other Big Ideas We Can’t Discuss Today
1.
If educators are concerned about specific student’s RC, then we must have a systematic
assessment process to address these concerns, starting with ruling out general reading
skill deficits, but also including interviews, observations, and diagnostic assessment.
See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In
A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda,
MD: National Association of School Psychologists.
•
If students have low general reading skills, RC strategy intervention should be time
limited and the use of such strategies should be actively taught, expected, supported,
and rewarded in Tier 1, general education and/or content area classes.
Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies.
American Educator, Winter, 39-50.
The Problem of “Fluency”
Experienced--and Often Powerful--Teachers Push Back
When R-CBM is (Inappropriately) Interpreted and
Communicated as “Fluency”
How Did This Happen?
– Errors in Judgment(s), Confusing Constructs with
Behaviors Tested
– Tendency toward Reductionism
An Example of Reading Curriculum-Based
Measurement (R-CBM)
It was a pretty good composition. I felt proud
knowing it was the best one at my school.
After I’d read it five times, I was impatient to
start reading it out loud. I followed the book’s
directions again. First I read the composition
out loud without trying to sound impressive,
just to hear what the words sounded like.
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Billy, 4th Grader
Oversimplifying the NRP
Phonemic Awareness
Phonemic Awareness
Phonics
Alphabetic
Understanding
Fluency
Fluency
Vocabulary
Comprehension
Vocabulary
http://www.nationalreadingpanel.org/
Comprehension
Strands are NOT
ALL
TheseBOXESSkills
Low Scores “in the
Box” Led to
General Reading
TEACHING the
Skill
Things in the Box
SEPARATELY
Consequences of (Mis) Interpretation as
Fluency
Teacher Push Back and Shallow Intervention
If R-CBM Interpreted As
Fluency
If Interpreted R-CBM(Correctly) As
General Reading Ability
Goal is to Read Fast
Goal is to Read WELL
Interventions Emphasize Speed
Interventions Emphasize Quality
Interventions are “Slices” or
Bandaids
Interventions are Integrated or
Bandages
Research Evidence of “FluencyDriven” Intervention or “Reading
Faster”?
Implementing repeated reading and wide reading interventions
without more formative intervention is not likely to be valuable
(p. 9)
Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school
students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10.
Our results indicate that repeated reading does not qualify as
an evidence-based or promising practice for students with or
at risk for learning disabilities
(p. 276)
Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading interventions for
students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.
A Student with a Significant Reading
Discrepancy
Potentially Severe Educational Need
Poor Reading Quality as Well as
Quantity
Reads Accurately?
Reads Efficiently with Automaticity?
Reads with Expression (Prosody)?
Effective Strategy for Unknown Words?
Errors Distort or Preserve Meaning?
Self Corrects Errors (Comprehension Self-Monitoring)?
Adjusts Pace When Text Difficulty Changes?
A Poor High School Reader
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Grade 8 Material < 10th percentile at
beginning of Grade 8
55
It’s Not an Empirical Issue--The Evidence is Compelling
and Overwhelming
Comprehension Measure
Criterion Measure
R-CBM
SAT Word Study
SAT Comprehension
Question Answering
Recall
.80
0.91
SAT Word Study
.66
SAT Comprehension
.82
SAT Word Study
.58
SAT Comprehension
Cloze
Correlation
SAT Word Study
SAT Comprehension
0.70
.71
0.72
Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special
Education, 9, 20-28.
It’s Not an Empirical Issue--The Evidence is Compelling
and Overwhelming
.88
.90
.85
.71
.72
Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory
analysis of its relation to reading. School Psychology Review, 21(3), 458-478.
But, I KNOW and HAVE SEEN Word
Callers!!!!!!!
The Research
The idea of word callers has gained popularity despite a lack of
evidence that applies “to an appreciable number of poor
readers” (Stanovich, 1986, p. 372).
Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual
differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.
But, I KNOW and HAVE SEEN Word
Callers!!!!!!!
The Practice
31 of 75 Third Grade Teachers responded to a postcard put in
their box asking them
“if you teach a third-grade student who can read fluently, but has
difficulty comprehending text.”
41%!
Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers'
judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.
But, I KNOW and HAVE SEEN Word
Callers!!!!!!!
Teachers Identified PAIRS of Kids in Their
Classrooms:
Word Callers (WC)
Read Fluently,
But Didn’t
Comprehend
Similarly “Fluent” Peers
Read Equally as Fluently
(Well) As WC,
But DID COMPREHEND
Teachers Predicted BOTH Pair’s R-CBM, Maze, and
Oral Comprehension Question Answering
Skills--THEN Both Pairs WERE TESTED
Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers'
judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.
The “Word Caller”
Teachers Were Partially Right!
Peers
Peers
WC
Almost 2
Standard
Deviations Lower
WCs “Comprehended” Less Well
WC
Almost 1.3
Standard
Deviations Lower
But WRC Was ALSO MUCH Lower
So Where Do This Leave Us...
1. Reading Comprehension is Complex and We Tend
to Oversimplify and Overgeneralize
•
Reading is Necessary, But Not Sufficient for
Understanding
•
Reading Comprehension Comes From Reading,
But is MORE Than Reading
“Comprehension” is Oversimplified
A-Rod hit the cover off of the ball, but
ended the game with a 6-4-3 double
play.
Inferential:
Factual:
Who is A-Rod?
What does “6” mean?
Why would people from
Beantown celebrate this?
Why would this event mean
different things in June than
October?
Another Common Example
The procedure is actually quite simple. First, you arrange things into different
groups. Of course, one pile may be sufficient depending on how much there is to
do. If you have to go somewhere else due to lack of facilities that is the next step,
otherwise you are pretty well set. It is important not to overdo things. That is, it is
better to do too few things at once than too many. In the short run, this may not
seem important, but complications can easily arise. A mistake can be expensive as
well. After the procedure is completed, one arranges the materials into different
groups again. Then they can be put into their appropriate places. Eventually, they
will be used once more and the whole cycle will have to be repeated. At first, the
whole procedure will seem complicated. Soon, however, it will become just another
fact of life. It is difficult to foresee any end the necessity for this task in the
173-word paragraph with 14 sentences of words that are very
4 one
letters)
and rated
immediateshort
future,(about
but then
can never
tell. as late Grade 6 by FleschKincaid
Let Me Add Some Information
How to Wash Clothes
The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile
may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of
facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is,
it is better to do too few things at once than too many. In the short run, this may not seem important, but
complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one
arranges the materials into different groups again. Then they can be put into their appropriate places.
Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole
procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to
foresee any end the necessity for this task in the immediate future, but then one can never tell.
The Bigger Deficits
Here
• Life Experience
• Content Knowledge
• Activation of Prior
Knowledge
• Knowledge about
Texts
And
Here
Languag
e
• Oral Language Skills
• Knowledge of Language
Structures
• Vocabulary
• Cultural Influences
Oral Reading is the
EASIEST
to Measure-- Fluency*
Reading
Knowledge
Let’s
Get This DownFor
andSome, the
And the MOST
Comprehension
The
Easiest
Thing
The Longer
Unmotivated Here Add MORE ToolsHardest
ThingIt
To Teach
Takes...
They’ll
Ever Do
• Motivation &
We Refer to It as
General Reading Skills
Engagement
• Active Reading
Strategies
• Monitoring Strategies
• Fix-Up Strategies
Metacognition
• Prosody
• Automaticity/Rate
• Accuracy
• Decoding
• Phonemic Awareness
*modified slightly from presentations by Joe Torgesen, Ph.D. CoDirector, Florida Center for Reading Research; www.fcrr.org
Comes From Reading, But Is Not
Reading
Reading Early, Well, and Wide (EWW) is Critical for Reading
Comprehension.
See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color become
successful readers. New York, NY: Scholastic.
Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, 8-15.
Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from research. In M. R.
Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI
(pp. 469-500). Bethesda, MD: National Association of School Psychologists.
Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English language
learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum Development.
http://www.youtube.com/watch?feature=player_embedded&v=RiP-ijdxqEc
Comes From Reading, But Is Not
Reading
Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic
Comprehension Strategy Instruction, Especially in Navigating
Expository/Informational Text and Vocabulary, Learning the “Right Words” the
“Right Way.”
This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem!
See:
Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary instruction. In
Scholastic (Ed.), Read About: . New York, NY
Kamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent Literacy:
Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education
Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education.
Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective
adolescent literacy programs. Education and Treatment of Children, 36, 161-184.
Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S., . . . Lesaux, N. (2007). Academic
literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC
Research Corporation, Center on Instruction.
We Need to Expand Our Assessment
Repertoire
We Need to Add a Variety of Ways to Authentically Assess Comprehension
in the Materials We Expect Students to be Able to Read and UNDERSTAND
See:
Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes
(Eds.), Best practices in school psychology IV (pp. 671-698). Bethesda, MD: National Association of School
Psychologists.
Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA:
Wadsworth Publishing Company.
Big Ideas
1.
When students read passages aloud for 1 minute, we are obtaining a holistic assessment of
general reading skills.
2.
A strong empirical relation has been demonstrated between R-CBM and measures of general
reading and reading comprehension (RC).
3.
Persuasive empirical data are not always persuasive of teacher opinion.
4.
General reading skills are necessary, but not sufficient for comprehension.
5.
RC is not a single “thing." Judgments are based on:
(a) What a student is expected to read.
(b) How the student will be assessed.
6.
RC is influenced by a number of important variables, including
(a) General reading ability
(b) Language, especially vocabulary and familiarity with cultural idioms
(c) Knowledge, especially content knowledge and knowledge of the world
(d) Metacognitive strategies, including motivation and interest