Overview
• Overview, history, and purpose of AIMSweb and
Curriculum-Based Measurement
• AIMSweb and RTI
• Administration, Scoring, and Interpretation
– Oral Reading Fluency
– MAZE
• Materials
• Data System
Overview of
Curriculum-Based Measurement (CBM)
and AIMSweb®
Mark R. Shinn, Ph.D.
Michelle M. Shinn, Ph.D.
Lisa A. Langell, M.A., S.Psy.S.
Evaluations to Inform Teaching—
Summative & Formative Assessment
Summative Assessment: Culmination measure. Mastery assessment.
Assessment after instruction. Pass/fail type assessments which summarize the
knowledge students learn. Typical summative assessments include:
• End of chapter tests
• High-stakes tests (e.g., State assessments)
• GRE, ACT, SAT, GMAT, etc. tests
• Driver’s license test
• Final Exams.
Formative Assessment: Process of assessing student achievement frequently
during instruction to determine whether an instructional program is effective for
individual students.
Informs:
• When students are progressing, continue using your instructional
programs.
• When tests show that students are not progressing, you can change
your instructional programs in meaningful ways.
Summative & Formative Assessment
Summative Assessment:
Characterized as assessment of learning.
Formative Assessment: Characterized as
assessment for learning.
(Citation: http://en.wikipedia.org/wiki/Summative_assessment)
Summative assessment tells
you what happened.
Formative assessment tells you
what’s happening.
Evaluations to Inform Teaching—
Diagnostic Assessment
Diagnostic Assessments: Measures that indicate specific skill strengths
and those areas needing improvement. Results may indicate skill areas
needing intervention/instruction. Programming may then address students’
needs.
Examples:
• Criterion-referenced assessments
• Cognitive assessments
• Rating scales
• Norm-referenced, standardized
assessments
• Tests may be based on the assessment of
cognitive skills, academic skills,
behavior, health, social-emotional wellbeing,
etc.
So, Where Does
Curriculum-Based Measurement (CBM) fit?
•
Summative?
•
Formative?
•
Diagnostic?
What is Curriculum-Based
Measurement (CBM)?
•
CBM is a form of Curriculum-Based Assessment (CBA).
•
CBM is the method of monitoring student progress through direct,
continuous assessment of basic skills.
•
CBM is used to assess skills such reading, spelling, mathematics, and written
language.
•
CBM probes require about 1 to 4 minutes to complete, depending on the skill
being measured.
•
Student performance is scored for speed and accuracy to determine
proficiency.
•
Because CBM probes are quick to administer and simple to score, they can be
given frequently to provide continuous progress data.
•
The results are charted to provide for timely evaluation based on hard data.
Origins of CBM
•
CBM was initially developed more than 20 years ago by Stanley Deno
and others at the University of Minnesota Institute for Research on
Learning Disabilities to develop a reliable and valid measurement system
for evaluating basic skills growth
•
CBM is supported by 30 years of school-based research
•
CBM is endorsed by the United States Department of Education as a
method for assessing student progress.
•
Starting in the area of reading, researchers have expanded to
investigate additional academic areas over the years. This includes
in-depth research and ultimately the publication of additional measures
in literacy, mathematics, and written language
•
Supporting documentation can be found in 100’s of articles,
chapters, and books available within the professional literature
describing the use of CBM to make a variety of important educational
decisions
Advantages of CBM
•
Direct measure of student performance
Deno, S.L. (1985). Curriculum-based measurement: the emerging alternative. Exceptional Children. 52(3):219-32.
•
Correlates strongly with “best practices” for instruction and
assessment
•
Correlates strongly with research-supported methods for
assessment and intervention
•
Focus is on repeated measures of performance
(This cannot be done with most norm-referenced and standardized
tests due to practice effect or limited forms.)
Common Characteristics of
General Outcome Measures
CBM involves the same kind of evaluation technology as other
professions.
Powerful measures that are:
• Simple
• Accurate
• Efficient indicators of performance that guide and inform a
variety of decisions
• Generalizable thermometer that allows for reliable, valid, cross
comparisons of data
General Outcome Measures (GOMs)
from Other Fields
Medicine measures height, weight, temperature,
and/or blood pressure.
Department of Labor measures the
Consumer Price Index.
Wall Street measures the Dow-Jones Industrial Average.
Companies report earnings per share.
McDonald’s® measures how many hamburgers they sell.
CBM is Used for Scientific Reasons
Based on Evidence
•
Reliable and valid indicator of student achievement
•
Simple, efficient, and of short duration to facilitate frequent
administration by teachers
•
Provides assessment information that helps teachers plan
better instruction
•
Sensitive to improvement of students’ achievement over time
•
Easily understood by teachers and parents
•
Improves achievement when used to monitor progress
Three-Tiered Assessment Model
3
2
1
Tier 2:
STRATEGIC MONITOR
Monthly monitoring for students
who are mild to moderate risk
for failure
Tier 3:
PROGRESS
MONITOR
Intensive monitoring
towards specific goals
for students at
significant risk for
failure
Tier 1:
BENCHMARK
Universal Screening for all students
Measures Currently Available via AIMSweb®:
•
Early Literacy
[K-1 benchmark, Progress Monitor (PM) any age]
– Letter Naming Fluency
– Letter sound fluency
– Phonemic Segmentation Fluency
– Nonsense Word Fluency
•
Early Numeracy (K-1 benchmark, PM any age)
– Oral Counting
– Number identification
– Quantity discrimination
– Missing number
•
•
•
•
•
•
Oral Reading (K-8, PM any age)
MAZE (Reading comprehension); (1-8, PM any age)
Math Computation (1-6, PM any age)
Math Facts (PM any age)
Spelling (1-8, PM any age)
Written Expression (1-8, PM any age)
All students in an academic
curriculum are
“benchmarked” three times
per year across any/all of
these assessment areas.
Administration and Scoring of
READING-CURRICULUM-BASED
MEASUREMENT (R-CBM)
for Use in General Outcome Measurement
Mark R. Shinn, Ph.D.
Michelle M. Shinn, Ph.D.
Reading - Curriculum-Based Measurement (R-CBM)
Students read aloud for 1 minute from Edformation’s Standard
Reading Assessment Passages of meaningful, connected text.
Number of words read correct and number of errors are counted.
Reported as WRC/errors
Benefits of Using Edformation’s Standard Reading Assessment
Passages
Are written to represent general curriculum or be “curriculum
independent.”
Allow decision making about reading growth, regardless of betweenschool, between-school-district, between-teacher differences in
reading curriculum.
Are graded to be of equal difficulty.
Have numerous alternate forms for testing over time without
practice effects.
Administration and Scoring of R-CBM
What Examiners Need To Do…
•
Before Testing students
•
While Testing students
•
After Testing students
Things you Need Before Testing
1. Standard Reading
Assessment Passage
Student Copy:
• No numbers
• Between 250-300
words (exception: 1st
grade)
• An informative first
sentence
• Same font style and
size
• Text without pictures
• Obtain from your
LAM
Things you Need Before Testing
2. Standard Reading
Assessment Passage
Examiner Copy:
•Pre-numbered so they
can be scored quickly
and immediately.
•Obtain from your LAM.
3. Tier 1 (Benchmark) R-CBM Probes:
1.
2.
3.
AIMSweb Manager provides
staff with copies of three gradelevel probes (teacher and
student copies).
FALL (Sept.): Staff administer
three, grade-level probes to
each student.
Report median score.
WINTER (Jan.): Repeat
administration of same three
probes to each student.
Report median score.
FALL (Sept): Grade 3
P01: It rained all day.
E
P02: Billy was sitting.
X
P03: Mama frog carried.
A
M
P
L
WINTER (Jan): Grade 3
P01: It rained all day.
P02: Billy was sitting.
P03: Mama frog carried.
E
4.
SPRING (May): Repeat
administration of same three
probes to each student.
Report median score.
SPRING (May): Grade 3
P01: It rained all day.
P02: Billy was sitting.
P03: Mama frog carried.
4. Additional Assessment Aids Needed Before Testing
A list of students to be assessed
Stop Watch (required—digital preferred)
Clipboard
Pencil Transparencies or paper copies of examiner passages
Dry Marker or Pencil
Wipe Cloth (for transparencies only)
Setting up Assessment Environment
Assessment environments are flexible and could include…
• A set-aside place in the classroom
• Reading station in the hall way
• Reading stations in the media center, cafeteria, gym, or
empty classrooms
• Computer lab, weight room, principal’s office, storage
closet, or wherever there’s a quiet space
Things You Need to do While Testing
Follow the standardized directions:
• R-CBM is a standardized test
• Administer the assessment with consistency
• Remember it’s about testing, not teaching
• Don’t teach or correct
• Don’t practice reading the passages
• Remember best, not fastest reading
• Sit across from, not beside student
R-CBM Standard Directions for
1 Minute Administration
1) Place the unnumbered copy in front of the student.
2) Place the numbered copy in front of you, but shielded so the student cannot
see what you record.
3) Say:
When I say ‘Begin,’ start reading aloud at the top of this page. Read
across the page (DEMONSTRATE BY POINTING). Try to read each
word. If you come to a word you don’t know, I will tell it to you. Be
sure to do your best reading. Are there any questions? (PAUSE)
4) Say “Begin” and start your stopwatch when the student says the first word. If
the student fails to say the first word of the passage after 3 seconds, tell them
the word, mark it as incorrect, then start your stopwatch.
5) Follow along on your copy. Put a slash ( / ) through words read incorrectly.
6) At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.”
7) Score and summarize by writing WRC/Errors
“Familiar” Shortened Directions
When students are assessed frequently and know the directions.
Say:
When I say ‘Begin,’ start reading aloud at the top of this page.
Things to do After Testing
Score immediately!
Determine WRC.
Put a slash (/) through incorrect words.
If doing multiple samples, organize your impressions of qualitative
features.
What is a Word Read Correctly?
Correctly pronounced words within context.
Self-corrected incorrect words within 3 seconds.
What is an Error?
Mispronunciation of the word
Substitutions
Omissions
3-Second pauses or struggles (examiner provides correct word)
What is not Incorrect? (Neither a WRC or an Error)
Repetitions
Dialect differences
Insertions (consider them qualitative errors)
Example
Juan finished reading after 1 minute at the 145th word, so he read 145
words total.
Juan also made 3 errors.
Therefore his WRC was 142 with 3 errors.
Reported as: 142/3.
R-CBM Activity:
Tallying Scores &
Calculating Median
Data: Get the MEDIAN score for student’s
3 passages: CORRECT WORDS
67 / 2
1 min.
85 / 8
1 min.
74 / 9
1 min.
Why use Median vs. Average?
Averages are susceptible to outliers when dealing with small number sets.
Median Score is a statistically more reliable number than average for R-CBM.
Data: Get the MEDIAN score for
student’s 3 passages: ERRORS
67 / 2
85 / 8
74 / 9
MEDIAN SCORE: 74/8
(Report in AIMSweb®)
NOTE: R-CBM is the only measure for which
the median score is calculated.
1 min.
1 min.
1 min.
Why use Median vs. Average?
Averages are susceptible to outliers when dealing with small number sets.
Median Score is a statistically more reliable number than average for R-CBM.
BENCHMARK DATA ACTIVITY
• Go to P. 2 of your trainee manual
• Calculate the Median Score for Hannah Berg
WRC
= Words Read Correct
ERRORS = Number of Errors
WRC/Errors
Example: 64/2
(All identifying information and scores are fictitious.)
54
(All identifying information and scores are fictitious.)
2
Practice Exercise 1: Let’s Score – P. 3 in trainee manual
Practice Exercise 1: Let’s Score
This student read 73 WRC/7 Errors
Practice Exercise 2: Let’s Score – P. 3 in trainee manual
Practice Exercise 4: Let’s Score
This student read 86 WRC/5 Errors
Paired Practice – Oral Reading
Fluency – P. 4 Trainee Manual
P. 4 in Trainee Booklet
First time: partner #1 answers, partner #2 scores
Second time: partner #2 answers, partner #1 scores
Administration and Scoring of
READING-MAZE (R-MAZE)
for Use in General Outcome Measurement
Power Point Authored by
Jillyan Kennedy
Based on Administration and Scoring of Reading R-MAZE for Use
with AIMSweb Training Workbook
By
Michelle M. Shinn, Ph.D.
Mark R. Shinn, Ph.D
R-MAZE is Used for Scientific Reasons
Based on Evidence
• It is a reliable and valid indicator of student achievement.
• It is simple, efficient, and of short duration to facilitate
frequent administration by teachers.
• It provides assessment information that helps teachers plan
better instruction.
• It is sensitive to the improvement of students’ achievement
over time.
• It is easily understood by teachers and parents.
• Improves achievement when used to monitor progress.
Curriculum Based Measurement Reading R-MAZE
CBM R-MAZE is designed to provide educators a more complete picture
of students’ reading skills, especially when comprehension problems are
suspected.
Area
CBM R-MAZE
Reading
Timing
3 minutes
Test Arrangements
Individual, Small
Group, or
Classroom Group
What is Scored?
# of Correct
Answers
Curriculum Based Measurement Reading R-MAZE (Continued)
• R-MAZE is a multiple-choice cloze task that students
complete while reading silently.
• The students are presented with 150-400 word passages.
• The first sentence is left intact.
• After the first sentence, every 7th word is replaced with three
word choices inside a parenthesis.
• The three choices consist of:
1) Near Distracter
2) Exact Match
3) Far Distracter
Sample Grade 4 R-MAZE Passage
Examples of R-MAZE
Administration and Scoring of CBM R-MAZE
What examiners need to do . . .
• Before testing students
• While testing students
• After testing students
Items Students Need Before Testing
What the students need
for testing:
• CBM R-MAZE
practice test
• Appropriate CBM
R-MAZE passages
• Pencils
Items Administrators Need Before Testing
What the tester uses
for testing:
• Stopwatch
• Appropriate
CBM R-MAZE
answer key
• Appropriate
standardized
directions
• List of students
to be tested.
Things You Need to do While Testing
Follow the standardized directions:
• Attach a cover sheet that includes the practice test so that students do
not begin the test right away.
• Do a simple practice test with younger students.
• Monitor to ensure students are circling answers instead of writing them.
• Be prepared to “Prorate” for students who may finish early.
• Try to avoid answering student questions.
• Adhere to the end of timing.
CBM R-MAZE Standard Directions
1)
2)
Pass R-MAZE tasks out to students. Have students write their names on the cover
sheet, so they do not start early. Make sure they do not turn the page until you tell
them to.
Say this to the student (s):
When I say ‘Begin’ I want you to silently read a story. You will have 3
minutes to read the story and complete the task. Listen carefully to
the directions. Some of the words in the story are replaced with a
group of 3 words. Your job is to circle the 1 word that makes the
most sense in the story. Only 1 word is correct.
3)
Decide if a practice test is needed. Say . . .
Let’s practice one together. Look at your first page. Read the first
sentence silently while I read it out loud: ‘The dog, apple, broke, ran
after the cat.’ The three choices are apple, broke, ran. ‘The dog apple
after the cat.’ That sentence does not make sense. ‘The dog broke
after the cat.’ That sentence does not make sense. ‘The dog ran after
the cat.’ That sentence does make sense, so circle the word ran.
(Make sure the students circle the word ran.)
CBM R-MAZE Standard Directions (Continued)
Let’s go to the next sentence. Read it silently while I read it out
loud. ‘The cat ran fast, green, for up the hill. The three choices
are fast, green, for up the hill. Which word is the correct word
for the sentence? (The students answer fast)
Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct
word fast. (Make sure students circle fast)
Silently read the next sentence and raise your hand when you
think you know the answer. (Make sure students know the correct
word. Read the sentence with the correct answer)
That’s right. ‘The dog barked at the cat’ is correct. Now what do
you do when you choose the correct word? (Students answer
‘Circle it’. Make sure the students understand the task)
That’s correct, you circle it. I think you’re ready to work on a
story on your own.
CBM R-MAZE Standard Directions (Continued)
4)
Start the testing by saying . . .
When I say ‘Begin’ turn to the first story and start reading silently.
When you come to a group of three words, circle the 1 word that
makes the most sense. Work as quickly as you can without
making mistakes. If you finish a/ the page/first side, turn the page
and keep working until I say ‘Stop’ or you are all done. Do you
have any questions?
5)
6)
7)
8)
9)
Then say, ‘Begin.’ Start your stopwatch.
Monitor students to make sure they understand that they are to circle only 1
word.
If a student finished before the time limit, collect the student’s R-MAZE task
and record the time on the student’s test booklet.
At the end of 3 minutes say: Stop. Put your pencils down. Please close
your booklet.
Collect the R-MAZE tasks.
CBM R-MAZE Familiar Directions
1)
After the students have put their names on the cover sheer, start the testing by
saying . . .
When I say ‘Begin’ turn to the first story and start reading silently.
When you come to a group of three words, circle the 1 word that
makes the most sense. Work as quickly as you can without
making mistakes. If you finish a/ the page/first side, turn the page
and keep working until I say ‘Stop’ or you are all done. Do you
have any questions?
2)
3)
4)
5)
6)
Then say, ‘Begin.’ Start your stopwatch.
Monitor students to make sure they understand that they are to circle only 1
word.
If a student finished before the time limit, collect the student’s R-MAZE task
and record the time on the student’s test booklet.
At the end of 3 minutes say: Stop. Put your pencils down. Please close
your booklet.
Collect the R-MAZE tasks.
Things to Do After Testing
•
•
•
Score immediately to ensure accurate results!
Determine the number of words (items) correct.
Use the answer key and put a slash (/) through incorrect
words.
CBM R-MAZE Scoring
What is correct?
The students circles the word that matches the correct word on the
scoring template.
What is incorrect?
An answer is considered an error if the student:
1) Circles an incorrect word
2) Omits word selections other than those the student
was unable to complete before the 3 minutes
expired
Making Scoring Efficient
1) Count the total number of items up to the last circled
word.
2) Compare the student answers to the correct answers on
the scoring template. Mark a slash [/] through incorrect
responses.
3) Subtract the number of incorrect answers from the total
number of items attempted.
4) Record the total number of correct answers on the cover
sheet followed by the total number of errors (e.g., 35/2).
Paired Practice : Let’s Score –
Trainee Manual P. 5
• P. 5 in Trainee Booklet
• First time: partner #1
answers, partner #2 gives
full directions (P. 18) and
administers passage on P.
22
• Second time: partner #2
answers, partner #1 gives
abbreviated directions (P.
20)
Accessing Your Materials
Two options:
- Print materials online
- Pull materials from folder at school and make copies
Accessing Materials Online
So
Accessing the Materials Folder
•
Your school will have one Benchmark package and one Progress Monitoring Package
•
Benchmarking Package contains:
•
Student Materials Folder for each grade level
• Contains the materials the student will have in front of them when testing for
each grade level (make one copy of each MAZE passage for each student
because they will be writing on them)
•
Teacher Materials Folder for each grade level
• Contains scoring booklet for each grade level (make one copy per student in
your class)
• Contains administration and scoring manual for each measure
• Contains MAZE answer key for benchmark passages (Grades 3-8)
•
Progress Monitoring Package contains:
•
One copy of each progress monitoring probe and answer key at each grade level
Distribution of Materials and
Collection of Data
• Make a plan of action:
• Who will distribute materials (i.e., scoring booklets, student
probes) at your school during benchmarking?
• Will each teacher print/copy their own materials? Or will one
person distribute the materials to all the teachers?
• When will your school begin benchmarking (a two-week window
recommended as the maximum time for each benchmarking
period)?
• Where will testing take place?
• What logistical barriers to testing exist and how will those be
handled?
Easy Score Entry
Once data are collected,
it is easily entered into
AIMSweb®’s
web-based software.
Simply type in the
scores! (pictured at left)
Dozens of reports are
then instantly available.
(Estimated time: 3-5 minutes.)
(All identifying information and
scores are fictitious.)
AIMSweb as a Program Evaluation Tool:
Benchmark 3x/year for Universal Screening—All Kids
(All identifying information and scores are fictitious.)
Michael Martin: A student with Benchmark
Data that indicates he is performing
significantly behind peers at his school.
Fall Benchmark Data for Michael Martin
Martin, Michael: Grade 5
(All identifying information and scores are fictitious.)
Box & Whiskers Graphs (box plots):
A 3-Step Explanation
1
AIMSweb commonly uses box plots to report data.
3
AIMSweb’s Box plots are
somewhat similar in
shape and representation
as to a vertical bell curve.
Michael
Martin
2
Above Average
Range
Average range of
Average Range
population included
(middle 50%)
in sample.
Below Average
Range
*In relation to user-defined comparison group
Above 90th
percentile*
90th percentile*
75th percentile*
Median (50th
percentile) *
25th percentile*
10th percentile*
Below 10th
percentile*
Target
Line
Discussion: Consider Michael’s R-CBM performance in
relationship to different AIMSweb® Comparison Groups
Grade 5:
Grade 5:
Michael’s School
Michael’s District
Grade 5: National
Aggregate Norms
(All identifying information and scores are fictitious.)
Michael appears to be performing below expectations
when compared to all three comparison groups.
• Consider modifying instructional program(s)
• Consider increasing frequency of assessment to assess
Grade 5: National
Gradeof5:
Grade 5:
efficacy
alternate program(s)
Aggregate Norms
Michael’s School
Michael’s District
• Review data regularly to assess progress
(All identifying information and scores are fictitious.)
AIMSweb National Aggregate Norm Table
AIMSweb Aggregate Norm Table
An Introductory Look at Additional
Benchmark Data
Individual Report: 3rd Grade Student
QUESTIONS:
• What does report
suggest about
Viviana’s progress?
• What does report
suggest about the
school’s progress for its
3rd grade students?
• What if you saw this
pattern in only one
school’s 3rd grade
within your district?
• What if you saw this
pattern across most
or all 3rd grade groups
in your district?
Data to Know that Things Went Well
QUESTIONS:
• What does report
suggest about
Heather A’s progress?
• SPRING: Compared to
Grade 3 peers at her
school, is Heather
performing in the:
- Well Above Average
Range?
- Above Average
Range?
- Average Range?
- Below Average
Range?
- Well-Below Average
Range?
Have Data to Know When Things Need Changing
QUESTIONS:
• What are possible reasons
why U. Boardman might not
be making progress?
• What might happen if
nothing is done to address
U. Boardman’s needs?
• Without this type of visual
data, collected at each
Benchmark period, do you
believe U. Boardman’s
stagnation would have been
quickly noticed otherwise?
Data to Know that Changes Made a Difference
Tier 2: Strategic Monitor
Strategic Monitor:
(1x/month)
Provides option to
increase assessment
frequency from three
times per year to once
per month for select
students.
Example of Strategic Monitor report containing
monthly data collected over a full school year
(All identifying information and scores are fictitious.)
Strategic Monitoring: October (Lindsey Hunter)
(All identifying
information and scores
are fictitious.)
TIER 3: Progress Monitor
3
Intensive monitoring of individualized goals for
students at-risk of failure
3
Progress Monitor (Tier 3):
Intensive assessment with adjustable
frequency that matches need
3
(All identifying information and scores are fictitious.)
Benchmark
(Tier 1):
3x per year
Strategic Monitoring
(Tier 2):
1x per month for select
students
at risk for educational
difficulties
Progress Monitor
(Tier 3):
Intensive assessment with
adjustable frequency that
matches need
(All identifying information and scores are fictitious.)
Compare a School to a Composite
Many More Reporting Options Available
The End