Collaborating with Teachers for
Next Generation Item Development
National Conference on Student
Assessment
June 29, 2012
Presentation Focus
• Engaging teachers in the process of writing
assessment items
 Overview of each state’s program
 Rationale for the approach involving teachers as
item writers
 Use of technology in the development process
 Benefits including professional development
 Lessons learned and next steps
 Next-generation assessment systems
Presenters
• John Moon, NeSA Project Director, Nebraska
Department of Education
• Andrew Middlestead, Test Development
Manager, Michigan Department of Education
• Patty McDivitt, Data Recognition Corporation
Item Development by
Nebraska Teachers
John Moon
NeSA Project Director
Statewide Assessment Office
john.moon@nebraska.gov
Overview of the Assessment
Program: Nebraska State
Accountability (NeSA)
•
•
•
•
•
Summative Assessment administered in Spring
Grades 3-8 and 11 for Reading and Math
Grades 5, 8, and 11 for Science
Grades 4, 8, and 11 for Writing
Teacher Developed Items
Overview of the Assessment
Program: Check4Learning
(C4L)
• Summative/Formative Assessment System
administered throughout the year
• State-level Item Bank of Multiple Choice Items
• For Reading, Math, and Science – all grades
• Practice Tests for Writing – all grades
• Teacher Developed Items
Balanced Assessment System
Aligned to State Standards
Statewide Assessments
(NeSA)
Interim/Benchmark Assessments
(C4L)
Classroom Assessment
(C4L)
• Both formative (C4L) and summative
(C4L/NeSA) assessments are valuable
and important
• Without both, the classroom assessment
system is not balanced
BALANCED CLASSROOM ASSESSMENT SYSTEM
FORMATIVE (C4L)
ASSESSMENTS
A process used by teachers
and students during
instruction that provides
immediate feedback to adjust
ongoing teaching and
learning to help students
improve their achievement
of intended instructional
outcomes.
SUMMATIVE (NESA &
C4L) ASSESSMENTS
A tool used after
instruction to measure
student achievement
which provides evidence
of student competence
or program
effectiveness.
Overview of the Assessment Program
Check4Learning: What is it?
• State-level item bank
of
• multiple-choice items
in
• reading, mathematics, and science
that are
• stored in the C4L online system
and
• used to generate online interim assessments to
“wrap around” NeSA tests.
How can C4L be used?
• Teachers, principals, or central office staff can
identify items
to
• generate mini tests for use at point of instruction
so that
• a check for learning can occur
at
• any time throughout the year.
Organization of Item Writing
• Table of Specifications (TOS) – What’s
needed?
 Examine indicators with highest number of
required items
 Set grade level goals for the item writing sessions
• Item Bank – What do we have?
 Inventory items at each grade by indicator
 Identify indicators needing additional items
 Examine DOK and difficulty levels for additional
items
Rationale for the Approach:
Work Session Item Development
– NeSA and C4L
• To understand a balanced assessment system
• To develop quality items for tested indicators
• To identify the characteristics of quality multiplechoice items
• To acquire adequate information to support others
on the balanced assessment system
A Quality Test . . .
• Accurately and
appropriately measures
what is relevant.
• Is consistent and precise.
• Does not put any group
at a disadvantage.
Validity
Reliability
Fairness
Quality Assessment Criteria
1. Assessments reflect the state standards.
2. The students have had an opportunity to learn
the content.
3. Assessments are free from bias or offensive
situations.
4. Assessment levels are appropriate for students.
5. The mastery levels are appropriate.
Item Writing Guidelines
•
•
•
•
Content
Style
Item Stems
Response Options
Item Writing Guidelines - CONTENT
• Should be clearly aligned to the identified
indicator
• Should not reference material or information
presented in another item
• Should focus on one idea or concept
• Should test meaningful content
Item Writing Guidelines - STYLE
• Spell out acronyms
• Minimize the use of “NOT” and “EXCEPT”
• Avoid absolutes (e.g., always, never) and
vague modifiers (e.g., best, worst)
• Use appropriate vocabulary
• Ensure correct grammar, punctuation,
capitalization, spelling
• Use active voice and present tense
Item Writing Guidelines – ITEM STEMS
• State clearly and concisely
• Use complete sentences with ending
punctuation in context
• Question appears at the end of the stem
• Use which before a noun and what before a verb
• Avoid negatives (capitalize if used)
• Do NOT use “Which of the following . . .”
Cognitive Demand
Three Different DOK Levels
DOK 1- Describe three characteristics of metamorphic rocks.
(Requires simple recall)
DOK 2- Describe the difference between metamorphic and
igneous rocks. (Requires cognitive processing to determine
the differences in the two rock types)
DOK 3- Describe a model that you might use to represent the
relationships that exist within the rock cycle. (Requires
deep understanding of rock cycle and a determination of
how best to represent it)
Item Difficulty
Difficulty is a reference to how many students
answer a question correctly.
“How many of you know the definition of exaggerate?”
DOK 1 – recall
If all of you know the definition, this question is an easy question.
“How many of you know the definition of prescient?”
DOK 1 – recall
If most of you do not know the definition, this question is a
difficult question.
Item Review Check List
• Item Stem
▫ Content – alignment and focus
▫ Style – clear and concise
•
•
•
•
Responses – plausible and homogenous
Key – correct answer
DOK – appropriate and correct
Difficulty – appropriate and correct
Item Development by
Michigan Teachers
Andrew Middlestead
Office of Educational Assessment and
Accountability in the Michigan
Department of Education
middlesteada@michigan.gov
Overview of the Assessment Program
 Michigan currently has 6 separate assessment
programs which it develops and administers
 Michigan Educational Assessment Program
(MEAP)
 Michigan Merit Examination (MME)
 MEAP-Access (2% population)
 MI-Access (1% population)
 English Language Proficiency Assessment(ELPA)
 Michigan Interim Assessments (K-12)
Overview of the Assessment Program
Michigan uses Michigan educators
throughout the development process of the
majority of test items that are needed for all
of these assessment programs.
Teacher-Written Items:
Development Model
 Michigan educators:
 Write items based on Michigan content
expectations and any guidance from Michigan
and DRC staff
 Review all items through both a Content and
Bias/Sensitivity review session
 Review field-test data from both a content and
bias/sensitivity perspective
Teacher-Written Items:
Development Model (continued)
 Michigan educators:
Michigan published children’s
authors:
Write commissioned passages for
use in our assessments.
Rationale for the Approach
• Michigan has been using educators to write and
review assessment items for up to 10 years.
• Having educators involved heavily in the test
development provides some sense of ownership
of the Michigan assessments their students see.
(though it doesn’t mean they like it  )
• Items are only developed by state or contractor
staff as a last resort in unique situations.
Use of Technology in the
Teacher-Written Item Development Process
• Michigan has developed its own online Item
Banking and Test Development system.
• Educators are able to work online in Item Writer
Training sessions, or independently on tasks
• Tasks are able to be done quicker and more
efficiently with far less paper (we use hardly any)
Use of Technology in the
Teacher-Written Item Development Process
Benefits
• The largest benefit I see in having
Michigan educators be so involved in our
item development process is:
 They can go back to their school and talk
about the process to hopefully increase
educator buy-in of the assessment
program.
Benefits (continued)
• Benefits I see in having our item
development online is:
 Faster turn-around time from item
writing, to reviewing, to consolidated
comments.
 Less paper
 Consistent with the times.
Lessons Learned
• Importing items and their attributes from
various sources into one item bank isn’t so easy.
• Transition time to realize efficiencies
• With such high educator involvement, regularly
evaluate your training materials and methods
Future Plans and Next Steps
 Goal of 100% commissioned passages
 12 Michigan published children’s authors have
been writing pieces across several assessment
programs for the past 18 months
 These authors are thrilled!
 Commissioned work has it’s benefits!
 Enhancing Michigan’s Item Banking System
 To provide more flexibility in program and item
types
 Technology-based items
Summary
Patty McDivitt
Data Recognition Corporation
pmcdivitt@datarecognitioncorp.com
Two Models for Teacher-Developed Items
 Nebraska
 Centralized training
and item writing
workshops; immediate
feedback provided
 Decentralized training
is not an option
 Teachers use a userfriendly MS Word
template; items
imported into the item
bank
 Michigan
 Centralized training
and item writing
workshops; immediate
feedback provided
 Decentralized training
is an option
 Teachers receive
training and write
directly inside the item
banking system
Two Models for Teacher-Developed Items
 Nebraska
 External formatting of
items for publication
 Once the item is
submitted and
accepted, the
involvement of the
teacher in writing the
item is done
 Michigan
 Internal formatting of
items for publication
 Once the item is
submitted and
accepted, the
involvement of the
teacher in writing the
item is done
Role of the Vendor in TeacherDeveloped Items
• Seek to provide support to the process
• Seek to ensure that the specific concerns of
assessment such as source of challenge are
considered without losing the intent of the writer
• Seek to ensure that the items adhere to style
specifications and pre-determined technical
quality
Next-Generation Assessment Systems
• Much as teachers have new and novel ways to approach
instruction in the classroom, teachers as writers will
display creativity in addressing the standards in new and
novel ways, including
• becoming invested in the deep understanding of the
standards and the development of the assessments
• helping to promote the link between assessment and
instruction in the classroom, and
• viewing assessments in a more positive light due to
peer involvement in the process.
Next-Generation Assessment Systems
• Teachers will become familiar with the use of
templates and/or the entire item-banking
process (shared item components).
• Teachers will use what they have learned in the
item-writing workshops to develop classroombased assessments.
Next-Generation Assessment Systems
 Teachers will become familiar in the use of itemwriting guidelines and item specifications.
 Teachers will gain understanding in the
structure of items designed to measure the
standards, including
 Evidence-based design items
 Selected-response items
 Technology-enhanced items
 Other
Next-Generation Assessment Systems
 Writer’s workshops (professional development)
 Focus upon collaboration among teachers
and/or the department
 Engage teachers in direct writing; providing
regular and direct feedback
 Engage teachers is peer review
 Empower teachers with responsibility and
ownership of the development process
Questions and Answers