STUDENT GROWTH
PROFESSIONAL
DEVELOPMENT FOR THE
PRINCIPAL AND TEACHER
EVALUATION PROCESS
May 2014
Survey Results


10 respondents from Advisory Committee
Top topics to cover:
 Designing
high quality assessments
 SLO’s
 Team

activities related to assessment creation
Top subtopics to cover:
 SLO’s:
discussion, process/creation, fairness
 Assessments: what is high quality, key elements,
selection
 Growth models: identification, implementation
Agenda for the Day





Student Growth Measures
Student Learning Objectives
Team Sharing Time
Assessment Development
Office Hours
Website with Resources
http://edanalytics.org/projects/illinois-studentgrowth
PERA Updates
PERA May update

Local Assessment Support (ISBE contract awarded)

Develop assessment literacy

Create local assessments, mostly in traditionally non-tested grades and subject

Run workshops, some on performance-based assessment, e.g. music,
career/technical



Pilot these workshops, then run with larger groups, and/or make into webinars
Guidebook on student growth (posted)

Includes examples of timelines and decision points and examples from other states
(build/borrow/buy assessments, set up data systems, set up SLOs, run PD)

Includes strong recommendation to have at least one pilot year
SLO template (posted)

Associated guidance shows that goals can be set by group (each starting point) or by
student, or both

Draft SLOs are posted in elementary math, music, HS English, independent living,
nutrition/culinary arts, consumer math

No guidance on how to make results comparable across teachers/schools/subjects.
PERA May Update


Guidance document on teachers of special education students, ELL, and early childhood
(discussed, not finalized or posted)

Districts have been advised by Charlotte Danielson not to create new frameworks, and to create
addenda instead. CPS has created these for special education, ELLs, arts, and physical education,
and one for early childhood education (ECE) will be released soon.

These are needed because administrators don't always have experience in all areas; they
illuminate specific considerations for these groups/acknowledge potentially different context
Lowest 20% of districts not yet determined


These must implement in 2015-16; Likely will be determined in summer 2014
State default model

Use one type II and one type III assessment, if a type II exists in that subject area and the process
for developing or selecting an assessment includes teachers, the assessment is aligned with the
curriculum, and is reliable and valid.

Guidance on how to meet these requirements will be forthcoming

Otherwise, use two type III assessments, with SLOs, one determined by evaluator, one by teacher

Note that the state model is only required in situations where a joint committee cannot reach
agreement on one or more aspects of student growth, and is only required for the parts where no
agreement was reached.

So if a joint committee agrees to everything but the percentage assigned to student growth, it
must use 50%.
PERA May Update

PEAC meetings are public
 Upcoming
meetings (all scheduled for 10 a.m. - 3 p.m.
at ISU Alumni Center, 1101 N Main, Normal):
 May 16 (may
be moved to Springfield)
 June
20
 July 18
 August 15

www.isbe.net/peac
Student Growth Measures
Student Learning Objectives
Value Tables
Simple Growth
Adjusted Growth
Value-Added
Student Learning Objectives (SLOs)

SLOs are goals that teachers and administrators set
using baseline data such as:
 Students’
performance on a prior year test
 Students’ performance on a diagnostic test (e.g., for
literacy or math skills)
 Prior year attendance data or discipline rates


Goals are monitored throughout the year and
success or failure determined at the end of the year
SLOs can be created by individual teachers, teams of
teachers, or the district.
Student Learning Objectives
Benefits
• Can be used with any assessment type
• Can be uniquely tailored
Drawbacks
• Not necessarily standardized or
comparable across educators
• Potentially very easy to game
• Potentially very hard to draw
meaningful conclusions
Implementation Requirements
• Parameters for educators to
follow
• Type I or Type or Type III
• Consideration of unintended
consequences and biases
Value Tables


Assigns point values to improvements across
categories (e.g., proficient to advanced)
Value tables can be weighted in many different
ways and cut points can be placed at creator’s
discretion.
Value Table Example
Cut Scores (from ISBE)
Example: Grade 4 to 5 Math

Focus on categories 1b and 2a (but possible for any adjacent categories in other
subjects and grades)

Grade 4: 1b covers 174 to 190, 2a covers 191 to 206
Grade 5: 1b covers 183 to 200, 2a covers 201 to 217







Student A: Moves from 174 to 200 (gain: 26 points, 85 value table points for
moving from 1b to 1b)
Student B: Moves from 190 to 190 (gain: 0 points, 85 value table points)
Student C: Moves from 190 to 183 (gain: -7 points, 85 value table points)
Should students A, B and C be weighted the same?
Student D: Moves from 174 to 182 (gain: 8 points, 20 value table points for
moving from 1b to 1a)
Should student C count more than student D?
Example: Grade 4 to 5 Math

Focus on categories 1b and 2a (but possible for any adjacent categories in
other subjects and grades)

Grade 4: 1b covers 174 to 190, 2a covers 191 to 206
Grade 5: 1b covers 183 to 200, 2a covers 201 to 217





Student E: Moves from 191 to 217 (gain: 26 points, 90 value table points
for moving from 2a to 2a)
Student F: Moves from 191 to 218 (gain: 27 points, 125 value table points
for moving from 2a to 2b)
Student G: Moves from 190 to 218 (gain: 28 points, 150 value table points
for moving from 1b to 2b)
Is the difference between students E, F and G substantial enough to justify
three widely different weights?
Value Tables
Benefits
• Uses two points in time—better than
attainment!
Drawbacks
• Does not separate teacher’s effect
on student growth from studentlevel factors
• Can lead to focus on “bubble kids”
• Biased if the point values do not
reflect the difficulty of making a
particular improvement
• Ignore improvements within
categories, and measurement error
• Easy to draw incorrect conclusions
based on category instability
Implementation Requirements
• Maintain data from two
consecutive years
• Type I or Type II (considerably
more challenging for Type III)
• Creation of categories
• Consideration of unintended
consequences and biases
Simple Growth


Here we are not referring to the general definition of
growth as outlined in state legislation, but a specific
type of growth model.
Simple growth is average gain in test scores across
students.
Simple Growth
Classroom
Gain in Score from
rd
th
3 to 4
1
11
2
13
3
12
4
7
5
20
6
19
7
15
8
22
9
18
10
29
11
35
12
33
Gain
High
Low
Is it fair to compare these
classrooms using simple growth?
3 Grade Score
4 Grade Score
Gain in Score from
rd
th
3 to 4
1
312
323
11
2
304
317
13
3
301
313
12
4
294
301
7
5
288
308
20
6
278
297
19
7
275
290
15
8
264
286
22
9
259
277
18
10
256
285
29
11
244
279
35
12
238
271
33
Classroom
rd
th
Test Score
Range
High
Low
Gain
High
Low
Simple Growth
Benefits
• Uses two points in time—better than
attainment!
Drawbacks
• Does not separate teacher’s effect
on student growth from studentlevel factors
• Ignores test measurement error
• May be harder for some students to
make gains—particularly high
achievers
• Easy to draw incorrect conclusions
based on test scale instability
Implementation Requirements
• Maintain data from two
consecutive years
• Type I or Type II (to be safe)
• Subtraction!
• Test where typical gain is the
same regardless of starting point
• Consideration of unintended
consequences and biases
Adjusted Growth

Adjusted growth measures growth in student scores
from one year to the next by looking at groups of
students (divided according to their performance on
a prior year test) and comparing their performance
on a post-test
Using Real Data to Account for the
Relative Difficulty of Making Gain
3 Grade Score
4 Grade Score
Gain in Score from
rd
th
3 to 4
Allen, Susan
312
323
11
Anderson, Laura
304
317
13
Alvarez, Jose
301
313
12
Adams, Daniel
294
301
7
Anderson, Steven
288
308
20
Acosta, Lilly
278
297
19
Adams, James
275
290
15
Atkinson, Carol
264
286
22
Anderson, Chris
259
277
18
Alvarez, Michelle
256
285
29
Abbot, Tina
244
279
35
Andrews, William
238
271
33
Student
rd
th
Test Score
Range
High
Low
Gain
High
Low
Using Real Data to Account for the
Relative Difficulty of Making Gain
rd
Student
th
3 Grade 4 Grade
Score
Score
Gain in Score
rd
th
from 3 to 4
Typical Growth
Student made at least
typical growth?
Allen, Susan
312
323
11
10
Y
Anderson, Laura
304
317
13
12
Y
Alvarez, Jose
301
313
12
13
Adams, Daniel
294
301
7
14
Anderson, Steven
288
308
20
16
Y
Acosta, Lilly
278
297
19
17
Y
Adams, James
275
290
15
18
Atkinson, Carol
264
286
22
20
Anderson, Chris
259
277
18
21
Alvarez, Michelle
256
285
29
26
Y
Abbot, Tina
244
279
35
28
Y
Andrews, William
238
271
33
30
Y
Y
Adjusted Growth
Benefits
• Takes students’ starting points into
account
Drawbacks
• Does not separate teachers’ effect
on student growth from
demographic factors
• Ignores test measurement error
Implementation Requirements
• Maintain data from two
consecutive years, and additional
historical data to determine
typical growth
• Type I or Type II (to be safe)
• More complex methods than
subtraction
• Consideration of unintended
consequences and biases
Value-Added

A value-added model:





takes a classroom of students’ pre-test scores and demographic
characteristics,
compares those students to others like them, and
predicts what their post-test scores would be assuming students had
an average teacher.
If the students’ actual teacher was able to produce more growth
than predicted in her students, she will have a high value-added
score. This teacher “beat the odds”.
This is the difference between actual student achievement and
the average achievement of a comparable group of students
(where comparability is determined by prior scores, at a
minimum).
Value-Added
Visual Representation
Actual student
achievement
scale score
Value-Added
Starting student
achievement scale
score
Predicted student achievement
(Based on observationally
similar students)
Year 1
(Prior-test)
Year 2
(Post-test)
Value-Added
Benefits
• Comprehensive measure that accurately
separates effects of educator on student
growth from other confounding factors
Drawbacks
• Requires the most data and more
complex modeling
Implementation Requirements
• Maintain data from two
consecutive years
• Large enough sample size (alone
or in a consortium)
• Type I or Type II
• Research team or statistical
capacity for calculation
• Statistical reference group score
and demographic data
• Consideration of unintended
consequences and biases
Capabilities Checklist
Please see the Capabilities Checklist handout
Goal Helper Tool
This is a goal helper, not a goal creator.
The goal helper provides a reality check
based on historical data.
Goal Helper


A tool to help you make realistic goals
Four Tabs
 Previous
Year Raw Data (Blue)
 Pre-Test Groups (Orange)
 This Year Raw Data (Red)
 Goal Output (Purple)

Note, the tool also includes seven (Gray) tabs; information in these tabs relates to the
workings of the tool and should not be altered.
Tab 1: Previous Year Raw Data
Column A
• Student ID
• This should be
unique by student
Column B
• Last Year Pre-Test Score
• Must be a numeric
value for this
student's Pre-Test
Column C
• Last Year Post-Test Score
• Must be a numeric
value for this
student's Post-Test
Data here is for all students in the district taking the
assessment at this grade level.
Tab 2: Pre-Test Groups
Desired Number of Pre-Test Groups
• Yellow Box (ONLY)
• Chose between 1 and 8
groupings
Information is provided in the
orange box for the Average Gain by
Pre-Test Score Range
Tab 3: This Year Raw Data
Column A
• Student ID
• This should be
unique by student
Column B
• Pre-Test Score
• Must be a numeric
value for this
student's Pre-Test
Data here is for ONLY the students in a particular class
or school taking the assessment at this grade level.
Tab 4: Goal Output
Column A
• Student ID
Column B
• Pre-Test
Column C
• Pre-Test Group
Column D
• Average Gain by Pre-Test
Group
Column E
• Post-Test
The purple box provides the projected group (class or
school at this grade level) average Post-Test score.
DuPage Goal Helper

To access the goal helper, please have one member
of your team contact Linda Kaminski at the DuPage
ROE: lkaminski@dupage.k12.il.us
Team Reflection Time
Please see Team Reflection Time Guiding Questions
Student Learning Objectives
What are SLOs? (A recap)
Student Learning Objectives (SLO) are
detailed, measurable goals for student academic
growth to be achieved in a specific period of time
(typically an academic year).
“A detailed process used to organize evidence of student growth
over a specified period of time.” – ISBE SLO webinar
Key Characteristics of SLOs*
Baseline Data and Rationale
What are student needs? What
baseline data did you review and what
did it demonstrate? What are student
starting points?
Learning Goal
What should students know and be
able to do at the end of your course?
How does this goal align to content
standards?
Assessments and Scoring
How will you measure the learning
goal? How will you collect data,
monitor, and score final student
outcomes?
Target Population
Which students will you target in the
SLO?
Expected Growth Targets
What is your specific target, or goal, for
student growth? Are goals different for
different student populations, based on
starting points?
Time Span
What is the timeframe for the SLO?
(typically a semester or a year)
Instructional Strategies
What instructional strategies will you
employ to support this SLO? How will
you modify strategies based on student
progress?
* Derived from ISBE SLO Guidebook and Template
SMART Goals
Does the SLO statement identify a specific
student population and growth target?
Specific
 Measurable
 Appropriate
 Realistic
 Time Limited

How is growth being measured? Does the SLO
identify a quantifiable growth target and a
specific assessment/evidence source?
Is the SLO appropriate for the grade
level/subject area? Are growth targets
appropriate for the student population?
Are goals and growth target realistic, given
the student population and
assessment/evidence source?
Does the SLO indicate the time period within
which the goal must be met?
Flexibility of Approaches to the
SLO Process
New York
More
Structured
Georgia
Ohio
Wisconsin
More Flexible
Assessment Selection

Flexible – Allows educators to select and/or
develop their own assessments to measure
student growth.

Structured – Requires the use of pre-approved,
standardized assessments
Target Setting

Statistical or Model Informed


Objective, or standardized


SLO targets are determined using a statistical model to predict
levels of student growth based on prior data
Standardized, or common, way to set growth targets for
teachers across classrooms, schools & districts
Subjective


Growth targets are set based on historical data, student needs,
and context.
Relies more on professional judgment
Scoring

Statistical or Model Informed
SLO scored using a statistical model that predicts levels of
student growth and set thresholds for final SLO ratings
 No scoring rubric needed


Objective, or standardized


Scoring rubric includes prescriptive criteria for assigning a
rating category based on student outcomes
Subjective

Scoring rubric includes broad and/or subjective language
for assigning a rating category based on student outcomes
Structured: New York

Assessment selection
 Requires
use of state test where available
 Provides list of state-approved assessments for
district use

Target setting
 Expectations

based on state-provided scale
Scoring
 Number
of students reaching the target goal
directly linked to final SLO score
New York State Scoring
New York State Scoring
Structured: Georgia

Assessments selection
 Teacher-selected,
but they must meet
minimal criteria

Target setting
 Teacher-selected,
but the state approves
the overall SLO goal

Scoring Rubric
 Each
possible score associated with % of
students that met the goal
Georgia (cont.)
Flexible: Ohio

Assessment selection
 Teacher-selected

Target setting
 Teacher-selected
 Attainment

discouraged
Scoring rubric
 Principals
must use the SLO scoring matrix to
score SLOs
Ohio (cont.)
Flexible: Wisconsin

Assessment selection
 Teacher-selected

Target setting
 Teacher-selected

& principal approved
& principal approved
Scoring rubric
 Outcomes
& process
 “Holistic” & self-scored
 Ultimately
principal determined
Trade-offs: Assessments
Flexibility
 Allows teachers to identify best assessment, BUT
 Lack of comparability of student outcomes
 Is
“passing” the same thing across different SLOs?
Standardization
 Provides clarity and potentially comparability, BUT
 Assessments may not directly align to classroom
teaching
Trade-offs: Target-setting
Flexibility
 Allows educators to consider individual student needs and
contextual factors, BUT
 Could potentially penalize educators who set ambitious
goals & reward those who set less rigorous goals
Standardization
 Provides consistency in SLO goals across educators, BUT
 Formulas may not be appropriate for all courses, students,
and assessments
Austin Example
Total Points – Pre-test Score = Predicted growth
2
100 – 70 = 15 points (or 85 post-test score)
2
Issue of growth difficulty
New York State Example

Standardization can work well
 NWEA
MAP
 Value-added model
Trade-offs: Scoring
Flexibility
 Supports professional judgment & takes extenuating
circumstances into account , BUT
 Subjectivity in defining what “all” or “most” students
means when assigning a rating
Standardization
 Allows for greater comparability across and within
schools
 Limits the degree to which professional judgment
enters into the SLO scoring process
Illinois?
Activity:
Consider the three major dimensions:
Assessment selection
 Target setting
 Scoring rubric

Think about how these dimensions work together in your
district to inform:
• Instruction
• Resource allocation
• High stakes decisions
SLO Dimensions
Scoring
rubric
Assessment
selection
Target
setting
Team Reflection Time
Lunch
Assessment Development
Agenda
Why use assessments?
 What makes a high quality assessment?
 Selecting high quality assessments
 Designing high quality assessments

Why use assessments?

Inform instruction
 Allows
the teacher to plan, support, monitor and
verify learning

Support resource allocations

High stakes decisions
Assessment Types as Defined by
PERA
Type I
Type II
Type III
An assessment that measures
a certain group of students in
the same manner with the
same potential assessment
items, is scored by a nondistrict entity, and is widely
administered beyond Illinois
An assessment developed or
adopted and approved by the
school district and used on a
district-wide basis that is
given by all teachers in a
given grade or subject area
An assessment that is
rigorous, aligned with the
course’s curriculum, and
that the evaluator and
teacher determine measures
student learning
Examples: Northwest Evaluation
Association (NWEA) MAP tests,
Scantron Performance Series
Examples: Collaboratively
Examples: teacher-created
developed common assessments, assessments, assessments of
curriculum tests, assessments
student performance
designed by textbook publishers
Need for Assessments in Illinois

ISBE requires a combination of
Type I and II (and possibly III) assessments for
principal evaluation.
 Type I, II, and III assessments for teacher evaluation.

What makes a high quality
assessment?

Selecting or constructing the right assessment for
the right purpose
 Does
the assessment align with the curriculum
standards?
 Does the assessment align with what is actually being
taught in the classroom?

Need appropriate assessments for SLOs and other
growth measures
Key Features of a High Quality
Assessment

Alignment: should cover the actual
standards/constructs to be taught in the classroom

Validity: measures what it claims to measure

Reliability: consistency (same test taker receives
same score)
How do I know if an assessment is
valid and reliable?

Ask yourself:
 Would
the same student, taking the test two days in a
row, be likely to get the same score? (Reliability)
 Does the test measure the curriculum I taught that
year? (Validity)
 Does the test cover all areas of the curriculum taught?
(Validity)
Assessments & Growth

Imprecise, inaccurate assessments lead to
imprecise, inaccurate growth measures
 Floor
& Ceiling effects
 Imprecision outside of proficiency thresholds
 Coarse scaling
Selecting Assessments for SLOs
Quality considerations:
 Content-alignment
 Stretch of growth
 Validity & reliability
Logistical considerations
 Timing (time of year and assessment duration)
 Test form
 Administration
 Scoring
SLO Assessment Checklist Activity
With your team:
Step 1: Identify one or two assessments that you think
teachers use/will use to write their SLO goals.
Step 2: Next, describe the key features of your
assessment in the Assessment Description form.
Step 3: Finally, use the SLO Assessment Checklist to
determine if your assessment meets the basic
requirements for an SLO
Designing Assessments

At what point should teachers develop their own
assessments?

How should teachers develop their own
assessments?
Hillsborough County Assessment
Development

Subject / Grade Coverage


Models from Art to Welding
Multiple Measures
Charlotte Danielson observational
ratings
 Combined use of student
outcomes and observational data
in state system


Data Quality

Student-Teacher Link
LAUSD Gates Project

Partnership between the Los Angeles Unified School
District, Education Analytics and the Gates Foundation
to develop teacher-developed assessments

Contributors include assessment design and evaluation
experts from around the country
Overall goal: To develop a replicable process for
supporting high-quality teacher-created assessments and
SLO development
LAUSD Teacher-Developed
Assessment Pilot


Piloting Fall 2014 & Spring
2015
30 to 40 teacher participants
5th grade Theatre
 High School Biology


Teacher Assessment Writing
Workshops
Assessment blueprint creation
 Item writing
 Item review and revision

LAUSD Assessment Development
Timeline
1
2
3
4
5
6
7
• Design: Prioritize Standards & Create Assessment Blueprint – Coordinators
• Write Items - Teachers
• Editorial & Bias Review – Education Analytics Team & Assessment Experts
• Content Review – Teachers & Coordinators
• Field Testing - Representative biology and theatre classrooms
• Analyze Data – Education Analytics Team
• Revise Items & Create Operational Tests – Teachers and Education Analytics
Team
Item Writing Process
Test Blueprint
Purpose:
 Communicate to teachers what is important to
teach
 Communicate with students and parents what is
important to learn
 Provide direction to the item writers and test
developers
 Ensure appropriate coverage of content on test
 Outline the cognitive complexity of the test items
Blueprint Development Process

Identify the benchmark or standards that will be
measured on the test
 Determine the



benchmarks that will not be tested!
Identify pre-requisite skills that should be
measured
Identify the weight of each blueprint entry
Determine the cognitive complexity of the items
that will be included on the test
Depth of Knowledge
Source: Webb, N.L., et al. (2005). Web Alignment Tool. Wisconsin Center of Education Research.
University of Wisconsin-Madison. Retrieved from http://www.wcer.wisc.edu/WAT/index.aspx
Theatre Example

Benchmark:
1.2 Identify the structural elements of plot (exposition,
complication, crisis, climax, and resolution) in a script or
theatrical experience.

What pre-requisite skills should be measured?


What is the level of cognitive complexity?



Students need to know the terms & definition
“Identify”
Level 2 on DOK: Skill/concept
What weight should we assign the standard?

20% of unit curriculum focuses on this content
Blueprint Example
Benchmark/ Recall
Content
Skill/Conce Strategic
pt
Thinking
Theatre: 1.2
XX
Extended
thinking
Weight
20%
Assessment item types
Constructed response
 Short answer
 Essays
 Labeling
 Performance-based tasks
Objective
 Multiple choice
 True/False
 Matching
 Fill-in
Why Performance Tasks?

Used to measure learning outcomes and
learning objectives that cannot be measured
well by objective tests

Suited for less structured problems, creation
of a product or a performance
Performance Tasks



Emphasis is on doing – not merely knowing; on
process as well as product
Goal is to be as authentic as possible
Can be narrow in definition or more broad and open
Examples: Oral presentations, online reading logs,
podcast, scientific experiment, excel graphing
Source: Brenda Lyseng, Minnesota State Colleges and Universities Center for Teaching and
Learning
Building a performance task
Learning Objectives
Which one do you want to assess?
Task
What is the core task you want
students to perform?
Source: Brenda Lyseng, Minnesota State Colleges and Universities Center for Teaching and
Learning
Building a performance task
What content standards do you want What knowledge will students
students to demonstrate?
demonstrate?
What performance standards do you What skills will students
want students to demonstrate?
demonstrate?
What will be their sources of
information?
Interviews, primary sources,
secondary, textbook
What type of product do you want?
Written report, oral report,
recommendation, graph
How will students work?
Individual, partner, team
Source: Brenda Lyseng, Minnesota State Colleges and Universities Center for Teaching and
Learning
Performance Task Rubric

Considerations:
 What
elements must be present to ensure high
quality?
 How many levels do I want?
 What is a clear description of each achievement level?
 Rubrics are for you and the students – ask for
feedback
Source: Brenda Lyseng, Minnesota State Colleges and Universities Center for Teaching and
Learning
Why Multiple Choice Items?




Efficient
Objectively scored
Tend to help with creating reliable tests
Can be very effective in producing information on
students’ conceptual knowledge, procedural
knowledge, and reasoning (Webb, 2006).
Parts of a Multiple Choice Item
What is the leading cause of death in the
United States?
Key
A.
B.
C.
D.
Cancer
Heart disease
Accidents
Diabetes
Stem
Distractors
*The Key and Distractors collectively are called the response options.
MC Item Formats

Correct answer
 One
option is indisputably correct and the other
options are indisputably incorrect
What year was George Washington elected
president?
A. 1783
B. 1789 *
C. 1790
D. 1795
MC Item Formats

Best answer
 One
or more options are correct to varying degrees;
however one option is unquestionably better than the
other options
According to the Mayo Clinic, what is the first thing you
should do if you burn your skin?
A. Take a pain reliever.
B. Put a burn ointment on your skin.
C. Call your doctor.
D. Run your skin under cool water. *
MC Item Formats

Negative format
 Examinee
is asked to identify which option is not
correct or which is the worst answer
 Use with caution!!
 Most negative format items can be converted into
correct answer or best answer formats
Which of the following is not one of the author’s
main points?
MC Item Formats

Multiple answer
 More
than one option is keyed as correct
 Examinee must identify all correct answers
Which of the following are recommended if you are trying to
reduce your blood pressure? Select all that apply.
A. Reduce your sodium intake *
B. Get 8 hours of sleep each night
C. Exercise regularly *
D. Increase your fluid intake
MC Item Formats

Complex format

Takes the options and regroups them into combined
answers
Which of the following are benefits of multiple choice items?
i.
They are efficient.
ii. They are objectively scored.
iii. Students like them.
A.
B.
C.
D.
i and ii
ii and iii
i and ii
i, ii, and iii
General Guidelines

Each item should reflect one content area
 What
is the important content the students should
know?
 Do not
 Use
test trivial content
your test blueprint!
 Identify
the relevant test objective that the item is
measuring
General Guidelines

Use correct grammar, punctuation, capitalization, and
spelling
For incomplete stems:
The process by which plants convert light energy into
chemical energy is called
A. cellular respiration.
B. photosynthesis. *
C. osmosis.
D. evaporation.
Guidelines for Writing the Item Stem

Keep the stem as brief as possible
 Be
clear and concise
 Avoid “window-dressing”
 Minimize reading time
 Stems that are unnecessarily wordy or contain
unneeded information can confuse test takers
Guidelines for Writing the Item Stem

Be sure the stem contains the main idea of your
item

When possible, avoid using negative words
 Not,
never, except, least, false
 EXCEPTION: Negative format item
 First
try to write the item as another format
Guidelines for Writing the Item Stem

Use terms like often, frequently, rarely, and
occasionally with caution
 These
words have different interpretation for different
examinees

Avoid being over specific or over general

Avoid opinion based items
 Qualify
any opinions
Guidelines for Writing the Item Stem

Use a visual stimulus in the item stem only if it is
necessary
A
graph, chart, picture, diagram etc.
 Including a stimulus that is not necessary can detract
from the purpose of your item
Identify the component of a neuron
that the arrow is pointing to.
A. Axon
B. Nucleus
C. Cell body
D. Dendrite *
Item Stems Needing Revision
The 19th amendment to the U.S.
Constitution, which was adopted
on August 18, 1920, granted
women the right to
Lengthy stem;
contains unnecessary
information
The 19th amendment to the
U.S. Constitution granted
women the right to
Romeo and Juliet
Too general; stem
does not include the
main idea
The main theme in
Shakespeare's Romeo and
Juliet is
The best movie of all time is
Stem is opinion based According to the American
Film Institute, the best
movie of all time is
Which of the following is not
found on Maslow’s hierarchy of
needs?
Negative stem
Which of the following is
found at the top of
Maslow’s hierarchy of
needs?
Guidelines for Writing the Options

Vary the position of the key
 Assign

the location of the key first
Make the distractors plausible
 What
are the common mistakes students make or the
common misconceptions they have?
 Do not use funny distractors
Guidelines for Writing the Options

Place the options in a logical order
 Numerical,

chronological, shortest to longest, etc.
Avoid using specific determiners
 Always,
never, only, etc.
In order for a bill to become a law it
A. should only be written by a U.S. Senator.
B. must always be signed by the president.
C. needs to pass both houses of Congress. *
D. can never be vetoed.
Guidelines for Writing the Options

Move any phrases that are repeated in all
options to the stem
If a number is squared,
A. the result is the number times two.
B. the result is the number plus itself. BETTER:
C. the result is the number times itself.
If a number is squared, the
D. the result is the original number.
result is the
A. number times two.
B. number plus itself.
C. number times itself. *
D. original number.
Guidelines for Writing the Options

Keep the options independent from one another
Which of the following has been shown to lower
cholesterol?
A. apples
B. yogurt
C. fruit
D. oatmeal *
Guidelines for Writing the Options

In general, avoid using “none of the above” and
“all of the above”
 None
of the above
 Often
not plausible to students
 Measures the ability to recognize incorrect answers
 All
of the above
 If
a student can identify that two alternatives are correct,
then they know that “all of the above” is the correct
answer
Guidelines for Writing Options:
Giving Away the Correct Answer

Key is too long or too specific
The term pessimism most nearly means
A. thoughtful.
B. happy.
C. the tendency to only see bad or undesirable outcomes. *
D. scared.
Guidelines for Writing the Options:
Giving Away the Correct Answer

Avoid one item giving away the answer to another
item
2.
A.
B.
C.
D.
In the passage, Iago is a
parrot. *
iguana.
flamingo.
snake.
4. The parrot in the story, Iago,..
Guidelines for Writing the Options:
Giving Away the Correct Answer

Avoid repeating words in the stem and the key
What is true of the Western Lowland Gorilla?
A. They are found on the continent of Asia.
B. The gorilla mostly consumes a vegetarian diet. *
C. Their lifespan is approximately 25 years.
D. Newborns weigh an average of 20 lbs. at birth.
Item Revision Activity
Activity:
Take out your Item Revision worksheet.
(1)
(2)
(3)
Identify the problem with the each question’s item(s)
Rewrite the questionable item to make it higher
quality
Share out
Items for Revision
Who painted the Mona Lisa?
A. Giovanni Bellini
B. Sandro Botticelli
C. Leonardo da Vinci *
D. Mickey Mouse
Problem: Funny, implausible distractor
Items for Revision
The humpback whale’s diet consists mainly of
A. krill and small fish. *
B. seals.
C. sharks and large fish.
D. mammals.
Problem: Dependent options
Items for Revision
Minnesota
A. is the 41st state.
B. has the robin as its state bird.
C. is known as the “Show-Me-State”.
D. Is the 21st most populous state in the U.S. with most
of its residents living in the Saint Paul/Minneapolis
metro area. *
Problem: stem is too general; main point is not conveyed
in the stem, key is too long and specific
Items for Revision
The first amendment of the U.S. Constitution grants
freedom of
A. speech.
B. religion.
C. press.
D. all of the above **
Problem: uses all of the above
Items for Revision
In the 1st book in the Harry Potter Series,
A. Harry, Ron, and Hermione kill a basilisk.
B. Harry, Ron, and Hermione only attend class.
C. Harry, Ron, and Hermione fight werewolf.
D. Harry, Ron, and Hermione find the Sorcerer's
Stone. *
Problem: repeated phrases in stem and options;
incorrect grammar; specific determiner
Office Hours