Michigan School Testing Conference
February 21, 2012
Dr. Ed Roeber
Kim Young
Dr. Ellen Vorenkamp

Who Are We?
Let’s speculate about the people in the room 
What one question might you ask to explore your notion?







Next 5 minutes, circulate around room
Name, professional role, district
Ask your question without comment or clarification and record data
Analyze data
What assumptions might you make about people in the room?
To what extent did your question give you the data you were looking for?
Who Are We?



Participants will recognize the need for quality classroom assessments including elements such as:
◦ Standard/Item Alignment
◦ Balance of Representation
◦ Target/Method Match
◦ Quality Items
◦ Test Blueprints
Participants will reflect on and modify
(where needed) current assessments
Outcomes

Setting the stage…
Table activity
Protocol – Chalk Talk
Center of chart paper write
Quality Assessments
Without comment…
What are your hunches about the need to build high quality assessments?


Think…Pair…Share
◦ What elements are necessary to assure quality common assessments?
 List these qualities
 Discuss why these are important
Key Questions


Validity Checklist
◦ Standard Alignment
◦ Balance of Representation
◦ Target/Method Match
◦ Quality Items
◦ Test Blueprints
Rubric Review


Activity
◦ Break it apart…see what you have…
Deconstructing Assessments




Are the assessment items tightly aligned with the standards?
Are there an equal number of items per standard? If not, is there “rationale”?
Are there enough items per standard to determine mastery?
Deconstructing Debrief

Please return in
15 minutes
Break


Knowledge – facts and concepts we want students to know

Reasoning – using what they know to reason and solve problems

Skills – students use their knowledge and reasoning to act skillfully

Products – use knowledge, reasoning, and skills to create a concrete product
Kinds of Learning Targets





Selected Response/Short Response
◦ True/false, multiple-choice, matching, fill-inthe-blank, short answers
Extended Response
◦ Essays, research reports and lab reports
Performance
◦ Public performances, investigations
Personal Communication through conversation/observation
◦ Oral exams, interviews, discussion groups
Method of Assessment

Target-Method Match
How well does your method of assessment match your target?
Target to be
Assessed Selected
Response/
Short-Response
Knowledge
Assessment Method
Extended -
Response
Performance
Assessment
Personal
Communication
Reasoning
Performance
Skills
Products




With an “elbow” partner…..
TMM Chart – fill in the grid
Which way may be best?
◦ Good match
◦ Partial match
◦ Not a good match
Target-Method-Match

Target-Method Match
How well does your method of assessment match your target?
Target to be
Assessed Selected
Response/
Short-Response
Knowledge Good match
Assessment Method
Extended -
Response
Performance
Assessment
Personal
Communication
Good match
Not a good match
Partial match
Reasoning Partial match
Performance
Skills
Products
Not a good match
Not a good match
Good match
Not a good match
Partial match
Good match
Good match
Good match
Good match Partial match
Not a good match

Target Method Match


In looking at items on your assessment, might there be an assessment method that could better capture evidence of student understanding of a standard?

What will you stay mindful of as you rethink or develop assessment items to assess standards?

Quality Items

Remember – the development of good items takes time and careful thought
General Item Writing
Guidelines

Parts of a Multiple-Choice
Item
Stem
Correct answer
(Key)
Distractors
(Incorrect Options or Foils)
What is the perimeter of a rectangular vegetable garden with dimensions 6 feet by 8 feet?
A 48 ft
B* 28 ft
C 24 ft
D 14 ft
22

3.
4.
5.
6.
7.
1.
2.
Align items to a standard
Target the appropriate Depth of
Knowledge
Use clear, concise language
Use correct grammar
Use appropriate reading level
Avoid the use of the word “you” and “I”
Avoid using synonyms within the item
General Guidelines

8.
9.
10.
Avoid unnecessary complexity
Don’t assume prior knowledge
Remember: Formatting matters: font sizes, distractor placement, etc.
General Guidelines



Open-ended statement, followed by (usually) 3 or 4 answer choices
Closed question, followed by
(usually) 3 or 4 answer choices
26

Examples
Open-ended stem
One of the factors of x 2 – 5x – 36 is
___ A x + 3
B x - 4
C x + 6
D* x - 9
Closed question stem
Which of the following is a factor of x 2 – 5x – 36 ?
A x + 3
B x - 4
C x + 6
D* x - 9
27

11.
12.
13.
14.
15.
Stuff the stem
Avoid redundancy
Avoid the use of negatives
Avoid clues in the stem
Ensure lead materials are essential to the item
General Guidelines
28

Stems With a Graphic/Stimulus
Lead The stem and leaf plot gives the ages of the people who answered survey questions after buying a pair of roller blades on an Internet auction.
Stem Leaf
5
6
7
3
4
1
2
7 7 8 8 8 8
9 0 1 3 3 5
2 4 6 7
1 3 3 5 7 9
9 4 5
2
0 3
Key:
3 2 means 32
Question
What is the median age of the people who answered the survey questions?
29

Parts of a Multiple Choice
Item
Stem
(Key)
Correct answer
Distractors
(Incorrect Options)
What is the perimeter of a rectangular vegetable garden with dimensions 6 feet by 8 feet?
A 48 ft
B* 28 ft
C 24 ft
D 14 ft
31

16.
17.
18.
19.
20.
Use direct, clear terminology
Use plausible distractors/foils
Use equal length and detail
Make all distractors equally attractive
Organize the options
General Guidelines for Writing
Response Options
32

21.
22.
23.
24.
25.
Have only one correct answer
Do not use overlapping answers
Vary placement of option choices
Good Items are fair items
Avoid using “All of the Above” and
“None of the Above”
General Guidelines
33



A constructed response item is an assessment item that asks students to apply knowledge, skills, and/or critical thinking abilities to real-world, standards driven performance tasks.
It requires a brief written response from students. They often have several parts.
Students have to write, draw, and/or explain their answers.
Constructed Response Items



Sometimes called “open-response” items, constructed response items are so named because they ask students to use their own thinking and background knowledge to develop answers without the benefit of any suggestions or choices.
Constructed response items often have more than one way to correctly answer the question.
Constructed Response Items


Constructed Response items are good to use when you want students to:
◦ Show their work
◦ Explain a process
◦ Compete a chart
◦ Perform a geometric construction
◦ Construct a graph
◦ Identify patterns
◦ Write an essay
Constructed Response Items
37



Tie constructed response items to higherlevel objectives.
This type of item is good to use when you want to test a skill that can’t be easily measured with a selected-response item.
HOTS
Constructed Response Items
38


Two primary types of constructed response items:
◦ Brief Constructed Response
◦ Extended Constructed Response
Constructed Response Items



Require about 1-3 minutes of student response time
Usually represented by one of the following 5 formats:
◦ Fill in the blank
◦ Short Answer
◦ Label a diagram
◦ Visual representation
◦ Show your work
Brief Constructed Response Items



Extended response items require students to provide evidence of understanding regarding a situation that demands more than a selected response or brief constructed response.
They usually involve 20-30 minutes of student response time
Extended Response Items


May require students to reflect and respond in a variety of contexts, such as:
 Write an essay from a prompt
 Take a position on a specific topic and support their stance
 Solve a problem
 Respond to findings of an investigation and/ or experiment
 Respond to written text
Extended Response Items


Guidelines
◦ Carefully word directions and prompts
◦ Allow sufficient time for completion
◦ Have resources necessary for item completion on hand and ready for use
◦ Share with students elements/characteristics of a successful response, where appropriate
Extended Response Items



When designing common assessments, use a variety of brief constructed response items…(these could include short answers, fill-in-the-blank, showyour-work and visual representations) as well as extended constructed response items.
Be sure they are aligned to appropriate
(usually higher-level) learning targets
Constructed Response Items
44


The item should be clear and specific about what students should do.

A Constructed response item may have several questions.

Allow for more than one way for students to respond.
Constructed Response Items
45


Include necessary visual representations such as charts, graphs, pictures, short readings, and cartoons.

Determine points possible for each item.
Constructed Response Items
46


Usually constructed response items are worth 2 or more points depending on the difficulty of the item and the task being performed.

Design a scoring protocol, based on the number of points possible, for each constructed-response item.

Scoring protocols are typically specific to each individual item
Constructed Response Items


Dot Activity
◦ Green = Item is good to go
◦ Yellow = Item may need to be modified
◦ Red = Item is not well-written and needs to be scrapped
Quality Item Hunt

Assessment Blueprints




Did you develop your assessment blueprint prior to developing your common assessment?
◦ Why is this desirable?
Have you reviewed or modified your test blueprint during the development process?
Does your or will your assessment reflect your intended blueprint?
Reflective Questions


Activity
◦ Put it back together…make changes as needed…
Reassemble Assessments

Now What? Next Steps


Ticket out the door…
Wrap Up; Evaluation




Dr. Ed Roeber, Michigan State U roeber@msu.edu
517.432.0427
Dr. Ellen Vorenkamp, Wayne RESA vorenke@resa.net
734.334.1318
Kimberly Young, MDE/BAA youngk1@michigan.gov
517.373.0988
Contact Information