The following materials were
prepared for use by CARS and
SASS consultants.
Finding Measures and Creating
New Ones
Presented by:
Kara Owens
Overview
o Creating your own instrument
o How to write items for your objectives
o Knowledge Based Objectives
o Attitudinal/Developmental Objectives
o Examples and Handouts
Overview of Writing Items
• General Guidelines
• Knowledge-Based Items:
–
–
–
–
Multiple-Choice
True-False
Matching
Short answer/fill in blank
• Attitudinal/Developmental Items
– Multiple Response/Checklist
– Ranking
– Order
But I don’t know how to write items!!
• Writing items gets easier with practice
• Don’t be frustrated if you find it
challenging, most people do
• It is much easier once the objectives have
been written
General Guidelines:
Where do I start?
• Start by creating a test blueprint or table of
specifications.
– This is a table that organizes objectives
according to their level of importance with
respect to a specific course or program
– Each objective is given a weight (percentage)
according to how important it is or how much
course time is spent covering that objective
General Guidelines:
Test Blueprint Example
Test Blueprint for Health 101
Objectives to be obtained by the end of Health 101:
Percentage
1.
Students should be able to recall the major food groups 10%
and their recommended daily servings.
2.
Students should be able to identify major muscles and 40%
provide examples of exercises to work those muscles.
3.
Students should be able to demonstrate knowledge of 40%
proper weightlifting techniques.
4.
Students should engage in activities that promote a 10%
healthy lifestyle.
Total Percentage:
100%
General Guidelines:
Test blueprint to test items
•
Based on the test blueprint, items should be
written to adequately cover the objectives
Health 101 Example:
– We decide that we want our instrument to be
20 items long, based on the test blueprint…
•
There would be two items (10%) for objectives 1
and 4 and eight items (40%) for objectives 2 and 3
General Guidelines:
Getting Started
•
Things we should know before trying to
write items:
1. Goals and objectives for the program or
course
2. The level of importance for each objective
(e.g. test blueprint)
3. An approximate number of items for the
total test
4. An approximate number of items for each
objective
Knowledge-Based Items:
The Key to Writing Good Items
Use your objectives to write your items!
– If your objectives are well-written (i.e.
observable, measurable, and specific) they
should help you focus the types of items that
can be written to assess them.
• Look at the verb!
Knowledge-Based Items:
Remember Bloom’s Taxonomy
Bloom’s Taxonomy
Less
Complex
More
Complex
Level
Description
Knowledge
Recognize facts, terms, and principles
Comprehension
Explain or summarize in one’s own words
Application
Relate previously learned material to new
situations
Analysis
Understand organizational structure of
material; draw comparisons and
relationships between elements
Synthesis
Combine elements to form a new original
entity
Evaluation
Make judgments about the extent to which
material satisfies criteria
Knowledge-Based Items:
Bloom’s Taxonomy Verbs
Bloom’s Level
Verbs
Knowledge
arrange, define,identify, label, list, match, name,
order, recall, recognize, reproduce, select
Comprehension
classify, describe, discuss, explain, express, indicate,
locate, report, restate, review, summarize,
Application
apply, demonstrate, dramatize, employ, give
examples, illustrate, schedule, sketch, solve, use
Analysis
analyze,categorize, compare, contrast, diagram,
differentiate, distinguish, illustrate, outline
Synthesis
arrange,collect, compose, construct, contrast, create,
design, formulate,organize, plan, predict
Evaluation
appraise, argue, assess, criticize, critique, defend,
estimate, evaluate, interpret, judge, rate
Knowledge-Based Items:
Using verbs to write items
• Verb
– will determine how complex the item should be
– Will determine what type of item is written (e.g. multiple-choice,
matching, true-false)
• Example:
Objective: Student will identify the types of questions to ask and
what not to ask during an informational
interview
Identify is at the first level of Blooms taxonomy, indicating that it
requires less complex, more general knowledge of the construct in
order to obtain this objective.
Knowledge-Based Items:
So how do I write an item to measure an objective?
•
•
Use the verb!
How can you assess if a student can
identify something?
1. Have them select it from a list
2. Have them pick it out from a scenario
Knowledge-Based Items: So how do I write
an item to measure an objective?
•
Example:
1.
Hilary is going to a job fair on campus where she plans
to meet with several hiring social work agencies, which
of the following is an appropriate question for her to
ask her interviewer?
A.
B.
C.
D.
What is the salary for a first year employee?
What are the job expectations you have for a new employee?
What kind of benefits do you offer to new employees?
What is your political party affiliation?
Knowledge-Based Items:
Example
• This question allows a student to demonstrate
knowledge of both what should and should not be
asked during an informational meeting.
• Now that you know how to use your objectives to
write items, you need some tips on constructing
clear items that measure examinee ability while
minimizing some common item writing errors.
• I will discuss some general guidelines for all
types of items and then provide direction for
writing items in specific formats.
Guidelines for
Knowledge-Based Items
•
The guidelines can be separated into 4
specific categories:
1. Language/Wording
2. Complexity
3. Item Patterns
4. Directions
Language/Wording
1.
Avoid bias (e.g., age, ethnicity, gender or disabilities).
2.
Avoid “loading” the questions by inadvertently incorporating your
own opinions into the items. This can bias the results and is
particularly important with attitudinal items.
 Example of “loading”:
Is there any reason to keep this program?
Yes
No
 Revised:
Does this program offer material not obtained in other courses?
Yes
No
Language/Wording
3. Avoid unclear or ambiguous wording.
 Example of unclear wording
Student that are not taking a psychology course but are taking a
psychology internship but have not yet fulfilled the requirements of
their practicum experience should come to the seminar but not attend
the training workshop.
True
False
4. Avoid the use of always, never, constantly.
Language/Wording
5.
Avoid double-barreled items (i.e., items that express
two or more ideas).



Example of a double-barreled item:
I like to work out at UREC at least 3-5 times a week.
Yes
No
If one disagrees with this statement, with which part does he or she
disagree? Does the person dislike working out at UREC? Does the
person not like to work out at all? Does he or she work out only 2
days a week instead or 3-5?
Revised
I like to work out.
I like UREC.
I like working out at UREC.
Yes
Yes
Yes
No
No
No
Language/Wording
6.
Avoid giving clues to the correct response through wording.


Example of giving clues:
When the two main characters in the book went outside they tossed
around an
A. baseball
B. tomato
C. apple
D. football
Revised
When the two main characters in the book went outside they tossed
around a(n)
Complexity
1. Be concise. Long, complex items can lead to
confusion.
2. Use the appropriate reading level for the
population that you are testing.
3. Be clear. Items should challenge the test-taker’s
knowledge, not the ability to uncover hidden
meanings.
Item Patterns
1.
Make sure that items are unrelated and do not give hints about the
correct answer to other items.
2.
Vary the position of the correct answer among the response options
(i.e., don’t have the correct answer most often in the “b” location).
To aid in this, make a frequency table of the number of times a
response was used, and try to use each option an equal number of
times.
Frequency Table
Correct Answer
# of times used
A
10
B
9
C
11
D
10
Total: 40 Items
40
Directions
1.
2.
3.
4.
All directions should be concise and easy to understand.
The use of bold type or a different font may help set off the
directions from the items by getting the test-taker’s attention.
Test directions:
– Refer to all items.
– May be read out-loud by the person administering the test.
– Should be placed at the top of the test.
– Include general information about the testing session and
approximately how long the test will last.
Item directions:
– Give specific directions about how to complete the items.
– Should be placed directly above the related section if the type
of item changes.
Close-ended Format
•
Examinee selects the correct answer from
choices provided.
•
We will review:
1. Multiple-Choice
2. True-False
3. Matching
Multiple-Choice Items
• For the bread, cereal, rice and pasta group, the food
pyramid recommends (stem)
• a.
2-3 daily servings (distractor)
• b.
3-5 daily servings (distractor)
• c. 4-6 daily servings (distractor)
• d.
6-11 daily servings (correct answer)
Multiple-Choice Items
1. Make answer choices brief, not repetitive
2. Offer at least 3 –5 answer choices. The aim is to provide
variability in responses; however, it should be noted that
“optimal” number of answer choices has been debated.
3. Write stems and answers in third person, rather than
second person.
 Example in the second person:
“What might happen if you get caught drinking on campus?”
 Revised in the third person:
“What might happen if a person gets caught drinking on campus?”
Multiple-Choice Items
4.
Use italics and boldface appropriately.
– Italics should be used for names of books,
periodicals, plays, poems, and films.
– Boldface should be used for emphasizing negatives.

Examples:
• In the novel The Catcher in the Rye, what was the main character’s
name?
• Which of the following is not an option when you receive a letter
from Judicial Affairs?
Multiple-Choice Items
5.
Be consistent when using “which” and “what” in the
question stem.
– “What” typically indicates that there is only one
right answer for the question.
– “Which” usually refers to “which of the following”.

Examples:
• What does UREC stand for?
Obviously there’s only one correct answer for this question.
• Which is a group fitness program offered by UREC?
There are several programs offered, but only one should be
listed as an answer choice.
Multiple-Choice Distractors
•
Distractors are alternative, incorrect answer
choices for multiple-choice items.
Distractors should:
1.
be related to the question and similar to the correct
answer choice.
2. not be easily identified as wrong choices.
3. be written in a similar style and manner (i.e. not
noticeably longer or shorter) as the correct choice.
4. attract examinees who have some knowledge of the
material, but who have not yet fully comprehended
all of the subject matter.
Distractors Should:
5. avoid the overuse all of the above, none of the
above or combinations such as A and B as
options. There are several reasons for avoiding
their use.
–
–
–
It is tempting to overuse them because they are easy to write.
Students with partial knowledge of the question may be able to
answer correctly by process of elimination. For example, a
student may know that two out of three response options are
correct.
These items may make it harder to discriminate between those
who fully know the subject matter and those who do not.
True-False Items
•
There are two types of true-false items:
1.
2.
right-wrong
yes-no.
Guidelines for Writing True-False Items:
1. Give clear directions for answering questions. Follow
the rules outlined under General Guidelines.
2. Write items such that they do not alert students as to the
correct answer. Alternate the length for both true and
false items to avoid this mistake.
3. Avoid using negatively worded items, as they tend to
lead to confusion. If the statement has to be negatively
worded, bold the negative word.
True-False Item Examples
Example:
Instructions
Please indicate whether you think the following statements for items
1-10 are True or False by circling the right answer.
Right-Wrong Question
Students are allowed to choose which dorm they want to live in on
campus.
True
False
Yes-No Question
If a student is placed in a temporary triple room, are they
guaranteed to be reassigned to a double room?
Yes
No
Matching Items
•
Matching items involve a set or list of related ideas and responses.
This format is often used when assessing a great deal specific
facts.
Guidelines for writing matching items:
1.
Provide specific directions regarding how and where to mark
answers.
2.
Limit the use of matching items. They often involve lower levels
of processing.
3.
To decrease the likelihood of guessing, provide more response
options than items in the list. (e.g., offering eight response options
for a list of five items).
Matching Items
Guidelines Continued:
5.
Make the list of responses or options uniform in type. In the
example given below, all of the responses are types of group
fitness classes; no other option types are listed.
6.
Keep the number of choices reasonable (e.g., between 5 and 12
options). When lists become long, it requires examinees to spend
too much time searching for answers.
7.
List response options in a set order. Words should be listed in
alphabetical order; dates and numbers can be arranged in either
ascending or descending order. This makes it easier for an
examinee to search for the correct answer.
Matching Example
Instructions:
Match the statements in items 1-5 with the appropriate group
fitness class (A-F) they describe from the list provided to the
right.
1. Workout which involves dancing
A. Athletic Box
2. Class includes aerobic moves at a moderate level
B. Cardio Blast
3. Workout includes drills and strength training
C. Cycle Fit
4. Workout involves boxing and athletic drills
D. Energy Circuit
5. Class involves flexibility and relaxation
E. Funk
F. Yoga Fitness
Alphabetical
order
More response options than items
Questions
• Any questions about the general
guidelines?
• Any questions about close-ended items?
• Any questions about knowledge-based
formats?
Constructed Response Items
•
Require the examinee to formulate and write
the answer to the question
•
Responses are typically a few words measuring
factual recall.
•
Types:
1.
2.
Short-answer
Sentence-completion
Guidelines
Guidelines for writing short-answer and sentencecompletion items :
1. The item should be written so that the examinee infers that there is
only one answer that is reasonable.
2. Phrase the item so that the examinee knows that the answer should
be concise.
3. Avoid items with several blanks in the sentence, which may be
confusing or unclear.
Example:
At Warren Hall and __________ on campus and students can go
to ___________.
Revised:
List three places where can students go to find out what movies are
playing on campus this semester?
Guidelines Continued
Guidelines Continued:
4. Place blanks near the end of the sentence to minimize
confusion.
5.
Make blanks the same length and long enough for the
longest answer to fit. If the blanks vary in length,
examinees may be able to determine the answer based
on the length of the line.
 Example:
If problems with a roommate arise, students should contact their
_________________.
Example
• Example of an objective:
– Students should be able to identify key employability
traits desired by employers.
• How can we write items using the various
formats for this objective?
• Break into groups and write items for the
objective using the various formats:
– Multiple choice
– True/False
– Matching
Example
• Multiple-Choice:
– Which of the following is the trait deemed MOST
important by MOST employers?
A. Interpersonal Skills
B. Leadership
C. Time Management
D. Flexibility
• True-False:
– Employers want new employees that focus on the solo
completion of tasks.
True
False
Example
Matching:
1. Working with others on a project
A. Flexibility
2. Working overtime to complete a project
B. Interpersonal Skills
3. Creating and documenting procedures for writing up a report
C. Leadership
4. Completing project on time
D. Organization
5. Supervising the progress of a group of coworkers
E. Reliability
F. Teamwork
G. Time Management
Example
• Now what if the objective were:
– Students should be able to illustrate how their life and
educational experiences have helped them develop key
employability traits.
• This objective will require a higher level question
because it a more complex objective.
 Example of Short answer question:
– How has this course and your prior work experiences
prepared you for a career in your desired field?
Example
• Now what if the objective were:
– Students should be able to demonstrate key
employability traits.
– This type of objective requires a performance
assessment to evaluate
Questions
• Any questions about knowledge-based
items?
• Questions about different formats and
guidelines?
Attitudinal/Developmental Items
• Measure self-reported feelings or interests
• No right or wrong answers
• This description will only include close-ended formats.
 Example Attitudinal Item:
I learned a great deal about my personal preferences from the Career and
Life Planning Course.
Strongly Disagree
Disagree
Neither Agree
Agree
Strongly Agree
Nor Disagree
 Example Developmental Item:
I have carefully thought out opinions regarding my place in the world
Strongly Disagree
Slightly Disagree
Slightly Agree
Strongly Agree
Attitudinal/Developmental Items
Guidelines for writing attitudinal/developmental
items:
1. As much as possible, statements and questions should
be written in the present tense. Items written in past or
future tense may result in confusion and mistaken
responses.
2. Statements should be clearly written and have only one
meaning or interpretation.
3. Instructions should mention that there are no right or
wrong answers.
4. Statements and questions should be short (no more than
20 words) and simple.
Guidelines Continued
Guidelines Continued:
5. Try to avoid the use of the words “if” or
“because”, which may complicate the sentence.
6. Avoid use of the following words: not, none,
never, all or always.
7. Response categories should not overlap.
8. Individual items should focus on one idea.
Types of
Attitudinal/Developmental Items
1. Checklist/Multiple-Response
1
2. Ranking Scales
3. Ordered Scales
2
3

Checklist
Example Checklist:
From the following list of events being considered by the University
Program Board, please check off ( ) all of the events that you would
consider attending.
•Comedy events
•Multicultural events
•Formal Dances
•Musical Events
•Halloween costume contest
•Plays
•International film festival
•Poetry readings
•Movies
•Talent show
Multiple Response
Example Multiple Response:
Which of the following on-campus movie showings do you attend?
Check all that apply.
Evening Shows
Afternoon Shows
I do not attend on-campus
movies
Ranking Scales
• Ranking scales are used to order or rate things as
they relate to one another.
• A respondent rank-orders a list based upon his or
her attitude regarding the topics listed.
• Limit rankings to no more than five items. When
there are too many items to be ranked, it may get
confusing and respondents may misnumber the
items.
Ranking Scales
Example of a Ranking Scale:
Please rank the following statements in order from
1(highest) to 4 (lowest) according to how well your advisor
succeeds in accomplishing the task.
Assists student in developing realistic goals
Refers student to available resources
Assists student in planning an appropriate program
Monitors student’s academic progress
Ordered Scales
•
Ordered scales are composed of items that are combined
to yield a score that provides a measurement of one’s
attitude concerning a particular construct.
Types of ordered scales:
1.
2.
3.
4.
5.
6.
Likert-type
Frequency
Satisfaction
Rating
Intensity
Comparison
7.
Influence scales
Ordered Scales
1.
Likert-type item:
o
I like to go on hikes.
Strongly
Disagree
Disagree
Neither Agree
Nor Disagree
Agree
Strongly
Agree
2. Frequency Item:
o
I enjoy going to the gym.
Never
Rarely
Sometimes
Frequently
Always
Ordered Scales
3. Satisfaction
o
Are you pleased with the courses being offered?
Very
Satisfied
Satisfied
Neither Satisfied
Nor Dissatisfied
4. Rating
o I thought the concert was…
Excellent
Good
Fair
Poor
Dissatisfied
Very
Dissatisfied
Ordered Scales
5. Intensity
o
The day after I drink, the side effects I
experience are:
None
Mild
Moderate
Severe
Very Severe
6. Comparison
o
When I go out, I drink…
Much more
than others
Somewhat more
than others
About the same
Somewhat less
Much less
as others
than others
than others
Ordered Scales
7. Influence
o
I think that my drinking behavior is…
Very big
problem
Big
problem
Moderate
problem
Small
problem
Very small
problem
No
problem
Mistakes
• Pay careful attention to what you are asking the
students to respond to with ordered scales.
• For example, DON’T use an intensity item with
a satisfaction response options
 Example:
The course assignments were tailored to the course material.
None
Mild
Moderate
Severe
Very Severe