Questioning Your Questions:
Critically Examining Science
Assessment Items
Presented by
Phyllis Kirkpatrick, senior science program coordinator
Barbara Taylor, senior science program coordinator
at NSTA’s 56th National Conference on Science Education, Boston, MA
1
Reflection
Do I have the knowledge and skills to generate
assessment items for:
Classroom assessments?
District (including benchmark) assessments?
State assessments?
2
Session Objectives
• Increase participants awareness of the
purposes and types of assessments.
• Introduce participants to assessment item
guidelines and test construction guidelines.
• Practice using assessment item guidelines to
edit selected items.
3
Looking at Assessments
What do YOUR tests TEST?
4
Purposes of Assessment
1.
Diagnose student strengths and weaknesses
2.
Assign grades
3.
Determine instructional effectiveness
4.
Influence the public’s perception of educational effectiveness
5.
Guide instructional decisions
6.
Hold schools accountable
7.
Sort, select, rank, and compare students and schools
8.
Evaluate instructional programs
9.
Allocate resources
10.
Match students to a special program
5
“The purpose of assessment
MUST BE to improve student
learning. Period.”
Cathy Seeley,
Past President of NCTM
9/94
6
Use a Variety of Assessments
Essays
Performance Tasks
Hands-on Tasks
Logbooks or Journals
Portfolios
Selected Response
7
Know where you are going
“To begin with the end in mind means to start
with a clear understanding of your
destination. It means to know where you are
going so that you better understand where
you are now so that the steps you take are
always in the right direction.”
Stephen R. Covey
The Seven Habits of Highly Effective People
8
Selected Response Questions
Remember—no test question is perfect; it can always
be made better.
9
Question: Does the item match the depth and
complexity of the identified standard?
Which of the following is a sex-linked characteristic?
A.
Blood type
C. Color blindness
B.
Eye color
D. Tongue rolling
What is the question asking the student to do?
The question is asking the student to recall.
10
Question: Does the item match the depth and
complexity of the identified standards?
Color blindness is a sex-linked characteristic.
What are the chances of a normal male and a
Color-blind female having a color blind son?
A. 0%
B. 25%
C. 75%
D. 100%
This question requires the
student to make a prediction
based on an understanding
of the heredity of sex-linked
characteristics.
11
Question: Is there only ONE arguably correct
response, regardless of the teaching?
Which of these will result if all colors are mixed?
A.
Black
C. Green
B.
White
D. Yellow
The question is ambiguous:
Are the colors ‘light’ or
‘pigment’?
12
Question: Do questions avoid negatives/prompts?
Which of these is NOT a characteristic of animals?
A.
B.
C.
D.
They
They
They
They
eat other organisms.
can move.
do not make their own food.
have cell walls.
13
Question: Do questions avoid negatives/prompts?
Which of these is a characteristic that all animals
have in common?
A.
B.
C.
D.
Unicellular
Photosynthetic
Multicellular
Carnivorous
This question requires the
student to identify the
characteristics of animals
and other organisms.
14
Question: Does the item ask a question? Does
it avoid incomplete thoughts?
________describes characteristics you can detect
with your senses.
A.Physical properties
B.Exterior properties
C.Appearance
D.Chemical properties
This item has multiple
correct answers.
15
Question: Are any of the answer options clued by the
words in the stem?
The Millennium Seed Bank is a project designed to
protect the future of __________.
A. Sperm
B. Eggs
C. Plant species
D. Animal species
The correct answer is
clued— most students
could relate seeds
to plants. Or, it is possible
that it can be answered
only if this is specifically
taught.
16
Reviewing and Editing Test Items
17
Test Items
1. The item stem should ask a question.
2. Items should be written in active voice using simple
sentences.
3. Artwork should provide useful information and be
necessary for answering the question.
4. Items should avoid using superfluous language.
5. Make sure that the information in the item stem and
options is factual.
18
6. The items should focus on important content, not on trivial
details.
7. The reading level of the item should be from a half year
below grade level to a half year above grade level. Keep in
mind that scientific terms tend to increase the reading level.
8. Items should avoid obvious biases.
9. Avoid using negatives in the stem such as not and least.
Edit these items to become except items, with except at the
end of the stem.
10. The items should be written clearly enough so that
there is only one arguably correct answer. The wording of
the stem should not allow for wiggle room.
19
Test Item Options
1. Item options need to be parallel. This means that the
length and wording of each option should be checked so
that students are not drawn to one option over the others.
2. Item options should not be clued by words in the stem.
3. If the options are numerical, they should be written in
ascending or descending order, not in random order.
4. All incorrect options should be logical. If they are
mathematical, they should be based on miscalculations
students are likely to make. The incorrect options should
give a teacher an indication of what the students were
thinking or the miscalculation they made that caused them
to choose the incorrect option.
20
5. In items that require calculations, check to make sure
that a correct answer cannot be obtained by doing the
mathematics the wrong way. In other words, a student
should not be able to randomly add numbers found in
the problem and get the correct answer.
6. Avoid using repetitious information in options.
7. Avoid having opposites as options.
8. Avoid overlap in options, especially in numerical answer
sets.
21
Practice on Your Samples
You have some sample items in your packet. Work
with a partner and be prepared to report out:
Which of the items need to be improved?
How did you and your partner make the item
better?
22
When Constructing a Test
1. Test items should cover the full range of Bloom’s taxonomy.
2. No more then 10% of test items should be at the knowledge
level.
3. Try to have at least 5 items for each main idea being tested.
4. Place easier items at the beginning and end of the test. The
harder items should be in the middle of the test.
5. Try to place items on the test so that there is adequate white
space. Do not place artwork/scenarios on a page different
(adjacent or back side) from the page the related item(s) is
on.
23
6. Distribute the answer choices randomly; try not to
have the same answer more then three times in row.
7. Make sure that the length of the test is appropriate.
Most students should be able to complete the test in
the class period. Rule of thumb—multiply the time it
takes for you to take the test and make the answer key
by 4 to roughly determine the average time your
students will need to complete the test.
8. Make sure items don’t key the answers to other items.
The stem of one item should not contain information
that answers another item.
24
Teachers should NOT have to
put aside GOOD INSTRUCTION
to prepare for a TEST…
25
Reflection
1.
2.
3.
What did I learn about my practices and beliefs
regarding assessment item design and test
construction?
What action steps would I like to take in assessment
item design and test construction this coming school
year?
What was the most important thing I took away from
this experience?
26
Resources
Marzano, Robert J., Pickering, Debra J., and Pollock, Jane E. (2001). Classroom
instruction that works: Research-based strategies for increasing student
achievement. Alexandria, VA: Association for Supervision and Curriculum
Development.
National Research Council. (2001). Classroom assessment and the National
Science Education Standards. Committee on Classroom Assessment and the
National Science Education Standards. J. Myron Atkin, Paul Black, and Janet
Coffey (Eds.). Center for Education, Division of Behavioral and Social Sciences
and Education. Washington, DC: National Academy Press.
Sevenair, John P., and Burkett, Allan R. (2001, Fall). “Materials to accompany a
presentation on multiple choice questions, examinations, and test banks.”
Unpublished paper, Xavier University of Louisiana. Available at
http://webusers.xula.edu/jsevenai/objective/ObjectiveTests.pdf
Stiggins, Richard J. (2002, June). “Assessment crisis: The absence of assessment
FOR learning.” Phi Delta Kappan. Retrieved November 16, 2005, from
http://www.pdkintl.org/kappan/k0206sti.htm
27
Contact Information
Phyllis Kirkpatrick
pkirkpat@mail.utexas.edu
Barbara Taylor
Barbara_taylor@mail.utexas.edu
28