Computation-based versus concept-based test
questions: High school teachers’ perceptions
Luanna Gomez and Daniel MacIsaac
Physics
Joseph Zawicki
Earth Sciences & Science Education
SUNY College at Buffalo
Buffalo, NY 14222
<gomezls@buffalostate.edu>
1
Abstract
The New York State Regents examination in physics is a
standardized assessment of high school students' competencies after
completing a year of introductory physics. The analysis of select items taken
from 1500 students will be provided. We have compared traditional
computational problems to less traditional conceptual problems to examine
the extent to which the response pattern provides insight to the difficulty of the
two types of questions. This discussion will form a context in which teachers'
perceptions of the nature of numeric and concept questions, of their relation to
physics understanding, and of their implications to physics instruction.
2
False dichotomies
We examined select items from recent New York State Regents
examinations, including response analysis from about 4500 high school
student papers. Student response data revealed the conceptual items
were amongst the most difficult, which was surprising to several
vociferous high school physics teachers interviewed.
3
Student population and context



Urban, suburban, and
rural school districts
NYS Regents exam in
physics
The secondary physics
exam is administered in
June to Grade 11 & 12
(N = 4500+ students)
4
Regents’ examination background

The exam‡:
• is aligned with New York State core curriculum¶;
• has been administered since 2007; and
• contains 3 sections (~ 60 items):
• multiple choice,
• constructed response (e.g. requires a short answer or
calculation), and
• extended constructed (e.g. requires a written passage
or multi-step calculation)
‡See,
for example,
http://www.nysedregents.org/testing/scires/regentsphys.html
¶The NYS core curriculum may be viewed at http:///www.nysed.gov
5
June 2008 secondary-level student
data (Example item 1)
Item 1
Major learning goal: To be able to
discern that measured quantities
can be classified as either a
vector or a scalar.

Discrimination index: D = 0.80
Concept multiple-choice question
The speedometer in a car does
not measure the car’s velocity
because velocity is a
(1)
vector quantity and has
magnitude and direction
(correct).
(2)
vector quantity and does not
have a direction associated with
it.
(3)
scalar quantity and has a
direction associated with it.
(4)
scalar quantity and does not
have and does not have a
direction associated with it.
6
June 2008 secondary-level student
data (Example item 2)
Item 2
Major learning goals: To
recognize that an object or system
has a kinetic energy associated
with its velocity, and that its
gravitational potential energy
depends solely on the relative
positions of the objects in that
system.

Discrimination index: D = 0.48
Concept multiple-choice question
A car travels from point A to
point B at constant speed up a
hill. As the car travels its
gravitation potential energy
(1)
increases and its kinetic energy
decreases.
(2)
increases and its kinetic energy
remains the same (correct).
(3)
remains the same and its kinetic
energy decreases.
(4)
remains the same and its kinetic
energy remains the same.
7
June 2008 secondary-level student
data (Example item 3)
Item 3
Major learning goals: To
recognize that an object or system
has a kinetic energy associated
with its velocity, and that its
gravitational potential energy
depends solely on the relative
positions of the objects in that
system.

Discrimination index: D = 0.75
Short calculation item
A 65kg pole vaulter wishes to
vault at a height of 5.5m.
Calculate the minimum amount
of kinetic energy the vaulter
needs to reach the height if air
friction is neglected and all the
vaulting energy is derived from
kinetic energy. [Show all work,
including the equation and
substitution with units.]
Correct answer: KE = 3500 J
(rounded to 2 significant figures)
8
June 2008 secondary-level student
data (Example items 4-6)
Major learning goal: Interpretation
of graphs
Item 4
 Discrimination index: D = 0.97
Item 5
 Discrimination index: D = 0.93
Item 6
 Discrimination index: D = 0.45
Multi-step calculation item
Item 4: Plot the data points for the dart’s
maximum vertical displacement versus
spring compression. [Use the information in
the data table (not shown).]
Item 5: Draw the line or curve of best fit.
Item 6: Using information from your graph,
calculate the energy provided by the
compressed spring that causes the dart to
achieve a maximum vertical displacement of
3.50m. [Show all your work, including
equation and substitution with units.]
(Solution not shown)
9
Teacher data collection (preliminary)
Individual teacher interviews (N ~ 3)
Teachers were asked whether they believed
computation-based (i.e., formula-driven) problems
were more or less challenging for students than
concept-based (i.e., qualitative) problems on the
Regents’ exam in physics/physical science—they
believed that the calculation problems were more
difficult than conceptual ones!
10
Tentative conclusion



Conceptual items may be more difficult than most
calculation-driven ones
Teachers may not appreciate the value of concept
questions and dismiss the results because they are
perceived as “trick” questions.
On the basis of these results, teachers may be
spending more time preparing students for the Regents
exam in physics with equations rather than concepts.
11
Implications
 Some
skill sets, such as inscription, remain
more difficult for students.
 Mundane skill sets, such as plotting points
and solving for commonly rehearsed
variables, appear to be readily achieved
12
Acknowledgements
SUNY Buffalo State College
Kathleen Falconer, Elementary Education and Reading
Western New York Regional Information Center
Timothy Johnson, Erie 1 BOCES
13