Constructed Response
Is it worth the effort?
Up Front Information
Mark Vanacore
Technology Coordinator
Albion Central School District
Formerly Teacher of
Physics, Earth Science, Chemistry,
Mathematics, Computer Programming,
Cisco Networking
Disclosure
I am 100% in favor of …
NYS State Regents Exams
Common Core Testing
Computer Based Testing
Making some changes in the format of State
Summative Assessments
There are things I wonder about…
Free Response Questions that aren’t
Math Questions that are allowed sometimes
but not others, on both exams
Science DBQ Good Bad ???
Vague Acceptables
Not at all Limited to Science Exams …
Things I like … Unexpected New
Questions…
Formative vs Summative
Assessments
Goals :
Formative - provide feedback to student and
teacher concerning the student’s ability to
respond to questions and problems.
Summative - A check to determine the level of
competency concerning the covered content.
Confusion about the Purpose of
State Exams
State exams are about determining
competency on the covered material.
Question:
Can we determine if a student understands
something with a multiple choice question?
Some Vocab Anchors
Anatomy of a Multiple Choice Question
A root question
An answer to the root question
Several Distractors
Vocab Anchors continued
Anatomy of a Constructed Response
Question
Some Information - Data, Tables, Passage
A Root Question
A Blank bit of Paper
A Scoring Guide with the intensions of the test
writer.
Some Testing Vocabulary
Reliability
Test - Retest - same results over time?
Parallel Forms - Form A vs Form B?
Inter-rater - Do all teacher grade the same?
*Internal Consistency ? Checks within test?
Validity Does it test what it says it tests?
Concerning State Science Tests
In General, State Science Exams ...
Pass all Reliability measures
Pass all Validity Tests
Methodology
The data was pulled from copies of the
scanable student forms.
The total points accumulated in each part of the
exam
The raw parts were converted in %’s for
comparison
Methodology
Exams Looked at took place in June 2014
Earth Science *
Living Environment *
Chemistry
Physics
Integrated Algebra *
Common Core Algebra *
Geometry
Trig
Data Concerning the 2014 Earth
Science Exams
Sample Size 96 Students Range of Scaled Scores 44 - 98
32 Mastery 85+
21 College Ready 75-84
24 Passing 65-74
14 Gandolfed 0-64
Comparison of MC to Overall Score
Correlation between the MC % Correct and the
overall Scale Score.
Correlation Coefficient .9596
Very High Correlation - Score well in the MC
Score Well on the Test
Correlation Graph
Correlation between CR and Overall
Score
Correlation between % of CR Points received
and overall score
Correlation Coefficient .9351
Still pretty High
Score well on the CR score well on the test
Correlation Graph
Correlation between Practical and
Total Score
Points Received on Practical compare to Scale
Score
Correlation Coefficient .6311
Much Lower but expected, the Practical is a
series of skills taught throughout the year which
kids learn well, range of scores 8-16 with an
average of 14/16 points.
What more can we learn?
MC to Score .9596
CR to Score .9351
Either score is good predictor of overall
performance.
But My lower kids need those Constructed
response questions…
Do they Really ???
I Subtracted the CR percent from MC Percent
example %MC-%CR = Some Number
Mastery Students Averaged -4.22
College Ready Averaged +1.52
Passing Averaged +3.96
Gandolfed Averaged +4.86
Take Away ...
The Test as is could be modified to eliminate
the Constructed response from the overall
score without significantly altering the outcome
of the test.
A simple “scantron” form would be sufficient to
determine the range of student level
Mastery-5 College Ready-4 Passing-3 Nearly
Passing-2 Not Passing-1
The Danger …
The impression that free response is not ever
needed.
Free Response is critical in Formative
Assessment
It is the most practical way to determine if the
student can complete complex processes to get
at an answer.
Getting a little more out of MC
MC with Explanations
What is the correct answer and how do you
know?
Of the Distractors which one is the most
correct and Why?
How could you change the root of the
question to make one of the distractors
correct?
Turning CR into MC
Ask Students to turn a Lab Question or CR
from an old exam into a MC Question.
Getting student to think like test writers may
help improve their scores.
Other Tests - Living Environment
Sample Size 59
Mastery 19
College 19
Passing 14
Gandolfed 7
Correl MC to Score .9592
Correl CR to Score .9110
Integrated Algebra Scores
Sample Size 132
Mastery 23
College 58
Passing 31
Gandolfed 20
Correl MC to Score .9106
Correl CR to Score .8033
Scatter Plot MC vs Scale Score
Scatter Plot CR vs Scale Score
Some Confusing Data
Diff MC - CR
Mastery +24
College +34
Passing +39
Gandolfed +31
It was a few points in science what happened?
Scatter Plot MC vs CR Integrated
Algebra
A change in Focus and
A Trap Revealed
Math teachers focused on MC questions
teaching tricks and tips to maximize the chance
of selecting correct response.
Including but not limited to back substitution of
distractors to find the answers.
Why? In Algebra you only need about 35
correct responses in part 1 to Pass the test.
Common Core Algebra Results
Sample Size 117
Mastery 1
College 6
Passing 69
Gandolfed 41
Correl MC to Score .8361
Correl CR to Score .7713
Diff MC to CR
Master -6 (1 Student)
College +19
Passing +21
Gandolfed +25
There are significant difference in
expectation between the Integrated
Algebra test and the Common Core
Algebra Test.
Common core question are more
about understanding and application
rather than plug and check
methodologies.
Summary Myth Busted
We do not really need elaborate Constructed
response questions to determine if kids have
met the criteria for being awarded a passing
grade and awarded Regents Credit.
Well designed MC questions can get the
information we need, and we have plenty of
really good questions across the content.
Summary Myth Confirmed
We still need free response question to ensure
that our kids have the skill needed to select the
correct multiple choice answer.
Understanding and being able to apply
knowledge is still important regardless of the
format of the summative assessment.
Raw Data Slides
www.albionk12.org/forum/presentations/
Spread Sheets in Excel
Slide Show in PowerPoint 2010
Credits
Statistician: Marge Innovera
Chairman, Math Dept. Horatio Algebra
Chairman, Staff Physics Dept. Victor Analysis
Chairman, Underemployment Study Group Art Majors
Chief Accountant Candace B. Rittenoff
Child Care Provider A. Hugh Nokitov
Complaint Line Operator Xavier Breath
Downsizing Consultant Candace Guy
Special Thanks to My Teachers for Copying the answer
sheets so I could collect the data.