BEYOND
MULTIPLE CHOICE
QUESTIONS
KRYSTAL K. HAASE, PHARM.D., FCCP, BCPS
ASSOCIATE PROFESSOR
TEXAS TECH UNIVERSITY HEALTH SCIENCES CENTER
SCHOOL OF PHARMACY
OBJECTIVES
• Describe the pro’s and con’s of using
different question formats other than
multiple choice.
• Identify and resolve common problems
when constructing open-ended questions.
• Develop standardized grading procedures
for open-ended questions.
• Discuss exam length and other challenges
when using open-ended question formats.
ASKING THE RIGHT
QUESTION
REVIEW
Bloom’s
Evaluation
High
Synthesis
Analysis
Application
Comprehension
Knowledge
Low
QUESTION TYPES
Recognition
Recall
Selection
Response
Construction
Response
QUESTION TYPES
Selection
Response
Construction
Response
True / False
Fill in the Blank
Matching
Short Answer
Multiple
Choice
Essay
TRUE / FALSE
• Pro’s
• easy to write
• easy to score
Evaluation
• limited ability to assess mastery
• high probability of guessing
• Best Use:
• Dichotomous, factual info
Bloom’s
• Con’s
Synthesis
Analysis
Application
Comprehension
Knowledge
MATCHING
• Pro’s
• Can assess a lot of info in a confined space
• Fairly low probability of guessing
Evaluation
• Assess recognition not recall
• Best Use:
• Knowledge recall
• Add additional “distractor” items
to increase rigor
Bloom’s
• Con’s
Synthesis
Analysis
Application
Comprehension
Knowledge
MULTIPLE CHOICE
• Pro’s
• Very versatile
• Easy to score
Evaluation
• More challenging to write
• Assess recognition over recall
• Best Use:
• Factual, conceptual, or
procedural information
Bloom’s
• Con’s
Synthesis
Analysis
Application
Comprehension
Knowledge
SHORT-ANSWER
• Pro’s
• Assess unassisted recall
• Relatively easy to write
Evaluation
• Only useful if you can give a
short answer.
• Must be worded carefully to
avoid scoring problems
• Best Use:
• Assessing information that you
expect to be memorized
Bloom’s
• Con’s
Synthesis
Analysis
Application
Comprehension
Knowledge
ESSAY
• Pro’s
• Con’s
• Take longer to answer
• Hard to grade fairly, consistently
• Take longer to grade
• Best Use:
• Assessing highest level objectives
Bloom’s
• Can test higher complex objectives
• Can test process / reasoning
• Realistic tasks
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
www.siop.org/workplace/employment%20testing/testformats.aspx
SUMMARY
• Each question type has pro’s and con’s
• Question type should be guided by the
learning objective to be assessed.
• Limited options for assessing higher
taxonomy
CONSTRUCTING OPENENDED QUESTIONS
WELL-DEVELOPED ESSAY QUESTIONS
• Mirror well-defined learning objectives
• Assess most appropriate content types
• Require content recall, evaluation, and
reasoning
• Are clearly written
• Provide boundaries
• Have well-defined grading criteria
http://testing.byu.edu/info/handbooks/WritingEffectiveEssayQuestions.pdf
APPROPRIATE CONTENT
• Content that justifies high-level mastery
• Construction, higher-order taxonomy
• Analysis (analyze, compare, contrast, interpret)
• Evaluation (evaluate, explain, justify)
• Synthesis (develop, construct, modify)
• Complex, multi-step thought processes
• Simulation of real-world processes
• “given a patient-case scenario”
If content / processes can be assessed by methods other
than essay questions, they probably should.
KEY ELEMENTS
• An ideal essay question requires students to:
• Recall facts
• Make an evaluative judgment or develop a novel
solution
• Explain reasoning behind response
• The question should include:
• Task
• Problem situation
WRITING FOR CLARITY
• Ensure your question requires higher-order
thinking
• Make sure the task is defined and focused
• Make sure the problem situation includes
adequate detail
SETTING BOUNDARIES
• Increasing structure prevents
• grading problems
• bluffing
• Avoid indeterminate questions
• Students can redefine and answer with info the
know well
• Give time / space limits
• Establish rules for answers
ESTABLISHING GRADING CRITERIA
• Must have specific criteria for grading
identified a priori
• Create a model answer
• Assign point values
• Peer review
• Identify essentials in response
• Determine whether partial credit is
allowable and how will be awarded.
• Grade blinded
DIFFERENT GRADING APPROACHES
• Comprehension / Understanding
• Screen responses for key elements
• Assign points for each element present
• Scores for presence of content only (potential bluffing)
• Reasoning / Complex Processes
• Must assess complete response (time consuming)
• Consider a rubric approach to limit subjectivity
• Problem – Solution
• Encourage requiring students to show work
• Can grade multiple steps in process
GRADING TIPS
• Grading scheme should be easily
interpreted by other graders
• Self-explanatory
• Point values that can easily be tallied
• When multiple graders, encourage frequent
communication and comparison of results
• Before grading, screen a sample of
responses for consistency
OTHER ISSUES
• Exam length
• Addressing unidentified “correct” answers
• Partial credit or all-or-none
• Practice examples
• Multiple examples
• Should not be the same questions as on exam