The 10 Most Common Testing Mistakes –
And How You Can Avoid Them
Presented by: Steven B. Just, Ed.D.
Lack of testable
learning objectives
© 2009 Pedagogue Solutions. All Rights Reserved.
Lack of testable learning objectives
© 2009 Pedagogue Solutions. All Rights Reserved.
Lack of testable learning objectives
 The learning objectives drive the creation of the
content
 Each objective should be concise
 Each objective should be “testable”
 The verb is crucial:
– Avoid “understand,” “know,” “be aware of”
– Use “define,” “calculate,” “list”…
© 2009 Pedagogue Solutions. All Rights Reserved.
Poorly-written learning objectives
Students should be able to:
 Understand the mechanisms of cardiac contraction
 Demonstrate a knowledge of basic cardiac
pathophysiology
 Integrate an understanding of cardiac physiology into
selling activities
© 2009 Pedagogue Solutions. All Rights Reserved.
Correct learning objectives
Students should be able to:

Describe the electrophysiology of the heart, including the cellular
anatomy and electrochemistry of contraction

Read a simple, normal electrocardiogram, and describe how the
parts of the recorded wave pattern relate to cardiac events

Identify the gross anatomy of the heart

List the basic categories of cardiac pathologies

Provide examples of the most commonly occurring cardiac
pathologies
© 2009 Pedagogue Solutions. All Rights Reserved.
Lack of testable learning objectives
 Ensure the questions are directly derived from the
learning content
 Separate those who have mastered the material from
those who haven’t
 Ensure real world competencies are assessed
 Ensure that all competencies are mastered
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified
purpose for exam
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam

Assessment
•Summative Assessment

Quiz
•Diagnostic Assessment

Test
•Rubric

Exam

Evaluation

Pretest

Post-test

Formative Assessment
•Performance Assessment
•Self-assessment
•High Stakes Assessment
•Certification
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Assessment:

A systematic process for returning results in order to describe
what students know or can do.

An ongoing process aimed at measuring and improving student
learning

Assessments can be in the form of a quiz, test, exam or
evaluation.
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Quiz:
 A low-stakes diagnostic assessment in which the
results are only to be used for self- or group-diagnosis
and prescription
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Test:
 A medium stakes formative assessment designed to
inform both the learner and (optionally) the instructor
of the learner’s level of knowledge at an intermediary
point in the instructional process. There are no longterm consequences for failure. Short-term
consequences may include required remediation
before proceeding with a learning activity
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Exam:
 A high-stakes summative assessment at the
completion of a learning experience for which there
are consequences for failure. Results of exams are
made available to the learner’s direct supervisor and
appropriate training department personnel. Exam
results may have career impacting consequences
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Evaluation:
 An assessment that measures, compares, and judges
 For example:
– Role play evaluations
– Smile sheets
– Evaluation of a training program
– Level 3 and 4 evaluations
© 2009 Pedagogue Solutions. All Rights Reserved.
No clearly identified purpose for exam
Assessment: A systematic process for returning results in order to describe what
students know or can do.
Quiz: A low-stakes diagnostic assessment in which the results are only to be used
for self- or group-diagnosis and prescription.
Test: A medium stakes formative assessment designed to inform both the learner
and (optionally) the instructor of the learner’s level of knowledge at an
intermediary point in the instructional process
Exam: A high-stakes summative assessment at the completion of a learning
experience for which there are consequences for failure.
Evaluation: An assessment that measures, compares, and judges
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions not job relevant
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions not job relevant
Questions to Ask
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions not job relevant
 Does this question test a learning objective?
 Is this knowledge required?
 Could this be looked up or must it be memorized?
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “double negative”
questions
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “double negative” questions
Lack of insulin production does not cause:
A. a lack of glucagon production *
B. depletion of fatty acid stores
C. unchanged brain cell uptake of glucose
D. diminished fat storage in adipose cells
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “double negative” questions
Which of the following is an effect of the LACK OF insulin
production:
A. increased conversion of glucose into energy
B. production of ketones from the metabolism of excess fats *
C. increased protein production
D. increased formation of glycogen
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at
Knowledge/Recall level
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Knowledge Question cues:
– List
– Identify
– Define
– Describe
© 2009 Pedagogue Solutions. All Rights Reserved.
Knowledge Question Example
List the order in which blood travels through the chambers of the
heart from the vena cava to the aorta:
a) right atrium, right ventricle, left atrium, left ventricle *
b) right atrium, left atrium, right ventricle, left ventricle
c) left atrium,right atrium, left ventricle, right ventricle
d) left atrium, left ventricle, right atrium, right ventricle
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Comprehension Question cues:
– Describe
– Predict
– Summarize
– Estimate
– Interpret
© 2009 Pedagogue Solutions. All Rights Reserved.
Comprehension Question: Example
An ultrasound image of a patient’s heart indicates definite
enlargement of the atria, and normal-appearing ventricles. The
overall size of the heart has not yet been determined. Which of
the following cardiomyopathies could describe this condition:
a) restrictive or dilated
b) hypertrophic or dilated
c) hypertrophic or restrictive *
d) dilated or hypertrophic
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Application Question cues:
– Apply
– Demonstrate
– Calculate
– Complete
– Classify
© 2009 Pedagogue Solutions. All Rights Reserved.
Application Question: Example
The isolated perfused frog heart serves as a good experimental
model of human heart contractility. What would you expect to
happen to the contractions of the frog heart if a solution with a
high calcium ion concentration were applied to it?
a) increased contractile force *
b) unchanged contractile force
c) increased rate of contraction
d) unchanged rate of contraction
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Learning Objectives and
Questions Must be Aligned
Within Bloom’s Taxonomy
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
 Knowledge LO
– You will be able to define arthritis.
 Knowledge Question
– Choose the best definition for the term arthritis
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
 Comprehension LO
– You will be able to identify examples of thrombosis.
 Comprehension Question
– Which of the following is an example of thrombosis?
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level

Application LO
– You will be able to apply your knowledge of key pivotal trial results
by stating critical supporting data points for your drug.

Application Question
– You are in conversation with a physician about the safety data for
your product. Identify the critical supporting data points from the
pivotal trial results.
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Objective/
Content Area
Knowledge
(Level 1)
Comprehension
(Level 2)
Application
(Level 3)
# of items
(percent)
Totals
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions all at Knowledge/Recall level
Objective/
Content Area
Knowledge
(Level 1)
Anatomy/
Physiology
Disease State
Application
(Level 3)
14
Comprehensio
n
(Level 2)
12
11
14
5
Market
10
15
5
Managed Care
14
10
6
Selling Skills
16
11
3
65 (~40%)
62 (~40%)
23 (~20%)
Totals
4
# of items
(percent)
30
(20%)
30
(20%)
30
(20%)
30
(20%)
30
(20%)
150 (100%)
© 2009 Pedagogue Solutions. All Rights Reserved.
Failure to analyze
exam data
© 2009 Pedagogue Solutions. All Rights Reserved.
Failure to analyze exam data

Do you need to know how well your questions discriminate between weak and
strong students?
–

Do you need to know what percent of students answered each question
correctly?
–

Difficulty level
Do you need to know where students have misinformation?
–

Point-biserial correlation
Choice distribution
Do you need to know where the group has strengths and weaknesses?
–
Score by learning objective/topic
© 2009 Pedagogue Solutions. All Rights Reserved.
Failure to retest
periodically
© 2009 Pedagogue Solutions. All Rights Reserved.
Ebbinghaus Curve of Forgetting
© 2009 Pedagogue Solutions. All Rights Reserved.
Ebbinghaus Curve of Forgetting
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score

The Higher Authority Method:
– “Our Vice President said it should be 90”

The Committee Method:
– “90 seems about right”

The Received Wisdom Method:
– “I don’t know how or when it got set, but it’s always been 90”
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
The Angoff Method:
A Defensible Method For
Setting Passing Scores
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
 Step 1: Identify three to five judges who are
familiar with the competency covered by the
test.
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
Step 2:

Does the question test the learning objective?

Are all choices plausible?

Is the identified correct answer correct? Could other choices
plausibly be correct?

Is the question properly written according to the rules of valid
question writing?

Is the question job relevant?
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
 Step 3: For each item on the test each judge
estimates the probability that a minimally competent
person would get it right.
 Step 4: Sum the probabilities of each judge
 Step 5: Average the judges’ scores
© 2009 Pedagogue Solutions. All Rights Reserved.
Invalid passing score
Item
Judge 1
Judge 2
Judge 3
1
.75
.80
.85
2
.80
.90
1.00
3
.75
.75
.90
4
.90
.90
.80
5
.95
.75
.85
Total
4.15
4.10
4.40
Percent
83%
82%
88%
Averaging the totals for each Judge
Cut Score= 84%
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “All of the Above”
questions
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “All of the Above” questions
Which of the following waves, complexes and intervals can
appear on an electrocardiogram:
A. P
B. QRS
C. QT
D. All of the above *
© 2009 Pedagogue Solutions. All Rights Reserved.
Using “All of the Above” questions
Which of the following waves, complexes and intervals DOES
NOT appear on an electrocardiogram:
A. P
B. QRS
C. QT
D. U*
© 2009 Pedagogue Solutions. All Rights Reserved.
No testing strategy
© 2009 Pedagogue Solutions. All Rights Reserved.
No testing strategy
 Governance and administration
 Legal issues
 Make expectations explicit and public
 Determine methods of testing
 Establish assessment frequency
 Assessment security
 Define terms
© 2009 Pedagogue Solutions. All Rights Reserved.
No testing strategy
 Create fair, valid and reliable
assessments
 Determine cut (passing) scores
 Remediation and consequences
 Re-certification
 Program evaluation/Item analysis
© 2009 Pedagogue Solutions. All Rights Reserved.
Questions/Comments
Presented by:
Steven B. Just, Ed.D.
sjust@pedagogue.com
609-921-7585 x12