Course Objectives
1. Recognize the scheme of integrated student assessment
2. Construct a standardized efficient MCQ
3. Analyze the quality of MCQ in the post exam
Evolution of Student Assessment Following Implementation of an
Integrated Medical Curriculum: Contribution to Improved Educational
Outcomes
Gregory K. Asimakis, Michael A. Ainsworth, Judith F. Aronson, Ann W.
Frye, Steven A. Lieberman & Jeffrey P. Rabek
The University of Texas Medical Branch, School of Medicine
Galveston, Texas, USA
Medical Science Educator Volume 21 : No. 2 - articles
lessons learned
“Curriculum Change without Assessment Change
Equals Little Change in Learning”
“modifying existing multiple choice “
“moved from a departmentally-based to
centrally coordinated curriculum, “
“workshops to train the course faculty in test
blue-printing, item writing, developing integrative
questions, and analyzing item statistics”
Welcome
Assessment coordinators
Objectives
1-Align the learning objectives of all levels (module
,year ,college)
2-Identify appropriate assessment methods.
3-Characterize efficient and effective assessment
processes.
4-Describe the way of Implementation of the
assessment plan
5-revise the plan by data analysis
Methods of assessment
Knowledge much ?
level?
Skills
level
Attitudes with student
patient
staff
Circumstances
Methods of assessments
•
Ideal test pass all those deserve to pass and
fail all those who deserve to fail
•
So
○ Use multiple methods to assess each
learning outcome.
○ Include qualitative as well as quantitative
measures.
MODIFIED ESSAY QUESTION MEQ
Please answer all the questions in sequence.
Mrs. Sami, a 50 years old lady complains of lethargy, weight loss and excessive vaginal
discharge, has been admitted to the gynae ward on which you work.
Q.1.
What are the three most likely diagnosis.
a.
b.
c.
Q.2.
List three specific questions which would help you distinguish between these
possibilities.
a.
b.
c.
Q.3.
A pap smear ordered by her general practitioner 2 month back revealed CIN II.
Write down the clinical examinations you want to do.
a.
b.
c.
Low Blood Pressure
a.
b.
c.
d.
e.
Adrenaline
Anaphylaxis
Atenolol
Autonomic neuropathy
Cardiogenic shock
f.
g.
h.
i.
j.
EMQ
Glyceryl trintitrate
Gram negative sepsis
Hypovolaemic shock
Morphine
Noradrenaline
The following patients are all found to have low blood pressure on examination. Choose the
most likely underlying cause for each case. Each answer option may be used once, more than
once or not at all.
1.
A 64-year old man presents with a 2-hour history of sudden onset of severe
abdominal pain radiating to his back. He has a history of treated hypertension and angina. On
examination he is drowsy and pale, has cold extremities with a weak radial pulse of 136/min and
a BP of 66/38.
(h)
2.
A 72-year old woman with a history of ischaemic heart disease and chronic atrial
fibrillation undergoes an emergency laparotomy for a perforated sigmoid colon. Twenty four
hours after surgery, her conscious level begins to fall and her urine output decreases to less
than 15ml/hour. On examination she has warm and clammy hands and feet, and has a bounding
radial pulse of 124/min and BP of 76/48. Prior to surgery her pulse and blood pressure were
normal.
(g)
3.
A 35-year old man is admitted with a right lower lobe pneumonia. He is treated with
intravenous Amoxicillin and oral Erythromycin. Shortly after the drugs have been administered
he complains of difficulty breathing and develops a florid rash over his whole body. He
becomes markedly hypotensive and you notice that his lips are swollen.
(b)
Quality assurance principles
•
Reliability
•
Validity.
•
Educational impact
•
Feasibility
•
Acceptability
•
Formative. For,Self. As ,summative. Of
How to increase reliability
•
Increase number of items
•
Increase time of testing
•
Large range of score
•
Using high discriminatory item
How to improve validity?
•
By choosing the correct questions and form of
question.
Knowledge
MCQ.
•
•
Validity.
Low
•
Reliability. High.
ESSAY..
High.
Low.
Attitude
•
Low reliability (behavioral expression
•
Variable validity best is workplace assessment
Way of implementation
of assessment plan
Data Analysis
•
Let the data sing
•
Difficulty index
•
Discrimination test
•
Test item analysis
summary
•
integrated assessment is a team based work
•
planning :concept ,objectives,methods,quality quide
,analysis revision ,and making decisions
How to Construct Effective Multiple-Choice
Tests
March 22, 2010
Objectives
• List what can be tested with multiple-choice
tests.
• Define the anatomy of a multiple-choice item.
• Explain guidelines for constructing multiplechoice items.
• Review examples of effective and ineffective
multiple choice tests.
• Evaluate MCQs items after exam(post tests)
• Identify MCQs technical flaws(practical)
When to Use
• To test a variety of levels of learning
• When you have a large number of individuals
taking the test
• When you have time to construct the test items
• When time is limited for scoring
(Clegg & Cashin, 1986)
Strengths
Weaknesses
Easily marked
Very hard to set well
testing factual knowledge
not effective for higher order
thinking & skills…..
run on-line.
guessing
save time and resources
Female students
Formative +summative
Danger of assessing
Planning a Test
• Use a test blueprint.
• Identify major ideas and skills
rather than specific details.
• Choose format :
best answer
correct answer
(Nilson, 2010)
Test Matrix
Anatomy of a Multiple-Choice item
Each item has:
• stem
• Lead- in question
• Answer and alternatives
correct or keyed option
several distractor options
1.
2.
3.
4.
Choose an important concept
Write the stem
Write the correct answer (key)
Develop distractors
common misconceptions
errors that could be made
plausible, yet less important information
similar in style, length to the key
every distractor should be reasonable
(Clegg & Cashin, 1986)
Format stems as
• problems, scenario or tasks
Clear
central idea
Avoid window dressing (excessive verbiage).
Positively structured
• Include most of informations in the stem so
that the options can be short
The Stem
• USE familiar language
• PROVID sufficient information
• DO NOT create tricky items by omitting essential
information
• DO NOT add extraneous information
• AVOID statements that fail to present a complete
thought or question.
• ELIMINATE excessive wording and irrelevant
information
Writing Stems
When using incomplete statements avoid beginning with the
blank space.
___ is the least severe form of behavior disorder.
A. Psychosis
B. Panic disorder
C. Neurasthenia
D. Neurosis
Better:
The least severe form of behavior disorder is ___ .
(Ory & Ryan, 1993)
Please check
Lead -in Question
• Different clinical tasks can be tested
• Can be done with the same stem (cloning of
question)
•
•
•
•
•
History
Diagnosis
Investigations
Management/Treatment/Drug therapy
Counseling
Lead-in Question
• Try asking questions that lead to clinical decision
making
Which one of the following…………?
– … is the most likely diagnosis?
– … investigations would you now order?
– …is the next step in the work-up and
management of this patient?
– … is the most important step in the initial
management of this patient?
EXCEPT
Item testing clinical decision making
A 62 year-old woman with a history of confusion
and constipation comes to the office for a followup visit. Laboratory investigations reveal a serum
calcium of 2.9mmol/L, a creatinine of 146
µmol/L, and a hemoglobin of 108g/L. Which one
of the following is the most likely diagnosis?
1. Hyperparathyroidism
2. Chronic renal failure
3. Multiple myeloma
4. Vitamin D intoxication
5. Renal cell carcinoma
Item testing clinical decision making
• A 62 year-old woman with a history of confusion
and constipationcomes to the office for a followup visit. Laboratory investigations reveal a serum
calcium of 2.9mmol/L, a creatinine of 146
µmol/L, and a hemoglobin of 108 g/L. Which one
of the following would help confirm the diagnosis?
1. Parathyroid hormone
2. Serum protein electrophoresis
3. 25-OH vitamin D
4. Serum creatinine
5. Abdominal ultrasound
Writing Item Alternatives
• Three - five is adequate.
• only one is the right answer.
• Vary the location .
• logical or numerical order.
• independent, not overlapping.
Writing Item Alternatives
•
homogeneous
• grammatical structure.
• equal in length
• None-of-the-above carefully.
• Avoid All-of-the-above.
Writing Item Alternatives
• AVOID
• negatives such as NOT // I don’t know
• clues to the right answer, such as:
o specific determinates including always, never, completely,
and absolutely.
o clang associations, choices identical to or resembling
words in the stem.
o conspicuous correct choice.
o pairs or triplets of options that clue the test-taker to the
correct choice.
o blatantly absurd, ridiculous options.
• Logical cues
• A 47-year old man present with an acute
episode of psychosis. Which one of the
following treatment would you consider
prescribing?
• 1. Alprazolam
• 2. Lorazepam
• 3. Haloperidol*
• 4. Diazepam
What is a “bad” MCQ?
If students do not answer the question correctly
because it is:
• too difficult or
• too easy or
• unclear,
edited or deleted
Thank you