Review of Year 1 Immunology/Virology
• Course occurs in the winter term of Year 1
• Course Directors – Charles Sentman, Ph.D.
David Bzik, Ph.D.
• Course has 44 curricular hours
• Course was last reviewed in May 2012
Recommendations from 2012 Review
• Incorporate more active modes of learning/new
methods of pedagogy into the course (i.e. reduce %
of lectures)
• Rewrite/create learning objectives for each session
to clearly articulate expectations to students
• Revise assessment questions; verify that they link
back to major course objectives
• Faculty development is needed for some faculty that
are not meeting expectations with regard to teaching
Recommendations from 2012 Review
• Reorganize course to create a better flow for the
material; consider matching order of topics used in
the text (specific to Immunology)
• Identify at risk students early and provide support,
e.g. review sessions (specific to Immunology)
• Update/edit notes; ensure lecture content and notes
are in agreement (specific to Virology)
• Provide an opportunity where biologic data are
observed, measured and analyzed (specific to
Virology)
Course Objectives – Immunology
There are 17 course objectives that fulfill Geisel
competencies as follows:
• 13 address specific knowledge in the preclinical
domain, and seem appropriate
• 4 address communication skills
• 3 address components of professionalism, although
only 1 is mapped to competency 4 in Ilios
• 1 addresses habit of inquiry/personal improvement
Currently no course objectives are mapped to Geisel
competencies 2 (clinical skills) and 6 (health care systems).
Course Objectives – Virology
could map to 4
could map to 4
Course Objectives – Virology
There are 19 course objectives listed in Ilios that fulfill
Geisel competencies as follows:
• 19 address specific knowledge in the preclinical
domain, and seem appropriate
• 1 addresses habit of inquiry/personal improvement,
but is currently not mapped to competency 5
Currently no course objectives are mapped to Geisel
competencies 2, 3, 4 and 6.
could map to 5b
Course Objectives
• Course objectives are provided to the students in the
syllabus for both parts of the course, and are clearly
written using measurable verbs
• Course objectives correlate well with the content in
the Step 1 brochure
• Did not find a published list of objectives by a
national organization in the field
Session Objectives
Recommendation 2012: Rewrite/create learning objectives for
each session to clearly articulate expectations to students
• Session objectives are provided at the top of each set of
notes (all except sessions on Herpes viruses, Tumor
viruses, Poliovirus)
• Most session objectives are written clearly using
measurable verbs (all except several immunology
sessions that use words like “know” and “understand”)
Objectives addressing LCME topics
• Health of Populations: Course includes discussions of
TB and RA, and how they affect the health care
system and society; course discusses vaccines in the
context of societal needs and problems.
• Basic and Ethical Principles of Clinical and
Translational Research: Course includes discussions
of cases that occurred prior to the emergence of
more modern views regarding basic scientific and
ethical principle of clinical and translational research,
including ways in which it is conducted, evaluated,
explained to patients, and applied to patient care.
Objectives addressing LCME topics
• Gender and Cultural Biases in Students Themselves:
Access to vaccination, access to diagnosis, access to
treatments, and risk of virus diseases are strongly
influenced by gender, racial/ethnic, and cultural
biases.
• Instruction in Medical Ethics and Human Values:
Not included in the course as this time.
Assessing integration in the curriculum
• The curriculum database was used to evaluate the
integration of major topics in the course across the
curriculum (e.g. Immunodeficiency, Rheumatoid Arthritis,
HIV, Viral Hepatitis, Influenza)
• In the past our curriculum has been weak in issues
related to the elderly, so we also assessed whether the
course discussed changes of immune function with age
Immunodeficiency
• present in two Year 1 courses (Immunology/Virology
and General Pathology)
• present in three Year 2 courses (Organ-based Pharm,
SBM Infectious Diseases, SBM Neurology)
• present in two Year 4 courses (Advanced Medical
Science, Clinical Pharmacology and Therapeutics)
Lymphoma
• present in one Year 1 course (Microbiology)
• present in one Year 2 course (SBM Hematology)
Rheumatoid Arthritis
• present in one Year 1 course (General Pathology)
• present in two Year 2 courses (SBM Bone and CT, On
Doctoring)
HIV
• present in two Year 1 courses (Immunology/Virology
and General Pathology)
• present in three Year 2 courses (Organ-based Pharm,
SBM Infectious Diseases, SBM Neurology)
• present in two Year 4 courses (Advanced Medical
Science, Clinical Pharmacology and Therapeutics)
Viral Hepatitis
• present in one Year 1 course (Immunology/Virology)
• present in one Year 2 course (SBM GI)
Influenza
• present in four Year 1 courses (Immunology/Virology,
Microbiology, General Pathology and HCDS)
• present in one Year 2 course (Organ-based
Pharmacology)
Age-related changes in Immunity
• searched using “geriatrics”, “elderly”, “aging”, “age
process”
• nothing mapped to Year 1 Immunology/Virology
• course directors should consider including at least
one of these words in one of their objectives so that
this material is detected in a search
Summary regarding Objectives
• Course objective listed in ILIOS are provided to the
students; several opportunities exist for mapping existing
objectives to additional Geisel competencies
• Most sessions now have objectives that are provided to
students; several sessions still need objectives, and
several sessions need to rewrite their objectives using
measurable verbs
• Searches in ILIOS regarding major topics in the course
demonstrated some redundancy with material in other
courses; much of this redundancy is planned and
appropriate, however the course directors should verify
this with other course directors
Course Learning Opportunities
Recommendation 2012: Incorporate more active modes
of learning/new methods of pedagogy into the course
(i.e. reduce % of lectures)
Immunology
• Lecture 21 hrs. (some are interactive) – 88%
• Discussion groups 3 hrs. – 12%
Virology
• Lecture 20 hrs. (some are interactive) – 100%
• Addition of a web-based laboratory
Pedagogy for AY 2012-13 is basically the same as that for AY 2011-12
Course Learning Opportunities
Recommendation 2012: Identify at risk students early
and provide support, e.g. review sessions (specific to
Immunology)
• Optional review sessions are offered to the class
• Course director contacts students who are struggling
Course Learning Opportunities
Recommendation 2012: Provide an opportunity where
biologic data are observed, measured and analyzed
(specific to Virology)
• A web-based laboratory was added to the Virology
course
Summary regarding Pedagogy
• The percentage of traditional lectures in the course is
higher than is desired (goal 40-50% of course hours);
the course director should consider ways to
incorporate more active pedagogies into the course
• Opportunities exist for students to discuss unclear
material with the faculty (review sessions)
• A web-based laboratory provides an opportunity for
students to observe, measure and analyze biologic
data
Assessment
• Course is divided into Immunology and Virology
components
• Immunology (IMM) component
– Two Quizzes (each 10% of course grade)
• Each covers 12 sessions
• Multiple choice 20-22 questions each
– Small Group Discussion (5% of course grade)
• 3 session, each student presents in one session
• Presentation by randomly-selected group of 4-5 students
• Graded by faculty preceptor with written feedback to students
Assessment
• Virology (VIR) component
– Two Quizzes (each 12.5% of course grade)
• Each covers 8 sessions (2 additional sessions assessed only on final exam)
• Multiple choice, 20 questions each
• Final Exam (50% of course grade)
– Covers immunology and virology (equal distribution)
– Multiple choice, 80 questions
• Overall Assessment Distribution
– Quizzes (4)
– Final Exam
– Small Group Discussion
45%
50%
5%
Assessment – IMM Question Formats
Immunology Quizzes (Q1 and Q2)
• All single best answer with 4 or 5 options
• In general, questions are clearly worded
• Only 1 of 42 items used a clinical scenario to stage
the question
• Three of 42 items used “negative” formats (All of the
following EXCEPT)
• Many of the questions used lists of T/F statements,
rather than one best answer lists
– A small number of items (3/42) were stated T/F questions (“Which of
the following statements are correct?”)
Assessment – IMM Question Content
Immunology Quizzes (Q1 and Q2)
• Most questions assessed recall of facts, and only a
few questions required application of knowledge:
Quiz 1 (IMM):
~18/22 assessed recall
Quiz 2 (IMM):
~14/20 assessed recall
• No apparent integration between multiple
session objectives
• Coverage is distributed evenly across the
sections
Assessment – IMM Correlation with objectives
Immunology Quizzes (Q1 and Q2)
• Questions evaluated correlated well with specific
session objectives
• Ex: Quiz 1 (11 sessions + 1 small group), 22 Questions
– 21 of 22 questions clearly map to session objectives
– 68 session objectives, ~29 covered on quiz
– Several questions had more than 1 mapped objective,
although no cross-over between sessions/topics
– Minimal overlap (only ~2-3 objectives mapped to more
than 1 question)
Assessment – VIR Question Formats
Virology Quizzes (Q3 and Q4)
• All single best answer with 4 or 5 options
• In general, questions are clearly worded
• 14 of 40 items used a clinical scenario to stage the
question
• No questions used “negative” formats (All of the
following EXCEPT)
• Many of the questions used lists of T/F statements,
rather than one best answer lists
– A small number of items (2/40) were stated T/F questions (“Which of
the following statements are correct?”)
Assessment – VIR Question Content
Virology Quizzes (Q3 and Q4)
• Many questions assessed recall of facts; a few
questions required application of knowledge:
Quiz 3 (VIR):
~12/20 assessed recall
Quiz 4 (VIR):
~17/20 assessed recall
• No apparent integration between multiple
session objectives
• Coverage is distributed evenly across the
sections
Assessment – VIR Correlation with objectives
Virology Quizzes (Q3 and Q4)
• Questions evaluated correlated well with specific
session objectives
• Ex: Quiz 3 (4 x 2hr sessions), 20 Questions
– 20 of 20 questions clearly map to session
objectives
– 38 session objectives, ~24 covered on quiz
– Few questions had more than 1 mapped objective
– Minimal overlap (only ~1-2 objectives mapped to
more than 1 question)
Assessment – Exam Question Formats
Final exam (includes both IMM and VIR)
• All single best answer with 4 or 5 options
• Some items were statements (i.e., a symptom,
followed by a list of potential causative agents)
without an explicit question
• Only 4 of 80 items used a clinical vignette to
introduce the item
• Four of 80 items used “negative” formats (All of the
following EXCEPT)
• Many items used lists of T/F statements, rather than
one best answer lists
Assessment – Exam Question Content
Final exam (includes both IMM and VIR)
• Many of questions assessed recall of facts:
~75% assessed recall
~25% assessed application of knowledge
• No apparent integration between multiple
session objectives
• Coverage is distributed evenly across the
disciplines (IMM and VIR) and sections
Assessment – Correlation with objectives
• Balanced treatment of topics (i.e., equal division between IMM
and VIR, and items maps across the COs)
• IMM portion of exam (40 items)
– ~ 45 session objectives covered (out of ~144)
– Several questions had more than 1 mapped objective, although no crossover between sessions/topics
– Minimal overlap (only ~2 objectives mapped to more than 1 question)
• VIR portion of exam (40 items)
– ~ 46 session objectives covered (out of ~80)
– Several questions had more than 1 mapped objective, although no crossover between sessions/topics
– Minimal overlap (only ~2 objectives mapped to more than 1 question)
Assessment – Small Group Presentations
• Students worked in groups (4-5, randomly selected) to
research an assigned immune-related disease condition and
lead oral presentations to fellow students
• Format: ~30 min of presentation by group, followed by ~25
min for discussion and questions, driven by a) suggested
discussion questions provided to the presenters, and b)
questions from their peers
• Students are completely responsible for conducting the
sessions
Assessment – Small Group Presentations
• Presentations are scored by faculty, based on the following
criteria:
– Knowledge of topic
– Coverage of immune mechanisms, pathophysiology, epidemiology, diagnosis,
and treatment
– Quality of presentation
– Professionalism
– Response to questions from peers
– Teamwork
• Students in the audience provide written feedback to the
groups (de-identified and presented in aggregate)
• This activity accounts for 10% of the IMM grade (5% of final
course grade)
• All members of the group receive the same grade
Summary regarding Assessment
• Quizzes and exam questions were clearly written
• All quizzes and the final exam use MCQ
– a fair number of T/F or “except” type questions
– Most items assessed recall, although application questions were more prevalent on the
exam
– Very few questions used clinical vignette stems
• Students are provided answer keys and access to the quizzes, including
detailed explanations of the answers
• Exam provided broad coverage of both IMM and VIR topics with equal
distribution
• There was no overlap between IMM and VIR content in the questions;
within disciplines, majority of the questions covered a single LO (no
integration of topics)
• Small groups provide opportunity for students to develop presentation
and discussion skills with feedback from faculty and peers
Measures of Quality – AAMC GQ
“Indicate how well you think that instruction in Immunology/Microbiology prepared
you for clinical clerkships and electives.” [1=poor; 2=fair; 3=good; 4=excellent]
BASIC SCIENCES
Geisel
mean 2008
Geisel
mean 2009
Geisel
mean 2010
Geisel
mean 2011
Behavioral Science
3.1
3.3
3.2
3.3
Biochemistry
2.7
2.8
2.7
2.5
Biostatistics/Epidemiology
2.7
2.8
2.9
3.2
Genetics
2.9
2.8
2.9
2.8
Gross anatomy/Embryology
3.4
3.5
3.6
3.5
Histology
2.8
3.0
2.8
2.9
Immunology
2.8
3.0
2.9
3.0
Microbiology
3.1
3.4
3.1
3.2
Neuroscience
3.2
3.2
3.2
On Doctoring
3.3
3.3
Pathology
3.1
Pathophysiology of Disease
Measur
es of
Quality
– AAMC
GQ
Geisel
mean 2012
All schools
means 2012
3.3
3.1
2.6
2.6
3.2
2.8
2.8
2.8
3.6
3.4
3.1
2.9
3.1
3.1
3.3
3.1
3.0
3.0
3.2
3.5
3.4
3.5
3.4
3.1
3.2
3.1
3.4
3.3
3.6
3.7
3.5
3.5
3.5
3.5
Pharmacology
3.1
3.5
3.4
3.1
3.1
3.0
Physiology
3.5
3.5
3.6
3.6
3.5
3.4
Measures of Quality – Step I
2009*
2010*
2011*
2012*
Means
09-12
Biochemistry
0.50
0.30
0.40
0.30
0.38
Biostatistics/Epidemiology
0.65
0.90
0.73
0.43
0.68
Genetics
0.40
0.30
0.48
0.28
0.37
Gross anatomy/Embryology
0.50
0.35
0.53
0.33
0.43
Histology/Cell Biology
0.53
0.30
0.40
0.37
0.40
Microbiology/Immunology
0.63
0.42
0.41
0.31
0.44
Pathology
0.55
0.35
0.42
0.26
0.40
Pharmacology
0.63
0.15
0.39
0.22
0.35
Physiology
0.67
0.32
0.47
0.38
0.46
TRADITIONAL CORE DISCIPLINES
*values reported for core disciplines are SD above the US/Can mean for Geisel mean scores
Measures of Quality – Course Reviews
scale [1=poor; 2=fair; 3=good; 4=very good; 5=excellent]
Immunol
2011
(35%)*
Immunol
2012
(96%)*
Virology
2011
(23%)*
Virology
2012
(96%)*
Overall satisfaction of course
1.71
3.19
2.95
3.65
Overall usefulness of lectures
1.70
2.99
2.38
3.58
Overall usefulness of small groups
2.41
2.83
N/A
N/A
Overall usefulness of course materials
2.09
3.15
3.45
3.44
Congruence of assessment questions to
material emphasized in course
1.87
2.63
2.90
3.02
*student participation rate on course evaluation
Measures of Quality – IMM Course Reviews
• Strengths:
sample comments:
•
•
•
•
“Dr. Mullin's was the highlight of this course. His lectures were dynamic and
clarified the material. I also found the review session led by Dr. Mullins to be very
helpful. This was the strongest aspect of the course.”
“What is GREAT about this course is that you provided page numbers of textbooks
and board review questions that correlated with each lecture. Every other course
in the school could benefit from this approach.”
“I really enjoyed how the lecture material mirrored the chapters from the Abbas
text book. When I was confused during lecture, the corresponding readings from
the book were essential.”
“In addition to Levinson and the USMLE-type questions in the back of the
text…provided most of the material we need to master in a digestible format.”
Measures of Quality – IMM Course Reviews
• Suggestions for Improvement:
sample comments:
•
•
•
•
•
“Extra information is like overeating. It stresses the system without providing
added nutrient benefit. The right amount of food and information causes the best
absorption.”
“Wade's lectures were very difficult to follow and his notes are long-winded and
repetitive. In comparison, Dr. Mullins's notes are clear and concise and his lectures
were interesting, easy to follow, and seemed to highlight the most important
material.
“…I often got more confused while reading his lecture notes than if I'd read the
textbook.”
“The material stressed did not feel like the material that was tested.”
“The most frustrating part of the course was the lack of consistency between the
exams and the information presented within the lecture.”
Summary Immunology Measures of Quality
•
•
•
•
•
•
•
•
Dr. Mullins: Engaging lectures, organized presentation of materials, concise notes with a
summary, unambiguous expectations for testing
Many students found the hypersensitivity lecture to be very good
Students appreciated Dr. Mullin’s engaging lecture style accompanied by his well organized,
succinct notes
Most students discussed the lack of consistency between professors in their lectures and
materials provided
Students found the textbooks very useful, the questions provided by the Levinson book and
the content in Abbas
– This is a double edged sword. Students praised Dr. Mullins notes for being concise,
highlighting key points, and not needing outside resources to understand the material
Students felt that the material tested did not match the material emphasized in class
Students would benefit from a summary at the end of each lecture that reiterated the key
points
Students appreciated that lecture objectives, slides, and textbooks were at times mapped
together, so that when looking at a slide you saw the objective that pertained to that slide or
when reading lecture notes there was a list of slides that corresponded to that paragraph of
notes
Measures of Quality – VIR Course Reviews
• Strengths:
sample comments:
• “Dr. Bzik's lecture notes - the bold and underlined words and the concise notes,
made for a very efficient study tool.”
• “Summaries given out in the end of the course was helpful in organizing different
viruses in my head.”
• “The subject matter is very interesting to me and I liked that the professors
attempted to match their questions to the types of questions asked on the National
Boards exam.”
• “I also really appreciated the inclusion of ethical, global health and historical issues
into the curriculum.”… “Instead of memorizing all the time you are actually piecing
together whys and hows during the process.”
• The material was taught clearly, tested fairly, and was straightforward…I felt that
the exams were representative of the material, and that I could get the information
easily by watching lecture and using the excel spreadsheet of viruses.”
Measures of Quality – VIR Course Reviews
• Suggestions for Improvement:
sample comments:
•
•
•
•
•
•
“The quiz questions were really frustrating. Some of them had multiple right answers, some of them
we really never covered in class. I really didn't think these quiz questions were a good
representation of what I learned in class.”
“It was difficult to grasp which details were important and which were not important to know.”
“…Dr. Witters practice exams, as a readiness blackboard quiz with detailed explanations of why
something is right AND why it is wrong, would help tremendously in focusing in on thinking and
rationalizing through questions correctly.”
“…if we need to read the Levinson material because it will not be taught to us in class but it is
quizzable then let us know at the beginning of the class not a couple of days before a quiz.”
“Not until after I read a few chapters of Levinson, did I realize there are about 10 things we need to
know for every virus: viral structure, replication cycle, disease, pathogenesis, transmission,
epidemiology, detection, treatment, vaccines. If each lecturer could make an effort to address each
of these aspects of the virus in their lecture material, it would make studying for exam much easier.”
Dr. Leib’s “..slides were very disorganized and haphazard and with no notes to accompany them,
there was no way to piece the information together. Some viruses had incubation times, other
didn't. Some had treatments, others didn't.”
Summary Virology Measures of Quality
• There was an incongruity between the expectations of quizzable
material versus supplemental material and lectures
– “…For each of the three lectures, there was a different emphasis placed on the
value of slides, notes, and the required textbook.”
• Lack of understanding as to what are the key points
– Students would appreciate a summary of each lecture of the
most important ideas
• Lecture notes that were not proof-read and were out of date, i.e.
didn’t match lecture slides
• Scheduling lab assignments in a different week from quizzes
• Proof reading quizzes more thoroughly to match class objectives
• Lecture notes with slides
Summary of Recommendations
• Course objectives are clearly written and correlate
well with objectives in the Step I brochure
• Some session objectives need to be rewritten;
session objectives need to be provided for every
activity in the course
• The issue of redundancy regarding common diseases
should be explored by the course directors to ensure
that the redundancy is planned
• The course is heavily lecture-based; the course
directors need to introduce more engaged/active
forms of pedagogy into the course
Summary of Recommendations
• Assessment questions have been improved since the
last review of the course, however further revision is
needed; students continue to cite frustration with
the content on assessments (i.e. the material
assessed does not match what is emphasized in the
course)
• Expectations regarding the course need to be clearly
communicated to students; some improvement is
needed in the clarity of course materials