Review of SBM/ID Course
• Scheduled for November 2014 in
Subcommittee
• Scheduled for January 2014 in full MEC
• Ad Hoc Course Review Group included Dave
Nierenberg, Stephanie Morton, Chris Rees
Basic Factors
• Course occurs in Y2, Term 5 (April and May)
• This is the end of the academic year
Students are a bit tired, but good time for a wrap-up course
The are worried about Step 1
The other major course going on is SBM/Reproduction
• Course Director – Tim Lahey, MD
• Course has 49 curricular hours (which includes
time for readiness/self-assessment quizzes)
• Also has one 4-hr final exam, which also
includes items from Pharmacology
Course Objectives 2014, Part 1
#
Objective
Exam
1
Relate key pathogens to major syndromes in ID
18
2
Identify and explain where new and useful biomedical information comes from
and how it is discovered
1
3
Discuss key topics in the clinical manifestations, diagnosis, treatment, and
prevention of major ID syndromes
40
4
Review uses of imaging, molecular diagnostics in the management of ID as well
as role of public health measures in the prevention of ID
6
5
Recognize vital role of prevention in control of ID
8
6
Describe the impact of social, economic, cultural, and personal factors upon the
health of the individual, manner in which people perceive helath and illness,
and health of groups
2
7
Explain role of complex medical procedures (e.g. LP, thoracentesis) in the
diagnosis and management of common ID’s
3
8
Recognize role of gram stain, culture, urinalysis and imaging in workup and
treatment of major infections diseases
12
Course Objectives 2014, Part 2
#
Objective
Exam
9
Recognize impact of variations in access to healthcare on the epidemiology of
major infectious diseases
0
10
Review the role of the physician in population-level interventions designed to
prevent major infectious diseases
0
11
Communicated collegially and effectively with physician colleagues (and others)
verbally in writing, and in the EMR
NA
12
Meet professional responsibilities fully
NA
13
Adhere to high ethical and moral standards, accept personal responsibility for
actions, accept constructive criticism, respect patient confidentiality
NA
14
Take responsibility for one’s own medical education; develop habits of
mindfulness and reflection
NA
15
Contribute constructive feedback during peer review
NA
16
Identify and evaluate relevant information about evidence-based, cost-conscious
strategies; apply this information to patient care and to continuous updating of
skills
NA
17
Empathize with patient concerns, respect patient points of view and traditions
NA
Course Objectives – Content Review
• Course objectives are provided in the syllabus on the
first page front of iBook
• Course objectives are written in the correct format
for the most part; a few could have clearer action
verbs inserted
• Need to “split out” objectives 1 and 3
– These could be combined and then expanded to 11 objectives,
corresponding to the 11 “modules” of the course (e.g.
Pneumonia/URI, UTI and STD)
• Need to specify and ensure that objectives 9-17 are
assessed via small group peer-to-peer assessments
Objectives: Step I Brochure
• Content seems to cover of the content advertised by the NBME
• Major modules seem to be major important areas of ID in the US:
–
–
–
–
–
–
–
–
–
–
–
Skin, soft tissue, bone
Parasitology
Pneumonia and URI
UTI and STD
Diarrhea and Foodborne Illness
Bacteremia and Endocarditis
HIV
Fungal infections and Immunizations
TB and other mycobacteria
Infections in special populations
CNS infections and Sepsis
Objectives: National Organization
• Tim is not aware of a national discipline-oriented
organization in their field that has published course
objectives, thus no comparison was made
Session Objectives
• Session objectives are provided in the course materials
for each session;
• However, many of these session objectives are not in
optimal format with measurable outcomes; for example
– “Review key concepts in drug susceptibility testing…”
– “Identify key concepts in ID including approach to antibiotics”
– “To learn the distinction between colonization and infection”
• All session objectives map back to one or more course
objectives
Overlap with other courses
• There is and should be some overlap of concepts and
content with Y1 courses, especially Medical
Microbiology and Virology
• There should be some overlap with many of the Y2
SBM courses (for example, the SBM/CV course
discusses clinical aspects of endocarditis)
• We searched in Ilios Inspector for “overlap terms”
such as:
– HIV
– Meningitis
• See following slides
Let’s look at coordination across years: “HIV”
Year
Course
objective
Lists “HIV”
Course level
MeSH hit
For HIV
Course has
session with
“HIV” in title
Session learning objectives
Contain “HIV”
1
Path x 1
Virology x 1
Pathology x 1
Virology x 1
Virology x 9
Pathology x 2
2
None
Pharmx x 1
SBM/ID x 2
Pharmacology x 1
SBM/ID x 1
SBM/ID x 18
Pharm x 3
SBM/Neuro x 3
PBLx1
3
None
None
None
None
4
AMS x 1
None
CPT x 2
CPT x 7
Let’s look at coordination across years: “meningitis”
Year
Course
objective
lists
“meningitis”
(0)
Course level
MeSH hit
for
“meningitis”
(3)
Course has
session with
“meningitis” in
title
(2)
Session learning objectives
Contain “meningitis”
1
None
None
None
Microbiology x1
2
None
SBM/Neuro x1
SBM/ID x1
SBM/ID x2
SBM/ID x7
SBM/Neuro x6
PBL x1
3
None
None
None
None
4
None
Neurology x1
None
CPT x2
Neurology x1
(18)
What were the session objectives?
Course
Session
Listed Session Objectives
Y1 Microbiology
Hemophilus and neisseria lab
Provisionally identify neisserial pathogens in mock CSF, and
explain the importance of provisional identification…
Y2 Neuro
Infectious diseases
Be able to treat the most common infections of the CNS, esp viral
and bacterial meningitis
Y2 Neuro
Infections diseases
Identify etiologies of chronic meningitis…
Y2 Neuro
Neuropathology: ID
Name the bacterial species most frequently causing meningitis in
infants, children, and adults
Y2 Neuro
Neuropathology: ID
Describe the pathology of bacterial meningitis and brain abscess
and distinguish them
Y2 Neuro
Neuropathology: ID
Describe tuberculous meningitis and differentiate it from
bacterial meningitis
Y2 ID
Meningitis
Review the behavioral and epidemiological contributors to
meningitis
Y2 ID
Meningitis
Recognize the clinical manifestations, diagnosis and treatment of
meningitis
Y2 ID
Meningitis
Discuss the role of vaccination in the prevention of meningitis
What were the session objectives?
Course
Session
Listed Session Objective
Y2 ID
Meningitis
Describe the role of LP in the diagnosis of meningitis
Y2 ID
Meningitis
Review role of CSF cultures in making diagnosis
Y2 ID
Brain infections
Identify major pathogens associated with brain abscess, meningitis, and
encephalitis, and their empirical treatments
Y2 ID
Conference #5:
Meningitis and sepsis
Review major pathogens responsible for most cases of meningitis and sepsis, and
their empirical therapies
Y2 ID
Conference #5:
Meningitis and sepsis
Describe the epidemiology of meningitis and sepsis
Y2 ID
Conference #5:
Meningitis and sepsis
Describe the clinical manifestations, diagnosis, and treatment of meningitis and
sepsis
Y2 ID
Conference #5:
Meningitis and sepsis
Discuss the role of cultures and other microbiological techniques in the diagnosis of
sepsis and meningitis
Y2 ID
PBL case on
immunocompromise
List the signs of meningitis in a pediatric patient that would lead to a lumbar
puncture
Y3
None
None
Y4 CPT course
Review of major antibiotic
drug classes
Explain the concept of empiric drug therapy, and describe the concept of “most
likely pathogens” for major infections such as pneumonia, urosepsis, and meningitis
Y4 CPT course
Case studies of drug
treatment of major
infections
Explain the concept of empiric drug therapy, and describe the concept of “most
likely pathogens” for major infections such as pneumonia, urosepsis, and meningitis
1. Summary regarding Objectives
• Most of the course objectives are written in
appropriate language, and appear to be appropriate
• However, objectives 1 and 3 are too broad, and need
to be subdivided (perhaps merged and expanded
into 11 objectives matching course modules)
• Objectives 9 and 10 should be emphasized more on
the final exam
• Objectives 11-17, if real, MUST be evaluated in some
other way (e.g. by student or faculty assessment in
small groups), and should be consolidated if possible
Course Learning Opportunities
Type
Standard lecture
Independent on-line module
Hours %
7
14%
14
28%
Small group conference
On-line readiness quiz and formative feedback
Quiz and case reviews and discussion (large session)
8
4
8
16%
8%
16%
Laboratory (parasitology and microbiology)
PBL case
Interprofessional discussion: social determinants of
health
4
3
1
8%
6%
2%
Peer assessment of small group performance
Total
0
49
0%
100%
Course Learning Opportunities
• The course director has made a significant effort to
dramatically reduce the use of “standard” lectures (down
to 14%)
• There is much more use of independent (recorded, web
based) learning modules
• Also use of conferences, readiness quizzes, and large
group discussions of of the quiz material and case
reviews
• New idea of creating 11 “modules” with a standard
sequencing of sessions
• This was an experiment, and we will need to look to
scores and comments from students as to how well they
adapted to this very different style
Assessment for 2014 edition
• There was one final exam, with 59 ID questions (and another 25
contributed from the pharmacology course)
– This exam counted for 76% of the final course total points in SBM/ID
– Up to 24 points this year were available from the readiness quizzes
• The final exam content will distribute more evenly across course
objectives once course objectives are re-written as suggested
• Course objectives 11-17 should be evaluated via peer-to-peer assessments
• Question types included:
– Vignettes
39 out of 59
– Factoids
18 out of 59
– Visual ID
1 out of 59
– Analyze data
1 out of 59
– Negative stems
2 out of 59
Assessment – Readiness quizzes
•
•
•
•
There were 8 readiness quizzes, each for 30 min
Each quiz contained 10 items
Each quiz was allotted a total of 30 min
Students could take each quiz once or twice,
not more
• Students received (per quiz) 3 points for best
score of 8 or greater, 1 point for score 4-7, and 0
points for score of 3 or less
• All told, these quizzes counted for up to 24% of
the final course score
Assessment - Conferences
• Currently, conference attendance and participation
account for 8% of the total course grade each (16%
total).
• Faculty give students credit for small group
engagement but do not provide narrative feedback
• The course requires students to provide anonymous
feedback on small group performance to 3 peers but
this did not occur in 2014
• Due to a technical glitch on Blackboard, the anonymity of
these evaluations was compromised, and no feedback was
ultimately provided. (Will try again in 2015)
3. Summary regarding Assessment
• There are no major problems with the assessments in the
course – questions are written in good formats
• Correlation between course assessment and stated objectives
was good except in the one year when a computer glitch
precluded peer assessments of small group performance
• Conferences provide opportunities for students to apply their
knowledge and discuss the material further with a faculty
facilitator, and faculty assess student “engagement” but do
not provide narrative assessments.
•
Note: For 2014-15, in all SBM courses, we will be handling conference engagement
and participation, and self-assessment quizzes, differently; passing performance
will be required in three different areas (knowledge tested on final exam;
participation and engagement in conference; self-assessment and readiness in
readiness quizzes, for example)
Measures of Quality – Ratings 2013-14
• 83% of class completed survey
• Ratings ranged from 3.79 (overall course quality) up to
4.13 (course director’s ability)
• Overall mean score for this course was 4.07
• This placed SBM/ID 7th out of 13 courses in Y2
• Scores for lecturers/large group leaders were 3.37 up to
4.66, with seven of the 11 rated at 4.0 or higher
• Scores for conference leaders ranged from 3.73 to 5.0
• Color key: green/yellow/pink=upper/middle/lower third
4. Measures of Quality – Ratings 2014-15
Measures of Quality – Student Comments
iBook
• Positive Comments (44)
–
–
–
–
Can see all important topics in one place (18)
Loved Dr. Lahey’s chapters (5)
Clear objectives, bolded sentences (5)
Excellent organization (2)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
iBook
• Suggestions for improvement [solicited in a separate question] (87)
– Provide iBook chapters for all topics (endocarditis, pediatric infections,
osteomyelitis) (17)
– Include more embedded tables, pictures, videos, and questions (16)
– Standardize the quality of notes (12)
• Bulleted lists not helpful (vs. prose) (13)
• Some chapters too long (Dr. Lahey, parasites) (8)
– Trouble differentiating important information – more
bolding/italicizing/boxing (8)
– Some chapters need editing for grammar; some students who printed the
iBook felt the 12 point font size was too small (6)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Small Group
• Positive Comments (6)
– Good interactions with classmates (3)
• Criticisms (55)
– Classmate presentations not helpful (14)
• Verbatim from iBook (8)
–
–
–
–
Generally not helpful (7)
Too time consuming (4)
Too much inter-group variability (3)
Tough to know what was relevant (1)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Small Group
• Suggestions for improvement (26)
–
–
–
–
–
–
–
–
–
–
–
Include directed, structured questions for the whole group to answer (8)
Make small groups longer (4)
More lecture material, fewer cases (2)
Have smaller groups (2)
Make groups more engaging (2)
Eliminate small groups to give extra time for independent study (2)
Have fewer small groups (2)
Faculty facilitators should talk more (2)
Use small groups for quiz review (1)
Have multiple cases per small group (1)
Focus on diseases instead of cases (1)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Quizzes
• Positive Comments (11)
– Forced students to keep up with material (1)
– Good way to check learning (2)
• Suggestions for improvement (11)
–
–
–
–
–
Quizzes should be made available for review on Blackboard (5)
Quiz deadlines should be later and/or on a different day (3)
Quiz questions should be improved (1)
Send out email reminders for quizzes (1)
Email PowerPoints for review sessions before class (1)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Quiz Reviews
• Positive Comments (5)
• Criticisms (21)
– Not useful (13)
• Suggestions for improvement (8)
– Use time for more comprehensive review (5)
– Use online module for quiz review instead (2)
– Review new questions instead of old questions (1)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Overall Organization/Structure
• Positive Comments (24)
–
–
–
–
Emphasis on independent, active learning (8)
Expectations/objectives/clarity of important concepts (5)
Structure of video + ibook -> small group -> quiz -> review (4)
Reduced class time (1)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Overall Organization/Structure
• Criticisms (25)
– Too few lectures (12)
• People liked the lectures that were included (6)
– Too much self-directed learning (7)
• “With this course, I think it is important to make the distinction between
"learning" and "being taught." I feel strongly that I learned about ID this term,
while at the same time, I feel as though I was taught almost nothing in this
course. Instead, I was given a 250 page book and told to memorize its
contents.”
– Did not like overall organization (some were very passionate about this) (6)
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Overall Organization/Structure
• Suggestions for improvement
– More short videos (17)
• People really liked the short videos that were provided (6)
• “Some of my classmates expressed that they truly learn better through lectures.
With this in mind, it might be worthwhile to provide more mini lectures
emphasizing the main topics in the course (such as the youtube videos Dr.
Lahey created) in order to provide these students with an appropriate
supplement to the i-book and small group sessions.”
• “If the course is attempting to move to be more small group oriented, I think
what would really enhance the course would be 20-min videos presenting the
essentials of each iBook chapter. Previous years' Echo lectures tended to be a
bit long and tedious to watch on our own, especially if they were discussionbased.”
(number in ( ) = number of comments made)
Measures of Quality – Student Comments
Miscellaneous
• Positive Comments
– Dr. Lahey (14)
• “Dr. Lahey is a magician”
• “Every time Dr. Lahey spoke it was like tasting ice cream for the first time. He is
incredibly engaging, intelligent, and passionate about his work.”
• “Dr. Lahey is one of a kind, amazing teacher. I feel like everything he touches
turns to gold.”
–
–
–
–
–
Exam was fair, counted for less of the overall course grade (2)
Faculty (1)
Labs (1)
Interactive sessions? (1)
Final Review Class (could be moved earlier) (1)
Measures of Quality – Student Comments
Miscellaneous
• Criticisms
– Too much reading, too little time (5)
• How long did the required independent study modules take to complete (more
than the time listed on the schedule, less then the time listed on the schedule,
etc.)?
–
–
–
–
–
–
–
–
–
Longer (31) – generally MUCH longer (>3 hours)
Slightly longer (4)
About right (6)
Less time (2)
Variable (7)
Unclear what to focus on for final exam (5)
Too little correlation with material on boards (5)
Not enough practice questions (3)
“The issue with the peer evaluations was very unfortunate”. (1)
Measures of Quality – Other Student Comments
• The position of this course (at the end of Year 2) was perfect going into
the Step 1 study period, given that much of the content covered in the
class (i.e., antibiotics and micro-organisms) requires a good deal of
memorization
• There was a significant amount of content in the iBook, as well as
in the ECHO recordings from the previous year, that students were
told to NOT review for the purpose of the SBM ID course. Given the
proximity of this course to Step 1, this left many people anxious,
feeling as though they were missing some critical pieces of ID
knowledge that would appear on Step 1, but which were not
covered elsewhere in the curriculum (i.e., parasitology)..
5. Student Recommendations
• iBook
– Provide iBook chapters for all topics
– Include more embedded tables, pictures, videos, and questions
– Standardize the quality of notes: convert bulleted lists to prose, edit
down particularly long chapters
– Differentiate important information with more
bolding/italicizing/boxing
– Edit for grammar, increase font size
• Small group
– Include directed, structured questions for whole group to answer
5. Student Recommendations
• Quizzes:
– Make quizzes available for review on Canvas
– Consider changing quiz deadlines
• Quiz Reviews:
– Create short online module for quiz review
– Consider using in-class time for more comprehensive review or
lecture
• Organization/Course Structure
– Consider creating more short videos to provide more guidance to
students
– Revise time estimates for independent learning modules
• PRELIMINARY SUGGESTIONS
FOR COURSE IMPROVEMENT
FOR AY 2014-15
SUMMARY
• Overall this is a strong course, with excellent and timely content, good
organization, a very strong course director, several bold innovations in
progress, and many strengths identified by students
• WE are making suggestions for small but deliberate improvements in several
areas:
– Re-writing some of the course level learning objectives
– Re-writing some of the session level learning objectives
– Making sure that all course level objectives are formally assessed with several
assessed in small groups and not on the final exam
– Variety and emphasis of learning opportunities
– Structure and content of final exam
– How other sessions of the course are organized and evaluated
– Coordination of content with other courses, other years
– Some specific student concerns about balance of learning opportunities, and
meeting learning needs of different style learners
– Making sure that time demands of course are not excessive relative to
Reproduction course needs
1. Summary regarding Objectives
• Most of the course objectives are written in
appropriate language, and appear to be appropriate
• However, objectives 1 and 3 are too broad, and need
to be subdivided (perhaps merged and expanded
into 11 objectives matching course modules)
• Objectives 9 and 10 need more attention on the final
exam
• Objectives 11-17, if real, MUST be evaluated in some
other way (e.g. by student or faculty assessment in
small groups), and should be consolidated if possible
2. Summary regarding sessions
• Improve wording of session objectives
• Enhance proportion of time in topic reviews spent in
faculty interactive large group discussion of key
concepts
3. Summary regarding Assessment
• There are no major problems with the assessments in the course –
questions are written in good formats
• Correlation between course assessment and stated objectives was good
except in the one year when a computer glitch precluded peer
assessments of small group performance
• Conferences provide opportunities for students to apply their knowledge
and discuss the material further with a faculty facilitator, but the faculty
don’t provide evaluation of students at present
• Course director needs to make sure that each of the course objectives is
assessed in some appropriate fashion
•
Note: For 2014-15, in all SBM courses, we will be handling conference engagement
and participation, and self-assessment quizzes, differently; passing performance
will be required in three different areas (knowledge tested on final exam;
participation and engagement in conference; self-assessment and readiness in
readiness quizzes, for example)
4. Measures of Quality – Ratings 2014-15
5. Student Recommendations
• iBook
– Provide iBook chapters for all topics
– Include more embedded tables, pictures, videos, and questions
– Standardize the quality of notes: convert bulleted lists to prose, edit
down particularly long chapters
– Differentiate important information with more
bolding/italicizing/boxing
– Edit for grammar, increase font size
• Small group
– Include directed, structured questions for whole group to answer
5. Student Recommendations
• Quizzes:
– Make quizzes available for review on Canvas
– Consider changing quiz deadlines
• Quiz Reviews:
– Create short online module for quiz review
– Consider using in-class time for more comprehensive review or
lecture
• Organization/Course Structure
– Consider creating more short videos to provide more guidance to
students
– Revise time estimates for independent learning modules
COURSE DIRECTOR RESPONSE
Thanks to Dave Nierenberg, Stephanie
Morton and Chris Rees for their time,
effort and wisdom
COURSE DIRECTOR RESPONSE
SBM ID hours in 2013
SBM ID hours in 2014
35 large group sessions
14 large group sessions
4 interactive reviews
8 interactive reviews
3 labs
3 labs
7 small groups
8 small groups
3 PBL
3 PBL
48 total contact hours
36 total contact hours
Strong performance after format change
SBM ID in 2013
SBM ID in 2014
Final exam mean 83.5
Exam mean 83.1(P=0.7)
Marginal 4
Marginal 5
Failure 1
Failure 0
Mean final grade 86%
Mean final grade 88%*
* P=0.005, largely driven by small group and quiz scores
We are making good progress
2013
2014
Course Director Response: Changes in 2015
• Objectives:
– Divide course objectives by condition (pneumonia, meningitis, etc)
– Specify which objectives are assessed in peer assessments
– Align course objectives better to course material and assessments
• Organization/Course Structure
– Clarify in course syllabus that independent learning modules are not
meant to be the sole review time
– Convert remaining 2 topics from PowerPoints to iBook chapters
– Encourage engagement in non-lecture-based learning
Course Director Response: Changes in 2015
• Quiz Reviews:
– Apportion more review session time to faculty concept review
• Small group sessions:
– Facilitated discussions of main case + “what if” cases instead of selfdirected student presentations
• Study tools:
– Create more short videos to augment in class sessions
– Quiz deadlines adjusted to the extent possible within a tight
schedule
– Add more study questions and if time figures into iBook
Revised Course Objectives, page 1
#
Objective
1
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of skin and soft tissue infections including osteomyelitis
2
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of upper and lower respiratory tract infections
3
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of urinary tract infections and sexually transmitted diseases
4
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of meningitis and other infections of the nervous system
5
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of HIV and associated opportunistic infections
6
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of tuberculosis and other mycobacterial infections
7
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of parasitic and tropical infections
Revised Course Objectives, page 2
#
Objective
8
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical treatment
and prevention of diarrheal and food-borne infections
9
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of bacteremia, endocarditis and sepsis
10
Review, recall and discuss pathogenesis, clinical manifestations, diagnosis, empirical
treatment and prevention of fungal infections, infections of special populations such
as children and immunocompromised hosts.
11
Review and discuss in an interprofessional context physician intervention to redress
the social determinants of health with special emphasis on HIV
12
Review, recall and discuss the importance of antibiotic stewardship
13
Communicate collegially and effectively with colleagues
14
Exhibit professionalism via enthusiastic engagement in learning including via peer-topeer teaching and reflection on constructive feedback
Course Director Response
• New objective assessment:
– 1-10, 12 assessed in quizzes and final exam
– 11 discussed in required large group discussion
– 13 &14 assessed in small group faculty + peer-to-peer assessment