SOURCES OF OUTCOME DATA
• Internal measures
– End of course &
clerkship surveys
– End of year surveys
– Faculty survey
– Universal Student
Rating of Instruction
(USRI)
– Certifying exam scores
– Canadian Graduate
Questionnaire
• External measures
– MCC
– Resident program
directors’ evaluation of
graduates
– NBME-Comprehensive
Basic Science Exam
– CaRMs
– Alumni
– LCME report
– Alberta
Universities/Colleges
Graduate Employment
Survey
A Jones, Associate Dean – University of Calgary
Overall Rating of First Year Courses
Excellent 5
4.5
V.Good
4
3.5
Good
3
2.5
Fair
2
1.5
Poor
1
POM
2000
Blood
2001
MSK
2002
CV
Resp
2003
Renal
2004
Endo
2005
Integr
2006
RMEBM
2007
Mean Scores on Certifying Evaluations (System
Courses-Yr 1; SB vs CP)
85
Mean Score
80
75
SB (93-96)
CP (97-07)
70
65
60
POM
Blood
MSK
Card
Resp Renal
A Jones, Associate Dean – University of Calgary
Endo
Mean Scores on Certifying Evaluations (System
Courses-Yr 2; SB vs. CP)
85
Mean Score
80
75
SB (93-96)
CP (97-06)
70
65
60
Neuro
Mind
Repro
A Jones, Associate Dean – University of Calgary
GI
ALBERTA LEARNING GRADUATE EMPLOYMENT
SURVEY 2004 GRADUATES FROM 2002 MEDICINE
Usefulness of Your Education in Achieving:
•
Research Skills
•
Working with Others
•
A Desire to Learn More93%
•
Learn Independently
97%
•
Awareness of Ethical
Issues
97%
80%
97%
A Jones, Associate Dean – University of Calgary
ALBERTA LEARNING GRADUATE
EMPLOYMENT SURVEY 2004
GRADUATES FROM 2002
MEDICINE
• Satisfaction with the quality of
teaching in your program?
100%
• Satisfaction with overall quality
of your educational experience 100%
• University of Alberta
83%
A Jones, Associate Dean – University of Calgary
ALBERTA LEARNING GRADUATE
EMPLOYMENT SURVEY 2004
GRADUATES FROM 2002
MEDICINE
• I would recommend the same
program of study to
someone else.
• Satisfaction with
Relevance of Courses
100%
96%
A Jones, Associate Dean – University of Calgary
Overall Quality of Education at U of C by Faculty:
% Satisfied or Very Satisfied
100
90
80
70
60
50
40
30
20
10
0
FA
HU
SS
CC
SC
HA
ED
KN
EN
NU SW LA MD
Data Source: 2002 Alberta Universities/Colleges’ Graduate
Employment Survey re: 2000 Grads
Strongly
Agree
Agree
5
4.5
4
3.5
No
Opinion
U of C 05
All Schools 05
3
2.5
Disagree
2
1.5
Strongly
Disagree
1
1
2
3
4
5
6
1: I am confident that I have acquired the clinical skills required to begin a residency program
2. I have the communication skills necessary to interact with patients and health professionals
3. I have basic skills in clinical decision making and the application of evidence based
information to medical practice
4. I have the fundamental understanding of the issues in social sciences of medicine
5. I have the ethical and professional values that are expected of the profession
6. I have the fundamental understanding of the basic disease mechanisms, clinical
presentations and principles of diagnosis and management for common conditions
Data Source:
Canadian Graduate
Questionnaire 2005
“I AM SATISFIED WITH THE QUALITY OF MY MEDICAL
EDUCATION”
70
60
Percent
50
U of C 03
All Achools 03
U of C 04
All Schools 04
U of C 05
All Schools 05
40
30
20
10
0
Strongly Agree
Agree
No Opinion
Disagree
Strongly
Disagree
A Jones, Associate Dean – University of Calgary
Data Source:
Canadian Graduate
Questionnaire 2003,
2004 & 2005
MCC TOTAL SCORE: 1992-2005
580
Mean Total Score
560
540
520
U of C
Canadian
500
480
460
440
92 93 94 95 96 97 98 99 0
1
2
3
Class
A Jones, Associate Dean – University of Calgary
4
5
Performance on national exams
Performance on LMCC – Clinical Reasoning:
1994-2002
560
550
Mean Score
540
530
520
U of C
Canadian
510
500
490
480
470
94
95
96
97
98
99
0
1
Class
A Jones, Associate Dean – University of Calgary
2
720 – U of C
888 - Canadian
Grads/Canadian Trained
A Jones, Associate Dean – University of Calgary
328 – U of C
111 - Canadian
Grads/Canadian Trained
MCC Subscale & Final Scores– Class 2003
Canadian (84), International Students (7), Total (91)
560
540
520
Mean Score
500
480
U of C Can
International
460
U of C Total
MCC Mean
440
420
400
380
360
Cleo
Med
O & G Peds Phello
Psy
Surg
MCQ
CRS
A Jones, Associate Dean – University of Calgary
Final
Percent
CaRMS: PERCENT OF MATCHED STUDENTS MATCHING TO
FIRST CHOICE DISCIPLINE IN 1ST ITERATION
CLASSES 2001- 2005
100
95
90
85
80
75
70
65
60
55
50
Mem
Dal
McG
Ott
Qu
1
TO
2
McM
3
4
West
Man
Sk
AB
Cal
BC
5
A Jones, Associate Dean – University of Calgary
Data Source:
CaRMS
RESIDENT DIRECTORS’ ASSESSMENT OF GRADUATES
(PGY1)
“Overall Performance - ability to function as a resident with a full workload”
100
90
80
Percent
70
64
62
60
62
60
56
50
44
40
32
30
37
32
40
20
10
6
4
1
0
2000
2001
Weaker than most
2002
Similar to most residents
2003
2004
Stronger than most
Class 2000 N = 50 (71%); Class 2001 N = 45 (68%) Class 2002 N = 40 (57%); Class 2003 N = 79(90%)
Class 2004 N = 76 (82%)
A Jones, Associate Dean – University of Calgary
Data Source:
Program
Directors’ Survey
Cl
in
ic
al
an
ag
em
en
t
Pr
of
Sk
es
ill
si
s
on
al
De
m
ea
Hu
no
m
ur
an
is
tic
Q
ua
lit
Pr
ie
s
es
en
ta
tio
n
Sk
ill
s
Se
lfAw
Ps
ar
en
yc
es
ho
s
so
ci
al
Se
ns
iti
Pe
vi
rfo
ty
rm
an
ce
O
ve
ra
ll
Pa
tie
nt
M
Ju
dg
m
en
t
Cl
in
ic
al
Kn
ow
le
dg
e
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
M
ed
ic
al
Percent
Resident Program Directors’ Assessment
of 2006 Graduates
Weaker
Similar
Stronger
Data Source: Program
Directors Survey
A Jones, Associate Dean – University of Calgary
Resident Program Directors’ Assessment
of 2005 Graduates
100%
90%
80%
60%
50%
40%
30%
20%
10%
an
ce
O
ve
ra
ll
vi
ty
oc
ia
l
os
Ps
yc
h
Pe
rfo
rm
Se
ns
iti
es
s
w
ar
en
Se
lfA
n
nt
at
io
st
ic
an
i
al
Hu
m
Similar
Sk
ill
s
ie
s
Q
ua
lit
ea
no
De
m
ic
al
Pr
of
es
si
on
Cl
in
Weaker
Pr
es
e
ur
s
Sk
ill
en
t
an
ag
em
M
Pa
tie
nt
ic
al
Cl
in
ed
i
ca
lK
no
Ju
d
w
le
d
ge
gm
en
t
0%
M
Percent
70%
Stronger
Data Source: Program
Directors Survey
A Jones, Associate Dean – University of Calgary
Resident Program Directors’ Assessment
of 2004 Graduates
100%
90%
80%
60%
50%
40%
30%
20%
10%
an
ce
O
ve
ra
ll
vi
ty
oc
ia
l
os
Ps
yc
h
Pe
rfo
rm
Se
ns
iti
es
s
w
ar
en
Se
lfA
n
nt
at
io
st
ic
an
i
al
Hu
m
Similar
Sk
ill
s
ie
s
Q
ua
lit
ea
no
De
m
ic
al
Pr
of
es
si
on
Cl
in
Weaker
Pr
es
e
ur
s
Sk
ill
en
t
an
ag
em
M
Pa
tie
nt
ic
al
Cl
in
ed
i
ca
lK
no
Ju
d
w
le
d
ge
gm
en
t
0%
M
Percent
70%
Stronger
Data Source: Program
Directors Survey
A Jones, Associate Dean – University of Calgary
UNDERGRADUATE MEDICAL
EDUCATION ALUMNI SURVEY
CLASSES of 1992-2002
A Jones, Associate Dean
OVERALL OPINION OF THE
UNDERGRADUATE MEDICAL
EDUCATION PROGRAM ALUMNI SURVEY
CLASSES 1992-2002
• 97% Satisfaction with the UME program at
University of Calgary
• 90% Felt prepared or very prepared for Post
Graduate Training
• 98% Would advise their child or child of a
relative or friend interested in Medicine to
apply to the University of Calgary
A Jones, Associate Dean – University of Calgary
OPNION OF ALUMNI 1992 – 2002
CURRICULUM STRENGTHS AND WEAKNESSES
Instruction
Strength
Neither Strength or Weakness
Bedside Clinical Correlation
92%
8%
88%
11%
Small Group Problem
Solving
86%
12%
Classroom Patient
Presentation
83%
15%
Learning Experiences
Appropriate to Educational
Objectives
80%
18%
Lectures
63%
31%
Integration of Basic and
Clinical Sciences
53%
33%
Clinical Instruction Overall
Why Curriculum renewal is
Important
“A curriculum is like water. It has the
tendency to seek the lowest level of energy
it can reach, and without constant renewal,
it will stagnate and become putrid. To avoid
stagnation alone is justification for action.”
Acad Medicine Sept 1998
A Jones, Associate Dean – University of Calgary
Barriers to Medical School Curriculum
Changes
Listed by North American Academic Deans:
•
•
•
•
•
Already crowded curriculum
Inadequate funding
Faculty resistance
Professional ‘turf’ issues
Scheduling conflicts
Graber et al. Acad Medicine 1997
A Jones, Associate Dean – University of Calgary
Curriculum –
A Planned Educational Experience
• Define the outcome measures.
• Create an evaluation system to be sure these
outcomes are realized.
• Develop the pathways to get to these outcomes.
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
Goals for a Revised Curriculum
A revised curriculum has to be consistent with
available information on clinical problem
solving and reflect basic principles of adult
learning.
A Jones, Associate Dean – University of Calgary
Medical students don’t remember or can’t use
the knowledge they learned in the traditional
basic science courses because the knowledge
is structured into mental organizations that are
not useful in the clinic
Barrows, 1985
A Jones, Associate Dean – University of Calgary
Problem Based Learning –
Benefits
•
•
•
•
•
Activate prior knowledge
Learn in context of clinical problem
Interest in learning stimulated
Self directed learning encouraged
Life long learning encouraged
Schmidt - Norman
A Jones, Associate Dean – University of Calgary
Problem Based Learning Concerns
•
•
•
•
Problem solving skills are not augmented
Significant gaps in knowledge occur
Incorrect integration of basic sciences
Tendency to engage in backward reasoning
Albanese; Mitchell
Academic Medicine
A Jones, Associate Dean – University of Calgary
Clinical Reasoning
• Clinical Reasoning and clinical knowledge
are interdependent.
• Effective problem solving requires a large
store of relevant knowledge.
• Clinical expertise is linked to depth and
organization of clinical knowledge.
A Jones, Associate Dean – University of Calgary
Problem Solving Skills In Medicine
Research has proven that experts in
specific domains learn knowledge and
problem solving skills for each problem
simultaneously. That is, knowledge
acquisition and clinical reasoning go
hand-in-hand.
Schmidt et al 1992
Efforts to Help Students Improve
Clinical Reasoning
Education must focus on the
development of adequate knowledge
structures. Teaching, coaching,
supervising must strongly encourage and
nurture actual knowledge organization of
the students.
Knowledge keeps no better than
fish
Alfred North Whitehead 1929
Clinical Reasoning and Small
Group Cases
It is useful to select one model of
clinical reasoning and base the tutorial
discussion on it. The precise model is less
important than its generic use as a
framework to structure the flow of
discussion. It later serves as a fall-back
strategy in complicated clinical situations.
Structure of Medical Knowledge in
Memory Categories and Prototypes
Both medical textbooks and classroom teaching
abound in the limitless presentation of detailed lists
of disorders. More often, both fail to provide a
categorization scheme that is best suited for their
retrieval in a clinical problem solving situation.
Bordage
Med Educ 1984
Types of Curricula
•
•
•
•
Disciplinary
Systems-based
Problem-oriented
Clinical Presentations based
Clinical Presentation Curriculum
Faculty
Identify
Identify
Represented by
Clinical Presentation
Core Competencies for
Clinical Presentation
Curriculum
Committee
Develop
Schematic Problem
Solving Pathway
For the Process of
Enabling
Basic Science
Objectives
Terminal
Objectives
Plans and
Monitors
Curriculum
Course Content
Together Represent
Teaching Methods
Clinical Reasoning
Graduation
Competencies
Guidelines
for
Learning Content
Evaluation
Steps in Development and Dissemination of
Clinical Presentation Objectives
1. Selection of clinical problem.
2. Classification system developed to help organize
knowledge needed to solve the clinical problem.
3. Key Features; Discriminating features identified
of prototypic prevalent disorders.
Steps in Development and Dissemination of
Clinical Presentation Objectives
4. Objectives and problem solving schemes
developed.
5. Distribution to Faculty for balanced input from
teachers generalists, specialists, and biomedical
scientists.
6. Endorsements of objectives.
Steps in Development and Dissemination of
Clinical Presentation Objectives
7. Dissemination of objectives.
8. Encouragement of implementation of objectives in
teaching, learning, clinical practice and problem
solving
9. Monitor and evaluate the translation of objectives
and problem solving schemes into practice.
Clinical Reasoning
Student Identifies Clinical
Presentation
Broad Classification of Problem
Schematic
Problem
Solving
Pathway
Identify Causal Alternatives and
Discriminating Key Factors
Differential Diagnosis
Diagnosis
Management Plan
The Scheme
• Causal Categories
- pre, post and renal causes of acute renal failure
• Diagnoses
- specific diagnoses for each causal category
• Basic sciences
- Integral part
- Timely presentation of content
“Ask any physician of 20 years standing how he has
become proficient in his art and he will reply, by
constant contact with disease; and he will add that
the medicine he learned in schools was totally
different from the medicine he learned at the
bedside.”
Wm. Osler 1932
Bleeding Tendency/Bruising
Hx PE DDx Invest. NatHx Mgmt
General Objectives
Thrombocytopenia
Disordered
Platelet Function
Congenital
Coagulation Disorders
Acquired Coagulation
Disorders
Vascular
Abnormalities
W Surgery
Student
U
Radiology
T
Psychiatry
S
Physiology
R
Pharmacology
Q
Pediatrics
O
Pathology
Oncology
V
a
N
M Office of Medical Education/Informatics/Culture, Health and Illness
L
Office of Medical Bioethics
K
Obstetrics & Gynecology
J
Neuroscience
I
Microbiology
H
Medicine
G
Immunology
F
Genetics
E
Family Medicine
D
Community Health Sciences/Nutrition/Prevention
C
Biochemistry
B
Anesthesia
A
Anatomy
Natural History, Prevention,
Bleeding
Physical
Differential
Prognosis &
Treatment &
History
Investigation
Tendency/
Examination Diagnosis
Complications Complications
Bruising
of Condition
of Treatment
b
General
Objectives
c
Thrombocytopenia
d
Disordered Platelet
Function
e
Congenital
Coagulation
Disorders
f
Acquired
Coagulation
Disorders
g
Vascular
Abnormalities
0
1
2
3
4
5
6
Schematic Problem Solving Pathway
Clinical Presentations
Bleeding Tendency/
Bruising
Platelets
Decreased
Number
Abnormal
Function
Coagulation
Congenital
Acquired
Vascular
Congenital
Acquired
Broad Classification
of Problem
Causal Alternatives
and Discriminating
Key Factors
Differential Diagnosis
Pharmacology
(ASA, Heparin)
Histology
(Bone Marrow)
Anatomy
(Spleen)
Basic Science
Objectives for
Bruising and
Bleeding
Genetics
(Hemophilia)
Physiology
(Hemophilia)
Pathology
Immunology
(Vessels)
(ITP, Vasculitis)
Basic Science or Biomedical
Knowledge in the Undergraduate
Program
The purpose of basic science teaching is to
provide a scientific foundation for tasks of
clinical practice such as diagnosis and
therapeutics. The essential challenge of
balancing depth of understanding with
breadth of coverage remains.
(See p. 35, Fig. 4.1)
W Surgery
Student
U
Radiology
T
Psychiatry
S
Physiology
R
Pharmacology
Q
Pediatrics
O
Pathology
Oncology
V
a
N
M Office of Medical Education/Informatics/Culture, Health and Illness
L
Office of Medical Bioethics
K
Obstetrics & Gynecology
J
Neuroscience
I
Microbiology
H
Medicine
G
Immunology
F
Genetics
E
Family Medicine
D
Community Health Sciences/Nutrition/Prevention
C
Biochemistry
B
Anesthesia
A
Anatomy
Natural History, Prevention,
Bleeding
Physical
Differential
Prognosis &
Treatment &
History
Investigation
Tendency/
Examination Diagnosis
Complications Complications
Bruising
of Condition
of Treatment
b
General
Objectives
c
Thrombocytopenia
d
Disordered Platelet
Function
e
Congenital
Coagulation
Disorders
f
Acquired
Coagulation
Disorders
g
Vascular
Abnormalities
0
1
2
3
4
5
6
Fever
Fever
< 2 weeks
> 2 weeks
Infectious
Bacterial
Viral
ACUTE VISUAL LOSS
PreRetinal
• Corneal edema (glaucoma)
• Vitreous hemorrhage (Diabetes)
Retinal
• Acute Macular Lesion (hemorrhage)
• Retinal Detachment (spontaneous)
• Retinal Artery Occlusion (carotid emboli)
Post Retinal
• Optic Neuritis (MS)
• Ischemic Optic Neuropathy (Temp. Arteritis)
• Occipital Infarction/Hemorrhage
Chrichton, Verstraton, Fletcher
Fluctuating Altered Cognition
Delirium and Confusional States
In the Head
•CNS Infections
•Seizures
•Hypertensive Enceph
•Psychiatric Disorders
Out of the Head
•Toxins
•Metabolic Derangements
•Systemic Organ Failure
•Physical Disorders
Strengths of Year I-II Curriculum
• Approaches to problem solving – clinical
reasoning
• Early clinical exposure
• Small group teaching
• Clinical correlation; patient presentations
• Basic science integration with problem
solving
• IST
• Medicine 440; elective time
• Communication; physical examination
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
Weakness of Year I-II Curriculum
• Exams not always reflective of teaching
‘emphasis’
• Lack of pharmacology
• Faculty not promoting core documents;
teaching to objectives not always clear
• Small group teaching variable
• Problem solving with schemes course
dependent
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
The Curriculum
First Year Courses
Aug
Jan.2
Oct Nov Dec 19
P FOR MBlood-GI
(Course I)
MSKDermOpthENT
(Course
II)
H
O
L
I
D
A
Y
2
w
ks
CVRESP
(Course
III)
Mar. 26 Mar. 29
RenalEndocrine
(includes
Obesity P for
M)
(Course IV)
June -July 18
H
O
L
I
D
A
Y
2
w
k
s
E
L
E
C
T
I
V
E
2wk
s
MEDICAL SKILLS PROGRAM
RMEBM-Health Promotions-Disease Prevention-Population Health
Number of weeks for First Year = 45 weeks
A Jones, Associate Dean
R
E
W
R
I
T
E
S
The Curriculum
Second Year Courses
July
18
E
L
E
C
T
I
V
E
4
wks
Aug. 18
NeuroscienceAging
(Course V)
Oct. 10
XMAS
H
O
L
I
D
A
Y
Infant-ChildReproductionGenetics
(Course VI)
2 wks
A Jones, Associate Dean
Jan. 3
MindFamily
(Course
VII)
Feb. 6
Mar. 5
Integrative
Course
Introduction
to Clerkship
Teaching Methods
• Lectures for rapid acquisition of key
content
• Small group case based learning
- In depth self-directed (or guided) learning
- Review, reinforcement, practice and feedback
- Problem solving, motivation, pertinence
• Clinical correlation
- Bedside sessions
Clinical Reasoning
Clinical reasoning does not develop in
isolation: it is associated with increasingly
refined and elaborated medical knowledge.
Problem solving is domain-specific and not
generic, so the challenge for medical
educators is not only to make explicit the
process of reasoning but also to identify the
necessary content.
Schmidt et al 1990
Kassirer 1995
Usual Sequence of Instruction
• Presentation and scheme shown
- Case based or otherwise
• Lectures or PBL sessions planned
- In depth knowledge and acquisition
- Basic and clinical sciences
• Small group sessions for reinforcement
- Shorter case scenarios for review
- Clinical correlation
Advantages of New Curricular Structure
Curriculum
• Courses will be linked to graduation
objectives and UME program philosophy
of teaching, learning and evaluation.
• Linkage of courses will integrate CP
better and reduce redundancies.
Task Force Report
Advantages of New Curricular Structure
Curriculum
• Clinical presentation list will be revisited
and clerkships will adopt appropriate
presentations.
Task Force Report
Scheme Use Reported by First & Second Year Students
(Classes of 2007 and 2006)
Strongly
Agree
5
4.5
4
3.5
Neutral
3
2.5
2
1.5
Strongly
Disagree
1
Learning
Prob-Solving
1st Yr
Often Referred
2nd Yr
Integration
Data Source: Classes 07 & 06 yr
end CP curriculum evaluation
MAP OF DEPARTMENT INVOLVEMENT IN UME CURRICULUM*
COURSES IN YEARS I AND II
Department
I
II
III
IV
HPOP
V
Medicine
X
X
X
X
X
X
Family Med
X
Pediatrics
X
X
X
VI
Int.
Intro
Clerk
MedSkills
X
X
X
X
X
X
X
X
?
Psychiatry
Surgery
X
X
X
X
Neurosci
Anatomy
X
Oncology
X
X
X
X
X
X
X
X
?
X
X
X
X
X
Medical
Genetics
Anesthesia
X
X
Community
Health
ER
X
X
X
Obst-Gyn
Pathology
VII
X
X
X
X
X
X
X
X
NUMBER OF HOURS TAUGHT IN THE
UNDERGRADUATE MEDICAL PROGRAM YEARS I-II
GFT and CLINICAL FACULTY
Hours
Relative Dept Size Average
Teaching hrs per Member
Medicine
1844
7
Family Med
1189
7
Pediatrics
694
5
Neuroscience
514
8
Psychiatry
458
4
Surgery
437
3
Pathology
257
4
Anatomy
252
10*
Oncology
242
3
Obst-Gyn
155
6
Com Health
78
1
Med Genetics
51
4
LEARNING METHODS USED DURING
YEARS I AND II UNIVERSITY OF CALGARY
UNDERGRADUATE PROGRAM
Small Groups/Bedside: 1/3
Large Groups/Classroom: 1/3
Self Directed Study: 1/3
A Jones, Associate Dean
SUMMARY OF STRUCTURED TEACHING HOURS
UNDERGRADUATE MEDICAL EDUCATION PROGRAM
YEARS I AND II 2004-2005
Hours
%
Non-GFT
5316
68
GFT
2453
32
TOTAL
7769
100
*Medicine 440, Summer Elective not included
A Jones, Associate Dean
Ownership of the Curriculum
"Faculty members own what is taught in the
curriculum - they own the content. The Associate
Dean and Curriculum Committee are responsible for
the methods and the effectiveness. Faculty give the
course, the Associate Dean has to give the degree.”
Academic Medicine
Vol. 73, Pg. 54
The Seven Most Dangerous Words in
Medical Education
“…but we’ve always done it this way”
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
Prototype Timeline for a Major
Educational Change
1. Decide what to do.
2. Build support for the idea.
3. Acquire resources.
4. Do it wrong the first time.
5. Do it wrong the second time.
6. Do it passably the third time.
7. Do it reasonably well the fourth time.
Friedman Acad Medicine 1993