Establishing Content Validity of a Novel
Written Examination to Assess Medical
Students on the Surgery Clerkship
A Berlin MD, A Reinert BA, A Swan-Sein MEd PhD,
R Nowygrod MD, A Fingeret MD
Agenda
•
•
•
•
•
•
Needs assessment
Exam Development
Exam Design
Methodology
Results
Discussion
Needs Assessment
• Historical assessment of students
– Clinical evaluations
– Unstructured oral exam
– NBME surgery subject exam
• Goals of new evaluation tool
– Objective and reliable
– Congruent with educational objectives
– Based on ACGME core competencies
Exam Design
• Novel Exam: Clinical Skills Exam (CSE)
– 20+ clinical scenarios
• 5 randomly selected per exam
– Short answer or essay response
– Administered over 3 hours
– ExamSoft computer software
• Grading
– Rubric
– Senior residents
Example 1
• Identify
operation
• Obtain surgical
consent
• Determine
order
• Describe steps
shown
Example 2
• Generate differential
based on history
• Describe risk factors
• Interpret radiographic
study
• Identify next step in
management
Methodology
• July, 2011 - February, 2013
– n = 262
• Analysis
– Descriptive statistics
– Multivariate analysis
• Comparison data for validity assessment
–
–
–
–
USMLE Step 1 and 2CK
NBME Subject Exams in Medicine and Surgery
Clerkship clinical evaluations
Clerkship order
Results: Descriptive Statistics –
CSE Content and Score Distribution
45
40
Communication Anatomy
12%
8%
35
30
History
13%
Physical
9%
Differential
8%
Radiology
10%
Frequnecy
Management
40%
25
20
15
10
5
0
48 54 60 66 72 78 84 90 96
Score (%)
Results: Trend Analysis –
CSE and NBME Surgery Subject Exam
Score (%)
Clinical Skills Examination
100
90
80
70
60
50
p = 0.45
p = 0.32
Median
Mean +SD
Mean -SD
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Score
NBME Surgery Subject Examination
100
90
80
70
60
50
p = 0.04
p = 0.02
Median
Mean +SD
Mean -SD
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Results: Multivariate Statistics Validity Analysis
CSE
Linear
Correlation Regression
r
p value
NBME Surgery
0.36*
0.82
NBME Medicine
0.01
0.54
Step 1
0.34*
0.82
Step 2
0.44*
0.007
Clerk - Clinical
0.48*
0.001
Reasoning
Clerk - Fund of
0.56*
0.004
Knowledge
* = p < 0.001
NBME
Correlation
r
Linear
Regression
p value
0.08
0.55*
0.56*
0.20
0.02
0.004
0.43*
0.09
0.63*
< 0.001
Results: Multivariate Statistics –
Validity Analysis
CSE
NBME
Bivariate
Analysis
p value
Bivariate
Analysis
p value
Logistic
Regression
p value
ANOVA
0.30
< 0.001
0.01
OB/Gyn prior
X2
0.34
< 0.001
0.02
Medicine prior
X2
0.69
< 0.001
0.03
Clerkship order
Results: Student Feedback on CSE
9
8
7
6
5
4
3
2
1
Objectives
Relevance
Clarity
Feedback
Overall
Clerkship
Results: Student Feedback - Word Cloud
Limitations
–Study
• Generalizability
• Statistical analysis
Future Directions
• Future directions
– New content
– Open access
– Computer algorithm based grading
– Use as formative assessment mid-clerkship
Conclusions
• CSE is a useful adjunct to NBME surgery
subject exam
– Independent of clerkship order
– Course goals and objectives
– ACGME core competencies
• Questions / Comments