The Role of Multiple Mini Interviews
(MMI) in selection
Dr Jon Dowell, Admissions Convenor,
Dundee Medical School
1
Overview
 Rationale behind MMI for selection
 A brief history
 What do MMIs assess
 How do MMIs work
 Performance in UK context
 Feedback from applicants and assessors
 The future?
Primus
Medicine has ‘previous’
Alexander Monros
1720 - 1846
Secundus
"I dislike him & his lectures so
much that I cannot speak with
decency about them. He is so
dirty in person & actions."
Tertious
Rationale
 Personal qualities considered ‘important’ but not reliably
assessed by interview and without much predictive validity.
(Goho + Blackman 2006. cognitive r 0.06, clinical r 0.17)
 Introduced for medicine at McMaster, Canada (Reiter, Eva et al).




Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7
Predicted OSCE performance ß 0.44 (interview ns)
Clerkship ratings ß 0.57 (interview ns)
MCC Part 1 (selected components) ß 0.3-0.4 (interview ns)
Increasing detail emerging
A cost efficiency comparison between the multiple mini-interview and traditional
admissions interviews.
Set up costs, running equates
Rosenfeld JM, Reiter HI, Trinh K, Eva KW.
Adv Health Sci Educ Theory Pract. 2008 Mar;13(1):43-58.
The multiple mini-interview: how long is long enough?
10 – 8 – 5 mins
Dodson M, et al. Med Educ. 2009 Feb;43(2):168-74
Factors affecting the utility of the multiple mini-interview in selecting candidates
for graduate-entry medical school.
8Q reliability 0.7
Roberts et al. Med Educ. 2008 Apr;42(4):396-404 applicant
22% variance
Should candidate scores be adjusted for interviewer stringency or leniency in the
multiple mini-interview?
Rasch Modeling
Roberts C, Rothnie I, Zoanetti N, Crossley J. Med Educ. 2010 Jul;44(7):690-8
Increasingly widespread
Countries
• Canada,
• Australia
• UK
• US
Disciplines
• UG Medicine
• UG Veterinary
• UG Dentistry
MMI in Dundee
7 minute x 10 stations
Reliability
– 2008/9 n 450
– 2009/10 n 500
– 2010/11 n 600
0.7
0.69
0.69/.67 (0.88)
– 2011/12 Dental school collaboration
MMI in St Georges
5 minute stations x 8 Stations
Muriel Shannon +Aileen O’Brien
Reliability
– 2009/10
– 2010/11
n 1078
n 1333
0.69
MMI in Belfast
5 minute x 9 stations
Keith Steele
Reliability
– 2010/11
n 500
0.56
MMIs typically aim to assess
– Interpersonal and communication skills (inc empathy)
– Teamwork and leadership
– Preparation and motivation
– Critical thinking, problem solving
– Ethical/moral reasoning
– ‘Integrity’
How?
Practicalities
Capacity (n = 600)
20 applicants per ‘Run’ per session
Two teams in parallel (80/day)
8 days total
Staffing (2 academic leaders + 2 admin)
For each run
5 faculty (interview)
5 student or Simulated Patients (interactive)
2 ‘actors’ (employed junior students)
3 ‘role players’ (students or SP)
Preparation = big job
Stations - Drafted and piloted with students
•
•
•
•
Assessor instructions
Candidate instructions
Resources (script, shapes, materials, UCAS forms etc)
Actor training
Training at start of every session
30 min general
20 min station specific
Station/Assessor/
Candidate Information
Domain Scores
Overall Judgement
Red Flag & Narrative
Performance of Dundee
MMI
Score Distribution
Station Reliability
1
ItemTotal
Correl
0.34
Alpha if
Item
Deleted
0.67
2
0.31
0.67
3
0.17
0.70
4
0.37
0.66
5
0.43
0.65
6
0.47
0.64
7
0.34
0.67
8
0.28
0.67
9
0.38
0.66
10
0.39
0.66
Overall Alpha 0.69
Domain Reliability
Number of
Stations
Number
of Scores
Cronbach’s
alpha
Communication
9
12
.767
Critical Thinking
6
6
.453
Moral Reasoning
4
4
.170
Prep +Motivation
3
4
.515
Teamwork
2
3
.395
Integrity
1
1
-
Gender difference - Yes
Interactive
One - one
Females performed significantly better than males.
2009 Female average 109 /150 vs 105 p = < .01 Effect size: 0.14
Applicant Group difference?
2009 – 2010
2010 - 2011
Apologies but we are intending to
eventually publish from this data
so have removed it from circulated
slides.
Assessors Survey 2008-9
116 /201 (58%) assessors completed online survey.
• 91% thought the process fair.
• 88% thought applicant stress moderate or less.
• All stations rated highly relevant, except for the
‘Question of Ethics’ where opinions divided.
Candidate Survey 2008-9
• 324 /433 (75%) completed online survey.
• 94% felt MMI was ‘fair’.
• 90% felt it is a ‘valid way to assess candidates’.
• 71% preferred MMI to traditional interview.
• ‘Student Counselor’ (role play) most enjoyable
We are happy
but where next?
Hawk-Dove Effect
STAT I O N S
A
1
2
3
4
5
6
B
C
D
E
P. H. Harasym, Ph.D.
7
8
9
10
Dundee 2011 Rasch vertical ruler
MFRM creates a
common scale for
candidate ability,
examiner stringency,
item difficulty and ......
Comparing raw and
‘fair’ scores for 350
offers 34 (9.7%)
different candidates
selected.
Conclusions and Next Steps
MMI appear very promising:
• Set up ‘costs’ considerable
• Running costs comparable (if students used)
• Student involvement positive
• Applicants are positive
• Predictive validity emerging and encouraging.
Outstanding issues:
• Rotating content.
• Application of IRT (DIFF, Fair scores etc)
• Oh yes – and the bloopers are great.
Thank you
Discussion?