Examiner training for OSCEs
Katharine Boursicot
Barts and the London, QMUL
School of Clinical Medicine, University of Cambridge
Trudie Roberts
University of Leeds Medical School
Programme
•
•
•
•
Principles of OSCEs for examiners
Video marking
Marking live stations
Strategies for enhancing examiner
participation in training
http://www.ltsn-01.ac.uk
Academic principles of
OSCEs
• The basics
• What is an OSCE?
• More academic detail
• Why use OSCEs?
• The role of examiners
• Examiners in OSCEs
http://www.ltsn-01.ac.uk
The basics
• For examiners who don’t know about
OSCEs
• A brief reminder for those who are familiar
with OSCEs
http://www.ltsn-01.ac.uk
What is an OSCE?
•
•
•
•
Objective
Structured
Clinical
Examination
http://www.ltsn-01.ac.uk
OSCE test design
Station
http://www.ltsn-01.ac.uk
OSCEs - Objective
• All the candidates are presented with the
same test
http://www.ltsn-01.ac.uk
OSCEs - Structured
• The marking scheme for each station is
structured
• Specific skill modalities are tested at each
station
•
•
•
•
History taking
Explanation
Clinical examination
Procedures
http://www.ltsn-01.ac.uk
OSCEs – Clinical Examination
• Test of performance of clinical skills: not a
test of knowledge
• the candidates have to demonstrate their
skills
http://www.ltsn-01.ac.uk
More academic detail
• Why use OSCEs in clinical assessment?
• Improved reliability
• Fairer test of candidate’s clinical abilities
http://www.ltsn-01.ac.uk
Why use OSCEs in clinical
assessment?
• Careful specification of content
• Observation of wide sample of activities
• Structured interaction between examiner
and student
• Structured marking schedule
• Each student has to perform the same
tasks
http://www.ltsn-01.ac.uk
Characteristics of
assessment instruments
• Utility =
•
•
•
•
•
Reliability
Validity
Educational impact
Acceptability
Feasibility
Reference
Van der Vleuten, C. The assessment of professional competence:
developments,research and practical implications Advances in Health
Science Education 1996, Vol 1: 41-67
http://www.ltsn-01.ac.uk
Test characteristics
• Reliability of a test/
measure
• reproducibility of scores
across raters, questions,
cases, occasions
• capability of differentiating
consistently between
good and poor students
http://www.ltsn-01.ac.uk
Sampling
Domain of Interest
Test Sample
Test Sample


http://www.ltsn-01.ac.uk
Reliability
• Competencies are highly domain-specific
• broad sampling is required to obtain
adequate reliability
• across content i.e. range of cases/situations
• across other potential factors that cause error
variance i.e.
• testing time, examiners, patients, settings, facilities
http://www.ltsn-01.ac.uk
OSCE : blueprint
History
Explan
Exam
Procedure
CVS
Chest pain
Disch drugs
Cardiac
BP
RS
Haemoptysis
Smoking
Resp
Peak flow
GIS
Abdo pain
Gastroscopy Abdo
Repro
Amenorrhoea Abnormal
smear
Cx smear
NS
Headache
Eyes
MS
Backache
Hip
Generic
Pre-op
assess
Consent for
post mortem
PR
Ophthalmosc
IV cannulation
Blood trans rea
http://www.ltsn-01.ac.uk
Test characteristics
• Validity of a
test/measure
• the test measures the
characteristic (eg
knowledge, skills) that
it is intended to
measure
http://www.ltsn-01.ac.uk
Model of competence
Professional authenticity
Does
Shows how
Behaviour~
skills/attitudes
Knows how
Knows
Cognition~
knowledge
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S67.
http://www.ltsn-01.ac.uk
Validity of testing formats
Professional practice assessment
Does
Shows how
Knows how
Knows
Performance assessment: OSCEs,
long/short cases, OSLERs, etc
Problem-solving assessment:
EMQs, SEQs
Knowledge assessment: MCQs
http://www.ltsn-01.ac.uk
Test characteristics:
Educational impact
Relationship between assessment and learning
Curriculum
Assessment
Teacher
Student
http://www.ltsn-01.ac.uk
Test characteristics
• Feasibility
• cost
• human resource
• physical resources
http://www.ltsn-01.ac.uk
Test characteristics
• Acceptability
• tolerable effort
• reasonable cost
• Acceptability
•
•
•
•
•
•
doctors
licensing bodies
employers
patients/consumer groups
students
faculty
http://www.ltsn-01.ac.uk
The role of examiners in OSCEs
•
•
•
•
General
Types of stations
Standard setting
Practice at marking
http://www.ltsn-01.ac.uk
The role of examiners in OSCEs
• To observe the performance of the student
at a particular task
• To score according to the marking
schedule
• To contribute to the good conduct of the
examination
http://www.ltsn-01.ac.uk
The role of examiners in OSCEs
• It is NOT to:
•
•
•
•
•
Conduct a viva voce
Re-write the station
Interfere with the simulated patient’s role
Design their own marking scheme
Teach
http://www.ltsn-01.ac.uk
Types of OSCE stations
•
•
•
•
History taking
Explanation
Clinical examination
Procedures
http://www.ltsn-01.ac.uk
Communication skills
• Stations involving patients, simulated
patients or volunteers
• Content vs process i.e
what the candidate says
vs
how the candidate says it
http://www.ltsn-01.ac.uk
Clinical skills
• People
• Professional behaviour
• Manikins
• Describe actions to the examiner
http://www.ltsn-01.ac.uk
The examiner’s role in
standard setting
• Use your clinical expertise to judge the candidate’s
performance
• Allocate a global judgement on the candidate’s
performance at that station
• Remember the level of the examination
http://www.ltsn-01.ac.uk
Global scoring
Excellent pass
Very good pass
Clear pass
Borderline
Clear fail
http://www.ltsn-01.ac.uk
Borderline method
Test score distribution
Checklist
1. Hs shjs sjnhss sjhs sjs sj

2. Ksks sksmsiqopql qlqmq q q
qkl
3. Lalka kdm ddkk dlkl dlld
4. Keyw dd e r rrmt tmk
5. Jfjfk dd


6. Hskl;s skj sls ska ak akl ald
Borderline score distribution
7. Hdhhddh shs ahhakk as
TOTAL

Pass, Fail, Borderline P/B/F
Passing score
http://www.ltsn-01.ac.uk
Regression based standard
Checklist
1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld
4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as



TOTAL
X = passing score


Checklist
Score
X
Overall rating 1 2 3 4 5
1
2
3
4
5
=
=
=
=
=
Clear fail
Borderline
Clear pass
V Good pass
Excellent pass
1
2
3
Clear
fail
Borderline
Clear
pass
4
V Good
pass
5
Excellent
pass
http://www.ltsn-01.ac.uk
Practice at marking
• Videos
• Live stations
• Mini-OSCE
http://www.ltsn-01.ac.uk
http://www.ltsn-01.ac.uk
Strategies for enhancing
examiner participation
1.
2.
3.
4.
CME
Job plan/ part of contract
Specific allocation of SIFT
Experience for post-graduate
examinations
5. Payment
http://www.ltsn-01.ac.uk