Evaluating the impact of
assessing doctors’
performance
Dr Julian Archer
NIHR Career Development Fellow
& Clinical Senior Lecturer in Medical Education,
Director of CAMERA, Consultant Paediatrician
Evaluating the impact of
assessing doctors’ performance
Outline


Why evaluate impact of assessment?
Programmatic research into medical
revalidation
Some earlier work
GMC revalidation evaluation framework
Why evaluate impact?
APA Validity
framework – Five
headings





Content
Response process
Internal structure
Relationship to other
variables
Consequences
Medical education

Predominately all
validity areas except
consequences
Why?

Consequences are
Complex
Socio-political
Not positivistic
Programmatic approach to evaluation
EVALUATING IMPACT OR
‘CONSEQUENCES’
What is Revalidation?
Research
Questions
Setting
Method
Stage 1 - Policy
Stage 2 - Practice
What is the meaning of
Revalidation?
What are the intended and unintended
consequences of Revalidation in
practice?
The
Literature
Systematic
Review
Policy
makers
Ward/Theatre
/Lab
GP
Private
Practice
Interviewee
drawings
Appraisees &
Appraisers
Focus groups
Video
appraisals
Visual
anthropology
Activity Theory
Conversation
Analysis
Stage 3 - Public
Mental
Health
Appraisee
Interviews
What is the impact of
Revalidation on the
relationship between the
profession and society?
Media
Journalist
Interviews
Analysis of
Media
Patients
Patient
Focus
Groups
Methodology
Discourse analysis
Outputs
•To provide an
understanding of
Revalidation translational
to policy, education and
practice
•To shape and propose positive
consequences while minimising negative
consequences of Revalidation
•To disseminate positive responses about
Revalidation in order to shape future
national policy
•Ascertain the messages
that the public are getting
from the media
•Develop communication
about Revalidation more
cost effectively
What is Revalidation in policy? - To explore the origins, definitions, and
potential purpose of Revalidation
STAGE 1
Methods
Policy review

starting from the Merrison report (1975)
31 unique elite interviews

leading members of Revalidation policy
development ranged from the leaders past
and present of main stakeholder bodies such
as the GMC, AoMRC, BMA, NHS Employers,
4 DHs
Discourse
REGULATION
Driven by a professional
movement for reform
Driven by medical scandal
Internally motivated
Evolutionary Development
Appraisal
Formative Development
Ongoing evaluation (process)
Up-to-date
E N T S
Restore/maintain confidence by
continuing to elevate standards
I
The Quality Agenda: for patients
P A T
PROFESSIONALISM
Externally motivated
Revolutionary change required
The Safety Agenda: for patients
Reassurance by measuring against
a fixed standard
Clinical Governance
Summative Judgment
Point-in-time Decision (product)
Fit-to-practise
Working with the GMC
DEVELOPING EVALUATION
FRAMEWORKS
Methods – evaluative cycle
Step 1: Engage
stakeholders
Step 2: Describe the
programme
Step 3: Focus the
evaluation question
Step 4: Gather credible
evidence
Literature
Interviewed major
stakeholders
Worked with GMC
(Main stakeholder)
Develop evaluative
questions
Develop possible
methods for an initial
pilot
Hansen HF. Choosing Evaluation
Models: A Discussion on
Evaluation Design. Evaluation.
2005;11:447-462.
Stakeholder interview themes
Importance of evaluation

“people who go through the process…without any
thought of what it actually represents and how it might
actually add value to the system”
What is revalidation for?

Drive-up standards (performance improvement) vs.
picking out ‘bad apples’
Appraisal

“to get inside the closed door of appraisal”
RO judgements

“I’m not sure exactly how the RO’s decision is made”
Summary
Impact or consequences of assessment is
an important part of validity evidence
Complex
Programmatic research required (drawing
on theoretical perspectives)
Acknowledgements and thanks
Colleagues


Sam Regan de Bere
CAMERA team
Funders



NIHR
The Health Foundation
General Medical
Council
Evaluating the impact of
assessing doctors’
performance
julian.archer@plymouth.ac.uk
camera.pupsmd@plymouth.ac.uk
www.plymouth.ac.uk/peninsula/research/camera
Twitter: @CAMERAPUPSMD