Jocelyn Lockyer PhD
Senior Associate Dean, Education
Professor, Department of Community Health
Sciences
University of Calgary
1


No financial conflicts of interest to report
College of Physicians and Surgeons of Alberta—
Physician Achievement Review Program
◦ Since 1996
◦ Instrument development, testing, and assessment
2
Collaborators
 CPSA
◦ John Swiniarski, Bryan Ward (deceased), Trevor
Theman
◦ Physician Performance Committee

University of Calgary
◦ Research Team..Claudio Violato, Herta Fidler
◦ Physician Advisor..Ray Lewkonia

Pivotal Research Inc
◦ Steve Dennis
3
◦ Describe the Alberta Physician Achievement Review
(PAR) Program
◦ Discuss the evidence supporting PAR as a tool for
screening and quality improvement
◦ Describe dissemination and uptake by others within
and outside Canada
◦ Provide an opportunity to answer questions about
PAR
4



Multisource feedback program (aka 360
degree evaluation)
Questionnaire based assessment
Feedback about observable behaviors is
obtained every 5 years from
◦ 8 Physician colleagues (peers, referring MDs,
referral MDs)
◦ 8 Co-workers (e.g., nurses, pharmacists, dieticians)
◦ 25 Patients
◦ Self
5
MSF
Medical colleagues
Self-assessment
Co-workers
Patients
6

First level screen which might
◦ Provides physicians with data about their practices to guide
improvement
◦ Identify need for further assessments

Source of unique and useful data that can inform
many of the CanMEDs roles, particularly
◦
◦
◦
◦



Communicator
Collaborator
Professional
Medical expert
Relatively inexpensive
Adaptable
Repeat administration (5 years)
7







General/family physicians (revised
2010)
Surgical specialists
Medical specialists
◦ Pediatrics
◦ Psychiatry
◦ All other medical specialists
Anesthesiologists
“Episodic care” providers
Diagnostic Imaging
Laboratory Medicine
8
Diagram 1
The College’s PAR Program
PAR Survey
Response
Feedback
Self-Analysis
PAR
Committee
Review
Advice to
Physician
Individualized
Assessment
Office Peer
Review
Competence
Assessment
9
Good assessment should offer
1. Validity or coherence
2. Reproducibility or consistency (reliability)
3. Equivalence
4. Feasibility
5. Educational effect
6. Catalytic effect
7. Acceptability
 Norcini et al, Criteria for good assessment: Consensus
statement and recommendations from the Ottawa 2010
Conference, Medical Teacher 2011
10
Criteria
Evidence
Validity
Developed by physicians with input from physicians,
health care professionals and patients.
Factor analyses showing items correlate in intended ways
to form scales.
Positive but low correlation between sources.
Reliability
22-25 patients, 8 colleagues and 8 co-workers for
reliability suitable for formative (lower stakes)
assessment based on G- and D-studies.
Equivalence
Comparisons in Nova Scotia and Alberta.
Comparisons by different schools of graduation.
Acceptability
High response rates from all sources.
>70% of physicians report obtaining valuable feedback.
11
Criteria
Evidence
Educational
effect
49% report making a change in practice as a direct result
of PAR.
Most changes on patient care, communication with
patients, colleagues, and co-workers and stress
management
Improvement in scores when tested 5 years apart.
Catalytic effect
Not studied.
12
Total
9816
Family medicine
4355
Surgeons
1161
Anesthesiologists
Medicine (and subspecialties)
472
1506
Pediatricians
442
Psychiatrists
617
Episodic care
842
Diagnostic imaging
271
Laboratory medicine
150
13




Dissemination
◦ >25 publications in peer review journals
Adopted
◦ Nova Scotia (NS-PAR)
◦ Manitoba (M-PAR)
Being Tested
◦ British Columbia
◦ Council of Teaching Hospitals of Ontario
◦ Royal College of Physicians and Surgeons of Canada
 Practice eligible route
Tested
◦ Medical Council of Ireland
◦ Netherlands (Internal Medicine)
◦ Singapore (Emergency Medicine)
14



Questionnaire based assessment in which feedback
about observable behaviors is provided by other
physicians, co-workers (e.g., nurses, social workers),
and patients
Program that meets criteria for a ‘good assessment’
Initiative that can be adopted and used by others but
attention needs to be paid to
◦ Purpose
 Quality improvement/formative assessment
◦ How it functions in different settings with special reference to
how physicians use the data to inform practice change
15
16