AECOM COGME Seminar
Implementing the Competencies
360 Degree Evaluations
Catherine C. Skae, MD
Director, Pediatric Residency Program
Children’s Hospital at Montefiore
Outline
 Definition of 360 degree evaluation
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
Outline
 Definition of 360 degree evaluation
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
Definition
 Consist of measurement tools completed by
multiple people in a person’s sphere of
influence
 Evaluators are superiors, peers,
subordinates, patients, and families
 Two practical challenges: constructing the
surveys and orchestrating data collection
Outline
 Definition of 360 degree evaluations
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
ACGME Toolbox
Suggested Best Methods of Evaluation
1= Most Desirable:
Patient Care- Work within a team
Practice-Based Learning & ImprovementUse of information technology
Facilitate learning of others
ACGME Toolbox
Suggested Best Methods of Evaluation
1= Most Desirable:
ProfessionalismEthically sound practice
Sensitive to culture, age, gender, disability issues
Systems-based Practice- Understand interaction of
their practices w/ the larger system
Advocate for patients within the healthcare system
ACGME Toolbox
Suggested Best Methods of Evaluation
2= Next Best Method:
Patient Care- Caring & respectful behaviors
Counsel & educate patients and families
Performance of medical procedures
Interpersonal & Communication SkillsCreation of therapeutic relationships
Listening skills
Professionalism- Respectful, altruistic
Syst-Based Practice- Practice cost-effective care
ACGME Toolbox
Suggested Best Methods of Evaluation
3= A Potentially Applicable Method:
Patient Care- Interviewing;
Developing and Carrying out Management Plans
Medical KnowledgeInvestigatory & Analytic thinking
Knowledge and Application of Basic Sciences
Practice-Based Learning & ImprovementAnalyzing own practice for needed improvements
Outline
 Definition of 360 degree evaluations
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
FacultyInpatient/ Elective
Intern/Resident
FacultyContinuity Clinic
FacultyInpatient/ Elective
Intern/Resident
FacultyContinuity Clinic
Senior
Residents
Junior
Residents
Intern 360 Degree Evaluation
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
CHILDREN’S HOSPITAL AT MONTEFIORE
EDIATRIC ROTATION EVALUATION
JUNIOR/SENIOR RESIDENT: _________________________
INTERN: ________________________________ (1, 2= below expectations; 3 = meeting expectations; 4, 5 = exceeding expectations)
ALL 1,2, AND 3 RATINGS REQUIRE COMMENTS AT END OF FORM
I. PATIENT CARE
Fails to gather essential and accurate information via history and P.E. 1
Experiences difficulty in developing and carrying out plans
1
appropriately
Has difficulty taking responsibility and lacks accountability for decisions
Has difficulty performing technical procedures
1
2 3
2 3
4 5 N/A
4 5 N/A
1 2 3 4 5 N/A Demonstrates responsibility and accountability for decisions
2 3 4 5 N/A Performs technical procedures skillfully
II. MEDICAL KNOWLEDGE
Does not demonstrate independent investigatory and analytic thinking 1 2 3 4 5 N/A
Does not have adequate knowledge base of basic and clinical sciences 1 2 3 4 5 N/A
III. INTERPERSONAL SKILLS AND COMMUNICATION
Does not use effective listening skills to elicit information
Does not work effectively as member of health care team
Has difficulty in creating & sustaining therapeutic relationships
Gathers essential and accurate information via history /P.E.
Develops and carries out patient management plans
1 2
1 2
1 2
Demonstrates independent investigator and analytic thinking
Knows and applies basic and clinical sciences
3 4 5 N/A Uses effective listening skills to elicit information
3 4 5 N/A Works effectively as a member of a health care team
3 4 5 N/A Creates and sustains therapeutic relationships
FacultyInpatient/ Elective
Intern/Resident
FacultyContinuity Clinic
Senior
Residents
Physician
Assistants/
Nurses
Junior
Residents
FacultyInpatient/ Elective
Intern/Resident
FacultyContinuity Clinic
Senior
Residents
Social
Workers
Physician
Assistants/
Nurses
Junior
Residents
Social Work 360 Degree Evaluation
•
•
•
•
•
•
ALBERT EINSTEIN COLLEGE OF MEDICINE- CHILDREN’S HOSPITAL AT MONTEFIORE
•
PEDIATRIC ROTATION EVALUATION
SOCIAL WORKER: _________________________
RESIDENT: _______________________
(1,2 = below expectations; 3 = meeting expectations; 4,5 = exceeding expectations)
•
•
DATE OF EVALUATION: ___________________________
•
•
I. MEDICAL KNOWLEDGE
Knows how to make appropriate Social Services referrals 1
2
3
4
5
N/A
II. INTERPERSONAL SKILLS AND COMMUNICATION
Communicates well with Social Workers
1
2
3
4
5
N/A
III. PROFESSIONALISM
Exhibits a positive attitude toward work
1
2
3
4
5
N/A
Attends and contributes to Interdisciplinary Team Rounds 1
2
3
4
5
N/A
•
•
•
•
•
•
•
•
•
•
•
•
•
PLEASE ADD COMMENTS:
FacultyInpatient/ Elective
Patients/
Families
Social
Workers
Intern/Resident
FacultyContinuity Clinic
Senior
Residents
Physician
Assistants/
Nurses
Interns/ Junior
Residents
Standardized
FacultyInpatient/ Elective
Patients
Patients/
Families
Social
Workers
Intern/Resident
FacultyContinuity Clinic
Senior
Residents
Physician
Assistants/
Nurses
Interns/ Junior
Residents
Outline
 Definition of 360 degree evaluations
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
Results/ Scholarship
Assessing Competencies of Pediatric Residents in an
Outpatient Setting.
Sandra Braganza, Iman Sharif, Philip Ozuah.
Objectives:
To determine whether the assessment of pediatric residents’
ACGME competencies by peers, support staff, and parents
differs from the assessment by preceptors alone.
To identify particular ACGME competencies for which
differences between preceptor and peer/staff/parent
assessments are most pronounced.
Results/ Scholarship
Methods: 14 statements assessing:
Interpersonal & Communication Skills- Communication and
Empathy
Professionalism- Respect & Integrity
Results/ Scholarship
Results: 271 surveys completed
Compared to preceptors, resident peers reported more
problems with Communication and Empathy.
Preceptors noted more problems with Communication than
did parents.
Parents were more likely to report problems with Empathy
and Respect than were preceptors.
Conclusion: 360 degree assessment was useful in eliciting
different evaluations of competencies by different
observers that interact with pediatric residents.
Results/ Scholarship
Parental Assessment of Quality of Ambulatory Care: How
Do Attendings and Residents Compare?
Andrew D. Racine, Sheila Liewehr, Laura Weingart, Scott D.
Krugman. Eastern Society for Pediatric Research. March
2005.
Objective: To compare the quality of pediatric ambulatory
care reported by parents of children with resident versus
attending primary care providers.
Results/ Scholarship
Results: 473 surveys were returned
No differences were found between residents and
attendings in aggregate scores for any of the 5 quality
domains
But on 2 individual items: does the doctor explain things to
your satisfaction, and can the doctor arrange for referralsattending scores were higher.
Conclusions: Parental assessment of the overall quality of
care delivered by pediatric residents compares favorably to
that delivered in similar settings by attending pediatricians.
Results/ Scholarship
Determining the Essential Components of Professionalism
and Interpersonal/ Communication Skills.
Laura Dattner, Philip Ozuah. Pediatric Academic SocietiesMay 2005.
Objective: To determine what pediatric patients and their
parents consider to be essential components of
professionalism and interpersonal/ communication skills.
Results/ Scholarship
Methods: Anonymous web-based survey for hospitalized
patients and their parents. Respondents were asked eight
questions about the residents who cared for them.
Results/ Scholarship
Results: 253 resident evaluation surveys were completed.
Overall, residents received very positive evaluations.
Questions which predicted a global score of 5 were:
“Does Dr. X. speak to you in ways that are clear and easy
to understand?”
“Does Dr. X. include you in decisions and planning of
care?”
“Does Dr. X. listen to your concerns?”
Results/ Scholarship
Conclusions: Essential components of professionalism and
interpersonal/ communication skills which independently
predicted an excellent global score were:
listening to concerns
including families in decisions and planning of care
clear communication
Results/ Scholarship
 Can We Rely on Adolescent Standardized Patients to
Assess Communication Skills?
 Reliability of Adolescent Standardized Patients in
Assessing Professionalism.
 The Utility of Unannounced Standardized Patients in
Medical Education.
Philip Ozuah, Marina Reznik, and Sandra Braganza.
Pediatric Academic Societies- May 2005.
Results/ Scholarship
 Adolescent standardized patients (SP) are a reliable
method for use in assessing residents’ competency in
communication skills.
 There was a strong correlation between faculty and
adolescent SP’s ratings of professionalism.
 Residents displayed lower levels of professionalism in
unannounced standardized patient encounters.
Outline
 Definition of 360 degree evaluations
 ACGME Suggested Best Methods
 Organization/ Timing
 Results/ Scholarship
 Future
The Future
•
•
•
•
•
Peers
Fellows
Inpatient Nurses
Unit Clerks
myevaluations.com
Thank You!