Session # B2c
October 17, 2014
The Use of Standardized Patients To Assess
Behavioral Health Consultant Core Competencies
Natalie Levkovich, CEO, Suzanne Daub, LCSW, Neftali Serrano, PsyD
Health Federation of Philadelphia
Collaborative Family Healthcare Association 16th Annual Conference
October 16-18, 2014
Washington, DC U.S.A.
Faculty Disclosure
Please include ONE of the following statements:
I/We currently have or have had the following relevant
financial relationships (in any amount) during the past
12 months:
– Neftali Serrano, Lead consultant, primarycareshrink.com
Learning Objectives
At the conclusion of this session, the participant will be able to:
Describe the novel methodology for assessing BHC core
competencies
Identify the BHC competencies that were assessed
List the application of findings for quality improvement
Define the benefits of an ongoing practice-based approach
to training and assessment for incumbent BHCs
Bibliography / Reference
Creating a Simulated Mental Health Ward: Lessons Learned. Rossetti J, Musker K, Smyth S, Byrne E, Maney C, Selig
K, Jones-Bendel T. J Psychosoc Nurs Ment Health Serv. 2014 Sep 12:1-7. doi: 10.3928/02793695-20140903-02.
Consumer and Relationship Factors Associated With Shared Decision Making in Mental Health ConsultationsMatthias
MS, Fukui S, Kukla M, Eliacin J, Bonfils KA, Firmin RL, Oles SK, Adams EL, Collins LA, Salyers MP. Psychiatr Serv.
2014 Sep 15. doi: 10.1176/appi.ps.201300563.
Twelve tips for asking and responding to difficult questions during a challenging clinical encounter., Soklaridis S, Hunter
JJ, Ravitz P., Med Teach. 2014 Sep;36(9):769-74. doi: 10.3109/0142159X.2014.916782. Epub 2014 Jul 14.
.
'Thinking on my feet': an improvisation course to enhance students' confidence and responsiveness in the medical
interview. Shochet R, King J, Levine R, Clever S, Wright S. Educ Prim Care. 2013 Feb;24(2):119-24.20.
Improving physician-patient communication through coaching of simulated encounters.
Ravitz P, Lancee WJ, Lawson A, Maunder R, Hunter JJ, Leszcz M, McNaughton N, Pain C. Acad Psychiatry. 2013 Mar
1;37(2):87-93. doi: 10.1176/appi.ap.11070138.
Learning Assessment
A learning assessment is required for CE credit.
A question and answer period will be conducted
at the end of this presentation.
Rationale
• Description of network: The Health Federation
of Philadelphia is a regional network of
community health centers
• Over 40 Behavioral Health Consultants across
15 organizations participate in a community of
practice
• Need for meaningful standardization of
competencies
History
• Partnerships developed between HFP,
Philadelphia College of Osteopathic
Medicine and the Thomas Scattergood
Foundation to enable the project
Simulation Protocol
• Develop rating tools and standardized patient
cases
• PCOM trained standardized patients
• Simulation occurred at PCOM lab over two days
• 21 BHCs, 3 Actors (1 male), 2 Raters
• Simulations were observed and recorded
List of Tools Used/ Created
Case
Mock Medical Record
BHC Rating Scale
Working Alliance Inventory (WAI) SP and Self
Rating Tool
Documentation Template
Post-Simulation BHC Feedback Survey
Simulation Tools
Results of the Simulation
● Poor interrater reliability on expert scale
noted (mean average difference in pass rate
per question, 25%)
● BHC disagreement with expert rater was
normally distributed with a mean
disagreement percentage of 38% +/-15
Distribution of Disagreement Percentage Between Expert Rater & BHC
Results of the Simulation
● Items that BHCs rated
themselves worst in (≈ 61+%
pass rate) included:
o aspects related to
communication with PCP
o functional understanding of
patient concern
o integration of medication
and medical comorbidities
● Items that expert raters marked
as poorest (≈27+% pass rate)
included:
o SOAP note skills,
communication with PCP,
o motivational interviewing
skills and
o collaborative planning with
patient
Results of the Simulation
● Congruence of BHC and patient actors’ perceptions of
the consult was high and generally positive
● BHCs rated themselves slightly lower in general
compared to patient ratings
Results of the Simulation
A: The patient and I agreed on the steps to be taken to improve his / her
situation.
B: The patient and I agreed on the steps to be taken to improve his / her
situation.
D: I have doubts about what we were trying to accomplish in the session.
I: I have doubts about what we were trying to accomplish in the session.
BHC Feedback
Strong agreement that:
• Experience was well organized
• Experience was useful for competency
development
• Patient feedback will help shape future practice
• Rater feedback targeted clinical skills for
improvement
Feedback For Standardized Patient Exercise
1.
2.
3.
4.
Improve interrater reliability through training of raters
Use more dispassionate raters when possible
Develop rating tool rubric
Simulation may be best for non-novice BHCs
Feedback For Network Training
1. Focus training on
a. motivational interviewing
b. integration of health behavior change/ medical
comorbidities
c. formalize SOAP note training
2. Improve integration and use of standardized
assessment tools