Session #E5a
Saturday, October 12, 2013
Anything Goes?
Developing a New Measure of Fidelity for Behavioral
Health Providers in Integrated Primary Care
Gregory P. Beehler, Ph.D., M.A.,
Clinical Research Psychologist, VA Center for Integrated Healthcare
Jennifer S. Funderburk, Ph.D.,
Clinical Research Psychologist, VA Center for Integrated Healthcare
Katherine M. Dollar, Ph.D.,
Technical Assistance Specialist, VA Office of Mental Health Operations
Collaborative Family Healthcare Association 15th Annual Conference
October 10-12, 2013
Broomfield, Colorado U.S.A.
Faculty Disclosure
We have not had any relevant financial relationships
during the past 12 months.
1. Define fidelity of implementation as applied to
behavioral health providers in Integrated Primary
2. Identify potential sources of low and high fidelity
among behavioral health provider behaviors in
primary care
3. Describe the psychometric properties and intended
use of a new measure of fidelity for behavioral
health provider behaviors in primary care
4. Describe VA behavioral health provider reactions to
a field test of a new measure of fidelity
Presentation Overview
1. Background: IPC and fidelity
2. Study 1: Developing a measure of protocol
3. Study 2: Psychometric validation of the PPAQ
4. QI Project: The PPAQ Minikit
5. Implications and Future Directions
• The role of Behavioral Health Providers (BHPs)
in Integrated Primary Care (IPC) is complex
– The competencies of BHPs are broad1,2
– BHPs may have limited background and training
related to integrated care
– BHPs face a number of practice barriers that can
lead to variability in how IPC is enacted locally3
• Despite the growing popularity of IPC, the
evidence base remains limited, but growing4-6
1. Pomerantz et al, 2009; 2. APA Workgroup, 2013; 3. Beehler & Wray, 2012; 4. Cigrang, et al., 2009; 5. Bryan, et al, 2009; 6. Ray-Sannerud,
et al, 2012.
Assessing Provider Behavior
• How well is IPC being implemented by BHPs? How do
we describe, measure, and monitor their behavior?
• Fidelity is the most commonly reported measure of
implementation outcomes1
• Fidelity refers to the degree to which an intervention is
delivered as intended
– Also referred to as treatment integrity or quality of
• It is a construct that appears in many evaluation
frameworks, such as RE-AIM2
1. Proctor, et al., 2011 ; 2. Glasgow, et al., 1999
Measuring Provider Fidelity
• Fidelity can be assessed at several levels
– molar (program-level)
– molecular (provider-level, or protocol adherence)
• Protocol adherence is the degree to which a
provider utilizes certain procedures and
engages in specific tasks and activities1
– Are patients receiving the intervention’s active
ingredients from the provider?
– The “bottom line” of implementation fidelity2
1. Perepletchikova & Kazdin, 2005; 2. Carroll et al, 2007.
Conceptual Model:
Fidelity of Implementation
Patient, Provider, Clinic,
and System Barriers and
Primary Care
Protocol Adherence
Patient Outcomes:
Distress, Coping,
Adherence, etc.
Carroll C, Patterson M, Wood S, Booth B, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation
Science. 2007; 2: 40.
Why Invest in Fidelity Assessment?
• Develop IPC measures:
– Currently, there are very few validated measures of
provider behaviors
• Advance IPC research:
– Linking provider behaviors to patient outcomes
• What formulation of IPC do we use?
• Training and Implementation:
– Can a protocol adherence measure assist BHPs in
understanding what specific behaviors constitute high
quality practices?
Study 1:
Developing a measure of protocol
Delphi Study Approach
• Goal: Build consensus on critical BHP
behaviors that should be included in a
measure of protocol adherence
• Design: Modified Delphi study
– Structured, iterative process of collecting and
summarizing opinions from content experts
– Often used to address problems of clinical practice
when there is incomplete knowledge
– Improves content validity for instrument
Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. (2013). Developing a measure of provider adherence to improve the
implementation of behavioral health services in primary care: A Delphi study. Implementation Science, 8, 19.
Delphi Study Approach
• A preliminary set of 56 items were developed by
the research team over six months
– Items represented critical provider behaviors related
to clinical services or collaboration
– Items based on a qualitative review of empirical and
theoretical works
• Each item was classified by the team based on
relevance for IPC:
– Essential (consistent with the IPC model; required for
good practice)
– Prohibited (inconsistent with the IPC model; should be
– Compatible (neither essential nor prohibited)
Delphi Study Participants
• Delphi panel sizes vary considerably
– We aimed for complete participation from 20 experts
• 33 IPC experts from VA, Department of Defense,
and academic health centers were invited to
– Purposefully sampled based on clinical,
administrative, or research experience
• 25 experts (76% response rate) participated in
three rounds of emailed surveys
Selected Participant Characteristics
n (%)
Educational background
23 (92)
1 (4)
1 (4)
Primary occupational setting
10 (40)
Department of Defense
7 (28)
Academic health center
8 (32)
Author/Co-author of peer reviewed paper*
IPC empirical research
11 (46)
Other IPC paper
13 (54)
Administrative lead for implementing IPC
*n=24 due to missing data.
17 (68)
Delphi Process Results
• Participants provided qualitative feedback on all items
and also suggested new items
• Participants were asked to classify each item as
essential, prohibited, or compatible
– Items that reached ≥80% consensus were considered
– 19 items met consensus in Round 1
– 26 additional items met consensus in Round 2
– 9 additional items met consensus in Round 3
• Final results at the end of Round 3:
– 38 essential, 10 prohibited, and 6 compatible behaviors
Delphi Process Example
PPAQ Item Examples
• The resulting 54-item instrument was named the
Primary Care Behavioral Health Provider Adherence
Questionnaire (PPAQ)
• Item 11 (essential)
My progress notes in the shared medical record include focused
recommendations for the Primary Care Provider and/or primary
care team
• Item 44 (prohibited)
I typically see patients for 10 or more appointments per episode
of care
• Item 31 (compatible)
I conduct follow-up appointments via telephone when
• A modified Delphi study was used to reach consensus on 54
critical provider behaviors related to the delivery of IPC
• 93% of original items developed by the research team were
included in the final instrument representing several IPC
Clinical scope and interventions
Practice and session management
Referral management and care continuity
Consultation, collaboration, and interprofessional
• The expert panel ensured content validity, but reliability
and construct validity required testing with frontline
Study 2:
Psychometric validation of the
Primary Care Behavioral Health Provider
Adherence Questionnaire (PPAQ)
PPAQ Study Objectives
Goal: To evaluate the psychometric properties of the PPAQ
using a sample of behavioral health providers (BHPs) working
in VA primary care settings
• BHPs working solely in the CCC (co-located, collaborative
care) role would have higher fidelity compared to those
working in either care management-only, or CCC and care
management combined roles
• BHPs working in CCC settings at VA Medical Centers would
evidence higher fidelity scores than BHPs working in CCC
settings at VA Community-Based Outpatient Clinics
PPAQ Study Objectives
Hypotheses (continued)
• BHPs with a longer history of providing CCC services would have
higher fidelity ratings due to mastery over CCC-specific roles
compared to those BHPs with a shorter history of CCC work
• BHPs who self-reported higher levels of knowledge/skills
related to CCC would show higher levels of fidelity
• BHPs who self-reported greater access to material and
resources to deliver CCC would show greater fidelity
• BHPs who endorse a CBT orientation over other forms of
psychotherapy would demonstrate higher fidelity
• Look at PPAQ scores with scores from the Level of Integration
Measure (LIM) (a self-report measure)
PPAQ Study Design
• Email invitation to online survey consisting of consent
information, background information, and questionnaires
– BHP Background Questionnaire
• Relevant professional background characteristics
• Knowledge and access to resources consistent with CCC
• 54-item, 5-point Likert-like scale of BHP protocol adherence to the
CCC model
• 3 subscales: PPAQ-E (38 essential items), PPAQ-C (6 compatible items),
PPAQ-P (10 prohibited items)
• 35-item, 5-point Likert-like scale reflecting level of behavioral and
physical health integration in primary care
• 6 subscales addressing aspects of integration
Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K.M. (2013). Psychometric assessment of the Primary Care Behavioral
Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine, 1, 13.
PPAQ Study Participants
• Potential participants identified by previous
participation in PC-MHI-related research or who
attended PC-MHI educational trainings from 2009-2011
• 580 BHPs who provided clinical services in primary care
for at least 25 % of their duties, had an active VA email
account, and with sufficient time to complete a brief
(i.e., <20 min) online survey were invited to participate
• 173 BHPs provided complete responses to the
survey, yielding a 30 % response rate (on par with
other electronic survey studies)
Selected Participant Characteristics
Educational background
Current PC-MHI role
Both CCC and care management
Care management only
n (%)
80 (46)
41 (24)
19 (11)
19 (11)
14 (8)
83 (48)
75 (43)
2 (1)
13 (8)
Participant Characteristics continued
• Majority of BHPs endorsed a CBT orientation
• Among those endorsing a CCC role:
– 26% felt they lacked resources to provide CCC
– 94% agreed that they had the knowledge and
skills to provide CCC
– Encouraging, but due to limited variability, this
variable was not included in subsequent analyses
PPAQ Results
• Reliability
– PPAQ-E (alpha=0.92) and PPAQ-P (alpha=0.70) had strong
internal consistency
• Known-groups validity
– Among BHPs in CCC-only or combined roles, those who had
sufficient resources to enact CCC had higher PPAQ-E scores
– Greater adherence to CCC essential behaviors was associated
with CBT orientation and greater perceived resources for CCC
• Convergent and discriminant validity
– Highly significant low to moderate correlations of PPAQ-E and
LIM total score (r=0.52, p<0.001) and subscales (ranging from
0.35 to 0.49, p<0.001) suggest good convergent validity
– Non-significant correlations between PPAQ-P and LIM total
score and subscales suggest good discriminant validity
– Lack of correlation between PPAQ-E and PPAQ-P (r=0.10,
p=.197) suggest good discriminant validity
• Relied on self-report for the nature of the professional
role (CCC, care manager, or both)
• Based sampling on previous participation in CCCrelated trainings or research, thus the participants may
report increased knowledge or familiarity with CCC
– May also have social desirability bias
• Validation in non-VA samples is needed
• Methodology did not allow investigation of multiple
forms of construct validity
• No well-accepted “gold standard” measures of CCC
adherence with which to compare
• The PPAQ has emerged as the first reliable selfreport instrument of BHP fidelity that has
demonstrated strong evidence of criterion
• Implications: beneficial for provider training (selfassessment), program monitoring, and clinical
• PPAQ has the potential to identify service delivery
gaps, but still important to maintain a stance in
which fidelity assessment is used to support
BHPs’ efforts to improve their capacity to deliver
evidence-based, patient-centered care
Quality Improvement Project:
The PPAQ Minikit
From Measure to Minikit
• The PPAQ is a psychometrically-sound self-report
measure that allows BHPs to assess their usual clinical
practices in terms of protocol adherence
– But what would make the PPAQ more useable for frontline
• Goal: Develop a fidelity-focused toolkit for CCC
providers based on the PPAQ
– Aim 1: Prepare the PPAQ Minikit components
– Aim 2: Pilot the PPAQ Minikit with CCC providers to
evaluate acceptability and utility of the PPAQ Minikit as a
diagnostic assessment that directs future implementation
Beehler, G. P., & Dollar, K.M. (2013). Employing an Evidence-Based Fidelity Tool to Identify Implementation Support Needs and
Preferences among Co-located, Collaborative Care Providers. MH QUERI QLP 55-020.
Aim 1: Develop PPAQ Minikit
• Work completed:
– Audio-enabled presentation providing background
on the PPAQ and instructions for using the Minikit
– PPAQ self-report form
– Scoring template and interpretation guide
PPAQ Self-Report Form and
Scoring Template and Interpretation Guide
PPAQ Interpretation Guide
Interpreting the PPAQ by Item
Addressing Practice Domains
Moving Forward:
A Process for Change
Complete Scoring
Template and
Interpretation Guide
Initial PPAQ selfadministration
1. Prioritize areas for change
Identify areas for improvement based on PPAQ
item analysis
Rank order areas by feasibility
Start small: Select one or two behaviors to address
2. Partner with advocates and peers
Reach out to PC-MHI Champion, colleague, or
Join local/regional peer support networks
Request consultation from CIH or PC-MHI PO
3. Access educational and clinical resources
Review PC-MHI PO resources
Review CIH practice tools and educational
Consider PC-MHI relevant handbooks and journals
4. Quality Improvement
Specify attainable practice change goals
Conduct P-D-S-A cycles to reach individual goals
Use a team-based approach for larger clinical goals
Aim 2: Piloting
• Methods
– Convenience sample of 10 VA BHPs
– Pilot the PPAQ Minikit components available via
– Complete a telephone interview to identify key
strengths and weaknesses related to acceptability,
quality, and likelihood of practice change
• Calculate estimates of acceptability
• Identify themes related to knowledge and practice
Preliminary Results
• Strengths:
– Overall usability and acceptability of the Minikit as a
diagnostic self-assessment was rated highly
– Minikit item content was perceived as very reflective
of the IPC model
– The tone of the Minikit was described as non-punitive
and balanced because it identified areas of strength as
well as weaknesses
– Participants noted that the Minikit would be highly
useful for BHPs new to the IPC/CCC role to clarify “do’s
and don’ts”
• Mature BHPs also found it to be a helpful reminder
to avoid drift
Preliminary Results
• Weaknesses:
– Limited familiarity with Excel was a barrier to usability
of Scoring Template and Interpretation Guide
– Current form of the Minikit may be seen as a one-time
assessment rather than a tool to direct on-going
– Improving adherence remains challenging in some
contexts due to clinic and system barriers (e.g., limited
time for QI projects or self-study or unwillingness of
primary care providers to fully collaborate)
Summary and Implications
• The PPAQ is the first reliable, validated measure
of BHP protocol adherence
• The PPAQ can be used in future effectiveness
research to link provider behaviors to patient
• The PPAQ may be used alone or with other
program development activities (e.g., facilitation
and coaching) to monitor provider behavior
• The PPAQ Minikit has shown early good potential
to be a provider-focused, self-assessment and
quality improvement package
Audience Q&A:
Disrupting Your Status Quo?
• Is your status quo “anything goes”?
How do you rate on these PPAQ sample items using the following
1=Never, 2=Rarely, 3=Sometimes, 4=Often, and 5=Always
1. During clinical encounters with patients, I see patients for 30
minutes or less.
26. I provide education to the primary care team on behavioral
health issues (e.g., presentations and handouts).
28. I provide family or couples therapy for 10 or more
appointments per episode of care.
How do you rate on these PPAQ sample items using the following
1=Never, 2=Rarely, 3=Sometimes, 4=Often, and 5=Always
32. During a patient appointment, I provide full length
empirically supported treatments, such as Prolonged Exposure
or Dialectical Behavior Therapy.
43. I meet with a patient for greater than 50 minutes to gather a
full psycho-social history and comprehensive psychiatric
48. I participate in primary care based clinical pathways for
common health conditions, such as chronic pain or comorbid
depression and cardiovascular disease.
• PPAQ Team
• Support
• Kyle Possemato, Ph.D.
• Christina Vair, Ph.D.
• Margaret Dundon, Ph.D.
• VA Center for Integrated
Healthcare Pilot Grant
• VA Mental Health Quality
Enhancement Research
Initiative (QLP 55-020)
Session Evaluation
Please complete and return the
evaluation form to the classroom monitor
before leaving this session.
Thank you!

PPAQ - Collaborative Family Healthcare Association