Evaluation of Patient Centered Medical
Home Practice Transformation Initiatives
Benjamin F. Crabtree
& Workgroup Collaborators






Sabrina M. Chase
Jeanette R. Goyzueta
Carlos Roberto Jaén
Rebecca A. Malouin
William L. Miller
Christopher G. Wise





Paul A. Nutting
Susan M. C. Payne
Michael T. Quinn
Gordon D. Schiff
Laura A. Schmidt
Overview
 Methods
 Logic
Model
 Recommendations
 Evaluation Design Consideration
Workgroup Methods




Initial series of May/June 2009 conference
calls to develop preliminary logic model
Presentation and discussion of logic model
at Commonwealth Evaluators Meeting at the
2009 AcademyHealth Conference
Iterative process of editing/revising written
drafts of assumptions and recommendations
via email and conference calls during
summer/fall 2009
Preparation and submission of manuscript
for peer-review in March, 2010
Logic Model
Assumptions
Inputs
Activities
Outputs
Outcomes
Impact
Practice
Capability
Assessments
Multimethod
Assessment
Process (MAP)
Good
Teamwork
Improved
Clinical
Quality
System/External
Environment
Assessment
Environmental
Scan
Change
Management
Skills
Advanced
PCMH
Practices are Complex
Adaptive Systems
Change needs to be both
“bottom up” and “top down”
Transformation to PCMH
requires communication
& relationships
Change models are required
t guide
to
id ttransformation
f
ti
Most practices lack effective
change models & knowledge
of process improvement
Mostt practices
M
ti
lack
l k sufficient
ffi i t
“adaptive reserve” to sustain
transformational change
Patient
Centered
Medical
Home
Attributes
Stakeholder
Motivation &
Leadership
Leadership
Training
Facilitative
Leadership
Improved
Patient
Experience
Teambuilding &
Collaboration
Practice
Improvement
Meetings
Action/Reflection
Cycles
Improved
Physician/Staff
Satisfaction
Practice
Change Model
Improvement
Process
Sensemaking
Enhanced
Efficiency
Most practices require some
type of external facilitation
There are both internal and
external barriers to change
7 Fundamental Evaluation
Requirements
1.
2.
Evaluations should critically examine
tthe
e PCMH
C
models
ode s being
be g implemented
p e e ted
and identify areas that require
modification over time.
Evaluations need ongoing embedded
data collection over an extended
period of time that captures how and
why implementation strategies
change.
change
7 Fundamental Evaluation
Requirements (Continued)
3.
4.
Evaluations must not only to
capture details of how individual
features are implemented, but
also how they
y interact with and
impact each other over time.
Evaluations need to understand
how and why roles evolve, why
they
y do not, and what future
training is required.
7 Fundamental Evaluation
Requirements (Continued)
5.
Evaluations need to identify not
only the effectiveness of
individual components (e.g.
patient registries,
registries physician
dashboards, patient portals, etc.),
but also how they are actually
used and how they integrate with
other practice components.
components
7 Fundamental Evaluation
Requirements (Continued)
6.
7.
Evaluations must capture not only
changes within individual practices,
but ways practices interface with other
entities such as specialists, hospitals,
and referral services.
services
Evaluations must contribute to better
understanding
u
de sta d g what
at resources
esou ces a
are
e
required to change the delivery of care
and which default options work best
when resources limit the
implementation process.
Evaluation Design
g & Measurement
Considerations

Evaluation designs require mixedmethods (qualitative & quantitative)
data from multiple levels of the
organization and its environment.





Overall health care environment
“Medical neighborhoods”
g
Practice or clinic
Clinicians and practice staff
Patients
Quantitative Measures


Other Commonwealth Evaluator’s
Collaborative workgroups
Stange KC, Nutting PA, Miller WL,
J
Jaen
CR,
CR Crabtree
C bt
BF,
BF Flocke
Fl k S,
S Gill
JM. Defining and measuring the
P ti t C t
Patient-Centered
d Medical
M di l Home.
H
J
Gen Intern Med, 2010; 25(6):601612.
Qualitative/process data





Describe the richness of the context, initial
conditions (both practice & medical
neighborhood)
Capture intervention and model modifications
to understand what worked, why and how, as
well as unanticipated consequences
Describe baseline and changes in “adaptive
g leadership,
p, communication,,
reserve” ((e.g.
relationship, etc.)
Describe/understand interdependencies of
practice features and how they impact each
other
Describe roles of practice participants and
h
how
these
h
change
h
(or
( need
d to change)
h
)
Emergent, MultiMulti-Method Design
Evaluation Design Strategies

SQUIRE Guidelines Outcome Reporting






Explain actual course of the intervention
Document degree of success
Describe how/why initial plan evolved
Document changes in processes of care
and patient outcomes
A basic
b i evaluation
l
i
strategy
A more comprehensive evaluation
strategy (but probably only realistic
for well funded comprehensive
projects)
j t )
A Basic Evaluation Design





Include a combination of brief
observations and targeted interviews
at different points in time
Site
S
te visits
s ts to a purposeful
pu pose u sample
sa p e of
o
practices
Online or written diaries
Follow-up observation and interviews
Quantitative measures
A Comprehensive Evaluation Design
Scientific Steering Committee
Project Management Team
Local Experts
Quantitative Outcomes
Team
Instrument Design
Data Collection
Data Management
Data Analysis
National Advisory Committee
External Consultants
Practice Redesign Team
Change Management Facilitators
Embedded Process Evaluation
Q lit ti Outcomes
Qualitative
O t
Team
T
Qualitative Observers
Interviewers
Data Management
y
Data Analysis
Laying the Foundation
for an Evolving PCMH
Bottom Line: PCMH Models
must continue to evolve