Dr. Karen Linkins, IBHP, Tides Center
Dr. Benjamin Miller, University of Colorado
Dr. Lynda Frost, Hogg Foundation for Mental Health
Dr. Becky Hayes Boober, Maine Health Access Foundation





1) Become familiar with strategies to evaluate
a complex health initiative;
2) Explore strategies for advocating with
policy makers;
3) Understand how to use data related to
quality health care interventions to create
compelling messages;
4) Gain insights on policy development and
leveraging; and
5) Share lessons learned and practical tools.
Performance Accountability Measures




How much did we do?
How well did we do it?
Is anyone better off?
Friedman, M. (2005). Trying hard is not good enough: How to produce
measurable improvements for customers and communities. FPSI Publishing.
How much did we do?
# Customers served
(by customer
characteristic)
How well did we do it?
% Common Measures
% Activity-specific
Measures
Is Anyone Better Off?
# Skills/Knowledge
#Attitude/Opinion
# Behavior
# Circumstance
#Improved Health
Outcomes
Is Anyone Better Off?
% Skills/Knowledge
% Attitude/Opinion
% Behavior
% Circumstance
% Improved Health
Outcomes







Who are our “customers”?
How can we measure if our “customers” are
better off?
How can we measure if we are delivering services
well?
How are we doing on the most important of
these measures?
Who are the partners who have a role to play in
doing better?
What works to do better, including no-cost and
low-cost ideas?
What do we propose to do?
 Did
we treat you well?
 Did we help you with your
problem?
 Performance
Accountability
Questions
 Population Accountability
Questions
What are the stories that
can influence policy?
Benjamin F. Miller, PsyD
Director of the Office of Integrated Healthcare Research and Policy
Department of Family Medicine
University of Colorado Denver School of Medicine





Sometimes in the face of innovation we
lose sight of our ultimate goal – to
change healthcare.
We focus on the problems rather than
recognize what is working.
We focus on meeting immediate needs
(e.g. financial) rather than plan for long
term success.
We slip into “protective mode” and
forget why we started the innovation to
begin with.
We stop seeing the other innovators
around us and focus on ourselves
rather than the larger community or
larger field.
But first
Brilliance
Brilliance
Brilliance
Brilliance
Brilliance






What we do (models)
What data we collect (clinical)
What we call ourselves (integrated)
What we need for sustainability (money)
Who we talk to (ourselves)
What we want (change)

Measuring integrated mental health (what is
that exactly?)
◦ There is no gold standard “tool”
◦ Consistency across sites (e.g., documenting mental
health diagnosis)



The evidence is lacking and the field is in
need of knowledge around the “elements”
HUGE scope
Financial sustainability (or the business case)
Mental Health Presentation
Medical issues
with
psychosocial
barriers to care
Psychosocial
Support Services
Medical issues
requiring
behavioral or
psychological
intervention
Behavior Change
Education &
Evidence-Based
Treatments
Mental Health
and Substance
Use
Presentations
Mental and
Physical Health
Multimorbidity
Severe Mental
Illness and/or
Substance
Abuse
Mental health
treatment plan
Coordination
of mental and
physical health
treatment
plans
Full
coordination
with specialty
care
Example Targeted Service Response




Two pots of money
Workarounds are often viewed as the solution
We don’t know what we don’t know (but we
think we know what we don’t know)
Turf wars and bad feelings
What we need to consolidate (or integrate)
Clinical data
Language
Financial data
What we measure
How we track and measure
what we do
Better community connections
and state to state connections
(and collaborations)
Shared and consistent
evaluation plans for integration
projects
Case study
Miller, B. F., B. Teevan, et al. (2011). "The importance of time in treating mental
health in primary care." Families, systems & health : The journal of collaborative
family healthcare 29(2): 144-145.






Time spent with patient
Time spent with other providers
Assigning monetary amounts to time (and or
patient volume)
Assessing changes in time and volume
Assessing value and outcomes
Learning about what patients use more time
and benefit from integrated initiatives
Case study
12%
Screening
Depression Diagnosis and
Percent of Visits with
14%
Depression
10%
Diagnosis
8%
6%
Depression
4%
Screening
2%
0%
FP
IM
Primary Care Specialty
Phillps, R. L., B. F. Miller, et al. (2011). "Better Integration of Mental Health Care
Improves Depression Screening and Treatment in Primary Care." American
Family Physician 84 (9): 980.





Number of patients identified
Number of patients treated
Number of patients who improve from
treatment
Comparing rates of identification to rates of
diagnosis (accuracy)
Using screening tools repeatedly for
treatment tracking
In summary






Be heard
Know what policy solutions can help lead to
sustainability (including financial)
Begin to collect some of the same data
Make sure our data are put into the medical
record in such a way it can be extracted
Have an entity that can pull it all together
Be compelling, be accurate, be timely
Thank you
Karen W. Linkins, PhD
Project Director
Integrated Behavioral Health Project
Tides Center
“Change is disturbing when it is done to us, but exhilarating
when it is done by us” (Elizabeth Moss Kanter, Professor,
Harvard Business School)
Many
different definitions of systems change
exist, but they share common elements: policies
and practices, resources, relationships, power and
decision-making, values, attitudes, skills,
governance, and supportive policies and reforms.
Systems change is dynamic, developmental, nonlinear, and complex.
The target of change is the system, not the
individual.

System change is defined as: changes in
organizational culture, policies and
procedures within and across organizations
that enhance or streamline access, and
reduce or eliminate barriers to needed
services by target populations.
Changes that endure beyond the funded
project that lead to any or all of the
following:

◦
◦
◦
◦
Increased Access
Improved Quality
Enhanced Efficiency
Increased Consumer Empowerment
Systems Change
Example
Increased Access
Changes in clinic operational policies (e.g., electronic
open scheduling and wait time monitoring, expanding
specialty staffing (telepsychiatry))
Improved Quality
Improve provider capacity to meet patient needs by
learning new skills and knowledge through distance
learning
Enhanced Efficiency
Data sharing across PC and BH providers to increase
identification and care coordination
Increased Consumer
Empowerment
Access to personal health record; use of technology to
facilitate client support groups



Stakeholder interests
Initiative goals, including desired outcomes
and impacts
How findings will be used, e.g.:
◦
◦
◦
◦

Educate policy makers
Disseminate best practices
Change local systems and policies
Support sustainability plans and garner new funding
sources
Available resources for the evaluation





Providers: Individual patient outcomes, panel
management
Clinics/Clinic Systems: Population health management,
administrative metrics (e.g., cycle times, provider
productivity, patient and provider satisfaction), billing,
culture change
Policy Makers: Cost and other administrative metrics
Community: Prevention, community health and
wellness, healthy behaviors, consumer engagement
Foundations: Alignment with strategic priorities, return
on investment, grantee accountability






Step
Step
Step
Step
Step
Step
1:
2:
3:
4:
5:
6:
Engage stakeholders
Describe the program
Focus the evaluation design
Gather credible evidence
Justify conclusions
Ensure use and share lessons learned





What will be evaluated? (program, context)
What aspects of the program will be considered in
assessing program performance?
What standards (i.e., type or level of performance)
must be reached for the program to be considered
successful?
What evidence will be used to indicate how the
program has performed?
How will the lessons learned be used to improve
public health effectiveness?



Design types: experimental, quasiexperimental, and observational designs.
No design is better or best in all
circumstances.
Design and methods should be matched to
the interests of targeted stakeholders (e.g.,
foundation, grantees, policymakers).



Design drives what counts as evidence, how
data are gathered, what claims can be made,
who needs to be involved, and what data
management systems are needed.
Mixed method designs are most effective
because each method has biases and
limitations.
During the course of an evaluation, methods
might need to be revised or modified.



Complex initiatives require significant
investments of time, resources and energy
to create common ground for change.
Programs often become so focused on
immediate implementation issues (client
“fixes”), the long-term vision for systems
change becomes lost or deferred.
Balancing the funder’s need for
accountability/rigor in reporting with
developing and maintaining authentic
relationships with grantees.



Data collection must be relevant.
Data should not be collected unless they are
shared and fed back to those responsible for
collection.
Evaluation should be clearly connected to
longer term outcomes. Failure to do so limits
buy-in, understanding, and a greater sense of
accountabilitytothe process.
Initiative Goals: Create a more responsive and
integrated system of care to increase access and
reduce costs for individuals with co-morbid
conditions (MH & chronic conditions)

Patient focused

System Focused
◦ Address patients’ needs, improve health outcomes
◦ Reduce reliance on ED resources for care that is more
effectively provided in less costly, community-based
settings
◦ Reduce ED volume and diversion time, and avoidable
inpatient use
◦ Encourage financing and policies that promote coordinated,
cross system, multidisciplinary care and integration of
services
Foundations
Oversight
Group
• Project Officers/Program Staff
• Policy Staff
• Evaluation Staff
Program Office
Evaluation Team
Grantees & Collaboratives
• Community-Based Organizations
• Hospitals
• Public Health, Housing/Homeless Programs, Mental
Health, Substance Abuse, MediCal, Criminal Justice


Participatory approach
Three evaluation phases of the evaluation
◦ Planning
◦ Implementation Process
◦ Outcomes and Promising Practices (“What Works”)

Multi-level, pre-post design
Frequent Users Initiative
Interventions
Intermediate
Outcomes/Changes
Long Range
Impacts
Enrolled TP Clients
• Outcomes
Planning Grants
Other Activities
• Meetings/Convening
s
• Other activities
• Service utilization
• Costs
Organizations
• Policies and
practices
• Data systems
• MOUs
Implementation Grants
• Changes in services
• Structure
County System
Service Delivery Change
• Client-based: Compare
enrolled clients & TP at
beginning and end of grant
period (utilization and cost)
• System-based: MIS analysis
of changes in the patterns of
service utilization and costs
system wide
(e.g., Intensive Case
Management)
• Intensity
• Data systems
• Financing
• Collaborations
Broader FUI Initiative
• New services
• Restructuring
Broad Systems
Change
• County
• State
• Policy papers
• Other activities
State Level
• Laws and
regulations
• Budget and
TP = Target Population
Measures

Cost and utilization (ED, inpatient and other systems as
available)
Clinical measures of health and functioning
Stability (e.g., income and insurance enrollment)
Service intensity (frequency and duration)

Strength of partnerships and collaborations




Policy and systems change (evidence of improved
coordination, streamlined access, permanent policy changes
to address/eliminate barriers)



Participatory orientation
◦ Balancing research rigor with “what’s reasonable
and feasible” – selecting outcome measures and
data collection strategies that matched capacity
and didn’t over burden staff
◦ Developing and maintaining meaningful
stakeholder participation (on-going
communication)
◦ Establishing and maintaining trust of programs to
ensure buy-in and data integrity
Defining/operationalizing multi-level outcomes
Ensuring evaluation findings aligned with and
relevant to information needs of various
stakeholders – at the “right time”




Client centered interventions: challenge of
programs/ models balancing individual client
“fixes” vs. permanent programmatic and systems
change
Data accuracy and consistency
Data availability and linkage capability
Mis-match of Foundation and Grantee Goals -Foundations wanted systems and policy change,
but funded local interventions


Findings were compelling and rigorous
enough to use for policy development
(Medicaid Waiver and other legislation).
The combination of quantitative and cost
data, as well as qualitative process and
outcome data created a strong and policy
relevant story of sustainable systems change.
Lynda Frost
Director of Planning and Programs
Hogg Foundation for Mental Health




20 years of research on collaborative care
model framed grant program on integrated
healthcare
Large conference highlighted research and
grantees’ work
Grantees engaged in advocacy around
reimbursement, other issues
Evaluation of grant program gathered statespecific outcome data and identified barriers
to implementation




Foundation convened key stakeholders to
identify barriers to implementation
One stakeholder lobbied for creation of
“Integration of Health and Behavioral Health
Workgroup”
Legislation mandated broad group of
appointed workgroup members
Resulting report described “best practices”
and recommended next steps



Foundation signed agreement with DHHS
Office of Minority Health to examine
integrated healthcare as a means of
eliminating health disparities in racial and
ethnic minority populations and persons with
limited English proficiency
Developed consensus report drawing on
practice-based evidence
Held large conference to share results; OMH
will release report with other national reports
Becky Hayes Boober
Program Officer
Maine Health Access Foundation

Maine Health Access Foundation—2 stories
◦ Middle of the night sentence embedded in budget
shifting hospital-based outpatient BH care from
Section 45 to Section 65.

“Medical Care - Payments to Providers 0147
Initiative: Reduces funding from reducing
reimbursement for outpatient substance
abuse and mental health services to
MaineCare Section 65 rates effective July 1,
2012.”





Know what is happening (rule making draft)
Take action (Work with DHHS to slow process)
Explore alternatives
Partner (Maine Hospital Association and
legislators)
Monitor
 IFS
Committee Invitation
◦ Budget shortages
◦ Messaging is important
(Endowment is 1/10 of 1% of
what is spent annually in Maine
on health care costs)





Build relationships, partnerships.
Be proactive.
Tell a compelling story.
◦ Human element (sans drama)
◦ Data
◦ Cost effectiveness
◦ Resulting outcomes
Embed into other key endeavors.
Identify key leverage points (employers)
540 Maine Primary Care Practices
82 NCQA PCMH
Recognized
Practices
Payers:
•Medicare
•Medicaid
• Commercials
(Anthem,
Aetna, HPHC)
26 Maine
PCMH Pilot
Practices
20 Pilot
Phase 2
Practices
100 MaineCare
Health Home
Practices
14 FQHCs
CMS APC
Demo
Payer:
Medicaid
Payer:
Medicare
540 Maine Primary Care Practices;
53 Community Behavioral Health
Agencies; 30 SA Agencies
ACOs: Pioneer and
Employer-Based
Beacon
82 NCQA PCMH
Recognized
Practices
BH HIT Support
26 Maine
and grant
PCMH Pilot
Practices +
20 new
MeHAF IC
grants, TA
~100 MaineCare
HH Practices??
Payment
reform
grants;
ACOs
Community
Care Teams
SAMHSA
Health
Home
14 FQHCs
CMS APC
Demo
(Medicare)
FQHC
expansion
ACA
DHHS Valuebased
contracting
AHRQ Academy
Section
1703
The Kid’s good.
The New Yorker. March 21, 2011
What are your
experiences?

In your small group, select a policy change
you would like to see happen. Develop a
messaging plan.




What compelling human interest stories will
build the case?
What data do you have that will help build a
compelling story? What data do you still
need? How will you get it? Present it?
How will you involve patients/families?
Who are potential partners (current and
needed)?


Share 1 key idea about messaging.
Share 1 key strategy for influencing policy,
using data/stories.




Benjamin.miller@ucdenver.edu
@miller7
occupyhealthcare.net
Lynda.frost@austin.utexas.edu
karen@desertvistaconsulting.com
bhboober@mehaf.org