Center for Health Care &
Policy Research
CREATING AND DISSEMINATING NEW SCIENTIFIC KNOWLEDGE TO
HELP DECISION-MAKERS DEVELOP COST EFFECTIVE SERVICES AND
PROGRAMS THAT IMPROVE PEOPLE'S HEALTH.
Outline
Problems that CHCPR researchers focus on (using science to improve practice and policy)
Example of one large project (AF4Q) as an illustration of the types of work in CHCPR
A bit about CHCPR and where we are headed
Potential opportunities for collaboration with the Prevention Center and its affiliates
Discussion
Problems to Address
High Costs
◦ Over $8,000 annually per capita
◦ $16,351/family or $5,429/individual policy
Unequal Access
◦ Uninsured and Underinsured
◦ Disparities
Asymmetric Information
◦ Available information on price/costs, quality/safety
◦ Slow to incorporate the advantages of IT
Distorted Incentives
◦ Supply Side – fee based reimbursement (over/underutilization)
◦ Demand Side – third party insurance (overutilization)
Problems to Address (cont.)
Short-Term/Acute Care Focused
◦ Specialty vs. Primary/Preventive Care
Fragmented Delivery Systems
◦ No accountable entity for patients
◦ Efficiency not rewarded (sometimes penalized)
Special Interests
◦ Big business at 17% of GDP
Heavily Regulated While Unabashedly Capitalistic
◦ Seeking innovation while maintaining entitlements
5
Looking Back: What We Could Have Saved if We Had Matched the
Next Highest Country (Switzerland)
Note: Per capita spending amounts adjusted for differences in cost of living, total U.S. savings adjusted for inflation.
Source: D. Squires, The Road Not Taken: The Cost of 30 Years of Unsustainable Health Spending Growth in the United States,
(New York: The Commonwealth Fund Blog, March 2013).
http://forces4quality.org/about-us
AF4Q Communities
AF4Q Programmatic Areas
Working Locally to Transform the System
In each community, the alliances are using funding and TA to move quality forward at the local
level, through activities in five important areas of focus:
Performance measurement and public reporting: using common standards to measure the quality of care
doctors and hospitals deliver to patients, and making that information available to the public.
 Payment Reform
Consumer engagement: encouraging patients to be active and effective managers of their health care.
Quality improvement: implementing techniques and protocols that doctors, nurses and staff in hospitals
and clinics can follow to raise the level of care they deliver to patients.
 Reducing Disparities
AF4Q Evaluation Goals
Provide inferences about the effect of the AF4Q program as a whole
and specific cases
Identify key lessons about AF4Q program design, implementation,
and effectiveness
Expand the knowledge base around community level health
improvement strategies and the collaborative/multi-stakeholder
approach
Evaluation Approach
Start with logic model or “theory of change”
Identify specific research questions
Build data collection and analysis plans based on the logic model
◦ quantitative, qualitative and mixed methods
◦ formative and summative components
Account for time – significant change evolves rather than happens instantly
Apply a standard of “proof” or “evidence” consistent with what the program evaluation
field demands
Disseminate formative and summative findings to target audiences
AF4Q Logic Model
5 Overarching Research Questions
1.
Are AF4Q communities able to develop and sustain a community-wide infrastructure to achieve
high quality and affordable health care?
2.
Does transparency about health care quality and costs increase in AF4Q communities relative to
non-AF4Q communities?
3.
Does the level of consumer engagement increase more in AF4Q communities relative to nonAF4Q communities?
4.
Does the AF4Q program yield improved levels of health care quality in participating communities
relative to non-AF4Q communities?
5.
Do health care disparities decrease in AF4Q communities relative to non-AF4Q communities and
do the local health care systems promote and support a culture of equity?
Common QI Evaluation Questions
Did ‘it’ (the program or policy) work?
Did the program happen?
◦ If so, what was the effect size?
◦ What was the dose?
◦ What mechanism(s) led to the effect?
◦ Did it vary across sites/markets and if so why?
◦ If not, why didn’t it work?
In which context and under what
conditions did it work?
Should any changes be made to the
program or to the initial theory of change?
Can it be spread? Under what conditions?
Data Sources for the AF4Q Evaluation
Tracking
◦ Systematic tracking of Alliance activities and
relevant health information and activities in the
Alliances’ communities
◦ e.g., availability of public reports, CE activities
and state policy proposals
Key Informant Interviews
◦ In person site visit interviews with multiple
stakeholders
◦ Phone interviews
◦ Over 1,000 to date
Surveys
◦ Consumer survey
◦ Physician survey
◦ Alliance Survey
Secondary Data
◦ Dartmouth Atlas
◦ BRFSS
◦ H-CAHPS
◦ Area Resource File
◦ Others
Alliance Tracking
Alliance Tracking
Alliance Survey
Alliance Survey
Secondary Data
Sources
Key Informant
Interviews
Key Informant
Interviews
Physician Survey
Consumer Survey
Alliance Tracking
Alliance Survey
Key Informant
Interviews
Key Informant
Interviews
Alliance Tracking
Key Informant
Interviews
Secondary Data
Sources
Consumer Survey
Types of Analyses/Evidence
Descriptive – describe and measure the activities (interventions) of the “alliances”
◦ Single case descriptions
◦ Comparative case descriptions
Conceptualization – Develop models for new concepts and measurement of these concepts
◦ E.g., “QI Infrastructure”, “Alignment”, and “Consumer Engagement”
Document trends on key outcomes and associate changes in outcomes with interventions
o For program as a whole and communities individually
Assess causality – test whether changes in outcomes happened due to AF4Q
◦ Involves some sort of “control” strategy
◦ Not simple
AF4Q Evaluation Multiphase Design
Also referred to as Methodological Triangulated Design
Sub designs (e.g., exploratory sequential) with independent methodological
integrity to study various programmatic areas
◦
◦
◦
◦
◦
◦
Performance Measurement and Public Reporting
Consumer Engagement
Quality Improvement
Disparities Reduction
Governance
Cross-cutting Topics (alignment, payment reform, etc.)
Complimentary to each other to attain goals of overall research design
Stages of Evaluation
Summative Evaluation
(2013-2016)
Formative/ Summative
Transition
Formative Evaluation
(2007-2009)
Evaluation
Development
(2006-2007)
• Regional Market Pilot
Evaluation Conducted
• AF4Q Evaluation Plan
Developed
• AF4Q Evaluation Proposal
Submitted and Funded
• Initial Key Informant
Interview, Alliance Survey,
and Consumer Survey
Instruments Developed
• AF4Q Logic Model
Developed
•PR Tracking Summaries
Disseminated
•Alliance Survey Baseline and
Round II Reports
Disseminated to
Communities
•Consumer Survey Round I
Reports Disseminated to
Communities
•Community Snapshots
Disseminated to
Communities
•Physician Survey Baseline
Reports Disseminated to
Communities
•1 Paper Published
(2010-2013)
• Outcomes Measures
Defined
• Midterm Report Provided
to RWJF
• 30 Papers Published,
Including AJMC Special
Issue
•AF4Q Impact Report Provided
to RWJF (2015)
•Presentation of Findings and
Dissemination Strategy
Planning Conducted with
RWJF, NPO and AF4Q
Communications
•Presentation of Findings
Conducted at AF4Q National
Meeting (2015)
•Other Presentations
Conducted Upon Invitation
• 35 Presentations Given
•Comprehensive AF4Q Journal
Issue Published (2016)
• 4 Research Summaries
Published
•AF4Q Research and Evaluation
Conference Held
• Alliance Survey Round III
Reports Disseminated to
Communities
•Consumer Survey Round III
Reports Disseminated
• Consumer Survey Round
II Reports Disseminated
to Communities
•Physician Survey Rounds II
and III Reports Disseminated
•Alliance Survey Rounds IV and
V Reports Disseminated
•30 Presentations Given
•Approximately 8 Research
Summaries Published
•5 Research Summaries
Published
•Approximately 30 Papers
Published
Common Evaluation Challenges
Ambition level of program sponsor (“stretch
goals”)
External validity
Measurement of key program concepts
Lack of cooperation from program
stakeholders
Few degrees of freedom
Program sponsor does not ‘stick with it’
Spillover and temporal trend (e.g., Hitech and
ACA)
Degree to which program “must be successful”
and power of the local anecdote or the PR firm
Variation in intervention strength (fidelity)
and attribution
Adequacy of resources relative to program size
(scale)
Summarizing across multiple measurements
Being a team player and academic home
valuation
Appropriate time horizon to observe expected
impact
How Can CHCPR Help?
Getting Research
Funded
Getting Research Done
Promoting Learning to
Strengthen Research
How Can CHCPR Help?
Getting Research
Funded
Identifying Funding Opportunities
 Conduct Individualized Funding Searches
 Connect Researchers
Getting Research Done
Promoting Learning to
Strengthen Research
Developing & Submitting Proposals

Organize Proposal Timelines

Create Budgets

Prepare Proposal Content
How Can CHCPR Help?
Getting Research
Funded
Getting Research Done
Managing Finances
 Strategic Budgeting
 Financial Compliance & Fiscal Responsibility
 Day-to-Day Purchasing & Accounting
 Subcontract/Consultant Administration
Promoting Learning to
Strengthen Research
 Payroll Management & Human Resources
How Can CHCPR Help?
Getting Research
Funded
Doing Research
 Staffing a Project
Getting Research Done
 Managing Research Projects
 IRB Compliance & Data Security
 Collecting & Analyzing Data
Promoting Learning to
Strengthen Research
 Using Technology & Facilities
Qualitative Data Services
 Develop key informant interview protocols
 Schedule and conduct telephone and in-person key informant interviews; our team has
scheduled and conducted more than 1,000 key informant interviews
 Track and monitor project activities, collect documentation data and maintain extensive
information databases to inform project research and evaluation
 Manage digital voice audio files and transcripts, including coordination with transcription
firms
 Develop codebooks and code data using popular qualitative software packages
 Analysis of qualitative data using various techniques
Quantitative Data Services
 Experience with web, mail, and telephone surveys
 Design and develop survey instruments
 Manage coordination with external survey vendors
 Identify and obtain publicly available and restricted use secondary data
sources
 Analyze survey data using SPSS, SAS, and Stata — including linear and nonlinear regressions, summary statistics, and creating graphs for visual analysis
How Can CHCPR Help?
Getting Research
Funded
Getting Research Done
Generating Visibility & Creating Impact
 Identifying & Understanding Research
Audiences
 Communicating & Promoting Findings
Promoting Learning to
Strengthen Research
 Gathering User Feedback
How Can CHCPR Help?
Getting Research
Funded
Keeping Current in Health Services Research
 Health Services Research Colloquium
 Workshops & Conferences
Getting Research Done
 Research Training for Students
Developing Professional Networks
Promoting Learning to
Strengthen Research

Access to Data Sets & Professional Services

Catalyzing Research Collaborations
Funding Opportunities
Let’s Partner!
DE N NIS S C A N LON – DXS 6 2@PSU.EDU
CHE L S EA LUN D ERS – CK L 1 6 @ P SU.EDU
B R I G I T T L E I T ZEL L – B J H 1 5 6 @PSU.ED U