Maximizing the Impact of Quality
Measurement Research on Policy
and Practice
Patrick S. Romano, MD MPH
UC Davis Center for Healthcare Policy and Research
Center for Studying Health System Change
AcademyHealth Quality Interest Group
May 27, 2010
Role of AcademyHealth
 Includes researchers, policymakers,
practitioners and others
 Mission is to bring together these multiple
disciplines to foster development of a
climate of awareness of health care quality
issues and to integrate that awareness into
a more comprehensive understanding of
pathways for effective measurement and
policy development.
From research to policy
 Researchers develop quality measures
 Researchers extend, test, and validate
quality measures
 Researchers apply quality measures to
understand differences and changes in
quality over time, across providers, across
markets, etc.
“Best practices” in development
 Identify the goal(s) of measure – why this topic?
- Align with an existing conceptual framework (e.g., IOM domains)
and NPP or other priorities, as appropriate
- Build on evidence (process affects relevant outcomes, outcome is
affected by processes)
- Identify, expose, understand, reduce variation in performance






Start from published literature and experience
Develop precisely specified definition
Test definition with actual data
Assess face/consensual (content) validity
Revise and retest definition with actual data
Assess other performance characteristics
- Reliability
- Criterion, predictive, and construct validity
- Usability, actionability, susceptibility to undesired effects
Example: AHRQ Quality Indicators
INITIAL
EMPRICAL ANALYSES
AND DEFINITION
LITERATURE REVIEW
USER DATA
PANEL EVALUATION
FURTHER
EMPIRICAL ANALYSES
REFINED DEF.
FURTHER REVIEW?
FINAL DEFINITION
So you think you have a quality measure…
 Does this measure have a life beyond your
own research?
 If so, what are the potential future
applications of your measure?
-
Other research studies
Policy and program evaluations
Quality monitoring and improvement efforts
Public reporting programs
Value-based purchasing programs
Maintenance, maintenance
It’s not so easy
 Measures need to be continuously maintained
and updated
- Annual ICD-9-CM and CPT updates
- ICD-10-CM starting October 2013
 Measures may need to be respecified for use in
different settings or with different data sets (e.g.,
Medicare, VA, all-payer)
 Measures need to be tested for reliability and/or
validity in these different settings and different
data sets (or caveat emptor)
 Consider measure performance at the
patient/episode level versus the provider level
General alternatives
 Maintain yourself (or through your own
organization or employer)
 Partner with (or lobby for adoption by) a
recognized steward
- Health care and provider organizations
- Public agencies
 Copyright and license
 Copyright and sell (transfer ownership)
Implications of selling
 You no longer have any responsibility for
updating, maintaining, or otherwise supporting
the measure (and you get some $ )…
BUT
 You no longer have the ability to modify the
measure based on new research findings or
feedback from researchers and users in the field
(unless established by contract)
 You have no control over how the measure is
packaged and marketed by the vendor
 Licensing fees are unpredictable, likely to drop
Finding a partner
 University or academic research institute
 Provider organizations (especially AMA
Performance Consortium for Performance
Improvement, ANA, specialty physicians)
 Accrediting organizations (NCQA, TJC)
 Health/disease advocacy organizations
 Public agencies (local, state, AHRQ, CMS,
VA, etc.)
Leading stewards of NQF-endorsed measures
(web search 5/26/2010)
Organization
Number
Center for Medicare & Medicaid Services
National Committee for Quality Assurance
American Medical Association-PCPI
AHRQ
The Joint Commission
Society for Thoracic Surgeons
ActiveHealth Management, Inc.
Resolution Health, Inc.
Health Benchmarks, Inc.
(N=615)
117
97
96
53
28
27
27
25
10
Endorsement, adoption, approval
 Only the National Quality Forum (NQF)
formally “endorses” measures
 Hospital Quality Alliance (HQA)
 AQA Alliance
 AMA Physician Consortium for
Performance Improvement
 National Quality Measures Clearinghouse
(NQMC) “includes” measures based on 9
“inclusion criteria”
Life cycle of measures: CMS
Life cycle of measures: NCQA
Conclusion
 Developers and evaluators of quality
measures CAN have an impact on policy
and practice
 Publish and promote
 Establish partnerships and relationships
(contractual or otherwise)