STAKEHOLDER –DEFINED RESEARCH
DESIGNING STUDIES THAT REFLECT THE PERSPECTIVES OF
PATIENTS, CLINICIANS AND PAYERS
EPG Grand Rounds, University of Maryland
February 6, 2014
Patricia Deverka, MD, MS
Center for Medical
Technology Policy
The Center for Medical Technology Policy (CMTP) is an independent ,non-profit
501(c) (3) organization that seeks to advance health care innovation and
effectiveness by improving the quality, relevance, and efficiency of health care
research. CMTP works on methods, infrastructure and policy to support the
conduct of comparative effectiveness research that generates information to
assist patients, clinicians, and payers in making informed clinical and health policy
decisions.
OVERVIEW
•
•
•
•
Rationale for engaging stakeholders
Definitions and approach
Case examples
Generative discussion
THE EVIDENCE PARADOX
• How to reconcile:
• ~18,000 RCTs are published each year*
• A growing number of non-experimental studies
• Many systematic reviews, health technology
assessments, clinical guidelines conclude that the
available evidence is limited or studies are poor quality
• Up to 60% of clinical recommendations made by ACC or AHA
based on expert opinion and/or low quality studies
• Systematic review of off-label uses of 19 FDA-approved oncology
drugs (428 pages, several thousand trials) “Because of the paucity of
high quality evidence, the data available – though voluminous –
may have little meaning or value for informing clinical practice”
* Chalkidou, Tunis, Whicher, et al. The role for pragmatic, randomized controlled trials (pRCTs) in comparative effectiveness research.
Clin icalTrials .Published online before print July 2, 2012, doi: 10.1177/1740774512450097
REASONS EVIDENCE PRODUCED BY CURRENT CLINICAL
RESEARCH ENTERPRISE NOT TRANSLATED INTO
PRACTICE
• Differences between settings where research is conducted and
where medicine is practiced
– Patient population
– Interventions, including usual care
– Providers, referral patterns access to care
• Failure to (be able to) report how treatment effects vary in individual
patients and subgroups
• Underrepresentation of children, women, elderly, ethnic & racial
minorities, patients with comorbidities
• Research priorities, study questions, endpoints, etc.
defined by researchers and funders, not end users
THE CER HYPOTHESIS
• Gaps in evidence will be reduced with
increased guidance from payers, patients and
clinicians in study design
• A functional definition of CER would be
research designed in light of meaningful
engagement of these decision makers
DEFINITIONS OF CER AND PCOR
Definition of CER
Definition of PCOR
The generation and synthesis of evidence that
compares the benefits and harms of alternative
methods to prevent, diagnose, treat, and monitor
a clinical condition or to improve the delivery of
care. The purpose of CER is to assist consumers,
clinicians, purchasers, and policy makers to make
informed decisions that will improve health care
at both the individual and population levels.
Patient-Centered Outcomes Research (PCOR) helps
people and their caregivers communicate and make
informed health care decisions, allowing their voices
to be heard in assessing the value of health care
options.
Source: Institute of Medicine
PCOR has the following characteristics:
• Actively engages patients and key stakeholders
throughout the research process.
• Compares important clinical management options.
• Evaluates the outcomes that are most important to
patients.
• Addresses implementation of the research finings
in clinical care environments.
Source: PCORI
WHAT IS UNIQUE ABOUT CER?
• Many CER studies will require an understanding of the
trade-offs between internal validity and increased
generalizability, relevance, feasibility and timeliness
• The right balance is not solely a scientific issue, it’s also
a social judgment about an acceptable level of
uncertainty, involving multiple stakeholders
• Process to achieve this with stakeholder input is evolving
THEORY OF STAKEHOLDER ENGAGEMENT
Evidence suggests that engaging stakeholders in research increases:
Relevance and Content
Relevance and Content
Knowledge Exchange
Application of Evidence
Dissemination Avenues
BARRIERS TO INVOLVING STAKEHOLDERS IN CER
• Confusing terminology, lack of standard definitions
• Timing; restrictions on availability of stakeholders
• Training needs for all stakeholders to maximize participation
• Concerns that process will add time and costs to project plans
• Lack of shared conceptualization of what it means to
“successfully” or “effectively” involve stakeholders in research
• Limited data regarding impact; systematic evaluation rare
Sources: Guise, O'Haire, McPheeters, et al. A practice-based tool for engaging stakeholders in future research: a synthesis
of current practices. J Clin Epidemiol. 2013 Jun;66(6):666-74. doi: 10.1016/j.jclinepi.2012.12.010. Epub 2013 Mar 13. and
CMTP experience
ADDRESSING THE BARRIERS
• Literature review
– Biomedical
– Social science
• Practical experience based on projects involving
stakeholders
• Drafted definitions and conceptual model
• Review and revision by an expert panel
– Patient and Consumer Advisory Council
– NICE Patient and Public Involvement Program and Citizen
Council in the UK
• Applied it to a complex multi-stakeholder project
Diverse Roots of
Public Participation Activities
CBPR*
• Reducing health disparities
• Social change & action
Public policy
• Environmental
planning
• Nuclear power
• Biotechnology
Health care
• Health technology
assessment
• FDA
• Health research (UK)
*Community-based participatory research
TYPOLOGY OF STAKEHOLDER ENGAGEMENT
STAKEHOLDER
ENGAGEMENT
“LIGHT”
STAKEHOLDER
ENGAGEMENT
Source: Nass, Levine, and Yancy. Methods for Involving Patients in Topic Generation for Patient-Centered Comparative Effectiveness
Research –An International Perspective
STAKEHOLDERS
Individuals, organizations, or communities
that have a direct interest in the process
and outcomes of a project, organization, or
policy.
Deverka, Lavallee, Desai, et al. Stakeholder participation in comparative effectiveness research: defining a framework for
.
effective engagement. J Compar Effect Res 2012; 2:181-94
TARGET STAKEHOLDER GROUPS FOR CER
STAKEHOLDER ENGAGEMENT
A process of actively soliciting the knowledge,
experience, judgment and values of individuals
selected to represent a broad range of direct interests
in a particular issue, for the dual purposes of:
1) Creating a shared understanding;
2) Making relevant, transparent, and effective
decisions.
CONCEPTUAL MODEL FOR STAKEHOLDER ENGAGEMENT IN CER
Types of evidence
Analytic-Deliberative Model
Inputs
Values
Research
Professional Experience
Patient and consumer knowledge and experience
Methods of combining evidence
Methods
Quantitative
Questionnaires
Delphi method
Multi-Criteria Mapping
Value of Information modeling
Qualitative
Facilitated
workshops/meetings
Stakeholder decision analysis
Decisions
Outputs
Process
CER
Topic generation
Research priorities
Study designs
Evidentiary thresholds for clinical and health policy decision
making
Implementation strategies
Outcomes
 Meta-criteria, Trust, Respect, Accountability, Legitimacy,
Fairness, Competence
 Change in Knowledge/attitudes
 Change in CER project decisions (e.g. choice of interventions,
study design, funding priorities)
 More useful evidence for clinical and health policy decision
making
 More efficient use of healthcare resources
 Improved health outcomes.
DECIDING WHICH STAKEHOLDER GROUPS TO
INVOLVE IN A PROJECT
• What topic(s) does the research address?
• What health care decision is the research meant to
inform?
• Who are the decision makers responsible for these
decisions?
• Who are the individuals and groups that are
affected by these decisions?
Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes
research. JGIM 2012;27(8):985-91.
WHEN TO INVOLVE PATIENTS AND OTHER
STAKEHOLDERS IN RESEARCH?
•
•
•
•
•
•
•
•
Topic identification and refinement
Priority-setting
Writing proposals (including deciding research methods)
Reviewing research conduct
Interpretation of findings
Dissemination of information
Implementation
Evaluation
Curtis, Slaughter-Mason, Thielke, et al. PCORI Expert Interviews Project: Final Report. Portland, OR: Center for
Evidence-based Policy.
Oregon Health & Sciences University.
STAKEHOLDER
INVOLVEMENT IN
GOVERNANCE
•
•
•
•
•
•
•
•
Decisionmaking
Data access
Data security
Data privacy
Conflicts of
interest
Consent
Communication
Coordination
with other
committees
Infrastructure*
• Assess needs
• Recommend data
elements
• Identify applications
• Monitor
Research
Education
Adapted from: Rein A, Holve E, Hamilton Lopez M, and Winkler J. A
framework for patient and consumer engagement in evidence
generation,” EDM Forum,Academy-Health, September 2012.
•
•
•
•
• Generate & prioritize
research questions
• Design studies
• Data collection,
processing &analysis
• Results interpretation
Translation
Dissemination
Evaluation
Continuous learning
healthcare system
*Refers to the creation/modification of the Patient
Engagement Platform needed to support both clinical
management and the conduct of PCOR
PATient-centered Involvement in
Evaluating the effectiveNess of TreatmentS
PATIENTS
Aims of PATIENTS
1) Foster sustainable partnerships with local, regional, and
national communities of diverse patients and healthcare
systems
2) Conduct and expand PCOR in partnership with patients
and healthcare delivery systems
3) Advance dissemination and implementation strategies for
PCOR findings
23
Vision
The PATIENTS Program vision is that its projects will:
• Further the process of UM institutional transformation for
“MPowering the State” in the area of health
• Eliminate health disparities within Baltimore, throughout
Maryland, and across the nation
• Align with the spirit of the NIH roadmap for transformative
and interdisciplinary research
24
University of Maryland has broad
participation
• UMB professional schools
– Pharmacy
– Medicine
– Nursing
– Social Work
– Dentistry
– Law
• UM College Park
25
PATIENTS Partners
26
Innovation in the PATIENTS Program
• Conducting PCOR with continuous patient and stakeholder
engagement
• Translating research into practice
• Continuous development through formative and impact
evaluation
• Bidirectional learning
• Sustainability
27
The Ten-Step Process for Conducting CER
Public
Announcements
1. Topic Solicitation
2. Prioritization
3. Framing the Question
Delphi Process
Patient
Forums
Based on: Mullins CD, Abdulhalim AM, Lavallee DC.
Continuous Patient Engagement in Comparative Effectiveness Research.
JAMA 2012; 307(15): 1587-8.
28
The Ten-Step Process for Conducting CER
4. Selection of
Comparators and
Outcomes
In-person
Meetings
Telephone
Calls
5. Creation of Conceptual
Framework
6. Analysis Plan
Focus Group
Inter-views
Electronic
Social Media
7. Data Collection
Teleconferences
Based on: Mullins CD, Abdulhalim AM, Lavallee DC.
Continuous Patient Engagement in Comparative Effectiveness Research.
JAMA 2012; 307(15): 1587-8.
29
The Ten-Step Process for Conducting CER
8. Reviewing & Interpreting Results
Teach-Back Method
9. Translation
10. Dissemination
Media
Critique Documents
(e.g. Patient Guides)
Based on: Mullins CD, Abdulhalim AM, Lavallee DC.
Continuous Patient Engagement in Comparative Effectiveness Research.
JAMA 2012; 307(15): 1587-8.
30
Education and Training:
Engaging Partners
Bidirectional Learning
Community Partners
Stakeholder Engagement
Cultural Competence
Using Online Platforms
UM Faculty and Staff
Research Methods
Manuscript Writing
Grant Writing
Examples:
• PatientsLikeMe: Social Media
• BSBHS/Riverside Heath System: Implementation
31
EXAMPLES:
STUDY DESIGNS REFLECTING VIEWS OF
DECISION MAKERS
TWO EXAMPLES
• Genomic Testing in Cancer
(CANCERGEN)
• Evidence Guidance Documents
– Molecular Dx in Cancer
THE PROMISE: TRANSFORM CANCER CARE
Molecular diagnostic (MDx) tests
have the potential to transform
oncology practice by helping
physicians classify and manage
various cancers
• Diagnose and stage cancers
• Help guide therapy selection and
dosing
• Assess treatment response
• Aid in detection of residual or
recurrent disease
34
CANCERGEN STRUCTURE
Thariani R, Wong W, Carlson JJ, et al.
Prioritization in Comparative
Effectiveness Research: The
CANCERGEN Experience in Cancer
Genomics. Medical Care 2012;
50(5):388-393
Figure 1: CANCERGEN project milestones including landscape analysis, stakeholder assessment and final
selection.
RANKING OF TESTS
No Order
1. ERCC1 Expression Testing for Platinum-Based
Adjuvant Therapy in NSCLC
2. EGFR Mutation Testing for Erlotinib Maintenance
Therapy in Advanced NSCLC
3. EGFR gene copy number (FISH) testing and first-line
cetuximab therapy in stage IV or recurrent NSCLC
Final Order
1. ERCC1
2. BCTM
4. BRAF mutation testing in Colorectal Cancer
5. Genetic Expression Profile (GEP) in Multiple
Myeloma (MM) to Identify Patients with Poor
Prognosis
3. EGFR mutation
6. Breast CA Tumor Markers for Detection of
Recurrence After Primary Breast Cancer Therapy
Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN)
LESSONS LEARNED
• Full participation of all stakeholders on highly technical
topics is possible with adequate preparation
• Multi-modal approaches are necessary
– Engagement method should be matched to particular
study question
– Stakeholders were open to novel methods (e.g., VOI), but
more work needs to be done to ensure full benefits of the
approach are fully realized
• Possible to engage most stakeholders for a multi-year
project
– Federal officials are the most difficult
CASE STUDY:
EFFECTIVENESS GUIDANCE DOCUMENTS
A STAKEHOLDER-DRIVEN APPROACH TO
IMPROVING THE EVIDENCE BASE
Molecular Diagnostics (MDx) In Oncology
DECISION MAKERS’ KEY QUESTIONS FOR CANCER MDX
TESTS
1. Does the MDx test provide correct information? (analytic
validity)
2. How well does the test result correlate with clinical
outcome? (clinical validity)
3. Does use of the MDx test lead to improved patient
outcomes as compared with the alternative? (clinical
utility)
4. Does use of the MDx test lead to greater value as
compared with the alternative? (cost-effectiveness)
PROBLEM
• Analytic validity and clinical validity now available for an
increasing number of MDx tests
• Clinical utility largely unknown for most MDx tests
• Uncertain clinical utility has consequences for patients and
health care system
– Decreases quality through inconsistent or unnecessary use of tests
– Wastes health care resources
WHAT’S NEEDED
•
•
•
•
Structured data about MDx test use
Stakeholder-driven process
Clear evidentiary standards for clinical utility
Willingness to consider range of methods,
outcome measures that are relevant to real-world
clinical decisions
THE RESPONSE: EFFECTIVENESS GUIDANCE DOCUMENTS
• Provide specific recommendations on the design
of studies intended to inform decisions by
patients, clinicians and payers
• Developed for specific clinical conditions and
categories of technologies
• Based on a structured, transparent, multistakeholder process led by CMTP
• Aim to balance internal validity, relevance,
timeliness and feasibility
•
provide decision-makers with a reasonable level
of confidence that the intervention improves net
health outcomes
• Analogous and complementary to FDA guidance
• Targeted to researchers working in industry or
academic settings
44
TECHNICAL WORKING GROUP
•
•
•
•
•
•
Academic researchers 2
Industry 3
Payers 2
Research funders 1
Policy makers 1
Patient advocate 1
RECOMMENDATIONS: 10 COVERING CLINICAL VALIDITY
AND CLINICAL UTILITY
• MDx test development follows phases similar to the phases of drug
development
• Recommendations have been organized around these phases
• Biomarker discovery (Phase 0) and the assessment of population impacts
(Phase 5) go beyond the scope of this EGD
46
HOW EGDS MIGHT BE USED
• By test developers and researchers in designing studies
• By payers in evaluating evidence submitted for
coverage and reimbursement
• By guidelines developers in judging quality of evidence
and strength of recommendations
• By research funding organizations in evaluating grant
proposals
• By patient advocacy and other groups generating
guidance for patients
SUMMARY
• Stakeholder engagement is essential for fulfilling the objectives of
CER
• Research funding requests and support need to account for resources
required to meaningfully implement engagement activities
• The terminology and methods are being developed and tested worked
for CER
• Methods need to be tailored to the particular phase of research, but
stakeholders should be involved throughout the process
• Careful attention to communication at multiple levels is critical to ensuring
true collaboration and a respectful, accountable process
• There is a growing body of examples of rigorous methods of SE being
applied to CER
• Evaluation is critical for measuring impact and process improvement
Extra Slides
CMTP: PRINCIPLES FOR INVOLVING PATIENTS IN
COMPARATIVE EFFECTIVENESS RESEARCH
• Each CER-related project includes patient representatives.** Examples of
such projects include identifying research topics, setting priorities,
developing questions to be studied, designing study protocols and
establishing methodological standards.
• Project leaders recruit a diverse group of patients for whom the project topic
is relevant.
• Project leaders, patients and other stakeholders make their mutual
expectations for patient involvement known to each other.
• The project team, other stakeholders and patients disclose potential conflicts
of interest.
• Project budget includes appropriate remuneration for patients and support
for their participation, including training, stipends, travel and lodging, and
other resources critical for their full involvement in the project.
**Includes patients, family caregivers and consumers representatives of all types
CMTP: PRINCIPLES FOR INVOLVING PATIENTS IN
COMPARATIVE EFFECTIVENESS RESEARCH……..CONT’D
• The project team and other stakeholders recognize and respect the
different skills, knowledge and experience of patients. Patients recognize
and respect those of the other participants.
• The project team communicates regularly with patients throughout the
life of the project to ensure the quality and sustainability of the
involvement process.
• The project team obtains periodic assessments from patients and other
stakeholders to evaluate the effectiveness of the engagement process and
inform the design of future research involving patients.
• Project reports and publications describe in the methods sections how
patients were involved in research.
• The project team and other stakeholders work with patients to present
study findings in a way that can be easily understood by patients.