ePRO Evolution and
Emerging Best Practice
Adam Wood
PDSM PRO Workshop 2010
Content
•
•
•
•
Defining PROs
Where are we now?
Why ePRO?
Obstacles
– Emerging Best Practice
• What about….?
• For reference
13 April 2015
confidential
Page 2
Defining PROs
Defining PROs
• “A PRO is a measurement of any aspect of a
patient’s health status that comes directly from the
patient” - FDA
– Patient diaries
– Questionnaires
– Disease specific measures
• ePRO is any electronic implementation
– IVRS
– Hardware device (e.g. PDA, tablet)
– Digital pen
13 April 2015
confidential
Page 4
Where Are We Now?
PROs are all round us
• CenterWatch estimate they are used in 75% of
trials
– Visit based measures more common that remote
reported diaries
• Some PROs are more important than others
• Growing need for Health Economic data
– Proving efficacy is one thing, persuading payers to fund
your treatment can be quite another
– Increasing trend to integrate Health Economics data
alongside efficacy data in Ph II/III
13 April 2015
confidential
Page 6
ePRO is Mainstream
• A number of proven vendors with strong track
records
• Best practice is well established
– Success is vastly more likely than failure
• Regulators are positive about PROs
– “Some treatment effects are known only to the patient”*
• ePRO is an integral part of PROs
– Not a separate, special case
– And it’s not EDC
* FDA PRO Guidance.
13 April 2015
confidential
Page 7
ePRO and Regulators
• Regulators in Europe and the US are keenly
supportive of electronic PROs
– invivodata regularly interacts with EMA and FDA
• Drugs approved based on ePRO data worldwide
– Japan (at least one)
– Europe (multiple)
– US (multiple)
• invivodata inspected by regulators in all 3
13 April 2015
confidential
Page 8
FDA PRO Guidance
• Finalised in 2009; drafted in 2006
– Defines new “rules of engagement” for use of PROs
• Questions it triggers
–
–
–
–
–
Conceptual framework
Application of scale to patient population
Patient input into PRO design
Reliability, validity, ability to detect change
Translations
– And, oh yes….mode of administration (p or e)
13 April 2015
confidential
Page 9
FDA “Spoke and Wheel”*
i.
Hypothesize Conceptual Framework
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Determine intended application\characteristics (types
of scores, mode and frequency of administration)
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
•
•
•
•
v. Modify Instrument
•
•
•
•
ii. Adjust Conceptual
Framework & Draft
Instrument
Change wording of items,
populations, response
options, recall period, or
mode of administration/
data collection
Translate & culturally adapt
to other languages
Evaluate as appropriate
Document all changes
PRO
↕
Claim
•
•
•
Prepare protocol & statistical analysis
plan (final endpoint & responder
definition)
Collect & analyze data
Evaluate treatment response using
cumulative distribution of response &
responder definition
Document interpretation of treatment
benefit in relation to claim
•
•
•
•
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Generate new items
Create instrument
Select recall period,
response options, & format
Select mode of
administration /
data collection
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, format, scoring, procedures &
training materials
Document measurement development
* Figure from: Patrick DL, Burke LB, Powers JH, Scott JA, Rock EP, Dawisha S, et al. FDA Paper Draft:
Patient-Reported Outcomes to Support Medical Product Labeling Claims. Value in Health 2007
13 April 2015
confidential
Page 10
FDA on Patient Compliance
(Federal Register, Vol. 71, No. 23; Feb. 3, 2006)
“If a patient diary or some other
form of unsupervised data entry is
used, the FDA plans to review the
protocol to determine what
measures are taken to ensure
that patients make entries
according to the study design
and not, for example, just before a
clinic visit when their reports will
be collected."
(Pg. 10; 334-337 – emphasis added)
13 April 2015
confidential
Page 11
Why ePRO?
Is This Familiar?
Missing data
Ambiguous data
Conflicting data
Extraneous data
From: “Technology Solutions for the collection of patient reported outcomes data”. Bill Byrom. European
Pharmaceutical Contractor, 2004.
13 April 2015
confidential
Page 13
Benefits of ePRO
• Who hasn’t had a paper diary nightmare?
• Process benefits on the theme of cleaner data
quicker
–
–
–
–
–
More complete – high compliance
Improved integrity – date:time stamps
Increased consistency – logic checks
No free or extraneous text
Automatically coded
• Don’t forget better science - far greater compliance
with protocol
– Who’d want to defend paper diary data to a regulator?
13 April 2015
confidential
Page 14
eDiary v Paper
NCI & invivodata Diary Methods Study
Stone et al. (2002): British Medical Journal
invivodata eDiary†
N = 40
Instrumented Paper Diary*
N = 40
†Actual
*Contained hidden
photosensor
13 April 2015
eDiary used
in 2002 Study
confidential
Page 15
eDiary v Paper
Results: eDiary vs. Paper Diary
Stone et al. (2002): British Medical Journal
• Paper Compliance
100
– Reported: 90%
– Actual: 11%
80
60
• eDiary Compliance
40
– Reported: 94%
– Actual: 94%
20
0
Paper
eDiary
Reported compliance
Actual compliance
13 April 2015
confidential
Page 16
Obstacles
Obstacles and overcoming them
• Fear of change
– We’ve always done it this
way
– Precedent not acceptable to
regulators
13 April 2015
confidential
Page 18
Obstacles and overcoming them
• Lack of awareness of
modern ePRO
– No longer something like
this….
– Or this….
13 April 2015
confidential
Page 19
Device ePRO examples
13 April 2015
confidential
Page 20
More obstacles
• Not enough time?
– Not enough time for what?
– Not enough time to plan for success = planning for
failure
• Sites won’t like it?
– Where’s the evidence for this?
– Who pays them?
– Doesn’t stop us using EDC widely
13 April 2015
confidential
Page 21
PRO Scale Migration
• Consensus has emerged on best practice
• Documented by ISPOR “PRO Good Research
Practices Task Force”
• Published in Value in Health
– Coons et al, 1098-3015/08
– “Recommendations on Evidence Needed to Support Measurement
Equivalence between Electronic and Paper-Based Patient-Reported
Outcome (PRO) Measures: ISPOR ePRO Good Research Practices
Task Force Report”
13 April 2015
confidential
Page 22
Levels of Modification
Level of
Modification
Rationale
Minor
The modification can be justified on the basis
of logic and/or existing literature. No change
in content or meaning.
Moderate
Based on the current empirical literature, the
modification cannot be justified as minor.
May change content or meaning.
Substantial
There is no existing empirical support for the
equivalence of the modification and the
modification clearly changes content or
meaning
Adapted from Shields et al.
13 April 2015
confidential
Page 23
Examples of Modification
Level of
Modification
Examples
Minor
1) Non-substantive changes in instructions (e.g., from
circling the response to touching the response on a
screen).
2) Minor changes in format (e.g., one item per screen
rather than multiple items on a page).
Moderate
1) Changes in item wording or more significant
changes in presentation that might alter interpretability.
2) Change in mode of administration involving different
cognitive processes (e.g., paper [visual] to IVR
[aural]).
Substantial
1) Substantial changes in item response options
2) Substantial changes in item wording
Adapted from Shields et al.
13 April 2015
confidential
Page 24
Evidence Required
Level of
Modification
Level of Evidence
Minor
Cognitive debriefing
Usability testing
Moderate
Equivalence testing
Usability testing
Substantial
Full psychometric testing
Usability testing
Adapted from Shields et al.
13 April 2015
confidential
Page 25
Patients can’t use it?
• Severe Parkinson’s Disease patient
13 April 2015
confidential
Page 26
Living with the PRO Guidance
Emerging Best Practice
• PRO Strategy
– PRO Dossier
– PRO input into protocol
– An Endpoint Development Process
13 April 2015
confidential
Page 28
“Begin with the end in mind”
Labeling Claims
Disease/Conceptual Model
Measurement Strategy
Instruments/Items
Conceptual Framework
Endpoint Model
Endpoints
13 April 2015
confidential
Page 29
A Few Words on Instruments
• When is a PRO instrument adequate to support labeling
claims?
– “The adequacy of a PRO instrument as a measure to support
medical product labeling claims depends on its documented
measurement properties that demonstrate the instrument is ‘fit for
purpose’” (Burke, 2008).
• “Fit for purpose” means that the instrument is specific to and defined by the
patient population and the specified disease, treatment, and selected
endpoints.
– In this circumstance, “instrument is defined as a means to
capture data (i.e., a questionnaire) plus all the information and
documentation that supports its use” (Burke, 2008).
13 April 2015
confidential
Page 30
Impact of FDA PRO Guidance
• Study Endpoints and Labeling Division (SEALD) are
reviewing programs where labeling claims based upon
PRO
• Feedback from FDA specifically cites PRO Guidance
• Expectations for PRO activity and documentation being
driven by Guidance
• PRO Submission/dossier template
• Created a need for practical resources to better understand
the nature of PROs
– best practices that facilitate the effective development of PRO
instruments.
13 April 2015
confidential
Page 31
Common Pitfalls/Myths
• We can go at risk and then just ask the FDA afterthe-fact
• Lack of patient-interview based data
– Needed to support the importance and relevance of the
concepts and items in the PRO instrument
• The instrument is ‘valid’ in patient population A,
therefore it is OK to use in patient population B
• We can wait until Ph III to address the PRO
instrument issues
13 April 2015
confidential
Page 32
Endpoint Development Process
• A tool for meeting scientific and regulatory needs
Ongoing Synthesis of
Material
Conceptual
Model
Development
13 April 2015
Concept
Justification
Measurement
Strategy
confidential
Instrument
Identification
Conceptual
Framework
Endpoint
Model
Page 33
Endpoint Development Process
Ongoing Synthesis of
Material
Conceptual
Model
Development
Concept
Justification
Measurement
Strategy
Identify
Program
Goals & Claims
Expert
Input
Focus of
Assessment
Identify
Relevant
Population
Patient
Reports
Identify
Relevant
Drug Effects
Empirical
Evidence
Identify
Relevant
PRO Concepts
13 April 2015
Conceptual
Framework
Endpoint
Model
Identify Existing
Instruments
Theoretical
Validation
Evidence/Plan
Study
Design
Requirements
Interval of
Assessment
Modify Existing
Instruments
Psychometric
Validation
Evidence/Plan
Construction
of
Endpoints
Recall
Period
Identify
Alternate
Measurement
Strategies
Timing &
Schedule of
Assessment
Develop Novel
Instruments
confidential
Instrument
Identification
Page 34
Conceptual Model:
Specifying & Substantiating Concepts
Signs/
Symptoms
Disease
Related
Impact
General
Impact
Drug Action
Concept Justification
Patient
Population
Relevant
Disease
Process
Impact 1A
• Signs
• Symptoms
Impact 2A
Impact 1B
Diagnosis
13 April 2015
confidential
Page 35
Concept Justification
Concept
Expert
input
Empirical
Data
Patient
Interviews
Payor
Relevance
Symptom
Symptom A
Symptom B
Symptom Impact
Symptom Impact A
Symptom Impact B
Yes indicates that this justification criterion has been satisfied
No indicates that this justification criterion has not yet been satisfied
TBD indicates To Be Determined
13 April 2015
confidential
Page 36
Conceptual Framework
PRO
Concept
Label Claim
PRO
Concept
PRO
Concept
Item 1
Item 2
Item 3
Item 4
Item 5
Label Claim
PRO
Concept
13 April 2015
confidential
Item 6
Page 37
PRO Strategy Pointers
• Begin thinking of PRO strategy early in
development
• Interact with regulators
• PRO EDP provides structure for elements of PRO
Strategy and Dossier
• Think about preparation for trial implementation
13 April 2015
confidential
Page 38
What about….?
What about ePRO Design
• ePRO Solution Design
– Maximise value of new modalities
• Don’t simply recreate the weaknesses of paper instruments
– Less missing data
– More precise measures
– More frequent, reliable measures
13 April 2015
confidential
Page 40
FDA’s view on PRO Design
(Federal Register, Vol. 71, No. 23; Feb. 3, 2006)
“PRO instruments that require
patients to rely on memory …
may threaten the accuracy of
the PRO data. It is usually better
to construct items that ask
patients to describe their current
state than to ask them to compare
their current state with an earlier
period or to attempt to average
their experiences over a period of
time.”
(Pg. 11; 339-343 – emphasis added)
13 April 2015
confidential
Page 41
What about…?
• Psychometric validation
– Luckily the FDA have written us an abridged text book
on psychometrics
– It’s called the FDA PRO Guidance
– Applies to paper as well
– Eliciting the patient perspective is a central tenet
13 April 2015
confidential
Page 42
Psychometric Validation
• Large proportion of the FDA PRO Guidance refers to
psychometric aspects
• With any PRO you need to show evidence that:
– “The adequacy of a PRO instrument as a measure to support
medical claims depends on its development history and
demonstrated measurement properties”
– Reliability
– Validity
– Ability to detect change
– Interpretability
• A specialised area that applies as much to paper as
electronic
13 April 2015
confidential
Page 43
What about Linguistic Validation?
• Translations (to a high standard)
– Full title: linguistic validation & cultural adaptation
•
•
•
•
Best practice well established
Plenty of vendors to choose from
Timelines typically 8 – 11 weeks
Applies to paper as well
13 April 2015
confidential
Page 44
For Reference
Scientific Improvements
• Patients are answering questions at the time of
clinical interest
– In other words as specified in the protocol and not just
before a clinic visit
• Patients give more complete answer sets so
improving statistical power (ref: ACT 2004)
• ePRO allows more sophisticated and targeted
questioning
• Competing compounds are now likely to be using
ePRO
– Who wants to be seen using inferior methods.
13 April 2015
confidential
Page 46
Regulatory Motives
• EMA and FDA regularly accept ePRO data for
primary and secondary endpoints.
– This is not something they view with suspicion.
• Recently BfArM (the German regulatory authority)
explicitly directed a sponsor to use ePRO rather
than paper
• The FDA PRO Guidance sets out clear
expectations for the use of any form of PROs
(paper or electronic)
– http://www.fda.gov/downloads/Drugs/GuidanceComplian
ceRegulatoryInformation/Guidances/UCM193282.pdf
13 April 2015
confidential
Page 47
Features of Different ePRO Types
Feature
(incomplete list)
IVRS
eDiary
ePen
Automated coding of data



Time-stamps



Time windows


Alarms / reminders


Branching logic


Prevent question skipping


Prevent extra “writing”



Graphics
Replace Clinician Interviews

Verbal Recordings

High-frequency assessments

Episodic, time-based reporting


Illiterate subjects
13 April 2015

confidential
Page 48
Thank You
Adam Wood
e: awood@invivodata.com
 +44 78 5492 8430