ISPOR PRO TASK FORCE:
USING MIXED MODES TO CAPTURE PRO DATA IN CLINICAL TRIALS
Background and Issue Description
As stated in the US Food and Drug Administration (FDA) Guidance for Industry titled Patient-Reported Outcome
Measures: Use in Medical Product Development to Support Labeling Claims, “We intend to review the
comparability of data obtained when using multiple data collection methods or administration modes within a
single clinical trial to determine whether the treatment effect varies by methods or modes” (FDA, 2009).
However, the “PRO Guidance” does not discuss ways to ensure or enhance the comparability of the data when
mixed data capture modes are used. The proposed PRO Mixed Modes Task Force will develop recommendations
that address this important PRO measurement issue.
The incorporation of the patient perspective in the evaluation of medical products (i.e., drugs, biologicals, devices)
has been increasingly viewed as important, if not essential. Medical products aimed at relieving patients’
symptoms and/or improving levels of self-reported functioning will require patient-reported outcomes (PROs) as
endpoints in clinical trials. As stated in the FDA’s PRO Guidance, “Use of a PRO instrument is advised when
measuring a concept best known by the patient or best measured from the patient perspective” (FDA, 2009).
There is no doubt that the release of this PRO Guidance by the FDA has focused increased attention on the
scientifically sound measurement of PRO endpoints in clinical trials. Contemporaneous with the increasing
attention on the appropriate use of PRO measures as efficacy endpoints, the use of technology in clinical trials
expanded substantially. There is growing recognition of the many advantages of electronic data capture
technologies for PRO measures (i.e., ePROs), including less subject burden, avoidance of secondary data entry
errors, easier implementation of skip patterns, date and time stamping, and more accurate and complete data
(Stone et al., 2002). Migrating from paper to electronic data collection has been one of the most significant
movements in the PRO measurement field.
Alongside the emergence of many new ways of capturing PRO data is the need to assure measurement equivalence
across and among these modes of administration and data capture (Coons et al., 2009). This becomes especially
important if multiple modes are utilized within a single trial. It is clear from the quote from the PRO Guidance
provided in the Problem Statement that the FDA anticipated this situation would occur. A prior ISPOR task force
report addressed the evidence necessary to demonstrate measurement equivalence between electronic and paperbased PRO measures (Coons et al, 2009). Although the task force recommendations focused on the migration of
paper-based PRO measures to electronic platforms, the same principles apply to the assessment of measurement
equivalence across and among all PRO data collection modes. However, the use of mixed modes of data capture
within clinical trials was not substantively addressed.
The goal of the PRO Mixed Modes Task Force is to develop recommendations regarding good research practices
for studies in which PRO data capture and/or administration modes are mixed within a single trial, or across trials
that are intended for direct comparison. The primary objective is to address issues that must be considered to
avoid sources of measurement error that materially impact the measurement properties of the instrument being
used to capture PRO endpoints in clinical trials. Our intent is to provide practical means of optimizing data
integrity when mixed modes are unavoidable or when the benefit of mixed modes is perceived to justify the risk.
For the purpose of this task force report, it is necessary to address the distinction made in the PRO Guidance
between PRO instrument administration modes and data collection methods. According to the Guidance,
administration mode refers to self- vs. interviewer-administered, while data collection method refers to the tool
used for capturing the data such as paper-based questionnaires, web-based data entry, interactive voice response
systems (IVRS), or any of the other ePRO devices (FDA, 2009).
When discussion turns to the issue of “mixing” or using more than one of these modes or methods in a clinical
trial, we find that the terminology distinction is potentially misleading, because the term “mixed methods” in the
larger PRO field refers to mixing qualitative and quantitative methods in research, and is not associated with
multiple methods of data capture. On the other hand, the PRO field has a long history of using the term “mixed
modes” to refer to both administration as well as data capture (i.e., ePRO vs. paper). Therefore to simplify the
discussion in this paper, we will use the term “mode” in the context of both modes of administration per the
guidance and modes of data capture per the PRO field.
Figure 1 presents examples of modes of administration and data capture to illustrate the universe of ways that
PRO data collection can be accomplished along with examples of sources of variability between these different
modes. Regardless of whether different modes are being considered, the comparability of the data obtained via
the original and alternative data collection approaches must be assured. If a new or different data collection
approach introduces response bias, the appropriateness of pooling data within a trial or comparing data across
trials is in question.
Figure 1. Modes of PRO administration and data capture, and the sources of variability
that may result from mixing modes
Mode of
administration
1. Selfadministered
Direct patient report
considered PRO
2. Interviewer
administered
Considered PRO if
items read verbatim
and patient answer
recorded without
interpretation
Mode of data
capture
Paper
Handheld
Tablet /Netbook
IVRS
Web via
computer
Web via phone
In person – paper
In person- Tablet
Over the phone
-paper
ComputerAssisted
Telephone
Interview (CATI)
Sources of variability
between methods *
Variation due to items
being seen or heard; how
they appear on page or
screen; number of items
visible on page or screen
at one time, how
responses are presented,
and how patients are to
input answer
Variation due to direct or
indirect presence of
interviewer; and
variation across
interviewers (e.g. age,
gender, personality)
Sources of
variability between
modes
Patient may alter
response due to
presence of
interviewer (e.g.
social desirability);
and variation
across interviewers
(e.g. age, gender,
personality)
* Shown here are key sources of variability between methods; for an exhaustive list see: …..
Note: Proxy or informant reporting is not considered a PRO
The focus of the task force report will be on mixed modes of data capture of PRO instruments. While there are
similar concerns with mixed modes of administration, the likelihood of differences between self and interviewer
administration is increased in comparison to mixed modes of data capture, and introduces issues such as social
desirability that are beyond the scope of this task force. Likewise, the task force will not address comparisons
between PRO and clinician or other observer ratings or proxy ratings, as there is much documentation of lack of
concordance between these measures and patients’ own ratings such that equivalence is not expected (Byrom &
Mundt, 2005; Debrota et al., 1999; Petrova et al, 2000).
The methodological purists among us recommend that PRO data capture modes not be varied within a single
clinical trial or between trials that seek to provide comparable data. In general, anything that has the potential to
introduce measurement error into a trial should be avoided (Streiner and Norman, 2006). Measurement error is,
in essence, noise (error variance) that reduces statistical power and attenuates the ability of the trial to detect real
change (i.e., treatment effect) in the trial endpoint. Clinical trial designs should avoid as many sources of error
variance in the data as possible. Potential error variance can be introduced into the trial design by different data
collection modes used within the trial that do not provide comparable data (i.e., the modes lack sufficient
measurement equivalence.) However, although it may not be optimal, mixing of PRO data collection modes
within trials does occur and has to be addressed pragmatically.
While mixed modes can and do occur in all research settings, the primary focus of this paper is on clinical trials as
the stakes and risks are highest for sponsors and investigators in this setting. The greatest risk when using mixed
modes for the PRO endpoint(s) within clinical trials is to find nonequivalence between the modes which could be
the difference between success and failure for the trial. Given the magnitude of the possible outcome, how can
mixed modes occur during clinical trials? Mixed modes may occur within and between the following levels: 1)
drug development programs, 2) clinical trials within a program; 3) countries within a clinical trial; 4) sites within
a trial; 5) patients within a site; and 6) within a patient.
Mixed modes occurring between drug development programs or clinical trials within a program are often the
result of evolving technology. New and better methods of PRO data capture emerge or regulatory requirements
change, and PRO modes must change during the drug development program. The implementation of mixed
modes in these situations can be carefully planned and executed with supporting validation studies to
demonstrate the equivalence of modes.
Usually the mixing of modes by countries within a clinical trial is the result of access issues. Not all of the
countries in the trial may have widespread access to the technology being implemented for PRO administration
(e.g. internet for web-based PRO measures). In such cases, specific countries within multi-country studies may
collect PRO data using one mode while the remaining countries use another. Equivalence will need to be
demonstrated across mode and country.
Modes can also be mixed among participating clinical sites within a trial for a variety of reasons such as access to a
specific technology or site capacity. Patients within sites may use mixed modes to complete PRO measures for
several reasons-- ability, health state, or patient preference. Moreover, the site may have a preferred mode for
individual patients. Finally, some patients may begin in one mode but switch to another as a backup in cases of
device loss or failure, or inability to access the electronic version. This situation can be difficult to detect unless
there is a mechanism to note how the PRO data was captured.
All of the above situations can add measurement error to the PRO endpoints and while purists recommend the
situations should be avoided, the reality is that sometimes it’s unavoidable.
Specific topics in the task force report will include:
•
Issues with mixed modes of data capture in clinical trials
o Potential risks
o Potential benefits
•
•
Strategies for appropriate use of mixed modes in clinical trials
Analytical approaches for evaluating mixed modes data