FDA Approach to Review of
Outcome Measures for Drug
Approval and Labeling:
Content Validity
Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials
(IMMPACT)
April 17-18, 2014
Elektra Papadopoulos, MD, MPH
Study Endpoints and Labeling Development (SEALD)
Office of New Drugs (OND)
Center for Drug Evaluation and Research (CDER)
Disclaimer
The views expressed in this presentation are those
of the speaker, and do not necessarily represent
an official FDA position.
2
Outline
• Elements of the review of content
validity
• Common instrument design
shortcomings
3
ELEMENTS OF THE REVIEW OF
CONTENT VALIDITY FOR CLINICAL
OUTCOME ASSESSMENTS
4
5
Content validity
• Content validity is the extent to which the content of
an instrument represents important aspects of a
given concept for an intended use and for a defined
target population
• Establishing content for a new instrument may
involve both qualitative and quantitative research
methods. Qualitative data are essential for
establishing content validity of a COA.
• Input from the target population of respondents is
essential
6
Content validity review
• Content validity review necessitates review of each
element of an instrument
• Instrument: A means to capture data plus all the
information and documentation that supports its use
e.g.,
– User manual
– Instructions
– Mode of administration (e.g., self administration or interviewer
administered)
– Data collection method (e.g., paper-based, computer assisted)
– Scoring
7
Sources of evidence to support
content validity
• Literature review
• Expert input
– Clinical
– Instrument development
• Patient input
– Concept elicitation interviews
– Cognitive interviews
8
Sample criteria for evaluating items
9
Patrick DL et al; Value in Health; 2011
COMMON DESIGN SHORTCOMINGS
IN ITEM DEVELOPMENT
10
Common instrument design
shortcomings
•
•
•
•
Global ratings of a complex concept
Ratings of change
Assessment of an unobservable concept by an observer
Measuring indirect impacts without also measuring
direct
• Inappropriate recall period
• Combining more than one concept within a single item
(e.g., double-barreled items)
11
Global ratings of a complex concept
• Example:
– Overall, based on all your years of medical practice, how
severe is this patient’s overall health condition?
• It is a problem because:
– Respondents consider unknown/unspecified aspects of
the patient’s condition and assign a value/rating in an
unstandardized manner
• To avoid the problem:
– Select/develop instruments that include separate items or
scores for each important sub-concept (e.g., sign or
12
symptom) relevant to the concept of interest
Ratings of change
• Example:
– Compared to your pain before you had cancer, please rate
your pain today
• It is a problem because:
– Respondents are required to recall their previous state and
make a comparison to some period in the past
– Ratings of change do not describe the patient’s current
state
• To avoid the problem
– Use items that ask about the patient’s current state in a
non-comparative way
13
Assessment of an unobservable
concept by an observer
• Example:
– How severe is your child’s pain
• It is a problem because:
– Assessments of patients by other individuals should be
limited to observable concepts; ratings of a patient’s
feelings/symptoms/mental states cannot validly be done
by a person other than the patient
• To avoid the problem:
– Use items that ask about observable behaviors (e.g. How
frequently did your child cry?)
14
Measuring indirect impacts without
also measuring direct
• Example:
– I am afraid I won’t have enough time to reach the
bathroom. I worry my incontinence will get worse.
• It is a problem because:
– Emotions such as “bother”, “distress”, “afraid” do not
measure the effect of treatment on core disease
symptoms
• To avoid the problem:
– Use a more direct measure (e.g., log of the number of
incontinence episodes)
15
Inappropriate recall period
• Example:
– For a condition where itch fluctuates daily: Over the past
four weeks, on average, how severe was your itching?
• It is a problem because:
– The recall period requires not only accurate recall, but
also complex mental tasks (e.g., averaging of experiences
that may fluctuate over a lengthy period of time)
• To avoid the problem:
– Use items with an appropriate recall period for the
population, concept and clinical trial context
16
Combining more than one concept
within a single item
• Example:
– Today, how many times did you cough up blood OR
sputum?
• It is a problem because:
– Does not allow discrimination of which concept the
respondent is rating
• To avoid the problem:
– Use one concept per item
• Today, how many times did you cough up blood?
• Today, how many times did you cough up sputum?
17
To Illustrate (Lack of) Content Validity
• Study population = primarily bed-ridden, minimal physical
activity (e.g., walking to the bathroom) very difficult
• Concept of interest to evaluate treatment benefit = physical
functioning
• Clinical Outcome Assessment = patient-reported
questionnaire to assess physical functioning
• Question in the assessment = “Do you have trouble running
to the bus?”
18
To Illustrate (Lack of) Content Validity
• Study population reports that mobility is a significant problem
• While interviewing patients, it is identified that their biggest
concern/challenge is the ability to stand up from a chair and
start walking
• Performance measure used in clinical trials: 6 minute walk
test
• No assessment of the relevant component of mobility that is
meaningful to this population of patients in their daily lives
19
SF-36 Physical Function Domain
20
Carpal Tunnel
Release
Hip Joint
Replacement
21
Patient report of “ability to perform
daily activities”
FDA Guidance: Patient Reported Outcome Measures: Use in Medical
Product Development to Support Labeling Claims
22
Challenges with the use of actigraphy
• What aspect of physical activity is measured by
actigraphy?
• What types of data (qualitative and quantitative)
would be needed to link actigraphy results to a
meaningful aspect of daily functioning in the
patient population of interest?
• For example, patients with lower back pain are
limited in some aspects of functioning that are not
captured by acceleration data such as the ability to
stand in one place or lift/carry objects
23
Summary
• Content validity is a critical measurement property
that pertains to all COAs
• Inadequate content validity may lead to:
– inability to detect a treatment effect and
– inability to interpret/describe a treatment effect
• Careful consideration of instrument design is
critical to ensure the COA can be interpreted and
accurately described in product labeling
24