Designing Health-Related
Quality of Life Measures
Ron D. Hays, Ph.D.
February 2, 2015 (9:00-11:50 am)
HPM 214, Los Angeles, CA
Ad Hoc Preference Score Estimates
Comprehensive Geriatric Assessment (n = 363
community-dwelling older persons) lead to
improvements in SF-36 energy, social
functioning, and
Physical functioning (4.69 points) in 64 weeks
 Cost of $746 over 5 years beyond control
group

Keeler, E. B., et al. Cost-effectiveness of outpatient geriatric
assessment with an intervention to increase adherence. Med
Care, 1999, 37 (12), 1199-1206.
Is CGA worth paying for?
Change in QALYs associated with 4.69 change in
SF-36 physical functioning scale
r = 0.69 -> b = .003
 QWB = 4.69 x .003 = .014
 .014 x 5 yrs. = 0.07 QALYs
 Cost/QALY: $10,600+

<$20,000 per QALY worthwhile
[0.69x (0.103)/(23.5) = .003]
Listed below are a few statements about your relationships
with others. How much is each statement TRUE or FALSE
for you?
1. I am always courteous even to people who are
disagreeable.
2. There have been occasions when I took advantage of
someone.
3. I sometimes try to get even rather than forgive and
forget.
4. I sometimes feel resentful when I don’t get my way.
5. No matter who I’m talking to, I’m always a good listener.
Definitely true; Mostly true; Don’t know;
Mostly false; Definitely false
Give yourself 1 point for each item that
you answered as shown below
(Possible score range is 0-5)
- I am always courteous even to people who are
disagreeable. DEFINITELY TRUE
- There have been occasions when I took
advantage of someone. DEFINITELY FALSE
- I sometimes try to get even rather than forgive
and forget. DEFINITELY FALSE
- I sometimes feel resentful when I don’t get my
- way. DEFINITELY FALSE
- No matter who I’m talking to, I’m always a good
listener. DEFINITELY TRUE
How many people scored?
•
•
•
•
•
•
•
0 points? _______
1 point? _______
2 points? _______
3 points? _______
4 points? _______
5 points? _______
Mean was 1-2 points in MOS
End goal is measure that is
“Psychometrically Sound”
• Same people get same scores
• Different people get different scores and differ in
the way you expect
• Measure works the same way for different
groups (age, gender, race/ethnicity)
• Measure is practical
7
First law of survey development:
Only do it when necessary
8
http://www.ispor.org/TaskForces/PROInstrumentsUse.asp
Developing a new PRO instrument is a labor and time
intensive task.
Use of existing instruments is generally preferable to
developing a new instrument or modifying an instrument.
If a PRO instrument is modified, additional validation studies
may be needed to confirm the adequacy of the modified
instrument’s measurement properties. The extent of
additional validation recommended depends on the type of
modification made.
The FDA intends to consider a modified instrument as a
different instrument from the original and will consider
measurement properties to be version-specific.
http://www.ispor.org/workpaper/FDA%20PRO%20Guidance.pdf
9
PRO Iterative
Development Process
10
Documentation
+ Chronology of all item development activities
+ Protocols for qualitative interviews, focus groups, cognitive interviews
and other research used to identify concepts, generate items, or revise
an existing instrument, including training of interviewers
+ Development of response options, modes of administration and scoring
+ Size, characteristics, location, and (if requested) transcripts of each
qualitative interview and focus group
+ Documentation on how saturation was achieved (i.e. no new information
was obtained from additional qualitative interviews or focus groups)
+ Description of any pilot test, including cognitive interviewing, cognitive
interview transcripts (if requested)
+ Versions of the instrument at various milestones of development
+ Item tracking table that list the source of each item in the final
instrument, and how it changed during development
+ A summary statement of qualitative research in support of content
validity of the PRO instrument
11
D. Patrick et al, Value in Health 2007, 10, S125-37
Second law: Know thy respondent
12
Chapter 1.2 (Elaine McColl)
“seeking lay input on the meaning of health
and the impact on quality of life of living with
an index condition is now becoming the
norm. A further advantage of lay
involvement is that the developer can gain a
greater insight into the language and
terminology used by the target respondents,
and can thus ensure that the vocabulary
used in the instrument is culturally
appropriate.”
13
Measurement Steps
•
•
•
•
•
•
Review literature
Focus groups
Define constructs
Draft items
Pretest (cognitive interviews)
Revise and finalize items
14
Literature Review
Edgell, E. T., et al. (1996). A review of health-related quality-of-life
measures used in end-stage renal disease. Clinical Therapeutics,
18, 887-938.
Coons, S. J., et al. (2000). A comparative review of generic quality
of life instruments. PharmacoEconomics, 17, 13-35.
Golomb, B., Vickrey, G., & Hays, R. D. (2001). A review of healthrelated quality-of-life measures in stroke. PharmacoEconomics,
19, 155-185.
Castle, N.G., Brown, J., Hepner, K. A., & Hays, R. D. (2005).
Review of the literature on survey instruments used to collect data
on hospital patients’ perceptions of care. Health Services
Research, 40 (6 Pt. 2), 1996-2017
Shen, J., Johnston, M., & Hays, R. D. (2011). Asthma outcome
measures. Expert Reviews of Pharmacoeconomic and Outcomes
15
Research, 11, 447-453
Search Internet
http://www.library.ucla.edu/libraries/biomed
http://www.ingentaconnect.com/
http://www.google.com/
http://www.rand.org/health/surveys_tools.html
http://www.sf36.com/
http://www.proqolid.org/
http://healthmeasurement.org/
http://chime.med.ucla.edu/
http://www.ncbi.nlm.nih.gov/pubmed/
16
Focus Groups
• Discuss feelings, attitudes, perceptions
• Learn
– Vocabulary and thinking patterns
• Conversational meeting
– Moderator and 6-12 people
– Questions posed
– Group synergy
– Economical
17
Focus Group Logistics
•
•
•
•
•
•
•
HSPC/IRB
Recruitment and payment
Moderator and note-taker
Visitors
Audio-record (backup)
Transcript
Background questionnaire
Focus group excerpts follow
18
Conservation of Energy
• Because of Hepatitis C, Jenny has had to set
“boundaries” with people. For example, she has had to
close her office door and say “no” at times.
• She has had to figure out how much energy she has to
get to the finishing line. “If I don’t have enough energy to
get to the finishing line, it means I won’t be available for
everyone else.”
rhays:
Example of unique issue that would be missed if survey only asked whether Jenny cut down on work
instead of asking whether she adjusts her work behavior to ensure she doesn’t deplete her energy
reserve.
Question 14a
19
Longer Recovery Period
• Candy has a full-time job fixing earthquake
damage and “needs the rest of the day to
be able to do that.”
• She engages in heavy work, but it takes
her several hours to recover from it.
Rhays: Question 15d, 16e, 25g
20
Work life accommodation
and physical limitations
• Manny is a self-employed builder who has owned his
business for 25 years. He goes to bed at about 9:00 and
gets up about 5:30 am. He wakes up several times
during the night.
• Manny has changed from participating actively on the job
site with his employees to only setting up the job and
supervision.
Rhays: Quest. 4a-d may not capture this very well.
21
Symptoms of disease
• Since he went off interferon, Manny has had a lot of
muscle fatigue. His legs cramp when he surfs.
• He can’t arm wrestle his 12-year old son, etc. Forcing
himself to do even 10 pushups now is difficult.
• Manny has also been getting little pimples on his legs
(“cryo globulins”) due to inadequate blood circulation.
Rhays: Questions 12a, b, n
22
Side effect of medicines
• When taking interferon Manny said that he
tended to get angry and yell and scream at
his wife.
• Manny also indicated that he gets a
stomach ache daily somewhere between
noon and 2 pm whether or not he has
eaten. “All of the sudden I feel nauseous.”
Rhays: Questions 14c, 14f, 27a
23
Daily unpredictability;
change in relationships
• Tammy lost her best friend as a result of
hepatitis C, because the friend didn’t like
her last minute cancellations caused by
the disease.
• However, it has brought her very close to
her spouse—he is very supportive and
even “babies” her now.
Rhays: Question 14e, 16d, 26e
24
Keeping disease secret
from others
• Fred only recently told other people that
he had hepatitis C (other than his wife).
He told a few people and recommended
that they get checked for it themselves.
• He is more comfortable telling people now
because he has more information about
the disease and can explain what it is.
Rhays: Questions 26e, i, m
25
Third law: Practice before you play
“Cut and try, see how it looks and sounds, see
how people react to it, and then cut again, and
try again” Converse & Presser (1986, p. 78)
Identify problems with
–
–
–
–
Comprehension of items (stem/response options)
Retrieval of information
Skip patterns
Response burden
26
Draft Items
27
Target Respondent--Educational Level
• Literacy expert
• Readability indices
Morales et al. (2001).
Readability of CAHPS 2.0
child and adult core surveys.
In M. L. Cynamon & R. A.
Kulka (eds), Seventh
Conference on Health Survey
Research Methods (pp. 8390). DHSS Publication No.
(PHS) 01-1013, Hyattsville,
Maryland.
28
Flesch-Kincaid Grade Level
FK GL = 0.39 * (n of words/n of sentences) + 11.8 * (n of syllables/n of words)
– 15.59
•
Driven by sentence length and syllables per word
•
U.S. school grade level (e.g., 8.0 implies that 8th grader can understand the
document).
•
Possible minimum = -3.4
– Green eggs and ham averages 5.7 words per sentence and 1 syllable per word
– (FK GL = -1.3)
Challenges
• Items have fragments such as preamble
phrases
• Response choices
In general, would you say your
health is:
• FK GL = ?
• Fragment not scored because not
recognized by computer as a complete
statement, declaration, or question that
ends with a period, exclamation mark, or
question mark.
In general, would you say your
health is excellent, very good,
good, fair, or poor?
• FK GL = 5.9
Items in Grid Format
The following questions are about activities you
might do during a typical day. Does your health
now limit you in these activities? If so, how
much?
1) Vigorous activities, such as running, lifting heavy objects,
participating in strenuous sports
Yes, limited a lot; Yes, limited a little; No, not limited at all.
Instructions separately
The following questions are about activities
you might do during a typical day. Does
your health now limit you in these
activities? If so, how much?
• FK GL = 5.9
The following questions are about activities
1
3
2
1
2
4
you might do during a typical day. Does
1
1
1
2 1 3
1
1
your health now limit you in these activities?
1
1
1
2 1 1 1
4
If so, how much?
1 1 1
1
 26 words, 3 sentences, 8.6 words/sentence, 40
syllables (0.39 * 8.6) + (11.8 * 40/26) – 15.59 = 5.9
Slightly revised instructions
The following questions are about activities
you might do during a typical day. Does
your health now limit you in these
activities?
• FK GL = 8.0
The following questions are about activities
1
3
2
1
2
4
you might do during a typical day. Does
1
1
1
2 1 3
1
1
your health now limit you in these activities?
1
1
1
2 1 1 1
4
 22 words, 2 sentences, 11 words/sentence, 36
syllables (0.39 * 11) + (11.8 * 36/22) – 15.59 = 8.0
Item
Vigorous activities, such as running, lifting heavy objects,
participating in strenuous sports
===
Does your health now limit you in vigorous activities, such
as running, lifting heavy objects, participating in
strenuous sports?
• FK GL = 13.5
EQ-5D Instructions (9.3)
By filling in one circle in each group below,
please indicate which statements best
describe your own health state today.
Flesch-Kincaid for Items
Paz, S. H., Liu, H., Fongwa, M. N., Morales, L. S., & Hays, R. D. (2009).
Readability estimates for commonly used health-related quality of life surveys.
Quality of Life Research,18, 889-900. .
Median
Range
HUI
9.4
6.1—12.0
QWB
8.5
1.2—12.0
SF-36
5.1
0.6—12.0
EQ-5D
4.4
0.5—10.3
Fourth law: Keep it simple and short
41
Use only as many words, items and
response options as needed
42
Survey Instructions
Thank you for taking the time to fill out this survey. The purpose of this survey
is to learn about your experiences as a cancer patient. The information you
provide is very important. It will help to improve cancer services for other
patients.
Many of the questions ask about your experiences at your “Cancer Center.” A
Cancer Center refers to the hospital, center, or institute where you receive
most of your cancer care. A Cancer Center also refers to the doctors,
nurses, and other health care professionals who work with the hospital,
center, or institute. In some places, the Cancer Center is all in one building.
In other places the doctors, nurses, and other health care professionals who
work with the Cancer Center are in different locations.
Many of the questions ask about your “Cancer Care Team.” A cancer care
team refers to the doctors, nurses, and other health care professionals who
provide your cancer care. Your cancer care team might also include social
workers, counselors, patient navigators and others who help with your
cancer care.
You may feel that some questions are easier to answer and some are harder to
answer. Please remember that there are no right or wrong answers! We
want to know about your experiences, both good and bad. If you truly do
not know the answer to a question, then it is okay to check the box that says
“Don’t know.”
43
Modified
The purpose of this survey is to learn about your
experiences as a cancer patient, both good and bad.
Some of the questions ask about your “Cancer Center”
and others ask about your “Cancer Care Team.”
A Cancer Center is the hospital, center, or institute where
you receive most of your cancer care and includes all the
health care professionals who work there.
A Cancer Care Team refers to the doctors, nurses, and
other health care professionals (social workers,
counselors, patient navigators, etc.) who help provide
your cancer care.
44
Use a recall period that is as
short as possible, but long enough
•
•
•
•
•
Now
Last 24 hours
Last 4 weeks (last month)
Last 6 months
Last 12 months
45
3-5 response options is enough
Strongly correlated with items administered
using more responses options
Miller, D. G. (1956). The magical number seven, plus or minus
two: Some limits on our capacity for processing information.
Psychology Review, 2, 81-96.
46
Response options
•
•
•
•
•
None of the time
A little of the time
Some of the time
Most of the time
All of the time
A good bit of the time
47
Choice of Response
Alternatives
• Respondents assume:
– Average behavior (attitude) is located in
middle of scale
– Extreme behavior located in extremes of
scale.
48
How Tall Is Bond?
• Very tall (+3)
• Moderately tall (+2)
• Tall (+1)
• Short (-1)
• Moderately short (-2)
• Very short (-3)
(r = 0.94)
49
How Many Hours a Day Do You
Watch TV? (A)
•
•
•
•
•
•
Up to ½ hour
½ hour to 1 hour
1 hour to 1.5 hours
1.5 to 2 hours
2 hours to 2.5 hours
More than 2.5 hours
50
How Many Hours a Day Do You
Watch TV? (B)
•
•
•
•
•
•
Up to 2.5 hours
2.5 hours to 3 hours
3 hours to 3.5 hours
3.5 hours to 4 hours
4 hours to 4.5 hours
More than 4.5 hours
51
Percentage of Sample Who
Reported Watching TV > 2.5
Hours Per Day
• A: 16%
• B: 38%
52
PRO Iterative
Development Process
53
Goals of Pretesting
• Identify potential problems
– Item stem or response options
– Comprehension
– Retrieval of information
– Skip patterns
– Response burden
54
Cognitive Interviews
• “Think aloud”
• Intermittent probes
• Retrospective recall
55
Think Aloud
• Participant verbalizes what
they are thinking when they
complete a survey or are
interviewed.
56
Intermittent Probe (A)
• Do you have one person you consider to
be your personal doctor or nurse? Yes/No
• Is this person a medical doctor, a nurse, or
what?
57
Intermittent Probe (B)
• Other than a medical doctor, nurse
practitioner, physician’s assistant, nurse,
or nurse-midwife, is there anyone else you
consider to be a health care professional?
58
Intermittent Probe (C)
Is “health care professional” the way you would refer
to this collection of people or would you use
different words to describe a medical doctor, nurse
practitioner, physician’s assistant, nurse, or nursemidwife?
59
Intermittent Probe (D)
• Questions 20 and 21 were similar (read
both questions again). Which answer
choices were easiest for you? Never to
always or Poor to Excellent?
• Why?
• Do you think these two questions measure
the same thing?
60
Intermittent Probe (E)
• In the last 6 months, how many visits did
you make for yourself to a doctor’s office
or clinic? (Do not include staying
overnight in a hospital, or visits to a
dentist.)
• Tell me about these visits. Who did you
visit each time? Did you include visits to
chiropractors?
61
Intermittent Probe (F)
• In the last 6 months, how often do you
think the chiropractors you saw listened
carefully to you?
• What do you think of when I said “listened
carefully”? Would you have answered
differently if I had asked you “how often do
you think the chiropractors you saw
listened to you?”
62
Retrospective Recall
• Ask probes after
the interview or
survey is
completed.
63
2nd Class Assignment
• Conduct and summarize 5 cognitive
interviews with a self-administered
HRQOL survey instrument. Your written
summary should be no more than 3 pages
in length.
http://chime.med.ucla.edu/qualitative/
http://hpm214.med.ucla.edu/
64
After all this upfront work,
then you can …
• Collect data
• Analyze data
• Write-up results
65
PRO Iterative
Development Process
PRO Iterative
Development Process
Fifth law: Believe the survey
respondent, but only so much
68
Listed below are a few statements about your
relationships with others. How much is each
statement TRUE or FALSE for you?
- I am always courteous even to people who
are disagreeable.
- There have been occasions when I took
advantage of someone.
- I sometimes try to get even rather than
forgive and forget.
- I sometimes feel resentful when I don’t get
my way.
- No matter who I’m talking to, I’m always a
good listener.