Scripts for QOL Outcomes in Stroke
What I’m Showing/Doing
Slide 1
Slide 2
Slide 3
What I’m Saying
Hello everyone. Before I begin, let me tell you a little about myself. I’m a
researcher in the Dept of MSS here at NU. My research has centered on
measuring health outcomes and using outcome data to improve the
quality of care and patient health. Today I am here to talk about QOL
outcomes in stroke.
But first I’d like to give special thanks to two of my colleagues here at NU,
….. who were kind enough to share some of the slides I am going to
present today.
Everyone in stroke health care--patients, clinicians, researchers, policy
makers and, hopefully you, is interested in answering this question:
[CLICK] What are the effects of stroke and its treatments?
[NEXT]
Slide 4
We can conceptualize these effects in various ways. We may be
interested in effects on the body, which we would discover through
biological and physiological measures describe effects on the body. But
all stakeholders want to know about effects on people’s lives. [CLICK]
These effects include symptoms, how well people are functioning,
general perceptions of health and how health or disease affects quality
of life.
Slide 5
What is Health related quality of life? One definition is that it is “the
……… There are a couple of important points in this definition. One is
that QOL is multidimensional. It includes physical, mental and social wellbeing. The other important point is that QOL is subjective. (CLICK)
[TRANSITION] It needs to be understood from the person’s viewpoint.
Slide 6
Slide 5
Slide 6
Slide 7
Slide 8
These types of outcomes are known as patient-reported outcomes, or
PROs. QOL is one type of PRO. So as we go through my presentation, I
may sometimes refer to PROs rather than HRQL.
[TRANSITION]
Why should you care about outcomes? And why PROs in particular?
Click to ruler slide. PROs are important for both research and clinical
care.
These challenges were appreciated by NIH many years ago.
[TRANSITION]
Therefore, the NIH supported several measurement initiatives to address
1
.
Slide 9
Slide 10
Slide 11
Slide 12
Slide 13
these challenges. These initiatives include Neuro-QOL, PROMIS, the NIH
Toolbox, EXAMINER, and PhenX.
The four measure development projects, Neuro-QOL, PROMIS, the NIH
Toolbox, EXAMINER, shared many of the same goals – to develop
psychometrically sound measures that could be used across diverse
populations, that had minimal cost, and which utilized modern test
development methods.
This slide shows how the initiatives compare in terms of target
populations, respondent types, domains assessed, and language
availability.
As we move on to other presentations in this series, it is important that
you are familiar with some of the terms I will be using.
• Item = question or statement a patient answers
• Instrument = collection of items
• Item bank = type of instrument. A large collection of items
measuring one domain. Any/all items can be used to provide a
score. Makes computer adaptive tests possible.
• Legacy = existing instrument that is “gold standard” or a
commonly used instrument
• Domain = feeling, function or perception you want to measure
(e.g., anxiety, physical function, general health perceptions)
This concludes my presentation on PROs and NIH supported
measurement development initiatives. In the next presentation, I will
introduce you to Neuro-QOL, a set of patient-reported outcome
measures for people with neurological conditions.
Thank you! This work has been supported through the National Institute
for Neurological Disorders and Stroke.
2
Lecture 2b: What is Neuro-QOL?
Slide 1
Hi! Welcome to this presentation about Neuro-QOL , a patient-reported
outcome measurement system for people with neurological disorders. This is
the second of five presentations on this topic and here we will focus on what
Neuro-QOL is and what domains it measures.
Slide 2
Neuro-QOL refers to a set of PRO measures that were developed using a
standard methodology. Neuro-QOL also includes the software that administers
the measures.
Slide 3
Neuro-QOL was initially targeted for use in 5 adult and 2 pediatric conditions.
The adult conditions were epilepsy, ALS, multiple sclerosis, Parkinson’s disease,
and stroke. The pediatric conditions were epilepsy and muscular dystrophies.
The conditions were chosen on the basis of disease prevalence, impact on
quality of life; whether or not there were promising treatments available or in
the pipeline, and the likelihood of seeing change in HRQL.
Slide 4
These are the domains of health assessed by Neuro-QOL. Neuro-QOL assesses
aspects of physical, mental and social health.
Slide 5
The mental health domain includes measures of anxiety, depression, stigma and
cognitive function for both adult and pediatric patients. There are also measures
of positive affect and well-being for adults and an anger measure for children
and adolescents. Communication is a scale rather than an item bank. End of life
concerns is in an early stage of development.
Slide 6
There are two adult measures of social health – ability to participate in roles and
activities and satisfaction with those roles and activities. There is only one fully
developed pediatric measure – interactions with peers.
Slide 7
Neuro-QOL includes 3 different types of instruments – computer adaptive tests
or CATs, short forms, and scales. These instrument types vary in terms of mode
of administration, precision and length. Most, but not all, of the fully
developed instruments constructed as item banks are generally available as
CATs and short forms. CATs are administered adaptively with participants
receiving a variable number of items and item content. A short form is of
fixed length with all participants responding to the same items.
Slide 8
Slide 9
Slide 10
Here is a list of the available Adult measures and the type of instrument they
are. As you can see, all of the adult instruments except for Communication are
item banks.
Here is a list of the available pediatric measures and the type of instrument they
are. Two of the pediatric measures are scales rather than item banks.
This concludes this presentation on Neuro-QOL. In the next presentation in the
series, I will describe how Neuor-QOL was developed.
Thank you! This work has been supported through the National Institute for
Neurological Disorders and Stroke.
3
Lecture 2c: How were Neuro-QOL instruments developed?
Action
Script
Slide 1
Hi! Welcome to this presentation about the
development of the Neuro-QOL measurement system.
Slide 2
Neuro-QOL followed the same measure development
process as PROMIS. Some you may have seen this slide
before. It’s been shown in other presentations on PRO
measurement systems. It is an excellent summary of the
process we followed in order to establish our initial item
pools and then to get them ready for field testing. The
top of the slide, in yellow font, shows the item selection
and development process. As you can see it included
input from patients and experts, a literature review,
selection of existing items and the writing of new items.
Each item pool was then reviewed by experts, underwent
translation, went thru cognitive testing with patients and,
in some cases, was informed by analysis of secondary
data sets. The bottom half of the slide reflects field
testing activities and subsequent development of the
final measures.
Slide 3
When developing Neuro-QOL items, we used 2 different
reference periods and 2 different types of response
options. The recall time frame was either one week or
current. If one week, we asked participants to respond in
terms of “the Past 7 days” There were 3 different ways
we asked about current function. With respect to
response options, in some cases we were interested in
how intensely a participant experienced something. In
other cases, we wanted to know the frequency of their
experience.
Slide 4
We conducted three waves of field testing for
Neuro-QOL. The first two were conducted in
order to calibrate and psychometrically assess
the item banks and scales.
• The first, Online clinical testing in adults (n=500)
and children (n=100) with Neuro-QOL conditions
included those domains that were more relevant
to clinical populations such as stigma and
fatigue. These item banks were calibrated using
respondents with the target neurological
conditions.
• Items in the remaining domains were calibrated
using an English and Spanish general population
sample recruited by an Internet panel company.
4
•
There were 3000 adults and 1500 children in that
sample.
Finally we did multisite clinical validation testing
of IRT-calibrated short forms and the scales
(n=800 baseline; n=400 7 day; n=400 180 day) –
Slide 5
We conducted the last phase of field testing, from
January 2009 through June 2010, to evaluate the
reliability, validity and responsiveness of Neuro-QOL
short forms and scales in clinical populations. A total of
581 adult and 113 pediatric patients were recruited from
the five adult and two pediatric neurological populations
targeted by Neuro-QOL. Proxies for stroke (N=84) and
the two pediatric samples (N=113) also participated.
Administration of Neuro-QOL and concurrent measures
(both cross-disease and disease-specific), physician
ratings and chart review were conducted at baseline and
at a 180-day follow up (to assess responsiveness). Testretest reliability of the Neuro-QOL Short Forms was
evaluated at 7 days. This slide lists the number of
patients with each neurological condition (and proxies)
who completed each assessment.
Slide 6
Participants were recruited from multiple sites across the
US. Recruitment took place at Children’s Memorial
Hospital, Cleveland Clinic Foundation, DartmouthHitchcock Medical Center, NorthShore University
HealthSystem, Northwestern University Feinberg School
of Medicine, Rehabilitation Institute of Chicago,
University of California – Davis, University of Chicago,
University of Puerto Rico, and University of Texas Health
Science Center.
Thank you! This work has been supported through the
National Institute for Neurological Disorders and Stroke.
Slide 7
5
Lecture 2d: Reliability of Neuro-QOL instruments
Slide 1
Hi! Welcome to this presentation about the psychometric properties of NeuroQOL instruments. This is the first of two presentations on this topic. Here we
will talk about reliability.
Slide 2
What is reliability? In research, reliability means consistency or repeatability.
Let’s pretend you are an archer, shooting at a bullseye. If you have a reliable
arrow then each time you shoot, the arrow will always hit the same spot, or at
least very close to the same spot. (CLICK)
Slide 3
Slide 3
Slide 6
Slide 7
Slide 8
There are different kinds of reliability. Internal consistency reliability refers to
the consistency of results across items within a test, or, how well items that
reflect the same construct give similar results. It is often evaluated as
Cronbach’s coefficient alpha. Coefficients of .70 or higher are considered
acceptable. IN the first column of this slide, you can see the range of Cronbach’s
alpha for all of the Neuro-QOL short forms for the different clinical groups. In all
cases, they were greater than .70. Test-retest reliability means that if you give a
test more than once, the results or scores should be the same provided there
has been no change in what you are measuring. This kind of reliability is often
evaluated using Intraclass Correlation Coefficients (ICCs.). ICCS of .70 or higher
are also considered acceptable. In the second column, you see the range of ICCs
for Neuro-QOL short forms per disease. A few of the short forms were less
reliable than we would like.
Now, Neuro-QOL was developed using Item Response Theory (IRT) in addition to
classical test theory methods. IRT extends the concept of reliability by showing
that reliability is not uniform across the entire range of test scores. We can think
of reliability in terms of how much error is associated with measurement. Scores
at the edges of a test’s range, for example, usually have more error associated
with them, and are therefore less reliable.
IRT uses the test information function to show the degree of precision at
different values of scores. This shows the test information function for the
Neuro-QOL fatigue bank. Scores are on the x axis and the amount of error is
shown on the right. As you can see, the fatigue measure is less reliable at the
upper and lower ranges. (Scale score is a raw score that has undergone some
transformation in order to allow scores to be shown on a consistent scale)
This is the test information function for the stigma bank. Again, reliability is
better in the middle range of scores than at either end.
6
Slide 4
The previous slides showed error in terms of scale scores. But NeuroQOL uses a T-score metric. T-scores are standardized scores with a
mean of 50 and a standard deviation (SD) of 10. Thus, a person who has
a T-score of 60 is one SD above the average of the referenced
populations, either the US general population or clinical populations
These are the test information functions for all of the adult item banks shown in
terms of T-Scores. . The shaded areas represent the measurement ranges
where a reliability of .95 is possible. As you can see, these forms are most
reliable in the mid-range of function, generally 1 or 2 standard deviations from the
mean.
Slide 11
This brings the presentation on Neuro-QOL reliability to an end. More detailed
information on the reliability of Neuro-QOL measures can be found in the NeuroQOL final and technical reports, available on the Neuro-QOL website. In the next
presentation, I will discuss the validity and responsiveness of Neuro-QOL. Thank
you! This work has been supported through the National Institute for
Neurological Disorders and Stroke.
Lecture 2e: Validity of Neuro-QOL instruments
Action
Script
Slide 1
Hi! Welcome to this presentation about the psychometric
properties of Neuro-QOL instruments. This is the second
of two presentations on this topic. Here we will focus on
validity and responsiveness.
Slide 2
A valid measure assesses what it is supposed to measure.
So a happiness test measures happiness, not energy. If
we think of it in terms of shooting arrows, then a valid
arrow means you hit what you want to hit, the bullseye!
Slide 3
There are different ways to assess validity. One way is to
show that a new measure correlates with established
measures of the same construct. This is known as
convergent validity.
Slide 4
This is an example of convergent validity of Neuro-QOL
measures for stroke patients. We assessed the
convergent validity between Neuro-QOL and external
measures using Spearman correlations. We used the
following guidelines to interpret magnitude: < 0.30:
Nominal; 0.30 to 0.49: Small; 0.50 to 0.69: Medium; 0.70
to 1.0: Large. We expected weaker relations between
measures of dissimilar constructs and stronger
associations between measures of similar or identical
ones. The pattern of relationships shown here provides
initial evidence of validity. However, showing validity is
an ongoing process and we expect additional evidence as
other investigators use Neuro-QOL.
7
Slide 5
Slide 6
Slide 7
Slide 8
Slide 8 continued
We also looked at convergent validity across diseases.
This graph shows convergent validity of the two NeuroQOL adult physical function measures in terms of their
correlations with the Barthel Index. The Barthel is a
measure of basic mobility and self-care Activities of daily
living.
Another way to examine validity is to see if a measure
differentiates people categorized into groups known to
differ on a similar construct. We used scores on the EQ5D to divide people into no, moderate or severe anxiety
groups. Average T-scores on the Neuro-QOL anxiety
measures were significantly different for each group both
at baseline and after 6 months.
Now we are going to talk about another important
property of PRO measures – responsiveness. Responsive
measures show change when a real change has occurred.
One way to assess change is to ask people to rate how
much they have changed over a specific period of time.
We used people’s global rating of change to categorize
people as better, the same, or worse.
This graph shows the responsiveness of several NeuroQOL measures compared to some other measures given
at the same time. The standardized response mean
(SRM) is one of several available effect size indices used
to gauge the responsiveness of scales to clinical change.
As you can see, these particular measures demonstrated
better responsiveness than some other legacy measures.
This ends the presentation on Neuro-QOL validity and
responsiveness. More detailed information on these
psychometric properties of Neuro-QOL measures can be
found in the Neuro-QOL final and technical report,
available on the Neuro-QOL website.
Thank you! This work has been supported through the
National Institute for Neurological Disorders and Stroke.
Lecture 2f: Selecting a Neuro-QOL instrument
Action
Script
Slide 1
Hi! Welcome to this presentation about how to select a
Neuro-QOL instrument.
Slide 2 – first line
There are several factors to consider when choosing a
Neuro-QOL measure (CLICK)
Slide 2 – first factor
First, what age group do you want to assess? Neuro-QOL
contains pediatric and adult measures. The instruments
can also be given as proxy measures. If your participants
8
Slide 2 – second factor
Slide 2 –third factor
Slide 2 –fourth factor
Slide 3
are less than 18 years old you can use either the pediatric
or parent-proxy versions of Neuro-QOL. Or both. You can
also use the proxy version of Neuro-QOL measures when
a participant, whether a child or an adult, is unable to
complete the measure because of cognitive limitations.
(CLICK)
You should also decide on which domains you want to
measure. Neuro-QOL assesses aspects of physical,
mental, and social health, but the specific aspects
measured may differ for adults and children. (CLICK)
Another question to ask yourself – What mode or modes
of administration do I want to use? Neuro-QOL can be
given on paper, through on or offline computer
administration, or via telephone or in-person interview.
Your choice of mode may influence what Neuro-QOL
instrument type you can use. For example, Neuro-QOL
short forms can be administered by computer or by
pencil and paper. However, Neuro-QOL CATS require
computerized administration. (CLICK)
Finally, you should think about your need for brevity
versus precision. For example, CATs are usually more
precise than a fixed form of the same length.
This concludes the presentation on selecting a NeuroQOL instrument
Thank you! This work has been supported through the
National Institute for Neurological Disorders and Stroke.
9