Joumal of Merr*l Hedrh,
August ?0ll; ?0(4): 136 "346
informa
healthcar€
Selecting *utcoilre fireasures in rnental healthl the views
of service r:sers
effective (Cay e
may be greater
desired outco
Differences ha
about which
srudies that
intervendons
findings, avai
MIKE J. CRAWFORDI, DAN ROBOTHAM2, IAVANYA THANAI,
suE PATTERSONT, TlM lrEA\IERr, ROSEMARY BARBER3, TIL WYKES4,
& DIANA ROSE4
interv*ntions
userr (Hanley
In recent ye
(Trauer et a1.,
I
plays in the
Facuky of Medicine, lmperial College Londot't, Cente ;for MentcJ Heabk, f-o*d<t*, {)K,zMental
Haalth Foundarion, Loadon, IlK,3Section af l\rbtic Heatth, lJnfuenitj: of Sheffiekt, SheJrte&, l"lK, aarl
aHeal* Smtice and Populatbn Research, Kings Oatlege Lottd.on, Lo*tlox, IJK
Stationery
increasingly
Choice of
clinical trials
Atrstra*t
Bachgrotmd. U*le is known abaut service users' vicrvs of mcasures uscd to evaluate trcatments in
mcntal healrh.
Ailli.l., To idenrify the viervs cf people with psychcsis and af'cctiyc disordcr abnut rhc relevancc lnd
acceptability of commonly uscd outcome measurcs.
rldel&rdi- Twtnty-four widely uscd outcome measure$ wer.c preselted to expert groups of srvice
usei.s. H*tniral g*up meth*d* xere used to dcvclop consciftus about thc appropriatencss of cach
$lea$taa, Comrnerts made by seryice users about how ortco:rr:s sheruld be assessed were also
reto:d*d"
fta:*/$. Group
membert *xpressed celncem about :he abiliry of somc outcome mcasurcs !o capturc
rheir experiences, Patient-rated measurcs were as*erscd as more rclevant and appropriate than statl'
ra{ed me&iui!:, and the need tc examine negative as ruell as tbe positive ellects of trearments rl,us
empharised. Specific cot:ccms were raised about some widely used measure; including the Global
Assestment of Functioning and the European Quality of Life scale.
Cotrclusiotts. !(/e eonsider it csrcndal that srrr"ice uscrs' views arc raken into account wher: selecdng
mfasures to evaluate $estnrent outcomes. Providing insight inta r,'icws af users of mcntal hcalth
srrr.ices, our findings serye as a startinEr point for discussion.
Kepvords:
in determining
on rouulle
Viatss, attitwles af ca*su,arer$, usw involaunent, {ruttonlcs, naeasurcntett
Most
views of service
are largely
appropriateness
have been used
disorders, due
health services.
Methsd
We compiled e
$ervice$ foi Pr
resources and
(Ame*can
Mental
megt€res were
PAG *rerbErt
preseat*d to
different expe
Backgrouad
the grolp
The developmeflt of effective reatmeRts for heal:h-related problems re quircs acrive input
fram people who use services (Parridge & Scadding, 2004). One area where active
sen"ice user involvement is f,ssential is in deciding rvhich outcome measures should be
used to examine the effec:iyeaess of new reatments (Entwi*tle et al., 1998; The James
Und Alliance, 20A7). Prer.ious studies have demc*strated differences in the way that
users and providers o'f health care iudge whether interventioss and Seatments are
Conespondence; Mike J. Crau.ford, Readcr in *lcnral l.Iealth Scruices Re'scarch, Deplrtmena of Pslchological lledirine, Faculry
of tr{edicinc, lmprrial Cal]ege l*nelon, Claybrach Cenue, 37, C)ayfxqrok Rotd kndon W6 8l-N, UK"'I'el: .+.0207-3116-12)3.
Fa:: +0?0?'386'I ?16, E-nrail: m,crss{srd&limperial.ac,ul
ISSN 0961-8237 print/lSSn* l360-0567 qalin! f0 201 I Infonla UK, 1.td.
DOtr
I 0.3 I
0q/Og518Z\7,2A1
1.5'7 7 1 t
4
be prolected bv copvriqht law
ffitle
17, U.S. Code
development
and Rethink);
undertaking
worked
up by the
PAG
measure, data
had been ured
outcome
Health (20$8)'
Sen:lce uter views af o*tcowe
informa
,lrcasures
137
heahalcare
effecrive (Cay et al., 1975; Prusoffet al., 1972; Rcr*ru'ell et
: the views
al',
1997)- T?rese differences
mental health, where views about the nature of ill heahh and the
desired ourcomes of trrarment may be more contentir:us (Faulkner & Thomas' 2002).
Differences have been reported between urers and providers ol menral heal:} servic*r
about which outcomes matter most (Crawford et al., 2008; Fischer el al., 2002) and
may tle grester
srudies
in
that have actively involved service users .eport changing the way that
(Irivedi & Wykes, 20*2). Despitt these
interventions were subsequently evaluated
F{ANA"
(3,
TIL
WYKES4,
lon, {JK, Mental
findings, available evide:rce suggests that most $tudies examining lhe effectiv*ness of
interven:ions and rratments are conducted wi€r little or no involvement from service
users {Hanley et al., 2001).
ln recent years, outcome rrlsasuremeat has gained an even great€r level of impofia*ce
(Trauer et al., 2009). In addition ro rhe central role thal ttre selection of outcome measEres
plays in tlre design of clinical tixls, greater emphasis has been placed sn findings fiom rrials
evaluatc trEatments in
ln derermining heal:h policy (Ham et s1., 1995), It has been argued that a grca:e: emphasi:
on routine outcomf rneasurement in clinical services can increase service qualiry (The
Stationery Office, 2008a). In the National Health Service (l*lHS), funding for sen'ices is
increasingly linked to the ou:comes rlrat services achisve Qepartment of Health, 2002),
Choice of apprcpriate ourcomes is therefore of majcr importance to bodr rhe quality cf
clinical rials and to efforts to improve rhe qualiry of mentalhealth services,
Most outcome me&$ures have been developed wirhout dire* input from serrice users and
rout the relevancc and
are largely unknown. $i/e therefore set out to obtain views of service users abour lhe
perr groups of tcrvice
t"r.
2
2ld, Sh{frteV,
rpropriateness
UK, and
of
each
re assesstd were also
gie,ns of service useffi about rhe appropriateness af outcome rneasure$ t}lat are widely *sed
appropriateness of rvidely used oulcome measlres. 11Ie focussed on outcome measures that
Ueen used {o evaluate interv?*tions and seririces for people with psycha:is and mood
disor6ers, due to the prominence of these forms of mental disorder among users af mental
heal*l srrvices.
e mcasufes to caplure
rppropliare than staff-
cts of treatments lvas
including the Global
:count rvhcn s*lecring
sers of mentai hcalth
rfienl
lqurres actlve rnput
area wh*re acdve
reasures should be
,
1998; The James
:s in the way that
rld reatments are
ro)ogical il{ediciae, [:aculty
lK. Tel: +020?-386-121i.
Methcd
$(e cornpiled a list of outcome mca$ures t]rat have been used to et'aluate treaffnen* and
services fcr pegple wi{r mood disorders aad psychosis by searching lexiboc}ks and online
resources and idendfoing measures used ir: published and ongoing clinical tri*ls
(American Psychiayic Associarion, 2005; Groth-Marnar, 2003; National Institute for
Mental Healrh, 2008; saia:ovic & Ramirez, 2003; Thomley & Adams, 1998). These
measures were presenrccl to a Proiect Advisory Group (PAG) for discussion' Views of
pAG rnembers wete used to draw up a shortlist of outcome measures that were thea
presen:ed to expert groups of service use*, The P.{G had nine mernbers with a range of
iiff"..n, experiences and qualifications: three were full-time res€archrrs; two membert of
the group wolked as clinicat acadernic$; two member* wcrked ia researh and
deuelopmint within voluntary sector organisations lThe Mental Heath Foundation
and liethink); and the lasr two member$ were selice users with experience of
All members of *re group had
Sen/ice User Involvenrtnt se1
on
Group
Research
part
of
a
Clinical
tvorked toge*rer as
lrletwork'
Research
Heal:h
Mental
up by the
pAG members were presented rvi:h a document that provided a descripdon of *te
rleasure, data oa reliabiliry and validiry, and informatioa about how often :};* measure
had been used in previous srudies, w'het}ler it had been included i* recent gilidance on
otrrcome measuremenr i* mrntal health published by rhe Narional lnstitute for Mentql
F{ealth (2008), and national guidance o* the treatrlerrl of schizophrenia and depression
underraking research and service development activities.
318
M. J. Cratufunl
et *1.
(I.Iational Insdrute for Clinical Excellence, 2002, 20Cr4), PAG members were asked to
consider the f*11owing factors when shortlisting *re rnea$ures:
l.
2.
3,
during the
outc0rne measu
aspects
The frequency with which rhey have been used in previous studies;
Evidence of their currentiy being used in ongoing reeearch and seruice evaluations;
The psychomerric propenies of the measure.
Whenever more than one version of the same outcorne measure was available, we used the
one most frequen:ly u$ed in previously published srudies,
Expe*
graups
Group members vrere recruited through the Menral Health Research Network, four nqrianal
asked fcrr volunteers who had :rsed
$ec*ndary care mental health services snd hsd direct experience of mood disorder and.lor
pychosis. We aimed to recruit between 8 and l2 people to each expcrt group ancl to ensure
representation of men and women of different ages and *om di{Iereat ethnic and cultural
backgrounds. Previour iavolvement in a resEarch study was considered helpful, but was not
a requirement io become a group member" The firsr group meeting was set up to examine
outcome rfleasures used in studies tn mood disqrders and tl:e second focussed on psychosis.
In *ddition to examining measure$ to &ssess mood, :he firs: graup also assessed general
rneilEl health measures and measure$ assessing social fuacdoning, The second group
exa$ined instrumeRg u$ed to assess side eflects cf rnedication and qualiry of lifb m*asures
in addition to those specifically designed to ssse$s pslohotic syrnptoms.
Potential group members had often experienced both mood disorder and psychosis and
chose which group they wanred to auend, Members of the group were sent wrinen
inforrnarisn prior to the mceting" At the start of the meeting, group members were asked to
provide wrirten informed consent to participate in the srudy. Mt-'mbers of rhe expert groups
werc paid {100 for t*kiag part in each dayJong meeting. EJrical approval rvas obtained
prior to the srart of the study from lnperial College Research Ethics Csmmittee.
We ccmmenced the expert groups with s general iatroducrion in which we presented the
backgraund and ailrts of the study and described the format of the me*ting. We rhen asked
group membe$ to complete a short quesdonnaire rhat sought inforrnation relared to
demographics, use of mental health services and previous involvement in research. trVe used
a nominal group tec}nique flones & Hunter, 1995) ta try to develop consensus sbour the
apjropriatenets of the differen: outcome measures prcsented. \(Ie slarled by presenting each
oxtcotne mea$ure by prcviding s short surnmary cf r}te backgrour:d and aims of the
measure, what it aimed to assess and how the measure was completed. lfe gave earh group
member a copy of tht measure and &en asked n:embers to lead rhrough them and make an
i*iti*l radng of dreir appropriateness on afl li*point Likefl scale, Each outcome measure
was rhen disc*ssed and feedback on the mean respofise sf th* group in round I was
provided. Members were then asked to rate the appropriateness of each measure again, in
light of the initiai mean group ssore. During each group, a service user rescarcher kept
a wriren record of key discusrion points,
men$l he*l& charities aad :wa local user groups. \Ve
of
Results
Iiile identified t
and mood di
the expert
excluded
therefore incl
(Tennant et al,
Twenry-fivr
group and 15
Participants
lemale and 14
themselves as
as other. All
services, with
evaluarioI1.
Median r*d
rauns exercr$e
(as indicated
rating, eight o
rating of 3 or
ofthe expert
measure$ that
No measure of
Comments w*de
Pnssible
from an anal
A reries af fea
measufe.
for outcome
and in son:e
i
researchers qf
sured that
they were
rnight be com
Recovery Star
experiencer of
Several
Data
Concerns
analT,sis
expressed
A summary measure of the expert group$, initial and fiaa1 ratiags for each outcome measure
was calculated using the median radng and the interquartile range. Quatitative data collected
t law (Title 17, U.S. Code
h
Tweaty-two
(EQ-$D),
outcome$ they
'itr
g:
=
Serrrice r$er uiew$ of outcotne
nbers were asked to
trcasures
fr
339
during the group meetings were used to exarnine views of serrice usets nbout the use af
outcome measures, to help interpret the quantitative data we ccilecred and Io describc
a$pects of outcome measures that service users believe m*kes :hem appropriate.
al
*e:
=
il
Iii
ies;
senice evaluarionsl
vailable, \.{,e used th€
etwork, four national
nteers who hsd used
tood disorder and,ror
. group and to ensure
t ethnic ard cultursl
. helpful, but was not
as set up to examine
|cussed on psychosis.
rlso assessed general
The second grcup
ality of tife measures
s.
:r and psychosis and
)
were sent wdfter
:mbers were asked to
of the expert groups
proval was obtained
lommittee.
ich we presented the
:ting. \Ve dren asked
brmation relate.l
to
ia research. lVe uscd
consensus about the
d by presendng each
td and aims of
the
We gave each group
h them and make an
.h outcorne measure
up in round
I
was
:h measure again, in
user researcher kept
;h outcome measure
itative dara collegcd
Rssults
!fle identified I 32 ourcome me[rures that have been use d in studies of pcaple with psychosis
and mo*d discrders. PAG members agreed a shorrlist of 22 outeclmr measures to present to
the expert groups, Howcvcr, PAG members were concerned that our search silategy had
exctuded measures r{eveloped in conrul:ation with senrice users but not in common u$e. 1Fe
therefore included two gdditional measures - the Watwick-Edinburgi Well-being scale
(fennant et al.,20O71and thc Rccovery Star (MacKeith & Burns, 2008),
Twenty-ive people arendrd the two no:ainal groups. Twelve pe$ple att*nded the first
group and 15 attended the second (rwo of thery had also participared in tl:e first grnup).
Participants raaged ia age from 26 to 66 (mean:44) years, Eleven (44%) participantt were
female and 14 (56%) were male. In terms of ethnicity, 15 (60%) parti{ipants idenrified
themselves as Brirish white, 5 (20%) as Bri:ish black,3 (I2%) *s 3ri:ishAsian aad ? (l%)
as other. All 25 panicipants had previous con{act with recondary care mentsl heslth
services, with most (N:22,88%) having had more than l0 year$ contact with servic*s.
Twenry-two (88%) parricipants had had some previous involvement in research or sen'ice
evaluation,
iii
{}
Median raricg$ for each outcome measure (with interquarrile range) in the initial and fin*l
rating exercise are preseored in Table I. l*vels of agreement were higher for the inal radag
(as indicatcd b1,a smaller intcrqdartile range for most outcome measures). After thc final
raring, eighf outcome measures had a median score Of ? or more and five had a m*dian
rating of 3 or lower. The highest ratings were for side elfecrs of medicarion, and rnembers
of the expert group ol mood disorder stated that they would have liked the chance to see
rneasures that had been used [o as$ess side effccts of mood stabilisers and antideprf,s$aflts.
No measure of social functioning achieved a ${orc higher rhan 5.
Comntertts made
tu
b1t
expert Eroup mentbers
Possible reasons why individual outcome measures achieved the ratir:gs they did emerged
fiom an analysi$ of the qualitarive data that were coll*ctsd during ttrese two meerings.
A series of features were identified as contributing t$ the appropriatenes$ of an ourcome
measure. These are summarised in Table II. Group members expressed a ttrong preference
for ourcome measures that src rated i:y scrvice user$ tlemselves. They expressed surprise,
and in some instances disbelief, rhat outcome sleasur€$ based entirely on the judgmen:s of
researchers or clinicians could be used to judge a person's response to lreatment. Oth*rs
$1qted *rat people may not be able to rnake ratings without help from others tt dmes when
they were most unwell. Some group members supported rhe idea that outcomt mfasures
migh: be completed by service users working together with staff (when completing the
Recovery Srar), although otl'lers felt that oulcome measures should be based entirely on the
experiences of the service user.
Several discussions rookplace about the length ofquestionnaires or a$sessfilenr schedules.
Concerns were expressed at the length of some measures, but greater concerns \#ere
c-rpre$sed about very short outcome measures such as the European Qualiry of Life scale
GQ-5D), wtlich group members believed were too short to proprrly assess the complex
outcomes they are designed to measure. An exception ts lhis was measures of side effects of
I
I
M, J.
o
nt
d
Crav,tford
d
et al.
h
^^^^?n
Ff F?,4
r f Ef naTir
?+cisn+
rhnh$s
u
.E
6@nnNclelaohNrra
hr$ho6fi6F€nQr
r
r
(
r
B
?f
s
?9c ?t c??
rnrnfr +6
mnn6n
o\O
Oe+rnrS
h
z
Is comp*hra:il*
Avoids qu*d*ns
nr\6t^
r
A6-u_1r-€6llra
r
a.,i
.tu
r Lnn€?
srn
rrh
i, E
E E*
r
Y'4 sLLNL
lE=lE=o n.i
aln
! 01
:[*]5t
!rH;rE
IE.EEET ErE#E{i{EE;r:i
!22.ai ;n*{'2l,fr: EsiiE
d
-
SrErrI
!
5E€EEP€EE€E€l aala:E
TEEEE:E TTTE?6JETT?I= :EB-EE
Trtr€:
TTTTE
ET E41i1,r r I !€
p a'"
c'd,
s c r'n
c c c c c'c
c
c 2 t c.2
E
=
UOEUUU
,
H
:
UO&trEuUUzUEu!
H
i
-
--a*oo
cl 61 6-nrn
H
H
:
O,JOUO
++NTiNci€iGi-*
-
=
-
-
F
H
Y
H
-
h
\IU
G
:
!-*;
^A^
i{i *
@
-
-aErre--N--On+
ArB
Bq
UF
coo.Ee!.E
+;ariJO
-
ei
?
c
ie
-Ptii tigf?i?{g iii i t?
j
Ftt ga;,I?E?i E*ir# ; rar' l
c
a
E
c
ifrEEa f f A iTiuut !F; fri,ElUA [€;
*e! fi
AE€E\E?frt tl'; A*;t
N
il|iii
o
U
,E
ff * ig ; ri H s I s EE I Ig5Es$gs3E I
g.
.?
il
a.2u
:UU
q13,"ZH
'rc.i)
U
z
|-.
tJ
for,
Should be tsed by
me[sure
rnedicarirn,
range
of
Anarher
'negadve'items,
upse$ing to br
health.
questions about
not endorsiag
In both
for a 'good' ou
functioning. Fcr
judgements
time indica*d
Varying views
Some group
purpose 0f
measure$ such
willingness to
imponsnt m
Concerning
was not s$
much as
tht
Sensitivity to the
and respect were
the intportance
cEY6-ES
Discussion
!.8!=*lu
14b!I-3€
O,iO,iO6O
By usiag
services, we
17, U.S. Code
a
be asked
valued the i
trtr
o:Eq?qt
'i-s;G;*E
-j
!
I
tie
complet* poor
made ti:em
=i
=fi
::iE
6
Makes not* of
Includes space
Some group
3
Ei
a*
*
assumed rhat
because some
fi
Er$ #^r_a?s{;6ff;
qf
Itearures
lncludes'poritivcn
n
d
ll.
Should be based o:r
6r
dh
^^^q^t
accY?4
af99?n?9fff,r4
crPff
333399'cgtousr!,o59tr9g
3gr]9cl
6
Tatrlc
u$er I)i{t$s
ITYTY
1tnot6
1nnmfr
]e
I
Table
ll,
mca$tr{es
Fsatures of an appropriate oulcome me8su.€ according to gtoup mettltlcrs
Should be bared on Faliena rather than staif-rated judgemetrs
Includes'positire'
r'€ cc h
trl
+alral
nn€$
n
ol oattofie
as
well as'negadve' iten:t
Is comprehensive * aeither too lolg nor loo shcn
Avoids quesrions tlral *re iarrusiv* abor: private issu*s such as sex life
Males note of r]:e time and place rvhere the outlome is mcasured
lncludes space for'added comments'
Shoulj he used by staffwho have good interpenoaal *kills and have been properly trainetl in the u*e ofthe ou{come
measul€
If self completed, is presenred in * professional manner
a.
,Z
=!
iBEs
-t!trL
.d;t
E
E:
!!
c;4fUU
il dtr.!
,?^ef^
rt)
z^
!i{
IE
CO
^uc
5 a,/
lEc
icu
-k
=xua
'r='e
tr
;. ul; i
t;i€?
&-i0
n:edication, which group mernbers fel: needed to be long in order to firliy capture the wide
range of tregative effects of treatment$ that group member$ had experienced.
Another rrcuxrent theme concerned the dominance of what tveaf, perceived as too man5'
'negative' i:ems in most questionnaires. Many group memb.rs remarked drat they found it
upsetting to be asked long lists of questions about difficulties associated *"i& menlal ilt
health. Conversely, Th+ Warwick-Edinburgh Well-being scale was commended for *sking
quesdons abour e$pects of good rnen:al health, whers poor emotional health is indicated by
not endorsing *lese'posi:.ive' items.
In both group$, member$ raised concerns about ouacome measures that set out crilfxia
for a 'good' outcome. This *pplied panicularly to measures of qualiry of life and sociat
functioning, Fcr ins:ance, some grcuF members expressed the view that it should not be
assumed thar people who gor on well with family rnembers had better social funcdonit{'
because some people made a coiircious choice nst to have con:act uith family mimbers.
Some group rnembers coilsidered it inappropriate that measures were graunded in
judgements about which aspects of relatioaships with others or ways that people spe$d their
rime indicated a'better'ourcome, Group member* suggested, instead, that people should
be asked wherhcr people wete happy witl these aspects of lheir livesVarying views were expressed abour inclusion of items on sensidve matters) such as sex.
Some group members srared rhar they would not be happy to answer such quesxani for the
purpose of assessing outcomes and considered the use of intrusive' questions in outcome
measures such as *re Social Funcrioning Questionnaire likely to impact on pesplc's
willingness to take part in research. Others disagreed artd felt that issues such ss sex were
important to qualiry of life and needed to be included.
Concerning the process of outcome rneasurement group members repeaiedly staa€d that it
was nat so much the content or length or ar interview or questionnaire t}lat mattered, as
much as rhe personal qualities of &c clinician or researcher conducting the assessmrtt' As
one group mernber $Bted that an outcome r*easure was'only as goad es the drlct$r who uses it'.
Seasitivity:o the needs of people experiencing mental disuess and being reated wirh digairy
arid respect were seen as pardcularly important. At a practical level, group menlbers suessed
the imponance of presentarion of documents. Reporting experiences *f being asked to
complete poor photocopies of questionnaires, gtoup members sm:ed that poor presentatioa
made tlem wonder about how seriously their respolses would be take* or whether services
valued the information they were being asked to prorride.
Discussion
o
.l
l1
o
o
:.'.
By using nclminal group melhods with expeft groups comprising users of mental health
seruices, we have genercted data on service u$er pempecrives on the appropriateness of
ri
'i ri
a:i
at
+
tii
347,
M. J. Crar*ford et al.
widely used outcome measures, G;oup members wcre clear thar the outcomes of
interventions and teatments needed to be assessed uring paticnt-rated measures, ?hey
of cornrnurly
mental healti
highlighted the importance of assessing negadve as well as posirive effects of intervendons
and ftearmenls and were critical of outcome rneasurs$ that presented people wirh only
negatively phrased questions. Many commen$ made try sen'ice users concerned the
acceptable
process
of collecting outcome measures rather lhar the contrnt of individual msacures.
The personal qualiries and training of staff using queetionaaires and checklists were
highlighted as imporrant to the experience, as were the presentation and format of
Warwiek-Edi
of m*ntel
The 24 *u
que6tionnaires.
Alttrough measures of medication side effects assessed by &e group rvere all relatively
highly regarded, ir is of concern *tat some of the most widely used outcome rneasures in
mentn: health, including the EQ-5D and the Clobal Assessment of Functioning received
very low ratings. ln addition to being staff rated, the Global Assessmenr of Fu*crioning
it made ;*nsr
measures fur
was criticised far conflation of funcdoning wift mental heaith in a single score. Sorne
group membf,r$ lommented that there had been times when firnctioning was r*asonably
good despirt poor mental health, a::d others reported continuing to functir:n poorly cven
after symptoms of emotional disress had subsided. Group members quesrioned whether it
rvas posrible lo as$ess *ese outcomes with a single score. Tt:is concern has also been
raised by clinicians a*d researchers using the Global Assessmeat of Funcdoning (Wiiliams,
apparent
chance.
schizophreni
quality of life
would have
Finally, the
did not use
measurement,
place al thr
help develop
mental health.
2005).
[ntplicatiotx of
Study stte*gths and limit*tions
Although 1v€ were able to include many measures iaxommonly :sed ro a$sess outcomes of
people wiir mood disorders and psychosis in studies in rhe UK, many qthers were not
included. Moreover, several of tle as*essed outcome measures are available in differing
farmats. Iilfe attempted to use the versions of these outco:ne me*surfs that have beea most
wldely used, for example, we asked group members to cansider the appropriateners of the
l2-irem General ilealth Questionnaire, rather than *re 28- or 60-item version of rhis
questionnaire (Goldberg & liflilliarns, 1991) and the 34-item version of the CORE-OM
but do not knaw how different versions would have been rated (Evans et al., 2002). ln a
timilar vein, u'hile a self-co:npleted version of Psychotic Symptom Rating Scales is available
we presented group members wirh the :taff-rated version. Given the preference of group
members for patierrt-rated outcome measures, the self-completed version of the measure
may have received a higher a:edian ratingThe criteria establirhed for develaping our shortlist were designed to ensure that we
examined o*rcrlme measures that have been widely used. This meant that meesures that
have only receody beea developcd wers gcrterally excluded. An enfortunate consequence of
this was that a range of recently developed outcome measures that have been developed in
parmership with service users were excluded (Al1ott, 2005; Rose et al", 2009). ?he only
measure that we examined whlch nrade explicit reference to incorporating views of *en'ice
users during its devei*pment was Ltre $fanvick-Edinburgh riilell*being scale flennant et al.,
200?), It is :herefore of note that d:is outcome measllra *chieved one of the highest rating by
grorp members"
Although differences in the median level of appropriateness of outcome measures were
seen ia this ::udn we have not used starisilcal tests to assess the srengrh of rhe difference,
This is because the study was no: d*signed !o ary ro make decisions about whether a
particder ou:{cmr measure was 'appropriate' or nat, but to pror.ide a dercription of rhe
views of people with psychosis and affectiyc disorder about the relevance and accepmbility
th:
One of
rated
ourcome
been raised
healrh
on staff-rated
(HONOS) (F
rated ou
challenged
measured
suengrhen
Ongoing
(Irauer &
Our findi*g
accords wi*r
beyond tide
psycholcgical
would have
stabiliserr and
as
well
as
2008).
Group
they were
health care
reported
(Blount et al.,
residendal
Service user rirl.tls of outwme
hat *re outcornes of
rated measures. They
:ffects of interventions
ited people with only
users concerned the
: individual measures.
and checklists were
tadon and format of
)up were all relatively
outcome measures in
Funcdoning received
sment r:f Fulctioning
a singlr score. Some
,oning was reasonably
function poarly even
q*estioned whether it
oncern has also becn
'uncdoning
flVilliams,
rlrcd.sar${ 34}
i*t knolr whether users of
mental heakh services iudge the Warwick-Edinburgh Weil*being lcalr s more relevant and
accep:able a$$etsmenr of general &ental heal:h than the General Health Questionnaire, and
apparenr diflerences in the medign ratings {sr these two measures coald have aris*n by
chance. However, we did find that service users who took part {n the expert group$ Iiked tht
Warwick-Ediaburgh Well-being scale and valued the way that i: asks about positive aspects
oI mentaI hcalth.
The ?4 outcome measures we assf,ssed were dir.ided between the two expefi pa:lels. $(hile
it made sense ro ask people rvith experiencc of being treated for dtpression to rate oiltcome
rneasures for mood disorders, and for people with psychosis to ret* outcorne measures in
schizophrenia, either panel could have been asked to rate ge*eric outc€me measures of
quality of life and social funcrioring, erc. $7e do not know whether ratings of the*e measures
would have differed had they been presented to a different panel.
Finally, the quali:adve data we collecred were in the form of conte$poraneous notes. We
did not use formal q*alitative research merhods to explore service user views about sutcome
rgeasursment, While we werf, sble to capture the maia themes of the discussions thst took
place ar the two rnf,edngs, we believe that future qualitative research should be conducted ro
help develop a L,elter understanding of service user views sf outcome meail:temetll in
mental health.
of commonly used outcome measures. For inttance, we do
Implicarions
oJ
fi.ndings
One of the clearesr findings of the study was :he pref€rerce of group members for patient-
to assess outcomes of
nany others were not
available in differing
s that have been most
rppropriateness of the
t-item version of this
rn of *re C0RE-OM
tns et al., 2002). In a
ting Scales is availsble
: preference of group
:rsion of lhe rneasure
ed to ensure that we
a{ lhat measures that
unate consequence of
we been devcloped in
al., 2009). The only
'ating views qf service
scale (Tennant et al,J
'f the highest rating by
icome measures were
rgth of the difference.
ons about whether a
: a description of the
rnce
ard acceptabiliry
rated outcome measures. This'Snding cofluasts wi& extensive re]iarce on staFrated
ourcome mea$rre$ seen in mental health services (Gilbody et al., 2002). Coacerns have
been rais{id about :he reliabiliqy and validiry of parie::t-rated outcome measures in mental
heafuh (Becker rr al., 1993), and current efforts to examine $ervice quality are often centred
on $taff-rated outcome measures such as the Health of rhe Narion Outcome Scale
(HONOS) (Fairbaim, 200?). This contrasts wirh other areas of health care where padent*
rated outcome mea$urcs are the norm. The service users rvho took part in dris st*dy
challenged the idea thar the qualiry of senices or outcomes of interventiotrs c{}uld be
measured appropriately by using staff-rated mea$ure$ and we believe that :hese data
strenglhen the argument for using patienr-rated outcorre measures in mental healrh.
Ongoing research to develop and refine a patient-rated HONOS is therefore t$ be welcomed
(Traaer & Callaly, 2G02).
Our findiag that serrice users are conccrned with assessment of side effects of trtatment
acc$rd$ with o*rer studies Qxe et al., 2000; Shumway e: a1., 2003). We extend this however
beyond side efiecrs of medicarion to include examination of the negadye tffects of
psychglogical treatments. Members of the group focusred on rnood disorder srated that drey
rvould have liked ro have seen an outcome measure assessing the side eflects of rnood
stabilisers and antidepre$sar:ts. Tbese findings rupport the inclusion of measures of ncgadve
as well as posidve effects of in:enrentions and treatment used in mental health (Rose et al.'
2oo8).
Group members were also cortcemrd abou{ the length of maay of the outcome r:leasures
they were asked to exBmine. Similar concerns were aiso raised by lay peopltr patients and
health care professionals who took part irl a previous survey of rhe acceptability of selfreported oulco{le measur€s used to assess the health of people with personality distrder
(Blounr et al., 2002). In tlat srudy, 52 people including ? lay people and 14 ex-useffi of a
residenrial reatment service for people with personality disorder were asked to commrnt on
744
M. J. Crawford
et al.
frve outcome measure$. lfhile shorter mea$ures were generally rated more positively than
longcr ones, both 1ay people and professionals who took part in this study critisied the nvo
shortest outcome messures as 'too simplistic' (Biount et al., 2002).
Concerns have lcng been expressed about the validiry and uti1iry of rvidely used measures
af qualiry of life (The Stationery Office, 2008b), While outcome measures such as rhe
EQ-5D have bsen specificaily dev*laped to allow comparison across medical settings,
members of our expert groups expressed the view that this measure rvas insufficiently
de:ailed to capture the quali:y of lile of people with mental disorders. They preferred Ionger
measures such as rhe World Health Organisation * Qualiry qf Life OfHO-Qol) rhat includes
items addressing a broader range of factors iudged by seruice r:ssrs [o be imporant when
assessing quali:y of life.
Group members rated measures relating to social funcdoning particularly poorly, Ssrvice
users had ideatified social functioning as a dtmain that was particuiarly important to assess
when iudging the ou:ccmes of interventions and treatment$ (Perry & Gilbody, 2009).
However, group members were cridcal of the norma{ve narure of drese measures, Group
members told us *rat rhey wanted to be asked abour whether they were satisf,ed with
different aspecls of their social funcrioning rather than having assumptions made that fewer
contacts with others or inyolvement in fewer occupational actiyities was necessarily an
indicator of a poorer outcome, Some group memb€ni also raised coocems ab+ut inclusion of
'intrusive' items such as questions about sex life in such outcome measures. $fhile there are
already a large number of outcome measures used in mental health, we believe thgt these
findings suggest that a measure of social functioning that focuses on areas of funcricrn thar
$ewice users consider appropriate and relevant and does nct use normative standards is
t*
rcquired.
"-
Asked :o assess varjous outcome measures widely used in studies of peopie with psychosis
and mood disorder, users of mental health services considered some to be more appropriate
than others. Orr da:a empha$ise the imporrance service u$er$ place on padent-rared
*ulco'Ile measures, Service users who took part in our rxper: gloups srated that studies
should exqmine t}e negative as weI as the positire effects of interventions and treatrnents.
Concems were raised about some widely used ourcome measures such as EQ-5D and
Global Asees:ment of Functioning (GAF) and about all the four measures of social
fuactioning thrt we examined,
Acknowledgments
are grateful to Angela Sweeny, Alison Faulkner, Joanna Fcx anel Jayasrec Kalathil for
their help with data collection and analysis er:d to Janey Antoniou, Neil Armstrong,
Valerie Baker, Terry Bo*ysr, Humphrey Greave*, Michael Knight, Gary Molloy,
Augusto Mon:eiro, Ros l'{ewnham! Graham Peacock, Kay Sheldon, Roger Smith,
Caroline Thom*s, Jennifer Trite, l,auren Wright and othrr$ who aftended the expert
uaitr,Bi:r
A]lott, Ir. {2005).
f^tanuworlt ard
Amerjcan Psychiarri*
Becker, M., Diamond,
persons wirh
3louu, C., Evans C.,
Beyond psychomr
Cat, 8.L., Small, W.P
ulcet. Lrrncet, J,
Crau{ord, M.J., Price,
bsscd services
far
Department of Health.
finrwisde, Y.A.,
health research
Svans, C,, Connell, J.,
slandnglised brie{
P$,chiorn, ,firil
Fairbairn, A. (2S0?).
I\vchiatric'l:'
Faulkner, A,, &
1go,
t*7.
Fischer, E.P.,
treatment of schi
Gilhody, S.M.,
Nario:ral surtey.
Goldtrerg, D., &
Groth-Mamat, C.
Ham, C., Huntcr, !,J.,
Haaley, 8., Trueldal*,
conducdag, firdri
519-523.
Corclutions
lfe
References
gI{tup meetings.
The study was funded by a proiect grant &om the Mentai Health Research Network.
Declaration of interest: The authors report no conflicts of interesr. The authors alonf
are
responsible for the coDrent and writing of the paper.
be protected bv copvriqht law (Title 17, U.S. Code
Jones, J.,
& Hunter,
Jorma!,31],,38i"
l,ee, 'I'.T'., Ziegler, I,
for health
MacKeith, J., &
National Institute for
'l'reatmant and
Nadonal Instirute fr:r
Carc. London:
Naticnal lnstitute fot
Partrid8c, N., &
prioriti* Ibr
Perry, A., & Gilbr:dY,
delieloprjtt$t cEe
Prusoff, 8.A., Klerman
repofi
rn
Rose, D., SweeneY, A.,
mcrsure of
Rose, D., \{'}'kes, ?.,
cognitive remcdir
I{l,ulrrcrnc
Rothwell. P.&{,,
sectional studY of
Medical ioumttl, Sl
Saiaroric,
M.' & Rami
Seruice user uians oJ outconte
I more positively than
rrudy critisied the two
widely used measure*
measures such as tle
'oss medical settings,
ure was insufficiently
They preferred longer
{O-Qol) *rar includes
to be irnportant when
:ularly poorly, Service
ly important to asses$
y & Gilbody,
2009).
lese measures. Gr<lup
:y were satisfled uri*r
tions made that fewer
:s was necessarily an
:rns about iaclusion of
sures. l*'hile rhere are
we believe that *rese
areas of function that
ormative standards is
References
'l'hearv,
Allrrrr, P. (2005). Recover-v. In D. SaUah & M. Clarke Gds,), /leseari:A antl !,k**!op*uttr in Mental Hc*ltlt:
Elsevitr
Science.
Oxford:
?'rartsu.rrk und L{rvlelr.
Arneri*rn l.,sychiatric Association. (20$5). Hatdioo h oJ Ptythiad,; Measurcs. Washingron DC: APA.
Becker, *1,, Di*mon{, R,, & Sainfort, F. (19S3). A new patient focused indcx frrr measuring rlr:glity of lifi: in
persons with severe and persistenl menral illness. Qualfui of l-ifa flarearr:lt, 2, 219-2rl .
Blount, C., Eyans C., Birch, S,, Warrel, l:,, & Nonon K, (2002). 'fhe propertics of sell-rep*rt rese.lrch rarssures:
SeSourl psychomerrics. Pilcim/a,gy and Pryrhotherapj': 'l'heral:*, R*torch and ltracil:, TJ, 151*164.
Cay, E,.L., Srnall, V.P., Neilson, I., & Henderson, A, (l S75). Pati*rts assessrnent of the result nf lurgery for ptptic
ulcer. lartcet, /, 29-3:.
Crawford, M.J., Price, K., Rutter, l)., Moran, P., T1rer, P., Bateman, A., et al. (2008). Dedic*ttd comlnunirybased services for sdults *ith trrersonalit-v clisorder: Delphi study. Rirish Jutnut aJ Pslthiaul, l9j, 742'143.
Departmenr oi Herlth. (2002J. Rej;::rrirg:\Ty'S Iiruarcic! l;'lows: ]ntroclucing Pcyrtert by R*uJls' London: $oH.
Enrwistle, V.A., Renfreu,, M.J., Yearley, S., Forrester, J., & Lam*nt,'f. (1S98). I.ay perspec:iver: Advantager Ior
health researeh.
Psyrhiatic'lizantent,
Faulkner, A., & 'Ihomas,
5 1
i Jayasree Kalathil for
ou, Neil Armstrong,
{ght, Gary Molloy,
Idon, Roger 5mith,
anended the expert
tesearch Nenvork.
Ihe authors alone are
1
3, 1'6,
f.
Consensus methods for medical and health sertices reseerch. Eitisk Medkal
& Hunrer, D. (lSSr).
lountal,3ll' 180.
l-ee,
EQ-5) and
of social
3!6,463 466.
S-523.
lace on patien:-rated
ps stated that studies
tions ald treatments.
measures
hledbal Joumal,
(2tl02)- Uscr-led research an.l eaidence-based medicine, {Snrish Jountal of l?4,chiatn"
.180,:*1.
Fiscl:er, E.lr., Shumwa_v, M., & L)wen, R.R, {2002). Priorities of consumcrs, prnviders, and family:ner:rbers in lhe
trca tment o f schi:ophrenia . Ps-vc h i n ti c Scnar:cs, -iJ, 7 Z4-7 ?,s.
Gilbody, S,M., Hous*, A.0., & Sheldon, T.A, {3002). Psychiarists in the L:K dc nnt use outcome! measures.
National sun'ey. Snr*trr -/o unul of I'stchiarn', 180, I 0l - 103.
Gotdbug, D., & Villiam:r, p. (1991). A Llscr's Guide to thc Gm*al Health Questionaaire. 'Windror: NFER-Nelst:n.
Grtr*r-Mamat, G. (2003). Handbook oJ l\*lwlagical Arrerrixerl i:loboken' NJ: John Wiley & Sans.
Ham, C., Hunter, D.J,, & Rotrinson, R. {l S95}. Eridence based policlmaking. Srjrri/r .tr{"t/rral Joumat, 310,7 l'7?.
Hanley, fi,, Truesdale, A., King, A., Elbourne, D., & Chalmcrs' l. i2001). hvolring consumers in d*signing,
conducdng, and inrerprering ranrlorniseil controlled rials: Questionnaire turvey, Bntis* Medlrrtt.ftnrrnal, -122,
Jcrnes, J.,
such as
tsnti,,sh
Evans, C., Connell, J., Iiarkham, M., Margison, I;., McGrath, C', ,\lellor-Clark, J., & Audin, K, (2002). Towsr'Js a
sranclardised briel culcome measure: Psychometric properties and utiliry of ttre COI{,.E-OM. ^Bnaii Jtrrnra/ or/
I,q,clli1 1t1,, 180, 5 1 '.50,
Fairbaim, A. (2007). Fayrnenr by results in menral h*alth: The cu:Tenl $ute of play in England. Ad**nces in
people with psychosis
: be more appropria:e
r
rrre&sares 145
T.T., Ziegler, J.K., Sommi, R., Sugar, C., j!{ahmoud, R., & lxnert, I-.A. (2000). Comparis*r: of prefererees
for hcrlth outcomes in sclriztrphrenia among st:keholder 6iroups. Jaunnl dltqrhiturit Rt:earch, -?4, 201-210.
l!{acKeith, J., & Burns, S. (l0t}8). *lenral Hedrh Rerrrrcry Srar. l:rndon: Mentsl He*ltli Prtrviders Forum.
Narion.tl lnsdrure flor Clinical lxcellence. l2OO2). Chtictl 1-tridtline L Stkizophrai*: Cox lnrt:tz'entitlns in tha
'I'reutnunt cnd Mauog$ntrn oJ Schi:ophrcnia h }>iruary a*J Jicardrlnr Care. bndon: NICE.
Narjr:nal lnsrirute firr Clilical Excellence. (2004). Deprrssr'on: lllanagemext { !}epr*tion in Pinary ard Secotulu\
Caru. I-ondon: NICE.
National lnstitute for l{ental Health, {200S). Al,:nral tlealth Auuot ttt ()nnpendi*nt. l:lndo*: NIiVIHE.
?artridge, N., & Scaddi:rg, J. (2004)" The James I.jnd Alliaace: Pa:ierrs and clinicians should joi*tly idendft'lheir
prioritics for clinical *tals. Lancet, i64, lq73'-1924.
perry', A., & Gilbody, S. (Xlo.q). User-defined orrcomes in menfal healrh: A qualirative study and cotrsensu$
dovelopment exercise- Jtrarral tl fuIcntal Hcdtt, 18, 411-421.
prusoIl, 8.A., Klerrnal G,I.,, & Paykql, E,S. (1c72). Ccncordsnce bet*'een clinical asssssments and paticnts'sclfreport in depression. Archfuses o.f Gtn*a! Ptyhiutry' 2,54{,-152.
Rose, D., Sweeney, A., Letse, tr{., Clemenr, S., Jones, I.R,, Burns, T., et al' (2009). Developing a uscr-ge:rerated
mcasure of contiauity of care: Brief repaa. Aua Pq"chiatrica Scandinauica, 119, 1?a'324.
Rose, D., Wykes, T,, Farrier, D', Doran, d'-M., Sporle, T., & Bcgner, D. (2008)- !ftxtt do clienrs think of
cognitirc remediarion drrrapy? A consumer-led investigation olgatisfaction and side effects,.{lrencart Jauruul o.[
l>qschiunic Rekohilit"at.iorr, /
I,
I
I
I
-.20i1.
Rorhwctl, P.M., lvl,ellorvell,2., Wong, C.K,, & I)orma::' ?.J. (lS9?). Doct*rs 3n{i patjents don'l *gree: Crnss
secrional srudy lfpatienrs' ald d*ctors' perceptions and assessmenn ofdis*bility in multiple sclcnrsis. Snnii
tlledical ltumal, "?r'r' I 580-l 583,
Saiaror.ic, M., & Ramirez L.F. (2003). Rarr'ng Scale.r in Mertal Healtlt {?,td Ed.). Hudson, Ohio: I-exi-Comp.
346
M. J, Crawford
Joumal of
et al,
August
20ll;2
Shumway, M., Sauuders, 'I'., Shern, D., Piner, 8., l)owns, A., Burbine, T., et al. (2003). Preferences fer
schizophrcnia Iteatmrnl ouxcomes among public policy n:akers, consumers, families, and provide m. I'qrhiatri
.lerrn'ces, .54, I
lt+l128.
Tennant, R., Hiller, L., fislwick, R., Platt, S., Joseph, S., $fleicb, S., el al. (2007). The 1ii/u'wick-E<linburgh Mental
\fdl-being Scale |ffE"!lWBS): Dwelopment and UK valjdadon. Hc<thh and Quality oJ Ltle Outcontes, 5, 63.
The James Und Allisnce. (i00?). The research cycle. Rerrirved from http://*'*r,r'.lindalliance.org/
The Stationery Ofdce. (2008a). High Qua[iry Car* $tr A* NHS Ns:yt Stage Rr-r,reeu I,it*l Report- l]rndon: The
Strtioaery O{frce.
The St*tion*ry Office. (2008b). Halse of Canr*ns Heahh Conuritue: r\tt,rlxil/ lrxtitute lor Heshh afid Clinical
Ij:cel!*r*,, F'irst Repon a/Sassmn 200748. lxrncion: The Stationery 0{ftce.
'Ittcmlcy, 8., & Adams, C. (:998). Content end qurlity of 2000 €ottralled trials in *chizophrenia over 50 years.
Birish Medid Joumal, il7, t l8l-l I84.
Traler, ?., & Callaly, T. (2002). Concordance benveen metrtal:y ill padents and their case manage.s using rhe
Health of rhe Nadou Outcome Scales (FIolrlC)$). Arctalasian ll1chiat1, tA,24*28.
Trauer, T., Callaly, 1'., & Hcrrman, H. (2000). A*irudes oifienral healrh rtaff to rourine outcofie {neasurement.
]o*rxal o! Menal Heahh, I& 288-?97.
Trivedi. P., & !i'y,kes, T. (2002). From passive subiects to equal part*er:: Qualitative review of user involvemenr in
research. Bitish !um! of Psychiarry', 181,468172.
Williams, J.B.W. (2005). Menral health status, functio,liog ar:d disabilities measures. ln Anterican Psychiaric
Asstrciation. (Ed.), Hnnrl&oo,t o_f Prychiatic Measure! {pp. 93-1 l5). Washingron DC: Americsn Psyehiarric
Associarion.
Patients'
PAI.IL
t
Doparttrcnr o!
Glatnargtn
Psychiatrist in
Abatract
Bachgxtm;d.
However, mSny
minoriry feel
Ailn. To
Method,
Intcrviews wcre
ftesldr,
found ta be i
from medicsl
either a form of
Conclusion,
interpersoaal
little choice.
Ke.yword*:
Intruduction
The National
therapy (ECI)
illness,
declined in
{Department
(Challiner &
as rts
ofacdon
(
Asa
number of
2005) are
Most ECT
legisladon
Corespondcnec
Singapore 574421
tssN 0963-823?
1)OI:
17, U.S. Code
l0.ll
$r