Advantages and disadvantages of
the use of best-worst scaling in the
field of health
Terry Flynn PhD
MRC HSRC, Bristol
ICEPOP Programme
Outline
•
•
•
•
•
•
What is best-worst scaling?
How has it been used in HSR to date?
Application: dermatology trial
Application: quality of life
Advantages and disadvantages
Areas for research
ICEPOP Programme
Traditional DCEs
• Discrete Choice Experiments
increasingly used in HSR
• Respondents choose a preferred
specification of the good or service
• Aim is to obtain quantitative estimates of
utility (benefit) associated with different
attribute levels describing the good or
service
ICEPOP Programme
The issue of interest here
Generally dermatology patients would prefer:
• Being seen by a consultant-led team rather
than a GP with part-time interest in
dermatology
• An appointment this week to one in 3 months
But suppose the choice is between an
appointment this week with a GP specialist
and one in 3 months with a consultant.
Which do patients value most? Doctor
expertise or waiting time?
ICEPOP Programme
An example
Appointment A
Appt this week
GP specialist
Easy to get to
(S)he is thorough
You pay £5
Appointment B
Appt in 3 months
Consultant
Difficult to get to
Isn’t thorough
You do not pay
Which appointment would you choose?
ICEPOP Programme
Application 1
Estimating preferences for
aspects of a dermatology
appointment
ICEPOP Programme
Dermatology trial example
Best Appointment A

You will have to wait one month
for your appointment
Getting to your appointment is
difficult and time-consuming
Consultation will be as thorough
as you would like
Doctor is an expert who has
been treating skin complaints for
at least five years
ICEPOP Programme
Worst

Best-Worst Scaling
• Devised by Finn & Louviere (JPPM 1992)
– introduced to health care by McIntosh & Louviere
(HESG 2002)
– statistical proof paper Marley & Louviere (J Math
Psych 2005)
– ‘user guide’ by Flynn et al (JHE 2006)
• Differs from traditional DCEs in the nature of
the choice task
• Individuals choose the best and the worst
attribute based on the levels displayed in a
given specification
ICEPOP Programme
Dermatology trial
• Patients who had been referred to secondary
care for skin complaint
• Postal questionnaire
• Randomly assigned to short version (8 DCE
scenarios) or long (16)
• 202 out of 240 q’airres returned (139
complete)
• Each scenario is a SINGLE consultation
described by waiting time, expertise of
doctor, ease of attending and thoroughness
ICEPOP Programme
Attributes & levels
•
Waiting time
–
–
–
–
•
3 months
2 months
1 month
1 week
Doctor expertise
– Part time specialist (GPSI)
– Full time specialist (consultant)
•
Ease of access
– Easy
– Difficult
•
Individualised care
– Thorough
– Not thorough
ICEPOP Programme
Attribute levels
3.5
3
2.5
First
Second
Third
Fourth
2
1.5
1
0.5
0
Time
ICEPOP Programme
Doctor
Ease access Indiv care
Attribute impacts
2.5
2
1.5
Impact
1
0.5
0
Time
ICEPOP Programme
Doctor
Ease access Indiv care
BWS estimated differences
1.2
1.0
0.8
0.6
First
0.4
Second
0.2
Third
0.0
Fourth
-0.2
-0.4
Time
ICEPOP Programme
Doctor
Ease
access
Indiv
Care
Multinomial (conditional) logit
analysis
• Effect of patient characteristics (clinical or
sociodemographic) upon preferences
• Separate effects of age/sex etc upon
attribute importance from effects upon level
scales
• Independent variables are version of effects
coding – epidemiological example: mean
effect across both sexes is estimated, with
effect code giving additional effect for one
sex (the other is this multiplied by minus 1)
ICEPOP Programme
Fully adjusted MNL results
Attributes
Waiting time
Dr
Convenience
Indivcare
Levels
wait3m
wait2m
wait1m
wait0m
drpttime
drfulltime
convhard
conveasy
indivno
indivyes
Estimate
Std Error
z
p>|z|
|
|
|
|
1.342555
.5544422
.3628801
.1117852
.1045399
.1053237
12.01
5.30
3.45
0.000
0.000
0.001
1.12346
.3495477
.1564495
1.561650
.7593367
.5693108
|
|
|
|
|
|
|
|
|
|
-1.958953
-1.117335
.2137621
2.862526
-1.470253
1.470253
-1.185982
1.185982
-2.843362
2.843362
.1605818
.1493553
.1457884
.1035633
.102335
.1205684
-
-12.20
-7.48
1.47
-14.20
-11.59
-23.58
-
0.000
0.000
0.143
0.000
0.000
0.000
-
-2.273687
-1.410066
-.0719779
-1.673234
-1.386555
-3.079671
-
-1.644218
-.8246039
.499502
-1.267273
-.9854091
-2.607052
-
ICEPOP Programme
[95% confidence interval]
Higher education
Attributes
educ_dr
educ_conv
educ_indiv
Levels
educ_3m
educ_2m
educ_1m
educ_0m
educ_drpt
educ_drft
educ_convh~d
educ_conve~y
educ_indivno
educ_indivye
Estimate
Std Error
z
p>|z|
|
|
|
-.1317751
.0564573
.0145355
.0923188
.0860605
.0862812
-1.43
0.66
0.17
0.153
0.512
0.866
-.3127166
-.1122183
-.1545725
.0491665
.2251328
.1836435
|
|
|
|
|
|
|
|
|
|
-.4883798
-.2318958
.2727426
.4475330
-.1920152
.1920152
-.3173861
.3173861
-.4161934
.4161934
.1332613
.1232679
.1212335
.085256
.0854444
.0982534
-
-3.66
-1.88
2.25
-2.25
-3.71
-4.24
-
0.000*
0.060
0.024*
0.024*
0.000*
0.000*
-
-.7495672
-.4734964
.0351292
-.3591139
-.4848541
-.6087665
-
-.2271924
.0097049
.5103559
-.0249165
-.1499182
-.2236203
-
ICEPOP Programme
[95% confidence interval]
Scoring 7+/30 on skin severity
Estimate
Std Error
z
Attributes
score7_dr |
score7_conv |
score7_indiv |
-.3202987
-.1181401
-.1738758
.0886460
.0826505
.0823243
-3.61
-1.43
-2.11
0.000*
0.153
0.035*
-.4940416
-.2801322
-.3352284
-.1465559
.0438519
-.0125232
Levels
score7_3m
score7_2m
score7_1m
score7_0m
score7_drpt
score7_drft
score7_con~d
score7_con~y
score7_ind~n
score7_ind~y
-.1215269
-.2264255
.022866
.3250864
.1038593
-.1038593
.1480246
-.1480246
.2354545
-.2354545
.1246303
.116925
.1132425
.0744481
.0727303
.0826623
-
-0.98
-1.94
0.20
1.40
2.04
2.85
-
0.330
0.053
0.840
0.163
0.042*
0.004*
-
-.3657979
-.4555942
-.1990853
-.0420564
.0054759
.0734394
-
.122744
.0027433
.2448173
.2497749
.2905734
.3974696
-
|
|
|
|
|
|
|
|
|
|
ICEPOP Programme
p>|z|
[95% confidence interval]
Implications for dermatology
• Policies to improve ‘process’ aspects of
the consultation will benefit higher
sociodemographic groups most
• Policies to improve waiting times will
benefit those patients who they
themselves feel most affected by their
skin condition
ICEPOP Programme
Statistical issues
• MNL is (usually) a first step
– Is there heterogeneity?
– Likely covariates that characterise it?
• More complex methods?
– Mixed logit
• what distributional assumption?
• lots of parameters in BWS: 72 possible pairs here
– Latent class analysis
• Non/semi parametric
ICEPOP Programme
Application 2
Estimating tariffs for the ICECAP
quality of life instrument for older
people
ICEPOP Programme
Heterogeneity
It’s one thing to know what the ‘average’
preference for an impaired health state
is in the population……but
suppose the poor/ill regard that state as
being particularly dreadful – any
decision to take (or not take) this into
consideration requires us to find out if
the poor/ill have different preferences
ICEPOP Programme
Heterogeneity (2)
• The use of population-level tariffs might
mean some interventions are deemed
cost-ineffective when for the poor/ill
they are highly cost-effective
• Even if we don’t want to move away
from population-level provision society
should have the data to debate this
ICEPOP Programme
Aim
• To produce a set of ‘tariffs’ for the 45=1024
possible quality of life scenarios that a British
older person might experience
• An older person could tick the box to indicate
which of 4 levels (s)he is experiencing for
each of 5 questions
– e.g. before the meals-on-wheels service a score
of 0.6 on a zero to one scale
– after the meals-on-wheels service a score of 0.75
on a zero to one scale
ICEPOP Programme
The ICECAP quality of life
instrument
• Four levels
–
–
–
–
all;
a lot (many);
a little (few);
none
• Example: role
o I am able to do all of the things that make me feel valued
 I am able to do many of the things that make me feel valued
o I am able to do a few of the things that make me feel valued
o I am unable to do any of the things that make me feel valued
ICEPOP Programme
The ICECAP quality of life
instrument (contd)
Similarly for:
Attachment
Security
Enjoyment
Control
ICEPOP Programme
(love and friendship)
(thinking about the future
without concern)
(enjoyment and pleasure)
(independence)
A complete quality of life state
I can have all of the love and friendship that I
want
I can only think about the future with a lot of
concern
I am able to do many of the things that make
me feel valued
I can have a little of the enjoyment and pleasure
that I want
I am able to be completely independent
ICEPOP Programme
The best-worst scaling study
• 315 completed interviews (478 approached
to take part)
• 255 had complete best-worst data
• Average length of interview: 35 minutes
Administered in older person’s own home
• All had participated in Health Survey for
England (HSE)
• Data available from previous round of HSE
(6-12 months previous) included
sociodemographic and health (n=226)
ICEPOP Programme
Statistical design
• Respondents randomised to:
– Orthogonal main effects plan in 16
scenarios or
– Its foldover
ICEPOP Programme
Best
Example quality of life scenario
Worst
You can have a lot of the love and friendship
that you want
You can only think about the future with a lot of
concern
You are unable to do any of the things that make
you feel valued
You can have a little of the enjoyment and
pleasure that you want

You are able to be independent in a few things
ICEPOP Programme

Population-level BWS
estimates (n=255)
Control
Enjoyment
4
3
Role
2
1
Security
Attachment
0
1
2
3
4
5
6
Values rescaled so lowest value (control level 1)
equals zero
ICEPOP Programme
Heterogeneity in ICECAP
0.8
0.7
y = 0.0099x -1.134
R2 = 0.7583
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
ICEPOP Programme
0.2
0.3
0.4
0.5
0.6
0.7
Latent class analysis
• Performed on the choice data
• Conditional logit results for each class
• No adjustment for covariates
– Need to know first of all if subgroups who are
internally homogeneous exist
– Then see if we can characterise these in terms of
health/wealth/other factors
• Covariate-adjusted conditional logit
regressions (1-class) suggested there was
heterogeneity…
ICEPOP Programme
LCA results
Table 1: Latent class analysis summary results
Model
Number LogBIC(LL) Paramete R-squared df
classes likelihood
rs
1
1 class -7411.89 14926.77
19
0.021
2
2 class -7078.65 14368.69
39
0.037
3
3 class -6865.71 14051.23
59
0.132
4
4 class -6782.53 13993.28
79
0.149
5
5 class -6713.34 13963.31
99
0.164
6
6 class -6654.38 13953.8
119
0.199
7
7 class -6606.25 13965.96
139
0.207
ICEPOP Programme
207
187
167
147
127
107
87
Statistical vs policy significance
Attachment
Class 1
Class 2
Class 3
N=107
N=78
N=32
Pop
I can have all of the love and
friendship that I want
0.2153 0.3134 0.2899 0.254
I can have a lot of the love
and friendship that I want
0.1835 0.2888 0.2702 0.233
I can have a little of the love
and friendship that I want
0.1079 0.1562 0.1237 0.134
I cannot have any of the love
and friendship that I want
-8E-04
-0.044 0.0121
-0.013
I am able to be completely
independent
0.2443 0.1702 0.1287 0.2094
I am able to be independent
in many things
0.1885 0.1692
I am able to be independent
in a few things
0.0984 0.1076 0.0502 0.1076
0.197 0.1848
I am unable to be at all
independent
Control
ICEPOP Programme
-0.069
-0.023
-0.041
-0.051
Who are these people?
• Can distinguish class three easily:
disproportionately:
– Male
– Without any qualifications
– Married (but only at 10% level)
But so what? Class 1 vs class 2….?
ICEPOP Programme
Class 1 versus class 2
• Difficult to distinguish them
– Having had a total joint replacement was
predictor for class 2 (more bothered about
attachments than control)
– Being unable to climb 12 stairs was predictor for
class 1 (more bothered about control than
attachments)
• Work with UTS researchers to investigate
alternative characterisations of clustering
ICEPOP Programme
Advantages of BWS
• All attribute levels on the same scale
• More data
– Estimate attribute impacts
– Understand heterogeneity more easily;
distributional assumptions not needed when have
individual respondent utilities
• Use as a method to get a random utility
theory consistent set of rankings
• Easier choice task?
• Simpler statistical design
ICEPOP Programme
Disadvantages of BWS
• The problem of the numeraire (money)
• Conditional not unconditional demand
– Nest within a DCE and adjust for different
random utility components
• Getting individual respondent models
not practical in many contexts
ICEPOP Programme
Future research in Best-Worst
methods
• Individual patient preferences
– clustering using other taxonomic methods
– investigate decision rules (lexicographic
preferences)
• Estimating attribute importance (rather than
simply impact)
– Alternative conceptualisation of utility
• Anchoring (the unconditional demand issue)
ICEPOP Programme
Investigating Choice
Experiments for the
Preferences of Older People
(ICEPOP)
Professors Joanna Coast (Birmingham)
Jordan Louviere (UTS)
Tim Peters (Bristol) &
Dr Terry Flynn
We would like to thank Dr Tony Marley for comments and assistance
5
4
3
2
1
0
0
.2
.4
.6
tariff
ICEPOP Programme
.8
1
5
4
3
2
1
0
0
.2
.4
.6
tariff
ICEPOP Programme
.8
1
Bristol sample
198 of the 1024 QoL states represented in Bristol
------------------------------------------------------------Percentiles
Smallest
1%
.3477733
.1051461
5%
.5968553
.2584114
10%
.6542614
.2636209
Obs
810
25%
.7704228
.2659647
Sum of Wgt.
810
50%
75%
90%
95%
99%
.8608195
.9135509
.9852881
1
1
ICEPOP Programme
Largest
1
1
1
1
Mean
Std. Dev.
.8291571
.1323457
Variance
Skewness
Kurtosis
.0175154
-1.44312
6.231093
ICECAP sample (313)
137 of the 1024 QoL states represented in BWS study
------------------------------------------------------------Percentiles
Smallest
1%
.2659647
0
5%
.5297861
0
10%
.6329976
.1483945
Obs
313
25%
.7576444
.2659647
Sum of Wgt.
313
50%
75%
90%
95%
99%
.8515525
.9135509
.9623603
.9982446
1
ICEPOP Programme
Largest
1
1
1
1
Mean
Std. Dev.
Variance
Skewness
Kurtosis
.8137987
.1524833
.0232512
-2.020194
9.229487
Random Utility Theory
• Let latent utility for item i be:
Ui = di + ei
Ui = latent utility, di = explainable portion & ei =
unexplainable portion.
• Probability that i is chosen:
P(i | Cn) = P[(di + ei) > (dj + ej)]  j  Cn,
if e’s ~ EV1 (0, 2)  McFadden’s MNL model:
P(i | Cn) = exp(di) / j exp(dj)
ICEPOP Programme