Comparison of Adaptive Conjoint Analysis and Simple Multiattribute Rating Theory
Dwayne Ball and David L. Olson, University of Nebraska
Application
To compare simplified versions of conjoint analysis and multiattribute utility theory to
identify practice attributes that primary care physicians believe would help them provide
optimal care for elderly patients. Given the large number of potential attributes and levels in
the case, full conjoint analysis methods involve too many questions of too great a complexity for
subjects. This proof-of-concept proposal has the purpose of comparing relative accuracy and
subject burden among full profile ratings based conjoint analysis, adaptive conjoint analysis, and
the simple multiattribute rating theory.
Motivation for the Comparison
The elderly population will continue to expand rapidly until the middle of the 21st century. If
Objective 1-5 of Healthy People 2010, to “increase the proportion of persons with a usual
primary care provider” is to be met, primary care physicians will need to increase the volume of
older patients they provide care for. There are some indications that physicians may not
welcome a larger volume of elderly patients. Between 30% and 50% of primary care physicians
limit the number of elderly patients they provide care for (Damiano, Cykert, Lee, Gillis, Geiger).
In addition, many physicians find elderly patients more difficult to treat (Damiano) and many
feel unprepared to care for them (Cantor, Perez). Physicians have also reported that Medicare
policies (Damiano, Cykert, Lee), time pressure, and inadequate community resources (Pereles)
are sources of frustration in caring for older patients. Because elderly patients often have
cognitive impairment and physical disabilities, their needs are often greater than younger
populations. In order for physicians to provide patient centered care for this population, they
must feel able to meet their particular needs. Little research has been done to determine the
needs and preferences of primary care physicians in providing care for this population. The
overall research motivation is to identify ways to make medicare patients more attractive to
physicians. The approach we propose is to identify the preference functions of a sample of
subjects who are physicians in this field. Identifying attributes of importance will enable better
understanding of compensatory features that will lead to greater acceptance of medicare patients
on the part of physicians. This knowledge will enable more rational organization of practices
and selection of public policy alternatives.
An intermediate objective of the overall research program is to identify what makes individual
patients burdensome to practicing physicians. There are 15 patient attributes identified as
potentially important in physician decisions to accept geriatric patients.
B. BACKGROUND AND SIGNIFICANCE
Significance of the proposed research
Between 1995 and 1998, the number of people 65 and older making ambulatory care visits to
physicians’ offices in the United States increased from 168,135 to 197,085 1 and this number will
continue to rise. The impact of population aging on health care has been of concern to clinicians
and policymakers for many years.2-4 In 1998, an entire issue of Health Services Research was
devoted to “Organizational Issues in the Delivery of Primary Care to Older Americans.”5
Because the number of available geriatricians remains very small, general internists and family
physicians will be the main providers of primary care to elders.6 These primary care physicians
do not appear ready to receive an increasing elderly patient population: between 30% and 50% of
primary care physicians limit the number of Medicare patients in their practices,7-11 and they are
more likely than specialists to do so.12
Because we know so little about primary care physicians’ needs and preferences in providing
care to elderly patients, we cannot make informed efforts to improve medical coverage of this
class of patient. As a result of the proposed study, we expect to determine those factors that
make medicare patients burdensome to physicians. This is significant because with this
knowledge, we and others will be able to target interventions toward the areas of greatest
concern to physicians and thereby enhance their ability to provide high quality patient centered
care to an increasing volume of older patients. This in turn will help assure access to primary
care for elderly people. The knowledge gained in this study will inform the future development
of new approaches to practice organization, educational programs, and health care policy that are
likely to increase physicians’ capability to provide care for a larger volume of elderly patients.
Conjoint analysis, relatively new to health services research, is a rigorous way of determining
respondents’ preferences. This technique takes subject preference input and fits a function using
regression or similar mathematical modeling tools. It has been used with increasing frequency to
determine patient preferences for treatment (Fraenkel 2001, Ryan 1997,). It has also been used in
studies of physicians’ preferences for practice and job characteristics (Scott 1999, Gosden 2000).
Conjoint analysis has strong theoretical grounding in mathematical psychology (). It has been
used extensively in marketing research to establish what factors influence the demand for
different commodities and what combinations of attributes would maximize sales (Cattin 1982,
Green and Krieger, 1988; 1989; 1992). Ball and Marquardt (1991) applied conjoint analysis to identify
preferences of venture capitalist preferences over investment opportunities. Over the last several years
it has been used with growing frequency in health [economics] and has shown high internal
consistency () [and predictive validity?] (). Important advantages of conjoint analysis for this
study are: 1. it simulates actual decision making by using scenarios with varying levels of the
attributes in question. Because respondents actually make choices in this setting,… we will be
able to ascertain how physicians weigh the value of patient characteristics and resources. 2. it
can estimate the relative importance of different characteristics of patients and situations to the
physician respondents, and 2. it can show what tradeoffs physicians are willing to make between
characteristics when they must decide on which practice attributes are most conducive to caring
for elderly patients.
Comparison of Methods
The most complete and widely accepted method to elicit preference is full profile ratings based
conjoint analysis. This method is complete in that all possible interactions can be examined.
However, the number of questions asked of a subject physician would be well beyond a
reasonable number. Fractional factorial conjoint analysis is a simpler version including all
measures at least some minimal number of times. By assuming away partial interactions, this
approach provides a thorough study if interactions are absent, with fewer responses required of
subjects.
However, in the population to be studied, there are expected to be high levels of interaction. The
number of attributes (15) and levels yield a high number of parameters to estimate (21).
Therefore, it would be attractive to apply other methodologies in an initial screening analysis,
with the intent of:
a) reducing the number of attributes important in the average physician preference function
b) examining the relative accuracy of alternative methodologies in case they might provide
sufficient accuracy for the complete study.
There are two alternative preference elicitation methodologies proposed for examination in this
study. Adaptive conjoint analysis is a method supported by computer software marketed by
Sawtooth Software, Inc. There is evidence that full profile ratings based conjoint analysis and
fractional factorial conjoint analysis do not accurately identify attributes that are important.
Adaptive conjoint analysis may provide a more accurate metric, and certainly does require less
input burden on the part of subjects. Another methodology is multiattribute utility theory
(MAUT). A simplified version of MAUT is the Simple Multiattribute Rating Theory (SMART).
The proposed proof-of-concept research is to compare these alternative methodologies with full
profile ratings based conjoint analysis in order to evaluate the number of questions and time
required of subjects, and to assess relative accuracy based upon analysis of preference function
consistency obtained (measured by the consistency of attribute level values, as well as rank
reversal of attribute importances, and rank reversal of a test set of patients).
The research design is to apply all three methods with a set of 36 subject physicians. These
subjects will be from the University of Nebraska Medical College system.
Method Background
Conjoint analysis has proven to be a valuable tool in the medical field in a number of
applications (Ryan, 1999). Spoth, et al. (1996) applied conjoint analysis for family-focused
prevention program preferences. Gosden et al. (2000) applied conjoint analysis to survey
general practitioner preferences for practice and job characteristics. Ryan and Farrar (2000)
argued that the application of conjoint analysis by patients in selecting health care options could
be expanded. Fraenkel et al. (2001) applied the adaptive form of conjoint analysis to patient
preferences over treatment options. Ryan and Farrar noted the danger of inaccurate data. Ryan
et al. (1998) examined issues of ordering effects, internal validity, and internal consistency in the
application of conjoint analysis in health care.
One of the practical difficulties in using conjoint analysis is that subjects are asked for a lot of
input information. The tedium and abstractness of preference questions can lead to inaccuracy
on the part of subject inputs (Larichev, 1992). Human subjects have been noted to respond
differently to different framing contexts as well (Kahneman and Tversky, 1979).
Conjoint analysis takes subject preference input and fits a function using regression or similar
mathematical modeling tools. Conjoint analysis has been widely used in marketing research
(Green and Krieger, 1988; 1989; 1992). Ball and Marquardt (1991) applied conjoint analysis to
identify preferences of venture capitalist preferences over investment opportunities. Conjoint
analysis is well-developed in the marketing field.
Multiattribute utility theory (MAUT – Keeney & Raiffa, 1976; Fishburn, 1989) approaches
identification of preference functions in a different way. While conjoint analysis fits a function
over a set of preference indications over presented alternatives, MAUT seeks to identify the
function which can be applied to any number of alternatives. As there are a number of conjoint
analysis methodologies, there also are a number of MAUT methodologies (Olson, 1996;
Larichev & Olson, 2001).
The proposed research is to identify the feasibility of applying MAUT approaches to medical
preference situations. Dr. Adams will be able to identify decisions based upon preference in the
medical field. Dr. Ball will be able to identify alternative conjoint analysis approaches. Dr.
Olson will be able to identify alternative MAUT approaches, and to mathematically compare
results based upon simulation scenarios to identify the relative degree of accuracy sacrificed
through the MAUT approach relative to conjoint analysis, as well as the degree of input from
subjects (expected to be much less for MAUT). This information will provide valuable
understanding of this tradeoff, and will enable identification of medical problem contexts that
can better be served by faster methods, or more accurate but much slower methods.
Comparative Results
Adaptive conjoint analysis (ACA) software was obtained from Sawtooth Software, Inc.’s web
site, loaded with a personal computer selection problem. There were seven attributes as given in
Table 1. Attribute levels are given in Table 1 listed in order of best to worst. (For attributes
Brand and Optical Drive, no order of preference is implied by order.)
Table 1: Data Set
Attribute Brand
Memory
(RAM)
512 MB
Hard
Drive
80 GB
Monitor
1.7 Ghz
256 MB
1.5 Ghz
128 MB
1.1 Ghz
0.9 Ghz
Source: www.sawtoothsoftware.com
60 GB
40 GB
20 GB
17”
15”
Best
Dell
HP
IBM
Gateway
Processor
speed
2.0 Ghz
21”
Keyboard
Optical
Drive
ergonomic CDRW/DVD
standard
DVD
CD-RW
CD
The comparison we conducted begins with a SMART analysis. The reason for this is for control
purposes. We are trying to measure differences between the methods. Our design was to
conduct a SMART analysis, and infer responses to ACA based upon SMART output.
The SMART methodology begins with development of relative weights. This is done by
anchoring each attribute in terms of the best and worst measures imaginable, and then applying
the swing weighting method (Edwards and Barron, 1994). Here we supplement this approach by
conducting the analysis from the perspective of both the most important and least important
attributes, and selecting final weight estimates as rough averages of these two estimates. Results
are given in Table 2:
Table 2: Swing Weighting
Ranked
Worst
Best
attributes
Processor 700 Mhz 2.0 Ghz
speed
128 MB
512 MB
RAM
Floppy
CD-R/
Optical
(none)
DVD
drive
14”
21”
Monitor
10 GB
80 GB
Hard
drive
future
Keyboard palm
Fred’s
Eagle
Brand
Best=100
100
Implied
weights
0.269
Worst=10 Implied
weights
300
0.229
Rough
average
0.250
80
70
0.215
0.188
280
250
0.214
0.191
0.215
0.190
60
50
0.161
0.134
220
200
0.168
0.153
0.165
0.140
10
2
0.027
0.005
50
10
0.038
0.008
0.030
0.010
First swing weights were estimated by identifying that attribute most important to swing from
worst performance to best. In some cases worst performances were taken from options
imaginable outside of the given data set. The same could have been done for the best
imaginable, although in this case the best performance measure from the data set was used for
the best rating. Rough averages were obtained for all but the least important attribute, which was
used to balance the sum of rough averages to add to 1.0.
The next step was to assign partial utilities for each attribute level. Numbers used are given in
Table 3.
Table 3: Attribute Partial Utility Scores
Score
Processor RAM
Optical drive
speed
1.0
2.0 Ghz
512 MB CDRW/DVD
0.9
1.7 Ghz
CDRW
0.8
1.5 Ghz
0.7
1.1 Ghz
CD
0.5
256 MB
0.4
DVD
0.3
900 Mhz
0.1
0
700 Mhz 128 MB floppy
Monitor
21”
17”
Hard
drive
80 GB
60 GB
Keyboard
40 GB
standard
Future
Eagle
ergonomic Dell, HP
15”
14”
Brand
Gateway
IBM
20 GB
10 GB
Palm
Fred’s
In SMART, direct assessment can be used. This allows nonlinear utilities, as with hard drives
and processor speeds. It also allows assignment of utility scores to discrete attributes, such as
optical drive, keyboard, and brand. Here the assessor found DVD optical drives unattractive
nuisances encouraging replacement of current music collections with newer technology at an
exorbitant price. Assessment of brands was purely personal prejudiced perception of the
assessor.
This information provides a means to assess relative value of any personal computer system
consisting of combinations of the seven attribute levels given. In fact, this information was used
as the basis of responses to ACA software. The partial contribution inferred for each feature was
as given in Table 4.
Table 4: Partial Utility Contributions to Value
Processor speed
Utility score
2.0 Ghz
1.0
1.7 Ghz
0.9
1.5 Ghz
0.8
1.1 Ghz
0.7
900 Mhz
0.3
RAM
512 MB
1.0
256 MB
0.5
128 MB
0
Optical Drive
CD-RW/DVD
1.0
CD-RW
0.9
CD
0.7
DVD
0.4
Monitor
21”
1.0
17”
0.9
15”
0.3
Hard Drive
80 GB
1.0
60 GB
0.9
40 GB
0.7
20 GB
0.1
Keyboard
Ergonomic
0.9
Standard
0.7
Brand
Dell
0.9
Hewlett Packard
0.9
IBM
0.3
Gateway
0.7
Times weight (0.250)
0.250
0.225
0.200
0.175
0.075
Times weight (0.215)
0.215
0.108
0
Times weight (0.190)
0.190
0.171
0.133
0.076
Times weight (0.165)
0.165
0.149
0.050
Times weight (0.140)
0.140
0.126
0.098
0.014
Times weight (0.030)
0.027
0.021
Times weight (0.01)
0.009
0.009
0.003
0.007
ACA Input
The Sawtooth ACA system was used, seeking to input information in compliance with the
SMART analysis just presented. ACA asks for input on a 1-7 Likert scale, allowing entry of
degree of preference. Initially, ratings of the three categorical data variables (brand, keyboard,
and optical drive) were entered on this 1-7 scale with 7 being best. Table 5 gives entries for this
phase.
Table 5: Entries of Relative Value for Categorical Data
Brand
Value Keyboard
Value
Dell
6
Standard
5
HP
6
Ergonomic
6
IBM
2
Gateway
5
Optical Drive
CD
CD-RW
DVD
CD-RW/DVD
Value
5
6
3
7
Note that most people would consider the DVD optical drive as an advance over rewriteable CDROMs. However, this subject’s personal bias was against DVDs, for reasons given above.
The next input allows the user to rank order swing preferences, presenting the extreme cases for
each attribute in turn, and asking for assignment of a value on the 1-7 scale (1 bad, 7 good). To
match the SMART input, the entries given in Table 6 were made.
Table 6: Swing Ratings for Attributes
Attribute
Worst Measure
Brand
IBM
Processor speed
900 Mhz
Memory (RAM)
128 MB
Hard drive
20 GB
Monitor
15”
Keyboard
Standard
Optical drive
CD-ROM
Best Measure
Dell
2 Ghz
512 MB
80 GB
21”
Ergonomic
CD-RW/DVD
Rating
1
7
6
3
4
2
5
ACA next asks for comparisons of attribute levels, beginning with pairs of attributes, then adding
more complex tradeoffs of three attributes. Responses are again on a 1-7 scale, only now 4
represents equivalent preference, and 1 or 7 extreme preference for one set of attribute values or
the other.
The first choice was between a system with RAM of 256 MB and Processor speed of 1.1 Ghz,
vs. RAM of 512 MB and Processor speed of 900 Mhz. The partial utility from SMART for the
first pair was 0.282, while the partial utility for the second set was 0.285. This was a minimal
difference, so an entry of 4 was made, indicating equivalence. The cutoffs used were
indifference (rating of 4) if the difference in weighted partial utility was 0.030 or less, 3 or 5
(depending upon which weighted partial utility was greater) for differences greater than 0.030
but less than or equal to 0.035, 2 or 6 for differences greater than 0.035 but less than or equal to
0.075. Table 7 shows the entries made.
Table 7: Preference Assessments
Choice 1
256 MB
CD
15” mon
21” mon
Std KB
Ergo KB
Gateway
512 MB
Dell
20 GB
Proc 1.1
256 MB
CDRW/DVD
20 GB
80 GB
IBM
1.7 Ghz
15” mon
128 MB
2.0 Ghz
1.7 Ghz
40 GB
Ergo KB
Std KB
Part val
0.282
0.238
0.241
Choice 2
512 MB
CD-RW
21” mon
Proc 0.9
128 MB
DVD
Part val
0.285
0.171
0.246
Diff.
0.003
0.067
0.005
Entry
4
2
4
0.184
0.161
0.030
0.308
0.282
0.149
0.454
17” mon
Ergo KB
Std KB
HP
128 MB
HP
60 GB
40 GB
40 GB
Dell
900 Mhz
17” mon
512 MB
1.7 Ghz
CD
Ergo KB
60 GB
CD
0.251
0.125
0.030
0.217
0.180
0.345
0.484
0.067
0.036
0
0.091
0.102
0.196
0.030
6
2
4
1
1
7
4
0.385
0.128
1.1 Ghz
60 GB
Std KB
Ergo KB
21” mon
Gateway
0.366
0.160
0.019
0.032
4
5
DVD
Std KB
80 GB
CDRW/DVD
17” mon
Dell
Finally, ACA asked the user to rate four computer systems measured on five attributes on a 0100 scale, with 100 being best. This set included nadir (option 1) and ideal (option 2) solutions.
Choices and input are given in Table 8:
Table 8: Assessment of Options
Attribute
Option 1
Processor speed
900 Mhz
Memory (RAM) 128 MB
Optical reader
DVD
Monitor
15”
Hard drive
20 GB
Weighted partial 0.216/0.960
utility
Assessment
23
Option 2
2.0 Ghz
512 MB
CD-RW/DVD
21”
80 GB
0.960/0.960
100
Option 3
900 Mhz
128 MB
DVD
21”
80 GB
0.461/0.960
50
Option 4
2.0 Ghz
512 MB
DVD
15”
20 GB
0.601/0.960
61
Since only five of the attributes were presented, total weighted partial utility of the ideal solution
(option 2) was only 0.960.
ACA Results
ACA returned attribute importances, which are not weights, but consider both importance and
scale differences. These are displayed against weights obtained from SMART.
Table 9: Attribute Importance Results
Attribute
Processor speed
Memory (RAM)
Optical drive
Monitor
Hard drive
SMART weight
0.25
0.21
0.19
0.17
0.14
rank
1
2
3
4
5
ACA importance
26
27
12
11
10
rank
2
1
3
4
5
Keyboard
Brand
0.03
0.01
6
7
1
6
7
6
There are two anomalies in that the relative importance of processor speed and memory are
reversed, and those of keyboard and brand are as well. However, there clearly is a strong
relationship. The difference can be explained by the fact that the SMART input used different
anchors for all attributes except for Memory (RAM). Thus attribute Processor speed had a
shorter range in ACA than in SMART, which reduced its importance. Also, in SMART the
Keyboard attribute range was very small, reducing its importance below that of Brand.
The next output from ACA was Attribute Utilities. These are compared in Table 10.
Table 10: SMART Utility Scores versus ACA Attribute Utilities
Processor speed
SMART Utility score
ACA Attribute Utility
2.0 Ghz
1.0
75
1.7 Ghz
0.9
89
1.5 Ghz
0.8
18
1.1 Ghz
0.7
-32
900 Mhz
0.3
-150
RAM
512 MB
1.0
63
256 MB
0.5
-37
128 MB
0
-26
Optical Drive
CD-RW/DVD
1.0
27
CD-RW
0.9
31
CD
0.7
29
DVD
0.4
-86
Monitor
21”
1.0
65
17”
0.9
-29
15”
0.3
-36
Hard Drive
80 GB
1.0
-1
60 GB
0.9
19
40 GB
0.7
-34
20 GB
0.1
17
Keyboard
Ergonomic
0.9
23
Standard
0.7
-23
Brand
Dell
0.9
-15
Hewlett Packard
0.9
26
IBM
0.3
0
Gateway
0.7
-11
There were a few anomalies. Those for Brand entries were probably due to the very low weight
on that attribute, which made it inconsequential in selections.
Ball, A.D. & Marquardt, R.A., How do venture capitalists rank potential investments? The role
of government incentives. Regional Science Perspectives 21:1, 1991, 50-66.
Edwards, W. and Barron, F.H. SMARTS and SMARTER: Improved simple methods for
multiattribute utility measurement. Organizational Behavior and Human Decision Processes
60, 1994, 306-325.
Fishburn, P.C., Foundations of decision analysis: Along the way. Management Science 35, 1989,
387-405.
Fraenkel, L., Bodardus, S, & Wittink, D.R., Understanding patient preferences for the treatment
of Lupus Nephritis with adaptive conjoint analysis. Medical Care 39:11, 2001, 1203-1216.
Gosden, T., Bowler, I., & Sutton, M., How do general practitioners choose their practice?
Preferences for practice and job characteristics, Journal of Health Service Research Policy
5:4, 2000, 208-213.
Green, P.E. & Krieger, A.M. Choice rules and sensitivity analysis in conjoint simulators. Journal
of the Academy of Marketing Science 16:1, 1988, 114-127.
Green, P.E. & Krieger, A.M. Recent contributions to optimal product positioning and buyer
segmentation. European Journal of Operational Research 41, 1989, 127-141.
Green, P.E. & Krieger, A.M. Modeling competitive pricing and market share: Anatomy of a
decision support system. European Journal of Operational Research 60, 1992, 31-44.
Kahneman, D. & Tversky, A., Prospect theory: An analysis of decision under risk. Econometrica
47:2, 1979, 263-291.
Keeney, R.L. & Raiffa, H. Decision with Multiple Objectives: Preferences and Value Tradeoffs.
New York: Wiley, 1976.
Larichev, O.I., Cognitive validity in design of decision-aiding techniques. Journal of MultiCriteria Decision Analysis 1:3, 1992, 127-138.
Larichev, O.I. & Olson, D.L. Multiple Criteria Analysis in Strategic Siting Problems. Boston:
Kluwer Academic Publishers, 2001.
Olson, D.L., Decision Aids for Selection Problems. New York: Springer, 1996.
Ryan, M., A role for conjoint analysis in technology assessment in health care? International
Journal of Technology Assessment in Health Care 15:3, 1999,m 443-457.
Ryan, M. & Farrar, S., Using conjoint analysis to elicit preferences for health care. British
Medical Journal 320, 3 June 2000, 1530-1533.
Ryan, M., McIntosh, E., & Shackley, P., Methodological issues in the application of conjoint
analysis in health care. Health Economics 7, 1998, 373-378.
Spoth, R., Ball, A.D., Klose, A., & Redmond, C. Illustration of a market segmentation technique
using family-focused prevention program preference data. Health Education Research 11:2,
1996, 259-267.