An Open Problem: Characterizing Random
Utility Models of Best-Worst Choice
A. A. J. Marley
University of Victoria, Victoria
DIMACS Workshop on Polyhedral Combinatorics of Random Utility, May 2006
Research Supported by
• The Australian Research Council (ARC)
• The Natural Sciences and Engineering
Research Council (Canada)
Summary
• An illustrative empirical study
• Definitions and theoretical examples
• Random utility models for best-worst
choice
• Characterizing random utility models of
best worst choice
• (Further theoretical results on Thurstonian
models of best-worst choice)
Public Issues
•
•
•
•
•
•
•
•
•
•
•
Preserving the environment
Quality medical care
Cost of living
Crime and public safety
National unity/Canada’s future
Level of taxation
Poverty in Canada
Drinking and driving
A quality food supply
The drug problem
Food safety
Question & Layout for Best-Worst
Task (example shows 2 questions)
Question & Layout Used for First Best-Worst Task
Identify the public issue that causes you the most concern and the public issue that
causes you the least concern in each of the following sets of issues.
Check only one issue for each of the Least and Most Columns.
LEAST
ISSUE
MOST
LEAST
ISSUE
Preserving the environment
Cost of living
Level of Taxation
A quality food supply
Quality medical care
Quality medical care
The drug problem
Poverty in Canada
MOST
Fractional factorial design for choice sets (Best-Worst task)
Issue
10
11
12
# times
present
1
2
3
4
5
6
7
8
9
Food safety
0
0
1
0
0
1
1
1
0
1
0
1
6
Preserving the environment
1
0
0
1
0
0
0
1
1
1
0
1
6
Cost of living
0
1
0
0
1
0
1
0
1
1
0
1
6
Level of taxation
1
0
1
0
0
0
1
0
1
1
1
0
6
National unity/Canada’s future
0
0
1
0
1
0
0
1
1
0
1
1
6
Crime & public safety
0
0
0
1
0
1
1
0
1
0
1
1
6
A quality food supply
0
1
1
1
0
0
0
0
0
1
1
1
6
Quality medical care
1
1
0
0
0
0
1
1
0
0
1
1
6
Drinking & driving
0
0
0
1
1
0
1
1
0
1
1
0
6
The drug problem
1
0
0
0
1
1
0
0
0
1
1
1
6
Poverty in Canada
0
1
0
0
0
1
0
1
1
1
1
0
6
# Issues in set
4
4
4
4
4
4
6
6
8
8
8
8
Comparison of Most-Least totals Vs
MNL estimates
2.5
2
Most-Least Totals
1.5
1
0.5
0
-1.000
-0.500
0.000
-0.5
0.500
-1
-1.5
-2
MNL Estimates
1.000
1.500
Summary
• An illustrative empirical study
• Definitions and theoretical examples
• Random utility models for best-worst
choice
• Characterizing random utility models of
best worst choice
• (Further theoretical results on Thurstonian
models of best-worst choice)
Consistent Random Ranking
Models
And the Open Problem
Observations
• A consistent random ranking model on (all the
subsets of) a set T has consistent margins
• Conversely, if a set of best-worst choice
probabilities satisfy a random ranking model
(next slide), then one can define a set of best
and worst choice probabilities through the
margins of the ranking probabilities
• Thus, the major task is to characterize a random
ranking model on best-worst choice probabilities
Summary
• A major open problem is to characterize the
class of (best, worst and) best-worst choice
probabilities that satisfy a consistent random
ranking model
• We have a characterization of the maxdiff model
for best-worst choice. This model is the most
frequently used in applications
• The maxdiff model (with its ‘natural’ extension to
best and worst choices) is not compatible with a
consistent random ranking model
Thank you for listening
Summary
• A major open problem is to characterize the
class of consistent random ranking models of
best, worst and best-worst choice
• We have a characterization of the maxdiff model
for best-worst choice. This model is the most
frequently used in applications
• The maxdiff model (with its ‘natural’ extension to
best and worst choices) is not compatible with a
consistent random ranking model
Thank you for listening