Survey Measurement of
Preference Parameters
Presentation by Miles Kimball
Osaka University, 2004
Outline
• What We Want to Know
• A New Approach: Survey Measurement of
Preference Parameters
• Challenges in the Survey Measurement of
Preference Parameters
• Where to Go from Here
What We Want to Know: A Partial
Wish-List We All Have in Common
• Time Preference
• Elasticity of
Intertemporal
Substitution
• Altruism/Envy
• Labor Supply
Elasticities
• Risk Aversion
• Prudence
• Temperance
• Flow Utility
• Overall Present
Discounted Value of
Utility
• Marginal Propensity
to Consume
• Shadow Interest Rate
• Shadow Wage
• Risk Aversion of
Value Function
The Need for a New Approach
• Despite decades of research using data on
actual economic choices, there is still no
consensus on the values of many basic
preference parameters.
• In the absence of persuasive identification of the
values of preference parameters, convenient
calibrations continue to reassert themselves.
– log utility
– additive separability
– zero or infinite labor supply elasticity
Advantages of Hypothetical
Experiments
• Exogenous
• Large enough to overcome frictions
• Simplified situations highlight different
aspects of preferences
• Close connection to economic theory
• Close connection to economic intuition
The Disadvantage of Hypothetical
Experiments
• Many economists have doubts about
whether people can really answer these
questions meaningfully.
• These doubts deserve to be taken
seriously, without letting them paralyze the
research program.
• This requires dealing seriously with the
cognitive limitation of respondents and
measurement error.
Challenges in the Survey
Measurement of Preference
Parameters
•
•
•
•
Cognitive limitations of respondents
Measurement error
Linking to economic theory
Survey Mechanics
Evidence that Survey Respondents
Have Important Cognitive
Limitations
• Question order and the precise wording of
questions often matter.
– First response on paper and pencil questions, last
response in phone interviews
– Yes-man effect
– Status quo bias
– Anchoring (effects of the response categories)
– Overweighting of recent events
– Guessing what the interviewer would think was
reasonable
Strategies for Dealing with
Respondents’ Cognitive Limitations
• Get many people’s reactions to a question, particularly
the reactions of professional survey methodologists.
• Make a vignette (story) for the hypothetical situation that
is as concrete and believable as possible given the
objective.
• Break the question down into pieces that are each
individually easy to understand—even at the cost of
complicating the later analysis of the question.
• When the question still seems too hard, go back to the
drawing board and design a new question to get at the
desired concept.
A Slogan
• To maximize the reliability of survey
answers to difficult questions, substitute
researcher effort for respondent effort
wherever possible.
How to Gauge How Seriously
Cognitive Limitations Might Be
Affecting Responses
• Pretest.
• Look at nonresponse rates.
• Debrief respondents. (“Why did you answer the
way you did on that question?”)
• Try different variants of the same basic question.
– Vary the wording
– Vary the order
– Vary cutpoints and the unfolding sequence
Measurement Error
• Even after doing everything possible to make the
questions as easy as possible for respondents to
understand and answer, there will still be
measurement error.
• Fortunately, econometric techniques can deal
with this measurement error.
• The appropriate econometric techniques are
different from those we are used to because
survey measurement of preference parameters
often yields more information about the
characteristics of the measurement error than
other econometric applications.
Linking to Economic Theory
• Economic theory gives precise cardinal
definitions of elasticities and other parameters
that make questions about preference
parameters quantitative.
• This distinguishes economics from psychology,
which typically has only ordinal concepts.
• Enjoying this strength of economic theory
requires designing hypothetical situations to
match the economic theory as closely as
possible.
Survey Mechanics
• Getting a representative sample.
• Getting a high response rate in order to avoid
serious biases.
• Taking into account the advantages and
disadvantages of different modes.
– In person (highest quality, highest cost)
– Mail (low response rates a problem, at least in the
U.S., and no interviewer feedback on how
respondents are handling questions)
– Phone (often the interior optimum)
– Internet (emerging method with great potential)
Case Study #1: Risk Aversion
• Question Design:
– Risk over permanent income to get at the
underlying utility function.
– Discrete choices to reduce the cognitive
burden.
– Rewritten to avoid status quo bias.
– Vignette to motivate the hypothetical situation.
Case Study #1: Risk Aversion
(continued)
• Implementation
– Asked in multiple waves to get a measure of
measurement error. Imputation of a cardinal
value to each category taking this into
account.
– Econometric adjustment to OLS that takes
into account the measurement error and its.
– Starting point and order of unfolding varied.
– Small risk questions.
– Variant questions for retired respondents.
Case Study #2: Labor Supply
• Question Design:
– We use the theoretical relationship relationship
between income and substitution effects.
– Income effects are the easiest for people to think
about.
– Winning the lottery is an obvious vignette, since
people really think about that possibility.
– The lottery is large and an easily understood size.
– The decision is broken down into quit or not, then if
not quit, reduce hours or not and by how much.
– The question sequence is designed to match the
family structure.
Case Study #2: Labor Supply
(continued)
• Implementation
– Asked in multiple waves of the HRS to allow
us to get at measurement error. (This aspect
is still unanalyzed.)
– We make an analytical and econometric
adjustment for the special status of quitting.
– We have put some variant questions on the
Survey of Consumers, but due to lack of
funding for survey time, only a fraction of
everything that should be looked at.
Case Study #3: Time Preference
and the EIS
• Question Design:
– Choice over consumption profiles to get at
preferences. (In contrast to choices over money that
get at the individual’s shadow interest rate.)
– Limitation to two time periods and discrete choices to
reduce the cognitive burden. Elicitation of second
choices to gain more total information.
– Bar graphs to make the numbers more vivid.
– The preamble is a compromise, trying to control for
inflation and health care costs but not try the
respondent’s patience.
Case Study #3: Time
Preference and the EIS
(continued)
• Initial Implementation
– Initial implementation on the Wave I of the
HRS had an inconsistency between the
consumption growth rates and the ABCDE
labels. Also there was not enough resolution.
In person interviews allowed use of graphs.
Case Study #3: Time
Preference and the EIS
(continued)
• Revised Implementation
– The HRS has a mailout that has a reasonable
response rate because of the preexisting relationship.
We implemented a revised version of the question
with more resolution and a systematic relationship
between ABCDE labels and consumption growth
rates.
– In the ongoing analysis, we have statistically modeled
status quo bias (in this case the tendency to continue
answering the same letter, regardless of true
underlying preferences) and measurement error. The
procedure for dealing with the measurement error
also easily handles the discrete choice aspect.
Case Study #3: Time
Preference and the EIS
(continued)
• Internet redesign and implementation
– Bob Willis has headed a project
experimenting with internet interviewing.
– This will allow us to test variants of the time
preference/intertemporal substitution
question, including a variant with continuous
choice over consumption bars using a mouse.
– This is still in the design stage. Technical
hurdles remain.
Where to Go From Here
• Issues in the Choice of a Strategy
• Overall Strategy Recommendation
• High Priority Topic Areas
Issues in the Choice of a Strategy
• Key Question #1: How much time and
effort should be spent nailing down each
preference parameter? (The tradeoff
between thoroughness and getting many
things done all at once.)
• Key Question #2: What is the best mode?
• Key Question #3: What is the best way to
tap into existing wisdom and resources for
survey measurement?
Overall Strategy Recommendation
#1: Put Thoroughness First
• Our discipline of Economics is skeptical enough of
survey measures of preference parameters that it is
important to be thorough and do the best job possible in
measuring each preference parameter to have an
impact. Getting many things done all at once is of no
avail if the answers are not seen as credible.
• Even a booster of survey measures of preference
parameters should only be fully persuaded of the results
after doing the most careful design, implementation and
analysis possible.
• In the future these measures might guide the policy for
nations.
• Since surveys in many nations can use the measures, it
is worth paying a high fixed cost at the design stage.
Overall Strategy Recommendation
#2: Phone Survey Initially +
Eventually an Internet Survey.
• Getting a representative sample is quite
valuable given the interest in the population
average values of preference parameters. This
can be achieved well be either phone or inperson surveys, but phone surveys are cheaper
and more practical.
• Internet surveys can do everything mail surveys
can do and more and are more flexible for
experimental work. This is an emerging mode.
Efforts are being made to figure out how to get
more representative samples in this mode.
Overall Strategy Recommendation
#3: Piggyback on Existing Surveys
• The Survey of Consumers provides important
services
– The Survey of Consumers handles the practical side
of pretesting, getting a representative sample and
doing the interviewing.
– It provides expertise in question design based on long
experience.
– Its staff is easy to work with. They are used to riders.
Overall Strategy Recommendation
#3: Piggyback on Existing Surveys
(continued)
• The Survey of Consumers has several structural
advantages
– It provides basic demographic data for free, as well as
some expectational data.
– It has a panel structure.
– It is in the field every month, so it is possible to have a
useful feedback loop.
– It is part of an academic center of survey research,
with access to all the intellectual resources of the
Survey Research Center.
Overall Strategy Recommendation
#3: Piggyback on Existing Surveys
(continued)
• The Survey of Consumers is a good launching
pad for other efforts.
– An existing question could generate a representative
sample of internet users under 50.
– Questions that are successful on the Survey of
Consumers and generate interesting analyses are
natural candidates for modules or even core inclusion
on the HRS and PSID.
– Once questions are fully developed, the machinery of
the Survey of Consumers can be used to run an
entirely new phone surveys if desired.
High Priority Topic Areas (In Order
of Priority, as I See It)
•
•
•
•
•
Happiness
Aspects of Declining Marginal Utility
Labor Supply Elasticities
The Marginal Propensity to Consume
Aspects of the Value Function
Happiness
• Recent research suggests that studying the
time-series properties of happiness is crucial
• Because it is in the field continuously, and has
information on the exact day people answer
questions adding a few questions to measure
happiness on the Monthly Survey of Consumers
would be of great value in seeing the effects of
news events on happiness.
• Most other questions can be asked once or
twice---this really should be on every month, for
as far in the future as can be arranged.
• The needed measures exist.
Aspects of Declining Marginal
Utility
• In the usual theory, all of the following are said to
reflect primarily the rate at which the marginal
utility of consumption declines
–
–
–
–
–
risk aversion
resistance to intertemporal substitution
the income elasticity of the statistical value of a life
the long-run wage elasticity of labor supply
the elasticity of altruism with respect to subsidies or to
leaky buckets
Aspects of Declining Marginal
Utility (continued)
• Each of these suggests a different survey
measure.
• If the usual theory is correct, data using all these
different survey measures would allow one to pin
down the rate at which the marginal utility of
consumption declines much more accurately.
• If the usual theory is incorrect, these separate
measures will indicate not only the failure, but
the nature and direction of the failure.
• The state of the art is well advanced for these
kind of survey measures. This set of
measurements could be done right away.
Beyond Declining Marginal Utility
• Some of the techniques for measuring the
rate at which marginal utility declines (u’’)
can be adapted to measure higher
derivatives of the utility function in a direct
way.
• The value of such measurement is great
because it has been even harder to
identify these higher order parameters by
existing data.
Labor Supply Elasticities
• The income elasticity of labor supply, the Frisch elasticity
of labor supply and the long-run elasticity of labor supply
are all of great practical importance.
• Separate measures of these different concepts are
possible, allowing a test of the theoretical relationship
between them.
• Also of interest:
– the relationship between consumption and labor in the utility
function
– the effect of job characteristics on labor supply
– endogenous age of retirement
• The state of the art is also getting to be well-advanced
here, though these are unavoidably more difficult than
the typical question in the previous topic area.
The Marginal Propensity to
Consume
• The MPC turns out to be quite difficult to
measure. Our initial efforts had serious
problems.
– The reported MPC’s for permanent and transitory
shocks were quite close to each other.
– When asked to distribute permanent income between
consumption, saving for early retirement, ordinary
saving, giving to relatives and giving to others, the
average MPC was about 24%.
– We interpret this as the 1/n phenomenon.
The Marginal Propensity to
Consume (continued)
• There is a variety of evidence in the literature for mental
accounting.
• Shapiro and Slemrod have used the Survey of
Consumers to study the impact of tax rebates, tax cuts
and withholding changes. The results are intriguing but
raise as many questions as they answer.
• Basic research is needed before we can accurately
measure the MPC.
• The intellectual resources exist to pursue that basic
research given funding for the needed survey data.
Aspects of the Value Function
• In measuring preference parameters, the idea is
to construct hypothetical situations that abstract
from aspects of an individual’s situation other
than preferences.
• It is also interesting to obtain measures that mix
together aspects of preferences and aspects of
the situation---especially when the relevant
aspects of preferences are separately measured
for comparison.
• When we know how to measure these well, time
series could be especially interesting.
Aspects of the Value Function
(continued)
• Examples:
– The absolute risk aversion of the value
function (risk version over dollar or yen
gambles)
– The marginal propensity to consume
– The shadow interest rate
– The shadow wage.
Nonstandard Arguments of the
Value Function
• This is quite speculative, but one could
imagine trying to measure nonstandard
arguments of the value function. For
example:
– intensity of work effort
– habits
– cumulated fatigue
Summary
• Survey measurement of preference
parameters has great promise.
• More generally, many things can be
measured that have never been measured
before.
• If we build a strong foundation for the
future, survey methods can transform the
conduct of economics in the 21st century.
• There are fascinating places to start.