Digitized by the Internet Archive
in
2011 with funding from
Boston Library Consortium
Member
Libraries
http://www.archive.org/details/dopeoplemeanwhatOObert
DEWEY
B31
1415
.Odr
Massachusetts Institute of Technology
Department of Economics
Working Paper Series
PEOPLE MEAN WHAT THEY SAY?
IMPLICATIONS FOR SUBJECTIVE SURVEY DATA
DO
Marianne Bertrand and
Sendhil Mullainathan
Working Paper 01 -04
January 2001
Room
E52-251
50 Memorial Drive
Cambridge,
MA 02142
This paper can be downloaded without charge from the
Social Science Research Network Paper Collection at
http://papers.ssrn.com/paper.taf7abstract id=260 131
1
Massachusetts Institute of Technology
Department of Economics
Working Paper Series
PEOPLE MEAN WHAT THEY SAY?
IMPLICATIONS FOR SUBJECTIVE SURVEY DATA
DO
Marianne Bertrand and
Sendhil Mullainathan
Working Paper 01 -04
January 2001
Room
E52-251
50 Memorial Drive
Cambridge,
MA 02142
This paper can be downloaded without charge from the
Social Science Research Network Paper Collection at
http://papers.ssrn.com/paper.taf7abstract id=260 1 3
MG 2 2
20o7
LIBRARIES
DO PEOPLE MEAN WHAT THEY SAY?
IMPLICATIONS FOR SUBJECTIVE SURVEY DATA
Marianne Bertrand and Sendhil Mullainathan*
Many
ing.
surveys contain a wealth of subjective questions that are at
Examples include "How important
yourself?", or
"How
satisfied are
is
leisure time to you?",
you with your work?"
one data source that economists rarely use. In
fact,
"How
glance rather excit-
satisfied are
Yet despite easy
you with
availability, this is
the unwillingness to rely on such questions
marks an important divide between economists and other
This neglect does not come from disinterest.
first
social scientists.
Most economists would probably agree that
the variables these questions attempt to uncover are interesting and important. But they doubt
whether these questions
elicit
meaningful answers. These doubts
skepticism rather than on evidence.
are,
however, based on a priori
This ignores a large body of experimental and empirical
work that has investigated the meaningfulness of answers to these questions.
objective in this paper
is
to
summarize
this literature for
Our primary
an audience of economists, thereby
turning a vague implicit distrust into an explicit position grounded in facts. Having summarized
the findings, we integrate
them
into a
measurement
error
framework so as to understand what
they imply for empirical research relying on subjective data. Finally, in order to calibrate the
extent of the measurement error problem,
we perform some simple empirical work using
subjective questions.
I.
EVIDENCE ON SUBJECTIVE QUESTIONS
Cognitive Problems
specific
We
begin by summarizing the experimental evidence on
1
people answer survey questions.
can
affect
how people
A
set of
from the ordering of questions: whether question
One
cognitive factors affect the
One
X
is
first
preceded by question
reason for this ordering effect
may
elicit
certain
is
general?" and
first,
when
A
second issue
"How happy
that
is
later answers. In a
you with
life in
the dating question
came
are
came second, they were
basically
Apparently, the dating question induced people to focus on one aspect of their
an aspect that had undue
Another cognitive
researchers
it
or vice versa can
that people attempt to provide
When
often do you normally go out on a date?"
the answers to both were highly correlated but
uncorrelated.
life,
"How
Y
memories or attitudes, which then influence
striking study, respondents were asked two happiness questions:
comes
interesting manipulation
answers consistent with the ones they have already given in the survey.
prior questions
way
experiments has shown that simple manipulations
process and interpret questions.
substantially affect answers.
how
effects
effect is the
on their subsequent answer.
importance of question wording.
"Do you think the United States should
compared responses to two questions:
forbid public speeches against democracy?"
and "Do you think that the United States should
allow public speeches against democracy?" While
yes, public speeches
In one classic example,
more than
half of the respondents stated that
should be "forbidden," three quarters answered that no, public speeches
should not be "allowed"
.
Evidence of such wording
effects are
extremely common.
Cognitive problems also arise due to the scales presented to people.
German respondents were
asked
how many hours
the respondents were given a scale that begin with
of
<
TV
|
In an experiment,
they were watching per day. Half of
an hour and then proceeded
in half
hour
increments ending with
the
first five
The
hours.
other respondents were given the
answers were compressed so that
respondents given the
and a half of
4|+
TV
first set
per day.
32%
it
began with
—
TV
viewing from the
stating that the survey's purpose
An
%
of the
more than two hours
of the respondents given the second set of response alternatives
scale.
2 hours range, suggests to subjects that this
the scale
scale except
l\ hours. Only 16
of response alternatives reported watching
reported watching more than two hours and a half of
be inferring "normal"
<<
same
is
TV
The
per day. Respondents thus appear to
first scale,
amount
to estimate the
of
amount
TV
of
with a finer partition in the
viewing
TV
is
common.
In fact,
viewing greatly diminishes
effect.
even more fundamental problem
is
that respondents
may make
little
mental
effort in
the relevant information or by
answering the question, such as by not attempting to recall
all
not reading through the whole
As a consequence, the ordering of
list
of alternative responses.
response alternatives provided matter since subjects
alternatives in a
most and
list.
may simply
pick the
first
or last available
In the General Social Survey, for example, respondents are asked to
least desirable qualities that a child
surveyed people and gave them this
that subjects would rate the
first
list
may have
in either the
out of a
GSS
list
list
the
of 13 qualities. Researchers
order or in reverse order.
They found
or last listed qualities, whatever they were, as most important.
Social Desirability
Beyond purely cognitive
issues, the social
nature of the survey procedure also appears to
play a big role in shaping answers to subjective questioning. Respondents want to avoid looking
bad
in front of the interviewer.
A
famous example
is
that roughly
25%
of non-voters report
.
having voted immediately after an election. This over-reporting
value norms of political participation the most and those
Other studies have noted that
if
who
among
those that
originally intended
on voting.
is
strongest
one adds to a voting question a qualifier that "Many people do
not vote because something unexpectedly arose...," the discrepancy rate between self-reported
voting and actual voting drops.
Another example can be found
in the self-reporting of racial attitude.
Much
evidence sug-
gests people are unwilling to report prejudice. For example, reported prejudice increases
when
respondents believe they are being psychologically monitored for truth telling and decreases
when the survey
administered by a black person.
is
Non- Attitudes, Wrong Attitudes and Soft Attitudes
Perhaps the most devastating problem with subjective questions, however,
that attitudes
may
A
not "exist" in a coherent form.
ingly,
apart, the
55%
same subjects were asked about
the possibility
indication of such problems
is
that
For example, in two surveys spaced a few
measured attitudes are quite unstable over time.
months
first
is
their views
of the subjects reported different answers.
on government spending. Amaz-
Such low correlations at high frequencies
are quite representative.
Part of the problem comes from respondents' reluctance to admit lack of an attitude. Simply
because the surveyor
about
it.
is
asking the question, respondents believe that they should have an opinion
For example, researchers have shown that large minorities would respond to questions
about obscure or even
A second,
fictitious issues,
more profound, problem
such as providing opinions on countries that don't
is
that people
may often
exist.
be wrong about their "attitudes"
People
may
not really be good at forecasting their behavior or understanding
why they
did what
they did. In a well-known experiment, subjects are placed in a room where two ropes are hanging
from the ceiling and are asked to
tie
the two ropes together.
The two ropes
are sufficiently far
With no
apart than one cannot merely grab one by the hand and then grab the other one.
other information, few of the subjects are able to solve the problem. In a treatment group, the
experimenter accidentally bumps into one of the ropes, setting
solve the
problem
one on an up
arc.
in this case: subjects
final
swinging.
it
was the jostling by the experimenter that led them to the
and related problem
is
cognitive dissonance.
Subjects
and past
may
attitudes.
asked afterwards
how they
liked the task, those
enjoyment. They likely reason to themselves,
for
"If
I
who
solution.
report (and even
feel)
In one experiment, in-
dividuals are asked to perform a tedious task and then paid either very
it
people
and grab
see that they can set the ropes swinging
attitudes that are consistent with their behavior
When
Many more
Yet when they are debriefed and asked how they solved the problem, few of
the subjects recognize that
A
now
it
little
or a lot for
it.
are paid very little report greater
didn't enjoy the task,
why would
I
have done
nothing?" Rather than admit that they should just have told the experimenter that they
were leaving, they prefer to think that the task was actually interesting. In this case, behavior
shapes attitudes and not the other way around.
II.
A MEASUREMENT ERROR PERSPECTIVE
What do
these findings imply for statistical work using subjective data?
Let us adopt a
measurement error perspective and assume that reported attitudes equal true attitudes plus
some
error term,
A =
A*
+
e.
Statistically,
we
readily understand the case
where
e is
white
The above evidence however suggests two important ways
noise.
error in attitude questions will be
more than white
noise. First, the
in
which the measurement
mean
of the error term will
not necessarily be zero within a survey. For example, the fact that a survey uses "forbid" rather
than "allow"
Second,
in a question will affect answers.
many
of the findings in the literature
suggest that the error term will be correlated with observable and unobservable characteristics
of the individual.
groups
(e.g.
For example, the misreporting of voting
higher in certain demographic
is
those that place more social value on voting).
There are two types of analysis that can be performed with subjective variables:
attitudes to explain behavior or explaining attitudes themselves.
surement
affects
both types of analyses.
attitudes to explain behavior.
the true model
time,
is
Y represents
Yu
— a+
First,
Specifically,
(3Xit
an outcome of
suppose we estimate
where
Y —
it
a
+ bX + cA
it
it
,
while
represents individuals,
t
represents
represents observable characteristics,
Z
represents
t
X
examine how mismea-
will
suppose we are interested in using self-reported
+ jA* + 5Z
interest,
We
using
it
,
i
unobservable characteristics, and we assume for simplicity that
the estimated coefficient c compare to 7 given what
Z
is
orthogonal to X.
How
we have learned about measurement
will
error
in attitude questions?
White
zero.
noise in the
The
first
measurement of
measurement problem
A
will
produce an attenuation
c will
now
i.e.
listed above, a survey fixed effect, will
as long as the appropriate controls (such as year or survey specific
The second problem,
bias,
correlation with individual characteristics
X
a bias towards
produce no bias
dummies) are included.
and
Z
will create a bias:
include both the true effect of attitude and the fact that the measurement error
in
A
is
Hence, assuming that measurement error problems are
correlated with unobservables.
not dominant, subjective variables can be useful as control variables but care must be taken in
interpreting
it.
The estimated
Let us
to the second type of analysis,
themselves. For example,
suppose we estimate
is
is
self-reported. This
A =
it
a
+ bX +
it
e,
play a
while the true model
variables.
much more important
is
A*
t
= a + f3X + -yZ
it
role. Specifically,
now
X. For example, suppose we
As noted
earlier, this
X
is
far
helps predict "attitude"
more
means
only predicting the measurement error in attitude. So, one cannot argue as one
is
a problem that
it
is
is
much harder
is
a good thing, irrespective of causality. Second, this
to solve than an omitted variable bias problem. For example,
hard to see how an instrumental variable could resolve this
instrument that affects
X
But the
the fact that a rich background affects the reporting of the preference for
did before that simply helping to predict
that
.
the fact that measurement error
severely bias
severe than in the previous analysis. First, the fact that an
little if it is
it
bias.
money. Such a correlation could thus be purely spurious. Notice that this problem
very
2
increase loneliness. Specifically,
see that those from rich backgrounds have a greater preference for money.
reflect
closely related
measurement of attitudes no longer causes
correlated with individual characteristics will
might simply
is
but also the
where we are attempting to explain attitudes
we might ask whether high work hours
In this setup, the white noise in the
now
the attitude
effect of attitude
problem that we often encounter even with perfectly measured
now turn
other biases
does not only capture the
how
that influence
effect of other variables
to the causality
coefficient
X but not the measurement of attitude.
will likely affect
measurement
in a causal sense.
issue.
One would need an
But the above evidence
This makes
it
tells
us
very unlikely that such
an instrument could be found in most contexts.
To summarize, interpreting the experimental evidence
provides two important insights.
measures
may be
First, if the
in a
measurement
measurement error framework
error
is
small enough, subjective
helpful as independent variables in predicting outcomes, with the caveat that
the coefficients must be interpreted with care. Second, subjective variables cannot reasonably be
used as dependent variables given that the measurement error likely correlates in a very causal
way with the explanatory
III.
variables.
HOW MUCH NOISE IS
THERE?
This leaves the important quantitative question:
the subjective questions
we might be
interested in?
how much white
Can we
noise error
in fact gain
is
there in
anything by adding
responses to subjective questions to our econometric models?
To
assess this,
in school in
we turn
to the
High School
&:
Beyond's Senior sample, which surveyed seniors
1980 and then followed them every two years until 1986. This sample provides us
with a set of subjective and objective variables in each of these waves.
In the
first
8
columns of Table
1,
we
correlate answers to a set of attitude variables with
future income (thereby removing mechanical correlations with current income).
Table corresponds to a separate regression. The dependent variable
row
1,
we add
as control the sex, race
cell in
the
log(salary) in 1985.
In
and educational attainment of the respondent. Answers
to the subjective questions clearly help predict individual income.
very intuitive.
is
Each
A
set of correlations are
People that value money or a steady job more earn more.
social goals such as correcting inequalities
around them earn
less.
People that value
People that have a positive
Maybe somewhat
attitude towards themselves earn more.
care about their family earn substantially more.
intriguing,
Even more
we
find that people that
intriguing, people that value leisure
time also earn more. The second row shows that respondents' attitudes do not simply proxy
for objective family
income
3,
background characteristics. Controlling
in the senior year
for parents' education
and family
does not weaken the predictive power of the attitude variables. In row
attitude questions stay predictive of future income even after one controls for
we show that
current individual income.
As a whole,
these results suggest that noise does not dominate the measurement of these sub-
jective questions. Attitudes actually predict
characteristics.
Of
we
course,
income even beyond past income and background
are not arguing for causality, merely that attitude variables add
explanatory power.
Finally,
one might wonder to what extent these variables are conveying any information
beyond fixed individual
characteristics.
School and Beyond survey.
but also add person fixed
We
effects.
In
row
4,
we
exploit the panel nature of the
High
rerun the standard regressions with lagged attitude measures
Most
of the effects previously discussed disappear, except for
the importance of work and the importance of having a steady job (marginally significant).
therefore does not appear that changes in attitudes have as
themselves.
much
It
predictive power as attitudes
Thus, while these attitude questions are helpful in explaining fixed differences
between individuals, changes
in
reported attitudes are not helpful in explaining changes in
outcomes.
In
column
9,
we
investigate
whether answers to reservation wage questions are correlated
with future income. Are individuals that report higher reservation wage today
more
in the future?
We
see a very strong relationship
likely to
earn
between reservation wage and future
income, even after controlling for the individual's education, sex and race. This holds true even
if
controls for family background (row 2) or family
we add
background and current income (row
3).
However, changes in reported reservation wages do not help predict changes
4).
In
Changes
in reported reservation
Finally, in
column
is
we
when
trying to predict individual income.
wages however provide no information about changes
we ask whether answers
10,
future job turnover. Again,
IV.
income (row
summary, answers to reservation wage questions do appear to capture some unobserved
individual characteristics and might be worth including
whole"
in
in income.
to job satisfaction questions help predict
find that people's self- reported satisfaction with their job "as a
3
a strong predictor of their probability of changing job or not in the future.
CONCLUSION
Four main messages emerge from this discussion.
First,
a large experimental literature by
and large supports economists' skepticism of subjective questions. Second, put
in
an econometric
framework, these findings cast serious doubts on attempts to use subjective data as dependent
variables because the
measurement error appears to correlate with a
and behaviors. For example, a drop
in reported racism over
reluctance to report racism.
much
Since
data as dependent variables, this
note, these data
may be
is
time
may
simply
reflect
an increased
of the interesting applications would likely use these
a rather pessimistic conclusion.
useful as explanatory variables.
interpreting the results since the findings
large set of characteristics
may not be causal.
10
Third, and on a brighter
One must,
however, take care in
Finally, our empirical
work suggests
that subjective variables are in practice useful for explaining differences in behavior across
individuals.
Changes
in
answers to these questions, however, do not appear useful in explaining
changes in behavior.
REFERENCES
Bertrand, Marianne and Mullainathan, Sendhil.
"Do People Mean What They Say?
Implications For Subjective Survey Data." Mimeo, University of Chicago, 2000.
Sudman, Seymour; Bradburn, Norman M. and Schwarz, Norbert. Thinking
The Application
of Cognitive Processes to Survey
Methodology
.
San Francisco:
about questions:
Jossey-Bass
Publishers, 1996.
Tanur, Judith
New
M.
Questions About Questions: Inquiries into the Cognitive Bases of Surveys
York: Russell Sage Foundation, 1992.
11
.
TABLE
Log Wage
Log Wage
Value
Value
Work
Money
Effect of Attitude Questions on Future
Dependent Variable:
Log Wage
Log Wage
Log Wage
Log Wage
1:
Value
Steady
Value
Family
Value
Value
Friends
Leisure
Job
Outcomes'1
Log Wage
Log Wage
Log Wage
Stayer
Value
Positive
Reservation
Satisfied
Wage
with
Social
Towards
Causes
Self
Job
Additional Controls:
Demographics
Demographics
Family Background
.08
.08
.13
.07
-.01
.09
-.08
.07
.17
.08
(.03)
(.02)
(.02)
(-02)
(.02)
(.02)
(.02)
(.02)
(.03)
(.01)
.07
.08
.12
.06
-.01
.08
-.08
.06
.17
.08
8c
(.03)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.03)
(.01)
Demographics
Family Background
U Log Wage in 1983
.07
.05
.12
.03
.02
.06
-.06
.05
.14
.08
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.03)
(.01)
.06
-.015
.03
.003
.00
.02
-.01
-.00
-.00
_
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
(.02)
&:
Person F.E.
Notes;
1.
2.
3.
Data Source: High School and Beyond (Seniors). Demographic characteristics include education, sex and race. Family background characteristics
include father education, mother education and family income in senior year (7 categories). "Stayer" is a dummy variable which equals one if there
os no job change between second and third follow up. "Work"
Each
cell
Except
in
corresponds to a separate regression. Standard errors are in parentheses.
row
4,
outcomes are from the third follow-up survey and attitudes are from the second follow-up. Row
The regressions in row 4 include survey fixed effects.
available survey periods.
12
4
reports panel regressions on
all
Footnotes
*
1.
Graduate School of Business, University of Chicago,
Due
to space constraints,
we
will just
NBER
and CEPR;
MIT and NBER.
mention two books that are a good source
for a
review of the experimental evidence: Tanur (1992) and Sudman, Bradburn and Schwarz
(1996).
A
fuller list of references
can be gotten
in the full version of this
paper (Bertrand
and Mullainathan 2000).
2.
An
extreme example of
this occurs
variable of interest itself as
is
report a lower preference for
when
the measurement error
is
correlated with the
suggested by cognitive dissonance. For example, people
money
if
they are making
less
money. This
is
may
a case of pure
reverse causation.
3.
In this case,
we
was only asked
are not able to study a fixed effect
in the
model
as the job satisfaction question
second and third follow up of the data.
13
561*5
037
k
>
Date Due
Lib-26-67
MIT LIBRARIES
3 9080 02246 0791