“WOULD YOU STILL HANG UP ON THIS POLL IF YOU KNEW . . . :”
EXPERIMENTS TO IMPROVE THE DESIGN OF
POLITICAL MESSAGE TESTING POLLS
Thomas M. Guterbock
Director, Center for Survey Research
University of Virginia
TomG@virginia.edu
Deborah L. Rexrode
Research Analyst, Center for Survey Research
University of Virginia
dlr3r@virginia.edu
Samantha Luks
Vice-President for Special Projects
YouGovPolimetrix
sam.luks@yougov.com
Presented at the Annual Meeting of the
American Association for Public Opinion Research
Hollywood, Florida
May 15, 2009
UPDATED ABSTRACT
May 2009
Political message testing polls occasionally generate trouble: complaints to
AAPOR, negative press attention, denunciations from opposing campaigns, and criticism
from others in the survey profession. These polls also present ethical issues, since they
may not always adequately secure informed consent. Some generate large numbers of
break-offs from annoyed respondents, suggesting that significant numbers are unsatisfied
with their experience in interviews of this type.
The exact form of message-testing surveys varies substantially across firms and
campaigns. However, there has been no published research about which features of these
surveys might affect the experience of those who respond. We report on a multi-factor
experiments testing whether specific changes in the design of a message testing poll can
produce improvement in the perception of the interview among respondents.
The vehicle for these experiments is a message testing poll about the generic 2010
Congressional election. Three features of the questionnaire are manipulated in a trial
fielded by Polimetrix/YouGov in March 2009. (1) An abrupt transition from ordinary
preference questions to persuasive or “push” questions is tested against more informative
transitions that forewarn the respondent that politically charged messages will ensue. (2)
The sequencing and degree of balance (between positive messages about one party’s
candidate for Congress and negative messages about the other) is varied. (3) The
questions that are asked after each persuasive item are varied, ranging from questions
about changed voting intention to less direct questions that ask about the convincingness, believability and importance of each item.
The dependent variables are responses to a closing interrogatory in which
respondents assess the fairness and availability of the questions, whether they felt fully
informed about the interview, their degree of concern with partisan use of the survey
results, whether the interview was comfortable or stressful, and their willingness to be
interviewed in the future. We present and discuss results showing that these judgments
are strongly affected by the features of the questionnaire. The results demonstrate that
respondent survey experience in message testing polls can be improved by betterment of
poll design.
2
SUMMARY OF EXPERIMENTAL DESIGN
FACTOR 1
TRANSITIONAL INTRODUCTION
Control or Generic Group:
No Introduction
Intro 1:
Here are some statements you might hear from a political candidate running for office. Your
thoughts and opinions in response to these statements are an important part of our research.
Intro 2:
The following statements are the sort that you might hear in a political campaign, or a campaign
commercial These statements are different from what you might expect from a usual poll or
survey. You might not agree with these statements, and some are negative. We are testing
people’s reactions to these statements and we would like to warn you that these statements could
cause some people to react strongly. This is ok. Your thoughts and opinions in response to these
questions are an important part of our research.
FACTOR 2
DESIGN BALANCE
Each design will present three blocks of message items. The first block will contain messages
about the favored party, and the second and third blocks will contain messages about the
opposing party. The strongest or least “fair” negative messages will be included in the third
block.
Unbalanced design: 6 positive messages about favored party.
7 negative items about opposing party
7 more negative items about opposing party
Partially balanced design:
4 positive and 2 negative messages about favored party
2 positive and 5 negative messages about opposing party
1 positive and 6 negative messages about opposing party
3
FACTOR 3
TEST QUESTIONS
Test 1:
When you hear the following statements about the Democrats in Congress, does knowing about
this make you more likely or less likely to vote for a Democratic party candidate, and how
strongly do you feel about that?
1 Much more likely
2 Somewhat more likely
3 Somewhat less likely
4 Much less likely
5 No opinion
Test 2:
When you read the following statements about the Democrats in Congress, how convincing are
they? (Positive statements)
1 Very convincing
2 Somewhat convincing
3 Somewhat unconvincing
4 Not convincing
5 No opinion
When you read the following statements about the Democrats in Congress, how much doubt do
they create? (Negative statements)
1 Serious doubt
2 Some doubt
3 Very little doubt
4 No doubts
5 No opinion
Test 3:
When you read the following statements about the Democrats in Congress, how believable do
you think this statement is?
1 Very believable
2 Somewhat believable
3 Somewhat unbelievable
4 Not believable
5 No opinion
Considering these same statements, for you as a voter, how important is it for you to know this
information?
1 Very important
2 Somewhat important
3 Not important
4 No opinion
4
EXAMPLE OF PARTISAN STATEMENTS
Republican
Republicans have supported the war in Iraq which has taken a devastating toll on our military, our
economy, and our reputation around the world. (Neg – War)
The Republican Party has always been firm and strong regarding the War in Iraq. A withdrawal
without completely securing our gains would be irresponsible. (Pos – War)
Republicans are working hard to strengthen our economy by advancing free trade with growing
economies around the world. (Pos – Foreign Trade)
Because of the strong Republican support for the unlawful invasion of Iraq several years ago,
America is now at odds with the rest of the world. (Neg – Diplomacy)
Republicans act as if peace in the Middle East is not an achievable goal. Therefore, they do not try to
facilitate any understanding and compromise. (Neg – Diplomacy)
Democratic
Democratic leaders’ emphasis on troop withdrawal in Iraq is undermining all that we have
accomplished there. If we pull out now, we may lose everything we have worked for. (Neg – War)
Democrats in Congress say we must strengthen America’s security by starting to reduce our troops in
Iraq in a responsible way and work with other nations to bring stability. (Pos – War)
Democrats work hard to keep American jobs on American soil. (Pos – Foreign Trade)
While diplomacy is appropriate in some circumstances, the democrats are too soft when it comes to
defense and need to recognize that military action is sometimes the best option. (Neg – Diplomacy)
There will never be peace in the Middle East as long as terrorists are not fully and universally
condemned, yet Democrats refuse to stand up to the Palestinians. (Neg – Diplomacy)
5
ATTRIBUTES OF QUESTIONS
Thinking about the questions you were asked related to Congress…
{FUTURE}
If we contacted you in the future to do another survey like this, do you think you would
participate?
1
2
3
4
Would definitely participate
Would probably participate
Would probably not participate
Would definitely not participate
{FAIR}
How fair were the questions about Congress?
1
2
3
4
5
Very fair
Somewhat fair
Somewhat unfair
Very unfair
Not sure/Don’t know
{BELIEVE}
How believable was the information presented about Congress?
1
2
3
4
5
Very believable
Somewhat believable
Somewhat unbelievable
Not believable at all
Not sure/Don’t know
{STRESS}
Was answering these questions comfortable or stressful? Did answering feel…
1
2
3
4
5
Very comfortable
Somewhat comfortable
Somewhat stressful
Very stressful
Neither comfortable nor stressful
{HOWUSED}
Do you think these survey results will be used to promote or oppose a particular party’s
congressional campaign?
a. Yes [GO TO SUPPORT]
b. No [GO TO MISLED]
6
{SUPPORT}
How concerned are you that the survey results might be used to promote a particular party’s
congressional campaign?
1
2
3
4
5
Very concerned
Somewhat concerned
Not very concerned
Not concerned
Not sure/Don’t know
{EXPECTED}
Thinking back to the introduction to this survey, were the questions what you expected?
1
2
3
4
Exactly what I expected
Close to what I expected
Somewhat different than I expected
Not at all what I expected [GO TO MISLED]
SCRIPT FOR PANEL PARTICIPANT DEBRIEFING AT THE COMPLETION OF THE
SURVEY:
A final note about this survey:
This survey was part of an academic experiment conducted by Polimetrix on behalf of the Center
for Survey Research at the University of Virginia to better understand how voters respond to
surveys that test political campaign messages. Such surveys are frequently conducted by
campaign consultants of both major political parties.
The University does not conduct partisan polling or research for parties or candidates. This
research does not reflect the political views of Polimetrix, the Center for Survey Research, or the
University of Virginia. This study has been approved by the Institutional Review Board of the
University.
The design of the study involves an equal number of participants from each political party. The
purpose of this survey is to test methods that would improve the practices of political polls and
improve the experience for respondents like you. The study utilizes different sequencing of
questions with some versions of the survey having a stronger negative or positive emphasis.
Some versions favor Republicans; some favor Democrats. Your reactions to these various
methods will help us develop more effective and responsible research methods for conducting
these types of polls.
Thank you very much for your assistance with our research
7
ATTRIBUTES BY TREATMENTS*
Crosstab
fairr Fairness of
questions about
Congress--recoded
1.00 very unfair
2.00
3.00
4.00
5.00
Total
Count
% within s elec ttreat
compare generic with
the better t reat ments
s omewhat unfair
Count
% within s elec ttreat
compare generic with
the better t reat ments
not s ure/don't know Count
% within s elec ttreat
compare generic with
the better t reat ments
s omewhat fair
Count
% within s elec ttreat
compare generic with
the better t reat ments
very fair
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
selectt reat compare generic
with the better treatments
2.00
balanc ed with
1.00
any transit ion
control/gener and alt ernate
ic treatment
tes t questions
73
10
Total
83
39.0%
4.2%
19.5%
30
20
50
16.0%
8.4%
11.7%
13
16
29
7.0%
6.7%
6.8%
28
75
103
15.0%
31.4%
24.2%
43
118
161
23.0%
49.4%
37.8%
187
239
426
100.0%
100.0%
100.0%
selectt reat compare generic
wit h the better treat ments
2.00
balanc ed with
1.00
any transit ion
control/gener and alt ernate
ic treat ment tes t questions
59
7
Total
66
Crosstab
believer Believabilit y of
info presented about
Congress-recoded
Total
1.00 not believable at all
Count
% within select treat
compare generic with
the bet ter t reatments
2.00 s omewhat
Count
believable
% within select treat
compare generic with
the bet ter t reatments
3.00 not s ure/ don't know Count
% within select treat
compare generic with
the bet ter t reatments
4.00 s omewhat
Count
believable
% within select treat
compare generic with
the bet ter t reatments
5.00 very believable
Count
% within select treat
compare generic with
the bet ter t reatments
Count
% within select treat
compare generic with
the bet ter t reatments
31.6%
2.9%
15.5%
31
31
62
16.6%
13.0%
14.6%
11
23
34
5.9%
9.7%
8.0%
57
106
163
30.5%
44.5%
38.4%
29
71
100
15.5%
29.8%
23.5%
187
238
425
100.0%
100.0%
100.0%
*Differences between treatments on all six of the attributes are significant.
8
Crosstab
expectedr
Questions
matched
expectations
1.00 not at all what I
expected
2.00 s omewhat
different than I expected
3.00 close to what I
expected
4.00 exactly what I
expected
Total
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
selecttreat compare generic
with the better treatments
2.00
balanced with
1.00
any transition
control/gener
and alternate
ic treatment
tes t questions
60
31
Total
91
32.8%
13.0%
21.6%
53
58
111
29.0%
24.3%
26.3%
53
115
168
29.0%
48.1%
39.8%
17
35
52
9.3%
14.6%
12.3%
183
239
422
100.0%
100.0%
100.0%
Crosstab
misuse
concern
that results
will aid a
campaign
Total
.00 won"t be used
Count
% within select treat
compare generic with
the bet ter t reat ments
1.00 not c oncerned, not Count
sure
% within select treat
compare generic with
the bet ter t reat ments
2.00 not very c onc erned Count
% within select treat
compare generic with
the bet ter t reat ments
3.00 somewhat
Count
concerned
% within select treat
compare generic with
the bet ter t reat ments
4.00 very c onc erned
Count
% within select treat
compare generic with
the bet ter t reat ments
Count
% within select treat
compare generic with
the bet ter t reat ments
selectt reat compare generic
wit h the better treat ments
2.00
balanc ed with
1.00
any transit ion
control/gener and alt ernate
ic treat ment tes t questions
58
102
Total
160
31.2%
43.2%
37.9%
25
42
67
13.4%
17.8%
15.9%
21
36
57
11.3%
15.3%
13.5%
36
34
70
19.4%
14.4%
16.6%
46
22
68
24.7%
9.3%
16.1%
186
236
422
100.0%
100.0%
100.0%
9
Crosstab
stress r int erview
comfortable or
stress ful--recoded in
ordinal sequence
1.00 very comfortable
2.00 s omewhat
comfortable
3.00 neither c omfortable
nor stressful
4.00 s omewhat st ress ful
5.00 very stres sful
Total
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
Count
% within s elec ttreat
compare generic with
the better t reat ments
selectt reat compare generic
with the better treatments
2.00
balanc ed with
1.00
any transit ion
control/gener and alt ernate
ic treatment tes t questions
40
97
Total
137
21.4%
40.4%
32.1%
32
60
92
17.1%
25.0%
21.5%
49
47
96
26.2%
19.6%
22.5%
51
32
83
27.3%
13.3%
19.4%
15
4
19
8.0%
1.7%
4.4%
187
240
427
100.0%
100.0%
100.0%
Crosstab
futurer
Participate in
future
surveys -reversed
1.00 would definitely
not participate
2.00 would probably
not participate
3.00 would probably
participate
4.00 would definitely
participate
Total
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
Count
% within selecttreat
compare generic with
the better treatments
selecttreat compare generic
with the better treatments
2.00
balanced with
1.00
any transition
control/gener
and alternate
ic treatment
tes t questions
6
0
Total
6
3.2%
.0%
1.4%
12
2
14
6.5%
.8%
3.3%
38
70
108
20.4%
29.2%
25.4%
130
168
298
69.9%
70.0%
70.0%
186
240
426
100.0%
100.0%
100.0%
10