Showcard use and mixed-mode administration in
longitudinal surveys: Are the data comparable?
Beth Fisher, Ned English, Catherine Haggerty – NORC at the University of Chicago
ABSTRACT
Making Connections is a study of ten disadvantaged US urban communities,
funded by the Annie E. Casey Foundation. This poster examines one aspect
of survey administration that has implications for longitudinal surveys using a
mixed-mode design: differences in responses given in person with the aid of
showcards compared to those given over the telephone without showcard
use and the substantive impact of multi-mode administration.
In the first wave of data collection for Making Connections, all interviews
were completed in person. Due to cost constraints, subsequent interviews
were completed by telephone when a telephone number was available and
the respondent was willing to participate by telephone. When interviews
were conducted by telephone, all response options for each question were
read to the respondent. If the interview was completed in person, response
options were read and respondents were shown a card that enumerated the
response options. Questions selected for showcard facilitation were those
with a long list of possible responses or those with a scaled response.
Showcards are frequently used in surveys because they ease the burden of
remembering response options and therefore improve data quality by
reducing satisficing or primacy and recency effects (Holbrook, Green, and
Krosnick, 2003).
We compare the substantive data collected within and across the two modes
of administration between the in-person only group and the inperson/telephone group to explore the potential unintended consequences
of attempts to ease respondent burden through the use of showcards and
other interview aids. We draw from prior comparisons of interview modes
(Roberts, Jäckle, and Lynn 2007, etc.) to investigate differences associated
with telephone and in-person interviewing for questions where showcards
were used and where they were not.
SCALE QUESTIONS - COMPARISONS
BETWEEN MODES
DEMOGRAPHICS
W2 In Person
Total Number of Households
W2 by Phone
54% (1172)
46% (1025)
2.87
2.70
47
44
49.06
50.08
Percentage of Female Respondents
65
70
Percentage of Hispanic Respondents
35a
24a
Mean Size of Household
Percent of Households with Children
Mean Age of Respondent
Percentage of Respondents with a High
School Degree or Higher
65b
79b
Percentage of Respondents who are
Employed
51
54
Percent of Households That Have Had
their Phone Cut Off in the Last Twelve
Months
18c
8c
Percentage of Respondents Rated as
Having Fair or Poor Understanding of
Questions by the Interviewer
14d
11d
Percent of Respondents Rated as Being
Friendly and Interested by the Interviewer
98
• Making Connections is a neighborhood-based longitudinal survey
• Interviews completed in person only in W1, by phone and in
person in W2
• Mode in W2 was based on respondent preference i.e. not random
• Questions concerning neighborhood engagement, children, income
and assets, and perceptions of neighborhood
• Approximately 45% of households stayed in the same housing unit in
both waves and of those 90% had the same respondent in both
waves
• Of these households, about half completed a second interview over
the phone
Research Questions:
• How does this affect responses to scales and questions with many
response categories?
• Does this make a difference in responses given between waves and
for different modes?
Summary of Chart types for Scaled Response Options
No. of questions with
differences between new
categories
Collapsed Into
following
categories,
compared
between modes
Scale Type
NonSignificant
1) Five options, all
listed on showcard
Selecting ‘short’ or
‘long’ option
(eg. agree vs
strongly agree)
2) Seven options
from 1 to 7.
Agree/disagree,
satisfied/dissatisfied,
or likely/unlikely with
three categories
labeled
Selecting labeled
category or an
unlabeled
category
3) Seven options
from 0 to 6 for
rare/common
occurrence with
three categories
labeled, including
‘does not occur’
Significant
3
3
Selecting labeled
category or an
unlabeled
category
96
5
10
Higher proportion of
in-person Rs selected
‘long’ responses for
all questions
5
Higher proportion of
in-person Rs selected
labeled response
categories for all
questions
6
Higher proportion of
in-person Rs selected
labeled response
categories for all
questions but one,
which had essentially
equal proportions
Live in closeknit home *
60%
* Subscript between pairs indicates significant difference at p<.05
50%
People willing
to help
neighbors *
40%
People
generally don't
get along *
30%
20%
People do not
share my
values *
10%
Q6.48 - Percent of Respondents in Wave 2 Reporting How
They Would Pay for Emergency Bill by Wave 2 Mode
0%
Strongly
Agree
Savings*
Credit card*
Agree
Neither Disagree Strongly
Neither
Disagree
Agree nor
Disagree
People in
neighborhood
can be trusted *
* Indicates difference is significant at p<.05. Comparisons of differences outlined in chart above.
Borrow from family
Q1.5 - Telephone – Proportion of Responses by
Category for Five Point Scale Wave 2 Questions
Borrow from friends
*
Sell something
Get payday loan*
In person - Yes
By phone - Yes
Difference
Pawn something*
Borrow from bank*
60%
Positive difference is
associated with a higher
proportion of Telephone
respondents answering
yes to the particular
category
Would not pay
Mutual Assistance Savings
Other*
10%
20%
30%
40%
50%
30%
Agree
Neither
Neither Disagree Strongly
Agree nor
Disagree
Disagree
People do not share my
values
People in neighborhood
can be trusted
-6%
-4%
-2%
0%
2%
4%
6%
8%
Neighborhood is safe
for children*
3
Strongly
Agree
Dental ins. for R*
In person - Yes
By phone - Yes
Difference
Life insurance
Long term disability ins.*
Feel safe at home at
night*
2
Q1.5 - Difference Between In-Person and Telephone
Proportions for Five Point Scale Wave 2 Questions
Dental ins. for family
Analysis of Scale Questions
• Compare the categories selected by respondents interviewed in
person and those interviewed by phone in W2 for a variety of
questions
• Based on observed patterns, collapse responses into two categories
• Use binomial tests for significance testing of proportions in the two
collapsed categories
• Using W2 mode categories, compare W1 responses in the same
manner to look for similar patterns in responses
People generally don't
get along
Neither Agree
nor Disagree
Disagree very
strongly
Strongly
Agree
Health ins. for family
Analysis of List Questions
• Compare the proportion of “yes” responses given for lists of questions
that use showcards between respondents interviewed in person and
those interviewed by phone in W2
• Calculate differences to identify trends in response patterns
• Use binomial tests for significance testing of proportions for “yes”
responses for each category in list questions
• Using W2 mode categories, compare W1 responses in the same
manner to look for similar patterns in responses
Agree
0%
* Indicates difference is significant at p<.05
Positive difference
associated with
increased proportion of
in-person respondents
selecting category.
Agree
Negative difference
associated with
increased proportion of
telephone respondents
selecting category.
Paid sick days
Disagree
Unpaid sick/leave days*
Paid vacation*
Feel safe in
neighborhood during
day
Do not
have feelings
either way
Would stop to speak to
someone asking for
directions*
5
Neither
Agree
nor
Neither
Disagree
Retirement program*
-10%
People willing to help
neighbors
Q2.5 - Difference Between Wave 2 In-Person and Telephone
Group Proportions for Seven Point Scale Wave 1 Questions
20%
60%
Health ins. for R*
Identify households
• Do not include any households that completed an interview through a
proxy, or for which an interview was completed with a non-English
speaker without a translated instrument in either wave.
• Use households that have not moved between waves and have the
same respondent selected in both Waves 1 and 2 (n=2,197)
• Compare W1 and W2 responses based on the mode they completed
the survey in during Wave 2 (in person or phone)
Live in close-knit
neighborhood*
Strongly
Agree
-8%
Q6.8 - Percent of Respondents in Wave 2 Reporting
Benefits They Receive from Employment by Wave 2 Mode
METHODOLOGY
Q1.5 - Difference Between Wave 2 In-Person and Telephone
Group Proportions for Five Point Scale Wave 1 Questions
Strongly
Disagree
10%
0%
SCALE QUESTIONS – COMPARISONS
BETWEEN WAVES
Disagree
50%
40%
Consumer finance enterprise*
-10%
Trend
Q1.5 - In Person – Proportion of Responses by
Category for Five Point Scale Wave 2 Questions
LIST QUESTIONS – COMPARISONS
BETWEEN MODES AND WAVES
INTRODUCTION
SHOWCARD EXAMPLES
Most children go trickor-treating in this
neighborhood
6
Agree very
strongly
-8%
-6%
-4%
-2%
0%
2%
4%
6%
Criminal activity
comimtted by people
living outside of
neighborhood
Strongly Disagree
0%
10%
20%
30%
40%
50%
60%
70%
80%
-15%
* Indicates difference is significant at p<.05
-10%
-5%
0%
5%
10%
15%
Q2.5 - Difference Between In-Person and Telephone
Proportions for Seven Point Scale Wave 2 Questions
Q6.8 – Percent of Respondents in Wave 1 Reporting
Benefits They Receive from Employment by Wave 2 Mode
Neighborhood is safe
for children
Disagree very
strongly
Health ins. for R*
Feel safe at home at
night
2
Health ins. for family
Feel safe in
neighborhood during
day
3
Dental ins. for R*
W2 In-person, W1 - Yes
Dental ins. for family
W2 By phone, W1 - Yes
Life insurance
Do not have
feelings either way
Would stop to speak
* asking for
to someone
directions
Difference
Long term disability ins.
5
Most children go trickor-treating in this
*
neighborhood
Retirement program*
Paid sick days*
6
Unpaid sick/leave days*
Agree very
strongly
Paid vacation
-10% 0% 10% 20% 30%
* Indicates difference is significant at p<.05
40%
50%
60%
70%
80%
-8%
-4%
0%
4%
8%
Criminal activity
comimtted by people
living outside of
neighborhood *
* Indicates difference is significant at p<.05. Comparisons of differences outlined in chart above.
DISCUSSION
Overall, there are response differences between the two groups. However, it is
important to note that the interview mode was determined through non-random
procedures resulting in selection bias. Because of this, demographic or other factors
may be responsible for some of the differences in response patterns between modes.
For list questions, mode does not appear to make a substantial difference when taking
W1 response patterns into consideration. The differences between proportions of each
group responding ‘yes’ to an item may be the result of demographic differences
between the two mode groups. With respect to scaled question responses, how
showcard use influences the data appears to depend on the particular scale. For
questions where all options are listed on the showcard, it seems to encourage
selection of ‘longer’ responses among in-person respondents. For other scales,
however, in-person respondents selected options that were labeled on the showcards
in higher proportions than those interviewed by telephone, who would not be
influenced by the visual cues associated with labels on the showcards.
Future research based on this preliminary investigation should include models to
control for interview mode and demographic variables. As for practical implications, it
is possible to build in procedures in subsequent waves where showcards are mailed to
respondents in case they complete by phone. However, based on this research,
inclusion of certain lists or scales in the showcards may not be beneficial. An
experimental design would help to determine how showcards affect response patterns
and also help to determine the showcard presentation that best facilitates unbiased
data capture.