ACSI
American Customer Satisfaction Index
TM
Does Interviewing Method Matter?
Comparing Consumer Satisfaction Results across Internet
and RDD Telephone Samples
Forrest V. Morgeson III, Ph.D.
Director of Research, American Customer Satisfaction Index
Barbara Everitt Bryant, Ph.D.
Research Scientist-Emerita, University of Michigan
Reg Baker
President, Market Strategies International
Presented at the 66th Annual American Association for Public Opinion Research Conference
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Research Questions and Findings
• Research Questions: Does interview method matter? Do the
results produced in a multi-industry consumer satisfaction
study differ significantly across a sample collected through
RDD/probability sampling and telephone interviewing, and
one collected via online panel/nonprobability sampling and
Internet interviewing?
• Research Design: We utilize a multi-method sample of
consumer satisfaction data, structural equation modeling
techniques, and two tests of difference to investigate the
significance of differences in survey responses across
samples drawn and interviewed using these two methods
• Findings: While some differences are observed, interview
method only marginally impacts the means of the survey
responses or the parameter estimates from the structural
models. Overall, the findings suggest that mixed-method
interviewing is feasible and reliable for consumer-oriented
survey research projects
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Overview of the ACSI
• Established in 1994, ACSI is the only standardized measure of
customer satisfaction in the U.S. economy, covering
approximately 225 companies in 45 industries and 10
economic sectors; companies measured account for roughly
one-third of the U.S. GDP
• 100+ departments and agencies of the U.S. federal
government also measured on an annual basis, along with
local and state government measures
• Results from all surveys are published monthly in various
media and on the ACSI website, www.theacsi.org
Structure of the ACSI
National
ACSI
Utilities
Information
Transportation &
Warehousing
Energy
Utilities
Airlines
U.S.
Postal
Service
Express
Delivery
Accommodation &
Food Services
Health Care &
Social Assistance
Newspapers
Motion Pictures
Broadcasting
TV News
Software
Fixed Line
Telephone
Service
Wireless
Telephone
Service
Cable &
Satellite TV
E-Business
Manufacturing/
Durable Goods
Hotels
Limited-Service
Restaurants
Full-Service
Restaurants
Hospitals
Public
Administration/
Government
Retail Trade
Manufacturing/
Nondurable Goods
News &
Information
Portals/
Search
Engines
Social Networking
Personal
Computers
Electronics
(TV/VCR/DVD)
Major
Appliances
Automobiles
& Light
Vehicles
Cellular Telephones
Food
Manufacturing
Pet Food
Soft Drinks
Breweries
Cigarettes
Apparel
Athletic Shoes
Personal Care
& Cleaning
Products
Finance &
Insurance
Local
Government
Federal
Government
E-Commerce
Banks
Life Insurance
Health
Insurance
Property &
Casualty
Insurance
Supermarkets
Gasoline
Stations
Department &
Discount Stores
Specialty Retail
Stores
Health &
Personal Care
Stores
Retail
Brokerage
Travel
The ACSI Model and Methodology
• In ACSI methodology, customer satisfaction is imbedded in a system of relationships, and
analyzed as part of a structural equation model. The model produces two critical pieces
of data useful to researchers and firms/agencies:
• The model provides mean scores (on a 0-100 scale) for each measured composite or
latent variable
• The model provides parameter estimates (or path coefficients) indicating what most
strongly influences satisfaction, and in turn how satisfaction influences future consumer
behaviors
Customer
Complaints
Perceived
Quality
• Overall
• Customization
• Reliability
Customer
Expectations
• Overall
• Customization
• Reliability
• Complaint Behavior
Perceived
Value
• Price Given Quality
• Quality Given Price
Customer
Satisfaction
• Satisfaction
• Comparison w/ Ideal
• Confirm/Disconfirm
Expectations
Customer
Loyalty
• Repurchase Likelihood
• Price Tolerance
(Reservation Price)
ACSI Data Collection
• Each year, including all private sector, public sector and
custom research projects, ACSI collects approximately
125,000 interviews of consumers
• From 1994 through 2009, nearly all of this data (with a few
exceptions for e-commerce companies) was collected over
the telephone using random-digit-dial probability sampling and
CATI
• Beginning in 2010, and following pilot testing that produced
promising results, ACSI moved to a multi-method interviewing
approach, with roughly half the data for any measured
company/government agency collected using RDD probability
sampling and CATI, and the other half collected using a
nonprobability panel of double opt-in respondents interviewed
online
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Extant Research
• While a handful of studies comparing results for samples
interviewed online to samples interviewed over the telephone
exist,* these studies have focused almost exclusively on
political opinions, voter preference, etc.
• There remains very little research into what differences (if any)
are likely to be observed across these two interviewing
methods for consumer-oriented data, where a significant
portion of data collection and survey research is focused
*Chang, L. and J.A. Krosnick (2009). “National Surveys via RDD Telephone Interviewing Versus The Internet: Comparing
Sample Representativeness and Response Quality,” Public Opinion Quarterly, 73(4), 641–678.
Fricker, S., M. Galesic, R. Tourangeau and T. Yan (2005). “An Experimental Comparison of Web and Telephone Surveys,”
Public Opinion Quarterly, 69(3), 370-392.
Vannieuwenhuyze, J., G. Loosveldt and G. Molenberghs (2010). “A Method for Evaluating Mode Effects in
Mixed-Mode Surveys,” Public Opinion Quarterly, 74(5), 1027-1045.
Findings from the AAPOR Online Task Force
• Findings from the AAPOR Online Task Force* suggest that
there is no theoretical basis for assuming that samples drawn
from nonprobability online panels are representative of a
larger population, and that therefore results may differ when
compared to an RDD probability sample interviewed over the
telephone
• However, this research also concludes there may be
instances in which online panels are useful and reliable, and
we conduct a series of empirical tests to see if customer
satisfaction data (ACSI) is such a case
*Baker, R. et al. (2010). “Research Synthesis: AAPOR Report on Online Panels,” Public Opinion Quarterly, 74(4), 711–781.
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Research Questions
• From the perspective of the ACSI project and its methodology,
two questions regarding multi-method interviewing are most
relevant and important:
• Do mean scores exhibit significant differences between a
sample interviewed online when compared to a sample
interviewed using RDD/CATI?
• Do model parameter estimates exhibit significant differences
between a sample interviewed online when compared to a
sample interviewed using RDD/CATI?
Data
• To seek answers to our research questions, we utilize a
sample of data consisting of approximately 9000 interviews
• Roughly half of these cases were collected via Internet
interviewing (from a sample balanced to Census
demographics from a large online panel (the Research Now
panel)), and the other half collected using RDD and CATI,
allowing us to test the similarities/differences produced by
these two interviewing methods
• The ACSI model (shown earlier) was estimated independently
for each industry and each interviewing method, producing
distinct mean scores and estimates (path coefficients)
facilitating these comparisons
Data
• The data represent consumer responses to questions
measuring satisfaction (and the other modeled variables) with
companies and industries in six NAICS sectors (for more
information on the companies included in the sample, see
Appendix A):
–
–
–
–
–
–
Apparel manufacturing (Manufacturing/nondurable goods)
Personal computers (Manufacturing/durable goods)
Fast food restaurants (Food services)
Insurance (Finance and insurance)
Supermarkets (Retail)
Wireless phone service (Information)
Tests of Difference
• To test for significant differences in mean scores across the
two interviewing methods for each ACSI variable in each of
the industries included in the sample, independent sample ttests were utilized
• To test for significant differences in parameter estimates for
the structural model for each of the industries included in the
sample, chi-square difference tests were utilized, with
parameters constrained to equality and significant chi-square
statistics indicative of significant parameter estimate
differences
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Results and Findings
• Across all of the tests – which included comparisons of 36
sets of mean scores across the two interviewing methods, and
54 sets of model parameter estimates – some significant
differences were observed
• In total, 36% of the mean scores (13 of 36) compared across
the two modes exhibited significant differences. Scores
skewed higher on the Internet, with 9 of 13 significant
differences reflecting “better” ratings among Internet
respondents (i.e. higher ratings, fewer complaints)
• Moreover, 39% of the model parameter estimates (21 of 54)
from the structural models compared across the two methods
exhibited significant differences
• (Two industry examples follow. All test results provided in
Appendix A)
Example 1: Supermarket Industry Results
Supermarket Industry
Telephone
Supermarket Industry
Internet
Variable
Expectations
N
784
Mean
79.08
N
790
Mean
80.24
Quality
784
80.43
790
79.43
Value
783
76.54
790
77.34
Satisfaction
784
76.38
790
75.59
Comp. (%)
782
10.87
788
10.53
Loyalty
782
76.37
786
82.60
Sig. Diff.
***
Path Coefficient
Tele.
Internet Sig. Diff.
Expect. → Quality
0.776
0.833
Quality → Value
0.528
0.629
Expect. → Value
0.196
0.111
Value → Sat.
0.444
0.481
Quality → Sat.
0.372
0.505
**
Expect. → Sat.
0.195
0.051
**
Sat. → Comp.
-0.286
-0.308
Comp. → Loyalty
0.045
-0.016
Sat. → Loyalty
0.616
0.638
• For the tests for this industry, one variable mean score of the
six tested was significantly different across the two samples,
while two of nine parameter estimates were significantly
different
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Example 2: Wireless Industry Results
Wireless Industry
Wireless Industry
Telephone
Variable
Internet
Sig. Diff.
N
Mean
N
Mean
Expectations
Quality
475
478
75.69
75.88
490
493
80.94
78.80
Value
470
72.70
488
71.54
Satisfaction
478
71.17
492
71.20
Comp. (%)
475
30.95
485
21.65
**
Loyalty
473
69.31
462
74.14
*
***
*
Path Coefficient
Tele.
Internet Sig. Diff.
Expect. → Quality
0.775
0.56
**
Quality → Value
0.85
0.998
**
Expect. → Value
Value → Sat.
Quality → Sat.
0.042
0.457
0.48
-0.058
0.529
0.476
Expect. → Sat.
Sat. → Comp.
0.053
-0.621
0.005
-0.601
Comp. → Loyalty
Sat. → Loyalty
-0.033
0.942
-0.037
0.96
• For the tests for this industry, four of the variable mean scores
exhibited significant differences, with scores skewing higher
(and complaint rate lower), and two of the parameter
estimates exhibited significant differences
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Results and Findings
• The above are “hard tests” of multi-method interviewing. As
many projects (including ACSI) have not traded telephoneonly for Internet-only interviewing, a “fairer” test is to compare
the telephone interview results to the mixed-method, mixedsample results
• For these tests, the results are more promising. Looking only
at differences in mean scores, of the 36 sets of means
compared only 11% (4 of 36) exhibited significant differences
• (Two industry examples follow. Full results for these tests are
included in Appendix A)
Example 3: Mixed-Sample vs. Telephone-Only
Mixed-Sample
N
Mean
Telephone
Sig. Diff.
N
Mean
Apparel
Industry
Expectations
957
83.99
475
84.14
Quality
957
85.12
475
86.33
Value
958
82.28
475
84.05
Satisfaction
Comp. (%)
Loyalty
957
955
950
81.26
1.47
79.78
475
475
473
83.16
0.63
79.52
Expectations
1156
83.51
556
82.94
Quality
1157
82.40
556
81.44
Value
1153
82.49
553
82.35
Satisfaction
1157
78.81
556
77.78
Comp. (%)
1147
12.64
553
15.91
Loyalty
1158
74.05
557
71.76
*
PC Industry
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Discussion Agenda
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Conclusions
• While some differences in both mean scores and model
parameter estimates are exhibited when comparing
telephone-only interviewing to Internet-only interviewing, the
differences account for a minority in both cases
• The results are even more promising when comparing mean
scores for telephone-only and mixed-method interviewing;
only a small fraction of the comparisons are significantly
different in this case
Implications and Future Research
• These tests provide evidence for the feasibility and reliability
of mixed-method sampling for consumer-oriented survey
research projects
• For projects working with this kind of data, both means scores
and model estimates appear to be relatively stable across
interviewing methods
• However, because we examine only consumer-oriented data,
those working with dissimilar types of data should perform
tests similar to ours to examine the reliability of mixed-method
interviewing, as results may vary
• Research expanding the types of data tested should help
market researchers determine the feasibility of multi-method
interviewing for particular client engagements
Appendix A: Supplemental Results and Information
Interview Data by Industry/Company
Industry
Companies
Apparel
Liz Claiborne; VF Corporation; Levi Strauss;
Jones Apparel Group; Hanesbrands
Personal Computers
Compaq; Apple; Hewlett Packard; Dell; Acer
Fast Food
Wendy’s; KFC; Little Caesar Enterprises;
Domino’s; Taco Bell; Pizza Hut; Burger
King; McDonald’s; Papa John’s; Starbucks
Insurance
Farmer’s Group; Allstate; State Farm;
Geico; Progressive; MetLife; Prudential;
New York Life; Northwestern Mutual Life
Supermarkets
Publix; Winn-Dixie; Supervalu; Safeway;
Wal-Mart; Kroger; Whole Foods
Wireless Service
Verizon; AT&T; Sprint Nextel; T-Mobile
Apparel and PC Industries Results
Telephone
Internet
Sig. Diff.
N
Mean
N
Mean
Expectations
475
84.14
482
83.83
Quality
475
86.33
482
83.93
*
Value
475
84.05
483
80.54
**
Satisfaction
475
83.16
482
79.39
**
Comp. (%)
475
0.63
480
2.29
*
Loyalty
473
79.52
477
80.05
Expectations
556
82.94
600
84.03
Quality
556
81.44
601
83.28
Value
553
82.35
600
82.63
Satisfaction
556
77.78
601
79.76
Comp. (%)
553
15.91
594
9.60
**
Loyalty
557
71.76
601
76.17
**
Apparel
Industry
PC Industry
Path Coefficient
Apparel Industry
Tele.
Internet
Sig. Diff.
Expect. → Quality
Quality → Value
Expect. → Value
Value → Sat.
Quality → Sat.
Expect. → Sat.
Sat. → Comp.
Comp. → Loyalty
Sat. → Loyalty
PC Industry
0.625
0.721
-0.031
0.449
0.415
0.069
-0.034
-0.216
0.772
0.778
0.847
-0.020
0.350
0.553
0.051
-0.057
0.014
0.908
**
Expect. → Quality
Quality → Value
Expect. → Value
Value → Sat.
Quality → Sat.
Expect. → Sat.
Sat. → Comp.
Comp. → Loyalty
Sat. → Loyalty
0.690
0.772
0.027
0.397
0.551
0.074
-0.547
-0.016
0.971
0.636
0.924
-0.104
0.419
0.630
-0.041
-0.475
-0.002
1.155
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
*
*
*
**
*
**
**
Fast Food and Insurance Industries Results
Path Coefficient
Telephone
N
Mean
Internet
N
Sig. Diff.
Mean
Fast Food
Industry
Internet
Sig. Diff.
Expect. → Quality
0.748
0.876
***
Quality → Value
0.694
0.647
Expect. → Value
0.115
0.197
Value → Sat.
0.350
0.425
**
**
Fast Food Industry
Expectations
1150
78.45
1169
79.98
Quality
1150
80.32
1169
80.59
Quality → Sat.
0.630
0.511
Value
1149
80.48
1170
80.25
Expect. → Sat.
0.051
0.116
Satisfaction
1150
75.65
1170
75.24
Sat. → Comp.
-0.347
-0.376
Comp. (%)
1149
8.09
1162
8.00
Comp. → Loyalty
0.049
-0.019
Loyalty
1145
74.44
1156
78.21
Sat. → Loyalty
0.825
0.797
Insurance
Industry
*
Tele.
***
*
Insurance Industry
Expect. → Quality
0.672
0.710
Expectations
970
81.72
1047
82.47
Quality → Value
0.703
0.820
**
Quality
973
83.81
1046
82.93
Expect. → Value
0.212
0.120
*
Value
966
79.83
1039
78.40
Value → Sat.
0.379
0.521
***
Satisfaction
971
79.68
1047
77.97
Quality → Sat.
0.527
0.481
Comp. (%)
976
7.48
1048
6.20
Expect. → Sat.
0.076
0.007
Loyalty
942
76.41
1006
78.11
Sat. → Comp.
-0.386
-0.308
Comp. → Loyalty
-0.083
-0.037
Sat. → Loyalty
0.813
0.929
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
*
**
Mixed-Method vs. Telephone-Only Means Tests (1)
Mixed-Method
Telephone
N
Mean
N
Mean
Expectations
2319
79.22
1150
78.45
Quality
2319
80.46
1150
80.32
Value
2319
80.36
1149
80.48
Satisfaction
2320
75.44
1150
75.65
Comp. (%)
2311
8.05
1149
8.09
Loyalty
2301
76.33
1145
74.44
Expectations
2017
82.11
970
81.72
Quality
2019
83.35
973
83.81
Value
2005
79.09
966
79.83
Satisfaction
2018
78.79
971
79.68
Comp. (%)
2024
6.82
976
7.48
Loyalty
1948
77.29
942
76.41
Sig. Diff.
Fast Food
Industry
*
Insurance
Industry
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Mixed-Method vs. Telephone-Only Means Tests (2)
Mixed-Method
Telephone
Sig. Diff.
N
Mean
N
Mean
Expectations
1574
79.66
784
79.08
Quality
1574
79.93
784
80.43
Value
1573
76.94
783
76.54
Satisfaction
1574
75.98
784
76.38
Comp. (%)
1570
10.70
782
10.87
Loyalty
1568
79.49
782
76.37
**
Expectations
965
78.36
475
75.69
*
Quality
971
77.36
478
75.88
Value
958
72.11
470
72.70
Satisfaction
970
71.19
478
71.17
Comp. (%)
960
26.25
475
30.95
Loyalty
935
71.70
473
69.31
Supermarket
Industry
Wireless
Industry
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the
Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.