Betting on Customer Satisfaction
How to carry out your own survey
data collection and analysis.
Jim Colton
Mentoring Specialist
Lou Johnson
Technical Training Specialist
Tuesday
9:15 a.m. – 10:15 a.m.
ASQ 2010 Lean & Six Sigma Conference
March 8 - 9, Phoenix, AZ
© 2009 Minitab, Inc.
Betting on Customer Satisfaction
Agenda:
Effective Use of Survey Data Collection and Analysis
Best Practices in Survey Design and Execution
Casino Survey Design, Goals and Execution
Casino Customer Satisfaction Data Analysis
Questions & Discussion
References
*** Non-disclosure: All survey results and scenarios are based on the authors’ actual experience. Data,
units, variable names, etc have been changed for demonstration purposes only to protect company propriety.
© 2009 Minitab, Inc.
Customer Satisfaction / Market Research ---- Failure
1985 New Coke Launch:
“Project Kansas” found
sweeter New Coke
preferred over Pepsi.
After launch in April, cans
of New Coke emptied in
the streets in protest.
3 months later, Coke loses
its historical edge in sales
to Pepsi. Coke Classic reintroduced.
In a frequently retold story (see Matthews), an elderly woman at a Marietta, Georgia supermarket confronts the
Coca-Cola deliveryman as he restocks the shelves. As he attempts to put New Coke bottles on it, she hits him
with her umbrella, yelling "It tastes like sh__" A nearby counterpart from Pepsi begins to snicker, only to be told
in turn, "You stay out of it! This is family business! Your stuff tastes worse than sh__”
** all information on this page from http://en.wikipedia.org/wiki/New_Coke#cite_note-Schindler-4
© 2009 Minitab, Inc.
Customer Satisfaction / Market Research ---- Success
In 40 years, Best Buy becomes the world’s largest
provider of consumer electronics*:
“customers who were highly satisfied across
© 2009 Minitab, Inc.
multiple touch points spend nearly 50% more over
the next 90 days.
“employees account for almost two thirds of
customer satisfaction”
“Even when the in-store experience is excellent, a
negative perception of the very next touch point
can result in nearly a 60% drop in customer
satisfaction.”
* CFI Group, White paper, Building Multi-Channel Customer Satisfaction and
Loyalty, 2008, list of findings.
Survey Data Collection/ Market Research --- Applications
Monitor customer satisfaction over time
New product development – what features do future
customers value and are willing to pay for
Market share – what portion of the current market is loyal
to each major provider and what factors are considered
in that decision
Product improvements – which characteristics of a
product or service are helping customer satisfaction and
which need to be improved.
© 2009 Minitab, Inc.
Customer / Market Surveys – Best Practices
Direct Request for the Information of Interest
How likely is it that your would recommend our company to a friend or colleague?*
Not at all Likely
1
Very Likely
2
3
4
5
6
7
Will you stay with Holiday Inn Express on your next overnight visit?
Yes
No
Not available
* Reichheld, Fredrick F. D.C. (2003).
© 2009 Minitab, Inc.
The One Number You Need to
Grow, Harvard Business Review, December, p2 -10.
Customer / Market Surveys – Best Practices
Accurate Measurement of the Response
*
5 pt scale is very common in
satisfaction surveys, followed by
the 9 pt. Odd number of ratings
leaves the center for a neutral
response, does not force an
opinion where one may not
exist**.
How likely is it that you would return to Lucky 7 on your next gambling outing?
Not at all Likely
1
Very Likely
2
3
4
5
* Merging Elements , white paper, Customer Satisfaction Surveys – Questions Answered, www.merging elements.com
** Allen, Elaine & Seaman, Christopher, A., (2007) Likert Scales and Data Analyses, ASQ
Quality Progress, July , Statistics Roundtable.
© 2009 Minitab, Inc.
Customer / Market Surveys – Best Practices
Clearly Defined Goal
• Goal: “gauge the usage and sophistication of Design for Six
Sigma projects, curriculum, training and certification”
• 10,000 DFSS surveys sent / 53 responses
© 2009 Minitab, Inc.
Customer / Market Surveys – Best Practices
Clearly defined goal
• Goal: determine the if a language translator would improve
customer satisfaction with bilingual customers.
• 1200 requests sent to software installers / 176 responses
Individual 95% CIs For Mean Based on Pooled StDev
Level
N
Mean StDev ------+---------+---------+---------+--English
59 3.8840 0.4991
(----*-----)
French
18 3.8327 0.5276 (--------*---------)
German
17 3.8529 0.7110 (---------*---------)
Japanese 26 4.0464 0.3801
(-------*-------)
Protugese 11 4.2215 0.3907
(-----------*-----------)
Spanish
45 3.9993 0.5315
(-----*-----)
------+---------+---------+---------+--3.75
4.00
4.25
4.50
Customer Satisfaction
Conclusion
The $150,000 spent on the translator would not be money well spent. Bilingual
and English speaking customers had the same level of customer satisfaction
© 2009 Minitab, Inc.
Customer / Market Surveys – Best Practices
Effective Survey Length
• Many analyses require answers to all questions. The number
of useful data points in the analysis can be a function of how
many questions you ask.
• Large survey provider:
10 – 15 questions max for a telephone survey
• GAP analysis asks for level of satisfaction and then level of
importance for each quality characteristic, looking for “gaps”
between the two. Doubles survey length.
• Length of the survey should show proper respect for the survey
recipient’s time.
Survey participants should sense the value of their responses
Sharing survey results should engage the respondent.
© 2009 Minitab, Inc.
Customer / Market Surveys – Best Practices
Pre-test Survey / Pre-analyze Results
• It all depends how you ask the question!
Burger King vs McDonalds *
• 3 to 1
• 50 / 50
• 1 to 5
vs
Making the mistake with 20 – 40 pretest subjects can save trauma of finding out that
thousands of survey participants did not understand the questions.
• Analyze the data from 5 fake surveys. Can your survey
customer apply the results of your analysis to their business
problem?
Pareto Chart of Failure Modes
1800
70
60
1200
50
40
900
660
600
300
277
154
153
150
131
e
at
129
126
78
74
72
65
64
n
a
10
0
0
t
or
* Zikmund, William, G. (1999) Essentials of Marketing Research,
Harcourt Brace & Company, Ft Worth, TX, p259. .
30
20
231
Percent
Rating
1500
d
d
ata
or
ri g
or
po
t
or
upp
n
tio
en
io
ct
er
th
m
rg
te
se
m
d
iv
la
si g
O
en
ba
ca
u pp
o
for
t g sele
as
ad
s
m
6
IC
to
alo
w
d
lly
tS
A
no
IC
-b
ple
nd
ct
g
M
A
is
an
cia
ce
en
je
o pe
D
M
ur
m
s ta
n in
ct
an
ss
t im d ata
ro
sc
D
ai
so
ge
for
ne
fin
no
d er
rp
ct
o
oje
re
Tr
to
N
pr
un
je
al
oo
in
adi
a na
le
n ot
e
tion
't
P
M
sm
litt
P ro
ed
l re
th
sn
io n
olu
o
rc
na
ct
S
oo
oe
to
fo
e te
le
T
ti o
td
ct
pl
ct
za
se
je
je
m
en
ni
ct
co
P ro
ga
je
em
P ro
r
to
ag
O
e
Pro
an
im
M
T
Rating
277 231 154 153 150 131 129 126
78
74
72
65
64
660
P ercent
12
10
7
6
6
6
5
5
3
3
3
3
3
28
C um %
12
21
28
34
41
46
52
57
60
64
67
69
72
100
© 2009 Minitab, Inc.
Survey Analysis Overview
Goals:
1. Improve the customer’s overall experience
2. Increase their likelihood of returning
3. Increase their likelihood of recommending Lucky7 Casino.
Objective: Determine which of the 150+ survey questions
drive the response to 3 critical survey questions:
1.
2.
3.
© 2009 Minitab, Inc.
Overall Experience
Likely to Return
Likely to Recommend.
Overview
Possible Minitab Analysis Tools:
1. Ordinal Logistic Regression in MINITAB
2. Stepwise Regression in MINITAB
3. Partial Least Squares in MINITAB
Each of these analysis tools attempts to find the set of
survey questions that drive the 3 critical survey
questions.
© 2009 Minitab, Inc.
Ordinal Logistic Regression
Advantage
• It is the only approach that correctly assumes the responses
are ordered categories.
Disadvantage
• If two questions are highly correlated, this analysis will likely
exclude one from the final model even if they both correlate
highly with the response.
© 2009 Minitab, Inc.
Stepwise Regression
Advantage
• It’s fast and simple.
Disadvantages
• It assumes a Normal distribution for the response.
• If two questions are highly correlated, it will likely exclude one
from the final model even if they both correlate highly with the
response.
© 2009 Minitab, Inc.
Partial Least Squares
Advantages
• Allows multiple responses.
• Can handle many predictors.
• Highly correlated predictors are not an issue.
Disadvantages
• No significance testing for individual questions.
© 2009 Minitab, Inc.
Challenge with All 3 Approaches
If one question is not answered by a respondent, the
entire row is ignored in the analysis.
© 2009 Minitab, Inc.
Strategy
Step 1: Import, format, and clean the data.
Step 2: Identify the survey questions with the least
amount of missing data.
Step 3: Use Partial Least Squares (PLS) on the set of
questions with the least missing data.
Step 4: Use Ordinal Logistic Regression (OLR) to
evaluate the importance of all questions individually.
Step 5: Create graphs to display the results.
© 2009 Minitab, Inc.
Step 1: Importing and Checking Data
Check for outliers
© 2009 Minitab, Inc.
Step 2: Bar Chart of Sample Size (N)
© 2009 Minitab, Inc.
Step 3: Partial Least Squares
Include the 29 questions
identified in Step 3 together in
one PLS model.
The session window output
includes R-squared values for
each response:
• Overall Experience: 45%
• Likely to Return: 35%
• Likely to Recommend: 38%
These R-squared values
represent the percentage of
variation explained in the
responses by the 29 questions
entered in the PLS model.
Model Selection and Validation for Overall_Experience
Components
1
2
3
X Variance
0.251765
0.291199
0.343480
Error SS
1679.63
1622.01
1593.77
R-Sq
0.419781
0.439686
0.449442
Model Selection and Validation for Likely_To_Return
Components
1
2
3
X Variance
0.251765
0.291199
0.343480
Error SS
1439.85
1337.33
1331.84
R-Sq
0.292853
0.343204
0.345900
Model Selection and Validation for Recommend
Components
1
2
3
X Variance
0.251765
0.291199
0.343480
Error SS
1892.56
1759.30
1759.26
R-Sq
0.333366
0.380309
0.380323
© 2009 Minitab, Inc.
Partial Least Squares Importance
Overall Experience
© 2009 Minitab, Inc.
Partial Least Squares Importance
Likely to Return
© 2009 Minitab, Inc.
Partial Least Squares Importance
Likely to Recommend
© 2009 Minitab, Inc.
Additional Partial Least Squares Graph
Loading Plot
© 2009 Minitab, Inc.
Step 4: Ordinal Logistic Regression
Logistic Regression Table
Apply OLR to each question
in a separate analysis.
Predictor
Const(1)
Const(2)
Const(3)
Const(4)
Players_Club_FH
Coef
-5.40424
-3.83727
-1.93093
0.722390
-0.119351
SE Coef
0.135424
0.0655857
0.0342311
0.0293175
0.0073626
Z
-39.91
-58.51
-56.41
24.64
-16.21
P
0.000
0.000
0.000
0.000
0.000
Log-Likelihood = -17616.793
Test that all slopes are zero: G = 265.047, DF = 1,
P-Value = 0.000
© 2009 Minitab, Inc.
Ordinal Logistic Regression Importance
Overall Experience
© 2009 Minitab, Inc.
Ordinal Logistic Regression Importance
Likely to Return
© 2009 Minitab, Inc.
Ordinal Logistic Regression Importance
Likely to Recommend
© 2009 Minitab, Inc.
Step 5: Most Important Questions
The four most important questions in predicting the rating for
Overall Experience, Likely to Return, and Likely to Recommend
are:
1. Treated Like VIP
2. Winning Player at Lucky7
3. Gambling Time at Lucky7
4. Percent of Casino Visits Lucky7
Is was somewhat surprising that the Hotel and Restaurants
questions did not show-up as key drivers.
© 2009 Minitab, Inc.
Step 5: Most Important Questions
Overall Experience
© 2009 Minitab, Inc.
Step 5: Most Important Questions
Likely to Return
© 2009 Minitab, Inc.
Step 5: Most Important Questions
Likely to Recommend
© 2009 Minitab, Inc.
Information in Missing Data
Whether or not someone answers a question may contain
information.
For each survey question, a column was created indicating whether
or not the question was answered.
All these missing data variables together explain less than 2% of the
variation in each of the 3 critical responses.
© 2009 Minitab, Inc.
Additional Analysis
© 2009 Minitab, Inc.
Improving the Survey
Look for redundant questions.
© 2009 Minitab, Inc.
Conclusions
Treated Like VIP – Made it easier to qualify for the VIP club.
Winning Player at Lucky7 – Geared advertising towards showing a
winning experience at Lucky7.
Gambling Time at Lucky7 – Offered some lower stakes tables during
off peak times.
Percent of Casino Visits Lucky7 – Made generous offers for next visit
to first time visitors.
© 2009 Minitab, Inc.
References
Hayes, Bob, E., (2008) Measuring Customer Satisfaction and Loyalty, Third Edition: Survey Design,
Use, and Statistical Analysis Methods, ASQ Press, ASQ, Milwaukee, WI.
Whitehead, Jason C., (2006) Solving the Data Quality Problem, White paper, TriTruns Innovation,
Arlington, VA.
Zikmund, William G., (1999) Essentials of Marketing Research. Harcourt Brace & Company,
Orlando, FL.
© 2009 Minitab, Inc.
Betting on Customer Satisfaction
Jim Colton
Mentoring Manager
Lou Johnson
Technical Training Specialist
Thank you for your time and participation.
Questions?
© 2009 Minitab, Inc.