Session 6. Survey Design and Sampling Strategy
MKTG 3010 MARKETING RESEARCH
1
Homework - Psychology beats business
training when it comes to entrepreneurship
(https://www.economist.com/news/business/21729454-among-smallbusiness-owners-togo-least-psychology-beats-business-training-whenit-comes)
What is the research question?
Which experimental design was used?
What is the treatment? How to calculate the
treatment effect?
What is the dependent variable?




2
Warning: We are getting more and more quantitative
Define the Information Needed
Design the Exploratory, Descriptive, and/or Causal Phases of the Research
Specify the Measurement and Scaling Procedures
Construct a Questionnaire
Specify the Sampling Process and the Sample Size
Develop a Plan of Data Analysis
3
Four Types of Primary Scales
Primary Scales
Ratio
Scale
Interval
Scale
Ordinal
Scale
Nominal
Scale
4
Lowest level of
measurement
Highest level of
measurement
Review 1. Please identify the type of scale being used and the permissible statistics that can be
used.
1) What is your age?
2) What is your postal code?
3) How much money did you spend last year on make-ups?
4) To what extent do you agree or disagree with this statement,
“I do not feel comfortable describing myself as beautiful.”
Completely disagree
Completely agree
1
2
3
4
5
5) On an average weekday, how much time do you spend on exercising?
i.
Less than 15 minutes
ii.
15 to 30 minutes
iii.
31 to 60 minutes
iv.
More than 60 minutes
5
Scaling Techniques
Scaling
Techniques
Noncomparative
Scales
Comparative
Scales
Continuous
Rating Scales
Paired
Comparison
Rank
Order
6
Constant
Sum
Itemized
Rating Scales
Likert
Semantic
Differential
Stapel
Comparative scales involve the direct comparison
of stimulus objects.
- e.g. Do you prefer Pepsi or Coke?
- Comparative scale data must be interpreted in relative
terms and have only ordinal or rank order properties.
In noncomparative scales, each object is scaled
independently of the others in the stimulus set.
- e.g. How do you feel about Coke?
- The resulting data are generally assumed to be interval
or ratio scaled.
Review 1.
How shall we measure people’s attitude toward
iPhone 8 using comparative scaling technique?
2.
How shall we measure people’s attitude toward
iPhone 8 using non- comparative scaling
technique?
8
Define the Information Needed
Design the Exploratory, Descriptive, and/or Causal Phases of the Research
Specify the Measurement and Scaling Procedures
Construct a Questionnaire
Specify the Sampling Process and the Sample Size
Develop a Plan of Data Analysis
9
Key Issues in Survey Design

Structure of the survey
Order of information


Respondent-driven design

Understanding the psychology of survey response
•
Question wording
−
10
Are they answering what we’re asking?
Review 
We need to include the following questions
in the survey. Please arrange them in a
meaningful order.





11
A. “What is your age?”
B. “What brand of skis do you own?”
C. “What feature do you like best about skis?”
D. “Have you been snow skiing in the past
twelve months?”
E. “Following are 10 characteristics of snow
skis. Please rate your ski on each of the
characteristic using the scale below.”
The Psychology of Survey Response
“Self-Reports”,
N. Schwarz
Am Psych 1999
Comprehension
Retrieval
Judgment
Response
12
Review -
Do you find anything wrong with the
following questions? Why?
1.
2.
3.
13
“Do you believe that McYummys offers fast and
courteous service?”
Which brand of shampoo do you use?
Do you think any brand is better than Gap?
II. Failures of Retrieval

When in doubt: people guess-timate.


Due to passage of time
Due to uncertainty of actual timing of event

Timing of event may be uncertain (allergies?)
When mundane: people guess-timate
 When salient: people over-estimate


Telescoping
Retrieval
14
Error from Inability to Recall
Years suffering allergies
25
1) What % have suffered for 10 years?
2) What % have suffered for 9 years?
20
15
%
10
5
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
Years
Retrieval
15
Priming:
Context Affects Retrieval

Cues in the environment affect what’s
available in memory.


Survey-related content
General attitudes
Retrieval
16
Addressing Memory Issues

Use a reasonable time frame:
“How
many
cartons
of orange
juice
did you
buy in
the
“How
many
cartons
of orange
juice
did you
buy
last
”
thisyear?
week?”
“Is this a normal number for you?”

Use available cues:
“Sinceyou
last aThanksgiving,
were you
“Were
driver or passenger
in a driver
car thatorwas
passenger
a car
that was
involved
in an accident?”
involved ininan
accident
in the
last year?”
• Be aware of the limits of human memory!
Retrieval
17
III. Issues in Judgment:
Constructed Preferences






Order effects: Anchoring, contrast
Demand Effect
Framing
Halo effect
Priming
Context effects: choice set
Judgment
18
Hmmm…let
me figure out
what I think.
Anchoring / Contrast Effects
1--2--3--4--5--6--7
Hate it
Love it
□ Squid Ink Ice Cream
□ Lemon Tart
□ Chocolate Cake
□ Tiramisu
□ Crème Brule
□ Vanilla Ice Cream
□ Lemon Tart
□ Chocolate Cake
□ Tiramisu
□ Crème Brule
Comprehension (past information) Affects
Judgments
Judgment
19
Demand Effect
“What do I think you want me to say?”

Are you in favor of stem cell research?
69% YES (NBC news)
 24% YES (Conference of Catholic Bishops)
 70% YES (Juvenile Diabetes Foundation)


57% don’t know enough to say (Gallup)
Source: “Unbearable Lightness of Public
Judgment Opinion Polls”, NYT 2001
20
Leading Questions:
“What kind of person are you?”
“Doyou
youthink
thinkthat
thatpatriotic
Americans
should buy
 –“Do
Americans
should
buyautomobiles
imported automobiles
when
imported
or automobiles
madethat
in
would
put American labor out of work?
this country?
___
yesimported
___
___
no domestic
___
___
don’t
know
___
don’t
know


Colgate
your
favorite
toothpaste?”
–“Is“What
is your
favorite
toothpaste
brand?”
Avoid leading or biased questions
Judgment
21
Framing

“Sony fired 4% of its work force.”

“Sony was able to retain 96% of its work
force.”
Judgment
22
Context Effects: Compromise
HD Space
Preference for the
middle option
RAM
Judgment
23
IV. Response Scale Induced Bias
How well does this brand’s product perform:
Sony
….
Not Well
1
2
3
4
Very Well
5
Rate how you feel about the following brands:
Sony
….
Not Well
-2
-1
Very Well
0
Response
24
1
2
Scale Usage Bias

Rate Apple on 20 attributes on a 5 point scale:
Avg=1.2

Avg=3.0
What does this mean?
She likes Apple more than he does
 She has positive scale usage bias (rates more favorable)
 She has extremeness usage bias (rates high and low only)
 He was uninvolved or uninformed
Alternative: Rankings remove all (good & bad) usage variance


Response
25
Minimizing Survey Bias:
Best Practices

Plan order carefully



Multiple measures





26
Rotate items
Split the sample
Ratings and rankings
Different wording
Consistent methods for comparisons
Provide (realistic) context
Recognize limitations: realistic expectations
Complexity of Questionnaire Design
Looks easy.
 Very difficult.
 No rules can guarantee flawless
questionnaire.
 Questionnaires by skilled researchers may
have drawbacks.
 Researchers may discover questionnaire
flaws after data collection.

Think Ahead: How Can You Tell
Whether They “Get It”?
“Don’t Know” / “Not Apply” option
 Open-ended responses



“Other (specify)”
Probing questions


“Why?”
But… coding is costly and
time consuming
 Multiple
methods
_______________________
_______________________
Bad Respondents

Causes:
Hurried
Irked
Inexperienced
Imposter (“Gaming”)
Clues In the Data
Response time
 Patterns of response (all ‘3’)
 Blank / nonsensical open-ends
 Traps





Usage of low incidence / bogus products
Verification: “Please check ‘5’ ”
“Recall test”
What’s the right cut-off?
Pre-Testing

Dry run on a small sample





sample members should be actual members of the
target population
ensures the questions you intend are being
answered
ensures that questions are understood by
respondents
screens for problems with flow patterns
Verbal protocol questionnaire testing
(“think aloud”)
Define the Information Needed
Design the Exploratory, Descriptive, and/or Causal Phases of the Research
Specify the Measurement and Scaling Procedures
Construct a Questionnaire
Specify the Sampling Process and the Sample Size
Develop a Plan of Data Analysis
32
Sampling Strategy
Where Do We Get Data From?
33
Large Sample Size vs.
Probability Design Sampling
2 Mil+
34
“Landon in a Landslide,” History Matters
Sampling Strategy
Define the Population
Determine the Sampling Frame
Select Sampling Technique(s)
Determine the Sample Size
Execute the Sampling Process
I. Define the Target Population



The target population is the collection of
elements or objects that possess the information
sought by the researcher and about which
inferences are to be made.
Who do you want to generalize to?
The target population should be defined in terms
of




36
Elements – the object that the information is desired
Sampling units – element, or entity contains the
element
Extent - geographical boundaries
Time – the time period of interest
Example: Revlon want to sample women over 18
years of age
Time Frame:
Upcoming
Summer
Sampling Unit:
Households
with 18 year
old females
37
Extent:
Domestic
United States
Element:
18 year old
females
II. Determine the Sampling Frame




A sampling frame is a representation of the
elements of the target population.
It consists of a list or set of directions for
identifying the target population.
How can you get access to them?
e.g.



38
telephone book
a mailing list purchased from a commercial
organization
Yellow page
III. Select Sampling Techniques
Sampling
Techniques
Nonprobability
Sampling
Techniques
39
Probability
Sampling
Techniques

Probability


Non-Probability


Population elements are selected in a non-random
manner
Advantages of Probability Samples


40
Every element has a known, non-zero probability of
inclusion in the sample
Allows quantification of sampling error
Generally more representative
Probability Sampling Techniques
Probability Sampling Techniques
Simple Random
Sampling
41
Systematic
Sampling
Stratified
Sampling
Cluster
Sampling
Nonprobability Sampling
Nonprobability Sampling Techniques
Convenience
Sampling
42
Judgmental
Sampling
Quota
Sampling
Snowball
Sampling
Simple Random Sampling
Each element of the population has the same
known nonzero probability of inclusion
Example:
Random selection
from voter
database
43
Simple Random Sampling
Mechanics



Table of random numbers
Computer generated random numbers
Random digit dialing
Advantages
• Easy to implement
• Does not always require a list
(e.g. random-digit dialing)
44
Disadvantages
• Less efficient than stratified
sampling
• May be more expensive than
cluster sampling
Is Random Sampling
“Random Enough”?

Can get outcomes that don’t “seem”
random.
1 out of 16 chance
OR
1 out of 16 chance
→“Inefficient”
45
Systematic Sampling
kth

Every
element from the list

Systematically spreading the
sample through
the population list

Sampling efficiency depends
on the ordering of the list
(e.g. sorted on key variable)
46
Example:
Every 5th voter
from registered
voter list
Leveraging Sample Structure: Stratified vs.
Cluster Sampling
Stratified
Stratified
Cluster
Cluster
Grouping:
homogenous
Strategy:
randomly sample within
randomly select clusters
e.g.
5 voters at each precinct
every voter at 5 precincts
Tradeoff:
more statistically efficient
cheaper
47
heterogenous
Sheena Iyengar: How to Make
Choosing Easier
48







1. How shall we classify the research method in the grocery
store study?
2. How would the number of choices affect consumer decision
in the “Jam” study? Do the findings reveal causal relationship?
3. For the financial study, what was the finding regarding the
participation of the plan? Is the relationship monotonic? For the
financial study, what was the finding regarding the choice of the
plan?
4. Do the findings in 3 &4 reveal causality? (Elaborate our
answers.)
5. What are the consequences of choice overload? And how
shall we overcome it?
6. What do you learn from the last study of choosing a car?
How does it relate to what we learned so far?
7. How does this video relate to the last one by Malcolm
Gladwell?
49