MKMR 310: Marketing Analytics
Week 4: Sampling, Survey Error, and Qualtrics
Pablo Azar
Weatherhead School of Management
Case Western Reserve University
1/27/2025
MKMR 310
1
Today’s agenda
• Recap
• Sampling
• Survey Errors
1/27/2025
MKMR 310
2
Recap: Measurement and scales
• There are several types of measures of central tendency, including:
• Mean: The arithmetic average of a set of numerical values, calculated by summing up all the values
and dividing by the number of values.
• Median: The middle value of a set of numerical values when they are arranged in order. If there is an
odd number of values, the median is the middle value, and if there is an even number of values, the
median is the average of the two middle values.
• Mode: The most frequent value in a set of data.
• Mean is a statistical measure of central tendency that is used to describe the average
value of a set of numerical data. While nominal and ordinal scale data may be
represented using numbers, they are not numerical data in the sense that they do not
have a mathematical meaning.
1/27/2025
MKMR 310
3
Recap: Measurement and scales
• In descriptive research, we want quantitative facts, opinions, and attitudes so we can
make comparisons and inferences.
• Quantitative data requires measurement: Assigning numbers or labels to concepts.
• Example: assign a 1 if the respondent is Male and a 2 if the respondent is Female.
• The measurement scale is the type of information provided by numbers.
• Each of the four scales (i.e., nominal, ordinal, interval, and ratio) provides a different type of
information.
• Each scale of measurement has certain properties which in turn determines the
appropriateness for use of certain statistical analyses.
1/27/2025
MKMR 310
4
Recap: Summary of measurement scale
Properties
Nominal
Ordinal
Interval
Ratio
Unique Name
✓
✓
✓
✓
✓
✓
✓
✓
✓
Preserve Order
Equal intervals
✓
Natural zero
1/27/2025
MKMR 310
5
Recap: Name that scale!
• Nominal, Ordinal, Interval, Ratio
1/27/2025
MKMR 310
6
Recap: Name that scale!
• Nominal, Ordinal, Interval, Ratio
1/27/2025
MKMR 310
7
Recap: Name that scale!
• Nominal, Ordinal, Interval, Ratio
1/27/2025
MKMR 310
8
Recap: Name that scale!
• Nominal, Ordinal, Interval, Ratio
1/27/2025
MKMR 310
9
Quiz
Q. Which of the following is NOT correct about measurement scale?
• The amount of information increases from nominal to ratio scales.
• Ratio data can always be converted into a nominal data.
• The burden imposed on respondent increases from ratio to nominal scales.
• The number of statistical techniques you may use to analyze data increases from nominal to ratio
scales.
• The Likert scale is usually treated as an interval scale in social science.
1/27/2025
MKMR 310
10
Marketing research process
1/27/2025
Planning
1. Problem
Identification
2. Determine the
Research Type
Implementing
3. Design the Data
Collection Method
4. Sampling and
Data Collection
Informing
5. Analyze and
Interpret the Data
6. Prepare and
Report the Findings
MKMR 310
11
Sampling
Sample and error
• A sample refers to a smaller, manageable version of a larger group. It is a subset
containing the characteristics of a larger population.
• You should think about questionnaire and representative of your sample.
1/27/2025
MKMR 310
13
Definition
1/27/2025
Probability
Sampling
MKMR 310
Non-Probability
Sampling
14
Sampling in survey research
• Sampling is a statistical process that involves selecting and surveying individuals from
a particular population.
• Target population: The entire group of people from whom information is needed.
• Sample: A subset of the population of interest.
• Sampling: The process of selecting units (e.g., people) from a population of interest.
• Sampling Frame: A list of the population elements from which we select units to be
sampled.
1/27/2025
MKMR 310
15
Example: Sample for T-shirt
• Suppose the University Bookstore wants to know potential customers’ interest in a new
product, PBL t-shirt. Who should we target to gather data on potential customer
interest in the new PBL t-shirt?
• Sample (100 people): A subset of the population of interest
1/27/2025
MKMR 310
16
Representativeness of the sample
• A sample should be representative of the population, thereby generalizing
the results back to the population.
• Good sample = representative sample = unbiased sample.
• Q. Why is it important to have a representative sample?
• 3 factors that influence the representativeness:
1/27/2025
MKMR 310
17
Representativeness of the sample
• A sample should be representative of the population, thereby generalizing
the results back to the population.
• Good sample = representative sample = unbiased sample.
Q. Why is it important to have a representative sample?
• 3 factors that influence the representativeness:
• Sampling procedure
• Sample size
• Response rate
1/27/2025
MKMR 310
18
Sampling definitions
1. Probability Sampling
2. Non-Probability Sampling
A sample in which every
element in the population has a
known statistical likelihood of
being selected.
Any sample in which little or no
attempt is made to get a
representative cross-section of
the population.
• Subjective procedure
• Sample is not always
representative
• Objective procedure
• Strict procedures to follow
• Yield a representative sample
1/27/2025
MKMR 310
19
Classification of sampling plans
1/27/2025
MKMR 310
20
Definition
1/27/2025
Probability
Sampling
MKMR 310
NonProbability
Sampling
21
Simple random sampling
• A sample is picked randomly, and every member has an equal opportunity to be
selected
• Enumerate the sampling frame
• Assign numbers to each element
• Randomly select units
• Simple random sampling is a widely used and well-regarded method of data collection
because of its simplicity, flexibility, representativeness, and proven validity
1/27/2025
22
MKMR 310
Systematic sampling
• You select members systematically—say, every tenth member—at that
particular time or event
• Enumerate the sampling frame
• Assign numbers and produce a skip interval k
• Choose every kth element in the sampling frame
• The beginning number is randomly chosen within the skip interval
1/27/2025
MKMR 310
23
Systematic sampling
• Systematic sampling is a useful option when the population is organized
and when efficiency and simplicity are desired, but care must be taken to
avoid introducing bias by choosing the right sampling interval.
1/27/2025
MKMR 310
24
Stratified sampling
• The population is divided into mutually exclusive groups based on
characteristics that they share (such as age or race); then random
samples are drawn from each group.
1. Enumerate the sampling frame.
2. Split the sampling frame into mutually
exclusive, homogeneous subgroups.
3. Conduct simple random or systematic sampling
on every subgroup.
1/27/2025
MKMR 310
25
Stratified sampling
• In general, stratified sampling is preferred over random sampling when the
goal is to ensure that the sample is representative of specific subgroups
within the population.
• For example, we want to survey people's opinions on a new product, and
we have a list of 1000 people who are divided into 2 age groups: 18-34 and
35+. We use stratified sampling to ensure that the sample accurately
represents both age groups.
1/27/2025
MKMR 310
26
Cluster sampling
• Mutually homogeneous yet internally heterogeneous groupings are evident in a
statistical population.
1. Enumerate the sampling frame.
2. Split the sampling frame into mutually
exclusive, heterogeneous subgroups.
3. Select a random sample of the
subgroups.
1/27/2025
MKMR 310
27
Cluster sampling
• In general, cluster sampling is preferred over random sampling when the population is
large or dispersed, and when the goal is to reduce the cost and complexity of the study
while still achieving a representative sample.
• For example, we want to survey people's opinions on a new product, and we have a list
of 100 neighborhoods in a city. Instead of surveying every person in the city, we use
cluster sampling to make the process more manageable. We first divide the 100
neighborhoods into 10 clusters of 10 neighborhoods each. Then, we randomly select 3
clusters to survey.
1/27/2025
MKMR 310
28
Advantages and disadvantages
• Advantages
• Generally will produce representative samples
• Avoids conscious bias
• Disadvantages
• Requires a sampling frame, which is not always available
• Often takes longer and costs more to select
1/27/2025
MKMR 310
29
Definition
1/27/2025
Probability
Sampling
MKMR 310
NonProbability
Sampling
30
Convenience sampling
• Operational Plan: Participants are selected based on their accessibility and
willingness to participate
• Example: Survey being conducted in a shopping mall, where participants are approached and asked
to complete the survey on the spot
• The participants in this case are selected based on their availability and willingness to
participate, rather than through a random or systematic method
• Representativeness: Mostly results in a sample that may not accurately reflect the
population, leading to biased results
1/27/2025
MKMR 310
31
Judgement sampling
• Operational Plan: Find willing participants based on the researcher’s
judgment criteria
• Example: A market researcher wants to study the attitudes of luxury car owners
toward a new car model. They can use judgment sampling to select a sample of
luxury car owners who they believe are representative of the target population
• Representativeness: This can be representative if the researcher uses
high-quality criteria and adheres to it strictly
1/27/2025
MKMR 310
32
Quota sampling
• Operational Plan: Find willing participants to represent various characteristics known
about the target population. For example, 50/50 male/female
• Representativeness: This can be representative if the researcher uses high-quality
criteria and adheres to it strictly
• Q. What is the difference between stratified sampling and quota sampling?
1/27/2025
MKMR 310
33
Quota sampling
• Operational Plan: Find willing participants to represent various characteristics known
about the target population. For example, 50/50 male/female
• Representativeness: Can be representative if the researcher uses high-quality criteria
and adheres to them strictly
• Q. What is the difference between stratified sampling and quota sampling?
VS.
1/27/2025
MKMR 310
34
Snowball sampling
• Operational Plan: Find one person who fits the characteristics of interest, then ask that
person to generate names of others with the same characteristics
• Representativeness: Unlikely to be representative
1/27/2025
MKMR 310
35
Advantages and disadvantages
• Advantages
• Samples can be drawn quickly and easily
• No sampling frame is necessary
• Good for exploratory research
• Disadvantages
• Samples can include irrelevant units
• Can’t generalize to the population of interest
• Can’t evaluate sampling error
1/27/2025
MKMR 310
36
Developing a sampling plan
• Define the target population
• Obtain a sampling frame (for probability sampling plans)
• Design the plan (needed size, method)
• Draw the sample (and validate)
• Resample, if necessary
1/27/2025
MKMR 310
37
Survey Error
Survey error
Every 10 years, the US Census Bureau conducts a census to determine the number of
people living in the united states.
Q. What kinds of survey errors can you name in this clip?
https://www.google.com/search?q=nbc+the+census+snl&ei=vHv6YfmkLpK2qtsPxtO1iAY&ved=0ahUKEwj58tOUheH1AhUSm2oFHcZpDWEQ4
MKMR 310
1/27/2025
dUDCA4&uact=5&oq=nbc+the+census+snl&gs_lcp=Cgdnd3Mtd2l6EAMyBQghEKABOgcIABBHELADOggIIRAWEB0QHkoECEEYAEoECEYYAFA0
WL8DYPEEaAFwAngBgAGKAogB6ASSAQUxLjIuMZgBAKABAcgBCMABAQ&sclient=gws-wiz
39
Measurement accuracy
Measured value = “True Value” + Error
• What we measure as a response to a questionnaire item consists of the “true” answer
we are interested in, plus some error.
• Remember: All measurement contains error.
• This error can be random (due to sampling) or systematic.
• It is important to reduce the amount of error so that the customer responses can more
accurately reflect the true value.
1/27/2025
MKMR 310
40
Total error
1/27/2025
MKMR 310
41
Sampling error
• Differences in the results between the population of interest and the sample
• A difference between the “truth” and what is measured based on using a sample from the
population
• Results from only using a subset (i.e., sample) of the population, and not the entire
population
• Because the sample is randomly selected, sampling error is also called “random error”
• Unless a census is done, there is always a certain amount of random error: Nearly
unavoidable in survey research
• This means that we have to use statistics to help us learn from the data
1/27/2025
MKMR 310
42
Non-sampling error
• A difference between the “truth” and what is measured based on problems with the
design and collection of the survey
• Because it is avoidable, non-sampling error is also called “systematic error”
• Instrument Error
• Sample Design Error
• Non-Response Error
• Response Error
1/27/2025
MKMR 310
43
1. Instrument error
• Instrument error refers to inaccuracies or biases in the data that are due to problems
with the design or administration of the survey or study
• Instrument error can occur when the survey or study instrument, such as a
questionnaire or interview guide, is poorly designed, or when there are problems with
the administration of the survey or study
• Examples of instrument error include:
• Ambiguous or confusing questions
• Bias in the question wording
• Poorly designed response scales
• Technical problems with the survey instrument
1/27/2025
MKMR 310
44
1. Instrument error
• To reduce the impact of instrument error, it is important to carefully design and pre-test
survey questions to ensure that they are clear, concise, and accurately reflect the
information that is being sought
• Additionally, it may be useful to pilot the survey with a small sample of individuals to
identify and correct any problems with the questions before the survey is administered
more widely
1/27/2025
MKMR 310
45
2. Sample design error
• The population has been miss specified (i.e., the wrong target market has been
selected)
• The sample is not representative (e.g., incorrect sampling frame)
• Improper selection:
• Respondents are not in the sampling frame
• The screening mechanism isn’t working properly
1/27/2025
MKMR 310
46
3. Non-response error
• We know that not everyone will respond to a survey
• Non-response error exists when the people who don’t respond are systematically
different from those who do (e.g., measuring customer satisfaction with a mail survey)
• This results in a sample that is not representative of the population
1/27/2025
MKMR 310
47
4. Response error
• Careless Responding: Respondents do not carefully read instructions, wording of the
question, and/or the response options
• Social Desirability Responding: Caused by respondents’ desire, either conscious or
unconscious, to gain prestige or appear in a different social role
• Acquiescence: Tendency to agree/disagree with items regardless of content (also
called “yea-saying” or “nay-saying”)
• Extreme Response/Midpoint Responding: Tendency to use the extremes or midpoint of
a scale (related to the response range and if a midpoint response is available)
• Memory issues: the limitations or inaccuracies in an individual's recall of past events,
experiences, or information
1/27/2025
MKMR 310
48
Name that error!
• My sister hates AT&T, so she gives them low scores on each customer satisfaction
question without even reading the questions
• Sampling Error
• Non-sampling Error
• Instrument Error
• Sample Design Error
• Non-Response Error
• Response Error
1/27/2025
MKMR 310
49
Name that error!
• You decide to conduct a mall-intercept-style study at Beachwood Place to study the
attitude of all Americans toward drinking beer
• Sampling Error
• Non-sampling Error
• Instrument Error
• Sample Design Error
• Non-Response Error
• Response Error
1/27/2025
MKMR 310
50
Name that error!
• Too embarrassed to admit they drink and drive, 10 subjects lied about their drinking
habits on your alcohol-use survey
• Sampling Error
• Non-sampling Error
• Instrument Error
• Sample Design Error
• Non-Response Error
• Response Error
1/27/2025
MKMR 310
51
Name that error!
• The Weatherhead School of Management sends out a survey to all undergraduate
students asking about satisfaction with class sizes. 5% of students respond. The
Weatherhead School of Management analyzes the resulting data and reports that an
estimated 80% of students are unhappy with the size of classes.
• Sampling Error
• Non-sampling Error
• Instrument Error
• Sample Design Error
• Non-Response Error
• Response Error
1/27/2025
MKMR 310
52
Order Bias
• The order in which questions are asked matters
• Using the funnel approach and randomizing question options helps minimize order
bias
• Example: Please rate on a 5-point scale your satisfaction with the following items
(5=very satisfied, 1=very dissatisfied)
1/27/2025
MKMR 310
53
Collect Primary Data: Qualtrics
Collecting the data
• Online survey software and questionnaire tools
• Marketing Research Firm (access to a large and diverse participant population)
1/27/2025
MKMR 310
55
Qualtrics
• Qualtrics is the premier Experience Management (XM) platform. The platform is
designed to optimize research around the customer, employee, product, and brand
experiences of your customers, constituents, and colleagues
• https://case.edu/utech/help/knowledge-base/qualtrics
• cwru.qualtrics.com
• Qualtrics provides great step-by-step instructions to get you started
1/27/2025
MKMR 310
56
How to register
1/27/2025
MKMR 310
57
Next time
Please register on the Qualtrics webinar page until the next class
1/27/2025
MKMR 310