Experimental Design: Part II
MAR 6648: Marketing Research
February 8, 2010
Overview
• Let’s review experiments!
• What can experiments do that other
techniques can’t?
• How do experiments get implemented in
marketing?
An experiment
An experiment?
• The owner of two McDonalds franchises here
in Gainesville wants to see if transactions run
more quickly if he uses both drive-thru
windows or only one. He picks one restaurant
to use both windows at all times for a month,
and the other he has closed at all times for a
month. He finds that the drive-thru that uses
both windows has notably faster service
times.
An experiment?
An experiment?
A Case Study in Causal Investigation
• Broken Windows Theory (Kelling & Wilson,
1982)
– “If a window in a building is broken and is left
unrepaired, all the rest of the windows will soon
be broken”
• Implication: People see the building as the sort of place
where it is OK to break windows
• Implication #2: If you fix the window, people see this as
a place where people behave nicely
• Broad sweeping implication: If you eliminate petty
crimes, people will stop perpetrating major crimes
A Case Study in Causal Investigation
• Primary concern: The theory is only interesting if
it is causal. Consider the alternatives:
– 1. Reverse causation: When you reduce major crimes,
people are also less likely to commit minor crimes
(e.g., if I stop people from stealing cars, I also stop
them from breaking windows)
– 2. Third variable: If I remove all the criminals from
circulation, there will be fewer broken windows AND
fewer stolen cars
• Now, given the methodologies we have thus far,
how would you try to evaluate the theory?
A Case Study in Causal Investigation
• Possibilities:
– Qualitative Data
– Survey Design
– Observational Research
– Archival Research/Data Mining
• Archival Analysis:
• Crime in New York City
Primary Conclusion:
“… an increase in the size of the police force generates a decrease in
robberies and burglaries.”
Corman and Mocan, 2000
A Case Study in Causal Investigation
• Possibilities:
– Qualitative Data
– Survey Design
– Observational Research
– Archival Research/Data Mining
– Experimental Design
1. Experimentation is the conscious manipulation of one or more variables by the experimenter
in such a way that its effect on one or more variables can be measured.
2. The variable being manipulated is called the independent variable (a.k.a. cause).
3. The variable being measured is called the dependent variable (a.k.a. effect).
4. Elimination of other possible causal factors: i.e., the research design should rule out the
other factors (exogenous variables) as potentially causal ones.
5. This is typically done through random assignment to condition
Experimental Design: Example 1
• Independent Variable
– “Policing Disorder” vs. Control
• Dependent Variable
– Service calls for five serious crimes
Control
Policing Disorder
Braga and Bond, 2008
Results
But what limitations do we see in this?
Experimental Design
Experimenters attached a paper
flyer to each bicycle and recorded
whether or not people dropped
the flyer or took it with them.
Control
Disorderly Setting
Keizer, Lindenberg, & Steg, 2008
Results
Percentage Littering
Control
33%
69%
Disorderly Setting
Expt. 1
Independent Variable
Dependent Variable
Graffiti
Littering
Further Experiments
Independent Variable
Dependent Variable
Expt. 1
Expt. 2
Expt. 3
Expt. 4
Graffiti
Illegally Parked Bicycles
Unreturned Shopping Carts
Setting off Fireworks
Littering
Trespassing
Littering
Littering
Expt. 5
Expt. 6
Graffiti
Litter
Stealing
Stealing
But what are the shortcomings of this design?
Key Points
• Experimentation is necessary to infer causality
• A poorly designed experiment will not allow you to
infer causality
• A good experiment should achieve:
– High internal validity through appropriate choice of
experimental design.
– High external validity by keeping the experimental setting
as close to the real marketing environment as possible.
– There is a trade-off here…
• A good control group is often a key requirement of a
good experimental design.
How should experimentation be used
in marketing?
Focus is on detecting causal
relationships between variables
New Customer
Service Program
?
Customer Satisfaction
Causal Research in Marketing
• Many examples of the need for causal effects
in Marketing
• 4P’s alone...
• How do we actually identify causal effects?
Key tool: Experiments
How to Run Experiments?
• How do we actually run experiments?
• In the sciences there is a long history of lab
experiments
• In a lab it is relatively easy to control external
conditions that might affect the validity of the
experiment
• What is the situation in Marketing?
Experiments in Marketing
 Usually take the form of a comparison
between a test and (at least one) control
group
 Experiments are frequently run as field
experiments
Test Group
New
Marketing
Tactic
Control Group
Old Marketing
Tactic
Customer Pool
Example: Price Experiment
• A catalog retailer selling women’s apparel
• Conducted price experiment to estimate
demand curves
• One control group and 4 test groups
• Each group consisted of 15,000 randomly
sampled customers
Version 1
Version 2
Version 3
Version 4
Version 5
The Concept of Validity
• Internal Validity:
– Refers to the ability of the experiment to unambiguously
show a cause and effect relationship, i.e., to what extent
can we attribute the effect that was observed to the
experimental variable and not other factors?
• External Validity:
– Refers to the extent to which the results of the experiment
can be generalized from the experimental environment to
the environment of the decision maker; i.e., the real world
• There is a trade-off between internal and external
validity, from a managerial perspective.
Test Markets
• Many uses:
– Controlled introduction of a new product
– Change of pricing strategy
– Change of product design
•
•
•
•
Choose representative markets
Often as long as one year
Expensive – but highly informative
Three Types:
– Simulated Test Markets
– Controlled Test Marketing
– Sell-In test Marketing
Simulated Test Marketing (STM)
• (a.k.a Laboratory Test Markets)
• Simulates an actual test market to estimate
– initial purchase rates
– ultimate repeat purchase rates
• Advantages: Compared to true test markets…
–
–
–
–
Fast (3 months)
Relatively cheap ($250,000)
Flexible
Impressive accuracy rates
• Limitations
– assumes preference data and purchase/repurchase decisions are valid
predictors of what would actually happen in the market place
– convenience sample
– attrition
• Despite this, laboratory test markets are one of the biggest success
stories in market research
Example: Typical STM Procedure
Step 1: Recruit
Qualified
Shoppers
Step 2: Background Questions
Familiarity, Preferences, Usage
Step 4: Simulated Shopping
• Respondents given money
• Invited into mock/real store
• Where they may buy any item
Step 3: Screening of Ads
Ad for Target Product +
Others
Step 6: Reinterview
Step 5: Debriefing
• Choices recorded (TRIAL RATE)
• Reasons for (non) purchase
• Non buyers given free sample
• Contacted after few weeks
• Product attitude, usage,
satisfaction
• Repurchase (REPURCHASE RATE)
• Intended & Actual
Controlled Test Marketing
• Cities for distribution is prearranged
• Purchases of a panel of consumers are monitored
through scanner data
• Example
–
–
–
–
IRI BehaviorScan
3,000 households in 7 cities
ID card presented to supermarket
In-store conditions - price, promotion, displays controlled and monitored
– Device on TV allows channel selection to be
monitored and ads to be substituted
Sell-In Test Marketing
• Cities where product is sold just as it would be
in a national launch
• Must gain distribution space
• Issues
– selecting the test cities
– implementing and controlling the test
– timing
– evaluative measures
– costs
Six U.S. Common Test Markets
Downsides to Test Markets?
Other Downsides?
Key Points
• Experimentation is very useful in marketing to
determine true causal effects of marketing
decisions
• Test markets are the primary example,
although smaller scale experiments are
common
– Test markets represent a trade off between
internal and external validity
– They have other downsides, too
Summary
• Experiments are awesome, right?
• They can demonstrate causality, which makes
them useful tools for marketers
• When you design an experiment, keep in mind
the balance you are looking for with regard to
internal and external validity
– Test markets give you a lot of external validity,
which can be good and bad