Experimental Design: Part I
MAR 6648: Marketing Research
February 3, 2010
Overview
• What are the basic features of an experiment?
• How do those features get implemented in a
real experiment?
• How do we adapt experiments to meet our
goals and resources?
1. Experimentation is the conscious manipulation of one or more variables by the
experimenter in such a way that its effect on one or more variables can be
measured.
2. The variable being manipulated is called the independent variable (a.k.a. cause).
3. The variable being measured is called the dependent variable (a.k.a. effect).
4. Elimination of other possible causal factors: i.e., the research design should rule
out the other factors (exogenous variables) as potentially causal ones.
5. This is typically done through random assignment to condition
An example of an experiment
• Suppose you want to know whether commercials
make people enjoy TV shows less
• This means you’ll want to have some shows
without commercials, and some shows with them
– Therefore, commercials (or not) is the independent
variable
• And you’ll want to measure enjoyment of the TV
shows they watch
– Therefore, enjoyment is the dependent variable
Conditions
• Not in terms of what you can and can’t do…
• Each independent variable (or combination of
IVs) is called a condition
Condition 1
Condition 2
Hypotheses
• Experimentation is essentially the process of trying to
determine which of two hypotheses is not false
• The null hypothesis:
– H0: Usually that there are no differences between
conditions
• The alternative hypothesis:
– H1: usually that there is a difference between conditions
• P-values in stats essentially represent the likelihood
that we found evidence for H1 by chance alone
Confirmation Bias
• We are inclined to confirm our beliefs but less
inclined (or able) to disconfirm them
• A real world example:
– Business managers don’t keep track of those they
don’t hire
• Why?
– Theories lead to unwarranted confidence
– Inability to search out disconfirmation
– Fixation or mental set
Control condition
• Control conditions allow us to see that our
manipulations caused (or didn’t cause) a change
in the dependent variable
• Usually a control condition is just no
manipulation
– This is sometimes done by adjusting when you run
your manipulation
• Sometimes, though, you want to compare your
new manipulation to what’s typically done now
– The control condition may be the standard or default
Random Assignment
• This essentially means that any one
participant is equally likely to be in any
condition
– Usually you put your conditions in random order,
and assign participants in the order that they
“arrive”
– Computers now allow you to assign people on the
spot
• Randomizer.org or random.org are good sources
An example of an experiment
• The hallmark of an experiment is random
assignment to conditions
– Let’s say the groups (the commercial watchers and the
people who watch it straight through) now look
different!
– Random assignment means that the two groups
should not have differed systematically at the start
– It also means that only your independent variable was
different between groups
• Random assignment and manipulation of the IV
mean that you can infer that the IV causes a
change in the DV
An example of an experiment
• Question: do commercials make you enjoy a TV
show less? Do people correctly predict this?
• Randomly assign your participants to groups
– Half will predict how they enjoy a TV show with or
without them, half will actually experience it and
report how they feel
– Half will watch a TV show with commercials, half will
watch the same show without them
• Measure enjoyment or predicted enjoyment
An example of an experiment
Nelson, Meyvis, & Galak, 2009
Example
• Objective: GAP wishes to gauge whether new
more aggressive sales techniques employed
by store assistants increase sales
• What is the best experimental design?
Experiment 1
• Design:
– 50 stores are sampled at random and assistants
are trained in the new approach
Metric =
Average sales for the 50 stores
in the next six months
MINUS
Average sales for the 50 stores
in the prior six months
Notation
• X = Exposure of a sample to the independent
variable (i.e., what we manipulate – “treatment”)
• O = Observation of measurement of the
dependent variable (i.e., what we measure / want
to affect)
• Movement through time is represented by the
horizontal arrangement of Xs and Os from left to
right.
Experiment 1: One group – before after
Causal Effect of X = O2-O1
Problems with this design?
•
•
•
•
History or maturation
Defensiveness
Mortality
Instrumentation
Experiment 2
• Design:
– 50 stores are sampled at random and the
assistants are trained in the new approach
– Another 50 stores are sampled at random as
control
Metric =
Average sales for the 50 test stores
in the next six months
MINUS
Average sales for the 50 control stores
in the next six months
Experiment 2: Two group – only after
Causal Effect of X = O2-O1
Problems with this design?
Experiment 3
• Design:
– 50 stores are sampled at random and assistants
are trained in the new approach
– Another 50 stores are sampled at random as
control
Average sales for the 50 test stores in the next six
Metric =
months
Average sales for theMINU
50 test stores in the prior six
S
months
MIN
Average sales for the 50 control stores in the next
US
six months
Average sales for theMINU
50 control stores in the prior
S
six months
Experiment 3: Two group – before after
Causal Effect of X = O4-O3 – (O2-O1)
More Advanced Experiments
• We have so far mainly looked at simple
experiments
• But often we need to test several variables
• When deciding on a marketing plan for a new
product there are many factors involved
Factorial Design
• Suppose we wish to test both product price
and web-design for an e-business
Full
Factorial
Design!
Price
Design
$9.99
Design 1
Design 2
$14.99
$19.99
Interactions and main effects
Factorial Design
• What do we do if we have many factors and
levels?
• Example:
– 5 prices, 4 product designs, 3 ad-copies  5*4*3
= 60 experimental cells!
• Solution: Use a fractional factorial design
– Only use a subset of all 60 cells in experiment
– Rely on regression analysis to extrapolate
Latin Squares
1st ad
2nd ad
3rd ad
4th ad
Group #1
Positive ad,
Male speaker
Positive ad,
Female
speaker
Negative ad,
Female
speaker
Negative ad,
Male speaker
Group #2
Positive ad,
Female
speaker
Negative ad,
Male speaker
Positive ad,
Male speaker
Negative ad,
Female
speaker
Group #3
Negative ad,
Male speaker
Negative ad,
Female
speaker
Positive ad,
Female
speaker
Positive ad,
Male speaker
Group #4
Negative ad,
Female
speaker
Positive ad,
Male speaker
Negative ad,
Male speaker
Positive ad,
Female
speaker
Latin Squares
1st ad
2nd ad
3rd ad
4th ad
Group #1
α
β
δ
γ
Group #2
β
γ
α
δ
Group #3
γ
δ
β
α
α
γ
β
Group #4
δ
An experiment?
• Steve was interested to see how much labels on
wine bottles affect how much people enjoy the
wine inside them. At a party, he served the wines
like normal, leaving the bottles out for people to
pour from, labels still on. He asked everyone to
indicate which wine they liked the best. At the
next party he threw, he poured the wine into
decanters, so that his guests couldn’t see the
labels when they poured the wine. They again
indicate which wine they liked best, and they had
different preferences from the last party.
An experiment?
• The owner of two McDonalds franchises here
in Gainesville wants to see if transactions run
more quickly if he uses both drive-thru
windows or only one. He picks one restaurant
to use both windows at all times for a month,
and the other he has closed at all times for a
month. He finds that the drive-thru that uses
both windows has notably faster service
times.
An experiment?
An experiment?
Summary
• Experiments are very useful for determining
causality
– The main hallmarks of experiments are random
assignment to condition, manipulation of the
independent variable, and a control group
– There are many different types of experiments,
which vary largely on whether they are run within
or between subjects (or both), when the
manipulation is run, and how many conditions are
used