7 Experimentation and
Test Markets
What is an Experiment?
• Experiment: research approach in which one
variable (called an experimental, treatment,
independent, or explanatory variable) is
manipulated and the effect on another variable
(referred to as a dependent variable) is observed.
• Example: the effect of advertising on sales or
the effect of price on sales.
– Advertising/price: experimental variable
– Sales: dependent variable
Demonstrating Causation
• Causal research: research designed to
determine whether a change in one variable
likely caused an observed change in another.
• To demonstrate causation (that A likely caused
B), one must be able to show 3 things:
1. Correlation or concomitant variation
2. Appropriate time order of occurrence
3. Elimination of other possible causal factors
Demonstrating Causation
• Concomitant variation
– To provide evidence that a change in A caused a
particular change in B, one must first show that
there is correlation between A and B; in other
words, A and B must vary together in some
predictable fashion.
– This might be a positive (i.e. advertising and sales)
or inverse relationship (i.e. price and sales).
Demonstrating Causation
• Appropriate time order of occurrence
– To demonstrate that A caused B, one must be able
to show that A occurred before B occurred.
– Example: to demonstrate that a price change had
an effect on sales, you must be able to show that
the price change occurred before the change in
sales was observed.
Demonstrating Causation
• Elimination of other possible causal factors
– The most difficult thing to demonstrate in marketing
experiments is that the change in B was not caused by
some other than A.
– Example: suppose a company increased its advertising
expenditures and observed an increase in sales. It is
possible that the observed change in sales is due to
some factor other than increase in advertising: a
major competitor might have decreased advertising
expenditures, or increased price, or pulled out of the
market, so one or a combination of other factors may
have influenced sales.
Experimental Setting
Experiments can be conducted in a laboratory or in a field
setting. Most experiments in the physical sciences are
conducted in a laboratory setting; while many
marketing experiments are field experiments.
• Laboratory experiments: experiments conducted in a
controlled setting.
• Field experiments: tests conducted outside the
laboratory in an actual environment, such as a
marketplace.
Experiment Validity
• Validity: the degree to which an experiment actually
measures what the researcher was trying to measure.
The validity of a measure depends on the extent to
which the measure is free from both systematic and
random error.
– Internal validity: extent to which competing explanations
for the experimental results observed can be ruled out.
– External validity: extent to which causal relationships
measured in an experiment can be generalized to outside
persons, settings, and times.
Field experiments offer a higher degree of external validity
and a lower degree of internal validity than do laboratory
experiments.
Extraneous Variables
Examples of extraneous factors/variables that pose a threat to
experiment validity are:
• History: intervention, between the beginning and end of an
experiment, of outside variables or events that might change
the dependent variable.
– Example: reading potential sales of Prego whie Ragu
is having its promotional campaign may not get the
accurate sales reading.
• Maturation: changes in subjects occurring during the
experiment that are not related to the experiment (a function
of time) but that may affect subjects’ response to the
treatment factor. It includes getting older, hungrier, more
tired, and etc.
Extraneous Variables
Examples of extraneous factors/variables that pose a threat to
experiment validity are:
• Instrument variation: changes in measurement instruments (e.g.
interviewers or observers) that might affect measurements.
• Selection bias: systematic differences between the test group and
the control group due to a biased selection process.
• Mortality: loss of test units or subjects during the course of an
experiment, which may result in a nonrepresentativeness.
Example: in a study of music preferences of the population, if
nearly all the subjects under the age of 25 were lost during the
course of the experiment, then the researcher would likely get a
biased picture of music preferences at the end of the experiment.
As a result, the finding would lack external validity.
Extraneous Variables
Examples of extraneous factors/variables that pose a threat to experiment
validity are:
• Testing effect: effect that is a by-product of the research
process itself.
Example: measuring attitude toward a product before exposing
subjects to an ad may act as a treatment variable, influencing perception
of the ad.
• Regression to the mean: tendency of subjects with extreme
behavior to move toward the average for that behavior during
the course of an experiment.
Example: the researcher might have chosen people for an
experiment group because they were extremely heavy users of a
particular product/service. In such situations, their tendency to move
toward the average behavior may be interpreted as having been caused
by the treatment variable when in fact it has nothing to do with the
treatment variable.
Extraneous Variables
Controlling Extraneous Variables
Causal factors that threaten validity must be controlled in some
manner to establish a clear picture of the effect of the
manipulated variable on dependent variable.
4 basic approaches are used to control extraneous factors:
• randomization: random assignments of subjects to treatment
conditions to ensure equal representation of subject characteristics.
• Physical control: holding constant the value or level of extraneous
variables (i.e. age, income, lifestyle) throughout the course of an
experiment.
• Design control: use of the experimental design to control
extraneous causal factors.
• Statistical control: adjusting for the effects of confounded variables
(extraneous causal factors) by statistically adjusting the value of the
dependent variable for each treatment condition.
Experimental Design, Treatment, and
Effects
• Experimental design: test in which the researcher has
control over and manipulates one or more
independent variables.
An experimental design includes 4 elements:
1. The treatment, or experimental, variable
(independent variable) that is manipulated
2. The subjects who participate in the experiment
3. A dependent variable that is measured
4. Some plan or procedure for dealing with extraneous
causal factors
Experimental Design, Treatment, and
Effects
• Treatment variable: independent variable that is
manipulated in an experiment.
– Manipulation: a process in which the researcher sets
the levels of the independent variable to test a
particular causal relationship.
– Example: to test the relationship between price
(independent v.) and sales (dependent v.), a researcher
might expose subjects to 3 different levels of price and
record the level of purchases at each price level.
As a variable that is manipulated, price is the single
treatment variable, with 3 treatment conditions/
levels.
Experimental Design, Treatment, and
Effects
• Experimental effect: effect of the treatment
variable on the dependent variable.
• Example: 3 different markets are selected to test
3 different prices (treatment conditions) for 3
months.
Market
 1
 2
 3
Price
Experimental effect (change in sales)
2% lower  increased less than 1%
4% lower  increased by 3%
6% lower  increased by 5%
Limitations of Experimental Research
• High cost of experiments
• Security issues
Competitors can decide whether and how to respond
or stolen concepts that were being tested in the
marketplace.
• Implementation problems
Example: a regional marketing manager might be
very reluctant to permit her market area to be used
as a test market for a higher price that might lower
sales for the area.
Selected Experimental Designs
• Pre-experiment designs
Designs that offer little or no control over extraneous factors. Studies using
this designs are often difficult to interpret because they offer little or no
control over the influence of extraneous factors.
– One-shot case study design: no pretest observations, no control group, and an
after measurement only (O1)
X
O1
– One-group pretest-posttest design: pre (O1)- and postmeasurements (O2) but
no control group
O1
X
O2
X: the exposure of an individual/a group to an experimental treatment
O (observation): the process of taking measurements on the test units
(individuals, groups of individuals, or entities whose response to the
experimental treatments is being tested: individual/group of consumers, retail
stores, total markets, and etc.)
Selected Experimental Designs
• True experimental designs
Research using an experimental group and a control group, to which test
units are randomly assigned (R). These experimental designs are superior
because randomization takes care of many extraneous variables.
– Before and after with control group design: involves random
assignment of subjects or test units to experimental and control
groups and pre – and postmeasurements of both groups.
Experimental group:
(R)
O1
X
O2
Control group:
(R)
O3
O4
– After-only with control group design: involves random assignment of
subjects or test units to experimental and control groups, but no
premeasurement of the dependent variable.
Experimental group:
(R)
X
O1
Control group:
(R)
O2
Selected Experimental Designs
• Quasi-experiments
Studies in which the researcher lacks complete control over the scheduling of
treatments or must assign respondents to treatments in a nonrandom manner.
These designs frequently are used in marketing research studies because cost and field
constraints often do not permit the researcher to exert direct control over the
scheduling of treatments and the randomization of respondents.
– Interrupted time-series design: research in which repeated measurement of an effect
“interrupts” previous data patterns. Ex: the use of consumer purchase panels to make periodic
measurements of consumer purchase activity (Os) with introducing a new promotional
campaign (X) .
O1 O2 O3 O4 X O5 O6 O7 O8
– Multiple time-series design: interrupted time-series design with a control group, then
researchers can be more certain in their interpretation of the treatment effect. Ex: an
advertiser were testing a new ad campaign in a test city, that city would constitute the
experiment group and another city (that was not expose to the new ad campaign) would be
chosen as the control group.
Experimental group:
O1 O2 O3 O4 O5 O6
Control group:
O1 O2 O3 O4 O5 O6
It is important the test and control cities be roughly equivalent in regard to characteristics related
to the sale of the product (e.g. competitive brands available)
Test Markets
• Test market: real-world testing of a new product or
some element of the marketing mix using an
experimental or quasi-experimental design.
• Test-market studies are designed to provide info.
regarding to the following issues:
– Estimates of market share and volume.
– The effects that the new product will have on sales of
similar products (if any) already marketed by the company.
(cannibalization rate)
– Characteristics of consumers who buy the product.
– The behavior of competitors during the test.
Test Markets
Types of test markets
• Traditional (standard) test market: testing the product and
other elements of the marketing mix through regular
channels of distribution which normally take 6 months or
more.
• Scanner (electronic) test market: research firms have
panels of consumers who carry scannable cards for use in
buying particular products, esp. those sold through grocery
stores. These panels permit us to analyze the
characteristics of those consumers who buy and don’t buy
the test products such as demographic data, pas purchase
history, or even media viewing habits.
Test Markets
Types of test markets
• Controlled test markets are managed by research suppliers
who ensure that the product is distributed through the
agreed upon types and numbers of outlets. The research
suppliers pay distributors to provide the required amount
of shelf space for test products and carefully monitor sales
of the product in these controlled test markets.
• Simulated test markets (STMs): simulations of the types of
test markets noted above. They can be conducted more
quickly than other approaches, at a lower cost, and can
produce results that are highly predictive of what will
actually happen.
Test Markets
Decision to conduct test marketing
1. Weigh the cost and risk of failure against the probability
of success and associated profits
2. Consider the likelihood and speed with which
competitors can copy your product and introduce it on a
national basis.
3. Consider the investment required to produce the
product for the test market vs. the investment required
to produce the product in the quantities necessary for a
national rollout.
4. Consider how much damage an unsuccessful new
product launch would inflict on the firm’s reputation.
Test Markets
Steps in a test market study
1. Define the objective of the test
2. Select a basic approach (type of test market)
3. Develop detailed test procedures (basic positioning
approach, the actual commercials to be used, pricing
strategy, media plan, and etc.)
4. Select test markets
5. Execute the plan
6. Analyze the test results throughout the test period
(purchase data, awareness data, competitive
response, source of sales)