Experimental Design
Playing with
variables
The nature of experiments
allow the investigator to control the
research situation so that causal
relationships among variables may
be evaluated
 One variable is manipulated and its
effect upon another variable is
measured, while other variables are
held constant

So… you’ve decided to do an
experiment

Decisions… decisions… decisions
Decision 1: Independent Variable?
value is changed or altered
independently of other variables
 hypothesized to be the causal influence
 categorical or continuous (?)

Experimental Treatments:
 alternative manipulations of the
Independent Variable
Experimental and Control Groups

Control Group
 Experimental
25
Groups
20
 there can be more
15
than one treatment
IV
level of the
10
treatment
Independent
5
Variable (basic or
0
ControlExperimental
Exp 3 Groups
factorial)
 there can be more
than one IV
Decision 2: Dependent Variable
The criterion or standard by which the
results are judged
 It is presumed that changes in the
Dependent Variable are the result of
changes in one or more Independent
Variable
 the choice of Dependent Variable
determines the type of answer that is
given to the research question

Decision 3: Test units/unit of
analysis
The subjects or entities whose
responses to the experimental
treatment are being measured
 People are the most common test unit
in business research

Decision 4: Extraneous variables
A number of extraneous or “other”
variables may affect the dependent
variable and distort the results
Conditions of constancy:
 When extraneous variables cannot be
eliminated we strive to hold Extraneous
Variables constant for all subjects

But, what about ___________?

Problems… problems…
IMPACT OF THE RESEARCH SITUATION
Demand Characteristics: experimental design
procedures that unintentionally hint to subjects about
the experimenter’s hypothesis
 rumour
 instructions
 status and personality of researcher
 unintentional cues from experimenter
 experimental procedure itself
 Setting: Field versus Laboratory
Field versus Laboratory

Field experiments: usually used to
fine-tune strategy and determine
sales volume

Laboratory: used when control over
the experimental setting is more
important
Experimental Design
effects….
The Hawthorne effect
Subjects perform differently just because they
know they are are experimental subjects
Western Electric’s Hawthorne Plant 1939 study
of light intensity
The Guinea Pig effect
exhibit the behaviour that they think is expected
Potential Solutions:
run experiment for a longer period
use a control group
Deception (?)
Experimental Treatment Diffusion
 if treatment condition perceived as very desirable
relative to the control condition, members of the
control group may seek access to the treatment
condition
 Potential Solutions:
-have control group in another site
-of course, this introduces new variables!
John Henry Effect
 legend of black railway worker
 control group overcompensates
 Potential Solutions:
 don’t do threatening experiments
 don’t set up obviously competitive situations
 don’t tell control group that they are control group
• conduct in another location somewhere else
• unfortunately, produces new variable of different location,
neighbourhood, etc.!
Resentful Demoralization of Control
Group

Control group artificially demoralized if perceives
experimental group receiving desirable treatment
being withheld from it

Potential Solutions?
 what about giving control group some perk to compensate?
 don’t tell them they are control group! (but what about
informed consent?)… Use of Placebo… use of blinding…
Getting control….

Design decisions

Physical Control
– Holding the value or level of extraneous
variables constant throughout the course of
an experiment.

Statistical Control
– Adjusting for the effects of confounding
variables by statistically adjusting the value
of the dependent variable for each
treatment conditions.

Design Control
– Use of the experimental design to control
extraneous causal factors.
Blinding
• Blinding is utilized to control subjects knowledge of
whether or not they have been given a particular
experimental treatment
• double-blind experiment
• secrecy
• but then violate principle of informed consent
• screen out or balance number of placebo reactors in
treatment & control groups
Sampling
Who and How
And How to Screw It
up
Terms
Sample
 Population (universe)
 Population element
 census

Why use a sample?
Cost
 Speed
 Sufficiently accurate (decreasing
precision but maintaining accuracy)
 More accurate than a census (?)
 Destruction of test units

Stages in the Selection of a
Sample
Step 7: Conduct
Fieldwork
Step 6: Select
Sampling units
Step 2: Select
The Sampling
Frame
Step 1: Define the
the target population
Step 5: Determine
Sample Size
Step 3: Probability
OR Non-probability?
Step 4: Plan
Selection of
sampling
units
Step 1: Target Population
The specific, complete group
relevant to the research project
 Who really has the information/data
you need
 How do you define your target
population


Bases for defining the population of
interest include:
• Geography
• Demographics
• Use
• Awareness
Operational Definition

A definition that gives meaning to a
concept by specifying the activities
necessary to measure it.


“The population of interest is defined as
all women in the City of Lethbridge who
hold the most senior position in their
organization.”
What variables need further definition?
Step 2: Sampling Frame
 The
list of elements from which
a sample may be drawn.
Also known as: working population.
 Examples?

Sampling Frame Error:

error that occurs when certain
sample elements are not listed or
available and are not represented in
the sampling frame.
Sampling Units:

A single element or group of
elements subject to selection in the
sample.
Primary sampling unit
 Secondary sampling unit

Error: Less than perfectly.
representative samples.

Random sampling error.

Difference between the result of a sample and
the result of a census conducted using
identical procedures; a statistical fluctuation
that occurs because of chance variation in the
selection of the sample.
…Error

Systematic or non-sampling error.
 Results from some imperfect aspect of
the research design that causes
response error or from a mistake in the
execution of the research
 Examples: Sample bias, mistakes in
recording responses, non-responses,
mortality etc,.
…Error

Non-response error.

The statistical difference between a
survey that includes only those who
responded and a survey that also
includes those that failed to respond.
Step 3: Choice!

Probability Sample:

A sampling technique in which every
member of the population will have a
known, nonzero probability of being
selected
Step 3: Choice!

Non-Probability Sample:
Units of the sample are chosen on the
basis of personal judgment or
convenience
 There are no statistical techniques for
measuring random sampling error in a
non-probability sample. Therefore,
generalizability is never statistically
appropriate.

Classification of Sampling
Methods
Sampling
Methods
Probability
Samples
Systematic
Cluster
Nonprobability
Stratified
Simple
Random
Convenience
Judgment
Snowball
Quota
Probability Sampling
Methods

Simple Random Sampling
the purest form of probability sampling.
 Assures each element in the population
has an equal chance of being included in
the sample
 Random number generators

Sample Size
Probability of Selection = Population Size

Advantages
minimal knowledge of population needed
 External validity high; internal validity
high; statistical estimation of error
 Easy to analyze data


Disadvantages
High cost; low frequency of use
 Requires sampling frame
 Does not use researchers’ expertise
 Larger risk of random error than stratified


Systematic Sampling
An initial starting point is selected by a
random process, and then every nth
number on the list is selected
 n=sampling interval
 The number of population elements
between the units selected for the
sample
 Error: periodicity- the original list has a
systematic pattern
 ?? Is the list of elements randomized??


Advantages
Moderate cost; moderate usage
 External validity high; internal validity
high; statistical estimation of error
 Simple to draw sample; easy to verify


Disadvantages
Periodic ordering
 Requires sampling frame


Stratified Sampling
Sub-samples are randomly drawn from
samples within different strata that are
more or less equal on some characteristic
 Why?
Can reduce random error

More
accurately reflect the
population by more proportional
representation

How?
1.Identify variable(s) as an efficient
basis for stratification. Must be known
to be related to dependent variable.
Usually a categorical variable
2.Complete list of population elements
must be obtained
3.Use randomization to take a simple
random sample from each stratum

Types of Stratified Samples
 Proportional Stratified Sample:

The number of sampling units drawn
from each stratum is in proportion to
the relative population size of that
stratum
 Disproportional

Stratified Sample:
The number of sampling units drawn
from each stratum is allocated
according to analytical considerations
e.g. as variability increases sample
size of stratum should increase

Types of Stratified Samples…
 Optimal allocation stratified sample:
The number of sampling units drawn from
each stratum is determined on the basis of
both size and variation.
 Calculated statistically


Advantages
Assures representation of all groups in
sample population needed
 Characteristics of each stratum can be
estimated and comparisons made
 Reduces variability from systematic


Disadvantages
Requires accurate information on
proportions of each stratum
 Stratified lists costly to prepare


Cluster Sampling
The primary sampling unit is not the
individual element, but a large cluster of
elements. Either the cluster is randomly
selected or the elements within are
randomly selected
 Why? Frequently used when no list of

population available or because of cost
Ask:
is the cluster as heterogeneous as
the population? Can we assume it is
representative?

Cluster Sampling example
You are asked to create a sample of all
Management students who are working in
Lethbridge during the summer term
 There is no such list available
 Using stratified sampling, compile a list of
businesses in Lethbridge to identify
clusters
 Individual workers within these clusters
are selected to take part in study


Types of Cluster Samples
 Area sample:

Primary sampling unit is a
geographical area
 Multistage

area sample:
Involves a combination of two or more
types of probability sampling
techniques. Typically, progressively
smaller geographical areas are
randomly selected in a series of steps

Advantages





Low cost/high frequency of use
Requires list of all clusters, but only of
individuals within chosen clusters
Can estimate characteristics of both cluster and
population
For multistage, has strengths of used methods
Disadvantages


Larger error for comparable size than other
probability methods
Multistage very expensive and validity depends
on other methods used
Classification of Sampling
Methods
Sampling
Methods
Probability
Samples
Systematic
Cluster
Nonprobability
Stratified
Simple
Random
Convenience
Judgment
Snowball
Quota
Non-Probability Sampling
Methods

Convenience Sample
The sampling procedure used to obtain
those units or people most conveniently
available
 Why: speed and cost
 External validity?
 Internal validity
 Is it ever justified?


Advantages
Very low cost
 Extensively used/understood
 No need for list of population elements


Disadvantages
Variability and bias cannot be measured
or controlled
 Projecting data beyond sample not
justified.


Judgment or Purposive Sample

The sampling procedure in which an
experienced research selects the sample
based on some appropriate characteristic
of sample members… to serve a purpose

Advantages
Moderate cost
 Commonly used/understood
 Sample will meet a specific objective


Disadvantages
Bias!
 Projecting data beyond sample not
justified.


Quota Sample

The sampling procedure that ensure that
a certain characteristic of a population
sample will be represented to the exact
extent that the investigator desires

Advantages
moderate cost
 Very extensively used/understood
 No need for list of population elements
 Introduces some elements of
stratification


Disadvantages
Variability and bias cannot be measured
or controlled (classification of subjects0
 Projecting data beyond sample not
justified.


Snowball sampling

The sampling procedure in which the
initial respondents are chosen by
probability methods, and then additional
respondents are obtained by information
provided by the initial respondents

Advantages
low cost
 Useful in specific circumstances
 Useful for locating rare populations


Disadvantages
Bias because sampling units not
independent
 Projecting data beyond sample not
justified.
