Comparison of Observational, and Experimental Research
Strategies
Type of Study
Subtype
Purpose
Methodology
Observational
Sample
Comparative
Survey
observational studies
to learn about
to observe or measure
some large
the relationship
actual (e.g.,
between two variables
biological)
(or among several
population
variables)
Representative sample (subgroup)
chosen from the population for study
(questioning or examination).
Differences between groups in both
(or all) variables occur naturally.
Bias and confounding are possible
Bias and
Confounding
Intervention? No intervention, manipulation, or
treatment
Experiment
to measure the effect of an
explanatory variable on the
response or outcome variable
treatments assigned at random
to experimental units (or vice
versa) to avoid bias and
confounding.
Bias and confounding usually
avoided if done properly
explanatory variable = an
intervention, manipulation, or
treatment applied to the
experimental unit
response or outcome variable = a
characteristic of the experimental
units that is observed after the
treatment is applied
Results Conclusions
drawn from the
sample
generalized to
the population.
Causality?
Reference
Document1
Can confirm the
existence and measure
the strength of the
relationship under the
conditions of the
study.
Results do not imply cause and effect.
Experiment
Baldi & Moore (2009), Chapter 7
1
Results do imply cause and effect
(if done properly).
Baldi & Moore (2009), Chapter 8
2/6/2016
Examples comparing Survey, a Comparative Observational Study, and an Experiment
Variables of interest

Study time (hr)

Score on test (%)
Sampling

Study A
o 150 students randomly selected from registration records
o Each student contacted and asked how many hours they studied for their
last test, and their test score on the last test

Study B
o 150 students from Psych 101 required to schedule 1 hour appointment at
psychometrics lab to study a five page report and then take a test about it
o 50 students given 5 minutes to study, 50 given 15 minutes, 50 given 25
minutes

Study C
o 150 students questioned at McComas Hall (rec center)
o Each student asked how many hours they studied for their last test, and
their scores on the last test
Document1
2
2/6/2016
Confounding, Bias, and Sampling Designs
Confounding Two variables (explanatory variables or unobserved “lurking” variables) are
confounded when their effects on a response variable cannot be distinguished
from one another (from Baldi and Moore, 1999, Section 7.1, Observation versus
experiment)
Example
Storks and Births
Example 1 of Confounding
Table 1-A
The two explanatory variables, Flow and
Depth, are confounded. We cannot tell
whether the change in Algae
concentration is caused by Flow, Depth,
or a combination of the two.
Table 1-A. Effects of Flow and Dept
on Algae Concentration (g/l)
Algae
Conc (g/l)
12.31
Flow
rapid
Depth
shallow
moderate
medium
3.14
slow
deep
0.21
Table 1-B. Effects of Flow and Dept on Algae
Concentration (g/l)
Depth
shallow
medium
deep
rapid
12.31
?
?
moderate
?
3.14
?
slow
?
?
0.21
Document1
3
Flow
Table 1-B
is a rearrangement of Table 1-A, to
expose the confounding. Confounding
occurs when some combinations of levels
of the two explanatory variables have
been left out of the study.
2/6/2016
Example 2 of No Confounding
Table 2-A
The two explanatory variables, Flow and
Depth, are not confounded. We can see
that the response, Algae Concentration, is
explained predominantly by Flow rather
than Depth.
Flow
rapid
rapid
rapid
moderate
moderate
moderate
slow
slow
slow
Document1
Depth
shallow
medium
deep
shallow
medium
deep
shallow
medium
deep
Algae
Conc (g/l)
12.31
12.75
13.22
4.21
3.14
5.59
0.18
0.19
0.21
Table 2-B. Effects of Flow and Dept on Algae
Concentration (g/l)
Depth
Flow
Table 2-B
is a rearrangement of Table 2-A, to
expose the lack of confounding. When all
combinations of the levels of the two
explanatory variables are included, there
is no confounding.
Table 2-A. Effects of Flow and
Dept on Algae Concentration (g/l)
4
shallow
medium
deep
rapid
12.31
12.75
13.22
moderate
4.21
3.14
5.59
slow
0.18
0.19
0.21
2/6/2016
Bias
The design of a study or a variable in a study is biased if it systematically favors
certain outcomes or values (from Baldi and Moore, 1999, Chapter 7, with
embellishment)
Examples of Bias

Self-reported Age (yrs) of women in a speed dating context.

Self-reported Income ($) of men in a speed-dating context.

Range of a population variable when measured in a sample.
Sampling Designs
1. A simple random sample (SRS) of size n is a sample of individuals from a population
chosen in such a way that every set of n individuals has equal chance to be the sample
selected.
2. Stratified Random Sampling
3. Cluster Sampling
4. Systematic Sampling
5. Multi-Stage Sampling
Golde I. Holtzman, Department of Statistics, College of Arts and Sciences, Virginia Tech (VPI)
Last updated: 2/6/2016 © Golde I. Holtzman, all rights reserved.
Document1
5
2/6/2016