IS 4800 Empirical Research Methods
for Information Science
Class Notes March 13 and 15, 2012
Instructor: Prof. Carole Hafner, 446 WVH
hafner@ccs.neu.edu Tel: 617-373-5116
Course Web site: www.ccs.neu.edu/course/is4800sp12/
Parametric Statistics (numeric variables)
Assumes a (near-enough-to) normal population
distribution so these parameters make sense:
μ = the population mean (unknown)
σ2 = the population variance
σ = the population standard deviation
Samples of size N are used to estimate these parameters
M is the sample mean used to estimate μ
Calculate: M = Σ X
N
Relationship Between Population
and Samples When a Treatment
Had No Effect
Population

Sample 1
M1
Sample 2
M2
3
Relationship Between Population
and Samples When a Treatment
Had An Effect
Control
group
population
c
Treatment
group
population
t
Treatment
group
sample
Control
group
sample
Mc
Mt
4
What we must decide
Which one of these diagrams to believe ?????
How to express belief in the first diagram
How to express belief in the second diagram
How do we make that decision ?
• How far apart do the sample means need to be?
• We calculate this relative to information about the
variance !! ?
• Using a criterion alpha which is our tolerance
for being wrong !!
Estimating population variance
SS = Σ (X - M)2
SD2 = Σ (X - M)2
“Sum of Squares”
Sample variance
N
S2 = Σ (X - M)2
N–1
= SS
Estimated population variance
N-1
σ2M = true variance of the sample means
=
σ2
(unknown)
N
S2M = estimated variance of the sample means =
S2
N
Why do we care about the variance of the
sample means ?
• Sampling Distribution
– The distribution of means of every possible sample taken
from a population (with size N)
• Sampling Error
– The difference between a sample mean and the population
mean: M - μ
– The standard error of the mean is a measure of sampling
error (std dev of distribution of means)
Understanding numeric measures
• Sources of variance
– IV
– Other uncontrolled factors (“error variance”)
• If (many) independent, random variables with the
same distribution are added, the result approximately
a normal curve
– The Central Limit Theorem
8
The most important parts of the
normal curve (for testing)
5%
Z=1.65 9
The most important parts of the
normal curve (for testing)
2.5%
Z=-1.96
2.5%
Z=1.9610
Hypothesis testing – two tailed
• Hypothesis: sample (of 1) will be significantly
different from known population distribution
• Example – WizziWord experiment:
– H1:  WizziWord   Word
– a = 0.05 (two-tailed)
– Population (Word users):  Word =150, s=25
– What level of performance do we need to see before we can
accept H1?
11
Hypothesis testing – two tailed
• Hypothesis: sample (of 1) will be significantly
different from known population distribution
• Example – WizziWord experiment:
– H1:  WizziWord   Word
– a = 0.05 (two-tailed)
– Population (Word users):  Word =150, s=25
– What level of performance do we need to see before we can
accept H1?
• Must see performance >1.96 stddevs above mean = 199
• BUT, also if performance < 1.96 stddevs below mean = 101
• Will reject H0.
12
Standard testing criteria for
experiments
• a = 0.05
• Two-tailed
13
Don’t try this at home
• You would never do a study this way.
• Why?
– Can’t control extraneous variables through
randomization.
– Usually don’t know population statistics.
– Can’t generalize from one individual.
14
Sampling
Mean?
Variance?
Population

Sample of size N
Mean values from all possible
samples of size N
aka “distribution of means”
s2
X

M=
SD 2 =
2
(
X

M
)

N
MM = 
s M2 =
N
s2
N
ZM = ( M -  ) / s M
Z tests and t-tests
t is like Z:
Z=M-μ/ s
M
t=M–0/ S
M
We use a stricter criterion (t) instead of Z
because S is based on an estimate of the
M
population variance while s Mis based on a
known population variance.
T-test with paired samples
Given info about
population of change
scores and the
sample size we will
be using (N)
We can compute the
distribution of means
?
=0
S2 est s2 from sample = SS/df
Now, given a
particular sample of
change scores of
size N
S2M = S2/N
We compute its mean
and finally determine
the probability that this
mean occurred by
chance
t=
M
SM
df = N-1
t test for independent samples
Given two
samples
Estimate population
variances
(assume same)
Estimate variances
of distributions
of means
Estimate variance
of differences
between means
(mean = 0)
This is now your
comparison distribution
Estimating the Population Variance
S2 is an estimate of σ2
S2 = SS/(N-1) for one sample (take sq root for S)
For two independent samples – “pooled estimate”:
S2 = df1/dfTotal * S12 + df2/dfTotal * S22
dfTotal = df1 + df2 = (N1 -1) + (N2 – 1)
From this calculate variance of sample means: S2M = S2/N
needed to compute t statistic
t test for independent samples, continued
Distribution of differences
between means
This is your
comparison distribution
NOT normal, is a ‘t’
distribution
Shape changes depending on
df
df = (N1 – 1) + (N2 – 1)
Compute t = (M1-M2)/SDifference
Determine if beyond cutoff score
for test parameters (df,sig, tails)
from lookup table.
Effect size
• The amount of change in the DVs seen.
• Can have statistically significant test but small
effect size.
21
Power Analysis
• Power
– Increases with effect size
– Increases with sample size
– Decreases with alpha
• Should determine number of subjects you need ahead
of time by doing a ‘power analysis’
• Standard procedure:
– Fix alpha and beta (power)
– Estimate effect size from prior studies
• Categorize based on Table 13-8 in Aron (sm/med/lg)
– Determine number of subjects you need
– For Chi-square, see Table 13-10 in Aron reading
22