What would it take to Change your Inference? Quantifying the
Discourse about Causal Inferences in the Social Sciences
Kenneth A. Frank
Help from Yun-jia Lo and Mike Seltzer
AERA workshop April 4, 2014 (AERA on-line video – cost is $95)
Motivation
Statistical inferences are often challenged because of uncontrolled bias. There may be bias due to
uncontrolled confounding variables or non-random selection into a sample. We will answer the
question about what it would take to change an inference by formalizing the sources of bias and
quantifying the discourse about causal inferences in terms of those sources. For example, we will
transform challenges such as “But the inference of a treatment effect might not be valid because of preexisting differences between the treatment groups” to questions such as “How much bias must there
have been due to uncontrolled pre-existing differences to make the inference invalid?”
Approaches
In part I we will use Rubin’s causal model to interpret how much bias there must be to invalidate an
inference in terms of replacing observed cases with counterfactual cases or cases from an unsampled
population. In part II, we will quantify the robustness of causal inferences in terms of correlations
associated with unobserved variables or in unsampled populations. Calculations for bivariate and
multivariate analysis will be presented in the spreadsheet for calculating indices [KonFound-it!] with
some links to SPSS, SAS, and Stata.
Format
The format will be a mixture of presentation, individual
exploration,
and group work. Participants may
Correlational
Framework
include graduate students and professors, although all must be comfortable with basic regression and
Impact of a Confounding variable
multiple regression. Participants should bring their own laptop, or be willing to work with another
Thresholds
for inference and % bias to invalidate
Internal
validity
student who has a laptop. Participants may choose to
bring to
the course an example of an inference
The
counterfactual
paradigm
Replacement
Cases
Framework
from a published
study or their
own work,
as well asExample:
data analyses
they
currently conducting.
Effect
of are
kindergarten
retention
overview
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen
Court curriculum
Materials to download: spreadsheet for
calculating
indices
[KonFound-it!]
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the
framework
powerpoint with
examples
and
calculations
Correlational Framework
Motivation
• Inferences uncertain
• it’s causal inference not determinism
• Instead of “you didn’t control for xxx”
• Peronal: Granovetter to Fernandez
• What would xxx have to be to
change your inference
• Promotes scientific discourse
Correlational Framework
Impact of a Confounding variable
• Informs pragmatic policy
Thresholds
for inference and % bias to invalidate
Internal validity
overview
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Replacement Cases Framework (40 minutes to reflection)
Thresholds for inference and % bias to invalidate and inference
The counterfactual paradigm
Application to concerns about non-random assignment to treatments
Application to concerns about non-random sample
Reflection (10 minutes)
Examples of replacement framework
Internal validity example: Effect of kindergarten retention on achievement
(40 minutes to break)
External validity example: effect of Open Court curriculum on achievement
Review and Reflection
Extensions
Extensions of the framework
Exercise and break (20 minutes: Yun-jia Lo, Mike Seltzer support)
Correlational Framework (25 minutes to exercise)
How regression works
Impact of a Confounding variable
Internal validity: Impact necessary to
invalidate an
inference
Correlational
Framework
Example: Effect of kindergarten retention
on achievement
Impact
of a Confounding variable
Thresholds
for inference and % bias to invalidate
Exercise (25 minutes: Mike Seltzer supports
on-line)
Internal
validity
counterfactual paradigm
External
Replacement
validity (30
Cases
minutes)
Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
combining
estimates
from different populations
Correlational
Framework
External
validity
External
validity
exampleOpen Court curriculum
overviewexample:
effect
of
Open
Court
curriculum
on
achievement
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Materials for course
https://www.msu.edu/~kenfrank/research.htm#causal
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Quick Survey
Can you make a causal inference
from an observational study?
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Answer: Quantifying the
Discourse
Can you make a causal inference
from an observational study?
Of course you can. You just might be wrong.
It’s causal inference, not determinism.
But what would it takeCorrelational
for theFramework
inference
Impact of a Confounding variable
to be wrong?
Thresholds
for inference and % bias to invalidate
Internal validity
overview
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
I: Replacement of Cases
Framework
How much bias must there be to
invalidate an inference?
Concerns about Internal Validity
• What percentage of cases would you have
to replace with counterfactual cases (with
zero effect) to invalidate the inference?
Concerns about External Validity
• What percentage of cases
would you have
Correlational Framework
to replace with cases from
anof aunsampled
Impact
Confounding variable
Thresholds
for inference and % bias to invalidate
Internal
validity
population (with zero effect)
to
invalidate
The
counterfactual
paradigm
Replacement Cases Framework Example: Effect of
kindergarten retention
Internal validity example: kindergarten retention
theCorrelational
inference?
Framework
External validity
overview
Conclusion
External validity exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
What Would It Take to Change an Inference? Using Rubin’s Causal Model to
Interpret the Robustness of Causal Inferences
Abstract
We contribute to debate about causal inferences in educational research in two
ways. First, we quantify how much bias there must be in an estimate to invalidate
an inference. Second, we utilize Rubin’s causal model (RCM) to interpret the bias
necessary to invalidate an inference in terms of sample replacement. We apply our
analysis to an inference of a positive effect of Open Court Curriculum on reading
achievement from a randomized experiment, and an inference of a negative effect
of kindergarten retention on reading achievement from an observational study. We
consider details of our framework, and then discuss how our approach informs
judgment of inference relative to study design. We conclude with implications for
scientific discourse.
Keywords: causal inference; Rubin’s causal model; sensitivity analysis;
observational studies
Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013. What would it take to Change an
Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Education,
Evaluation and Policy Analysis. Vol 35: 437-460.
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
http://epa.sagepub.com/content/early/recent
overview
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Estimated Effect
9
8
7
6
5
}
4
3
% bias
necessary
to
invalidate
the
inference
above threshold
{
below threshold
Threshold
2
1
0
A
overview
B
Study Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Quantifying the Discourse: Formalizing
Bias Necessary to Invalidate an Inference
δ =a population effect,
ˆ =the estimated effect, and
δ# =the threshold for making an inference
An inference is invalid if:
ˆ   # 
(1)
An inference is invalid if the estimate is greater than the
threshold while the population value is less than the threshold.
Defining bias as ˆ -δ, (1) implies an estimate is invalid if and
only if:
bias (ˆ)  ˆ   #
(2)
Correlational Framework
Expressed as a proportion of the estimate,
invalid
if:
Impact ofinference
a Confounding
variable
Thresholds
for inference and % bias to invalidate
Internal validity
#
#
The counterfactual paradigm
Replacement
(ˆ Cases
 ) Framework

Example: Effect of kindergarten retention
ˆ
Internal validity example: kindergarten retention
% bias ( ) 
 1
Correlational Framework
ˆ
External
validity
validity
exampleOpen Court curriculum
ˆ External
overview Conclusion 
example:of
effect
of Open Court curriculum
Extensions
the framework
inference invalid if
ˆ   #  ,
subtract ˆ
ˆ  ˆ   #  ˆ   ˆ
 0   #  ˆ   ˆ
multiply by -1
 0  ˆ   #  ˆ  
substitute bias=ˆ  
 0  ˆ   #  bias
 bias >ˆ   # > 0
as % of estimate, divide by ˆ
bias ˆ   #
>
>0
ˆ
ˆ


Correlational Framework
Impact of a Confounding variable
#
Thresholds
for inference and % bias to invalidate
 %bias >1 
>0
Internal validity
ˆ
 Cases Framework The counterfactual paradigm
Replacement
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Estimated Effect
9
8
7
6
5
ˆ
}
4
3
% bias
necessary
to
invalidate
the
inference
ˆ
above threshold
below threshold
{
δ#
Threshold
2
1
0
A
% bias (ˆ) to
overview
B
Study Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
# Internal validity
(ˆ   # )
 The
4
1 paradigm
Replacement Cases Framework
#
invalidate=
1
Example:
1counterfactual
 Effect
 33%
ˆ retention
ˆ
of kindergarten
retention
scale


by

Internal
validity
example:
kindergarten
ˆ
ˆ
6 validity
3

 External
Correlational Framework
External validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Robustness and Replicability
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Original study effect size versus replication effect size (correlation coefficients).
data
https://osf.io/ezcuj/wiki/home/
Open Science Collaboration Science 2015;349:aac4716
Published by AAAS
Interpretation of % Bias to
Invalidate an Inference
% Bias is intuitive
Relates to how we think about
statistical significance
Better than “highly significant” or “barely
significant”
But need a framework for interpreting
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Framework for Interpreting % Bias to
Invalidate an Inference: Rubin’s Causal
Model and the Counterfactual
1) I have a headache
2) I take an aspirin (treatment)
3) My headache goes away (outcome)
Q) Is it because I took the aspirin?
A) We’ll never know – it is counterfactual – for the
Correlational Framework
individual
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
This is theReplacement
Fundamental
Problem
of Causal
The counterfactual
paradigm
Cases Framework
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational
Framework
External
validity
Inference
External
validity
exampleOpen Court curriculum
overview
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Definition of Replacement Cases as Counterfactual:
Potential Outcomes
t
c
Definition of treatment effect for individual i:  i  Yi  Yi
Yi t  value on outcome if unit received treatment
Yi c  value on outcome if unit received control
Fundamental problem of causal inference is
that we cannot simultaneously observe
t
Yi and Yi
overview
c
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Fundamental Problem of Inference and
Approximating the Counterfactual with
Observed Data (Internal Validity)
9
10
11
3
4
5
overview
6?
6?
6?
But how well does
the observed data
approximate the
counterfactual?
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Symbolic: Fundamental Problem of
Inference and Approximating the
Counterfactual with Observed Data (Internal
Validity)
overview
Yt|X=t
Yc|X=t
Yt|X=c
Yc|X=c
6?
6?
6?
But how well does
the observed data
approximate the
counterfactual?
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Approximating the Counterfactual
with Observed Data But how well does
the observed data
approximate the
counterfactual?
Difference
between
8
1
counterfactual
9
1
10
1
values and
3
observed values
4
for the control
5
implies the
6
9
treatment effect
of 1
is overestimated
Correlational Framework
as 6 using
Impact of a Confounding variable
observed control
Thresholds
for inference and
% biaswith
to invalidate
Internal validity
cases
mean
The
counterfactual
paradigm
Replacement Cases Framework Example: Effect of kindergarten
of 4 retention
Internal validity example: kindergarten
retention
overview
Correlational Framework
Conclusion
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Using the Counterfactual to Interpret %
Bias to Invalidate the Inference
9
10
11
0
6
6
0
6
0
3
4
5
6.00
4
6
How many cases
would you have to
replace with zero
effect counterfactuals
to change the
inference?
Assume threshold is
4 (δ# =4):
1- δ# / ˆ
=1-4/6=.33 =(1/3)
overview
Correlational Framework
The inference would be invalid
if you
replaced variable
33% (or 1 case)
Impact of
a Confounding
for was
inference
and % bias effect.
to invalidate
with counterfactuals for Thresholds
which
there
no treatment
Internal
validity
The counterfactual
paradigmeffect)=
ˆ +%replaced(no
New estimate=(1-%
replaced)
Replacement
Cases Framework

Example:
Effect of kindergarten
retention
Internal
validity
example:
kindergarten
retention
ˆ
(1-%replaced)
 =(1-.33)6=.66(6)=4
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Which Cases to Replace
• Think of it as an expectation: if you
randomly replaced 1 case, and
repeated 1,000 times, on average
the new estimate would be 4
• Assumes constant treatment effect
•
conditioning on covariates and
interactions already Correlational
in the Framework
model
Impact of a Confounding variable
• Assumes cases carryThresholds
equal
weight
for inference
and % bias to invalidate
Internal validity
overview
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Estimated Effect
9
8
7
6
5
ˆ
}
4
3
% bias
necessary
to
invalidate
the
inference
ˆ
above threshold
below threshold
{
δ#
Threshold
2
1
0
A
% bias (ˆ) to
overview
B
Study Correlational Framework
Impact of a Confounding variable
To invalidate
Thresholds
for inference and
% bias to the
invalidate
# Internal validity
(ˆ   # )
 The
4
1
inference, replace 33% of
paradigm
Replacement Cases Framework
invalidate=
1
Example:
1counterfactual
 Effect
 33%
of kindergarten retention
cases with counterfactual
validity
retention
6 validity
3 example: kindergarten
ˆ
ˆInternal
Correlational Framework
External
data with
zero
effect
External validity exampleOpen
Court
curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Fundamental Problem of Inference and
Approximating the Unsampled Population
with Observed Data (External Validity)
9
10 Yt|Z=p
11
3
4 Yc|Z=p
5
overview
How many cases
would you have to
replace with cases
with zero effect to
change the
inference?
Assume threshold is:
δ# =4:
1- δ# / ˆ
6
6
6 Yt|Z=p´
=1-4/6=.33 =(1/3)
6
6 Yc|Z=p´
Correlational Framework
6
Impact of a Confounding variable
for
6
4Thresholds
0 inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Estimated Effect
9
8
7
6
5
ˆ
}
4
3
% bias
necessary
to
invalidate
the
inference
ˆ
above threshold
below threshold
{
δ#
Threshold
2
1
0
A
% bias (ˆ) to
overview
B
Study Correlational Framework
Impact of a Confounding variable
To and
invalidate
thetoinference,
Thresholdsvalidity
for inference
% bias
invalidate
(ˆ   # )
 # TheInternal
4
1
replace 33% of cases with
paradigm
invalidate=
1
 1Example:
counterfactual
 Effect
33%of
Replacement Cases Framework
kindergarten
retention
cases
from unsampled
ˆ
ˆ Internal
kindergarten
retention
6 validity
3 example:


Correlational Framework
population
with
zero
effect
External
validity
External validity exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Review & Reflection
Review of Framework
Pragmatism  thresholds
How much does an estimate exceed the threshold
 % bias to invalidate the inference
Interpretation: Rubin’s causal model
• internal validity: % bias to invalidate 
number of cases that must be replaced with
counterfactual cases (for which there is no
effect)
• external validity: % bias to invalidate 
number of cases that must be replaced with
unobserved population (for which there is no
effect)
Correlational Framework
Reflect
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
Which part is most confusing to you?
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Is there more than one interpretation?
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview
Discuss Conclusion
with a partner or two
example:of
effect
of Open Court curriculum
Extensions
the framework
Example of Internal Validity from Observational Study :
The Effect of Kindergarten Retention on Reading and Math
Achievement
(Hong and Raudenbush 2005)
1. What is the average effect of kindergarten retention policy? (Example
used here)
Should we expect to see a change in children’s average learning
outcomes if a school changes its retention policy?
Propensity based questions (not explored here)
2. What is the average impact of a school’s retention policy on children who
would be promoted if the policy were adopted?
Use principal stratification.
Correlational Framework
Hong, G. and Raudenbush, S. (2005). Effects
Retention
Impact of
of aKindergarten
Confounding variable
Thresholds
for inference
and
% bias to invalidate
Policy on Children’s Cognitive Growth
in Reading
and
Mathematics.
Internal
validity
The counterfactual paradigm
Replacement
Cases
Framework
Educational
Evaluation
and
Policy Analysis.
Vol.
27,ofNo.
3, pp. 205–224
Example:
Effect
kindergarten
retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Data
Early Childhood Longitudinal Study Kindergarten cohort
(ECLSK)
US National Center for Education Statistics (NCES).
Nationally representative
Kindergarten and 1st grade
observed Fall 1998, Spring 1998, Spring 1999
Student
background and educational experiences
Math and reading achievement (dependent variable)
experience in class
Parenting information and style
Teacher assessment of student
School conditions
Correlational Framework
Analytic sample (1,080 schools that do Impact
retainofsome
children)
a Confounding
variable
Thresholds
for inference and % bias to invalidate
471 kindergarten retainees
Internal validity
counterfactual paradigm
Replacement
Cases Framework The
10,255 promoted
students
Example: Effect of kindergarten retention
overview
Correlational Framework
Conclusion
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Effect of Retention on Reading Scores
(Hong and Raudenbush)
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Possible Confounding Variables
(note they controlled for these)
Gender
Two Parent Household
Poverty
Mother’s level of Education (especially relevant for
reading achievement)
Extensive pretests
measured in the Spring of 1999 (at the beginning
of the second year of school)
standardized measures of reading ability, math
ability, and general knowledge;
indirect assessments of literature, math and
general knowledge that include aspects of a
Correlational Framework
child’s process as well as product;
Impact of a Confounding variable
teacher’s rating of the child’sThresholds
skills
inforlanguage,
inference and % bias to invalidate
Internal validity
math, and
science
counterfactual paradigm
Replacement
Cases Framework The
Example: Effect of kindergarten retention
overview
Correlational Framework
Conclusion
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculating the % Bias to Invalidate the Inference:
Obtain spreadsheet
From https://www.msu.edu/~kenfrank/research.htm#causal
Choose spreadsheet for calculating indices
spreadsheet for calculating indices [KonFound-it!]
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement
Cases Framework The
Access spreadsheet
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Kon-Found-it: Basics
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Obtain t critical, estimated effect and standard error
n=7168+471=7639;
df > 500,
t critical=-1.96
Estimated effect
( ˆ) = -9.01
Standard
error=.68
From: Hong, G. and Raudenbush, S. (2005).
Effects of Kindergarten Retention Policy on
Children’s Cognitive Growth in Reading and
Framework
Mathematics.Correlational
Educational
Evaluation and Policy
Impact of a Confounding variable
Analysis. Vol. 27,
No. 3, pp.
205–224and % bias to invalidate
Thresholds
for inference
overview
Internal validity
The
counterfactual paradigm
Replacement Cases Framework Example:
Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
covariates
Page 215
Correlational Framework
Impact of a Confounding variable
Df=207+14+2=223
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Add 2 to include the intercept and retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Kon-Found-it: Basics
% Bias to Invalidate
Estimated effect
( ˆ) = -9.01
Standard
error=.68
n=7168+471=7639
Covariates=223
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
Specific calculations
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculating the % Bias to Invalidate the Inference:
Inside the Calculations
δ# =the threshold
for making an inference =
se x tcritical, df>230 =
.68 x -1.96=-1.33
[user can specify
alternative threshold]
}
overview
% Bias necessary to invalidate
inference
= 1-δ#/ ˆ
Correlational Framework
=1-1.33/-9.01=85%
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
85% ofparadigm
the estimate must be
counterfactual
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity
kindergarten
dueexample:
to bias to
invalidateretention
the
Correlational Framework
External
validity
External validity
exampleOpen Court curriculum
inference.
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Using the Counterfactual to Interpret %
Bias to Invalidate the Inference
9
10
11
0
6
6
0
6
0
3
4
5
6.00
4
6
How many cases
would you have to
replace with zero
effect counterfactuals
to change the
inference?
Assume threshold is
4 (δ# =4):
1- δ# / ˆ
=1-4/6=.33 =(1/3)
overview
Correlational Framework
The inference would be invalid
if you
replaced variable
33% (or 1 case)
Impact of
a Confounding
for was
inference
and % bias effect.
to invalidate
with counterfactuals for Thresholds
which
there
no treatment
Internal
validity
The counterfactual
paradigmeffect)=
ˆ +%replaced(no
New estimate=(1-%
replaced)
Replacement
Cases Framework

Example:
Effect of kindergarten
retention
Internal
validity
example:
kindergarten
retention
ˆ
(1-%replaced)
 =(1-.33)6=.66(6)=4
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Example Replacement of Cases with Counterfactual Data to Invalidate Inference of
an Effect of Kindergarten Retention
Correlational Framework
Counterfactual:
Impact of a Confounding
variable
Retained
Promoted
No effectThresholds for inference
and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal
example: kindergarten retention
Original cases that
were validity
not replaced
Correlational Framework
External
validity
External validity
Replacement counterfactual
casesexampleOpen
with zero effectCourt curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Original distribution
Interpretation
1) Consider test scores of a set of children who were retained that are considerably
lower (9 points) than others who were candidates for retention but who were in fact
promoted. No doubt some of the difference is due to advantages the comparable
others had before being promoted. But now to believe that retention did not have
an effect one must believe that 85% of those comparable others would have
enjoyed most (7.2) of their advantages whether or not they had been retained.
This is even after controlling for differences on pretests, mother’s education, etc.
2) The replacement cases would come from the counterfactual condition for the
observed outcomes. That is, 85% of the observed potential outcomes must be
unexchangeable with the unobserved counterfactual outcomes such that it is
necessary to replace those 85% with the counterfactual outcomes to make an
inference in this sample. Note that this replacement must occur even after
observed cases have been conditioned on background characteristics, school
membership, and pretests used to define comparable groups.
Correlational Framework
Impact of a Confounding variable
3) Could 85% of the children have manifest an
effect because
of unadjusted
Thresholds
for inference
and % bias to invalidate
Internal validity
differences (even
after
controlling
for
prior
achievement,
motivation
and
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
background) rather than retention itself?
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Evaluation of % Bias Necessary
to Invalidate Inference
Compare bias necessary to invalidate inference with bias accounted for
by background characteristics
1% of estimated effect accounted for by background characteristics
(including mother’s education), once controlling for pretests
More than 85 times more unmeasured bias necessary to invalidate
the inference
Compare with % bias necessary to invalidate inference in other studies
Use correlation metric
Correlational Framework
• Adjusts for differences in scale
overview
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
% Bias Necessary to Invalidate Inference based on Correlation
to Compare across Studies
r 
t
(n  q  1)  t 2

13.25
(7639  223  1)  (13.25) 2
 .152
t taken from HLM: =-9.01/.68=-13.25
n is the sample size
q is the number of parameters estimated
threshold= r 
#
t critical
2
(n  q  1)  t critical

1.96
(7639  223  1)  1.962
 .023
Where t is critical value for df>200
% bias to invalidate inference=1-.023/.152=85%
overview
Correlational Framework
Impact of a Confounding variable
Accounts
for changes in regression
Thresholds
for inference and % bias to invalidate
Internal validity
coefficient and
standard error
counterfactual
paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Because
t(r)=t(β)kindergarten retention
Internal
validity example:
Correlational Framework
External
validity
External validity exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Estimated Effect
9
8
7
r
6
5
}
4
3
% bias
necessary
to
invalidate
the
inference
r
above threshold
below threshold
{
r#
Threshold
2
1
0
A
B
Study Correlational Framework
Impact of a Confounding variable
Thresholdsvalidity
for inference and % bias to invalidate
(r  r # )
r #TheInternal
4
1
paradigm
#
% bias (r ) to Replacement
invalidate=Cases Framework
 1  Example:
1counterfactual
  Effect
 33%
of kindergarten
retention
scale
r

r
by r
Internal 6
validity
example: kindergarten retention
r
r
3
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Kon-Found-it: Basics
% Bias to Invalidate
Estimated effect
( ˆ) = -9.01
Standard
error=.68
n=7168+471=7639
Covariates=223
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
Specific calculations
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
% Bias Necessary to Invalidate Inference based on Correlation
to Compare across Studies
t critical
threshold= r 
#
r 
2
(n  q  1)  t critical
t
(n  q  1)  t 2


1.96
(7639  223  1)  1.962
13.25
(7639  223  1)  (13.25) 2
 .023
 .152
% bias to invalidate inference=1-.023/.152=85%
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Compare with Bias other
Observational Studies
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
% Bias to Invalidate Inference for observational studies
on-line EEPA July 24-Nov 15 2012
Kindergarten
retention effect
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Exercise 1 : % Bias necessary to Invalidate an Inference
for Internal Validity
Take an example from an observational study in
your own data or an article
Calculate the % bias necessary to invalidate the
inference
Interpret the % bias in terms of sample
replacement
What are the possible sources of bias?
Would they all work in the same direction?
What happens if you change the
sample size
Correlational Framework
standard error
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
degrees of freedom
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Debate your
inference with a partner
Correlational Framework
External validity
overview
Conclusion
External validity exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Application to Randomized Experiment:
Effect of
Open Court Curriculum on Reading Achievement
Open Court “scripted” curriculum versus
business as usual
917 elementary students in 49 classrooms
Comparisons within grade and school
Outcome Measure: Terra Nova
comprehensive reading score
Correlational Framework
of a Confounding
variable
Borman, G. D., Dowling, N. M., and Schneck, C. (2008).Impact
A multi-site
cluster randomized
field trial of
Thresholds
for
inference
and
%
bias to invalidate
Open Court Reading. Educational Evaluation and Policy Internal
Analysis,validity
30(4), 389-407.
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
49
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Value of Randomization
Few differences between groups
But done at classroom level
Teachers might talk to each other
School level is expensive (Slavin, 2008)
Slavin, R. E. (2008). Perspectives on evidence-based research in
education-what works? Issues in synthesizing educational program
evaluations. Educational Researcher, 37, 5-14.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
n=27+22=49
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Differences between Open Court
and Business as Usual
Difference across grades: about 10
units
7.95 using statistical model
“statistically significant” unlikely
(probability < 5%) to have occurred by
chance alone if there were really no
differences in the population
Correlational Framework
Impact
of a Confounding
variable
But is the Inference about
Open
Court
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement
Cases Framework The
Example: Effect of kindergarten retention
valid in other
contexts?
Internal validity example: kindergarten retention
overview
Correlational Framework
Conclusion
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Obtaining # parameters estimated, t critical, estimated effect and
standard error
3 parameters
estimated,
Df=n of classrooms# of parameters
estimated=
49-3=46.
t critical = t.05, df=46=2.014
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
The counterfactual paradigm
Replacement Cases Framework
Example:
EffectStandard
of kindergarten
retention
Estimated
effect
error=1.83
Internal
validity example:
kindergarten
retention
Correlational Framework
validity
( ˆ)External
=External
7.95validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Quantifying the Discourse for Borman et al:
What would it take to change the inference?
δ =a population effect,
ˆ =the estimated effect = 7.95, and
δ # =the threshold for making an inference =
se x tcritical, df=46 =1.83 x 2.014=3.69
% Bias necessary to invalidate inference =
1- δ #/ˆ =1-3.69/7.95=54%
54% of the estimate must be due to bias to invalidate the
inference
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculating the % Bias to Invalidate the Inference:
Inside the Calculations
t critical= 2.014
standard error =1.83
ˆ =the estimated effect = 7.95 sample size=49
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculating the % Bias to Invalidate the Inference:
Inside the Calculations
δ# =the threshold for making an inference =
se x tcritical, df=46 =1.83 x 2.014=3.686
% Bias necessary to invalidate inference =
1-d#/d =1-3.686/7.95=54%
54% of the estimate must be due to bias to invalidate the inference.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
% Exceeding Threshold for Open
Court Estimated Effect
below threshold
9
ˆ  7.95
8
Estimated Effect
above threshold
}
7
6
5
4
54 % above threshold=1-3.68/7.95=.54
δ# =3.68
3
2
54% of the estimate
must be due to bias to
invalidate the inference
1
0
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
OCR
overview
Fundamental Problem of Inference and
Approximating the Unsampled Population
with Observed Data (External Validity)
9
10 Yt|Z=p
11
3
4 Yc|Z=p
5
overview
How many cases
would you have to
replace with cases
with zero effect to
change the
inference?
Assume threshold is:
δ# =4:
1- δ# / ˆ
6
6
6 Yt|Z=p´
=1-4/6=.33 =(1/3)
6
6 Yc|Z=p´
Correlational Framework
6
Impact of a Confounding variable
for
6
4Thresholds
0 inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Interpretation of Amount of Bias Necessary to
Invalidate the Inference: Sample
Representativeness
To invalidate the inference:
54% of the estimate must be due to sampling bias to
invalidate Borman et al.’s inference
You would have to replace 54% of Borman’s cases (about
30 classes) with cases in which Open Court had no effect
to invalidate the inference
Are 54% of Borman et al.’s cases irrelevant for nonvolunteer schools?
We have quantified the discourse about the concern of
external validity
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the
Open Court Curriculum
Business as Usual
overview
Unsampled Population:
No effect
Open Court
Correlational Framework
Original volunteer cases
that
not replacedvariable
Impact
ofwere
a Confounding
Thresholds
for inference
andwith
% bias
to invalidate
Replacement cases
from
non-volunteer
schools
no treatment
effect
Internal
validity
The
paradigm
Original
distribution
forcounterfactual
all volunteer cases
Replacement Cases
Framework
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
The Fundamental Problem of External
Validity
Before a randomized experiment:
People believe they do not “know” what generally works
People choose treatments based on idiosyncratic conditions -- what
they believe will work for them (Heckman, Urzua and Vytlacil, 2006)
After a randomized experiment:
People believe they know what generally works
People are more inclined to choose a treatment shown to generally
work in a study because they believe “it works”
The population is fundamentally changed by the experimenter (BenDavid; Kuhn)
The fundamental problem of external validity
Correlational
Framework
the more influential a study the more
different
the pre and post
Impact
of a post
Confounding
variable
populations, the less the results apply
to the
experimental
Thresholds
for inference and % bias to invalidate
Internal validity
population
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal
validity example:
kindergarten retention
All the more
so if it isFramework
due to the design
(Burtless,
1995)
Correlational
External validity
overview
Conclusion
External validity exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Sampson, R. J. (2010). Gold standard myths: Observations on the
experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500. page 495
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Sampson, R. J. (2010). Gold standard myths: Observations on the
experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500. page 496
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Comparisons across Randomized Experiments (correlation metric)
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Distribution of % Bias to Invalidate Inference for
Randomized Studies EEPA:
On-line Jul 24-Nov 5 2012
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Review & Reflection
Review of applications
Concern about internal validity: Kindergarten
retention (Hong and Raudenbush)
• 85% of cases must be replaced counterfactual data
(with no effect) to invalidate the inference of a negative
effect of retention on reading achievement
– Comparison with other observational studies
Concern about external validity: Open Court
Curriculum
• 54% of cases must be replaced with data from
unobserved population to invalidate the inference of a
positive effect of Open Court on reading achievement in
non-volunteer schools
– Comparison with other randomized experiments
Correlational Framework
Reflect
Impact of a Confounding variable
for inference and % bias to invalidate
Internal validity
Which part is most confusing to you? Thresholds
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Is there more than one interpretation? Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview
Discuss with aConclusion
partner or two
example:of
effect
of Open Court curriculum
Extensions
the framework
Exercise 2 : % Bias Necessary to Invalidate an Inference
for External Validity
Take an example of a randomized experiment in
your own data or an article
Calculate the % bias necessary to invalidate the
inference
Interpret the % bias in terms of sample
replacement
What are the possible sources of bias?
Would they all work in the same direction?
What happens if you change the
sample size
Correlational Framework
standard error
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
degrees of freedom
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Debate
your inference with a new partner
Extensions of the Framework
Ordered thresholds for decisionmaking
Alternative hypotheses and scenarios
Relationship to confidence intervals
Related techniques
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Ordered Thresholds Relative to Transaction Costs
Definition of threshold: the point at which evidence from a study
would make one indifferent to policy choices
1. Changing beliefs, without a corresponding
change in action.
2. Changing action for an individual (or family)
3. Increasing investments in an existing program.
4. Initial investment in a pilot program where none
exists.
Correlational Framework
5. Dismantling an existing program
and replacing
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
it with Replacement
a new program.
Cases Framework The counterfactual paradigm
overview
Correlational Framework
Conclusion
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
How District Leaders Really Make Decisions, and About What
from Design-Based Implementation Research: A Strategic Approach to Scaling
and Sustaining Educational Innovation: William R. Penuel
University of Colorado
Philip Bell
University of Washington
Use of Research Evidence
Improve PD
4.17
Improve existing programs
3.97
To select standards focus
3.93
Find concepts to inform improvement efforts
3.90
Improve district policy implementation
3.90
Challenge an adopted program
3.83
Mobilize support for adopted program
3.76
Choose between programs
3.72
Justify an already-made decision
3.62
1
2
3
4
5
Sampson, R. J. (2010). Gold standard myths: Observations on the
experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500.
Tolerance for Bias and Return on
Investment (high stakes counter
intuitive)
Low Return
Medium Return High
(charter schools) (KindergartenRe (New drug for
tention)
bad disease)
Low allocation
20-35%
5-20%
0-5%
Medium
Allocation
35-50%
20-30%
5-10%
High allocation
50-65%
30-40%
10-15%
New feature
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative Hypotheses: Non
Zero Null Hypothesis
Non-zero null hypotheses (for kindergarten retention)
H0:δ> −6.
se x tcritical, df=7639=.68 x (−1.645)= −1.12 (one tailed test).
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative Hypotheses: Non
Zero Null Hypothesis
δ# = −6−1.12=−7.12
1− δ #/ˆ =1− (−7.12/−9)=.21.
21% of estimated effect would have to be due to bias to invalidate
inference for H0:δ> −6.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative Scenario: Failure to
Reject Null when null is False
(type II error)
Assume the data are comprised of two
subsamples, one with an effect at the
r
threshold (r#) and the other (
) with 0
effect, and define  as the proportion of
the sample that has 0 effect.
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example:
Effect of kindergarten retention
#
Internal validity example: kindergarten retention
Correlational Framework
External
validity
xy
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
rxy =(1-)r +
overview
r
 .
Sample Replacement: Estimate Does
not Exceed the Threshold
r
Set xy=0 and solve for :
=1- rxy /r#
If  of the cases have 0 effect then if you replace
them (with cases at the threshold) then the overall
estimate will be at the threshold.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies:
Reevaluating the evaluation of training programs." Journal of the American statistical
Association 94, no. 448 (1999): 1053-1062.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative Scenario: Alternative Thresholds for
Inference
Use δ# = −4
To Invalidate the inference, 56% of the
estimate would have to be
due to bias if the
Correlational Framework
threshold for inference is -4.
Impact of a Confounding variable
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative Scenario: Non-zero Effect in the
Replacement (e.g., non-volunteer) Population
Inference invalid if 1-πp<(δp − δ#)/(δ p − δ p´ ).
If δ p´ = −2, and δ#=3.68 and δ p =7.95 (both
as in the initial example for Open Court).
Inference is invalid if 1-πp<(7.95 – 3.68)/(7.95
− −2 ) =.43
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
inference invalid if more than 43% of the
sample were replaced with cases for
which the effect of OCR was −2 .
overview
Exercise: Different Scenarios
Using an inference you have identified
or one of the ones I presented, use the
spreadsheet to evaluate:
1) Alternative hypotheses
2) Type II errors
3) Alternative thresholds
Correlational Framework
4) Non-zero effects in replacement
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
cases
Internal validity
The counterfactual paradigm
Replacement Cases Framework
overview
Correlational Framework
Conclusion
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Applications of % Bias to
Invalidate
Carrico, A. R., Vandenbergh, M. P., Stern, P. C., & Dietz, T. (2015). US
climate policy needs behavioural science. Nature Climate Change, 5(3), 177179.
Frank, K. A., Penuel, W. R., & Krause, A. (2015). What Is A “Good” Social
Network For Policy Implementation? The Flow Of Know‐How For
Organizational Change. Journal of Policy Analysis and Management, 34(2),
378-402.
Dietz, T. (2015). Altruism, self-interest, and energy consumption. Proceedings
of the National Academy of Sciences, 112(6), 1654-1655.
Beland, L. P., & Kim, D. (2014). The effect of high school shootings on
schools and student performance (No. 2015-05).
Asensio, O. I., & Delmas, M. A. (2015). Nonprice incentives and energy
conservation. Proceedings of the National Academy of Sciences, 112(6),
E510-E515.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Out 6-15-15
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Relationship between the Confidence Interval and % Bias Necessary to Invalidate
the Inference of an Effect of Open Court on Comprehensive Reading Score
Lower bound of confidence interval “far from 0”
}
δ#
ˆ
0 1 2 3 4 5 6 7 8 9 1 1 1 1
0 1 2 3
}
Confidence
Interval
estimate exceeds thre
by large amount
0 1 2 3 4 5 6 7 8 9 1 1 1 1
ˆ
0 1 2 Correlational
3
Framework
overview
Impact of a Confounding variable #
δ to invalidate
Thresholds
for inference and % bias
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Related Techniques
Bounding (e.g., Altonji et, Elder & Tabor, 2005; Imbens 2003; Manski)
lower bound: “if unobserved factors are as strong as observed factors, how small
could the estimate be?”
• Focus on estimate
% robustness: “how strong would unobserved factors have to be to invalidate
inference?”
• Focus on inference, policy & behavior
External validity based on propensity to be in a study (Hedges and O’Muircheartaigh )
They focus on estimate
We focus on comparison with a threshold
Other sensitivity (e.g., Rosenbaum or Robins)
Characteristics of variables needed to change inference
We focus on how sample must change.
• Can be applied to observational study or RCT
Other Sources of Bias
Violations of SUTVA
Correlational Framework
• Agent based models?
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Measurement error
Internal validity
The counterfactual
paradigm
• Just Replacement
another source
Cases
of bias
Framework
(minor concern
for examples
here)
Example:
Effect of kindergarten
retention
Internal validity example: kindergarten retention
Differential treatment
effects
Correlational
Framework
External
validity
External
validity
exampleOpen Court curriculum
overview
• Use Conclusion
propensity scores to differentiate,
then apply
indices
example:
effect
of Open Court curriculum
Extensions
of
the
framework
New directions
•
Non-linear (e.g., logistic) or multilevel regression
• Inference made from estimate/standard error
(not deviance)
• Can be thought of as weighted least squares,
assumes weights not related to replacement
of cases
• Propensity score matching
• How many bad matches would you have to
have to invalidate inference (Rubin liked this
idea)
Correlational Framework
Impact
of a Confounding
variable
• Value added: for a teacher orThresholds
school
rated
as
for inference and % bias to invalidate
Internal validity
ineffective,
how many
students
The counterfactual
would you
paradigm
have
Replacement
Cases Framework
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
to replace
with average
tovalidity
achieve
an Court curriculum
Correlational
Framework students
External
validity
External
exampleOpen
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
effective rating?
II: Correlation Framework
Concerns about internal validity
At what levels must an omitted variable be correlated with a predictor and
outcome to change an inference about the predictor?
Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a
Regression Coefficient." Sociological Methods and Research, 29(2), 147194
Concerns about external validity
What must be the correlation between predictor and outcome in the
unsampled population to invalidate an inference about an effect in a
population that includes the unsampled population?
*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample
Representation. Sociological Methodology. Vol 37, 349-392. * co first
authors.
Combined application (with propensity
scores) Framework
Correlational
Impact of a Confounding
Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos,
Marisa variable
Cannata, Linda
Thresholds
for
inference
and
bias to invalidate
Internal
validity Influence:%National
Chard, Ann Krause, Raven McCrory. 2008.
Extended
Board
The
counterfactual
paradigm
Replacement Cases Framework Example: Effect of kindergarten retention
Certified Teachers
as Help Providers. Internal
Education,
Evaluation,
and Policy
validity
example: kindergarten
retention
Correlational
Framework
External
validity
Analysis. Vol 30(1): 3-30.
External validity exampleOpen Court curriculum
overview
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Making an Inference from an
Observational Study
• Gather data on people as they naturally choose
“treatments”
• Measure covariates, especially pretests
• Randomly sample from a larger population – can
do because not assigning to treatments
• Examples
• Coleman et al., Catholic schools effect
• Hong and Raudenbush: effects of kindergarten
Correlational Framework
retention
Impact of a Confounding variable
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
93
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Adjusting for Confound to get Correct Effect
Replacing observed cases with linear counterfactuals
6
6
6
To replace observed control cases to get estimated effect of 4
instead of uncontrolled mean difference
of 6, use
confound: control group
Correlational
Framework
disadvantaged from the start
Impact of a Confounding variable
fordo
inference
and %
bias to
invalidate
But how
we obtain
these
estimates?
Internal
validity
y     s   confound Thresholds
0
y
overview
1
2
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
validity example: kindergarten retention
6Correlational
 4 x s  1Framework
x confound Internal
External
validity
External validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Control for Confound ↔Adjusting the Outcome
Replacing observed cases with linear counterfactuals
6
6
6
y   0  1s   2 confound
y  6  4 x s  1 x confound
example: 3=6  4 x 0  1 x -3
3=6-3
add 3 to each side
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
 3+3=6  3  3
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
control for conound
 adjust the outcome
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Regression!
SAS Syntax for reading in toy counterfactual data
DATA one;
input y s confound;
cards;
9 10
10 1 0
11 1 0
3 0 -3
4 0 -2
5 0 -1
run;
Biased estimated
proc reg data=one;
model y=s ;
run;
proc reg data=one;
model y=s confound;
run;
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
y   0  The
1Example:
s counterfactual
  2confound
paradigm
Replacement Cases Framework
Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational
Framework
y

6

4
xExternal
s  1validity
xvalidity
confound
Unbiased estimate
External
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Regression!: SPSS Syntax for reading in toy counterfactual
DATA LIST FREE / y s confound.
data
Begin DATA .
9 10
10 1 0
11 1 0
3 0 -3
4 0 -2
5 0 -1
End DATA .
Biased estimated
REGRESSION
/MISSING LISTWISE
/STATISTICS COEFF OUTS R ANOVA
/CRITERIA=PIN(.05) POUT(.10)
/NOORIGIN
/DEPENDENT y
/METHOD=ENTER s.
REGRESSION
/MISSING LISTWISE
/STATISTICS COEFF OUTS R ANOVA
/CRITERIA=PIN(.05) POUT(.10)
/NOORIGIN
/DEPENDENT y
/METHOD=ENTER s confound.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
y   0  The
1Example:
s counterfactual
  2confound
paradigm
Replacement Cases Framework
Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational
Framework
y

6

4
xExternal
s  1validity
xvalidity
confound
Unbiased estimate
External
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
How Regression Works:
Partial Correlation
Partial Correlation: correlation between s and y,
where s and y have been controlled for the confounding variable
rs·y|cv 
rs·y  rs·cv  ry·cv
1 r
2
y·cv
1 r
2
s·cv

.96490  .86603  .92848
1  .86603
2
1  .92848
2
 .86601
sd ( y | cv)
1.265
ˆ
1  rs·y|cv
 .86601
4
sd (s | cv)
.274
Correlational Framework
sd ( y | cv)  sd ( y ) 1  ry2c  3.406 1  .928482 Impact
1.265 of a Confounding variable
Thresholds
for inference
and % bias Sas
to invalidate
Spss syntax
syntaxt
Internal validity
2
2
counterfactual paradigm
sd (s | cv)  Replacement
sd (s) 1  rsc Cases
.547 1Framework
 .86603 The
.274
proc corr
data=one;
CORRELATIONS
Example: Effect
of kindergarten
retention
Internal validity example: kindergarten retention
var yss confound
confound;
/VARIABLES=y
Correlational Framework
External
validity
External
validity
exampleOpen
Court
curriculum
run; NOSIG
/PRINT=TWOTAIL
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
How Regression Works:
Partial Correlation
Partial Correlation: correlation between s and y,
where s and y have been controlled for the confounding variable
rs·y|cv 
rs·y  rs·cv  ry·cv
1 r
2
y·cv
overview
1 r
2
s·cv

.31483  .19170  .76147
1  .76147
2
1  .19170
2
 .26536
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference
and % bias Sas
to invalidate
Spss syntax
syntaxt
Internal validity
counterfactual paradigm
Replacement Cases Framework The
proc corr
data=one;
CORRELATIONS
Example: Effect
of kindergarten
retention
Internal validity example: kindergarten retention
var yss confound
confound;
/VARIABLES=y
Correlational Framework
External
validity
External
validity
exampleOpen
Court
curriculum
run; NOSIG
/PRINT=TWOTAIL
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculation of a Partial
Correlation in KonFound-it!
rs·y|cv 
rs·y  rs·cv  ry·cv
1  ry2·cv 1  rs·2cv
overview

.31483  .19170  .76147
1  .76147 2 1  .191702
 .26536
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Derivation of Partial from Matrix Algebra
(Normal Equations)
n
ˆ1 
 ( x  x)  ( y
i 1
i
i
 y)
n
 ( xi  x)
2
i 1
Multivariate: Β=[X’X]-1X’Y
1 rx1x 2 
X 'X  
,

 rx1x 2 1 
for
n

 ( x  x)  ( y
i 1
i
n
i
 y) / n
2
(
x

x
)
/n
 i

covariance of x and y
variance of x
i 1
For 2 variables that are standardized:
rx1x 2 
 1
1  r 2 1  r 2 
 rx1 y 
x1 x 2
x1 x 2
1


X 'X 
, X'Y=  
 rx1x 2

1
 rx 2 y 


2
1

r
1

r

x1 x 2 
x1 x 2

Correlational Framework
2
sd
(
y
)
1

r
r

r
r
r

r
r
r

r

r
Impact
of
a
Confounding
variable
y  cv
 y s  cv
s·y
s·cv
y ·cv
1 , X ' X 1X'Y  x1 y x 22y x1x 2  s  y cvThresholds

for
inference
and
%
bias
to
invalidate
2 Internal validity 2
2
2
1

r
1

r
The
counterfactual
paradigm
1

r
1

r
sd
(s)
1

r
ReplacementxCases
Framework scv Example: Effecty·of
1x 2
cv kindergarten
s·cv
retention s cv
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Defining the Impact of a Confounding Variable
rs·y  rs·cv  ry·cv sd ( y )
1 
2
1  rs·cv
sd (s)
The "impact" is the product:
impact (of cv on rs·y )  rs·cv  ry·cv
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
s·y |cv
2 TheExample:
2 paradigm
counterfactual
Replacement Cases Framework
Effect of kindergarten retention
Internal validity example: kindergarten retention
s·cv
Correlational Framework y ·cv External
External
validity
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
r
overview
rs·y  rs·cv  ry·cv

1 r
1 r
Regression Works: of Teaching
The Impact of aHow
Enhancement
Impact of a Confounding Variable on a Regression Coefficient
through leadership on Correlation Between
rs∙y=.96490
Board Certification
and Help Provided
S
Board
Certification
s
rscv=.17
rcv∙s=.86603
rrsc∙scvxr×r
c∙yy
cv
.866x.928
CV
cv
Enhancement of
Impact weights
the relationship
Y
y
Help between cv and y
r
s
y=.86601
rsy|cv
=.18
Providedby the
relationship
between c and s:
the stronger the
relationship
between cv and
y, the more
rrycv =.07
important the
cv∙y=.92848
relationship
between cv and
Correlational Framework
s.
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
teaching through
Replacement
leadership Cases
overview
Alternative Conceptualization
(overlapping variance)
(rx2∙y|x1)2
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Everything old is new again
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Regression works…
As long as you have the right
confounding variables
Linear relationships assumed
Does this work in practice?
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Regression in Practice: Observational studies with
controls for pretests work better than you think
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate
answers? A randomized experiment comparing random to nonrandom assignment. Journal of the
American Statistical Association, 103(484), 1334-1344.
Quantify how much biased removed by statistical
control using pretests in a given setting
Sample: Volunteer undergraduates
Outcome: Math and vocabulary tests
Treatment:
• basic didactic,
• showing transparencies
• defining math concepts
Correlational Framework
OLS Regression with pretests removes 84% Impact
to 94%
of a Confounding variable
for
of bias relative to RCT!! Propensity by strataThresholds
not
quite
asinference
good and % bias to invalidate
Internal
validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity
example:
kindergarten
retention
See also Concato
et al.,
2000 for a comparable
example
in medical
research
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Supporting findings
Berk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433
Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a
true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results
from the two approaches are effectively identical’’ page 191.
Cook, T. D., P. Steiner, and S. Pohl. 2010. Assessing how bias reduction is influenced by covariate choice, unreliability
and data analytic mode: An analysis of different kinds of within-study comparisons in different substantive
domains. Multivariate Behavioral Research 44(6): 828-47.
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational
study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4), 463-479.
Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of
research designs. New England Journal of Medicine, 342(25), 1887-1889.
Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce
comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724–
750.
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A
randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association,
103(484), 1334-1344.
Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in
selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.
Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in
controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages 250-267. more than
just pretest.
Correlational Framework
Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER
Impact of a Confounding variable
working paper 14607.
Thresholds
for inference and % bias to invalidate
35. Kane, T., & Staiger, D. (2008).
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example:
Effect
of kindergarten
retentionof
Bifulco, Robert . "Can Nonexperimental Estimates Replicate Estimates
on
Random
Assignment
in Evaluations
InternalBased
validity
example:
kindergarten
retention
School Choice? A Within‐Study
Comparison."
Journal of Policy Analysis
and Management
31, no. 3 (2012): 729-751.
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
Reports
64% to 96% reduced with pre-test
overview
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Cook, T. D., P. Steiner, and S.
Pohl. 2010. Assessing how
bias reduction is influenced
by covariate choice,
unreliability and data
analytic mode: An analysis
of different kinds of withinstudy comparisons in
different substantive
domains. Multivariate
Behavioral Research 44(6):
828-47.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
New paper on Multiple
Approaches (from Tom Dietz)
http://arxiv.org/pdf/1412.3773v1.pdf
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Reflection
What part if most confusing to you?
Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and
solutions
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
When Regression Doesn’t Work
Instrumental variables?
• Alternative assumptions, may be not better
– Exclusion restriction
– Strong instrument
– Large n
Propensity scores?
No better than the covariates that go into it
• Heckman, 2005; Morgan & Harding, 2006, page
40; Rosenbaum, 2002, page 297; Shadish et al.,
2002, page 164);
• How could they be better than covariates?
– Propensity=f(covariates).
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
ROSENBAUM
Conclusion
example:of
effect
of OpenPAUL
Court curriculum
Extensions
the framework
Paul Holland says:
overview
Sensitivity Analysis:
What Must be the Impact of an Unmeasured
Confounding variable invalidate the
Inference?
Step 1: Establish Correlation Between
predictor of interest and outcome
Step 2: Define a Threshold for Inference
Step 3: Calculate the Threshold for the
Impact Necessary to Invalidate the Inference
Step 4: Multivariate Extension,
with other
Correlational Framework
Covariates
Impact of a Confounding variable
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Obtain t critical, estimated effect and standard error
bivariate model:
y   0  1s (or x)   2unobserved confound
reading achivement   0  1retention   2unobserved confound
n=7168+471=7639;
df > 500,
t critical=-1.96
overview
Estimated effect
( ˆ) = -9.01
Standard
error=.68
Correlational Framework
From: Hong, G. and
Raudenbush,
S. (2005).variable
Effects of
Impact
of a Confounding
Kindergarten Retention
Policy
Children’s
Thresholds
for on
inference
andCognitive
% bias to invalidate
Internal
validity
Growth in Reading
and
Mathematics.
Educational
The counterfactual paradigm
Replacement Cases
Framework
Example:
of No.
kindergarten
retention
Evaluation
and Policy
Analysis.Effect
Vol. 27,
3, pp. 205–224
Internal
validity
example:
kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Establish Correlation and Threshold for Kindergarten Retention on
Achievement
Step 1: calculate treatment effect as correlation
r 
t
(n  q  1)  t 2

13.25
(7639  2  1)  (13.25) 2
 .150
t taken from HLM: =-9.01/.68=-13.25
n is the sample size =7639
q is the number of parameters estimated ( predictor+omitted variable=2)
Step 2: calculate threshold for inference (e.g., statistical significance)
threshold= r 
#
t critical
2
(n  q  1)  t critical

1.96
(7639  2  1)  1.962
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Where t is critical value for df>200
r# can also be defined in terms of effect sizes
overview
 .022
Step 3a: Calculate the Threshold for the
Impact Necessary to Invalidate the Inference
y   0  1 x   2 confound (cv)
Assume rx∙cv =ry∙cv (which maximizes the impact of the confounding
variable – Frank, 2000). Then impact= rx∙cv x ry∙cv = rx∙cv x rx∙cv = ry∙cv
x ry∙cv , and
rx·y|cv 
rx·y  rx·cv  ry·cv
1 r
2
y·cv
1 r
2
x·cv

rx·y  impact
1  impact
Set rx∙y|cv =r# and solve for k to find the threshold for the impact
of a confounding variable (TICV). Sometimes I say ITCV.
rx·y  r # TICV  .150  .022  .130
TICV 
1 | 022 |
1 | r # |
Magnitude of
Magnitude of
overview
Magnitude= .130
Correlational Framework
impact of an unmeasured Impact
confound
> .130 →variable
inference invalid
of a Confounding
for inference
andinference
% bias to invalidate
impact of an unmeasuredThresholds
confound
< .130 →
valid.
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Maximizing impact
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Step 3b: Component correlations
.150  .022
TICV 
 .130
1 | 022 |
Magnitude= .130
If rx∙cv = ry∙cv =r, then impact= rx∙cv x ry∙cv =r2 .
Component correlations for threshold
r 2  .130  r= .130  .361
The magnitude of each correlation (rx∙cv Correlational
, ry∙cv ) mustFramework
be greater than .36 to
change inference. The correlations must have
opposite
signs. variable
Impact
of a Confounding
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Calculating the % Bias to Invalidate the Inference:
Obtain spreadsheet
From https://www.msu.edu/~kenfrank/research.htm#causal
Choose spreadsheet for calculating indices
spreadsheet for calculating indices [KonFound-it!]
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement
Cases Framework The
Access spreadsheet
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
KonFound-it! Spreadsheet
User enters values in yellow
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside the Calculations
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside the Calculations
Step 1: calculate treatment effect as correlation
r 
t
(n  q  1)  t 2

13.25
(7639  3)  (13.25) 2
 .150
Step 2: calculate threshold for inference (e.g., statistical significance)
threshold= r 
#
overview
t critical
(n  q  1)  t

1.96
(7639  3)  1.96
2 Correlational Framework
2
critical Impact of a Confounding variable
 .022
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside the Calculations: Step 3a
TICV 
rx·y  r
#
1 | r # |
Step 3b
overview
.150  .022
TICV 
 .130
1 | 022 |
r 2  .130  r= .130  .361
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Estimated Effect
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
.99
.88
.77
.66
.55
.44
.33
.22
.11
00
r
}
% bias
necessary
to
invalidate
the
inference
r
above threshold
below threshold
{
r#
A
Threshold
B
Study Correlational Framework
Impact of a Confounding variable
Thresholdsvalidity
for inference and % bias to invalidate
(r  r # )
r #TheInternal
4
1
paradigm
#
% bias (r ) to Replacement
invalidate=Cases Framework
 1  Example:
1counterfactual
  Effect
 33%
of kindergarten
retention
scale
r

r
by r
Internal 6
validity
example: kindergarten retention
r
r
3
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Estimated Effect
Figure 1
Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
9
.9
.8
8
.7
7
.6
6
.5
5
.4
4
.3
3
.2
2
.1
1
00
scale r  r # by 1-r #
above threshold
r
below threshold
r#
r
Threshold
r#
A
B Framework
Correlational
Study
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
#
Replacement
Cases Framework The
Example:
Effect of kindergarten retention
(r  r ) (.4  .2)
(.8  .6)
Internal
validity example: kindergarten retention
scale by =Correlational
 Framework .25 orExternal
 .50
External
validity
#
validity
exampleOpen Court curriculum
1 r
1  .2
1  .6
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Transforming % Bias to Replace to ITCV Hmmmm….
(r  r # )
ITCV  scale (r  r ) by 1  r :
1 r#
(r  r # )
#
%bias to replace  scale (r  r ) by r:
r
r
(r  r # )
r
(r  r # )
So %bias to replace x

x

 ITCV
#
#
#
1 r
r
1 r
1 r
r
#
 %bias to replace =ITCV when

1

r

r
1
#
1 r
Toy data: r=.6, r #  .4, r+r #  .6  .4  1, so % bias =ITCV:
#
#
(r  r # ) (.6  .4)
%bias to replace=

 .33
r
.6
r
(.6  .4) .6
(.6  .4) ( r  r # )
.2
%bias to replace x

x Correlational


  .33  ITCV
Framework
1 r#
.6
1  .4
1  .4
1 r#
.6
overview
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Reflection
What part if most confusing to you?
Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and
solutions
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Multivariate Extension
Z (pretest,
mother’s education)
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside the Calculations: Multivariate
y   0  1s   2 Z  3unobserved confound
reading achivement   0  1retention   2 ( pretest , mothered )   2unobserved confound
Step 1M: calculate treatment effect as correlation with DF correction
r 
t
(n  q  1)  t 2

13.25
(7639  3  223)  (13.25) 2
 .152
DF correction
Step 2M: calculate threshold for inference
(e.g.,
statistical significance)
Correlational
Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
critical
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten
2
2 retention
Correlational Framework
External
validity
External validity exampleOpen Court curriculum
critical
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
threshold= r 
#
overview
t
(n  q  1)  t

1.96
(7639  3  223)  1.96
 .023
Step 3M: Multivariate Extension,
with Covariates
k=rx ∙cv|z× ry ∙ cv|z
Following bivariate approach: maximizing the impact with covariates z in the
model implies
r  r#
ITCV | z 
x·y | z
|z
1 | r| #z |
and rx·cv| z =ry·cv| z = ITCV | z
And components
ry·cv  ry·cv| z
rx·cv  rx·cv| z
overview
1  R 1  R  
1  R 1  R  
2
y·z
2
cv·z
2
Ry2·z Rcv·
z
2
x·z
2
2
2
Correlational
Framework
cv·z
x·z cv·z
R R
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Assumption: Omitted Confound
Independent of Observed
Covariates
2
cv·z
R
 0 by assumption (can be altered in spreadsheet)
Gives maximal credence to skeptic challenging inference
ry·cv  ry·cv| z
rx·cv  rx·cv| z
overview



1  R 1  R  
2
2
2
1  Ry2·z 1  Rcv·

R
R
z
y·z cv·z  ry·cv | z
2
x·z
2
cv·z
2
Rx·2 z Rcv·
z  rx·cv | z


1  R 
1  Ry2·z
2
x·z
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Step 3M: Multivariate Extension, with Covariates
Following bivariate approach: maximizing the impact with covariates z in the model implie
rx·y | z  r| #z
.  .152  .023
ITCV | z 

 .132,
#
1 | r| z |
1 | .023 |
and rx·cv| z =ry·cv| z = ITCV | z =. .132=.364
ry·cv  ry·cv| z
And
rx·cv  rx·cv| z
1  R   .364 1  .201  .325
1  R   .364 1  .275  .310
2
y·z
2
x·z
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
2
Internal validity
Rcv·

0
by
assumption
(can
be
altered
in
spreadsheet)
counterfactual paradigm
z
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Obtaining R2 YZ
:
2
YX |Z
r
2
YZ
R


overview
2
Y .X ,Z
R
R
2
YZ
1 r
2
YZ
R
2
YX |Z
2
YX |Z
r
r
2
Y .X ,Z
1
R 
2
YZ
R
2
YX |Z
2
YX |Z
r
r
2
Y .X ,Z
1
.152  .22

 .201
2
.152  1
2
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside Calculations: Obtaining R2 YZ
:
2
YZ
R

R
2
YX |Z
2
YX |Z
r
overview
r
2
Y .X ,Z
1
.152  .22

 .201
2
.152  1
2
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Inside Calculations: Obtaining R2
:
xZ
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Extra Sufficient Statistics for Obtaining R2xZ
Already have Se(β1)
Page 210
std(y),
Page 208
% retained= 471/7639=.061
Var(x)=.061*.94=.057, std(x)=.24
Residual variance=55+88=143
Initial variance=13.532=183
R2 =1-143/183=.22
Correlational
Framework
page
217
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal
or validity example: kindergarten retention
number of covariates
Correlational Framework
External
validity
External validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
page 216:
Multivariate Calculations in Spreadsheet
overview
User enters
Std(y)
Std(x)
R2
# of covar
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Obtaining R2 xZ
:
2
RXZ
1
ˆ y2 1  RY2. X , Z 

ˆ x2 df Se(ˆ1 )
overview

2
1
13.532 1  .22 
.24 (7639  241) .68 
2
2
 .275
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
What must be the Impact of an Unmeasured
Confound to Invalidate the Inference?
If |impact of unobserved variable| > .132 (or .130
without covariates) then the inference is invalid
If |r x cv|z |= |ry cv|z|, then each would have to be greater
than impact of unobserved variable 1/2 =.36 to
invalidate the inference.
*Because effect is negative, components take
opposite signs
*Correlations must be partialled for covariates z.
Multivariate adjustment, to invalidate
the inference:
Correlational Framework
|ry cv| > .325 and |r x cv| >.310
Impact of a Confounding variable
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Links to sas and stata code
sas program for calculating indices and
related measures
stata code for calculating indices (beta
version – thanks to Yun-jia Lo)
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Alternative: Regression Coefficient and Standard Error
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Maximizing Expression
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Comparing Regression vs Correlation ITCV (example from
Frank, 2000)
Regression (page 155)
Correlation (pages 160,182; Frank, Sykes et al., 2008)
ITCV | z 
rx·y| z  r| z#
1 | r| z# |
overview

.1809  .0579
 .1305
1 | .0579 |
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Exercise 3: Impact Threshold
1)Identify a statistical inference in an article you are
interested in.
2) Describe possible confounds/alternative explanations
that could bias the estimate
3) Note the sample size and t-ratio
recall t=estimate/[se(estimate)]
4) Calculate robustness of inference using
http://www.msu.edu/~kenfrank/papers/calculating%20indic
es%203.xls
5) If you have the data calculate the multivariate ITCV
6) What happens if you change the
estimate
Correlational Framework
standard error
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
sample size
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
null hypothesis
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview
6) Discuss
with a partner Extensions
Conclusion
example:of
effect
of Open Court curriculum
the framework
Applications of Impact Threshold in Education and Sociology (also some in accounting)
Frank, 2000. ITCV=.228.
Frank, K.A., Zhao, Y., Penuel, W.R., Ellefson, N.C., and Porter, S. 2011. Focus, Fiddle and Friends: Sources of
Knowledge to Perform the Complex Task of Teaching. Sociology of Education, Vol 84(2): 137-156.
Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory.
2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy
Analysis. Vol 30(1): 3-30.
Frisco, Michelle, Muller, C. and Frank, K.A. 2007. Using propensity scores to study changing family structure and
academic achievement. Journal of Marriage and Family. Vol 69(3): 721–741
*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37,
349-392. * co first authors.
Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods
and Research, 29(2), 147-194
Crosnoe, Robert and Carey E. Cooper. 2010. “Economically Disadvantaged Children’s Transitions into Elementary
School: Linking Family Processes, School Contexts, and Educational Policy.” American Educational Research Journal
47: 258-291. ITCV =.32
Crosnoe, Robert. 2009. “Low-Income Students and the Socioeconomic Composition of Public High Schools.” American
Sociological Review 74: 709-730. ITCV=.03
Augustine, Jennifer March, Shannon Cavanagh, and Robert Crosnoe. 2009. “Maternal Education, Early Child Care, and
the Reproduction of Advantage.” Social Forces 88: 1-30. ITCV=.4,.23
Maroulis, S. & Gomez, L. (2008). “Does ‘Connectedness’ Matter? Evidence from a Social Network Analysis
within a Small School Reform.” Teachers College Record, Vol. 110, Issue 9.
Cheng, Simon, Regina E. Werum, and Leslie Martin. 2007. “Adult Social Capital: How Family and Community Ties
Shape Track Placement of Ethnic Groups in Germany.” American Journal of Education 114: 41-74.
Correlational Framework
William Carbonaro1 Elizabeth Covay1 School Sector and Student
Achievement
in the Era of variable
Standards Based
Impact
of a Confounding
Reforms. Sociology of eductaion vol. 83 no. 2 160-182 .
Thresholds
for inference
and % bias to invalidate
Internal
validity
Chen, Xiaodong, Frank, Kenneth A. Dietz, Thomas, and Jianguo Liu.
2012. Weak
Ties, Labor Migration, and
The counterfactual
paradigm
Environmental Impacts:
Toward a Sociology
of Sustainability.
& Environment.
Vol. 25 no. 1 3-24.
Replacement
Cases
Framework Organization
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
see also
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Pan,
W., and Frank, K.A. (2004). "An Approximation to the Distribution of the Product of Two Dependent Correlation
overview
example:
effect
of Open Court curriculum
the framework
Coefficients." Journal Conclusion
of Statistical Computation and Simulation,Extensions
74,
419-443 of
Out 6-15-15
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Extensions Logistic
Simulation:
• Kelcey, B. 2009. Improving and Assessing Propensity Score Based
Causal Inferences in Multilevel and Nonlinear Settings.
Unpublished doctoral dissertation, University of Michigan
• Ichino, Andrea, Fabrizia Mealli, and Tommaso Nannicini. 2008.
“From temporary help jobs to permanent employment: what can we
learn from matching estimators and their sensitivity? ” Journal of
Applied Econometrics 23:305–327.
• Nannicini, Tommaso. 2007. “Simulation–based sensitivity analysis
for matching estimators.” Stata Journal 7:334–350.
Table
• David J. Harding. 2003. “Counterfactual Models of Neighborhood
Effects: The Effect of Neighborhood Poverty on High School
Dropout and Teenage Pregnancy.” American Journal of Sociology
109(3): 676-719.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Extensions: Multilevel
If omitted confound is at level 1, no concerns. Covariates include
level 2 fixed or random effects
If omitted confound at level 2:
Seltzer, M. H., Frank, K. A., & Kim, J. (2007). Studying the
Sensitivity of Inferences to Possible Unmeasured Confounding
Variables in Multisite Evaluations. Paper presented at invited
session at AERA 2007.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Extensions: Impact Not Maximized and Impact Curve
Impacts above
the blue line
invalidate the
inference
Smallest impact
overview
Correlational Framework
of a Confounding variable
to invalidate inference: rxcv=ryImpact
=.364
cv
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Extensions: Impacts Before and After Pretests
If k > .109 (or .130 without covariates) then the inference is invalid
Correlational Framework
Impact of strongest measured covariate (student approaches to learning) is -.00126 (sign
Impact
of of
a Confounding
indicates controlling for it reduces estimated
effect
retention) variable
Thresholds
for inference and % bias to invalidate
Internal validity
The counterfactual paradigm
Replacementconfound
Cases Framework
Example:
Effect
oftimes
kindergarten
Impact of unmeasured
would have
to bevalidity
about
100
greaterretention
than
the
Internal
example:
kindergarten
retention
impact of Correlational
the strongestFramework
observed covariateExternal
to invalidate
the inference. Hmmm….
validity
External validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Effect of Retention on Achievement After Adding
each Covariate
Controls
Est
Se
t
School
-21.24
.63
-33.49
School+Pre2 (spring Kindergarten)
-12.01
.45
-26.48
School+Pre2+(Pre2-Pre1)+
-12.10
.47
-26.28
School+Pre2+(Pre2-Pre1)+
Momed
-12.00
.47
-26.26
School+Pre2+(Pre2-Pre1)+
Female
-12.07
.46
-25.18
School+Pre2+(Pre2-Pre1)+
2parent
-12.01
.46
-26.27
Correlational
-12.04Framework.46
-26.16
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement
Casesbased)
Framework The
Hong and Raudenbush
(model
-9.01
Example:
Effect of .68
kindergarten-13.27
retention
Internal validity example: kindergarten retention
Correlational
External
validity
External
validity
exampleOpen Court curriculum
n=10,065,
R2 =.40Framework
overview
example:
effect
of Open
Court
Extensions
of
the framework
Note: 1Conclusion
year’s growth is about 10 points,
so retention
effect
> 1curriculum
year growth
School+Pre2+(Pre2-Pre1)+
poverty
Consider Alternate Sample
(External Validity)
Causal Inference concern:
We cannot assert cause if the effect of is not
constant across contexts.
Statistical Translation:
Would the inference be valid if the sample
included more of some population (e.g. nonvolunteer schools) forCorrelational
whichFramework
the effect was not
as strong?
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal
validity
Rephrased
for
robustness:
what
must
be the
counterfactual
paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity
example: kindergarten
retention
conditions
in
the
alternative
sample
to
invalidate
Correlational Framework
External
validity
External validity exampleOpen Court curriculum
overview
the inference?
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Consider Alternate Sample
(External Validity)
Define  as the proportion of the sample that
is replaced with an alternate sample.
r is correlation in unobserved data
Rxy is combined correlation for observed and unobserved data:
Rxy=(1-)rxy +  rxy .
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
validity
retention
*Frank, K. A.
and Min, K.
2007. Indices of Internal
Robustness
forexample:
Sample kindergarten
Representation.
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
Sociological
Methodology.
Vol
37,
349-392.
* co first authors.
overview
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Thresholds for Sample
Replacement
Set R =r# and solve for rxy:
If half the sample is replaced (=.5), original
inference is invalid if rxy < 2r#-rxy
Therefore, 2r#-rxy defines the threshold for replacement:
TR(=.5)
If rxy=0, inference is altered if π> 1-r#/rxy . Therefore
1-r#/rxy defines the threshold for replacement
TR(rxy=0)
Assumes means and variances are constant across samples, alternative calculations available.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Thresholds for Sample
Replacement: Toy Example
If half the sample is replaced (=.5), original
inference is invalid if rxy < 2r#-rxy
If r#=.4 and rxy =.6 then inference invalid if
rxy < .2
(2 x .4-.6=.2)
If rxy=0, inference is altered if
π> 1-r#/rxy.
If r#=.4 and rxy =.6 then inference invalid if
π> .33
(1-.4/.6=.33)
Assumes means and variances are constant across samples,
alternative calculations
available.
Correlational
Framework
overview
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Interpretation
Replacing cases with null hypothesis
cases  % bias to invalidate
How much would you have to replace
the data with flat line (null hypothesis)
cases to invalidate?
How much would you have to disturb
the data to invalidate the
inference?
Correlational Framework
overview
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Fundamental Problem of Inference to an
Unsampled Population (External Validity)
9
10
11
3
4
5
overview
But how well does
the observed data
represent both
populations?
8
8
8
6
6
Correlational Framework
6
Impact of a Confounding variable
for inference and % bias to invalidate
4Thresholds
6
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Empirical example of Thresholds for Replacement:
Effect of Open Court (Borman et al)
TR(=.5)= 2r#-rxy|z =2(.29)-.54=.03.
Correlation between Open Court and reading achievement
would have to be less than .03 to invalidate inference if
half the classrooms in the sample were replaced (e.g.,
with classrooms with no effect).
TR(rxy =0)= 1-r#/rxy|z =1-(.29/.54)=.47
About 47% (abut 23 classrooms) would have to be replaced
with others for which Open Court had no effect (rxy =0) to
invalidate the inference in a combined sample.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Review of Correlational Framework
Statistical control  impact of
a confound
Internal validity: Impact
necessary to change an
inference
External validity: unobserved
correlations necessary to
change an inference
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Exercise 4: Robustness for Sample
Representativeness (External Validity)
1)Identify a statistical inference in your own work or in the literature for
which there is concern about the external validity
2) Identify possible populations for which the effect may not apply
3) Note the t-ratio and sample size
4) Calculate robustness of inference using
http://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls
5) Interpret the % bias to replace
6) How do the indices change when you change the
sample size?
t-ratio?
standard error?
null hypothesis?
Correlational Framework
Impact
of a Confounding
variable
7) Discuss with a new partner your
inference
and
how
robust
you
think
Thresholds
for
inference
and
%
bias
to
invalidate
Internal
validityroles.
it is. Partner can challenge. Then
change
The
counterfactual
paradigm
Replacement Cases Framework Example: Effect of
kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Out 6-15-15
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Link back to Sample Replacement :
Toy example
If rxy=0, inference is altered if π> 1-r#/rxy.
If r#=.4 and rxy =.6 then inference invalid if
π> .33
(1-.4/.6=.33)
Assumes means and variances are constant across samples, alternative calculations available.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Fundamental Problem of Inference and
Approximating the Counterfactual with
Observed Data (External Validity)
9
10
11
3
4
5
overview
How many cases
would you have to
replace with cases
with zero effect to
change the
inference?
Assume threshold is:
δ# =4:
1- δ# / ˆ
6
6
6
=1-4/6=.33 =(1/3)
6
6
Correlational Framework
6
Impact of a Confounding variable
for
6
4Thresholds
0 inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Linear Adjustment to Invalidate
Inference
Linear model distributes
change over all cases
9
10
11
0
6
6
0
6
0
3+6=9
+3=6
4 +2=6
5 +1=6
6.00
4
6
How many cases
would you have to
replace with zero
effect counterfactuals
to change the
inference?
Assume threshold is
4 (δ# =4):
1- δ# / ˆ
=1-4/6=.33 =(1/3)
overview
Correlational Framework
The inference would be invalid
if you
replaced variable
33% (or 1 case)
Impact of
a Confounding
for was
inference
and % bias effect.
to invalidate
with counterfactuals for Thresholds
which
there
no treatment
Internal
validity
The counterfactual
paradigmeffect)=
ˆ +%replaced(no
New estimate=(1-%
replaced)
Replacement
Cases Framework

Example:
Effect of kindergarten
retention
Internal
validity
example:
kindergarten
retention
ˆ
(1-%replaced)
 =(1-.33)6=.66(6)=4
Correlational
Framework
External
validity
External validity exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Conclusion
Applications of frameworks
Limitations
There will be debate about inferences:
quantify
Assumptions as the bridge between
statistics and science
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Control for Confound ↔Adjusting the Outcome
Replacing observed cases with linear counterfactuals
6
6
6
y   0  1s   2 confound
y  6  4 x s  1 x confound
example: 3=6  4 x 0  1 x -3
3=6-3
add 3 to each side
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
 3+3=6  3  3
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
control for conound
 adjust the outcome
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
overview Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Applications of Frameworks
% bias to invalidate
Any estimate, threshold
Assume equal effect of replacing any case
Counterfactual
Good for experimental settings (treatments)
Think in terms of replacement of cases
Correlational
Good for continuous predictors
Correlational Framework
Impact of a Confounding variable
Think in terms of correlations
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Both can be applied to Internal and External
overview
Validity
Limitations on Inference
Without random sample and random assignment causal
inferences are uncertain. They are inferences.
Observation study
Internal validity: omitted variables
Randomized Study
External validity: fidelity, attrition
Paradox of external validity
If results of randomized study influence behavior, then inherent
limit on external validity
• Prior to study, subjects sign up because of fit, political
pressure, etc
• After study, subjects sign up because they think it
generally works
Framework
Inferences will be debated based onCorrelational
differences
in
Impact of a Confounding variable
Experience
Thresholds
for inference and % bias to invalidate
Internal validity
Power
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Goals
overview
Correlational Framework
Conclusion
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
It’s all in how you talk about it!
Do best method you can
Include relevant controls!
But science is as much in the nature of the
discourse as the method
Virtue epistemology
• Greco, 2009; Kvanig, 2003; Sosa, 2007
What would it take to invalidate the inference?
• Frank, K.A., Duong, M.Q., Maroulis, S., Kelcey, B.
under review. Quantifying Correlational
DiscourseFramework
about Causal
of a Confounding
Inferences from RandomizedImpact
Experiments
and variable
Thresholds
for inference and % bias to invalidate
Internal
validity
Observational Studies in Social
Science
Research
The counterfactual paradigm
Replacement Cases Framework
overview
Correlational Framework
Conclusion
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
Quantify what it Would Take to
Change the Inference
• Objections to moving from statistical to causal inference
– No unobserved confounding variables
– Treatment has same effect for all
• Robustness indices quantify how much must assumptions must be
violated to invalidate inference.
• No new causal inferences!
– robustness indices merely quantify terms of debate regarding
causal inferences.
• Can be used with any threshold.
• Basically, characterizing size of p-value or t-ratio, instead of “barely
significant” versus “highly significant”
• Can be used (theoretically) for any t-ratio
– Discuss: Statistical inference as threshold?
Correlational Framework
Impact
of a Confounding variable
• Difficult to defend tenuous effects (e.g., p
=.049).
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Assumptions are the bridge between statistical and
causal inference
Assumptions
Statistical Inference
Causal Inference
Cornfield, J., & Tukey, J. W. (1956, Dec.), Average Values of Mean Squares in Factorials.
Annals of Mathematical Statistics, 27(No. 4), 907_949.
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
In Donald Rubin’s words
“Nothing is wrong with making
assumptions; on the contrary, such
assumptions are the strands that join the
field of statistics to scientific disciplines.
The quality of these assumptions and their
precise explication, not their existence, is
the issue”(Rubin, 2004, page 345).
Ken adds: and we should talk about the inferences in terms of the
sensitivity to the assumptions
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Paul Holland and Don Rubin
This [the use of randomization to alleviate the
problems of untestable assumptions] should not be
interpreted as meaning that randomization is
necessary for drawing causal inferences. In many
cases, appropriate untestable, assumptions will be
well supported by intuition, theory or past evidence.
In such cases we should not avoid drawing causal
inferences and hide behind the cover of
uninteresting descriptive statements. Rather we
should make causal statements that explicate the
underlying assumptions and justify
them as well as
Correlational Framework
possible.”
Impact of a Confounding variable
overview
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
Reflection
What part if most confusing to you?
Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and
solutions
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
'Crosnoe, Robert' <crosnoe@austin.utexas.edu>; rwerum2@unl.edu; Muller, Chandra L <cmuller@austin.utexas.edu>; bollen@unc.edu;
simon.cheng@uconn.edu; drew@drewdimmery.com; bradley.blaylock@okstate.edu; fabiog@ntu.edu.sg; shevlin@u.washington.edu;
ben.b.hansen@umich.edu; chosman@umich.edu; paul.holland200@att.net; efjuncai@cityu.edu.hk; fbrykho@cityu.edu.hk; mjs3@princeton.edu;
gabriela.olivares@uni.edu; klugman@temple.edu; mgaddis@live.unc.edu; Arthur.Kraft.1@city.ac.uk; HuaiZhang@ntu.edu.sg; jeeseonkim@wisc.edu;
michael.weiss@mdrc.org; mgangl@ssc.wisc.edu; dharding@umich.edu; dlarcker@stanford.edu; trusticus@london.edu; tad61@columbia.edu;
mfeng@katz.pitt.edu; chanli@katz.pitt.edu; gpainter@usc.edu; mcorreia@london.edu; jcore@mit.edu; ksmith1@unl.edu; ssbsup@okstate.edu; Spiro Maroulis
<Spiro.Maroulis@asu.edu>; 'Ben Kelcey' <ben.kelcey@gmail.com>; 'Minh Duong' <mdthetahat@gmail.com>; mseltzer@ucla.edu; cwinship@wjh.harvard.edu;
'James Moody' <jmoody77@soc.duke.edu>; 'William Carbonaro' <William.Carbonaro.1@nd.edu>; 'Mark Berends' <Mark.Berends.3@nd.edu>; Min Sun
<misun@uw.edu>; Richard.Taffler@wbs.ac.uk; machteld.vandecandelaere@ppw.kuleuven.be; annowens@usc.edu; keongkim@cityu.edu.hk;
smithas@rhsmith.umd.edu; jwhitaker@richmond.edu; pkr@pkr.in; eminor1@nl.edu; r-korajczyk@kellogg.northwestern.edu; R.H.Cuijpers@tue.nl;
liuyqphilly@gmail.com; ranxu@msu.edu; sarahgaley@gmail.com; toveave@gmail.com; chentin4@msu.edu; linqinyun1001@gmail.com;
aaron.samuel.zimmerman@gmail.com; torphyka@gmail.com; loyunjia@msu.edu; ichiench@msu.edu; croninge@umd.edu; djklasik@umd.edu; David
Williamson Shaffer <david.williamson.shaffer@gmail.com>; dkaplan@education.wisc.edu; Eric Camburn <ecamburn@education.wisc.edu>; Geoffrey Borman
<gborman@education.wisc.edu>; georgek@uic.edu; stephen.morgan@jhu.edu; Cavanagh, Shannon E <scavanagh@austin.utexas.edu>; Dedrick, Robert
<Dedrick@usf.edu>; John Lockwood <jrlockwood@ets.org>; Lou Mariano <Lou_Mariano@rand.org>; Joshua Cowen <jcowen@msu.edu>;
pay2n@virginia.edu; T Gmail <tdietzvt@gmail.com>; Yu Xie <yuxie@umich.edu>; smithas@rhsmith.umd.edu; Konstantopoulos, Spyros <spyros@msu.edu>;
Kimberly S. Maier <kmaier@msu.edu>; Schmidt, William <bschmidt@msu.edu>; Houang, Richard <houang@msu.edu>; reckase@msu.edu;
knaebleb@uwstout.edu; dutters@uwstout.edu; mcavoy@npg.wustl.edu; sgupta@msu.edu; ricklaux@psu.edu; dlynch@bus.wisc.edu;
JMathieu@business.uconn.edu; deshon@msu.edu; carms@wharton.upenn.edu; d.petmezas@surrey.ac.uk; 'liuyqphilly@gmail.com'; 'Sarah Galey'
<sarahgaley@gmail.com>; 'Jihyun Kim' <toveave@gmail.com>; 'qinyun Lin' <linqinyun1001@gmail.com>; 'Aaron Zimmerman'
<aaron.samuel.zimmerman@gmail.com>; 'Kaitlin Obenauf' <torphyka@gmail.com>; 'loyunjia@msu.edu'; 'I-Chien Chen' <ichiench@msu.edu>; 'Kim Jansen'
<jansenki@msu.edu>; 'zixi chen' <zixichen68@gmail.com>; 'litenglo@msu.edu'; 'John Lane' <lanejoh3@msu.edu>; 'Guan K. Saw' <sawguan@msu.edu>;
'Soobin Jang' <soobinjang@gmail.com>; 'kmartin@msu.edu'; 'Ran Xu' <ranxu@msu.edu>; 'Christopher Lee' <cdeanlee@umich.edu>; 'Rachel White'
<whitera3@msu.edu>; Angelo Joseph Garcia <angelo.giuseppe63@gmail.com>; 'Tingqiao Chen' <chentin4@msu.edu>; linamav@ime.usp.br;
aansari@utexas.edu; pgoldsmith@tamu.edu; vpanov@ecko.uran.ru; astghik.mavisakalyan@curtin.edu.au; Youngs, Peter Alexander (pay2n)
<pay2n@eservices.virginia.edu>; Susanna Loeb <sloeb@stanford.edu>; sean.reardon@stanford.edu; Guanglei Hong <ghong@uchicago.edu>;
sraudenb@uchicago.edu; kfinnigan@warner.rochester.edu; let2119@tc.columbia.edu; wendychan2016@u.northwestern.edu; yp2266@tc.columbia.edu;
estuart@jhsph.edu; croninge@umd.edu; Christina Prell <cprell@umd.edu>; jqeaston@spencer.org; lauramd@gse.upenn.edu; jons@gse.upenn.edu;
slu@ccsr.uchicago.edu; james_kim@gse.harvard.edu; jessaca.spybrook@wmich.edu; rmaynard@gse.upenn.edu; ssadoff@ucsd.edu; hmay@udel.edu;
McCaffrey, Daniel F <dmccaffrey@ets.org>; marisa.a.cannata@vanderbilt.edu; gzamarro@uark.edu; psteiner@wisc.edu; bschneid@msu.edu;
s.a.c.gorard@durham.ac.uk; amanda.carrico@vanderbilt.edu; omar.asensio@ucla.edu; machteld.vandecandelaere@ppw.kuleuven.be; William Penuel
<William.Penuel@colorado.edu>; Allison Atteberry <Allison.Atteberry@colorado.edu>; Henry, Adam D - (adhenry) <adhenry@email.arizona.edu>; Adam
Gamoran <agamoran@Wtgrantfdn.org>; Eric Grodsky <egrodsky@ssc.wisc.edu>;
'Jim Spillane' <j-spillane@northwestern.edu>;
s-ispaCorrelational
Framework
landa@northwestern.edu; Rob Greenwald <wald@sree.org>
Emails of Users
overview
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework
% Bias to Replace for Value
Added
Value added based on average change in student test
scores.
If value added falls below a threshold, teacher identified
as incompetent
What % of change scores would have to be replaced
with average change scores to invalidate inference of
incompetent teacher
If Hanushek argues schools could be improved by
replacing weakest teachers, then we can ask how
many students would weakest
teacher
have to replace
Correlational
Framework
Impact of a Confounding variable
to improve score
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Framework
Students Replacement
could beCases
changed
byThe
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
overview
Teacher
teaching
Conclusion
External
validity
External
validity
exampleOpen Court curriculum
example:of
effect
of Open Court curriculum
Extensions
the framework
better
Replacement of cases can be
thought of as reweighting
overview
Correlational Framework
Impact of a Confounding variable
Thresholds
for inference and % bias to invalidate
Internal validity
counterfactual paradigm
Replacement Cases Framework The
Example: Effect of kindergarten retention
Internal validity example: kindergarten retention
Correlational Framework
External
validity
External
validity
exampleOpen Court curriculum
Conclusion
example:of
effect
of Open Court curriculum
Extensions
the framework