Where are we going?
• What to do if no RCT, RD, ITS or sophisticated
matching is possible?
• We describe and analyze principle of pattern
matching to improve the basic workhorse design by a
feature other than high quality matching
• Illustrate what you might do if there is no pretest at
all and so not even a work-horse design is possible
• Before that, a bit of a summary
Yesterday 1
• Do not match from extremes unless forced to
• Then match using a reliable set of measures,
moving to propensity score framework, latent
variables, or statistical reliability-adjustments
to handle unreliability
• But you will still have problem of possible
specification error!
Yesterday 2
• In selecting non-equivalent control group:
• Use local focal matching to reduce the degree of
initial non-comparability though you cannot expect
total non-comparability
• Sometimes the control group so formed will not
differ from what would have been achieved with
random assignment, at least on observables
• At other times, the initial group non-equivalence will
be reduced for when you come to do statistical
analysis to “control” initial differences
Yesterday 3
• When there is initial non-equivalence on observables:
• Theory and empiricism about the constructs (covariates)
in the “true”selection model helps, as does careful
measurement thereof
• Measures in multiple other domains helps too,
• Unclear whether ANCOVA or propensity scores do better-Shadish et al, Glazerman et al.--though demographic
variables alone and Heckman IV models do not fare well.
Propensity scores preferred on theory alone.
• You can never be sure of the final causal conclusion,
though
Implication
• In designing research you do well to avoid the
workhorse if you can, though it is modal in current
educational practice
• Can you add prior pretest waves?
• Can you add any of the other design elements, some
mentioned in our ITS discussion and others we
discuss today
• How can you design your way out of reliance on a
simple design with non-equivalent groups and
pretest and posttest?
When there is no Pretest on the same
scale as Outcome
• Do randomized experiment--Abcedarian, PerryPreschool, Head Start, Early Head Start, CLIO, Even
Start and Sesame Street
• If not possible, do everything possible to make
control group focally local
• Add design elements to rule out alternative
explanations of a possible causal relationship
• Here’s an example
Minton’s Dissertation
•
•
•
•
•
Object: Evaluate Sesame St in 1st year
Problem 1: Program already launched
Problem 2: No pretest possible
Problem 3: No money for original data collection
Setting: One kindergarten in NJ that built SS into its
day and that has records on children and their
families plus annual PPV assessment
Question 1: What control group is
possible?
• What control group to find, given program was very
popular in its first year.
• Why is popularity a problem?
• Neighborhood kids who did not view
• Next-door kids of same age who not view?
• Older siblings in general
• Older sibs attending same kindergarten within last N
years
• Older sibs attending same kindergarten last 2 years
Achievemen
t scores
Minton (1975) Sesame Street Study - 1
Older
control
Younger
Sesame Street
Achievemen
t scores
Minton (1975) Sesame Street Study - 2
Older
control
Letter
skills
Non-letter
PPVT skills
Younger
Sesame Street
Minton (1975) Sesame Street Study - 3
Achievemen
t scores
Letter skills – high viewers
Older
control
Letter skills – low viewers
Non-letter skills–low viewers
Non-letter skills–high viewers
Younger
Sesame Street
What has happened here?
• Single causal hypothesis of SS effective made
to have multiple data implications
• These are meant to rule out alternative
hypotheses and not to recreate same bias
• These implications here in the form of a
difference in difference in differences
• Collect data and test hypothesis
Another Example
• How the Introduction of TV affected Library
Circulation
Parker et al. (1966) Effects of TV - 1
Library
circulation per
capita
Short interrupted time series
19 19 19
45 49 53
Fiction book
circulation
1975
Parker et al. (1966) Effects of TV - 2
Library
circulation per
capita
Short interrupted time series with control
19 19 19
45 49 53
Early TV communities
Late TV communities
1975
Parker et al. (1966) Effects of TV - 3
Library
circulation per
capita
ITS with switching replication
1949 interruption
1953 interruption
19 19 19
45 49 53
1975
Parker et al. (1966) Effects of TV - 4
Library
circulation per
capita
ITS with switching replication and control
19 19 19
45 49 53
Fiction
Fact
Fiction
Fact
1975
What has happened here?
Combine an ITS with non-equivalent DVs and
switching replications
What alternative interpretations can you come up
with?
How plausible are these?
• Have we seen this before with RD and ITS?
• One general causal hypothesis has multiple
implications in the data
• Predicted hypothesis as multiple differences of
differences; as higher order interactions
Reynolds and West’s (1987) “Ask for the
Sale” Experiment
From all stores selling lottery tickets, some stores
volunteered (or not) to post a sign reading “Did we
ask you if you want a Lottery ticket? If not, you get
one free”. So this is a basic nonequivalent control
group design, with the control matched on zip
code, store chain, and pretest ticket sales.
NR O1
X
O2
----------------------------NR O1
O2
The Outcome of the Basic Design
But there might be many reasons besides
Adding a Nonequivalent DV
•
They added three nonequivalent dependent
variables, showing that the intervention
increased ticket sales but not sales of gas,
cigarettes, or grocery items.
Adding Multiple Baselines - recasting
as ITS Design
• They located some stores in which the
treatment was initiated later than in other
stores, or initiated and then removed, and
found that the outcome tracked the
introduction of treatment over time while
sales in the matched controls remained
unchanged
Adding Multiple Pretests and Posttests
•
They added multiple pretests and posttests
by examining mean weekly ticket sales for
four weeks before and four weeks after the
treatment started.
The Point is:
• To use the choice of additional design
elements to rule out more alternative
interpretations, hopefully all that can be
currently identified
• The goal is ruling out plausible alternative
interpretations, and it can also be reached via
keeping the pattern of results constant but
varying the number and type of comparisons
involved
Main NAEP 4th grade math scores
by year and proficiency standards
D & J Results: 4th Grade Math
Main NAEP 4th grade math scores by year:
Public and Catholic schools
Main NAEP 4th grade math scores by year:
Public and Other Private schools
Trend NAEP 4th grade math scores by year:
Public and Catholic schools
Student Enrollment
Catholic
Other Private
Public
1994
5.73
4.72
89.55
1996
5.67
4.74
89.60
1998
5.58
4.87
89.56
2000
5.38
4.81
89.81
2002
5.26
5.13
89.61
2004
4.88
4.93
90.18
2006
4.56
5.07
90.37
Warning!
• This pattern matching strategy requires:
• Clear causal hypothesis - relevance to
discontinuity
• Careful measurement - reliability and ceiling
or floor effects
• Large samples (or large effects) because
hypothesis is of a complex statistical
interaction
• How lucky Minton was!
Examples from you of the Basic WorkHorse Design
• Let us take some from you and see if they can
be improved by adding design elements.
Design Elements to be combined:
Assignment
• Random Assignment
• Cutoff-Based Assignment
• Researcher-controlled Matching -- of many
kinds in econometric literature
Design Elements to be
combined:Treatments
•
•
•
•
Switching Replications
Reversed Treatments
Removed Treatments
Repeated Treatments
Design Elements to be combined:
Measurement
•
•
•
•
•
•
Single Pretest
Pretest Time Series
Proxy Pretests
Retrospective Pretests
Moderators with predicted Interactions
Measuring Threats to Validity
Design Elements to be combined:
Comparison Groups
•
•
•
•
•
Single Non-Equivalent Groups
Multiple Non-Equivalent Groups
Twins/Siblings
Cohorts
Other Focal, Local Comparison Groups
Golden Rules (1)
• You can’t put right through statistics what you
have done wrong by design
• Statistical adjustments work better the less
non-equivalence there is to adjust away in the
first place
• Since the work horse is so prevalent but so
problematic, how can we complexify the
design through adding design elements
Golden Rules (2)
• First, Do an experiment; if not
• Do Regression-discontinuity study. If not,
• Do ITS with some sort of a comparison series.
If not
• Do study combining multiple design element,
preferably with focal local intact controls, case
matching on many covariates, reintroduction
of treatment at new time, non-equivalent DVs,
etc.
Golden Rules (3)
Don’t be bamboozled by fancy models in
Greek clothing. Always translate them into
structural design elements before evaluating
their likely validity. That will reveal what you
have got
• Remember you only control for the reliably
measured part of any construct, not the
construct itself.
Evaluation, formative
On a scale from 1 to 6, with 6 being high, please rate the
following and then indicate how you would improve what we
did.
Contact with Valerie about the workshop
Accommodations
Food
Curriculum Content
Curriculum Relevance to your current or anticipated work
Quality of Instruction
Any other Suggestions for Improvements?
Grant Opportunities
at the Institute of Education Sciences
Allen Ruby, Ph.D.
Associate Commissioner
Policy & Systems Division
National Center for Education Research
Take Away
• IES has a small number of education grant programs
–
–
–
–
–
Matrix of topics/goals within grant programs
Range of work: exploration, development, evaluation
Focus is on improving student outcomes
Competitions are held twice a year (6/24/10 and 9/16/10)
Competitive peer review using an absolute standard
• Take advantage of available information
– http://ies.ed.gov (Request For Applications, abstracts of
projects, webinars, resources for researchers webpage)
– Program officer (listed in RFA)
IES Structure
Standards &
Review
National
Center for
Education
Statistics
Office of the
Director
National
Center for
Education
Evaluation
National Board
for Education
Sciences
National
Center for
Education
Research
National
Center for
Special Ed
Research
Overall Research Objectives
• Develop or identify education interventions
(practices, programs, policies and approaches)
that enhance academic achievement and that
can be widely deployed
• Identify what does not work and thereby
encourage innovation and further research
• Understand the processes that underlie
variations in the effectiveness of education
interventions
Final Outcomes of Interest are for Students
Preschool
• School readiness
• Developmental outcomes for infants and toddlers with
disabilities
Kindergarten through Grade 12
• Academic outcomes in reading, writing, math and science
• Behaviors, interactions, and social skills that support learning in
school and successful transitions to post-school opportunities
• High school graduation
• Functional skills for independent living of students with
disabilities
Postsecondary: enrollment, persistence, and completion
Adult Education: reading, writing, and math (basic, secondary,
and EL)
Key Dates
Application Letter of Intent
Deadline iesreview.ed.gov
Application
Package
Start
Dates
www.grants.gov
6/24/10
4/29/10
4/29/10
3/1/11
to
9/1/11
9/16/10
7/19/10
7/19/10
7/1/11
to
9/1/11
Research and Research Training Grant
Programs
• Education Research Grant Programs
• Special Education Research Grant Programs
• Statistical and Research Methodology in Education
• Evaluation of State and Local Education Programs
and Policies
• Postdoctoral Research Training Grant Programs
• National Research and Development Centers
Education Research Grants Programs (84.305A)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Reading and Writing
Mathematics and Science Education
Cognition and Student Learning
Teacher Quality (Reading & Writing; Math & Science)
Social and Behavioral Context for Academic Learning
Education Leadership
Education Policy, Finance, and Systems
Postsecondary Education
English Learners
Early Learning Programs and Policies
Education Technology
Adult Education
Organization and Management of Schools and Districts
Analysis of Longitudinal Data to Support State & Local
Education Reform
Special Education Research Programs (84.324A)
• Early Intervention and Early Childhood Special
Education
• Reading, Writing, and Language Development
• Mathematics and Science Education
• Social and Behavioral Outcomes to Support Learning
• Cognition and Student Learning in Special Education
• Professional Development for Teachers and Related
Service Providers
• Special Education Policy, Finance, and Systems
• Transition Outcomes for Special Education
Secondary Students
• Autism Spectrum Disorders
5 Research Goals
1. Exploration: examine relationship between
malleable factors and education outcomes
2. Development and Innovation: iteratively develop
new or improved education interventions
3. Efficacy & Replication: evaluate the efficacy and
feasibility of an intervention
–
Efficacy Follow-up Studies
4. Scale-up Evaluation: evaluate the impact &
feasibility of interventions at scale
–
Scale-up Follow-up Studies
5. Measurement: develop and validate assessments
Exploration
• Explore malleable factors that are associated
with better student learning and education
outcomes
– Malleable factor: something that can changed by
the education system - student, teacher, or school
characteristics, or an education program or policy
• Explore factors that mediate or moderate the
relationship between malleable factors and
student outcomes
• Small primary data studies, secondary
analyses, and meta-analyses
Development and Innovation
• Develop new interventions (e.g., instructional
practices, curricula, teacher professional
development, policies)
• Demonstrate the feasibility of the intervention for
implementation in an authentic education delivery
setting
• Collect pilot data on promise of intervention to
achieve intended outcomes
Efficacy and Replication
• Test whether or not fully developed
interventions are effective under specified
conditions and with specific types of students
• Take place under supportive conditions , e.g.
homogenous sample, high assistance
• Studies using random assignment to
intervention and comparison conditions are
preferred where feasible
• New this year: Efficacy follow-up studies
Scale-up Evaluation
• Test whether interventions are effective when
implemented under typical conditions.
• As implemented by practitioners and with
sufficiently diverse samples to support
generalizability.
• Studies using randomized assignment to
treatment and comparison conditions are
preferred whenever they are feasible.
• Added this year: Scale-up follow-up studies
Measurement
• Develop and validate assessments or other
measurement tools
– Typically to be used by practitioners (e.g.,
screening, progress monitoring, and outcome
assessment) but can also be for researcher use
– Validation of non-student measures must involve
student outcomes (e.g. Teacher Quality)
– Program specific, e.g., cost-accounting under
Education Policy, Finance, and Systems
• Not for evaluating an assessment used as an
intervention
• The measure is the primary product
Maximum Grant Duration and Typical Size
Grants include both direct and indirect costs
• Exploration: 2 years, $100,000 to $350,000 per yr
10%
– Up to 4 years & $400,000 for primary data collection
• Development: 3 years, $150,000 to $500,000 per yr 50%
• Efficacy: 4 years, $250,000 to $750,000 per yr
26%
– Follow-up: 3 years, $150,000 to $400,000 per yr
• Scale-up: 5 years, $500,000 to 1.2 million per yr
2%
– Follow-up: 3 years, $250,000 to $600,000 per yr
• Measurement: 4 years, $150,000 to $400,000 per yr 12%
Information Sources
• IES website: http://ies.ed.gov
• Obtain IES Request for Applications
– Describes type of research that can be done and substance
of what to include in application
– http://ies.ed.gov/funding
• IES Grants.Gov Application Submission Guide
– Step by step instructions on how to apply
– http://ies.ed.gov/funding
• Obtain Application from and Submit Application to
– www.grants.gov
– 84.305A (regular education) & 84.324A (special education)
• Initial IES Contact (feel free to contact)
– Allen.Ruby@ed.gov
• Handout includes information covered in this presentation