Bridging the Research-Practice Gap: Teacher
Preparation and Evidence-based Education
Ronnie Detrich
Wing Institute
Goals for Today
• Discuss basic characteristics and processes of
evidence-based education.
• Review the legal and ethical basis for evidence-based
education.
• Highlight issues for higher education.
October 1957
USSR launched
Sputnik.
U. S. Education quickly
blamed.
Modern reform efforts
began.
1983
A Nation at Risk
American students
not performing
well.
Education quickly
blamed.
The Nations Report
Card created.
1994
Goals 2000
All students will start
school ready to
learn.
High school
graduation rate ≥
90%.
All students in
grades 4, 8, & 12 will
demonstrate
competency in
challenging subjects.
2001
No Child Left
Behind
By 2014 every student
will be at grade level.
Instructional methods
will be scientifically
based.
Educators will be held
accountable for
outcomes.
Everybody’s Talking
• Teacher accountability often offered as solution for
education troubles.
• Numerous proposals to pay and evaluate teachers
based on student performance.
o Proposals usually linked to performance on high stakes
tests.
Function of Accountability Systems
• Assure that all students are benefitting from
educational services.
• Student performance primary source of feedback to
educators about effects of interventions.
Scope of the Problem
Research to Practice
• Gap concern in many
disciplines.
• Education is not excluded.
• Scientist/Practitioner model
aimed to close the gap.
Gap or Chasm?
Scurvy in the British Royal Navy:
An Example of the Research to Practice Gap
James Lancaster
first experiment
demonstrating how
to prevent scurvy.
1601
John Lind again
experimentally
demonstrated the
effectiveness of citrus in
preventing scurvy.
British Navy adopted
policy to have citrus on all
ships in the Royal Navy.
1747
1795
Research to Practice Issues
• The lag time from efficacy research to
effectiveness research to dissemination is 10-20
years. (Hoagwood, Burns & Weisz, 2002)
• Only 4 of 10 Blueprint Violence Prevention
programs had the capacity to disseminate to 10+
sites in a year. (Elliott & Mihalic, 2004)
How Big is the Gap?
550 named interventions for children and adolescents
Empirically evaluated
Behavioral
Kazdin (2000)
Cognitivebehavioral
Evidence-based interventions are less likely to be used than interventions for
which there is no evidence or there is evidence about lack of impact.
Goals for Evidence-based Practice in Education
• At its core the EBP movement is a consumer
protection movement.
o It is not about science per se.
o It is a policy to use science for the benefit of
consumers.
o “The ultimate goal of the ‘evidence-based movement’
is to make better use of research findings in typical
service settings, to benefit consumers and society….”
(Fixsen, 2008)
How is Education to Increase the Use of
Evidence-based Practices?
• No Child Left Behind (NCLB) requires that
interventions used to improve educational
performance are based on scientific research.
o In NCLB there are over 100 references to scientific
research.
• Individuals with Disabilities Education Improvement
Act [IDEIA] (2004) requires interventions that are
scientifically based instructional practices.
How is Education to Increase the Use of
Evidence-based Practices?
• Specific requirements of IDEIA include:
o Pre-service and professional development for all who
work with students with disabilities to ensure such
personnel have the skills and knowledge necessary to
improve the academic achievement and functional
performance of children with disabilities, including the use
of scientifically based instructional practices, to the
maximum extent possible.
How is Education to Increase the Use of
Evidence-based Practices?
• Scientifically based early reading programs, positive
behavioral interventions and supports, and early
intervention services to reduce the need to label
children as disabled in order to address the learning
and behavioral needs of such children.
How is Education to Increase the Use of
Evidence-based Practices?
• The Individualized Education Program (IEP) shall
include a statement of the special education and
related services and supplementary aids and
services, based on peer-reviewed research to the
extent practicable, to be provided to the child, or on
behalf of the child, and a statement of the program
modifications or supports for school personnel that
will be provided for the child.
How is Education to Increase the Use of
Evidence-based Practices?
• In determining if a child has a specific learning
disability, a local education agency may use a process
that determines if a child responds to a scientific,
research-based intervention as part of the evaluation
procedures.
The Ethical Basis
• Most national psychological and educational
organizations have ethical standards requiring
science-based practices to address problems.
o American Psychological Association Ethical Standard 2.04:
 Psychologists’ work is based on the established scientific and
professional knowledge of the discipline.
The Ethical Basis
• National Association of School Psychologists
o Standard III F 4.
 School psychology faculty members and clinical or field supervisors
uphold recognized standards of the profession by providing
training related to high quality, responsible, and research-based
school psychology services.
The Ethical Basis
• National Association of School Psychologists
o
Standard IV 4.
 School psychologists use assessment techniques, counseling and
therapy procedures, consultation techniques, and other direct and
indirect service methods that the profession considers to be
responsible, research-based practice.
The Ethical Basis
• Behavior Analyst Certification Board
o Standard 2.09a
 The behavior analyst always has the responsibility to recommend
scientifically supported, most effective treatment procedures.
Effective treatment procedures have been validated as having both
long-term and short-term benefits to clients and society.
o Standard 2.09b
 Clients have a right to effective treatment (i.e., based on the
research literature and adapted to the individual client).
Another Ethical Responsibility
• Education services are largely funded through public
dollars (taxpayers dollars).
• There is an implicit assumption that the money will
be spent for the public good.
• A fiduciary responsibility exists when one person or
organization is charged with managing another
person’s money.
Another Ethical Responsibility
• The primary responsibility of a fiduciary is to act
solely for the benefit of the other party.
• Being a fiduciary carries the weight of ethical
conduct.
• Those of us charged with providing education
services have a fiduciary responsibility to assure
that the taxpayers are receiving the greatest
possible return on their investment.
How Do We Meet Our Fiduciary Responsibility?
• Interventions that have an evidence base are more
likely to produce positive effects for students.
o Does not assure positive outcomes but increases the
probability.
How Do We Meet our Fiduciary Responsibility?
• The impact of a non-evidence based intervention
is unknown.
o Using a non-evidence-based intervention when
evidence-based interventions available may constitute
unethical practice.
o Use a non-evidence based intervention should be
considered research
 All of the safe-guards afforded research participants and their
families should be in place.
o Conducting research with tax-dollars provided for
education services may constitute a violation of our
fiduciary responsibility.
Becoming Evidence-based
• Clearly, the intent of Congress, the U.S. Department
of Education, and Office of Special Education
Programs is to rely on interventions that have a
scientific basis.
• Professional organizations place great value on
scientific knowledge.
• What does it mean to be evidence-based?
What is Evidence-based Practice?
• Evidence-based practice has its roots in medicine.
o Movement has spread across major disciplines in
human services:
 Psychology
 School Psychology
 Social Work
 Speech Pathology
 Occupational Therapy
What Is Evidence-based Practice?
Professional
Judgment
Best available
evidence
Client Values
Sackett et al (2000)
Client
Values
Professional
Available
Judgment
Evidence
• EBP is aBest
decision-making
approach that places
emphasis on evidence to:
o guide decisions about which interventions to use;
o evaluate the effects of an intervention.
Phases of Evidence-based
Intervention
Identify
Identify
Evidence-based
Intervention
Evaluate
Evaluate
Implement
Implement
Identify
What is Evidence-based Education?
• The term “evidence-based” has become
ubiquitous in last decade.
o Often used interchangeably with empirically supported
and best practice.
o No consensus about what it means.
o At issue is what counts as evidence.
o Federal definition emphasizes experimental methods.
 Preference for randomized trials.
 Definition has been criticized as being positivistic.
Identify
What Counts as Evidence?
• Ultimately, this depends on the question being
asked.
o Qualitative methods are best for answering social
validity questions.
• In EBP the goal is to identify causal relations
between interventions and outcomes.
o Experimental methods do this best.
Identify
What Counts as Evidence?
• Even if we accept causal demonstrations to be
evidence, we have no consensus.
o Randomized Clinical Trials (RCT) have become the “gold
standard.”
o There is controversy about the status of single subject
designs.
 Most frequently criticized on the basis of external validity.
 WWC has recently established standards for SSDs .
 No well established method for calculating effect sizes.
Identify
Distinguishing Between Evidence-based and
Empirically Supported
• Evidence-based refers to practices that have been
validated through a systematic review.
o Can involve meta-analysis.
o WWC reviews are systematic reviews.
• Empirically supported refers to practices that have
received support in peer reviewed journals.
o Have not been systematically reviewed to establish
strength of evidence.
Identify
Applying Best Available Evidence Construct
• If validated intervention available adopt it.
o Assuming it is appropriate to context.
• If no validated intervention select empiricallysupported intervention.
• If no empirically-supported interventions develop
intervention based on principles.
o Principles of scientific reading.
o Principles of behavior.
Identify
Applying EBP Framework
• Professional judgment involved at all levels of
selection an intervention.
• Interventions must fit competencies of those
implementing.
• Interventions must be acceptable to consumers and
fit values.
Identify
How Are Evidence-based Interventions
Identified?
• Identification is more than finding a study to
support an intervention.
• Identification involves distilling a body of
knowledge to determine the strength of evidence.
Identify
How Are Evidence-based Interventions
Identified?
• Distillation requires standards of evidence for
reviewing the literature.
o Standards specify:
 the quantity of evidence
 the quality of evidence
Identify
Continua of Evidence
Quantity of the Evidence
Meta-analysis
(systematic review)
Repeated Systematic
Measures
Single Case Replication
(Direct and Parametric)
Threshold
Convergent Evidence
of
Evidence
Quality of the Evidence
Current “Gold Standard”
High Quality
Randomized Controlled Trial
Single Case Designs
Semi-Randomized Trials
Well-conducted
Clinical Studies
Uncontrolled Studies
Expert Opinion
Various Investigations
Single Study
General Consensus
Personal Observation
Janet Twyman, 2007
Identify
How Are Evidence-based Interventions
Identified?
• Two approaches to validating interventions
o Threshold approach:
 Evidence must be of a specific quantity and quality before an
intervention is considered evidence-based.
 What Works Clearinghouse
» Meets evidence standards.
» Meets evidence standards with reservations.
» Does not meet standards at this time.
o Hierarchy of evidence approach:
 Strength of evidence falls along a continuum with each level
having differential standards.
 National Autism Center
»
»
»
»
Established
Emerging
Unestablished
Ineffective/Harmful
Identify
Identify
Evidence-based
Intervention
Evidence-based
Intervention
Identify
No Agreed upon Standards
Intervention X
Validated
Standard 1
Standard 2
Not Validated


How are consumers to decide?
Identify
Ineffective
Effective
Effective
Ineffective
Assessed Effectiveness
Actual Effectiveness
Effective
Effective
Ineffective
Ineffective
True
Most
likely with
False
hierarchy approach
Positive
Positive
Most
likely with
False
threshold approach
Negative
True
Negative
Identify
Choosing Between False Positives and False
Negatives
• At this stage, it is better to have more false
positives than false negatives.
False Negatives:
Effective interventions will
not be selected for
implementation.
As a consequence, less
likely to determine that they
are actually effective.
False Positives: Progress
monitoring will identify
interventions that are
not effective.
Role of Higher Ed in Identifying Evidence-based
Practices
• Conduct systematic reviews to identify effective
practices.
o Multiple organizations are have processes for validating
interventions.







What Works Clearinghouse
Best Evidence Encyclopedia
Campbell Collaboration
Center for Evidence-based Policy
National Autism Center
Promising Practices Network
CEC developing standards for review.
Role of Higher Ed in Identifying Evidence-based
Practices
• Independent reviews for journals:
o Evidence-based Communication Assessment and
Intervention
o Browder, et al., 2006: Research on Reading Instruction for
Individuals with Significant Cognitive Disabilities,
Exceptional Children.
• Expand dissemination efforts.
o Specify “audience.”
 Different audiences may require different methods.
o Expand methods.
Diffusion of Innovation
Rogers, Diffusion of Innovation, 2003
• Diffusion of innovation is a social process, even more
than a technical matter.
• The adoption rate of innovation is a function of its
compatibility with the values, beliefs, and past
experiences of the individuals in the social system.
Principles for Effective Diffusion:
Improving the Odds (Rogers, 2003)
• Innovation has to solve a problem that is important
for the “client.”
• Innovation must have relative advantage over
current practice.
• It is necessary to gain support of the opinion leaders
if adoption is to reach critical mass and become selfsustaining.
• Innovation must be compatible with existing values,
experiences and needs of the community.
Principles of Effective Diffusion:
Improving the Odds
• Innovation is perceived as being simple to
understand and implement.
• Innovation can be implemented on a limited basis
prior to broad scale adoption.
• Results of the innovation are observable to others.
Q&A
Implement
Implementation: Bridging the Research-Practice
Gap
• Is identifying evidence-based interventions
sufficient to meet regulatory and ethical
requirements?
o Intent of both legal and ethical guidelines is to have
positive impact.
 Evidence-based interventions are assumed to give to give us
that chance.
• Identification is necessary but not sufficient to
assure that intervention will be effective.
Implement
What We Know
• Teachers are primary means of exposure to
interventions.
• Students will not benefit from effective practices if
they are not exposed to them.
• Data suggest that preparation programs are not
preparing trainees to use evidence-based practices.
Implement
16%
What Are Universities Teaching
About Formative Assessment?
14%
% of Course Syllabi
12%
10%
Sample of 13
Elementary Teacher Programs
8%
14%
6%
4%
2%
0%
0%
Progress Monitoring, Formative
Assessment, or Ongoing Assessment
Dibels
Response to Intervention and Teacher Preparation, Spear-Swerling, 2008
Implement
Are We Training Educators to Know What to
Do?
Survey of School Psychology Directors of Training
(Shernoff, Kratochwill, & Stoiber, 2003)
Evidence-based interventions
29%
directors Knowledge
41%
programs
Training
Implement
Teacher Preparation Programs Under Scrutiny
By almost any standard, many if not most of the
nation's 1,450 schools, colleges, and departments of
education are doing a mediocre job of preparing
teachers for the realities of the 21st century
classroom.
Arne Duncan, Secretary of Education
Teacher Preparation Programs Under Scrutiny
Teacher education is the Dodge City of the education
world…it is unruly and disordered. There is no
standard approach to where and how teachers
should be prepared.
Levine, 2006
Teacher Preparation Programs Under Scrutiny
The nation’s leading educators concede that there is
presently very little empirical evidence to support
the methods used to prepare the nation’s teachers
Walsh, 2006
Is this really our choice?
Implement
Teacher Preparation Programs Under Scrutiny
• National Council on Teacher Quality reviewing all
1400+ teacher preparation programs in U.S.
o Graded on 17 standards.
 Classroom management
 Instructional Practices in reading, math, and ELL.
 Field work experience.
 Assessment practices.
 Instructional Design
Scientific Methods of Instruction
Assessment and Data Interpretation
Scientific Basis of Early Reading
University Accountability
Implement
Implementation: Where Good Interventions Go
to Die
• Implementation is not important unless it is done
with integrity.
• Evidence-based drug education programs are
implemented with integrity only 19% of the time.
(Hallfors & Godette, 2002)
o This may be a generous estimate.
o No reason to assume that other interventions fare better.
Implement
Why Treatment Integrity is Important?
• Kovaleski, Gickling, Morrow, & Swank (1999)
o Evaluated high vs low implementation of Instructional
Support Teams (IST).
 School-wide organizational change.
 Students benefited from IST processes only when implemented
with high fidelity.
 Implementing with low fidelity resulted in no better outcomes for
students than control group not exposed to IST processes.
 Having structures in place was not sufficient to assure high fidelity.
 Fidelity assessed one time per year.
Implement
Why Treatment Integrity is Important?
• Horner (2005)
o Effect of high fidelity vs low fidelity on office discipline
referrals.
 Schools that implemented with high fidelity had 25% fewer
office referrals for major rule violations than schools that did
not meet fidelity criterion.
 Fidelity measures taken 2 times per year.
• If we want to increase the integrity of
implementation we need a different training
model.
Effects of Training
OUTCOMES
(% of Participants who demonstrate knowledge, demonstrate new
skills in a training setting,
and use new skills in the classroom)
Knowledge
Skill
Demonstration
Use in the
Classroom
Theory and
Discussion
10%
5%
0%
..+Demonstration in
Training
30%
20%
0%
…+ Practice &
Feedback in
Training
60%
60%
5%
…+ Coaching in
Classroom
95%
95%
95%
TRAINING
COMPONENTS
Joyce and Showers, 2002
Mortenson & Witt, 1998
Implement
Challenges for Teacher Preparation Programs
• Adoption or adaptation?
o Must we implement exactly as prescribed or can we
adjust to fit local circumstances?
• Research suggests that programs are almost
always adapted.
• Presumably adapted to improve outcomes.
o Some adaptation for other reasons:
 Better fit teaching style.
 Do not like some elements of program.
Implement
Challenges for Teacher Preparation Programs
• If adapted is program still research-based?
o If teacher’s allowed to adapt then program more
acceptable.
o Teacher’s made better adaptation of reading programs if
they were well grounded in principles. (Klingner, Vaughn,
Hughes, & Arguellas, 1999).
Implement
Challenges for Teacher Preparation
• Teach principles of instruction or teach specific
methods of instruction.
o Principles of reading: 5 elements of scientifically based
instruction.
 Reading Mastery contains all elements.
o Not all schools use Reading Mastery.
 If train to Reading Mastery then may not be skilled at
implementing other reading programs.
 Principles are always expressed as part of a package.
 Not all packages created equally.
 There are more reading programs than is feasible to teach to preservice teachers.
Implement
Challenges for Teacher Preparation
• If emphasize specific curricula:
o May not be able to adapt because do not have principles
of instruction to guide decision making.
Training Methods for New Teachers
• How do we increase generalization from pre-service
to classroom?
o Clinical training Model gaining traction.
 Will require reorganization of much of training and field work
experience.
 How skills are taught at pre-service level.
 How mentors interact with trainees.
 How mentors are selected.
 How competencies of mentors are defined and evaluated.
 Length of training?
Additional Training Content: Seven Questions
Parents Should Ask and Why They Are Important
1. What is the evidence-base for this intervention?
o What is known about the effects for children like mine?
o What is known about effects in settings like this?
Seven Questions Parents Should Ask and Why
They Are Important
2. What are the other options?
•
Both Picture Exchange Communication System and
Sign Language are evidence-based for non-verbal
children with developmental disabilities.
o Usually practitioners recommend one without discussing
options.
Seven Questions Parents Should Ask and Why
They Are Important
3. Why is this intervention being recommended?
•
A good answer describes how the recommended
intervention is a good fit for a particular child.
o A less desirable answer is anything that does not describe
how it is the best match for this child.
Seven Questions Parents Should Ask and Why
They Are Important
4. What are the risks?
• All interventions have risks.
o A good answer reflects that risks have been considered
and steps taken to minimize when possible.
o Both PECS and Sign Language have risks:
 PECS requires the child always have access to the communication
system which can be difficult to transport.
 Sign language requires a community of others who sign.
Seven Questions Parents Should Ask and Why
They Are Important
5. Do those responsible for implementing have the
necessary training, skills, and knowledge to
implement with adequate levels of treatment
integrity?
Seven Questions Parents Should Ask and Why
They Are Important
6. Are the resources available to implement with
adequate levels of treatment integrity?
• Most commonly cited reasons for failing to
implement with integrity:
o Lack of training
o Lack of materials
o Lack of time
Seven Questions Parents Should Ask and Why
They Are Important
7. How will the effectiveness of the intervention be
evaluated?
o How will the family be kept informed about effects?
o When shall we meet again to review progress?
Evaluation of Intervention
Evaluate
Evaluating Evidence-based Interventions
Progress Monitoring
• Implementation of evidence-based intervention does
not assure success.
o Necessary to evaluate impact in local context.
 No intervention will be effective for all students.
 Cannot predict who will benefit.
o Progress monitoring is practice-based evidence about
evidence-based practices.
o Consistent with legal requirements and ethical standards.
Evaluate
Ethical Standards and Progress Monitoring
• National Association of School Psychologists
o Standard IV C 1b.
 Decision-making related to assessment and subsequent
interventions is primarily data-based.
o Standard IV 6.
 School psychologists develop interventions that are appropriate to
the presenting problems and are consistent with the data collected.
They modify or terminate the treatment plan when the data
indicate the plan is not achieving the desired goals.
Evaluate
Ethical Standards and Progress Monitoring
• Behavior Analysis Certification Board
o Standard 4.04
 The behavior analyst collects data or asks the client, clientsurrogate, or designated other to collect data needed to assess
progress within the program.
o Standard 4.05
 The behavior analyst modifies the program on the basis of data.
Evaluate
Legal Requirements for Progress Monitoring
• Fundamental to IEP process.
o Must report on same schedule that grades are reported in
general education.
• Response to Intervention is accepted as alternative
means for determining eligibility for Learning
Disability classification.
o Progress monitoring is the heart of RTI.
 All students routinely and systematically monitored to assure
adequate progress is occurring.
Evaluate
Why Teach Progress Monitoring?
• Progress monitoring 2-5/week in math and reading:
o
o
o
o
4 times as effective as 10% increase in per pupil spending;
6 times as effective as voucher programs;
64 times as effective as charter schools;
6 times as effective as increased accountability.
Yeh (2007)
Evaluate
Why Teach Formative Assessment?
Hattie, Visible Learning, 2009
Fuchs & Fuchs, 1986
Evaluate
Evaluating Evidence-based Interventions
• Curriculum based measurement is a powerful means
for evaluating impact of academic interventions.
o Scores on CBM correlated with scores on high stakes test.
 Can be used to predict how students will perform on state-wide
tests.
Evidence-based Education,
Evaluate
Progress Monitoring and Treatment Integrity
• Student data provides feedback about progress.
• If we know about adequacy of treatment integrity
then can make decisions:
o Adequacy of intervention
o Adequacy of implementation
 If implementation is inadequate then focus should be on
improving educator behavior.
 If implementation is adequate then focus should be on changing
intervention so student can succeed.
 Decisions can be made about increasing or decreasing intensity of
intervention.
Visual Aids Can Enhance Data Interpretation
Grade Level Standard
Aim Line
Trend Line
Evaluate
Outcome
Negative
Positive
Continue Intervention
Negative
Change Intervention
Low
High
Integrity
Low
High
Positive
Unknown reason
Unknown reason
• Other life changes?
• Intervention problem?
• Unknown
intervention?
• Implementation problem?
• Intervention is
effective?
Evaluate
Where are We?
• Being evidence-based is the law and it is ethical
conduct; however, it is not as easy as it sounds.
• A common definition of evidence protects
consumers.
• The research to practice gap limits the impact of
evidence-based education.
o The science of implementation is in its infancy.
Evaluate
Where are We?
• Pre-service training should change to reflect current
policy.
o Current methods are not producing desired outcomes.
o Changes in both method of training and content.
Thank You
Copies May be Downloaded at
www.winginstitute.org