RTI Professional Learning Webinar Series:
Using Research to Select and Design
Effective Interventions – Session 4
October 15, 2009
Our session will start momentarily. While you are waiting, please do the following:
Enter/edit your profile information by going to:
•Tools - Preferences - My Profile…
•Fill out the info on the “identity” tab and click “OK”
•To view the profile of another use, hover your mouse over his or her name in the
participants window
Configure your microphone and speakers by going to:
•Tools – audio – audio setup wizard
Confirm your connection speed by going to:
•Tools – preferences – connection speed
RTI Professional Learning Webinar
Series:
Using Research to Select
and Design
Effective Interventions
.
Session #4 10/15/09 3-4:30 pm
Critical Review of Research Studies
on Interventions
2
Counting ALL Participants
After you enter your profile information to
join the webinar as usual:
1. Go to the Chat Window
2. Type your district name and school name
or organization name or “parent”
3. Type the names of every person in
attendance with you
4. Send it to “This Room”
3
Who are we?








Teachers
Administrators
Counselors or Social Workers
County Office or DOE
Parents
PK-5, 6-8, 9-12, post-secondary
Statewide Organizations
Other?
4
Welcome and
Introductions
 John Wight, GaDOE
 Dr. Kimberly Anderson, REL-SE at SERVE
Center at UNCG
5
REGIONAL EDUCATION
LABORATORY- SOUTHEAST (REL-SE)
Operated by the SERVE Center at UNCG
Executive Director
Dr. Ludwig van Broekhuizen
Toll Free: 800-755-3277 www.serve.org
6
Serving Alabama, Florida, Georgia, Mississippi,
North Carolina, and South Carolina
Georgia liaison: Dr. Kim Anderson
kanderson@serve.org
404-657-6174
7
The Regional Educational Laboratory (REL)
System
8
REL-SE MISSION
To serve the educational needs of the
Southeast, using applied research,
development, dissemination, and
training and technical assistance, to
bring the latest and best research and
proven practices into school
improvement efforts.
9
REL-SE Services
1. Outreach and dissemination of research,
evaluation, and policy information
2. Technical assistance to SEAs and LEAs
3. “Issues & Answers” publications
4. Randomized Controlled Trial experimental
studies on interventions of relevance to our
region
•
AMSTI/Alabama and K-PAVE/Mississippi
10
Series Framework: EBDM Cycle
11
WEBINAR SERIES OVERVIEW
SESSION 1
SESSION 2
• Series
Introduction
• RTI
Overview
INTRODUCTION
SBR &
EBDM
• Understand
Scientifically
Based
Research
and the
EvidenceBased
DecisionMaking Cycle
SESSION 3
• Understand
methodologies
used in ed.
research, and
their
strengths and
weaknesses
for
determining
“what works”
TYPES OF
RESEARCH
METHODS
12
SESSION 4
CRITICAL
READING
PROTOCOL FOR
STUDIES ABOUT
INTERVENTIONS
• Understand
how to
assess the
rigor and
findings of
studies
• Practice
using the
Critical
Reading
Protocol
Webinar Series Overview
sessions 5 & 6
• IES
Practice
Guide RTI
Math
• IES
Practice
Guide RTI
Reading
STATE OF
RESEARCH
ON RTI
session 7
INTEVENTION
REVIEW
PROTOCOL
•Determine the
desired
outcomes,
program
features,
implementation
issues, and
extent of
evidence on an
intervention.
13
session 8
•Learn
what
Fidelity of
Implementation is
and how to
monitor it
effectively.
FIDELITY OF RTI
IMPLEMENTATION
session 9
PROMISING
PROGRESS:
LOOKING
FORWARD
• Learn from
colleagues
in GA how
they put the
concepts
and tools
from this
series into
action.
TODAY’S OBJECTIVES
1. Reach back to Session 3
Types of Research Methods handout posted on GaDOE website
2. Understand how to assess the rigor and findings of
studies on interventions
Critical Reading Protocol For Studies About Interventions handout
3. Practice using the Critical Reading Protocol for Studies
About Interventions
Carnegie Learning 2001 & Witzel Multisensory Algebra studies
4. Debrief the Protocol process and discuss the relevance of
this type of process for RTI decision making
5. Preview session #4
14
Here we go!
Objective #1
Reach back to Session #3:
Types of Research Methods
15
We discussed 5 types of research methods.
What do you remember?
What thoughts/questions remain?
2 min. Chat
Facilitators lead their groups
Individuals utilize chat box





Descriptive
Correlational
Quasi-Experimental
Experimental
Meta-Analysis
16
Quick Poll 1: Recall
Studies using which method can make
claims that the intervention studied caused
changes in student achievement?
A.
B.
C.
D.
E.
Descriptive
Correlational
Quasi-Experimental
Experimental
Meta-Analysis
17
Quick Poll 2: Recall
Studies using which method can discover
relationships between variables (e.g., a
particular instructional strategy and
academic achievement)?
A.
B.
C.
D.
E.
Descriptive
Correlational
Quasi-Experimental
Experimental
Meta-Analysis
18
Quick Poll 3: ID This Research Design
 Citation: Bennett, J., Lubben, F., & Hogarth, S. (2006).
Bringing science to life: A synthesis of the research
evidence on the effect of context-based and STS
approaches to science teaching. Science Education, 10,
347-370.
 Research Question: What is the average effect across
studies examining the impact of context-based and STS
approaches to science teaching?
 Findings: The 17 studies suggested that context-based
and STS approaches to science teaching improved
attitudes towards science as compared to control groups.
19
What type of research design do you
think the researchers used? Why?
A.
B.
C.
D.
E.
Descriptive
Correlational
Quasi-Experimental
Experimental
Meta-Analysis
20
Quick Poll 4: ID This Research Design
 Citation: Rainey, L., Harman, P., & McColskey, W. (1996).
A report of the integrated science student impact study at
the University of Alabama. Greensboro, NC: SERVE Center
 Research Question: Do students in the Integrated Science
Program (“ISP”) perform significantly better on performance
assessments as compared to students not enrolled in the
ISP?
 Methods: Students in the ISP and not in the ISP were
matched based on race, gender, and academic ability.
 Findings: In a test of science process skills, students
enrolled in the ISP performed significantly better than
students not in the ISP.
21
What type of research design do you
think the researchers used? Why?
A.
B.
C.
D.
E.
Descriptive
Correlational
Quasi-Experimental
Experimental
Meta-Analysis
22
Objective #2
Put our knowledge of research
methods together with other key
information to assess the rigor and
findings of studies on interventions
23
Critical Reading Protocol for Studies
About Interventions
 Looks at key elements of a research study’s
design and results in order to determine its
quality and rigor.
 This protocol can be used in the process of
compiling information about interventions and
strategies, and determining “what works.”
 This protocol is in the GaDOE RTI manual.
24
Critical Reading Protocol for Studies
About Interventions
See Protocol handout
1. Authorship: Who is the author? Is it the
vendor or a third party? Who funded the
study? Is there an obvious bias?
2. Sample: What was the sample? How
closely do the participants in the study
mirror your population?
25
Critical Reading Protocol for Studies
About Interventions
3. Research design: What kind of design
did the researchers use? What kind of
questions can this design answer (and
not answer)? How were the groups
formed?
*use your Types of Research Methods
handout
26
Critical Reading Protocol for Studies
About Interventions
4. Results: What kinds of outcomes were
reported on? Are the results
statistically significant? Practically
significant?
5. If you were to implement this
intervention what outcomes could you
reasonably expect to achieve?
27
Objective #3
Practice using the Critical Reading
Protocol for Studies About Interventions
 Participants read sample program evaluation
publications and utilize Protocol to determine
the quality and rigor of the study
 Then, whole group discussion
28
Sample Publication #1:
Carnegie Learning 2001 Cognitive Tutor
 1 min. introduction to this publication
 10 min. read and answer Protocol questions
(in small facilitated groups and/or in chat box)
 10 min. whole group discussion: What did we
learn about this research study and its
application for our RTI decision making?
29
POLL: Protocol Question #1
Authorship: Who is the author? Is it the vendor or
a third party? Who funded the study? Is there an
obvious bias?
A. The author is/has a relationship to the vendor,
and as such may be biased
B. The author is a third party, and as such should
be neutral
C. Cannot determine
D. We have a question about this…?
30
POLL: Protocol Question #2
Sample: What was the sample? How closely do
the participants in the study mirror your
population?
A. The participants closely mirror our population
B. The participants are not similar to our
population
C. Cannot determine
D. We have a question about this…?
31
POLL: Protocol Question #3
Research design: What kind of design did the
researchers use? What kind of questions can
this design answer (and not answer)? How
were the groups formed?
A.
B.
C.
D.
The design was quasi-experimental
The design was correlational
The design was experimental
Cannot determine or we have a question about
this…?
32
POLL: Protocol Question #4
Results part 1: What kinds of outcomes were
reported on?
A.
B.
C.
D.
Results were Texas End of Course Exam
Results were local assessments
Cannot determine
We have a question about this…?
33
POLL: Protocol Question #4
Results part 2: Are the results statistically
significant? Practically significant?
A. Results were statistically significant
B. Results were statistically AND practically
significant (in our view, the difference between
scores of C.T. students and comparison group
students would justify C.T.’s cost, for us)
C. Cannot determine
D. We have a question about this…?
34
POLL: Protocol Question #5
If you were to implement this intervention what
outcomes could you reasonably expect to
achieve? Why?
A.
B.
C.
D.
We would expect to achieve similar results
We would not expect to achieve similar results
Cannot determine
We have a question about this…?
35
Sample Publication #2:
Witzel 2005 Multisensory Algebra
 1 min. introduction to this publication
 10 min. read and answer Protocol questions
(in small facilitated groups and/or in chat box)
 10 min. whole group discussion: What did we
learn about this research study and its
application for our RTI decision making?
36
POLL: Protocol Question #1
Authorship: Who is the author? Is it the vendor or
a third party? Who funded the study? Is there an
obvious bias?
A. The author is/has a relationship to the vendor,
and as such may be biased
B. The author is a third party, and as such should
be neutral
C. Cannot determine
D. We have a question about this…?
37
POLL: Protocol Question #2
Sample: What was the sample? How closely do
the participants in the study mirror your
population?
A. The participants closely mirror our population
B. The participants are not similar to our
population
C. Cannot determine
D. We have a question about this…?
38
POLL: Protocol Question #3
Research design: What kind of design did the
researchers use? What kind of questions can
this design answer (and not answer)? How
were the groups formed?
A.
B.
C.
D.
The design was quasi-experimental
The design was experimental
The design was descriptive
Cannot determine or we have a question about
this…?
39
POLL: Protocol Question #4
Results part 1: What kinds of outcomes were
reported on?
A. Results were SC’s “PACT” test scores
B. Results were scores on a local assessment
designed specifically for use in the study
C. Cannot determine
D. We have a question about this…?
40
POLL: Protocol Question #4
Results part 2: Are the results statistically
significant? Practically significant?
A. Results were statistically significant
B. Results were statistically AND practically
significant (in our view, the amount of difference
between scores of M.A. students and
comparison would warrant M.A.’s cost, for us)
C. Cannot determine
D. We have a question about this…?
41
POLL: Protocol Question #5
If you were to implement this intervention what
outcomes could you reasonably expect to
achieve?
A.
B.
C.
D.
We would expect to achieve similar results
We would not expect to achieve similar results
Cannot determine
We have a question about this…?
42
OBJECTIVE #4
Debrief the Critical Reading Protocol
process, and
Discuss the relevance of this type of
process as we consider
purchasing/creating interventions
43
Debrief & Discuss
3 min. Chat
Facilitators lead their groups
Individuals utilize chat box
1. What do we think about the Protocol?
2. How does this Protocol process compare to how we
normally think about the research behind
interventions we are considering?
3. What is the value (or not) of this type of process as
we consider purchasing/creating interventions?
44
Objective #5
Preview session #4
October 28, 2009 3-4:30pm
State of Research on RTI Math:
What do we know? What do we not know?
Dr. Sybilla Beckmann, UGA
45
Webinar Series Overview
sessions 5 & 6 session 7
• IES
Practice
Guide
RTI Math
• IES
Practice
Guide
RTI
Reading
STATE OF
RESEARCH
ON RTI
INTEVENTION
REVIEW
PROTOCOL
• Determine the
desired
outcomes,
program
features,
implementation issues,
and extent of
evidence on
an
intervention.
session 8
•Learn
what
Fidelity of
Implementa
-tion is and
how to
monitor it
effectively.
FIDELITY OF RTI
IMPLEMENTATION
46
session 9
PROMISING
PROGRESS:
LOOKING
FORWARD
• Learn from
colleagues
in GA how
they put the
concepts
and tools
from this
series into
action.
http://ies.ed.gov/ncee/wwc/publications/practiceguides/
47
Accessing session #5
materials and webinar
 John Wight
48
Thank you for being here today!
Dr. Kim Anderson
– kanderson@serve.org
– 404-657-6174
The SERVE Center/REL-SE
– 800-755-3277
– www.serve.org
John Wight, GaDOE
 jwight@doe.k12.ga.us
 404-656-0478
49