What the Data Tell Us
CAPSES
October 2010
Jack States
Randy Keyworth
The Wing Institute
What Evidence Should We
Trust?
The Fallibility of Professional Judgment:
a false sense of accuracy
CRITERIA FOR
TREATMENT CHOICES
CLIENT
PHYSICIAN
77%
22%
Your demonstrated track
record of success based on
data you have gathered
systematically and regularly
39%
92%
Results of controlled
experimental Studies
37%
92%
Your intuition (gut feeling)
about what will be
effective
Gambrill and Gibbs, 2002
The Fallibility of Professional Judgment:
errors in reasoning
Common errors in reasoning that can effect perceptions and decisions.
Circular Reasoning
Shifting the Burden of Proof
Analogy
Non-Sequitur
Self-Referential Fallacy
Humor
Post Hoc
Ad Hominem
Extrapolation
Red Herring
Sidestepping/Avoiding the Question
Circumstantial
Equivocation
Suppressed (Stacking the Deck)
Guilt by Association
False Dichotomy
Statistics
Best-in-Field Fallacy
Lying
Jumping to Conclusions
Authority
Traditional Wisdom
Non-Experimental
Qualitative Research
QUALITATIVE CASE STUDY RESEARCH COMPONENTS AND STRUCTURE
OF A CURRICULUM-BASED MENTORING PROGRAM AT THE MIDDLE
SCHOOL LEVEL, GOLIGHTLY, M (1996)
Outcomes of the Study
This study has been conducted to discover the components and structure of a curriculum-based
mentoring program at the middle school level. A second purpose of the study was to discover common
traits and characteristics eighth grade mentoring students exhibit. A major outcome of this study is that
it may lead to developments in curriculum, which include mentoring programs to build strong
leadership qualities in young people. A curriculum including mentoring could successfully employ older
students to serve as positive role models for younger children. Middle school mentors would have
valuable opportunities to serve as leaders, while further building character in themselves. Younger
students in a peer mentoring program would benefit by spending time with a caring guide who shows
concern for their well being. Younger students could benefit as well, academically, as older mentors
would be available to offer them assistance and support. These mentors can fill the role of the missing
role model and caregiver for younger students who need positive examples and guidance.
From the findings in this research study, several important ideas have evolved which could help
educators interested in school based mentoring programs. Findings in this study have revealed traits
the mentoring students feel are important in service as positive role models. These traits include
responsibility, helpfulness, caring, respect for self and others, and a sense of ethics. The mentoring
program in this study was shown effective in continuing the development of these qualities in
participating students. Through findings in this study, educators will have the design and structure
needed to implement such a program within an existing curriculum. Through student participation in
this type of mentoring program, students will find opportunities to serve in their school and community
while continuing to develop effective traits such as responsibility, helpfulness, and respect.
Correlational Design
Single Subject Design
Bob - Out of Seat
18
Frequency of out of Seat
16
14
12
10
8
6
4
2
0
A
B
A
Baseline
Intervention
Baseline
B
Intervention
Randomized Control Trial (RCT)
An experimental design used to establish a
cause and effect relationship.
In an RCT the investigators randomly
assign eligible subjects into groups to
receive or not receive one or more
interventions that are being compared.
How to Interpret Results:
Effect Size
Cohen’s d
Effect Size
Small
d = .2
Medium
d = .5
Large
d = .8
Meta-analysis
• A literature review that establishes a single
effect
• Offers a quantitative alternative of the
magnitude of the effect – Effect Size
Examples of Meta-analyses
Common Educational Practice
(Hattie, 2009)
1
0.9
0.8
0.88
0.77
Large .8
0.74
0.73
Effect Size
0.6
Medium .5
0.4
Small .2
0.2
0.06
0.04
0.01
0
-0.2
-0.4
-0.6
-0.16
-0.34
Continua of Evidence
Quantity of the Evidence
Meta-analysis
(systematic review)
Repeated Systematic
Measures
Single Case Replication
(Direct and Parametric)
Threshold
of
Convergent Evidence
Evidence
Various Investigations
Quality of the Evidence
Current “Gold Standard”
High Quality
Randomized Controlled Trial
Single Case Designs
Semi-Randomized Trials
Well-conducted
Clinical Studies
Uncontrolled Studies
Expert Opinion
General Consensus
Single Study
Personal Observation
Janet Twyman, 2007
What is the Current State of
Education Research?
Insufficient number of rigorous
studies
• What Works Clearinghouse
• Campbell Collaborative
• American Educational Research
Association
What is the Current State of
Education Research?
Outcome Measures Used to Evaluate
Four vs Five Year Preparation Programs
4
# of Studies
3
3
3
3
3
2
2
1
1
1
No studies
examined
teaching
outcomes
0
0
Ratings
Preparation
Program
Ratings of
Teachers
Satisfaction
Academic
Teachers
Qualifications Enter Field
Retention
Effectiveness Criteria
0
Teacher
Direct
Student
Knowledge Observation Achievement
Less than 1% offer
qualify as rigorous
SOURCE: SRI (2004)
Field Experience Research
Research Designs
Assessment Methods
Number of Studies
50
45
45
40
35
30
25
22.5
20
17.5
15
12
10
7
5
0
3
0
Research Design Methods
AERA Report : Studying Teacher Education, 2005
What is the Focus of Most
Reform Effects?
•Home
•Staff Structural
•Student
Reforms
•Systems
School Size
Elementary School Size
United States
490
480
Average Enrollment
470
460
450
440
430
420
410
400
1983 1984 1985 1988 1989 1990 1991 1992 1993 1994
1995 Years
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
School
Class Size
Effect Sizes for Class-Size Reduction Star Project
(Randomized Trial)
0.4
0.35
Effect Sizes
0.3
0.25
Reading
0.2
Math
0.15
0.1
0.05
0
Kindergarten
1st Grade
2nd Grade
3rd Grade
Little Gain at an
Annual Cost of
More Than $1.5
Billion
How Effective are Charter Schools
Traditional
Public vs.
CharterComparison
School Growth
As Percentage
of Charter
Total Schools
(NAEP
Scores)
in the U.S.
0.5
5.0%
0.45
Percentage of Total
Effect Size
0.4
4.0%
0.35
0.3
3.0%
0.25
.2 = small effect size
0.2
2.0%
0.15
Public Schools minimally
out-perform Charter
Schools
There are more
than 5,000 Charter
schools in the
United 0.17
States
0.11
0.1
1.0%
0.05
0.0%0
1999-00 2000-01 Reading
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07
Math 2007-08 2008-09
What We Know
- No Child Left Behind mandates full state certification
- In 2004-05 approximately 1 in 7 teachers did not meet standard
Difference in Student Scores for Alternative Credential
Compared to Traditional Credential Teachers
0.6
0.5
This study found
no benefit to
student
achievement
Effect Size
0.4
0.3
0.2
0.1
0
0.01
Reading
-0.03
Math
-0.1
Constantine, J., Player D., Silva, T., Hallgren, K., Grider, M., and Deke, J. (2009). An Evaluation of Teachers Trained Through Different
Routes to Certification, Final Report (NCEE 2009- 4043). Washington, DC: National Center for Education Evaluation and Regional
Assistance, Institute of Education Sciences, U.S. Department of Education.
What We Know
82,000 teachers are NBPTS certified nationwide
0.25
Comparison of NBPTS Certified Teachers to Non-certified
Teachers on Student's Math Achievement
Effect Size
0.2
National Certification
Has Minimal Impact on
Improving Student
Achievement
0.15
0.1
0.07
0.05
0.07
0.05
0.04
0.025
0
-0.01
-0.05
Goldhaber et
al. (2005)
Cavaluzzo
(2004)
McColskey et Clotfelter et Harris & Sass Sanders et al.
al. (2005)
al. (2006)
(2006)
(2005)
Studies
National Board Certification and Teacher Effectiveness: Evidence from A Randomized Assigned Experiment, December 2008
What We Know
No rigorous research is available regarding achievement
and National Council for the Accreditation of Teacher
Education (NCATE)
Comparison of NCATE Trained Teachers vs. Non-NCATE
Teachers Passing the PRAXIS II (1995-97)
% Graduates Pass the PRAXIS II
92%
91%
Rigorous
Research on
Student
Achievement
Absent
90%
88%
86%
84%
84%
82%
80%
NCATE Institution
Non-NCATE Institution
What We Know
The impact of structural interventions has been disappointing
Cost Benefit Analysis of Common Interventions
Effectiveness Cost Ratio
0.016
0.014
0.0143242
0.012
Yeh Calculations
Wing Institute Calculation
0.01
0.008
0.006
0.004
0.002
0.00025286
0
Rapid
assessment
(2 to 5 X/
week)
0.00007419
Accountability 10% Increase
(High-Stakes in spending
Testing)
0.00000591
0.00000062
0.000214
Vouchers
Charter
schools
Class-size
Interventions
(Effectiveness Cost Ratio = Effect Size/Cost Per Student)
Yeh, S. S. (2007). The cost-effectiveness of five policies for improving student achievement. American Journal of Evaluation,
28(4), 416-436.
What Works?
Effective Classroom
Instruction
What We Know
Teachers are very important: Effect Size .20 - .40
7% - 21% of student gains can be attributed to teacher effectiveness
(Nye et al., 2004)
Teacher Effectiveness – Gains in 8th grade math
90%
83%
80%
Percent Proficient
70%
Source: Sanders and Rivers 1996
60%
50%
40%
30%
29%
20%
10%
0%
3 Years Ineffective Teacher
3 Years Effective Teacher
What We Know About Teaching
Training Method that Produces the Best Results
100%
Percentage of Participants
90%
80%
1. Theory & Discussion
70%
60%
2. Plus Demonstration
50%
40%
3. Plus Practice & Feedback
30%
4. Plus Coaching
20%
10%
0%
Knowledge
Skill
Demonstration
Use in
Classroom
Degree of Skill Competency
Student Achievement Through Staff Development, Joyce & Showers, 2002
What Makes a Difference
M. C. Wang, G. D. Haertel, and H. J. Walberg, 1994
What We Know - About Assessment
Size Effect of Formative Evaluation Methods
(Progress Monitoring) on Student Achievement
1
0.9
0.8
0.91
Large
0.7
Effect Size
0.7
0.6
0.5
0.4
0.3
0.26
0.2
0.1
0
Data Collected
Data Graphed
Data Analyzed
Following Rules
Effects of Systematic Formative Evaluation: A Meta-Analysis, Fuchs & Fuchs, 1986
The Impact of What Happens in the Classroom
What Do We Know About Reading
The Impact of What Happens in the Classroom
Examples of Classroom Variables
Curricula
0.8
0.73
0.67
0.7
0.6
0.6
Medium
0.52
0.5
0.43
0.45
0.4
0.4
0.3
0.06
0.08
Perceptual Motor
0.1
Whole Language
0.2
Vocabulary
Phonics
Science
Math
Writing
Feedback
Teaching
Strategies
0
Classroom
Management
Effect Size (d)
0.6
Visible Learning, Hattie, J (2009)
What Do We
Know About
Teacher Preparation?
What We Know
There are no data to know how widely coaching is employed
Field Experience Research Topics
90
84
Number of Studies
80
70
The principal
measure was
“Attitude Change”
and only 3 out of 107
were experimental
60
50
40
30
20
10
10
2
3
3
Coaching
Student
Achievement
Knowledge
5
0
Methods
Subject Matter
Implementation Implementation
Study Outcome
AERA Report : Studying Teacher Education, 2005
Attitude
Change
What We Know
- No Child Left Behind mandates subject matter competency
- Subject matter effect size: .06 - .12
Teacher Subject Matter Training and Student Achievement
Strength of Evidence
Strong
Minimal
2
Subject
matter impact
negligible
1
Insignificant
0
Math
(Secondary)
Math
(Elementary)
Science
Reading
Arts
Other
Subject Matter
Literature Review
Studying Teacher Education: The Report of the AERA Panel on Research and Teacher Education (2005)
Creating Effective Teachers: Concise Answers for Hard Questions (2003)
What Schools Are Teaching
Formative Assessment in Connecticut Preparation
Programs
16%
% of Course Syllabi
14%
12%
Sample of 13
Elementary Teacher Programs
10%
8%
14%
6%
4%
2%
0%
0%
Progress Monitoring, Formative
Assessment, or Ongoing Assessment
Dibels
Response to Intervention and Teacher Preparation, Spear-Swerling, 2008
How Are Teaching Candidates Assessed?
Methods for Assessing Teachers in Connecticut Survey
60%
% of Syllabi
50%
no examples
based on
achievement
50%
46%
40%
32%
30%
30%
21%
20%
10%
0%
Tests and quizes
Reflective
Running record
learning journal
Observation of
child or lesson
Planning a unit
Response to Intervention and Teacher Preparation, Spear-Swerling, 2008
In Conclusion
• Research is a guide to what works
• Review research with a critical eye
• Select interventions that work with your
population
• Avoid the quick fix
• Select interventions that are cost
effective
• Implement interventions as designed