Whole Day First Grade Program:
Effectiveness, Sustaining, &
Building a Base for Going to Scale
Jeanne Poduska, ScD
Center for Integrating Education and
Prevention Research in Schools,
American Institutes for Research
Presented at: New Directions for
Mental Health and Drug Abuse Effectiveness
and Dissemination Trial Research and Methodology
October 28, 2005
Contributors and Support
Sheppard G. Kellam, M.D. 1
Jeanne Poduska, Sc.D. 1
C. Hendricks Brown, Ph.D. 2
Carla Ford, Ph.D. 1
Amy Windham, Ph.D. 1
Natalie L. Keegan 1
John Reid, Ph.D. 3
Bonnie Copeland, Ph.D. 4
Linda Chinnia, 4
National Institute on Drug Abuse,
Division of Epidemiology, Services and Prevention Research,
Prevention Research Branch
1R01 DA015409 and 1R01 DA019984 (and support from NICHD)
1
American Institutes for Research,
Center for Integrating Education and Prevention Research in Schools
2 University of South Florida, Prevention Science Methodology Group
3 Oregon Social Learning Center
4 Baltimore City Public School System
Phases of Education and
Prevention Trials
Sustaining
Sustaining
Systemwide
Systemwide
Going to Scale
Going-to-Scale
Sustainability
Sustainability
Effectiveness
Effectiveness
Efficacy
Efficacy
Developmental Epidemiology:
A Strategy for Prevention Research
Community Epidemiology
Life Course Development
Preventive Interventions
Research roots of WD
• First Generation of Baltimore trials tested separately
Mastery Learning (ML) aimed at reading, and Good
Behavior Game (GBG) aimed at classroom management
of student behavior in First Grade.
– Each had short-term impact on its target; GBG had long
term impact on very high risk children.
• Second Generation tested combination of instructional
improvement and GBG and showed impact on reading
and behavior into middle school, still room for
improvement.
– A family/classroom program also seemed promising.
• Oregon Prevention Research Center tested family/
classroom partnership (LIFT program) and found impact
among higher risk children.
Add Outcome Slide
How do you Establish Community &
Institutional Partnerships?
1. Analyze social/political structure
2. Learn the vision of problems and priorities
3. Identify mutual self-interest within and across leadership
4. Fit your interest under the vision of leadership
5. Request ad-hoc oversight committee of leaders
6. Work through trust
This process takes about a year
Functions of a
Community & Institutional Board
•
Establish vision and priorities
•
Examine/critique/approve/support proposed programs
•
Communicate constituents’ concerns, values, priorities, and
acceptable language to program leaders
•
Communicate program needs to constituent leaders
•
Continually assess absent constituencies
•
Work toward institutionalizing programs
Baltimore City Public
School System Partners
Patricia Welch, Ph.D., Chair, BCPSS Board of School Commissioners
Bonnie Copeland, Ph.D., Chief Executive Officer
Linda Chinnia, Chief Academic Officer
Ben Feldman, Ph.D., Research, Evaluation & Accountability Officer
Sue Cutter, Area 1 Interim Academic Officer
James Smith, Area 2 Academic Officer
Gwen Cleage, Curriculum and Instruction Officer
La Verne Sykes, Director, Parent and Community Involvement
Deborah Wortham, Ph.D., Director of Professional Development
Delores Alexander, Coordinator of Reading
Marietta English, President, Baltimore Teachers Union
James Gittings, Public School Supervisors and Administrators Association
20 Schools, Principals, and Teachers Areas 1 & 2
Three Foci of Baltimore Whole Day
First Grade Program (WD)
• Improve reading instruction in all academic
subjects thru the day, based on prior
Baltimore trials.
• Improve classroom behavior management,
based on prior Baltimore trials.
• Improve parent/classroom partnership re
reading and behavior, based on prior
Baltimore and Eugene, Oregon trials.
Lessons Learned From First Two
Generations of Trials in Baltimore
Without a system to mentor, model, and
monitor teacher practices over time,
practices are not sustained.
A multi-level institutional structure with
mentoring and monitoring is required to
implement interventions with fidelity.
Teachers need support from principals;
principals from area leaders; area leaders
from chief academic and executive
officers.
Goals of WD Trial
• Can we find synergy from the combination?
• Can the practices be sustained over time?
• Can practices be scaled up by school system
with same effect?
Baltimore Whole Day First Grade
Program Analytic Model
Classroom
Behavior
Management
Teachers’ Effective
Academic
Instruction
Effective
Family-Classroom
Partnerships
Whole Whole
Day First
Grade
Day
Education and
Prevention Program
Decreased
Aggressive,
Disruptive
Classrooms
Classrooms
Poor
Improved
Reading
Skills
Reading Skills
Decreased Later
Substance Abuse
Decreased
Individual
Individual
Aggression
Aggression
Increased
Poor
Achievement
Achievement
Decreased
Depressive
Depressive
Symptoms
Symptoms
Other mediating or moderating variables:
• Family structure and poverty
• Deviant peers
•
• School building
•
• Community economic health,
•
resources, drugs, and violence
Decreased
Later Conduct & Antisocial Personality
Disorders
School Success
& Decreased
Drop-Out
Decreased Later
Depressive
Disorders
Multi-Level Structure to Sustain
Practices and Go to Scale
BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioners
Chief Executive Officer
Chief Academic Officer
Professional
Development
Officer
Curriculum &
Instruction
Officer
Research and
Evaluation Office
Area
Academic
Officers
Principals
Teachers
Lead
Coaches
Coaches
Core Team
BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioner
Chief Executive Officer
Chief Academic Officer
Professional
Development
Officer
Curriculum &
Instruction
Officer
Research and
Evaluation Office
Area
Academic
Officers
Principals
Teachers
Lead
Coaches
Coaches
Schools Committee
BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioner
Chief Executive Officer
Chief Academic Officer
Professional
Development
Officer
Curriculum &
Instruction
Officer
Research and
Evaluation Office
Area
Academic
Officers
Principals
Teachers
Lead
Coaches
Coaches
Within-school Random Assignment* of
Teachers/Classrooms and Children in the
Whole Day First Grade Program
IN EACH SCHOOL:
Random
Teachers randomly
assigned to
classrooms
Children randomly
assigned to
classrooms
*Brown & Liao. AJCP, 1999.
SC (a)
SC (b)
WD
Random
Effectiveness Design of the
Whole Day First Grade Program:
Follow 2 Cohorts of 1st Graders to 3rd Grade
2003-04
2004-05
2005-06
12
Teachers
WD
Cohort 1 Grade 1
Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
12
Teachers
SC
Cohort 1 Grade 1
Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
Remaining
Teachers
Sustainability Design of the
Whole Day First Grade Program:
Follow Teachers over Cohorts of 1st Graders
2003-04
12
Teachers
WD
2004-05
2005-06
Cohort 1 Grade 1
Cohort 2 Grade 1
Cohort 3 Grade 1
12
Teachers
SC
Remaining
Teachers
Cohort 1 Grade 1
Cohort 2 Grade 1
Scaling-Up Design of the
Whole Day First Grade Program:
3 Cohorts of 1st Graders
2003-04
12
Teachers
WD
2004-05
2005-06
Cohort 1 Grade 1
Cohort 2 Grade 1
Cohort 3 Grade 1
12
Teachers
SC
Trained in
WD
Remaining
Teachers
Trained in
WD
Cohort 1 Grade 1
Cohort 2 Grade 1
Cohort 3 Grade 1
Cohort 3 Grade 1
Design of the
Whole Day First Grade Program
2003-04
2004-05
2005-06
12
Teachers
WD
Effectiveness
Effectiveness
Sustainability
Effectiveness  
Sustainability
Cohort 1 Grade 1
Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
Cohort 3 Grade 1
12
Teachers
SC
Effectiveness
Effectiveness
Sustainability
Effectiveness  
Scaling Up (to WD)
Cohort 1 Grade 1
Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
Cohort 3 Grade 1
Remaining
Teachers
Scaling Up (to WD)
Cohort 3 Grade 1
Primary Measurement Objectives
in the WD Trial
•
Implementation
•
Proximal effects of intervention on student
behavior, achievement, and psychological wellbeing
•
Longer term (distal) effects
•
Teacher practices
•
Hypothesized or known mediating and moderating
variables
•
Measures of the multi-level structure
Fidelity of Whole Day First Grade Program
Implementation Checklist Excerpt
CLASSROOM BEHAVIOR MANAGEMENT INDICATORS
QUALITY
RATING
20. Instructs students on class
rules by:
YN
 Announcing the game and
reviewing rules.
YN
 Setting and adhering to time
limits for the game.
YN
 Providing appropriate
independent assignments
during the game.
YN
Classroom Behavior Management Global Rating:
Global Rating is not necessarily the sum of individual quality ratings.
1
2
3
4
5
6
1
2
3
4
5
6
Measures of Effectiveness
Social Adaptational Status
OUTCOME
MODE
ACADEMIC ACHIEVT
Student interview
MEASURE
Woodcock Johnson
Houston Word Reading Scales
Teacher Interview
Teacher Observation of Classroom
Adaptation (TOCA)
Record review
School records, Standardized scores
Time on-task/off-task
Classroom Obs.
Teacher Observation/Student
Engagement (TOSE) - Foorman
Aggressive behavior
Classroom Obs.
Teacher Interview
TOSE
TOCA
Disruptive behavior
Classroom Obs.
Teacher Interview
TOSE
TOCA
Concentration prob’s
Classroom Obs.
Teacher Interview
TOSE
TOCA
Social Isolation
Classroom Obs.
Teacher Interview
TOSE
TOCA
BEHAVIOR
Measures of Effectiveness
Psychological Well-being
OUTCOME
MODE
MEASURE
Depression
Student interview Direct Clinical Observation
Baltimore How I Feel
Anxiety
Student interview Direct Clinical Observation
Baltimore How I Feel
Flat affect
Student interview Direct Clinical Observation
Hyperkinesis
Student interview Direct Clinical Observation
Bizarre behavior
Student interview Direct Clinical Observation
Teacher Practices
Observations by Independent Observers
(3x per year)
• Teacher Practices
• Timed Observation of Instruction
(Foorman)
• Global ratings
Measures of Multi-Level
System to Support Sustaining
Practices and Scaling-Up
• Defined expectations for each level of the
multi-level structure along four
organizational variables: power/structure,
tasks, resources, human factor
• Attendance logs and observers’ ratings of
occurrence and quality for expectations
• 1 Core Team; 2 AAOs; 12 school teams
Monitoring and Mentoring
Feedback Loop
Monitoring
Data from
Monitoring
Teacher
Practices
Effective
Mentoring
Chief Academic
Officer
Core Team
Area Academic Officers &
Lead Coaches
School Coaches / Principals
Mentoring
Fidelity of Whole Day First Grade Program
Implementation Checklist Excerpt
CLASSROOM BEHAVIOR MANAGEMENT INDICATORS
QUALITY
RATING
22. Plays the Good Behavior Game by:
Y N
 Dividing students into balanced teams.
Y N
 Reviewing class rules and announcing
the game.
Y N
 Setting and adhering to time limits for
the game.
 Placing checks on the board when
individual students break rules.
 Providing rewards to winning teams.
Classroom Behavior Management Global Rating:
Global Rating is not necessarily the sum of individual quality ratings.
Y
Y
Y
1
2
3
4
5
6
1
2
3
4
5
6
N
N
N
Level of Implementation
Patterns of Quality of GBG
Implementation in
12 Intervention Schools: 2004-05
High
4-6
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
11
18
25
36
Low
1-3
Week of Implementation
Level of Implementation
Patterns of Quality of GBG
Implementation in
12 Intervention Schools: 2004-05
High
4-6
Low
1-3
●●
●●●
●●●●
●●
●
11
18
25
Week of Implementation
●●●●
36
Level of Implementation
Patterns of Quality of GBG
Implementation in
12 Intervention Schools: 2004-05
High
4-6
Low
1-3
■■
■■
11
18
■
■
■■
25
36
Week of Implementation
Level of Implementation
Patterns of Quality of GBG
Implementation in
12 Intervention Schools: 2004-05
High
4-6
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
▲▲▲
●●
■■
●●●
■
●●●●
Low
1-3
●●●●
■■
●●
●
■
■■
11
18
25
36
Week of Implementation
Professional Development Model as
Moving to Scaling Up
• AIR (Carla Ford) transitioning from
classroom/school based focus to multi-level
professional development
• Carla monitoring coaches; coaches
monitoring teachers
• 3 Area Lead Coaches being trained fall 2005;
train teachers in spring 2006
• Developing monitoring system
Summary—
Prevention and Service Researchers
Researcher/Community Institution Partnerships
Defining intervention and practices with precision
Defining and measuring multi-level structure for
sustaining and scaling-up practices
Continual monitoring of practices deliberately
linked to professional development
Dual design requires equal emphasis on
measurement of effectiveness and going to
scale
Summary—
Statisticians and Methodologists
• Modeling mediation of teacher practices
• Relationship of observational data on teacher
practices and checklist of teacher practices:
thick to thin data
• Modeling multi-level structure data
• Help us prepare for missing data in follow-up!