Presentation at National Early Childhood Inclusion Institute
Chapel Hill, NC
May 21, 2014
Patricia Snyder, PhD
Professor and David
Lawrence Jr. Endowed
Chair in Early Childhood
Studies
University of Florida
Mary Louise Hemmeter, PhD
Professor
Department of Special
Education
Vanderbilt University
Moderator: Pamela J. Winton, PhD
Senior Scientist & Director of Outreach
FPG Child Development Institute
Elena Soukakou, PhD
Senior Lecturer
University of Roehampton,
London, UK
 Welcome
and introduction
 Overview of three measures
 Question & answer after each
measure
 Facilitated discussion
Teaching Pyramid Observation Tool (TPOT)
(Fox, Hemmeter, & Snyder, 2013)
Available from Paul Brookes Publishing
Embedded Instruction Observation System --Teacher Version
(EIOS-T)
(Crowe, Snyder, Crow, Mullin, & Embedded Instruction Project, 2011)
Available from the authors
Inclusive Classroom Profile (ICP)
(Soukakou, 2012).
Available from the author

Copyright © 2013 by Elena P. Soukakou
Dr. Elena Soukakou, Author

Measures inclusive, classroom-level practices
that support the individualized needs of
children with disabilities
 Structured
observation measure
 1–7 point rating scale
 12 items
Young children
with disabilities
can experience
low quality in
classes that are
otherwise rated
as being of
high quality
Wolery, et al., 2000
National Professional Development
Center on Inclusion
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Adaptation of space and materials
Adult involvement in peer interactions
Adult guidance of children’s activities and play
Conflict resolution
Membership
Relationships between adults and children
Support for social communication
Adaptation of group activities
Transitions between activities
Feedback
Family-professional partnerships
Monitoring children’s learning
 As
a research tool
 As a classroom evaluation tool
 As a professional development
tool
National Professional Development Center on Inclusion

Children with identified disabilities
in the context of classroom
activities and social interactions
with adults and peers

Teachers, teacher assistants,
specialists
National Professional Development Center on Inclusion

Observation

Teacher interview

Document review
National Professional Development Center on Inclusion
Exploratory research
2. Conceptualization and domain delineation
3. Item generation
4. Expert review
5. Pilot study in the UK
See Soukakou, E. P. (2012). Measuring quality in inclusive preschool
1.
classrooms: Development and validation of the Inclusive Classroom Profile
(ICP). Early Childhood Research Quarterly, 27(3), 478-488)
 1st pilot study in the UK showed promising
results on reliability & validity (Soukakou, 2012)
 2nd pilot study in the US in collaboration
with:
NC Department of
Instruction,
Exceptional Children
National Professional Development Center on Inclusion
 Did assessors learn to use the ICP with
accuracy?
 What is the evidence for reliability and
validity?
 Did assessors find the ICP useful and
acceptable for program evaluation?
National Professional Development Center on Inclusion
 51 inclusive classrooms in one state
 Public Pre-K (5),
Head Start (13),
Developmental Day programs (13),
Other child care centers (20)
 150 children with disabilities
 Mean age of children= 4.43 years
National Professional Development Center on Inclusion
 51 ICP assessments
 50 ECERS-R assessments
 Assessor survey for gathering
data on ICP acceptability
National Professional Development Center on Inclusion
Results: Inter-Rater Reliability
ITEM
ICP 1 Adaptation of Space, Materials and Equipment
ICP 2 Adult Involvement in Peer Interactions
ICP 3 Adult Guidance of Children’s Play
ICP 4 Conflict Resolution
ICP 5 Membership
ICP 6 Relationships between Adults and Children
ICP 7 Support for Communication
ICP 8 Adaptations of Group Activities
ICP 9 Transitions between Activities
ICP 10 Feedback
ICP 11 Family-Professional Partnerships
ICP 12 Monitoring Children’s Learning
ICC
.62
.78
.11
.70
.84
.75
.51
.72
.95
.60
.99
.99
Results: Rank-Order Correlations
Between ICP and ECERS
ECERS-R Scale
Space and Furnishings
Personal Care
Language and Reasoning
Program Structure
Activities
Interactions
Parent and Staff
ECERS Total Score
Note: *p<.05, **p<.01, ***p<.001
ICP Total Score
0.48***
0.21**
0.47***
0.29*
0.30*
0.38**
0.38**
0.48***
Mean(SE)/B(SE)
Child Care
3.67 (0.15)a
Developmental Day
5.12 (0.19)b
Head Start
4.64 (0.19)b
Public Pre-K
4.76 (0.30)b
Note: Means not sharing superscripts are significantly different.
On a 1–5 point scale, 4 assessors:
 Rated the importance of the ICP constructs measured
very highly (m= 5)
 Would highly recommend the ICP measure to others
(m=5)
 Found the measure easy to administer (m= 4)
 Felt well prepared after the reliability training
observations (m=4)
National Professional Development Center on Inclusion
 Assessors established adequate
administration and reliability proficiency
upon training.
 Evidence for construct validity.
 Differences in quality across types of
programs
 Assessors found the ICP easy to use and
useful for program evaluation
National Professional Development Center on Inclusion

Training program for users. Online overview
materials at
http://npdci.fpg.unc.edu/measuring-qualityinclusion-inclusive-classroom-profile

Professional development curriculum for PD
providers/consultants
National Professional Development
Center on Inclusion
ICP training available
at PDC@FPG
PDC@FPG
http://pdc.fpg.unc.edu
Putting Knowledge to Work
Mary Louise Hemmeter
Lise Fox
Patricia Snyder
The Pyramid Model:
Promoting Social and Emotional Competence and Addressing
Challenging Behavior
Tertiary
Intervention:
Few Children
Secondary
Prevention:
Some Children
Universal
Promotion:
All Children
27
TPOT Score Sheet
& TPOT Manual
TPOT Sample Items
http://products.brookespublishing.com/Teaching-Pyramid-Observation-Tool-TPOT-for-Preschool-Classrooms-Set-Research-Edition-P727.aspx
28


The TPOT was developed to measure the fidelity with which
teachers implement Pyramid Model practices
Provides information that can be used to:
 Describe “quality” of implementation of TPOT practices
 Compare implementation within and across
teachers/classrooms
 Support program-wide implementation and improvement
activities
 Identify needs of teachers for training and implementation
support

Observations
 Conducted for
a minimum of 2 hours
 Must observe centers or free play, at least one teacherdirected activity, and the transitions between activities
 Focus of observation is primarily lead teacher’s
implementation of practices, but consider all adults

Interviews
 For
those practices that might not or cannot be
observed during the 2-hour observation

Key Practice Items: Multiple indicators associated with
each item


Each indicator rated yes, no, or N/O (only when noted)
Red Flags
 Each

item rated yes or no
Using Effective Strategies for Responding to
Challenging Behavior
 Item
only scored when challenging behavior observed
 Includes three indicators for responses to each incident of challenging
behavior
 Each of these three Indicators rated as yes or no for each incident
Observation items
1. Schedules, routines, and
activities (SR)
2. Transitions between activities
(TR)
3. Supportive conversations (SC)
4. Promoting engagement (ENG)
5. Providing directions (PD)
6. Collaborative teaming (CT)
7. Teaching behavior
expectations (TBE)
8. Teaching social skills and
emotional competencies (TSC)
Observation and interview items
9.
Teaching friendship skills (FR)
10.
Teaching children to express
emotions (TEE)
11.
Teaching problem-solving
(TPS)
Interview items
12. Interventions for children
with persistent challenging
behavior (PCB)
13. Connecting with families
(COM)
14. Supporting families in using
Pyramid Model practices
(INF)
Item
Indicator
No Opportunity
33
34
35
Interview Questions for Teaching
Friendship Skills
37
38
Defining the Behavior – What to Note


Primarily Head Start classrooms
TPOT observations






50 classrooms
2 raters
3 occasions
300 total TPOTs
Generalizability study to look at
dependability of scores across items,
raters, occasions of measurement
CLASS
(Pianta, LaParo, & Hamre, 2008)
 50 classrooms
 Administered between 2nd and 3rd TPOT
observation
 Inter-observer agreement for 34% (n = 17)
 Convergent score validity
Work reported was supported, in part, by
Institute of Education Sciences grant
(R324A07212) to Vanderbilt University. The
information and opinions expressed are
those of the authors, not the funding
agency.
40



Less than .01% of
variance on key practice
indicators due to raters
<1% of variance on key
practice indicators due
to occasion
6.12% of variance on
key practice indicators
due to
classroom/teacher








Phi coefficient = absolute
decisions
G coefficient = relative decision
Phi key practice indicators = .89
G key practice indicators = .95
Phi red flags = .76
G red flags = . 84
Good news!
Dependability in rank ordering of
classrooms and dependability in
scores across raters, occasions
Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013).
Developing and gathering psychometric evidence for a fidelity instrument.
Journal of Early Intervention, 35, 150-172.
41
N = 50
Emotional Support
(ES)
Classroom
Organization (CO)
Instructional Support
(IS)
TPOT Key
Practices
.70
.73
.76
Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013). Developing and
gathering psychometric evidence for a fidelity instrument. Journal of Early Intervention,
35, 150-172.
42





TPOT Key Practices subscale and every CLASS dimension
and domain
TPOT Red Flags subscale and every CLASS dimension
and domain (negative relationships)
General teaching items on TPOT (SR, TR, SC, ENG, PD)
and each CLASS dimension and domain
Most targeted teaching items on TPOT (TBE, TSC, TEE,
TPS, FR) and Instructional Support CLASS domain
TPOT Connecting with Families with each dimension and
domain on CLASS
Additional detail in Chapter 7 in TPOT manual
43
 Noteworthy
correlations
TPOT
Key Practices subscale and overall ECERS-R
10 of the 14 TPOT key practice items and overall
ECERS-R
TPOT Red Flags subscale and overall ECERS-R
(negative relationships)
Additional detail in Chapter 7 in TPOT manual
44
Note. Adapted from Steed and Pomerleau (2012). N = 31 classrooms.
a = Seven environmental items included on pilot version of TPOT.
* = p < .05 ** = p < .01
Additional detail in Chapter 7 in TPOT manual
45
Use of TPOT in Potential Efficacy Study (N = 40)
100
Total TPOT score
90
80
Intervention
70
60
50
Comparison
40
30
Coaching
Training
20
10
0
Sept.
1
Nov.
2
Feb.
3
Apr.
4
Figure 1. Mean TPOT scores across 4 waves. Total TPOT indicators = 108. Wave 4 [t(40.03)=6.80, p<.001, Cohen’s d=2.6)
Hemmeter, M.L., Fox, L., Snyder, P., & Algina, J. (2011, April). Efficacy of a classroom-wide model for promoting social-emotional
development and preventing challenging behavior. Paper presented at the annual meeting of the American Educational Research
46
Association, New Orleans, LA.
Adjusted Means
Intervention
Control
Cohen’s d
Effect Size
Social
88.6
84
.41
Problem Behavior
108.7
115.5
-.52*
Social
103.8
96.4
.46*
Problem Behavior
95.2
99
-.29
Target Children
Non Target
Children
Additional Results from Potential Efficacy Study:
Target Child Social Interaction Behaviors

Using the TPOT in coaching
 Running
TPOT
 Formal TPOT
 Goal setting/action planning

Using the TPOT program wide
 Monitor
implementation of PW implementation
 Plan professional development

Using the TPOT in monitoring/evaluation
Percentage of Indicators Observed for Key Practice Items
Initial - 8 Teachers
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
November - 8 Teachers
April - 8 Teachers
Work reported was supported, in part, by Institute of Education Sciences grant
(R324A070008) to the University of Florida. The information and opinions expressed
are those of the authors, not the funding agency.
Scoring Sheet
Manual


Continuous event observational coding system
Used by teachers to record the frequency and
accuracy of complete learning trials embedded
within ongoing classroom activities, routines, and
transitions
 Adult
or environmentally arranged antecedents to elicit a
targeted child behavior
 Whether the target child behavior occurred
 Consequences and additional help (if appropriate) provided
following child behavior
Adapted from Embedded Instruction Observation
System (EIOS) - Research Version 2.0 (Snyder et al., 2010)
 EIOS
was primary dependent measure in a Goal 2 study
funded by IES*
 Adapted from the the Teacher Performance Rate and
Accuracy Scale (Ross, Singer-Dudek, & Greer, 2005)
 Designed to quantify interlocking three-term contingencies or
“learn units”
*Institute of Education Sciences Project No. R324A070008: Impact of Professional Development on Preschool
Teachers’ Use of Embedded Instruction Practices
Three-Term
Contingencies for
Teacher
Instructional
Components
Three-Term
Contingencies for
Child
Teacher Antecedent 1
Child [reaches for ball]
Teacher Behavior 1
Teacher: “Say ball.”
Child Antecedent
Teacher Consequence 1/
Teacher Antecedent 2
Child: “Ball.”
Child Behavior
Teacher Behavior 2
Teacher: [gives child a
Child Consequence
Teacher Consequence 2
Child [plays with ball]
balI]
Note. Table adapted from Ross et al., 2005
From: Barton, E., Bishop, C., & Snyder, P. (in press). High quality
instruction through complete learning trials: Blending intentional
teaching with embedded instruction. Young Exceptional Children
Monograph.
Individualized
Priority Learning
Targets
IEP Content
Targeted Curricula
(e.g., early literacy, social emotional)
General
Curriculum
Commercial
Universal
General
Curriculum
Locally
Developed
Early Learning Foundations
EIOS-T
EIOS-R
Focused on documenting
 Occurrence of learning
trials
 Categories of antecedents,
consequences and error
corrections
Focused on documenting
 Occurrence of learning
trials
 Category of antecedent
delivered
 Teacher-delivered
 Environmentally
 Adult-delivered
arranged
 Peer-delivered
 Environmentally

arranged
Procedural accuracy of
learning trial components

Presence or absence of
complete learning trial
components



Simplified and reduced EIOS-R codes
Developed coding manual and scoring sheet
Developed and piloted training
 Piloted with
preschool teacher who participated in Goal 2
study


Revised coding manual based on teacher feedback
Piloted with three preschool teachers in a multiple
baseline across teachers single-subject experimental
study (Bishop, Snyder, & Crow, 2014)
EIOS-T Codes

Multiple baseline across three teachers singlesubject experimental study (Bishop et al., 2014)
 When
provided feedback about their coding accuracy, 2 of 3
teachers were able to record accurately the occurrence of
learning trials
 When provided with feedback about their coding accuracy,
all 3 teachers were able to record accurately the fidelity with
which they implemented complete learning trial
components
 When teachers began to accurately record the occurrence
and accuracy with which they delivered CLT components,
the fidelity with which they implemented CLTs increased
How are these measures
similar?
Could these measures be used in an
integrated fashion? How would that
integration occur?
What is the role of measures such as these in terms of
supporting inclusion within the broader early childhood
quality movement?
THANK YOU
Head
Start
Child Public
Care Pre-K
Dev
Day
Total
Mean
Mean
Mean
Mean
Mean
Course hours in special education
9.25
1.18
1.50
16.08
7.52
Number of years of
teaching child(ren) with a disability
10.77
5.95
4.40
5.48
6.91
Number of years of teaching in EC
13.62
12.85
5.60
8.10
11.12
* Based on teacher report
National Professional Development Center on Inclusion
Head
Start
Child
Care
Public
Pre-K
Dev.
Day
Total
Mean
Mean
Mean
Mean
Mean
Number of children in classroom
17.15
16.55
15.00
16.38
16.51
Number of children with IEP in
classroom
2.62
2.15
2.40
4.69
2.94
Age of youngest child (Yrs)
3.54
3.20
4.00
3.69
3.49
Age of oldest child (Yrs)
4.54
4.30
4.60
4.62
4.47
Number of adults in classroom
2.38
1.70
2.20
3.46
2.37
Children with IEP/adults ratio
1.13
1.22
1.07
1.43
1.23
All children/adults ratio
7.66
10.91
7.10
5.02
8.21
ECERS-R score
4.95
4.58
5.14
5.31
4.92
ICP score
4.64
3.67
4.76
5.12
4.39
National Professional Development Center on Inclusion

Speech and language (38%)

Developmental delay (37%)

Autism (12%)

Other health impairment (5%)

Sensory impairment (4%)

Multiple disabilities (1%)

Orthopedic impairment (1%)

Don’t know (2%)
* Based on teacher report

88% of classrooms had at least one child with a
moderate or severe level of disability in at least
one area. *
* Based on teacher responses using an adaptation of the
ABILITIES Index (Simeonsson & Bailey, 1991).
National Professional Development Center on Inclusion
Children receiving majority of
services in classroom
Head
Start
Child
Care
Public
Dev.
Pre-K
Day
Total
17.65%
37.21%
8.33%
67.21%
57%
 In North Carolina, the percentage of children receiving the majority of
specialized services in the classroom is 50.97%*
 Nationally, the percentage of children receiving the majority of specialized
services in the classroom is 41.67%*
*OSEP Report to Congress, 2011
National Professional Development Center on Inclusion
Average Percent Occurrence Agreement
No Training
EIOS-T
Training
EIOS-T
Feedback
Natalie
12 (0 – 20)
62 (38 – 75)
90 (75 – 100)
Rhonda
31 (9 – 50)
36 (9 – 50)
47 (18 – 76)
Brenda
33 (0 – 66)
41 (20 – 63)
90 (78 – 100)
Average Percent Component Agreement
No Training
EIOS-T
Training
EIOS-T
Feedback
Natalie
N/A
66 (40 – 83)
88 (80 – 100)
Rhonda
N/A
66 (33-83)
83 (69-100)
Brenda
N/A
67 (39-89)
82 (61-83)
Percentage of Procedurally Correct Learning Trials