Scaling Progress in Early
Childhood Settings
(SPECS)
STEPHEN J. BAGNATO, Ed.D., NCSP
Professor of Pediatrics & Psychology
Director, Early Childhood Partnerships
Director, SPECS (PAPREKA/PEIOS)
Children’s Hospital of Pittsburgh/UCLID Center
University of Pittsburgh School of Medicine
steve.bagnato@chp.edu
www.uclid.org
SPECS Program Evaluation
Research Team Leaders:
PAPREKA and PEIOS
Stephen J. Bagnato, Ed.D., NCSP, Director
Candace Hawthorne, Ph.D., OTR/L, Coordinator
Ilene Greenstone, MA, Coordinator
Pip Campbell, Ph.D., OTR/L, Coordinator
Assisted by Western and Eastern PA Research Teams
What is the Authentic
Assessment Alternative to
Conventional Testing in
Early Childhood?
John T. Neisworth
Stephen J. Bagnato
John Salvia
Frances M. Hunt
Inauthentic Measurement in
Early Childhood
“Much of developmental psychology [early
childhood assessment] as it now exists
is the science of the strange behavior of
children with strange adults in strange
settings for the briefest possible periods
of time.” (Bronfenbrenner, 1979, p. 19)
Authentic Assessment
in Early Childhood






Natural observations of ongoing child behavior
in everyday settings and routines vs. contrived
arrangements;
Reliance on informed caregivers (teachers,
parents, team) to collect convergent, multisource data across settings;
Curriculum-based measures linked to program
goals, content, standards, & expected outcomes;
Universal design; equitable assessment content
and methods;
Intra-individual child progress supplemented by
inter-individual normative comparisons;
NAEYC/DEC/HS & PA DAP Assessment
Standards & Practices
Are There Professional and
Pennsylvania Standards for
Authentic Assessment in
Early Childhood/Early
Intervention?
Selected Professional Standards
for Early Childhood Assessment
(DEC, 2004; NAEYC, 1997, HS, 2000)









Reliance on developmental observations-ongoing observational
assessments overtime
Performance on “authentic, not contrived, activities”
Integration of assessment and curriculum
Child progress on past performances as the reference, not group
norms
Choose materials that accommodate the child’s special functional
needs
Use only measures that have high treatment validity
Rely on curriculum-based measures as the foundation or “mutual
language” for team assessments
Defer a diagnosis until evaluation of a child’s response to a
tailored set of interventions
Use scales with sufficient item density to detect even small
increments of progress
6 “Best Practice” Criteria for Authentic Assessment
in Pennsylvania’s ECE Programs
1.
Purpose:
2.
Method:
3.
Context:
4.
Process:
5.
Standards-based:
6.
Parent Partnerships:
Assess for program planning not diagnosis or
exclusion; eliminate “readiness” testing practices
“No tabletop testing”; Deemphasize scores; observe
and assess functional skills that link to the curriculum standards
Observe and record evidence of natural, ongoing
child development and behavior in typical, everyday routines not
contrived settings
Rely on teachers/caregivers to observe and record
child progress 2-3 times each year
Align all assessments and their item
content with ELS and curricula; link assessment with expected
outcomes
Enable parents to have central role
in providing observational data on progress
Measures for PEIOS/PAPREKA
Research: Balance of Attributes

Tension to balance research rigor with
utility; Choice of measures based on
following elements:
1.
Simplicity
Authenticity
Utility
Evidence-base
Standards-referenced
Functional content
Sensitivity
2.
3.
4.
5.
6.
7.
The Pennsylvania Early
Intervention Outcomes Study
(PEIOS)
Documenting the Benefits of Early Intervention
Supports in PA to Fulfill State and Federal
Mandates
What are the missions,
research questions and
authentic measurement
design for PEIOS?
PEIOS “Fast Facts”








Aim: Document early intervention outcomes for state &
federal mandates
County agencies and MAWAs in 6 PA Regions mapped to
PQP
Random selection
Collect data on entry-level functioning compared to
performance at followup
Use both/either an existing measure and a common
functional measure across programs: ABAS II
File reviews to code program and service intensity
Multiple research strategies to analyze interrelationships
among program intensity and child/family outcomes
Classify outcomes by OSEP/ECO categories
PEIOS Outcome and Research
Measures

Child Measure [Caregiver]:
Adaptive Behavior Assessment System II
(ABAS; Harrison & Oakland, 2004)
 Child Measure [Caregiver]:
Program-Identified Measure (BDI; DC;
COR)
 Program Measure [PEIOS Team]:
Program Specs (Bagnato, 2005)
 Program Measure [PEIOS Team]:
Developmental Specs (Bagnato, 2005)
Adaptive Behavior Assessment
System II (ABAS, 2003)

Multi-dimensional observational & judgmentbased rating scale (3-point) of functional
competencies
 Ages: 0-89years
 Early childhood forms: Parent & Teacher/Provider
 Nationally standardized: Ages 0-5=2100; all forms
and all ages=5270
 Norm-referenced scores: General Adaptive
Composite (100,15); Subskills (10,3)
 Excellent technical research base; disability
studies; Aligns with DSM IV; AAMR categories
 Psychological Corporation
ABAS Rating Format
0……..Is not able [can’t; too young; physical limits]
1……..Never/almost never when needed [prompts]
2……..Sometimes when needed [with/out help]
3……..Always/almost always when needed [before]
G……..Check if you guessed
ABAS Domains:
Parent Form (241); Teacher (216)










Communication
Community Use
School Living/Home Living
Functional Pre-Academics
Health & Safety
Leisure
Self-Care
Self-Direction
Social
Motor
ABAS Disability Research:
Clinical and Matched Control Samples












Mental Retardation
Developmental Delay
Biological Risk Factors (e.g., prematurity; drugs)
Motor and Physical Impairments
Language Disorders
Autism and PDD
Learning Disability
ADHD
Alzheimer’s Disease
Neuropsychological Disorders
Behavior and Emotional Disorders
Deaf and Hard of Hearing
PEIOS Program Evaluation Research
& Measurement Model

January 2006: Identify and train PEIOS program
evaluation liaisons
 January-February 2006: Random selection of
children
 January-February 2006: Train on use of the ABAS
II (if chosen) with liaisons, teachers, others
 January-April 2006: Collect EI-entry child data
 January-April 2006: Conduct file reviews to
document program intensity
 May 2006: Collect child progress data using
ABAS II or program-chosen measure (each
September and May)
TIME IN
INTERVENTION
CHILD & FAMILY PROGRESS
OUTCOMES
PROGRAM
INTENSITY
PEIOS LONGITUDINAL REPEATED MEASURES
REGRESSION RESEARCH DESIGN AND TIMELINE
REGION 1
September
May
REGION 2
REGION 3
ABAS II
Program Scale
ABAS II
Program Scale
D-SPECS
P-SPECS
D-SPECS
P-SPECS
What are the federal
OSEP/ECO outcome
indicators and reporting
timelines for PEIOS?
OSEP/ECO Child Outcome
Indicator Domains
1.
2.
3.
Positive social-emotional skills (including
social relationships)
Acquisition and use of knowledge and skills
(including early language/communication; early
literacy)
Use of appropriate behaviors to meet one’s
needs
Domains, sub-domains, and item content of
measures are mapped to these integrated
functional areas
Sample LaRosa Database
OSEP/ECO Child Outcome Indicator
Metrics
1.
2.
3.
4.
5.
6.
% of children who reach or maintain functioning
level comparable to same-age peers
% of children who improve functioning toward
same-age levels
% of children who did not improve functioning
% of children maintaining own rate and preventing
regression
% of children showing specific curricular skill
improvements compared to own previous skill levels
% of children whose developmental progress profiles
exceed own pre-intervention (maturational) expectations
and those of their local EI peer group (IEI; CEI; PCI, HLMEAPS)
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of Programming
[Pooled HR/DD groups; n = 104; p<.000; 48th%ile>68th%ile; 95% confidence interval]
850
832
DOCS total score
825
790
800
ECI
800
775
761
777
776
750
expected
757
725
735
729
700
683
675
684
upper margin
of error of
expectation
713
650
2.30
3.00
3.20
3.40
3.60
3.80
Age (in years)
4.30
5.00
OSEP/ECO Family Outcome Indicators
[Draft Reconciliation]

Based on ratings on the Family Outcomes
Survey (in development)
1.
Understand their child’s strengths, abilities, and
special needs
Know their rights and advocate effectively for
their children
Help their children develop and learn
Have support systems
Access desired services, programs, activities in
their community
2.
3.
4.
5.
Timelines for State Reporting of Child
Outcome Data to OSEP
(Recent Report from OSEP/ECO National Meeting , Washington, DC,
1/12-13/06





December 2005 SPP: Measurement plan
submitted
February 2007 APR: Report on EI-entry
child data only—no progress data wanted
February 2008 APR: 1st progress report
February 2009 APR: 2nd progress report
February 2010 APR: 3rd progress report
What is the collaborative
model for training and
implementation in PEIOS and
PAPREKA?
DIRECTOR
PAPREKA COORDINATORS:
Western PA
Eastern PA
RESEARCH SYSTEMS
DATA MANAGER
STATISTICIAN
PROGRAMMER
EVALUATION
ASSISTANTS
REGIONAL PA
CONSULTANT
COORDINATORS
LIAISON
REGIONAL
CONSULTANT
SPECS TEAM
COUNTY
MAWA
SCHOOL DISTRICT
PARTNERSHIPAdministrators
TEACHERS
CAREGIVERS
EVALUATION
ASSISTANT
The Pennsylvania
Pre-Kindergarten Analysis
(PAPREKA)
A 4-Year Independent Program Evaluation
Research Collaborative to Document the Impact
and Outcomes of Partnership for Quality PreKindergarten (PQP)
SPECS Pennsylvania Early Childhood
Intervention Outcome Studies
www.uclid.org; Early Childhood Partnerships: SPECS






Heinz Pennsylvania Early Childhood Initiatives
(ECI) [1997-present) (Bagnato, etal., 2002; 2005)
Pennsylvania Pre-Kindergarten Analysis
(PAPREKA; 2005-2009] (Bagnato etal. 2005)
Pennsylvania Early Intervention Outcomes Study
(PEIOS; 2005-2008) (Bagnato etal., 2005)
TRACE Center of Excellence for Early Childhood
Assessment (2002-2007; Dunst, Trivette, Bagnato)
The Efficacy of a Direct Instruction Add-On to a
DAP Curriculum in 4KIDS at Braddock (20052007)
Pennsylvania Preschool Integration Initiative
(PAPII; 1989-1993) (Bagnato & Neisworth, 1993)
SPECS
Evaluation of Heinz Pennsylvania ECI
Outcomes (1997-2004)












Nearly 4000 at-risk children
Urban and rural communities: Pittsburgh, Erie, Central PA,
York, Lancaster, Beaver
School district collaborations
Steady developmental gains
Delayed group gains
Social-behavioral gains, even for disorders
Early school success for >400 children: K-5
Reduced grade retention and special education
Program quality improvements
Improvements in parenting skills
Evolving community networking
Analysis of time-in-intervention effects (“dosage”)
What are the missions,
research questions and
measurement design for
PEIOS?
PAPREKA Missions for PQP

Document child and program outcomes attributable
to PQP
 Analyze the comparative impact of various school
district/community ECE partnership models on child
and program outcomes
 Analyze the early school success of PQP children
during their K year
 Field-validate the PA Early Learning Standards and
develop an ELS assessment rubric
 Set the stage for future followup of PQP children into
early grades and linkage with PSSA results
PAPREKA Research
Questions for PQP
1.
2.
3.
4.
5.
PQP children will show a pattern of actual progress which
will outpace their maturational expectancies.
Program quality and interactive teaching styles will predict
child progress and outcomes.
Certain types of partnership models will better predict child
outcomes and program quality.
For both children at-risk and with delays, those who
participate and remain engaged in the PQP programs for
the longest periods of time (“dosage”) will show the most
significant progress.
PQP children will demonstrate early school success in
preschool and kindergarten based upon three criteria: blind
teacher assessments on the BSSI; greater % of ELS
attainments; and reduced grade retention and special
education placement rates compared to historical district
benchmarks.
PARTNERSHIP
MODEL
TIME IN
INTERVENTION
CHILDREN’S
EARLY SCHOOL
SUCCESS OUTCOMES
PROGRAM
ASPECTS
What is the authentic
measurement approach and
timeline for PAPREKA?
Natural learning
environment
“...the ongoing/routine/typical
circumstances and contexts
of a child”…the child’s natural
“developmental ecology” that
forms the environmental basis
for real-life early learning and
adaptation (OSEP, 1998).
SPECS Approach for PAPREKA

Design research model through collaboration with
community partners (“participatory action
research”)
 Train teachers for ongoing child assessments
 Conduct “authentic assessments” in everyday
preschool settings using natural observations
 Provide feedback to teachers & parents for
individualized early learning plans
 Evaluate progress on “Key Performance
Indicators (KPI): child, program quality, teacherchild interactions, & early school success
outcomes
 Classify type of partnership model in each site
 Assess K achievements of children
 Research impact of PQP
PAPREKA Measures
1.
Basic School Skills Inventory 3 (BSSI-3)
(Hammill, etal, 1998)
2.
3.
4.
Preschool and Kindergarten Behavior
Scale (PKBS) (Merrell, 2003)
Early Childhood Environment Rating
Scale (ECERS-R) Screener (Cassidy etal, 2005)
Teaching Styles Rating Scale (TSRS)
(McWilliam, etal, 1996)
5.
PA Early Learning Standards (ELS)
Rubric (in development)
Sample MIS Database
A Sample of
Mary’s Skills
22 months old
Some Things I’m Really Good At

put 2 words together (examples: 'Want more,'
'Get down')
 refer to myself by name
 participate in and enjoy nursery rhymes and
finger games like 'Humpty Dumpty' or 'Itsy
Bitsy Spider'
Some Things I’m Working On
 copy sounds made by others
 drink from a regular cup without help
 answer simple questions
Some Things I Want to Learn Next
 dress myself, but I may need some help or
supervision
 share toys with other children when asked by
an adult
 count two or three objects (saying the number
as I touch the object)
Database
Merged Document
Class Snapshot #2
Student Progress on Selected Assessment
Items
January 2003 DOCS Assessments
Sunshine Center Young Toddlers
Highlighted
& Bolded
Item #’s
Areas of
More Program
Support
Item
#
Item Description
24 to 35 months
draw a straight line from top to
295 bottom
use one hand to open and close
296 scissors
usually use plurals and past
297 tense correctly when speaking
use some verb endings ('-s' for
plurals, '-ed' for past, '-ing' for
298 present)
299 tell if self is a boy or a girl
bend at the waist to pick up
something off the floor (does not
300 squat)
match to the colors red, blue,
301 green, and yellow
pull down my pants to use the
302 bathroom
1
2
3
4
5
6
7
8
9
Briana
Keisha
Mark Akeem Felix Katy Carlos
J.
Ned T. S.
Ali M. S.
A.
J.
B.
S.
gt
n
gt
gt
gt
gt
n
gt
gt
gt
gt
gt
n
n
gt
gt
n
gt
n
n
n
n
gt
gt
gt
gt
n
gt
gt
n
n
gt
gt
Basic School Skills Inventory
(BSSI)









Learning readiness skills for children
Authentic teacher observational ratings
Ages: 48-107 months (Pre-3rd grade)
6 Domains: Spoken language; Reading; Writing;
Math; Behavior; Daily living
Standard and T-Scores (100/15; 50/10)
Functional skills/benchmarks for learning
Graduated scoring: 0, 1, 2, 3 (mastery)
Norms = 757 children; 5 states
PRO-ED
BSSI Subscale Samples






Spoken Language
Uses complete sentences when talking
Listens to and retells a story in sequence
Initiates and maintains conversations with others
Reading
Recognizes upper/lower case letters
Names letters when sounds are spoken
Has basic site vocabulary of 5 words
BSSI Subscale Samples






Writing
Writes from left to right
Writes first name without a model
Writes single letters when asked (b, h, m, t, a, e)
Mathematics
Counts objects in set of fewer than 10
Counts aloud from 1-20
Understands concepts of 1st, 2nd, 3rd
BSSI Subscale Samples






Classroom Behavior
Makes friends easily
Takes turns
Uses teacher feedback to improve learning
Can attend to activity for 5 minutes
Daily Living Skills
Enters and exits school by self
Assumes responsibility for own belongings
BSSI Rating Scale
When completing the BSSI, a four-point
rating scale is used to rate each behavior
or skill:
0 (Does not perform)
1 (Beginning to perform)
2 (Performs most of the time)
3 (Performance indicates mastery)
National Normative Comparisons for ECI Childrens’
Early Learning Abilities: Second Semester Grades K-1
Basic School Skills Inventory-Revised
150
Mean Standard Score
140
130
120
110
100
90
80
70
60
Spoken Reading Writing
language
Math
Classroom
behavior
Domain
Daily
living
skills
Overall
Social
skills
Problem Academic
behaviors competence
% Learning Skill s Achieved
Acquisition of Kindergarten Precursor Skills Matching PA Early
Learning Standards:
Achievement in Final 6 Months of ECI
100
Time 1
90
Time 2
80
70
60
50
40
30
20
10
0
Asks
Understands
Counts
Names
letters
Writes
letters
Names
printed
letters
Knows/
"reads"
signs
Names
items
Retells
Tells
meanings
"Reads"
Total
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of Programming
[Pooled HR/DD groups; n = 104; p<.000; 48th%ile>68th%ile; 95% confidence interval]
850
832
DOCS total score
825
790
800
ECI
800
775
761
777
776
750
expected
757
725
735
729
700
683
675
684
upper margin
of error of
expectation
713
650
2.30
3.00
3.20
3.40
3.60
3.80
Age (in years)
4.30
5.00
Preschool and Kindergarten
Behavior Scales (2003)








36-72 months
Dual focus on social skills and problem behaviors
Competencies/concerns with treatment validity
Cooperation, interaction, independence, and
behavior domains (attention, withdrawal)
Parent and professional ratings
Norms = 2,855
Graduated scoring and SS
PRO-ED
Preschool and Kindergarten
Behavior Scales (2003)
SOCIAL SKILLS
Never
Rarely
Sometimes
Often
Works or plays independently
0
1
2
3
Follows instructions from
adults
0
1
2
3
Shows self-control
0
1
2
3
Participates in family or
classroom discussions
0
1
2
3
Follows rules
0
1
2
3
Takes turns with toys and
other objects
0
2
3
1
Preschool and Kindergarten
Behavior Scales (2003)
PROBLEM BEHAVIOR
Never
Rarely
Sometimes
Often
Acts impulsively without
thinking
0
1
2
3
Must have his/her own way
0
1
2
3
Is restless and “fidgety”
0
1
2
3
Withdraws from the company
of others
0
1
2
3
Is overly sensitive to criticism
or scolding
0
1
2
3
Disrupts ongoing activities
0
1
2
3
Social-Behavioral Progress Pattern for ECI Children:
Behavior Disorder Group (18%) T1-T5
150
Time 1
Time 3
Time 5
Mean Standard Score
140
130
120
110
100
90
80
70
60
50
Social Skills
Problem Behavior
Domain
Roles for Authentic
Assessment in PAPREKA
Teachers
[September and May]
PAPREKA Team
[September and May]
(Note: January for 2006)
(Note: January for 2006)

BSSI-3
 PKBS

ECERS-R Screen
 TSRS
Linking Assessment to State Early
Childhood Outcomes:
PA Early Learning Standards (ELS)








Approaches to Learning
Creative Arts
Language & Literacy
Logical-Mathematical
Personal-Social
Program Partnerships
Science
Social Studies








Demonstrate initiative &
curiosity
Expresses self through
movement & music
Understands language
sounds
Develops space-shape
concepts
Develops self-control skills
Form school partnerships
with ECE
Acquire knowledge about
matter and living things
Understand role of self in
community
Linking BSSI Items to the
PA Early Learning Standards
BSSI Area or
Domain
BSSI Item
Spoken
Language
Answers questions
correctly after listening to a
story [SL 11]
Reading
Follows printed
instructions (reads and
follows directions) [R 20]
Classroom
Behavior
Demonstrates a readiness
to respond when his or her
turn comes during group
activities [CB 11]
Daily Living
Skills
Follows directions relating
to paper-and-pencil tasks
[DL 10]
PA EL Standard
(for 4- & 5-year-olds)
Develop and expand
listening and
understanding skills
[RL 1]
Linking BSSI Items to the
PA Early Learning Standards
BSSI Area or
Domain
Mathematics
BSSI Item
PA EL Standard
(for 4- & 5-year-olds)
Assigns the correct
numeral to a set of objects
[M 7]
Learn about numbers,
numerical representation,
and simple numerical
operations [LM 1]
Understands the concepts
of “first”, “second”, and
“third” [M8]
Daily Living
Skills
Tells time within 5 minutes
from analog watch or clock
face [DL 19]
Overview of Programmatic
Measures: ECERS & TSRS
ECERS-R Screener
 16 core items
 Two factors:
1. ActivitiesMaterials (9)
2. LanguageInteraction (7)
TSRS
 20 items
 Focus on teacherchild interaction
during instruction
 8 teaching strategy
variables: Redirects,
Introduces, Elaborates,
Follows, Informs,
Acknowledges, Praises,
Affect
PAPREKA LONGITUDINAL REPEATED MEASURES
REGRESSION RESEARCH DESIGN AND TIMELINE
PARTNERSHIP
MODEL 1
September
May
PARTNERSHIP
MODEL 2
PARTNERSHIP
MODEL 3
BSSI-3
PKBS
BSSI-3
PKBS
ECERS-R
TSRS
ECERS-R
TSRS
Roles of Implementation Sites in
the Partnership:
Implement Assessment & Gain
Incentive Funds







Designate PQP Evaluation Liaison
Train with PAPREKA team to learn assessments
Train teachers in assessments
Gain signed parent consent
Ensure completion of assessments by teachers
2x/year
Collect completed assessment forms and
transfer hard copies to PAPREKA Coordinator
Show teachers how to use feedback letters
Roles of PAPREKA Team for
Implementation Sites








Meet with partnership to form trusting
collaboration
Maintain confidentiality
Train liaison and teachers in authentic
assessment “best practices”
Complete programmatic observations 2x/year
Collaborate to classify type of partnership in
each program
Produce child feedback letters
Issue 1 year-end “report card”
Analyze aggregate data on child outcomes
PAPREKA Team Roles with
The PQP Planning Sites






Schedule collaborative meetings to develop our
partnership early as a basis for future work
Offer early training sessions on assessment
purposes and eventual evaluation model and
process
Demonstrate computer child feedback options for
programs to guide teaching and quality
improvements
Consult with partnerships to facilitate their
eventual implementation proposal
Collect extant information from sites on various
programmatic features
Begin process of classifying sites by type of
partnership model
Not everything that can be
measured counts, and not
everything that counts can be
measured (Einstein, 1951)
The MisMeasure of Man
(Stephen J. Gould, 1981)
“We pass through this world but
once. Few tragedies can be more
extensive than the stunting of life,
few injustices deeper than the denial
of an opportunity to strive or ever
hope, by a limit imposed from
without, but falsely identified as lying
within” (p.28).