Taking Stock:
Assessing & Improving
Early Childhood Learning
and Program Quality
Board of the Massachusetts
Department of Early Education and Care
May 13, 2008
Dr. Sharon Lynn Kagan
Columbia University & Yale University
Overview
 Impetus for the Work
 Challenges
 Proposed System Design
 Differing Viewpoints on Using Child
Assessment Data for Local Agency
Accountability
 Action Steps
 Implications
Part I
Impetus for the Work
Growing Importance of ECE
 Throughout the United States and
countries around the world, Early
Childhood Education is ascending to
prominence.
 States are investing more dollars in
the field, with growth in virtually
every segment of ECE, including:
– Direct Services
– Professional Development
– Quality Rating Systems
Growing Investments
 These growing investments are
matched by calls to assess that the
investments are paying off.
 Are programs delivering the services
they say they are delivering?
 Are children achieving the outcomes
desired?
 Are the investments yielding
appropriate returns?
Growing Emphasis on ECE
 Practitioners also are interested in
results, recognizing that:
– The early years come only once for a
child
– The early years are critical to long term
development
– Teachers are the keys to making
programs high quality
– Teachers and policy makers can learn
from data about children.
Interest in Accountability
 Interest in Accountability is not
limited to ECE alone.
 All human service agencies are being
asked to take on a results orientation.
 Business and industry has provided a
lead in this area and fueled efforts
toward greater efficiency and quality.
 There is a prevailing ideology that
accountability provides the
information to make progress.
Many Factors Provide Impetus
 So there is a collision of interest that
propels a focus on outcomes and
accountability:
– New data that attests to the importance
of the early years
– New investments and the need to see if
they are making a difference.
– New movement toward accountability
as an elixir for quality improvement.
Part II
Challenges
Four Challenges
1. Structural Challenges
2. Conceptual Challenges
3. Technical Challenges
4. Resource Challenges
Challenges: Structural
 Fragmented non-system of programs
for preschool-aged children
 Disjointed early childhood and public
education policies
Multiple Standards and Assessments
Child Care
Head Start
State Pre-K
Special Education
Program
Quality
Standards
State Licensing Standards
(50 states)
Quality Rating Systems (QRS)
(13 states + 29 pilots)
Program
Performance
Standards
State Program
Standards
(39 states)
IDEA regulations
State program
standards
Assessing
Program
Quality
Licensing Visits
QRS Assessments
(13 + 29)
PRISM Reviews
Program
Monitoring
(30 states)
State Program
Monitoring
Standards
for
Children’s
Learning
Early Learning Guidelines
(49 states)
Head Start Child
Outcomes
Framework
Early Learning
Guidelines
(49 states)
3 functional goals
Child
Assessments
No current requirements
National
Reporting
System
Pre-K Assessments
(13 states)
Kg. Assessments
(17 states)
States report % of
children in 5
categories on 3
goals
Research/Evaluations
Yes
Yes
Yes
Yes
Kindergarten to Grade 3 Standards, Assessments, Data
Challenges: Structural
 Costs, burdens, confusion of multiple
standards, assessments, & reports.
 Multiple new initiatives all at once.
 Pre-K–K-3 disconnects:
– Pre-K assessments aren’t transferred to
schools.
– Standards, assessments, curricula aren’t
aligned.
Challenges: Conceptual
 Early childhood assessment training
and practice vs. standards-based
assessment and curriculum.
 Early childhood reliance on program
standards/data vs. expanded interest
in children’s learning.
Challenges: Technical
 Need appropriate assessment tools
and methods to report on:
– Progress/status of young children in all
domains of learning and development
– Young ELLs and children with
disabilities
– Program quality in diverse local
agencies
Challenges: Resources
 Limitations and inequities in
funding for:
– Programs
– Infrastructure
 Risk that accountability efforts
ignore and exacerbate inequities in
resources
Task Force Charge
 Recommendations for a state
accountability system for early
education programs for prekindergarten children and linkage to
standards-based assessments in
kindergarten-grade 3.
Origins
 Conceived by The Pew Charitable Trusts
as part of their Advancing Quality Pre-K
For All initiative.
 Additional funding from the Foundation
for Child Development and the Joyce
Foundation.
 Task Force convened in fall, 2005, report
release fall, 2007.
 Presentation reflects progress-to-date.
Task Force Members
 Dr. Sharon Lynn Kagan, Chair
 Dr. Eugene Garcia, Vice-Chair
– Dr. W. Steven
Barnett
– Ms. Barbara
Bowman
– Dr. Mary Beth
Bruder
– Dr. Lindy Buch
– Dr. Maryann Santos
de Barona
– Ms. Harriet Dichter
– Mr. Mark Friedman
– Dr. Jacqueline
Jones
– Dr. Joan Lombardi
– Dr. Samuel Meisels
– Ms. Marsha Moore
– Dr. Robert Pianta
– Dr. Donald Rock
Part III
Proposed System Design
Framing Beliefs
 Accountability is here to stay.
 Programs should be held to performance
standards that are documented and
verified.
 Assessments should inform policy
decisions and be tied to program
enhancement efforts.
 Current approaches to accountability and
assessment must be reformed.
State Accountability & Improvement
System Design
Infrastructure
Early Learning &
Program Quality Standards
Program Rating &
Improvement
Professional Development
Data Management &
Reporting
Assessment/Program Improvement Approaches
APPROACHES
I
CHILD POPULATION
II
PROGRAM POPULATION
III
STATE PROGRAM
EVALUATION
IV
LOCAL AGENCY
QUALITY
CORE
QUESTION
How well are all children
progressing in learning and
development?
What is the quality of all
early childhood programs?
What is the quality and how
well are children progressing
in specific state programs?
What is the quality in local
agencies?
HOW DATA
IS USED
- Oversight of state
investments/initiatives
- Planning new
investments/initiatives
- Baseline information for K-12
education planning
- Oversight of state
investments/initiatives
- Planning new
investments/initiatives
- Baseline information for K-12
education planning
- Program-wide improvement
efforts
- Refining standards/policies
- Appropriations decisions
Pre-K-Grade 3 Alignment and Linkages
- Technical assistance to
individual agencies.
- Awarding incentives and
recognition to local agencies for
program improvements
- Decisions on funding local
agencies
Infrastructure
System Infrastructure
Early
Learning &
Program
Quality
Standards
Program
Rating &
Improvement
Professional
Development
Data
Management
& Reporting
Infrastructure
 Early Learning and Program
Quality Standards
– Alignment between:
• Standards, assessment systems and
curricula
• Standards between ages and grades
• State and federal program structures
and funding streams
• Child and program standards
Infrastructure
 Program Rating & Improvement
– Assesses and reports on the quality of
all forms of early education programs
– Provides technical assistance and
professional development to improve
quality
– May provide public
recognition/incentives to reward
higher levels of quality
Infrastructure
 Professional Development System
– Links informal training with formal
education, provides career pathways,
links education and compensation.
– Supports training on assessment
administration, analysis and use.
Infrastructure
 Data Management and Reporting
– All-in-one place data on:
• Children
• Programs
• Workforce
– Unified system of child identification
numbers.
– Provides for quality assurance of data
and assessments.
Assessment Approaches
 States vary in:
– What they want to know
– How they plan to use data
– Available resources
 States may implement one or any
combination of options.
 Report includes cautions/safeguards
for each option.
Assessment/Improvement Approaches
APPROACHES
Child
Population
Program
Population
State
Program
Evaluation
Local Agency
Quality
CORE
QUESTION
How well are
all young
children
progressing in
learning and
development?
What is the
quality of all
early
education
programs?
What is the
quality and
how are
children
progressing in
specific state
programs?
What is the
quality in local
agencies?
All Four Approaches Discuss:
 Questions that can be Answered
 Data to be Collected
 Designs for Data Collection
 Uses of the Data
 Challenges and Cautions
I. Child Population Approach
 How well are all young children
progressing in learning and
development?
– Data:
• Demographic data collected on representative
sample of all young children
• Comprehensive data on health, well-being, preschool enrollment
– Designs:
• At Kindergarten Entry
• At Ages 1, 3, 4
• Longitudinal ECLS State Strategy
I. Child Population Approach
 How well are all young children
progressing in learning and
development?
– Uses:
• Planning interagency investments/initiatives
• Legislative oversight
• Baseline information for public education
– Challenges/Cautions:
• Gaining access to unrolled preschoolers
• Can’t use data to maker inferences about preschool programs’ impacts/quality
• No causal attributions
• Expensive and not routinely done
II. Program Population
Approach
 What is the quality of services
in all early childhood
programs?
– Data:
• Program quality
• Workforce
• Public investments
– Design:
• Data collected from states
Program Improvement
(PRS/QRS) System
II. Program Population
Approach
 What is the quality of services in all
early childhood programs?
– Uses:
• Planning interagency investments/initiatives
• Legislative oversight
• Baseline information for public education
– Challenges/Cautions:
• Need to consider quality of programs in light of
investments in them
• Need to be sensitive to changes in program
quality
• Unable to attribute causality to program quality
rankings
III. State Program Evaluation
Approach
 What is the quality and how well are
children progressing in specific
programs?
– Data:
• Sample of centers that represent the universe of
programs to be studied
• Comprehensive data on children, program, and
teachers
• Data aligned with program mandates and
standards
– Design:
• Standard Evaluation Design-program and
instrument identification, data collection and
analysis
• Program Rating System Design-uses data for
program information from the PRS/QRS
III. State Program Evaluation
Approach
 What is the quality and how well are
children progressing in specific
programs?
– Uses:
• Program-wide improvement efforts
• Refining standards/policies
• Appropriations decisions
– Challenges/Cautions:
• Attribution of single program effects due to
many children’s participation in multiple
programs (e.g., lack of pure control group)
• Pre-mature evaluation inappropriate
• Implementation fidelity to program design
IV. Local Agency Quality
Approach
 What is the quality in local
agencies?
– Data:
• Program quality data in relation to state
standards
• Observations of teaching/learning
opportunities
– Design:
• Use PRS/QRS data
• On rotational basis
– 1/3 annually
– Low performing programs
IV. Local Agency Quality
Approach
 What is the quality in local
agencies?
– Uses:
• Technical assistance to individual local
agencies
• Awarding incentives and public
recognition
– Challenges/Cautions:
• Heavy burden on state agencies
• Heavy costs to do classroom observations
• States need to develop technical assistance
to address program needs/weaknesses
Pre-K—Grade 3 Integration
 Align standards, assessments and
reporting on:
– Children’s progress
– Quality of teaching/learning
opportunities
 “Vertical” teams of
teachers/managers to:
– Review assessment information
– Enrich learning experiences and teaching
strategies
 Joint professional development
Part IV
Differing Viewpoints on
Using Child Assessment Data
for Local Agency
Accountability
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 Task Force reached an easy
consensus on:
– Infrastructure components
– Four assessment/improvement
approaches
– Linking of pre-kindergarten through
grade 3
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 Some members said this wasn’t
enough.
 Needed more information:
– How well are children in local agencies
learning?
– How does what they are learning
compare with state standards?
– Are some local agencies unusually
effective in fostering learning progress?
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 To do this, some members of the Task
Force advocated collecting and reporting
data on all children's performance and to
distill it so it could be used by local
agencies for planning and improvement.
 They recognized that we can have children
in good programs who don’t do well, and
they contend that program quality is NOT
a proxy for child outcomes.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 These Task Force members
recognized the huge challenges
associated with this:
1. When we disaggregate data, it will be
held to a higher standard.
2. The field needs better tools
3. The field needs well-trained reliable
data collectors
4. The field needs systematic ways of
collecting, cleaning, and analyzing
data.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 Most importantly, these Task Force
members noted, we need to be very
careful not to misuse data:
– Not to teach to the test.
– Not to use data to label or place
children.
– Not to defend programs.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
 The Task Force members who
support the use of agency-level child
assessment data acknowledged the
challenges, but contended, over time,
states would benefit by using both
child and program quality data.
 They recommended nine crucial
safeguards.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
1. Collect enriched data on the
programs, including teacher-child
interactions, nature of curriculum
interventions, and nature of teacher
training.
2. Collect detailed data on the children,
including prior ECE experience and
primary language.
3. Impel the state to align its assessments
to all domains in standards.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
4. Select approaches that include direct
observation.
5. Base reporting on progress.
6. Collect data at more than one point in
time.
7. Stagger strategies for data collection:
–
–
One third in each of 3 years
Priority accorded to low-quality programs
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
8. Do not, under any circumstances,
report any data on individual
children.
9. Use the data collected to make
program improvements.
SAFEGUARDS #8 AND #9 ARE, BY
FAR, THE MOST IMPORTANT
ONES RECOMMENDED.
Differing Viewpoints on Using
Child Assessment Data on
Local Agencies
While there was consensus on
the safeguards recommended,
the Task Force did not reach
consensus on offering this a
separate approach.
Part V
Action Steps
Action Steps: Legislatures
 Provide adequate funding for
programs and infrastructure for
ongoing assessments and program
improvements.
Action Steps: State Agencies
 Develop a strategic plan for early
childhood accountability and program
improvement system.
 Create a robust, positive, and rigorous
culture for early childhood
accountability efforts.
 Enable local Pre-K–3 partnerships.
Action Steps: Federal
Government
 “Harmonize” information systems.
 Fund research and development
for better assessment tools.
 Support ongoing longitudinal
research on children and
programs.
Action Steps: Local Agencies
 Create opportunities for teachers
and managers to review
assessments and enhance children’s
learning opportunities.
 Initiate dialogue with local school
districts.
The Benefits
 For Children: Enhanced learning
opportunities and improved outcomes
 For Legislators: Better data to guide
state policies and investments
 For Teachers/Directors: Targeted
and well-resourced professional
development and program
improvement efforts
 For the Early Childhood
Profession: Enhanced public
awareness and credibility
Action Steps: Follow-Up
 The Pew Charitable Trusts approved
a grant to the Council of Chief State
School Officers (CCSSO) to
disseminate the report and work with
selected states in implementing
recommendations.
 Project will build on CCSSO efforts
to improve state education standards,
assessments, and data systems for all
children—preschool through high
school and beyond.
Part VI
Implications
Implications
 Acknowledge the intensity of
sentiment regarding potential misuse
of child assessment data.
 Attend to the infrastructure and the
resources to do the collection.
 Think systemically.
Implications
 Need more work on tools that are
aligned with standards and
curriculum.
 Need more work on hard-to-capture
domains in assessment construction.
 Need far more work on the
considerations regarding English
Language Learners and children with
disabilities.
Implications
 Need states to focus on effective
accountability systems.
 Such accountability systems should be
designed to meet the state’s needs.
 Such accountability systems should take
into consideration state resources (human,
technical, and fiscal).
 Every state should develop and begin
implementing an ECE accountability
system that provides data and respects
families, teachers, children, and child
development principles.