SSIP Phase I:
Data Analysis
Part C/619 State Accountability
Priority Area
April 8, 2014
Disclaimer
This SSIP presentation and supplemental materials
were developed prior to OSEP’s publication of the
final SPP/APR package
Webinar Goals
• Participants will leave the webinar with a basic
understanding of:
– Phase I: Data Analysis process
– Resources and strategies that can support states
in the Data Analysis process
– Considerations for engaging stakeholders in the
Data Analysis process
3
Data Analysis Requirements
A description of how the state analyzed key
data to:
(1) select the State-identified Measurable
Result.
(2) identify root causes contributing to low
performance.
The description must include information
about:
(1) how the data were disaggregated by
multiple variables.
(2) if applicable, any concerns about the
quality of the data and how the state will
address these concerns.
(3) If applicable, methods and timelines
related to any additional data to be
collected and analyzed.
4
Measurable Result Requirements
• May, but need not, be an SPP/APR
indicator or a component of an
SPP/APR indicator.
• Must be clearly based on the data
and state infrastructure analyses.
• Must be a child-level outcome in
contrast to a process outcome.
• May be a single result or a cluster of
related results.
5
Baseline and Targets Requirements
• States must establish baseline
(expressed as a percentage)
with the Measurable Result
(FFY 2013 data)
• States must set targets
(expressed as percentages)
for each of five years FFY
2014 – FFY 2018
• FFY 2018 target must be
higher than FFY 2013 baseline
6
Fundamentals of Data Analysis
• EIA
– Evidence
– Inference
– Action
• Starting with questions
7
Fundamentals of Data Analysis
• EIA
– Evidence: just the #s
– Inference: interpretation
– Action: steps to be taken
8
Starting with Question(s)
• Where are areas of lower
performance? (analysis by
variables)
• Geographic areas of the state
• Child/family characteristics
• Program characteristics
9
Starting with Question(s)
• Where are areas of lower performance?
(analysis by child characteristics)
– Does our program serve some children more effectively than others?
» Do outcomes vary for children with different racial/ethnic
backgrounds?
» Are outcomes different for Dual Language Learners as compared
to mono-language learners?
10
Starting Points
Potentially starting
with:
• An Issue (e.g.
shifting
demographic)
• An Initiative
• Child outcomes
data
11
Questions and Existing Initiatives
• What is the state performance in social emotional
development compared to other outcome areas?
• What have our trends been in the area of social
emotional development for young children?
• Are there certain areas of the state that have lower
performance in the area of social emotional
development for young children?
• Do social emotional outcomes differ by child
characteristics (race/ethnicity, socioeconomic
status, geographic region of the state)?
12
Broad Data and Infrastructure
Analysis
• Purpose – Explore child results (and potentially
the related family results) and practices that
would be justifiable targets for improvement
• Goal – Assemble evidence to substantiate to
leadership and stakeholders why you selected a
particular result
• Strategies – Analysis of child results and related
data to identify areas of lower performance
13
Types of Broad Data Analysis
Analysis of child outcomes data
• By summary statement
• State data compared to national data
• Local data comparisons across the state
• State trend data
• Analysis by race/ethnicity, disability, income
Analysis of related family outcomes data
• State data compared to national data
• Local data comparisons across the state
• State trend data
• Analysis by race/ethnicity, income, length of time in program
• Linked to child outcomes data
Resource: Broad Data Analysis
Template
• Purpose: to look at
how children in the
state are performing
relative to national
data, across years,
within the state and
by comparisons
across programs
http://ectacenter.org/eco/assets/docs/SSIP_child_outcomes_broad
_data_analysis_template_FINAL.docx
15
Resource: Broad Data Analysis
Template
16
Resource: Broad Data Analysis
Template
17
Meaningful differences calculator
• Purpose: to look
at statistical
significance of
change in state SS
data from year to
year; and allow
comparison of
local to state
http://ectacenter.org/eco/pages/summary.asp#meaningfuldiffcalc
18
Resource: Analyzing Child Outcomes
Data for Program Improvement
• Quick reference tool
• Consider key issues,
questions, and
approaches for analyzing
and interpreting child
outcomes data.
http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutc
omesData-GuidanceTable.pdf
19
Guidance Table
20
What is the likely child result that
will be the focus of your SSIP?
•
•
•
•
•
Social Relationships
Knowledge and Skills
Action to Meet needs
All three of the above
Something other than above
21
In-Depth Data and Infrastructure
Analysis
• Purpose – Conduct further analysis exploring the link
between the practices and infrastructure and the
child result.
• Goal – Gather sufficient evidence to link specific
practices and infrastructure to child results (to
inform needed improvement strategies).
• Strategies - Subgroup analysis, comparisons of
programs, “root cause analysis,” local data drilldown, narrative summary of analysis
22
Identify Root Causes Contributing to
Low Performance
• Analyze data at the local level
• Identify factors contributing to low
performance (including infrastructure)
• Contributing factors:
– Explain why you have
the problem
– Point to how the problem
can be addressed
23
Identify Root Causes Contributing to
Low Performance
• Identify barriers for each
contributing factor
– What is standing in the
way of addressing this
contributing factor?
– Why hasn’t it been
addressed to date?
24
Resource: Subgroup Analysis
Template
• Purpose: to provide
states with table
shells for subgroup
analyses that have
proven useful in
understanding
predictors of child
outcomes.
http://ectacenter.org/eco/pages/usingdata.asp
25
Subgroup Analysis Example
26
Local Contributing Factor Tool
http://ectacenter.org/~docs/eco/ECO-C3-B7-LCFT.docx
http://ectacenter.org/~docs/topics/gensup/14-ContributingFactorResults_Final_28Mar12.doc
27
LCFT: Question Categories
System/
Infrastructure
Practitioner/
Practices
Policies/ procedures
Competencies of
staff
Funding
Implementation of
effective practices
Training/TA
Time
Supervision
Resources
Data
Supports
Personnel
28
Data Quality
• Not the focus of the SSIP
• But must be addressed
in the SSIP
– Describe data quality
issues
– Describe data quality
improvement efforts
Data Quality: Pattern Checking
• Checking
predictable
patterns to
help
determine ‘red
flags’ to be
investigated in
the data.
http://ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf
Data Quality Profiles
• The profiles include
information about:
– State vs. national
– Data quality criteria
used for national
analysis
• Completeness of data
• Progress categories
patterns
– Trends over time
Contact: Abby Winer abby.winer@sri.com
Data Analysis State Example
Virginia Part C
Beth Tolley
Kyla Patterson
Starting Point
• Child and family outcome data
• Stakeholder Input
– State Interagency Coordinating Council (VICC)
– Local System Managers
Preparation
• Child Outcomes Broad Data Analysis Template
• Data Quality Profile
• Support from ECTA and DaSy
Process
• Overview of SSIP
• Powerpoint presentations and handouts with
data and analysis questions
• Large and small group discussion and input
Broad Analysis Questions
• Does our state’s data look different than the
national data?
• Are our state outcomes trends stable over time?
– Is the data trending upwards?
– Is the data trending downwards?
• Is our state performing more poorly in some
outcomes than others?
• Are the outcomes similar across programs?
• What about data quality? Can we be confident
in our data?
Child Outcomes: National vs. State
FFY11 and State FFY12
90%
80%
70%
60%
50%
National FFY11
Virginia FFY11
40%
Virginia FFY12
30%
20%
10%
0%
SS1 - SE
SS1 - KS
SS1 - TA
SS2 - SE
SS2 - KS
SS2 - TA
National Vs. State
Meaningful Differences
Virginia Trends
Virginia Trends
State (n=3555)
N (n=64)*
A (n=22)*
J (n=35)*
BB (n=46)*
NN (n=7)
T (n=24)*
CC (n=33)*
H (n=28)*
MM (n=27)*
K (n=79)*
E (n=32)
W (n=124)*
FF (n=146)*
G (n=125)*
S (n=102)
Y (n=556)
V (n=90)
D (n=39)
R (n=84)
B (n=100)
Q (n=133)
JJ (n=61)
L (n=70)
HH (n=46)
M (n=114)
II (n=107)
I (n=122)
KK (n=37)
P (n=135)
U (n=164)*
X (n=129)*
GG (n=113)*
LL (n=52)
Z (n=40)
DD (n=125)*
EE (n=14)
O (n=181)*
C (n=32)
F (n=48)*
AA (n=40)*
Child Outcomes: Local vs. State
FFY 2012: Actions to Meet Needs, Exited within Age Expectations
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Data Quality Elements
– Completeness of data
• number of children reported for the outcome/number who exited
• Virginia’s results: average= 65%; range for Local Systems = 17% 100%
– Expected Patterns for Progress Categories
Category a
0
>10%
Category e
<5%
>65%
• Virginia’s state date is within these parameters for all three
outcomes
– Child Outcomes State Trends Over Time
• As noted on previous slides, Virginia’s results do not show wide
variations which would trigger concerns about data quality
Family Outcomes:
State Trends over Time
Family Outcomes: Local vs. State 2012-2013
4C: EI has helped the family help their child develop and learn
Identification of Area of Concern
• Based on broad data analysis of child and
family outcome data
• VICC and LSM identified same area of
concern:
– Outcome 3C – Use of appropriate behaviors
(taking action) to meet needs
In-Depth Data Analysis
Purpose:
•To identify the specific measurable result
•To identify root causes and contributing factors
– why is it happening?
Stakeholder Questions
• Does the child’s reason for eligibility impact
results on this outcome?
• Does age at entry or length of time in EI
impact results?
• Does use of Part B entry ratings as Part C exit
ratings have an impact?
• Is there consistent understanding of the
developmental areas involved in determining
a rating on this indicator?
Next Steps
• Disaggregate data by child characteristics
– Age at entry
– Length of time in service
– Race/ethnicity
– Reason for eligibility
• Analyze data by local system
• Continue to ask why questions
Methods
• National Resources, such as:
– SSIP subgroup analysis template
– Analyzing child outcomes for program
Improvement
• Joint analysis with Local System Managers
through regional meetings
• Back to the VICC in June
Section 619 and the SSIP
Some considering:
•
•
•
•
619 incorporated into the Part B
619 focus for the Part B
Part C and Section 619
Coordination across 0-21
53
Richard Henderson
Evelyn S.Dunstan
Johnson
Shannon
Early Childhood and Interagency Coordinator
Division of Special Education
Division of Student Achievement and School Improvement
Idaho State Department of Education
Idaho State Systems Improvement Plan
Focus Shifting to Performance
K-12 Priorities
Indicator 3: Participation and Performance on Statewide
Assessments
Indicator 5: Participation/Time in General Education Settings (LRE)
Indicator 14: Post School Outcomes
Big B
Priorities
619
Alignment
Indicator
3
Indicator
7
Indicator
5
Indicator
6
Social and Emotional Outcome Data
100
90
80
70
60
Outcome A1
50
Outcome A2
40
30
20
10
0
FFY 2009
FFY 2010
FFY 2011
FFY 2012
Parents
Child
Care
Providers
Social and
Emotional
Development
Head
Start
Programs
School
Districts
The Protective Factors Framework
•
•
•
•
•
Parental Resilience
Social Connections
Concrete Support in Times of Needs
Knowledge of Parenting and Child Development
Social and Emotional Competence of Children
Statewide Systematic Implementation of
Social and Emotional training across
Programs
Other activities
• Send out a Statewide surveys to: Special
Education Preschools, Head Start, and Child
Care Providers to define need and readiness
• Early Childhood Coordinating Council has
adopted this as priority for Head Start
subcommittee
• Attend a day long statewide planning meeting
as preconference to NAEYC
Contact Information:
Shannon Dunstan
Early Childhood & Interagency Coordinator
Idaho State Department of Education
Division of Student Achievement and School Improvement
Division of Special Education
(208) 332-6908
sdunstan@sde.idaho.gov
Summary
Fundamentals of Data Analysis
• Starting with a question (or questions)
• Evidence, Inference, Action process
Broad data Analysis
• To explore child results and related family results and practices that
would be justifiable targets for improvement
• Resources: Broad Data Analysis Template, Analyzing Child Outcome
Data, Meaningful Differences Calculator
In-depth Data Analysis
• To look at local level data and identify causes of low performance
• Resources: LCFTs, Subgroup Analysis
Data Quality
• Describe concerns and how the state will address these concerns.
• Resources: State Profiles, Pattern Checking Document
Contact Information
Christina Kasprzak, ECTA/DaSy
Christina.Kasprzak@unc.edu
Anne Lucas, WRRC/ECTA
Anne.Lucas@unc.edu
Cornelia Taylor, ECTA/DaSy
cornelia.taylor@sri.com
Megan Vinh, WRRC
Mvinh@uregon.edu
Abby Winer, ECTA/DaSy
Abby.winer@sri.com
64
IDEA DATA
CENTER
Thank you for your attention!
This is the second in a series on SSIP presented in 2014. Resources
related to this call and other presentations in the series are available at
the following URL:
http://ectacenter.org/~calls/2014/ssip/ssip.asp