CESA 6
IMPROVING THE QUALITY OF
CHILD OUTCOMES DATA
Ruth Chvojicek – Statewide Part B Indicator 7
Child Outcomes Coordinator
OBJECTIVES
•
•
To discuss the significance of and strategies for
improving child outcomes data quality
To look at state, CESA and District data patterns as
one mechanism for checking data quality
2
QUALITY ASSURANCE:
LOOKING FOR QUALITY DATA
I know it is in here
somewhere
ONGOING CHECKS FOR DATA QUALITY
Good training and assessment
Before
Efficient data systems
Timely and accurate data entry
Ongoing supervision of
implementation
During
Feedback to implementers
Refresher training
Review sample of completed COSFs
After
Pattern checking analysis
4
PROMOTING QUALITY DATA •
Through data systems and verification, such as
•
•
•
•
Monthly data system error checks e.g. missing &
inaccurate data
Monthly email data reminders
Indicator Training data reports
Good data entry procedures
5
LOOKING AT DATA
6
USING DATA FOR PROGRAM
IMPROVEMENT = EIA
Evidence
Inference
Action
7
EVIDENCE

Evidence refers to the
numbers, such as
“45% of children
in category b”

The numbers are not
debatable
8
INFERENCE
How do you interpret the #s?
 What can you conclude from the #s?
 Does evidence mean good news? Bad news? News
we can’t interpret?
 To reach an inference, sometimes we analyze data in
other ways (ask for more evidence) “Drill Down”

9
INFERENCE
 Inference
is debatable -- even reasonable
people can reach different conclusions
 Stakeholders (district personal) can help with
putting meaning on the numbers
 Early on, the inference may be more a
question of the quality of the data
10
ACTION
•
•
•
•
•
Given the inference from the numbers, what
should be done?
Recommendations or action steps
Action can be debatable – and often is
Another role for stakeholders
Again, early on the action might have to do
with improving the quality of the data
11
PROMOTING QUALITY DATA
THROUGH DATA ANALYSIS
12
THE THREE OUTCOMES
Percent of preschool children with IEPs who demonstrate improved:
Positive Social Emotional Skills
Acquisition & Use of Knowledge &
Skills
Takes Appropriate Action to Meet
Needs
7-POINT RATING SCALE
PLEASE REFER TO HANDOUT – “THE BUCKET LIST”
1
2
3
• Child does not yet
show functioning
expected of a child
his or her age in any
situation.
• Child’s functioning
does not yet
include immediate
foundational skills
upon which to build
age-appropriate
functioning.
• Child functioning
reflects skills that
developmentally
come before
immediate
foundational skills.
• Child’s functioning
might be described
as like that of a
much younger
child.
• Child occasionally
uses immediate
foundational skills
across settings and
situation. More
functioning reflects
skills that are not
immediate
foundational than
are immediate
foundational.
• Child does not yet
show functioning
expected for a child
of his or her age in
any situation.
• Child uses
immediate
foundational
skills, most or all of
the time, skills upon
which to build ageappropriate
functioning.
• Functioning might
be described as like
that of a younger
child.
4
• Child shows
occasional ageappropriate
functioning across
settings and
situations. More
functioning is not
age-appropriate
than ageappropriate.
5
6
7
• Child shows
functioning
expected for his or
her age some of the
time and/or in
some settings and
situations. Child’s
functioning is a mix
of age-appropriate
and not ageappropriate
behaviors and skills.
• Childs functioning
might be described
as like that of a
slightly younger
child.
• Child’s functioning
generally is
considered
appropriate for his
or her age but there
are some
significant
concerns about the
child’s functioning
in this outcome
area. These
concerns are
substantial enough
to suggest
monitoring or
possible additional
support.
• Although ageappropriate, the
child’s functioning
may border on not
keeping pace with
age expectations.
• Child shows
functioning
expected for his or
her age in all or
almost all
everyday
situations that are
part of the child’s
life. Functioning is
considered
appropriate for his
or her age.
• No one has any
concerns about the
child’s functioning
in this outcome
area.
PATTERN CHECKING - CHECKING TO SEE IF RATINGS
ACCURATELY REFLECT CHILD STATUS
•
We have expectations about how child outcomes data
should look
Compared to what we expect
• Compared to other data in the state
• Compared to similar states/regions/school districts
•
•
When the data are different than expected ask follow
up questions
15
QUESTIONS TO ASK

Do the data make sense?


Am I surprised? Do I believe the data? Believe some of
the data? All of the data?
If the data are reasonable (or when they become
reasonable), what might they tell us?
16
PATTERNS WE WILL BE CHECKING TODAY
1.
2.
3.
4.
5.
6.
7.
8.
Entry Rating Distribution
Entry Rating Distribution by Eligibility Determination
Comparison of Entry Ratings Across Outcomes
Entry/Exit Comparison by CESA
State Entry Rating Distribution by Race/Ethnicity
State Exit Rating Distribution
Progress Categories by State/CESA
Summary Statements by State/CESA
17
SMALL GROUP DISCUSSION QUESTIONS:
1.
2.
3.
4.
5.
What do you notice about your local data?
What stands out as a possible ‘red flag’?
What might you infer about the data?
What additional questions does it raise?
What next steps might you take?
18
PREDICTED PATTERN #1
Children will differ
from one another
in their entry
scores in
reasonable ways
(e.g., fewer scores
at the high and
low ends of the
distribution, more
scores in the
middle).
Rationale:
Evidence suggests EI and
ECSE serve more mildly
than severely impaired
children (e.g., few
ratings/scores at lowest
end). Few children
receiving services would
be expected to be
considered as
functioning typically
(few ratings/scores in
the typical range).
19
State 11-12 Entry Rating Distribution
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1
3.6%
10.8%
9.3%
13.8%
21.4%
20.9%
20.2%
Outcome 2
4.7%
14.2%
16.3%
19.7%
27.8%
14.2%
3.1%
Outcome 3
3.3%
7.5%
8.5%
11.5%
17.3%
21.2%
30.7%
PREDICTED PATTERN #2
Groups of children with more severe disabilities
should have lower entry numbers than
groups of children with less severe
disabilities.
21
STATE 11-12 ENTRY RATING ELIGIBILITY PERCENTAGES
Autism
Cognitive Disability
Deaf-Blind
Emotional Behavioral Disability
Hearing Impairment
Orthopedic Impairment
Other Health Impairment
Significant Developmental Delay
Specific Learning Disability
Speech or Language Impairment
Traumatic Brain Injury
Visual Impairment
3.7%
0.9%
0.0%
0.9%
0.8%
0.9%
3.4%
21.2%
0.2%
67.6%
0.1%
0.3%
Outcome 1
Autism
Cognitive Disability
Deaf-Blind
Emotional Behavioral Disability
Hearing Impairment
Orthopedic Impairment
Other Health Impairment
Significant Developmental Delay
Specific Learning Disability
Speech or Language Impairment
Traumatic Brain Injury
Visual Impairment
1
20.4%
24.6%
0.0%
4.5%
3.4%
13.8%
17.3%
7.3%
6.3%
0.4%
36.4%
0.0%
2
38.9%
33.3%
0.0%
21.2%
8.5%
10.8%
21.7%
24.5%
0.0%
4.0%
36.4%
19.0%
3
14.3%
24.6%
50.0%
12.1%
5.1%
6.2%
14.2%
18.3%
12.5%
5.8%
9.1%
14.3%
4
17.9%
10.1%
50.0%
39.4%
5.1%
6.2%
23.6%
24.7%
12.5%
9.6%
9.1%
19.0%
5
7.1%
5.8%
0.0%
18.2%
40.7%
24.6%
15.4%
17.5%
25.0%
23.8%
9.1%
4.8%
6
1.4%
1.4%
0.0%
4.5%
18.6%
23.1%
6.3%
5.6%
31.3%
27.9%
0.0%
33.3%
7
0.0%
0.0%
0.0%
0.0%
18.6%
15.4%
1.6%
2.0%
12.5%
28.6%
0.0%
9.5%
Outcome 2
Autism
Cognitive Disability
Deaf-Blind
Emotional Behavioral Disability
Hearing Impairment
Orthopedic Impairment
Other Health Impairment
Significant Developmental Delay
Specific Learning Disability
Speech or Language Impairment
Traumatic Brain Injury
Visual Impairment
1
16.1%
37.7%
0.0%
0.0%
5.1%
16.9%
19.7%
8.6%
6.3%
1.4%
45.5%
14.3%
2
32.5%
42.0%
0.0%
4.5%
13.6%
12.3%
20.5%
26.7%
6.3%
8.7%
36.4%
23.8%
3
13.9%
14.5%
50.0%
4.5%
16.9%
9.2%
17.3%
23.6%
25.0%
14.4%
0.0%
9.5%
4
20.7%
2.9%
50.0%
12.1%
18.6%
6.2%
10.6%
21.2%
31.3%
20.2%
9.1%
0.0%
5
9.6%
2.9%
0.0%
19.7%
27.1%
21.5%
18.1%
14.2%
12.5%
34.1%
9.1%
28.6%
6
6.4%
0.0%
0.0%
39.4%
13.6%
23.1%
8.3%
4.6%
12.5%
17.6%
0.0%
23.8%
7
0.7%
0.0%
0.0%
19.7%
5.1%
10.8%
5.5%
1.1%
6.3%
3.5%
0.0%
0.0%
Outcome 3
Autism
Cognitive Disability
Deaf-Blind
Emotional Behavioral Disability
Hearing Impairment
Orthopedic Impairment
Other Health Impairment
Significant Developmental Delay
Specific Learning Disability
Speech or Language Impairment
Traumatic Brain Injury
Visual Impairment
1
11.8%
27.5%
0.0%
1.5%
5.1%
21.5%
20.5%
6.1%
6.3%
0.5%
45.5%
4.8%
2
29.3%
49.3%
0.0%
4.5%
6.8%
16.9%
15.4%
16.6%
6.3%
2.4%
9.1%
14.3%
3
18.9%
13.0%
0.0%
7.6%
6.8%
20.0%
17.7%
17.4%
6.3%
4.4%
18.2%
14.3%
4
19.3%
4.3%
50.0%
18.2%
10.2%
16.9%
16.5%
23.2%
0.0%
7.1%
27.3%
14.3%
5
12.5%
4.3%
50.0%
37.9%
16.9%
16.9%
18.5%
21.4%
25.0%
16.1%
0.0%
19.0%
6
5.4%
1.4%
0.0%
15.2%
25.4%
7.7%
9.1%
12.0%
37.5%
26.0%
0.0%
33.3%
7
2.9%
0.0%
0.0%
15.2%
28.8%
0.0%
2.4%
3.3%
18.8%
43.5%
0.0%
0.0%
PREDICTED PATTERN #3
Functioning at entry in one outcome is related
to functioning at entry in the other
outcomes.
For cross tabulations we should expect most
cases to be in the diagonal and the other to
be clustered on either side of the diagonal.
29
Outcome 1
Outcome 2
1
2
3
4
5
6
7
1
13
4
1
2
6
47
20
9
9
5
1
3
4
5
6
7
17
33
21
27
13
7
6
13
51
52
22
18
2
2
25
70
91
84
3
2
14
13
48
123
1
1
1
2
10
Outcome 2
Outcome 3
1
2
3
4
5
6
7
1
12
4
1
2
4
31
10
3
1
26
34
5
3
4
15
20
42
5
2
5
1
9
31
49
61
22
6
7
9
15
41
84
43
3
3
8
25
121
135
12
Outcome 1
Outcome 3
1
2
3
4
5
6
7
1
13
2
1
1
2
4
30
6
3
1
1
3
1
21
29
10
6
2
4
14
19
32
13
5
1
5
1
9
10
48
77
22
6
6
7
3
5
22
57
71
37
1
1
5
19
81
197
PREDICTED PATTERN #4
Large changes in
status relative to
same age peers
between entry and
exit from the
program are
possible but rare.
When looking at the
Entry/Exit Rating
comparison for
individual children we
would expect very few
children to increase
more than 3 points.
33
Outcome 1 Exit
Outcome 1 Entry
1
1
2
3
4
5
1
2
1
1
1
4
3
6
6
2
3
1
1
4
6
2
3
4
6
5
4
11
20
1
4
18
2
19
2
3
1
4
5
6
7
1
2
6
7
1
Outcome 2 Entry
Outcome 2 Exit
1
2
3
6
7
1
2
1
1
2
1
3
6
4
12
2
2
1
4
5
9
10
1
6
10
4
5
10
10
15
6
1
9
15
3
4
7
4
5
1
1
Outcome 3 Exit
1
2
3
4
5
6
3
3
1
1
1
1
3
2
4
1
2
2
3
5
6
4
2
3
9
6
5
1
5
7
13
6
1
1
2
21
4
32
Outcome 3 Entry
1
2
3
7
1
7
PREDICTED PATTERN #5
If children across
race/ethnicity
categories are
expected to
achieve similar
outcomes, there
should be no
difference in
distributions
across
race/ethnicity.
Note:
Wisconsin began
gathering race/ethnicity
data for Indicator 7 on
July 1, 2011. This
impacts the data on the
graphs being reviewed
today.
37
Outcome 1
Asian
Black
Hispanic
American Indian Alaskan
Hawaiian Other Pacific Islander
Two or More Races
White
1 2 3 4 5 6 7
4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%
6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%
4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%
2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%
0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%
4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%
2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%
Outcome 2
Asian
Black
Hispanic
American Indian Alaskan
Hawaiian Other Pacific Islander
Two or More Races
White
1 2 3 4 5 6 7
5.7% 14.9% 24.1% 20.6% 22.0% 12.8% 0.0%
7.1% 20.0% 23.6% 19.7% 20.4% 7.6% 1.6%
7.6% 19.3% 22.8% 20.7% 19.8% 8.0% 1.7%
2.7% 21.8% 17.3% 27.3% 21.8% 9.1% 0.0%
10.5% 15.8% 15.8% 42.1% 10.5% 0.0% 5.3%
4.5% 12.6% 12.6% 23.4% 30.6% 10.8% 5.4%
3.6% 11.8% 13.3% 19.1% 31.2% 17.1% 3.9%
Outcome 3
Asian
Black
Hispanic
American Indian Alaskan
Hawaiian Other Pacific Islander
Two or More Races
White
1 2 3 4 5 6 7
4.3% 7.8% 5.7% 11.3% 15.6% 22.0% 33.3%
6.0% 10.5% 11.5% 13.5% 18.8% 20.3% 19.5%
3.6% 9.9% 8.6% 13.0% 16.8% 25.6% 22.5%
2.7% 5.5% 13.6% 15.5% 22.7% 20.0% 20.0%
15.8% 5.3% 21.1% 15.8% 15.8% 5.3% 21.1%
2.7% 4.5% 7.2% 13.5% 20.7% 22.5% 28.8%
2.7% 6.6% 7.8% 10.6% 17.0% 20.5% 34.9%
PREDICTED PATTERN #6
Children will differ from one another in their
exit scores in reasonable ways. (At exit
there will be a few children with very high
or very low numbers.
41
42
OSEP PROGRESS CATEGORIES
43
PROGRESS CATEGORIES
PLEASE REFER TO HANDOUT “CHILD OUTCOMES DATA CONVERSION”
Percentage of children who:
a. Did not improve functioning
b. Improved functioning, but not sufficient to move
nearer to functioning comparable to same-aged
peers
c. Improved functioning to a level nearer to same-aged
peers but did not reach it
d. Improved functioning to reach a level comparable to
same-aged peers
e. Maintained functioning at a level comparable to
same-aged peers
44
PREDICTED PATTERN #7
Children will differ
from one another
in their OSEP
progress
categories in
reasonable ways.
Note – A graph of this
predicted pattern
should have a similar
distribution expected in
entry & exit ratings (bell
curve).
46
2010-2011 Outcome 1
Progress Category Percentages
45.0
40.0
35.0
Axis Title
30.0
25.0
20.0
15.0
10.0
5.0
0.0
National
A
1.0
B
7.0
C
31.0
D
44.0
E
17.0
Wisconsin
1.5
11.5
21.3
28.4
37.4
2010-2011 Outcome 2
Progress Category Percentages
45.0
40.0
35.0
Axis Title
30.0
25.0
20.0
15.0
10.0
5.0
0.0
National
A
1.0
B
8.0
C
32.0
D
42.0
E
17.0
Wisconsin
1.2
15.2
28.9
41.2
13.6
2010-2011 Outcome 3
Progress Category Percentages
50.0
45.0
40.0
35.0
Axis Title
30.0
25.0
20.0
15.0
10.0
5.0
0.0
National
A
1.0
B
5.0
C
23.0
D
44.0
E
27.0
Wisconsin
1.0
8.0
12.3
29.5
49.2
SUMMARY STATEMENT #1
PLEASE SEE HANDOUT – “SUMMARY STATEMENTS FOR TARGET SETTING
1.
Of those children who entered the program
below age expectations in each Outcome, the
percent who substantially increased their
rate of growth by the time they exited the
program.
c + d___
a+b+c+d
51
SUMMARY STATEMENT #2
2.
The percent of children who were
functioning within age expectations in each
Outcome by the time they exited the
program.
d + e__
a+b+c+d+e
52
2010-2011 Outcome 1
Social Emotional
100
90
80
70
Axis Title
60
50
40
30
20
10
0
SS1: Greater than Expected Growth
SS2: Exited Within Age Expectations
National
90.8
61.3
Wisconsin
79.3
65.7
2010-2011 Outcome 2
Knowledge & Skills
100
90
80
70
Axis Title
60
50
40
30
20
10
0
SS1: Greater than Expected Growth
SS2: Exited Within Age Expectations
National
89.4
59.2
Wisconsin
80.7
54.7
2010-2011 Outcome 3
Action to Meet Needs
100
90
80
70
Axis Title
60
50
40
30
20
10
0
SS1: Greater than Expected Growth
SS2: Exited Within Age Expectations
National
91.3
71.1
Wisconsin
82.1
78.6
Wrap Up
&
Next Steps
58
TAKE HOME MESSAGE
If you conclude the data are not (yet) valid, they
cannot be used for program effectiveness, program
improvement or anything else.
SO
What do you if the data are not as good as they
should be?
Answer: Continue to improve data collection
through ongoing quality assurance
KEY TO GOOD DATA
Have a good
outcome
measurement
60
QUALITY CHECKS DURING DATA COLLECTION
Ongoing supervision of
implementation
During
Feedback to implementers
Refresher training
61
QUALITY REVIEW OF COSF TEAM DISCUSSION:
•
•
•
•
•
•
Team discussion
Parent input
Documentation of rating
discussion
Multiple assessment
sources
Child’s everyday
functioning – Not test
scores
Child’s full range of
functioning (AE/IF/F)
Ongoing Supervision:
• Feedback to teams
critical
• Refresher training
• Beware of “auto pilot”
and “drift”
Methods:
• Observation
• Videos
62
QUALITY REVIEW OF COMPLETED COSFS
•
•
•
•
•
•
Complete?
Adequate evidence?
Match the outcome area?
Based on functional behaviors?
Across settings and situations?
Ratings consistent with evidence?
63
NEXT STEPS?
1.
2.
3.
What steps might you take after today’s discussion?
What other data patterns would be interesting /
helpful to look at and interpret?
What types of TA/support/follow up might be
appropriate next steps?
64
Thank You for
Participating Today!
For more information:
www.collaboratingpartners.com
http://sped.dpi.wi.gov/sped_spp-preout
http://projects.fpg.unc.edu/~eco/index.cfm