Summary
Briefing
State of Florida
Child and Family
Services Review
-April 1 through September 30
What are the CFSRs?
• Amendments to the Social Security Act in 1994 mandated the U.S. Department
of Health and Human Services (USDHHS) to determine a process for ensuring
conformity with titles IV-B and IV-E of the Social Security Act
• The Adoption and Safe Families Act of 1997 directed USDHHS to develop a set
of outcome measures that could be used to assess state performance in
achieving the goals of safety; permanency; and child and family well-being
• In 2000, a final rule was published in the Federal Register that established a
review system for monitoring state child welfare programs
• This system, called the Child and Family Services Reviews (CFSRs), is
administered by the Children’s Bureau (a division of of USDHHS)
CFSR History
• CFSR Round 1
– Conducted between 2001 and 2004
– Data indicators to measure state performance consisted of 6 items (2 safety/4
permanency)
• CFSR Round 2
– Conducted between 2007 and 2010
– Data indicators consisted of 6 items, but the permanency standards were
expanded to 15 different measures that were distilled into four composites
• CFSR Round 3
– Began in 2015
– Data indicators have been revised and consist of 7 items (2 safety/5
permanency)
CFSR: Overall Goals
• Ensure conformity with title IV-B and IV-E child welfare
requirements
• Determine what is happening to children and families
who have contact with the child welfare system
• Support states to enhance their capacity to improve
outcomes and systems for children and families
Seven Outcomes
Safety
1. Children are, first and foremost, protected from abuse and neglect.
2. Children are safely maintained in their homes whenever possible and
appropriate.
Permanency
3. Children have permanency and stability in their living arrangements.
4. The continuity of family relationships and connections is preserved for
children.
Child and Family Well-Being
5. Families have enhanced capacity to provide for their children's needs.
6. Children receive appropriate services to meet their educational needs.
7. Children receive adequate services to meet their physical and mental
health needs.
Seven Systemic Factors
1.
2.
3.
4.
5.
6.
7.
Statewide Information System
Service Array
Case Review System
Staff Training
Quality Assurance System
Agency Responsiveness to the Community
Foster & Adoptive Parent Licensing, Recruitment, & Retention
CFSR3: What Has NOT Changed
• The process:
1. Data profile (3 years data trends and national standards)
2. Statewide assessment
3. Onsite review
• Assessment based on the same 7 outcomes and 7 systemic
factors using national standards and measures of expected
performance
• States found to be “not in conformance” will continue to
have the opportunity to improve their programs before
facing the possibility of financial penalties
Principles of Reviews
• The reviews are a collaborative effort between the
federal and state governments.
• The reviews examine state programs from two
perspectives.
– First, they assess the outcomes of children and families served
by the state’s child welfare agencies.
– Second, they examine identified systemic factors that affect
the ability of state child welfare agencies to help children and
families achieve positive outcomes.
Principles of Reviews
• The review process collects information from a variety of
sources so the Children’s Bureau can make determinations
about a state’s performance. These sources include
– the statewide assessment (and by cross-reference, the Florida Child
and Family Services Plan or Annual Progress and Services Reports);
– statewide data indicators;
– case records;
– case-related interviews with children, parents, foster parents,
caseworkers, and other professionals; and
– interviews with Tribes, partners, and stakeholders, as necessary.
Principles of Reviews
• The reviews capture state program strengths and areas
needing improvement. They include a program
improvement process that states use to make
improvements, where needed, and build on an agency’s
identified strengths.
• The reviews promote the development of Program
Improvement Plans designed to strengthen Florida’s capacity
to create positive outcomes for children and families. The
reviews promote ongoing state self-evaluation of programs
and outcomes.
Principles of Reviews
• The reviews are best supported by a state’s maintaining and
enhancing its quality assurance system through a continuous
quality improvement approach so that ongoing
measurement of service quality can promote continuous
improvement in outcomes for the children and families
served by the state.
• The reviews, and the results thereof, emphasize
accountability. While the review process includes
opportunities for Florida to make program improvements
before having federal funds withheld for nonconformity,
significant penalties are associated with the failure to make
the identified improvements needed to improve outcomes.
Importance of Collaboration
• The Child and Family Services Reviews promote change through
collaboration that begins between the federal and state governments as
they assess the effectiveness of child welfare agencies in serving
children and families, and continues between child welfare agency
leaders and their internal and external partners.
• Federal and state staff partner throughout the Child and Family Services
Reviews process, but final decisions are the responsibility of federal
staff.
• Most important, this collaborative process should result in changes that
promote improved outcomes for children and families.
Structure of Reviews
• The Child and Family Services Reviews involve a two-phase process:
– (1) a statewide assessment and
– (2) an onsite review as required by 45 CFR § 1355.33. If needed, a state will
develop and implement a Program Improvement Plan to improve upon areas
identified as not in substantial conformity.
• In the first phase, a statewide assessment is completed, using statewide
data indicators and other data to evaluate the programs under review
and examine the outcomes and systemic factors subject to review.
• The second phase of the review process is an onsite review, which
includes case reviews, case-related interviews for the purpose of
determining outcome performance, and, as necessary, stakeholder
interviews that further inform the assessment of systemic factors.
CFSR Process
States submit data to to two systems: the National Child Abuse and Neglect Data System (NCANDS) and the
Adoption and Foster Care Analysis and Reporting System (AFCARS)
The Children’s Bureau (CB) of the Administration for Children and Families sends each state a State Data
Profile of the state’s performance relative to national standards for three years prior to the review period
Using the most recent year of data from the State Data Profile, states develop a Statewide Assessment, a
report on child and family outcomes and systemic factors, and submits it to CB
Onsite review is conducted based on the results of the statewide assessment and involves stakeholder
interviews and case reviews. States meeting specific criteria can conduct their own case reviews using the
federal review instrument
States held responsible for meeting “substantial conformity” on safety, permanency, and well-being
outcomes and systemic factors
Key Steps in the Review Process
• State completion and submission of the Statewide Assessment Instrument to
the Children’s Bureau by March 25, 2016
• Joint preparation for the onsite review, including:
– Participation in planning conference calls
– Discussion of review sites, review team structure, and Reviewers
– Sampling activities
– Managing logistics for the onsite review
– Selection of stakeholder interviewees and scheduling of stakeholder and case specific
interviews
Key Steps in the Review Process
– Case Debriefing and results discussion
– Completion of the onsite review, including case reviews, stakeholder interviews, quality
assurance of information, reconciliation of findings, and conducting debriefings/results
discussion
– Children’s Bureau analysis of review data and issuance of the Final Report with
determinations of substantial conformity
– State development of the Program Improvement Plan, as necessary, in consultation with the
Children’s Bureau
– Children’s Bureau approval of the Florida's Program Improvement Plan
– Joint evaluation of progress in meeting Program Improvement Plan goals
– State Program Improvement Plan completion
CFSR ONSITE REVIEW
Rolling Monthly Sample Period
•
Case reviews will be conducted April 1, 2016-September 30, 2016.
Review Months
April 2016
Rolling Monthly Sample
Periods*
4/1/2015 to 9/30/2015
Periods Under Review
4/1/2015 to Date of Review
May 2016
5/1/2015 to 10/31/2015
5/1/2015 to Date of Review
June 2016
6/1/2015 to 11/30/2015
6/1/2015 to Date of Review
July 2016
7/1/2015 to 12/31/2015
7/1/2015 to Date of Review
August 2016
8/1/2015 to 1/31/2016
8/1/2015 to Date of Review
September 2016
9/1/2015 to 2/29/2016
9/1/2015 to Date of Review
NOTE:
The sample period is April 1, 2015-September 30, 2015.
Add 45 days for in-home services sample periods
CFSR3: What Has Changed
• Allows for ACF to approve a state conducted review which allows
the state to complete the case reviews and case specific
stakeholder interviews.
• ACF approved Florida’s application to be a state conducted review
in December 2015.
• Case reviews will be conducted jointly by region and CBC QA staff.
CBC QA staff will lead the case review process.
• Samples increased from 60 to 80 cases statewide.
• The Children’s Bureau has created a web based case review
system known as the Online Monitoring System and all CBCs
currently have completed the federal training and are conducting
case reviews in the system.
• ACF approved Florida’s application to be a state conducted review
in December 2015.
• All CBCs will review cases served in-home and out-of-home.
• Samples increased from 60 to 80 cases statewide.
Total Reviews by Region
Based on CBC percentage of the statewide total for each placement type
Out-of-Home Care
Northea s t Regi on
Centra l Regi on
Suncoa s t Regi on
Southea s t Regi on
South Regi on
Child Protective
Statewide
N=30,049
Percentage of
chi l dren i n OHC
by Regi on
Percent refl ects
the number of
chi l dren i n OHC
i n ea ch regi on
di vi ded by the
total number of
chi l dren i n OHC
s tatewi de
Ca s es to
Revi ew
N=55
2776
9.24%
5
612
4239
14.11%
8
6771
22.53%
7835
Investigations
Statewide
N=5,164
Totals
In Home Services Families
Sample
Sample Split
Out-of- In Home
Home
Servi ce
Ca re
Ca s es
1
Statewide
N=11,555
Percentage of
chi l dren i n
OHC by
Regi on
Percent
refl ects the
number of
chi l dren i n
OHC i n ea ch
regi on
di vi ded by the
total number
of chi l dren i n
OHC
s tatewi de
Ca s es to
Revi ew
N=25
Total
Ca s es to
Revi ew
N=80
330
942
8.15%
2
7
5
2
1230
1226
2456
21.25%
5
13
8
5
12
1470
1420
2890
25.01%
6
18
12
6
26.07%
14
1234
775
2009
17.39%
5
19
14
5
5296
17.62%
10
1038
1219
2257
19.53%
5
15
10
5
3132
10.42%
6
807
194
1001
8.66%
2
8
6
2
55
25
DCF Region
Northwes t Regi on
In Home
Services
Families
Statewide
N=6,391
55
25
Reviews by CBC and Month
Northwest and Northeast Regions
UPDATED 12/10/2015 AS A FOLLOW UP TO THE MASC CONFERENCE CALL. TOOK 1 OHC CASE
FROM CNSWF AND ADDED 1 IN HOME CASE TO CNSWF. BRINGS TOTALS TO 55 OHC CASES
AND 25 IN HOME CASES
Review Period
April 1 through September 30, 2016
April
2016
Regional
Community
% of Regional
Total OHC
Based Care
OHC
Lead Agency
In Home
Service
Cases
% of Total
May
2016
OOH
In-Home
Out of
In home
Reviews 55 Reviews 25 Home Care
Cases
June
2016
Out of
In home
Home Care
Cases
July
2016
Out of
In home
Home Care
Cases
Out of
In home
Home Care
Cases
August
2016
September
2016
Total for the
Review Period
Out of
In home
Home Care
Cases
Out of
In home
Home Care
Cases
Out of
In home
Home Care Cases
Big Bend CBC
1104
39.77%
179
29.25%
2
1
1
0
0
1
0
0
1
0
0
0
0
0
2
1
Families First
Network
1672
60.23%
433
70.75%
3
1
0
0
1
0
0
1
0
0
1
0
1
0
3
1
Northwest
Region
2776
5
2
1
0
1
1
0
1
1
0
1
0
1
0
5
2
Community
Partnership
for Children
Family
Integrity
Program
Family
Support
Services N.
612
1315
31.03%
298
24.23%
2
1
1
0
0
0
1
0
0
1
0
0
0
0
2
1
281
6.63%
35
2.85%
1
0
0
0
0
0
1
0
0
0
0
0
0
0
1
0
1252
29.54%
548
44.55%
2
2
1
0
0
1
0
0
1
0
0
1
0
0
2
2
Kids First of
Florida
335
7.90%
78
6.34%
1
1
0
1
0
0
0
0
0
0
0
0
1
0
1
1
Partnership
for Strong
Families
1055
24.89%
271
22.03%
2
1
0
0
0
1
1
0
0
0
1
0
0
0
2
1
Northeast
Region
4238
1230
100.00%
8
5
2
1
0
2
3
0
1
1
1
1
1
0
8
5
Reviews by CBC and Month
Central and Suncoast Regions
Brevard
Family
Partnership
1030
15.21%
199
13.54%
2
1
0
0
1
0
0
1
0
0
0
0
1
0
2
1
CBC Central
Florida
1749
25.83%
430
29.25%
3
2
1
0
0
0
1
0
0
1
1
0
0
1
3
2
Community
Based Care
Seminole
549
8.11%
125
8.50%
1
0
0
0
0
0
0
0
0
0
1
0
0
0
1
0
Heartland for
Children
1538
22.71%
190
12.93%
3
1
0
1
1
0
0
0
1
0
1
0
0
0
3
1
Kids Central
Inc.
1905
28.13%
526
35.78%
3
2
1
0
0
0
0
1
0
1
1
0
1
0
3
2
Central
Region
6771
12
6
2
1
2
0
1
2
1
2
4
0
2
1
12
6
Children's
Network of
SW Florida
Eckerd
Pinellas/Pasc
o
1470
1764
22.51%
352
28.53%
3
2
1
0
1
0
0
1
1
0
0
1
1
0
3
2
2263
28.88%
296
23.99%
4
1
1
0
0
1
1
0
1
0
1
0
0
0
4
1
Eckerd
Hillsborough
2723
34.75%
418
33.87%
5
1
0
1
1
0
1
0
1
0
1
0
1
0
5
1
Sarasota
YMCA
1085
13.85%
168
13.61%
2
1
1
0
0
0
1
0
0
0
0
0
0
1
2
1
Suncoast
Region
7835
14
5
3
1
2
1
3
1
3
0
2
1
2
1
14
5
1234
REviews by CBC and Month
Southeast and South Regions
ChildNet
Broward
2785
52.59%
563
54.24%
5
3
1
1
1
1
1
0
1
0
1
0
0
1
5
3
ChildNet
Palm Beach
1487
28.08%
268
25.82%
3
1
0
0
1
0
0
1
1
0
0
0
1
0
3
1
Devereux
1024
19.34%
207
19.94%
2
1
0
0
0
0
0
0
1
0
0
1
1
0
2
1
Southeast
Region
5296
1038
10
5
1
1
2
1
1
1
3
0
1
1
2
1
10
5
Our Kids
3132
807
6
2
1
0
1
1
1
0
1
0
1
0
1
1
6
2
Southern
Region
3132
807
6
2
1
0
1
1
1
0
1
0
1
0
1
1
6
2
STATEWIDE DATA INDICATORS
Statewide Data Indicators:
CFSR2 vs. CFSR3
• No composites
• Fewer and simpler measures
• Greater reliance on entry cohorts
CFSR3 Indicators
• Safety
– S1: Maltreatment in foster care
– S2: Recurrence of maltreatment
• Permanency
– P1: Permanency in 12 months for children entering foster care
– P2: Permanency in 12 months for children in foster care for 12 to 23
months
– P3: Permanency in 12 months for children in foster care for 24
months or more
– P4: Re-entry to foster care
– P5: Placement stability
Methods
• Cohorts
– In foster care during the 12-month period (S1)
– Children who were victims of a substantiated report of
maltreatment during the 12-month period (S2)
– Children entering care during the 12-month period (P1, P4, P5)
– In foster care on the first day of the year (P2, P3)
• Measures
– Percent (S2, P1, P2, P3, P4)
– Count/rate per day of foster care (S1, P5)
• Accounts for time at risk for the outcome (maltreatment in care or placement
moves) using the total number of days eligible children were in care
S1: Maltreatment in Foster Care
Of all children in foster care during a 12-month period, what is the
rate of victimization per day of foster care?
• What’s changed?
– Rate of maltreatment per child days in foster care vs. percentage of
children not maltreated in foster care
– Includes all maltreatment types by any perpetrator vs. just maltreatment
by foster parents/facility staff
• Includes all days in foster care during the year (across episodes)
• Multiple incidents of substantiated maltreatment for the same
child are included in the numerator
Calculating Rate Per Day of Foster Care:
Maltreatment in Foster Care
Cohort: Children in Care Between Oct 2012 – Sep 2013
Child A
Days in care: 275
Instances of maltreatment: 0
Child B
Days in care: 45
Instances of maltreatment: 1
Child C
Days in care: 310
Instances of maltreatment: 2
Child D
Days in care (episode 1): 95
Instances of maltreatment: 0
Days in care (episode 2): 188
Instances of maltreatment: 0
1
Denominator: total days in care
275 + 45 + 310 + 95 + 188 = 913
2
Numerator: instances of maltreatment
0+1+2+0+0=3
3
Calculate rate of maltreatment per day in care
3 / 913 = 0.003286
4
Multiply by 100,000
0.003286 * 100,000 = 328.6
328.6 victimizations per
100,000 days in foster care
Maltreatment in Foster Care:
Example – Children in Care Oct 2012-Sep 2013
Child A
Child B
Child C
12/17/12
1/15/13
11/24/12
5/9/12**
3/21/13
in care
2/19/13
In care
1/4/13
9/25/13
9/19/13*
n/a
n/a
n/a
n/a
Age at entry/1st day of year
17
3
8
12
13
Total days in care
275
45
310
95
188
0
1
2
0
1
n/a
1/18/13
1/30/13
4/15/13
n/a
3/21/13***
Entry date
Exit date
18th birthday
Substantiated reports
Date of report
Child D (1) Child D (2)
*Day count ends at 18th birthday
**Day count starts on the first day of the 12-month period
***Instance of maltreatment is excluded because it was reported within 7 days of foster care placement
S2: Recurrence of Maltreatment
Of all children who were victims of a substantiated report
of maltreatment during a 12-month reporting period,
what percent were victims of another substantiated
maltreatment allegation within 12 months of their initial
report?
• What’s changed?
– Window is 12 months vs. 6 months
– Recurrence vs. no recurrence
P1: Permanency in 12 Months
for Children Entering Care
Of all children who enter foster care in a 12-month
period, what percent discharged to permanency within
12 months of entering foster care?
• What’s changed?
– Expanded definition of permanence includes reunification,
adoption, or guardianship vs. reunification only
– Includes all children entering foster care during the year vs.
just those who were removed for the first time
– Entry cohort window is 12 months vs. 6 months
P2: Permanency in 12 Months
for Children in Care for 12-23 Months
Of all children in foster care on the first day of the 12month period, who had been in foster care (in that
episode) for 12-23 months, what percent discharged to
permanency within 12 months of the first day?
• What’s changed?
– New measure with an intermediate time period (between 12
and 23 months)
P3: Permanency in 12 Months
for Children in Care for 24+ Months
Of all children in foster care on the first day of the 12month period, who had been in foster care (in that
episode) for 24 or more months, what percent discharged
to permanency within 12 months of the first day?
• What’s changed?
– nothing
P4: Re-entry to Foster Care
Of all children who enter foster care in a 12- month
period and are discharged within 12 months to
reunification or guardianship, what percent re-entered
foster care within 12 months of their date of discharge?
• What’s changed?
– Entry cohort (denominator includes all children who enter
care during the year and exit within 12 months) vs. all children
who exit during the year
– Includes exits to reunification and guardianship vs.
reunification only
P5: Placement Stability
Of all children who enter foster care in a 12- month
period, what is the rate of placement moves per day of
foster care?
• What’s changed?
– Entry cohort vs. all children in care for less than 12 months
– Controls for time in care by constructing a moves/placement
day vs. the number of moves per child
– Accurately accounts for actual number of moves vs. the prior
“2 or more” indicator
Calculating Rate Per Day of Foster Care:
Placement Stability
Cohort: Children Entering Care Between Apr 2013 – Mar 2014
Child A
Days in care: 342
Placement moves: 2
Child B
Days in care: 196
Placement moves: 0
Child C
Days in care (episode 1): 35
Placement moves: 1
Days in care (episode 2): 167
Placement moves: 1
Child D
Days in care: 154
Placement moves: 0
1
Denominator: total days in care
342 + 196 + 35 + 167 + 154 = 894
2
Numerator: placement moves
2+0+1+1+0=4
3
Calculate rate of moves per day in care
4 / 894 = 0.00447
4
Multiply by 1,000
0.00447 * 1,000 = 4.5
4.5 placement moves per
1,000 days in foster care
Placement Stability:
Example – Children Entering Care Apr 2013-Mar 2014
Child A
Child B
Entry date
4/23/13
6/22/13
5/4/13
10/15/13
8/30/13
Exit date
in care
1/4/14
6/8/13
in care
in care
18th birthday
n/a
n/a
n/a
n/a
2/1/14
Age at entry/1st day of year
17
3
8
12
13
Total days in care
342
196
35
167
154*
Placement moves
2
0
1
1
1
5/1/13
9/30/13
n/a
5/12/13
12/26/13
2/27/14**
Date of move
*Day count ends at 18th birthday
**Placement move excluded because it occurred after the 18th birthday
Child C (1) Child C (2)
Child D
ASSESSING PERFORMANCE
Measuring States’ Performance
• National standards for each indicator are set at the national
observed performance for that particular indicator (which is often
similar to the national average)
• State performance will be measured using a multi-level statistical
model, risk-adjusted for select child- and state-level
characteristics
– The goal is to minimize variation in outcomes due to factors over which
states have little control (e.g., the average age of children in foster care in
the state)
– Accounts for factors that differ across states and that can affect outcomes
regardless of the quality of services the state provides
• Each state’s risk-adjusted performance will be compared to the
national standard to determine substantial conformity
Multi-Level Model
• State performance assessed using a multi-level
statistical model based on the nested nature of the data:
children (level 1) living within a state (level 2)
• Model accounts for variation:
–
–
–
Across states in terms of case mix: the age distribution of children
served for all indicators and the state’s foster care entry rate for P1
and P4
Across states in terms of the number of children they serve
Between states in terms of child outcomes
Risk Adjustment Variables
INDICATOR
AGE*
ENTRY RATE**
S1: Maltreatment in Foster Care
Age at entry/on first day
--
S2: Recurrence of Maltreatment
Age at initial
victimization
--
P1: Permanency in 12 Months for Children Entering Foster
Care
Age at entry
Entry rate/1,000
P2: Permanency in 12 Months for Children in Foster Care 1223 Months
Age on first day
--
P3: Permanency in 12 Months for Children in Foster Care 24
Months+
Age on first day
--
Age at exit
Entry rate/1,000
Age at entry
--
P4: Re-Entry to Foster Care in 12 Months
P5: Placement Stability
*Age defined as birth-3 months, 4-11 months, and each year from age 1 through 17
**Entry rate per 1,000 uses the Census year (July 1) closest to the 12-month period the child entered
foster care as the population denominator
Risk Standardized Performance (RSP)
• The RSP is the ratio of the number of “predicted”
outcomes over the number of “expected” outcomes,
multiplied by the national observed performance
• These values are calculated from the results of the
multi-level model which encompasses the case mix
across all states
• Each state’s RSP can be compared directly to the
national observed performance to determine state
performance vs. an “average” state
Generating the RSP
Example: Exits to Permanency in 12 Months (Entry Cohort)
1. Estimate the number of predicted outcomes
– The predicted number of outcomes the state would have (on average)
based on the state’s performance with its observed case mix
State A:
500 predicted exits
to permanency
State B:
9,500 predicted exits
to permanency
State C:
4,420 predicted exits
to permanency
2. Estimate the number of expected outcomes
– The number of outcomes that would be expected if children were served
by the average state
State A:
425 expected exits
to permanency
State B:
10,000 expected exits
to permanency
State C:
4,500 expected exits
to permanency
Generating the RSP (cont.)
Example: Exits to Permanency in 12 Months (Entry Cohort)
3. Divide the number of predicted outcomes by the number of
expected outcomes
State A:
500 / 425 = 1.176
State B:
9,000 / 10,000 = 0.900
State C:
4,420 / 4,500 = 0.982
3. Multiply this ratio by the national observed performance (40.4%
for P1)
State A:
1.176 * 40.4% = 47.5%
State B:
0.900 * 40.4% = 36.4%
State C:
0.982 * 40.4% = 39.7%
Higher than expected
permanency rate
Lower than expected
permanency rate
Lower than expected
permanency rate
Measuring States’ Performance
Example: Exits to Permanency in 12 Months (Entry Cohort)
• Approximate 95% interval estimates are calculated around each
state’s RSP
55.0%
State A
52.3%
50.0%
47.5%
State C
45.0%
42.8%
40.0%
35.0%
30.0%
25.0%
State B
43.1%
39.7%
38.5%
36.4%
34.3%
36.3%
Categorizing States’ Performance
Example: Exits to Permanency in 12 Months (Entry Cohort)
• Each state’s interval estimate is compared to the national
observed performance. Performance is categorized as:
55.0%
50.0%
State A
Higher than national
performance
47.5%
State C
45.0%
State B
40.0%
35.0%
30.0%
25.0%
39.7%
National Standard: 40.4%
36.4%
Lower than national
performance
PIP REQUIRED
No different
from national
performance
Meeting the National Standard
• States meeting national standards (those whose performance is
no different or exceeds the standard) during the PIP monitoring
period will be credited as having met their goals
• States that fail to meet the national standard will be required to
include that indicator in their PIP
• Baseline for the PIP is the state’s observed (rather than riskadjusted) performance on the indicator for the most recent year
of data available before the beginning of PIP implementation
• Performance goals and thresholds are based on the state’s
observed performance in the three most recent years of data
Setting Program Improvement Goals
• Improvement goals for each of the data indicators are
calculated from improvement factors, which are:
– Based on the state’s observed performance over the three
most recent years of data
– Multiplied to the state’s baseline for each indicator needing
improvement
• Thresholds are calculated for companion measures (P1
and P4) and are the inverse of the improvement goals.
Calculating Improvement Factors/Goals
Example: Exits to Permanency in 12 Months (Entry Cohort)
1. Generate estimates of recent performance using 7 values:
Florida’s P1 Performance
1. Year 1 (observed)
39.2%
2. Year 2 (observed)
37.9%
3. Year 3 (observed)
36.4%
4. Years 1-3 (average)
37.8%
5. Years 1 & 2 (average)
38.5%
6. Years 1 & 3 (average)
37.8%
7. Years 2 & 3 (average)
37.1%
Calculating Improvement Factors/Goals
Example: Exits to Permanency in 12 Months (Entry Cohort)
2. Estimate variability in recent performance:
– Resample (using bootstrapping) from the original 7 estimates to generate a
larger sample of 30 values
– From these 30 values, calculate a sample mean and standard deviation
– Repeat the bootstrapping process 1,000 times, generating 1,000 sample
means and 1,000 standard deviations
3. Calculate a grand mean (average of all 1,000 sample means) and
standard deviation (average of all 1,000 standard deviations)
Florida’s P1 Performance
Grand mean
37.8%
Average standard deviation (Avg S)
0.8%
Calculating Improvement Factors/Goals
Example: Exits to Permanency in 12 Months (Entry Cohort)
4. Calculate the improvement goal:
– Obtain the improvement factor by
dividing the grand mean + 2
standard deviations (or -2 standard
deviations depending on the
indicator) by the grand mean
– Multiply the improvement factor to
the baseline (year 3 performance)
5. Generate the threshold
factor/threshold
– Threshold calculation is the inverse
of the improvement goal (minus 2
standard deviations)
Florida’s P1 Performance
(37.8% + (2 * 0.8%)) / 37.8%
39.5% / 37.8% = 1.045
Improvement Factor = 1.045
1.045 * 36.4% = 38.0%
Improvement Goal = 38.0%
(37.8% - (2 * 0.8%)) / 37.8%
36.1% / 37.8% = 0.955
0.955 * 36.4% = 38.0%
Threshold = 34.7%
Companion Measures
• Using Florida as an example, in order to
meet the PIP goal for P1, the state must:
– Increase the percentage of children
achieving permanency to 38.0%
– Not allow the percentage of children reentering care to exceed 10.6% (the
threshold for Re-entry to care – P4)
• Conversely, to meet the PIP goal for P4,
CA must reduce the percentage of
children re-entering care to 8.2%, but
cannot allow the percentage of children
achieving permanency to decline further
than 34.8%
Entry Cohort:
Permanency
in 12 months
Re-entries to
Foster Care
Adjusted Goals/Thresholds
• For improvement goals/thresholds that turn out to be too
aggressive to be attainable or too modest to be meaningful, caps
and floors are set on the improvement factors based on the
distribution of all states’ improvement factors
– Cap (maximum) is set at the 50th percentile
– Floor (minimum) is set at either the 20th or the 80th percentile, depending
on whether the desired outcome is a higher or lower value
– States whose improvement factors exceed the cap or fall short of the floor
will have their improvement factors replaced by the maximum or minimum
improvement factor for that indicator (these are “adjusted improvement
goals”)
National Standards
IMPROVEMENT
FACTOR
NATIONAL
STANDARD
INDICATOR
MINIMUM
MAXIMUM
S1: Maltreatment in Foster Care
8.04*

0.922
0.849
S2: Recurrence of Maltreatment
9.0%

0.953
0.910
P1: Permanency in 12 Months for Children Entering Foster Care
40.4%

1.035
1.057
P2: Permanency in 12 Months for Children in Foster Care 12-23
Months
43.7%

1.040
1.074
P3: Permanency in 12 Months for Children in Foster Care 24+
Months
30.3%

1.034
1.080
P4: Re-Entry to Foster Care in 12 Months
8.3%

0.912
0.867
4.12**

0.953
0.912
P5: Placement Stability
*Rate per 100,00 days in care
**Rate per 1,000 days in care
Florida’s Performance
RSP*
IMPROVEMENT
FACTOR
GOAL
OBSERVED PERFORMANCE
INDICATOR
YR3
YR1
YR2
S1: Maltreatment in Foster Care
8.11
7.86
7.97
11.66
0.922
7.35
S2: Recurrence of Maltreatment
9.2%
9.1%
9.2%
12.0%
0.953
8.8%
P1: Permanency in 12 Months for
Children Entering Foster Care
39.2%
37.9%
36.4%
37.7%
1.045
38.0%
P2: Permanency in 12 Months for
Children in Foster Care 12-23 Months
41.7%
42.8%
39.8%
39.5%
1.044
41.6%
P3: Permanency in 12 Months for
Children in Foster Care 24 Months+
22.6%
22.7%
22.2%
22.4%
1.034
23.0%
P4: Re-Entry to Foster Care in 12
Months
7.7%
8.6%
9.4%
11.5%
0.876
8.2%
P5: Placement Stability
4.32
4.03
3.81
3.78
*Risk Standardized Performance
(baseline)
Standard Met – no PIP
CHANGES TO THE STATEWIDE
ASSESSMENT & ONSITE REVIEW
Statewide Assessment
and Integration with the CFSP
CFSP/ASPR
CFSR/PIP
Separate/Redundant
Assessments
States can
refer to
CFSP/APSR
and update
info as
needed
Integrated Statewide
Assessment
Onsite Review: Stakeholder Interviews
• Reserved for systemic factors where level of conformity is unclear
from the statewide assessment (or CFSP/APSR)
• The scope of stakeholder interviews conducted will vary based on
demonstrated level of functioning on each of the systemic factors
in the statewide assessment
• Additional stakeholder interviews for a systemic factor may not be
required (with the exception of service array)
• Where statewide assessment data is insufficient to determine
substantial conformity, the joint federal-state team will determine
which stakeholder interviews are necessary to gather additional
information during the onsite review
Onsite Review: Case Reviews
• Florida has met specific criteria and is approved to conduct case
reviews using the revised federal CFSR onsite review instrument
• State policies, procedures, and other materials were reviewed by
the Administration for Children and Families prior to the decision
• The Children’s Bureau will participate in state’s case review
process
Criteria for Using State Case Review Process
• The state must:
– Conduct annual (at least) internal case review process
assessing statewide performance using a uniform sampling
methodology
• Minimum of 65 cases served during the sample period (40 foster care
cases and 25 in-home cases)
– Have a process in place for ensuring accurate and consistent
case review ratings
– Use the federal instrument, rating guidance, and instructions
Conclusion
• Questions and Comments?
– Contact info
Sources
•
CFSR Round 3 Statewide Data Indicators
–
•
CFSR Technical Bulletin #7
–
•
•
http://kt.cfsrportal.org/action.php?kt_path_info=ktcore.actions.document.view&fDocumentId=73607
http://kt.cfsrportal.org/action.php?kt_path_info=ktcore.actions.document.view&fDocumentId=72431
Federal Register Notice: Statewide Data Indicators and National Standards for Child and Family
Services Reviews
–
https://federalregister.gov/a/2014-09001
–
Final rule: http://www.gpo.gov/fdsys/pkg/FR-2014-10-10/pdf/2014-24204.pdf
Sustaining the Momentum: The Next Round of Reviews
–
http://kt.cfsrportal.org/action.php?kt_path_info=ktcore.actions.document.view&fDocumentId=72464
–
Children's Bureau Plan for CFSR Statewide Data Indicators and National Standards
•
•
http://kt.cfsrportal.org/action.php?kt_path_info=ktcore.actions.document.view&fDocumentId=72553
For additional information: https://training.cfsrportal.org/resources/3044