Children & Families Services Review (CFSR)
Presented By: Alba Quiroz Garcia, CDSS
July 31, 2014
Material from the May 2014 Administration for Children and Families, CFSR Briefing for States, Seattle, WA
Recap, what is the CFSR?

The federal government created the CFSR in response to
the 1997 Adoption and Safe Families Act. It is an
assessment of a state’s performance related to child
welfare.

Indicators and standards are used along with information
from a sample of cases to inform the federal government
about determinations of whether a state is in substantial
conformity with certain safety and permanency outcomes
and to assess each state’s ability to achieve Safety,
Permanency, and Well-being for children and families.
2
Why a CFSR information session in 2014?
With Rounds 1 and 2 of the CFSR completed, Round 3 is
near and modifications are occurring presently.

This session is to help inform the field about
plans for statewide data indicators, national
standards, and the case review process that will be used in
the next round of the CFSR.

3
Timeframe for California’s CFSR Round 3
California’s Round 3 CFSR is scheduled
for February through September 2016.

Some states will begin Round 3 of the CFSR as early as
Fall 2014. This means that changes still pending for the
redesign of the indicators need to be finalized rapidly.

4
Why CFSR modifications?

With regard to data indicators & standards, the CB heard
feedback from the field that the process could be improved by
using different methods and measures that better reflect state
practices and improvements. The challenges with prior
indicators & measures ranged from methodological issues, ease
of interpreting composite numbers, and concerns about
comparability of state data.

Also, the CB now has a greater reliance on entry cohorts,
fewer & simpler indicators, and they are dropping composites
in favor of single item indicators.

Release of information memorandum to states on establishing
and maintaining a Continuous Quality Improvement (CQI)
system.
5
CQI
California is seeking to submit a letter of intent to the CB to use our case
review process as part or the next CFSR.
For the state’s case review process to be considered, the following must be
met:
1. Use the Federal Onsite Review Instrument (ORI). This tool collects and
documents outcomes to Safety, Permanency and Well-Being.
2. Submit a letter of intent by July 2015 to the CB. The letter must be
accompanied by a plan that meets all requirements outlined in the
federal register (released April 2014).
3. Be able to replicate a sample from a recent AFCARS period.
4. Use a sample period of April 1-Sept 30, 2015 .
The CB is not certifying the CQI process, rather they are approving or not
approving our state’s case review process. Approval status is expected as
early October 2015, but no later than December 2015.



6
CQI Continued…

If the state is approved, there is no limit on how many
cases the state can review during the timeframe.

However, if a state chooses or has to do a traditional
CFSR, the CB will only review 65 cases total (statewide).

For Round 3, case reviews do not need to be done in
pairs as was the case in the previous two CFSR Rounds.
7
Systemic Factors and Statewide Indicators

The seven (7) Systemic Factors and Statewide Indicators
remain present for CFSR Round 3.

Both will be utilized to determine if the state is in
substantial conformity and meeting federal targets for
Safety, Permanency and Well-Being.
8
Systemic Factors
There will be a focus on systemic factor functioning in
Round 3. In CFSR Rounds 1 and 2 the focus was on
having “requirements in place.”
Seven (7) Systemic Factors:









9
1. Statewide Information System
2. Case Review System
3. Quality Assurance System
4. Staff and Provider Training
5. Service Array and Resources Development
6. Agency Responsiveness to the Community
7. Foster/Adoptive Parent Licensing, Recruitment & Retention
Systemic Factors Continued…

The CB can make a determination about substantial
conformity for 6 of the 7 systemic factors using the data
provided in the Statewide Assessment, with the exception
of Service Array. Service Array cannot be determined
through the statewide assessment and will require
stakeholder interviews (there is no way that states can
get around this).

Since the CB can make determinations about substantial
conformity for most, but not all systemic factor, there is a
possibility of reduction in scope and number of
stakeholder interviews.
10
Systemic Factors Continued…

At least 95% of applicable cases must meet the target to
be determined “In Substantial Conformity.”

For Round 3, there is a reduction in Systemic Factor
Items:

11
In CFSR Round 1 and 2 there were 22, for Round 3, there are
18.
Six (6) statewide data indicators will serve
as basis for determining whether or not
states meet national standards.

Two (2) for Safety
I.
II.

Maltreatment
Re-Report of Maltreatment
Four (4) for Permanency
I.
II.
III.
IV.
12
Percent of the entry cohort that achieves permanency in 12
months.
Percent of those children who subsequently re-enter care.
Percent of children and youth who have been in care for 2
years or longer who achieve permanency in the next 12
months.
A placement stability indicator for children entering care
Safety Indicators-Children are protected
from abuse and neglect.
1. Maltreatment in Foster Care (FC)
This asks, “Of all children in care during a 12-month
period, what is the rate of victimization per day of FC?”
 The denominator looks at all children in care during a 12month period and looks at the total number of days these
children spent in care as of the end of the year.
 The numerator looks at those same children who are in the
denominator and counts the total number of substantiated or
indicated reports of maltreatment (by any perpetrator) that
occurred during a foster care episode.
 Includes all perpetrator types and controls for amount of time
spent in FC.
13
Safety Indicators-Children are protected
from abuse and neglect.

II. Re-Report of Maltreatment



14
Of all children with a screened-in report of alleged
maltreatment in a 12-month period (regardless of disposition
type), what percent had another screened-in report within 12
months of their initial report?
Expanded timeframes: from 6 months to 12 months
Rather than limit the indicator to include victims only, the CB
includes all screened-in reports of alleged maltreatment that
have reached disposition.
Safety Indictors Continued…

Rather than limit the indicator to substantiated cases only, the
CB is expanding this indicator to count all children with
screened-in reports of alleged maltreatment.


15
Why? The CB believes that looking at repeat reports, regardless of
disposition type, is a viable measure of an agency's’ attempts to
prevent maltreatment, based on research indicating that families with
screened-in but unsubstantiated reports are at a high risk of rereport.
Using all screened-in reports rather than limiting the [indicator] to
substantiated cases makes states more comparable to each other
and even against their own performance over time, when considering
the growing implementation of differential response (DR), where a
substantial percentage of cases may not receive a formal
investigation.
Permanency Indicators-Children have
permanency and stability in
their living situations.

Rather than have composites made up of multiple
measures, each of these indicators will be assessed
individually on a particular cohort of children in FC for
the initial determinations of compliance. In some cases,
for purposes of monitoring during the PIP, the CB will
treat another indicator as a companion indicator (this is
new).
16
I. Entry Cohort Permanency in 12 Mo.

This first indicator asks, “Of all children who entered care in a 12-month
period, what percent discharged to permanency within 12 months?”

The difference here is the attention on permanency, so this means that
while the majority of discharges in a short time frame will be reunification,
the CB will also count exits to living with a relative, adoption, or
guardianship as a success on this indicator.

In the last CFSR, the CB had a similar measure that looked at entry
cohorts, but was limited to first-time entries. The CB is now counting all
entries during the year and expanded the timeframe from a 6 to12 month
cohort. The past CFSRs also only counted reunifications.

By expanding the denominator to a full year, the CB can achieve more
stable and reliable estimates, which is important because those children
achieving permanency on this indicator will also serve as the denominator
for the re-entry indicator.
17
II. Re-Entry Permanency Indicator




The second indicator looks at children in the entry cohort
who discharged to permanency within 12 months, and tracks
them for another 12 months after exit to see whether they
re-enter foster care.
It excludes exits to adoption because they cannot be as
reliably tracked.
For all exits to reunification, living with a relative, or
guardianship the CB wants to be sure they were not done
without adequate planning or services to support a permanent
placement, resulting in a greater risk of re-entry to care.
This indicator is limited to the entry cohort, and does not
include all exits to reunification that occur during the year, as it
did before. The denominator now including exits to
guardianship and this represents another change.
18
III. Children in Care 2+ Yrs.
Permanency 12 Mo.

Looks at the cohort of children in FC on the 1st day of a
year who already have been in FC for 2 years or longer.

The CB is interested in knowing what proportion of this
cohort achieves permanency within 12 months of the 1st
day? So the focus is on getting these children and youth
to permanent homes. Again, the CB will count all
discharges to reunification, living with a relative, adoption,
or guardianship as a success.
19
IV. Placement Stability


Has been refined to have one indicator instead of three.
The proposed indicator looks at the rate of placement moves
in the first year of FC, for the entry cohort.


Similar to the way the CB calculates maltreatment in FC, the
denominator for this indicator is the sum of all days children spent in
FC, to control for various lengths of time in care. For the numerator,
the CB looks at how many times children were moved from one
placement to another, while they were in care.
This is actually cumulative across placement episodes in the
same year. The CB will NOT count their initial placement as a
move, but any change is counted. While the CFSR Round 2
indicators did not differentiate children who moved twice
from those who had moved more times, this indicator counts
every move to accurately capture the rate of movement,
rather than the number of children affected.
20
National Standard-Methodology

It is set at the national observed performance for each
statewide indicator.

Example: For most of the indicators it is a simple formula: the
number of children in the nation with the outcome divided by
the number of children in the nation eligible for the outcome.


In reviewing the permanency-by-12-months indicator... Assume we
have 220,000 children in the nation that achieve permanency by 12
months out of 500,000 children in the nation that were eligible. So
quite simply, the national standard for this indicator is 44%. This will
often be similar to the national average.
According to CB analysis, most states will need to
improve on at least one indicator.
21
Data Sources

There is continued reliance on AFCARS and NCANDS
data. The CB will also link the two using the child ID.

Performance on these indicators will be provided to
states via data profiles, to assist in monitoring Program
Improvement Plans (PIP) that may address one or more
of these indicators.
22
Data Sources Continued

One criticism of the measurement techniques used in prior
CFSR rounds was a heavy reliance on exit cohorts and pointin-time cohorts, rather than using prospective entry cohorts.
Exit cohorts and cross-sectional cohorts can distort system
performance outcomes because they represent biased subsets
of all children who are served by the agency in a year.

For example, looking only at children who exit foster care in a
given year ignores those children who did not leave. Looking
only at those in foster care at the end of the year (or at any
point in time) biases the sample to include more children with
long lengths of stay in foster care. Entry cohorts can avoid
these problems and provide a more fair assessment of overall
system performance. Thus, the CB has shifted toward a greater
focus on entry cohorts.
23
Foster Care Exclusions

Youth who are already age 18 or older at the start of the
indicator (that is, at the age of entry into care or who are
already 18 at the outset of an indicator) will be excluded.


Although CA and other states do provide FC to youth age 18
and older, there is no consistent inclusion of this population
across states.
Any FC episode lasting less than 8 days will be excluded
from these indicators. These very short-term placements
are more prevalent in some states than in others, so this
is another way to eliminate a bias and make states more
comparable in those outcomes.
24
Measuring States’ Performance-Methodology

The goal of risk adjustment is to minimize the differences
in outcomes due to factors over which the state has little
control. It takes into account the characteristics of each
state’s case mix, something that has been a concern for
stakeholders and people in the field for many years.


25
For example, a state’s children may be at greater risk for a
poor outcome simply because of their age or history, which
might be a characteristic of the child population in that state.
By performing risk adjustment, the CB can take into account
factors that differ across the states and can influence the
outcomes, regardless of the quality of care a state provides.
Measuring States’ Performance-RiskAdjustment Possible Variables

Age of child, as an example, is a risk adjustment variable
that the CB is considering, others being considered by the
CB are: child’s gender/sex, number of prior removals,
state FC entry rate.

The CB will finalize the variables after they receive and
review public comments.
26
Measuring States’ Performance
Risk-Adjustment Continued…

Risk Adjustment—Maltreatment in Foster Care Example

27
Use maltreatment in FC, hypothetically the average age of a
child in FC across the country is 10 years old. In state A, the
average age is 5 years old. State A can perform worse than
other states simply because of the age of its children, putting
them at more risk. Since we know from field analysis of
NCANDS data and from the vast literature on maltreatment,
that younger children are more at risk for maltreatment,
regardless of the quality of care being provided….So in this
case, risk adjustment would control for the age of the children
in the state compared to the average age of the children across
the nation.
Measuring States’ Performance—Confidence
Intervals

For example, 45%, plus or minus 2%, would
represent the interval estimate. The CB can
be 95% confident that a state’s overall
performance in the long run lies
somewhere between the lower and upper
limit of the state’s confidence interval.

28
Example: State A, their risk-adjusted
performance was 45%, but because of some
error and uncertainty, the CB can be 95%
confident that their true performance is
somewhere between 43% and 47%.
Measuring States’ Performance-Categorizing
States

So the confidence interval is used to help determine
where a state is relative to the national standard. The CB
will categorize states into three groups:





No different than the nation,
Higher than the nation, or
Lower than the nation
States that fail to meet the national standard will be
required to include that indicator in a PIP.
Regardless of which indicators are included in the state’s
PIP, the CB will provide each state with a data profile that
includes information on the state’s performance on all of
the indicators.
29
Companion Indicators

In addition to setting targets for all statewide
data indicators that may be included in a PIP,
there will also be thresholds established for
companion indicators.

The companion indicators are entry cohort
permanency in 12 months and the re-entry
indicator…because efforts to improve on the
entry cohort permanency in 12 months
could have the consequence of increasing the
rate of re-entry, states should not get worse
than the threshold for re-entry.
30
Resources

http://www.childsworld.ca.gov/PG1520.htm

https://training.cfsrportal.org/resources
31