Pro Bono Publico: Avoiding Child Welfare Data Abuse
Panel on Meaningful Measurement
in the Context of Litigation and Consent Decrees
Daniel Webster, MSW, PhD
Center for Social Services Research, University of California at Berkeley
Isabel Blanco
Deputy State Child Welfare Director, South Carolina
Andy Barclay
Fostering Court Improvement, Atlanta, GA
Susan Smith, PhD
Casey Family Programs Data Advocacy Group
Presentation Originally Created by:
Barbara Needell, MSW, PhD
Melissa Correia, MSW
Emily Putnam-Hornstein, MSW, PhD
Bryn King, MSW
AAPWA Preconference Symposium
Demonstrating Sustained Well Being of Children and Youth
in a Compliance Environment
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
San Diego, CA
September 9, 2012
The Current Placement System*
(highly simplified)
the foster care system
CHILD IN
a bunch of
stuff happens
CHILD OUT
*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out
of home placement experiences. Chicago: American Evaluation Association Annual Conference
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Tracking Child Welfare Outcomes
rate of allegations/
substantiated allegations
home-based services
vs.
out of home care
reentry to care
permanency
through reunification,
adoption, or
guardianship
counterbalanced
indicators of system
performance
length
of stay
stability
of care
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
use of least
restrictive
form of care
positive attachments to
family, friends, and neighbors
Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L.
(2002). Measuring Outcomes in Child Welfare. Chapel
Hill: Jordan Institute for Families
There are three kinds of lies:
Lies, Damned Lies and Statistics
^
Abused Statistics
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Public Data: Putting it All Out There
• PROS:
– greater performance accountability
– community awareness and involvement, encourages
public-private partnerships
– ability to track improvement over time, identify areas
where programmatic adjustments are needed
– Region/region and region/county collaboration
– transparency
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Public Data: Putting it All Out There
• CONS:
– Potential for misuse, misinterpretation, and
misrepresentation
– Available to those with agendas or looking to create a
sensational headline
– Misunderstood data can lead to the wrong policy decisions
– “Torture numbers, and they’ll confess to anything”
(Gregg Easterbrook)
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Six Ways to Abuse Data
(without actually lying!):
1) Compare Apples and Oranges
2) Use ‘snapshots’ of Small Samples
3) Rely on Unrepresentative Findings
4) Logically ‘flip’ Statistics
5) Falsely Assume an Association to be Causal
6) Rely on Summary Statistics
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
1) Compare Apples and Oranges
Two doctors in San Diego, CA…
Doctor #1
Doctor #2
Lowest mortality rate?
2/1000
20/1000
What if Doctor #1 is a podiatrist and Doctor #2 is a cardiologist?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
2) Data Snapshots…
Crime in San Diego, CA…
Number of Crimes
Period 1: 76
Period 2: 51
No change.
Average
= 73.5
Crime jumped by 49%!!
Period 3: 91
Crime dropped by 16%
Period 4: 76
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
2) Data Snapshots…
A politician recently claimed that 92.3% of all
the jobs lost since Obama took office were lost
by women.*
*http://www.forbes.com/sites/susanadams/2012/04/12/romney-claims-on-womens-job-loss-paint-a-misleading-picture/
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
3) Unrepresentative Findings…
Survey of people in San Diego, CA…
90% of respondents stated that they support using tax
dollars to build a new football stadium.
The implication of the above finding is that there is overwhelming
support for the stadium…
But what if you were then told that respondents had been
sampled from a list of season football ticket holders?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
4) Logical “Flipping”…
Headline in U-T San Diego:
60% of violent crimes are committed by men who
did not graduate from high school.
“Flip”
60% of male high school drop-outs commit violent
crimes?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
5) False Causality…
A study of San Diego residents makes the
following claim:
Adults with short hair are, on average, more than 3 inches
taller than those with long hair.
Hair Length
X
Height
Gender
Finding an association between two factors does not mean that
one causes the other…
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
6) Reliance on Summary Statistics
The average human has one breast and one testicle.*
Disaggregation is required for this analysis to be really useful.
* ~Des McHale www.quotegarden.com/statistics.htm
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Disaggregation
• One of the most powerful ways to work with data…
• Disaggregation involves dismantling or separating out groups
within a population to better understand the dynamics
• Useful for identifying critical issues that were previously
undetected
Aggregate Permanency Outcomes
Race/Ethnicity
County
Age
Placement Type
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
In San Diego County, entry rates are highest for infants.
Entries are higher for African American children
for almost all ages.
Rate of First Entry into Out of Home Care in 2011
by Age, Gender, & Race
45
40
35
30
Rate per
1,000
Children
Total
25
Black Female
20
Black Male
15
White Female
10
White Male
5
0
<1
1
2
3
4
5
6
7
8
9
Age
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
10
11
12
13
14
15
16
17
Response to Data Maltreatment?
• must have the will to weather the storm(s)…
• continued efforts to frame the data, educate
interested media, policymakers, and others
– what do these findings mean?
– how can these data be used to gain insight into where
improvements are needed?
• agencies must be proactive in discussing both the
“good” and the “bad” (be first, but be right).
– be transparent
– if not playing offense…playing defense
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Questions?
Daniel Webster
510.290.6779
dwebster@berkeley.edu
http://cssr.berkeley.edu/ucb_childwelfare/
Needell, B., Webster, D., Armijo, M., Lee, S., Dawson, W., Magruder, J., Exel, M., CuccaroAlamin, S., Putnam-Hornstein, E., Williams, D., Simon, V., Hamilton, D., Lou, C., Peng, C.,
Moore, M., King, B., Henry, C., & Nuttbrock, A. (2012). Child Welfare Services Reports for
California. Retrieved 8/22/2012, from University of California at Berkeley Center for
Social Services Research website. URL: <http://cssr.berkeley.edu/ucb_childwelfare>
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley