Using Program Data to
Improve APS Performance
Or How I Learned to Be A Geek
and Enjoy It
Karl Urban
Director of Performance and Policy Development
Texas Adult Protective Services
What are the Objectives of This
Presentation?
• Participants will learn about quantitative
and qualitative measures of APS
performance.
• Participants will learn how to use APS
program data for employee accountability
and performance improvement.
• Participants will learn about the
technological support systems for using
APS program data.
What Will You Get Out of This
Presentation?
• This session will:
– provide a review and discussion of data for measuring
APS performance;
– explain how employees and managers are
encouraged and empowered to use data at the
individual, unit and program level; and
– demonstrate the technology systems and tools for
accessing and using data.
• Participants will not become geeks but they may
just become better managers and decisionmakers.
Why Am I Qualified to Talk About
This?
• I am a policy wonk by education and
disposition
• My prior experience with performance
management using data
– Texas Performance Based Planning and
Budgeting System
– State Unit on Aging/AoA/GPRA
• My current job in Texas
Why Is This Topic Important for
APS Programs?
• The GAO just recommended that the
federal government explore collecting
performance information from APS
systems.
• Our experience in Texas suggests APS
programs are a potentially data rich
environment that can be used for
accountability and program improvement.
What Are My Biases
on Managing With Data?
• It is a means, not the ends.
• Data helps you to ask the right questions;
rarely does it give definitive answers to
policy or operational questions.
• The rewards of managing by data are
usually worth the effort, but you need to
pay attention to potential unintended
consequences.
What is the Status of APS
Programs Today?
• “It was the best of times; it was the worst of
times”
– Excitement – and a little apprehension – around EJA
and attendant national focus
• How do we help the federal government know what is
important to measure? How do you actually do it across
disparate state and county-based systems?
– Most state and local programs are hurting financially
• How do we use performance information to help do more
with less?
• What else do you think is critical in the APS
world these days?
• Texas is really, really big, very culturally
diverse, and an extremely red state.
…and Jerry Jones is actually
from Arkansas
What Does the Texas APS’ Inhome Program Do?
•
•
•
•
•
APS investigates allegations of abuse, neglect and exploitation of
vulnerable adults in the community age 65 or older or people with
disabilities age 18-64.
– APS clients live in their own home, family homes or unlicensed
congregate settings such as personal care homes or room and
boards.
– APS conducted 82,802 in-home investigations in FY 2010.
If needed, APS arranges for or pays for services to alleviate abuse
and neglect, such as respite, and emergency assistance with food and
shelter.
– APS arranged or paid for services in 42,926 cases in FY 2010.
APS also makes referrals to law enforcement and to guardianship
programs.
Employs over 800 direct delivery staff with a total budget of almost $45
million
APS is the safety net for the safety net in Texas
10
What Does It Mean to Be the
Safety Net for the Safety Net?
Sexual Abuse
0.1%
Suicidal Threat
0.7%
Emotional
Abuse
4.4% Exploitation
2.4%
Medical
Neglect
18.2%
Physical
Neglect
61.8%
Mental Health
Neglect
9.7%
Physical Abuse
2.6%
FY 2010 Cases By
Type of ANE
What Do Texas APS’ MH&IDD
Investigations Do?
• APS investigates abuse, neglect, and exploitation of
clients receiving services in the following stateoperated and/or contracted settings:
–
–
–
–
–
State Supported Living Centers
State Hospitals
State Centers
Community mental health/mental retardation centers
Privately-operated Intermediate Care Facilities for Persons with
Mental Retardation
– Mental retardation waiver providers
•
APS completed 9,922 MH&IDD related investigations in FY 10.
•
Employs about 150 staff with a total budget slightly over $8 million
12
What Does the APS Program Look
Like from a Metric Perspective?
Department of Family and Protective Services
Adult Protective Services
14
Department of Family and Protective Services
Adult Protective Services
15
Department of Family and Protective Services
Adult Protective Services
16
Department of Family and Protective Services
Adult Protective Services
17
Department of Family and Protective Services
Adult Protective Services
18
Department of Family and Protective Services
Adult Protective Services
19
Department of Family and Protective Services
Adult Protective Services
20
Department of Family and Protective Services
Adult Protective Services
21
Texas has a robust system of
using qualitative (case reading)
and quantitative (performance)
data for employee accountability
and program improvement.
So?
What is Qualitative Data?
• Based on case reading by 7 quality assurance
specialists
– 2 (closed) cases per worker per month
– Scored against standards based on policy
• Examples:
– Was the case initiated in accordance with policy?
– Was the client assessment tool completed appropriately?
– Did the investigator collect all the necessary evidence?
• Rolls up into 3 scales
– Investigation: how good was the investigation?
– Outcome: was the client protected and services delivered?
– Process: did we cross our Is and dot our Ts?
• Rolls up into worker, unit, region, statewide data
What is Quantitative Data?
• Based on documentation in case management system
by workers
• Extensive system of available reports from MRS
(Management Reporting System)
• Examples:
– Timeliness of documentation, face-to-face contacts
– Case durations
• There are multiple levels of reports:
– Twelve month summaries that are used for evaluation purposes
and
– Monthly reports that provide a case detail list for the months
summarized on the twelve timeliness reports
– There are also quarterly, weekly and even daily reports
How Does APS Use
Program Data?
• Provide data to inform employee and
program accountability.
• Provide data for managing and enhancing
the program.
• Provide data to assess and enhance
quality of casework and investigations.
How Does APS Use Data for
Employee Accountability?
•
•
•
•
Provide data to inform employee and
program accountability.
Achieve client-centered outcomes (intake,
investigations, risk assessment, delivery of
services).
Inform job performance standards.
Establish method of accountability, including
awareness and procedures for monitoring.
Provide information for conducting periodic
employee performance reviews.
Task
Performance Standard
Data Source
Task 1: Case
Initiation
(Quantitative)
Caseworker will initiate
each case within 24
hours.
Average for Timeliness in INV Stage Closed
Investigations - 24 Contact during the review
period
(5) Distinguished = Above 98%
(4) Commendable = 96%-98%
(3) Competent = 94%-95%
(2) Needs Improvement = 92%-93%
(1) Needs Major Improvement = Below 92%
(See Data Warehouse Report
-> APS -> Employee
Development -> INV
Timeliness in INV Stages
Closed (12 Month) - INV APS
11 % 24 Contact Met)
Task 2: Investigation
(Quantitative)
Caseworker will make
face-to-face contacts
within policy timeframes
for the priority.
Average for Timeliness in INV Stage Closed
Investigations - Face to Face Contact during the
review period
(5) Distinguished = Above 94%
(4) 92%-94% = Commendable
(3) Competent = 89%-91%
(2) Needs Improvement = 80%-88%
(1) Needs Major Improvement = Below 80%
(See Data Warehouse Report
-> APS -> Employee
Development -> INV
Timeliness in INV Stages
Closed (12 Month) - INV APS
11 % FTF Contact Met)
Task 4: Investigation
(Qualitative)
Caseworker will fully
investigate allegations
and all factors that
present a threat to the
client's health or safety.
Investigation Scale average for the sample of
cases read during the review period.
(5) Distinguished = Above 93%
(4) Commendable = 87%-93%
(3) Competent = 80%-86%
(2) Needs Improvement = 73%-79%
(1) Needs Major Improvement = Below 73%
(See APS Case Reading
System -> Reports -> Three
Scales Report ->
Investigation
Scale
Investigation Scale
Score - %)
Task 6: Service
Delivery (Qualitative)
Caseworker will work to
alleviate the client's
state of abuse/ neglect/
exploitation and remove
threats to client's health
and safety by designing
and implementing, a
service plan that
addresses problems
identified during the
investigation.
Client Outcome Scale average for the sample of
cases read during the review period.
(5) Distinguished = Above 93%
(4) Commendable = 87%-93%
(3) Competent = 80%-86%
(2) Needs Improvement = 73%-79%
(1) Needs Major Improvement = Below 73%
(See APS Case Reading
System -> Reports -> Three
Scales Report -> Client
Outcome Scale Client
Outcome Scale Score - %)
Task 7: Process
Compliance
(Qualitative)
Caseworker will meet
deadlines, perform
consultations and
notifications, and
document the casework
according to policy.
Process Compliance Scale average for the
sample of cases read during the review period.
(5) Distinguished = Above 93%
(4) Commendable = 87%-93%
(3) Competent = 80%-86%
(2) Needs Improvement = 73%-79%
(1) Needs Major Improvement = Below 73%
(See APS Case Reading
System -> Reports -> Three
Scales Report -> Process
Compliance Scale Process
Compliance Scale Score - %)
How Does APS Use Data for Managing
and Enhancing the Program?
• Systemic/periodic data collection
(management reporting system/tools).
• Enhance employee performance.
• Measure the operationalization of
certain policies.
• Regional/unit/worker development.
FY 2010 In-Home Program Performance Dashboard
Process
*Case
CompliInitiation
ance
% Inv
***
Progress Caseload
Svc
Aver Daily
Region
Outcome
1
2
3
4
5
6
7
8
9
10
11
State
94%
94%
90%
93%
89%
89%
93%
84%
95%
89%
94%
91%
98%
99%
94%
96%
94%
93%
96%
87%
98%
92%
96%
94%
93%
96%
90%
92%
91%
89%
93%
84%
95%
88%
94%
91%
97%
97%
95%
97%
94%
96%
94%
90%
97%
91%
98%
95%
95%
97%
89%
95%
90%
91%
95%
87%
95%
92%
95%
92%
91%
92%
87%
90%
80%
77%
85%
63%
89%
84%
91%
76%
87%
96%
87%
86%
76%
83%
90%
70%
93%
79%
87%
85%
32
21
45
38
42
36
41
49
30
53
45
40
51
29
49
47
51
47
49
81
37
51
52
51
83
50
94
85
93
83
90
130
67
104
97
91
83%
78%
65%
76%
77%
72%
59%
64%
80%
71%
68%
70%
13%
6%
13%
3%
8%
8%
13%
5%
12%
11%
6%
9%
6%
10%
15%
9%
11%
15%
14%
12%
9%
11%
11%
12%
73%
56%
34%
61%
58%
49%
37%
47%
61%
48%
53%
49%
33
23
41
31
39
30
36
43
24
51
37
36
State FY10
State FY09
State FY08
State FY07
State FY06
91%
91%
93%
91%
90%
94%
92%
91%
89%
90%
91%
88%
86%
85%
88%
95%
95%
94%
94%
95%
92%
91%
90%
87%
84%
76%
71%
70%
58%
52%
85%
80%
NA
NA
NA
40
43
44
47
52
51
51
54
68
67
91
94
98
115
118
70%
73%
73%
74%
75%
9%
11%
11%
11%**
12%
12%
11%
12%
12%
49%
52%
51%
51%
52%
36
32
30
36
51
Casereading Database
*FTF
Contact
Sum
*Monthly
Inv
SD
Supervisor Rapid
Doc.
Validation
Status
Duration Duration INV + SD
Rejection Closure
Time. ****
Rate
Contact
Mean
Mean Duration
Rate
Rate
Investigation
HHSC Data
Data Warehouse
In-home Performance Summary: Region 3 Compared to State
Measure
2006
2007
2008
2009
Number of Intakes
Region
State
Average Investigation Duration (Days)
Region
State
Averaged Service Delivery Duration (Days)
Region
State
Timeliness of Documentation Rate (%)
Region
State
Average % of Investigations Pending 60+
(Days)
Region
State
Rapid Closure Rate
Region
State
Monthly Status Contacts Rate
Region
State
Face-to-Face Initiation Rate
Region
State
Overall Case Reading Scores
Investigation 3 Scale Region
State
Outcome 3 Scale Region
State
2010
13,605
82,029
13,016
77,238
14,307
83,605
15,817
89,489
18,888
103,401
65
58
62
52
60
49
52
48
51
45
74
67
85
71
59
54
45
51
49
51
NA
NA
NA
NA
79%
83%
84%
83%
87%
84%
30
27
31
23
28
23
22
23
23
19
14%
11%
15%
12%
14%
12%
14%
12%
15%
12%
47%
48%
55%
58%
64%
70%
72%
71%
87%
82%
80%
95%
84%
94%
87%
94%
87%
95%
89%
92%
88%
90%
89%
90%
89%
91%
87%
89%
93%
93%
90%
91%
90%
91%
91%
92%
90%
91%
94%
94%
Monthly Summary of Pending APS State Supported Living Center Investigations
DATE
Under
10
Days
10 Days or
More
10+ Days with
Extension
10+ Days without
Extension
Percent 10+
Days
Percent 10+ Days
without Extension
Sunday
1-May-2011
2-May-2011
22
23
22
1
51.11%
2.22%
3-May-2011
20
23
23
0
53.49%
0.00%
4-May-2011
25
23
22
1
47.92%
2.08%
5-May-2011
24
23
22
1
48.94%
2.13%
6-May-2011
26
24
24
0
48.00%
0.00%
How does APS Use Data to Assess
and Enhance Quality of Casework
and Investigations?
• Appropriate notifications to law enforcement and/or
other entities.
• Capture critical components of casework.
• Establish clear investigation performance standards.
• Ensure compliance with documentation
requirements.
• Establish process for reviewing staff decisions.
• Reporting to external and/or internal Stakeholders.
Where Does All This Data
Come From?
• The intranet, of course
– Case reading database
– MRS data warehouse
• Accessible to:
– EVERYONE!
For Discussion:
What are the consequences
- positive and negative –
of using data to drive
accountability and performance?
What are the consequences - positive and
negative - of using data to drive
accountability and performance?
• Provide data to inform employee and program
accountability
–
• Objective, 3rd party, consistent, measures compliance with the most
important policy
• Valuable resource for conversations between supervisors and
employees about employee’s development
–
• When used in evaluations, QA data is perceived as punitive, not
constructive
• Case reading sample size is small at employee level
• What should determine employee performance: subjective,
inconsistent supervisors or biased, inconsistent case readers?
What are the consequences - positive and
negative - of using data to drive
accountability and performance?
• Provide data for managing and enhancing the
program
–
• “What gets measured gets moved”
–
• “All management cares about is ‘the numbers’”
What are the consequences - positive and
negative - of using data to drive
accountability and performance?
• Provide data to assess and enhance quality
of casework and investigations.
–
• Zero in on cause and effect by
disaggregating and examining
• Track and trend key issues
–
• Data must be accurate and reliable
(“faceplate game”)
Karl Urban
Director of Performance and Policy
Development
Texas Adult Protective Services
karl.urban@dfps.state.tx.us
512.438.5518