The Linkages Evaluation Coaching Project Amy D’Andrade San Jose State University School of Social Work May 19, 2015 Presentation Outline 1. Introduction – Evaluation basics – Our process 2. Results – – – – 3. 4. Implementation Process investigation Case Studies [Outcomes] and Dissemination What was it like? Concluding thoughts Introduction Steps in Evaluation ID team Adjust practice Analyze data Develop logic model Gather data What is a Logic Model? “.. a simple diagram that communicates the rationale for why a program will work.” - Linkages Toolkit PROGRAM ACTIVITIES LONG TERM OUTCOME INPUTS OUTPUTS Program resources Activities Participation/ What we invest What we do Who we reach OUTCOMES Short Medium What results we obtain Longterm INPUTS ACTIVITIES Linkages Team Development of coordinated case plan DPSS staff CWS staff Service providers Families CFPIC DPSS staff attends TDMs Regular communication between DPSS and CWS workers OUTCOMES Case plans and casework draw on knowledge and resources of both divisions Services offered not duplicative Greater service availability/ fewer waitlists Services offered provide for more family needs Training for CWS staff on Linkages Identification of shared families All eligible families receive Linkages services [To Linkages activities 1,2,3] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS Two Approaches to Evaluation Outcome Evaluation: Assess the impacts or outcomes of programs INPUTS ACTIVITIES Linkages Team Development of coordinated case plan DPSS staff CWS staff Service providers Families CFPIC DPSS staff attends TDMs Regular communication between DPSS and CWS workers OUTCOMES Case plans and casework draw on knowledge and resources of both divisions Services offered not duplicative Greater service availability/ fewer waitlists Services offered provide for more family needs Training for CWS staff on Linkages Identification of shared families All eligible families receive Linkages services [To Linkages activities 1,2,3] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS Two Approaches to Evaluation Process Evaluation: Assess the delivery of programs INPUTS ACTIVITIES Linkages Team Development of coordinated case plan DPSS staff CWS staff Service providers Families CFPIC DPSS staff attends TDMs Regular communication between DPSS and CWS workers OUTCOMES Case plans and casework draw on knowledge and resources of both divisions Services offered not duplicative Greater service availability/ fewer waitlists Services offered provide for more family needs Training for CWS staff on Linkages Identification of shared families All eligible families receive Linkages services [To Linkages activities 1,2,3] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS Evaluation Components I.Implementation Tracking: Be able to track what proportion of Linkages cases receive primary Linkages activities, and how that changes over time II.Process Investigation: Understand why CWS/CMS communication falters or doesn’t occur III.Case Studies: Illuminate why and how Linkages activities, when received, make a difference for families IV.Outcomes Description: Identify relevant child welfare and W2W outcomes of cases receiving Linkages V.Dissemination: Share results and lessons learned with stakeholders Evaluation Coaching Support Intensive coaching with 3 focus counties: Madera, Los Angeles, Tulare – Initial in-person visit – On-line or phone coaching sessions – On-going email conversations and phone calls – Concluding site visits and presentation of results/report/recommendations So far with Coaching counties: 1. Identify program activities 2. Brainstorm areas of concern, interest, and related questions 3. Prioritize and select evaluation questions 4. Operationalize service outputs 5. Identify data sources 6. Collect data – in progress 7. Analyze data – in progress 8. Report results – in progress Results I. Implementation Tracking: Be able to track what proportion of Linkages cases receive primary Linkages activities, and how that changes over time. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Greater service availability/ fewer waitlists Services offered provide for more family needs Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS LA Linkages – FP Protocol I. Implementation Concern: Are eligible cases being identified? Data collection on 8 of 14 sites: • 30 FP cases randomly selected from each site • Checked CW data forms for eligibility code used to refer to Linkages • Checked eligibility for W2W/CalWORKS per DPSS system % Correctly Identified: Of those DPSS determined to be eligible, what % were identified as eligible by CW? 38% 0% 44% 50% 55% 54% Site 1 67% 70 % 90% 100% Site 6 % Correctly Identified: Of those DPSS determined to be eligible, what % were identified as eligible by CW? 38% 0% 44% 50% 55% 54% Site 1 67% 70 % 90% 100% Site 6 What steps might the county take based on this data? INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Greater service availability/ fewer waitlists Services offered provide for more family needs Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS Implementation Tracking Tulare and Madera Case ID SW W2W EW Tandem Visit Coord Case Plan Comm Occurred Smith Jenny Redd Clyde Downes Tom Garcia Yes 1/23/15 Yes 1/23/15 Yes Barnes Donna Todd Clyde Downes Maria Thomas Craig Mike Dunn Margo David Tom Garcia Yes 3/15/15 Percentage of eligible active cases receiving Linkages activities - May What might the county do based on this data? Percentage of cases receiving all Linkages activities over time II. Process Investigation: Why does CWS/CMS communication falter or not occur? INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Greater service availability/ fewer waitlists Services offered provide for more family needs Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS Identified all workers on cases for which Linkages activities had not occurred Created and sent email survey to these workers When a child welfare case worker receives a new case that is Linkageseligible, the worker is to reach out and connect with the Welfare-to-Work case manager - invite them to a staffing, or just establish communication. This doesn't seem to have happened on a recent case of yours. Worker 1: Worked for the agency over 5 years Do you remember why it didn't happen? What got in the way for you? What hinders you or gets in the way of your making this connection? “With changes occurring, processes get lost.” “It's difficult to find time when both parties can meet and discuss the case.” What might increase the likelihood that you would take this action? Is there anything else you would like to say about this issue? “Establish a process that we can all work with.” “Having us meet to discuss the case is important and needed.” Worker 2: Worked for the agency 1-5 years Do you remember why it didn't happen? What got in the way for you? “I don't see the need/value of taking that action.” What might increase the chance that you would take this action? “If there was a more effective way of communicating that there is a link.” What hinders you or gets in the way of your making this connection? “There are many service providers that need to be contacted and referrals to be completed and it would be helpful if the WTW case manager were to reach out to the SW instead of the other way around.” Is there anything else you would like to say about this issue? [no response] Worker 3: Worked for the agency less than 1 year Do you remember why it didn't happen? What got in the way for you? What hinders you or gets in the way of your making this connection? “Not sure what you are referring to.” “The Linkages worker does not come to the meetings.” What might increase the likelihood that you would take this action? Is there anything else you would like to say about this issue? “?” “If I knew what you were referring to, I could answer.” What might the county do based on this data? III. Case Studies: Understand and describe why and how Linkages activities, when received, make a difference for families INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Greater service availability/ fewer waitlists Services offered provide for more family needs Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2WSingle and CWmother SWs) Training for CWS staff on Linkages Agencies Fresno Academy Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS with 4 children all under the age of 5. Father was in prison. Mother had Monthlyserious Linkagessubstance abuse problem. Had not committee in many staffyears (if at all). meeting workedIncreased Court Child Abuse Council Case plans and casework draw on knowledge and resources of both divisions Identification of shared families knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Court Child Abuse Council Tandem monthly meetings (W2W and CW SWs) Regular communication parent (W2WTwo and CW SWs) Greater service availability/ fewer waitlists Services Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS family, five children removed when offered provide for both parents were taken tomore jail.family Mother was pregnant. Once released fromneeds jail, the family Monthly Linkages committee had great difficulty finding housing due to Increased staff meeting parents’ lowknowledge incomeon(father undocumented), value of multiple prior evictions, and criminal records. collaboration, Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Linkages program More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Identification of shared families Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer waitlists Families better able to complete case plan requirements Increased case closure CWS / DPSS supports DPSS “One thing for me, anytime that we had a staffing, I always wanted to have the W2W worker involved…because…(he Families Decreased more likely to re-entry might) be aware of something Services receivethat I CWS / offered renewal might not be aware needed of.” provide for more family needs Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS “I think Cesar got quite a few emails from me, wanting to find out what’s Increased staff to (the mom) and whether available knowledge on value of option or not…The fact that it’s an collaboration, we’re in the same building sitting close Linkages program by each other makes it very easy to ask questions and be able to plan…” More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Training for CWS staff on Linkages Agencies Fresno Academy Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS “We kind of help back each other up in a way. If I would forget something with the family and Cesar went to see them the next Increased day,staff he would send me an email – ‘Hey knowledge on Maria, I went to go visit them and you value of collaboration, forgot…’ And I think we were just better able Linkages program to help the family because we had each other.” [CW SW] Case Study 1 Court Child Abuse Council Case plans and casework draw on knowledge and resources of both divisions Identification of shared families More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting Court Child Abuse Council Training for CWS staff on Linkages Agencies Fresno Academy Identification of shared families Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS “…Once she got her children back she wanted to start working – so we looked to see if she could receive childcare through Linkages, through W2W.” [CW SW] Increased staff knowledge on value of collaboration, Linkages program Case Study 1 “She’s now working at [fast food] about 30 hours a week. Critical to keep that job is child care. So we secured child care with a relative, and we can Morethat families pay relative,[Toand that’s what we’re doing.” Linkages receive activities 1-4]SW] [W2W Linkages services INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Tandem monthly meetings (W2W and CW SWs) Case plans and casework draw on knowledge and resources of both divisions Regular communication (W2W and CW SWs) Greater service availability/ fewer waitlists Services offered provide for more family needs Monthly Linkages committee “I meeting Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS think the family was encouraged to see all the Increased staff knowledge on people involved value of in dealing with the different issues.” collaboration, [W2W SW] Linkages Training for CWS staff on Linkages “I think program it kind of gave them peace of mind that not families on the case, but so was Cesar. That only I wasMore working [To Linkages receive Identification of 1-4] just made them feel activities more supported from our Linkages shared families services department.” [CW SW] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee meeting “It’s Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS stillIncreased in progress. We’re still assisting with child care staff knowledge on because she has an open W2W case. She’s not value of collaboration, generating enough income to get off aid, but of course Linkages a much better position now because of …the Training for CWSshe’s in program staff on Linkages earnings that she generates. She’s able to take care of More families her needs better. And [To at Linkages the end of this year, she’s be receive Identification of activitiesTax 1-4] Credit. That’s probably able to claim Earned Income Linkages shared families services something she hasn’t seen in a while - or never.” Case Study 1 INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth “This past CFPIC Tandem monthly meetings (W2W and CW SWs) Regular communication (W2W and CW SWs) Monthly Linkages committee month was the meeting Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer waitlists Services offered provide for more family needs Families better able to complete case plan requirements Increased case closure CWS / DPSS Families more likely to receive needed supports Decreased re-entry CWS / renewal DPSS first Increased monthstaff that the family paid their own rent. knowledge on The program was able to give them a month, and that enabled them value$800 of Court collaboration, to save money to get a car…since W2W helped them with essential Linkages Child Abuse items…they Training were able to get other items on their own…a little table with program for CWS Council staff on Linkages some chairs, a little TV, and now a couch….they’ve felt empowered Agencies because now they’re doing itMore onfamilies their own…The work we completed [To Linkages receive Identification of activities 1-4] Fresno together was far more than justLinkages a patch. We enabled them to get a little shared families Academy services bit ahead of the game, and they are going to maintain momentum.” INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative Families better able to complete case plan requirements Increased case closure CWS / DPSS Families Regular “This particular mom, she was extremely more likely to communication Services free from receive committed and dedicated to being (W2W and CW offered needed child constantly calling SWs) welfare services. She wasprovide for supports Decreased re-entry CWS / renewal DPSS CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC Tandem monthly meetings (W2W and CW SWs) meeting Training for CWS staff on Linkages Agencies Fresno Academy Greater service availability/ fewer waitlists more family or texting me or Cesar. She kept us as needs Monthly Linkages accountable as we kept her, if not more.” committee Court Child Abuse Council Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs Development of coordinated case plan Services offered not duplicative CW SWs Supervisors Exec Team Community/ Employers Families, Parents, Youth CFPIC “The W2W worker did a lotGreater with this Tandem monthly service meetings (W2W family. As soon as we got them availability/ and CW SWs) meeting Training for CWS staff on Linkages Agencies Fresno Academy Families better able to complete case plan requirements waitqualified, he was able to fewer secure the lists Families Regular housing….Because without him, no one more likely to communication Services have rented to this family… he receive (W2W andwould CW offered needed SWs)met with the landlord andprovide for was able to supports more family explain how the program worked needs – and Monthly Linkages committee that accelerated everything.” Court Child Abuse Council Case plans and casework draw on knowledge and resources of both divisions Identification of shared families Increased staff knowledge on value of collaboration, Linkages program More families receive Linkages services [To Linkages activities 1-4] Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS What might the county do based on this data? I. Outcomes Description: Identify relevant child welfare and W2W outcomes of cases receiving Linkages Dissemination I. All-county Webinars II. Convening Presentation III. Linkages County Team Meeting IV. County Stakeholder Presentation V. Other Ideas? What was it like? County perspective Coach perspective Concluding Thoughts Re. Linkages 1.Implementation concerns well-founded – tracking efforts + practice adjustments + ongoing training are needed 2.When families receive Linkages activities, it can work very well 3.Better connections needed between IT/Data folks and practice Re. Evaluation 1. Evaluation takes time! (We were overambitious) 2. The focus, structure, and expertise provided by the coaching process was helpful to the evaluation effort 3. Evaluation coaching process would have benefited from more in-person meetings Questions? amy.dandrade@sjsu.edu