From Outcomes to Impact How to Create Change in Family Violence Dr. Robbie Babins-Wagner CEO, Calgary Counselling Centre Adjunct Professor, Faculty of Social Work, University of Calgary • For at least the past 10 years funders of nonprofit organizations have required outcome data as a measure of accountability. • Agencies are increasingly reporting on program outcomes on an annual basis. FCSAA Power of Prevention Conference, November 29. 2013 • The outcome agenda has brought greater accountability to the sector but there exists a gap in moving from outcomes to impact • How do we improve the overall outcomes for the clients that we serve. FCSAA Power of Prevention Conference, November 29. 2013 My interest outcome and evaluation began almost 30 years ago when I worked in health care and extended to my work at Calgary Counselling Centre. It lead to my completing my doctorate in outcomes and now is a prime focus of my work at CCC and the collaborative work I am doing with colleagues through the Center for Clinical Excellence in Chicago FCSAA Power of Prevention Conference, November 29. 2013 • It is also a focus of my work through the United Way of Calgary’s Leading Boldly Network • The Network is focused on – Building new capacity in Calgary for Collaborative Social Innovation (CSI), and – Making progress on complex problems that are beyond the mandate of any individual organization. FCSAA Power of Prevention Conference, November 29. 2013 • Research suggests that most agencies have some outcome data • But even with having the data, we have not been successful in achieving significantly better outcomes at a magnitude that matches the need in critical areas such as mental health, child outcomes, school completion and more. FCSAA Power of Prevention Conference, November 29. 2013 • Its suggested that – we have not marshalled the full extent of available knowledge and applied it to complex problems… – which would in turn generate new knowledge to address some of our toughest social problems. FCSAA Power of Prevention Conference, November 29. 2013 Frustrations 1. On the part of non-profit leaders who have not been able to find support (financial and non-financial) from funders for collecting and using information to manage to outcomes. 2. On the part of funders who have become frustrated with non-profits who have no means of determining whether they re doing what they say they do. FCSAA Power of Prevention Conference, November 29. 2013 • The research is clear, it take a lot on the part of both agency leaders and funders to support the changes being required to solve complex problems. • Having a strong outcome platform is the first step for agencies. FCSAA Power of Prevention Conference, November 29. 2013 • The focus on having a strong outcome platform requires a difficult cultural transition from simply having a genuine interest in improvement to truly infusing outcome thinking into the way we manage our organizations FCSAA Power of Prevention Conference, November 29. 2013 • Some funders suggest that to do this properly, requires a full 10% of their available funds for direct investment (on their parts) while at the same time supporting agencies to alter their culture and develop the human and IT systems necessary to manage to outcomes (Morino, 2010). • Where this was initially tested, only 5 of 12 (42%) agencies achieved this kind of transformative change FCSAA Power of Prevention Conference, November 29. 2013 • There are funders (in the US) who are investing in non-profit leaders who embrace the value of great information, even it they haven't had the external support to build the systems they need for collecting and using information . FCSAA Power of Prevention Conference, November 29. 2013 • Funders are increasingly focused on outcomes that lead to larger impact: • They are: – asking grant applicants to demonstrate a predisposition for using information to guide operations – providing longer term funding, with clear expectations that it will take at least 2-3 years of intensive work to create a true outcome focused culture – learning to be more flexible in how to support this work FCSAA Power of Prevention Conference, November 29. 2013 • Funders have also learned: – They cant impose this type of change – They need to give the agencies the time and space to do it their own way, if agencies choose to do it at all – The results aren’t going to sustain if the agencies don’t own the process and resulting systems – This work isn’t for everyone. FCSAA Power of Prevention Conference, November 29. 2013 • And when non-profit agencies elect to do this work they: – do a better job of meeting their missions – become more focused and disciplined in managing their organizations – Are enabled towards a path toward managing to outcomes – strengthen internal culture of the agency FCSAA Power of Prevention Conference, November 29. 2013 Moving from outcomes to impact will bring forth the following results: • staff will develop better skills in their core capacity (i.e. counselling) • program managers will develop better people management skills and have the tools they need • board members will have the information to provide more effective oversight All of which lead to better results for clients FCSAA Power of Prevention Conference, November 29. 2013 Steps to get the process started: 1. Determine who you will get feedback from clients (service users) 2. Learning must be the primary goal of the phase one where you – Collect information about the problem – Collect information about possible solutions 3. Apply rigor within reason 4. Be practical 5. Create a learning culture FCSAA Power of Prevention Conference, November 29. 2013 • The Problem Despite all the right intentions many non-profits do not have the benefit of; – good information – tools to determine where they are headed – mapping a logical course towards an objective, and – course-correct when required. FCSAA Power of Prevention Conference, November 29. 2013 – While these are not new problems, current economic conditions locally, provincially federally and globally suggest that we are likely to see an increased focus on results to support decreased funds for social and human services – The research suggests that organizations which demonstrate meaningful, lasting impact will be the focus of increased investment over the next decade. FCSAA Power of Prevention Conference, November 29. 2013 • There is also the suggestion that individual organizations who can demonstrate impact will be brought together with other like minded organizations to work together in disciplined ways toward collective impact (Kania & Kramer, 2011). • An additional focus will be a focus on working across silo’s - within organizations, across organizations in the community and within government FCSAA Power of Prevention Conference, November 29. 2013 As non-profits focus on improved client benefit, we need to: – clarify the results (outcomes) we are trying to achieve – collect the information that can best help us navigate towards these outcomes – differentiate between operational performance (overhead costs) and organizational effectiveness (results or outcomes) – ensure we have the culture to support our work FCSAA Power of Prevention Conference, November 29. 2013 In Sum • Many non-profits have no reliable way to know whether they re on track to deliver on their promise • We need to invest in continuous collection and use of information (data) to guide program/agency decisions and operations • This change requires a significant culture shift within an organization – it is primarily about culture and people (not numbers) FCSAA Power of Prevention Conference, November 29. 2013 In Sum (cont.) • We must focus on why measure and on what to measure – not just on how to measure • The non-profit needs to drive the change and be the primary beneficiary of it • Reasonableness and common sense must guide both agency and funder investment decisions FCSAA Power of Prevention Conference, November 29. 2013 Tris Lumley, head of strategy for New Philanthropy Capital (London) suggests that “Great organizations …. are build around great data. Data that [allow] then to understand the needs they address, what activities are likely to best address these needs, what actually happens as a result of these activities, and how to allocate resources and tweak what they do for even greater impact. Too often, funders set the agenda with their own requirements [and]cripple the organizations they are trying to help. FCSAA Power of Prevention Conference, November 29. 2013 So how do we do this? How do we manage to outcomes? FCSAA Power of Prevention Conference, November 29. 2013 5 Steps • Develop clarity through reflection and analysis on what change you are trying to create • Gain specificity on how you will accomplish this change • Determine what information (hard and soft) will be most helpful for gauging whether you are on or off track to achieve that change • Collect and use this information to plan, make important decisions, track, course-correct and improve • Combine the steps above with good practice judgement and keen discernment, which are more important than any single metric . FCSAA Power of Prevention Conference, November 29. 2013 Of most importance • Gain clarity on what change you are trying to achieve • Develop specificity on how you will accomplish this change • Determine what information will be most helpful for gauging whether you are on or off course • Collect and use this information as the basis for understanding what’s working, planning, decision making and improvement • Lastly, the technology behind the systems is imporntat and necessary but is not nearly as important as the mindset of the leaders who put these systems in place FCSAA Power of Prevention Conference, November 29. 2013 Organizational Culture • Recruit culture leaders • Walk the talk • Know what you stand for (core beliefs & guiding principles) • Be able to answer the question “To what end” – improved client results → community impact • Ensure that everyone is moving in the same direction • Be clear and direct about expectations • Encourage self-improvement and personal growth FCSAA Power of Prevention Conference, November 29. 2013 A Case Example FCSAA Power of Prevention Conference, November 29. 2013 The OQ 45 • • • • a 45-item self-report measure takes about five minutes to complete grade 6 reading level provides a total score, based on all 45 items, as well as three subscales: symptoms of distress, interpersonal relationships and social role functioning • scores range from 0 – 180, with a clinical cut-off of 63 FCSAA Power of Prevention Conference, November 29. 2013 Oq 45.2 Total Score 160 140 120 Score 100 Score 80 Cutoff Score 60 40 20 0 59 60 61 62 63 64 65 66 67 68 Session Number FCSAA Power of Prevention Conference, November 29. 2013 69 70 Clinical Significance • Criteria for clinically significant change (Jacobson & Truax, 1991) are used to make judgements of client outcomes • Clients who change in a positive or negative direction by at least 14 points are regarded as having made “reliable change” • The clinical cut-off of 63 differentiates clients in the dysfunctional range from those of a functional population • Very high clinical standard to meet (very unlikely to happen by chance) • Most clinicians are satisfied with a smaller point change in outcome suggesting positive or negative movement. FCSAA Power of Prevention Conference, November 29. 2013 Four Categories of Change for the OQ Category Definition No Change Any score that sows neither reliable (14 point) improvement or deterioration. Deteriorated The client’s score is at least 4 points worse than the first OQ ( reliable negative change) Improved The client’s final OQ score is at least 14 points less than their first session OQ Score Recovered The client’s final OQ score is at least 14 points less than their first session score and the client’s final OQ score is at or below the clinical cut-off of 63. FCSAA Power of Prevention Conference, November 29. 2013 N Deteriorated % No Change % Improved % Recovered % Total Improved & Recovered 14.1 35 Baseline 6072 CCC 2009 2052 147 7.2 1021 49.8 471 23.0 413 20.1 43.1 CCC 2010 2284 173 7.6 1085 47.5 519 22.7 509 22.3 45 CCC 2011 2469 160 6.5 1152 46.7 612 24.8 545 22.1 46.9 CCC 2012 2288 148 6.5 1023 44.7 597 26.1 520 22.7 48.8 Total 16680 1237 7.4 7909 47.4 3771 22.6 3390 20.3 42.9 8.2 56 20.9 FCSAA Power of Prevention Conference, November 29. 2013 2004 - 2012 Average OQ at First Session Average OQ at Last Session Average Change Full Time n=2759 68.74 S=25.13 59.42 S= 26.61 -12.62 S=20.76 Associate n=1528 69.73 S=25.96 59.94 S=27.45 -12.62 S=20.63 Resident N=5377 75.47 S=25.69 64.99 S=27.70 -13.25 S=21.73 Intern N=19153 75.95 S=26.46 65.51 S=28.66 -13.41 S=21.46 Total N=18,817 74.07 S=26.14 63.81 S=28.04 -13.32 S=21.35 FCSAA Power of Prevention Conference, November 29. 2013 Why Family Violence • Everyone is vulnerable to family violence • It is a significant social problem (Johnson, 2006) • The financial cost to Canadian Society – In 2009, the total economic impact of spousal violence was estimated at $7.4 Billion (Zhang, Hoddenbagh, McDonald& Scrim, 2012) FCSAA Power of Prevention Conference, November 29. 2013 The 2013 Stats Can report (based on 2009 data). • Only 22% of victims reported violence to the police • In Canada, in 2008, 334,000 individuals were victims of self reported violence at the hands of an intimate partner – 6% of the adult population • Alberta has the second highest rate of self reported abuse, followed by Saskatchewan. • The consequences are significant for victims & children as well as the abuser FCSAA Power of Prevention Conference, November 29. 2013 Demographic Predictors: • Younger • Living in a common-law relationship • Living in a step family • Income levels and education levels were not predictors of family violence FCSAA Power of Prevention Conference, November 29. 2013 CCC Male Family Violence Clients (2008 – 2012) On average • 35 years old, range of 18 – 76 • Median income: $25,000-35,000 • Marital status: Single (41.1%); Married/Common-Law (38.9%); Separated/Divorced (19.9%) FCSAA Power of Prevention Conference, November 29. 2013 Calgary Counselling Centre’s Responsible Choices for Men Program The primary goal is to assist men to become violent free Major objectives include: o decreasing all forms of abusive behaviour, o accepting responsibility for one’s behavior, o increasing self-esteem, o improving family relations, o decreasing stress, o increasing empathy towards the victim, and o stopping abuse towards children FCSAA Power of Prevention Conference, November 29. 2013 Program Elements: • Counselling to engage in process – Specialized 14 week (30 hour) Group – Counselling for any outstanding issues • Outcome measures • OQ data is collected on a session by session basis during counselling, at the beginning of group, at the end of group and at the end of counselling FCSAA Power of Prevention Conference, November 29. 2013 Responsible Choice for Men Outcomes (2008-2012): • 1545 clients (individual + group) • Of these clients, 607 went on to participate in a specialized family violence group whereas 938 proceeded with counselling • Average number of counselling sessions for those who participated in counselling and group = 6 – The range of counselling sessions for those who did not participate in group was 1-41. – The range of counselling sessions for those who participated in group was 1-23. FCSAA Power of Prevention Conference, November 29. 2013 RCM Counselling Average only Results First 2008 - 2012 Session OQ Average Change Last Session Score OQ Results n=606 53.4 41.7 t(602) = 13.31, p <.001* Range of OQ scores 2-144 1-153 -11.7 FCSAA Power of Prevention Conference, November 29. 2013 Combined Programming RCM N=299 N=292 Unadjusted Adjusted Score score Change Score Change Score Unadjusted Adjusted First Counselling Session 45.2 85.7 First Session of Group 33.1 77.9 -11.78 Last Session of Group 28.6 71.4 -4.5 -6.5 From first session of counselling to last session of group ----------- --------- -16.6 -14.3 FCSAA Power of Prevention Conference, November 29. 2013 Results – Results – Unadjusted Adjusted scores scores t(299) = 12.79, p <.001* t(291)=10.6, p<0.001* Combined Programming N=299 First Counselling Session 45.2 First Session of Group 33.1 -12.1 Last Session of Group 28.6 -4.5 From first session of counselling to last session of group Change Score -16.6 Results (Unadjusted) t(299) = 12.79, p <.001* FCSAA Power of Prevention Conference, November 29. 2013 Combined Programming N=292 First Counselling Session 85.7 First Session of Group 77.9 -7.8 Last Session of Group 71.4 -6.5 From first session of counselling to last session of group Change Score -14.3 Results – adjusted for social desirability t(291) = 10.63 p <.001* FCSAA Power of Prevention Conference, November 29. 2013 Category Unadjusted Individual Counselling RCM CCC Cases US Baseline Study Deteriorated 43 (7%) 648 (7.6%) 8.2% No Change 299 (50%) 4055 (47.4%) 56% Improved 183(30%) 1933 (22.6%) 20.9% Recovered 78(13%) 1915 (22.4%) 14.1% Total Improved or Recovered 261(43) 3848 (45%) 35% 637 8,551 6,072 Total FCSAA Power of Prevention Conference, November 29. 2013 Category From first RCM Counselling session to first group session CCC Cases US Baseline Study 31 (5.7%) 648 (7.6%) 8.2% No Change 305 (56.1%) 4055 (47.4%) 56% Improved 168 (30.9%) 1933 (22.6%) 20.9% Recovered 40 (7.4%) 1915 (22.4%) 14.1% Total Improved or Recovered 208 (38.8) 3848 (45%) 35% 544 8,551 6,072 Deteriorated Total FCSAA Power of Prevention Conference, November 29. 2013 Category Unadjusted From first Family Violence counselling session to last group session CCC Cases US Baseline Study Deteriorated 11 (7.2.%) 648 (7.6%) 8.2% No Change 62 (40.8%) 4055 (47.4%) 56% Improved 58 (38.2%) 1933 (22.6%) 20.9% Recovered 21 (13.8%) 1915 (22.4%) 14.1% Improved or Recovered 79 ( 52%) 45% 35% 152 8,551 6,072 Total FCSAA Power of Prevention Conference, November 29. 2013 Category Adjusted for social desirability From first Family Violence Counselling session to last group session CCC Cases US Baseline Study Deteriorated 11 (7.2%) 648 (7.6%) 8.2% No Change 62 (40.8%) 4055 (47.4%) 56% Improved 58 (38.2%) 1933 (22.6%) 20.9% Recovered 21 (13.8%) 1915 (22.4%) 14.1% 79(52%) 45% 35% 152 8,551 6,072 Improved or Recovered Total FCSAA Power of Prevention Conference, November 29. 2013 Combined N=152 Programming to (unadjusted) counselling follow-up First Counselling Session Change Score Results 45.3 First Session of Group 33.1 -12.2 Last Session of Group 30.1 -3 Last Counselling Session 28.8 -16.5 FCSAA Power of Prevention Conference, November 29. 2013 T(151)=9.4, p<0.001* Conclusion – Overall results support continued use of this framework of individual and group – Decreasing the number of clients doing individual counselling only – Will work with the program to support best results for clients FCSAA Power of Prevention Conference, November 29. 2013 robbie.wagner@calgarycounselling.com www.calgarycounselling.com FCSAA Power of Prevention Conference, November 29. 2013