From Outcomes to Impact
How to Create Change in Family
Violence
Dr. Robbie Babins-Wagner
CEO, Calgary Counselling Centre
Adjunct Professor, Faculty of Social Work,
University of Calgary
• For at least the past 10 years funders of nonprofit organizations have required outcome
data as a measure of accountability.
• Agencies are increasingly reporting on
program outcomes on an annual basis.
FCSAA Power of Prevention Conference,
November 29. 2013
• The outcome agenda has brought greater
accountability to the sector but there exists a
gap in moving from outcomes to impact
• How do we improve the overall outcomes for
the clients that we serve.
FCSAA Power of Prevention Conference,
November 29. 2013
My interest outcome and evaluation began
almost 30 years ago when I worked in health
care and extended to my work at Calgary
Counselling Centre.
It lead to my completing my doctorate in
outcomes and now is a prime focus of my work
at CCC and the collaborative work I am doing
with colleagues through the Center for Clinical
Excellence in Chicago
FCSAA Power of Prevention Conference,
November 29. 2013
• It is also a focus of my work through the
United Way of Calgary’s Leading Boldly
Network
• The Network is focused on
– Building new capacity in Calgary for Collaborative
Social Innovation (CSI), and
– Making progress on complex problems that are
beyond the mandate of any individual
organization.
FCSAA Power of Prevention Conference,
November 29. 2013
• Research suggests that most agencies have some
outcome data
• But even with having the data, we have not been
successful in achieving significantly better
outcomes at a magnitude that matches the need
in critical areas such as mental health, child
outcomes, school completion and more.
FCSAA Power of Prevention Conference,
November 29. 2013
• Its suggested that
– we have not marshalled the full extent of available
knowledge and applied it to complex problems…
– which would in turn generate new knowledge to
address some of our toughest social problems.
FCSAA Power of Prevention Conference,
November 29. 2013
Frustrations
1. On the part of non-profit leaders who have
not been able to find support (financial and
non-financial) from funders for collecting and
using information to manage to outcomes.
2. On the part of funders who have become
frustrated with non-profits who have no
means of determining whether they re doing
what they say they do.
FCSAA Power of Prevention Conference,
November 29. 2013
• The research is clear, it take a lot on the part
of both agency leaders and funders to support
the changes being required to solve complex
problems.
• Having a strong outcome platform is the first
step for agencies.
FCSAA Power of Prevention Conference,
November 29. 2013
• The focus on having a strong outcome
platform requires a difficult cultural transition
from simply having a genuine interest in
improvement to truly infusing outcome
thinking into the way we manage our
organizations
FCSAA Power of Prevention Conference,
November 29. 2013
• Some funders suggest that to do this properly, requires
a full 10% of their available funds for direct investment
(on their parts) while at the same time supporting
agencies to alter their culture and develop the human
and IT systems necessary to manage to outcomes
(Morino, 2010).
• Where this was initially tested, only 5 of 12 (42%)
agencies achieved this kind of transformative change
FCSAA Power of Prevention Conference,
November 29. 2013
• There are funders (in the US) who are
investing in non-profit leaders who embrace
the value of great information, even it they
haven't had the external support to build the
systems they need for collecting and using
information .
FCSAA Power of Prevention Conference,
November 29. 2013
• Funders are increasingly focused on outcomes
that lead to larger impact:
• They are:
– asking grant applicants to demonstrate a
predisposition for using information to guide
operations
– providing longer term funding, with clear expectations
that it will take at least 2-3 years of intensive work to
create a true outcome focused culture
– learning to be more flexible in how to support this
work
FCSAA Power of Prevention Conference,
November 29. 2013
• Funders have also learned:
– They cant impose this type of change
– They need to give the agencies the time and space
to do it their own way, if agencies choose to do it
at all
– The results aren’t going to sustain if the agencies
don’t own the process and resulting systems
– This work isn’t for everyone.
FCSAA Power of Prevention Conference,
November 29. 2013
• And when non-profit agencies elect to do this
work they:
– do a better job of meeting their missions
– become more focused and disciplined in
managing their organizations
– Are enabled towards a path toward managing to
outcomes
– strengthen internal culture of the agency
FCSAA Power of Prevention Conference,
November 29. 2013
Moving from outcomes to impact will bring forth
the following results:
• staff will develop better skills in their core
capacity (i.e. counselling)
• program managers will develop better people
management skills and have the tools they need
• board members will have the information to
provide more effective oversight
All of which lead to better results for clients
FCSAA Power of Prevention Conference,
November 29. 2013
Steps to get the process started:
1. Determine who you will get feedback from
clients (service users)
2. Learning must be the primary goal of the phase
one where you
– Collect information about the problem
– Collect information about possible solutions
3. Apply rigor within reason
4. Be practical
5. Create a learning culture
FCSAA Power of Prevention Conference,
November 29. 2013
• The Problem
Despite all the right intentions many non-profits do
not have the benefit of;
– good information
– tools to determine where they are headed
– mapping a logical course towards an objective,
and
– course-correct when required.
FCSAA Power of Prevention Conference,
November 29. 2013
– While these are not new problems, current
economic conditions locally, provincially federally
and globally suggest that we are likely to see an
increased focus on results to support decreased
funds for social and human services
– The research suggests that organizations which
demonstrate meaningful, lasting impact will be the
focus of increased investment over the next decade.
FCSAA Power of Prevention Conference,
November 29. 2013
• There is also the suggestion that individual
organizations who can demonstrate impact will
be brought together with other like minded
organizations to work together in disciplined
ways toward collective impact (Kania & Kramer,
2011).
• An additional focus will be a focus on working
across silo’s - within organizations, across
organizations in the community and within
government
FCSAA Power of Prevention Conference,
November 29. 2013
As non-profits focus on improved client benefit,
we need to:
– clarify the results (outcomes) we are trying to
achieve
– collect the information that can best help us
navigate towards these outcomes
– differentiate between operational performance
(overhead costs) and organizational effectiveness
(results or outcomes)
– ensure we have the culture to support our work
FCSAA Power of Prevention Conference,
November 29. 2013
In Sum
• Many non-profits have no reliable way to know
whether they re on track to deliver on their
promise
• We need to invest in continuous collection and
use of information (data) to guide
program/agency decisions and operations
• This change requires a significant culture shift
within an organization – it is primarily about
culture and people (not numbers)
FCSAA Power of Prevention Conference,
November 29. 2013
In Sum (cont.)
• We must focus on why measure and on what
to measure – not just on how to measure
• The non-profit needs to drive the change and
be the primary beneficiary of it
• Reasonableness and common sense must
guide both agency and funder investment
decisions
FCSAA Power of Prevention Conference,
November 29. 2013
Tris Lumley, head of strategy for New Philanthropy
Capital (London) suggests that
“Great organizations …. are build around great data. Data
that [allow] then to understand the needs they address,
what activities are likely to best address these needs,
what actually happens as a result of these activities, and
how to allocate resources and tweak what they do for
even greater impact. Too often, funders set the agenda
with their own requirements [and]cripple the
organizations they are trying to help.
FCSAA Power of Prevention Conference,
November 29. 2013
So how do we do this?
How do we manage to outcomes?
FCSAA Power of Prevention Conference,
November 29. 2013
5 Steps
• Develop clarity through reflection and analysis on what
change you are trying to create
• Gain specificity on how you will accomplish this change
• Determine what information (hard and soft) will be
most helpful for gauging whether you are on or off
track to achieve that change
• Collect and use this information to plan, make
important decisions, track, course-correct and improve
• Combine the steps above with good practice
judgement and keen discernment, which are more
important than any single metric .
FCSAA Power of Prevention Conference,
November 29. 2013
Of most importance
• Gain clarity on what change you are trying to achieve
• Develop specificity on how you will accomplish this
change
• Determine what information will be most helpful for
gauging whether you are on or off course
• Collect and use this information as the basis for
understanding what’s working, planning, decision
making and improvement
• Lastly, the technology behind the systems is imporntat
and necessary but is not nearly as important as the
mindset of the leaders who put these systems in place
FCSAA Power of Prevention Conference,
November 29. 2013
Organizational Culture
• Recruit culture leaders
• Walk the talk
• Know what you stand for (core beliefs & guiding
principles)
• Be able to answer the question “To what end”
– improved client results → community impact
• Ensure that everyone is moving in the same direction
• Be clear and direct about expectations
• Encourage self-improvement and personal growth
FCSAA Power of Prevention Conference,
November 29. 2013
A Case Example
FCSAA Power of Prevention Conference,
November 29. 2013
The OQ 45
•
•
•
•
a 45-item self-report measure
takes about five minutes to complete
grade 6 reading level
provides a total score, based on all 45 items, as
well as three subscales: symptoms of distress,
interpersonal relationships and social role
functioning
• scores range from 0 – 180, with a clinical cut-off of
63
FCSAA Power of Prevention Conference,
November 29. 2013
Oq 45.2 Total Score
160
140
120
Score
100
Score
80
Cutoff Score
60
40
20
0
59
60
61
62
63
64
65
66
67
68
Session Number
FCSAA Power of Prevention Conference,
November 29. 2013
69
70
Clinical Significance
• Criteria for clinically significant change (Jacobson & Truax,
1991) are used to make judgements of client outcomes
• Clients who change in a positive or negative direction by at
least 14 points are regarded as having made “reliable
change”
• The clinical cut-off of 63 differentiates clients in the
dysfunctional range from those of a functional population
• Very high clinical standard to meet (very unlikely to
happen by chance)
• Most clinicians are satisfied with a smaller point change in
outcome suggesting positive or negative movement.
FCSAA Power of Prevention Conference,
November 29. 2013
Four Categories of Change for the OQ
Category
Definition
No Change
Any score that sows neither reliable (14 point)
improvement or deterioration.
Deteriorated
The client’s score is at least 4 points worse than the
first OQ ( reliable negative change)
Improved
The client’s final OQ score is at least 14 points less
than their first session OQ Score
Recovered
The client’s final OQ score is at least 14 points less
than their first session score and the client’s final
OQ score is at or below the clinical cut-off of 63.
FCSAA Power of Prevention Conference,
November 29. 2013
N
Deteriorated
%
No
Change
%
Improved
%
Recovered
%
Total Improved
& Recovered
14.1
35
Baseline
6072
CCC
2009
2052
147
7.2
1021
49.8
471
23.0
413
20.1
43.1
CCC
2010
2284
173
7.6
1085
47.5
519
22.7
509
22.3
45
CCC
2011
2469
160
6.5
1152
46.7
612
24.8
545
22.1
46.9
CCC
2012
2288
148
6.5
1023
44.7
597
26.1
520
22.7
48.8
Total
16680
1237
7.4
7909
47.4
3771
22.6
3390
20.3
42.9
8.2
56
20.9
FCSAA Power of Prevention Conference,
November 29. 2013
2004 - 2012
Average OQ at First
Session
Average OQ at Last
Session
Average Change
Full Time
n=2759
68.74
S=25.13
59.42
S= 26.61
-12.62
S=20.76
Associate
n=1528
69.73
S=25.96
59.94
S=27.45
-12.62
S=20.63
Resident
N=5377
75.47
S=25.69
64.99
S=27.70
-13.25
S=21.73
Intern
N=19153
75.95
S=26.46
65.51
S=28.66
-13.41
S=21.46
Total
N=18,817
74.07
S=26.14
63.81
S=28.04
-13.32
S=21.35
FCSAA Power of Prevention Conference,
November 29. 2013
Why Family Violence
• Everyone is vulnerable to family violence
• It is a significant social problem (Johnson,
2006)
• The financial cost to Canadian Society
– In 2009, the total economic impact of spousal
violence was estimated at $7.4 Billion (Zhang,
Hoddenbagh, McDonald& Scrim, 2012)
FCSAA Power of Prevention Conference,
November 29. 2013
The 2013 Stats Can report (based on 2009 data).
• Only 22% of victims reported violence to the police
• In Canada, in 2008, 334,000 individuals were victims
of self reported violence at the hands of an intimate
partner – 6% of the adult population
• Alberta has the second highest rate of self reported
abuse, followed by Saskatchewan.
• The consequences are significant for victims &
children as well as the abuser
FCSAA Power of Prevention Conference,
November 29. 2013
Demographic Predictors:
• Younger
• Living in a common-law relationship
• Living in a step family
• Income levels and education levels were not
predictors of family violence
FCSAA Power of Prevention Conference,
November 29. 2013
CCC Male Family Violence Clients
(2008 – 2012)
On average
• 35 years old, range of 18 – 76
• Median income: $25,000-35,000
• Marital status: Single (41.1%); Married/Common-Law
(38.9%); Separated/Divorced (19.9%)
FCSAA Power of Prevention Conference,
November 29. 2013
Calgary Counselling Centre’s Responsible
Choices for Men Program
 The primary goal is to assist men to become violent
free
 Major objectives include:
o decreasing all forms of abusive behaviour,
o accepting responsibility for one’s behavior,
o increasing self-esteem,
o improving family relations,
o decreasing stress,
o increasing empathy towards the victim, and
o stopping abuse towards children
FCSAA Power of Prevention Conference,
November 29. 2013
Program Elements:
• Counselling to engage in process
– Specialized 14 week (30 hour) Group
– Counselling for any outstanding issues
• Outcome measures
• OQ data is collected on a session by session basis during
counselling, at the beginning of group, at the end of group
and at the end of counselling
FCSAA Power of Prevention Conference,
November 29. 2013
Responsible Choice for Men Outcomes
(2008-2012):
• 1545 clients (individual + group)
• Of these clients, 607 went on to participate in a
specialized family violence group whereas 938
proceeded with counselling
• Average number of counselling sessions for those who
participated in counselling and group = 6
– The range of counselling sessions for those who did not
participate in group was 1-41.
– The range of counselling sessions for those who participated
in group was 1-23.
FCSAA Power of Prevention Conference,
November 29. 2013
RCM Counselling Average
only Results
First
2008 - 2012
Session
OQ
Average
Change
Last Session Score
OQ
Results
n=606
53.4
41.7
t(602) = 13.31, p
<.001*
Range of OQ
scores
2-144
1-153
-11.7
FCSAA Power of Prevention Conference,
November 29. 2013
Combined
Programming
RCM
N=299
N=292
Unadjusted Adjusted
Score
score
Change
Score
Change
Score
Unadjusted
Adjusted
First
Counselling
Session
45.2
85.7
First Session of
Group
33.1
77.9
-11.78
Last Session of
Group
28.6
71.4
-4.5
-6.5
From first session
of counselling to
last session of
group
-----------
---------
-16.6
-14.3
FCSAA Power of Prevention Conference,
November 29. 2013
Results –
Results –
Unadjusted Adjusted
scores
scores
t(299) = 12.79,
p <.001*
t(291)=10.6,
p<0.001*
Combined
Programming
N=299
First Counselling
Session
45.2
First Session of
Group
33.1
-12.1
Last Session of
Group
28.6
-4.5
From first session
of counselling to
last session of
group
Change
Score
-16.6
Results (Unadjusted)
t(299) = 12.79, p <.001*
FCSAA Power of Prevention Conference,
November 29. 2013
Combined
Programming
N=292
First Counselling
Session
85.7
First Session of
Group
77.9
-7.8
Last Session of
Group
71.4
-6.5
From first session
of counselling to
last session of
group
Change
Score
-14.3
Results – adjusted for social
desirability
t(291) = 10.63 p <.001*
FCSAA Power of Prevention Conference,
November 29. 2013
Category
Unadjusted
Individual
Counselling RCM
CCC Cases
US Baseline Study
Deteriorated
43 (7%)
648 (7.6%)
8.2%
No Change
299 (50%)
4055
(47.4%)
56%
Improved
183(30%)
1933
(22.6%)
20.9%
Recovered
78(13%)
1915
(22.4%)
14.1%
Total Improved
or Recovered
261(43)
3848
(45%)
35%
637
8,551
6,072
Total
FCSAA Power of Prevention Conference,
November 29. 2013
Category
From first RCM
Counselling
session to first
group session
CCC Cases
US Baseline Study
31 (5.7%)
648 (7.6%)
8.2%
No Change
305 (56.1%)
4055 (47.4%)
56%
Improved
168 (30.9%)
1933 (22.6%)
20.9%
Recovered
40 (7.4%)
1915 (22.4%)
14.1%
Total Improved or
Recovered
208 (38.8)
3848
(45%)
35%
544
8,551
6,072
Deteriorated
Total
FCSAA Power of Prevention Conference,
November 29. 2013
Category
Unadjusted
From first Family
Violence
counselling session
to last group
session
CCC Cases
US Baseline Study
Deteriorated
11 (7.2.%)
648 (7.6%)
8.2%
No Change
62 (40.8%)
4055
(47.4%)
56%
Improved
58 (38.2%)
1933
(22.6%)
20.9%
Recovered
21 (13.8%)
1915
(22.4%)
14.1%
Improved or
Recovered
79 ( 52%)
45%
35%
152
8,551
6,072
Total
FCSAA Power of Prevention Conference,
November 29. 2013
Category
Adjusted for
social
desirability
From first Family
Violence
Counselling
session to last
group session
CCC Cases
US Baseline Study
Deteriorated
11 (7.2%)
648 (7.6%)
8.2%
No Change
62 (40.8%)
4055
(47.4%)
56%
Improved
58 (38.2%)
1933
(22.6%)
20.9%
Recovered
21 (13.8%)
1915
(22.4%)
14.1%
79(52%)
45%
35%
152
8,551
6,072
Improved or
Recovered
Total
FCSAA Power of Prevention Conference,
November 29. 2013
Combined
N=152
Programming to
(unadjusted)
counselling follow-up
First Counselling
Session
Change
Score
Results
45.3
First Session of Group 33.1
-12.2
Last Session of Group
30.1
-3
Last Counselling
Session
28.8
-16.5
FCSAA Power of Prevention Conference,
November 29. 2013
T(151)=9.4, p<0.001*
Conclusion
– Overall results support continued use of this
framework of individual and group
– Decreasing the number of clients doing individual
counselling only
– Will work with the program to support best results
for clients
FCSAA Power of Prevention Conference,
November 29. 2013
robbie.wagner@calgarycounselling.com
www.calgarycounselling.com
FCSAA Power of Prevention Conference,
November 29. 2013