o
ag
MONITORING AND EVALUATION OF A
MULTI-AGENCY RESPONSE TO
HOMELESSNESS
Paper to the Ireland Evaluation Network
23 May 2008
Peter Wilkins
The Office of the Auditor General for Western Australia
o
ag
Evaluating the complex
• A research theme of the International
Evaluation Research Group
www.inteval-group.org/
• A forthcoming book ‘Evaluating the
Complex: Attribution, Contribution and
Beyond’ Edited by Robert Schwartz, Kim
Forss and Mita Marra
• A new interest in effectiveness evaluations of
overarching policy initiatives
The Office of the Auditor General for Western Australia
o
ag
Evaluating the complex (2)
• Complex policy initiatives mostly fail standard
evaluability assessment.
• Objectives problematic, little evidence of causal
linkage
• Non-linear links of interventions, contextual
variables and outcomes
• Attribution a major challenge, even when macrolevel outcomes are measurable
• Editors have identified 5 approaches that might be
helpful: cluster evaluation; multi-level evaluation;
quantified logic models; comparative community
studies; thematic evaluations
The Office of the Auditor General for Western Australia
o
Origins
of
Homelessness
Initiatives
ag
• Task Force created in July 2001 to develop a plan of action
• The Chairperson of the Taskforce was from the not-forprofit sector
• Also to provide advice on the development of benchmarks
and social indicators relating to homelessness and
recommend a plan to monitor the progress of the
implementation strategy”.
• Taskforce reported in January 2002 and made 68
recommendations relating primarily to nine State agencies.
• It recommended that implementation include tracking the
extent to which the Actions have been implemented. It also
called for Independent Review “The non-government sector is keen
to see an independent review undertaken after a two-year period to assess the
outcomes for people who are homeless. This is viewed as part of open
government and the commitment to work in partnership with the community.”
The Office of the Auditor General for Western Australia
o
Origins
of
Homelessness
Initiatives
(2)
ag
• State government response in May 2002, accepting
majority of recommendations.
• Committed to a further $32 million over the four years
2001-02 to 05-06 for a set of initiatives
• Report recognised the complex environment and
circumstances surrounding homelessness. For instance, it noted
that people who are homeless have typically experienced difficulties in their
lives and that often homelessness arises because of these difficulties including
family conflict, mental illness, interruptions to stability through imprisonment,
long term care or institutional living, family violence, and problematic alcohol
and other drug use.
• Three themes of the Government's response to the
Taskforce's recommendations: Better options for housing;
Vulnerability and transition and Stability in housing.
The Office of the Auditor General for Western Australia
o
ag
Monitoring Committee
• Established a monitoring committee to oversee and report on the
implementation of the State Homelessness Strategy, and to
– Develop performance indicators to measure Government
performance in responding to the Taskforce recommendations.
– Prepare an annual report to Cabinet Standing Committee on
Social Policy, against these performance indicators, with the first
report due in May 2003.
– Prepare an evaluation report on the implementation of the
recommendations for submission to Cabinet in December 2003.
• Monitoring Committee membership included representatives of five
key departments, the Chairperson of the Taskforce, from ATSIC and
the Office of the Auditor General [1]. It met regularly and its major
outputs were 17 reports: 12 quarterly progress, 3 performance
indicators & 2 evaluations.
[1] For much of its life the OAG member who participated as an observer was
the author of this paper.
The Office of the Auditor General for Western Australia
o
ag
First Evaluation
• Focused “primarily on formative issues, ie improving processes rather
than evaluating success”
• Sought to address 6 key questions: Implementation; Timeliness; Shortfalls; Impacts; Program logic; and Performance indicators, and found:
– Initiatives requiring the development of strategies or the forging of
agreements had generally made satisfactory progress but that progress on
initiatives involving legislation, policies or systems has been a little more
mixed.
– Nearly all the performance indicators developed by the Monitoring
Committee were useful
– Areas for further consideration included: no mechanism for showing what
other initiatives might be beneficial; developmental performance
indicators; and no single measure of the level of homelessness
– draft Program Logic Model (Figure 1) & PIs associated with a Hierarchy
of Outcomes (Figure 2)
• Four main recommendations: continue to ensure that agency and
community support & engagement continue; cross-government
strategies to further breakdown the "silo" effect; review of two of the
PIs; and a further review of the Strategy
The Office of the Auditor General for Western Australia
o
Program Logic for Government
ag Response to Homelessness
Source: Cummings, R and Moir, S (2003).
The Office of the Auditor General for Western Australia
o
ag
The Hierarchy of Indicators
Source: WA State Homelessness Strategy - Report Against Performance Indicators - As at 30 June 2005.
The Office of the Auditor General for Western Australia
o
ag
Second Evaluation
• Sought to assess impact & effectiveness of the initiatives in
the strategy and addressed: the extent to which commitments had
been implemented; effects of the initiatives on other major service deliverers;
outcomes for clients and other affected groups; whether funded initiatives
should be continued, ceased or changed & the service system’s response had
improved; effectiveness of the strategy in addressing the priority outcomes;
how well coordinated across Government.
• Report identified that the evaluation was atypical in that it
is reviewing outcomes of the individual initiative and
strategic levels.
• Noted components of a monitoring and an impact
evaluation:Monitoring Evaluation is carried out to ensure that a
project remains on track and to guide continuous improvement.
– Impact Evaluation is used to assess the impact of a settled project
• Main findings were generally positive: majority of
initiatives successfully implemented or underway, good
outcomes for clients.
The Office of the Auditor General for Western Australia
o
ag
Second Evaluation (2)
Extracts of the 2006 Evaluation Report: Major Findings against some Key
Evaluation Questions
• To what extent were the commitments made in the Government
Response to the State Homelessness Taskforce (Putting People First)
implemented?
Work on thirty of the initiatives completed, progress cf 25 initiatives completed in 2003. Overall, 58
of the 67 are reported to be progressing or completed satisfactorily. The results of the evaluation
reflect a general satisfaction with the Strategy as a whole amongst stakeholders.
• How effective has the Strategy been in addressing the Priority
Outcomes?
A program logic analysis confirmed that the initiatives are consistent with the priority outcomes. The
data collected in the study, as well as client feedback and stakeholder views reported here also
support this conclusion
• How has the Strategy been Implemented in the context of the
Government’s Social Policy Agenda and how well has it been
coordinated across Government?
Widespread agreement among stakeholders that Strategy fits within the Government’s social policy
agenda. Some success, still a long way to go. Still clients who are missing out on services, waiting too
long for services & being passed from department to department.
The Office of the Auditor General for Western Australia
o
ag
Second Evaluation (3)
CONCLUSION
• The evidence suggests beginning to break the cycle of homelessness
although not possible at present to assess their cost effectiveness.
Where current funding or policies do not provide for effective
servicing, a cycle of homelessness remains as clients reappear in the
service system.
• Overall, the Strategy is beginning to show some positive measurable
outcomes in the headline indicators….The improvements so far,
however, are modest. This is to be expected because of the magnitude
of the issue being addressed. Perhaps biggest weakness is improving
accommodation services for individuals suffering from mental
illness…..While there has been a small improvement in indicators such
as Homeswest waiting lists per head of population, the absolute level of
homelessness is still high and more needs to be done if it is to be
substantially improved….
• The evaluation indicated that while much was achieved, much
remained to be done.
The Office of the Auditor General for Western Australia
o
ag
Second Evaluation (4)
• As the Strategy was coming towards an end, it also became
apparent that there were significant pressures on the
availability of affordable housing including public housing
stock linked to a natural resources driven economic boom
in the State. The Government has acknowledged these
pressures and indicated that it is seeking to address them
• As the website for the 2006 evaluation notes: “In 2006, the
booming economy is placing more people in work…. Costs
are increasing to buy and build properties and rents are
rising. People with minimal resources, both financial and
personal are being squeezed out at the bottom end of the
housing market. This places an added strain on the safety
net services”
The Office of the Auditor General for Western Australia
o
Evaluation Linked to Monitoring
ag
• Distinctive feature of the implementation process has been the
intertwining of monitoring & evaluation and a synergistic relationship
between the two
• 2006 evaluation specifically addresses effectiveness by reviewing
performance indicator information collected and reported as the
"Hierarchy of Indicators"
• The relative sophistication of the approach can be identified from the
findings of a recent report of the Auditor-General of New South Wales
(2007) who found that “… we were unable to determine how well the
government is responding to homelessness statewide. This is because
there are no statewide performance measures or targets on
homelessness. Also there is limited benchmarking, and no formal
means of spreading information on homelessness initiatives and
projects.”
• The approach entails risks of over-simplifying a complex situation &
creating a view that attribution has been resolved. Other analytical
frameworks (Figure 3).
The Office of the Auditor General for Western Australia
Housing Analytical Framework for Planning
o
ag
Source: WA Department of Housing and Works (2005).
The Office of the Auditor General for Western Australia
o
Evaluation
and
Attribution
(2)
ag
• The 2nd report observes that the evaluation is more challenging than
that of a single program as many factors are at play in the same arena
as the strategy
–
“Political, cultural, social and economic trends act on service delivery agencies at a
macro level. In addition, there are a multitude of different projects and programs
at work, a multitude of organisations at work independently of any particular
initiative. Should changes in social outcomes occur, the challenge is to be able to
distinguish between at least three possibilities: the change is occurring because of
the Strategy; the change is occurring for another reason apart from the Strategy; or
the changes occurring partly due to or inspite of the Strategy.”
– It further notes that “while positive change in social indicators may indicate a
successful strategy, it is generally impossible to map such a one-to-one relationship
between activity and result. The use of professional judgment of stakeholders can
be useful to help interpret data.”
• It makes a contribution to understanding causality without
establishing it. Based on all the evidence it was reasonable to conclude
that the program was an important contributing factor to the changes
observed.
• The report noted that the evaluation “has attempted to identify
measurable impact where possible, and where this is not possible, it
has used the program logic analysis and Hierarchy of Indicators to
demonstrate changes in headline indicators which in turn that are
expected to impact on the priority outcome”.
The Office of the Auditor General for Western Australia
o
ag
Evaluation and Attribution
• Attribution in simple terms is about causality, about answering the
question: Are observed results the consequence of the intervention
rather than other programs or external factors?
• Mayne 2001 talks of "the problem of attribution" and notes that more
difficult than measuring outcomes is determining just what
contribution a specific program made to the outcome, and how much
of the success (or failure) can be attributed to the program.
• Pinpointing causality between individual interventions and outcomes is
in most cases problematic, and doing so for a multiplicity of
interventions is nigh on impossible.
• The State Homelessness Strategy is complex as a result of multiple
interventions operating at different levels in different ways, and having
multiple stakeholders
• The second evaluation sought to address “whether funded initiatives
should be continued, ceased or changed” which implies a need for
some understanding of causality
The Office of the Auditor General for Western Australia
o
Effectiveness and Cost-Effectiveness
ag
• Research group is looking at “The Effectiveness and CostEffectiveness of Homelessness Prevention and Assistance Programs”
using the Western Australian experience as its area of study.
– Effectiveness the extent to which the programs improved the outcomes of those they
assist; cost-effectiveness looks at the outcomes delivered for the net dollar spent
including ‘net return’ compared with alternative uses of the same resources.
– Primary field work a longitudinal client-based survey and a one-off survey of
agencies primarily looking at costs. Costs determined using one of two scenarios:
clients of the homelessness assistance service compared with the population
average; and clients of each service in target categories compared with clients who
did not fall into one of these categories
• Preliminary findings: the programs produced positive outcomes for
clients in the period immediately following the provision of support
and did so at low cost of delivery relative to the delivery of other
services
• "…. if homelessness programs were able to reduce the utilisation of
health and justice facilities by clients of homelessness programs down
to population rates of utilisation the savings achieved would pay for the
homelessness programs several times over. This suggests that there is
potential for homelessness programs to be dramatically cost-effective."
The Office of the Auditor General for Western Australia
o
agEffectiveness & Cost-Effectiveness (2)
• Outcomes assessed across a range of dimensions: changes in
clients status measures such as labour force, level of income etc;
changes in the utilisation of homelessness programs services and nonhomelessness programs services; changes in self-assessed satisfaction
and Program-specific client outcome indicators
– Programs supporting those leaving prison proving beneficial in lowering
rates of recidivism and making positive contribution to the lives of
individuals who have previously returned to prison
– On average, the annual cost of health services for a community centre
client is $10,217 per person per year greater than the population average,
and for Justice services $3810 per person per year greater than the
population average
• Value of annual "population offsets" is at least 2.7 times greater than
the annual program costs, a significant potential for cost savings from
providing assistance.
• The preliminary report notes that the Final Report "will include an
examination of client outcomes 12 months down the track when most
clients will have left support."
Flateau, P et al (2007). “The Effectiveness and Cost-Effectiveness of Homelessness
Prevention and Assistance Programs: A Report of Preliminary Findings” Australian
Social Policy Conference, Sydney. www.sprc1.sprc.unsw.edu.au/ASPC21007/
The Office of the Auditor General for Western Australia
o
ag
A Client Based Model
Source: Flateau, P et al (2006). The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance
Programs Positioning Paper, Melbourne: Australian Housing and Urban Research Institute (AHURI). www.ahuri.edu.au
The Office of the Auditor General for Western Australia
o
ag
A Non-Client Model
Source: Flateau, P et al (2006). The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance
Programs Positioning Paper, Melbourne: Australian Housing and Urban Research Institute (AHURI). www.ahuri.edu.au
The Office of the Auditor General for Western Australia
o
ag
Concluding Comments
• Intertwining of monitoring & evaluation
• The second evaluation enabled reflection by the
evaluator and stakeholders on the question of
causality without making purity of approach a
barrier to forward movement of the Strategy
• Ongoing research has the potential to build
understanding of cost effectiveness
• See paper at www.aes2007.com.au/papers.php
The Office of the Auditor General for Western Australia
o
ag
Contact details
Dr Peter Wilkins
Assistant Auditor General
Office of the Auditor General
4th floor Dumas House
2 Havelock St
WEST PERTH
Western Australia 6005
Phone: +61 8 9222 7588
Mobile: +61 407 380 772
Fax:
+61 8 9322 5664
Email: peter.wilkins@audit.wa.gov.au
The Office of the Auditor General for Western Australia