Evaluation and Policy:
How Are They Connected?
Geoff Nelson, Psychology, Wilfrid Laurier University
Tim Aubry, Psychology, University of Ottawa
Aimee Watson, Mental Health Commission of Canada
Agenda
2:45-3:00 – introductions and expectations for the workshop
3:00-3:10 – policy
3:10-3:20 – evaluation
3:20-3:30 – knowledge transfer
3:30-3:50 – small group discussion
3:50-4:00 – reporting back on small groups
Policy
The renewal and repurposing of HPS
• March 21, 2013, federal budget announcement, Economic Action Plan
2013 proposes $119 million per year over five years, nearly $600
million in total, to the Homelessness Partnering Strategy using a
“Housing First” approach.
• Building on the successful At Home/Chez Soi project, the Government
will continue to work in partnership with provinces and territories,
communities, the private sector and other stakeholders, such as the
Federation of Canadian Municipalities, to reduce homelessness.
Economic Action Plan 2013 proposes $119 million per year over five
years for the Homelessness Partnering Strategy using a Housing First
approach. Over the coming months, the Minister of Human Resources
and Skills Development will work with Government partners at the
community level to confirm the implementation details of this
approach.
Policy
The renewal and repurposing of HPS
• The evidence shows that the Housing First approach: can be
implemented across Canada; improves the lives of those who are
homeless and have a mental illness; and makes better use of public
dollars, especially for those who are high service users—as
demonstrated by a reduction in the use of hospital in-patient care,
emergency rooms, police detentions and justice services.
• The project will develop a body of evidence to help provinces and
territories provide services to homeless people living with a mental
illness. In addition, funding arrangements are being established to
ensure that people participating in the project will have supports in
place once the project is completed.
Policy
How did this renewal and repurposing of HPS happen?
Evaluation and trusted messengers
• At Home/Chez Soi research and MHCC knowledge translation
• The conservative heartland – the Alberta experience and the
emergence of the Canadian Alliance to End Homelessness led by Tim
Richter
• Looking south of the border – US research on Housing First, support
from the G.W. Bush administration, development of 10-year plans to
end homelessness in many US cities
Policy
How did this renewal and repurposing of HPS happen?
Political will
•
•
•
•
Dissatisfaction with progress in reducing homelessness in Canada
Review of HPS provided an opportunity to refocus its approach
Desire to do a better job of reducing homelessness in Canada
An existing funding mechanism and program (HPS)
Program Evaluation (Rossi et al., 2004)
Definition
“The systematic collection of information about the
activities, characteristics, and outcomes of
programs”
Why Evaluate?
1. to make judgements about the program,
2. to improve program effectiveness,
3. to inform decisions about future programming”
Make-Up of a Program
PROGRAM MISSION
NEEDS
GOALS &
OBJECTIVES
VALUES &
PRINCIPLES
PROGRAM
PROCESSES
PROGRAM
OUTCOMES
INPUTS
INITIAL
OUTCOMES
ACTIVITIES
INTERMEDIATE
OUTCOMES
OUTPUTS
LONGER-TERM
OUTCOMES
RELATIONSHIP TO THE THREE MAIN TYPES OF PROGRAM EVALUATION
NEEDS
ASSESSMENT
IMPLEMENTATION /
PROCESS
EVALUATION
IMPACT / OUTCOME
EVALUATION
Housing First Program
Mission
- Reduce
Homelessness
Program Processes
HOUSING
FIRST
Outcomes
Individual:
+ Housing Stability
+ Functioning
+ Quality of life
Community:
- Chronic & episodic
homelessness
Housing First: Assessment of Fidelity
Housing First:
Assessment of Fidelity Options
1. Self-asessment of fidelity (Gilmer,
Stefanic, Sklar, & Tsemberis, 2013)
2. External assessment of fidelity
(Mental Health Commisssion of
Canada, 2014)
Outcome Monitoring
“The ongoing collection and analysis of
information on indicators of how well
programs are performing with regard to the
achievement of results” (Rossi, Freeman, &
Lipsey, 1999).
Types of Outcome Measures




Standardized self-report measures
Simple counts of client status
Collateral rating scales
Consumer satisfaction measures
Housing First: Outcome Indicators &
Potential Measures
1. Housing status of clients (Simple count)
2. Community functioning (Multnomah Community
Ability Scale – Collateral rating scale)
3. Quality of Life (Lehman’s Quality of Life Interview
– Standardized self-report measure)
4. Consumer Satisfaction Measure – (Service
Satisfaction Scale)
Community-Level Outcomes
Point-in-Time Counts
A PIT is a snapshot of sheltered and unsheltered
homeless people in a community on a single night.
Why do a count?
 Provides a baseline
 Identifies needs of the populatioin
 Enables you to measure community level
outcomes
 Helps identify priority populations
Canadian
Point-in-Time Count
Methodology and Toolkit
Piloted in Seven
Cities in Alberta
October 16, 2014
Who to Count?
OPERATIONAL CATEGORY
1
2
3
4
1
Unsheltered
Emergency
Sheltered
Provisionally
Accommodated
At-Risk of
Homelesness
LIVING SITUATION
COUNTED
o
People living in public or private spaces without
consent or contract
✔
1.2
People living in places not intended for permanent
human habitation
✔

Emergency overnight shelters for people who are
homeless
✔
2.2
Violence-Against-Women (VAW) shelters
✔
2.3
Emergency shelter for people fleeing a natural
disaster or destruction of accommodation due to
fires, floods etc.
3.1
Interim Housing for people who are homeless
3.2
People living temporarily with others, but without
guarantee of continued residency or immediate
prospects for accessing permanent housing.
3.3
People accessing short term, temporary rental
accommodations without security of tenure
3.4
People in institutional care who lack permanent
housing arrangements.
3.5
Accommodation / Reception centres for recently
arrived immigrants and refugees
4.1
People at imminent risk of homelessness
4.2
Individuals and families who are precariously housed.
✔
✔
• Screening
Mandatory
questions
• Consent
• Gender
• Age
• Ethnicity
• Migration
• Immigration
• Homelessness History
• Veteran Status
Using Program Evaluation to Develop and
Improve Housing First Programs
1.
2.
3.
4.
Conduct regular fidelity assessments
Integrate outcome monitoring into service delivery
Conduct regular point-in-time counts
Provide training of service providers on data
collection
5. Provide regular feedback on collected data
6. Keep analyses simple
7. Make program improvements based on collected
data
Knowledge Exchange
“Evidence of effectiveness alone is rarely enough to ensure adoption of
interventions” (Leff & Mulkern, 2002)
What is Knowledge Exchange?
The Canadian Institutes of Health Research defines KE as: a dynamic and
iterative process that includes synthesis, dissemination, exchange and
ethically-sound application of knowledge to improve the health of
Canadians, provide more effective health services and
products and strengthen the health care system
www.cihr‐irsc.gc.ca/e/29418.html
Knowledge Exchange
How do you do it?
Moving Knowledge to Action:
• What?
• Who?
• How?
Synthesis - 1 and 3 pagers, summaries, early findings
Dissemination
Exchange
- media, community events, social media
- Engagement, training, collaboration
Knowledge Exchange
Influencing Policy
Lessons from the At Home/ Chez Soi experience:
Throughout At Home/Chez Soi, the project team engaged in an ongoing
process to work with government. Strategies included:
• Worked with all levels of government (municipal, provincial, federal)
• Regularly held in-person meetings and briefings throughout course of
project
• Developed briefing material- PowerPoint presentations, Early Findings
reports
• Used key messages to ensure consistent and focused messaging
• Resulted in strong government engagement in At Home project sites
and ongoing policy change
Focusing the Frame
A Participant Photography Project
- 1 and 3 pagers, summaries, early findings
- media, community events, social media
My Kitchen
View
from my Window
Advisory panels, collaboration,
meetings
__________________________
________________________
I love to cook. On the street it's hard to get a
I am so grateful for this view. I
decent meal. Now my fridge is full and I can
am also grateful to not have
choose what and when I eat. I'm so much
anyone able to knock on my
healthier now, and I get so much satisfaction in
window at all hours of the night.
doing something I love to do. I can even invite
I have some sense of safe.
people over for a meal. Beats the hell out of a
food kitchen.
-
Focusing the Frame
A Participant Photography Project
-
Strings
____________________
The name “chaos theory” comes from the
fact that the systems that the theory
are apparently disordered. But
1 and 3 pagers, describes
summaries,
early findings
Chaos Theory is really about finding the
underlying order in seemingly random
data.
- media, community events, social media
-
The “butterfly effect” is a sensitive
dependence on meetings
initial conditions. Just a
Advisory panels, collaboration,
small change in the initial conditions can
drastically change the long term behaviour
of a system...
I am the system
Small Group Discussion Questions
• How can your communities incorporate evaluation into your Housing
First programs?
• What knowledge transfer strategies could you use with policy-makers
in your jurisdictions to promote the adoption of Housing First