Proposal and Program
Evaluation
Ivan Yau
Critical Reflections
 Different
Purposes / Targets
lead to different Strategies and
Demands on rigorousness of
program evaluation – to public, visual
documentation and stories can be used
effectively (have a sense that these are
representative), to academic and funder,
generally, statistical analysis is used (but should
have meaningful no.!!)
Critical Reflections
 Poor
in Program Evaluation in
general so far!! – these can be
reflected in program evaluation published
with poor objectives, evaluation methods
and its interpretation. So what are your
roles as social workers?
Critical Reflections

Evaluation on Program Effectiveness
are linked to ‘accountability’ and
‘monitoring’ both. If impact / effectiveness can
be demonstrated (accountability), our service can only be
monitored by output.

Program Evaluation is a progressive
and goal-driven process, not a
repetitive routine measure, e.g. after a
rigorous evaluation on effectiveness, only certain kind of
QC or a more in-depth evaluation is required.
Further development
Some examples
1.
Hong Kong Family Welfare Society
(1989) A research report on cognitivebehavioral Group Therapy. - further developed
into more refined program on clients with more severe
M.I. symptoms, parent group and other with more skill
deficits
2.
Chronic Disease Self Management
Program (CDSMP) – more vigorous screening is
required to ensure the commitment in the program and
reduce drop-out rate
Critical Reflections
Evidence – biased preference
1. Numbers – even converting subjective

experience into Numbers!!
2. Counting
and Rating
3. Statistical Analysis (T-test, signtest,…logic of significant level and
hypothesis testing)
Program Logic One

Linear (common, but oversimplified)
Problem
Evaluation
Program Logic Two

Holistic View: Analysis of Target and Problem, Mechanism of
Change (Intervention), Objectives, Evidence and Expected
changes
Program Logic Three

From SSLD perspective:
Program Evaluation
Objectives:
1. Change
2. Important
3. From SW concern / perspective!!
 Effective? - Evidence (changes
observed) is consistent with the
objective.

Illustration – Program Logic






跨越鴻溝 – 促進親子溝通計劃（2001）
針對大埔區青少年不斷離家出走的現象
合作伙伴：警方（轉介）
潛伏危機：受誘惑而去吸毒、販毒、賣淫
服務目標所受的影響：減少再次離家出走的情況
社會控制的議程：加派警力巡查、對離家出走的
青少年/家長提出警告、向多次離家出走青少年
的家庭提出保護令的申請、……
Illustration – Program Logic
現象分析：親子缺乏溝通是一般離家出走少年的
起源？
 理論：青少年成長理論、家庭發展歷程、……
 進一步探索：
1. 青少年離家後在甚麼地方流連？有何活動？跟甚
麼人連繫？（滿足甚麼需要？）
2. 這些青少年的家庭狀況？他們對家庭的觀感？家
庭關係？（家庭無法滿足他們甚麼需要？）

Illustration – Program Logic
一些可能的需要：
 進一步探索：
1. 社群連繫（歸屬感、關顧、認同…)
2. 刺激、歡樂、成功感、…
 滿足需要的其他途徑……?

Program Plan

What are common problems in
program / session plan?
Program Plan

1.
2.
Common Problems
Linear without contingent thinking
Not aware of mechanism of change
Program Plan
Mechanism of Change

Some Illustrations
1.
Skill training
2.
Cognitive restructuring – getting familiar
– Parenting skill training as an
example  structured exercise, brief lecture,
demonstration, role-play and feedback, experimentation
in real life
with ABC (differentiating DT /DT, aware of
undesirable impact of DT), identifying own DT,
disputing these through empirical enquiry, selfrestructuring process
Mechanism of Change
Some Illustrations
3. Self Understanding / acceptance –

selfexploration on critical incidents in trustful environment,
feedback from others after intensive interaction,
connecting with others deeply and comparing the
difference
4.
生死教育
– reflection on the edge of life / death
experience or stimulated experience
Program Plan
Program Evaluation
Some Clarifications
Intervention is different from
Objective / Effectiveness.
 Process review is useful for
improving the practice.
 Client’s satisfaction is Not the
program effectiveness.

Objective
Usually viewed as ‘Prevention of
Undesirable Consequence’ /
‘Alleviation of the (social) Problem’
 Important and Desirable – from
who’s perspective? Client’s?
 Influence from stake-holder?
Theory-driven? Social Control?

Effectiveness and Change
Extent: Magnitude in relation to all
participants (sometimes Magnitude
alone is sufficient!)
 Confidence: Change is not the result
of chance
 Causal relationship: can hardly be
established

Intervention
Intervention must link to mechanism
of changes (theory-driven and
common sense!!) and desirable
outcome.
 Group proposal – brief session
outline first, and further detailed
session plan as the group evolves
gradually.

Effectiveness I
Participants’ change (pre-post, post only)
 Issues of measurement: validity (face
validity), reliability (items-correlation) and
sensitivity
 Changes reach statistically significant level
 Single-item comparison
 Scale construction: literature review, local
validation, feedback from participants,…
Effectiveness II
Outcome Indicator
 Criteria for success: how come?
 Baseline makes comparison feasible
(indicating the change)
 Meaningful (and significant) to reflect the
accumulated effort, or indirect evidence
to achieve the ultimate goal (community
level intervention)
Different Levels – G & P

1.
2.
3.
4.
Group and Programs
Specific target group
Focused, limited intervention
Operationalization of Objectives –
pre-post measures, post-measures
Use of statistical analysis to assess
significant changes
Different Levels - Project

1.
2.
3.
Project
Specific target group with rather
long-term intervention
Focus on several key outcome
indicators
Pre-post measures are also feasible
Different Levels - Service
Service
1. Diversified target groups with longterm intervention and general /
vague objectives (IFSC, DEC,…)
2. Aggregated data is always useful
3. Focused on major groups, or use of
randomizations if necessary

Political Agenda

Making use of client’s views in
advocating for changes, e.g. IFSC,
CSSA, SW field work supervision, 自力更新服務

Connecting impact of the program
with advocacy / policy change, e.g. 類
風濕性關節炎自我管理課程、中學義工服務、小學學校
社工服務、兒童之家服務、…
Rounding Up

Aggregate data (client’s satisfaction, objective reached,
target complaints, etc) is useful for service
evaluation, key outcome indicator (decrease
in A & E of child injury due to child neglect, decrease in
suicide) for project evaluation, while
accumulated impact
(make use of community
resources by NA, support network, no. of high risk case
receiving service) for both.
Rounding Up

Evaluation should be better linked to
continuous improvement (don’t repeat the
evaluation every year), service development
(promotion of a better service model) or social
movement (advocacy) (e.g. changing concepts of
chronically ill patient – ownership of health, instead of
replying on medical treatment).
Rounding Up
Evaluation can be and should be
Political – particularly help marginalized
group to voice out their views / needs.
 Social workers should:
1. Be able to link up the objective,
intervention and outcome.
2. Be quipped with themselves of different
ways / strategies for program evaluation.

Reference




陳錦棠（2006） 社會服務成效評估：程序邏輯模式之應
用. 香港：香港基督教女青年會
羅致光等編（1999）靑少年成長與活動案例. 香港：香港
基督敎服務處。
羅致光主編（1992）靑少年中心活動成效評估 : 方法與案
例. 香港：香港基督敎服務處
United Way of America (2006) Measuring Program
Outcomes: A Practical Approach