January, 2014
Evidence-Based Research in Canada
 National research agenda to better
understand “what works” in career services
 3 calls for proposals from HRSDC over 6 years
 Source of major international envy among
Career Development Practitioners (CDPs) and
Researchers
2
Why Evidence-Based Practice?
 Ethics: Clients deserve interventions that have
demonstrated success
 Confidence: Practitioners should know that a
given intervention will result in a predictable
outcome
 Security: Being able to provide evidence of
success increases ability to obtain and keep
funding for services and secure jobs for
CDPs/Employment Counsellors
3
Why Evidence-Based Practice?
 Outcome-focused intervention
 Evidence of client change
…contrast with…
 Client flow
 Ease of access
 Practitioner time use
It’s easy to measure variables that don’t address
client change
4
5
Session Objectives
 Highlight results of 3 recent research projects: What
have we learned?
 Explore how a select number of LMI and career
development resources could be integrated into
everyday evidence-based practice
 Explain where we are now and next steps
 Direct you to additional research studies
6
Before we start…….
 What evidence do you currently collect?
 What is done with the evidence
currently collected?
7
Reflective Questions….
 What did the research do?
 What was learned?
 What was most surprising?
 What are the implications for practice?
8
RESEARCH PROJECT #1:
SSESSING THE IMPACT OF LMI ON CAREER DECISION MAKING
AND WORK SEARCH
Research Team
Canadian Research Working Group in EvidenceBased Practice (CRWG),
Canadian Career Development Foundation (CCDF)
In partnership with
New Brunswick Post-Secondary Education & Labour,
Saskatchewan Advanced Education, Employment & Labour
9
WHAT IS THE PROBLEM?
 There is very little literature about the direct
contribution of LMI to employment/career
decision-making (Savard & Michaud, The Impact of LMI
on Career Decision-Making Process: Literature Review, FLMM,
2005)
 Several questions remain unanswered:
 How do people actually use LMI?
 What (if any ) assistance would be helpful?
10
RESEARCH QUESTIONS
 If client needs are assessed and clients are given LMI
consistent with their needs,
To what extent does assistance by a service
provider enhance their effective use of LMI?
OR
To what extent is independent self-help a
sufficient process for clients to use LMI
effectively?
11
METHOD
 We prepared “guided” LMI packages (take-
home booklets and Resource Centre
binders) on:
 Career Decision Making:
 Know yourself
 Know the Labour Market
 Put it all Together
 Job Search:




Check for “Fit”
Get Ready
Search for Work
Get a Job
12
INTERVENTION
All participants in the study:

Received a needs assessment interview & completed an initial, pre-program
survey

Received an LMI package specific to their identified employability need

Were randomly assigned to either a self- directed intervention delivery
method or an assisted self-directed intervention delivery method.

Were given an orientation to the Resource Centre which they could freely
use on their own

Completed a pre and post-pre survey

Received a cash honorarium and certificate of participation
13
INTERVENTION (CONT)
The self-directed group:
worked independently for 3 weeks, making
use of the materials and the Resource
Centre
returned in Week 4 for their exit interview
14
INTERVENTION (CONT)
The assisted self-help clients received:
 two additional AIS (Advice and Information)
interviews (20-30 minutes) in weeks 1 and 3
focused on helping them understand,
interpret and apply the LMI to their own
situations and /or access additional LMI
returned in Week 4 for their exit interview
15
SAMPLE BY PROVINCE
Province
Saskatchewan
New Brunswick
English
New Brunswick
French
Intervention Type
Delivery
Total
Independent
Assisted
CDM
20
15
35
JS
23
25
48
Province Total
43
40
83
CDM
28
15
43
JS
12
13
25
Province total
40
28
68
CDM
15
17
32
JS
11
9
20
Province total
26
26
52
109
94
203
Total for 3 groups
16
WHAT DID WE MEASURE
The Dependent Measures for the data analysis were:
 General ability to use LMI
 Knowledge
 Clear vision of what I want in my career future
 Knowledge of print and online resources
 Skill
 Have effective strategies for keeping myself motivated
 Have a realistic action plan
 Personal Attributes
 Optimism about what lies ahead re meeting my career
goals
 Confidence in my ability to manage future career
transitions
17
EXAMPLES OF QUESTIONS USED IN PRE AND IN POSTPRE SURVEYS
Knowing what you know now,
how would you rate yourself before the workshop, and how would
you rate yourself now?
Before
After
Av
e
Ave
1. A clear understanding of what I need to do to move forward in
my career.
21 57 48 16 9 1.57 1 3 28 66 53 3.11
2. A clear vision of what I want in my career future.
24 37 40 39 11 1.84 1 7 29 63 51 3.03
3. Reviewed my past work, education and experience so that I
know what skills and strengths I have.
11 33 57 35 14 2.05 2 3 21 68 56 3.15
4. A list of possible options that may fit with what I want in my
career future.
23 49 47 23 9 1.64 1 4 25 66 54 3.12
5. Knowledge of print and online resources that help me to
research career/employment options.
32 55 42 16 5 1.38 1 1 21 53 74 3.32
6. Confidence that career-related employment opportunities
actually exist that fit with what I want in my career future.
19 41 58 25 8 1.75 1 10 27 58 55 3.03
18
DESCRIPTIVE RESULTS
 Changes in competency self assessments:
Pre program:
 30% and 58% of the respondents rated competency Not OK
 5% rated their competency as Exceptional
Post program:
 1%– 9% rated competency Not OK
 39% rated their competency as Exceptional
The amount of change was similar
across all three dimensions of the survey, knowledge, skills,
and personal attributes
all demonstrated about
the same amount of change.
19
DESCRIPTIVE RESULTS (CONT.)
 Of particular note are items that suggest increased ability to self-
manage their careers, such as:
 A clear understanding of what I need to do to move forward
in my career.
 A clear vision of what I want in my career future.
 Effective strategies for keeping myself motivated to achieve
my career/employment goals.
 Confidence in my ability to research career, employment, and
training options that are available
 Confidence in my ability to manage future career transitions.
20
DIFFERENTIAL RESULTS-TOTAL SCORE
For group as a whole:
• significant increase in
overall ability to use
LMI
• neither intervention
was more conducive to
one manner of delivery
compared to the other
 Both CDM and JS groups had significant increases across time
 Change in CDM group was significantly larger than in JS group
 Participants in the JS group had higher scores than participants in the CDM group,
likely indicating that JS participants were more familiar with using LMI before the
project began.
 Participants receiving assistance demonstrated greater change across time than did
those in the independent mode
Similar pattern for all subscales
21
ATTRIBUTION FOR CHANGE
To what extent would you say that any changes in the ratings on the previous pages
are a result of your participation in this research project, and to what extent were
they a function of other factors in your life?
Program
English
French
mostly
other
factors
somewhat
other
factors
3
0
3
0
uncertain
somewhat
this program
mostly
this
program
15
1
47
25
80
26
22
STUDY REPEATED IN
FRANCOPHONE NEW BRUNSWICK
Overall,no significant differences in the
results of the anglophone and francophone
samples with the exception that assisted
francophone clients reported increased gains
over the anglophone clients although both
groups reported more positive change in the
assisted condition.
23
FOLLOW UP INTERVIEWS (1 WEEK AND 4
MONTHS AFTER INTERVENTION)

“I was surprised at how much information there was”

“Initially I thought I could not do this, but I found out I could”

“There’s a lot to learn”

“I got more focused on my goal”

“I got clearer about what I wanted to do”
24
CLIENT CRITICISMS
 The program needed to be longer given the number of
resources available. (222)
 Working full time made it very hard […]. With a little
more help it would have been much easier (242)
 There was too much information and that made
making a career decision more difficult […] very hard
to do this alone (209)
25
WHAT DID PRACTITIONERS SAY?
 “The structure helped me be more focused and concrete”
 “I never really thought about intake of clients as assisted or self-
help—now it’s built into every intake I do”
 “I used to be maternal and forget that it was the client’s
decision and I got more involved than I needed to. Now I ask
the clients to be more responsible”
 “Now I give more homework tasks and I’m more specific. The
clients felt more focused and so do I”
26
WHAT DID THE RESEARCH DO?
Isolated LMI as an intervention
Isolated giving information and advice from
employment counselling
Organized LMI into a career development
learning sequence
Tested self-help versus a combination of selfhelp, information giving and advice
27
WHAT WAS LEARNED?
 Tailored LMI embedded in a learning process results in knowledge
and skill acquisition as well as the capacity for self-management
 LMI appropriate for a client’s specific need (opposed to general
LMI) appears to support engagement and action
 For many clients, a little (or no) professional LMI support is
enough
 Structure and timelines appear to motivate action and a sense of
progress
 Giving clients hands-on tools appears to motivate more than
money
28
WHAT WAS MOST SURPRISING
The extent of positive client change in knowledge,
skill and personal attributes in both conditions of
self-help and supported self-help
The value clients placed on the LMI materials and
the degree to which they used them
The positive impact of “organized” LMI and LMI
“specific to client need” versus generalized LMI
29
WHAT ARE THE IMPLICATIONS FOR
PRACTICE?
To be discussed……..
30
Research Project #2:
Assessing the Impact of Career Resource and
Supports Across the Employability Dimensions
 Drs. Dave Redekopp and Bryan Hiebert
 Alberta Human Services
 Manitoba Entrepreneurship, Training
and Trade
 Canadian Career Development
Foundation (CCDF)
31
What is the Problem?
 There is very limited research about the direct
contribution of career development learning resources
on client progress toward or achievement of
employability.
 There are many “assumptions” made about self-help
resources but very little actual evidence of their impact.
 There is much “talk” about labour market attachment
but on investigation it remains a very “wooly” construct.
32
This Study…
Developed comprehensive career development coaching
guides (way beyond LMI) covering all employability
dimensions excepting career growth:
 CAREER DECISION MAKING: Not yet clear about work/learning goal;
needs to understand personal skills, interest, values and personality and
more about different potential occupations in order to develop a vision
for a preferred future
 SKILLS ENHANCEMENT: Has career goal, but needs to explore skills and
qualifications needed for the work they want and how to acquire them
 JOB SEARCH: Has career goal and necessary skills/qualifications for
desired work; needs skills, strategies and tools necessary to conduct a
successful job search
 JOB MAINTENANCE: Has difficulty keeping work; may need to address
personal/life issues (e.g. health, behaviour, transportation, child care) or
may need to work on the skills/attitudes required for success at work.
The last part of this Guide is appropriate for someone already working,
but wanting to improve their chances of keeping the job.
33
This Study…
 Randomly assigned clients to one of two treatment
conditions:
 Practitioner launched (in which they had their needs assessed,
were oriented to the appropriate Guide and then worked
independently with their Resource Guide for 4 weeks)
 Practitioner launched and supported (in which they also
worked with their Career Consultant for 4 weeks of “regular”
services (i.e. 2-4 sessions on average) using the Guide as
appropriate but not restricted in their interventions
 Examined differential impact across treatment conditions
on client knowledge, skills, personal attributes and
labour market outcomes
34
Developed Two New Indices…
Self Help Index(self-help-o-meter) to see if:
 The propensity for self-help can be measured
 The ability to self-help affects client outcomes
Labour Market Attachment Index to see if:
 Specific variables related to the client’s work history
and life situation can be measured
 These variables affect client outcomes
Research Question
If clients are given a comprehensive needs assessment to determine
their employability need(s), what is the differential effect of
independent and consultant-supported career development
resources on clients who are weakly attached to the labour market
versus those who are strongly attached to the labour market?
 Will all clients demonstrate positive change?
 Will clients who are strongly attached to the LM show more
positive change?
 Will clients who are supported by a consultant show more
positive change?
 Will weakly attached clients struggle more in the independent
condition?
36
Sample Composition
Province
Alberta = 115
Dimension
CDM = 35
SE = 25
JS = 42
JM = 13
Manitoba = 113
CDM = 44
SE = 19
JS = 43
JM = 7
Totals
Total sample = 228
CDM = 79 (35%)
SE = 44 (19%)
JS = 85 (37%)
JM = 20 (9%)
Delivery
Independent = 17
Supported = 18
Independent = 16
Supported = 9
Independent = 21
Supported = 21
Independent = 10
Supported = 3
Independent = 21
Supported = 23
Independent = 10
Supported = 9
Independent = 23
Supported = 20
Independent = 4
Supported = 3
Independent = 122 (54%)
Supported = 106 (46%)
37
Key Findings
Career Decision Making Total Score
Delivery
PLR
PLSR
Total
n
34
38
72
Before
After
23.71(11.38) 45.50(10.93)
24.92(11.41) 49.05(8.86)
24.35(11.98) 47.38(9.98)
Summary: Main Effect for Delivery: F(1,70)=1.32; p=.26
Main Effect for Time: F(1,70)=253.72; p<.01
Delivery X Time Interaction: F(1,70)=.66; p=.42
Total
34.60
36.99
Key Findings
Skill Enhancement Total Score
Delivery
PLR
PLSR
Total
n
21
16
37
Before
After
16.38(21.50) 93.95(8.05)
50.19(18.75) 88.88(17.61)
56.54(20.85) 91.76(13.10)
Summary: Main Effect for Delivery: F(1,35)=.48; p=.04
Main Effect for Time: F(1,35)=69.33; p<.01
Delivery X Time Interaction: F(1,35)=.51; p=.48
Total
77.67
69.53
Key Findings
Job Search Total Score
Delivery
PLR
PLSR
Total
n
41
33
74
Before
After
60.51(23.76) 97.46(13.52)
53.79(23.24) 97.97(12.69)
57.51(23.61) 9769(13.07)
Summary: Main Effect for Delivery: F(1,72)=.71; p=.40
Main Effect for Time: F(1,72)=260.49; p<.01
Delivery X Time Interaction: F(1,72)=2.07; p=.16
Total
78.99
75.88
Key Findings
Job Maintenance Total Score
Delivery
PLR
PLSR
Total
n
12
5
17
Before
After
55.42(12.30) 73.50(9.70)
56.20(18.38) 77.80(14.96)
55.65(13.74) 74.94(11.14)
Summary: Main Effect for Delivery: F(1,15)=.18; p=.68
Main Effect for Time: F(1,15)=28.03; p<.01
Delivery X Time Interaction: F(1,15)=.19; p=.67
Total
64.58
67.00
Attribution of Change
 91% of clients in all interventions attributed their
positive change partly (40%) or mostly (51%) to their
participation in the study
 92% had a clear plan for next steps after the 4 week
intervention period
 98% planned to use their Guide again and/or shared it
with friends and family
42
In the Words of Clients…
 The Guide was hugely beneficial. It was a profound and amazing experience.
 My current job is not a good fit for me, but the job I have just been offered (after






working with the CDM guide) is the PERFECT fit for me.
The guide really clarified for me what I want to do in my work – what is important
to me. It helped me to realize I have the qualifications and [know]what I love to do.
It re-connected me with myself.
The guide was a catalyst for action. I knew in the back of my mind for a long time
that I wasn’t happy in my work, but it was the guide that made me do something
about it.
I think the guide is pretty universal for Canada. Combined with the skills I got from
my employment counselor, I feel more confident and feeling like I’m going in the
right direction. It was only 4 weeks but sometimes that’s all you need!
Definitely not feeling the way I did before the guide.
This experience was phenomenal. I’m so glad I got to be part of it! I’m grinning ear
to ear!!
I think high school students should get this.
43
Results
 Statistically and clinically significant positive
impact of career interventions across ALL subscores and across ALL employability dimensions
– compelling evidence of positive changes in
clients as a result of career development
coaching interventions
44
Results (cont’d)
 More positive trends for supported group but
differences are not statistically significant
(surprising and troubling result until overly caring
practitioners admitted to “creaming” the sample with
the best of intentions and researchers realized too late
that practitioners really do care!!)
45
What Did the Research Do?
 Developed and tested impacts of career development
coaching resources on changes in specific employability
dimensions
 Simulated regular employment services to the extent
possible
 Attempted to “unpack” two poorly understood
constructs—capacity to self-help and labour market
attachment
46
What We Learned
 Many clients can benefit significantly from selfhelp resources when they are matched to their
need and they are “launched” – more so than
we anticipated!
 This is especially noteworthy given the short
intervention period of 4 weeks and the “reallife” setting in which the changes occurred.
47
What We Learned (cont’d)
 Self-help coaching guides provided after a thorough needs assessment
and orientation are effective and can be used as a first line of
intervention, saving valuable practitioner time for clients who really
need it, or potentially opening a promising perspective on online career
services.
 More research is needed to test the “Self-Help Index” and a “Labour
Market Attachment Index”. These tools could not be effectively tested
due to the unanticipated “creaming” of clients who, in the view of the
practitioners, might have had difficulty with self-help materials (if
assigned to the independent group) and/or might fall at the low
extreme of labour market attachment.
48
What Was Most Surprising?
 The statistically and clinically significant amount of
change experienced by clients – especially given the
short intervention period
 The number of clients interviewed after the study who
credited the guides for significant positive life changes
 The extent to which practitioners remained committed
to the research despite significant additional demands
on their time
49
What Are the Implications for
Practice?
 To be discussed……..
50
Research Project #3:
Common Indicators
Transforming the Culture of Evaluation in Career
and Employment Services
Saskatchewan Abilities Council
Saskatchewan Department of the Economy
New Brunswick Post-Secondary Training, Education and Labour
Goss-Gilroy Inc.
Canadian Research Working Group on Evidence Based Practice in
Career Development (CRWG)
Life Role Development Group (LRDG)
CCDF
51
What Have We Learned So Far About
Client Change?
 What “inputs”/ “precursors” impact/influence client
change?
 What “processes”/ “interventions” impact/influence
client change?
 What “outcomes” result from the interactions
between “inputs” and “processes”?
52
Research Progress to Date
Input  Process  Outcome
Resources available
• Self-help Index
• Labour Market
Attachment Index
•
•
•
•
LMI Booklets
AIS interviews
Employability
Dimensions
Resource
Packages
Practitioner
coaching using
Resource
Packages
Indicators of client change
1. Learning outcomes
• Ability to use LMI
• Ability to use career
coaching resources
• Employment/career vision
• Strategies to stay
motivated
• Skills across all
employability dimensions
2.
Personal attribute outcomes
• Changes in optimism and
confidence
3.
Impact outcomes
• Impact of #1 & #2 on
client’s life, e.g.,
employment status,
employment retention,
training status, fit with goal
and vision
53
What is the Problem?
 We have data/evidence of positive impacts from
specific interventions for certain kinds of clients
 We do not have data/evidence of positive impacts
from employment services overall
 We do not have a data gathering tool that can be
used to gather common data across divergent
employment service settings
54
Therefore the Next Step:
Common Indicators Project
 Key research questions:
 What common indicators are applicable across
different contexts, different client groups, different
agencies and different interventions?
 What statements of service effectiveness can be
made by tracking common indicators?
and
55
IF Numbers Permit…
the Ultimate Question:
 What kinds of interventions in
what contexts produce what kinds
of results?
56
Participants
 New Brunswick Post-Secondary Training, Education and
Labour (30 practitioners; 137 clients)
 Saskatchewan Abilities Council
(18 practitioners; 154 clients)
New Brunswick
Campbellton/Bathurst
Dieppe/Moncton
Fredericton
Miramichi
Péninsule acadienne
Edmunston
Southwest
Southeast
Saskatchewan
Moose Jaw
Regina
Saskatoon
Swift Current
Yorkton
57
What Common Indicators are Applicable?
How we Investigated
 Literature Review
 Focus Groups
 Compare the two
 Accept those in common and most robust
58
Focus Group Questions:
Indicators of Client Change: (Outcomes)
 If clients have benefitted from career and employment
services, they have experienced some changes in in their lives.
They are not in the same position they were when they started.
Some changes may be obvious—they have a job which they did
not have before; they have enrolled in a training program
which they had not before for example. What else indicates
that clients or the situations of clients have changed as a result
of career and employment services?
 What clues do you look for to tell you that clients and/or client
situations are changing?
59
Contributors to Client Change:
(Process)
 What do you do or what do you provide
that you think contributes to these
changes in the lives of clients?
60
Influences on Staff, Clients/External:
(Inputs)
 What do you think influences your ability to help
clients and/or their situations to change?
 What do you think influences the ability of clients
to change themselves or their situations?
 What external factors (not in the control of clients
or staff) influence the changes achieved through
career and employment services?
61
PRIME Structure
(Performance Recording Instrument for Meaningful Evaluation)
Input  Process  Outcome
Resources available
•
•
•
•
•
Practitioners:
Background;
Experience
Availability of and
access to
Community
resources
Availability of and
access to
Employment
Opportunities and
Training
Client Employment
History and
Potential
Client Life
Circumstances
Working alliance, client
engagement
Specific interventions in:
Job Readiness
Career Decision Making
Skill Enhancement
Work Search
Work Maintenance
Life Circumstances
Goal Setting, Action
Planning, Action Steps
Support provided in personal
attributes:
self-esteem, self
efficacy, well-being, self
confidence, selfawareness
Progress ratings (+ or -) in all
of the above
Indicators of client change
1. Learning outcomes
• Changes in
Knowledge and
Skills
• Changes in
Personal attributes
2.
Impact outcomes
• Impact of #1 & #2 on
client’s life, e.g.,
employment status,
training status,
quality, fit, standard
of living
3. Working Alliance and
Client engagement
4. Actions underway to move
forward: Work in
Progress
62
Practitioner/Research Partner Tasks
 Recruit clients into the study (NO criteria);
 Follow all research protocols
 Use PRIME each time you see a client to assess
and record any changes (+ or -) in:





Goals and Action Plans
Employability Learning Needs/Changes
Personal Attribute Needs/Changes
Life Circumstance Needs/Changes
Working Alliance and Client Engagement
63
Practitioner/Research Partner Tasks
 Work with each client for 6 weeks following
Assessment interview and client entry into the
research
 Conduct an Exit from Research interview and use
the system to record/assess:





Goals and Action Plans
Employability Learning Needs/Changes
Personal Attribute Needs/Changes
Life Circumstance Needs/Changes
Working Alliance and Client Engagement….AND
64
Practitioner/Research Partner Tasks
 Labour Market Outcomes:
 Employment or Training Status
 Fit of Employment or Training with skills,
qualifications, vision
 Adequacy of salary
 Linked to future opportunity
 Actions Underway to move forward: Work in
Progress
 Have clients complete post-pre questionnaire
65
Pre-Employability/Job Readiness
Client needs assistance to :



Not
Not
at all much
A
little
Quite
a lot
A
lot
Identify and clarify future direction
(e.g., training, education, employment
or change in life circumstances goal)
Identify personal strengths/resources
that support future direction (e.g.,
training, education, employment or
change in life circumstances goal)
Resolve specific
challenges/vulnerabilities that may
impact on future direction (e.g.,
mortgage, public transit, day care etc.)
66
Life Circumstances
Indicators of challenges in client life circumstances are in
Not Not
A
evidence:
at all much little
 Improved housing is needed
 Improved transportation is needed
 Improved capacity to work and/or study is needed
 Increase in sense of responsibility for own choices and
behaviours is needed
 Increase in ability to set short and long term goals is
needed
 Increase in understanding expectations and demands of
employers is needed
 Reduction in destructive behavior is needed
 Following medical and medication protocols is needed
 Increased openness to change is needed
 Increased access to constructive and positive support
systems is needed
 Improved relationships with family and friends are needed
 Increased trust in other people is needed
 Other: _________________________________
Quite
a lot
A lot
67
Personal Attributes Example
Client needs assistance in the following
areas:
Ability to Self-Manage:
 More info
Developing stronger Self-Esteem:
 More info
Improving sense of Well-Being: i.e.:
 More info
Developing stronger Self-Efficacy: i.e.:
 More info
Increasing Self-Awareness:
 More info
 Other (Please specify):
________________________________
Not
Not
at all much
A
little
Quite A lot
a lot
68
Self-Esteem Attribute

more info: Self-Esteem:
“a person’s overall emotional
evaluation of his or her own
worth, an attitude to self and a
judgement of oneself. Selfesteem encompasses beliefs
(e.g., I am competent; I am
worthy) and emotions (pride,
shame). Self-esteem is the
positive or negative evaluation
of the self”

Demonstrates an optimistic outlook

Sees self as competent and able

Has confidence in ability to interact with others

Uses positive self-talk

Acts with little hesitation

Raises few objections

Asks questions

Articulates skills assertively

Makes eye contact

Is hopeful

Takes reasonable risks
69
Working Alliance
The client and I:



Not at Not A little Quite
all
much
a lot
A lot
established a climate of trust
and comfort in working
together
arrived at a goal that is owned
by the client
agreed on the action plan steps
to help achieve the client’s goal
70
Client Engagement:
The client:

participated actively in the
interview

was focused on achieving results
Not at
all
Not
much
A little Quite
a lot
A lot
71
How Much Change in Personal Attributes
(e.g., self-efficacy and self-esteem)
Occurs For Clients Within 6 Weeks Of
Intervention?
 Of 154 clients for whom practitioners rated client
personal attribute needs before and after, 40%
improved, 14% worsened and 46% experiencing no
change.
72
Contrast with Clients
50
Percentage of Clients
45
40
35
30
25
Before
20
After
15
10
5
0
Very Low
Low
Moderate
High
Need for Help with Personal Attributes -- Client Assessment
73
How Much Do Clients Learn in 6 Weeks
(Regardless of Employability Dimension)?
Practitioners:
70
% of Clients
60
50
40
30
20
10
0
Worsened
Unchanged
Improved
Change in Employability Dimension Need -- Practitioner
Assessment
74
% of Clients
How Much Do Clients Learn in 6 Weeks
(Regardless of Employability Dimension)?
Clients:
100
90
80
70
60
50
40
30
20
10
0
Worsened
Unchanged
Improved
Change in Employability Dimension Need -- Client Assessment
75
How Does the Working Alliance Affect
Learning Outcomes?
Practitioner
Learning Change
Low Alliance/
Engagement
Moderate Alliance/ High Alliance/
Engagement
Engagement
(# of Clients)
(# of Clients)
(# of Clients)
Improved
2 (50%)
12 (48%)
81 (62%)
Unchanged
1 (25%)
7 (28%)
28 (21%)
Worsened
1 (25%)
6 (24%)
22 (18%)
76
How Does the Working Alliance Affect
Learning Outcomes?
Clients
Learning
Change
Low Alliance/
Engagement
Moderate Alliance/ High Alliance/
Engagement
Engagement
(# of Clients)
(# of Clients)
(# of Clients)
4 (100%)
20 (95%)
243 (95%)
Unchanged
0 (0%)
1 (5%)
4 (2%)
Worsened
0 (0%)
0 (0%)
8 (3%)
Improved
77
What Labour Market Outcomes Are Achieved
with 6 Weeks of Intervention?
Overall, 31% of the 313 clients for whom outcome data exist were employed,
17% were in a training/education program and 7% were waitlisted for a
program by the end of the intervention period. Where 65% of clients were
not employed, in a program or on a waitlist before the intervention, 45%
were after the intervention period, a decrease of almost one-third.
70
% of Clients
60
50
40
30
Before
20
After
10
0
Employed
In program
Waitlisted
Not employed
or in program
78
How Are Improvements in Learning
Outcomes Related to Improvements in
Personal Attribute Outcomes?
 Almost all clients (90%) whose learning improved also improved
their personal attributes, in practitioners’ eyes. At the diagonal
end of the table, the majority (65%) of clients whose attributes
worsened also experienced decreases in learning.
 Effectively all clients (99%) experiencing improved attributes
also experienced improved learning.
Improved Learning
Unchanged Learning
Worsened Learning
Improved
Attributes
54 (90%)
1 (2%)
5 (8%)
Unchanged Worsened
Attributes Attributes
29 (45%)
4 (20%)
26 (41%)
3 (15%)
9 (14%)
13 (65%)
79
Question #1:
Applicable Common Indicators
The following items were shown in this study to
have statistically significant connections to
employment outcomes:
Input Indicators
Employability Dimension
need/ competence
(composite & 5 dimension
scores)
Responsibility
Support
Personal attributes
Process Indicators
Working alliance/ client
engagement
80
Question #2: Connecting Inputs, Processes
and Outcomes
 Working alliance/client engagement is a predictor
of both employment outcomes and learning (as
measured by reductions in need for help with
employability dimensions) and is a likely predictor
of improvements in personal attributes (e.g., selfesteem, self-efficacy)
 Over ¾ of clients who obtain employment report
finding employment consistent with their skills
and vision
 (N.B. 70% of sample had not exited services by end
of study)
81
 The more clients need assistance with Skill
Enhancement, the less likely that they will be
employed in 6 weeks
 The stronger clients perceive their personal
attributes, career decision making and work
search skills the more likely they will be employed
in 6 weeks
 Personal attributes improve over 6 weeks
 Learning improves over 6 weeks
 Client learning is highly related to client change in
personal attributes
82
 The ability to assume personal responsibility and
to have access to strong support systems appear
to be among the most influential life
circumstances with respect to outcomes
 Practitioners were overall cautious in reporting
client change; clients were not. More study is
needed but it is clear that assessing change is
complex and subtle—it is not a linear process
with simple cause and effect connections
83
Other Findings:
Model is robust but can be simplified and
improved:
 The number of indicators under each employability
dimension can be significantly reduced and give the same
result
 Only two indicators under Life Circumstances emerged as
significant—the ability to assume personal responsibility
and have access to strong support systems. These
indicators can be significantly reduced
 Working Alliance and Client Engagement can be combined
as one indicator only
 A number of technical improvements to the system have
been identified
84
What Was Learned?
 We can collect common indicators across service
settings, essential for the field to develop
convincing evidence based data (still need larger
samples and longer data collection)
 We have a pathway to measuring client progress in
a meaningful way and a line of sight to BEGIN to be
able to connect interventions with client change
and with successful labour market outcomes
85
What Was Learned?
 There remains much to do but there are strong
indications that the approach can move us from a
“what does not work” evaluation culture to a “what
works”—a real transformation
 A transformed evaluation culture must be predicated
on buy-in from all levels and full recognition of
complexity of evaluation and of client change; this is
NOT a simple process and it is far from linear!
86
What Was Most Surprising?
 Reports from practitioners that using PRIME
made them more reflective about their own
practice
 Despite having to enter data twice,
practitioners “hung in” with the research.
This MATTERS!!
87
Reflective Questions….
 What have you learned?
 What most surprised you?
 What do you think are the implications
for practice?
 What next steps do you see as
important?
88
Other Research Studies
 Go to: www.ccdf.ca
 Click on Projects; Current Projects; From Research to
Practice Symposium; Symposium Program
 One page summaries of 8 additional applied research
studies
 Diverse populations: grades 6-12;
unemployed/underemployed university graduates; social
assistance recipients; laid off workers; disadvantaged
adults
89
Thank you!
Lynne Bezanson and Sareena Hopkins
l.bezanson@ccdf.ca
s.hopkins@ccdf.ca
www.ccdf.ca
www.crwg-gdrc.ca
90