New Ways
to
Evaluate Counselling Effectiveness
Bryan Hiebert
Adjunct Professor
Department of Educational Psychology & Leadership Studies
University of Victoria
Professor Emeritus in Education
University of Calgary
1
What is counselling all about?
Counselling
or
Counselling Psychology
Clinical Psychology
Social Work
etc.
1. Write down the words or phrases that
come to your mind
• 1 minute
2
Overview
1. Background
2. An alternate to RTCs
3. A framework for demonstrating value
4. Alternate evidence gathering tools
Purpose
1. An alternate way of looking at demonstrating the
value of human services
2. Intended to stimulate discussion
3
Evaluation is important
1. But counsellors do not evaluate their work
with clients
2. 1993 study
• 40% never evaluated their work with clients
• 35% evaluated with client during interview
3. 2005 study
• 33% did not answer
• 56% reported data
 Client flow, counsellor time use, etc.
4
How to address the problem
We need an approach that is:
1. Comprehensive enough to include what is needed
2. Simple enough for people to use
3. Incorporates evaluation into standard practice
• Your own practice becomes your data source for
predicting client outcomes

Each client is a n = 1 experiment
• Across time and across clients, counsellors
acquire ability to make predictions
linking interventions & outcomes
Local Clinical Scientist (Professional Practitioner)
5
Evidence-Based Outcome-Focused Practice
Input  Process  Outcome
Resources
Actions
Client change
Need to link process with outcome
6
Evidence-Based Outcome-Focused Practice
Input

Process

Outcome
Indicators of client change
1. Learning outcomes
• Knowledge and skills linked to intervention
2. Personal attribute outcomes
• Changes in attitudes
• Intrapersonal variables
(self-esteem, motivation, independence)
3. Impact outcomes
• Impact of #1 & #2 on client’s life,
e.g., employment status, enrolled in training
• Societal, economic, relational impact
7
Outcomes of Counselling
1. Client learning outcomes
•
•
Knowledge
Skills
+
Precursors
Personal Attributes
2. Impact on client’s life
•
•
•
Client presenting problem
Economic factors
Third party factors
8
Precursors
Intervene between learning outcomes
& impact outcomes
1. Attitude
2. Motivation
3. Self-esteem
4. Stress
5. Internal locus of control
6. Belief that change is possible
Changes counsellors mentioned
That were being observed but not reported
9
Evidence-based Outcome-focused Practice
Input

Process
 Outcome
Activities that link to outputs or deliverables
Generic interventions
•
Working alliance, microskills, etc.
Specific interventions
1. Interventions used by service providers
•
•
Skills used by service providers
Home practice completed by students
2. Programs offered by school
3. Involvement by 3rd parties
4. Quality of service indicators
•
Stakeholder satisfaction, including students
10
Evidence-based Outcome-focused Practice
Input 
Process
 Outcome
Resources available
1. Staff
•
Number of staff, level of training, type of training
2. Funding
•
Budget
3. Service guidelines
•
Agency mandate
4. Facilities
5. Infrastructure
6. Community resources
11
Outcome Focused Evidence-Based Practice
Input  Process  Outcome
Need to link process with outcome
1. What will I do?
2. What are the expected client changes?
 What do I expect clients to learn?
 What sorts of personal attributes do I want my
clients to acquire?
 What will be the impact on their lives?
3. How will I tell?
12
Outcome Focused Evidence-Based Practice
RESOURCES AVAILABLE






Context: Structure of
opportunity
Staff: Number of staff, level of
training, type of training
Funding: Budget
Service guidelines
Facilities
Infrastructure
Community resources
ACTIVITIES THAT LINK TO
OUTCOMES OR DELIVERABLES
Generic interventions
 Working alliance, microskills, etc.
Specific interventions
Client
 Context
 Needs
 Goals
OUTCOMES

 Strategies linked to specific client problems
(stress, grief, depression, career, etc.)
 Client home practice
 Other
Programs & Workshops
 Facilitation guides
 Intervention manuals
External Referral
INDICATORS OF CLIENT (LEARNER) CHANGE
1. Learning outcomes
 Changes in knowledge and skills linked to the program or intervention used
 Progress Indicators
End Result Indicators
2.
Personal attribute outcomes
 Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.
 Progress Indicators
End Result Indicators
3.
Impact Outcomes
 Changes in the client’s life resulting from application of learning
13
Evidence gathering
1. A Framework for evaluation
• Learning from the evaluation experts
2. Gathering evidence of our impact
• Documenting what works and why
3. Developing evidence gathering tools
• Need to use informal evidence
Some examples
14
Assessment as Decision Making (vs. Judgement)
Please use a two-step process
1. Would you say that your level of mastery of
the attribute under considerations is
unacceptable
00
1
acceptable
2
3
44
1. Then assign the appropriate rating





0 = really quite poor
1 = just about OK, but not quite
2 = OK, but just barely
4 = really very good
3 = in between barely OK and really good
15
Problem with skill self-assessment
1. Participants asked to rate their skill
(or knowledge) before and after a program
2. Often, pre-workshop scores are high
and post-workshop scores are lower
• People find out as a result of the workshop that they knew
less than they thought or had less skill than they thought
• Based on the new awareness, post-scores are lower
3. People don’t know what they don’t know
4. How can we get around this problem?
16
Assessing Learning & Attribute Outcomes
Post-Pre Assessment
1. We would like you to compare yourself now and before
the workshop. Knowing what you know now, how
would you rate yourself before the workshop, and how
would you rate yourself now?
2. Please use a two-step process:
• Decide whether the characteristic in question is
acceptable or unacceptable, then
• assign the appropriate rating
unacceptable
00
1
acceptable
2
3
44
17
Applied Career Transitions Program
(on-line program for unemployed university grads)
For Module 1
•
•
•
•
•
All together there were 10 (items) x 29 (participants) = 290 ratings
Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings
Unacceptable Ratings decreased from 50% to 14%
Pre: 6 Exceptional Ratings – Post: 130 Exceptional Ratings
Exceptional Ratings increased from 2 to 44% of the participants
18
Results: Impact outcomes
Module 1
• 23 out of 29 had found a job
• 10 of the jobs lined up well with career vision
Module 2
• 4 out of 6 had found a job
• 3 of the jobs lined up well with career vision
19
Post-Pre Assessment:
Bridging Health Care with Self-Care
1. Knowledge of stress and stress control
2. Knowledge of how to manage personal change
3. Knowledge of nutrition and nutrition control
-------------1. Level of stress
2. Level of nutrition (high=healthy)
3. Level of fitness
4. Confidence in ability to manage personal
change
20
Participant Self-Assessed Change
Stress & Stress Control
Knowledge
Level
Response category
Start
Finish
Start
Finish
very low
17%
0%
4%
9%
somewhat low
30%
0%
4%
35%
neither high nor low
26%
0%
13%
26%
somewhat high
21%
70%
39%
26%
very high
4%
30%
39%
4%
21
Post-Pre Results
Bridging Health Care with Self-Care
Stress & Stress Control
Start of Program
End of Program
80
70
60
50
40
Knowledge
30
Stress
20
10
0
80
70
60
50
40
30
20
10
0
Vertical axis indicates % of participants
22
Post-Pre Results
Bridging Health Care with Self-Care
Manage Personal Change
Start of Program
End of Program
80
80
70
70
60
60
50
50
40
40
30
Knowledge
30
20
Ability
20
10
10
0
0
Vertical axis indicates % of participants
23
Attribution for Change
1. To what extent would you say that any
changes depicted above were a function of
participating in our program, and to what
extent were they a function of other factors
(e.g., job, personal circumstances, etc.)
mostly
other
factors

somewhat
other
factors

0%
0%
uncertain

9%
somewhat mostly
this
this
program program


26%
57%
24
Participant Comments
1. This program made me realize that change is possible for
me to make, and that even a small effort can bring
significant benefits. When I joined the program, I thought
that only a superhuman effort would result in any positive
benefits, and that thought itself was a barrier to making a
behavioural change. Now I know that “I can do it” and that
knowledge itself is half the battle
2. This is a fantastic program. The biggest benefit for me is
that I am starting to see myself as being a more active
person, rather than a “couch potato”. I appreciated the
focus of the program being on life change and wise
choices, rather than on diet and weight loss. I
25
Tracking client goal attainment
1. Clients are different
• Live in different contexts
• have different goals
• require different interventions
2. How can we consolidate the results?
26
Goal Attainment Scaling
1. A bit better than expected
1. About what I expected
+2
+1
0
1. A bit worse than expected
-1
1. Much worse than
expected
-2
Observable Indicators
1. Much better than expected
27
Thought Listing
Stimulus question
1. For Clients:
•
•
•
What are the main factors contributing to your
presenting problem?
What needs to happen in order for you to achieve your
goals?
What are the barriers you need to overcome?
• For counsellors:

•
What is counselling all about?
What factors are responsible for client change?
2. Write down all the words or phrases that come to
mind in response to this question.
28
Cognitive Mapping Task
1. Complete the thought listing task
2. Transfer each item to a “Post-it” sticker, 1
item per sticker.
3. Arrange the stickers on a large sheet of graph
paper
• Place related concepts close to each other
4. Draw connecting lines between stickers that
represent related concepts
5. Draw a circle around any clusters of concepts
• label each circle
29
Cognitive Mapping-Example
What is
counselling
all about?
Scoring scheme:
http://cmap.ihmc.us
30
Relaxation Monitoring Sheet
Date/Time
Start
Indicator
Finish
Difference
pulse rate (beats per minute)
breathing rate (beats per min.)
finger temperature (degrees)
Feelings, body sensations, etc.
Thoughts, perceptions, imagery, etc.
.
31
Informal Evaluation
of
Headache, Pain,
and
Related Affective States
32
Self-Monitoring Headache
0 - No headache
1 - Low level,
only enters awareness when you think about it
2 - Aware of headache most of the time,
but it can be ignored at times
3 - Painful headache, but still able to continue job
4 - Severe headache,
difficult to concentrate with demanding tasks
5 - Intense incapacitating headache
33
Headache Monitoring Grid
Level
5
4
3
2
1
0
06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05
(time of day)
Before treatment
After treatment
34
Headache: Indicators of Success
Measure
Beginning
End
4.1
0.6
Number of intense (level 4 or 5 )
headaches
3
0
Hours of intense (level 4 or 5 )
headache
13
0
Hours headache free
0
10
Average headache level
35
Self-Monitoring Stress
0 - No stress
1 - Low level,
only enters awareness when you think about it
2 - Aware of stress level most of the time,
but still able to stay focused on job
3 - Very stressful,
beginning to interfere with what you are doing
4 - Severe stress,
creates major interference with what you are doing
5 - The most stress you ever feel
36
Self-Monitoring Confidence
0 - No confidence at all in your ability to handle the
situation
1 - Low confidence, but you might get lucky and it
will turn out OK
2 - Sort of confident, you think you might be able to
handle it, but probably not
3 - Pretty confident, you think you have a chance
of this working out OK
4 - Very confident, you’ll probably be able to pull
this off
5 - Completely confident about what you are doing
37
Self-Monitoring Job Satisfaction
0 - No satisfaction at all with your job
1 - Somewhat satisfied, there are some
23-
45-
things about your job that are OK
Satisfied enough – as jobs go, this one
is as good as any
Mostly satisfied, occasionally there are
things about your job that you actually
like
Very satisfied, but there are occasional
things bother you
Completely satisfied with your job,
feeling a sense of self-fulfillment
38
Self-Monitoring Motivation
5 - Very motivated-it's the most important thing for
me to do today
4 - Very motivated, but something might come up
to interfere
3 - Quite a bit motivated, I think I will end up
doing some of it
2 - I sort of care and I might get around to doing
some work today
1 - If I run out of things to do, I'll try looking in
some of my books
0 - I couldn't care less if I ever did my school work
39
Using Self-Talk to Control Anger
With cognitively-challenged adults
in residential settings
Hiebert, B., & Malcolm, D. (1988). Cognitive
strategies for mentally handicapped clients.
In B. O'Byrne, (Ed.). Natcon-14 (pp. 289-298). Toronto,
On: Ontario College Counsellor's Association.
(Reprinted in Resources In Education, June 1989, ERIC
Document Reproduction Service No. Ed 302800)
40
Picture Stimulus: Pushing
41
Picture Stimulus: Hitting
42
Picture Stimulus: Kicking
43
Interview Protocol for Assessment
1.
2.
3.
4.
5.
6.
7.
8.
9.
I am going to show you 3 pictures and ask you
some questions about each picture.
Look at this picture.
Tell me how many people are in the picture.
What are the people doing in the picture?
Point to the person that is being pushed/kicked/hit
If you were the person being pushed/kicked/hit,
how would you feel?
If you were the person being pushed/kicked/hit,
what would you feel?
If you were the person being pushed/kicked/hit,
what would you think?
If you were the person being pushed/kicked/hit,
what would you do?
44
Interview Scoring Sheet
Statement
Feeling
Anger
Non
anger
Thinking
Non
facilitative
Facilitative
Action
Aggressive
Non
aggressive
1–5
6 … how would you
feel?
7 … what would
you feel?
8 … what would
you think?
9 … what would
you do?
45
Field Test Results: Picture Stimuli
Response Category
Test
time
Pre
test
Post
test
Group
Feeling
Thinking
Action
Anger
Non
anger
Facilitative
Non
facilitative
Aggressive
Non
aggressive
Male
10
19
6
8
8
9
Female
13
13
6
3
14
6
Total
23
32
12
11
22
15
Male
8
18
15
0
0
20
Female
10
16
11
0
2
15
Total
18
34
26
0
2
35
46
Frequency of Anger Outbursts
10 Mentally Handicapped Residents
47
Monitoring Your Self-Talk
Con
 
Pro
  





48
Changing Your Self-Talk
Con

Pro








49
Self-monitoring Stress
Subjective Units of Disturbance
50
Self-monitoring Stress
Subjective Units of Disturbance
51
Self-monitoring Stress
Subjective Units of Disturbance
52
Emotional Thermometer
53
Outcome-Focused Evidence-Based Practice
Outcomes
Processes
Learning
Personal
Impact
Decision making
Skill training
Stress control
Self-management
Other
54
Linking Process and Outcome
We have evidence that the facilitator followed the
program
We have evidence about which participants followed the
program more (vs. less) closely
• We could compare the results of those who follow the
program compared to those who did not
We have evidence of how much change took place
Learning
We have a connection
counsellor process  client engagement  outcomes
Attribute
Impact
Participants attribute change to the program
A solid link between program and outcome
55
Professional Identity:
What we do defines who we are
1. Most practitioners define their job as delivering
services
• So … they do not evaluate
the impact of their services on clients
2. What is counselling all about?
3. Needs to include BOTH process and outcome
• What will I do to facilitate client change?
+
• How well is it working?
56
To demonstrate value, we need to develop
Culture of evaluation:
We need to reach the state where
•
•
•
•
Identification of outcomes is an integrated part of providing
services
 Without efficacy data, counselling services are
vulnerable
 It is in our best interest to gather evidence attesting to
the value of the services we provide
Measuring & reporting outcomes is integrated into practice
 Informal data are acceptable
Reporting outcomes is a policy priority
Outcome assessment is a prominent part of practitioner
education
This needs to be a priority in all sectors
57
Multiple Effects of
Relevant Evaluation Practices
1. On client
•
•
•
Increased motivation
Increased awareness of effectiveness
How am I doing at this?
2. On counsellor
•
•
•
•
Increased effectiveness
Increased confidence
Increased motivation
Increased job satisfaction
3. On agency
•
Value of services
58
What is counselling all about?
Counselling
or
Counselling Psychology
Clinical Psychology
Social Work
etc.
1. Write down the words or phrases that
come to your mind
• 1 minute
59
New Ways
to
Evaluate Counselling Effectiveness
Questions? Comments?
Bryan Hiebert
Adjunct Professor
Department of Educational Psychology & Leadership Studies
University of Victoria
Professor Emeritus in Education
University of Calgary
http://www.bryanhiebert.ca
60