Towards measuring learning
experiences as outcomes of
psychosocial Intervention
Willem Melief
Why the development of the learning
concept?
A The intention to improve the scientific base of
the outcome measurement instrument Sater
B The consistent finding of learning experiences
as outcomes in the application of Sater
What is Sater ?
A standardized system to measure Benefits of
Social Work Intervention Experienced by
Clients
Developed by Verwey-Jonker Institute:
Willem Melief
Meta Flikweert
Norbert Broenink
Developed on Requests of
stakeholders
Regular requests for research about
•outcomes of social work
•client experiences
•client satisfaction
By stakeholders
•Agencies for Social Work
•Representatives of clients
•Financiers and government
Characteristics of SATER
• The use of client problem definitions
• Client experiences linked to problem
categories and intervention aspects
• Clients enter answers in computer
questionnaire for optimal privacy
• Standard table and graph output produced by
a few standard SPSS command files
• Standard report format
Measurement of ‘Client
Experiences’
Two elements
1 Experienced benefits in relation to the
experienced problems
Benefits  results, outcomes
2 Satisfaction about personal treatment
and treatment process
.
Sater Data
•Clients of 6 agencies
5 in a pilot; 1 in a further application
•Database of appr. 1275 client interviews
•6 agency reports
Benefit categories
Problems partly or totally solved
= Prob. Sol.
Learned to accept and live with unchangeable
conditions
= Acceptance
Learned to handle problems
= Handling
Experienced support and understanding
= Sup/underst
Experienced Benefits
in %; N = 250 (Gron) 1025 (Pilot)
60
50
40
Groningen
5 pilot orgs
30
20
10
0
Prob.sol
Handling
None
Change oriented benefit
categories
• Problem solving
• Learning
– Learning to handle
problems including
solving them
– Learning to accept
and live with
Unchangeable
reality
• 33%
• 50+ %
• 20 + %
Arguments in favour of the further
development of learning
•The frequent occurrence in benefits
reported by clients
•Connection to social work methodology
•Perspective of lasting, repeatable effects
of intervention
•Connections to neighbouring fields
– psychotherapy
– psycho-education
Questions to be answered
about learning effects
1 What are the learning effects that
correspond to the sort of learning
reported by the clients in our research?
2 How can we measure these learning
effects? .
Kinds of learning
• Remembering
historical
dates
• Remembering the periodic
table of chemical elements
• Acquiring skill in mental
arithmetic
• Acquiring skill in writing
correct language
• Mastering the art piano
playing
• Becoming a accomplished
football player
• Acquiring
skills
in
controlling ones violent
mental impulses
• Finding out how to relate
to other people
• Acquiring skills in solving
(specific) problems
• Acquiring skills in staying
out of (specific kinds of)
trouble
Psychosocial learning
Knowledge elements
Behavioral skills
Attitudinal elements
Thought procedures
Adequate handling
of the
Psychosocial situation
Repertoire of Learned
Elements
At disposition of person
Selection from
repertoire of elements
matching the
Psychosocial situation
Causes for inadequate handling of
the situation
• Missing elements in the repertoire
• Shortcomings of elements in the repertoire:
– Lack of information
– Insufficient skills
– Insufficient training in practice
• Shortcomings in the ability to assess the situation,
• Shortcomings in the control and co-ordination of the use of
elements from the repertoire.
• Factors blocking adequate use of elements:
– Overwhelming emotions (fears, anger, depression, etc)
– Serious psychopathology
– Serious shortcomings in elementary conditions
– Serious uncontrollable behaviour of other people
Main helping strategies
• Removing the blockades first
– Shortcomings in elementary conditions
– Overwhelming emotions
– etc
to free the road for:
• Teaching clients to accomplish lasting results:
–
–
–
–
necessary information
skills
ability to assess the situation
ability to apply the right element from his repertoire
Literature Search
• Social Work Literature
– Very little direct or systematic treatment of learning
effects
– Description of intervention outcomes that can be
interpreted as learning
for example behaviour modification directed treatment
– A Dutch approach of learning as model for
emanicipatory social work and empowerment
• Social Science Literature :
– The work of the Dutch psychologist and scholar A.D. De Groot,
who intoduced the concept of fundamental learning
experiences.
Traditional measurement of learning
effects
• In traditional educational research
Learning effects = behavioural effects
• Acquired knowledge
• Skills
• Attitudes
• This approach is very valuable !!, but
Coverage problem
but is not enough
The limitation to only measuring
behavioural effects causes a
Coverage problem =
• subjective fundamental learning
experiences are not measured
(covered)
Fundamental Learning Experiences
• Conscious experience of person that:
– He has learned what behaviours (skills,
attitudes) he has at his disposition = ready
to be used
– He has learned when certain behaviour is
adequate and when not
– He has learned to flexibly use available
behaviours adapted to new situations
Fundamental Learning Experiences
• Have as much to do with things
(knowledge behaviour) that are always
as they are and reactions that always
have to be the same:
as with
• Exceptions, things that are (sometimes)
different and reactions that sometimes
have to be different.
I have learned something about:
UNIVERSAL
EXISTENTIAL
Rules of the world
Surprises of the world
Things that are always Other, new things that exist,
WORLD what they are
unexpected
possibilities
facts, etc.
Situation S leads always Situation S does not always
to response R and not R* lead to response R; R* is
possible
AB
CD
Rules about myself
Surprises about myself
Personal regularities
Other, new possibilities open
SELF
In situation S I will to me
always react with R and It is possible that in situation
never with R*
S I will react with R* and not
always with R
Relevance for social work
• Terminology and content are very similar to our
own rudimentary way of thinking.
• Problems of clients relate to shortages in
conscious application of behavioural elements
due to:
– lack of insight in what one has at one’s
disposition
– lack of skill to assess the situation and select
the right reaction to it.
Preliminary conclusions
• Measuring learning experiences by clients are
important;
– Provides core insights that are missing in
current outcome research
– This information cannot be measured in
another way
• There is a promising connection to theoretical
thinking
• Antagonism in the research and professional
world is a serious obstacle
Opposing positions
Pro objective measurement :
-What is not objectively measurable does not count
(is irrelevant)
-Subjective experiences of clients are not to be
trusted (have no validity)
versus
Against objective measurement
-reduces the uniqueness of the client
-subjective data are too important to be discarded
-mostly strong preference for qualitative research
Supplementary Model
• Quantitative outcome measurement using
objective measurement = OK!!
– Base of evidence base methods
But should be supplemented by:
• Quantitative measurement of subjective
experiences, which = also OK
– Learning experiences
– Satisfaction
• Qualitative research = certainly also OK
The future
• Expand the theoretical base for measuring
learning experiences
– On preference including contacts with scholars
I missed in other countries.
• Convert the very crude part about learning
experiences in the Sater system to a more
sophisticated measuring tool
• To correlate experience outcomes with
outcomes of objective measurement