Challenges to
Measuring Practice
Performance
“Why, What & How?”
Dr. Sharon Vitali
Oxford Brookes
School of Health & Social Care
We want to know…
 Is
this student academically eligible
for a Diploma or a Degree in this
discipline?
 Is
this student competent to practice
as a novice in the field?
 Are
our assessment (academic &
practice) mechanisms valid & reliable?
Assessment of
Practice
Why?
http://digitalfox.binaryden.net/?p=55
Assessment of
Practice
What?
Defining Competence

From the Latin word ‘competens’

to “…be fit, proper or qualified”
The Oxford Library of Words and Phrases (1998)
Defining Competence

a competent learner is "…one who has
reached a defined standard of
performance".
Freeman & Lewis (1998:314)
Defining Competence

an amalgam of knowledge, skills and
values

must be demonstrated

measured against a benchmark of
standards that have been agreed upon.
The History of Competency Based
Education & Training (CBET)


competency based education and
training originated in the 1960's and
1970's
popular theoretical fields of
behaviourism and functional analysis
O'Hagan (1996)
The History of Competency Based
Education & Training (CBET)


McClelland (1973), argued that
standard intelligence tests and aptitude
tests had limited validity in their ability
to predict job performance
AND ….suggested use of competencybased testing instead.
The History of Competency Based
Education & Training (CBET)

outcomes = workplace expectations

what a person can do rather than length
of type of training
Toohey, Ryan, McLean & Hughes (1995)
The History of Competency Based
Education & Training (CBET)


refers to skills and knowledge that can
be transferred and applied to new
situations and environments
results in a recognisable, consistent and
portable qualification
Toohey, Ryan, McLean & Hughes (1995)
The Debate Surrounding
(CBET)

Early implementation




Multiple diverse opinions
Disagreement
Very broad guiding principles
Lack of valid or reliable instruments for
measurement
Hyland (1994, 1995)


Findings negative
Academic outrage and resistance
The Debate Surrounding
(CBET)


More popular with ‘task or skill based’
disciplines where activities and
standards of practice are easily defined,
articulated and behaviourally
measurable.
Education, Nursing, Midwifery,
Physiotherapy & Occupational Therapy
Critics of CBET

“…viciously reductionist…minimalist
and impoverished conception of human
thought and action… (which does not)
account adequately for key aspects of
human reasoning, understanding, and
learning.”
Hyland (1995)
Critics of CBET

competency based education and
training is a “…narrow technicist
approach to education which defines
knowledge in the light of bureaucratic
and corporate needs”
Collin (1991:45)
Gambrill (1983); Erault(1994)
and Lishman (1998)



Agree with the competency based approach
except
remind us that high levels of professional
decision-making are necessary for the
delicate and complicated work required in
social work
express concern about the transferability of
skills, the nature of complex critical thinking
and the role that professional values and
attitudes
CBT…


Is a market and government driven
philosophy
Attempts to ensure ability and
accountability thereby enabling
efficient and global recruiting of crucial
segments of the workforce.
FIT FOR PRACTICE…?



Akin to driver’s license
Specific knowledge and skills, which
are believed to be essential, are defined
along with the required level of
performance.
The individual then acquires the
knowledge and skills through a variety
of sources and methods.
FIT FOR PRACTICE…?


Candidate must demonstrate the
required benchmark.
If they fail to attain the benchmark in
any area, they have the option of reacquiring the knowledge/skill and
practicing performance until they are
able to demonstrate attainment of the
benchmark.
FIT FOR PRACTICE…?



No penalty for barely reaching adequacy,
neither is there any reward in excelling.
Simply a license to legally perform a
specific activity in society.
License holder subject to disciplinary
measures upon infractions, and the
license may be revoked should the
misdemeanour be significant enough.
How do we know a graduate is
‘fit for practice’ ?

Test their knowledge

Test their skills

Test their values & attitudes
…against….
The National Occupational
Standards Measurements (SW)
Key Roles (6)
Units of Competency (21)
Occupational Benchmarks (77)
Assessment of
Practice
How?
CRITERION:
Putting the client at ease and establishing a rapport
Management of the interview and sequence of questions
Clarity of questions
Active and open listening
Accurate collection of facts
Awareness and appropriate response to emotional
distress
Respectful empathy
Anti-oppressive attitude
0
1
2
3
4
5
Please rate the student’s performance using
the following gradients:
0 = no evidence of the skill
1 = minimal evidence of the skill, awkward or unsuccessful
attempt
2 = some attempt at employing the skill, with mixed success
3 = good attempts but not smoothly integrated
4 = consistent and smooth use of skill
5 = excellent demonstration of the skill

ALERT = concern that student is demonstrating or
communicating a value base or attitude which is in conflict the
value base and code of practice for social workers and cannot
be explained by their novice status or nervousness.
Exercise One



Watch this video clip and score the
student on the criterion, against the 05 gradient
Turn to the person next to you and
compare your scores for each criterion
Report back the difference between
your scores
Challenges to Measurement of
Practice


Surrogate Markers
Who is measuring what, when & how?


Criterion


Validity, Reliability & Bias
Clarity of Articulation of the knowledge, skill
or value (knowledge behaviour & attitude)
Gradients

(Benner, 1982)
Clarity of Articulation & Translation of
gradients
SURROGATE MARKERS



Surrogate Markers
Is what we are measuring a ‘surrogate’
for something else, from which we
imply or assume a conclusion
i.e. written work vs. practice
VALIDITY




How valid is the measurement
(technique and / or instrument)
Is it measuring what we think/say it is
measuring?
i.e. if I take my weight…on a set of
scales that will tell me how much I
weigh…but not how much of me is fat
or if I am obese
BMI Scales/ Fat meters
TYPES OF VALIDITY
Face
validity
Concept Validity
Construct Validity
Conclusion Validity
Internal Validity
RELIABILITY



Reliability… Sharon’s rubber ruler rule
If this measurement
(technique/instrument) is used by
different people, at different times, in
different situations, etc…does the
measurement come out the same?
INTER-RATER RELIABILITY?
Validity & Reliability
Copyright ©2006, William M.K. Trochim
www.socialresearchmethods.net/kb/index.php
Measurement Errors
and the impact of Bias
Accurate & precise
Imprecise but accurate
Precise, inaccurate, biased
Exponential influence
note how bias >> imprecise measurement in
making for type I error
Measurement Errors
Some general considerations
Imprecise but accurate
Variance = Σ (θ
¯ - θ)2/(n-1)
Precise, inaccurate, biased
Bias = ¯θ - θo
¯¯¯¯¯¯
Std.dev=√variance
inaccuracy indicator = bias2 + var
note how bias >> imprecise measurement in
making for type I error
Accurate & precise
Accuracy = 1/(bias2 + var)
Benner’s (1982) Application of the
Dreyfus Model of Skill Acquisition

Gradients in the development of a skill:





Novice
Advanced Beginner
Competent
Proficient
Expert
Measurement Instrument: the
RICET
Reflective
Ipsative
Competency
Evaluation
Tool
The RICET




(1)
Piloted in Canada 1998
Quantifies & tracks the acquisition of
competencies
Identifies discrepancies in measures of
competencies (ipsative, practice
teacher, tutors, SU/C)
Useful for structuring Learning
Contracts or PDP
The RICET




(2)
Knowledge, Skills & Values
Multiple settings for measurement:
ipsative, classroom, laboratory,
portfolios, & practice learning settings
Scored independently by student, tutor,
practice teacher & SU/C*
No self-referencing to prevent
tendency toward incremental gains
The RICET


Ontario Government Training Course
PQCCA 2000-2001



(3)
Produced statistically significant findings of
changes in levels of competency
SCIE OSWE Project 2005-2009
Currently being used for several
research projects
The RICET





(4)
Triangulates the measurements to reduce bias, and
identify discrepancies
Identifies individual deficits and/or
curriculum/practice learning gaps
Audit tool for efficient rolling curriculum reviews
Enables inter-rater reliability training & testing
Fast, easy inexpensive administration
 can be web-based
 easily transferred to schematic presentation
Consider your/the student’s current state of
competence (knowledge/skills) and indicate a score
(from the guide below) for each of the learning
outcomes listed.
0 = Can not produce any evidence of competence.
1 = Understands the learning outcome, but can
produce only limited or no evidence of appropriate
attempts to put it into practice. Much more
knowledge/practice needed.
2 = Understands, and can offer evidence of tentative
attempts to integrate into current knowledge/skill
base.
Consider your/the student’s current state of
competence (knowledge/skills) and indicate a score
(from the guide below) for each of the learning
outcomes listed.
3 = Demonstrates competence with some
regularity.
4 = Advanced understanding and demonstrating
adequate level of integration of knowledge, skills,
and appropriate application.
5 = Clearly understands and demonstrates
consistent and appropriate application of
knowledge and skills in practice.
Small Group Activity:
Validity of our Measurement

Are we using surrogate measures?



If so … what is the evidence that they are
valid?
How Valid are our “Criterion” as measures
of what a competent novice should be
able to do?
How well articulated are the Criterion
(Competencies/ Capabilities)?

Observable & Measurable?
Small Group Activity:
Reliability of our Measurement


How Reliable is our Measurement
Instrument?
How well articulated and translated are
the gradients?



Observable & Measurable?
How much variability is there in their
application?
How can we improve inter-rater
reliability?
Exercise Two:


Working in nine small groups
Take the Professional Capability
assigned you and try and tease out the
following questions
Professionalism

Identify and behave as a professional
social worker committed to
professional development.
Values and ethics

Apply social work ethical principles and
values to guide professional practice.
Diversity

Recognise diversity and apply antidiscriminatory and anti-oppressive
principles in practice.
Rights, justice and economic
wellbeing

Advance human rights and promote
social justice and economic wellbeing.
Knowledge

Apply knowledge of social sciences, law
and social work practice theory.
Critical reflection and
analysis

Apply critical reflection and analysis to
inform and provide a rationale for
professional decision-making.
Intervention and skills

Use judgement and authority to
intervene with individuals, families and
communities to promote
independence, provide support and
prevent harm, neglect and abuse.
Contexts and organisations

Engage with, inform, and adapt to
changing contexts that shape practice.
Operate effectively within your own
organisational frameworks and
contribute to the development of
services and organisations. Operate
effectively within multi-agency and
inter-professional settings.
Professional leadership

Take responsibility for the professional
learning and development of others
through supervision, mentoring,
assessing, research, teaching,
leadership and management.
References:





Benner P (1982) From Novice to Expert The American Journal of
Nursing 82(3)
Bloom, M. & Fischer, J. (1982). Evaluating Practice: Guidelines for
the Accountable Professional. New Jersey: Prentice-Hall Inc.
Collins, M. (1991) Adult Education as Vocation London, Routledge.
Csernick, R. Gordon, K. & Vitali, S. (2000) The Practice of Field Work:
What Social Work Interns Experience In the Field Paper Presented
Joint Conference of the International Federation of Social Workers
and the International Association of Schools of Social Work.
Montreal, Quebec, IFSW.
Csiernick, R., Vitali, S., and & Gordon, K. (2000b) Student and Field
Instructor Perceptions of a Child Welfare Competency-Based
Education and Training Project Canadian Social Work 2(2), pp. 5364.
References:







DfEhttp://www.education.gov.uk/swrb/a0074240/professionalstandards-for-social-workers-in-england
Erault, M. (1994) Developing Professional Knowledge and
Competence. London, Falmer Press.
Freeman, R. & Lewis, R. (1998) Planning & Implementing
Assessment London, Kogan Page Limited.
Gambrill, E. (1983) Casework: A Competency-Based Approach
Englewood Cliffs, Prentice Hall.
Hyland, T. (1994). Competence, Education and NVQs: Dissenting
Perspectives London, Cassell.
Hyland, T (1995). IN Hodkinson, P. & Issitts, M ed. The Challenge of
Competence London, Cassell.
Lishman J (1998) Personal & Professional Development IN Adams R,
Dominelli L & Payne M. eds Social Work: Themes Issues and Critical
Debates. London, Macmillan.
References:

McClelland, D. (1993). Testing for Competence Rather Than For
Intelligence. American Psychologist 28 pp.1-14

National Occupational Standards



O'Hagan, K. ed. (1996) Competence in Social Work Practice: A
Practical Guide for Professionals. London & Bristol, Jessica Kingsley
Publishers.
Oxford Library of Words & Phrases (1998) The Oxford Library of
Words & Phrases (1998) Oxford, Oxford University Press.
Toohey, S. Ryan, G. McLean, J. & Hughes, C. (1995), Assessing
Competency Based Education and Training: A Literature Review.
Australian and New Zealand Journal of Vocational Education and
Training pp. 86-117.
References:

Vitali S & Buchanan A, Competency Based Education in Social Work
IN: Vanderhoeven JL & Chulakov D.(EDS) (2006) Social Work in
Schools. A Comparative Perspective from England (UK), Flanders
(Belgium), Perm Region (Russian Federation) and Sachsen-Anhalt
(Germany). (Issues in European Education Series, n° 7). Antwerpen


Apeldoorn: Garant.
Vitali S (2005) "Lessons from the Front Line: Evaluating the Post
Qualification Child Care Award" D.Phil Dissertation, Oxon
Vitali S Can we measure and track the acquisition of competency on
a professional social work degree? IN Burgess H & Carpenter J ED
(2010) The Higher Education Academy for Social Policy & Social
Work Monograph Number 2: The Outcomes of Social Work
Education:developing evaluation methods
Vitali S (2011) The Acquisition of Professional Social Work
Competencies Social Work Education Vol 30(2) 236-246.