PARTICIPANT NOTES
AVOIDING THE PITFALLS OF ASSESSMENT
Sydney
Friday 17 August, 2012 (9.30am-4pm)
Level 30, 9 Castlereagh Street Sydney (offices of WSAA)
Do you find assessments a challenge when things go wrong?
Are you unsure about your assessments sometimes?
Worried that that you might be missing something?
What is the purpose of the workshop?
This workshop is designed to provide you with the opportunity to discuss the critical points
above as they are all linked and are the keys to avoiding assessment problems. With other
assessors from the water industry, you will discuss the challenges faced and identify strategies
to assist in avoiding these challenges.
Outcomes
At the end of the session you will be able to:





Follow ‘good’ assessment practice
Gather quality evidence
Use flexibility in your approach to assessment
Work effectively and efficiently with participants
Use effective strategies for dealing with ‘pitfalls’ you may encounter
Acknowledgement
Some of the information contained in these notes has been adapted from the Department
of Education and Communities workshops.
Where are you at?
In your group introduce yourself to others, if necessary, and then discuss the following
questions. From this discussion identify 3 outcomes your group would like to achieve from the
workshop.
1. What do you find most difficult about organising and planning assessment, eg,
developing assessment strategies and plans, selecting appropriate tools, organising
resources and people, etc?
2. Do you have any difficulties in working with assessment candidates and other
stakeholders in the assessment process – whether during the consultation, briefing,
assessment, review or feedback phases of the process?
3. Do you have difficulty identifying ‘quality evidence’?
4. Do you find it difficult to deal with challenges in assessment?
5. Can you confidently make assessment decisions, or do you sometimes doubt your
own judgements about candidate competency?
6. What overall concerns do you have about assessment and about your own abilities
as an assessor?
Avoiding the Pitfalls in Assessment
August 2012
Page 2
Good practice in assessment
‘Good practice in assessment’ can mean a number of things, however at its heart its about
achieving consistency in assessment practice and being confident that we have processes,
tools and people (assessors) capable of delivering consistent, high quality assessment
outcomes.
As we know assessment is:

Process driven

The collection of quality evidence

Making decisions based on this evidence against the relevant unit/s of competency
Activity – Group Brainstorm-Key components of good
assessment practice
Avoiding the Pitfalls in Assessment
August 2012
Page 3
Key components of quality assessment practice include

Unpacking training packages including analysing units of competency

Identifying relevant pathway for student

Using variety of assessment methods that relate to the workplace and the job

Making sound decisions

Using ‘holistic’ assessment

Ensuring the principles of assessment are met

Ensuring the rules of assessment are adhered to

Using a sound assessment process

Having qualified, experienced assessors

Customising, clustering to suit
Add to the list:
Avoiding the Pitfalls in Assessment
August 2012
Page 4
PARTICIPANT NOTES
Role and responsibilities of an assessor
As in any profession we as assessors are required to fulfil a role and with that come
responsibility to ourselves as assessors, the candidate, the RTO and any others
involved in the assessment process.
Activity – Small Group
1. When planning competency based assessment what do you see as your role
& responsibilities?
2. When conducting competency based assessment what do you see as your
role & responsibilities?
3. When reviewing and recording the assessment what do you see as your role &
responsibilities?
4. Is there a limit to those responsibilities? If so what are they?
Avoiding the Pitfalls in Assessment
August 2012
Page 5
PARTICIPANT NOTES
Role and responsibilities of a candidate
The candidate also has a role to play in the assessment process and often the way
the assessor fulfilled their role will impact on the candidate.
Activity – Small Group
Think about your experience with candidates and consider…...
1. What does the candidate do in the assessment process?
2. Do you think we, as assessors, should empower the candidates? If so how, if
not why not?
3. Who has the major role to play in the process, the candidate or the assessor?
Why?
4. If you had to list 3 major areas the candidate was responsibility for in the
assessment process what would they be?
Avoiding the Pitfalls in Assessment
August 2012
Page 6
PARTICIPANT NOTES
Activity- Using your unit/s to assist you
The key area when using units of competency is to have a clear picture of the
requirements of the unit and what evidence you may gather. One simple but
effective way to do this is using highlighters.
1. Read through the unit and try to identify the best methods of gathering
assessment evidence. Assign each method a different coloured highlighter
pen.
2. Focusing on one type of assessment at a time, highlight all the key parts of the
unit that you will assess using that method.
3. Repeat step 2 above for each method of assessment. Some items will be
included in more than one method of assessment. Is it likely that appropriate
emphasis will be given to critical aspects of performance?
4. Transfer the information onto a matrix if necessary, eg, if you need an
electronic record of your analysis of the unit.
Avoiding the Pitfalls in Assessment
August 2012
Page 7
PARTICIPANT NOTES
Quality Evidence
Assessors often worry about collecting enough good quality evidence to feel
confident about their assessment decisions. At a more fundamental level, it can
even be confusing to decide whether a document or an opinion actually constitutes
evidence at all. Assessment evidence is valid, current, authentic data, from a variety
of sources, and provides sufficient evidence of performance, over a period of time,
to make a judgement about candidate competence.
Evidence
Candidate’s explanation of how to do it,
or parts of it, and their answers to specific
questions.
Specific, competency-related references,
preferably on letter-head, signed and
dated by appropriate person.
Performance reviews/appraisals concise,
relevant to the competencies being
assessed.
Emails and minutes of meetings attended
by the candidate which provide proof of
work undertaken.
Recently produced products (anything
from a report to a cake or building) and
proof they have been produced by the
candidate.
NOT Evidence
(although some of it could become
evidence)
Candidate’s verbal assurances they can
do it “I’ve been doing it for years.”
Personal references making general
comments about character.
Job descriptions/position descriptions,
especially if not signed off.
Journals and diary entries written by the
candidate.
Unauthenticated products which the
candidate cannot prove are their work.
Old products – especially if methods of
work have changed.
Unauthenticated pictures, video or
photographs of work or products.
Authenticated (eg, signed off) and
dated photographs or videos of work in
progress or products.
‘Evidence’ by exception – “I must be
working safely – I’ve still got all my
fingers!”
Observed performance.
Observation of inappropriate
simulation/role play (eg, swimming on dry
land)
Rules of Evidence
The main differences between evidence and non-evidence relate to the rules for
assessment evidence:
Currency
Can the assessor be confident the candidate can still perform to the standard
demonstrated by the evidence?
Authenticity



Can the assessor be confident they are looking at the candidate’s own work?
Are qualifications, references and licences presented by the candidate
authentic documents?
Avoiding the Pitfalls in Assessment
August 2012
Page 8
PARTICIPANT NOTES
Validity

Is the evidence sufficiently relevant to the competencies being assessed?
Sufficiency


Is there enough evidence to cover all components of competency – task skills,
task management skills, contingency skills and job/role environment skills?
Does the evidence show competency over a period of time?
The notion of product and process; critical aspects, AQF levels all play a part in
quality evidence.
See checklist in attachment.
Your notes:
Avoiding the Pitfalls in Assessment
August 2012
Page 9
PARTICIPANT NOTES
Dimensions or components of competence
Component/ Dimension
Lower order
Example
Higher order Example
Task skills-undertaking a specific
workplace task
Using a computer
Writing a training
program
Saving a document
on the computer
while printing
another
Planning a training
development project
from needs analysis to
trial run, evaluation and
finalization.
(Performing to an acceptable level)
Task management skills- managing a
number of different task to complete a
whole work activity
Prioritising work
Contingency management skills –
responding to problems and irregularities
when undertaking a work activity, such as:
Computer breaks
down
*Breakdowns
Virus is detected on
external storage
media
*Changes in routine
*Unexpected results or outcomes
Managing the different
requirements of a
range of stakeholders
Coping with changing
priorities and
rescheduling.
*Difficult or dissatisfied customers
Job/role environment skills – dealing with
the responsibilities and expectations of the
work environment when undertaking a
work activity, such as:
*Working with others
*Interacting with clients and suppliers
*Complying with standard operating
procedures
*Observing enterprise policy and
procedures
Avoiding the Pitfalls in Assessment
August 2012
Working with other
computer
operators or others
in the team when
expected
Ensuring safe
working practices
Working in line with
specific
organization
requirements
Working with a range
of stakeholders
Working within industry
regulatory
requirements for
training
Developing training
which suits the
organization culture
Page 10
PARTICIPANT NOTES
Pitfalls for assessors
As an assessor you should be aware that there are a number of pitfalls and errors that
assessors can make when conducting assessments.

Marking down the middle
This is the tendency to judge candidates as being in the middle of a rating scale. This
does not identify strengths and weaknesses of the candidate'’ skills.

The Halo Effect
This is the tendency to be influenced by the personality of the candidate. If they have
an impressionable effect, the assessor may rate the candidate highly. The converse is
known as the Horn Effect, where the assessor may judge the candidate as not yet
competent because they have a low impression of them.

Failing to observe
If the assessor is inexperienced they may fail to observe aspects of performance such
as non-verbal cues in an interaction.

Failing to record
This is the tendency not to record all aspects of the assessment because of the belief
that it may be recorded accurately later. Everyone forgets things and the assessor
may just forget a critical requirement.

Mirror effect
This is the tendency to judge a candidate as competent because the candidate
thinks similarly to the assessor or acts in a similar manner.
From Certificate IV in Assessment & Workplace Training (BSZ98) Australian National
Training Authority
 Communication
Assessors require a high level of communication skills to effectively and efficiently
work through the assessment process. One of the big pitfalls for assessors is ‘arguing’
about evidence.
Getting caught up in arguments about candidates’ evidence is a big pitfall for
assessors. Explaining evidence in advance and reiterating this through the assessment
process will probably avoid the problem. However, in cases where this doesn’t work, it
is well for assessors to be ready for ‘accusations’.
Activity – Group discussion
What is your experience with the above?
Avoiding the Pitfalls in Assessment
August 2012
Page 11
PARTICIPANT NOTES
Activity – Avoiding pitfalls!
Working in small groups:

Discuss the common pitfalls described below regarding assessment
evidence.

Relate them to your own experience

Think of examples you may have and discuss these with your group

Discuss recommended strategies

Record major findings/conclusions/issues on the butchers paper provided

Review the strategies from the other groups and contribute to a whole group
discussion
Pitfall
Avoiding the Pitfalls in Assessment
August 2012
Strategy
Page 12
PARTICIPANT NOTES
Common pitfalls when gathering assessment evidence
As has been indicated, getting caught up in arguments about candidates’
evidence is a big pitfall for assessors. Explaining evidence to the candidate well in
advance and reiterating this through the assessment process will probably avoid
the problem. However, in cases where this doesn’t work, it is well for assessors to
be ready for accusations, which may be expressed in different ways. For
example:
The ‘personal’
‘you know me, you know what I can do – isn’t that evidence enough’
‘we work together every day – what more do you want?’
‘my record speaks for itself, anyone will tell you that’
‘are you questioning my honesty and integrity?’
It’s not fair
‘Polly presented the same evidence and you allowed that’
‘you know I haven’t had time to collect any more evidence’
‘my supervisor hates me and wouldn’t let me use the report I wrote’
‘I wasn’t told that I had to have those minutes signed off’
‘that’s all the team evidence I’ve got – the team kept me out of everything and
wouldn’t tell me what was going on’.
‘most of the evidence I have is commercial in confidence and I’m not allowed to
use it’
Experience speaks
‘I’ve been doing this for 15 years – I must be able to do it’
‘I’ve been using this equipment for the last five years – what do you mean I’m out
of date?’
‘I’ve been a member of my professional association for 20 years – if that’s not
evidence, I don’t know what is’.
‘look, I’ve always done it this way, it’s always worked – why on earth would I
change?’
‘I know OH&S rules say that, but when you are as experienced as I am, you can
safely take short cuts’
‘How dare you question my degree, just because it is 25 years old’
I teach it
‘Of course I can do it: I teach it’
‘When you have been teaching as long as I have, you could apply these skills to
any situation anywhere’
‘I don’t see that disciplining student team members is any different from
disciplining work colleagues’
Avoiding the Pitfalls in Assessment
August 2012
Page 13
PARTICIPANT NOTES
I’m not given the opportunities
‘I know my Frontline Management evidence is only at Level IV, but I don’t see
how I can be expected to produce Level V evidence, when I’m always
overlooked when it comes to acting up in a job’
Looking for an out
‘I refuse to jump any more of these hurdles you keep putting up – my evidence
speaks for itself – I’m going to the union’
‘I want another, more experienced assessor’
‘I’m going to appeal’
Taken from the DET workshop Assessment Evidence 2003
Your notes:
Avoiding the Pitfalls in Assessment
August 2012
Page 14
PARTICIPANT NOTES
Communication strategies
Assessors confronted with declarations and accusations such as these need
effective communication strategies to defuse the situation and return to positive
mode. Strategies could include:

Avoiding being drawn into an argument

Using ‘I’ messages to explain what is required (eg: as an assessor, I have to
produce actual evidence to support my decision; I cannot rely on ‘word
of mouth information)

Repeat positive suggestions to engage candidate in more positive
discussion

Reassuring the candidate (eg: Of course I believe you: we just need some
evidence to back us up)

Advising the candidate about how to strengthen their evidence

Suggesting options for additional supplementary evidence, with examples

Where appropriate, follow up concerns the candidate may have about
their access to evidence records

Acknowledging the value of experience and helping the candidate
translate it into evidence

Responding positively rather than defensively (eg: We can certainly
consult another assessor – or the union representative – the union is very
supportive of this process)
Communicating positively with candidates about appropriate assessment
evidence enhances the evidence gathering process and facilitates evaluation of
evidence.
Decision making
Despite all the preparation, evidence gathering and review, assessors find making the
decision a challenging and sometimes lonely task. Many worry that they “still won’t
get it right”.
Below are some tips to reassure you as an assessor, and the judgements you make








Have good quality evidence, be able to explain it and show where there are
gaps(if necessary)
Be able to explain what is not quality evidence
Work closely with other assessors and technical experts
Call in another assessor when necessary
Provide honest feedback to the candidate throughout the assessment
process, so that the final decision does not come as a surprise
Be able to explain that “not yet competent” is not a final result and that
positive outcomes are still possible
Avoid appeals by helping the candidate prepare an action plan for
improvement
Accept that it is impossible to be right every time
Avoiding the Pitfalls in Assessment
August 2012
Page 15
PARTICIPANT NOTES

Know that the organisation for which you work has strategies in place for
reviewing, comparing, and evaluating the assessment process, tools and
evidence contributing to judgements made
All this said, in the end, the assessor’s decision will stand or fall on the evidence that
supports it. If assessors gather and analyse quality evidence in accordance with
approved assessment policy and procedures, their processes will withstand scrutiny.
Avoiding the Pitfalls in Assessment
August 2012
Page 16
PARTICIPANT NOTES
Avoiding pitfalls
Based on our discussions what would you consider would be some
‘good practice’ tips to avoid pitfalls?
Avoiding the Pitfalls in Assessment
August 2012
Page 17
PARTICIPANT NOTES
Attachments
Example Quality Evidence Checklist
Characteristics of Quality
E vi d e n c e
Does Evidence
M e et t h i s
Characteristic?
Yes/No
Comments – why evidence is or
isn’t adequate or appropriate
Appropriate to AQF level
Includes critical aspects of
evidence
Incorporates underpinning
knowledge and skills
Incorporates employability
skills
Demonstrates task skills
Demonstrates task
management skills
Demonstrates
contingency
management skills
Demonstrates job or role
environment skills
Provides a picture of
consistent performance
over time
Valid
Sufficient
Current
Authentic
Avoiding the Pitfalls in Assessment
August 2012
Page 18
PARTICIPANT NOTES
Level
Summary
Level 1
Level 2
Level 3
Level 4
Level 5
Graduates at this level will have
knowledge and skills for initial
work, community involvement
and/or further learning
Graduates at this level will have
knowledge and skills for work in
a defined context and/or further
learning
Graduates at this level will have
theoretical and practical
knowledge and skills for work
and/or further learning
Graduates at this level will have
specialised knowledge and skills
for skilled/paraprofessional work
and/or further learning
Knowledge
Graduates at this level will have
foundational knowledge for
everyday life, further learning
and preparation for initial work
Skills
Graduates at this level will have
foundational cognitive, technical
and communication skills t
undertake defined routine
Graduates at this level will have
basic factual, technical and
procedural knowledge of a
defined area of work and
learning
Graduates at this level will have
basic cognitive, technical and
communication skills to apply
appropriate methods, tools,
materials and readily available
informati
Graduates at this level will have
factual, technical, procedural
and some theoretical knowledge
of a specific area of work and
learning
Graduates at this level will have
a range of cognitive, technical
and communication skills to
select and apply a specialised
range of methods, tools,
mate
Graduates at this level will have
theoretical and practical
knowledge and skills for
specialised/ and/or skilled work
and/or further learning
Graduates at this level will have
broad factual, technical and
some theoretical knowledge of a
specific area or a broad field of
work and learning
Graduates at this level will have
a broad range of cognitive,
technical and communication
skills to select and apply a range
of methods, tools, materials and
simple issues and problems
Graduates at this level will have
technical and theoretical
knowledge in a specific area or a
broad field of work and learning
Graduates at this level will have
a broad range of cognitive,
technical and communication
skills to select and apply
analyse information to complete
routine and non-routine activities
solutions to a limited range of
predictable problems
provide and transmit solutions to
predictable and sometimes
unpredictable problems
and transmit solutions to
Application of
knowledge
and skills
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy in highly
structured and stable contexts
and within narrow parameters
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy and
limited judgement in structured
and stable contexts and within
narrow parameters
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy and
judgement and to take limited
responsibility in known and
stable contexts within
established parameters
to a variety of predictable and
sometimes unpredictable
problems
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy,
judgement and limited
responsibility in known or
changing contexts and within
established parameters
Application of
knowledge
and skills
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy,
judgement and defined
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy, welldeveloped judgement and
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy, welldeveloped judgement,
adaptability and responsibility as
a practitioner or learner
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy, expert
judgement, adaptability and
responsibility as a practitioner or
learner
require self-directed work and
broad parameters to provide
specialist advice and functions
parameters to provide specialist
advice and functions
Taken from AQF Implementation Handbook 2011
Avoiding the Pitfalls in Assessment
August 2012
Page 19
transmit information and skills to
others
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy,
judgement and defined
responsibility in known or
changing contexts and within
broad but established
parameters
Graduates at this level will apply
knowledge and skills to
demonstrate autonomy,
authoritative judgement,
adaptability and responsibility as
an expert and leading
practitioner or scholar