Assessment Validation –
Why and How
Tony Auciello 18th October 2012
Ambition in Action
Session objective
/ Gain a common understanding with regard to
validation requirements for the VET sector and
Sydney Institute
Assessment validation
Assessment Design
Competency Mapping
www.sit.nsw.edu.au
Ambition in Action
What is assessment validation?
Process of comparison and evaluation of
assessment tools/processes against competency
standard/s and bench marks to ensure that
assessment tools meet the Principles of Assessment
and the Rules of Evidence.
www.sit.nsw.edu.au
Ambition in Action
Assessment Validation Activities
/ Assessment Validation activities involve teachers
developing a shared interpretation of competency
standards and comparing assessment methods
and tools. This process is to encourage dialogue
between teachers and with industry about
assessment practice.
www.sit.nsw.edu.au
Ambition in Action
Teachers Role in Assessment Validation
/ Teachers are required to participate in assessment
validation activities and record the changes that
they make to assessment tools and activities.
www.sit.nsw.edu.au
Ambition in Action
Assessment Validation Procedure
/ Assessment Validation involves planning
assessment validation activities on an annual
basis, conducting assessment validation and
improving assessment strategies, processes, tools
and practices in a continuous improvement cycle
using a risk management approach
www.sit.nsw.edu.au
Ambition in Action
NVR Standards for the operation of RTOs
The new Standards for nationally regulated RTOs (the SNRs), formerly the AQTF
standards, require RTOs to systematically validate assessments.
15.5
Assessment including Recognition of Prior Learning (RPL):
(a) meets the requirements of the relevant Training Package or
VET accredited course; and
(b) is conducted in accordance with the principles of
assessment and the rules of evidence; and
(c) meets workplace and, where relevant, regulatory
requirements; and
(d) is systematically validated.
www.sit.nsw.edu.au
Ambition in Action
Industry Engagement about Assessment
/ The Australian Vocational Education and Training system is
“industry-led”. Industry engagement is built into the VET
system at a number of levels.
For example:
/ Training Package Development: Industry Skills Councils
establish industry led committees to ensure that Training
Packages are developed and amended to reflect the needs
of industry.
www.sit.nsw.edu.au
Ambition in Action
NVR Standards for the operation of RTOs
/ The new Standards for nationally regulated RTOs (the SNRs), formerly the
AQTF standards, require RTOs to engage effectively with industry in the
preparation of their delivery and assessment strategies, and the development
and validation of assessment tools.
Standard 15.3 states:
/ Staff, facilities, equipment and training and assessment materials .... are
consistent with the requirements of the Training Package or VET accredited
course and ... training and assessment strategies and are developed through
effective consultation with industry.
Standard 16.4 states:
/ Employers and other parties who contribute to each learner’s training and
assessment are engaged in the development, delivery and monitoring of training
and assessment.
www.sit.nsw.edu.au
Ambition in Action
So who is “Industry”?
/ The former National Quality Council (now known as
National Skills Standards Council - NSSC) describes
industry as “including, but not limited to, business owners
and employers and their representative bodies, employees
and unions, and their representative bodies”. (NQC Training
Package Glossary, 2009). In our context, industry also
includes community organisations, government departments
and other RTOs.
www.sit.nsw.edu.au
Ambition in Action
What are the advantages of validation?
/ Consistency of interpretation of competency
standards and benchmarks
/ Continuous improvement
/ Builds confidence in VET products
/ Customer service
/ Compliance
www.sit.nsw.edu.au
Ambition in Action
What do we validate?
/
/
/
/
assessment methods
assessment tools
assessment procedures
assessment decisions
High Risk Units first
www.sit.nsw.edu.au
Ambition in Action
Underpinning Knowledge
When Validating
/
/
/
/
/
Principles of Assessment
Rules of Evidence
Employability skills
AQF standards
Training Package/units competence
www.sit.nsw.edu.au
Ambition in Action
Principles of Assessment
/ Validity - The process assesses what it claims to assess. Validity requires that:
/ assessment against the units of competency
/ assessment of knowledge and skills must be integrated with their practical application
/ judgement of competence must be based on sufficient evidence.
/ Flexibility – To be flexible, assessment should: reflect the candidate’s needs,
provide for recognition of competencies no matter how, where or when they have been
acquired, draw on a range of methods appropriate to the context, competency and the
candidate.
/ Fairness - Fairness in assessment requires: consideration of the individual
candidate’s needs and characteristics and any reasonable adjustments that should be
applied.
/ Reliability - Refers to the consistency of the interpretation of evidence and the
consistency of assessment outcomes. Reliability requires a standard benchmark of
assessor competence and relevant vocational competence.
www.sit.nsw.edu.au
Ambition in Action
Rules of Evidence
/ Validity - The process assesses what it claims to assess. Validity requires that:
/ assessment against the units of competency
/ assessment of knowledge and skills must be integrated with their practical application
/ judgement of competence must be based on sufficient evidence.
/ Currency - Relates to the age of collected evidence. Competency requires
demonstration of current performance.
/ Sufficiency - Relates to the amount of evidence collected.
/ Authenticity - Relates to ensuring the evidence is from or of the candidate and
not another person.
www.sit.nsw.edu.au
Ambition in Action
Employability skills
/ Employability skills are the non-technical skills and
knowledge necessary for effective participation in
the workforce, and can include skills such as
communication, self-management, problem solving
and teamwork. They are also sometimes referred
to as generic skills, capabilities, enabling skills or
key competencies.
www.sit.nsw.edu.au
Ambition in Action
AQF Qualifications
/ The Australian Qualifications Framework (AQF) is
the national policy for regulated qualifications in
Australian education and training. It incorporates
the qualifications from each education and training
sector into a single comprehensive national
qualifications framework.
www.sit.nsw.edu.au
Ambition in Action
AQF Qualifications
www.sit.nsw.edu.au
Ambition in Action
The future of assessment
/ all new training packages assessments work-based
or simulated environment
/ COAG directions external validation
www.sit.nsw.edu.au
Ambition in Action
Clustering
/ Clustering units that combine to form an integrated
work activity/task.
/ Clustering generic elements across units
/ Clustering common themes across units
www.sit.nsw.edu.au
Ambition in Action
Assessment Design
Visualise the competency being performed in
the workplace
Locate Benchmarks
Identify Evidence Requirements
Identify Assessment Methods
Design Tools (instrument +instruction) and
map
www.sit.nsw.edu.au
Ambition in Action
Tips - designing quality assessments
Visualise a competent person in the workplace
www.sit.nsw.edu.au
Ambition in Action
Dimensions of competency
/ Task skills
Performing the task/job to the required standard.
/ Task management skills (variables)
Able to do more than one thing at a time and managing the
tasks correctly.
/ Contingency management skills
Responding appropriately to irregularities and breakdowns
in routine within a job or workplace.
/ Job/role environment skills (outcomes)
Able to deal with the responsibilities and expectations of the
work environment.
www.sit.nsw.edu.au
Ambition in Action
Examples Assessment methods
/
/
/
/
/
/
/
/
Observation
Questioning
Portfolios
Work samples
Third party feedback
Simulations, role plays
Presentations
Self assessment
www.sit.nsw.edu.au
Ambition in Action
Mapping
What is mapping?
Why do you need to do this?
The Steps in Mapping.
1. Analyse the Unit of competency
2. Select the assessment tool
3. Map the components of the UOC to the assessment
tools chosen
www.sit.nsw.edu.au
Ambition in Action
Analyse the Unit of Competence
The following diagram shows the range of information taken into account at this stage of the
development process:
www.sit.nsw.edu.au
Ambition in Action
How to identify assessment methods or
evidence gathering techniques
www.sit.nsw.edu.au
Ambition in Action
Sample Mapping Document
www.sit.nsw.edu.au
Ambition in Action
www.sit.nsw.edu.au
Assessment
Task
Parts of Unit Assessed
E=element, PC = perf
criteria
1.3, 1.4, 1.5
2.2, 2.3
3.2, 3.3, 3.4,
AmbitionEvidence
in Action
Part of Unit
Assessed
Guide
S =Skills and
K=knowledge
CA=
K1-K4
CA2
K5-K6
CA1
Critical
Aspects
Simulated case
study and
written or
verbal
questions.
E1 PC 1.2,
E2 PC 2.1,
E3 PC 3.1,
3.5, 3.6
E4 PC 4.3,
E5 PC 5.1,
Workplace
observation or
simulated
observation
E2 PC 2.1, 2.2, 2.3
E3 PC 3.1, 3.2, 3.3, 3.4
E4 PC 4.1, 4.2, 4.4, 4.5
K1-K6
CA1
S1-s4
CA2
www.sit.nsw.edu.au
Questioning
E1 PC 1.2, 1.3, 1.4, 1.5
K1-K4
CA2
4.4
5.2
Ambition in Action
Streamlining
/ Streamlining / Refining Assessment
Validation
Assessment Validation Record
Assessment Validation Checklist
This checklist is to be used as a guide when validating assessments. This checklist should be used as part of the
assessment validation meeting. This document should be kept as part of the record of evidence of assessment validation.
Faculty name:
Section name:
Qualification name and number:
Course No:
This form is used for validating assessments. This coversheet is to be attached to each validated assessment and stored in appropriate faculty/section team share folder or
dedicated local storage location. Where modification is required a revised assessment tool is also to be attached.
Faculty name:
Section name:
Qualification name and number:
Course No:
Training package name and number:
Training package name and number:
Units (List below individual Units or Clusters of Units to be validated both core & elective )
Unit name and number:
Units (name and
number)
Pre Assessment
Use this checklist as part of an assessment validation meeting to evaluate assessment tools, tasks or strategies that you
have developed
Learning outcomes
/elements
(from Training package)
/ less forms
Assessment Activity
The evidence requested is benchmarked against the current Training
Package or Curriculum Document
The assessment task is based on current workplace activities and
contexts (industry/ legislative currency)
The assessments is fair, flexible, valid and reliable
The level of difficulty is appropriate to the outcomes (performance
criteria) being assessed and sufficient time is allocated
The instructions to the assessors are clear, concise and accurate to
ensure consistent, reliable and fair results for all candidates
The instructions to the candidates are clear and sufficient, including
task requirements, time allowed and additional aids allowed
The assessment tool is suitable for use in other assessment contexts
eg, recognition, trainees, workplace
Reasonable adjustment can be made to the assessment strategy to
allow for candidates with special needs
The assessment tool documentation includes a cover page with
Sydney Institute logo, section name, course name/no, unit/module
names/numbers and assessment event name
The assessment is written in plain English
Version control policy has been followed
Comments:
www.sit.nsw.edu.au
Yes
Category
of
Validation
Assessment validation outcome
Modification
required
Completion
date
Select
1
Principles of assessment are required to ensure quality outcomes. Assessments should be fair, flexible, valid and reliable
as follows:
Validity means that the evidence relates to the unit competency, addresses essential skills and knowledge, dimensions of
competency and employability skills.
Reliability means that the assessment tool and process will produce consistent outcomes when applied by a range of
assessors in a range of contexts.
Fairness means that the assessment will not disadvantage any person and will take into account the characteristics of the
person being assessed.
Flexibility means that the assessment tool and process allows for assessment in a range of assessment contexts.
2
3
Comments
Comments:
Faculty Assessment Validation Team:
LTV (Lead Team validator)
Position:
name:
TVM 1 (Team Validation Member)
Position:
name:
TVM 2 (Team Validation Member)
name:
Position:
Validator category and organisation name: Select
Signature:
Date
Validator category and organisation name: Select
Signature:
Date
Validator category and organisation name: Select
Signature:
Date
Location of
documentation
Ambition in Action
Under Review
www.sit.nsw.edu.au
Ambition in Action
For more information follow the links
/ Competency Based Assessment Publications
/ Training Packages @ Work
/ Training.gov.au
www.sit.nsw.edu.au