TVET Australia
Assessment, validation and
moderation
A power point presentation developed by the NQC to support
information sessions on assessment, validation and moderation
Contact
NQC Secretariat
TVET Australia
Level 22/ 390 St Kilda Road Melbourne Vic 3004
Telephone: +61 3 9832 8100
Email: nqc.secretariat@tvetaustralia.com.au
Web: www.nqc.tvetaustralia.com.au
Disclaimer
This work has been produced on behalf of the National Quality
Council with funding provided through the Australian Government
Department of Education, Employment and Workplace Relations and
state and territory governments. The views expressed in this work
are not necessarily those of the Australian Government or state and
territory governments
Acknowledgement
This presentation was designed to support the interactive information
sessions that formed part of the NQC’s communication and
dissemination strategy: NQC products: validation and communication.
Reports and materials which focus on validation and moderation may
be downloaded from the NQC website at
http:www.nqwc.tvetaustralia.com.au/nqc_publications
This work was produced for the National Quality Council by
Andrea Bateman, Quorum QA Australia Pty Ltd
Chloe Dyson, Quorum QA Australia Pty Ltd
Quality of assessment
Setting the Scene
 Concerns about the quality of assessments and
comparability of standards across the VET
sector
 OECD (2008) Reviews of VET (Australia)
 NQC (2008) Industry Expectations of VET
 Service Skills SA (2010) VETiS Project
Today’s workshop




Assessment
Developing Assessment Tools
Competency Mapping
Simulated Assessment
 Engaging industry
 Assessment Quality Management Framework
 Validation and moderation
 System considerations
 Diverse settings
NQC resources
• Guide for Developing Assessment Tools
• Assessment Facts Sheets
• Simulated Assessment
• Making Assessment Decisions
• Peer Assessment and Feedback
• Quality Assuring Assessment Tools
• Assessor Partnerships
• Systematic validation
• Assessor Guide: Validation and Moderation
Session 1: What is assessment?
 Purposeful process of systematically gathering,
interpreting, recording and communicating to
stakeholders, information on student performance.
Assessment Purposes
 Evaluative
 designed to provide information to evaluate institutions
and curriculum/standards – primary purpose is
accountability
 Diagnostic
 Produce information about the candidate’s learning
 Formative
 Produce evidence concerning how and where
improvements in learning and competency acquisition
are required
 Summative
 Used to certify or recognise candidate achievement or
potential
Assessment Purposes
 Assessment for learning occurs when teachers use inferences
about student progress to inform their teaching (formative)
 Assessment as learning occurs when students reflect on and
monitor their progress to inform their future learning goals
(formative)
 Assessment of learning occurs when teachers use evidence of
student learning to make judgements on student achievement
against goals and standards (summative)

http://www.education.vic.gov.au/studentlearning/assessment/preptoyear10/default.htm
Session 2: Developing Assessment
Tools
NQC Products
• Guide for Developing Assessment Tools
• Assessment Facts Sheets
• Simulated Assessment
• Making Assessment Decisions
• Peer Assessment and Feedback
• Quality Assuring Assessment Tools
• Assessor Guide: Validation and Moderation
http://www.nqc.tvetaustralia.com.au/nqc_publications
Impact
Changes to definitions within the NQC publications
 AQTF 2010 User Guide documentation; and the
 Training Package Development Handbook
Reliability
Validity
Assessment tool
Validation
Moderation
Key Stages – developing assessment tools

identify and describe the purposes for the assessment

identify the assessment information that can be used as evidence of
competence/learning

identify a range of possible methods that might be used to collect
assessment information

define the contexts for interpreting assessment information in
ways that are meaningful for both assessor and candidate

determine the decision making rules

define procedures for coding and recording assessment information

identify stakeholders in the assessment and define their reporting
needs.
Essential Characteristics Assessment Tool
An assessment tool includes the following components:
 The context and conditions for the assessment
 The tasks to be administered to the candidate
 An outline of the evidence to be gathered from the
candidate
 The evidence criteria used to judge the quality of
performance (i.e., the assessment decision making
rules); as well as the
 The administration, recording and reporting
requirements.
Ideal Characteristics













The context
Competency mapping
The information to be provided to the candidate
The evidence to be collected from the candidate
Decision making rules
Range and conditions
Materials/resources required
Assessor intervention
Reasonable adjustments
Validity evidence
Reliability evidence
Recording requirements
Reporting Requirements
Competency Mapping
 The components of the Unit(s) of Competency that the
tool should cover should be described. This could be as
simple as a mapping exercise between the components
within a task (eg each structured interview question) and
components within a Unit or cluster of Units of
Competency. The mapping will help determine the
sufficiency of the evidence to be collected as well as the
content validity.
Advice regarding competency mapping can be found in the NQC Assessor Guide:
Validation and Moderation
Competency Mapping
Recording
14.
13.
feedback
Reasonable
adjustment
Roles
9.
12.
Resources
8.
Timelines
tools
7.
11.
methods
6.
Appeals
information
5.
10.
Purpose
4.
2.
1.
3.
Integrate
Plan Assessment Activities and Processes
Specify
TAEASS401A
characteristics
Workplace Project: 14 tasks to be performed by the candidate
Elements

Determine
Assessme
nt
Approach



Prepare
the
Assessme
nt Plan



Develop
Assessme
nt
Instrumen
ts

















Required skills




Cognitive
Interperso
nal skills
Research
and
evaluation
skills
Communi
cation
skills
Interperso
nal skills




















Component of Unit(s) of Competency
BSBITU201A Produce simple word processed documents
Methods
Elements
Evidence Guide
Elements 1.1
Required Skill and
Knowledge
RS3, RK3
1
Practical demonstration
2
Third party report
Elements 1 – 3
RS1-5, RK 1-4
CA1, CA2
3
Workplace documents
Elements 1-3 especially
1.1, 2, 3
RS1-5, RK 1-4
CA2
Competency Mapping:
Steps in the process
 Step 1:
 Step 2:
 Step 3:
Unpack the unit of competency to
identify its critical components.
For each assessment method, list the
tasks to be performed by the
candidate.
For each assessment method, map
the critical components of the unit
to each assessment task.
Refer to NQC Assessor Guide: Validation and Moderation
Level of specificity in mapping –
Risk Assessment
Risk can be determined by consideration of:
 Safety (eg potential danger to clients from an incorrect
judgement)
 Purpose and use of the outcomes (eg selection
purposes)
 Human capacity (eg level of expertise and experience
of the assessors)
 Contextual (eg changes in technology, workplace
processes, legislation, licensing requirements and/or
training packages)
Decision Making Rules
 The rules to be used to:
 Check the quality of the evidence (i.e. the
rules of evidence)
 Judge how well the candidate performed
on the task according to the standard
expected
 Synthesise evidence from multiple
sources to make an overall judgement
Additional advice – refer to Fact Sheet 2
Reasonable Adjustments
 This section of the assessment tool should
describe the guidelines for making reasonable
adjustments to the way in which evidence of
performance is gathered without altering the
expected performance standards (as outlined in
the decision making rules).
Simulated assessment
 For the purposes of assessment, a simulated workplace is one
in which all of the required skills are performed with respect
to the provision of paid services to an employer or the public
can be demonstrated as though the business was actually
operating.
 In order to be valid and reliable, the simulation must closely
resemble what occurs in a real work environment.
 The simulated workplace should involve a range of activities
that reflect real work experience. It should allow the
performance of all of the required skills and demonstration of
the required knowledge.
Ref: AQTF definition (refer to Activity Handout), Assessment Fact Sheet 1
Activity 1: Engaging Industry
 In your groups discuss what input employers (you might wish
to specify a vocational area) could provide to develop valid
assessment tools and processes.
 For the following scenarios, note down 2/3 questions you
could ask employers and how the responses will inform the
development or review of assessment tools and/or processes.
Relevant NQC support materials:
•Industry Enterprise & RTO Partnership
•Assessment Fact Sheets: Assessor Partnerships
•Assessor Guide:Validation and Moderation
Activity 2: Self Assessment
 In groups of 3, review the assessment tool using the
self assessment checklist from the NQC (2009)
Implementation Guide (Template A.1, p. 45).
 Identify any gaps in the tool?
 Discuss the pros and cons of including such additional
information within the tool?
Tool Review
 Has clear, documented evidence of the procedures for collecting, synthesising,
judging and recording outcomes (i.e., to help improve the consistency of
assessments across assessors [inter-rater reliability]).
 Has evidence of content validity (i.e., whether the assessment task(s) as a
whole, represents the full range of knowledge and skills specified within the
Unit(s) of competency.
 Reflect work-based contexts, specific enterprise language and job-tasks and
meets industry requirements (i.e., face validity).
 Adheres to the literacy and numeracy requirements of the Unit(s) of
Competency (construct validity).
 Has been designed to assess a variety of evidence over time and contexts
(predictive validity).
 Has been designed to minimise the influence of extraneous factors (i.e.,
factors that are not related to the unit of competency) on candidate
performance (construct validity).
Tool Review
 Has clear decision making rules to ensure consistency of judgements across
assessors (inter-rater reliability) as well as consistency of judgements within an
assessor (intra-rater reliability).
 Has a clear instruction on how to synthesise multiple sources of evidence to
make an overall judgement of performance (inter-rater reliability).
 Has evidence that the principles of fairness and flexibility have been adhered
to.
 Has been designed to produce sufficient, current and authentic evidence.
 Is appropriate in terms of the level of difficulty of the task(s) to be performed
in relation to the skills and knowledge specified within the relevant unit(s) of
Competency.
 Has outlined appropriate reasonable adjustments that could be made to the
gathering of assessment evidence for specific individuals and/or groups.
 Has adhered to the relevant organisation assessment policy.
Quality Checks
 Panel
 Pilot
 Trial
Refer to Fact Sheet 4, Quality assuring assessment tools
Session 3: Assessment Quality
Management
NQC Products
• Code of Professional Practice: Validation & Moderation
• Implementation Guide: Validation and Moderation
• Assessment Facts Sheets
• Quality Assuring Assessment Tools
• Systematic Validation
• Assessor Partnerships
• Assessor Guide: Validation and Moderation
http://www.nqc.tvetaustralia.com.au/nqc_publications
Validation
 Validation is a quality review process. It involves
checking that the assessment tool produced valid,
reliable, sufficient, current and authentic evidence to
enable reasonable judgements to be made as to
whether the requirements of the relevant aspects of
the Training Package or accredited course had been
met. It includes reviewing and making
recommendations for future improvements to the
assessment tool, process and/or outcomes.
NQC Implementation Guide: Validation and Moderation 2009
Outcomes of validation
Recommendations for future improvements
Context and conditions for the assessment
Task/s to be administered to the candidates
Administration instructions
Criteria used for judging the quality of performance (e.g. the
decision making rules, evidence requirements etc)
 Guidelines for making reasonable adjustments to the way in
which the evidence of performance was gathered to ensure
that the expected standard of performance specified within
the Unit(s) of Competency has not been altered
 Recording and reporting requirements.




Moderation
 Moderation is the process of bringing assessment
judgements and standards into alignment. It is a
process that ensures the same standards are applied
to all assessment results within the same Unit(s) of
Competency. It is an active process in the sense that
adjustments to assessor judgements are made to
overcome differences in the difficulty of the tool
and/or the severity of judgements.
NQC Implementation Guide: Validation and Moderation 2009
Outcomes of moderation
 Recommendations for future improvement and
adjustments to assessor judgements (if required) and
 Recommendations for improvement to the assessment
tools
 Adjusting the results of a specific cohort of candidates
prior to the finalisation of results and
 Requesting copies of final candidate assessment results
in accordance with recommended actions.
Validation vs Moderation
Features
Validation
Moderation
Assessment Quality
Management Type
Quality Review
Quality Control
Primary Purpose
Continuous improvement
Bring judgements and standards into
alignment.
Timing
On-going
Prior to the finalisation of candidate
results
Focus
Assessment tools; and
Candidate Evidence (including
assessor judgements) (desirable only)
Assessment tools, and; Candidate
Evidence, including assessor
judgements (mandatory)
Type of Approaches
Assessor Partnerships
Consensus Meetings
External (validators or panels)
Consensus Meetings
External (moderators or panels)
Statistical
Outcomes
Recommendations for future
improvements
Recommendations for improvements;
and
Adjustments to assessor judgements (if
required)
Types of Approaches - Statistical
 Limited to moderation
 Yet to be pursued at the national level in VET
 Requires some form of common assessment task at the national level
 Adjusts level and spread of RTO based assessments to match the level and
spread of the same candidates scores on a common assessment task
 Maintains RTO-based rank ordering but brings the distribution of scores
across groups of candidates into alignment
Strength
 Strongest form of quality control
Weakness
 Lacks face validity, may have limited content validity
Types of Approaches - External
 Types
 Site Visit Versus
 Central Agency
Strengths
 Offer authoritative interpretations of standards
 Improve consistency of standards across locations by identifying local bias
and/or misconceptions (if any)
 Educative
Weakness
 Expensive
 Less control than statistical
Types of Approaches – Assessor
Partnerships
 Validation only
 Informal, self-managed, collegial
 Small group of assessors
 May involve:
 Sharing, discussing and/or reviewing one another’s tools and/or
judgements
 Benefit
 Low costs, personally empowering, non-threatening
 May be easily organised
 Weakness
 Potential to reinforce misconceptions and mistakes
Ref: Implementation Guide, Assessment Fact Sheet 5
Types of Approaches - Consensus
 Typically involves reviewing their own & colleagues assessment tools and
judgements as a group
 Can occur within and/or across organisations
 Strength
 Professional development, networking, promotes collegiality and sharing
 Weakness
 Less quality control than external and statistical approaches as they can
also be influenced by local values and expectations
 Requires a culture of sharing
Systematic Validation (consensus)
Indicators
Yes/No
Action
Is there a plan for assessment validation (including validation of RPL
assessment) in place?
Does your plan:
•Determine the sample of units of competency to be validated over
a set period of time
•Provide dates for proposed validation activities
•Include details about who will participate in assessment validation,
including the Chair of consensus panels, if relevant
•Include a strategy to ensure that all relevant staff are involved
•Identify what processes and materials will be used for
implementing and recording the outcomes of assessment validation
Does your RTO have terms of reference in place to guide the work of
consensus panels?
Does your RTO have validation materials (policy, procedure, forms)
in place that cause participants to engage effectively in validation?
Does your RTO have a process for monitoring the action taken as a
result of validation?
Does your RTO have a process and plan in place for reviewing the
effectiveness of assessment validation?
Ref: Assessor Guide
System considerations
 What is the most appropriate approach to validation?
Condition
Suggested approach
Whenever my RTO conducts internal validation
few opportunities for improvement arise
Consider including external representation on
your validation panel
Our assessors are contractors and cannot come
to validation consensus meetings because my
RTO can’t afford to pay for their time and some
are located interstate
Consider establishing assessor validation
partnerships at your local level, but ensure that
improvements identified are recorded and fed
back to other assessors and formalised
Our RTO conducts high risk units related to
licensing, where the licensing authority has
mandated the use of assessment tools it provides
Consider consensus moderation, ideally with
external representation on your panel.
Our RTO is new and assessors do not have a lot
of experience
Consider inviting an external person with
expertise in assessment tool design to validation
consensus meetings
Assessment Quality Management
Quality Assurance
(Input approach)
Examples include:

Industry competency standards
as the benchmarks for
assessment

National assessment principles

Minimum qualifications for
assessors (i.e., TAE40110)

Development of a Professional
Code of Practice

Standardisation of reporting
formats

Assessment Guidelines and
Policy Documents

Benchmark examples of varying
levels of performances

Assessment tool banks

Common assessment tasks

Exemplar assessment tools

Panelling, Piloting and/or Trialling
of assessment tools.

Professional development
programs/workshops for
assessors
Quality Control
(Outcome approach)
Examples include:
 Moderation in which adjustments to
assessor judgements are made to
overcome differences in the difficulty
of the assessment tool and/or
severity of the judgement.
Quality Review
(Retrospective approach)
Examples Include:

Monitoring and auditing of registered
training organisations

Review and validation of assessment
tools, processes and outcomes to
identify future improvements.

Follow-up surveys with key
stakeholders (e.g., student destination
surveys, employer feedback on how
well the assessment outcomes
predicted workplace performance).
Quality management in diverse
settings
 Identified barriers:
 Structural (i.e., the organizational and resource aspects)
– financial, variations of definitions across key documents
 Process (i.e., the practices and activities that take place)
– rolling enrolments, partnering arrangements,
workloads
 Personal factors (i.e., the attitudinal, assessment literacy
and expectations of the key players).
 Strategies deployed by RTOs
Refer to Handout – Quality management processes in
diverse settings.
Activity 3: Assessment Quality
Management
Approach
Strategies implemented
 Activity 4: Assessment Quality Man
Quality assurance
Quality control
Quality management
Ref: Implementation Guide
Andrea Bateman
Director
Education Consultant
Quorum QA Australia Pty Ltd
Chloe Dyson
Director
Education Consultant
Quorum QA Australia Pty Ltd
Email: andrea@batemangiles.com.au
Phone: 0418 585 754
Email: chloed@alphalink.com.au
Phone: 0408124825
Principal author
Associate Professor Shelley Gillis
Deputy Director
Work-based Education Research Centre
Victoria University
Email: shelley.gillis@vu.edu.au
Phone: 0432 756 638