User Choice compliance audit information
session – Training and Assessment evidence
requirements webinar transcript
User Choice Webinar session – July 2015
Slide 1
Good morning everyone and welcome to the first of hopefully many User Choice information sessions.
My name is Drew Clark and I have been with the department about 21 years and have been conducting
compliance audits for about 13 years. I am joined today by Rod Harris who has been with the department
for over 18 years and brings a wealth of experience to the Market Quality audit team. Rod will be assisting
me today to not only keep me on track but to also answer any of your questions as we go through.
These information sessions are an opportunity for the department to provide our Pre-qualified suppliers
with clarification on specific areas of contention, areas where a significant number of non-compliances are
identified through the compliance audits conducted by Market Quality.
Slide 2
The department initially provided an expression of interest through its ‘Contractor Connector’ monthly
newsletter which was followed up with a survey issued to interested parties seeking feedback on the main
areas of concerns when it comes to meeting compliance against the User Choice funding program.
The results of this survey identified approximately 72% of respondents were seeking clarification on
Assessment retention including on-the-job verification and 54% were looking for further information
regarding the evidence requirements for training participation. So these areas will be what we focus on
today.
Slide 3
So the information we will cover today will be training participation and assessment.
Within training participation, we will run through:
• Predominant method of delivery
• What training evidence needs to be retained
• How much evidence needs to be retained and for how long
• Volume of learning
In relation to assessment, we will cover:
• What assessment evidence to be retained
• How much evidence needs to be to be retained and for how long
• What is on-the-job verification
• Hard copy evidence versus electronic evidence
Slide 4
The majority of the information we will run through today is located within one of three source documents:
PQS Agreement, User Choice 2010-16 Policy and the PQS Compliance Audit Evidence Guide (2010-16)
for the Further Education and Training Act 2014 or SRTO evidence guide.
I have included on the current screen a couple of very useful websites relating to the User Choice funding
program. The first is the location for the User Choice PQS Agreement along with the relevant User Choice
funding policies and the second is the location of the SRTO evidence guide.
While you unfortunately can’t click on the screen you will notice the web addresses popping into the chat
screen. As you would have been advised, this session is being recorded and will be available on the
department’s website along with the transcript and the PowerPoint slides.
While we will do our very best to address any questions you have today, anything we can’t get to will be
covered within a Frequently Asked Questions section on the department’s website when the webinar is
published.
Slide 5
So before we kick this session off, it would be great if everyone could please use the arrow icon to indicate
on the map of Queensland where their Head Office is located.
For any interstate attendees please indicate the interstate box.
Slide 6
As I mentioned previously, the two compliance areas we will go through today relate to training and
assessment and the evidence that must be retained to support any claims for payment submitted by a
supervising registered training organisation (or SRTO). And because training comes before assessment, I
thought we would start off with ‘Training Participation’.
The topics we will be cover relate to:
• What training evidence needs to be retained broken down between the different methods of delivery
o Face to face trainer led
o Online learning
o Other
• How much evidence does an SRTO need to keep and for how long
• And Volume of Learning
Slide 7
You will hear throughout this section of the webinar reference to the term “predominant method/mode of
delivery”.
As AVETMIS Standards only allow SRTOs to report one delivery identifier against any particular unit of
competency, the SRTO must determine which delivery mode best aligns to how the training of apprentices
and trainees (the students) is being conducted.
This is also important as the delivery identifier you report to the department could affect the amount of
funding you are entitled to claim.
Information regarding the predominant method of delivery could be something captured within the SRTOs
learning and assessment strategy or other documents and would highlight the range of training methods
and the amount of time the average student would require before they are ready to be assessed.
So to give you all a bit of practice, I have come up with the following scenario…
In the scenario outlined here, the SRTO trainer is traveling to the student’s workplace for the purpose
of providing face to face training which takes two hours which would align to AVETMISS Delivery
identifier ‘30’ – workbased delivery.
Version 1 – July 2015
The student is then told to work through the rest of the training materials accessed through a student
portal on the SRTO’s website. It is estimated this should take the average student about 8 hours to
complete. This would align with AVETMISS Delivery identifier ‘20’.
You would think this appears simple but you may be surprised by the number of SRTOs that get this wrong.
So what do you think would be the predominant method of delivery?
In the bottom right of your screen a poll has appeared – please take a few moments to complete –
Poll question: “What is the predominant method of delivery?”
Poll Options:
A. Delivery identifier ‘20’
B. Delivery identifier ‘30’
C. Other Delivery identifier
Poll Results:
Answers
Results
%
A
Delivery identifier ‘20’
14/32
44
B
Delivery identifier ‘30’
9/32
28
C
Other Delivery identifier
4/32
13
No Answer
6/32
19
Slide 8
With this in mind, could you please put circles around the location/s where you provide User Choice funded
training?
Slide 9
So now we look at our first type of delivery method which is SRTO led face to face training.
This delivery method would relate to either AVETMISS Delivery mode identifier ‘10’ or ‘30’, depending on
the training location, and is used where the SRTO is predominantly delivering face to face training to
students at its premises (code ‘10’) or at the student’s workplace (code ‘30’).
Evidence required supporting training participation as outlined within the SRTO evidence guide must
contain:
•
•
the name of the student/s
the unit or module code and/or name – this information is essential as User Choice funding is paid
against individual units of competency and where evidence of a student’s participation in training can’t
be validated, recovery of funding may occur.
Further, if SRTOs decide to cluster a number of units under a topic then some sort of mapping document
must be provided which demonstrates where the actual unit of competency is being delivered.
Slide 10
•
•
•
The date training occurred which is checked to verify the accuracy of AVETMISS data submitted to the
department and supports that training was conducted within the term of the student’s Training Contract
Location – again another essential piece of information that must be captured and will support the
AVETMISS postcode submitted and will subsequently validate any location loading claimed by the
SRTO.
Trainer’s signature and date so we know who has conducted the training on the SRTO’s behalf
And probably the most important bit of information that must be captured…
• The signature of the apprentice or trainee – this validates to the department that training has been
provided and the student has been engaged in learning.
Without this signature from the student, any withdrawn claims for payment submitted to the department will
not be validated and recovery of funding will occur.
Version 1 – July 2015
Slide 11
To retain sufficient evidence of training participation, the SRTO must develop a process to capture this
information irrespective of where the face to face training occurs or the number of students involved.
To help streamline the requirements to capture this evidence, the department will now accept the signature
of the student on the initial record of training participation for each individual unit of competency.
Subsequent records of training participation will only require the student’s name and the signature of the
trainer to validate attendance. This evidence will be sufficient to substantiate the submission of AVETMISS
data relating to withdrawn claims for payment.
Slide 12
The next delivery method relates to AVETMISS Delivery mode identifier ‘20’
This is used where the predominant method of delivery is aligned to students accessing their learning
through an electronic means -- such as training resources through the web based programs or via a
student portal on the SRTO’s webpage.
We see this type of delivery method in situations where the SRTO:
• is physically separated from the student by distance
• but is still able to provide guidance and feedback to the student through:
o online chats,
o emails or
o even over the phone.
More recently we have noted situations where SRTOs are undertaking an electronic training delivery
method at their own premises. This involves students attending a classroom environment to complete their
online learning and will often see students working through different units of competency or even different
qualifications at the same time but with access to a SRTO trainer (tutor) to help them through the unit
should assistance be required.
In the department’s view, this method of delivery, even though it is undertaken in a classroom is still
considered as AVETMISS Delivery identifier ‘20’ – Electronic based delivery.
Now you will notice the evidence requirements to support this type of delivery are significantly less than the
SRTO led face to face training covered previously.
This is because the use of online or electronic training resources is designed to capture information
regarding:
• every date and time the student logs into the system and
• what learning resources were accessed during that time.
Further, the unique login record allocated to each student is sufficient to address the requirements for the:
• name of the student as well as acting as a signature.
For this type of delivery, information regarding the location of training is not required as the department has
determined that all online delivery (which is AVETMISS Delivery identifier ‘20’) is to be reported as the
location the training has been co-ordinated from which will typically be the SRTO head office.
Slide 13
The last type of delivery method available would relate to AVETMISS Delivery identifier ‘40’ or Other
Delivery and is used where the predominant method of delivery does not align with one of the other
AVETMISS delivery modes.
Version 1 – July 2015
At times Market Quality has identified this type of delivery will be used where the student is undertaking
self-paced learning through the use of training materials sent by the SRTO. The evidence required to be
retained is similar to face to face training delivered by the SRTO. However, in this situation the SRTO
trainer is not present.
To ensure this information is captured you will need to rely on the student to record when they undertook
the training for you. It is also very important to note that the issuance of training materials does not
constitute training.
So the information to be captured must include:
• the name of the student/s again for obvious reasons
• the unit or module code and/or name – to support what the student has been working on and to validate
the submission of any withdrawn claims should the student decide not to continue.
Slide 14
•
•
•
the date training occurred
the Location of training to again support the accuracy of the AVETMISS postcode data submitted to the
department
the signature of the apprentice or trainee – to validate that they were engaged in learning.
And the big difference which is…
• the signature of the workplace supervisor or employer – this is used to validate that the workplace has
allowed the student the opportunity to complete training resources.
Slide 15
So now we have covered the range of delivery methods but the obvious question remains…
What do I keep?
Clause 9 of the PQS Agreement covers the requirements for records retention relating to User Choice
funding training and assessment and states “Records of each student’s participation in training (and
assessment) for each unit of competency including records of commencement of educational content,
attendance and progression”.
That pretty much says it all right there –
100% of units of competency for 100% of students - and it’s significantly different from ASQA retention
requirements.
So how long does an SRTO need to keep this evidence? The answer is six years from the end of the term.
The term being the period of time your organisation holds its PQS Agreement. The current User Choice
PQS Agreement has an end date of 30 June 2016.
Slide 16
The ASQA Users Guide to the Standards for Registered Training Organisations 2015 details your
obligations in relation to addressing the volume of learning requirements outlined by the Australian
Qualifications Framework.
For your information, we will put the ASQA website into the chat box for your reference
http://www.asqa.gov.au/users-guide-to-the-standards-for-registered-training-organisations-2015/about-thestandards-for-rtos/standard-one/clauses/clauses-1.1--1.4.html
The department strongly recommends you familiarises yourself with the standards and the requirements of
the AQF to ensure compliance with ASQA.
Version 1 – July 2015
Currently, volume of learning is not an area that Market Quality monitors as a part of the department’s
compliance audit process. However, this may be something the department reviews in the future as the
focus on quality training outcomes evolves.
Further, the Department of Education and Training currently has a memorandum of understanding with
ASQA and advice will be forwarded to the National VET Regulator should audits conducted by Market
Quality identify any instances of concern regarding the duration of training provided to apprentices and
trainees.
Slide 17
This draws to an end the section regarding training participation evidence. I really hope this was useful and
everyone got something out of it.
{Again, if you have any questions about the range of evidence for any of the delivery methods covered,
please use the chat function at the bottom right of your screen. Any questions that Rod doesn’t get to we
will do our best to address at the end of the webinar.}
Slide 18
The sections we are going to cover when it comes to assessment evidence are:
• What assessment evidence needs to be retained and what does an auditor look for?
• Hard evidence versus electronic or scanned evidence
• How much evidence has to be retained and for how long?
• What is on-the-job verification?
And because of the time we have allocated for this webinar, I will only be focussing on the evidence
retention requirements needed to validate Competency Achieved claims for payment or AVETMISS
Outcome identifier ‘20’
Slide 19
What must I keep? The SRTO must retain full and complete evidence covering all the assessment
instruments used to determine that student is competent at the individual unit level. This may include all of
these methods:
• Short answers questions
• Case studies
• Projects or assignments
• Verbal responses
• Role plays
• Observation / Practical demonstration
Or even other assessment gathering methods I haven’t mentioned.
Within the following situation an SRTO has decided only a couple of assessment methods would be used
to determine competence. These methods are:
• short answer questions including multiple choice and true/false questions
• two case studies prompting an essay style response to different scenarios and
• an observation of skills.
Down in the bottom right another poll will shortly appear – please have a read and indicate which answer
you believe is correct.
Poll question: “What assessment must be retained?”
Poll Options:
A) The coversheet that records competency achieved
B) All forms of assessment
C) A sample of one theory and practical assessment
D) None at all
Version 1 – July 2015
Poll Results:
Answers
Results
%
A
The coversheet that records competency achieved
2/32
9
B
All forms of assessment
29/32
91
C
A sample of 1 theory and practical assessment
1/32
3
D
None at all
0/32
0
No Answer
3/32
9
Slide 20
So what actually does the auditor look for when examining the student files…?
•
Full and complete assessment evidence
Market Quality auditors will firstly determine the range of assessment evidence based on information
contained within the apprentice/trainee’s training plan and/or the SRTO’s learning and assessment
strategy. The auditor will look for each nominated method of assessment and will examine this
evidence for indication the student has attempted all assessment requirements.
Instances were assessment has not been fully completed are likely to result in the recovery of funds.
Similarly, where one or more nominated method of assessment evidence is missing, for example – only
one of the two required case studies has been retained, or no evidence of practical skills has been
captured; a recovery of funds will occur.
Slide 21
•
Evidence assessment has been marked
You would be surprised the number of times Market Quality have seen assessment evidence fully
completed by the student but which contains no details or proof the assessment evidence has actually
been reviewed and marked as satisfactory by the assessor. The department has determined that the
signature of the assessor only on a coversheet is insufficient to support assessment has been
adequately reviewed and marked.
Slide 22
•
Signature of apprentice or trainee
To support the authenticity of the assessment, the apprentice/trainee must sign, at the very least, the
assessment coversheet verifying the completed assessment is their own work etc. Ideally, the student
should be signing each assessment instrument as additional proof that it was their work that was submitted
Slide 23
•
Competent at the unit level
The assessor first makes a determination or judgement that the student has satisfactorily addressed the
requirements of each individual assessment instrument. The culmination of satisfactory outcomes for each
assessment will then support the judgement of competence against the unit of competency.
For example, a student can’t be deemed competent at the unit level if they have satisfactorily completed
the theory assessment component but are unsatisfactory in addressing the practical skills assessment
Slide 24
•
Training package requirements (if applicable)
Now a Market Quality auditor will advise you as part of the opening interview that we are not there to
conduct an ASQA audit. This means the User Choice compliance audit will not breakdown the
assessment instrument to ensure that all the knowledge and skills requirements of the unit has been
addressed.
Version 1 – July 2015
However, if a training package unit of competency contains specific assessment requirements which
are often located within the critical aspects of assessment. The audit will investigate to ensure that
sufficient evidence has been retained that addresses the outcome of the unit.
A really good example would be:
Slide 25
SITHIND301 Work effectively in hospitality service
This is a core unit from the Certificate III in Hospitality Operations qualification which is a Queensland User
Choice funded traineeship. After all the information regarding Elements and Performance criteria and
required skills and knowledge.
The Critical aspects of assessment indicate “evidence of the ability to… integrate technical skills and
provide quality hospitality service to customers for a minimum of 36 complete service periods (shifts)”.
In this instance, Market Quality auditors would expect to sight evidence which supports the trainee has
demonstrated the range of hospitality skills over 36 shifts. This is often seen captured in a log book or
similar document and would detail:
•
•
•
the date and time of the shifts,
the duties during the individual shifts
signature of both the trainee and workplace supervisor
Slide 26
Another example from the Construction, Plumbing and Services training package
CPCCCA3003A Install flooring systems
The Critical aspects of assessment for this unit indicate among other things the requirement for the student
to…
• complete as a minimum one bearer and joist system to specific standards
• install a tongue and groove fitted strip flooring surface and an approved wet area floor system to a
bathroom area
• and install a system of similar size using either a sheet or strip platform system.
So in addition to the necessary theory assessment component, the unit indicates at least four assessments
need to occur to address the practical skills component. While we are not analysing the validity of
assessment, again as this is the role of ASQA, Market Quality will ensure that sufficient assessment has
been conducted and retained to verify claims for payment.
Please be aware that in cases where Market Quality identifies issues of missing assessment or
assessment which is demonstrably inadequate, in addition to a recovery of funds, Market Quality will also
provide a summary of the audit findings to ASQA.
Slide 27
This should be a nice and easy one by now… and to check that no one fell asleep I have our last poll
regarding retention of evidence which incorporates both training and assessment.
Poll question: “What is the term of retention of training participation and
assessment records?”
Poll Options:
E) Just for the appeal period
F) 12 months after assessment completed
G) Six years from the end of the term of the Agreement
H) Destroy the same day
Version 1 – July 2015
Poll Results:
Answers
Results
%
A
Just for the assessment appeal period
0/32
0
B
12 months after assessment completed
0/32
0
C
6 years from the end of the term of the Agreement
27/32
84
D
Destroy the same day
0/32
0
No Answer
5/32
16
That’s right, simply put; all assessment (and training) evidence needs to be retained for each student, for
each unit of competency for 6 years from the end of the term of the Agreement. If you think about it this
way, SRTOs that held a 2006-2010 User Choice contract still have until 1 July 2016 before they are able to
dispose of any training and assessment evidence.
Slide 28
What is On-the-job verification?
The main thing we need to remember about apprenticeships and traineeships is that it is an Employment
based training pathway which is distinctly different from fee for service qualifications undertaken via
institutional based training or even other departmental funded programs such as Certificate 3 Guarantee.
In relation to User Choice, there are two pieces to the assessment puzzle. Firstly the Formal assessment
conducted by SRTO, for example the short answer questions, assignments and role plays. And then
there’s the evidence which supports the judgement of workplace performance or the on-the-job verification
by the employer or supervisor.
Both pieces of the puzzle must be satisfactorily completed and retained as evidence by the SRTO prior to
the submission of claims for payment at the individual unit of competency level.
Evidence of the on-the-job verification can be captured one of two ways.
• Third party report completed and signed by the workplace supervisor
In relation to a third party report, SRTOs are advised a copy and paste from a unit’s Elements and
Performance criteria will be identified as a non-compliance and could possibility lead to a recovery of
funds and likely referral to ASQA. It is not up to the employer to breakdown and interpret units of
competency.
A third party report must outline a list of observable tasks that a competent student should be able to
demonstrate within the workplace to provide the employer or supervisor with a foundation against which
to make their judgement of workplace performance.
•
Alternatively retention of on-the-job verification may be captured within a copy of the relevant page from
the training record signed by all parties.
If this is your chosen method of retention when it comes to on-the-job verification evidence you need to
remember the training records is the property of the student and must remain with them at all times.
The risk SRTOs run is if the student exits the training contract for whatever reason and takes the
training record with them as the SRTO all your evidence of on-the-job verification has been lost.
Systems would need to be implemented to manage this risk.
Finally, Market Quality have been approached on a number of occasions regarding advice about the
sufficiency of using file notes to document telephone or face to face conversations with the employer
regarding the student’s workplace performance for the purpose of evidencing on-the-job verification. SRTO
have been advised this will not be accepted at audit and recovery of funding will occur if this is identified.
Slide 29
So who can sign off against the On-the-job verification?
Version 1 – July 2015
The on-the-job verification forms an integral part of the overall assessment process, and where the SRTO
uses a third party report (or similar document) the on-the-job verification must be completed by someone
who is authorised within the workplace to make this determination. This will typically be the employer or
qualified supervisor.
However, please note that the auditor will examine the completed Employer Resource Assessment for
each student to ensure the person or persons nominated as qualified supervisors on this document is the
same person providing the on-the-job verification. Therefore the Employer Resource Assessment must
capture the details for all supervisors, including the employer, who are qualified to make this judgement.
If it is identified the person signing the on-the-job verification is not noted as a qualified supervisor, the
SRTO will be required to provide further evidence that it has implemented a process of integrity to support
the on-the-job verification has been completed by someone qualified to make the appropriate judgements
against workplace performance. It would be unlikely that an auditor would accept the on-the-job verification
for an apprentice being signed off by the receptionist.
SRTO are also reminded that trainees cannot supervise other trainees who are currently
undertaking the same traineeship.
Slide 30
So how does an SRTO store all of this evidence?
In the end the decision remains yours as the SRTO to determine what works best for you. The department
does not outline how records are to be maintained, only what must be retained. Evidence may be fully kept
in hard copy in plastic sleeves within manila folders, retained fully online within a student management
system, or scanned and saved into PDF files or a combination of all three. It’s whatever works best for you.
If electronic or scanned evidence is going to be retained then the SRTO must ensure these records are full
and complete to be able to support claims for payment:
• Must be legible
• Electronically captured signatures from apprentices and trainees must be individually obtained at each
instance. Signatures from students must not be captured and stored and later used to copy and paste
signatures on the assessment items.
If this is found to have been occurred the department may seek the recovery of funds.
Slide 31
So the information we have covered today was training participation and assessment.
Within training participation, we ran through:
• Predominant method of delivery
• What training evidence needs to be retained
• How much evidence needs to be retained and for how long
• Volume of learning
In relation to assessment, we covered:
• What assessment evidence to be retained
• How much evidence needs to be to be retained and for how long
• What is on-the-job verification
• Hard copy evidence versus electronic evidence
Slide 32
I am happy to spend a few minutes at this point to address any general questions or comments about
assessment retention requirements.
Please use the raise hand icon to indicate you have a question to ask:
Version 1 – July 2015
•
•
•
please nothing specific to individual cases
please do not mention any names and
remember this session is being recorded for later publication.
Slide 33
Thank you very much for your participation in this User Choice compliance audit information session.
Please take the time to complete the short survey which will appear on your screen after you leave this
session.
Thank you again.
Version 1 – July 2015