SINGAPORE WORKFORCE SKILLS QUALIFICATIONS SYSTEM (WSQ)
GUIDE TO CONTINUOUS IMPROVEMENT REVIEW (CIR)
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 1
Contents at a Glance
Introduction
Content
Part I:
Overview of Quality Assurance Framework
Part II:
Continuous Improvement Review (CIR)
– Essential Elements of Review
Part III:
Roles and Responsibilities of Approved Training Organisation
(ATO)
Annex A : Steps-by-Steps Process for ATO
Annex B : Self-assessment Checklist
Annex C : 2 Year Plan
Annex D: CIR Presentation by ATO
Annex E : CIR Outcome Measures
Pg
4
8
17
23
28
35
37
39
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 2
Introduction
T
he field of adult training is about providing quality courses to cater to the learning
needs of the adult. A quality course would be able to enhance the learning
experience of the adult and contribute to the adult in more ways than we can ever
imagine. Some examples of the contribution would be 1) increase employability, 2)
improve skills set, and 3) innovate work processes.
However, as an Approved Training Organisation (ATO), providing quality courses
are just one of the many responsibilities you are expected to do. It is really a case of
simple logic. To provide quality output i.e. quality courses, you need to have quality
input.
What is quality input you might ask?
To answer the question, we strongly recommend you to explore this guide. We
assure you that it will be a meaningful learning journey, and you will be able to take
away more than you can imagine.
The bottom-line is that this guide serves more than just preparing your organisation
for the Continuous Improvement Review (CIR). It aims to guide your organisation in
establishing a strong quality Continuous Education and Training (CET) infrastructure
system within your organisation.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 3
Part I
Overview of
Quality Assurance Framework
In this part …
Part 1 deals with the WSQ quality assurance framework and the components
existing within the framework. On the application side, this part introduces the
actions required by any organisation embarking on the journey of becoming a WSQ
Approved Training Organisation (ATO). It is a brief introduction of the different
processes involved. The objective of Part 1 is to allow you to understand the journey
you have taken to becoming an ATO and examine the processes that you would
have gone through before the CIR.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 4
Part I
Overview of
Quality Assurance Framework
In This Part
Introduction to the Quality Assurance Framework – What it means to you
The Process within the Framework
The Concept of WSQ That You Just Have
to Know
If you are a WSQ ATO, the term WSQ would not be unfamiliar to you. From the
moment you have chosen to be a WSQ ATO, you should have a basic
understanding of WSQ. However, if it happens to slip your mind at the point you are
reading this guide, we are providing you with a quick note to refresh your memory.
The Singapore Workforce Skills Qualifications (WSQ) is a robust and integrated
continuing education and training system. Designed to build industry competencies,
it is founded on international best practices and validated by industries and
employers. WSQ caters to adult workers who have widely diverse training needs; it
offers a wide range of certification and qualification ranging from Certificate to
Graduate Diploma.
Quality assurance provides the backbone for the WSQ. A strong quality assurance
framework is important to build up public confidence in the WSQ.
So as a provider under the WSQ framework, you are recognised by the public to be
providing courses that are based on a robust quality CET system.
It is thus a joint responsibility between yourself and WDA to continuously contribute
and improve on the robustness and quality of the CET system.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 5
Mechanism to contribute and improve
WSQ quality
WDA has developed the quality assurance framework as a mechanism to uphold the
quality of WSQ. In addition, this framework serves to continuously improve the
quality of WSQ.
The quality assurance framework is made up of two processes:
i)
Pre-delivery Approval – There are 2 stages of accreditation under the
pre-delivery approval, 1) the organisation approval accreditation, and 2)
the course accreditation. The objectives of both accreditations are
similar; that is to ensure that organisation has the capability to delivery
courses that satisfy the competency requirements of the WSQ and is
accessible to workers through appropriate delivery modes and
resources. The process also ensures that the course will be delivered
by a quality training organisation using qualified adult educators. Upon
approval, the following status will be conferred:(a)
An organisation approved to deliver stipulated training
and/or assessment services under the WSQ is known as an
Approved Training Organisation (ATO);
Henceforth the term Approved Training Organisations or ATOs will
be used to refer to WSQ accredited organisations.
At this stage, we would assume that you have gone through the above mentioned
process and the 2 stages within the process. If you have not, we recommend you to
download a copy of our accreditation guide (www.wda.gov.sg) so as to better
understand the details behind the accreditation.
However, do not stop your reading of this guide. It is still relevant for you to
understand the types of mechanism WDA is adopting to ensure the quality of the
WSQ frameworks.
Moving on, the second process under the quality framework is none other than the
theme of this guide – Continuous Improvement Review (CIR).
ii)
Continuous Improvement Review (CIR) – Upon approval (organisation
and course), every ATO is to maintain and upkeep the standard of
delivery in training and assessment. In addition, every ATO is to pursue
continuous improvement to uplift their quality of design and delivery.
WDA will periodically validate the internal quality assurance system and
capability of each ATO under CIR.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 6
It is important to note you are not taking this journey alone. Throughout the journey,
WDA will assist ATO to develop their capabilities in designing and delivering of
quality training and assessment under WSQ.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 7
Part II
Continuous Improvement Review (CIR)
– Essential Elements of Review
In this part …
Here is where you learn about the essential criteria of review under the Continuous
Improvement Review (CIR) framework. We will begin by explain the purpose of the
CIR. From there, we will look in the areas which we will review during the CIR. Along
the way, you will find out about the specific components that we are evaluating under
each of the essential criteria.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 8
Part II
Continuous Improvement Review (CIR)
– Essential Elements of Review
In This Part
The Purpose of CIR
The Essential Elements of Review
Continuous Improvement Review
Continuous Improvement Review is one of the extremely important process in the
quality assurance framework that WDA puts in place to maintain and improve the
quality of WSQ. CIR enables WDA to discern the gaps in quality pertaining to WSQ.
As an ATO, you could use CIR as a quality check tool to improve your internal CET
structure. It is thus valuable to understand the criteria in the CIR system so as to
maximise the usage of it to improve your organisation’s capability and quality as an
ATO.
Why CIR?
The main objectives of CIR are:
i) Inculcate the practice of seeking continuous improvement in the delivery of
training and assessment;
ii) Identify best practices and role models amongst the ATO; and
iii) Nurture ATO to be WDA’s partners in providing quality continuing education
and training.
Therefore, CIR is really a process for both WDA and the ATO to critically reflect on
the practices adopted by the ATO. Such reflections provide opportunities for the ATO
to refine its CET system, and thus contributing to the overall improvement of WSQ.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 9
Essential Criteria of Review
All ATO is subjected to audit on the following essential criteria for CIR:
1. Course Quality Assurance System
i) System to ensure courseware and assessment are aligned to WSQ,
pedagogically sound and contextualised
ii) System to manage profiling of learners to ensure that the course(s) is fit for
purpose
iii) System to manage training activities
iv) System to manage assessment activities
2. Plan-Do-Check-Act (PDCA) System
i) System to ensure PDCA is implemented regularly for the
comprehensiveness of training, assessment and management system
3. Outcome Evaluation System
i) System to monitor and analyse L1 outcomes
ii) System to monitor and analyse L2 outcomes
iii) System to monitor and analyse L3 outcomes
iv) System to monitor and analyse L4 outcomes
4. Adult Educator Management System
i) HR system to ensure most qualified adult educators are recruited and
inducted
ii) HR system to monitor, appraise and manage performance of adult
educators
iii) HR system to train and develop adult educators
5. WSQ Related Administrative System
i) Learner support system
ii) System to ensure accurate and timely submission of assessment
results
iii) WSQ certificate printing and distribution system
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 10
iv) Maintain up to date profile in Skillsconnect
6. Viability of Organisation
i) Governance and management of organisation
ii) Financial health of organisation
7. Free from Breach of Terms and Conditions
i) Records of organisation free breach of WDA terms and conditions.
System-Evidence-Outcome Approach
WDA is adopting the system-evidence-outcome (SEO) approach in the evaluation of
the CIR criteria. The focus on the SEO approach is because there is strong
correlation between sound systems and the quality of outcomes. It has to be reiterated here that the continuation of your approval status/es is/are subjected to the
outcome of WDA’s scheduled CIR audit which would be graded, based on the
scoring matrices for S-E-O.
Definition of S-E-O
1. Systems: a set of interacting or interdependent entities forming an integrated
whole, giving a macro perspective. Components within the system generally have
interconnectivity i.e. functional and structural relationships.
2. Evidences: a thing or things helpful in forming a conclusion or judgment:, for
instance, the broken window was evidence that a burglary had taken place.
3. Outcomes: refers to the way something turns out; result; consequence.
To ensure that you are on the same page as us, we have provided an example of SE-O institute for courseware development. (Please see pg. 13 for the given example)
You have to note that in the given example, the system is represented by a process
flow chart. You can represent your system in many forms e.g. a standard operating
manual, a policy guide etc.
The key in developing a comprehensive system is how concise and systematic the
information is presented. To achieve this, you first have to set your organisation
SMART (Specific, Measurable, Agreed upon, Realistic and Time-based) targets
when drafting any system at the outset.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 11
Succinctly, each system devised by you should address the following 5 questions:
i)
When
ii)
Who
would be involved in the system; i.e., the personnel who is
directly involved in either the overseeing or implementing of the system
iii)
What
approach/tools would the system used; i.e., the specific
approach/tools that the system would be using in its course of
implementation.
iv)
Why
did ATO have this system; i.e., the rationale/objective of the
system.
v)
How
would the system be devised and implemented; i.e., effective
start date of system, frequency of system implementation.
would each process within the system be evaluated and follow
up on; i.e., any necessary follow up actions/evaluation are to be
addressed accordingly in the system.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 12
Example of S-E-O
System: Process flow for Courseware Development
Courseware
Development Team *
Industry Information, Market
Research, Course Evaluation
Forms, Companies’ Feedback
Conduct research and survey the market and
customers’ needs
Perform thorough study of the course objectives and assess
capabilities such as trainer’s skills, facilities, equipment
(Refer Note 1)
Costing
Perform simple costing to evaluate if the course is
profitable
Profitable
No
Case Closed
Yes
Courseware Developer
(Refer Note 2)
Develop the Course Programme Design and Overview,
Trainers’ Guide, Lesson Plan and Course Materials as
per WSQ standards (1st draft to be submitted within 1
month to the Courseware Development Team)
Courseware
Development Team
WSQ Programme Design and
Overview, Trainers’ Guide,
Lesson Plan,
Course Materials
Submit to Courseware Development Team for
approval
No
Approved by Team
* Team consists of Principal, Training
Manager and Trainers
Course Developer to rework
based on Team’s directions
(See Note 3)?
Yes
A
Continue on pg 2 of 2
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 13
Effective Date: 1 Jul 10
Pg 1 of 2
Doc No: XYZ-QA-01
Version : 1.0
XYZ Pte Ltd
Title : Course Development Process
A
Training Manager
Brief the appropriate trainers on the training concept
before trainers proceed to conduct the new course.
Proceed to conduct the course
Course
Evaluation
Form
(Trainee) – Form 1
Course
Evaluation
Form
(Trainer) – Form 2
Post-Course Profiling – Form 3
Perform design validation by analysing course
evaluation forms / interviewing trainees and conducting
post-course profiling for 3 sessions to gather
participants’ profiles/feedback on the new course
Trainers
Changes in course
design?
Yes
Follow up by Courseware Developer to
amend and update the relevant
documents
and
affected
staff
accordingly
No
Proceed with next course
Note 1
For any deficiencies of trainer’s
skills, suitable trainers shall be
sent for training or training could
be outsourced to qualified parttime trainers.
Training facilities shall be properly
planned.
Any
new
training
equipment/products required can
be bought, pending the costing
calculation.
Note 2
New course developers will have
to attend pre-development briefing
conducted
by
the
Course
Development Team. The minutes
of the briefing and the attendance
must be recorded and compiled for
future reference.
Note 3
For WSQ programmes, any new
courses or revisions done to
training
materials
must
be
submitted to WDA for approval
Effective Date: 1 Jul 10
Pg 2 of 2
Doc No: XYZ-QA-01
Version : 1.0
XYZ Pte Ltd
Title : Course Development Process
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 14
It is always useful to enhance each flowchart and all the supporting form/s with
document names, document numbers, effective dates and version control tracking
mechanisms.
As the scoring for the CIR criteria is based on 3 components – system, evidence and
outcome, having a comprehensive flowchart means that the ATO would score for
system.
The Evidence in this specific example would be the following:
- Industry Information, Market Research, Course Evaluation Forms, Companies’
Feedback, etc.
- Course Evaluation Form (Trainee) – Form 1, Course Evaluation Form (Trainer) –
Form 2, Post-Course Profiling – Form 3, etc.
The Outcome in this specific example would be the following:
- Good L1 score as trainees find the course relevant;
- Repeat corporate clients being served (as the organizations know what they are
getting if they are requesting for the same training the second time).
Please note that the under this specific example, the ATO has to devise Forms 1, 2
and 3 as well as document the Industry Information, Market Research, Companies’
Feedback, etc as evidence of deployment of the system so as to score points under
the evidence matrix.
Likewise, outcome/s would have to be presented as well-collated L1 scores by
course or trainer on a periodic (e.g., quarterly, half-yearly, etc) basis. While the list of
clients served, especially established corporate clients or walk-ins through words of
mouth, etc if available would be useful in securing points in the outcome matrix.
In addition, it is important to note that there should be
clear and concise documentation for all the criteria under
the S-E-O. Any S-E-O without documentation will affect the
CIR evaluation outcome.
You have come to the end of Part II of this guide. At this point, you should have a
general idea of the seven essential criteria of review in the CIR:
i) Course Quality Assurance System
ii) Plan-Do-Check-Act (PDCA) System
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 15
iii) Outcome Evaluation System
iv) Adult Educator Management System
v) WSQ Related Administrative System
vi) Viability of Organisation
vii) Free from Breach of Terms and Conditions
You should also have a general understanding of the S-E-0 approach adopted by
WDA in the evaluation of the essential criteria. However, if you still have doubts in
your mind, you can always email or drop a call to the officer-in-charge of your
organisation to seek their advice.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 16
Part III
Roles and Responsibilities
of ATO
In this part …
In the third part of the guide, we will be talking about the preparation requires by an
ATO/ATC/AAC when being notified of a CIR by WDA. We will talk about the things
that you are required to prepare after you have received the CIR notification from
WDA, and the things that you are required to do consistently as an ATO.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 17
Part III
Roles and Responsibilities
of an ATO
In This Part
The Roles and Responsibilities of an ATO
Scope of CIR
ATO Continuous Improvement Efforts
Outcomes of CIR
Fees and Feedback
The Roles and Responsibilities
Together We Play a Part
–
It is important to note that you do not journey alone when you are an ATO. WDA will
provide the necessary assistance to aid you along the way. Thus, before the CIR,
during the CIR, and even after the CIR, WDA seeks to work closely with you to
improve your overall quality as an ATO. Some of the aids that WDA would provide to
you in building you quality are:
i)
ii)
iii)
iv)
v)
Validate the internal quality assurance of ATOs
Build up the competencies of the ATOs through feedback
Issue CIR audit report for CIR audit conducted
Highlight ATO’s strengths and areas for improvement
Facilitate ATO’s participation in capability development initiatives and
workshops
However, as an ATO, you need to have the correct attitude in building up your own
quality. As the saying goes, “Your Attitude Determines Your Altitude”, you would
have to put in effort to improve your own ability and quality as an ATO. Some of the
things you are recommended to do constantly are:
i) Practice internal quality assurance
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 18
ii) Ensure Plan-Do-Check-Action (PDCA) system is in place and is
diligently making continuous improvements to meet the required quality
or for better quality, especially in areas where required by WDA from
previous CIR audits conducted
iii) Ensure quality of accredited courseware and adult educators are
consistent with actual training and/or assessment
iv) Ensure evidences of all systems implementation, and dates of training
and assessment are available for audit when requested
You have learned about the criteria of review and the S-E-0 that WDA used in CIR
evaluation. It is time to look into the activities that you are required to do on the day
of the CIR.
Scope of CIR
In part II of the guide, we have gone through the essential criteria of review, and the
expected responsibilities of an ATO. We will now move on to look into the specific
things you will be required to do when you received the CIR notification from WDA
and on the actual day of the CIR.
When you received the CIR notification email from WDA (one month before the CIR
date), you are required to response to the notification email within 5 workings days
providing the following:
i) A series of available dates and time for the CIR
ii) A training and/or assessment schedules
iii) A self assessment checklist (see Annex B)*
iv) A 2 year plan (see Annex C) (Applicable to only ATO offering public
courses)*
*This is applicable only for public ATOs and is to be provided within 2 weeks
from the date of the notification
Upon receiving your response with the attached documents, the office-in-charged of
your CIR will then get in touch with you to finalise on the CIR date and time based on
the series of dates and time provided by you. The officer may also request more
information based on the 2 attached documents that you are required to submit.
Once the date and time of the CIR is finalised, you can look forward to the actual
CIR date and prepare the necessary presentation and materials for the CIR.
According to the stated essential criteria in part II, you will be required to make a
presentation based on each criteria. The presentation is recommended to be kept
under 15mins (see Annex D for the breakdown of the presentation process). You
need to have the supporting evidences ready for verification during the CIR audit.
The review covers the following:
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 19
i) Showcase of all the implemented systems based on the essential
criteria of review e.g. relevant SOP manual
ii) Specific evidence of each implemented system, e.g. training
observation checklist with requirements clearly communicated to all
adult educators, or learners’ profiling checklist in determining
contextualisation of courseware, etc.
iii) Outcome of each implemented system, e.g. systematic documentation
of all CVs and qualifications of qualified adult educators (against
declaration submitted at accreditation), or assessment results with
analysis to reflect L1 and L2 outcomes, etc.
It is important that you prepare the relevant evidences and supporting documents in
advance before your scheduled CIR audit by WDA. The reason of the preparation is
really quite simple - you do not want to be running around getting the documents and
indirectly reflecting the possibility of an inadequacy in your administrative system.
Your Continuous Improvement Efforts
Specifically, ATO is required to practice continuous improvement in its ongoing
delivery of training and assessment under WSQ. Evidence of these efforts should be
available or demonstrated in the form of PDCA through any of the following:
i) Quality of design and delivery of courseware, e.g. regular reviews to
ensure currency, for continuous improvement, contextualisation to be
relevant to a particular industry or target audience etc;
ii) Quality of trainers, assessors and curriculum developers, e.g. training
and development plan and implementation, performance management
system etc.;
iii) Assessment Management System, e.g. regular reviews and
improvements made to internal quality assurance system particularly
for the management and conduct of assessments, ranging from how
assessments are designed and conducted to ensure the four principles
of assessments – reliability, fairness, validity and flexibility as well as
positive and negative features experienced in conducting the
assessments, and refinements made, assessment validation etc.;
iv) Administrative System (supporting training and assessments for
accurate and timely records submission to WDA and certificate
issuance to trainees);
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 20
v) Tracking of outcomes, e.g. end-of-course evaluation, attrition rate,
passing rate, surveys conducted etc.
You can refer to Annex A for the step-by-step of CIR for ATO to have an overview of
the CIR processes.
Outcomes of CIR
Possible Capability Development Initiatives
From the CIR pilots conducted, WDA has collated some common areas of
improvement arising from findings of the ATOs piloted. These are taken into
consideration for capability development initiatives that are being planned. Some of
the initiatives in the pipeline include the conduct of regular seminars to equip ATOs
with the knowledge to professionalise internal practices, which can be in the form of
monthly briefings, one-to-one consultations, or sharing of best practices and
outcomes of CIR in groups. ATOs can also look forward to the development of a
guidebook to assist them to set up a training system and implement continuous
improvement practices.
The CIR will further provide information and evidence that will enable ATOs and
WDA to identify areas of strength as well as areas in need of focused attention for
planned improvement in the short, medium and long term. With these needs
identified, WDA can then design suitable interventions to assist the ATOs.
Continuation of Approval Statuses
In addition to capability development, the CIR also replaces the need for application
by ATO for renewal or extension of WSQ statuses, i.e. the CIR determines if the
approval status of the ATO should be continued.
The continuation of approval statuses underpinned by the results of CIR is guided by
the CIR Outcome Measures. Other considerations determining the continuation of
approval statuses besides the periodic CIR are also guided by the CIR Outcome
Measures.
The complete CIR Outcome Measures is represented in Annex E.
Fees
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 21
An annual fee of $321 (inclusive of GST) is payable per ATO for CIR. CIR fees are
payable from 1 Apr 2007 via e-payment by ATO upon receipt of WDA’s payment
advice.
Feedback
WDA welcomes any feedback to seek continuous improvement and fine tune the CIR
system. Please send your comments to Ms Chiang Han Meng at
chiang_han_meng@wda.gov.sg.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 22
Annex A: Step-by-Step Process of CIR for
ATO
Step
Action Needed
Remarks
Documentary
Evidence
ATOs are
scheduled for CIR
and notified by
WDA through email
1 month before the
CIR
Step 1:
Receipt of
CIR Notice
Upon receipt of
WDA’s CIR
notification, ATO is
to respond to WDA
with the completed
self-assessment
checklist (Annex B)
and a 2 year plan
(Annex C)
(Applicable to only
ATO offering public
courses)
WDA Officer will
evaluate the ATO’s
readiness for the
CIR
WDA Officer
advises on
allocated
documentation
submission
schedule and/or
range of duration
for possible site
audit and possible
training/assessment
observation
sessions.
Timeline
According
to CIR
schedule
Training and
assessment
schedules
ATO is to
responding
to WDA
within 5
working
days upon
receipt of
CIR
notification
ATO is also to
indicate a series of
available dates and
time in the
response to the CIR
WDA Officer will
notification.
finalise the CIR
date and time with
the ATO based on
For areas which
the provided series
ATO does not have of dates and time
a system or not
from the ATO
implemented yet,
ATO is to submit an
action plan
providing details on:
a) Gaps
b) Follow-up
actions to
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 23
Step
Action Needed
Remarks
Documentary
Evidence
Timeline
address
gaps
c) Time needed
to implement
changes.
ATO is to provide
training/assessment
schedules and
anticipate a period
(and not a
stipulated date and
time) when the site
visit would possibly
be conducted
Step 2:
Preparing
for CIR
Audit
ATO to prepare a
Standard CIR
15-min presentation checks to include :
outline and
coverage advised in 1. Course
Quality
Annex D
Assurance
ATO to get ready
System
required evidences
and supporting
2. Plan-Dodocuments and
Check-Act
when informed
(PDCA)
about the site audit,
System
ATO to avail
3. Outcome
facilities for site
Evaluation
audit. WDA-QAD
System
will go down to
ATO’s training site
4. Adult
to conduct CIR site
Educator
audit.
Management
System
5. WSQ
Administrativ
e Related
System
6. Viability of
Organisation
1. SOP
manual/flow
chart etc, of
the various
systems
2. CVs and
evidence
showing
qualifications
of trainers,
assessors and
curriculum
developers
declared
3. Process of
review by ATO
to ensure
currency of
courseware,
relevance to
industry, target
audience and
contextualisati
on
4. Administrative
System –
Show
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 24
Step
Action Needed
Remarks
7. Free from
Breach of
Terms and
Conditions
8. Evidence-ofImplementati
on of the
Various
System
Documentary
Evidence
Timeline
evidences of
accurate and
timely
assessment
submission
and
certification
issuance
processes
5. Outcomes
tracking, e.g.
attrition rate,
9. Outcome(s)
passing rate,
of the Various
surveys
System
conducted etc
6. All evidence of
implementatio
n of the
various
systems
7. Documentation
of outcomes of
all the various
systems
Supplementary
evidence sources
or
supporting
documents which
may be requested
for the CIR audit
are:
1. Course files
(course
materials
submitted at
point of course
application,
includes
Assessment
Plan, Slides,
Participants’
Handouts and
Trainer Guide)
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 25
Step
Action Needed
Remarks
Documentary
Evidence
Timeline
2. Course
attendance
records
3. Training and
assessment
records
Step 3:
During site
audit
15
minutes The management
presentation
by representative
should be present
ATO
during the site audit
Site audit and MR
interview
Conduct audit of
training premises (if
applicable)
Step 4 :
Training/
assessment
Observation
ATO to avail
training/assessment
sessions for WDA
Officer’s
observation
Step 5 :
Go through areas
(strengths and
weaknesses)
highlighted by WDA
Officer
After the
audit
ATO’s slides and The site
handouts
for audit
would
presentation.
normally
All SOPs related not take
to
the more than
implemented
half a day
systems
WDA
Officer will
brief
provider
and
discuss
follow-up
actions if
necessary
WDA
Officer to
prepare
CIR report
Step 6 :
Receive
CIR audit
report and
outcome
Study findings
the report.
in
Take note of
rectifications and
timeline in CIR
CIR
outcome
measures
(shown in
Annex E)
will apply –
where
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 26
Step
letter
Action Needed
Remarks
Documentary
Evidence
outcome letter and
take corrective
actions
Timeline
ATO has
cleared,
taken
action or
failed to
take
corrective
actions
within
stipulated
time/does
not
respond
* Timeline for CIR cases is 50 working days from the date of site audit.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 27
Annex B: Self-assessment Checklist
Proposed
CIR Date:
Name of ATO:
Name of Management Representative (MR):
Contact Details of MR:
(Email)
(Tel No)
(O)
(Hp)
Section A
Are you able to show evidence for all the items in the checklist? If your answers are all ‘Evident’ in Section A, your organisation is likely to be ready for a CIR
audit. Please note that the list of evidence suggested for each area is NOT EXHAUSTIVE.
Area of Review 1 – Course Quality Assurance System
Criteria
Interpretation of Criteria
Implemented System
Evidence of System
Implementation
Outcome of System
Implementation
Evident (E) / Not Evident (NE)
1.1) System to ensure
courseware and assessment are
aligned to WSQ, pedagogically
sound and contextualised
•
•
System in place to manage developers
Process in place to conduct internal
evaluation and validation of
courseware to ensure alignment to
WSQ standards
-E
-E
-E
- NE
- NE
- NE
-E
-E
-E
- NE
- NE
- NE
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 28
•
1.2) System to manage profiling
of learners to ensure that the
course(s) is fit for purpose
•
•
1.3) System to manage training
activities
•
1.4) System to manage
assessment activities
•
System in place to facilitate
contextualisation of courseware to
ensure its currency and relevancy, etc
-E
-E
-E
- NE
- NE
- NE
Process in place to ensure systematic
profiling of learners, e.g. did the learner
meet the prerequisite of the course:
1. literacy level
2. numeracy level
3. required assumed
knowledge/skills, etc
-E
-E
-E
- NE
- NE
- NE
Process in place ensure to only learners
with the appropriate aptitude/attitude
are enrolled into the course
-E
-E
-E
- NE
- NE
- NE
System in place to continuously
monitor the training processes to
ensure:
1. it is in accordance to CBT
principles, and
2. it is in accordance to industry
practices i.e. ensuring quality
and currency of training
-E
-E
-E
- NE
- NE
- NE
System in place to continuously
monitor the assessment processes to
ensure it is in accordance to CBA
principles, i.e., valid, reliable, fair, and
flexible
-E
-E
-E
- NE
- NE
- NE
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 29
Area of Review 2 - Plan-Do-Check-Act (PDCA) System
Criteria
Interpretation of Criteria
Implemented System
Evidence of System
Implementation
Outcome of System
Implementation
Evident (E) / Not Evident (NE)
2.1) System to ensure PDCA is
implemented regularly for the
comprehensiveness of training,
assessment and management
system
•
Demonstration of establishment in
PDCA. This may include:
1. Organization management
2. Training management
3. Assessment management
-E
-E
-E
- NE
- NE
- NE
Area of Review 3 – Outcome Evaluation System
Criteria
Interpretation of Criteria
Implemented System
Evidence of System
Implementation
Outcome of System
Implementation
Evident (E) / Not Evident (NE)
3.1) System to monitor and
analyse L1 outcomes
•
•
•
•
Clear articulation of indicators, this is to
include trainee satisfaction rate.
-E
-E
-E
- NE
- NE
- NE
Trainee satisfaction rate minimally on
these 4 areas:
1. Training Programme
2. Course Material
3. Trainers
4. Facilities and Logistical support
Analysis of obtained data
Follow up action(s) on analyzed data
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 30
3.2) System to monitor and
analyse L2 outcomes
3.3) System to monitor and
analyse L3 outcomes
3.4) System to monitor and
analyse L4 outcomes
•
Clear articulation of indicators, this is to
include
1. Passing rate
2. Attrition rate
Analysis of obtained data
Follow up action(s) on analyzed data
Clear articulation of indicators on
impact of transfer of learning, this is to
include:
1. Extent of candidate’s behaviour
and/ or capability improvement
as indicated by candidate
2. Extent of candidate’s behaviour
and/or capability improvement
as indicated by employer
Analysis of obtained data
•
Follow up action(s)on analyzed data
•
Clear articulation of indicators with
analysis of obtained data on impact of
training on business, which may
include:
1. volumes,
2. values,
3. percentages,
4. timescales,
5. return on investment,
6. other quantifiable aspects of
organisational performance, for
instance; numbers of complaints,
staff turnover, wastage, noncompliance, quality ratings,
accident ratings, zero-defect
ratings, growth, retention, etc.
•
•
•
•
-E
-E
-E
- NE
- NE
- NE
-E
-E
-E
- NE
- NE
- NE
-E
-E
-E
- NE
- NE
- NE
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 31
Area of Review 4 – Adult Educator Management System
Criteria
Interpretation of Criteria
Implemented System
Evidence of System
Implementation
Outcome of System
Implementation
Evident (E) / Not Evident (NE)
4.1) HR system to ensure most
qualified adult educators are
recruited and inducted
4.2) HR system to monitor,
appraise and manage
performance of adult educators
4.3) HR system to train and
develop adult educators
•
•
•
System in place to recruit and induct
qualified adult educators
-E
-E
-E
- NE
- NE
- NE
System in place to manage the
performance of adult educators
-E
-E
-E
- NE
- NE
- NE
System in place to facilitate adult
educators ongoing professional
development
-E
-E
-E
- NE
- NE
- NE
Area of Review 5 – WSQ Related Administrative System
Criteria
Interpretation of Criteria
Implemented System
Evidence of System
Implementation
Outcome of System
Implementation
Evident (E) / Not Evident (NE)
5.1) Learner Support System in
place
•
Learner support system that covers:
1. Course advisory
2. Progression pathway
3. Course schedule
-E
-E
-E
- NE
- NE
- NE
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 32
5.2) System to ensure accurate
and timely submission of
assessment results
•
•
5.3) WSQ certificate printing and
distribution system
5.4) Maintain up to date profile
in Skillsconnect
•
•
4. Fees and refund policy
5. Retest/Appeal procedures
6. Career advisory (if applicable)
7. Placement support (If applicable)
Assessment data are submitted in
accordance to prescribed timeline
within 2 weeks from the date of the
assessment, unless otherwise approved
Submission of results must be accurate
WSQ cert is printed and distributed
according to agreed service timeline:
1. Cert must be printed and
distributed to trainees within two
months from date of assessment,
unless otherwise approved
2. Distribution of original certificate
must be accurate
Profile information in Skillsconnect
must maintain the most up-to-date
information. This include:
1. Organization profile
2. Course profile
3. Adult educators profile
-E
-E
-E
- NE
- NE
- NE
- NA
- NA
- NA
-E
-E
-E
- NE
- NE
- NE
- NA
- NA
- NA
-E
-E
-E
- NE
- NE
- NE
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 33
Section B
If answer to any question below is ‘Yes’, please elaborate under Remarks column.
S/N
Questions
1
Are there changes in the management level personnel for the past one year?
2
Is there any major instability in the organisation’s financial health for the past
one year?
Has your organisation violated any WSQ terms and conditions?
3
Yes
No
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 34
Remarks
Annex C: 2 Year Plan
(Applicable to only ATO offering public courses)
ATO is required to submit proposal on the training and assessment delivery on a 2-year
timeline, this is to ensure that the ATO have a clear mission and vision on their involvement
in WSQ. The proposal must indicate the following:
i) Identified one specific WSQ framework, e.g., Security WSQ;
ii) Intended scope of delivery within the framework, e.g. bundling of SOAs for
the 2 licensing modules at the Certificate in Security Operations (CSO) level,
full qualification of the CSO, etc;
iii) Target market and proposed clientele mixed, eg., training only Malaysian
learners;
iv) Existing clientele and new clientele over the past year who dropped out, eg.,
Malaysian trainees who had attained at least 3 GCE ‘O’ Levels including
English Language or its equivalent; and Malaysian trainees with language
difficulties and cultural adjustment issues (ascertained through screening);
v) Proposed marketing and outreach strategies, eg. advertising in the Malaysian
print media and newspapers, course previews and career fairs at Malaysian
recruitment centres, etc;
vi) Intended annual headcount to be trained, eg. at least 25 trainees for the 2
licensing modules at the CSO level, (or 50 Statements of Attainments [SOAs])
per month; and
vii) Proposed innovative learning and assessment strategies, eg. Compact Disks
(CD) would be deployed for intensive trainee revisions on the practical
performances before assessments at the centralized assessment centre,
Polwel Co-operative Society Limited.
viii)Risk assessments by the ATO. These include:
•
•
•
Financial Risk
Operations Risk
Strategic Risk
To facilitate the submission of the proposal by the ATO, ATO can use the provided template
in the following page. ATO is advised to provide between 200 and 250 words for the
proposal.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 35
Template for 2-year plan
Applicants are advised to provide between 1000 and 1250 words for the proposal.
i. Specific WSQ framework
Identified existing WSQ framework, and/or new WSQ framework(s) with intention of embarking
into.
ii. Scope of delivery
Intended scope of delivery within the framework (existing and/or new). Indicate the WSQ modules
e.g. interact with guest, deliver service excellence, workplace safety and health.
iii. Target market and proposed new clientele-mix
•
•
The trainees and the intended proportion Eg., training only Singaporean and local PRs.
Target Corporate Clients (indicate clienteles’ names).
iv. Existing clientele
•
•
The existing ratio of Singaporeans/PRs trainees in relation to foreign trainees.
Existing Corporate Clients.
v. Proposed outreach strategies
Proposed marketing and outreach strategies, eg. advertising in the print media and newspapers,
course previews and career fairs, etc.
vi. Estimated annual headcount to be trained and projected number of SOAs
Eg. at least 25 trainees for the 2 licensing modules at the Customer Service Officer level, (or 40
Statements of Attainments [SOAs]) per month.
vii. Proposed learning and assessment strategies
Eg. Small group tutorials cum buddy scheme for learners with low learning retention ability and
compact disks (CD) would be deployed for intensive trainee revisions on the practical
performances before assessments at the centralised assessment centre.
viii. Risk assessments by the applicant
Eg. Security ATOs would face a declining no. of trainees should their clients convert to in-house
training or install the following IT infrastructure:
- Bio-metric access and manpower scheduling system;
- Remote CCTV surveillance system; and
- Visitor management system.
The ATO would hence face a challenge to meet the initial proposed training target/s. ATO to
suggest and elaborate how to mitigate this risk.
ix. Remark/Other Comments
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 36
Annex D: CIR Presentation by ATO
The Continuous Improvement Review (CIR) will commence with a
presentation by the ATO outlined as follows. The presentation can be in the
form of a powerpoint presentation or briefing notes. The Management
Representative (MR) must be present to conduct the presentation or provide
clarifications. The presentation or brief notes will be part of the documentary
evidences of the CIR. Evidences can include the following:
1.
2.
Course Quality Assurance System
i) Course files (course materials submitted at point of course application,
includes Assessment Plan, Slides, Participants’ Handouts and Trainer
Guide)
ii) Summary page of courses with updated changes and what
were the specific affected materials.
iii) E.g. 1 “Interact with Guests” - Lesson plan has been
contextualised.
iv) E.g.2 “Prepare for Service” – Assessment plan has been
amended.
v) Tracking mechanism used by ATO
Adult Educator Management System
i) Performance management of trainers/assessors
ii) Training and development of trainers/assessors
iii) Description of supervision mechanism for trainers
and
assessors under supervision
3.
Outcome Evaluation System
i) Update on actions taken based on organisation audit or previous CIR
ii) Continuous review and feedback mechanisms in place and
implementation
iii) System to monitor and analyse L1 to L4 outcomes with detailed
analyzed of collected data
4.
WSQ Related Administrative System
Learner support system that covers course advisory, progression
pathway, course schedule, fees and refund policy, retest/appeal
procedures, career advisory, placement support (if applicable)
ii) System in place to ensure accurate and timely submission of
assessment results
iii) WSQ certificate printing (if applicable) and distribution system
iv) Course training and assessment records
i)
5.
Plan-Do-Check-Act (PDCA) System
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 37
i)
6.
Demonstration of establishment in PDCA.
Viability of Organisation
i) Update on governance and management of ATO
ii) ATO’s financial health for the past one year
7.
Free from Breach of Terms and Conditions
i) Records of ATO for complaints or compliments received by ATO or on
ATO
*Presentation should be kept to 15 mins.
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 38
Annex E: CIR Outcome Measures
Result of CIR
(can be either A or B)
Outcome
Not Addressed Within
Advised Timeline
Not Addressed After
Reminder
A) Cleared CIR with no
action required
Continuation of approval status with new validity
period, with or without new applications
B) CIR with action required
Reminder letter with
rectification within 5 working
days
Revoke of WSQ ATO
status.
Ad-hoc CIR
Not Addressed Within
Advised Timeline
Not Addressed After
Reminder
Reminder letter with
rectification within 5 working
days
Revoke of WSQ ATO
status.
(outside of CIR
schedule)
C) Triggered by Other
Factors
i)
Complaints with
action required
ii) Misrepresentation
iii) Inaccurate or late
report submission
iv) Breach of WDA
Terms and
Conditions
Issued by: Quality Assurance Division
Singapore Workforce Development Agency
Version:
1.7
Effective: 1 Sep 10
Version Control
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 39
No
Date
Changes Made
Version No.
1
31 May 07
Fee revised to $315 (pg. 6)
1.2
2
21 Sep 07
Fee revised to $321 (pg. 6)
1.3
3
5 Jan 09
Revision of areas of review
1.4
Added Essential Elements of Review
(pg.12)
Added System-Evidence -Outcome
Approach (pg. 13)
4
1 Aug 09
Added Annex B: Self-assessment
Checklist (pg. 25)
1.5
Removed NSRS
Added Example of S-E-O (pg.13)
5
1 July 10
Added Annex C: 2 year plan (pg.35)
1.6
Revised submission timeline for self
assessment checklist and 2 year plan
(p.19)
Added criterion 1.2 into “Course
Quality assurance system”
Revised Outcome Measures, i.e.
“Revoke of WSQ ATO status” under
column “Not Addressed After
6
1 Sep 10
Reminder” (p.39)
1.7
© Singapore Workforce Development Agency
WDA-WSQ-CIR-GUIDE Effective date: 01 Sep 2010 version 1.7
Page 40