The ASEAN University Network Quality Assurance Journey:
Challenges and Lessons Learned
Tan Kay Chuana
Ong Chee Binb
Zhou Qic
a
Director, Office of Quality Management, National University of Singapore
b
AUN-QA Expert
c
Consultant, Office of Quality Management, National University of Singapore
Abstract
Over the last three years, the ASEAN University Network has been building up a network for quality
assessment at the educational program level. There is a slow but steady rise of a regional assessment
framework that ASEAN country member can embrace. There have been many challenges in developing
this instrument. This report documents the journey.
Keywords: ASEAN University Network, quality assessment, quality assurance, program assessment
Introduction
The primary aim of this report is to share the journey that the ASEAN (Association of Southeast Asian
Nations) University Network on Quality Assurance (AUN-QA) took in the last three years to implement
its QA framework. This report highlights also key challenges and lessons learned during the
implementation journey.
Over the last three years (Dec 2007 – Dec 2010), the AUN-QA had carried out 10 actual quality
assessments for 23 undergraduate programs in seven AUN member universities. The implementation
journey can be characterized as I3, encompassing initiation, implementation, and improvement.
Numerous challenges were encountered before, during, and after the implementation of the QA
framework. They are valuable lessons for QA organizations and agencies in implementing their own
quality assurance framework.
Background of the AUN-QA
The AUN is a network of 17 member universities in South-east Asia, established in 1995 with an
agreement signed by the Ministers of 10 ASEAN countries responsible for higher education. Today, the
AUN has 26 member universities. The AUN recognizes the importance of quality in higher education
and the need to develop a holistic quality assurance system that can raise academic standards and enhance
education, research and service among AUN member universities. In 1998, it mooted the AUN-QA
network which aimed to develop QA mechanism to uplift and enhance higher education standards among
its members. Following the establishment of the AUN-QA, was the development of the AUN-QA
Guidelines and the AUN-QA Manual for the Implementation of the Guidelines in 2004 and 2006,
respectively. The launch of these documents signaled the start of the AUN-QA implementation journey
in 2007. This journey encompasses initiation, implementation and improvement.
Initiation Phase
The initiation phase involved an on-site pilot quality assessment of Burapha University in May 2007. The
aim of the exercise was to put into practice the “Manual for the Implementation of the Guidelines” by
assessing the implementation of the AUN-QA model for internal quality assurance at the institutional and
program levels.
An assessment team composed of Chief Quality Officers (CQOs) from five leading ASEAN universities
with experience in quality assurance, conducted an on-site quality assessment first at the institutional
level, and then at a program in the Faculty of Engineering. The assessment was carried out with the
1
participation of CQOs and officials from other member universities, followed by a workshop to review
the process and results of the assessment. The pilot assessment provided valuable insights into the
interpretation of the AUN-QA criteria, assessment process and interviews, format and content of selfassessment reports, visit itinerary, and pre-assessment and post-assessment activities.
Arising from the pilot assessment, the following challenges were identified:
·
·
·
·
language is acknowledged as a barrier to assessment
time is a key factor if a complete and effective assessment is to be carried out
subjectivity of the assessment ratings and ambiguity of some AUN-QA criteria
lack of a structured assessment process and assessment templates to capture assessment findings
for continuous learning and improvement.
In overcoming the above challenges, the following were developed to facilitate the implementation of the
AUN-QA framework:
·
·
·
·
a set of planning and reporting templates was designed and standardized for use in the actual
quality assessment at the institutional, IQA system, and program levels
a systemic approach to quality assessment using the Plan-Do-Check-Act (PDCA) approach
a set of guidelines on assessment itinerary and interview with staff, students, alumni, and
employers
a set of requirements for writing a self-assessment report (SAR) and key evidences including the
use of English as a language for assessment, interview, and reporting.
Implementation Phase
Before the start of the implementation phase, five experienced CQOs from Singapore, Malaysia,
Thailand, and the Philippines who were involved in the initiation phase as assessors. The implementation
of the actual quality assessment at program level was on a voluntary basis. The initial programs proposed
for assessment were mainly undergraduate engineering and business. The first actual quality assessment
at the program level was carried out at the University of Malaya in Dec 2007. Biomedical Engineering
and Computer Aided Design/Manufacturing Engineering were the first two programs to be assessed.
The first actual quality assessment provided the platform for the PDCA assessment process, itinerary, and
interview guidelines and templates to be evaluated. The outcome of the first actual quality assessment
was encouraging. This led to an overwhelming request for voluntary assessment. As the assessment
team acquired more experiences and credibility, the quality assessments were expanded to include nonengineering and non-business undergraduate programs. Between Dec 2007 and May 2012, the AUN
carried out 14 assessments at seven universities in 34 undergraduate programs (see Table 1 below).
Table 1: Assessments carried out by the AUN over the period Dec 2007 to Dec 2010
No.
Period
University
Programme
1st
Dec 2007
University of Malaya
2nd
Aug 2008
De La Salle University
3rd
Dec 2008
Universitas Indonesia
4th
Dec 2008
Institut Teknologi Bandung
5th
Oct 2009
Universitas Gadjah Mada
6th
Dec 2009
Vietnam National University - Hanoi
2












Biomedical Engineering
Computer-Aided Design /Manufacture Engrg
Chemical Engineering
Applied Economics
Civil Engineering
Mechanical Engineering
Physics
Pharmacy
Pharmaceutical Sciences
Chemistry
Medical Education
Information Technology
No.
Period
University
Programme
7th
Dec 2009
Vietnam National University – Ho
Chi Minh City
8th
Oct 2010
Universitas Indonesia
9th
Nov 2010
De La Salle University
Dec 2010
Jun 2011
Vietnam National University - Hanoi
Universitas Gadjah Mada
th
Nov 2011
De La Salle University
th
Dec 2011
Vietnam National University – Ho
Chi Minh City
14th
May 2012
Vietnam National University - Hanoi
10th
11
12
13
th






















Information Technology
Computer Science & Engineering
Electronics & Telecommunications
Architecture
Electrical Engineering
Chemistry
Metallurgy & Material Engineering
Chemistry
Psychology
Literature
Economics
Biology
Geology Engineering
Civil & Environmental Engineering
Physics
Applied corporate Management
Software Technology
Biotechnology
Manufacturing Engineering
Vietnamese Studies
Bachelor of Science in Chemistry
Fast Track Bachelor of Arts in English Teacher
Education
In the implementation phase, the AUN-QA guidelines and criteria, processes, and templates, were
rigorously tested. Valuable lessons were learnt after each round of assessment. The AUN Secretariat and
the assessors encountered new challenges in the face of rising quality assessment requests from the AUN
member universities. These challenges were:
·
·
·
·
·
·
·
a pressing need to expand and develop a pool of qualified AUN-QA assessors
different stages of QA development and maturation among the AUN member universities may
slow down the overall effort to raise academic standards and enhance education, research, and
service within ASEAN
a need for continuous learning and improvement
a need to improve the efficiency and effectiveness of the assessment process
a scheme to recognize programs and universities which have been assessed
a set of guidelines to govern the assessment process and assessors
a need to harmonize the AUN-QA assessment framework within and outside ASEAN countries
and universities.
Improvement Phase
The new challenges that surfaced in the implementation phase led to the improvement phase. The
improvement phase was a conscious effort made by the AUN Secretariat and senior CQOs to address the
challenges identified in the implementation phase as well as to prepare the AUN-QA for the next step
towards achieving its objectives. The following paragraphs document initiatives and programs developed
to address the challenges.
Development of Qualified AUN-QA Assessors
Immediately following the first assessment at the University of Malaya, the AUN realized the need to
build and expand the assessment capacity and capabilities which calls for the training of new assessors to
expand the pool of qualified AUN-QA assessors. Two assessors training workshops were held in 2008
and 2009, where 19 new assessors were trained. This increased the pool of AUN-QA qualified assessors
from 5 to 25.
3
Enhancing QA in Cambodia, Laos, Myanmar, and Vietnam (CLMV)
To further leverage on the capabilities of ASEAN-6 (namely Singapore, Malaysia, Thailand, Indonesia,
the Philippines, and Brunei Darussalam), the AUN-QA strives to narrow the QA development gap
between the ASEAN-6 and the ASEAN-4 (namely Cambodia, Laos, Myanmar, and Vietnam). This is
done through knowledge transfer and information exchange between the ASEAN-6 and the ASEAN-4.
With support from the Japan-ASEAN Integration Fund (JAIF), the AUN-QA undertook a project entitled
“Trainings on Enhancing Quality Assurance in CLMV Countries”. This training covered the AUN-QA
framework and assessment process. The objectives of the project were:
·
·
·
·
develop and strengthen QA systems in CLMV countries through the use of the AUN-QA
Guidelines and the AUN-QA Manual
build up QA personnel team in CLMV countries
improve and upgrade the quality of education management in CLMV countries
assist and sustain the AUN-QA network.
Three workshops were held in Cambodia, Laos, and Vietnam in 2010. A 4th workshop will be held in
Myanmar in 2011. After the workshops, the CQOs from the CLMV countries are expected to be able to
set up their own quality assurance system in their institutions and/or perform self-assessment at either the
institutional, internal quality assurance system, or program level. The workshop included representatives
from the Ministries of Education in the CLMV countries. The JAIF project is expected to bring
harmonization of quality assurance in higher education in ASEAN to the next level.
Continuous Learning and Improvement
Continuous learning and improvement is the essence of quality assurance activities. The AUN-QA took
the liberty to institutionalize this through the assessment process by including areas for improvement in
the assessment report. A feedback mechanism was also implemented to solicit improvement ideas from
assessors and member universities after each assessment. A benchmarking database is being planned to
facilitate learning and enhancing QA best practices among the member universities.
Improving AUN-QA Assessment Efficiency and Effectiveness
After three years of rigorous testing of the AUN-QA framework at the program level, a revision of the
AUN-QA Manual was undertaken by an appointed Documentation Review Committee to improve
assessment efficiency and effectiveness. Following several rounds of meetings and feedback, the AUNQA framework at the program level was eventually revised. The criteria were consolidated and refined,
and the number of criteria and statements was reduced. A comparison between the original and the
revised AUN-QA framework at program level is summarized in Table 2 below.
Table 2: Comparison between the original and revised AUN-QA framework
Original Criteria
1.
2.
Revised Criteria
Goals and objectives; expected learning outcomes
Programme specification
3. Programme content
4. Programme organisation
5. Didactic concept and teaching/
learning strategy
6. Student assessment
7. Staff quality
8. Quality of support staff
4
1.
3.
Expected learning outcomes
Program specification
3.
Program structure and content
4.
Teaching and learning strategy
5.
6.
7.
Student assessment
Academic staff quality
Support staff quality
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Student quality
Student advice and support
Facilities and infrastructure
Quality assurance of teaching/
learning process
Student evaluation
Curriculum design
Staff development activities
Feedback stakeholders
Output
Stakeholders satisfaction
8.
9.
10.
11.
Student quality
Student advice and support
Facilities and infrastructure
Quality assurance of teaching and learning
process
12.
13.
14.
15.
Staff development activities
Stakeholders feedback
Output
Stakeholders satisfaction
To further enhance the assessment efficiency and effectiveness, a web-based information management
system is being proposed. The IT system is expected to:
·
·
·
·
·
improve assessment efficiency and effectiveness
improve assessment transparency
share the knowledge gained and best practices documented
reach out to universities, QA agencies, and the general public within and outside ASEAN
hasten harmonization of quality assurance framework and practices.
An illustration of the web-based IT system is documented in Figure 1 below.
Figure 1: Illustration of web-based IT system
AUN-QA Certification Scheme
To encourage and recognize universities for taking the bold step in implementing the AUN-QA
framework and subjecting themselves to be assessed by AUN-QA assessors, a certification scheme is
proposed. Applying university that meets the minimum requirements would be qualified for either the
distinguished, certified or provisional AUN Quality Label with a validity period of either 1 or 5 years.
Guidelines for AUN-QA Assessment and Assessors
5
With the growing number of quality assessments and to facilitate the implementation of the AUN-QA
certification scheme, a set of operating guidelines to govern the roles, responsibilities, and requirements
of the applying university (assessee), assessors, observers, and staff of the AUN Secretariat, is needed.
The Guidelines aim to provide a common frame of reference and accountability to the AUN quality
assessment process and to stakeholders. They are formulated to fulfill Section 3.8 of the Standards and
Guidelines for Quality Assurance in the European Higher Education Area, Third Edition, 2009. The
European standards and guidelines for external quality agencies on accountability procedures are
reproduced below:
·
·
·
a published policy for the assurance of the quality of the agency itself, made available on its
website
documentation which demonstrates that:
o the agency’s processes and results reflect its mission and goals of quality assurance
o the agency has in place, and enforces, a no-conflict-of-interest mechanism in the work of
its external experts
o the agency has reliable mechanisms that ensure the quality of any activities and material
produced by subcontractors, if some or all of the elements in its quality assurance
procedure are subcontracted to other parties
o the agency has in place internal quality assurance procedures which include an internal
feedback mechanism (i.e., means to collect feedback from its own staff and
council/board); an internal reflection mechanism (i.e., means to react to internal and
external recommendations for improvement); and an external feedback mechanism (i.e.,
means to collect feedback from experts and reviewed institutions for future development)
in order to inform and underpin its own development and improvement
a mandatory cyclical external review of the agency’s activities at least once every five years.
To ensure that the guidelines are effectively implemented, an AUN-QA Council is being proposed. The
functions of the council are expected to be:
·
·
·
·
·
to formulate, govern, and review directions and policies for the operation of the AUN-QA
network
to formulate and review guidelines, criteria, and documentation for AUN-QA models, assessment
process, assessor appointment, and certification scheme and fees
appoint AUN assessors and develop the system for training and certifying them
endorse assessment reports and certification status
revoke the certification status of a university if it fails to honor and fulfill its public and social
duties, undertakings and obligations to its stakeholders including the AUN-QA network.
Harmonization of the AUN-QA Framework Within and Outside of ASEAN
The AUN-QA activities have been driven not only through increasing collaboration among its member
universities, but also including exchanges among QA agencies and organizations within and outside of
ASEAN. This includes collaboration and exchanges with the ASEAN Quality Assurance Network
(AQAN), SEAMEO Regional Centre for Higher Education and Development (SEAMEO RIHED), the
German Academic Exchange Service (DAAD), and the European Association for Quality Assurance in
Higher Education (ENQA). These collaboration efforts are expected to hasten the harmonization of the
AUN-QA framework within and outside ASEAN.
AUN-QA Training for Non-Member Universities
The established AUN-QA framework and criteria certainly draw attention to larger audience. With a
growing number of enquiries on the AUN-QA framework from non-member universities across the
region, AUN-QA started to offer training courses for non-member universities in 2011. The training team
6
has conducted two sessions of workshop in October 2011 and April 2012. Either attracted more than 25
participants from various universities. The third workshop will be organized in Bangkok, December 2012.
Further enhancing QA in Cambodia, Laos and Myanmar (CLM)
Following the successful organization of workshops to enhance QA in CLMV, AUN-QA gained support
from the Asia Development Bank to conduct training courses to set up QA systems and further enhance
QA capabilities in CLM. Two phases of workshop have been planned. The first phase aims to enable the
QA personnel in respective universities to complete a self-assessment report for a chosen programme.
The second phases focuses on QA assessment skills. The first phase will take place in August 2012 and
the second phase will take place in November 2012.
The landscape of QA in AUN member universities
In order to have a broad overview of the AUN-QA actual assessment results and identify the common
strengths and weaknesses, an analysis of the AUN-QA assessment results from 2007 to 2011 has been
completed. The analysis reveals the gaps of QA status in AUN member universities. Average
performances in each of AUN-QA criteria are compared. In addition, differences in engineering
programmes and non-engineering programmes are summarized.
Diagnosis tool for Self Assessment Report
Diagnosing tool for self-assessment report is developed to help department to evaluate their SAR. The
tool is essentially a checklist which consists of the areas that department commonly neglect. Using this
tool could enable department to improve the quality of their SAR by incorporating more solid evidences
for actual assessment.
Lessons Learned
From the AUN-QA experience, the successful establishment of quality assurance processes in its member
universities can be attributed to the following critical success factors:
·
·
·
·
·
·
high engagement, involvement, and participation of stakeholders in the development and
implementation of the AUN-QA guidelines and criteria
collaborate with regional and international QA organizations and agencies to facilitate the
harmonization of quality assurance practices
the initiation phase allowed the early detection of problems in implementing the AUN-QA
guidelines and criteria. This enables the smooth implementation and conduct of the actual quality
assessments
voluntary basis creates commitment to adopt the AUN-QA Guidelines and criteria by member
universities
build and expand capacity and capabilities progressively to take advantage of the lesson learnt in
the earlier AUN-QA activities and assessments
continuous development and improvement of quality assurance processes ensures that the system
is robust, as well as to stay current in the development of quality assurance in higher education.
The above learning points provide useful lessons and a road map for quality assurance bodies and
universities to pursue in starting and sustaining their own QA practices.
References
ASEAN University Network (2010). Guidelines for Implementation of the AUN-QA Manual. From
AUN website, Retrieved April 1, 2011 from
http://www.aunsec.org/site/main/web/index.php?option=com_content&view=article&id=219%3Aaun-qaguideline&catid=154%3Aaun-qa-project&Itemid=191&lang=en
7
ASEAN University Network (2010). Manual for the Implementation of the Guidelines. From AUN
website, Retrieved April 1, 2011 from http://www.aunsec.org/site/upload/qa/QA_Manual.pdf.
European Association for Quality Assurance in Higher Education (2009). Standards and Guidelines for
Quality Assurance in the European Higher Education Area, Third Edition. From ENQA website,
Retrieved April 1, 2011 from http://www.enqa.eu/files/ENQA%20Bergen%20Report.pdf.
8