Report of External
Evaluation and Review
AWI International Education Group
Confident in educational performance
Confident in capability in self-assessment
Date of report: 1 July 2015
Contents
Purpose of this Report................................................................... 3
Introduction ................................................................................... 3
1. TEO in context.......................................................................................... 3
2. Scope of external evaluation and review .................................................. 5
3. Conduct of external evaluation and review ............................................... 6
Summary of Results ...................................................................... 7
Findings ...................................................................................... 11
Recommendations ...................................................................... 23
Appendix ..................................................................................... 24
MoE Number:
7387
NZQA Reference:
C18212
Dates of EER visit:
24-26, 31 March 2015
Final report
2
Purpose of this Report
The purpose of this external evaluation and review report is to provide a public
statement about the Tertiary Education Organisation’s (TEO) educational
performance and capability in self-assessment. It forms part of the accountability
process required by Government to inform investors, the public, students,
prospective students, communities, employers, and other interested parties. It is
also intended to be used by the TEO itself for quality improvement purposes.
Introduction
1. TEO in context
Name of TEO:
AWI International Education Group
Type:
Private training establishment (PTE)
First registered:
21 December 2000
Location:
Level 2, 520 Queen Street, Auckland
Delivery sites:
No delivery sites in addition to main physical
address.
Courses currently
delivered:
•
General English and IELTS Preparation
(Level 3)
•
National Diploma in Computing (Level 5)
•
AWI Diploma in Business (Level 7)
•
AWI Diploma in Computer Science and IT
(Level 7)
Code of Practice signatory:
Yes
Number of students:
International: 166 students in 2014; no domestic
students
Number of staff:
13 full-time equivalents
Scope of active
accreditation:
AWI has the necessary domain consents to assess
to enable it to deliver the four programmes it is
currently approved to deliver.
Distinctive characteristics:
AWI had students from Sri Lanka, South Korea,
Pakistan, Nepal, China and India. Most students
come from India. Most are male and over 18 years
Final report
3
of age.
Recent significant changes:
Changes to the AWI Diploma in Computer Science
and IT (Level 7) were approved and notified by
NZQA on 24 March 2015.
AWI has recently appointed a new campus
director, quality and academic manager, and joint
programme leader for delivery of the AWI Diploma
in Computer Science and IT and the National
Diploma in Computing. The previous academic
manager (the equivalent position is now campus
director) and programme leader for the AWI
Diploma in Business left AWI.
Previous quality assurance
history:
At the previous external evaluation and review
(EER) in 2014, NZQA was Not Yet Confident in
AWI’s educational performance and Not Yet
Confident in its capability in self-assessment.
The NZQA national external moderation results
(NEMR) report dated 9 September 2014
commented that AWI had met requirements in the
system moderated (computing levels 5-8), for the
second consecutive year.
AWI did not meet all of the requirements of the
Immigration New Zealand online visa system audit
in November 2014. Immigration New Zealand
commented that the concerns noted were easily
rectified and was confident that AWI would be able
to implement the recommendations immediately.
Since the Immigration New Zealand audit,
processes have been revised to ensure 100 per
cent compliance at the next audit in approximately
July 2015.
NZQA withdrew AWI’s accreditation to deliver the
New Zealand Diploma in Business (NZDipBus)
(Level 6) and New Zealand Institute of
Management Diploma in Management NZIM
Diploma in Management (Level 5) effective 18
June 2014, because AWI had consistently not met
most of the national external moderation
requirements and NZQA was not confident that
students were being assessed fairly, validly,
consistently or appropriately against the learning
outcomes of the qualification being awarded.
Final report
4
NZQA notified, with AWI’s agreement, on 4
December 2014 that AWI was not to enrol any new
students on its AWI Diploma in Business (Level 7)
until it applied for changes that addressed
concerns NZQA had and the changes were
approved by NZQA. AWI applied for significant
changes which were approved by NZQA and
notified on 12 February 2015, conditional on AWI
complying with assessment and moderation
conditions within the first six months and
successfully undergoing a monitoring visit in 12
months. NZQA revoked the condition preventing
AWI enrolling any new students on its AWI
Diploma in Business (Level 7), 18 February 2015.
Approximately 59 of AWI’s 147 students were
enrolled on its level 7 business diploma at that
time, but that number was expected to grow once
AWI could enrol new students.
Other:
At the time of the EER, AWI had no active current
risk conditions and was not subject to any
monitoring and compliance activities other than
those referred to above as part of the NZQA
approval of changes to the AWI Diploma in
Business.
2. Scope of external evaluation and review
The following focus areas were agreed by NZQA and AWI:
•
AWI Diploma in Business (Level 7)
This focus area was chosen based on the recent history of this programme, as
previously mentioned, and the number of students enrolled and the expected
growth in numbers if and when changes to this programme, including assessment
and moderation, were approved by NZQA and AWI was able to enrol new students.
AWI changes were approved by NZQA and AWI was allowed to enrol new students
in February 2015. The evaluation team also wanted to see how well AWI was
transitioning students into this significantly changed and newly approved level 7
business qualification.
•
National Diploma in Computing (Level 5)
AWI Diploma in Computer Science and IT (Level 7) was a focus area in the
previous EER. The level 5 qualification provides a pathway to the level 7
qualification. NZQA was aware that AWI was mapping the current level 5
qualification to the proposed new New Zealand Diploma in Technology Technical
Final report
5
Support (Level 5) (120 credits) arising from the Targeted Review of Qualifications.
With most of the computing field unit standards expiring, AWI was considering
changing to module/paper-based delivery if no replacement standards were
available. (Since the scoping the life of the unit standards and qualification has
been extended through to 2017.)
•
International student support
All AWI students were international students.
In accordance with NZQA policy, the scope also included the following mandatory
focus area:
•
Governance, management and strategy.
3. Conduct of external evaluation and review
All external evaluation and reviews are conducted in accordance with NZQA’s
published policies and procedures. The methodology used is described fully in the
web document Policy and Guidelines for the Conduct of External Evaluation and
Review available at: http://www.nzqa.govt.nz/providers-partners/registration-andaccreditation/external-evaluation-and-review/policy-and-guidelines-eer/introduction.
The TEO has an opportunity to comment on the accuracy of this report, and any
submissions received are fully considered by NZQA before finalising the report.
Three evaluators conducted the EER over three days at the Queen Street,
Auckland site. Prior to the EER visit and AWI’s submission of its self-assessment
information, the lead evaluator visited the site and met with the chief executive, the
campus director and members of the senior management team. The evaluators
interviewed the chief executive, campus director, academic and quality assurance
manager, projects manager and marketing managers. They also interviewed
current students, programme leaders and teaching staff.
During the visit, the evaluators spoke to graduates and other key stakeholders,
including local advisory committee members, external moderator(s), employers and
international student agents. Prior, during and after the visit, AWI provided a wide
range of documentation. Three days after the final day of the visit, the lead
evaluator and one team evaluator met the chief executive, campus director,
business programme leader and projects manager to clarify information previously
tabled and raised and discussed during the visit. At the meeting, AWI provided
further information which included homestay audits, staff appraisals, internal
moderation reports and a detailed explanation of achievement data.
Final report
6
Summary of Results
Statement of confidence on educational performance
NZQA is Confident in the educational performance of AWI International
Education Group.
•
Students at AWI have high programme and qualification completion rates. For
example, in 2014, 94 per cent of the students achieved the National Diploma in
Computing (Level 5), an improvement on 2012 and 2013. In 2014, 97 per cent
of the students achieved the AWI Diploma in Business (Level 7), also an
improvement on 2012 and 2013. The reliability of the achievement data from
2013 and 2014 and into 2015 has improved since the previous EER, and gave
the evaluators confidence in the results. This was a consequence of AWI
meeting NZQA’s external assessment and moderation requirements.
•
Each programme has positive graduate outcomes. Most students who
graduate from the lower-level computing qualification go on to the level 7
related programme (from 57 per cent in 2012, to 42 per cent in 2013, to 94 per
cent in 2014), which is a desirable outcome because it leads to most students
gaining information technology-related employment positions. The proportion of
AWI level 7 business qualification graduates who gained employment in
business-related positions increased from 45 per cent in 2012 to 86 per cent in
2013 and 92 per cent in 2014, as a consequence of AWI increasing support and
providing access to more internship and employment opportunities.
•
AWI has met the external requirements of NZQA with regard to the Diploma in
Business so that it can again enrol new students in that programme. It has
focused on reconfiguring new senior management and programme leader roles
and recruiting staff with the appropriate knowledge, experience and skills;
getting its reduced programme portfolio right; and making its learning
environment more student and staff-friendly.
•
Suitably qualified staff work collegially and effectively, and those who teach
engage in regular, structured teacher observations, appraisals and professional
development. Throughout the most recent staff changes, AWI has reviewed its
programmes and practices and developed quality processes to ensure effective
teaching, consistency in meeting national standards and good educational
performance.
•
Local advisory committee members and other key stakeholders such as
external moderators, employers and agents provide a range of educational and
professional expertise, advice and support. This helps ensure AWI is delivering
relevant and up-to-date knowledge and skills. The stakeholder contributions
include workshops, seminars, access to work experience and employment
opportunities for students.
Final report
7
•
The needs of all AWI international students, predominantly male and Indian, are
being met, and evaluator interviews confirmed student surveys which reflect a
high level of student satisfaction with their educational progress; over the past
two years the PTE has been meeting the requirements of the Code of Practice
which apply to its students.
•
AWI has a number of tools to ensure that students’ English language does not
present any barriers to their learning and success. It has made a full-time,
qualified and experienced person available on campus at all times to support
students with their English, although currently this resource is not being used as
widely or effectively as it might be in what is essentially an area that is optional
for students.
•
AWI has achieved a lot in the 12 months since its previous EER. It has
followed a comprehensive and regular improvement and reporting plan.
Supported by external advice, it has strengthened its assessment and
moderation and continues to do so as it reviews and improves its programmes
and a wide range of its practices. It has been buoyed by success, reflected in
external endorsement of its progress in assessment and moderation and
approvals of its improvements and programmes. AWI is aware that these
activities are not yet complete, but staff dynamics are positive and the PTE has
made sufficient progress in organisation-wide change and improvement to
enable NZQA to be confident in AWI’s educational performance.
Final report
8
Statement of confidence on capability in self-assessment
NZQA is Confident in the capability in self-assessment of AWI International
Education Group.
•
The evaluators were aware that AWI had recently satisfied the external
requirements imposed by NZQA related to the delivery of the AWI level 7
business diploma programme. The evaluators noted that AWI has also
applied what it has learned to its own internal practices and self-assessment,
leading to worthwhile improvements.
•
Recent senior management, programme leader and teacher appointments
and structured teacher observations, appraisals and programme reviews
integrated with other tools and processes, such as student and staff
feedback and valid assessment and moderation practices, have helped
strengthen AWI’s educational performance. AWI monitors attendance and
learner achievement closely, and the organisation’s processes readily
highlight any concerns students have. In general, such information is
accurate and clearly reported, and is usually shared among staff in
appropriate ways.
•
AWI reconfigured roles and appointed staff to ensure it had the appropriate
systems to deliver effective educational performance and ensure all learner
achievement was underpinned by robust assessment and moderation. To
this end, AWI is comprehensively reviewing all its programmes and many of
its practices. Internal and external moderation and NZQA approvals of
improvements have supported the evaluators’ view that AWI’s selfassessment is effective and contributing to a range of improvements.
•
All staff, including marketing staff, support the pastoral care and well-being
of students.
•
AWI gathers a range of achievement data and information. A major
challenge AWI faces, like most organisations, is rationalising reporting
achievement by calendar year or cohort and reconciling completions with
retentions. AWI understands this and is developing its understanding,
practice and use of its rich data to maximise the value to AWI and its key
stakeholders.
•
Positive student progress was evident, and there were some very positive
graduate pathway and employment outcomes, although employer feedback
itself was limited by the methodology used and would benefit from further
development.
•
AWI’s self-assessment is largely effective and currently supported by a raft
of surveys. Understandably, there are still some areas for improvement, but
the evaluators noted that these are being managed effectively. For example,
AWI is continuing to ensure all programmes and all assessments are
Final report
9
aligned with the appropriate level based on external advice and moderation.
The PTE intends to address a lack of visible version control of some of its
documents.
•
The verification of students’ IELTS (International English Language Testing
System) levels and assessment of their English language levels is
sometimes variable as part of entry needs analysis, and AWI has taken
initiatives to address and enhance practice in this area (see Findings 1.5).
The reasons for the low uptake of English language support were not
sufficiently considered in determining how well AWI’s programmes and
activities, including using authentic and New Zealand business case studies,
are matching students’ needs.
•
AWI has introduced encouraging initiatives which include listening to
students and staff through open forums and regular meetings, implementing
a student representative system, making wise staff appointments, reviewing
the membership of the advisory committee, continuing to seek external
advice − including its own external as well as internal moderation − and
reviewing all programmes.
•
Regular online student surveys capture useful and timely information across
a range of relevant areas including facilities, the learning environment,
learning, teaching and assessment, and guidance and support. Feedback,
including from these surveys, has contributed to improvements such as
repainting the campus and providing access to coffee and cold drinks.
•
Having necessarily focused on meeting external requirements, AWI needs
more time to further develop and strengthen its self-assessment. It is aware
of most of the matters that require further attention and is addressing them.
It has made encouraging progress within the past six months. Nevertheless,
based on what the evaluators have observed, and considering the progress
made, NZQA is confident in AWI’s capability in self-assessment.
Final report
10
Findings1
1.1 How well do learners achieve?
The rating for performance in relation to this key evaluation question is Adequate.
The rating for capability in self-assessment for this key evaluation question is
Adequate.
AWI shows high levels of programme and qualification completion rates in the level
5 computing and level 7 business programmes. Student progress and qualification
completion rates are monitored and reported to students on a regular basis based
on achievement data gathered and processed by the PTE and underpinned by
increasingly more reliable assessment and moderation than previously.
The National Diploma in Computing (Level 5) programme provides a pathway
towards the related level 7 diploma, and almost all students achieve the
qualification (see Table 1). While 2013 and 2014 achievement data is supported by
sound external moderation results, the results for 2012 call into question the validity
of achievement for that cohort. AWI has put in place a range of assessment and
moderation improvements, some programme-specific, since the previous EER.
Table 1. National Diploma in Computing (Level 5)
2012
2013
2014
Enrolled
20
29
37
Completed
7
11
17
Rollover to next
academic year
12
16
19 (in progress)
Failed/withdrew
1
2
1
Based on the National Diploma in Computing (Level 5) data provided as part of the
2015 EER, it is significant that between 2012 and 2014 new enrolment rates
increased slightly, rollovers increased slightly, qualification completion rates
increased from 88 to 94 per cent, and withdrawals decreased from 12 to 3 per cent.
AWI attributes lower withdrawal rates to better recruitment information based on
visits by marketing staff to students’ home countries, marketing staff contacting
potential students while the students were still in their home country, and improved
pastoral care and support.
For the level 7 AWI Diploma in Business, student progress and programme and
qualification completion rates are high, with AWI providing sound reasons for the
1
The findings in this report are derived using a standard process and are based on a targeted
sample of the organisation’s activities.
Final report
11
few failures to achieve the qualifications, such as not passing all assessments, and
reasons for withdrawals, such as students having to return home, obtaining a work
visa or going to study at another PTE (see Table 2). AWI has put in place a range
of assessment and moderation improvements, some programme-specific, since the
previous EER.
Table 2. AWI Diploma in Business (Level 7)
2012
2013
2014
Enrolled
61
98
119
Completed
(qualification)
22
47
59
Rollover to next
academic year
36
46
56
3 withdrew
5 (3 withdrew, 2
completed the
programme but did
not pass all
assessments)
4 (2 withdrew, 2
completed the
programme but did
not pass all
assessments)
Failed/withdrew
Based on the data provided as part of the 2015 EER, it is significant that between
2012 and 2014 new enrolment rates declined slightly, rollovers increased slightly,
qualification completion rates increased from 88 to 97 per cent, and withdrawals
decreased from 12 to 3 per cent. AWI attributes lower withdrawal rates to the same
factors as for the National Diploma in Computing (Level 5).
For the General English and IELTS Preparation programmes, AWI had three
students in 2015 with clear pathway plans to achieve the required IELTS results.
This would enable them to gain entry into their chosen careers and enrol in three
other institutions providing relevant programmes; their tutor was working with them
to achieve these goals. In 2014, AWI had three IELTS students who all exited with
the required scores and were accepted into three programmes in other institutions.
Student records from 2012, 2013 and 2014 showed participation in English
programmes by 100, 14 and 13 students respectively, with positive student
achievement by some students including other than attaining targeted IELTS
scores for entering tertiary study, such as improving their English in their current
studies. AWI attributes the initial decline in numbers to a downturn in the South
American and Korean markets, but the evaluators noted that previous English
students had also been Chinese. AWI agreed with the evaluators that the full-time
English staff member was a valuable but underused resource. AWI could
investigate the reasons for the underuse and consider and implement changes to
increase use, to benefit students’ studies and employment in New Zealand.
AWI follows up graduates where possible and presented a convincing range of
graduate outcomes of gaining employment in business-related positions within New
Zealand or overseas.
Final report
12
AWI has not identified any significant differences in programme and qualification
achievement according to age, gender or ethnicity. Data such as the dates and
numbers of initial assessment and reassessment successes and grades are used
to identify areas for improvement (for example APA referencing, business
vocabulary) and this has resulted in some improvements in programme delivery,
teaching and better integration of content and specialised language by levels in
response to areas of individual learner needs. This also supports the need for
better use of AWI’s English resource to further strengthen learner achievement
(academic support started in 2015 as a consequence of teacher concerns about the
English abilities of some students).
Educational performance information is closely monitored, compared year by year
and used to make improvements. As a consequence, AWI has made significant
progress in improving student attendance and reducing the possibility that
plagiarism could pass undetected, although the evaluators consider the
approaches taken could be more consistent and systematic. AWI is continuing to
manage the quality of its assessment through improving its internal and external
moderation practices which are vital to the validity of its assessment data and
recognition of student achievement.
1.2 What is the value of the outcomes for key stakeholders, including
learners?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is Good.
Recent level 7 graduates interviewed by the evaluators clearly described their
progress from enrolment at AWI, through part-time employment while studying, and
on to graduation and local employment ‒ many into areas that used their business
management knowledge and experience gained while studying. Graduates also
provided some good examples of transfer of skills and knowledge gained during
their studies into the New Zealand workplace in positions not always directly related
to their studies and aspirations at the time.
Graduates, local advisory committee members and employers confirmed AWI
student survey information that the transferable skills students developed during
their studies (confidence in oral presentations, teamwork and employment-seeking
skills, in particular) are valued and assist many graduates in securing employment
and settling in New Zealand. The level of job-search assistance increased in 2014
and 2015 in response to student requests in surveys and direct to staff. Current
students, graduates, advisory committee members and employers attributed the
fact that over 90 per cent of students were working part-time to a range of initiatives
– these included the provision of curriculum vitae (CV) preparation, mock interviews
and student and staff referrals provided by all staff and at least one advisory
committee member. Initiatives also included internship programmes initiated by the
Final report
13
marketing department to help students gain valuable New Zealand work experience
while still studying.
AWI monitors outcomes by telephone survey of graduates, and most recently AWI
successfully contacted all graduates. This information indicated that an
increasingly high proportion of AWI students achieve their study and work goals.
Some exceptional examples of graduate outcomes were provided, including a
graduate who has employed other AWI graduates and now offers internships to
others who wish to gain local experience. AWI also had some evidence of other
graduates successfully continuing their studies at a higher level, including New
Zealand degree level. There was some evidence that students who returned to
their home country found their AWI qualification and work experience gained in
New Zealand assisted them in finding employment.
AWI programmes offer students the opportunity to graduate work-ready, and
graduates and employers report that success in this area is high and improving.
Accessing internships is still a work in progress, but immediate networks are
developing including through staff, teachers, local advisory committee members
and other key stakeholders. Current students, supported by the experience of
graduates, were aware of the value of related job-specific, part-time work while
studying – and the reality of having to wait for some time after graduation to gain a
position such as assistant manager or manager (and in the case of computing, the
necessity of achieving the higher level 7 qualification, offered by AWI, to gain
employment).
Wherever possible, AWI teachers take a practical approach to learning in their use
of questions, as well as individual and team approaches, which are also valuable
for the workplace, as part of student learning. They also use New Zealand case
studies wherever practicable. Some students the evaluators spoke to were keen to
mix with other kiwis to enhance their learning experience – this is something AWI
could consider further as part of English support and helping students to come to
terms with the New Zealand accent and the reality of social interactions at large
and as part of the workplace beyond their AWI experience.
Final report
14
1.3 How well do programmes and activities match the needs of
learners and other stakeholders?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is Good.
NZQA withdrew AWI’s accreditation for the NZDipBus (Level 6) and NZIM Diploma
in Management (Level 5). AWI has four other approved programmes leading to the
level 3-7 qualifications it is not currently delivering. It intends to review these
programmes and consider their future use in 2015-2016 after focusing on the four
programmes it is currently delivering.
Most students enrol in AWI programmes to gain an international qualification and
work experience and to seek a post-study work visa and find longer-term
employment in New Zealand. In fact, reliable data shows that 45 per cent of AWI
Diploma in Business graduates obtained qualification-related employment in 2012,
86 per cent in 2013 and 92 per cent in 2014. Similar information indicates that the
completion of the AWI Diploma in Computer Science and IT (Level 7) is leading to
similar but not as high rates of related employment, and AWI and its students are
aware that such levels of employment by graduates of the National Diploma in
Computing (Level 5) are much lower. This is why most of these graduates go on to
the level 7 qualification as a worthwhile pathway.
Programme pathways available to AWI students support progress to higher levels
of knowledge and skills and link to Australian and New Zealand labour market
analyses of core competencies and skills shortages in specific business and IT
areas.
Useful local advisory committee input into AWI’s knowledge of industry needs
occurs, and has helped AWI respond to a changing market with its programmes
over the years. Effective ongoing links with education agents who are very
supportive of AWI has led to a student population from a range of countries, which
students clearly value, and demonstrates a met need for those students. Recent
AWI visits to overseas markets, including India, has gathered useful information
and strengthened recruitment of suitable students.
Future planning has been constrained by the need to meet external requirements
and the Targeted Review of Qualifications process for business and computing
qualifications. For example, AWI was mapping the current level 5 qualification to
the proposed New Zealand Diploma in Technology Technical Support (Level 5)
(120 credits) arising from the Targeted Review of Qualifications. With most of the
computing field unit standards expiring, AWI was considering changing to
module/paper-based delivery if no replacement standards were available. However,
recently the life of the unit standards has been extended through to 2017, which
gives AWI more time and flexibility to consider how it will most effectively continue
to match the needs of students and other stakeholders with regard to information
technology and computing at different levels.
Final report
15
Student survey analysis and graduates interviewed support the view that the goals
and needs of prospective students are now more effectively determined by AWI
from the point of possible enrolment onwards. Learning and pastoral care and
career and employment needs are being met by all AWI staff including marketing.
The sharing of initial student needs assessment and ongoing teacher information
has improved, although the necessity for an academic needs assessment remains.
AWI is meeting NZQA requirements (apart from for the NZDipBus and NZIM
Diploma in Business for which it no longer has accreditation), and has gained new
approvals. It is continuing to improve assessment and moderation, apply what it
has learned, and is reviewing all its programmes – including outcomes descriptors,
prescriptors, graduate profiles and practices. All this is helping AWI to ensure it is
matching the needs of its students and stakeholders. As mentioned, there is still
room to develop and improve individual students’ English language proficiency, and
the provision of support and activities to help students adjust to New Zealand
culture, accent and workplace practices. Visiting marae and learning the haka are
part of this approach.
1.4 How effective is the teaching?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is
Adequate.
AWI’s learning environments are well planned and structured, focused on the
benefits and needs of learners. Teachers are well qualified and experienced and
work collegially together in a way that makes the most of their respective strengths.
Their approach is highly student-centred and based on approaches that are
reactive and responsive, proactive and interactive, and focused on meeting the
needs of the students. Modes of delivery are clearer, more detailed and flexible
than they were, and effectively integrate theory and practice.
Students the evaluators interviewed confirmed that student satisfaction surveys
indicated that teachers and students relate well to each other, learning activities
and resources effectively engage students, and students have opportunities within
the classroom – and most of them in the workplace – to apply the theoretical
knowledge and practical skills they have gained. These characteristics help
support attendance, which is essential to student progress and which is closely
monitored. Students compared their current teachers very favourably with their
previous teachers.
Student and staff satisfaction surveys are conducted, supervised and processed by
the academic and quality assurance manager. Response rates and satisfaction
rates are high, and students highlighted that teachers encouraged and gave them
confidence to ask questions and challenge and express themselves, and were
available to provide help and support when needed. AWI actively identifies student
Final report
16
and teacher satisfaction and responds promptly to the information gathered to
address any issues. Student satisfaction feedback is used as part of teacher
appraisal and peer observation to support, develop and improve teacher
effectiveness. Performance appraisals and peer observation include evaluating
how effective changes have been in teaching practice. Professional development
includes assessment and moderation and is readily available and supported by
AWI.
Assessment is more flexible and based on when individual learners are ready. It is
being used to provide timely feedback to students on their progress. This includes
individual and class feedback on assignments and first drafts, which students said
they found helpful in the business diploma programme. The evaluators found
evidence of improved assessment and moderation practices, with programme
leaders and teachers following comprehensive policies and procedures to
strengthen internal moderation. There was significant improvement in external
moderation results for 2013 and 2014 computing unit standard assessments at
levels 5-8, as well as for the first two papers of the recently approved and revised
business diploma. As mentioned, AWI decided to revisit all its newly developed
material in light of NZQA feedback on one submitted assessment pack, and it
recorded all changes; this reflects good practice. While true/false and multiple
choice questions have been removed from one of the business diploma papers
based on the feedback from external moderation, the cognitive appropriateness of
the short answer questions and clarity of the assessment rubric at level 7 are
examples of areas that are open to challenge and AWI needs to further consider.
Also as mentioned, although business and computing teaching is practically based,
and uses appropriate New Zealand and real-world examples, the opportunities for
some students to apply their learning to real-world business environments is still
limited. Students desire to have more interactions with other New Zealanders and
their culture and accents, including from other educational institutions and
organisations. AWI is aware of the low uptake of the English language support it
has provided. More information about why this is the case could help AWI address
students’ needs in this area even better and maximise student use of the language
support resource.
Self-review and reporting of individual student progress and achievement at regular
staff meetings is occurring and is reasonably robust. Students confirmed that
AWI’s attendance monitoring was strict and actions were taken when necessary.
Teachers were aware of incidents of plagiarism and sanctions were applied
following fair processes, but more consistent practices would be helpful and safer.
AWI is addressing this in information and advice to students, and in assessment
and moderation practices, software use and professional development.
Final report
17
1.5 How well are learners guided and supported?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is
Adequate.
AWI provides students with effective guidance and support. This starts as early as
at the point of recruitment in India (AWI’s largest market), which AWI has visited
and checked the quality of the English medium of instruction in schools in some
regions to be better informed when considering applications. While English
language entry requirements are being met, AWI is able to identify those students,
including from remote rural areas, who may need additional English support. AWI
makes available such support but, as mentioned, this is not currently being
accessed widely by students. This may be a result of a number of factors which
could include students not wishing to be identified, as well as what is being
provided and how. Some English support is provided by teachers as part of their
programmes and their academic support, as well as by the marketing team. AWI
has addressed or is addressing mainly minor matters identified by a recent
Immigration New Zealand audit of online visa applications.
AWI has supportive agents in India and New Zealand. Based on student
satisfaction surveys and direct feedback, AWI has improved its orientation which
includes information about living in Auckland (and driving but not alcohol liquor sale
laws as required by the Code of Practice), and succeeding in studies. More recent
surveys have reflected higher levels of satisfaction with this area. Staff members
are able to speak to students in their first language about information they need to
understand before they sign off, otherwise AWI students have access to first
language speakers. Students are tested on matters they must understand before
they sign off on pastoral care, which is good practice that could be extended to
needs assessments.
Students who have been at AWI for varying periods of time, and who were
interviewed by the evaluators, observed how supportive, responsive and inclusive
all the staff at AWI are. Students who had been at AWI the longest compared AWI
now very favourably with what it used to be like. Students confirmed that teachers
are readily available and accessible and report to them regularly on their progress.
All staff contribute to student guidance and care, and staff meeting minutes show
that all staff have been informed about the Code of Practice, although not all are
fully conversant with it.
Homestay accommodation is available and AWI contracts this out. Few students,
most of whom are over 18 years of age, take up homestay accommodation (none in
the past two years), and instead flat or board with relatives or friends. AWI does
not carry out police vetting for homestays and has contracted the responsibility to
its contracted organisation (which is also expected to report the vetting to AWI),
although the Code of Practice states clearly that it is the direct responsibility of the
Code signatory. Where under 18-year-olds are staying with family members, the
Final report
18
Code of Practice states that the signatory may still wish to conduct police checks –
AWI does not do so unless specifically requested by parents. AWI staff visit
students at their residences and offer out-of-working hours assistance. This also
applies to checking smoke detectors wherever international students stay, and AWI
has done this since the EER visit. At least three staff members are available 24/7
should students need to contact someone in an emergency. AWI is not fully aware
of where the students who live out of Auckland stay during the week, and it would
be good practice to become so.
AWI follows up student attendance rigorously, requiring evidence of the reasons
why students are absent from class. AWI expects 100 per cent attendance and
acts fairly but appropriately, including issuing warning letters as required. The
evaluators noticed that AWI did not actually include the details of attendance in
these letters, or student attendance expectations in the student handbook. The
inclusion of such information would further strengthen AWI’s already strong
processes in this area. As mentioned, the evaluators were provided with examples
where sanctions around plagiarism were applied usefully and fairly (assessments
were rated as zero and students had to undergo another assessment of the same
content), and the processes and practices could be documented and applied more
consistently.
Student activities include celebration of festivals and other activities like learning the
haka. Students appreciated AWI’s support and guidance about the New Zealand
labour market and job searching, CV and interview workshops. As mentioned,
students are also keen to have more contact with other New Zealand students, their
kiwi accent, and possibly a wider pool of work opportunities. AWI is yet to consider
this.
1.6 How effective are governance and management in supporting
educational achievement?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is
Adequate.
AWI has been through a significant process of further rationalisation and
downsizing since the previous EER 12 months ago. In December 2014, NZQA
notified AWI, with its agreement that AWI was not to enrol any new students on its
AWI Diploma in Business (Level 7) until it applied for changes that addressed
NZQA’s concerns. AWI made significant changes which were approved by NZQA
in February 2015, conditional on AWI complying with assessment and moderation
conditions within the following six months and successfully undergoing a monitoring
visit in 12 months.
These actions contributed to a range of impacts that went beyond a fall in total
student numbers and a pause in AWI Diploma in Business (Level 7) numbers. AWI
Final report
19
is focused on the four programmes it is currently delivering and is not currently
delivering four other programmes it still has approvals for. It intends to review
these programmes and consider their future use in 2015-2016.
AWI realised that assessment and moderation were two related areas that needed
significant attention, although not the only ones. Aware that it needed to act quickly
to consolidate its education provision and strengthen its self-assessment, AWI
established a new management team and organisational structure in which some
roles were rationalised and responsibilities clarified and people with the appropriate
knowledge, experience and skills were appointed. The AWI chief executive and
new campus manager appointed in October 2014 had significant governance over,
and a major role in making these changes. There is clear evidence that AWI was
and is responding to external requirements, but it has also applied what it has
learned across the board in the organisation and responded to internal as well as
external information and made changes that have an impact on educational
performance and have an important bearing on the business. Regular minuted
management meetings and an open-door policy have involved all staff and students
who have focused on educational performance, self-assessment and student
satisfaction.
AWI has effectively transitioned students into the recently approved new AWI
Diploma in Business (Level 7), achieved external moderation approval for the first
two papers of the new qualification, continued to develop the membership and
functionality of AWI’s local advisory committee, improved the physical learning
environment by painting it and providing improved amenities for students, and
introduced international students to the haka. Such developments, and others
mentioned elsewhere in this report, are appreciated by the students and staff.
Areas that AWI still needs to address include: making learner achievement clearer
and more useful for learners, teachers and other stakeholders; continuing to
develop workplace and internship opportunities based on successes to date;
communicating attendance expectations more clearly and providing more
information in warning letters to support the robust monitoring and actions on
attendance; ensuring students access and make maximum use of the academic
and English language support it has made available; making changes to how it
administers Code of Practice requirements, however minor; developing and
documenting more consistent approaches, policies and processes to identify cases
of plagiarism and act on them based on the effective practices it already has in
place; and considering the introduction of activities with New Zealand students to
help the AWI students become familiar with the New Zealand accent.
AWI’s current purpose and direction are very clear: consolidate current reduced
educational provision without losing sight of other opportunities. The creation of a
board arising from the EER visit will support the chief executive and effectively help
AWI to achieve this. AWI’s leadership is strong and effective and has resulted in
courageous, wise and well-informed appointments of key new staff. Resources are
expected to continue to be appropriately and effectively allocated. Staff feel highly
valued. Despite recent and current constraints, AWI has shown its capability to
Final report
20
meet external requirements and anticipate and/or respond to change. Its current
self-assessment is showing characteristics that are consistent with being
continuous, ongoing, authentic and transparent, and identifying and leading to
worthwhile improvements. However, AWI’s self-assessment is not yet fully
comprehensive and robust – it is a work in progress and it is too early to be able to
confirm its effectiveness and capability.
Final report
21
Focus Areas
This section reports significant findings in each focus area, not already covered in
Part 1.
2.1 Focus area: Governance, management and strategy
The rating in this focus area for educational performance is Good.
The rating for capability in self-assessment for this focus area is Adequate.
2.2 Focus area: International student support
The rating in this focus area for educational performance is Good.
The rating for capability in self-assessment for this focus area is Adequate.
2.3 Focus area: AWI Diploma in Business (Level 7)
The rating in this focus area for educational performance is Adequate.
The rating for capability in self-assessment for this focus area is Adequate.
2.4 Focus area: National Diploma in Computing (Level 5)
The rating in this focus area for educational performance is Good.
The rating for capability in self-assessment for this focus area is Good.
Final report
22
Recommendations
NZQA recommends that AWI:
•
Further develop the analysis, common understanding, communication and
use of student achievement data.
•
Continue to action expert external assessment and moderation feedback
and review of assessment levels of the level 7 programmes.
•
Continue to ensure all teaching staff are engaged with assessment and
moderation.
•
Continue to develop ways in which information gathered during initial and
subsequent student needs assessment is shared and acted on by teaching
staff supported in their ‘student-centred learning’ approach.
•
Continue to develop relationships to provide students with further directly
applied or work-integrated learning.
•
Continue to review practices in relation to English language entry
requirements against NZQA published policy and guidelines to ensure
consistent application.
•
Consider ways to encourage and persuade students to take advantage of
the English teacher resource that AWI has made available.
Final report
23
Appendix
Regulatory basis for external evaluation and review
External evaluation and review is conducted according to the External Evaluation
and Review (EER) Rules 2013, which are made by NZQA under section 253 of the
Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary
Education, Skills and Employment.
Self-assessment and participation in external evaluation and review are
requirements for maintaining accreditation to provide an approved programme for
all TEOs other than universities. The requirements are set through the NZQF
Programme Approval and Accreditation Rules 2013, which are also made by NZQA
under section 253 of the Education Act 1989 and approved by the NZQA Board and
the Minister for Tertiary Education, Skills and Employment.
In addition, the Private Training Establishment Registration Rules 2013 require
registered private training establishments to undertake self-assessment and
participate in external evaluation and review, in accordance with the External
Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration.
The Private Training Establishment Registration Rules 2013 are also made by
NZQA under section 253 of the Education Act 1989 and approved by the NZQA
Board and the Minister for Tertiary Education, Skills and Employment.
NZQA is responsible for ensuring non-university TEOs continue to comply with the
rules after the initial granting of approval and accreditation of programmes and/or
registration. The New Zealand Vice-Chancellors’ Committee (NZVCC) has
statutory responsibility for compliance by universities.
This report reflects the findings and conclusions of the external evaluation and
review process, conducted according to the External Evaluation and Review (EER)
Rules 2013.
The report identifies strengths and areas for improvement in terms of the
organisation’s educational performance and capability in self-assessment.
External evaluation and review reports are one contributing piece of information in
determining future funding decisions where the organisation is a funded TEO
subject to an investment plan agreed with the Tertiary Education Commission.
External evaluation and review reports are public information and are available
from the NZQA website (www.nzqa.govt.nz).
The External Evaluation and Review (EER) Rules 2013 are available at
http://www.nzqa.govt.nz/assets/About-us/Our-role/Rules/EER-Rules.pdf, while
information about the conduct and methodology for external evaluation and review
can be found at http://www.nzqa.govt.nz/providers-partners/external-evaluation-andreview/policy-and-guidelines-eer/introduction/.
Ph 0800 697 296
E qaadmin@nzqa.govt.nz
www.nzqa.govt.nz
Final report
24