2003/04 Financial review of the
New Zealand Qualifications
Authority
Report of the Education and Science
Committee
Contents
Recommendation
2
Introduction
2
Financial and performance management
2
Operational performance and accountability
3
Variability in scholarship results
3
New “distinction award”
4
NCEA and students’ achievement
4
Entrance into university
5
Accreditation audit of Te Wananga o Aotearoa
6
Approach to this financial review
7
Evidence and advice received
7
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
New Zealand Qualifications Authority
Recommendation
The Education and Science Committee has conducted the financial review of the 2003/04
performance and current operations of the New Zealand Qualifications Authority (NZQA)
and recommends that the House take note of its report.
Introduction
The 2003/04 financial year was a challenging one for the authority, because of the
simultaneous implementation of NCEA (the national certificate of education achievement)
to year 13, and the introduction of scholarship examinations. NZQA informed us that the
implementation of NCEA level 3 qualifications jointly with the scholarship examination
was a policy decision.
There were significant developments in several other areas:
•
A New Zealand register of quality-assured qualifications, listing all approved
qualifications available in New Zealand’s tertiary institutions and secondary schools,
was established.
•
Industry-based audit standards were developed in conjunction with industry training
organisations.
•
Implementation of the information systems work programme project continued, and
is due for completion in 2006.
We were told that Qualifications Recognition Services, formerly known as the Qualification
Evaluation Services, continues to provide responsive services in a rapidly changing
international environment.
Financial and performance management
In an unqualified audit from the Office of the Auditor-General, NZQA again received
ratings of “excellent” and “good” for its financial and service performance management. 1
As noted below, we consider that the authority's service performance management control
environment could be improved by the inclusion of measures around the implementation
of NCEA.
Financial performance
In the last two financial years NZQA has posted deficits in excess of $3 million. In this
financial year the authority achieved a net surplus of $14.271 million.2 We were told that
factors contributing to the surplus included efficiencies introduced by the authority, an
unexpected increase in the number of secondary school candidates sitting examinations,
1
2
Office of the Auditor-General, Briefing on the New Zealand Qualifications Authority, 2 February 2005, p.4.
New Zealand Qualifications Authority, Annual Report 2003/04, p.24.
2
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
candidates sitting examinations at more than one level, and an increase in the number of
examination candidates from wananga. There was also a reduction in expenditure.
The authority receives funding from the Government and from fees charged. We were
informed by the authority that once the implementation of NCEA stabilises, the NZQA
should be able to budget and forecast more accurately.
Fees
The fees for national qualifications for senior secondary school students have been reduced
from $150 per student in 2004 to $75 in 2005.3 Fees are set by NZQA in consultation with
the Government. Seeking an assurance from the authority that the fee structure is
financially sound, we were informed that the authority has no plans to alter the fee
structure at present.
Operational performance and accountability
On 15 February 2005 the Government initiated two reviews: one of the setting and
management of the 2004 Scholarship examinations, to be completed by 29 April 2005, and
one of the performance of NZQA, to be completed by 31 July 2005. The Associate
Minister of Education also announced that he was establishing a taskforce to advise on the
New Zealand Scholarship examination.
We asked the authority if the Government had previously expressed any concerns about
NZQA’s performance. We were informed that while there is an ongoing dialogue between
the chief executive and the minister, NZQA were unaware of serious Government
concerns with its performance. The chairperson of NZQA informed us that he had been
apprised recently of the possibility of a review, and expressed a willingness to cooperate.
NZQA also informed us that it was initiating a number of reviews. These include
•
an internal review by the risk committee of the authority (we were assured by the
chairperson that every effort will be made to ensure the independence of this
review.)
•
inquiries into a number of matters by the chief executive, including the failure to
“red-flag” an email sent to the minister’s office
•
an external and international review of the examination process.
Variability in scholarship results
In the recent scholarship examinations 300 of the 900 students sitting the English
examination attained a scholarship, compared with only 9 of the 1,000 biology students.
We heard that a number of factors affect variability in scholarship results. They included
the new curriculum, the standards set, the preparation of teachers, the particular
examination questions, and the marking schedule.
The Secondary Education Team in the Curriculum, Teaching and Learning Group of the
Ministry of Education coordinates the development of achievement standards and
3
Response to written questions, 15 February 2005, question 10.
3
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
assessment resources for NCEA and the New Zealand Scholarship examinations. They
also provide professional development for teachers in preparation for these qualifications.
The authority sets and marks the examination questions. Throughout the year the authority
published mock scholarship examination questions and a marking schedule on its website.
We were informed that as the scholarship examination is a standards-based assessment, the
removal of inter-subject scaling may also have contributed to variability in the results.
Furthermore, we were told some schools had entered students who were not capable of
achieving a suitable standard to merit scholarships.
New “distinction award”
We sought clarification from the authority of the sequence of events that led to the
Associate Minister of Education developing a new category of “distinction award” for
those students who performed exceptionally well at NCEA level 3 but did not receive a
scholarship. We were told that the events occurred as follows:
•
December 24: NZQA completed the processing of 4 million results.
•
January 14: provisional scholarship results and data were emailed to the Minister’s
office. No analysis of the data had been undertaken.
•
January 17: some preliminary analysis of scholarship data, noting variability in some
subjects, was emailed to the Minister’s office. This email was not “red-flagged”. The
data was not analysed in detail.
•
January 18: the chief executive received some scholarship data and raised some
concerns. NZQA began to work closely with the Ministry of Education.
•
The scholarship data was analysed and the degree of variability determined.
•
A cabinet paper was prepared for the Minister setting out options for offsetting the
variability in the results of the scholarship examination.
•
January 26: the Minister of Education announced the awarding of certificates of
distinction.
We note that the sequence of events will be examined during the course of the
Government’s review.
NCEA and students’ achievement
NCEA is a partnership between the Ministry of Education, which sets the curriculum and
standards for a qualification; the teachers who teach it; and the NZQA, which sets the
examination or assessment procedures, arranges the marking for external examinations and
records the results. A central feature of the NCEA is that it is standards-based; this means
that the qualification achieved by a student should reflect their actual performance, and
therefore results are not scaled.
We were interested in how NCEA was addressing student achievement at the lower end of
the qualification spectrum. We were informed that NCEA has enabled such students to
engage in learning, which has encouraged some of them to stay longer in school, and
students were leaving school with meaningful qualifications.
4
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
Variation in results
The authority informed us that factors influencing the variability of cohort results in unit
standards or achievement standards include:
•
the standards
•
those who develop and examine course material
•
the assessment schedule
•
the range of external achievement standards available to students
•
the characteristics of a cohort, such as size, gender proportions, and ability.
Achievement standards are derived from the curriculum, and assessment items are
designed to ascertain whether these standards have been achieved. Highly variable results
within and between subjects indicate a need for intervention. Variability of results within a
subject suggests intervention is needed in the curriculum, student guidance, or teaching
areas. High levels of inter-subject variability may indicate that interventions are required in
standard-setting, examinations, or marking. Intervention of these kinds may also be
appropriate for addressing high inter-year variability.
Comparing standards-based assessment results
We note that in our 2002/03 financial review of the NZQA we asked whether the
standard-based achievement results can be used effectively for the purpose of comparing
schools.
Standards-based assessment measures the progress of a student’s learning. We were
informed that comparisons between students and schools can be based on the number of
“excellents” and “merits” a chieved.
Given the variability of achievement results across different standards we have doubts
about the validity of such comparisons.
Public confidence in NCEA
We have received a copy of AC Nielsen’s 2004 survey of public confidence in NCEA. We
are dissatisfied with the authority’s response to our request for an explanation as to why it
did not include public confidence in NCEA measures in its 2003/04 statement of service
performance.
Entrance into university
Criteria for admission to a university are published on the authority’s website. These
criteria were set by NZQA and the universities in response to concerns held by universities
that some students were entering university without satisfactory literacy and numeracy
standards, and consequently not achieving.
We are concerned at the consistency in the pass rates in literacy and numeracy standards
required for university entrance. We expect that these issues will be considered as part of
the inquiry into NCEA.
5
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
Accreditation audit of Te Wananga o Aotearoa
The authority has recently conducted an accreditation audit on the Green Light Learning
for Life programme of Te Wananga o Aotearoa and we were told that a report is being
finalised. We await this report with interest.
6
2003/04 FINANCIAL REVIEW – NEW ZEALAND QUALIFICATIONS AUTHORITY
Appendix
Approach to this financial review
We met on 2 February, 9 February 2005, and 9 March 2005 to consider the financial review
of the New Zealand Qualifications Authority. We heard evidence from the New Zealand
Qualifications Authority and received advice from the Office of the Auditor-General.
Committee members
Hon Brian Donnelly (Chairperson)
Lynne Pillay (Deputy Chairperson)
Dr Ashraf Choudhary
Deborah Coddington
Hon Bill English
Helen Duncan
Simon Power
Bernie Ogilvy
H V Ross Robertson
Metiria Turei (non-voting member)
Evidence and advice received
In addition to the 2003/04 annual report and statement of intent, we considered the
following evidence and advice during this financial review:
•
Office of the Auditor-General, Briefing on the New Zealand Qualifications
Authority, 2 February 2005.
•
Responses to Part 1 written questions, 19 January 2005.
•
Responses to Part 2 written questions, 15 February 2005.
•
Reponses to Part 3 written questions, 28 February 2005.
7