REPORT ON THE PERFORMANCE OF THE
NEW ZEALAND QUALIFICATIONS AUTHORITY
IN THE DELIVERY OF SECONDARY SCHOOL
QUALIFICATIONS
BY THE
REVIEW TEAM
LED BY DOUG MARTIN
31 July 2005
CROWN COPYRIGHT
ISBN 0-478-24499-1
TABLE OF CONTENTS
Executive Summary......................................................................................................... 1
Recommendations............................................................................................................ 7
Introduction.................................................................................................................... 11
Review Process and Methodology ................................................................................ 12
Discussion of Findings ................................................................................................... 13
3.1
A
B
Governance, monitoring and reporting of NZQA performance ............... 13
Governance ...................................................................................................... 13
Monitoring and reporting................................................................................. 19
3.2
The adequacy of NZQA capability including planning, systems
and processes to undertake its role............................................................... 23
Methodology for Assessing Performance................................................................ 23
A People Capability ............................................................................................. 23
B Planning ........................................................................................................... 28
C Systems ............................................................................................................ 30
3.3
The nature and extent of the involvement of, and responsiveness to,
key stakeholders in the decision-making of the NZQA .............................. 34
Methodology for Assessing Stakeholder Engagement ............................................ 34
Current Situation...................................................................................................... 34
Findings.................................................................................................................... 37
3.4
Whether, and if so how, the NZQA is providing leadership
in examination processes ............................................................................... 39
Purpose and History of Assessment......................................................................... 39
Experience to Date with NCEA ............................................................................... 41
Variation in the Externally Assessed Achievement Standards ................................ 42
Causes of Variation in Externally Assessed Achievement Standards ..................... 44
Recommended Approaches to Managing Variation in External Assessment ......... 46
Internally Assessed Achievement Standards ........................................................... 51
3.5
Such other matters as the Reviewer considers relevant ............................. 56
Separately Assessed Standards ................................................................................ 56
The Amount of Assessment in New Zealand .......................................................... 56
Fragmentation .......................................................................................................... 57
Numbers of Achievement Standards and Grades .................................................... 57
Annex 1: Terms of Reference for Review by the State Services Commissioner........... 59
Annex 2: Organisations and individuals consulted........................................................ 61
Executive Summary
1
This report forms Part Two of the review of the New Zealand Qualifications Authority
(NZQA). It follows the report by the Review Team on the 2004 Scholarship released
on 6 May 2005 (Part One). Part Two focuses on the performance of NZQA in
introducing a new system of secondary schools qualifications, the National Certificate
of Educational Achievement (NCEA). It covers a number of dimensions including
governance, capability, stakeholder management, and professional leadership in
examination processes.
2
In carrying out this review, the Associate Minister of Education asked the Review
Team to focus on whether the apparent variation in NCEA results both between
subjects and between years (where valid comparisons are possible) reflects deficiencies
in NZQA systems. This issue is considered as part of the assessment of NZQA’s
professional leadership in examination processes (Term of Reference 3.4).
3
The Review Team consulted widely within the sector during the course of the review.
Some 40 secondary schools with diverse characteristics were visited, with the team
mostly meeting the Principal, the Principal’s Nominee and Heads of Department of the
school. Meetings were held with representatives of education sector organisations,
employer groups, the Board and key staff of NZQA, officials from the Ministry of
Education, the Education Review Office and the Treasury, and a range of academics
and researchers. Sixty-six written submissions were also received.
4
A number of submitters raised issues more associated with the design of NCEA than
implementation. Some common themes emerged. These would be more appropriately
considered as part of the Stocktake that will shortly be undertaken under the
stewardship of the Secondary Leaders’ Forum. The Review Team was concerned that
these themes be captured as they were offered in good faith in the course of the
consultation. They are therefore summarised under Term of Reference 3.5.
5
A clear message from the majority of schools visited and from key stakeholders is that
notwithstanding difficulties with implementation, NCEA is enhancing learning
outcomes for students and improving teaching practice. There was a strong desire
expressed within the school sector and by all of the representative organisations to
address the issues that have emerged from the first three years of implementation of
NCEA and move on. Some schools and education specialists continue to express
concerns about aspects of the design of NCEA and the impact that these have had on
the reliability of assessment. Nonetheless there is acceptance that there is no going
back.
6
In assessing the performance of NZQA, the Review Team acknowledges that the
Board, the Acting Chief Executive and senior management are currently addressing a
number of the issues that have impacted negatively on the performance of NZQA.
Whilst much remains to be done to give effect to these intentions, a momentum has
been established to lift the performance of NZQA. The appointment of a new Chief
Executive with the required skills and competencies will do more than any other single
factor to lift the performance of NZQA and to strengthen the implementation of NCEA.
7
As noted in Part One the introduction of NCEA in 2002 represented an unprecedented
change in teaching, learning and assessment in New Zealand’s senior secondary
1
schools. Given the scale of the change involved the Review Team believes that the
implementation path should extend over five years or more. Within this period
interventions are required that recognise the time it takes for the achievement standards
to be fully developed and understood.
8
The Review Team has made a number of recommendations based on its findings. In
addition, reflected throughout this report are a number of suggestions for improvements
in NCEA implementation and in NZQA performance. These have arisen as a result of
the consultation process with stakeholders and also from the team’s analysis under each
of the Terms of Reference.
9
Term of Reference (TOR) 3.1 required the Review Team to assess the governance,
monitoring and reporting of NZQA’s performance in the school sector. The NZQA
Board was asked to provide a self-assessment of its own performance, which it did by
drawing on the consolidated findings of a self-review undertaken in August 2004 and
an independent review commissioned by the Board in response to difficulties with the
2004 Scholarship. As a result of these reviews, the Board has adopted 17 measures
directed at strengthening its governance. The Review Team supports these initiatives.
10
Two of the Board’s findings merit particular attention, namely, the need for the Board
to have a more strategic focus, and the need for NZQA to provide more tailored
communication on NCEA to learners, parents, schools, employers and representative
groups.
11
In respect of the former, a key finding of the Review Team is that the NZQA Board’s
primary focus has been on the logistics of implementing NCEA at a detailed
operational level. This is a massive undertaking involving external and internal
assessment at three levels, as well as the externally assessed scholarship. Each year it
amounts to more than 150,000 candidates and approximately 1.9 million individual
examination booklets. The business operations risks are therefore significant and the
focus of the Board on these risks was understandable. The broader change
management issues that arose during this period of major transition for teachers,
students and parents did not get the attention they needed. Neither did internal
organisational development issues.
12
In particular, the Board did not appreciate that the broadly defined policy settings for
NCEA afforded NZQA considerable discretion in determining both the approach to
implementation and the speed of the transition to full standards based assessment. Nor
did it appreciate the need for caution in implementation, given the scale of the change,
the relative paucity of resources for teachers in the initial stages (e.g. properly
moderated exemplars), and the fact that the development of the standards were at an
early stage and would be subject to revision as more experience was gained.
13
This under-appreciation of NCEA’s implementation was not an issue for NZQA alone.
The Ministry of Education did not convey clear expectations of a measured
implementation process to manage the risks of transition. Nor were these picked up at
the operational policy interface between the two agencies. As Part One of the review
on the 2004 Scholarship noted, the boundaries between the Ministry of Education’s
overarching policy role and NZQA’s more restricted operational policy role are not
clear. Furthermore, NZQA’s policy capacity was reduced from the mid-1990s so that it
2
was not well placed to analyse and address the full implications of NCEA
implementation.
14
In the event, the implementation path chosen by NZQA was a very steep one. The
safety nets that had developed under the ‘old system’ to ensure a degree of stability in
assessment were stripped away, and too much reliance was placed on the sufficiency
and efficacy of standards that were still in the early stages of development and subject
to differing interpretations.
15
Compounding this, the communication to different stakeholders such as students,
parents, schools and employers was not sufficiently tailored to their needs. The level of
understanding amongst parents and employers in particular remains poor. This issue is
covered more fully under TOR 3.3.
16
During the course of the review, the team met frequently with the Acting Chair of the
Board (Catherine Gibson), the Acting Chief Executive (Karen Sewell), and the Board
Member with particular expertise in secondary education (Margaret Bendall). The
purpose was to gain a common understanding of the issues associated with the NZQA’s
approach to assessment and of the measures necessary to address those issues. The
engagement has been very constructive and the Review Team is confident that the
Board will act to ensure that the extent of variability in the external assessment of
NCEA in 2005 is within acceptable tolerances, and more generally further strengthen
the implementation of NCEA in 2006 and beyond. The measures that the Review Team
believe are necessary to achieve this are identified under TOR 3.4.
17
In light of this additional workload, the Review Team believes that the capability of the
Board should be strengthened, either though the appointment of an additional member
with experience in secondary education, or by the Board augmenting its capability
through the appointment of an advisor on assessment.
18
One of the most important tasks confronting the Board over the next few months is the
appointment of a new Chief Executive. The Board’s record over the past nine years in
recruiting and retaining chief executives with the required skills has not been good.
This is evidenced by the fact that the organisation has had eight changes in leadership at
chief executive level in as many years. This has meant that NZQA has not developed
into a mature and high performing organisation.
19
The damage created by unstable leadership has been considerable. The Board, in
approaching the appointment of a new Chief Executive, must analyse NZQA’s
requirements for the next three to five years to develop a job description and a person
specification for the Chief Executive that reflects the challenges ahead. It must then
engage in a rigorous search and selection process to ensure that a candidate that is well
suited to the position is appointed. Having said this, the Review Team recognises that
the Board is fully aware of the need for stability in the leadership of NZQA and is
taking advice on best practice appointment processes.
20
The principal instrument for monitoring the performance of NZQA, the 2004/2005
Purchase Agreement with the Minister of Education, did not pick up through the
quarterly reporting processes the difficulties with the implementation of NCEA which
prompted this review. This is because the Purchase Agreement failed to adequately
define the implementation risks and express these in output and performance measures,
3
relying too heavily on quantitative measures of performance. The Output Agreement
for 2005/2006 better captures the implementation risks. However, the Review Team
recommends that some of the principal findings of this review should be reflected in
new output and performance measures and incorporated into the Output Agreement
through a negotiated variation to the Agreement. The Acting Chair of the Board has
indicated to the Minister of Education the need for a variation to the Output Agreement
and the Minister has acknowledged this.
21
Term of Reference 3.2 required the Review Team to assess the capability of NZQA to
undertake its role, including its planning, systems and processes. The overarching
finding of the Review Team is that the lack of stable leadership at the Chief Executive
level has prevented NZQA from developing into a mature and high performing
organisation and that this is reflected across most dimensions of capability.
22
The allocation of senior management responsibilities has been subject to continual
change as different chief executives have come and gone, and until recently an interim
arrangement was in place. This has created considerable uncertainty and has
encouraged a ‘silo’ mentality in which different sub-cultures have been allowed to
flourish. Nonetheless, the current allocations are unwieldy and in urgent need of
review to better align them with NZQA’s strategic direction.
23
The business operations of NZQA have operated under considerable pressure for three
years and under very compressed timeframes. This has meant that fully developed
systems, and a culture of performance that embraces the notion of individual
accountability, have not been embedded in the organisation.
24
The Executive Management Committee (which comprises the senior management
team) gave scant attention to what was one of the major challenges confronting NZQA,
the implementation of NCEA. The burden of implementation has fallen mainly on
middle management. The management experience of NZQA’s senior management team
is variable and management skills and competencies require further development.
25
The review of the allocation of senior management responsibilities recommended by
the Review Team will provide the opportunity to address these aspects of senior
management capability.
26
NZQA’s overall capability needs strengthening to support the implementation of
NCEA. In the Secondary Education Group (SEG) more assessment expertise is
required (this should include access to external advice). Its business support operations
urgently need to be strengthened and made less reliant on contractors. In addition SEG
should consider ways of improving its core competencies in public management.
NZQA’s operational policy capability, which is now separate from the Secondary
Education Group, requires further development. This will include a stronger
relationship with the Ministry of Education.
27
NZQA’s key planning documents, the Statement of Intent (SOI) and the supporting
business plans, are too aspirational, do not convey a sense of business priorities and are
insufficiently focused on NCEA. The 2006/2007 SOI and supporting business plans
should expressly incorporate the initiatives to strengthen the implementation of NCEA
and provide a basis for evaluating the effectiveness of these.
4
28
Finally, NZQA’s internal systems for financial management, information technology
and human resources need further upgrading and consolidating as a matter of urgency
to support a modern and high performing organisation. Much of this work is already
under way.
29
Term of reference 3.3 required the Review Team to examine the nature and extent of
involvement of, and responsiveness to, key stakeholders in the decision-making of
NZQA. The Review Team received consistent feedback that NZQA needs to be able to
engage more proactively and credibly with stakeholders and be prepared to hear and act
on constructive criticism. NZQA needs to develop ways of building high quality
stakeholder feedback loops and more effectively integrating these into its work.
30
Early communication by NZQA and the Ministry to parents and employers about
NCEA was poor. As a result the level of awareness and understanding of the new
secondary school qualifications outside of the immediate school environment is low
and could be considerably improved through targeted and ‘plain English’
communication. Many schools produce their own high quality material for parents but
a consistent approach is needed. The Review Team understands the Board has already
moved to address this issue.
31
Term of reference 3.4 required the Review Team to assess NZQA’s professional
leadership in examination processes, including the extent to which variability in NCEA
results reflected deficiencies in NZQA systems.
32
The key finding is that the implementation path chosen by NZQA to full standards
based assessment has been too steep. The safety nets that ensured stability and
reliability in the old system were displaced without comparable alternatives. Too much
reliance has been placed on the self-sufficiency of standards that are continuing to
develop and change. Teachers, examiners and markers were asked to interpret the
standards consistently before the standards were adequately internalised. This has
produced a degree of variability in the results of external assessment that is outside of
public and professional tolerances.
33
In contrast, the internally assessed achievement standards show far less variation of
results, although this picture is a national one and does not assure high consistency
from school to school or teacher to teacher. This has led to questions about the
sufficiency of the moderation applied.
34
The Review Team acknowledges that NZQA is already taking action to reduce the
extent of variation in external assessment. These actions focus on strengthening preassessment processes.
35
The Review Team’s recommendations are intended to add to and strengthen the
positive actions that NZQA is already taking by including a number of initiatives post
assessment. The recommendations are:
•
to implement a simplified form of item testing (‘equating’) for the 2006
examination cycle. This activity identifies stable patterns from current results as
the basis for designing and pre-testing future assessment items.
5
•
to define and bring forward normative boundaries to function as a safety net for
the four grades of Non-Achieved, Achieved, Achieved with Merit and Achieved
with Excellence for the 2005 examination cycle;
•
to withhold the release of examination results for two to three weeks to allow time
for an analysis to verify that they are fair and accurate; and
•
to continue to develop ways to report aggregated NCEA results at the subject
and/or programme level consistent with the flexible choices offered by NCEA,
that provide the quick information and potential comparisons with past
examination results that employers and the public seek.
36
Finally, the Review Team recommends a number of initiatives to strengthen the
external moderation of internal assessment by schools. These initiatives address the
perception that the current approach is not sufficient to ensure national consistency.
37
Term of Reference 3.5 required the Review Team to comment on any matter relevant
to the foregoing terms of reference. During the course of the consultation, a number of
issues concerning the design of NCEA were raised with the Review Team. The issues
were raised in good faith and the team believes that it is useful to capture them for
consideration by the Secondary Leaders’ Forum in the course of the planned Stocktake
of NCEA this year. The Review Team has briefly recorded the issues along with some
of their pros and cons, but has not arrived at a conclusion other than to note that opinion
on many of them is divided.
38
The issues recorded include:
39
6
•
the relationship between the number of separately assessed standards and overall
reliability of external assessment;
•
the fact that New Zealand is unique in having four assessment cycles with external
examinations in the last three years of secondary schooling;
•
the extent to which the curriculum is fragmented or split and how this affects
learning and assessment; and
•
the numbers of achievement standards within each subject and the number and
application of grades.
With a change of the magnitude of NCEA, refinement based on continuous review and
evaluation is not only to be expected but is also good change management practice.
Recommendations
The Review Team recommends that:
3.1 In respect of the governance, monitoring and reporting of NZQA
performance
1
The Board mandates the Acting Chief Executive to bring forward proposals designed to
ensure that the extent of variability in the results of the external assessment of NCEA in
2005 is within acceptable professional and public tolerances.
2
Following the completion of the 2005 assessment cycle, the Board mandates the Acting
Chief Executive to bring forward proposals to further strengthen the implementation of
NCEA in 2006 and beyond.
3
The Board’s capability in secondary education be strengthened either by the
appointment of a new member with experience of secondary education or through the
appointment of an assessment advisor to the Board.
4
In approaching the appointment of a new Chief Executive, the Board recognises the
damage that has occurred from the considerable instability in the leadership of NZQA
at chief executive level over the past decade.
5
The Board undertakes an analysis of the organisation’s requirements over the next three
to five years to inform the development of the job description, person specification and
performance agreement for the new Chief Executive.
6
The 2005/06 Output Agreement is varied to incorporate some additional output and
performance measures to address the following issues: reducing variability in the results
of external assessment, ensuring national consistency in internal assessment by schools
and better targeting of communication to employers and parents. In addition the Output
Agreement should incorporate output and performance measures relating to NZQA’s
policy and research functions.
7
The Ministry of Education review its monitoring capability in order to enhance the
capability of TAMU to monitor the implementation of government policy.
8
The Ministry of Education review the location of its policy and monitoring roles with a
view to effecting a better separation.
3.2 In respect of the adequacy of NZQA capability including planning,
systems and processes to undertake its role
9
The new Chief Executive as a matter of urgency undertakes a review of the allocation
of senior management responsibilities and capability with the view to better aligning
them with the needs of the organisation.
10
Immediate action is taken to strengthen NZQA’s capability to manage the business
support operations for the 2005 external assessment cycle and beyond.
7
11
Immediate action is taken to strengthen the assessment expertise within the Secondary
Education Group to support the necessary changes to the approach to external
assessment.
12
NZQA continues to develop its operational policy capability and capacity to meet future
NCEA implementation requirements, including strengthening the relationship with the
Ministry of Education.
13
The Statement of Intent for 2006/07, and supporting organisation business plan, set out
the key priority goals and actions that give effect to the initiatives to strengthen the
implementation of NCEA in accordance with Recommendation 2.
14
The Output Agreement for 2006/07 contain new output and performance measures that
monitor the effectiveness of the initiatives to strengthen NCEA implementation in
accordance with Recommendation 2.
15
The Board gives priority to upgrading NZQA’s internal systems and processes in order
to support improved performance.
3.3 In respect of the nature and extent of the involvement of, and
responsiveness to, key stakeholders in the decision-making of the NZQA
16
NZQA develop more effective ways of capturing qualitative feedback on the
implementation of NCEA through the use of focus groups, face to face interviews and
other mechanisms to enhance organisational learning.
17
NZQA ensures that it has a consistent, sustained and tailored approach to
communication with stakeholders about secondary school qualifications (with
appropriate feedback loops), particularly for parents and employers.
3.4 In respect of whether and if so how the NZQA is providing professional
leadership in examination processes
18
19
NZQA build on the actions it is currently taking to strengthen pre-assessment
moderation, by undertaking two further initiatives:
a
Implementing a simplified form of item testing (‘equating’) for the 2006
examination cycle. This activity identifies stable patterns from current results to
be the basis for designing and pre-testing future assessment items. This process
can begin once the 2005 assessment has been completed.
b
In parallel but on a longer timeframe, NZQA should examine the scope for
applying item response testing which can identify the difficulty and
differentiating ability of individual assessment items. The outcomes of this
project are unlikely to be available for practical use before 2008.
NZQA strengthen its post-assessment moderation by urgently defining normative grade
boundaries which can function as a safety net for the 2005 external assessments:
a
8
Bands of accepted tolerances should allow each grade some, but not excessive,
movement. Results showing variation beyond these bands should be brought
within tolerances unless there is a defendable explanation for that variation to
ensure that the results are fair to students.
b
As part of this activity, the indicative 70% pass rate for NCEA Level 1 should be
confirmed, and equivalent indicative boundaries be defined to identify a pass rate
for Levels 2 and 3. This will establish a marker for other grade boundaries to
guide post-assessment moderation.
20
The timing of the publication of results be determined by the principles of accuracy and
fairness. NZQA should provide advice to the Government on a timeframe for releasing
results which reasonably guarantees these principles. In practice this may extend the
current timeframes by two to three weeks (possibly into early February). Consideration
should also be given to an earlier staging of Level 3 NCEA and Scholarship
examinations.
21
NZQA continue to develop ways to report aggregated NCEA results at the subject
and/or programme level consistent with the flexible choices offered by NCEA.
22
NZQA review in 2006 the policy that schools are responsible for reporting when
students have attempted and failed to achieve, and develop a nationally consistent
approach to reporting non-achievement - of both internal and external achievement
standards.
23
NZQA revisit advice received earlier on the ways it can improve the validity of
moderation by randomly sampling student work and opening all standards to selection
for moderation.
24
To provide additional safety, NZQA employ data from the external achievement
standards to identify schools or subjects with significantly outlying results from their
internal achievement standards and focus moderation on these schools and/or subjects.
3.5 In respect of such other matters as the Reviewer Considers Relevant
25
The design issues discussed under Term of Reference 3.5 be referred to the Secondary
Leaders’ Forum for consideration as part of the Stocktake of NCEA.
9
10
Introduction
40
In February 2005 the Associate Minister of Education, Hon David Benson-Pope, asked
the State Services Commissioner to review the adequacy of the setting and management
of the 2004 Scholarship and, more widely, the performance of New Zealand
Qualifications Authority (NZQA) in the delivery of the secondary school qualifications
[CAB Min (05) 5/14]. The Associate Minister of Education requested the review under
Section 11 of the State Sector Act 1988. This request was in response to surprise at the
degree of variability of results between subjects for the 2004 Scholarship, the fact that
relatively few Scholarships were awarded, and apparent variability of results in the
National Certificate of Education Achievement (NCEA).
41
The Deputy State Services Commissioner, Tony Hartevelt, asked Doug Martin of
Martin Jenkins and Associates to lead the review. Dr Alan Barker an independent
education and public management consultant based in Hong Kong, assisted Mr Martin.
They have been supported by staff of the State Services Commission. The Review
Team has approached the Review in two parts: the implementation of the 2004
Scholarship and the performance of NZQA. The report on the 2004 Scholarship was
delivered to the Deputy State Services Commissioner on 29 April and released publicly
on 6 May 2005.
42
This report covers the second part of the review, the performance of NZQA in
delivering secondary school qualifications. Part Two of the Terms of Reference (TOR)
(attached as Annex 1) ask for a review of how well the NZQA is undertaking its role in
respect of the school sector qualifications, with particular attention to:
3.1
the governance, monitoring and reporting of NZQA performance;
3.2
the adequacy of NZQA capability including planning, systems and processes to
undertake its role;
3.3
the nature and extent of the involvement of, and responsiveness to, key
stakeholders in the decision-making of the NZQA;
3.4
whether and if so how the NZQA is providing professional leadership in
examination processes; and
3.5
any other matter which the reviewer considers relevant to the foregoing question.
4
To make recommendations concerning these matters.
43
Part Two also covers the question of variability of results in NCEA Levels 1 – 3. This
is in response to a request by the Associate Minister of Education that the Review focus
on whether the apparent variation in NCEA results both between subjects in a year, and
between years where valid comparisons could be made, reflect deficiencies in NZQA
systems. This matter is covered under TOR 3.4.
44
Recommendations are made under each of the relevant Terms of Reference (TOR).
Most of these relate to the performance of NZQA but some relate to the Ministry of
Education. Under TOR 3.5, the Review Team has summarised some common themes
emerging from the consultation relating to the design of NCEA. The issues were raised
in good faith and the team believes that it would be useful to capture them for
consideration by the Secondary Leaders’ Forum in the course of the planned Stocktake
of NCEA this year.
11
Review Process and Methodology
45
In undertaking Part Two of the review, the Review Team consulted widely within the
sector to gain a thorough understanding of the way in which different groups perceived
the performance of NZQA in delivering school qualifications. This involved talking
with a similar but slightly larger group of stakeholders than for Part One. A list is
attached as Annex 2.
46
In order to gain a first hand account from schools of the quality of their engagement
with NZQA in implementing NCEA, members of the Review Team also visited
approximately 40 secondary schools around the country. A widely representative
group of schools was chosen covering deciles 1-10, single sex and co-educational,
urban and rural, Wharekura, state and independent. As for Part One a list of structured
questions based on the Terms of Reference was sent to all those interviewed ahead of
each meeting.
47
A call for submissions was made in March 2005 with a deadline for Part Two initially
the end of April. This was extended to 31 May 2005. Forty-nine submissions were
received from teachers or groups of teachers and schools; seven from representative
organisations; five were received from parents; three from education academics; and
two from past or present employees of NZQA.
48
The Review Team met with the NZQA Board, the Acting Chief Executive, senior
managers and staff directly involved in NCEA. It also met with four former Chief
Executives and three former Board members of NZQA. The Ministry of Education, the
Education Review Office, the Treasury, the Department of the Prime Minister and
Cabinet and the State Services Commission were also consulted during the course of
the review.
49
The Review Team has engaged in some depth with a range of education assessment
academics around the country to better understand their perspectives on NCEA
implementation and to draw out their ideas and views on external assessment in
particular. The report reflects this engagement particularly under TOR 3.4.
50
In the course of this review, the Review Team has had regard to the Education Review
Office report 1 on the first two years of NCEA implementation and the PPTA report on
NCEA. 2 The findings of this review on the implementation of NCEA are broadly
consistent with both these reports.
51
This report also acknowledges and draws on where appropriate the Education Sector
Review which was released publicly on 18 July 2005.
52
Finally, the Review Team wishes to thank all those people who have contributed to this
review, either by meeting with members of the team or by sending in submissions. The
information made available has been invaluable in enabling the Review Team to look
across the spectrum of secondary education assessment and take account of at times
widely diverging perspectives. The findings of this review attempt to take a balanced
and fair approach.
1
2
Progress of NCEA Levels 1 and 2, and Readiness of Schools to Implement Level 3, February 2004
Teachers Talk About NCEA, March 2005
12
Discussion of Findings
53
The detailed findings of the Review Team are discussed below under each of the Terms
of Reference (TOR) 3.1 – 3.5
3.1 Governance, monitoring and reporting of NZQA performance
A
Governance
Current Situation
Statutory arrangements
54
NZQA is a statutory Crown entity established under Part XX of the Education Act
1989, and is subject to the relevant provisions of that Act and of the Crown Entities
Act. NZQA is a Crown agent for the purposes of the Crown Entities Act, and so must
give effect to relevant government policy directions by the Minister of Education
(Crown Entities Act s103). As a Crown agent, NZQA is expected to have a
comparatively close working relationship with the Minister and to be responsive and
proactive in giving effect to government policy.
Board roles and responsibilities
55
Under the legislation, the Board’s role is to serve as NZQA’s governing body, with the
authority, in NZQA’s name, to exercise NZQA’s powers and to perform its functions
(Crown Entities Act s 25). In carrying out this role, the Board has four principal
responsibilities. These are:
a
To appoint the Chief Executive – the Board is required to impartially select a
suitably qualified person for the position of Chief Executive and is not authorised
to delegate this power (s 254B of the Education Act).
b
To ensure that all decisions relating to the operation of NZQA are made under the
authority of the Board.
c
To set the strategic direction of NZQA jointly with the Minister of Education –
the principal instruments for this are the Statement of Intent and Output
Agreement (formerly the Purchase Agreement).
d
To carry out its duties in accordance with the Crown Entities Act 2004.
56
It is a reasonable expectation that the Board, based on good practice, would regularly
review CEO performance, although this is not specified in the legislation.
57
The Education Act 1989 requires NZQA to establish a consistent approach to the
recognition of qualifications in academic and vocational areas (s 247) and to ensure that
mechanisms are in place to guarantee that different institutions, secondary schools,
private training establishments providing nationally recognised courses have
assessment procedures that are fair, equitable, consistent and in keeping with required
standards (s 253).
58
Under the Crown Entities Act 2004, which came into effect in January 2005, the Board
and Board Members have collective and individual duties owed to the Minister, and the
entity, respectively. The individual board members duties are to:
13
59
60
•
Comply with the Crown Entities Act and the Education Act (s 53).
•
Act with honesty and integrity (s 54).
•
Act in good faith and not pursue his or her interests at the expense of the entity’s
interests (s 55).
•
Act with reasonable care, diligence, and skill (s 56).
•
Not disclose information other than in carrying out the entity’s functions or as
permitted by the law (s 57).
The collective Board duties are to:
•
Ensure that NZQA acts in a manner consistent with its objectives, functions,
current statement of intent, and output agreement (if any) under Part 4 of the
Crown Entities Act (s 49).
•
Ensure that NZQA performs its functions efficiently and effectively and in a
manner consistent with the spirit of service to the public (s 50).
•
Ensure that NZQA operates in a financially responsible manner, in accordance
with s 51.
•
Ensure to the extent of its powers that the restrictions and obligations relating to
Crown entity subsidiaries and acquiring shares or interests in other companies,
bodies corporate, trusts (etc) in ss 96-101 are complied with.
Given that these duties are new, the Review Team notes them as relevant for the further
development of governance arrangements.
Structure of the Board
61
The entity’s governing body, the Board, must have between eight and ten board
members appointed by the Minister after consulting with anyone he thinks fit “having
regard to the interests of industry, the professions, and the authorities and bodies that
are respectively responsible for providing compulsory and post-compulsory education”
(s 249 of the Education Act).
62
Currently the Board has nine members, one of whom has been designated Acting Chair
for the remainder of 2005. The Board has had five chairs including the current Acting
Chair over 15 years. The most recent Chair held the position for two years from May
2003 to May 2005.
Systems and Processes
63
14
The Board has recently approved new Terms of Reference for its activities based on the
Education Act (1989) and the Crown Entities Act 2004. The Board worked from late
2004 to refine its committee structure and now has four committees: Quality Assurance,
Audit and Finance, Human Resources, and an Appeals Committee (this latter
committee meets on an ad hoc basis only). It is also reviewing its operating procedures
to take account of the new Crown entity legislation. A Review Implementation
Committee was set up initially to oversee the changes recommended in the Board’s
independent review of the 2004 Scholarship examination (referred to as the Minter
Ellison report). 3
64
The Board meets monthly, except for January, and more frequently if required.
65
The Minter Ellison report highlighted the need for the Board to adopt best practice
governance principles. The Board has reported that the Review Implementation
Committee is in the process of examining appropriate criteria to meet this expectation.
Methodology for assessing performance of Board
66
The approach by the Review Team involved:
•
a review of the Board’s self-assessment of performance in the school sector;
•
a review of Board systems and processes in respect of the relevant legislation; and
•
having regard for best practice governance arrangements that could reasonably
have been expected to apply.
67
The Review Team wrote to the Chair of the Board in May 2005 requesting the Board to
undertake a self-assessment of performance in the school sector. For the purposes of its
self-assessment the Board provided the team with a letter, a copy of the Minter Ellison
report on the 2004 Scholarship and the August 2004 Report on the Evaluation of the
Board conducted by the then Chair.
68
Relevant documentation was requested from the Board on the way it organises and runs
its business. This included its committee structure, terms of reference for each of the
committees and terms of reference for the Board.
69
The Review Team reviewed the draft Strategic Plan 2005-2008, the Statement of Intent
2005/06 – 2007/08, the Purchase Agreement to 30 June 2005 (and previous Purchase
Agreements), the selection process for chief executive appointment and the
performance review documentation between the Board and the CEO.
70
In view of the legislative roles and responsibilities of the Board, the Review Team
considered what governance criteria might reasonably have been applied by the Board
over the last few years. A paper by the Australian National Audit Office (ANAO)
offers sound guidance for public sector governance. 4 ANAO identified and defined six
core governance principles: accountability, transparency/openness, integrity,
stewardship, leadership and efficiency. These are fully compatible with the new Crown
Entities Act and have therefore been adapted to the New Zealand setting to provide a
framework for this review (shown in italics). The definitions are set out in Box 1
below. They have been slightly re-ordered to reflect the priorities that have emerged in
this review.
3
4
Report Regarding 2004 Scholarship Issues, 5 May 2005
Public Sector Governance Volume 1, Better Practice Guide, Framework, Processes and Practices, July 2003
15
Box 1: Governance Principles
Leadership sets the ‘tone at the top’, and is absolutely critical to achieving an organisation-wide
commitment to good public sector governance and management practices and the desired results.
Accountability is the process whereby public sector organisations, and the individuals within them, are
responsible for their decisions and actions and submit themselves to appropriate external scrutiny. It is
achieved by all parties having a clear understanding of those responsibilities and having clearly defined
roles through a robust structure. In effect, accountability is the obligation to answer for a responsibility
conferred. This responsibility extends across a range of concerns, including probity and ethics as well as
the effective and efficient implementation of programmes and encompasses a range of processes.
Transparency/Openness is required to ensure that stakeholders can have confidence in the decisionmaking processes and actions of public sector organisations, in the management of their activities, and in
the individuals within them. Being open, through meaningful consultation with stakeholders and
communication of full, accurate and clear information, leads to effective and timely action and stands up
to necessary scrutiny. In the New Zealand context, where the State Services Commissioner has recently
been granted authority to set minimum standards of integrity and conduct that are to apply in Crown
entities, the concept of transparency requires entities to be able to provide reasons to explain and justify
their decision, i.e. they should be able to demonstrate that they were ‘fair and reasonable’ in the
circumstances. 5
Integrity comprises both straightforward dealing and completeness. It is based on honesty and
objectivity and high standards of propriety and probity in the stewardship of public funds and resources
and management of an entity’s affairs. It is dependent on the effectiveness of the control framework,
influenced by relevant legislation and ultimately by the personal standards and professionalism of the
individuals within the entity. It is reflected both in the entity’s decision-making procedures and in the
quality of its financial and performance reporting. In the New Zealand context, the State Services
Commissioner’s interest in Crown entity integrity and conduct focuses on how they act as instruments of
executive government, and in how their action (or inactions) affect the public’s trust not only in each
entity but in government as a whole.
Stewardship – Public officials must answer for the exercise of their powers to Ministers, who in turn are
democratically answerable to the nation. The resources they use are held in trust and are not privately
owned. Officials are therefore stewards of those powers and resources. It is important to govern public
sector organisations so that their capacity to serve government and the public interest is maintained and
improved over time. This includes financial sustainability and the efficient and effective management of
resources, as well as less tangible factors, such as maintaining the trust placed in the organisation and/or
Government as a whole.
Efficiency, that is the best use of resources to further the aims of the organisation, as agreed with
Ministers and as set out in the relevant accountability documents, with a commitment to evidence-based
strategies for improvement. Efficiency thus requires objectivity and the application of the merit
principle. For example, in carrying our public business, including making public appointments,
awarding contracts, or recommending individuals for rewards and benefits, holders of public office
should make choices on merit.
71
In developing its findings the Review Team took these principles into account. Given
the Board has the responsibility of appointing the Chief Executive as well as overseeing
a major change, the Review Team notes that the leadership principle carries more
weighting than is given by ANAO.
Findings
72
The Review Team has aligned its findings with the Board’s legislative responsibilities
listed in paragraph 55. It has also taken account of the recommendations contained in
the Minter Ellison report and the Board’s action on these to date.
5
Integrity and Conduct: Setting Standards for Crown Entities – A document for discussion with Crown entities, April
2005 (available at www.ssc.govt.nz)
16
Appointment of Chief Executive
73
Appointing and retaining a new Chief Executive is one of the most important tasks
confronting the Board over the next few months. The Board’s record over the past nine
years in recruiting and retaining chief executives with the required skills has not been
good. This is evidenced by the fact that the organisation has had eight individuals in
nine chief executive or acting chief executive positions since October 1996. This has
created instability and has meant that NZQA has not developed into a mature and high
performing organisation. This lack of maturity is reflected across all of the dimensions
of capability. This theme is expanded under TOR 3.2.
74
The Review Team acknowledges that the Board is seeking external advice on its
current selection and recruitment process to benchmark it against best practice. An
important first step in the recruitment process is for the Board to undertake an analysis
of the organisation’s requirements over the next three to five years in order to inform
the development of the position description and the person specification. The Board
must then engage in a rigorous search and selection process to ensure that a candidate is
appointed who is suited to the position.
75
The performance agreement established for the incoming Chief Executive will be
crucial in establishing the appropriate balance between governance and management
and more broadly instilling a culture of performance and accountability throughout
NZQA. The Board should review the current performance agreement to ensure that it
meets the future needs of the organisation.
Strengthening Governance and Strategic Direction
76
As noted earlier, the Board of NZQA was asked to provide its own assessment of
performance in the school sector and it did this by drawing on the consolidated findings
of a self review undertaken in August 2004 and the independent review commissioned
by the Board in response to difficulties with the 2004 Scholarship.
77
In its self-assessment for this review the Board highlighted a number of measures it had
begun in order to strengthen governance. Action had started in late 2004 with the
review of the committee structure and associated terms of reference. This work was put
on hold pending the reviews of the 2004 Scholarship but has since been completed and
a new committee structure has been put in place. The Board’s Review Implementation
Committee is overseeing the changes recommended by the Minter Ellison review and is
monitoring the implementation of the State Services Commission reviews and the
decisions by Cabinet on Scholarship. The Committee has created an implementation
timeline, acknowledging that some things need to be acted on quickly and others will
take time.
78
The Board has adopted 17 measures to strengthen governance. The Review Team
strongly supports these measures and of these believes that the following list should be
given priority.
•
The need for the Board to become more strategic in focus.
•
The need for the Board to be better connected with the Minister. The Acting Chair
and Acting Chief Executive now meet regularly with the Minister.
17
•
The need to adopt best practice governance principles.
•
Ensuring tailored communication to students, parents, schools, employers and
representative groups. This is under action.
•
Consideration of whether the Board composition is appropriate.
•
Re-alignment of the Board’s role to the new Crown Entities Act.
•
Review of the Board’s approach to risk management and adopting a “no
surprises” practice throughout the organisation.
79
The need for the Board to take a more strategic focus is of particular importance. A key
finding of the Review Team is that the Board’s principal focus has been on the sheer
logistics of implementing NCEA. This is a massive task involving four levels of
external assessment and the production of 1.9 million individual examination booklets
for 150,000 students each year. The business operations risks were considerable and the
focus of the Board on the management of these risks was understandable. However,
the broader change management issues that arose during this period of major transition
for teachers, students and parents did not get the attention they needed. Neither did
internal organisational development issues.
80
In particular, the Board failed to appreciate that the broadly defined policy settings for
NCEA afforded NZQA considerable discretion in determining both the approach to
implementation and the speed of transition to full standards based assessment. Nor did
it appreciate the need for caution in implementation, given the sheer scale of the
change, the relative paucity of resources for teachers in the initial stages (e.g. properly
moderated exemplars), and the fact that the development of standards were at an early
stage and in some cases would be subject to quite major revision.
81
Crucial decisions about the approach to implementation were taken by officials within
the Secondary Education Group without reference to the Executive Management
Committee, the Chief Executive or the Board. The lack of stable leadership at the Chief
Executive level in NZQA since 1996 and the uncertainties created by continual changes
in the allocation of senior management responsibilities probably contributed to this by
encouraging a ‘silo’ mentality to flourish.
82
The implementation path chosen was a very steep one. The safety nets that had
developed under the old system to ensure a degree of stability in assessment were
stripped away and not replaced by more appropriate ones. Undue reliance was
therefore placed on the efficacy of standards that were still in the early stages of
development and subject to varying interpretations. Little regard was paid to the fact
that the transition to full standards based assessment would take five years or more, and
that a flatter transition path was therefore warranted. The primary consequence of this
was a degree of variability in external assessment results that is outside of both public
and professional tolerances (see the discussion under TOR 3.4).
83
During the course of the review, the team has met frequently with the Acting Chair of
the Board (Catherine Gibson), the Acting Chief Executive (Karen Sewell), and the
Board Member with particular expertise in secondary education (Margaret Bendall).
The purpose was to gain a common understanding of the issues associated with
NZQA’s approach to assessment and the measures necessary to address those issues.
The engagement has been very constructive and the Review Team is confident that the
18
Board will provide the Acting Chief Executive with the necessary mandate to bring
forward initiatives that will ensure that the extent of variability in the external
assessment of NCEA in 2005 is within acceptable tolerances and more generally further
strengthen the implementation of NCEA in 2006 and beyond to better reflect the nature
of the transition. The refinements to the approach to assessment that the Review Team
believes are necessary are identified under TOR 3.4.
84
The Review Team therefore recommends that the Board mandate the Chief Executive
to bring forward initiatives to ensure that the extent of variability in the external
assessment of NCEA in 2005 is within acceptable tolerances. Following the
completion of the 2005 assessment cycle, the Board should mandate the Chief
Executive to recommend ways of further strengthening the implementation of NCEA in
2006 and beyond.
85
The foregoing reinforces the importance of the Board stepping back from involvement
in operational issues and assuming a more strategic focus. As well as more effectively
managing the Crown’s risks in implementing such major change, it would send the
right signals to people with the appropriate skills and experience who might consider
applying for the position of Chief Executive.
Board Membership
86
Notwithstanding the difficulties with the implementation of NCEA, the Review Team
is impressed with the overall calibre of the Board and the skills and experience each
member brings. Feedback from individual Board members suggests that the Board still
has some way to go in functioning as a unified team, but this aspect of Board capability
is clearly improving.
87
The Board needs to strengthen its governance of the implementation of NCEA. In light
of this the appointment of an additional member with experience in NCEA should be
considered. Up until comparatively recently, the Board included two members with
contemporary experience in secondary education, but the departure of one of those
members on the expiry of her term has reduced the complement to one. Alternatively,
the Board could augment its capability by appointing an assessment advisor.
Alignment with the Crown Entities Act
88
The Board also needs to ensure that its operations are fully aligned with the new Crown
Entities Act that came into force in January 2005. The Review Team also suggests that
in seeking to adopt best practice governance arrangements that the core public sector
governance principles set out in Box 1 provide a sound basis for the Board.
B
Monitoring and reporting
Current situation
89
NZQA’s performance is monitored by the Tertiary Advisory and Monitoring Unit
(TAMU). TAMU was established in early 1998 as part of the Ministry of Education. It
is located in the Tertiary, Curriculum, Teaching and Learning Group. This group
combines tertiary and secondary education policy and operations, spanning a large
number of functions.
19
90
TAMU is responsible for monitoring the performance of both the Tertiary Education
Institutions (TEIs) and the education sector Crown entitites on behalf of the Minister of
Education. It concentrates largely on Crown risk. The nature of this risk varies
significantly between these types of organisations. For example, the main focus of TEI
monitoring is financial, reflecting the very substantial Crown investment in those
organisations. On the other hand, the principal risks for the education sector Crown
entities such as NZQA relate to the implementation of government policy, which is
much more qualitative in nature.
91
TAMU also provides the Minister with advice on the selection and appointment of
Board members and more broadly with the building of capability of boards. The
emphasis is more towards the needs of the TEIs.
92
The Minister sends a Letter of Expectation to the Chair of the Board at the start of the
accountability cycle in February/March each year. TAMU monitors the formal
accountability documents, that is, the Statement of Intent (SOI) and the Output
Agreement, against the Letter of Expectation. In practice this is carried out through an
iterative process of engagement between NZQA and TAMU, with TAMU commenting
on successive drafts of the SOI and Output Agreement.
93
NZQA reports quarterly to the Minister as well as annually. NZQA sends its draft
reports to TAMU for comment before they are finalised and signed off by the Board.
Once the report is completed, TAMU writes a covering submission to the Minister
commenting on the report. TAMU drafts a response from the Minister based on this
analysis to the Chair of the Board, to provide formal feedback to NZQA.
94
The need for sound relationships is integral to the iterative process used to develop and
monitor the accountability documents. Recently a senior level relationship manager
has been appointed by the Ministry to augment TAMU’s expertise. This should aid the
move to a more qualitative and less compliance focused approach. NZQA has this year
also appointed a new manager of strategy and planning with the view to improving the
quality of the accountability documents. This person is also tasked with improving
relationships to help achieve this objective.
95
The Crown Entities Act requires agencies to strengthen their qualitative approach in the
accountability documents. In practice this will mean a shift from the more compliance
oriented Purchase Agreement to a results oriented Output Agreement. This change will
require a much stronger performance and outcomes approach and demonstration of the
actions required to achieve the desired results.
Methodology for assessing performance
96
20
There is no specific legislative mandate for the monitoring regime, except in the most
general sense under the provisions of the Public Finance Act. The monitoring of
Crown entities across government has grown out of a need to manage fiscal risk in
agencies that are at arm’s length from Government and receive government revenue.
The Review Team, in talking with the central agencies and other departments is aware
that many departments with monitoring responsibilities for Crown entities are finding
the development of adequate ways to monitor their activities a challenging task.
Findings
97
As noted above TAMU’s responsibilities are made more challenging by the diverse
nature of the risks confronting the organisations it monitors. For the TEI’s, the principal
risks are financial, and TAMU is well placed to exercise judgements about the
management of those risks. Whilst financial risks are also relevant to education Crown
entities such as NZQA, the principal risks relate to the implementation of government
policy, and it is clear that further work is required in the development of output and
performance measures that adequately capture those risks.
98
The Purchase Agreement for 2004/2005 did not adequately reflect the sustained nature
of the implementation task confronting NZQA and the associated risks. These included
the risks of:
99
•
excessive variability in the results of external assessment;
•
stakeholders having a low level of understanding of the nature of the new
secondary school qualifications as a result of poorly targeted communication;
•
external moderation of internal assessment not being sufficiently robust to ensure
national consistency in internal assessment; and
•
NZQA being unresponsive to stakeholder concerns.
Too much emphasis was placed on quantitative measures of activity and not sufficient
on qualitative measures directed at achieving good outcomes. Qualitative measures
would better capture the risks of a major implementation task. It is not surprising
therefore that the monitoring regime failed to provide any warning signs of the
difficulties that prompted this review.
100 The Output Agreement for 2005/2006 (which replaces the Purchase Agreement) has a
wider range of performance measures that better capture these risks. However, the
Review Team recommends the incorporation of further output and performance
measures in the Output Agreement to reflect the key findings of this review. This can
be achieved by way of a variation to the Agreement. The Review Team understands
that the Chair of the Board of NZQA has signalled to the Minister an intention to seek
variations to the Output Agreement and the Minister has acknowledged this.
101 The Review Team has found that there is an excessive degree of variability in the
results of some of the externally assessed standards and that these have arisen directly
from the systems employed by NZQA in the operation of the assessment cycle (refer
TOR 3.4). A specific output and performance measure in the examinations and
assessment section (3.1.7 to 3.1.9) should be included to ensure that the extent of
variability in the results of external assessment in 2005 and beyond is within public and
professional tolerances. This would clearly establish NZQA’s accountability for the
accuracy of the results of external assessment, and enhance the ability of the monitoring
system to detect problems at an early stage.
102 It would also be desirable to revisit the output and performance measures relating to the
monitoring and review section (3.1.5 and 3.1.6) to make it clear that the outcome
sought from external moderation is national consistency in internal assessment by
schools. The stakeholder communications section of the Output Agreement (3.1.10 to
3.1.12) should also be broadened to include effective communications with
21
stakeholders outside of the immediate school environment, notably parents and
employers whose level of understanding of the new secondary school qualifications is
low.
103 Finally, there is no reference in the Output Agreement to the policy and research roles
of NZQA. As noted in TOR 3.4, research directed at evaluating both the outcomes of
internal and external assessment under NCEA and the effectiveness of different
approaches to assessment is vital to achieving continuous improvement in NCEA
outcomes.
104 The Review Team recommends that in order to make the qualitative improvements in
the Output Agreement described above, the Ministry of Education review its capability
within TAMU. While the new emphasis on relationship management will help there
will need to be a corresponding move to enhance the capability of TAMU to monitor
the implementation of government policy.
105 The Review Team also observes that the monitoring and policy advice role in
secondary education is currently located within the same group, the Tertiary,
Curriculum, Teaching and Learning Group of the Ministry of Education. The Group’s
policy advisers on secondary education also have major responsibilities in the
implementation of NCEA and work alongside NZQA officials in the Joint Overview
Group, the Secondary Leaders’ Forum, and other collaborative groups. As noted in the
report on the 2004 Scholarship, the effectiveness of NZQA is at least in part a function
of the effectiveness of Ministry of Education in managing and contributing to the
interface. This creates obvious tensions with the role of the Ministry in monitoring the
performance of NZQA on behalf of the Minister. The relationship between the two
organisations in secondary education is not ‘arms length’, but rather they are ‘joined at
the hip’.
106 It is acknowledged that synergies can be gained by TAMU having close access to
policy and that TAMU operates independently in the formulation of its advice to the
Board and the Minister. Nonetheless the Review Team believes that the Ministry
should give further consideration to the way in which it can better separate its
monitoring and policy advisory roles in the implementation of NCEA.
107 On a related note, the recent and parallel Education Sector Review highlighted two
governance issues. Specifically, the Review recommended that “Treasury and SSC
should investigate, by October 2005, options for greater support for departments which
‘host’ Crown entities to develop selection strategies and provide induction training and
information for board members, and for improvement in the quality of monitoring by
those ‘host’ departments. (This is wider than the education sector.)” This Review
Team supports both those initiatives in relation to the Ministry’s monitoring of NZQA.
22
3.2 The adequacy of NZQA capability including planning, systems and
processes to undertake its role
Methodology for assessing performance
108 In approaching this Term of Reference a common methodology was applied across all
dimensions of capability. As for the Board, the Review Team requested that senior
management undertake a self-assessment to inform this part of the review. The Review
Team also met with the senior managers as a group and individually with those senior
managers who have particular responsibility for school sector activities. The Review
Team sought the views of a wide range of sector stakeholders on the performance of
NZQA and these views are reflected in the discussion that follows. The Review Team
had regard to the core governance principles identified in TOR 3.1 and the extent to
which these principles underpin performance.
109 Capability was analyzed across several dimensions: people capability, planning and
systems.
A
People Capability
Current Situation
110 NZQA has nearly 400 people working on its staff. 281 of these are Full Time
Equivalent staff and 96 are contractors. The average length of service for permanent
staff is 3 years and 2 months. Turnover is currently 18.1% based on resignations only.
Turnover (on the same basis) has been consistently high over the last five years,
peaking in 2002 at 26.6%. 6 There is a heavy reliance on contractors especially in
business support and specialist areas such as information technology.
111 The broad allocation of responsibilities at the senior management level is depicted in
Chart 1 below. The Board noted and supported this allocation, which on the
organisation chart shows nine direct reports to the Chief Executive, in November 2004.
Although the two newest appointments were made in April 2005 the new arrangements
have never been fully implemented. Currently there is one vacant position (General
Manager Corporate Services) and two of the senior roles have not been confirmed by
the Board and are filled by acting managers. These are General Manager Strategic
Development and General Manager Approvals, Accreditation and Audit.
6
The average for the Public Service is 12% but this includes retirements, dismissals and deaths. A consistent level to
aim for would be between 15-18% based, on similar agencies.
23
Figure 1: Organisation Chart
NZQA Board
Acting Chief
Executive
Board Services and
Audit
Executive Assistant
Chief Advisor Maori
GM
Policy, International
& Research
(Acting GM)
Strategic
Development
GM
Corporate Services
(Vacant)
Manager
Communications and
External Relations
CFO
Finance
GM
Secondary Education
GM
Tertiary Education
GM
Qualifications
Services
(Acting GM)
Approvals,
Accreditation and
Audit
Director Human
Resource Services
CIO
Information Services
112 Currently there are 12 direct reports to the Chief Executive. This is because the
vacancy in the Corporate Services Group Manager position has resulted in three third
tier managers in the corporate area reporting directly to the Chief Executive. Prior to
the introduction of the current structure, an interim structure had been in place for a
period of time. The Executive Management Committee comprises eleven of the direct
reports to the Chief Executive plus the Communications and External Relations
Manager.
113 The Secondary Education Group (SEG) was established in 2001 in order to focus on
the introduction of NCEA. Prior to this secondary education was part of the National
Operations Group. Currently the Secondary Education Group has 66 FTEs, five fixed
term employees and 13 contractors. The turnover rate is less than the whole of
organisation rate at 10%. The average length of service is also higher than the
organisation average at four years and two months.
114 Each of the four business units which make up SEG has a manager who reports to the
Group Manager. The units are:
•
Schools liaison, Monitoring and Reporting.
•
Assessment.
•
Secondary Operations and Development.
•
Customer Resource Services.
115 The Review Team has focused the capability of two groups of people for NCEA
implementation. The first group comprises the key internal staff in SEG who liaise
direct with schools and provide the mission critical support functions for external
assessment. These are the School Relationship Managers (SRMs), the National
Assessment Facilitators (NAFs) and the Development and Operations staff. The second
24
group comprises the external professionals contracted to provide the expertise for
internal and external assessments.
116 Table 1 below profiles the current SRM and NAF teams.
Table 1: Profile of NAF and SRM Team
Number
SRM
NAF
15
12
Average Years Teaching
21.28
17.60
Average Length of Service
4.81
3.50
117 As can be seen in Table 1 the SRMs have on average 21.28 years of teaching
experience and there is a reasonable degree of stability within the team. More generally
the feedback from schools is that the SRM initiative has been very effective. Schools
almost without exception reported to the Review Team that their SRM was helpful,
prompt in answering queries and had made a large contribution to a better relationship
between the schools and NZQA. The role of the SRM team and how it might be further
developed is further discussed under TOR 3.3.
118 Table 1 also shows that the depth of teaching experience amongst the NAFs is sound –
on average 17.6 years. The principal role of the NAFs is to facilitate the external
assessment system and processes. This role does not, however, appear to be well
understood by the sector. Consistent feedback from schools was a requirement for
access to subject expertise which they expected would be available from the NAFs.
However the NAFs are not recruited to be subject experts per se (see TOR 3.3 for
further discussion of this matter). The NAF and SRM teams are currently being
strengthened through the appointment of additional staff.
119 In the Secondary Operations and Development unit there is a heavy reliance on
contractors. For example, in the Development Team 11 out of 12 positions are filled by
contractors. There is now a strategy in place to reduce the reliance on contractors and
appoint staff into permanent positions. Turnover of permanent staff is also a problem.
120 The external group comprises a large number of professionals (predominantly teachers)
who are contracted to set the external assessments (examination papers) and mark
student scripts. Another group is contracted to moderate the internal assessments to
ensure national consistency. In 2004 SEG contracted on a part time basis 100
examiners, 300 Panel Leaders (who coordinate the marking across the country) and
1700 markers. Currently there are 358 moderators for SEG to manage who are also
contracted on a part time basis. These functions are managed by the Assessment unit.
121 The Review Team also reviewed the profile of a sample of contracted professionals
who mark exams and moderate internal assessments. The Review Team focused on the
subjects most frequently mentioned as being problematic in submissions and during
school visits. These were: English, Biology, Geography, and Technology. The profile
reveals that the professionals concerned have extensive teaching experience in their
subjects and in the marking of exams and moderation of internal assessment processes.
This is illustrated in table 2 below.
25
Table 2: Profile of professionals involved in external and internal assessment processes for four
subjects
% of
% with
previous
Years of % Involved
%
Years
previous experience
Number
previous
in exam
7
HODs
Teaching experience
in
experience 8 setting
in this role associated
role
Panel
11
82
17.6
64
82
5.6
18
Leaders
(External)
Markers
29
38
16.5
79
97
7.1
14
(External)
Moderators
(Internal)
8
63
24.5
100
88
7.4
25
122 Many schools raised concerns about Technology in particular. When the data above
was analysed by subject, it was clear that in Technology there was difficulty in
maintaining enough depth to adequately cover at all three levels the range of its subspecialities or ‘contexts’. There are seven of these: Technology Generic, Biotechnology, Food Technology, Information and Communication Technology,
Electronic and Control Technology, Materials Technology and Structures Technology.
The external assessment of Technology has only one panel leader and six markers for
the three levels of assessment and the seven ‘contexts’.
Findings about people capability
123 The main finding under people capability is that the lack of stable leadership at Chief
Executive level over nine years has had a major impact on the performance of the
organisation. This has affected the ability of NZQA to develop into a mature and high
performing organisation across virtually all of the dimensions of organisational
capability. It is acknowledged that many of the issues highlighted above are now being
addressed by the Board and the Acting Chief Executive.
Some general observations on culture
124 The Review Team found that the culture of NZQA has been shaped by constant change
and uncertainty in its leadership. One comment made to the Review Team was that
there are as many sub-cultures in NZQA as there have been Chief Executives.
Certainly there appear to be a number of different ‘worldviews’ that are determined
either by belief or by professional background or both. This translates at lower levels
of the organisation into inconsistency of approach. There is no doubt that NZQA staff
members are very committed to what they do and to improving outcomes for learners in
New Zealand. What is missing are a clear corporate interpretation of NZQA’s role as
the result of a sustained internal and external debate, and a complementary focus on
performance and individual accountability.
125 The dissension in some quarters that has surrounded the introduction of NCEA has led
to a somewhat defensive culture. While this is understandable in human terms it is
7
Full details on years of teaching experience were not available. Details were available for 73% of panel leaders,
66% of markers and 25% of moderators.
Full details on years of previous experience were not available. Details were available for 100% of panel leaders,
76% of markers and 63% of moderators.
8
26
important that staff move beyond this now and get on top of the issues discussed in this
report. Not all staff members seem aware that they are State Servants working to
implement government policies and therefore directly accountable to the government of
the day through their Board. Where the implementation of NCEA has eroded public
confidence, it is the staff who have direct individual responsibility to restore that
confidence through high quality analysis, action and excellent stakeholder management
(see the discussion under TOR 3.3). This change must be led and modelled by the
Board and by senior managers.
Senior management capability
126 A direct consequence of the lack of stable leadership at Chief Executive level has been
instability in the allocation in senior management responsibilities. For much of the
period of the implementation of NCEA an interim structure was in place and this
created uncertainty within the organisation. This has meant NZQA has tended to
operate in silos at the expense of the whole organisation view and different sub-cultures
have developed. There is very little focus on performance or acceptance of individual
accountability.
127 Despite a recent re-alignment of senior responsibilities the current allocations are
unwieldy and not well aligned to NZQA’s strategic direction. There are too many
direct reports for an organisation the size of NZQA and there is insufficient recognition
of business priorities. The Board in supporting the current allocation of senior
management responsibilities expressed concern about the number of direct reports.
128 The senior management meets as an Executive Management Committee (EMC). This
committee has functioned at the operational rather than the strategic level. One of the
major delivery tasks of NZQA, the implementation of NCEA, has not been addressed
collectively by EMC. Crucial strategic decisions on the approach to implementation
have been made within the Secondary Education Group. The burden of implementing
NCEA has therefore been carried disproportionately by middle management.
129 The extent of public sector management experience in the EMC is variable and needs
strengthening, as do the basic skills in management through development programmes.
130 The Review Team therefore believes that there is a need to further review the senior
management allocations. This will provide the opportunity to address the various
issues of capability at the senior management level. However, this review should await
the appointment of the new Chief Executive, and proceed as a matter of priority after
that.
Capability of the Secondary Education Group
131 As noted above, the technical competence of the professional staff within the
Secondary Education Group (SEG) is good. The profile of both SRMs and the NAFs
demonstrates that both groups have considerable depth of teaching experience. Both
groups are in the process of recruiting additional staff to further enhance their
capability, in response to the Report on the 2004 Scholarship. As the NAFs are not
recruited as subject experts, NZQA will need to work through how to fill this gap as
well as meeting its own immediate business needs to manage the large number of
professionals it requires to run four sets of external assessments each year.
27
132 SEG needs more assessment expertise (this should include access to external advice) in
order to give effect to the recommended changes in the approach to external
assessment, and to position the Group to better analyse the initial results of external
assessment and take corrective action. Beyond this, the main technical capability gap is
a lack of consistent access by schools to subject expertise. SEG, in conjunction with
the Ministry of Education, needs to explore how to best meet this need.
133 The Secondary Operations and Development unit, which provides the business support
processes for the external assessment of NCEA, has critical capacity and capability
gaps due to the reliance on contractors and turnover of permanent staff. NZQA has
identified and is putting in place a new structure to stabilise the unit, reduce reliance on
contractors and ensure that crucial gaps are back-filled. The Review Team supports the
action that is being taken to strengthen NZQA’s capability to manage the business
support operations for the 2005 external assessment cycle and beyond.
134 SEG needs to manage a significant number of contracted professionals (examiners,
markers, moderators) to support the external and internal assessment processes. This is
a significant number of professionals to be managed during a very compressed time
period, requiring not only very efficient systems and processes but also very clear
directions about how to carry out their roles and the outcomes expected.
135 In addition SEG should consider ways of improving its core competencies in public
management.
Operational Policy Capability
136 Beyond the Secondary Education Group, NZQA’s operational policy capability needs
to be further developed to support the implementation of NCEA, including
strengthening the relationship with the Ministry of Education. In the report on the 2004
Scholarship, the Review Team commented on the operational policy capacity of NZQA
(paragraph 34). NZQA had substantial policy capacity when it was established but this
capacity was progressively reduced from the 1990s because it created tensions with the
Ministry’s overarching responsibility for policy. This left NZQA with little or no
capacity to undertake analysis and develop options for managing complex operational
policy matters and risks.
137 A Policy, International and Research Group has now been established and capability
and capacity issues are being addressed. The Review Team supports this move to
strengthen operational policy capability and capacity. This is an area that will need to
be monitored on an ongoing basis to ensure that the operational policy requirements of
NZQA are being met.
B
Planning
Current situation
138 NZQA’s planning processes include the formal accountability documents, the
Statement of Intent (SOI) and the Output Agreement, and the business plans produced
by separate groups. Currently the Board also produces a three year Strategic Plan for
internal purposes.
139 In reviewing the planning documents, the Review Team looked for a clear sense of
what NZQA wanted to achieve and why (links to its legislation), its priorities, how it
28
would go about achieving these and over what timeframe, how it would assess risk, and
how it could evaluate its progress.
Findings about planning
140 The Statement of Intent (SOI) 2005/06 – 2007/08 is beginning to be more focused on
results than previous SOIs but the Review Team believes that this document needs to
become less aspirational. Instead it must identify specific business objectives and
activities that demonstrate how NZQA intends to achieve results. It also needs to
identify priorities and not try to do too much. NCEA implementation needs a much
more prominent place in the SOI.
141 The SOI 2005/06 – 2007/08 includes a Foreword by the Minister that sets the strategic
context: “a high quality education system is critical to our place in the global
knowledge economy and to our domestic social and economic wellbeing.” This is
followed by a Foreword by the Chair that is largely focused on how the organisation
intends to work over the two years covered by the document. It refers to the two
external reviews on the 2004 Scholarship and the way in which their findings have been
integrated into the SOI. The Government Goals are listed and there is a section on
environmental influences. The SOI also includes a section on risk management.
142 There is a section on the outcomes that NZQA wants to achieve and how they link to
the relevant Government Goals. An overarching outcome (“All learners achieving to
their potential”) and three intermediate outcomes have been identified. This is followed
by four strategies, which are new, and three output class statements. The Output
Classes remain the same as the previous year. The four strategies are:
Strategy 1: Drive Excellence and Quality in All Areas of Delivery.
Strategy 2: Provide Strategic Leadership to the Sector.
Strategy 3: Enhance Confidence and Trust in the Work of the Authority
Strategy 4: Increase Use of and Access to Knowledge and Research.
143 The Review Team supports the focus on effective business processes and systems in
Strategy 1, the focus on working to better communication with stakeholders in Strategy
3 and the emphasis on knowledge and research in Strategy 4. Strategy 2 however,
seems to be more the business of the Ministry of Education. If it does its part well in
delivery secondary school qualifications, NZQA will be providing leadership
appropriate to its role in the sector. It is acknowledged that the strategies were agreed
by the Board in February 2005. They will need to be updated for next year in light of
the findings of this report and the findings of the Education Sector Review.
144 Overall the Review Team identified two major gaps in the document. There is an
absence of clear intervention logic which links the outcome statements, the strategies
and the outputs being delivered by NZQA. There is also an absence of the practical
programmes and activities that the organisation will take to ensure that NCEA
implementation is carried out to the standard required. NCEA is a first order priority
for NZQA and this must be reflected in its SOI and Output Agreement to a level of
detail that provides assurance that the organisation both understands and can act on the
relevant issues.
29
145 The performance measures under Output Class 03, the Administration of National
Assessment for Secondary Qualifications need to be enhanced in next year’s SOI in line
with the recommendations in this report for changes to the Output Agreement 2005/06.
The Review Team has commented in some detail on the Output Agreement above
under Monitoring and Reporting. The SOI and the Output Agreement together have
considerable potential to influence the performance of NZQA because they are
negotiated with the ‘host’ Ministry, the Ministry of Education, acting for the Minister
of Education.
146 The SEG Business Plan for 2005/06 contains a combination of a wide range of key
outputs (a total of 12) and four specific areas of focus.
147 The four areas of focus are:
•
To implement the Cabinet decisions on Scholarship 2005.
•
To report on the contributing causes of variation in candidate cohort performance
in intra subject, inter subject and inter year to the appropriate agencies and groups
to minimize any potential disadvantage to candidates.
•
More effective communication with schools (and the sector generally) on major
assessment curriculum, teaching and learning issues to emerge from the external
assessment round (i.e. around sector perception and readiness).
•
To reduce the operational input of staff and to enhance their professional
involvement in debating and acting on important assessment and communication
issues.
148 These are worthwhile additions to the business plan and demonstrate a commitment to
address some of the key implementation issues. However, no specific actions are linked
to these areas of focus. The Business Plan contains more than 30 key milestones, only
a handful of which can be linked back to the focus areas. Risks are however identified.
149 Finally the Review Team notes that the Statement of Intent should become the principal
strategic planning document developed jointly with the Minister of Education, as
required by the new Crown Entities Act. While a separate strategic plan is likely to
become redundant, it is critical that the organisation undertakes a sound strategic
planning process.
C
Systems
150 Many of NZQA’s current systems and processes have outlived their usefulness and
need to be upgraded to reflect the requirements of a modern State sector organisation.
It will be important to prioritise business objectives and to build systems which support
the delivery of these priorities in a way that is realistic and sustainable. Assessing
whether there is sufficient resource to do so will be an important task for the new Chief
Executive.
151 The Review Team examined the IT, HR, and financial management systems. The
systems are in the process of being reviewed and upgraded, although there remain some
gaps. The Review Team has chosen to reflect both progress and areas to strengthen in
the discussion below which combines both a description of the current situation and the
team’s findings.
30
Information Technology
152 The IS (Information Services) unit of NZQA has undertaken a significant challenge to
meet the business requirements associated not only with the implementation of NCEA
but also those for the wider organisation. The IS challenges in 2004 were as great for
NCEA as they were in 2002 when Level 1 was introduced. Furthermore these demands
have been met during a period of major change of the IT infrastructure. IS has played a
major role in the business planning organisation-wide over the last three years because
of the need to be clear about priorities for IT development.
153 The ability of IS managers to take a strategic approach ultimately resulted in a
successful business case to Government for capital funding to develop the new IT
system known as the e.QA programme. NZQA has been able to upgrade its systems,
essentially replacing all hardware and all applications, and has adopted a web-centric
approach to their IT needs making them, in some respects, a leader in the State
Services. Web-centric services means that there is a need for around the clock support.
NZQA’s public website has recently been judged second best in government by an
independent review. 9 This has been a huge change management exercise in its own
right which has dovetailed with the NCEA change management operation.
154 The considerable progress that has been made has come at the cost of best practice
governance and management standards. This is recognised by IS managers who freely
admit that the IS has worked under considerable pressure for three years and now needs
to move to ‘bed in’ the new systems. In the case of NCEA this is expected to take
another two to three years. The development of the new infrastructure and web-based
delivery system has required approximately twice as many staff in the IT area.
Numbers of IS staff are currently between 90 and 100, with around half of these
contractors, some of whom who are still in business critical positions.
155 Senior managers are well aware of the risks created and recently commissioned an
independent review. The report outlines the governance and development issues that
need to be addressed, some of them urgently. Senior management has accepted their
recommendations and is acting on them, a response which the Review Team supports.
There are still risks for IS as the focus of the implementation moves from development
to consolidation of the change.
Human Resources
156 Over the last 18 months there has been a considerable investment of time and energy to
upgrade HR systems. Since early 2004 the Human Resources unit has developed and
implemented a number of initiatives that are refocusing the business unit away from a
payroll and recruitment operation to a broader organisational capability and capacity
service. These initiatives include: training and coaching for managers, the embedding
of a revised remuneration system, the introduction of a performance management
system and recruitment and retention strategies. The HR Business Plan for 2005/06
continues this focus. Work is also under way on building future organisation capability
and capacity, including succession planning and leadership development.
157 The training programme for managers, as well as aimed at the new systems for
remuneration and performance management, has included such topics as coping with
9
www.e-govwatch.org.nz
31
stress, select committee training, media training, ministerial writing, team building and
induction. Twenty staff are currently completing national certificates or diplomas in
management. Training to break down silos between groups and build a more
collaborative culture is proposed.
158 Because of the high turnover and the workforce demands created by large amounts of
change management (notably for NCEA and e.QA implementation), necessitating the
high use of contractors, much of the HR unit’s energies continue to be diverted into
recruitment and retention. Also, there is evidence that not all managers have
participated in the useful training offered.
159 An important gap in HR capability however is the lack of an HR information
management system. This needs to be addressed to ensure that the senior management
team is able to make better informed decisions on issues of capability and capacity.
Scoping for this is being included in the wider Financial Management Information
System project (see below under Financial Management).
160 The largest impediment to effective HR practice remains the unstable leadership
described earlier. A new Chief Executive and their senior management team will need
to embrace and build on these initiatives to create a more highly trained, stable and
effective workforce. Once future planning and accountability documents are in place
which prioritise business goals and activities, the HR framework for increasing overall
organisational capability should be aligned to them. For SEG this will require the
alignment of competencies and experience to ensure that the right people are in place to
deliver the NCEA changes proposed in TOR 3.4.
Financial Management
161 Financial reporting and analysis capabilities at NZQA are currently limited by out of
date and inadequate information systems. This means that there is a lack of accurate
forecasting and budgeting information that can be provided to managers. Managers
have tended to respond to this situation by being fiscally conservative and risk averse
and this has in turn led to a history of financial surpluses.
162 The current Financial Management Information System (FMIS) is 14 years old and in
urgent need of replacement. This has been recognised by the Board. In December
2004 management reported to the Board that the FMIS was inadequate for NZQA’s
needs. The Board asked the Chief Executive to undertake a scoping exercise to replace
the FMIS. The first part, an analysis to determine the specific needs of the
organisation, is under way. It is due to be completed in September 2005 when it will be
reported to the Board.
163 In addition the Board initiated a Budget Process Review in December 2004. This
review identified the need for the organisation to put in place a Medium Term Financial
Strategy that has:
32
•
a clear set of business drivers;
•
base assumptions;
•
a consistent long term framework for financial planning; and
•
scenario planning.
164 The strategy is on track to be completed in September 2005 and aims to:
•
Provide a comprehensive view of NZQA’s financial position over the baseline
period (to 2009).
•
Answer the question “does NZQA have sufficient financial resources to
undertake its role?”
•
Enable NZQA to price budget initiatives to a level acceptable to central agencies
and Ministers, including sensitivity analysis.
•
Deliver a robust and defensible model for cost recovery.
•
Deliver “product costing” level information.
165 These two initiatives will be brought together in September 2005 for the Board to
consider the type of FMIS needed and to approve these specifications. The next step
will be to go to the market to explore actual options for a new system. Implementation
is planned for 1 July 2006. The new Chief Financial Officer who began with NZQA in
April 2005 is managing this work.
166 NZQA has become a major technology user and provider over the last few years. This
has placed new demands on the balance sheet and particularly on asset management.
As part of the financial strategy outlined above NZQA is planning to address
technology replacement cycles and introduce asset management over the long term.
167 The December 2004 Budget Process Review also recognised the need to ensure that all
budget holders, managers and administrative staff with budget responsibilities receive
financial management training. A training programme that is tailored to the different
needs of these groups is currently being put in place for September and October 2005.
Further training will also accompany the introduction of the new FMIS, which will
reinforce the initial training.
168 It will be important to ensure that performance incentives in the future for managers are
geared to accuracy rather than risk aversion. The Review Team understands that senior
managers are considering ways to make this change.
169 The Review Team fully supports these initiatives to improve overall forecasting,
budgeting, reporting and analysis.
33
3.3 The nature and extent of the involvement of, and responsiveness to, key
stakeholders in the decision-making of the NZQA
Methodology for assessing stakeholder engagement
170 The Review Team consulted widely with stakeholders in the school sector and beyond
to inform all aspects of this review. Structured questions based on the Terms of
Reference were provided ahead of each meeting, tailored to the different groups. There
were broadly four groups that the Review Team engaged with in some depth. These
were schools, education specialists, representative organisations such as the PPTA and
NZSTA, and employers’ groups.
171 Stakeholders raised a number of important professional issues, some of them highly
technical. These are discussed under TOR 3.4, where several are the basis of
substantive recommendations by the Review Team. Some of these are policy matters
and have been noted under TOR 3.5.
Current Situation
Schools
172 Members of the Review Team visited approximately 40 schools around the country.
The coverage was diverse, as set out in Review Process and Methodology, and reached
approximately 10% of all secondary schools. The Review Team met with Principals,
Principals’ Nominees and Heads of Departments.
173 The majority reported that in their experience NCEA is enhancing learning outcomes
for students and improving teaching practice, even at this early stage. In a minority of
schools teachers made it plain that they had opposed the introduction of standardsbased assessment. Some of these schools have since revised their views and are now
more positive; others are still reluctant players. Schools however, regardless of their
stance on assessment, recognised that NCEA is here to stay, and there was a degree of
convergence on the issues that need to be addressed.
174 The introduction of NCEA has required schools to put in place considerable
administrative resource and support systems like IT upgrades to handle the data
transactions. Most schools are handling this well. They reported that the tangible
benefit of the rich data and information on student learning and teaching available to
schools justified the administrative load. The data, however, needs to become more
reliable before its full potential for school performance is realised (see TOR 3.4).
175 Two features of NCEA implementation impressed the Review Team. Schools reported
that the introduction of School Relationship Managers (SRMs) has improved the
relationship between schools and NZQA considerably. Several schools suggested that
more ‘face time’ with the SRM would be worthwhile, say three times a year (February,
July and October) instead of once a year in February. This could be regional and would
be an excellent way of sharing professional experience and ideas between schools.
176 The second feature is the position of Principal’s Nominee which is a pivotal role in
each school. This person is typically a senior teacher and member of the school
management team, who is responsible for the interface with NZQA. The Principal’s
Nominee carries administrative, facilitation and professional leadership responsibilities
34
and has become essential to the smooth running of NCEA. Where the Principal’s
Nominee was proactive, positive and energetic the school benefited greatly. One
suggestion was that NZQA could establish and publish a profile of an effective
Principal’s Nominee.
177 Consistent feedback from schools was a requirement for better access to subject
expertise. Schools tend to look to the NAFs for this expertise. This is somewhat
problematic as the NAFs are employed as facilitators to manage the professional side of
running external assessments, which is a huge task. This is discussed further under
findings.
178 Many schools reported that they tried to teach and assess too many standards in the
beginning and have now pulled back. For example they are more likely now to aim for
20 credits per subject rather than 24. Several teachers observed that the experience
with the Sixth Form Certificate was similar and it took several years to settle into a
realistic pattern.
179 Another major change for schools is the frequency with which they need to interact
with both the Ministry of Education and NZQA. There was concern in some schools
that the central education agencies do not recognise the extent of the requirement they
put on schools and the timing of this interaction needs to be fine-tuned. Two positive
suggestions were made:
•
NZQA should aim to release examiners’ reports by March each year so that
adjustments to teaching practice can be made. Any later and the opportunity to
make changes for the current year on this feedback is diminished.
•
Updated standards and moderated exemplars be provided in a timely fashion. It
was suggested that there should be a cut off date of no later than October in the
preceding year. 10
180 Schools have taken an important role in developing an understanding of NCEA
amongst parents. Many but not all schools prepare comprehensive written material on
NCEA for parents. In light of negative public perceptions, several schools reported the
difficulties they were experiencing in communicating the details of NCEA and indeed
the advantages for students. In general schools would like more assistance in this area.
Education specialists
181 This group included academics, researchers and education measurement experts.
Almost all of the comment received by the Review Team, oral and written, is reflected
in TOR 3.4 or TOR 3.5. Some additional themes highlighted by this group are worth
recording, principally for NZQA and the Ministry to consider how best to act on them.
These include:
•
10
While NZQA has exercised very good operational leadership, conceptual
leadership of NCEA has been missing. NZQA needs to extend its capability to
Early on in NCEA implementation unmoderated standards and exemplars were posted on the web and these caused
confusion and frustration for teachers when student work was criticised for not reflecting the most recent standard
and/or exemplar. This latter problem seems to be largely historical as the Ministry and NZQA now have integrated
the NCEA website. Changes to standards and exemplars are systematically signalled in the newsletter to Principals’
Nominees.
35
focus on the big picture and to ask critical questions whilst at the same time
managing operations.
•
More assessment expertise is needed in NZQA to create a critical mass of
capability in that area and more use should made of external assessment experts.
•
The Ministry needs to ensure that it has sufficient curriculum experience to be
credible in each subject.
•
The standards in some subjects need to be strengthened. This will require the
curriculum in that subject to be fully up to date, as the two are linked.
•
NZQA and the Ministry need a stronger ‘customer focus’ when liaising with the
sector.
Representative groups
182 The Review Team met with a wide range of sector groups (see Annex 2 for the full
list). Most of the comments from these groups mirrored the comments from schools
and have also been incorporated into TOR 3.4. An additional view was that the School
Relationship Managers as a group could more systematically provide an overview of
what is happening in schools at any given time on any given topic. This information
could feed into the analysis on operational policy matters.
183 Another matter that was raised by stakeholder groups, but interestingly not by schools,
was the matter of university entrance. Two issues are relevant. The first is a concern
that the curriculum is being fragmented by the separate assessment of achievement
standards which may be compromising the preparation of students for university study.
The second is that the variability in the results of external assessment is creating a less
predictable environment for universities in selecting top entrants to limited entry
programmes.
184 A joint secondary and tertiary group is currently reviewing the common standard for
entrance to university to ensure that it is fit for purpose. The group is reviewing the
scope and breadth of the Achievement Standards currently on the approved list that
contribute to the overall standard of entrance and will make recommendations on any
modifications.
185 The Review Team notes that most schools it visited are tightly managing courses for
individual students to ensure that they meet the requirements for entrance into
university.
Employers
186 The employers groups support the principle of NCEA because it provides better
pathways into further education and into the workforce. However, the initial support
for NCEA has been compromised by inadequate communication to employers, who
have generally found the results difficult to interpret. As with other stakeholders, the
variability in results for external assessment is a cause for concern, as is the risk of any
lack of national consistency in internal assessment. Employers want to be able to rely
on the rigour of the school leaving qualification irrespective of the school attended.
36
187 Employers acknowledge that schools with students who traditionally were regarded as
poor performers under the old system are performing better under NCEA because there
are more opportunities to succeed.
Findings
188 Overall the Review Team found that there was a strong desire expressed within the
school sector and by all of the representative and stakeholder organisations to address
the issues that have emerged from the first three years of implementation of NCEA and
move on.
189 The Review Team received consistent feedback that NZQA needs to be able to engage
more proactively and credibly with stakeholders and be prepared to hear constructive
criticism. It is not just a matter of stakeholders ‘not understanding’ NCEA, although
some communications have been insufficiently well targeted to their intended
audiences. Rather, NZQA has not always viewed the feedback as a constructive
contribution to organisational learning. NZQA also needs to be careful that it seeks
qualitative, nuanced feedback. The use of written questionnaires is not the best method
for this. The Review Team recommends that NZQA develop ways of seeking
constructive, qualitative stakeholder feedback and views that it can use to refine its
systems.
190 As noted above many schools indicated that they wanted better access to subject
expertise. Teachers look to the National Assessment Facilitators (NAFs) for this
expertise although the primary function of the NAFs is to ensure the smooth running of
external assessments. In some cases this may mean facilitating access to subject
expertise. NZQA, in conjunction with the Ministry of Education, will need to address
this matter. Some expertise may be able to be drawn from subject associations but any
such arrangement would need to ensure that the advice is consistent. This is discussed
further under TOR 3.4
191 Early communication by NZQA and/or the Ministry to parents and employers was poor
with the result that the level of awareness and understanding of the potential of NCEA
amongst stakeholders outside of the school environment is low and could be
considerably improved through targeted and ‘plain English’ communication. Many
schools produce their own high quality material for parents but a consistent approach is
needed and NZQA could do more to facilitate this. The Review Team understands the
Board has already moved to address this issue. NZQA now has an opportunity to work
with employers’ groups to enhance their understanding of NCEA.
192 Stakeholder groups made suggestions for improved communication with the sector
regarding NCEA. Some of these relate to NZQA and others relate to the Ministry of
Education. Suggestions include:
•
NZQA should use focus groups or other more nuanced forms of acquiring
feedback on NCEA, rather than questionnaires that provide only small amounts of
qualitative information.
•
NZQA and/or the Ministry should undertake research into student experience
with NCEA and the pathways they are choosing to support the anecdotal evidence
that students have more choice and are gaining better learning outcomes than
under the old system.
37
38
•
Although there was a lot of positive feedback about the NCEA website, some
schools felt that in time the website could be re-designed to be more ‘customer’
friendly.
•
Feedback loops could be improved, e.g. schools provide comment to the Ministry
as input to the review of standards but receive little or no feedback about the
actual changes made.
3.4 Whether, and if so how the NZQA is providing leadership in examination
processes
193 The Associate Minister of Education sought assurance that the review would assess
whether the apparent variation in NCEA results both between subjects in a year, and
between years where valid comparisons could be made, reflect deficiencies in NZQA
systems. This issue lies at the heart of NZQA’s leadership of examination processes.
194 The Review Team has found that there is an excessive degree of variability in the
results of some of the externally assessed standards and that these have arisen directly
from the systems employed by NZQA in the operation of the assessment cycle which
have not allowed for a smooth transition to full standards based assessment. The team
also believes that the system of external moderation of internal assessment by schools
needs strengthening to ensure national consistency.
195 In approaching this Term of Reference the Review Team found it necessary to examine
in some depth technical assessment issues. The Review Team has therefore:
•
drawn on analysis undertaken by Professor Terry Crooks 11 on the variability of
NCEA Level 1 results in selected subjects;
•
discussed the extent and causes of variation in the results of externally assessed
achievement standards; and
•
identified approaches to managing this variation in the future.
196 The Review Team also addressed the issue of whether the current system of external
moderation of internal assessment is sufficiently robust to ensure national consistency.
Feedback from schools identified this as a matter of some concern.
197 As noted in Part One, the introduction of NCEA represented an unprecedented change
in teaching, learning and assessment in New Zealand’s senior secondary schools. It
must be acknowledged that the implementation of NCEA has been a massive
undertaking and NZQA deserves credit for what has so far been achieved, as does the
teaching profession. Given the scale of the change involved with the introduction of
NCEA in 2002, the Review Team believes that the implementation path should extend
over five years or more. Within this period, interventions are required that recognise
the time it takes for the achievement standards to be fully developed and understood.
Purpose and History of Assessment
198 The Review Team has been struck by the relatively low level of understanding by the
key stakeholders of the new secondary school qualifications. The reasons for the shift
in the approach to assessment in New Zealand secondary schools since the 1960s are
therefore summarised below. Before that, it is worth briefly considering the purpose of
assessment.
199 Assessment performs two main functions: assessment of learning and assessment for
learning. Assessment of learning (often described as summative assessment) measures
a student’s current achievements and capabilities. It takes a snapshot at a point in time.
A series of such snapshots can be used over time to map progress and identify further
11
Professor Crooks is Co-Director of the Educational Assessment Research Unit at the University of Otago.
39
learning needs. By contrast, assessment for learning (often described as formative
assessment) aims to provide feedback to enhance student development. It is usually
continuous in delivery and often less structured than a planned formal assessment
event.
200 In practice the two forms of assessment work together, but assessment systems tend to
favour one over the other. Until the 1960s New Zealand’s senior secondary system
emphasised summative assessment primarily to select students who would proceed to
university and comparatively less attention was paid to formative assessment. The
summative assessment was predominantly norm-referenced and focused on the relative
performance of different students in examination events which suited its selective
function. The results were predictable (reliable), partly because the proportion of
students who would pass and the spread of results were pre-determined by the normreferenced model. 12
201 This emphasis suited the circumstances of the time. National examinations served as
filtering devices. Those students who remained in school typically sat the School
Certificate (SC) examinations, usually in four or more subjects. The exams were
marked on a percentage correct basis and subject marks were statistically scaled against
each other so that the total group of students taking any pair of subjects had similar
mark distributions in those two subjects. A further scaling applied to all subjects to
achieve an agreed mean mark, and a desired spread of marks (the ‘bell curve’) was
obtained. In the 1960s the mean mark was about 50% and about half the candidates
passed. University Entrance examinations and later the University Bursaries (UB)
examinations applied similar procedures. Passes at 50% or better in four subjects were
required for admission to university. Good performance in five subjects, relative to the
total pool of students who took the examinations, resulted in the award of a
Scholarship, ‘A’ Bursary or ‘B’ Bursary.
202 The strength of summative assessment lay in its capacity to select the top students for
advancement but it also had the undesirable effect of discouraging other students from
remaining at school. In the 1960s, fewer than 30% reached Form 6 (Year 12), and
fewer than 10% Form 7 (Year 13). By 1990 there was a different external environment
and more than 70% stayed on to year 12, and about 35% to year 13. Schools were
looking to develop alternative programmes to the examinations because a growing
proportion of students who remained at school did not intend to go to university.
203 This steady shift in the approach to assessment over several decades explains a lot
about NZQA’s role. It has led the change to a more comprehensive approach to
assessment, focusing assessment on the collection of evidence appropriate to establish
whether a student has acquired knowledge or can practise certain skills. This requires
the identification of the standards for knowledge and skills in the first place, and
standards-based (or criterion-referenced) assessment has largely replaced normreferenced assessment. Standards-based assessment relies on the judgements of peers
to set the standards and to weigh the evidence. It acknowledges that assessment of
human endeavour is not an objective science, though scientific disciplines can help in
the formation of evidence-based judgements.
12
The norm referenced model scaled examination results to fit a predetermined distribution (usually a bell curve).
40
204 These changes to assessment in New Zealand schools were not isolated but tied to
broader, student-centred approaches to learning. The full package included a widened
range of curriculum and guidance, multiple pathways to diverse further learning,
flexible qualifications structures, and diverse methods of achieving credentials through
credits which can be accumulated and transferred.
205 The development of standards-based assessment during the 1990s amounted to a
significant change in New Zealand’s approach to knowledge and learning in the
secondary school environment. It needed to be accompanied by huge cultural and
practice changes in schools, as well as significant shifts in thinking by parents,
employers and the community at large.
206 The change was contentious, however, and the debate became very polarised. The
model, NCEA, that was ultimately implemented in 2002 was a compromise between
the various parties, balancing internal and external assessment in roughly equal
proportions, and having three successive cycles of assessment in the last three years of
secondary schooling, plus scholarship. This is very substantial by international
standards.
207 The polarisation of the debate over senior secondary qualifications can be traced back
over two decades. The compromise reached over NCEA was a pragmatic response to
the history of division within the sector, rather than being based on first principles. In
practice NCEA was shaped by intensive input from subject specialists, sector leaders
and wide sector consultation. The Qualifications Development Group, a dedicated
NCEA development team established by the Ministry of Education in 1999, led this
process.
Experience to Date with NCEA
208 Three years on, the majority of feedback received by the Review Team from schools
and representative organisations is that NCEA is breeding a culture of success for
students who would have previously failed under the old system as well as enhancing
learning outcomes for all students. NCEA also empowers teachers to make the
judgements they were trained for and is further developing the profession. Some
schools and education specialists continue to express concerns about aspects of the
design of NCEA and impact that these have had on reliability of results. Nonetheless
there is acceptance that there is no going back.
209 However, most schools and organisations believe that the implementation path to
standards based assessment has been too steep. The safety nets that ensured stability
and reliability in the old system were displaced and there were few comparable
alternatives. Too much reliance was placed on the self-sufficiency of standards that
were continuing to develop and change, and on the capacity of teachers, examiners and
markers to interpret the standards consistently before they had been adequately
internalised. This has produced a degree of variability in the results of external
assessment that is beyond public and professional tolerances. In parallel, the external
moderation of internal assessment by schools was felt by many to be insufficiently
robust to ensure national consistency.
210 The Review Team agrees that the implementation path has been too steep and has
overlooked some essential safety nets. Correcting one imbalance in assessment has
41
inadvertently created another. While formative assessment is the core of education,
summative assessment is its twin, just as reliability is the twin of validity. Insufficient
attention has been paid to securing a balance between these elements and attaining
reliable summative assessment to secure public confidence.
211 The evidence for this, and the responses that the Review Team sees as necessary to
correct the imbalance, are canvassed below.
Variation in the Externally Assessed Achievement Standards
212 The three years of NCEA implementation allow results to be compared across those
years and the comparison shows that there is significant variation in a large number of
externally assessed achievement standards from one year to the next. The variations
relate not just to students who did not achieve the standards but also to the students who
received ‘merit’ or ‘excellence’. In some instances there are also noticeable variations
between different externally assessed standards within the same subject. This is
disturbing, but a positive note is that there is far less variation in the results of internally
assessed achievement standards, which are different standards from the externally
assessed achievement standards.
213 Professor Crooks has analysed NCEA Level 1 in English, Mathematics, Science and
Geography. His tables show the consistent picture for internally assessed achievement
standards:
Table 3: Percent Achieving Merit or Excellence on Internal Standards in the Four Subjects
Standard
2002
2003
2004
90052
33.6
31.2
31.9
90058
36.8
35.3
37.9
90059
34.6
33.8
35.1
90060
37.6
34.3
32.3
90206
44.1
42.1
35.0
90207
52.1
50.5
54.6
90208
43.2
39.5
42.2
90149
39.6
42.0
39.6
90150
39.7
41.7
40.0
90193
28.9
33.5
34.8
90186
49.4
51.3
50.3
90187
37.4
34.4
31.2
Note: the numbers of students graded ‘not achieved’ on the internal achievement standards
had to be estimated for 2003 and 2004 because these were not fully reported by schools in
2003 and not reported at all in 2004.
214 By contrast, however, a much larger variation in the externally assessed achievement
standards is shown in the following table:
Table 4: Percent Achieving Merit or Excellence on External Standards in the Four Subjects
Standard
2002
2003
2004
90053
20.9
17.7
17.0
90054
20.8
17.1
15.9
90055
27.1
19.0
15.0
90056
17.1
15.4
17.5
90057
51.3
35.0
16.1
90202
16.6
22.5
22.5
90203
6.1
9.9
21.9
42
Standard
90204
90205
90147
90148
90151
90152
90153
90194
90188
90189
90190
90191
90192
2002
22.6
39.5
34.7
31.1
35.3
32.0
40.4
17.1
13.0
27.7
6.9
13.1
17.3
2003
12.1
23.8
16.9
23.3
31.3
33.8
18.0
28.2
17.3
26.8
6.7
13.0
16.6
2004
19.9
24.5
27.6
7.4
39.6
17.2
23.6
23.6
8.6
17.3
17.9
18.7
9.1
215 When the internal and external assessment results are combined the overall pattern of
variation is still large:
Table 5: Maximum Year to Year Fluctuations in the Percentage Achieving Merit or Excellence on
the 12 Internal and 24 External Standards in the Four Subjects
Maximum change in
Percent of Internal
Percent of External
percentage
Standards
Standards
> 30 %
0
5
20 – 29.9%
0
10
15 – 19.9%
0
20
10 – 14.9%
0
25
5 – 9.9%
33
25
0 – 4.9%
67
15
216 Professor Crooks concludes that the extent of variation in the externally assessed
standards is excessive. The Review Team agrees.
217 The much more satisfactory picture for internal standards is encouraging, but it is
important to appreciate that that the picture is a national one and does not assure high
consistency from school to school or teacher to teacher.
218 Nonetheless, the results on internal assessment are consistent with the view, well
supported by research and education literature, that teachers are the best source of
information on the abilities and performance of their students.
219 Crooks’ findings on internal assessment are borne out by other studies. For instance,
research on the Waikato Certificate of Studies in English, a rival qualification to
NCEA, comes to the same conclusion. 13 The Certificate’s moderator, Professor Cedric
Hall, 14 found that classroom teachers tended to be more reliable as markers than
external examiners. 15
220 This finding should not come as a surprise because assessment reliability increases with
the amount of evidence that is collected, as in internal assessment. Variation is more
likely to be high when it is confined to a single event of assessment (like an external
13
14
15
The Waikato Certificate was originally an NZQA approved programme whose credits counted towards NCEA.
There is now doubt about this.
Professor Hall is Deputy Dean of the Faculty of Education at the Victoria University of Wellington.
Professor Hall also comes to other conclusions that are critical of NCEA.
43
examination). But because NCEA relies on a balance of internal and external
assessment the latter must have integrity.
Causes of Variation in Externally Assessed Achievement Standards
221 Variation in unmodified examination results is not new or unusual. It was present in
the past SC and UB examinations, though it was not as obvious in these examinations
as it is with NCEA. Two factors mainly account for this. The UB examinations were
scaled after marking to conform to normative expectations of what students would
typically achieve, and the un-scaled SC results were ‘soft-scaled’ using normative
expectations before, during and after marking. Both examinations also reported their
results by a single, aggregated mark which ‘evened-out’ the inevitable variations
between questions. By contrast, NCEA reports unmodified student results for
individual standards, which are not aggregated into a single subject result. Neither of
the controlling factors – normative adjustments and an aggregated result – is available
to NCEA as it is currently operated.
222 NZQA has identified several specific causes of variation in NCEA:
•
Level 1 achievement standards were reviewed by the Ministry of Education
during 2003. New versions were used for assessment in 2004 and changes were
made to the examinations to match the new version of the standard.
•
There were achievement standards where examiners considered the 2004
candidates to be better prepared than in 2003, resulting in fewer students not
achieving the standards. The improvements came about from better prepared
students following the publication of examiners’ reports on the previous years’
examinations.
•
There were adjustments to the examinations using feedback to improve practices.
NZQA cite the Level 2 achievement standard 90380 Read unfamiliar texts and
analyse the ideas and language as an example. In 2003 examiners considered
that the paper was below the required standard and sought correction for 2004.
The proportion of candidates who did not achieve this achievement standard in
2004 was more in line with other externally assessed Level 2 English
achievement standards than it was in 2003.
•
There were shifts in the cohort size taking a particular standard that differed from
the changes in the total number of students. This suggested that a higher or lower
number of the more able students may have entered the standard, or a number of
repeating students may have entered.
•
In some subject areas the number of external standards available changed
between 2003 and 2004, which contributed to the variation in results in any
individual achievement standard as students moved from one standard to another.
223 NZQA also believes that NCEA uncovers behaviour previously unnoticed. Students
now understand how many credits they require and consequently do not attempt some
standards in an examination session. Between 25-30 per cent of examination scripts
were handed in blank in the 2004 English standard 90722 Respond critically to
Shakespearean drama studied. While this may surprise, it has always been the case
that students select the questions they answer. The Rector of the University of
Aberdeen noted in 1874, “students work to pass, not to know, and outraged science
takes her revenge: they do pass and they don’t know”.
44
224 These causes for variation are important to identify and the NCEA assessment system
shows a positive capacity to expose and analyse them – something which happened
rarely in the past with norm-referenced statistical moderation and aggregate marks.
Because there was the comfort of a bald mark people quickly moved on.
225 However, the most important source of variation, the inevitable variation that comes
from different interpretations of questions and the answers provided, is only partially
acknowledged by NZQA, as was the case with the 2004 Scholarship.
226 Even very small changes in the way a question is asked can produce large differences in
the answers. Professor Warwick Elley 16 has compared successive cohorts of university
students who answer the same questions, with those who answer lightly modified
questions (e.g. “What is a holophrase?” changed to “Complete: A holophrase
represents…”). He demonstrates that the cohorts answering the same questions show a
mean average variation of 2.2%, but the cohorts answering the slightly modified
questions show a variation of 24.8%.
227 When NZQA does discuss variation in questions it focuses on the differing
interpretations of examiners and markers in applying the assessment schedule. NZQA
believes that the stability of the personnel involved (85% of the examiners of
achievement standards that have been questioned were the same in 2003 and 2004) and
the close contact between panel leaders and examiners controls this source of variation.
228 The PPTA differs and reports complaints by panel leaders of insufficient opportunity to
meet with markers. “Marking is done under huge pressure with probably not enough
‘guinea pig’ scripts being marked before marking is finalised … and there appears to be
no liaison between the leaders of the panels for the different standards being assessed
which can lead to inconsistencies…”. To its credit NZQA has responded positively to
these points, exploring the formalisation of the contacts between the panel leaders and
examiners, and expanding the benchmarking by using a greater number of guinea pig
scripts before marking begins.
229 Yet neither the PPTA nor NZQA shape the issue fully because it is not only teachers
and assessors who differently interpret questions. Students also differ in their
interpretations and this manifests directly in widely varying answers. This is a ‘blind
spot’ in NZQA’s mirror which may explain some gaps in its current strategies for
managing variation.
230 NZQA focus is primarily on pre-assessment moderation techniques to achieve common
understanding and interpretations of the standards by teachers, students, examiners and
markers before an assessment occurs. It also aims to ensure consistent quality through
accrediting a school’s internal quality assurance processes 17 in conjunction with the
more comprehensive role played by the Education Review Office. From these premoderation measures NZQA anticipates that “…the level of non-achievement will be
reduced through better guidance and preparation of students, professional development
of teachers, and ongoing communications around the expectations for assessments”.
16
17
Emeritus Professor Elley is a widely published expert in educational measurement.
Via the Quality Assurance Standard for Accreditation of Secondary Schools.
45
231 This approach is positive and necessary to achieve a common understanding by all
relevant parties before an assessment to reduce variation. Yet it does not fully address
the effects of variation after they have manifested in student results. This results in
NZQA having comparatively weaker post-assessment moderation strategies for
managing variation. 18
232 Most assessment systems internationally balance pre- and post-assessment moderation
measures. For instance, research conducted for the Western Australian Curriculum
Council recommends pre-moderation (applying item response theory which tests the
suitability of assessment items or questions) backed by statistical moderation. This
dual approach is not intended to replace standards-based assessment, but supports the
direct use of clearly defined standards. Western Australia proposes to continue
statistical moderation until evidence shows that it is no longer necessary. 19
233 A similar precautionary approach has been suggested by a number of New Zealand
schools. One principal believes that “in time the New Zealand educational community
will develop a shared common understanding and interpretation of the standards, but
the Level 1 NCEA experience shows that this will take many years. During this time
there needs to be a safety net or guidelines to ensure that the variations are within
acceptable limits and the future of our students is not left to chance”.
Recommended Approaches to Managing Variation in External Assessment
234 The Review Team’s recommendations concentrate on strengthening the front and back
ends of external assessment to control variation, and making some changes to the way
results are aggregated and reported. The measures proposed are part of a package,
some additional features of which are discussed later. The team acknowledges that
NZQA is already taking action to reduce the extent of variation in external assessment.
235 The Review Team’s recommendations are therefore intended to augment and
strengthen positive actions that NZQA is already taking. NZQA’s actions include:
18
19
46
•
Careful appointment processes and training for examiners and markers.
•
Enhancing examiners, panel leaders and markers’ expectations of cohort
performance by providing feedback from the teacher examination surveys and the
guinea-pig scripts and assessment schedules from the previous year’s
examinations. Unless told otherwise, these guinea-pigs and assessment schedules
accurately reflect the standard. The aim is to produce an examination and
assessment schedule requiring the same levels of performance from candidates
and is akin to the team’s recommendation for an enhanced equating procedure.
NZQA’s view of variation helps to explain two of its positions. When analysing variation, NZQA identifies 15
Level 1 and 19 Level 2 externally assessed standards in which the variation in the percentage of students not
achieving in 2003 and 2004 was 20% or more. Twenty percent is a large value and in business would undermine
confidence in most processes. By contrast, the PPTA refers to a boundary of 5%, which in the context of cut-off
scores may be too tight. It seems that 10% would be more realistic.
A second statement from NZQA is even more surprising. It confirms that: “…the Level one achievement standards
in which the percentage of students that did not achieve the standard in 2004 is similar to that of 2002, indicating that
it may be the 2003 result that was aberrant. The 2004 result for the 2004 achievement standards is also more in line
with that of other external achievement standards in the same subject than they were in 2003”. In other words, the
variation in 2003 is noted and corrected in the following year, but its consequences for the 2003 students are glossed
over.
International best practice in outcomes-based assessment related to post-compulsory education, Tognolini, Andrich
and Ball, 2001.
•
Requiring all previously trained Panel Leaders to attend top-up training in
September 2005.
•
Increasing the proportion of guinea pig scripts which are test-marked to refine the
marking schedule, and considering the profile of candidate performance indicated
by this test. This information will be provided to the NAF before clearance for
marking to begin.
•
Panel Leaders who are not Examiners will be briefed by the Examiner to achieve
as complete an understanding as possible. Dialogue will occur between the
Examiner and Panel Leader through the NAF.
•
Enhanced analysis of interim results. Any concerns about the profile of cohort
performance will be signaled to the NAF along with an explanation for these
results, supported by a completed check-list of possible factors. In future, Panel
Leader’s reports may include these comments, which may also be subsequently
included as part of the Examiner Report, and available in draft form to future
examination setters. These actions are consistent with the recommendation
below, to use examination data to moderate the results.
•
Reporting across subject groupings (as recommended by the Review Team).
•
Increasing the communication to teachers through a new ‘Assessment Matters’
circular.
•
Reviewing international literature on variation.
•
Conducting more analysis of actual student behaviour in examinations.
•
Calling together a symposium of assessment experts.
236 The Review Team strongly supports these actions and our further recommendations are
either a clarification or deepening of what NZQA intends. The first point of attention is
on the front end of assessment, before students sit the examinations.
237 Western Australia’s safety net uses item response theory which assumes that each
candidate possesses an underlying ability which can be scored, placing them on an
ability scale. Items are then tested to see whether a candidate gets each individual item
correct or not. Each item is measured by its difficulty and its capacity to discriminate
between different candidates’ abilities. The likelihood that a correct answer may be
guessed rather than intended is also calculated. This requires sophisticated
mathematical modeling, now made easier by computers.
238 Item testing improves ‘classical’ assessment which focuses on the examinee’s raw test
score. This test score is the sum of the scores received on the items in the test. By
contrast, item testing looks at whether each individual item is correct or not, rather than
the raw test score. For example if two students get five out of the eight questions
correct but the first four questions are harder than the last four, then it matters which
five questions are correctly answered. The student who does better on the hard
questions should get a higher grade.
239 Although the assumptions behind this technique are not accepted uniformly by
educationalists, several academics have suggested that NZQA should employ it. The
main constraint is practical: item testing is difficult to employ when it is faced with the
many and frequent assessments of NZ, and when questions must change each year.
47
Subjecting each item to response testing is cumbersome and costly, particularly if every
item or question requires testing before assessment is conducted. If the item testing is
applied to the current assessment results before they are released, then time must be
available for comprehensive analysis before the release of results. In the Review
Team’s previous report on the 2004 Scholarship it was noted that the timeframe was
already too compressed and intense in New Zealand. Item response testing would
make this worse.
240 It has been further suggested that the ‘bookmarking’ methodology could augment item
testing. This is a procedure developed during the asTTLe project. 20 Items are ranked
according to difficulty determined by the performance of students. A panel of teachers
reviews each item and is asked to place a ‘bookmark’ to differentiate between ‘groups’
of items that a student at the threshold of a performance level should master and those
items that are not necessary to master.
Bookmarking particularly highlights the
processes and skills in a progression. It can also be adapted to an examinee-centred
approach which ranks students around the cut scores, rather than items. The particular
value of this approach is its focus on standards setting without recourse to normreferencing, and the fact that it produces replicable evidence that can secure the
standards at the crucial cut-off points. However, it shares the time and resource
intensity of item response testing.
241 Item testing may be manageable, however, if pre-testing samples the questions and it
occurs in another country like Australia. A sampling approach might also work in
after-assessment analysis if more time is inserted between assessments and the
reporting of results.
242 It is therefore recommended that NZQA explore the possibilities offered by item
response testing. This will not be an immediate panacea. NZQA is fast approaching
the examination season for 2005 and research is only possible in 2006. If feasible, a
pilot will be necessary in 2007 with wider migration only beginning in 2008 at the
earliest.
243 Consequently, the Review Team recommends that a simplified form of item testing be
examined immediately, to test whether it can be implemented next year (2006). This
idea was suggested by Professor Crooks. It requires analysis of the current results from
the three years of NCEA to identify stable patterns in an achievement standard, or even
one year of acceptable results which can provide a base from which potential items for
next year’s assessment could be constructed. The package of the past assessment items
and the new items would then be pre-tested on a number of students in New Zealand
schools. Because the achievement standards are relatively small, the risk of
contamination (‘leaking’) by pre-testing is minor and controllable.
Science 90191 provides an example of what is required:
Table 6: Science 90191: Describe aspects of physics
Year
2002
2003
2004
20
% Pass (Achieved
or Better)
73.4
49.7
71.9
% Merit or Better
% Excellence
15.1
13.0
18.7
2.2
2.4
2.2
Assessment Tools for Teaching and Learning, www.tki.org.nz
48
244 This achievement standard was inconsistent at the Achieved level in 2003 but
reasonably consistent at other levels over the three years. The assessment items of
2002 and 2004 (and the items that test M and E in 2003) can be used to develop new
items that are closely related. The new items can be adjusted according to the
outcomes of the pre-testing, and then used with confidence. If unusual variation occurs
in the subsequent results of the actual examination, causes can be identified (and
hopefully rectified) and the students’ results adjusted so that students do not shoulder
the effects of consequences beyond their control.
245 Again, practical impediments may limit this approach which is known as ‘equating’.
Any pre-testing incurs costs, and the right timing may be hard to achieve (pre-tests have
to occur early in the year, though that issue may fade after the first year of pre-testing,
when a staggered approach can be taken). Establishing what is an acceptable pattern or
example may also prove contentious, which will slow down the implementation. In
some cases there is no obvious pattern and Level 3 has only one year of results. Panels
of experts may have to decide what the acceptable assessment rubrics are.
246 The Review Team’s next recommendation therefore shifts to the back end of
assessment and is more immediate: that NZQA define and bring forward normative
boundaries to function as a safety net for the three grades of Non-Achieved, Achieved,
Achieved with Merit and Achieved with Excellence. This will give practical guidelines
to NZQA’s intention to increase the analysis of results. In effect, the decision by the
Secondary Leaders’ Forum to aim for a 70% pass rate for NCEA Level 1 sets one of
these boundaries. However, the pass boundaries for Levels 2 and 3 remain vague at
present and need to be settled urgently. There is some guidance from the overall pass
rates in 2003 and 2004. The normative boundaries will allow outlying assessments to
be identified, and at present NZQA provides no indication of how they will otherwise
do this in an efficient manner.
247 Setting the boundaries may be more difficult for Merit and Excellence where the
standards differ noticeably between subjects, reflecting their different characteristics.
However, the Achieved/Not Achieved boundary is more stable because it is based on
achievement of the outcomes of the National Curriculum Statements. The generalised
nature and variable quality of the curriculum statements pose some difficulties, though
the assessment rubrics formulated in the asTTle project may help.
248 These statistical boundaries should not be excessively precise, narrow or set in
concrete. Rather, the statistics should be indicative and incorporate a level of tolerance.
They should contribute to analysis, highlight unacceptable variation and function as a
tool among a range of other tools for moderation. If not, statistical moderation will
become a convenient default, proving the blanket coverage of the past which concealed
the true nature of variation. The very real and rich data which is emerging form the
exposure of variation will be lost and the education system and students will suffer.
249 It is vital that the statistics should guide but not pre-determine. They should also be
there to inform the analysis. The Review Team is concerned that NZQA currently
makes less than optimal use of statistics. Many of the staff convey impressive personal
insights using data but these should be systemic. The statistics should be immediately
generated and to hand and inform professional adjudication by respected peers who
must analyze their meaning, weigh implications, identify (if possible) the causes of
49
variation, and make the judgments which balance the interests of students with
information for future improvement to the system.
250 The issues at stake are illustrated by ENGLISH 90057: Reading of unfamiliar texts
Table 7: ENGLISH 90057: Reading of unfamiliar texts
Year
2002
2003
2004
% Pass (Achieved
or Better)
89.9
79.1
55.9
% Merit or Better
% Excellence
51.3
35.0
16.1
2.9
2.7
0.5
251 This standard shows a strong trend towards getting harder. This needs urgent attention
and NZQA appreciates this. A normative adjustment to correct the trend could be
misconstrued as yet another swing, which NZQA will need to manage by effective
communication.
252 Defining the normative boundaries for the three affirmative grades of Achievement,
Merit and Excellence will inform exemplars, and assessment rubrics can be developed
and posted to increase the internalizing of standards. As Professor John Hattie 21 says:
“We can restore confidence in standards-based assessment to the degree we dependably
communicate the standards.” 22
253 The Review Team recommends that the examination results should not be sent to
students, parents, schools, or published more generally until the results are fair and
accurate. Time must be available to NZQA to complete its analysis and make
adjustments if necessary. This may require the intervention of a subject panel or set of
experts. The current pressures on NZQA which squeeze the time between marking
which is usually completed by Christmas (for logistical reasons discussed in the report
on the 2004 Scholarship) and publishing results as early as possible in January distorts
the assessment system and endangers fairness. Something has to give.
254 Schools have traditionally exerted pressure to obtain the results in order to assign
students to the next year’s classes. They could equally make use of the other
substantial sources of information on their students to make these decisions. The
universities will also have to wait for two to three weeks longer and have indicated that
this does not pose an insurmountable problem. In practice this may mean between two
to three more weeks than the current timeframes for considered analysis (possibly into
early February).
255 The Review Team also believes that better compromises can be reached over the
staging of the external assessments at the end of the teaching year. For instance Level 3
NCEA examinations and Scholarship could be staged earlier than Levels 1 and 2
because they mark the end of a programme of learning and the great majority of
students will not be returning to school.
256 The Review Team’s next recommendation reinforces actions NZQA is taking to
aggregate NCEA results. Aggregating marks to control variation acts as a disguise but
it will take much of the heat out of the debate. Both School Certificate and University
21
22
Professor John Hattie is a member of the Faculty of Education at the University of Auckland.
Professional Opinion on NCEA, in a paper commissioned by the Ministry of Education, 2005.
50
Bursaries examinations availed themselves of the statistical principle of adding lowcorrelated items to produce a regression that reduces variability. As currently reported,
each achievement standard is reported separately, and NCEA has no combined,
aggregate mark to smooth the variations between questions within the subject and the
variation is nakedly reported.
257 The effect of the ‘aggregation control’ is demonstrated by NZQA when external
assessment results from achievement standards in a subject are roughly aggregated and
compared to aggregated School Certificate subject scores. The comparison is imperfect
because the two are not directly parallel (one is norm-referenced and the other
standards-based) but the aggregated outcomes are more consistent than comparisons of
totals from three hour examinations with individual standards from shorter
examinations:
Table 8: External assessment pass rates
Subject group
NCEA
Level 1
2003
Accounting
64%
English
60%
Mathematics
61%
Science
59%
Economics
69%
Geography
57%
History
63%
NCEA
Level 1
2004
71%
56%
68%
58%
70%
67%
70%
School Cert 1997 – 2001
(lowest and highest)
57 - 71%
56 - 65%
51 - 62%
57 - 65%
57 - 68%
58 - 68%
68 - 73%
258 Aggregation is largely a reporting rather than a design issue. Schools should advise
students on suitable achievement standards to enrol in and structure a coherent
programme around those achievement standards. These standards can then report an
aggregate grade for subjects, and possibly for the programme, to provide the quick
information and comparisons with past examination results that some employers and
the public seek. A single graphic presentation of results on one page, with broad
fields/subjects in which the student has obtained credit, the levels at which credits had
been attained in each field/subject, and the proportion of standards at each level/subject
in each field at merit or excellence levels would greatly assist parents and employers.
259 NZQA is already modelling a new results package which includes some of these
elements. However, internally assessed achievement standards still do not report nonachievement. There is more on this point later.
Internally Assessed Achievement Standards
Moderation
260 Moderation worries some schools which feel that the system is too light to achieve
national consistency. Several of them, supported by academic commentators, propose
that the external examination be used to moderate school based assessments, and
subjects with a large internally assessed component (e.g., English, Māori, art, music,
social studies), operate with the average and range of their internally assessed grades
strictly controlled and limited. If the external examination proves to be an
inappropriate method in a subject, a moderation test or common assessment task could
be used.
51
261 This view is driven by a concern that the design of NCEA does not integrate or blend
the internal and external approaches to assessment. Instead the achievement standards
are either one or the other – assessed internally or separately assessed externally. This
means that the weaknesses of each approach to assessment - internal assessment lacks
the objectivity provided by an external examination and external assessment lacks the
improved reliability and validity offered by internal assessment - is not softened by the
other.
262 A blend of internal and external assessment is something that a future review of the
design of NCEA could consider. At present the changes may not need to go that far
because many of the concerns about moderation appear transitional, reflecting the early
stages of implementation. As NCEA ‘beds in’, the Review Team observes schools
becoming more familiar and accustomed to receiving accreditation for internal systems.
This system is derived from the tertiary sector and stresses quality assurance on the
principle of ‘trust but verify’. This places more responsibility on the teaching
profession than the past secondary sector system of external quality control through
examinations.
263 Indeed, it is this shift towards more professional responsibility that causes the concerns
about moderation. As already noted, some schools are worried about the consistency of
judgement between schools for internal assessment. The Review Team strongly
supports NZQA’s encouragement of greater trust in the judgement of the teaching
profession, but also believe that some safety nets may improve the moderation system,
especially at its outset when the new practices are taking hold.
264 To clarify this point it is useful to provide a fuller picture of the moderation system.
There are two complementary dimensions. To be accredited, schools must have
policies, systems and processes which comply with national standards. NZQA carries
out a review known as managing national assessment (MNA) on a two year cycle,
although the review can be yearly for schools with serious problems. By the end of
2005, every school will have had at least two MNA reports. The school’s systems must
cover:
•
The routine critique of assessment materials to be used.
•
Verification of assessment decisions.
•
Appreciating national standards and keeping up to date with them.
•
Ensuring the authenticity of students’ work.
•
Facilitating opportunities for students to display evidence of achievement.
•
Conveying assessment information to students and other stakeholders.
•
Addressing concerns identified by the external moderation process.
265 This internal responsibility of the school is then tested by the external scrutiny of up to
20% of the standards assessed each year by the school. The scrutiny is carried out by
subject moderators contracted and trained by NZQA. This sample size allows the
School Relationship Managers (SRMs) to target areas of concern, and accede to
teachers’ requests for additional standards to be moderated to reassure their
judgements. The assessment task, the marking schedule and the student’s work are
scrutinised and moderators comment on the validity of the assessment activity, and
52
evaluate the assessment judgements to provide a ‘second opinion’ on them. A report is
provided to NZQA and to the school. The moderators’ work is itself checked by the
national moderators. Annually, the SRM collates the reports into a summary and
advises schools of the issues arising from it. The MNA process then checks that
concerns identified have been followed up.
266 There is an appeal process available for teachers who disagree with the moderator’s
report, where the moderation findings are reviewed by the national moderator. The
appeal is promoted as a ‘third opinion’ which further enhances the professional debate,
but it has been little used. In 2003 there were 79 appeals out of 12,833 activities that
were moderated and 25 appeals were upheld.
Seventy-nine percent of assessor
judgements were consistent with national standards from 54,778 examples of student
work.
267 NZQA has made several improvements since moderation began. Moderation for unit
standards (the original model) evaluated only the proposed assessment activity or task.
This implied that the validity of the task was central to moderation rather than the
combination that is now addressed which moderates the assessment activity or task, the
marking schedule and the student work.
268 A further weakness was that schools frequently failed to attach student work because
actual assessments which generate the work had not taken place at the time of the
school’s moderation submission. In 2003 almost 60% of moderation submissions
lacked student work. NZQA has now changed the cycle to establish a 14-month
moderation period, independent of the calendar year for each school, to emphasise that
the student samples do not have to come from the current calendar year.
269 At the same time, NZQA strengthened the negotiation between the SRM and the school
on which standards should be moderated. The goal (to be achieved by early 2006) is
for no moderation submission to be made without student sample work. Although
schools can choose which results to submit, NZQA continues to persuade schools and
teachers that borderline decisions submitted for moderation produce the most
satisfactory professional outcomes.
270 Further improvements include a more accessible moderators’ report form; promoting
the appeal process; auditing external moderation policy and processes by an
independent (and possibly international) organization; increasing communication of
assessment issues identified through external moderation – to be discussed in
‘Assessment Matters’ circulars to schools, in the annual SRM Principal Roadshows,
and through MNA visits to schools; and identifying moderators who make ‘poor’
judgements and putting remedial practices in place for them.
271 The Ministry of Education and NZQA are discussing the possibility of NZQA
contracting teachers to moderate and provide advice and local support. The contracted
moderators would divide their time between teaching and contracted moderation
services. Their activity would complement and enhance the role of Support Services
Advisors. NZQA is also considering employing full-time national moderators with
access to a website chat room (or equivalent) and to subject national associations.
272 These improvements deserve full support but there are several further actions which the
Review Team believes would strengthen the safety net for moderation.
53
273 First, schools select the examples of student work (but not the other material), raising
concerns that only ‘safe’ marking decisions are put forward, a perception that has been
reinforced by the unfortunate practice of some school principals who have used
moderation outcomes as a measure of teacher competence. A report commissioned by
NZQA in 2001 (known as the Rhoades report) suggested ways in which random
sampling of student work rather than school selection could improve the validity of
moderation. There are reasons why NZQA have opted for selection which include
shifting student populations and enrolments in achievement standards, but selection
lowers the statistical validity because of possible bias.
274 Secondly, the achievement standards for moderator scrutiny are selected by the school
and the SRM. This is directed by key principles, such as focusing on Level 3 first
(because it is higher stakes and schools have less implementation experience at this
level), and ‘red flag’ areas identified by previous moderation. Nonetheless the 2001
report provided reasons why exposing all the achievement standards for potential
selection improves the moderation validity, and it offered practical suggestions about
how this could be done.
275 The Review Team accepts that in an optimal system teachers are best positioned to
identify borderline cases for scrutiny, and positive, professional moderation invites the
discussion that ensues. NZQA also rightly notes that cheating the moderation system is
unlikely because there are so many eyes – the moderators, national moderators, school
nominees with the moderation responsibility for their schools, HODs, Principals, and
other teachers in the numerous subject cluster groups. But it is a simple reality that new
systems are viewed with suspicion and because unpredictable effects frequently occur,
it is sensible to have safety nets that maximise stability.
276 The current system of moderation cannot guarantee year to year consistency and the
repeated use of exemplars and equated models of assessment mean that there will be a
growing rate of achievement, not all of which is attributable to improved student
performance.
277 Consequently, the Review Team also recommends, as elsewhere, that NZQA should
use the rich data it gathers from the external achievement standards to identify schools
or subjects with significantly different results from internal achievement standards.
This is similar to the procedures used to ensure adequate consistency between schools
accrediting University Entrance. The data would not directly moderate internals with
externals, which would be questionable given the different skills and knowledge tested
by each form of assessment. However, while the differences are very obvious in some
subjects, such as speaking and writing in the languages, in other subjects like
mathematics the difference may not be as significant, and more direct comparisons
would have validity.
278 Better use of data could focus moderation on outlying schools or one or more targeted
subjects. NZQA has statistics identifying which subjects are weaker in internal
assessment (e.g. Biology and Technology have significantly lower rates of acceptable
assessor judgements).
279 The Review Team recommends that NZQA urgently reviews its policy which makes
internal assessment the sole responsibility of the schools, who accept the duty of
54
reporting to caregivers where students have attempted and failed to achieve. The
reality is more complex because NZQA runs a national system of secondary school
qualifications and cannot undermine its responsibility to collect and report data on
those qualifications. A change in this policy will allow NZQA to address its policy of
not reporting non-achievement of internally assessed achievement standards.
Reporting Non-Achievement
280 There has been repeated concern about the apparently inconsistent reporting of nonachievement. NCEA does not report non-achievement for the internally assessed
achievement standards but non-achievement is reported for the externally assessed
achievement standards. This has been a strong point of contention with parents and
employers.
281 Questions about reporting or not reporting ‘not achieved’ results arose when the 2002
student results were used to establish a league table of school performance. In the
ensuing debate, varying practices in schools over reporting ‘not achieved’ results of
internally assessed standards emerged, further complicated by the growing range of
qualifications offered to students as the Qualifications Framework was implemented in
schools. The league tables proved unreliable and concerns were expressed from all
sides about the dependability of the data.
282 To adjudicate the matter NZQA carried out a consultation process with schools, other
education groups and some members of the public. The clear preference of
stakeholders was to report only achievement on students’ interim result notices, in line
with the reporting practices on the Record of Learning (which is a cumulative record of
learner achievements).
283 Consequently, NZQA decided not to record or report ‘not achieved’ results of internally
assessed standards, and to report ‘not achieved’ results of externally assessed standards
only on schools’ and candidates’ secure web results pages, and in publicly available
national and school results statistics. It aimed to ensure the availability of external
examination results to students, parents, families, schools and the broader community
which have a legitimate interest in them, while controlling the inappropriate use of
aggregate data to compare schools. Schools would also continue to report both
‘achieved’ and ‘not achieved’ results of internally assessed standards in their own
school reports.
284 The reasons behind these decisions and the actual availability of the results data have
since been lost. The Review Team supports the decision to report only the standards
achieved (or better) on each student’s Record of Learning, but non-achievement of all
internal and external achievement standards should be reported on the results notices
sent to students and their schools, and in the statistics provided by the NZQA. Without
this provision, parents and others feel misled, and it is impossible to properly track
what is happening nationally. This is a matter that NZQA should give priority to in
2006.
55
3.5 Such other matters as the Reviewer Considers Relevant
285 During the course of the consultation, a number of issues concerning the design of
NCEA were raised with the Review Team. The issues were raised in good faith and the
team recommends that they be referred to the Secondary Leaders’ Forum for
consideration as part of the planned Stocktake of NCEA this year. In so doing, the
Review Team is not expressing a view on the merits or otherwise of the issues raised
other than to note that opinion is divided on many of them.
286 With a change of the magnitude of NCEA, refinement based on continuous review and
evaluation is not only to be expected but is also good change management practice.
Separately Assessed Standards
287 Separately assessed standards are seen by some as heightening the potential for
variation because the reliability of assessment results bears a strong relationship to the
amount of evidence that is collected. The more evidence the better the reliability, and
in the case of one-off examinations, the longer the paper the better the reliability. The
assessment time given to one achievement standard in the examination is typically less
than an hour (the majority of externally examined NCEA papers have a suggested
length of 30 to 60 minutes). This is seen as insufficient evidence to be reliable,
endangering the validity of decisions about the cut-off points for grades such as
Achieved or Excellence. 23
288 It is argued that fewer externally assessed standards would focus the three hour
examination on, say, three standards rather than up to six at present and hence increase
the reliability of assessment and reduce variation caused by shorter tests. This would
require the consolidation of external standards into larger standards than the current
ones.
The Amount of Assessment in New Zealand
289 Many of those consulted pointed out that New Zealand is unique in having four
assessment cycles with external examinations in the last three years of senior secondary
schooling. This is not seen as sustainable over the medium to long term. By contrast,
the predominant international practice is for one external assessment, usually at exit
from the secondary school. Even Ireland, which is educationally conservative, is
seeking to reduce its current two examinations to one. Managing the assessment loads
constitutes a major challenge to both NZQA and schools.
23
Assessment measurement theory can explain this. On a reliability scale where 0 represents no reliability and 1.0
represents perfect reliability, a value over 0.8 is considered to be satisfactory. This value has an associated standard
error of measurement of approximately 6.3%, normally doubled to ensure a 95% confidence. This means that an
examination score of 50% is subject to a margin of plus or minus 12.6%. In effect, 50% is the mid point in a range
of 37.4% to 62.6%.
When this margin is applied to a cut-off point or grade boundary, the level of achievement of the student is between
12.6% above and below the cut-off point or grade boundary. When the assessment is short and the evidence is
limited as it is for a single achievement standard, this range can more than double and a mark of 50% can be
interpreted as somewhere between 25% and 75%. The margin compounds when the reliability estimate is less than
0.8, increasing the standard error of measurement, meaning that a student awarded merit could equally have been
awarded ‘achieved’ or ‘excellence’.
56
Fragmentation
290 A number of those consulted feel that the separate assessment of each achievement
standard sacrifices the whole or integrated nature of a curriculum, which is broken into
fragments. Essential cross-curriculum concepts and skills are therefore not taught.
Professors Hall, Elley and Marsh 24 argue that the design of the course must come first,
while decisions on assessment should arise out of this design. They argue that the
current focus on achievement and unit standards leads teaching and learning in the
reverse direction—assessment comes first and course design then follows. They see this
as at the heart of the fragmentation issue surrounding NCEA.
291 On the other hand, many schools strongly support the freedom of choice provided by
NCEA and believe it simply calls for new practices in designing and delivering an
integrated programme. Many principals believe that the ability to build an NCEA
qualification over two or three years has contributed to a significant retention in Year
13 for 2005. Students are not rigidly grouped into year levels; rather the three years
comprise a single frame where students can ‘multi-level’ their studies.
292 While modularising the curriculum can fragment and atomise ‘whole’ subjects or
bodies of knowledge, in practice, curricula, courses and qualifications have always
been divided into parts. The real debate is not about the dividing the curriculum but
whether the size and number of modules suit and maintain the connections necessary
for integrity, and whether the reporting of results is at the modular level or aggregated.
Numbers of Achievement Standards and Grades
293 A number of schools have expressed the view that the number of standards in some
subjects is too great. Typically there are between five to eight achievement standards.
Eight might be too many when viewed against most overseas practice where five to six
outcomes are the norm.
The Western Australia Curriculum Council report
recommended:
For each course of study there should be five or six outcomes that will typically be
content- or skills-based. Each outcome should have a minimum of five standards
of performance. Five levels should be enough for purposes of certification. In the
case when the same performances are used for competitive tertiary entry, it may
be necessary to increase the number of gradations.
294 Some schools have also commented that the number of levels of achievement is too
small (currently four: Not-Achieved, Achieved, Achieved with Merit and Achieved
with Excellence), particularly at Level 3. They argue that this small range does not
adequately differentiate between students of different ability. They argue that it would
be desirable to have an expanded set of grade levels for NCEA Level 3 where the
stakes are higher and entry to higher education is involved.
295 Professors Hall, Elley and Marsh suggest a mark/grade scale of at least 10 points, to
help employers and tertiary institutions make better selection decisions, and to provide
more motivation for brighter students. They recommend an approach like the Waikato
Certificate of Studies-English which reports results on a 10-point scale and includes the
24
Professors Hall, Elley and Marsh submitted a joint submission to the review. Professor Reginald Marsh is Emeritus
Professor at Victoria University of Wellington and like his colleagues an expert in educational measurement.
57
aggregation of results across assessment components to yield an overall course grade.
Reliability estimates for the overall course grade typically reach 0.9 (Hall, 2004).
296 Others see the problem as being not the number of grade levels but the pass/fail model
used in NCEA for each of the grades. A student either meets the criteria or not and
meeting the criteria usually means meeting all of the criteria set down for a particular
grade within the achievement standard. This point can be overstated because a partial
answer to a difficult question (a Merit or Excellence question) can contribute to an
Achieved grade, but a firm threshold particularly affects the Achieved/Not-Achieved
grades. It is argued that when a student has to meet all the criteria this has the
undesirable effect of determining that student’s grade by the criterion in which he or
she is least proficient.
297 Many teachers feel this is too rigid because educational development rarely relies upon
acquiring a complete set of competencies to progress to the next level. It is argued that
a fairer system would allow some credit for performances which do not meet all the
criteria. As with most complex tasks, there are degrees of performance and a student’s
work rarely perfectly matches the performance indicators.
58
Annex 1
New Zealand Qualifications Authority
Terms of Reference for Review by the State Services Commissioner
Introduction
The New Zealand Qualifications Authority ("NZQA") is a Crown entity established under
Part XX of the Education Act 1989 "to establish a consistent approach to the recognition of
qualifications in academic and vocational areas"(s247).
NZQA's functions include the administration of the National Certificate of Educational
Achievement (NCEA) and other school, trade and vocational assessment. In 2004, NZQA
instituted a new Level 4 qualification, New Zealand Scholarship, to be the basis for the
allocation of university scholarship awards.
Background to Review
New Zealand Scholarship was introduced at the same time as NCEA Level 3. The new Level
4 stand-alone qualification was designed to enable the award of scholarships on the basis of
examination. Previously, scholarships had been awarded to the top 3 or 4 percent in
University Bursary examinations.
Students sat the New Zealand Scholarship examinations in November 2004, and results were
mailed to them in January 2005, and later made available on the NZQA website. The results
showed a much smaller number of scholarships awarded this year than expected and widely
differing achievement rates in different subject areas. The results gave rise to public concern
about the administration of the new qualification, and its fairness to otherwise high achieving
students.
The Associate Minister of Education has asked the State Services Commissioner to review
the adequacy of the setting and management of the 2004 Scholarship examinations and, more
widely, the performance of NZQA. The Minister's request is made under section 11 of the
State Sector Act 1988.
Terms of reference
Scholarship examinations
1
To review the 2004 New Zealand Scholarship examinations with particular regard to:
1.1
whether the roles and responsibilities of the various agencies involved were clearly
defined;
1.2
whether the identification and addressing of issues arising in designing and
implementing the new system was adequate;
1.3
whether the processes to set and moderate the standards for the qualification were
adequate;
1.4
whether the administration of the examinations against those standards, including
review of the processes for setting, moderating, and marking the examination were
adequate;
1.5
whether communication and guidance to teachers and students concerning New
Zealand Scholarship were adequate;
1.6
any other matter which the reviewer considers relevant to the foregoing questions; and
2
To make recommendations concerning these matters.
NZQA’s Performance
3
Given the purpose and role of the NZQA to provide quality assured qualifications, to
review how well the NZQA is undertaking its role in respect of the school sector
qualifications, with particular attention to:
3.1
the governance, monitoring and reporting of NZQA performance;
3.2
the adequacy of NZQA capability including planning, systems and processes to
undertake its role;
3.3
the nature and extent of the involvement of, and responsiveness to, key stakeholders in
the decision-making of the NZQA;
3.4
whether and if so how the NZQA is providing professional leadership in examination
processes;
3.5
any other matter which the reviewer considers relevant to the foregoing question.
4
To make recommendations concerning these matters.
Reviewer
The State Services Commissioner will appoint a suitable person or persons to carry out the
review and to report to him, pursuant to section 25 of the State Sector Act.
Under section 25 of the State Sector Act, the Commissioner and persons he appoints under
section 25(2) of the State Sector Act have the same powers to summon witnesses and to
receive evidence as are conferred on a Commission of Inquiry by the Commissions of Inquiry
Act 1908.
Timing
The reviewer will report and make recommendations on:
•
the questions concerning the introduction of the New Zealand Scholarship qualification
by 29 April 2005, and
•
the performance of the NZQA by 31 July 2005.
The two parts of the review have separate reporting dates to enable any findings and
recommendations concerning the New Zealand Scholarship examinations to be taken into
account, as far as will be possible, in the administration of the 2005 examinations.
V A (Tony) Hartevelt
Deputy State Services Commissioner
60
Annex 2
Organisations and individuals consulted
New Zealand Qualifications Authority Board
New Zealand Qualifications Authority Senior Managers
Ministry of Education
Education Review Office
The Treasury
Department of the Prime Minister and Cabinet
State Services Commission
Former Chief Executives and Board Members of NZQA
Business New Zealand
Employers and Manufactures Association
Independent Schools of New Zealand
Institutes of Technology and Polytechnics of New Zealand
New Zealand Catholic Education Office
New Zealand School Trustees’ Association
New Zealand Vice Chancellors’ Committee
New Zealand Post Primary Teachers’ Association
New Zealand Post Primary Teachers’ Association Principals’ Council
School Principals’ Association of New Zealand
Secondary Schools
New Zealand Council for Educational Research
Education Specialists from a range of Tertiary institutions including:
The University of Auckland
Manukau Institute of Technology
Massey University
Victoria University of Wellington
Canterbury College of Education
The University of Otago
61