Summary: Trial Evaluation Report - Model for Quality Assuring Mātauranga
Māori Qualifications and Programmes of Study
In 2010, the New Zealand Qualifications Authority (NZQA) released for public
comment a proposed model for the quality assurance of mātauranga Māori
qualifications and programmes of study.1
A consultation summary report was subsequently published which proposed further
work including further model development, and a trial and evaluation with selected
providers.2
Throughout February-April 2011, a trial was conducted using draft tools, processes,
and resources developed specifically for the purpose of evaluating mātauranga Māori
programmes. The evaluation was not a trial of the programmes themselves. Five
tertiary education organisations (TEOs) took part, offering programmes at both
diploma and degree level. The participating organisations were:
a) Te Wānanga o Raukawa
b) Waiariki Institute of Technology
c) Eastern Institute of Technology
d) Te Whare Wānanga o Awanuiarangi
e) Te Pū Wānanga o Anamata.
An evaluation team comprised of a Lead and Supporting Evaluator and NZQA
observers and TEO management, staff and contributors participated in the trial. The
process involved:
a) Organisations gathering evidence of programme self-assessment systems,
processes and practices in place and submitting or presenting relevant
documentation to the Lead Evaluator prior to the on-site visit.
b) Pre-scoping hui between a TEO and lead evaluator to agree on a specific date,
approach / process for the on-site visit.
c) On-site visit comprising of: discussions with a range of people; observation of
class practice, student activities, resources, etc.; and reading of relevant
documentation.
d) During the trial, each on-site visit was followed by a de-brief hui between NZQA
and the TEO. These hui provided an opportunity for TEOs to provide NZQA with
feedback on their experiences of the trial process.
Finally, trial evaluation reports were written solely for the purposes of testing the
efficacy of the trial evaluation tools and their ability to gather and report evaluation
findings appropriately.3
Quality Assuring Mātauranga Māori Qualifications and Programmes of Study
An evaluation of the trial was conducted by an independent contractor, FEM (2006).
The purpose of the evaluation was to further inform the development of a model for
quality assuring mātauranga Māori qualifications and programmes of study. The
evaluation had two specific focus areas:
 How effective are the model’s toolkit and methodology in quality assuring
selected matauranga Māori programmes of study?
NZQA (2010) Quality Assuring Mātauranga Māori Qualifications and Courses: Consultation Paper March 2010.
NZQA (2010) Quality Assuring Mātauranga Māori Qualifications and Courses - Consultation Summary Report: p. 8.
3
The draft reports were produced as part of the trial process and are confidential to the respective participants.
1
2

To what extent, and in what ways, does NZQA have the capacity and capability to
implement the model? What nature and level of capacity and capability is
required to prepare for implementation in the future?
The report’s findings are summarised below:
1.
Key strengths of the draft toolkit used in the trial (KEQs, indicators and performance criteria)
included:
 Its conceptual validity and alignment with ‘what matters’ in both NZQA and mātauranga
Māori provider worldviews.
 Use of te reo Māori in key elements such as the KEQs and performance rubrics went
some way toward making the tools more relevant and acceptable to mātauranga Māori
providers (with considerable room for further improvements).
2.
Key strengths of the processes used in the trial included:
 Alignment with good kaupapa Māori engagement principles and practices as well as with
standard EER processes.
 Up-front investment in relationships established credibility with TEOs and demonstrated
an understanding of relationship development as central to good process with providers.
 Demonstration of NZQA’s willingness to be guided by the tikanga of providers and adapt
processes to meet their expectations.
3.
Two key areas where the model’s toolkit and methodology can be significantly improved for
implementation are:
a) Further developing and strengthening the engagement approach, including building in
processes for:
 Sufficient face to face engagement to: build effective relationships; ensure
understanding of the process; agree protocols for engagement; identify priority focus
areas; and offer flexibility for TEOs to ‘tell the story their way’.
 Assessing TEOs’ understanding and perceptions about the purpose, content and
processes of mātauranga Māori evaluative quality assurance; providing information
(spoken and written); and checking / verifying provider understanding prior to the
evaluation.
 On-going support to assist providers in building long-term capacity and capability in
self-assessment and quality assurance.
b) Improving the clarity of the toolkit and methodology for TEOs and evaluators by:
 Developing simplified written information to assist TEOs in understanding and
preparing for the evaluation.
 Increasing the extent of te reo Māori translation used in the toolkit and any supporting
resources developed.
 Developing simplified guidelines for evaluators use.
4.
Key strengths of NZQA capacity and capability were:
a) A demonstrated commitment to continuous improvement and making the model work for
mātauranga Māori providers, together with a capable team of evaluators.
b) NZQA’s success in developing capacity and capability in its pre-existing pool of
evaluators, and its credibility within the Māori evaluation community, that such a pool was
available to participate in the trial.
5.
There remains considerable room to develop and grow the mātauranga Māori evaluator
pool, including:
a) Long term, developing a pool of seasoned evaluators with sufficient depth of knowledge
and experience to implement mātauranga Māori evaluative quality assurance.
b) Over the short to medium term, investment in: growing evaluator capacity; providing ongoing skill development opportunities; and resourcing.
c) Exploring the offer from the triallists to: work alongside pool evaluators when conducting
evaluations; and utilising TEOs as a source of potential mātauranga Māori evaluators.
Triallist Feedback
Trial participants also continue to provide valuable comment and feedback on the
proposed approach to quality assuring mātauranga Māori qualifications and
programmes of study. Points raised include:

A participatory approach to arriving at evaluative conclusions is not evident in the
final reporting processes or report format, language and structure.

There is a need for a Māori world view to be reflected in the model, otherwise it
simply looks and feels like the existing external evaluation and review approach.

Evaluators capable of engaging, evaluating, reporting and managing relationships
are critical for ensuring an enriching and empowering quality assurance
experience from beginning to end.

To be “fit for purpose”, the quality assurance of mātauranga Māori qualifications
and programmes of study must be based on kaupapa Māori principles. These
principles then need to be reflected throughout the model’s tools, processes and
methodologies.

The model needs to be able to recognise and take account of the distinctiveness
of each TEO and how they respond or meet the needs of their key stakeholders,
including ākonga, their whānau, hapū, iwi and wider communities.
Next Steps
The report’s findings and feedback from key stakeholders will continue to inform
NZQA’s approach to the quality assurance of mātauranga Māori qualifications and
programmes of study. This will include:

Development of a plan identifying short-, medium- and long-term strategies to
support the sustainable development of mātauranga Māori evaluative quality
assurance (MM EQA) capacity and capability.

Strengthen MM EQA by embedding, at its core, key mātauranga Māori principles,
then ensuring they are reflected in the MM EQA tools, processes and
methodologies.

Work closely with trial participants and other key stakeholders in making final
improvements to the MM EQA tools, processes and methodologies.

Producing documentation and supplementary resources that provide information
in a clear and concise way, for clients and audiences, including NZQA analysts
and evaluators, qualification developers and TEOs.