NZQA
NZQA -Initial PTE Assessment- Report 2602 2010.pptx
1

1. What you asked for
2. What we did
3. Our findings and summary
4. Our recommendations
5. Implications of recommendations and other issues
6. Next steps
NZQA -Initial PTE Assessment- Report 2602 2010.pptx
2

1.1 To conduct an urgent independent high level assessment of the policies and procedures relating to NZDipBus with a view to highlighting the issues and identifying possible remedial paths in respect of defaults currently evident.
1.2 Consider the implications on the overall quality assurance systems underpinning the NZDipBus qualification and make findings and recommendations relative to enhancing the overall quality assurance system, including comment relative to: i. The adequacy, accuracy and processing of the reported and monitored data collected.
11.
Developing remedial options in respect of NZDipBus and the overall assurance system.
111.
The appropriateness of NZQA response to adverse PTE performance/behaviour.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
3

(cont.)
1.3 The review was defined as:
1.
Desk based and to be completed within a period of three weeks. ii. To provide a pathway for more intensive analysis, review and solution formulations.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
4

In conducting this high level review we have:
1. Been briefed by DCE Quality Assurance and the DCE Advisor.
2. Interviewed a range of management personnel (see Appendix 1).
3. Read and reviewed a range of documentation relative to NZDipBus and the current 2010 QA processes.
4. Sought to draw on regulatory QA comparisons.
5. Drawn on our own material relative to other quality assurance systems.
6. Drawn initial findings/ drafted recommendations/ and sought to test them with NZQA.
7. Met with Immigration New Zealand Business Relationship Adviser.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx 5

• NZQA's primary function is to protect the standing of the individual qualifications, the qualifications assurance system, NZQA (the regulator), the tertiary sector and New Zealand's educational interest.
• It is virtually impossible to have a failsafe quality assurance system.
• Fraud is particularly difficult to detect.
• All regulators face the fact there could be points of failure on their watch.
• The most critical factor is to promptly remedy failure and strengthen the system in the light of the learning.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
6

1. NZQA's current quality assurance system is comprehensive and not broken.
2. An ideal regulatory quality assurance system contains incentives and sanctions which serve to align the interests of the assurer and the assured. Each party has significant interest in maintaining system compliance.
3. There is much to commend in NZQA's current (2009) QA programme
4. There is high commonality between the NZDipBus processes and the generic systems for registration, accreditation, approval, selfassessment, moderation, reporting and communication.
5. NZDipBus policies and guidelines make this qualification susceptible to misuse (high cross credits entitlement).
6. NZQA's QA system would benefit from greater deterrence influence.
7. Legislative provisions exist to allow NZQA to impose provider conditions on registration and accreditation. This ability could be used to address the current capability and response imbalances and interest misalignment.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx 7

(cont.)
8. In terms of the QA system, our initial assessment is:
~ Quality assurance philosophy
~
~
Gate keeping into the system
Validation of service offering
(courses/unit standards, etc)
~ Risk monitoring
~ Assessment
~ Monitoring
~ Moderation
~ Evaluation and review
~ Dynamic balance
(alignment of interests)
~ Powers in default, and sanctions
- Comprehensive could benefit from greater risk based differentiation
- Good
- Good
- Satisfactory to good
- Needs greater validation
- Focus change being effected
- Needs risk refinement enhancements
- Early days/sound approach
- Needs refinement
- Needs enhancing
9. The tertiary sector (and the PTE sub-sector) comprises a diverse range of providers.
10. The high trust, high accountability philosophy is commendable and generally appropriate for the sector.
11. The adoption of the annual capability and performance statutory attestation process provides grounds to lodge a criminal case where default (perjury).
8
NZQA- Initial PTE Assessment- Report 2602 2010.pptx

12. The QA monitoring process is multi-faceted:
Moderation
Specific monitoring for NZDipBus
Assessment reporting and analysis
Reactive risk response (i.e. complaints, etc)
Annual reporting and attestation
Aspect focused external evaluation and review
Full entity focused external evaluation and review
There are benefits in better facilitating the collection and collation of sector intelligence
Stronger enforcement and prosecution powers and capability would more effectively protect the system
Market based intelligence and/or whistle blowing should be encouraged to enable industry monitored integrity
There is an effective relationship with Immigration NZ that could be extended and formalised into a MOU
NZQA should be able to respond promptly to identified breaches and use secured powers to stabilise the position, prevent further damage, and protect learner records
13. NZQA to act promptly when material default is identified to minimise damage to the qualifications system, the regulator, and the national interest.
14. There is a marked contrast between the interests of ITPs and their students and for example some new PTEs focused on the foreign education market.
15. The prime interests of some students may not necessarily fully relate to the qualification but also to the quantum of points available for a residency application.
16. The imperative to act promptly to safeguard the qualifications system is currently thwarted by notice periods and the principles of natural justice.
9
NZQA- Initial PTE Assessment- Report 2602 2010.pptx

• Our findings have initiated a suite of recommendations. They have been devised so they can integrate with the current QA programme and provide the necessary strengthening and rigour necessary. While few on their own represent major change or investment, collectively they define a new focus, a revised orientation, and a materially changed approach. There exists now an imperative to act to safeguard the interests of a range of stakeholders.
NZQA as the regulator is the appropriate party.
• The aim is to operate a successful and cost effective QA system that fulfils the function of protecting the standing of the qualification, the system, the regulator (NZQA), the sector, and national interests.
• Our objective with these, recommendations is to seek to refine and enhance the system to enable greater quality assurance at lower cost and lower risk.
We consider there is a need to revisit and revise the respective rights and obligations of the key stakeholders and create an enduring and positive tension.
• NZQA's current quality assurance system is comprehensive and not broken.
It will benefit from a series of refinements that serve to strengthen the information collection and analysis, co ordination, capability and breach response options.
• We recommend NZQA consider introducing the following refinements:
NZQA- Initial PTE Assessment- Report 2602 2010.pptx 10

1. Act promptly on these recommendations following review and assessment to reduce ongoing risk of reputational damage. NZQA to develop an implementation plan to integrate the recommended refinements into the existing quality assurance framework.
2. Strengthen and rebalance the monitoring and response components of the current quality assurance system following a risk based approach.
3. Recognise the different incentives of different providers and adopt a risk based approach for different providers (high risk - higher assurance activity, higher compliance costs).
4. Create up to four risk categories and apply these to PTE entities.
5. Use current legislative provisions in the Education Act to impose where applicable, conditions relative to a range of aspects including: a. Provision of a bond or surety, or personal guarantee. b. NZQA's right to copy or retrieve academic records in the face of easily evidenced condition default. c. Independent verification of academic achievement (random validation). d. Student enrolments to agree to participating in any NZQA quality assurance processes. e. Tightening of prior learning recognition where risk high (limit NZDipBus to 4 credit papers). Special case applications to NZQA to assess, where material hardship can be evidenced. f. Independent assessment of relevance of prior learning where risk high g. Changed frequency, focus and coverage of monitoring visits.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
11

(cont.)
5. Design a specific set of registration conditions, responsibilities, powers and obligations for each risk profile (becomes the new 'bar').
6. Strengthen remedial powers by imbedding rights, and obligations and powers into the registration and accreditation agreements to protect brand.
7. Revise the approach to moderation, either for risk related categories and introduce random selection and variable sample size.
8. Tertiary records to require a broader set of report data and more regular reporting from providers.
9. Report data to be analysed and assessed and KPis compared between similar providers, e.g. average number of cross credits granted per qualification.
10. Introduce strictly controlled personally assigned access rights to the
Tertiary Records database and require a two or three party approval system.
11. Introduce a separate step (for assessments) prior to awarding a qualification (undertake a range of personal checks including with
Immigration New Zealand).
12. Enter into an agreement with Immigration NZ relative to acceptable institutions and courses eligible for visa entry and residency point credits.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
12

13. Encourage the collation and analysis of market intelligence.
Strengthen the accessibility and confidentiality of the whistle blower facility. Assess the suitability of the Independent Integrity plus line to add valuable sector intelligence.
14. Insert into the registration conditions the right for NZQA, on evidence of certain defaults, to place the provider on a 'watch list', such watch list likely to result in Immigration NZ withdrawing residency credit entitlement.
15. Work in a staged manner to develop the PTE review programme.
Other instances of malpractice may not have the same modus operandi but produce similar adverse impacts. The review programme should consider all potential areas of malpractice.
16. Establish a provider database (CRM as part of the new SNAP system) holding a single version of the 'key data' and facilitate input from all divisions. Database to hold a range of searchable result or assessment fields that will enable management action, e.g. 'assessed level of competence in the organisation' or 'likelihood of failure' (low, medium and high).
NZQA -Initial PTE Assessment- Report 2602 2010.pptx 13

17. At certain risk levels require a surety or bond to be provided to underpin the costs should default occur. Gain the ability to levy costs or fines.
18. Material statutory attestation breaches to lead to criminal prosecution with cases prepared by NZQA or specialist contracted investigators and case preparers. When completed to be tabled with Police.
19. NZQA initiate without delay a programme of change to incorporate all or most of these recommendations. The programme to be staged to enable early low cost gains. The rationale for the investment being the reduction in the level of risk to individual learners, PTE businesses, qualifications, the quality assurance system and the economic activity. The progression will need a sponsor, a clear vision, a budget and a time frame, and the specific assignment of responsibilities and accountabilities.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
14

• The following implications and issues require further definition, analysis and review.
1. Reconciling the heightened risk based approach, including the differential registration conditions with current operating high trust philosophies.
2. The ability to use current legislatively empowered 'conditions' and other clauses to implement many of the recommendations needs specialist legal assessment.
3. To define the provider database structure and content, including the extent of Tertiary Records analysis, and the system's points of input and reporting.
4. To further review the current 2010 risk based monitoring approach and programme, PTE reporting requirements, tertiary sub-sector analysis, and evaluation and review approach to identify what refinements could address risk identified in this high level assessment.
NZQA -Initial PTE Assessment- Report 2602 2010.pptx
15

1. Gain feedback from project sponsor and incorporate in report.
2. Present to management and incorporate feedback.
3. Summarise outstanding issues and plan to resolve.
4. Workshop and brainstorm most appropriate approach to PTE review programme. Incorporate DipBus assessment learnings.
5. Draft design, build and implementation plan comprising six phases: i. Project establishment and tasking of team. ii. In depth assessment of report and implications. iii. Assess feasibility and select best integrated approach. iv. Build and implement revised QA system on staged basis. v. Implement stages. vi. Maintain supporting stakeholder communication throughout stages. vii.Review and evaluate changes and outcome.
6. Work with independent assessor to gain continuity and review experience and expertise.
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
16

• Mr Tim Fowler- DCE Quality Assurance
• Mr Michael Steer- Advisor to DCE Quality Assurance
• Ms Kathy Mclaren - Manager Registration, Approvals and
• Mr Tony Duffin - Manager, Risk and Compliance
• Ms Lynda Glogau -Manager Qualifications, Development and
• Ms Val Jolly - Manager Moderation
• Mr Grant Adam - NZQA Solicitor
• Mr Paul Stone - Manager External Evaluation Review (tbc)
• Mr Barry Madgwick - Team Leader Tertiary Records
• Ms Celia Coombes - Relationship Manager Immigration NZ
NZQA- Initial PTE Assessment- Report 2602 2010.pptx
17

DRAFT- Commercial in Confidence
Date
Programme manager
Completion
NZQA- QA Upgrade Programme schedules 120410 a el(11).xlsx
I
Title