Portfolio Committee on Basic
Education
Report on the Quality
Assurance of the NSC
12 February 2013
Dr Mafu S Rakometsi
WHAT CAN WE LEARN FROM THE NSC
RESULTS?
Introduction to context, principles,
approaches and processes
Dr Mafu Rakometsi - CEO of Umalusi
Regulatory Framework
Quality Assurance of Assessment
•
NQF Act Section 27 (h)
The QC must develop and implement policy and
criteria for assessment for the qualifications on its
sub-framework.
Section17 of the GENFETQA Act
(5) The Council must, with the concurrence of the Director-General and
after consultation with the relevant assessment body or education
institution, approve the publication of the results of learners if the
Council is satisfied that the assessment body or education institution
has—
(i) conducted the assessment free from any irregularity that may
jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for
conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is
required to comply with in order to obtain a certificate; and
(iv) complied with every other condition determined by the Council.
Framework for QA of Learner Achievement

•
Based on established and existing practices in
assessment for certification
Prescribed components of External
assessment (examinations) and Site based/
internal / continuous assessment
• Use of systems, processes, and procedures to
evaluate, inspect, monitor and report on
examination systems, processes and
procedures of public and private assessment
bodies.
Framework for Quality Assurance of Assessment

Evaluation and /or accreditation of assessment bodies

Periodic inspection of assessment systems

Ongoing monitoring of assessment systems

Quality assurance of external examinations through:

Moderation of examination question papers

Monitoring and moderation of SBA

Monitoring the conduct of examinations

Moderation of marking

Standardization of assessment outcomes
Approval for the release of Results
Approval is based on the following requirements:
•
The examinations are conducted compliant to the
applicable policies regulating the conduct and
administration of the examinations
•
At the time of approval, there is no serious
irregularity which could undermine the credibility of
the examinations.
Quality Assurance of the DBE 2012
National Senior Certificate Examination
Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments
Moderation of question papers
PURPOSE

To ensure that the question papers are of the required standard
(standard captured in the NCS and SAG’s)


To ensure that the question papers are relatively:
- fair
- reliable
- representative of an adequate sample of the curriculum
- representative of relevant conceptual domains
- representative of relevant levels of cognitive challenge
External moderators
Moderation of the question papers
Approach

Question papers set by panel of examiners –
DBE

Internally moderated by DBE

Externally moderated by Umalusi

Subsequent moderations and approval.
Moderation of Question papers
Number of NSC 2012 question papers moderate
Number
of
subjects
Number of
papers
Approved/
Conditionally
approved at
1st
moderation
Approved/
Conditional
ly
approved
at 2nd
moderation
Approved/
Conditiona
lly
approved
at 3rd
moderatio
n
4th
moderatio
n and
beyond
62
(including
the nonofficial
languages
and LO)
Nov 2012 -132
27.3% (36)
56.8% (75)
15.2%(20)
0.7% (1)
Mar 2013 -130
27,7% (36)
54.6 %(71)
13.9% (18)
3.8% (5)
Moderation of the question papers
Criteria

Technical Criteria

Internal moderation – quality of internal moderator reports.

Content coverage

Cognitive Demand

Marking Guidelines

Language and Bias

Adherence to Policies and guidelines

Predictability
Moderation of the question papers
Findings: Areas of Good Practice



Percentage of question papers and memoranda approved after
first and second moderation ( Nov 2012- 79,6% ; Mar 13 –
72,3%)
Simultaneous moderation of final and supplementary question
papers.
Standard of Internal moderation
Moderation of the question papers
Findings: Areas of Concern



Adherence to timeframes and impact on quality
of setting and moderation.
Question papers requiring more than four
moderations.
Role of the external moderator.
Moderation of internal /continuous
assessment


Internal assessment refers to any assessment
conducted by the provider, the outcomes of
which count towards the achievement of the
qualification
panels of moderators / subject specialists
Moderation of internal assessment (cont.)
Purpose of Umalusi’s verification:





To verify the rigour and appropriateness of the DBE moderation
process – linked to DBE plans
Ascertain the degree to which assessment bodies/provinces are
attempting to ensure standardisation across
Ascertain the standard and quality of the tasks
Determine the extent and quality of internal moderation and
feedback.
Determine the reliability and validity of the assessment
outcomes
Moderation of Internal Assessment
Phase 1 (June/July 2012)- focus on assessment
instruments (teacher files)
PROVINCIAL EDUCATION
DEPARTMENT
Subjects
Eastern Cape
KwaZulu- Natal
Limpopo
Mpumalanga
Mathematics, Life Sciences,
Physical Sciences
Accounting, English FAL, History,
Geography, Life Orientation,
Moderation of Internal Assessment
Phase 1: Findings
•
Quality and Standard of Assessed tasks

Content Coverage – mostly appropriate

Cognitive demand and difficulty levels –standards variable
mainly at lower cognitive levels,

Over reliance on previous question papers.

Projects and practical investigations- not cognitively
balanced, more theory than practice.

Marking tools – not applied consistently

Teachers subject knowledge often inadequate

Physical Education Task (PET) inappropriately assessed.
Moderation of Internal Assessment
Phase 1: Findings
•
Internal Moderation

Moderation instruments – compliance vs. quality

School moderation – seriously neglected

Monitoring vs. Moderation
•
Feedback and Support – general support by subject advisors.
Very little developmental feedback.
Moderation of Internal Assessment
Phase 1: Areas of Good Practice
•
Use of Common Tasks
•
Diagnostic Analysis to improve performance
•
Compliance with Internal Assessment protocols.
Moderation of Internal Assessment
Phase 2 (October/November 2012)- focus on learner
evidence and teacher files.
PROVINCIAL EDUCATION
DEPARTMENT
Subjects
All 9 PED’s
Mathematics, Life Sciences,
Physical Sciences
Accounting, English FAL, History,
Geography, Life Orientation,
Moderation of Internal Assessment
Phase 2: Findings
•
Quality and Standard of Moderation

Brief unclear – focus on marking only and not on quality of
task.

Shadow marking vs. moderation
•
Marking of the Tasks

Marking tools – not applied consistently

Teachers subject knowledge often inadequate

Challenges with marking extended writing and research
projects.

Marking too lenient
Moderation of Internal Assessment
Phase 2: Findings
•
Learner performance

Difficulty with higher-order questions

Ability to express/respond in English a determining factor

Extremely poor performance in certain subjects e.g. Maths,
LO Common Assessment Task,

Evidence suggests that very little effective teaching and
learning is taking place
Moderation of Internal Assessment
Phase 2: Findings
•
Internal Moderation at School, Cluster and District Levels.

Varying quality of Moderation

Limited moderation of candidates work – shadow marking

Little evidence of moderation of tasks.

Lack of mark sheets or incorrect recording of marks.
Moderation of Internal Assessment
Phase 2: Areas of Good Practice
•
Use of Common Tasks
•
Diagnostic Analysis to improve performance
•
Compliance with Internal Assessment protocols.
•
Evidence of moderation at the various levels.
•
Introduction of National Moderation
Monitoring of Examinations
•
State of readiness
•
Conduct of examinations
•
Marking
State of Readiness
Monitored the DBE audit visits (March/April) and evaluation visits
to all
Comprehensive approach that monitors exam systems (including
SBA)
Monitoring of Examinations
State of readiness
Findings
•
All provinces have examination systems in
place
•
Vacant posts and use of contract staff is a
major concern.
Monitoring of the writing phase.
No of exam centres
No of exam centers
No of Umalusi monitors
monitored by Umalusi per province
6623
207
36
Findings
•
General Management of the examinations

Generally examinations conducted in line with policy

Isolated instances of non-compliance (suitability of venue,
noise, security, contingency plans)

Shortage of question papers and answer books

Provincial Monitoring
Monitoring of the marking phase
Total
No of
marking
centers
(2010)
No of
marking
centers
(2011)
No of
marking
centers
(2012)
No of
marking
centers
monitored by
Umalusi
127
123
118
87
Monitoring of the marking phase
Findings
•
In general marking processes in line with policy
•
Concern with lack of security at some marking centres
•
Concern with the appointment and training of markers
•
Concern with the quality and standard of marking and internal
moderation.
•
Number of marking centres – security risk
Verification of marking
PURPOSE
•
•
Moderation of marking determines the standard and
quality of marking and ensures that marking is
conducted in accordance with agreed practices
Umalusi engages the following during the
moderation of marking
1. Pre-marking/memorandum discussion:
centralised memo discussions recommended - this
will ensure consistency across marking centres
2. Moderation of marking (off-site and on-site)
Verification of marking

Memoranda discussion meetings

Approved and signed off finalized memoranda/
marking guidelines for all the NSC subjects

Concern with non attendance and pre- memo
discussion preparations

Concern with timing and scheduling of memo
discussions.

Practice of marking and training in the memo
discussions impacts positively on standard of
marking.
Verification of marking
Provinces
Subjects
Centralised
All 9
Gateway subjects + 4
others
On-site
Eastern Cape; Gauteng;
KZN; Limpopo; Northern
Cape
13 in total
Findings
Findings
• Adherence to Marking memo
• Marking generally accurate – problem with
marking extended writing and essays
Verification of marking
• Quality of marking varied. Question marking improved
consistency.




Competency of markers to mark interpretation type
questions.
Internal moderation
No unauthorised changes to marking guidelines
reported
Training of chief markers and internal moderators at
memo discussions having a positive impact on
marking.
Standardisation and Verification of
Resulting
•
Provision of GENFETQA – Council may adjust raw
marks.
•
International practice – large scale assessment
systems
•
Standardisation – process used to mitigate the effect
of factors other than learners knowledge and
aptitude on the learners performance.
•
Sources of variability – difficulty in question paper,
undetected errors, learner interpretation of questions
Statistical moderation
Standardisation

58 subjects standardised

Raw marks accepted: 41 subjects

Moderated Upward : 5 subjects

Moderated Downward : 12 subjects
Verification of the Resulting Process
Planned
Subject structures
Candidate registration
Status
Subject structures verified.
System verified during state
of readiness visits.
Generation of mark Sheets
Capturing of Marks
Standardisation data &
Booklets
Capturing of adjustments
Monitored
Monitored
Data sets received and
verified
Verified
Statistical moderation &
resulting
Verified
Examination Irregularities



The majority of irregularities were of a technical nature and
these were reported to Umalusi according to the established
channels.
Some irregularities were as a result of registration-related
problems, e.g. candidates nor appearing on mark sheets, some
registered for incorrect subjects.
Umalusi represented on NEIC
Major irregularities:

Alleged leakage of Question Papers in KZN
WHAT CAN WE LEARN FROM THE NSC
RESULTS?
STANDARDISATION DECISIONS
DBE NSC 2012
SUBJECT
Life Sciences
Physical Sciences
Mathematics
Mathematical Literacy
Mathematics: Probability;
Data Handling
ADJUSTMENT
CA( 0 at 0 scale to -8 at 23.
scale to 0 at 70, scale to +3 at
119, scale to 0 at 171, scale to
-7 at 239, scale to -1 at 291,
scale to -7 at 300)
0 at 0, scale to -12 at 102,
block -12 from 102 to 252,
scale to 0 at 300
Raw
0 at 0, raw to 114, scale 0 to
+9 from 114 to 240, scale to 0
at 300
Raw
SUBJECT
ADJUSTMENT
Geography
0 at 0, scale to -6 at 96, block
-6 from 96 to 180, scale to 0 at
240, raw to 300
Life Orientation
Hospitality Studies
Raw
Raw
Tourism
½ CA( 0 at 0, scale to -12 at
121, scale to 0 at 300)
Agricultural Science
Consumer Studies
Raw
Raw
SUBJECT
Agricultural Management
Practices
Raw
Agricultural Technology
Raw
Music
Raw
Dance Studies
Raw
Design
0 at 0, scale to -12 at 102,
scale to -6 at 246, scale to 0 at
300
Dramatic Arts
ADJUSTMENT
½ CA(o at o, scale to -15 at 66,
scale to 0 at 300)
SUBJECT
Visual Arts
ADJUSTMENT
Raw
Religion Studies
Raw
Information Technology
Raw
Computer Applications
Technology
Raw
Civil Technology
Electrical Technology
0 at 0, scale to -9 at 90, block
-9 from 90 to 180, scale to 0 at
300
½ CA(0 at 0, scale to -15 at
211, scale to 0 at 300)
SUBJECT
ADJUSTMENT
Mechanical Technology
Raw
Engineering Graphics and
Design
Accounting
Business Studies
Economics
Raw
0 at 0, scale to -9 at 78, block 9 from 78 to 222, scale to 0 at
300
Raw
Raw
SUBJECT
ADJUSTMENT
History
Raw
Afrikaans HL
0 at 0, scale to +4 at 116,
block of +4 from 116 to 247,
scale to 0 at 300.
Afrikaans FAL
Afrikaans SAL
Raw
Raw
English HL
0 at 0, scale to +8 at 112,
block of +8 from 112 to 240,
scale to 0 at 300.
SUBJECT
English FAL
ADJUSTMENT
0 at 0, scale to -6 at 51, block
of -6 from 51 to 189, scale to 0
at 300.
English SAL
Raw
IsiNdebele HL
Raw
IsiNdebele FAL
IsiZulu FAL
Raw
0 at 0, scale to +8 at 112,
block of +8 from 112 to 232,
scale to 0 at 300.
Raw
IsiZulu SAL
Raw
IsiZulu HL
SUBJECT
ADJUSTMENT
Setswana HL
Raw
Setswana FAL
Raw
Setswana SAL
Siswati HL
Raw
Raw
Siswati FAL
Raw
Siswati SAL
Raw
IsiXhosa HL
Raw
IsiXhosa FAL
Raw
IsiXhosa SAL
Raw
SUBJECT
Xitsonga HL
ADJUSTMENT
½ CA (0 at 0, scale to -13 at
74, scale to 0 at 300)
Xitsonga FAL
Raw
Tshivenda HL
Tshivenda FAL
½ CA(0 at 0, scale to -12 at
115, block -12 from 115 to 153,
scale to 0 at 300)
Raw
SUBJECT
ADJUSTMENT
Sepedi HL
Sepedi FAL
Raw
Raw
Sesotho HL
Raw
Sesotho FAL
½ CA(0 at 0, scale to -15 at
300)
Sesotho SAL
Raw
Quality Assurance of the IEB 2012
National Senior Certificate Examination
Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments
Moderation of Question papers
Number of IEB NSC 2012 question papers
moderated
Number
of
subjects
Number of
papers
Approved/
Conditionally
approved at
1st
moderation
Approved/
Conditional
ly
approved
at 2nd
moderation
Approved/
Conditiona
lly
approved
at 3rd
moderatio
n
4th
moderatio
n and
beyond
39
Nov 2012 -69
39.7% (28)
54.4% (37)
4.4%(3)
1.5% (1)
Moderation of the question papers
Findings: Areas of Good Practice


Percentage of question papers and memoranda approved after
first and second moderation ( Nov 2012- 94.1% )
Standard of Internal moderation
Moderation of Internal Assessment
Phase 2 (November 2012)- focus on learner evidence
and teacher files.
IEB
Subjects
Mathematics, Maths Lit, Life Sciences,
Physical Sciences
Accounting, English HL, History, Life
Orientation, Business Studies, Economics
Moderation of Internal Assessment
Findings: Areas of Good Practice
•
Regional and national moderation is in place and well
executed
•
The marking of tasks of a highly acceptable standard
Findings: Areas of Concern
•
Feedback to the learners
•
Assessment of Physical Education Tasks in LO – distance
education learners
Monitoring of the writing phase.
Findings
•
General Management of the examinations

Generally examinations conducted in line with policy

Isolated instances of non-compliance (candidate
identification)

Lack of Irregularity Registers

Assessment Body Monitoring
Verification of marking
8 Subjects
Findings
• marking and moderation of scripts is extremely
thorough
• Marking generally accurate
• practice of double marking is highly commendable
• Moderation doesn’t always involve full moderation of a
script
Statistical moderation
Standardisation

58 subjects standardised

Raw marks accepted: 44 subjects

Moderated Upward : 9 subjects

Moderated Downward : 4 subjects
WHAT CAN WE LEARN FROM THE NSC
RESULTS?
STANDARDISATION DECISIONS
IEB NSC 2012
SUBJECT
Accounting
Business Studies
Computer Application
Technology
Consumer Studies
ADJUSTMENT
0 at 0, scale to -6 at 96; block 6 from 96 to 212, scale to -5 at
245, back to 0 at 300)
½ CA(0 at 0, scale to -15 at
139, block of -15 from 139 to
208, scale back to 0 at 300)
½ CA (Raw from 0 to 101,
scaled from 0 at 101 to +7 at
219, scaled to 0 at 300)
Raw
Design
Raw
Dance Studies
Raw
SUBJECT
Dramatic Arts
Economics
Engineering Graphics and
Design
Geography
ADJUSTMENT
Raw
½ CA(0 at 0, scaled to +10 at
45, block +10 from 45 to 102,
scaled to 0 at 300)
½ CA ( 0 at 0, scale to +6 at
41, block of +6 from 41 to 79,
scale to 0 at 300)
CA(0 at 0, scale to +12 at 36,
scale to -7 at 59,scale to -1 at
121, scale to +8 at 173, block
+8 from 173 to 200, scale to 0
at 300)
SUBJECT
History
ADJUSTMENT
CA (0 at 0, scale to -18 at 56,
scaled to -25 at 109, scaled to
0 at 300
Information Technology
Raw
Hospitality Studies
Raw
Tourism
Raw
Visual Arts
½ CA ( 0 at 0, scale to -15 at
89, block -15 from 89 to 222,
scale to 0 at 300)
Music
Raw
SUBJECT
ADJUSTMENT
Religion Studies
Raw
Life Orientation
Raw
Life Sciences
Raw
Mathematical Literacy
Raw
Mathematics
Mathematics P3
Raw
Raw
SUBJECT
Physical Science
English FAL
English HL
Afrikaans FAL
ADJUSTMENT
½ CA ( 0 at 0 scale to -7 at 28,
scale to 0 at 68, block +1
from68 to 74, scale to 0 at110,
scale to -7 at 241, scale to 0 at
300)
½ CA (0 at 0, scale to +15
at64, scale to +6 at 221, scale
to +10 at 258, scale to 0 at
300)
Raw
½ CA (0 at 0, scale to -10 at
147, block of -10 from 147 to
175, scale back to 0 at 300)
SUBJECT
ADJUSTMENT
Afrikaans HL
Raw
Sepedi FAL
Raw
Sesotho FAL
Raw
Setswana FAL
Raw
Siswati FAL
Raw
Siswati HL
Raw
IsiXhosa FAL
IsiZulu HL
Raw
Raw
SUBJECT
IsiZulu FAL
IsiZulu SAL
Arabic SAL
ADJUSTMENT
CA( 0 at 0, scale to -16 at 146,
scale to -6 at 166, block -10
from 166 to 242, scale to 0 at
300)
Raw
French SAL
Raw
½ CA( 0 at 0, scale to -13 at
51, scale to -1 at 174, scale to 5 at 206, block -5 from 206 to
279, scale to 0 at 300)
German HL
Raw
German SAL
Raw
SUBJECT
ADJUSTMENT
Gujurati SAL
Raw
Hebrew SAL
0 at 0, raw to 180, scale to -6
at 246, scale to 0 at 300
Hindi FAL
Raw
Hindi SAL
Raw
Italian SAL
Raw
Latin SAL
Raw
SUBJECT
ADJUSTMENT
Modern Greek SAL
Raw
Portuguese HL
Raw
Portuguese SAL
Raw
Spanish SAL
Raw
Tamil SAL
Raw
Urdu FAL
Raw
Urdu SAL
Raw
Serbian HL
Raw
SUBJECT
ADJUSTMENT
Equine Studies*
Raw
Sport and Exercise Science* Raw
Maritime Economics*
Raw
Nautical Science
Raw
Advanced Programme
Mathematics
Advanced Programme
English
½ CA( Block +5)
Raw
Annexure to Approval Letter to
DBE
Specific concerns regarding the quality assurance of the NSC
examination and assessment:


Non-adherence to timeframes for the submission of question
papers must be addressed. It is important to note that for the
credibility of the NSC examination it is vitally important that
every effort is made to adhere to agreed deadlines for setting
and moderation of question papers. Any delays affect Umalusi
in its quality assurance exercise.
In addition to what is said above, every effort should be made to
set question papers of an appropriate quality and standard so
that they can be ideally approved at first moderation.

Low levels of performance in key subjects. This issue need to
be looked into and addressed so as not to compromise the
standard of the NSC.
• Umalusi is concerned with the problem in Life Orientation with
awarding of marks at the 80% level where there is clear
evidence of tampering with the marks. The DBE is urged to send
a strong message of warning to the provinces indicating that
should this problem persist in the future all learners at the 80%
mark will be adjusted downward.


The quality of School-Based Assessment (SBA) continues to
be a concern. There may be several plausible reasons for this
poor quality of SBA, but it is clear that there is too little control
over work that is done.
It is evident from our reports that progress has been made over
the years with regard to the standard of marking. There is
however room for improvement in the selection and training of
markers, chief markers, senior markers and internal
moderators. Competency tests will go a long way in the
identification of competent markers.
Conclusion


The findings of the quality assurance processes are a clear
indication of a maturing system that has, on the one hand,
made positive strides towards improvement in certain areas of
assessment and examination, but, on the other hand, still has a
few challenges that need to be addressed.
The quality assurance of each of these processes presented
above was conducted based on Umalusi criteria. Umalusi uses
criteria that are subjected to constant review and refinement, to
ensure that they are in line with current trends in assessment
and examinations.
Conclusion…




In general Umalusi is pleased with the manner in which the
2012 NSC examination was administered.
Umalusi acknowledges that a number of technical irregularities
were reported, but these were addressed in a fitting manner.
Umalusi takes this opportunity to express appreciation to the
national & provincial departments of education for their
concerted effort in ensuring a credible examination.
Umalusi expresses appreciation also to all the relevant
stakeholders for the necessary support given in line with
Umalusi quality assurance initiatives.
Thank you!