OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the
Qualifications

December 2010
Ofqual/10/4787
Contents
Executive summary .................................................................................................... 2
Monitoring of the OCR Level 2 Nationals in ICT ......................................................... 6
Scheme of assessment........................................................................................... 6
Validity of assessment ............................................................................................ 7
Assessment process .................................................................................................. 9
Internal assessment ................................................................................................ 9
Moderation ............................................................................................................ 10
Monitoring of other OCR Nationals qualifications ..................................................... 11
Appendix A: List of observed OCR meetings............................................................ 12
Centre moderation visits.................................................................................... 12
Centre Assessor Guidance and Training (CAGT) ............................................. 12
Appendix B: Schemes of assessment ...................................................................... 13
Qualifications ........................................................................................................ 13
Units...................................................................................................................... 14
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Executive summary
This report combines the Office of Qualifications and Examinations Regulation
(Ofqual) scrutiny monitoring findings for the Level 2 1 Nationals in ICT qualifications,
with the outcomes of the agreed Oxford Cambridge and RSA Examinations (OCR)
action plan completed in August 2010.
In the scrutiny monitoring, Ofqual found inconsistencies in the guided learning hours
and comparability of demand across units. As a consequence, there was
considerable variation in the amount of work and level of skill required to complete
the different units, which significantly undermined the comparability of different routes
through the qualification.
In the majority of units reviewed, we found that there was limited potential for
learners at the higher end of the ability range to demonstrate what they know,
understand and can do.
There were also inconsistencies in some of the assessment criteria 2 , where the
requirements were more challenging at merit than at distinction 3 . During the review of
learner work, we found inconsistencies in assessor marking, which in turn revealed
issues with the effectiveness of the moderation process.
In January 2010, OCR submitted Level 2 ICT qualifications for accreditation onto the
Qualifications and Credit Framework (QCF). Ofqual found these qualifications did not
meet fully the requirements of the Regulatory Arrangements for the Qualifications
and Credit Framework (2008) and requested that they were amended and
resubmitted. Following discussions and further amendment to the units, it became
apparent that there would be a substantial amount of work for OCR to transfer these
qualifications to the QCF. As a result, OCR requested that the Level 2 ICT suite of
qualifications remain on the National Qualifications Framework (NQF). The actions
taken within this report relate to the final NQF submission.
After reviewing the final changes OCR made to the unit specifications, marking grids
and guidance for centres and assessors, Ofqual and its regulatory partners (the
Department for Children, Education, Lifelong Learning and Skills (DCELLS) in Wales
1
2
Level 2 qualifications are those that are equivalent to GCSE grades A*–C.
Assessment criteria are the discrete sub-tasks a learner completes to meet the overall task for each
assessment objective. A number of assessment objectives make up the requirements of a unit.
3
Learners can achieve a pass, merit or distinction in OCR Nationals.
Ofqual 2010
2
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
and the Council for the Curriculum, Examinations and Assessment (CCEA) in
Northern Ireland) were able to accredit the revised qualifications for first teaching
from September 2010.
OCR made significant changes to the revised Level 2 ICT suite of qualifications in
order to gain this accreditation – revising the content of some of the units and
reducing the guided learning hour (GLH) values. The new qualifications offered for
first teaching from September 2010 are therefore substantially different from the
qualifications on which this report is based.
OCR has completed the actions in the agreed action plan. However, OCR submitted
other subjects in the Nationals suite of qualifications at the same time as the Level 2
ICT qualifications. We will continue to monitor and review other OCR Nationals
qualifications to ensure that the issues addressed in the Level 2 ICT qualifications
have been addressed, as appropriate, in any other qualifications offered by OCR.
Ofqual 2010
3
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Introduction
The Office of Qualifications and Examinations Regulation (Ofqual) is the independent
regulator of qualifications, examinations and assessments in England and of
vocational qualifications in Northern Ireland. Our work will ensure that learners get
the results their work merits and that the qualifications they receive count, both now
and in the future.
The OCR Level 2 Nationals in ICT suite of qualifications operates within a regulatory
framework, which is set out in The Statutory Regulation of External Qualifications in
England, Wales and Northern Ireland (QCA 2004).
Ofqual carries out a programme of monitoring activities each year to assess the
performance of awarding organisations against the regulatory criteria. One of these
activities is scrutiny, which is an in-depth study of the assessment process.
The scrutiny of a qualification aims to:

determine whether the required qualification criteria and regulatory
requirements have been met

determine whether the assessments were fair and effective in measuring
achievement by learners in respect of the stated assessment objectives and
learning outcomes

determine whether the procedures designed to ensure consistency of practice
and comparability of standards are implemented effectively

identify any aspects of the specification that appear to have constrained fair,
effective and reliable assessment

identify good practice that is worthy of encouragement and dissemination to
promote continuing improvement in the quality of the qualification.
Each scrutiny involves a team of independent specialists, who are experts in their
subject. The team observes meetings held by the awarding organisation, analyses
assessment instruments and associated mark schemes, and reviews a sample of
learners’ work.
Following each scrutiny, Ofqual reports to the awarding organisation indicating how,
if necessary, the awarding organisation should improve its provision. The awarding
organisation produces an action plan in response, which details how and when any
issues will be addressed.
Ofqual 2010
4
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Ofqual reported findings from the scrutiny of Level 2 Nationals in ICT to OCR in July
2009.
Our findings were based on evidence from the observation of different aspects of the
assessment process, consideration of assessment materials (qualification
specification, unit specifications, marking criteria and guidance materials issued by
the awarding organisation) and learners’ work. Details of the meetings observed are
provided in Appendix A.
In September 2009, OCR provided an action plan in response to these findings. As
part of their action plan, OCR supplied Ofqual with revised unit specifications and
mark grids, plus draft assignments that demonstrated the steps OCR had taken to
address the issues highlighted in the findings report. These steps are evaluated in
this report under the sections entitled ‘Action taken’.
Ofqual 2010
5
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Monitoring of the OCR Level 2 Nationals in ICT
The OCR Level 2 Nationals in ICT Centre Handbook defines the OCR National
qualifications as a:
‘suite of qualifications [that] provides learners with high quality, industry-relevant
qualifications geared to the specific requirements of key sectors. They are
vocationally-related qualifications that provide valuable opportunities for
individuals to develop skills and gain underpinning knowledge and understanding
which will support entry into work or progression to further studies through Further
Education or Higher Education. The OCR Nationals are attractive, practicallybased qualifications intended to stimulate and interest learners. They support
achievement of Key Skills and relate to national occupational standards thereby
providing an ideal progression to National Vocational Qualifications (NVQs) once
individuals are in suitable employment’.
Source: OCR Level 2 Nationals in ICT Centre Handbook
Scheme of assessment
The OCR Level 2 Nationals in ICT were accredited for first teaching in September
2006. The suite is made up of four qualifications that share 23 units. These are all
internally assessed and subject to external moderation by an OCR visiting
moderator. The four qualifications that may be taken are as follows:
Qualification
4
Guided
learning
hours (GLH)
SCAAT 4
equivalences
at A* to C
grade
OCR Level 2 National First Award in ICT
90
1 GCSE
OCR Level 2 National Award in ICT
180
2 GCSEs
OCR Level 2 National First Certificate in ICT
270
3 GCSEs
OCR Level 2 National Certificate in ICT
360
4 GCSEs
Equivalency figures (SCAAT points) are calculated by the Qualifications and Curriculum
Development Agency (QCDA) and are applied to all eligible approved qualifications for use in
compiling the School and College Achievement and Attainment Tables (SCAAT) for England. They
are designed to provide an indication of the degree of successful attainment for a qualification in
relation to other qualifications within the existing national frameworks.
Ofqual 2010
6
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Unit 1 (ICT Skills for Business) is compulsory within each qualification. All other units
are optional and are completed in either 60 GLH or 30 GLH. To be awarded a
qualification, learners must complete and pass Unit 1 and additional optional units to
make up the precise number of guided learning hours required for that qualification.
Learners receive an overall grade of pass, merit or distinction for each unit
completed. When learners 'cash in' their completed units for certification, they are
allocated a number of points per unit, based on the size of each unit and the grade
received for that unit. Learners who have passed sufficient units are awarded the
qualification. The grade is based on their accumulated points score. A detailed
summary of the scheme of assessment and associated points thresholds is provided
in Appendix B.
The scrutiny considered assessment materials from units 1, 2, 4, 5, 7, and 20 to 23.
Where possible, learner work was considered at pass, merit and distinction for each
of the above units. However, the number of samples of learner work provided by the
awarding organisation, particularly at pass and merit, was limited.
Validity of assessment
Ofqual found that there was considerable variation in the amount of work and level of
skill required to complete the different units, which significantly undermined the
comparability of different routes through the qualification. We noted that the 30 GLH
(group C) units reviewed required a lower proportion of higher-order thinking skills,
such as planning and evaluation, compared to the 60 GLH (group B) units reviewed.
The combined level of demand for two group C units (equivalent to 60 GLH) was
therefore lower than that required for a single group B unit (60 GLH).
Assessment objectives 5 lacked guidance on how much learners were permitted to
make use of ‘wizards’ and templates included in software. Where these were used,
the demand of many tasks was significantly reduced.
Comparisons between Unit 1 (ICT Skills for Business), which is compulsory, and
other 60 GLH units also revealed issues of comparability in demand. We found that
the level of demand for Unit 1 was not appropriate for a qualification at Level 2
(equivalent to GCSE grades A*–C). Tasks did not provide sufficient opportunity to
stretch more-able learners, as requirements to carry out higher-order skills were
minimal – for example, there was little scope for learners to take design decisions.
Furthermore, some higher-level tasks demonstrated in learner work (such as
5
Assessment objectives are a series of tasks that, together, form the requirements for a unit, against
which learners’ work is marked.
Ofqual 2010
7
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
providing parameter queries in Microsoft Access) were not in the assessment
objectives and therefore were not credited.
Overall, Unit 1 was judged to be more prescriptive than the other 60 GLH units
reviewed. All units reviewed, with the exception of Unit 1, focussed on one final
product, whereas in Unit 1, the assessment objectives covered different tasks
required in an office environment, such as using email or searching the internet. The
fragmented nature of the assessment criteria in Unit 1 meant that although the level
of demand of this unit was too low, the unit was more challenging in terms of the
organisation required by both learners and assessors in order to demonstrate the
required evidence.
Inconsistencies in the assessment criteria 6 were also noted – for example, in
assessment objective 5 of Unit 1, where the marking criteria for the task to create an
effective business spreadsheet were more challenging at merit than at distinction.
The criteria for merit required learners to create an effective business spreadsheet,
displaying accurate figures, and containing a title and appropriate row/column
headings, whereas the criteria for distinction only required learners to create an
effective business spreadsheet displaying accurate figures. Pass criteria for this unit
were judged insufficiently demanding for Level 2 and learners of only moderate ability
were able to achieve a distinction.
Unit 4 (Design and Produce Multimedia Products) contains five assessment
objectives that take learners through the design, production and review of a
multimedia product. The nature of the product was not clearly defined, although
OCR’s guidance material did suggest a number of software tools that could be used
to create the product including Microsoft PowerPoint, as well as specialised
multimedia software. Although the task involved compiling media taken from other
sources into a product rather than creating it from scratch, designing and assembling
a multimedia product using specialised software could give all learners the
opportunity to demonstrate the extent of their abilities. However, learners could
complete this unit by producing a simple PowerPoint presentation. As a result, the
level of demand of the unit would not be comparable across the different software
options available. The assessment objectives for this unit also include a ‘review’
stage that is significantly less rigorous than the ‘evaluation’ required for some other
units.
6
Assessment criteria are the discrete sub-tasks a learner completes to meet the overall task for each
assessment objective.
Ofqual 2010
8
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Action taken
In April 2010, OCR submitted draft units and model assignments for the Level 2
Nationals in ICT qualifications on the National Qualifications Framework (NQF) as
part of the accreditation for the revised qualifications. Ofqual found that the units and
assessments had gone some way to provide opportunities for all learners to
demonstrate their abilities. However, the changes had not gone far enough to ensure
that the qualifications had options that are comparable in terms of the nature and
volume of work required and the level of intellectual demand. In response, OCR
reduced the number of guided learning hours on all units in this suite of qualifications
to bring the requirements into line with similar qualifications at Level 2. OCR
resubmitted the amended units in April 2010. As a result of the improvements to units
and model assignments, and the reduction in the GLH of individual units to 20 and 40
GLH, the revised suite of qualifications was accredited for first teaching from
September 2010.
Assessment process
All units in the Level 2 Nationals in ICT suite are internally assessed. Learners
compile a portfolio of evidence that demonstrates their achievement of the
assessment objectives. Portfolios are graded by centre assessors. Centres' grades
are confirmed via a system of moderation visits made by OCR moderators.
Moderation visits serve two purposes: firstly, they verify a centre's grading decisions;
secondly they offer advice to the centre about how to improve assessment and
learner performance.
Internal assessment
In order to meet an assessment objective within a unit, learners need to satisfy all of
the pass criteria. If all pass criteria are achieved, the learner’s work is then graded at
pass, merit or distinction according to the internal assessor’s judgement of how much
of the pass, merit and distinction criteria the learner has achieved.
No guidance was provided on the weighting to be applied to each criterion and to
each of the assessment objectives. This meant that assessors were left to make
subjective judgements as to which elements of the task were most important and
which grade constituted the ‘best-fit’ to the learner’s achievement.
We judged this ‘best-fit’ grading method insufficient to ensure consistency in
standards between assessors. In Unit 1, we found that the assessment criteria were
not sufficiently clear to ensure consistent application and that the ‘best-fit’ approach
Ofqual 2010
9
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
was interpreted in different ways. For example, one learner awarded merit had
narrowly failed to meet all the distinction criteria (missing by just one half point) yet
another learner was awarded a distinction having only met a few of the criteria.
Examples of inconsistencies in marking were also noted in other units.
There were also concerns regarding the way in which differentiation between
different grades across many of the units was achieved. Differentiation between
pass, merit and distinction grades was, in some cases, achieved by increasing the
range and sophistication of functions that must be used while producing the product
or products. This was an effective and appropriate method of differentiation.
However, in other cases, differentiation was achieved by merely increasing the
number or size of the outputs to be produced. This was not an appropriate method of
differentiation.
Action taken
In March 2010, OCR provided Ofqual with a comprehensive range of materials from
their revised centre INSET training programme and added additional materials to
their website, such as frequently-asked questions and exemplars.
In April 2010, OCR submitted improved mark grids as part of the NQF accreditation
process and subsequently improved their Centre Handbook for these qualifications.
The revised Centre Handbook includes a more comprehensive section on the
assessment and grading of evidence and provides clearer statements about what
has to be completed (for example, that all requirements of pass grade descriptions
must be met at all levels) and how the ‘best-fit’ approach to merit and distinction
grading operates and how it should be implemented.
The combination of these improvements is likely to help assessors meet their
responsibilities by providing them with more information and guidance on how to
ensure assessment requirements can be interpreted and applied consistently.
The impact of these changes will not be apparent immediately and we will review
them through the future monitoring of these and other OCR National qualifications.
Moderation
OCR’s document Instructions for Visiting Moderators for OCR Nationals 20082009 describes the sampling principle for Nationals qualifications as: "every
unit, every assessor, the full range of grades available". Before visiting a centre,
the moderator selects a sample of five portfolios for each unit based on the list
of learners provided by the centre. This sample for each unit must include
portfolios marked by each assessor who has undertaken marking for that unit.
Ofqual 2010
10
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Where more than five assessors have worked on a unit, the sample must be
expanded.
After the sample is selected, the guidance instructs moderators to select a smaller
number of portfolios to review. The guidance suggests limiting the number of
portfolios to review to five or six, depending on the number of units being moderated,
though the sample must take in the work of each assessor within each unit they have
assessed.
The sampling guidance only instructs moderators to review portfolios for all units and
for all assessors. Sampling requirements do not always require moderators to select
sample sizes related to the learner entry at each centre and as such the rules were
seen to be too restrictive. Ofqual was concerned that the quantity of work considered
at some visits was too small, considering the size of the entry.
Action taken
OCR has reviewed and revised the training and standardisation materials for training
moderators. The information now provided to moderators is more comprehensive
and includes an improved section on sampling. OCR has also revised the sampling
arrangements for moderators to ensure the widest possible coverage of different
grades in the sample, without compromising the initial focus on covering all units and
all assessors.
The impact of these changes will not be apparent immediately and we will review
them through the future monitoring of these and other OCR National qualifications.
Monitoring of other OCR Nationals qualifications
As a result of this scrutiny, Ofqual required OCR to take into account the findings
from the OCR Level 2 Nationals in ICT scrutiny to determine if areas of improvement
could also be applied to other OCR qualifications. Ofqual is working with OCR on
revisions to other Nationals qualifications for first teaching in September 2011 and
September 2012.
Ofqual 2010
11
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Appendix A: List of observed OCR meetings
Meeting title
Location
Date
Standardisation of Moderators
Nottingham
7th November 2008
New Moderator Training
Coventry
5th December 2008
Centre moderation visits
3rd April 2008, 4th April 2008, 21st April 2008, 2nd May 2008, 21st May 2008, 9th
June 2008.
Centre Assessor Guidance and Training (CAGT)
Title
Location
Date
Get Started: Successful First Delivery
London
28th April 2008
Get Ahead: An In-depth Look at Unit 1 London
(ICT Skills for Business)
16th May 2008
Get Ahead: Enhancing Delivery for
Experienced Practitioners
Plymouth
22nd May 2008
OCR ICT Nationals Conference
Birmingham
11th June 2008
Title
Location
Date
Get Ahead: Effective Assignment
Writing
Exeter
5th March 2008
Generic OCR Nationals CAGT events
Get Started: Portfolio Building and
London
Internal Moderation for OCR Nationals
Ofqual 2010
13th May 2008
12
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Appendix B: Schemes of assessment
Qualifications
Four qualifications are available in the OCR Nationals Level 2 in ICT qualification
suite:
Qualification
Guided
learning
hours (GLH)
SCATT 7
equivalences
at A* to C
OCR Level 2 National First Award in ICT
90
1 GCSE
OCR Level 2 National Award in ICT
180
2 GCSEs
OCR Level 2 National First Certificate in ICT
270
3 GCSEs
OCR Level 2 National Certificate in ICT
360
4 GCSEs
Learners receive an overall grade of pass, merit or distinction for each unit
completed. No numeric grades are awarded. When learners 'cash in' their completed
units for certification, they are awarded a number of points based on the size of each
unit and the grade received, as follows:
Unit size
Pass
Merit
Distinction
30 GLH
0.5 points
1 point
1.5 points
60 GLH
1 point
2 points
3 points
7
See footnote 3
Ofqual 2010
13
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Learners who have passed sufficient units are awarded the qualification with a grade
based on their accumulated points score, as follows:
Qualification
GLH
Pass
Merit
Distinction
First Award
90
1.5 points
2.5 points
3.5 points
Award
180
3 points
4.5 points
7 points
First Certificate
270
4.5 points
6.5 points
10.5 points
Certificate
360
6 points
9.5 points
14 points
Units
Learners must all take Unit 1 and then each select a combination of 60 GLH and
30 GLH units from the optional units to make up the exact number of guided learning
hours required for each qualification. (An asterisk indicates the units sampled as part
of the monitoring.)
Group A – mandatory unit
GLH
Unit 1*
60
ICT Skills for Business
Group B – optional units
GLH
Unit 2*
Webpage Creation
60
Unit 3
Digital Imaging – Plan and Produce Computer Graphics
60
Unit 4*
Design and Produce Multimedia Products
60
Unit 5*
Desktop Publishing
60
Unit 6
Spreadsheets – Design and Use
60
Unit 7*
Databases – Design and Use
60
Unit 8
Technological Innovation and E-commerce
60
Unit 9
Customer Support
60
Unit 10
Planning and Supporting Telecommunications
60
Unit 11
Career Planning for IT
60
Unit 12
Work Experience in IT
60
Unit 13
IT Systems and User Needs
60
Unit 14
Installation of Applications
60
Ofqual 2010
14
OCR Level 2 Nationals in ICT
Findings and Actions from the Monitoring of the Qualification
Group B – optional units continued
GLH
Unit 15
System Testing and Maintenance
60
Unit 16
Repair and Decommission of IT Equipment
60
Unit 17
CAD and CAM
60
Unit 18
Application of Computer Control
60
Group C – optional units
GLH
Unit 19
Application of Data Logging
30
Unit 20*
Creating Animation for the WWW using ICT
30
Unit 21*
Creating Computer Graphics
30
Unit 22*
Creating Sound Using ICT
30
Unit 23*
Creating Video
30
Ofqual 2010
15
Ofqual wishes to make its publications widely accessible. Please contact us if
you have any specific accessibility requirements.
First published by the Office of Qualifications and Examinations Regulation in 2010
© Crown copyright 2010
Office of Qualifications and Examinations Regulation
Spring Place
2nd Floor
Coventry Business Park
Glendinning House
Herald Avenue
6 Murray Street
Coventry CV5 6UB
Belfast BT1 6DN
Telephone 0300 303 3344
Textphone 0300 303 3345
Helpline 0300 303 3346
www.ofqual.gov.uk