A690966
Page 1 of 44
20
Indicators for literacy and numeracy
provision and gain
Version 0.4 - June 2012
Version control
Version
Date
Change
Draft 01
September 2011
Draft 02
October 2011
Includes elements on need for indicators.
Draft 03
26 October 2011
Revise elements on need for indicators after discussion with Bryce
Draft 04
December 2011
Added literature review section
Draft 05
February 2012
Edited for readability and sense
Draft 06
February 2012
Changes accepted, revisions for conciseness and flow.
Draft 07
February 2012
Reid Hodges feedback on Jason Dowse changes.
Consultation with ITPs
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 2 of 44
Contents
Executive Summary ........................................................................................................ 3
Introduction ..................................................................................................................... 4
Feedback Sought ............................................................................................................ 5
Measuring LN Performance ............................................................................................ 6
Role of an LN Gain Indicator .................................................................................... 6
Relationship to Funds .............................................................................................. 6
Indicator development principles .............................................................................. 7
The approach taken to develop an LN Performance Indicator .................................. 7
Programmes and sequences.................................................................................... 8
Proposed Indicator #1: LN Gain Indicator ................................................................... 10
Proposed Indicator #2: LN Percent Assessed Indicator............................................. 12
Proposed Indicator #3: LN Provision Indicator ........................................................... 13
Summary........................................................................................................................ 13
Appendix A: Definitions and Business Rules SAC ..................................................... 15
Definition of Sequences ......................................................................................... 15
Joining assessments to sequences/programmes ................................................... 17
Calculating gain...................................................................................................... 21
Proposed Indicator #1: LN Gain Indicator ............................................................... 24
Proposed Indicator #2: LN Percent Assessed Indicator .......................................... 29
Proposed Indicator #3: LN Provision Indicator ........................................................ 31
Dimensions sourced from the SDR and course qualification register...................... 35
Dimensions sourced from the LNAAT .................................................................... 35
Appendix B: Definitions and Business Rules FFTO and YG ...................................... 37
Appendix C: Definitions and Business Rules ILN/WPL .............................................. 37
Appendix D: Definitions and Business Rules ITR....................................................... 37
Appendix E: Abbreviations........................................................................................... 37
Appendix F: LNAAT and assessments ........................................................................ 39
Appendix G: Research on Performance Indicators .................................................... 41
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 3 of 44
Executive Summary
1
As a result of additional investment by Government the number of
programmes that include literacy and numeracy has increased. However,
evidence about literacy and numeracy learning gain has remained limited.
2
The Tertiary Education Commission (TEC) intends to get a better
understanding of the levels of provision of literacy and numeracy education as
well as the outcomes being achieved by learners engaged in this education. It
intends to use three indicators to achieve this and invites your feedback on
these and the associated business rules. The proposed indicators are:
a
A measure of the difference between initial and progress scores from the
Literacy and Numeracy for Adults Assessment Tool (LNAAT or the
Assessment Tool) where that is statistically significant. This will be known
as the Literacy and Numeracy (LN) Gain Indicator;
b
An indicator of the percentage of learners in literacy and numeracy
education who receive literacy and numeracy assessments; and
c
A measure of the proportion of level 1-3 LN SAC/ITF funded LN related
provision in courses with Embedded Literacy and Numeracy as a
percentage of all level 1-3 provision each provider delivers.
3
This paper includes the draft definitions for the proposed indicators and
discusses how they will be calculated and other options for a gain indicator
that were considered but not taken forward. The proposed indicators are
applicable for Tertiary Education Organisations (TEOs) receiving funding for
literacy and numeracy courses. The paper describes the general approach
and the appendices describe the specific definitions and business rules for
individual funding sources.
4
The TEC is developing the indicators as part of a larger programme of work.
That programme includes the publishing of performance information,
consideration of how performance information is to be used in investment
processes and the supply of information for TEOs‟ benchmarking purposes.
The indicators will inform discussions between the TEC and TEOs and the
TEC‟s investment process. The indicators will also be useful for TEOs that
wish to obtain a better understanding of their performance and how it may
change over time.
5
The TEC is taking a phased approach to the development and use of the
indicators. After receiving your feedback the TEC will make final decisions
regarding the indicators. The next phase will include further work on data
quality and understanding the information produced by the indicators and
subject to that process, expects them to be introduced for the 2013 academic
year.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 4 of 44
Part A
Introduction
6
The vision for a more literate New Zealand was laid out in the 2001 New
Zealand Adult Literacy Strategy More than Words. It identified literacy as an
enabler for New Zealanders‟ full participation in all aspects of life. The 2006
international Adult Literacy and Life Skills (ALL) survey found that 1.1 million
New Zealanders (43 per cent of adults aged 16–65 at that time) had literacy
and numeracy skills below those needed to participate fully in a knowledge
society. It showed Māori adults have lower literacy and numeracy skills than
the general population, and that Pasifika adults are also over-represented1.
7
To address these issues, Budget 2008 provided additional funding for new
literacy and numeracy learning opportunities. The TEC has published the
Action Plan for Literacy, Language and Numeracy. That document outlined a
systematic approach to building tertiary education sector literacy and
numeracy capability.
8
As a result of the additional investment the number of programmes that
include literacy and numeracy has increased. However, evidence regarding
literacy and numeracy learning has remained limited.
9
The TEC has been working on resolving this and had the benefit of feedback
on an earlier draft of this paper from a consultative group of five Polytechnics
and Institutes of Technology.
10
The TEC proposes to develop and use a LN Gain Indicator measuring how
well individual TEOs are able to lift literacy and numeracy levels. It will use as
its reference base LNAAT assessment scores of learners near the start and
near the finish of programmes of learning where the difference (gain) is
statistically significant. This document discusses the rationale for a LN Gain
Indicator and the proposed approach to calculating it. The TEC is also
proposing to introduce:
11
1
a
An indicator of the percentage of learners engaged in literacy and
numeracy education who receive literacy and numeracy assessments;
and
b
A measure of the proportion of level 1-3 LN related provision in courses
with Embedded Literacy and Numeracy as a percentage of all level 1-3
provision each provider delivers.
Details on specific definitions and rules are documented in Appendix A for
SAC funded provision. Information on definition and business rules for other
funding sources will be released subsequently for separate feedback.
However, these will follow the general approach used here so TEOs and ITOs
are encouraged to provide feedback at this point. Abbreviations used in this
For further information, see TEC, Getting Results in Literacy and Numeracy, September 2010.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 5 of 44
paper are listed in the appendices along with further information on the
LNAAT and research used to inform this work.
Feedback Sought
12
The TEC is seeking feedback on the proposed indicators, especially
regarding:
•
Opportunities to improve their definition;
•
Areas requiring further clarity from the TEC, for example in the proposed
business rules;
•
TEO practices that may affect data quality or lead to potential unintended
consequences from the use of the indicators; and
•
Several specific questions which are included in the Summary section,
paragraph 62.
13
A summary of the responses and further information on the process for
finalising the indicators will be released after feedback has been received.
14
Please contact the TEC Service Centre at servicecentre@tec.govt.nz by
Friday 22 June 2012 with your feedback on this document and include the
words “LN definitions and rules feedback” in the subject when sending in
email feedback on the document.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 6 of 44
Measuring LN Performance
Role of an LN Gain Indicator
15
Since 2008, Government has invested in building the LN infrastructure and
funded TEOs to provide embedded and intensive LN to provide opportunities
for literacy and numeracy learning. TEOs with learners participating in
programmes which include embedded LN are required to use the LNAAT to
measure learner literacy and numeracy levels near the beginning and near
the end of each programme.
16
The TEC is developing the LN Gain Indicator as part of a larger programme of
work. That programme includes the publishing of performance information,
consideration of how performance information is to be used in investment
processes and the supply of information for TEOs‟ benchmarking processes.
17
LN education, for the purposes of this document, means activities tertiary
education organisations (TEOs) undertake that contribute specifically to the
enhancement of learner literacy and numeracy.
18
Learner capability captured by the LNAAT provides information on the extent
of TEO assessment of learners as they enter programmes and their gain in
literacy and numeracy over time.
19
In combination with other data sources, the LNAAT will allow TEO student
assessment to be monitored against the number of learners funded for LN
provision, the extent of gain achieved by individual learners and by
programme, TEO, sub-sector, sector and other key dimensions such as Māori
and Pasifika. Māori and Pasifika people present with greater adult literacy and
numeracy needs, but are not over-represented in literacy and numeracy
provision. It appears that literacy and numeracy levels may be a key
impediment to Māori and Pasifika achieving at higher levels of education.
20
The TEC is taking a phased approach to the development and use of the LN
Gain Indicator. After receiving your feedback the TEC will make final
decisions regarding the LN Gain Indicator. The next phase will include further
work on data quality and understanding the information produced by the LN
Gain Indicator and subject to that process, expects it will be introduced for the
2013 academic year.
Relationship to Funds
21
The development work on the indicators has been focused on investment via
the Student Achievement Component Fund (SAC) but the same broad
approach is proposed to LN provision funded through other funding
mechanisms. At this stage the TEC proposes to produce the indicators for
each of the funded areas but will explore whether aggregate indicators can be
developed to provide a cross-fund view of LN performance and provision.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 7 of 44
Indicator development principles
22
Principles associated with the LNAAT were an overriding consideration of our
work to-date, especially the principle that compliance needs to be minimised
along with any interference with the professional teaching practices of
educators. The TEC also approached this work with the intention of avoiding
any additional assessment of learners.
23
In addition to those principles, the TEC has tried to ensure the indicators:
Use existing information where possible to minimise compliance and
administration costs;
Take the least complex approach possible so TEOs can replicate the
calculations;
Are valid;
Are useable regardless of funding source;
Are fair (e.g., use measures of improvement taking the learner‟s beginning
capability into consideration, rather than pure measures of outcome);
Allow comparisons between TEOs; and
Allow tracking of Māori and Pasifika attainment in literacy and numeracy.
24
The TEC has worked to these principles as far as possible but acknowledges
that the proposals outlined in this document may have impact on providers
that currently do not administer LN assessments to every LN learner as
required. In addition, some complexity has been unavoidable because of the
need to define what a “programme” is with respect to LN learning where that
concept is not already in use. This is discussed further in the section that
deals with “Programmes and Sequences”. The TEC will mitigate the impact of
complexity via clear definitions and business rules, by working with providers
to assist them in understanding and replicating the proposed indicators and
being transparent in its approach and intentions.
The approach taken to develop an LN Performance Indicator
25
26
2
The TEC has reviewed published information to identify2 how other entities
measure performance of literacy / numeracy programmes. That work
suggests there are two key approaches to consider:
a
Measurement of attainment against a standard/benchmark/threshold; or
b
Measurement of change in scores.
The TEC is not proposing to take a benchmark/threshold based approach at
this stage because of the complexities in developing a threshold that would be
The TEC thanks the Ministry of Education Library for conducting the literature search, during
August and September 2011. This search was not meant to be comprehensive. It was
targeted at the use of assessment measures as the basis for an indicator of a performance,
usually of a literacy/numeracy programme or an institution delivering the programme.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 8 of 44
fair to all providers given their different size, characteristics of provision and
learner characteristics. Accordingly, it intends to pursue a gain based
indicator which will measure the capability gain by learners as measured by
an initial and progress assessment. Further information on research of the
different approaches to measuring performance is included in Appendix G.
Programmes and sequences
27
The TEC requires that LN learners it funds are assessed near the beginning
and near the end of programmes of learning. Where the concept of LN
programmes do not exist in the data (for instance with SAC programmes), the
TEC proposes to approximate them. This involves grouping LN courses that
occur within 90 days of each other into “sequences”.3
28
The TEC proposes to adopt the sequence concept for the following funding
sources:
29
a
SAC;
b
Foundation Focused Training Opportunities (FFTO); and
c
Youth Guarantee (YG).
Funds with an existing programme concept where the indicators will be
applied (with suitable adaptation to the detail) include:
a
Work Place Literacy (WPL);
b
Work Place Literacy employee targeted;
c
Intensive Literacy and Numeracy(ILN);
d
Intensive Literacy and Numeracy Targeted English for Speakers of other
Languages;
e
Industry Training Fund;
f
Modern apprenticeships; and
g
Trade Academies4.
30
LN courses that occur further than 90 days apart are assumed to be unrelated
to each other. This means that TEOs can plan to assess learners over the
course of the sequence or programme rather than for every course. This is
an approach which we would like specific comment on.
31
Initial and progress assessments are expected in the first and second halves
of sequences/programmes respectively. These are taken as the earliest and
3
The TEC would also like TEOs to provide via their feedback, information on how
qualifications relate to programmes- is there usually only a single qualification with LN
content per qualification?
4
Trade academies are provided by a number of different types of TEOs as well as ITOs.
Reporting their LN related performance will be by the relevant existing reporting
mechanisms, eg via the Single Data Return for SAC funded providers.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 9 of 44
the highest scoring assessment in the respective halves. The earliest
assessments are used because they show the learners‟ initial competence.
The second assessment point could be either the most recent assessment or
the highest assessment after the mid-point. The highest scores are used
because of the risk that reduced learner motivation may lead to lower
assessment
scores
for
some
learners
near
the
end
of
sequences/programmes. The highest score in the second half of the
programme is more likely to reflect a learner‟s actual competence.
32
Because learners can be assessed at more than one TEO, the TEC looks for
assessments at the TEO of enrolment preferentially. If none are found,
assessments for the same learner at another TEO can be substituted.
33
The difference between initial and progress assessment scores in a particular
assessment area indicates gain achieved. The tool reports gain within a
range of values in reflection of the uncertainty inherent in learner
assessment. Statistically significant positive gain indicates that there can be
confidence that a difference in scores reflects real improvement in a learner‟s
competence.
34
The data also shows the regression rather than improvement of scores, either
within or between sequences. The current approach is to treat any gain
within a sequence/programme that is not positive and statistically significant
as zero gain. No improvement is considered to have been achieved. Data on
the frequency with which this occurs will be examined further.
35
Where a sequence is complete within a single year, the gain indicator will
measure incremental gain. Where a sequence spans multiple years, then the
sequence will measure the cumulative gain across those years.
36
The issues of incremental versus cumulative gain do not arise for noncontinuing sequences/programmes because they have unique initial and
progress assessments.
37
There are a number of questions related to sequences which we would like
you to consider. These are outlined in the Summary section, paragraph 62.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 10 of 44
Part B
38
Please note that the analysis documented below was undertaken on data for
the 2011 calendar year from Single Data Returns (SDR) and was limited to
SAC funded results for numeracy assessments.
39
The literacy and numeracy information development project is on-going.
While all reasonable care has been taken in preparing the data that informs
this paper, all data quoted or cited is yet to be finalised and is therefore
subject to change. This is due to the developmental nature of the work
involved.
40
The TEC will be in a position to supply finalised information in due course that
reflects all proposed business rules.
41
The following is a brief discussion of the proposed indicators. Detailed
definition of these indicators can be found in Appendix A.
Proposed Indicator #1: LN Gain Indicator
42
The TEC is proposing to develop and use as a gain indicator a measure that
is based on the LNAAT score changes and the count of learners involved in
LN. The numerator is the count of statistically significant positive gain results
and the denominator is the count of all LN gain results for that Provider, year
and assessment area (as measured between two assessments). Gain will be
measured between initial and progress assessments in programmes or
sequences.
43
Our analysis revealed that the proposed indicator produces a separation in
the observed performance of providers. However, it was also observed that
some providers had such low numbers that the indicator would have a high
level of volatility year on year. Therefore, if this indicator is adopted
consideration could be given to only applying it to providers with more than a
minimum number of learners. Alternatively, the indicator could be calculated
but the TEC and providers would interpret it in the knowledge that for low
numbers of learners, it might not be informative. A further consideration is the
privacy of individuals, which could be compromised when reporting the
performance for providers with low numbers of learners. TEOs may wish to
consider this further and whether some information should be restricted to
TEOs only, rather than being publicly reported- for example for providers with
fewer than 30 learners.
44
The TEC also considered a number of other candidates for the LN Gain
Indicator including:
a
The proportion of learners attaining the next higher step;
b
The proportion of learner gain greater than a threshold; and
c
The proportion of EFTS in LN Courses achieving statistically significant
gain.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
5
Page 11 of 44
45
The first option was eliminated as a candidate because step change may or
may not be associated with statistically significant gain and would not
therefore provide an accurate indication of the true level of gain. The second
option was eliminated because of the difficulties in setting a threshold gain
level, although the concept of what is “reasonable gain” may be considered
further.
46
The EFTS based indicator was analysed in more depth and compared to the
count based gain indicator using the SAC EFTS delivered for a learner by
provider for LN courses and between sequence start and finish dates. The
first question to be considered was whether the EFTS based or count based
indicators produced markedly different results. The results for providers using
the two different indicators were plotted, as shown below5.
47
This analysis shows that the results of the two indicators are highly correlated
and means that either indicator could be used as they will produce very
similar results. The next dimensions to be considered related to ease of
calculation, reproducibility by TEOs and simplicity. Both indicators are
relatively easy to produce (although there are complexities with the sequence
concept) and TEOs will have access to the key information to produce either
indicator including:
a
NSN;
b
Assessment area;
Using 2011 data for numeracy assessments, gain is cumulative.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 12 of 44
c
LNAAT scores;
d
Funding source;
e
Volume delivered with the unit of measure reflecting the funding source.
For example for the SAC fund the unit will be EFTS, for the ITF it will be
Standard Training Measures; and
f
Ethnicity (eg. Māori, Pasifika, Other).
48
In addition, TEOs will be able to derive the sequence start and finish dates
and derive the level of gain. TEOs with provision that is structured into
programmes, will be able to determine programme start, mid and end-points.
TEOs will also be able to determine using standard tools, whether the level of
gain for an individual learner is statistically significant. On balance, the TEC
concluded that the count based indicator is simpler and more meaningful to
explain as it is built on learners rather than volume of delivery.
49
The proposed indicator also lends itself to additional analysis, such as
analysing the level of gain by learners associated with their initial LN step and
progression and whether Māori and Pasifika are attaining literacy skills at least
on a par with other literacy and numeracy learners and links to further
pathways of learning.
Aggregate or by assessment area
50
An aggregate gain indicator could be weighted to reflect the numbers of gain
results in each assessment area or could be un-weighted and simply be the
average of the indicators for each of the assessment areas. A weighted
indicator would provide a more accurate picture of provider performance than
an un-weighted indicator. However even a weighted indicator may be of
limited use because the TEC expects that it and providers will want to
understand LN performance for each assessment area. In addition, because
LN provision can be via multiple finding sources this adds another layer of
complexity. Accordingly, the TEC proposes that initially it generate a count
based LN gain indicator for each of the assessment areas and for each
funding source. TEOs may wish to comment on whether an aggregate
indicator could be developed and if so, how.
51
The TEC is not proposing to benchmark providers at this stage. However, the
TEC is planning further work on the LN Gain Indicator and will consider
whether we need to use standardisation to account for factors influencing gain
and will discuss this with the sector as it progresses.
Proposed
Indicator
52
Indicator
#2:
LN
Percent
Assessed
Measuring gain requires learners to have both an initial and a progress
assessment. If a learner does not undertake at least one initial and one
progress assessment per assessment area then their gain cannot be
measured or the contribution of a provider identified with respect to those
learners. The TEC is proposing an indicator in addition to a gain indicator
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 13 of 44
which will measure the percentage of learners who have received two
assessments in the course of a sequence/programme- an initial assessment
and a progress assessment.
53
The indicator is calculated as the number of learner enrolments with the
required number of initial and progress assessments for each assessment
area, divided by the number of learner enrolments in sequences or
programmes for the reporting year of interest.
54
This indicator would be calculated on an aggregate basis for each provider for
each year.
55
Some learners present below level 1 of the learning progressions and are not
assessed using the LNAAT and instead assessed using “starting points”.
However their enrolments are included in the percent assessed indicator. The
TEC would like your comments on possible options to address this issue.
These are outlined in the Summary section, paragraph 62.
56
If the LN Percent Assessed indicator is adopted it will be used alongside the
LN Gain Indicator to inform providers and the TEC regarding the level of
learner participation, achievement and TEO performance.
Proposed Indicator #3: LN Provision Indicator
57
The TEC wants to improve its understanding of the extent of LN delivery. To
do this it intends to use an indicator that measures the proportion of level 1-3
SAC funded and industry training funded provision with embedded literacy
and numeracy as a percentage of all level 1-3 provision each provider
delivers. It will use the unit of measure appropriate to the funding source, eg
EFTS for SAC funded provision and STMs for industry training provision. It
may also use an aggregate indicator that reports on provision regardless of
funding source. The TEC will use this information to inform policy work and in
the course of the investment process.
Summary
58
Additional investment by Government has increased the number of
programmes that include literacy and numeracy and the TEC wants to obtain
a better understanding of this provision and its outcomes.
59
It has examined a number of options and intends to use three indicators to
achieve this:
a
A measure of the difference between initial and progress scores from the
Assessment Tool. This will be known as the Literacy and Numeracy (LN)
Gain Indicator;
b
An indicator of the percentage of learners engaged in literacy and
numeracy education who receive literacy and numeracy assessments;
and
c
A measure of the proportion of level 1-3 SAC/ITF funded LN related
provision as a percentage of all level 1-3 provision each provider delivers.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 14 of 44
60
The TEC is developing the indicators as part of a larger programme of work.
That programme includes the publishing of performance information,
consideration of how performance information is to be used in investment
processes and the supply of information for TEOs‟ own benchmarking
processes.
61
The TEC is seeking feedback on the proposed indicators by 22 June 2012,
especially regarding:
62
63
•
Opportunities to improve their definition;
•
Areas requiring further clarity from the TEC, for example in the proposed
business rules; and
•
TEO practices that may affect data quality or lead to potential unintended
consequences from the use of the indicators.
The TEC is also seeking your feedback on the following specific design
questions:
a
Should sequences and programmes that are less than 90 days in duration
be excluded from the gain calculations?
b
The current proposal is that the initial assessment can occur up to 90
days before a sequence starts, is this „look-back‟ period long enough and
if not, how far back should we extend this to?
c
Some learners present below level 1 of the learning progressions and are
not assessed using the LNAAT but are instead assessed using “starting
points”. However their enrolments are included in the percent assessed
indicator. Two possible approaches to addressing this discrepancy
include changing the assessment tool to enable reporting of learners who
are on the starting point or, all starting point learners could receive the
vocabulary assessment and the TEC could use this as a proxy for
identifying those learners and removing them from the dataset. The TEC
would like TEOs to provide their feedback on these, or other, options.
d
The TEC would also like TEOs to provide via their feedback, information
on how qualifications relate to programmes - is there usually only a single
qualification with LN content per programme?
The TEC is taking a phased approach to the development and use of the
indicators. After receiving your feedback the TEC will make final decisions
regarding the indicators. The next phase will include further work on data
quality and understanding the information produced by the indicators and
subject to that process, expects them to be introduced for the 2013 academic
year.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 15 of 44
Appendix A: Definitions and Business Rules SAC
Definition of Sequences
What it reports
1
Sequences will be created as a proxy for programmes where
programmes do not exist as a concept. Sequences provide an
approximation of an LN programme by grouping LN courses together.
2
The purpose of the sequence is to provide a timeframe over which to seek
initial and progress assessments.
Rules for calculating the sequence
3
The following are the rules used in calculating the sequence.
4
Use the „master‟ National Student Number (NSN), if the same student has
multiple NSNs.
5
Select courses from the course enrolment file from the SDR where
a
Embedded literacy and numeracy flag = Yes
b
Course funding source code = 01
c
Course register level is 1-3
d
Exclude any Te Reo Maori medium and ESOL courses using NZSCED
codes:
i
70115
ii
70120
iii
70124
iv
70128
v
91501
vi
91502
vii
91504
viii
91506
ix
100106
6
These are the LN courses.
7
Allocate all LN Courses with 90 days or less interval between them into
sequences.
8
Allocate LN Courses to separate sequences where the interval between them
is greater than 90 days, numbered from the earliest course start date to the
latest.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 16 of 44
9
Each sequence contains course enrolments for one NSN enrolled at one TEO
accessing funding from one Course Funding Source Code.
10
Assign sequences to an „LN Reporting Year‟ based on the year of the latest
course end date in a calendar year.
11
Sequences with course end dates in multiple years are assigned to multiple
LN Reporting Years. These define continuing sequences.
12
Define the sequence start date as the earliest of any course start dates in the
sequence.
13
Define the sequence end date as the latest of any course end dates in the
sequence.
14
Define the sequence midpoint as the date halfway between the start and end
dates of the sequence.
Which data to use
Table 1: Minimum data used to produce the sequence
Period of interest
Year
Example
Reporting year
year n
2012
Year-to-date data produced
at any time in reporting year
Most recent SDR data from
year n or earlier
2012 or earlier
Full year data availability
Finalised December SDR for
each year up to year n
February 2013
Worked example
15
The following section provides a worked example of the functioning of the
sequence and the selection of assessments. The diagram below displays the
possible permutations for sequence construction.
16
The following cases cover the possible scenarios under which sequences are
constructed for a single learner at a single TEO for a single funding source:
a
Case 1: A learner enrols in 3 LN courses in a calendar year with no
gaps between them of more than 90 days (blue boxes). The earliest
and latest LN course start and end dates define the duration of the
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 17 of 44
sequence (blue line). The sequence is denoted sequence number 1 for
2010
b
Case 2: The same learner enrols in 2 LN courses the following year
these LN courses have no gaps between them of 90 days or more. The
gap between these LN courses and the LN courses in case 1 (grey
boxes) is also less than 90 days, hence those courses become part of
this sequence. This is an example of a continuing sequence. The
sequence is denoted sequence number 1 for 2011
c
Case 3: The learner enrols in another LN course that spans 2 calendar
years. There are no other LN courses within 90 days. This results in a
sequence of one course. Because the end date of the course is in
2012, the sequence is denoted sequence number 1 for 2012.
d
Case 4: The learner enrols in yet another LN course and there are no
other LN courses within 90 days. This results in another sequence of
one course. The sequence is denoted sequence number 2 for 2012.
Data quality issues impacting the indicator and implications for TEO data
submission practices
17
Assessments will be required near the beginning and near the end of
sequences.
18
Only courses with embedded literacy and numeracy flag (ELN) = “Yes” are
used. TEOs will need to ensure that this flag is correct for each course at time
of SDR submission.
19
TEOs will likewise need to ensure that register levels 1-3 are recorded
correctly for their courses at time of SDR submission.
Joining assessments to sequences/programmes
What it reports
20
Joining assessment to sequences allows learner engagement with the tool to
be reported and gain calculated.
Rules for joining assessments to sequences/IT programmes
21
The following are the rules used in joining assessments and sequences:
a
Use the sequence definition in paragraphs 3-5;
b
Use the LNAAT assessment data.
22
Define the „test date‟ as the Assessment Submitted Date unless the
Assessment Type is non-adaptive for printing, in which case „test date‟ is
Assessment Start Date.
23
To determine initial assessments for cumulative gain, join assessments to the
sequence using the following hierarchical priority, for each assessment area:
a
Using test date, join the earliest assessment by test date before the
midpoint or up to 90 days prior to the sequence start date if the
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 18 of 44
assessment was administered by the TEO of enrolment. This is the
initial assessment; if there is no such assessment:
b
Using test date, join the earliest assessment before the midpoint or up
to 90 days prior to the sequence start date if the assessment was
administered by a TEO that is not the TEO of enrolment. This is the
initial assessment; if there is no such assessment:
c
There is no initial assessment.
24
To determine initial assessments for incremental gain, use the steps outlined
in paragraph 23, however use the most recent progress assessment from any
earlier LN Reporting Year with statistically significant gain.
25
To determine progress assessment, join assessments to the sequence using
the following hierarchical priority, for each assessment area:
a
Using test date and assessment score, join the earliest occurrence of
the highest assessment score on or after the midpoint if the assessment
was administered by the TEO of enrolment. This is the progress
assessment; if there is no such assessment:
b
Using test date and assessment score, join the earliest occurrence of
the highest assessment score on or after the midpoint if the assessment
was administered by a TEO that is not the TEO of enrolment. This is
the progress assessment; if there is no such assessment:
c
There is no progress assessment.
Which data to use
Table 2 describes the availability of data used to join year to date and full-year
assessments and sequences.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 19 of 44
Table 2: Minimum data used to join assessments and sequences
Period of interest
Year
Example
Reporting year
year n
2012
Year-to-date data produced
at any time in reporting year
Most recent SDR data from
year n or earlier
2012 or earlier
Year-to-date data produced
at any time in reporting year
Most recent assessment data
from year n - 8 weeks or
6
earlier
2012 less 8 weeks
Full year data availability
Finalised December SDR for
each year up to year n
February 2013
Full year data availability
Most recent assessment data
from year n - 8 weeks or
6
earlier
March 2013
6
Due to the submission of back dated non-adaptive for print assessments, there is up to 8
weeks of lag between a learner sitting an assessment at the assessment becoming
available in the LNAAT. Hence 8 weeks must be allowed for to ensure full reporting on a
given time period.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 20 of 44
Worked example
26
For non-continuing sequences and continuing sequences without prior
statistically significant gain:
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 21 of 44
Data quality issues impacting the indicator and implications for TEO data
submission practices
27
TEOs will need to ensure that NSNs are currently entered and aligned
between the LNAAT assessments and SDR enrolments.
Calculating gain
What it reports
28
Gain reports the differences in a learners scores in a given assessment area
over time. It is calculated as the progress assessment score minus the initial
assessment score. It tells us the increase or decrease in a learners score in
an assessment area
29
A test of statistical significance of the gain score indicates the level of
confidence that the difference in score reflects real improvement in learner
competency.
30
Calculations for gain, gain score error, and the statistical significance of gain
follow in Formulae 1,2 and 3.
Formula 1: Calculation of gain
Gain = Progress Score – Initial Score7
Formula 2: Calculation of gain score error
Gain Score Error
=
Formula 3: Calculation of the statistical significance of the gain
Gain is statistically significant when gain is positive and
Gain Score Error > Gain Score x 1.6458
7
Where the sequence is a continuing sequence the most recent progress assessment data
from an earlier sequence with statistically significant gain can be substituted for the initial
assessment data.
8
This is the criterion at a 95% confidence interval.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 22 of 44
Which data to use
31
Table 3 describes the availability of data used to calculate the year to date
and full-year gain.
Table 3: Minimum data used to calculate gain
Period of interest
Year
Example
Reporting year
year n
2012
Year-to-date data produced
at any time in reporting year
Most recent SDR data from
year n or earlier
2012 or earlier
Year-to-date data produced
at any time in reporting year
Most recent assessment data
from year n - 8 weeks or
9
earlier
2012 less 8 weeks
Full year data availability
Finalised December SDR for
each year up to year n
February 2013
Full year data availability
Most recent assessment data
from year n - 8 weeks or
9
earlier
March 2013
Worked examples
9
32
The following section provides a worked example of the calculation of gain
based on the selection of associated assessments for a learner at a single
TEO with both continuing and non-continuing sequences. The diagram in the
next section displays the scenarios.
33
The following cases cover the possible scenarios for learner sequence
enrolment and assessment selection for gain:
a
A learner enrols in any sequence and is not assessed by any provider;
b
A learner enrols in any sequence and only either an initial or a progress
assessment is found at any provider;
c
A learner enrols in a sequence in year n and has an initial and progress
assessment with at least one provider;
d
The same learner enrols in a continuing sequence in year n+1 and has
a progress assessment with at least one provider; and
e
The same learner enrols in a sequence in year n+2 and has an initial
and a progress assessment with at least one provider.
Due to the submission of back dated non-adaptive for print assessments, there is up to 8
weeks of lag between a learner sitting an assessment at the assessment becoming
available in the LNAAT. Hence 8 weeks must be allowed for to ensure full reporting on a
given time period.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 23 of 44
34
Cases 1 and 2 do not permit the calculation of gain to be made and are
contrary to expectation. Cases 3-5 are summarised in the diagram below.
35
In case 3 gain is 140 – 95 = 45. For the sake of example this is a statistically
significant positive gain.
36
The case 3 sequence continues into 2011. This makes the case 4 sequence
a continuing sequence. The standard gain calculation gives 150 – 95 = 55.
This is denoted the cumulative gain for a continuing sequence since it gives
the gain accumulated since the beginning of the continuing sequence.
37
Because there is an earlier assessment with statistically significant gain in the
2010 LN Reporting Year, we are permitted to calculate the incremental gain
for the continuing sequence since that event. In this case, the gain is
calculated as 150 – 140 = 10. For the sake of example this is not a
statistically significant gain.
38
If the 2010 sequence had also continued into 2012, we could once again
substitute the 2010 progress assessment as the initial assessment for 2012
since it is our last reference point of a score that reflects statistically
significant gain.
Data quality issues impacting the indicator and implications for TEO data
submission practices
39
The rules requiring certain assessment areas for SAC influence how we
analyse performance from year to year. Table 4 describes the different yearto-year rules used to determine which learners are expected to be assessed
in which assessment area.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 24 of 44
Table 4: Assessment requirements for SAC funded ELN
Reporting Year
Assessment areas requiring Initial
and Progress assessments
Learners NOT requiring
Progress assessments
2011
Reading OR Numeracy
Level 6 for Numeracy
Level 5+ for Reading
2012
Reading OR Numeracy OR Writing
Level 6 for Numeracy
Level 5+ for Reading
Level 5+ for Writing
2013
TBC
TBC
40
For example, a learner who is assessed in 2011 at level 6 for Numeracy at
the initial assessment (or any assessment in a previous year) is exempt from
all future assessments in that assessment area.
41
These are collectively designated the „no re-test required‟ thresholds. Note
that learners reaching the no re-test threshold at their progress assessment
are reported within the reporting year but no testing will be required for the
assessment area in following years. The learner will not be reported again in
the assessment area for any year after a threshold has been met. The
learner is not reported in the assessment area within the reporting year if the
threshold is met at the initial assessment.
42
This means that learners are automatically excluded from counts in both the
numerator and denominator of any calculation that involves the exempt
assessment area. The learner will still be counted under any non-exempt
assessment area and in any years prior to the exemption taking effect.
Proposed Indicator #1: LN Gain Indicator10
What it reports
43
The LN Gain Indicator is used to monitor the extent to which students are
achieving statistically significant positive gain per assessment area. (see
Formula 4).
Formula 4: Percentage of Learners with statistically significant positive gain
Number of learner enrolments in LN sequences with statistically significant positive
gain
Number of learner enrolments in LN sequences with initial and progress
assessments
Rules for calculating LN Gain Indicator
44
10
The following rules are used for calculating % learners with statistically
significant positive gain:
This indicator does not rely on the rules relating to assessment areas requiring
assessments each year.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 25 of 44
a
Count the number of learner enrolments in sequences with statistically
significant positive gain in each assessment area. Exclude exempt
learners for the assessment area. This is the numerator.
b
Count the number of learner enrolments in sequences with both initial
and progress assessments in each assessment area. Exclude exempt
learners for the assessment area. This is the denominator.
c
Calculate the indicator for each assessment area.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 26 of 44
Worked example
45
Table 5 shows mock data sourced from the LNAAT. For the sake of the example all enrolments at the TEO are included in the table
Table 5: Mock up of source data from LNAAT (vocabulary assessment area excluded)
Gain Error
Alpha
387621030
238
61 Step 1
595
80 Step 4
357* 101 253
33 Step 1
218
33 Step 1
-35
47 632
44 Step 4 -
Beta
111213168
684
30 Step 5
497
47 Step 3
-187 56 878
73 Step 6
440
39 Step 2
-438
83 693
42 Step 4
463
38 Step 3
-230 57
Gamma
322134852
540
28 Step 4
564
39 Step 3
463
38 Step 3
-101 54
Delta
428680840
513
28 Step 2 -
-
-
-
Epsilon
384299958
756
39 Step 6 -
-
-
-
Zeta
175058730
736
35 Step 5 -
-
-
-
Eta
473301007
369
29 Step 1
193
30 Step 1
-176 42 748
37 Step 6 -
-
-
-
824
65 Step 6 -
Theta
472496524
580
29 Step 4
692
29 Step 6
112* 41 817
46 Step 6 -
-
-
-
631
33 Step 4
481
33 Step 3
-150 47
Iota
232831665
768
37 Step 6 -
-
-
-
Kappa
233333248
659
31 Step 4 -
-
-
-
Lambda
472496520
687
32 Step 5 -
-
-
-
452
33 Step 3
556
33 Step 3
104* 47
Mu
271681346
431
30 Step 1 -
-
-
-
Nu
120105280
641
112
29 Step 5
47 Step 1
-
-
-
-
-
-
--
--
--
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
-
Gain
(* indicates statistical significance)
Progress Step
Progress Standard Error
Progress Score
Initial Step
Initial Standard Error
Initial Score
Gain Error
Gain
(* indicates statistical significance)
Writing
Progress Step
Progress Standard Error
Progress Score
Initial Step
Initial Standard Error
Initial Score
Gain Error
Gain
(* indicates statistical significance)
Reading
Progress Step
Progress Standard Error
Progress Score
Initial Step
Initial Standard Error
Numeracy
Initial Score
TEO Name
NSN
NSN
Name
--
-
--
A690966
Name
Page 27 of 44
NSN
Numeracy
472496517
Omicron
472496526
Pi
167847074
530
32 Step 3 -
-
-
-
Rho
333237096
200
37 Step 1 -
-
-
-
Sigma
56869075
404
30 Step 1 -
-
-
-
Tau
11878836
325
29 Step 5
48 Step 1
-
-
Writing
Xi
Upsilon
681
Reading
-
-
--
-
452
33 Step 3
897 108 Step 6
445* 113
452
33 Step 3
720
47 Step 5
268* 57
452
33 Step 3
428
34 Step 3
-24 47
-
472496514
SUMMARY
Number of
learner
enrolments in
sequences
(excluding no
re-test)
13
Count of learner enrolments
with initial assessments
(excluding learners with ‘no
re-test’)
Number of learner
enrolments in LN sequences
with any gain / initial and
progress assessments
(excluding learners with ‘no
re-test’)
Number of learner
enrolments in LN sequences
with statistically significant
positive gain (excluding
learners with ‘no re-test’)
9
7
4
8
1
2
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
7
0
3
A690966
46
Page 28 of 44
Using Formula 4 for Numeracy:
Number of learner enrolments in LN sequences with statistically significant positive gain
Number of learner enrolments in LN sequences with initial and progress assessments
= Count(Alpha, Theta)
=2
divided by
= Count(Alpha, Beta, Eta, Theta)
=4
= 2/4
= 50%
47
Using Formula 4 for Reading:
Number of learner enrolments in LN sequences with statistically significant positive gain
Number of learner enrolments in LN sequences with initial and progress assessments
= Count(“none”)
=0
divided by
= Count(Alpha) [Beta is exempt]
=1
= 0/1= 0%
48
Using Formula 4 for Writing:
Number of learner enrolments in LN sequences with statistically significant positive gain
Number of learner enrolments in LN sequences with initial and progress assessments
= Count(Lambda, Xi, Omicron)
=3
divided by
= Count(Beta, Gamma, Theta, Lambda, Xi, Omicron, Upsilon)
=7
= 3/7 = 43%
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 29 of 44
Proposed Indicator #2: LN Percent Assessed Indicator
What it reports
49
The percent assessed indicator is used to monitor the extent to which students
are taking initial and progress assessments for the required assessment areas.
50
It is calculated as the number of learner enrolments with the required number of
initial and progress assessments for each assessment area, divided by the
number of learner enrolments in sequences for the reporting year of interest (see
Formula 5).
Formula 5: Percentage with initial and progress assessments for required
assessment areas
Number of distinct learner enrolments in LN sequences with an initial and a progress
assessment
Number of learner enrolments in LN sequences
Rules for calculating LN Percent Assessed Indicator
51
Count the number of sequences with at least one initial and one progress
assessment in the same assessment area, for any one of the required
assessment areas. Do not include in the count any learners with a prior „no retest‟ required in ALL of the required assessment areas. This is the numerator.
52
Count the number of sequences. Do not include in the count any sequence
where a learner has „no re-test‟ required in ALL of the required assessment
areas. This is the denominator.
53
Calculate the indicator.
Data quality issues impacting the indicator and implications for TEO data
submission practices
54
There should be at least one progress assessment for every LN learner
(excluding those designated „no re-test‟) for each calendar year they are enrolled.
Worked example
Please see Table 5 for source data.
55
In 2011 the rules for SAC require either reading or numeracy to be assessed.
Therefore under 2011 rules the indicator would be calculated as follows:
Number of distinct learner enrolments with initial and progress assessments in
reading or numeracy, excluding any learners with „no re-test required‟ in both
reading and numeracy
=count(Alpha, Beta, Eta, Theta)
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 30 of 44
=4
divided by
Number of learner enrolments in LN sequences excluding any learners with „no
re-test required‟ in both reading and numeracy:
= count(Alpha, Beta, Gamma, Delta, Epsilon, Zeta, Eta, Theta, Iota, Kappa,
Lamda, Mu, Nu, Xi, Omicron, Pi, Rho, Sigma, Tau, Upsilon)
=20
=4/20= 20%
In 2012 the rules for SAC require either reading or numeracy or writing to be
assessed. Therefore under 2012 rules the indicator would be calculated as
follows:
Number of distinct learner enrolments with initial and progress assessments in
reading or numeracy or writing, excluding any learners with „no re-test required‟
in all three assessment areas.
=count(Alpha, Beta, Gamma, Eta, Theta, Lambda, Xi, Omicron, Upsilon).
=9
divided by:
Number of learner enrolments in LN sequences excluding any learners with „no
re-test required‟ in all three assessment areas.
= count(Alpha, Beta, Gamma, Delta, Epsilon, Zeta, Eta, Theta, Iota, Kappa,
Lamda, Mu, Nu, Xi, Omicron, Pi, Rho, Sigma, Tau, Upsilon)
=20
=9/20 = 45%
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 31 of 44
Proposed Indicator #3: LN Provision Indicator
What it reports
56
Percentage of LN Provision is an indicator of the proportion of EFTS in level 1-3
SAC funded courses that have literacy and numeracy provision embedded.
Formula 6: Percentage LN Provision
Sum of EFTS delivered in register level 1-3 courses with ELN = Yes
Sum of EFTS delivered in register level 1-3 courses
Rules for calculating LN Provision Indicator
57
The following rules are used to calculate percentage LN Provision.
58
Use the sequence definition rules in paragraphs 3-5.
59
Sum the EFTS consumed by each distinct learner in each course or part of a
course delivered within a calendar year where the ELN flag = Yes
60
Sum the EFTS consumed by each distinct learner in each course or part of a
course delivered within a calendar year; and
61
Calculate the indicator.
Which data to use
Table 6 describes the availability of data used to calculate the LN Provision Indicator.
Table 6: Minimum data used to produce the provision indicator
Period of interest
Year
Example
Reporting year
year n
2012
Year-to-date data produced
at any time in reporting year
Most recent SDR data from
year n or earlier
2012 or earlier
Full year data availability
Finalised December SDR for
each year up to year n
February 2013
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 32 of 44
Worked example
62
Table 7 shows a mock-up of course data from which the percentage of LN provision can be calculated for embedded SAC provision:
Table 7: Mock course level data sourced from SDR via DW
Provider
Code
Provider Name
Register
Level
Literacy Numeracy
NSN
1
Literacy
Numeracy
Flag
Y
0001
Mace Academy
0001
0001
Part of Literacy Numeracy Measure
167847074
CPCCPCPR
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
Mace Academy
2
N
Part of Literacy Numeracy Measure
167847074
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0001
Mace Academy
2
N
Part of Literacy Numeracy Measure
167847074
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
0.0417
0001
Mace Academy
Y
Part of Literacy Numeracy Measure
11878836
CPCCPCPR
6402 Provide Resuscitation Level 2
0.0083
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1123
The Future
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
N
Part of Literacy Numeracy Measure
11878836
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
0001
Mace Academy
0001
Mace Academy
0001
Mace Academy
1
1
N
4
Y
2
Y
Course
Code
Course Title
6402 Provide Resuscitation Level 2
Total Course
EFTS
Delivered
0.0083
0.0417
0.25
0.0417
0.25
0.25
11878836
1123
The Future
0.0417
Not Part of Literacy Numeracy Measure
333237096
NCFS19136
19136 Demonstrate the Social Requirements
for Daily Living in the Rural Ind
0.0167
Not Part of Literacy Numeracy Measure
333237096
NCFS19137
19137 Describe the Opportunities,
Advantages, and Disadvantages of Rural Em
0.0167
Part of Literacy Numeracy Measure
333237096
1121
Introduction
0.0417
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
63
Page 33 of 44
Calculation step 1. Identify all level 1-3 courses delivered (shaded green and peach in the above table) and sum the EFTS delivered to
each NSN enrolled in each course. This is shown below:
Provider
Code
Provider Name
Register
Level
Literacy Numeracy
NSN
1
Literacy
Numeracy
Flag
Y
0001
Mace Academy
0001
Mace Academy
0001
Part of Literacy Numeracy Measure
167847074
CPCCPCPR
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
Mace Academy
2
N
Part of Literacy Numeracy Measure
167847074
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0001
Mace Academy
2
N
Part of Literacy Numeracy Measure
167847074
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
0.0417
0001
Mace Academy
1
Y
Part of Literacy Numeracy Measure
11878836
CPCCPCPR
6402 Provide Resuscitation Level 2
0.0083
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1123
The Future
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
N
Part of Literacy Numeracy Measure
11878836
1122
Skills For Today
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
The Future
0.0417
0001
Mace Academy
19136 Demonstrate the Social
Requirements for Daily Living in the Rural
Ind
Introduction
0.0167
0001
Mace Academy
1
N
2
Y
Course
Code
11878836
1123
Not Part of Literacy Numeracy Measure
333237096
NCFS19136
Part of Literacy Numeracy Measure
333237096
1121
Course Title
6402 Provide Resuscitation Level 2
Total
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
Total Course
EFTS
Delivered
0.0083
0.0417
0.25
0.0417
0.25
0.25
0.0417
1.1586
A690966
64
Page 34 of 44
Calculation steps 2. Identify all level 1-3 courses delivered with embedded literacy and numeracy (shaded green in the above table)
and sum the EFTS delivered to each NSN enrolled in each course. This is shown below:
Provider
Code
Provider Name
Register
Level
Literacy Numeracy
NSN
Course
Code
Course Title
1
Literacy
Numeracy
Flag
Y
0001
Mace Academy
0001
Mace Academy
0001
Part of Literacy Numeracy Measure
167847074
CPCCPCPR
6402 Provide Resuscitation Level 2
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
0.0417
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
167847074
1123
The Future
0.0417
0001
Mace Academy
1
Y
Part of Literacy Numeracy Measure
11878836
CPCCPCPR
6402 Provide Resuscitation Level 2
0.0083
0001
Mace Academy
Y
Part of Literacy Numeracy Measure
11878836
1123
The Future
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1121
Introduction
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
11878836
1123
The Future
0.0417
0001
Mace Academy
2
Y
Part of Literacy Numeracy Measure
333237096
1121
Introduction
0.0417
2
Total
65
Calculation steps 3. Using Formula 6:
Sum of EFTS delivered in register level 1-3 courses with ELN = Yes
Sum of EFTS delivered in register level 1-3 courses
Now divide the EFTS delivered at levels 1-3 courses with ELN flag = yes by number of all EFTS delivered at levels 1-3
= 0.3919/1.1586
= 34%
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
Total Course
EFTS
Delivered
0.0083
0.3919
A690966
Page 35 of 44
Dimensions sourced from the SDR and course qualification register
66
The following dimensions, as described in the TEC Definitions and Rules for
Educational performance indicators11 will be used with LN information:
a
b
c
d
e
Gender;
Age;
Ethnicity;
Course Funding Source Code; and
The embedded literacy and numeracy flag.
Dimensions sourced from the LNAAT
English First Language flag
67
The English First Language flag indicates whether a learner has English as a first
language (=Yes) or not (=No), or if the value is unknown (=Unknown).
68
The most recent value of the flag from any provider is used. This flag can have
different values for the same learner at different providers.
Assessment Area
69
The Assessment Area12 is recorded by the LNAAT against each assessment. It has
one of four values:
a
b
c
d
Numeracy;
Reading;
Writing; and
Vocabulary.
Assessment Strand
70
An assessment strand is a sub-set of the assessment area and includes13,14:
a
Numeracy
i
ii
iii
b
General Numeracy
Number Knowledge
Number Strategies and Measurement
Reading
11
http://www.tec.govt.nz/Documents/Reports%20and%20other%20documents/measuring-studentachievement-rules-and-definitions.pdf
12
http://literacyandnumeracyforadults.com/Assessment-Tool-Support/Glossary/Assessment-Strand
13
http://literacyandnumeracyforadults.com/Assessment-Tool-Support/Glossary/Assessment-Strand
14
http://literacyandnumeracyforadults.com/The-Learning-Progressions
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 36 of 44
i
ii
iii
iv
c
Writing
i
d
Write to Communicate
Vocabulary
i
71
Read with Understanding
Language and text features
Comprehension
Reading critically
Vocabulary.
An assessment strand is a sub-set of the assessment area and includes15,16:
Step
72
The Step is recorded by the LNAAT against each assessment. For Reading, Writing
and Numeracy, it records the level of the learning progression17 that a learner has
attained in an assessment as numeric values 1-6. Vocabulary assessments do not
map to the steps and instead use the terms „Emerging‟, „Expanding‟ and „Extended‟
to describe learners progress.
Assessment Type
73
This flags whether the assessment adjusts to the student‟s answers so that the
questions reach the level of the student‟s literacy or numeracy abilities. There are
three types18:
a
b
c
d
Adaptive (adjusts automatically);
Non Adaptive (does not adjust automatically);
Non Adaptive for Printing19 (does not adjust automatically); and
Snapshot (adjusts automatically).
74
The types of assessment to be administered are determined by the LNAAT rules and
if these change then any changed requirements will be flowed through to the
business rules for measuring gain.
75
Further detail on the LNAAT can be found in Appendix F.
15
http://literacyandnumeracyforadults.com/Assessment-Tool-Support/Glossary/Assessment-Strand
http://literacyandnumeracyforadults.com/The-Learning-Progressions
17
http://literacyandnumeracyforadults.com/The-Learning-Progressions
18
http://literacyandnumeracyforadults.com/Assessment-Tool-Support/Glossary/Assessment-Type
19
Note: these assessments‟ results may be marked and input into the LNAAT up to 8 weeks after
calendar year end.
16
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 37 of 44
Appendix B: Definitions and Business Rules FFTO
and YG
This appendix will be released as soon as possible and you will have three weeks
from the date of it release to comment. Once all the appendices have been released
and feedback received, they will be consolidated into a single LN definitions and
business rules document.
Appendix C: Definitions and Business Rules ILN/WPL
This appendix will be released as soon as possible and you will have three weeks
from the date of it release to comment. Once all the appendices have been released
and feedback received, they will be consolidated into a single LN definitions and
business rules document.
Appendix D: Definitions and Business Rules ITR
This appendix will be released as soon as possible and you will have three weeks
from the date of it release to comment. Once all the appendices have been released
and feedback received, they will be consolidated into a single LN definitions and
business rules document.
Appendix E: Abbreviations
Abbreviation Term
Abbreviation
Term
DW
TEC Data Warehouse
NZSCED
New Zealand Standard
Classification of
Education (the field of
study for courses and
qualifications)
EFTS
Equivalent full-time student
OTEP
Other Tertiary Education
Provider
ELN
Embedded Literacy and
Numeracy
PBRF
Performance-Based
Research Fund
EPI
Educational Performance
Indicator
PTE
Private Training
Establishment
ILU
Industry Liaison Unit (name
of database holding
enrolments and completions
for industry training)
Institute of Technology and
Polytechnic
QAC code
Qualification Award
Category code (used to
denote formal and nonformal qualifications)
Rural Education Activities
Programme
ITP
REAP
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 38 of 44
LN
Literacy and Numeracy
SAC
Student Achievement
Component Fund
LNAAT
Literacy and Numeracy for
Adults Assessment Tool
(The Assessment Tool)
National Student Number
SDR
Single Data Return
STM
Standard Training
Measure
NZQA
New Zealand Qualifications
Authority
TEC
The Tertiary Education
Commission
NZQF
New Zealand Qualifications
Framework
TEO
Tertiary Education
Organisation
NSN
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 39 of 44
Appendix F: LNAAT and assessments
1
The LNAAT was developed for a New Zealand context drawing on a review of
the international literature of approaches to measuring gain from literacy and
numeracy education. That review assisted with identifying the most
appropriate interpretation of “real” gain and a requirement to use statistically
significant gain within appropriate confidence intervals. It also highlighted the
applicability of “effect size” and “bias corrected effect size” statistics in the
determination of the relative magnitude of the effect of an intervention.
2
The TEC has built on this research and more than 2,000 assessment items
have been trialled with thousands of adults. The LNAAT that has been
designed as a result will demonstrate the added value of programmes,
regardless of the level at which learners begin. The sophistication of the item
development and calibration means that assessments are rigorous and
comparable. For the first time, providers are able to show what their
programmes are achieving and will be able to compare their performance with
others.
3
The LNAAT has been available for use since 2010. The TEC has required the
use of the LNAAT for all learners enrolled in courses that include literacy and
numeracy components embedded in them since the beginning of 2011.
4
The LNAAT provides assessments in four areas:
a
numeracy;
b
reading;
c
writing; and
d
vocabulary.
5
Assessments can be performed using four assessment types: adaptive, nonadaptive, non-adaptive for print and snapshot formats.
6
Adaptive assessments must be taken online. They are considered adaptive
because the assessment adapts the step level at which the questions are
pitched according to user performance within the test.
7
Non-adaptive and non-adaptive for print assessments require the tutor to
select the difficulty of the assessment in terms of the range of steps covered,
unless the assessment is for writing. These step ranges are 1-3, 2-5 and 4-6.
There is anecdotal evidence that some TEOs may be inappropriately post
testing some learners using assessments at step 4-6 difficulty where step 2-5
should have been used, resulting in lower than expected second
assessments and the tool showing negative gains. For this reason, it is
important that educators are aware of the consequences of selecting
assessments with difficulties inappropriate for their learners.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
8
20
Page 40 of 44
Snapshot assessments are briefer versions of the adaptive assessments and
require the tutor to select a step level “threshold”20. With fewer questions
asked in the snapshots, there is a greater measurement uncertainty
associated with learners‟ assessment scores. When the scores have greater
measurement uncertainty; the difference in scores must be larger in order to
show statistically significant difference. This may therefore be a factor
influencing gain performance monitoring.
http://literacyandnumeracyforadults.com/Assessment-Tool-Support/Glossary#glossaryterm-8721
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 41 of 44
Appendix G: Research on Performance Indicators
1
The TEC has reviewed published information to identify21 how other entities
measure performance of literacy / numeracy programmes. That work
suggests there are two key approaches to consider.
Measurement of attainment against a Standard
2
One approach is to measure the degree to which learners attain a set
standard. This standard could be national norms, desired outcomes, specific
assessment scores (or degree of improvement in scores), a mean or median
across a jurisdiction, or expected benchmarks or competency levels set by
national policy. There are a number of examples where performance is
measured against a set standard, including:
The Ministry of Education‟s new 2014 requirements around literacy and
numeracy specify that school students need to achieve ten credits each
from literacy/numeracy unit standards. For university entrance, students
need to achieve ten credits (five in reading, five in writing) at level two or
higher achievement standards, or from an academic literacy common
assessment tool (CAT) at level three.22
The Education Review Office monitors National Certificate of
Educational Achievement (NCEA) results and data from assessment
tools (such as Assessment Tools for Teaching and Learning (asTTLe),
Participatory Assessment Tool (PAT), Supplementary Test of
Achievement in Reading (STAR), National Education Monitoring Project
(NEMP) to measure how well students are achieving literacy and
numeracy standards compared to national norms or standards.23
The National Reporting System in the United States on performancebased funding of adult basic education uses a variety of indicators using
outcome benchmarks, such as “number of students completing one of
more levels” (not specified), “attain GED”,24 or “enter employment”.
Other indicators used in this system use other standards such as the
21
The TEC thanks the Ministry of Education Library for conducting the literature search,
during August and September 2011. This review is not meant to be comprehensive. It was
targeted at the use of assessment measures as the basis for an indicator of a performance,
usually of a literacy/numeracy programme or an institution delivering the programme.
22
Tony Turnock, Manager, Secondary Outcomes, Schooling, Ministry of Education, personal
communication. The standards are more complicated than described in the text, but the
point here is that they involve the attainment of a minimum number of credits.
23
Education Review Office 2011 Evaluation Indicators for School Reviews Wellington: ERO,
p.15.
24
Graduate Equivalency Degree, equivalent to a high school diploma, and could also refer to
an assessment that if passed equates to the Graduate Equivalency Degree.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 42 of 44
mean of State performance (“percentage of students passing the GED
will be equal to or greater than the state average”).25
United States “National Report Cards” on higher education (at State and
national levels) use the per cent of learners scoring at or above
“proficient” on various assessments.26 The Australian Language,
Literacy and Numeracy Program 2010-2013 uses an indicator that
included “all blocks of 200 hours [of training] achieve satisfactory
progress or above” (italics added, progress measurement not stated).27
Measurement of change in scores
3
The second approach is to use the difference in scale scores between two
assessment points. The difference could be any difference in scores,
“significant” learning gains, an attainment of a higher level of competency
associated with a score, or choosing a threshold of a gain in score. Some
examples of such indicators can be seen in the Even Start Family Literacy
Project, implemented in several States in the United States as described
below:
A typical measure is “The percentage of Even Start adults who achieve
significant learning gains on measures of reading/English language
acquisition as measured by the Comprehensive Adult Student
Assessment System (CASAS) and the Tests of Adult Basic Education
(TABE)”28 (italics added. Some States do not allude to the amount of
gain expected.)
Hawaii‟s Even Start program has “At least 70% of the adult participants
will demonstrate a scale score gain of at least 4 points on each of the
subtests of CASAS”, and “at least 80% of the adult participants will
increase their literacy and numeracy skills at least one level on the EFF
(Equipped for the Future, an American literacy/numeracy program)
Subject Tables as judged by portfolio analysis at the initial and final
assessments.”29
25
See, for example, graphs on achievement on:
www.nrsweb.org/docs/infocus/report%20cards/NY%20annualreportrockeller.pdf, from the
Adult Education Report, Adult Education Workforce Development Team, New York State
Department of Education; U.S. Department of Education, Office of Vocational and Adult
Education, 2010 Technical Assistance to States on Performance-Based Funding Final
Project Report (by MPR Associations) Washington DC; U.S. Department of Education,
Office of Vocational and Adult Education, Division of Adult Education and Literacy 2007
Desk Monitoring: Improving Program Performance (by American Institutes for Research)
Washington DC.
26
The National Center for Public Policy and Higher Education 2008 Measuring Up 2008 The
National Report Card on Higher Education.
27
Department of Education, Employment and Workplace Relations 2010 Language, Literacy
and Numeracy Program 2010-2013 Program Guidelines Australia pp 104-105.
28
US Department of Education, Office of Elementary and Secondary Education, Even Start
Program, http://www2.ed.gov/programs/evenstartforumula/performance.html.
29
Hawaii‟s Even Start Statewide Initiative Grant, Hawai‟i family Literacy Consortium,
Performance Indicators: Summary. http://literacynet.org/hflc/indicators.pdf p.2
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 43 of 44
Illinois Even Start program‟s indicator for adult basic education is “At a
minimum, sixty-five per cent (65%) of Even Start adults who have
completed at least 75 hours of Adult Basic Education (ABE) or Adult
Secondary Education (ASE) instruction will demonstrate at or above the
average specified test score gains on the Test of Basic Adult EducationReading (TABE-R), based upon the entry test score for new students
and the most recent test score dated on or after March 3rd of the
preceding project year for continuing students.” There is a table of
specified test score gains according to beginning level, so adults judged
to be at “Beginning Adult Basic Education” are expected to show a test
score gain of 30 points, while adults who are “High intermediate ABE”
are expected to gain 20 points. “Low advanced ABE” adults are
expected to gain 10 points on the TABE-R after 75 hours of
instruction.30
4
When indicators are defined, they usually allude to conditions that could
impact the indicator. For example, Illinois‟s indicator contains several
conditions that impact on how the indicator is calculated:
Adults only are included, as children can also participate in Even Start
and may have less exposure to learning than adults; they are assessed
using a different assessment tool;31
A minimum number of hours of instruction is required –in Illinois this is
75, but the number can vary from 12 (MPR Associates, footnote 5) to
200 (Australia, Department of Education, Employment and Workplace
Relations -DEEWR, footnote 6).
prior instruction in the program (“new” versus “continuing” students),
initial level of competence (those at higher levels are expected to evince
less gain in score on the progress assessment).
5
Motivation could also impact an indicator based on performance on
assessments. For example, Hawaii‟s Even Start program includes the
following outcome indicators:
“At least 60% of the adult participants for whom a higher education or
work-related goal is judged by the staff to be appropriate at the time of
initial enrolment, and who enter at Novice +1 level or above on rubrics
in reading, writing, math, problem-solving, plan, interpersonal skills of
cooperation, will, within one calendar year of leaving the family literacy
program, enrol in higher education, obtain a job, receive a promotion or
30
Illinois Family Literacy Performance Indicators for Adults 2010,
http://www.isbe.state.il.us/earlychi/pdf/adult_indicators.pdf
31
For example, the Peabody Picture Vocabulary Test-III, US Department of Education, Office
of Elementary and Secondary Education, Even Start Program, op cit,
http://www2.ed.gov/programs/evenstartforumula/performance.html
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4
A690966
Page 44 of 44
upgrade in employment, and remain enrolled or hold the achieved
position for at least 6 months”32 (italics added).
“At least 60% of the adult participants for whom high-school-level
certification is an appropriate goal as demonstrated by rating of
intermediate Level or above on Rubrics in Reading, Writing, Math, and
Problem Solving on the initial assessment, will receive the appropriate
credential”33 (italics added).
6
Adjusting for these conditions can be via calculating them as a 3-year (for
example) rolling average (MPR Associates, footnote 5) or by weighting the
indicator (National Center for Public Policy and Higher Education, footnote 6).
The TEC will work to understand the impact these conditions may have on
the use of the information produced by the indicators.
32
Hawaii‟s Even Start Statewide Initiative Grant, Hawaii family Literacy Consortium,
Performance Indicators: Summary. http://literacynet.org/hflc/indicators.pdf p.2.
33
Hawaii‟s Even Start Statewide Initiative Grant, http://literacynet.org/hflc/indicators.pdf ibid.
Indicators for literacy and numeracy provision and gain– June 2012 version 0.4