What’s the adult L&N story?
The results from international
assessments of adult literacy and
numeracy skills
Juliette Mendelovits
CEET 17th Annual National Conference
Friday 1 November 2013
Monash Conference Centre, Melbourne
A literacy deficit
Half of all adults in Tasmania cannot read or write
properly, and many of their children are following
in their footsteps as badly needed school reforms
are frustrated. Sarah Dingle investigates.
A report by the Australian Bureau of Statistics for 2011-2012
shows half of all Tasmanians aged 15 to 74 are functionally
illiterate, and more than half are functionally innumerate—
meaning they don’t have the skills needed to get by in the
modern world, like filling out forms, or reading the
instructions on their prescription.
ABC Radio National, Background Briefing,
Sunday 22 September 2013
WHAT’S BEHIND THE HEADLINES?
Tasmanian 'illiteracy' and
'innumeracy' levels
15 to 74 year olds
2006 - ALLS
Document literacy
ALLS 2006
Numeracy
Below Level 3
Tasmania
51%
ALLS 2006
Tasmania
Below Level 3
56%
2011-12 PIAAC
Literacy
Numeracy
PIAAC 2011-12
Below Level 3
PIAAC 2011-12
Tasmania
49%
Tasmania
Below Level 3
58%
Australian 'illiteracy' and
'innumeracy' levels
Document literacy
ALLS 2006
Numeracy
Below Level 3
ALLS 2006
Below Level 3
Tasmania
51%
Tasmania
56%
Victoria
49%
Victoria
54%
Australia
47%
Australia
53%
Literacy
Numeracy
PIAAC 2011-12
Below Level 3
ALLS 2006
Below Level 3
Tasmania
49%
Tasmania
58%
Victoria
46%
Victoria
54%
Australia
44%
Australia
55%
According to ALLs [and PIAAC?]
About 50% of
Australian adults
are functionally
illiterate and
innumerate
How can this be true?
What does ‘below Level 3’ mean?
According to the international ALLS reports,
Level 3 is
 ‘a suitable minimum level for coping with the
increasing demands of the emerging knowledge
society and information economy’
A Level 2 literacy item from PIAAC
Level 2 literacy description (226 to 275 points)
At this level, the medium of texts may be digital or printed, and texts may comprise
continuous, non-continuous, or mixed types. Tasks at this level require respondents
to make matches between the text and information, and may require paraphrasing or
low-level inferences. Some competing pieces of information may be present. Some
tasks require the respondent to
• cycle through or integrate two or more pieces of information based on criteria;
• compare and contrast or reason about information requested in the question; or
• navigate within digital texts to access and identify information from various parts of
a document.
A Level 3 numeracy item from PIAAC
Level 3 numeracy description (276 to 325 points)
Tasks at this level require the respondent to understand mathematical information
that may be less explicit, embedded in contexts that are not always familiar and
represented in more complex ways. Tasks require several steps and may involve the
choice of problem-solving strategies and relevant processes. Tasks tend to require
the application of number sense and spatial sense; recognising and working with
mathematical relationships, patterns, and proportions expressed in verbal or
numerical form; and interpretation and basic analysis of data and statistics in texts,
tables and graphs.
PIAAC: an overview
The Programme for the International Assessment of Adult
Competencies (PIAAC)
• An international survey of adults 16-64 years of age (15 to 74 year
olds in Australia)
• Commissioned by the OECD, implemented internationally by a
consortium led by Educational Testing Service (ETS) (USA)
• Administered in Australia by the ABS
• Measures the domains of
• literacy (reading)
• numeracy
• problem solving in technology-rich environments (PSTRE)
• Modes of administration
• paper and pen
• computer
• Administered in 25 OECD countries (including Australia) and subnational entities in 2011-12
• Comparable with the results of the Adult Literacy and Life Skills
Survey ??
Results from PIAAC
Means for Australia and selected participating
countries
Australia
Canada
England/ N.Ireland
Japan
Korea
United States
Average
Literacy
Mean scale
score
280
273
272
296
273
270
273
Level 2: 226 to 275 points
Level 3: 276 to 325 points
Numeracy
Mean scale
score
268
265
262
288
263
253
269
PSTRE
% at Level 2
or 3
38
37
35
35
30
31
34
Results from PIAAC
% at each literacy level for Australia and selected
participating countries
Australia
Canada
England /
N.Ireland
Japan
Korea
United
States
Average
Below Level 1
Level 1
Level 2
Level 3
Level 4/5
3
4
3
9
13
13
29
32
33
39
37
36
17
14
13
1
2
4
4
11
14
23
37
33
49
42
34
22
8
12
3
12
33
38
12
CONSTRUCTION OF THE LITERACY
AND NUMERACY REPORTING SCALES
Items and test-takers are placed on the
same scale
High Achievers
2
1
0
-1
Low Achievers
-2
-3
|
|
X|
|
X|
XX|
XX|7
XX|
XXX|34
XXX|19
XXXX|23
XXXXXX|12
XXXXXX|
XXXXX|11 22
XXXXXX|26 27 33
XXXXXXX|10 14 30 32
XXXXXXX|2 5 20
XXXXXXXX|1 4 6 21 29
XXXXXXXXXX|8 16
XXXXXXX|9 31
XXXXXXXXX|3
XXXXXXXX|18 25
XXXXXXXX|13
XXXXXXX|15
XXXXXXXX|
XXXXXX|
XXXXXXX|
XXXX|24
XXXX|
XXXXX|
XXX|28
XX|17
XX|
XX|
X|
X|
X|
X|
|
Difficult items
Easy items
Relationship between difficulty of assessment
items and proficiency of adults
Anchoring of tests
The Rasch model derives comparable scores
for two different assessments via the
inclusion of 'anchor' items.
Anchoring can be used:
 across test forms (e.g. Rotated designs)
 across populations (e.g. Different countries)
 across time (e.g. Different Adult literacy cycles –
ALLS and PIAAC)
Example of
anchoring
between two
populations of
different
ability
Different tests
Common
items
Same Scale!
A map of item difficulty and test-takers’
proficiency
Once items and adults are placed on the same scale,
proficiency levels are defined
Level 5
2
Level 4
1
Level 3
0
Level 2
-1
Level 1
Below Level 1
-2
-3
|
|
X|
|
X|
XX|
XX|7
XX|
XXX|34
XXX|19
XXXX|23
XXXXXX|12
XXXXXX|
XXXXX|11 22
XXXXXX|26 27 33
XXXXXXX|10 14 30 32
XXXXXXX|2 5 20
XXXXXXXX|1 4 6 21 29
XXXXXXXXXX|8 16
XXXXXXX|9 31
XXXXXXXXX|3
XXXXXXXX|18 25
XXXXXXXX|13
XXXXXXX|15
XXXXXXXX|
XXXXXX|
XXXXXXX|
XXXX|24
XXXX|
XXXXX|
XXX|28
XX|17
XX|
XX|
X|
X|
X|
X|
|
PIAAC literacy Level 4
Tasks at this level often require respondents to perform multiple-step
operations to integrate, interpret, or synthesise information from complex
or lengthy continuous, non-continuous, mixed, or multiple type texts. ...
PIAAC literacy Level 2
... Tasks at this level require respondents to make matches between the text
and information, and may require paraphrasing or low-level inferences.
Some competing pieces of information may be present. . ...
PIAAC literacy Level 1
Most of the tasks at this level require the respondent to read relatively short
digital or print continuous, non-continuous, or mixed texts to locate a single
piece of information. ...
Described literacy levels from PIAAC
Level 2
... Tasks at this level
require respondents to
make matches between
the text and information,
and may require
paraphrasing or low-level
inferences. Some
competing pieces of
information may be
present. . ...
WHAT DOES PIAAC TELL US ABOUT
THE DISTRIBUTION OF AUSTRALIANS’
SKILLS?
Sex and skills
Australians aged 15 to 74
Percentage
PIAAC literacy level by sex
45
40
35
30
25
20
15
10
5
0
Males
Females
Below Level 1 / Level 1
Level 2
Level 3
Level 4/5
PIAAC level
PIAAC numeracy level by sex
40
35
Percentage
30
25
20
Males
15
Females
10
5
0
Below Level 1 / Level 1
Level 2
Level 3
PIAAC level
Level 4/5
Employment and skills
Australians aged 15 to 74
PIAAC literacy level by employment status
45
40
35
Percentage
30
25
Employed
20
Not employed
Total
15
10
5
0
Below Level 1 /
Level 1
Level 2
Level 3
PIAAC Level
Level 4/5
SO WHAT ABOUT THE HEADLINES?
How can all of this be true?
PISA
ALLS
According to PISA
About 15% of
Australian 15-yearolds are below the
required standard of
reading and maths
NAPLAN
According to ALLS
about 50% of
Australian adults
are below the
minimum standard
of literacy and
numeracy
According to
NAPLAN about 6%
of Australian Year 9
students are below
the required
standard of literacy
and numeracy
Comparability
of scales?
A project sponsored by the
Victorian Department of
Education and Early
Childhood Development
and the Australian Council
for Education Research
Preliminary
findings:
Literacy
Questions?