Making the most of PIAAC:
Preliminary investigation of adults’ numeracy
practices through secondary analysis of the
PIAAC dataset
Diana Coben
University of Waikato
New Zealand
dccoben@waikato.ac.nz
Barbara Miller-Reilly
University of Auckland
New Zealand
Paul Satherley and David Earle
Ministry of Education
New Zealand
ALM22 conference: Opening Our Mathematical Eyes
1-2 pm, 13th July 2015
What is PIAAC?
• Programme for the International Assessment of Adult
Competencies, aka The Survey of Adult Skills
www.oecd.org/site/piaac/
• PIAAC assesses key cognitive and workplace
competencies in:
– Literacy
– Numeracy
– Problem solving in technology-rich environments
• Collects information on how often people undertake a
range of (literacy and) numeracy activities at work and
in everyday life
• New Zealand’s results due 2016 (PIAAC Round Two)
What are we doing?
• Exploring what secondary analysis of
PIAAC data can tell us about what adults
do regarding numeracy
• Reviewing accessibility and userfriendliness of the data
• Encouraging beginners and experienced
practitioners and researchers to explore
the PIAAC dataset
Making the most of secondary data sources
“A number of secondary data sources […] are
currently under utilised for education research
on adults.
One reason for this is that a huge time
investment is needed by the researcher to
analyse such data, as the data sets are large
and complex.
Making available better guidance specific to
education researchers would be helpful to
encourage greater use of these data sets.”
(Vignoles, 2007)
Numeracy in PIAAC
“the ability to access, use, interpret and
communicate mathematical information and
ideas in order to engage in and manage the
mathematical demands of a range of situations
in adult life…”
“Numerate behaviour involves managing a
situation or solving a problem in a real context,
by responding to mathematical
content/information/ideas represented in
multiple ways.”
(OECD, 2012, p. 34)
PIAAC data related to numeracy practices
• PIAAC directly measures numeracy skill
through a numeracy assessment
• Provides average scores on the PIAAC
numeracy scale for whole 16-65 year old
population, or subgroups
• Background Questionnaire - administered
face-to-face in the respondent’s home by
an interviewer who enters the answers into
a laptop computer
PIAAC Background Questionnaire
Covers educational background, skills used at work (for
those currently or recently in employment) and in other
contexts such as the home and the community, e.g.:
– Demographic characteristics – age, gender, ethnicity, country of
birth, first language, language spoken most often at home
– Other personal characteristics – learning dispositions, selfassessed health status
– Education and training characteristics
– Work characteristics
– Self-assessed mathematics skills for work
– Self-assessed reading and writing skills for work
– Skill use in everyday life – literacy activities, ICT activities
May help answer interesting questions when analysed
together with numeracy practice information
The numeracy practices questions in the PIAAC
Background Questionnaire
Interviewer asks respondent how often s/he undertakes 7
different numeracy practices in work or everyday context:
• In your job, how often do you usually calculate prices, costs or
budgets?
• Use or calculate fractions, decimals or percentages?
• Use a calculator – either hand-held or computer based?
• Prepare charts, graphs or tables?
• Use simple algebra or formulas?
• Use more advanced mathematics or statistics such as calculus,
complex algebra, trigonometry, or use of regression techniques?
The same questions are asked, in the past tense, for
people who are not currently working, but who worked in
the last 12 months
Frequency… and its limitation
• The five frequency options are:
–
–
–
–
–
never
less than once a month
less than once a week but at least once a month
at least once a week but not every day
every day
• Limitation: questions only measure
frequency; they do not measure intensity or
complexity of numeracy activity
Types of analysis and analysis issues
• A range of types of analysis are possible to help answer
different questions about adults’ numeracy practices.
• First steps often simple univariate tabulations - provide a
‘big picture’ view
• Next steps may be multivariate analysis - allows better
understanding of complex associations between factors
• PIAAC dataset can provide in-depth understanding of
associations between various factors
• Multivariate analyses:
– can show which factors are most strongly associated with
frequent numeracy practice, and
– can provide numerical measures of, e.g., how much increase in
frequency of numeracy practice is associated with one unit of
measured numeracy skill, whilst holding other factors constant
Secondary analysis of PIAAC data
Secondary analysis of PIAAC data is:
•
•
•
•
undertaken later in time than primary analysis
more in-depth and multivariate
less headline-focused and descriptive
brings together different parts of PIAAC dataset to generate new
knowledge and understanding of associations and relationships
• provides empirical findings to build or refine conceptual or statistical
models
• often theme-based, rather than general in focus
Primary PIAAC analysis:
• includes average scale scores in PIAAC skill domains for key
demographic and socioeconomic groups
• aligns with international headline reporting
What PIAAC data cannot tell us
• PIAAC data cannot tell us to what extent frequent
numeracy practice causes high numeracy skill
• Analytical writing should clearly distinguish associations
from causes and not use ambiguous language such as
leads to, brings about, influences, is linked with, etc.
• It may be that numeracy practice, opportunity or
requirement to undertake numeracy practice, and
numeracy skill, are all mutually reinforcing
• Even where the data show a strong association between
two factors (e.g., numeracy practice and numeracy skill)
whilst controlling for other factors, we cannot infer that a
change in one factor will result in a change in the other –
either on an individual or a group level
What PIAAC can tell us about adults’ numeracy
activities
• How often do people perform numeracy activities at
work?
• How often do people perform numeracy activities in
everyday life?
• What are the patterns for different groups in numeracy
activities at work?
• What are the patterns for different groups for numeracy
activities in everyday life?
• What similarities or differences exist for numeracy
activity at work and in everyday life?
• What similarities or differences exist for numeracy and
literacy activity at work?
What PIAAC can tell us about adults’ numeracy activities
(continued)
• What similarities or differences exist for numeracy and
literacy activity in everyday life?
• What are the patterns of work-related numeracy activity
together with ICT activities?
• What are the characteristics of people who seem to be
matched or mismatched on numeracy skill at work and
numeracy activity? Is ‘field of study of highest
qualification’, or ‘occupation’ related to the match or
mismatch?
• What changing patterns for young people’s numeracy
activities can we see on their pathways from school to
tertiary education to work?
• How have patterns of numeracy activity at work changed
over time?
What PIAAC data are publicly available and
how can they be accessed?
• Information on PIAAC ‘Public Data & Analysis’ is at
http://www.oecd.org/site/piaac/publicdataandanalysis.htm
• OECD PIAAC Gateway website ‘Data Tools › Datasets and
Tools’ http://piaacgateway.com/datasets/ gives users tools
needed to analyse the PIAAC dataset
• Guide: What You Need to Consider (AIR PIAAC Team, nd)
https://piaac.squarespace.com/s/WorkingWithPIAACData_
022015.pdf
• Online Distance Learning Data Training (DLDT) modules
http://nces.ed.gov/training/datauser/#PIAAC cover
statistical procedures and methods of analysis
• FAQ webpage http://nces.ed.gov/surveys/piaac/faq.asp
• Interactive ‘Q & A. Your questions, our answers’
athttp://piaacgateway.com/your-questions-our-answers/
PIAAC Data Explorer (PDX)
• PIAAC Data Explorer (PDX)
http://piaacdataexplorer.oecd.org/ide/idepiaac/variables.aspx
-web-based application for accessing PIAAC
data, does not require advanced statistical
knowledge or specialist software
• point-and-click interface for creating statistical
tables and charts and exploring levels of
adult skills and demographics
• 4 steps to go through when using PIAAC
Data Explorer, supported by DLDT modules
PIAAC International Data Explorer
Step 1
• We chose “Numeracy” as our Subject
• Note that “Adults (16-65)” only option for Age at this point
• After choosing “Numeracy”, fuller screen appears with a
choice of dataset to use: “PIAAC 2012”, “ALL 2003” or
“All years/studies”. We chose “PIAAC 2012”
• Then choice of 72 Categories and Groups.
– Under the Category “Scale Scores”, and Sub-Category “Skills”,
the Measure “PIAAC Numeric: Numeracy” is already selected.
– Under the Group “International”, we chose the Jurisdiction
“OECD average”
• Choose “Select Variables” to move to Step 2 to select
the independent variable(s)
STEP
1
Select Criteria
Select Variables
Edit Reports
Build Reports
Language:
English
STEP 1:
Select criteria from each drop-down menu to begin. Additional options related to your selections will appear. Then select measures, jurisdictions, and years based on
available data.
Subject:
Age:
Numeracy
View all (72)
Category
View Selected (1)
Sub Category
Scale scores
Measure
Reset
Adults (16-65)
Search:
All
Years/Studies
Help
PIAAC 2012
Go
ALL 2003
PIAAC Numeric: Numeracy details
Skills
Skill use
Reading
components
Reading
component timings
Full population
estimates
International
Group
background
questionnaire
Derived
International
variables
X
Jurisdiction
All
Years/Studies
PIAAC 2012
ALL 2003
OECD Average
Selected countries
OECD National Entities
OECD Sub-National Entities
Partners
Select Variables
About
Accessible Version
Data Explorer Technology
PIAAC International Data Explorer
Step 2
• Under the Category (and Sub-Category) “Major reporting group”
we chose the variable “All Adults”.
• In addition, under the category “International Background
Questionnaire”, and Sub-Category” “Skill use – literacy &
numeracy”, we chose the following six variables:
•
•
•
•
•
•
“Skill use work - Numeracy - How often - Calculating costs or budgets”
“Skill use work - Numeracy - How often - Use or calculate fractions or percentages”
“Skill use work - Numeracy - How often - Use a calculator”
“Skill use work - Numeracy - How often - Prepare charts graphs or tables”
“Skill use work - Numeracy - How often - Use simple algebra or formulas”
“Skill use work - Numeracy - How often - Use advanced math or statistics”
• Now that the variables are selected, choose “Edit Reports” to
move to Step 3.
STEP
2
Select Criteria
Select Variables
Edit Reports
Build Reports
STEP 2:
Select at least one variable from the category list below. View the list of all available variables, view by selected variables only, or search variables by keywords. Years
selected will override previous selections.
Subject, Age: Numeracy, Adults (16-65)
Jurisdiction: OECD Average
Measure: PIAAC Numeric: Numeracy
Year/Study: PIAAC 2012
View all (334)
Category
Sub Category
Major
reporting groups
Major
reporting groups
Help
Reset
View Selected (7)
Variable
Search:
All Years/Studies
PIAAC 2012
Go
ALL 2003
All adults details
Person resolved gender from BQ and QC check
(derived) details
Education - Highest qualification - Level details
Current status/work history - Subjective status
details
Current work - Employee or self-employed details
Background - Born in country details
Age groups in 5-year intervals based on LFS
groupings (derived) details
Age in 10 year bands (derived) details
International
background
questionnaire
Education
Current status
Current work
Last job
Skill use at
work
Edit Reports
PIAAC International Data Explorer
STEP 2 (later choices)
International background
questionnaire
Education -->
etc
Skill use at work -->
Skill use - literacy &
numeracy -->
Skill use work - Literacy - --- etc
Skill use work - Numeracy - How often
- Calculating costs or budgets
Skill use work - Numeracy - How often
- Use or calculate fractions or
percentages
Skill use work - Numeracy - How often
- Use a calculator
Skill use work - Numeracy - How often
- Prepare charts graphs or tables
Skill use work - Numeracy - How often
- Use simple algebra or formulas
Skill use work - Numeracy - How often
- Use advanced math or statistics
Skill use work - ICT - --- etc
PIAAC International Data Explorer
Step 3
• The PIAAC International Data Explorer then
generated seven possible reports.
• These seven reports can be reviewed, edited,
deleted or copied. New reports can be created,
new format and statistics chosen.
• We selected Averages and Percentages from
the drop down menu “Statistics Options”.
• We chose “Build Reports” to move to Step 4 to
view the completed reports.
STEP
3
Select Criteria
Select Variables
Edit Reports
Build Reports
STEP 3:
Preview and edit existing reports using the action links next to each report name. Create new reports, set format and statistic options. (New and copied reports will appear
at the bottom of the report list.)
Subject, Age: Numeracy, Adults (16-65)
Jurisdiction: OECD Average
Measure: PIAAC Numeric: Numeracy
Variables: All adults, Skill use work - Numeracy - How often - Calculating costs or budgets, Skill use work - Numeracy - How often - Use or calculate fractions
or percentages, Skill use work - Numeracy - How often - Use a calculator, Skill use work - Numeracy - How often - Prepare charts graphs or tables, Skill use
work - Numeracy - How often - Use simple algebra or formulas, Skill use work - Numeracy - How often - Use advanced math or statistics
Year/Study: PIAAC 2012
Create New Report
Report
All
Format Options
Help
Reset
Statistics Options
Action
Measure
Variable
Year/Study Jurisdiction
Statistic
Report 1
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy All adults
PIAAC 2012 OECD Average Averages
Report 2
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Calculating costs or budgets
PIAAC 2012 OECD Average Averages
Report 3
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use or calculate fractions or percentages PIAAC 2012 OECD Average Averages
Report 4
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use a calculator
PIAAC 2012 OECD Average Averages
Report 5
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Prepare charts graphs or tables
PIAAC 2012 OECD Average Averages
Report 6
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use simple algebra or formulas
PIAAC 2012 OECD Average Averages
Report 7
Preview | Edit
Delete | Copy
PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use advanced math or statistics
PIAAC 2012 OECD Average Averages
Build Reports
About
Accessible Version
Data Explorer Technology
PIAAC International Data Explorer
Step 4
• For each of the six variables of Numeracy Skills used at
work, a report is generated.
• Each report lists Averages of the Numeracy scale scores
and Percentages of the population (each with Standard
Errors) for the response categories (Never; Less than once a
month; Less then once a week but at least once a month; At least
once a week but not every day; Every day).
• At Step 4 each report table can be viewed by selecting
the report name from the drop-down menu. It is possible
to generate charts and complete significant tests for any
table.
• We chose “Export reports” to download selected reports
as Excel worksheets (or Word document or HTML).
STEP 4
Select Criteria
Select Variables
Edit Reports
Build Reports
STEP 4:
View each report table by selecting the report name from the drop-down menu. Create report types to edit and preview, each tab created represents one report type to
export.
Help
Subject, Age: Numeracy, Adults (16-65)
Jurisdiction: OECD Average
Measure: PIAAC Numeric: Numeracy
Variables: All adults, Skill use work - Numeracy - How often - Calculating costs or budgets, Skill use work - Numeracy - How often - Use or calculate fractions or
percentages, Skill use work - Numeracy - How often - Use a calculator, Skill use work - Numeracy - How often - Prepare charts graphs or tables, Skill use work Numeracy - How often - Use simple algebra or formulas, Skill use work - Numeracy - How often - Use advanced math or statistics
Year/Study: PIAAC 2012
Select Report: Report 3
Export Reports
Table
Chart
Significance Test
Averages and percentages for numeracy, age by Skill use work - Numeracy - How often - Use or calculate fractions or percentages [GQ03C], year and jurisdiction: PIAAC 2012
Less than once a week but at least once
Never
Less than once a month
a month
At least once a week but not every day
Standard
Standard
Standard
Standard
Standard
Standard
Standard
Standard
Year/Study Jurisdiction Average
Error Percentage
Error Average
Error Percentage
Error Average
Error Percentage
Error Average
Error Percentage
Error Average
PIAAC 2012
OECD
Average
257
(0.3)
46
(0.2)
283
(0.6)
9
(0.1)
291
(0.6)
8
(0.1)
293
(0.5)
13
NOTE: The Numeracy scale ranges from 0 to 500. Detail may not sum to totals because of rounding. Some apparent differences between estimates may not be statistically significant.
SOURCE: Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of Adult Competencies (PIAAC), 2012.
About
Accessible Version
Data Explorer Technology
(0.1)
290
Analysis of the reports generated
on this run
• Firstly, analysing statistics on Numeracy skill-use
in a work context for the first three skills on the
IDE list of 6:
– Calculating costs or budgets
– Use, or calculation of, fractions or percentages
– Use of a calculator
• Then, in the next 2 tables, we’re analysing
statistics for the last three Numeracy skills
– Preparation of charts, graphs or tables
– Use of simple algebra or formulas
– Use of advanced math or statistics
NB: se is Standard Error
Average of Numeracy scale scores and Percentage of adults shown for
3 frequency options of 3 Numeracy skill use at work variables
Less than once a week but at
least once a month
Never
Average (se)
Percentage (se)
Average (se)
Percentage (se)
Every day
Average (se)
Percentage (se)
280 (0.4)
21 (0.1)
Frequency of Calculating costs or budgets in a work context
264 (0.3)
49 (0.2)
290 (0.7)
8 (0.1)
Frequency of Use, or calculation of, fractions or percentages in work context
257 (0.3)
46 (0.2)
291* (0.6)
8 (0.1)
290* (0.4)
23 (0.2)
8 (0.1)
283* (0.3)
37 (0.2)
Frequency of Use of a calculator in a work context
254 (0.4)
30 (0.2)
284* (0.7)
 For each Numeracy skill differences between all percentages and between most
averages are statistically significant (not those marked with *)
 Percentages of people in work who never use a calculator at work, or use one
everyday are markedly different from those for the other two Numeracy skills.
 These statistics for Numeracy skills may indicate that adults with the lowest skill are
choosing (or being chosen) to work in jobs that entail infrequent activity. This is in
keeping with research on the British Cohort Studies, which also showed that
people’s skills diminish with lack of use (Bynner & Parsons, 1998, 2000).
Average of Numeracy scale scores and Percentage of adults shown for
3 frequency options of the other 3 Numeracy skill use at work variables
Less than once a week but at
least once a month
Never
Average (se)
Percentage (se)
Average (se)
Percentage (se)
Every day
Average (se)
Percentage (se)
Frequency of Preparation of charts, graphs or tables in a work context
261 (0.3)
60 (0.2)
298 (0.5)
10 (0.1)
294 (0.8)
6 (0.1)
292 (0.5)
17 (0.2)
297 (1.4)
2 (0.0)
Frequency of Use of simple algebra or formulas in a work context
260 (0.3)
55 (0.2)
296 (0.7)
7 (0.1)
Frequency of Use of advanced math or statistics in a work context
270 (0.2)
86 (0.1)
306 (1.0)
3 (0.1)
 For these Numeracy skills differences between all percentages and between all
averages are statistically significant within each skill.
 For Use of advanced math or statistics the percentages of adults choosing
‘Never’ is highest, and for ‘Everyday’ lowest, frequency options – similar results
occur for the other two more complex skills above.
 Averages of Numeracy scale scores for Use of advanced math or statistics are
higher within most response categories which suggests that people with strong
numeracy skills have jobs that entail frequent numeracy activity.
Some concluding thoughts
• Overall, we found our exploration of the
PIAAC dataset stimulating and challenging
in equal measure
• While the scale and complexity of the
dataset may appear daunting we would
encourage you to undertake your own
exploration
• Our advice is: start with something simple
and take it from there
References
•
•
•
•
•
•
•
•
AIR PIAAC Team. (nd). What You Need to Consider Before Working with PIAAC
Data. Retrieved 3 May, 2015, from
http://static1.squarespace.com/static/51bb74b8e4b0139570ddf020/t/54e773d7e4b03
28389a39b36/1424454615327/WorkingWithPIAACData_022015.pdf
OECD. (2011). PIAAC Conceptual Framework of the Background Questionnaire Main
Survey. Paris: OECD.
http://www.oecd.org/site/piaac/PIAAC(2011_11)MS_BQ_ConceptualFramework_1%2
0Dec%202011.pdf
OECD. (2013). Technical Report of the Survey of Adult Skills (PIAAC). Paris: OECD.
www.oecd.org/site/piaac/_Technical%20Report_17OCT13.pdf
OECD, PIAAC, www.oecd.org/site/piaac/
OECD, PIAAC, Public Data & Analysis (includes link to PIAAC International Data
Explorer) http://www.oecd.org/site/piaac/publicdataandanalysis.htm
PIAAC Numeracy Expert Group. (2009). PIAAC Numeracy: A conceptual framework
OECD Education Working Paper (Vol. 35). Paris: OECD Publishing. http://www.oecdilibrary.org/education/piaac-numeracy-a-conceptual-framework_220337421165
PIAAC Gateway: Datasets and Tools http://piaacgateway.com/datasets/
Vignoles, A. (2007). The Use of Large Scale Data-sets in Educational Research.
Retrieved 10 July, 2015, from http://www.tlrp.org/capacity/rm/wt/vignoles