PISA for Development
Technical Strand 1:
Contextual questionnaires
Overview of PISA instruments
(constructs, indices and variables)
EDU/DCD
PISA for Development
Contextual questionnaires

Purpose of PISA contextual
questionnaires

How are they used?

How they are developed

What will PISA for Development seek to
do?

Questions for discussion
2
PISA for Development
Purpose of PISA contextual questionnaires
 Student and school questionnaires are part of the
core assessment design of PISA
 Countries may also opt for additional
questionnaires:
-
Parents, Teachers, Health, Time Use, ICT
familiarity, Career…
 Student questionnaires: ~ 30 minute* standalone document (students respond after test questions)
 School questionnaires: ~ 30 minute stand-alone
document (school authority responds before or on day of testing)
* Rotated questionnaires in 2012 to cover additional material
PISA for Development
Purpose of PISA contextual questionnaires
 Contextualise the performance results in
reading, mathematics and science*
 Allow analyses of equity issues across sub-groups
and populations
 Provide indications of possible policy
interventions
 Some information collected is used to scale
cognitive assessment data
 Country-specific questions can be included for
additional analysis
* Additional domains such as financial literacy and problem-solving
PISA for Development
Purpose of PISA contextual questionnaires
 Have been used in 74 countries/economies
 In more than 40 languages
 Publicly available
PISA 2009 Student Questionnaire
PISA 2012 Student Questionnaire
PISA 2012 School Questionnaire
PISA 2009 School Questionnaire
PISA Contextual Questionnaires
How they are used
Levels of an education system
This is the PISA
Analytic Framework
that guides data
collection and
analysis
Outputs and
Outcomes
impact of learning
Policy Levers
Antecedents
shape educational
outcomes
contextualise or
constrained policy
Individual
learner
Quality and
distribution of
knowledge & skills
Individual attitudes,
engagement and
behaviour
Socio-economic
background of
learners
Instructional
settings
Quality of
instructional delivery
Teaching, learning
practices and
classroom climate
Student learning,
teacher working
conditions
(other institutions)
Output and
performance of
institutions
The learning
environment at
school
Community
and school
characteristics
Country or
system
Social & economic
outcomes of
education
Structures, resource
allocation
and policies
National educ.,
social and
economic
context
Schools
PISA Contextual Questionnaires
How they are used
This is how the PISA
Analytic Framework
is operationalised for
data collection and
analysis
Main data collection
Student Questionnaire, School
Questionnaires and System Level*
Outcomes
Students
Classrooms
Schools
Country
Processes
Antecedents
Variables included in instruments:
• General recurring variables (all cycles)
• Domain recurring variables (main domain
every 3 years)
• Extension variables (specific cycles)
• System-level data
* Options include questionnaires geared for Teachers, Parents,
Health, Time Use, ICT familiarity, Career
PISA Contextual Questionnaires
System-level data collection
Standards and procedures for data collection in PISA
These are examples of
the types of information
collected at the system
level:
•
•
•
•
•
•
•
•
•
•
 Questionnaires for non-OECD economies
 Panel of experts to complete some data collection
Ratio: students / teaching staff
Spending /student
Sample of relevant areas to
Instruction time (by age)
address in developing
contexts
Teachers’ salaries
(by education, experience)
Evaluation & accountability
Educational stratification
• Within-country disparities
Decision-making
• Quality, supply and
distribution of educational
at different levels
materials and resources
School choice
• (Extra-)tuition
Changes in education
• Parental support
policy
• Contractual arrangements
Others (…)
of school teaching staff
8
PISA Contextual Questionnaires
School questionnaires
This is the type of information
collected through the school
questionnaires, that can then be
cross-referenced with other
sources
•
•
•
•
•
•
•
•
•
•
•
•
•
•
School resources, policies and practices
These are the constructs that will be
reviewed as part of the technical work of
PISA for Development
Grades served
Public-Private
Funding sources
Community type
Enrollment (boys, girls)
Average class size
Non-native speakers
Grouping/tracking
Teachers, by qualifications
Teacher shortages
Physical building limitations
Instructional materials
Teacher attitudes, expectations, relations
Computer/network/web access
•
•
•
•
•
•
•
•
•
•
•
•
Absenteeism, disruptions
Teacher morale
Student morale
Teaching environment
Time for special programmes
Non-native speaker
offerings
Admissions requirements
Assessment types
Uses of assessment
Assessment of teachers
Decision making
authority
External decision makers
9
PISA Contextual Questionnaires
Student questionnaires
Student attitudes, behaviours and approaches to learning
General and domain-related processes
These are examples
of the types of
non-cognitive
outcomes
on which PISA
collects
information
Sense of belonging at school
Student-teacher relations at school
Disciplinary climate in classes
Teacher support in subject-specific classes
Out-of-school activities (homework, etc...)
Self and domain-related cognitions
Anxiety towards “study subject”
Instrumental motivation to learn “study subject”
Interest in and enjoyment of “study subject”
Self-efficacy in “study subject”
Self-concept in “study subject”
Learning strategies
Others
Preference for competitive learning situations
Preference for co-operative learning situations
Control strategies
Perseverance
Health status, Time use
(…)
These are the
constructs
that will be
reviewed as
part of the
technical
work of PISA
for
Development
10
PISA Contextual Questionnaires
Examples of analysis
… How are questionnaire data used for
relevant analysis and comparative
policy insights?
(a few examples from PISA results)
11
550
500
2006
2003
Kyrgyzstan
600
2009
Indonesia
Albania
Georgia
Tamil Nadu-India
Himachal Pradesh-India
Reading performance
Chinese Taipei
Latvia
Lithuania
Russian Federation
Costa Rica
Serbia
Bulgaria
Uruguay
Romania
Venezuela-Miranda
Thailand
Malaysia
Colombia
Brazil
Montenegro
Mauritius
Jordan
Tunisia
Argentina
Kazakhstan
Panama
Peru
Azerbaijan
Shanghai-China
Hong Kong-China
Singapore
Liechtenstein
Macao-China
Croatia
Dubai (UAE)
Malta
United Arab Emirates
Trinidad and Tobago
Qatar
Examples of analysis
Range of performance among non-OECD countries
2000 PISA Results for
different years
Low-income economies ($1,025 or less)
Lower-middle-income economies ($1,026 to $4,035)
Upper-middle-income economies ($4,036 to $12,475)
High-income economies ($12,476 or more)
450
400
350
300
12
Mexico
Examples of analysis
PISA 2009 Results
School performance and socio-economic background
School performance and students’ socio-economic background within schools
Student performance and schools’ socio-economic background
Schools with a similar socio-economic profile
Score
600
Private school
Public school in rural area
Public school in urban area
550
OECD average
450
400
350
OECD average
Student performance
500
300
250
200
-3
Disadvantage
-2.5
-2
-1.5
-1
-0.5
0
PISA Index of socio-economic background
0.5
1
1.5
Advantage
2
Kyrgyzstan
Examples of analysis
PISA 2009 Results
School performance and socio-economic background
School performance and students’ socio-economic background within schools
Student performance and schools’ socio-economic background
Score
OECD average
600
550
OECD average
Student performance
500
Private school
Public school in rural area
Public school in urban area
450
400
350
300
250
200
-3
Disadvantage
-2.5
-2
-1.5
-1
-0.5
0
0.5
PISA Index of socio-economic background
1
1.5
2
Advantage
Policy
Examples
ofpractices
analysis
Policies and
R
R
System
E
School
Equity
Policy findings from contextual information
Learning climate
Policy briefs on
these issues can be
found in the PISA
in Focus series
Discipline

Teacher behaviour

Parental pressure

Teacher-student
relationships

Dealing with heterogeneity
Grade repetition




Prevalence of tracking
Expulsions



Ability grouping
(all subjects)



Standards /accountability
Nat. examination

Based
on the analysisof
of analysis
Examples
cognitive outcomes and
Policy insights using contextual
contextual information
information
Lessons from PISA
from successful
education systems
(equitable and improvers)
PISA Contextual Questionnaires
How they are developed

‘Crowd sourcing’ and collaboration
– PISA draws together leading expertise and institutions from
participating countries to develop instruments and methodologies…
… guided by governments on the basis of shared policy interests

Cross-national relevance and transferability of
policy experiences
– Emphasis on validity across cultures, languages and systems
– Frameworks built on well-structured conceptual understanding
of assessment areas and contextual factors

Continuous review and updating of assessment
frameworks (cycles)
– Guided by participating countries and various expert groups formed by
leading international experts in different areas related PISA
– Technical Advisory Group, Reading Expert Group, Mathematics EG,
Science EG, Questionnaire EG, and international contractors
17
PISA Contextual Questionnaires
How they are developed
PISA 2012 Questionnaires
• Student questionnaire: Rotated design (total of
65 questions – 25 core and 3 blocks of 27 questions)
• School questionnaire: 33 questions (5 additional
questions on financial education at school)
• Parental questionnaire: 25 questions
• ICT questionnaire: 12 questions
• Education careers: 14 questions
Linkages between cognitive and
non-cognitive outcomes
PISA Contextual Questionnaires
How they are developed
Purpose of the PISA Field Trial (for Main Study)
• Selection of cognitive test items to be included in the
PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF)
• Validation of psychometric equivalence of
translated instruments (and adaptations from source
versions)  issues addressed
• Identify constructs from contextual
questionnaires (indices and variables) that are
associated with performance for inclusion in the Main
Study
• Determine cross-national validity of the
questions in the questionnaire
• Technical standards ensure that data from participating
countries are internationally comparable
PISA Contextual Questionnaires
What will PISA for Development seek to do?
 Adapt and enhance contextual instruments to
“better fit” diverse contexts found in
developing countries while maintaining the
comparability with the main PISA scales and
international results
 Adapt existing constructs (and indices and
variables)
 Identify and introduce new constructs
(indices and variables)
 Confirm and validate enhancements
through field trials and main data collection in
participating countries
PISA Contextual Questionnaires
What will PISA for Development seek to do?
Examples of variables and indices
Student Questionnaire
Section 1: ABOUT YOU
Section 2: YOUR FAMILY AND YOUR HOME
Section 3: YOUR READING ACTIVITIES
Section 4: LEARNING TIME
Section 5: YOUR SCHOOL
Section 6: YOUR SCHOOL LESSONS AND SUBJECTS
Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS
Section 8: YOUR VIEWS ON <BROAD SCIENCE>
Section 9: CAREERS AND <BROAD SCIENCE>
Section 10: YOUR MATHEMATICS EXPERIENCES
PISA Contextual Questionnaires
Student questionnaires
Examples of variables and indices
Student Questionnaire
SECTION 2: YOUR FAMILY AND YOUR HOME
2009-ST08: Family structure
2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)
2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)
2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)
2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)
2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)
2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)
2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)
2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)
2009-ST17: Country of birth for student and parents (IMMIG)
2009-ST18: Age at arrival in country of test
2009-ST19: Language spoken at home
2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)
2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)
2009-ST22: Books in home (ESCS, HOMEPOS)
PISA Contextual Questionnaires
Student questionnaires – examples of indices
PISA index of Economic, Social and Cultural Status
(ESCS)
 Comparability across countries (not a countryspecific income indicator)
Highest occupational
status of parents
WEALTH
HISEI
CULTPOSS
Home possessions
HOMEPOS
HEDRES
Highest educational
level of parents
PARED
# of books
in home
PISA Contextual Questionnaires
Student questionnaires – examples of indices
PISA index of Economic, Social and Cultural Status
(ESCS)
Index of family wealth
a room of their own, a link to the Internet, a
dishwasher (treated as a country-specific
Index
cultural
possessions
item), aofDVD
player,
and three other
classic
literature,items
books
ofWEALTH
poetry
and
country-specific
(some
items
in
works
art their
(ST20)
ST20),ofand
responses on the number
Index
of home
educational
of cellular
phones,
televisions,resources
computers,
For
availability
of a desk, a quiet
carsexample,
and the rooms
with CULTPOSS
place
a computer
a bathtoorstudy,
shower
(ST21). that students can
use for schoolwork, educational software,
HEDRES
books to help students, technical
reference
books and dictionary (some items in ST20)
# of books
in home
PISA Contextual Questionnaires
Student questionnaires – examples of indices
Self-efficacy in science and mathematics
 2006-ST17: Self-efficacy in science
(SCIEEFF)
 2003-ST31: Self-efficacy in  Stays the same in PISA
2012
mathematics (MATHEFF)
Relationship between
constructs, indices and
variables (questions)
PISA Contextual Questionnaires
Student questionnaires – examples of indices
Self-efficacy in science
 2006-ST17: Self-efficacy in science
(SCIEEFF)
06-ST17
Q42 How easy do you think it would be for you to
perform the following tasks on your own?
(Please darken only one circle in each row.)
I could
do this
easily
a) Recognise the science
question that underlies a
newspaper report on a health
issue
1
I would
I could
struggle
do this to do this
with a bit on my I couldn’t
of effort
own
do this
2
3
4
PISA Contextual Questionnaires
Student questionnaires – examples of indices
Instrumental motivation to learn science

2006-ST35: Instrumental motivation to learn
science (INSTSCIE)
06-ST35
Q46 How much do you agree with the statements
below?
(Please darken only one circle in each row.)
Strongly
agree
a)
b)
Making an effort in my
science class(es) is worth it
because this will help me in
the work I want to do later
on
What I learn in my science
1
Agree
2
Disagre
e
3
Strongly
disagree
4
PISA for Development
How will we do this?
 Identify constructs in need of adaptation and
important constructs that are missing
 Obtain input and advice from experts in the field
(grounded in developing-country contexts)
 Obtain input and guidance from participating
countries
 Draw on existing work and empirical evidence
 Test the constructs (indices and variables) to ensure
comparability with PISA scales
PISA for Development
Technical Strand 1:
Contextual questionnaires
PISA for Development
Questions for discussion
 Are the intentions and rationale of this strand of work
clear and understood?
 Which components of the PISA questionnaires appear
to demonstrate the greatest need of adaptation?
 Where do you see the largest hurdle to overcome?
 What existing studies and evidence should contribute
to this strand of work?