1
1
Organisation for Economic Cooperation and Development (OECD)
What light does PISA shed on
student learning?
Selected results from PISA 2003
Paris
2 February 2005
Andreas Schleicher
Head, Indicators and Analysis Division
Directorate for Education
2
2
3
3
In the dark, all schools and education systems look the same…
But with a little light….
4
4
But with a little light….
…important differences become apparent….
5
5
Outline for today
1. Learning for tomorrow’s world

PISA – a new framework for evaluating the
preparedness of youths for the knowledge society
2. Where we are today

What PISA 2003 shows about the performance
of education systems
3. Where we can be

What the best performing countries show can be
achieved
4. How we can get there

Levers for policy that emerge from international
comparisons, with focus on assessment methods
6
Learning for Tomorrow’s World
PISA – a new framework for evaluating the preparedness
of youths for the knowledge society
PISA - The most comprehensive international
assessment of student competencies
7
7

Geographic and economic coverage



275,000 15-year-old students randomly sampled
43 countries in 2000, 41 and 2003, 59 in 2006
Subject matter coverage


Mathematics, Science, Reading, Cross-curricular competencies
Volume of questions
– 3½ hours of mathematics assessment
– 1 hour for each of reading, science and problem solving

Each student
– 2 hours on paper-and-pencil tasks (subset of all questions)
– ½ hour for questionnaire on background, learning habits, learning
environment, engagement and motivation



Variety of task formats

Open-constructed responses, multiple-choice

A total of 7 hours of assessment material
Depths
Target population: 15-year-olds in school
8
8
Deciding what to assess...
looking back at what students were
expected to have learned
OR
looking ahead to what they can do with
what they have learned.
For PISA, the OECD countries chose the latter.
9
9
Knowledge based society
What do we expect of key competencies?

Competency

Applying psycho-social resources…
– Cognitive, motivational, ethical, volitional, social
… to successfully meet complex demands in varied contexts

Key competencies






Apply to multiple areas of life
Lead to important and valued individual and social outcomes
Imply the development of a higher level of reflectivity and
mental complexity
Build on a combination of cognitive and non-cognitive
psychological resources
Can be learned – and taught
Key competencies operate as constellations
10
10
Three broad categories of
key competencies
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
PISA concept
of literacy
of technologies
Acting
Interacting
in
Accessing,
managing,
integrating
autonomously
diverse groups
and evaluating
written information
e.g.
e.g.
in order
to develop
andwithin
potential,
Acting
the bigger picture
Relating
wellones
to knowledge
others
and to participate in, and contribute to, society
Co-operating, working in teams Learning strategies
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
11
11
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
Reading literacy
(focus in 2000)
of technologies
Acting
Interacting in
Using, diverse
interpreting
autonomously
groups and reflecting
e.g.
on written
material
e.g.
Acting within the bigger picture
Relating well to others
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
12
12
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
Capitalising on the potential
Scientific literacy
(focus in 2006)
of technologies
Acting
Interacting in
Using scientific knowledge, identifying scientific
autonomously
diverse groups
questions, and drawinge.g.evidence-based conclusions
to
e.g.
Acting the
within
the bigger
picture
understand and
make well
decisions
about
natural
world
Relating
to others
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
13
13
Using “tools”
interactively to
engage with the world
To analyse, compare, contrast, ande.g.
evaluate
Using language, symbols and texts
Toinformation
think imaginatively
Interacting with
on the potential
MathematicalCapitalising
literacy
(focus in 2003)
of technologies
Acting
Interacting
in
Emphasis is on
mathematical
knowledge put into
autonomously
diverse groups
functional use in a multitude
of different e.g.
situations
e.g.
Acting within the
bigger picture
well to
others
in varied,Relating
reflective
and
insight-based
ways
Co-operating, working in Forming
teams and conducting life plans
Taking responsibility and
Managing and
resolving situations
conflicts
To apply knowledge
in real-life
understanding rights and limits
To communicate thoughts and ideas effectively
14
14
Mathematical literacy in PISA
The real world
The mathematical World
as an instrument to understand the
real world
Making the problem amenable
to mathematical treatment
A model of reality
Understanding,
structuring and
simplifying the
situation
A mathematical
model
Using relevant
mathematical
tools to solve
the problem
A real situation
Validating
the results
Mathematical
results
Real results
Interpreting
the mathematical results
15
15
Using “tools”
interactively to
engage with the world
PISA assessment of
– Problem-solving competencies
• Using cognitive processes to resolve real
situations where the solution path is not
immediately
the competencies
Acting
Interacting
in obvious and where
required
are not within autonomously
a single discipline.
diverse
groups
• PISA self-reports on:
–
–
–
–
Dispositions to learning
Learning strategies
Engagement with school
Self-concept
Development of assessments
16
16


Frameworks by international experts
Assessment materials






submitted by countries
developed by research consortium
screened for cultural bias
– by countries
– by an international expert panel
– items with prima facie cultural bias removed at this stage
internationally validated translations
trialled to check items working consistently in all countries
Final tests


items shown in trial to be culturally biased removed
best items chosen for final tests
– balanced to reflect framework
– range of difficulties
– range of item types (constructed response, multiple choice)
17
Securing an equitable distribution of
learning opportunities
Measured by the impact students’ and schools’ socioeconomic background has on performance – not merely by
the distribution of learning outcomes
18
18
High mathematics performance
Hong Kong-China
High average performance
Top-performers
Large socio-economic
disparities
Finland remained first in Liechtenstein
reading
and since
There
is more
to 2000
this
moved further in math andMacao-China
than
national income
science…

Countries
with higher
… and is now on a par with the Iceland
national
income
and better
East Asian
countries
that
educated
adult populations
were previously
unmatched in
Strong
sociotend
to perform
better…
math
and science
Ireland
economic impact on
but there
are exceptions.
… Also
the
Netherlands
is
student
performance
Progress
among the top-performers in
Poland

Other countries with
math
improvements
at least
… though not ininreading
andtwo Latvia
assessment
science. areas were
Belgium,
Czech
Republic

As
is the the
Flemish
Community
andBelgium
Germany
of

Russian Federation
… In Belgium and Germany it
Italy
wasLow
theaverage
top performers
who
performance
drove improvements.
Large socio-economic disparities
Average performance
High average performance
Finland
A
widening
gap
Korea
of 15-year-olds
in
540
High
social
equity
Netherlands
More improvement at the top
mathematics
Japan

Canada of the scale has widened the
Belgium gap between the top and
Switzerland
Australiabottom performers in the
New Zealand
520
OECD.
Czech Republic
Denmark
France
Sweden
Austria
Socially equitable
Germany
distribution of
500
Slovak Republic
learning opportunities
Norway
Luxembourg
Hungary
Progress
Spain  Poland
United States
480
raised it’s overall
performance in all four
assessment areas
… thanks to big
improvements among
Portugal
lower-performing
Low average performance
460
students in the wake of a
High
social
equity
major
reform
in 1999.
Low mathematicsGreece
performance
19
19
High mathematics performance
Hong Kong-China
Durchschnittliche
High average performance
High average performance
Finland
Schülerleistungen
im
Korea
Differences
in
socio-economic
540
Large socio-economic disparities Netherlands
High social equity
Liechtenstein
background pose
major challenges forBereich Mathematik
Japan
Canada
education systems
Belgium
Switzerland
Australia
Students whose parents have better-paid
New
Zealand
520
jobs, are better educated or have
more
Czech Republic
Iceland
“cultural” possessions in their homes tend
to
Denmark
France
perform better…
Sweden
Austria
socioSocially equitable
… ButStrong
the performance
advantage varies
Ireland
Germany

economic impact on
distribution of
500
–
Australia,
Canada,
Finland,
Iceland
and
Japan
Slovak Republic
student performance
learning opportunities
Norway
provide examples showing that it is possible to
Luxembourg
combine
quality and equity
Hungary
Poland
– In contrast, results for Belgium, Germany,
Spain
Latvia
United States
Hungary, he Slovak Republic and Turkey 480
reveal
large socio-economic inequalities in the
distribution of learning opportunities .
Portugal
Low average performance
Large socio-economic disparities
460
Russian Federation
Italy
Low average performance
High social equity
Low mathematics performance
Greece
20
Ensuring consistent performance
standards across schools
Between and within-school variation in performance
Iceland
Finland
Norway
Sweden
Poland
Denmark
Ireland
Canada
Spain
New Zealand
Australia
United States
Mexico
Portugal
Luxembourg
Switzerland
Greece
Slovak Republic
Korea
Czech Republic
Netherlands
Austria
Germany
Italy
Belgium
Japan
Hungary
Turkey
21
21
Is it all innate ability?
Variation in student performance
140
120
100
80
60
40
20
0
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
Is it all innate ability?
22
22
Variation in student performance in mathematics
In some countries, parents can rely on high
and ofconsistent standards across schools
Variation
100
80
performance within

In Canada,
schools
Denmark, Finland, Iceland and Sweden
average student performance is high…
… and largely unrelated to the individual schools in which
students are enrolled.
60
40
20
0
-20
1
11
Iceland
Norway
Sweden
Poland
Ireland
Canada
Spain
New Zealand
Australia
United States
Mexico
Portugal
Luxembourg
Switzerland
Greece
Slovak Republic
Korea
Czech Republic
Netherlands
Austria
Germany
Italy
Belgium
Japan
Hungary
Turkey
14
12
5
-80
Denmark
-60
Finland
Variation of
performance between
schools
-40
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
23
Bridging the gender gap
Performance, attitudes and motivation
24
24
Gender differences


In reading, girls are far ahead

In all countries, girls significantly outperform boys in reading

In most countries, boys outperform girls
In mathematics, boys tend to be somewhat ahead
… but mostly by modest amounts…
… and mainly because boys are overrepresented among topperformers while boys and girls tend to be equally represented in
the “at risk” group
– Within classrooms and schools, the gender gap is often larger

Strong problem-solving performance for girls suggests…
… that it is not the cognitive processes underlying mathematics that
give boys an advantage…
… but the context in which mathematics appears in school

Gender differences in interest and attitudes towards
mathematics are significantly greater than the observed
performance gap
– Girls report much lower intrinsic (though not instrumental)
motivation in mathematics, more negative attitudes and much
greater anxiety with mathematics…
… and this may well contribute to the significant gender difference
in educational and occupational pathways in mathematics-related
subjects
25
Creating strong foundations for
lifelong learning
Performance, attitudes and motivation
26
26
Student approaches to learning

The ability to manage one’s learning is both an
important outcome of education and a contributor to
student literacy skills at school


Different aspects of students’ learning approaches
are closely related



Learning strategies, motivation, self-related beliefs,
preferred learning styles
Well-motivated and self-confident students tend to invest in
effective learning strategies and this contributes to their
literacy skills
Immigrant students tend to be weaker performers
…
but they do not have weaker characteristics as learners
Boys and girls each have distinctive strengths and
weaknesses as learners


Girls stronger in relation to motivation and self-confidence
in reading
Boys believing more than girls in their own efficacy as
learners and in their mathematical abilities
27
27
Control strategies in mathematics
OECD average
Austria
Japan
Percentage of students
0
20
40
60
80
100
When I study for a mathematics test, I try to work out what
are the most important parts to learn.
When I study mathematics, I make myself check to see if I
remember the work I have already done.
When I study mathematics, I try to figure out which concepts
I still have not understood properly.
When I cannot understand something in mathematics, I always
search for more information to clarify the problem.
When I study mathematics, I start by working out exactly
what I need to learn.
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.9, p.375 and Figure 3.9, p.143.
28
28 Elaboration strategies in mathematics
OECD average
Poland
Germany
Percentage of students
0
20
40
60
80
100
When I am solving mathematics problems, I often think of new
ways to get the answer.
I think how the mathematics I have learned can be used in
everyday life.
I try to understand new concepts in mathematics by relating
them to things I already know.
When I am solving a mathematics problem, I often think about
how the solution might be applied to other interesting
questions.
When learning mathematics, I try to relate the work to things
I have learnt in other subjects.
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.11, p.377 and Figure 3.11, p.146.
Combined explanatory power of student learning characteristics
29
29 on mathematics performance and control strategies
Percentage of variance in student mathematics performance
that is explained by the combined effect of
Percentage of variance in student use of control strategies
that is explained by the combined effect of
-interest in and enjoyment of mathematics
-interest in and enjoyment of mathematics
-anxiety in mathematics
-control strategies
40
50
30
-anxiety in mathematics
20
10
0
%
0
10
20
30
40
50
Norway
Denmark
Poland
Sweden
Finland
Korea
New Zealand
Iceland
Canada
Slovak Republic
Czech Republic
United States
Portugal
Australia
Ireland
Greece
Turkey
Uruguay
Mexico
Germany
Switzerland
Luxembourg
Spain
Hungary
Austria
France
Italy
Japan
Belgium
Netherlands
OECD Average
United Kingdom
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 3.13, p.149.
30
How can we get there?
Levers for policy that emerge from international
comparisons
31
31
Sympathy doesn’t raise standards –
aspiration does

In many of the best performing countries


National research teams report a strong
“culture of performance”
– Which drives students, parents, teachers and
the educational administration to high
performance standards
PISA shows that students perceived a high
degree of teacher support
– Which should not be simply equated with
“achievement press”
32
32
Governance of the school system

In many of the best performing countries


Decentralised decision-making is combined with
devices to ensure a fair distribution of
substantive educational opportunities
The provision of standards and curricula at
national/subnational levels is combined with
advanced evaluation systems
– That are implemented by professional agencies

Process-oriented assessments and/or
centralised final examinations are complimented
with individual reports and feed-back
mechanisms on student learning progress
High
Mathematics performance
33
33
Hong Kong-China
Korea
Netherlands
Liechtenstein
High performance
Belgium
Low social equity
Finland
540
High
performance
Japan
Switzerland High
Australia
New Zealand
Canada
social equity
Macao-China
520
Czech Republic
Iceland
Denmark
France
Sweden
– Only
8% of schools
involved in appointing
Austria
Moderate impact
(OECD 64%)
Ireland
Germanyteachers
Some notes on Italy:
Strong impact of
social background on
performance
Hungary
Low performance
Low social equity
of
social background on
Slovak
Republic
– Only
2% of schools involved inperformance
determining
Norway
teacherLuxembourg
salary increases (OECD 38%)
Poland
– All schools involved
in establishing student
Spain
Latvia
United States
assessment
policies
(OECD 85%)
Low performance
– 84% of schools involved
in determining course
content (OECD
67%)
High social equity
Russian Federation
30
20
480
Italy
Portugal
Formulating the school budget and
deciding on budget allocations in schools
High degree of autonomy
Low degree of autonomy
500
460
Greece
Low
Performance
10
0 440
34
34
Organisation of instruction

In many of the best performing countries

Schools and teachers have explicit strategies
and approaches for teaching heterogeneous
groups of learners
– A high degree of individualised learning processes
– Disparities related to socio-economic factors and
migration are recognised as major challenges


Students are offered a variety of extracurricular activities
Schools offer differentiated support structures
for students
– E.g. school psychologists or career counsellors

Institutional differentiation is introduced, if at
all, at later stages
– Integrated approaches also contributed to reducing
the impact of students socio-economic background on
outcomes
High
Mathematics performance
35
35
Hong Kong-China
Finland
Korea
Netherlands
Liechtenstein
High performance
Belgium
Low social equity
Strong impact of
social background on
performance
High
performance
Japan
Switzerland High
Australia
New Zealand
Germany
Canada
social equity
Macao-China
520
Czech Republic
Denmark
France
Sweden
Austria
Ireland
Slovak Republic
Hungary
Poland
Iceland
Moderate impact of
social background on
performance
Norway
Luxembourg
United States
Spain
Latvia
Low performance
Low performance
Low social equity
540
Portugal
High
Russian Federation
Italy
480
social equity
460
Early selection
and institutional differentiation
High degree of stratification
Greece
Low degree of stratification
30
500
20
Low
Performance
10
0 440
37
37
Teacher support in mathematics
Students’ views
OECD average
United States
Germany
Percentage of students
0
20
40
60
80
100
The teacher shows an interest in every
student's learning.
The teacher gives extra help when students
need it.
The teacher helps students with their learning.
The teacher continues teaching until the
students understand.
The teacher gives students an opportunity to
express opinions.
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.
38
38
Student-related factors affecting school climate
Principals’ views
Percentage of students
OECD average
Korea
0
20
40
60
80
100
Student absenteeism.
Disruption of classes by students.
Students skipping classes.
Students lacking respect for teachers.
Student use of alcohol or illegal drugs.
Students intimidating or bullying other students.
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.2a, p.406 and Figure 5.2, p.216.
39
39
Teacher-related factors affecting school climate
Principals’ views
Percentage of students
OECD average
Denmark
0
20
40
60
80
100
Teachers low expectation of students.
Poor student-teacher relations.
Teachers not meeting individual students'
needs.
Teacher absenteeism.
Staff resisting change.
Teachers being too strict with students.
Students not being encouraged to achieve their
full potential.
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.4a, p.410 and Figure 5.4, p.220.
45
45
Assessment methods in the best-performing PISA countries
Standardised tests
Never(%)
1 to 2 times a year(%)
3 to 5 times a year(%)
Monthly(%)
More once a month(%)
100%
80%
60%
40%
20%
Zealand
New
Australia
Switzerland
Belgium
Canada
Japan
Netherlands
Korea
Finland
0%
46
46
Assessment methods in the top performing PISA countries
Teacher developed tests
Never(%)
1 to 2 times a year(%)
3 to 5 times a year(%)
Monthly(%)
More once a month(%)
100%
80%
60%
40%
20%
Zealand
New
Australia
Switzerland
Belgium
Canada
Japan
Netherlands
Korea
Finland
0%
47
47
Assessment methods in the top performing PISA countries
Judgemental ratings
Never(%)
1 to 2 times a year(%)
3 to 5 times a year(%)
Monthly(%)
More once a month(%)
100%
80%
60%
40%
20%
Zealand
New
Australia
Switzerland
Belgium
Canada
Japan
Netherlands
Korea
Finland
0%
48
48
Assessment methods in the top performing PISA countries
Student portfolios
Never(%)
1 to 2 times a year(%)
3 to 5 times a year(%)
Monthly(%)
More once a month(%)
100%
80%
60%
40%
20%
New Zealand
Australia
Switzerland
Belgium
Canada
Japan
Netherlands
Korea
Finland
0%
49
49
Assessment methods in the top performing PISA countries
Student assignements/projects/homework
Never(%)
1 to 2 times a year(%)
3 to 5 times a year(%)
Monthly(%)
More once a month(%)
100%
80%
60%
40%
20%
Zealand
New
Australia
Switzerland
Belgium
Canada
Japan
Netherlands
Korea
Finland
0%
50
50
One challenge – different approaches
The future of education
systems needs to be
“knowledge rich”
Informed professional
judgement, the teacher as
a “knowledge worker”
Informed
prescription
National
prescription
Professional
judgement
Uninformed
prescription, teachers
implement curricula
Uninformed professional
judgement
The tradition of
education systems has
been “knowledge poor”
51
51
Further information

www.pisa.oecd.org
– All national and international publications
– The complete micro-level database

email: pisa@oecd.org

Andreas.Schleicher@OECD.org
… and remember:
Without data, you are just another person
with an opinion
52
PISA country participation
OECD countries participating from PISA 2000
OECD countries participating from PISA from 2003
OECD partner countries participating from PISA 2000
OECD partner countries participating from PISA 2003
OECD partner countries participating from PISA 2006