OECD
World
Forum
onon
Key
Indicators
OECD
World
Forum
Key
Indicators
Statistics,
Statistics,Knowledge
Knowledgeand
andPolicy
Policy
Palermo,
10-13
November
2004
Palermo,
10-13
November
2004
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
1
Development, validation and
policy use of international,
comparative indicators on
education
Barry McGaw
Director for Education, OECD
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
2
OECD Indicators of Education Systems
• Framework for indicators
– current state of development
– longer-term strategic objectives
– comment on collaborative nature of the work
• Illustrations of development and use
– educational attainment
– quality of educational outcomes
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
3
Framework for OECD education statistics
Level
A
Individual
learner
Level
B
Instructional
settings
Level
C
Level
D
Schools, other
institutions
Country or
system
Domain 1
Domain 2
Domain 3
Outputs and
Outcomes
Policy Levers
impact of learning
shape educational
outcomes
Antecedents
contextualise or
constrain ed policy
Quality and
distribution of
Individ attitudes,
engagement and
Socio-economic
background of
knowledge & skills
behaviour
learners
Quality of
instructional
Teaching, learning Student learning,
teacher working
practices and
delivery
classroom climate
conditions
Output and
performance of
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
4
Well-established areas
Educational finance, students,
graduates,
Domain entrants,
1
Domain 2 personnel
Domain
• Well-established and build on national systems…
Level
A
Level
B
Level
C
Level
D
3
Outputs
anddefinitions,
Policy
– Internationally agreed
standards,
andLevers
methods. Antecedents
shape educational
contextualise or
– Joint OECD, UNESCO,
EUROSTAT collection.
Outcomes
outcomes
constrain ed policy
impact
of learning
…but need further
consolidation
– finance and graduates
needand
refined international
methodologySocio-economic
and more
Quality
Individ attitudes,
coherent national implementation.
Individual
distribution of
engagement and
background of
– improved meta-data.
learner
knowledge
& skills
…and an extension
of their
coverage, behaviour
e.g.
– Early childhood education.
Quality of
– International student mobility.
Instructional
settings
Schools, other
institutions
Country or
system
instructional
learners
Teaching, learning Student learning,
teacher working
practices and
delivery
classroom climate
conditions
Output and
performance of
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
5
Well-established areas
Level
A
Individual
learner
Level
B
Instructional
settings
Level
C
Level
D
Schools, other
institutions
Country or
system
Domain 1
Domain 2
Domain 3
Outputs and
Outcomes
Policy Levers
impact of learning
shape educational
outcomes
Antecedents
contextualise or
constrain ed policy
Quality and
distribution of
Individ attitudes,
engagement and
Socio-economic
background of
knowledge & skills
behaviour
learners
Quality of
instructional
Teaching, learning Student learning,
teacher working
practices and
delivery
classroom climate
conditions
Output and
performance of
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
6
Areas under development
Educational processes
Domain 1
Outputs and
Outcomes
statutory teacher salaries
• Annual collections
Level
A
Level
B
–
impact
of learning
– teachers’ working
conditions
– instruction time Quality and
Individual
distribution of
– pre-service training
learner collection
• Five-yearly
knowledge & skills
– distribution of decision-making
Quality of
responsibilities
Instructional
• Strategic objective instructional
settings
deliveryand
– survey on teachers, teaching
Level
D
Country or
system
Policy Levers
Antecedents
shape educational
outcomes
contextualise or
constrain ed policy
Individ attitudes,
engagement and
Socio-economic
background of
behaviour
learners
Teaching, learning Student learning,
teacher working
practices and
conditions
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
Output and
•
Quality
review
planned
Level Schools, other performance of
institutions
Domain 3
classroom climate
learning
C
Domain 2
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
7
Areas under development
Level
A
Individual
learner
Level
B
Instructional
settings
Level
C
Level
D
Schools, other
institutions
Country or
system
Domain 1
Domain 2
Domain 3
Outputs and
Outcomes
Policy Levers
impact of learning
shape educational
outcomes
Antecedents
contextualise or
constrain ed policy
Quality and
distribution of
Individ attitudes,
engagement and
Socio-economic
background of
knowledge & skills
behaviour
learners
Quality of
instructional
Teaching, learning Student learning,
teacher working
practices and
delivery
classroom climate
conditions
Output and
performance of
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
8
Areas under development
Level
A
Individual
learner
Level
B
Instructional
settings
Domain 1
Domain 2
Domain 3
Outputs and
Outcomes
Policy Levers
impact of learning
shape educational
outcomes
Antecedents
contextualise or
constrain ed policy
Quality and
distribution of
Individ attitudes,
engagement and
Socio-economic
background of
knowledge & skills
behaviour
learners
Quality of
instructional
delivery
Output and
other outcomes
Level Schools,
Educational
performance of
C
Level
D
institutions
– PISA established
institutions
Teaching, learning Student learning,
teacher working
practices and
classroom climate
conditions
The learning
environment at
Community
and school
school
characteristics
and policies
economic context
• 3-yearly surveys of 15-year-olds; benchmarking countries/regions
Social & economic
National educ,
Structures,
–
Longer-term
objective
Country or
outcomes of
social and
resource alloc
•
longitudinal
or
multi-cohort
to
identify
factors
related
to
success
system
education
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
9
Areas under development
Level
A
Individual
learner
Level
B
Instructional
settings
Level
C
Level
D
Schools, other
institutions
Country or
system
Domain 1
Domain 2
Domain 3
Outputs and
Outcomes
Policy Levers
impact of learning
shape educational
outcomes
Antecedents
contextualise or
constrain ed policy
Quality and
distribution of
Individ attitudes,
engagement and
Socio-economic
background of
knowledge & skills
behaviour
learners
Quality of
instructional
Teaching, learning Student learning,
teacher working
practices and
delivery
classroom climate
conditions
Output and
performance of
The learning
environment at
Community
and school
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
10
Areas under development
Domain 1
Domain 2
Domain 3
Outputs
and Policy Levers
Labour market and
social outcomes
Antecedents
institutions
contextualise or
Outcomes
– Evidence from
national labourshape
forceeducational
surveys
outcomes
constrain ed policy
of learning
• earningsimpact
and employment
• continuingQuality
education
andand training
Individ attitudes, Socio-economic
Individual
Level
… but to bedistribution
done to harmonise
them
of
engagement
and
background of
A
• consistent implementation of ISCED
learner
knowledge & skills
behaviour
learners
• international alignment of national taxonomies
Quality
of
Student learning,
Teaching, learning
– Strategic objective:
beyond
qualifications
to competencies
Instructional
Level
instructional
teacher working
• measure adult
competencies practices and
B
settings
• evaluate capacity
of education/training
to deliver competencies
delivery
conditions
classroom climate
• assess impact of competencies on economic & social outcomes
The learning
Community
Output and
…
currently
developing
a
detailed
strategy
Level Schools, other performance of
environment at
and school
C
Level
D
Country or
system
institutions
school
characteristics
Social & economic
outcomes of
Structures,
resource alloc
National educ,
social and
education
and policies
economic context
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
11
It is a collaborative effort…
• With countries
–
–
–
–
–
National Co-ordinators (34)
Technical Group (72)
Network on Educational Outcomes (37)
Network on Education and Socio-economic Outcomes (40)
Network on School Features and Processes (36)
• With other international organisations
–
–
–
–
EUROSTAT
UNESCO
World Bank
Euridyce
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
12
Data and indicators on educational attainment
with international comparisons that provoke
national policy debate.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
13
0
25
1st
15
10
5
55-64
extra for 45-54
extra for 35-44
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
Tu r
key
Aus
tri
Por a
t ug
al
Me
xi co
30
Ita l
y
St a
tes
No
rwa
y
De
nm
a
Ne
th e rk
rla n
ds
Ca
na d
a
Sw
ede
Sw
n
i
t
zer
Un
l
an
it ed
Ki n d
gd o
m
Aus
tral
Hu ia
ng a
ry
Ice
la n
d
Fi n
lan
d
Jap
an
Ge
r
Cze
ma
ny
ch
Re
pub
lic
Pol
an
Lu x
em d
bo u
rg
Ire l
a
Ne
w Z nd
e al
a nd
Fra
nce
Spa
in
Kor
ea
B
el
Slo
vak giu m
Re
p ub
l ic
Gre
ece
Un
it ed
Percentage
University-level attainment by age-group
40
35
2nd
3rd
20
22nd
13th
17th
extra for 25-34
Source: OECD (2004) Education at Glance, Table A3.3, p.71
14
With data on secondary school completion
suggesting further change in tertiary
completion rates…
-US has dropped from 1st to 10th
-South Korea has risen from 24th to 1st
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
15
Data and indicators on student learning
OECD/PISA
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
16
OECD/PISA
• Target populations
– 15-year-olds in school
• Assessments
– Reading literacy
• Using, interpreting and reflecting on written material
– Mathematical literacy
• Recognising a problem can be solved mathematically
• Mathematising problems to solve them
– Scientific Literacy
• Identifying scientific questions
• Knowing what counts as evidence for dealing with such questions
• Drawing evidence-based conclusions
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
17
PISA assessment schedule
2000
2003
2006
2009
2012
Reading
Reading
Reading
Reading
Reading
Maths
Maths
Maths
Maths
Maths
Science
Science
Science
Science
Science
Problem
solving
ICT literacy
?
?
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
18
PISA 2003 results will be released at
00.01 Paris time, Tuesday 7 December 2004.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
19
Participation in OECD/PISA
• PISA 2000
– 28 of then 29 OECD Member countries
– 15 Non-Members (some testing in 2002)
• PISA 2003
– 30 OECD Member countries
– 10 Non-Members
• PISA 2006
– 30 OECD Member countries
– 28 Non-Members
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
20
Comparisons of achievement levels.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
21
PISA 2000 Reading literacy means: OECD
600
550
500
450
400
K or
ea
Kin
gdo
m
Jap
an
Sw
ede
n
A us
tria
B el
gi u
m
Ice
l an
d
Nor
way
Fra
nce
Uni
ted
St a
tes
Den
ma
rk
Sw
itze
rlan
d
S
pai
Cze
n
ch
Rep
ubl
ic
It al
y
Ge
r ma
ny
Hun
gar
y
P ol
and
Gr e
ece
P or
tug
al
Lux
em
bou
rg
Me
xico
ted
Uni
nd
Ir el
a
tral
ia
d
A us
lan
Z ea
ada
New
Can
Fin
l an
d
350
Source: OECD (2001) Knowledge and skills for life: First results from PISA 2000, Fig. 2.4, p.53
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
22
325
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
Per
u
Fin
l an
d
Can
ada
New
Zea
la n
d
Aus
tral
ia
Irel
Hon
an d
gK
ong
- Ch
ina
Kor
Uni
ea
ted
Ki n
gdo
m
J ap
an
Sw
ed e
n
Aus
tria
Bel
giu
m
Ic e
land
Nor
way
Fra
nce
Uni
ted
Sta
te s
Den
ma
rk
Sw
i tze
rl an
d
Spa
Cze
in
ch R
epu
bli c
Ital
y
Ge
rm a
Li e
n
y
chte
nste
in
Hun
gar
y
Pol
and
G re
e ce
Rus
Por
tug
si an
al
Fed
era
tion
Lat
v ia
Is ra
el
Lux
em
bou
rg
Tha
il an
d
Bul
gar
ia
Rom
an i
a
Me
x ic o
Arg
ent
i na
Chi
le
Bra
FYR
z il
Ma
c ed
oni
a
Ind
on e
si a
Al b
ani
a
PISA 2000 Reading literacy means: All
600
575
550
525
500
475
450
425
400
375
350
Source: OECD, UNESCO (2003) Literacy skills for the world of tomorrow, Fig. 2.5, p.76
23
Fi n
lan
d
Ca
na d
a
Ne
wZ
e
Aus ala nd
tral
ia
Ire l
and
Kor
ea
Un
it ed
Ki n
Jap
gd o
an
m
Sw
ede
n
Aus
tria
Bel
giu
m
Ice
la n
d
No
rw a
y
Fra
nce
Un
it ed
St a
tes
De
nm
ark
Sw
itze
rl
Spa a nd
in
Cze
ch
Re
pub
Ita l
lic
y
Ge
rm a
ny
Hu
ng a
ry
Pol
and
G re
ece
Por
t ug
al
Lu x
em
bo u
Me
rg
xi co
% at each reading proficiency level
0%
Level 5
10%
20%
30%
40%
50%
60%
70%
Level 4
US is only average, on
average, but has
Korea has a high average Mexico has almost 45% below
relatively high proportion
performance but few atLevel Level
2 – and less than 60% of
3
at highest level.
the highest level.
15-year-olds in school.
Korea has less than
6% below Level 2.
Level 2
80%
Level 1
90%
100%
Below Level 1
Source: OECD (2001) Knowledge and skills for life:, Appendix B1, Table 2.1a, p.246
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
24
Comparisons of equity in outcomes.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
25
Social background vs reading performance
Student performance in reading
High
Low
Each 20,000
students in the
OECD area are
represented by one
dot in this diagram
There are disadvantaged students
who do well and advantaged ones
who do poorly BUT there is a strong
tendency for social advantage to be
associated with better performance
PISA Index of social background
Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
Social
Advantage
26
Social background and reading performance
High
Finland
Korea
United States
Spain
Germany
Student performance in reading
600
Countries differ markedly in
the equity of their outcomes.
550
500
450
This gap is in the order of
3 years of schooling.
400
350
300
Low
-2
-1.5
-1
-0.5
0
0.5
PISA Index of social background
Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
1
1.5
Social 2
Advantage
27
This kind of analysis is possible only with individual data.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
28
Reading mean vs social background
560
High quality
Low equity
Finland
High quality
High equity
540
Mean reading literacy .
Australia
United Kingdom
520
New Zealand
500
480
France
Switzerland
Czech Republic
Hungary
Germany
Korea
Japan
Sweden
Belgium
United States
Canada
Ireland
Austria
Norway
Denmark
Iceland
Spain
Italy
Poland
Greece
Portugal
460
440
420
Luxembourg
Low quality
Low equity
-25
-20
Low quality
High equity
Mexico
-15
-10
-5
0
5
10
15
20
25
Social equity
Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
29
Germany commissioned a multi-lateral study.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
30
Germany’s selection of countries to study
560
High quality
Low equity
Finland
High quality
High equity
540
Mean reading literacy .
Australia
United Kingdom
520
New Zealand
500
480
France
Switzerland
Czech Republic
Hungary
Germany
Korea
Japan
Sweden
Belgium
United States
Canada
Ireland
Austria
Norway
Denmark
Iceland
Spain
Italy
Poland
Greece
Portugal
460
440
420
Luxembourg
Low quality
Low equity
-25
-20
Low quality
High equity
Mexico
-15
-10
-5
0
5
10
15
20
25
Social equity
Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
31
Conclusions from German study
• Publication
– What makes school systems perform?
Seeing school systems through the prism of PISA
• Conclusions
– building a culture of achievement
• social and cultural disparities are central to innovation strategies
• quality and equity are addressed at the same time
– integrated not differentiated system
• with good quality support to students, teachers, schools
– shifting responsibility downward from central authority
• BUT with system monitoring of performance
• AND, PROBABLY, publication of results at the school level
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
32
Mean literacy by expenditure per student
Countries above
the line achieve
more than could
be expected,
given how much
they spend.
550
525
Finland
Ireland
Korea
Australia
UK
Belgium
France
500
Czech Republic
Hungary
Greece
Poland
475
450
425
Mexico
400
10,000
Japan
Sweden
Spain
Germany
Norway
Austria
USA
Denmark
Switzerland
Italy
Portugal
Countries below the line achieve
less than could be expected, given Denmark is among the high
spenders but is only average
how much they spend.
in achievement.
20,000
30,000
40,000
50,000
60,000
70,000
80,000
Cumulative expenditure per student to age 15 ($US equivalent PPP)
Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
33
Denmark commissioned OECD to undertake a comparative
policy review of Denmark, Finland, England and Alberta.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
34
Conclusions from review of Denmark
• Publication
– Quality and equity of schooling outcomes in Denmark
• Recommendations (35 in all)
– Learning standards, evaluation of students, schools
• clarify and raise expectations of student learning
• establish school improvement teams
• develop strategy for estimating value added by schools
– Roles and competencies of school leaders
• mandatory courses before appointment
• mentoring during full first year
– Professional development of teachers
• link initial and in-service education and research
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
35
Thank you.
OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004
36