The characteristics of high-performing education systems: Lessons from international comparisons Barry McGaw Melbourne Graduate School of Education, University of Melbourne Revolutions, Revelations & Reality PDN School Leaders’ Conference 2009 Gold Coast, 7 August 2009 There is a rising demand for high-level skills Changed demand for skills in the US 65 60 The dilemma for schools: The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource. 55 50 40 1960 ABN 34 117 491 228 mcgaw group pty ltd 45 1970 1980 Routine manual Nonroutine manual Nonroutine analytic Nonroutine interactive 1990 2002 Routine cognitive Autor, D., Levy, F. and Murnane, R. J., (2003) The skill content of recent technical change, Quarterly Journal of Economics 118, M.I.T. Press, Cambridge, pp.1279-1334. Levy, F. and Murnane, R.J. (2006), “How Computerized Work and Globalization Shape Human Skill Demands”, working paper, available at: http://web.mit.edu/flevy/www/computers_offshoring_and_skills.pdf. 3 The storyline so far… ABN 34 117 491 228 mcgaw group pty ltd There is a growing labour market demand for higher level skills. 4 How good is Australian school education? Using international comparisons Making international comparisons of achievement requires decisions about... whom to assess. ABN 34 117 491 228 mcgaw group pty ltd what to assess, 7 Deciding what to assess... looking back at what they were expected to have learned OECD/PISA chose the latter. IEA studies have chosen the former. ABN 34 117 491 228 mcgaw group pty ltd OR looking ahead to what they can do with what they have learned. NAEP intention was the latter (as for PISA); but it became more curriculum-oriented (as for IEA) 8 PISA defines science performance in terms of a student’s: Scientific knowledge and use/extrapolation of that knowledge to… …identify scientific issues, …explain scientific phenomena, and …draw evidence-based conclusions about sciencerelated issues Awareness of how science and technology shape our material, intellectual and cultural environments Willingness to engage with science-related issues ABN 34 117 491 228 mcgaw group pty ltd Understanding of the characteristic features of science as a form of human knowledge and enquiry For example When reading about a health issue, can students separate scientific from non-scientific aspects of the text, apply knowledge and justify personal decisions? 9 PISA defines science performance in terms of a student’s: Scientific knowledge and use/extrapolation of that knowledge to… …identify scientific issues, …explain scientific phenomena, and …draw evidence-based conclusions about sciencerelated issues Awareness of how science and technology shape our material, intellectual and cultural environments Willingness to engage with science-related issues ABN 34 117 491 228 mcgaw group pty ltd Understanding of the characteristic features of science as a form of human knowledge and enquiry For example Can students distinguish between evidence-based explanations and personal opinions? 10 PISA defines science performance in terms of a student’s: Scientific knowledge and use/extrapolation of that knowledge to… …identify scientific issues, …explain scientific phenomena, and …draw evidence-based conclusions about sciencerelated issues Awareness of how science and technology shape our material, intellectual and cultural environments Willingness to engage with science-related issues ABN 34 117 491 228 mcgaw group pty ltd Understanding of the characteristic features of science as a form of human knowledge and enquiry For example Can individuals recognise and explain the role of technologies as they influence a nation’s economy? Or are they aware of environmental changes and the effects of11 PISA defines science performance in terms of a student’s: Scientific knowledge and use/extrapolation of that knowledge to… …identify scientific issues, …explain scientific phenomena, and …draw evidence-based conclusions about sciencerelated issues Awareness of how science and technology shape our material, intellectual and cultural environments Willingness to engage with science-related issues ABN 34 117 491 228 mcgaw group pty ltd Understanding of the characteristic features of science as a form of human knowledge and enquiry Interest in science, support for scientific enquiry, responsibility for the environment This addresses the value students place on science, both in terms of topics and in terms of the scientific approach to understanding the world and solving problems. 12 Mimi and Dean wondered which sunscreen product provides the best protection for their skin. Sunscreen products have a Sun Protection Factor (SPF) that shows how well each product absorbs the ultraviolet radiation component of sunlight. A high SPF sunscreen protects skin for longer than a low SPF sunscreen. Mimi thought of a way to compare some different sunscreen products. She and Dean collected the following: • two sheets of clear plastic that do not absorb sunlight; • one sheet of light-sensitive paper; • mineral oil (M) and a cream containing zinc oxide (ZnO); and • four different sunscreens that they called S1, S2, S3, and S4. ABN 34 117 491 228 mcgaw group pty ltd Example question: Sunscreens 13 Example question: Sunscreens ABN 34 117 491 228 mcgaw group pty ltd Mimi and Dean included mineral oil because it lets most of the sunlight through, and zinc oxide because it almost completely blocks sunlight. Dean placed a drop of each substance inside a circle marked on one sheet of plastic, then put the second plastic sheet over the top. He placed a large book on top of both sheets and pressed down. Mimi then put the plastic sheets on top of the sheet of light-sensitive paper. Light-sensitive paper changes from dark grey to white (or very light grey), depending on how long it is exposed to sunlight. Finally, Dean placed the sheets in a sunny place. 14 ABN 34 117 491 228 mcgaw group pty ltd Question: Which one of these statements is a scientific description of the role of the mineral oil and the zinc oxide in comparing the effectiveness of the sunscreens? A. Mineral oil and zinc oxide are both factors being tested. B. Mineral oil is a factor being tested and zinc oxide is a reference substance. C. Mineral oil is a reference substance and zinc oxide is a factor being tested. D. Mineral oil and zinc oxide are both reference substances. 15 Question: The light-sensitive paper is a dark grey and fades to a lighter grey when it is exposed to some sunlight, and to white when exposed to a lot of sunlight. Which one of these diagrams shows a pattern that might occur? Explain why you chose it. A. ZnO has blocked the sunlight as it should and M has let it through. I chose A because the mineral oil needs to be the lightest shade while the zinc oxide is the darkest. Partial Credit: A. Gives a correct explanation for either the ZnO spot or the M spot, but not both. A. Mineral oil provides the lowest resistance against UVL. So with other substances the paper would not be white. A. Zinc oxide absorbs practically all rays and the diagram shows this. A because ZnO blocks the light and M absorbs it. ABN 34 117 491 228 mcgaw group pty ltd Full Credit: A. With explanation that the ZnO spot has stayed dark grey (because it blocks sunlight) and the M spot has gone white (because mineral oil absorbs very little sunlight). A B C D 16 Deciding whom to assess... grade-based sample OR age-based sample differences in school starting age differences in grade retention/promotion policies. IEA studies have chosen grade-based. ABN 34 117 491 228 mcgaw group pty ltd OECD/PISA chose the age-based NAEP started as age-based (as for PISA) became grade-based in 1980s (as for IEA) 17 Problem of age-based sample (IEA/PIRLS) 570 Sweden EnglandNetherlands 550 Canada(O,Q) 540 Italy 530 520 Hungary United States Germany Czech Republic Scotland Greece New Zealand France Slovak Republic Iceland 510 500 R = 0.55 R2 = 0.30 Norway 490 9.6 ABN 34 117 491 228 mcgaw group pty ltd Mean score in reading 560 Countries that perform best are those that test older students. 9.8 10.0 10.2 10.4 10.6 10.8 11.0 Mean age of students tested 18 OECD’s PISA assessment of the knowledge and skills of 15-year-olds ABN 34 117 491 228 mcgaw group pty ltd Coverage of world economy 83% 81% 77% 87% 86% 85% 19 What do international comparisons tell us? ABN 34 117 491 228 mcgaw group pty ltd 550 350 300 Finland Canada New Zealand Australia Ireland Hong Kong-China Korea United Kingdom Japan Sweden Austria Belgium Iceland Norway France United States Denmark Switzerland Spain Czech Republic Italy Germany Liechtenstein Hungary Poland Greece Portugal Russian Federation Latvia Israel Luxembourg Thailand Bulgaria Romania Mexico Argentina Chile Brazil FYR Macedonia Indonesia Albania Peru Mean reading results (PISA 2000) 600 Australia tied for 2nd with 8 others among 42 countries. 500 450 400 OECD (2003), Literacy skills for the world of tomorrow: Further results from PISA 2000, Fig. 2.5, p.76. 21 Australia’s ranking in OECD/PISA Reading Reading ranks PISA 2000: 4th but tied for 2nd PISA 2003: 4th but tied for 2nd PISA 2006: 7th but tied for 6th PISA 2000 PISA 2003 PISA 2006 Finland Finland Finland Korea Canada NZ Hong Kong Korea Canada NZ Hong Kong Korea Canada NZ Same as Australia ABN 34 117 491 228 mcgaw group pty ltd Ahead of Australia Behind Australia Hong Kong 22 560 Trends in reading performance Higher performers in Korea improved. Korea 550 Finland 540 Lower performers in HK improved. Hong Kong China 530 Canada New Zealand Australia 510 Changes for Finland, Canada & New Zealand are not significant. ABN 34 117 491 228 mcgaw group pty ltd 520 500 PISA 2000 OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 6.21, p.319. PISA 2003 PISA 2006 23 Trends in Australian reading performances 700 95th %ile 650 90th %ile 600 75th %ile 550 Mean 25th %ile 450 400 10th %ile 5th %ile 350 ABN 34 117 491 228 mcgaw group pty ltd 500 300 PISA 2000 PISA 2003 PISA 2006 OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 6.21, p.319. 24 ABN 34 117 491 228 mcgaw group pty ltd 550 350 300 Hong Kong-China Finland Korea Netherlands Liechtenstein Japan Canada Belgium Macao-China Switzerland Australia New Zealand Czech Republic Iceland Denmark France Sweden Austria Germany Ireland Slovak Republic Norway Luxembourg Poland Hungary Spain Latvia United States Russian Federation Portugal Italy Greece Serbia Turkey Uruguay Thailand Mexico Indonesia Tunisia Brazil Mean mathematics results (PISA 2003) 600 Australia tied for 5th with 8 others among 40 countries. 500 450 400 OECD (2004), Learning for tomorrow’s world: first results from PISA 2003, Fig. 2.16b, p.92. 25 Australia’s ranking in OECD/PISA Mathematics Mathematics ranks PISA 2000: 6th but tied for 3rd PISA 2003: 11th but tied for 5th PISA 2006: 13th but tied for 9th PISA 2000 Hong Kong Finland Hong Kong Korea Netherlands Japan ABN 34 117 491 228 mcgaw group pty ltd Ahead of Australia PISA 2003 Same as Australia Finland Korea Switzerland Canada Switzerland Canada Macao Japan PISA 2006 Taiwan Finland Hong Kong Korea Netherlands Switzerland Canada Macao Japan 26 None of those that moved ahead of Australia in mathematics from 2003 to 2006 improved significantly. Australia did not move significantly from 2003 to 2006. ABN 34 117 491 228 mcgaw group pty ltd Australia’s significant shift in ranking is a consequence of cumulative non-significant shifts in different directions. 27 Trends in Australian mathematics performances 700 95th %ile 650 90th %ile 600 75th %ile 550 Mean 25th %ile 450 10th %ile 400 ABN 34 117 491 228 mcgaw group pty ltd 500 5th %ile 350 PISA 2003 PISA 2006 OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 6.21, p.319. 28 ABN 34 117 491 228 mcgaw group pty ltd 550 350 300 Finland Hong Kong-China Canada Chinese Taipei Estonia Japan New Zealand Australia Netherlands Liechtenstein Korea Slovenia Germany United Kingdom Czech Republic Switzerland Macao-China Austria Belgium Ireland Hungary Sweden Poland Denmark France Croatia Iceland Latvia United States Slovak Republic Spain Lithuania Norway Luxembourg Russian Federation Italy Portugal Greece Israel Chile Serbia Bulgaria Uruguay Turkey Jordan Thailand Romania Montenegro Mexico Indonesia Argentina Brazil Colombia Tunisia Azerbaijan Qatar Kyrgyzstan Mean science results (PISA 2006) 600 Australia tied for 4th with 7 others among 57 countries. 500 450 400 OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 2.11b, pp.56-57. 29 Australia’s ranking in OECD/PISA Science Science ranks PISA 2000: 8th but tied for 3rd PISA 2003: 6th but tied for 4th PISA 2006: 8th but tied for 4th PISA 2000 Same as Australia ABN 34 117 491 228 mcgaw group pty ltd Ahead of Australia PISA 2003 PISA 2006 Finland Finland Hong Kong Canada Japan Korea Japan Korea Finland Hong Kong Canada Hong Kong Canada Japan Korea 30 The storyline so far… There is a growing labour market demand for higher level skills. International comparisons on quality in education show that: • Australian students are relatively high performing. ABN 34 117 491 228 mcgaw group pty ltd • The competition is not standing still. 31 The impact of raising expectations of low performers Variation of performance within schools OECD, UNESCO (2003), Literacy skills for tomorrow’s world: further results from PISA 2000, Table 7.1a, p.357. Iceland Sweden Finland Norway Spain Ireland Canada Denmark Korea New Zealand Australia United Kingdom Luxembourg United States Portugal Mexico Switzerland Italy Czech Republic Greece Poland Austria Hungary Germany Variation of performance between schools Belgium 110 100 90 80 70 60 50 40 30 20 10 0 -10 -20 -30 -40 -50 -60 -70 -80 ABN 34 117 491 228 mcgaw group pty ltd Variation in reading performance (PISA 2000) 33 OECD (2004), Learning for tomorrow’s world, Table 4.1a, p.383. Iceland Finland Norway Sweden Poland Denmark Ireland Canada -80 Spain -60 New Zealand Australia United States Mexico Portugal Luxembourg Switzerland Greece Slovak Republic Korea Czech Republic Netherlands Austria Germany Italy Belgium 80 Japan 100 Hungary Turkey ABN 34 117 491 228 mcgaw group pty ltd Variation in mathematics performance (PISA 2003) Variation of performance within schools 60 40 20 0 -20 -40 Variation of performance between schools 34 560 Trends in reading performance Korea 550 Finland 540 Hong Kong China 530 Canada New Zealand 520 Australia Poland 510 490 Lower and higher performers in Poland improved. Lower performers in Poland improved. 480 Changes for Finland, Canada & New Zealand are not significant. ABN 34 117 491 228 mcgaw group pty ltd 500 470 PISA 2000 OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 6.21, p.319. PISA 2003 PISA 2006 35 The storyline so far… There is a growing labour market demand for higher level skills. International comparisons on quality in education show that: • Australian students are relatively high performing. • The competition is not standing still. International comparisons on equity in education show that: ABN 34 117 491 228 mcgaw group pty ltd • Setting high expectations for all can improve low performers. 36 Elite student achievement is important too. Scientific excellence of 15-year-olds and countries’ research intensity Number of researchers per 1000 employed, FTE Finland 16 14 12 Sweden Japan 10 New Zealand Denmark United States Australia France Belgium Canada Korea Germany Luxembourg Switzerland Austria Spain United Kingdom Ireland Slovak Republic Portugal Poland Netherlands Greece Hungary Italy Norway 8 6 4 2 R2 = 0.70 Turkey Mexico 0 0 ABN 34 117 491 228 mcgaw group pty ltd 18 2 4 6 8 10 12 14 16 18 20 22 Percentage of students at levels 5 & 6 in PISA 2006 science OECD (2007), PISA 2006: Science Competencies for Tomorrow’s World, Vol. 1, p.51. 38 Equity matters too. % at each reading proficiency level: PISA 2000 Korea has relatively high mean but with few very high performers and very few low performers. 100 90 Level 5 80 70 Australia’s mean is high because of its relatively high percentage of very high-performing students. Level 4 60 50 40 Level 3 30 20 Level 2 10 0 Level 1 -20 -30 Australia has somewhat more low performing students than some high-performing countries around it. Peru Brazil Chile Argentina Mexico Romania Bulgaria Thailand -100 Source: OECD, UNESCO (2003) Literacy skills for the world of tomorrow, Table 2.1a, p.274 Luxembourg Israel Latvia Russian Federation Portugal Greece Poland Hungary Liechtenstein Germany Italy Czech Republic Spain Switzerland Denmark United States Norway France Iceland Belgium Austria Sweden Japan United Kingdom Korea Ireland Australia Canada -90 Finland -80 New Zealand -70 Hong Kong-China -60 Albania Below Level 1 Indonesia -50 FYR Macedonia -40 ABN 34 117 491 228 mcgaw group pty ltd -10 40 ABN 34 117 491 228 mcgaw group pty ltd 100 80 60 40 10 -40 -50 -60 -70 -80 -90 -100 Finland Hong Kong-China Canada Chinese Taipei Estonia Japan New Zealand Australia Netherlands Liechtenstein Korea Slovenia Germany United Kingdom Czech Republic Switzerland Macao-China Austria Belgium Ireland Hungary Sweden Poland Denmark France Croatia Iceland Latvia United States Slovak Republic Spain Lithuania Norway Luxembourg Russian Federation Italy Portugal Greece Israel Chile Serbia Bulgaria Uruguay Turkey Jordan Thailand Romania Montenegro Mexico Indonesia Argentina Brazil Colombia Tunisia Azerbaijan Qatar Kyrgyzstan % at each science proficiency level: PISA 2006 Level 6 90 Level 5 70 Level 4 50 Level 3 30 20 Level 2 0 -10 -20 Level 1 -30 Australia’s percentage of low performing students is similar to those in other relatively high performing countries around it. OECD (2007), PISA 2006: science competencies for tomorrow’s world, Vol. 1 - analysis, Fig. 2.11a, p.49. Below Level 1 41 Socioeconomic status & reading literacy (PISA 2000) Reading literacy ABN 34 117 491 228 mcgaw group pty ltd High Two indices of relationship: Social gradient Correlation or variance accounted for Social gradient: Magnitude of increment in achievement associated with an increment in social background (on average) Correlation: How well the regression line summarises the relationship Low PISA Index of social background Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308 Social Advantage 42 Social gradients for reading literacy (PISA 2000) High 600 Reading literacy Finland 500 Canada 450 Australia This gap is in the order of 3 years of schooling. 400 Germany 350 Steeper slope = less equitable results 300 ABN 34 117 491 228 mcgaw group pty ltd 550 -2 Low -1.5 -1 -0.5 0 0.5 1 1.5 2 Social PISA Index of social background Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308 Advantage 43 Social gradients for reading literacy (PISA 2000) 550 High quality 540 Low equity Finland New Zealand Australia United Kingdom 530 Reading literacy 500 490 Germany Hungary Korea Japan Sweden Belgium Austria France Norway United States Denmark Switzerland Czech Republic 510 480 Canada Ireland Iceland Spain Italy Poland Greece Portugal 470 460 450 Luxembourg 440 430 ABN 34 117 491 228 mcgaw group pty ltd 520 High quality High equity 420 -25 Low quality Low equity -20 Low quality High equity Mexico -15 -10 -5 0 5 10 15 Social equity (OECD regression slope - country regression slope) Source: OECD (2001) Knowledge and skills for life, Table 2.3a, p.253. 20 25 44 Social gradients for science literacy (PISA 2006) 575 High quality FinlandHigh equity High quality Low equity 550 New Zealand United Kingdom Czech Republic 500 France Germany Switzerland Austria Belgium Hungary Poland United States Slovak Republic Korea Ireland Sweden Denmark Luxembourg 475 Norway Greece Spain Italy Iceland Portugal 450 425 Turkey Low quality Low equity ABN 34 117 491 228 mcgaw group pty ltd Science literacy 525 Canada Japan Australia Netherlands 400 -15.0 -10.0 Low quality Mexico High equity -5.0 0.0 5.0 10.0 15.0 Social equity (OECD regression slope - country regression slope OECD (2007) PISA 2006: science competencies for tomorrow’s world, Vol 1 – analysis, Figure 4.6, p.184. 45 575 SES-science literacy correlations (PISA 2006) High quality High quality Low equity Finland High equity 550 New Zealand Australia Netherlands United Kingdom Germany Czech Republic Switzerland Belgium Ireland Austria Hungary Sweden Poland Denmark France United States Spain Luxembourg Slovak Republic Italy Portugal Greece Science literacy 525 500 475 Korea Norway Iceland 450 425 Turkey Low quality Low equity ABN 34 117 491 228 mcgaw group pty ltd Canada Japan 400 -10.0 -8.0 Low quality High equity Mexico -6.0 -4.0 -2.0 0.0 2.0 4.0 6.0 8.0 10.0 Social equity (% variation accounted for: OECD-country) OECD (2007) PISA 2006: science competencies for tomorrow’s world, Vol 1 – analysis, Figure 4.6, p.184. 46 The storyline so far… There is a growing labour market demand for higher level skills. International comparisons on quality in education show that: • Australian students are relatively high performing. • The competition is not standing still. International comparisons on equity in education show that: • Setting high expectations for all can improve low performers. • The disadvantaged are over-represented among low performers. ABN 34 117 491 228 mcgaw group pty ltd • Low performers are not left further behind than in other high-performing countries except to some extent in reading. 47 What do international comparisons tell us about Australian students’ engagement? Australian students’ engagement with science learning Rank among 57 countries ABN 34 117 491 228 mcgaw group pty ltd Aspect of engagement Level of engagement Difference in achievement associated with 1 unit difference in engagement General value of science 41st 2nd Personal value of science 37th 1st General interest in science 54th 13th Enjoyment of science 45th 2nd Self efficacy in science 13th 4th Self concept in science 43rd 1st Instrumental motivation to learn 32nd 3rd Future-oriented motivation to learn 42nd 3rd Involvement in science-related activities 53rd 2nd OECD (2007) PISA 2006: science competencies for tomorrow’s world, Vol 1 – analysis, Chapter 3, pp.122-163. 49 The storyline so far… There is a growing labour market demand for higher level skills. International comparisons on quality in education show that: • Australian students are relatively high performing. • The competition is not standing still. International comparisons on equity in education show that: • Setting high expectations for all can improve low performers. • The disadvantaged are over-represented among low performers. Australian students’ engagement with (science) learning: • They think they are good at science, as they are. • They don’t like it, don’t see the value of it, and don’t engage outside school. ABN 34 117 491 228 mcgaw group pty ltd • Low performers are not left further behind than in other high-performing countries except to some extent in reading. 50 Characteristics of the best performing systems Autonomy and standards 70 63 PISA science 50 41 40 46 30 Yes 20 10 Standards-based No external examinations 0 0 No ABN 34 117 491 228 mcgaw group pty ltd 60 Yes School autonomy (in teacher selection) 52 Characteristics of Finland High-quality teachers Schools responsible for all their students ABN 34 117 491 228 No grade repetition No school differentiation (before age 15) Early intervention for students needing it mcgaw group pty ltd More difficult to enrol in teacher education than medicine All 6-year trained with Masters on entry Largest % of students given extra support is in Grade 1 53 bmcgaw@unimelb.edu.au