PISA-PIRLS-Taskforce of IRA
International Reading Association
Annual Convention, May 9, 2011
(Orlando, FL, USA)
PISA 2009: A Critical Analysis of Trends
and Developments with Implications for
Adolescent Literacy Practice and Policy
William G. Brozo
George Mason University, Fairfax, Virginia, USA
Christine Garbe
University of Cologne, Germany
Gerry Shiel
St. Patrick's College, Dublin, Ireland
Map of Presentation
 Introduction of Presenters; IRA PIRLS/PISA Task Force
(Bill Brozo)
 Major Trends and Developments in PISA (Gerry Shiel)
 Key Findings – Ireland (Gerry Shiel)
 Key Findings – Germany (Christine Garbe)
 Key Findings – US (Bill Brozo)
 Conclusions: Implications for Adolescent Literacy Practice
and Policy (Bill Brozo)
 Discussion
2
Introduction of Presenters; IRA PIRLS/PISA
Task Force
William G. Brozo
wbrozo@gmu.edu
George Mason University, Virginia, USA
Bill Brozo
4
 Professor of Literacy at George Mason University, Fairfax,
Virginia, USA
 Degrees from the University of North Carolina and the
University of South Carolina
 Member of PISA/PIRLS Task Force since its inception in
2003
 Involved in international projects in the Balkans and Europe
and most recently in Oman
 Scholarship focuses on issues of adolescent literacy
PISA/PIRLS Task Force
5
 In 2003, The International Reading Association
Board of Directors requested that an International
Task Force be convened to consider the PISA 2000
findings
 Of particular interest to the board were the policy
and practice implications of PISA
 Original Task Force members in addition to me
included Keith Topping of Scotland, Renate Valtin of
Germany, Maria Dionisio of Portugal, and Cathy
Roller of IRA
PISA/PIRLS Task Force
6
 Generated reports and PowerPoint slide shows
available at the IRA website
 Given numerous presentations at national and
international conferences
 After a 2-3 year period of relative dormancy, the
Task Force was given new life in 2010 when the IRA
Board of Directors authorized its reconstitution to
coincide with findings from PISA 2009
PISA/PIRLS Task Force
7
 Current Task Force members include:
Gerry Shiel of Ireland; Christine Garbe of Germany;
Sari Sulkunen of Finland; Amby Pandian of Malaysia
 Bill Brozo serves as the chairperson of the Task
Force
Gerry Shiel
8
 Research Fellow since 1997 at the Educational
Research Centre at St. Patrick’s College in Dublin
 Consultant to OECD on Cycles II, III, and IV of PISA,
including PISA 2009
 Current chair of Federation of European Literacy
Association
 Author of numerous research, policy, and practical
publications related to reading literacy
Major Trends and Developments in PISA /
Key Findings Ireland
Gerry Shiel
gerry.shiel@erc.ie
St. Patrick‘s College, Dublin, Ireland
PISA Products – www.pisa.oecd.org
10
What is PISA?
11
 Programme for International Student Assessment
- Project of OECD
 International survey of achievement of 15-year-old students
 Reading literacy, mathematical literacy, scientific literacy
 Three-yearly cycles
- 2000, 2003, 2006, 2009, 2012. . .
 In 2009, 65 countries/economies participated in PISA
- 34 OECD member/candidate countries and 31 ‘partner’
countries = nearly 90 % of world economy
Purposes of PISA
12
 Describe the performance of 15-year olds in
reading literacy and in other areas
 Examine equity in performance within and
across countries, and identify factors associated
with equity
 Examine variables associated with reading
literacy (enjoyment of reading, engagement in
reading)
 Monitor trends in performance and related
indicators
Reading-related Components of PISA 2009
13
Component
Short Description / Comment
Print-based test of reading literacy
Includes 28 link items and 64 new items
School Questionnaire
Gathers background information on schools
(size, management structure, resources etc.)
Student Questionnaire
Includes questions on included items on
engagement in reading, enjoyment of reading,
use of reading strategies, frequency of reading a
range of print and digital texts, reading selfefficacy
Test of electronic reading
Involved simulated internet environment;
administered in 19 countries.
PISA Reading Literacy Framework
14
Mean Scores of Selected Countries Relative to
OECD Country Average (2009)
Above OECD Avg
Not Significantly Different
Below OECD Avg
Shanghai-China (556)
United States (500)
Italy (486)
Korea (539)
Sweden (497)
Spain (481)
Finland (536)
Germany (497)
Czech Rep (478)
Canada (524)
France (496)
Luxembourg (472)
New Zealand (521)
Ireland (496)
Austria (470)
Australia (515)
OECD Average (493)
15
Changes in Overall Reading Performance
(2000-09) (Selected Countries)
Significant Increase
No Change
Significant Decline
Chile (+40)
United States (-5)
Ireland (-31)
Israel (+22)
Canada (-10)
Sweden (-19)
Poland (+21)
Finland (-11)
Czech Republic (-13)
Portugal (+19)
S. Korea (+15)
Germany (+13)
Australia (-13)
16
Trends in gender differences in reading (20002009)
17
Average Difference Scores (all favor females)
60
55
51
50
40
30
40
39
29
29
25
40
35
32
35
2000
2009
20
14
10
0
Ireland
USA
OECD
Average
Finland
Germany
Korea
17
Aspects of Reading Engagement and Learning
Assessed in PISA
18





Engagement
Time spent reading for
enjoyment
Enjoyment of reading
scale (attitude to reading)
Diversity of print-reading
materials
Diversity of online reading
activities
Reading for school
•
•
•
•
•
Learning
Understanding and
remembering
Summarising
Memorisation strategies
Elaboration strategies
Control strategies
18
Association between Reading for Enjoyment and
Reading Performance (OECD Average)
540
40
37.4
532
527
527
35
520
30.3
30
504
500
25
480
20
17.2
Mean Score
15
460
460
10.6
10
440
4.5
420
5
0
Don't read for
enjoyment
30 minutes or less a day
30-60 minutes
Percent Students
1-2 horus
> 2 hours
19
Changes in Frequency of Reading for Pleasure
(2000-2009) – Percentages Who Read for Enjoyment
80
75
70
65
2000
2009
60
55
50
Ireland
USA
OECD 26
Average
Finland
Germany
Korea
20
Use of Reading Strategies
(OECD Average – Performance by Quarter)
Index of Understanding &
Remembering
Index of Summarising
21
Reading for Enjoyment and SES (Ireland)
22
Effects of Strategy Usage on Print Reading
Performance - IRELAND
23
Effect
45
40
35
39
38.9
35.2
30
27.6
25
Effect
20
15
10
7
5
5.9
0
Understand &
Remember
Summarize
Control Strategies
Memorisation
Strategies
Elaboration
Strategies
Socioeconomic
Status
Reflections on Ireland
24





Decline in overall reading performance
Demographic and other changes since 2000
Decline in reading for enjoyment
Effects of different reading strategies
Draft plan to improve literacy and numeracy
Key Findings – Germany
Christine Garbe
christine.garbe@uni-koeln.de
University of Cologne, Germany
Christine Garbe
26
 Professor of German Language and Literature at the
University of Cologne after many years at Leuphana
University, Lueneburg
 Coordinator of major Adolescent Literacy grant
Projects in Europe – ADORE, BaCuLit
 Initiator of an International ADOLESCENT LITERACY
NETWORK: www.alinet.eu
 Frequent author and presenter on topics related to
PISA and adolescent literacy
Major Trends and New Developments for German
Adolescents: Some good news
 Germany is one of the seven OECD-member
states that significantly improved in reading
competences
 In PISA 2000 Germany performed well below the
OECD-average (500) with 483 points; in PISA
2009 Germany reached 497 points and is thus
just above the OECD-average (493).
27
Major Trends and New Developments for German
Adolescents: some good news
 Germany improved its results on the overall
reading literacy scale mainly because of
improvements by the poor readers.
 In PISA 2000 the “students at risk” (performing
below level 2) were 22,6 %; in PISA 2009, this
reduced to 18,5 %. With regard to the highest
performers - level 5 (in PISA 2009 including level
6) - no progress was made: the share of students
declined from 8,8 to 7,6 %.
28
Major Trends and New Developments for German
Adolescents: some good news
 The improvement of Germany´s overall results
in reading literacy was brought about by
reducing the gap between the strong and poor
readers
 In particular, students with migrant backgrounds
improved considerably: they achieved 26 points
more than in PISA 2000, twice as much as the
whole German cohort (+ 13 points)
29
Major Trends and New Developments for German
Adolescents: not so good news
 Nevertheless the gap between German native
speakers and Second language learners /
students with migrant background is still too
large: German natives reach 514 points,
students with migrant background reach 470
points
 The difference of 44 points is equivalent to more
than one year of schooling
30
Major Trends and New Developments for German
Adolescents: not so good news
 German students are relatively weak
performers in the Reading literacy subscale
“Reflect and evaluate”, i.e. in the most complex
and demanding tasks
 With 491 points, Germany ranks at position 27
(out of 65), whereas in the overall reading scale
at position 20. Students of Ireland (position 16)
and the US (position 10) perform much better in
this aspect.
31
Major Trends and New Developments for German
Adolescents: not so good news
 Gender differences in reading did not change
significantly between 2000 and 2009
 Girls are performing 40 points better than boys
in overall reading competence. There are slight
differences in the three subscales:
 Access and retrieve information: 38 points.
 Integrate and interpret: 36 p.
 Reflect and evaluate: 44 p.
32
Major Trends and New Developments for German
Adolescents: not so good news
 Boys are over-represented in the lowest
proficiency levels:
 Below level 2 (poor performers) there are 18,5 %
German Students. Boys: 24 %, Girls: 12,6 %
 Reading for Enjoyment (outside of school) did not
increase in Germany: 41 % of German youth do not
read for pleasure (in 2000 42 %)
 Especially where boys are concerned: 50 %
compared with 25 % girls do not read for pleasure.
33
Key Findings - US
William G. Brozo
wbrozo@gmu.edu
George Mason University, Virginia, USA
Major Trends and New Developments for U.S.
Adolescents
 No state- or school district-level results are
available
 Overall, 15-year-olds in the United States
achieved a slightly but not significantly lower
score in 2009 (500) compared with 2000 (504)
 Up slightly but not significantly from 2003 (495)
35
READING LITERACY: GENDER
36
 Girls outperformed boys in reading literacy in
the United States as in every participating
country
 In 2000 the disparity between girls and boys in
the U.S. was 28 points; in 2009, there was a 25
point difference in overall achievement favoring
girls
 Girls overall achievement was 518 in 2000 and
513 in 2009 compared with boys 490 in 2000
and 488 in 2009
READING LITERACY: RACE
37
Race/Ethnicity
Score
U.S. Average
500
OECD Average
493
P < .05
White
525
X
Asian
541
X
Black
441
X
Hispanic
466
X
Shanghai-China
556
X
Republic of Korea
539
X
Finland
536
X
Hong Kong China
533
X
Singapore
526
X
READING LITERACY: SOCIOECONOMIC CONTEXTS
38
Percentage of Students Eligible for Free and Reduced-Price Lunch
Score
U.S. Average
500
OECD Average
493
Less than 10 percent
551*
10 – 29.9 percent
527*
25 – 49.9 percent
502
50 – 74.9 percent
471*
75 percent or more
446*
READING LITERACY: ENGAGEMENT
39
 The pattern for U.S. 15-year-olds is similar to the
pattern for all students on PISA
 Higher reading engagement, as demonstrated
by time spent reading and attitudes toward
reading, is related to higher achievement
READING LITERACY: ENGAGEMENT
40
 Students who do not read for enjoyment had a
score of 467 while those who read one, two, or
more hours per day had scores from 541-544
 Students who strongly agree with the statement
“I read only if I have to” had a score of 459,
while those who strongly disagree had a score of
552
 For students who view reading as a favorite
hobby, their score was 562, while those who do
not had a score of 466
READING LITERACY: READING STRATEGIES
41
 15-year-olds in the U.S. who use reading
strategies and processes regarded as effective
ways of aiding comprehension had higher scores
than those who did not
 Monitoring comprehension, determining
importance, connecting new information with
prior knowledge, summarizing, and questioning
were all related to higher achievement
READING LITERACY: READING STRATEGIES
42
 Students who almost always check their
understanding after reading had a score of 521,
while those who almost never do this had a score of
465
 Those who almost always try to identify the
important points while reading had a score of 532,
while those who almost never do this had a score of
436
 Students who relate new information to what
they’ve already learned had a score of 526, while
those who almost never do this had a score of 480
READING LITERACY: READING STRATEGIES
43
 Those who summarize what they read had
scores ranging from 513 – 519, while those who
rarely did this had scores between 460 – 480
 Those students who always ask their own
questions while reading had a score of 519,
while those who almost never do this had a
score of 445
Conclusions: Implications for Adolescent
Literacy Practice and Policy
William G. Brozo
wbrozo@gmu.edu
George Mason University, Virginia, USA
Implications for Adolescent Literacy Practice and
Policy: Gender
 More attention needs to be given to declining
reading achievement and motivation among boys,
particularly for boys of color and migrant
backgrounds
 Texts and instructional practices will need to be
culturally responsive and orchestrated in ways that
capture boys’ imaginations, sustain their attention,
and build competency
 Boys competencies with non-continuous and
alternatively formatted text may serve as bridge to
academic literacy
45
Implications for Adolescent Literacy Practice and
Policy: Race/SES
 PISA 2009 continues to show most of the lowest
performing 15-year-olds are poor and minority children
 For example, Asian-American and White students in the
U.S. have some of the best scores in the world, but Black
and Hispanic students rank near the bottom among OECD
countries
 Some argue that the United States runs separate and
unequal schools and neighborhoods leading to illeducated youth
 The conditions of the schools and neighborhoods for our
poor, African American, and Hispanic youth, they assert,
are not designed to develop high levels of reading
literacy
46
Implications for Adolescent Literacy Practice and
Policy: Race/SES
 Furthermore, we accept poverty, violence, drugs,
unequal school funding, uncertified teachers, and
de facto segregation in the schools that serve these
children and in the neighborhoods in which they
live
 These unequal conditions may be the major reason
we fall short in international comparisons when we
combine the scores of these ill-educated youth with
those of youth who enjoy better resources
 As long as these differences are allowed to exist,
some have argued, we will rank about average in
international comparisons
47
Implications for Adolescent Literacy Practice and
Policy: Reading Strategies
 The evidence is compelling for literacy curricula
that emphasizes critical reading processes
 This emphasis should be the focus right from the
start in our language and literacy schemes for
children
 The new Common Core State Standards
movement places emphasis on helping students
read and succeed with increasingly complex text
starting in the early grades
48
Implications for Adolescent Literacy Practice and
Policy: Reading Strategies
 To be successful with complex text, students will
need to be skillful and flexible with a range of
effective reading comprehension strategies,
such as questioning, summarizing, identifying
importance, connecting new content to prior
knowledge, and monitoring comprehension
49
Cross-National Implications
for Adolescent Literacy Policy and Practice
 Strive to bring males to achievement levels that
parody those of females
 Close the gap between the lowest and highest
performers
 Increase reading engagement for all youth
 Stress high-yield reading strategies
50
PISA-PIRLS-Taskforce of IRA
International Reading Association
Annual Convention, May 9, 2011
(Orlando, FL, USA)
Discussion