Using Student Assessment Results to
Improve Teacher Knowledge and
Practice
NESA FLC
24- 27 October2013
MARTINA BOVELL, B.A. Dip. Ed. Grad dip. Arts
(UWA)
Senior Research Fellow
www.acer.edu.au
80 years experience
Independent
Not for profit
Over 300 staff in seven offices
Melbourne, Sydney, Brisbane, Adelaide, Perth, India (New Delhi), UAE (Dubai)
Four goals:
Learners and their needs
every learner engaged in challenging learning opportunities appropriate to their readiness and
needs
The Learning Profession
every learning professional highly skilled, knowledgeable and engaged in excellent practice
Places of learning
every learning community well resourced and passionately committed to improving outcomes for all
learners
A Learning Society
a society in which every learner experiences success and has an opportunity to achieve their
potential
ACER and international assessment
Member of the International Association for
the Evaluation of Educational Achievement
(IEA).
Consortium leader for the OECD’s
Programme for International Student
Assessment (PISA) 1998 – 2012.
Purposes of assessment
• clarify educational standards
• monitor trends over time
• evaluate the effectiveness of
educational initiatives and programs
• ensure that all students achieve
essential skills and knowledge
International Schools’ Assesssment
http://www.acer.edu.au/tests/isa
Over 64 000 students from 312 schools participated in the ISA in
October 2012 and Feb 2013.
The ISA
Grades 3 – 10
•
•
•
•
Mathematics
Reading
Expository writing
Narrative writing
Grades 8 – 10
• Science online
• New in 2013-2014 testing cycle
How ISA results are reported
• All students who participate in ISA tests have their performance
measured against a single scale.
• There is a separate scale for each of the five domains being
assessed (Reading, Mathematics, Narrative Writing, Expository
Writing, and Science).
• The ISA scale score for a student is different from the “raw”
score that the student would get by adding up the number of
correctly answered questions on a test.
• The ISA scale allows meaningful comparisons of results between
different grade levels and between different calendar years even
though the tests administered are not the same.
• The ISA scales are based on those developed for the
Organisation for Economic Co-operation and Development’s
(OECD’s) Program for International Student Assessment (PISA).
Types of information
Whole grade mean and distribution
– all students, language background, gender
– this year and previous years (longitudinal comparisons)
• compared to all other ISA schools
• compared to all other Like schools (based on %s of students with ESB)
• For years 8 – 10, comparison with PISA results (Maths, Reading,
Science)
Classes within whole grades
– class mean and individual student scale scores
– performance by item classification within a domain
• compared to all other ISA schools
• compared to all other Like schools
• compared to other classes in your school
Item by item results for each student
– By item classification
– compared to other individuals
– compared to all other ISA schools
ISA Reports
• Paper-based
• Electronic
• tracking
• interactive
Science: online test delivery means
faster turn-around of results
“Improve teacher knowledge and
practice”
A growth model:
– acknowledges that each student is at some
point in their learning
– expects every student to make excellent
learning progress regardless of their
starting point
– assesses growth over time
ACER National School Improvement tool
(NSIT)
• Means of ACER’s mission of improving
learning
• Endorsed by Australian federal and state
governments
• Research-based
• Informs and assists schools’ improvement
agendas
• Available to all schools
http://www.acerinstitute.edu.au/home
Part of ACER’s Institute of Learning larger
school improvement project.
• The Institute’s services include
– Professional learning
– School review services
– Capacity building for school improvement
www.acer.edu.au/documents/NSIT.pdf
9. Schoolcommunity
partnerships
8. Effective
pedagogical
practices
7.
Differentiated
teaching and
learning
1. An explicit
improvement
agenda
2. Analysis
and
discussion of
data
National school
improvement
tool
6.
Systematic
curriculum
delivery
5. An expert
teaching
team
3. A culture
that
promotes
learning
4. Targeted
use of
school
resources
Review of a domain
Domain performance levels
Domain performance level: LOW
• Teachers do not systematically analyse
test and other data for their classes and
teachers make little use of data to
reflect on their teaching.
Domain performance level: MEDIUM
• An ad hoc approach exists to building
staff skills in the analysis, interpretation
and use of classroom data
Domain performance level: HIGH
• One or more members of staff have been assigned
responsibility for implementing the annual plan,
analysing the full range of school data, and
summarising, displaying and communicating student
outcome data for the school. The school has ensured
that appropriate software is available and that at
least these assigned staff have been trained to
undertake data analyses.
• Time is set aside (e.g. on pupil free days and in staff
meetings) for the discussion of data and the
implications of data for school policies and classroom
practices. These discussions occur at whole-school and
team levels.
Domain performance level: OUTSTANDING
• Data are used throughout the school to identify gaps in
student learning, to monitor improvement over time and to
monitor growth across the years of school.
• A high priority has been given to professional development
aimed at building teachers’ and leaders’ data literacy skills.
• Staff conversations and language reflect a sophisticated
understanding of student assessment and data concepts
(e.g. value-added; growth; improvement; statistical
significance).
• Teachers are given class test data electronically and are
provided with, and use, software to analyse, display and
communicate data on individual and class performances and
progress, including pre- and post-test comparisons.
• Teachers routinely use objective data on student
achievement as evidence of successful teaching.
Data literacy framework
US Dept of Education office of planning, evaluation and
Policy development (2011). Teachers’ ability to use data to
inform instruction: challenges and supports.
• STUDY:
• investigation of teachers’ thinking about student data by administering
hypothetical education scenarios accompanied by data displays and
questions to individual and small group interviews to teachers and
schools identified as exemplary in active data use (50 teachers, 72 small
groups)
• PURPOSES:
• 1. to investigate teachers thinking and reasoning independently about
data and how they build on each other's understandings when working
with data in small groups.
• 2. to isolate the difficulties, misconceptions and support needed.
What they found
• Teachers’ likelihood of using data is affected by
how confident they feel about their knowledge
and skills
• Working in small groups appears to promote
teachers’ engagement with data. Compared when
working individually, teachers were
– more likely to arrive at sound data interpretations
– more likely to use a wider range of skills when making
decisions about how to use and interpret data
– able to clarify and frame problems and correct data
interpretation errors
– more likely to enjoy discussing data
Data literacy skill areas:
US study framework
• Data location
– find relevant pieces of data in the data system or display
• Data comprehension
– understand what the data is saying
• Data interpretation
– figure out what the data mean
• Instructional decision making
– select an instructional approach to address the situation identified
in the data
• Question posing
– frame instructionally relevant questions that can be addressed by
the data
Bias when making decisions using data
Representative /Availability bias
• When judging the probability of something, we use some
preconceptions based on how similar two things are or the extent to
which an event matches our previous experience.
• e.g. irrelevant personal characteristics or stereotypes
Anchoring and adjustment bias
• Make a decision based on an initial calculation without following
though on the calculations.
• We ignore data that doesn’t agree with our preliminary decisions or
biases (we agree with what we expect).
Confidence in making the decision is not always associated
with quality of decision making
Group decision making can mitigate some of these biases.
PISA findings about decision making
PISA in focus #26 http://www.oecd.org/pisa/pisainfocus/
• “Countries vary in the way they use marks, but they all tend to
reward the mastery of skills and attitudes that promote learning.”
• “Teachers tend to give girls and socio-economically advantaged
students better school marks, even if they don’t have better
performance and attitudes than boys and socio-economically
disadvantaged students.”
• “It seems that marks not only measure students’ progress in school,
they also indicate the skills, behaviours, habits and attitudes that
are valued in school.“
What biases?
Narrative Writing assessment
Narrative Task
Same tasks and marking criteria for all grades
– Content 0 - 14: quality and range of ideas, development of plot,
characters and setting, the writer’s sense of audience and purpose, the
overall shape of the writing.
– Language 0 - 14:
sentence and paragraph structure, vocabulary and
punctuation, and the writer’s voice.
– Spelling 0 - 11:
considers phonetic and visual spelling patterns and the
kind of words attempted, and correctness.
Reporting
Content: 7 (max score 14)
Language: 6 (max score 11)
Spelling: 5 (max score 11)
Individual student report comment:
Level 5: Write a story with some developed detail in content and
using a variety of sentence forms. Spell correctly many words
from a student-level vocabulary.
Reading Framework
The ISA definition of Reading literacy derives from PISA:
Understanding, using, reflecting on and engaging with written
texts, in order to achieve one’s goals, to develop one’s
knowledge and potential, and to participate in society.
Goes beyond decoding and literal comprehension and
recognises the full scope of situations in which reading plays
a role in the lives of students from grades 3 to 10.
Three parts:
ASPECT
TEXT TYPE
TEXT FORMAT
Reading Framework
Retrieving Information:
Locating , selecting, and collecting one
or more pieces of information in a text.
Interpreting texts:
Making sense of the text, constructing
meaning making connections and
drawing inferences from one or more
parts of a text, e.g. cause and effect,
compare and contrast, category and
example. Involves information that is
not stated.
Reflecting:
Drawing on knowledge, ideas and values
external to the text to evaluate a text;
relating a text to one’s experience,
knowledge and ideas.
Continuous text format
composed of sentences that are, in
turn, organised into paragraphs. These
may fit into even larger structures
such as sections, chapters and books.
Narrative pieces, exposition,
description, argument and
instructional passages
Non–continuous text format
Essentially, texts composed of one or
more lists in which information is
presented in, e.g. tables, graphs, maps
and diagrams
Feedback from EARCOS
• This workshop was exactly what was needed to guide us. It
really helped the focus and the analysis of data. I hope ISA
continue to provide this service to support schools who use this
service.
• Very useful and I am motivated to do my work better.
• I felt that the workshop has enabled me to speak with some
authority on how we can unpack the ISA results. More
importantly I have an insight into how we can use these results
to better inform further decisions, ones that are based on
student achievement.
• Very useful. More of these sessions need to be provided to
help teachers understand how data can be used to improve
learning.
• Informative and well worth the time. Hands on, interactive
learning.
ISA Interactive diagnostic report
A guide to using the interactive diagnostic
report that was demonstrated during the
the conference session is downloadable at
http://www.acer.edu.au/documents/ISA_Using_the_Interactive_Diagnosti
c_Report.pdf
Interrogating data
Focus on the student
• Did the student perform as well as expected?
• Does the performance match expectations/reflect teacher
judgement about the student?
• What does the student’s response pattern show about the
strengths of the student?
• What does the student’s response pattern show about the
areas of concern for the student?
• Are any areas of concern preventing the student from
making progress? What might account for these?
Interrogating data
Focus on the group – group scores
• How does the group achievement relate to the bands?
• How does the class distribution against the bands match
expectations about the group?
• Did the group as a whole perform as well as expected?
• Does the relative order of students match expectations
about the students?
• What students have achieved higher than expected, or
lower than expected, in relation to others?
• Are there students in the group with similar achievements?
Interrogating data
Focus on the group – group scores
• Do students with similar scale scores have similar or
different response patterns?
• What assessment criteria do students perform well on?
• What assessment criteria do students perform less well on?
• What does the group’s response pattern show about its
strengths?
• What does the group’s response pattern show about its
areas of concern?
Interrogating data
• Focus on the teaching program
• Has any teaching impacted on the group’s results?
• Are there any areas of concern preventing the
whole group from making progress?
www.acerinstitute.edu.au/conferences/eppc
Presented by practitioners, for practitioners.
ACER recognises that, every day, teachers and school leaders
are responsible for improving learning among students.
This conference provides an opportunity to report on and
celebrate the improvements you have achieved within your
classes, across the whole school or within networks of schools.
Call for papers now open
School Improvement Tool Contact
Robert Marshall
Senior Project Director
Australian Council for Educational Research
19 Prospect Hill Road, Camberwell, Victoria
Australia 3124
Robert.Marshall@acer.edu.au
• +61 3 9277 5346
• 0439 665 965
Martina Bovell
Senior Research Fellow
Australian Council for Educational Research
7/1329 Hay Street, West Perth
Western Australia 6005
martina.bovell@acer.edu.au
+61 8 9235 4821
+61 439 926 277
References
• Barber, M. and Mourshed, M. (2007). How the world’s best-performing
school systems come out on top. Mckinsey and Co.
• Dweck, C.S. (2006). Mindset: The new psychology of success. New
York: Balantine Books.
• Fullen, M., Hill, P., Crevola, C. (2006). Breakthrough. California: Corwin
Press.
• International Baccalaureate Organisation. (2010). IB learner profile
booklet. www.ibo.org
• International Baccalaureate Organisation. (2010). Programme
standards and practices. www.ibo.org
• Masters, G. (2013). Towards a grown mindset in assessment. ACER
Occasional essays – 2013. www.acer.edu.au
• Tversky, A. and Kahneman, D. (1982).Judgement under uncertainty:
Heuristics and biases. In Judgement under uncertainty: Heuristics
and biases, eds. D Kahneman, P. Slovic and A.Tversky, 3-20. NY:
Cambridge University Press – cited in US Dept of Education office of
planning, evaluation and Policy development (2011). Teachers’ ability
to use data to inform instruction: challenges and supports .