Benő Csapó
http://www.staff.u-szeged.hu/~csapo/
Current Trends
in Educational Assessment in Hungary:
Research and Development
Prague, 26 November 2014
UNIVERSITY OF SZEGED
INSTITUTE OF EDUCATION
Outline
• Current trends in educational assessment
• Educational assessment in Hungary
• Developing an online diagnostic
assessments system
– a large-scale project at the University of Szeged
• Further plans to develop the educational
assessment in Hungary
Function of assessment:
providing feedback
“Measure what is measurable, and make
measurable what is not so.”
Galileo Galilei
“If you can not measure it,
you can not improve it.”
Kelvin
Current trends in educational
assessment
What is new?
(1) New constructs
(2) New contexts
(3) New instruments
(1) Assessment of new constructs
“21st Century Skills”
(2) Assessments in new context
Assessment for learning
• Formative and diagnostic assessment
• Embedded in learning and teaching
• Based on a better understanding of
learning and development
• Detailed, frequent
• Provides immediate feedback
• Helps to determine the directions of
intervention
(3) New Instruments
Technology-Based Assessment
•
•
•
•
•
Innovative item formats, improved validity
Improved objectivity and reliability
Improved precision
Automated scoring
Immediate feedback
TBA in PISA
• 2006, major domain: science
– Technology-Based Assessment of Science
– international option
• 2009, major domain: reading
– Electronic Reading Assessment
– international option
• 2012, major domain: mathematics
– Computer-Based Problem Solving
– innovative domain
• 2015, major domain: science
– Computer-based assessment at all domain
– Collaborative problem solving as innovative domain
USA: Assessment in the
“Race to the Top”
• “Adopting standards and assessments that
prepare students to succeed in college and
the workplace and to compete in the global
economy”
• “Building data systems that measure student
growth and success, and inform teachers and
principals about how they can improve
instruction”
Characteristics of the assessment
projects in “Race to the Top”
• Two grant categories
– Comprehensive Assessment Systems
– High School Course Assessment
• Coalition of states
• Two large coalitions received approximately
$330 million
• Partnership for Assessment of Readiness for
College and Careers (PARCC) $170 million
• SMARTER Balanced Assessment Consortium
(SBAC) $160 million
Educational assessment in
Hungary
History and context: Two research
and development communities
• Assessment centers at Budapest
– National Educational Institute (-1990)
• Hungary participated in the first IEA project in the 1970s and 1980s
– After 1990: Several reorganizations
– Today: Educational Authority
• International and national assessment programs
• Research and development centers at the University of
Szeged
– Test development projects from the 1970s
– Several research groups established in the 1980s and 1990s
– Today
• Research Group on the Development of Competencies (1996-)
– Hungarian Educational Longitudinal Program
• Center for Research on Learning and Instruction
– Developing Diagnostic Assessments
– Training of educational assessment experts (1991-)
Educational Assessments in Hungary
PIRLS
TIMSS
PISA
Grade
Diagnostic Assessment System
Diagnostic
Assessments
Assessment of Basic
Competencies
Matura
(Abitur)
Characteristics of the Assessment of
Basic Competences
• Reading and mathematics are the
assessment domains
• Every student is measured at grade
6, 8 and 10
• The data are longitudinally connected
• Sophisticated value-added models are used
• School-level results are published (‘soft
accountability’)
Developing Diagnostic Assessments
A project implemented by the
Center for Research on Learning and
Instruction University of Szeged
Two phases: 2009-2011, 2012-2014
edia.hu
http://edia.hu
English
http://edia.hu/?q=en
Publications / Books
Demo test
Online Diagnostic Assessment
System
• A complex system for supporting learning and teaching
• Based on the analysis of learning and developmental
processes, and the outcomes of learning
See -> framework development
• An online accessible assessment platform
See -> eDia
• An item bank containing thousands of items in
innovative formats
See -> demo test for item formats
• Teacher training
preparing the teachers for the utilization of the system
• Developing intervention materials (e.g. computer
games) for enhancing students’ skills if needed
Devising assessment frameworks for
reading, mathematics and science for
the first six grades
How an assessment framework looks
like?
PISA
- application
(literacy)
TIMSS
- content
- reasoning
- application
Diagnostic
- content
- reasoning
- application
A three-dimensional model for framework
development
GENERAL ABILITIES
Internal, psychological
There may be
unexploited
potentials here
LITERACY
External, social
expectations
Disciplinary content
CONTENT KNOWLEDGE – EXPERTISE
Curriculum-based assessment, 1st and 2nd IEA
Science and Mathematics Study
PISA, three main
literacy domains
Devising assessment frameworks:
working methods and products
• Broad national and international
cooperation
• Results published in 2x3 volumes (in
English and in Hungarian)
• Opening theoretical chapters (international
co-authors)
• Detailed standards by Hungarian expert
groups
Reading
Editors
Benő Csapó and Valéria Csépe
Pyschology
Literacy
Discipline
Leo Blomert
Wolfgang Schnotz
Krisztián Józsa
Valéria Csépe
Edit Katalin Molnár
János Steklács
Mathematics
Editors
Benő Csapó and Mária B. Szendrei
Reasoning
Literacy
Discipline
Terezinha Nunes
Lieven Verschaffel
Mária B. Szendrei
Benő Csapó
Csaba Csíkos
Júlia Szendrei
Science
Editors
Benő Csapó and Gábor Szabó
Reasoning
Literacy
Discipline
Philip Adey
Erzsébet Korom
Gábor Szabó
Benő Csapó
Mária B. Németh
Erzsébet Korom
Result of the Framework
Development
Three volumes, published both in
Hungarian and in English
May be downloaded in .pdf
The eDia platform
• The system runs at the servers of the
University of Szeged
• Item builder module
• Item bank unit
• Test delivery module
• Data bank
• Data analysis module
• Feedback module
The item development process
• Training the item writers (350+)
– assessment theory and item writing
– content of the assessment (frameworks)
– ICT and the eDia platform
•
•
•
•
•
Item writing
Reviewing by the group leader
Supervising by the „dimension chief”
Piloting
Determining the item parameters
Innovative item formats
• Text and voice for the grades 1-3
• Stimulus
–
–
–
–
text, voice
picture, animation
video
simulation
• Response capture
– everything what is possible on paper, keyboard,
mouse
– manipulating and moving objects on the screen, drag
and drop
– touchscreen solutions
Characteristics of the Online
Diagnostic Assessment System
• Serving ca. 600 000 students (grades 1-6)
• Communicating with ca. 20 000 teachers
• Making the service available in ca. 3 000
schools
• Managing ca. 10 000 test items
• Maintaining and analyzing students’ data
• Providing both students and teachers with
sophisticated feedback in user friendly
format
Assessments of the major domains
• Reading, mathematics and science are the
major domains
• Each major domain is measured in 3
dimensions in different scales
• Large item banks are used to measure the
major domains
Assessment of minor domains
• There are 16 minor domains in the system
– school readiness
– writing skills, musical abilities, English as a second
language
– health literacy, financial literacy, visual skills, civic
competencies
– combinatorial reasoning, inductive reasoning
– problem solving, learning to learn, ICT literacy,
creativity
– social skills, motivation
• Test batteries are prepared for the minor
domains
Plans for the next phase: 2015• Online developmental exercises
for helping slow students to catch up
• Gamification
• Visualization
• Integration with the regular instructional
processes
• Extending to the kindergarten population
The significance
of
early
education
The economists’
view
(The Heckman
curve)
The Basic
developmental
skills
psychologists’
Interest view:
Solid foundation
Attitudes for later
Motivation
learning
Can online assessments be used in
kindergarten?
YES!
Yes!
Further plans to develop
educational assessment in
Hungary
Digitizing the Assessments of Basic
Competencies
• Present problems
– only reading and mathematics is assessed
– expensive
– slow feedback
– difficult to interpret
• Plans for TBA
– adding science
– developing item banks
– immediate feedback
– visualizing the data to help the interpretation
Technology-based reform of the
‘matura’
• Present problems
– too many elective subjects
– low objectivity
– no comparable scales
– etc.
• Plans for the reform
– technology based assessments only
– 5 compulsory subjects: Hungarian language
and literature, history, mathematics, science,
English (no electives)
Conclusions
• Technology-based assessments work well in
kindergarten and primary schools
• Technology can be extended to other
domains and age groups, the precision,
objectivity and reliability can be improved
• Further work is needed to integrate the
assessments into the regular classroom
activities and educational processes
Thanks for your attention!
www.staff.u-szeged.hu/~csapo