Benő Csapó http://www.staff.u-szeged.hu/~csapo/ Current Trends in Educational Assessment in Hungary: Research and Development Prague, 26 November 2014 UNIVERSITY OF SZEGED INSTITUTE OF EDUCATION Outline • Current trends in educational assessment • Educational assessment in Hungary • Developing an online diagnostic assessments system – a large-scale project at the University of Szeged • Further plans to develop the educational assessment in Hungary Function of assessment: providing feedback “Measure what is measurable, and make measurable what is not so.” Galileo Galilei “If you can not measure it, you can not improve it.” Kelvin Current trends in educational assessment What is new? (1) New constructs (2) New contexts (3) New instruments (1) Assessment of new constructs “21st Century Skills” (2) Assessments in new context Assessment for learning • Formative and diagnostic assessment • Embedded in learning and teaching • Based on a better understanding of learning and development • Detailed, frequent • Provides immediate feedback • Helps to determine the directions of intervention (3) New Instruments Technology-Based Assessment • • • • • Innovative item formats, improved validity Improved objectivity and reliability Improved precision Automated scoring Immediate feedback TBA in PISA • 2006, major domain: science – Technology-Based Assessment of Science – international option • 2009, major domain: reading – Electronic Reading Assessment – international option • 2012, major domain: mathematics – Computer-Based Problem Solving – innovative domain • 2015, major domain: science – Computer-based assessment at all domain – Collaborative problem solving as innovative domain USA: Assessment in the “Race to the Top” • “Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy” • “Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction” Characteristics of the assessment projects in “Race to the Top” • Two grant categories – Comprehensive Assessment Systems – High School Course Assessment • Coalition of states • Two large coalitions received approximately $330 million • Partnership for Assessment of Readiness for College and Careers (PARCC) $170 million • SMARTER Balanced Assessment Consortium (SBAC) $160 million Educational assessment in Hungary History and context: Two research and development communities • Assessment centers at Budapest – National Educational Institute (-1990) • Hungary participated in the first IEA project in the 1970s and 1980s – After 1990: Several reorganizations – Today: Educational Authority • International and national assessment programs • Research and development centers at the University of Szeged – Test development projects from the 1970s – Several research groups established in the 1980s and 1990s – Today • Research Group on the Development of Competencies (1996-) – Hungarian Educational Longitudinal Program • Center for Research on Learning and Instruction – Developing Diagnostic Assessments – Training of educational assessment experts (1991-) Educational Assessments in Hungary PIRLS TIMSS PISA Grade Diagnostic Assessment System Diagnostic Assessments Assessment of Basic Competencies Matura (Abitur) Characteristics of the Assessment of Basic Competences • Reading and mathematics are the assessment domains • Every student is measured at grade 6, 8 and 10 • The data are longitudinally connected • Sophisticated value-added models are used • School-level results are published (‘soft accountability’) Developing Diagnostic Assessments A project implemented by the Center for Research on Learning and Instruction University of Szeged Two phases: 2009-2011, 2012-2014 edia.hu http://edia.hu English http://edia.hu/?q=en Publications / Books Demo test Online Diagnostic Assessment System • A complex system for supporting learning and teaching • Based on the analysis of learning and developmental processes, and the outcomes of learning See -> framework development • An online accessible assessment platform See -> eDia • An item bank containing thousands of items in innovative formats See -> demo test for item formats • Teacher training preparing the teachers for the utilization of the system • Developing intervention materials (e.g. computer games) for enhancing students’ skills if needed Devising assessment frameworks for reading, mathematics and science for the first six grades How an assessment framework looks like? PISA - application (literacy) TIMSS - content - reasoning - application Diagnostic - content - reasoning - application A three-dimensional model for framework development GENERAL ABILITIES Internal, psychological There may be unexploited potentials here LITERACY External, social expectations Disciplinary content CONTENT KNOWLEDGE – EXPERTISE Curriculum-based assessment, 1st and 2nd IEA Science and Mathematics Study PISA, three main literacy domains Devising assessment frameworks: working methods and products • Broad national and international cooperation • Results published in 2x3 volumes (in English and in Hungarian) • Opening theoretical chapters (international co-authors) • Detailed standards by Hungarian expert groups Reading Editors Benő Csapó and Valéria Csépe Pyschology Literacy Discipline Leo Blomert Wolfgang Schnotz Krisztián Józsa Valéria Csépe Edit Katalin Molnár János Steklács Mathematics Editors Benő Csapó and Mária B. Szendrei Reasoning Literacy Discipline Terezinha Nunes Lieven Verschaffel Mária B. Szendrei Benő Csapó Csaba Csíkos Júlia Szendrei Science Editors Benő Csapó and Gábor Szabó Reasoning Literacy Discipline Philip Adey Erzsébet Korom Gábor Szabó Benő Csapó Mária B. Németh Erzsébet Korom Result of the Framework Development Three volumes, published both in Hungarian and in English May be downloaded in .pdf The eDia platform • The system runs at the servers of the University of Szeged • Item builder module • Item bank unit • Test delivery module • Data bank • Data analysis module • Feedback module The item development process • Training the item writers (350+) – assessment theory and item writing – content of the assessment (frameworks) – ICT and the eDia platform • • • • • Item writing Reviewing by the group leader Supervising by the „dimension chief” Piloting Determining the item parameters Innovative item formats • Text and voice for the grades 1-3 • Stimulus – – – – text, voice picture, animation video simulation • Response capture – everything what is possible on paper, keyboard, mouse – manipulating and moving objects on the screen, drag and drop – touchscreen solutions Characteristics of the Online Diagnostic Assessment System • Serving ca. 600 000 students (grades 1-6) • Communicating with ca. 20 000 teachers • Making the service available in ca. 3 000 schools • Managing ca. 10 000 test items • Maintaining and analyzing students’ data • Providing both students and teachers with sophisticated feedback in user friendly format Assessments of the major domains • Reading, mathematics and science are the major domains • Each major domain is measured in 3 dimensions in different scales • Large item banks are used to measure the major domains Assessment of minor domains • There are 16 minor domains in the system – school readiness – writing skills, musical abilities, English as a second language – health literacy, financial literacy, visual skills, civic competencies – combinatorial reasoning, inductive reasoning – problem solving, learning to learn, ICT literacy, creativity – social skills, motivation • Test batteries are prepared for the minor domains Plans for the next phase: 2015• Online developmental exercises for helping slow students to catch up • Gamification • Visualization • Integration with the regular instructional processes • Extending to the kindergarten population The significance of early education The economists’ view (The Heckman curve) The Basic developmental skills psychologists’ Interest view: Solid foundation Attitudes for later Motivation learning Can online assessments be used in kindergarten? YES! Yes! Further plans to develop educational assessment in Hungary Digitizing the Assessments of Basic Competencies • Present problems – only reading and mathematics is assessed – expensive – slow feedback – difficult to interpret • Plans for TBA – adding science – developing item banks – immediate feedback – visualizing the data to help the interpretation Technology-based reform of the ‘matura’ • Present problems – too many elective subjects – low objectivity – no comparable scales – etc. • Plans for the reform – technology based assessments only – 5 compulsory subjects: Hungarian language and literature, history, mathematics, science, English (no electives) Conclusions • Technology-based assessments work well in kindergarten and primary schools • Technology can be extended to other domains and age groups, the precision, objectivity and reliability can be improved • Further work is needed to integrate the assessments into the regular classroom activities and educational processes Thanks for your attention! www.staff.u-szeged.hu/~csapo