Irish Presidency Conference
Better Assessment and Evaluation to Improve Teaching and Learning
19-20 March 2013
The Use of ICT for the Assessment
of Key Competences
Christine Redecker
Yves Punie
IPTS- Who we are
IPTS: Part of Joint Research Centre of
the EC: 7 Research Institutes across
Europe
Mission: “to provide customer-driven
support to the EU policy-making
process by developing science-based
responses to policy challenges that have
both a socio-economic as well as a
scientific/technological dimension”
IPTS Studies on
ICT for Learning
Future Learning Spaces in 2020
eLearning in the EU10 MS
- Digital Competence
Review ICT
- Active Ageing & Learning
Impact
Learning 2.0 in formal E&T
Learning 2.0 in informal settings
Creativity & Innovation in
schools in EU
The Future of Learning
Tell-Net Teacher Networking
2006
2007
2008
2009
2010
2011
2012
2013
ICT for the Assessment of Key Competences
Digital Competence
DC Follow up
Mainstreaming Creative Classrooms
1to1 Learning initiatives in Europe
Mapping Technologies for Learning
Open Education in Europe
Where we stand
1995
2000
2005
Generation 1
Generation 2
Automated administration
and/or scoring of
conventional tests
Adaptive Testing
2010
2015
2020
Generation 3
2025
Personalised learning
Generation 4
Continuous Data Mining & Analysis
Personalised
integrated
feedback and
Behavioural tracking
assessment
tutoring
Learning Analytics
Multiple Choice
Short Answer
Automated Feedback
Replication of complex
real life situations within a
test
Intelligent Tutors
Generation Re-invention
Transformative strategy: use technology to Games
change test formats
Virtual laboratories
Simulations
Computer-Based Assessment (CBA)
Virtual Worlds
Online collaboration
Peer assessment
Learning 2.0
Collaborative multimedia learning
environments
Technology-enhanced learning
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational
measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New
York: Macmillan
Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B.
McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Current use of ICT in
assessment in schools
There is scope for improvement
Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general
secondary education (ISCED 1, 2, 3), 2009/10
Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture
Executive Agency.
Examples
1st Generation CBA:
Problem-Solving
http://www.worldclassarena.org/v5/flash/09_y
ear_old/backtofront.swf
World Class Arena Sample Test: Problem
Solving (9 years)
Examples
Generation Re-Invention:
Problem-Solving
http://www.worldclassarena.org/v5/flash/13
_year_old/pyramids.swf
Problem Solving 13 years
Examples
1st Generation CBA:
Situational Judgement
EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm
EPSO sample assessment centre e-tray exercise
http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm
Examples
Generation Re-Invention:
e-tray exercise
Future eAssessment
Strategies
1995
2000
2005
Generation 1
Generation 2
Automated administration
and/or scoring of
conventional tests
Adaptive Testing
2010
2015
2020
2025
Data Mining & Analysis
Behavioural tracking
Learning Analytics
Multiple Choice
Automated Feedback
Intelligent Tutors
Generation Re-invention
Transformative strategy: use technology to
change test formats
Computer-Based Assessment (CBA)
Generation 3
Continuous
integrated
assessment
Generation 4
Short Answer
Replication of complex
real life situations within a
test
Personalised learning
Games
Virtual laboratories
Simulations
Personalised
feedback and
tutoring
Virtual Worlds
Online collaboration
Peer assessment
Learning 2.0
Collaborative multimedia learning
environments
Technology-enhanced learning
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational
measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New
York: Macmillan
Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B.
McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Examples
Towards Generation 3:
Datamining
Signals
http://www.itap.purdue.edu/tlt/sig
nals/signals_final/index.htm
Similar: GPS at Northern Arizona
University
http://resourceconnect.nau.edu/gr
ade-performance-status.asp
Examples
Towards Generation 3: Behaviour Tracking
Example: SNAPP
SNAPP is a software tool that
allows users to visualize the
network of interactions
resulting from discussion
forum posts and replies.
Teachers can rapidly identify
patterns of user behaviour.
SNAPP can extract all user
interactions from various
commercial and open source
learning management
systems, such as BlackBoard
and Moodle.
Examples
Towards Gen 4:
Intelligent Tutoring Systems
AutoTutor
https://www.youtube.com/watch?v=aPcoZPj
L2G8
GnuTutor (http://gnututor.com/)
OpenSource version of AutoTutor
https://www.youtube.com/watch?v=k6HDE
EnYuIA&feature=player_embedded
Technological trends
1995
2000
Generation 1
2005
2010
2015
2020
2025
Generation 2
Adaptive Testing
Automated administration
and/or scoring of
conventional
mainstream usetests
for high stakes testing
- further development of innovative test formats
- increased ease of use at school level
- trend towards automated scoring
→ cultural change:
increased acceptance of CBA
Personalised learning
Generation 3
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational
measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New
York: Macmillan
Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B.
McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Continuous
integrated
Multimedia collaborative
learning
assessment
environments support key competence
Generation 4
development.
- Peer and portfolio
Personalised
feedback and
assessment
tutoring evolve
naturally.
-Data mining and analysis will make
integrated assessment and tutoring possible
in the future.
We are in a transitional period: Moving from the “Testing
Paradigm” to the
(future)Re-invention
“Tutoring Paradigm” in which explicit
Generation
Learning 2.0
testing
could become
obsolete
Replication
of complex
real life
situations within a test
Computer-Based Assessment (CBA)
Collaborative multimedia learning
environments
Technology-enhanced learning
Assessing
Key Competences
Key Questions
1. Which ICT tools, formats, environments, devices etc. are
most suitable for the assessment of which Key
Competences?
2. How can ICT be used to support competence-based
assessment strategies that adequately capture the skills
and attitudes dimensions of all eight Key Competences and
their transversal nature?
Intelligent
Tutors etc.
Communication in the mother tongue
D
2
Communication in foreign languages
D
3a
Mathematical competence
D
3b
Basic competences in science and technology
4
Digital competence
5
Learning to learn
6
Social and civic competences
FS
7
Sense of initiative and entrepreneurship
FS
8
Cultural awareness and expression
S
FDS
FDS
FS
1
S
Simulations
Virtual worlds
games
Self
Assessment
FD
Peer
Assessment
S
ePortfolios
Quizzes /
simple
games
F = Formative Assessment
D = Diagnostic Assessment
S = Summative Assessment
CBA
Focus Areas
FS
FD
FS
FDS
FDS
FS
FS
FDS
FS
•PART 1
•INSPIRATION
Mother Tongue Literacy
1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
Matching spoken and written language
Foreign language learning
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
http://www.nickjr.de/games/107-swipers-buchstabier-buch
2 Communication in foreign languages
Mathematical games
3a
Mathematical competence
CBA (DS)
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Mathematical games using simulation for secondary school education
Source: http://hotmath.com/games.html.
Peer Assessment for
Collaborative Work
5 Learning to Learn
Example:
SparkPlus
http://spark.uts.edu.au/
CBA (F)
Peer Assessment (FS)
Self Assessment (FS)
Peer Assessment
using moodle
www.moodle.org
Moodle is a free web
application that educators can
use to create effective online
learning sites.
•
Moodle Statistics
Registered sites: 77,213
Countries: 229
Courses: 7,116,466
Users: 66,349,750
Teachers: 1,292,731
Enrolments: 56,306,274
Forum posts: 118,413,272
Resources: 63,891,942
Quiz questions: 147,057,434
http://www.youtube.com/watch?v=Tp8v2UBbmN4
ePortfolios
1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
2 Communication in foreign languages
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
8 Cultural Awareness and expression
http://sites.google.com/site/eportfolioapps/overview/model-k-12
ePortfolios
ePortfolios (FS)
Virtual worlds / games (FS)
ePortfolios
1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
2 Communication in foreign languages
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
8 Cultural Awareness and expression
ePortfolios (FS)
Virtual worlds / games (FS)
http://sites.google.com/site/eportfolioapps/overview/levels
Level 1 - ePortfolio as Storage
Collection:
Creating the Digital Archive (regularly – weekly/monthly)
A Focus on Contents & Digital Conversion
Level 2 - ePortfolio as Workspace
Collection + Reflection:
Immediate Reflection on Learning & Artifacts in Collection (regularly)
A Focus on Process & Documentation of Learning
Level 3 - ePortfolio as Showcase
Selection/Reflection/Direction (each semester? End of year?)
A Focus on Product & Documentation of Achievement
→ A (formative/summative) assessment approach on the basis of artefacts
→ Focus on production and reflection
ePortfolios meet
Virtual reality
8
Cultural Awareness and expression
ePortfolios (FS)
Virtual worlds / games (FS)
A game approach to arts
http://pheon.org; http://pheon.org/intro.html;
Pheon is a multimedia scavenger hunt that tests players’ ingenuity through
exploring the collection of the Smithsonian American Art Museum, making
Science:
Molecular Workbench
3b
Basic competences in science and technology
CBA (S)
Virtual Worlds / games (FS)
Simulations (FDS)
http://mw2.concord.org/public/part2/diffat/page3.cml
Simulations for
mathematics
3a
Mathematical competence
CBA (DS)
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Sample task from
US future New State Tests
(2014/15)
http://www.k12center.org/rsc/pdf/a-seachange-in-assessment-letter-size.pdf
Cognitive Tutor
(Carnegie)
3a
Mathematical competence
CBA (DS)
http://www.youtube.com/watch?v=RR4HPrGlUz
8&feature=related
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Science:
Quest Atlantis
3b
Basic competences in science and technology
Example:
Quest Alantis
Quest Atlantis (QA) is an
international learning and
teaching project that uses a 3D
multi-user environment to
immerse children, ages 9-16, in
educational tasks. QA combines
strategies used in the
commercial gaming
environment with lessons from
educational research on learning
and motivation.
CBA (S)
Virtual Worlds / games (FS)
Simulations (FDS)
Citizenship
6 Social and civic competences
CBA (DS)
Peer/Self Assessment (FDS)
Virtual Games / Simulations (FDS)
Global Conflicts
http://www.globalconflicts.eu/
Example: Child labour in
Bangladesh:
http://www.youtube.com/wat
ch?v=mf9Ju86IVYw
Educational game, designed to help teach concepts in citizenship, geography, and media.
Developed by Serious Games International, it has detailed lesson plans and assignments for students.
Examples
•PART 2
•CRITICAL INSPECTION
CBA for
Digital Competence
4
Digital competence
CBA (S)
Simulations (FDS)
Digital Competence
→ Focus on knowledge
Internet and Communication Core Certification (IC3) exam sample question
4
Digital competence
CBA (S)
Digital Competence: IC3
→ Focus on operational skills
Internet and Communication Core Certification (IC3) exam
sample question
http://www.certiport.com/Portal/common/htmllibrary/SalesVideo-Demos/demo_ic3.html
CBA for
Digital Competence
Simulations (FDS)
Entrepreneurship
Questionnaire
7 Sense of initiative and entrepreneurship
CBA (DS)
Peer/Self Assessment (FDS)
Virtual Games / Simulations (FDS)
Entrepreneurship Questionnaire
→ Focus on personality/disposition as a static fact
?
https://www.careeringahead.
com.au/survey/Entrepreneurs
hip/Generic/questions.asp
Conclusions
ICT offers powerful, engaging assessment formats, in particular:
CBA
Quizzes /
simple games
ePortfolios
Basic literacy
(foreign/mother tongue)
Creative writing
(foreign/mother tongue)
Basic mathematical
competence
Cultural awareness and
expression
Caveat:
Self / Peer
Assessment
Learning
to Learn
Simulations
Virtual worlds/ games
Scientific competence
Digital
competence
Social and civic
competences
Educational
software
Mathematical
competence
CBA lends itself to simplified, knowledge-based assessment
→ Pedagogy comes first! Careful selection of question formats
ICT Assessment approaches are underdeveloped for:
Learning to Learn
Sense of initiative and entrepreneurship
Social and civic competences
The Paradigm Shift
Era of Computer-Based Assessment (CBA)
Era of Technology-enhanced learning
Focus on: Efficient testing
1990
1995
CBA
(Gen.1)
Focus on: Personalised learning
2000
Adaptive
Tests
(Gen.2)
2005
2010
2015
2020
2025
Technology
Automatic
Scoring
Gen. ReInvention
Learning
2.0
Immersive
Learning
Learning
Analytics
Intelligent
Tutors
Pedagogy
Efficient
Testing
Validity of
Scores
Quick
Feedback
Authentic
Contexts
?
→ Primacy of Pedagogy:
Pedagogical guidelines for intergrated assessment
?
?
?
• Thank you very much for your attention!
• Christine Redecker
• Yves Punie
• Christine.Redecker@ec.europa.eu
• Yves.Punie@ec.europa.eu
http://ipts.jrc.ec.europa.eu/public
ations/pub.cfm?id=5719
further information ……
http://is.jrc.ec.europa.eu
Newsletters:
http://is.jrc.es/pages/newsletter.html
Publications:
http://is.jrc.ec.europa.eu/pages/Publications.html
Contact:
jrc-ipts-is-NEWS@ec.europa.eu