An Application of Standard-Setting
Methods in a Formative Assessment
in Digital Responsibility among
Norwegian Eight Graders.
Ove Edvard Hatlevik and Ingrid Radtke
www.iktsenteret.no
Agenda
• Introduction
• Digital skills in the curriculum
• Digital responsibility
• Experience with standardsetting
Norwegian Centre for ICT in education
• Governmental agency
• Activities
– Online guidelines for schools and educators
– Summarizing experiences with methods and tools
in the classroom
– Surveys and monitoring
Assessment
• Summative assessment
• Formative assessment
– For example voluntary learning supportive tests
– Feedback on how to improve
What to measure?
• Concept
– Digital competence or ICT literacy or …
• Framework
– E.g. the DIGCOMP framework (Ferrari, 2013) has
five areas with 21 dimensions and three
proficiency levels
– European Commission, Joint Research Centre, Institute for
Prospective Technological Studies
Different concepts
• ICT literacy (ETS, 2006; Ainley et al., 2007)
– Computer and Information literacy (Fraillon et al.,
2013)
– The International Computer and Information Literacy Study
(ICILS) 2013
•
•
•
•
Digital literacy (Aviram & Eshet-Alkalay, 2006 )
Digital competence (Erstad, 2006; Ferrari, 2012)
Internet skills (Van Deursen & Van Dijk, 2009 )
iSkills (ETS, 2008)
About the concepts
• Often tools or technology (e.g.
ICT/Digital/Internet) combined with learning
domains (e.g. skill, competence, literacy)
• Definitions often include modes (critically or
reflectively) or purposes (work or leisure)
One example of an international used
definition of CIL
• Computer and information literacy refers to:
“individual’s ability to use computers to
investigate, create, and communicate in order
to participate effectively at home, at school, in
the workplace, and in society” (Fraillon et al.,
2013, p. 17).
– The International Computer and Information Literacy Study
(ICILS) 2013
International frameworks
• iSkills (Educational testing service, 2006)
• ATC21s (Binkley et al, 2012)
• DIGCOMP (Ferrari, 2013)
• Computer and information literacy (Fraillon et al,
2013)
• Australian Curriculum, Assessment and Reporting Authority
(ACARA) (2012)
National initiatives to assess some of
these concepts / frameworks
•
•
•
•
•
Australia
Chile
Norway
Korea
U.S.
• International Computer and Information
Literacy Study 2013 (Fraillon et al., 2014)
National Curriculum (Norway)
• Digital skills as a “key competence”
• Sources describing digital skills
– A general framework of basic skills
– A general description within the subjects
– Explicit competence aims for the subjects
• The Norwegian Directorate for Education and Training
Digital skills …
• … involve being able to use digital tools, media
and resources efficiently and responsibly, […]
• … include developing digital judgement by
acquiring knowledge and good strategies for
the use of the Internet.
• The Norwegian Directorate for Education and
Training (2012, p. 12)
General framework of digital skills
To acquire and
process digital
information
To produce digital
information
Digital judgement
Communication
Example of a digital judgement task
from the ICILS 2013
Example of description: Digital skills in
Natural science involves
• … […] mastering
searching and
research strategies, learning to
evaluate sources critically and
selecting relevant information
about themes within natural sciences.
Examples of one competence
aims (social science):
• use digital tools to present work in the social
sciences and comply with the rules for
personal data protection and copyright.
Assessment team
• 7 employees at the ICTsenter
• Different skills and
background
- teaching-experience
- ict-skills
- research and statistics
- assessment skills
Assessment in Digital Responsibility
• One out of 5 formative assessments in digital
competence
• In cooperation with the Directorate of
Education
• Available from november 2015
Assessment development
• 3 rounds with piloting (and visiting schools)
• Analyses with classical test-thoery and IRT
• Standard setting with Angoff- and Bookmarkmethod
Definition Digital Responsibility
• Digital judgement means being able to use
digital tools, media and resources in a
responsible manner, and being aware of rules
for protecting privacy and ethical use of the
Internet.
• Search and process: … interpret digital
information appropriately and critically.
• Produce …. Be aware of copyright issues.
Digital Responsibility
• Items were developed in 6 content areas
– critical use of digital sources
– ownership of digital material
– safety issues
– personal integrity
– cyber bullying
– green data
Scoring and item-types
• scoring - each correct
answered item gives
one score.
• item-types - multiple
choice and multiple
response, knowledge
matrix, matching, pulldown list and drag and
drop.
Exsamples for Competence Aims
Norwegian
• use different types of digital
and paper-based
dictionaries
• be familiar with copyright
rules on the use of sources
Application of Digital Responsibility
• Use different types of digital
dictionaries
–
–
–
–
–
–
critical use of digital sources
safety issues
personal integrity
ownership of digital material
cyber bullying
green data
Development of Assessment
• Autumn 2013, discussion and small pilot with
2 classes
• Spring 2014, 90 items with 1395 students
from 36 schools
• Nov 2014, 60 items with 1062 students from
26 schools, representative sample
Final assessment – 50 items
Performance standards
• Three performance
standards
• Low – middels- high
• Lowest level was
divided into two
Performance standards
• Related to the items in
the test
• Not the ‘ideal’ student
Level 1
At this level, the pupil has basic skills in the area of digital judgment that encompasses
understanding of basic concepts and risks of using digital tools.
The pupil can make simple search for sources, have knowledge of TONE-term (method used
to estimate credibility, objectivity, accuracy and suitability of digital sources) and can choose
appropriate sources. When it comes to copyright pupils can be expected to have an
understanding about what is right or wrong when referring to sources and downloading
files.
Privacy:
• Simple assertions about what is allowed to publish when it comes to photos or
videos online
• What kind of information pupils should provide about themselves and what
dangers are associated with registration at various websites
• Storing data and using search engines from their mobile phone
• Simple concepts around privacy, like permission and consent
Safety:
• Protect your own password when logging onto someone else’s PC
• Dangers around using free wireless networks
Standard setting workshop
• 10 experts
• Help from Cito to
conduct the workshop
Expert panel
• Teacher with hands on experience with the
use of ICT in schools
• Other subject-matter experts
• Stakeholders?
About the workshop
• Introduction of 3 performance standards
• Training session with Angoff-method
• Both Angoff- and Bookmark-method were
used
Angoff method
• Suppose a group of 100
borderline students.
How many of them will
answer this item
correctly?
• Cut-off points. Sum
judgements.
• Reflections
Bookmark method
• List with ordered items
by difficulty after IRTanalyses
• Experts choose the
items which mark the
borderline for the levels
of the students.
• Cut-off points: average
of the bookmarks set by
experts.
Results
• About the same cutting point for the higher
level, level 2-3
• Higher cutting point for the lower level after
Angoff-method, level 1-2
• Standard-deviation
Decision-making
• Which factors would be
important?
1. Distribution of items and students
• Number of items to qualify for a certain level.
• Number of students on that level. (As seen
from the pilot-data.)
• Difficulty of items around the cutting-score.
2. More validity arguments
• Level 3 should be exclusive!
• Not too many students in Level 1!
Guidance material
• Group-level
• Item-level
• Organisation-level
Way forward!
• Limitations?
• Science versus individual judgements
• Validity – accept among teachers
• Aim is to improve teaching and learning!