Internet based assessments via collaborative problem
solving: The Assessment and Teaching of 21st Century
Skills (ATC21S™) Project
Patrick Griffin
Executive Director, ATC21S
PROJECT GOVERNANCE
EXECUTIVE BOARD
Patrick Griffin, University of Melbourne,
Executive DirectorATC21S
Michael Stevenson, Cisco Vice President Global Education, Former ATC21S Board
Chair (2009–2010)
Ministerial representatives from…
–
Australia:
–
Finland:
–
Singapore
Anthony Salcito, Microsoft Vice President
- Education, Former ATC21S Board Chair
(2010–2011)
–
USA:
–
Costa Rica:
Esther Care, University of Melbourne,
ATC21S International Research
Coordinator
–
Netherlands:
Shelly Esque, Intel Vice President - Legal
and Corporate Affairs , Current ATC21S
Board Chair (2011–2012)
PROJECT GOVERNANCE
ADVISORY BOARD
Patrick Griffin, University of Melbourne, Executive
Director ATC21S (Chair)
Andreas Schleicher, Organization for Economic
Co-operation and Development (OECD)
Seamus Hegarty, International Association for the
Evaluation of Educational Achievement (IEA)
Irina Bokova (Director General UNESCO)
Ray Adams, Technical Director PISA 2003-2012
Marc Durando, European Schoolnet
Esther Care, University of Melbourne, ATC21S
International Research Coordinator
Stuart Elliott, National Academy of Sciences
David Forster, International Testing Commission
Robin Horn, World Bank
Eugenio Eduardo Severin, Inter-American
Development Bank
National project managers from each founder
country and associate country
Task force member from each company:
Katrina Reynan, Director, Cisco
Martina Roth, Director, Intel
Greg Butler, Director, Microsoft
Focus: the Assessment and Teaching of 21st Century
Skills
 Traditional assessments may not be suited to measure many
21st century skills
 Goal is to develop new assessment approaches matched to
new C21 skills and to advise systems, schools and teachers on
the use of assessment data to help students develop higher
order performances
Phase 1
Conceptualise
C21 skills and
education
output needs
Phase 2
Skill
Identification
and
hypotheses
Phase 3
Development
and coding
via coglabs
Phase 4
Pilot studies
and trials
Phase 5
Dissemination
scale and
policy
CONCEPTUALIZING THE SKILLS
 Assemble experts
 Explore practical and technical needs in the classroom
Five working groups were established:

Defining 21st-century skills: Ms. Senta Raizen, WestEd

Methodological issues: Dr. Mark Wilson, University of
California, Berkeley

Technological issues: Dr. Beno Csapo, University of
Szeged, Hungary

Classrooms and formative evaluation: Dr. John
Bransford, University of Washington, and Dr. Marlene
Scardamalia, University of Toronto

Policy frameworks and new assessments: Dr. Linda
Darling-Hammond, Stanford University
 Define methods
 Create a new framework
21ST-CENTURY SKILLS DEFINED
ATC21S started by internationally defining 21st-century skills as
four broad categories.
Ways of
thinking
Ways of
working
Tools for
working
Living in the
world
Critical
thinking
Communication
Information
Literacy
citizenship
Problem
solving
collaboration
ICT
Literacy
Life and
Career
Learning to
learn
Personal
responsibility
Metacognition
Social
Responsibility
Problem
Solving
hypotheses
generalisations
Rules
Patterns
Elements
Collaborative
Problem
Solving
Cognitive
Social
Task
Regulation
Participation
Learning
through Digital
Networking
Intellectual
Capital
Social Capital
Producer
Knowledge
Building
Perspective
Consumer
Social
Regulation
Ways of
thinking
Ways of
working
Tools for
working
Living in the
world
Critical
thinking
Communication
Information
Literacy
citizenship
Problem
solving
collaboration
ICT
Literacy
Life and
Career
Learning to
learn
Personal
responsibility
Metacognition
Social
Responsibility
Problem
Solving
hypotheses
generalisations
Rules
Patterns
Elements
Collaborative
Problem
Solving
Cognitive
Social
Task
Regulation
Participation
Learning
through Digital
Networking
Intellectual
Capital
Social Capital
Producer
Knowledge
Building
Perspective
Consumer
Social
Regulation
Collaborative problem-solving
ATC21S Assessment and Teaching of 21st Century Skills
The University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org
Components
Collaborative
problem
solving
Social skills
Participation
Perspective
taking
Cognitive skills
Social
regulation
Task regulation
Knowledge
building
Ways of
thinking
Ways of
working
Tools for
working
Living in the
world
Critical
thinking
Communication
Information
Literacy
citizenship
Problem
solving
collaboration
ICT
Literacy
Life and
Career
Learning to
learn
Personal
responsibility
Metacognition
Social
Responsibility
Problem
Solving
hypotheses
generalisations
Rules
Patterns
Elements
Collaborative
Problem
Solving
Cognitive
Social
Task
Regulation
Participation
Learning
through Digital
Networking
Intellectual
Capital
Social Capital
Producer
Knowledge
Building
Perspective
Consumer
Social
Regulation
Participation skills over levels
Low



Middle 



High





Peripheral participation
Low subjective responsibility for outcomes of
collaboration, leading to lurking behavior
Simple epistemological beliefs (knowledge is perceived
as fixed and to be transmitted from teacher/textbook to
learner)
Activity in scaffolded environments
Responding to cues in communication
Medium subjective responsibility for outcomes of
collaboration
Developed epistemological beliefs (knowledge is
perceived as fixed, but can be elaborated through
communication and collaboration)
Initiating and promoting interaction
Activating and scaffolding others in participation
Ensuring equal participation rates among group members
High subjective responsibility for outcomes of
collaboration
Sophisticated epistemological beliefs (knowledge is
perceived as fluid, constructed, and inherently
social/collaborative in nature)
Perspective taking skills
Low





Middle





High





Low levels of empathy
High egocentric bias
Social projection (expectation of others as highly similar to
oneself)
Ignoring contributions from others
Contributions are not tailored to participants
Medium levels of empathy
Medium level of egocentric bias
Receptive ability (being able to understand what others want to
convey, e.g. from overhearing)
Contributions from others are taken into account
Contributions are moderately tailored to recipients
High levels of empathy
Low or no egocentric bias
Contributions from others are embraced and contextualized with
respect to collaborators’ opinions and skills
Eliciting contributions from others (e.g. through questions)
Contributions are tailored to recipients (audience design)
Social regulation skills
Low




Low tolerance for ambiguity
Competitive or individualistic social value orientation
Low readiness to negotiate joint understanding
Tendency to withdraw after conflict arises
Middle




Cooperative social value orientation
Attempts to negotiate joint understanding
Conflicts will be avoided
Initiation of compromises
High




Pro-social attitudes
Strategies for conflict resolution
Conflicts are regarded as productive tensions
Initiation of successful compromises
Task regulation skills
Low




Trial and error hypothesis testing
Unorganized sequence of solution attempts
Little or no goal setting
Variety of taskwork mental models will be ignored
Middle




High





Forward search through a problem space
Organized sequence of solution attempts
Setting of unspecific goals
Variety of taskwork mental models will be taken into
account
Reflective regulation
Forward and backward search through a problem space
Strategic oversight over collaborative strategy
Setting of specific goals
Variety of taskwork mental models will be harnessed
productively
Knowledge building skills
Low




Knowledge telling
Sharing of information
Isolated contributions
Lack of argumentation patterns
Middle




Critical analysis of information
Building on input from others
Adding artifacts
Forming of incomplete arguments
High



Knowledge transforming
Integration and synthesis of multiple artifacts
Forming of complete, proper arguments (explanatory
coherence)
Developin assessment Tasks for ATC21S

• Founder countries


• Associate and founder
countries

Draft -- Concept check
for reality
Panel – cognitive
laboratory for codes
Pilot – for
administration
Trials – for calibration
Tasks and Assignments to assess ICT Literacy
Poetry
(Graphic Organizers,
Creating and Listening
to Audio and video)
Arctic Trek
(Collaborative
Notebook, Information
Foraging)
ATC21S
• Twenty first century skills
Collecting the Data
Coding and scoring
Social Behaviour
U2L001A = presence of chat before any action
Cognitive Behaviour
U2L00 4A = All positions hav been covered (providing player had access to 3
balls)
U2L006A = sewuential placement of balls -6 combinations
LMRLMR
RMPRML
RMLLMR
LMRRML
LLMMRR
RRMMLL
Calibration and Interpretation
Element
Amplification
Low
Middle
High
Participation
1.Action
Activity within environment
2.Interaction
Interacting with, prompting and
responding to the contributions
of others
Undertaking and completing a
task or part of a task individually
3.Task completion
No or very little activity in
environment
Acknowledges
communication
Activity in scaffolded context
Maintains presence only
(lurking)
Identifies and attempts the
task
Perseveres in task as indicated by
repeated attempts or multiple
strategies
Contributions or prompts
from others are taken into
account
Contributions are modified
for recipient understanding in
the light of feedback
Contributions or prompts of others
are adapted and incorporated
appropriately
Contributions are tailored to
recipients (audience design)
Negotiates through differences in
perspective
Responding to cues in
communication
Activity in scaffolded and
unscaffolded contexts
Initiating and promoting
interaction or activity
Perspective taking
4.Responding adaptive
responsiveness
Ignoring, accepting or adapting
contributions of others
Contributions or prompts
from others are ignored
5.Audience awareness
Mutual modelling
Awareness of how to adapt
behaviour to increase suitability
for others
Contributions are not
tailored to participants
6.Negotiation
Achieving a resolution or
reaching compromise
No attempts to negotiate
joint understanding
Comments on differences in
perspective
7.Metamemory
Self concept
Recognising own strengths and
weaknesses
Notes own performance
Comments on own
performance in terms of
appropriateness or adequacy
Social regulation
8.Transactive memory
9.responsibility Initiative
11. Resource management
Comments on own performance
in terms of appropriateness or
adequacy in the context of the
task
Comments on performance of Comments on performance of
Recognising strengths and
Notes performance of
others in terms of
weaknesses of others
others
others in terms of
appropriateness or adequacy appropriateness or adequacy in
the context of the task
Assuming responsibility for
Undertakes activities largely Completes activities and
Assumes group responsibility as
ensuring aspects of task are
independently of others
reports to others
indicated by use of second
completed by the group
person plural or accepting
others’ contribution
Managing resources or people to Uses resources (or directs
Suggests that people or
Allocates people or resources to
complete a task
people) without
resources be used in part of a task through to completion
Element
Amplification
Low
Middle
High
Knowledge Building
20. Knowledge acquisition Follow path to gain knowledge Acquires knowledge as a
12. Rules “If …then”
(planning and executing)
Deliberate single actions to
acquire knowledge
Knowledge acquired through
multiple purposeful actions
Integrates and synthesises of
multiple pieces of information
result of being given it
directly
Formulating a course of action Activity is undertaken with Identifies short sequences of Identifies potential multiple
little or no prior formulation actions for a specific task
sequence routes for a complex
to address a problem or task
for a course of action
task
Making connections between
elements of knowledge
Focused on (acts/shares)
isolated pieces of
information
Building on input and
information from others
18. Hypothesis “what if…” Changing from one line of
(reflecting and monitoring reasoning or course of action
to another as information or
circumstances change
Maintains a single line of
approach
tries multiple options in light Reconstructs and reorganizes
of new information or lack
understanding of the problem in
of progress
light of ne information or
opinion.
19. Relationships
(representing and
formulating)
Task Regulation
15. Collecting Elements
Explore and understand
Does not recognize the need Identifies the need for
for further information
specific information related
o immediate activity
Identifies need for varied
information related to multiple
activities.
13. Systematicity
Implementing possible
solutions to a monitoring
progress
Trial and error hypothesis
testing in an unorganized
sequence of solution
attempts
Forward and backward search
through a problem space with
reflective solution attempts
17. Tolerance for
ambiguity
Accepting ambiguous
situations and exploring
options within these
Maintains only a presence in notes ambiguity and
situations where there is
suggests options
ambiguity
14. Organising (Problem
analysis)
Problem is stated as initially Problem is divided into sub
Analysing and defining a
problem in familiar language represented either explicitly problems
(i.e. Making the problem more or implicitly
manageable and meaningful)
Forward search through a
problem space with an
organized sequence of
solution attempts
Explores the problem space
Problem is divided into sub
problems and their inter
dependence is recognised
Dimensionality
Cognitive Strand
Level Descriptions
- cognitive
Level 5 Refined strategic application & problem solving
At this level, students can recognise previous errors and transfer and use that knowledge to more complex sub
task pages. They recognise resources which they concluse have caused previous errors and avoid these. They
are developing a level of confidence in dealing with the task and this allows them to correct or override their
partner’s responses/answers/ actions. Students share appropriate resources with their partner and use
consistent strategies that have they have identified as effective in previous steps in the problem solution. Any
planning and most strategies being implemented are within a team context.
Level 4 Systematic and methodical
At this level, students begin to solve problems through a collaborative process and use more effective coworking strategies. By now they can simplify the problem and review previous sub task implementations
several times to determine whether the rule is transferable in subsequent task pages. Exchanging information
between partners and using the shared information efficiently . Their sequencing and trial and error actions
require few attempts and are hence completed in an optimal amount of time/attempts. Actions appear to be
well thought out and planned and each action appears purposeful They can identify cause and effect and use
suitable strategies to gain a correct path solution. The student directs and guides the partner in appropriate
actions and resources.
Level 3 Strategic Planning Organising & Executing
At this level the student can complete lower difficulty tasks correctly and independently. They are able to
complete the final step or sub task and assist their partner in this process. They increase their working
together by planning strategies of working, goal setting and sharing their resources with their partner. The
student tends to focus on the relevant resources and disregards those that posed no benefit in previous trials.
The student is able to conclude mutually agreeable answers when required. In multi-page tasks the student
teds to identify successful approaches and resources and apply the same rules repeatedly. The student
continues to intensify the sequential investigations and systematic behaviour in subsequent task pages that
have increased in difficulty. This leads to attempts to test hypotheses together with the partner and to the
development of increasingly thorough trial and error behaviour.
Level 2 directed approach Systematic Trial & Error
At this level, the student identifies possible cause and effect of actions and demonstrated an initial
understanding of the task concept and begins testing hypotheses and rules. The student uses successively
enhanced sequential trials or systematic exploration to increase this knowledge.
they may try to alter the techniques they use to investigate or explore the change to the task they are making.
They start to share information with their partner, and to identify differences of resources and information
that each partner has. They seek information about their partners resource set and provide information about
their own to the partner. They reorganise information and begin to distinguish between the resources
relevant the irrelevant to the problem space.
Level 1 Trial and Error Exploration
Students at this level are primarily exploring the task environment, clicking on various resources often in a
random fashion. They may engage in singular / unilateral approaches to trial and error in an attempt to build
knowledge of the problem space. They participate individually in the activity and attempt to problem solve
through an apparent unsystematic guessing approach. Their strategy is limited to following the specific
instructions provided unable to proceed without instructions. They tend to repeat errors or reproduce
unproductive actions with no clear indication of advancing through the task for several attempts. They may
social
Social Strand
Level 5- Cooperation, Shared Goals and Appreciation
At this level, the student works collaboratively with partner for knowledge building and problem solving
solutions. The student engages with the partner early in the task and there is discussion of planning and
strategy. Feedback from partner is incorporated and/or adapted to identify solution paths or modify incorre
ones. Where there are conflicts (e.g. missing information or inaccurate information from partner), the stude
manages them successfully to reach common understanding / agreement with partner before proceeding o
possible solution path.
Level 4 Mutual Commitment
At this level, the student works collaboratively with partner throughout the problem solving process. He/She
promotes meaningful interaction with partner, exchanging information/ resources and approaches as well a
incorporating ideas from partner. The student also reports his/her own activities to partner. Differences in
points of view (e.g. inaccurate information / irrelevant resources) are managed much more effectively and
there are attempts to reach consensus to progress on the activity.
Level 3 Resolving Differences
At this level, the student becomes aware of the partner role early in the collaborative problem solving proce
and recognises the need to engage with partner for the problem solving task. The student initiates and
promotes interaction with partner; sharing resources, information and ideas. The student attempts to clarify
differences (e.g. irrelevant resources from partner) in order to reach agreement (not necessarily consensus)
progress on the task. The student acts as though they are aware of his/her own strengths and challenges in
collaborative problem solving.
Level 2 Awareness of Partnership
At this level, the student works independently on some parts of the activity but is also beginning to develop
awareness of partner role which can impact on the chances of completing the task successfully. He/she
interacts with partner and requests resources / information. The student also starts to tailor communication
with partner to improve mutual understanding of the communications exchange. The student may have
conflicts with his/her partner but there is an attempt to clarify differences.
Level 1 Clarification and Independence
At this level, the student is working independently on the activity using only his/her own ideas. There is limit
interaction with partner and this is mainly prompted by task instructions. The student acknowledges
communication cues by partner. However, the student’s communication with partner elicits requests for
clarification and restatement Disagreements lead to conflicts that are not resolved by the end of the activity
Developmental Progressions
Social Strand
Cognitive Strand
Level 5- Cooperation, Shared
Goals and Appreciation
Level 5 Refined strategic
application & problem solving
Level 4 Mutual Commitment
Level 3 Resolving Differences
Level 4 Systematic and
methodical
Level 3 Strategic Planning
Organising & Executing
Level 2 Awareness of
Partnership
Level 2 directed approach
Systematic Trial & Error
Level 1 Clarification and
Independence
Level 1 Trial and Error
Exploration
Feedback and Reporting
Reports
Reporting to Students
Learning Readiness
Real time reports will be
available, based on empirically
developed learning
progressions. They will identify
a point of readiness to learn for
each student.
This type of report will be linked
to teaching interventions
associated with readiness to
learn as indicated by the black
bar in the spine of the chart.
Instructional groups
Teacher development
Professional
development
modules
1.
2.
3.
4.
5.
Defining and
Assessing 21st
Century Skills
Using a
Developmental
Model
ATC21S
Assessments:
Getting Started
Interpreting Reports
Teaching and
Learning 21st
Century Skills
The System
An integrated system
–
–
–
–
Developmental progressions – standards referenced
Assessment tasks with automatic scoring
Teaching intervention and professional development
Reports for students, teachers and systems
Next steps
Access and Use
Validation studies
Expanded resources – mainstream curriculum
Other 21st century skills
ATC21S Assessment and Teaching of 21st Century Skills
The University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org
Transfer and Utility
• To what degree will 21st century skills such as
collaborative problem solving or ICT Literacy in
Learning Networks facilitate student learning?
– Are these skills teachable and learnable?
– Will the skills enhance learning in the classroom?
• To what degree might enhanced 21st century skills
contribute to adaptiveness of graduate students to
the workforce?
ATC21s New Users?
Follow up!
Implications
Conventional Assessment
• Correctness if the important
part of the response
• Response can indicate
reasoning ability of a
specific level
• Difficult to interpret various
incorrect responses
• Additipnal data is lost in the
focus on the correct
response
Future Applications
Task design
I.
Design tasks to capture specific components of cognitive
ability
I.
II.
III.
IV.
II.
Background data capture allows more user friendly and engaging
asssessments
I.
II.
III.
IV.
V.
VI.
III.
working memory
Processing speed
Pattern recognition
Systematicity’
Reduce test anxiety
counteract teaching to the test
Capture multiple variables
Reduce test time need
Increase assessment efficiency
Better, richer information for teachers
Applications in serious game industry
Roll Out and Dissemination
What is in the system?
• Two models - cloud and portable
• Prototype assessment tasks
• Developmental progressions
• Reporting and feedback for schools, teachers and students
• Professional development modules
• Specifications for developers
Portable and Cloud Systems
Portable version
Cloud version
Local adaptations
Centralised control/management
Decision-making autonomy
Centralised decision-making
Customisation enabled
Embed or link within existing systems
Local adaptations
Addition of tasks
No customisation
Local support structure
Centralised support structure
Flexible registration system
Rigid registration system
Flash
Less susceptible to differences across
browsers
Not currently supported on iPad*
HTML 5
More susceptible to differences
across browsers
PISA 2015
•
Collaborative problem solving to be tested in OECD countries in 2015.
PISA will test human to computer interaction
•
Collaborative Problem solving involving human to human maybe in
2018