Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems:
A peek into the future. In S. D. Craig, & D. Dicheva (Eds.), Proceedings of the Artificial Intelligence and Education (AIED '09)
Workshop on Intelligent Educational Games (pp. 99 – 109). Brighton, UK. http://myweb.fsu.edu/vshute/pdf/peek.pdf
Education, Writ Large
Education is a social process.
Education is growth. Education is
not a preparation for life;
education is life itself. –Dewey
Why Education Needs to Change
The world has changed a lot in the past 100 years. Education has not.
Classroom photo, 1910.
Classroom photo, 2010.
Why Assessment Needs to Change
Learning & Assessment Definitions
LEARNING: Lifelong process of
accessing, interpreting, and
evaluating information & experiences
into knowledge, skills, behaviors,
values, identities, and dispositions.
ASSESSMENT: Collecting,
analyzing, and interpreting info about
student understanding and/or
performance re: educational goals.
Various procedures used for
evaluating student work & learning.
Teach.
Stop.
Test.
Repeat.
Assessment
Collaborative Learning
Fuse Learning & Assessment
Assessments for
Learning
Informative
Assessments
Assistments
Educative
Assessments
ACED
• Black & Wiliam, 1998a; 1998b
• Stiggins, 2002
• Bass & Glaser, 2004
• Razzaq, Feng, Nuzzo-Jones, Heffernan, Koedinger,
Junker, et al., 2005
• Wiggins, 1998
• Shute, Hansen, & Almond, 2008
• You can’t fatten a hog by weighing it, or can you?
• http://myweb.fsu.edu/vshute/pdf/shute%202008_a.pdf
Themes
Seamless & Ubiquitous
Assessment
Comprehensive
Learner Models
Theme 1
Theme 2
Assessment Info for
Decision Making
Theme 3
Aligned along 3 dimensions: vertically, horizontally, and temporally.
Setting Up A Metaphor for Theme 1
On June 26, 1974, the first product
with a bar code was scanned at a
check-out counter.
It was a 10-pack of Wrigley's Juicy
Fruit chewing gum.
Today, the pack of gum is on
display at the Smithsonian
Institution's National Museum of
American History.
Theme 1 Metaphor
(via Pellegrino, Chudowsky, & Glaser, 2001)
Business
 Before 1974, businesses had to close down once a year
to take inventory.
 Barcodes enabled access to a continuous stream of
information to monitor inventory.
 Now businesses continue without interruption,
supporting real-time inventory management.
Theme 1 Metaphor
(via Pellegrino, Chudowsky, & Glaser, 2001)
School
 Schools shouldn’t have to interrupt normal instruction to
give external tests.
 Instead, assessment should be continual & invisible,
supporting real-time, JIT learning.
 This is the first of 3 themes envisioned, 20 years hence
(seamless & ubiquitous assessment).
Stealth Assessment
 Advances in measurement let us administer
evidence-based assessments to:
 Extract ongoing information from a learner
 Make accurate inferences of competencies
 React in immediate and helpful ways.
 Accomplished via automated scoring and
machine-based reasoning techniques.
 Seamlessly woven into the fabric of the
learning environment.
How Does It Work?
 Stealth assessment can be
used to improve learning:

Gathering evidence for accurate
estimates of competencies
 Info on competencies used by teachers,
the system/game, and/or students
 Current estimates of
competency levels integrated
into the game and displayed as
progress indicators.
 This elevates valued
competencies to the same
level as health and weapons!
Theme 1: Summary
“Seamless and Ubiquitous Assessment”
Idea: A continuous process fusing assessment and
learning.
Seamless: Removal of false boundaries between
learning and assessment.
Ubiquitous: Constant nature and need to
feedback back results of assessment into learning.
Theme 2: Comprehensive Models
Current
Goal State
State
Theme 2: Comprehensive Models
 Research needed to identify,
model, and assess 21st century
competencies as well as other
attributes, like affective
variables, values, dispositions,
etc.
 These will comprise rich
evolving profiles representing a
synopsis of what’s known, what
can be done, what’s believed,
preferred/valued, and felt.
ECD focuses on the evidence of competencies.
It provides a way to reason about assessment design, and
a way to reason about learner performance. As an assessment
framework, it can support both Theme 1 (seamless/ubiquitous
assessment) and Theme 2 (comprehensive models).
Assessment Design
Competency Model
What do you want to say about the
person?
Evidence Model
What observations would provide best
evidence for what you want to say?
Task/Action Model
Model
What kinds of tasks let you make the
necessary observations?
Design and Diagnosis
For more information about ECD, please see ECD for Dummies.
Example Conceptual Model
Example Computational Model
Summary of ECD
ECD should be used as the framework
for new assessments because it can:
 Yield valid assessments for different
purposes
 Provide accurate estimates of complex
competencies and dynamic performances
 Aggregate information from various sources
 Afford transparency to stakeholders via
evidentiary reasoning to support claims.
Wrapping It Up
 New competencies need to be
identified, modeled, assessed
and supported.
 Assessments need to be valid
and reliable to meet our
educational needs (e.g., ECD
and stealth assessment).
 Immersive games are fun,
and enable learning within
complex and relevant
environments.
Themes over time
Theme 2
Theme 1
Theme
20 years…
Seamless
assessment
Learning-related programs/devices that students use (games,
learning environments, applications) employ seamless assessment.
Ubiquitous
assessment
Different platforms are interoperable re: delivery, analysis, and
support of key comps, relevant data, and inferences linking them
altogether.
Comprehensive
Model: Identifying
key comps
Wide range of important skills and other attributes identified and
justified (e.g., causal reasoning, collaboration, critical thinking,
reflexivity, self-regulation, adaptability, focused attention, systems
thinking, interdisciplinary thinking, problem solving, compassion, etc.)
Comprehensive
Model: Modeling
Robust (conceptual & computational) models of all comps, as well as
their interactions. Vertical scales for key competencies (e.g.,
trajectories).
Comprehensive
Model: Assessing
Most assts intimately combined within learning experiences thus
rarely need summative asst (e.g., only for auditing purposes).
Assessment data accumulated into lifelong portfolios/profiles.
Major Funding Targets
Fusing assessment and learning
Understanding full complement of
characteristics involved with learning
Rendering assessments useful to all
parties
Thank you!
Questions?
Email:
vshute@fsu.edu
Website: http://myweb.fsu.edu/vshute
Collaboration
Critical Thinking
Curiosity
Grit
Go Back
Systems Thinking
Creativity
Empathy
Benefits
Retention/
Efficiency
Transfer
National
Trend
Validity/
Accuracy
Barriers
(or Opportunities!)
Where’s GeneralContext? izability
Buy-in
Interoperability
Privacy