Evidence-Centered Design and Cisco’s Packet
Tracer Simulation-Based Assessment
Robert J. Mislevy
Professor, Measurement & Statistics
University of Maryland
with
John T. Behrens & Dennis Frezzo
Cisco Systems, Inc.
December 15, 2009
ADL, Alexandria, VA
December 15, 2009
ADL
Slide 1
Simulation-based & Game-based Assessment

Motivation: Cog psych & technology
» Complex combinations of knowledge &
skills
» Complex situations
» Interactive, evolving in time, constructive
» Challenge of technology-based
environments
December 15, 2009
ADL
Slide 2
Outline
 ECD
 Packet
Tracer
 Packet Tracer & ECD
December 15, 2009
ADL
Slide 3
ECD
December 15, 2009
ADL
Slide 4
Evidence-Centered Assessment Design
Messick’s (1994) guiding questions:



What complex of knowledge, skills, or other
attributes should be assessed?
What behaviors or performances should
reveal those constructs?
What tasks or situations should elicit those
behaviors?
December 15, 2009
ADL
Slide 5
Evidence-Centered Assessment Design




Principled framework for designing,
producing, and delivering assessments
Process model, object model, design tools
Explicates the connections among
assessment designs, inferences regarding
students, and the processes needed to
create and deliver these assessments.
Particularly useful for new / complex
assessments.
December 15, 2009
ADL
Slide 6
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Layers in the assessment enterprise
From Mislevy & Riconscente, in press
Packet Tracer
December 15, 2009
ADL
Slide 8
Cisco’s Packet Tracer





Online tool used in Cisco Networking Academies
Create, edit, configure, run, troubleshoot networks
Multiple representations in the logical layer
Inspection tool links to a deeper physical world
Simulation mode
» Detailed visualization and data presentation

Standard support for world authoring
» Library of elements
» Simulation of relevant deep structure
» Copy, paste, save, edit, annotate, lock
December 15, 2009
ADL
Slide 9
Instructors and students can author
their own activities
December 15, 2009
ADL
Slide 10
Instructors and students can author
their own activities
December 15, 2009
ADL
Slide 11
Instructors and students can author
their own activities
Explanation
Experimentation
Packet Tracer & ECD
December 15, 2009
ADL
Slide 15
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Layers in the assessment enterprise
From Mislevy & Riconscente, in press
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Design structures: Student, evidence, and
task models. Generativity.
Assessment
Implementation
• Assessment
argument
Manufacturing
“nuts & bolts”:
authoring
automated
scoring
• tasks,
Design
Patterns
details, statistical models. Reusability.
Assessment Delivery

From Mislevy & Riconscente, in press
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
structures
What
is important about
this domain?
Application
to
familiar
assessments
Domain Analysis
What work and situations are central in this domain?
What KRs are central to this domain?
Fixed
competency
Domain Modeling
variables
Conceptual Assessment
Framework
Assessment
Implementation
Upfront
design of
Assessment
features
ofDelivery
response
classes

From Mislevy & Riconscente, in press
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Upfront design of features
of task situation – static,
Students interact with tasks,
performances evaluated,
implicit in task, not usually
feedback created. Fourtagged
process delivery architecture.
What
is important about
this domain?
Application
to
complex
assessments
Domain Analysis
What work and situations are central in this domain?
What KRs are central to this domain?
Multiple,
perhaps
Domain Modeling
configuredon-the-fly
competency
Conceptual
Assessment
Framework
variables
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
More
complex
evaluations
of features
of task
situation
What
is important about
this domain?
Application
to
familiar
assessments
Application
to
complex
assessments
Domain Analysis
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
Conceptual Assessment
Framework
Assessment
Implementation
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Some up front
Time
Some up front
design of
design of
Assessment
features
of Delivery
Students interact with tasks,
features of task
performances evaluated,
performance or
Evolving,
created. FourUnfolding
Macro features
Micro features feedback
Micro features
Macro features
situation;
others
effects;
others
process
delivery architecture.
situated
of performance
of performance interactive,
of situation
of situation
recognized
(e.g.,
performance
recognized
situation
From Mislevy & Riconscente, in press
agents)
(e.g., agents)

Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms
of assessment
argument. Conceptualization.
Evidence
Model(s)
Task Model(s)
Student Model
Scoring
Measurement
Model evidence, and
Conceptual Assessment ModelDesign structures: Student,
task models. Generativity.
Framework






Assessment
Implementation
X1
X1
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
X2
X2
for representing
…
details,
statistical
models. Reusability.

Object models

• Psychometric models
Assessment“competences/proficiencies
Delivery
Includes
Students interact with tasks,
•Simulation environmentsperformances evaluated,
feedback created. Four• Task templates
process delivery architecture.
• Automated scoring


From Mislevy & Riconscente, in press
1.
3.
5.
7.
xxxxxxx x
xxxxxxx x
xxxxxxx x
xxxxxxx x
2.
4.
6.
8.
xxxxxxxx
xxxxxxxx
xxxxxxxx
xxxxxxxx
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
PADI Domain
objectAnalysis
model for task/evidence models
Domain Modeling
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
Domain Modeling
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Graphical representation of
network & configuration is
How do we represent key aspects of the domain in
expressable as a text representation
terms of assessment argument. Conceptualization.
in XML format, for presentation &
work product, to support
Design structures: Student, evidence, and
automated
scoring.
task models.
Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
Domain Modeling
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
• Authoring interfaces
• Simulation environments
How do we•represent
key aspects
of the domain
in
Re-usable
platforms
& elements
terms of assessment argument. Conceptualization.
• Standard data structures
• IMS/QTI, SCORM
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
In Packet Tracer, Answer Network Serves
as base pattern for work product evaluation
Dynamic task models - Variable
Assignment: Initial Network
Similar to the Answer Network Tree.
When the activity starts, instead of using the
initial network as the starting values, the activity
will configure the network with the contents of
the variables.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
• Interoperable
elements
• IMS/QTI,
SCORM
Domain Modeling
• Feedback /
instruction /
Conceptual Assessment
reporting
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Conclusion



Behavior in learning environments builds
connections for performance environments.
Assessment tasks & features strongly related to
instruction/learning objects & features.
Re-use concepts and code in assessment,
» via arguments, schemas, data structures that are
consonant with instructional objects.

Use data structures that are
» share-able, extensible,
» consistent with delivery processes and design models.
December 15, 2009
ADL
Slide 30
Further information

Bob Mislevy home page
»
»
»
»

http://www.education.umd.edu/EDMS/mislevy/
Links to papers on ECD
Cisco NetPASS
Cisco Packet Tracer
PADI: Principled Assessment Design for Inquiry
»
»
NSF project, collaboration with SRI et al.
http://padi.sri.com
December 15, 2009
ADL
Slide 31