A view on breaking
barriers from a high
tech industry
John T. Behrens, Ph.D
Kristen E. DiCerbo, Ph.D.
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
1
It takes a village to think about
assessment (update)
Youn Young Choi, UMD
Dan Robinson, UT
AAron Crawford, ASU
Roy Levy, ASU
Bob Mislevy, UMD
Dennis Frezzo, Cisco
Mark Chen, Cisco
Philip Piety
Daisy Rutstein, UMD
Junhui Liu, UMD
Patti West, Cisco
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
2
Thanks also to
 Bob Lissitz and organizers
 State of Maryland
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
3
A play in 3 acts
 Who we are and where we come from
 Some barriers to be broken (with examples)
 Some game play (or is it work?)
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
4
Cisco Networking Academy
ICT Networking Skills Education for Individuals and Underserved
Communities
Large and Global
165+
Countries
900,000+ Students/Year
3 Million+ Students since inception
Diverse Students and Communities
Students: Diverse Age, Gender, and Challenging Circumstances
Communities: Mature and Developing Countries
Diverse Educational Institutions
Universities, Community Colleges, Vocational Schools, Secondary
Schools, Non-profit Organizations, Second Chance
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
5
Students by Region (900,000+ Students)
7%
2%
19%
14%
7%
1%
7%
% of Global Total
20%
Europe
18%
Latin America
17%
APAC
16%
U.S. and Canada
7%
Greater China
7%
Middle East
7%
Central and Eastern Europe
5%
Africa
2%
Russia and CIS
1%
Japan
Presentation_ID
19%
5%
18%
% of Worldwide Total as of July 31, 2010
%
Source: AME, jzinn_v1, Snapshot as of April 30, 2010
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
6
Students by Education Level by Region (900,000+ Students)
% of Global Total
20%
Europe
18%
Latin America
17%
APAC
16%
U.S. and Canada
7%
Greater China
7%
Middle East
7%
Central and Eastern Europe
5%
Africa
2%
Russia and CIS
1%
Japan
* See Notes Page for additional details on these definitions
Presentation_ID
Source: AME, jzinn_v1, Snapshot as of April 30, 2010
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
7
Need to do more than make stuff
Curricula
Product
Relationships
 14 courses
 Entry-level tech skills
 Students
 Instructor-led
 Academies
 Simulation and
visualization software
 Instructors
 Governments
 Hands-on experience
 Partners
 Assessments
 Educational process and
learning systems
 Metrics
 Services
 Larger server base
than cisco.com
 Support
 2.5 terabytes of data
 1M assessments/month
Infrastructure
Presentation_ID
Program Design
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
8
Comprehensive blended e-learning
Curriculum
Assessment
Gaming
 Explanatory text
 Knowledge focused questions
 Interactive media
 Student or instructor initiated
 Promotes motivation and
engagement
 Hands on labs
 Integrated reporting
 Promotes learning and practice
 Embedded assessment
 Rich feedback
 Provides larger context
 Rich feedback
 Simulation supported
 Rich feedback
 Simulation supported
Presentation_ID
 Simulation supported
© 2009
2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
9
Instructor Initiated
Student Initiated
A comprehensive assessment model
Presentation_ID
Formative
Assessment,
Measure
Understanding
Proficiency Flow
Formative
Assessment,
PerformanceBased
Quizzes, Testlets,
Feedback Activities
Packet Tracer,
Labs, Simulations
Formative
Assessment,
Measure
Understanding
Formative
Assessment,
PerformanceBased
Summative
Assessment
Proficiency Flow
Chapter Exams,
Practice Finals
Flash Rich Media, Packet
Tracer, Simulations,
PT Practice SBAs
Cert Practice Exams
© 2008 Cisco Systems, Inc. All rights reserved.
SBAs, Final Exams,
Midterms,
Packet Tracer
Cisco Confidential
10
Exams by month/year
1999
1,300,000
1,200,000
2000
1,100,000
2001
1,000,000
900,000
2002
800,000
2003
700,000
600,000
2004
500,000
2005
400,000
300,000
2006
200,000
2007
100,000
2008
Presentation_ID
November
September
July
May
March
January
0
© 2008 Cisco Systems, Inc. All rights reserved.
2009
Cisco Confidential
11
BABY IT’S A WILD WORLD…..
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
12
Audience Participation
Pretend its 1990 (20 years ago)
For each of the following tasks, consider how you would
accomplish it (in 1990).
1. See your niece (who lives in another country) play
with the present you sent her in real time.
2. Determine the height of Juday Creek in South Bend
Indiana this morning.
3. Determine the amount of cloud cover in Australia
today
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
13
Audience Participation
Jump back to the present (2010)
For each of the following
tasks, consider how you
would accomplish it today
1. See your niece play with
the present you sent in
real time.
2. Determine the height of
the Juday Creek in
South Bend IN
3. Determine the amount
of cloud cover in Perth,
Australia today
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
14
What’s changed in 20 years?
Remote sensors
NOAA, Urban cams, Web cams
Ubiquitous computing
e-commerce, e-government
Powerful computing (massively parallel computing –
The Large Hadron Collider at CERN will produce roughly 15
petabytes (15 million gigabytes) of data annually – enough to fill
more than 1.7 million dual-layer DVDs a year
Computers for computers
financial transaction; airline reservations
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
15
And you have it too
Video and digital image sensors in your cell phone
Social computing (vote for your favorite dancer with 1 million
other people)
Powerful computing given away for free
Google apps, Google mash ups, Google earth
Computers for computers
Airline system that texts your cell phone
Zotero
Computers in space
Tracking your location and giving you
suggestions based on your location
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
16
The important aspect of the digital
revolution
 Is not about digitization, computation and information
transfer
(though keep buying the equipment)
 It is about how our lives and work can be transformed
by the technological changes
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
17
Degree of Diffusion of the Technological
Revolution
Installation Period
Synergy
Coherent
Growth
Irruption
Intense
funding of
New Tech
Big
Bang
Presentation_ID
Deployment Period
Maturity
Idle money
moving to
new areas
Frenzy
Divorce
Between
paper &
real assets
Crash
Institutional
Recomposition
© 2008 Cisco Systems, Inc. All rights reserved.
Next
Big
Bang
Cisco Confidential
18
Assessment in the digital jungle
Digital Desert
Digital Jungle
Periodic samples of data
Ongoing ubiquitous data
Small samples of data
Dramatically large and ubiquitous
Special intrusive systems to get data
Data built into daily activity
Lack of data requires special focused
inputs
“Items” no longer exist
Absence of data requires inferential
stretch
Availability of data lessons need for
inference
Data scarcity leads to small sample
science (e.g models of expertise)
Data jungle leads to improved
understanding of detailed
mechanisms & rules (automated
automated scoring)
Disconnected intrusions
Data storage leads to continuous
improvement
Data outside classroom not even
considered
Data is data no matter where it is
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
19
Breaking boundaries in assessment
 Hands on vs simulation
 Curriculum vs assessment
 Formative vs summative
 Formal vs informal
 Game vs assessment
 Natural data vs tests
 Results vs communication
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
20
Assessment computing in the 20th
century
Question  Answer(options) Correctness Points *b j
Question  Answer(options) Correctness Points *b j
Question  Answer(options)  Correctness Points *b j
Question  Answer(options)  Correctness Points *b j
Question  Answer(options)  Correctness Points *b j
Question  Answer(options)  Correctness Points *b
j
Question  Answer(options)  Correctness Points *b
j
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
21
 Items as pieces
 Value of independence
 Scoring first
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
22
A More Flexible Idea
Administrator
If Z1 then Z2
--------------------
Select
Activity
Present &
Record
Max (f(x))
Activity
100 90
80
70
60
50
40
30
40
50
User
Record
Weights
& Models
X1
Evidence /
Observable
Synthesis
Activities
Meta-Data
Work
Product
Rules &
Structure
Evidence/
Observables
Feature
Identification
X2
X3
X1 = 1; X2 = 0, Xn=?
If A and B then X1 = 1
X4
If C then X2 = 0
Xn
Task Level Feedback
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
23
In Sum
Choose/Create
Next Activity
Update Skill
Profile
Presentation_ID
Start
Give
Activity
Score and
Give
Feedback
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
24
The adaptive exam model
Choose/Create
Next Activity to
optimize
information
Update Skill
Profile
Presentation_ID
Start
Give
Activity
Score and
Give
Feedback
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
25
The practice tutor model
Choose/Create
Next Activity for
best skill
change
Update Skill
Profile
Presentation_ID
Start
Give
Activity
Score and
Give
Feedback
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
26
The motivational game model
Choose/Create
Next Activity
best motivation
Update Skill
Profile
Presentation_ID
Start
Give
Activity
Score and
Give
Feedback
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
27
Check it out….
 The ECD delivery model is sufficiently flexible to show
a high level symmetry between assessment, games
and tutors.
 The principles of inference and meaning generation
from observation are articulated at a sufficiently broad
manner to cut across the disciplines.
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
28
How we use ECD and
technology to break
assessment boundaries
 Formal vs informal
 Curriculum vs assessment
 Hands-on vs hands-off
 Education vs “real world”
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
29
Packet Tracer cannot do everything a complex lab
can do, but it can do many things a basic CCNA
lab cannot:
 Visualization
 Scaling
 Rapid revision
 Off-line delivery
 Complex scoring
 Built in gaming
 Micro-world
authoring
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
30
Exploration & Experimentation
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
31
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
32
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
33
Complimentary Systems
Filling in the missing pieces
Fixed Response Questions
Packet Tracer
Based Assessments
Focus
Questions can pinpoint important
knowledge and key ideas in
specific parts of the curriculum.
Students demonstrate real-world
skills in simulated environment to
“put all the pieces together”.
Automated
Scoring
Question correctness easy to
understand and discuss. Grade
book supports transparency.
New versions of Packet Tracer
allow authoring of sophisticated
scoring rules to make complex
reports.
Rich Feedback
Multiple Choice Questions tied to
personalized feedback with
curriculum linking.
PT SBA provide detailed diagnostic
feedback regarding strengths and
weaknesses in complex problems.
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
34
Implication: Packet Tracer SBA Exam
Launch
Take Exam
Submit It
Automatically
Scored
View
Feedback
1
2
3
4
5
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
35
Packet Tracer SBA Detailed Feedback
Helps Students Identify Areas for Improvement
Proficiency Estimates
Presentation_ID
Detailed Scoring Feedback
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
36
Goal 8
Quiz
Simulation and feedback are threaded through the
learning progression
PT
Exam
Goal 7
Goal 4
Exam
SBA
Labs
Goal 5
PT Integration
Content Areas
Goal 6
Course 4
Presentation_ID
Chapter 1
Quiz
Chapter 2 …
Exam
Exam
Quiz
PT Integration
Course 1-3
PT
Labs
Goal 1
PT
PT Integration
Goal 2
Labs
Goal 3
…Chapter k
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
Final
37
But we are lazy and often stupid, so we want the computers to
figure out the rules and the patterns.
COMPUTING FOR
COMPUTERS
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
38
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
39
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
40
Hierarchical Cluster Analysis
 We can see which
“documents” are
most like other
“documents”
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
41
Patterns of Command Sequences
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
42
But learning doesn’t happen only through instruction
… And maybe learning can happen without suffering
CAN WE HAVE ASSESSMENT
WITHOUT SUFFERING?
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
43
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
44
Roll the demo here………..
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
45
Individual Performance
18
16
14
Money
12
Reputation
10
Business Sense
8
Configuration
6
Troubleshooting
Physical Labor
4
2
0
1
V1C3
V1C3 detail
Work Product
Features
Proficiencies
Presentation_ID
Max Users
Static IP Address on PC3
PC3 is in Network
Connectivity Tested
Business Sense
Configuration
Troubleshooting
Score
2
3
4
5
6
7
8
9
1
192.168.0.99
1
1
1
1
1
© 2008 Cisco Systems, Inc. All rights reserved.
300/300
Cisco Confidential
46
Pulling it all together
Bridging/Switching
OSI Model
Routed Protocols
Domain Knowledge
Routing Protocols
WAN Protocols
Network Modeling
Network
Management
LAN Technologies
Cisco Basics
Network Proficiency
Design
Implementation
Operation
Troubleshoot
Curriculum
Assessment
Gaming
Integration
 Explanatory Text
 Knowledge focused
questions
 Promotes motivation
and engagement
 Student or instructor
initiated
 Promotes learning and
practice
 Comprehensive skill
model provides
coherence
 Integrated reporting
 Provides larger context
 Rich feedback
 Rich feedback
 Simulation supported
 Simulation supported
 Interactive media
 Hands on Labs
 Embedded Assessment
 Rich Feedback
 Simulation supported
Presentation_ID
© 2009
2008 Cisco Systems, Inc. All rights reserved.
 Common use of
simulation supports
cross-activity transfer
 Possibilities for future
research
Cisco Confidential
47
Bridging/Switching
OSI Model
Routed Protocols
Domain Knowledge
Routing Protocols
WAN Protocols
Network Modeling
Network
Management
LAN Technologies
Cisco Basics
Network Proficiency
Design
Presentation_ID
Implementation
Operation
Troubleshoot
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
48
And don’t forget to report it in a way
that provides enablement
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
49
Conclusion
 The digital revolution places us in the midst of
incredible transformative social and economic change.
 Simulation and digital gaming is embedded in this
change.
 Best practices in assessment may help us move
forward in this area, the worst will not.
 Large amounts of data and computing power will allow
possibilities little imagined
 We can only take advantage of these possibilities if we
think comprehensively about ecosystems, design and
meaning.
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
50
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
51
Presentation_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
52