Mike Hruska
President/CEO
Problem Solutions
mike@problemsolutions.net
@mikehruska
The Landscape
Adaptive Learning:
“Adaptive learning systems will make education much more tailored
to each individual student. These systems use artificial intelligence
to customize lessons that match the individual student’s progress.”1
Experiential Learning:
“Experiential learning to build capabilities is one of the most
important elements of a successful company transformation.” 2
70/20/10 Rule:
“Redefining the blend to bring learning closer to the workplace—
and provide appropriate “scaffolding” for the learner’s needs—is
still a struggle for most organizations. 3
http://www.mckinsey.com/insights/mgi/in_the_news/empowering_teachers_and_trainers_through_technology
http://www.mckinsey.com/insights/operations/experiential_learning_whats_missing_in_most_change_programs
http://insights.ccl.org/wp-content/uploads/2015/04/BlendedLearningLeadership.pdf
Current State
• Many companies required learning is beyond
capabilities of current platforms
• The LMS is not “satisfying” needs
• Learning is beyond formal (experiential and
informal)
• Companies face cost and time challenges to
develop human capital and manage talent
• Data from systems could drive learning
experiences, but data isn’t typically connected
the BIG QUESTIONS…
How can we leverage performance data
to save time and money training
personnel?
How can we increase training
effectiveness by using data collected
along the continuum of training?
What are learning ecosystems?
“Learning ecosystems provide a combination
of technologies and support resources to
help individuals learn within an
environment“
Source: David Kelly: http://twist.elearningguild.net/2013/11/what-is-a-learning-ecosystem/
What are learning ecosystems?
Access From
Any Device
Personalized and
Brokered Content
•Just-in-Time
•Just-for-You
•RIGHT TIME
EXPERIENCES
Learn From:
•Recommendations
•Intelligent Tutors
•Mentors and
peers via social
networks
•Self-Discovery
Learn Using:
•Simulations
•LMS
•Web
•Games
•Virtual Worlds
•Intelligent content
6
Ecosystem Elements
Actors
Resources
Events
Signals
Sensors
Flows
Patterns
AFFORDANCES
Incoming Data
• Evaluate incoming
student
competencies to
inform planned
instruction
• Use data analytics to
discover trends over
time
• Discover which
methods of training
have had the most
impact
Current Data
• Dynamically optimize
training events in realtime based on
performance.
• Assess capability and
proficiency levels of
individuals, teams, and
groups
• Reveal competency gaps
and their sources:
- Trainee skills
- System faults
Outgoing Data
• Make sound
recommendations for
the next training events
based on performance
data.
• Report performance
data to the larger
organization.
• Discover which training
interventions caused
the greatest transfer to
the operational
environment.
Use Cases
Historical proficiency / Performance over time
Live performance view
Macro adaptation (learning path)
Micro adaptation (performance based)
Trends analysis
RECOMMENDATIONS
Key Questions
• Is this person capable to do X?
• What do they need to be ready to do X?
• Are my people capable/ready to do X? If not,
how can we prepare them?
• Who are the best people at X ? And why are
they so awesome?
• What does someone need to do more of? Less
of?
What is Interoperable Performance
Assessment (IPA)?
 Method of uniformly defining
and describing experience
and context to assess learning
and performance
 Ability to adapt training
across a variety of
environments, systems, and
modalities
 Observe, assess, evaluate, or
assert performance by
systems or observers
Our [Research] Approach
Build lightweight and useful tools that:
– Uniformly capture performance data AND
context
– Enable learning ecosystem functionality
• Allow the data to become immediately visible
• Enable adaptive systems to respond to the data
– Allow multiple lines of research efforts to build
upon work
– Inform early adopters and lower implementation
risk
Army Research Laboratory (ARL)
Supported Programs
Soldier Centered Army Learning Environment
•
•
•
Developed by Army Research Lab (ARL) and Army Research Institute (ARI)
Data-driven architecture and web service-based means to allow the integration
with new technologies
Supports training and education across multiple hardware platforms
Generalized Intelligent Framework for Tutoring
•
•
•
•
Developed by Learning in Intelligent Tutoring Environments (LITE) Laboratory
Supports Army’s vision of more efficient and effective learning
Computer-based tutoring framework to evaluate adaptive tutoring concepts,
models, authoring capabilities, and instructional strategies
Provides a generic tutoring capability, including remediation strategies based
on learner performance, to integrated learning environments
Human Performance Measurement
Language (HPML) Constructs
•
•
•
•
•
•
•
Experience
Position
Platform
Training Environment
Training Characteristics
Measure
Assessment
•
•
•
•
•
•
•
Project/Mission
Task
Subtasks
Competency
Objective
Standard
Knowledge/Skills
Reference: Stacy, W., Ayers, J., Freeman, J., & Haimson, C. (2006). Representing
Human Performance with Human Performance Measurement Language.
Washington, DC. Aptima, Inc.
xAPI and HPML
Elements of an xAPI Statement
• Actor
• Verb
• Object
• Context
• Results
• Extensions
Data via
web
services
Import
Tools
Games
Mobile
A/R
Virtual
Worlds
Adaptation
Personalization Multiple
Simulators
Tailoring
Analytics
Visualization
Efficacy
Efficiency
Effectiveness
Example Ecosystem
TOOLS
Encoding/data collection library
Reduces complexity to support xAPI
Implements encoding best practices
Describes rich performance context
Dynamic Link Library (.dll)
Individual and group performance
encoding support
• www.pipelinexapi.com
•
•
•
•
•
•
Current Research
• Efficacy research
• Using Pipeline with gunnery
and crew simulators
• Encoding individual
data/team
using xAPI
• Macro-adaptation
• Show effectiveness and ROI
basis for adaptation
19
Effectiveness Research
• Recently completed human subject
research using xAPI to adapt individual to
drive team outcomes
• In the team training, participants in the
Adaptive condition took nearly 40% less
time to and nearly 60% fewer scenarios
to achieve excellent qualification scores.
• Results will be published in Q4 2015 in
research journal – I/ITSEC
Lessons Learned
Competencies
• Profiles should be built around competencies.
• Systems should use competencies as an anchor for tracking learner
performance
Granularity
• A balance exists between capturing large amounts of data and
capturing meaningful data that is useful to other systems.
Standards and Interoperability
• Though data may be captured by a given system, it may or may not
be relevant to other systems.
• Most systems that adapt to the learner do so in a black box fashion
using proprietary models of the learner, domain, and data.
• Highly adaptive systems are typically complex and designed as
isolated systems that do not communicate or interoperate with
other digital learning systems.
Questions or want more info?
Mike Hruska
President/CEO Problem Solutions
mike@problemsolutions.net
@mikehruska
www.problemsolutions.net
APPENDIX SLIDES
Our work..
Support technical activities in and around
learning technologies
Span applied research to product development
Build startups and early stage technologies
Design and build products and solutions
Design and deploy new technologies
Things We’ve Done
Built the Experience API with Gov’t and Industry over last 5+ years
Built more open source tools for xAPI than any other gov’t program
Brandon Hall Gold Award Winning xAPI Product on xAPI
Thought Leader Articles in Learning Solutions Magazine (Feature in
June 2015) on Learning Ecosystems, xAPI and Design
CoAuthor of “Learning on Demand – ADL and the Future of eLearning”
CoAuthor of US Dept of Education “Ed Tech Startup Guide” (April
2015)
CoAuthor of Increasing Access through Mobile Learning (2014)
Learning Solutions Magazine
Industry Articles
Learning Ecosystems, Analytics, and xAPI
-http://www.learningsolutionsmag.com/articles/1766/elearning-authoring-taking-the-next-step-with-xapi
-http://www.learningsolutionsmag.com/articles/1761/amplifying-the-experience-api-xapi-camp-atdevlearn-2015
-http://www.learningsolutionsmag.com/articles/1745/are-you-an-isd-a-business-process-engineer-orboth
-http://www.learningsolutionsmag.com/articles/1693/learning-ecosystems-and-the-experience-api-xapicamp-recap
-http://www.learningsolutionsmag.com/articles/1722/xapi-and-analytics-measuring-your-way-to-success -http://www.learningsolutionsmag.com/articles/1523/ten-steps-to-plan--communicate-your-xapi-designto-a-web-developer
Training Magazine
What is the xAPI?
-http://www.trainingmag.com/content/what-experience-api
Elearning News - xAPI
-http://elmezine.epubxp.com/t/112009/33
ASTD T&D Magazine - SCORM Evolution
-http://www.astd.org/Publications/Magazines/TD/TD-Archive/2013/04/A-SCORM-Evolution
Future Learning in the DoD
-http://www.kmimediagroup.com/military-training-technology/440-articles-mtt/unlimited-access-learning
Published Research on xAPI
Hruska, M., Medford, A., Murphy, J. (2015). Learning Ecosystems Using the Generalized
Intelligent Framework for Tutoring (GIFT) and the Experience API. 17th International
Conference on Artificial Intelligence in Education (AIED 2015). Madrid, Spain. June 2015.
Amburn, C., Goodwin, G., Michael, H., Murphy, J. (2015). Developing Interoperable Data for
Training Effectiveness Assessment in Army Marksmanship Training. MODSIM World 2015.
Goodwin, G., Hruska, M., Murphy, J. (2015). Developing Persistent, Interoperable Learner Models
in GIFT. GIFT Sym3. Orando, FL, June 2015
Hruska, M., Long, R., Amburn, C. (2014). Human Performance Interoperability via xAPI: Current
Military Outreach Efforts. Simulation. Fall Simulation Interoperability Workshop, 14F-SIW-035,
Orlando, FL, September, 2014.
Hruska, M., Long, R., Amburn, C., Kilcullen, T., Poeppelman, T. (2014). Experience API and Team
Evaluation: Evolving Interoperable Performance Assessment. The Interservice/Industry Training,
Simulation & Education Conference (I/ITSEC). Volume: 2014.
Poeppelman, T., Hruska, Long, R., Amburn, C. (2014). Interoperable Performance Assessment for
Individuals and Teams Using Experience API. 2nd Annual Generalized Intelligent Framework for
Tutoring (GIFT) Users Symposium (GIFTSym). Pittsburgh, PA, June, 2014.
Hruska, M., Poeppelman, T. R., Dewey, M., Paonessa, G., Paonessa, M., Nucci, C., Ayers, J. (2013).
Interoperable Performance Tracking to Support Tailored Learning (Final Report). U.S. Army
RDECOM Army Research Laboratory (ARL) – Simulation Training Technology Center (STTC).
Poeppelman, T., Ayers, J., Hruska, Long, R., Amburn, C., Bink, M. (2013). Interoperable Performance
Assessment using the Experience API. The Interservice/Industry Training, Simulation & Education
Conference (I/ITSEC). Volume: 2013.
{ "id": "9a9c6f5f-5bb7-4ed1-b993-96e9cee3cd2f", "actor": { "objectType": "Agent", "name": "PVT Jacob
Daniels", "mbox": "mailto:name@us.army.mil" }, "verb": { "display": { "en-US": "experienced" }, "id":
"http://adlnet.gov/verbs/experienced" }, "object": { "objectType": "Activity", "id":
"activityId:uri/10th_CAB_AATX", "definition": { "name": { "en-US": "10th CAB AATX" }, "description": { "enUS": "On-Station - Area of Operations - Expert:15 degree safe firing range each way,Average:18 degree
safe firing range each way,Novice:20 degree safe firing range each way - This measure assesses the way in
which pilots approach the target when they get into the on-station area" } } }, "result": { "extensions": {
"resultExtensions:uri/extensions/trainingobjective": "Meet all Time on Target objectives Meet all Time on
Target objectives" }, "score": null, "success": false, "response": "Expert" }, "context": { "extensions": {
"contextExtensions:uri/AVCATT/role": "FAC",
"contextExtensions:uri/AVCATT/visual_scene_display_resolution": "20/10 visuals",
"contextExtensions:uri/AVCATT/cockpit_content": "Content and layout exactly replicates aircraft",
"contextExtensions:uri/AVCATT/visual_field_of_view": "180 horizontal FOV",
"contextExtensions:uri/AVCATT/g_cueing": "No G cues",
"contextExtensions:uri/AVCATT/axis_disturbance_motion_cues": "No disturbance motion cues" },
"contextActivities": { "parent": [], "grouping": [ { "id":
"contextActivities:uri/01_2_5179_perform_aerial_deliberate_attack_operations", "definition": { "name": { "enUS": "01-2-5179 Perform Aerial Deliberate Attack Operations" }, "description": { "en-US": "Destroy enemy
or force them to withdraw" } } }, { "id": "contextActivities:uri/support+troops+in+contact+(tic)",
"definition": { "name": { "en-US": "Support Troops in Contact (TIC)" }, "description": { "en-US": "Friendly
ground forces receiving effective fire" } } }, { "id": "contextActivities:uri/battlefield_communications",
"definition": { "name": { "en-US": "Battlefield Communications" }, "description": { "en-US": "Right
information, to the right person, at the right time" } } } ] }, "platform": "AH-64 - Apache" }, "timestamp":
"04/17/2014 16:19:27", "authority": { "objectType": "Agent", "name": null, "mbox": null } }