Multichannel Learning System (MLS)
Concept Evaluation Train-the-Trainer
Workshop
16 December 2014
Defense Institute of Security Assistance Management (DISAM)
Dayton, OH
Ms. Janie Glover
Naval Education & Training Security Assistance Field Activity (NETSAFA)
MLS Project Manager
Background
Short-term goal
■ Identify the best methods for providing distance education for
international military students
■ Evaluate the effectiveness of using multiple learning formats to
support Security Cooperation Education and Training Program
(SCETP) requirements.
Long-term goal
■ Develop an International Military Visitor Briefing (IMVB) capability,
based on the MLS Evaluation Results, that uses multiple training
approaches to support international visitors invited to the United
States.
2
Agenda – Day 1
Time
0830 – 0900
0900 – 0930
Registration
Welcome Remarks & Introductions
0930 – 1000
Multichannel Learning System (MLS) Brief
1000 – 1030
Using MLS Learning Approaches to Support
Humanitarian Assistance Requirements
Break
Measurement Approaches
Use of MLS for knowledge transfer within an Island
community
1030 – 1055
1055 - 1125
1125 - 1155
Topic
1200 - 1330
1330 - 1400
Lunch
Multichannel Learning in MoD Serbia
1400 - 1425
1425 - 1440
1440 - 1535
Individual Training & Education Research for the
Canadian Armed Forces
Break
MLS Research Protection Program
1545 – 1600
Day 1 Wrap-up
Presenter/Comments
Janie Glover (NETSAFA)/ Dr. Mark
Ahles (DISAM)/ Doug Howard
(MCSCG)
Janie Glover (NETSAFA) /Dr. Mark
Ahles (DISAM)
Dr. Aiman Al Sumadi / Dr. Asem Hiari
(JRMS)
Dr. Venkat Sastry (CU)
Saboohi Famili/ Guernsey College of
Further Education
(GCFE)
Biljana Presnall (Jefferson
Institute)/Goran Simic (Serbia Military
Academy)
Dr. Jerzy Jarmasz (DRDC)
Julie Howell (SSC Pacific)/ Dr. Andrea
Loesch (GIRAF PM)
Janie Glover (NETSAFA)/ Dr. Mark
Ahles (DISAM)/ Doug Howard
(MCSCG)
3
Organizational Chart, Stakeholders
& Participating Organizations
Organizational Chart
Advisor Team/Stakeholders
Ms. Janie Glover
Project Manager
• Office of the Secretary of Defense (OSD),
Coalition Warfare Program (CWP) Office
• Marine Corps Security Cooperation Group
(MCSGC)
• DoD Security Cooperation Education &
Training Program (SCETP) Organizations
• International Defence Organizations
Dr. Mark Ahles
Deputy Project Manager
Project Management Support
Lockheed Martin
International Training Team (LMITT)
Dwayne Eldridge
Learning Content
•
•
•
•
•
•
•
Canada (DRDC)
Germany (FKIE, CalTech)
Georgia (GRENA, ICP)
Jordan (JRMC)
Serbia (Jefferson Institute)
U.K. (DCTS, DAUK)
U.S. (LMITT)
Dr. Kristen Barrera
Testing & Evaluation
•
•
•
•
•
•
•
•
•
ONR Reserve Component, SSC Pacific
Canada (DRDC)
Germany (CalTech)
Georgia (GRENA, ICP)
Jordan (JRMC)
Serbia (Jefferson Institute)
U.K. (GCFE, CU)
U.S. (LMITT)
Germany (Siebenundvierzig ING GmbH &
Co. KG)
Julie Howell
Research Protocol
•
•
•
•
•
•
Germany (GIRAF PM)
U.S. (LMITT)
U.K. (DAUK)
Canada (DRDC)
Jordan (JRMC)
Serbia (Jefferson Institute)
5
NETSAFA Mission & Relationships
SECNAV
ASN RD&A
NAVY IPO
SECDEF
DSCA
International
Policy
Develop, execute and manage the Department
of Navy’s Security Assistance and Security
Cooperation training and education programs
that support the U.S. security strategy in
building partner capacity to build and
strengthen enduring international partnerships
Maritime
Engagement
Management
USCG/IA
USMC
MCSCG
6
NETSAFA’s Role
Material Cases
PME
Flight Training
Mission Training
COCOM / NCC
CTFP/CD/I-MET/1206
Mobile Teams
Technical Training
Training Providers
Requirements / Priorities
Country Teams
7
Defense Institute Of Security
Assistance Management (DISAM)
Department
Department of Defense
(DoD)(DoD)
Of Defense
Defense Security Cooperation Agency (DSCA)
Defense Institute of Security Assistance Management (DISAM)
8
DISAM Mission Statement
The Defense Institute of Security Assistance Management provides
professional education, research, and support to advance U.S. foreign
policy through Security Assistance and Cooperation
Tasks
■ Resident, non-resident, and online courses
■ Seminars and tutorials
■ Research, consultation, studies
■ Information dissemination
9
Continuous Learning (711HPW/RHAS)
Air Force Research
Laboratory (AFMC)
711th Human
Performance Wing
Human Effectiveness (RH)
Warfighter Readiness
Research Division (RHA)
■ Develop mission
relevant content for
learning
■ Measure training
outcomes in complex
mission environments
■ Accelerate
development of
mission critical
competencies
■ Optimize training value
of live, virtual, and
constructive
environments
10
Space and Naval Warfare
Systems Center PACIFIC
■ SSC Pacific has the mission to enable Information
Dominance for our Naval, Joint, National, and
Coalition warfighters through research, development,
delivery and support of integrated capabilities

Lifecycle development and support for military
Command Control, Communications, Computers,
Intelligence, Surveillance, Reconnaissance (C4ISR)
systems – from concept to fielded capability
■ Responsibilities for MLS

Institutional Review Board (IRB)
–

Assurance for the protection of the rights and welfare of research subjects
Provide/leverage operational test and evaluation expertise
■ Obtained Letter of Endorsement from Commanding Officer,
SSC Pacific (dated 12 Dec 2012)
11
ONR Global Mission Statement
Search globe for promising, emerging scientific research and advanced
technologies to enable ONR to effectively address current needs of the
Fleet/Force and investigate and assess revolutionary, high-payoff
technologies for future Naval missions and capabilities
Discovering the
Best Science
Enabling Global
Technical Awareness
Science & Technology
Collaborations
■
■
■
■
■
Innovate fundamental
research
Help shape the future
U.S. Naval investments
and strategies
Engaging global S&T
talent through
cooperation
■
Fundamental
research is universal
Continued
contributions to global
technology
awareness
■
■
Advancement of mutual
beneficial science
Support publication of S&T
research
Ensure DoN research
efforts in support of U.S.
Government policies
12
Office of Naval Research Reserve
Component (ONR RC) Mission Statement
The mission of Navy Reserve Science & Technology
Program is to leverage the unique combination of
technical expertise and warfare experience resident
within the Naval Reserve Force to assist the Office of
Naval Research and the Naval Research Laboratory
in providing optimal science and technology
solutions to the warfighter
Provides Depth
Unavailable in Active
Component
■ Specialized civilian skills
■ Advanced technical degrees
■ Significant Government and
Industry Acquisition
Experience
■ Warfighting and S&T
experience
Optimal
Projects
Navy
Operational
Knowledge
Both
Technical, S&T,
and Acquisition
Knowledge
ONR-RC Personnel
Aligns with ONR/NRL
Mission Priorities
Delivers Relevant Products
to the Fleet/Forces
■ Matched to Navy S&T
needs based on skills
■ Highly integrated with
supported commands
■ Regardless of unit
location/command
■ Force multiplier…
tremendous ROI
■ History of recalls to support
Navy Operations
■ High impact where technology
touches the Fleet
13
International Team
Canada
Jordan
■
■
Defence Research and Development
Canada (DRDC)
King Hussein Medical Center; Arab
Jordanian Center for Studies (ARJOCS)
Germany
■
Centre for eLearning Technology
(CalTech) of the Saarland University,
German Research Centre for Artificial
Intelligence and Higher School of
Technology and Economics; GIRAF PM
Services GmbH; Siebenundvierzig ING
GmbH & Co. KG
Georgia
■
Georgian Ministry of Defense (MoD);
Georgian Research and Educational
Networking Association (GRENA);
International Center for Advancement of
Research, Technology & Innovation (ICP)
Serbia
■
Center for Simulations and Distance
Learning of the Military Academy in
Belgrade/Jefferson Institute
U.K.
■
Defence Academy of the United Kingdom
(DAUK) at Cranfield University; Guernsey
College of Furthering Education (GCFE)
14
Project Goals, Objectives &
Timeline
Project Objectives
■ Develop a capability that ensures IMSPDB students have a positive
and successful experience in the U.S. along with their U.S.counterparts
■ Collaborate with international partners to identify the “best” methods
for providing “distance” education for international military students
■ Evaluate the learning effectiveness of using multiple learning formats
(e-publications, mobile applications, video and web-based) to support
Security Cooperation Education and Training Program (SCETP)
requirements
16
Project Goals
■ Identify the best methods for providing distance education for
international military students
■ Evaluate the effectiveness of having multiple learning formats to
support Security Cooperation Education and Training Program
(SCETP) requirements
17
Project Deliverables
■ A final report that evaluates the results of the MLS Concept Evaluation
and makes recommendations on the “best approach” for meeting
Indoctrination Training Requirements
■ A Strategy Document, in consonance with the Security Cooperation
Education and Training Program (SCETP) Community, which focuses
on the out-years (i.e., Phase III and Phase IV)
18
Timeline
Activity
Dec
Jan
Feb
Mar
Apr
May
Jun
Concept Evaluation
Train-the-Trainer
Workshop
Concept Evaluation
2 Jan – 16 Feb 2015
Input Translations
Data Sanitization
17 -28 Feb 2015
Data Analysis
MLS Final Report
(Draft)
MLS Final Report
And
Briefing
19
Concept Evaluation
Concept Evaluation Review
The Concept Evaluation (CE) will involve three groups
1. U.S. Government, Civil Servants and Contractors
2. International Military Students (IMS) attending training in the US
3. International partners – Academic, Government & Commercial
Organizations
CE Approach
■ The Testing & Evaluation Team developed the MLS Test Plan
■ Research Protocol Team implemented US/International Research
Protocol Processes, in consonance with the international country leads
■ Data Collection Portal developed by Germany
■ Group leads, identified above, seek potential MLS CE Volunteers
21
Concept Evaluation Review
Multiple Channels of
Learning Evaluated for
International Learners
Evaluations (20 COMPLETED
PER INTERNATIONAL PARTNER)
Learning FORMAT
Channel
e
b
m
v
e = e-Publication
b = web-based learning
m = mobile application
v = video
Learning DELIVERY
Channel
PC = personal computer
(PC/Mac/Linux)
T = tablet (iPad/
Android/Apple/etc.)
M = mobile device (smart
phones)
Germany
(Siebenundvierzig
ING GmbH & Co.
KG) will host the
MLS Data
Collection Portal
PC
T
M
• Registration
• Demographics
• Pre/Post Content Test
• User Feedback
• No content feedback
• Technology Used
• Learning Formats Selected
• Why?
• Channel selection
• Environmental concerns
Evaluators will be given user ID /
PIN to system. Will self-select
desired learning formats.
Each learning channel will provide
18-20 minutes of content on U.S.
civilian & military cultures.
Evaluators continuously review
learning formats.
Working Group Decisions
1. What questions to ask.
2. What demographics to
collect.
3. Input from partners on what
portion of the IMSPDB MLS
evaluation
4. Commitment for evaluators.
5. Concept Evaluation
Timeframe agreement.
Evaluation Results
22
Learning Content Working Group
■ Ensure the courses are developed according to the
optimum format (i.e., e-pubs, mobile apps, video and
web-based) to meet the learning objectives
■ Identify delivery formats to support traditional and nontraditional training approaches in order to support user
preferences
23
Testing & Evaluation Working Group
■ Conduct an evaluation of multiple learning approaches
(i.e., e-pubs, mobile apps, video and web-based) to
evaluate the effectiveness of meeting IMSPDB Training
Requirements
■ Make recommendations on how the lessons learned may
be used to support DoD and other USG International
Military Student Pre-Departure (IMSPDB )Training
Requirements
24
Research Protocol Working Group
■ Coordinate with other working groups to ensure all
content development and evaluation activities are
compliant with US/International requirements
■ Collaborate with international partners to ensure
Research Protocol processes are compliant, at a
minimum, with US requirements
■ Obtain approval for the Concept Evaluation by the SSC
Pacific Institutional Review Board
■ Monitor the Concept Evaluation process to ensure
compliance
25
Project Documentation
& Collaboration
Project Documentation
■ Introduction
■ Multichannel Learning System
(MLS) Project
■ Management Approach
■ Project Working Groups
■ Human Research Protocol
Program (HRPP)
■ Project Communication
■ Appendices A: Participating
Organizations
■ Appendices B: Acronym
27
JKO APAN MLS Portal
https://wss.apan.org/1539/JKO/mls/SitePages/Home.aspx
28
Project Update
Collaboration Activities
■ Capitalized on the expertise of the Subject Matter Experts (SMEs)
■ Developed a Testing & Evaluation Strategy on how to evaluate multiple
learning approaches (i.e., audio, video, web-based, mobile apps, and epublications)
■ Formalized the Research Protocol Strategy to ensure appropriate
approvals are received in a timely manner
■ Developed a shared understanding of each participant expectations for
the MLS Concept Evaluation
■ Ensured all Project Members are signed-up to the APAN MLS
Collaboration Portal
■ Ensured each Country Representative was involved in each Working
Group
■ Conducted preliminary testing of learning formats
30
CE Train-the-Trainer Workshop
Expectations
■ All participants will have a clear understanding of Concept
Evaluation Approach
■ US/International partners will ensure Research Protocol
Actions are accomplished
■ US/International partners will provide training to potential
CE volunteers, as required
31
Questions / Comments
Acronyms
Acronym
ADL
AFRL
AFSAT
ARJOCS
APAN
ASN RD&A
CalTech
CD
CDSA
CDTS
CE
COCOM
CoI
Acronym Meaning
Advanced Distributed Learning
U.S. Air Force Research Laboratory
U.S. Air Force Security Assistance Training Squadron
Arab Jordanian Center for Studies
All Pacific Area Network
Assistant Secretary of the Navy (Research, Development &
Acquisition)
Centre for eLearning Technology
Counter Drugs
Connecting Soldiers to Digital Applications
Center for Defence Technologies Studies
Concept Evaluation
Combatant Commanders’
Community of Interest
33
Acronyms
Acronym
CTFP
DAUK
DCTS
DIILS
DISAM
DOD
DoS
DRDC
DSAMS
DSCA
FKIE
GCFE
GRENA
HPW/RHA
Acronym Meaning
Combating Terrorism Fellowship Program
Defence Academy of the United Kingdom
Defence Centre for Training Support
Defense Institute of International Legal Studies
Defense Institute of Security Assistance Management
Department of Defense
Department of State
Defence Research and Development Canada
Defense Security Assistance Management System
Defense Security Cooperation Agency
Fraunhofer-Institute for Communications, Information
Processing, and Ergonomics
Guernsey College of Further Education
Georgian Research and Educational Networking
Association
U.S. Air Force Research Laboratory, Warfighter Readiness
Division
34
Acronyms
Acronym
HRPP
ICP
IMDSPDB
I-MET
IPO
JKO
JRMC
JSET
LMITT
LMS
MCSCG
MLS
Acronym Meaning
Human Research Protection Program
International Center for the Advancement of Research,
Technology & Innovation
International Student Pre-Departure Briefing
International Military Education and Training
International Program Office
Joint Knowledge Online
Jordanian Royal Medical Centre
Joint Security Cooperation Education and Training
Lockheed Martin International Training Team
Learning Management System
Marine Corps Security Cooperation Group
Multichannel Learning System
35
Acronyms
Acronym
MoD
MoLE
NCC
NETSAFA
ONR RC
ONRG
PAL
ROI
S&T
SAN
SCETP
Acronym Meaning
Ministry of Defence
Mobile Learning Environment
Naval Component Commands
Naval Education and Training Security Assistance Field
Activity
Office of Naval Research Reserve Component
Office of Naval Research Global
Personal Assistant for Learning
Return on Investment
Science & Technology
Security Assistance Network
Security Cooperation Education and Training Program
36
Acronyms
Acronym
SECDEF
Acronym Meaning
Secretary of Defense
SME
Subject Matter Experts
SSC
Space & Naval Warfare Systems Center
T&E
Testing and Evaluation
USCG/IA
U.S. Coast Guard International Affairs and Foreign Policy
USMC
United States Marine Corps
37