Draft Multichannel Learning System (MLS) Project Plan
Draft
Multichannel Learning System (MLS)
Project Plan
Prepared for:
Ms. Janie Glover
U.S. Naval Education and Training Security Assistance Field Activity
(NETSAFA) Technical Director
Project Manager
-- August 2013
1
Draft Multichannel Learning System (MLS) Project Plan
Table of Contents
Introduction ............................................................................................................................................. 3
Background ......................................................................................................................................... 4
Project Challenge ................................................................................................................................ 6
Current State of Learning Environments ............................................................................................ 7
Multichannel frameworks and MLS ................................................................................................... 7
Document Revision History ............................................................................................................... 8
Multichannel Learning System (MLS) Project ....................................................................................... 9
Overview ............................................................................................................................................ 9
Objectives ........................................................................................................................................... 9
Stage-Gate Approach .......................................................................................................................... 9
Concept ............................................................................................................................................. 10
Concept Evaluation........................................................................................................................... 10
Management Approach ......................................................................................................................... 11
Project Management ......................................................................................................................... 11
MLS Project Team............................................................................................................................ 12
Supporting Teams ............................................................................................................................. 12
Roles & Responsibilities .................................................................................................................. 12
US/International Organizations ....................................................................................................... 12
Project Working Groups ....................................................................................................................... 16
Learning Content Working Group .................................................................................................... 16
Testing and Evaluation Working Group ........................................................................................... 16
Human Research Protection Program ................................................................................................... 17
MLS Access Portal ............................................................................................................................... 18
Appendix ............................................................................................................................................... 20
Appendix A: Participating Organizations ......................................................................................... 21
Appendix B: Acronym Index............................................................................................................ 22
Index ..................................................................................................................................................... 23
Figures
Figure 1 - e-Learning Research Roadmap .............................................................................................. 5
Figure 2 - (Draft) Stage-Gate Approach ................................................................................................. 9
Figure 3 - Concept Evaluation (CE) Overview ..................................................................................... 11
Figure 4 - Organizational Chart ............................................................................................................ 12
Tables
Table 1: Document Revision History...................................................................................................... 8
ii
Draft Multichannel Learning System (MLS) Project Plan
Introduction
The Department of State (DoS), Department of Defense (DOD) and other US Government
organizations use training and education as a prime tool for achieving strategic objectives,
establishing rapport, opening communication, building capacity and maturing relationships with
Coalition and international partners. Training is the critical component in the International Security
Cooperation Program managed by the Naval Education & Training Security Assistance Field Activity
(NETSAFA), United States Marine Corps Security Cooperation Group (MCSCG), Air Force Security
Assistance Training Squadron (AFSAT), United States Coast Guard International Affairs and Foreign
Policy (USCG/IA), and the U.S. Army Security Assistance Training Field Activity (SAFTA). Defense
Security Cooperation Agency (DSCA) directs, administers and supervises the execution of the
Security Cooperation Programs managed by NETSAFA, MCSGC, AFSAT, USCG/IA and SATFA
and Defense Institute of Security Assistance Management (DISAM) provides professional education
to those staffs as well as Security Cooperation Offices.
In 2011 - 2012, a Mobile Learning Environment (MoLE) Coalition Warfare Program (CWP)
Project was initiated which focused on leveraging the global cellular network infrastructure, mobile
technologies and emerging mobile devices in order to facilitate the sharing of training content. As a
result of a two-year project that involved over 24 nations, the MoLE Project demonstrated the
capability to support training and education delivery via mobile devices, and showed that any new
content could be developed to support Department of Defense (DoD), Combatant Commanders
(COCOM) and other stakeholders by articulating and refining the requirements for the use case for
mobile learning.
The MoLE projects clearly showed that education and training conducted within mobile
learning domain could enable future military and civilian leaders to assume greater responsibility in
areas related to regional security cooperation, humanitarian assistance, peacekeeping, disaster relief,
medical disaster, piracy/smuggling interdiction, etc. It was an assessment of using mobile
technologies and showed that mobile applications, when developed appropriately (i.e., knowledge-,
community-, and user-centered) can be an effective learning medium for on-demand training. The
project, however; did not evaluate the effectiveness of learning material/content or the interoperability
among US/international partners.
The Multichannel Learning System (MLS) project will evaluate the effectiveness of using
multiple learning mediums (e.g., web-based learning, mobile applications, videos, e-publications, etc.)
as well as improve the tracking and traceability of joint training among coalition partners. It will
encompass a portfolio of multiple media systems, e.g., web-based, mobile learning applications,
videos, etc., in order to provide a more individualized approaches for the delivery of learning
courses/content to U.S. service members and our international partners. This will be accomplished
through a series of technical trials to a demonstration of these capabilities, which includes an
extensive proof-of-concept among all project participants.
The MLS Concept Evaluation (CE) Evaluation will include portions of the International
Military Student Pre-Departure Briefing (IMDSPDB) that is provided to international students who
receive training and education in the United States. As a result, these trials and the MLS CE should
enable the DOD and military training communities to maximize investments in international training
(i.e., an enhanced International Military Student Pre-Departure Briefing (IMDSPDB)) as well as
evaluate the effectiveness of using multiple mediums in meeting course/content learning outcomes.
These trials will also enable the DoD and military training communities to maximize investments in
international training, evaluate coalition interoperability capabilities, information sharing and
collaboration, as well as build new partnerships (e.g., Germany Ministry of Defense (MoD) and
India).
3
Draft Multichannel Learning System (MLS) Project Plan
The MLS Project has the endorsement of the Defense Security Cooperation Agency (DSCA),
based on the project’s capability to ensure that coalition partners have global access to the Military
Services’ state of the art online education and training programs. At the end of this project, the U.S.
will have an enhanced operational training capability with participating partner nations, and these
capabilities will be integrated into other Security Cooperation Education and Training Programs
(SCETP) managed by DSCA. DSCA and the Joint Staff will participate in the “U.S. Only MLS
SCETP Development and Integration Working Group,” as funding permits, to design and develop a
Concept Evaluation or prototype for the integration of this effort to implement any additional changes
to the Joint Knowledge Online (JKO), Security Assistance Network (SAN) or Defense Security
Assistance Management System (DSAMS).
Background
In 2008, the Commander, U.S. Naval Forces Europe (CNE) set a requirement for a
multinational, multilingual e-Learning environment that would support Allies, Coalition partners,
North Atlantic Treaty Organization (NATO) nations, Partnership for Peace (PfP) and selected African
nations. This requirement was based on meeting an urgent demand to provide training and education
capabilities that would meet the Combatant Commander’s (CCMD) requirement to effectively operate
in large areas of operations (AOR) where the most difficult challenge is the ability to train and
communicate due to the vast nature of the AOR’s with limited connectivity and infrastructure. To
meet this need, the Coalition Warfare Program (CWP) initiated a project, entitled the Multinational
Virtual Learning Environment (MVLE)1.
MVLE was based on an emergent requirement to establish an advanced distance/distributed
learning program, which would assist key regions of Europe and Africa in training future military and
civilian leaders. Since this was the first step in providing an e-Learning capability within the global
infrastructure, the MLVE Project focused on creating and strengthening partnerships in key nations in
the Black Sea Region (BSR)/South Eastern Europe (SEE), addressing DoD International Security
Cooperation objectives, using regional expertise in the African national, Baltic Sea, Caspian Sea,
Black Sea and Mediterranean Sea to assist in developing an advanced e-Learning capability. This
initial development and deployment effort involved ten countries2. The MLVE Project was an
important step in establishing a coalition e-Learning environment with non-traditional partners.
Through the use of a “stage gate approach,” the project showed that global partners could collaborate
to develop a capability based on common goals and objectives.
MVLE showed that education and training conducted within the “learning domain” could
enable future military and civilian leaders to assume greater responsibility in areas related to regional
security cooperation, peacekeeping, disaster relief, and piracy/smuggling interdiction. These efforts
were conducted in conjunction with the U.S. Department of Defense, U.S. European Command
(USEUCOM) and particularly the U.S. Navy in key regions of Europe. Although the project was quite
successful, it also identified several outstanding requirements that were needed in order to fully meet
CCMD Training Requirements and meet the needs of the individual user These requirements include:
(1) more functionality, (2) ease of use, (3) up-to-date learning content, (4) better, faster connectivity
and (5) more advance navigation.
1
2
Multinational Virtual Learning Environment (MLVE) Concept Evaluation Operational Evaluation Report; 1 October 2008
Azerbaijan, Bulgaria, Georgia, Germany, Moldova, Norway, Romania, Tadzhikistan, United Kingdom, United States
4
Draft Multichannel Learning System (MLS) Project Plan
2008
2009
2010
2011
2012
2013
MVLE CCIF/CWP Project
S-1000D API Interface Follow-on Project (N09-24)
General Topics
S-1000D API Interface (N08-35)
FY11-FY12 Coalition Warfare Program (CWP) (Mobile
Learning Environment (MoLE))
S&T Training Needs Analysis (N08-36)
Understanding and Assessing Cultural Differences (N09 -05)
What Do Learners Within An ADL Environment Need?
(N09-29)
Approaches in Azerbaijian to Regional Crisis Mgmt (N09-37)
Linguistic/Medical Terminology Assistance
(N11-02)
Swiss ADL Research & Protype Development
(N12-01)
Speech
enabled
Multimodal Enabled Advanced Distributed Learning (N08-14)
ADL Structuration and Complex Adaptive Systems (N09-36)
Understanding Cultural Differences When Deploying Speech-Enabled Capabilityes (N10-23)
What do learners in an m-Learning Environment Need? (N09-29)
m-SCORM Challenges (N09-35)
JFCOM Atlas
Pro Mobile
Learning (mLearning) and
Speechenabled
Capability
m-Learning
Bi-Lingual m-Learning: A Use Case (N09-39)
m-Learning S&T Requirements (N11-03)
Mobile Medical Development Project (N11-01)
Mobile Learning Contents for MSO (N11-37)
Collaborati
on
Projects
Mobile App Infrastructure for Cross Platform
Deployment (N11-38)
Coalition Chat Line (CCL)
Coalition Chat Requirements (Proposed)
Linguistics Requirements (Proposed)
Figure 1 - e-Learning Research Roadmap
After the MLVE Project ended, the Department of Defense (DoD) members3 of the MVLE
Working Group, identified several approaches to addressing the shortfalls identified post-MVLE
Project. As Figure 1 shows, these included additional science and technology research projects that
were focused on both the short-term requirements, which focused on providing a capability based on
the “low hanging fruit” of technology and on the long-term requirements, which focused on
capabilities needed by the DoD but had not yet matured (e.g., automatic language translation, speechenabled capabilities, etc.).
In working with the DOD Community, approximately ten e-Learning/advanced distributed
learning research efforts were initiated. The Office of Naval Research (ONR) and Office of Naval
Research Global (ONRG), focused more on the long-term goal, and investigated and evaluated the
effectiveness of a speech-enabled capability. As a result, both organizations funded a two-year
research project, entitled Multimodal Enabled Advanced Distributed Learning (ME-ADL), to assess
the use of speech as an enabled capability. The results show that using speech as an enabled
capability, by providing ‘snippets’ of learning content, is an effective way of ensuring that core
learning goals and objectives are fulfilled. It also showed that speech, as a learning medium, would
provide more interaction and engagement with multicultural environments.
In 2011, the Mobile Learning Environment (MoLE) was accepted as a Coalition Warfare
Program. Utilizing state-of-the-art in mobile technology, MoLE advanced e-Learning to a new level
and helped satisfy current training demands since most countries are using cellular networks, mobile
devices and emerging mobile application/service models to deliver a range of training experiences.
This capability, referred to as mobile learning or more commonly “m-Learning,” is through mobile
devices to wirelessly link remote and highly mobile users directly with resources to obtain training
and education.
3
USEUCOM, Office of the Secretary of Defence for Personnel & Readiness (OSD P&R), Deputy Director for Joint Staff (J7) Joint and
Coalition Warfighting (DD J7 JCW) Joint Knowledge Online (JKO), Commander Naval Forces Europe (CNE), U.S. Space & Naval
Warfare Systems Center Pacific (SSC-Pacific), SSC-Europe, SSC-Charleston, Office of Naval Research (ONR), Office of Naval Research
Global (ONRG), ONR Program 38 S&T
5
Draft Multichannel Learning System (MLS) Project Plan
The MoLE Project helped to address several of the shortfalls identified in the MVLE Project. It
leveraged the global cellular network infrastructure, mobile technologies, and emerging mobile
application/service models, which have resulted in building a mobile learning (m-Learning) capability
that integrated into the Deputy Director, Joint Staff (J-7) for Joint and Coalition Warfighting (DD J7
JCW) Joint Knowledge Online (JKO) portal. This facilitated the sharing of educational content
between US and multinational partners. .
Since the first e-Learning project, the Deputy Director for Joint Staff, Joint Coalition and
Warfighting (DD J7 JCW) Joint Knowledge Online (JKO) has made significant improvements to
support DoD/CCMD requirements (e.g., Language and Cultural Training, mobile learning, etc.).
However, there are still a few emergent requirements that are required to meet the full spectrum of
requirements. Since the conception of the MVLE Project, learning technologies have developed to a
point where the technology can personalize education, rapidly assess student learning, support social
learning, diminish learning boundaries (e.g., language training, cultural awareness, etc.), serous
games, intelligent tutoring, and provide alternative teaching methods and approaches to lifelong
learning.
In meeting the goal for a technology-enabled capability, the Multichannel Learning System
(MLS) Project was identified as a project to provide a more effective learning capability that would
meet the needs of U.S./international partners (e.g., regional engagement, exercise/scenarios, Security
Cooperation, Global Health Awareness, etc.). The goal of the MLS Project is to incorporate various
training modalities (e.g., audio, e-course, mobile apps, etc.) to advance the state of education and
training across DOD and the Federal Government.
Project Challenge
Training is critical to the success of a broad spectrum of deployed military operations. Most of
the current training methodologies and technologies are not well suited for addressing emergent
training requirements for the generation of learners (i.e., Generation X, Y and Z). This gap limits the
ability to meet the global training and education needs of today’s demographics as well as provide an
instructional medium that supports a personal approach.
Commanders, and other US defense-related organizations, that support international training,
are finding that live instructor-led training is oftentimes unsustainable due to a high-pace operational
environment, limited numbers of instructors and other budgetary constraints. To help bridge the gap,
the DOD has made significant investments in e-learning platforms. While traditional e-learning
systems are effective at delivering curriculum aimed at professional growth and development, they are
not ideally suited for portability, broad accessibility and/or real-time collaborative learning often
required during a plethora of military operations. As a result, technology with greater mobility is
required, not to replace existing technologies but to supplement them in order to extend the continuum
of training and education into environments not being sufficiently served.
It has been realized that the use of technology (e.g., social media, mobile apps, e-readers, audio
books, etc.) is so widespread that it must become an integral part of people’s lives and learning.
However, in many ways the technology is detached, and even isolated, from the human user. The
user, who the technology is supposed to serve, is subservient to the technology rather than the other
way around.
Technology needs to be ‘brain friendly’ in order to complement the cognitive process of the
human user. This is a critical requirement if the technology is going to be used, and if it is providing
an efficient and effective approach that is beneficial to the user and maximizes learning. This means
that the learning needs – course objectives – is the focus of the learning and not the, oftentimes
complex, navigation that is cognitively demanding.
6
Draft Multichannel Learning System (MLS) Project Plan
Current State of Learning Environments
Governments, enterprises, and commercial businesses are migrating towards the omnipresent
adoption of computational systems which adapt and personalize learning to individual needs.
Advanced computational technologies and algorithms are advancing in parallel with discoveries in
understanding human cognitive processes. The implications of these advances will dramatically form
the framework in which we develop instructional and learning strategies for the future.
The Military Services and the Educational Communities have continuously expressed a
commitment to improving each Service’s learning models by leveraging technology without
sacrificing standards and the quality of training while simultaneously complying with existing laws
and regulations. In looking at the learning continuum, the focus should be on providing dynamic
environments, blending physical and virtual learning as well as the social learning aspects to provide
an environment that focuses on meeting the needs of the individual learner and the learning outcomes.
This requires a re-look at the learning environment since the arrival of a new generation of recruits4
come from a technology-dense environment where most educational learning and experiences comes
from non-traditional learning approaches.
According to the ‘On Learning: Future of the Air Force Education and Training’ writings, in
addition to the widely accepted education model, Bloom’s Taxonomy of affective, psychomotor and
cognitive learning, learning must account for each of these domains and ensure education and training
also accounts for the emotional, physical, and mental aspects of learning. Therefore, the nature of
education and training dictates the right mix of training delivery in order to optimize and leverage the
use of new technologies to supplement, and where appropriate, replace “hands-on” training.
Therefore, the future of learning will be affected by the shifting paradigms in four areas – learners,
content, technology, and the geopolitical environment.
In just a few years, since the guidance document was written, the use of technology has
become so widespread that it has become an integral part of people’s lives and learning. However, in
many ways the technology is detached and even isolated from the human user. The user, who the
technology is supposed to serve, is subservient to the technology rather than the other way around.
Technology needs to be ‘brain friendly’ so it fits and complements the cognitive processes. This is
critical if technology is going to be used, and it the learning experience is to be efficient and effective.
If technology is not brain friendly it may not be used, or its use may be detrimental, or at least
cumbersome, rather than enhance learning.
Multichannel frameworks and MLS
In the last decade, the delivery of learning, education and training (LET) materials have been
revolutionized and new approaches to provide education and training content are available. E-learning
has become a collective term used to describe training by any electronic means (e.g., web-based,
mobile technologies, serious games, etc.) anytime and anywhere. This has resulted in the call for an
advanced state of LET referred to as “Personal Assistant for Learning (PAL)”. One of the key
attributes of PAL is tracking the experiences associated with learning or training performances across
different devices and formats while adapting to the learning styles of the individual – this is one of the
goals of the Multichannel Learning System (MLS) project.
In order to drive successful adoption of technology as a learning medium, we need to identify
the frameworks which will drive usability and efficacy. In “A Roadmap for Education Technology,”
the Computing Research Association identifies several key challenges which must be addressed in
order to successfully leverage the future computing technologies of tomorrow:
4
On Learning: The Future of Air Force Education and Learning; Air Education and Training Command, 30 January 2008
7
Draft Multichannel Learning System (MLS) Project Plan
1. Personalizing Education
a. Develop computational tools which systematically mimic a human tutor by
understanding an individual’s strengths, weaknesses, and motivations.
b. Understand learning models that are representative of what learners know and can do by
identifying pedagogy unique to each individual learner
c. Developing data models which explain themselves to the user by addressing interface
challenges between user and software
2. Assessing Student Learning
a. Evolve beyond the traditional “Teach/Stop/Test” learning model to a model which
assesses learners understanding seamless without formal test boundaries.
b. Understanding the characteristics and attributes that are truly representative of learning
competencies
3. Diminishing Boundaries
a. Understanding how to best support learning which occurs outside of traditional academic
settings
b. Understanding how to support the transition between formal and informal learning
environments without compromising learning effectiveness
4. Developing Alternative Teaching Modes
a. Understanding how software can support both social learning collaboration and content
coaching
b. Capturing the key processes by which learners generate and evaluate knowledge.
Technology-enhanced learning, which involved the use of multiple learning modalities like
audio books, avatars, e-readers, mobile apps, speech, tutoring, etc., must be built integrated to
maximize learning and meet the needs of the individual using the above frameworks are key
considerations. This also means that “learning needs” drive the technology and makes learning easier
since the learner to focus on the actual learning and not how to locate the learning material. Under
this approach, the user focuses on the learning objectives instead of how to acquire the information.
For example, many e-learning modules make navigation complex and demanding. This taxes the
brain resources for navigation through the learning module, rather than on the learning content.
Through the use of speech as an enabler, the user will be able to rapidly gain access to the learning
instead of focusing on the ‘next button’ so they can get to the screen that contains the course.
Therefore, a technology-enhanced capability needs to be flexible by using different approaches
to navigation through the learning journey, and one that strives to minimize the cognitive load on the
learner so the user can focus on the learning and not on figuring out how to navigate. By setting up
expectations via consistency, the technology can guide and help the users’ brain intuitively process
information; hence, reducing cognitive load. This means that learning should be engaging, interesting
and interactive and suited to meet the needs to the learners’ preference.
Document Revision History
The MLS Management Plan is a living document. As conditions change and lessons are
learned, this plan will be updated and coordinated with the critical stakeholders.
Versions
0.1
0.2
0.3
Date
1 December 2012
8 April 2013
1 August 2013
Description
Original
Rev 1
Rev 2
Status
Draft Version
Draft Version
Draft Version
Table 1: Document Revision History
8
Draft Multichannel Learning System (MLS) Project Plan
Multichannel Learning System (MLS) Project
Overview
The MLS Project Activities will be separated into phases, referred to as ‘stage-gates,’ which
aligns to fiscal years (i.e., Phase I (FY 2014) and Phase II (FY 2015)). To assure MLS Project Goals
and Objectives are met, a stage-gate approach will be finalized as a product of the initial MLS KickOff Review with participation from the international Participates who have volunteered to participate
based on letters of commitment toward the development, testing and deployment of a sustained
capability.
The MLS Project Management Plan will be finalized and all project task, deliverables, and
actions will be identified according to each project phase and incorporated into the updated MLS
Management Plan.
Objectives
The MLS Project has both a short-term and long-term strategic goal. The short-term goal is to
evaluate a capability that enables the MLS international partners to evaluate the effectiveness of using
a multichannel capability that will be used by more than 140 countries that participate in the Security
Cooperation Education and Training Program (SCETP).
The long-term goal is to assess this capability by using courses that were provided by the
Security Cooperation Programs, and other DoD Organizations, via the Joint Knowledge Online (JKO)
where long-term sustainability and interoperability is assured. This objective will permit the Security
Cooperation Education and Training Program (SCETP) Process to automatically track and report the
successful completion of multichannel-related training to Congress; and permit the SCET Working
Group to design and deliver an integrated MLS capability into the DD J7 JKO LMCS, Security
Assistance Network (SAN) and Defense Security Assistance Management System (DSAMS).
Stage-Gate Approach
The Multichannel Learning System (MLS) Project will commence in October 2013 based on
decisions by The Defense Institute for Security Assistance Management (DISAM) Deputy
Commandant, and the Naval Education and Training Security Assistance Field Activity (NETSAFA)
Technical Director based on Figure 2. The full timeline, milestones and deliverables will be
confirmed after the MLS Kick-Off Meeting.
Figure 2 - (Draft) Stage-Gate Approach
9
Draft Multichannel Learning System (MLS) Project Plan
After the MLS Kick-Off Project Meeting, Figure 2 will be modified to reflect international
participant research and development activities.
Concept
The Multichannel Learning System (MLS) Project provides a unique opportunity for the
Security Cooperation Education and Training (SCETP) Programs, led by the Defense Institute of
Security Assistance Management (DISAM) and the Naval Education and Training Security
Assistance Field Activity (NETSAFA), to significantly improved the International Military Student
Predeparture Brief (IMDSPDB), which is provided to both officers and enlisted personnel from more
than 140 countries. In the current format, which is more than ten years old, potential students are
provided access to the material via the DISAM Website, on a CD or provided via their respective
country’s U.S. American Embassy.
The MLS Project is seen as an opportunity to update the content, as well as the manner in
which it is made available, so foreign students traveling to the United States have an enjoyable
experience. For example, many of the students come from a wide variety of national backgrounds
(e.g., Europe, Asian Subcontinent, Pacific Rim, etc.) and experience both cultural and military shock.
This unfamiliarity of the US environment can cause problems in lieu of the demanding classroom
environment which they may face. In addition to the cultural and military shock, there may be
technological challenges (e.g., personal technology, limited access to the internet, etc.) encountered in
preparation for their US-based training. Therefore, the MLS Project will look at the importance of
contemporary pedagogical issues related to using various technologies (i.e., web-based learning,
mobile applications, video, e-publications, etc.); and conduct an in-depth analysis of the International
Military Student Pre-Departure Briefing (IMDSPDB) Indoctrination Training Program to obtain a
understanding of where the content aligns with the ‘learner journey’(e.g., pre-departure, in-transit and
post-arrival indoctrination requirements).
The project will consist of a multi-phased approach toward the development of an enhanced
IMDSPDB Indoctrination Training Program. Phase I will include the selection of a small subset of the
Indoctrination material that will be used as a test case. Based on the aforementioned objectives (i.e.,
pedagogical issues related to using various technologies, and understanding of where the content
aligns with the ‘learner journey’), the MLS Project Team will test the effectiveness of the content to
meeting ‘indoctrination material requirements’. The results of this test will be documented and
recommendations will be made regarding enhancements for the entire Indoctrination Training
Program.
Phase II, which will occur after mid-2014, will be a decision by DISAM on how the final
IMDSPBD Product will be upgraded to address the aforementioned objectives as well as develop the
content to ensure that it is optimum for each learner generation. Once the upgraded capability has
been developed, interested MLS participants are encouraged to participate in an evaluation of the
enhanced capability before it is fielded.
The MLS Project also has a long-term objective of developing a capability that permits
international participants access to courses other than the IMDSPDB Courses (e.g., Security
Cooperation, Trafficking in Persons, etc.). In addition to accessing additional courses, international
participants will have the course results pushed to the respective Reporting Databases. For example,
for the Security Cooperation Courses, the results will be pushed to the ‘Security Cooperation
Reporting Databases’ to meet DOD and other US Agency Requirements. For courses like the
Trafficking in Persons, which is a required course for all DOD personnel, the respective Service/DoD
Personnel System will be able to pull the results from JKO for entry into each person’s training
record.
Concept Evaluation
10
Draft Multichannel Learning System (MLS) Project Plan
The MLS Project has two goals; specifically, (1) Identify what is the best method to provide
distance education to prepare international military students for resident training in the United States;
and (2) Identify a ‘Joint’ solution for providing controlled access to online courses to support other
Security Cooperation Training Organizations via the Joint Knowledge Online (JKO) capability.
Figure 3 – Concept Evaluation (CE) Overview
In order to “Identify what is the best method to provide distance education to prepare
international military students for resident training in the United States,” Figure 3 (Concept
Evaluation (CE) Overview) applies. Each of the CE volunteers will receive an email informing them
of the link to the MLS CE Portal that will be used to obtain their acceptance to the MLS CE Informed
Consent as well as the collection of demographic data. Once this has been accomplished each
volunteer, based on the CE strategy developed by the Testing and Evaluation Working Group, will
take one or more courses identified by the Learning Content Working Group.
For the data collection component, each of the courses will consist of a pre-test, the content
being tested, a post-test, and a qualitative and quantitative comments questionnaire. The data
collection approach will permit the MLS Testing and Evaluation Team to analyze the data to see
which methods provided the optimum return-on-investment for the respective demographics. In
addition to the courses, additional content may be added to allow users to evaluate the effectiveness of
additional tools to support learning. For example, there are components of the International Military
Student Pre-Departure Briefing (IMDSPDB)) content which includes a checklist to help potential
students prepare for their US training. A checklist will be provided to evaluate it effectiveness in
checking and/or adding additional items.
In reference to “Can the MLS Project provide a ‘Joint’ solution for providing controlled
access to online courses to support other Security Cooperation Training Organizations via the Joint
Knowledge Online (JKO) capability”, this component will be addressed separately by the Integration
and Development Working Group, a US-only working group, that will look developing a Strategic
Roadmap for developing and integrating future Security Cooperation Training Reporting
Requirements into a repository system.
Management Approach
Project Management
The Multichannel Learning System (MLS) Project will consist of a Management Team and
three working groups (i.e. Learning Courses, Testing & Evaluation, and the US-only Integration and
Development); US Department of Defense (DOD) Advisors, and the prime subcontractors.
11
Draft Multichannel Learning System (MLS) Project Plan
Figure 4 - Organizational Chart
MLS Project Team
Project Manager – NETSAFA Technical Director
Project Management Support – Lockheed Martin International Training Team (LMITT),
Q&P Ltd
Learning Working Group – DISAM
Testing & Evaluation Working Group – 711th HWP/RHA, ONR RC
Supporting Teams
Learning Content Working Group – NETSAFA, DISAM, LMITT and Country Team
Delegates will participate to ensure the development of the learning content meets core learning
informational objectives.
Testing & Evaluation Working Group - 711th HPW/RHA, ONR RC, Defence Academy
United Kingdom (DAUK), Q&P Ltd, and a representative from each country will ensure the testing
and evaluation process meets the goals and objectives of the MLS Project as well as DoD Human
Research Protection Program (HRPP) Requirements.
Integration and Development Working Group – NETSAFA, LMITT, DD J7 JKO,
AFSAT, SATFA, MCSGC, USCG, DISAM, Security Assistance Network (SAN) Representative, and
Defense Security Assistance Management System (DSAMS) Representative. This working group
will develop a Strategic Roadmap for developing and integrating future Security Cooperation
Training Reporting Requirements into a repository system so data is available to Army Training
Requirements and Resources System (ATRRS) and DSAMS.
Roles & Responsibilities
To achieve both horizontal and vertical communications and integration among the
US/International Organizations, the following MLS Organizational Roles and Responsibilities were
defined.
US/International Organizations
12
Draft Multichannel Learning System (MLS) Project Plan
a.
U.S. Naval Education and Training Security Assistance Field Activity
(NETSAFA)
-
b.
Deputy Director, Joint Staff (J-7) for Joint Knowledge Online (DD J7 JKO)
-
c.
Co-Lead for the Testing & Evaluation Working Group
Participate in all MLS discussions & reviews
Participate in the MLS Collaboration Portal
Participate in the MLS CE
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Office of Naval Research Global (ONRG)
-
g.
Leads the Testing & Evaluation Working Group
Participate in all MLS discussions & reviews
Participate in the MLS Collaboration Portal
Participate in the MLS Concept Evaluation (CE)
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Office of Naval Research (ONR) Reserve Component (RC) Test & Evaluation
(T&E) Team
-
f.
Lead for the Learning Courses Working Group
Participate in all MLS discussions & reviews
Learning Courses MLS Collaboration Portal Lead
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
U.S. Air Force Research Laboratory, Warfighter Readiness Research Division
(711th HPW/RHA)
-
e.
Participate as a member of the Integration and Development Working Group
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Defense Institute of Security Assistance Management (DISAM)
-
d.
Project Leader for the Multichannel Learning System (MLS) Project
Facilitates MLS Management Meetings, Teleconferences, and related discussions
Participate in the MLS Collaboration Portal
Liaise with Stakeholders and other key personnel to ensure project goals and
objectives are addressed
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Participate in all MLS discussions & reviews
Provide co-funding to support MLS-related Science & Technology (S&T) and
Project Meetings
U.S. Naval Space & Warfare System Center Pacific (SPAWAR Pacific)
13
Draft Multichannel Learning System (MLS) Project Plan
h.
Advanced Distributed Learning (ADL) Initiative
-
i.
-
Participate in the MLS Project Reviews
Canadian Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
Germany: Centre for eLearning Technology (CalTech) of the Saarland
University, German Research Centre for Artificial Intelligence and Higher
School of Technology and Economics
-
l.
Learning Courses Advisor to the Learning Courses Working Group Lead
Participate in MLS discussions & reviews
Provides support to the Integration and Development Working Group
Participate in the MLS Collaboration Portal
Participate in the MLS CE
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Canada: Canadian Defense Academies
-
k.
Participate in all MLS discussions & reviews
Provide co-funding to support MLS-related Science & Technology (S&T) and
Project Meetings
Participate in the MLS CE
Adhere to the US Department of Defense (DOD) the Human Research Protection
Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics
Lockheed Martin International Training Team (LMITT)
-
j.
DoN Approval Authority for the Human Research Protection Program
(HRPP)/Research Ethics Protocol Documents
Participate in all MLS discussions & reviews
Participate in the MLS Collaboration Portal
Participate in the MLS CE
Participate in the MLS Project Reviews
German support for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
Germany: Fraunhofer-Institute for Communication, Information Processing,
and Ergonomics (FKIE)
-
Participate in the MLS Project Reviews
German support for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
m. Germany: GIRAF PM Services GmbH
-
Participate in the MLS Project Reviews
German Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
14
Draft Multichannel Learning System (MLS) Project Plan
n.
Georgia: Georgian Research and Educational Networking Association (GRENA)
-
o.
-
-
-
Participate in the MLS Project Reviews
Serbian Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
United Kingdom: Q&P, Limited
-
t.
Participate in the MLS Project Reviews
Jordan Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
Serbia: Center for Simulations and Distance Learning of the Military Academy
in Belgrade/Jefferson Institute
-
s.
Participate in the MLS Project Reviews
Indian Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
Jordan: King Hussein Medical Center
-
r.
Participate in the MLS Project Reviews
Provide participants for the MLS CE
India: Hindustan University Center for Defence Technology Studies (CDTS)
-
q.
Participate in the MLS Project Reviews
Georgian Lead for the resolution of Human Research Protection Program
(HRPP)/Research Ethics Requirement as discussed in Research Ethics
Provide participants for the MLS CE
Georgia: International Center for Advancement of Research, Technology &
Innovation
-
p.
Provide participants for the MLS CE
Provide support to the MLS Project Manager, DISAM Online Director, and the
Testing and Evaluation Leads
MLS Project Lead for Human Research Protection Program (HRPP)/Research
Ethics Requirements
Coordinates and participate in MLS discussions and reviews
United Kingdom: Defence Academy of the United Kingdom (DAUK) at
Cranfield University
-
Participate in the MLS Project Reviews
DAUK and European Union (EU) Representative for the resolution of Human
Research Protection Program (HRPP)/Research Ethics Requirement as discussed
in Research Ethics
Participate as an active member of the Testing & Evaluation Working Group
Provide participants for the MLS CE
15
Draft Multichannel Learning System (MLS) Project Plan
u.
United Kingdom: Tribal Education Ltd
-
v.
Provide Technical Support to the Learning Content Working Group
Participates in all MLS discussions and reviews
Develops technical architecture to support the MLS CE
Ensure Testing & Evaluation Plan is aligned with the MLS CE Informed Consent
United Kingdom: Sutton College of Learning Adults (SCOLA)
-
Participate in the MLS Project Reviews
Lead for ensuring the Human Research Protection Program (HRPP)/Research
Ethics Requirement as discussed in Research Ethics meet their academic
requirements
Provide participants for the MLS CE
Project Working Groups
The MLS Project will consist of two working groups that will ensure the project goals and
objectives are fulfilled, which will form the foundation of the project (i.e., Learning Content,
Technology & Transition, and Testing & Evaluation). The third working group, the Integration and
Development, will be a US-only team focused on meeting US-specific requirements and will not be
referenced any further in this project plan.
Learning Content Working Group
The Learning Courses Working Group will ensure the learning courses provided by the US and
the International Participates are segmented, as appropriate, to ensure they are developed according to
the optimum format (e.g., audio course, e-course, apps, etc.) to meet the learning objectives. Initially,
the goals and objectives of this working group include:
(1) Identify learning courses that are being provided by US/International Participates to
ensure course-specific learning objectives are met. Specifically,
-
Ensure courses being provided by US/International Participates can be provided
in multiple formats
-
International partners assist in adapting content to meet national requirements
-
Civil military cooperation content may also be included as appropriate
(2)
Identify the delivery formats that will support just-in-time learning through the use of
various formats in order to ensure a technology-enabled capability is provided that
complements traditional training and education systems as well as more of a
“personal assistance for learning” approach
(3)
Identify assessment tools and develop instructional measures of effectiveness based
on research being conducted by US/International partners
Testing and Evaluation Working Group
16
Draft Multichannel Learning System (MLS) Project Plan
The Testing and Evaluation (T&E) Team will focus on establishing the testing and evaluating
process that will show that the MLS Concept Evaluation meets the project goals and objectives.
Initially, the goals and objectives of this working group include:
(1)
Conduct an operational evaluation of technology-enabled capability (e.g., audio books,
e-readers, mobile apps, speech interface, tutoring, etc.), utility, usability and suitability
in a multinational environment; specifically:
- Coordinate with the Learning Courses Working Group and the Technology &
Transition Working Group to identify critical assessment issues
- Develop operational measures of effectiveness and performance
- Develop data collection and analysis plans to ensure these plans are sufficient to
fully evaluate the project’s technical components
- Coordinate the data collection and analysis plan, with the MLS Project Research
Protocol Administrator, to ensure it complies with the Human Research Protection
Program (HRPP)/Research Ethics requirements
(2)
Provide recommendations on how the project’s lessons learned may be incorporated
into future learning initiatives to ensure it meets the goals and objects of the
Department of Defense (DOD) and other US agency international training
requirements.
Human Research Protection Program
The Multichannel Learning System (MLS) Project is categorized5 as a research project since it
is a “systematic investigation, including research development, testing and evaluation, designed to
develop or contribute to generalizable knowledge” and data collection will be through intervention or
interaction with individuals. This data collection will occur using a number of sources (e.g., surveys,
data analysis, apps, etc.), and interactions will be developed in accordance with the Testing and
Evaluation Plan and disseminated in the MLS Concept Evaluation (CE) Informed Consent.
Although the data collection effort will not impinge on the privacy, confidentiality of the
participates, nor pose any risk to the individuals involved, the Informed Consent document will be
developed and utilized to ensure that each volunteer is aware of the purpose, duration of the Concept
Evaluation, procedures and processes, Testing & Evaluation Process, risk and discomforts, potential
benefit, voluntary Participate and withdrawal and confidentiality. In addition to these issues, for that
volunteers that are not proficient in English, the Informed Consent will be provided in their native
language.
During the first year of the project, each country will have a representative responsible for
ensuring all Human Research Protection Program (HRPP)/Research Ethics issues meet their
Country’s Requirements. Each will be required to have read the Belmont Report6, the Federal Policy
for the Protection of Human Subjects7, and provide documentation to ensure the research protocol
strategy, required by their respective country, is aligned with the MLS Informed Consent process.
From a US-perspective, the MLS Project is exempt from review since the project is ‘research
conducted on the effectiveness of or the comparison among instructional techniques, curricula, or
5
Title 45 CFR 46
http://ohsr.od.nih.gov/guidelines/belmont.html
7 Title 45 CFR 46, Subpart A
6
17
Draft Multichannel Learning System (MLS) Project Plan
classroom management methods’.8 However, since the MLS Project is a global project that involves
international participation, it will have an Institutional Review Board (IRB) to ensure that the testing
and evaluation process meets all requirements identified by the United States9, European Union10, and
each International Country participating. This process will be identified as a milestone that must be
completed during the first ten months of the project.
This is an important issue since every country participating in the project may, or may not, have
the same requirements as the US. For example, the European Union is more restrictive with regards
to the Data Protection Requirements and the Rights of Individuals. Accordingly, the EU Data
Protection Requirements state that ‘all study Participates present in a research project need to be
informed about the planned research use of the collected data independently of the type of data
collected’ – ‘Informed Consent is not just about Patients.11 This requirement means that individuals
need to be aware of the (a) Methods used for handling personal data, (b) Justification for requesting or
obtaining their data, (c) Duration of data use and storage, and a (d) Guarantee concerning the rightful
use of the data.
Therefore, to ensure that all Human Research Protection Guidelines are followed, each
International Country Representative involved in the MLS Project will be required to have one
representative that will ensure their respective country’s guidelines and requirements are fulfilled.
This individual will be responsible for coordinating with the appropriate Country Representative and
for obtaining written permission before they can participate in the MLS Concept Evaluation. Once
these letters have been obtained, the MLS Project will have both an Independent IRB and US
Government IRB to ensure no actions are outstanding.
MLS Access Portal
Access to MLS Project Collaboration is located at https://wss.apan.org/1539/JKO/mls. This
Community of Interest (CoI) provides a repository of project materials in order to ensure each
stakeholder and participant has access to relevant materials/briefings. To ensure information is
provided across the project, the following libraries are available:
- On the front page, participants will see a brief description of the MLS Project and the Project
Calendar. On the left-hand side of the portal, which is referred to as the “Quick Launch”, participants
have access to the Contacts, MLS Shared Documents, Learning Content Library, Mobile Learning
Library, Research Protocol Library, and Testing & Evaluation Library. The Management Team also
has access to the Management Library and the Integration Library, which is related to the US-only
Working Group (Integration and Development Working Group).
- Two “working group” libraries (i.e., Learning Courses, Testing & Evaluation) contain
relevant documents relevant to each working group. Each contains the title to the item as well as a
brief description.
- The Management library, which is only viewable by the Management Team, contains key
project documents and presentations. Adobe PDF documents of each can be viewed in the “MLS
Shared Documents” section.
8
32 CDR 219 (Sec 219.101)
DoD Directive 3210.7; DoD Directive 3216.2; DoD Directive 5400.11; SECNAVINST 3900.39D and 32 CFR 219; U.S.
Air Force Research Laboratory (AFRL) Instruction 40-402
10 EU Directive 95/46/EC; Ethical Review in SEVENTH Framework Programme (FP7); Guidance for Informed Consent
11 Ethical Review in SEVENTH Framework Programme (FP7); Guidance for Informed Consent
9
18
Draft Multichannel Learning System (MLS) Project Plan
- The Mobile Learning library contains presentations/documents related to developing mobile
capabilities as well as collaborations with the Joint Knowledge Online (JKO).
- The Research Protocol Library contains references/guidance instructions related to Human
Research Protocol Requirements. Once the MLS Research Protocol Documents and the CE Informed
Consent has been approved by any International Institutional Review Board (IRB), these documents
will also be added to this library.
19
Draft Multichannel Learning System (MLS) Project Plan
Appendix
20
Draft Multichannel Learning System (MLS) Project Plan
Appendix A: Participating Organizations
Canada
Georgia
Georgia
Germany
Germany
Germany
India
Jordan
Serbia
Serbia
United Kingdom
United Kingdom
United Kingdom
United Kingdom
United States
United States
United States
United States
United States
United States
United States
United States
United States
Canadian Defence Academies
Georgian Research and Educational Networking Association (GRENA)
International Center for the Advancement of Research, Technology & Innovation
Centre for eLearning Technologies (CalTech) of the Saarland University, German
Research Centre for Artificial Intelligence and Higher School of Technology and
Ergonomics
Fraunhofer-Institute for Communications, Information Processing, and Ergonomics
(FKIE)
GIRAF PM Services GmbH
Hindustan University Center for Defence Technologies Studies (CDTS)
King Hussein Medical Center
Center for Simulations and Distance Learning of the Military Academy in Belgrade
Jefferson Institute
Defence Academy of the United Kingdom (DAUK) at Cranfield University
Q&P, LTD
Sutton College of Learning Adults
Tribal Education Ltd
Advanced Distributed Learning (ADL) Initiative
Defense Institute of Security Assistance Management (DISAM)
Joint Knowledge Online (JKO)
Lockheed Martin International Training Team (LMITT)
Naval Education and Training Security Assistance Field Activity (NETSAFA)
Office of Naval Research Global (ONRG)
Office of Naval Research Reserve Component (ONR RC)
U.S. Air Force Research Laboratory (ARFL) Warfighter Readiness Research
Division (711 HPW/RHA)
U.S. Naval Space & Warfare Systems Center Pacific (SPAWAR Pacific)
21
Draft Multichannel Learning System (MLS) Project Plan
Appendix B: Acronym Index
AFRL
AFSAT
AOR
APAN
ATRRS
BSR
CCMD
CDTS
CalTech
CoI
CWP
DAUK
DISAM
DOD
DoS
DSAMS
DSCA
FKIE
GRENA
HPW/RHA
HRPP
IMDSPDB
IRB
JCW
JKO
LET
MCSCG
MLS
MoD
MoLE
MVLE
NATO
NETSAFA
ONR
ONRG
PAL
PfP
CE
SAFTA
SAN
SCETP
SCOLA
SEE
T&E
USCG/IA
USEUCOM
U.S. Air Force Research Laboratory
U.S. Air Force Security Assistance Training Squadron
Area of Responsibility
All Pacific Area Network
Army Training Requirements and Resources System
Black Sea Region
Combatant Commanders’
Center for Defence Technologies Studies
Centre for eLearning Technology
Community of Interest
Coalition Warfare Program
Defence Academy of the United Kingdom
Defence Institute of Security Assistance Management
Department of Defense
Department of State
Defense Security Assistance Management System
Defense Security Cooperation Agency
Fraunhofer-Institute for Communications, Information Processing, and Ergonomics
Georgian Research and Educational Networking Association
U.S. Air Force Research Laboratory, Warfighter Readiness Division
Human Research Protection Program
International Student Pre-Departure Briefing
Institutional Review Board
Joint and Coalition Warfighting
Joint Knowledge Online
Learning, Education and Training
Marine Corps Security Cooperation Group
Multichannel Learning System
Ministry of Defence
Mobile Learning Environment
Multinational Virtual Learning Environment
North American Treaty Organization
Naval Education and Training Security Assistance Field Activity
Office of Naval Research
Office of Naval Research Global
Personal Assistant for Learning
Partnership for Peace
Concept Evaluation
U.S. Army Security Assistance Training Field Activity
Security Assistance Network
Security Cooperation Education and Training Program
Sutton College of Learning Adults
South Eastern Europe
Testing and Evaluation
U.S. Coast Guard International Affairs and Foreign Policy
U.S. European Command
22
Draft Multichannel Learning System (MLS) Project Plan
Index
Acronym Index ..................................................................................................................................... 21
Background ............................................................................................................................................. 4
Concept ................................................................................................................................................. 10
Current State of Learning Environments ................................................................................................ 7
Document Revision History .................................................................................................................... 8
Human Research Protection Program ................................................................................................... 17
Introduction ............................................................................................................................................. 3
Learning Content Working Group ........................................................................................................ 16
Management Approach ......................................................................................................................... 11
MLS Access Portal ............................................................................................................................... 18
MLS Project Team ................................................................................................................................ 12
Multichannel frameworks ....................................................................................................................... 7
Objectives ............................................................................................................................................... 9
Overview ................................................................................................................................................. 9
Participating Organizations ................................................................................................................... 20
Project Challenge .................................................................................................................................... 6
Project Management ............................................................................................................................. 11
Project Working Groups ....................................................................................................................... 16
Concept Evaluation (CE) Framework ................................................................................................... 10
Roles & Responsibilities ....................................................................................................................... 12
Stage-Gate Approach .............................................................................................................................. 9
Supporting Teams ................................................................................................................................. 12
Testing and Evaluation Working Group ............................................................................................... 16
US/International Organizations............................................................................................................. 12
23