Multichannel Learning System (MLS)
Concept Evaluation Train-the-Trainer
Meeting Minutes
The Defense Institute of Security Assistance Management (DISAM)
Dayton, Ohio
16-17 December 2014
1
1. Introductions
On 16-17 December 2014, the Defense Institute of Security Assistance Management
(DISAM) hosted the Multichannel Learning System (MLS) Concept Evaluation (CE) Train-theTrainer Meeting. The meeting served several purposes; specifically, (a) obtain briefings from the
international partners on their research and development activities related to MLS, (b) provide an
update from the MLS Working Group Leads, and (c) to conduct a “Train-theTrainer” session in
preparation for the Concept Evaluation.
The MLS CE Train-the-Trainer Meeting consisted of US representatives from the Naval
Education & Training Security Assistance Field Activity (NETSAFA), DISAM, U.S. Marine
Corps Security Cooperation Group (MCSCG), U.S. Air Force Research Laboratory Human
Performance/Warfighter Readiness Division (711th HPW/RHAS), Jefferson Institute, and
Lockheed Martin International Training Team (LMITT). International representatives included
Defence Research and Development Canada (DRDC); Siebenundvierzig ING GmbH & Co. KG,
Germany; Jordanian Royal Medical Services (JRMS); and Serbia Military Academy Belgrade.
US representatives attending via Defence Connect Online (DCO) included US Army Security
Assistance Training Field Activity (SATFA) Training and Doctrine Command (TRADOC), U.S.
Space & Naval Warfare Systems Center Pacific (SSC Pacific), and LMITT. International
representatives participating via DCO included GIRAF PM Services GmbH, Germany; and
Cranfield University, United Kingdom.
2. Action Items
a. Variability Testing: A Variability test is needed to ensure the data collection questions
are asked in a consistent manner with an ease-of-understanding. PoC: Kristen Barrera; Status:
Open
b. MLS Data Collection Portal: Several modifications are required to the MLS Data
Collection Portal (i.e., use of terminology, Jordanian Flag, additional open-ended question,
automatic upload of MLS Registration PINs, and the evaluation process). PoC: Jacob Hodges/
Bjőrn Hayer; Status: Open
c. MLS Registration Email Process: The MLS Registration and Data Collection Process
requires modification so that CE volunteer receives only one email for the entire evaluation
process. PoC: Jacob Hodges/Julie Howell; Status: Open
d. CE Informed Consent: The MLS Concept Evaluation Informed Consent will be provided
to all international leads, for translation, once the SSC Pacific IRB has approved it. PoC: Jacob
Hodges/Julie Howell; Status: Open
2
e. MLS CE Train-theTrainer Briefing: A MLS CE Train-the-Trainer Briefing is needed to
support US/International organizations desiring to provide training to potential CE volunteers.
The CE Train-the-Trainer Briefing will include a brief project introduction, information on the
Learning Formats and the Course Material, the CE process, and the Data Collection approach.
PoC: Jacob Hodges; Status: Open; will be provided to US/International partners once the MLS
Data Collection Portal has been updated
f. Raw Data Files: All MLS US/International Participants that desire to have a copy of the
aggregate and raw data need to provide specifics on (a) where the data will reside, (b) where will
it be stored/safeguarded, and (c) what safeguards will be in place to ensure the data collected is
safe (i.e., locked-up, accessibility, storage, etc.). PoC: Julie Howell/Andrea Loesch/ Jerzy
Jarmasz SSC Pacific IRB; Status: Open; Under Review
g. Use of Terminology: Due to concerns regarding usage of key terms (i.e., e-pub, web
based, and content), all MLS Document and briefings will be modified to show e-pub as e-book,
web based as e-learning, and content as course material. PoC: Jacob Hodges; Status: Closed
h. Canadian Research Protocol Teleconference: A teleconference is needed with the
Canadian Research Ethics Chair to discuss Informed Consent issues, individual privacy, data
maintenance, and other “incidental findings”. PoC: Julie Howell/Jerzy Jarmasz; Status: Closed
i. User-preferred Learning Format Question: An additional open-ended question is needed
so that the analysis team can determine ‘if there was a learning format preferred that the
volunteer could not access. PoC: Jacob Hodges; Status: Closed
j. CE Informed Consent: The CE Informed Consent requires changes to reflect new
terminology and MLS Registration Process. PoC: Jacob Hodges/Julie Howell; Status: Closed
k. MLS CE Train-the-Trainer Briefings: Upload all MLS CE Train-the-Trainer Briefings
and documents to the JKO MLS Data Collection Portal. PoC: Jacob Hodges; Status: Closed
3. Discussion Items
Day 1 – 16 December 2014
a. Janie Glover, NETSAFA/MLS Project Manager, opened the meeting by welcoming
everyone attending in person and those attending via the DCO. She started by informing MLS
participants of the death on the third member of the Jordanian Royal Medical Services (JRMS)
Team, Lt Col Dr. Awnl Abu Lall. She provided her condolences to this family and stated he
would be missed. She also provided ‘get well wishes’ to Saboohi Famili, United Kingdom (UK)
3
Guernsey College of Further Education (GCFE), who is currently unable to attend since she is in
the hospital.
Publicly, Janie Glover wanted to provide a “Thank You” to a few people. Specifically,
(1) The DISAM Team, Deputy Commandant Dr. Ron Reynolds, his support staff, and
Mark Ahles for enduring all that we have went through in getting to this stage in the project.
(2) To Tink L’Ecuyer for setting-up this meeting and takes care of all the problems (e.g.,
logistics, arrangements, etc.);
(3) All of the International Partners for “hanging-in” and staying with the program in spite
of all the problems that the project faced. She further added that the MLS Project did have a plan,
faced many frustrations, lack of funding, shutdowns and portal issues;
(4) To the Germany Team (i.e., GIRAM PM Services GmbH and Siebenundvierzig ING
GmbH & Co. KG) and Kristen Barrera for continuing to pursue the formulation of the MLS
Coalition Warfare Program (CWP) Collaborating Funding Contract;
(5) To SSC Pacific and the Research Protocol (RP) Team for their diligence in developing
the RP strategy; and to
(6) The Security Cooperation Education & Training Program (SCETP) Community and
their Counterparts for continuing to support the MLS Project.
Janie Glover concluded by saying that she wanted to ensure everybody leaves the two-day
meeting with a shared understanding of how the CE will work, and that she really appreciated
everyone coming.
b. Mark Ahles, DISAM/Deputy MLS Project Manager, welcomed everyone to WrightPatterson Air Force Based (WPAFB), on behalf of the Deputy Commandant, Dr. Ron Reynolds.
He reminded the participants that DISAM is a teaching institution for international students
coming to the US, and that DISAM conducts research related to each Security Cooperation core
area. He added that this project is important since sharing information and research findings
across all spectrums is key to enhancing learning. He closed by thanking everyone for attending.
c. Doug Howard, MCSCG, said that although the Marine Corps is the smallest of the
Services, when international students arrive at the Marine Corps Schools for training they may
be the only international student attending. If we can help them to be better prepared for the
training experience, it is important. Any international student assistance is important and the
Marine Corps takes training very seriously. Therefore, the Marine Corps is supportive of MLS.
d. Janie Glover, kicked-off presentations by briefing Multichannel Learning System (MLS)
Concept Evaluation Train-the-Trainer Meeting. She reminded participants of the short-term and
long-term goals, added that the implications are high, and said other organizations can use what
4
the project has developed to support their training and education requirements. She then
reviewed the Organizational Chart, Stakeholders and Participating Organization. She provided a
recap of the International Team organizations and their involvement before reminding the
participants of the Project Objectives, Goals, Deliverables and Timeline after the Concept
Evaluation Train-the-Trainer Meeting
Janie Glover discussed three groups involved in the CE (i.e., International Military
Students (IMS); International Partners, and US Government, Civil Servants and Contractors).
She provided an overview of Figure 1, which shows the Learning Formats/Delivery, Data
Collection Portal, Evaluation Approach, and Collaborative Discussions amongst the
US/International participants.
Figure 1: Concept Evaluation Overview
Janie Glover then provided a brief overview of each Working Group and their
responsibilities in meeting project goals and objectives. She closed by highlighting the
collaboration activities, from the past year, and the CE Train-the-Trainer Meeting Expectations.
e. Dr. Aiman Al Sumadi, Jordan Royal Medical Services (JRMS), presented Using MLS
Learning Approaches to Support Humanitarian Assistance Requirements: A Jordan Case Study.
He started by providing recognition, and condolences, to a former colleague, LtCol Dr. Awnl
Abu Lall, who passed away on 8 December 2014. Aiman Al Sumadi then proceeded to provide
background information on JRMS’ Military, National, International and Humanitarian Role. He
highlighted the JRMS International Peace Delegates, UN Peacekeeping Forces Medical Teams,
the Deploying Field Hospitals to Disaster Areas and War Zones, and the Syrian Refugee Crisis.
He noted that due to the influx of Syrian Refugees, the Jordanian Guard Forces (JBGF) and
JRMS have to provide assistance for refugees at the 45-border crossing points see Figure 2.
5
Figure 2: Refugee Humanitarian Aid Process
Along this border, Aiman Al Sumadi noted there were eight clinics and three of them
receive refugees 24/7. At the crossings, the refugees undergo a Triage (i.e., assessed and
assisted, by JBGF soldiers qualified as Tactical Medics in addition to JMRS medical personnel)
and a Refugee Management Process. Based on the experience of JRMS in UN
Peacekeeping/Humanitarian Missions, there is a need for Crisis Management in order to contain
the physical and psychological impact on refugees – the importance of MLS Training
Capabilities.
Aiman Al Sumadi then discussed the need for MLS Learning Approaches to support
refugee assistance requirements. He highlighted a “Humanitarian Assistance Requirements to
Support Emerging Refugee Crisis Meeting” held on 6 June 2014 in Jordan that was sponsored by
the Office of Naval Research Global (ONRG). Physicians, university professors, lawyers, UN
Workers, UNHCR Representatives, Military Officers and Officials from Jordanian Ministry of
Health and Jordan’s Civil Defense attended the meeting. Discussion focused around medical and
non-medical issues and tools for communication/collaboration. However, the key requirement
identified during the meeting was technologies for training non-medical personnel in basic
medical issues not covered in tactical medic training courses. The ONRG-sponsored meeting
confirmed that research is needed to identify requirements on how to introduce humanitarian
medicine, refugee medicine and disaster medicine into the basic education and training
curriculum of medical schools.
Aiman Al Sumadi concluded by discussing the feasibility of MLS Learning in
Humanitarian Assistance. He said that there appeared to be a lack of entry and mid-level
qualifications. The only qualifications available to humanitarian workers in developing countries
are expensive master’s degree in a range of non-humanitarian courses. He highlighted that
training is an essential part of the professional development in providing life-saving relief
assistance in often-hazardous missions. He concluding by saying the learning experience may
6
involve multiple channels of learning, and that training should be organizationally oriented, and
provide certification to participants be either competency- or curriculum-based.
f. Dr. Venkat Sastry, UK Cranfield University, presented Measurement Approaches. He
started by reviewing the initial MLS Kick-Off Meeting discussions. He highlighted that
motivational factors and the course material mostly influence learning. He made note that the CE
course material might not be sufficiently strong enough for a rigorous statistical evaluation. He
then addressed the question ‘what are we trying to measure?’ In response to this question, he
referenced the MLS Project Objectives and the Evaluation Strategy taken by the US/International
partners.
The biggest challenge is the assistance needed from each country lead to ensure that the
volunteers evaluate more than the first three chapters in the course material more than two
learning formats. He expressed concern that many students may opt for a snippet evaluation (i.e.,
evaluate only parts of the content on learning format and then switch to another format in order
to meet the two-format request) versus completely evaluating each learning format (i.e.,
hyperlinks, search functions, etc.). Whilst he agreed that the International Military Visitor
Briefing (IMVB) has many benefits to multichannel learning approaches, it may be difficult to
show the importance based on a limited evaluation and the number of participants that will be
taking part.
Venkat Sastry concluded by proposing that Cluster Analysis be used so that the analysis
can show similarities and trending data. However, there is for many data; otherwise, the results
will be unpredictable. He closed by saying that the main challenge to the analysis team will be
conducting different kinds of analysis (e.g., format evaluations, likes and dislikes, etc.) and then
correctly interpreting them if there is not a large sample size.
g. Col Goran Šimíc, PhD, Serbian Military Academy Belgrade and Ms. Biljana Presnall,
Jefferson Institute presented MLS in MoD Republic of Serbia. Goran Šimíc started by saying
they recently joined the US’s Advanced Distributed Lab (ADL) Partnership, and they are
building formal documents. The Serbian Military Academy Belgrade has e-learning courses for
different platforms. They have their own ADL system, which is used to create apps, create user
accounts, and develop course content for both online and offline usage.
Col Goran Šimíc then introduced their e-Learning resources, which includes a Distance
Learning Portal, Learning Platform, Mobile Learning Project, Course Development, and Library
Digitization, as shown in Figure 3. He also discussed the International Cooperation and
Knowledge Exchange, and Serbia’s participation in the Multinational Environment.
Biljana Presnall introduced the Moodle LMS, the Mobile Learning Applications, and the
motivation behind the learning approach used by the Serbian Military Academy Belgrade. She
presented numerous graphics on the Mobile Learning Applications, both desktop and mobile
courses, which demonstrated their success in providing a mobile learning capability. She
concluded by discussing ongoing projects and their future plans. To obtain additional
information visit the Serbian Military Academy Belgrade website.
7
Figure 3: Military Academy Belgrade e-Learning Resources
h. Dr. Jerzy Jarmasz, Defence Research and Development Canada (DRDC), presented
Individual Training and Education (IT&E) Research for the Canadian Armed Forces. He started
by stating some of the things that Canada is doing to support IT&E might be useful to the MLS
Project. He discussed the Canadian Defence Academy (CDA) approach to multiple learning
capabilities.
As shown in Figure 4, the CDA uses a Knowledge Management Tool to validate and
support the learning process, starting from the Instructor Support to the Validation of Advanced
Learning Environments. Jerzy Jarmasz discussed several research projects (i.e., Framework for
the Application of Mobile Learning, Virtual Resource Centre, Mobile App – Small Arms
Instruction, Training Toolkit for the Comprehensive Approach, Neuro-Adaptive Training
Systems Development, Distributed Learning: Command & Control LiveSpace Lab (C2L2), the
OnLiNe Government Advanced R&D Environment (ONGARDE), and CAF Mobile App Store
(CAFMAS).
He highlighted several research projects. The Training Toolkit for the Comprehensive
Approach is the best way to push information out quickly (e.g., supporting quick deployments).
The C2L2 is enabling collaborative workspaces, which is rich in multimedia environments.
ONGARDE promotes collaboration and engagement where research, development and
evaluation activities occur, as well as supports the CAF Campus Learning Portal to enable
information sharing of projects, lessons learned and best practices.
8
Figure 4: CDA Thematic Themes
i. Julie Howell, SSC Pacific/Research Protocol Lead, presented MLS Concept Evaluation
(CE) Research Protocol (RP) Requirements. She started by introducing her organization for new
members of the team. She then identified the RP Team, which consisted of US Government
Leads, MLS Project Team and Country Representatives. She discussed the Terms of Reference
(ToR) to ensure everyone had the same understanding of the definitions (i.e., research, human
subjects, and private information). She reviewed the US, Canada, and EU Directives (i.e., US
Title 32 CFR 219.101, US Department of Defense (DoD) Directive 5400.11, DOD Directive
3216.2, Canada Tri-Council Policy Ethical Conduct for Research Involving Humans, EU
Directive 95/46/EC, EU Guidance on Informed Consent, UK Ministry of Defence JSP 536, etc.).
As shown in Figure 5, Julie Howell confirmed there were three groups of volunteers (i.e.,
International Military Students (IMS), US Government Civilian and Military, and International).
Each will receive an ‘Invitation to Participate’ email that provides their respective point-ofcontact (PoC) and a copy of the MLS Concept Evaluation Informed Consent. Within three days,
each potential human subject will receive the MLS Instructions email so each volunteer can to
participate.
She concluded by reiterating the importance of each organization providing a PoC. She
said the PoC’s purpose was is to ensure there was no influence on potential CE volunteers. The
PoC will be the person that provides the email address of any potential volunteer, and answers
any questions regarding the MLS CE process. Julie Howell also mentioned they were working
with the individual country representatives to obtain an International Investigator’s Agreement
(IIA), which is a document that ensures the international RP Leads are following the Belmont
Principles.
9
Figure 5: Email Instructions Process
Jerzy Jarmasz, DRDC, said that he was willing to coordinate a teleconference between
Julie Howell and his IRB lead to address any questions regarding informing volunteers of their
rights, how their information might be shared, where the data is going to reside, etc.
j. Dwayne Eldridge, DISAM/Learning Content Lead, presented MLS Learning Content. He
provided a brief reminder of the Learning Content Working Group Objectives before discussing
the International Military Student Pre-Departure Briefing (IMSPDB) Course Material that will
be contained in each of the four learning formats (i.e., e-book, e-learning, mobile app and video).
He said each learning format would contain all seven sections (i.e., Culture, Individualism,
Punctuality, Informality, Egalitarianism, Short-Term Mentality and Daily Life in America) of the
IMSPDB Course. However, the chapters on Culture, Individualism, and Punctuality will be
testable.
Dwayne Eldridge then stepped through the capabilities and features of each learning
format, as shown in Table 1. He concluded by reminding everyone that CE volunteers are
requested to evaluate two different learning formats in order to determine which learning formats
provide the optimum leaning opportunities for international student coming to the United States.
He also said that volunteers should be encouraged to evaluate all of the features of each learning
format (i.e., hyperlinks, search functions, start/stop, non-testable material, etc.).
Learning Format
Type
e-book
Viewing Capabilities
 Apple (epub)
 Android (epub)
 Amazon (mobi)
Features




10
Hyperlinks
Index
Search Function
Table of Contents
e-Learning
 Personal Computers
 Not available on Android
Devices and Apple iPad (Adobe
Format)
Mobile App
 Download using QR Code
 Simulated app for Apple and
Windows-based users
 Personal computers
 Tablets
Video
 Direct viewing via MLS
Data Collection Portal
 Hyperlinks to external
resources
 Start/Stop Features
 Easy to access
 Hyperlinks
 Direct viewing via MLS
Data Collection Portal
 Hyperlinks
 Start/Stop Features
Table 1: Learning Format Capabilities & Features
k. Janie Glover closed the first day’s meeting by providing each participant with a hardcopy
of Figure 6, in preparation of Day 2. She concluded by saying she thought the first day was
successful.
Figure 6: Concept Evaluation Engagement Discussion
Day 2 – 17 December 2014
a. Dr. Kristen Barrera, 711th HPW-RHAS/Testing & Evaluation Lead, discussed the MLS
Testing, Evaluation & Analysis Plan. She recapped the MLS Goals, Objectives, and the Testing
& Evaluation Objectives. She discussed the areas of evaluation (i.e., Technology, Usability or
ease-of-use, usefulness and utility, learning transfer and interest/desirability). One of her
concerns in evaluating the usefulness and utility of multiple formats; i.e., the ability to choose the
preferred format, was that some volunteers may not evaluate the full capabilities of each learning
format.
11
Kristen Barrera then discussed the six Concept Evaluation Criteria. These include
technology usage (i.e., personal computers, tablets, and mobile devices), usability (i.e.,
learnability, user satisfaction, and ease-of-use), learning (i.e., learning transfer, and sharing of
information), usefulness (i.e., benefits or availability of multiple formats), desirability or interest
(i.e., was the user engaged or interested), and utility (i.e., was the format easy-to-use, did the
features work). She then identified several test limitations and uncontrolled variables. For
example, the user limitations include users of Apple Devices since the e-Learning and video
content is in Adobe format. In addition, the mobile application will not be available for Apple or
Windows users.
The uncontrolled variables consist of (1) responses to questions and (2) learning formats.
Since each evaluation participant is not required to answer every question, it is expected there
will be some questions that volunteers decide not to answer. Another uncontrolled variable is the
learning format evaluations. Since volunteers are asked to select two different formats, they need
to understand that evaluating an Android and Kindle-formatted e-book is only one format
evaluation.
Figure 7 shows, each participant will register after receiving an email that provides access
to the MLS Data Collection Portal. Each volunteer is assigned a unique four-digit personal
identification number (PIN), which consists of a country code and research identification
number. Once the volunteer has accessed the portal, he/she will be required to (1) accept the
informed consent, (2) complete the initial survey questions (i.e., demographics, and pre-test
questions), (3) evaluate the learning formats, and (4) complete the final questions (i.e., post-test
questions, general questions, format-specific questions, and attribute questions). During this
entire process, no personal information or identifiable characteristic is collected on the volunteer.
12
Figure 7: Data Collection Process
Kristen Barrera then stepped through the Data Collection Questions (i.e., demographics,
general, and technical). From this discussion, several thoughts emerged. Janie Glover pointed
out that the Management Team was relying on the international leads to determine if the
questions were too hard or not understandable. Although the international partners helped
develop the questions, viewing all of the questions as a complete set, the US/international
partners needed to ensure there would not be any language problems or misunderstanding of the
questions intent. After extensive discussion on English language proficiency, each partner agreed
they would try to obtain volunteers with an acceptable reading proficiency, and that Grade 8 is
probably the benchmark. In response to concerns over the complexity of the questions, Kristen
Barrera stated that she would conduct a variability test on the questions.
Kristen Barrera then presented the MLS Analysis Approach for interpreting the findings.
She discussed how the Statistical Data Analysis would be presented, Data Analysis Approach,
and the Analysis Questions (i.e., utility, learning, interest, user satisfaction, usability, ease-ofuse, learnability of the formats, utility of the formats, and cross correlation analysis). She
concluded her discussion by recapping the projected evaluation timeframes, learning format
evaluation expectations, and format capabilities.
Immediately following Kristen Barrera’s discussion, a question was asked about the use
of terms provided throughout briefings and documents (i.e., e-pub, web-based, and content).
Whilst the project team has an understanding of the meanings of these terms, the volunteers may
not. Therefore, it was recommended that terms that are more common be used (i.e., e-book, eLearning, and course material).
b. Jacob Hodges, LMITT, and Bjőrn Hayer, Siebenundvierzig ING GmbH & Co. KG,
presented MLS Data Collection Portal. Bjőrn Hayer started by introducing himself and his
company since this was his first meeting with the MLS Project Team. He proceeded by
providing an overview of the Data Portal. He showed pictures of two tables (i.e., Results and
13
Summary) that captured the evaluation results. The Result Table presented the PIN, the
question, and the each response. The Summary Report showed that each question had a unique
identification number, the response to each question, and the number of responses per question.
He then provided an example of the data collections results using pie charts from the Summary
Charts page.
Bjőrn Hayer then showed the ‘Participants Progress’ page, which is a display of the
progress of each volunteer (i.e., activity status). This page reflects each PIN number that has
been registered in the database and its status (i.e., registered, finished pre-test, and finished). He
then showed User/Content section, which identifies the PIN along with the associated learning
format evaluated. He concluded by discussing the Registration Database, which is an automatic
upload for each volunteer receiving the MLS Registration email. He concluded the database
discussion by showing that the MLS Data Collection Portal provides complete anonymity, that
no email address, or identifying information is linked to the data collection process.
Jacob Hodges presented the Data Collection Approach. He started by discussing the
Concept Evaluation Recruitment Process and how the email addresses are obtained. Specifically,
(1) Each International Military Student (IMS) attending training in the US, according to
the Security Assistance Network (SAN) Web Training Database, is invited to participate in the
Concept Evaluation.
(2) An organizational PoC for the MLS Project (i.e., NETSAFA, DISAM, MCSCG, and
711 HPW/RHAS) will recruit US Government and Civil Servant Concept Evaluation volunteers.
(3) An International PoC, from each international partner participating in the MLS
Project, will recruit Concept Evaluation volunteers.
(4) Once Jacob Hodges receives the email addresses of volunteers, he will send each
volunteer an ‘Invitation to Participate’ email, which contains a copy of the approved MLS
Concept Evaluation Informed Consent Form. Each organizational PoC will also be included on
the email so that they are aware that the Concept Evaluation has started.
The next issue covered was the Data Collection Security. Jacob Hodges pointed out that
none of the data collected involves privacy or personal identification information. However, in
meeting the research protocol requirements of all countries involved, each MLS Concept
Evaluation Volunteer has certain rights; these include:
(1) They must be informed which data is being collected (i.e., demographics and
responses to questions).
(2) They must be informed why the data is being collected (i.e., identify the best methods
for providing distance education to international military students, and evaluate the effectiveness
of having multiple learning formats to support Security Cooperation Education and Training
Program (SCETP) requirements).
14
(3) They must be aware that no data will be collected which could possibly identify them.
Therefore, to ensure that all legal and organizational issues regarding the safeguarding of
data, each volunteer is required to read the MLS CE Informed Consent. This provides the
assurance to the potential volunteer that no US/International partner will gain access to the data
without appropriate approval from the SSC Pacific IRB Lead. It also states that only one person
will know the email address of each potential volunteer, which is on a two-password protected
flash disk.
In concluding the Data Collection Approach section, Jacob Hodges showed an example
of the MLS Instructions email. He showed that each mail provides an introduction (i.e., what is
expected, how long it will take, etc.), identifies the CE requirements (i.e. registration, informed
consent, demographics, etc.), evaluation/access (i.e., evaluation two different learning formats).
He then discussed the Final Instructions email, which will be sent to each volunteer within 24
hours after they have started evaluating the learning formats.
Extensive discussions then ensued as to why there was a 24-hour delay in sending the
second email, why two sets of emails are required, and are steps necessary. Jacob Hodges
responded by saying the 24 hour delay was to permit time for each volunteer to evaluation two
learning format and, and allow volunteers to return to the portal, using the MLS Registration
hyperlink, to review other learning formats, if they desired to do so. However, several
US/International partners wanted to conduct the evaluation at one time vs. having the potential
volunteers re-enter the portal.
A comment was also made that volunteers did not have to go through all of the content in
each learning format, but sample some of each. This was a concern raised by Kristen Barrera
during the Testing, Evaluation and Analysis Presentation. If each volunteer does not evaluation
each learning format, in its entirety (i.e., Table 1: Learning Format Capabilities), how will the
analysis team identify the best methods for providing distance education to international military
students?
The final decision was that the MLS Invitation process would be revised which required
only one email to kick-off the registration process. US/International PoC’s would be required to
let each potential volunteer know that they must evaluate both learning formats before selecting
the ‘continue the evaluation’ otherwise the volunteer would no longer have access to the learning
formats.
Jacob Hodges then provided an overview of the Data Portal Collection process. He
showed that each volunteer is required to “certify they have read, acknowledge, and accept the
terms and conditions of the Concept Evaluation Informed Consent.’ In cases where they may be
concerns over the volunteer’s understanding of the English language, a copy of the
acknowledgement statement and Informed Consent is provided in Arabic, Deutsch, French,
Georgian and Serbian. He then showed a graphic of what each volunteer would see once they
have entered the Data Collection Portal. On the left-hand side of the portal, a status bar informs
15
the volunteer where they are at in the evaluation process (e.g., demographics, pre-test, post-test,
etc.), with the corresponding area in bold print. The center of the screen will display the question
with optional responses.
Jacob Hodges concluded the Data Collection Portal section by showing how several areas
looked (i.e., learning formats page, learning format evaluations, and text fields). Bjőrn Hayer
concluded the MLS Data Collection Portal presentation by conducting a walkthrough of the
MLS Data Collection Portal so that all participants would have an idea how the process worked.
c. Jacob Hodges briefly outlined phases for conducting the CE as shown in Figure 8. The
intent of the discussion was to ensure US/International partners understood the process, so that
they would know when they could start CE Recruitment. In Phase I, the SSC Pacific IRB
Approval Letter will start the recruitment process. After CE volunteers have been identified,
each organization’s PoC will send an email to Jacob Hodges containing the email addresses of
CE volunteers.
Jacob Hodges will then send each CE volunteer an “Invitation to Participate” email. This
email will (a) introduce the project sponsors, (b) identify the project objective, (c) state the
purpose of the research study, and (d) provide the CE Informed Consent Form. This will permit
CE volunteers an opportunity to (a) read the CE Informed Consent and (b) to ask their respective
PoC for clarity on areas they do not understand. Within three-days of receiving the “Invitation to
Participate” email, each CE volunteer will receive the “MLS Instructions” email so that they can
start the evaluation process.
Figure 8: Concept Evaluation Phases
d. The final discussion involved the engagement of the international partners. Each
international partner was asked to discuss their approach to seeking CE volunteers and if they
planned to conduct their own train-the-trainer session.
16
(1) Canada: Use a combination of mass emails to potential volunteers who are likely to
participate and provide an MLS CE Awareness Briefing based on the material presented at this
meeting.
(2) Georgia: They have not developed a strategy for implementing the CE; private and
local universities are using some types of multichannel learning, and it is a high priority for the
Georgian MoD. For the CE, they plan to seek volunteer students from different universities, IT
personnel. Georgia will conduct awareness training using the materials provided.
(3) Jordan: JRMS will seek military personal, and they are looking at having about 30
volunteers with more for back up. The will conduct awareness training, which will make the CE
efficient.
(4) Serbia: Plans to seek CE volunteers that have both a military and civilian blend. They
will use the briefing material provided as an awareness briefing to potential participants.
e. The MLS CE Train-the-Trainer Meeting closed after Jacob Hodges presented a draft of
the Action Items (see paragraph 2. Action Items).
17