THE LEVEL OF LEARNING EFFECTIVENESS UTILIZING THE VIRTUAL
ACADEMY FOR THE FBI
by
Anthony R. Gonzalez
A Graduate Capstone Project
Submitted to the Extended Campus
in Partial Fulfillment of the Requirements of the Degree of
Master of Aeronautical Science
Embry-Riddle Aeronautical University
Extended Campus
Cincinnati, Ohio Resident Center
May 2007
THE LEVEL OF LEARNING EFFECTIVENESS UTILIZING THE VIRTUAL
ACADEMY FOR THE FBI
by
Anthony R. Gonzalez
This Graduate Capstone Project
was prepared under the direction of the candidate’s Project Review Committee Member,
Dr. Steven Harbour, Adjunct Associate Professor, Extended Campus,
and the candidate’s Project Review Committee Chair,
Dr. Edward Gordhammer, Associate Professor, Extended Campus, and has been
approved by the Project Review Committee. It was submitted
to the Extended Campus in partial fulfillment of
the requirements for the degree of
Master of Aeronautical Science
PROJECT REVIEW COMMITTEE:
Steven Harbour, Assistant Professor, Aeronautics
Committee Member
Edward Gordhammer, Ph.D.
Committee Chair
ii
ACKNOWLEDGEMENTS
Above all, I thank God for blessing me with the ability and resources to pursue
and complete this endeavor. Without His constant mercy, grace and help, I can do
nothing.
I also thank my wonderful wife, Elizabeth, for the patience and understanding she
exhibited for these past two and a half years. Her support during this journey has been
tremendous, even through the late nights, evening classes, and weekends that were often
full of homework and study assignments. My two children, Alexander and Julianna, also
showed their Dad a lot of support and interest in what I was studying and how I was
progressing. I pray that my love for education passes on to you both. Thanks, guys!
Appreciation also must go to my co-workers at the FBI. The encouraging
responses I received through the surveys, interviews and conversations were incredible.
Thanks also to Steven Harbour and Dr. Edward Gordhammer for their time,
suggestions and their expertise in helping me complete this project.
iii
ABSTRACT
Researcher:
Anthony R. Gonzalez
Title:
The Level of Learning Effectiveness Utilizing the Virtual Academy for the
FBI.
Institution:
Embry-Riddle Aeronautical University
Degree:
Master of Aeronautical Science
Year:
2007
The FBI, like many organizations, uses Web/Computer-Based Training for employees
and members of the law enforcement community. This method of distance education is
practical and economical. The FBI Virtual Academy is an Internet/Intranet-based
Learning Management System that provides “anywhere, anytime” training to its users.
Convenience, ease of use and inexpensive delivery are definite advantages of this
platform. However, problems exist because instructional designers fail to consider
principles of adult learning during course development. Disregarding these principles can
negatively influence the effectiveness of the training. This project explored the manner in
which employees use the FBI Virtual Academy and their preferred learning environment.
Analysis of interviews and surveys helped determine if Virtual Academy courses provide
effective transfer of learning.
iv
Table of Contents
Page
PROJECT REVIEW COMMITTEE
ii
ACKNOWLEDGEMENTS
iii
ABSTRACT
iv
LIST OF TABLES
viii
LIST OF FIGURES
ix
Chapter
I
II
INTRODUCTION
1
Background
1
Researcher’s Work Setting and Role
2
Statement of the Problem
3
Significance of the Problem
3
Limitations
5
Assumptions
5
Acronyms
6
REVIEW OF RELEVANT LITERATURE AND RESEARCH
7
Introduction
7
The History of Web/Computer-Based Training
7
The Adult Learner
8
Motivation
9
Interaction
11
Content
12
v
Transfer of Learning
13
Research Hypotheses
14
Hypothesis One
14
Hypothesis Two
14
Null Hypotheses
III
14
Null Hypothesis One
14
Null Hypothesis Two
14
RESEARCH METHODOLOGY
15
Research Model
15
Survey Population
15
Sources of Data
16
Data Collection Device
16
Distribution Method
17
Procedures
17
Treatment of Data
17
IV
RESULTS
19
V
DISCUSSION
24
Hypotheses
24
Hypothesis One
24
Hypothesis Two
25
Virtual Case File
VI
26
CONCLUSIONS
28
Office Distractions
28
vi
VII
Change of Environment
29
Confidence in Virtual Academy Courses
29
Teaching Methods Contrary to FBI Instruction Philosophy
30
Time
30
Timing of Survey
30
RECOMMENDATIONS
31
Improve Content and Increase Interaction
31
Public Relations
32
Increase Incentives
32
Increase Management Support
33
Encourage Discussion (Mandatory Courses)
33
Areas for Further Research
33
REFERENCES
35
APPENDICES
38
A
BIBLIOGRAPHY
38
B
DATA COLLECTION DEVICE
39
vii
LIST OF TABLES
Table 1 – Employee Preference of Virtual Academy or Classroom-Based Courses
Table 2 – Virtual Academy/Classroom-Based Courses vs. Job Confidence
viii
19
LIST OF FIGURES
Figure 1 – Virtual Academy Courses Increase Job Confidence
20
Figure 2 – Classroom Courses Increase Job Confidence
21
Figure 3 – Employees Prefer Online Course over Classroom-Based Environment
21
Figure 4 – Classroom-Based Teaching Is More Effective Than Web/Computer-Based
Training
22
Figure 5 – Virtual Academy Courses Are Effective
22
Figure 6 - Interaction is Important in a Learning Environment
23
ix
1
CHAPTER I
INTRODUCTION
Background
Web/Computer-Based Training (W/CBT) courses have existed for over a decade.
Advances in computer technology and increased Internet connection speeds have allowed
schools and corporations around the world to utilize Web/Computer-Based Training for
students and employees. In many instances, online learning, also known as e-Learning or
distance learning, has replaced the traditional classroom setting. Students and workers are
now able to complete a short training course or even a degree program on their
computers.
Organizations are investing hundreds of thousands of dollars to develop,
implement and maintain “in-house” training infrastructures. In 2004, companies in the
United States employing at least 100 people had a total training budget of $51.4 billion.
(Gitonga, 2006) In 2006, total training spending increased to $55.8 billion.
Learning technologies are widely used by companies of all sizes. Nearly 60% use
virtual classroom technologies, 40% have learning management systems, 25 – 30% use
application simulation and rapid e-learning tools, and 14% use learning content
management systems. (Business Wire, 2007)
The design of these systems enables employees to access training any time and
any place to fill potential skill and knowledge gaps that may be present. Another
important advantage for using Web/Computer-Based Training is economics. Companies
can potentially save millions of dollars in travel and training costs. Initial costs for a
2
quality Web-based training system can be high, but the benefits, especially long term, far
outweigh the price.
The FBI Virtual Academy is an Internet/Intranet-based Learning Content
Management System that provides online course instruction to educate employees. These
courses are in addition to traditional face-to-face classroom courses held at the FBI
Academy in Quantico, Virginia and other locations around the United States.
Since its inception in 2002, thousands of FBI and law enforcement personnel have
used Virtual Academy for in-service registration and continuing education, enhancing the
knowledge, skills and abilities needed to support the criminal justice community.
Researcher’s Work Setting and Role
The researcher works as a Training & Civic Liaison Specialist (TCLS) for the
Training Division of the FBI Cincinnati Field Office. There are 56 FBI Field Offices in
the United States and each has at least one such position. The duties and responsibilities
of this individual include coordinating, approving and recording the training activities
between FBI/law enforcement personnel and Virtual Academy. Each employee is able to
accomplish most of the actions on their own. However, the approval stage in the
registration process passes through the TCLS. Most of the time, information about new
courses, mandatory training and registration passes from Headquarters to the TCLS, who
then sends it out to the appropriate office personnel.
The researcher also holds an FBI Instructor Certificate. He is responsible for
analyzing, designing, developing, implementing and evaluating training in the Cincinnati
Division.
3
Interaction with employees and feedback occurs often. The researcher has been
able to develop a good sense of employee opinions towards Virtual Academy training
and classroom-based training. The interaction also allows the researcher to develop a
strong sense as to the operation and weaknesses of the FBI Virtual Academy. Feedback
also exposes TCLS employees to the attitudes, difficulties and successes encountered by
Virtual Academy users.
Statement of the Problem
Many corporations in the United States are moving from classroom-based training
to online Web/Computer-Based training. Again, one reason for this move is economics;
however, another major reason is the increasing demand for training itself. Over the past
several years, an increasing number of workers have stated that additional training is
necessary for them to maintain and/or build their trade skills, knowledge and abilities. In
addition, studies show that workers strongly prefer informal on-the-job training and selfpaced training to formal classroom training. (Horton, 2000)
The Virtual Academy is the FBI’s answer to addressing this need. The problem
lies in its delivery system. The design platform and appearance of Virtual Academy is
professional, visually pleasing and user-friendly. However, many courses contain weakly
presented subject matter that lacks interaction and does not take into consideration basic
adult learning theory principles. This drawback can seriously influence the effectiveness
any online training the FBI attempts to incorporate.
Significance of the Problem
Adults learn differently than children. It is apparent that some in the FBI Training
Development Division are creating simple computer-based training modules that do not
4
take into consideration adult learning principles. By disregarding these principles,
ineffective, or even negative learning transfer can occur, wasting valuable time and
resources. Many training programs, including the FBI Virtual Academy, use a traditional
training model that focuses more on the needs of the organization than on the needs of the
learners.
Many courses in Virtual Academy fail to present the content in a manner
consistent with these principles. Several times a year, courses appear that are mandatory
for all personnel to complete. The courses contain vital sensitive material that every
person working at the FBI needs to know and apply. Yet, many courses are noninteractive, and quite boring. Employees feel frustrated with the notion that they must
spend time out of their already busy day to complete these tedious tasks. The courses are
nothing more than textual information presented in a glorified slide show.
Every student must take an active role in the learning process in order for
effective learning transfer to occur. Unfortunately, this transfer is hindered by the fact
that little, if any active learning occurs in Virtual Academy. Employees navigate through
the text-laden modules, and complete assessments that may contain confusing, obscure
questions. In addition to its lackluster content, a number of employees have also
expressed frustration with Virtual Academy’s weak technical support and poor system
implementation.
The Strategic Human Capital Plan, published in 2006 by the Office of the
Director of National Intelligence, is very specific when it addresses the importance of
training. This Plan affects the military and 11 United States Intelligence agencies. It
states, “We must take a far more integrated, “corporate” approach to education, training,
5
and career development…Our professionals are critical to our mission, and we must
ensure they receive the very best education, training and career-long development,
defined and delivered by a “corporate” system of higher learning and education.” (ODNI,
2006, p.21)
Paralleling this philosophy, the current Administration launched E-Gov
Initiative1, encouraging “e-Learning investments as part of a strategic, systematic, and
continuous development of Federal government human capital.” (www.whitehouse.gov)
In its present state, the FBI Virtual Academy is not addressing these initiatives.
There are obvious economical benefits in examining this problem. There is also a
possibility that research in this area and addressing these issues may improve security and
morale within the FBI.
Limitations
The main limitation in this study is the fact that only employees based at the
Cincinnati Division completed the survey used in this paper. Ninety of the two hundred
employees in the Division responded to the survey, providing a more than adequate
sample size. That number, however, is not large enough to gain statistically reliable data
that reflects the entire Bureau population. A second limitation is that the study only
evaluates courses made available through the FBI Virtual Academy. It does not consider
any other Web/Computer-Based training programs.
Assumptions
There are three assumptions made by the researcher in this study. The first
assumption is that all employees, while employed by the FBI, have completed a class, inservice, and/or seminar performed in a traditional classroom setting. The second
6
assumption is that all survey results have been properly collected and entered in the
project spreadsheet. Finally, the researcher assumes that the survey results from the
Cincinnati Division are honest and unbiased.
Acronyms
FBI – Federal Bureau of Investigation
IDC – Instructor Development Course
VA – Virtual Academy
TCLS – Training & Civic Liaison Specialist
VCF – Virtual Case File
W/CBT – Web/Computer-Based Training
7
CHAPTER II
REVIEW OF RELEVANT LITERATURE AND RESEARCH
Introduction
This chapter will review some of the literature relevant to Adult Learning Theory
and the development of W/CBT courses.
The use of W/CBT education in the corporate world is increasing daily. While
there are some who are critical of this type of learning environment, advocates believe
(computer) delivered education can improve learning performance and courses delivered
online can be of similar quality to face-to-face format. (Ricciardi & Sipes, 2006)
Organizations and employees embrace the cost and convenience aspects of online
learning. However, there is no “one-size-fits-all” approach when designing and delivering
course content. Individuals have different learning styles that course designers and
instructors must consider. Computer-based instruction does not guarantee a rich learning
environment. (Caplan, 2004)
The History of Web/Computer-Based Training
W/CBT is a form of distance learning that can trace its roots back to
correspondence education developed in the mid-1800s. A remarkable social innovation at
the time, this method allowed those unable to attend conventional classes to receive
education and training.
Distance education has readily adopted new technologies to reach wider
audiences and teach effectively. Advances in technology throughout the 20th century
added capabilities that allowed teachers to broadcast live presentations to distant learners.
Radio, television and teleconferencing brought instructors and students closer together, in
8
spite of distance and/or political boundaries. W/CBT is the latest form of technology used
to advance distance learning. (Horton, 2000) In the late 1990s, vendors began developing
tools making the Internet easier to use. These tools made it easier to create very effective
and interactive training modules.
The ability to create and implement “anytime, anywhere” training has been
described as the biggest change in the way humans conduct training since the invention
of the chalkboard. The fact remains, however, that W/CBT changed the way we teach
people, but it does not change the way people learn.
The Adult Learner
Instructors of adult learners have unique challenges they must deal with. Sir
Winston Churchill once said, “I am always ready to learn, but I am not always ready to
be taught.” Adult learning theories are based upon the philosophy that adults are
responsible for their own learning and want to be engaged in the learning process.
Andragogy, a term used to describe the principles of adult learning, enables course
developers and instructors to build a more effective learning process for adults. (Knowles
et al., 2005)
The andragogic model asserts that five issues be considered and addressed in
adult learning. They include (1) letting learners know why something is important to
learn, (2) showing learners how to direct themselves through information, and (3) relating
the topic to the learners’ experiences. In addition, (4) adults will not learn until they are
ready and motivated to learn. Often this (5) requires helping them overcome inhibitions,
behaviors, and beliefs about learning. (Conner et al., 1995)
9
While the basic paradigm of andragogy has been around since the 1830s,
extensive work and development in this field did not begin until the 1920s. Eduard C.
Lindeman established the foundation of the theory of adult learning. Lindeman identified
several key assumptions about adult learners. One key assumption is that adults are
motivated to learn as they experience needs and interests that learning will satisfy.
According to Lindeman, adult curriculum should begin with the needs of the learners in
mind. It should start with the lives of the learners and it should look to the learner's own
experience as its most valuable resource. (Knowles, et al, 2005)
What else do we know about how adults learn versus how children learn? Is there
a difference between “conventional” education and adult education? Existing literature
indicates that there are distinct differences. This study will address four specific
characteristics as they pertain to the FBI Virtual Academy and adult learners: Motivation,
Interaction, Content and Transfer of Learning.
Motivation
Adults are motivated to learn as they experience needs and interests that learning
will satisfy. (Knowles, 1988, p. 31) Adult orientation to learning is life-centered.
Experience is the richest source for adult learning. Adults also have a deep need to be
self-directing. (Knowles et al., 2005 p. 40) In the 1950s, Malcolm Knowles expanded on
this work, identified new assumptions and assigned a list of characteristics to adult
learners.
Knowles (2005) developed six core principles: (1) the learner’s need to know, (2)
self-concept of the learner, (3) prior experience of the learner, (4) readiness to learn, (5)
orientation to learning, and (6) motivation to learn.
10
Motivation is the most overlooked aspect of instructional strategy, and perhaps the
most critical element needed for employee-learners. Even the most elegantly designed
training program will fail if the students are not motivated to learn. (Kruse, 2007)
Programs and course modules should be developed to foster the abilities known to result
in success online: self-discipline, motivation, responsibility and organization. (Cavanaugh
& Hughes, 2006)
Dick & Reiser (1989) point out that instructional sequences usually include at
least seven common elements: motivating the learner, explaining what is to be learned,
helping the learner recall previous knowledge, providing instructional material, providing
guidance and feedback, testing comprehension, and providing enrichment or remediation.
The ARCS Model, created by John Keller, suggests that four conditions are
necessary in order for a learner to be motivated. ARCS, which stands for Attention,
Relevance, Confidence and Satisfaction, are the conditions that, when integrated,
motivate someone to learn. (Fernandez, 1999) The first and single most important aspect
of the ARCS model is gaining and keeping the learner’s attention. (Kruse, 2007) Many
instructor and “train-the-trainer” courses emphasize the use of openers or “icebreakers”
to get their students attention. Once the instructor has it, maintain it by providing the
reason for the training: the Relevance. The FBI Instructor Development Course calls this
the “WIIFM” Principle, or, “What’s In It For Me.” Goals and benefits of the training
must be stated clearly and address how this new knowledge will affect their lives. Adults
tend to be more motivated toward learning that helps them solve problems in their lives
or results in internal payoffs. Using this new knowledge or skill set increases Confidence,
which also produces Satisfaction in the learners.
11
Interaction
In 1990, the Accounting Education Change Commission pronounced that adults
should be active participants in the learning process and not passive recipients of
information. (Ricciardi & Sipes, 2006) The term “active learning” has been a buzzword
in the academic community for over twenty years. Many educators agree that in order for
a learner to sort out, make sense of, and integrate new knowledge, he/she must have a
participative role in the learning process. Web browsing or navigating through a series of
pages in a training module does not require a great deal of active involvement. (Hoffman
& Ritchie, 1997, p.4) When developing an online curriculum, Hutton (1999) wondered,
“How would I create a sense of learning community that is so important to adult
learning?” Psychologist Kurt Lewin stated that learning is more effective when it is an
active rather than a passive process. In 1995, Joann Ward stated,
Passive Learning is called indoctrination. When we think critically, we become
active learners. Instructional products must challenge learners to be active
participants in the knowledge construction process, rather than passive recipients
of ‘pre-packaged’ knowledge.
Managers and Human Resource Development professionals have attributed past
failures of Web/Computer-Based training to un-motivated learners or worse. Historically,
these learning programs rarely offer a motivational environment or involve interaction.
(Conner et al., 1995) The concept that human beings are social animals was lost.
Interaction should also extend to the communication necessary when requiring
technical support. Without a live instructor, developers need to make sure that students
are comfortable with the hardware, software and/or any equipment used during the
12
training. This will allow the students to concentrate on learning. (Conner et al., 1995) The
study done by Gitonga (2006) found that having an open climate for communication was
a critical supporting factor for transfer of learning. Other studies specifically point to
Technological Support as being one of the most critical support factors when it comes to
transfer of learning.
Content
To refer back to the ARCS Model, adults prefer courses that focus heavily on the
application of the material to real-life, relevant problems.
Focus on the development of problem-solving and critical-thinking skills requires
that W/CBT programs be designed to accommodate the needs of the learner, giving
him/her the freedom to follow a unique path to learning in his/her own cognitive style.
(Brown, 2000, p. 3) Web-based learning tasks should require students to construct
meaning rather than repeat information they have read or heard. The instructor (course
designer) must assume the role of facilitator or coach and develop activities with
advanced organizers, hyperlinks, and appropriate scaffolding to help students in their
knowledge construction. (Brown, 2000, p. 4)
Clark (1983) felt rather strongly about the content of a course curriculum and its
method of delivery. The “media are mere vehicles that deliver instruction but do not
influence student achievement any more than the truck that delivers our groceries causes
changes in our nutrition…only the content of the vehicle can influence achievement.”
Additionally, no one wants to be bored while learning. If we do not care about a
topic, we are less likely to stick with it and continue to learn. If the content does not
address an adult’s need for motivation and relevance, they may get easily distracted.
13
(Conner et al., 1995) Designing relevant, interactive lessons is again, a challenge that
must be met by W/CBT developers. Statistics show that human retention rates during
lectures or reading is as little as 5-10%. Adding audiovisuals, demonstrations and practice
by doing can increase that rate up to 75%. When people apply that knowledge and teach
others, retention soars to over 95%. The Chinese saying holds true in adult learning: “Tell
me, I forget. Show me, I may remember. Involve me, I understand.”
Transfer of Learning
Transfer of learning occurs in many areas of our lives. It manifests itself not only
in an academic environment, but also at home, in the workplace, and even in our
communities. Transfer takes place whenever our existing knowledge, abilities and skills
affect the learning or performance of new tasks. (Taylor, 1997, p.5) Mosel (1957)
recognized three conditions that must exist for effective learning transfer: the content
must be applicable to the job, the student must learn the content, and the student must be
motivated to change their job behavior in order to apply what was learned. Factors also
exist which can hamper effective transfer: lack of reinforcement from supervisors,
environmental interference, and lack of organizational support.
We can evaluate the effectiveness of the transfer by measuring not only test
results, but also observing workplace behavior and the impact the training had on the
organization. Donald Kirkpatrick put forth his four-level model of evaluation in the
1970s. The last two levels are directed at behavior in the workplace and business results.
When evaluating their online training systems, two questions should be considered by
management: Have employees changed their workplace behavior because of the training,
and are their positive measurable changes in the way the organization conducts business?
14
Research Hypotheses
Some organizations implementing technology-based learning have discovered
that the learners lack the skills, motivation or confidence to engage in the required level
of self-directed learning. That may be the case at the FBI. This study investigates the
correlation between classroom courses and Web/Computer-Based Training and each of
their influence on effective transfer of learning. Based on the experience the researcher
has with the courseware in the FBI Virtual Academy, employee surveys and the
supporting literature, the study explores the following:
Hypothesis One
The instructional implementation of FBI Virtual Academy courses does not meet
the needs of FBI/law enforcement personnel.
Hypothesis Two
In a learning situation, FBI/law enforcement personnel prefer a traditional
classroom environment rather than an online course.
Null Hypotheses
Null Hypothesis One
The instructional implementation characteristics are adequate and meet the needs
of FBI/law enforcement personnel.
Null Hypothesis Two
In a learning situation, FBI/law enforcement personnel prefer online courses
rather than a traditional classroom setting.
15
CHAPTER III
RESEARCH METHODOLOGY
Research Model
A correlational model was used in this project. Its design took quantitative and
qualitative data and explored relationships connecting the FBI Virtual Academy,
classroom training and employee preferences of each setting.
Survey Population
The population for this survey consisted of a random selection of 100 employees
based in the Cincinnati Division of the FBI. Employees are classified as Special Agents
or Professional Support. One hundred surveys were mailed to these employees in March
2007. Ninety responses were received between 23 March and 4 May 2007. The
Cincinnati Division has Resident Agencies (RA’s) in several cities around southern Ohio.
Employees at these RA’s were included in the survey. The following is a breakdown of
survey distribution: Two (2) surveys were sent to each RA in Athens, Portsmouth, and
Cambridge. Seventeen (17) were sent to Dayton, Twenty-eight (28) to Columbus and
Forty-nine (49) to Cincinnati.
Additional demographical information is as follows: 52% of the respondents are
Special Agents, 48% Professional Support. 57% have been working for the FBI over ten
years. During business hours, 54% spend 3-6 hours and 31% spend 6-8 hours on the
computer every day. Nearly 75% of those surveyed have taken more than five online
courses through the FBI Virtual Academy, and 60% have taken an online course outside
of the Virtual Academy.
16
Sources of Data
The main source of data was collected using a 23-question survey. To introduce
and explain the survey’s purpose, a brief explanatory letter accompanied the survey. (See
Appendix B) Additional qualitative data was received through informal interviews with
other FBI Training & Civic Liaison Specialists, Training Coordinators, and FBI
Academy Instructors. These employees are at the center of Virtual Academy activity and
provided qualitative data, giving a more comprehensive picture to the research.
The Data Collection Device
The data collection device for this paper was a custom spreadsheet constructed by
the researcher in Microsoft Excel. The spreadsheet was comprised of four sections.
The first section recorded the demographics and corresponded with the first seven
questions of the survey. A data matrix was used to collect basic information on the
Virtual Academy users: agent vs. support, years at the FBI, time spent on computer, and
information regarding online training, including Virtual Academy. Each question used a
nominal scale to classify the respondents into the appropriate groups.
The second section also used a data matrix to record the answers to survey
question #8 through #19. A five point Likert scale was used in the survey to record the
levels of employee agreement with the question or statement. A number was assigned to
each point on the scale, and entered into the spreadsheet.
The worksheet cells in the third section contained formulae that performed basic
descriptive statistical analyses of the data in questions #8 through #19.
The final section contained graphs based on the specific questions used in this
study.
17
Distribution Method
Each survey was addressed and mailed to the individual employees in the
Cincinnati Division. The employees mailed their responses back in the enclosed return
envelope.
Procedures
To collect data regarding employee use of Virtual Academy, the researcher
developed a questionnaire to determine FBI employee views concerning the Virtual
Academy, its content delivery, subject matter and transfer of learning. The target of the
survey was the employees of the FBI Cincinnati Field Office. The survey only addressed
the Cincinnati Division. The intention was to use the survey results and the qualitative
data received from other FBI employees to show similar relationships around the Bureau.
The Virtual Academy is the same in all FBI offices and all employees are required to take
the same courses.
As the surveys were returned, the data was entered into the project spreadsheet.
Statistical information was automatically determined using the appropriate formulae.
After all of the project spreadsheet was complete, a statistical summary worksheet
was developed. Charts were then created to give a graphical representation of the results.
Treatment of Data
A standard statistical analysis was performed using the results obtained from the
survey. A Chi-square (χ2) test was used to count the number of responses that fell into the
particular predetermined categories. This non-parametric analysis was used because the
intention of this project was to ascertain and examine any relationships between Virtual
18
Academy courses, traditional classroom-based courses and their influence on effective
learning transfer. The relationships were then used to verify the researcher’s hypotheses.
19
CHAPTER IV
RESULTS
After reviewing the survey results, it was determined that in addition to the
demographical questions (#1-7), only questions #8-11 and questions #17-19 provided
adequate information necessary to test the researcher’s hypotheses. The chi-square tests
were used for questions #8-11 only.
The Likert scale used in the survey measured five response levels: strongly
disagree, disagree, undecided, agree and strongly agree. A response of “1” indicated
strongly disagree while a response of “5’ indicated strongly agree.
For the Chi-square tests, responses recorded as “strongly agree” and “agree” were
grouped together, as were the responses of “disagree” and “strongly disagree.”
Table 1
Virtual Academy/Classroom-Based Courses vs. Job Confidence
Agree Disagree Undecided Total
Question #8 – Virtual Academy courses have
increased my confidence in my job performance
Question #9 – Classroom based courses have
increased my confidence in my job performance
Total
31
28
31
90
68
7
15
90
99
35
46
180
Table 2
Employee Preference of Virtual Academy or Classroom-Based Courses
Agree Disagree Undecided Total
Question #10 – I prefer a quality online course
rather than a classroom environment
Question #11 – Web or Computer-Based
Training is not as effective as a classroom
Total
17
50
23
90
53
24
13
90
70
74
36
180
20
The mean (X̄ ), mode and median for question #8 were 3 with a standard deviation
(s) of 0.95. The standard error of the mean (σX̄ ) was 0.1.
The mean (X̄), mode and median for question #9 were 4 with a standard deviation
(s) of 0.90. The standard error of the mean (σX̄ ) was 0.09.
The mean (X̄ ), mode and median for question #10 were 2 with a standard
deviation (s) of 1.1. The standard error of the mean (σX̄ ) was 0.12.
The mean (X̄ ), mode and median for question #11 were 4 with a standard
deviation (s) of 1.22. The standard error of the mean (σX̄ ) was 0.13.
Figure 1
VirtualVirtual
Academ
y Courses
Increase
Job Confidence
Academy
Courses Increase
Job Confidence
Strongly
Agree
1%
Strongly
Disagree
7%
Agree
33%
Disagree
24%
Undecided
35%
21
Figure 2
Classroom Courses
Increase
Job Confidence
Classroom
Courses
Increase
Job Confidence
Strongly
Disagree
1%
Strongly
Agree
28%
Disagree
7%
Undecided
17%
Agree
47%
Preference for Online Course Over Classroom
Figure 3
Environm ent
Employees Prefer Online Course over Classroom-Based Environment
Agree
14%
Undecided
26%
Strongly
Agree
4%
Strongly
Disagree
20%
Disagree
36%
22
Figure 4
Classroom More Effective Than W/CBT
Classroom-Based Teaching Is More Effective Than Web/Computer-Based Training
Strongly
Disagree
4%
Strongly
Agree
26%
Disagree
22%
Undecided
14%
Agree
34%
Figure 5
Virtual Academ y Instruction Is Effective
Virtual Academy Courses Are Effective
Strongly
Agree
3%
Strongly
Disagree
3%
Disagree
13%
Agree
43%
Undecided
38%
Interaction Is Im portant (Classroom or W/CBT)
Figure 6
Interaction is Important in a Learning Environment
Strongly
Agree
30%
Disagree
14%
Undecided
6%
Agree
50%
23
24
CHAPTER V
DISCUSSION
Based on the results of the survey, the data produced indicates that FBI employees
in the Cincinnati Division prefer learning in a classroom-based environment rather than
online using the Virtual Academy.
The results summarize three aspects: confidence, effectiveness and interaction.
When asked which method of instruction increases confidence in job
performance, 75% of employees agreed that a classroom environment instills more
confidence as opposed to 34% who felt online courses have the same effect. Over onethird of the employees surveyed were undecided regarding the Virtual Academy courses.
Sixty percent of employees believe learning in a classroom is more effective than
online learning which was only preferred by 18% of employees.
Finally, the data indicated that employees feel very strongly about interaction.
Eighty percent believe that interaction is important in a learning environment.
Hypotheses
Hypothesis One
The instructional implementation of FBI Virtual Academy courses does not meet
the needs of FBI/law enforcement personnel.
This hypothesis should have been more specific in its definition of what “needs”
FBI and law enforcement personnel have in regards to job training. I do not feel it is out
of context or beyond the scope of this paper to suggest that all employees who must
participate in on-the-job training want the same features: effective, interesting and
applicable.
25
The responses from the survey show some, but not overwhelming support for
Virtual Academy. However, there is also a large “undecided” population that must be
included in this discussion. Virtual Academy has been functioning for five years.
Nevertheless, in the last couple of years, there have been an increased number of required
courses and upgrades to FBI computers. New courses have also been developed and
placed in the database. Employees have only recently begun to use Virtual Academy
more frequently. The “undecided” factor could be attributed to the fact that employees
have not used the system long enough, or completed enough coursework to see any
effect, positive or negative.
Despite the large number of undecided responses, the percentages and the
relationships obtained in the Chi-square test are enough to reject the null hypothesis and
support the premise that the Virtual Academy does not meet the needs of FBI/law
enforcement personnel.
Hypothesis Two
In a learning situation, FBI/law enforcement personnel prefer a traditional
classroom environment rather than an online course.
Fifty-six percent of those surveyed prefer the student-teacher face-to-face
setting when learning as opposed to the 18% who preferred the online experience. These
figures completely support the statement and are enough to reject the null hypothesis.
26
The study mentioned by Horton (2000) showed that workers strongly prefer
informal on-the-job training and self-paced training to formal classroom training.
In spite of research that indicates otherwise, why are FBI employees less likely to
prefer an online learning environment?
Virtual Case File
The aversion to the online training environment that FBI employees seem to
possess may point to a “corporate culture” issue that developed because of the FBI’s
failure to implement another type of computer program: The Virtual Case File (VCF).
The concept behind the VCF was a system designed to automate the way the FBI
handled their paper-based work environment. It was also an upgrade to the Automated
Case System (ACS), an obsolete case management system. Agents would introduce and
process forms into the database electronically, allowing agents and intelligence analysts
the ability to share vital investigative information. Rather than purchasing existing offthe-shelf software to build the system, the FBI was determined to write their own code
and invent their own program. Organizational behaviorists would identify this counterproductive attitude as the “not invented here” syndrome. Some corporations fall prey to
this mindset by refusing to use other products, ideas or procedures simply because that
company did not develop it.
As the FBI geared up for VCF implementation, the new concept was introduced
to the Bureau and employees were mandated to take blocks of online instruction that
prepared them for the new system. This training occurred in the midst of detrimental
factors that doomed VCF from the very beginning. Poor design, bug-ridden lines of code,
27
outdated hardware and a rush to completion were among the many reasons VCF failed.
After nearly $600 million in total costs, VCF was a bust.
Bureau employees were obviously not pleased with the results and morale took a
huge blow. Employees who were already reluctant to enter the information age were now
even more convinced that the “old fashioned way” was better and that an electronic
version of the existing file management system would be a total failure.
About the same time that VCF was experiencing its slow death, Virtual Academy
was implemented. Here was another program put into service by the FBI and touted as
the new way employees would receive training. The opposition ranged anywhere from
outright anger and frustration to total disregard. Trusting the FBI to create anything
“new” using its slow network and outdated computers, was not something employees
were ready or willing to do.
28
CHAPTER VI
CONCLUSIONS
Distance education has become a powerful and widespread instrument for
teaching adults. Competitive corporations have utilized this concept by purchasing and/or
developing online training courses for their employees. This method of “anywhereanytime” education allows companies to train their workers faster, cheaper and more
effectively than sending them away from the workplace. In addition, learning theory
research shows that adults prefer a self-directed learning environment once they are
introduced to it.
The FBI has implemented its own platform for online training. However, based on
this study, it appears that FBI employees fail to see any great significance in Virtual
Academy over a classroom based environment.
As discussed in the previous chapter, the researcher believes the VCF debacle had
a major impact on the way FBI employees feel about the Virtual Academy. There was
additional information collected in this study that provided evidence of other factors why
the Virtual Academy has not been embraced. In the course of the normal work setting,
the researcher noted comments made during informal conversations and interviews and
recorded the written comments made by several survey respondents.
Office Distractions
“Need a quiet room with a door to take Virtual Academy courses; too distracted at
desk.”
“…constant disruptions and interruptions from regular work make it impossible to
take VA serious.”
29
These are comments written on the surveys by the respondents. Those who work
in a busy office know that quiet and privacy are not features easily obtained. Open office
space or cubicles with co-workers surrounding you, ringing phones and overhead
announcements are some of the constant distractions an employee has to deal with. We
normally deal with these but it makes a horrible environment for a person trying to focus
on a computer-based training module.
Change of Environment
Simply stated, people enjoy getting out of the office. Most of the training offered
by the FBI occurs at the FBI Academy in Quantico, Virginia. There are training courses,
however, that are conducted at division offices and other locations around the United
States. The FBI also takes advantage of training companies such as SkillPath and Fred
Pryor Seminars. These companies typically provide one or two day training events at
hotels or conference centers. The demand for this type of training indicates that
employees enjoy learning in a venue other than the office.
Confidence in Virtual Academy Courses
Some FBI employees are firm believers in the time-tested method of a face-toface, interactive classroom environment. Eighty percent of the employees surveyed for
this study consider interaction extremely important in a learning environment. Another
comment written on a survey explained that most agents feel that Virtual Academy is an
easy “check the box” training solution. This attitude reflects some disregard for the
content of Virtual Academy courseware and its effectiveness.
30
Teaching Methods Contrary to FBI Instruction Philosophy
The FBI Instructor Development Course (IDC) places great emphasis on the
principles of Adult Learning and on Interaction. One of the guidelines set forth by FBI
Instructors is to create a learning environment that is active – not passive. To do this, the
instructor should use learning strategies that actively engage the students. This sounds
easy to do in a face-to-face setting since classroom conversation can certainly be
interactive. While the process may take more creativity and development, actively
engaging participants in an online course is very achievable.
Time
One of the reasons many corporations embrace online training for their employees
is to save time. This is especially true in large companies. When a large number of your
employees must receive training, it is impossible to have them away from work at the
same time. However, even without the time spent traveling and the time away from the
office, employees still find it difficult to take an online course. “VA is just another time
consuming periodic distraction,” wrote one employee.
Timing of Survey
At the time the survey for this study was distributed, several mandatory courses
had to be completed by all FBI employees. Many employees felt that the subject matter
of these courses had little, if anything to do with their particular jobs and therefore had no
value to them. There was no application of knowledge or skills the employees could
transfer to their position. The hard feelings that employees had at the time may have also
had an effect on their answers.
31
CHAPTER VII
RECOMMENDATIONS
In a recent poll of senior human resource professionals, it was found that almost
two-thirds – 62 percent – believe online learning will form the single largest part of their
organizations’ future training delivery. Employees agree, as 81 percent also believe they
will be doing the majority of their learning online in the future. (Silberstein, 2007)
Training trends in the corporate sector are obviously heading in this direction.
Virtual Academy and its resources have been available for several years, yet the
significant potential of its learning environment has yet to be realized. Therefore, what
steps can the FBI take in order to help employees take advantage of the training
opportunities Virtual Academy has to offer?
Improve Content and Increase Interaction
Improving the course content is the first and most important step that the Training
Development Unit should take. All instructors have basic responsibilities, which should
not change. The most fundamental role of an instructor is to design, develop and
implement a training event that causes another person to learn. Developing a
Web/Computer-Based learning environment is a large endeavor requiring many different
skills, and usually, many different people. That challenge, along with organizations
looking for an easy way to fulfill certain training requirements, can inhibit effective
learning transfer. Neglecting the basic tenets of Adult Learning Theory will ultimately
waste time and money. The waste that employees become aware of can create low morale
and a disregard for any attempt at career enhancement possibilities.
32
One such way to improve the content of Virtual Academy courses is to utilize an
Instructional Design Methodology similar to the one used by Ninth House, Inc. The
blended learning approach incorporates focused, interactive elements, simulations and
storytelling in their training modules. These methods have been proven to increase
retention and help employees understand the importance of properly applying the learned
skills. A Department of Justice survey found that 71% of Ninth House learners found the
courses to be equal or superior to other computer-based, CD-ROM, or Web-based
courses. In addition, 64% of those surveyed felt that the courses helped to improve the
ability to do their jobs. (www.ninthhouse.com) These figures contrast greatly with those
received in this study’s Virtual Academy survey.
Public Relations
Placing a positive image on the use of Virtual Academy is another important plan
that the Training Development Unit should carry out. The campaign should be substantial
– reiterating how Virtual Academy works, advertising the courses offered, and
emphasizing the importance of continuing education. At the Field Office level,
supervisors must be highly involved in this – convincing employees that self-directed
learning can be extremely beneficial really starts at the management level.
Increase Incentives
FBI employees are required by mandate to complete at least fifteen hours of
Continuing Education every year. This training can take place in a variety of ways,
including courses completed through Virtual Academy. A stronger link between online
courses completed and the FBI Performance Appraisal System may encourage more
33
participation. A possible improvement in this area could include a step and/or merit
increase for the employee, if a certain number of courses are completed.
Increase Management Support
Supervisors hold the most significant keys to resolving the problem of transfer of
training. They hold the primary responsibility for the most cited barrier – absence of
reinforcement on the job for the newly acquired skills and abilities. (Taylor, 1997)
Encourage Discussion (Mandatory Courses)
One of the established principles of learning is the principle of intensity:
Increasing the intensity of the material increases the likelihood of retention. The
researcher is not suggesting that the Bureau take a radical “activist” or overdramatic
approach, but the approach should be thought provoking. The FBI’s mission is “to protect
and defend the United States against terrorist and foreign intelligence threats and to
enforce the criminal laws of the United States." (www.fbi.gov) This is an extremely
important and serious mission, and if you get people thinking about the seriousness of a
topic, they will be more inclined to act accordingly. Employees should be encouraged to
want to learn more. Encourage topics and questions to be a part of discussions in the
office. Supervisors should take opportunities during meetings or training events to
discuss the important topics addressed in the mandatory courses.
Areas for Further Research
Additional questioning of employees could benefit the FBI and Virtual Academy.
The Training and Development Unit is continuing to push many aspects of the training
process into Virtual Academy: registration, supervisor approval, computer-based courses,
etc. This implementation without evaluation is intrusive to some and bothersome to
34
many. Finding out more about what Bureau employees need and how they want to learn
may help Virtual Academy realize its full potential.
Management and those who decide on course subject matter must use experienced
instructors when working on the design implementation of a program. During the
planning phase, effective learning transfer must be considered as part of the process.
This brings us to the ultimate question that needs to be answered in the scope of
this study: Just how effective is the training offered by Virtual Academy courseware?
Further research should be conducted in this particular area to see if there is empirical
evidence that effective transfer of learning is occurring. Notwithstanding the attitudes of
FBI employees towards VA, transfer of learning may be occurring, but exactly how
effective and permanent is it? Have the abilities, skills and attitudes of FBI employees
changed because of Virtual Academy training? The FBI needs to be more concerned
about how effective employee training is and less concerned about “checking a box” or
satisfying statistical requirements.
35
REFERENCES
Accounting Education Change Commission (AECC). (1989) The Accounting Education
Change Commission: Its History and Impact. Sarasota, FL: American Accounting
Association
Brown, B. (2000). Web-Based Training. Columbus, OH: ERIC Clearinghouse on Adult
Career and Vocational Education. (ERIC Document Reproduction Service No. ED
445 234)
Browning, J., Gambill, L., & Mory, E. (1998). Instruction on the Web: The Online
Student's Perspective. In SITE 98: Society For Information Technology & Teacher
Education International Conference, edited by S. McNeil et al. Charlottesville, VA:
Association for the Advancement of Computing in Education.
Clark, R. (1983). Reconsidering Research On Learning From Media. Review of
Educational Research. Volume 53, No. 4, pp. 445-459. Washington D.C.: American
Educational Research Association
Conner, M.L., Curry, K., DeVries, L., Forman, D., Wilmsmeyer, D., Wright, E., &
Zeider, C. (1995). Learning: The Critical Technology (6th ed.). St. Louis, MO: Wave
Technologies International
Federal Bureau of Investigaion. (2006). Instructor Development Program - Student
Learning Guide. Quantico, VA: Regional Training and Development Unit
Fernández, J. (1999). Attribution Theory and Keller’s ARCS Model of Motivation.
Retrieved March 10, 2007 from George Mason University Web site:
http://chd.gmu.edu/immersion/knowledgebase/strategies/cognitivism/keller_ARCS
36
Gitonga, J. (2006). Work Environment Factors Influencing the Transfer of Learning for
Online Learners. Champaign, IL: University of Illinois at Urbana. (ERIC Document
Reproduction Service No. ED 492 788)
Goldstein, H. (2005). Who Killed the Virtual Case File? IEEE Spectrum Online.
Retrieved July 13, 2007, from http://www.spectrum.ieee.org/sep05/1455
Hoffman, B., & Ritchie, D. (1997). Using Instructional Design Principles To Amplify
Learning on the World Wide Web. San Diego, CA: San Diego State University.
(ERIC Document Reproduction Service No. ED 415 835)
Horton, W. (2000). Designing Web-Based Training. New York, NY: John Wiley & Sons.
Jain, R. & Trandis, H. (1997). Management of Research and Development Organizations.
New York, NY: John Wiley & Sons.
Khan, B. (2005). E-Learning Quick Checklist. Hershey, PA: Idea Group Inc.
Knowles, M.S., Holton, E.F. III, & Swanson, R.A. (2005). The Adult Learner (6th ed.).
Burlington, MA: Elsevier.
Kruse, K. (2007). The Magic of Learner Motivation: The ARCS Model. Retrieved March
10, 2007 from http://www.e-learningguru.com/articles/art3_5.htm
Mosel, J. (1957). Why Training Programs Fail To Carry Over. Personnel. pp. 56-64
Office of the Director of National Intelligence (ODNI). (2006). The U.S. Intelligence
Community’s Five-Year Strategic Human Capital Plan. Washington, D.C.
Ricciardi, V., & Sipes, K. (2006). Online vs. Face to Face: Is There a Difference in How
Accounting and Finance Students Learn in an Online vs. Face-to-Face Setting?
Frankfort, KY: Kentucky State University
37
Shea, P. (2004). Plenary address on interaction. Sloan-C Online Research Workshop.
March, 2004; and presentation at the Sloan-C Workshop on Blended Learning,
University of Chicago, April 2005
Silberman, M. (1998). Active Training. San Francisco, CA: Jossey-Bass
Silberstein, N. (2007). On-the-Job Training Goes Online. Online Degrees: Your Guide to
Getting Ahead With e-Learning, Fall/Winter 2007, 30-32.
Swan, K. (2004). Relationships Between Interactions and Learning in Online
Environments.
Taylor, M. (1997). Transfer of learning: Planning Effective Workplace Education
Programs. Ottawa, Ontario, Canada: National Literacy Secretariat, Human Resources
Development Canada. (ERIC Document Reproduction Service No. ED 414 462)
38
APPENDIX A
BIBLIOGRAPHY
American Psychological Association. (2001). Publication manual of the American
Psychological Association (5th ed.). Washington, DC: Author.
Leedy, P. & Ormrod, J. (2005). Practical Research – Planning and Design (8th ed.).
Upper Saddle River, NJ: Pearson Prentice Hall.
Rosado, A, Dammier, E., Clark, R., & Rosenhammer, F. (Eds.). (2001). Guide to the
Graduate Research Project (5th ed.). Daytona Beach, FL: Embry- Riddle
Aeronautical University, Extended Campus.
TD Bank Financial Group. (2003). Measurement Standards – Training Evaluation and
Effectiveness Reporting. Toronto, Canada: TD Bank
39
APPENDIX B
DATA COLLECTION DEVICE
VIRTUAL ACADEMY SURVEY
My name is Tony Gonzalez. I am the Training Technician for the Cincinnati
Division.
I am working towards fulfilling my requirements for a Master of Aeronautical
Science degree from Embry-Riddle Aeronautical University. Within the degree, I have
focused a good portion of my study concentrating in the areas of Human Factors and
Education Technology.
One of my degree requirements is the completion of a Graduate Research Project.
This survey is the research instrument used to gather data for this project. Your assistance
in completing this survey will provide invaluable, anonymous data pertinent to this
research topic. My topic involves educational resources employers use to train
employees. In our case, one such resource is the FBI Virtual Academy.
This survey is for data collection purposes only. It is completely confidential and
strictly voluntary.
I fully understand the workload you all carry. If you can, please take a few
minutes to complete the survey as honestly and as candidly as you can and return the
sheet to me in the enclosed self-addressed, stamped envelope. I would like to receive as
many responses as I can by Wednesday, April 4th.
Thank you for your time and help.
Tony Gonzalez
40
VIRTUAL ACADEMY SURVEY
-----------------------------------------------------------------------------------------------------------For items 1 through 7, please CIRCLE THE APPROPRIATE ANSWER
-----------------------------------------------------------------------------------------------------------1.
Are you a Special Agent or Professional Support?
AGENT
2.
During an average workday, about how much time do you spend working on the
computer?
a.
b.
c.
d.
3.
SUPPORT
Less than 1 hour
1 – 3 hours
3 – 6 hours
6 – 8 hours
How long have you worked for the FBI?
a.
b.
c.
d.
Less than 3 years
3 – 5 years
5 – 10 years
Over 10 years
4.
How many Virtual Academy courses have you completed online?
a. 1 – 3
b. 3 – 5
c. More than 5
5.
Did any of the courses allow you to submit a post-course evaluation?
a. Often
b. Sometimes
c. Seldom
d. Never
41
6.
Did the Virtual Academy course(s) you took contain any quizzes or tests?
YES
7.
NO
Have you ever taken a Web-based or Computer-based training course other than
Virtual Academy?
YES
NO
For statements 8 through 23, CIRCLE A NUMBER from 1 to 5 that BEST
DESCRIBES your opinion or experience. Use the following scale:
Strongly Disagree
Disagree
Undecided
Agree
Strongly Agree
1
2
3
4
5
8.
Virtual Academy courses have increased my confidence in my job performance.
1
9.
2
3
4
5
Classroom based courses have increased my confidence in my job performance.
1
2
3
4
5
10. I prefer a quality online course rather than a classroom environment.
1
2
3
4
5
11. Web or Computer-based training is not as effective as a face-to-face classroom
environment.
1
2
3
4
5
42
Strongly Disagree
Disagree
Undecided
Agree
Strongly Agree
1
2
3
4
5
12. Spending time on the computer to complete required training is beneficial.
1
2
3
4
5
13. Being able to complete required training courses at work is convenient.
1
2
3
4
5
14. I feel confident navigating around Virtual Academy.
1
2
3
4
5
4
5
15. The Virtual Academy program is user-friendly.
1
2
3
16. Virtual Academy provides an adequate level of technical support.
1
2
3
4
5
17. The methods of instruction used in Virtual Academy are effective.
1
2
3
4
5
18. The content and delivery of Virtual Academy courses help motivate learners.
1
2
3
4
5
43
Strongly Disagree
Disagree
Undecided
Agree
Strongly Agree
1
2
3
4
5
19. Interaction is an important characteristic to me when I am in a training
environment (online or classroom).
1
2
3
4
5
The FBI requires employees to complete several Computer-based training courses every
year. With regard to the required training courses only:
20. The training received was important for my job.
1
2
3
4
5
4
5
3
4
5
3
4
5
21. The training received was relevant to my job.
1
2
3
22. The content was clear and organized.
1
2
23. The content was accurate.
1
2