Received: 6 November 2020
|
Accepted: 14 January 2021
DOI: 10.1111/bjet.13070
ORIGINAL MANUSCRIPT
Using a virtual lab network testbed to facilitate
real-­world hands-­on learning in a networking
course
Andy Luse1
| Julie Rursch2
1
Department of Management Science and
Information Systems, Oklahoma State
University, Stillwater, OK, USA
2
Department of Electrical and Computer
Engineering, Iowa State University, Ames,
IA, USA
Correspondence
Andy Luse, Department of Management
Science and Information Systems,
Oklahoma State University, 414 Business
Building, Stillwater, OK 74032, USA.
Email: andyluse@okstate.edu
Abstract
Engineering and technology educators continually
strive for realistic, hands-­
on laboratory exercises
to enhance their students’ learning. This research
describes the redesign of an undergraduate introductory computer networking course to include new
weekly virtual laboratory assignments that culminate
in a ‘real world’ final project of configuring a ‘corporate’ network. The use of an Internet testbed technology named ISEAGE allows students to design and
implement fully functional networks using public IP
space that is contained in the testbed. To the students, it appears as if they were directly connected
to the Internet while still being protected. This paper
shows that ‘real world’ projects using virtual lab technology can have a positive effect both on objective
networking knowledge, as well as subjective self-­
assessments of self-­
efficacy with regard to implementing the technology. It also demonstrates that
‘real world’ final projects encourage student thinking
at upper levels of Bloom's taxonomy.
KEYWORDS
computer networks, engineering education, virtual laboratories
© 2021 British Educational Research Association
1244
|

wileyonlinelibrary.com/journal/bjet
Br J Educ Technol. 2021;52:1244–1261.
USING A VIRTUAL LAB NETWORK TESTBED
|
1245
Practitioner notes
What is already known about this topic
• Computer and engineering courses require hands-­on labs.
• Due to online enrollment and the current COVID pandemic, these hands-­on labs
need to be taught virtually.
What this paper adds
• We detail a virtual lab environment for teaching hands-­
on skills in computer
networking.
• We longitudinally assess the virtual lab technology across multiple institutions.
Implications for practice and/or policy
• Other practitioners can implement this same technology for teaching hands-­on
networking concepts virtually.
INTRODUCTION
Weekly labs are used by departments across the United States as a way to build competencies, as well as enhance student cognition. Faculty members and curriculum designers
build each lab or homework problem focusing on the most basic concepts of the course. By
the nature of classroom schedules, online learning, and current issues with COVID, many of
these labs are utilising online virtual labs (Alioon & Delialioğlu, 2019) and other technological
innovations to assist in learning virtually (Altmeyer et al., 2020). Even with traditional classroom discussions and lectures to tie the pieces together, even the brightest students sometimes have problems connecting the dots between ideas and an even harder time using all
the separate weekly pieces to solve a large-­scale, real-­world problem. Add to this the added
complexity of enabling online instruction in virtual labs, and the ability to teach concepts is
even more exacerbated. The execution of moving students from learning singular concepts
and mastering singular tasks to full comprehension and the ability to create new ideas has
become even tougher in these virtual lab environments.
Faculty in engineering and technology departments across the United States subscribe
to the concepts espoused in Bloom's taxonomy (Anderson & Sosniak, 1994; Bloom et al.,
1956). This framework is used to shape curriculum, and provide educational activities that
encourage student upper-­level cognitive activity (Crompton et al., 2019). The objective is to
provide homework problems, classroom conversations and laboratory exercises that motivate the students to think beyond simple answers and solutions. The ultimate goal is to have
students take newly acquired concepts from course materials, synthesise them and create
their own unique ideas.
This research aims to answer the research question of whether ‘real world’ projects using
a novel virtual Internet lab technology testbed can have a positive effect on learning. The
paper shows the authors’ approach to encouraging deeper student cognition by creating
new, meaningful, weekly virtual labs that culminated in a ‘real world’ final project. The labs
were developed for an introductory networking class using the Internet-­Scale Event and
Attack Generation Environment (ISEAGE) testbed. These weekly labs allowed the student
to both increase their knowledge of networking concepts as well as develop self-­efficacy
(Bandura, 1986) with accomplishing networking tasks. The capstone final project allowed
students to use the skills they mastered during the weekly virtual lab exercises to design,
implement, configure and run their own networks in a ‘real world’ setting created within
the virtual lab testbed. To evaluate the success of the newly created virtual labs and final
1246
|
LUSE and RURSCH
project, student learning was measured using objective, subjective and qualitative methods.
The results of these measurements are reported in this paper.
THEORETICAL BACKGROUND
When undertaking course development or course redesign, Bloom's taxonomy is often a
framework that educators use to build their content. Bloom's taxonomy of educational learning objectives was originally developed in the early 1950s (Bloom et al., 1956) and updated
in the current century by Anderson and Krathwohl (Anderson & Krathwohl, 2001). Each
layer of the tiered framework represents the level of cognitive complexity of the tasks that
are asked of students. As progression is made up the levels of the model, it is expected that
students have mastered the lower thinking skills and are ready to undertake new challenges.
Table 1 shows the revised Bloom's taxonomy.
When educators redesign curriculum, the goal is to create experiences where students
move through the tiered learning objectives in a logical order. Students’ assignments are
created to allow students to master a lower level cognition skill before upper order thinking
skills are required. For example, freshmen-­level courses spend a large portion of the semester at lower level thinking skills of remembering, understanding, and, hopefully, applying to
familiarise the students with the concepts or basic building blocks. Upper division courses
for juniors and seniors can begin in the applying level of Bloom's taxonomy since they already have the working understanding of the building blocks and quickly move to the analysing and evaluating stages of a problem or content area. In the engineering discipline, the
senior design project or capstone project is the culminating project of their undergraduate
career and forces students into the creating cognitive skills level. Therefore, we hypothesise,
Hypothesis 1 Weekly skills-­mastery labs utilising the novel ISEAGE Internet testbed will
allow students to present cognitions at the upper (creating) level of Bloom's taxonomy.
Generally, labs are created at the third level of Bloom's taxonomy (Applying). At the end of
a lab, students should have executed a series of technical steps and achieved an end goal.
Well-­designed labs build self-­efficacy through the use of enactive mastery by personally
performing the task. Self-­efficacy measures a person's confidence that he or she can complete an activity in the future which, in turn, has been shown to lead to future success with
the task. Successful task completion relies not only on the individual's actual skills, but also
on his or her belief in himself or herself to perform the tasks (Bandura, 1986) or activities
(Lent, 2005). Whereas there are four primary ways individuals gain self-­efficacy (physiological states, verbal persuasion, vicarious experience, enactive mastery), it has been shown
TA B L E 1
Bloom's revised taxonomy (Anderson & Krathwohl, 2001)
Title words
Supporting words
Creating
Designing, constructing, planning, producing, inventing, devising, making
Evaluating
Checking, hypothesising, critiquing, experimenting, judging, testing, detecting,
monitoring
Analysing
Comparing, organising, deconstructing, attributing, outlining, finding, structuring,
integrating
Applying
Implementing, carrying out, using, executing
Understanding
Interpreting, summarising, inferring, comparing, explaining, exemplifying
Remembering
Recognising, listening, describing, identifying, retrieving, naming, locating, finding
USING A VIRTUAL LAB NETWORK TESTBED
|
1247
that enactive mastery influences the individual's self-­efficacy the most (Bandura, 1997). A
carefully constructed lab which culminates in a successful hands-­on interaction with the task
is one of the most common ways self-­efficacy is built (Scheibe et al., 2007). Additionally,
successfully completing the task numerous times increases the individual's self-­efficacy.
Compeau and Higgins (1995) developed one of the first methods for measuring self-­
efficacy within information systems. This work focused on creating a measure of computer
self-­efficacy that was thought to carry over into all areas of information systems. However,
later research demonstrated that a general computer self-­efficacy measurement did not indicate future success in specific task areas. Constructs for measuring self-­efficacy are now
developed within each specific domain of interest (Agarwal et al., 2000; Johnson & Marakas,
2000; Marakas et al., 2007). Since the authors’ work was in a networking course, self-­
efficacy questions focus on network concepts and skills gained from the weekly labs, as well
as the ‘corporate’ network final project as previous work shows necessary (Davazdahemami
et al., 2018; Luse et al., 2014). Given the above, we hypothesise,
Hypothesis 2 Weekly skills-­mastery labs utilising the novel ISEAGE Internet testbed will
positively increase student self-­efficacy with regard to networking and infrastructure.
Self-­efficacy provides self-­beliefs about personal ability that is important for successful
task completion (Bandura, 1986). An increase in self-­efficacy can lead an individual to improve his or her actual performance by increasing task-­specific self-­efficacy (Ramalingam
et al., 2004). Previous research has found that increases in self-­efficacy can lead to an increase in academic performance in information technology areas (Smith, 2002). With regard
to lab-­based instruction, hands-­on lab exercises intended to increase self-­efficacy through
enactive mastery have also been shown to increase academic performance (Luse et al.,
2020). Given this supported premise that increases in self-­efficacy lead to increases in academic performance, we hypothesise,
Hypothesis 3 Weekly skills-­mastery labs utilising the novel ISEAGE Internet testbed will
positively increase student academic performance regarding knowledge pertaining to
networking and infrastructure.
LEARNING ENVIRONMENT AND LABORATORY EXERCISES
The authors of this paper were presented an opportunity to redesign an introductory networking course. In this redesign, the authors wanted to focus on increasing student engagement in the course through hands-­on activities to motivate students to move beyond
answering simple networking questions to mastering the concepts presented in class and
being able to utilise them to construct their own unique final projects. The authors decided
to challenge the students to implement their own ‘corporate’ network in a ‘real-­world’ setting as a final project. Therefore, the authors designed a set of weekly hands-­on activities
which, after successful organising, testing and evaluating, allowed the student to construct
an entire ‘corporate’ network over the course of a semester.
Learning environment
The learning environment for the introductory networking course was built using two servers
running VMware; one to allow students to build their ‘corporate’ network and the other for the
ISEAGE testbed. VMware provides a centralised server to enable multiple virtual computers
1248
|
LUSE and RURSCH
that can be freely used by the students and can be interconnected to create a virtual network set-­up. The ISEAGE testbed was developed as a cybersecurity research testing tool;
however, because it provides a controlled environment, it has also been used in more than
45 cybersecurity courses and events (Luse et al., 2020; Rursch & Jacobson, 2013a, 2013b;
Rursch et al., 2010). However, the collaboration described in this paper is the first time it was
used as a virtual lab for an introductory networking (non-­cyber security) course.
An overarching reason for opting to use the ISEAGE testbed in this course was to provide routable IP networks that appear to the students as if their traffic is routed through the
Internet. If a student performs a traceroute to another student network, the output received
would indicate multiple routers were installed between itself and the other network very
much like what a network administrator would see in a real corporate network. Additionally,
the student networks are assigned IP addresses that are ‘borrowed’ from public ranges
used in the Internet, but, because of the air gap proxy, none of these public IP addresses
ever escape into real Internet traffic. As the only egress point for traffic originating inside
the student networks, the air gap proxy allows the protocols http, https and ftp to pass out of
the students’ networks, but restricts all other traffic. This allows students connectivity to the
Internet to update systems and/or install patches, as well as to conduct searches for information about network or operating system problems, however, all other traffic is contained
inside the testbed.
Another important reason for opting for the ISEAGE testbed was the ability to use the
system as a virtual lab in a completely online environment. Students would be able to interact with their network and machines as if they were sitting in front of physical machines,
routers and switches. The rise in online education was the primary reason behind this move;
however, the need for rapid deployment of an online virtual lab due to the COVID pandemic
proved to be an added bonus of the system. Students were able to transition seamlessly
from using the system in a co-­located lab environment to a completely virtualised lab system.
Laboratory exercises
The weekly labs for the students focused on implementing a specified set of network services. The students were told at the beginning of the semester that their weekly labs would
culminate in a final project where they would build their own ‘corporate’ network. To conduct
the weekly labs and ultimately build the ‘corporate’ network students were asked to create
their own teams of three to four. Space on the student server was allocated at the beginning
of the semester with each team being given three virtual machines to use for their weekly labs
and ‘corporate’ network. The first machine each team was given had Windows preinstalled
on it. This was to be used as their client machine on the network. The students were required
to install and configure a second machine with routing services installed to act as the corporate router for the network. Finally, the team installed Windows Server on the third machine,
which would run most of the networking services. These services included an active directory domain server (AD), domain name service (DNS), dynamic host configuration protocol
(DHCP), remote desktop protocol (RDP), web server, file transfer protocol (FTP) and email
server. These include many common network services that users are accustomed to on a
corporate network including network-­wide username/password utilisation (AD), the ability to
access resources using a human-­friendly name such as www.google.com (DNS), automatic
IP assignment to allow users to use their computer on the network (DHCP), remote machine
access when working from home (RDP), a corporate website (web server), file sharing between users (FTP) and email services using corporate email addresses (email server).
As part of the initial instructions about the final project, each team was given a DNS name,
along with IP address information belonging to the team. Table 2 shows an example of this
USING A VIRTUAL LAB NETWORK TESTBED
TA B L E 2
|
1249
Example team IP information
Domain name
IP range
Subnet mask
Default gateway
DNS
team1a.com
64.39.3.0
/24
64.39.3.254
64.39.3.200
information for Team 1a. This team would have a domain name of team1a.com. The team
would need to configure all of their services to use this domain. For example, they would be
required to have a webpage that was reachable by typing in www.team1a.com. The www
represents the hostname and the service that is running, while the team1a.com would be the
domain name. The team was also given an IP address range to use on their corporate network with one of those IP addresses designated as the default gateway for sending any data
destined outside their network. Furthermore, students were required to handle all requests
for services on the team1a.com network by configuring their own DNS server.
The final project included two different phases. The first phase required students to have
all services running, configured properly for their network, and correctly answering requests. Additionally, students were required to submit a report describing each service and
documentation for how a ‘normal end user’ would access each service. The documentation
included directions, as well as user accounts/passwords needed to access each service.
At the end of this phase the faculty member evaluated the services by reading the team
report and accessing the team services using the process detailed in their documentation.
The computer used for testing the student network services did not live in the student's ‘corporate’ network but was on a separate network in the virtual lab environment. This allowed
‘real world’ tests of using DNS, accessing the webpage, uploading/downloading files using
ftp, connecting to the client machine using remote desktop and sending/receiving email for
a user on the students’ ‘corporate’ network. The second phase of the final project required
each team to evaluate the services of another team. Each student team was assigned the
report of one other team's ‘corporate’ network. The student team would then assess the services as an outside user within the same virtual lab environment, just as the faculty member
did, and write a report evaluating the services of that group. This allowed each team to go
beyond set-­up and use the skills they had learned to actually evaluate the work of another
team.
STUDY DESIGN, MATERIALS AND METHODS
Subjects for this study included upper-­level undergraduate students from three sections of
an introductory computer networking course at two different large universities in the Midwest
and south-­central United States. The original course redesign occurred at university 1, but
when one of the authors moved to university 2, the labs and final project were used there
and provided a second location in which to test the course redesign. Both courses were
taught mostly identically as both were part of programmes in the information systems department at both universities. While there are slight differences in the cadre of courses required for an information systems degree at both universities, both of the courses included in
this study at both universities were upper-­level courses taken by juniors and seniors where
this was the first networking/infrastructure course for students at both universities and the
only prerequisite for both was the general introductory information systems course taken by
all students in the college.
The study consisted of a pretest given the first week of the class. This pretest included
questions to measure the student's networking knowledge, self-­efficacy with regard to networking, and previous experience with networking and general computing. A posttest was
given the final week of class consisting of the same questions used to measure the student's
1250
|
LUSE and RURSCH
networking knowledge and self-­efficacy with regard to networking. In addition, one open-­
ended question regarding the usefulness of the project was also administered.
The primary research analyses examined one independent variable and two dependent
variables both at two separate time points using a within-­subjects general linear model
(GLM). The first dependent measure utilises 20 one-­point questions from a CompTIA
Network+practice test and consists of a single score ranging from 0 to 20 representing their
score on the exam. The second dependent measure, task self-­efficacy (TSE), consists of
nine items. The items are designed to measure a subject's self-­efficacy with regard to setting up and maintaining specific network services. Two covariates of previous networking
experience and number of courses taken in the area of networking were used to nullify possible differences in these areas among subjects, which could confound the results.
A 20-­question mini test was administered the first week of each course and again the last
week. The items were the same at both time periods and taken from a sample Network+
exam. Furthermore, the university the student was attending was analysed for possible differences among participants between universities. The items were gathered from a test-­
bank of practice questions used as a study aid for those preparing for the network+ exam.
The researchers included both questions that directly coincided with the content of the project as well as questions on other network topics not explicitly covered in the project in an
attempt to not bias the questions towards only those concepts specifically covered in the
project. The questions are given in the Appendix.
A nine-­question self-­assessment was administered the first week of the semester and
again the last week to measure an individual's self-­efficacy with regard to the topics covered
by the course. The items were the same at both time periods and adapted from guidelines
set forth by previous research (Marakas et al., 2007) who suggested that task-­specific measures of self-­efficacy offer greater insight into individual self-­efficacy assessment due to the
context-­specific nature of the questions. These specific task-­specific self-­efficacy question
in relation to computer networking set-­up were used in previous research to help validate
the use of task-­specific measures of self-­efficacy (Davazdahemami et al., 2018). See the
Appendix for the specific questions used.
To better understand the impact of the final project on student learning, a qualitative
analysis was also conducted. While the objective CompTIA Network+ survey was used to
measure the student's knowledge of networking and the subjective questions about self-­
efficacy demonstrated the student's increase in feeling that they could undertake a similar
task in the future, neither of these measures looked at the deeper cognition that students
achieved by ‘pulling it all together’ with the final project of constructing an operational ‘corporate’ network. To this end an open-­ended question was asked of each individual at the
end of the course:
If we were to use the project in the future, what things did you find useful?
The context of the question was presented with the intent of the authors to improve the project in future iterations of the course. The authors were not only looking for tangible suggestions
to change the final project pragmatically, but also to evaluate these statements in terms of
learning themes, as well as categories along Bloom's taxonomy as a qualitative study.
RESULTS
In total, 68 students filled out both the pretest and posttest. This amounted to a 60% response
rate across the two universities. The sample size included 41 students at university 1 and
27 students at university 2. The percentage of female respondents at each university was
USING A VIRTUAL LAB NETWORK TESTBED
|
1251
15% and 22% respectively, which is slightly above the 12% average of previous research in
college enrollment numbers in technology areas for women (Zweben, 2012). Self-­reported
previous experience with regard to computer networking was minimal for both universities
(39% –­ none, 34% –­ little at university 1 and 41% –­ none, 33% –­ little at university 2) with
only 2% and 11% reporting quite a bit of experience at each university respectively. While
only the introductory information systems course was required as a prerequisite at both universities, the number of reported courses in the area already taken was significantly higher
at university 2 (4.17) as compared to university 1 (2.68) (t = −2.97, p = 0.01). Table 3 summarises the descriptive statistics for the dependent measures and covariates.
Assumptions
Assumptions for the GLM (Agresti, 2018) includes independence of the observations within
each of the two time points, but not across time points given the within-­subjects nature of
the test. This independence assumption was reached by having the subjects fill out the
questionnaires at exactly the same point in time, independently of all others in the room, and
with no outside resources. Another assumption of the GLM is the dependent variable must
have some sort of exponential distribution (eg, binomial, Poisson, multinomial, normal). Our
data showed that the dependent variable of Network+ score was not significantly different
from normal (Kolmogorov–­Smirnov = 0.075, p = 0.20). While the TSE measure was found
to have some non-­normality (Kolmogorov–­Smirnov = 0.13, p = 0.006), this non-­normality
was found to be somewhat influenced by instructor such that instructor 1 showed normality (Kolmogorov–­Smirnov = 0.12, p = 0.20) while instructor 2 showed some non-­normality
(Kolmogorov–­Smirnov = 0.14, p = 0.02). Given this, instructor was included as a covariate
in the analyses.
Network+
Table 4 shows that even after accounting for previous experience and number of courses
taken in networking, time was found to have a significant positive effect on participant score,
supporting Hypothesis 2. Upon deeper inspection, this difference was found to vary in
size depending on the university of the participant (F(1,61) = 45.22, p = 0.008). Looking at
Table 5, while a significant increase in score is found between time 1 and time 2 for university 1 (simple main effect –­F(1,61) = 58.90, p < 0.001) and university 2 (simple main effect
–­ F(1,61) = 5.07, p = 0.028), the rate of increase differs. Specifically, at time 1, there is a significant difference in scores between individuals at university 1 and university 2 (simple main
effect –­ F(1,61) = 17.28, p < 0.001), but at time 2 this difference no longer exists (simple main
effect –­ F(1,61) = 0.045, p = 0.83). So, while students at university 1 start out with a lower
TA B L E 3
Descriptive statistics
Pretest
n = 68
Net+
Posttest
TSE
# Courses
Experience
Net+
TSE
Mean
7.57
2.83
3.23
0.93
10.88
5.57
SD
3.23
1.50
2.06
0.92
3.49
1.20
Items
20
9
1
1
20
9
Range
0–­20
1–­7
N/A
1– ­4
0–­20
1–­7
Cronbach's α
N/A
0.93
N/A
N/A
N/A
0.97
1252
|
TA B L E 4
LUSE and RURSCH
Overall test for network+ scores by university with covariates
Effects
IV
Within-­subject
Time
Between-­subject
University
Experience
F(1,61)
p value
6.51
0.013
TA B L E 5
Time * University
0.10
5.97
0.017
0.09
39.45
<0.001
0.39
0.30
0.590
0.01
45.22
0.008
0.11
Courses
Cross-­level
Partial
η2
Simple main effects for network+ scores by university with covariates
IV1
IV2
F(1,61)
p value
Partial η 2
Times
University 1
58.90
<0.001
0.49
University
Time 1
University 2
Time 2
TA B L E 6
0.045
0.028
0.08
<0.001
0.22
0.832
0.00
Overall test for network+ scores by faculty member with covariates
Effects
IV
Within-­subject
Time
Between-­subject
Professor
p value
Partial η 2
8.84
0.005
0.19
F(1,37)
1.06
0.310
0.03
14.20
0.001
0.28
Courses
0.30
0.589
0.01
Time * Professor
0.83
0.370
0.02
Experience
Cross-­level
5.07
17.28
average score, the scores for individuals from both universities are almost identical by the
end of the course. Furthermore, the effect size for this increase at university 2 is medium in
size (η 2 = 0.08), but a very large effect size exists for the increase at university 1 (η 2 = 0.49)
(Cohen, 2013).
For greater verification of the results, two subsamples were analysed for the students at
university 1 (see Table 6). The sample from university 1 consisted of students from two different sections of the same course. These sections utilised the same weekly labs and final
project, but were taught by two different faculty members. The results show that no significant difference exists in scores for the participants between the two courses (F(1,37) = 0.83,
p = 0.37) with both exhibiting a significant increase in scores between time 1 and time 2
(F(1,37) = 8.84, p = 0.005) with a high effect size (η 2 –­0.19).1 This provides greater credence
to the assumption that the differences in scores are not due to professorial acumen or style
of the faculty member, but instead to the utilisation of the weekly labs and final project.
Task self-­efficacy (TSE)
Before analysing changes in task self-­efficacy, internal consistency was examined for both
time points with high internal consistency among the items at time one (Cronbach's α = 0.93)
and time two (Cronbach's α = 0.97). Table 7 shows that even after accounting for previous
USING A VIRTUAL LAB NETWORK TESTBED
TA B L E 7
|
1253
Overall test for task self efficacy by university with covariates
Effects
IV
F(1,61)
p value
Partial
η2
Within-­subject
Time
88.03
<0.001
0.59
Between-­subject
University
0.226
0.02
64.06
<0.001
0.51
Courses
0.33
0.568
0.01
Time * University
0.00
0.981
0.00
Experience
Cross-­level
TA B L E 8
1.50
Overall test for task self efficacy by faculty member with covariates
Effects
IV
F(1,37)
p value
Partial
η2
Within-­subject
Time
65.68
<0.001
0.64
Between-­subject
Professor
0.50
0.486
0.01
22.19
<0.001
0.38
Courses
0.19
0.669
0.01
Time * Professor
0.01
0.946
0.00
Experience
Cross-­level
experience and number of courses taken, time was found to have a significant positive
effect on participant score (F(1,61) = 88.03, p < 0.001) with a very high effect size (η 2 =
0.59), supporting Hypothesis 3. This increase in score was not found to vary by university
(F(1,61) = 0.00, p = 0.981) or by faculty member of the two different sections from university
1 (F(1,37) = 0.01, p = 0.946) (see Table 8). So, the students show a significant increase in
TSE regardless of the university they belong to or the instructor of the course.
Qualitative
The qualitative analysis utilised a thematic examination as discussed in previous qualitative
research (Fereday & Muir-­Cochrane, 2006). Thematic analysis involves a hybrid approach
that integrates data-­driven codes as well as theory-­driven codes based on social phenomenology (Schutz, 1972). This method uses the inductive approach of data-­driven thematic
analysis (Boyatzis, 1998) and the deductive approach of a priori codes from prior theory
(Crabtree & Miller, 1999). For this research, two researchers analysed the same answers
to the question above separately. The first researcher analysed the data using an inductive
approach whereby the researcher identified themes that emerged in the data. The second
researcher, who was familiar with Bloom's taxonomy, utilised the tenets of the taxonomy as
a priori themes and categorised the data according to these theoretical tenets.
This open-­ended question yielded 38 usable statements when analysed by the first coder
for emergent themes using an emergent coding perspective (Glaser & Strauss, 2009).
Table 9 shows the themes that were found.
A second coder read these same statements without access to the initial themes from the
first coder and placed each statement into one of the six categories of Bloom's taxonomy.
This approach yielded 45 useful statements. Table 10 shows these results.
Overall, the students at both universities showed the greatest number of cognitions at the
creating level, supporting Hypothesis 1. The most frequently mentioned theme focused on
the actual set-­up of the entire ‘corporate’ network in the final project. Comments that were
coded into this theme revolved around having the ability to create a full-­fledged network and
1254
|
TA B L E 9
LUSE and RURSCH
Themes discovered
Set-­up
University 1
University 2
13
9
Hands- ­on experience
7
6
Real-­world application
3
2
TA B L E 1 0
Bloom's levels identified as themes
University 1
University 2
Creating
13
10
Applying
7
7
Understanding
5
0
Evaluating
2
1
the ability to work with the variety of network services in a fully functional environment. For
example, one student noted:
The most useful part of the project was becoming more comfortable and confident in setting up and understanding a network. I think that by having each component assembled individually, it becomes much more apparent how the entire
network, and all network(s) work together.
These types of statements, which were thematically labelled as set-­up by the first coder,
were consistently coded by the second coder as the Creating category in Bloom's revised
taxonomy. They were classified as such since the students were building their own network
and their statements related their ability to put all the small pieces together to build the larger
‘corporate’ network.
The second most noted theme by the first coder related to gaining hands-­on experience.
One student stated:
I think that the most beneficial part of the project was the hands-­on experience.
Up until now, I had no experience on installing, setting up, and configuring a
server. This project has greatly expanded my knowledge on networking.
Statements like these were consistently categorised by the second coder as the Applying
stage of Bloom's revised taxonomy where students were carrying out network configurations or
running operating system installations.
A third and final theme from the first coder that started to emerge in some of the student
comments reflected the ‘real world’ application of the final project. One student commented:
I do believe the project is a great experience and should be done in the future, it
is one thing to look at paper and learn tidbits of info but much more beneficial to
actually be put in a real world application and create a network.
This type of statement was consistently coded by the second coder as Understanding in the
Bloom's revised taxonomy. The student could see the relationship between what was being
done in the lab environment as a reflection of the world which they were training to enter.
USING A VIRTUAL LAB NETWORK TESTBED
|
1255
The thematic method utilised here also allowed for multiple methods of triangulation in
order to establish validity for the study. Denzin describes four types of triangulation for qualitative research including (1) data, (2) methodological, (3) theoretical and (4) investigator
(2017). Data triangulation involves analysing data from different times, space and persons.
Given our analysis analysed multiple individuals from multiple universities across different
semesters, our data attains data triangulation for time, space and persons. Methodological
triangulation utilises multiple methods in order to better understand data. Our use of both an
inductive and deductive approach meets the requirement for methodological triangulation.
Theoretical (or perspective) triangulation involves utilising alternative theoretical schemes.
Our use of Bloom's taxonomy as a separate theory for analysis adheres to theoretical triangulation. Investigator triangulation involves the use of multiple individuals to analyse the
data. Our use of two independent observers meets the requirement for investigator triangulation. Finally, to test for reliability, two methods were used to examine inter-­rater agreement
and inter-­rater reliability. The percentage total agreement showed a very high inter-­rater
agreement of 94.7%. Furthermore, a Cohen's Kappa value of 0.90 showed a very high degree of inter-­rater reliability (Landis & Koch, 1977).
DISCUSSION
Virtual labs have become popular for usage in teaching courses that require hands-­on education within the constraints of online learning with the goal of students thinking at the upper
levels of Bloom's taxonomy. This research examined the use of a virtual internet environment whereby students could set-­up networking and services in a corporate environment
and interact with other student corporate environments. The weekly labs and the final project evaluated in this research were designed to move students beyond the normal lab space
of applying to the evaluating and creating stages of Bloom's taxonomy. Like others before,
this project used the weekly labs to build self-­efficacy in the students through enactive mastery. However, with the addition of the final project, the authors worked to provide a capstone
experience.
Three different assessment metrics were used to evaluate the virtual lab technology.
The first focused on pretest and posttest measures of individual achievement with regard to
networking topics covered during the semester. The second included pretest and posttest
individual self-­assessment of self-­efficacy regarding the skills taught with the weekly labs.
The third, was a posttest qualitative assessment of what the student felt was beneficial about
the project which was used as a metric for the depth of cognition in the responses and coded
as related to Bloom's taxonomy.
Results showed that while individual performance on the mini-­exam related to networking
topics increases over time, the size of increase depends on the university. The authors did
not find this difference in pretest scores to be surprising as the programme at university 2 is
more technical in nature as compared to the programme at university 1. What was surprising
is that, even with this initial difference, the posttest scores were not found to be significantly
different. This provides great promise for the use of the virtual lab technology to implement
a real-­world final project at other institutions. While the posttest scores of the individuals
at both universities increased, the weekly labs and final project provided an even greater
increase for those who started out with a lower pretest score. In other words, the entire experience proved to be an equaliser as both groups of students ended the course with the same
level of ability despite starting the course at different levels of proficiency.
While the first assessment was objective, the second assessment was a subjective
self-­assessment given by the student of their own self-­efficacy with the concepts from the
course. Self-­efficacy states that those individuals who feel as though they are able to do
1256
|
LUSE and RURSCH
something well will be more likely to be able to do it well in the future (Bandura, 1986). The
results showed that the self-­efficacy of the students with regard to the content covered by
the course increased significantly, with this increase being the same regardless of university. This again paints a picture of the virtual technology implementing weekly labs and a
‘real world’ project as being valuable.
Both the objective and subjective results did not depend on the faculty member teaching
the course, as both analyses showed a statistically significant increase in both the mini-­
exam and self-­efficacy scores that were not significantly different between two different
faculty members teaching the same course at university 1. This again provides credence to
the effectiveness of the virtual lab technology implementing weekly labs and final project as
a learning tool, regardless of who is teaching the material.
Finally, a qualitative analysis of an open-­ended question regarding the usefulness of the
virtual lab technology to the student was used to evaluate student cognition in relation to
Bloom's taxonomy. The three emergent themes pointed to students feeling the virtual lab
increased their overall comprehension of the course material. When these comments were
coded into taxonomy title words Creating, Applying and Understanding were reported by the
students.
Two interesting points are notable based on the qualitative analysis. First, the frequency
of comments in each category was relatively the same across universities (see again Tables
9 and 10). Second, the type of qualitative analysis used in this study points both to a somewhat novel process as well as interesting findings. Classic qualitative coding methods typically involve either an emergent perspective (inductive) where the coder starts with no
extant codes and lets the themes emerge from the data, or a more confirmatory perspective
(deductive) where the coder begins with a theory and codes the comments in-­line with the
theoretical components of the theory. Our analysis used a thematic analysis based on social phenomenology (Fereday & Muir-­Cochrane, 2006; Schutz, 1972) that involved both approaches. However, whereby previous research using this method utilises the same coder
for each method, we utilised separate coders that did not have interaction with the other.
What is interesting is that the themes emerging from the two coders showed a strikingly high
correlation between the matching themes of set-­up/creating, hands-­on experience/applying
and real-­world application/understanding as documented above.
IMPLICATIONS
Implications of this research are seen both for courses on the same topic as well as for
general lab-­based instruction. First, the use of the ISEAGE Internet testbed is shown to provide positive increases in both knowledge and feelings of efficacy with regard to networking
and infrastructure technology. These increases are shown for a semester-­long course on
networking and infrastructure using the ISEAGE system, as compared to previous research
only using the system for cyber defence competitions or courses. Given the free availability
of the testbed software (http://www.iseri​nk.org/), this research provides the opportunity for
others to implement the same technology for their own course, thereby rolling out the system
to enable students to have hands-­on training in networking and infrastructure.
For the wider community, this research demonstrates the use of real world, hands-­on
labs to increase student achievement. While other simulation-­based training has been used
in courses, this system does not simulate the technology, but instead allows the students
to interact with and configure the same systems that they will configure in the real world for
their jobs. Overall, this type of testbed provides corroboration for the use of such testbeds to
allow for actual interaction as opposed to modules simulating this interaction.
USING A VIRTUAL LAB NETWORK TESTBED
|
1257
LIMITATIONS AND FUTURE WORK
There are several areas for improvement and future work with regard to this research. First,
the qualitative assessment provides no comparison to other research. The purpose of the
qualitative assessment was to gauge the ability of the students to think at upper levels of
Bloom's taxonomy. While upper-­level cognitions were identified, the ability of the overall lab
to move students to upper levels of the taxonomy is not part of this research. Future research
should examine the ability of the proposed lab course as compared to similar courses that
do not implement the ISEAGE environment, in order to ascertain the movement of students
from lower to upper levels of the taxonomy with the added implementation of the lab.
While the authors plan to continue to use the mini-­exam and the self-­efficacy assessments in their on-­going continual improvement for the course, they recognise the need to
further refine the measurement tool evaluating the cognition of students at upper levels of
Bloom's taxonomy. For this work, the open-­ended question asking about the usefulness of
the project for the student provided formative work from which the authors will develop quantitative questions to more precisely measure the level of cognition the students achieve from
the redesigned labs. Further and more in-­depth qualitative assessment is needed to more
fully understand the area utilising a greater number of questions and probative sessions as
compared to the current research that only provides a first step.
Further, the measurement tools used in this introductory course are being refined to evaluate a masters level engineering capstone course focused on information assurance and
network security. This course is an existing course where graduate students design, implement, secure and defend a ‘corporate network’ over the semester. While the introductory
networking course involved inclusion of new concepts and material, the graduate level capstone course is a culminating experience where all previous coursework should be utilised in
the creation of the ‘corporate network’. This will provide a comparative estimate with regard
to all aspects of this research between different populations, including both the quantitative
and qualitative methods.
CONCLUSION
Overall, this research provides an evaluation of a virtual lab environment for teaching networking concepts. The results paint a very good picture of the virtual lab environment as a
promising technology for use in networking education. With the continued need for online
education, this research provides a virtual lab solution that continues to provide the necessary hands-­on experience needed for students in computer and networking technologies.
CO N FL I C T O F I NT E R EST
No conflict of interest exists.
DATA AVA I L A B I L I T Y STAT E M E NT
The data used in this manuscript can be obtained by contacting the first author at andyluse@okstate.edu.
ETHICS
Subjects were protected by securing the data on a private server. Identifying information
was only kept to enable pairing of pre and posttest data, which was then removed before
analysis.
1258
|
LUSE and RURSCH
E N D N OT E
1
Given the non-­significant result, a post hoc power analysis was run to ensure sufficient power to find a main
effect of professor if one existed. The post hoc analysis found power of 0.96, indicating a high degree of power
for finding a main effect if one truly existed.
REFERENCES
Agarwal R., Sambamurthy V., Stair R. M. (2000) The evolving relationship between general and specific computer
self-­efficacy: An empirical assessment. Information Systems Research 11(4):418–­430.
Agresti A. (2018) Statistical methods for the social sciences (5th ed.). Pearson.
Alioon Y., Delialioğlu Ö. (2019) The effect of authentic m-­learning activities on student engagement and motivation. British Journal of Educational Technology 50(2):655–­6 68.
Altmeyer K., Kapp S., Thees M., Malone S., Kuhn J., Brünken R. (2020) The use of augmented reality to foster
conceptual knowledge acquisition in STEM laboratory courses—­Theoretical background and empirical results. British Journal of Educational Technology 51(3):611–­628.
Anderson L. W., Krathwohl D. R. (2001) A taxonomy for learning, teaching and assessing: A revision of Bloom's
Taxonomy of educational objectives: Complete edition. Longman.
Anderson, L. W., & Sosniak, L. A. (1994). Bloom's taxonomy: A forty-­year retrospective. Ninety-­third yearbook of
the National Society for the Study of Education, Pt.2. University of Chicago Press.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice-­Hall.
Bandura, A. (1997). Self-­efficacy: The exercise of control. Freeman.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of education goals. Longman.
Boyatzis, R. (1998). Transforming qualitative information: Thematic analysis and code development. Sage
Publications.
Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic Press.
Compeau, D. R., & Higgins, C. A. (1995). Computer self-­efficacy: Development of a measure and initial test. MIS
Quarterly, 19(2), 189–­211. https://doi.org/10.2307/249688
Crabtree, B., & Miller, W. (1999). A template approach to text analysis: Developing and using codebooks. In B.
Crabreee & W. Miller (Eds.), Doing qualitative research (pp. 163–­177). Sage.
Crompton, H., Burke, D., & Lin, Y. C. (2019). Mobile learning and student cognition: A systematic review of PK-­12
research using Bloom's Taxonomy. British Journal of Educational Technology, 50(2), 684–­701. https://doi.
org/10.1111/bjet.12674
Davazdahemami, B., Luse, A., Scheibe, K. P., & Townsend, A. M. (2018). Training, self-­efficacy, and performance: A replication study. AIS Transactions on Replication Research, 4(1), 1–­18. https://doi.org/10.17705/​
1atrr.00023
Denzin, N. K. (2017). The research act: A theoretical introduction to sociological methods. Transaction Publishers.
Fereday, J., & Muir-­Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of
inductive and deductive coding and theme development. International Journal of Qualitative Methods, 5(1),
80–­92. https://doi.org/10.1177/16094​0 6906​0 0500107
Glaser, B. G., & Strauss, A. L. (2009). The discovery of grounded theory: Strategies for qualitative research.
Transaction Publishers.
Johnson, R. D., & Marakas, G. M. (2000). The role of behavioral modeling in computer skills acquisition: Toward
refinement of the model. Information Systems Research, 11(4), 403–­417.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics,
33(1), 159–­174.
Lent, R. W. (2005). A social cognitive view of career development and counseling. In S. D. Brown & R. W. Lent
(Eds.), Career development and counseling: Putting theory and research to work (pp. 101–­127). Wiley.
Luse, A., Brown, A., & Rursch, J. (2020). Instruction in 802.11 technology in online virtual labs. IEEE Transactions
on Education. https://doi.org/10.1109/TE.2020.2998701
Luse, A., Rursch, J. A., & Jacobson, D. (2014). Utilizing structural equation modeling and social cognitive career
theory to identify factors in choice of IT as a major. ACM Transactions on Computing Education, 14(3), 1–­19.
https://doi.org/10.1145/2623198
Marakas, G. M., Johnson, R. D., & Clay, P. F. (2007). The evolving nature of the computer self-­efficacy construct:
An empirical investigation of measurement construction, validity, reliability and stability over time. Journal of
the Association for Information Systems, 8(1), 16–­46. https://doi.org/10.17705/​1jais.00112
Ramalingam, V., LaBelle, D., & Wiedenbeck, S. (2004). Self-­efficacy and mental models in learning to program.
Paper presented at the Proceedings of the 9th annual SIGCSE conference on Innovation and technology in
computer science education.
USING A VIRTUAL LAB NETWORK TESTBED
|
1259
Rursch, J. A., & Jacobson, D. (2013a, October 23–­26). This IS child's play—­Creating a playground (computer network testbed) for high school students to learn, practice and compete in cyber defense competitions. Paper
presented at the Frontiers in Education, Oklahoma City, OK.
Rursch, J. A., & Jacobson, D. (2013b, October 23–­26). When a testbed does more than testing: The Internet-­
Scale Event Attack and Generation Environment (ISEAGE) –­Providing learning and synthesizing experiences for cyber security students. Paper presented at the Frontiers in Education, Oklahoma City, OK.
Rursch, J. A., Luse, A., & Jacobson, D. (2010). IT-­adventures: A program to spark IT interest in high school students using inquiry-­based learning with cyber defense, game design, and robotics. IEEE Transactions on
Education, 53(1), 71–­79. https://doi.org/10.1109/TE.2009.2024080
Scheibe, K. P., Mennecke, B. E., & Luse, A. (2007). The role of effective modeling in the development of self-­
efficacy: The case of the transparent engine. Decision Sciences Journal of Innovative Education, 5(1), 21–­42.
https://doi.org/10.1111/j.1540-4609.2007.00126.x
Schutz, A. (1972). The phenomenology of the social world. Northwestern University Press.
Smith, S. M. (2002). The role of social cognitive career theory in information technology based academic performance. Information Technology, Learning, and Performance Journal, 20(2), 1–­10.
Zweben, S. (2012). Computing degree and enrollment trends from the 2010–­2011 CRA Taulbee Survey.
How to cite this article: Luse, A., & Rursch, J. (2021). Using a virtual lab network
testbed to facilitate real-­world hands-­on learning in a networking course. British
Journal of Educational Technology, 52, 1244–­1261. https://doi.org/10.1111/bjet.13070
A PPE N D I X
Task-­Specific Computer Networking Self-­Efficacy (Davazdahemami et al., 2018)
1.
2.
3.
4.
5.
6.
7.
I can set-­
up DNS for a network.
I can set-­up a web server for individuals to view webpages.
I can set-­up DHCP to dynamically configure IP settings for client machines.
I can set-­up an email server to send and receive email for a domain.
I can set-­up Active Directory to allow network-­wide management.
I can set-­up FTP to allow for file sharing on my network.
I can set-­up a machine to provide routing of traffic between the outside world and my
network.
8. I can set-­up a network to allow the machines to connect with each other.
9. I can install and set-­up a server.
Which of the following tools is used for querying DNS server to obtain domain name or IP
address mapping?
• nbtstat
• tracert
• nslookup
• ipconfig
HTTPS: (Select 2)
• Uses SSL/TLS
• Does not encrypt the entire communication channel
• URLs begin with shttp://
• Runs on TCP port 443
• Runs on TCP port 143
Which of the following TCP ports is used for FTP data transfer?
• 23
• 20
• 21
• 22
1260
|
LUSE and RURSCH
Which of the following cabling types is used with BNC connectors?
• Twisted pair
• Fibre-­optic
• Unshielded twisted pair (UTP)
• Coaxial
IPv6 address consists of:
• 128 bits
• 32 bits
• 64 bits
• 48 bits
Which of the following is a private class C address?
• 10.0.5.24/8
• 192.168.0.55/16
• 192.168.0.1/24
• 172.16.1.5/8
Network Time Protocol (NTP) uses:
• UDP port 123
• UDP port 67
• TCP port 143
• TCP port 110
Which of the following port numbers are used by Simple Network Management Protocol
(SNMP)? (Select 2)
• 137
• 138
• 161
• 162
• 389
Wireless network name is also known as:
• DSSS
• IFS
• MAC
• SSID
Smurf attack is an example of: (Select best answer)
• SYN flood
• Three-­way handshake
• Distributed Denial of Service
• Denial of Service
The layers of the OSI model, from the top down, are:
• Application, presentation, session, transport, network, data link, physical
• Session, presentation, data transport, MAC, network, physical
• Physical, data link, network, transport, session, presentation, application
• Presentation, application, session, network, transport, data link, physical
• Application, encryption, network, transport, logical link control, physical
Fake logon screen on a system will most probably be a result of downloading and installing:
• Trojan horse
• Worm
• Botnet
• Adware
Which of the following is also known as an echo request?
• ICMP packet
• nslookup
USING A VIRTUAL LAB NETWORK TESTBED
|
1261
• tracert
• SNMP packet
Internet Group Management Protocol (IGMP): (select 2)
• Is used on IPv4 and IPv6 networks
• Is used for establishing multicast group memberships
• Is used on IPv4 networks
• Operates at Layer 5 of the OSI model
IMAP4: (Select 2)
• Runs on TCP port 110
• Runs on TCP port 143
• Is used for retrieving e-mail messages from e-mail servers
• Is used for sending e-mails between SMTP servers
• Offers less functionality than POP3
Which of the following protocols are used for retrieving e-mail messages from e-mail servers? (Select 2)
• POP
• SMTP
• IMAP
• OSPF
• SNMP
MAC address: (Select 2)
• Consists of 48 bits
• Has a part called OUI, which identifies the device (NIC)
• Consists of 64 bits
• Is also known as a physical address
Can be displayed by typing ipconfig command in Windows command-­line utility
Which of the following is an access method used in Ethernet networks?
• OSI
• CSMA/CA
• CSMA/CD
• SONET
When two communicating nodes can send and receive but not at the same time, the communication is described as:
• Full-­duplex
• Simplex
• Half-­duplex
• Multiplex
What is the default subnet mask for a class C network?
• 127.0.0.1
• 255.0.0.0
• 255.255.0.0
• 255.255.255.0
• None of the above