The effects of individual differences and visual instructional aids on
disorientation, learning performance and attitudes in a Hypermedia
Learning System
Abstract
Research suggests that certain visual instructional aids can reduce levels of disorientation and increase
learning performance in, and positive attitudes towards, HLS for learners with specific individual
differences. However, existing studies have looked at only one or two individual differences at a time,
and/or considered only a small number of visual instructional aids. No study has considered the impact
of the three most commonly studied individual differences – cognitive style, domain knowledge and
computer experience – on learning performance, disorientation and attitudes in a HLS incorporating a
full range of visual instructional aids. The study reported here addresses this shortcoming, examining
the effects of, and between, these three individual differences in relation to learning performance,
disorientation and attitudes in two HLS versions: one that incorporated a full set of visual instructional
aids and one that did not. Significant effects were found between the three individual differences with
respect to disorientation, learning performance and attitudes in the HLS that provided no instructional
aids, whereas no such effects were found for the other HLS version. Analysis of the results led to a set
of HLS design guidelines, presented in the paper, and the development of an agenda for future
research. Limitations of the study and their implications for the generalizability of the findings are also
presented.
Keywords: Hypermedia Learning; Individual Differences; Visual Support; Disorientation, Learning
Performance; Attitudes
1. Introduction
Hypermedia Learning Systems (HLS) are being increasingly used in Higher Education [55] to
support students’ access to learning material in a flexible way. A defining feature of HLS is that the
learning material is presented using a non-linear structure [29], allowing students to determine their
own path through the material [4]. Allowing learners to decide the sequence in which they encounter
1
the learning content has been suggested to offer improvements in learning and cognitive flexibility
[53]. However, some users have difficulty in navigating through HLS to find the information that they
need and, as a result, experience disorientation [6]. A consequence of disorientation is that learners can
miss at least some of the relevant content in the system, which may hinder their learning performance
[36]. Performing less well may lead to these learners having a negative attitude towards HLS and also
to have less interest in learning using these types of learning system [22].
Research findings suggest that not all learners are comfortable using, satisfied with or learn
effectively from HLS, implying that the value of HLS varies depending on the individual and may be
influenced by various characteristics of the learner [26]. This means that the individual differences that
these characteristics represent become important when designing and developing HLS. A range of
studies have looked at the relationship between individual differences and student learning in HLS,
with the results tending to show that individual differences, particularly cognitive style (most often
examined in terms of Field Dependence/Field Independence), domain knowledge and computer
experience, influence learners’ levels of disorientation, learning performance and attitudes in HLS.
Though widely studied individually or in pairs, the influence of, and relationships between, all of these
individual differences in relation to disorientation, learning performance and attitude have not been
looked in a single study.
It is argued that supporting individual differences in HLS to reduce learners’ disorientation would
be helpful, as it can improve learning performance and increase learning satisfaction [51]. A common
way to reduce disorientation is to provide instructional guidance, in the form of visual navigational aids
(e.g., maps) and a set of visual cues (e.g., breadcrumbs, highlighting of context, pagination and so on),
within the HLS. The relationship between individual differences and such instructional guidance has
been explored in many studies, yet there are no studies that have integrated a map and these visual cues
in a single HLS.
This paper reports a study that seeks to address these gaps in the research literature through by
examining the effects of three individual differences (cognitive style, domain knowledge and computer
experience) on disorientation, learning performance and attitudes in HLS with and without instructional
guidance in the form of visual navigational aids and a set of visual cues. The paper begins by
reviewing relevant literature related to individual difference and HLS use, in order to identify and to
frame three research questions that will address these gaps. The paper then sets out the methodological
2
approach to exploring the research questions, introducing the empirical study, its research design,
sample, the range of materials and instruments used, and the detailed experimental procedure.
The
paper then presents the results of the study and analyses the resulting data in order to provide answers
to the three research questions. The paper ends by framing a set of design guidelines from the analysis
of the data and by setting out directions for future work.
2. Background
As the World Wide Web (WWW) becomes ever more widely used as an educational platform [55],
HLS are gaining increased attention from researchers [5, 31]. One of the major reasons for moving
from traditional classroom-based learning to offering instruction through the use of HLS is that the
latter can present learning material in a non-linear structure [10]. Such non-linearity affords learners
greater flexibility in navigating the learning content and allows them to choose their own paths through
it to meet their learning goals [4]. Additionally, non-linearity allows learners to access and sequence
information in accordance with their individual needs [25]. Furthermore, allowing learners to have
control over their learning may also make them motivated to learn, improving their learning
performance and cognitive flexibility [53].
However, the flexibility offered by HLS may cause problems for some users. It has been argued
that not all users can ‘develop’ their own learning paths effectively to achieve their learning goals when
using HLS [27], and many studies have shown that users may experience disorientation – reflected as
questions of ‘where am I?’, ‘where have I been?’ and ‘where can I go next?’ – when seeking to
navigate through HLS [6].
Some studies have suggested that learners who encounter higher levels of disorientation may, in
turn, perform less well in learning tasks [36]. Separately and in combination, disorientation and poorer
performance in learning tasks may have an impact on learners’ attitudes towards HLS. Dringus [22],
for example, argues that when learners experience disorientation in HLS, and are hindered in their
learning performance, there have an increased chance of showing negative attitudes towards the nonlinear learning environment. As a result, such learners may feel less motivation to learn using HLS.
The range of reactions to being given freedom in terms of navigation may be explained by the
different characteristics that learners possess, meaning that the individual differences that these
characteristics represent are critical for effective HLS’ design. It has been suggested that individual
3
differences such as cognitive style [15], domain knowledge [8], and computer experience [55] are the
most commonly studied in research related to student learning and HLS use. Each of these individual
differences will be briefly introduced in the context of research into HLS.
The ways in which an individual thinks, memorizes, perceives, organizes, processes and presents
information is often referred to as cognitive style [47]. Among the different dimensions of cognitive
style that have been studied to date, Field Dependence (FD) and Field Independence (FI) are often
argued to be of interest, especially with respect to research that is related to HLS [12]. FI learners tend
to rely on internal references, adopt an active approach to learning and process information using an
analytical approach. Conversely, FD learners tend to rely on external references, adopt a passive
approach to learning and accept information in exactly the way it is presented to them [17, 58].
Witkins et al. [58] Group Embedded Figure Test (GEFT) and Riding’s [46] Cognitive Style Analysis
(CSA) are two common instruments used to identify a learner’s cognitive style.
The research related to cognitive style and HLS suggests that FI learners: prefer being given high
levels of freedom of navigation; experience less disorientation in HLS; perform well in learning tasks;
and have a positive attitude towards HLS compared to FD learners [16, 32, 54]. One explanation of FI
learners’ performance in HLS is that they are able to follow a restructuring approach more easily
because they are internally directed and tend to adopt an active approach to learning; and they can
extract relevant items from within the complex context offered within complex content, such as that
offered in HLS, because they are more analytical. In contrast, FD learners prefer to follow the
structure of the learning material because they are externally-directed and tend to adopt a passive
approach to learning; they also have difficulties extracting the relevant items within the complex
context because they are less analytical [17, 59].
Research into domain knowledge, mostly suggests that domain knowledge influences the degree of
disorientation and learning performance in HLS, with low domain knowledge (novice) learners
experiencing higher levels of disorientation and performing less well in HLS than high domain
knowledge (expert) learners [7, 23, 35, 57]. A general conclusion drawn in studies in this area to
explain this finding is that, novices are unfamiliar with the subject content, which makes it difficult for
them to impose a meaningful conceptual structure on the content compared to experts [13].
In terms of the research related to computer experience, studies suggest that compared with those
with lower levels of computer experience, learners with high levels of computer experience: prefer the
4
non-linear pathways that are normally offered in HLS; navigate effectively; take fewer steps to reach
the information they need in the tutorial; browse more pages and are able to reach more detailed levels
of the subject content; enhanced their time efficacy; and overall, perform well in learning tasks in the
HLS [30, 37, 52, 55]. It is argued that this is because a well-developed understanding of different
computer applications enables these learners to better navigate through the HLS to achieve their
learning goals. In contrast, the lack of skills related to the use of computers and their applications
makes it difficult for those with low levels of computer experience to successfully navigate through
HLS to find the information that they need [40, 52].
It is suggested that reducing the levels of disorientation experienced by learners with different
characteristics may improve their learning performance and, in turn, may show more interest in
learning using HLS [12, 13]. To reduce learners’ disorientation, the use of instructional aids, such as
maps which support visual navigational, has been suggested [5, 8, 13, 14], and a number of studies
have examined the effects of both maps and individual differences (such as those considered in this
paper) on learners’ levels of disorientation, learning performance and attitudes in HLS [3, 8, 33, 34,
45]. The results from these studies suggest that when maps are provided in HLS, learners’ levels of
disorientation are decreased and learning performance and positive attitude are increased. However,
these studies have not considered the effects of all of the three individual differences – cognitive style,
domain knowledge and computer experience – in combination with maps in HLS. Rather, they have
tended to examine the effects of only one or two of the individual differences presented in this study.
Studies which examine the effect of maps in relation to a wider set of individual differences may help
to determine whether maps reduce disorientation and increase learning performance and learning
satisfaction for all users with different combinations of the individual differences, or whether maps
create problems for specific groups.
In addition to maps, previous studies suggest that visual cues – such as breadcrumbs [1, 44],
graphic visualizations [39, 42], history-based mechanisms [24], context highlighting [20], page labels
[18], different colors link [42] and link annotations [9] – can also reduce disorientation and enhance
efficiency of learning performance and satisfaction in HLS [19, 49].
Despite the fact that these studies have explored the same set of individual differences considered in
this paper and have considered a range of these visual cues, an important gap remains. There are no
studies where cognitive style, domain knowledge and computer experience have been considered
5
together in relation to all of the visual cues identified earlier within a single HLS. This implies that the
combined effects of these three individual differences and all of these visual cues on learners’
disorientation, learning performance and attitudes in HLS have not been fully examined. Though
methodologically complex in terms of study design and subsequent analysis, addressing this gap may
provide results which can help designers and developers to gain a better understanding of the
relationships between HLS instructional aids (in the form of visual cues), individual differences and
learner disorientation, performance and attitude. This leads to the framing of the three research
questions addressed in the study reported in the remainder of this paper:

Research Question 1:
What are the effects of and between cognitive style (FD/FI), domain knowledge, and
computer experience on learners’ levels of disorientation when using a HLS that includes a
map and the defined set of visual cues and when using a HLS without any instructional aids?

Research Question 2:
What are the effects of and between cognitive style (FD/FI), domain knowledge, and
computer experience on learners’ learning performance when using a HLS that includes a map
and the defined set of visual cues and when using a HLS without any instructional aids?

Research Question 3:
What are the effects or and between cognitive style (FD/FI), domain knowledge, and
computer experience on learners’ attitudes when using a HLS that includes a map and the
defined set of visual cues and when using a HLS without any instructional aids?
3. Methodology
This section describes the research methodology that was used to address the three research
questions. An explanation of the research design is given, followed by a description of the sample, the
materials and instruments used in the study, the experimental procedure and the approach to data
analysis.
6
3.1 Research Design
To answer the three research questions that were proposed, this study adopted an experimental
research approach [48], in which a set of independent variables and dependent variables were identified
and used. The independent variables were the HLS and the users’ individual differences (cognitive
style, domain knowledge and computer experience). With respect to the HLS, two versions, one
without instructional aids and one with visual instructional aids (in the form of a map and a set of
visual cues) were required.
The dependent variables in this experimental study were learning
performance, disorientation and attitudes towards the HLS. A between-subjects design was used in this
study, with one half of the sample using the HLS that provided no instructional aids and the other half
of the sample using the HLS containing the set of visual instructional aids defined above.
It is
acknowledged that this research design is complex and has a wider range of variables than have been
used in other studies in the field. This brings significant issues in terms of analysis of the data and the
implications that can be drawn from them. These issues are discussed, and reflected on, in later
sections of the paper.
This study also aimed to gather detailed user information on individual differences, learners’
attitudes, feelings and preferences, their experience of disorientation, and their interaction behavior
with respect to the HLS that they used. To support the experimental study, a descriptive study was also
employed (using a qualitative approach), in which learners were observed, surveyed and interviewed
[41, 48]. However, for reasons of space, this paper will consider only the experimental study and its
results.
3.2 Sample
The sample was drawn from university students across London, UK. University students were
considered to be suitable participants because, as mentioned in sections 1 and 2, this study was
concerned with research related to students and HLS in Higher Education. Recruitment of the sample
was also supported by the existence of channels through which we were able to contact a large number
of students at universities in the London region.
Though the choice of university students as
participants can be seen as part of a purposive sampling strategy, the restriction of the recruitment of
participants to those from the London region introduce a ‘convenience’ characteristic to the sample
7
which is important to acknowledge, and which will be returned to in section 7 when the implications of
the findings with respect to the study’s limitations, are discussed.
192 participants took part in the study: split equally between the HLS that provided no instructional
aids and the HLS that contained the visual instructional aids. The participants were at undergraduate,
postgraduate taught and postgraduate research levels and were registered on a range of courses at
universities in the London region.
They were informed about the experimental study through a
discussion forum, notice boards in the Students’ Unions of universities across London and by word of
mouth. Participants were also told that taking part in this study would help them to learn about, and to
build their own web site, either for personal or business use, and that small incentives in the form of
soft drinks, sweets and snacks would also be offered at the end of the practical study.
Participants were tested for cognitive style (in stage 2, see section 3.4) and their levels of domain
knowledge and computer experience were assessed (in stage 3, see section 3.4). This allowed the
identification of participants with appropriate cognitive style types and experience profiles. Having
secured the overall sample, participants were then assigned to one of the two HLS conditions (with and
without instructional aids). The assignment of each individual to the HLS versions was random, but it
was ensured that the sample was balanced in terms of the different experience profiles represented
under each HLS treatment. There was an equal chance of an individual being assigned to each
experimental condition (represented by the two versions of HLS), thus increasing the internal validity
of this study.
Table 1 presents a summary of the characteristics of the participants in the two versions of the HLS.
In terms of gender, in the HLS that provided no instructional aids (version 1), 42 were male and 54
were female; in the HLS that provided visual instructional aids (version 2), 50 were male and 46 were
female. With regards to age, the majority of the participants were between 18 and 23: 60 in version 1;
and 71 in version 2. There were 19 participants aged between 24 and 29 (19 used version 1 and 14
used version 2); 20 were aged between 30 and 35 (12 used version 1; and eight used version 2); and
eight were aged 36 or over (five used version 1 and three used version 2).
<<Insert Table 1 here>>
8
Data on level of education were also gathered. For version 1, 52 participants were undergraduates,
27 were taught postgraduates (Masters) and 17 were research postgraduates (PhD). For version 2, 60
participants were undergraduates, 24 were taught postgraduates and 12 were research postgraduates.
The participants were studying different subjects: business (18 in version 1; 11 in version 2);
computing and mathematics (45 in version 1; 43 in version 2); engineering and design (7 in version 1;
14 in version 2); health sciences (10 in version 1; 10 in version 2); law (five in version 1;10 in version
2); and social sciences (11 in version 1; eight in version 2). A number of the participants in the study
were registered as having dyslexia (10 who used version 1 and 14 who used version 2).
Participants’ levels of domain knowledge, computer experience and cognitive style were also
determined. In terms of domain knowledge, 96 participants were novice (defined as having low levels
of domain knowledge) and 96 were expert (high levels of domain knowledge). With regards to
computer experience, 96 had low levels of experience and 96 had high levels. Finally, in terms of
cognitive style, 96 participants were FD and 96 were FI. The different experience profiles were
assigned to the two HLS conditions to give a balanced sample in terms of these three individual
difference types.
3.3 Materials and Data Collection Instruments
To conduct the experimental study a range of materials and data collection instruments were used.
The remainder of this section describes each of the materials and instruments.
3.3.1 The subject content of the HLS
The two versions of the HLS were developed by the researchers for the sole purposes of this study
and contained the same subject-based information as the learning content of the tutorial. The learning
content covered core aspects of the Extensible Hypertext Mark-up Language (XHTML) and was
structured into seven sections: (i) introducing XHTML; (ii) how to create, save and view XHTML
documents; (iii) basic XHTML formatting: (iv) creating lists; (v) how to use images; (vi) how to insert
links within a page; and (vii) how to create tables.
9
3.3.2 HLS with visual instructional aids
A HLS incorporating visual instructional aids was designed and developed.
The visual
instructional aids incorporated into the HLS took the form of: a structural map; different colors link;
breadcrumbs; pagination; page labels; annotation of links; graphic visualization; highlighting context;
and history based mechanism. The ‘map’ (see Figure 1) helped the user to navigate through the tutorial
to find the information that they needed in order to achieve their learning goals in three ways: it
presented a global overview of the Extensible Hypertext Mark-up language (XHTML) learning
content; it offered a representation of the relationships between the different information topics
presented in the HLS; and it offered a document structure of the information presented in the HLS.
The ‘different colors’ link (as shown in Figure 1) informed users about the pages that they had
visited. This approach prevented them from repeatedly opening the same page they had visited earlier,
hence, a save of time.
<<Insert Figure 1 here>>
The ‘link annotation’ cue (as shown in Figure 2) provided users with information about each link in
relation to the XHTML tutorial. The ‘breadcrumb’ visual orientation cue (see Figure 2) helped users to
identify their current location in the HLS and the path that had led them there. The ‘pagination’ visual
cue (as shown in Figure 2) allowed users to know how many pages there were in the section, the page
that they were currently on; the number of pages that they had viewed in the section; and the number of
pages left to view. The ‘page labels’ visual cue, in the form of headings and sub-headings (see Figure
2), helped users to locate particular information in the tutorial.
<<Insert Figure 2 here>>
The ‘graphic visualization’ cue (as shown in Figure 3) showed a global overview of the information
that was presented in the HLS, and offered a conceptual structure of the information presented in the
HLS. The ‘highlighting context’ visual cue that was provided in the graphic visualization (as shown in
Figure 3) showed disabled links as differently colored nodes, with the accompanying text provided in
different font sizes, styles and colors. Using the ‘highlighting context’ approach allowed users to be
10
alerted to: (i) their current position in the HLS; and (ii) their position in relation to the overall system
structure.
<<Insert Figure 3 here>>
The ‘history-based mechanism’ (as shown in Figure 4) allowed users to view and access the last
two visited pages. With the use of this visual cue, the chance of unintentionally opening previously
visited nodes was reduced.
<<Insert Figure 4 here>>
3.3.3 HLS without instructional aids
As part of this experimental study, a HLS without instructional aids was also designed and
developed, offering a non-linear structure that could be flexibly navigated, allowing users to set their
own learning paths in order to meet their learning goals (see Figure 5). In this version of the system
only one navigation support mechanism was provided – an index (as shown in Figure 6) which showed
an alphabetic list of the system’s links in relation to the XHTML content. The users were able to
access the links in any order. The same learning content – a tutorial on Extensible Hypertext Mark-up
Language (XHTML) – was used for both versions of the HLS. The tutorial comprised seven lessons,
each with a minimum of three and a maximum of seven sections.
<<Insert Figure 5 here>>
<<Insert Figure 6 here>>
3.3.4 Cognitive Style Analysis (CSA)
To identify learners’ cognitive styles, Riding’s [42] Cognitive Styles Analysis (CSA) was
employed. The CSA includes two sub-tests. In the first sub-test, learners were presented with items
containing pairs of complex geometrical figures from which they had judge whether the figures were
similar or different. In the second sub-test, learners had to indicate whether a simple figure was
contained within a complex one. The first sub-test required field-dependent capacity whereas the
second sub-test required field independence capacity. Following Riding’s [42] classification, FM
(Field Mixed), FD and FI users were identified with the following scores: those who scored less than
1.02 were classed as FD and those who scored 1.36 and above were classed as FI. Those falling
11
between 1.03 and 1.35 were classed as Field Mixed (FM); only FI and FD users were included in the
analysis.
3.3.5 Pre-test and Post-test Questions, and Practical Task Sheet
To determine the effects of the HLS’ visual instructional aids on participants’ learning performance
(in the form of declarative and procedural knowledge), a set of instruments were used. In line with the
approach taken in other studies in the field (for example, [11]; [23]; [36]; [37]; [56]), pre-test and posttest instruments were respectively applied to measure the participants’ declarative knowledge of the
learning material (XHTML) prior to and after the use of the prescribed HLS. Given the specific
learning material addressed in the HLS, the pre- and post-test instruments had to be developed
specifically for this study. Each test contained 20 multiple-choice questions, with each question having
four possible answers and a “I do not know” option. Participants were asked to circle the answer that
they thought was correct. The answers to all of the questions were available in the tutorial. Ensuring
that the pre-test and post-test questions were comparable was achieved by either rewriting the question
and providing the possible answers in a different order or, where appropriate, by substituting different
numbers or variables into the questions.
To assess procedural knowledge, a practical task sheet was designed and developed, again
specifically for this study to address the XHTML learning content. The task sheet comprised five
questions related to the construction of web pages. The key areas addressed by the questions related to
the insertion of images and links, building tables, formatting text, creating lists, and creating, saving
and viewing a document in the browser.
3.3.6 Questionnaires
Finally, three closed-question questionnaires were used in this study.
The first collected
information on: gender; age; course area and level; and disability. Participants’ levels of computer
experience, domain knowledge in relation the tutorial’s subject content (XHTML) and familiarity with
other programming languages (such as Hypertext Markup Language (HTML), Visual Basic, C++ and
Java) was also gathered, using a three-point Likert scale (1 representing “novice”, and 3 representing
“expert”).
12
The second questionnaire presented closed choice questions to measure participants’ levels of the
four types of disorientation: (i) I know my current location in the HLS; (ii) being on the current page, I
know where I was previously in the HLS; (iii) being on the current page, I know where to go next in
the HLS; and (iv) I know how to reach my desired location in the HLS.
The third questionnaire presented closed choice questions to rate attitude towards different aspects
of the HLS: structure; navigation; overall level of disorientation; dependency on and distraction by the
visual instructional aids; and overall satisfaction. A Likert scale was used for all questions in the
second and third questionnaire, with the scale items comprising 1 (strongly disagree) to 7 (strongly
agree). Participants were required to circle the response that most closely reflected their answer to each
question.
This use of closed choice questionnaires is evident in a number of other studies in the field (for
example, [2]; [21], in relation to disorientation; and [12]; [11]; [36]; [37]; [38]; [56], in relation to
attitudes and perception). Though there are several questionnaires which gather data on disorientation,
and attitudes and perception, this study had to develop its own instantiations in order to gather data
about participants’ use and perceptions of the specific HLS used in this study.
3.4 Procedure
The experimental study was conducted over a number of sessions in a computer usability laboratory
at Brunel University. Each session contained a small group of participants, each working individually
on a PC. The experimental study was divided into nine stages:
1.
Participants were given a brief explanation of the study;
2.
Participants took the CSA test to determine their level of field dependence (classified
into FD, FM and FI) according to their CSA score. Participants identified as being
FM did not undertake the subsequent stages;
3.
Participants completed the questionnaire to determine their level of computer
expertise, experience of XHTML and other programming languages and to gather
demographic information;
4.
Participants were given a maximum of 15 minutes to complete the pre-test (marked
out of 20) to determine their prior knowledge of XHTML;
13
5.
Participants were then randomly assigned to use one of the two HLS (with visual
instructional aids; and without instructional aids) and used the assigned HLS to
complete the online instructional content of the XHTML tutorial;
6.
Participants undertook the practical task to measure their learning performance (in the
form of procedural knowledge). They were allowed to use the HLS to find the
answers and the task was marked out of 50;
7.
Participants were given a maximum of 15 minutes to complete the post-test (again,
marked out of 20);
8.
The disorientation questionnaire was administered to measure participants’ levels of
the four types of disorientation;
9.
The attitude questionnaire was administered to record participants’ attitudes towards
the HLS.
3.5 Data Analysis
As already noted, each participant’s cognitive style was determined using the CSA instrument,
categorizing participants as Field Dependent (FD) or Field Independent (FI) (see stage 2, above). The
‘domain knowledge’ variable was categorized as taking one of two values: novice – signifying low
domain knowledge; or expert – signifying high domain knowledge.
Similarly, the ‘computer
experience’ variable was categorized as taking one of two values: novice – low computer experience;
or expert – high computer experience (see stage 3, above). Thus each participant had a profile of
individual differences – with one of two cognitive style types, one of two levels of domain knowledge,
and one of two levels of computer experience – from an overall set of eight possible profiles.
These profiles were used within the grouping strategy for the three-way ANOVA [28, 43] which
was used to determine whether there was a significant effect between the three individual differences
(cognitive style, domain knowledge and computer experience) on learners’ disorientation, learning
performance and attitudes when using the two versions of the HLS: one without instructional aids and
with visual instructional aids.
In addition to the ANOVA test, the Newman Keuls post-hoc test was used to identify whether or
not there were significant differences between the sub-groups.
The data gathered from the
disorientation questionnaires (in stage 8), attitude questionnaires (in stage 9) and achievement
14
tests/practical tasks (stages 4, 6 and 7) were analyzed using the Statistical Package for Social Science
[50]. The level of significance was set at p < 0.05.
4. Findings
This section presents and considers the findings that were gathered from the experimental study, the
aim of which was to address the three research questions. To this end, data gathered from the pre- and
post-tests tests, practical tasks and closed questionnaires were analyzed.
4.1 Description of participants
192 university students participated in the study: 96 of them used the HLS that provided no
instructional aids; the remainder used the HLS that incorporated visual instructional aids.
Since this study used a three-way ANOVA, providing a focus on three individual differences –
cognitive style (FD or FI), domain knowledge (high or low DK) and computer experience (high or low
CE) (as discussed in section 3.5), a minimum of 12 participants were needed in each group to ensure a
sample that had the potential for revealing significant results (see Table 2), giving a total sample
comprising 192 participants.
Table 2: Distribution of participants according to their cognitive style, DK and CE
HLS that incorporated visual instructional
HLS that provided no instructional aids
aids
FI
FD
Total
FI
FD
Total
Low DK and low CE
12
12
24
Low DK and low CE
12
12
24
Low DK and high CE
12
12
24
Low DK and high CE
12
12
24
High DK and low CE
12
12
24
High DK and low CE
12
12
24
High DK and high CE
12
12
24
High DK and high CE
12
12
24
Total
48
48
96
Total
48
48
96
Total number of participants = 192
15
4.2 Results related to disorientation
With regards to the HLS that provided no instructional aids, the three-way ANOVA revealed a
significant effect between cognitive style, domain knowledge and computer experience in relation to
participants’ levels of the four types of disorientation in the HLS. The analysis of the Newman Keuls
post-hoc tests also revealed some significant differences between the groups. The summary results of
the ANOVA and analysis of the post-hoc tests are presented in Table 2. For the group of participants
that had low DK and low CE, low DK and high CE, or high DK and low CE, cognitive style had a
significant impact. FD users with one of these experience profiles experienced higher levels of all four
types of disorientation than did FI users with the same experience profile. This is argued to be a major
finding of the study, though there are important issues relating to interpretation of this finding which
will be discussed in section 7.
<<Insert Table 2 here>>
However, for the group of participants that had high DK and high CE, cognitive style did not have a
significant effect: neither FD nor FI users in this group experienced higher levels of any of the four
types of disorientation in the HLS.
In the version of the HLS that incorporated visual instructional aids, the three-way ANOVA
identified no significant effect in relation to cognitive style, domain knowledge and computer
experience on the levels of the four types of disorientation. With regards to all four experience profiles
((i) low DK and low CE; (ii) low DK and high CE; (iii) high DK and low CE; and (iv) high DK and
high CE), cognitive style did not seem to have an impact: FD and FI users with the same experience
profile tended to agree equally with the statement that they did not experience higher levels of
disorientation in the HLS in relation to the four types.
Having considered disorientation, the next section reports the learning performance results in
relation to both versions of the HLS.
4.3 Results related to learning performance
For the HLS that provided no instructional aids, a three-way ANOVA revealed a significant effect
in relation to cognitive style, domain knowledge and computer experience on learning performance.
16
This was reinforced by the results of post hoc tests. Tables 3 summarizes the results. For the group of
participants that had either low DK and low CE, or low DK and high CE, cognitive style showed a
significant impact: FD learners with one of these experience profiles scored lower in the post-test and
in the practical task, achieved lower test gain scores (the difference between pre-test and post-test
scores), and took more time completing the tutorial and the practical task compared to FI learners with
the same experience profile.
<<Insert Table 3 here>>
However, for the group of participants that had either high DK and low CE, or high DK and high
CE, cognitive style showed no significant impact: both FD and FI learners with the same experience
profile performed equally well in the post-test and in the practical task, achieved lower gain scores, and
took almost the same amount of time completing the tutorial and the practical task.
With regards to the HLS that incorporated visual instructional aids, a three-way ANOVA revealed
no significant effect between cognitive style, domain knowledge and computer experience on learning
performance. For all four experience profile groups ((i) low DK and low CE; (ii) low DK and high CE;
(iii) high DK and low CE; and (iv) high DK and high CE), both FD and FI users with the same
experience profile performed well in the post-test and practical task, and took broadly the same amount
of time to complete the tutorial. Additionally, for the group of participants that had either low DK and
low CE, or low DK and high CE, both FD and FI users with the same experience profile achieved
higher gain scores. For the group of participants that had either high DK and low CE, or high DK and
high CE, both FD and FI users with the same experience profile achieved lower gain scores.
However, with respect to the length of time taken to complete the practical task, the results from the
three-way ANOVA revealed a significant result when considering cognitive style, domain knowledge
and computer experience.
Analysis of the post-hoc tests identified some significant differences
between the groups (see Table 4 for summary results). For the group of participants that had either low
DK and low CE, or low DK and high CE, cognitive style had a significant impact: FD users with one
of these experience profiles spent more time completing the practical task than FI users with the same
experience profile. However, for the group of participants that had either high DK and low CE, or high
DK and high CE, cognitive style did not have a significant impact, with both FD and FI users with the
17
same experience profile spending almost the same amount of time to complete the practical task as
each other.
<<Insert Table 4 here>>
Having considered the results of the ANOVA in relation to learning performance, the next section
reports the findings gathered from the attitudes questionnaires.
4.4 Results related to attitude
For the HLS that provided no instructional aids, the three-way ANOVA identified a significant
effect between cognitive style domain knowledge and computer experience on attitudes towards
structure, navigation, overall experience of levels of disorientation, and overall satisfaction. Analysis
of the post-hoc tests also showed significant differences between the groups.
Table 5 shows a
summary of the ANOVA and post-hoc tests results.
<<Insert Table 5 here>>
For participants with experience profiles of low DK and low CE, low DK and high CE, and high
DK and low CE, cognitive style had a significant impact: FD users with one of these experience
profiles showed more negative attitudes towards the structure, had more difficulties navigating through
the HLS and disliked the high levels of freedom of navigation that were offered in the HLS, reported
much higher levels of disorientation in the HLS, and showed greater dissatisfaction learning in the HLS
compared to FI users with the same experience profile. However, with regards to those users with high
DK and high CE, cognitive style showed no significant impact. Both FD and FI users with this
experience profile showed a positive attitude towards the structure, had no difficulties navigating
through the HLS and enjoyed the higher levels of freedom of navigation offered in the HLS, did not
experience disorientation, and reported being satisfied learning in the HLS.
In the version of the HLS that incorporated visual instructional aids, the three-way ANOVA
revealed no significant effects in relation to cognitive style, domain knowledge and computer
experience on participants’ attitudes towards the HLS in terms of their views of structure, navigation
(and navigation efficacy), levels of disorientation, distraction by the visual instructional aids, and
18
overall satisfaction. With regards to participants in all four experience profile groups ((i) low DK and
low CE; (ii) low DK and high CE; (iii) high DK and low CE; and (iv) high DK and high CE), cognitive
style had no significant impact: all participants showed a positive attitude towards the structure;
reported that they did not experience difficulties navigating through the HLS; tended to disagree
equally strongly with the statement that they experienced high levels of disorientation in the HLS;
responded that they were not distracted by the visual instructional aids when performing their learning
in the HLS; and reported being satisfied learning in the HLS.
However, the three-way ANOVA revealed a significant effect on the views related to dependency
on the visual instructional aids. Additionally, the analysis from the post-hoc tests identified some
significant differences between the groups. The summary results of the ANOVA and post-hoc tests in
relation to this issue are presented in Table 6. For the groups of participants that had low DK and low
CE, low DK and high CE, or high DK and low CE, cognitive style had a significant impact: FD
learners with one of these experience profiles depended more on the visual instructional aids to reduce
their levels of disorientation in the HLS than did FI learners with the same experience profile.
For the group of participants that had high DK and high CE, cognitive style showed no significant
impact, with both FD and FI learners with this experience profile responding that they did not depend
on the visual instructional aids to reduce disorientation in the HLS.
<<Insert Table 6 here>>
The analysis of the three-way ANOVA also found a significant effect between cognitive style,
domain knowledge and computer experience on the attitudes towards the levels of freedom of
navigation in the HLS. Analysis of the post-hoc tests identified significant differences between the
groups. The summary results of the ANOVA and the post-hoc tests in relation to this issue are
presented in Table 7. Similar results gathered in the version of the HLS that provided no instructional
aids were identified.
<<Insert Table 7 here>>
19
5. Discussion
This study aimed to examine the effects of and between cognitive style, domain knowledge and
computer experience in relation to learners’ disorientation, learning performance and attitudes when
using a HLS that provided no instructional aids and when using a HLS that incorporated visual
instructional aids (see research questions 1-3).
Before moving on, it is important to restate the
complexity of the findings and the limitations in inferences that can be drawn from them, given the
number of variables being considered.
Drawing on the preceding data analysis, this section seeks to identify the effects of the individual
differences when using different navigational support mechanisms, rather than trying to prove that
specific combinations of individual differences caused the effects. This may seem a semantic point,
but it is important. The focus is on gaining insights from the data that can have practical value for
designing systems which better support learners. As such, after the discussion, the paper will draw on
the findings to frame support for designers of such systems, with evidence that they will be helpful to
specific user groups while not hindering others. This section will be structured in relation to the three
research questions posed in section 2. A summary of the overall findings are presented in Table 8, for
the version of the HLS without visual aids, and Table 9, for the version with visual aids, for ease of
reference.
<<Insert Table 8 here>>
<<Insert Table 9 here>>
5.1 Research question 1
With regards to research question 1 and in relation to the HLS that provided no instructional aids,
the analysis of the disorientation questionnaires mostly suggested that there was a significant effect
between cognitive style, domain knowledge and computer experience on disorientation. Cognitive
style had a significant impact on learners’ disorientation for the groups of participants that had low DK
and low CE, low DK and high CE, or high DK and low CE. FD users with one of these experience
profiles experienced higher levels of disorientation in the HLS than did FI users with the same
experience profile, making it a relevant issue in HLS design. However, care must be exercised in
20
relation to this finding as there are alternative interpretations (other than there being an ordinal
interaction) which could be valid. This issue will be returned to in section 7.
Moving on to the remaining group, cognitive style did not have a significant impact for those users
with an experience profile of high DK and high CE, with both FD and FI users in this group
disagreeing with the statement that they experienced disorientation problems in the HLS. It is argued
that this is because their prior knowledge of the learning content of ‘XHTML’ and their expertise in
using computers supported them in navigating effectively through the HLS. Thus, when considered in
relation to other individual differences such as domain knowledge and computer experience, cognitive
style does not always influence disorientation.
In the version of the HLS that incorporated visual instructional aids – the map and set of visual cues
introduced earlier in the paper – the analysis of the disorientation questionnaires revealed no significant
effects between cognitive style, domain knowledge and computer experience in relation to
disorientation. For all four experience profile groups, cognitive style had no significant impact on
disorientation in the HLS, with neither FD nor FI users with the same experience profile agreeing with
the statement that they experienced high levels of disorientation. However, the analysis of the attitudes
questionnaires suggested that cognitive style had a significant effect on the use of the visual
instructional aids to reduce disorientation in the HLS. For the group of participants that had low DK
and low CE, low DK and high CE, or high DK and low CE, FD users with one of these experience
profiles depended more on the visual instructional aids to reduce disorientation in the HLS than did FI
users with the same experience profile. The study’s results suggest that while FD users with one of
these experience profiles are totally dependent on the visual instructional aids, this is not the case for FI
users with the same experience profile – that is, they are not totally dependent on these visual
instructional aids to reduce disorientation problems in the HLS. For those users with an experience
profile of high DK and high CE, cognitive style did not have a significant impact on the use of the
visual instructional aids in order to reduce disorientation problems in the HLS, with neither FD nor FI
users in this group being dependent on them. This study has shown that when considering cognitive
style, domain knowledge and computer experience together, it is not always the case that cognitive
style will influence the use and impact of visual instructional aids to reduce disorientation problems.
21
5.2 Research question 2
With regards to research question 2, and in relation to the HLS that provided no instructional aids,
the analysis of the achievement tests (post-test scores, test gain scores, practical task scores and time
efficacy) suggested that there was a significant effect between cognitive style, domain knowledge and
computer experience in relation to learning performance. For participants with experience profiles of
either low DK and low CE, or low DK and high CE, cognitive style had a significant impact, with FD
users with one of these experience profiles performing less well in the post-test and in the practical
task, achieving lower gain scores, and taking more time to complete the tutorial and the practical task
than FI users with the same experience profile. However, care should be exercised in relation to this
finding as the extent to which experience level and/or prior domain knowledge accounted for the
variance may mean that there is too little left for analysis of other interaction effects. This issue,
effectively one of statistical power, will be returned to in section 7. If, though, cognitive style is
significant in this context, it raises a concern in relation to learning performance for FD users in these
experience profile groups.
With regards to those users with experience profiles of either high DK and low CE, or high DK and
high CE, cognitive style revealed no significant effects, with both FD and FI users with the same
experience profile performing equally well in the learning tasks. This is to be expected given their high
domain knowledge prior to the use of the tutorial, as reflected in their higher pre-test scores. This
study has shown that when domain knowledge is high, irrespective of cognitive style, computer
experience and the mode of learning system used, learning performance will not be hindered. This
insight may help designers to gain an improved understanding of when and why cognitive style will not
influence learning performance.
In terms of the HLS that incorporated the visual instructional aids, for the group of participants that
had low DK and low CE, or low DK and high CE, cognitive style showed no significant effects on
learning performance in terms of post-test scores, test gain scores, practical task scores and time
efficacy (time taken to complete the tutorial), with both FD and FI users with the same experience
profile performing equally well in the post-test and in the practical task, achieving higher gain scores
and taking broadly the same amount of time to complete the tutorial. However, cognitive style had a
significant impact on time efficacy (the time taken to complete the practical task) for the learners who
had one of these two experience profiles (that is, low DK and low CE, or low DK and high CE): FD
22
learners with one of these experience profiles spent more time completing the practical task than FI
learners with the same experience profile. The study’s results suggest that, in general, the visual
instructional aids seemed to have supported the FD users with one of these two experience profiles in
reducing their higher levels of disorientation. This, in turn, led them to learn as effectively as FI users
with the same experience profile, in contrast to the findings for the same experience profile groups
using the version of the HLS that provided no instructional aids. These results suggest that the
relationship between HLS, visual instructional aids, individual differences and learning performance
needs to be carefully considered in the design of HLS.
Finally, for the participants that had either high DK or low CE, or high DK and high CE, similar
findings were identified in both versions of the HLS (i.e., with and without instructional aids) in
relation to cognitive style (as discussed earlier in this section).
These results suggest that when
considering cognitive style, domain knowledge and computer experience together, cognitive style will
not always influence the way that visual instructional aids are used to enhance a user’s learning
performance in the HLS.
5.3 Research question 3
With respect to research question 3 and in relation to the HLS that provided no instructional aids,
the analysis of the attitude questionnaires suggested a significant effect between cognitive style,
domain knowledge and computer experience in relation to learners’ attitudes. For the groups of
participants that had low DK and low CE, low DK and high CE, or high DK and low CE, cognitive
style showed a significant impact on attitudes: FD users with one of these experience profiles showed
more negative attitudes towards the HLS in terms of their views of structure, navigation and overall
satisfaction than did FI users with the same experience profile. This finding is supported by the earlier
discussion in relation to research questions 1 and 2, which suggested that FD users with one of these
three experience profiles were less comfortable learning in the version of the HLS that provided no
instructional aids than were FI users with the same experience profile. It seems that, unlike FI users
with one of these experience profiles, the FD users with same experience profile have difficulties
imposing a structure on the learning content or mapping a mental representation of the information that
is presented in the HLS to enhance their learning. This raises a concern in relation to attitudes for FD
users in these groups, and is an issue when seeking to develop guidance for HLS design. Finally, with
23
regards to those users with an experience profile of high DK and high CE, cognitive style did not have
a significant impact on attitudes, with both FD and FI users in this group showing a positive attitude in
terms of their views of the structure of, navigation in, and overall satisfaction with the HLS. It is
argued that this could be because their prior knowledge of the learning content of ‘XHTML’ and of
using computers permitted the users in this group, irrespective of their cognitive style, to be
comfortable learning in a non-linear learning environment.
This study has shown that when
considering other individual differences, for example domain knowledge and computer experience,
cognitive style will not always influence attitudes in HLS, with FD and FI users with an experience
profile of high DK and high CE showing the same preference for non-linearity features in the HLS.
This makes it an important area to consider in the HLS design.
With regards to the HLS that incorporated visual instructional aids, for participants in all four
experience profile groups, cognitive style did not have a significant impact on attitudes: both FD and FI
users with the same experience profile showed a positive attitude towards the HLS, responded that they
were not distracted by the visual instructional aids, and reported being satisfied learning in the HLS.
However, for the group of participants that had low DK and low CE, low DK and high CE, or high DK
and low CE, cognitive style showed a significant effect on the use of the visual instructional aids to
reduce levels of disorientation in the HLS. FD users with one of these experience profiles were more
dependent on the visual instructional aids to reduce their levels of disorientation in the HLS than were
FI users with the same experience profile. The visual instructional aids may have assisted the FD users
in these experience profile groups in reducing their levels of disorientation, and in turn, to learn
effectively in the HLS. This, in turn, may have led them to show a positive attitude towards this
version of HLS. The study’s results suggest that FD users in these experience profile groups must be
provided with guidance in the form of visual instructional aids, for example maps and visual
orientation cues in the HLS, otherwise they may show less interest learning in this non-linear learning
environment. For those users with high DK and high CE, cognitive style did not have a significant
effect, with neither FD nor FI users with this experience profile being dependent on the visual
instructional aids to reduce disorientation in the HLS.
6. Development of HLS design guidelines
In order for these findings to be useful in informing the design of HLS, it is important that they are
captured in a way that makes them accessible. As such, this section will present design guidelines that
24
draw on the findings and seek to capture the issues in a form that is useful to, and usable by, HLS
designers. Particular attention is paid to identifying issues which, on the basis of the findings and their
analysis, offer scope to support particular groups of users while not hindering other groups – in other
words they identify factors that can positively support some users while being benign for others. As
such, the guidelines seek to be generally applicable to HLS.
Based on the discussions of the findings from this study (as presented in preceding section),
designers need to be aware that while some users (for example FI users with low DK and low CE, low
DK and high CE, or high DK and low CE; and FD/FI users with high DK and high CE) have no
difficulties learning in HLS, others (for example FD users with low DK and low CE, low DK and high
CE, or high DK and low CE) struggle to learn effectively and, in turn, demonstrate negative attitudes
towards non-linear learning environments. The FD users with one of these experience profiles (low
DK and low CE, low DK and high CE, high DK and low CE) are largely dependent on learning
guidance in the form of visual instructional aids to perform their learning in HLS. However, the results
of this experimental study suggest that including these visual instructional aids will not have
detrimental effects for those users who are less dependent on them or who do not need them at all to
perform their learning in HLS (for example, FI with low DK and low CE, low DK and high CE, or high
DK and low CE; FD/FI with high DK and high CE). The remainder of this section will suggest ways
in which HLS may be designed to support FD users with one of the following experience profiles – low
DK and low CE, low DK and high CE, or high DK and low CE – to learn effectively in HLS but
without disturbing the other users (FI with low DK and low CE, low DK and high CE, or high DK and
low CE; FD/FI with high DK and high CE) who do not require additional support in using HLS.
6.1 Provide conceptual support in the form of graphical maps
FD users with experience profiles of low DK and low CE, low CE and high CE, or high CE and low
CE, experience higher levels of disorientation in HLS because they: have difficulty imposing a
structure on the learning content that is presented in the HLS; or have difficulty mapping a mental
representation of all the information that is presented in the HLS. In order to help FD users to reduce
these problems in HLS, designers need to provide visual instructional aids in the form of a graphical
map on every page.
The graphical map will help the learners to see a global overview of the
information (in the form of nodes/links) that is represented in the HLS. Additionally, it will help the
25
learners to gain a representation of the relationships between the different pieces of information
presented in the HLS. The conceptual structure of the information presented by the graphical map can
further assist them in navigating through the HLS to reach the information that they need in relation to
their learning goals. Referring to the graphical map will make these FD users confident that they are
successfully completing all of the learning materials in the HLS which are needed to achieve their
learning goals. This, in turn, can help them not only to complete all of the topics, but also to integrate
their knowledge so as to complete the tutorial effectively.
The map can also support FD users with high DK and low CE to navigate the tutorial to access
content in accordance with their information needs and navigation preferences, but without them being
disorientated in the non-linear environment. However, care does need to be taken when presenting the
graphical map on the same page where the learning content is presented as this can create two issues:
first, the graphical map may take up a lot of space on the page, which may prevent the learning content
from being clearly visible on the screen; and second, the graphical map may disturb other users who do
not need it (for example – FI users with low DK and low CE, low DK and high CE, or high DK and
low CE; FD/FI users with high DK and high CE ). In order to reduce the impact of these two issues,
zoom-in and zoom-out features can be added to the visual representation.
FD users with low DK and low CE, low DK and high CE, or high CE and low CE may still
encounter higher levels of disorientation if they do not know the following when they navigate through
the HLS: (1) where are they?; (2) what have they visited?; (3) where have they been (including
repeatedly visited pages) and how can they go back to these visited pages?; and (4) where to go next?
These issues may be particularly acute when high levels of freedom of navigation have been provided,
and when the tutorial consists of a large number of main sections, sub-sections and even sub-sub
sections. Maps may not reduce all of these four types of disorientation, making it an issue in relation to
the design of the HLS. It is therefore imperative to present and consider other methods that can be
applied to reduce these four types of disorientation problems in the design of the HLS.
6.2 Provide ‘breadcrumbs’ and ‘highlighting context’ visual cues to reduce the first type of
disorientation – ‘where are they’?
With respect to the first type of disorientation (‘where are they?’), FD users with experience
profiles of (i) low DK and low CE, (ii) low DK and high CE, or (iii) high DK and low CE) sometimes
26
have difficulties finding: (a) how far they have navigated in the HLS to their current location in the
HLS; and (b) their location within a specific section in the tutorial. In terms of the first issue,
‘breadcrumbs’ can be used as a visual cue to show these FD users how they reached their current
location in the HLS. For this to be effective, the link showing the current page should be disabled and
presented in a different color text. Placing the ‘breadcrumbs’ above the page title can clearly attract the
attention of these FD users. Additionally, a different background color can be used to differentiate the
‘breadcrumb’ visual cue from the learning content on the page. With regards to the second issue, one
solution is to clearly identify the learner’s current position on the graphical map – for example by
disabling and differently coloring the ‘current’ node and using a ‘hand’ icon with the accompanying
text ‘you are here’ to point to the node on the graphical map.
6.3 Use ‘check mark’ or ‘different link colors’ visual cues to reduce the second type of disorientation –
‘what have they visited?’
The second type of disorientation – ‘what have they visited?’ – is mostly experienced by FD users
with low DK and low CE, or low DK and high CE, who find it difficult to impose a structure on the
learning content in the HLS. A consequence of this disorientation problem is that the FD users in these
experience profile groups may open unnecessary nodes or fail to complete effectively all of the
learning materials needed to achieve their overall learning goals. One remedy for this issue is to
provide a check mark to indicate the pages that the user has visited in the HLS. Such a check mark (in
the form of a ‘tick’ symbol, for example) can be displayed inside the nodes (which represent the
information that is presented in the HLS) in the graphical map. Additionally, different colors could be
used for the tick symbols to differentiate the latest page from the previous pages visited. The tick
symbol should be of reasonable size, so that it is clearly visible inside the nodes in the graphical map.
If this approach is adopted, designers need to tell users, especially those with low CE, what the tick
symbol/colors refer to by, for example, annotating the symbols (activated when the mouse is rolled
over them).
Providing different link colors is another way of informing the FD users in these
experience profile groups of the pages that they have already visited.
6.4 Use the ‘history-based mechanism’ cue to reduce the third type of disorientation – ‘where have
they been and how can they go back to these destinations?’
27
The third type of disorientation is mostly experienced by FD users with low DK and low CE, low
DK and high CE, or high DK and low CE. FD users with one of these experience profiles have
difficulties finding where they have recently been in the HLS (including the pages that they have
opened more than once), especially on the rare occasions that they use a non-linear approach in relation
to their learning goals. This makes it harder for them to go back to these previously visited pages if
they need to. One way to reduce this problem is to provide a history-based mechanism. Through the
use of this visual technique, the FD users in these experience profile groups will be able to view a list
of the pages that they have visited in the HLS. Additionally, links could be provided to the visited
pages to help the FD users to access them directly should they need in order to accomplish their
learning goals. Designers will need to be aware, however, that displaying all of the visited pages
(including those that have been visited repeatedly) on one page may lead to space and scrolling
problems. A compromise may be to display, say, only the last 10 visited pages.
Even if this compromise approach is adopted, designers will need to be cautious about the historybased mechanism being displayed on the same page as the learning content because it may cause two
issues: first, the learning content may not be clearly visible, which may prevent some users from
learning effectively; and second, it may irritate other users, for example FD/FI users with high DK and
high CE, and FI users with low DK and low CE, low DK and high CE, or high DK and low CE. In
order to reduce these two issues, the history-based mechanism cue could be opened in a new pop-up
window, or zoomed in and out on the same page where the learning content is displayed.
6.5 Use ‘page labels’ and the ‘direct guidance’ visual cue to reduce the fourth type of disorientation –
‘where to go next?’
The fourth type of disorientation –‘where to go next in the HLS’ – is a common problem that faced
by some FD users, especially those with low DK and low CE, or low DK and high CE. One way to
help these FD users is to provide page labels in the form of headings and sub-headings on a page. This
can help these learners to take coherent paths in order to successfully complete their current learning
goals. Using different font sizes, styles and colors for the text that is used for the page labels will help
these users to differentiate the learning content from the page label visual cues. Additionally, the font
size that is used for the labels should not be too large otherwise the labels may take up too much space
on the page, causing a disturbance to other users who do not use them.
28
Another approach to alleviating the fourth type of disorientation is to provide direct guidance. A
pagination facility (which shows the current page) can be displayed at the bottom of the learning
content along with two arrow links: the backward arrow link to its left, and the forward arrow link to its
right. The backward arrow link will direct these FD users one page back from the current one, whereas
the forward arrow link will direct users one page ahead. Using this mechanisms to allow navigation
only one page either ‘side’ of the current page makes sure that these users will be opening appropriate
pages in relation to their current learning goals and should help alleviate disorientation.
7. Limitations of the study and future research
In attempting to answer the three research questions posed in this paper, certain issues may have
limited the generalizability of this study’s findings. The first limitation relates to the sample used in
the study, which was made up of university students. They were considered to be a suitable sample
group because the research relates to HLS and Higher Education and because most existing studies in
the area have also used university students as their subjects. It is important to note that university
students may be more educated than the general population, so the findings related to the use of the two
versions of the HLS may have been different had the sample been drawn from the general population.
To overcome this shortcoming, the study could be replicated using a sample drawn from this wider
group. This may encourage researchers to consider the effects of an even wider range of individual
differences alongside cognitive style, domain knowledge and computer experience. Other factors such
as age, educational level, and ability to use computers and their applications may prove worthwhile to
study.
Within the choice of university students (as part of the study’s purposive sampling approach), the
decision to recruit participants from only the London region may be seen as a limitation associated with
the sample. It is not necessarily the case that students from this region are representative of university
students nationally, although there is no specific bias of which we are aware in taking such a regional
recruitment approach. Since the study’s focus was on, and classified the participants in terms of,
cognitive style, prior knowledge and experience, the limiting of the sample to participants from the
London region should not have made a meaningful difference as the three different variables – and
levels within them – were appropriately represented in the sample. Conducting further studies with
29
university students from across the country would, however, address any criticism in this respect and
aid confidence in the generalizability of the findings to the wider UK HE student population.
It may also be useful to conduct comparable studies using different learning content in the HLSs.
Although the content in the tutorial reported in this study covered, for example, conceptual and
procedural information and we see no immediate reason why the choice of content should limit the
applicability of the findings to other subject domains, studies that attempt to replicate the findings with
other subject content structured and presented in the same way as for this study’s tutorial may be useful
to demonstrate generalizability.
A second limitation of this research is that, in both versions of the HLS, the XHTML learning
content was presented and explained using text and images. Some of the participants in this study were
registered as having dyslexia, and their presentation of the learning content may have hindered their
learning performance and may have caused them to show less interest in the HLS. The findings related
to learning performance and attitudes may have been different had an alternative approach to overcome
presenting aspects of the learning content (for example, the use of audio) been employed.
To
overcome this limitation, further studies could be designed to examine the effects of audio on
disorientation, learning performance and attitudes in relation to the same set of individual differences
considered in this study, but with a sample of learners who have dyslexia. The results may help to
identify whether using audio supports these learners in overcoming the issues associated with their
dyslexia.
It is also important to note the complexities in interpreting ordinal interactions identified in studies
such as the one reported in this paper. The interpretation of an identified ordinal interaction in a study
of individual differences is always complex and challenging, because the counterfactual is not clear,
making any conclusion drawn vulnerable to alternative interpretations. In this study, the suggested
ordinal interaction between cognitive style and disorientation in the HLS with no instructional aids for
three of the four experience profile groups (and the mitigation of the disorientation effects seen for the
same groups in the HLS when instructional aids were provided) may be down to factors other than
cognitive style. However, as noted in section 3.1, we undertook additional qualitative data collection
and analysis as part of the study and the findings here supported the interpretation of the ordinal
interaction. Whilst. these data cannot be fully reported given the space constraints of this paper, the
30
support from the qualitative data analysis gives us confidence in the interpretation of the data as
showing an ordinal interaction.
A further issue, relevant to the findings related to cognitive style and learning performance reported
in section 4.3 and discussed in section 5.2, is the strength of evidence and the threats to the validity of
inferences drawn from the data analysis. Since third order interaction was being explored in the study,
the issue of research power is exacerbated and it is fair to question whether the size of the sample was
large enough to give weight to the finding with respect to learning performance. Repeating the study
with a larger sample would be one way to address this concern about whether the study was
‘underpowered’ and to improve confidence in the validity of the findings.
Strength of evidence issues are ones that, given the methodological limitations – and overall
complexity – of designing and running studies of this kind, must be acknowledged. The findings
presented in this paper must, as a result, be viewed with caution, as the discussion in this section has
stressed.
An associated, wider limitation of the study relates to the complexity of the issues that it seeks to
explore. Experimental studies of three individual differences as independent variables in a single study
are rare. This is not surprising as the complexity of the design and analysis – and the care and caution
with which any results should be interpreted, and the scope for them to be criticized – perhaps makes
such studies unappealing to researchers. However, studies that concentrate only on a single individual
difference (often without controlling for others that have been argued to be important) also run the risk
of generating findings that have limited value or are open to strong challenge. We recognize the
methodological complexity and associated analytical difficulties associated with this study, and the
associated criticisms that can be levelled in relation to the strength of evidence produced, but feel that
seeking to overly simplify studies of this type also creates associated issues such as those noted above.
Providing full details of the approach that we have taken allows researchers to assign weight to the
findings based on their analysis of the study’s design and execution. We hope that it encourages others
to undertake similarly complex studies and add to the corpus of evidence in the field.
8. Conclusion
This paper has examined the effects of and between cognitive style, domain knowledge and
computer experience in relation to disorientation, learning performance and attitudes in two versions of
31
HLS: one that provided no instructional aids and one that incorporated visual instructional aids (in the
form of a map and a set of visual cues). Based on the analysis of the data gathered in the study, this
paper has framed a set of guidelines for the use of visual instructional aids in the design of the HLS that
aim to support designers in building systems that reduce disorientation, support effective learning and
enhance learning satisfaction, but do not have detrimental effects on those users who do not depend on
the additional support of visual aids in using HLS. Additionally, the identified limitations of this study
– and the caution raised over the interpretation and generalizability of the findings – are argued to be
useful for identifying areas of future research.
References
[1] Aery, Sean C. Breadcrumb Navigation Deployment in Retail Web Sites. A Master’s Paper for the
M.S. in I.S. Degree. July, 2007.
[2] Ahuja, J.S. and Webster, J. Perceived disorientation: an examination of a new measure to assess
web design effectiveness. Interacting with Computers, 2001. 14(1): p. 15-29.
[3] Alomyan, H. and Au, W. Exploration of instructional strategies and individual difference within
the context of web-based learning, International Education Journal, 4 (4), (2004), 86-92.
[4] Amadieu, F., Tricot, A., and Mariné, C. Individual differences in learning from hypermedia:
Learners’ characteristics to consider to design effective hypermedia. 6th International Conference
on Human System Learning - ICHSL, 14-16 may, Toulouse, (Toulouse: IEEE France Section),
(2008), pp. 1-9.
[5] Amadieu, F., Tricot, A., and Mariné, C.
“Prior Knowledge in Learning from a Non-Linear
Electronic Document: Disorientation and Coherence of the Reading Sequences.” Computers in
Human Behavior 25: (2009a), 381-88.
[6] Amadieu, F., Gog, F., Paas, F., Tricot, A., and Mariné. C. Effects of prior knowledge and conceptmap structure on disorientation, cognitive load, and learning. Learning and Instruction,
19 (2009b), pp. 376–386.
[7] Amadieu, F., Tricot, A., and Mariné, C. Exploratory Study of Relations between Prior Knowledge,
Comprehension, Disorientation and On-line Process in Hypertext. The Ergonomics Open Journal,
9, (2009c), pg 49-57.
32
[8] Amadieu, F., Tricot, A. and Mariné, C. Interaction Between Prior Knowledge And Concept-Map
Structure on Hypertext Comprehension, Coherence of Reading Orders and Disorientation.
Interacting with Computers, 22 (2), (2010), 88-97.
[9] Brusilovsky, P., Sosnovsky, S. and Yudelson, M. Addictive links: The motivational value of
adaptive link annotation in educational hypermedia. In: V. Wade, H. Ashman and B. Smyth (eds.)
Proceedings of 4th International Conference on Adaptive Hypermedia and Adaptive Web-Based
Systems (AH'2006), Dublin, Ireland, June 21-23, Springer Verlag, (2006), pp. 51-60.
[10] Calcaterra, A., Antonietti, A., and Underwood, J. Cognitive style, hypermedia navigation and
learning. Computers and Education. 44, 4, (2005), 441–457.
[11] Chen, S. Y. Evaluating the learning effectiveness of using Web-based instruction: An individual
differences approach. International Journal of Information & Communication Technology
Education, 1(1), (2005), 69–82.
[12] Chen S.Y. and Macredie R.D. Cognitive modelling of student learning in web-based instructional
programmes. International Journal of Human-Computer Interaction 17, (2004), 375–402.
[13] Chen, S. Y., Fan, J. and Macredie, R. D. Navigation in Hypermedia Learning Systems: Experts
vs. Novices. Computers in Human Behavior. 22(2), (2006), 251-266.
[14] Chen, N.S., Kinshuk, Wei, C.W and Chen, H.J. Mining e-learning domain concept map from
academic articles. Computers & Education, 50, 3, (2008), pp. 1009–1021.
[15] Chen, S. Y. and X. Liu. Mining students' learning patterns and performance in Web-based
instruction: a cognitive style approach, Interactive Learning Environments, 19:2, (2011), 179-192.
[16] Chen L-H. Web-based learning programs: use by learners with various cognitive styles. Comput
Educ 54: (2010), 1028–1035.
[17] Chou, H. W.
Influences of cognitive style and training method on training effectiveness.
Computers and Education, 37, (2001), 11–25
[18] Chung, P. H. Changing the Interface with Minimal Disruption: The Roles of Layout and Labels.
Doctoral dissertation, Rice University, Houston, TX, (2006).
[19] Davidovic, A., Warren, J., Trichina, E. and Lakes, M. Learning benefits of structural examplebased adaptive tutoring systems, IEEE Trans Educ 46 (2003), 241_251.
33
[20] De La Passardiere, B. and Durfresne, A. Adaptive navigational tools for educational hypermedia
[On-line].
Computer
Assisted
Learning.
Available:
http://mistral.ere.umontreal.ca/_dufresne/Publications/ical92.htm, (1992).
[21] Demirbilek, M. Effects of interface windowing modes and individual differences on disorientation
and cognitive load in a hypermedia learning environment. PhD. Dissertation, University of
Florida, (2005).
[22] Dringus, L.P. Towards active online learning: A dramatic shift in perspective for learners. The
Internet and Higher Education, 2(4), (2002), 189-195.
[23] Fan, J. P. Interface design for hypermedia learning systems: a study of individual differences and
hypermedia system features. Ph.D. dissertation, Brunel University, West London, UK, (2005).
[24] Fan, J.P. and Macredie, R.D. Gender Differences and Hypermedia Navigation: Principles for
Adaptive Hypermedia Learning Systems, Idea Group Publishing, Hershey, PA, (2006).
[25] Gerjets, P., Scheiter, K., Opfermann, M., Hesse, F. W. and Eysink, T. H. S. Learning with
hypermedia: The influence of representational formats and different levels of learner control on
performance and learning behavior. Computers in Human Behavior, 25, (2009), 360–367.
[26] Graf, S., Liu, T.-C. and Kinshuk. Analysis of learners’ navigational behavior and their learning
styles in an online course. Journal of Computer Assisted Learning, 26, (2010), 116-131.
[27] Hua, Q. and Pei-Luen, P.R. Method for Reducing Disorientation in Hypermedia Educational
systems. Tsinghua Science and Technology, Volume 14, Issues 5, (2009), 655-662.
[28] Karpinski. Factorial ANOVA (chapter 8): Higher Order ANOVA, (2006).
[29] Khalifa M. and Lam R. Web-based learning: effects on learning process and outcome. IEEE
Transactions on Education, 45, (2002), 350-356.
[30] Khosrowjerdi, M. and Iranshahi, M. Prior Knowledge and information-seeking behavior of PhD
and MA. Library and Information Science Research, In Press, Corrected Proof, (2011).
[31] Lajoie, S.P. and Azevedo, R.
Teaching and learning in technology-rich environments, P.
Alexander, P. Winne, Editors, Handbook of Educational Psychology, (2nd ed.), Erlbaum, NJ,
(2006), pp. 803–821.
34
[32] Lee, H. M., Cheng, Y. W., Rai, S. and Depickere, A. What affect student cognitive style in the
development of hypermedia learning system? Computers and Education, 45, (2005), 1–19.
[33] Lin, H. and Chen, L.
Discovering learning pattern in different cognitive style of learners.
Proceedings of the Third International Conference on Convergence and Hybrid Information
Technology, 2, (2008), 268–273.
[34] Minetou, C. G., Chen, S. Y. and Liu, X. Investigation of the use of navigation tools in web-based
learning: A data mining approach. International Journal of Human–Computer Interaction, 24(1),
(2008), 48–67.
[35] Mishra, P. and Yadav, A. Using hypermedia for learning complex concepts in chemistry: a
qualitative study on the relationship between prior knowledge, beliefs, and motivation. Education
and Information Technologies 11 (1), (2006), 33–69.
[36] Mitchell, T. J. F., Chen, S. Y. and Macredie, R. D. Cognitive Styles and Adaptive Web-based
Learning. Psychology of Education Review. 29(1), (2005a), 34-42.
[37] Mitchell, T. J. F., Chen, S. Y. and Macredie, R. D. Hypermedia learning and prior knowledge:
Domain expertise vs. system expertise. Journal of Computer Assisted Learning, 21(1), (2005b),
53–64.
[38] Mitchell, T. J. F., Chen, S. Y. and Macredie, R. D. The Relationship between Web enjoyment and
students’ perceptions and learning using a Web based tutorial. Learning, Media and Technology.
30(1), (2005c), 29-42.
[39] Mueller-Kalthoff, T. and Moeller, J. The effects of graphical overviews, prior knowledge, and
self-concept on hypertext disorientation and learning achievement. Journal of Educational
Multimedia & Hypermedia, 12(2), (2003), 117-134.
[40] Mustafa, K.
Individual Learner Differences In Web-based Learning Environments: From
Cognitive, Affective and Social-cultural Perspectives. Turkish Online Journal of Distance
Education-TOJDE, Volume: 6 Number: 4 Article: 2, (2005).
[41] Myers, M.D. Qualitative Research in Business & Management. Sage Publications, London,
(2009).
35
[42] Nielsen, J. Designing web usability: The practice of simplicity. USA: New Rider Publishing,
(2000).
[43] Pallant, J. SPSS, Survival Manual, (2001).
[44] Pardue, J.H., Landry, J.P. and Kyper, K. Look ahead and look-behind shortcuts in large item
category hierarchies: The impact on search performance. Journal of Interaction with computers,
21, 4, (2009).
[45] Parkinson, A., Redmond, J.A. and Walsh, C. Accommodating field-dependence: A cross-over
study. In: The 9th Annual Conference on Innovation and Technology in Computer Science
Education (ITiCSE 2004), Association for Computing Machinery, (2004), 72–76.
[46] Riding, R. Cognitive styles analysis – CSA administration. Birmingham: Learning and Training
Technology, (1991).
[47] Riding, R., and Rayner, S.G. Cognitive styles and learning strategies. David Fulton Publisher,
London, (1998).
[48] Ross, S.M., Morrison, G.R., and Lowther, D.L. Educational Technology Research Part and
Present: Balancing Rigor and Relevance to Impact School Learning. Contemporary education
technology, 1(12), (2010), 17-35.
[49] Spyridakis, J.H., Mobrand, K.A., Cuddihy, E., and Wei, C.Y. Using structural cues to guide
readers on the Internet. Information Design Journal, 15(3), (2007), 242-259.
[50] SPSS for Windows. Release 15.0: SPSS, (2001).
[51] Su, Y. and Klein, J. D. Effects of navigation tools and computer confidence on performance and
attitudes in hypermedia learning environment. J. Educ. Multimedia Hypermedia, 15(1), (2006),
87–106.
[52] Thatcher, A. Web search strategies: The influence of web experience and task type. Information
Processing and Management, 44(3), (2008), 1308–1329.
[53] Triantafillou, E., Pomportsis, A. and Stavros, D. The design and formative evaluation of an
adaptive educational system based on cognitive styles. Computers and Education, 41, (2003), pp.
87-103
36
[54] Umar, I. N. and Maswan, S. The effects of a web-based guided inquiry approach on students’
achievement. Journal of Computers, 2(5), (2007), 38–43.
[55] Waniek, J, and Schafer, T. The role of domain and system knowledge on text comprehension an
information search in hypermedia. Journal of Educational Multimedia and, Hypermedia v18 n2,
(2009), p221-240.
[56] Wang, A. The effects of varied instructional aids and field dependence and independence on
learners’ structural knowledge in a hypermedia learning environment: Ph.D. dissertation, Chio
University, (2007).
[57] White, R. W., Dumais, S.T. and Teevan, J. Characterizing the influence of domain expertise on
Web search. In Proceedings of WSDM, (2009), 132-141.
[58] Witkin, H. A., Moore, C.A., Goodenough, D.R. and Cox, P.W.
Field-dependent and field-
independent cognitive styles and their educational implications. Review of Educational Research,
47(1), (1977), 1-64.
[59] Witkin, H.A., and Goodenough, D.R. Cognitive Styles: Essence and Origins. New York:
International Universities Press, Inc, (1981).
Figure captions
Figure 1: A screen from the HLS, showing the map and link color visual aids
Figure 2: A screen from the HLS, showing the breadcrumbs, pagination, link annotation and
page labels orientation cues
Figure 3: An example of the graphic visualization and highlighting context visual cues
Figure 4: An example of the history-based mechanism visual cue
Figure 5: A screen from the HLS showing the non-linear structure
37
Figure 6: The HLS and its index navigation tool
Table captions
Table 1: Distribution of participants according to their cognitive style and levels of DK and CE
Table 1: Characteristics of the participants and their assignment to the versions of the HLS
Table 2: Analysis of variance of levels of disorientation
Table 3: Learning performance results
Table 4: Learning performance results related to the length of time taken to complete the
practical task
Table 5: Results related to attitudes towards the HLS
Table 6: Analysis of variance on views related to dependency on the visual instructional aids
Table 7: Analysis of variance on views of levels of freedom of navigation
Table 8: High level findings in relation to the HLS that provided no instructional aids
Table 9: High level findings in relation to the HLS that provided visual instructional aids
38