User Experience Evaluation
Through The Brain’s Electrical Activity
Akshay Aggarwal, Gerrit Niezen, and Harold Thimbleby
Swansea University
Swansea, Wales, United Kingdom
666532@swansea.ac.uk, g.niezen@swansea.ac.uk, harold@thimbleby.net
ABSTRACT
A novel system for measuring the user experience of any user
interface by measuring the feedback directly from the brain
through Electroencephalography (EEG) is described. We developed an application that records data for different emotions of the user while using any interface and visualises the
data for any interval during the task, as well as presenting various statistics and insight about the data. The application also
provides the points of mouse movement on any interface as
different coloured dots, where the colour represents the mental load at those points. This makes it easier to identify the
user experience based on emotions at exact points on the user
interface.
In experiments, the brain activity of participants was recorded
while they performed tasks on both a well-designed and
poorly designed user interface. Screen and mouse cursor
position were recorded, along with the values of several facial expressions and emotions extracted from the EEG. Users
were interviewed after the study to share their experiences.
For each study session analysis was done by comparing EEG,
screen recording and interview data. Results showed that
frustration, furrow and excitement values reflect user experience.
Author Keywords
Electroencephalography; EEG; Brain Computer Interface;
User Interface; User Experience (UX).
ACM Classification Keywords
H.5.m. Information Interfaces and Presentation (e.g., HCI):
Miscellaneous
General Terms
Human Factors; Design; Measurement.
INTRODUCTION
The design of any user interface is important for the effective
interaction between the user and the service it links to. There
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from Permissions@acm.org.
NordiCHI ’14, October 26 - 30 2014, Helsinki, Finland.
Copyright 2014 ACM 978-1-4503-2542-4/14/10. . . $15.00.
http://dx.doi.org/10.1145/2639189.2639236
are different rules and principles that guide the designers to
make the system easy to use, and several methods are used to
measure the user experience of a User Interface (UI) [1, 3].
One way is to run a usability test with some target users to
observe actual behaviour in some form of controlled environment [1]. Users perform a series of tasks, describing what
they are doing while doing it, and complete pre-test and posttest questionnaires. This method is useful as the pre-task
questionnaire provides the information about the user’s previous experience and familiarity with the interfaces and the
post-task questionnaire provides details on their experience
related to the design. By asking users to explain how to perform a specific task, insight is gained in how well they understood the necessary steps involved in achieving their goal. For
accurate results the questionnaires have to be designed very
carefully, which takes a great deal of skill. Another problem
with this method is that the users might not remember the details of what happened during the task and may not address
all the issues related to the interface that were experienced
during the test.
Eye-tracking during testing provides information on what
people look at and gives researchers more insight into how
people think, based on the eye-mind hypothesis that states
that people look at what they are thinking about [3]. This assumption means that researchers need be careful about their
conclusions about the cognition behind behaviour and requires careful study design.
Another evaluation method is the semi-structured interview,
which is a semi-formal discussion about the experience of
the user. This method gives good insight, as the interviewer
can deviate where necessary to order to obtain the needed information [3]. There are some limitations to this method, for
example the user might feel the pressure of answering questions quickly and avoid taking a long pause. If the interview is
less structured, it also becomes harder to analyse afterwards.
A number of inaccuracies that occur during user testing can
be overcome with the use of “emotional markers”. One of
the most efficient and easy to use techniques to record emotional markers is the use of Electroencephalography (EEG)
— EEG technology records the voltage fluctuations from the
ionic current flowing due to neuron activity. The emotional
marker technique involves recording the user’s brain waves
while they are using the interface and then analyse their experiences. This technique traces the actual mental state of the
user, which resolves issues with users not remembering their
feelings during the test, as in the case of post-task questionnaires or interviews. With this method we can get a better
estimate of the cognitive values of a user’s brain, with more
nuanced data such as cognitive load and focus. This paper
describes how EEG technology can be used to implement a
user experience evaluation method and a validation of its effectiveness and accuracy.
This paper describes the use of EEG technology along with
existing methods to improve user experience evaluation. This
approach overcomes the limitations and utilises the strengths
of the existing evaluation methods. It precisely identifies how
many and when various emotional events occurred while a
user uses an interface, thereby overcoming the limitations discussed above. Semi-structured interviews and/or post-task
questionnaires are used to obtain the reasons behind these
events.
RELATED WORK
There is an abundance of literature on using EEG to determine and classify different emotions and mental states and
try to co-relate different variables responsible for the cognition process in the brain [6, 13, 14]. The literature underscores the challenges in formulating a model that accurately
describes the relation and functioning between the variables
responsible for human emotion.
Horlings et al. [6] used EEG to capture the brain waves of
10 participants while they were looking at pictures with some
emotional content. Participants rated their emotion on a SelfAssessment Manikin (SAM), a non-verbal pictorial assessment technique that directly measures the pleasure, arousal,
and dominance associated with a person’s affective reaction
to a wide variety of stimuli [2]. Positive, neutral and negative emotional states were classified, achieving a classification rate of 37% and 49% for the valence and arousal state
respectively.
Emotiv Systems Australia [5] developed a headset called
EPOC that uses 14 electrodes to measure the brain’s electrical activity. Khushaba et al. [8] attempted to find out the nature of decision making using EEG through the Emotiv EPOC
headset. Participants were showed 72 sets of images, each set
having 3 images and for every set they had to click on the
image they liked. The images used different colours and patterns to more precisely determine the contribution of colour
and pattern to the decision making process. They found that
in general participants preferred either a certain colour or a
certain pattern. They also tabulated the frequencies of each
colour and pattern selection. The data were visualised according to the mutual information between the four EEG band
powers along different EEG channels.
A study [13] done to differentiate between the high alert,
normal alert and fatigue state of mind also used the Emotiv
EPOC headset. 16 participants were given a computer-based
task which was 180 minute long and involved mental arithmetic. The participant’s brain activity was recorded while
they were performing the task. The data was divided into a 15
minute span for classification. The results of this study shows
that for the first fifteen minutes of the study the participants
were in a state of high alertness, then for the next 35–40 minutes the alertness levels came down, which the researchers
called “normal alertness” and after this stage the participants
experienced mental fatigue as classified according to the data.
Vi and Subramanian [14] performed a study to detect the state
of “confusion,” where “confusion” describes the state where
the user is unsure about something he/she has done. To detect
this state the users were shown a series of five arrows in a sequence for one second and they had to memorise the pointing
direction of the middle arrow. After they answered, they were
asked how sure they were about their recent choice, with four
options: 1) Totally correct; 2) totally incorrect; 3) more towards correct side; and 4) more towards incorrect side. The
process was repeated, where the participants’ brain waves
were captured through the Emotiv EPOC headset. Datasets
where the participants selected option 3 or option 4 were considered as a “confusion” state. After signal processing and
analysis a system was trained to detect this “confusion” state.
Another study [9] used EEG to determine change in brain activity during reading and non-reading states. A series of presentation slides, with some having text and some blank, were
presented to participants for 30 seconds in the case of text
slides and 20 seconds in the case of blank slides. The final
results revealed that the “left hemisphere is dominant regarding reading tasks” [9]. They were able to clearly differentiate
between brain activity during reading and non-reading states.
Three researchers from the National University of Singapore
measured sleep quality as a basis for making a music recommendation system, that analyses the quality of sleep and then
suggests music to help the person sleep [15]. The researchers
collected EEG data to classify it into three categories, that is,
wakefulness, deep sleep or other sleep stages. Only the EEG
recordings were used and achieved very high accuracy rates
of between 87% and 99%.
USER STUDY
In order to design a system that can evaluate user interfaces
based on the user experience of the users while using them,
we needed to analyse the variables that reflect the user experience. Our user study was aimed at finding out two things:
1. Which expressions and emotions extracted through EEG
values reflect user experience?
2. How does the data of the emotions and expressions from
(1) vary according to the user interface design?
Our aim was to find the specific actions or specific parts of
the user interface that lead to the user experiencing negative
or positive emotions. For this study we recorded the values
for several emotions in order to find the ones that best reflects
user experience and are sensitive enough to clearly differentiate between good and bad user interface design.
Experimental setup
For the acquisition of EEG data we wrote a program that
uses some of the libraries provided by the “Research Edition”
Emotiv SDK [11]. The program extracts the data values for
emotions and facial expressions such as blink, left/right wink,
look left/right, furrow/raise eyebrow, smile, clench, left/right
smirk, short-term excitement, long-term excitement, meditation, engagement/boredom, frustration and laugh. For all
these emotions a value 0 represents an absence of that expression, whereas the value 1 represents the maximum value for
the expression. Values between 0 and 1 represent the strength
of the emotion. The program code also tracks the mouse cursor position so as to find the screen location and relate it with
the emotion at that time.
• Home — for returning to the home page at any time from
anywhere in the website
According to [7], Emotiv’s emotion models were built using data gathered from more than 100 participants. To induce emotions, activities such as playing games and watching
videos or still images were used. Videos of the participants
were recorded together with respiration, skin conductance,
heart rate and blood volume flow measurements as well as
EEG data collected using the Emotiv EPOC. Subjects completed questionnaires about their experience, after which the
data were analysed and labelled. Features known to be specific to particular emotions were then used to create the emotion models.
• Contact us — link to the page which does not show any
information but displays the text “please contact the site
administrator for further information”
The software application calculates and saves the values for
the emotions and expressions, as well as recording the time
and mouse cursor position for each entry. The application is
written in C++ using the Qt framework [10].
Screen recording software “Camtasia Studio” [4] was used to
record the screen while the participant was using the interface. All screen movements were stored as video, allowing
us to relate the activities of the user with the headset values
at the time of those activities. The “EPOC Control Panel”
software was used to set up the headset on the scalp of the
participant. This ensured the proper placement and connection of the sensors.
User interface used in study
To test the user experience of a user interface we created two
websites, one based on Shneiderman’s eight golden rules of
interface design [12], and another design where some of these
eight golden rules were clearly violated. For designing a bad
interface an ad hoc survey was performed in which users were
asked that what are the things they find bad about a user interface. In general two things were reported:
• Difficulty in finding something for which the user is
looking for
• No proper feedback for the user and the waiting time for a
process not shown
In addition to the things highlighted in the survey we also
decided to focus in more detail on two of the eight golden
rules[12]:
• Improper feedback to the user in case an error occurs
• Confusing design where the user cannot decide which
option to choose in order to proceed towards their desired
goal
• About — link to the page displaying information about
the bands
• Schedule — link to the page showing the date and place of
the performances during summer
The link to book the tickets is not provided on the home page
as a menu button. Instead the link to the tickets is a hyperlink
which is being provided at the bottom of the “About” page as
it is the least probable page to find a link for ticket booking.
After the user finds the link to the tickets they are directed
to the registration page, where they have to fill their primary
details to register before payment. Four buttons are displayed
which all appear to redirect to the payment page after filling in
the registration details. The buttons have the text “yes,” “go,”
“continue” and “proceed” respectively and only the button
with the text “Go” leads to the payment page, whereas the
other three buttons just refresh the page. The user’s confusion
state and disappointed state when the button will not respond
as the user expects was recorded.
After the users click the “Go” button they are redirected to
the payment page where, after entering their card details, they
click the “Pay” button at the bottom of a confirmation screen.
A bank payment screen appears, which displays text indicating to wait for four seconds while payment is being processed, while actually programmed to wait for eight seconds,
so as to make the user impatient.
Eight seconds after the payment gateway, an error screen appears displaying a message that the server has crashed and
then displays a long message of the technical details of the
system failure, with a link at the end which says, “click here
for an attempt to regenerate ticket.” When the user clicks on
this link the ticket is generated. This page is designed to violate the design rule which says if an error occurs it should
be friendly and should not display the technical details, but
should only display the error correction or recovery method.
Good design variant
In the good design variant most of the screens were kept the
same. The home page was changed to include a link for tickets as one of the menu bar options. The link to the ticket was
also provided as an image button with the text “Tickets.” On
the registration page there is only one button to proceed. After the users fill in the card details on the payment page, they
click on a Pay button. A confirmation screen appears which
shows the text “Wait for 4 seconds” and after four seconds the
ticket is generated.
Bad design variant
Method
The website is for a rock band concert. The design has a home
page consisting of four menu buttons:
Ten participants were recruited for the study. All participants
were university students with experience in booking tickets
online. Sessions were conducted with each participant individually. The task for each participant was to book and generate an e-ticket for a rock concert. The headset was set up on
the participant’s scalp, and screen and mouse cursor position
were recorded.
First, the participants used the badly designed website. They
were given a credit card to book the tickets, so as to make
the whole process seem more authentic. After each participant finished his/her task we conducted semi-structured interviews, where their experiences on the interface and the things
they felt were incorrect and hard to use were discussed. These
interviews were audio recorded for a later analysis together
with handwritten notes. After the semi-structured interview
for the bad design variant the participant was asked to do the
same task on the good design variant. Their screen and mouse
cursor movement were recorded, and at the end of the task
another semi-structured interview was conducted to discuss
their experiences.
Figure 1. Frustration values for a participant during event 1
Six out of ten participants attempted both the bad and good
design. Two of the remaining participants only attempted the
good design and the other two only attempted the bad design.
The reason for this distribution was to see the changes in the
emotion values for the same person while using the good design and the bad design. For example:
• A participant using the good design variant after the bad
design variant can change his/her normal response towards
that interface as he/she becomes familiar with the task.
• As participants were not told which design variant is good
or bad, they might start expecting error in the good design
as well based on their previous experience with the bad
design.
For each session data were recorded in three forms: 1) A CSV
file having the values for the emotion and expression of the
participant; 2) a video clip that shows the complete screen
activity of the user while doing the task; and 3) the audio
clip and notes that were made while doing the semi-structured
interviews.
Visualising the facial expression data showed that except
for furrow and smirk, the facial expression values hardly
changed, and even if they do change once or twice in a data set
of 3000 data values, it does not tell us anything about the user
experience. Before moving to further analysis we removed
the facial expression values blink, wink left, wink right, look
left, look right, eyebrow, laugh and clench.
A visualisation of emotional values showed that meditation,
boredom and long-term excitement did not show any variations after the tasks started, so these emotion values were also
removed from further analysis.
Table 1 shows how data analysis was performed for each participant. The events for which the analysis was done consist of all the important design scenarios, and also some extra
events for participants based on the variation in the data values. These were useful in finding other reasons for fluctuation
of the user’s emotion. Clip time, file time and spreadsheet
data row number for all events are included to simplify the
Figure 2. Frustration values for a participant during event 2
analysis of specific events. Reasons for furrow, frustration
and excitement values during the task were tabulated according to the user’s experience1 , and also by relating the recorded
video clip to the data values. Data values were separated into
specific intervals of interest. Another analysis was performed
where the data of all the participants for a specific emotion
during a specific interval were collected. This enabled us to
analyse the change in an emotion during a specific interval
and determine the general trend of the each emotion during
every interval.
Results
The results show that the value of “frustration” as taken from
the headset reflects the user’s mental load, which gives us
more information about the user experience. We have presented the results for the frustration value during three important events in figures 1, 2, 4 for the bad design variant.
The results show consistent high values for this emotional response during the three events by each participant.
Event 1: Finding the ticket link
The first event occurs between the start of the task and the
time when the user finds the link which is placed at the bottom
of the “About” page. The results show that the frustration levels of all the participants were high in user interface designs
1
as described by the user in the semi-structured interviews
Events
Spreadsheet
Row No
Recording 2
started
Task
65
started
Seen
612
ticket link
Clicked
650
on ticket
link
Started
832
writing
name
Finished
1072
filling
details
Click go
1317
Finished
1746
filling pay
details
Clip File
time time
Click pay
Click yes
Error
message
appear
1755
1817
1928
3.28 11:53:23
3.35 11:53:30
3.47 11:53:42
Click link
Ticket
2159
2227
4.17 11:54:12
Fell to 0.3
4.24 11:54:19
0.5
Frustration Reason for
value
Frustration
Furrow
value
Reason
for
furrow
1
Reading
S.T.E
Reasons
S.T.E
for
11:49:59
0.15 11:50:09 over 0.8
During ticket
link search
1.17 11:51:11
Reached Found link
1
1.21 11:51:15
1.40 11:51:34 Value 1
Filling basic
details
1
Filling details
1
Small
dropdown list
year
2.08 11:52:02
Around 0.9
2.35 11:52:29
3.27 11:53:22 1
over 0.8
Finding the
correct button
Filling payment details
During bank
screen and
error message
phase
Reading
Value 1
small
four times font error
message
After finding
recovery link
Table 1. shows the table used for analysing the user study data
From
17851900,
value 1
Excitement
about payment,
filling details,
bank screen,
completing the
task
Figure 3. Maximum frustration level during event 2
Figure 5. Maximum frustration values during event 3
Figure 4. Frustration values for a participant during event 3
Figure 6. Frustration values for participant who only attempted good
design variant
where the information is not organized or easily accessible by
the user.
Event 2: Correct button
The second event starts after the user fills in their primary
details to the time when they click the “Go” button. This
event shows the emotional response of the user during the
time they were confused about finding the correct button to
proceed. Figure 2 shows the line graph for the frustration values of a participant during this event. The results also show
a participant’s data who got stuck after filling the primary information and could not find the “go button” link, asked for
help and we had to guide this participant to click on the correct link, who then proceeded and completed the task. The
participant’s frustration level value reached the maximum as
shown in figure 3 and was constant until the end of the task
due to built-up frustration during this failure.
Our results for the frustration level during the second event
show that in the user interfaces where there is no clear navigation for the user, frustration levels raise. The continuous
rise and fall of the frustration values occurs due to every time
they clicked on the button, the participants expected it will
work similar to the “proceed” and “continue” buttons. With
those buttons the page actually refreshes, giving them a feel
of progress and lowers their frustration levels. As they realize that they are on the same page their frustration levels rise
again.
Event 3: Error message
The third event of interest is the point when the error message comes after the bank payment screen until the time the
user clicks on the link to regenerate ticket. Figure 4 shows a
line graph of the frustration levels of a participant during this
event. During our study one participant was unable to find
the correct link after filling in the primary details, and was
assisted in finishing the task. Due to the failure in finding the
link, the participant’s frustration level value shown in figure
5 was at the maximum throughout the rest of the task.
Participants who only attempted good design
As described in the Method section, some participants only
attempted the good design, as we wanted to differentiate between the results of a user who has already experienced the
bad design variant. After using the bad design variant, the
response to the good design variant was affected by the bad
design experience. Users reported that they were not confident whether their selection would work or not, and they also
built some negative expectations based on their previous experience. The study where the participants who attempted
only the good design variant were not familiar with the design of the website and they did not have expectations about
the functioning of the website. The results for a participant
who only attempted the good design variant is shown in figure 6. The rest of the results show that the frustration levels
of the participants who only attempted the task on the good
design did not go over 0.6.
Other reasons for frustration
1. During filling in basic and payment details: The results
show high values of frustration of participants while
entering their basic information on the registration page
and filling in the payment details from the credit card.
Participants explained that because they were asked not to
move their head and body (to make the data noise-free), it
was difficult for them to read and enter the credit card
details.
2. Participants wanted to finish very quickly.
Figure 7. Eyebrow furrow values for a participant with weak eyesight
3. Waiting time: The results also show that high levels of
frustration were seen when the participant had to wait
because of a slow browser; also when the payment was
being made they had to wait a few seconds for the bank
payment screen. Although all these results have nothing to
do with the interface design, they show that the higher
levels of frustration can also be achieved due to the other
factors and should not be confused with the interface
design itself.
Furrow
The values for eyebrow furrow were also analysed. It was
found that the furrow value is not related to the emotional
response, but that it reflects the points where the user made
a furrow expression, either because of difficulty in reading
or understanding something. The furrow values were also
shown when the participants were searching, in our case for
the link to book the tickets. Furrow value entries for the participants whose eye sight is weak was quite frequent and for
the participants with normal eye sight was less frequent.
Figures 7 and 8 show the furrow values for a participant with
weak eyes and normal eyes respectively. The value of the
furrow depends on the extent of the actual furrow made by
the user. The results show that the higher furrow values were
seen when the participant had to read text which was in a
smaller font size. The furrow expression also depends on the
type of expression a person gives for a certain situation. The
high furrow values show that the participant had a problem in
reading something, which could be due to a design where the
text font is small and not clearly visible to the user. It was
also found that the furrow value entries were less frequent in
the good design variant, where participants did not have to
search for links or read error messages.
Short-term excitement
The analysis of the short term excitement values show that
this emotion is not necessarily related to the interface design,
but related to the task that the user is doing. High values
for this emotion were seen when the user was excited about
the next step in the task. In our case it happened during the
payment procedure as the participants were excited about the
ticket generation. All the high peaks in the graph were seen
during the payment option or after completing the registration.
Figure 8. Eyebrow furrow values for a participant with normal eyesight
Users reported excitement when they finished filling in
payment details, as they were excited about completing
task. The short-term excitement values are higher when
interface is new for the user, as the user is excited about
next screen and the process.
the
the
the
the
Conclusion
One aim of the study was to explore and evaluate the expressions and emotions extracted through the headset whose values best reflect the user experience on any user interface. Another aim of the study was to study the variation produced
in the short-listed expressions and emotions, according to the
user interface design. It was found that frustration is the key
feature to measure user experience and the furrow expression
and short term excitement also reflect the mental states of the
user, which is helpful in relating the user’s progress on the interface with the user experience. The conclusion drawn from
this study is that the value of frustration increases with the increase in confusion, nervousness, focus, and concentration. It
was found in the bad design variant that the frustration levels
rose and reached the maximum value in all the three important bad design scenarios.
The frustration values were also raised because of factors like
having a slow browser and the waiting time during ticket generation. The rush to finish the task quickly also increased the
values of frustration. The frustration levels of the user were
quite low when performing the same task on the good design
variant, indicating that the frustration value is a sensitive and
Figure 9. A screenshot of the software application, showing frustration,
furrow and excitement values for a specific time interval
reliable measure. In the study all the high and low levels of
frustration were justified and there was no case where the rise
and fall in the values did not match with the user’s explanation about their experience.
The frequent value for the furrow expression is either because
of the user’s eyesight or from focusing on something specific.
The high furrow values can tell us that the user was either
unable to read something, unable to understand something or
was trying to search for something in order to progress. The
excitement values reflect the sense of achievement or curiosity about the outcome of the current action. The excitement
values are generally high when the user is either new to the interface or the user is close to completing the task. The results
obtained showed quite consistent variation for every participant in all the scenarios.
The results of the study helped us to determine the values that
reflect user experience. This classification was used to develop a software application, described in the next section,
that can be used to evaluate user experience in order to improve user interface design.
CASE STUDY
This section discusses the use of our system to evaluate a university website. This case study shows how the results, that
depict the user experience of the participants, can be used to
evaluate the design and usability of the website.
Figure 10. The navigation report generated by the software application
for the registration page of the “bad design”
Figure 11. The user experience report generated by the software application
1. the CSV (Comma-Separated Values) format file containing
the recorded values from the headset together with timestamps and mouse cursor positions;
2. the recorded video clip that is created using screen capture
software while the user is using the interface; and
3. the notes and audio clip from the semi-structured interviews that explains the reasons for the values described in
the CSV file.
We used a test scenario where the user was asked to use a
particular option of the website, and to use it the user must
be able to find the location of that option. This kind of test
scenario was helpful in evaluating the design and structure of
the website. The task was to find the tuition fees for the computer science postgraduate course for an international student.
The setup used was identical to that of the user study, that
is, recording the values of ’frustration’, ’furrow’ and excitement through the Emotiv EPOC headset along with the screen
recording and tracking of their mouse cursor position.
The user interface of the application is shown in figure 9, with
a graph indicating the three values i.e. excitement, furrow
and frustration for a specified time interval (0.05 seconds to 4
minutes 10 seconds). The application also provides the ability to see the user’s mouse navigation points on the web page
being tested, including the frustration level at those points,
as shown on the navigation report in figure 10. Red points
indicate high frustration, yellow points indicate medium frustration and green points indicate a relaxed state. A user experience report, shown in figure 11, is generated by the application that gives the minimum, maximum and average values
for all three emotions, together with the percentage of time for
the values of ’high mental load’, ’engagement’ and ’relaxed’
during the selected interval.
Software application
Method
We developed a software application that records and saves
data from the headset, as well as providing a number of different visualisations of the recorded data. Three types of data
are used by the application:
Two university students participated in the case study. The
participants started from the home page of the website and
had to navigate to the page where the fees structure for
the international postgraduate computer science student was
Experimental setup
shown. They were interviewed immediately after they completed their respective tasks. The data were visualised as soon
as the test session ended and the participants were asked to
describe their experiences. The reasons for all the important
points of interest were confirmed through semi-structured interviews.
Evaluation
The initial analysis of the data was done by splitting the
screen recording time into several intervals, and then for every interval relating the events during those intervals with the
graph generated for that interval. The reasons given by the
user were also related to the graph for the respective events.
By using this method we could identify the intervals where
the participant was relaxed, engaged or highly frustrated.
Results and discussion
The pathways selected by both the participants were different.
One participant started navigating through the menu bar options and the second participant used the search box to reach
the required page. We discuss the user experience of both the
participants separately.
Participant 1
The participant, when asked about high frustration values, reported confusion to decide whether to click on “postgraduate” or “international” as the postgraduate international student fees had to be found. The participant then clicked on
“Fees and funding,” and after searching for the “Tuition fees”
link they clicked on “Tuition Fees for International Students.”
A small piece of text appeared guiding the user to click if
they want to see international student fees. This link was very
small and the participant’s furrow levels were found to be at
the maximum value during this stage. The frustration level
of the participant also rose to the maximum value when, after
clicking on that link, a page appeared where the fees structure
was written for all courses in alphabetised order. The participant failed to notice the course listing for computer science
and then clicked away from this page. Later they entered the
computer science department page but could not find the fee
structure there. This participant was unable to finish the task.
The design of the course listings page could be improved by
using text spacing and size to create a visual hierarchy. The
structure of the data in the website could be revised as the
user had to follow too many links to reach the required page.
A well-designed information architecture makes a site easier
and more enjoyable to use [1].
Participant 2
The second participant started the study by directly searching
for the query “computer science postgraduate international
fee”, and from the results page clicking on the entry for the
computer science department. The participant then selected
the option “Postgraduate” from the sidebar menu. No direct
link for the fee was found there, but there was a small link
for the “postgraduate taught admissions” brochure. The participant clicked on it and a PDF document appeared that did
not have the fee details. The participant again searched for
international postgraduate fees and from the results he navigated to the page where another click on the “postgraduate taught course page” displayed the list of fees for all the
courses. High furrow values were seen when: 1) the brochure
was being read; 2) where the text is very small; and when
computer science is being searched for in the list of subjects.
The frustration was seen to reach the highest level when the
participant could not find the fee in the brochure.
The participant reported the user experience of the website to
be challenging. The average frustration value was 0.6, which
is not very high. The participant found the link by using the
search box, but it was disappointing for the participant not to
find a fee link on the computer science department’s page.
Discussion
The system was able to effectively measure the user experience of participants on the university’s website using the values of “frustration” and “furrow” values at all the important
points of interest, and based on the user experience we were
able to find some structural and design inconsistencies on the
website. It was found that the pages on the website were not
properly linked to all the necessary related pages.
The participants were supposed to find tuition fees for international postgraduates in computer science. According
to the interface the user can start by clicking on any of the
words present in the requirement, that is, postgraduate, international, fee, and computer science. All the pages should
be effectively linked, so that the user will be able to proceed
to navigate in the correct direction irrespective of which keyword was used at the start.
CONCLUSIONS AND FURTHER WORK
We designed and developed a novel method to measure user
experience on any user interface by measuring EEG feedback.
The results showed variation in three emotional and facial expression values for a specific design, and the other reasons
that could cause fluctuation in the values. By comparing:
• the expression values with the respective action or part of
the interface used at the same time,
• and the reason for the emotion as reported by the participant in the semi-structured interview after attempting the
task,
we were able to find patterns like the emotion value of the
participant was x when he/she was using y part of the interface
and the reason reported was z. The “frustration” values show
the mental load and the “furrow” values show the condition
when the options are not clearly visible, usually due to small
font size and/or suboptimal positioning.
One advantage of the system is that it is capable of highlighting the user experience on any specific part of the interface.
As the measurements are taken directly from the brain’s electrical activity, the system can detect events experienced by the
user during the test which they forget to share using any other
methods. This helps to overcome drawbacks of some of the
existing methods, like post-task questionnaires or interviews,
when used independently. The system combines EEG technology with the existing methods to improve the accuracy and
precision of user experience evaluation.
The system can be improved by automatically identifying
points of interest and presenting questions to the user about
their mental state and the reason for the fluctuation in values,
using the screenshots or video of those events. The system
could also be trained to detect more emotions and expressions
which reflects the user experience in a more accurate way.
REFERENCES
1. Jesmond Allen and James Chudley. Smashing UX
Design: Foundations for Designing Online User
Experiences. John Wiley & Sons, 2012.
8. Rami N. Khushaba, Luke Greenacre, Sarath Kodagoda,
Jordan Louviere, Sandra Burke, and Gamini
Dissanayake. Choice modeling and the brain: A study on
the electroencephalogram (EEG) of preferences. Expert
Systems with Applications, 39(16):12378 – 12388, 2012.
9. Inês Oliveira, Ovidiu Grigore, Nuno Guimarães, and
Luı́s Duarte. Relevance of EEG input signals in the
augmented human reader. In Proceedings of the 1st
Augmented Human International Conference, AH ’10,
pages 5:1–5:9, New York, NY, USA, 2010. ACM.
10. Qt Project. http:/qt-project.org/downloads/.
11. Emotiv Research edition sdk. http://www.emotiv.com/
upload/manual/Research%20Edition%20SDK.pdf/.
2. Margaret M. Bradley and Peter J. Lang. Measuring
emotion: the Self-Assessment Manikin and the
Semantic Differential. Journal of behavior therapy and
experimental psychiatry, 25(1):49–59, March 1994.
12. Ben Shneiderman. Designing the User Interface:
Strategies for Effective Human-computer Interaction.
Addison-Wesley Longman Publishing Co., Inc., Boston,
MA, USA, 1986.
3. Paul Cairns and Anna L. Cox. Research methods for
human-computer interaction. Cambridge University
Press, 2008.
13. Leonard J. Trejo, Kevin Knuth, Raquel Prado, Roman
Rosipal, Karla Kubitz, Rebekah Kochavi, Bryan
Matthews, and Yuzheng Zhang. EEG-based estimation
of mental fatigue: Convergent evidence for a three-state
model. In Dylan D. Schmorrow and Leah M. Reeves,
editors, Foundations of Augmented Cognition, volume
4565 of Lecture Notes in Computer Science, pages
201–211. Springer Berlin Heidelberg, 2007.
4. Camtasia Studio.
http://www.techsmith.com/camtasia.html/.
5. Emotiv Epoc. http://www.emotiv.com/.
6. Robert Horlings, Dragos Datcu, and Leon J. M.
Rothkrantz. Emotion recognition using brain activity. In
Proceedings of the 9th International Conference on
Computer Systems and Technologies and Workshop for
PhD Students in Computing, CompSysTech ’08, pages
6:II.1–6:1, New York, NY, USA, 2008. ACM.
7. Paul Salvador Inventado, Roberto Legaspi, Merlin
Suarez, and Masayuki Numao. Predicting student
emotions resulting from appraisal of ITS feedback.
Research and Practice in Technology Enhanced
Learning, 6(2):107–133, 2011.
14. Chi Vi and Sriram Subramanian. Detecting error-related
negativity for interaction design. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems, CHI ’12, pages 493–502, New York, NY, USA,
2012. ACM.
15. Wei Zhao, Xinxi Wang, and Ye Wang. Automated sleep
quality measurement using eeg signal: First step towards
a domain specific music recommendation system. In
Proceedings of the International Conference on
Multimedia, MM ’10, pages 1079–1082, New York, NY,
USA, 2010. ACM.