Lin, H. C. K., Su, S. H., Chao, C. J., Hsieh, C. Y., & Tsai, S. C. (2016). Construction of Multi-mode Affective Learning System:
Taking Affective Design as an Example. Educational Technology & Society, 19 (2), 132–147.
Construction of Multi-mode Affective Learning System: Taking Affective
Design as an Example
Hao-Chiang Koong Lin1, Sheng-Hsiung Su1*, Ching-Ju Chao2, Cheng-Yen Hsieh1 and
Shang-Chin Tsai1
1
Department of Information and Learning Technology, National University of Tainan, Tainan, Taiwan //
2
Department of Applied Foreign Languages, Tung Fang Design University, Kaohsiung, Taiwan //
koong000@ms39.hinet.net // ice.shsu@gmail.com // chingju@mail.tf.edu.tw // allen01816@hotmail.com //
maney100@hotmail.com
*
Corresponding author
ABSTRACT
This study aims to design a non-simultaneous distance instruction system with affective computing, which
integrates interactive agent technology with the curricular instruction of affective design. The research subjects
were 78 students, and prototype assessment and final assessment were adopted to assess the interface and
usability of the system. Prototype assessment consisted of heuristic assessment and the system usability scale,
while final assessment adopted the triangular cross-validation method: where the questionnaire for user
interaction satisfaction, observation, and interviews were used to explore the effect of learning and obtain
qualitative and quantitative information for analysis. According to the experimental results, the usability of the
non-simultaneous distance instruction system with affective computing was high; the respondents showed highlevel satisfaction regarding interaction with the affective learning system; the training game response
mechanism of the system could effectively improve the emotion of learning; there was a significant
improvement in the effect of learning based on the affective learning system.
Keywords
Affective computing, Affective tutoring system, Text emotion, Facial expression, Skin potential
Introduction
In recent years, the prevalence of smart devices demonstrates a lifestyle associated with computers in modern
society. Regarding different types of program, besides functions, humanity is also critical. The most significant
difference between human beings, computers, and machines is that human beings have emotion. Therefore, in order
to reinforce the humanity of computers and machines, they should approach human emotion in order to provide
proper feedback. With the popularity of the internet, non-simultaneous distant instruction gradually replaces
traditional instruction. Emotion is part of the key semantic information of interpersonal relationships. Positive
emotion leads to more successful learning processes (Ezhilarasi & Minu, 2012; Eyharabide & Amandi, 2012).
Instructional systems can automatically recognize a learners’ emotion and provide proper feedback, thus, it is a
significantly potential and essential research indicator (Islam, 2013).
With the prevalence of online learning, if computers can recognize learners’ learning emotion and maintain their
position emotion, it will result in learning efficiency and effectiveness (Kerr, Rynearson, & Kerr, 2006). In
interpersonal interaction, conversation and facial expression are the most direct methods to recognize others’
emotions; however, people can disguise these two emotions. Therefore, aside from introducing semantic and facial
expression emotional recognition, this study measures learners’ physical signals as indicators of emotional
judgments. Since physical signals are non-volitional physical reactions, the emotional state is the most objective. The
introduction of multi-mode Affective Computing can enhance the precision rate of emotional recognition in
Affective Tutoring Systems.
This study applies Affective Computing, as developed by the laboratory, to research the findings and development of
Intelligent Tutoring Systems. Through developed Affective Tutoring Systems, it expands different dimensions and
functions. The purpose is to strengthen learners’ trust and involvement in learning by a more advanced and complete
emotion recognition module. By complete and effective assessment, it demonstrates the value and existence of this
study.
ISSN 1436-4522 (online) and 1176-3647 (print). This article of the Journal of Educational Technology & Society is available under Creative Commons CC-BY-ND-NC
3.0 license (https://creativecommons.org/licenses/by-nc-nd/3.0/). For further queries, please contact Journal Editors at ets-editors@ifets.info.
132
Literature review
Affective tutoring systems
Changes of technology are rapid and important, and various kinds of digital technologies and techniques are
gradually introduced to instructional environments. As the traditional teachers’ one-to-many instruction model
changes, learning models have become diverse (Chen, Kao, & Sheu, 2003). ITS (Intelligent Tutoring Systems)
means to provide personalized instruction by computer analysis or direct feedback to students. ITS finishes different
instructional tasks by simulating teachers. According to different students’ characteristics and states, it indicates
various instructional methods. Affective Computing is a research field that deals with issues of emotions and
computers. Generally speaking, it is classified into four research levels: emotional recognition, emotional expression,
having emotion, and emotional intelligence. Most studies focused on emotional recognition (Picard & Klein, 2002).
Figure 1. Framework of affective tutoring (Source: Ammar et al., 2010)
Technology introduction reinforces class activities. In addition, teachers can observe the negotiation,
communication, cooperation, and interaction among students (Liaw, Chen & Huang, 2008; Infante et al., 2009;
Martin, Pastore, & Snider, 2012). However, without an appropriate learning strategy, learning effectiveness will not
be as expected (Peng et al., 2009). ATS (Affective Tutoring Systems) means the defection of students’ learning and
emotional states to offer proper emotional feedback and regulate students’ learning emotion (Mao & Li, 2010; Hsu,
Lin, Lin, & Lin, 2014). ATS is developed upon ITS, and aims to effectively adapt to students’ emotion by simulating
human beings (Ammar, Neji, Alimi, & Gouardères, 2010; Lin, Wang, Chao, & Chien, 2012). Although ATS is
developed in recent years, it is the first research that adapts to and recognizes emotion. Thus, this study reviewed this
study of Picard (Picard, 2000), who proposed a conceptual module that influences learning emotion, and constructed
a system that recognizes learners’ emotional state, provides appropriate feedback, and reinforces learners’ learning
(Lin, Chen, Sun, & Tsai, 2012; Lin, Hsieh, Loh & Wang, 2012).
With the Affective Computing technique, computers can recognize human emotions. ATS is considered as a
personalized training model. Lin, Tsai, Cheng, Chao, and Su (2014) combined Affective Computing with a webpage
system to develop Affective Computing on a webpage to provide adaptive learning for learners. Mao and Li (2010)
undertook an investigation into the key factors that influenced students’ satisfaction when using the affective
learning system, and found that the factors included the attitude of students, the effect of tutoring, the accuracy of
emotion recognition, the quantity of identifiable emotions, instructive action, and the usability of the system.
Sarrafzadeh, Alexander, Dadgostar, Fan, and Bigdeli (2008) developed a system targeting math for primary school
students. Through the instruction of a lifelike animated agent, the system analyzed the facial expressions of students
to identify their emotions, and showed the emotion of the animated agent. By incorporating affective computing into
133
a smart tutoring system, Ammar et al. (2010) detected and judged facial expressions and computed emotions, in
order to enhance the interaction between instruction and learners, develop students’ interest in learning, help them
absorb knowledge, and significantly increase the effect of learning through mutual assistance among learners. The
framework chart of the affective learning system is as shown in Figure 1.
Emotional recognition
Chinese semantic information
Emotion is the key semantic information of interpersonal interaction (Ezhilarasi & Minu, 2012). If emotion can be
precisely recognized, it will help to make decisions in better way (Lin, Wu, & Hsueh, 2014). The first condition of
text emotion recognition is to understand semantic content to acquire precise information. It should be based on
natural language processing and semantic analysis (Yan, Bracewell, Ren, & Kuroiwa, 2008). The assumption of
document frequency might lower the precise rate of classification, as many terms with high document frequency are
usually unused words or unimportant information (Basu & Murthy, 2012). Therefore, when selecting terms by
document frequency, other methods are usually adopted. Lu, Lin, Liu, Cruz-Lara, and Hong (2010) proposed the
automatic and hierarchical emotional semantic acquisition system, which is highly intensive. Through an
independent database, it automatically recognizes the subjects’ semantic emotional responses to the events of
sentences. Regarding classification of emotions, there are two principle methods: monitoring and non-monitoring
learning (Feldman, 2013). Studies on emotional analysis demonstrated that the importance of the terms, as calculated
by TF-IDF weights, is highly effective (Liu, 2012).
Facial expression
Facial recognition techniques are commonly applied in daily lives (Gunes & Piccardi, 2007), such as, automatic
facial focus, domestic security and recognition, and facial recognition guard systems, which unlock the door of facial
recognition. Camera shutters with multi-smiles is another extended technique of facial recognition. Therefore, it
demonstrates the common application of facial expressions and potential. According to faces and facial features,
there are generally six kinds of facial expressions: joy, anger, sadness, surprise, fear, and disgust (Ekman & Friesen,
1971; Ezhilarasi & Minu, 2012). Metri, Ghropade, and Butalia (2012) constructed a facial emotional recognition
system by facial features, as proposed by Ekman, and enhanced the effect of recognition of emotions through
physical poses. Figure 2 shows the steps of recognizing facial expressions. Hsu et al. (2014) integrated facial
recognition and semantic recognition by multi-mode for Affective Computing to develop an affective tutoring
system.
Figure 2. Steps of facial emotional recognition (Source: Metri, Ghorpade, & Butalia, 2012)
Skin conductance degree
Skin conductance is also called Galvanic Skin Response (GSR), and it means the electronic conductance on skin.
When people’s emotions change, their body actions, facial expressions, and physical reactions will change
accordingly. For instance, with stimulus, the secretion of skin sweat glands will influence changes of the Galvanic
Skin Response. We can measure the change of skin potential by Affectiva Q sensor. Through this instrument, we can
obtain the figures and analyze the different waves. The peak of the wave means severely positive and negative
emotional reactions, as shown in Figure 3.
134
Figure 3. Galvanic skin response
Research method
Research process and implementation steps
In order to probe into learning effectiveness and system usability of the multi-mode Affective Tutoring System, as
introduced in digital art materials, which include network art, dynamic video-audio technology, recording art,
software art, and new media art, 78 college students of a national university in southern Taiwan were invited as the
research subjects. Their educational levels were college and university to master. By prototype assessment and final
assessment, it evaluated the system interface and system usability. Figure 4 shows the system assessment process of
this study. The first step is to put forward a system concept model for the design of the prototype system. After the
design, the prototype system is assessed. The assessment of the prototype system consists of two parts--the usability
scale of the assessment system for average users, and the assessment of experts. The last step is the final assessment,
which adopts the triangular cross-validation method. The research analysis is based on questionnaires, observations,
and interviews.
Figure 4. Process of system assessment
Prototype system assessment
•
Assessment of average users--System usability scale: To obtain information regarding the subjective feeling of
average users during operation of the system, this study adopted the system usability scale (SUS) as an
assessment tool. Developed by the British Digital Equipment Co., Ltd. in 1986, the scale was designed to inform
enterprises of the general usability of their products and provide a low-cost, reliable, and fast method. It is a
135
•
•
five-level Likert scale, with each item including five options ranked in an ascending manner: (1) Strongly
disagree; (2) Disagree; (3) Average; (4) Agree; (5) Strongly agree. The scale comprises 10 items and adopts the
forward and backward cross-questioning method. Usually, respondents complete the scale without discussion
after operating the system. Through a formula, scoring will be converted into one with “100 points” as the full
mark, where a higher score indicates stronger satisfaction of respondents for the system.
Assessment of experts--Heuristic assessment: Heuristic assessment is an informal usability testing method used
to detect problems regarding usability in the design of a user interface, in order that these problems can be
regarded as a focused part of the redesign. In the heuristic assessment, experts follow a group of given usability
heuristics, and evaluate the constituents of a respondent interface to see if these constituents are consistent with
these heuristics (Nielsen, 1994). Meanwhile, Nielsen suggested inviting 3-5 evaluators, and predicted about 75%
of the problems regarding usability.
Final assessment—Triangulation: As a method of testing research information, triangulation adopts more than
two resources to obtain full understanding, and demonstrate a specific reference point or topic, with the aim of
enhancing the rigorousness and reliability of research. In general, it is recommendable to include three
information sources, which allows an evaluation with diverse perspectives, and provides a neutral stance when
two views are contradictory.
Affective tutoring systems
The system framework of this study is as shown in Figure 5:
Figure 5. System framework
Emotional recognition module
•
Semantic recognition module: By dialogue between the subjects and the interactive agent, this study conducts
Chinese semantic emotional recognition; and through the dialogue input by the subjects, the subjects’ emotional
state is immediately recognized. The construction is as shown below: (1) construction of emotional dictionary;
(2) semantic structure message processing: word segmentation rules, semantic structure, message processing,
keywords matching, and word segmentation; (3) acquisition of semantic emotion. Figure 6 shows the process of
semantic analysis.
136
Figure 6. Process of semantic analysis
•
•
Facial expression recognition module: By open library--EmguCV, this study develops a facial expression
recognition module. EmguCV is an OpenCV component packaged by C# for the development of a Visual Studio.
EmguCV not only has powerful image processing capacity, but is also an open and free library, thus, system
development is less difficult. The steps of facial expression recognition are shown as follows: (1) recognition of
human beings’ facial positions; (2) recognized facial features are compared to six kinds of facial expressions by
classification - HaarTraining and classification; (3) recognized facial expressions refer to emotions; in the
opposition situation, there is no emotion.
Physiological signal module: By Q sensor, this study includes the skin conductance information of the physical
signals in the system. Through a Bluetooth connection, Q sensor sets the device as a COM Port. By Visual
Studio C#, it obtains related figures. Figure 7 shows the process of physiological signal.
Figure 7. Process of physiological signal
•
Brainwave concentration and relaxation training module: By NeuroSky MindWave Mobile, this study conducts
brainwave concentration and relaxation training. When the subjects learn by this system, they rely on the
emotional recognition of the previous three kinds of emotional recognition modules. When the subjects have
negative emotion, the system will automatically accumulate the emotion. When a negative emotion
unexpectedly occurs, the system will automatically stop the teaching material and record the learning time in the
137
•
database. It then activates the Brainwave Visualizer, as developed by NeuroSky, for the subjects’ concentration
and relaxation training. With training effectiveness, the subjects can continue learning.
System interface: In this study, the system interface design is divided into 7 zones: function, teaching material,
interactive agent, semantic dialogue, facial expression, skin conductance signal, and system record. Figure 8
shows the layout of the system interface.
Figure 8. Layout of system interface
(a) Tools (see Figure 8 a): video control, which can be played, paused, and stopped; after-class questionnaires
include learning effectiveness, scale of system usability, and user interaction satisfaction questionnaires.
After learning the subjects, there can be after-class assessment and system assessment; regarding parameter
settings, the subjects can regulate facial expression recognition parameters. According to individual
conditions, it regulates recognition sensitivity. There are six kinds of recognition, and each kind includes
two regulation parameters; the description column provides system instructions, system history, and
laboratory introduction. The end button of this system is termination of the course. Three questionnaires
will be completed to finish the experiment.
(b) Teaching material (see Figure 8 b): the video playing of teaching material. The videos, as recorded by a
teacher in advance, are constructed for the subjects’ learning. Video of this experiment is based on system
design. Therefore, the affective design is treated as the instructional content. A combination of instruction
content and system allows the subjects to have profound learning experiences.
(c) System record (see Figure 8 c): records all activities of this system, including system starting time, change
of course playing, database connection records, results of facial expression recognition, semantic and
emotional recognition, Q sensor connection, and accumulation of negative emotion. The recording format is
“hour: minute: second. Milli-second recording”.
(d) Facial expression (see Figure 8 d): the subjects’ facial expressions are captured by webcam. When the face
and facial expression are captured, a square frame will be drawn. In the figure, the white frame is the
subjects’ captured face; the yellow frame is the subjects’ facial expression recognized as joy.
(e) Skin conductance (see Figure 8 e): the subjects’ skin conductance is captured by Q sensor. Captured
frequency is 32 times/every second. The system shows the total in the system and conducts emotional
sensing.
(f) Interactive agent (see Figure 8 f): the agent designed by this study: batman. According to the emotion
recognized by this system, it transfers the movement of facial expressions and interacts with the subjects in
a semantic dialogue zone.
138
(g) Semantic dialogue (see Figure 8 g): the subjects can interact with the interactive agent in this zone. The
system will conduct semantic recognition according to sentences input by subjects, and provide immediate
feedback.
(h) Researcher’s operation (see Figure 8 h): records the subjects’ time of system learning and current time. Two
buttons are function module controls. The researcher assists with the operation button.
•
Learning process: First of all, interactive agent module: this module is the bridge between a learning system and
the subjects. By an agent mechanism, the system can interact with the subjects and properly provide feedback.
Second, video course teaching material module: this module is the video course teaching material recorded by a
teacher in advance. It is based on video, and the subjects learn online, as in a classroom. Third, learning state
recording module: this module is the core of this system. The system automatically and completely saves the
subjects’ learning stages and uploads it to a database. Thus, the teacher can immediately recognize each
subject’s learning condition, and actively solve problems for the subjects after class in order to reinforce
learning effectiveness.
Experimental results and analysis
To determine if this non-simultaneous distance instruction system with affective computing could improve the effect
of learning, and evaluate the usability of the system, this study invited 78 undergraduates and graduates to participate
in the experiment. The respondents were divided into two groups--the Experimental group, which used the affective
learning system, and the Control group, which adopted the online webpage learning system. The course materials
were videos pre-recorded by the teachers of a university in Kaohsiung, Taiwan. Figure 9 shows the flow chart of the
experiment. Prior to learning, the respondents received a learning effectiveness assessment (pre-learning test), and
then watched a 16 minute instructive video. After learning, the respondents received another learning effectiveness
assessment (post-learning test), completed the questionnaire for user interaction satisfaction, and took the interview.
The entire experiment was recorded, and final assessment--triangulation was conducted after the experiment.
Figure 9. Flow chart of the experiment
139
The system counted the negative emotions captured in the learning of the Experimental group, and the information
was added into the learning status database. When the negative emotion reaches a certain level, the system will
suspend the course and start a brainwave concentration and relaxation training game. In concentration training,
stronger concentration will result in the explosion of a bucket; while sustainable and strong concentration will
maintain the explosion of the bucket, and the best explosion time will appear on the interface, as shown in Figure 10.
In relaxation training, if the respondents feel more relaxed, a balloon will start to rise; if the respondents maintain a
high-level of relaxation, the balloon will float above and rise higher, and the maximum height the balloon reaches
will appear on the interface, as shown in Figure 11 The respondents can decide if they want to continue brainwave
concentration and relaxation training; if they refuse to continue the training, they can shift the interface to the
affective learning system to continue learning.
Figure 10. Brainwave concentration training
Figure 11. Brainwave relaxation training
140
After learning, both the Experimental group and the Control group immediately received the learning effectiveness
assessment (post-learning test), and then completed the questionnaire for user interaction satisfaction. After
completing the questionnaire, the Control group finished its task in the experiment, while the Experimental group
finished its task only after the interview.
System usability analysis
At the prototype system development phase, this study invites 30 users for prototype system usability analysis.
Cronbach’s α acquired by the scale of system usability is .791, and the least reliability of .7 is accepted by this study,
as it demonstrates that the reliability of the questionnaires is acceptable. After transforming reverse responses into
positive responses in the scale of system usability, this study conducts item analysis, as shown in Table 1.
Noticeably, regarding Q4 and Q10, only 46.6% and 66.8% users, respectively, suggest that they do not need
assistance, and the subjects indicate that they can use the system without prior knowledge. Hence, the system
interface must be simplified for the subjects’ ease of use.
The researcher analyzed the questionnaire filled by the respondents and converted the backward questions in the
system usability scale into forward ones for analysis. According to the 5-point scale, the researcher selected the two
highest score -- the percentages of “4” and “5” for aggregation analysis. As is shown in the following table, 56.7% of
the respondents to Q1 were willing to use the affective learning system on a regular basis; 70.2% of the respondents
to Q2 did not believe that the system was too complicated; 83.3% of the respondents to Q3 thought that the system
was easy to use; 46.6% of the respondents to Q4 believed that the system entailed little assistance from technicians;
73.4% of the respondents to Q5 thought that all the functions of the system were well integrated; 83.3% of the
respondents to Q6 did not believe that the system was inconsistent; 86.6% of the respondents to Q7 thought that most
people would master the skills of using the system within a short time; 96.7% of the respondents to Q8 did not
believe that the system was too difficult to use; 96.7% of the respondents to Q9 were confident that they could use
the system; 66.8% of the respondents to Q10 did not think it necessary to acquire much knowledge to use the system.
Items
Mean
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
AVERAGE
3.73
3.80
4.13
3.20
3.87
4.10
4.20
4.40
4.27
3.80
3.95
Table 1. Analysis of system usability scale
5-point scale percentages (%)
Standard deviation
1
2
3
4
.740
0
0
43.3
40.0
.925
3.3
3.3
23.2
50.0
.681
0
0
16.7
53.3
1.064
6.6
20.2
26.6
40.0
.629
0
0
26.6
60.0
.759
0
3.3
13.4
53.3
.664
0
0
13.4
53.3
.563
0
0
3.3
53.3
.521
0
0
3.3
66.7
.847
0
6.6
26.6
46.6
.7393
.99
3.34
19.64
51.65
5
16.7
20.2
30.0
6.6
13.4
30.0
33.3
43.4
30.0
20.2
24.38
The scores in the scale were obtained according to the scoring of the system usability scale. They reflected the
comprehensive assessment of the respondents on the system usability and can be used for the comparison of usability
among different versions of the system. The total score (ranging from 0 to 100 points) of the questionnaire was
obtained according to the scoring of the system usability: (1) the score of the items labeled with odd numbers was
obtained by subtracted 1 from the original score; (2) the score of the items labeled with even numbers was obtained
by subtracted 5 from the original score; (3) the total score (ranging from 0 to 100 points) of the questionnaire was
obtained by first aggregating the scores of all items and then multiplying the aggregated score with 2.5. Based on the
scoring of the system usability scale, the researcher conducted statistics of questionnaire scores of the subjects. The
result is as shown in Table 2; where the mean is 73.75, the median is 70.00, the mode is 67.50, the standard deviation
is 11.14, and the minimum and maximum are 47.50 and 95 respectively. The score for the system usability is 70,
which indicates that most of the respondents were satisfied with the system usability. The average score for the
system is “73.75 points,” which shows that average respondents were satisfied with the usability of system.
Meanwhile, it was compared with Figure 12, which fell into the zones of “good” and “excellent,” respectively.
141
Number of samples
30
Table 2. Conversion results of scores in the system usability scale
Mean
Median
Mode
Standard deviation
Minimum
73.75
70.00
67.50
11.14
47.50
Maximum
95.00
Figure 12. Score distribution of the system usability (Source: Bangor, Kortum, & Miller, 2009)
User interaction satisfaction analysis
The questionnaire for user interaction satisfaction includes 6 dimensions: total use reaction, display of screen, terms
and system information, learning, system function and usability, and user interface. Each dimension includes 2~6
items. According to the subjects’ satisfaction, the rating is from 1~7. There are a total of 27 items. Cronbach’s α of
the user interaction satisfaction questionnaire is .949, and the least reliability of .9 accepted by this study. Thus, the
reliability of questionnaires results is good.
This study analyzes questionnaires responded by the subjects, and obtains the mean by the total scores of the items of
the dimensions divided by the number of items. It then conducts descriptive statistical analysis on the results, with
the analytical outcome as shown in Table 3. The means of the dimensions of the Experimental group are higher than
those of the Control group. Satisfaction with dimensions is at least 5 in the 7-point scale, meaning that the subjects’
subjective satisfaction with the design of the human-machine interface of the system is good.
Regarding mean, we compare satisfaction with the dimensions of the Experimental and Control groups. Satisfaction
with using Affective Tutoring Systems is higher than online webpage learning systems, which shows that, in the
same video material learning, satisfaction with Affective Computing recognition is higher than non-simultaneous
distant instructional systems. In addition, regarding the means of total use reaction, usability, and user interface, the
Experimental group is more significant than the Control group, which shows that users’ interaction satisfaction with
the system is high.
Table 3. User interaction satisfaction questionnaire--descriptive statistics
Experimental group (30 people)
Control group (48 people)
Dimensions of questionnaires
Mean
Standard deviation
Mean
Standard deviation
Total usage reaction
5.14
.83
4.40
.75
Display of screen
5.76
.92
5.09
.89
Term and system information
5.29
.69
4.78
.99
Learning
5.49
.83
5.00
.88
System function
5.37
.99
5.02
1.08
Usability and user interface
5.38
.89
4.81
.89
Average
5.405
.73
4.852
.74
According to questionnaire results and t testing of independent samples, this study attempts to determine if
satisfaction is different between the Experimental group and the Control group. Analytical results are as shown in
Table 4. Assessment results of user interaction satisfaction is Experimental group > Control group and
significance .002 < .05, meaning that the satisfaction of the two groups’ is significantly different. Based on previous
results, the Experimental and Control groups have significantly different satisfaction using the Affective Tutoring
142
System and online webpage learning system. Interaction satisfaction with Affective Tutoring Systems is higher than
online webpage learning systems.
Table 4. User interaction satisfaction questionnaires—t-test of independent samples
Number of
Standard
Mean
t value
Group
samples
deviation
Experimental group
30
5.405
.73
-3.238
Control group
48
4.852
.74
Note. **p < .01.
Significance
(two-tailed)
.002**
Analysis of learning effectiveness
Learning effectiveness before the experiment
To determine the difference in knowledge between the Experimental group and the Control group, the researcher
undertook independent sample t testing on the pre-learning scores of the respondents. As shown in Table 5,
significance (.38 > .05) indicates that there was no significant difference between the two groups before learning,
meaning that both groups shared similar knowledge before the experiment.
Table 5. One-way ANOVA of learning effectiveness before the experiment
Number of
Standard
Mean
t value
Group
samples
deviation
Experimental group
30
47.00
16.43
.878
Control group
48
50.21
15.23
Significance
(two-tailed)
.383
Learning effectiveness after the experiment
The researcher undertook one-way ANOVA of the post-learning scores of the respondents to determine if there was
any significant difference between the two groups, and the result is as shown in Tables 6 and 7. The average postlearning scores of the Experimental group and the Control group were 73.00 and 63.13, respectively, which is higher
than the pre-learning average score of 47.00 and 50.21, respectively. The significance (.019 < .05) of the postlearning score manifests that the instruction played a significant role in the improvement of learning effectiveness.
Group
Experimental group
Control group
Total
Source of variance
Inter-group
In the group
Total
Note. *p < .05.
Table 6. Descriptive statistics of the post-experiment learning effectiveness
Number of samples
Mean
Standard deviation
30
73.00
13.17
48
63.13
20.02
78
66.92
18.26
Table 7. One-way ANOVA of the post-experiment learning effectiveness
Quadratic sum
Freedom
Mean quadratic sum
F
1800.288
1
1800.288
5.734
23861.250
76
313.964
25661.538
77
Significance
.019*
The researcher subtracted the pre-learning score from the post-learning score, and conducted percentage conversion
for independent sample t testing, and the result is as shown in Table 8. The mean of the increase in the learning
effectiveness of the Experimental group is 94.94%, whereas, that of the Control group is 29.60%. The significance of
(.033 < .05) means that there was significant increase in the learning effectiveness of the two groups, which indicates
that the increase in the learning effectiveness of the Experimental group was greater than that of the Control group,
and the respondents could improve their learning effectiveness with the affective learning system.
143
Group
Experimental group
Control group
Note. *p < .05.
Table 8. Comparison of the increase in learning effectiveness
Number of
Standard
Standard
Mean
t value
samples
deviation
error
30
94.94
158.09
28.86
-2.228
48
29.60
35.98
5.19
Significance
(two-tailed)
.033*
Results of the interview
After the experiment, the respondents took an interview that lasted for 3 to 5 minutes. The researcher wrote the
respondents’ answers to the above interview questions, which have the following results:
• Most respondents would first turn to their partners for help, and then their teachers, when encountering problems
in learning. The majority of the respondents thought it was difficult to communicate with teachers, and thus,
would not regard teachers as the first choice when they sought help. Only a few respondents did not turn to
teachers or partners for help; instead, they sought help on the Internet.
• Most respondents thought that active help from teachers would facilitate their learning; however, only a limited
number of teachers can notice the problems of students and voluntarily offer help in the current educational
environment. A few respondents believed that they would not seek help until they face problems, and that active
help from teachers might upset or embarrass them.
• Most respondents thought it necessary to keep the mechanism of Question (2). According to them, not all
students would actively seek help, thus, the mechanism was good. A handful of students still preferred to seek
help on the Internet, as they thought such a method could improve their learning and memory.
• Most respondents thought that the response from the interactive agent was too limited, and that an interactive
agent would give the same response to similar emotion.
• Most respondents believed that the brainwave training game was interesting; however, some thought the game
was so difficult that they could not fulfill the objectives, and thus, felt frustrated; while others were deeply
fascinated by the training, and hoped to break the record kept by themselves or others.
• Most respondents thought that what was taught was boring, and such a mechanism could help them refocus on
what was to be taught later. Only a few respondents believed that the game interrupted their learning, was of
little help in their learning, and they felt reluctant to continue learning once they started to play the game.
According to the results of the interviews, most respondents were satisfied with the affective learning system, and
thought that the learning mechanism, as proposed by the researcher, was effective and necessary. In emotion
recognition, the number of keywords for emotion recognition of Chinese meaning was too small, and the system
failed to give responses to many daily expressions; the recognition of facial expression was too sensitive; the
information about facial expressions used for classification was inadequate, and errors were likely to occur when the
system judged a face for the first time. One respondent thought that the interactive agent distracted his attention
during learning, and that the brainwave game would interrupt learning.
Conclusion and future studies
This study incorporated affective computing into a non-simultaneous distance instruction system, and adopted the
multi-mode affective perception module to detect the emotions of the respondents during learning. The curricular
collocation system design used affective design as the learning material to provide respondents with more impressive
learning experiences. Prototype assessment and final assessment were employed to discuss the usability of the
system, as well as satisfaction with the system. The former involved heuristic assessment based on expert evaluation,
where a system usability scale was used to analyze the usability of the revised system; the latter adopted
triangulation, where methods such as questionnaires for user interaction satisfaction, and observations, as well as
interviews, were used to explore satisfaction with the system; the researcher designed the assessment of learning
effectiveness according to the learning material in order to analyze the learning effectiveness of the respondents.
Based on the experiment results, the researcher have come to the following conclusions according to the objective
and topic of this study:
144
•
•
•
•
With the non-simultaneous distance instruction system, the researcher integrated affective computing with the
instruction of a video course, and developed the affective learning system. The usability of the system scored
73.75, meaning that average users were satisfied with the usability of the system. Therefore, the researcher
believes that the usability of the non-simultaneous distance instruction system with affective computing was
high.
This study used triangulation to investigate respondents’ satisfaction with the affective learning system. In the
interviews, most respondents said that the system was simple and easy to operate, the interactive agent
mechanism made learning interesting, and enhanced their passion for learning. Hence, the researcher thinks that
most of the respondents were satisfied with the affective learning system.
According to the results of affective computing, the researcher collected the negative emotions of the
respondents, and started the training game at an appropriate time in order that the respondents could change their
emotion during learning. Most of the respondents believed that such a mechanism was helpful to learning at a
later stage, thus, the researcher believes that the training game in this system can effectively improve emotion
during learning.
This study aims to develop an affective learning system based on a non-simultaneous distance instruction
system. To determine if the system could maintain and enhance the learning effectiveness of the existing system,
the researcher divided the respondents into two groups, namely, the Experimental group and the Control group,
and conducted simultaneous assessment of learning effectiveness. According to the experiment, the affective
learning system not only maintained the existing learning effectiveness, but also improved the learning
effectiveness of most respondents. Therefore, this research believes that the affective learning system can
effectively enhance learning effectiveness.
The research results and conclusions show the possibility of the popularization of this affective learning system, and
offers suggestions for relevant future research.
Suggestions for the affective learning system
During the experiments, the researcher found that the response mechanism of the interactive agent was a key factor
that could effectively increase the respondents’ use of the system. If the recognition and accuracy of the existing
affective module are enhanced, respondents will feel a stronger intention to use the system, and will not feel bored in
the later stages of learning. Regarding the Chinese meaning emotion recognition module, the researcher thinks that
more daily expressions can be added to strengthen the understanding of the system, and that a diverse response
mechanism can be added to lengthen the interaction between respondents and the interactive agent. Meanwhile, a
large quantity of daily life facial expressions should be collected in order to enrich the database, which in turn will
enhance the accuracy of real-time facial expression recognition. Regarding the physiological signal recognition
module, the researchers thinks that, if it is impossible to buy specialized devices, such as an affective lab, the
physiological signal recognition module should be removed in order to promote the efficacy of the entire system.
Regarding the brainwave concentration and relaxation module, the researcher believes that both concentration and
relaxation are important factors that influence learning emotion, and that if the two are incorporated into the realtime detection of the system, it will significantly facilitate learning. With non-simultaneous distance instruction
concepts, this study adopted video-audio materials for instruction. It is suggested that future researchers make full
use of the functions of the non-simultaneous distance instruction system, and add curricular interaction modules,
such as a discussion and curricular interaction sections in order to enhance the interaction between teachers and
students, stimulate an actual instruction environment, and strengthen the on-site experience.
Suggestions for video-audio course material
The video-audio course material in this study was the PPT of the textbook instructed by teachers, but without the
reality of face-to-face instruction; hence, the researcher thinks that interactive materials, such as augmented reality,
should be added to involve respondents in the instruction and reduce boredom. The researcher believes this affective
learning system will attract more users. Additionally, pictures of teacher instruction can be added into the videoaudio course section to enhance the on-site experience. Regarding the assessment of learning effectiveness, the
researcher suggests that future researchers follow the mode, as this study involved the demonstrations of experts,
which indicate a high-level of reliability and validity.
145
Suggestions for the interactive agent
The interactive agent of this study is a singular role, thus, the researcher suggests that a paper doll system be added
by future research in order that respondents will have an alternative. Meanwhile, additional accessories for the
interactive agent can be added to enhance respondents’ identification with the interactive agent, and thus, improve
their learning experience. In terms of the emotion of the interactive agent, more emotional behaviors, such as
jumping with pleasure, can be added to reinforce the visual experience.
References
Ammar, M. B., Neji, M., Alimi, A. M., & Gouardères, G. (2010). The Affective tutoring system. Expert Systems with
Applications, 37, 3013-3023.
Basu, T., & Murthy, C. A. (2012). Effective text classification by a supervised feature selection approach. In 2012 IEEE 12th
International Conference on Data Mining Workshops (pp. 918-925). doi:10.1109/ICDMW.2012.45
Bangor, A., Kortum, P., & Miller, J. A. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale.
Journal of Usability Studies, 4(3), 114-123.
Chen, Y. S., Kao, T. C., & Sheu, J. P. (2003). A Mobile learning system for scaffolding bird watching learning. Journal of
Computer Assisted Learning, 19(3), 347-359.
Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social
psychology, 17(2), 124-129.
Eyharabide, V., & Amandi, A. (2012). Ontology-based user profile learning. Applied Intelligence, 36(4), 857-869.
Ezhilarasi, R., & Minu, R. I. (2012). Automatic emotion recognition and classification. Procedia Engineering, 38(0), 21-26.
Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82-89.
Gunes, H., & Piccardi, M. (2007). Bi-model emotion recognition from expressive face and body gestures. Journal of Network and
Computer Applications, 30, 1334-1345.
Hsu, K. C., Lin, H. C. K., Lin, I. L., & Lin, J. W. (2014). The Design and evaluation of an affective tutoring system. Journal of
Internet Technology, 15(4), 533-542.
Islam, A. K. M. N. (2013). Investigating e-learning system usage outcomes in the university context. Computers & Education, 69,
387-399.
Infante, C., Weitz, J., Reyes, T., Nussbaum, M., Gómez, F., & Radovic, D. (2009). Co-located collaborative learning video game
with single display groupware. Interactive Learning Environments, 18, 177 -195.
Kerr, M. S., Rynearson, K., & Kerr, M. C. (2006). Student characteristics for online learning success. The Internet and Higher
Education, 9(2), 91-105.
Liaw, S. S., Chen, G. D., & Huang, H. M. (2008). Users’ attitudes toward Web-based Collaborative learning systems for
knowledge management. Computers & Education, 50, 950-961.
Lin, H. C. K., Wang, C., H., Chao, C. J., & Chien, M. K. (2012). Employing textual and facial emotion recognition to design an
affective tutoring system. The Turkish Online Journal of Educational Technology, 11(4), 418-426.
Lin, H. C. K., Chen, N. S., Sun, R. T., & Tsai, I. H. (2012). Usability of affective interfaces for a digital arts tutoring system.
Behaviour & Information Technology, 33(2), 105-106.
Lin, H. C. K., Hsieh, M. C., Loh, L. C., & Wang, C. H. (2012). An Emotion recognition mechanism based on the combination of
mutual information and semantic clues. Journal of Ambient Intelligence and Humanized Computing, 3(1), 19-29.
Lin, H. C. K., Wu, C. H., & Hsueh, Y. P. (2014). The Influence of using affective tutoring system in accounting remedial
instruction on learning performance and usability. Computers in Human Behavior, 41, 514-522.
Lin, H. C. K., Tsai, S. C., Cheng, Y. C., Chao, C. J., & Su, S. H. (2014). Usability evaluation of affective tutoring systems on web
page. Mitteilungen Klosterneuburg, 64(6), 27-40.
Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), 1-167.
146
Lu, C. Y., Lin, S. H., Liu, J. C., Cruz-Lara, S., & Hong, J. S. (2010). Automatic event-level textual emotion sensing using mutual
action histogram between entities. Expert Systems with Applications, 37(2), 1643-1653.
Mao X., & Li, Z. (2010). Agent based affective tutoring systems: A Pilot study. Computers & Education, 55, 202-208.
Martin, F., Pastore, R., & Snider, J. (2012). Developing mobile based instruction. Techtrends: Linking Research and Practice To
Improve Learning, 56(5), 46-51.
Metri, P., Ghorpade, J., & Butalia, A. (2012). Facial emotion recognition using context based multimodal approach. International
Journal of Emerging Sciences, 2(1), 171-182.
Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen, & R. L. Mack (Eds.), Usability Inspection Methods. New York, NY: John
Wiley & Sons.
Peng, H. Y., Chuang, P. Y., Hwang, G. J., Chu, H. C., Wu, T. T., & Huang, S. X. (2009). Ubiquitous performance-support system
as Mindtool: A Case study of instructional decision making and learning assistant. Educational Technology & Society, 12(1), 107120.
Picard, R. W. (2000). Affective computing. Cambridge, UK: MIT Press.
Picard, R. W., & Klein, J. (2002). Computers that recognise and respond to user emotion: Theoretical and practical implications.
Interacting with Computers, 14(2), 141-169.
Sarrafzadeh,A., Alexander, S., Dadgostar, F., Fan, C., & Bigdeli, A. (2008). “How do you know that I don’t understand?” A Look
at the future of intelligent tutoring systems. Computers in Human Behavior, 24, 1342-1363.
Yan, J., Bracewell, D. B., Ren, F., & Kuroiwa, S. (2008). The Creation of a Chinese emotion ontology based on HowNet.
Engineering Letters, 16(1), 166-171.
147