DEVELOPMENT OF EFFICIENT
ALGORITHMS FOR
BIOFEEDBACK SIGNAL
PROCESSING USING EMBEDDED
SYSTEM
THESIS
Submitted
In fulfilment of the requirements of the degree of
DOCTOR OF PHILOSOPHY
By
TANU SHARMA
PHDENG10021
Supervised by
DR. BHANU KAPOOR
Department of Computer Science and Engineering
CHITKARA UNIVERSITY
HIMACHAL PRADESH, INDIA
i
CHITKRA UNIVERSITY, HIMACHAL PRADESH
DECLARATION FORM BY THE STUDENT
I hereby certify that the work which is being presented in this thesis
entitled “Development of efficient algorithms for biofeedback
signal processing using embedded system” is for fulfilment of the
requirement for the award of degree of Doctor of Philosophy
submitted in Chitkara University, Barotiwala ,Solan, Himachal
Pradesh, India is an authentic record of my work carried out under the
supervision of Dr. Bhanu Kapoor, Professor, Chitkara University,
Himachal Pradesh,India.
The work has not formed the basis for the award of any other degree
or diploma, in this or any other institute or university. In keeping with
the ethical practice in reporting scientific information, due
acknowledgements have been made wherever the findings of others
have cited.
(Signature)
(Tanu Sharma)
ii
CHITKRA UNIVERSITY, HIMACHAL PRADESH
CERTIFICATE FROM SUPERVISOR
I hereby certify that the thesis entitled “Development of efficient algorithms
for biofeedback signal processing using embedded system” submitted by
TanuSharma,Regd.No.PHDENG10021
to Chitkara University, Barotiwala ,
Solan, Himachal Pradesh, India in fulfilment for the award of degree of Doctor of
Philosophy is a bona fide record of research work carried out by her under my
supervision .The content of this thesis ,in full or in parts,have not been submitted
to any other Institution or University for the award of any degree or diploma.
(Signature)
Dr.Bhanu Kapoor
iii
ACKNOWLEDGEMENT
To discover, analyze and to present something new is to venture on an unknown
path towards an unexplored destination is an arduous adventure unless one gets a
true torchbearer to show the way. This research work, being an illuminative
research would not have been possible without the guidance my guide, Dr.Bhanu
kapoor, Professor, Chitkara University, Himachal Pradesh,India for his constant
help and motivation which helped me a lot in handling this thesis and completing
it in due time.
I wish to express my sincere thanks to Honourable Chancellor Dr.Ashok Chitkara,
Pro-Chancellor Dr. Madhu Chitkara for giving opportunity to contribute in area of
research. I am very much thankful to Vice Chancellor Brig(Dr)RS Grewal. A
special thanks to Dr. Varinder Kanwar and Dr. Rajneesh Sharma, who always
motivated and encouraged to achieve goals. I must acknowledge Dr. Sudhir
Mahajan, it was under his tutelage that I developed a focus and became interested
in vision.
A thanks also goes to those who provided me with statistical advice at times of
critical need :Ms.SapnaSaxena, Ms Disha,Ms.Neha Kishore and Mr.Ashok
Kumar. My family who encouraged me and prayed for me throughout the time of
my research, I would also like to thank my family for the support they provided
me in my life.This thesis is heartily dedicated to my daughter “Anaaya” who gave
me time to do research by giving her precious mother-daughter time.
Last but not the least; I would like to thank God for not letting me down and
showing me the silver lining in the dark clouds.
Tanu Sharma
iv
LIST OF PUBLICATIONS
Published/Presented
•
SHARMA, T. & KAPOOR, B.2014 Data analysis by using machine
learning algorithm on controller for estimating emotions.,"International
Journal on Computational Science & Applications(IJCSA), Published
volume 4,number 6. ), vol. 4, pp. 19-31, DOI:10.5121/ijcsa.2014.4602
•
SHARMA, T. & KAPOOR,B.2014, Emotion prediction by using
intelligent machine learning algorithms . “International Journal of
Applied Engineering Research (IJAER)(In process of publication)
•
SHARMA, T. & KAPOOR, B. Emotion estimation of physiological
signals by using low power embedded system.
Proceedings of the
Conference on Advances in Communication and Control Systems-2013,
2013. Atlantis Press.
•
SHARMA, T. & KAPOOR, B. Intelligent data analysis algorithms on
biofeedback signals for estimating emotions. Optimization, Reliabilty,
and Information Technology (ICROIT), 2014 International Conference on,
2014. IEEE, 335-340.
Paper Accepted
•
SHARMA, T. & KAPOOR, B.2014,“Estimation of emotion through biosignals by using portable and low power embedded system”, International
Journal of Scientific & Engineering Research (ISSN 2229-5518)
•
SHARMA, T. & KAPOOR, B .2014 "Energy Efficient Emotion Prediction
by Using Intelligent Machine Learning Algorithms on Controller"
v
International
Journal
of
Electrical
and
Computer
Engineering
(IJECE)(ISSN: 2088-8708)
•
SHARMA, T. & KAPOOR, B.2014,” Machine learning algorithm on
controller to predict emotions” Proceedings of the Conference IEEEICCIC 978-1-47 99-3974-9,2014
•
SHARMA, T. & KAPOOR, B.2014."Detection and Prediction of emotions
by using hybrid Machine learning algorithms in embedded System"
International Conference on Recent cognizance in wireless communication
& image processing-ICRCWIP-2014.
Paper communicated
•
SHARMA, T. & KAPOOR, B.2015," Cost efficient emotion prediction
system by using intelligent machine learning algorithms on embedded
system"
International
Journal
of
Biomedical
Engineering
and
Technology.( ISSN: 1752-6426)
•
SHARMA, T. & KAPOOR, B.2015, " Energy efficient emotion prediction
system by using intelligent machine learning algorithms on embedded
system" Indian Journal of Science and Technology(ISSN: 0974-6846) IC
Value : 5.02
•
SHARMA, T. & KAPOOR, B.2015, Intelligent system Predicting
emotion" Journal of Information Science and Engineering (ISSN: 10162364)
vi
ABBREVIATIONS
GSR
Galvanic Skin Response
BVP
Blood Volume Pulse
ANS
Autonomous Nervous System (ANS)
R
Resistance
V
Voltage
EEG
Electroencephalograph
EMG
Electromyography
EDR
Electrodermal response
PGR
Psychogalvanicreflex
SCR
Skin conductance response
I
Current
K
Kilo
A/D
Analog to digital converter
LCD
Liquid Crystal Display
µc
Microcontroller
PEROM
Programmable and erasable read only memory
I/O
Input output
RAM
Random Access Memory
CPU
Central process unit
VCC
Supply voltage
GND
Ground
ALE
Address Latch Enable
PROG
program pulse input
vii
PSEN
Program Store Enable
EA
External Access Enable
VPP
Programming enable voltage
SFR
Special Function Register
LED
Light Emitting Diode
DC
Direct Current
viii
LIST OF TABLE
Table 2.1
External Components…………………………………. 46
Table 2.2
Age-related ranges of heart beat....................................
53
Table 2.3
Different Types of Plethysmographs………………….
54
Table 2.4
Details of each Pin…………………………………….
68
Table 2.5
Voltage Reference Generator Bit................................... 73
Table 3.1
Maximum a Posteriori” (MAP) Decision Rule……….. 96
Table 3.2
Different ranges of the psychophysiological signal…... 97
Table 3.3
Calculations…………………………………………… 98
Table3.4
Comparative study…………………………………….
104
Table 4.1
Change in Autonomic Activity………………………..
109
Table 4.2
Questions with Emotional Content…………………...
113
Table 4.3
Questions with Neutral Content………………………
114
Table 5.1
Budget for the Project…………………………………
137
Table 5.2
Comparison of the Existing Machines and Proposed
Work………………………………………………….
ix
150
LIST OF FIGURES
Page No.
Fig.1.1
Nervous System……………………………………………..
Fig.1.2
Diagrammatical view if ANS with biofeedback……………. 4
Fig.1.3
Branches of the Biofeedback technology…………………...
6
Fig.1.4
Emotional Intelligence Branches……………………………
9
Fig.1.5
Non-adaptive method Source……………………………….
12
Fig.1.6
Adaptive method Source……………………………………. 13
Fig.1.7
Intelligent method…………………………………………...
15
Fig.1.8
State Transitions…………………………………………….
16
Fig 1.9
Significant moments in the history of biofeedback................
20
Fig. 2.1
Block Diagram of Conventional Biomedical
Instrumentation System..........................................................
4
35
Fig. 2.2
System Design Process...........................................................
39
Fig. 2.3
Product Design of the Biofeedback system............................
40
Fig. 2.4
Proposed acquisition system of physiological data and
detect emotions.......................................................................
41
Fig. 2.5
Wire Protocol.......................................................................... 44
Fig. 2.6
Circuit Diagram......................................................................
45
Fig. 2.7
Skin Anatomy.........................................................................
47
Fig. 2.8
Skin conductance measured through the sweat glands of
finger tips................................................................................
49
Fig. 2.9
Voltage Divider....................................................................... 51
Fig. 2.10
Structure of the heart............................................................... 52
Fig. 2.11
Arrangement of a plethysmograph.........................................
x
56
Fig. 2.12
Relative absorption levels of infrared light of skin................. 57
Fig. 2.13
Representation of the Photoplethysmograph waveform......... 57
Fig. 2.14
Arrangement of light source and light sensitive detector:
Transmittance method ............................................................ 58
Fig. 2.15
Arrangement of light source and light sensitive detector:
Reflectance method ................................................................ 59
Fig. 2.16
Temperature Sensor................................................................
62
Fig. 2.17
EZ430-F2013 - MSP430 16-bit microcontroller USB Stick..
64
Fig. 2.18
Architectural view of MSPF2013...........................................
65
Fig. 2.19
PCB diagram........................................................................... 66
Fig. 2.20
Pin Diagram of MSP430F2013(Mainoddin and Usha, 2014)
68
Fig. 2.21
Analog-To-Digital Conversion..............................................
71
Fig 3.1
Sequence of emotion………………………………………...
82
Fig 3.2
Plutchik’s Model …………………………………………..
86
Fig 3.3
Russell’s Model ……………………………………………
86
Fig 3.4
Generic process of emotion identification…………………
88
Fig 3.5
Machine learning approach………………………………..
90
Fig.3.6
Classification Methods…………………………………….
91
Fig 3.7
Two Conditional Events …………………………………..
95
Fig 3.8
Emotion States diagram with transition probability..............
101
Fig. 4.1
Two-dimensional emotion models with four quadrants……
110
Fig. 4.2
Data analysis and subject assessment for emotion estimation 117
Fig. 4.3
Variation in GSR……………………………………………
122
Fig. 4.4
Variation in Blood volume Pulse (BVP)……………………
123
Fig. 4.5
Variation in Temperature…………………………………… 123
xi
Fig. 5.1
ProComp5 Infiniti System T7525M………………………..
131
Fig. 5.2
GSR/Temp 2x………………………………………………
132
Fig.5.3
Holter Monitor………………………………………………
133
Fig. 5.4
The Digital HRM……………………………………………
134
Fig. 5.5
Bedside Monitor MPM 5533……………………………….
135
Fig. 5.6
Cardiomon CCM900……………………………………….
136
Fig. 5.7
Bioview HRV monitor……………………………………..
136
Fig. 6.1
The electrode placement …………………………………..
142
Fig. 7.1
Basic Biofeedback System…………………………………
149
Fig. 7.2
Work flow of implemented Algorithms…………………….
151
xii
LIST OF ALGORITHMS
Algorithms2.1
Efficient Algorithm for A-D conversions.........................
72
Algorithm 3.1
Best first search algorithm………………………………
89
Algorithm 3.2
HYBRID-NAV-MAR…………………………………..
101
Algorithm 5.1
Purposed and applied Sleep/awake algorithm…………... 128
xiii
CONTENTS
INTRODUCTION
1
1.1
Introduction to the research work…………………...
1
1.2
Autonomic nervous system………………………….
3
1.3
Biofeedback
5
1.3.1 Biofeedback Modalities……………………….
6
1.4
Psychophysiology……………………………………
8
1.5
Psychophysiology and Emotions……………………
8
1.6
Biofeedback Embedded System…………………….
10
1.7
Issues of biofeedback devices………………………
10
1.7.1
Multiple Interventions....................................
11
1.7.2
Smart/Intelligent Devices...............................
11
Chapter - 1
1.8
1.7.3 Cost…………………………………………….
14
Solutions to the Problem…………………………….
14
1.8.1
Biofeedback modalities intervention...............
14
1.8.2
Research Contribution by Integrating ………
15
Intelligent Methods for Data Analysis
and Decision Making
1.8.3
Contributions of the work to make the
design cost-
1.9
17
effective and power- efficient
Need of the work…………………………………….
xiv
17
1.10
Objective of the work………………………………..
19
1.11
Literature Survey ……………………………………
19
1.12
Organization of thesis……………………………….
31
Chapter – 2
EMOTION DETECTION ARCHITECTURE &
33
SYSTEM
2.1
Introduction………………………………………….
33
2.2
Architectural View of Conventional Model…………
34
2.2.1
Subject……………………………………….
35
2.2.2
Transducer/Sensor…………………………...
35
2.2.3
Signal Conditioner…………………………..
36
2.2.4
Display System………………………………
36
2.2.5
Control System………………………………
36
2.2.6
Working of conventional model…………….
37
2.3
Problems in Conventional Architecture……………..
38
2.4
Proposed Model (Emotion Detection Model)……….
38
2.4.1
System Design Process………………………
38
2.4.2
Steps for Designing the Emotion Detection
40
System
2.5
Proposed Design and How It Works………………
41
2.6
Circuit Diagram of the Proposed System ………….
44
2.7
SIGNAL PROCESSING…………………………….
46
2.7.1
46
Connectivity of input signal with sensors
2.7.1.B Blood Volume Pulse (BVP)………………..
xv
55
2.8
2.9
Chapter – 3
2.7.1. C Skin Temperature…………………………
60
Microcontroller Overview
63
2.8.1
MSP430F2013 Architecture…………………
65
2.8.2
Modular design ……………………………..
66
2.8.3
The SD16 A Sigma-Delta ADC……………..
70
Conclusion…………………………………………… 77
EMOTION INTELLIGENCE METHODOLOGY
78
AND ANALYSIS
3.1
Introduction………………………………………….
78
3.2
Background and Related Works…………………….
78
3.3
Emotions and Emotional intelligence……………….
79
3.4
Methods for Recognizing Emotions…………………
82
3.5
Emotion Model………………………………………
84
3.6
Emotion Estimation Methodology
87
3.6.1
Generic process of emotion identification
87
3.6.2
Machine learning algorithms for emotion
90
prediction
3.6.2.A Naïve Bayes…………………………………
91
3.6.3
100
HYBRID-NAV-MAR………………………..
3.6.4. Implementation of HYBRID-NAV-MAR
100
3.6.5
Markov Model……………………………….
102
Comparative study…………………………………..
105
EMPIRICAL STUDY AND ANALYSIS
107
4.1
Generalities…………………………………………..
107
4.2
Benchmark construction of experimentation ……….
110
4.2.1
Experimental Study_ S0_1…………………..
110
4.2.2
Experiment_ S0_2…………………………… 113
4.2.3
Experiment_ S0_3…………………………… 115
3.7
Chapter – 4
xvi
4.3
4.4
Chapter – 5
5.3
5.4
Chapter – 6
DATA ANALYSIS …………………………………. 116
4.3.1
Data Acquisition……………………………..
117
4.3.2
Normalization and feature extraction………
118
4.3.3
Classification Methods……………………..
119
4.3.4
Observations
121
CONCLUSIONS
123
POWER EFFICIENCY AND SYSTEM COST
125
5.1 Introduction
125
5.2 Energy efficiency
126
5.2.1
Proposed Low-Power model
127
5.2.2
Power Analysis
129
Expenditure Effectiveness
130
5.3.1
Biofeedback Machines
130
5.3.2
GSR2/Temp Biofeedback System
131
5.3.3
ECG MACHINES
132
5.3.4
134
5.3.5
Multiparameter Bedside Monitor (MPM
5533)
Cardiomon CCM900
5.3.6
BioView HRV Monitor
136
5.3.7
Resources of Proposed work
137
135
Conclusion
138
EMOTION RECOGNITION DEVICE USER
139
GUIDE
6.1
General guidelines
139
6.2
Technical background
140
6.3
Using the skin response in the biofeedback training
141
6.4
Hardware/Software Set-up
142
6.4.1
143
Biofeedback training by using the proposed
xvii
device
6.4.2
Standard Controls
144
6.4.3
Sensor features:
144
6.4.4
Precautions and safety
145
CONCLUSION
146
7.1
Introduction
146
7.2
Conclusions of the study
147
7.3
Limitations of the Study
149
7.4
Future scope
150
REFERENCES
153
Appendix – I
166
Chapter - 7
168
Appendix - II
174
Appendix – III
xviii
ABSTRACT
The most unstoppable and uncontrollable aspect of the mental state
of humans is emotion. The emotions cannot be changed by a device but an
effort can be made to predict or estimate emotion. The estimating of
emotions is likely to be helpful because emotion is regarded as one of the
profound factors, which influence the everyday life activities that can help
us judge or choose between the available choices; and can alert us to avoid
danger. The estimation of emotions can be done by using different
methodologies, such as detecting facial expressions, voice estimation in
stress, and measuring the changes in the physiological signals
(temperature, heart rate, and GSR). This research focuses on the
physiological parameters, which are measured using the sensors to decide
the emotive status of a human being. The signals obtained using these
measurements are authentic; though not alterable, these cannot be
concealed during the measurement process, as these are generated due to
the activation of the sympathetic nerves of Autonomous Nervous System
(ANS).The
bio-sensors
have
the
advantage
of
monitoring
the
physiological parameters of the body (the physiological parameters are
directly controlled by the autonomous nervous system and are affected by
emotions).
When different emotions are experienced by a human body then
physiological
changes
are
observed
in
terms
of
the
skin
conductance(GSR), blood volume pulse (BVP), and temperature. The
proposed system design is a low power, portable, and cost effective
embedded system for estimating the emotions based on collected data. It
senses different emotions by using the machine learning algorithms
embedded within a microcontroller (MSP430F2013).The Naïve Bayes
classifier is one of the probability models that incorporates the class
conditional assumptions and gives an output in the form of predicted
emotion based on the data collected in the past. The algorithm’s accuracy
xix
improves as more the system collects more data. This predicted emotion
by the Bayesian method becomes the input for the next machine learning
algorithm, the Markov model. The Markov model is also implemented in
microcontroller for predicting the emotion based on collected data. That
means that both the algorithms are fused together to make them work in a
hybrid form. For a proper fusion, a new algorithm called HYB-NAVMAR was designed to support the hybrid form. Our experimental results
with the implementations of these algorithms on a TI MSP430 shows that
the emotion can be predicted reasonably accurately as based on the
collected data and the accuracy of prediction improves as the system
collects
more
xx
data.
CHAPTER 1
INTRODUCTION
1
Chapter 1
INTRODUCTION
A foundation and the body of knowledge on which this research work stands are
discussed
in
this
psychophysiology,
chapter.
the
It
relation
introduces
of
the
basics
psychophysiology
of
biofeedback,
with
emotion,
psychophysiological signals using biofeedback technique, and biofeedback
embedded systems which detect the emotions of humans. This chapter also gives
an overview of research and literature survey.
1.1 Introduction to the research work
Emotion is regarded as one of the key factors and it influences everyday
life activities that can help us judge or choose between the available
choices; and can alert us to avoid danger etc. Most of the time emotion is
acquired unwittingly in human beings depending upon the situation and
circumstances. The recognition of emotion is therefore necessary to know
more about the human behaviour(Elfenbein, 2002).
As a result, emotion recognition has become popular in the field of human
computer interface community, which can play an important role in the
improvement of the interaction between humans and computers. If the
emotion recognition and elicitation is done in an effective way, it is
possible to make humans interact with computers the same way the
humans interact with each other. Human Computer Interaction (HCI) and
Human Robotic Interaction (HRI) are the major areas where emotional
intelligence can be implemented(Cowie et al., 2008).
Emotion recognition by device can be done in various ways, such as
affective computing techniques and psychophysiological measurement.
Affective computing is applicable in the recognition and synthesis of facial
expression, voice inflection, gestures, and postures(Sauter et al.,
2010).However, this method is not reliable, as the individuals can hide
their emotions and their affective states of mind. In order to overcome this
2
problem, the use of psychophysiological measurement was introduced.
This technique is now being used actively to acquire such emotional states
using different biosensors, that is, Galvanic Skin Response (GSR), Blood
Volume Pulse (BVP), Electromyography (EMG), Electrocardiography
(ECG), temperature, and few others. The signals obtained using these
measurements are authentic, thus not alterable and cannot be concealed
while measuring them, as they are generated due to the activation of
sympathetic nerves of Autonomous Nervous System (ANS)(Hagemann et
al., 2003). Signals from the biosensors are then processed using different
signal processing techniques and methods to convert these into a feasible
data
(commonly
stated
as
psychophysiological
data).
The
psychophysiological data are obtained by recording signals from the
human participants.
This research proposes machine intelligence that includes emotional
intelligence and expresses outcomes toward this goal: “developing a
machine's ability to recognize human emotional state from physiological
signals.” To make it work as machine, intelligent popular machine
learning algorithms are employed including the Naïve Bayes Classifier
and the Markov model. Both of these algorithms were implemented and
also in a hybrid form. The Naïve Bayesian used here include class
conditional independence assumptions that are based on probability
models. As such, the predictions about emotion is based on data collected
by the system in the past and should improve with increase in collected
data.
The output of Naïve Bayes comes in the form of emotion. For the future
prediction of next emotion is handled by the Markov model. It depends
upon the accuracy of the prediction by the Bayesian method. In order to
combine both algorithms, a new algorithm has been designed called the
HYB-NAV-MAR. Some of the psychophysiological signals (bio-signals)
like GSR and BVP have been used inputs for the designed embedded
system. The extraction of psychophysiological data was done by using
bio-sensors. As part of the designed embedded system, a portable and a
3
cost-effective
microcontroller
from
Texas
Instruments,
MSP430F2013,(Davies, 2008) was used in this research. The designed
system has capability and intelligence to monitor patient (subject) different
level of emotions automatically and simulation of these emotions can be
seen through a computer.
1.2
Autonomic nervous system
The nervous system is originally split into two divisions: : the peripheral
nervous system and the central nervous system (Loewy and Spyer,
1990).The central nervous system consists of all of the neurons and this
receives information from each organ and tissue within the body. It also
analyzes the information and sends appropriate responses back to the
organs. ANS(Autonomic Nervous System) acts as a control system within
the body, maintaining and carrying out all tasks that fall below the level of
consciousness. This involves tasks such as the process of digestion, the
beating of the heart, and the respiratory rate.Although this can be
consciously controlled, under most circumstances it is an automated and
non-conscious process. In order to maintain the balance there are two
divisions of the autonomic nervous system: the parasympathetic division
and the sympathetic division. The sympathetic nervous system kicks into
gear when energy expenditure is necessary (ex: during times of excitement
or stress). Because of this, it has earned the nickname the “fight or flight
response.” This system can do several things such as increasing blood
pressure and heart rate, stimulating the secretion of adrenaline, and
increasing the blood flow to the skeletal muscles(Gabella, 2001).
The parasympathetic nervous system (shown in Fig.1) returns our body
back to homeostasis. It kicks in when energy reserves can be conserved
and saved for later use. This system is capable of increasing salivation,
digestion, and storage of glucose and it can slow down heart rate, as well
as decrease respiration.
4
Fig 1.1 Nervous System(Brodal, 2004)
The autonomic nervous system is the subsection of the peripheral nervous
system that regulates body activities that are usually not under conscious
control. Biofeedback method completes the loop among autonomic
autonom
functions and conscious awareness.
Spinal
Cord
Brain
Effectors
Human Senses
and/or Assistive
Biofeedback providesprovides “Close Loop”
Fig1.2 Diagrammatical view if ANS with biofeedback(Corwin
biofeedback(Corwin and Williams,
2008)
The primary emotions of fear, surprise, sadness, disgust, anger, joy, etc.,
can be mixed to produce more complex emotional experiences. The left
hemisphere of the brain primarily processes positive emotions. Negative
emotions are processed in the right hemisphere.
hemisphere. The amygdale provides a
"quick and dirty" pathway for the arousal of fear that bypasses the cerebral
5
cortex. Body changes that occur during emotion are caused by hormone
adrenaline and activity in the ANS.
The sympathetic branch of the ANS is primarily responsible for arousing
the body; the parasympathetic branch for quieting it. Emotional arousal
involves changes in heart rate, blood pressure, GSR, etc(Westerink et al.,
2008).
Recognizing emotions is not just dependent upon facial expressions.
Rather, there are many more kinds of signs or cues, such as: biofeedback
modalities, voice, gestures, and actions. Numerous biofeedback modalities
which exist are; temperature change retrieved via fingertip thermometers;
electromyography (EMG) in which muscle contraction is calculated;
resistance of skin influenced by sweat is evaluated (GSR); cardiovascular
activity is measured via heart rate.
Emotion depends upon the activities in the ANS of the individual. As
emotion varies, it brings variations in sympathetic nerves of ANS in
excited condition. The sweat is secreted from the sweat glands and with
this a change in GSR is observed. The heart rate is also under constant
control of autonomic nervous system, which also affects heart rate(Zald,
2003).
1.3 Biofeedback
Biofeedback(Schwartz and Andrasik, 2003) is a treatment technique in
which people are trained to improve their health by using signals from
their own bodies. It works on the principle that we have the innate ability
and potential to influence the automation functions of our body through
the exertion of will and mind. By using the biofeedback, a person can
learn to change their body’s reaction, in a way that improves strength of
body, and this can be done by using an electronic device that measures and
indicates diverse things that are happening inside the body. Two branches
of the biofeedback area can be defined: Ubiquitous-biofeedback and
Clinical-biofeedback.
6
Biofeedback
Clinical
Biofeedback
Ubiquitous
Biofeedback
Fig. 1.3: Branches of the Biofeedback technology(Kim et al., 2006)
Clinical biofeedback is performed under the direction of a health
practitioner, usually in a clinical setting. The subject is guided during
exercises where she or he is made responsive to key physiological
parameters and assisted through different procedures to control them.
These sessions are normally bound by time and the setting in which they
are performed (typically a clinic).
Ubiquitous biofeedback, on the other hand, is a continuous process that is
neither bound by time nor setting. For instance, a system designed to
monitor mental stress can be activated almost anytime and anywhere the
individual feels that her or his mental stress level tends to increase (e.g.
work, classroom, and car)(Kim et al., 2006).
1.3.1
Biofeedback Modalities
Several physiological processes can be observed by biofeedback
applications. By using various sensors for different parameters and the
possibility of recording every parameter at a time can be achieved.While
designing the biofeedback device one should take care that each
biofeedback modality has different displays that are clearly specified, so
that one doesn’t get puzzled.(Schwartz and Andrasik, 2003) Some of the
more common modalities are given below:
GSR (Galvanic Skin Response): Galvanic Skin Response (GSR)
(Bitterman and Holtzman, 1952, Westeyn et al., 2006)is a measure of
7
sweat gland activity. Most people are familiar with having cold, clammy
hands under stressful circumstances, such as meeting new people or
having to perform before an audience. The coldness comes from
constriction of the smooth muscles surrounding the blood vessels
(measured by hand temperature), while the dampness is caused by eccrine
gland activity. The eccrine glands secrete a salty solution in response (Shi
et al., 2007)to emotions and stress.
BVP (Blood Volume Pulse): Blood Volume Pulse is measured with the
phasic change in blood volume which varies with each heartbeat, heart
rate, and heart rate variability (HRV).It consists of beat-to-beat differences
in intervals between successive heartbeats. It is captured by technique
called
Photoelectric
Plethysmography,
also
known
as
photoplethysmography (PPG).It is a non-invasive method used to measure
the heart rate by determining the blood volume changes in the skin’s
periphery (finger-tip, ear-lobe) by the photo-electric method(Poh et al.,
2011).
Temperature: It is measured by sensors attached on the ring fingers. The
temperature modality indicates the contraction of the smooth muscles
around the blood vessels and determines that how much blood reaches the
fingertips. Few sensors of measuring temperature are: Thermistors,
Thermocouple, Resistance temperature detectors, etc. The sensor is
selected depends upon the requirements(Lakin, 1998).
Neurofeedback: Brain waves are calculated by the electroencephalograph
(EEG). EEG is comprised of various bandwidths: Theta (4-7 Hz), Alpha
(8-12 Hz), Beta (13-20 Hz), and Gamma (21+). In general EEG training is
to implement range of motion among bandwidths, so that the client knows
what every bandwidth feels like and how to use every state for its
characteristic benefits. Normally, beta and gamma are useful for directed
movement and getting things done; alpha is useful in situations where
8
relaxed vigilance (such as meditation); and theta is useful for creative, day
dreamy generation of imagery (theta is sometimes called start of
unconscious)(Evans and Abarbanel, 1999).
EMG (electromyography): Muscle activity is exacted by the EMG, which
detects the electrical activity occurring within certain muscles, specifically
the trapezius (shoulder) and temporalis (jaw and scalp) muscles. Muscle
tension indicates stress. For example, it is common for people to react to
the anxiety of anger by clenching their teeth. To measure EMG, the skin is
cleaned and adhesive sensors with a gel are attached to the shoulder
muscles(Lucovnik et al., 2011).
1.4 Psychophysiology
Psychophysiology is the branch of psychology which is concerned mainly
with the physiological response to mental processes. Psychophysiology
has been defined as the interaction of mind and body and is the study of
different physiological manifestations (e.g. facial expression, heart beat
etc.) of emotional states.(Andreassi, 2000) Physiological psychology (i.e.
psychophysiology) studies how physiological variables such as brain
stimulation (independent variables) that can affect other (dependent)
variables. The independent variable can be considered as a stimulus (e.g.,
an image of bleeding man) and the dependent variable can be
physiological measures (e.g., heart rate, skin conductance(Hubbard et al.,
2002).. This research here focuses on dependent variables such as the
GSR, the BVP, and the body temperature.
1.5 Psychophysiology and Emotions
Emotions are generated implicitly when any stimulus elicits some feeling,
which strikes on the brain (psychology) and makes alterations on physical
response. The interpretation of psychophysiological signals along with
emotion has strong correlation. Moreover, in psychophysiology, it has
been found that emotions and physiology (skin conductance, muscle
9
contraction and relaxation, heartbeat, blood pressure etc.) are closely
c
related and influence each other(Laparra-Hernández
other
Hernández et al., 2009).
2009) Emotion
recognition from psychophysiological measurement and classification of
these emotions can be used as input for robotic behaviour. The use of
psychophysiological data on humans interacting
interacting with robots is a more
recentand emerging method(Larsen
method
et al., 2008),, however, there is only
limited research conducted on the use of psychophysiological measures.
Emotions
ons can be defined as a mental state that occurs impulsively without
any effort and is complemented by physiological variations. It is precisely
produced by cognitive procedure, subjective feelings, physiological
arousal, motivational trends, and developmental
developmental reactions. Humans can go
through different emotions on a higher scale that can regulate through
various emotional experiences; such types of feature are named as
Emotional Intelligence (EI). (Mayer et al., 2001)Emotional
Emotional intelligence
has four features that are also known as branches (Ciarrochi et al., 2000)
These branches are:
Fig.1.4: Emotional Intelligence Branches
•
Perceiving (observing) emotion
•
Usage of emotions to facilitate thought
•
Managing emotions
•
Understanding emotions
This work explains an application based upon perceiving emotions, as
perceiving emotion has the capability to identify emotion in oneself and
others. It can also identify the difference between honest and dishonest
10
emotions. An emotion consists of physiological changes, adaptive
behaviour, emotional expressions and emotional feelings. Emotions can be
disruptive, but overall they help us to adapt and survive.
1.6 Biofeedback Embedded System
Embedded systems (Noergaard, 2012)have witnessed an incredible growth
in the last few decades.. Almost all of the fast developing sectors like
automobile, aeronautics, space, rail, mobile communication, and electronic
payment
solutions
have
witnessed
increased
use
of
embedded
technologies. Embedded systems have greatly influenced the growth of
medical devices, making mechanization and miniaturization possible. The
use of embedded systems in these devices has metamorphosed these
instruments into portable, smarter and network-enabled devices with
sophisticated analytical capabilities, delivering high precision and
accuracy. (Hock, 2000a)
In this research hardware and software instrumentation development and
signal processing approach was used to detect the emotion level of a
subject (human). To check the device's performance, a certain set of
experiments were done.
1.7 Issues of biofeedback devices
Over the last decade, many researchers have carried out studies addressing
this affective sensing challenge, such as the attempts at emotion
identification through facial expression (Allen et al., 2001), body gesture
(Hock, 2000b), and speech processing (Ron, 1997) in isolation or in
combinations(Wheeler and Jorgensen, 2003). Among the diverse affective
sensing approaches, the monitoring and analysis of physiological signals is
measured as a particularly promising technique for affective assessment,
since these signals are inherently controlled by the subject’s ANS, which
means they are less susceptible to environmental interference or voluntary
masking .
11
Biofeedback devices are generally used exclusively in therapeutic settings.
This can be a disadvantage, as they are not available for patients (Wong et
al., 2001)for more than a few hours a day. We will next describe some
similar issues which were analysed by the researchers.
1.7.1 Multiple Interventions
The paucity of information concerning multiple biofeedback modalities
intervention(Koh et al., 2008) has provided an area for research. Issues of
existing multiple biofeedback modalities are their size (bulky), complex
circuit and complex design, which leads to high power consumption. In
personal cases, a patient can’t even think to buy such a device for personal
use at home. An example for such cases can be if any patient is paralyzed
and their caretaker wants to know emotion or any biofeedback, then they
have to take that patient to a hospital or they have to buy an expensive
device.(Kopka and Crawford, 2004)
1.7.2 Smart/Intelligent Devices
The problem of the control and correction of the human functional state is
of essential today. The rise of human interaction with sophisticated
equipment leads to the growth of nervous emotional load and formation of
steady stressful status, development of neurosis, and psychosomatic
diseases.
The
elimination
of
such
distresses
that
are
wholly
pharmacological is often unapproachable because of contra-indications. As
a result, psychophysiological methods of the person’s emotional state
correction have been widely developed. One of the largest groups of such
methods
is
indirect
biofeedback
treatment
(Drechsler
et
al.,
2007)(biofeedback training).
It is possible to split methods of emotional state correction and, in
particular, biofeedback training, into three category: non-adaptive,
adaptive and intelligent(Unakafov, 2009).
Non-adaptive methods: In such techniques, there is no mechanism of a
procedure’s correction depending on results of influence on the patient.
12
The composition of non-adaptive
non
method is presented in Fig.1.5, where
TCA is Training Control Algorithm, the basis of the technique.(Shenoy
technique.
et
al., 2006)
Fig.1.5: Non-adaptive method Source:(Unakafov,
(Unakafov, 2009)
Influence alteration is made by the expert (the psychologist, the
psychophysiologist
chophysiologist or the psychotherapist) based on an objective (the
control of a patient state) and/or a subjective advice (interrogation of the
patient). Common drawback of non-adaptive
non adaptive methods is their selective
efficiency in case of simple methods, such as a high load on the specialist
who is carrying out the practice in case of sophisticated methods(Clancy
methods
et
al., 2002).Essential
.Essential increase of biofeedback training efficiency is probable
at the expense of approach personalization: maintenance of the highest
patient’s sensitivity to perception of internal sensations, use of individually
selected modes of the biofeedback
biofeedback training to resolve these problems,
adaptive methods are use.(Lehrer
use.
et al., 2000).. They include mechanisms
13
of automatic modification of influence on the patient, depending on his/her
state.
Adaptive methods provide a very high effectiveness of training. In tough
cases flexibility of the technique appears deficient and the result depends
on the experience and professional skills of the specialist carrying out the
process.(Shenoy
(Shenoy et al., 2006)The
2006)The structure of an adaptive technique is
shown in Fig.1.6 below. Now, the requirement for highly skilled
psychophysiologists
gists is growing and it is not completely satisfied.
Fig. 1.6: Adaptive
A
method Source: (Unakafov, 2009)
Intellectual method:
method To compensate for the need of specialists, to
facilitate their work and raise its effectiveness, intellectual methods
(Unakafov, 2009)can
2009)can be used. The intellectual method includes the
method of training conditions specialist estimate (an artificial intellect,
AI/Machine learning algorithms).The formulation and understanding of
intellectual methods is a latest problem, which is weakly resolved at
present. There are some general ideas
ideas regarding formulation of intellectual
methods in a amount of sources (for example,(Ron,
example,
1997)),
), however, this
14
problem is not completely solved yet(Ron, 1997).In this research work, an
intelligent hardware-software biofeedback system is the focus and it uses
psychophysiological signals.
1.7.3 Cost
The use of biofeedback devices in therapeutic setting is expensive. As a
result of high cost, it becomes doubtful whether it is more effective than
the learning of relaxation techniques. Due to its therapeutic settings, one to
one sessions are conducted which can again make this method expensive.
In personal biofeedback devices, some of the costs for the technical
equipment are also high.(Brunelli et al., 2006)
1.8 Solutions to the Problem
A technical device might be a good solution for the issues described
above. The idea for an application would help create awareness about
emotions to find solutions. To automate the proposed method, it is
necessary to develop the mechanism that can estimate different
psychophysiological signals and detect the emotions of humans based on
the data it has collected in the past.
1.8.1 Biofeedback modalities intervention
This research has proposed a design and developed a low power
consuming, cost-effective embedded system through which humans can
measure and analyze different parameters of ANS (GSR, Temperature, and
Heart rate) of a person and display it on any of the output devices.
Multiple parameters of ANS are applied and implemented on the model,
like the GSR, heart rate, and temperature. MSP430F2013 is the
microcontroller that uses a lowpower architecture and improves the power
efficiency of system. The algorithm designed by researchers, the
sleep/awake algorithm, is also implemented for enhancing power
efficiency of the system. The device is intended for portable and energy-
15
efficiency will be of key importance for its usage over long periods of time
without a need to recharge or replace the battery.
1.8.2 Research Contribution by Integrating Intelligent Methods for
Data Analysis and Decision Making
Implementation of intelligent methods provides automatic development of
influence strategy and, if necessary, delivery of recommendations to the
specialist
list who is carrying out the process. Their structure is presented in
the Fig 1.7. The training process can be carried out in both modes: manual
(control under doctor) and automatic (with machine learning algorithms).
Thus, unlike the adaptive methods, which
which are only a convenient tool for
the specialist, intellectual methods will not only simplify his/her work, but
also replace the specialist.
Fig.1.7: Intelligent method
For data analysis part, we have implemented different machine learning
algorithm that run on the MSP430 microcontroller. These are:
1. Naïve Bayesian
2. Markov Model
3. HYB-NAV
NAV-MAR
16
Naïve Bayesian: This algorithm is applied to predict the emotional state
based on the data collected in the past. Bayesian is responsible for
decision-making and inferential statistics by using probabilities of various
emotions as seen over the past under certain circumstances. The number of
possibilities for the ANS data is extremely large so as the system trains
more the predictions are likely to get better.
Markov Model: Markov model is based on the Markov chain. It models
the state of a system with a random variable changing over time. The
Markov property suggests that the distribution for a variable depends only
on the distribution of the previous (past) state. In Markov, when changes
in states are observed those occurrences are called transitions. The
probability is associated with each state change and those changes are
called transition probabilities. The process starts in one of the above
shown states and moves successively from state to state(S-Stress, JJoyful, and C-Calm).This helped to predict future, based on current
emotion.
Fig.1.8: State Transitions
HYBRID-NAV-MAR: The output of the Bayesian algorithm acts as input
to the Markov model. The input for the algorithm was the different
possible events and the different state to have transition between events
(GSR/BVP/Temperature) and states (Stress/Joy/Calm). This is a new
17
algorithm which has helped to create linking between Bayesian and the
Markov model.
1.8.3 Contributions of the work to make the design cost-effective
and power-efficient:
This research proposes a biofeedback based system “emotion detection,”
which is a high quality, low-cost system answering the advanced needs of
the clinical systems. The pursuit of this research is also to explore the
techniques which may lead to the development of computer systems
enriched with affective awareness.
In order to make it cost effective, a circuit was designed accordingly and
total hardware/software to keep costs low. In order to improve the energyefficiency of the system, data processing algorithm using TI MSP430 have
been implemented and has also given a cost-effective and highly portable
solution.
1.9 Need of the work
When person confront with various demanding situations - our bodies
respond in much the same way as the "fight or flight response"(Jansen et
al., 1995).A person automatically prepares also to fight the stress or to run
from it. Sometimes, one needs to control different emotional situations
which can lead the person suffering them to dangerous situations, in both
the medium and short term.
During our life, as person confront the various stressors(Eckenrode, 1984)
that
occur
each
day,
we
respond
by constantly tensing
and
relaxing(Kahneman et al., 2004). Ultimately, after each instance of
tensing, we cease to return to our actual level of physiological relaxation.
Consequently, through the years we establish a stair-step pattern. We
adjust to rising levels of physiological action(Nagai et al., 2004). In so
doing, we drop familiarity with deeper levels of relaxation and get used to
greater levels of stress as the norm. This habituation to needless
18
physiological activity has a wearing consequence and can cause such
conditions as high blood pressure, headaches, digestive problems, and
other diseases(Chrousos, 2009). This method is used in analyzing
variables like heart rate, GSR, blood pressure and the occurrence of certain
patterns in electroencephalogram(Greenhalgh et al., 2010, Lin et al., 2012,
Chanel et al., 2009).
Classically, biofeedback exercises take place in a clinic under the
supervision of a doctor or therapist(Piepoli et al., 2011). Nonetheless, in
recent years, many works have challenged this classical notion by enabling
such exercises from various non-clinical settings (e.g. home, work,
car)(Meule et al., 2012). These systems are therefore aimed at achieving
geographical biofeedback ubiquity. However, to the best of our awareness,
no attempts have been made to celebrate the concept of non-clinical
biofeedback into a reference model. Such a model should provide an
abstract structural representation of the various components at play in a
typical non-clinical biofeedback application. Note that these applications
operate mostly independently from a clinician and interface directly with
the user. Therefore, specific adjustments and additions are required to
accommodate direct interface with the user. Consequently, the
formalization of the reference model provides a standardization of the
various components involved and the goal and scope of such systems. One
of the principle objectives of this thesis is to present this reference model.
Also, this model focuses on the concept of temporal biofeedback ubiquity.
To the best of our knowledge, biofeedback exercises in most existing
works are performed during time-bound sessions. A user of such systems
commits her or his entire concentration to the exercise. Nonetheless, we
advocate the principle of continuous monitoring of physiological data. The
users go about their day while the system operates in the background.
19
1.10 Objective of the work
The main objective of this work is to design and develop an embedded
system which can be used for analysis of emotions and other physical
activity parameters:
1. To design and develop a low-power consuming, cost-effective
embedded system through which we can measure and analyze different
parameters of a person's ANS (GSR/ temperature/heart rate/EEG) and
display it on any of the output devices.
2. Put together an accurate analog-to-digital domain conversion.
3. Implement intelligent data analysis algorithms on embedded processor
such as TI MSP430.
4. Implement energy-efficient data processing algorithm using TI
MSP430 in creating a cost-effective and highly portable solution
1.11 Literature Survey
The word ‘biofeedback’ was not coined until 1969, but the idea has been
known for thousands of years in the form of meditation and a variety of
yoga techniques(Yucha and Montgomery, 2008). Biofeedback was first
introduced in rehabilitation and physical therapy more than 35 years ago.
It appeared to be the effective support of the therapy, demonstrating
positive changes in function of patients in a lot of clinical conditions. For
example, yogis have been deliberately controlling their ANS (such as
slowing their heart rate or rising their body temperature) by analysing their
body’s performance. It is believed that the foundation of biofeedback
research was recognized in the 1930s when progressive relaxation
methods (Martin and Johnson, 2006) and autogenic training (Miu et al.,
2009) were established. These techniques involve a roitine practice that
lasts for a certain time gap (for example, 15 minutes) during which the
practitioner replicates a set of visualizations and releases tension in the
muscles to induce a state of relaxation. Such practices, supplemented with
information relating to the mind and body (collected by electronic
20
sensors), would later form the foundation for biofeedback(Morone and
Greco, 2007).
C. Jung
Relationship
between
psychology
and
physiology
H.S Black
Negative
feedback in
electronics
J. Basmanjian
EMG biofeedback
A.
Rosenblueth
N. Wiener
Feedback in
cybernetics
N.E. Miller
Operant
conditioning
of the
autonomic
nervous
system
functioning
J. Kamiya
Subjects can
increase
amplitude of
alpha waves
R.Caton
Bioelectrical
brain
activity
as result
of
stimuli
H. Berger
Electrical
correlates of
brain activity
in human,
EEG
(Caton,
1875)
1920s
1930s
1960s
1970s
B. Sterman
Sensorimotorrhythmprotocol
in epilepsy
Fig 1.9: Significant moments in the history of biofeedback
(Healey and Picard, 2005) : This paper provides the basis of designing of
an intelligent driving system that adjusts the driving condition by taking
stress levels of a driver into account. It provides various techniques for
collecting and analysing physiological data during real-world driving tasks
to determine a driver’s relative stress level. Following physiological
signals,
electromyogram,
electrocardiogram
skin
resistance,
and
respiration were estimated continuously while drivers followed a set route
through open roads in the greater Boston area.
Data from 24 drives of at least 50-minute durations were collected for
analysis. The data were analysed in two ways. The analysis used features
from 5-min intervals of data during the rest, highway, and city driving
conditions to discriminate three levels of driver anxiety with an precision
of over 97% across multiple drivers and driving days.
21
(Westland, 2011): This work reviews the olden times of designs for a
fussy branch of affective technologies that get electrodermal response
readings from person subjects. Electrodermal response equipment have
gone through frequent improvements to improved measure these nervous
responses, however still fall short of the capabilities of today’s skill. This
author has analysed diverse issues from which current avatar of
electrodermal reaction measurement suffers, from five that tend to
confound the mining of meaningful affective data streams from the
calibrations. Electrodermal reactions unadventurously have been labourintensive. Protocols and record of subject reactions were recorded on
separate documents, forcing stable shifts of attention among scripts,
electrodermal measuring devices and of observations and subject
reactions. These troubles can be solved by collecting supplementary data
and connecting it in a computer interface. That is, by adding associated
sensors to the basic electrodermal resistance reading to disentangle: (1)
body resistance; (2) skin conductance; (3) grip actions; other (4) factors
affecting the neural dispensation for instructing the body. The current
paper has analyzed, in depth, the mainly widely used low-cost technology
for measuring nervousness and emotional state: the electrodermal
response. It has been argued that, with a high-quality understanding of the
method behind electrodermal response, it is doable to precisely monitor a
number of inner emotional activities, and with the right set of sensors, to
split the differing sources of change and nervousness over time. Since the
accurate capture modulation of emotional state is a promising element in
explore of wealthier more immersive gaming, it is argued that such
measurements can considerably adjoin to the brilliance and the magnetism
of gaming innovations in software.
(Wagner et al., 2005): This paper gives the possible of physiological
signals as consistent channels for emotion detection to which only a little
amount of attentiveness has been paid up to this aim, as compared to
audio-visual emotion channels such as words or facial expressions. All
vital stages of an mechanical recognition system are talked about, from the
22
recordings of a physiological statistics set for a feature-based multiclass
classification. In order to gather a physiological statistics set from
numerous subjects over many weeks, the author employed a musical
introduction method that extemporaneously guides subjects to actual
emotional states, without using a deliberate laboratory setting. Fourchannel
biosensors
were
occupied
to
calculate
electromyogram,
electrocardiogram, skin conductivity, and respiration variations.
A vast range of physiological features was occupied from miscellaneous
analysis domains, including time/frequency, sub-band spectra, entropy,
geometric analysis, etc., which are proposed so as to discover the top
emotion-relevant features and to attach them with emotional states. The
top features taken are exposed in detail and their efficiency was confirmed
by the classification results. Classification of four melodic emotions
(positive/high arousal, negative/low arousal, negative/high arousal, and
positive/low arousal) is finished by using an complete, probabilistic linear
discriminator analysis (pLDA).
Moreover, by exploiting a dichotomic property of a2D emotion model, the
writer created a newest EMDC scheme so as to intensely improve the
accuracy of the four emotion classes. Using this technique, he actually
obtained a maximum of 13 percent improved correctness for all subjects.
However, the recognition accuracy of subject-independent classification
(70 percent for four classes) was not equivalent to the subject-dependent
cases (95 percent for four classes). The major reason can probably be
ascribed to the intricate dissimilarity of non-emotional entity contexts
between the subjects rather than to any contradiction of ANS differences
between emotions.
(Plotnikov et al., 2012), The preparation of this work was to observe the
use of EEG for user position monitoring in games, attempting to support
dependability with consumer deployment and thus exploit a tool alike to
the ones that are soon likely to emerge in the game control market. Passive
brain–computer interaction (BCI) can give useful information to identify a
23
user’s state and anticipate intentions, which is necessary to keep
adaptively and personalization. Specified the vast variety of viewers, a
game’s capability of adapting to dissimilar user profiles—in exacting to
keep the play in flow—is critical to make it ever more enjoyable and
satisfying. They have performed a user testing exploiting a four electrode
electroencephalogram (EEG) tool alike to the ones that are soon likely to
demonstrate on the market for game control. They have performed a
spectral description of the video-gaming experience, also in contrast with
other responsibilities. The results presented in this work give to advancing
the state of the art of information in many aspects. Author has performed a
spectral classification of the video-gaming experience, in grouping with
other tasks, with exacting attention to flow, which is a key factor of the
gaming practice. Results authenticate that subdivision of brain frequencies
in bands (and the consideration of coherences as well) is an important
feature-definition principle based on area facts (while band combinations,
such as the “attention ratio,” do not add to performance), and that the most
informative bands are those around low beta for discriminating with
gaming conditions (mid beta for discriminating gaming from other tasks).
One of the key actions for the BCI, further than medical applications
agenda, is require holding the integration of BCIs with existing gaming
hardware and software. This research suggests that a real-time user flow
monitoring
system—including
standard
hardware
for
signal
accomplishment and a processing software module as a component of the
game engine architecture—could become a common feature of novel
production adaptive computing systems.
(Nourbakhsh et al., 2012), In this paper, writer aims to perform a
comprehensive study on Galvanic Skin Response (GSR) which has lately
attracted researchers’ consciousness as a prospective physiological pointer
of cognitive load and emotions. It has usually been investigated through a
single or a few measures and in one investigational situation. The author
has assessed GSR data captured from two experimentations, one including
text reading tasks and the other using arithmetic tasks, each imposing
24
several cognitive load levels. In this study, they investigated different time
and frequency-domain features of GSR in multiple difficulty levels of
arithmetic and reading experiments. A normalisation was functional to
omit the subject-dependency of GSR information. The results show that
normalization efficiently improves the significance of distinction between
the cognitive load levels for mean and accumulative GSR and the spectral
features. They have examined temporal and spectral features of GSR
against diverse task difficulty levels. ANOVA test was applied for the
statistical evaluation. Obtained results show the strong outcome of the
explored features, especially the spectral ones, in cognitive workload
measurement in the both studied experiments. They have said that their
future work will include applying machine learning techniques and
assessing the arrangement of other physiological features in cognitive load
detection.
(de Santos Sierra et al., 2011), this paper proposes a stress-detection
method based on physiological signals. Concretely, galvanic skin response
(GSR) and heart rate (HR) are proposed to give in sequence on the state of
mind of a person, due to their non-intrusiveness and non-invasiveness.
Besides, specific psychological experiments were designed to induce
tension properly onto individuals in order to acquire a database for
training, validating, and testing the planned system. Such a system is based
on fuzzy logic, and it described the behaviour of a person under stressing
stimuli in terms of HR and GSR. The stress-detection accuracy obtained is
99.5% by acquiring HR and GSR during a period of 10s, and what is
more, rates over 90% of success are achieved by decreasing that
attainment period to 3–5 s.
This paper also comes up with a proposal that accurate stress detection
only requires two physiological signals, namely, HR and GSR, and the fact
that the proposed tension-detection method is suitable for real-time
applications. Finally, this system may be applicable in scenarios related to
aliveness discovery (e.g., detecting if an individual is accessing a
biometric system with an amputated finger), civil systems (e.g., driver
25
control), withdrawing money from a cash dispenser, electronic voting
(e.g., someone is forced to commit a certain vote), and so forth.
A wide variety of scenarios can advantage from this approach due to its
non-invasiveness, the likelihood of it being embedded on present security
systems, and its possibility in detecting stress in real time, jointly with the
capability of being mutual into other stress-detection methods based on
computer-vision algorithms. It has discussed future work which focuses on
incorporation with mobile devices.
(Tronstad et al., 2010), in this paper four electrode gels were checked with
regards to sorption characteristics and electrical properties. Skin resistance
time series were gathered from 18 test subjects during relaxation, exercise
and recovery, wearing various pairs of electrodes contra laterally on the
hyposthenia and the T9 dermatome. A little pressure test was applied on
the T9 electrodes. Impedance frequency sweeps were taken on the T9
electrodes the same day and the next, parameterized to the Cole model.
ANOVA on the first skin conductance level change, show response
amplitude,
recovery offset
and
pressure-induced
varies
revealed
considerable differences among gel types.
The wetter gels caused a higher positive level vary, a better response
amplitude, larger recovery offset and greater pressure induced artifacts
compared to the solid gels. Sweating on the T9 site led to negative skin
resistance responses for the wetter gels. Correlations were found among
the desorption measurements and the initial skin conductance level change
(hypothenar: R = 0.988 T9: R =0.901) RMANOVA on the Cole
parameters revealed a significant decrease in Rs of the most resistive gel.
Clinical implications are discussed such as the pressure artifacts which are
more applicable in situations where the patient is moving or physically
active during the recordings than for controlled setups in the laboratory.
Pressure on the electrodes is also likely to occur through sleep
monitoring.Pushing or pulling on the electrodes may give transient
26
artifacts which could be detected as particular responses, but the greatest
source of error comes from the electrodes which cause changes that stay
after the pressure is no longer applied.
(Edelberg and Wright, 1964), this study tested the theory that the palmer
galvanic skin response (GSR) involves the sweat gland and an epidermal
component each responding preferentially according to the demands of the
behavioural circumstances. Their relative donations were determined by
comparison of simultaneous GSR's from areas with high vs. low
concentrations of sweat glands and with direct measurement vapour
production as well. Stimuli were tones and lights which were either
alerting signals or execution signals for a perceptual or a motor (reaction
time) task. The population unexpectedly showed greater relative sweat
response to the alerting signal for the reaction time task than to the
associated execution signal (71 out of 94 S's). Individual subjects, but not
the population as a whole, differentiated significantly between alerting and
execution signals for the perceptual task. Results supported the hypothesis
that two components are present in the palmer GSR and that these
manifest stimulus response specificity, but they were inconclusive
regarding the nature of the class of stimuli to which each responds. The
difference cannot be one of a preparation for motor as opposed to nonmotor activity.
(Zhai and Barreto, 2006), in this paper a stress discovery system is
developed based on the physiological signals monitored by non-invasive
and non-intrusive sensors. The development of this emotion detection
system involved three stages: experiment setup for physiological sensing,
signal pre-processing for the extraction of affective features and affective
recognition using a learning system. Four signals: Blood Volume Pulse
(BVP), Galvanic Skin Response (GSR), Pupil Diameter (PD) and Skin
Temperature (ST) are monitored and analyzed to differentiate affective
states in a processor user.
27
A Support Vector Machine is used to perform the supervised classification
of affective states between “stress” and “relaxed.” Outcomes show that the
physiological signals monitored do, in fact, have a strong correlation with
the changes in emotional state of our experimental subjects when tension
stimuli are applied to the interaction situation. It was also found that the
pupil diameter was the most significant affective state indicator, compared
to the other three physiological signals recorded.
(Shi et al., 2007), In this paper, they attempted to describe the use of
physiological measure, namely Galvanic Skin Response (GSR), for
objectively evaluate users’ stress and arousal levels while using unimodal
and multimodal versions of the same interface. It has investigated the
relevance of GSR as an objective indicator of user’s cognitive load and
proposes a number of GSR features that can provide further insights into
the experienced level of cognitive load.
Preliminary and partial analysis of GSR data from user experiments has
shown that mean GSR across users increases as cognitive load increases.
In addition, it suggests users experienced lower cognitive load levels when
using a multimodal interface instead of a unimodal interface (such as
speech-only interface or gesture-only interface).
Cross-examination of GSR data with multimodal data annotation showed
promising results in explaining the peaks in the GSR data, which are found
to correlate with sub-task user events. This interesting result verifies that
GSR can be used to serve as an objective indicator of user cognitive load
level in real time, with a very fine granularity.
For future work, they firstly would like to complete the GSR analysis for
all eleven subjects who participated in the experimentation, and perform
further significance tests on the mean and accumulated GSR data. They
also desired to explore in a more rigorous way the correlation between
user’s GSR variation and interactive behaviour, especially when using
multimodal interfaces.
28
(Haapalainen et al., 2010), in this paper, they collected data from multiple
sensors and compared their ability to assess cognitive load. Their focus
was on visual perception and cognitive speed-focused tasks that influence
cognitive abilities general in ubicomp applications. They evaluated the
usefulness of a wide range of psychophysiological signals in processing
cognitive load in six different elementary cognitive activities. Four of the
tests were chosen to address the PS factor while each of the other two tests
targeted one of the other factors, SC and FC. Results demonstrated that,
for each participant, a psychophysiological signal was found that can be
used to precisely discriminate (74%) tasks of low and high level of
difficulty, and following that, levels of low and high cognitive
consignment in participants.
They initiate that across all participants, the electrocardiogram median
absolute variation and median heat flux computations were the most
precise at distinguishing between low and high levels of cognitive load,
providing a classification correctness of over 80% when used mutually.
Their contribution is a real-time, aim, and generalizable technique for
assessing cognitive load in cognitive tasks usually found in ubicomp
systems and situations of divided notice level of difficulty, and following
that, levels of low and high cognitive load in contributors.
They found that across all participants, the electrocardiogram median
absolute deviation and median heat flux measurements were the most
accurate at distinguishing between low and high levels of cognitive load,
providing a classification accuracy of over 80% when used together. Their
contribution is a real-time, objective, and generalizable method for
assessing cognitive load in cognitive tasks commonly found in ubicomp
systems and situations of divided attention.
(Tarvainen et al., 2001), in this paper principal module analysis is used for
the analysis of the inducing GSR, which is a simple technique of capturing
the autonomic nerve reaction as a parameter of the sweat gland role. Any
stimulus able of an arousal consequence can evoke the response and the
29
amplitude of the response is more dependent on the surprise consequence
of the stimulus than on the physical stimulus strength. Basis functions are
observed from the Eigen decomposition of the information correlation
matrix. As PCA is the best mean square fit of a set of orthogonal functions
to the set of measurements, the explanation will depend upon the nature of
calibrations.
The dimensionality of measurements can be estimated by the number of
basic functions required to estimate measurements in certain correctness.
Hence the Eigen values, corresponding to used functions, are a measure of
similarity. The method was tested using 20 healthy subjects and 13
psychotic patients. Eleven surprising auditory stimuli were delivered at
asymmetrical intervals and evoked GSRs were recorded from the hand.
Observed similarities between adjacent waveforms were more remarkable
within healthy subjects. Response waveforms were usually unaltered for
healthy subjects, but there was a tendency of habituation. Observed reduce
in amplitudes was 67-99% within healthy subjects. For psychotic patients
wave shapes were random and amplitudes were usually smaller.
(Mahdis et al.,2012),This paper is an study on negative emotions states
identification by employing of Fuzzy Adaptive Resonance Theory (FuzzyART) considering the changes in actions of autonomic nervous system
(ANS). Specific psychological experiments were designed to provoke
suitable physiological responses on human in order to acquire a suitable
database for training, validating and testing the proposed process. In this
research, the three physiological applied signals are Galvanic Skin
Response (GSR), Heart Rate (HR) and Respiration Rate (RR). The initial
experiment which is named Shock was designed to resolve a criterion for
the change of physiological signals of each individual. In the next one, a
arrangement of two sets of questions has been asked from the subjects to
provoke their emotions. Ultimately, Physiological responses were
analyzed by Fuzzy-ART to recognize which question excites the negative
30
emotions. Detecting negative emotions from neutral is obtained with total
accuracy of 94%.
(Natascha Esau et al.,2005)Existing emotion recognition applications
usually differentiate between a small number of emotions. However this
set of so called basic emotions varies from one application to another
depending on their according needs. In order to support such differing
application needs an adaptable emotion model based on the fuzzy
hypercube is presented. In addition to existing models it supports also the
recognition of derived emotions which are combinations of basic
emotions. We show the application of this model by a prosody based fuzzy
emotion recognition system.
(Ryu et al., 2008), in this paper, they evaluated the results of the
conductive rubber electrode to utilize the electrode of wearable health
monitoring machine. These electrodes were made the rubber electrode
integrated a good conductive metal compound and had several viscosity.
In addition, conductive yarn was validated to replace connecting wire
among electrode and measurement machine. For the continuous health
measurements, physiological signal must be able to calculate for a long
time in daily life. Consequently, the health monitoring machine was made
as small as possible, and sensors must have a little effect by motion
artifact. To decrease impedance among the user skin and the electrode, a
conductive gel was applied to the metallic electrode, thus facilitating
converting ion current flowing in a living organ into an electric current.
Though, for profit disposable electrode using the conductive gel incurred a
skin problem such as a reddish skin and stinging pain, when used to
compute electrocardiogram (ECG) for a long time. The ECG signal
measured by the conductive yarn rubber electrode cable was a superior
quality. However, conductive yarn must be isolated to be used in clothing.
31
1.12 Organization of thesis
The research presented in this thesis aims to uncover the problems of an
embedded system in biofeedback by using intelligent machine learning
algorithms and making them work in hybrid form. An embedded system is
developed
which
can
measure
different
psychophysiological
signals(GSR/BVP/Temperature) and intelligently analyse these signals and
give output in form of emotions(JOY/STRESS/CALM) based on the data
collected by system in the past.Based upon these emotions, future emotion
of that subject is also analyzed through data analysis algorithms. This
system is cost-effective, portable, and consumes low power. This
introductory chapter explains the general perspective in which computing
of emotion takes place. It illustrates some of the problems of definitions in
this area and also general needs of work.
The remainder of this thesis is organised as follows:
Chapter 2 introduces the method which was devised to enable the
monitoring of emotion. This chapter gives architectural view on the
proposed automated affective detection process based on data collected by
the system. It also presents details of hardware and software aspects. It
covers the first two objective of research, which reflects how a system can
be designed using multiple parameters and analog-to-digital conversion
algorithms.
Chapter 3 presents emotion extraction methods classifiers used in machine
learning algorithms.
This chapter describes an implementation of
intelligent data analysis algorithms on embedded systems such as TI
MSP430. It also represents a novel algorithm used to make machine
learning algorithm work in hybrid form. This wraps up the third object of
the research.
Chapter 4 provides detailed information regarding minimizing cost
function. This chapter gives a comparative study with existing devices and
biofeedback solutions to fulfil the requirements of portability and low cost
32
efficiency. For power efficiency, an additional algorithm (Sleep/awake
algorithm) has beendesigned along with the overall implementation of the
system using energy-efficient TI MSP430 microcontroller. This describes
the fourth objective of research.
Chapter 5 explains exploratory experiments because it is next used to
address questions of technological feasibility. In this research four
different experiments were conducted and case studies of same are
described in this chapter. It reports groups of users and find significant
performance and also satisfaction for users of affective system. The
chapter discusses the findings and implications of the experimental work.
Chapter 6 provides user guidelines and safety measures of the final
product, as it is intended to give assistance to people using a particular
system. The chapter contains essential instructions on the use of the
system. Such system can be used by a at home user.
Chapter 7 describes my conclusions and contribution. It presents a
summary of the benefits and limitations of my work, and the issues and
opportunities arising from it for future work.
33
CHAPTER 2
EMOTION DETECTION
ARCHITECTURE
AND
SYSTEM
34
Chapter 2
EMOTION DETECTION ARCHITECTURE &
SYSTEM
This key chapter reflects the first two objectives of this research, which covers
design and development of a system for the computer/non-computer users to
detect emotion though the psychophysiological signals (GSR/BVP/Temperature).
This is done to fulfil the affective sensing requirements of a prospective affective
computing system. This work deals with the instrument/system, the methodology
used for the acquisition of the signals from the subjects, and the procedure that
how this information is sent to the microcontroller (MSP430F2013). In this
chapter, research also reflects the implementation of a new proposed model,
which is one solution to the conventional models and gives an accurate analogue
to digital domain conversions. This chapter discusses hardware and software
aspects of the proposed model
2.1 Introduction
Although the action of the autonomic nervous system cannot be controlled
directly, it can be inclined in an indirect way by two mechanisms called
conditioning and biofeedback.(Kandel, 2014, Lang, 2014, Grossman et al.,
2013) Biofeedback is a therapeutic method in which people are trained to
improve their health by using the signals from their own bodies. Physical
therapists use the biofeedback method to help the stroke victims regain
movement in the paralyzed muscles.(Tate and Milner, 2010) Psychologists
use it to help the tense and anxious clients learn to relax. Specialists in
many different fields use the biofeedback method to help their patients
cope with the pain. Biofeedback is a means for relieving the ache, gaining
control of our body procedures to augment relaxation, and developing a
good health and more comfortable life patterns. Clinical biofeedback
follows the same principle, using specialized instruments to monitor
35
diverse physiological processes as they occur. The patterns on a computer
screen and the audio tones that go up and down imitate the changes as and
when they happen in the body system being monitored. (Morris and
Guilak, 2009)
Example: Biofeedback provides us the data about ourselves by the means
of peripheral instruments. Using a thermometer to measure our
temperature is a common example of biofeedback.(Smalls et al., 2009)The
biofeedback training publicizes us with the activity in our diverse systems
in a body so we may discover to control this activity to relieve stress and
improve health. Many stress-related illnesses (such as headaches and low
back pain) occur due to the over activation of the physiological systems in
a response to the stressful events.(Ulrich-Lai and Herman, 2009)
The biofeedback training is an educational procedure for knowledge the
particular mind/body skills. Learning to identify the physiological
reactions and varying them is not unlike knowledge how to play the piano
or tennis – it requires practice. Through practice, we become familiar with
our own exclusive psychophysiological prototypes(Kreibig et al., 2007)
and responses to stress, and learn to control them rather than having them
controls us. A microcontroller-based system is designed to pick up the
electrical signals, such as pulse, GSR, and temperature, froma human body
to condition it according to the requirement and then to display the
patient’s condition.
2.2 Architectural View of Conventional Model
The primary purpose of any medical instrumentation system is to measure
or determine the presence of some physical quantity that may, in some
way assist the medical personnel to make better diagnosis and treatment.
Any conventional medical device would comprise the subsequent
model.(Yamashita et al., 2007)
36
Fig. 2.21: Block Diagram of Conventional Biomedical Instrumentation
System(LI et al., 2013)
2.2.1 Subject
The Subject is the individual body, which generates a range of signals.
Research/investigation on the human body can either be interventional
(trial) or observational (test article). It incorporates both the collection and
analysis of data in order to answer the specific questions. Human subject
research often involves surveys, questionnaires, and interviews(Sawday,
2013).
2.2.2 Transducer/Sensor
A transducer converts one form of energy to another form. The main
function of the transducer is to provide a usable output in response to the
subject, which may be a precise physical quantity, property, or condition.
Essentially, the sensor converts a physical signal to an electrical signal.
Depending on the transducer, the production produced can be in the
appearance of voltage, current, resistance, or capacitance. The sensor
should be minimally invasive and interfere with the living system with
minimum extraction of energy. The most important function of the
transducer is to provide a usable output signal(Wang et al., 2005).
2.2.3 Signal Conditioner
For interfacing analog signals to the microprocessor/microcontroller, a
data acquisition system is used. The function of the system is to obtain
and digitize the information, often from the hostile clinical environments,
37
without any degradation in the resolution or correctness of the signal.
The signal conditioner converts the output of the transducer into an
electrical quantity suitable for the operation of the display or recording
system. Signal conditioning typically includes functions, such as
amplification, alteration from analog to digital, or signal transmission
circuitry. The buffer amplifier helps in increasing the sensitivity of the
instruments by amplifying the original signal or its transuded form. The
A/D converter carries out the procedure of the analog to digital; the
higher the digit of bits, the higher the accuracy of conversion. Since
software expenses generally far exceed the hardware costs, the
analog/digital interface structure must permit software efficient transfers
of data and command the status signals to avail the full capability of the
microcontroller(Cao et al., 2006).
2.2.4 Display System
The display system provides a noticeable demonstration of the quantity.
It may be on the chart recorder, on the screen of a cathode tube, in a
numeric form, or an LCD display(Anttonen and Surakka, 2005).
2.2.5 Control System
This system controls all operations of the device. It consists of
microprocessor/microcontroller and embedded software to provide the
necessary controls. The control logic provides the necessary interface
among the microprocessor system and the elements of the attainment unit
to provide the essential timing control. It has to sample the data at correct
time, make sure that the correct analog signal is selected, initiate the A/D
conversion
procedure,
and
signal
to
the
microcontroller
or
microprocessors on completion of the conversion(Schima et al., 2006).
2.2.6 Working of conventional model
Each time you scratch an itch, clutch a snack when you are hungry, or
use the bathroom when you feel the need, you are responding to the
38
biofeedback cues from your body about your physiologic state. With the
biofeedback training, however, you are cued by the sensors that are
attached to your body. This data is conveyed by the visual displays or
sounds. Using imagery and mental exercises, subject (human) learn to
use the feedback provided by the sensors as a measure of success and
then you study to control these functions. With practice, subject can learn
to "tune in" without instrumentation and you can control these purpose.
For example, in a biofeedback training session for annoyance,
temperature sensors are first attached to subject hands, then to his/her
feet and ultimately to forehead, if needed. The subject goal would be to
increase blood flow away from the brain by raising the temperature in
his/her hands or feet. Other sensors strength monitor electro-dermal or
galvanic skin response to determine how simply person sweat or get
"goose bumps" because this affects subject ability to alter his/her skin
temperature. To warm up hands and feet, subject might imagine basking
in the sun on a beach while listening to a script like "I feel warm, my
hands are growing warm and heavy" or any external stimuli can be used
e.g watch movie etc. After training session, subject would be sent home
with this script on an audiotape and small thermometers to use for your
everyday practice(Martin et al., 2007).
2.3
Problems in Conventional Architecture
Although instruments based on medial has shown to do tremendous good
for the mankind, still there are some uncovered issues to be solved. The
following are the common architectural issues: (Darwish and Hassanien,
2011)
•
Complexity
•
Signal parameter support (e.g. only temperature)
•
High power consumption
•
Bulkiness
39
2.4
Proposed Model (Emotion Detection Model)
The architecture of this emotion detection monitoring system is a novel
model and with more portability, less complexity, low price and, more
power efficiency it is a solution to many problems with the earlier
conventional models. This system includes the mechanism of stimulation,
the readings, the measurements, and finally the estimation of the emotional
state (anger, happiness, etc.) of a person. This system takes multiple inputs
from the body and can intelligently analyse those inputs for predicting the
emotions.
2.4.1 System Design Process
Design and excellence are an essential part of any biofeedback product.
Taking for example the microcontroller-based system: ergonomics,
aesthetics, and engineering have been considered concurrently as part of
the design process as shown in the Fig. 2.2 of the System Design Process.
Ergono
mics
Design
Aesthetic
Design
Product
Engineer
ing
Design
Fig. 2.22: System Design Process
The product must be intended with a user-friendly control panel. Its
display should seem natural and easy to recognize. This feature can be
addressed by using only a single input connector for each of the
parameters' methodically programmed and developed user-interface with
the peripherals. During the product design the subsequent design
parameters were considered:
40
Aesthetics: This is the outward look of a product; attention must be paid
to the aesthetics both in the form design and control panel.(Green, 2007)
Reliability: The functional reliability of the system and the electronic
control can be increased substantially by the use of an intelligent µc, well
calibrated
and
standardized
sensors
and
conditioning
processes.(Narayanan and Xie, 2006)
Maintainability: To ensure an easy maintenance of the system, the design
must incorporate the easy removability of different parts so that the
various parts can be re-assembled quickly for carrying the routine repairs.
For an easy maintainability, the system cards must be designed in a
modular form with the standard reliable connectors.(Kopetz, 2008)
2.4.2 Steps for Designing the Emotion Detection System
The design steps for the design of a standalone biofeedback device are
shown in the Fig. 2.3. System identification explains the process and
discovers the relationship between input and output. Requirements
determine the needs or conditions to be met for a new or altered product,
taking into account the possibly conflicting requirements of various
stakeholders, such as beneficiaries or users. Functional design specifies the
sub-processes that are required in the system.
User Requirement of Biofeedback System
Requirement Analysis
Abstract Design microcontroller based
Design Indent of Biofeedback System
Design Transformation
for forming design, control panel design, and
electronicdesign
Final
41Product
Fig. 2.23: Product Design of the Biofeedback system
2.5 Proposed Design and How It Works
The architecture of the emotion estimation monitoring system, shown
below in the Fig 2.4, has a mechanism to measure the different biomodalities or bio-signals (BVP/GSR/Temperature).
In the designed
product, the validation of subject is done at priority. It fetch the bio-signals
from the subject and then sends it to the MCU (MSP430F2013).(Sharma
and Kapoor, 2013)
Fig. 2.24: Proposed acquisition system of physiological data and detect emotions.
42
As shown in this figure, the proposed design of the system covers different
requirements:
•
Portability
•
Less Cost
•
Energy Efficiency
•
Intelligent Analysis
•
Multi-parameter Support
•
User Friendly
•
Home Product
•
Analysis of Simulations by Doctors
As shown in the Figure 2.4, the system is divided into Part A (below the
red bar) and Part B (above the red bar).
PART A: Covers all the research objectives. This part has the capability to
sense different bio-signals, convert analog signal to digital, data
processing, and do intelligent data analysis (explained in the chapter 3).
Final output is shown by using different coloured LEDs. The RED LED
reflects Stress, YELLOW LED reflects Calmness, and GREEN LED
reflects Joy. The PART A fulfils the research purpose. This part is not
complex and is easy to use. Any user can use the proposed device at home
by following a few simple instructions. Signal processing and control is
explained in the next sections of this chapter.
PART B: This is designed and developed for an extra functionality.
This second part, which is above the red bar, is specially designed for the
doctors. By adding this part, the doctors can check simulations on a
monitor and have the detailed readings of a patient for the records. In
GSR, variable voltage according to the body resistance is fed into the
MCU MSP430 for an analog processing. The output is sent to the 8051
microcontroller through a 2 wire designed protocol, and the final result is
further sent to the PC from a serial port using the UART Communication.
Equally,the BVP Sensor that measuresthe Blood Volume Pulse Rate is
43
integratedin the circuit and gives the high pulse in synchronization with
the heart rate. A light is passed into the human finger with an LED, which
reflects back from the amount of blood. The phototransistor receives the
amount of light and gives the output voltage that is fed into the MSP430
microcontroller for the analog processing. The output is sent to the 8051
microcontroller through a 2 wire designed protocol, andthe final result is
sent to the PC from a serial port using the UART Communication. In the
Temp Sensor (LM-35), MSP microcontroller has a built-in temperature
sensor with which the temperature is measured directly using the internal
SD16 of MSP. The output,as a digital value, is sent to the 8051
microcontroller via a 2 wire self-designed protocol; and the final result is
further sent to the PC from a serial port using the UART Communication.
To interface the two MCUs, an isolator circuit was formed because these
MCUs work on different voltages and direct interfacing was not
possible;due to this, the opto-couplers were used to send or receive the
data from either side. Another MCU(8051)(Mazidi et al., 2006) was
required to finally send the data to PC, as there was no UART
Communication Protocol present in the MSP Microcontroller.
2 Wire Communication Protocol: 14 pin Microcontroller MSP430F2013 is
portable but has limitations. It has limited pins.So, to solve this, a protocol
was designed,that is, two wires Communication Protocol. This protocol
can send 16-bit data in one transmission. The communication is one way
only, that is, it can send data only from MSP to 8051, not vice versa. The
data is sent using two pins named DATA and CLOCK. The data that is to
be sent is broken into 16 bits. Then one by one the bit starting from LSB is
placed on the DATA bit .A total of 16 times the clock will go low for
sensing the full 16-bit ADC sample.
From the receiver end, as soon as the clock is received, the interrupt mode
stores the present bit from the DATA pin. So it keeps on storing the bits as
received and makes a full value when 16 clocks are received and then it
clears all other variables. This way, a complete sample of 16 bits could be
sent from MSP to 8051 using only two PINS.
44
Fig. 2.25: Wire Protocol
Optocoupler: It is used to provide the isolation between MSP and 8051, as
MSP works on 3V and 8051 works on 5V. Due to this, they need isolation.
The general purpose of the optocouplers consists of a gallium arsenide
infrared emitting diode driving a silicon phototransistor in a 6-pin
6
dual inline package(Quinones
(Quinones and Joshi, 2007).
2007)
The Part B is good for analysing only the simulation. This part makes the
system more complicated and disturbs its portability. It also consumes
more power and requiresthe involvement of a doctor. So, after testing this
module, the researcher has kept this part as optional and maintained the
full focus on the Part A only.
2.6 Circuit Diagram of the Proposed System
The initial move of the hardware design is to place the hierarchy
hierarch of the
elements. It is rational to follow the hierarchical order when looking for
the way to connect them collectively. Once all the components are picked
and the respective footprints are found in the software, the component
placement and wiring can commence.
commence. This is an intuitive part of the
design, and certainly takes a few iterations before the “close-to-optimal”
“close
solution is found.
45
Fig. 2.26: Circuit Diagram
The pin configuration typically can be achieved by adding several external
components. The parallel I/O ability of the MSP430 allows the
configuration to control the outside world by connecting to the external
hardware. As explained previously, the PART A fulfils the research
purpose and the PART B is designed
designed and developed for an extra
functionality. So, the circuit diagram above explains the PART A alone.
The functions of the components to the microcontroller MSP430F2013 are
listed in the Table below.
Table 2.4: External Components
Pin No
2,3,5
External Components
Description
LED
Three LEDs are attached to display the output of the
system in the form of three different emotions:
Green = Joyful
Yellow = Calm
Red = Stress
46
1
4
6
Vcc
Thermistor
Electrodes
1.8 V-3.6 V Supply voltage during the program
execution
Temperature Sensor
GSR Sensor
9
Light source (LED) and
light detector (photo
diode)
BVP Sensor
2.7 SIGNAL PROCESSING
Signal acquisition is carried out within the input voltage range of the
analog-digital converter (ADC). The task of the ADC is to digitise the
analog voltage with a resolution high enough to represent the original
signal. In other words, the quantisation is a process of mapping a
continuous range of values by a finite set of integer values.(Luecke,
2005) Following are the various steps for acquiring the data from a
human body:
2.7.1 Connectivity of input signal with sensors
A biofeedback system needs to deliver and receive information from the
user. In order to receive the data derived from the user's physiological
signals, we must use a variety of sensors. Each of these sensors will
account for a particular physiological signal. This system supports
different parameters, and every parameter has its own sensor with a
specific sensing technique.(Ahmed et al., 2011)
2.7.1.1 Galvanic skin response (GSR)
Galvanic skin response (GSR), also known as electrodermal response
(EDR), psychogalvanic reflex (PGR), or skin conductance response
(SCR), is a technique of measuring the electrical resistance of the
skin.(Villarejo et al., 2012) EDRs are the changes in the electrical
properties of a person’s skin caused by an interaction between the
environmental events and the individual’s psychological state. Various
electrical properties like conductance (SC), resistance (SR), potentials
47
(SP), impedance (SZ), and admittance (SY) are observed. These variations
can be sensed in the different parts of the body (the palm of the hands is of
utmost interest). Variations in the ionic content of the various skin layers,
depending upon the amount of sweat and hence upon the sweat glands'
activity, are accountable for these changes.
The electrical conductance of the skin is measured by the silver electrodes
(GSR sensor), which derives the variation from skin’s moisture level. The
sympathetic nervous system controls the sweat glands, thus making the
skin’s conductance a good indicator of physiological arousal.
Structure and Galvanic Skin Function
The skin is a selective barrier that serves the function of preventing the
entry of any foreign matter into the body and selectively facilitating a
passage for materials from the bloodstream to the exterior of the body.
There are two forms of sweat glands present in the human body: the
apocrine and the eccrine. The latter is of primary interest to the
psychophysiologists. The primary function of the eccrine sweat glands is
thermoregulation. However, according to Edelberg(Nagai and Critchley,
2008), the sweat glands on the palm and plantar surfaces are more
responsive to the psychological sweating than other areas. Figure 2.7
below shows the anatomy of the eccrine gland and various layers of
skin.(Milad et al., 2007)
48
.
Fig. 2.27: Skin Anatomy(Amirlak et al., 2011)
The skin has electric properties that can change relatively quickly and are
closely related to the psychological process.(Carlson and Carlson, 2012)
These changes in the skin’s conductance and electrodermal activity (EDA)
(Boucsein, 2012)are related to the variations in the eccrine sweating.
Sweat act like an electrolyte. As the sweating increases, the skin pores
start filling with the sweat making the skin more conductive. Autonomic
nervous system (ANS) has the sympathetic branch that controls the eccrine
sweating; therefore, the skin conductance reflects the rise of the
sympathetic ANS, which accompanies different psychological processes.
Skin conductance and EDA have been applied in a wide array of research,
serving as indicators of such processes as awareness, habituation, arousal,
and cognitive effort in the different sub-domains of psychology and
interrelated disciplines. In judgment and decision making study, the skin
conductance is often used as an indicator of emotional arousal and
affective processes.
GSR Measurement
Galvanic skin response is a non-intrusive and easy to apprehend
physiological signal, which is being explored for the emotion sensing.
Human skin is a good conductor of electricity and when a weak electrical
49
current is delivered to the skin, changes in the skin’s conduction of that
signal can be measured. GSR is a method of regulating the internal
physical process by giving a biofeedback, which is effective in the
treatment of phobias, anxiety, and to increase the relaxation process of the
subject during the hypnosis.(Pradeep et al., 2008)
Fig. 2.28: Skin conductance measured through the sweat glands of finger
tips(Mandryk et al., 2006)
The variable that is measured is either skin resistance or its reciprocal, that
is, skin conductance. GSR is measured in milli volts (mV). According to
Ohm’s Law, skin resistance (R) is equal to voltage (V) applied between
the two electrodes on the skin divided by current passed through the skin
(I). The Law can be expressed as R=V/I.(Rudenko et al., 2013) The GSR
is extremely sensitive to the emotions in some persons;anger, startle
response, fear orienting response, and feelings are all among the emotions
that may produce some kind of similar GSR responses. GSR measurement
is also becoming common method in the hypnotherapy and psychotherapy
practices.It can be implemented as a method of extracting depth of
hypnotic trance prior to the commencement of the suggestion therapy.
When a traumatic situation is experienced by the client (for example,
during hypnoanalysis), immediate changes in galvanic skin response can
show that the client is experiencing an emotional arousal. It is also applied
in the behaviour therapy to measure the physiological reactions, such as
fear.
Range of GSR<5 Kohms indicates a high level of brain arousal and >25
Kohms indicates a low arousal and withdrawal from mind (calm level).
The GSR is measured most conveniently at the palms of the hand, where
50
body has the highest concentration of sweat glands. The measurement is
made using a DC current source. The Galvanic Skin Response (GSR) is a
measure of the skin's conductance between the two electrodes. The
electrodes are typically attached to the subject's fingers or toes using the
electrode cuffs, or to any other part of the body using a Silver-Chloride
electrode patch. To measure the resistance, a small voltage is applied to
the skin and the skin's current conduction is measured.(Sharma and
Kapoor, 2013, Jeon et al., 2007)
The skin conductance is considered to be a function of the sweat gland
activity and the skin's pore size. An individual's baseline skin conductance
will vary for many reasons, including the gender, diet, skin type, and
situation. The sweat gland activity is partly controlled by the sympathetic
nervous system. When a subject is startled or experiences anxiety, there
will be a fast increase in the skin's conductance (a period of seconds) due
to the increased activity in the sweat glands (unless the glands are
saturated with sweat).
GSR Sensor
Extremely pure silver electrodes (having silver with purity of 99.999%)
are used to measure the GSR. Electrodes are small plates that apply a safe
and imperceptibly tiny voltage across the skin.
There is saturation effect: when the duct of the sweat gland fills, there is
no longer a possibility of further increase in the skin conductance. The
excess sweat pours out of the duct andthe sweat gland activity increases
the skin's capacity to conduct the current passing through it. The changes
in the skin conductance reflect the changes in the level of arousal in the
sympathetic nervous system. It was observed that the Analogic Digital
Converser saturates at 2.35 V. The microcontroller has a built-in ADC of
16 bits with a resolution of:
2.35/65535=3.5v
(1)
The Galvanic Skin Response oscillates between 10 kΩ and 10 MΩ
(Sharma and Kapoor, Villarejo et al., 2012), as can be seen in the existing
51
studies about the skin conductance obtained from the different applied
voltages .As ADC has a resolution of 3.5 V and the minimum tension is
136 mV, an operational amplifier does not have to be included. This
concept helped in achievingthe third objective, that is, energy efficiency.
A person’s skin acts as a resistance to the passage of electrical current. By
placing two electrodes on the fingers, we can calculate the GSR. To find
out the value, one resistance was used, as it can be seen in the fig 2.9, in
series with the skin resistance to form a voltage divider.
Fig. 2.29: Voltage Divider.
VO =
R2
RS + R2
(2)
Where, Rs is the resistance of the skin.
It can be observed that the Vo output tension is inversely proportional to
the value of the skin resistance. The more stressed the person is, the more
his/her hands will sweat, so his/her resistance will decrease. Therefore, we
can conclude that the more stress the person is under, the higher output
voltage will be.
2.7.1.2Blood Volume Pulse (BVP)
Blood Volume Pulse is the phasic variation in the blood volume with
each heart rate, heartbeat, and heart rate variability (HRV). (Chambers et
al., 2005)It consists of beat-to-beat differences in the intervals between
successive heartbeats. During the systole stage, the muscles of the
ventricles contract and force the blood to flow from the ventricles into
the arteries. The rate of heart contractions over a given time period is
52
defined as the Heart Rate. It is usually expressed in beats per minute
(bpm).(Fox
(Fox et al., 2007)
Heart rate is one of the human body’s vital sign that tells the medical
personnel about the extremity of the casualties’ physiological conditions.
It is one of the simplest and the most informative cardiovascular
parameters. With the observation that heart rate fluctuation is related to
various cardiovascular disorders, the analysis of the heart rate has
become a widely used tool in the assessment of the behaviour of the
heart.
Fig. 2.30:: Structure of the heart(Borazjani
heart
et al., 2010)
Blood passes through the heart in two phases. The phase where the
ventricles are filled with blood is referred to as the "diastole" stage.(Veress
stage.
et al., 2005) The pumping of the blood out of the ventricles is referred to
as the "systole" stage.(Kazui
stage.
et al., 2006) During the systole stage, the
blood flows from the ventricles of the heart into the small arteries. The
difference in the size of the opening of the ventricles and the arteries
53
causes a burst of pressure. This pressure wave expands the arterial walls as
it travels and is felt as the "pulse".(Sutton-Tyrrell et al., 2005)
BVP Measurement
The heart rate varies between individuals. The normal human heart rate at
rest is 60 to 80 bpm. At rest, an adult has an average heart rate of 72 bpm.
The athletes normally have a lower heart rate than less active people.(Poh
et al., 2010)The heart rate also varies with age. Children normally have a
higher heart rate of approximately 90 bpm.
Table 2.5: Age-related ranges of heart beat(Fink et al., 2009)
Age
Beats per minute
Newborn
90-100
10 years
80-90
10+ and adults
60-80
Bhattacharya, Kanjilal(Shi et al., 2009, Bhattacharya et al., 2001) stated
that non-invasive techniques can be used to determine the human body's
cardiovascular condition. It was addressed that the qualitative assessment
of the overall clinical status of the cardiovascular dysfunctions can be
determined non-invasively. Various techniques and devices have been
used to measure the heart rate in humans. The pressure sensors measure
the changes in the pressure level near the heart or the vibrations produced
by the heart. The sound sensor measures the changes in the sounds near
the heart, and light sensors detect the changes in the optical property of the
blood. There are various methods to measure the heart rate, such as
Mechanical method, Electrical Signal Detection, Optical method, and
Plethysmograph. The most common and accurate technique, which is used
these days is Plethysmograph.
PLETHYSMOGRAPH (Allen, 2007)is a combination of the Greek word
"plethysmos," meaning increase and "graph," meaning to write.
Plethysmograph was developed in the 1960’s and 1970’s by the psychophysiology researches.(Fleming, 1980) It is an instrument that is used to
54
determine the variations in the blood volume or the blood flow in the
body. These transient changes occur with each heart beat. (Lacey and
Lacey, 1978)
There are several different types of plethysmograph, which vary according
to the type of transducers that is being used. The common types include:
air, impedance, photoelectric, and strain gauge plethysmograph. Each type
of plethysmograph measures the change in the blood volume in a different
manner.(Shimazu et al., 1989, Cheang and Smith, 2003) Various
plethysmographs are explained in the table below:
Table 2.6:Different Types of Plethysmographs.(Terry, 2005)
Types
Methodology
Air
Uses an air-filled cuff. Measures the rate of change of
forearm volume, which correlates with the change in
the blood volume.
Impedance
Uses low frequency alternating current applied through
the electrodes. Measures the change in the electrical
impedance, which corresponds to the change in the
blood volume.
Uses photodetectors. Measures the intensity of the
transmitted and reflected light, which demonstrates the
volume change in the blood perfusion.
Uses a rubber tube filled with mercury. Measures the
changes in limb circumference, which relates to the
changes in the blood volume.
Photoelectric
Strain gauge
BVP Sensor
The heart rate sensor used in this research is based on the principle of the
Photoelectric plethysmography method. This methodis also known as
photoplethysmography (PPG) and is an optical measurement technique
used for detecting the blood volume changes in the micro vascular bed of
tissues. This method uses a light source (LED) to illuminate the skin and a
light detector (photo diode) to detect the changes in the optical properties
due to the change in the blood volume. This method has become very
55
popular in the medical field, especially, in the pulse oximetry due its
simple, non-invasive, and unobtrusive monitoring.(Barreto et al., 1995)
It measures the heart rate by determining the blood volume changes in the
skin periphery (finger-tip and ear-lobe) by the photo-electric method.
Compared to the other types of plethysmograph methods mentioned in the
Table 2.3, PPG technique is simple to use, easy to set up, and low in
cost.(Allen, 2007)
Dr. Nolan (Wallace et al., 2011) proposed that photoplethysmography is a
non-invasive technique that can be used to measure the variations in the
heart rate.
“A PPG can prove to be quite helpful in measuring the HRV. There is
some exciting research going on in determining HRV using PPG. The
analysis of signal from PPG has great potential for enriching the
interpretation of HRV.”
A plethysmograph consists of:
i.
A light source, which illuminates the tissue.
ii.
A light sensitive detector, which detects the amount of light
transmitted from the tissue.
56
Fig. 2.31: Arrangement of a plethysmograph(Stojanovic and Karadaglic, 2007)
Photoplethysmography (PPG) works by placing an individual’s finger tip
or ear-lobe between two parts of a transducer consisting of a light source
and a light sensitive detector. A beam of infrared light is projected towards
the detector. The blood in the finger or ear-lobe scatters the light in the
infrared range, and the amount of light reaching the detector is inversely
related to the volume of blood in the skin periphery.(Kamal et al., 1989,
Elgendi, 2012)
PPG is based upon the premise that all living tissues and blood have
different light absorbing properties. The infrared light is absorbed well in
blood whereas, weakly in the tissues.(Sundararajan, 2010)
The Figure 2.12 shows the absorption level of the infrared light in the
living tissues and blood. When the blood vessels in the finger dilate, the
increased blood flow allows less light to reach the photo-detector and
when the blood vessels contract, the blood flow is decreased and increased
light reaches the photo-detector.
57
Fig. 2.32: Relative absorption levels of infrared light of skin
The photoplethysmograph waveform: The photoplethysmograph waveform
does not resemble the pulse seen in an electrocardiogram (which is used to
record the electrical activity of the heart). However, the periodicity of the
signal is unchanged and the photoplethysmographic waveform can be
effectively used to detect the changes in the heart rate.(Peper et al., 2010)
Dicrotic Notch
Anacrotic Limb
Time
Waveform (mV)
Fig. 2.33: Representation of the Photoplethysmograph waveform(Peper et
al., 2010)
The upstroke, called the anacrotic limb, is abrupt due to the contraction of
the ventricle (systole). The downstroke is more gradual and corresponds to
the elastic recoil of the arterial walls. The downstroke regularly shows a
fluctuation that is known as the dicrotic notch. This is due to the vibrations
58
set up when the aortic valves shuts. (Maffei, 2012)Each time the heart
muscle contracts, blood is ejected from the ventricles and a pulse of
pressure is transmitted through the circulatory system. This pressure pulse
while travelling through the vessels causes the vessel-wall displacement,
which is measurable at various points of the periphery circulatory system.
Two methods are commonly used to measure the heart rate by the optical
method. These are:
1. Transmittance method
2. Reflectance method
Transmittance method: In the transmittance method, the light source and
the light sensitive detector are mounted in an enclosure that fits over the
patient’s fingertip or ear-lobe.(Algorri et al., 2013)
Fig. 2.34: Arrangement of light source and light sensitive detector: Transmittance
method
Light is transmitted through the finger tip of the patient’s finger and the
output of the light sensitive detector is determined by the amount of light
reaching the detector. With each contraction of the heart muscles, blood is
forced to the extremities and the amount of blood in the finger increases.
This alters the optical density with the result that the light transmission
through the finger reduces and the resistance of the light sensitive detector
increases accordingly.
59
Reflectance method: This method is used in this research. The
arrangement
used
in
the
reflectance
method
of
photoelectric
plethysmography is shown in the Fig 2.15. In the reflectance method, the
light sensitive detector is placed adjacent to the light source. Part of the
light rays emitted by the light source is reflected and scattered from the
skin tissues and falls on the photodetector.(Park et al., 2013)
Fig. 2.35: Arrangement of light source and light sensitive detector:
Reflectance method (Park et al., 2013)
The quantity of light that is reflected is determined by the tissue backscattered or the absorbed optical radiation. The output of the photodetector
varies in proportion to the volume changes of the blood vessels.
The signal from the heart rate sensor is then sent to a part of the
microcontroller where all the processing takes place for the beats per
minute (BPM) value calculation. The timer is programmed in an autoreload mode, so that it overflows at a regular interval and generates an
interrupt at 10µsec intervals.
In order to generate the interrupts at 10µsec interval, the reload value for
the timer had to be calculated for a system clock of 22.118 MHz. The
timer low byte (TL0) operates as a 16-bit timer while the timer high byte
(TH0) holds the reload value. When the count in TL0 overflows, the timer
flag is set and the value in TH0 is loaded into TL0. The TH0 value was
calculatedusingthe following equation:
(3)
Tsysclk = 60
Fsysclk is a system clock frequency of 22.118 MHz
i.e.
Tsysclk =
.
=0.045µsec
(4)
2.7.1. C Skin Temperature
The human skin is an organ made up of a layer of tissues that protect the
underlying muscles and organs. As skin comes in a direct contact with the
surroundings, it plays a vital role in protecting the inner body from the
external threats. The skin is the largest organ of the human body, as it
covers the whole body and has the largest surface area. It weighs more
than any single organ of the body. (Kenefick et al., 2010)The skin has two
major layers: the epidermis and the dermis. These layers are made of the
different types of tissues and have different functions. The epidermis is the
outer-most layer and the dermis lies below the epidermis and contains a
number of structures that are responsible for lubrication, water-proofing,
softening, and anti-bactericidal actions.
The skin temperature is an effective indicator when it comes to evaluate
the human sensations. Kataoka et al. (Shuto et al., 2011) investigated the
relationship between the stressful tasks and the skin temperature. It was
found that the skin temperature falls when stress, tension, or other
sensations occur; because the blood flow decreases due to the factors lie
blood vessel constriction. This was most noticeable at the extremities, such
as fingertips and nose. Similarly, according to Blessing(Ootsuka et al.,
2011), the net heat transfer between the individual and the external
environment varies according to the amount of the blood flowing through
the skin, which is regulated as an intrinsic component of the body
temperature control. The non-metabolic factor influencing the cutaneous
blood flow is a sympathetically mediated vigorous vasoconstriction
initiated when the individual perceives a potentially dangerous
environmental event. Yamakoshi et al.(Yamakoshi, 2013) studied driver’s
awareness level using the skin temperature. The researchers measured the
facial skin temperature, including the truncal and peripheral site, of
61
healthy volunteers during simulated monotonous driving. They found that
the sympathetic activity, that is, peripheral vasoconstriction
vasoconstriction was increased
during the monotonous driving situation, which resulted in a significant
gradual drop in the peripheral skin temperature.
Temperature Sensor and Measurement
Various equipments and instruments have been used in the past for the
body temperature
mperature measurement. The most common device to measure the
body temperature is a thermometer. Thermometer is a combinationof two
Greek words; "thermo," which means heat and "meter," which means
measure. Therefore, a thermometer is a device that detects the
t change in
the heat level and converts it into a temperature value.(Boano
value.(Boano et al., 2011)
2011
There are different types of thermometers. The most common ones include
mercury-in-glass,
glass,
infrared,
gas,
plastic strip,
and
liquid crystal
thermometers. However, the mercury-in-glass
mercury glass thermometers have widely
been used for the clinical purposes. The other
other devices that are used for
measuring the temperature include thermocouples, thermistors, resistance
temperature detectors (RTD), and silicon band gap temperature sensors.
All these temperature measuring devices are designed to measure the
temperature forr specific objects or environments. The temperature can be
measured using different scales. The most common temperature scales
used and accepted internationally are the Kelvin or Absolute, Centigrade
or Celsius, and Fahrenheit scale. (Yin et al., 2010)
Fig. 2.36:
2.
Temperature Sensor(Yu et al., 2010)
62
Choose R1 = –VS / 50 µA
VOUT = 1500 mV at 150°C
VOUT = 250 mV at 25°C
VOUT = –550 mV at –55°C
In this research LM35 is used as a temperature sensor. The LM35 series
are precision integrated-circuit temperature sensors with an output voltage
that is linearly proportional to the Centigrade temperature. Thus the LM35
has an advantage over the linear temperature sensors that are calibrated in
Kelvin, as the user is not required to subtract a large constant voltage from
the output to obtain a convenient Centigrade scaling. LM35 does not
require any external calibration or trimming to provide the typical
accuracies of ±¼°C at the room temperature and ±¾°C over a full −55°C
to +150°C temperature range. A low cost is assured by trimming and
calibration at the wafer level. The low output impedance, linear output,
and precise inherent calibration of the LM35 sensor make interfacing to
readout or control circuitry especially easy.
The device is used with single power supplies, or with plus and minus
supplies. As the LM35 sensor draws only 60 µA from the supply, it has
very low self-heating of less than 0.1°C in the still air. The LM35 is rated
to operate over a −55°C to +150°C temperature range, while the LM35C is
rated for a −40°C to +110°C range (−10° with improved accuracy). The
LM35 series is available in the hermetic TO transistor packages, while the
LM35C, LM35CA, and LM35D are also available in the plastic TO-92
transistor package. The LM35D is also available in an 8-lead surfacemount small outline package and a plastic TO-220 package.
There are a number of devices available for monitoring or observing the
human temperature. In this research, the aim was to go for a low-cost,
compact, reliable, and accurate temperature sensor that is capable of
monitoring the skin temperature with ease and comfort.
As stated earlier, the output from the temperature sensor is an analog
voltage. This output signal from the sensor is used as the input for the
smicrocontrollerthrough the analog port pin. The microcontroller is
63
programmed to perform the required processing and conversion from a
voltage value into a temperature value. The relationship between the
voltage value and the temperature value is calculated by the following
equation:
T(°C) =
(5)
∆/∆
Where
Vos=Ds offset, 509mv
∆V/∆T=Typical output gain,+6.45 mV/°C
2.8 Microcontroller Overview
The popular of all the electrical systems today employ some sort of
microcontroller technology. A microcontroller’s inexpensive, flexible,
and self-sufficient design permits it to command almost any modern task
that employs some form of embedded systems. From cars to refrigerators
to handheld devices, microcontrollers play a dominant role in the
development of many different products for many different companies.
In this research, the Microcontroller used is MSP430F2013.The
MSP430F2013 includes a 16bit CPU, 16-bit timer,16-bit Sigma Delta
Analog-to-Digital converter, brownout detector,Watchdog timer, USI
module supporting SPI and I2C serial communication standards, and five
low power modes drawing as little as 0.1µA standby current. TI’s UltraLow Power microcontroller, MSP430, uses EZ430, a USB stick
implementation of a full development kit that includes power supply, I/O
access, additional debugging hardware, and few extra peripherals.
64
Fig. 2.37:: EZ430-F2013
EZ430
- MSP430 16-bit
bit microcontroller USB Stick
The MSP430 (the controller for the EZ430) employs a Reduced
Instruction Set Computer architecture (RISC) CPU. The eZ430-F2013
eZ430
is a
complete MSP430 development tool including all the hardware and
software to assess the 16-bit
16 bit mixed signal microcontroller MSP430F2013
MS
and to develop a complete solution that works in a suitable USB stick form
factor. The eZ430-F2013
eZ430 F2013 supports the Code Composer Studio and IAR
Embedded Workbench Integrated Development Environments to provide a
full emulation with the option of designing
des
using a stand-alone
alone system or
detaching the removable target board to integrate into an existing design.
The USB port provides enough power to operate the ultra-low-power
ultra
MSP430, so no external power supply is required.
2.8.1 MSP430F2013 Architecture
Architectu
The MSP430CPU has a 16-bit
16 bit RISC architecture that is highly clear to the
application. All operations, other than the program-flow
program flow instructions, are
performed as registered operations in conjunction with seven addressing
modes for source operand and four addressing modes for destination
operand. The CPU is integrated among 16 registers that provide a reduced
instruction execution time. The register-to-register
register register operation execution
time is one cycle of the CPU clock. Four of the registers, R0 to R3, are
respectively
pectively designated as the stack pointer, constant generator, program
65
counter, and status register. The remaining registers are called the generalgeneral
purpose registers. The peripherals are connected to the CPU using the
address, data, and control buses. It can
can be handled with all
instructions.(Frederic
(Frederic et al., 2013)
Fig. 2.38:: Architectural view of MSPF2013(Megalingam
MSPF2013(Megalingam et al., 2011)
The MSP430 von-Neumann
von Neumann architecture has one address space shared
with the special function registers (SFRs), peripherals, RAM, and
Flash/ROM memory. The device-specific
device specific data sheets are available for the
specific memory maps. The code accesses are always performed on the
even addresses. The data can be accessed as bytes or words. The
addressable memory space currently is 128 KB.
The
he CPU incorporates sixteen 16-bit
16 bit registers. R0, R1, R2, and R3 have
dedicated functions. The 16-bit
16
program-counter
counter (PC/R0) points to the
next instruction to be executed. The stack pointer (SP/R1) is used by the
CPU to store the return addresses of the subroutine
subroutine calls and interrupts.
The status register (SR/R2), used as a source or destination register, can be
used in the register mode only addressed with word instructions. The RISC
instruction set of the MSP430 has only 27 instructions. The constant
generator
erator allows the MSP430 assembler to support 24 additional and
66
emulated instructions. The twelve registers, R4 to R15, are generalgeneral
purpose registers. All of these registers can be used as data registers,
address pointers, or index values; and can be accessed
accessed with byte or word
instructions. Seven addressing modes for the source operand and four
addressing modes for the destination operand can address the complete
address space with no exceptions.
2.8.2 Modular design
The following PCB diagram shows the arrangement
arrangement of the hardware on
the EZ430. Notice how the actual MSP430 attaches to the debugging and
USB interfacing hardware through a 4-pin,
4
JTAG port.
Fig. 2.39:
2.
PCB diagram(Zantis, 2012)
Dedicated embedded emulation logic resides on the device itself and is
accessed via JTAG using no additional system resources.
The benefits of embedded emulation include:
• Unobtrusive development and debugging with full-speed
speed execution,
breakpoints, and single-steps
single steps in an application are supported.
• Development is in-system
in system subject to the same characteristics as the
final application.
• Mixed-signal
signal integrity is preserved and not subject to
to the cabling
interference
The eZ430-F2013
F2013 can be used as a stand-alone
stand alone development board.
Additionally, the MSP-EZ430D
MSP EZ430D target board may also be detached from
the debugging interface and integrated into another design. The plastic
enclosure can be removed
remove to expose the MSP-EZ430U
EZ430U debugging
67
interface and the MSP-EZ430D
MSP EZ430D target board. The MSPEZ430D target
board can be disconnected from the debugging interface by gently pulling
the two boards apart. The target board can be used in a stand-alone
stand
design
by interfacing
erfacing to the 14-pins
14 pins of the MSP430F2013. The holes in the MSPMSP
EZ430D target board provide a direct access to each pin of the
MSP430F2013. The MSP-EZ430U
MSP EZ430U debugging interface may also be used
as a standard Flash Emulation Tool for all devices in the MSP430F20xx
MSP43
family of the microcontrollers. The target boards can be designed and
flashed using the MSP-EZ430U
MSP EZ430U debugging interface and other supported
MSP430F20xx devices.(Gaspar
devices.
et al., 2010)
Fig. 2.40: Pin Diagram of MSP430F2013(Mainoddin
MSP430F2013(Mainoddin and Usha, 2014)
There is one 8-bit
bit I/O port implemented—port
implemented
P1—and
and two bits of I/O
port P2. All individual I/O bits are independently programmable. Any
combination of input, output, and interrupt condition is possible. The
edge-selectable
selectable interrupt input capability is available for all the eight bits
of port P1 and the two bits of port P2. The read/write access to the portport
control registers is supported by all instructions. Each I/O has an
individually programmable
programm
pull-up/pull-down resistor.(Mainoddin
(Mainoddin and
Usha, 2014) Following is the table that describes
describes the details of each pin of
the microcontroller that is used here:
68
Table 2.4: Details of each Pin
Pins
P1.0/TACLK/ACLK/C
A0
Details
General-purpose digital I/O pin Timer_A, clock signal
TACLK input ACLK signal output Comparator_A+,
CA0 input
General-purpose digital I/O pin Timer_A, capture:
P1.1/TA0/CA1
CCI0A input, compare: Out0 output Comparator_A+,
CA1 input
General-purpose digital I/O pin Timer_A, capture:
P1.2/TA1/CA2
CCI1A input, compare: Out1 output Comparator_A+,
CA2 input
P1.3/CAOUT/CA3
General-purpose digital I/O pin Comparator_A+, output
/ CA3 input
General-purpose digital I/O pin SMCLK signal output
P1.4/SMCLK/C4/TCK
Comparator_A+, CA4 input JTAG test clock, input
terminal for device programming and test
General-purpose digital I/O pin Timer_A, compare:
Out0 output ADC10 analog input A5 USI: external
P1.5/TA0/CA5/TMS
clock input in SPI or I2C mode; clock output in SPI
mode JTAG test mode select, input terminal for device
programming and test
General-purpose digital I/O pin Timer_A, capture:
P1.6/TA1/A6/SDO/SCL
/TDI/TCLK
CCI1B input, compare: Out1 output ADC10 analog
input A6 USI: Data output in SPI mode; I2C clock in
I2C mode JTAG test data input or test clock input
during programming and test
General-purpose digital I/O pin ADC10 analog input
P1.7/A7/SDI/SDA/TDO
A7 USI: Data input in SPI mode; I2C data in I2C mode
/TDI+
JTAG test data output terminal or test data input during
programming and test
XIN/P2.6/TA1
XOUT/P2.7
Input terminal of crystal oscillator General-purpose
digital I/O pin Timer_A, compare: Out1 output
Output terminal of crystal oscillator General-purpose
digital I/O pin
69
RST/NMI/SBWTDIO
Reset or non maskable interrupt input Spy-Bi-Wire test
data input/output during programming and test
Selects test mode for JTAG pins on Port1.The device
TEST/SBWTCK
protection fuse is connected to TEST. Spy-Bi-Wire test
clock input during programming and test
VCC
Supply voltage
VSS
Ground reference
2.8.3 The SD16 A Sigma-Delta ADC
An ADC takes an analog signal as an input and then converts that analog
signal into a digital stream of bits depending on its reference voltage,
precision, and resolution. An n-bit ADC (A/D converter) provides 2n
discrete quantization levels corresponding to various specified analog
input signal amplitude range. There exist a number of A/D conversion
techniques varying in complexity and speed. The outputs from each sensor
are analog in nature. The output signal from the sensor is used as an input
into the analog port pin of the microcontroller. The MPS430F2013 is
equipped with an analog-to-digital (ATD) conversion system that samples
an analog (continuous) signal at regular intervals and then converts each of
these analog samples into its corresponding binary value using a sigmadelta modulation technique. As MSP430F2013 is having an in-built ADC
(SD16 A Sigma-Delta), so the microcontroller is programmed to perform
the required processing and conversions.
The SD16 A is a single-converter 16-bit, analog-to-digital conversion
module implemented in the MSP430x20x3 series. It is made up of one
sigma-delta analog-to-digital converter and an internal voltage reference. It
has eight fully differential multiplexed analog input channels, of which
three are internal. The operation of the sigma-delta converters is totally
different from the successive-approximation ADCs. The idea behind them
is to reduce the analog-to-digital conversion to 1 bit 1 and to take the
samples a few orders faster than the desired sample rate to compensate for
its very poor resolution. The magnitude of the analog input is then
70
represented by the mean value of the produced fast bit-stream. The
average is then digitally processed to output the samples at the specified
rate. The Fig 2.20 shows the architecture of a sigma-delta converter. It can
be broken down into two parts: the first, with the feedback loop, is
responsible for the analog-digital conversion, whereas, the second converts
the fast bit-stream to the desired sample rate.(Zantis, 2012)
.
Subtrator
Integrator
+
ADC
Analogue
Input
fs
fm
fm
Low-Pass
Filter
Decimator
Digital
Input
DCA
Fig. 2.41: Block diagram of a sigma-delta
A/D converter
Modulator
Decimation
Filter
Fig. 2.21:Analog-To-Digital Conversion
The analog-to-digital conversion is done by a 1-bit second-order sigmadelta modulator. A single-bit comparator within the modulator quantizes
the input signal among the modulator frequency fM. The resulting 1-bit
data stream is averaged by the digital filter for the conversion outcome.
The bit-rate of the first part is called the modulator or oversampling
frequency (fm). This is typically much faster than the sample rate (fs) at
the digital output. The decimation filter is a comb type digital filter with
selectable oversampling ratios (OSR = fm/fs) of up to 1024. The filter is
also called sinc filter because its frequency response is alike the sinc(x) =
sin(x)/x function. The comb filter is the sigma-delta converter’s
characteristic feature, which has to be taken into account through the
design stage. One may think that it is a downside, however, when it comes
to anti-aliasing or notch filtering, it can be utilised by a sensible software
design. The ADC converts the ∆V = V+ − V− voltage difference among a
pair of inputs, rather than the voltage between a single input and the
ground. If this feature is not required, the V− should be tied to the ground.
The sigma-delta converters often give a programmable gain amplifier
71
(PGA) on their inputs, which may eliminate the need for an additional
external operational-amplifier. These are the plain op-amps with the
feedback resistors, and they do not provide high input impedance. Their
analog input voltage range is dependent on the actual gain setting, which
can be increased up to 32 in the SD16 A. The maximum full-scale range
for Vref = 1.2V and GAINPGA = 1 is ±VF SR, where VF SR is defined
by:
VFSR =
/
!"#
=
./
= ±0.6V
(7)
A side effect of the averaging applied in the digital sinc3 filter is that the
output does not react promptly to the change of the input. It needs 4
periods of Ts to elapse until the reliable value appears. This is called
latency, and probably sets the most severe limitation of sampling
frequency when more than one channel is used.
The F2013 SD16_A conversion system consists of an 8-channel
multiplexed input anda 16-bit output sigma delta analog-to-digital
converter block. Its features include a software selectable internal/external
voltage, up to a 1.1 MHz modulator input frequency, and a selectable lowpower conversion mode. The converter block is software programmable to
perform either single or continuous conversions into a 16-bit output
register that is called the SD16MEM0 register. The SD16_A module must
be initialized using its two control registers, the SD16 control and channel
control (SD16CTL & SD16CCTL0) registers. When the SD16_A module
is not actively converting, it automatically shuts down to preserve the
power while putting together an accurate analog to digital domain
conversion. The following algorithm was used in this research for
converting the analog signal to a digital one.
72
Algorithm 2.1: Efficient Algorithm for A-D conversions
STEP 1: SD16CTL = SD16REFON + SD16SSEL_1;
// Internal Voltage Ref ON and Clock Division
STEP 2:SD16CCTL0 = SD16UNI;
// Changing SD16 to Unipolar Mode
STEP 3:SD16INCTL0 = SD16INCH_1;
// Selecting Input channel
STEP 4:SD16CTL = SD16REFON + SD16SSEL_1;
// Internal Voltage Ref ON and Clock Division
STEP 5:SD16CCTL0 = SD16UNI;
// Changing SD16 to Unipolar Mode
STEP 6:SD16INCTL0 = SD16INCH_N;
// Selecting Input channel
The SD16CTL register: The SD16_A Control Register is mainly
responsible for the selection of the clock source, the division of the clock
into the sigma delta modulator, and the enablement of the internal voltage
reference.
The SD16 Clock Source Select (SD16SSELx) (Bits 5 – 4): The clock
source to be divided is selected using the clock source select bits, much
like the timer module.
The SD16 Reference Generator ON (SD16REFON) (Bit 2): The SD16_A
module can use an internally provided reference voltage for the
modulation or it can be provided as a user-specified voltage reference
through the specified ports. The internally provided reference voltage has
a value of 1.2 V and is used when the SD16REFON bit in the SD16CTL
register is set to 1.
73
Table 2.5: Voltage Reference Generator Bit
SD16REFON Bit
Internal Voltage Reference
0
Reference OFF
1
Reference ON
The SD16CCTL0 register: The Channel Control 0 Register is responsible
for the conversion mode, the data output settings, the oversampling ratio,
and all interrupt settings. There are two modes – Bipolar and Unipolar. In
this research,only the Unipolar mode was required. The mode is selected
as follows:
Bipolar Mode, SD16UNI = 0
Unipolar Mode, SD16UNI = 1
SD16INCTL0: The analog input into the machine is configured using the
Input Control (SD16INCTL0) and Analog Input Enable (SD16AE)
registers. Setting the SD16AE bits, enable the analog circuitry for the
particular differential pair of input pins and disable any digital circuitry
that might be linked to that pin.
SD16INCHx: The SD16INCTL0 Register is dependable for setting the
selected input channel and the SD16INCHx Bits (0 – 2) are responsible for
selecting the analog input to be modulated.
Key Features of MSP430F2013 Microcontroller:
eZ430-F2013 development tool including a USB debugging interface and
detachable MSP430F2013 target board has the features below:
•
LED indicator
•
14 user-accessible pins
•
eZ430 debugging and programming interface
•
Supports development with all 2xx Spy Bi-Wire devices
(MSP430F20xx, F21x2, F22xx)
74
•
Supports eZ430-T2012 and eZ430-RF2500T target boards
•
Removable USB stick enclosure
•
Low Supply Voltage Range 1.8 V to 3.6 V
•
Ultra-Low Power Consumption
•
Active Mode: 220 µA at 1 MHz, 2.2 V
•
Standby Mode: 0.5 µA
•
Off Mode (RAM Retention): 0.1 µA
•
Five Power-Saving Modes
•
Ultrafast Wake-Up from the Standby Mode in less than 1 µs
•
16-Bit RISC Architecture with 62.5 ns Instruction Cycle Time
•
16-Bit Timer_Awith Two Capture/Compare Registers
•
On-Chip Comparator for Analog Signal Compare Function or Slope
A/D
•
16-Bit Sigma-Delta A/D Converter With Differential PGA Inputs
and Internal Reference
Kit Contents
The evaluation kit contains everything that is needed to develop and run
applications for the MSP430 microcontrollers. It includes:
One eZ430-F2013hardware set, which is housed inside a plastic
enclosure that may be opened in order to separate the MSP-EZ430D
target board from the MSP-EZ430U debugging interface
One MSP430 Development Tool CD-ROM, which contains several
documents including the following related to the eZ430-F2013:
•
MSP430x2xx Family User's Guide
•
MSP-FET430 FLASH Emulation Tool User's Guide
75
•
MSP-FET430 FLASH Emulation Tool User's Guide Errata
•
eZ430-F2013 User's Guide
•
IAR Embedded Workbench Kickstart Version
•
Code Composer Studio MCU Edition
Software Design
To develop the application software for the data storage tag, the IAR
Embedded Workbench is used. The IAR Embedded Workbench is a set of
development tools for building and debugging the embedded applications
using assembler, C, and C++. The 16 bit MSP430 devices from Texas
Instrument are supported by the IAR tool. The IAR development tool can
generate a binary file that can be downloaded on the microcontroller. The
status of all the interval registers related to the microcontroller’s
peripherals has already been discussed in the MCU architecture.There are
two drivers available to continue with the software development process.
The IAR tools provide the facility to simulate the device operation without
any hardware. This feature allows the designer to start developing the
software for the application even before any hardware is built. The second
option is to debug the hardware with the emulator, that is, the USB shaped
device.
The emulator is a complete set of developing tools that provide all the
hardware and software to evaluate the MSP430-F2013 microcontroller.
This USB stick shaped device is compatible with the IAR embedded
workbench integrated development environment (IDE). The IAR tool is
used to compile the application software for the prototype board. The
debugging interface contains a USB port and a Spi-By Wire Interface that
is incorporated to download the binary version of the software on the
microcontroller. The primary function of the watchdog timer (WDT+)
module is to perform a controlled system restart after the software problem
occurs.
76
Software Tools
The software for the EZ430, IAR Embedded Workbench, comes free with
the purchase of the tool. Though it is a “kickstart” version, (which meansit
has a 4kB limit of code), the standard microcontroller with which it comes
is limited to 2KB of memory. The IAR carries both a C compiler and an
assembler. The code size limitations would be an issue if the
microcontroller class was taught for the development in the C
programming language, but the fact that the EZ430 is used for an
introductory course in the assembly language makes the limitations nonrestrictive.
To create a new project, select Project>Create New Project. In the dialog
box that appears, choose "MSP430" in the Tool chain and "Empty project"
in the Project templates. The empty project appears in the Workspace
window on the left-hand side. Before adding any files to the project, the
workspace should be saved by File>SaveWorkspace;provide a valid file
name. Choose File>Add Files to open a dialog box in which the files can
be selected; click open to add the files to the project. After the
programming, the application needs to be downloaded. However, you
must first choose Project>Rebuild All to finish the compiling and linking.
2.9 Conclusion
In this chapter, a comprehensive work on the design and development
creativity was conducted. This chapter demonstrated a research through
hardware implementation. GSR, BVP, and Temperature displayed the
capable results for use in identifying and differentiating the physiological
arousal. This chapter also discussed the proposed architecture and the
design implementation in detail. This proposed architecture was designed
for making the system portable, easy to use, and intelligent. This chapter
has provided a detailed explanation of the first two objectives.
77
Chapter 3
EMOTION
DETECTION
METHODOLOGY
AND ANALYSIS
This chapter talks about investigations made on emotion-specific ANS
responses and recognition using classification algorithms. Chapter
describes methodology by implementing two different types of machine
learning algorithms for data classification. These are Naïve Bayesian and
Markov model-based algorithms implemented on a Texas Instruments
MSP430 microcontroller. Beside these algorithms, HYBRIB-NAV-MAR,
a hybrid of the two basic algorithms, is also implemented on the
microcontroller.
3.1 Introduction
The main focus of this research is developing an intelligent system that
can identify its users’ emotional states(Craig et al., 2004). Despite the
importance of emotions in our lives, many Human-Machine-Interfaces
(HMI) completely lack the “emotional intelligence”(Goleman, 2006).
Emotional intelligence helps us to understand and manage our own
emotions as well as other people’s emotions towards us. Emotion
recognition is one of the important steps towards the emotional
intelligence in an advanced human-machine interaction(Shibata et al.,
1997). Recently, emotion recognition using the physiological signals(Kim
et al., 2004) has been performed by various machine learning algorithms,
as physiological signals are important for the emotion recognition abilities
of the human-computer systems(Picard et al., 2001). The purpose of this
research study is to classify three different emotional states (joyful, calm,
and stress) with respect to the physiological signals using several machine
learning algorithms, which will further enable us to predict the future
emotion based on data collected by the system so far.
78
3.2 Background and Related Works
To understand an emotion recognition system, there are some important
concepts that have been discussed in this section. Emotion(Lazarus, 1991)
is a complicated state of feeling for the mankind, for example, sorrow,
fear, joy, and hate. In physiology, emotion is often related with the arousal
of the nervous system and may be accompanied by the physiological
changes, such as bigger respiration or heartbeat(Bonnet and Arand, 1997).
The major difference between mood and emotion is the duration of the
feeling. A fine or terrible mood may last for one or two days, but an
emotion may last just for a few seconds or maybe minutes.
In 1983, Gardner gave a widely held notion that intelligence is a unitary
capacity for the logical reasoning possessed by every individual to a
greater or lesser extent; the concept of multiple intelligences has strongly
affected psychology, educational theories, and neuroscience.(White and
Gardner, 1983) This idea has also entered the computer science, where it
has been used to support the establishment of the affective computing, or
the computing that arises from and deliberately influences emotions
(Picard, 1997).
Affective computing lies on the border of artificial intelligence and
anthropomorphic interface design. It aims at enriching the rule-based
systems of artificial intelligence with emotional modules to recognize the
user emotions and to give machines emotions. Emotional intelligence
(Salovey et al., 2000) is regarded as the ability to perceive and express
emotions, to understandably manage and use them, and to foster the
personal well-being. The concept subsumes Gardner’s interpersonal and
intrapersonal intelligences in a unique emotional space, so as to
differentiate the specific emotional competencies from the social ones.
Machines do not have emotions; these cannot feel happy on seeing us, sad
when we go, or bored when we don’t give them enough interesting input.
But “Emotional Intelligence” could address several problems that exist
today, while enabling the invention of better technologies for the future. In
79
some contexts, for example, medical monitoring or use of other wearable
systems with physiological sensors can be natural and comfortable to work
with. These can help in sensing information, such as changes in the heart
rate, muscle tension, temperature, skin conductance, and more. For
example, if a patient is wearing a heart monitor for health monitoring or
for fitness tracking then the technology can potentially measure the heartrate variability changes associated with the cognitive and emotional stress.
Physiological information has been shown to carry the information that
changes with different emotions (Ekman, Levenson et al. 1983) and such
information can be used to build classifiers for an individual’s affective
state.
3.3 Emotions and Emotional intelligence
Emotions are a complex state of feeling that affect/bring about the
physical and psychological changes that control our behaviour. The
physiology of emotion is directly linked to the arousal of the nervous
system due to various states apparently, to particular emotions.
Emotional Intelligence (EI)(Mayer et al., 2000) is the ability to monitor
emotions of one's own and other people's that classify between different
emotions and label them appropriately, and to use the emotional
information to guide thinking and behaviour. Emotional intelligence
encompasses several abilities that can hierarchically be ordered in four
branches composed of several sub-skills organized according to their
complexity.(Mayer et al., 2003)
1 Identifying / perceiving emotions
2 Using emotions
3 Understanding emotions
4 Managing emotions.
80
This research explains the contribution in the field of perceiving emotions.
Perceiving emotion has the ability to identify the emotion in oneself and in
others; it also has the ability to tell the difference between the honest and
dishonest emotions. At the basic level, there is the ability to perceive,
appraise, and express the emotions accurately.
These are the basic information-processing skills in which the relevant
information consists of feeling and mood states. The emotions can be
identified in one’s own physical and psychological states as well as in
other people and objects. Basic skills also include the ability to express
emotions and needs related to these feelings; and evaluating accuracy and
honesty in the expression of the feelings. The second branch, using
emotion, refers to the use of emotions as thinking facilitation. Different
emotions induce different information-processing styles; hence, emotional
states can be harnessed by an individual towards a number of ends, such as
stimulating creativity and
problem
solving.
The third
element,
understanding emotion, concerns essential knowledge about the emotional
system. The most fundamental competencies at this level, concern the
ability to label the emotions with words, perceive the causes and
consequences of emotions, understand how different emotions are related,
and interpret the complex feelings. This knowledge contributes to the
fourth branch, mood maintenance, which is regarding the regulation of
emotion and mood repair strategies. In order to put the knowledge into
action, people must develop further competencies. They must be open to
feelings - pleasant as well as unpleasant. Then, they need to practice and
become adept at engaging in behaviours that bring about the desired
feelings in themselves and in others. Design explains a real-time
monitoring system that is capable of estimating different emotions from
different biofeedback signals, especially for the people who cannot express
their emotions, for example, the paralysis stricken patients. This system
will be user–friendly as shown in the following diagram.
81
Fig 3.1: Sequence of emotion
Although people perceive that stress can have negative impact on health
and well-being,
being, they normally do not take any action to prevent stress or to
manage it. Effectively detecting the stress well in-time
in time not only provides a
way for the people to understand their stress condition better but also
provides the physicians with more reliable data for intervention and stress
control. Identifying the stress level using the psychological sensors has
been a hot research topic in the recent years. The existing studies have
shown that psychosocial stress can be recognized by the physiological
information of human being. The physiological information can be
acquired by using the biological or physiological sensors(Healey
sensors
and
Picard, 2005),, such as Elecardiagram (ECG), Galvanic
Galvanic Skin Response
(GSR), Electromyogram (EMG), and Respiration (RESP).
3.4 Methods for Recognizing Emotions
Monitoring the emotions is significant, as it contains the information that
can assist in improving human well-being.
well being. It is also important to observe
observ
the emotions as these are the perceptions of bodily changes and can assist
in identifying any medical condition at an early stage, that is, before it
becomes serious. Emotion regulation is an important skill for coping with
social and personal troubles(Gross
troubles(Gross and Thompson, 2007).
2007) Emotion
recognition has also become an important subject in the human-machine
human
interaction scenario in the present times.
times. Various methods have been used
in the past to detect human emotions. The most commonly used techniques
are explained below:
82
Emotion Recognition Using Text(Wu et al., 2006):Emotion recognition
using text has become a popular method these days, especially when it
comes to human-machine interaction. Textual information is not only an
important communication medium that exists in books, newspapers,
websites, emails etc, but also a rich source of emotion. Different
approaches have been used to recognize and evaluate these emotions. The
most common approach uses the natural language processing techniques,
which in-turn extract emotions and sentiments by analyzing the text input.
Zhang et al developed a semi-automatic acquisition technique to obtain the
emotion information using a sentence or text.(Chuang and Wu, 2004)
Emotion Recognition Using Facial Expressions(Busso et al., 2004, Zhao
and Pietikainen, 2007):The basic idea of emotion recognition using facial
expression is to segment facial images into various regions of interest. The
common regions taken into account include movements of cheek, chin,
wrinkles, eyes, eyebrows, and mouth. Different classification techniques
are
then
applied
to
differentiate
between
different
types
of
emotions.(Cowie et al., 2001) developed an intelligent emotion
recognition system, interweaving psychological findings about the
emotion representation with analysis and evaluation of the facial
expressions.
Emotion Recognition Using Speech(Cowie and Cornelius, 2003):Emotion
recognition from speech has become increasingly popular, as it has
become an important part of the affective computing and can help in
improving the human-machine interaction. It can also be used in various
applications, such as call centre conversation analysis, entertainment,
indexing of audio files based on emotions, and many more.
Emotion Recognition Using Body Movements and Gestures(Gunes and
Piccardi, 2007): Ginevra et al. proposed an approach for the detection of
four emotional states (anger, joy, pleasure, and sadness) based on the
analysis of body movement and gesture expressivity . They used nonpropositional movement qualities (amplitude, speediness, and variability
83
of movement) to infer emotions and investigate the role of movement
expressivity versus the shape in gesturing. Their proposed method
analyzed the emotional behaviour based on the direct classification of time
series and on a model that provides indicators explaining the dynamics of
significant motion cues.
Emotion Recognition Using a Data-Driven Fuzzy Inference System(Chul
M et al.,2003) This paper explores the detection of domain-specific
emotions using a fuzzy inference system to sense two emotion categories,
negative and nonnegative emotions. The input features are a mixture of
segmental and suprasegmental acoustic information; feature sets are
selected from a 21-dimensional feature set and implemented to the fuzzy
classifier. Such kind of model are mostly implemented with factors which
not predictable like voice, body gestures etc.
Emotion Recognition Using Physiological Signals(Horlings et al., 2008):
Emotion recognition using biosensors has recently become popular not
only for the interaction between humans but also in the human-machine
interactions.
Biosensors
have
the
advantage
of
monitoring
the
physiological parameters of the body; these physiological parameters are
directly controlled by the autonomous nervous system and are affected by
the emotions. These sensors can collect various signals including heart
rate, skin conductance, electrocardiogram, blood volume, and temperature;
and then can evaluate the emotions based on the changes taking place.
This research has evaluated the emotion recognition using the
physiological signals, as this method gives true readings where the subject
cannot manipulate the data. The machine was developed with a motive to
extract the emotions from the paralyzed people who cannot express their
emotions.
3.5 Emotion Model
There are different emotion perspectives. The first perspective describes
that some emotions are present in humans from infancy, in the sense that
84
those emotions can be adapted later on to a specific value, without
crossing a particular threshold; this is the discrete emotion theory and is
also named as Ekman’s basic emotions. The second perspective is the
dimensional theory that categorizes all kinds of emotions in a 2dimensional space, postulating that every emotion has two aspects: a
cognitive (Valence) and a physiological (Arousal) component.
Many theorists define the emotional models according to various
dimensions. Paul Ekman’s theory has six basic emotions: happiness,
sadness, anger, disgust, fear, and surprise (Ekman, 1992). He explains that
the basic emotions allow particular characteristics to be expressed in
various degrees.
Fig 3.2: Plutchik’s Model (Plutchik et al., 1997)
Pulkkinen’s model of emotional regulation presents a map of emotions
that are divided into two dimensions: (a) high self-control vs. low self
control and (b) inhibition vs. expression. The model identifies 11
emotions: reserved, passive labile, aggressive, active, responsive,
constructive, stable, thoughtful, compliant, anxious, and impulsive.
Pulkkinen also divides the emotions into four behavioral clusters: A, B, C,
85
and D. These prototypes are used for behavioral regulation. In his study,
he defines the eight primary emotions as: surprise, anticipation, sadness,
joy, trust, fear, disgust, and anger. The eight primary emotions are
arranged as four pairs of opposites: trust-distrust, joy-sadness, fear-anger,
and surprise-anticipation.
Fig 3.3: Russell’s Model (Russell and Feldman Barrett, 1999)
In the Russell’s model, shown in the above fig 3.3 it can be seen that the
emotions are divided by arousal and valence dimensions. The vertical axis
signifies arousal and the horizontal axis represents valence. In this model,
each emotion can be recognized by its varying degree of arousal and
valence. The eight primary emotions are defined as: active/arousal,
peppy/enthusiastic,
happy/pleased,
consent/calm,
quiet/passive,
sluggish/tired, sad/gloomy, and jittery/nervous (Russell and Feldman
Barrett, 1999). The centre of the circle means a neutral state. This model
has been widely used by various emotion classification tests and emotional
facial expression recognition (Remington et al., 2000).
86
3.6 Emotion Estimation Methodology
Emotion estimation or prediction can be done with data analysis by using
machine learning.(Preisach et al., 2008).In this research, data is analyzed
by machine learning algorithms embedded in microcontroller. The process
of data analysis is done in two phases; first phase is that when sensed data
is computed by algorithms for predicting emotion and second when final
predicted data output is collected to form training data. This chapter
explains first phase in the following sections.
3.6.1 Generic process of emotion identification
The process of detecting emotion using the physiological sensors normally
consists of three major phases. First, features are extracted from the raw
physiological sensor data using feature extraction algorithms. In order to
effectively identify the stress level or patterns, many features are to be
extracted from a variety of physiological sensors. Second, the most
relevant features are selected by using some feature selection heuristics.
More features extracted does not necessarily mean better performance of
stress identification. On the other hand, more features may bring-in the
useless information or even the misleading information. Selecting fewer
features and predicting emotion patterns as accurately as possible is a
challenging research work to do. Third and final, based on the selected
features, an information fusion procedure is applied to identify the stress
level or patterns. The fusion of methods used is Naïve Bayes (NB) and
Markov model (MM).
87
Raw Data
GSR/BVP/Temperature
Number of orienting responses
Extraction
Normalized mean
Correlation-based
Selection
Feature Selection
Naïve Bayes
Combining
algorithms
Hybrid-Nav-Mar
Markov Model
Emotional State
Stress/Calm/Joyful
Fig 3.4: Generic process of emotion identification
Numerous factors affect the success of machine-learning algorithms on a
given task. The demonstration and quality of the example data is the first
and prime. The feature subset selection is the process of identifying and
removing as much irrelevant and redundant information as possible.
Machine learning has taken inspiration from both the pattern recognition
and statistics. The feature selection algorithms perform a search through
the space of feature subsets. Selecting a point in the feature subset space,
from where to begin the search, can affect the direction of the search. One
88
option is to begin with no features and successively add attributes. An
exhaustive search of the feature subspace is prohibitive for all but a small
initial number of features. With N initial features, there exist 2N possible
subsets. The heuristic search strategies are more feasible than exhaustive
ones and can give good results, however, these do not guarantee finding
the optimal subset. The heuristic search strategies are the ones that have
been used for the feature selection.
The feature subsets evaluation is the single biggest differentiating factor
among the feature selection algorithms for machine-learning. These
algorithms use heuristics based on the general characteristics of the data to
evaluate the merit of the feature subsets. This method, called the
wrapper(Karegowda et al., 2010) uses an initiation algorithm along with a
arithmetical re-sampling technique, for example, cross-validation to
estimate the final correctness of the feature subsets.
A local change is simply the addition or deletion of a single feature from
the subset. When the algorithm considers only additions to the feature
subset, it is known as forward selection. The best first search(Yang and
Honavar, 1998) can back-track to a more promising previous subset and
continue the investigate from there. Given enough time, a best first search
will explore the entire search space.
Algorithm 3.1: Best first search algorithm
1
Initiate with the OPEN list having the start state, the
CLOSED list empty, and BEST←start state.
2
Let s = arg max e(x) (get the state from OPEN with the
highest evaluation).
3
Remove s from OPEN and add to CLOSED.
4
If e(s) ≥ e(BEST), then BEST ← s.
5
For each child t of s that is not in the OPEN or CLOSED list,
evaluate and add to OPEN.
6
If BEST changed in the last set of expansions, go to 2.
7
Return BEST
89
3.6.2 Machine learning algorithms for emotion prediction
Machine learning (Baldi and Brunak, 2001)is closely related to and often
overlaps the statistics which are computational; a discipline, which also
focuses on prediction-making. It dynamically learns to make accurate
predictions based on the past observations. The effectiveness of these
heuristics is then investigated by using the fusion computational
probabilistic algorithms for the final prediction within a microcontroller.
Input /Training Data
Machine Learning
Algorithm
Classification
Rule
Predicted
Classification
Fig 3.5: Machine learning approach(Baldi and Brunak, 2001)
Three broad categories of machine learning classification approaches
involve (a) unsupervised (b) supervised and (c) reinforcement learning of
the datasets in one of the possible classification states as shown in Fig.3.6.
In the present problem supervised learning approaches have been used.
The supervised learning is adopted because of its compatibility in
modelling and controlling dynamic systems.
In general, the classifiers are ranged from the linear classifiers (Logistic
regression, LBNC, Linear SVM) to the non-linear classifiers (K-NN, SVM
with poly kernel of degree 2, Decision Trees); and there is another called
probabilistic classifier (Naïve Bayes).
90
Classification Methods
Supervised
Learning
Unsupervised
Learning
Reinforcement
Learning
• Input (training
data) and output
(target data) both
labeled
• Classifier function
for discrete output
• Regression
function for
continuous output
• Finds hidden
structures in
unlabeled data
• Forms natural
clusters based on
similarity
• No error or
reward signal
• Trial-and-error
based approach
• Involve a
sequence of steps
• Decisions at each
stage affect the
decisions taken at
next steps
Fig.3.6: Classification Methods
In this research, emotion recognition using the physiological signals has
been performed by various machine learning algorithms, such as Naive
Bayesian, HYB-NAV-MAR, and Markov Model; and implemented within
a microcontroller. In the proposed model, to fulfil the need of
probabilities, the Naïve Bayes probabilistic classifier was used. For the
future predictions, the Markov model was implemented within the
microcontroller. By combining both the algorithms, a new algorithm,
HYBID-NAV-MAR, was designed and developed by the researchers.
Details of the different classification methods are given below:
91
3.6.2.A Naïve Bayes
Naïve Bayes is a classification method, which relies on the Bayes rule for
making the predictions, and it assigns the example x to the class label ωi
with the largest posterior probability P(ωi | x). Naïve Bayes relies on a
simplifying assumption that the predictive attributes are conditionally
independent given the class. This assumption specifically simplifies the
computational complexity that is associated with the estimating class
conditional probabilities, which are estimated for each attribute separately
(Theodoridis&Koutroumbas, 2006). Naïve Bayes has shown to offer a
competitive performance against other widely used and more sophisticated
classification methods, such as decision trees and neural networks (George
& Pat, 1995; Theodoridis&Koutroumbas, 2006). It is also a useful
classifier that can easily deal with the high dimensional data because of its
short training time (Hand et al., 2001).
The Bayesian Classification represents a supervised learning method as
well as a statistical method for classification. It assumes an underlying
probabilistic model and allows us to capture the uncertainty about the
model in a principled way by determining the probabilities of the
outcomes. It can solve various diagnostic and predictive problems.
Following are the uses of Naïve Bayes classification:
Naïve Bayes text classification(Zhang and Li, 2007), The Bayesian
classification is used as a probabilistic learning method (Naive Bayes text
classification). Naive Bayes classifiers are among the most successful
known algorithms for learning to classify text documents.
Spam filtering(Androutsopoulos et al., 2000) is the best known use of
Naive Bayesian text classification. It makes the use of a naive Bayes
classifier to identify the spam e-mails.
Hybrid Recommender System(Burke, 2002), Naive Bayes Classifier and
Collaborative Filtering Recommender Systems are used to apply the
92
machine learning and data mining techniques for filtering the unseen
information and to predict whether a user would like a given resource.
Online applications(Haffner et al., 2005), This online application has been
set up as a simple example of supervised machine learning and affective
computing. Using a training set of examples, which reflect nice, nasty, or
neutral sentiments, we are training Ditto to distinguish between them.
Naïve Bayes is further divided into three types classification(Lewis, 1998)
• Independence:
Example:
Suppose there are two events:
T: Manju teaches the class (otherwise it’s Anand’s)
R: It is rainy
“The rain levels do not depend on and do not influence who is teaching.”
Theory:
From P(R | T) = P(R), the rules of probability imply:
_ P(~R | T) = P(~R)
_ P(T | R) = P(T)
_ P(T ^ R) = P(T) P(R)
_ P(~T ^ R) = P(~T) P(R)
_ P(T^~R) = P(T)P(~R)
_ P(~T^~R) = P(~T)P(~R)
• Conditional Independence
Example:
93
Suppose we have these three events:
_ T: Lecture taught by Manju
_ A: Lecturer arrives late
_ R: Lecture concerns circuits
Suppose: Anand has a higher chance of being late than Manju. Anand has
a
higher chance of giving circuits lectures.
Theory:
R and A are conditionally independent given T is for all x, y, z in (Allen et
al., 2001):
P(R=x ½ T=y ^ A=z) = P(R=x ½ T=y)
(1)
More generally: Let S-1 and S-2 and S-3 be sets of variables. Set-ofvariables S-1 and set-of-variables S-2 are conditionally independent given
S-3 is for all assignments of values to the variables in the sets; P(S-1’s
assignments ½ S-2’s assignments & S-3’s assignments)= P(S-1’s
assignments ½ S-3’s assignments).
Input is sent to Bayesian for the Bayesian reasoning and predicting
emotion. This algorithm was applied to predict the emotional state.
Bayesian is responsible for decision making and inferential statistics by
using the probabilities techniques(Gelman et al., 2014, Sivia and Skilling,
1996). Bayes Theorem:
P(h | D) =
•
P( D | h) P(h)
P( D)
(2)
Conditional Probability:
The probability of an event occurring due to the reality that a diverse event
has already occurred is known as conditional probability. Let’s say if
94
the conditional probability of an event H is the probability that the event
will occur given the information that an event F has already occurred.
If events H and F are dependent, then the probability of the intersection of
F and H (the probability that two events occur) is defined by
P(F and H) = P(F)P(H|F). (3)
From this explanation, the conditional probability P(H|F) can easily be
obtained by dividing P (F and H) by P(F):
P( h | F ) =
P( F and H )
P( F )
(4)
F
H
Fig 3.7: Two Conditional Events (Reflecting headache with Flu)
H = “Have a headache”
F = “Coming down with Flu”
P(H) = 1/10
P(F) = 1/40
P(H|F) = ½
“Headaches are rare and flu is rarer, but if you are coming down with flu
there’s a 50-50 chance that you will have a headache.
P(H|F) = Fraction of flu-inflicted World in which you have a headache =
=
#World with Flu and headache Area of " H and F" region P( H ^ F ) (5)
=
=
#World with Flu
Area of " F" region
P( F )
95
The conditional probability of the diverse classes gives the values of
attributes of an unidentified sample. Then the classifier will predict that
the sample belongs to the class having the maximum posterior probability
in that case. Instance is represented by an n-dimensional feature vector
(x1,x2,…,xn). The sample is classified as class from a set of probable
classes C. According to the “maximum a posteriori” (MAP) decision rule:
Table 3.1: “Maximum a Posteriori” (MAP) Decision Rule
Classify(x1,x2,…..xn) = argmax p(C=c)∏,-. pxi|C * c
D: Set of tuples
Each Tuple is an ‘n’ dimensional attribute vector
X : (x1,x2,x3,…. xn)
Let there be ‘m’ Classes : C1,C2,C3…Cm
Classifier predicts X belongs to Class Ci if
P (Ci/X) > P(Cj/X) for 1<= j <= m , j <>i
Maximum Posteriori Hypothesis
P(Ci/X) = P(X/Ci) P(Ci) / P(X)
Maximize P(X/Ci) P(Ci) as P(X) is constant With many attributes,
computing of this is expensive to evaluate P(X/Ci).
Naïve Assumption of “class conditional independence”
P (X |Ci)=∏,. px |Ci
P(X/Ci) = P(x1/Ci) * P(x2/Ci) *…* P(xn/ Ci)
The conditional probability in the above method is obtained by the
estimate of the probability mass function using the training data of that
situation. Moreover, the independent assumption may not be a realistic
model of the probabilities concerned(Rish, 2001).
In this research, the Conditional Probability is preferred. In the designed
logic, different conditions (Very High, High, Normal, Low, and Very
Low) are considered. These conditions are associated with various
96
parameters (GSR, BVP, and Temperature) for extracting the emotions of
human. Various parameters with the different conditions were considered
EVENT 1(A) and Different Emotions (JOY, CALM, and STRESS) were
EVENT 2(B).
Naïve Bayes calculations: To apply the Naïve Bayes algorithm, different
ranges have been considered for each psychophysiological signal
(GSR/BVP/Temperature). Each signal has been further categorized by
various health conditions. These ranges have been confirmed and verified
by national and international doctors. Details of each doctor are given in
the appendix 1.
Table 3.2: Different ranges of the psychophysiological signal
Conditions
GSR
BVP
Temperature
Critical High
>=70 Kohms
>=120BPM
>=103°C
High
>=35 to < 70 Kohms
>=90 to <120BPM
>=99 to <103 °C
Normal
>=25 to < 35 Kohms
>=70 to < 90 BPM
>=98 to < 99 °C
Low
>=15 to < 25 Kohms
>=50 to < 70 BPM
>=96 to <98 °C
Critical Low
<15 kohms
< 50 BPM
< 96 °C
As shown in the above table, there are five conditions applied on three
different signals. So, the possibility table has been designed with respect to
53 with the different possibilities as shown in the annexure 2. Total
Possibilities expected from the above mentioned table are 53=125.
Following is a probability calculation table of each emotion, which is
calculated from the data table mentioned in the Annexure 2.
In this, each value detected is based on the Conditional Probability Naïve
Bayes technique, because the conditional probability of the event A is the
probability that the event will occur given the information that an
event B has already occurred.
97
Table 3.3: Calculations
JOYFULL: P (J) =12/125=0.096
Critical
HIGH
HIGH
NORMAL
LOW
Critical LOW
GSR
P (J|GCH) =.5
P (J|GH) =.5
P (J|GN) =0
P (J|GL) =0
P (J|GCL) =0
BVP
P (J|BCH) =0
P (J|BH) .33
P (J|BN) .33
P (J|BL) .33
P (J|BCL) =0
TEMPERATURE
P (J|TCH) =0
P (J|TH) =0
P (J|TN) =.5
P (J|TL) =.5
P (J|TCL) =0
STRESS : P (S) =6/125=.048
GSR
P (S|GCH) =0
P (S|GH)=0
P (S|GN) =0
P (S|GL)=1
P (S|GCL)=0
BVP
P (S|BCH) =0
P (S|BH) =.33
P (S|BN) =.33
P (S|BL) =.33
P (S|BCL) =0
TEMPERATURE
P (S|TCH) =0
P (S|TH)=0
P (S|TN) =.5
P (S|TL)=.5
P (S|TCL)=0
CALM: P (C) =6 /125=.048
GSR
P (C|GCH) =0
P(C|GH)=0
P(C|GN) =1
P (C|GL)=0
P (C|GCL) =0
BVP
P (C|BCH)=0
P (C|BH) =.33
P (C|BN) =.33
P (C|BL) =.34
P (C|BCL) =0
TEMPERATURE
P (C|TCH) =0
P (C|TH)=0
P (C|TN) =.5
P(C|TL) =.5
P (C|TCL)=0
By taking an example of one case:
If,
X (GSR=25, BVP=75, TEMP=98)
P ( X | J ) = P ( X 1 | J ). P ( X 2 | J ). P ( X 3 | J )
*
/0|./
/0|./
/0|1./1
/0
/0
/0
X
X
P (X | J ) =
(1)
0 × .33 ×12
=0
12
P ( X | S ) = P ( X 1 | S ). P ( X 2 | S ). P ( X 3 | S )
98
*
/2|./
/2
/2|./ /2|1./1
X
X
/2
/2
(2)
(0 × .33 × .5)
=0
6
P (X | S) =
P ( X | C ) = P ( X 1 | C ). P ( X 2 | C ). P ( X 3 | C )
*
/3|./
/3|./
/3|1./1
/3
/3
/3
X
X
P ( X | C) =
(3)
(1 × .33 × .5)
= .16
6
Hence, the maximum value will be considered as detected emotion. So, the
answer is CALM in the above mentioned example case. Naïve Bayes
predicts an output in the form of the probability of emotion with the
maximum
possibility.
This
technique
computes
the
conditional
probabilities of the diverse classes given the values of attributes of an
unknown sample and then the classifier predicts that the sample belongs to
the class having the peak posteriori probability. It should be noted that
these probabilities are dependent on the data collected so far and as we
collect more data from various experiments then the accuracy of the
prediction is likely to improve.
The predicted emotion by Naïve Bayes becomes an input for the next
implemented algorithm, that is, Markov Model. Markov predicts future
emotion based on the current emotional state of a human. For the
compatibility and interfacing between both algorithms, a new algorithm,
HYBRID-NAV-MAR, has been designed and developed by the
researchers.
3.6.3 HYBRID-NAV-MAR
This algorithm was developed to combine Naïve Bayes and Markov model
algorithms for obtaining the meaningful results. The output of Naïve
99
Bayes, that is, an emotional state, is given to another algorithm (Markov
Model) for the future prediction.
3.6.4. Implementation of HYBRID-NAV-MAR
Implementing Markov has been a challenge as the algorithm is used to
model a random system that changes the states according to a transition
rule that only depends on the current state. It undergoes transitions from
one state to another on a state space. The probabilities associated with
various state changes are transition probabilities. These are the most
important aspects of Markov for the future prediction. The transition
probabilities rely only on the current position and not on the manner in
which the position was reached. For example, the transition probabilities
from 10 to 9 and 10 to 11 are both 0.5, and all other transition probabilities
from 10 are 0. These probabilities are independent of whether the system
was previously in 9 or 11.
To have transition probabilities between various states (Stress/Joy/Calm)
with respect to each event (GSR/BVP/Temperature) a HYBRID-NAVMAR algorithm has been designed and implemented, within a
microcontroller, by the researchers. This has added more intelligence to
the system. Following is the designed transition probability algorithm,
which has helped the system to work in the hybrid form and has combined
both Naïve bayes and Markov model.
Algorithm 3.2: HYBRID-NAV-MAR
100
Step 1: Accept data for occurrences of each respective event in metrics GSR[3,5],
BVP[3,5] , TEMP[3,5]
Step 2: for i= 0 to 2
for j =0 to 4
emot_mat[i][j] = GSR[i,j] +BVP[i,j] + TEMP[i,j]
endfor
endfor
Step 3: Compute the probability transition
transition from emot_mat for each state.
Fig3.8: Emotion States diagram with transition probability
The emotions can be seen as a state of crisis that is preceded by arousal
due to an external stimulus. Something that tends to create distractions can
be considered as an external stimulus. After the stress factor (the stressor)
disappears, the body relaxes,
relaxes, gets calm, and returns to a normal condition.
The change between the two states can be sudden or incremental;
typically, arousal is more rapid and relaxation takes considerably longer.
Various emotions are categorized and are based upon the degree of arousal
a
from low to high and valence, that is, positive to negative emotions. Due
to the variation in the emotions, the above mentioned algorithm has three
matrices, which have been designed for each event with respect to three
different emotional conditions
condition (Normal/Low/high).
101
3.6.5 Markov Model
Markov model(Schuller et al., 2003) is based on the Markov chain that
models the state of a system with a random variable that changes from
time to time. The Markov property suggests that the distribution of current
variable depends only on the distribution of the previous (past) state. In
Markov, the changes in the states are called transitions. The probability is
associated with each state that changes and are called transition
probabilities.
The process starts in one of the above shown Fig3.7 states and moves
successively from one state to another. Each move is known as a step. If
the chain is presently in the state Si, it moves to the next state Sj with a
probability denoted by Pij. This probability does not depend on the
previous states of the chain. The probabilities p ij are generally known as
transition probabilities.
Markov chain collects the random variables Xt, where the index t runs
through 0, 1, ...n having the property that gives the present; based on that
the future is predicted. In this, the future is conditionally independent of
the past. In other words,
P (Xt-j|X0=i0, X1=i1……Xt-1) =P (Xt=j|Xt-1=it-1)
(7)
Example: If person is in stress and have to check for other emotions
then:
Stress to Calm Probability will be:
P (Tn|Tn-1,Tn-2,…..,T1) (1)
T=time;
=P (T3=calm|T2=Stress,T1=Stress)*P(T2=Stress|T1=Stress)
=P (T3=calm|T2=Stress)*P(T2=Stress|T1=Stress)
=.14*.18
102
=.0252
=.0252*100
=2.52%
Stress to Joy Probability will be:
P (Tn|Tn-1,Tn-2,…..,T1)
T=time;
=P (T3=Joy|T2=Stress,T1=Stress)*P(T2=Stress|T1=Stress)
=P (T3=Joy|T2=Stress)*P(T2=Stress|T1=Stress)
=.10*.18
=.018
=.018*100
=1.8%
Probability of calm is high, which means that the next emotion can be
Calm based on the probabilities of the data collected. One can have a
threshold value of the probability above which a state is predicted to be
highly likely.
3.7 Comparative study
Indeed, the Table 3.3 at the end of this section identifies many
chronologically ordered studies that measure physiological signals and
analyze these signals by the recognition algorithms. There is budding
evidence indeed that emotional states have their corresponding specific
physiological signals that can be mapped correspondingly. Most of the
researches follow the methodology given below:
(i) Analyze different body signal(s) (e.g., skin conductance, heart rate)
103
(ii) Use different emotion elicitation method(s) (e.g., mental imagery,
movie clips)
(iii)Work with varying number of subjects
(iv) Classify emotions according to the different method(s) of analysis
(v) Show the different results for various emotions.
Table3.4: Comparative study
Authors
Emotion
elicitation
method
Subjects
Signals
measured
Data
analysis
technique
Result
(Lisetti and
Nasoz,
2004)
Difficult
task solving
58
undergraduate
students of an
introductory
psychology
course
Cardiovascular
activity (heart
rate and blood
pressure)
ANOVA
and
ANCOVA
Systolic and
diastolic
blood
pressure
responses
were greater
in the difficult
standard
condition than
in the easy
standard
condition for
the subjects
who received
the highability
feedback.
(Gläscher
and
Adolphs,
2003)
Difficult
problem
solving
32 university
undergraduates
(16 males and
16 females)
Skin
conductance,
objective task
performance,
self-report
MANOVA
Skin
correlation/ conductance
regression
amplified at
analyses, the start of the
ANOVA
trial, but
decreased by
the end of the
trial for the
hardest
condition.
(Collet et
Neutrally
and
30 people (16
females and 14
Skin
conductance,
104
Friedman
variance
Electrodermal
responses
al., 1997)
expressively
loaded
slides
(pictures)
and elicited
Happiness,
surprise,
anger, fear,
sadness, and
disgust
males)
skin potential,
skin
resistance,
skin
temperature,
skin blood
flow, and
instantaneous
respiratory
frequency
analysis
distinguished
13 emotion
pairs out of
15. Skin
resistance and
skin
conductance
perturbation
duration
indices
separated 10
emotion pairs.
Though,
conductance
amplitude
could
distinguish 7
emotion pairs
(Tarvainen,
2004)
11 auditory
stimuli
mixed with
some
standard
and target
sounds
elicited with
emotion i.e.
Surprise
20 healthy
controls (as a
control group)
and 13
psychotic
patients
GSR
Principal
component
analysis
clustered
by centroid
method
78% for all,
but 100% for
the patients
(Vyzas and
Picard,
1998)
Personal
imagery
(Happiness,
sadness,
irritation,
fright,
disgust,
surprise,
neutrality,
platonic
love,
romantic
love)
A healthy
graduate
student with
two years of
acting
experience
GSR, heart
rate, EEG, and
respiration
Sequential
floating
forward
search
(SFFS),
Fisher
Projection
(FP) and
hybrid (FP
and SFFS)
81% for
hybrid SFFS
and Fisher
method with
40 features
and 54% rate
with 24
features
(Nasoz et
al., 2004)
A sluggish
computer
37
undergraduate
Skin resistance
and blood
HM
models
prototype
recognition
105
game
interface
(Frustration)
and graduate
students
volume
pressure
worked
significantly
better than
random
guessing
while
discriminating
between
regimes of
possible
frustration
from regimes
of much less
likely
frustration.
3.9 Conclusion
The development of affective sensing and monitoring system through
various machine learning algorithms with their methodology is discussed
in this chapter. The processing and feature extraction methods applied to
the PD, GSR and BVP signals were outlined in the corresponding sections.
106
Chapter 4
EMPIRICAL STUDY AND ANALYSIS
The research describes that the development of an emotion detection
approach is based on the automatic monitoring of physiological signals
using a microcontroller. There are three main aspects of this study: (a)
experimentation setup for the physiological sensing, (b) signal processing
to sense the affective state, and (c) affective computing using the machine
learning algorithms. This chapter focuses on the empirical study of this
research. The physiological signals were concurrently recorded and
coordinated by the hardware and software combination throughout the
whole experimentation to analyze the potential concurrent changes that
occurred due to the sympathetic activation of aroused emotion. The goal of
this chapter is to define the experimental setups for the data collection,
which will be further used for emotion prediction.
There are number of patients exhibiting autonomic disorders. All
autonomic tests include their physiological background, indications,
contra-indications, the entry conditions that must be fulfilled before the
subject is allowed to take the test, the instrumentation, the activities flow
performed during the test, and the exceptions which might cause test.
4.1 Generalities
A setup and a corresponding protocol were defined and implemented
while performing the experiments. Those protocols are:
•
Provide an appropriate stimulus, capable of eliciting stress in the
subjects participating in the experiment.
•
Provide appropriate variation in each output data.
•
Provide proper coordination of all the software and hardware
components that are involved in the experimental process.
107
•
Record the GSR, BVP, and temperature signals with all the necessary
time markers.
•
This design is not suitable for the people having a disease called
hyperhidrosis(Ogorevc et al., 2013), which causes excessive sweating.
It is a drawback of the system.
The complete implementation of the system for experimental with the
coordinated software and hardware components are described in the
Chapter 2.For validation and data collection, three sets of different
experiments were conducted with totally different scenarios by using same
strategy and protocol.
Scenario _1 (S0_1):- In the first scenario, the experiments were done in a
multinational company for an improvement in the daily activities of the
staff of the company. The company’s interest was to target the weak
performers. After the discussion, permission was granted by the company
for the betterment of the employees. This helped the staff to work on their
emotional aspect.
Scenario _2 (S0_2):- In the second setup, the experiments were done by a
doctor on the hundred odd patients (subject) in a hospital; each subject
having different age, gender, and medical background. The experiment
was also included the paralyzed people to understand that how everybody
cannot express the emotions. This helped the doctor to see an exact mind
state of a patient for the better treatment.
Scenario _3 (S0_3):-In this scenario, a set of audio/video clips was
successfully used as stimuli, in the real-time. Different audio or video
songs of different languages were played and then even with the choice of
the subject. The songs experimentally were found to be triggering-off the
specific emotion. Various subjects were asked to listen to the clips and
subjectively feedback was measured for detecting the emotion arousal.
108
This work was to design and develop a real-time monitoring system that
can be used to estimate different emotions, especially for the people who
cannot express their emotions, such as the people suffering from a
paralyzed body. The expected values of the different biofeedback
modalities are mapping with different ranges of emotion areas mentioned
in the Chapter 3. Different emotional expressions produce different
changes in the autonomic activity; following are the examples of various
activities:
Table 4.1: Change in Autonomic Activity(Ekman et al., 1983, Kreibig, 2010)
Emotions
GSR
BVP
Temperature
Anger
Decreases
Increased
Increases
Fear
Increases
Increased
Decreases
Happiness
No Change
Normal
No Change
Stress can be seen as a state of crisis that is preceded by arousal due to an
external stimulus. An external stimulus can be considered something that
tends to create distractions. Once the factor causing stress (the stressor)
disappears, the body gets relaxed and calm; and then returns to a normal
state.
This considers a simplified setting by assuming that the person is either in
the normal state or in a stressed state. The change between the two states
can be sudden or incremental; typically, arousal is more rapid and
relaxation takes considerably longer.(Fontaine et al., 2007) We can see
that the various emotions are categorized; the emotions are based upon the
degree of arousal from low to high and valence i.e. positive to negative of
emotions are shown in Fig 4.1(Schmidt and Trainor, 2001). All the
features were selected from the training data which was extracted from
real-time experimental data set.
109
Arousal
Sadness
GSR
Joyful
Stress
Neutral
Calm
Valence
BVP
Fig. 4.1: Two-dimensional emotion models with four quadrants
The record was collected based on the experiments consisting extraction of
GSR, BVP, and temperature signals and stored within the microcontroller.
According to emotion model(Ohme et al., 2009)following are expected
outcomes from various activities of experiments:
• When stress will increases then at same time the skin conductivity GSR
will decrease and HR/BVP will increase
• When joyfulness is decreased then the skin conductivity GSR will
increase and HR/BVP will (increase/decrease)
• When calmness, there will be then no change
The realistic interest of these experiments was to predict the state for
statistics collection (samples). This was done to have those samples
available for the testing that were never presented to the system during the
training phase. This data was collected and analyzed in the controlled
settings with the designed hardware and with appropriate algorithms
embedded in the microcontroller. Data sets of all experiments are given in
the attached Appendix 3.
110
4.2 Benchmark construction of experimentation
4.2.1 Experimental Study_ S0_1
Aim:
The experiment was to determine the change in the emotional level of a
subject responding to a task given by a company and to a series of
questions with emotional content.
Participants and stimuli:
An experimental setup was established at the sitting place of the subject,
where the daily tasks were performed. As mentioned in the procedure, the
readings were taken at the time when the subject was performing the
assigned task (office work). Both the genders, twenty odd male and female
subjects ranging from the age 20 to 55, were considered. Initially, the
subjects under neutral conditions were measured; that served as the
reference for us to estimate the variance of the values in the different
emotional states.
Once the state was reached, the subject tended to be in that state for a
finite amount of time. The total time of stimulus for each emotion was
between 2-3 minutes and with a gap of 2 minutes between different
emotions. So, to fulfill the criteria, emotion was estimated thrice.
Few sample questions with different emotional content are given below:
•
How long you have worked as a subordinate?
•
How do you rank your overall job profile?
•
Are you satisfied with your pay package?
•
Does your job profile justify your hard work?
•
Are you satisfied with the services provided by the company?
•
Do you think that you deserve more in life?
111
Procedure:
1.
Selected subjects, by the company itself, should be asked to go for
hand wash with soap and water, and then get the hands dried
properly.
2.
Subject should be healthy (that is, no fever etc.).
3.
Subject should be without any alcohol intake.
4.
The GSR,BVP, and temperature sensors should be attached to the
distal finger segment of two non-adjacent fingers
5.
The subject should sit comfortably without any external stimuli
disturbance.
6.
During the experiment, subject should not be allowed to have
water, as it can change the emotion.
7. Two different measurements should be taken in this experiment: (a)
while regular daily task, and (b) while carrier satisfaction
interview.
8. Readings should be taken three times: (a) before the task, (b)
during the task, and (c) after the task. Average of these three values
would be considered as actual value.
9. Different emotions should be detected, as it would also help in
professional growth (by building strong emotions)
4.2.2 Experiment_ S0_2
Aim:
Skin conductance orienting response (SCOR) in childhood, habituation is
absent at age 3 but apparent at age 4 and increases thereafter to peak at age
6 and then levels off.(Gao et al., 2007, Kylliäinen and Hietanen, 2006).
112
This experiment was designed for all age group above 6 and to determine
the change in the emotional level of a subject while responding to the
doctors for the questions with the emotional content.
Participants and stimuli:
An experiment setup was established according to the comfort-level of a
doctor. Total ten questions, five with neutral content and five of an
emotional nature, were asked from the subject. The doctor instructed the
subject to sit quietly and answer each question honestly in one word. The
subject was instructed not to give explanation on any answer. Questions
were asked according to the age factor. Once the state was reached, the
subject tended to be in that state for a finite amount of time. The total time
of stimulus for each emotion was between 2-3 minutes. Experiment was
carried for 15 days on different subjects and sometimes subjects were also
intentionally repeated for the better judgment.
Table 4.2: Questions with Emotional Content
Age 6 to 12
Age 12 to 35
Age above 35
Does being alone at night
frighten you?
Are you in love?
Do you ever cry?
Has anyone ever beaten
you?
Do you ever cry?
Do you recall your young
days?
Are you scared of Ghosts?
Do you feel there is someone
who understands you?
Have you ever seen a tragic
accident?
Do you feel scared during
the exams days?
Do you have any best friend?
Are you satisfied with your
achievements in the life?
How do you handle the
exam pressure?
Are you satisfied with your
career?
Whom you miss the most in
your life and why?
Table 4.3: Questions with Neutral Content
Age 6 to 12
Age 12 to 35
Age above 35
Do you like burger?
Is it Monday today?
Do you have a car?
Do you like watching TV?
Do you like holidays?
Do you have a House?
Which day is today?
What is your hobby?
Do you have kids?
113
Which is your favorite Which sport does you like the Are you a foodie?
game?
most?
Do you like coloring?
Who is your best friend?
Which
dish?
is
your
favorite
Procedure
1. The subjects coming for the daily checkup should be asked to go for the
hand wash with soap and water, and the get the hands dried properly.
2. Subject’s health should not be critical (e.g. fever etc.)
3. Subject should be ready without any alcohol intake.
4. The GSR, BVP, and temperature sensors should be attached to the surface
of the distal finger segment of two non-adjacent fingers.
5. The subject should sit comfortably without any external stimuli
disturbance.
6. During the experiment, subject should not be allowed to have water, as it
can change the emotion.
7. Two different measurements should be performed in this experiment: (a)
during the regular daily task, and (b) during the carrier satisfaction
interview.
8. Different emotions should be detected, as it would also help doctor for the
better understanding.
4.2.3 Experiment_ S0_3
Aim
The experiment was performed to determine that how the audio/video
clips may result in a high subject agreement in terms of the elicited
emotions (that is, sadness, anger, surprise, fear, and amusement). Twentyone movies, in three groups, were played for the participants. Each group
of seven clips was meant to extract different emotion (Stress, Joyful, and
Calmness).
114
Participants and stimuli
An experiment was done on 20 undergraduate/graduate students from
different streams: electronics & communication, computer science, and
civil. The subjects participated in the study all mutually. The subjects were
informed that after the experiment they had to fill out a questionnaire
where they had to answer the demographic items. Then the subjects were
informed that they would be watching various movie clips geared to elicit
emotions and during each clip, they would be prompted to answer the
questions about the emotions that they experienced while watching the
scene. They were also asked to respond according to the emotions they
experienced. A slide show played the various clippings and, after each one
of the clips, a slide was presented asking the participants to answer the
survey items for the previous scene. During the above measurement, the
subject was advised to abstain from all physical work, and needed to
concentrate on listening to the clips. The total time of stimulus for each
emotion was between 4 to 5 minutes, with minimum gap of 1 minute
between different stimuli, during which the music was put off and the
subject was advised to come to normal, sip water, munch on a snack etc.
For each scene, four questions were asked. The questions are:
•
Which emotion did you experience from this video clip?
•
How would you rate, on a five point scale, the intensity of the
sentiment that you experienced?
•
Whether you experienced any other emotion at the same intensity or
advanced, and if so, specify what that feeling was?
•
Have you seen that clip before?
Procedure
1. The subjects, who volunteered for the experiment, were asked to go for
the hand wash with soap and water, and get the hands dried properly.
115
2. Subject should be healthy (that is, no fever etc.).
3. Subject should be ready without any alcohol intake.
4. The GSR, BVP, and temperature sensors should be attached to the
surface of the distal finger segment of two non-adjacent fingers.
5. The subject should sit comfortably without any external stimuli
disturbance.
6. The readings should be taken three times: (a) before the task, (b)
during the task, and (c) after the task. Average of these three values
would be considered as actual value.
7. Different emotions should be detected, as it would help in gathering
the accurate training data.
4.3 DATA ANALYSIS
Data Analysis is the process of reducing/filtering the large amounts of
collected data in a way so that the data makes sense. To do this, the
hardware was designed and developed with a capability to do the data
analysis and data storage within the hardware. The following fig 4.2
represents the general structure of the proposed system.
Emotion
Induction
Measuring physiological
variables
Emotion
Estimation
Subject Assessment and
Data Analysis
Fig. 4.2: Data analysis and subject assessment for emotion estimation
116
4.3.1 Data Acquisition
The information was gathered based on the above mentioned experiments.
The sensors were attached to the fingers of the individuals to
simultaneously acquire the BVP, GSR, and Temperature signals by means
of a recording mechanism. The purpose of these experiments was to focus
on both main stressing tasks, namely Talk Preparation (TP) and
Hyperventilation (HV). Each experiment was divided into four steps,
which are described in the subsequent subsections:
1) First step (FS_1) consisted of attaching sensors to the persons, and after
a variable period of time when the subject was asked to calm down, an
acquirement was performed according to the procedure mentioned
above.
2) Hyperventilation (HV): later, the person was required to breathe
intensely and speedy for every 2-3 seconds, indicated by the
experimenter. This task was performed until the subject evidently
perceived the changes in his/her corporal sensations. It was in this
moment
exactly
when
GSR/BVP/Temperature
was
sampled,
representing an obvious behaviour of physiological signals under a
tensing situation.
3) Talk Preparation (TP): After HV, the subject was asked to take a break
and then was asked to prepare the answers to the questions mentioned
in the above experiments. The subject was given one or two minutes to
prepare for the answers; signals were sampled again during a period of
90 seconds, representing a stressing situation.
4) In the final step (FS_2), the experimentation comes to an end by
acquiring the emotions from the subject. It is significant to state that for
the sake of independence in the order of the tasks.
4.3.2 Normalization and feature extraction
117
The procedures described above resulted in a set of physiological records
(total 160 physiological records). The differences among the number of
data sets for each emotion class are due to the data loss for the data of
some participants during various segments of the experiment. In order to
compute the number of variations in the physiological responses, the data
was normalized for every emotion, as the participants went from a calm
state to the state of experiencing a specific emotion. Normalization is also
important for minimizing the individual differences among participants in
terms of their physiological responses while experiencing a specific
emotion. The composed data was normalized by using the average value
of the corresponding information type gathered during the relaxation
period for the same participant. An example of normalization for the GSR
values is as follows:
Normalized Data = raw_data – raw_relaxation_data
Raw_relaxation data
(1)
After the data signals were normalized, features were extracted from the
normalized data. Four features were extracted for each data signal type:
maximum, minimum, mean, and variance of the normalized data. The
information was stored in a three dimensional array of real numbers:
1
The subjects who participated in the experiment
2
The emotion classes (stress, joyfulness, and calmness)
3
Extracted features of statistics signal types (minimum; maximum;
mean; and var iance of GSR, temperature, and BVP).
Every slot of the array consists of one exact feature of a precise data signal
type, belonging to one exact participant while s/he was experiencing one
precise emotion. (e.g., a slot carries the mean of normalized skin
temperature assessment of, say, the participant number 1 while s/he was
experiencing tension, whereas, another slot, for example, contains the
variance of normalized value of the participant number 5 while s/he was
experiencing calmness). As mentioned, features were extracted for each
118
data type and then supervised learning algorithm was implemented that
took these features as input and interpreted them for final prediction.
4.3.3 Classification Methods
Classifiers are compared on the experimental data. The Naïve Bayes
classifiers are trained and tested on the individual and multiple subjects.
Later than all the features were extracted, these were provided as
contribution to the learning systems, which were trained to differentiate
the tension state. The training data has been classified into two different
sets in order to evaluate that how activity information may influence the
results of a stress inference. One set of training data includes only the
GSR/BVP/Temperature related features, while the second set also includes
the accelerometer information. We also evaluated the classification
performance for the between-subjects datasets and within-subject datasets.
A cross-validation analysis was applied on the resulting models. The entire
dataset was used to generate several types of the physiological response
models. These models included the models of changes to all
GSR/BVP/Temperature response. For a cross-validation, the original
sample is randomly partitioned into k equal size sub-samples; of
these k sub-samples, a single sub-sample is retained as the validation data
for testing the model, and the remaining (k – 1) sub-samples are used as
the training data. (Abu-Nimeh et al., 2007)The cross-validation process is
then repeated k times (the folds), with each of the k sub-samples used
exactly once as the validation data. The k results from the folds can then
be averaged (or otherwise combined) to produce a single estimation.
According to the cross validation strategy, the original data is first divided
into 10 equal subsets. Sequentially, one subset is tested using the classifier
trained on the remaining subsets. This process is repeated until every
instance has been used exactly once for testing. The overall success rate
for a classifier is then evaluated as the number of correct classifications
divided by the total number of feature sets tested:
119
45567859:8;< *
3 =->-=-,
= ,?@ = A
(2)
Considered mean, minimum, maximum, and standard deviation of skin
conductance and peak height; the total number; and the cumulative
amplitude, rising time, and energy of startle responses in a segment. These
features were initiated useful in the earlier studies. The Naïve Bayes
classifiers are based on the probability models that integrate class
conditional assumptions (Quattoni et al., 2004) We basically estimate the
probabilities that an object from each class will fall in every cell of the
discrete variables (every probable discrete value of the vector variable X),
and then we employ Bayes theorem to create a classification. This
technique computes the conditional probabilities of the diverse classes
given the values of attributes of an unidentified sample and then the
classifier will calculate that the sample belongs to the class having the
maximum posterior probability. If an instance is represented by an ndimensional feature vector, (x1, x2,…, xn), a sample is classified to a class
c from a set of probable classes C according to the highest posteriori
(MAP) decision rule, mentioned in chapter 3.
Classify (a1, a2,…..an) = argmax p(C=c)∏,-. pxi|C * c (3)
The conditional probability in the above calibration is obtained from the
estimates of the possibility mass function using the training data. Even
though the self-determination assumption may not be a practical model of
the probabilities involved, it may still permit relatively correct
classification performance.
4.3.4 Observations
In this section, the results from all three experiments are discussed. The
situations and emotions where there occurs a great arousal, such as horror
and melancholy were easy to identify, whereas the lower arousal emotions,
such as joy and sadness were meagerly distinguishable. The present work is
an attempt to such an end and hopes to find out the methods and ways to
120
achieve the goal of affective communication. This experiment has a
drawback that it is not based on the natural / real emotional states, but the
induced emotions are being observed and analyzed. The other factor of
importance is the emotional responses that are purely dependent upon the
regulation capability of the individual. The signals from the experimental
subjects were gathered and diverse features were extracted. The prediction
performance was evaluated using 10-fold cross validation: 10 samples were
pulled out as the test samples, and the residual samples were used to train
the classifiers. The objective was to develop and train a system that accepts
the various physiological variables as input and predicts the participant’s
affective state. Few examples of the statistics variation are shown below:
GSR Variation
Fig. 4.3: Variation in GSR
121
BVP Variations
Fig. 4.4: Variation in Blood volume Pulse (BVP)
Temperature Variation
Fig. 4.5: Variation in Temperature
122
4.4 CONCLUSIONS
The results from the experiments illustrate a promising correlation among
the emotional tension and the monitored physiological signals. The tests
performed with the classifiers have recognized the user emotional states on
the basis of the features extracted from the physiological signals. These
results have exposed that, below the controlled conditions, the
simultaneous
monitoring
and
simultaneous
processing
of
three
physiological signals: BVP, GSR, and ST are complete success. This work
corresponds to the data collected in the controlled laboratory settings.
However, the controlled setting in a laboratory is not suitable for mobile
emotion monitoring, because the physical activity affects the measured
physiological
signals.
The
automated
induction
of
an
accurate
physiological response was followed by the prediction models. It is
interesting to know that for predicting all three parameters the accuracy
levels were surprisingly high. The physiological responses follow directly
from the changes in affect and thus can be used as the key predictors of an
affective state. Although biofeedback devices can be used to obtain actual
physiological signals, it may be impractical to require the users to
biofeedback equipment and then deploy an additional hardware with the
applications.
123
Chapter 5
POWER EFICIENCY AND SYSTEM COST
Cost-effectiveness includes the data concerning the reduction in physician
visits and/or medication use, decrease in the medical care cost, reduction
in the hospital stays, reduction in mortality, and enhanced quality of life.
The evidence suggests that the multi component behavioural treatments
are cost-effective in all dimensions(Schafer et al., 2011). The hardware
and software structural design (communication protocols, power
organization policies, and application-level control) has been tuned to
optimize the price, battery autonomy, and real-time performance that is
required for this function. This chapter talks about the factors that show
that how this study has given benefits in terms of cost and power.
5.1 Introduction
The most expensive investment to set up a biofeedback system is the
equipment (Brunelli et al., 2006, Sugar et al., 2007). The price of the
apparatus can dwarf the price of all the training and mentoring you need to
go through. The time-commitment needed to essentially get a device
working can be overpowering. Most of the people do not enter this field
since they see the implausible cost of the multi-channel equipment with
tons of whistles and bells and then realize that its use is simply
unintelligible. It is important to know how to assess what equipment is
suitable for intended use. To have a cost-effective product, one wants to
pay very less for software or hardware with other capabilities. To have
good physiological systems, person need to record the physiological
activities to treat diverse disorders and then to recover the functioning of
the healthy clients; this is probable when you take a suitable biofeedback
course with very low power consumption and less cost.
As before stated, there is no need to buy a very expensive machine capable
of recording many more channels than you will ever use – as long as you
are certain you won’t enlarge your use of biofeedback techniques and
124
types of patients. In general, if the quality of the hardware and software is
kept equal with more channels, the cost and energy effectiveness will be
immense.
5.2 Energy efficiency
Energy efficiency is an approach of managing and restraining the growth
in energy consumption or of working on low power. Something is more
energy efficient if it delivers more services for the same energy input, or
the same services for the less energy input.
Power management has become a very important research area and
various approaches have been anticipated. Power efficiency has become a
crucial issue in the current computing systems. For the mobiles and other
portable devices, battery is one dominant constraint with a fixed energy
budget. For the high-performance servers, the ever-increasing power
consumption brings in not only the tremendous difficulty with high cost of
building and operating the cooling system, but also the reliability
concerns. Present work focuses on the performance of the low standby
power circuits. The use of the applications with the embedded devices is
increasing day by day. The users want more functionality in smaller size
and with longer battery life. The developments in the area of technology
for batteries could not kept pace with those in the processing power and
storage.
One of the important factors is power dissipation. It is a compute of the
rate at which the power is lost from an electrical method. When an electric
current works on a performer, the internal energy of that conductor
increases which causes its temperature to increase above the ambient
(surrounding) temperature. This additional causes the power to dissipate
away from the conductor into the surroundings during the process of heat
transfer. The rate of this heat shift (joules per second) is known as 'power
dissipation' (in watts).
The Microcontroller used in this research work is TI’s MSP430 family of
ultra-low-power MCUs that consist of several devices featuring different
sets of peripherals targeted for the various applications. The architecture,
125
combined with the several low-power modes is optimized to achieve an
extended battery life in the portable measurement applications. There are
two ways for making the system energy efficient; these ways are discussed
next:
Static Power Management: This management deals with regulating the
power consumption in the inactive periods while preserving the states of
OS & applications according to the pre-defined policies. It requires user
interaction to reactivate the system, for example, sleep hibernation and
suspend. In this research, the concept of the static power management is
applied and explained in the section below.
Dynamic Power Management:
It refers to implementing the power
management schemes while programs are running. It manages the power
of the peripheral devices through the device drivers and operating system.
5.2.3 Proposed Low-Power model
This section discusses few parameters that were taken care to increase the
energy-efficiency. Those parameters are:
• Using the Sleep/awake algorithm
• Using a high resolution ADC
After doing certain test, it was observed that the analog to digital
conversion saturates at 2.35V.MSP430 is having ADC of 16 bits so the
resolution is:
2.35 E
* .035HE
65536
ADC in used microcontroller has a resolution of .035mV and the
minimum tension is 136mV, with this it can be analyzed that there is no
need of operational amplifier. By removing operational amplifier has
given an advantage to system by consuming less power.
126
This concept helped in achieving an energy efficient objective. Moreover,
algorithm 5.1 (Sleep/awake algorithm) is designed to achieve more power
efficiency. The algorithm is described below:
Algorithm 5.1: Purposed and applied Sleep/awake algorithm
STEP 1: Fetching Input
STEP 2: Input for conversion to digital form
STEP 3: Waiting for data sample to be converted
STEP 4: Data Analysis
STEP 5: Calculations of results
STEP 6: Send to PC for final data
STEP 7: Repeat from the step 1. By default, it will be in low
power mode; converts only when demanded or a signal is sent
from PC to microcontroller.
STEP 8: Wait for a signal from PC to start the process
If yes
Conventional process will be followed
Else
Check for signal / Step 1 will be followed.
5.2.4 Power Analysis
The power consumption in the current design will start with the memory.
The current drawn by the MCU depends on the mode of its operations.
The number of peripherals used also effects the current consumption. If we
enable more peripherals for our use then the current consumption will
increase. For MSP430F2013 microcontroller, the typical current
127
consumption is 220uA at 2.2V in the active mode (by assuming that the
MCU peripherals are operating at their maximum limitation). The
maximum load current on the reference output voltage buffer of
MSP430F2013 is equal to 1mA at 3.6V; that makes the power
consumption to 3.6mW.There are some losses in the circuitry of the
MSP430F2013 microcontroller as well, that is, the Standby Mode (0.5 µA)
with power consumption of .013mW.Static power consumption is the
product of the device leakage current and the supply voltage. This static
power consumption is defined as quiescent, or PS, and can be calculated
by following equation:
PS =VCC x ICC
Where:
VCC = supply voltage
ICC = current into a device
Icc, Active Current/Operation = 17mA.
Vcc, Operating Voltage = 3.6V
Power Consumption during programming the memory,
(In active Mode) PS = 3.6 x17= 61.2mW
(1)
(Standby mode) PS = 3.6 x 0.013 = .46 mW
(2)
Sleep/Awake algorithm is implemented in Standby mode by using
5.3 Expenditure Effectiveness
In a normal biofeedback session, electrodes attached to the skin send
information to a small monitoring box that translates the data into a tone
varying in the pitch, a visual meter varying in the brightness, or a
computer screen that shows lines moving across a grid. The sensors are the
leads and the signal transformers that attach to the patient are classically
specialized for simply a few types of signals. They can increase the price
of a machine by several hundreds of dollars – possibly thousands. The
128
same goes for the multiple channels.
channels. Few examples of the similar systems
are given below:
5.3.1 Biofeedback Machines
ProComp5 Infiniti(Andziulis
Infiniti
et al., 2009) is housed in an ergonomicallyergonomically
designed case and requires only a USB port to attach to any IBM
compatible PC. ProComp5 Infiniti™ has the identical inputs as ProComp
Infiniti, but omits the last 3 channels. The first two sensor channels
provide an final signal
signal fidelity (2048 samples per second) for presentation
the raw EEG, SEMG, EKG, and HR/BVP signals, while the residual 3
channels sample the data at the rate of 256 samples per second for the
slower signals, such as respiration, temperature, and force. Not only can
ProComp5 Infiniti™ capture information in the real time by connecting
straight to the PC via its fiber-optic
fiber optic cable, but it can also store the
information on a Compact Flash memory card for uploading afterwards to
the PC; or it can also use optional
optiona long-range
range compact flash module
telemetry anytime it is desirable. Unlike the competitive equipment, all
channels can be used with any grouping of sensors, including EEG, EKG,
RMS SEMG, Skin Conductance, Heart Rate, Blood Volume Pulse,
Respiration, Goniometer,
Goniometer, Force, Inclinometer, and Torsiometer.
Fig. 5.1: ProComp5 Infiniti System T7525M
ProComp5 Infiniti System:
•
TT-USB
USB interface unit
129
•
Fiber-optic cables (1’ and 15’)
•
Four alkaline “AA” batteries
•
Sleek fabric storage and carrying case
The price of the invention is $2964 (177840/-INR).
5.3.2 GSR2/Temp Biofeedback System
The GSR2/Temp(Yazdani et al., 2012) is the same unit as the GSR2. It
monitors the Galvanic Skin Response from the finger tips to monitor the
changes in the autonomic nervous system that reflect tension or relaxation.
In addition to the GSR2 functionality, the GSR2/Temp also includes a
Thermistor to monitor the finger temperature and a meter to display the
changes in the temperature. The GSR2/Temp actually has two biofeedback
units in one with both of these capabilities. It also includes a hands-free
GSR finger sensor.
Fig. 5.2: GSR/Temp 2x
Biofeedback with the GSR2/Temp gives you this unique dual monitoring
capability to help you take your relaxation response deeper. Knowing your
physical reaction helps you in changing your responses. The GSR2/Temp
measures the ups and downs in the autonomic nervous system activity by
measuring the subtle changes in the moisture level of your fingers; the
130
often discussed Galvanic Skin Response. The cost of the product is Rs
16,433 /5.3.3
ECG MACHINES
A portable ECG machine measures the activity of the heart. The heart rate
can be measured by finding the interval between the R-R impulses.
Following are the examples of some of the portable ECG machines
available in the market.
Holter Monitor
A Holter monitor(Hanke et al., 2009) is a small portable ECG device
named after its inventor, the American biophysicist Norman Holter. The
Holter monitor is broadly used to measure the heart rate. The device
records the electrical activity of the heart, typically over a 24-hour period,
while the patient keeps a diary recording the activities and any symptoms
felt. The ECG recording is then analyzed, and irregular heart activity is
correlated with the patient’s record of the activities and symptoms.
Fig.5.3: Holter Monitor
Though widely used, these units have some drawbacks. These units do not
have the facility of providing the information about the variations in the
heart rate over a span of time. The Holter monitors do not provide any
information/feedback to the user when the vital sign preset limits are
131
exceeded. Moreover, these units are expensive, cumbersome, and cannot
be used for the long-term monitoring.
The cost of the Holter monitors ranges from Rs.50,000 to Rs.2 lakh
typically.
The Digital Heart Rate Monitor
The Digital Heart Rate Monitor (Hanke et al., 2009), manufactured by
Electronic Engineering Corporation, is a cost-effective device that uses
three ECG electrodes to monitor the cardiac rate. The heart rate is
determined by the principle of electrocardiography. The device can
measure the heart rate from 20bpm to 300bpm.
Figure 5.4: The Digital HRM
Features:
• Detection Principle: Electrocardiography
• Electronic circuitry: Digital Integrated circuit
• Heart Rate Range: 20 to 300bpm.
• Power: 220V/50Hz mains supply
• The Digital Heart Rate Monitor does not indicate the variations in the
heart rate
The cost is the digital HRM is Rs.14,000/-.
132
5.3.4 Multiparameter Bedside Monitor (MPM 5533)
MPM 5533 is a multi-parameter monitor manufactured by BPL(Henneman
et al., 2006). The MPM 5533 measures the heart rate by both the
electrocardiography and photoplethysmography technique. The measuring
range of the device is 30bpm to 250bpm.
Features
•
Detection Principle: Electrocardiography and Photoplethysmography
•
Adult to Neonatal usage
•
Large 10.4" colour TFT display
•
Non-fading solid traces of graphical wave forms on the 320 x 240
graphic LCD screen
Fig. 5.5: Bedside Monitor MPM 5533
5.3.5
•
Heart rate range: 30bpm to 250bpm
•
Visual alarm indicator
•
Graphic & Tabular Trend Information
•
Network Connection for Central Monitoring
•
Power Consumption: <18VA
•
Optional Built-in Strip Chart Recorder
•
Cost: Rs.5 lakh typically.
Cardiomon CCM900
133
Cardiomon CCM900 (NAIDU et al.), is a portable ECG monitor
developed
by L&T, India.
Features
•
Low cost simple ECG monitor
•
Uses 3 lead ECG electrodes
•
Status screen display with auto-set of all alarms
•
Trending of heart rate
•
Cost : Rs.87,900
Fig. 5.6: Cardiomon CCM900
5.3.6
BioView HRV Monitor
The Bioview model monitor (Heinze et al., 2012) for HRV is designed for
use in conjunction with a standard IBM compatible PC to record a
patient’s ECG and heart rate variability.
Features
•
Detection Principle: Electrocardiography (ECG)
•
Records Heart Rate Variations
•
Displays Graphical Trends of HRV on PC
•
Connected through RS-232.
134
Fig. 5.7:Bioview HRV monitor
Cost: Rs.1.75 lakh (excluding the cost of the PC)
The comparison of the portable ECG monitors and HRV equipments
available in the market with the proposed work is presented in the next
section.
5.3.7
Resources of Proposed work
One of the objectives of the work was to reduce the cost. A
microcontroller’s inexpensive, flexible, and autonomous design allows it
to command almost any contemporary task that employs embedded
systems. The complete EZ430 package, including the tool and the
software, trades at $20 (approx. Rs.1200). The additional hardware
package could total about $25 - $30 (approx. Rs.1500/-). An EZ430
package consists of:
•
A CD (full copy) of Texas Instruments for the MSP430 including
training labs, presentations, and professional material.
•
Advanced MSP430 development kit (compiler and hardware)
Table 5.1: Budget for the Project
Necessary
Hardware
Quantity
Price (in
Rupees)
Remarks
EZ430
1
1500/-
With complete
package, including the
compiler
Connector
2
100/-
Electrodes
2
150/-
135
LED
4
20/-
photo diode
1
5/-
LM35
1
95/-
Hardware Subtotal
Software
1870/0/- (included in EZ430 package)
Other charges
100/-
Total
1970/- round off 2000/-
5.4 Conclusion
This chapter discussed the benefits of the implemented research that
includes the different aspects of TI’s low cost, fixed-point, and ultra lowpower chips that are often used as smart components. The major
influences are in favor of the low-power consumption, small package size
and less cost. The integrated analog and processing power of the
MSP430F2013 family provides a low cost yet powerful MCU solution that
can be used in various applications. The Sleep/Awake algorithm designed
and developed in this research has helped in less consumption of power.
For more details, see the following conclusion table.
Table 5.3: Comparison of the Existing Machines and Proposed Work
PARAMETERS
Detection Principle
HOLTER
MONITORS
ECG
DIGITAL HR
MONITOR
BEDSIDE
MONITOR
(MPM5533)
ECG
CARDIOMON
(CCM900)
BIOVIEW
Alarm on Portable Power Supply
Cost
Detection of
Abnormality
No
Yes
1.5 volt AA Rs.50,000 to
alkaline
2 lakh
battery
No
Yes
2-7 V
Rs.20,000
ECG & PPG
Yes
Yes
ECG
Yes
Yes
100 - 240
VAC, 47-63
Hz, 1.1 – 0.45
A
12 VDC
ECG
No
Yes
230 V / 50 Hz Rs.1.75 lakh
136
Rs.5 lakh
Rs.87,900
No
Yes
excluding the
cost of PC
3.6V – 6.5V Rs. 177840/-
Yes
Yes
9V Battery
Rs 16,433 /-
Yes
Yes
9V Battery
Rs 11,524
YES
YES
1.8 V
Rs 40005000
HRV MONITOR
ProComp5
Infiniti™
GSR2/Temp 2X
EEG, EKG, SEMG and
HR/BVP signals
GSR/BVP
GSR
GSR2 Biofeedback
Relaxation System
Proposed Work GSR/BVP/Temperature
137
CHAPTER 6
EMOTION
RECOGNITION
DEVICE
USER
GUIDE
This guide introduces the Blood volume Pulse (BVP), Galvanic Skin
Response (GSR) and Temperature biofeedback detecting emotion Training
System. This device is designed to develop the skills in body awareness
and self-regulation practice. The guide has two purposes: first, the
explanation of what emotion detection device actually is, and the second,
the training on how to use the machine. Specifically, the physiological
parameters that are being measured in this are: Skin Conductance
(SC),(Zantis, 2012) Volume of blood in each pulse, and Temperature. The
Sympathetic Nervous System, which is one of the two branches of the
ANS causes quite rapid increases in the physiological parameters. Subjects
(Human) measure changes in the skin’s conductivity due to a stimulus,
whether it is a picture, smell, sound, or touch.
6.1 General guidelines
The Skin Temperature, Galvanic Skin Response (GSR), and Blood
Volume Pulse (BVP) feedbacks are measured from the fingers. The
fingers are highly sensitive to the emotions, such as Hypertension, Stress,
Phobias, and Anxiety in the human body. This equipment can measure the
different parameters and based on that it can display the results/emotions
on the screen (LCD/LED). The activity of the glands is determined by the
autonomic nervous system, which contains two major subunits: (a) the
parasympathetic nervous system and (b) the sympathetic nervous system.
The glands of the skin are solely controlled by the sympathetic nervous
system, making them a good indicator for the inner strain and stress. The
sympathetic nervous system reacts to the stress stimuli by activating all the
“emergency functions” of the body; bringing it to a state of heightened
responsiveness, that is, increased pulse rate and blood pressure. With these
changes comes the effect of “wet hands” on which our measurement relies.
138
A scientific theory for this effect assumes that our ancestors needed it to
have a firmer grip on things, for example, in a flight or pursuit through a
difficult
terrain.
When
the
threatening
situation
is
over,
the
parasympathetic nervous system becomes dominant; the pulse rate, blood
pressure, and glucose level start falling. The body enters a resting state to
allow the recuperation, and the hands become dry again.
The skin response is a quite universal tool for the biofeedback training. It
is widely used in the therapy of anxiety, panic disorders, and specific
phobias. Further fields of use are high blood pressure, tinnitus, and sleep
disorders. If you suffer from a serious disorder or medical condition,
always consult a professional physician or therapist, and do not attempt a
treatment on your own. Measurements are done by placing two electrodes
on two fingertips of the same hand. The dark colored lower side of the
electrode should be in good skin contact. The goal of the feedback training
is twofold: (a) reduction of the permanent, that is, basic level of stress and
(b) reduction of the immediate stress response to a particular
stimulus.(Ohme et al., 2009)
6.2 Technical background
The philosophy of technology is based on each sensor’s ability to store its
own data on an internal flash memory chip and micro-controller. The
sensor uses an internal voltmeter to detect the micro changes in the skin’s
resistance and conductivity. When a stimulus is sensed, the sympathetic
nervous system reacts causing many physiological changes including the
release of miniscule amounts of sweat from the sweat glands. These small
changes in the skin’s moisture level allow for an electrical current to pass
through both skin and tissue more easily. In BVP the LED must face the
light detector in order to detect the light as it passes through the tissues.
The probe emits a light when the machine is switched on. You must check
that you can see the light to make sure the probe is working properly.
6.3 Using the skin response in the biofeedback training
139
For a successful training, we need a quiet and comfortably tempered room
with convenient seating and clothing; and without the phones and any
other sources of distraction. You should avoid all the conditions that can
make you sweat out of purely the physical reasons, such as an intense
physical activity before the training or an intense sunlight and heat. To
obtain the comparable results, you should try to keep the initial and
ambient conditions constant throughout all the training sessions.
Wrap the two electrodes around the upper or middle phalanges of your
index and middle fingers of the same hand. The dark lower side of the
electrodes must be in good skin contact. It is recommended to use the nondominant hand (that is, the left hand for right handed people and vice
versa), because the skin tends to be a little less callused n the nondominant hand. Attach the cables and wrap the tape around the clips to
ensure a firm contact. Similarly, BVP consists of two parts: (a) the light
emitting diodes (LEDs) and (b) a light detector (called a photo-detector).
The beams of the light pass through the tissues from one side of the probe
to the other. The blood and tissues absorb some of the light that is emitted
by the probe. The light absorbed by the blood varies with the saturation of
hemoglobin. The photo-detector detects the transmitted light as the blood
pulses through the tissues and based on this, the MCU calculates a value.
The grip of the tape on the sensors should neither be too firm to block the
blood circulation nor too loose to let the sensors slip and move around.
The subject should keep the hand down onto a comfortable support; in a
calm and relaxed resting position.(Nestoriuc et al., 2008)
6.4 Hardware/Software Set-up
You can start setting up the device set-up by clicking a small push button.
One of the options is 'Simulator'. In the simulator mode, you can learn how
to use the software without the encumbrance of sensors and leads etc. The
application simply uses the recorded data as a source. The skin of the
subject should be clean and free from any oily substances, for example, the
suntan lotion. Some subjects have very dry skin that can make it difficult
140
for the sensors to capture a good signal. Users with very dry skin will have
to be treated differently. The finger contacts should be snug on two
adjacent fingers – usually the index and the second finger – but not too
tight to be uncomfortable. Make sure that the two sensor contacts are in
good contact with the skin.
Once set up, the program starts running and sends the data across to be
stored in MCU for the future analysis. On the same time, the simulations
can also be observed on the attached PC, if so required.
Steps to use the instrument
•
Plug in the USB port to switch on the device.
•
Place the fingers on each sensor with a delay of 3 seconds.
Fig 6.1 The electrode placement (Marmor et al., 2009)
One should keep the sensors at a distance, so that they do not touch each
other. The electrodes must be separated first and then fingers placed on
them.
•
Take off the fingers from the sensors.
•
This part is capable of sensing the different bio-signals, converting the
analog signals to the digital ones, processing & analyzing the data, and
finally showing the output using LED. The RED LED reflects Stress,
the YELLOW LED reflects Calmness, and the GREEN LED reflects
Joyfulness.
141
•
At this stage, if you remove the fingers from the sensors, the LCD
displays “SYSTEM READY TOUCH SENSOR.”
•
System is again ready and you can ask the next subject for the analysis.
6.4.1 Biofeedback training by using the proposed device
First stage: observe, test, and determine the initial status
Record your state for the 30 seconds at rest without influencing the
measurement. Relax as best as you can and do not watch the measured
values, as it can compromise a true measurement. If you find your skin
conductance continuously rising without any reason, you may have
attached the electrodes too firmly, making you sweat beneath. At this
point, the humidity should be directly noticeable. If necessary, dry your
hands and reattach the electrodes with a little more slack.
Second stage: Targeted Biofeedback training with the skin response
The second stage consists of multiple sessions. Start the measurement and
watch the values for a while. Then try to bring the values down through an
active and conscious intervention. There are many ways to do this, such as
breathing in a controlled and calm pattern; and techniques of muscle
relaxation or autosuggestion. The exact way is up to you; your knowledge
on relaxation and your will to experiment. The device gives you a realtime feedback of even the smallest effects. This can change the emotion,
for example, from stress to calmness.
Third stage: deliberate provocation, relaxation, and stress coping
Actively use the stress stimuli (stressors) to improve your ability to deal
with them. Because of its immediate feedback and sensitivity, the skin
response is particularly a useful tool to work with direct provocation. It
helps in knowing that the amplitude of the skin reaction is proportional to
the intensity of the stressor. The training begins with a period of rest, so
start the measurement and relax for a couple of minutes. A selective
142
stressor should be applied, such as sound or image with a negative
connotation. Almost every person knows about the certain things or
situations that can cause distress for him or her. As an example: if you
have trouble speaking out loud before a larger group of people, try to
imagine the situation and pretend holding a speech before a large group.
Under the influence of such a stressor, you are likely to see a surge in the
skin response.
6.4.2 Standard Controls
Main control is with MCU that acquires the data, does extraction by using
a code, analyzes the data, and then displays the output as certain emotion
with the help of the 3 LEDs (where the RED is for STRESS, the GREEN
is for JOY, and the YELLOW is for CALMNESS). Each factor in the
device is self-explanatory. The control sets the time period at which the
data is written to the session data file. It is particularly useful for the
coaches and the therapists. The events, thus marked are represented in the
long term chart by a vertical white line.
6.4.3 Sensor features:
•
Fully digital data.
•
Rugged plastic ergonomic case.
•
Push button switch for Start/Stop experiments in the off-line mode.
•
LED indicator of experiment status (blinks while collecting the data).
•
Pre-calibrated sensing equipment.
Note: product is intended for the educational and medical use only.
6.4.4 Precautions and safety
•
Do not allow any liquid substance into either the GSR sensor or
probes.
143
•
Wash hands prior to using the device for the best results.
•
After use, gently wipe away any foreign material from the sensor.
•
Store in a box at the room temperature, keeping out of the direct
sunlight.
•
Do not use or store the device in the dusty or dirty areas; its moving
parts and electronic components can be damaged.
•
Do not store the device in the hot areas. High temperatures can shorten
the life of the electronic devices
•
Power off the device when not in use.
•
This device is not water-resistant. Keep it dry.
•
Use only in the normal position as explained in the product
documentation.
144
Chapter 7
CONCLUSION
In order to ensure that a healthcare service adapts a real-time intelligent
system while interacting with the patients, a reliable algorithm is required
to classify the emotional states of a patient. This chapter is based upon the
overall conclusions drawn from this research work. It also includes the
conclusions drawn from the emotion intelligence, by implementing
different intelligent methods on the adaptive and non-adaptive systems for
estimating the emotion value. Moreover, this chapter also describes the
future scope of this research work.
7.1 Introduction
In the general engineering terms, feedback is used to control a process. If
this concept is applied to the biological processes within the body, it is
known as biological feedback or biofeedback. A variable produced by the
procedure is measured and compared with the reference value; based on
the differences, an action is taken to bring the variable equal to the
reference value. The body functions that are controlled by the autonomic
nervous
system
are
generally
not
subject
to
the
voluntary
control.(Krassioukov et al., 2012) In fact, most of these body functions are
not concisely perceived. However, it has been found that if some suitable
methods can be used to measure these functions, and if the information
pertaining to the magnitude of these functions can be conveyed to the
subject, a certain degree of voluntary control can be exercised over some
of these body functions. Biofeedback is not completely understood, but
presently, it is being used in the clinical treatments. Many different
physiological processes have been evaluated for a possible control by the
biofeedback methods including EEG, EMG, heart rate, blood pressure,
145
GSR, BSR, and temperature. (Uchino, 2006)There have been a number of
experimentations in the use of biofeedback for the secondary effects.
A major shortage of the potable, cost effective, and power efficient
intelligent methods has motivated a design for the direct interaction with
patients. There is a need to develop a system that is capable of detecting
and responding to the emotional states of a patient and facilitating the
positive patient experiences in the healthcare sector. Analysis of the
effects on the emotional states of the patient has revealed that the
physiological signals (GSR, BVP, and Temperature) must have a utility
for promoting the positive arousal and valence states. Afterwards, an
automatic prediction of the users' emotion-based machine learning
algorithms (Naïve Bayes and Markov Model) is made.
7.2 Conclusions of the study
The biofeedback device includes a sensor capturing the body variables
(bio-signals) that is to be analyzed by the biofeedback process. The
magnitude of the measured variable is converted into a appropriate visual
or auditory cue to be accessible to the subject. Occasionally, it is necessary
to provide an additional signal processing among the measurement and
feedback part of the instrumentation. In the system that is being designed
here, the changes in the physiological signals (due to the changes in the
body conditions) are captured and processed for the data analysis; and then
are displayed in a form of emotion. Biofeedback has already been
represented by some to be the purest form of “self-control”. The
accomplishment of biofeedback depends on the interpretation of the data
and training of the subjects, so that they can use the results successfully.
Some people believe that they perform better while under stress, but the
fact is, that’s the rare case. In fact, research has shown that the stress
makes a person more likely to make mistakes. (Kouzes and Posner, 2010)
Besides, the stress makes a person forget where he/she has put the keys.
146
Stress can also have a dramatic impact on your health. It can make you
sick, ruin your teeth, ruin your heart, make you fat, make you look older,
or can also weaken your immune system.
Emotions play a pivotal role while dealing with the patients who are going
through stress. The main aim of this study is to detect and estimate the
emotions. These emotional characterizations can help improve the
performance of the recommendation and retrieval systems. The analysis
and evaluation directions in this thesis consist of methodology and the
results of the emotion recognition methods employed to detect the
emotions.
The detection of a new emotional state by a classification algorithm may
serve as an effective method for the severing patient states and behavior
adaptation to promote the positive patient healthcare experiences. This
research has developed a new computational algorithm (HYBRID-NAVMAR) for the human emotional state classification to facilitate the
doctors/therapist in predicting the emotion. The physiological responses,
such as Skin temperature, blood volume pulse (BVP) and galvanic skin
response (GSR) have been found to be obvious and valid indicators of
valence and arousal. Such physiological signals from the patients were
monitored as real-time system in the hospitals. It is possible to extract the
physiological measures and to classify the emotional states during the
patient interaction with a developed system.
147
Sensor
A/D Converter
Micro-
LCD
(display)
Controll
er
Power
Supply
Fig.7.1 Basic Biofeedback System
All experiments were done while treating different subjects. By
monitoring the physiological signals, the emotion values were
collected for the participants during the experiment trials. The process
involved: (1) physiological feature extraction (2) feature selection and
(3) a machine learning model of the emotional states. A proposed
algorithm was applied to elucidate the relationships among the
physiological responses and patient emotional states. Analysis of the
patient emotional states revealed individual to detect emotions
(Calmness, Stress, and Joyful).
7.3 Limitations of the Study
The research findings revealed the classification models for predicting the
emotional states to depend upon the user physiological characteristics.
However, this research did not examine these factors consistently across
various studies, such as facial, vocal, or motion. The responses were not
collected from the older participants. The effects of skin moisture and
subcutaneous fat thickness on the ability of the skin to dissipate heat in
young and old subjects, with and without diabetes, at three different
environmental room temperatures were measured.(McLellan et al., 2009)
148
7.4 Future scope
This information can provide a basis for the real-time adaptation of
the new methodology for the development of some applications, such
as robotics behavior to optimize the patient emotional experiences in,
for example, medicine delivery tasks. A future research should focus
on integrating different technologies. If implemented in robotics, the
predicted emotional state obtained from the classification model
should be transferred as an input to the behavior adaptation system to
determine the appropriate expressions.
149
Signal sensing
INPUT FROM HUMAN BODY
(GSR/BVP/Temperature)
A/D Conversions
NO
If input
successfull
fetched
With Gap of 2
Minutes, repeat the
process
YES
Naïve Bayesian
Emotion (JOY/CALM/STRESS)
Probability matrix:
P (Tn|Tn-1,Tn-2,…..,T1)
HYBRID-NAV-MAR
P (Xt-j|X0=i0, X1=i1……Xt-1)
=P (Xt=j|Xt-1=it-1)
Markov Model
Future Emotion
Fig.7.2 Work flow of implemented Algorithms
150
REFERENCES
Abu-Nimeh S, NAPPA D, WANG X and NAIR S (2007) “A comparison of
machine learning techniques for phishing detection” Proceedings of the antiphishing working group’s 2nd annual eCrime researcher’s summit, ACM, pp. 6069
AllenJ, Harmon-jone and Cavender H(2001) Manipulation of frontal EEG
asymmetry through biofeedback alters self-reported emotional responses and facial
EMG. Psychophysiology,pp 38, 685-693
Andreassi, J. L. (2000) Psychophysiology: Human behavior & physiological
response, Psychology Press
Androutsopoulos I, Koutsias J, Chandrinos K. V, Paliouras G. and Spyropoulos C.
D. (2000). An evaluation of naive bayesian anti-spam filtering. arXiv preprint pp
cs/0006013
Andziulis A, Bielskis A, Denisovas V, Ramašauskas O, Bielskien÷ J, Bielskis, P
and Guseinovien÷ E.(2009) An Approach of Creating of an Intelligent Mobile
Tutoring Eco–social Laboratory for Assisted Recreation. Electronics and Electrical
Engineering.–Kaunas: Technologija, pp. 91.
Baldi P and Brunak S( 2001)Bioinformatics: the machine learning approach, MIT
press
Bitterman M & Holtzman W. H( 1952) Conditioning and extinction of the galvanic
skin response as a function of anxiety. The Journal of Abnormal and Social
Psychology, pp 47, 615
Bonnet M and Arand D(1997) Heart rate variability: sleep stage, time of night, and
arousal influences. Electroencephalography and clinical neurophysiology pp.102,
390-396
Brodal P(2004) The central nervous system: structure and function, Oxford
University Press
Brunelli D, Farella E, Rocchi L, Dozza M, Chiari L. and Benini L.(2006) “Biofeedback system for rehabilitation based on a wireless body area network”.
Pervasive Computing and Communications Workshops,. PerCom Workshops 2006.
Fourth Annual IEEE International Conference on,. IEEE, 5 pp.-531
Burke R (2002) Hybrid recommender systems: Survey and experiments. User
151
modeling and user-adapted interaction,pp. 12, 331-370
Busso C, Deng Z, Yildirim S, Bulut M, Lee C M, Kazemzadeh A, Lee S, Neumann
U. and
Narayanan S. (2004) “Analysis of emotion recognition using facial
expressions, speech and multimodal information”
Proceedings of the 6th
international conference on Multimodal interfaces,. ACM, pp. 205-211
Caton, R. (1875) Electrical Currents of the Brain. The Journal of Nervous and
Mental Disease, 2,pp 610
Chanel G, Kierkels J, Soleymani, M. & Pun, T (2009) Short-term emotion
assessment in a recall paradigm. International Journal of Human-Computer
Studies, pp.67, 607-627
Chrousos G. P. (2009) Stress and disorders of the stress system. Nature Reviews
Endocrinology, pp.5, 374-381
Chuang Z.-J. and Wu, C.-H (2004) Multi-modal emotion recognition from speech
and text. Computational Linguistics and Chinese Language Processing, pp.9, 45-62.
Ciarrochi J. V, Chan A. Y. and Caputi P. (2000) A critical evaluation of the
emotional intelligence construct. Personality and Individual differences, pp.28,
539-561.
Clancy E, Morin E. L. and Merletti R. (2002) Sampling, noise-reduction and
amplitude
estimation
issues
in
surface
electromyography.
Journal
of
Electromyography and Kinesiology, pp 12, 1-16
Corwin E. J. and Williams L. (2008) Handbook of pathophysiology.
Cowie R. and Cornelius R. R.(2003) Describing the emotional states that are
expressed in speech. Speech communication, pp. 40, 5-32.
Cowie R, Douglas-Cowie E., Karpouzis K, Caridakis G, Wallace M. and Kollias, S.
(2008) Recognition of emotional states in natural human-computer interaction.
Multimodal user interfaces. Springer.
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W.
& Taylor, J. G. (2001) “ Emotion recognition in human-computer interaction”
Signal Processing Magazine, IEEE, 18, 32-80.
Craig, S., Graesser, A., Sullins, J. & Gholson, B. (2004) Affect and learning: an
exploratory look into the role of affect in learning with AutoTutor. Journal of
educational media, 29, 241-250.
152
Davies, J. H. (2008) Msp430 Microcontroller basics, Elsevier.
De Santos Sierra A., Ávila C. S., Guerra Casanova, J. & Del Pozo, G. B. (2011) “
A Stress-Detection system based on physiological signals and fuzzy logic”
Industrial Electronics, IEEE Transactions on, pp 58, 4857-4865.
Drechsler R., Straub M Doehnert, M. Heinrich, H., Steinhausen, H.-C. & Brandeis,
D. (2007) Controlled evaluation of a neurofeedback training of slow cortical
potentials in children with attention deficit/hyperactivity disorder (ADHD). Behav
Brain Funct, pp.3, 35.
Eckenrode J.( 1984) Impact of chronic and acute stressors on daily reports of mood.
Journal of personality and social psychology,pp. 46, 907.
Edelberg, R. & Wright, D. J. (1964) Two galvanic skin response effector organs
and their stimulus specificity. Psychophysiology, pp.1, 39-47.
Ekman P., Levenson R. W. & Friesen W. V. (1983) Autonomic nervous system
activity distinguishes among emotions. Science, 221, 1208-1210.
Elfenbein, H. A. (2002) Emotional Intelligence and the Recognition of Emotion
from Facid Expressions.
Evans J. R. & Abarbanel A. (1999) Introduction to quantitative EEG and
neurofeedback, Elsevier.
Fontaine J. R., Scherer K. R., Roesch E. B. & Ellsworth P. C. (2007) The world of
emotions is not two-dimensional. Psychological science, 18, 1050-1057.
Gabella, G. (2001) Autonomic nervous system. eLS.
Gao Y., Raine A., Dawson M. E., Venables P. H. & Mednick S. A.(
2007)Development of skin conductance orienting, habituation, and reorienting from
ages
3
to
8
years:
A
longitudinal
latent
growth
curve
analysis.
Psychophysiology,pp. 44, 855-863.
Gelman A., Carlin J. B., Stern H. S. and Rubin D. B. (2014) Bayesian data
analysis, Taylor & Francis.
Goleman D. (2006) Emotional intelligence, Bantam.
Greenhalgh J., Dickson R. and Dundar Y. (2010) Biofeedback for hypertension: a
systematic review. Journal of hypertension, pp.28, 644-652.
Gros J. J. and Thompson, R. A. (2007) Emotion regulation: Conceptual
foundations.
153
Gunes, H. & Piccardi, M. (2007) Bi-modal emotion recognition from expressive
face and body gestures. Journal of Network and Computer Applications, pp.30,
1334-1345.
Haapalainen, E., Kim, S., Forlizzi, J. F. & Dey, A. K. (2010) “Psycho-physiological
measures for assessing cognitive load”. Proceedings of the 12th ACM international
conference on Ubiquitous computing,. ACM, pp.301-310.
Haffner, P., Sen, S., Spatscheck, O. & Wang, D. Acas: (2005) “Automated
Construction Of Application Signatures.” Proceedings of the ACM SIGCOMM
workshop on Mining network data, ACM, 197-202.
Hagemann D, Waldstein S. R. and Thayer, J. F.( 2003) Central and autonomic
nervous system integration in emotion. Brain and cognition, pp. 52, 79-87.
Hanke T, Charitos E. I, Stierle U., Karluss A., Kraatz E., Graf B., Hagemann A.,
Misfeld M. and Sievers, H. H. (2009) Twenty-four–hour Holter monitor follow-up
does not provide accurate heart rhythm status after surgical atrial fibrillation
ablation therapy up to 12 months experience with a novel permanently implantable
heart rhythm monitor device. Circulation, pp.120, S177-S184.
Healey J. A. and Picard R. W. (2005) “Detecting stress during real-world driving
tasks using physiological sensors”. Intelligent Transportation Systems, IEEE
Transactions on, pp.6, 156-166.
Heale J. A. & Picard R. W. (2005) Detecting stress during real-world driving tasks
using physiological sensors. Intelligent Transportation Systems, IEEE Transactions
on, pp.6, 156-166.
Heinze C., Trutschel U. & Golz M. (2012) The relation of heart rate variability
measures to psychophysiological variables with circadian rhythmicity. Biomedical
Engineering/Biomedizinische Technik, pp.57, 237-240.
Henneman M. M., Bax J. J., Schuijf J. D., Jukema J. W., Holman E. R., Stokkel M.
P., Lamb H. J., De Roos A. & Van Der Wall, E. E. (2006) Global and regional left
ventricular function: a comparison between gated SPECT, 2D echocardiography
and multi-slice computed tomography. European journal of nuclear medicine and
molecular imaging, 33, 1452-1460.
Hock, A. G.( 2000). Biofeedback system for monitoring the motion of body joint.
Google Patents.
Hock, A. G. (2000)Biofeedback system for sensing body motion and flexure.
154
Google Patents.
Horlings, R., Datcu, D. & Rothkrantz, L. J. Emotion recognition using brain
activity. Proceedings of the 9th international conference on computer systems and
technologies and workshop for PhD students in computing, 2008. ACM, 6.
Hubbard, J. A., Smithmyer, C. M., Ramsden, S. R., Parker, E. H., Flanagan, K. D.,
Dearing, K. F., Relyea, N. & Simons, R. F. (2002)Observational, physiological, and
self–report measures of children’s anger: Relations to reactive versus proactive
aggression. Child development, 73, 1101-1118.
Jansen, A. S., Van Nguyen, X., Karpitskiy, V., Mettenleiter, T. C. & Loewy, A. D.
(1995). Central command neurons of the sympathetic nervous system: basis of the
fight-or-flight response. Science, 270, 644-646.
Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N. & Stone, A. A.( 2004)
A survey method for characterizing daily life experience: The day reconstruction
method. Science, 306, 1776-1780.
Karegowda, A. G., Jayaram, M. & Manjunath, A.(2010)Feature subset selection
problem using wrapper approach in supervised learning. International journal of
Computer applications, 1, 13-17.
Kim, K. H., Bang, S. & Kim, S. (2004) Emotion recognition system using shortterm monitoring of physiological signals. Medical and biological engineering and
computing, 42, 419-427.
Kim, Y. B., Yoo, S. K. & Kim, D. (2006) Ubiquitous healthcare: Technology and
service. Intelligent Paradigms for Assistive and Preventive Healthcare. Springer.
Koh, C., Young, C., Young, J. & Solomon, M. (2008) Systematic review of
randomized controlled trials of the effectiveness of biofeedback for pelvic floor
dysfunction. British Journal of Surgery, 95, 1079-1087.
Kopka, A. & Crawford, J. (2004)Cricoid pressure: a simple, yet effective
biofeedback trainer. European journal of anaesthesiology, 21, 443-447.
Kouzes, J. M. & Posner, B. Z. (2010) The truth about leadership. San Francisco,
CA: Jossey-Bass.
Krassioukov, A., Biering-Sørensen, F., Donovan, W., Kennelly, M., Kirshblum, S.,
Krogh, K., Alexander, M. S., Vogel, L. & Wecht, J. (2012) International standards
to document remaining autonomic function after spinal cord injury. The journal of
155
spinal cord medicine, 35, 201-210.
Kreibig, S. D. (2010) Autonomic nervous system activity in emotion: A review.
Biological psychology, 84, 394-421.
Kylliäinen, A. & Hietanen, J. K. (2006)Skin conductance responses to another
person’s gaze in children with autism. Journal of autism and developmental
disorders, 36, 517-525.
Lakin, B. (1998)Thermostat with remote temperature sensors and incorporating a
measured temperature feature for averaging ambient temperatures at selected
sensors. Google Patents.
Laparra-Hernández, J., Belda-Lois, J., Medina, E., Campos, N. & Poveda, R.
(2009) EMG and GSR signals for evaluating user's perception of different types of
ceramic flooring. International Journal of Industrial Ergonomics, 39, 326-332.
Larsen, J. T., Berntson, G. G., Poehlmann, K. M., Ito, T. A. & Cacioppo, J. T.
(2008)The psychophysiology of emotion. Handbook of emotions, 3, 180-195.
LAZARUS, R. S. 1991. Emotion and adaptation, Oxford University Press.
LEHRER, P. M., VASCHILLO, E. & VASCHILLO, B. 2000. Resonant frequency
biofeedback training to increase cardiac variability: Rationale and manual for
training. Applied psychophysiology and biofeedback, 25, 177-191.
LEWIS, D. D. 1998. Naive (Bayes) at forty: The independence assumption in
information retrieval. Machine learning: ECML-98. Springer.
LIN, G., XIANG, Q., FU, X., WANG, S., WANG, S., CHEN, S., SHAO, L.,
ZHAO, Y. & WANG, T. 2012. Heart rate variability biofeedback decreases blood
pressure in prehypertensive subjects by improving autonomic function and
baroreflex. The Journal of Alternative and Complementary Medicine, 18, 143-152.
Loewy, A. D. & Spyer, K. M. (1990) Central regulation of autonomic functions,
Oxford University Press New York.
Lucovnik, M., Maner, W. L., Chambliss, L. R., Blumrick, R., Balducci, J., NovakAntolic, Z. & Garfield, R. E. (2011)Noninvasive uterine electromyography for
prediction of preterm delivery. American journal of obstetrics and gynecology, 204,
228. e1-228. e10.
Marmor, M., Fulton, A., Holder, G., Miyake, Y., Brigell, M., Bach, M. & Vision, I.
S. F. C. E. O. (2009) Iscev Standard for full-field clinical electroretinography (2008
156
update). Documenta Ophthalmologica, 118, 69-77.
Martin, G. & Johnson, C. L. (2006)The boys Totem town neurofeedback project: A
pilot study of EEG biofeedback with incarcerated juvenile felons. Journal of
Neurotherapy, 9, 71-86.
Mayer, J. D., Caruso, D. R. & Salovey, P. (2000) Selecting a measure of emotional
intelligence: The case for ability scales.
Mayer, J. D., Salovey, P., Caruso, D. R. & Sitarenios, G. (2001)Emotional
intelligence as a standard intelligence.
Mayer, J. D., Salovey, P., Caruso, D. R. & Sitarenios, G. (2003). Measuring
emotional intelligence with the MSCEIT V2. 0. Emotion, 3, 97.
Mclellan, K., Petrofsky, J. S., Bains, G., Zimmerman, G., Prowse, M. & Lee, S.
(2009) The effects of skin moisture and subcutaneous fat thickness on the ability of
the skin to dissipate heat in young and old subjects, with and without diabetes, at
three environmental room temperatures. Medical engineering & physics, 31, 165172.
Meule, A., Freund, R., Skirde, A. K., Vögele, C. & Kübler, A. (2012)Heart rate
variability biofeedback reduces food cravings in high food cravers. Applied
psychophysiology and biofeedback, 37, 241-251.
Miu, A. C., Heilman, R. M. & Miclea, M. (2009) Reduced heart rate variability and
vagal tone in anxiety: trait versus state, and the effects of autogenic training.
Autonomic Neuroscience, 145, 99-103.
Morone, N. E. & Greco, C. M. (2007)Mind–body interventions for chronic pain in
older adults: A structured review. Pain Medicine, 8, 359-375.
Nagai, Y., Critchley, H., Featherstone, E., Trimble, M. & Dolan, R. (2004) Activity
in ventromedial prefrontal cortex covaries with sympathetic skin conductance level:
a physiological account of a “default mode” of brain function. Neuroimage, 22,
243-251.
Nestoriuc, Y., Martin, A., Rief, W. & Andrasik, F. (2008) Biofeedback treatment
for
headache
disorders:
a
comprehensive
efficacy
review.
Applied
psychophysiology and biofeedback, 33, 125-140.
Noergaard, T. (2012)Embedded systems architecture: a comprehensive guide for
engineers and programmers, Newnes.
157
Nourbakhsh, N., Wang, Y., Chen, F. & Calvo, R. A.( 2012) Using galvanic skin
response for cognitive load measurement in arithmetic and reading tasks.
Proceedings of the 24th Australian Computer-Human Interaction Conference,
ACM, 420-423.
Ogorevc, J., Geršak, G., Novak, D. & Drnovšek, J. (2013). Metrological evaluation
of skin conductance measurements. Measurement, 46, 2993-3001.
Ohme, R., Reykowska, D., Wiener, D. & Choromanska, A. (2009)Analysis of
neurophysiological reactions to advertising stimuli by means of EEG and galvanic
skin response measures. Journal of Neuroscience, Psychology, and Economics, 2,
21.
Ohme, R., Reykowska, D., Wiener, D. & Choromanska, A. (2009) Analysis of
neurophysiological reactions to advertising stimuli by means of EEG and galvanic
skin response measures. Journal of Neuroscience, Psychology, and Economics, 2,
21.
Picard, R. W., Vyzas, E. & Healey, J. (2001) Toward machine emotional
intelligence: Analysis of affective physiological state. Pattern Analysis and
Machine Intelligence, IEEE Transactions on, 23, 1175-1191.
Piepoli, M. F., Conraads, V., Corra, U., Dickstein, K., Francis, D. P., Jaarsma, T.,
Mcmurray, J., Pieske, B., Piotrowicz, E. & Schmid, J. P.( 2011) Exercise training in
heart failure: from theory to practice. A consensus document of the Heart Failure
Association and the European Association for Cardiovascular Prevention and
Rehabilitation. European journal of heart failure, 13, 347-357.
Plotnikov, A., Stakheika, N., De Gloria, A., Schatten, C., Bellotti, F., Berta, R.,
Fiorini, C. & Ansovini, F. (2012)Exploiting real-time EEG analysis for assessing
flow in games.
Advanced Learning Technologies (ICALT), IEEE 12th
International Conference on, IEEE, 688-689.
Poh, M.-Z., Mcduff, D. J. & Picard, R. W. (2011) Advancements in noncontact,
multiparameter physiological measurements using a webcam. Biomedical
Engineering, IEEE Transactions on, 58, 7-11.
Preisach, C., Burkhardt, H., Schmidt-Thieme, L. & Decker, R. (2008) Data
Analysis, Machine Learning and Applications: Proceedings of the 31st Annual
Conference of the Gesellschaft Fü r Klassifikation EV, Albert-LudwigsUniversität Freiburg, March 7-9, 2007, Springer Science & Business Media.
158
Quattoni, A., Collins, M. & Darrell, T. (2004)Conditional random fields for object
recognition. Advances in neural information processing systems,pp. 1097-1104.
Rish, I.( 2001) An empirical study of the naive Bayes classifier. IJCAI 2001
workshop on empirical methods in artificial intelligence, IBM New York,pp. 41-46.
RON, S. 1997. Speech-based biofeedback method and system. Google Patents.
Ryu, C. Y., Shin, S. C., Nam, S. H., Kang, J. H., Song, Y. S., Yim, T. G., Kim, S.
& Kim, Y. T.(2008)Wearable physiological signal detection module and
measurement apparatus having the same. Google Patents.
Sauter D. A., Eisner F., Calder A. J. & Scott, S. K. (2010) Perceptual cues in
nonverbal vocal expressions of emotion. The Quarterly Journal of Experimental
Psychology, 63, 2251-2272.
Schafer A. M., Rains J. C., Penzien D. B., Groban L., Smitherman, T. A. and
Houle, T. T. (2011) Direct costs of preventive headache treatments: comparison of
behavioral and pharmacologic approaches. Headache: The Journal of Head and
Face Pain, pp.51, 985-991.
Schmidt L. A. and Trainor L. J.(2001) Frontal brain electrical activity (EEG)
distinguishes valence and intensity of musical emotions. Cognition & Emotion,
pp.15, 487-500.
Schuller B., Rigoll G. & Lang M. (2003 )Hidden Markov model-based speech
emotion recognition.
Acoustics, Speech, and Signal Processing, 2003.
Proceedings.(ICASSP'03). IEEE International Conference on, pp.. IEEE, II-1-4 vol.
2.
Schwartz, M. S. & Andrasik, F. E. (2003) Biofeedback: A Practitioner's Guide,
Guilford Press.
Shenoy, P., Krauledat, M., Blankertz, B., Rao, R. P. & Müller, K.-R. (2006)
Towards adaptive classification for BCI. Journal of neural engineering, 3, R13.
Shi, Y., Ruiz, N., Taib, R., Choi, E. & Chen, F.( 2007) Galvanic skin response
(GSR) as an index of cognitive load. CHI'07 extended abstracts on Human factors
in computing systems, ACM,pp. 2651-2656.
Shibata, T., Yoshida, M. & Yamato, J. Artificial emotional creature for humanmachine interaction.
Systems, Man, and Cybernetics, (1997)Computational
Cybernetics and Simulation., 1997 IEEE International Conference on,. IEEE,
159
pp.2269-2274.
Sivia, D. & Skilling, J. (1996) Data analysis: a Bayesian tutorial. AMC, 10, 12.
Sugar, T. G., He, J., Koeneman, E. J., Koeneman, J. B., Herman, R., Huang, H.,
Schultz, R. S., Herring, D., Wanberg, J. & Balasubramanian, S. (2007)Design and
control of RUPERT: a device for robotic upper extremity repetitive therapy. Neural
Systems and Rehabilitation Engineering, IEEE Transactions on, 15, 336-346.
Tarvainen, M. P., Koistinen, A. S., Valkonen-Korhonen, M., Partanen, J. &
Karjalainen, P. A. (2001) Analysis of galvanic skin responses with principal
components
and
clustering
techniques.
Biomedical
Engineering,
IEEE
Transactions on, 48, 1071-1079.
Tronstad, C., Johnsen, G. K., Grimnes, S. & Martinsen, Ø. G. (2010)A study on
electrode gels for skin conductance measurements. Physiological measurement, 31,
1395.
Uchino B. N. (2006) Social support and health: a review of physiological processes
potentially underlying links to disease outcomes. Journal of behavioral medicine,
29, 377-387.
Unakafov, A.(2009)Analysis and modeling of the galvanic skin response
spontaneous component in the context of intelligent biofeedback systems
development. Measurement Science Review, 9, 36-41.
Wagner, J., Kim, J. & André, E. From physiological signals to emotions:
Implementing and comparing selected methods for feature extraction and
classification.
Multimedia and Expo, 2005. ICME 2005. IEEE International
Conference on, 2005. IEEE, 940-943.
Westerink, J. H. D. M., Tuinenbreijer, C., Schut, M. H. & Van Den Broek, E. L.
2008. Computer program product, device and method for measuring the arousal of
a user. Google Patents.
Westeyn, T., Presti, P. & Starner, T. ActionGSR: A combination galvanic skin
response-accelerometer for physiological measurements in active environments.
Wearable Computers, 2006 10th IEEE International Symposium on, 2006. IEEE,
129-130.
Westland, J. C. 2011. Electrodermal Response in Gaming. Journal of Computer
Networks and Communications, 2011.
Wheeler, K. R. & Jorgensen, C. C. 2003. Gestures as input: Neuroelectric joysticks
160
and keyboards. IEEE pervasive computing, 2, 56-61.
White, R. & Gardner, D. 1983. Physostigmine prolongs the elementary event
underlying decay of inhibitory postsynaptic currents in Aplysia. The Journal of
neuroscience, 3, 2607-2613.
Wong, K. S., Fung, K. Y., Fung, S. M., Fung, C. W. & Tang, C. H. 2001.
Biofeedback of pelvic floor muscles in the management of genuine stress
incontinence in Chinese women: randomised controlled trial. Physiotherapy, 87,
644-648.
Wu, C.-H., Chuang, Z.-J. & Lin, Y.-C. 2006. Emotion recognition from text using
semantic labels and separable mixture models. ACM transactions on Asian
language information processing (TALIP), 5, 165-183.
Yang, J. & Honavar, V. 1998. Feature subset selection using a genetic algorithm.
Feature extraction, construction and selection. Springer.
Yazdani, A., Lee, J.-S., Vesin, J.-M. & Ebrahimi, T. 2012. Affect recognition based
on physiological changes during the watching of music videos. ACM Transactions
on Interactive Intelligent Systems (TiiS), 2, 7.
Yucha, C. & Montgomery, D. 2008. Evidence-based practice in biofeedback and
neurofeedback, AAPB Wheat Ridge, CO.
Zald, D. H. 2003. The human amygdala and the emotional evaluation of sensory
stimuli. Brain Research Reviews, 41, 88-123.
Zantis, F. P. 2012. Development of a low-power-signal acquisition device for
signals lower than 25 Hz.
Zhai, J. & Barreto, A. (2006. ) “Stress detection in computer users based on digital
signal processing of noninvasive physiological variables.” Engineering in Medicine
and Biology Society, EMBS'06. 28th Annual International Conference of the IEEE,
2006. IEEE, pp.1355-1358.
Zhang H. & Li, D. (2007) Naive Bayes text classifier. Granular Computing, 2007.
GRC 2007. IEEE International Conference on, IEEE, 708-708.
Zhao G. & Pietikainen M. (2007). Dynamic texture recognition using local binary
patterns with an application to facial expressions. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 29, 915-928.
161
162
APPENDIX-I
166
LIST OF CONSULTED DOCTORS
1.
Dr Gurjeev Rattan
2.
Dr.Vipun
3.
Dr. Akhil Chopra
4.
Dr. Chetna Sharma
5.
Dr.Anil Chopra
167
APPENDIX-II
168
PROBABILITY TABLE
Sr.No
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
GSR
BVP
Temperature
OUTPUT
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
High
Critical Low
EMERGENCY
High
High
EMERGENCY
Low
Critical High
EMERGENCY
Low
Critical Low
EMERGENCY
Low
High
EMERGENCY
Normal
Critical High
EMERGENCY
Normal
Critical Low
EMERGENCY
Normal
High
EMERGENCY
High
Low
JOY
169
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
Normal
JOY
Low
Low
JOY
Low
Normal
JOY
Normal
Low
JOY
Normal
Normal
JOY
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
High
Critical Low
EMERGENCY
High
High
EMERGENCY
High
Low
EMERGENCY
High
Normal
EMERGENCY
Low
Critical High
EMERGENCY
Low
Critical Low
EMERGENCY
170
43
44
45
46
47
48
49
50
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
51
High
52
High
53
High
54
High
55
High
56
High
57
High
58
High
59
High
60
61
62
63
64
65
66
67
High
High
High
High
High
High
High
Low
High
EMERGENCY
Low
Low
EMERGENCY
Low
Normal
EMERGENCY
Normal
Critical High
EMERGENCY
Normal
Critical Low
EMERGENCY
Normal
High
EMERGENCY
Normal
Low
EMERGENCY
Normal
Normal
EMERGENCY
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
High
Low
Low
Normal
Normal
High
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
Critical Low
Critical High
Critical Low
Critical High
Critical Low
High
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
ILLNESS
171
68
69
70
71
72
73
74
75
High
High
High
High
High
High
High
High
Low
76
Low
77
Low
78
Low
79
Low
80
Low
81
Low
82
Low
83
Low
84
Low
85
86
87
88
89
90
91
92
93
94
95
96
97
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
Normal
Low
Low
High
Normal
High
Normal
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
High
Low
Low
Normal
Normal
High
Low
Normal
High
High
Low
172
ILLNESS
ILLNESS
JOY
JOY
JOYFUL
JOYFUL
JOYFULL
JOYFULL
High
High
Low
Normal
Low
Normal
Normal
Low
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
Critical Low
Critical High
Critical Low
Critical High
Critical Low
High
High
High
Low
Normal
Low
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
ILLNESS
ILLNESS
ILLNESS
STRESS
STRESS
STRESS
98
99
100
Low
Low
Low
Normal
101
Normal
102
Normal
103
Normal
104
Normal
105
Normal
106
Normal
107
Normal
108
Normal
109
Normal
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Normal
Low
Normal
Normal
Critical
High
Critical
High
Critical
High
Critical
High
Critical
High
Critical
Low
Critical
Low
Critical
Low
Critical
Low
Critical
Low
High
High
Low
Low
Normal
Normal
High
Low
Normal
High
High
Low
Low
Normal
Normal
173
Normal
Low
Normal
Critical High
STRESS
STRESS
STRESS
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
EMERGENCY
Critical Low
EMERGENCY
High
EMERGENCY
Low
EMERGENCY
Normal
EMERGENCY
Critical High
Critical Low
Critical High
Critical Low
Critical High
Critical Low
High
High
High
Low
Normal
Low
Normal
Low
Normal
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
EMERGENCY
ILLNESS
ILLNESS
ILLNESS
RELAX
RELAX
RELAX
RELAX
RELAX
RELAX
APPENDIX-III
174
SAMPLE EXPERIMENTS
BVP
TEMPERATURE
Emotions
Subject 1
GSR
Low
Low
Low
Anxiety
Subject 2
Low
High
Low
Anxiety
Subject 3
High
Low
Low
RELAX
Subject 4
High
High
Low
JOYFULL
Subject 5
Normal
High
Low
RELAX
Subject 6
Normal
Low
Low
RELAX
Subject 7
High
Normal
Low
JOYPULL
Subject 8
Low
Normal
Low
Anxiety
Subject 9
High
High
Normal
JOYFULL
High
Low
Normal
RELAX
Subject 11
Normal
Normal
Normal
RELAX
Subject 12
Low
Low
Low
Anxiety
Subject 13
Normal
Normal
Normal
RELAX
Subject 14
High
High
Low
JOYFULL
Subject 15
Low
Low
Normal
Anxiety
Subject 16
Low
High
Normal
Anxiety
Subject 17
Normal
Normal
Normal
RELAX
Subject 18
Low
Low
Low
Anxiety
Subject 19
High
High
Low
JOYFULL
Subject 20
Normal
Normal
Normal
RELAX
Subject 21
Normal
Low
Low
RELAX
Subject 22
Low
Low
Normal
Anxiety
Subject 23
Low
Low
Low
Anxiety
Subject 24
High
Low
Normal
RELAX
Subject 25
High
Normal
Low
JOYPULL
Subject 26
Low
Low
Low
Anxiety
Subject 27
High
Low
Normal
RELAX
Subject 28
High
Low
Normal
RELAX
Normal
Normal
Normal
RELAX
Subject 30
High
High
Low
JOYFULL
Subject 31
High
Low
Normal
RELAX
Subject 32
High
Low
Normal
RELAX
Subject 33
Normal
Normal
Normal
RELAX
Subject 34
High
Low
Normal
RELAX
Subject 35
Low
Low
Low
Anxiety
Subject 36
Low
High
Low
Anxiety
Subject 37
Normal
Normal
Normal
RELAX
PERSON
Subject 10
Subject 29
Gender
Female
Male
175
Subject 38
High
Low
Normal
RELAX
Subject 39
Low
High
Low
Anxiety
Subject 40
Normal
Normal
Normal
RELAX
176