User Experience:
Beyond Cognition and Emotion …
Matthias RAUTERBERG
Eindhoven University of Technology – TU/e
The Netherlands
2013
Interaction Paradigms in Computing
Cultural computing
Cross cultural-interaction
Social computing
Community-mediated-interaction
Cooperative computing
Computer-mediated-interaction
Personal computing
Man-machine-interaction
1960
© Matthias Rauterberg, 2013
1980
2000
Eindhoven University of Technology
2020
time
2/27
What is Culture?
Culture is the integration pattern of human behavior that includes
- attitudes,
- norms,
- values,
- beliefs,
- actions,
- communications and language
- institutions of a race, ethnic, religious and/or social group.
The word culture comes from the Latin root colere (to inhabit, to cultivate, or to honor). In
general, it refers to human activity; different definitions of culture reflect different theories for
understanding, or criteria for valuing, human activity. Anthropologists use the term to refer to
the universal human capacity to classify experiences, and to encode and communicate them
symbolically. They regard this capacity as a defining feature of the genus Homo.
REF: Rauterberg M. (2006). From personal to cultural computing: how to assess a cultural experience. In: G. Kempter & P. von Hellberg (eds.)
uDayIV--Information nutzbar machen (pp. 13-21). Lengerich: Pabst Science Publisher.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
3/27
Cultural Computing:
Attitudes
Norms
Values
Beliefs
Etc.
© Matthias Rauterberg, 2013
conscious
unconscious
Eindhoven University of Technology
Attitudes
Norms
Values
Beliefs
Etc.
4/27
REF: Salem B., Nakatsu R., Rauterberg M. (2009). Kansei experience: Aesthetic, emotions and inner balance. International Journal on Cognitive Intelligence and
Natural Intelligence, vol. 3, no. 2, pp. 18-36.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
5/27
Daniel Kahneman
Map of Bounded Rationality: A Perspective on Intuitive Judgement and Choice .
Nobel Prize Lecture, 8 December 2002 [PDF]
© Matthias Rauterberg, 2013
Eindhoven University of Technology
6/27
Sandy PENTLAND
© Matthias Rauterberg, 2013
Eindhoven University of Technology
7/27
* propositional representation
* synchronization via global workspace
phenomenal consciousness
* internal processing
verbal behaviour
nonverbal
behaviour
unconsciousness
* parallel processing in modular subsystems
multi modal actuator output
consciousness
multi modal sensory input
REF: Hofmann, W., & Wilson, T. (2010). Consciousness, introspection, and the adaptive unconscious. In B. Gawronski & B. K. Payne
(Eds.), Handbook of Implicit Social Cognition: Measurement, Theory, and Applications (pp. 197-215). New York: Guilford Press.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
8/27
After 1910 the discoveries of
Carl Gustav JUNG
about the collective unconscious and the related archetypes were challenging.
Jung dreamt a great deal about the dead, the land of the dead, and the rising of the dead. These
represented the unconscious itself -- not the "little" personal unconscious that Freud made such a
big deal out of, but a new collective unconscious of humanity itself, an unconscious that could
contain all the dead, not just our personal ghosts. Jung began to see the mentally ill as people who
are haunted by these ghosts, in an age where no-one is supposed to even believe in them. If we
could only recapture our mythologies, we would understand these ghosts, become comfortable with
the dead, and heal our mental illnesses.
(1875-1961)
© Matthias Rauterberg, 2013
Eindhoven University of Technology
9/27
© Matthias Rauterberg, 2013
Eindhoven University of Technology
10/27
How to design for the unconscious?
A collaborative project between two PhD students
Leonid IVONIN
•
•
•
Processing of
physiological signals
and application of data
mining methods
Shared activities
• Generation of ideas
•
Identification, selection,
and preparation of
audiovisual stimuli for
elicitation of psychological
states
•
Selection of appropriate
questionnaires
•
Presenting application
(ArcheBoard)
• Statistical analysis
• Writing of articles
Development of
technical infrastructure
for the experiments
Sensing application
(ArcheSense)
© Matthias Rauterberg, 2013
Huang-Ming CHANG
Eindhoven University of Technology
11/27
The State of the Art
Mapping
2.Elicitation 3.Recognition
Affective
Stimuli
Emotional
Responses
1.Explicit Emotions
(e.g. joy, anger, sadness, etc.)
© Matthias Rauterberg, 2013
Eindhoven University of Technology
12/27
The State of the Art
Mapping
Elicitation
Recognition
Smile
The emotion of Joy
© Matthias Rauterberg, 2013
Eindhoven University of Technology
13/27
Problem of Current States
What induces emotions?
• Perceptual Quality
Black, White
• Physical Content
Appraisal
Emotion
Dove
• Symbolic meaning
Peace, Spirit, etc.
• Barrett, L. F. (2012). Emotions are real. Emotion, 12(3), 413–29.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
14/27
Study 1 - method
Hypothesis: Psychological states related to archetypal stimuli would lead to different
patterns of physiological activations.





Number of participants:
• 34
Two types of stimuli:
• Visual (pictures)
• Auditory (sounds)
Five categories of stimuli:
• Archetypal
• Positive-relaxing
• Positive-arousing
• Neutral
• Negative
Six stimuli per category
Physiological data:
• Heart Rate
• Galvanic Skin Response
• Skin Temperature
Examples of visual stimuli
The findings indicated a significant relationship between the categories of stimuli
(including archetypal) and physiological signals.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
15/27
Study 2 - method
Hypothesis: Various archetypal experiences lead to recognizable patterns of physiological
activations that could be differentiated using computational intelligence algorithms.






Number of participants:
• 36
Two types of stimuli:
• Film clips with explicit emotions
• Film clips with archetypal experiences
Five categories of explicit stimuli:
• Neutral
• Amusement
• Fear
• Joy
• Sadness
Physiological data:
• ECG (HR + HRV)
• Skin conductance (response + level)
• Respiration
• Skin temperature
Duration of each stimulus:
Approximately 5 minutes
Self-report ratings after every stimulus
© Matthias Rauterberg, 2013
 Eight categories of archetypal stimuli:
• Anima
• Animus
• Hero departure
• Hero initiation
• Hero return
• Mentor
• Mother
• Shadow
Eindhoven University of Technology
16/27
Study 2 - results
Dynamic patterns of the heart rate responses of the participants to the film clips presentations.
The mean values and 95% confidence intervals of the HR are represented with the bold
lines and the vertical bars for each of the psychological condition.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
17/27
Study 2 - results
Classification Performance Achieved with Different Methods
kNN
SVM
Naïve Bayes
LDA
Archetypal experiences
Classification rate
78
82
85.5
83
Cross-validated
classification rate
74
75.5
79.5
79.5
Explicit emotions
Classification rate
77.6
75.2
78.4
77.6
Cross-validated
classification rate
72
71.2
74.4
74.4
Archetypal experiences and explicit emotions
© Matthias Rauterberg, 2013
Classification rate
62.8
55.7
72.6
69.8
Cross-validated
classification rate
49.8
68.3
57.2
61.2
Eindhoven University of Technology
18/27
ArcheSense tool
ArcheSense is a tool for evaluation of human experience with
products or media based on physiological data of people.
A tool for
evaluation of
unconscious
human
experience
More information about ArcheSense can be found at http://hxresearch.org.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
19/27
Study 3 - method
Hypothesis: Various archetypal experiences lead to recognizable patterns of physiological
activations that could be differentiated using computational intelligence algorithms.







Number of subjects:
• 23
Two types of stimuli:
• Film clips with explicit emotions
• Film clips with archetypal experiences
Five categories of explicit stimuli:
• Neutral
• Active-pleasant
• Active-unpleasant
• Passive-pleasant
• Passive-unpleasant
Physiological data:
• ECG (HR + HRV)
• Skin conductance (response + level)
Duration of each stimulus:
• Approximately 1 minute
Number of stimuli per category: 3
Self-report ratings after every stimulus
© Matthias Rauterberg, 2013
 Seven categories of archetypal stimuli:
• Anima
• Hero departure
• Hero initiation
• Hero rebirth
• Hero return
• Mentor
• Mother
• Shadow
Eindhoven University of Technology
20/27
© Matthias Rauterberg, 2013
21/6
Reborn from Fire
and Thunder Strom
Rain drops down
“God is in the
rain.”
© Matthias Rauterberg, 2013
Reborn from Fire
Eindhoven University of Technology
22/27
Study 3 - results
Comparison of the classification accuracy achieved using the self-report questionnaires
and the physiological data (between-subject classification).
Categories of the film clips
Number
of states
Self-reports
Physiological data
Anima, hero departure, hero initiation, hero rebirth, hero
return, mentor, shadow
7
28.0
36.7
Anima, hero departure, mentor, shadow
4
42.0
53.3
Anima, hero initiation, mentor, shadow
4
43.1
57.1
Anima, hero rebirth, mentor, shadow
Anima, hero return, mentor shadow
4
4
38.4
40.6
52.9
56.1
Active-pleasant, active-unpleasant, neutral, passivepleasant, passive-unpleasant
5
50.4
50.7
Active-unpleasant, neutral, passive-pleasant, passiveunpleasant
4
64.9
57.2
Archetypes
Explicit
emotions
Classification methods: k-nearest neighborhood (kNN), support vector machine
(SVM), naïve Bayes, linear discriminant analysis (LDA), and Adaptive Boosting with
decision trees (AdaBoost). Only the best accuracy is reported.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
23/27
Overview
Study #1:
•
•
•
•
•
•
‘Exploratory’ study
Pictures and sounds (presentation for 6 sec)
1 archetype, 4 explicit emotions
Between-subject design
Statistically significant results
23.3% classification accuracy (5 classes)
Study #2:
•
•
•
•
•
•
Film clips (length 5 min)
8 archetypes, 5 explicit emotions
1 stimuli per class
Between-subject design
Statistically significant results
79.5% classification accuracy (8 classes)
Study #3:
•
•
•
•
•
•
Film clips (length 1 min)
7 archetypes, 5 explicit emotions
3 stimuli per class
Between-subject and within-subject designs
Statistically significant results
57.1% classification accuracy (4 classes, between-subject),
70.3% classification accuracy (7 classes, within-subject)
© Matthias Rauterberg, 2013
Eindhoven University of Technology
24/27
Performance of Predictive Models
Classification Rate in General
Recognition
Technique
Affective Stimuli
Archetypal
Explicit Emotions
Symbols
Self-report Data
(Conscious)
Poor
Physiological Data
(Unconscious)
Good
<
=
Superb
Good
State of the Art
Emotion toward
Archetypal Symbols
© Matthias Rauterberg, 2013
Implicit Emotion
Eindhoven University of Technology
25/27
Three main conclusions can be drawn:
(1) Conscious and unconscious reactions are different.
(2) The unconscious is sensitive to archetypes.
(3) The unconsciousness is at least important as the consciousness.
© Matthias Rauterberg, 2013
Eindhoven University of Technology
26/27
Thank you for your attention.
A door goes open to a new world…
© Matthias Rauterberg, 2013
Eindhoven University of Technology
27/27
Publications
Downloads at http://www.idemployee.id.tue.nl/g.w.m.rauterberg/references.html#P
Journals
1.
Ivonin, L., Chang, H.-M., Chen, W., & Rauterberg, M. (2013). Unconscious emotions: Quantifying and
logging something we are not aware of. Personal and Ubiquitous Computing, 17(4), 663–673.
Conference
1.
2.
3.
4.
5.
6.
Chang, H.-M., Ivonin, L., Chen, W., & Rauterberg, M. (2011). Lifelogging for hidden minds: Interacting
unconsciously. In J. C. Anacleto, S. Fels, N. Graham, B. Kapralos, M. S. El-Nasr, & K. Stanley (Eds.),
Entertainment Computing - ICEC 2011 (LNCS, vol. 6972) (pp. 411–414). Springer Berlin Heidelberg.
Chang, H.-M., Ivonin, L., Diaz, M., Catala, A., Chen, W., & Rauterberg, M. (2013). Experience the world
with archetypal symbols: A new form of aesthetics. In N. Streitz & C. Stephanidis (Eds.), DAPI/HCII
2013 (LNCS vol. 8028) (pp. 205–214). Springer.
Chang, H.-M., Ivonin, L., Chen, W., & Rauterberg, M. (2013). Feeling something without knowing why:
Measuring emotions toward archetypal contents. INTETAIN’13 Intelligent Technologies for Interactive
Entertainment ( In press). Mons, Belgium.
Chang, H.-M., Ivonin, L., Diaz, M., Catala, A., Chen, W., & Rauterberg, M. (2013). What do we feel
about archetypes: self-reports and physiological signals. EUSIPCO’13 European Signal Processing
Conference ( In press). Marrakech, Morocco.
Ivonin, L., Chang, H.-M., Chen, W., & Rauterberg, M. (2012). A new representation of emotion in
affective computing. In J. Jia (Ed.), Proceeding of 2012 International Conference on Affective
Computing and Intelligent Interaction (Lecture Notes in Information Technology Vol. 10) (pp. 337–343).
Taipei, Taiwan: Information Engineering Research Institute.
Ivonin, L., Chang, H.-M., Chen, W., & Rauterberg, M. (2013). Automatic recognition of the unconscious
reactions from physiological signals. In A. Holzinger (Ed.), SouthCHI 2013 (LNCS Vol. 7946) (pp. 16–
35). Springer Berlin Heidelberg.
Publications
Magazines
1.
Ivonin, L.: Measurement and interpretation of consumers’ experiences in neuromarketing.
Neuromarketing Theory and Practice. Issue 5 (April), 20-21 (2013)
Other
1.
2.
3.
Ivonin, L., Chang, H.-M., Chen, W., Rauterberg, M.: Measuring archetypal experiences with
physiological sensors. SPIE Newsroom. Online, DOI: 10.1117/2.1201301.004669 (2013)
Ivonin, L., Chen, W., Rauterberg, M.: Narratives of Unconscious: Emotion Recognition from
Autonomic Reaction Patterns. ICE Summer School Report (2012)
Ivonin, L., Chen, W., Rauterberg, M.: Technology for Understanding Human Mind and
Behavior. ICE Summer School Report (2011)
Submissions under review
Journals
1.
2.
3.
4.
Chang, H.-M., Ivonin, L., Chen, W., & Rauterberg, M. (2013). From symbolic meanings to
emotions: A new strategy for selecting affective stimuli to discover unknown emotions.
Psychological Reports, Under review.
Chang, H.-M., Ivonin, L., Chen, W., & Rauterberg, M. (2013). From mythology to psychology:
Identifying archetypal symbols in movies. Technoetic Arts, (in press).
Ivonin, L., Chang, H.-M., Diaz, M., Catala, A., Chen, W., & Rauterberg, M. (2013). Observing
archetypal experiences of consumers with physiological measurements. Journal of Consumer
Psychology, Under review.
Ivonin, L., Chang, H.-M., Diaz, M., Catala, A., Chen, W., & Rauterberg, M. (2013). Beyond
cognition and affect: sensing the unconscious. Behaviour and Information Technology, Under
review.
Conference
1.
Chang, H.-M., Ivonin, L., Diaz, M., Catala, A., Chen, W., & Rauterberg, M. (2013). Unspoken
emotions in myths: archetypal symbolism as a new resource of emotional design. DPPI‘13
Designing Pleasurable Products and Interfaces. (Under review).