Research Methods –
Measuring User Experience
What might we measure in
relation to user experience?
Measures of User Experience
•
•
•
•
Experience of a specific emotion
Experience of a type of emotional response
Experience of a type of pleasure
Experience of “flow state”
Lazzaro: Four Keys to More
Emotion without Story
•
•
•
•
Hard Fun
Easy Fun
Serious Fun
People Fun
• Emotions: fear,
surprise, disgust,
naches/kvell, fiero,
schadenfreude,
wonder
Frome – Game Generated
Emotion
• Game Emotions
– Emotions of competition
• Narrative emotions
– Emotions from engaging with artwork
• Artefact emotions
– Emotions of aesthetic evaluation
• Ecological Emotions
– Response to what the artwork represents
Player Pleasures
LeBlanc
Crawford
Sensation
Fantasy/Exploration
Fantasy
Fantasy/Exp
Nose Thumbing
Narrative
Fantasy/Exp
Challenge
Proving Oneself
Exercise
Fellowship
Social Lubrication
Discovery
Fantasy/Exp
Expression
Need for Ack.
Submission
Exercise
Csikszentmihalyi - Flow
20:33
Neilsen – Usability Attributes
•
•
•
•
•
Learnability
Memorability
Efficiency
Errors and their severity
Subjective satisfaction
Juul & Norton
• Different from
productivity-based
software
• User challenge
/difficulty is
expected (sought
out)
• Challenge can be in
any aspect of the
games, including the
interface
Mandryke
Heuristic Evaluation
• Traditionally – examining
compliance with
recognised usability
principles
Neilsen – Usability Heuristics
•
•
•
•
•
•
•
•
•
•
Visibility of system status
Match between system and real world
User control and freedom
Consistency & standards
Error prevention
Error diagnosis and recovery
Recognition rather than recall
Flexibility & efficiency of use
Aesthetic and minimalist design
Help and documentation
Pinelle, Wong & Stach –
Game Usability
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Unpredictable / inconsistent response to user’s actions
Does not allow enough customization
Artificial intelligence problems
Mismatch between camera/view and action
Does not let user skip non-playable content
Clumsy input scheme
Difficult to control actions in the game
Does not provide enough information on game status
Does not provide adequate training and help
Command sequences are too complex
Visual representations are difficult to interpret
Response to user’s action not timely enough
Desurvire, Caplan & Toth – Heuristic
Evaluation for Playability (HEP)
•
•
•
•
Gameplay
Game story
Game mechanics
Game usability
Physiological Data
•
•
•
•
•
•
Galvanic skin response (GSR)
Respiration
Blood volume pulse (BVP)
Heart rate variation (HRV)
Electromyography (EMG)
Pupil dilation (PD)
• Arousal (GSR, Resp, BVP, HR)
• Mental effort (HRV, PD, EMG)
• Valance (EMG, HRV, PD)
Electrodermal Activity
• Galvanic skin response (GSR)
• Measures variation in electrodermal activity
between tonic baseline and phasic responses
• Uses eccrine sweat glands – palms of hands and
soles of feet
Cardiovascular
• Blood pressure – pressure needed to push blood
through circulatory system
• Blood Volume – how much blood is being pushed
around
• Heart rate – number of beats per minute
• Heart rate variability – change in heart rate
Muscles
• Electromyography – measure of muscle activity
– Brow
– Jaw
– Cheek
Arousal
•
•
•
•
Increases in galvanic skin response
Increased respiration
Decreased blood volume pulse
Increased heart rate
Mental Effort
•
•
•
•
•
Decreased heart rate variability
Greater pupil dilation
Increases in jaw clenching or brow-raising
Increased respiration rate
Decreased variability of respiration rate
Positive vs. Negative
Emotions
•
•
•
•
•
Valance of an emotion
Facial muscle analysis of brow and cheek
Heart rate,
Irregularity of respiration
Pupil diameter
Physiological Data Advantages
• Continuously collected to evaluate process not just
outcome
• Doesn’t interfere with experience
• High bandwidth – lots of data
• Can be used to infer underlying emotions
Physiological Data Disadvantages
•
•
•
•
High variability between individuals
Sensor error, interference and noise is prevealent
Requires baseline and normalization techniques
Can be invasive and impact performance
System Gathered Data
•
•
•
•
•
•
Time on task
Number/type of errors
Choices made
Number of times help system used
Number of time area/page visited
Any user input
Research Case Study:
Red-eye Removal
• Eastman Kodak – Removal of
red-eye defect from images in
direct print kiosks
Red-Eye: Pre-Artefact
• Research, evaluation/review of existing systems
• Scoping parameters for system design - range of
size of pupils with red-eye defect
• Negotiated system requirements and specifications
– Touch screen
– Screen resolution
– Amount of zoom
Red-Eye: Building Artefacts
• System captured data
– Time on task – how long to adjust each of three
– How many something was undone and what was undone
Red-Eye: User Testing
• 24 participants – Kodak factory workers variety of
ages and gender
• Three versions of the system – all participants used
all
• Variation in order that the versions were tested
• Used talk aloud – video recorded sessions
• Post test questionnaire – subjective/qualitative
Red-Eye: Data Analysis
•
•
•
•
Time on task analysis
Error rates/types
Speak aloud comment classification
Which did users say they preferred/found easiest
• Correlation between:
– Order used and user preference
– Order used and time on task
– Order used and speak aloud comment types
Sources
• http://www.nngroup.com/articles/ten-usabilityheuristics/
• http://www.useit.com/papers/heuristic/heuristic_ev
aluation.html
• http://userbehavioristics.com/downloads/usingheuri
stics.pdf
• http://userbehavioristics.com/downloads/usingheuri
stics.pdf
• http://mi-lab.org/wpcontent/blogs.dir/1/files/publications/uxInGames_K
oeffel_et_al.pdf
• Crawford (1982) “Why do people play games?”
in The Art of Computer Game Design. [online]
Available
at: http://www.scribd.com/doc/140200/ChrisCrawford-The-Art-of-Computer-Game-Design (Last
Accessed 31 January 2013)
• Frome, J. (2007) "Eight Ways Videogames Generate
Emotion" in Situated Play, Proceedings of DiGRA
2007 Conference. [Online] Available
at http://www.digra.org/dl/db/07311.25139.pdf(Last
Accessed 28/01/13)
• Lazzaro, N. (2004) Why we play videogames: Four
keys to more emotion without story. XEODesign.
[Online] Available
at: http://xeodesign.com/xeodesign_whyweplayga
mes.pdf (Last Accessed 7 Feb 2013)
• Pinelle, D., Wong, N., Stach, T. (2008) “Heuristic
Evaluation for Games: Usability Principles for Video
Game Design” in Proceedings of the ACM
Conference on Human Factors in Computing
Systems (CHI 2008), 1453-1462.
(http://hci.usask.ca/publications/2008/p1453pinelle.pdf)
• Isbister, K. & Schaffer, N. eds. (2008) Game usability:
advice from the experts for advancing the player
experience. London: Morgan Kaufmann.
• http://www.jesperjuul.net/text/easydifficult/
• http://armorgames.com/play/4309/this-is-the-onlylevel