Outline:
Game User Experience Evaluations
 Case Study 1: NBA Live 10
 Case Study 2: NHL 11
 Take Away
 Q&A

Veronica Zammitto
Game User Experience Evaluation
Common Current Techniques
New Techniques
Qualitative Methods:
• Interview
• Focus Groups
• Think Aloud
• Survey
Quantitative Methods:
• Psychophysiological signals
(‘biometrics’)
• Eye Tracking
• Telemetry
• Subjective information
interpreted by an expert.
• Answers “Why” questions.
• Measureable, objective,
continuous information.
• Answers “What” questions
Veronica Zammitto
Mixed Method Approach for
Evaluating Sports Games

Triangulating:
Eye Tracking
• In-depth understanding
of User Experience (UX).
• Support design decisions.
Interviews
Survey
UX
Telemetry
Psychophysiology
Veronica Zammitto
Veronica Zammitto
Veronica Zammitto
Veronica Zammitto
Case Study 1: NBA Live10

NBA gameplay issues  to identify:
 successful and unsuccessful
gameplay aspects.
 emotional profile of the player
 engagement and emotions
 attentional focus

UX throughout the game, and for certain events.
Veronica Zammitto
Eye Tracking
Hardware + software  X, Y on screen
 Tracking users’ gaze can reveal the
player’s focus.
 DIYS, ~US$ 4,000 to 80,000

Veronica Zammitto
Veronica Zammitto
Eye Tracking

By using ET we can identify where players’
attention is.




Fixation
Saccades
Gaze Movement
Patterns
Veronica Zammitto
Aspects
Attention
Play Style 2
70 % on Core Gameplay Elements
(ball carrier, court assessment)
Time
Play Style 1
50 % on Core Gameplay
Elements (ball carrier, court
assessment)
50 % on non-core Gameplay
Elements (audience, players on
the bench, coaches)
Clock is not checked often.
Focus more on core gameplay
elements during the last minute.
Facts
Focus more on core gameplay
elements during the last minute
(might be missed)
Low or lack of interest.
Medium
Mainly use simple pass.
Almost no use of play calling.
Need to be provided with
strategies
Need of positive reinforcements
from coach and teammates.
Effective, looking mainly at icons
and logos.
Medium-high
Fair use of direct-pass.
Fair use of play calling.
Applies strategies in his games.
Performance
In-game Adv
30 % on non-core Gameplay
Elements (audience, players on the
bench, coaches)
Clock is checked often.
Medium interest.
Positive reinforcement from players
at the bench and coach.
Effective, looking mainly at icons and
logos
Veronica Zammitto
Aspects
Attention
Play Style 1
Play Style 2
50 % on Core Gameplay Elements 70 % on Core Gameplay Elements
(ball carrier, court assessment)
(ball carrier, court assessment)
Time
50 % on non- Core Gameplay
Elements (audience, players on
the bench, coaches)
Clock is not checked often.
30 % on non- Core Gameplay
Elements (audience, players on the
bench, coaches)
Clock is checked often.
Focus more on core gameplay
elements during the last minute.
Facts
Focus more on core gameplay
elements during the last minute
(might be missed)
Low or lack of interest.
Medium
Mainly use simple pass.
Almost no use of play calling.
Need to be provided with
strategies
Need of positive reinforcements
from coach and teammates.
Effective, looking mainly at icons
and logos.
Medium-high
Fair use of direct-pass.
Fair use of play calling.
Applies strategies in his games.
Performance
In-game Adv
Medium interest.
Positive reinforcement from players at
the bench and coach.
Effective, looking mainly at icons and
logos
Veronica Zammitto
Aspects
Attention
Play Style 2
70 % on Core Gameplay Elements
(ball carrier, court assessment)
Time
Play Style 1
50 % on Core Gameplay
Elements (ball carrier, court
assessment)
50 % on non- Core Gameplay
Elements (audience, players on
the bench, coaches)
Clock is not checked often.
Focus more on core gameplay
elements during the last minute.
Facts
Focus more on core gameplay
elements during the last minute
(might be missed)
Low or lack of interest.
Medium
Mainly use simple pass.
Almost no use of play calling.
Need to be provided with
strategies
Need of positive reinforcements
from coach and teammates.
Effective, looking mainly at icons
and logos.
Medium-high
Fair use of direct-pass.
Fair use of play calling.
Applies strategies in his games.
Performance
In-game Adv
30 % on non- Core Gameplay
Elements (audience, players on the
bench, coaches)
Clock is checked often.
Medium interest.
Positive reinforcement from players
at the bench and coach.
Effective, looking mainly at icons and
logos
Veronica Zammitto
Aspects
Attention
Play Style 2
70 % on Core Gameplay Elements
(ball carrier, court assessment)
Time
Play Style 1
50 % on Core Gameplay
Elements (ball carrier, court
assessment)
50 % on non- Core Gameplay
Elements (audience, players on
the bench, coaches)
Clock is not checked often.
Focus more on core gameplay
elements during the last minute.
Facts
Focus more on core gameplay
elements during the last minute
(might be missed)
Low or lack of interest.
Medium
Mainly use simple pass.
Almost no use of play calling.
Need to be provided with
strategies
Need of positive reinforcements
from coach and teammates.
Effective, looking mainly at icons
and logos.
Medium-high
Fair use of direct-pass.
Fair use of play calling.
Applies strategies in his games.
Performance
In-game Adv
30 % on non- Core Gameplay
Elements (audience, players on the
bench, coaches)
Clock is checked often.
Medium interest.
Positive reinforcement from players
at the bench and coach.
Effective, looking mainly at icons and
logos
Veronica Zammitto
Play Styles video
Veronica Zammitto
Triangulating Eye Tracking & Survey
Knowledge of
the rules of
basketball
Knowledge of
the NBA Live
title
Gamer type
0%
80 %
20 %
80%
Play Style 1
- expert
- Intermediate
- The Basics
- very familiar.
20% - familiar.
40 % - Hardcore
60 % - Casual
Main platform 80 % - Console
use
20 % - PC
Games bought
12.2
per year
An intermediate knowledge of the rules of
basketball but very familiar with the NBA
Live franchise.
Profile
Summary
Leaning towards casual player who plays
games a couple times per week, and prefers
FPS.
Play Style 2
60 % - Expert
40 % - Intermediate
0% - The Basics
80% - very familiar.
20% - familiar.
60
40
80
20
%
%
%
%
-
Hardcore
Casual
Console
PC
7.4
Expert knowledge of the rule of basketball
and very familiar with the NBA Live
franchise.
Leaning towards hardcore players who play
video games everyday but across many
genres.
Veronica Zammitto
Telemetry
Lay Up
Dunk
Pass
Call Time Out
Switch Players
Steal

Hooks in the game engine that flag and
time stamp pre-defined events.

Players’ in-game behavior



Statistical analysis
Visualizations
Machine learning algorithms
Veronica Zammitto
Events Performed
by Players
Through Time
in NBA 10
Veronica Zammitto
Events Performed by Players Through Time
in NBA 10
Veronica Zammitto
Passes sent by players
Veronica Zammitto
Players’ Scoring Location
58.6 %
91.4 % of
the shots
Veronica Zammitto
AI Scoring Location
75 %
Veronica Zammitto
What Went Right – NBA study

Better understanding of :




The new techniques were proven to provide useful
data to development.



players styles and demographics.
In-game behavior
Identification of emotions (next slides)
Rethink the role of game elements. I.e., coach.
Create tutorials for court observation based on eye tracking
data
Worthy of further investment to continue with studies
Veronica Zammitto
What Went Wrong

Large scope of the NBA study




Impacted synchronization of the usability study
with production’s delivery schedule.
The study should have been subdivided into mini
assessments to achieve a quicker turn around.
Low involvement of production in the project.
Manual coding:

Time consuming.
Veronica Zammitto
Case Study 2: NHL 11

Same techniques used for NBA

Adjustments from lessons learnt:

Narrower focus: “Game Presentation”

Front-End Visualizations (Overlays).


Cut Scenes (NIS):


I.e.: Do players look at information provided in the UI?
Are they watched or skipped?
High involvement with the development team


Meetings with development for a ‘statement of work’.
Iterative process with development

Helps to define the root of usability questions.
Veronica Zammitto
Front End: Overlays
Scoreboard
Faceoff
Line Change-Strategy
- Player
Penalty
Line Change – Strategy
- AI
Offside Warning
Fight Stamina AI
Fight Stamina Player
Fight Controls
Icing Warning
Pulling Home Goalie
Player's Name -Overlay
Scouting Report
End of Period Stats
Veronica Zammitto
Scoreboard
Faceoff Overlay
Offside Warning
Icing Warning
Penalty Overlay
Line ChangeStrategy Player
Overlay
Line ChangeStrategy AI
Overlay
Fight-Controls
Fight-Stamina
Player
Fight-Stamina AI
Player’s Name
Overlay
Scouting Report
End of Period
Stats
Veronica Zammitto
Scoreboard
Faceoff Overlay
Offside Warning
Icing Warning
Penalty Overlay
Line ChangeStrategy Player
Overlay
Line ChangeStrategy AI
Overlay
Fight-Controls
Fight-Stamina
Player
Fight-Stamina AI
Player’s Name
Overlay
Scouting Report
End of Period
Stats
Veronica Zammitto
Scoreboard
Faceoff Overlay
Offside Warning
Icing Warning
Penalty Overlay
Line ChangeStrategy Player
Overlay
Line ChangeStrategy AI
Overlay
Fight-Controls
Fight-Stamina
Player
Fight-Stamina AI
Player’s Name
Overlay
Scouting Report
End of Period
Stats
Veronica Zammitto
Scoreboard
Faceoff Overlay
Offside Warning
Icing Warning
Penalty Overlay
Line ChangeStrategy Player
Overlay
Line ChangeStrategy AI
Overlay
Fight-Controls
Fight-Stamina
Player
Fight-Stamina AI
Player’s Name
Overlay
Scouting Report
End of Period
Stats
Veronica Zammitto
Overlays Distribution
1st Period
Frequency
725
Percent
30.2
2nd Period
796
33.2
3rd Period
842
35.1
OT
Total
38
1.6
2401
100.0
Frequency
Fight
Percent
14
.6
Gameplay
2038
84.9
NIS
349
14.5
Total
2401
100.0
Veronica Zammitto
Percent
Ignored
75.7
Observed
24.3
Total
100.0
Only ¼ of the overlays are actually observed,
the other 75% are ignored.
Veronica Zammitto
Percentage of Observed and Ignored Overlays
during different game sections.
Quality and sensitive information that helps the player
has more chances to be looked during NISes.
Veronica Zammitto
Overlay “Map” in NHL 11
with Observed and Ignored proportions
Veronica Zammitto
Veronica Zammitto
Overlay
End of Period Stats
Observation Count Observation Length
(avg. visits per
(avg. in seconds per
appearance)
appearance)
6.36
2.43
Scouting report
4.09
1.71
Faceoff-Overlay
2.69
0.2
Icing Warning
1.67
0.06
Offside Warning
2.54
0.19
Penalty Overlay
2.59
0.21
Pulling Home Goalie
5.5
0.28
Fight Controls
3.25
0.95
Fight Stamina AI
1.5
0.28
Fight Stamina Player
3.25
0.23
Line Change-Strat - AI
1.86
0.26
Line Change-Strat - Player
3.27
0.36
Player's Name - Overlay
3.53
0.78
How people observed
the overlays.
Veronica Zammitto
Non-Interactive Sequence (NIS)
Event
NISes
Scripts - Subsets
Veronica Zammitto
NIS
Percentage Average per
match
Watched
37.15
11.03
Skipped
62.85
18.67
100
29.7
Total
NIS
Canned
Context Sensitive
Replay
Percentage
16.29
37.64
46.07
% watched
31.72
47.16
30.98
NIS’ Map
% skipped
68.28
52.84
69.02
Veronica Zammitto
NISes by event and type
Veronica Zammitto
The effectiveness of NIS is a combination of its type
(Canned, Context Sensitive, and Replay), the event that
triggers the sequence, and its frequency.
Veronica Zammitto
Psychophysiology

Infer emotions from
physiological data.


Arousal: engagement,
excitement, magnitude of
emotions.
Valence: positive (fun) or
negative (frustration)
(Biometrics)
arousal
Frustrating
Exciting
Emotional
Valence
Boring
Fun
Veronica Zammitto
Galvanic Skin Response (GSR)

Psychological arousal.

Skin’s conductance
increases when a person
becomes excited, stressed,
or anxious.
Veronica Zammitto
Veronica Zammitto
Electromyography (EMG)

Emotional valence.

Sensors capture and amplify
muscles’ contractions.

Facial muscles:


Smiling (zygomatic) = Positive emotions
Frowning (corrugator) = Negative emotions (or)
Cognitive load
Veronica Zammitto
Facial EMG
Veronica Zammitto
Veronica Zammitto
Player’s positive emotional reaction when scoring in NBA Live 10
Veronica Zammitto
Isomorphism
between telemetry
and GSR
Veronica Zammitto
Emotional Profiling of NHL 11
Arousal =
exciting,
engagement.
Veronica Zammitto
Levels of Arousal
in NHL 11: Tendency of
increasing arousal
throughout the game.
Veronica Zammitto
Player 2 vs AI 0
Player 6 vs AI 2
“I had a good game, but I should
have readjusted the AI level.
To me it’s also a challenge to score a
lot of goals when I know I’m going
to win, then it’s how many.
So, you established your own
achievement.
Yes, I wanted to get 6 at least by
the end of the game.”
Veronica Zammitto
NHL 11:
Arousal
profile
NHL’s Arousal Profile
Veronica Zammitto
Arousal-Valence Space
Arousal =
excitement
Valence =


positive
emotions
Negative
emotions
Veronica Zammitto
Veronica Zammitto
Veronica Zammitto
Veronica Zammitto
Veronica Zammitto
Take Away

New possibilities in user experience, mixed method with:
 Eye-tracking
 Biometrics
 Telemetry

Identifying the concrete information, cross-references.
 Emotions
 Engagement

 Interview
 Survey
 Attention
 In-game behavior
Information that supports design decisions.
Veronica Zammitto
Thanks!

Questions?
Veronica Zammitto