Using games to model and
recognize interactions
LREC 2012 Tutorial
Kostas Karpouzis
National Technical University of Athens
kkarpou@cs.ntua.gr
schedule
• User models for game play
–
–
–
–
Modelling prototypical player and non-player characters
Emotion and games: inducing and expressing emotions
User, environment and interaction context
Modelling and measuring player satisfaction
schedule
• Game-based corpora
–
–
–
–
Elements of captured user expressivity
Design issues in game-based corpora
Inducing emotion with games
Measuring behavioural, affective and cognitive aspects
during game play
schedule
• Emotion, affect and behaviour
– Available modalities in natural human-computer
interaction
– Your body (and face, hand, speech) is the controller
– Behavioural cues, non-verbal interaction
– What else can we recognize? What can’t we?
schedule
• Lessons learned
– CALLAS: capturing affect in arts and entertainment apps
– Serious games for observing and learning social skills: the
Siren project
• Games for corpora collection
– the platformer case: Super Mario Bros clone
– the FPS case: Cube 2
– the racing case: TORCS
yours truly
• Post-doc in Humaine Network of Excellence
• Since:
– CALLAS (emotion in arts and entertainment)
– Feelix Growing (affect-aware robots)
• Currently:
– Siren (serious games for conflict resolution in school
environments)
– ILearnRW (serious games for children with dyslexia and
dysorthographia)
why games?
• Toys provide our first human-“machine” interface!
• Games are everywhere these days!
– Computers, consoles/TV, mobile phones
– Browser and Facebook/Google+ do not require installation
– ‘Freemium’ business model attractive to both gamers and
developers
– Huge interest in industry and academia
• Lots of funding from EU projects 
why games?
• Constrained environment  easier to record
people
• Novel modalities: gestures, body movement,
speech (lexical and prosody)
– Ideally associated with what happen in the game
• (Possibly) novel interaction paradigms, i.e. not
WIMP
why games?
• The concept of Flow
– a state of concentration or complete absorption with the
activity at hand and the situation. It is a state in which
people are so involved in an activity that nothing else
seems to matter (Csikszentmihalyi,1990)
– “Being completely involved in an activity for its own sake.
The ego falls away. Time flies. Every action, movement,
and thought follows inevitably from the previous one, like
playing jazz. Your whole being is involved, and you're
using your skills to the utmost”
while in flow…
• Gamers are more expressive
• May show less (or no) social inhibition
– Spontaneous expressivity
• Include expressive modalities not usual in
other HCI contexts (e.g. gesturing)
• Positive or negative expressions may be
exaggerated
while in flow…
or
(if things go
wrong)…
while in flow…
a bit of theory
the quest for Flow (and fun)
a theory of fun
• Raph Koster
– lead designer of Ultima
Online
– creative director of Star Wars
Galaxies
– http://www.theoryoffun.com
/theoryoffun.pdf
– http://www.raphkoster.com
a theory of fun
• ‘all games are edutainment’
– ‘some games teach spatial relationships (e.g.
Tetris)’
– ‘some teach you to explore (Super Mario)’
– ‘some teach you how to aim (FPSs)’
– some teach you to share/cooperate (Farmville)
– ‘players seeking to advance in a game will always
try to optimize what they are doing’
a theory of fun
a theory of fun
a theory of fun
• ‘We talk so much about emergent gameplay,
non-linear storytelling, or about playerentered content. They’re all ways of increasing
the possibility space, making self-refreshing
puzzles’
• So, what is it that makes a game ‘fun’?
flow revisited
• the ‘holy grail’ of
game design
• just the right amount
of challenge
• making a game very
hardgamers quit
• making a game very
easygamers bored
flow revisited
• it’s not about the
graphics
• or the controller
• or the franchise (e.g.
sports games)
• just ask Rovio
– makers of Angry Birds
– $80M/yr, 600M dl’s
flow revisited
• ‘smart’ games adapt
to player skill and
engagement
• keeping them coming
back for more
• at the end of the
day…
generating content
• we want to make games
harder or easier to
match player skill
• predefined levels (e.g.
‘easy’ / ‘expert’)
• still we have to define
what ‘easy’ means
generating content
• game level as a
multi-parameter
function  produce
game content
procedurally
• e.g. number and size
of gaps, number of
opponents, etc.
generating content
• Java implementation
• Mario AI
Championship
• www.marioai.org
• cf. Yannakakis &
Togelius @ IEEE TAC
what about serious games?
• ‘the opposite of play
is not work, it’s
depression’
• Dr. Stuart Brown
• Serious Play 2008
(hosted @ TED.com)
you may want to watch…
• Jane McGonigal: Gaming can make a better
world, Feb 2010
you may want to watch…
• A gamer just before an ‘epic win’!
what about serious games?
• our first activity as
children
• ‘fun’ and ‘flow’ are a
given
what about serious games?
• our first activity as
children
• ‘fun’ and ‘flow’ are a
given
• best material to
teach social skills
what about serious games?
• our first activity as
children
• ‘fun’ and ‘flow’ are a
given
• best material to
teach social skills
• but schools fail to
capitalize on that
gamification
• the best one way to influence player behaviour
• include game design elements in non-game contexts
gamification
• image by Sebastian Deterding
• Bunchball white paper: http://info.bunchball.com/gamification-101/
gamification
• in Fourquare, users
earn points for
check-ins and other
activities
• leaderboards and
badge display
enhance competition
gamification
• replace ‘check-in’ with,
e.g., ‘recycle’
– gamification in the real
world
gamification
• replace ‘check-in’ with,
e.g., ‘recycle’
– gamification in the real
world
• what happens when
badges and rewards
are taken away?
– open research question
in a nutshell
• games provide challenge and fun to players
– or should be adapted to do so
• fun not always equal to entertainment
– the case of serious/learning games
• player experience: function of skill,
performance and challenge
user modelling
player personalities
• Richard Bartle
– co-creator of MUD1
(the first MUD)
• Bartle Test of Gamer
Psychology
– series of questions to
players of MMOs into
categories based on
gaming preferences
player personalities
player personalities
• Achievers: players who prefer to gain
points, levels, equipment and other
concrete measurements of succeeding
in a game
• Explorers: players who prefer
discovering areas, creating maps and
learning about hidden places
player personalities
• Socializers: players who choose to play
games for the social aspect, rather than
the actual game itself
• Killers: players who like to …‘club’ other
players
– They thrive on competition with other
players, and prefer fighting them to
scripted computer-controlled opponents.
player personalities
• Bartle Quotient totals 200% across all
categories, with no single category exceeding
100%
• A person may score "100% Killer, 50%
Socializer, 40% Achiever, 10% Explorer“
– indicating preference to fight people compared to
other aspects of gameplay
• but…
player personalities
• these are self-reported characteristics
– mostly refer to what players would prefer to do
and not necessarily what they actually do when
playing
• game genre-specific
– and different for single- and multi-player gaming
• offer little in terms of adaptation
– mostly refer to game mechanics and features
back to the drawing board
• what can we model?
– and how?
• definition of ‘affective computing’
– ‘affective Computing is computing that relates to,
arises from, or deliberately influences emotion or
other affective phenomena’ -- Roz Picard, 1995
– ‘a set of observable manifestations of a
subjectively experienced emotion’ -- MerriamWebster’s dictionary
observable manifestations
observable manifestations
hypothesis
• ‘shallow’ treatment
– i.e. not as far as ‘personality’, sticking to ‘affect’
• identify/track user reactions
– facial expressions and gestures, body movements
and stance, hand and body expressivity (for
whole-body interaction)
• relate those to events in the game
hypothesis
• ideally, we could identify the players’ stress
level (via the ‘observable manifestations’) and
their skill level (via their performance)
• and cluster those to identify player types
– for the particular game genre!
• or use them to adapt the game
– make it easier for players ‘in distress’
– or harder for players in the verge of boredom
hypothesis
• why bother with both affect and
performance?
• why are players standing still?
– is it flow (immersion) or boredom?
• or why do they move around?
– is it immersion (e.g. in a racing game) or lack of
engagement?
• remember: Flow  skill AND engagement
hypothesis
pitfalls!
• happy face  low stress?
– may indicate irony
• sad face  high stress  need to adapt?
– some serious games may need to induce negative
emotions to get things running
– e.g. in a conflict resolution game
– more on this later
we’ve covered affect;
what else is there?
• cognitive models (Gray, 2007)
– evaluate why players behave in the way that they do, or
conversely to control computer-driven AI (Funge, 1999)
• cultural model
– “collective programming of the mind which distinguishes
the members of one group or category of people from
another” (Hofstede, 1996)
– not necessarily related to player origin or descent (e.g.
sub-cultures)
we’ve covered affect;
what else is there?
• group models
– differences of equal treatment (favourability)
when interacting with individuals from the ingroup (i.e. the group the agent belongs to) or outgroup
we’ve covered affect;
what else is there?
• learner model
– students motivation strongly linked to learning (Malone
and Lepper, 1987)
– demographic information and personality  understand
and predict the student’s learning behavior
– demographics (gender), personality (Big 5: openness,
conscientiousness, extraversion, agreeableness,
neuroticism), goal orientation (performance based on
result or mastery on skill), and presence (involvement with
the system) (McQuiggan et al., 2010)
in a nutshell
• games provide an ideal medium to induce and
capture affective interactions
• well-designed games bring out different (and
valuable!) reactions from players
• gaming is a multi-faceted activity
– thus, player models are usually detailed
• player affect tells us a lot about the game
lessons learned
the CALLAS and Siren projects
capturing affect in arts
and entertainment
• The aim of the
CALLAS project was
to identify the role of
affect in artistic and
entertainment apps
• Close the affective
loop between user
and ‘machine’
your body is the controller(™)
your body is the controller(™)
• ‘Whole Body
Interaction’ is a novel,
affect-aware medium
– OK, MS Kinect admittedly
took it to the masses
image by Exertion Games Lab /
Floyd Mueller
• mainstream exertion
games have been
around since the 90s
(e.g. Konami's Dance
Dance Revolution)
your body is the controller(™)
• sometimes it’s not
about which gesture you
produce
– unless it carries affective
meaning (e.g. ‘thumbsup’)
• but about how you
produce it
– greet a stranger vs your
significant other
your body is the controller(™)
• Kinect (or simple image
processing techniques)
 track motion of
hands or whole body
• Caridakis, Castellano,
etc. study ‘expressivity
features’
– speed, spatial extend,
fluidity, etc.
your body is the controller(™)
• perceptual studies
indicated strong
correlation with human
annotators
– especially wrt activation
levels
• culturally dependent?
– initial studies verify
cultural clichés about
expressivity
conflict resolution games
• Siren aims to
produce a conflict
resolution serious
game
– for 10-14 y.o.
children
– in school
environments
conflict resolution games
conflict resolution games
The life cycle of conflict (Swanstrom and Weissmann, 2005)
conflict resolution games
• during escalation, negative emotions are present
• cannot use neg. emotions to indicate stress  adaptation
conflict resolution games
• rather, use estimated emotion to identify where players
are in this figure (which phase)
conflict resolution games
• and produce content to ‘push’ users towards descalation
• learning objective of the game!
conflict resolution games
• sensed affect can be used to identify player
performance
– i.e. whether players actually ‘move’ towards
resolving the conflict
• but which emotions are relevant?
• negative vs positive
• is that enough for all game genres?
the Siren database
• Super Mario revisited
• Idea: correlate facial
expressivity with
game events
– facial expressions,
cues (e.g. lip biting),
head movement
(expressivity, eye
gaze)
the Siren database
• self-reported
frustration and
engagement
– for pairs of games
• no direct (objective)
measurement of
difficulty level
the Siren database
• about 80 volunteers
• mostly students
– not necessarily game
players
• interesting results
– experienced players
move less
– men move less
the Siren database
• about 80 volunteers
• mostly students
– not necessarily game
players
• interesting results
– experienced players
move less
– men move less
– paper in ‘Multimodal
corpora’ workshop
the Siren database
• prototypical player
types?
– e.g. ‘Iceman’ who hardly
moves but performs
superbly
– or ‘dancers’ who ‘flow’
along with Super Mario’s
motion
• culture-related?
other game genres
• Cube 2 is an open
Source (ZLIB license)
First-Person Shooter
• capture game events
• create game content
on the fly
– http://sauerbraten.org
other game genres
• TORCS (The Open Racing
Car Simulator) is a portable
multi platform car racing
simulation
• competitions for
faster/smoother driving,
demolition derby!
– http://torcs.sourceforge.net
other game genres
• TORCS (The Open Racing
Car Simulator) is a portable
multi platform car racing
simulation
• competitions for
faster/smoother driving,
demolition derby!
– http://torcs.sourceforge.net
in a nutshell
•
•
•
•
•
player affect is genre-dependent
reflects many qualities from the user model
many open research questions
single- vs multi-player
easy to find people to play games
– yay!
• large AV datasets difficult to put together
– as always 
thank you!
Kostas Karpouzis
kkarpou@cs.ntua.gr