Designing Tools
Enhance
Interactive
Experiences
& the development
process
Magy Seif El-Nasr
College of IST
Penn State University
Interactive Entertainment:
Industry Impact
Interactive Entertainment:
Industry Impact
Interactive Entertainment:
Industry Impact
Game industry profits:



2002: $1 billion more than movie industry
2004: EA reported $1.43 billion quarterly earning
2005: Nintendo $4 billion and expects to sell at
least 2 million Wiis in US alone
Importance of Game Engines + tools
Training
Entertainment
Devil May Cry II
Education
Virtual Aquarium,
NCSU
Leaders project, ICT
code
Tools
Assembly
.radix 16 ;all numbers hexadecimal
start: mov #41,@#8264 ;system variable
loop: jsr r4,print1
.asciz "HELLO WORLD! "
br loop
print1:jsr pc,@#1248 ;print string (ROM call)
rts r4
C#
System.Writeline (“Hello World”);
Learning and Computer literacy Impact
Several Researchers believe that
“video games provide an easy lead-in to
computer literacy” – Cassell 2000
High Schools students and Modding
Books on Modding for teens gaining popularity
Interactive Entertainment:
Motivation and Learning
“I want to learn how to make an awesome
video game. I would like to learn everything
about technology or at least more than I did”
- middle school girl
“I want to make my characters talk, build a
world, and make interesting stories”
- another middle school girl
Impact and Importance of the tools
Facilitate development: games, training,
health therapy application
Facilitate learning: media, computing literacy
Facilitate Creativity
Two thrusts of my research
Tools for building
engaging interactive
experiences
Game Engines +
Tools as learning
environments
Two thrusts of my research
Tools for building
engaging interactive
experiences
Game Engines +
Tools as learning
environments
Focus: visual design and character design
Based on cinematic and theatric theory
Using CI and AI techniques
My Research
Tools




Lighting
Camera (in progress)
Character (in progress)
Dance (in Progress)
Why Lighting?
Lighting is Everything

Mii
•
•
•
•
•
•
•
BS in Computer Science
Directed in Theatre
Graphics Design in Advertisement
MS in Computer Science
Studied Psychology and Neuroscience
PhD in Computer Science
2 years studied acting & lighting design at
Northwestern
Why Lighting?
Dramatic Tension
Visual Focus
Style
Feel of the space
Time of Day
Period
visibility
Lighting in Traditional Media
Game Lighting
Static Lighting Design
Manually setting light layout
(light maps)
x
Advantages:


realistic
Controllable
=
Images from Lightmaps (static shadowmaps) article written by Kurt Miller from:
http://www.flipcode.com/articles/article_lightmaps.shtml
Disadvantages:


Don’t adapt to variations
in the environment
Requires much memory
Image from Max Payne
Game Lighting
Static + Dynamic Lighting
Manually scripting
Lighting effects
Dynamic Character
Lighting
Real-time Shadows
Dynamic user control of
some lights
Game Lighting
Static + Dynamic Lighting
Game Lighting
Static + Dynamic Lighting
Advantages:
More realistic
Adaptable to change
Disadvantages:
Effects are scripted and rely on very careful
design
Restricted
Based on realism
Compositional Balancing is done at design time
Problems: No modulation
Screenshot from Mission 21
Devil May Cry
Saturation Graph
for Screenshot
Problems (Blade of Darkness): No
adaptation to game Play
Problem 1: unpredictability
Develop a lighting plot or setup based on:
Narrative configuration:


Story
Conflict/dramatic tension
Timing and dramatic progression
Physical Configuration:


Camera orientation
and position
Characters positions
and orientations
Requiring dynamic adaptation
depend
on user
Two Goals
Dynamic Adaptive Lighting:
Better gameplay
Facilitate new Interactive models
High-Level authoring of lighting:
Faster prototyping
Exploration of design space
Developing such as a
Design Tool is Hard
Why?
Technical Issues of Dynamic Lighting
Artistic control for style
low-level balancing of context and
lighting parameters but provide
continuity
ELE – Expressive Lighting Engine
A lighting system that:
Intelligently adjusts
lighting in real-time accommodate
context and effect
Based on cinematic
& theatric theory
Allow artist to control
lighting at a high-level
ELE
ELE
Artistic
Constraints
Previous
State
Current
State
WAMP (World Action
Message Protocol)
Allocation Subsystem
Game/Rendering
Engine
Angle Subsystem
LAMP (Lighting Action
Message Protocol)
Color Subsystem
The three subsystems: use optimization to find best solution
given context, desired effects, state, and artists’ constraints
ELE
ELE
Artistic
Constraints
Previous
State
Current
State
WAMP (World Action
Message Protocol)
Allocation Subsystem
Game/Rendering
Engine
Angle Subsystem
LAMP (Lighting Action
Message Protocol)
Color Subsystem
The three subsystems: use optimization to find best solution
given context, desired effects, state, and artists’ constraints
Automatic light allocation
find best allocation
given artistic
constraints:






Modeling
Depth
Visibility
Visual Continuity
Visual Focus
Low vs. high key
Algorithm for dynamic allocation
1.
2.
3.
Calculate visible area
Divide the zone into overlapping areas
Allocate a number of lights to areas,
given that p : L  A minimize:
popt 
arg max(vV ( p)  d D( p)  m M ( p)  vcVC ( p))
p
visibility
depth
modeling
Visual
continuity
Layout of lights
ELE
ELE
Artistic
Constraints
Previous
State
Current
State
WAMP (World Action
Message Protocol)
Allocation Subsystem
Game/Rendering
Engine
Angle Subsystem
LAMP (Lighting Action
Message Protocol)
Color Subsystem
The three subsystems: use optimization to find best solution
given context, desired effects, state, and artists’ constraints
Lighting angle Selection
Select azimuth, elevation angles, given
artistic constraints :





Visual Continuity
Motivation of direction
Visibility
Modeling
Mood
ELE finds best light angles to meet goals
Automating cinematic lighting design
Choosing key light azimuth angle
Optimizes:

cost(k , s, k , m) 
v (1  V (k , s))   k  k  m k  m  l min k  li

i
visual
mood
motivation
continuity
k is the key light angle, k- is angle previous frame
m is the mood angle desired, V is visibility
v is the cost of deviation from best visibility angle
- is the cost of visual continuity
l is the cost of deviation from realistic direction
m is the cost of deviation from ideal mood angle
visibility &
modeling
•
•
•
•
•
•
Evaluation of key light angle
visibility & modeling
Light
s
Nose
line
k
Camera
From: Millerson’s
The technique of lighting for television and Film, 1991
We described these rules as :
V (k , s)  sin(k ) cos(s)
ELE: Angle Subsystem
Artist's setting
120
Costs
100
80
60
40
20
Showing character’s facial
expressions and gestures
on
tr
as
t
C
oo
d
M
Vi
si
bi
lit
y
0
ELE: Angle Subsystem
Artist's settings
120
Costs
100
80
60
40
20
on
tr
as
t
C
oo
d
M
Showing character’s facial
expressions and gestures, high
tension
Vi
si
bi
lit
y
0
ELE: Angle Subsystem
Artist's settings
120
Costs
100
80
60
40
20
Emphasizing mood and
mystery
oo
d
M
on
tr
as
t
C
Mood Angle = side angle
Vi
si
bi
lit
y
0
ELE: Angle Subsystem
Artist's settings
120
Costs
100
80
60
40
20
Emphasizing mood
on
tr
as
t
C
oo
d
M
Mood Angle = side angle
Vi
si
bi
lit
y
0
ELE
ELE
Artistic
Constraints
Previous
State
Current
State
WAMP (World Action
Message Protocol)
Allocation Subsystem
Game/Rendering
Engine
Angle Subsystem
LAMP (Lighting Action
Message Protocol)
Color Subsystem
The three subsystems: use optimization to find best solution
given context, desired effects, state, and artists’ constraints
Choosing Colors
Color (Hue + saturation
+ intensity)
Compose colors for
different areas on the set
Choosing Colors
Adjust colors to accommodate desired artistic
constraints:
 Depth
 Dramatic Intensity
 Dramatic focus
 low vs. high key setting
 Specific author-suggested
Hue, Saturation, Lightness, color Warmth for
focus, non-focus, and background
Palette restrictions specifying style
+ maintain visual continuity and style
Choosing Colors
Optimize:
cost (c , c )  d  D (c )  d  
t
t 1
t
2
depth
c  contrast  (c )    
t
p( c ) 
t
2
contrast
Palette constraints
t
t 1
Visual
vc E (ci , ci ) 
i{ f ,n ,b}
continuity
t
I (ci ), Artist’s desired color
i{ f ,n ,b}
parameters


Choosing Colors
s  S (c )  si  
i
I (c ) 
t
i
2
t
i
h  H (c )  hi  
2
t
i
i
l  L(c )  li  
2
t
i
i
w W (c )  wi 
i
t
i
Saturation
Hue
Lightness
2
Warmth
Calculating Color Warmth
Based on warmth perception
Warm
Used linear fit to psychophysical data:
  R    0.008 
  
warmth  G     0.0006 
  B    0.0105

  
T
R
G   0.422
 
 B 
Cool
ELE: Color Subsystem
Lightness (NF)
Contrast (F – NF)
Warmth (NF)
for low tension, realistic colors
th
ar
m
W
C
Li
gh
tn
e
warm tones,
low color contrast
on
tr
as
t
90
80
70
60
50
40
30
20
10
0
ss
Ideal Values
Artist's setting
ELE: Color Subsystem
th
ar
m
W
C
on
tr
as
t
40
30
20
10
0
ss
for night scenes,
or emphasize character
80
70
60
50
Li
gh
tn
e
cool tones,
low color contrast
Ideal Values
Artist's setting
ELE: Color Subsystem
th
ar
m
W
ss
on
tr
as
t
C
for high tensions scenes
100
90
80
70
60
50
40
30
20
10
0
Li
gh
tn
e
warm,
high intensity contrast
Ideal Values
Artist's setting
Two Goals
Dynamic Adaptive Lighting:
Better gameplay
Facilitate new Interactive models
High-Level authoring of lighting:
Faster prototyping
Exploration of design space
Use Light as tension meter
Dynamic Intelligent
Cinematic Lighting
can make a difference in games
Demo (Dynamic Lighting)
Demo (Static Lighting)
Results of Pilot Study – during
CHI 2005
Non-players liked the visual representation
Liked representation of information through
lighting
Some players said: it made the game too easy
Some gamers said: lighting was disturbing
Adapting Difficulty of Spotting
enemies
Adaptive Visual Focus
can make a difference in games
Two Goals
Dynamic Adaptive Lighting:
Better gameplay
Facilitate new Interactive models
High-Level authoring of lighting:
Faster prototyping
Exploration of design space
Elevator
Elevator
EDISON Edge Detection and Image
SegmentatiON [Christoudias et al.
2002]
Mean-shift approach
[Comaniciu and
Meer 2002]
Design Galleries
[Marks et al. 1997]
User adjusting Lighting in Games like
The Sims or The Movies
A comment from an
educator:
“This can be used as
a way of promoting
Media Literacy”
Classes – Using the Tools
400-level interdisciplinary
course on Interactive
Narrative, tools:




C#
Flash
ELE
Interactive Narrative
Engine
Classes – Using the tools
400-level Game Design,
tools:



Unreal + ELE
Half Life 2
Torque
Classes – Exploring the theory
400-level Interdisciplinary
course on Design of
Immersive Environments
Future Directions
Tools for building
engaging interactive
experiences
More on lighting
Lighting: industry partners
Continue on Projects:



Camera
Character
Dance
Current Projects
Ambient Intelligence and Dance
designing tools to allow artists to dictate aesthetic
effect
Intelligent Systems
Pressure +
Physiological
Sensors
Current Projects
Comedia dell’arte in Second Life
Character, gesture, and camera tools
Virtual Actor
Training
Current Projects
Lockheed Martin
Lighting
visual design
camera
character
Living
Hypothesis
Future Directions
Game Engines +
Tools as learning
environments
Use more of the research
tools
Impact of the use of the
research tools
Specifically our work on
Games and Modding
Classes – Using Game Modding
Middle/high school Gaming
for Girls
Future Work:



Camera tool
Lighting
Character
In Conclusion
Conclusion
Game Industry is a growing industry
Game tools are important
Devil May Cry II
Leaders project, ICT
In my research I develop new tools to


Enhance design process
Enhance interactive experience
Application of these tools: Education,
Training, and other applications
Questions?
http://faculty.ist.psu.edu/
seifel-Nasr/
magy@ist.psu.edu
Modding and Learning
Transfer of Computer Science, Math, Art
Concepts
Different engines promote different concepts,
and require different pre-reqs
Increase self efficacy of high-school female
students
Materialize abstract concepts
Learn something about themselves
Different engines are popular among different
groups
Concepts Promoted thru Modding
Essential Concepts
 Assumptions
of engines need to
be understood, e.g.
Parallel processing in
Warcraft III

Concepts you need know to operate the
engine,
Co-ordinate system
Color, contrast, etc. to use ELE
Assignment-dependant concepts, e.g.

Character Modeling
Modding and Learning
Transfer of Computer Science, Math, Art
Concepts
Different engines promote different concepts,
and require different pre-reqs



Graphics: Vector Math, Geometry, Animation
Programming: Parallel processing, event
programming, Object Oriented Programming
Arts: Architecture Design, map design, visual
Composition, Lighting, camera movement, etc.
Game Projects involved
Project Management
Group Work
iteration design
Prototyping
Critique
Game Design concepts:



Mechanics
Rules
etc
What do students learn when they
are building games using the
current tools?