Musical Metacreation: Papers from the 2012 AIIDE Workshop
AAAI Technical Report WS-12-16
Mezzo: An Adaptive, Real-Time
Composition Program for Game Soundtracks
Daniel Brown
University of California at Santa Cruz
twitteringmachine2000@yahoo.com
that evoke the conditions and relationships of these
elements. Story states that occur in a game are musically
conveyed by employing or withholding normative musical
features. This creates various states of markedness, a
concept which is defined in semiotic terms as a valuation
given to difference (Hatten 1994). An unmarked state or
event is one that conveys normativity, while an unmarked
one conveys deviation from or lack of normativity. A
succession of musical sections that passes through varying
states of markedness and unmarkedness, producing various
trajectories of expectation and fulfillment, tension and
release, correlates with the sequence of episodes that
makes up a game story’s structure.
Mezzo uses harmonic tension and formal regularity as
its primary vehicles for musically conveying markedness;
it is constantly adjusting the values of these features in
order to express states of the game narrative. Motives are
associated with characters, and markedness with game
conditions. These two independent associations allow each
coupling of a motive with a level of markedness to be
interpreted as a pair of coordinates in a state space (a
“semiotic square”), where various regions of the space
correspond to different expressive musical qualities
(Grabócz 2009).
Certain patterns of melodic repetition combined with
harmonic function became conventionalized in the
Classical Era as normative forms, labeled the sentence,
period, and sequence (Caplin 1998, Schoenberg 1969).
These forms exist in the middleground of a musical work,
each comprising one or several phrase repetitions and one
or a small number of harmonic cadences. Each musical
form has a normative structure, and various ways in which
it can be deformed by introducing irregular amounts of
phrase repetition to make the form asymmetrical. Mezzo’s
expressive capability comes from the idea that there are
different perceptible levels of formal irregularity that can
be quantitatively measured, and that these different levels
convey different levels of markedness.
Abstract
Mezzo is a computer program designed that procedurally
writes Romantic-Era style music in real-time to accompany
computer games. Leitmotivs are associated with game
characters and elements, and mapped into various musical
forms. These forms are distinguished by different amounts
of harmonic tension and formal regularity, which lets them
musically convey various states of markedness which
correspond to states in the game story. Because the program
is not currently attached to any game or game engine,
“virtual” gameplays were been used to explore the
capabilities of the program; that is, videos of various game
traces were used as proxy examples. For each game trace,
Leitmotivs were input to be associated with characters and
game elements, and a set of ‘cues’ was written, consisting of
a set of time points at which a new set of game data would
be passed to Mezzo to reflect the action of the game trace.
Examples of music composed for one such game trace, a
scene from Red Dead Redemption, are given to illustrate the
various ways the program maps Leitmotivs into different
levels of musical markedness that correspond with the game
state.
Introduction
Mezzo is a computer program designed by the author that
procedurally writes Romantic-Era-style music in real time
to accompany computer games. It was motivated by the
desire for game music to be as rich and expressive as that
written for traditional media such as opera, ballet, or film,
while still being procedurally generated, and thus able to
adapt to a variety of dramatic situations. To do this, it
models deep theories of musical form and semiotics in
Classical and Romantic music. Characters and other
important game elements like props and environmental
features are given Leitmotivs, which are constantly
rearranged and developed throughout gameplay in ways
Copyright © 2012, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
68
traces were used as proxy examples, each ranging from
about six to 13 minutes long. Videos of scenes from three
games have so far been used: a level of Red Dead
Redemption, in which a cowboy must herd cattle and stop a
train robbery; a short episode from Star Wars: The Old
Republic, in which a Jedi Knight must decide whether or
not to give in to his romantic attraction to an alien; and a
walkthrough of the entire last level of Super Mario 3D,
where the protagonist Mario must rescue a princess held
captive by a monster atop a castle turret.2
For each game trace, a set of Leitmotivs was written for
each character and game element I considered significant,
which were then loaded into the program as MIDI files.
Each game trace was also given a set of harmonic
progressions taken from a work by a Romantic-Era
composer which seemed appropriate for the game: Liszt
for Red Dead Redemption (because of its pastoral
character), Chopin for Star Wars scene (because it’s a love
scene), and Wagner for Super Mario (because of its
overblown, mythological nature). Then a short set of
‘cues’ was written, consisting of a set of time points at
which a new set of game data would be passed to Mezzo.
To compose music for a certain state of a game, Mezzo
uses a number of different settings. Any characters and
elements associated with Leitmotivs which are involved in
the current scene are passed to the program, as well as the
characters’ states. Mezzo is designed to work with games
in which some normative state can be defined for a
character, be it measured by health or goal achievement or
some other interpretation, and the distance from this
normativity can be measured.
Another small set of states associated with the player’s
character is also passed to the system. These states are
related to the measure of normativity, but must also be
interpreted according to the character’s relation to other
characters and the game story. These states are defined as
stability (when the player is not currently facing a problem
to overcome), battling (when the player is in direct conflict
with another character who is present), questing (when the
player is trying to achieve some goal, but is not directly
faced with an opposing character), failing (when the
character has made some irreversible mistake, like getting
killed), succeeding (when the character has completely
overcome some problem, like destroying an opponent), and
introduction (when a character first appears and is not in
any of the other states).
Finally, features relating to the level of activity are used
by Mezzo to determine aspects of the musical texture, like
how energetic it will be, how much space will occur
Overview of the Compositional Process
The actual amount of musical information that must be
input by a user is very small. First, the Leitmotivs that will
be associated with characters and game elements must be
specified; these can be as long as the user desires, but
produce the most interesting outcomes when they are only
two to four measures long.
Second, the harmonic
vocabulary is given by inputting a small number of chord
progressions (10 or so), although these progressions are not
the ones that are actually used in Mezzo’s compositions.
The program extracts information regarding voice-leading
and levels of acoustic tension from the input progression,
and then constructs its own progressions during gameplay
(using a genetic algorithm) that are stylistically similar to
those that were input. This allows the user to establish a
general harmonic language––by taking progressions from a
particular composer’s work, say, such as Wagner or
Chopin––without having to author any specific expressive
details herself. The program takes care of that.
Composition in Mezzo is a two-step process: first build
forms, then deform them according to stochastic
constraints. Both of these processes generate expressive
features in the music being composed. The form-building
stage composes chord progressions with the appropriate
length, formal organization, cadence type, and amount of
harmonic tension, and builds forms by mapping
appropriate Leitmotivs and accompanying textures to them.
In the second stage, each form that has been composed is
mapped to a data structure that sets the stochastic
constraints on its formal regularity, determining which, if
any, formal sections will be repeated or omitted, and to
what extent. This stochastic model is a generalization of
the methods of deformation used by Classical and
Romantic composers, and attempts to maintain continuity
with their procedures while projecting them into a modern,
interactive context. Furthermore, it extends this concept to
function in the open-ended setting of a game. Each time a
form is stated, it will, with some probability, be organized
differently from previous times, and there will be no
pattern to the way the organization changes from statement
to statement. However, this randomness is controlled by
the stochastic variables, so that a certain quality of
irregular formal organization will always be met.
Composition Examples for Virtual Gameplay
Mezzo is currently not attached to any game or game
engine, although development in this area is underway.1
To explore the capabilities of the program, “virtual”
gameplays have been used; that is, videos of various game
2
At the time this paper was written, these videos were available at the
following URLs:
http://www.youtube.com/watch?v=LSIdhm_EOec
http://www.youtube.com/watch?v=X1ZWJdbdsCM
http://www.youtube.com/watch?v=-9-WfJLlhXE
1
Mezzo is written in Python, and uses Max as an interface for sending
game signals to the Python code and handling playback. Communication
between Python and Max is implemented using Open Sound Control.
69
œœ. . œ. .
.
œ œœ œ. œ. ‰ œ. œ. œ. œ œœ.
4
&4
& 44 œ
œ..
3
33
œ.
.
œœ
œœ.
.
33
œœ
4 ‰ j
&
& 444 ‰ œœœj œœœœ ... œœœ œœœ
œ œ œœ œœ
?
? 44 ‰‰ œœ œœ .... œ
4 JJ
&
&
3
jj jj
œœœœ œœ.œœ ‰‰
œ.œ .. .
.
33
33
œ
!
œ
3
3
w
œ
‰ œœj
‰ œj
œ
‰ Jœœ
‰ Jœ
œ
œ
3
3
œœ
œœ
œ
œœ .. œœ œœ
œœ œœ
œœœ.... œœ œ
œœ .. œœ œ
3
Œ
3
Œ
œœ
œœœœ
œœ
3
3
3
33
œœ
simile
simile
œœ
j 3 33 j j ‰ 3 ?
œœ3 jœœ
œ3 œ
œ œ œœ œ œœjœœj ‰ ?
œœ
œœ œ
œ ˙˙
j
.
œ
˙
?
œ
‰ œ œœ .. œœ ˙˙
œ j œ .. œ ˙
?
‰ œ œœ .œœ ˙
œœ œ .
‰ œJœ œ . œ ˙
3
3
‰ œJ
3
œœ œ
œ
œœ œœ œœœ œ œ œ3œ œ œ œœ
œ œ œ œ œ œœ œ œ œ œ œ
3
3
3
œ3 œ
3
3
œ œ Œ
œ
Œ
33
ŒŒ
œœ
œ
œœœœœœœ
œœ
œœœœœ
3
3
3
3
3
œœœ œ ‰3 ‰‰ 3 œ 3 ‰‰ œœœjjœœœ.. .
œ œœ œœœœ ‰œ œ œœœ œœœœ
œ œ.
œ œ œ œ œ
œœœ œœœ.. .
œ œ
‰‰ Jœ œ .
œœ œ œœœœœ ŒŒ
J
‰‰ jj œ ... œœœœ ˙˙˙˙˙
œœ œœœœ... œœ ˙
œ œ
œ˙
œ
‰‰ œ œœ.. œ ˙
œJ
J
œœ
œ
œœ
!!
33
33
3
& Œ3 j 3 j 3 j 3 j
& Œ œj œ œ j œ œ j œ œ j
? œ œœ œœ œœ
?w
3
jj
œ œœœ
œ.œ œ.œ. .œ.
.
33
!
!
&
33
33
œœœœ œ œœ œœœ
œœ
œ œ œœœœ œœ œœ œœœœœœœœ œœœœ jjœœ JœJœ œœœœ jj œœ# #œœœœ
œœœœ
œœœœ œœ
œœœœ œœœœ
33
33
33
33
33
!!
œœ œ œ œœ
œ
œœ œ œ œœ œ œ ‰ œ œœ œ œœ œœœ œœœ ‰ œ Jœœœ œœœ ‰ ‰ œœ # œ œ
œ œ ‰ œ œ œ œ ‰ œ J œ 3 ‰ ‰ 3# œ œ
3
!
&
Œ
œœœ.œ. œœ.œœ.
œœ.
œ.œ.bbœ.œ. ..
œœ
J ‰ ‰ ŒŒ JJ ‰‰ ‰‰
œœ
œœ
j œœ ˙˙˙
‰‰ œœœj œœœœœ... ... œœœœ ˙˙˙
œ œ
œœœ œœ. . œœ ˙˙
‰‰ Jœ
œœœ
œ
œœœ
œ
œœ
œ
ŒŒ
œ
3
3
3
!!
33
ΠΠ3 j 3j 3j 3j
œœjœ œ œjœ œ œjœ œ œj
œ œ œ
3
3
3
3
ww
J
œœ
3
œœ œ
œ œ œœ
œ œ œœ
œ œ œœ
œœ œœ œœœ œ œ œœ3 œ œœœ œœœœœ œœœ œ œ œœœœœ œœœ œ œ œœœœœ œœœœ œ œœœ œœ œœ œœ œ
œœœœ œœœœ
œ œ œ œ œœœ œœœ œ 3 œ œ3 3 œ œ3
3
3
3
3
3
3
3
œ
œ
œ
œ
3 œ
3 œ
3
3
3
3
3
3
3
3
œ
œ3
œ
œ
œ œ œ Œ
œ œ œ œ
œ
j œœ
œ jœ jœ j
œ œ œ
Œ 3 œj œ œ œ Œ œ œ œ œ & ‰ œœ jœœ .. œœœœ
œ jœ jœ j J œ J œ J œ J
j
Œ œ œ œ œ Œ œ œ œ œ & ‰ œœ œœ .. œ œ
J J J J
3
3
3
3
3
3
3
& ‰ œj œ œœ œœ
w
w
œ ..
& ‰œ j œ œœ œœ
w
w
œœ œ ..
œ
!!
œ
3
œ Œ
3
3
3
3
œ
œ
œœ
œœ
œ
œœ
œ
?
?
?
œ
Œ Jœ œ œJ œ Jœ œ Jœ
Œ3 œ œ3 œ œ3 œ œ3 œ
?w
3
J J
3
3
J J
3
w
Example 1a: Cattle motive organization at cue 1
œœ
& œ œ. œ œ.
œœ
& œ œœ .. œœ œœ ..
?
œœ ww
œ w
Jœœ ww
œ w
J
œœ
œ œ. œ œ.
œœœœ œœœ... œœœ œœœ...
œœ
œ
Jœ
œœ
JJœ
œ œœ .. œœ œœ .. œ
œ œœ .. œœ œœ .. œ w
J
?
J
œ
? Œ œj œ œj œ œj œ œj Œ œ œ œ œ œ œ œ & œœ œ œ œœ œ œ œœ œ œ œ œ œœ
J J J J
3
œ
œ
j
œ
j
œ
j
j
œ œ œ œ œ œ œ œœ œ œœœ œ œ œ œ œ œ œ œ
?
? Œ œ œ œ œ Œ J J J J & œ œ œ œœ Œ œ
w 3 3 3 w3 3 3 3
3
œœ 3 3 3
?
œ
œ Œ
w
w 1b: Cowboy motive
Example
organization at cue 1
3
3
3
3
3
3
3
3
3
3
3
3
ww
w
wwww
œœ
œ
œœœœ
w
œœ
œ
œœœœ
œœ .. œœ ˙˙
œœœ. ..œœœ ˙˙˙
œ. œ ˙
œœ
œ
œœœœ
œœ .. œœ ˙˙
œœœ. ..œœœ˙˙˙
œ. œ ˙
œœ .. œœ œœœ
œœœ. ..œœœœœœœ
œ. œ
œœ .. œœ
œœœ. ..œœœ
œ. œ
œ
œ
œ œ. œ œ œ. œ
œœœ œ œ. œ. œœ œœ œ œ. œ. œ
œ. œ œ. œ
œ œ. œ ˙
œ œ. œ ˙
œ œ. œ œ œ. œ œ œ. œ œ œ. œ
œ. œ ˙
œ. œ ˙
œ. œ œ. œ
œ. œ œ. œ
œœ j
œ
œ
œ
œ
œ
œ
œ
œ
œ
œœ j œœ œ œ œ œœ œ œ œ ? Œ J œ J œ J œ J Œ J œ J œ J œ J Œ J œ Jœ œ Jœ œ Jœ Œ Jœ œ Jœ œ œJ œ Jœ
œœ Jœ œ
œœ jœœœ œ œœ œj œœ œ œ œ ? Œ œJ œ Jœ œ œJ œ œJ Œ œJ œ Jœ œ œJ œ œJ Œ Jœ œ Jœ œ Jœ œ Jœ Œ Jœ œ Jœ œ œJ œ Jœ
œ œœ Jœ œ Œ
w3 3 3 3 w 3 3 3 3 w 3 3 3 3 w 3 3 3 3
3
3
3
3
œ
œ
œ Œ
w
w
w
w
3
3
3
3
70
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
b
& b bb
bb b b
&
bb
&bb
bb b
& b
b
& b bb
bb b b
&
? bb b
b
? bb
bb
b
& b bb
bb b b
&
? bb
bb
? bb
bb
? bb b
b
? bb b
b
? bb b
b
? bb b
b
‰ j œ . œœ œœœ
.
‰ œj œœ . œœœ œœœ
œ œ. œ
‰ j
‰ œœj œœ .. œ œœ
œœ œœ .. œ œœ
Ó
˙
Ó
˙
!
!
œœ
œœ
œœ
œœ
?
?
‰ j œ . œœ œœœ
.
‰ œj œœ . œœ œœœ
œ œ.
‰ j
‰ œœj œœ .. œœ œœ
œœ œœ .. œœ œœ
˙
Ó
˙
Ó
!
!
œœ
œœ
œœ
œœ
‰ j œ . œœ œœœ
.
‰ œj œœ. œœœ œœœ
œ œ. œ
‰ j
‰ œœj œœ .. œ œœ
œœ œœ .. œ œœ
˙
Ó
˙
Ó
!
!
>œ >œ . >œ >œ . >œ n >œœ .. >œœ
‰ . n>œœ >œœ .. >œœ >œœ .. >œœ n >œœ.. >œœ
œ œ œ œ
? ‰ . n œœR œ .. œœ œ .. œœ œ .. œ
œ œ. œ œ. œ
‰ . Rœr
‰ . œ>r œ œ œ
> œ> œ> œ>
w
nw > > >
ww>
nn ww>
w
nw
>
!
>
œœ œœœ œœœ œœœ œ
!
flœ flœ flœ flœ flœ flœ flœ flœ flœ flœ œfl œfl
fl fl fl fl fl fl fl fl fl fl fl fl
!
!
!
!
‰ j œ . œ n ˙˙
nœ
‰ j œ . œ n ˙˙
nœ
‰ j
nœ. œ ˙
‰ n œœj œ . œ
n
œ œœ .. œœ ˙
nœ
n˙
Ó
n˙
Ó
!
!
œœ
œœ
œœ
œœ
A œœ
A œœ
‰ œœ
‰ œœ
œ.
œ.
œœ ..
œœœ...
œ.
œ œ.
œ œ.
œœ œœ ..
œœœ œœœ...
œ œ.
œœ Œ œœ
œœJ Œ œœ
J
œœ
œJ
œ
Jj
œœ
j
œœ
˙˙ ..
˙˙˙...
˙.
˙.
˙.
œ
œ
œœ
œœ
œœ
œ
œ
‰ œœ
‰ œœ
œœ
œœ
œœ
œœ
œœ Œ
Jœ
œ Œ
J
œ
œ
˙
˙
nw
nnww>
nw !
>
!
œœ .. œ
œœœ... œœ
œ. œ
?
œ . œœ
?
œ . œœ
˙˙
˙˙
˙˙
˙˙
œœ œœœ œ œœ œœœ œ œœ œœœ œœœ œœœ œ
œflœfl œflœflœfl œfl œflœfl œflœflœfl œfl œflœfl œflœflœfl œflœflœfl œflœflœfl œfl
fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl
>œ >œ . >œ >œ . >œ n >œœ .. >œœ
>œ >œ . >œ >œ . >œ n >œœ .. >œœ w>
w>
w>
‰ . n>œœ >œœ .. >œœ >œœ .. >œœ n >œœ.. >œœ ‰ . n>œœ >œœ ..>œœ >œœ ..>œœ n >œœ..>œœ n>ww
n>ww
n>ww
œ œ. œ œ. œ œ. œ
œ œ. œ œ. œ œ. œ w
w
w
‰ . n œœR œœ .. œœ œœ .. œœ œ . œ ‰ . n œœR œœ .. œœ œœ .. œœ œ . œ n ww
n ww
n ww
‰ . Rœr
‰ . Rœr
w
w
w
‰ . œ>r œ œ œ
‰ . œ>r œ œ œ
>
>
> œ> œ> œ>
> œ> œ> œ> w>
w
w
>
>
>
>
>
>
>
>
>
nw
nw
!
!
!
nnww>
nnww>
!
!
!
nw
nw
!
!
!
!
!
>
>
!
!
!
!
!
!
!
!
!
?
œœ
œ
œ
Ó
Ó
!
!
w>
n>ww
w
n ww
!
!
!
!
w
ww>
w
>
w
>
w
>
w>
n>ww
w
n ww
w
>
w
> !
w
>
w
> !
!
!
!
!
!
!
Example 2a: Cowboy motive organization at cue 5
b
& b bb
bb b
& b
b
& bb b b
b
& bb
œœ
œ œ œ œ
œ
œœ ‰ œ œ œœ œ œ œ œ œ œ
œ
œ
œ
œ
3œ
‰ œ œ3 œ 3œ œ œ 3œ œ
3
3
j 3 3
œ 3 œ œ 33 œj œ33 œj Œ 3
œœ œœœj œœœ œj œ œj Œ
œ œœ œœ œœ œœ
Œ œJ œ œJ œ œJ œ Jœ
œ œ œ œ
Œ 3 J œ 3 J œ 3J œ 3J
? bb b
? b b bb
b
? bb b 3
? b b bb w
b
w
bb
& bb b
b
& bb
b
& bb b b
& b bb
3
3
3
œœ
œ œ œ œ
œ
œœ ‰ œ œ œœ œ œ œ œ œ œ
œ
œ
œ
œ
3œ
‰ œ œ3 œ œ3 œ œ œ3 œ
3
3
j 3 3
œ 3 œ œ 33 œj œ33 œj Œ3
œœ œœœj œœœ œj œ œj Œ
œ œœ œœ œœ œœ
Œ œJ œ œJ œ œJ œ Jœ
œ œ œ œ
Œ 3 J œ 3 J œ 3J œ 3J
3
w
w
œœ œ œ œ œ
œ
œ
œœœ œ œ œœ œ œœ œœ œ œ œ œ œœ
œ 3 œ œ œ3 œ 3œ œ œ 3œ œ
3
3
œ 33n j œœ 3 œ 33 œj Œ 3
œ
œ
œœ j œœ œœ œœj Œ
œ n œœ
œ œ
3
j
j
œ
Œ 3 œ œ œ œj œ œj
Œ œj œ 3 œj œ 3 œj œ 3 œj
? bb
b
? b b bb
b
? bb
bb
? bb b w
b
w
3
3
3
3
3
3
œœ œ œœ œ
œœ œ3 œœ b œœ œ
bœ 3 œ
3
œœ 3
œœ œ
œ
œ
œ œ
Œ J bœ J œ J
œ
œ œ
Œ 3 J bœ 3 J œ 3J
œ bœ
œ bœ œ
"3 œ
"œ 3
œ
w
w
3
3œ
œœ j œ 3œ œœ j œ n œ3 œ
œœ 3 œœj œœ œJ œœ 3 œœj œœ œ n œ œ
œœ œ J œœ œœ
3
3
Œ
œ
œ œ œ Œ
œ
œ
3
Œ 3 œj œ œj œ œj œ œj &
Œ œj œ 3œj œ 3œj œ 3 œj &
w
w
3
3
3
3
3
3
œ œœ œ ‰ œ œœJ
œ
œ
œ œœ œ ‰ œ œJ
œ3
3
3
3
Œ3
œ33œj œ3
Œ
œœ œœj œ
œœ
œ œJ Œ Jœ œ œJ
œ 3 œJ Œ 3Jœ œ 3 œJ
3
3
w
w
3
3
3
3
Example 2b: Cattle motive organization at cue 7
71
œœ œ œ œ œ œ
œœ œ œ œ œ œ œ œ œœ
œ3 œ œœ œ œ œœ œ œ œ œœœ
3
3
œ
œ
3
œ
œ
3
!
!
!
!
3
3
!
!
!
!
n œœ œ œ œ œ œ œ3 œ œ œ œ
œ
n œœ œ œ œœ œ œ n œœ œ œ œ œ
3
3
3
nœ
3
3
Œ3
nœ
nœ nœ
œ Œ
nœ
œœ
3
3
Œ n œ œ 3 œJ œ œJ œ 3 œJ
J
œ
Œ 3n œJ œ œJ œ 3 œJ œ œJ
?
œœ 3 ‰
‰3
3
3
œœ œ ‰ œ œ œ ‰ œ œ œ œ œœ ?
œ3 œ œœ œ œ œ3 œ œœ
œ œ
3
œ 3 œJ œ œœ 3 ‰ Œ
œœ ‰ Œ
w
J
œ
w
3
œœ ‰ ‰ b œ
œ œ
œœ ‰ ‰ b œ œ
3
œ3
3
3 j j 3
‰
3œ œ
œ œœj œœj ‰
œœ œ
œ œJ œ Jœ
œ 3œJ œ 3 Jœ
3
3
œ
œ
Œ
Œ
3
3
3
œœ œ œ œ œ
œ
œ œ œ œ œ œœ
œœ œ œ œœ œ œ œ œœ
œ 3 œ œ œ3 œ œ œ3 œ œ œ3 œ
3
3
œ33n œjœœ3 œ33 œj Œ3
œœ œjœœ œœ œœj Œ
œ n3 œœ
œ œ
Œ 3 œjœ œjœ œj œ œj
Œ œjœ 3 œjœ 3œj œ 3œj
3
3
w
w
3
N œœ œ œ œœ œ œœ œ œ œœ œ
N œœ œ3 œ œœ œœ œœ œ3 œ œœ œœ
œ3
œ3
œœ œ œ œœ œ œ œœ œ œ œœ œ œ
œœ œ3 œ œœ œ3 œ œœ œ3 œ œœ œ3 œ
œ
œ
œ
œ
Œ œJ œ œJ œ œJ œ œJ
Œ 3 œJ œ 3 œJ œ 3 œJ œ 3 œJ
œ
œ
œ
œ
Œ Jœ œ Jœ œ œJ œ Jœ
Œ 3Jœ œ 3 Jœ œ 3œJ œ 3 Jœ
3
w
w
3
œ3
œ
3
3
3
œ3
œ
3
3
w
w
3
œ
œ
3
3
3
3
œ
œ
3
3
between statements of Leitmotivs, what registers they will
occur in, etc.
Each time point in a script was associated with a set of
values for each of the states listed above. This process, of
course, assumes an amount of interpretive ability on the
part of an actual game, which here is being done by a
human author. While this is currently a drawback of the
design of the demonstration, the interpretive assumptions
being made are reasonable; this is because the states a
game must pass Mezzo, as described above, are well
defined in terms of explicit game elements.
A Max patch was used to trigger each set of values to
Mezzo at each associated time point, and this process and a
video were begun at the same time. The music was
composed for four MIDI channels, all set to a piano sound.
Each time the program is run alongside a video, the music
is similar, although it is never the same.
For the Red Dead Redemption scene, ten cues were
determined for which the scene changes:
mapped to non-normative––that is, highly marked––
musical settings. First shown is a section using the
cowboy’s motives during cue 5, when his horse has been
shot and he is under great duress. Second shown is the
cattle’s motive at cue 7, in which the herd is highly
disorganized, presenting the cowboy with the problem of
coercing them back together. In both of these sections, the
motives are organized in a highly non-normative way, and
the harmonic tension is high, in order to express the highly
marked states the characters are experiencing.3
Further Avenues for Research
While many important expressive elements are not
handled by Mezzo, it still offers a framework for
procedurally composing cohesive music that adapts to
dramatic elements in a game in real time. Elements such
as orchestration, motivic development, and expressive
playback (rubato, dynamics, etc.) are certainly necessary to
make this an effective music engine for game
development. Furthermore, a standardized communication
protocol for mapping game states to musical settings needs
to be developed and implemented. However, the program
as it now stands is a proof of concept of a method of
procedurally composing music that is expressive and
interesting. It also offers a foundation to which more
implementations of musical expressiveness can, and
hopefully will be, added in the future.
1: cattle introduced; cowboy rides into herd
2: explosion on horizon; cowboy rides toward it
3: cowboy encounters train robbery in progress
4: gunfight between cowboy and train robbers
5: cowboy’s horse shot; cowboy looks for
another horse; train robbers flee
6: cowboy mounts new horse and bids farewell
to train passengers he rescued from robbers
7: cowboy attempts to marshal cattle that have
scattered during commotion of train robbery
8: cowboy tries to corral one errant cow back to
herd; gives up, rides off with remaining cattle
9: the herd, minus one cow, reaches the pasture
10: cowboy rides home, mostly successful
References
BioWare Austin and BioWare Edmonton. Star Wars: The Old Republic.
BioWare, 2008.
Brown, Daniel. 2012. Expressing Narrative Function in Adaptive,
Computer-Composed Music. D.M.A. diss, Department of Music,
University of California at Santa Cruz, Santa Cruz, CA.
Two types of Leitmotivs were input, corresponding to
the player’s character (a cowboy), and the herd of cattle.
The player’s character had four similar Leitmotivs, which I
composed, and the herd only one, which was taken from a
Liszt scherzo. The musical examples given below show
some of the resulting music that was written for these cues.
The cowboy’s motives and the cattle’s can be distinguished
in the following way: the cowboy’s are made up of block
chords in dotted-eighth- and dotted-sixteenth-note
rhythms; the cattle’s motive is made up of scherzo-like
arpeggiated triplets.
Example 1 shows their organization at cue 1; here, the
formal organization is very regular––both the cowboy’s
and the cattle’s motives are organized as normative
sentences, and the harmonic tension is very low. This
evokes the relatively normative state that both the player
and the cattle are in: the player is faced with a problem
(herding the cattle to pasture), but the game has just begun,
and the problem does not seem dire.
Example 2 shows music written in later stages of the
game, in which the same Leitmotivs are used, but are now
Caplin, William. 1998. Classical Form: A Theory of Formal
Functions for the Instrumental Music of Haydn, Mozart, and
Beethoven. New York: Oxford U Press.
Hatten, Robert S. 1994. Musical Meaning in Beethoven:
Markedness, Correlation, and Interpretation. Bloomington:
Indiana U Press.
Grabócz, Márta. 2009. Musique, narrativité, signification. Paris:
L’Harmattan.
Nintendo EAD Tokyo. Super Mario 3D Land. Kyoto: Nintendo, 2011.
Rockstar San Diego. Red Dead Redemption. New York: Rockstar Games,
2010.
Schoenberg, Arnold. 1969. Structural Functions of Harmony.
New York: W.W. Norton & Co.
3
A full analysis of the music composed for this game trace is in Chapter 7
of (Brown 2012).
72