Motions: Hazel Smith
(text); Will Luers
(image and coding)
Roger Dean (music)
Musico-Literary Miscegenations:
words and music synergies in new
media writing ``1
Hazel Smith
Writing and Society Research Centre
University of Western Sydney
Trajectories
• Focus: words and music relationships in new media writing.
• Relationship between words and sound in the new media writing
field neglected by critics. Tend to be more interested in the
relationship between word and image.
• Word/sound relationships have tended to be less explored by new
media writing practitioners than word/image.
• Aspirational as well as analytic.
Grigar, Dene. "The Role of Sound in Electronic Literature." trAce Online Writing Center. Spring 2006.
Barber, John F. Internet radio and electronic literature: locating the text in the act of listening, Electronic Book Review,
http://www.electronicbookreview.com/thread/electropoetics/internetradio
2
What is new media writing?
• New media writing (electronic literature, digital writing, networked and
programmable writing) is a form of literature that exploits the possibilities of
new technologies to evolve the way we write and read literature.
• Usually has screened words, may also have spoken words.
• Some features: kineticism, interactivity, processing of text, textual
variability, multimedia.
Hayles, Katherine N. Electronic Literature: New Horizons for the Literary. Notre Dame: University of Notre Dame
Press, 2008.
3
Will Luers (video), Roger Dean (sound), Hazel Smith (text),
Film of Sound
Cordite Poetry Review , 2013.
• http://cordite.org.au/ekphrasis/film
-of-sound/
4
Words and music: formal and cultural crossings
• Cross-overs of words and music in new media writing
can create hybrid structures which produce emergent
meanings.
• These hybrid structures can facilitate various forms of
cultural crossing and cross-cultural encounter.
• Conceptualise this process of formal and cultural
crossing as musico-literary miscegenation (Sexual
inter-racial relationships).
5
word
imagetext
image
textsound
sound
Musicoliterary
miscegena
tion in new
media
writing
Cultural processes
Transnationalism
Cross-cultural encounter
Cultural borrowings and
appropriations
Translation
Intermedia Processes
Parallelism
Coordination
Interactive variability
Algorithmic Synaesthesia
Heterogeneity
Semiotic and perceptual exchange
Intermedia
hybrid space
ethnicit
y
culture
Nation
state
Musico-literary
miscegenated
space
Intracultural
hybrid space
Formal aspect ; Zbikowski; conceptual blending
• Formal aspect of the model; builds on theories of musical multimedia, work
in the area of musico-literary discourse: Nick Cook, Laurence Kramer, Lawrence
Zbikowski.
• Zbikowski: draws on ideas about conceptual blending from cognitive
linguistics (Turner and Fauconnier) and maps them to the way words and music
interrelate.
— generic space, input spaces, blended space.
• differences from Zbikowski; multimedia not song and opera; interested in
juxtapositions, frictions, hybrids, not blends; cultural framework.
Zbikowski, L. M. (2002/3 ). Music Theory, Multimedia, and the Construction of Meaning. Integral 16/17, 251-268.
Cook, N. (2001). Analysing musical multimedia, Oxford: Oxford University Press.
Kramer, L. (2002). Music Meaning: Toward a critical history, Berkeley: University of California Press.
7
New media writing: what kinds of
sound/music?
(1)
• Ecological sounds (sounds taken from or mimicking the
environment, from everyday life).
• Often have an iconic or illustrative connection with the
text.
Clarke, E.F. Ways of Listening: An Ecological Approach to the Perception of Musical Meaning, Oxford University Press, 2005
8
Example (ecological sound )
MD Coverley Afterimage
• http://califia.us/Afterimage/end.htm
9
New media writing: what kinds of
sound/music? (2)
• Musical sounds: often used for their affective, emotional,
atmospheric or immersive properties.
— from acoustic instrumental music to electroacoustic
music. (Film of Sound)
— independent piece of music, separately composed;
music composed for the piece including algorithmically
generated music.
Smith, H. soundAFFECTs: translation, writing, new media, affect in Sounds in Translation eds. Amy Chan , Canberra:
Australian National University Press.
10
New media writing: what kinds of
sound/music?
• Voiced
(3)
sounds:
—straight
— technologically manipulated
— computer - synthesised voices
Smith, Hazel , and Roger T. Dean. "Voicescapes and Sonic Structures in the Creation of Sound
Technodrama, Performance Research 8.1 (2003): 112-23.
11
Intermedia processes
Word/sound juxtapositions in new media writing
the intermedia hybrid space (1)
Coordination, Parallelism, Interactive Variability, Algorithmic Synaesthesia,
Heterogenity, Semiotic and Perceptual Exchange (may be more than one type of
conjunction in any one piece)
Coordination:
— associative, representational, illustrative linking between sound
and words
— (often ecological sounds (e.g. MD Coverley Afterimage ).
Most emergence when associations are multiple and are indirect.
Parallelism:
— music and screened text run continuously together but are
independent entities. The text or music or both, may be algorithmically
organised to be different each time the piece is played, so there may be different
points of coordination each time.
12
Brian Kim Stefans, Suicide in an Airplane 2014
— Example of parallelism: Brian Kim Stefans (Asian American writer),
Suicide in an Airplane 2014 written in Flash, music Suicide in an
Airplane by Leo Ornstein, futurist composer 1919.
— Source text for the words is an article from The New York
Times ; American combat operations in Afghanistan; algorithms for the
text chose different words each time from the source text.
— Different points of coordination between the words and
music each time.
Energetic music, futurist celebration of mechanisation, engines and
flight; a text which can be read as a critique of the bombings in
Afghanistan. Parallelism can bring out both similarities and
differences.
13
Example 2 (Brian Kim Stefans: Suicide in an Airplane, 2013)
• http://www.arras.net/?p=572..
14
Intermedia processes
Word/sound juxtapositions in new media writing
the intermedia hybrid space (2)
Interactive variability:
— the user can interact with, and vary, the sound (possibly in addition
to the text: Wilks Tailspin, Bouchardon Loss of Grasp ).
— Or the performer can interact with the sound during real-time
performance.
Creates constantly changing relationship between text and sound.
Algorithmic synaesthesia:
— Music and words may be composed using the same algorithmic
basis for both; the algorithms from the words can be translated into algorithms
for sound (e.g. Smith, Dean, Brewster, soundAFFECTs; Cayley Translation).
Dean, R.T., et al. The mirage of real-time algorithmic synaesthesia: Some compositional mechanisms and research
agendas in computer music and sonification. Contemporary Music Review
15
Hazel Smith, Anne Brewster (text, Roger Dean
(sound and coding) : soundAFFECTs, 2004.
• On CD Rom accompanying, Hazel Smith The
Erotics of Geography, Tinfish Press, 2008.
16
Intermedia processes
Word/sound juxtapositions in new media writing
the intermedia hybrid space (3)
Heterogeneity:
— Sound and text composed of multiple, heterogeneous and dynamic elements.
Continuously couple and uncouple in different ways.
— The more heterogeneous the elements, the more emergence . (Luers, Dean, Smith
Film of Sound).
Semiotic and perceptual exchange:
— Different media seem to take on each other’s characteristics when juxtaposed.
— Words highly referential, while music is more abstracted. When words and sound
are juxtaposed become more aware of sonic/abstracted aspect of words, referential aspects of
sound.
— perception as well as semiotics.
(e.g. Young Hae Chang Heavy Industries)
Smith, H. & Dean, R. (1997). Improvisation, Hypermedia and the Arts Since 1945, London and New York: Harwood Academic.
c17c
Young Hae Chang Heavy Industries: Nippon
(music: Thelonius Monk quartet , Japanese Folk
Tale, 1967)
• http://www.yhchang.com
18
Young Hae Chang Heavy Industries: Nippon (2)
• Conscious use of semiotic and perceptual exchange – the
movement of the language is programmed to respond to the
rhythm of the music.
•
after the opening the Japanese is moving with the piano and
the saxophone with the English.
•
the word-music exchange is not only creating a semiotic
dialogue, it is also creating a cultural interaction.
19
word
imagetext
image
textsound
sound
Musicoliterary
miscegena
tion in new
media
writing
Cultural processes
Transnationalism
Cross-cultural encounter
Cultural borrowings and
appropriations
Translation
Intermedia Processes
Parallelism
Coordination
Interactive variability
Algorithmic Synaesthesia
Heterogeneity
Semiotic and perceptual exchange
Intermedia
hybrid space
ethnicit
y
culture
Nation
state
Musico-literary
miscegenated
space
Intracultural
hybrid space
Intercultural processes and the intercultural hybrid
space:
Musico-literary miscegenation
Mixings/crossings of word and sound can and may facilitate
cultural crossings :
— crossing of national borders (transnationalism)
— crossing over of different ethnicities and ethnic
cultures
— various forms of cultural encounter (western, nonwestern cultures)
— various forms of cultural appropriation
21
Musico-literary miscegenation
Results in musico-literary miscegenation:
• Reactivating the word miscegenation more positively:
— maps the ethnic, erotic and reproductive onto the mixing of
words and music
— brings the histories of colonialism, cultural imperialism and
unequal power relationships to bear on our thinking about the wordsound relationship
— in intermedia/multimedia work, word and music
relationships most successful when non-hierarchical.
22
Theoretical Background
Cross–cultural aspect of the model builds on:
• theories of globalisation/cosmopolitanism (e.g. George Ritzer — glocalisation and
grobalisation as the two faces of globalisation, Ulrich Beck, Robert Fine, Berthold
Shoene). Posthuman cosmopolitanism.
• theories of biraciality (Zack, Funderberg, Piper, Romano, Sollers); cross cultural
encounter, hybridity (e.g. Paul Gilroy, Stuart Hall, Robert Young). Work of Jackie Lo
and Helen Gilbert.
• the positive retrieval by others of the idea of miscegenation (Dunning; Kim
Stefans)
Three examples which show different forms of cultural crossing: cross cultural
encounter, transnationalism; cultural appropriation.
Lo, Jacqueline, and Helen Gilbert. "Toward a Topography of Cross-Cultural Theatre Praxis." TDR/The Drama Review 46.3 (2002): 31-53.
Dunning, Stefanie K. Queer in Black and White: Interraciality, Same Sex Desire, and Contemporary African American Culture. Indiana: Indiana
University Press, 2009.
Schoene, B. (2009). The Cosmopolitan Novel, Edinburgh: Edinburgh University. Press.
Smith, H. A Posthuman Cosmopolitanism in new media writing, Hyperrhiz: new media cultures, 09, http://www.hyperrhiz.net/hyperrhiz09/essays/aposthuman-cosmopolitanism-and-new-media-writing.html
Ritzer, G. (2007). The Globalization of Nothing, London: Pine Forge Press.
Zack, N. (1994). Race and Mixed Race, Philadelphia: Temple University Press.
Kim Stefans (2013) Viva Miscegenation: New Writing (MakeNow Books)
23
Example. Young Hae Chang Heavy Industries:
Operation Nukorea 2003 /
• http://www.yhchang.com
24
Operation Nukorea
•
Cross-cultural group based in Seoul. Young Hae Chang, Korean artist and translator; Marc Voge.
American poet. Strong interest in the word-music relationship.
•
Formally, movement of text constructed to respond to two different rhythmic levels of the music:
the regular pulse of the ostinato; the rubato effects of the melody: semiotic and perceptual
exchange.
• Text about an imagined devastating attack by North Korea on South Korea, precipitated by a preemptive US attack on North Korean nuclear facilities. Text about east-west antagonisms.
• Juxtaposed with US jazz pianist Bill Evans’s “Peace Piece”,. Everyone Digs Bill Evans, 1958, followed by
a trio item “What is there to say”. From Everyone Digs Bill Evans, 1958.
•
Thematically the music is meditative, calm and reconciliatory, opposite to content of the text.
• Juxtaposition of different media speaks to the conflict between east and west: emergent meaning.
— Whatever the level of aggression between nation states and different cultures, always the
possibility of peace, exchange and cooperation.
25
Example. Translation John Cayley (text) Giles
Perring (music)
• http://collection.eliterature.org/1/
works/cayley__translation.html.
First published 2004.
26
Example: John Cayley/Giles Perring Translation (2)
•
John Cayley: British new media artist currently living in the US.
The text undergoes ‘transliteral morphing’ — the building up of,
and substitution of letters, within each individual textual block;
moves between English, German and French; and often mixes
languages and inhabits an in-between language.
• The main text that is translated is ‘ Translation is removal from
one language into another through a continuum of
transformations. Translation passes through continua of
transformation, not abstract ideas of identity and similarity’. ‘On
Language as such’ Benjamin.
27
John Cayley/Giles Perring: Translation (3)
• Music, ‘sung vocabularies’, and the words share the same algorithms
as the music (algorithmic synaesthesia).
• The music cursor scans the text and is programmed to respond
musically to what is happening linguistically.
• the translation between the media both facilitates and reinforces the
translation between languages, and emphasises the broader cultural
dynamic of transnational (European) crossings and inter-relationships.
Emergence — the endless process of translation between languages,
music and cultures, and the inhabitation of an in-between language.
28
Motions: Hazel Smith (text), Will Luers (image and
coding), Roger Dean (music) 2014
http://taylorstreetstudio.com/motions/
• About human trafficking and contemporary slavery (draws on many different
genres: poetry, narrative, documentary, photography). Interactive multimedia
book, different each time you read it.
• Juxtaposes texts and images with music (through parallelism, interactivity
illustration, heterogeneity)
• Four sections of music: flight sounds, sampled Macedonian music , train
sounds, Americanised music. Also cross-fertilisation of the American and East
European musics.
• Uses transnational appropriation of sounds (e.g. sampling of Macedonian
music).
• Uses transnational appropriation to evoke other cultures but also their
suppression. Brings out the darker, exploitative side of
globalisation/cosmopolitanism/transnationalism.
29
Motions (Smith, Luers, Dean)
http://taylorstreetstudio.com/motions/
30
Coda
• Different formal crossings between words and music can
facilitate cultural crossings.
• this crossover creates the musico-literary miscegenated space.
• model reflects sonic-verbal interactions in pre-existing new
media writing but can suggest future possibilities.
31