Algorithmic Visual Music
Jun Yong Moon
I. Introduction
Motion graphics show us new visuals everyday. Sound takes big portion of it. Artists visualize it in
various ways, and the interaction of sound and visuals is an important element that entertains us. For this
reason, I am looking for an advanced method of visualizing music. How should I combine image with
sound to impress people? In a technical aspect, I am convinced that computer codes will be greatly helpful
for my purpose. The new version of iTunes visual equalizer is a good example. The algorithm of this
software effectively analyzes music and expresses it to dynamic visuals and I would like to use this
technology. Thus, for my final project, I will design an algorithmic visual music using code. For this work,
I will research the visual music precedents from early the 1900’s and see how other artists have
corresponded visuals to sound and how technology has served their desires.
II. Early Visual Music Artists
In this chapter, I will look back on the history of visual music. Numerous painters already had tried to
express sound in a static image before the invention of film. However I would rather only write about
moving images for this research, since I believe only time-based medium can represent music. Kerry
Brougher states that “Painters, who had been confined to merely suggesting motion and rhythm through
static fragments, could now create flowing movements and rhythmic schemes that shape time, thereby
bringing visual art closer to music.”1 The moving image technology has been even more developed since
the invention of computers, which have been expanding the scope of artistic expression. Before the
computer arts, we will see how early visual music artists approached the image and sound in this chapter.
Kerry Brougher mentions Viking Eggeling as one of the first distinctive artists using this new film making
technology in early the 1900’s.2 Although it looks like a clumsy animation in today’s perspective, and
having no relationship with sound, Kerry Brougher explain the importance of Eggeling’s work in the visual
music history :
Symphonie Diagonale(1924) is composed of stark white shapes on a black background. His shapes,
although abstract, conjure associations with stringed musical instruments, musical staff lines, and
machine parts. As they metamorphose from one state into another, they seem to create a completely
new language that lies somewhere between the visuals and the aural. Artist Laszlo Moholy-Nagy
wrote that “Eggeling discovered the all-prevailing, revolutionary importance of an esthetic of time in
film.” And that “his experiments at first leaned upon musical frames of reference, such as the division
of time, regulation of tempi, and over-all structure.”3
His discovery was followed and further developed by some visual music pioneers such as Oskar Fishinger,
Len Lye, Stan Brakhage, John and James Whitney, Jordan Benson.
1
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 100.
2
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 100.
3
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 100-1.
Oskar Fischinger's works show good examples of successful early visual music elements. In Circles(1933),
we can see a good example of matching space and the sound. We can see a sense of sound spreading out in
space from the particles scaled depending on the sound. In Murattl Marches On (1934) and Murattl
Privat(1935), he characterized cigarettes, which dance and march depending on music. In Muntz TV(1953),
he even synchronized typographies to sound. He probably shows the most elaborate integration of sound to
image among the early artists I will cover in this research. However, I would like to rather emphasize on
this fact: According to Kerry Brougher, “he used the term ‘body’ to describe some of the forms in his work
– ‘Blue body,’ ‘red little bodies’ - suggesting characters acting out a kind of drama involving complex
transformations from one state to another. Most of his films begin with relatively simple geometric forms
that, through expansion and contraction, begin to transform on multiple planes, climaxing in pyrotechnical
explosions of color flicker.”4 I notice that he brought a flow of narrative into visual music. Each visual
element in his works appears on the stage and acts some kind of motion before they disappear. Even though
they don’t suggest any representational meaning, we fill as similar filling as real dancers on the stage and
follow their movement with eyes being curious about what’s going to happen next. They entertain us in
each moment they appear in this way and finally, developing a drama into climax. On the other hand,
Fischinger also tried to generate music himself. According to Kerry Brougher, he experimented on his own
images themselves to generate sound : “By using a modified camera that was capable of recording black
geometric shapes painted on scrolls right onto the film soundtrack, he was able to turn these visual
‘ornaments’ into sound.”5 Obviously, we can see he had greatly emphasized on the synchronization of
visuals and sound, and furthermore, even tried to blur the boundary between them. We will see numerous
other artists who have been doing this kind of effort to enhance the relationship of visuals and aural later.
Another artist, Len Lye, tried different method of drawing images directly on the celluloid of film.
According to Kerry Brougher, “He combined this ‘direct film’ approach with other techniques such as
solarization, multiple exposures, new color printing techniques, optical printing, traveling mattes, and
scratching, with text messages. He fused the abstract images he made directly on film with representational
documentary footage for advertising films.”6 He already realized as similar visuals as today’s motion
graphics with these techniques and all of these visuals succeeded to interact with sound effectively in his
films such as A Colour Box(1935), Rainbow Dance(1936), Trade Tattoo(1937), and Clolour Flight(1937).
Especially, I want to emphasize on his use of patterns depending on sound – for example, in Rainbow
Dance - since they form a beautiful harmony with music by effectively visualizing the complex bit of
music.
III. Early Algorithmic Visual Music
While Oskar Fischinger and Len Lye are connecting their visual elements with the sound in every single
frame, some artists focused on algorithmic animations. Such kind of animations generated by a set of rules
provided them complex visual density, which is similar to the complex combination of sound elements in
music. I will talk about John and James Whitney, and Jordan Belson who used these animation techniques
in their visual music.
In the technical aspect, the Whitney’s work process is really interesting and impressive that it had been
already realized in the early 1900’s.
John Whitney constructed an eight-millimeter optical printer, a system of creating sound from the
motion of a pendulum. It was more intricate than any of earlier synthetic-sound experiments.
Pendulums of differing lengths fitted with adjustable weights were attached by a wire to an aperture
4
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 110.
5
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 125.
6
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 111.
in the camera, which recorded their movements, thereby creating sound from motion. Using this
system, the Whitneys were able to create a four-octave range of electronically produced tones. In Five
film Exercises(1943-44), the Whitneys synched this optically produced sound to images that were
produced by light shot through a stencil system devised by James. The results are a completely new
type of visual music; in both appearance and sound, the works are electronic, aggressive, grating, and
almost ruthless in their assault on eye and ear. The images and sound seem inextricably linked. One is
not a result of the other; rather, sound is image, and image sound, with no fundamental difference.
Their works advanced visual music in one sudden shocking step during World War II. 7
One of their works, Lapis (1963-1966), shows well their powerful visual style. Countless particles swirl
around the center on the basis of their algorithms, sometimes randomly scattering and sometimes forming
mandala patterns, which remind us some kind of spiritual symbols. From the eternally repeating, but
carefully refined motions of patterns, we feel like flying endlessly into the center of cosmic. At this point, I
want to talk about cosmic visuals, which not only Fischinger the Whitneys’, but also another numerous
artists have been pursuing. According to Kelly Brougher, from the ancient Greeks on, even scientists such
as Pythagoras and Newton have been trying to find out the principal of universe from music. 8 This purpose
is closely related to why the Whiteys’ so concentrated on the technology as well. The numberless micro
particles forming infinite patterns in Lapis would not have been possible without his optical devices. For
James Whitney, complex forms and motion was essential to visualize the property of music, or to visualize
the way music impress our emotion. According to Kerry Brougher, he was certain that the solution lay in
technology. He said that “searching ahead toward a single-handed effort to invent a machine that would be
used as an instrument for composing this liquid architecture.”9
I absolutely agree with the idea about the similarity between cosmic structure and music. I am convinced
that music can not be visualized by a single frame of image, since the music is generated in a certain length
of time. Moreover, forming visual space is essential in visual music, since sound itself physically echoes in
space. It can not be visualized by a simple visual element as well, since a music piece comprises complex
combination of sound elements. Thereby, changing themselves continuously, these complex combinations
form eternal and spiritual feelings. Ultimately, the cosmic visuals might be the one which can satisfy all of
these premises. Furthermore, only an algorithmic animation can effectively generate such complex visuals.
Another artist, Jordan Belson, beautifully realized this cosmic visuals “through the use of both old and
new technology, including standard animation, optical printing, lasers, and liquid crystals.”10 Even though
both of Whitney’s’ and Billson’s works were abstract, if Whitney’s’ works were mathematical and logical,
Billson’s works were more similar to nature. His work, Samadhi(1966-67) makes spectators feel like
floating in a mysterious dream, which seems to be continuing in an eternal space and time, with the images
like cloud, sky, wind, sunshine, undersea or waves. Besides, in the music aspect, “often created by Belson
himself, the music-to-image relationship became extremely tight. As he explained, ‘You don't know if
you're seeing it or hearing it.’” 11
IV. Contemporary Artists
We can see how many artists have pursued tight relationship between image and sound. Some created
music themselves while some even devised new instruments, which generated sound and image
7
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 125.
8
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 132.
9
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 132-3.
10
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 110.
11
Kerry Brougher, “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900, ed. by Kerry Brougher, Jeremy strick, Ari
wiseman, and Judith Zilczer, (New York : Thames & Hudson, 2005), 110.
simultaneously for those purposes. Such desire of artists can be now realized more effectively with
powerfully advanced high technology. I'll talk about these contemporary artists and see what they are
pursuing in this chapter.
A Japanese visual music artist, Ryoichi Kurokawa shows impressive live performances. According to
him, “He composes time based sculpture with digitally generated materials and field recorded sources, and
the minimal and the complexities coexist there. Kurokawa accepts sound and imagery as a unit not as
separately, and constructs very exquisite and precise computer based works with the audiovisual language.
That shortens mutual distance and the synchronization of sound and visual composition.”12 One of his main
techniques in cm: av_c, Parallel Head, Rheo is an exactly synchronized combination of motion and sound,
which provides him a unique visual style corresponding with experimental electric music.
Carsten Nicolai has been doing lots of live visual music performances with a name of Alva Noto.
According to him, “He works intensively in the transitional area between art and science. He seeks to
overcome the separation of the sensual perceptions of man by making scientific phenomenon like sound
and light frequencies perceivable for both eyes and ears.”13 His performances, most of graphics and sounds
are generated by computer, shows an intelligent aesthetic since we can find a logic how visuals and sound
are related and affect each other.
A Japanese electronic composer, who often collaborates with Nicolai, Ryoji Ikeda shows interesting
synthetic sound. According to him, “He focuses on the minutiae of ultrasonic, frequencies and the essential
characteristics of sound itself. His work exploits physical property of sound, its causality with human
perception and mathematical dianoia as music, time and space. Using computer and digital technology to
the utmost limit, Ikeda has been developing particular "microscopic" methods for sound engineering and
composition.”14 With Carsten Nicolai, he worked a collaborative project cyclo, which examined error
structures and repetitive loops in software and computer programmed music, with audiovisual modules for
real time sound visualization. He also did live performances for his albums +/- (1996), 0 degrees (1998)
and Matrix (2000), which showed unique style of electric sound. His visuals seem to be logically
explaining the algorithm of sound generation, which effectively emphasizes his sound itself.
While those artists are using new computer technologies for their works, Golan Levin focuses on
designing of a system itself for generating image and sound simultaneously. Scribble (2000: Golan Levin,
Gregory Shakar and Scott Gibbons) is a live audiovisual concert. According to him, “Performed on custom
software, Scribble features tightly-coupled sounds and dynamic abstract visuals which are at times
carefully scored, and at other times loosely improvised. It is performed on the Audiovisual Environment
Suite (AVES), a set of seven interactive systems, which allow people to create and perform abstract
animation and sound, simultaneously, in real time. Each instrument is an experimental attempt to design an
interface.”15
For the last, I can't help mentioning Granular-Synthesis, a co-project of Kurt Hentschlaeger and Ulf
Langheinrich. In Modell 5(1994-96), we can see that even the simplest relationship between visuals and
sound can provide powerful impression when those elements are corresponded in a creative way. I am also
convinced that when spectators can sympathize with the intuitive relationship of visuals and music, it can
be successful.
V. Prototype
12
Ryoichi Kurokawa, http://www.ryoichikurokawa.com
13
Carsten Nicolai, http://www.carstennicolai.com
14
Ryoji Ikeda, http://www.ryojiikeda.com
15
Golan Levin, http://www.flong.com
The term of synaesthesia means the unity of the senses and, by extension, the arts. Sensory perception of
one kind may manifest itself as sensory experience of another. Thus, when two feelings from different
senses are united, one emphasizes one another or these two are mixed together into a new feeling
entertaining us. I want to provide viewers this experience of visual music, not merely maximizing the
emotion of music and providing visual entertainment. I am convinced that this is the ultimate purpose of
my project. Thus, to make people sympathize, the main design object of this project is performing an
intuitive combination of visuals and sound. I will explain my conclusions from the above precedents to
achieve this goal.
Most of all, I will maximize the drama of the combination of image and sound. I should carefully analyze
the music from the single moments to the entire sequence of music and I should focus, release, and develop
the emotion of music with my visual elements.
In a visual aspect, I'd like to emphasize on motions in a 3 dimensional space. We saw numerous of artists
who have been fascinated with cosmic visuals and I agree with the idea that only spatial motions can
represent the property of music. Contrary, I feel a limit of a 2 dimensional motion from some of the
precedents above. Obviously, the advantage of the computer graphics at this moment is the capability of
rendering 3 dimensional images. I will maximize this capability and I expect it will achieve the powerful
synaesthesia of space and music.
On the other hand, among those above precedents, I would like to aim for Jordan Benson's visual style,
which is on the fence between abstraction and representation. I want to deliver meaningful images to
viewers. I don't want my work to carry on a vague lump of emotion. I saw enough how boring a mere
abstractive visual music was. I want viewers to understand mine. Like Bensons, my visuals will be abstract,
but will use representational metaphors. As we can see from the above contemporary precedents, these
days’ codes, which are much easier, are now supporting artists with more realistic images. We can now
easily draw not only simple lines or dots but also meaningful images with the Whitneys’ mathematical
algorithms. Even though the Whitneys’ algorithmic animations successfully visualized the property of
music, but they were not meaningful enough to make people sympathize with it.
VI. Conclusions
Visual Music has a long history as its main medium, a film. Numerous kinds of methods, technologies,
visuals, and sound have been tried to make those two different fields of art interact to each other. As I
mentioned above, these are necessary for a successful visual music: intuitive relationships of image and
sound, modulating drama, spatial motions, and meaningful algorithms. The newly developed computer
technology will help artists to achieve better results and now artists are trying to get something additional
from the combinations of sound and visuals. Ultimately, I hope this project to be a step for developing my
motion graphics. I will maximize the possibility of code and apply it to the process of producing motion
graphics. This algorithmic process, capable of analyzing music and generating motions, will be a great tool
for me seeking for an advanced step of motion graphics.
References
Woolman , Matt. Sonic Graphics : Seeing Sound. New York : Rizzoli, 2000.
Brougher , Kerry. “Visual-Music Culture” In Visual Music : Synaesthesia in Art and Music since 1900,
edited by Kerry Brougher, Jeremy strick, Ari wiseman, and Judith Zilczer, 88-179. New York : Thames &
Hudson, Washington, D.C. : Hirshhorn Museum, Los Angeles : Museum of Contemporary Art, 2005
Fischinger, Oskar. Oskar Fischinger, Ten Films 1900-1967. S.l. : Center for Visual Music, 2006
Starr , Cecile, Oskar Fischinger, and Toshifumi Kawahara. The World of Oskar Fischinger : pioneer of
abstract and advertising animation. Montvale, N.J. : Laserdisc Corporation of America, 1986
Starr, Cecile, John N Whitney, and Toshifumi Kawahara. The World of John Whitney : Pioneer of Visual
Music. Montvale, N.J. : Laserdisc Corporation of America, 1984
Lye, Len. Rhythms. Paris, France: Re: Voir Video, 2001
Belson , Jordan, John Luther Adams, Atlanta Singers, Arctic Chamber Orchestra. Samadhi and other films.
Montauk, NY : Mystic Fire Video distributor 1989
Center for Visual Music. http://www.centerforvisualmusic.org
UBU WEB. http://ubu.com
Kurokawa , Ryoichi. Ryoichi Kurokawa. http://www.ryoichikurokawa.com
Nicolai, Carsten. Carsten Nicolai. http://www.carstennicolai.com
Nicolai, Carsten.. Alva Noto. http://www.alvanoto.com
Ikeda, Ryoji. Ryoji Ikeda. http://www.ryojiikeda.com
Levin , Golan. Flong. http://www.flong.com
Hentschlaeger , Kurt, and Ulf Langheinrich. Granular Synthesis. http://www.granularsynthesis.info/ns