Toward Synthesized Environments:
A Survey of Analysis and Synthesis Methods
for Sound Designers and Composers
Ananya Misra, Perry Cook
Princeton University
Motivation
• Multitude of sounds in the sonic landscape
• Multitude of algorithms
• Better knowledge of a variety of algorithms
=>
empowerment to create rich sound scenes
Some Related Surveys
• Smith. “Viewpoints on the History of Digital Synthesis.”
ICMC 1991.
• Pope. “A Taxonomy of Computer Music.” Contemporary
Music Review 1996.
• Tolonen et al. “Evaluation of Modern Sound Synthesis
Methods.” Tech. Rep. 1998.
• Vercoe et al. “Structured Audio: Creation, Transmission,
and Rendering of Parametric Sound Representations.”
Proc. IEEE 1998.
• Widmer et al. “Sound and Music Computing: Research
Trends and Some Key Issues.” JNMR 2007.
This one: from the perspective of creating complex
environmental sound scenes or compositions
Overview
• Abstract synthesis algorithms
• Synthesis from “scratch”
• Synthesis from existing sounds
– Concatenative techniques
– Additive synthesis
– Subtractive synthesis and other techniques
• Analysis not for synthesis
Disclaimer: No taxonomy is clean or 1-1 from
method to class
Abstract synthesis algorithms
Oscillators
• From analog days
Sine wave
Triangle wave
Sawtooth wave
Alarm clock
Examples by ChucK
Frequency Modulation
• Modulation of one oscillator’s phase by
another’s output
Basic example
Doorbells
Examples by ChucK
More
• Circle maps as nonlinear oscillators (Essl,
ICMC 2006)
• Errant sound synthesis: “the potential of any
algorithm cast into the audio range” (Collins,
ICMC 2008)
• Auditory display and sonification
Synthesis from scratch
Synthesis from physical or perceptual
models, without the raw material of
existing audio samples
Physical models
High-level parametric control over synthesized
sound
• Plucked string model, waveguides and more
• Reed and bowstring models, singing voice
synthesis, percussive sounds
• Synthesis ToolKit => PeRColate (Max/MSP),
ChucK, SuperCollider
• Real-world contact / motion sounds
Modal synthesis
Gait modeling
Cook, 2002
Perceptual models
Give desired perceptual characteristics
• For speech and singing (Cook, CMJ 1996):
– Formant synthesizers
– Formant wave functions (FOFs) (Rodet,
CMJ 1984)
• General: Feature-based synthesis (Hoffman,
ICMC 2006)
=>
Synthesis from existing sounds
Creation of sound from existing sound,
synthesis by analysis
Concatenative techniques
• Rearrangement of samples in the time
domain
• Wavetable synthesis
• Concatenative synthesis (Schwarz, JNMR
2006):
–
–
–
–
Source sound segmented into units
Target sound
Set of unit descriptors
Unit selection algorithm
Concatenative techniques
• Applications: singing, instruments, audio
mosaicing
• Granular synthesis: concatenating usually
short “sound grains” (Truax, 1990)
Concatenative techniques: granular
synthesis
• Formant wave functions, when using arbitrary
sound samples
• Dictionary-based methods with time-localized
waveforms (Sturm, ICMC 2008) => analytical
counterpart
• TAPESTREA: granular synthesis by
parametrically looping transformed events
Concatenative techniques for sound
textures
• Splitting soundscapes into syllable-like
segments (Hoskinson, ICMC 2001)
• Soundscape generation from database of
annotated sound files (Birchfield, ICMC 2005)
• Fast sound texture synthesis using overlapadd (Fröjd, ICMC 2007)
From Fröjd&Horner, 2007;
algorithm offered in TAPESTREA
Additive synthesis
Spectral analysis, addition of resulting signals
• Channel vocoder: bank of bandpass filters
• Phase vocoder: FFT to get phase as well as
magnitude of each frequency band
• Pitch and time transformations (offered in
TAPESTREA)
• Cross-synthesis
Additive synthesis: sinusoidal modeling
• Speech signals can be modeled using a few
sinusoids (McAulay 1986, Quatieri 1986)
• Spectral modeling synthesis (Serra, 1989):
sines + noise
• Lemur/Loris, CLAM, SMS, SPEAR, AudioSculpt
• Used to extract and transform some
environmental sounds in TAPESTREA
Transformed windchimes
Baby chorus / cacophony
Sines + Transients + Noise
Decompose into transients (brief, noisy events)
as well as sines and noise
• Transient / onset detection techniques:
– Time-domain envelope following (in TAPESTREA)
– Comparing energy envelopes of original and
residual noise signals (Levine, 1998)
– Comparing energy in short and long signal
segments (Verma, 1998) (in TAPESTREA)
– Frequency-domain techniques
Subtractive synthesis and linear predictive
coding
• Filtering a signal to shape it by subtracting
unwanted components
• LPC: Source-filter model where next = linear
combination of previous samples (Atal, 1970)
• Musical composition (Lansky, 1989)
• Sound texture synthesis (Athineos, 2003;
Zhu, 2004)
Noise-excited LPC
Time-frequency LPC;
Athineos&Ellis, 2003
Other tools for sound textures
• Wavelet-tree learning (Dubnov, 2002) (offered
in TAPESTREA)
• Parametrically modeling and transforming
stochastic components (Miner, 2002)
• Inferring statistical distributions of events
(Zhu, 2003)
Analysis not for synthesis
Content-based analysis methods not
necessarily designed for synthesis.
May provide information to guide
synthesis algorithms.
Analyze to…
• Represent an audio signal in structurally or
perceptually meaningful ways
• Understand and use a collection of sounds on
a global level
Representing a signal
• Source separation: Computational auditory
scene analysis (Ellis, 1992; Melih, 2000)
– Automatic and manual grouping of partials offered
in TAPESTREA
• Source separation: Multiple fundamental
frequency estimation (Klapuri, 2004)
• Blind source separation (Hoffman, 2009)
• Music transcription
Understanding a collection
• Comparison actions: content-based
classification, search, recommendation, …
• Automatic timbre recognition
• Music information retrieval tools, e.g.
MARSYAS
Conclusions
Advantages of parametric synthesis
algorithms
• Many algorithms may contribute to one
simple piece
• Many available tools: programming
languages, specialized libraries, graphical
software
• Compression
• Ability to re-render over and over with
changes, interactively or in real-time
• Rich palette of techniques for composers
Taxonomy of methods by sounds
Sound/Goal
Methods
Abstract
FM, non-linear oscillators, feature-based synthesis,
wavetables, granular synthesis
Acoustic instruments Wavetables, waveguides / physical models, granular
synthesis, additive synthesis
Contact sounds
Physical models
Cross-synthesis
LPC, vocoders
Pitch/time
transformations
LPC, vocoders, granular synthesis, additive synthesis
Pitched sounds
Additive synthesis, granular synthesis, FM, oscillators
Singing voice
FM, formant synthesis, FOFs, granular synthesis, additive
synthesis
Speech
Formant synthesis, FOFs, granular synthesis, vocoders,
additive synthesis, LPC
Textures and
soundscapes
Granular synthesis, LPC, stochastic and wavelet-based
methods
Transients
Onset detection, physical models, granular synthesis,
sines+transients+noise models