Class notes are reminders of some of the main points discussed in class. They are not always presented in a rigorous format.
Information Theory.
Claude Shannon and Weaver, W. - The Mathematical Theory of Communication (Urbana, 1949).
Norbert Wiener – Cybernetics (1948).
Louis Brillouin – Science and Information Theory (1956)
Abraham Moles – Information Theory and Esthetic Perception (U of I Press, 1958).
Information as a measurable quantity. Art/music as a communication process: messages are transmitted through channels
that have a certain, measurable capacity. Information is regarded as a value, not considered for its significance. The
efficiency η of the channel is given by:
η =
(Itransmitted – Ireceived) / Itransmitted
The value of information is related to the unexpected, unforeseeable/unpredictable (i.e. probability), original. Originality
and significance are distinct and intelligibility is the inverse of originality ( 1 / originality). Originality is a function of the
improbability of the message:
where I = improbability = 1 / ω (ω = probability of occurrence).
H = K log I
Logarithms. The logarithm of a number to a given base is the power or exponent to which the base must be raised in
order to produce the number. Here are some examples:
100 = 1
101 = 10
102 = 100
log10 1 = 0
log10 10 = 1
log10 100 = 2
20 = 1
21 = 2
22 = 4
log2 1 = 0
log2 2 = 1
log2 4 = 2
100.69897 = 5
log10 5 = 0.69897
10-1 = 0.1
log10 0.1 = -1
Since log 1/ω = log 1 – log ω = 0 – log ω, the expression above becomes:
H = -K log ω
which mirrors Boltzmann's formula for entropy (second law of Thermodynamics):
S = k log ω
where S = measure of disorder and ω = probability of a given state
In the case of binary choices (yes/no questions):
H = -K log ½ = K log2
it defines the unit of information, bit, binary choice: H = -log2 ω.
Abraham Moles: “The quantity of information transmitted by a message is the binary logarithm of the number of choices
necessary to define the message without ambiguity”.
H = -Nt ∑ pi log2 pi
with ∑ from i=1 to i=n
pi = probability of occurrence of symbols drawn from a repertoire
n = symbols in the repertoire
Nt = length of sequence in which the symbols are assembled
N (replacing Nt) could also be the rate of events / second
or M = Nt = the total number of elements in the sequence
Complexity. Both organisms and machines have structure. “To understand them is to understand the structure” (Moles).
The only universal dimension of structure independent of its elements is complexity, the multiplicity of possible situations.
In perceptual terms, it is related to variety – and linked to probability.
Maximum information. ∑ pi log pi might be considered a weighting (averaging) which implies that the maximum
originality or information occurs when
pi = p = 1/n
then
H = -log2 1/n
and
H = k log2 n
and
(equiprobability of occurrence)
∑ pi log2 pi = n pi log2 pi
(with ∑ from i=1 to 1=n)
for n → ∞, H → ∞
Maximum number of possible choices implies maximum entropy or information.
Relative information and redundancy. If H1 is the information content of a given message and H m = -log2 1/n is the
maximum entropy for the same message, the ratio
H1 / H m
is the relative information of the message independent of its length. Redundancy is the complementary magnitude
1 - H1 / H m
Redundancy is a measure of “wastage” of symbols in transmittal of information. Redundancy is 0 when H 1 = Hm and is
(almost) equal to 1 when H1 is very small or 0
We can now draw the following comparisons:
H entropy, disorder
information
originality
unforeseeability
aperiodicity
novel output
complex
1 - H1 / Hm structure, order
redundancy
banality
foreseeability
periodicity
intelligible form
simple
Repetition → regularity → degree of coherence/predictability. Hence, “the amount of periodicity is then a degree of order”
(Moles). Consider pitch vs. noise and meter vs. aperiodic rhythm.
Semantic and esthetic information. Semantic messages are expressible in symbols and determine translatable, logical
decisions. Esthetic messages determine interior states and are untranslatable. In the 1960s it was thought that information
theory is a tool that can deal with perception and with esthetics in a rational, precise way. “Information Theory invades a
territory reserved for philosophy” - Louis Brillouin.
Musical applications. Xenakis uses ataxy (entropy) as another dimension of his screens (see his Markovian music), a
dimension of the musical vector space. Also, consider the following:
complex chords: the more elements they have, the more information, more difficult to understand → perceived as dissonant
enharmonic modulation/chords: create an ambiguous situation, unexpected resolution to a far tonality, high entropy
(information); redundancy produced by the common language (only certain chords allowed) and voice leading. The
resolution releases information.
complex polyphony and polyrhythms: difficult to follow every stream in works by Ives or integral serialists. Xenakis'
criticism – too much information, beyond a certain threshold, music becomes incomprehensible as such. His solution: move
to a different (higher order) level, textures governed by stochastic distributions.
tone-row music: new non-repeating pitches → novelty, information; repetition of intervals → redundancy; combinatoriality
adds to both structure and complexity.
Debussy – creates a gentle, sensual background → redundancy paired with novel scales, chords → information and complex
timbres.
Schoenberg – originality in tone-row idea, klangfarbenmelodie, sprechstimme → information; redundant background
(traditional rhythmic structure, form, orchestration).
Webern – very high originality, strict structure → very high redundancy; extremely complex. Because of high redundancy,
intelligible.
Cage – huge entropy, no organization; there is a limit to the attention (apperceptual limit) above which the channel shuts off.
Why does that not happen here ? 1. Our mind projects form, structure, organization even when there is none; 2. events are
few and spaced in time giving us time to assimilate the sounds (which is what Cage asks us to do anyway).
Timing and pacing of information becomes very important.
Wagner – leitmotives are introduced at the beginning of the opera in an one-to-one reltionship with the character, object or
action they represent: someone tells a story and we hear them in clear association. In later acts they might be in
counterpoint with what a character says on stage (Lohengrin) or depicting unexpected mental associations (Parsifal).
Herbert Brün: the role of the composer is to retard the decay of information. As one listens to a piece it slowly reveals
itself, its structure, the rules of the game; repeated auditions accelerate the process – the more one knows about it, the less
information is derived. There is nothing one can do about this decay but the composer can find strategies that will insure a
slowing down, a retardation of this process. Complexity plays a role: the information delivered by a pop tune decays much
faster than the information of a Beethoven string quartet.