Research When
Uncertainty is a Certainty
Jim Hazy
Adelphi University
Garden City, NY
To those who do not know mathematics it is
difficult to get across a real feeling as to the
beauty, the deepest beauty, of nature…. If you
want to learn about nature, to appreciate nature,
it is necessary to understand the language
through which she speaks to us.
The Character of Physical Law (1965) Ch. 2
- Richard Feynman 1918 - 1988
Some Theoretical Points
Information Theory Implies Uncertainty
– What is entropy anyway?
Nonlinear dynamical systems (NDS)
– Linearization - linear “thinking” often works!
Local versus global prediction
– Until it doesn’t!  New information created
Bifurcations & Catastrophes
Sensitivity to Initial Conditions (SIC), Divergence (along
emergent dimensions) & Deterministic Chaos
Why Information Theory?
An informal interpretation
“Listening” or “watching” for what is happening “signals”
when “noise” or uncertainty is a certainty
– Recognize information: Some events are predictable, some surprising
– Gathering “new information” about the events in the environment
To gather new information, one must probe for it
– Observer’s “probability model” predicts outcomes - looking for outcomes
– Entropy “maps” a model into “questions” to glean info from noise
Perfect Predictability, p = 1  No New Info after events
– For example: Death is permanent.
– Event entropy = 0
Surprise, p < 1 New Info is available after event
To clarify this important concept, let the “predictive model” for a fair coin flip be:
The variable X =
Heads = 1 with probability 1/2
Tails = 0 with probability 1/2
Now there is an event, a coin flip that we “assume” is fair.
Here is the question: In the ensemble of all possible outcomes (1 or 0) above, how
much information is needed to determine which actually occurred?
The entropy of X in a space of 2 outcomes 1 or 0 is:
H(X) =
- ½ log ½ - ½ log ½
-(.5)*(-1) - (.5)*(-1)
.5
+
.5
= 1 bit is learned about the state of environment
Here’ how to think about this: Based upon the above “model”, it will take only one
“yes” or “no” question (base 2), or “probe” or “experiment,” to determine what actually
happened during the event. One checks the coin on the ground for heads or tails.
In the above example, an observer guessing “was it heads?” will be correct 50% of
the time; if wrong, one knows it’s “tails” with 100% confidence. As a result the
expected number of “yes” or “no” questions is 1 bit.
H(p) | p(x)  real numbers as follows:
H(X) = - Σ p(x) log2 p(x)
x=χ
http://pandasthumb.org/pt-archives/entropy.jpg
To clarify this important concept, the following example calculation is taken from
Cover and Thomas (2006, 15), let the ensemble have 4 possible states post event:
X=
a with probability 1/2
b with probability 1/4
c with probability 1/8
d with probability 1/8
The entropy of X is:
H(X) = - ½ log ½ - ¼ log ¼ - 1/8 log 1/8 – 1/8 log 1/8 = 7/4 or 1.75 bits
One way to think about this is that for the thoughtful investigator it will on average
take fewer than two “yes” or “no” questions, or “probes” or “experiments,” to
determine the precise outcome within the ensemble, that is, which future happened.
In the above example, an observer guessing, “was it “a”?” will be correct 50% of the
time, so 50% of the time it takes one question; if wrong, guessing “b” will then be
right 50% of the time, so 50% of 50% or 25% of the time it takes 2 questions; if
wrong, guessing “c” will be right 50% of the time; and if this third guess is also wrong,
then it is “d” 100% of the time. Thus, the remaining 25% of the time it takes 3
questions. The expected number of “yes” or “no” questions is = 0.5*1 + .25*2 + .25*3
= 1.75 questions or bits. (There is a theorem that says entropy ≤ # questions).
This “regularity” in the random variable – a consistent mix of predictability versus
surprise -- in the physical environment can be identified with 1.75 yes/no experiments
in the informational environment. This is a “model” of an uncertain environment
wherein one can know what happened with 1.75 bits of info.
Experimenting to Gather Information
Modeling, Experimentation
& Analysis
Uncertainty
About
“Phenomenon A”
Mutual
Information
from
“Surprises”
Uncertainty
About
Phenomenon B
H(X)
This line indicates information gathered from the
inherent uncertainty in predicting the system
being observed; there are always surprises from
random events: “A COIN FLIP”
L)
H μ(
)
L
H(
E
+
=E
*
(u )
L
hμ
The blue is transient information embedded in
the complexity of the system & available as
events unfold to improve our models:
LEARNING THAT THE FLIP IS FAIR
T
This curve indicates an observer’s naïve model
of the phenomenon resulting from insufficient
observation, that is, “ignorance” about the
“system”: “I FEEL LUCKY”
Length L
Line Slope = entropy rate = “new mutual information” per observation or symbol L
Line Intercept = E = Excess entropy includes observed info thru & improved model
One must learn the language of the symbols, L, “spoken” by the system.
Adapted from Feldman, McTague, & Crutchfield (2008).
Panel 1
Panel 2
A linear stability
A Better, Nonlinear
model enables local
Model Clarifies
approximation but doesn’t
Robustness
fully recognize constraints
of Stability
Panel 3
An even better
bifurcation model
opens new possibilities
Prior constraints limit maximum performance potential
even when human organizing dynamics are optimal
Qualitative breakthroughs in
performance due to innovation
overcome prior constraints
Phenomenon
Stability models
for given levels of
people &
resources needed
or supported
Model
Transformation
model for
additional people
& resource flow
needed/supported
Model_N
State_N
Innovation
Dynamics
…
Accuracy of
Prediction
Model Driving
Agents
Choices, a
Model_3
State_3
Innovation
Dynamics
Model_2
State_2
Innovation
Dynamics
Model_1
Structural
Complexity
Of
Organizing
State, b
State_1
Innovation
Dynamics
Model_0
State_0
(Surie & Hazy, 2006)
Thank You
References
Cover, T, & Thomas, J. A. (2004). Elements of Information Theory 2nd Edition. Hoboken, NJ: Wiley-Interscience
Crutchfield, J (1994). Is anything ever new? Considering emergence. In G. Cowan, D. Pines & D. Meltzer 9eds.),
Complexity: Metaphors, Models, and Realty 515-537, Reading, Massachusetts: Addison Wesley Publishing.
Crutchfield J.P. & Feldman, D.P. (1997). Statistical complexity of simple one-dimensional spin systems, Physics
Review E 55 (2), 1239-1242.
Crutchfield J.P. & Feldman, D.P. (2003). Regularities unseen, randomness observed: The entropy convergence
hierarchy. Chaos. 15: 25-54.
Epstein, J. M. (1997). Nonlinear dynamics, mathematical biology, and social science (Vol. IV). Reading,
Massachusetts: Addison-Wesley Publishing.
Feldman D. P., & J. P. Crutchfield (1998). Statistical measures of complexity: Why? Physics Letter A, 238 (415),
244-252.
Feldman, D. P., McTague, C. S., & Crutchfield, J. P. (2008). The organization of intrinsic computation; Complexityentropy diagrams and the diversity of natural information processing. Chaos 18, 043106-1.
Gell-Mann, M. (1995). The Quark and the Jaguar: Adventures in the simple and the complex. New York: Henry
Holt & Company.
Haken, H. (2006). Information and self-organization: A macroscopic approach to complex systems. Berlin:
Springer.
Hazy, J. K. (2008). Toward a theory of leadership in complex adaptive systems: Computational modeling
explorations. Nonlinear Dynamics, Psychology and Life Sciences 12(3), 281-310.
Osorio, R., Borland, L. & Tsallis, C. (2004). Distributions of High-Frequency Stock Market Observables. In M.
Gell-Mann & C. Tsallis, (Eds.) Nonextensive Entropy – Interdisciplinary Applications. Oxford: Oxford University
Press.
Prigogine, I. (1997). The end of certainty: Time, chaos, and the new laws of nature. New York: Free Press.
Prokopenko, M., Boschetti, F. and Ryan, A. J. (2008). An information-theoretic primer on complexity, selforganization and emergence. Complexity (Online) 9999 (9999) NA.
Rubinstein, A. (1998). Modeling Bounded Rationality. Cambridge: The MIT Press.
Ruelle, D. (1989). Chaotic Evolution and Strange Attractors. Cambridge: Cambridge University Press.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technology Journal 27. 379-423,
623-657.
Simon, H. A. (1962). The architecture of complexity. Paper presented at the Proceedings of the American
Philosophical Society 106, No. 6.
Sterman, J. D. (2000). Business Dynamics: Systems thinking and modeling for a complex world . Boston: Irwin
McGraw-Hill.