Research When Uncertainty is a Certainty Jim Hazy Adelphi University Garden City, NY To those who do not know mathematics it is difficult to get across a real feeling as to the beauty, the deepest beauty, of nature…. If you want to learn about nature, to appreciate nature, it is necessary to understand the language through which she speaks to us. The Character of Physical Law (1965) Ch. 2 - Richard Feynman 1918 - 1988 Some Theoretical Points Information Theory Implies Uncertainty – What is entropy anyway? Nonlinear dynamical systems (NDS) – Linearization - linear “thinking” often works! Local versus global prediction – Until it doesn’t! New information created Bifurcations & Catastrophes Sensitivity to Initial Conditions (SIC), Divergence (along emergent dimensions) & Deterministic Chaos Why Information Theory? An informal interpretation “Listening” or “watching” for what is happening “signals” when “noise” or uncertainty is a certainty – Recognize information: Some events are predictable, some surprising – Gathering “new information” about the events in the environment To gather new information, one must probe for it – Observer’s “probability model” predicts outcomes - looking for outcomes – Entropy “maps” a model into “questions” to glean info from noise Perfect Predictability, p = 1 No New Info after events – For example: Death is permanent. – Event entropy = 0 Surprise, p < 1 New Info is available after event To clarify this important concept, let the “predictive model” for a fair coin flip be: The variable X = Heads = 1 with probability 1/2 Tails = 0 with probability 1/2 Now there is an event, a coin flip that we “assume” is fair. Here is the question: In the ensemble of all possible outcomes (1 or 0) above, how much information is needed to determine which actually occurred? The entropy of X in a space of 2 outcomes 1 or 0 is: H(X) = - ½ log ½ - ½ log ½ -(.5)*(-1) - (.5)*(-1) .5 + .5 = 1 bit is learned about the state of environment Here’ how to think about this: Based upon the above “model”, it will take only one “yes” or “no” question (base 2), or “probe” or “experiment,” to determine what actually happened during the event. One checks the coin on the ground for heads or tails. In the above example, an observer guessing “was it heads?” will be correct 50% of the time; if wrong, one knows it’s “tails” with 100% confidence. As a result the expected number of “yes” or “no” questions is 1 bit. H(p) | p(x) real numbers as follows: H(X) = - Σ p(x) log2 p(x) x=χ http://pandasthumb.org/pt-archives/entropy.jpg To clarify this important concept, the following example calculation is taken from Cover and Thomas (2006, 15), let the ensemble have 4 possible states post event: X= a with probability 1/2 b with probability 1/4 c with probability 1/8 d with probability 1/8 The entropy of X is: H(X) = - ½ log ½ - ¼ log ¼ - 1/8 log 1/8 – 1/8 log 1/8 = 7/4 or 1.75 bits One way to think about this is that for the thoughtful investigator it will on average take fewer than two “yes” or “no” questions, or “probes” or “experiments,” to determine the precise outcome within the ensemble, that is, which future happened. In the above example, an observer guessing, “was it “a”?” will be correct 50% of the time, so 50% of the time it takes one question; if wrong, guessing “b” will then be right 50% of the time, so 50% of 50% or 25% of the time it takes 2 questions; if wrong, guessing “c” will be right 50% of the time; and if this third guess is also wrong, then it is “d” 100% of the time. Thus, the remaining 25% of the time it takes 3 questions. The expected number of “yes” or “no” questions is = 0.5*1 + .25*2 + .25*3 = 1.75 questions or bits. (There is a theorem that says entropy ≤ # questions). This “regularity” in the random variable – a consistent mix of predictability versus surprise -- in the physical environment can be identified with 1.75 yes/no experiments in the informational environment. This is a “model” of an uncertain environment wherein one can know what happened with 1.75 bits of info. Experimenting to Gather Information Modeling, Experimentation & Analysis Uncertainty About “Phenomenon A” Mutual Information from “Surprises” Uncertainty About Phenomenon B H(X) This line indicates information gathered from the inherent uncertainty in predicting the system being observed; there are always surprises from random events: “A COIN FLIP” L) H μ( ) L H( E + =E * (u ) L hμ The blue is transient information embedded in the complexity of the system & available as events unfold to improve our models: LEARNING THAT THE FLIP IS FAIR T This curve indicates an observer’s naïve model of the phenomenon resulting from insufficient observation, that is, “ignorance” about the “system”: “I FEEL LUCKY” Length L Line Slope = entropy rate = “new mutual information” per observation or symbol L Line Intercept = E = Excess entropy includes observed info thru & improved model One must learn the language of the symbols, L, “spoken” by the system. Adapted from Feldman, McTague, & Crutchfield (2008). Panel 1 Panel 2 A linear stability A Better, Nonlinear model enables local Model Clarifies approximation but doesn’t Robustness fully recognize constraints of Stability Panel 3 An even better bifurcation model opens new possibilities Prior constraints limit maximum performance potential even when human organizing dynamics are optimal Qualitative breakthroughs in performance due to innovation overcome prior constraints Phenomenon Stability models for given levels of people & resources needed or supported Model Transformation model for additional people & resource flow needed/supported Model_N State_N Innovation Dynamics … Accuracy of Prediction Model Driving Agents Choices, a Model_3 State_3 Innovation Dynamics Model_2 State_2 Innovation Dynamics Model_1 Structural Complexity Of Organizing State, b State_1 Innovation Dynamics Model_0 State_0 (Surie & Hazy, 2006) Thank You References Cover, T, & Thomas, J. A. (2004). Elements of Information Theory 2nd Edition. Hoboken, NJ: Wiley-Interscience Crutchfield, J (1994). Is anything ever new? Considering emergence. In G. Cowan, D. Pines & D. Meltzer 9eds.), Complexity: Metaphors, Models, and Realty 515-537, Reading, Massachusetts: Addison Wesley Publishing. Crutchfield J.P. & Feldman, D.P. (1997). Statistical complexity of simple one-dimensional spin systems, Physics Review E 55 (2), 1239-1242. Crutchfield J.P. & Feldman, D.P. (2003). Regularities unseen, randomness observed: The entropy convergence hierarchy. Chaos. 15: 25-54. Epstein, J. M. (1997). Nonlinear dynamics, mathematical biology, and social science (Vol. IV). Reading, Massachusetts: Addison-Wesley Publishing. Feldman D. P., & J. P. Crutchfield (1998). Statistical measures of complexity: Why? Physics Letter A, 238 (415), 244-252. Feldman, D. P., McTague, C. S., & Crutchfield, J. P. (2008). The organization of intrinsic computation; Complexityentropy diagrams and the diversity of natural information processing. Chaos 18, 043106-1. Gell-Mann, M. (1995). The Quark and the Jaguar: Adventures in the simple and the complex. New York: Henry Holt & Company. Haken, H. (2006). Information and self-organization: A macroscopic approach to complex systems. Berlin: Springer. Hazy, J. K. (2008). Toward a theory of leadership in complex adaptive systems: Computational modeling explorations. Nonlinear Dynamics, Psychology and Life Sciences 12(3), 281-310. Osorio, R., Borland, L. & Tsallis, C. (2004). Distributions of High-Frequency Stock Market Observables. In M. Gell-Mann & C. Tsallis, (Eds.) Nonextensive Entropy – Interdisciplinary Applications. Oxford: Oxford University Press. Prigogine, I. (1997). The end of certainty: Time, chaos, and the new laws of nature. New York: Free Press. Prokopenko, M., Boschetti, F. and Ryan, A. J. (2008). An information-theoretic primer on complexity, selforganization and emergence. Complexity (Online) 9999 (9999) NA. Rubinstein, A. (1998). Modeling Bounded Rationality. Cambridge: The MIT Press. Ruelle, D. (1989). Chaotic Evolution and Strange Attractors. Cambridge: Cambridge University Press. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technology Journal 27. 379-423, 623-657. Simon, H. A. (1962). The architecture of complexity. Paper presented at the Proceedings of the American Philosophical Society 106, No. 6. Sterman, J. D. (2000). Business Dynamics: Systems thinking and modeling for a complex world . Boston: Irwin McGraw-Hill.