On the Intrinsic Value of Information
Amos Golan
Department of Economics
Director of Info-Metrics Institute
American University
Radu Balan
Department of Mathematics, Center for Scientific Computation
and Mathematical Modeling, Norbert Wiener Center
University of Maryland
31 October 2014
Recent Innovations in Info-Metrics
American University , Washington D.C.
Acknowledgments
Thanks to sponsor:
2
1. Motivation
What the problem is NOT about:
• Information: Choose your own preferred definition of
information (entropy):
– Shannon’s: 𝐼 𝑉 = − 𝑁
𝑘=1 𝑝𝑘 log 2 (𝑝𝑘 )
– Hartley’s: 𝐻 𝑉 = log 2 (𝑁)
– Renyi’s: 𝐻𝛼 𝑉 =
• Complexity:
1
log 2
1−𝛼
𝑁
𝛼
𝑝
𝑘=1 𝑘
– Information/Entropy
– Kolmogorov complexity: shortest length of a universal
coder
3
What this talk IS about: VALUE of Information
• Approach:
– We start from a set of axioms to construct value of
information functions
– We investigate existence and uniqueness of such
functions
– We parameterize various classes of value of
information functions
• Ingredients that may be added at a later time:
– Partial information
– Contradictory information
4
2. Axiomatic
Let X denote a finite information set
𝑋 = 𝑥1 , 𝑥2 , … , 𝑥𝑁
Examples:
• Chapters in a book
• Books in a library
• Entries in the Webster’s dictionary
• Set of logical propositions in a logical system
• List of candidates in an election
5
A value of information function (voif) 𝑣 assigns a number
(value) to a subset 𝐴 of 𝑋:
𝑣: 𝒫 𝑋 → ℝ
Definition. A Type A value of information function is a
function 𝑣: 𝒫 𝑋 → ℝ that satisfies:
P1. (Normalization) 𝑣 ∅ = 0 , 𝑣 𝑋 = 1 ;
P2. (Positivity) 𝑣 𝐴 ≥ 0 for all 𝐴 ∈ 𝒫(𝑋) ;
P3a. (“Concavity Type A”) (Super-additivity)
𝑣 𝐴 ∪ 𝐵 ≥ 𝑣 𝐴 + 𝑣 𝐵 − 𝑣(𝐴 ∩ 𝐵)
for every 𝐴, 𝐵 ∈ 𝒫(𝑋).
We denote by ℱ𝐴 (𝑋) the set of Type A v.o.i.f.’s.
Note ℱ𝐴 𝑋 ⊂ 0,1
2𝑁
⊂ℝ
2𝑁
6
Comments:
• P1+P2: nonnegative and normalized to [0,1]
• Reasonable assumptions provided no contradictory
information, and that X provides full information.
• “Super-additivity”:
– For disjoint sets 𝐴 ∩ 𝐵 = ∅,
𝑣 𝐴 ∪ 𝐵 ≥ 𝑣 𝐴 + 𝑣(𝐵) (*)
Formalizes the statement “value of a system is larger
than the sum of values of its parts”
– For non-disjoint sets, e.g. 𝐴 = 𝐵, (*) cannot be
extended trivially to:
𝑣 𝐴 = 𝑣 𝐴 ∪ 𝐴 ≥ 𝑣 𝐴 + 𝑣 𝐴 = 2𝑣(𝐴)
The formal axiom P3a removes the value of intersection.
7
Axiomatic – Part 2
The concavity hypothesis admits a different form:
Definition. A Type B Value of Information Function is a
function 𝑣: 𝒫 𝑋 → ℝ that satisfies:
P1. (Normalization) 𝑣 ∅ = 0 , 𝑣 𝑋 = 1 ;
P2. (Positivity) 𝑣 𝐴 ≥ 0 for every 𝐴 ∈ 𝒫(𝑋) ;
P3b. (“Concavity Type B”):
𝑣 𝐴 ∪ 𝐵 ≥ 𝑣 𝐴 + 𝑣(𝐵)
for every 𝐴, 𝐵 ∈ 𝒫(𝑋) so that 𝐴 ⊄ 𝐵 and 𝐵 ⊄ 𝐴.
We denote by ℱ𝐵 (𝑋) the set of Type B v.o.i.f.’s.
Note ℱ𝐵 𝑋 ⊂ 0,1
2𝑁
⊂
𝑁
2
ℝ
8
Discussion of P3b:
P3b is a different generalization of the inequality (*) ,
which is P3a for disjoint sets.
In general:
• P3a implies a growth at least linearly in terms of
subset size : 𝑣 𝐴 ≥ 𝑐|𝐴|
• P3b implies a growth at least exponentially in terms
of subset size: 𝑣 𝐴 ≥ 𝑐1 𝑒 𝑐2 |𝐴| .
9
Type A VOIF: Extreme Points (1)
We seek extreme points (vertices) of the convex set ℱA (X).
“Dictatorial” Extreme Points
In fact for any ∅ ≠ 𝐴 ⊂ 𝑋, the following type A VOIF
1 𝑖𝑓
𝐴⊂𝐵
𝜑𝐴 : 𝒫 𝑋 → ℝ , 𝜑𝐴 𝐵 =
0 𝑖𝑓 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
is an extreme point in ℱ𝐴 (𝑋).
Explicitly: if 𝜑𝐴 = 𝑡𝑣1 + 1 − 𝑡 𝑣2 for some 0 < 𝑡 < 1 and 𝑣1 , 𝑣2
type A voif’s then 𝑣1 = 𝑣2 = 𝜑𝐴 .
10
Type A VOIF: Extreme Points (2)
Another class of extreme points in ℱ𝐴 𝑋 :
“Democratic” (Equalitarian) Extreme Points
For every 2 ≤ 𝑑 ≤ 𝑁 − 1, the following type A VOIF
𝜑
𝑑
𝐴 =
𝐴 −𝑑+1
𝑖𝑓
𝑁−𝑑+1
0
𝑖𝑓
𝐴 ≥𝑑
𝐴 <𝑑
is an extreme point.
11
Type B VOIF: Extreme Points (1)
The situation is slightly more complicated. So far we
identified the following classes of extreme points:
The “dictatorial” voif’s:
For each subset 𝐴 ∈ 𝒫(𝑋),
Ψ𝐴,1
2 𝐵 −𝑁
𝐵 =
0
𝑖𝑓
𝑖𝑓
𝐴⊊𝐵
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
A second type of dictatorial voif:
Ψ𝐴,2
2 𝐵 −𝑁
𝐵 = 2 𝐴 −𝑁+1
0
𝑖𝑓
𝑖𝑓
𝑖𝑓
𝐴⊊𝐵
𝐴=𝐵
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
12
Type B VOIF: Extreme Points (2)
The “democratic” (“equalitarian”) Type B VOIF extreme
points are given by:
Ψ(𝑑)
2 𝐴 −𝑁
𝐴 =
0
𝑖𝑓
𝑖𝑓
𝐴 ≥𝑑
𝐴 <𝑑
where 2 ≤ 𝑑 ≤ 𝑁.
Note Ψ (𝑑) factors through the cardinal function.
13
4. An Application: List Rankings
Consider 𝑋 = 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒1 , … , 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑁 and let 𝑣: 𝒫 𝑋 →
ℝ represent an assignment of preferences.
Assume v is a Type B VOIF (suitable for “game changer”):
𝑣 𝐴 = max 𝑣 𝐴 ∖ 𝑖 + 𝑣(𝐴 ∖ 𝑗 ) ,
𝐴 ≥2
𝑖,𝑗∈𝐴
𝑖≠𝑗
1
Assume 𝑣 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑘 = 𝑎𝑘 for 1 ≤ 𝑘 ≤ 𝑁, where each
𝑍
(𝑎1 , … , 𝑎𝑁 ) is drawn randomly from a fixed distribution, and 𝑍 is
the normalization factor so that 𝑣 𝑋 = 1.
Problem: How many distinct rankings can be achieved on 𝒫(𝑋)
and what are their probability distributions.
14
List Rankins: Results
We consider the case of uniform distribution for
individual preferences 𝑎1 , … , 𝑎𝑁 .
Results over 1,000,000 simulations:
𝑋 =𝑁
2
3
4
5
6
7
4
8
16
32
64
128
#𝑃𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝑅𝑎𝑛𝑘𝑖𝑛𝑔𝑠
= 𝒫 𝑋 −2 !
= 2𝑁 − 2 !
2
720
8.717 1010
2.652 1032
3.147 1085
2.372 10211
# 𝐴𝑐ℎ𝑖𝑒𝑣𝑒𝑑 𝑅𝑎𝑛𝑘𝑖𝑛𝑔𝑠
2
12
480
197379
?
?
𝒫 𝑋
= 2𝑁
15
List Rankings: Results (2)
N = 3 , 2N = 8 , (2N-2)!= 720 , Achieved=12
16
List Rankings: Results (3)
N = 4, 2N = 16 , (2N-2)!= 8.717 1010, Achieved=480
17
List Rankings: Results (4)
N = 5 , 2N = 32 , (2N-2)!= 2.652 1032 , Achieved=197379
18
5. Conclusions
• In this talk we studied the class of concave (super additive)
nonnegative normalized functions quantifying value of
information. Motivation: Value of Information.
• We show their set form a convex set.
• We identified classes of extreme points (vertices).
• We presented an application on list rankings.
19
Thank you!
20