Physical Fluctuomatics
2nd Mathematical Preparations (1): Probability and statistics
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
kazu@smapip.is.tohoku.ac.jp
http://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics (Tohoku
University)
1
Probability
a. Event and Probability
b. Joint Probability and Conditional
Probability
c. Bayes Formula, Prior Probability and
Posterior Probability
d. Discrete Random Variable and
Probability Distribution
e. Continuous Random Variable and
Probability Density Function
f. Average, Variance and Covariance
g. Uniform Distribution
h. Gauss Distribution
Physics Fluctuomatics (Tohoku
University)
2
Event, Sample Space and Event
Experiment: Experiments in probability theory means that
outcomes are not predictable in advance. However, while
the outcome will not be known in advance, the set of all
possible outcomes is known
Sample Point: Each possible outcome in the experiments.
Sample Space：The set of all the possible sample points in the
experiments
Event：Subset of the sample space
Elementary Event：Event consisting of one sample point
Empty Event：Event consisting of no sample point
Physics Fluctuomatics (Tohoku
University)
3
Various Events
Whole Events Ω:Events consisting of all sample points
of the sample space.
Complementary Event of Event A: Ac=Ω╲A
Defference of Events A and B: A╲B
Union of Events A and B: A∪B
Intersection of Events A and B: A∩B
Events A and B are exclusive of each other: A∩B=Ф
Events A, B and C are exclusive of each other:
[A∩B=Ф]Λ[B∩C=Ф]Λ[C∩A=Ф]
Physics Fluctuomatics (Tohoku
University)
4
Empirically Definition of Probability
Definition by Laplace: Let us suppose that the total
number of all the sample points is N and they can occur
equally Likely. Probability of an event A with N sample
points is defined by p=n/N.
Statistical Definition: Let us suppose that an event A
occur r times when the same experiment are repeated R
times. If the ratio r/R tends to a constant value p as the
number of times of the experiments R go to infinity, we
define the value p as probability of event A.
r
 p R   
Pr A  p
R

Physics Fluctuomatics (Tohoku
University)
5
Definition of Probability
Definition of Kolmogorov: Probability Pr{A} for every
event A in the specified sample space Ω satisfies the
following three axioms:
Axion 1:
PrA  0
Axion 2:
Pr  1
Axion 3: For every events A, B that are exclusive of
each other, it is always valid that
PrA  B  PrA PrB
Physics Fluctuomatics (Tohoku
University)
6
Joint Probability and Conditional Probability
Probability of Event A
Pr{A}
Joint Probability of Events A and B
PrA, B  PrA  B
Conditional Probability of Event A
when Event B has happened.
PrA, B
PrB A 
PrA
 PrA, B  PrB APrA
Physics Fluctuomatics (Tohoku
University)
A
B
7
Joint Probability and Independency of Events
Events A and B are independent of each other
PrA, B  PrAPrB
In this case, the conditional probability
can be expressed as
PrB A  PrB
A
A
B
B
Physics Fluctuomatics (Tohoku
University)
8
Marginal Probability
Let us suppose that the sample space  is expressed by
Ω=A1∪A2∪…∪AM where every pair of events Ai and Aj is
exclusive of each other.
M
PrB   PrAi , B
i 1
Marginal Probability of Event B for
Joint Probability Pr{Ai,B}
PrB   PrA, B
Simplified Notation
A
Ai
B
Marginalize
A
B
Summation over all the possible events in which every pair
of events are exclusive of each other.
Physics Fluctuomatics (Tohoku
University)
9
Four Dimensional Point Probability
and Marginal Probability
Marginal Probability of Event B
PrB   PrA, B, C , D
A
Marginalize
C
D
A
B
C
D
Physics Fluctuomatics (Tohoku
University)
10
Derivation of Bayes Formulas
PrA, B  PrA BPrB
Physics Fluctuomatics (Tohoku
University)
11
Derivation of Bayes Formulas
PrA, B  PrA BPrB
PrA, B  PrB APrA
Physics Fluctuomatics (Tohoku
University)
12
Derivation of Bayes Formulas
PrA, B  PrA BPrB
PrA, B  PrB APrA
PrA, B
PrA B 
PrB
Physics Fluctuomatics (Tohoku
University)
13
Derivation of Bayes Formulas
PrA, B  PrA BPrB
PrA, B  PrB APrA
PrA, B PrB APrA
PrA B 

PrB
PrB
Physics Fluctuomatics (Tohoku
University)
14
Derivation of Bayes Formulas
PrA, B  PrA BPrB
PrA, B  PrB APrA
PrA, B PrB APrA
PrA B 

PrB
PrB

PrB  
A
PrB APrA
PrA, B

PrA, B
A
Physics Fluctuomatics (Tohoku
University)
15
Derivation of Bayes Formulas
PrA, B  PrA BPrB
PrA, B  PrB APrA
PrA, B PrB APrA
PrA B 

PrB
PrB

PrB  
PrB APrA

PrB APrA
PrA, B  PrB APrA

PrA, B
A
A
A
Physics Fluctuomatics (Tohoku
University)
16
Derivation of Bayes Formulas
PrA, B  PrA BPrB
A
PrA, B  PrB APrA
B
PrA, B PrB APrA
PrA B 

PrB
PrB

PrB  
PrB APrA

PrB APrA
PrA, B  PrB APrA

PrA, B
A
A
A
Physics Fluctuomatics (Tohoku
University)
17
Bayes Formula
PrA B 
PrB APrA
 PrB APrA
Prior
Probability
A
A
Posterior Probability
It is often referred to as Bayes Rule.
B
Bayesian Network
Physics Fluctuomatics (Tohoku
University)
18
Probability and Random Variable
We introduce a one to one mapping X(A) from every
events A to a mutual different real number. The
mapping X(A) is referred to as Random Variable of A.
The random variable X(A) is often denoted by just the
notation X.
Probability of the event X=x that the random variable X
takes a real number x is denoted by Pr{X=x}. Here, x is
referred to as the state of the random variable X．The set
of all the possible states is referred to as State Space.
If events X=x and X=x’ are exclusive of each other, the
states x and x’ are excusive of each other.
Physics Fluctuomatics (Tohoku
University)
19
Discrete Random Variable and
Continuous Random Variable
Discrete Random Variable:
Random Variable in Discrete State Space
Example:{x1,x2,…,xM}
Continuous Random Variable:
Random Variable in Continuous State Space
Example：(−∞,+∞)
Physics Fluctuomatics (Tohoku
University)
20
Discrete Random Variable and
Probability Distribution
Let us suppose that the sample  is expressed by
Ω=A1∪A2∪…∪AM where every pair of events Ai and Aj
are exclusive of each other.
We introduce a one to one mapping X:Ai xi
(i=1,2,…,M).
If all the probabilities for the events X=x1, X=x2,…, X=xM are
expressed in terms of a function P(x) as follows:
PrX  x  Px x  x1 , x2 ,, xM 
Random Variable
State Variable
State
the function P(x) and the variable x are referred to as
Probability Distribution and State Variable, respectively.
Physics Fluctuomatics (Tohoku
University)
21
Discrete Random Variable and
Probability Distribution
Probability distributions have the following properties:
0  Pxi   1 i  1,2,, M 
M
 Px   1
i 1
i
Normalization Condition
Physics Fluctuomatics (Tohoku
University)
22
Average and Variance
Average of Random Variable X : μ
M
  E X    xi Pxi 
i 1
Variance of Random Variable X: σ2
M
  V X    xi    Pxi 
2
2
i 1
：Standard Deviation
Physics Fluctuomatics (Tohoku
University)
23
Discrete Random Variable and
Joint Probability Distribution
If the joint probability
Pr{(X=x)∩(Y=y)}= Pr{X=x,Y=y}
is expressed in terms of a function P(x,y) as
follows:
PrX  x, Y  y  Px, y 
P(x,y) is referred to as Joint Probability Distribution.
X
 
Y 
Probability Vector
 x
 
 y
Physics Fluctuomatics (Tohoku
University)
State Vector
24
Discrete Random Variable and
Marginal Probability Distribution
Let us suppose that the sample  is expressed by
Ω=A1∪A2∪…∪AM where every pair of events Ai and Aj are
exclusive of each other.
We introduce a one to one mapping X:Ai xi (i=1,2,…,M).
M
PY  y    P xi , y 
Marginal
Probability
Distribution
i 1
PY  y    P x, y  Simplified Notation
x
Summation over all the possible events in which every pair of events are exclusive of each other.
 P( x, y)  1
x
y
Normalization Condition
Physics Fluctuomatics (Tohoku
University)
25
Discrete Random Variable and
Marginal Probability
Marginal Probability of High Dimensional Probability Distribution
Marginal Probability Distribution
PY  y    Px, y, z, u 
x
z
u
X
Y
Z
U
Marginalize
Physics Fluctuomatics (Tohoku
University)
26
Independency of
Discrete Random Variable
If random variables X and Y are independent of each others,
Px, y   P1 xP2  y 
Joint Probability
Distribution of Random
Variables X and Y
Marginal Probability
Distribution of
Random Varuiable Y
 P1 ( x)   P2 ( y)  1
x
y
Probability Distrubution of
Random Variable Y
Probability Distribution of
Random Variable X
PY  y    Px, y   P2  y 
x
Physics Fluctuomatics (Tohoku
University)
27
Covariance of
Discrete Random Variables
Covariance of Random Variables X and Y
CovX , Y    xi   X y j  Y Pxi , y j 
M
N
i 1 j 1
 X  E[ X ]   xi Pxi , y j  Y  E[Y ]   yi Pxi , y j 
M
M
N
i 1 j 1
i 1 j 1
Cov[ X , X ]  V [ X ]
Covariance Matrix
N
Cov[Y , Y ]  V [Y ]
Cov[ X , Y ] 
 V[ X ]

R  
V[Y ] 
 Cov[Y , X ]
Physics Fluctuomatics (Tohoku
University)
28
Example of Probability Distribution
exp ax 
x  1
P( x) 
2 cosh a 
EX   tanh a 
E[X]
VX   1  tanh a 
2
Physics Fluctuomatics (Tohoku
University)
0
a
29
Example of Joint Probability Distributions
exp axy
x  1, y  1
P ( x, y ) 
4 cosh a 
EX   0 VX   1
Cov[ X , Y ]  EXY 
Cov[X,Y]
0
a
 tanh a 
Physics Fluctuomatics (Tohoku
University)
30
Example of Conditional Probability Distribution
Conditional
Probability of Binary
Symmetric Channel
1 x , y
P( y x)  p
x  1, y  1
exp axy

1  p  
2 cosh a 
x,y
1 1 p 

a  ln 
2  p 
Physics Fluctuomatics (Tohoku
University)
31
Continuous Random Variable and
Probability Density Function
For a random variable X defined in the state space
(−∞,+∞), the probability that the state x is in the
interval (a,b) in expressed as
Pra  X  b  Pr   X  b Pr   X  a
F  x   Pr   X  x
Distribution Function
Pra  X  b  F b  F a     x dx
b
a
Probability Density Function
dF x 
 x  
dx
Physics Fluctuomatics (Tohoku
University)
32
Continuous Random Variable and
Probability Density Function
 x  0    x  



 x dx  1
Normalization Condition
Physics Fluctuomatics (Tohoku
University)
33
Average and Variance of
Continuous Random Variable
Average of Random Variable X
  EX    x x dx


Variance of Random Variable X
  V X   
2


x     x dx

Physics Fluctuomatics (Tohoku
University)
2
34
Continuous Random variables and
Joint Probability Density Function
For random variables X and Y defined in the state
と probability
Y の状態空間
において状
space確率変数
(−∞,+∞),Xthe
that(−∞,+∞)
the state
vector (x,y)
態 xregion
と y が区間
(a,b)×(c,d)
にある確率
is in the
(a,b)(c,d)
is expressed
as
Pra  X  b   c  Y  d 

d
c

b
a
 x, y dxdy
Joint Probability Density Function
 
   x, y dxdy 1
Normalization Condition
 
Physics Fluctuomatics (Tohoku
University)
35
Continuous Random Variables and
Marginal Probability Density Function

Y  y     x, y dx

Marginal Probability Density Function of
Random Variable Y
Physics Fluctuomatics (Tohoku
University)
36
Independency of
Continuous Random Variables
Random variables X and Y are independent
of each other.
 x, y   1 x2  y 
Joint Probability
Density Function
of X and Y

  1 ( x)dx  1

   2 ( y)dy  1
Probability Density Function of Y
Probability Density Function of X

Marginal
Probability Density
Function Y
Y  y     x, y dx   2  y 

Physics Fluctuomatics (Tohoku
University)
37
Covariance of
Continuous Random Variables
Covariance of Random Variables X and Y
CovX , Y   
 





x


y



x
,
y
dxdy
X
Y
 
 X  E[ X ]  
Y
 

 E[Y ]   
 
 
 
Cov[ X , X ]  V [ X ]
Covariance Matrix
x x, y dxdy
y x, y dxdy
Cov[Y , Y ]  V [Y ]
Cov[ X , Y ] 
 V[ X ]

R  
V[Y ] 
 Cov[Y , X ]
Physics Fluctuomatics (Tohoku
University)
38
Uniform Distribution U(a,b)
Probability Density Function of Uniform Distribution
b  a 
 x   
 0
1
a  x  b 
   x  a, b  x  
ab
E X  
2
2

b  a
V X  
12
p(x)
(b-a)-1
Physics Fluctuomatics (Tohoku
University)
0 a
b
x
39
Gauss Distribution N(μ,σ2)
Probability Density Function of Gauss Distribution
with average μ and variance σ2
(  0)
 1
2
 x  
exp   2 x        x  
 2
 p(x)
2 2
1
E X   
VX   
The average and the variance are derived
by means of Gauss Integral Formula
2
0
μ
x
 1 2
 exp   2  d  2

Physics Fluctuomatics (Tohoku
University)
40
Multi-Dimensional Gauss Distribution
For a positive definite real symmetric matrix C, twoDimensional Gaussian Distribution is defined by
 1
x  X 
1 
 
 x, y  
exp   x   X , y  Y C 
 y  Y  
2 2 det C  2
1
   x  ,    y  
The covariance matrix is given in terms of the matrix C as follows:
Cov[ X , Y ] 
 V[ X ]

  C
V[Y ] 
 Cov[Y , X ]
by using the following d -dimentional Gauss integral formula
 

 1  T 1   
d
       exp   2 
C  d 

Physics Fluctuomatics (Tohoku
University)
2 
det C
41
Law of Large Numbers
Let us suppose that random variables X1,X2,...,Xn are
identical and mutual independent random variables with
average . Then we have
1
Yn  ( X 1  X 2    X n )   (n  )
n
Central Limit Theorem
We consider a sequence of independent, identical distributed
random variables, {X1,X2,...,Xn}, with average  and variance 2.
Then the distribution of the random variable
Yn 
1
( X1  X 2    X n )
n
tends to the Gauss distribution with average  and variance
2/n as n+.
Physics Fluctuomatics (Tohoku
University)
42
Summary
a. Event and Probability
b. Joint Probability and Conditional
Probability
c. Bayes Formula, Prior Probability and
Posterior Probability
d. Discrete Random Variable and
Probability Distribution
e. Continuous Random Variable and
Probability Density Function
f. Average, Variance and Covariance
g. Uniform Distribution
h. Gauss Distribution
Physics Fluctuomatics (Tohoku
University)
Last Talk
Present Talk
43
Practice 2-1
Let us suppose that a random variable X takes binary
values 1 and the probability distribution is given by
exp ax 
x  1
P( x) 
2 cosh a 
Derive the expression of average E[X] and variance V[X]
and draw their graphs by using your personal computer.
Physics Fluctuomatics (Tohoku
University)
44
Practice 2-2
Let us suppose that random variables X and Y take
binary values 1 and the joint probability distribution is
given by
exp axy
x  1, y  1
P ( x, y ) 
4 cosh a 
Derive the expressions of Marginal Probability
Destribution of X, P(X), and the covariance of X and
Y, Cov[X,Y].
Physics Fluctuomatics (Tohoku
University)
45
Practice 2-3
Let us suppose that random variables X and Y take
binary values 1 and the conditional probability
distribution is given by
1 x , y
P( y x)  p
1  p 
 x,y
Show that it is rewritten as
exp axy
P( y x) 
2 cosh a 
Hint
p  exp ln p
 x, y
1 1 p 

a  ln 
2  p 
1
 1  xy
2
x  1, y  1
cosh(c) is an even function for any real number c.
Physics Fluctuomatics (Tohoku
University)
46
Practice 2-4
Prove the Gauss integral formula:


0

 1 2
exp    d 
2
 2 
Hint
R
R
 1 2
 1 2
 1 2
exp    d  2 lim  exp    d
 exp   2  d  Rlim
   R
R   0
 2 
 2 

 2 lim
R  
R
R
0
0
 
 1 2 1 2
exp      d d  2 lim
R  
2 
 2
Physics Fluctuomatics (Tohoku
University)
 1 2 
r 1 exp   2 r dr
47
Practice 2-5
Let us suppose that a continuous random variable X takes any
real number and its probability density function is given by
 1
2
p x  
exp   2 x    
 2

2 2
1
   x  
Prove that the average E[X] and the variance V[X] are given by
E X   
VX   
2
Draw the graphs of p(x) for μ=0, σ=10, 20, 40 by using
your personal computer.
Physics Fluctuomatics (Tohoku
University)
48
Practice 2-6
Make a program for generating random numbers of
uniform distribution U(0,1). Draw histgrams for N
generated random numbers for N=10, 20, 50, 100 and
1000.
1
x
rand()
randmax
In the C language, you can use the function rand()
that generate one of values 0,1,2,…,randmax,
randomly. Here, randmax is the maximum value
of outputs of rand().
Physics Fluctuomatics (Tohoku
University)
49
Practice 2-7
Make a program that generates random numbers of Gauss
distribution with average  and variance σ2. Draw
histgrams for N generated random numbers for N=10, 20,
50, 100 and 1000.
Hint:
For n random numbers x1,x2,…,xn generated by any probability
distribution, (x1+x2+…+xn )/n tends to the Gauss distribution with
average m and variance σ2/n for sufficient large n. [Central Limit
Theorem]
First we have to generate twelve uniform random numbers x1,x2,…,x12
in the interval [0,1].
Gauss random
number with average
1
2
12
0 and variaince 1
  x  x   x  6
σξ+μ generate Gauss random numbers with average μ and variance σ2
Physics Fluctuomatics (Tohoku
University)
50
Practice 2-8
For any positive integer d and d d positive definite
real symmetric matrix C, prove the following ddimensional Gauss integral formulas:



 1   T 1  
       exp   2    C   
 
Hint:


 
d 

2 d det C
By using eigenvalues λi and their corresponding

eigenvectors ui (i=1,2,…,d) of the matrix C, we have
 1 0

 0 2
C  U 0 0

 
0 0

0
0
3

0
 0

 0
 0 U 1

  
 d 
 

U  u1 , u1 ,, ud 
Physics Fluctuomatics (Tohoku
University)
51
Practice 2-9
We consider continuous random variables X1,X2,…,,Xd.
The joint probability density function is given by

px  
 1   T 1   
exp   x    C x   
d

2  det C  2
1


 x 1 


 
   x2 
d 


x




,



  


 
x 


 d



Prove that the average vector is  and the covariance
matrix is C.
Physics Fluctuomatics (Tohoku
University)
52
Summary
a. Event and Probability
b. Joint Probability and Conditional
Probability
c. Bayes Formulas, Prior Probability and
Posterior Probability
d. Discrete Random Variable and Probability
Distribution
e. Continuous Random Variable and
Probability Density Function
f. Average, Variance and Covariance
g. Uniform Distribution
h. Gauss Distribution
Physics Fluctuomatics (Tohoku
University)
53