CSE 473/573
Computer Vision and Image
Processing (CVIP)
Ifeoma Nwogu
Lecture 27 – Overview of probability concepts
1
Schedule
• Last class
– Object recognition and bag-of-words
• Today
– Overview of probability models
• Readings for today:
– Lecture notes
2
Random variables
• A random variable x denotes a quantity that is
uncertain
• May be result of experiment (flipping a coin) or a real
world measurements (measuring temperature)
• If observe several instances of x we get different
values
• Some values occur more than others and this
information is captured by a probability distribution
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
3
Discrete Random Variables
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
4
Continuous Random Variable
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
5
Joint Probability
• Consider two random variables x and y
• If we observe multiple paired instances, then some
combinations of outcomes are more likely than
others
• This is captured in the joint probability distribution
• Written as Pr(x,y)
• Can read Pr(x,y) as “probability of x and
– where x and y are scalars
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
6
Joint Probability (2)
• We can definitely have more than two random
variables and we would write Pr(x,y,z)
• We may also write Pr(x) - the joint probability of all
of the elements of the multidimensional variable
x = [x1,x2….xn] and if y = [y1,y2….yn]
then we can write Pr(x,y) to represent the joint
distribution of all of the elements from
multidimensional variables x and y.
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
7
Joint Probability (3)
2D Hinton diagram
Joint distributions of one discrete and one continuous variable
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
8
Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
9
Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables
Scale and
Normalization
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
10
Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
11
Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables
Works in higher dimensions as well – leaves joint distribution between
whatever variables are left
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
12
Conditional Probability
• Conditional probability of x given that y=y1 is relative
propensity of variable x to take different outcomes given that
y is fixed to be equal to y1.
• Written as Pr(x | y=y1)
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
13
Conditional Probability
• Conditional probability can be extracted from joint probability
• Extract appropriate slice and normalize
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
14
Conditional Probability
• More usually written in compact form
• Can be re-arranged to give
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
15
Conditional Probability
• This idea can be extended to more than two
variables
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
16
Bayes’ Rule
From before:
Combining:
Re-arranging:
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
17
Bayes’ Rule Terminology
Likelihood – propensity for
observing a certain value of
x given a certain value of y
Posterior – what we
know about y after
seeing x
Prior – what we know
about y before seeing x
Evidence –a constant to
ensure that the left hand
side is a valid distribution
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
18
Independence
• If two variables x and y are independent then variable x tells
us nothing about variable y (and vice-versa)
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
19
Independence
• If two variables x and y are independent then variable x tells
us nothing about variable y (and vice-versa)
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
20
Independence
• When variables are independent, the joint factorizes into a
product of the marginals:
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
21
Probabilistic graphical models
22
What is a graphical model ?
A graphical model is a way of representing probabilistic
relationships between random variables.
Variables are represented by nodes:
Conditional (in)dependencies are
represented by (missing) edges:
Undirected edges simply give
correlations between variables
(Markov Random Field or
Undirected Graphical model):
Directed edges give causality
relationships (Bayesian Network or
Directed Graphical Model):
Graphical Models
• e.g. Bayesian networks, Bayes nets, Belief
nets, Markov networks, HMMs, Dynamic
Bayes nets, etc.
• Themes:
– representation
– reasoning
– learning
• Original PGM reference book by Nir Friedman
and Daphne Koller.
Motivation for PGMs
• Let X1,…,Xp be discrete random variables
• Let P be a joint distribution over X1,…,Xp
If the variables are binary, then we need O(2p)
parameters to describe P
Can we do better?
• Key idea: use properties of independence
Motivation for PGMs (2)
• Two variables X and Y are independent if
– P(X = x|Y = y) = P(X = x) for all values x, y
– That is, learning the values of Y does not change prediction
of X
• If X and Y are independent then
– P(X,Y) = P(X|Y)P(Y) = P(X)P(Y)
• In general, if X1,…,Xp are independent, then
– P(X1,…,Xp)= P(X1)...P(Xp)
– Requires O(n) parameters
Motivation for PGMs (3)
• Unfortunately, most of random variables of
interest are not independent of each other
• A more suitable notion is that of conditional
independence
• Two variables X and Y are conditionally
independent given Z if
– P(X = x|Y = y,Z=z) = P(X = x|Z=z) for all values x,y,z
– That is, learning the values of Y does not change prediction of X once we know
the value of Z
In summary….
Probability theory provides
the glue whereby the parts are combined,
ensuring that the system as a whole is consistent, and providing
ways to interface models to data.
The graph theoretic side of graphical models provides both an
intuitively appealing interface by which humans can model highlyinteracting sets of variables as well as a data structure that lends
itself naturally to the design of efficient general-purpose
algorithms.
Many of the classical multivariate probabilistic systems studied
in fields such as
statistics, systems engineering, information theory, pattern
recognition and statistical mechanics are special cases of the general
graphical model formalism -- examples include mixture models, factor
analysis, hidden Markov models, and Kalman filters.
Slide Credits
• Simon Prince – Computer vision, models and
learning
29
Next class
• More on Graphical models
• Readings for next lecture:
– Lecture notes to be posted
• Readings for today:
– Lecture notes to be posted
30
Questions
31