Probability Refresher
Events
• Events as possible outcomes of an experiment
• Events define the sample space (discrete or continuous)
– Single throw of a dice: {1, 2, 3, 4, 5, 6}
• Combinations (union & intersection) of events also define
events
E = E1 OR E2 = E1  E2
E = E1 AND E2 = E1  E2
Mutually exclusive events: E1  E2 = 
Partition (of sample space ): Set of mutually exclusive events
that cover the entire sample space
– Complementary events: E & F are complements of each other if
E  F =  and E  F = 
–
–
–
–
2
Events & Probabilities – (1)
• P{E}: Odds that event E occurs (P{} = 1)
• Union Law: P{EF} = P{E}+P{F}-P{EF}
 P{EF} ≤ P{E}+P{F} with equality only if E and F are mutually
exclusive (complementary events are mutually exclusive)
• Conditional probability
– P{E|F} = P{EF}/P{F}  P{EF} = P{E|F}P{F}
• Independence
– E & F are independent if P{EF} = P{E}P{F}
• Conditional independence
– E & F are conditionally independent given G if
P{EF|G} = P{E|G}P{F|G}
3
Events & Probabilities – (2)
• Law of total probability: For any partition
F1,…,FN, of the
sample
space
(
F
=
)
i
i
N
N
PE   PE  Fi    PE | Fi PFi 
i 1
• Bayes Law
i 1
PE | F PF 
PF | E 
PE
– Prove using definition of conditional probability
• Combining Bayes Law and Law of total probability
PF | E 
PE | F PF 
N
 PE | F PF 
i 1
i
i
4
Example – Anti-virus s/w Test
• We know that our s/w is 95% accurate, i.e.,
– P{positive | virus} = 0.95 (true positive) and P{negative | virus} = 0.05
– P{negative | no virus} = 0.95 (true negative) and P{positive | no virus} = 0.05
• We also know that on average 1 out every 1,000 computers is infected with a
virus, i.e., P{virus} = 0.001
• What are the odds that a computer that tests positive is infected with a virus?
– p = P{has virus| positive}
– Bayes Law: p = [P{positive | virus}P{virus}]/P{positive}
– Bayes Law + Total Probability Law: Replace P{positive} with
P{positive} = P{positive | virus}P{virus}+P{positive | no virus} P{no virus}
P{positive} = 0.950.001+0.050.999 = 0.0509
– This gives p = 0.950.001/0.509 = 0.0187, i.e., less than 2%
5
Random Variables
• Basically a mapping of events to numbers
– Typical notation: X (random variable), x (value)
• Cumulative distribution function (c.d.f.): FX(a) = P{X ≤ a}
– Note that by definition FX() = 1
• Complementary distribution: FX(a) = 1- FX(a) = P{X > a}
• Discrete and continuous r.v.’s
– Probability mass function (p.m.f) vs. probability density function (p.d.f.)
– Expectation & higher moments
• Discrete r.v.:
EX    xpX x,
• Continuous r.v.:
EX    xf X x dx,


 
E X i   x i p X x 
 

E X   xi f X x dx
i

• Variance: Var(X) = E[(X – E[X])2] = E[X2] – E2[X] (by linearity of expectation –
more on this soon)
6
Joint Probability
• Discrete r.v.: Joint probability mass function PX,Y(x,y)
– PX,Y(x,y) = P{X=x AND Y=y)
– PX(x) = ΣyPX,Y(x,y) and PY(y) = ΣxPX,Y(x,y)
• Continuous r.v.: Joint density function fX,Y(x,y)
b1 b2

a1 a2
f X ,Y x, y dxdy  Pa1  x  b1 AND a2  y  b2 




f X x    f X ,Y x, y dy and fY  y    f X ,Y x, y dx
• If X and Y are independent r.v.’s (X  Y)
– Discrete: PX,Y(x,y) = PX(x)PY(y)
– Continuous: fX,Y(x,y) = fX(x)fY(y)
– E[XY] = E[X]E[Y]
7
Conditional Probabilities & Expectations
• Discrete r.v.:
– Conditional p.m.f. of X given A
P X  x   A
p X | A x   PX  x | A 
PA
– Conditional expectation of X given A
P X  x   A


EX | A   xpX | A x   x 
PA
x
x
• Continuous r.v.:
 f X x 
if

– Conditional p.d.f. p X | A x    PX  A

0
x A
otherwise
– Conditional expectation of X given A
1
EX | A   xf X | A x dx   xf X | A x dx 
xf X x dx


A
A
8
PX  A

More on Expectation
• Expected value from conditional expectation
– Discrete r.v.: E[X] = ΣyE[X|Y=y]P{Y=y}
• More generally: E[g(X)] = ΣyE[g(X)|Y=y]P{Y=y}
– Continuous r.v.: E[X] = yE[X|Y=y]fY(y)dy
• More generally: E[g(X)] = yE[g(X)|Y=y]P{Y=y}
• Linearity of expectation: E[X+Y] = E[X] + E[Y]
• Linearity of variance for independent r.v.’s
– If X  Y then Var(X+Y) = Var(X) + Var(Y)
9
Random Sum of Random Variables
• Let X1, X2, X3,… be i.i.d. random variables and
N be a non-negative, integer-valued random
variable, independent of the Xi’s
N
• Define S  i 1 X i
• Find expressions for E[S] and Var(S)
– Condition on N and use linearity of expectation
– For variance, use the fact that
Var(S|N = n) = nVar(X)
10