Probability and Stochastic Processes
Chapter 1
Experiments, Models, and Probabilities
Hyundong Shin
Communications and Coding Theory Laboratory
(CCTLAB)
1
Section 1.1
Applying Set Theory to Probability
(Mathematical Basis of Probability)
2
Basic Definitions
i Set: a collection of things
i Elements: the things that make up the set
i Representations:
A = {KHU, SNU, KAIST}
{
}
C = x 2 x = 1,2, 3, 4,5
i Inclusion (∈): a relationship between an element and a set
x ∈ A means "x is an element of a set A."
i Subset (⊂): a relationship between two sets
A ⊂ B : the set A is the subset of the set B.
i Set Equality (=)
A = B : iff A ⊂ B and A ⊃ B
3
Two Elementary Sets
i Universal set (S):
includes all of the elements of all of the sets in the study.
i Null set (φ ): has no element, i.e.,
{ }.
ex) Rolling a dice:
A = {even } = {2, 4,6}
B = {odd} = {1, 3,5}
S = {1,2, 3, 4,5,6}
C = {Roll x > 6} = {
}=φ
φ ⊂ A,B ⊂ S
4
Set Algebra (Operations on the Sets):
i Union: A ∪ B
i Intersection: A ∩ B
i Complement: Ac
i Difference: A− B = A ∩ Bc = A− ( A ∩ B)
5
Exclusiveness and Exhaustiveness
i Mutually Exclusive:
A collection of sets A1 , A2 ,…, An are mutually exclusive iff
Ai ∩ A j = φ, i ≠ j ∈ {1,2,…,n}.
c.f.) If there are only two sets in the collection, the sets are
called "disjoint".
i Collectively Exhaustive:
A collection of sets A1 , A2 ,…, An are collectively exhaustive iff
n
 Ai = A1 ∪ A2 ∪∪ An = S.
i=1
6
1.1 Comment: Mutually Exclusive Sets
A collection of sets A1, . . . , An is mutually exclusive
if and only if
A
Ai \ Aj = ?,
B
i 6= j.
(1.1)
The word disjoint is sometimes used as a synonym
for mutually exclusive.
1.1 Comment: Collectively Exhaustive Sets
A1
A2
A collection of sets A1, . . . , An issets,collectively exhaustive collectively exhaustive if and only if
A1 [ A2 [ · · · [ An = S.
A5
A4
A3
(1.2)
7
1.1 Comment: Partitions
A1
A2
A3
A4
A collection of sets A1, . . . , An is a partition if it is
both mutually exclusive and collectively exhaustive.
8
9
Ex) Coin-tossing: Toss a coin two times.
10
11
12
Quiz 1.1
Monitor three consecutive packets going through a Internet router. Based
on the packet header, each packet can be classified as either video (v) if it
was sent from a Youtube server or as ordinary data (d). Your observation
is a sequence of three letters (each letter is either v or d). For example,
two video packets followed by one data packet corresponds to vvd. Write
the elements of the following sets:
A1 = {second packet is video},
A2 = {all packets are the same},
B1 = {second packet is data},
B2 = {video and data alternate},
A3 = {one or more video packets}, B3 = {two or more data packets}.
For each pair of events A1 and B1, A2 and B2, and so on, identify whether
the pair of events is either mutually exclusive or collectively exhaustive or
both.
13
Quiz 1.1 Solution
A1
B1
A2
B2
A3
B3
= {vvv, vvd, dvv, dvd}
= {vdv, vdd, ddv, ddd}
= {vvv, ddd}
= {vdv, dvd}
= {vvv, vvd, vdv, dvv, vdd, dvd, ddv}
= {ddd, ddv, dvd, vdd}
Recall that Ai and Bi are collectively exhaustive if Ai [ Bi = S. Also, Ai and Bi are
mutually exclusive if Ai \ Bi = . Since we have written down each pair Ai and Bi above,
we can simply check for these properties.
The pair A1 and B1 are mutually exclusive and collectively exhaustive. The pair A2 and
B2 are mutually exclusive but not collectively exhaustive. The pair A3 and B3 are not
mutually exclusive since dvd belongs to A3 and B3 . However, A3 and B3 are collectively
exhaustive.
14
Section 1.2
Probability Axioms
15
16
17
18
19
20
21
22
23
24
25
Section 1.3
Conditional Probability
26
A Priori Probability P [A ]
The probability measure P ⎡⎢ A⎤⎥ of the event A reflects the knowledge
⎣ ⎦
of the occurrence of A prior to performing an experiment.
P ⎡⎢ A⎤⎥ : a priori probability of A
⎣ ⎦
i P ⎡⎢ A⎤⎥ ≈ 1 or P ⎡⎢ A⎤⎥ ≈ 0: little amount of uncertainty
⎣ ⎦
⎣ ⎦
i P ⎡⎢ A⎤⎥ ≈ 1 2 (little information): large amount of uncertainty
⎣ ⎦
27
Conditional Probability P ⎡⎢ A B⎤⎥
⎣
⎦
i Discribes the knowledge of A when we know that B has
occurred but we still do not know the precise outcome.
i Read as "the probability of A given B".
28
29
A respectable probability measure relative to a
sample space that consists of all the outcomes in B.
30
32
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Inference: “Reasoning about causes when we observe effects”
53
40
54
55
Example. Binary Symmetric Channel (BSC)
Input
0
0
1
1
Output
What is the probability that the input was 0 when the output is 1 ?
56
42
57
(If events A and B have nonzero probabilities):
equivalent to
P éë A Bùû = P [ A ]
P éë B Aùû = P [ B]
¹ Disjoint P [ AB] = 0
5844
59
60
61
62
63
64
65
66
67