Lecture 5: Basic Probability Theory Donglei Du (ddu@unb.edu) Faculty of Business Administration, University of New Brunswick, NB Canada Fredericton E3B 9Y2 Donglei Du (UNB) ADM 2623: Business Statistics 1 / 55 Table of contents 1 Probability Theory What is probability? Random Experiment (RE) Set/events operations How to Interpret Probabilities? Rules of Counting Donglei Du (UNB) ADM 2623: Business Statistics 2 / 55 Layout 1 Probability Theory What is probability? Random Experiment (RE) Set/events operations How to Interpret Probabilities? Rules of Counting Donglei Du (UNB) ADM 2623: Business Statistics 3 / 55 What is probability? Probability is a measure of the likelihood that an event in the future will happen. It can only assume a value between 0 and 1; A value near zero means the event is not likely to happen; A value near one means it is likely to happen. Donglei Du (UNB) ADM 2623: Business Statistics 4 / 55 Random Experiment (RE) RE is a process that satisfies the following propositions: the process can be repeated as many trials as you want the outcome of any trial is uncertain well-defined set of possible outcomes each outcome has a probability associated with it Donglei Du (UNB) ADM 2623: Business Statistics 5 / 55 An example Toss a coin once: This process can be repeated as many times as you want (repetitive nature) Nobody knows whether Head or Tail will appear in any particular tossing (uncertain outcome) However, we know either Head or Tail must appear (Well-defined sample space) Now what is the chance you will get a Head if you toss a coin now? Donglei Du (UNB) ADM 2623: Business Statistics 6 / 55 Some Terminologies Outcome: is a particular result of a random experiment Sample Space: is the collection or set of all the possible outcomes of a random experiment Event: is the collection of one or more outcomes of an experiment Donglei Du (UNB) ADM 2623: Business Statistics 7 / 55 Example Toss one coin in very trial The sample space is S = {H, T}. An event is the occurrence of head: E = {h}. Flip two coins in very trial The sample space has two outcomes S = {(H,H), (H,T), (T,H), (T,T)}. An event is the occurrence of a head in the first coin: E = {(H, H), (H, T )} Roll one die in very trial The sample space contains 6 outcomes S = {1, . . . , 6}. An event is the occurrence of an even number: E = {2, 4, 6}. Roll two dice in very trial The sample space contains the 36 outcomes S = {(1, 1), . . . , (6, 6)}. An event is the occurrence of two dice with sum equal to 4: E = {(1, 3), (2, 2), (3, 1)} Donglei Du (UNB) ADM 2623: Business Statistics 8 / 55 Venn Diagram A Venn diagram or set diagram is a diagram that shows all possible logical relations between a finite collection of sets. The Venn diagram for the above die example. Donglei Du (UNB) 1 3 5 2 4 6 ADM 2623: Business Statistics 9 / 55 An example Toss two coins in very trial: This process can be repeated as many times as you want (repetitive nature) Every trial contains the outcome of two coins Origin First Flip p H T Donglei Du (UNB) Second Flip p H HH T H HT TH T TT ADM 2623: Business Statistics 10 / 55 Set operations Union Intersection: Complement: Donglei Du (UNB) ADM 2623: Business Statistics 11 / 55 Union Sample Space A B A B Union: The union event of two events A and B is denoted as A ∪ B, which consists of all outcomes in either A or B or in both. Namely event A ∪ B occurs if either A or B occurs. Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then E1 ∪ E2 = {1, 2, 3, 5} Donglei Du (UNB) ADM 2623: Business Statistics 12 / 55 Intersection Sample Space A A B B Intersection: The intersection event of two events A and B is denoted as A ∩ B, which consists of all outcomes in both A and B or in both. Namely event A ∩ B occurs if both A and B occur. Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then E1 ∩ E2 = {1, 3} Two events A and B are mutually exclusive if A ∩ B = ∅. Donglei Du (UNB) ADM 2623: Business Statistics 13 / 55 Complement Sample Space A A Complement: The complement event of an event A is denoted as Ā, which consists of all outcomes that are not in A. Namely event Ā occurs if A does not occur. Example: Roll a die: E1 = {1, 3, 5}. Then Ē1 = {2, 4, 6} Donglei Du (UNB) ADM 2623: Business Statistics 14 / 55 Probabilities defined on events For any random experiment with sample space S, the probability of any event is P (E) satisfying (i) 0 ≤ P (E) ≤ 1. (ii) P (S) = 1. (iii) (special addition rule) For any pair-wise mutually exclusive events E1 , E2 , . . . ! ∞ ∞ [ X P En = P (En ). n=1 Donglei Du (UNB) i=1 ADM 2623: Business Statistics 15 / 55 Examples Toss one fair coin in very trial: If we assume a head is equally likely to appear as a tail, then 1 2 Toss one biased coin in very trial: If we assume a head is twice as likely to appear as a tail, then P (H) = P (T ) = 1 2 P (H) = , P (T ) = . 3 3 Toss one fair die in very trial: If we assume all sides are equally likely to appear, then 1 P (i) = , i = 1, . . . , 6. 6 From (iii) in the previous slide, we must have 1 P ({1, 3, 5}) = P (1) + P (2) + P (3) = . 2 Donglei Du (UNB) ADM 2623: Business Statistics 16 / 55 Probability Distributions defined on sample space A (discrete) probability distribution on a given sample space is a table of all disjoint outcomes and their associated probabilities. Example: Toss two fair coins in very trial: The table below is the probability distribution for the sample space S = {HH, HT, T H, T T }: Outcome HH HT TH TT Donglei Du (UNB) Prob. 0.25 0.25 0.25 0.25 ADM 2623: Business Statistics 17 / 55 Rule of Probability Complement Rule Addition Rule Multiplication Rule Donglei Du (UNB) ADM 2623: Business Statistics 18 / 55 Rule of Complement For any event A: P (Ā) = 1 − P (A) Donglei Du (UNB) ADM 2623: Business Statistics 19 / 55 Rule of Addition For any two events A and B: P (A ∪ B) = P (A) + P (B) − P (A ∩ B). If events A and B are mutually exclusive: P (A ∪ B) = P (A) + P (B). The above can be extended to any number of mutually exclusive events A1 , . . . , An : P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ). Donglei Du (UNB) ADM 2623: Business Statistics 20 / 55 Rule of Addition: Example Toss two fair coins in very trial and assume that the four possible outcomes S = {HH, HT, T H, T T } are equally likely to happen. Let A = {HH, HT } be the event that the first coin falls head, and B = {HH, T H} be the event that the second coin falls head. Problem: Find the probability that either the first coin falls head or the second coin falls head? Solution: P (A ∪ B) = P (A) + P (B) − P (A ∩ B) = Donglei Du (UNB) ADM 2623: Business Statistics 3 1 1 1 + − = . 2 2 4 4 21 / 55 Rule of Multiplication Joint probability Conditional probability Donglei Du (UNB) ADM 2623: Business Statistics 22 / 55 Joint probability Joint probability: For two events A and B defined on the same sample space, the joint probability of events A and B is P (A ∩ B). Example: Roll one die in very trial. Let A = {1, 2, 3} and B = {1, 3, 5} be two events. Then the joint probability of A and B is P (A ∩ B) = P ({1, 3}) = Donglei Du (UNB) ADM 2623: Business Statistics 2 1 = . 6 3 23 / 55 Represent Joint probability as contingency table Joint probability distribution is given in a tabular form, called contingency table. The probability distribution for each variable is called the marginal distribution. Example: A survey of undergraduate students in the Faculty of Business Management at UNB revealed the following regarding the gender and majors of the students: Male Female Marginal Accoun. 150/750 175/750 325/750 IB 150/750 160/750 310/750 HR 50/750 65/750 115/750 Marginal 350/750 400/750 1 Example: What is the Probability of selecting a Female Accounting student? Solution: P (F ∩ A) = 175/750 = 23.33% Donglei Du (UNB) ADM 2623: Business Statistics 24 / 55 Conditional probability Conditional probability: For any event A and B, the probability of event A, given the occurrence of event B: P (A|B) = P (A ∩ B) . P (B) Example: What is the probability of selecting a Female, given that the person selected is an International Business major? Solution: P (F |I) = 160/310 = 51.6% Compare to the unconditional probability: P (F ) = 400/750 = 50% Donglei Du (UNB) ADM 2623: Business Statistics 25 / 55 Independent events: two events Two events A and B are independent if and only if P (A|B) = P (A) or equivalently P (A ∩ B) = P (A) × P (B). Example: the events F = {(F, A), (F, IB), (F, HR)} and IB = {(F, IB), (M, IB)} are not independent, since P (F |IB) = 160/310 = 51.6% 6= 50% = 400/750 = P (F ) Donglei Du (UNB) ADM 2623: Business Statistics 26 / 55 Independent events: more than two events n events A1 , . . . , An are independent if and only if for any r ≤ n: P (A1 ∩ . . . ∩ Ar ) = P (A1 ) × . . . × P (Ar ). Example: (Pairwise independent events may not be independent overall): Roll a four-faced die once. Let A1 = {1, 2}, A2 = {1, 3} and A3 = {1, 4} be three events. Then 1 4 1 P (A1 ∩ A3 ) = P (A1 ) × P (A3 ) = 4 1 P (A2 ∩ A3 ) = P (A2 ) × P (A3 ) = 4 1 1 P (A1 ∩ A2 ∩ A3 ) = 6= = P (A1 ) × P (A2 ) × P (A3 ) 4 8 P (A1 ∩ A2 ) = P (A1 ) × P (A2 ) = Donglei Du (UNB) ADM 2623: Business Statistics 27 / 55 Rule of Multiplication For any two events A and B: P (A ∩ B) = P (A|B)P (B). The above can be extended to any number of events A1 , . . . , An : P (A1 ∩ . . . ∩ An ) = P (A1 )P (A2 |A1 ) . . . P (An |A1 ∩ . . . ∩ An−1 ). If events A and B are independent: P (A ∩ B) = P (A)P (B). The above can be extended to any number of independent events A1 , . . . , A n : P (A1 ∩ . . . ∩ An ) = P (A1 ) . . . P (An ). Donglei Du (UNB) ADM 2623: Business Statistics 28 / 55 Rule of Multiplication: Example Example: Draw three cards with replacement i.e., draw one card, look at it, put it back, and repeat twice more. Problem: Find the probability of drawing 3 Queens in a row: Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a Queen. Then the three events are independent of each other. P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 )P (Q3 ) = Donglei Du (UNB) ADM 2623: Business Statistics 4 4 4 × × ≈ 0.00046. 52 52 52 29 / 55 Rule of Multiplication: Example Example: Draw three cards without replacement i.e., draw one card, look at it, keep it, and repeat twice more. Problem: Find the probability of drawing 3 Queens in a row: Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a Queen. Then the three events are dependent of each other. P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 |Q1 )P (Q3 |Q1 ∩ Q2 ) 4 3 2 = × × ≈ 0.00018. 52 51 50 Donglei Du (UNB) ADM 2623: Business Statistics 30 / 55 More examples on rules of probability Example: Draw one card from a deck of 52 cards Problem: What is the probability of getting a red card or a Queen? Solution: P (R ∪ Q) = P (R) + P (Q) − P (R ∩ Q) = Donglei Du (UNB) ADM 2623: Business Statistics 26 + 4 − 2 7 = ≈ 0.538 52 13 31 / 55 More examples on rules of probability Example: Two men throw their identical hats into the center of the room at a party. Then the hats are mixed up and each man randomly selects a hat. Problem: What is the probability that none of them selects his own hat. Solution: Let Ai (i = 1, 2) be the event that the ith man selects his own hat. Then the desired probability can be calculated via the complement rule. The complement event is that at least one man selects his own hat: P (A1 ∪ A2 ) = P (A1 ) + P (A2 ) − P (A1 ∩ A2 ) = 0.5 + 0.5 − P (A1 )P (A2 |A1 ) = 1 − 0.5(1) = 0.5 Therefore none of them selects his own hat is P A1 ∪ A2 = 1 − P (A1 ∪ A2 ) = 1 − 0.5 = 0.5 Donglei Du (UNB) ADM 2623: Business Statistics 32 / 55 Another method for solving the above problem Solution: Let Ai (i = 1, 2) be the event that the ith man selects his own hat. Therefore none of them selects his own hat is P Ā1 ∩ Ā2 = P (Ā1 )P (Ā2 |Ā1 ) = 0.5(1) = 0.5. Donglei Du (UNB) ADM 2623: Business Statistics 33 / 55 Bayes’ Theorem Sample Space A1 B A1 An B A2 B A2 …… An Given an event B, and a set of mutually exclusive and exhaustive events A1 , . . . , An , and we know the prior probabilities P (A1 ), . . . , P (An ). Then the posterior is: P (A1 |B) = Donglei Du (UNB) P (B|A1 )P (A1 ) P (B|A1 )P (A1 ) + . . . + P (B|An )P (An ) ADM 2623: Business Statistics 34 / 55 Bayes’ Theorem: Example Example: 25 percent of residents in an area leaves their garage doors open when they left their home. The chances of being raided are 5 percent and 1 percent for those who leave their doors open, and who do not leave their doors open, respectively. Problem: Suppose one resident was robbed, what is the probability that he or she originally left his or her door open? Solution: Let B = robbed residents, A1 = open doors residents, A2 = close door residents. We know that P (A1 ) = 0.25, P (A2 ) = 0.75, P (B|A1 ) = 0.05, P (B|A2 ) = 0.01. Donglei Du (UNB) ADM 2623: Business Statistics 35 / 55 Bayes’ Theorem: Tree diagram B A1 0.05 B | A1 0.25 A 1 0 75 0.75 A2 B | A1 0.95 0.01 B A1 B A2 B | A2 B | A2 0.99 Donglei Du (UNB) ADM 2623: Business Statistics B A2 36 / 55 Bayes’ Theorem: Example Therefore P (A1 |B) = = P (B|A1 )P (A1 ) P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) 0.0125 0.05(0.25) = = 0.625. 0.05(0.25) + 0.01(0.75) 0.02 Similarly we can calculate P (A2 |B) = Donglei Du (UNB) 0.01(0.75) 0.0075 = = 0.375. 0.05(0.25) + 0.01(0.75) 0.02 ADM 2623: Business Statistics 37 / 55 Bayes’ Theorem: Example Example: An insurance company classifies drivers as good, medium, or poor risks. Drivers who apply to for insurance fall into these three groups in the proportions: 30%, 50%, and 20%, respectively. The probabilities of a good-risk, medium-risk, and poor-risk drivers will have an accident are 0.01, 0.03, and 0.10, respectively. Problem: Suppose the company sells Mr. Brophy an insurance policy and he has an accident. What is the probability that Mr. Brophy was a good-risk driver? Solution: Let B = accident drivers, A1 = good drivers, A2 = medium drivers, and A3 = poor drivers. We know that P (A1 ) = 0.3, P (A2 ) = 0.5, P (A3 ) = 0.2; P (B|A1) = 0.01, P (B|A2 ) = 0.03, P (B|A3 ) = 0.10. Donglei Du (UNB) ADM 2623: Business Statistics 38 / 55 Bayes’ Theorem: Tree diagram B A1 0.01 B | A1 B | A1 0.99 0.3 A1 A2 0.5 02 0.2 B A1 B A2 0 03 0.03 B | A2 B | A2 0.97 A3 B A2 B A3 0.10 B | A3 B | A3 0.90 Donglei Du (UNB) 2013/9/17 B A3 Donglei Du: Lecture 1 ADM 2623: Business Statistics 39 / 55 Bayes’ Theorem: Example Therefore P (A1 |B) = = P (B|A1 )P (A1 ) P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + P (B|A3 )P (A3 ) 0.01(0.3) 0.03 = ≈ 0.079 0.01(0.3) + 0.03(0.5) + 01.0(0.2) 0.038 Similarly we can calculate P (A2 |B) = P (A3 |B) = 0.03(0.5) 0.015 = ≈ 0.395 0.038 0.038 0.10(0.2) 0.02 = ≈ 0.526 0.038 0.038 Now we know that Mr. Brophy is mostly likely a poor-risk driver. Donglei Du (UNB) ADM 2623: Business Statistics 40 / 55 How to Interpret Probabilities There are two ways to Interpret probability: Objectivists Subjectivists Donglei Du (UNB) ADM 2623: Business Statistics 41 / 55 How to Interpret Probabilities: Objectivists Objectivists: assign numbers to describe some objective or physical state of affairs. The most popular version of objective probability is frequentist probability, which claims that the probability of a random event denotes the relative frequency of occurrence of an experiment’s outcome, when repeating the experiment. This interpretation considers probability to be the relative frequency ”in the long run” of outcomes. Donglei Du (UNB) ADM 2623: Business Statistics 42 / 55 How to Interpret Probabilities: Subjectivists Subjectivists: assign numbers per subjective probability, i.e., as a degree of belief. The most popular version of subjective probability is Bayesian probability, which includes expert knowledge as well as experimental data to produce probabilities. The expert knowledge is represented by some (subjective) prior probability distribution. The data is incorporated in a likelihood function. The product of the prior and the likelihood, normalized, results in a posterior probability distribution that incorporates all the information known to date. Donglei Du (UNB) ADM 2623: Business Statistics 43 / 55 Rules of Counting Multiplication Rule Permutation Rule Combination Rule Donglei Du (UNB) ADM 2623: Business Statistics 44 / 55 Multiplication Rule If one thing can be done in M ways, and if after this is done, something else can be done in N ways, then both things can be done in a total of M*N different ways in that stated order! Donglei Du (UNB) ADM 2623: Business Statistics 45 / 55 Permutation Rule A counting technique that is used when order is important A permutation of r objects chosen from n objects is a group of any r objects, when order is important Pnr = Donglei Du (UNB) n! (n − r)! ADM 2623: Business Statistics 46 / 55 Example Example: You are assigned the task of choosing 2 of your 6 classmates to serve on a task force. One will act as the Chair of the task force, and the other will be the Secretary. Problem: In how many ways can you make this assignment? Solution: P62 = Donglei Du (UNB) 6! = 30 (6 − 2)! ADM 2623: Business Statistics 47 / 55 Combination Rule A counting technique that is used when order is NOT important A combination of r objects chosen from n objects is a group of any r objects, when order is not important Cnr = Donglei Du (UNB) n! r!(n − r)! ADM 2623: Business Statistics 48 / 55 Example Example: You are assigned the task of choosing 2 of your 6 classmates to serve on a task force. Responsibilities are evenly shared. Problem: In how many ways can you make this assignment? Solution: C62 = Donglei Du (UNB) 6! = 15 2!(6 − 2)! ADM 2623: Business Statistics 49 / 55 The Twin Paradox (Probabilistic Pigeonhole Principle) Example: The Statistics Professor wants to play a little game with his students. “I bet that there are two of you who have the same birthday! What do you think”. Several students reply immediately: “There are 366 possible birthdays, so you could only conclude this if there were at least 367 of us in the class! But there are only 50 of us, and so you would lose the bet for sure!” Problem: Let us calculate the probability of winning by the professor. Donglei Du (UNB) ADM 2623: Business Statistics 50 / 55 The Twin Paradox (Probabilistic Pigeonhole Principle) Solution: Let A be the event of winning by the professor. Then its complement is the event of wining by students, which is equivalent to the event that all n = 50 students having different birthdays. P (Ā) = = number of different birthdays for 50 students number of birthdays for 50 students 50 C366 366! 366 × . . . × 316 = = ≈ 0.03. 50 50 366 /50! (366 − 50)!366 36650 Therefore the teacher has a much higher chance of wining: P (A) = 1 − 0.03 = 97%.! Donglei Du (UNB) ADM 2623: Business Statistics 51 / 55 The Twin Paradox (Probabilistic Pigeonhole Principle) Here are the probability of wining by the professor for different n, the number of students. n Prob. 10 12 20 41 30 71 40 81 50 97 60 99 Donglei Du (UNB) ADM 2623: Business Statistics 52 / 55 Case: conditional probability and market direction prediction Probability of consecutive up or down price moves G*#2)354"?+67"@#/3)"J;3*I6" Data: S&P 500 from Jan 1950 to May 2015 K"# K"#$"%" #$"! !B" K"#$"!B" @1A"BCD"BE!F" Figure: Probability of consecutive up and down price movements Donglei Du (UNB) ADM 2623: Business Statistics 53 / 55 Case: conditional probability and market direction prediction Probability of 4th up day (A), given 3 consecutive up days (B) is P (A|B) = Donglei Du (UNB) P (A ∩ B) P (A) 0.086 = = = 53.1% P (B) P (B) 0.162 ADM 2623: Business Statistics 54 / 55 Case: conditional probability and market direction prediction G*#212+)+034"#$"@1*:36"@#;34" G*#2)354"?+67"@#/3)"J;3*I6" The probabilities are all roughly 53% – which is the same as the probability of having a ‘normal’ up day. Q"# Q"#$"%" #$"! !B" Q"#$"!B" @1A"BCD"BE!F" Figure: Probability of consecutive up and down price movements Donglei Du (UNB) ADM 2623: Business Statistics 55 / 55