Probability_1

1 P ROBABILITY1.1Probability Theory and Statistical InferenceWhenever we have available just a sample of data from a population - and this will be the typical situation – then we need to be able to infer the (unknown) values of the population characteristics from their estimated sample counterparts. This is known as statistical inference and requires the knowledge of basic concepts in probability and an appreciation of probability distributions. 6.2 Basic Concepts in ProbabilityMany, if not all, of you will have had some exposure to the basic concepts of probability, so our treatment here is accordingly brief. We will begin with a few definitions, which will enable us to establish a vocabulary.∙ An experiment is an action, such as tossing a coin, which has a number ofpossible outcomes or events , such as heads (H ) or tails (T ).∙ A trial is a single performance of the experiment, with a single outcome.∙ The sample space consists of all possible outcomes of the experiment. Theoutcomes for a single toss of a coin are []T H ,, for example. The outcomes in the sample space are mutually exclusive , which means that the occurrence of one rules out all the others: you cannot have both H and T in a single toss of the coin. They are also exhaustive , since they define all possibilities: you can only have either H or T in a single toss.∙ With each outcome in the sample space we can associate a probability , whichis the chance of that outcome occurring. Thus, if the coin is fair, the probability of H is 0.5.With these definitions, we have the following probability axioms∙ The probability of an outcome A , ()A P , must lie between 0 and 1, i.e. ()10≤≤A P ,(1.1)∙ The sum of the probabilities associated with all the outcomes in the samplespace is 1. This follows from the fact that one, and only one, of the outcomesmust occur, since they are mutually exclusive and exhaustive, i.e., if i P is the probability of outcome i occurring and there are n possible outcomes, then∑==n i i P 11(1.2)∙ The complement of an outcome is defined as everything in the sample spaceapart from that outcome: the complement of H is T , for example. We will write the complement of A as A (to be read as ‘not -A ’ and is not to be confused with the mean of A !). Thus, since A and A are mutually exclusive and exhaustive()()A P A P -=1(1.3)Most practical problems require the calculation of the probability of a set of outcomes, rather than just a single one, or the probability of a series of outcomes in separate trials, e.g., what is the probability of throwing three H in five tosses? We refer to such sets of outcomes as compound events .Although it is sometimes possible to calculate the probability of a compound event by examining the sample space, typically the sample space is too complex, even impossible, to write down. To calculate compound probabilities in such situations, we make use of a few simple rules.The addition ruleThis rule is associated with ‘or’ (sometimes denoted as ⋃, known as the union of two events): ()()()()B A P B P A P B A P and or -+=(1.4)Here we have introduced a further event, ‘A and B ’, which encapsulates the idea of the intersection ()⋂ of two events.Consider the experiment of rolling a six-sided die. The sample space is thus []6,5,4,3,2,1, 6=n and 61=i P for all 6,,2,1 =i . The probability of rolling a five or a six is ()()()()6and 5656or 5P P P P -+=Now, ()06and 5=P , since a five and a six cannot simultaneously occur (the events are mutually exclusive). Thus()31061616and 5=-+=PIt is not always the case that the two events are mutually exclusive, so that their intersection does not always have zero probability. Consider now the experiment of drawing a single playing card from a standard 52=n card pack. The sample space is the entire 52 card deck and 521=i P for all i . Let us now compute the probability of obtaining either a Queen or a Spade from this single draw. Using obvious notation()()()()S Q P S P Q P S Q P and or -+=(1.5)Now, ()4=Q P and ()5213=S P (obtained in each case by counting up the number of outcomes in the sample space that are defined by each event and dividing the result by the total number of outcomes in the sample space). However, by doing this, the outcome representing the Queen of Spades (Q and S ) gets included in both calculations and is thus ‘double counted’. It must therefore be subtracted from()()S P Q P +, thus leading to (1.5). The events Q and S are not mutually exclusive, ()521and =S Q P , and()5216521134or =-+=S Q PThe multiplication ruleThe multiplication rule is associated with ‘and’. Consider the event of rolling a die twice and asking what the probability of obtaining fives on both rolls is. Denote this probability as ()215and 5P , where we now use the notation j i to signify obtaining outcome i on trial j : this probability will be given by ()()()3616161555and 52121=⨯=⨯=P P PThe logic of this calculation is straightforward. The probability of obtaining a five on a single roll of the die is 61 and, because the outcome of one roll of the die cannot affect the outcome of a second roll, the probability of this second roll also producing a five must again be 61. The probability of both rolls producing fives must then be the product of these two individual probabilities. Technically, we can multiply the individual probabilities together because the events are independent.What happens when events are not independent? Consider now drawing two playing cards and asking for the probability of obtaining a queen of spades (QS 1) on the first card and an S on the second (S 2). If the first card is replaced before the second card is drawn then the events 1QS and 2S are still independent and the probability will be ()()0048.0521521352421==⨯=⨯S P Q PHowever, if the second card is drawn without the first being replaced (i.e., there is a ‘normal’ deal) t hen we have to take into account the fact that, if 1QS has occurred, then the sample space for the second card has been reduced to 51 cards, 12 of which will be S : thus the probability of 2S occurring is 5112, not 5213. The probability of 2S is therefore dependent on 1QS having occurred and thus the two events are not independent. Technically, this is a conditional probability, denoted ()12QS S P and in general ()A B P , and the general multiplication rule of probabilities is, for our example, ()()()00452.05112521 and 12121=⨯=⨯=Q S P QS P S QS Pand, in general, ()()()A B P A P B A P ⨯= andOnly if the events are independent is ()()B P A B P =. More formally, if A and B are independent, then, because the probability of B occurring is unaffected by whether A occurs or not ()()()B P A B P A B P == and ()()()A P B A P B A P ==Combining the rulesConsider the following example. It is equally likely that a mother has a boy or a girl on the first (single) birth, but a second birth is more likely to be a boy if there was a boy on the first birth and similarly for a girl. Thus ()()5.011==G P B Pbut, for example,()()6.01212==G G P B B Pwhich implies that ()()4.01212==B G P G B PThe probability of a mother having one child of each sex is thus()()()()()()()()()()4.04.05.04.05.0and or and boy 1 and girl 11211212121=⨯+⨯=+==B G P B P G B P G P G B P B G P PCounting rules: combinations and permutationsThe preceding problem can be illustrated using a tree diagram , which is a way of enumerating all possible outcomes in a sample space with the associated probabilities. Tree diagrams have their uses for simple problems (consult one of the textbooks for more discussion on them), but for more complicated problems they quickly become very complex and difficult to work with.Suppose we have a family of five children of whom three are girls. To compute the probability of this event occurring, our first task is to be able to calculate the number of ways of having three girls and two boys, irrespective of whether successive births are independent or not. An ‘obvious’ way of doing this is to write down all the possible orderings, of which there are ten: GGGBB GGBGB GGBBG GBGGB GBGBGGBBGGBGGGBBGGBGBGBGGBBGGGIn more complex problems, this soon becomes difficult or impossible, and we then have to resort to using the combinatorial formula . Suppose the three girls are ‘named’ a , b and c . Girl a could have been born first, second, third, fourth or fifth, i.e., any one of five ‘places’ in the ordering:a ?????a ?????a ?????a ?????aSuppose that a is born first; then b can be born either second, third, fourth or fifth, i.e., any one of four places in the ordering: ab ???a ?b ???a ??b ?a ???bBut a could have been born second, etc., so that the total number of places for a and b to ‘choose’ is 2045=⨯. Three places remain for c to choose, so, by extending the argument, the three girls can choose a total of 60345=⨯⨯ places between them. This is the number of permutations of three named girls in five births, and can be given the notation !2!5121234534535=⨯⨯⨯⨯⨯=⨯⨯=Pwhich uses the factorial (!) notation. 6035=P is six times as large as the number of possible orderings written down above. The reason for this is that the listing does not distinguish between the girls, denoting each of them as G rather than a , b or c . The permutation formula thus overestimates the number of combinations by a factor representing the number of ways of ordering the three girls, which is !3123=⨯⨯. Thus the formula for the combination of three girls in five births is ()()101212312345!2!3!5!33535=⨯⨯⨯⨯⨯⨯⨯⨯===P CIn general, if there are n children and r of them are girls, the number of combinations is()!!!!r n r n r P C r nr n-==Bayes TheoremRecall that ()()()A B P A P B A P ⨯= andor, alternatively,()()()A PB A P A B P and =This can be expanded to be written as()()()()()()()()()()B P B A P B P B A P B P B A P A P B P B A P A B P ⨯+⨯⨯⨯=which is known as Bayes theorem . With this, we can now answer the following question: given that the second birth was a girl, what is the probability that the first birth was a boy, i.e., what is ()21G B P ? Noting that the event 1B is 1G , Bayes theorem gives us()()()()()()()()()4.05.06.05.04.05.04.011211211221=⨯+⨯⨯=⨯+⨯⨯=G P G G P B P B G P B P B G P G B PThus, knowing that the second child was a girl allows us to update our probability that the first child was a boy from the unconditional value of 0.5 to the new value of 0.4.Definitions of probabilityIn our development of probability, we have begged one important question: where do the actual probabilities come from? This, in fact, is a question that has vexed logicians and philosophers for several centuries and, consequently, there are (at least) three definitions of probability in common use today.The first is the classical or a priori definition and is the one that has been implicitly used in the illustrative examples. Basically, it assumes that each outcome in the sample space is equally likely , so that the probability that an event occurs is calculated by dividing the number of outcomes that indicate the event by the total number of possible outcomes in the sample space. Thus, if the experiment is rolling a fair six-sided die, and the event is throwing an even number, then the number of outcomes indicating the event is three (the outcomes []6,4,2), the total number of outcomes in the sample space is six []6,5,4,3,2,1, so that the required probability is 2163=.This will only work if, in our example, we can legitimately assume that the die is fair. What would be the probability of obtaining an even number if the die was biased, but in an unknown way? For this, we need to use the relative frequency definition. If we conduct an infinite number of trials of an experiment (that is, ∞→n ), and the number of times an event occurs in these n trials is k , then the probability of the event is defined as the limiting value of the ratio n k .Suppose that even numbers were twice as likely to occur on a throw of the die as odd numbers (the probability of any even number is 92, the probability of any odd number is 91, so that the probability of an even number being thrown is 32). However, we do not know this, and decide to estimate this probability by throwing the die a large number (10,000) of times and recording as we go the relative frequency of even numbers being thrown. A plot of this ‘cumulative’ relative frequency is shown overleaf. We observe that it eventually ‘settles down’ on 32, but it takes quite a large number of throws (around 6,000) before we get a really good estimate of the probability..62.64.66.68.70.72.74.76.7810002000300040005000600070008000900010000P r o p o r t i o n o f e v e n n u m b e r sN umber of rollsWhat happens if the experiment cannot be conducted more than once, if at all, e.g., if it is a ‘thought experiment’? We can then use subjective probability , which assigns adegree of belief on an event actually occurring. If we believe that it is certain to occur, then we assign a probability of one to the event, if we believe that it is impossible to occur then we assign a probability of zero, and if we think that it has a ‘good chance’ of occurring then we presumably assign a probability that is greater than 0.5 but less than 1, etc.No matter what definition seems appropriate to the experiment and outcome at hand, fortunately all definitions follow the same probability axioms, and hence rules, as those outlined above.。