Statistical sciences. Schools of thought. Resources for the course. Bayesian Methods - Introduction

Transcription

1 Bayesian Methods - Introduction Dr. David Lucy d.lucy@lancaster.ac.uk Lancaster University Bayesian methods p.1/38 Resources for the course All resources for this course can be found at: lucy/courses/cfas416/cfas416.html All files should be downloaded to a directory in your H: drive for those who are at Lancaster University. For those who are not make sure that you place them on the Universities server. Sometimes pen drives do not work with these machines - we can sort out files you wish to keep later. Bayesian methods p.2/38 Statistical sciences The debate on a statistical framework has been going on for a long time. Centers around the idea of probability. Is probability: 1. A measure of population frequencies? 2. A measure of belief? Bayesian methods p.3/38 Schools of thought Two schools of thought: 1. Frequentist - thinks of probability in terms of population frequencies. That is a probability applies to long runs of similar events. Consequently it is difficult for the frequentist to think of single events. 2. Bayesian - for the Bayesian probability is a degree of belief in an outcome or an event. Easy for the Bayesian to think of a probability as a property attached to singular events. Bayesian methods p.4/38

2 Schools of thought Frequentists - tend to calculate: 1. Confidence intervals. 2. p-values. Bayesians - tend to calculate: 1. Credible intervals. 2. Probabilities. Many non-statisticians interpret confidence intervals in the same way as a credible intervals. As an interval in which a given probability occurs. Bayesian methods p.5/38 Schools of thought Ultimately the differences are: Frequentists - are unwilling to ascribe probabilities to events, they can only ascribe probabilities to observations. Means you cannot comment on events. Bayesians - do make probabilistic statements about both events and observations. The difference is largely metaphysical, and thus not open to solution through any mathematical enquiry. Bayesian methods p.6/38 Frequentist test For example: a simple significance test : 15 people who reported headaches were given a placebo. Of those 6 reported their headache vanishing within half an hour. 15 people who reported a headache were given aspirin. Of those 10 reported their headache vanishing within half an hour. Does aspirin have a beneficial effect upon headaches? fictitious data for demonstration purposes. Bayesian methods p.7/38 Frequentist test How to answer this problem? Is the observation of 10 from 15 headaches vanishing unusual in some way if we expect 6 to vanish under normal circumstances. need to calculate the sampling distribution for the null hypothesis. For the null hypothesis we suggest that there is no underlying difference in the rate of headache vanishing. Bayesian methods p.8/38

3 Frequentist test Sampling distribution under the null hypothesis: Is binomial - that is concerns the number of positive observations from so many trials. In this case positive means headache gone within half an hour. We calculate the binomial distribution for 0 to 15 positives, from 15 trials with a probability of a positive of 6 in 15, or 0.4. This gives the distribution for all the observations if the null hypothesis is true. Bayesian methods p.9/38 Binomial distribution The binomial distribution: Pr(X = k) = n! k!(n k)! pk (1 p) (n k) For a sample of n trials, k positives, each success happening independently with probability p.! indicates the factorial expansion, so, for example: 5! = = 120 Note: 0! = 1, and 1! = 1. Bayesian methods p.10/38 Binomial test So if I want to know the probability that I will observe 0 positives from a sample of 15 I substitute 0 for k, 15 for n and 0.4 for p into the binomial expression: Pr(X = x) = n! k!(n k)! pk (1 p) (n k) Pr(X = 0) = 15! 0!(15 0)! 0.40 (1 0.4) (15 0) = 15! 0!15! 0.40 (0.6) (15) = (0.6)(15) Bayesian methods p.11/38 Binomial test For the probability that I will observe 1 positive from a sample of 15, I substitute 1 for k and the rest as before, so: Pr(X = 1) = 15! 1!(15 1)! 0.41 (1 0.4) (15 1) = 15! 14! 0.41 (0.6) 14 = (0.6) 14 = Or about half a percent. Bayesian methods p.12/38

4 Binomial test As there are 15 individuals in the sample we must do this 16 times (don t forget that 0 is a possible value) to cover all values for the numbers of people whose headaches vanish within half an hour. Fortunately we have computers to do all that for us. Best displayed as a graph. Bayesian methods p.13/38 B(0 : 15,15,0.4) probability density number of individuals Bayesian methods p.14/38 p-value The sum of the probabilities for 10, 11, 12, 13, 14 and 15 people whose headaches vanish within half an hour is This means that the probability of observing 10 or more people whose headaches vanish is 0.033, or 3%. This is conditioned on, that is we presume that, the null hypothesis is true. Bayesian methods p.15/38 p-value This probability is called a p-value. A p-value is the probability of observing a specific value of our test statistic, in this case the number of people whose headaches subside within half an hour, given the truth of the null hypothesis. This probability is about 3%. 3% is very small were the null hypothesis true. These observations provide little support for the null hypothesis. To put it another way we might reject the null hypothesis. Bayesian methods p.16/38

5 Hypothesis tests All hypothesis tests have the same underlying structure: 1. Define a distribution for the observations under the null hypothesis. 2. Make some observations. 3. Calculate the probability of the observations, or any observations more extreme, given the truth of the null hypothesis. 4. Make some decision about the truth, or otherwise, of the null hypothesis. Bayesian methods p.17/38 Problems There are a number of problems with these sorts of hypothesis tests: 1. Calculate the null distribution, a hypothesis you are not fundamentally interested in. We re actually more interested in the alternative hypothesis. 2. Arbitrary acceptance of null hypothesis. Conventional limits 5% and 1%. 3. Limited support for null hypothesis does not logically entail support for the alternative hypothesis. 4. Reliance on large samples and long-run statistics. Bayesian methods p.18/38 Bayesian frameworks Bayesian approaches allow more easily: Probabilities as degrees of belief to be calculated for single, non-repeatable, events. Ideal for work in a variety of sciences as events, even if part of a class of events, are classes we wish to know about. Bayesian methods p.19/38 Laws of probability let us recap at a fairly fundamental level: 1. n i=1 Pr(A i) = 1 - if Ai s are mutually exclusive and exhaustive. 2. Pr(A or B) = Pr(A)+Pr(B) - we saw this in the example where we had events of 10, 11, 12 etc. people whose headaches subsided in half an hour. 3. Pr(A and B) = Pr(A) Pr(B) - iff A and B are mutually independent. Bayesian methods p.20/38

6 Third law of probability Let us suppose I wanted to know the probability of the joint event of a head followed by a tail from two throws of a fair coin: 1 st throw 2 nd throw joint outcome H H HH H T HT T H TH T T TT Probability of getting a head with first throw, 0.5, probability of getting a tail with second throw 0.5, probability of getting a head followed by a tail is = 0.25, or 1/4. Bayesian methods p.21/38 Third law of probability Works also if the coin is biased: Let us say the coin falls 75% on its head. It therefore falls 25% on its tail. so for HT joint event we get = % This relies upon independence between the two events. Here the outcome from the first coin toss cannot influence the outcome from the second coin toss. Bayesian methods p.22/38 Third law of probability Where independence can be assumed the version of the third law is called the product rule. Applied in DNA evidence evaluation as: 1. Alleles are considered independent of one another on a loci (Hardy-Weinberg equilibrium). 2. Alleles considered independent between loci (linkage equilibrium). Unfortunately not all the events which form joint events can be considered independent. In reality very few are. Bayesian methods p.23/38 Third law of probability Probability systems in which non-independence has to be taken account of occur even in fairly simple dice throwing events. From a single throw of a fair, six-sided, die. What is the probability of throwing together: 1. An even number. 2. A number less than four. Bayesian methods p.24/38

7 Third law of probability The probability of: Getting an even number is 0.5 (2,4,6 from 6) A number less than four is also 0.5 (1,2,3 from 6) Assuming independence we simply multiply the two together to get the probability of the joint event, so: = 0.25, or 1/4. Is this correct? Bayesian methods p.25/38 Third law of probability Bayesian methods p.26/38 Third law of probability We need to use the conditional probability: The probability of getting a number less than four is 0.5. That is: from {1,2,3,4,5,6} only {1,2,3} are less than four. From our set of outcomes less than four what is the probability that the outcome is even? We have the set {1,2,3}, and the number 2 is even, so the probability of getting an even number is 1/3 from our reduced set of numbers which are less than four. Multiply these together, 1/2 1/3 = 1/6, which is correct. Bayesian methods p.27/38 Conditional probability The probability 1/3 from the example has a special name. It is called a conditional probability. It is called conditional because the condition for it to be true is that we looked only at those possible outcomes which were less than four. Before the third law could be stated: Pr(A,B) = Pr(A) Pr(B) For a more generally applicable formulation we need to write: Pr(A, B) = Pr(A) Pr(B A) Bayesian methods p.28/38

8 Conditional probability Pr(B A) we have said is the conditional probability. Pr(B A) is the probability of B given A has already occurred. Sometimes called the probability of B conditioned on A. the means conditioned on. To the left of the, called the conditioning bar, is the event for which the probability is required, to the right of the conditioning bar is the condition which must be met. Thus Pr(event to find the probability of condition to be met) Bayesian methods p.29/38 Conditional probability The joint probability of of observing an outcome which is less than four, and is even, is the same as the joint probability of observing an outcome which is even, and less than four. Probability of even is 1/2. Even numbers are {2,4,6}. Of the even numbers only 2 is also less than 4. So Pr(A B) = 1/3 Multiplying these together we get 1/2 1/3 = 1/6 which is the same answer for the joint event. Bayesian methods p.30/38 Unconditional probability It has been pointed out that all probabilities are conditional. The probability that a die roll gets a six is conditional on it being fair and possessing six, or more, sides. The probability that a coin lands heads uppermost about half the time is conditional on it being fair. In fact the term fair indicates the probability that the coin lands heads uppermost about half the time - circular definition. Bayesian methods p.31/38 Unconditional probability There are however a number of probabilities which, in context, we do not explicitly indicate any conditioning for. For example, for the toss of a fair coin I would, by convention, write: Pr(x = T) = 1/2. To be technically accurate I should write: Pr(x = T I) = 1/2, where I indicates a list of conditions. This gets tedious, but is sometimes used in the literature where necessary. Bayesian methods p.32/38

9 Data Table of the occurrence of the rhomboid Foss in men and women. From Rogers et al. (2000) Journal of Forensic Sciences. 45(1): Bayesian methods p.33/38 Joint probability Pr (sex and fossa) present absent male female Bayesian methods p.34/38 Unconditional probability Pr (sex) male 0.67 female 0.33 Bayesian methods p.35/38 Unconditional probability present absent Pr (fossa) Bayesian methods p.36/38

10 Conditional probability Pr (fossa given sex) present absent male female Bayesian methods p.37/38 Conditional probability Pr (sex given fossa) present absent male female Bayesian methods p.38/38