Objectives. 6.3, 6.4 Quantifying the quality of hypothesis tests. Type I and II errors. Power of a test. Cautions about significance tests

Transcription

1 Objectives 6.3, 6.4 Quantifying the quality of hyothesis tests Tye I and II errors Power of a test Cautions about significance tests Further reading: htt://onlinestatbook.com/2/ower/contents.html

2 Toics: Understanding ower Understand how a decision is made in a statistical test. Understand one can always make a wrong decision. But the chance of this can be quantified. Understand Tye I error Tye II error Power. Now how to determine samle size to obtain certain desired ower. Learning cumbersome calculations is not necessary but using and interreting the as is.

3 Evaluating a testing rocedure In a statistical test, one of two decisions are made. Reject null. Do not reject null. Rejection of null, deends on how far the samle mean is from the oulation under the null. The distance is determined by the t or z-value. From the t or z-value a -value is evaluated. The testing rocedure deends on two choices made by the ractitioner. First, the significance level α. Decision: If -value less than α, we reject null. Second, the samle size (which the ractitioner collects).

4 Evaluating a testing rocedure We need to understand how these two factors may have influenced the result (the decision made in the test). When designing your exeriment: we need to know whether the test can reject the null for `imortant alternatives. The alternative is true, but the variability in the data together with the small samle size may mean that detecting the alternative is very difficult. If we conduct the test and we do not reject the null we need to investigate whether the test could `easily rejected the mean an imortant alternative mean were true. We will be making some calculations, you do not have to do these calculation in the exam. You do need to understand the ideas behind them, the concets and the conclusions drawn.

5 Next we consider this little guy.

6 A real story A few years ago while I was execting that little creature, the doctor wanted to determine whether I had gestational diabetes. There exists accurate medical tests to determine if a atient has gestational diabetes, but this takes time and is also costly. Instead, a screening rocess is done to identify likely candidates. The screening rocess is based on a taking a few blood samles and testing whether there is `evidence' of gestational diabetes. Gestational diabetes is diagnosed if the mean glucose level, μ, is greater than 140. The mean level is not known. Here are the results of my blood tests: 139.3,137.1,146.9, The samle mean was What do you think, is there evidence of gestational diabetes? If there is, then the atient is given more invasive tests.

7 Since glucose samles are biological it is assumed to be drawn from a normal distribution. The standard deviation can be estimated from the data, but this will mean the test is not very sensitive to detecting gestational diabetes. Instead a very accurate estimate of the standard deviation is obtained from a blood bank and it is assumed σ = 4. Since σ = 4 we will use the normal distribution. Since 4 blood samles are taken the standard error is 4/ 4 = 2. Since the glucose samle is normally distributed, the samle mean is normally distributed and x N( µ, 2) {z}? μ is the level of glucose in her blood. A erson has gestational diabetes if μ > 140.

8 What not to do A bad rocedure is to directly comare the samle mean with 140 Bad decision: x >140 x ale 140 Go for further testing Don t go for further testing This decision rule would be too costly. If a women has a healthy mean of μ = 140, there is a 50% chance her samle mean will be greater than 140. Using this as the decision rule will lead to too many so called `false ositives, i.e. healthy atients referred for tests. To understand why recall that the data is normal so the average is normal x N( µ, 2) {z}? and the normal distribution is symmetric about µ.

9 We illustrate the above with some examles. x N( µ, 2) {z}? On the left we have the distribution of the samle mean for two healthy atients. If the true mean is µ=138. The roortion of samle means over 140 is about 16%. If the true mean is µ=140, there is a 50% chance the samle mean is greater than 140. One the left we have the distribution of the samle mean for two unhealthy atients. We see the chance of the samle mean being over 140 deends on their true mean level.

10 Detecting Gestational diabetes via testing As we can only rove the alternative we need to state what are looking for in the alternative. If we want to discount the ossibility of the atient being healthy without gestational diabetes (rove gestational diabetes), we set the hyotheses as: H 0 :μ 140 (healthy) against H A : μ>140 (has gestational diabetes). Given the data we decide how lausible it is to obtain that data under the assumtion the null is true (the -value). If the -value small (the null is imlausible) and we reject the null and determine that it is lausible she has gestational diabetes. We recall from Chater 7, that an alternative but equivalent method to using a -value is to determine a boundary where the decision is made. x If is over the boundary, the null is rejected (and further tests done).

11 Constructing the decision boundary Suose the 5% significant level is used. This means if the -value is less than 5% we reject the null. Using the normal tables (since we are not estimating the standard deviation from the data) we know that the area to the right of a standard normal which gives 5% is Therefore the area to the right tail of 140 which gives 5% is = x > This gives an automatic way to do the test. If the samle mean is greater than we reject the null. Remember using this rule there is a 5% chance of falsely diagnosing a healthy atient (i.e., the roortion of healthy eole diagnosed is 5%).

12 The rejection region for this test Imlementing the rule: This means that when a doctor takes 4 blood samles she evaluates the samle mean (average) and comares this number to If this average is less than the doctor cannot reject the null. In other words, the atients is not diagnosed with gestational diabetes. On the other hand, if the average is greater than than it is believed the atient has gestational diabetes and more tests are done. Examles: A atient has measurements: 36.2, 36.8, 39.11, 39.27, the samle mean is Since < 140 < we definitely cannot reject the null. A atient has measurements: 143,143.2,145,144 average is Since 143.8> we reject the null, atient goes on to have further tests.

13 Tye I errors: Examle my case H 0 : μ 140 against H A : μ > 140. We do the test at the 5% level. My samle means was Since is greater than the null was rejected (the -value is less than 5%) To calculate the -value. Make a z-transform = ( )/2 = Since the alternative is ointing right, we calculate the area to the right of 1.65, which from the tables is 4.9%. 4.9% < 5%, therefore I was referred for more tests. Fortunately for me, subsequent medical tests showed that I did not have gestational diabetes. My samle mean of was a little extreme, which triggered the medical tests. I was an examle of a Tye I error; The decision where we falsely reject the null hyothesis when the null is true.

14 q There is a 4.9% chance of a healthy atient yielding the samle 139.3,137.1,146.9,149.9 (and the samle mean 143.3). The 5% significance level means that by using this level, 5% of healthy atients will be referred for extra tests. The insurance comany is hay. 5% is better than giving all atients being given costly invasive tests. The insurance comany would be even haier if they could reduce this level to 1%. Only 1% of healthy atients would have been given unnecessary tests. Since my -value was 4.9%, and 4.9%> 1% level I would not have been given the unnecessary medical tests. But there is a cost to reducing the significance level to 1%.

15 By reducing the significance level, the doctor is less likely to identify atients who actually have gestational diabetes. We require smaller -values to reject the null. This means more evidence against the null is required (we can demonstrate this using the Statcrunch alet later on). We need a balance. A large significance level means more healthy atients will be given unwanted tests. But a large number of unhealthy atients will be caught. A small significance level means fewer healthy atients will be given tests, but less unhealthy atients will be identified.

16 The decision rule and the errors made Truth H 0 true, no Gestational diabetes H A true, Gestational diabetes Decide not to reject null Correct decision Incorrect decision Decide to reject null, H A true Incorrect decision Correct decision We can quantify the chance of our making the correct or wrong decision: Truth H 0 true, no Gestational diabetes H A true, Gestational diabetes Decide not to reject null Correct decision, chance = (1 ) Tye II error, chance = Decide to reject null, H A true Tye I error, chance = Correct decision, chance =(1 ) q We have already covered the tye I error (usually denoted α), this is the chance of rejecting the null when it is true (chance of diagnosing gestational diabetes when the atient is healthy). The chance of this haening is the re-set significance level. In this examle it is 5%.

17 Tye II errors and ower Truth H 0 true, no Gestational diabetes H A true, Gestational diabetes Decide not to reject null Correct decision, chance = (1 ) Tye II error, chance = Decide to reject null, H A true Tye I error, chance = Correct decision, chance =(1 ) q q q The Tye II error (usually denoted β) is the chance of not detecting gestational diabetes when the atient has gestational diabetes (not rejecting the null when the alternative is true). Power is the chance of correctly rejecting the null (diagnosing gestational diabetes when erson has gestational diabetes), Power = 1 β Common mistake: Remember α and β are not related linearly: ie. β (1-α).

18 Power: Detecting gestational diabetes Recall the hyothesis H 0 : μ 140 against H A : μ > 140. Now, we discuss the issue of rejecting the null, when the alternative is true (this is known as ower). This is the same as the statistical test working! Power = 1 (the chance of NOT detecting the alternative when a atient has diabetes) = 1 Tye II error. To do the calculation we need to use alternatives which are of articular interest. Aim: Can the statistical test detect seriously ill atients. To see whether it can, we need to make what is known as a ower calculation. Severe gestational diabetes is when μ is far, far larger than 140. This calculation has nothing to do with data. It is a method of checking how good the actual statistical test is. In this examle, it assesses, how good the test is in diagnosing ill atients.

19 Motivation: Power = 1 Tye II error The sugar level of a regnant women after taking a sugary drink has mean level μ. If the mean μ is above 140, she has gestational diabetes. The roblem is we don t know her mean level. So we make a decision by doing a hyothesis test H 0: μ 140 against the alternative H A: μ > 140. We do the test at the 5% level. Based on the average of four blood samles, if her samle mean (based on 4 blood samles) is greater than (see revious calculation) there is evidence to suggest she has gestational diabetes and a roer medical test is done is our decision threshold Peole who do have diabetes can also have a samle mean which is less than , these are eole who are let through the net. Our statistical test has not caught this erson, so she will not be given treatment did we make a mistake? Are we not treating a erson who may be ill? To answer this question we use the following as.

20 We will make use of the following a tools: Free Alet (made by Dr. Kolodziej): htt://shiny.stat.tamu.edu:3838/eykolo/ower/ If you use this alet make sure the range is secified correctly (by changing lower bound of lot to 135 and uer bound of lot to 150) else it will look strange. This a is a visual tool. It does not give robabilities. The ower calculator in Statcrunch: Instructions: Go to Stats -> Z-stats -> One-samle -> Power/Samle size (this alet will do the ower calculation for you). We will go through it in class (for the column Difference you need to ut the difference between the null and alternative of interest). You do not need to learn the calculation, you simly have to understand the the lots I am going to show you.

21 Examle: Power for a atient with mild diabetes μ=142 This atient has mean level 142 (mild diabetes). The chance of this being detected is the red area to the left of This robability is the area to the RIGHT of =0.64 This is about 25% (look u z-tables). That is there is a 25% chance of detecting gestational diabetes when the mean is 142 (the ower in this case). It s not huge, but when the diabetes is mild it robably does not matter. Remember there is a 75% chance of not detecting diabetes in this atient (this is a Tye II error).

22 Examle: Power for a atient with moderate diabetes μ=145 This atient has mean level 145 (moderate diabetes). The chance of this being detected is the red area to the left of This robability is the area to the RIGHT of = 0.86 This is about 80% (look u z-tables). That is there is a 80% chance of detecting gestational diabetes when the mean is 145. This means there is a 20% chance of not detecting diabetes for this atient. Whether this matters or not deends on the ractitioner. Will the atient be at risk if her diabetes is not detected?

23 Examle: Power for severe diabetes μ 148 This atient has mean level 148 (severe diabetes). The chance of this being detected is the red area to the left of This robability is the area to the RIGHT of = 2.36 This is about 99.0% (look u z-tables). That is there is a 99% chance of detecting gestational diabetes when the mean is 148. This means there is a 1% chance of not detecting diabetes for this atient. Thus there is a very good chance the test will detect some one who has severe diabetes.

24 The 99% we have calculated means that 99% of women who have severe will be detected using the statistical test. To increase the detecting rate either: Increase the Tye I error Or increase samle size. We consider these two cases below.

25 Increasing the Tye I error q To increase the detecting rate: Increase the significance level from 5-10%. This will move the rejection oint from to = (1.28 corresond to the 90% on the normal tables). This means that a erson with a samle mean greater than is referred for further tests. The `cost of this method is that 10% of women without gestational diabetes will be referred for more tests.

26 If you like calculations The chance of detection is the area to the RIGHT of This is the robability to the RIGHT of z = = 2.72 Which is 99.7%. There is a 99.7% of detecting the severe diabetes, but a 10% of falsely diagnosing a erson who is healthy.

27 Increasing the samle size An alternative method to increasing ower without changing the significance level is to increase the samle size. By increasing the samle size we increase the reliability of the samle mean (by decreasing the standard error). This is best seen with a calculation, increase the samle size from 4 to 9, construct the oint at which rejection occurs and again the ower.

28 By increasing the samle size: q We ush the rejection oint closer to 140: q We also make the samle means more centered about the atients mean level. q q q This combination means the chance of detecting severe diabetes is the area to the right of = If you like calculations This is the area to the right of which is % = 4.36 This means with an extremely high robability we will detect severely ill atients. Even the chance of detecting moderate diabetes (μ=145) increases from 79% to 98.21%.

29 Question Time It is known that the healthy mean resting ulse and standard deviation of Sidney the chamion horse is µ= 30 and σ = 6. Sydney looks sick and his owner checks for an elevated ulse by testing the hyothesis H 0 : µ 30 vs H A :µ>30. He measures the ulse twice and does a statistical test at the 5% level. A ulse is considered extremely high the mean ulse level is µ= 50. Using the lot below what is the chance of an extremely high ulse NOT being detected? (A) less than 5% (B) 5% (C) 95% (D) More than 95% htt://

30 Increasing ower q To summarize, there are two feasible ways to increase ower: Increase the significance level, but this will result in more false ositives. Increase the samle size, this also has a cost, since it may be costly to take several samles (we look into this in a few slides). A less feasible method: Another way to increase ower (but does not have any real meaning esecially for this examle), is to move the alternative further from the null. In other words, increase 148 to say 150. However, this really has no ractical meaning, since severely ill eole are those who level is over 148. Choosing the Tye I error, α, is a balancing act between the risk of a Tye I error and the risk of a Tye II error, β. Reducing α (tye I error) always increases b (tye II error).

31 Why ower calculations are imortant The ower of a test of hyothesis with fixed significance level α is the robability that the test will reject the null hyothesis when the alternative hyothesis is true. That is, the ower is 1 b. In other words, ower is the robability that the data gathered in an exeriment will be sufficient to reject a null hyothesis that is false. Knowing the ower of your test is imortant: When designing your exeriment: select a samle size large enough to detect an effect of a magnitude you think is meaningful. For examle in the gestational diabetes examle it was imortant the test should detect μ 148. When your test ends u being non-significant: check that your test would have had enough ower to detect an effect of a magnitude you think is meaningful. If the ower is large, then it is unlikely the alternative of interest is true.

32 Examle 1: Not rejecting the null: Gestational diabetes 6 blood samles are taken to test for gestational diabetes. H 0 : μ 140 against H A : μ > 140 with σ=4. The samle mean of these 6 blood samles is 142. There is no evidence to reject the null hyothesis. The atient is not given further treatment and sent home. Could we have missed a atient who actually had severe gestational diabetes (μ 148)? q We need to do a ower calculation. Using the Statcrunch a we see that From the ower calculator we see that the chance of missing a erson with such severe diabetes is = = 0.056% This is a very small, chance. Thus we can be reassured we have not missed a seriously ill atient.

33 Examle 2: Low otassium Hyokalemia is diagnosed when the blood otassium level is below 3.5mEq/dl. The otassium in a blood samle varies from samle to samle and follows a normal distribution with unknown mean but standard deviation is known to be 0.2. The hyothesis of interest is H 0 : μ 3.5 against H A : μ < 3.5. Tyically, the average of 9 blood samles are taken and the test is done at the 5% level. Question: Will the test be able to detect a critically ill atient with otassium level below 3.3? We translate the above into statistical seak. It asks whether the test will have very high ower (over 99.5%) for detecting eole whose mean level μ 3.3? Answer: The oint of rejection is when the samle mean is less than =3.39

34 Power calculation In this case, the difference between null and alternative of interest is = 0.2. The standard deviation is 0.2 and the samle size is 9 blood samles. The calculator gives 91.23% chance of catching (detecting eole) who have mean level of 3.3 or below. And if you like calculating: The robability of the samle mean being less than 3.39 when the true mean μ=3.3 is the area to the LEFT of the z-transform: z = =1.35 (0.2/3) Which is 91.1%

35 Alication A erson is tested for low otassium using the above rocedure. The test is done at the 5% level (using 9 blood samles). There is no evidence to reject the null, should we be worried that we have missed someone who is severely ill (mean level below 3.3)? The ower calculation shows that the chance of missing a erson who is severely ill is 8.77%. The medical rofessional needs to decide whether this is too high and the consequences of it. If the doctor does decide it is too high then the doctor will either need to collect more bloods samles or do the test at higher significance level (change the level from 5% to 10%).

36 Question Time The recommended amount of exercise a erson should do is 150 minutes or more. If the mean amount of exercise done in a town is 110 minutes or less, the town is determined to be at high risk. In Hearn, 8 individuals were randomly samled and the hyothesis H 0 : μ 150 against H A : μ < 150 tested (the researchers were looking to see if they were doing less than the recommended amount of exercise). Based on this samle, the - value for this hyothesis test based on the data is 15%. A ower analysis was done to assess the test's ability to detect high risk towns, where the mean is 110 minutes or less (the results of the alet are given below).

37 htt://

38 Power Examles (tests at the 5% level) Suose you want to rove someone does not have gestational diabetes. The hyothesis is 4 blood samles are taken and the standard deviation is 4. Use Statcrunch to calculate the chance the testing rocedure will give the all clear to someone whose mean level is μ=136? What is the tye II error for this mean? Suose you want to see if red wine increases olyhenol levels. The hyothesis is The standard deviation is 4. If 10 volunteers are used. What is the ower (ability of test) to detect μ = 5. If 100 volunteers are used. What is the ower (ability of test) to detect μ = 5. If 100 volunteers are used. What is the ower (ability of test) to detect μ = -5? In a tomato acking lant we test the mean H 0 : µ = 227g The standard deviation is 10. H 0 : µ 140 H A : µ<140 H 0 : µ ale 0 H A : µ>0 H A : µ 6= 227g

39 Each test is done using n = 20 tomato boxes. How good is the rocedure at detecting the machine malfunction μ = 234g? Each test is done using n = 20 tomato boxes. How good is the rocedure at detecting the machine malfunction μ = 220g? Each test is done using n = 40 tomato boxes. How good is the rocedure at detecting the machine malfunction μ = 220g?

40 Review: Tye I and Tye II errors A Tye I error is made when the null hyothesis is rejected but the null hyothesis actually is true. (But we do not know that it is true.) The robability of making a Tye I error is the significance level α. The conclusion for the cola study may have been a Tye I error. A Tye II error is made when the null hyothesis is not rejected but the null hyothesis actually is false. (We do not know that it is false.) The robability of making a Tye II error is labeled b, but it is not a fixed number because it deends on the actual truth (true value of μ). The conclusion for the tomato study may have been a Tye II error. Here, error does not mean we made an avoidable mistake. It simly means we got misleading evidence (by chance alone).

41 Choosing a larger α makes it more likely that H 0 will be rejected and thus more likely that a Tye I error will occur (if H 0 is true) but less likely that a Tye II error will occur (if H 0 is false). On the other hand, choosing a smaller α makes it less likely that H 0 will be rejected and less likely that a Tye I error will occur (if H 0 is true) but more likely that a Tye II error will occur (if H 0 is false).

42 Question Time In court we test the hyothesis H 0 : Innocent vs H A: Guilty. Which statement(s) are correct? (A) Using a very low significance level will lead to less innocent eole being convicted but also less guilty eole being convicted. (B) Using a 5% significance level will result in 5% of guilty eole being convicted. (C) Using the 5% significance level will result in 5\% of innocent eole being convicted. (D) [A] and [C] (E) [B] and [C] htt://

43 Examles of Tye I and Tye II errors Recall the gestational diabetes examle: H 0 : μ 140 against H A : μ > 140 The truth is never known, if the atients samle mean has a -value less than 5% (or equivalent the samle mean is greater than143.28), then the atient is diagnosed with gestational diabetes and further tests are done. If the -value is greater than 5% nothing is done. My case was an examle of a Tye I error, where, as a healthy atient, I was diagnosed with gestational diabetes since my -value was 4.9%. A atient with gestational diabetes who has not been diagnosed with gestational diabetes, because her -value is larger than 5% is an examle of a atient who `falls through the net and is an examle of a Tye II error.

44 What is statistical significance? We say the results of a test is statistically significant if the - values is less than a re-decided significance level. Statistical significance only says that the evidence for H a is substantial enough to reject H 0, because it is unlikely that the observed difference from H 0 was due just to random samling. Statistical significance may not be ractically imortant. That s because statistical significance does not tell you about the magnitude of the effect, it only tells you that there is an effect. An effect could be too small to be relevant or imortant. And with a large enough samle size, significance can be reached even for the tiniest effect. This is an examle of imractical significance. This is when making confidence intervals for the mean can be more useful. The confidence interval tells how big the effect actually is.

45 Examle: Imractical significance Let us return to the gestational diabetes examle H 0 : μ 140 against H A : μ > 140. Suose a atient s mean level is μ= By definition this means she has gestational diabetes. But we can see that it is only very slightly greater than 140: though by definition she has the condition, for all ractical uroses she does not. For samle sizes of order 5, 10, 15, it would not be ossible to diagnose her condition (ie. difficult to reject the null). But this is not an issue, since a mean level so close to 140 her condition is so mild it does not require treatment Suose we could take 50,000 blood samles. In this case the ower calculation shows that there is a very large chance of rejecting the null, even when her diabetes is extremely mild. However, with such mild diabetes it is unnecessary to treat the atient. For very large samle sizes we are likely to reject the null even with the alternative is very close to the null and not very useful.

46 Interreting effect size: It deends on the context There is no rule on how big an effect has to be in order to be considered meaningful. In some cases, effects that may aear to be trivial can be very imortant. Examle: Imroving the format of a comuterized test reduces the average resonse time by about 2 seconds. Although this effect is small, it is imortant since this is done millions of times a year. The cumulative time savings of using the better format is gigantic. Always think about the context of the study. Try to lot your results, and comare them with a baseline or results from similar studies.

47 Do not ignore lack of significance Consider this rovocative title from the British Medical Journal: Absence of evidence is not evidence of absence. Having no roof of who committed a murder does not imly that the murder was not committed. Failing to find statistical significance only results in not rejecting the null hyothesis. This is very different from actually acceting the null hyothesis. Lack of significance does not imly that there is no effect. The samle size, for instance, could be too small to overcome large variability in the oulation. Lack of significance also does not imly that there is nothing of interest in your study. A test for a oulation mean or for comaring means does only that; it does not consider other characteristics of the oulation(s) such as standard deviation(s).

48 Interreting significance level and P-value, again The P-value is the roortion of all ossible random samles that would favor H a by as much as or more than our actual samle does, assuming H 0 is true. Since H 0 is rejected when the P-value is less than the significance level α, α is the roortion of samles that would have enough evidence to reject H 0 assuming H 0 is true the Tye I error chance. If an investigator always uses α = 5% then for 5% of the tests he does when H 0 is true, he will reject H 0. He does not know when this haens! He only knows that H 0 has been rejected and what significance level he used. Once comuted and decided, any individual conclusion either is or is not a correct statement about the true oulation arameter value. It is not random. The P-value is not the robability that the conclusion you have comuted and decided is correct.

49 Accomanying roblems associated with this Chater Quiz 12 Homework 5 (Q7) Homework 6 (Q1 and Q2)