What do you think of the following research? I m interested in whether a low glycemic index diet gives better control of diabetes than a high

Transcription

1 What do you think of the following research? I m interested in whether a low glycemic index diet gives better control of diabetes than a high glycemic index diet. So I randomly assign 100 people with type 2 diabetes to follow either a low GI or a high GI diet for one year, and I measure the change in glycated hemoglobin (a measure of plasma glucose over the long term) for these people. The change in glycated hemoglobin between the diet groups is not statistically significant.

2 What do you think of the following research? I m interested in whether a low glycemic index diet gives better control of diabetes than a high glycemic index diet. So I randomly assign 100 people with type 2 diabetes to follow either a low GI or a high GI diet for one year, and I measure the change in glycated hemoglobin (a measure of plasma glucose over the long term) for these people. The change in glycated hemoglobin between the diet groups is not statistically significant. However On looking at the data, I notice that for subjects in the low GI group, the change in glycated hemoglobin is greater in females than males. So I carry out a statistical test using data from the low GI group, with null hypothesis that the mean change for females is equal to the mean change for males. It s statistically significant!! Nobel prize, here I come

3 Back to the planning stages of the study ER waiting times study What we know: Last year s average wait time was 128 minutes. We plan to collect a sample of size 64 patients. Suppose the sample standard deviation (38 minutes) is the population standard deviation. That is, suppose σ=38 minutes. You may assume that the conditions for the nearly normal model for the sample mean are satisfied. a) What values could the sample mean be so that we will reject the null hypothesis? b) Calculate the probability of a Type II error if the mean wait time is now minutes. c) Do you think that you have sufficient power?

4

5

6

7 Analogies for Power: Power is like a flashlight, and significance is hiding in the corner of a room. If your flashlight has enough light (power) you will see significance that is truly there, but if your flashlight is too dim (not enough power) you won't see significance even when it is truly there. Just as the power of a pair of binoculars measures how well they help you resolve a blob into the actual separate objects, so does the power of statistical test reflect your ability to distinguish real differences. Dogs sniffing drugs: To show that there are no drugs in your house you want a dog with a good nose, but who doesn't find anything (high power, negative study). Using a dog with a cold may not find any drugs, but won't be convincing either (low power, negative study).

8

9 Multiple statistical tests: Suppose researchers carried out 80 separate, independent statistical tests, of which 2 were significant at the 5% significance level. 1. If all of the null hypotheses are true, each test has probability of being significant at the 5% significance level. 2. What is the distribution of the number of tests that are significant? 3. What is the probability that 2 or more of the tests are significant?

10 An application of statistical testing: Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction Craig M. Bennett 1, Abigail A. Baird 2, Michael B. Miller 1, and George L. Wolford 3 1 Psychology Department, University of California Santa Barbara, Santa Barbara, CA; 2 Department of Psychology, Vassar College, Poughkeepsie, NY; 3 Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH INTRODUCTION GLM RESULTS With the extreme dimensionality of functional neuroimaging data comes extreme risk for false positives. Across the 130,000 voxels in a typical fmri volume the probability of a false positive is almost certain. Correction for multiple comparisons should be completed with these datasets, but is often ignored by investigators. To illustrate the magnitude of the problem we carried out a real experiment that demonstrates the danger of not correcting for chance properly. METHODS Subject. One mature Atlantic Salmon (Salmo salar) participated in the fmri study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. Task. The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing. Design. Stimuli were presented in a block design with each photo presented for 10 seconds followed by 12 seconds of rest. A total of 15 photos were displayed. Total scan time was 5.5 minutes. Preprocessing. Image processing was completed using SPM2. Preprocessing steps for the functional imaging data included a 6-parameter rigid-body affine realignment of the fmri timeseries, coregistration of the data to a T 1 -weighted anatomical image, and 8 mm full-width at half-maximum (FWHM) Gaussian smoothing. A t-contrast was used to test for regions with significant BOLD signal change during the photo condition compared to rest. The parameters for this comparison were t(131) > 3.15, p(uncorrected) < 0.001, 3 voxel extent threshold. Several active voxels were discovered in a cluster located within the salmon s brain cavity (Figure 1, see above). The size of this cluster was 81 mm 3 with a cluster-level significance of p = Due to the coarse resolution of the echo-planar image acquisition and the relatively small size of the salmon brain further discrimination between brain regions could not be completed. Out of a search volume of 8064 voxels a total of 16 voxels were significant. Identical t-contrasts controlling the false discovery rate (FDR) and familywise error rate (FWER) were completed. These contrasts indicated no active voxels even at relaxed statistical thresholds (p 0 25)

11 Title: Neural correlates of interspecies perspective taking in the postmortem Atlantic Salmon METHODS Subject: One mature Atlantic Salmon participated in the fmri study. The salmon was approximately 18 inches long, weighted 3.8 lbs, and was not alive at the time of scanning. Task: The task administered to the salmon involved completing an open ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine the individual in the photo must have been experiencing. Analysis: Voxel wise statistics on the salmon data were calculated through an ordinary least squares estimation of the general linear model

12 Results: Several active voxels were discovered in a cluster located within the salmon s brain cavity. The size of this cluster was 81 mm 3 with a cluster level significance of P< Out of a search volume of 8,064 voxels, a total of 16 voxels were significant.

13 Moral of the story: If you investigate enough things, you ll find something statistically significant! How to avoid Type I errors ( false discovery ): Understand how they can happen. Plan carefully: o Decide what you will investigate before gathering the data. o Carry out a power analysis to make sure you have sufficient sample size so that you have enough power to detect what you want to. Use statistical procedures that are appropriate for the data.(independent? Satisfy the distributional assumptions?) If you will be carrying out multiple exploratory analyses, learn about procedures designed to control for the false discovery rate. Be honest when reporting results. Be careful not to over state any findings.

14 The first three things you should do with your data, according to Stats: Data and Models 1. Make a picture. 2. Make a picture. 3. Make a picture.