1 The conceptual underpinnings of statistical power

Transcription

1 1 The conceptual underpinnings of statistical power The importance of statistical power As currently practiced in the social and health sciences, inferential statistics rest solidly upon two pillars: statistical significance and statistical power. The two concepts, both of which are expressed in terms of probabilities (i.e., how likely events are to occur), are so integrally related to one another that it is almost impossible to consider them separately. Statistical significance, the first pillar, is a probability level generated as a byproduct of the statistical analytic process. It is computed after a study is completed and its data are collected. It is used to estimate how probable the study s obtained difference or relationship (which is called its effect size) would be to occur by chance alone. Based in large part upon Sir Ronald Fisher s (1935) recommendations, this probability level is often interpreted as an absolute standard. If it is 0.05 or below, the results are said to be statistically significant and the researcher has, by definition, supported his/her hypothesis. If it is 0.06 or above, then statistical significance is not obtained and the research hypothesis is not supported. Statistical power, on the other hand, is computed before a study s final data are collected. It involves a two-step process: (a) hypothesizing the effect size which is most likely to occur based upon the study s theoretical and empirical context and (b) estimating how probable the study s results are to result in statistical significance if this hypothesis is correct. Said another way, statistical significance is used to ascertain whether or not a given effect size can be interpreted as being reliable enough to allow the scientific community to accept a hypothesis once a study is completed. Statistical power, in contrast, is used to ascertain how likely a study s data are to result in statistical significance before the study is begun. It is, in effect, a hypothetical or projected test of statistical significance conducted before an investigator has access to data. Although reliance upon this genre of hypothesis testing is not without its critics (Neyman & Pearson, 1933), the convention of determining whether or not a given effect size is statistically significant (hence can be 1

2 THE CONCEPTUAL UNDERPINNINGS OFSTATISTICAL POWER considered reliable enough to support a researcher s hypothesis) is almost universally employed in empirical research. Even when confidence intervals are substituted for statistical significance, research consumers still check to see whether a zero effect size resides within that reported confidence interval which normally occurs only when statistical significance is not achieved and hence is synonymous therewith. In effect, then, researchers have no choice other than to subject their data to statistical analysis and report the resulting significance level in one form or another. There is, in fact, something rather comforting to many scientists about the definitiveness of a decision-making process in which truth is always obtained at the end of a study and questions can always be answered with a simple yes or no and not prefaced with such qualifiers as perhaps or maybe. 1 Without delving into the philosophical wisdom of this process, its almost universal acceptance in day-to-day scientific practice has resulted in an ironic twist of fate: almost everyone analyzes their data to ascertain whether it is statistically significant but a large number of researchers 2 either do not ascertain how likely they are to obtain this statistical significance prior to conducting these studies or at least conduct them with an insufficient amount of power. What makes this situation ironic is that, to a very real extent, the achievement of statistical significance has become a prime determinant of scientific success and a primary scientific objective in itself. This is because an investigation which does not achieve statistical significance (a) does not support its authors hypotheses, (b) is often not considered to be as reliable as research which does, and (c) is not as likely to be published. From both a pragmatic and a scientific point of view, therefore, it behoves anyone interested in conducting empirical research to do everything legitimately possible to design his/her research in such a way that it has a reasonable chance of obtaining statistical significance and to a large extent this is synonymous with designing research with a reasonable amount of power. In effect, then, this entire volume is dedicated to showing researchers how to determine the probability of obtaining statistical significance when designing their empirical investigations. Said another way, this book is dedicated to the second pillar upon which the scientific inferential process rests: statistical power. Said one final way, this book is dedicated to enabling scientists to be successful in the conduct of their scientific endeavors. Statistical power is therefore of paramount importance, not only because its consideration is a necessary condition of achieving success in scientific research, but also because it constitutes a procedural facet which is largely under the individual scientist s control. As such, it is not only 2

3 THE EFFECT SIZE CONCEPT professionally self-destructive to design research which does not have a high probability of success, it is unethical to do so for the simple reason that all scientific investigations consume scarce societal resources be these monetary in nature or reflected in terms of the time and effort required of their participating human subjects. Since such investments are made on the premise that scientific investigations have the potential of contributing to society and the quality of the human condition, to conduct these investigations without optimizing their chances of success is tantamount to consuming scarce resources under false pretenses. The book s approach to power This book is unique in the sense that it represents the most comprehensive treatise presently available on statistical power, yet the only statistical knowledge it assumes on the part of its users is a conceptual understanding of (a) the arithmetic mean, (b) the standard deviation, and (c) the normal curve. Even this prerequisite knowledge is probably not essential for the actual determination of a study s statistical power, but it is necessary in understanding the process itself. The remainder of this chapter, then, is largely dedicated to providing the reader with a conceptual basis for understanding what statistical power means. To do this it is necessary to first introduce the concept of a study s effect size, after which we will demonstrate how power is actually computed using only the mean, standard deviation, and the normal curve as prerequisite concepts. Chapter 2 presents 11 key design factors which an investigator may manipulate to increase the statistical power of an empirical investigation and Chapter 3 illustrates the use of the power tables presented in this book and provides a number of guidelines regarding the conduct of a power/ sample size analysis. Each of the remaining chapters is dedicated to one of the discrete statistical procedures that can be used to analyze the full spectrum of hypotheses and research designs employed in present day scientific practice. The effect size concept The most integral statistical component in the power analytic process is the concept of a study s effect size, which is nothing more than a standardized measure of the size of the mean difference(s) among the study s groups or of the strength of the relationship(s) among its variables. Although this book requires no computation whatever to conduct a power analysis, it is 3

4 THE CONCEPTUAL UNDERPINNINGS OFSTATISTICAL POWER always necessary to be able to conceptualize a study s most likely outcome in terms of a hypothesized effect size. To examine this concept in a little more detail, therefore, let us assume the simplest possible example: a two-group design with (a) one group receiving an experimental treatment (E), (b) a separate group serving as the control (C), and (c) a single continuous dependent variable (i.e., a measure which can be appropriately described by the mean). In such a study, the intuitively most obvious indicator of the experimental treatment s success will be the size of the difference between its mean and the mean of its control group. Unfortunately, this particular indicator is dependent upon a number of factors including (a) the scale with which the dependent variable is measured and (b) how heterogeneous the research sample turns out to be. Since the primary benefit to be derived from a power analysis is at the design stage, which occurs before subjects are selected and data are collected, it is obviously advantageous to employ an a priori measure of effect size that is independent of the type of dependent variable(s) and the types of subjects the investigator will be using. Fortunately, by expressing the potential mean difference between any two groups in terms of standard deviation units we are able to achieve an effect size measure which is completely independent of both the dependent variable s measurement scale and the type of sample that happens to be selected for the study. This scale- and distribution-free measure of effect size (ES) is expressed by the following formula: Formula 1.1. Effect size formula for two independent groups M ES E M C SD pooled Although computationally quite simple, this formula produces a descriptive statistic of great power and generality. It allows us, for example, to compare directly the effects resulting from two completely different experimental treatments performed on two different samples employing two completely different dependent variables. This very useful characteristic emanates from the standard deviation concept itself, which it will be remembered is, along with the mean, the key component of one of the most successful of all mathematical models: the normal (or bell-shaped) curve. An ES, then, is nothing more than a mean difference expressed in standard deviation units, which means that it is exactly comparable to a z-score (i.e., a standardized score in which the mean is zero and the standard deviation is 1.0). An ES of 1.0 therefore implies that one group s mean differs from that of 4

5 THE EFFECT SIZE CONCEPT another by one standard deviation (which is comparable to a z-score of 1.0). An ES of 0.50 implies that one group s mean differs from that of another by one-half (0.50) of one standard deviation (and can be interpreted exactly the same way as an individual who achieves a z-score of 0.50, meaning that he/she scored one-half of a standard deviation above the mean of his/her reference group). From a research perspective, then, the ES is a superior concept to either the mean or the standard deviation in the sense that it is completely independent of the dependent variable s scale of measurement. The actual value of the standard deviation, like that of the mean, is dependent upon the scale of measurement and the characteristics of the sample. The number of standard deviations a score is from its mean, or the number of standard deviations (or standard deviation units) two scores are from one another, however, is not specific to the particular attribution being described or compared because the ES is expressed in terms of standard deviation units. To illustrate, let us consider some hypothetical results emanating from two separate experiments: one designed to influence the quantitative subtest of the Scholastic Aptitude Test (SAT), which we will assume has a mean of 500 and a standard deviation of 100, and one designed to change attitudinal scores measured by a single item Likert scale (mean 3.00; SD 1.00). Now obviously we could never directly compare the raw mean differences generated by two studies designed to affect such disparate variables. Regardless of how successful the intervention designed to change attitudes was, for example, the experiment as designed could never result in a mean difference greater than 4.00 (since a Likert scale item can only range between 1 and 5). We would hope, however, that an experimental treatment worth testing would differ from its control by considerably more than 4 SAT points. This is where standard deviation units become so helpful in estimating power. Suppose the first experiment resulted in an experimental vs. control (E vs. C) difference of 50 SAT points while the second yielded a mean difference of only one-half of a Likert scale point. At first glance there might seem to be a major discrepancy between the relative potency of these two interventions. As indicated in the computation in Figure 1.1, however, dividing by the appropriate standard deviation indicates that the strength or size of the two interventions is identical when expressed in terms of the ES. This example demonstrates one of the most useful attributes of the ES concept: it provides a distribution-free statistic by which the results (or hypothesized results) of any experiment can be described and by which any two experiments can be directly compared. To be truly useful to investigators who are interested in estimating the amount of power available at the 5

6 THE CONCEPTUAL UNDERPINNINGS OFSTATISTICAL POWER Table 1.1. The ES as an index of the proportion of subjects helped by the intervention ES % of experimental control subjects Experiment 1 (SAT) Experiment 2 (LS) M ES E M C M 0.50 ES E M C SD 100 SD 1.0 Figure 1.1. The ES as a distribution-free measure. SAT, Scholastic Aptitude Test; LS, Likert Scale. design phase of a study, however, it is necessary for them to be able to hypothesize what will be the most likely value for the ES they will obtain once their research has been completed. To do this, it is quite helpful for them to be able to visualize what an ES means. The most straightforward way of doing this is probably to think in terms of mean differences and standard deviations and apply Formula 1.1. For investigators who have trouble doing this, there are other ways of thinking about the ES. One of these is to ignore the size of the mean difference likely to accrue from an experiment and to think only in terms of the percentage of subjects that the intervention is likely to help in comparison with the control. This index is far from perfect, since automatically (i.e., due to chance alone) 50% of the experimental subjects will score as well as or better than the mean of the control subjects with no intervention at all. Still, if an investigator thinks that his/her intervention will result in, say, approximately two-thirds of the experimental group scoring higher than the mean of the control group as a function of that intervention, Table 1.1 could be used to translate this estimate to a hypothesized ES of approximately

7 THE EFFECT SIZE CONCEPT Table 1.2. The ES as a measure of shared variance (r 2 ) or the Pearson r ES r 2 r Other researchers find it more convenient to conceptualize the ES in terms of r 2 or the amount of variance in the dependent variable which is explained in terms of the independent variable. (The independent variable in the present case is defined as whether or not an individual receives the intervention.) Thus, a researcher who thinks in terms of percentage of variance explained (or in terms of the Pearson r, which is a measure of association between two variables) can use Table 1.2 to convert such an estimate directly to an ES, although this index too has its limitations. Most experimental researchers, for example, do not visualize their effects in these terms and have difficulty conceptualizing what it means to say that 6% (a medium ES of 0.50) of the dependent variable is shared by variations in group membership. Finally, those investigators who prefer to think in terms of proportions or success rates may use Table 1.3 to conceptualize the ES. Thus, if the E vs. C average success rate is 50%, an ES of 0.50 translates to an approximate 25% superiority for the intervention as compared to the control. If the E vs. C average success rate is either below or above 50%, then the E vs. C difference (E C) for any given ES is less than the estimates presented in Table 1.3. Hypothesizing an appropriate effect size. Regardless of how an ES is visualized, it is still necessary for an investigator to hypothesize what the most likely value he/she is to obtain from the study being designed prior to conducting a power analysis. This is, if anything, the concept s Achilles heel, 7

8 THE CONCEPTUAL UNDERPINNINGS OFSTATISTICAL POWER Table 1.3. The ES as a measure of differences in proportion (E vs. C average 0.50) ES E vs. C success rate C E E C because many investigators simply throw up their hands in frustration at this step, asking How can I know what the ES will be before I conduct the study? The answer, of course, is that they cannot although they can certainly formulate a hypothesis regarding this value. In reality, the process of estimating a study s ES is similar to the process of formulating a classical hypothesis; the only difference is that this hypothesis states how much better one group will be than another. There are basically three ways in which this is done: (1) The first is by conducting a pilot study and using the ES obtained therefrom. In some cases this is not optimal because of the small number of subjects typically available for pilot studies. Preliminary studies involving a study s primary variables should always be run as a matter of course, however, and the resulting data can often provide relatively good ES estimates. (This is especially true if a relatively large scale preliminary effort involving the same conditions which will be employed in the final study (e.g., the existence of a randomly assigned control group) are employed.) (2) Another method of hypothesizing an appropriate ES is by carefully reviewing similar studies conducted by other investigators and ascertaining the types of results obtained therein. The increased availability of meta-analyses (i.e., review articles which actually compute mean ES values for research studies testing the same basic hypotheses) makes this a relatively attractive option. 8

9 (3) In the absence of a pilot study or sufficiently detailed information from the literature to hypothesize an ES, a very reasonable strategy involves using Jacob Cohen s (1988) ES recommendations. Professor Cohen suggests that researchers hypothesize a small (0.20 standard deviation units for two-group studies), medium (0.50 SD units), or large (0.80 SD units) ES based upon their knowledge of the variables involved. In the absence of specific insights, he recommends choosing a medium ES of This latter advice has historically been adopted by a great many researchers and now has an extremely impressive empirical rationale following Lipsey & Wilson s (1993) seminal meta-analysis of 302 social and behavioral meta-analyses in which they found the average ES of over individual research studies. Incredibly they found the average ES to be exactly 0.50, which leads us to paraphrase Professor Cohen s original advice (at least for social and behavioral research): When in doubt, hypothesize an ES of THE MEANING OFPOWER The meaning of power Once an ES is hypothesized, power becomes quite easy both to compute and to conceptualize. To gain a clearer understanding of what power means, let us consider a typical power analysis. Let us assume that an investigator wishes to evaluate a treatment s success in decreasing patients headache pain as measured by a visual analog scale, that she/he hypothesizes that the most likely ES to accrue would be 0.50, and knows she/he has enough subjects to achieve an N per group of 64. Note that the discussion that follows is equally applicable to determining the sample size needed to achieve a desired level of power or the maximum detectable effect size emanating from a fixed sample size and desired level of power: these are all key, integrally related components and are all subsumed under the concept we are referring to as power and the process we are referring to as a power analysis. To ascertain how much power would be available for such an experiment, all the researcher would be required to do is decide what statistical procedure would be appropriate for this particular experiment. Since only two groups are involved and the outcome variable is continuous in nature, an independent samples t-test is appropriate. The investigator would therefore turn to Table 4.1 at the end of Chapter 4 and locate the power level at the juncture of the 0.50 ES column and the N/group row closest to 64 (i.e., 65) which would yield a power estimate of 0.81, which in turn would lead our investigator to conclude correspondingly that, she/he would 9

10 THE CONCEPTUAL UNDERPINNINGS OFSTATISTICAL POWER have an approximately 80% chance of achieving statistical significance (or of rejecting a false null hypothesis) at the 0.05 level with a sample size of 64 subjects per group if indeed the hypothesized ES of 0.50 were appropriate. To communicate the results of this particular power analysis, our investigator would include a statement something like the following in his/her research report or grant proposal: A power analysis (Bausell & Li, 2002) indicated 64 subjects per group would yield an 80% chance (i.e., power 0.80) of detecting an ES of 0.50 between the experimental and control conditions using an independent samples t-test (two-tailed alpha 0.05). In non-statistical language, what this is telling us is that, if everything goes as planned, our investigator will have an 80% chance of achieving statistical significance. In other words, if the experimental treatment is indeed capable of producing a gain of one-half of the dependent variable s standard deviation as compared to no treatment at all (i.e., if the hypothesized ES is accurate), then eight out of ten properly performed experiments using 64 subjects per group will result in statistical significance at the 0.05 level. Although certainly efficient, there is a decided black box quality to this process. Since the computational procedures for power are not difficult, however, perhaps actually seeing how power could be obtained without the help of tables or computer programs may be helpful in conceptualizing what it really means to say that a study has an 80% chance of achieving statistical significance. The calculation of power. Re-examining the above statement indicates that our researcher has hypothesized an ES of one-half of a standard deviation (i.e., (M E M C )/SD), expects to employ 64 subjects in both the experimental and control groups, and will use a significance level of Actually computing the power available for such a study, then, would require access to only three sources of information: (1) The critical value of the t-statistic (i.e., t cv ) which would be necessary to achieve statistical significance at an alpha level of 0.05 if the N/group employed were equal to 64. This information can be found in tables present in almost all elementary statistics texts or from commonly used statistical packages such as SPSS or SAS and would be equal to This value, then, is the necessary t we must obtain in order to achieve statistical significance. (Note that there are no probabilities here: if a t of 1.98 is obtained using 64 subjects per group the researcher will (i.e., p 1.00) obtain statistical significance.) 10