Optimizing Research Payoff

Transcription

1 64917PPSXXX1.1177/ Miller, UlrichOptimizing Research research-article216 Optimizing Research Jeff Miller 1 and Rolf Ulrich 2 1 University of Otago and 2 University of Tübingen Perspectives on Psychological Science 216, Vol. 11(5) The Author(s) 216 Reprints and permissions: sagepub.com/journalspermissions.nav DOI: / pps.sagepub.com Abstract In this article, we present a model for determining how total research payoff depends on researchers choices of sample sizes, α levels, and other parameters of the research process. The model can be used to quantify various tradeoffs inherent in the research process and thus to balance competing goals, such as (a) maximizing both the number of studies carried out and also the statistical power of each study, (b) minimizing the rates of both false positive and false negative findings, and (c) maximizing both replicability and research efficiency. Given certain necessary information about a research area, the model can be used to determine the optimal values of sample size, statistical power, rate of false positives, rate of false negatives, and replicability, such that overall research payoff is maximized. More specifically, the model shows how the optimal values of these quantities depend upon the size and frequency of true effects within the area, as well as the individual payoffs associated with particular study outcomes. The model is particularly relevant within current discussions of how to optimize the productivity of scientific research, because it shows which aspects of a research area must be considered and how these aspects combine to determine total research payoff. Keywords optimizing research payoff, sample size, power, false positives, replicability Research efficiency is important in virtually every area of science, because the capacity to carry out research is limited by the available time, money, and personnel. In many research areas, there are also limits on the available study material, such as meteor fragments, cells of rare types, or plots of land with particular soil types. In medical research, there may be relatively few patients with a disease under study. In studies on animals, ethical considerations suggest that studies should be kept as small as possible (e.g., Sagarin, Ambler, & Lee, 214). Finally, in situations where research results would have immediate societal value, studies should be as efficient as possible so that the research goals may be achieved and benefits obtained as quickly as possible. In recent years, there has been great alarm over the possibility that current research practices are inefficient, and there have been many recommendations about how efficiency could be increased (e.g., Chalmers et al., 214; Chalmers & Glasziou, 29; Ioannidis et al., 214; Macleod et al., 214). To a large extent, the alarm has arisen because of emerging evidence that many published findings in the scientific literature may not actually reflect the true state of the world (e.g., Ioannidis, 25b; Jager & Leek, 214; Ledgerwood, 214a; Nosek et al., 215; Simmons, Nelson, & Simonsohn, 211; Vul, Harris, Winkielman, & Pashler, 29; Wacholder, Chanock, Garcia-Closas, El ghormli, & Rothman, 24; for an historical review, see Finkel, Eastwick, & Reis, 215). Some of these erroneous findings, often referred to as false positives (FPs), appear to arise because researchers try many alternative data analyses looking for statistically significant results a practice sometimes referred to as p hacking (e.g., Head, Holman, Lanfear, Kahn, & Jennions, 215; John, Loewenstein, & Prelec, 212; Simmons et al., 211). In addition, the tendency of journals to publish only statistically significant results is known to increase the frequency of FPs within the published literature (e.g., Bakker, Van Dijk, & Wicherts, 212; Fanelli, 212; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995). Naturally, concerns over the unexpectedly high frequency of FPs have led to calls for changes in both the questionable practices themselves and in the incentives that promote them (e.g., Button et al., 213; Fiedler, Kutzner, & Krueger, Corresponding Author: Jeff Miller, Department of Psychology, University of Otago, PO Box 56, Dunedin 954, New Zealand miller@psy.otago.ac.nz

2 Optimizing Research ; Ioannidis, 25b, 214; Koole & Lakens, 212; Nosek, Spies, & Motyl, 212; Vul et al., 29). These concerns have also led to the development of methods for identifying and counteracting the effects of these practices on meta-analyses (e.g., Egger, Smith, Schneider, & Minder, 1997; Francis, 212; Ioannidis & Trikalinos, 27; Peters, Sutton, Jones, Abrams, & Rushton, 26; Simonsohn, Nelson, & Simmons, 214b; Sutton, Abrams, Jones, Sheldon, & Song, 2; Ulrich & Miller, 215; van Assen, van Aert, & Wicherts, 215; Wolf, 1986) and to methods for assessing and ethically disclosing the effects of the practices on statistical results (e.g., Sagarin et al., 214). Thus, the recently increased awareness of FPs has resulted in numerous positive steps that can and should be taken to reduce the frequency and impact of FPs in the literature. Although it is clear that these steps can reduce the frequency of FPs in the scientific literature, there is still the fundamental problem that some FPs must arise even if researchers do everything correctly (i.e., they use no questionable research practices). Some FPs are simply inevitable in any research area that relies on statistical inference, for reasons that are well known and are reviewed in the following section. Two of the factors influencing the frequency of FPs the level of statistical significance and statistical power are determined by the researchers own choices. Naturally, then, concern over FPs has led to recommendations that researchers make these choices in such a way as to minimize FP frequency (e.g., Asendorpf et al., 213; Button et al., 213; Ioannidis, 25b; Johnson, 213; Nosek et al., 212). Unfortunately, the same choices that minimize FPs necessarily have other consequences, some of which are undesirable (e.g., Fiedler et al., 212; Fiedler & Schott, in press; Finkel et al., 215; Friston, 212; Ioannidis, 214; Johnson, 213; Mudge, Baker, Edge, & Houlahan, 212; Sbarra, 214). In particular, as will be elaborated later, minimizing FPs also tends to slow the discovery of real effects, known as true positives (TPs). This means that the optimal choices must strike a good balance between reducing the frequency of FPs and maintaining a good rate of TPs a strategy referred to as the error balance approach by Finkel et al. (215). This can only be done within a detailed model in which the overall research payoff is assessed by considering both the costs associated with FPs and the benefits associated with TPs. To contribute to the ongoing discussion about how research practices should be modified to address concerns about FPs, this article presents such a model of total research payoff and shows how this model can be used to investigate which researchers choices would tend to optimize that payoff. In the next three sections, we review the relevant background on statistical hypothesis testing, the inherent rates of incorrect decisions such as FPs, and the costs associated with minimizing FPs. This review is organized primarily around the technique of null hypothesis significance testing (NHST), which is the standard approach in most fields, but the same basic issues also arise within alternative Bayesian procedures (e.g., DeGroot, 1989, pp ). Following that review, we present a model for research payoff and show how the model can be used to optimize payoff within a research area. Among other things, the model shows how the optimal choices for researchers depend on the characteristics of their research area, and this means that it is impossible to identify a universally optimal set of choices that would apply across all areas. Nonetheless, the model is useful in highlighting the factors that researchers must consider in making their choices. NHST For researchers using NHST, the set of all possible studies can be conceptualized as a research scenario like that shown in Figure 1. Depending on the interests of a metaanalyst concerned with topics like research efficiency and FP rates, this set of studies might be fairly tightly constrained (e.g., highly cited clinical trials examined by Ioannidis, 25a), or it might include a rather broad set of topics (e.g., studies in neuroscience examined by Button et al., 213; or studies carried out with the hope of publication in one of three specific psychology journals, examined by Open Science Collaboration, 215). Each individual study within the scenario tests a specific null hypothesis (H ) according to which a particular effect is absent. In some studies the effect is actually absent (i.e., H is true; left side of Fig. 1), whereas in others a true effect is actually present (i.e., H is false; right side of figure). At the end of each study, the statistical analysis leads either to a positive decision that a true effect is present, known as rejecting H (shaded rectangles), or to a negative decision that a true effect may not be present, known as failing to reject H (unshaded rectangles). Thus, four study outcomes can be distinguished, depending on the presence or absence of the true effect under test and the positive or negative decision based on the data. Two of the outcomes represent correct decisions (TPs and true negatives [TNs]), whereas two represent incorrect decisions (FPs and false negatives [FNs]). The possibility of incorrect decisions both FPs and FNs is an unavoidable feature of statistical research, because the results are influenced by random error as well as by the true state of the world. Table 1 depicts the probabilities of the different outcomes shown in Figure 1, both in general and for the numerical example illustrated in the figure. As can be seen from the formulas in the table, the probabilities of the four mutually exclusive and exhaustive outcomes within NHST are determined by three underlying parameters:

3 666 Miller, Ulrich Effect absent (H true) FP TN Effect present TP FN (and journal editors) in many research areas. In terms of the outcomes depicted in Figure 1, Pr( FP ) α = Pr( FP ) + Pr( TN ). (1) 2. The conditional probability of obtaining a negative result given that a true effect is present. This parameter is generally known as the Type II error probability and denoted β, but we will denote it as β(d, n s, α) to emphasize that its value depends on the true effect size d (e.g., as parameterized by Cohen, 1988), the sample size n s, and the chosen α level. In terms of the outcomes shown in Figure 1, Pr( FN ) β( d,n s, α) = Pr( TP ) + Pr( FN ). (2) Fig. 1. A schematic depiction of a given research scenario and the four possible outcomes of individual studies within that scenario. The total area within the large outside rectangle represents all possible studies that might be carried out within that scenario. Studies testing for effects that are absent (i.e., H is true) are shown on the left, and those testing for effects that are present are shown on the right, but of course the researcher does not know whether a particular contemplated effect falls on one side or the other. The result of each study is either a positive decision (i.e., conclude that the effect is present; shaded rectangles), or a negative decision (i.e., conclude that the effect may be absent; unshaded rectangles). True positive (TP) and true negative (TN) decisions are correct; false positive (FP) and false negative (FN) decisions are errors. The figure is drawn so that the areas of the four outcome rectangles (FP, TN, TP, and FN) are proportional to their probabilities under the parameter values shown in Table The conditional probability of obtaining a positive result given that no true effect is present. This parameter is variously known as the alpha (α) level, the Type I error probability, the significance level, or the p level. An important requirement of NHST is that researchers must choose this value before collecting the data, and it is conventionally chosen to be 5% by researchers The α level and sample size determining the Type II error probability are, of course, always known to the researcher. As is considered further in the General Discussion, the true effect size is generally not known precisely but can often be estimated with meta-analysis (e.g., Richard, Bond, & Stokes-Zoota, 23). The complement of the Type II error probability, 1 β(d, n s, α), is known as the power level, and it is the conditional probability of obtaining a positive result given that a true effect of size d is present. 3. The unconditional probability of a true effect. This parameter is sometimes called the base rate of true effects, π 1, and it represents the proportion of all possible studies within a given research scenario for which the true effect is present. This probability is reflected in Figure 1 as π = Pr( TP ) + Pr( FN ). (3) 1 Like the true effect size, the base rate of true effects is also generally unknown, but researchers have a number of ways of estimating its value, as is considered further in the General Discussion. Table 1. Probabilities Corresponding to the Four Possible Individual Study Outcomes (False Positive or FP, True Negative or TN, True Positive or TP, and False Negative or FN) for the Research Scenario Depicted in Figure 1 Outcome Probability Numerical example in Figure 1 FP Pr(FP) = (1 π 1 ) α.9.5 = 4.5% TN Pr(TN) = (1 π 1 ) (1 α).9.95 = 85.5% TP Pr(TP) = π 1 [1 β(d, n s, α)].1 (1.6) = 4.% FN Pr(FN) = π 1 β(d, n s, α).1.6 = 6.% Sum 1 1% Note: The probabilities shown in the numerical example correspond to the relative areas of the four outcome rectangles shown in Figure 1, and these represent a research scenario with α =.5, power [1 β(d, n s, α)] =.4, and a base rate proportion of true effects π 1 =.1. For this scenario, the rates of FPs and FNs are: R fp = Pr(FP) / [Pr(FP) + Pr(TP)] = 4.5 /( ) = 52.9%, and R fn = Pr(FN) / [Pr(FN) + Pr(TN)] = 6. /( ) = 6.6%.

4 Optimizing Research 667 Rates of FPs and FNs Discussions of FPs often focus on the proportion of all positive results within a research area that are false, known as the rate of FPs (R fp, also sometimes called the false positive report probability or the false discovery rate; Benjamini & Hochberg, 1995; Wacholder et al., 24). 1 Formally, the rate of FPs is the conditional probability that a result is false, given that it is a positive result, defined as FP R fp = Pr( ) Pr( FP ) + Pr( TP ). (4) A common misconception about the FP rate is that it should be at most 5% if researchers use the α =.5 level for hypothesis testing (e.g., Cohen, 1994; Falk & Greenbaum, 1995; Oakes, 1986; Pashler & Harris, 212; Pollard & Richardson, 1987). Instead, FP rates can be much larger even approaching 1% for purely statistical reasons. Considering that there may be additional effects of editorial policies and of questionable research practices beyond the purely statistical causes of FPs, some estimates put the FP rate at more than 9% of published findings in certain research areas (e.g., Ioannidis, 25b). The idea that the rate of FPs is at most the 5% α level is a misconception because these are two fundamentally different conditional probabilities. The probabilities are similar in that both increase directly with the proportion of false positive results, as can be seen by noting that the numerators of Equations 1 and 4 are identical. However, the two probabilities have different denominators. The α level indicates the number of FPs relative to the total number of experiments in which H is true [i.e., Pr(FP) + Pr(TN)], whereas the rate of FPs measures the number of FPs relative to the total number of experiments in which H is rejected [i.e., Pr(FP) + Pr(TP)]. Thus, the α level and the rate of FPs are only equal when TPs and TNs have equal probabilities [i.e., Pr(TP) = Pr(TN)], as can be seen by comparing the relevant fractions directly:? R fp = α Pr( FP )? Pr( FP ) =. Pr( FP ) + Pr( TP ) Pr( FP ) + Pr( TN ) This comparison shows that the rate of FPs exceeds the α level whenever TNs are more common than TPs. For example, TNs are much more common than TPs within the research scenario summarized in Table 1 and depicted in Figure 1, and this results in 52.9% FPs even though α =.5. In general, the rate of FPs tends to be (5) large when true effects are rare, simply because there are few opportunities for TPs in that case. As another example, when a researcher tests new cancer treatment drugs, only a small proportion of the candidate drugs may actually be beneficial (e.g., π 1.1), and this produces a relatively high FP rate (e.g., Ioannidis, 25b; Wacholder et al., 24). In an extreme case where all studies tested true H s (e.g., ESP research), true positive results would be impossible (i.e., Pr(TP) = ), and therefore all positive results would be FPs (i.e., R fp = 1). Conversely, FPs would tend to be rare in research scenarios in which most studies tested for true effects (i.e., where π 1 is large). For instance, with an ideal research scenario in which researchers based their studies on perfect theories, all predicted effects would be present and FPs would be impossible (i.e., Pr(FP) =, thus R fp = ). In a scenario for which 8% of studies tested for true effects, the rate of FPs would be 3%, assuming the same α level and Type II error probability used in Table 1. Unfortunately, the base rate of true effects π 1 is difficult to determine, leading to uncertainty about what FP rate should be expected on purely statistical grounds. As studies can erroneously result in FNs as well as FPs, several authors have emphasized that one should also consider the rate of FNs, defined as FN Rfn = Pr( ) Pr( FN ) + Pr( TN ) (e.g., Fiedler et al., 212; Friston, 212; Mudge et al., 212; Vadillo, Konstantinidis, & Shanks, 216). For the research scenario illustrated in Table 1, for example, the rate of FNs is R fn = 6/( ) = 6.6%, meaning that 6.6% of all negative results represent studies in which researchers failed to detect a true effect. FNs may also be costly, as they reflect missed opportunities. For example, it would clearly be a serious error to dismiss a drug as therapeutically ineffective when it was actually helpful. To minimize both FPs and FNs requires an understanding of how the rates of these errors depend upon the characteristics of the underlying research scenario under discussion. Figure 2 shows how the rates of FPs and FNs depend on the key parameters of α level, power 1 β(d, n s, α), and the base rate of true effects π 1. The figure illustrates four important points. First, there are strong effects of the base rate π 1 on the rates of FPs and FNs, and these effects go in opposite directions. Comparing the panels on the left shows that the rate of FPs tends to be larger when true effects are rare (i.e., π 1 =.1 in Fig. 2A versus π 1 =.9 in Fig. 2E). This is because there are more opportunities for FPs when most studies test true H s. In contrast, examination of the panels on the right shows that the rate of FNs tends to be larger when true (6)

5 668 Miller, Ulrich A: π 1 =.1 α: B: π 1 =.1 R fp.4 R fn C: π 1 =.5 D: π 1 = R fp.4 R fn E: π 1 = F: π 1 = R fp.4 R fn Power Power Fig. 2. Rate of false positives {R fp = Pr(FP)/[Pr(FP) + Pr(TP)], left panels} and rate of false negatives {R fn = Pr(FN)/[Pr(FN) + Pr(TN)], right panels} as a function of the α level, study power [1 β(d, n s, α)], and the base rate probability that a true effect is present π 1. effects are more common (Fig. 2F versus 2B), and this happens because there are more opportunities for FNs when most studies test for true effects. Second, the researcher s α level has a strong effect on the FP rate, as can be seen within Figures 2A, 2C, and 2E. As expected, the rate of FPs decreases when the α level is reduced, because H is rejected in a smaller proportion of the studies in which it is true. Somewhat counterintuitively, the researcher s α level has very little effect on the FN rate for a given power level (Figs. 2B, 2D, and 2F). The insensitivity of the rate of FNs to α appears inconsistent with the standard wisdom that decreasing α makes it harder (i.e., less likely) to detect true effects (e.g., DeGroot, 1989; Fiedler et al., 212; Friston, 212; Mudge et al., 212), but there is a simple explanation for the apparent inconsistency. The standard wisdom applies when the effects of decreasing α are considered across studies with a fixed sample size; for that situation, decreasing α also decreases power. In contrast, Figure 2 displays the effect of changing the α level across studies with a fixed power level. This figure thus depicts the situation in which sample sizes are increased to compensate for the power loss that would otherwise result from the change in α level. If sample size rather than power appeared on the x-axes of Figures 2B, 2D, and 2F, then smaller αs would indeed produce visibly larger false negative rates at each sample size. Third, as can be seen within each panel, the FP and FN rates both decrease as power increases. These

6 Optimizing Research 669 decreases are to be expected because increasing power simultaneously increases the number of TPs (which reduces the ratio R fp in Eq. 4) and also reduces the number of FNs (which reduces the ratio R fn in Eq. 6). Fourth, there are interactions among the α level, power, and base rate parameters in the sense that the effect of each parameter depends on the values of the other parameters. For example, both the α level and power have larger effects on the rate of FPs when the base rate of true effects is low than when it is high (Figs. 2A vs. 2E). This pattern is to be expected because of the increased opportunity for FPs when true effects are rare, as mentioned previously. Analogously, power has a smaller effect on the rate of FNs when true effects are rare than when they are common (Fig. 2B vs. 2F), because there is little opportunity for FNs when true effects are rare. Study Costs Based on the fact that α level and power influence the rates of FPs and FNs, as shown in Figure 2, it seems natural to suggest that researchers should seek to maximize the accuracy of their study results by reducing their α levels and by using larger samples to increase power (e.g., Colhoun, McKeigue, & Smith, 23). For example, Schimmack (212) argued that the most logical approach to reduce concerns about Type I error is to use more stringent criteria for significance (p. 553), and he suggested that a value of α =.1 be used in particularly important studies. Similarly, Johnson (213) recommended shifting to α =.5 or α =.1, noting that under some conditions more stringent standards will thus reduce false-positive rates by a factor of 5 or more (p ). Many others have emphasized the importance of increasing sample sizes to enhance study power and thereby reduce FPs (e.g., Asendorpf et al., 213; Button et al., 213; Ioannidis, 25b; Nosek et al., 212). In contrast, however, some have argued against large sample sizes on various grounds. As noted earlier, ethical considerations may warrant keeping sample sizes as small as possible for example, when animals are sacrificed to perform the study (e.g., Lakens, 214; Sagarin et al., 214). In addition, large sample sizes have been criticized for yielding sufficiently high power to detect very small effects that might better be ignored (Friston, 212; Lenth, 21). The latter criticism might not apply to theoretical development where small effects could be quite important, however, and it can also be overcome by using the data to estimate the sizes of effect as well as their existence. More critically, the main argument against decreasing α and increasing power is that these changes lead to more costly studies. In many studies, the most straightforward way to increase power is to increase sample size, and especially large sample size increases are needed with especially small α levels. Figure 3 illustrates the problem by showing what sample size n s is required to achieve various levels of power for a given α level and true effect size d. Assuming finite study resources (e.g., sample sizes), the strategy of decreasing α and increasing power consumes more resources per study, and this reduces the number of studies that can be conducted. As Lakens and Evers (214) put it, Because running studies with large sample sizes is costly, and because the resources that a researcher has available are finite, a researcher is forced to make a tradeoff between the number of studies that he or she runs and the power of these studies (p. 288). To choose the optimal sample size for any research program, then, the benefits of increasing the power for each individual study must be balanced against the reduction in the number of studies conducted. This can only be done within the context of a quantitative model integrating the benefits and costs of the various correct conclusions and errors that might be made. A Model for Research We introduce the model by considering the question of what sample size a researcher should choose. 2 As will become apparent, this very practical problem includes all of the essential ingredients needed to address questions about the optimal level of α, the optimal power, the optimal false positive and false negative rates, and indeed the optimal values for many other parameters of the research scenario. As will be considered in the General Discussion, this model could in principle be applied within a very broad context (e.g., an entire research area, such as social psychology ). To make the presentation simpler and more concrete, however, we will start by considering how it would apply within the much narrower context of a hypothetical team of researchers carrying out initial screening studies of the effectiveness of various drugs, using experimental/control group designs analyzed with two-sample t tests. 3 Because of resource limitations and other practical constraints, this research team can test only a limited total number of participants, n max, with the available funding. 4 In general, if the team uses experimental and control groups of size n s to screen each drug, they can screen only k = n max (2 n s ) drugs. Thus, there is a tradeoff between the number of participants used in testing each drug and the number of drugs tested (e.g., Lakens & Evers, 214), leading to the question of whether it is better to test more drugs with fewer participants per drug or fewer drugs with more participants per drug. That is, how should the research team choose the sample size n s to maximize its total research payoff? To identify the optimal sample size, it is necessary to have a measure of the total research payoff P T associated

7 67 Miller, Ulrich Required n s α: A: d =.2 B: d =.5 with the research team s scientific efforts (cf., Ioannidis, 214, Table 2). We assume that each of the four possible outcomes in Figure 1 (i.e., TP, FP, TN, FN) is associated with its own individual outcome payoff value (i.e., P tp, P fp, P tn, P fn ), where correct decisions (TP, TN) are associated with positive individual outcome payoffs and incorrect decisions (FP, FN) are associated with negative ones. As is considered further in the General Discussion, these individual outcome payoff values would depend on many factors and they probably vary widely across research areas. Similar individual outcome payoff models have been used in the evaluation and optimization of diagnostic testing (e.g., Swets, Dawes, & Monahan, 2) and personnel selection (e.g., Taylor & Russell, 1939), among other areas. In such areas, it is sometimes possible to estimate fairly directly the payoffs associated with the different individual outcomes. Given the individual outcome payoffs, the research team s expected total payoff using a given sample size, E[P T (n s )], can represented as Required n s Required n s C: d = Power Fig. 3. Functions showing the sample size (n s, per group) required to obtain the indicated level of power 1 β(d, n s, α) when using the indicated α level. For this example, computations were carried out for studies analyzed using a two-tailed, two-sample t test with the indicated true effect size (d). Power was computed as the probability of finding a significant result in the same direction as the true effect (i.e., positive observed d values). Note that the scale of the vertical axis differs across panels. n Pr( TP ) P +Pr( ) P max tp FP fp E [ PT ( ns )] =, (7) 2 ns +Pr( TN ) Ptn +Pr( FN ) P fn where Pr(TP), Pr(FP), Pr(TN), and Pr(FN) are the probabilities of the individual study outcomes, and P tp, P fp, P tn, and P fn are their payoffs. The weighted sum of the individual outcome probabilities and payoffs is the expected payoff for a single study, and the total number of studies that can be conducted with sample size n s is n max (2 n s ). To illustrate how this model can be used to calculate and optimize overall research payoff, it is necessary to assume particular values for the parameters within an example research scenario. The parameter values would certainly differ across scenarios, though, so it is only possible to illustrate the model for some particular and necessarily somewhat arbitrary choices. Thus, we want to emphasize that we intend to illustrate the types of conclusions that the model could support after appropriate parameter values were determined, but we do not intend to offer general conclusions that would apply across all research scenarios. We first illustrate the model with an arbitrary set of individual outcome payoffs P tp = 1, P fp = 1, P tn =, and P fn = which we refer to as the simplistic payoffs. These individual outcome payoffs make explicit the idea of assigning zero values to negative results an idea that seems implicit in existing publication biases (i.e., the practice of publishing mainly significant results; see, e.g., Fanelli, 212; Franco, Malhotra, & Simonovits, 214), as unpublished negative results are likely to have little impact. These payoffs also reflect the arbitrary assumption that the

8 Optimizing Research 671 Table 2. Illustration of the Computational Steps Used to Determine the Optimal Sample Size, Power Level, Rate of FPs, and Rate of FNs for a Given Research Area n s 1 - β(d, n s, α) Pr(TP) Pr(FP) Pr(TN) Pr(FN) E[P T (n s )] R fp R fn Note: The computations reflect a two-sample t-test research scenario with α =.5, a base rate of true effects π 1 =.5, an effect size of d =.5 when the effect is present, individual outcome payoffs of P tp = 1, P fp = 1, P tn =, and P fn =, and a total of n max = 1, available participants across all studies. (These are the same parameters used to compute the α =.5 curves shown in Figures 4E, 5E, 6E, and 7E.) For each sample size n s, the first step is to compute power 1 β(d, n s, α) from the noncentral t-distribution, as is described in the Appendix Computational Details in the online Supplemental Material. Second, the individual outcome probabilities Pr(TP), Pr(FP), Pr(TN), and Pr(FN) are computed from α, π 1, and β(d, n s, α) using the equations shown in Table 1. Third, the expected total research payoff P T is computed with Equation 7. Within this research scenario, the computations indicate that the maximum total expected payoff is E[P T (n s )] = 3.974, which is obtained with n s = 8, so n s = 8 is the optimal sample size. The optimal power level is the value of 1 β(d, n s, α) =.152 associated with n s = 8. The rates of FPs R fp and FNs R fn can be computed for each sample size with Equations 4 and 6, and the rates associated with the optimal sample size are R fp =.141, and R fn =.465. gain contributed by one TP is exactly offset by the loss associated with one FP. Table 2 illustrates the computations needed to apply the model within the example research scenario with its particular combination of statistical test (i.e., two-sample t test), effect size d, base rate of true effects π 1, and α level. For each possible sample size n s, the associated power can be computed from the assumed true effect size d and α level using standard techniques (see Computational Details in the online Supplemental Material). The individual outcome probabilities Pr(TP), Pr(FP), Pr(TN), and Pr(FN) can then be computed using their formulas shown in Table 1, and the expected total payoff E[P T (n s )] can be computed from Equation 7. Once the expected total payoff is computed in this manner for each possible sample size, the optimal sample size is easily identified as the one producing the largest expected payoff. Figure 4 shows expected total research payoff, E[P T (n s )], computed using the simplistic individual outcome payoffs. The different panels represent nine distinct research scenarios differing in the base rate probability with which true study effects are present (π 1 ) and in the size of the true effect (d) when it is present, but having in common the simplistic individual outcome payoffs. As suggested by Cohen (1988), we parameterized the true effect size as the ratio of the mean true effect to the standard deviation of the individual data values (d), with effect sizes of d =.2,.5, or.8 generally regarded as small, medium, and large, respectively. Note that the range of the vertical axis differs across the three rows of this figure, because the expected payoff depends strongly on the base rate of true effects. The research team s problem, of course, is to choose the sample size n s that maximizes its expected total payoff, and the results shown in each panel of the figure directly address that problem for a research team working in the depicted scenario. For example, consider the effect of sample size shown in Figure 4I, which depicts a research scenario in which true effects are relatively common and the effects are relatively large when they are present (i.e., π 1 =.9 and d =.8). Depending on the α level, payoff can peak at a sample size of anywhere from approximately 5 to 5. According to this model, it would seem that smaller sample sizes are inefficient because they do not provide enough information about a given drug, whereas larger sample sizes are inefficient because they reduce the number of drugs that can be tested. For the scenario illustrated in Figure 4I, it is also striking that the expected total payoff is larger for α =.5 than α =.1 and α =.1. Thus, the best overall strategy within this scenario is to use α =.5 with a sample size of five. Evidently, the more stringent α values waste resources because they require larger samples to obtain reasonable power (Fig. 3). Note that the expected payoffs for the different α levels converge at a sample size of approximately 6; for this sample size and true effect size, power is nearly 1. for all three of these αs.

9 672 Miller, Ulrich 2 A: π 1 =.1; d =.2 B: π 1 =.1; d =.5 C: π 1 =.1; d =.8 2 α: D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = n s n s n s Fig. 4. Expected total payoff, E[P T (n s )], as a function of α level and sample size (n s ) for nine research scenarios (A I) differing in the base rate probability that a true effect is present (π 1 ) and in the size of the effect when it is present (d). s were computed from Equation 7 using individual outcome payoffs of P tp = 1, P fp = 1, P tn =, and P fn =. Computations were carried out for studies analyzed with two-sample t tests. The existence of a clear peak in the function relating expected research payoff to sample size that can be seen in Figure 4I is generally consistent across the distinct research scenarios depicted in the different panels, with rather small samples often tending to be optimal under the simplistic individual outcome payoffs used for this figure. Comparisons across the different panels also reveal strong effects of the base rate and the true effect size. Expected total payoffs tend to be larger when true effects are more common (i.e., larger π 1 ), because more TPs can be found when more true effects are present. Furthermore, payoffs tend to be larger with larger true effects, because power tends to be greater with larger effects. What might perhaps be surprising and is certainly disappointing is that the expected total payoff can actually be negative when the base rate is low and the effect size is small (Fig. 4A). The negative total payoff means that the indicated sample sizes and α levels would produce more FPs than TPs, so researchers would actually be better off conducting no studies at all than conducting studies with these values. In this scenario, the most stringent α level works best because it produces

10 Optimizing Research 673 fewer FPs, but even it produces only a small positive expected total payoff (.48) with the optimal sample size. In the other panels, though, the least stringent α =.5 value is superior. This α level, which was advocated by Fisher (1926), has traditionally been standard in many research areas. What are the optimal levels of power, rate of FPs, and rate of FNs? The preceding analysis of sample size can be extended to determine the optimal level of study power, the optimal rates of FPs and FNs, and indeed the optimal value for any other function of the probabilities depicted in Figure 1. Such extensions are useful because they allow researchers to consider the more general questions of what level of power, what rate of FPs, and what rate of FNs would be optimal within a given research area. The extensions are possible because the choice of sample size uniquely determines all of the other values, as is illustrated in Table 2. Therefore, once the optimal sample size is chosen within a given research scenario, the corresponding optimal values of these other variables are also determined. In Table 2, for example, the optimal sample size is n s = 8. This sample size s associated power level,.152, is therefore the optimal power level for researchers working under this scenario. Of course, these researchers could increase power by increasing the sample size, but the payoff computations indicate that this would be a bad idea under this scenario with its assumed simplistic individual outcome payoffs. More total payoff would be lost by performing fewer studies than would be gained by having higher power within each study. Figure 5 shows how the optimal power level within each scenario can be identified by plotting the expected total payoff, E[P T (n s )], as a function of study power rather than sample size. In terms of the columns of Table 2, Figure 5 simply plots the expected total payoff on the vertical axis against power on the horizontal axis. Because power is monotonic with sample size, these plots essentially involve stretching and relabeling the x-axes relative to Figure 4. In addition, vertical comparisons between the curves for different α levels within a panel have a different meaning in this figure than in Figure 4. In the earlier figure, such vertical comparisons were between studies with the same sample size but different power levels, whereas in this figure the comparisons are between studies with the same power levels but different sample sizes. Perhaps the most surprising result in Figure 5 is that expected payoffs within a scenario can be optimal even with rather low values of power. The optimal level of power tends to be higher when the base rate of true effects is lower, but even with the relatively small base rate of π 1 =.1, the optimal value of power can be below.9, and power levels down to approximately.5 may perform almost as well due to the flatness of these curves. In some of the cases with larger base rates, the optimal power levels are actually below.5, meaning that expected total payoff is maximized despite the fact that true effects are missed in more than half of the studies testing for them. Although it seems intuitively surprising that such a low level of power could be optimal, the computations reveal that in this scenario the additional power gained by increasing sample size is inadequate compensation for the reduced number of studies that can be conducted. This is, of course, partly because of the nonlinear relationship between sample size and power shown in Figure 3. The model can also be used to address the ongoing discussion of FP and FN rates considered in the introduction (e.g., Ioannidis, 25b; Jager & Leek, 214; Simmons et al., 211). Just as each sample size has an associated power level, it also has associated rates of FPs and FNs that can be computed using Equations 4 and 6 (see, e.g., the R fp and R fn columns of Table 2). Thus, the expected total payoff within each scenario can be plotted against these rates, and it is possible to see which rate is associated with the greatest payoff within each scenario. Figures 6 and 7 show expected research payoffs, again computed with the simplistic individual outcome payoffs, plotted as functions of the rates of FPs and FNs, respectively. The most dramatic implication of these figures is that the optimal rates of FPs and FNs can be quite different depending on the base rate of the true effects. For example, Figures 6A, 6B, and 6C show that researchers conducting studies in an area with a low base rate of true effects (i.e., π 1 =.1) can maximize their expected total payoff by choosing sample sizes that will yield as many as approximately 15% to 4% FPs, depending on their α levels. In contrast, the optimal rate of FPs is far lower when effects are more common (i.e., π 1 =.5 or.9; Figs. 6D 6I), and it can even drop below 1% in some cases. Similarly, Figure 7 shows that the optimal rates of FNs also vary tremendously across research scenarios. FN rates below 5% are optimal when true effects are rare (i.e., π 1 =.1), but the optimal FN rate can exceed 8% when true effects are common (i.e., π 1 =.9). The bottom line, then, is that although the optimal FP and FN rates are determined by a complex interplay of the characteristics of the research scenario (i.e., base rate, individual outcome payoffs, etc.), these optimal rates can still be identified using the model. Different individual outcome payoff values As was already mentioned, the expected total payoffs depicted in Figures 4 through 7 were computed using the

11 674 Miller, Ulrich 2 A: π 1 =.1; d =.2 B: π 1 =.1; d =.5 C: π 1 =.1; d = α: D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = Power Power Power Fig. 5. Expected research payoff as a function of α level and study power for the same nine scenarios and payoff computations depicted in Figure 4. simplistic individual outcome payoffs P tp = 1, P fp = 1, P tn =, and P fn = as an example. The earlier conclusions about optimal sample sizes, α levels, power, et cetera are thus specific to those simplistic payoffs. To gain some insight into how those conclusions might differ with other individual payoffs, we briefly consider in this section what would be optimal with a very different set of payoffs. As is considered further in the General Discussion, identifying appropriate individual outcome payoffs is an extremely difficult problem, especially since the values seem likely to vary widely across research areas. Here, we recalculated total payoffs using a set of individual outcome payoffs that were sensitive to negative findings as well as positive ones: P tp = 4, P fp = 8, P tn = 1, and P fn = 2. These negative-sensitive payoffs were selected based on the rationale that, in practice, negative findings should probably also be given some weight, because any given research program is also likely to derive both benefits from true negative findings and costs from FNs (e.g., Mudge et al., 212). Indeed, Fiedler et al. (212) even argued that, compared to FPs, false negatives often constitute a more serious problem (p. 661). Specifically, we chose these example negative-sensitive values to reflect

12 Optimizing Research A: π 1 =.1; d =.2 B: π 1 =.1; d =.5 C: π 1 =.1; d =.8 2 α: D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = R fp R fp.1.2 R fp Fig. 6. Expected research payoff as a function of α level and the rate of false positives (R fp ) for the same nine scenarios and payoff computations depicted in Figure 4. Note that the scales of the horizontal axes have been adjusted to reflect the different rates of false positives in the different scenarios. scenarios in which (a) positive results are approximately four times as important as negative ones, and (b) the cost of an incorrect decision is approximately twice as large as the benefit of a correct decision. Figures 8 and 9 show results obtained with these negative-sensitive individual outcome payoffs, and there are clearly major differences between these results and those obtained with the simplistic payoffs (Figs. 4 5). One rather trivial difference is that when the base rate of true effects is low (i.e., π 1 =.1), total payoffs are much higher with the negative-sensitive individual outcome payoffs than with the simplistic ones. This is simply because the negative-sensitive payoffs reward correct negative decisions whereas the simplistic payoffs do not.

13 676 Miller, Ulrich 2 A: π 1 =.1; d =.2 B: π 1 =.1; d =.5 C: π 1 =.1; d =.8 2 α: D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = R fn R fn R fn Fig. 7. Expected research payoff as a function of α level and the rate of false negatives (R fn ) for the same nine scenarios and payoff computations depicted in Figure 4. Note that the scales of the horizontal axes have been adjusted to reflect the different rates of false negatives in the different scenarios. The more important consequence of the change in payoffs is the change in the optimal values of sample size and power. For example, when true effects are rare (i.e., π 1 =.1), relatively high power is optimal for the simplistic individual outcome payoffs (Figs. 5A 5C), whereas comparatively low power is optimal for the negativesensitive individual outcome payoffs (Figs. 9A 9C). The reverse happens when true effects are common (i.e., π 1 =.9); in these situations, relatively low power is optimal for the simplistic individual outcome payoffs (Figs. 5G 5I), whereas high power is optimal for the negativesensitive payoffs (Figs. 9G 9I). Naturally, similar qualitative differences due to the individual outcome payoffs can be seen when comparing the optimal FP

14 Optimizing Research A: π 1 =.1; d =.2 α: B: π 1 =.1; d =.5 C: π 1 =.1; d = D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = n s n s n s Fig. 8. Expected research payoff as a function of sample size (n s ) and statistical significance level (α), for the same nine scenarios and payoff computations depicted in Figure 4. Total payoffs were computed from Equation 7 using the negative-sensitive individual outcome payoffs of P tp = 4, P fp = 8, P tn = 1, and P fn = 2. rates or the optimal FN rates (not shown). Clearly, the dramatic effects of the individual payoffs on the optimal values of sample size, power, etc., indicate that the exact payoffs must be considered explicitly in order to justify any recommendations concerning these values. Extensions The proposed model is only a first approximation to the measurement of research payoff, but its basic principles are general enough to be extended well beyond the limited set of research scenarios examined so far. For example, to this point, we have considered only studies analyzed with the two-sample t test for a difference between two means, but the model can also be easily extended to other statistical tests (see Computational Details in the online Supplemental Material), and the results are quite similar. Thus, we suspect that conclusions about optimal research strategies may depend little on the particular statistical test under consideration. Another way to generalize the model presented here is to consider more complex research scenarios. For

15 678 Miller, Ulrich 15 A: π 1 =.1; d =.2 B: π 1 =.1; d =.5 C: π 1 =.1; d = α: D: π 1 =.5; d =.2 E: π 1 =.5; d =.5 F: π 1 =.5; d = G: π 1 =.9; d =.2 H: π 1 =.9; d =.5 I: π 1 =.9; d = Power Power Power Fig. 9. Expected research payoff as a function of α level and study power for the same nine scenarios depicted in Figure 4. Total payoffs were computed from Equation 7 using the negative-sensitive individual outcome payoffs of P tp = 4, P fp = 8, P tn = 1, and P fn = 2. example, we have considered only scenarios in which there is either no effect or an effect of a fixed size, but it is easy to handle scenarios in which effects of various different sizes might be present. As an example, Varying Effect Sizes in the online Supplemental Material shows how the model can be extended to a scenario in which effects of different sizes could be present and in which the payoffs for TPs and FNs depend on the effect sizes. The extended model can be used to compute the expected total payoff as a function of α level and sample size for such scenarios, and it can thus again to help identify the optimal values. So far, our illustrative calculations have only addressed situations in which the research scenario is the same for all studies. Real research is more complicated than this, especially because the outcome of one study often informs the selection of the next study, but the model can also be extended to such situations. For one thing, the individual outcome payoff values associated with a particular individual study should perhaps not only reflect the possible impact of that study per se but also reflect the contribution of that study to its wider literature. 5 In a contemplated drug study, for example, the negative weight assigned to an FP might reflect not only the costs