Organizational significance

Transcription

1 STRATEGIC ORGANIZATION Vol 6(2): DOI: / Copyright 2008 Sage Publications (Los Angeles, London, New Delhi and Singapore) SO!APBOX EDITORIAL ESSAYS Organizational significance J. Myles Shaver University of Minnesota, USA As a field, we have become overly focused on the hypothesis testing aspect of statistical inference and have largely neglected interpreting the estimates that underlie our hypothesis tests (e.g. Shaver, 2006, 2007; Starbuck, 2006). In this essay, I argue that assessing the magnitude of estimated effects (i.e. effect size) is only a first step in assessing the importance of empirical findings. Of greater importance is translating effect sizes into a discussion of organizational significance. I contend that assessing organizational significance requires substantial care and judgment. Because of this, it is incumbent upon us as authors to communicate the rationale underlying our judgment. Not only does this convey the foundation from which we derive our interpretations, it provides richer descriptions of the phenomena that we study, and can encourage debate as to what are important vs unimportant organizational effects. The following pages present what I believe are important considerations to aid judging organizational significance. The descriptions and examples demonstrate why assessing organizational significance requires that we go well beyond the mechanical reporting of statistical significance and even beyond the reporting of effect sizes. Beyond statistical significance to effect sizes The concern with focusing solely on statistical significance when reporting empirical results is that a statistically significant effect does not necessarily reflect a meaningful or important effect (e.g. Cohen, 1994; McCloskey and Ziliak, 1996). To understand this, it is useful to revisit the role of hypothesis testing in statistical inference. Hypothesis tests assess if an estimated parameter is consistent with a value of that parameter in the underlying population from which our sample is drawn (i.e. the maintained or null hypothesis). The case for looking beyond statistical significance is best illustrated by example. Consider flipping a coin. A coin is fair if the probability of heads equals the probability of tails, which equals 0.5. I select a coin and start with the maintained hypothesis that the coin is fair. I flip the coin 1000 times and 51 percent of the flips land heads. Is the coin fair? Likely and a binomial test 185

2 186 STRATEGIC ORGANIZATION 6(2) with the null hypothesis that the coin is fair would support that intuition ( p.55). 1 Now I flip the coin 10,000 times and get 51 percent heads. At this point, I would doubt the coin is fair because a coin that flips heads 51 percent of the time is not a fair coin (i.e. 51 percent 50 percent) and with this many flips I would expect the sampled percentage to better reflect the true percentage of the coin. A binomial test that the coin is fair would support that assessment ( p.05). Now I flip the coin 100,000 times and still find that 51 percent of the outcomes are heads. I can be almost certain that the coin is not fair and the binomial test would corroborate this conclusion ( p ). Knowing that a coin is unfair is only part of the story because an unfair coin that flips heads 51 percent of the time is very different to an unfair coin that flips heads 80 percent of the time. However, if I focus solely on statistical significance, I can reach the conclusion that a coin that flips heads 51 percent of the time (or even 50.1 percent of the time) is not fair with a large-enough sample. This illustrates the concern of relying solely on hypothesis tests. Hypothesis tests inform whether or not we should refute the maintained hypothesis; they do not necessarily inform whether the deviation from the maintained hypothesis has any practical importance. Bringing the example into the realm of empirical strategy and organization research, consider interpreting a coefficient estimate from OLS. The standard hypothesis test reported in statistical-package output reflects the null hypothesis that 0 (i.e. the independent variable does not affect the dependent variable). Therefore, when the p-value of a coefficient estimate is.02, the interpretation is that there is a 98 percent chance that the estimate does not come from an underlying population where the effect is zero assuming that the test is correctly specified. However, concluding that an effect is non-zero is different from concluding that an effect is meaningful. It is for this reason that I have encouraged that we pay closer attention to the magnitude of the coefficient estimates we examine (i.e. the effect size) (Shaver, 2007). 2 Beyond effect sizes to organizational significance Although assessing the magnitude of coefficient estimates helps escape the myopia of focusing only on significance tests, it does not address the question: what constitutes an important effect size? 3 Due to the nature of the phenomena that we study in strategic organization research, there is no single generalizable metric to assess if an effect is important or not. For example, I cannot universally say that increasing the expected probability of an organizational event by X percent is important or unimportant whether X is 1, 10, 50 or 90. This assessment is contingent on what I am studying. Likewise, whether or not an action that increases firm revenues by $10,000 is meaningful or not depends on the context. Explicitly

3 SHAVER: ORGANIZATIONAL SIGNIFICANCE 187 tackling this issue is what underlies assessing organizational significance. Moreover, judgment and interpretation provide the foundation for this assessment. Assessing organizational significance requires judgment to interpret the magnitude of effects, taking into account the organizational context. Return to the example of an organization action that increases firm revenues by $10,000. This effect would likely be trivial in the context of Fortune 100 companies; yet might be important in the context of startup firms. As one can see, interpreting the effect within the context of the sample can aid in assessing organizational significance. Therefore, presenting and referring to additional descriptive statistics is a straightforward and useful action to facilitate the interpretation of organizational significance. Contextual factors that aid the interpretation of organizational significance need not be restricted to quantitative descriptions of the variables within our samples. Qualitative information about the context can often aid assessing organizational significance. Returning to the previous example, if the revenue increase translates entirely into profits due to the nature of the business, one could argue that the effect would be more meaningful than if only a fraction of the revenue translates into profits. Likewise, in an organizational setting where revenues are assumed to be almost completely determined by external factors due to regulation, demonstrating any type of managerial impact on revenue generation could constitute an organizationally significant effect. In these situations, authors qualitative understanding about their research settings can inform the translation of effect size to a discussion about organizational significance. Both quantitative and qualitative information about the organizational context can aid assessing organizational significance. Additional complications when interpreting organizational significance Assessing organizational significance is further complicated because the phenomena we study are often characterized by categorical outcomes, rare events and heterogeneous samples (e.g. firms). Each of these conditions places additional demands on interpreting how independent variables affect the dependent variables that we study. Therefore, these conditions require additional care and judgment when translating effect sizes into assessments of organizational significance. The ambiguity of assessing what is an important effect size becomes pronounced for categorical dependent variables for two reasons. First, the organizational importance of falling into one category vs another can vary greatly depending on the definition of the variable. Consider the following categorical dependent variables: organization failure and CEO turnover. Although CEO turnover can be an important organizational event, organization failure is

4 188 STRATEGIC ORGANIZATION 6(2) arguably much more important. Consequently, the threshold I would consider when interpreting an organizationally significant effect would be lower in the case of organizational failure than for CEO turnover. How much lower? Again, this is a matter of judgment. However, I could undertake the following thought experiment to try and calibrate. What is the magnitude increase in the likelihood of CEO turnover I would trade for a 1 percent decrease in the likelihood of organizational failure? 4 With categorical dependent variables, organizational significance reflects both the impact of the independent variable on the dependent variable (i.e. effect size) and the organizational importance of the dependent variable. The second complication of interpreting organizational significance with categorical dependent variables is that effect sizes in such cases provide information on how an independent variable affects the expected probability of a categorical outcome. However, with a categorical dependent variable, one of the outcomes is realized not the expected probability of the outcome. For example, the underlying probability that a startup firm succeeds might be 2 percent. However, each startup either fails or succeeds. The distinction between realized outcomes and expected probability of outcomes is particularly important when strategic actors do not sample repeatedly. For example, an individual entrepreneur might only invest in one startup, whereas a venture capitalist might invest in 100. For the entrepreneur that starts only one business, the underlying expected probability of success is not what he or she realizes. His or her business is either a success or not. In contrast, although any individual business that the venture capitalist backs either succeeds or fails, their portfolio of businesses can reflect the expected probability of success. In this example, discussions about expected values might be meaningful if my focus is on venture capitalists; yet less meaningful if my focus is on entrepreneurs. The distinction between expected probability and realized outcomes is especially important when categorical outcomes are rare events. Consider the example where we predict CEO turnover and we find an effect that increases the likelihood of CEO turnover by 0.5 percent (i.e. the marginal effect is 0.5 percent). 5 This increase has different practical implications if the base rate of CEO turnover is 0.5 percent vs 50 percent. One percent is 0.5 percent more than 0.5 percent. Moreover, it doubles the expected probability of CEO turnover. However, while 50.5 percent is 0.5 percent more than 50 percent, it increases CEO turnover by a tiny fraction of the underlying incidence. Because CEOs either turn over or not, doubling the relative expected probability of turnover can have a different practical implication. With categorical dependent variables, assessing organizational significance requires that we consider the base rate at which the event occurs in the population. I would like to stress that considering the underlying rate that an event occurs in the population does not mean that we do so mechanically and suspend judgment when assessing organizational significance. For example, the expected probability that I win Powerball (a multi-state lottery) equals

5 SHAVER: ORGANIZATIONAL SIGNIFICANCE 189 This is certainly a rare event. However, doubling the chance that I win (i.e. expected probability) makes no practical difference to my actually walking away with the jackpot (i.e. the realized outcome) because there are still a lot of zeros after the decimal place! Another complication when translating effect sizes to a discussion of organizational significance arises with heterogeneous samples. For example, even within a sample of publicly traded US firms there is usually great heterogeneity in assets, market capitalization, profits, employment, R&D spending and many other organizational attributes. Although heterogeneous samples are advantageous for invoking variance in the attributes that we wish to study, they also pose the following challenges when interpreting effect sizes. First, with heterogeneous samples there can be advantages to scaling dependent variables to facilitate comparison across very different observations. For example, if my dependent variable is market capitalization, the market capitalization of GE and Microsoft are difficult to compare to other firms even if these firms are publicly traded US companies. For that reason, I might find it beneficial to examine a scaled measure of market capitalization such as the market-to-book ratio. The advantage of doing so is that it makes the heterogeneous firms more comparable. Thus, it could take into account that rounding errors for very large firms are meaningful decision criteria for small firms. Therefore, ratios can be a valid way to measure decision criteria. For instance, increasing market-to-book might have the same impact on organizational decisions for different sized firms. However, if the impact of the decision is the focus of a study, then nonscaled values might be more valid measures even in samples of heterogeneous firms. For example, if I want to study reduction in pollution emissions, then the non-scaled value of emission reduction might be a better measure than the ratio of emission reductions. This is because DuPont reducing emissions by 1 percent is likely a lot more meaningful than a small private chemical company reducing pollution by 50 percent. Scaled variables pose additional demands when interpreting organizational significance. Although scaling variables can help reveal organizationally significant effects, they can also mask them. Second, heterogeneous samples can complicate the interpretation of marginal effects. With non-linear estimators (e.g. probit and Poisson regression models), marginal effects depend on where on the estimated curve an observation lies. If firms lie in different places on the curve, it will often not make sense to assess the marginal effect for the average firm being studied. With linear estimators, unless explicitly modeled, the marginal effect does not vary depending on the attributes of the firm being studied. However, the marginal effect might not be meaningful to interpret with the restriction that they do not vary over a heterogeneous sample. Heterogeneous samples demand that we interpret effect sizes at multiple points within our sample in order to assess organizational significance.

6 190 STRATEGIC ORGANIZATION 6(2) Organizational significance is not R 2 R 2 is the proportion of variance of the dependent variable explained by the independent variables. R 2 is bound by zero and one. When R 2 equals zero, it means that the independent variables explain none of the variance in the dependent variable. When R 2 equals one, the independent variables explain all variation in the dependent variable. Although organizationally significant effects can explain a substantial proportion of the variation in dependent variables and organizationally non-significant effects can explain little proportion of the variation in dependent variables, this is not always true. For this reason, determining organizational significance requires interpreting effect sizes and not focusing on R 2. The following arguments demonstrate why. Because R 2 is a proportion, its value is a function of the numerator and the denominator. Therefore, R 2 is a function of the overall variation in the dependent variable, in addition to the variance explained by the independent variables. This consideration can be important because in strategic organization research many important phenomena like firm performance are complex and multifaceted. What affects firm performance? Possibly hundreds or thousands of different effects, which in part explains why there can be such great heterogeneity in firm outcomes. It would be ridiculous to expect to model and measure factors that would explain almost all of the variance in a dependent variable like firm performance. Nevertheless, within this variance there might exist several important effects that individually do not explain a large proportion of the total variance. For example, because there is substantial variation in profitability, it might be difficult to estimate a model with high R 2. However, in this situation it is still possible to estimate effects of meaningful magnitude. The reason that the R 2 would not be high is that other meaningful effects are not included in the model. 6 Again, this situation is plausible with complex dependent variables like firm performance. Consider another example, which mirrors the previous situation. It is possible to imagine a dependent variable that has little variation and the range over which the dependent variable varies has no practical importance. For example, I have two machines in my factory that fill 100 ml bottles of water. Due to the calibration of the machines, one fills to ml (99 times out of 100 due to some randomness) and the other to ml (99 times out of 100 due to some randomness) and I know which machine fills to each level. Knowing the machine that filled the bottle would almost perfectly predict the exact amount of water in each bottle and thus result in a high R 2. However, explaining almost all of the variation might have little practical importance. If this was the case, then I would not conclude that this is an organizationally significant effect even though the R 2 is large. The preceding examples demonstrate how focusing on R 2 can mislead when interpreting organizational significance. However, in each of these cases

7 SHAVER: ORGANIZATIONAL SIGNIFICANCE 191 interpreting effect size leads to more informed interpretations of organizational significance. I would like to stress that my point is that effect size is what informs organizational significance, and there is not a one-to-one correspondence to important effect size and high R 2. Depending on the context, important effects do not necessarily explain a large proportion of variance of the dependent variable. Likewise, effects that explain a large proportion of variance of the dependent variable are not necessarily important effects. Therefore, effect sizes rather than R 2 are key to assessing organizational significance. What is the importance of R 2 if it is not determining organizational significance? It reflects how well the model predicts the dependent variable. If one s goal is prediction (e.g. I want to predict the performance of firms) vs identifying relationships between predicted variables (e.g. foreign firms perform better than indigenous firms), then assessing R 2 would be appropriate. However, most research in strategic organization involves testing theories that predict relationships between variables vs trying to most accurately predict organizational outcomes. Conclusion Assessing organizational significance demands that we do more than mechanically transfer t-tests and coefficient estimates from a statistical package output into our papers. It requires that we first calculate effect sizes, and then take into account the nature of the phenomena that we study and the context of the samples that we examine in order to assess their importance. It requires judgment, interpretation, deliberation and care. A benefit of assessing organizational significance is that it requires that we take our empirical estimates back to the phenomena that we study. Given the nature of what we study, we cannot divorce our statistical analyses from the context that generates them. Tightening this linkage will hopefully have the desired impact that we can better talk to each other and to managers about the relevance of our research. I hope the considerations set out in this essay help stimulate thought of how to translate empirical results into a discussion of organizational significance. I view the points that I raise as considerations rather than rules. Strict adherence to rules can trap us into suspending our judgment. The goal of discussing organizational significance is that we allow judgment to play a more central role in the interpretation of empirical findings. Acknowledgments I appreciate helpful comments from Gary Dushnitsky, Adam Fremeth, Xavier Martin, Rob Salomon, David Souder, Minyuan Zhao and the coeditors of Strategic Organization.

8 192 STRATEGIC ORGANIZATION 6(2) Notes 1 From the binomial distribution, the probability that I flip exactly x heads out of n total flips with a fair coin ( n x ) 0.5 n. The two-tailed test in this case is the probability that x 510 or x Shaver (2007) describes several practical considerations when calculating effect sizes in strategy and management research. 3 Although Starbuck (as cited in Barnett, 2007: 117) also stresses the need to go beyond assessing statistical significance and assess substantive importance, he provides little guidance on how to assess this beyond to think about the practical or theoretical significance of empirical findings. 4 Although similar to the previous example, this is a different issue. In the previous example, I had to gauge whether increasing revenues by $10,000 was important. Here, the metric of a dollar is well defined. The judgment involved comparing this well-defined metric to the same metric with respect to total revenues. The point I wish to highlight in this example is that there is not a well-defined metric for what it means to realize a categorical organizational event and the underlying importance can vary greatly depending on the event. 5 To be clear in the use of the term, the marginal effect demonstrates the increase in expected probability. A marginal effect of X means the expected probability increases by X, not that the expected probability changes by the underlying probability multiplied by X. Moreover, I focus on interpreting marginal effects rather than the odds ratio or the log odds when considering categorical variables, because I believe understanding chances in the expected probability provides more meaningful interpretations than changes in the ratio of an event occurring to it not occurring (i.e. the odds ratio) or the natural logarithm of that ratio (i.e. log odds). 6 Of course, how these unobservable effects correlate with the included variables affects the validity of inference we can draw (e.g. Shaver, 1998). References Barnett, M. L. (2007) (Un)learning and (Mis)education through the Eyes of Bill Starbuck: An Interview with Pandora s Playmate, Academy of Management Learning and Education 6(1): Cohen, J. (1994) The Earth is Round (p.05), American Psychologist 49: McCloskey, D. N. and Ziliak, S. T. (1996) The Standard Error of Regression, Journal of Economic Literature XXXIX: Shaver, J. M. (1998) Accounting for Endogeneity When Assessing Strategy Performance: Does Entry Mode Choice Affect FDI Survival?, Management Science 44(4): Shaver, J. M. (2006) Interpreting Empirical Findings, Journal of International Business Studies 37(4): Shaver, J. M. (2007) Interpreting Empirical Results in Strategy and Management Research, Research Methodology in Strategy and Management 4: Starbuck, W. (2006) The Production of Knowledge. Oxford: Oxford University Press. J. Myles Shaver is the Pond Family Chair in the Teaching and Advancement of Free Enterprise Principles at the Carlson School of Management, University of Minnesota. His research interests revolve around corporate strategy choices and their impact on

9 SHAVER: ORGANIZATIONAL SIGNIFICANCE 193 performance. In particular, his research focuses on the management and economics of international expansion and corporate expansion through diversification and mergers and acquisitions.to date, Myles s work has been published in the Academy of Management Review, International Journal of Industrial Organization, Journal of Economics and Management Strategy, Journal of International Business Studies, Management Science, Small Business Economics, Strategic Management Journal, Strategic Organization, Asia-Pacific Journal of Management and various books. Address: th Avenue South, Suite 3 365, Minneapolis, MN 55455, USA. [ mshaver@umn.edu]