Since many political theories assert that the

Transcription

1 Improving Tests of Theories Positing Interaction William D. Berry Matt Goler Daniel Milton Floria State University Pennsylvania State University Brigham Young University It is well establishe that all interactions are symmetric: when the effect of X on Y is conitional on the value of Z, the effect of Z must be conitional on the value of X. Yet the typical practice when testing an interactive theory is to (1) view one variable, Z, as the conitioning variable, (2) offer a hypothesis about how the marginal effect of the other variable, X, is conitional on the value of Z, an (3) construct a marginal effect plot for X to test the theory. We show that the failure to make aitional preictions about how the effect of Z varies with the value of X, an to evaluate them with a secon marginal effect plot, means that scholars often ignore evience that can be extremely valuable for testing their theory. As a result, they either unerstate or, more worryingly, overstate the support for their theories. Since many political theories assert that the effects of variables vary epening on the social, political, economic, or strategic context, moels specifying interaction among variables are ubiquitous across all subfiels of political science. 1,2 A consequence is that conitional hypotheses such as X has a positive effect on Y that gets stronger as Z increases are extremely common. It is well establishe that interactive statistical moels containing multiplicative terms, such as XZ, are appropriate for evaluating such conitional hypotheses (Aiken an West 1991; Clark, Gilligan, an Goler 2006; Frierich 1982; Wright 1976). 3 A number of authors in recent years have offere valuable avice on how to improve research-testing theories positing interaction by properly specifying the expecte conitionality in a statistical moel an effectively presenting an interpreting the results (Braumoeller 2004; Brambor, Clark, an Goler 2006; Kam an Franzese 2007). However, even researchers following this avice often ignore valuable empirical evience that can be easily erive from their estimate moel an as a result, fail to assess all of the preictions generate by their theory. The result is that many researchers either unerstate, or, more worryingly, overstate the empirical support for their conitional theories. The inaequacy of many empirical tests of conitional theories can be trace to the tenency of scholars positing interaction between two variables to conceive of these variables as having ifferent roles within the theory. One variable, Z, is typically viewe as the conitioning variable, the role of which is to moify the impact of the other variable, X, on the epenent variable, Y. Certainly, when X an Z interact, it is reasonable to conceive of Z as conitioning the effect of X on Y. However, it makes little sense to view X an Z as having funamentally ifferent theoretical roles by esignating one of the variables as a conitioning variable an the other as not. This is because, logically, all interactions are symmetric (Brambor, Clark, an Goler 2006; Kam an Franzese 2007). In other wors, if Z moifies the effect of X on Y, then X must moify the effect of Z on Y. Some might view conceiving of one variable as the conitioning variable an failing to acknowlege the symmetry of interaction as merely a semantic 1 The ata an all computer coe necessary to replicate our results are publicly available at the authors web sites. Stata 11 was use for all statistical analyses. 2 In a systematic examination of three leaing journals (American Journal of Political Science, American Political Science Review, an Journal of Politics) from 1996 to 2001, Kam an Franzese (2007, 7 8) fin that fully 24% of articles employing statistical methos teste theories preicting interaction. 3 We treat the terms theory positing interaction, interactive theory, an conitional theory as synonymous. The Journal of Politics, Vol. 0, No. 0, xxx 2012, Pp oi: /s Ó Southern Political Science Association, 2012 ISSN

2 2 william. berry et al. 4 As of January 2009, 44 publishe articles in the ISI Web of Knowlege atabase cite BCG s (2006) article an present at least one marginal-effect plot. problem. We emonstrate, however, that this practice can have pernicious consequences, leaing researchers to ignore empirical evience relevant for testing their theory. In a much-cite 2006 article, Brambor, Clark, an Goler [hereafter BCG] emonstrate that political scientists can greatly increase their ability to impart substantively meaningful information from interactive moels by using parameter estimates to construct a marginal effect plot for an inepenent variable, i.e., a graph that shows how the marginal effect of the variable varies with the value of another variable. Scholars have respone in large numbers to BCG s call to incorporate marginal effect plots into their analyses. Inee, within three years of the appearance of BCG s article, at least 44 publishe papers presente such plots. 4 This has ramatically improve the interpretation of statistical results from interactive moels in the literature. Ironically, however, BCG s article may have inavertently encourage its reaers to make the mistake of viewing one variable as the conitioning variable. Although BCG (2006, note 9) correctly observe that interactive moels are symmetric an that the marginal effect of each inepenent variable is a meaningful quantity of interest, they go on to imply that analysts might reasonably establish one of these quantities as the focus of theoretical interest an prouce only one marginal effect plot. Inee, of the 44 papers we ientifie that present marginal effect plots to evaluate a theory positing interaction, 39 (89%) present only a single plot, showing how the estimate effect of one variable varies with the other. We accept as a funamental principle that scholars estimating a statistical moel shoul use the estimation results to assess as many of the theory s implications as possible. We show that for those testing theories positing interaction between two inepenent variables, this often means eriving an testing preictions about how the marginal effect of each inepenent variable varies with the value of the other not all of which can be evaluate by inspecting a single marginal effect plot. It is important to note that we are not suggesting that researchers testing a hypothesis about how the marginal effect of X varies with Z shoul manufacture a secon hypothesis about how the marginal effect of Z varies with X when their theory generates no preictions beyon those alreay incorporate in the first hypothesis. We are simply observing that many conitional theories propose by political scientists generate more preictions than can be teste with a single marginal effect plot, an that in this situation, when the researcher limits consieration to a single plot, she subjects her theory to a weaker test than is possible given the ata available. In the next section, we consier the implications of the inherent symmetry of interactive moels for theory testing in more etail. In particular, we emonstrate why it can be angerous for a researcher with a conitional theory to limit consieration to preictions that can be evaluate with a single marginal effect plot. The basic insight is that any observe relationship between Z an the marginal effect of X is always consistent with a wie variety of ways in which the marginal effect of Z varies with X, someofwhichmaybeinconsistent with the unerlying conitional theory. This means that proposing a hypothesis about how the effect of X varies with Z an assessing it by examining just a marginal effect plot for X often constitutes a weak test of the conitional theory unerlying the hypothesis. Supplementing this hypothesis with a secon one about how the effect of Z varies with X that can be evaluate by inspecting a marginal effect plot for Z can ramatically narrow the range of relationships that are consistent with one s unerlying theory, thereby strengthening the empirical test. We then turn to practical avice on eriving an testing hypotheses from conitional theories. In particular, we iscuss issues that arise when evaluating empirical evience in favor of, or against, conitional theories by examining several prototypical sets of results one might get when estimating an interactive moel. Many of these issues have not been aequately aresse in the existing literature, leaving some reaers uncertain as to how to evaluate the level of empirical support for conitional theories. Next, we illustrate our central points by replicating two of the numerous publishe stuies that seek to test a conitional theory but that present a marginal effect plot for only one of the two variables preicte to interact. In one replication, constructing a secon marginal effect plot reveals aitional support for the researcher s theory. In the other, a secon plot reveals evience contrary to the analyst s theory. Throughout the article, we offer avice on how to maximize the information portraye in marginal effect plots, an before concluing, we summarize our recommenations.

3 improving tests of interactive theories 3 Implications of the Symmetry of Interaction for Theory Testing Suppose we have a conitional theory in which X an Z interact in influencing some continuous epenent variable, Y, such that the effects of X an Z can be capture with the following linear-interactive moel: 5 FIGURE 1 A Plot of the Marginal Effect of X on Y against Z when b X an b XZ in equation (1) are Positive Y ¼ b 0 þ b X X þ b Z Z þ b XZ XZþ e: ð1þ This moel involving a single prouct, or multiplicative, term is the most common specification of interaction in political science. In this moel, the marginal effect of is ¼ b X þ b XZ Z: ð2þ As equation (2) clearly inicates, unless the coefficient for the prouct term, b XZ, is zero, the marginal effect of X is conitional on the value of Z. 6 To emphasize this conitionality in what follows, we enote the marginal effect of X as ME(X Z). In turn, we let ME(X Z 5 z) enote the marginal effect of X on Y when Z equals the specific value z. The marginal effect plot for X in Figure 1 epicts the relationship between ME(X Z) an Z in equation (2) when b X an b XZ are both positive. The plot illustrates that (1) when Z 5 0, the marginal effect of X on Y is b X, an (2) ue to the constant slope, b XZ, of equation (2), the marginal effect of X changes by b XZ for every unit increase in Z. Note that the interactive moel specifie in equation (1) is symmetric in X an Z. In other wors, the fact that the marginal effect of X on Y is conitional on Z logically guarantees that the marginal effect of Z on Y must be conitional on X. Inee, the marginal effect of Z is given by: ¼ b Z þ b XZ X: ð3þ 5 For simplicity, we assume that there are no other covariates in the moel. However, all claims in this article hol with any number of aitional variables as long as none of them interacts with either X or Z. Although we focus on moels with continuous epenent variables, our avice is equally applicable to moels with limite epenent variables such as logit an probit. 6 Note that the expression in equation (2) implies that the marginal effect of X on Y is conitional on the value of Z but not on the value of X. This property stems from the linear functional form for the interaction specifie in equation (1). In interactive moels with a nonlinear functional form, in contrast, the marginal effect of X on Y necessarily varies with both the value of X an the value of Z (Brambor, Clark, an Goler 2006, 77). This implies that the marginal effect of Z is b Z when X is zero an changes by b XZ for every unit increase in X. Thus, it is evient that the coefficient on the prouct term, b XZ, inicates both the slope of the relationship between ME(X Z) an Zanthe slope of the relationship between ME(Z X) an X. As such,we must recognize that Z conitions the effect of X on Y,anX conitions the effect of Z on Y. It is this inherent symmetry that makes it misleaing for scholars to esignate X or Z as the conitioning variable an the other variable as the one being conitione. 7 We recognize that in some settings it can be very tempting to conceive of one variable as the conitioning variable. For example, when one variable, X, is continuous an the other, Z, is ichotomous, it seems natural to think in terms of the effect of X on Y being ifferent in one context (Z 5 0) than in another (Z 5 1), thereby establishing Z as the conitioning variable. However, the fact remains that the effect of the binary variable Z also varies with X. Although the inherent symmetry of interactions is well-ocumente (Brambor, Clark, an Goler 2006; Kam an Franzese 2007), the implications of such symmetry for theory testing have been largely overlooke. Recall that the prouct term coefficient, b XZ, in equation (1) inicates both the slope of the relationship between ME(X Z) an Z an the slope of the relationship between ME(Z X) an X. This implies that if a researcher with a conitional theory presents 7 We shoul note that the inherent symmetry of interactions is not the result of the particular linear-interactive specification that we use in equation (1); all interactive specifications are symmetric (Kam an Franzese 2007, 16).

4 4 william. berry et al. a clearly state proposition about how the marginal effect of X on Y varies with Z, then she is also implicitly introucing a hypothesis about how the marginal effect of Z on Y varies with X. Thus, on the surface, it may seem unnecessary even reunant for the researcher to explicitly state an aitional hypothesis about ME(Z X). However, this intuition is incorrect. Equation (2) or equivalently, a marginal effect plot for X completely characterizes how the marginal effect of X on Y varies with Z in the linear-interactive moel of equation (1). Similarly, equation (3) or a marginal effect plot for Z fully elineates how the marginal effect of Z on Y varies with X. Note, however, that although equations (2) an (3) share a common slope, b XZ, equation (2) or its epiction as a marginal effect plot for X provies no information about the value of the intercept, b Z,inequation(3).Hence,a marginal effect plot for X oes not establish the sign (positive or negative) or the magnitue of the marginal effect of Z at any value of X. This is critically important because ifferent values for this intercept imply quite ifferent ways in which the marginal effect of Z is conitional on X. It may be the case that only some of these ways are consistent with the researcher s unerlying conitional theory. To illustrate, suppose that one has a conitional theory in which X an Z interact to influence Y. In particular, the theory preicts that the marginal effect of X is always positive an that the magnitue of this positive effect increases with Z. In other wors, both b X an b XZ in equation (1) are expecte to be positive. The marginal effect plot in Figure 1 is consistent with these theoretical claims. But what exactly oes the fact that b X an b XZ are positive tell us about the marginal effect of Z on Y? All we can infer from this information is that the plot of ME(Z X) will have the same positive slope as the plot of ME(X Z). However, a wie variety of conitional relationships among X, Z, an Y are still possible even after this slope is establishe. To see this, suppose that the plot of ME(X Z) in Figure 1 has an intercept, b X, of 0.10 an a slope, b XZ, of If we assume arbitrarily that the values of both X an Z range from 0 to 100 in the population of interest, then this plot implies a conitional relationship in which the marginal effect of X is 0.10 when Z is at its lowest value an (100)(0.004) when Z is at its highest value. In Figure 2, we epict three quite ifferent conitional relationships among X, Z, any that are all consistent with this marginal effect plot for X where b X an b XZ On the left of Figure 2 are three-imensional (3-D) plots of Y against X an Z. These plots permit one to visualize how the two inepenent variables jointly influence Y. To the right of each 3-D plot is the associate plot of ME(Z X) againstx. Akeyfeatureto note about these marginal effect plots is that although they share the same slope, 0.004, the value of the intercept, b Z, is ifferent in each. In Figure 2a, b Z is 0.20, inicating that the marginal effect of Z is 0.20 when X 5 0. The fact that b XZ is positive means that the marginal effect of Z on Y is always positive but that this positive effect strengthens as X increases, reaching 0.60 when X achieves its maximum value of 100. This is reflecte in the 3-D plot by the slope of Y against Z being positive both in the left rear vertical plane (i.e., when X 5 100) an the right front vertical plane (i.e., when X 5 0), but the slope being more steeply positive in the rear. In Figure 2b, the intercept, b Z, is sufficiently negative (-0.60) that the marginal effect of Z remains negative at all values of X espite the positive value for b XZ. In this scenario, the negative effect of Z eclines in strength with increases in X, reaching when X obtains its maximum value. This is mirrore in the corresponing 3-D plot by the slope of Y against Z being negative both in the left rear plane (i.e., when X 5 100) an the right front plane (i.e., when X 5 0), but the slope being more steeply negative in the front. Figure 2c is similar to Figure 2b in that the intercept, b Z, is negative (-0.20). However, its negative value is sufficiently small in magnitue that the marginal effect of Z eventually becomes positive once X is large enough. In this scenario, the marginal effect of Z is when X 5 0. Z s negative effect ecreases in magnitue as X increases until ME(Z X) reacheszero when X As X increases past 50, the marginal effect of Z becomes positive an grows in strength, reaching 0.20 when X In the associate 3-D plot, note that the slope of Y against Z is negative in the right front plane (i.e., when X 5 0) but positive in the left rear plane (i.e., when X 5 100). Figure 2 illustrates quite ramatically how a single marginal effect plot for X can be consistent with very ifferent conitional relationships among X, Z, any. Assume one s theory preicts that the relationship among X, Z, any shoul be like the one epicte in Figure 2a. It is ifficult to imagine someone with this theory claiming empirical support if the estimate relationship looks like that shown in either Figure 2b or Figure 2c. The plots shown in Figure 2b an Figure 2c epict funamentally ifferent processes by which Y is jointly etermine by X an Z. For example, in

5 improving tests of interactive theories 5 FIGURE 2 Three Conitional Relationships Among X, Z, an Y Consistent with the Plot of ME(X Z) in Figure 1 (Assuming b X an b XZ ) Figure 2a, Y is maximize when X an Z are both at their maximum, an Y is minimize when X an Z are both at their minimum. In Figure 2c, Y is also greatest when X an Z are both at their maximum, but Y is smallest when X is minimize while Z is maximize. In Figure 2b, Y is largest when X is maximize an Z is minimize, an Y is smallest when Z is at its maximum an X is at its minimum. Yet if one limite the empirical evience examine to an estimate plot of ME(X Z) showing a positive intercept an a positive slope, as in Figure 1, one might claim support for one s conitional theory,

6 6 william. berry et al. ignorant of the inconsistent evience that woul be apparent from an inspection of a plot of ME(Z X). Thus, even when there is strong empirical support for a hypothesis about how the marginal effect of X on Y varies with Z base on an estimate plot of ME(X Z), a failure to use one s conitional theory to erive an aitional hypothesis about how the marginal effect of Z varies with X (beyon a preiction about the value of b XZ ) an inspect a marginal effect plot for Z may mask either (1) aitional evience in support of the theory, or more worryingly, (2) evience inconsistent with the theory. It is important to recognize that once one constructs a theory positing interaction between X an Z in influencing Y specific enough to establish the signs of the intercept an slope of a plot for ME(X Z), one nee not eman a great eal more of the theory to generate aitional preictions about ME(Z X) that woul permit a stronger test of the theory. For example, assume once more that one s theory preicts a plot of ME(X Z) taking the form of Figure 1, with both a positive intercept an a positive slope. We have seen that, by itself, this preiction is consistent with all three plots of ME(Z X) in Figure 2. But if thetheoryweretopreictaitionallythatz has a positive effect on Y when X is at, say, its highest (or, in fact, any) value, then this woul imply that Figure 2b for which ME(Z X) is negative throughout is inconsistent with the theory. If, in contrast, the theory were to preict that Z has a positive effect on Y when X is at its lowest value, then both Figures 2b an 2c woul be eliminate as possibilities. In both of these cases, supplementing an estimate plot of ME(X Z) with one of ME(Z X) woul allow for a stronger test of the unerlying conitional theory. Deriving an Testing as Many Preictions as a Conitional Theory Allows We now offer some practical avice on eriving an testing hypotheses from conitional theories that can be accurately specifie with the linear-interactive moel of equation (1). Five Key Preictions Ieally, a theory positing interaction between X an Z in influencing Y woul be strong enough to preict the precise magnitue of the effect of each of X an Z at every possible value of the other variable. Of course, theories in political science are very rarely strong enough to generate such specific preictions. However, we believe that conitional theories in the literature are typically strong enough to generate five basic preictions about the marginal effects of X an Z on Y: 8 1. P X jzmin : The marginal effect of X is [positive, negative, zero] when Z is at its lowest value. 2. P X jzmax : The marginal effect of X is [positive, negative, zero] when Z is at its highest value. 3. P ZjXmin : The marginal effect of Z is [positive, negative, zero] when X is at its lowest value. 4. P ZjXmax : The marginal effect of Z is [positive, negative, zero] when X is at its highest value P XZ : The marginal effect of each of X an Z is [positively, negatively] relate to the other variable. Note that by calling for researchers to state preictions about what happens when X an Z are at their lowest an highest values, we o not imply that analysts shoul necessarily focus greatest attention on estimate marginal effects at these extreme values. Inee, as we note below, when there are few observations at these extremes, estimates of marginal effects at these values are less relevant for testing the theory than estimates of marginal effects at more central values for X an Z. Rather, we call for preictions at the extremes simply because if one assumes linearity as in equation (1), or at least monotonicity, such preictions automatically imply preictions at values between the extremes. 10 The preictions outline above nee not be presente as five separate hypotheses. Inee, with careful 8 These preictions are base on the case in which an author s conitional theory conforms to a moel of the form shown in equation (1). More complex conitional theories woul prouce testable preictions of a ifferent form an require an alternative moel specification. 9 Two issues regaring these preictions are worth noting. First, when Z is ichotomous, the preictions P ZjXmin an P ZjXmax shoul be state in terms of the response of Y to a iscrete change in Z rather than in terms of the marginal effect of Z. Thisisbecausethe concept of a marginal effect makes sense only when it is possible to conceive of an infinitesimally small change in Z. The preictions P X jzmin an P XjZmax shoul be state similarly when X is ichotomous. Secon, when any of these preictions points to a zero effect, in which one inepenent variable has no effect at an extreme value of the other, scholars nee to think very carefully about whether the functional form of equation (1) properly specifies the expecte nature of the interaction (see the appenix). 10 Of course, when one inepenent variable say X is ichotomous, the highest an lowest values of X are the only two possible values for X, an thus the preictions P ZjXmin an P ZjXmax together escribe the marginal effect of Z at all possible values of X.

7 improving tests of interactive theories 7 phrasing, all five preictions can be subsume in a single hypothesis about how the marginal effect of X varies with Z an a single hypothesis about how the marginal effect of Z varies with X. This is illustrate in the following pair of hypotheses: H X Z : The marginal effect of X on Y is positive at all values of Z; this effect is strongest when Z is at its lowest an eclines in magnitue as Z increases. HZ X : The marginal effect of Z on Y is positive when X is at its lowest level. This effect eclines in magnitue as X increases; at some value of X, Z has no effect on Y. AsX rises further, the effect of Z becomes negative an strengthens in magnitue as X increases. Note that H X Z implies that the marginal effect of X is positive at both the lowest an highest values of Z, thereby offering preictions P X jzmin an P X jzmax.h Z X states that the marginal effect of Z is positive at X s lowest value an negative at X s highest value, thereby offering preictions P ZjXmin an P ZjXmax.Thereisno nee to state a separate hypothesis that each inepenent variable is negatively relate with the marginal effect of the other because such a preiction that of P XZ is implicit in both H X Z an H Z X. Thus, in combination, H X Z an H Z X inclue all five preictions we recommen an offer as complete a escription of the expecte interaction between X an Z as one coul offer for a linear-interactive moel without preicting specific magnitues for marginal effects at specific values of the inepenent variables. In general, scholars who propose a theory shoul seek to test as many of the theory s implications as possible. When it comes to interactive theories that can be accurately specifie by the linear moel of equation (1), this requires making, an then testing, as many of the five preictions liste above as possible. Later, we illustrate this recommenation by revisiting two recent stuies estimating an interactive moel one in comparative politics an one in international relations an consiering whether each utilizes the moel s coefficient estimates to test all of the preictions that the author s theory generates. Before we o this, though, we briefly iscuss several issues that arise when evaluating empirical evience in favor of, or against, conitional theories. Some Prototypical Results When Testing Interactive Moels Suppose we want to evaluate the empirical support for the conitional theory from which hypotheses H X Z an H Z X in the previous section are erive following the avice we have offere. We woul estimate equation (1) an then use the moel s coefficients to construct marginal effect plots for both X an Z. Clearly, the evience in favor of the theory woul be greatest in the case where we fin strong support for each of the five preictions mae by hypotheses H X Z an H Z X. This woul involve fining that the point estimates for ME(X Z 5 z min ), ME(X Z 5 z max ), an ME(Z X 5 x min ) are all positive, statistically significant, an substantively significant an that the point estimates for ME(Z X 5 x max ) an b XZ are both negative, statistically significant, an substantively significant [where min an max refer to the minimum an maximum observe values of a variable in the sample]. Below, when we use the term significant without any qualification, it is meant to imply that both statistical an substantive significance have been establishe. 11 But shoul we require that all of these conitions be met before we claim any empirical support for our conitional theory an reject our theory if any of the conitions is not achieve? Ultimately, we believe that this is an unrealistically strong stanar for empirical evience an that it woul be a mistake to treat all situations in which at least one of these conitions fails to be met as equivalent. Although firm knowlege that one of the five preictions from earlier is false woul be sufficient logical grouns for concluing that the unerlying theory is false, it is important to remember that statistical tests cannot tell us with certainty whether any of the preictions is false; all they offer is information about the risks of a false inference if one rejects the null hypothesis that a quantity of interest equals a particular value, usually zero. For this reason, it is inappropriate to establish har an fast rules about what combinations of evience regaring the five preictions constitute support for the unerlying conitional theory. Nevertheless, we can examine several prototypical sets of results one might get when estimating an interactive moel taking the form of equation (1), 11 Unless we explicitly state to the contrary, statistically significant in this article implies significantly ifferent from zero at some specifie significance (a) level. When we say that a point estimate is substantively significant, we mean that its value is large enough to be eeme of nontrivial magnitue. We recognize that the minimum magnitue require for substantive significance is subjective an that there is no single correct way of establishing substantive significance. In our replication of a stuy by Alexseev (2006) later in the article, we illustrate one potentially useful strategy for emonstrating the substantive significance of interactive relationships. For more on the important ifference between statistical an substantive significance, see Achen (1982, 41 51).

8 8 william. berry et al. an for each, assess the extent to which we woul feel comfortable claiming support for the unerlying conitional theory given the empirical evience presente. To groun the iscussion, assume we seek to test the theory generating hypotheses H X Z an H Z X. A strong test woul require that we use the moel s coefficient estimates to evaluate all five of the preictions containe in H X Z an H Z X. However, for illustrative purposes, we simplify matters in the iscussion that follows by focusing on hypothesis H X Z an the three preictions that it contains: (1) ME(X Z 5 z min ). 0, (2) ME(X Z 5 z max ). 0, an (3) b XZ, 0. Six ifferent prototypical sets of results are portraye in Figure 3 in the form of a marginal effect plot for X. The ashe curves aroun the marginal effect line epict a 95% confience interval, thereby ientifying the values of Z at which the marginal effect of X is statistically significant. However, since we want the plots to convey information about substantive significance as well, we ientify the values of Z at which the marginal effect of X on Y is significant (i.e., both statistically an substantively significant) by making the horizontal axis bol at these values. 12 Uner each plot, we also inicate whether the coefficient on the prouct term, b XZ,is significant. This last piece of information is not usually inclue in publishe marginal effect plots but is critical for etermining whether there is empirical evience of interaction between X an Z, i.e., for testing preiction P XZ. 13 This is because, as equation (4) remins us, b XZ inicates the strength of 12 We aopt this bol axis convention here because it is useful for portraying an iscussing hypothetical results about generic X, Y, an Z variables. We o not recommen that researchers aopt this convention when reporting actual research results. 13 Note that whether this is true for moels with limite epenent variables like logit an probit epens on the epenent variable of conceptual interest. There are two possible epenent variables of conceptual interest when estimating a binary logit or probit moel: (1) an unboune latent variable, Y*, assume to be measure by the observe ichotomous variable, Y, an (2) the probability that Y equals one, Pr (Y 5 1). When one s epenent variable of interest is the unboune Y*, then the prouct term coefficient, b XZ, reflects the extent of interaction. However, this is not the case when the epenent variable of interest is Pr (Y 5 1). Inee, when the epenent variable is Pr (Y 5 1), one cannot etermine whether there is interaction between X an Z by inspecting the coefficient on the prouct term (or any single term). The fact that the marginal effect of each of X an Z on Pr (Y 5 1) is not linearly relate to the other variable means that preiction P XZ must be evaluate by estimating the marginal effects of X an Z at ifferent values for the inepenent variables an assessing how they change as the values of the inepenent variables change (Ai an Norton 2003; Berry, DeMeritt, an Esarey 2010; Norton, Wang, an Ai 2004). the relationship between both (1) ME(X Z) an Z, an (2) ME(Z X) @Z@X ¼ b XZ: ð4þ To facilitate reaers seeing as much statistical evience relevant for testing a theory positing interaction as possible, we recommen that scholars routinely report the estimate prouct term coefficient an a t-ratio or stanar error for this coefficient in their marginal effect plots. Consier first the plot in Figure 3a. The marginal effect of X is positive an significant across the observe range of Z, anb XZ is negative an significant. This plot provies unambiguously strong evience for hypothesis H X Z because each of its three preictions receives strong empirical support. Next, consier the plot shown in Figure 3b. The only ifference here is that the marginal effect of X is no longer significant when Z is at its highest value. However, because H X Z preicts that the marginal effect of X on Y eclines in magnitue as Z increases, which leaves open the possibility of a weak effect by the time Z gets large, we are not particularly trouble to fin that ME(X Z 5 z max )fails to be significant. Thus, in this situation, we woul conclue that there is strong support for H X Z even though the value for ME(X Z 5 z max ) is not significant. 14 The plot shown in Figure 3c provies a more ambiguous case. As before, the significant coefficient on the prouct term represents clear evience that the marginal effect of X is conitional on Z as preicte. The ifference is that the range of values for Z for which the marginal effect of X is positive an significant is now smaller than in Figure 3b, an the point estimate for ME(X Z 5 z max ) is actually negative. In this scenario, we are not terribly concerne that the point estimate for ME(X Z 5 z max ) takes the wrong sign because this value is statistically an substantively insignificant. We woul argue that how supportive these results are of hypothesis H X Z epens on the percentage of observations having values of Z at which the marginal effect of X is positive an significant, i.e., for which Z, z in the figure. The higher this percentage, the more 14 Ieally, the theory unerlying H X Z woul be strong enough to generate a preiction about whether the marginal effect of X on Y shoul (1) remain strong even when Z reaches its maximum, or (2) ecline to near zero when Z is maximize. In the former case, the theory woul preict that X has a significant effect on Y when Z 5 z max. But in the latter case, it woul preict an insignificant effect when Z 5 z max. Amittely, theories in political science are rarely capable of yieling such a fine istinction.

9 improving tests of interactive theories 9 FIGURE 3 Plots of ME(X Z) Reflecting Several Prototypical Sets of Empirical Results incline we woul be to accept the empirical evience as supportive. Of course, the minimum percentage high enough to justify a claim of support is subjective. As a result, we recommen that scholars report the percentage of observations that fall within the region of significance. Inee, it woul be very helpful if researchers woul provie a frequency istribution for the variable plotte on the horizontal axis so that reaers can assess for themselves the relative ensity of observations across the range of X. We illustrate how such a frequency istribution might be incorporate into a marginal effect plot when we report the results of two replications in the next section. When it comes to evaluating conitional theories, one practice that we strongly avise against is getting into a counting game in which one s conclusion is base strictly on the number of preictions for which there is statistical support. For example, consier the plot shown in Figure 3. This plot provies statistical confirmation for two of the three preictions containe in H X Z, namely that ME(X Z 5 z min ) is positive an that b XZ is negative. The fact that b XZ

10 10 william. berry et al. is significant provies strong empirical evience of interaction between X an Z. Importantly, though, the plot suggests that this interaction takes an appreciably ifferent form than that preicte by hypothesis H X Z.AlthoughX has the expecte positive effect when Z is low, X has a significant negative effect when Z gets large. We believe that scholars shoul not sweep this kin of inconsistency with the hypothesis uner the rug by claiming a healthy batting average of 0.667, with two of the three preictions confirme. Evience that when Z is high, increases in X yiel substantial ecreases in Y rather than the preicte increases strikes us as sufficient to raise serious concerns about the conitional theory unerlying the hypothesis. Figure 3e illustrates a more extreme case in which claiming support for H X Z base on two of the three preictions receiving statistical support woul be unwarrante. In this case, the marginal effect of X is positive an significant across the entire observe range of Z, thereby inicating support for the preictions that ME(X Z 5 z min ) an ME(X Z 5 z max ) are positive. However, although b XZ is negative as preicte, it lacks statistical significance an the nearly flat marginal effect line inicates that the magnitue of b XZ is substantively trivial. In essence, there is no evience of appreciable interaction between X an Z. Inee, this sort of plot with a marginal effect line slope slightly upwar or ownwar is exactly what we woul expect to fin if we were to estimate equation (1) when each of X an Z has a strong positive effect on Y but their effects are aitive rather than interactive. Thus, the evience in Figure 3e seriously challenges the theory preicting that X an Z interact in influencing Y. We now consier a final set of prototypical results shown in Figure 3f. Once again, the line plotte is intene to be nearly flat. The effect of X on Y is substantively insignificant at all values of Z, but statistically significant when Z, z. The fact that the marginal effect of X changes from statistically significant when Z, z to statistically insignificant when Z $ z might seem to suggest that there is interaction between X an Z. Inee, BCG (2006, 74) imply precisely this when they claim that a situation in which the marginal effect of X on Y is statistically significant for some values of Z but not for others might be interprete as a sign of meaningful interaction even when the coefficient on the prouct term is statistically insignificant. However, this is incorrect. The nearly flat line in Figure 3f represents a case in which the marginal effect of X has a t-ratio barely above the threshol for statistical significance when Z is low an a t-ratio barely below the threshol when Z is high. If one capitalizes on the fact that ME(X Z) changes from statistically significant to not as Z surpasses z to claim evience of interaction, one is placing too much reliance on an arbitrarily chosen level of statistical significance. If this level were set slightly higher, ME(X Z) woul be statistically significant over the entire range for Z. If the level were set slightly lower, the marginal effect woul not be statistically significant at any value of Z. The more relevant information is that the coefficient on the prouct term, b XZ, is not statistically significant an is of small magnitue. As we showe in equation (4), this inicates that the marginal effect of X varies only trivially with Z, an on this basis we shoul reject the theory positing interaction unerlying H X Z. Two Replications We now illustrate our central points by replicating two stuies chosen from the many that test a conitional theory but that present a marginal effect plot for just one of the two variables hypothesize to interact. In one replication, examining the secon marginal effect plot lens aitional support for the researcher s theory. In the other, the secon plot provies evience that contraicts the author s theory. Revealing Aitional Evience in Favor of the Theory Being Teste Kastner (2007) examines how conflicting interests an the strength of omestic actors with internationalist economic interests affect the level of trae between countries. Previous stuies inicate that bilateral trae tens to be lower when countries have conflicting political interests. As Kastner notes, though, there is consierable variation across country yas in the extent to which conflicting interests lea to reuce bilateral trae. His explanation for this variation centers on the strength of omestic actors who benefit from trae. Specifically, Kastner argues that although leaers generally want to reuce trae with countries that o not share their interests, some leaers are constraine in their ability to o this by the presence of strong omestic actors with internationalist economic interests. As Kastner puts it, the negative effects of conflict on commerce shoul be less severe when internationalist economic interests have strong political clout omestically (2007, 670). Unable to measure the strength of internationalist interests in a ya irectly, Kastner uses the extent of trae barriers in the countries (Trae Barriers) as a proxy variable that is inversely relate to the strength

11 improving tests of interactive theories 11 of these interests. If we enote the extent of conflict between two countries by Conflict an their level of bilateral trae by Trae, Kastner s hypothesis can be state as follows: H Conflict Barriers : The marginal effect of Conflict on Trae is negative at all values of Trae Barriers; this negative effect is weakest when Trae Barriers is at its lowest level an strengthens in magnitue as Trae Barriers increases. Kastner tests his conitional theory using annual ata from 76 countries from 1960 to 1992 an an OLS moel with an interactive specification taking the form of equation (1): Trae ¼ b 0 þ b C Conflict þ b B Trae Barriers þ b CB ðconflict 3 Trae BarriersÞ þ bcontrols þ e; ð5þ where Controls is a vector of control variables. The coefficient on the prouct term, Conflict 3 Trae Barriers, is negative an statistically significant at the 0.01 level, with a t-statistic of Using the parameter estimates from his moel (Table 1, Moel 1, 676), Kastner prouces a plot showing how the marginal effect of Conflict on Trae varies with the level of Trae Barriers. We reprouce this marginal effect plot in a slightly moifie form in Figure 4a. 15 Base on the plot,as well as the statistically significant negative coefficient on the prouct term, Kastner claims empirical support for his theory. We avise researchers who propose a theory positing interaction between two variables, X an Z, to use the theory to generate as many of the five key preictions liste earlier as the theory allows regaring the marginal effects of X an Z on Y. Kastner s 15 We were able to replicate Kastner s results perfectly. Our marginal-effect plot iffers from his (Figure 1, 677) in four respects. Rather than plotting the marginal effect of Conflict on Trae on the vertical axis, as we o, Kastner plots the change in Trae as Conflict increases from its 15 th percentile in the sample to its 85 th percentile. Given the linear form of equation (5), this ifference in scaling the vertical axis is superficial because one scaling is a linear transformation of the other. Secon, Kastner plots percentiles for Trae Barriers in the sample along the horizontal axis. We saw no goo reason to istort the scale for Trae Barriers by using percentiles rather than the actual values. This ifference in scaling for the horizontal axis explains why our plot is linear, but Kastner s is not. Thir, we plot the marginal effect of Conflict on Trae over the entire range of values for Trae Barriers in the estimation sample, whereas Kastner plots it only over the values for Trae Barriers that fall between the 20 th an 80 th percentiles. Fourth, we have ae a shae rectangle to our plot; we explain the purpose of this below. hypothesis, H Conflict Barriers, offers three of these preictions: P CjBmin : The marginal effect of Conflict on Trae is negative when Trae Barriers is at its lowest value. P CjBmax : The marginal effect of Conflict on Trae is negative when Trae Barriers is at its highest value. PCB : The marginal effect of each of Conflict an Trae Barriers is negatively relate to the other variable. However, Kastner s hypothesis is silent about the expecte value (positive, negative, or zero) of the marginal effect of Trae Barriers at the highest an lowest values of Conflict. Before we consier the marginal effect of Trae Barriers on Trae, we reevaluate the empirical support for preictions P CjBmin,P CjBmax,anP CB. Assuming that the statistically significant negative coefficient for the prouct term in equation (5) is also substantively significant, there is unambiguous support for preiction P CB. 16 In other wors, there is clear evience that the marginal effect of Conflict on Trae is negatively relate to the value of Trae Barriers, as Kastner hypothesizes, an (ue to the symmetry of interactions) that the marginal effect of Trae Barriers is negatively relate to the value of Conflict. But is this conitionality consistent with preictions P CjBmin an P CjBmax? On the one han, Figure 4a shows that the marginal effect of Conflict is negative an statistically significant when Trae Barriers takes on its largest observe value, thereby supporting preiction P CjBmax. On the other han, preiction P CjBmin fails to receive empirical support. Contrary to expectation, the marginal effect of Conflict is positive an statistically significant when Trae Barriers is at its smallest observe value, an inee, at all values less than Overall, the marginal effect plot for Conflict closely resembles the prototypical plot shown in Figure 3. This raises concerns about the conitional theory unerlying the hypothesis being teste because the estimate marginal effect is statistically significant in the wrong irection at one en of the horizontal axis. Kastner offers no explanation for why an increase in conflict shoul lea to increase bilateral trae when omestic actors with internationalist economic interests are strong, i.e., when trae 16 Kastner oes not explicitly evaluate the substantive significance of the estimate effects he reports. Rather than unertake our own assessment, for our illustration we simply assume that statistical significance implies significance (i.e., both statistical an substantive significance).

12 12 william. berry et al. FIGURE 4 Marginal Effect Plots Designe to Evaluate the Conitional Theory Presente by Kastner (2007) Our replication of Kastner s analysis illustrates the importance of constructing a marginal effect plot that shows how the effect of X on Y varies over the entire observe range of Z. Kastner plots the marginal effect of Conflict only for values of Trae Barriers between the 20 th an 80 th percentiles; this interval is inicate by the shae rectangle in Figure 4a. Note that in this restricte range for Trae Barriers, the estimate marginal effect of Conflict on Trae, although positive for low values of Trae Barriers, is never positive an statistically significant. Thus, although the full marginal effect plot reveals values for Trae Barriers at which there is clear evience of an unexpecte positive effect of Conflict on Trae, the restricte plot masks the existence of these values an makes it appear as if the estimate positive effect of Conflict never achieves statistical significance even at the lowest values for Trae Barriers. Inee, Kastner s restricte plot more closely parallels the prototypical plot shown in Figure 3c, which we argue earlier potentially offers support for the hypothesis being teste epening on the percentage of sample observations falling into the region of significance. Consier the results in Figure 4a more closely. The marginal effect of Conflict is negative an statistically significant when Trae Barriers excees Superimpose over the marginal effect plot is a histogram portraying the frequency istribution for Trae Barriers; the scale for the istribution is given by the vertical axis on the right-han sie of the graph. The histogram shows that 55.4% of the country yas in Kastner s sample fall into this range of statistical significance. At the other extreme, the effect of Conflict is positive an statistically significant when Trae Barriers is less than Of the sample observations, 14.5% lie in this range. 17 Although these latter observations, which are inconsistent with barriers are low. In our view, the statistically significant positive effect of Conflict when Trae Barriers is less than 3.16 shoul not be ismisse as a trivial inconsistency; rather, it is an important piece of evience to consier alongsie the support for preictions P CjBmax an P CB when evaluating Kastner s theory. 17 The fact that there are few observations at low levels of Trae Barriers means that the evience that Conflict has a statistically significant positive effect on Trae when Trae Barriers is low may rest heavily on the moel s linearity assumption. (Note that reaers woul be completely unaware of this issue in the absence of a histogram showing the earth of sample observations with low values of Trae Barriers. This highlights the importance of incluing in a marginal effect plot information about the istribution of the variable epicte on the horizontal axis.) Unless one believes that there is a strong a priori theoretical justification for the linearity assumption, one shoul be skeptical about rawing strong inferences concerning the marginal effect of Conflict at low levels of Trae Barriers without subjecting this assumption to empirical scrutiny. In the online appenix at we o precisely this by estimating quaratic an cubic versions of Kastner s moel, thereby relaxing the linearity assumption. The evience that Conflict has a statistically significant positive effect on Trae when Trae Barriers is low is robust to these alternative specifications.