Untangling Selection Effects in Studies of Coercion

Transcription

1 Untangling Selection Effects in Studies of Coercion Eugene Gholz Assistant Professor Lyndon B. Johnson School of Public Affairs University of Texas Daryl Press Associate Professor Department of Political Science University of Pennsylvania Abstract: For more than a decade, scholars have recognized that studies of coercion are plagued by selection effects. Analyses that fail to account for the strategic decisions that lead countries to initiate (or not) crises will yield biased results. Unfortunately, recent attempts to improve research design to account for selection effects are flawed. We use a formal model to demonstrate that selection effects create complex, non-monotonic relationships between key parameters (e.g., defender interests and power) and observable crisis outcomes. Whether the real connections between these variables and deterrence are positive, negative, or non-monotonic, scholars will observe complex non-monotonic relationships in datasets of crisis dynamics, greatly complicating empirical analyses of coercion. We describe a better set of research approaches both quantitative and qualitative that scholars can use to mitigate the problems of selection effects as they study coercion. We provide two short case studies to illustrate how the recommendations for qualitative research can be carried out.

2 For years scholars of international politics have labored to test theories of coercion. The main body of this research uses data on crisis outcomes to determine whether any of a host of variables such as formal alliances, domestic political institutions, public statements by leaders, military force balances affect the odds of successful deterrence or compellence. 1 The goal is to identify factors that affect the likelihood of wars (that is, to understand deterrence), to discover what makes countries accede to adversaries' demands (that is, to understand compellence), and to understand when countries issue explicit threats or resort to brinkmanship to pursue their foreign policy goals (that is, to understand crisis initiation). Selection effects greatly complicate efforts to test theories of coercion, because they make the easily observable cases of crisis interaction a non-representative sample of countries. The problem arises because many potential challengers are deterred before they initiate a crisis, and many defenders cede the issue at hand rather than take a stand they are likely to abandon later. Datasets of crises, in which by definition both challengers and defenders have forgone opportunities to concede, over-represent the most motivated challengers and defenders. 2 The implication for international relations scholarship is profound: the statistical relationships that are observable in the data on crisis outcomes may differ significantly from the actual relationships that exist between key variables and successful coercion. 3 In this paper we demonstrate that several apparently plausible research strategies for studying coercion despite the selection effects do not work. James Fearon proposes reversing the sign of the expected empirical relationships; Paul Huth and Todd Allee 1 For a useful review, see Huth Fearon Smith DRAFT DO NOT CITE 1

3 recommend using an off-the-shelf selection estimator like the Heckman probit. 4 We use a formal model of crisis dynamics to show the selection effect more precisely than previous work. The selection effect is considerably more complex than scholars have assumed: under some conditions, selection effects strongly influence the types of countries that appear in observable datasets of crisis interactions, but under other conditions, the selection effect is relatively weak. Our model reveals how changes in parameter values affect both the real rate of successful coercion and the rate that scholars can observe in datasets of crisis outcomes. Overall, the selection effect should lead scholars to observe a non-monotonic relationship between key independent variables and deterrence success, whether the underlying real relationship is positive, negative, or non-monotonic. The results of our model should not be confused with previous work on nonmonotonic relationships in crisis dynamics. 5 For example, according to some models of crisis dynamics, the relationship between the balance of power and the probability of war should be non-monotonic, because the identity of challengers and defenders is endogenous. As weak defenders become more powerful, war becomes less likely until the defender becomes so strong that it might actually choose to start a war, in essence becoming a challenger. Models that incorporate this dynamic, though, address the real relationship between power and the probability of war, and their authors assume in their attempts to empirically test them that the real relationship maps in a straightforward way to the relationships that we can observe in datasets on crisis outcomes. The implication of our model is that this assumption is not correct, hence their attempts to empirically test their models are suspect. Whatever real relationship exists between the balance of power 4 Fearon 1994; Huth and Allee Bueno de Mesquita, Morrow, and Zorick DRAFT DO NOT CITE 2

4 or signals of interest and successful coercion whether linear or not, monotonic or not the observable relationship is complex and non-monotonic. 6 The problems that we identify with the current approach to studying coercion are not quibbles. They raise fundamental questions about a decade of empirical scholarship on coercion. Variables that actually have a powerful effect on the real rate of deterrence success, or the real rate of success at compellence, may have been overlooked. Even worse, mistakes in interpreting selection effects and the predictions of models of coercion can lead to dangerous inferences that are exactly opposite of the truth. We offer suggestions to improve research design in studies of coercion using both quantitative and qualitative methods. One promising route is for scholars to use statistical estimators that are derived directly from the payoffs and structure of a crisis game. 7 Custom estimators like this would serve as alternatives to the off-the-shelf estimators like probit and logit that are widely used in the international relations literature. This approach directly captures the effects of strategic interaction (and hence internalizes the selection effect) while maintaining the traditional advantages of statistical research. The downside of this research design is that it places high demands on data quality and on scholars' confidence in their precise model specification. 6 Bueno de Mesquita, Morrow, and Zorick 1997 use a quadratic term in a logit model to test their model's prediction of a non-monotonic relationship between the balance of power and the probability of war. Signorino 1999 argues that their choice of logit (shared by many other scholars in their studies of crises) is not appropriate for datasets that are generated through strategic interactions. However, Signorino's tests do not help us to understand whether logit fails due to the underlying strategic relationship, due to the complex selection effect, or both. Signorino 1999 shows that there is a problem with off-the-shelf estimators, but he cannot explain the source of the problem. This paper shows the crisis dynamics much more clearly. Our suggestions for improving research design complement Signorino's suggestions. 7 Lewis and Schultz See also Signorino 1999; Smith DRAFT DO NOT CITE 3

5 A second promising approach using large-n datasets is for scholars to replace crisis outcomes with crisis initiation as their dependent variable. 8 This research design mitigates problems caused by selection effects, and the relationship between key independent variables and the decision to issue a challenge should at least be monotonic, but this approach has its own limitations: it allows scholars to answer only some of the important questions about coercion, and it does not solve all selection effects problems. Although neither of these two quantitative approaches is perfect, each can be used to strengthen scholars' understanding of coercion. The third promising research approach is almost universally overlooked as a method for mitigating selection effects: case studies. The central problem in studying coercion results from the indeterminate relationship between actual rates of successful coercion and observable patterns of crisis outcomes. To avoid that problem, scholars can study the process of decision making rather than the outcomes of crises. Scholars can use archival material to directly observe how leaders assess the seriousness of a given threat. What did the leaders discuss? Did they debate the significance of the adversary s domestic political institutions, note the existence or absence of formal alliances, and believe the adversary s public statements? Or did other factors weigh more heavily during their deliberations? Of course, selection effects will still lurk in the backgrounds of these case studies: crises will involve a non-representative sample of countries. But the selection effect will have a less pernicious impact because inferences are not being 8 See, for example, Leeds DRAFT DO NOT CITE 4

6 drawn from the frequency of coercive success but from the types of information that leaders focus on, discuss, and debate during crises. 9 The remainder of this paper is divided into four sections. The first section describes the canonical model of military deterrence crises and explains why it is susceptible to selection effects. The second section shows that the implications of the multi-stage model for crisis outcomes are more complex than past work has revealed and details the problems that the complexity poses for empirical studies. The third section demonstrates how case studies can mitigate the problems of selection effects. The conclusion emphasizes the importance of the full specification of the formal model, the necessity of carefully linking the design of statistical analyses to the formal model, and the value of careful case studies for better tests of theories of coercion. Selection Effects in Studies of Deterrence Scholars usually model deterrence interactions as occurring in two stages: general deterrence and immediate deterrence. General deterrence takes place during non-crisis periods when one country (a challenger) considers threatening a second country (the protégé); typically a third country in the model (the defender) may come to the protégé s aid. 10 General deterrence succeeds invisibly when prospective challengers decide not to threaten the protégé; if a threat is issued, general deterrence has failed. Once a protégé 9 Case studies have been criticized specifically for their vulnerability to selection effects: researchers may choose non-representative or biased cases (King, Keohane, and Verba 1994). These critics are correct, and case study researchers must choose their cases carefully. But this researcher-induced selection bias is separate from the selection effect introduced by the strategic behavior of countries (Collier, Mahoney, and Seawright 2004). Properly chosen case studies can dramatically mitigate the impact of the selection effects that plague datasets on crises. 10 Deterrence theorists distinguish between direct and extended deterrence. Direct deterrence refers to efforts to prevent attacks on oneself; extended deterrence is an effort to prevent attacks on others. The text describes extended deterrence situations, but this argument applies to direct deterrence, too. In those cases, the issue over which the challenger threatens can be considered the "protégé. DRAFT DO NOT CITE 5

7 is threatened, the defender must decide whether to respond by issuing a deterrent warning or by quietly conceding. If the defender promises to come to the protégé's aid, an immediate deterrence crisis begins. A subsequent attack by the challenger would constitute a failure of immediate deterrence; if the challenger instead refrains from attacking (backs down from its earlier threat), then immediate deterrence has succeeded. 11 Rational deterrence theory suggests that a powerful defender with a strong interest in a given protégé should be more effective at deterring attacks. Early quantitative analyses of deterrence crises applied that theory. 12 In their empirical tests, scholars expected to find a positive correlation between immediate deterrence success (i.e., a challenger's decision to back down during a crisis) and variables that reflect the defender's capabilities and interest in the protégé. We call this expected positive relationship the "traditional prediction" of deterrence theory. The straightforward approach for testing theories of deterrence was challenged by scholars, notably including James Fearon, who realized that selection effects distort the easily observable data on crisis dynamics. 13 To make valid inferences, scholars must consider the choices that challengers and defenders make before crises begin. Other things being equal, countries are more likely to threaten protégés if potential defenders are too weak to resist effectively or are likely to give in without a fight. Only highly 11 Some critics of the model (e.g., Lebow and Stein 1989 and 1990) question whether the absence of a threat necessarily reflects general deterrence success and whether a challenger s decision to back down during a crisis means that immediate deterrence necessarily succeeded. Sometimes countries even those that have issued threats have no interest in attacking, irrespective of any deterrence calculations. These critics raise important points. In this paper, however, we limit ourselves to critiquing the logic behind the research design of many empirical studies of coercion. For that purpose, we assume that investigators can correctly define the boundaries of each observation in their datasets. Later in this paper, we argue that careful case studies can avoid the problems of selection effects; they can also reduce the danger of misidentifying a challenger s lack of interest as deterrence. 12 See, for example, Huth and Russett 1984 and 1990; Huth, Gelpi, and Bennett The seminal works are Fearon 1992, 1994, and DRAFT DO NOT CITE 6

8 motivated challengers (i.e., those who would rather fight than accept the status quo) will threaten protégés in the sphere of powerful and credible defenders. In datasets of crises, therefore, challenger motivation will be correlated with defender credibility and power. The implication of this argument is both counter-intuitive and profound: the defenders who look most fearsome will usually fail at deterrence during crises. Credible and capable defenders might succeed at general deterrence, but scholars only observe immediate deterrence in datasets of crisis outcomes. Consequently, when scholars estimate the relationship between variables that reflect the defender's capabilities and credibility and immediate deterrence success, they should expect to find a negative correlation. We call this the "selection effects prediction" of deterrence theory. 14 Insert Figure 1 The selection effects argument is clearer when it is illustrated formally. The model in figure 1 depicts a deterrence crisis in four stages and describes the payoffs that the challenger and defender receive for each outcome. 15 First a potential challenger decides whether to threaten a protégé or simply accept the status quo; the value of the 14 Fearon 1994 expects a negative correlation between immediate deterrence success and variables that reflect defender interest but a positive correlation with variables that reflect defender power. This distinction is based on the assumption that although challengers update their assessments of defender interest during crises, they do not learn about defender power. We disagree; challengers can learn about both defender interest and power from crisis behavior. Therefore, the most internally consistent version of the selection effects prediction would treat interests and power similarly: expecting both variables to be inversely related to immediate deterrence success in crises. We use the latter version of the selection effects prediction, but our model results in the next section contradict both versions. 15 The model is based on Fearon 1992 with modifications explained in the text and the appendix. This fourstage model allows for uncertainty about both the challenger and the defender: the power and / or interests of both actors can be modeled as private information, so that each must act based on his predictions about what his adversary will do. Other scholars (e.g., Schultz 1999; Lewis and Schultz 2003) have used threestage models or even a two-stage model (Signorino and Yilmaz 2003) because they are simpler to solve mathematically yet still demonstrate the importance of strategic behavior for empirical studies of coercion. However, these simpler models eliminate the possibility of defender bluffs and therefore diverge from reality. That choice substantively affects the calculations of forward-looking challengers in the model. The four-stage model is the simplest that captures challengers' and defenders' simultaneous incentives to misrepresent their motivations and capabilities (the real-world situation). DRAFT DO NOT CITE 7

9 Fight War (1-p)*(A C ) - F C, (1-p)*(-A D ) - F D Figure 1: A Four Stage Deterrence Encounter C Don t Threaten Threaten D Status Quo (0, 0) Don t Mobilize Mobilize C Def. Acquiesces (A C, -A D ) Don t Attack Attack D Chal. Backs Down (-R C, R D ) Don t Fight C = Challenger D = Defender Note: Payoffs are described in text. Def. Backs Down (A C +R C, -A D -R D )

10 status quo is normalized to zero for both countries. If the challenger decides to threaten and the defender concedes (chooses not mobilize ), the challenger seizes the protégé and gains A C ; the defender loses A D. A C and A D represent the challenger and defender's levels of interest in the protégé. If, on the other hand, the defender mobilizes, the challenger must decide whether to back down or attack. Backing down, however, is not free. The challenger would suffer an audience cost equal to R C if he were to retreat from his threat; the defender would enjoy a foreign policy victory, receiving R D. If the challenger does attack, the defender has a final choice to make: he can back down, or he can carry through on his deterrent threat and fight to defend the protégé. Backing down entails the loss of the protégé and also an audience cost (-A D -R D ), while the challenger gets A C +R C for seizing the protégé and defeating the defender. The final outcome arises if the defender decides to fight: both the challenger and defender receive their expected value for war, which is a function of the probability that the defender will win (p), the value of the protégé, and the cost of fighting (F D for the defender and F C for the challenger). 16 For the defender, the expected value for war reduces to (1-p)(-A D ) F D ; 17 the challenger receives (1-p)(A C ) F C for fighting. 16 The original formulation in Fearon 1992 uses a composite "value for war" parameter rather than separating the power-related variables (probability of winning the war and the cost of fighting) from the interest variables (A C and A D ). This conflation of power and interests makes it difficult to follow the mechanisms by which real-world independent variables affect outcomes. For example, researchers interested in the effect of democracy on international relations have suggested that democracy might make countries more sensitive to the costs of war (reducing the value for war by increasing F in our model) and that democracies might be more likely to win their wars (increasing the value for war by changing p in our model). A composite value for war parameter complicates efforts to test these theories. Furthermore, each country s value for war presumably is directly related to the value that it assigns to the protégé, yet the composite "value for war" specification does not take that into account. Finally, using the value for war as the outcome payoff at the bottom of the tree hides a relationship between the challenger's payoffs and the defender's payoffs: the probability that the challenger will win a war is just one minus the probability that the defender will win the war (not accounting for ties), meaning that the payoffs at the bottom of the tree should be correlated. Fearon 1997 revises the payoff for war along the lines used here, but other recent efforts to model crisis interaction have continued to use the less transparent formulation (Schultz 1999; Lewis and Schultz 2003). DRAFT DO NOT CITE 8

11 With complete information, the game tree in Figure 1 has only three possible outcomes: the status quo, defender acquiesces, and war. If a defender is unwilling to fight (and the challenger knows it), the challenger will always threaten, and the defender will always acquiesce (i.e., "not mobilize"). If, on the other hand, a defender values the protégé highly enough or is powerful enough to have a high expected value for war, both the challenger and the defender will know that the defender will fight if the challenger attacks. A challenger then would have only two options: either 1) accept the status quo or 2) threaten, attack, and fight a war over the protégé. Under complete information, therefore, immediate deterrence never succeeds. Immediate deterrence can only succeed when a challenger either bluffs or probes impossible strategies with complete information. 18 Immediate deterrence is possible, however, if there is incomplete information. Challengers and defenders may not know their relative military capabilities (p) or the cost of a war (F C and F D ). More often in studying crisis behavior, scholars have focused on incomplete information about interests: a country can never be sure about the value its adversary places on a given protégé. 19 For example, not knowing the true level of the defender's interest creates an incentive for unmotivated challengers to initiate crises as a way to gain information all the while knowing that they will back down if the defender mobilizes. The challenger's goal in issuing the threat is to find out if the defender cares 17 The defender s payoff for war is [p*(0) + (1-p)*(-A D )] F D, which reduces to the expression in the text. 18 A bluff is a threat that a country knows it will not carry out if an adversary issues a counter-threat. A probe is a threat whose execution depends on the intensity of the adversary's response. 19 Huth Many empirical articles try to estimate the importance of various signals of interest. For example, does signing a formal alliance treaty increase the defender's credibility, hence increasing deterrence? Do public statements by leaders (like President Kennedy's famous "Ich bin ein Berliner") increase credibility and deterrence? Do tripwire deployments of troops that are too small to affect the probable outcome of a war send a strong signal of defender interest? These and other independent variables (e.g., measures of a defender's intrinsic interest in a protégé) are all presumed to correlate with a challenger's estimate of the defender's level of interest in a protégé. DRAFT DO NOT CITE 9

12 enough about the protégé to be willing to pay the cost of mobilizing; if the defender is willing to pay that cost, then the challenger can update and increase its assessment of the probability that the defender also cares enough about the protégé to be willing to fight for it. 20 In the model with uncertainty, immediate deterrence succeeds when a challenger who is simply probing or bluffing encounters a defender who mobilizes and is relatively likely to be willing to fight. The formal model described above can incorporate the assumption of incomplete information, thereby demonstrating the selection effect. If a challenger does not know how much the defender values the protégé (i.e., the actual value of A D ), it must use its best estimate of the defender's level of interest, K, to calculate the expected value of a bluff or probe. 21 When K is big, the expected value of a bluff is small because the defender appears relatively likely to mobilize and fight. 22 Therefore in datasets of crises, a credible defender (e.g., high K) is unlikely to be paired with a bluffer; a less credible defender could face either a bluffer or a highly motivated challenger. Because the rate of immediate deterrence is determined by the ratio of bluffers to committed challengers, 20 It is likely that challengers and defenders also gain information about the power variables during a crisis. For example, a challenger might learn how much of the defender's military it was willing to deploy to the protégé country, how smoothly the defender's troops were mobilized, and how many of the defender's allies were willing to mobilize, too. All of that information would allow the challenger to update its assessment of the probability that it would win a war over the protégé. The model could be readily recast to make p rather than A D and A C the incomplete information parameter. 21 Many of the tools of statecraft available to defenders in extended deterrence situations are ways of signaling their level of interest, thereby affecting K. For example, signing a mutual defense pact presumably would increase the value of K. 22 Challengers only benefit by bluffing when the defender chooses not mobilize. But as K increases, the challenger believes that fight becomes more attractive to the defender relative to surrendering the protégé. The derivative of the payoff for fight with respect to A D is greater than the derivative of the payoffs of both don t mobilize and don t fight" (that is, p-1 > -1). DRAFT DO NOT CITE 10

13 actions that increase K should correlate negatively with the observed rate of immediate deterrence success the selection effects prediction. 23 In sum, recent scholarship on deterrence, increasingly attuned to selection effects, argues that previous analyses systematically misinterpreted their data on deterrence. But the studies that model selection effects offer good news: if we consider the strategic behavior that led countries into crises, we can correct our interpretation of statistical tests of deterrence theory. Specifically, those attributes of a crisis that correlate with immediate deterrence failure should be emulated by potential defenders, because they are successfully screening out all but the highly motivated (undeterrable) challengers before crises even begin. In other words, the results that the early studies of deterrence produced should be reversed; the signs predicted for coefficients in deterrence theory regressions should be "flipped." This simple correction allegedly helps us to understand the true relationship between various independent variables and deterrence success. A More Complete View of the Multi-Stage Model By recognizing the danger of selection effects in data on deterrence, scholars have identified a critical flaw in early studies. Unfortunately, scholars have drawn the wrong empirical predictions from the multi-stage model of deterrence, leading to incorrect interpretations of data on crisis outcomes. The signs of the coefficients of estimated relationships between independent variables and immediate deterrence success may be misunderstood, and for many samples, the estimates will be biased toward zero. 23 The reason that unmotivated challengers usually do not threaten a highly credible defender that is, the reason for the selection effect is that there are costs associated with backing down during a crisis (R C and R D ). If prospective attackers faced no costs from backing down, then there would be no selection effect. On audience costs, see Schultz 2001; Fearon DRAFT DO NOT CITE 11

14 The central problem is that the traditional predictions i.e., that greater defender power, interest in the protégé, and credibility make immediate deterrence success more likely and the selection effects predictions i.e., the exact reverse are both sometimes correct. 24 In other words, actions that strengthen general deterrence (reduce the number of challenges) will sometimes cause the observed probability of immediate deterrence successes to increase, and other times the same actions will cause the rate of immediate deterrence success to decline. Scholars will find it very difficult to determine a priori which situation applies for a given sample or whether each situation applies for a subset of the data. The result is that deterrence theory makes no determinate predictions about patterns of immediate deterrence success in scholars' datasets, and scholars cannot test specific hypotheses about coercion (e.g., whether local military force advantages bolster deterrence) by drawing straightforward inferences from patterns of crisis outcomes. A detailed look at the effects of an increase in defender credibility shows its two countervailing effects on the likelihood of immediate deterrence. According to the logic of the selection effects prediction, it reduces the frequency of immediate deterrence success by reducing the expected value of bluffing and therefore the pool of bluffers who decide to issue threats. At the same time, though, an increase in defender credibility also reduces the expected value of attacking after the defender mobilizes, because it seems more likely to the challenger that the defender will fight. As a result, some challengers that might have been willing to attack against a less-credible defender instead will only 24 In his excellent article on the democratic peace, Schultz 1999 notes in passing a non-monotonic relationship between key variables in his model and war likelihood. Schultz s model is not intended to capture the complete dynamics of deterrence crises (e.g., for simplicity he omits the stage in which defenders can bluff); however, his results are generally consistent with our finding. See also Lewis and Schultz DRAFT DO NOT CITE 12

15 probe, and if they face defenders who actually mobilize, those challengers will back down. Against a less credible defender, they would have been undeterrable, but the increase in defender credibility turned them into examples of immediate deterrence success. In sum, increasing defender credibility both reduces and increases the number of bluffers in the observable dataset. The net effect of changes in credibility on immediate deterrence success depends on the relative magnitude of the two effects. If, for example, challenger audience costs (R C ) are very big, then the pool of bluffers should shrink rapidly when defender credibility rises; in this case the selection effect prediction is correct, and increasing defender credibility will correlate with less immediate deterrence. But if audience costs are small, the pool of challengers who actually plan to attack may shrink more quickly than the pool of bluffers, in which case increases in defender credibility will lead to more immediate deterrence success. The other parameters in the game tree i.e., the costs of fighting, the probability of defender victory, and the baseline level of challenger and defender interest in the protégé similarly affect the responses of bluffers and committed attackers to an increase in defender credibility, changing the relative composition of the observed pool of challengers in a dataset. The net effect on the predicted correlation between defender credibility and immediate deterrence success is ambiguous. Insert figure 2 Figure 2 demonstrates the countervailing effects graphically. The line depicts the range of potential values for A C, the challenger's interest in the protégé, from the lowest possible interest at the left to the highest at the right. We assume that the defender's precrisis estimate of the challenger's interest (J) lies in the center of the range of possible DRAFT DO NOT CITE 13

16 Figure 2: Indifference Points between Challenger Strategies Challenger Strategy Don t Threaten Threaten/Don t attack Threaten/attack Challenger Indifference points A C - J-α A C1 A C2 J+α Challenger: Status quo seeker Bluffer Motivated attacker

17 values and that the defender's uncertainty about his estimate (α) correctly delimits the width of the interval of possible levels of challenger interest. 25 Two particular points are indicated in the figure: A C1 is defined as the value for A C at which a challenger is indifferent between adopting the strategies not threaten and threaten/not attack. 26 The actual value of A C1 can be calculated in terms of the other payoffs on the figure 1 game tree (see the appendix). Similarly, A C2 is the value of A C at which a challenger is indifferent between the strategies of threaten/not attack and threaten/attack. In the figure, the probability of immediate deterrence success is the ratio of the distance between A C1 and A C2 to the distance between A C1 and J+α. An action taken by a defender prior to a crisis that increases its credibility (e.g., something that increases K) has two effects on potential challengers' calculations. First, because the action makes the defender appear more likely to mobilize, the incentive for a challenger to bluff declines. A C1, therefore, moves to the right. 27 This shift of A C1 is why the selection effects literature argues that credible defenders deter most bluffers from issuing a threat. But an increase in defender credibility also means that the defender is more likely to fight for the protégé rather than choose not fight after a challenger attacks. Therefore, only a highly motivated challenger will actually attack when facing a credible defender. In figure 2, A C2 moves to the right as defender credibility increases. 25 In this model α is public knowledge. We use α to reflect both the defender s uncertainty about the challenger s actual interest in the protégé and the challenger s uncertainty about the defender s true level of interest. 26 A challenger would choose the strategy threaten/not attack in the hope that the defender would not mobilize. 27 One way to think of this is that as defender credibility increases, the marginal bluffer decides that bluffing is not worth it i.e., as defender credibility increases, it takes a greater value of A C to make a challenger indifferent between not threatening and bluffing ( threaten/not attack ). DRAFT DO NOT CITE 14

18 Unless one knows the relative distance that A C1 and A C2 shift as defender credibility increases, one cannot determine the net effect on the probability of immediate deterrence success. If A C1 shifts more quickly than A C2, the proportion of bluffers in crises will drop, and successful immediate deterrence will become less common. This is the selection effects prediction. But if A C2 shifts more quickly, rising credibility will increase the likelihood of immediate deterrence success, as suggested by the traditional prediction. The problem for scholars who study deterrence is that the net effect of increases in credibility on deterrence outcomes depends on the precise values of the other parameters in the game tree. The appendix demonstrates the complexity of these relationships. For a wide range of values for the magnitude of audience costs, costs of fighting, probability of defender victory in a war, and uncertainty in the adversaries' predictions of each other's level of interest in the protégé, we can choose values of the other variables such that an increase in defender credibility will either increase or decrease the probability of immediate deterrence success. Without very precise measurements of all of these variables, scholars cannot know whether deterrence theory predicts a positive or negative correlation between actions that signal greater defender interest in its protégé and immediate deterrence success. Insert figures 3 Figure 3 shows the rate of immediate deterrence success as a function of the defender s power (p) and its apparent interest in the protégé (K) under a range of circumstances. Panel 1 shows the relationship between K and IDS for a range of DRAFT DO NOT CITE 15

19 Figure 3: Complex Relationships between power (p), interests (K), and immediate deterrence success (IDS)

20 parameter values that might describe typical cases. 28 Panel 2 shows the relationship between p and IDS for the same sets of parameters. Panel 3 illustrates the relationship between K and IDS in a defense dominant world, meaning that the expected costs of fighting are greater for the challenger than the defender. Panel 4 presents the relationship between p and IDS in a rapacious world: with these values, bluffing is rampant because the cost of backing down is low, and war is common because the cost of fighting is much smaller than the potential spoils of victory. In all four panels, the relationship between the variable of interest and IDS is non-monotonic and quite complex. 29 There are three key points to take from these graphs. First, an increase in K or p can result in either a substantial increase or decrease in the rate of immediate deterrence success. Therefore, steps that a defender takes that successfully signal its interest in a protégé or its ability to successfully defend a protégé could generate either higher or lower rates of observable deterrence success (IDS). Second, for some parameter values (e.g., F C =9 in panel 1 and R C =5 in panel 3), the relationship between the defender s apparent interest, K, and immediate deterrence success is essentially flat for wide ranges of parameter values, meaning that even when deterrence is working (i.e., challengers are threatening and attacking less often than they would have at lower levels of K), no evidence of this successful deterrence will appear in data on crisis outcomes. Finally, the relationship between K and IDS is a function of the size of K; as K varies, the 28 We consider these values to be typical because audience costs are smaller than the costs of fighting (except for the F C =3 line), and because the costs of fighting are smaller than the value of the protégé (except for the lowest values of K in Panel 1). Scholars may disagree about what constitute typical values, but the general shapes of these lines appear with a range of parameter values. 29 Most of the curves end before reaching the right limit of the graph (in Panel 3 the curves end at values of K that range from 18.0 to 18.8, though this is difficult to see). This occurs because for some parameter values there are no crises. For example, if for a given set of parameters everyone knows that even the most interested defender is unwilling to fight (A D2 >K+ ), then all challengers will threaten but no defenders will mobilize, so there will be no crises. DRAFT DO NOT CITE 16

21 relationship between K and IDS changes. 30 Similar results can be seen in the panels relating IDS to p. These graphs illustrate a serious problem for analyses that attempt to draw inferences about theories of coercion by observing immediate deterrence outcomes. For example, studies that regress immediate deterrence success on either indicators of a defender s interest in a protégé or indicators of a defender s power will not produce meaningful results. 31 If a study assumes that the selection effects prediction is correct but inadvertently samples cases in which the actual relationship between K (or p) and immediate deterrence success is positive, the analysis will tend to fail theories that are correct and possibly confirm those that are wrong. If the sample comprises observations in which the relationship between K or p and IDS is relatively flat, then variables that have great significance as causes of K or p and hence great significance for coercion will appear to be irrelevant. And if the study happens to examine a sample that includes both positive correlation and negative correlation cases (for example, cases that cross over a local maximum of the probability of immediate deterrence success), the estimated coefficient relating defender interest or power to immediate deterrence success will be biased towards zero. These latter two cases will be statistically indistinguishable from 30 If the relationship between K and IDS were always concave down, scholars could execute a weak test of various theories of deterrence using a quadratic specification in a regression equation: the squared term on the relationship between K and IDS should never have a positive coefficient. Unfortunately for some parameter values (e.g., the left part of the Fc=3 curve in Panel 1), the relationship is concave up. We thank Bear Braumoeller for discussion of this point. 31 Signorino 1999 also shows that strategic interaction between countries can cause problems for attempts to estimate crisis models, specifically for studies using logit (and probit). Signorino's paper draws on a version of the Bueno de Mesquita et al. model: the true relationship between power and the probability of war is non-monotonic because of the endogeneity of the identity of challengers. His results are based on a complete information game (with uncertainty about crisis outcomes generated because each country is assumed to make errors in its strategic choices at a publicly known rate) so while his article does an excellent job of showing examples of the problems with using off-the-shelf estimators in the presence of strategic interaction, the assumption of perfect information is not realistic (see also Lewis and Schultz 2003). The incomplete information model developed in this article better captures actual crisis dynamics. DRAFT DO NOT CITE 17

22 cases in which the independent variables genuinely have no relationship to immediate deterrence success. These results support neither the traditional prediction of deterrence theory nor the selection effects prediction. 32 The addition of control variables to off-the-shelf quantitative estimators cannot solve the selection effects problems. Controlling for the value of the other parameters in the model (e.g., F C, F D, etc.) merely accounts for the effects of those parameters on IDS, not for their effects on the shape of the relationship between K and IDS or p and IDS. Adding control variables would assume that there is a single, "true" relationship between the study variables and IDS and that each observation, once the effects of the control variables are factored out, would contribute additional information about those true relationships. Unfortunately that assumption is not warranted: there is no single function that relates K or p to IDS. In effect, large datasets on crises almost certainly encompass multiple causal relationships between K, p, and IDS, and consequently the observations cannot simply be pooled. 33 Even in a dataset with observations drawn randomly from all values of K, p, and the other parameters, estimators that compute an "average" relationship between the independent variables of interest and IDS, hoping to "wash out" the effects of the various relationships between intervening variables like K and IDS, would not yield meaningful results. Scholars have also been tempted to try to mitigate the selection effects problem by using sophisticated two-stage estimators proposed by Heckman and others. 34 A recent 32 Note that the results in Figure 3 also contradict Fearon s version of the selection effects prediction, which suggests a monotonic negative relationship between IDS and interest variables (K) and a monotonic positive relationship between IDS and power variables (p). 33 Collier and Mahoney Huth and Allee 2002; Smith 1996; Nooruddin DRAFT DO NOT CITE 18

23 article on research design explicitly endorses this trend. 35 These analyses separately estimate relationships among variables during two stages of a crisis: they first study the decision to initiate a crisis and then study behavior during a crisis conditional on the prior decision to initiate a crisis in the first stage. The models are based on the hypothesis that the error term in the first estimate is correlated with the error term of the second estimate. Accounting for correlation in error terms is important, but it does not address the key problem in research design described in this article: even if there were no correlation in the error terms, we would still not know the right functional form to estimate at either stage. 36 The relationships at each stage are non-linear, and their shapes depend on all of the model parameters. Simply using a selection model estimator does not help us to determine whether to expect an increase in defender credibility will deter more potential challengers at the first stage (crisis initiation) or at the second stage (crisis behavior). In other words, putting aside the problems with the error terms, scholars do not know a priori what coefficients to expect and what functional form to look for in their statistical analyses of crisis behavior. In sum, Fearon s insight about selection effects made a substantial contribution to scholars understanding of pitfalls in studies of deterrence. Unfortunately the hurdles that stand before scholars are even higher than Fearon and others realized: both the traditional and the selection effects predictions about the relationship between defender credibility and immediate deterrence success should obtain in datasets of crisis outcomes. 35 Huth and Allee In a similar vein, Smith 1999 also argues that at least two separate problems (censoring and interdependence of observations) plague datasets on crises. Accounting for correlation of error terms at best would solve one of the problems. DRAFT DO NOT CITE 19

24 Furthermore, simple solutions such as using off-the-shelf selection estimators do not solve this problem. Mitigating the Problems of Selection Effects with Case Studies Given that selection effects make it difficult to make straightforward predictions about crisis outcomes, what should scholars do to study coercion empirically? One approach is to apply better statistical methods to datasets of crisis outcomes or to new datasets tailored to account for the selection effects. We will address some of these possibilities in the next section. A promising alternative is to study the process of decision-making rather than the outcomes of crises. This section builds on the general observation of Collier, Brady, and Seawright that studies of the causal process gain their inferential leverage in a different way than studies of "dataset observations." 37 Examining the decision-making process allows researchers to (1) directly observe the variables that scholars typically measure indirectly (such as decision-makers estimates of their adversaries' power, interests, and credibility), and (2) directly observe which tools of statecraft influenced those estimates. The key point is that most quantitative studies of coercion use patterns of crisis outcomes to draw inferences about how credible, powerful, or committed each country appeared to its adversary (and hence about the relative effectiveness of the steps each country took to signal those attributes); these inferences are dubious because of the non-monotonic relationships explained in the previous section. By studying the decision-making process, however, scholars can avoid reliance on inferences about what a given rate of IDS implies about a country s 37 Collier, Brady, and Seawright DRAFT DO NOT CITE 20

25 credibility, power, or interests. Studying the decision-making process therefore avoids the most serious problems posed by selection effects for studies of coercion. 38 Many theories of coercion can be tested using evidence about the decision making process. For example, scholars believe that leaders' assessments of their adversaries' credibility have an important effect on coercion, so scholars would like to know what influences decision-makers' assessments. Specifically, a credible defender is one that the challenger believes is likely to fight rather than back down if confronted with a choice at the fourth node of the game tree in Figure 1. Credibility then depends on model parameters like K, p, and F D. 39 Ideally scholars could study the effects of various tools of statecraft on K, p, F D, and the other model parameters, and they could then learn about both the causes of credibility and crisis dynamics. In the empirical record, decision-makers do not speak in the language of the model, but they frequently estimate their adversary's overall credibility and other variables. Scholars can translate these estimates into the variables that are important for the theories that they want to test, and they can also examine what evidence decisionmakers used as they made their assessments. Did they consider alliances, their adversaries' past behavior, the balance of military capabilities, the personality traits of specific leaders, or other factors? Scholars can read the internal memos and the transcripts from closed-door meetings to "listen in" on the secret deliberations. Armed with direct measures of credibility gleaned from examining the decision-making process, 38 Although decision-making processes can be studied using either quantitative or qualitative techniques, we highlight case study research designs because they have been overlooked as an approach to avoid selection effects. To be clear, scholars have frequently used case studies to study crisis decision-making and deterrence, but most of the past qualitative studies, like their quantitative counterparts, draw key conclusions from crisis outcomes and are therefore vulnerable to the problems introduced by selection effects. See, for example, George and Smoke Credibility is the challenger's estimate of the probability that the defender will fight. We give a mathematical expression for this probability, labeled as y, in the appendix. DRAFT DO NOT CITE 21