Rethinking historical controls

Transcription

1 Biostatistics (2001), 2, 4,pp Printed in Great Britain Rethinking historical controls STUART G. BAKER Biometry Research Group, Division of Cancer Prevention, National Cancer Institute, EPN 344, 6130 Executive Blvd MSC 7354, Bethesda, MD , USA KAREN S. LINDEMAN Department of Anesthesiology, The Johns Hopkins Medical Institutions SUMMARY Inference from traditional historical controls, i.e. comparing a new treatment in a current series of patients with an old treatment in a previous series of patients, may be subject to a strong selection bias. To avoid this bias, Baker and Lindeman (1994) proposed the paired availability design. By applying this methodology to estimate the effect of epidural analgesia on the probability of Cesarean section, we made two important contributions with the current study. First, we generalized the methodology to include different types of availability and multiple time periods. Second, we investigated how well the paired availability design reduced selection bias by comparing results to those from a meta-analysis of randomized trials and a multivariate analysis of concurrent controls. The confidence interval from the paired availability approach differed considerably from that of the multivariate analysis of concurrent controls but was similar to that from the meta-analysis of randomized trials. Because we believe the multivariate analysis of concurrent controls omitted an important predictor and the meta-analysis of randomized trials was the gold standard for inference, we concluded that the paired availability design did, in fact, reduce selection bias. Keywords: All-or-none compliance; Concurrent controls; Meta-analysis; Propensity scores; Randomized trials. 1. INTRODUCTION The gold standard for evaluating the effects of medical interventions is the randomized controlled trial. Unfortunately, in some situations, a satisfactory randomized trial is not feasible. Expanding on Byar (1980), some reasons include (i) enrollment of sufficient numbers of patients would take too long, (ii) the cost or required effort would be excessive, (iii) controversy exists over various ethical issues or (iv) the study could not be blinded, and an unblinded analysis would introduce bias. When a randomized trial cannot be implemented, general approaches for inference with comparative studies include (i) multivariate adjustments for concurrent controls and (ii) historical controls. Unfortunately, results from both approaches can be seriously biased. Multivariate adjustments can be biased if they fail to adjust for an important predictor (see, for example, Byar (1980)). Traditionally when making inference from historical controls, one compares the new treatment in a current series of patients with the old treatment in a previous series of patients, which can give biased results if subjects with certain risk factors related to outcome are more or less likely to receive the new treatment (see, for example, Pocock (1983)). To whom correspondence should be addressed c Oxford University Press (2001)

2 384 S. G. BAKER AND K. S. LINDEMAN To strengthen inference from historical controls Baker and Lindeman (1994) proposed the paired availability design in the context of a proposed study of the effect of labor epidural analgesia on the probability of Cesarean section (CS). This is an important topic because (i) CS, especially during labor, increases maternal morbidity and mortality when compared with vaginal delivery (National Institute of Child Health and Human Development, 1981), (ii) epidural analgesia represents an effective form of pain relief not matched by other agents, and (iii) each year 1.6 million women in the United States receive epidural analgesia for pain relief during labor (Hawkins et al., 1997; Lieberman, 1999). Because of insufficient data to implement their analysis, Baker and Lindeman (1994) focused on design issues and sample size calculations for the number of required studies. More recently, many studies of the effect of epidural analgesia on CS have been published. These include studies which could be analyzed via the methods for the paired availability design, as well as randomized trials, and multivariate observational studies. Using data from these studies, we extended our previous work in two ways. First, we generalized the paired availability design to include two different types of availability and to accommodate data from multiple time periods. Second, we compared the results of the current paired availability analysis with those from a multivariate analysis of concurrent controls and from a meta-analysis of randomized trials. 2. PAIRED AVAILABILITY DESIGN The basic paired availability design has three characteristics: (i) a comparison of all eligible subjects at a later time with all eligible subjects at previous time using as an intervention the availability, not the receipt, of treatment at a hospital, (ii) a model to estimate the effect of receipt of treatment, and (iii) a meta-analysis to combine results from all the hospitals. In Section 5, we extend (i) to multiple time periods. In applying the paired availability design to study the effect of epidural analgesia on the probability of CS, we identified 11 hospitals or medical institutions which reported changes in availability of epidural analgesia (Table 1). Because the technique of epidural analgesia stabilized in the late 1970s, we only included studies with data after We first consider the case of two time periods as in the original formulation of Baker and Lindeman (1994). To put the analysis on firmer foundation, we consider the following scenarios for a change in availability of epidural analgesia over time. Fixed availability. Epidural analgesia is available at a certain time during the first time period, such as the daytime, and is also available at another time, such as the evening, during the second time period. Alternatively, epidural analgesia is only available for a certain class of patients in the first time period, such as very sick patients, and is more widely available in the second time period. Random availability. More anesthesiologists are available to provide epidural analgesia during the second time period than the first. When a patient arrives at the hospital, the availability of epidural analgesia is random, depending on whether or not the anesthesiologist is working elsewhere in the hospital. 3. THOUGHT EXPERIMENT With the two scenarios for availability in mind, consider the following thought experiment. Suppose each woman who delivered in the first time period had entered the hospital at a random time during the second time period and vice versa. Under fixed availability, we assume that a women would arrive at the same time of day in either time period. Under random availability, we assume the same probability

3 Rethinking historical controls 385 Table 1. Paired availability design. Studies ordered from smallest to largest sample size Study Times Number Fraction who received Estimate (s.e.) Epidural CS Both Johnson and Rosenfeld (1995) (0.143) Frolich et al. (1997) (0.196) Lyon et al. (1997) (0.048) Robson et al. (1993) (0.026) Gribble and Meier (1991) (0.022) Larsen (1992) (0.044) Fogel et al. (1998) (0.015) Yancy et al. (1999) (0.014) Mancuso (1993) (0.018) Newman et al. (1995) (0.031) Dailey (1999) (0.025) Data read from a graph Based on model for multiple times. of availability throughout the time period. Under either fixed availability or random availability, we can classify subjects by whether or not they would have received epidural analgesia in each time period. The classification depends on the preference of the subject and the availability of epidural analgesia. Let R be a random variable which denotes this classification; it has four realizations: a, always-receivers, who would receive epidural analgesia in either time period; c, consistent-receivers, who would not receive epidural analgesia in the time period with less availability and would receive it in the time period with greater availability; i, inconsistent-receivers, who would receive epidural analgesia in the time period with less availability and would not receive it in the time period with greater availability; n, never-receivers, who would not receive epidural analgesia in either time period. For some subjects we will not be able to deduce the realization for R but this is not necessary for our analysis. The variable R has its origins in Baker and Lindeman (1994) and is similar to a variable independently proposed by Angrist et al. (1996).

4 386 S. G. BAKER AND K. S. LINDEMAN 4. FRAMEWORK FOR INFERENCE Having presented the thought experiment, we can now outline our framework for inference. Let Y = 1 if a subject has a CS and 0 otherwise. Let D = 1 if a subject receives epidural analgesia and 0 otherwise. Ideally one would compare outcomes after all the patients received D = 1, and then, after turning back the clock, the same patients received D = 0. In our notation the comparison would be pr(y = 1 D = 1) pr(y = 1 D = 0). This study design would eliminate the possibility that different outcomes for D = 1 and D = 0 resulted from differences inherent to the patients. Although this study design is impossible to implement, it sets a standard for evaluation. Using a traditional approach to historical controls, one would compare CS probabilities among subjects who received epidural in the later time, T = 1, with those who received it in the earlier time, T = 0. In our notation the comparison is pr(y = 1 D = 1, T = 1) pr(y = 1 D = 0, T = 0). Estimation based on this comparison is subject to selection bias because subjects who receive epidural analgesia may have certain risk factors, such as painful labor, that make it more likely from them to have CS even if epidural analgesia has no effect (Hess et al., 2000). Now consider the paired availability design. Let Z = 0 denote the time period before a change in the availability of epidural analgesia, and let Z = 1 denote the time period after a change in availability. The paired availability design directly compares the probability of CS in all eligible subjects at Z = 0 versus the probability of CS in all eligible subjects when Z = 1, namely, pr(y = 1 Z = 1) pr(y = 1 Z = 0). Because this comparison is based on all subjects, it avoids the selection bias associated with traditional historical controls. However pr(y = 1 Z = 1) pr(y = 1 Z = 0) estimates the effect of increased availability on the probability of CS, not the effect of epidural analgesia on probability of CS, which is of primary interest. We will show that with additional assumptions, we can use the paired availability design to estimate the effect of epidural analgesia on the probability of CS among a special subset of the population, namely consistent-receivers. In our notation, this comparison is θ = pr(y = 1 D = 1, R = c) pr(y = 1 D = 0, R = c). (4.1) In an ideal situation, (4.1) is a comparison of outcomes after all the consistent-receivers received D = 1, and then, after turning back the clock, the same consistent-receivers received D = 0. In practice (4.1) is still a causal effect because of assumptions which state that assignment to Z (which determines D in consistent-receivers) is essentially random and some additional assumptions. Because R is a baseline covariate, estimation based on θ avoids selection bias associated with traditional historical controls. Note that θ strictly applies only to consistent-receivers. In practice, one might argue that θ correctly measures the effect in all subjects. We will argue that the assumptions necessary for valid inference from θ are plausible, much more so than those required for valid inference with traditional historical controls. 5. ASSUMPTIONS To appropriately base inference on (4.1) we require the following assumptions. These assumptions modify those in Baker and Lindeman (1994), putting them on firmer foundation by incorporating both fixed and random changes in availability and including more supporting evidence. The following assumption, or a variant, is necessary with any type of study involving possible confounding over time. ASSUMPTION 1(STABLE CARE) There were no systematic changes obstetric practice, unrelated to epidural analgesia, which could affect the probability of CS. We restrict Assumption 1 to systematic changes, because combining estimates from multiple hospitals will average the effect of random changes in obstetric practice over time. In support of Assumption 1,

5 Rethinking historical controls 387 with the exception of abstracts by Robson et al. (1993) and Newman et al. (1995) which did not discuss the matter, all of the studies reported that obstetric practice did not vary over time. In addition, three considerations mitigate the possibility of substantial bias from a systematic change in obstetric practice over time. First, the studies spanned a period of 15 years and included hospitals in both North America and Europe, so it is unlikely the same systematic change, if any, occurred in all studies. Second, in all the studies utilization of epidural analgesia increased over time with the exception of Johnson and Rosenfeld (1995) in which epidural availability decreased following initiation of state-funded health care insurance. Therefore, a systematic change over time would have the opposite effect in Johnson and Rosenfeld (1995) compared with the other studies. Third, as we will discuss, in the study by Dailey (1999) with multiple time periods, we tested for a change in effect over time and found it was not statistically significant. The second assumption, or a variant, is also necessary with any study with possible confounding over time. Here it has important implications regarding the thought experiment. ASSUMPTION 2(STABLE POPULATION) There are no changes between the two time periods in the characteristics of the population related to the probability of CS in the absence of epidural analgesia. The joint distribution of whether or not a subject requests epidural analgesia, the arrival time at the hospital, and time course of labor is the same in both time periods, so the distribution of R does not depend on the time period: pr(r = r Z = 0) = pr(r = r Z = 1). In support of Assumption 2, with the exception of abstracts by Robson et al. (1993) and Newman et al. (1995) which did not discuss the matter, all the studies reported that the patient population did not vary over time. In addition, many of the studies involved a closed population served by an army medical center or the only hospital in a geographic region. It would be unlikely that a women in labor would go to another, considerably less convenient, hospital in order to receive epidural analgesia. As in Baker and Lindeman (1994) and Angrist et al. (1996), in which it is called an exclusion restriction, we make the following assumption which allows us to peel away information from subjects who would not change treatment in the thought experiment and thus estimate the quantity of interest. ASSUMPTION 3(NOEFFECT OF TREATMENT TIMING) For always-receivers and never-receivers, the probability of CS does not depend on the time period. In our notation, pr(y = 1 R = r, Z = 1) = pr(y = 1 R = r, Z = 0), for R = a, n. For always-receivers, the time during the course of labor when epidural analgesia is initiated depends on the availability of epidural analgesia and hence differs by time period. For example, under random availability, a subject in Z = 1 might receive an epidural sooner than if she were in Z = 0, simply because an anesthesiologist was available sooner. Therefore Assumption 3 would be violated if the initiation time of epidural analgesia affected the probability of CS. However, after randomizing subjects to early or late initiation of epidural analgesia, Chestnut et al. (1994) found that time of initiation did not affect the probability of CS. For never-receivers, one possible complication is that some women will not receive epidural analgesia due to rapid delivery and might have a different probability of CS than women who receive opioid analgesia. However, if Assumption 2 holds, the fraction of women with rapid deliveries should be similar for Z = 0orZ = 1, so the probability of CS will be the same for all never-receivers. We require one final assumption. ASSUMPTION 4(STABLE PREFERENCES) There is no change in preference for epidural analgesia over time. For fixed availability, there are no inconsistent receivers: namely, pr(r = i) = 0. For random availability, the effect of epidural analgesia on consistent-receivers is the same as on inconsistentreceivers: namely, θ = γ, where γ = pr(y = 1 D = 1, R = i) pr(y = 1 D = 0, R = i). Assumption 4 would be violated if new information became available in the second time period, such as a report warning of the risks from epidural analgesia, which would alter preferences for epidural

6 388 S. G. BAKER AND K. S. LINDEMAN Table 2. Thought experiment: fixed availability and stable preferences Request Time of Available Available epidural day in Z = 0 in Z = 1 R = a always-receiver yes daytime yes yes R = c consistent-receiver { yes evening no yes R = n never-receiver no daytime yes yes no evening no yes R = i is not possible with stable preferences and fixed availability Table 3. Thought experiment: random availability and stable preferences Request Available Available epidural in Z = 0 in Z = 1 R = a always-receiver yes yes yes R = c consistent-receiver yes no yes R = i inconsistent-receiver { yes yes no R = n never-receiver yes no no no yes/no yes/no analgesia. Technically, Assumption 4 is also needed to justify Assumption 2 that the distribution of R is the same over time. We do not know of any violations of Assumption 4. For fixed availability, Assumption 4 is the monotonicity assumption in Angrist et al. (1996) and was the assumption used in Baker and Lindeman (1994). Under fixed availability, the only way for some subjects to become inconsistent-receivers is for preferences to change over time (Table 2), which Assumption 4 rules out. For random availability, Assumption 4 is predicated on the fact that both inconsistent- and consistent-receivers would always request epidural regardless of time period (Table 3); the only difference is whether or not, by chance, an anesthesiologist were available to provide the epidural analgesia. 6. ESTIMATION FOR TWO TIME PERIODS Invoking the assumptions, we can obtain a simple estimate for (4.1) Starting with the identity pr(y = 1 Z = z) = r pr(y = 1 Z = z, R = r) pr(r = r Z = z), and invoking Assumption 2, we obtain pr(y = 1 Z = z) = r pr(y = 1 Z = z, R = r) pr(r = r). Invoking Assumption 3, gives pr(y = 1 Z = z) = pr(y = 1 R = r) pr(r = r) r=a,n + pr(y = 1 Z = z, R = r) pr(r = r) (6.1) r=c,i Using (6.1), the difference in probabilities of CS between the two time periods is pr(y = 1 Z = 1) pr(y = 1 Z = 0) =[pr(y = 1 Z = 1, R = c) pr(y = 1 Z = 0, R = c)] pr(r = c) +[pr(y = 1 Z = 1, R = i) pr(y = 1 Z = 0, R = i)] pr(r = i). (6.2)

7 Rethinking historical controls 389 Essentially, Assumption 3 has peeled away information from always-receivers and never-receivers, which was uninformative because there was no change in treatment. Noting that realizations for Z and R imply a realization for D, we rewrite (6.2) as pr(y = 1 Z = 1) pr(y = 1 Z = 0) = θ pr(r = c) ψ pr(r = i), (6.3) where ψ = pr(y = 1 D = 1, R = i) pr(y = 1 D = 0, R = i) and θ was defined in (4.1). Invoking Assumption 2 and the definition of a, i, and c, we can write pr(d = 1 Z = 0) = pr(r = a) + pr(r = i) and pr(d = 1 Z = 1) = pr(r = a) + pr(r = c), which together imply Dividing (6.4) by (6.3) gives pr(d = 1 Z = 1) pr(d = 1 Z = 0) = pr(r = c) pr(r = i). (6.4) pr(y = 1 Z = 1) pr(y = 1 Z = 0) pr(d = 1 Z = 1) pr(d = 1 Z = 0) pr(r = c)θ pr(r = i)ψ =. pr(r = c) pr(r = i) By invoking Assumption 4, which says that pr(r = i) = 0 for fixed availability or θ = ψ for random availability, we can express θ as the ratio of the difference in probabilities of CS to the difference in probabilities of epidural analgesia, θ = pr(y = 1 Z = 1) pr(y = 1 Z = 0) pr(d = 1 Z = 1) pr(d = 1 Z = 0). (6.5) Baker and Lindeman (1994) gave a related proof for fixed availability. In a different context, Angrist et al. (1996) gave a related proof based on equivalent assumptions. Let n zdy denote the number of subjects in time period z who receive d and have outcome y, and let p zdy = n zdy /n z++. The maximumlikelihood estimate is θ = (p 1+1 p 0+1 )/(p +11 p +01 ), with approximate asymptotic variance of ( z p z+1 p z+0 /n z++ )/(p +11 p +01 ) 2 (Baker and Lindeman, 1994). 7. ESTIMATION FOR MULTIPLE TIME PERIODS In some studies the change in epidural use occurred gradually and results were reported at multiple time periods, Z = 0, 1, 2,...,k. One approach to analyzing these data is to separately combine the data from the first k/2 time periods and the last k/2 time periods and treat it as a two-period design. However, if data {n zdy } are available, as in the largest study by Dailey (1999), we can utilize all the information to obtain better estimates. For each pair of successive time periods, z and z + 1, we define a variable R z with realizations {a, c, i, n}, as with variable R. Let θ z = pr(y = 1 D = 1, R z = c) pr(y = 1 D = 0, R z = c). If the assumptions cover multiple time periods, we can extend (6.5) to θ z = pr(y = 1 Z = z + 1) pr(y = 1 Z = z) pr(d = 1 Z = z + 1) pr(d = 1 Z = z). (7.1) To summarize the effect of epidural analgesia on the probability of CS, we want to combine the estimates for θ z into an overall estimate for θ. One possibility is to estimate θ z separately for each time period and combine the results by taking a weighted average. The difficulty is that the estimates for each time period z are not independent, making it difficult to determine the optimal weights. Instead we formulate the following model and compute the maximum-likelihood estimate when θ z = θ. Define the marginal probabilities, pr(y = 1 Z = 1) = γ and pr(d = 1 Z = z) = α z,sopr(y = 1 Z = z) = γ + θ(α z α 1 ).

8 390 S. G. BAKER AND K. S. LINDEMAN Including a nuisance parameter β z for one of the joint probabilities of Y and D, we can write the joint probabilities of outcome and analgesia as The kernel of the likelihood is pr(y = 1, D = 1 Z = z) = β z, pr(y = 0, D = 1 Z = z) = α z β z, pr(y = 1, D = 0 Z = z) = γ + θ(α z α 1 ) β z, pr(y = 0, D = 0 Z = z) = 1 γ θ(α z α 1 ) α z β z. L = k (1 γ θ(α z α 1 ) α z β z ) n z00 z=0 (γ + θ(α z α 1 ) β z ) n z01 (α z β z ) n z10 β n z11 z, (7.2) which we maximized using a Newton Raphson algorithm. For the data in Dailey (1999), we obtained θ = with a standard error of In order to investigate whether the effect of epidural on the probability of CS changed over time, we also fit a model with θ z. There was no evidence to reject the more parsimonious model: the deviance increased by 14.8 on nine degrees of freedom, ( p = 0.10), and { θ z }={ 0.11, 0.15, 0.03, 0.04, 0.01, 0.02, 0.00, 0.03, 0.03} indicated no trend over time. As expected, fitting the model to multiple time periods reduced the standard error. With five time periods of 2 years each, the estimated effect was with a standard error of With two time periods, it was with a standard error of COMBINING ESTIMATES OVER ALL STUDIES In ten studies we used the formulation for two time periods because either (i) the availability of epidural analgesia changed suddenly (Johnson and Rosenfeld, 1995; Lyon et al., 1997; Gribble and Meier, 1991; Larsen, 1992; Fogel et al., 1998), (ii) only data from two time periods were reported (Robson et al., 1993; Frolich et al., 1997; Yancy et al., 1999), or (iii) only marginal data {n z+y, n zd+ } were reported (Mancuso, 1993; Newman et al., 1995). As discussed previously, in the largest study by (Dailey, 1999), we had the necessary data to use the formulation for multiple time periods. To combine the estimates from each study (see Table 1 and Figure 1), we used a weighted sum based on a random-effects model (Dersimonian and Laird, 1986) and a paired permutation distribution, treating the time periods as in the observed order or reversed. For each estimate, let p denote the probability of the same order and hence the same estimate, and 1 p the probability of the reverse order and hence negative of the estimate. Under the null hypothesis, p = 0.5. Applying the computational technique of Follman and Proschan (1999), we inverted hypothesis testing to obtain a confidence interval. Our estimate was 0.00 with a 95% confidence interval of ( 0.06, 0.05). 9. MULTIVARIATE ADJUSTMENT FOR CONCURRENT CONTROLS An alternative method for estimating the effect of epidural analgesia on the probability of CS is a multivariate adjustment for concurrent controls. Let S = 1 denote the subjects who received epidural and S = 0 denote those who did not. We include S to indicate that different subjects received D = 1 and D = 0. A comparison is based on pr(y = 1 D = 1, S = 1,w) pr(y = 1 D = 0, S = 0,w) where, ideally, given risk factors w, there are no unknown risk factors with different distributions for D = 0 and D = 1. The best published multivariate adjustment for studying the effect of epidural analgesia on the

9 Rethinking historical controls 391 Paired Availability Design Meta-Analysis of Randomized Trials study 95% CI size difference in epidural rates trial 95% CI Johnson % Muir1 Frolich % Thorp Lyon % Bofill Robson % Philipson Gribble % Muir2 Larsen % Clark Fogel % Sharma2 Yancy % Loughnan Mancuso % Newman Sharma % Dailey % Gambling summary summary size 50 50% 93 96% % % % % % % % % Fig. 1. Paired availability design and meta-analysis of randomized trials. difference in epidural rates probability of CS in observational data was a propensity score analysis of 1733 subjects by Lieberman et al. (1996). This study considered many more variables than any other observational study of epidural analgesia and CS: maternal age, race, insurance, pre-pregnant weight, height, birth weight, gestational duration, dilation at admission, initial rate of cervical dilation, station of fetal head at admission, active management, ruptured membrane at admission, maternal chronic hypertension, and maternal pregnancy hypertension. Unlike many of the other observational studies, Lieberman et al. (1996) did not include oxytocin as a predictor of the probability of CS. Because epidural analgesia likely affects the utilization of oxytocin, oxytocin is an outcome variable and thus inappropriate as a baseline predictor. To convert the results in Lieberman et al. (1996) from an odds ratio to a difference, we applied the method of subclassification of propensity scores (Rosenbaun and Rubin, 1984) to data from Figure 2 in Lieberman et al. (1996). With this approach we estimated that epidural analgesia increases the probability of CS by 0.10 with 95% confidence interval of (0.07, 0.13). Unlike the paired availability analysis, the multivariate adjustment for concurrent controls indicated that epidural analgesia has a large effect on the probability of CS. We believe the reason for this discrepancy is that the multivariate analysis omitted an important predictor. According to Danilenko- Dixon and Van Winter (1997),... even the most meticulous statistical methods will be unable to recognize, quantify, and account for all potential confounding variables. For instance, potential confounders that are very difficult to measure or impossible to quantify and control include the parturient s perception of and physiological response to pain and the size, shape, and degree of laxity of the maternal pelvis especially in relation to the size, shape, presentation, and station of the fetal head, both during the onset of labor and with respect to changes occurring the progression of labor. Lieberman et al. (1996) admitted we cannot rule out the possibility of residual confounding by an unmeasured and uncontrolled factor. For example we did not have information regarding pelvic size. However, Lieberman et al. (1996) argued that for some other factor not controlled in our analysis to be responsible for the association we have noted, it would have to be very strongly associated with epidural analgesia use and cesarean delivery. There are no obvious candidates apart from the factors we have measured.... The association of such a factor with epidural analgesia and cesarean delivery would need to be at least as large as the association

10 392 S. G. BAKER AND K. S. LINDEMAN Table 4. Randomized trials. Studies ordered from smallest to largest sample size Numbers Proportions Study Size Epidural CS Epidural CS Estimate (s.e.) Muir et al. (1994) control (0.132) study Thorp et al. (1993) control (0.069) study Bofill et al. (1997) control (0.071) study Philipsen and Jensen (1989) control (0.066) study Muir et al. (2000) control (0.046) study Clark et al. (1998) control (0.068) study Sharma et al. (2000) control (0.031) study Loughnan et al. (2000) control (0.010) study Sharma et al. (1997) control (0.018) study Gambling et al. (1998) control (0.029) study Compliance not reported in the abstract, so full compliance was assumed. we observed between epidural analgesia and cesarean delivery for it to be a competing explanation for our findings. We believe that intense pain early in labor could be such a factor. Women with greater pain early in labor are more likely to request epidural analgesia and are also at higher risk for operative delivery (National Institute of Child Health and Human Development, 1981; Wuitchik et al., 1989; Hess et al., 2000), which would increase the apparent effect the effect of epidural analgesia on the probability of CS (Rosenbaum, 1989). 10. VALIDATION: AMETA-ANALYSIS OF RANDOMIZED TRIALS Our gold standard for evaluation is the randomized trial. We identified ten randomized controlled trials of epidural analgesia since 1980 which reported data on CS rates in each randomized group, either in an article or an abstract (Table 4). We dropped from our meta-analysis the study by Nikkola et al. (1997) because there were no cases of CS in either arm, which made it difficult to include in the calculations. Because Nikkola et al. (1997) randomized only 20 patients, we believe the information loss was minimal. In this setting, let Z = 0 if a subject were randomized to no epidural, and Z = 1 if randomized to epidural. The intent-to-treat estimate is pr(y = 1 Z = 1) pr(y = 1 Z = 0). By virtue of the randomization, the distribution of unknown risk factors is the same in the two groups. In some trials many subjects randomized to the epidural group did not receive epidural analgesia, and many randomized to the control group received epidural analgesia. Previous meta-analyses were based on intent-to-treat (Halpern et al., 1998; Howell, 1999; Zhang et al., 1999). In the presence of noncompliance, analysis by intent-to-treat estimates the effect of assignment to the epidural group, not quantity of interest, the effect of epidural analgesia. To adjust for this noncompliance, we used the same framework as with the paired

11 Rethinking historical controls 393 Table 5. Effect of epidural analgesia on the probability of CS Estimate 95% confidence interval Paired availability design 0.00 ( 0.06, 0.05) Meta-analysis of randomized trials 0.02 ( 0.02, 0.08) Multivariate adjustment of concurrent controls 0.10 (0.07, 0.13) availability design. This approach is similar to that in Angrist et al. (1996). For each study, we obtained an estimate using (4.1) (see Figure 1 and Table 4). Randomization ensures that Assumptions 1 and 2 hold. Assumption 3 holds for the same reason as with the paired availability design. In Loughnan et al. (2000), a small number of subjects received both opioid and epidural analgesia, and it is necessary to also assume no interactive effect on the probability of CS. Under randomization, Assumption 4 says no subject would receive epidural if randomized to no epidural and receive no epidural if randomized to epidural, for which we believe there would be few exceptions. Our summary estimate, based on a random effects meta-analysis with a paired permutation distribution (Follman and Proschan, 1999), was 0.02 with a 95% confidence interval of ( 0.02, 0.08). The estimate and confidence interval were similar to those from the paired availability design, providing strong support for the validity of inference based on the paired availability design. 11. DISCUSSION As with traditional historical controls, the paired availability design assumes no changes over time which could confound the effect of treatment. We considered three types of temporal assumptions. Assumption 1 (stable care) ensures one is measuring the effect of treatment. Assumption 2 (stable population) ensures the same distribution of unknown covariates. Assumption 4 (stable preference), which is not often considered, ensures no new information which would alter the type of subject selecting a treatment. There was no evidence of any gross violation of these assumptions. Traditional historical controls must also assume that subjects who receive the old treatment in the previous time period have the same distribution of unknown covariates as subjects who receive new treatment current time period. As a much more plausible alternative, the paired availability design requires Assumption 3 (no effect of treatment timing), which makes it possible to peel away the effect of treatment in always- and neverreceivers. By combining the results of multiple studies, the paired availability design dampens the effect of random violations of the assumptions. The striking similarity between the confidence intervals based on the meta-analysis of randomized trials and the paired availability design, particularly in comparison with confidence interval from the multivariate adjustment for concurrent controls (Table 5), provides strong support that the paired availability design reduces selection bias. In interpreting results from all the approaches, one should realize that it was not possible to blind the obstetricians to the type of analgesia. An obstetrician who knew that a patient received epidural analgesia might have been more likely to perform a CS because of (i) a preconceived notion that epidural analgesia increases the probability of CS or (ii) a perception that the patient was ready for anesthesia and surgery for CS by the presence of the epidural catheter. The lack of blinding would most likely have increased the apparent association between epidural analgesia and the probability of CS. Because we found only a small effect of epidural analgesia on the probability of CS, we do not believe the lack of blinding had much, if any, noticeable effect. As one referee pointed out, the key idea is that the availability of treatment may vary with time in ways that have little to do with the characteristics of the patients, so it is better to base the analysis of historical controls on changes in availability of treatment rather than on changes in receipt of treatment

12 394 S. G. BAKER AND K. S. LINDEMAN as is traditionally done. Another referee noted that because differences in availability are more plausibly like a coin toss compared to differences in the actual taking of treatments, one could use multivariate adjustments, such as propensity scores, to better estimate the effect of changes in availability of treatment, and then make additional adjustments to estimate the effect of receipt of treatment. We agree that it best to adjust for observed covariates, if they are available and note that, alternatively, one could directly incorporate covariates into the likelihood in (7.2). However, in this application, the additional covariate data were not generally available and would be difficult or impossible to obtain in many of the studies. Our findings of a small effect of epidural analgesia on the probability of CS are clinically important because a small increase in the probability of CS would likely deter few women from selecting the most effective form of pain relief for labor. According to Chestnut (1997), Even if it were shown that the contemporary use of epidural analgesia results in a small increase in the CS rate, it is unclear how many women who now choose epidural analgesia would voluntarily opt for nothing or intravenous opioid analgesia, which is less effective and provides marked sedation. It is also economically important because some insurance companies have considered withholding reimbursement for epidural analgesia because of perceived high probability of CS (Chestnut, 1997). ACKNOWLEDGEMENTS The authors thank two anonymous referees for their insights, Patricia Daily for invaluable data from the Mills-Pennisula hospital, and Barbara Leighton and Steven Segal for helpful comments and assistance in locating appropriate studies. REFERENCES ANGRIST, J. D., IMBENS, G. W. AND RUBIN, D. B. (1996). Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association 91, BAKER, S. G. AND LINDEMAN, K. S. (1994). The paired availability design: a proposal for evaluating epidural analgesia during labor. Statistics in Medicine 13, BAKER, S. G. (1997). Compliance, all-or-none. In Kotz, S., Read, C. R. and Banks, D. L. (eds), The Encyclopedia of Statistical Science, Update Vol. 1. New York: Wiley, pp BAKER, S. G. (1998). The paired availability design: an update. In Abel, U. and Koch, A. (eds), Nonrandomized Comparative Clinical Studies, Dusseldorf: Symposion, pp BIRNBACH, D. J., GRUNEBAUM, A., STEIN, D. J., KARGACIN, B., KURODA, M. M. AND THIS, D. M. (1997). Does epidural analgesia protect against Cesarean section in nulliparous patients? A retrospective review of patients, Abstract (Presented at the Annual Meeting of Society of Obstetric Anesthesiology and Perinatology) (Hamilton, Bermuda). BOFILL, J. A., VINCENT, R. D., ROSS, E. L., MARTIN, R.W.,NORMAN, P.F.,WERHAN, C.F.AND MORRISON, J. C. (1997). Nulliparous active labor, epidural analgesia, and cesarean delivery for dystocia. American Journal of Obstetrics and Gynecology 177, BYAR, D. (1980). Why data bases should not replace randomized clinical trials. Biometrics 36, CHESTNUT, D. H. (1997). Epidural analgesia and the incidence of Cesarean section. Time for another close look. Anesthesiology 87, CHESTNUT, D. H., MCGRATH, J. M., VINCENT, R. D., PENNING, D. H., CHOI, W.W.,BATES, J.N. AND MCFARELANE, C. (1994). Does early administration of epidural analgesia affect obstetric outcome in nulliparous women who are in spontaneous labor? Anesthesiology 80,

13 Rethinking historical controls 395 CLARK, A., CARR, D., LOYD, G., COOK, V. AND SPINNATO, J. (1998). The influence of epidural analgesia on cesarean delivery rates: a randomized, prospective clinical trial. American Journal of Obstetrics and Gynecology 179, DAILEY, P. (1999). Personal communication. DANILENKO-DIXON, D. AND VAN WINTER, J. T. (1997). The effect of epidural analgesia on labour and delivery. Postgraduate Obstetrics and Gynecology 17, 1 8. DERSIMONIAN, R. AND LAIRD, N. M. (1986). Meta analysis of clinical trials. Controlled Clinical Trials 7, FOLLMAN, D. A. AND PROSCHAN, M. A. (1999). Inference in random effects meta-analysis. Biometrics 55, FOGEL, S., SHYKEN, J. M., LEIGHTON, B. L., MORMOL, J. S. AND SMELTZER, J. (1998). Epidural labor analgesia and the incidence of cesarean delivery for dystocia. Anesthesia and Analgesia 87, FROLICH, M. A., ORTH, V., KNITZA, R., FINSTERER, U., HEPP, H. AND PETER, K. (1997). Does epidural analgesia reduce the incidence of operative delivery? Abstract (Presented at the Annual Meeting of Society of Obstetric Anesthesiology and Perinatology) (Hamilton, Bermuda). GAMBLING, D.R.,SHARMA, S.K.,RAMIN, S. M., LUCAS, M.J.,LEVENO, K.J.,WILEY, J.AND SIDAWI, J. E. (1998). A randomized study of combined spinal-epidural analgesia versus intravenous meperidine during labor. Anesthesiology 89, GRIBBLE, R. K. AND MEIER, P. R. (1991). Effect of epidural analgesia on the primary Cesarean rate. Obstetrics and Gynecology 78, HALPERN, S., LEIGHTON, B. L., OHISSON, A., BARRETT, J. F. R. AND RICE, A. (1998). Effect of epidural verses parenteral opioid analgesia on the progress of labor. A meta-analysis. Journal of the American Medical Association 289, HAWKINS, J.L.,GIBBS, C.P.,ORLEANS, M.,MARTIN-SALVAJ, G.AND BEATY, B. (1997). Obstetric anesthesia work force survey, 1981 versus Anesthesiology 87, HESS, P. E., PRATT, S. E., SONI, A., SARNA, M. C. AND ORIOL, N. E. (2000). An association between severe labor pain and Cesarean delivery. Obstetric Anesthesia 90, HOWELL, H. J. (1999) Epidural versus non-epidural analgesia for pain relief in labour, Cochrane Review Oxford: The Cochrane Library, update software. JOHNSON, S.AND ROSENFELD, J. A. (1995). The effect of epidural anesthesia on the length of labor. The Journal of Family Practice 40, LARSEN, D. D. (1992). The effect of initiating an obstetric anesthesiology service on rate of Cesarean section and rate of forceps delivery. Abstract (Presented at the Annual Meeting of the Society of Obstetric Anesthesia and Perinatology). LIEBERMAN, E. (1999). No free lunch on labor day. The risks and benefits of epidural analgesia during labor. Journal of Nurse-Midwifery 44, LIEBERMAN, E., LANG, J., COHEN, A., D AGOSTINO, R., DATTA, S. AND FRIGOLETTO, F. (1996). Association of epidural analgesia with cesarean delivery in nulliparas. Obstetrics and Gynecology 88, LOUGHNAN, R. A., CARLI, F., ROMNEY, M., DORE, C. J. AND GORDON, H. (2000). Randomized controlled comparison of epidural bupivicaine versus pethidine for analgesia in labor. British Journal of Anesthesia 84, LYON, D., KNUCKLES, G., WHITAKER, E. AND SALGADO, S. (1997). The effect of instituting an elective labor epidural program on the operative delivery rate. Obstetrics and Gynecology 90, MANCUSO, J. J. (1993). Epidural analgesia in an army medical center: impact on Cesareans and instrumental

14 396 S. G. BAKER AND K. S. LINDEMAN deliveries, Abstract 13 (Presented at the Annual Meeting of the Society for Obstetric Anesthesiology and Perinatology) (Palm Springs). MUIR, H. A., SHULKLA, R., LISTON, R. AND WRITER, D. (1996). Randomized trial of labor analgesia: a pilot study to compare patient-controlled intravenous analgesia with patient-controlled epidural analgesia to determine if analgesic method affects delivery outcome. Abstract. Canadian Journal of Anaesthesia 43, A60. MUIR, H., BREEN, T.,CAMPBELL, D., HALPERN, S., LISTON, R.AND BLANCHARD, W. (2000). A multi-center study of the effect of analgesia on the progress of labor, Abstract A 23 (Presented at the Annual Meeting of the Society of Obstetric Anesthesiology and Perinatology) Montreal Canada. NATIONAL INSTITUTE OF CHILD HEALTH AND HUMAN DEVELOPMENT (1981). Report No Bethesda, MA: NICHD. NEWMAN, L. M., PEREZ, E. C., KROLICK, T. J. AND IVANKOVICH, A. D. (1995). Labor analgesia, Cesarean anesthesia, and cesarean delivery rates for deliveries from 1988 through Abstract. Anesthesiology 83, 3A, A967. NIKKOLA, E. M., EKBLAD, U. U., KERO, O. O., ALIHANKA, J.J.M.,MARKKU, A.O.AND SALONEN, M. A. O. (1997). Intravenous fentanyl PCA during labour. Canadian Journal of Anaesthesia 44, PHILIPSEN, T.AND JENSEN, N. (1989). Epidural block or parenteral pethidine as analgesia in labour; a randomized study concerning progress in labour and instrumental deliveries. European Journal of Obstetrics and Gynecology and Reproductive Biology 30, POCOCK, S. J. (1983). Clincial trials. A Practical Approach. Chichester: Wiley. ROBSON, M., BOYLAN, P., MCPARLAND, P., MCQUILLAN, C. AND O NEILL, M. (1993). Epidural analgesia need not influence the spontaneous vaginal delivery rate. American Journal of Obstetrics and Gynecology January, Abstract 240, 364. ROSENBAUN, P.R.AND RUBIN, D. R. (1984). Reducing bias in observational studies using subclassification of the propensity score. Journal of the American Statistical Association 79, ROSENBAUM, P. R. (1989). The role of known effects in observational studies. Biometrics 45, SHARMA, S. K., SIDAWI, J. E., RAMIN, S. M., LUCAS, M.J.,LEVNO, K.J.AND CUNNINGHAM, G. (1997). Cesarean delivery: a randomized trial of epidural versus patient-controlled meperidine analgesia during labor. Anesthesiology 87, SHARMA, S. K., LEVENO, K. J., MESSICK, G., ALEXANDER, J. M., SIDAWI, J. E. AND WILEY, J. (2000). A randomized trial of patient-controlled epidural versus patient-controlled intravenous analgesia during labor. Abstract A 22 (Presented at the Annual Meeting of the Society of Obstetric Anesthesia and Perinatology) (Montreal, Canada). THORP, J.A.,HU, D.H.,ALBIN, R. M., MCNITT, J., MEYER, B.A.,COHEN, G.AND YEAST, J. D. (1993). The effect of intrapartum epidural analgesia on nulliparous labor: a randomized, controlled, prospective trial. American Journal of Obstetrics and Gynecology 169, WUITCHIK, M., BAKAL, D. AND LIPSHITZ, J. (1989). The clinical significance of pain and cognitive activity in latent labor. Obstetrics and Gynecology 73, YANCY, M. K., PIERCE, B., SCHWEITZER, D. AND DANIELS, D. (1999). Observations on labor epidural analgesia and operative delivery rates. American Journal of Obstetrics and Gynecology 180, ZHANG, J., KLEBANOFF, M. AND DERSIMONIAN, R. (1999). Epidural analgesia in association with duration of labor and mode of delivery: a quantitative review. American Journal of Obstetrics and Gynecology 180, [Received 30 June, 2000; revised 11 September, 2000; accepted for publication 28 September, 2000]