The article by Stamou and colleagues [1] found that

Transcription

1 THE STATISTICIAN S PAGE Propensity Score Analysis of Stroke After Off-Pump Coronary Artery Bypass Grafting Gary L. Grunkemeier, PhD, Nicola Payne, MPhiL, Ruyun Jin, MD, and John R. Handy, Jr, MD Providence Health System, Portland, Oregon The article by Stamou and colleagues [1] found that the stroke rate after coronary artery bypass grafting (CABG) was lower for off-pump CABG than for conventional, on-pump CABG. Postoperative stroke occurred in See also page % of on-pump patients and only 1.2% of off-pump CABG patients, for an (unadjusted) odds ratio (OR) of 2.1 for on-pump versus off-pump CABG (see Appendix for definition of OR). Because the two patient groups were not similar with respect to potential risk factors for stroke, they used a logistic regression, which produced an adjusted OR of 1.6. As an adjunctive analysis, they computed a propensity score and used it to find on-pump matches for 72% of the off-pump CABG patients. For these matched subsets the OR for postoperative stroke was 1.8. This expository article briefly describes propensity scores and demonstrates another way of using them to compare treatments. We apply this method to another data set, and in the process corroborate the findings of Stamou and colleagues. Propensity Score: The Concept Propensity score analysis is about 20 years old, but has only recently begun to be used to any extent in medical research (Fig 1). One of the originators of this method recently wrote an overview article for physicians [2], and another recent review by Eugene Blackstone [3] provides a thorough discussion of this method. Randomized studies are considered the highest level of evidence for comparing a treatment with a control. The essential feature of such studies is that some random mechanism controls treatment assignment, therefore each patient has the same probability of receiving the treatment, thus protecting against a biased comparison due to patient selection. Many factors influence which treatment patients receive in a nonrandomized observational study. Physicians usually select the treatment for a particular patient based on his or her characteristics (age, severity of disease, type of anatomy, and so on). If we knew this Address reprint requests to Dr Grunkemeier, 9155 SW Barnes Rd, #33, Portland, OR 97225; ggrunkemeier@providence.org. physician-specific selection rule, and had available the characteristics of each patient that went into this decision, we could compute the probability of his or her getting one of the two treatments. Suppose we did not know the selection rule, but had available data on a group of patients, including which treatment they received. Then we could approximate the rule by producing a risk model. We regularly use logistic regression to produce risk models that use patient characteristics to predict the probability of postoperative events such as death (although we really know whether each patient in the study actually died or not). The propensity score is just the result of a multivariable risk model for the event treatment. For each patient it provides the probability he or she received the treatment (although we already know that, too). In comparing on-pump versus off-pump CABG, we will pattern our model on the one by Stamou and colleagues and make on-pump the treatment and offpump CAB the control. Then, the propensity score is simply the probability that a patient, given his or her particular set of characteristics, received on-pump CABG. A group of patients with the same propensity score are thus equally likely to have been assigned to the on-pump treatment. And, therefore, they were also equally likely to have been assigned to off-pump CABG. In fact, some of them got on-pump and some off-pump CABG, just as if they had been randomly allocated to Fig 1. The number of papers from a Medline search using the text string propensity score according to the year of publication. This term has had a recent increase, although it is still not widely used by The Society of Thoracic Surgeons Ann Thorac Surg 2002;74: /02/$22.00 Published by Elsevier Science Inc PII S (02)

2 302 STATISTICIAN S PAGE GRUNKEMEIER ET AL Ann Thorac Surg PROPENSITY SCORE ANALYSIS 2002;74: and remove more than 90% of the bias due to each of the covariates [5]. Fig 2. Off-pump coronary artery bypass grafting (OPCAB) utilization by hospital. The percentage of off-pump coronary artery bypass graftings from 1988 to 2000 varied greatly among the nine hospitals. The total number of operations is shown below each bar. Hospitals with larger numbers of cases tended to have lower off-pump coronary artery bypass grafting usage. Hospitals were put into four groups depending on off-pump coronary artery bypass grafting usage; the low utilization hospitals ( 10% off-pump coronary artery bypass grafting) were not used for the comparison of off-pump coronary artery bypass grafting to on-pump results. whichever treatment they actually received. We might think of them as randomized after the fact. Matching Propensity scores can be used in different ways in the analysis. Stamou and colleagues used matching [4]. They had a large number of on-pump CABG patients (78%) and a smaller number of off-pump CABG patients (22%). They used logistic regression analysis to calculate a propensity score, and then for each off-pump CABG patient they found the on-pump patient with the closest propensity score. They were able to find matches for 72% of the off-pump CABG patients, and then used a second logistic regression to compute the OR for stroke between these two matched samples. Providence Off-Pump CABG Experience Providence Health System includes nine cardiac surgery programs in four western states that submit data to a common database and meet regularly to discuss quality improvement initiatives. During 1998 to 2000, 7,955 isolated CABG (no previous or concomitant valve replacement) were performed, 15% of which were off-pump CABG. Interestingly, the use of off-pump CABG ranged from 1% to 72% among the nine hospitals (Fig 2). This wide variation in off-pump CABG usage justifies continued research inquiries into the relative efficacy of these two surgical approaches. This variation also makes hospital one of the most important risk factors for offpump CABG. There was a natural grouping of hospitals into four groups according to off-pump CABG usage: low, moderate, medium, and high (Fig 2), and the (unadjusted) stroke rates were lower for off-pump CABG in each group (Fig 3). Overall, the stroke rates were 2.2% versus 1.2% (arrows in Fig 3), for an unadjusted OR (95% confidence interval) of 2.0 (1.1, 3.5), almost identical to that of Stamou and colleagues. To compute a propensity score for on-pump CABG, we excluded hospitals in the low utilization category, and performed a logistic regression to determine risk factors for on-pump in the other three usage groups. The unadjusted OR for postoperative stroke was 2.1 in this subset of patients (Appendix). Table 1 contains a brief description of the characteristics of the patients. The propensity score model included hospital, younger age, urgency, prior CABG, and number of grafts, all highly significant. This simple model was primarily for demonstration purposes. To compute a final model, nonsignificant (p 0.50) Stratification Stratification is another way to use the propensity score to adjust comparisons and reduce bias [5]. It has the advantage of potentially using all of the patients. (Stamou and associates were only able to enroll 32% of all patients using the matching technique.) With this method, the propensity scores are consecutively ordered (both groups together), and divided into groups, usually five, of equal size. The result is that the patients in each quintile all have similar probabilities of receiving the treatment. It has been shown that by dividing the propensity scores into quintiles, it is possible to achieve balance between treatment groups within each quintile, Fig 3. Percentage of postoperative stroke by off-pump coronary artery bypass grafting (OPCAB) versus on-pump for hospitals with low, moderate, medium, and high off-pump coronary artery bypass grafting usage. In each group, the stroke percentage was lower for patients with off-pump coronary artery bypass grafting. The arrows indicate the mean values for all hospitals: 2.2% for on-pump and 1.1% for off-pump coronary artery bypass grafting. (CVA cerebrovascular accident; Post-op postoperative.)

3 Ann Thorac Surg STATISTICIAN S PAGE GRUNKEMEIER ET AL 2002;74: PROPENSITY SCORE ANALYSIS 303 Table 1. Comparison of Off-Pump Coronary Artery Bypass Grafting and On-Pump Populations in the Moderate to High Off-Pump Coronary Artery Bypass Grafting Usage Hospitals Characteristics Treatment Group [S] Off-pump CABG On-pump Patients 990 (32%) 2,120 (68%) Mean age (yr) [S, P] Female gender [S] 28.5% 26.2 Prior events Stroke [S] 8.3% 7.7% Coronary surgery [P] 5.8% 9.1% Urgency [P] Elective 67% 63% Urgent 29% 29% Emergency 4% 8% Number of grafts [P] 1 12% 3% 2 23% 12% 3 32% 31% 4 25% 33% 5 7% 14% 6 1% 7% Hospital Off-pump CABG usage [P] Moderate 18% 82% Medium 45% 55% High 72% 28% CABG coronary artery bypass grafting; P variables used to create the propensity score, from a logistic regression for the outcome onpump operation; S variables used in a standard logistic regression for the outcome postoperative stroke. variables can be used [6], and other refinements made [3]. Nevertheless, the c-statistic (the area under the receiver operating charateristics curve) of 0.84 indicated quite good discrimination, not surprising considering the help from the hospital variable. The propensity scores for the off-pump CABG patients were in general lower than for the on-pump patients, but with much overlap (Fig 4). The quintile groups had more comparable scores (Fig 5), but at the expense of having unbalanced numbers between treatments within each strata, favoring off-pump CABG in the lower strata and heavily favoring on-pump in the upper strata. The stroke rate was lower in the off-pump CABG groups in each of the first three quintiles, but there were too few patients in the upper two quintiles for an adequate comparison. This is the penalty for achieving the balance in propensity scores. Nominally, the common OR from the stratified results, using the Mantel-Haenszel method [7] was 2.7 (1.3, 5.8); however, a test of homogeneity of OR among the strata (barely) failed. Fig 4. Boxplots of the propensity scores for off-pump coronary artery bypass grafting (OPCAB) and on-pump patients, excluding the low usage hospitals. For each distribution, the interquartile range (the values for the 25th and 75th percentiles) form the lower and upper edges of the box. The horizontal line within the box indicates the median (50th percentile) and the outer horizontal lines at the end of the vertical lines indicate the maximum and minimum. The number of patients in each group is shown below the horizontal axis. age, female gender, prior stroke, and on-pump CABG had a c-statistic of 0.74, indicating fair discrimination. The OR for on-pump in this model was 2.7 (1.3, 5.6), which compared nicely with the pooled value from the propensity analysis. This adjusted OR was higher than the raw OR, because off-pump CABG patients had higher percentages of the other risk factors in the model (Table 1). Hospital was not significant in this model, and using all hospitals, including low off-pump CABG usage, produced virtually the same regression model. A third way of using propensity score is to use it in a logistic regression, either by itself or in conjunction with Conventional Logistic Regression for Stroke As did Stamou and colleagues, we performed logistic regression analysis with postoperative stroke as the outcome. A model with the significant risk factors of older Fig 5. Boxplots of the propensity scores for off-pump coronary artery bypass grafting (OPCAB) and on-pump patients, excluding the low usage hospitals, grouped according to quintile of propensity score. There is fairly close agreement within each quintile. The number of patients in each group is shown below the horizontal axis.

4 304 STATISTICIAN S PAGE GRUNKEMEIER ET AL Ann Thorac Surg PROPENSITY SCORE ANALYSIS 2002;74: Table 2. Comparison of Odds and Probabilities (Using Data From Providence Health System Cardiac Surgery Programs With Moderate to High Off-Pump CABG Usage) Characteristics Postoperative Stroke No Yes Total Stroke Probability Stroke Odds Off-pump CABG On-pump 2, , Ratio (On-pump/off-pump CABG) RR OR CABG coronary artery bypass grafting; OR odds ratio; RR relative risk; a model such as the one above. The propensity score was not a significant addition to the above model and forcing it into the model changed the OR for off-pump CABG only slightly, to 2.6. Comment The propensity score methodology shares the limitations of all risk models. It can only account for factors that are available. To be successful may require many variables; one of the original examples used up to 74 variables to produce the score [5]. The result of the regression is only an estimate of the propensity, not the true propensity itself. And at best, propensity scores are only approximations of the real thing, randomization. Stamou and associates used propensity scores to produce matched samples, but could not use all of the patients; the matched samples included only 32% of the patients. We excluded patients operated on in low offpump CABG usage hospitals, and produced quintiles using the others. But two of the quintiles had numbers too small in one of the groups to make satisfactory comparisons. Conventional logistic regression analysis produced a more satisfying result, and all of the patients could be used. Propensity score analysis was recently used in two articles comparing valve repair versus replacement for ischemic mitral valve disease. One of the articles was able to use it successfully [8], whereas the authors of the other article tried to but were unable to find satisfactory matches [9]. This prompted an editorial comment by Craig Miller in the same issue, questioning this apparent inconsistency [10]. Propensity score analysis may or may not be helpful in a particular comparison study. A recent review of the topic concluded: Finally, it is important to note that we are not advocating the use of only propensity scores in analyses of observational studies, rather we are encouraging the use of propensity scores in addition to traditional methods of analysis [6]. Stamou and associates used propensity scores and traditional methods that both yielded a significant OR for postoperative stroke in favor of off-pump CABG, and our analysis of an independent data set did the same. Providence Health System provided data from the cardiac surgery programs at: Providence Anchorage Medical Center, Alaska; Providence Everett Medical Center, Washington; Providence Campus, Swedish Medical Center, Seattle, Washington; Providence St. Peter Hospital, Olympia, Washington; Providence Yakima Medical Center, Yakima, Washington; Providence Portland Medical Center, Portland, Oregon; Providence St. Vincent Medical Center, Portland, Oregon; Providence St. Joseph Medical Center, Burbank, California; Providence Holy Cross Medical Center, Mission Hills, California. We also thank Vicki Anderson for managing the database. References 1. Stamou SC, Jablonski KA, Pfister AJ, et al. Stroke after conventional versus minimally invasive coronary artery bypass. Ann Thorac Surg 2002;74: Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med 1997;127: Blackstone EH. Comparing apples and oranges. J Thorac Cardiovasc Surg 2002;123: Rosenbaum PR, Rubin DB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. AmerStat 1985;39: Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. JASA 1984;79: D Agostino RB Jr. Tutorial in biostatistics: propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998;17: Lachin JM. Biostatistical methods: the assessment of relative risks. New York: John Wiley & Sons, Gillinov AM, Wierup PN, Blackstone EH, et al. Is repair preferable to replacement for ischemic mitral regurgitation? J Thorac Cardiovasc Surg 2001;122: Grossi EA, Goldberg JD, LaPietra A, et al. Ischemic mitral valve reconstruction and replacement: comparison of longterm survival and complications. J Thorac Cardiovasc Surg 2001;122: Miller DC. Ischemic mitral regurgitation redux to repair or to replace? J Thorac Cardiovasc Surg 2001;122: Appendix Probabilities and Relative Risks Probabilities are computed as the number of patients who had an event divided by the total number of patients. For example, in the moderate-to-high off-pump CABG usage groups, 49 of 2,120 on-pump patients had a postoperative stroke (Table 2). Thus, the probability is 49/ 2, (this is an approximate equality, as the

5 Ann Thorac Surg STATISTICIAN S PAGE GRUNKEMEIER ET AL 2002;74: PROPENSITY SCORE ANALYSIS 305 answer is rounded off to 4 decimal places). Multiplying this by 100 converts it to a percentage of about 2.3%. The ratio of two probabilities is called a risk ratio or relative risk (RR). Odds and Odds Ratios Odds are the number of patients who had an event divided by the number of patients who did not have the event. For the on-pump Providence Health System patients this is 49/2, , slightly larger than the probability. For the off-pump CABG patients it is 11/ Odds are always larger than probabilities, but when the probabilities are close to zero, as above, the difference between them is small. The ratio of two odds is called an odds ratio (OR). The (unadjusted) OR for stroke comparing Providence Health System on-pump to off-pump CABG is / (Table 2), and the RR is Odds are similar to probabilities for low-risk events (such as postoperative stroke), and are often used because the adjusted ORs can be determined from logistic regression coefficients. Logistic Regression Logistic regression is a method of incorporating multiple risk factors into a model to estimate a binary outcome, in our case stroke. The output consists of coefficients for each risk factor: positive means a risk (as for age, usually) and negative means a protective factor (as for ejection fraction, usually). Taking the exponential function of a coefficient gives the (adjusted) OR for that factor. If it is a risk factor, the OR will be greater than one, and if it is protective the OR will be less than one. A logistic regression for postoperative stroke with the single risk factor on-pump would yield the raw OR for on-pump given in Table 2. Confidence Intervals There is, in general, a duality between confidence intervals and p values. If the 95% confidence interval for an OR does not contain the value one, then that OR is significantly different from one at the 5% level (p 0.05). Conversely, if the 95% confidence interval for an OR does contain the value one, then that OR is not significantly different from one (p 0.05). Confidence intervals, however, are more informative than p values; they not only indicate the statistical significance, but also the clinical range of values. This is true for both significant and nonsignificant results.