Approximately 20% of patients with non small cell

Transcription

1 Estimating Lung Resection Risk: A Pilot Study of Trainee and Practicing Mark K. Ferguson, MD, Jennifer D. Stromberg, BS, and Amy D. Celauro, MS Section of Cardiac and Thoracic Surgery, Department of Surgery, The University of Chicago, Chicago, Illinois Background. Most surgeons believe that experiencebased risk estimates for major lung resection are reliable. Elements that influence such estimates are poorly understood. Methods. Clinical vignettes were created for patients who underwent lung resection; 48 patients who had major complications were matched to 48 patients without complications. Ten senior surgical trainees and 9 practicing thoracic surgeons blinded to outcomes estimated the risk of complications using a seven-point scale (uninformed estimates). After review of a calculated risk score, risk was again estimated (informed estimates). Results. Risk estimates did not differentiate between patient groups with and without complications (4.8 versus 4.9; p 0.94 for trainees; 4.5 versus 4.2; p 0.21 for practicing surgeons). The accuracy of predicting complications was only fair, but was better for practicing surgeons than for trainees (58% versus 51%; p 0.041). Risk estimates correlated moderately well with baseline pulmonary function and possibly with age, but not with performance status or extent of resection. Knowledge of a calculated risk score resulted in more frequent alterations of trainee risk scores, improved interobserver agreement in both groups, and aligned trainee and practicing surgeon estimates more closely. Conclusions. Surgeon estimates are not accurate in predicting lung resection complications using vignettebased, matched-pair methodology. Practicing surgeons and trainees base risk estimates on limited objective clinical data. Trainee estimates are more susceptible to modification by a standard risk score than are estimates of practicing surgeons. Prospective studies are necessary to further explore the etiology, accuracy, and utility of surgeon risk estimates. (Ann Thorac Surg 2010;89: ) 2010 by The Society of Thoracic Accepted for publication Dec 11, Presented at the Fifty-sixth Annual Meeting of the Southern Thoracic Surgical Association, Marco Island, FL, Nov 4 7, Address correspondence to Dr Ferguson, Department of Surgery, The University of Chicago, 5841 S. Maryland Ave, MC 5035, Chicago, IL 60637; mferguso@surgery.bsd.uchicago.edu. Approximately 20% of patients with non small cell lung cancer are treated with major lung resection for potential cure. Perioperative risks associated with major lung resection are not negligible; complications and operative mortality add substantially to the cost of care for these patients, doubling the duration and cost of hospitalization [1]. Efforts have been made to decrease the incidence of adverse events after major resection for lung cancer through risk reduction, improved patient selection, and risk management. Improved patient selection has been facilitated by identification of specific risk factors, generation of management guidelines, and development of specific risk scoring systems [2]. There is limited use of management guidelines and scoring systems in the clinical management of patients with surgically treatable lung cancer, partly because of the belief on the part of many physicians that surgeon judgment about an individual patient s risk is likely to be more accurate than any of these systems. However, the accuracy of physician estimates in this setting is unknown, and elements that influence physician risk estimates are poorly understood. Variable accuracy of physician estimates of risk has been demonstrated in other types of surgery, including vascular, hepatobiliary, and gastrointestinal operations [3 6]. We investigated physician estimates of risk for major lung resection to determine the effects of level of training, which elements of a clinical vignette are related to risk estimates, and how knowledge of an objective risk score influences trainee and practicing surgeons estimates of risk. Material and Methods Patients were selected from a prospectively collected database of individuals who had undergone major lung resection for non small cell lung cancer from 1995 through 2004 and who had not received induction therapy. Fifty patients who had major postoperative complications were paired with closest-matched complicationfree patients from the database based on age, sex, forced expiratory volume in the first second expressed as a percent of predicted, diffusing capacity of the lung for carbon monoxide (Dlco%), extent of resection, and American Joint Commission on Cancer stage [7]. Clinical vignettes comprising three or four short paragraphs were abstracted from these patients records; any personal identifying or outcomes information was eliminated. Each vignette included patient age, sex, and race; a brief history of present illness including symptoms at the time of presentation; past medical history; findings on physical examination; a description of results of tests 2010 by The Society of Thoracic /10/$36.00 Published by Elsevier Inc doi: /j.athoracsur

2 1038 FERGUSON ET AL Ann Thorac Surg SURGEON ESTIMATES OF LUNG RESECTION RISK 2010;89: including computed tomography of the chest and abdomen, positron emission tomography scan, any pertinent cardiovascular evaluation, and pulmonary function studies; and a description of surgical therapy including the lobe(s) removed. Results of cardiopulmonary exercise tests such as 6-minute walk or peak oxygen consumption were not available for most patients and were not included in the vignettes. Two pools of subjects were used in this study. All 12 senior residents in general surgery at The University of Chicago Medical Center who had completed at least 2 months of training on the Thoracic Surgery Service within the past 2 years were invited to participate (trainee surgeons). A group of 14 thoracic surgeons who were currently in practice at academic medical centers in the United States or Europe and who were known personally by the senior author were also invited to participate (practicing surgeons). Subjects consented via under auspices of a protocol approved by The University of Chicago Internal Review Board. Subjects were asked to read each case vignette and then estimate the relative risk (not otherwise specified) of postoperative complications anticipated for that patient on a scale ranked from 1 (much less than normal risk) to 7 (much more than normal risk), with normal risk indicated as 4 (uninformed estimate). On the second sheet of the patient vignette a calculated risk score, based on the EVAD system [8], was indicated. This risk score is the sum of points assigned (ranging from 0 to 4 for each clinical variable) for age, forced expiratory volume in the first second expressed as a percent of predicted, and diffusing capacity of the lung for carbon monoxide expressed as a percent of predicted, with the final score ranging from 0 to 12. With this information at hand, the subjects were asked to again estimate the patient s risk on the seven-point scale (informed estimate). Subjects were instructed not to revise their original estimates after seeing the calculated risk score, and were instructed not to revise any of their prior estimates after seeing further case vignettes. Case vignettes were randomly ordered in each participant s packet of vignettes. The patient matching process largely eliminated the ability of physicians to focus on obvious predictors of risk, shifting the reliance of risk prediction to more subtle clues contained in the vignette that were not included in the EVAD score. Subjects uninformed and informed risk estimates were assessed both as separate groups (surgical trainee, practicing surgeon) and collectively. Because there was a difference in the range of estimates used by the trainee and practicing physician groups, raw numerical data were standardized as z scores and then transformed to a scale of 0 to 1 (transformed estimates). Continuous data were analyzed using paired and unpaired Student s t tests. Categorical data were compared with 2 methods. Correlation coefficients were calculated to assess the relationship of risk estimates to clinical values. Receiver operator characteristic analysis was performed to assess the accuracy (area under the curve [AUC]; an AUC of 0.55, for example, can be interpreted as an accuracy of 55%) of risk estimates. Kappa scores were calculated to assess interobserver agreement for each patient and were then averaged across the patient pool [9]. Data are expressed as mean standard deviation. A probability value of less than 0.05 was considered statistically significant. Results Risk estimates for 96 vignettes were analyzed; two pairs of vignettes were inadequately matched for purposes of the study. Of 12 trainee surgeons who were invited to participate, 10 completed the estimates successfully, 1 declined, and 1 provided estimates that were not internally consistent and were discarded. Of 14 practicing surgeons who were invited to participate, 9 agreed to participate and provided their estimates. Vignette patients with and without complications were similar (Table 1) except for a higher incidence of good performance status among patients without complications. Problems experienced by patients in the complication group included pulmonary (21 overall; 2 prolonged ventilation, 12 pneumonia, 9 prolonged air leak, 8 lobar collapse requiring intervention), cardiovascular (20 overall; 9 postoperative intravenous inotropic drug for hypotension, 2 pulmonary embolism, 2 myocardial infarction, 16 supraventricular arrhythmia), other (30 overall; 4 wound infection, 6 empyema, 4 bronchopleural fistula, 27 other including gastrointestinal bleeding, renal insufficiency, chylothorax), and operative mortality (11). Most patients who experienced complications exhibited more than one complication. The level of training was related to risk estimation. Trainee surgeons uninformed risk estimates were signif- Table 1. Comparison of Vignette Patients With and Without Postoperative Complications Variable No Complications (n 48) Complications (n 48) p Value Men 56% 60% 0.68 Age (y) PS 0 or 1 83% 60% FEV 1 % Dlco% Fraction resected a ppofev 1 % ppodlco% BMI Diabetes 22.9% 16.7% 0.44 Hypertension 50.0% 43.8% 0.54 Prior MI 4.4% 10.4% 0.26 Stage I 77% 73% 0.64 a Fraction of lung resected based primarily on the number of functional segments removed. BMI body mass index; Dlco% diffusing capacity for carbon monoxide expressed as a percent of predicted; FEV 1 % forced expiratory volume in the first second expressed as a percent of predicted; MI myocardial infarction; ppo predicted postoperative; PS Eastern Cooperative Oncology Group performance status.

3 Ann Thorac Surg FERGUSON ET AL 2010;89: SURGEON ESTIMATES OF LUNG RESECTION RISK Table 2. Risk Estimates for All Patients by Surgeon Category Estimates All Trainee Practicing p Value Trainee vs Practicing Raw values Uninformed Informed p value uninformed vs informed a a a Transformed values Uninformed Informed p value uninformed vs informed a 0.11 a a 1039 a Paired Student s t test. icantly higher than those of practicing surgeons for both raw and transformed estimates (Table 2). This difference persisted for informed raw estimates, but the difference between trainee surgeon and practicing surgeon transformed informed estimates was not statistically different. The ability to differentiate between patients with and without complications based on the matched-pair patient vignettes was poor. Uninformed risk estimates for all physicians were similar for patients without and with complications (raw estimates, versus ; p 0.54; transformed estimates, versus ; p 0.52). Neither trainee surgeons nor practicing surgeons estimated risk as higher for patients who experienced complications (Table 3). The accuracy of physician prediction of complications overall was only moderate for uninformed raw estimates (AUC, 0.55; 95% confidence interval, 0.44 to 0.65); this worsened somewhat for informed raw estimates (AUC, 0.52; 95% confidence interval, 0.41 to 0.62; p 0.29). Similar findings were evident for transformed estimates (uninformed AUC, 0.55; 95% confidence interval, 0.44 to 0.65; informed AUC, 0.52; 95% confidence interval, 0.42 to 0.63; p 0.40). Accuracy was significantly better for practicing surgeons compared with trainee surgeons for raw and transformed estimates (Table 4). Knowledge of an objective risk score resulted in informed physician risk estimates that more closely correlated with the objective risk score; correlation coefficients for all surgeons increased from for uninformed raw risk estimates to for informed raw risk estimates. The objective risk score influenced informed risk estimates for both trainee and practicing surgeons, albeit somewhat differently. More estimates were altered (1- point change in raw estimates) between the uninformed and informed estimates in the trainee surgeon group (458 of 960; 48%) than in the practicing surgeon group (322 of 864; 37%; p 0.001). Trainee surgeons were more likely to increase their risk estimates (17% versus 10%) and less likely to retain the original risk estimate (53% versus 63%), whereas trainee surgeons and practicing surgeons were equally likely to decrease their risk estimates (30% versus 27%; p 0.001). The absolute magnitude of changes in individual estimates was similar between the groups regardless of whether the informed estimates were higher or lower. Raw and transformed risk estimates for practicing surgeons increased significantly; raw risk estimates for trainee surgeons increased but their transformed risk estimates did not demonstrate a significant change. Knowledge of the objective risk score did not importantly influence the accuracy of estimates for either group. Interobserver agreement for uninformed estimates was better for practicing surgeons than for trainee surgeons ( versus ; p 0.001); the strength of agreement for both was categorized as fair. Kappa scores for informed estimates improved significantly for both groups (p for each). There was no difference between the informed scores for practicing surgeons and trainee surgeons ( versus ; p 0.22); the strength of agreement for both groups was categorized as moderate. The relationship between clinical variables and surgeons uninformed risk estimates demonstrated that some variables were much more strongly associated with Table 3. Risk Estimates for Patients With and Without Complications by Surgeon Category Trainee Practicing Estimates No Complications Complications p Value No Complications Complications p Value Raw values Uninformed Informed Transformed values Uninformed Informed

4 1040 FERGUSON ET AL Ann Thorac Surg SURGEON ESTIMATES OF LUNG RESECTION RISK 2010;89: Table 4. Accuracy of Physician Prediction of Complications a Estimates Trainee Practicing p Value (trainee vs practicing) Raw values Uninformed 0.50 (0.40 to 0.60) 0.58 (0.47 to 0.68) Informed 0.49 (0.38 to 0.59) 0.55 (0.44 to 0.65) p value uninformed vs informed Transformed values Uninformed 0.51 (0.41 to 0.61) 0.58 (0.47 to 0.68) Informed 0.49 (0.39 to 0.60) 0.55 (0.44 to 0.65) p value uninformed vs informed a Based on area under the curve using receiver operator characteristic analysis; values in parentheses represent 95% confidence intervals. risk estimates than were others (Table 5). Pulmonary function and possibly patient age correlated with estimates of risk, whereas neither the extent of lung resection nor the performance status was statistically related to risk estimates. Stage was related to risk estimates for practicing surgeons but not for trainee surgeons. Comment Initial forays into assessing the likelihood of postoperative pulmonary incapacity and death after major lung resection began in the early 1950s using timed spirometric values [10, 11]. Subsequently, advanced age and diffusing capacity were identified as independent predictors of pulmonary and other risks [12 15]. Additional factors that likely influence risk but have been less well evaluated include cardiovascular disease, renal dysfunction, poor nutritional status, and poor performance status. who perform lung resections are generally familiar with these risk factors, although they are not all used by all surgeons in generating risk estimates for individual patients, and how each factor is weighted likely varies considerably. The risk factors listed above fail to incorporate a vital element in risk estimation: the surgeon s overall impression of an individual patient s risk based on in-person interactions with that patient. How surgeons develop the ability to estimate risk, what factors are included in their risk assessments, and how accurate their predictions of risk are remain unknown. In an effort to standardize the assessment of risk, a variety of unique scoring systems, evaluation algorithms, and general scoring systems have been applied to lung resection patients [8, 16 22]. Most such systems are only moderately accurate in their ability to predict outcomes for patient populations, and have not been shown to be useful in the estimation of risk for individual patients. Part of the explanation for this is the absence of any data regarding the surgeon s personal impression of risk for an individual patient. For these and other reasons, many surgeons believe that their judgment about an individual patient s risk is likely to be more accurate than any of the existing scoring systems. In reality, the accuracy of surgeon judgment compared with objective methods of risk assessment varies depending on the clinical situation and the method of assessment [3 6]. Judgment about the appropriateness or anticipated outcomes of surgery is influenced by a variety of nonclinical factors, including training level, and can vary with location, time, and recent events [3, 23, 24]. These findings suggest that such judgments are inconsistent and potentially unreliable. As a result, we elected to study surgeon estimates of risk of major lung resection to assess the effects of training level, the factors that appear to affect such judgment, and the influence of a standard risk score on subsequent estimates of risk. The overall accuracy of physician estimates of risk of major lung resection was only fair. This level of accuracy is worse than that reported in other surgical studies, Table 5. Relationship Between Uninformed Estimates and Clinical Variables a Clinical Variable All Physicians Trainee Practicing Age (0.003) (0.001) (0.024) FEV 1 % ( 0.001) ( 0.001) ( 0.001) Dlco% ( 0.001) ( 0.001) ( 0.001) Fraction of lung preserved (0.95) (0.224) (0.284) Performance status (0 or 1 vs other) b Stage (I vs other) b a Correlation coefficient (p value) unless otherwise specified. b 2 p value comparing dichotomized performance status and stage to risk estimates categorized into quartiles. Dlco% diffusing capacity of the lung for carbon monoxide expressed as a percent of predicted; FEV 1 % forced expiratory volume in the first second expressed as a percent of predicted.

5 Ann Thorac Surg FERGUSON ET AL 2010;89: SURGEON ESTIMATES OF LUNG RESECTION RISK which are in the mid-70% range [25]. The lack of accuracy is likely related in part to the use of clinical vignettes in our study compared with actual patients in other studies. In addition, using matched-pair techniques to control for the most obvious risk factors, forcing surgeons to focus on more subtle predictors of risk, undoubtedly influenced the overall accuracy of their estimates. Finally, the use of a broad category of any complication in our study likely provided a less defined outcome for estimating risk compared with the use of more specific targets in other studies. The level of surgeon experience had an important impact on the results. Trainee surgeons had a lower level of agreement among their risk estimates than did practicing surgeons. Based on the raw estimates of risk, trainee surgeons had a higher baseline level of risk estimation than did practicing surgeons. This difference persisted despite transforming the estimates of the groups to similar scales. Practicing surgeons had more accurate estimates of risk than did trainee surgeons, an effect that persisted when the ratings were transformed. The fact that practicing surgeons estimates were more closely aligned and were more accurate than trainee estimates suggests that, despite using matched-pair techniques to neutralize commonly accepted risk factors, the vignettes nevertheless contained useful information (performance status, comorbidities) that importantly influenced risk estimates. The finding that practicing surgeons performed better than did trainee surgeons is expected, and is a finding similar to those of other studies, in which level of training or experience was associated with more accurate estimation of risk or outcomes [26, 27]. Both groups of physicians used a limited number of clinical clues to arrive at their risk estimates. Trainee and practicing surgeon estimates correlated fairly well with forced expiratory volume in the first second expressed as a percent of predicted and diffusing capacity of the lung for carbon monoxide expressed as a percent of predicted, moderately well with patient age, and poorly with performance status and extent of resection. Cancer stage correlated well with risk estimates among practicing physicians but poorly among estimates of trainee physicians. Similar variability in weighting of clinical variables has been reported previously [4, 26, 28]. The finding that a limited number of clues were the basis for risk estimation suggests that a more formal process of reviewing clinical data, including the use of such tools as algorithms that require entry of clinical data in a stepwise decision tree, may improve the accuracy of practicing surgeons risk assessment. Analysis of how trainee surgeons use of variables that are associated with adverse outcomes differs from practicing surgeons use may illuminate methods for improving the education of trainees and identify ways of influencing trainees to think like the best practicing surgeons. Revealing an objective risk score to subjects had an interesting influence on their subsequent risk estimates. A common response of practicing surgeons was to decrease their risk estimates, whereas a large number of trainee estimates were either increased or decreased. These changes substantially improved interobserver agreement among risk estimates, particularly in trainee surgeons. These findings underscore the malleability of trainee surgeon risk estimates, and suggest that accurate objective risk scores may be useful in aligning estimates of risk among relatively inexperienced physicians with those of experienced surgeons [26, 28]. It is not surprising that knowledge of an objective risk score had no effect on the accuracy of risk predictions. The risk scoring system used in this study, and other risk scores for major lung resection, have levels of predictive accuracy that are not dissimilar to that of the practicing thoracic surgeons in this study [8]. This study has notable shortcomings that should be considered when evaluating our findings. The number of participants was small, resulting in potential type 2 errors in some of the analyses. The definition of target outcomes for risk estimation was intentionally vague, which may account for some of the large interobserver differences evident in the risk estimates. Subjects scored a large number of vignettes twice, a process that could result in participant fatigue and reduce the relative accuracy of the estimates. The use of an objective risk score of moderate accuracy prevented exploration of whether an objective score can meaningfully improve physician estimates of risk. The use of pair matching for important clinical variables resulted in the need for surgeons to focus on clues more subtle than age and lung function to estimate risk. Finally, the use of clinical vignettes, rather than basing estimates on patients whom the surgeons have personally evaluated, resulted in an unreliable assessment of overall accuracy of risk estimates. Surgeon estimates of risk for major lung resection, when based on case-matched clinical vignettes, were only moderately accurate. Experienced surgeons have better overall accuracy than trainee surgeons in estimating risk. Both trainee and practicing surgeons use only a few clinical clues in making their risk estimates. Knowledge of an objective risk score influences risk estimates more for trainee surgeons than for practicing surgeons, decreases interobserver differences in both groups, and brings trainee estimates of risk more in line with estimates of practicing surgeons. Prospective studies are necessary to further explore the etiology, accuracy, and utility of surgeon risk estimates. Methodology should include preoperative estimates of risk for actual patients and a more clinically accurate objective risk score. References Wang J, Olak J, Ultmann RE, Ferguson MK. Assessment of pulmonary complications after lung resection. Ann Thorac Surg 1999;67: Ferguson MK. The rationale for developing scoring systems for clinical practice. Thorac Surg Clin 2007;17: Rutkow IM, Starfield BH. Surgical decision making and operative rates. Arch Surg 1984;119: Timmermans D, Kievit J, van Bockel H. How do surgeons probability estimates of operative mortality compare with a decision analytic model? Acta Psychologica 1996;93:

6 1042 FERGUSON ET AL Ann Thorac Surg SURGEON ESTIMATES OF LUNG RESECTION RISK 2010;89: Markus PM, Martell J, Leister I, Horstmann O, Brinker J, Becker H. Predicting postoperative morbidity by clinical assessment. Br J Surg 2005;92: Hartley MN, Sagar PM. The surgeon s gut feeling as a predictor of post-operative outcome. Ann R Coll Surg Engl 1994;76(6 Suppl): American Joint Commission on Cancer. AJCC cancer staging manual, 6th ed. New York: Springer-Verlag, 2002; Ferguson MK, Durkin AE. A comparison of three scoring systems for predicting complications after major lung resection. Eur J Cardiothorac Surg 2003;23: McGinn T, Wyer PC, Newman TB, et al. Measures of observer variability (kappa statistic). Can Med Assoc J 2004; 171: Gaensler EA. Analysis of the ventilation defect by timed vital capacity measurements. Am Rev Tuberc 1951;64: Gaensler EA, Cugell DW, Lindgren I, Verstraeten JM, Smith SS, Strieder JW. The role of pulmonary insufficiency in mortality and invalidism following surgery for pulmonary tuberculosis. J Thorac Surg 1955;29: Ginsberg RJ, Hill LD, Eagan RT, et al. Modern thirty-day operative mortality for surgical resections in lung cancer. J Thorac Cardiovasc Surg 1983;86: Ferguson MK, Little L, Rizzo L, et al. Diffusing capacity predicts morbidity and mortality following pulmonary resection. J Thorac Cardiovasc Surg 1988;96: Markos J, Mullan BP, Hillman DR, et al. Preoperative assessment as a predictor of mortality and morbidity after lung resection. Am Rev Respir Dis 1989;139: Ferguson MK, Reeder LB, Mick R. Optimizing selection of patients for major lung resection. J Thorac Cardiovasc Surg 1995;109: Epstein SK, Faling LJ, Daly BD, Celli BR. Predicting complications after pulmonary resection: preoperative exercise testing vs a multifactorial cardiopulmonary risk index. Chest 1993;104: Melendez JA, Barrera R. Predictive respiratory complication quotient predicts pulmonary complications in thoracic surgical patients. Ann Thorac Surg 1998;66: Pierce RJ, Copland JM, Sharpe K, Barter CE. Preoperative risk evaluation for lung cancer resection: predicted postoperative product as a predictor of surgical mortality. Am J Respir Crit Care Med 1994;150: Berrisford R, Brunelli A, Rocco G, et al. The European Thoracic Surgery Database project: modelling the risk of in-hospital death following lung resection. Eur J Cardiothorac Surg 2005;28: Charlson ME, Pompei P, Ales KL, McKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987;40: Birim O, Maat APWM, Kappetein AP, van Meerbeeck JP, Damhuis RAM, Bogers AJJC. Validation of the Charlson comorbidity index in patients with operated primary nonsmall cell lung cancer. Eur J Cardiothorac Surg 2003;23: Brunelli A, Fianchini A, Al Refai M, Gesuita R, Carle F. Internal comparative audit in a thoracic surgery unit using the physiological and operative severity score for the enumeration of mortality and morbidity (POSSUM). Eur J Cardiothorac Surg 2001;19: Rutkow IM. Surgical decision making. The reproducibility of clinical judgment. Arch Surg 1982;117: Dale W, Hemmerich J, Ghini EA, Schwarze ML. Can induced anxiety from a negative earlier experience influence vascular surgeons statistical decision-making? A randomized field experiment with an abdominal aortic aneurysm analog. J Am Coll Surg 2006;203: Woodfield JC, Pettigrew RA, Plank LD, Landmann M, van Rij AM. Accuracy of the surgeons clinical prediction of perioperative complications using a visual analog scale. World J Surg 2007;31: Jacklin R, Sevdalis N, Darzi A, Vincent CA. Efficacy of cognitive feedback in improving operative risk estimation. Am J Surg 2009;197: Chatterjee S, Ng J, Kwan K, Matsumoto ED. Assessing the surgical decision making abilities of novice and proficient urologists. J Urol 2009;181: Jacklin R, Sevdalis N, Harries C, Darzi A, Vincent C. Judgment analysis: a method for quantitative evaluation of trainee surgeons judgments of surgical risk. Am J Surg 2008;195: DISCUSSION DR JOE B. PUTNAM (Nashville, TN): The authors have a significant experience with risk assessment in thoracic surgery patients. This manuscript builds upon that experience. The authors present an objective model to measure risk assessment based upon the expertise of the evaluator. A good separation of assessment-of-risk experience was shown between novices (fourth-year surgery residents) and experts (attending thoracic surgeons). The authors provided objective data through vignettes, and assessment of risk was made by all participants. The authors identified that experts assess risk a bit better than novices, but both were somewhat inaccurate in this smallsample study. In contrast to risk algorithms of populations applied to the individual patient, this study used annotated clinical data for an individual to predict risk. Malcolm Gladwell in his book Blink discussed how an expert thinks and how an expert makes decisions. Experts will make complex decisions based on multiple factors. Pattern recognition, identification of important elements, and organization of data are all useful in assessing risk. Intuitive assessment of risk from patient-specific variables may become more important as surgeons develop experience. Evaluation of risk assessment using vignettes creates an objective measure for the role of subjective data in assessing and recognizing risk. In the manuscript you identified that the objective risk score influenced the risk estimate. In other words, when the participants were supplied with a risk score, they tended to revise their data a bit. Still, it did not reach a significant level. Could you determine the role of the subjective information in revising this risk assessment? Secondly, would you plan to use this information to teach or improve risk assessment for your residents and colleagues? While we all in this room value the science of medicine, we applaud your unique efforts to refine the art of medicine. Thank you. DR FERGUSON: Thanks, Bill. A portion of the manuscript wasn t included in my talk, and that is the use of an objective score to see how it might influence estimates of risk. Unfortunately, the objective score, like many, wasn t very accurate in estimating risk, and so the influence of the score was to elevate the kappa scores, but they still would not be considered good, even after knowledge of the objective risk score. Part of what it did was bring the good risk estimates down into a more average area, because the score influenced the surgeons to decrease estimates originally in the high risk range and increase estimates originally in the low risk range.

7 Ann Thorac Surg FERGUSON ET AL 2010;89: SURGEON ESTIMATES OF LUNG RESECTION RISK In terms of how this is going to be used to teach, I think it is a little soon to begin to apply this until we understand better how some of the objective factors do and perhaps should influence risk estimate, but also we need to explore better how the subjective factors do. And so I have a project ready to go that has individual surgeons estimate risk preoperatively, then pairing those estimates with the postoperative outcomes for those patients. There are a lot of hurdles, as you can imagine, to getting that approved, but hopefully it will be under way relatively soon. DR KEITH S. NAUNHEIM (St. Louis, MO): Doctor Ferguson, I very much enjoyed the presentation. But I would just like to ask how much of the risk model can actually be predicted from the objective parameters? I know in many of the prior risk assessment models there is only a certain percentage of the actual complications that can be accounted for with any specific risk model. Obviously, none of the risk models are perfect, but how accurate do you think you can make a risk model? Are we actually going to be able to accurately and reproducibly quantitate risk? patients on all the different parameters that I listed. So it was done by hand, essentially subjectively from a relatively large database. DR MEYERS: But then once you had those matched pairs, what did you do with the matched pairs? DR FERGUSON: They were developed into separate individual patient vignettes. So we had the history of present illness, objective findings, past medical history, and then the operation that was performed and the pathologic stage. Nothing was mentioned about operative outcomes. And then the surgeons and trainees were asked to assign a risk based on that parameter. It was essentially paragraphs removed from my initial patient encounter letter. DR MEYERS: So then you had 48 matched pairs of vignettes, so there were 96 vignettes that they had to go through? DR FERGUSON: Right DR FERGUSON: The best risk models currently have an accuracy in the mid-60% range. So looking at just objective parameters, at least the ones that we currently understand, they are not adequate for using clinically to predict outcomes. My proposal is that some estimates based on physician judgment could be added to the objective risk model in order to come up with an overall risk that includes all the objective parameters and the subjective parameters to hopefully improve our patient selection process. DR BRYAN FITCH MEYERS (St. Louis, MO): Could you just explain the methodology in one particular area, how you used the matched pairs, where there were patients with similar characteristics that had a complication or not? I didn t understand exactly how that went into the final process. DR FERGUSON: I developed a very simplified score based on FEV 1 (forced expiratory volume in the first second), Dlco (diffusing capacity of the lung for carbon monoxide), and age, and then began to use that score to nearest neighbor match DR MEYERS: Well, it just seems that you might have been setting them up for that 50% correct rate by having the pairs matched on the factors that you would normally use to assess risk. DR FERGUSON: I think that is a good point; you could make the argument that this underscores my contention that the subjective factor is very important and needs to be explored further in how you assess risk outside of these objective parameters. DR WILLIAM A. BAUMGARTNER (Baltimore, MD): Mark, I enjoyed your talk and I think maybe you answered my question. I would have to look at the written vignettes, but it seems to me that most of us need to directly see the patient to be able to really create in your mind what the risk is for that patient, taking in all the factors. DR FERGUSON: Exactly. It is the eyeball factor that is missing here, which is why we are designing this new study that will include the subjective factor.