Eur J Vasc Endovasc Surg (2010) 39, 220e224 Poor Inter-observer Agreement on the TASC II Classification of Femoropopliteal Lesions * T. Kukkonen a, *, M. Korhonen b, K. Halmesmäki a, L. Lehti b, M. Tiitola b, P. Aho a, M. Lepäntalo a, M. Venermo a a Department of Vascular Surgery, Helsinki University Central Hospital, PO Box 340, FI-00029 HUS, Helsinki, Finland b Department of Radiology, Helsinki University Central Hospital, PO Box 340, FI-00029 HUS, Helsinki, Finland Submitted 21 August 2009; accepted 7 November 2009 Available online 2 December 2009 KEYWORDS TASC; Classification; Femoropopliteal; Arterial lesions; Inter-observer agreement Abstract Objectives: This study aims to evaluate the reproducibility of femoropopliteal TASC II classification and to analyse the influence of an educational intervention on inter-observer agreement. Design: This is a validation study. Materials: This study included 200 consecutive angiograms of femoropopliteal arterial lesions. Methods: Seven investigators evaluated the first 100 angiograms, independently aided by the available TASC guide. Thereafter, the intervention included a discussion of the 25 most problematic cases, initially by a panel of 22 vascular surgeons, and later by the seven investigators to clarify grading principles. In the second stage, the 100 remaining cases were evaluated independently. A multi-rater variation of Brennan and Prediger s free-marginal kappa (k free ) was used to calculate inter-observer agreement. Results: There were lesions not fitting any of the TASC classes. Total agreement among all seven investigators was reached in 7% and 19% of the cases before and after the intervention, respectively. In the first stage, k free was 0.32 between all observers (range between two observers k free Z 0.11e0.54). The intervention increased the agreement to k free Z 0.49 (range: 0.20e0.56). Agreement between the two observers was 38e69% (mean 49%) before the intervention and 51e73% (mean 61%) thereafter. Conclusions: TASC II classification for femoropopliteal lesions allows individual interpretations, and the common use of this classification as a basis for decision making and reporting outcomes could therefore be questioned. ª 2009 European Society for Vascular Surgery. Published by Elsevier Ltd. All rights reserved. * This paper was presented at the XXIII Annual Meeting 3e6 September, 2009, European Society for Vascular Surgery, Oslo, Norway. * Corresponding author. Tel.: þ358 50 4279842; fax: þ358 9 47173548. E-mail address: tiia.kukkonen@hus.fi (T. Kukkonen). 1078-5884/$36 ª 2009 European Society for Vascular Surgery. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.ejvs.2009.11.008
TASC II: Inter-observer Agreement 221 Figure 1 TASC II classification. The TransAtlantic Inter-Society Consensus for the management of peripheral arterial disease (TASC and TASC II 1,2 ) aims to recommend the preferable methods of revascularisation for different arterial lesions by decreasing variation in the management of individual patients with identical conditions. The TASC II classification is widely used as a contemporary guide to aid in decision making concerning lower limb ischaemia when choosing between endovascular and surgical techniques. TASC I and II classifications are used in scientific reports as a means to characterise patient populations and treated lesions 3e5 as well as in reporting the outcome of a specific intervention in stratified groups of lesions. 6,7 However, the present TASC II classification may cause confusion, as the grading of lesions can be cumbersome, allowing individual interpretations of a number of lesions. This, of course, diminishes the applicability of the classification. To the best of our knowledge, there are no published papers on inter-observer agreement on the TASC classifications. The aim of this study was to evaluate the reproducibility of femoropopliteal TASC II classification between vascular surgeons and interventional radiologists, in addition to analysing the influence of an educational intervention on inter-observer agreement. Materials and methods This validation study comprised 200 consecutive angiograms on femoropopliteal arterial lesions treated with endovascular procedures during 2005e2006 at the Department of Vascular Surgery at the Helsinki University Central Hospital. During the first stage, two senior vascular surgeons, two vascular surgical trainees, two angioradiologists and one angioradiologist in training evaluated the first 100 angiograms independently in one session aided by the TASC II guide available (Fig. 1). Thereafter, the intervention included a discussion of the 25 most controversial cases firstly by a panel of 22 vascular surgeons and vascular trainees, and, secondly, by the seven investigators in order to clarify the cases, to agree on TASC II classification principles and to discuss cases not directly belonging to any the of TASC II classes. In the second stage, the 100 remaining cases were evaluated independently by the seven investigators in a manner identical to the first stage.
222 T. Kukkonen et al. Table 1 Lesion types not included in the TASC II classification. Multiple stenoses or occlusions (some >5 cm) totaling <15 cm Heavily calcified occlusion 5 cm but 15 cm involving the infrageniculate popliteal artery Single SFA and popliteal stenosis >10 cm, but 15 cm involving the infrageniculate popliteal artery Single occlusion of >15 cm but 20 cm Total occlusion (5 cm but 15 cm) of the popliteal artery and 1e2/3 trifurcation vessels Single occlusion of SFA >20 cm not involving the popliteal artery A multi-rater variation of Brennan and Prediger s freemarginal kappa (k free ) 8 was used to calculate a changeadjusted measure of agreement using the Online Kappa Calculator (www.justusrandolph.net/kappa/). A k free of 0.70 or above was considered to indicate adequate interobserver agreement. Classes B3 (single or multiple lesions in the absence of continuous tibial vessels to improve inflow for a distal bypass) and C2 (recurrent stenoses or occlusions that need treatment after two endovascular interventions) require additional information besides angiograms to be set. Results In the first evaluation, free-marginal k (k free ) between all observers was 0.32 with a range of 0.11e0.54 between two observers, a mean of 0.27 and a median of 0.15. The intervention increased inter-observer agreement (k free Z 0.49; range between two observers 0.20e0.56, mean 0.37, median 0.36), but it still remained below the adequate inter-rater agreement (0.70). Agreement between two observers varied from 38% to 69% (mean 49%, median 51%) before the intervention and from 51% to 73% (mean 61%, median 61%) thereafter. Total agreement between all seven investigators was reached only in 7% and 19% of the cases before and after the intervention, respectively. Even among three experienced raters who had rated more than 500 angiograms per person prior to this study, the mean k free was 0.26 (median 0.28) before the intervention and 0.49 (median 0.49) following the educational intervention. There are several lesion types which do not belong to any of the TASC II classes (Table 1) (Fig. 2). In an attempt to increase agreement, combining classes improved interobserver reliability, if classes A and B or B and C were combined (Table 2). Discussion The TASC classification has been a welcome attempt to create a uniform system to describe arterial lesions. 3e7 However, in clinical practice, the use of TASC classification often raises questions and disagreement when a lesion of an individual patient is to be classified. Yet, classification of arterial lesions can be difficult, since most cases of peripheral arterial disease requiring intervention are largely characterised by more than one lesion at more than one level, as the authors of the TASC II document have stated. Classification schemes are limited by the necessity to focus on individual lesions or segments with different wall characteristics. 2 In the present study, we have focused on the femoropopliteal segment alone. Figure 2 An example of a lesion not fitting to any of the TASC II classes.
TASC II: Inter-observer Agreement 223 Table 2 Results of combining classes. First stage Second stage Classes k free Reliability % k free Reliability % A, B, C, D 0.32 49.2 0.49 61.4 A þ B, C, D 0.52 68.3 0.71 a 80.8 A, B þ C, D 0.48 65.5 0.62 74.6 A, B, C þ D 0.28 52.2 0.46 63.8 a Adequate inter-observer agreement. The guidelines of the classification in the Second TransAtlantic Inter-Society Consensus Document for the treatment of peripheral arterial disease have been found controversial at times in both clinical and scientific settings. 9 This provided an incentive to study the use of TASC II classification in daily practice. The poor inter-rater agreement in the first part of the study came as no surprise, because the lesions were classified by individuals on the basis of self-education using the printed TASC II document. However, even after the two educational interventions, where all the difficult lesions and common rules were carefully discussed and agreed upon, inter-observer agreement remained poor. There are some shortcomings related to the TASC II classification. It combines stenoses and occlusions and does not outline the grade of stenosis it refers to. Furthermore, lesions can be complex and consist of segments with milder and more severe stenoses and occlusions, leading to differences in interpretation regarding lesion severity and length as well as the number of lesions (single or multiple) (Fig. 3). Defining heavily calcified lesions or the chronicity of occlusions only by viewing angiograms is also a potential source for inter-observer variability. To increase the agreement, a classification system should include definitions of a stenosis (a commonly used definition is that a haemodynamically significant stenosis is a lesion narrowing the lumen of an artery by 50% or more in an angiogram). The second issue with a reference to single or multiple is more difficult and almost impossible to define strictly. A definition stating that a single lesion is a lesion where the arterial segment is stenosed by 50% or more for the entire length of the lesion could be used. Moreover, some of the instructions for the classification are contradictory. For instance, TASC II D1 (chronic total occlusions of CFA or SFA (>20 cm, involving the popliteal artery)) caused confusion. It is not clear whether SFA occlusions need to both be >20 cm in length and have a popliteal involvement, or whether either one is enough to justify D1 classification. Furthermore, some of the written instructions differ from the illustrations included in the classification. TASC II B3 refers to lesion(s) in the absence of continuous tibial vessels to improve inflow for a distal Figure 3 A lesion with several potential classifications.
224 T. Kukkonen et al. bypass. In the illustration, the anterior tibial artery is drawn patent. TASC II D2 refers to lesions with chronic total occlusion of the popliteal artery and proximal trifurcation vessels, while the illustration evokes an idea of only a distal popliteal occlusion. Here, it is unclear, what is meant by a total occlusion, as total occlusion may refer to an occlusion involving the entire length of the popliteal artery as well as to 100% stenosis of any length. Moreover, several lesion types do not fit to any of the TASC classes, which leads to individual interpretations in an attempt to classify them, thereby decreasing interobserver agreement. In these cases, no matter how strictly we tried to agree on the definitions in the guidelines, interobserver agreement remains poor. It is very difficult to make recommendations that cover all types of lesions because of the complexity and variability of the atherosclerotic lesions in this region. A more complex classification system would be required to address this problem, but, conversely, it might lead to decreased applicability of the classification system in daily clinical practice. A weakness of the current study, and one possible explanation for poor inter-observer agreement, was that the length of the lesions was not accurately measured with an electronic measure during the classification process. Furthermore, increasing the time spent studying each case might have improved the agreement. However, the outcome was no better among a subgroup of raters who had rated hundreds of angiograms together and were very familiar with this particular classification system. The morphological classification of atherosclerotic lesions is different between the two TASC documents. 1,2 The authors state in the TASC II document that TASC II classification schemes have been modified from the original TASC guidelines to reflect inevitable technological advances. A key issue is whether classifications should stay unaltered to allow reliable comparisons over time and between different techniques, letting the recommendations change as a reflection of evolving techniques. This kind of evolution in the classification system leads to confusion and weakens the implementation of the system in clinical practice over time. It also makes comparison between published results impossible. However, the original TASC classification was rather simple and did not include crural vessels or multiple lesions, therefore making the development of this classification comprehensible. To diminish the problem of a modified classification system, the classification used (TASC I/TASC II) should, at the very least, be clearly mentioned in vascular publications. 9 All classification systems are vulnerable to interpretation differences. For instance, the intra- and inter-observer reproducibility of well-accepted and extensively used fracture classifications has been found poor to the extent that comparisons between studies have been considered unjustified. 10,11 We conclude that the TASC II classification for femoropopliteal lesions allows wide individual interpretations and, therefore, the common use of this classification as a basis for decision making and reporting outcomes could be questioned. Conflict of Interest/Funding None. References 1 Management of Peripheral Arterial Disease (PAD). TransAtlantic Inter-Society Consensus (TASC). Eur J Vasc Endovasc Surg 2000; 19(Suppl. A):S1e250. 2 Norgren L, Hiatt WR, Dormandy JA, Nehler MR, Harris KA, Fowkes FG, et altasc II Working Group. Inter-Society Consensus for the Management of Peripheral Arterial Disease (TASC II). Eur J Vasc Endovasc Surg 2007;33(Suppl. 1):S1e75. 3 Lepäntalo M, Laurila K, Roth WD, Rossi P, Lavonen J, Mäkinen K, et alscandinavian Thrupass Study Group. PTFE bypass or thrupass for superficial femoral artery occlusion? A randomised controlled trial. Eur J Vasc Endovasc Surg 2009;37: 578e84. 4 Kedora J, Hohmann S, Garrett W, Munschaur C, Theune B, Gable D. Randomized comparison of percutaneous Viabahn stent grafts vs prosthetic femoralepopliteal bypass in the treatment of superficial femoral arterial occlusive disease. J Vasc Surg 2007;45:10e6. 5 Biskup NI, Ihnat DM, Leon LR, Gruessner AC, Mills JL. Infrainguinal atherectomy: a retrospective review of a single-center experience. Ann Vasc Surg 2008;22:776e82. 6 Al-Khoury G, Marone L, Chaer R, Rhee R, Cho J, Leers S, et al. Isolated femoral endarterectomy: impact of SFA TASC classification on recurrence of symptoms and need for additional intervention. J Vasc Surg; 2009 [e-pub]. 7 Korteweg MA, van Gils M, Hoedt MT, van der Valk PH, Tutein Noltenius RP, Avontuur JA, et al. Cryoplasty for occlusive disease of the femoropopliteal arteries: 1-year follow-up. Cardiovasc Intervent Radiol 2009;32:221e5. 8 Randolph JJ. Free-marginal multirater kappa (multirater k free): an alternative to Fleiss fixed-marginal multirater kappa. Finland: Joensuu; 2005. 9 Pedrini L. TASC II guidelines are little help in practice. Vasc News; 2009:4. 10 Belloti JC, Tamaoki MJ, da Silveira Franciozi CE, Gomes dos Santos JB, Balbachevsky D, Chap Chap E, et al. Are distal radius fracture classifications reproducible? Intra and interobserver agreement. Sao Paulo Med J 2008;126:180e5. 11 Siebenrock KA, Gerber C. The reproducibility of classification of fractures of the proximal end of the humerus. J Bone Joint Surg Am 1993;75:1751e5.