reproducibility of the interpretation of hysterosalpingography pathology

Transcription

1 Human Reproduction vol.11 no.6 pp , 1996 Reproducibility of the interpretation of hysterosalpingography in the diagnosis of tubal pathology Ben WJ.Mol 1 ' 2 ' 3, Patricia Swart 2, Patrick M-M-Bossuyt 1, Marc van Beurden 2 and Fulco van der Veen 2 'Department of Clinical Epidemiology and Biostatistics and 2 Department of Obstetrics and Gynaecology, Academic Medical Centre, University of Amsterdam, Amsterdam, The Netherlands 'To whom correspondence should be addressed at: Department of Clinical Epidemiology and Biostatistics, Academic Medical Centre, University of Amsterdam, PO Box 227, 1 DE Amsterdam, The Netherlands. b.w.mol@amc.uva.nl The aim of the study was to estimate the inter- and intraobserver reproducibility of the interpretation of hysterosalpingography (HSG) in the diagnosis of tubal pathology, and associate reproducibility with diagnostic accuracy. Four observers evaluated 143 HSGs twice, on proximal tubal obstruction, distal tubal obstruction, hydrosalpinx and peritubal adhesions. Diagnostic laparoscopy with chromopertubation was considered to be the reference strategy. Reproducibility (inter- and intra-observer agreement) was expressed in terms of K-values. Accuracy was expressed in terms of sensitivity, specificity and likelihood ratios, K- values for reproducibility between observers were almost perfect for proximal obstruction, substantial for distal obstruction and hydrosalpinx, and moderate for adhesions. K-values for reproducibility within observers were almost perfect for proximal obstruction and substantial for distal obstruction, hydrosalpinx and adhesions. HSG had a high specificity for proximal obstruction, but a low sensitivity. Distal obstruction, absence of hydrosalpinx and adhesions had a poor accuracy. The likelihood ratio for the presence of hydrosalpinx was high. In conclusion, proximal tubal obstruction detected on HSG changes the pre-test probability of proximal tubal obstruction from 16 to 5%. Proximal tubal patency detected on HSG changes the pre-test probability of proximal tubal patency from 16 to 9%. It is unlikely that a lack of reproducibility of the interpretation of proximal tubal patency is responsible for the low sensitivity; alternative explanations are artefacts occurring while performing HSG or an imperfect reference strategy diagnostic laparoscopy. HSG is of limited use in diagnosing distal tubal obstruction and hydrosalpinx, and has no value in the detection of peritubal adhesions. Key words: accuracy/diagnostic laparoscopy/hysterosalpingography/reproducibility/tubal pathology Introduction The first reports on the test characteristics of hysterosalpingography (HSG) appeared early in the 197s (Duignan et al, ; Keirse and Vandervellen, 1972; Maathuis et al, 1972; Swolin and Rosencrantz, 1972). Since then, many studies on the diagnostic performance of the HSG have been published. Recently, a meta-analysis of all studies comparing HSG with diagnostic laparoscopy with respect to the detection of tubal pathology has been performed (Swart et al, 1995). This metaanalysis showed that the sensitivity of HSG in the diagnosis of tubal obstruction was 65%, with a specificity of 83%. HSG appeared to be an inappropriate method for detecting peritubal adhesions. A lack of reproducibility in test results is a potential explanation for a low sensitivity and specificity, i.e. accuracy, of the corresponding diagnostic procedure (Kraemer, 1992). If observers disagree on the reading of a test result, the test is unlikely to have a very good accuracy, hi addition, a poor reproducibility limits the clinical value of this procedure. If a low reproducibility can be held responsible for the low sensitivity of the HSG in the diagnosis of tubal obstruction and the poor accuracy in the detection of peritubal adhesions, introducing guidelines for interpretation and training observers may enhance the diagnostic quality of HSG. This study reports on the within- and between-observer reproducibility and accuracy of the HSG, and the association between both, for the detection of tubal pathology. Materials and methods The study was performed in the infertility department of the Academic Medical Center in Amsterdam, The Netherlands. All consecutive patients in whom HSG was performed between May 1985 and November 1987 were included. Patients whose file or X-ray was not retraceable, and patients who did not undergo laparoscopy, were excluded. HSG was performed in patients with primary and secondary infertility, as part of the routine infertility work-up. HSG was done in the follicular phase, with a water-soluble contrast medium (telebrix), by the staff of the department. A spasmolytic was not used. Patients were lying on their backs during the whole procedure. Fluoroscopy was recorded on videotape and four X-rays were taken on each occasion. In 1994, all X-rays were evaluated twice by four members of the infertility department in two sessions with a time-interval of 3 months. Fluoroscopy was not available for the observers. The participants were a professor in gynaecology, two gynaecologists of the section of infertility, and a resident in gynaecology. None had previously been involved in performing the HSGs. The results of the diagnostic laparoscopy were not available to the participants at the time of HSG evaluation. To avoid memory bias, the X-rays were shown in a different order during the second session. The observers were not allowed to discuss HSG results during and between sessions. At the time this study was conducted, there were no written guidelines for interpretation of HSG in our department European Society for Human Reproduction and Embryology

2 Reprodudbility of interpretation of HSG Table I. Results of inter- and intra-observer reproducibility and accuracy (with 95% confidence intervals) for the four items of hysterosalpingography K-value interobersver reproducibility K-value intraobserver reproducibility LR (+) b LR(-) C LR (inadequate for interpretation) d Proximal tuba] obstruction Distal tubal obstruction Hydrosalpinx Peritubal adhesions.85 (.81-.9).69 (.66-O.73).64 (.6-.68).55 ( ).89 ( ).72 ( ).68 ( ).65 (.6-.69) 6. ( ) 2.1 ( ) 5.8 ( ) 1.8 ( ).6 ( ).43 ( ).64 ( ).61 ( ).21 ( ).31 ( ) 1.8(13-2.2) "For proximal tubal obstruction, sensitivity was 44% (37-52%) and specificity was 92% (91-94%). ''Likelihood ratio of a positive test result T-ikelihood ratio of a negative test result. ''Likelihood ratio of test inadequate for interpretation. The observers evaluated the HSG on four items, for each tube separately: (i) proximal tubal obstruction, i.e. absence of filling of the tube, (ii) distal tubal obstruction, i.e. at least partial filling of the tube, without swelling of the tube and without contrast in the peritoneal cavity, (iii) hydrosalpinx, i.e. at least partial filling of the tube, with swelling of the tube and without contrast in the peritoneal cavity, and (iv) peritubal adhesions, i.e. contrast in the peritoneal cavity, but bowel contours not visible. Each item was diagnosed as 'present' or 'absent'. Moreover, an item could be diagnosed as 'inadequate for interpretation' if no contrast could pass as a consequence of a proximal abnormality. For instance, if a tube was diagnosed as 'proximal obstructed', the items 'distal obstruction', 'hydrosalpinx' and 'peritubal adhesions' were 'inadequate for interpretation'. Reproducibility for each of the four evaluated items was expressed using K-statistics. A K-value of indicated agreement beyond chance, a K-value of 1 indicated % perfect agreement between observers. The reproducibility in the case of K-values between.-.2 was regarded as 'slight', between.2-.4 as 'fair', between.4-.6 as 'moderate', between.6-.8 as 'substantial' and between.8-1. as 'almost perfect' (Landis and Koch, 1977). K-values and 95% confidence intervals (CI) for inter- and intraobserver reproducibility were calculated (Fleiss, 1981) with four and two raters respectively. For the calculation of the interobserver reproducibility, only the results of the first session were used. In addition, if an item could be diagnosed as 'present', 'absent' or 'inadequate for interpretation', K-values were calculated for the interand intra-observer reproducibility of these three results separately. Diagnostic laparoscopy with chromopertubation was the reference strategy for the calculation of diagnostic accuracy. Diagnostic laparoscopy was always performed after the HSG, with a.double puncture technique, by a resident in gynaecology, supervised by a staff member. Methylene Blue was injected at room temperature through a Foley catheter into the uterine cavity. The amount of Methylene Blue was variable, depending on the time necessary to assess tubal function. Abnormal findings were recorded on videotape. Diagnostic laparoscopy was evaluated independently from the results of HSG. Results of diagnostic laparoscopy were reported systematically in the medical files. Accuracy (with 95% CI) was calculated as the mean accuracy of the four observers, and expressed in terms of sensitivity, specificity and likelihood ratios. Accuracy calculations were based solely on results of the first judgement session of the HSGs. Presence of disease on HSG (tubal obstruction/hydrosalpinx/peritubal adhesions) was regarded as a positive test, whereas absence of disease was regarded as a negative test Therefore, sensitivity was defined as the percentage affected tubes correctly detected by HSG. Specificity was defined as the percentage of unaffected tubes properly classified by HSG. A 2X2 table was constructed for proximal tubal obstruction, and sensitivity, specificity and likelihood ratios were calculated. For distal tubal obstruction, hydrosalpinx, and peritubal adhesions, a 3X3 table was constructed because some HSGs were diagnosed as 'inadequate for interpretation'. Consequently, likelihood ratios of presence of abnormality, absence of abnormality and 'inadequate for interpretation' were calculated. To assess the relationship between accuracy and reproducibility, we stratified the results of the interpretation of HSG for the amount of agreement. The results of the two sessions were added, so there were eight observations on each item per tube. Results Between May 1985 and November 1987, 365 patients underwent HSG. In 21 patients diagnostic laparoscopy with chromopertubation was also performed. The files and/or X-rays of 58 of them could not be retrieved. As a consequence HSGs of 143 patients were included in the study, so each observer evaluated 286 tubes. At diagnostic laparoscopy, the prevalence of proximal obstruction was 16%, the prevalence of distal obstruction 29%, the prevalence of hydrosalpinx 13% and the prevalence of adhesions 24%. Reproducibility In Table I, inter- and intra-observer reproducibility statistics are presented for individual observers. The overall K-values (with 95% CI) for inter-observer reproducibility were.85 (.81-.9) for proximal obstruction,.69 ( ) for distal obstruction,.64 (.6-.68) for hydrosalpinx, and.55 ( ) for adhesions. For distal obstruction, hydrosalpinx and peritubal adhesions, the interpretation of an item could be divided in three different categories. In these cases it was possible to calculate the reproducibility for each interpretation separately. For distal obstruction, the K-values of the test results 'present', 'absent' and 'inadequate for interpretation' were.53,.71 and.85 respectively. For hydrosalpinx 'present', the K-value was fair,.24. For hydrosalpinx 'absent' and 'inadequate for interpretation' K-values were.61 and.71 respectively. The test result 'adhesions present' was hardly reproducible (K-value.18). The results 'adhesions absent' and 'adhesions inadequate for interpretation' were more reproducible:.55 and.7 respectively. The overall K-values (with 95% CI) for intra-observer reproducibility were.89 ( ) for proximal obstruction,.72 ( ) for distal obstruction,.68 ( ) for hydrosalpinx, and.65 (.6-.69) for adhesions (Table I). Intra-observer reproducibility of the separate test results of 125

3 B.WJJVlol et al I No disease D Disease Number of observers that diagnose disease 14 -I "I No Disease D Disease I 1 Number of observers that diagnose hydrosalpinx 25 2 No Disease D Disease 15 5 Number of observers that diagnose proximal tubal obstruction Number of observers that diagnose peritubal adhesions Number of observers that diagnose distal tubal obstruction Figure 1. (a) Perfect test with maximal accuracy and maximal reproducibility; (b) accuracy andreproducibilityfor proximal obstruction; (c) accuracy and reproducibility for distal obstruction; (d) accuracy and reproducibility for hydrosalpinx; (e) accuracy and reproducibility for adhesions. distal obstruction for the test result 'absent' was substantial (K-value.72) and for the results 'present' and 'inadequate for interpretation' moderate,.57 and.88 respectively. For hydrosalpmx 'present', the K-value was fair,.37. For hydrosalpinx 'absent' and 'inadequate for interpretation' this was.69 and.72 respectively. For the result 'adhesions present' the K-value was only fair (K-value.38), and for the results 'adhesions absent' and 'adhesions inadequate for interpretation' substantial,.66 and.73 respectively. Accuracy Data on accuracy of HSG are presented in Table I. Likelihood ratios and 95% CI were calculated for all four items. Proximal obstruction had an overall sensitivity of 44% (37-52%) and an overall specificity of 92% (91-94%). For distal obstruction the likelihood ratio of a positive test was 2.1 ( ) and the likelihood ratio of a negative test was.43 ( ). For hydrosalpinx the likelihood ratio of a positive test was 5.8 ( ). However, the likelihood ratio of a negative test was 126

4 Reprodudbility of interpretation of HSG.64 ( ). For adhesions, the likelihood ratio of a positive test was 1.8 ( ) and the likelihood ratio of a negative test.61 ( ). Reproducibility and accuracy Figure 1 shows a histogram for each item, in which the HSG diagnosis on each tube is stratified by the amount of agreement The white part of the bars represents the presence of abnormality on laparoscopy, whereas the black part represents a normal laparoscopy. Figure la shows the ideal situation: if, according to the reference standard, disease is present, all observers consider disease to be present, and vice versa. For the item proximal obstruction the eight observers strongly agreed in their interpretation, resulting in only a few tubes in which no complete agreement was reached. Therefore, the lack of accuracy for this item could not be explained by lack of reproducibility of the interpretation of proximal obstruction on HSG. Figure lb shows that the lack of accuracy was caused by false positive and false negative test results at the extremes (either % positive or % positive), probably caused by artefacts that occurred while performing HSG. For distal obstruction, hydrosalpinx and adhesions, there were many HSG with <% agreement (Figure lc, d and e). Moreover, the % positive and % negative diagnoses for distal obstruction contained a considerable amount of falsepositive and false-negative test results. Lack of accuracy of distal tubal obstruction can be explained by both lack of reproducibility and artefacts in the performance of HSG. Discussion This study showed that the within and between reproducibility of the interpretation of proximal tubal obstruction on HSG is almost perfect (K.86 and.88 respectively). For this item, HSG has a sensitivity of only 44% with a specificity of 92%, compared to diagnostic laparoscopy. These data are in line with the results of our meta-analysis, where summary pointestimates of 65% for sensitivity and 83% for specificity were calculated for proximal obstruction (Swart et al, 1995). A finding of proximal tubal obstruction on HSG changes the probability of proximal obstruction from 16 to 59b. A finding of proximal tubal patency changes the probability of proximal tubal patency from 16 to 9%. The high reproducibility of the interpretation of proximal obstruction on HSG suggests that the low sensitivity of HSG cannot be improved by better interpretation of HSG. For this reason, training of observers or the introduction of guidelines is unlikely to improve reproducibility and accuracy. There remain three other possible explanations for the lack of sensitivity. Laparoscopy is not a perfect reference standard. Alternatively, the lack of accuracy might be due to the fact that the observers were only given access to X-rays, rather than fluoroscopy, or to artefacts occurring while performing HSG. Artefacts might include premature ending of the procedure, insufficient pressure because of vaginal reflux or differences in muscle tonus of the tubes. In that case, sensitivity can only be improved by optimizing the technique for performing HSG. The obtained inter- and intra-observer reproducibility of the interpretation of distal obstruction was only substantial (K.69 and.72 respectively). As expected, it is more difficult to assess overflow of contrast into the peritoneal cavity, then filling of the tube. Compared to proximal obstruction, there was more disagreement. Likelihood ratios were 2. for presence of distal tubal obstruction and.43 for the absence of tubal obstruction. This combination limits the use of HSG in detecting distal tubal patency. For evaluating distal tubal obstruction, training of observers or the introduction of guidelines might improve reproducibility and therefore accuracy of the diagnosis of distal tubal obstruction. As for distal tubal obstruction, inter- and intra-observer reproducibility of the interpretation of hydrosalpinx on HSG were only 'substantial' (K.64 and.68). The likelihood ratios of a positive and a negative test were 5.8 and.64 respectively. This means that HSG is a useful test to detect hydrosalpinx, but an imperfect test to rule out hydrosalpinx. In our metaanalysis hydrosalpinx was not studied, due to a lack of data. Here also, training of observers or the introduction of guidelines could improve reproducibility and therefore accuracy of the diagnosis. A likelihood ratio of 1.8 for a positive test and of.61 for a negative test were calculated for adhesions, in accordance with the results of our meta-analysis. HSG therefore is unreliable for the evaluation of peritubal adhesions. Overall, inter-observer reproducibility was moderate (K.55) and intraobserver reproducibility just substantial (K.65). However, inter-observer reproducibility of the test result 'adhesions present' was only 'slight', and its intra-observer reproducibility 'fair'. This lack of reproducibility confirms our earlier conclusion that the HSG is of no clinical significance for the diagnosis of peritubal adhesions. In conclusion, we can say that HSG has limited use in detecting tubal patency, defined as the absence of proximal and distal tubal obstruction. However, it is able to detect proximal tubal occlusion. The lack of sensitivity is not caused by a lack of reproducibility. The usefulness of HSG for detecting distal tubal obstruction is also limited. HSG is a good test to detect hydrosalpinx, but not very useful in ruling out this condition. Improvement of reproducibility through the introduction of guidelines might improve accuracy and therefore the practical value of the HSG with regard to distal obstruction and hydrosalpinx. HSG is of no use for the detection of adhesions. Acknowledgements The authors wish to thank F.B.Lammes, MD, PhD and H.V.Hogerzcil, MD for the interpretation of the HSGs, and W.M.Ankum, MD, PhD and J.V.Th.H.Hamerlynck, MD, PhD, for critically reading the manuscript References Duignan, N.M., Jordan, J.A., Coughlan, B.M. and Logan-Edwards, R. (1972) One thousand consecutive cases of diagnostic laparoscopy. J. Obstet. GynaecoL Br. Comm., 79, Fleiss, J.L. (1981) Statistical Methods for Rates and Proportions. John Wiley and Sons, New York, pp Keirse, MJ.N.C. and Vandervellen, R. (1972) A comparison of hysterosalpingography and laparoscopy in the investigation of infertility. Obstet GynecoL, 41,

5 B.WJJvIol et al Kraemer, H.C. (1992) Evaluating Medical Tests. Sage Publications, Newsbury Park, California, pp Landis, J.R. and Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, Maathuis, J.B., Horbach, M.M. and Van Hall, E.V. (1972) A comparison of the results of hysterosalpingography and laparoscopy in the diagnosis of Fallopian tube dysfunction. FertiL SleriL, 23, Swart, P., Mol, B.WJ., van der Veen, E, van Beurden, M., Redekop, W.K. and Bossuyt, P.M.M. (1995) The accuracy of hysterosalpingography in the diagnosis of tubal pathology, a meta-analysis. FertiL Steril, 64, Swolin, K. and Rosencrantz, M. (1972) Laparoscopy vs. hysterosalpingography in sterility investigations, a comparative study. FertiL Steril., 23, Received on January 11, 1996; accepted on March 14,