Use of IOTA simple rules for diagnosis of ovarian cancer: meta-analysis

Transcription

1 Ultrasound Obstet Gynecol 2014; 44: Published online 12 October 2014 in Wiley Online Library (wileyonlinelibrary.com). DOI: /uog Use of IOTA simple rules for diagnosis of ovarian cancer: meta-analysis N. NUNES*, G. AMBLER, X. FOO*, J. NAFTALIN*, M. WIDSCHWENDTER and D. JURKOVIC* *Gynaecological Diagnostic Outpatient Treatment Unit, University College Hospital, London, UK; Department of Statistical Science, University College London, London, UK; Department of Women s Cancer, Institute for Women s Health, Elizabeth Garrett Anderson, University College London, London, UK KEYWORDS: adnexal masses; IOTA simple rules; meta-analysis; meta-regression; ovarian cancer; ovarian tumors; pattern recognition; sensitivity; specificity; ultrasound operator level ABSTRACT Objectives To present data on prospective evaluation of the International Ovarian Tumor Analysis (IOTA) simple-rules tool for the diagnosis of ovarian cancer and to perform a meta-analysis of studies that utilized the same diagnostic method. Methods In the present study a level-ii ultrasound operator systematically assessed the tumors of women with an ultrasound diagnosis of adnexal tumor(s) according to the IOTA simple-rules protocol to determine the risk of the tumor being malignant. The results of simple rules were compared with the pattern recognition method and with histological findings. This validation study was included in the subsequent meta-analysis, for which we searched MEDLINE, EMBASE and Cochrane from the publication of the first study in The terms used were simple rules, simple rules ovarian, ovar tumor and ultrasound. Quality assessment was performed using the modified Quality Assessment of the Diagnostic Accuracy of Studies (QUADAS-2) checklist. Random effects meta-analysis was used to calculate pooled estimates of sensitivity and specificity for the simple-rules tool, and meta-regression was used to investigate heterogeneity across the studies. Results Three hundred and three women were included in the validation study with 168 (55.4%) benign, 19 (6.3%) borderline and 116 (38.3%) malignant tumors on histological examination. The rules were applicable in 237 (78.2%) of the tumors and for these tumors, sensitivity was 96.2% (95% CI, %) and specificity was 88.6% (95% CI, %). Six of the 88 discovered studies were included in the meta-analysis along with the current validation study, which resulted in inclusion of a total of 3568 patients. When the meta-analysis was performed the pooled sensitivity (when the rules were applicable) was 93% (95% CI, 90 96%) (I 2 = 32.1%) and the pooled specificity was 95% (95% CI, 93 97%) (I 2 = 78.1%). Heterogeneity was observed across the studies. was higher and specificity lower in the study populations in which the prevalence of malignant tumors was greatest. Conclusion The simple rules protocol could be used in 76 89% of tumors and is an accurate test for the diagnosis of ovarian cancer. Assessment by an ultrasound expert is required when the protocol cannot be applied. Copyright 2014 ISUOG. Published by John Wiley & Sons Ltd. INTRODUCTION Accurate preoperative ultrasound characterization of adnexal pathology helps to plan patients care and optimize the surgical approach. The International Ovarian Tumor Analysis (IOTA) group is a multicenter collaboration whose aim is to design tools for the preoperative diagnosis of ovarian cancer that can be used by non-expert ultrasound operators. By using standardized terms and definitions to describe morphological features of ovarian tumors, the simple-rules model of observed ultrasound features was designed 1,2. To determine the nature of an adnexal tumor there are five benign features and five malignant features. If one or more benign features are found in the absence of any malignant features then the tumor is defined as benign. If one or more malignant features are found in the absence of any benign features, then the tumor is defined as malignant. If no features are seen or if both malignant and benign features are observed then the tumor is unclassified or indeterminate. In the original study the Correspondence to: Mr D. Jurkovic, Gynaecology Diagnostic Outpatient Treatment Unit, Lower Ground Floor, Elizabeth Garrett Anderson, University College Hospital, London, NW1 2BU, UK ( Davor.Jurkovic@uclh.nhs.uk) Accepted: 30 May 2014 Copyright 2014 ISUOG. Published by John Wiley & Sons Ltd. SYSTEMATIC REVIEW

2 504 Nunes et al. rules were applicable in 76% of cases with a sensitivity of 93%, specificity of 90%, positive likelihood ratio (LR+) of 9.45 and negative likelihood ratio (LR ) of Level-II ultrasound operators are non-experts who tend to describe morphological features of tumors without attempting to determine their nature 3. Level-III operators use subjective analysis called pattern recognition to determine the nature of an ovarian tumor. Pattern recognition is established as the most accurate method of analysis, but it takes many years to gain the experience required to use the test in routine clinical practice 4,5. The aim of this study was to present new data on the diagnostic accuracy of the simple-rules protocol for the diagnosis of ovarian cancer and to include the simple-rules data in a meta-analysis. METHODS Patients with adnexal tumor (n = 555) Surgery/biopsy indicated and performed (n = 335) Women not pregnant (n = 331) Surgery/biopsy performed within 120 days (n = 303) Excluded because surgery/biopsy not indicated and performed (n = 220) Excluded because pregnant (n = 4) Excluded because surgery/biopsy not performed within 120 days (n = 28) This was a prospective single-center study performed over 33 months from May 2009 to January 2012 in a general gynecology clinic within a tertiary unit. A single level-ii ultrasound operator (N.N.) assessed all consecutive patients diagnosed with an adnexal tumor using a Voluson E8 ultrasound machine (GE Medical Systems, Zipf, Austria). Assessment included a detailed history and ultrasound examination using both transvaginal and transabdominal approaches, and this was always done prior to obtaining, and therefore without knowledge of, a histological diagnosis. In accordance with the simple-rules protocol the benign features were: B1, unilocular cyst; B2, presence of solid components (largest diameter < 7 mm); B3, presence of acoustic shadowing; B4, smooth multilocular tumor with largest diameter < 100 mm; and B5, no blood flow (color score 1). The malignant features were: M1, irregular solid tumor; M2, ascites present; M3, at least four papillary structures present; M4, irregular, multilocular solid tumor with largest diameter 100 mm; and M5, very strong blood flow (color score 4). Tumors that displayed both benign and malignant features were unclassified. An expert ultrasound operator who used subjective pattern recognition to classify the tumors as benign or malignant subsequently examined all patients. The expert was blinded to the results of the simple-rules assessment, and management of the women was based solely on the findings of pattern recognition and the patient s wishes. The level-ii operator was not trained in pattern recognition and was discouraged from using it. Women who had had a hysterectomy and who were aged 50 years were defined as postmenopausal, in keeping with the literature. Pregnant women, those unable to undergo a transvaginal scan and those whose surgery date exceeded 120 days from the date of the ultrasound scan were excluded, as was the case with the original study 2 (Figure 1). In women with bilateral tumors, both were assessed and categorized. If either tumor was categorized as malignant then the malignant tumor was Women included in study (n = 303) Figure 1 Flowchart of eligibility for entry to study to assess patients with an adnexal tumor diagnosed on ultrasound using the International Ovarian Tumor Analysis simple rules (current study). used in the final analysis. If the assessment of one tumor was indeterminate and that of the other was benign, then the tumors were categorized as indeterminate. Surgical treatment options included a laparoscopic cystectomy, oophorectomy and laparotomy and debulking surgery. Pathologists were blinded to the simple-rules categorization and were only aware of the opinion of the expert or the surgeon. Only patients with a histological diagnosis were included in the final analysis. All tumors were classified according to the criteria recommended by the International Federation of Gynecology and Obstetrics 6. Though histologically different, borderline tumors were analyzed together with invasive malignant tumors, as in the original study. Our local research and development department determined that formal ethical committee assessment and approval were not required for the study because ultrasound examination of all adnexal tumors was routine departmental practice, including morphological analysis of the tumors using the IOTA terms, and also because therapeutic decisions were not based on the simple-rules results. Guidance on the Standards for the Reporting of Diagnostic Accuracy (STARD) initiative was followed in the conduct, analysis and reporting of our study 7. Meta-analysis Data sources Articles were identified from a number of electronic databases. A search was performed on MEDLINE using

3 Meta-analysis of IOTA simple rules for ovarian cancer 505 Citations retrieved from electronic and manual searches of references (n = 88) Excluded (n = 74): Duplicates (n = 18) Did not include simple-rules protocol (n = 56) Full manuscripts retrieved for detailed evaluation (n = 14) Primary articles included in systematic review (n = 6) Excluded (n = 8): Retrospective and interobserver agreement study (n = 1) 13 Lack of original data (summary) (n = 1) 15 A protocol (n = 1) 16 Data duplication (n = 3) 9,11,12 Interobserver agreement study and non-consecutive (n = 1) 14 Simple-rules data combined with alternative scan protocol (n = 1) 8 Additional article (present study) included in analysis (n = 1) Articles included in systematic review (n = 7) Figure 2 Flowchart of study selection process for meta-analysis of use of International Ovarian Tumor Analysis simple rules for identification of malignant adnexal tumors. the terms simple rules and adnexal, simple rules and ovarian and simple rules and ultrasound. A search was also performed on EMBASE using the terms simple rules after using terms ovar tumor and ultrasound. A Cochrane search was also performed for ovary and tumor and diagnosis. Studies from 1 st January 2008 until 8 th November 2013 were included for review and further manual searching was done of references for the included articles. Our own data were included as one of the studies. selection Two reviewers selected articles (N.N. and D.J.), and N.N. read all abstracts. To be included in this meta-analysis, the study had to include data on use of the simple-rules protocol when applied to adnexal tumors for the diagnosis of ovarian cancer. Women must have had surgery and a histological diagnosis for comparison as the gold standard or the reference standard (Figure 2). Only prospective studies and preferably those with a clear indication that the sample selection was random or consecutive were included. This was to avoid selection bias and improve the applicability and generalizability of the results. After choosing relevant articles, the full text articles were read and 2 2 contingency tables were created with the data extracted independently by two authors (N.N. and G.A.). Assessments were performed based on the modified Quality Assessment of the Diagnostic Accuracy of Studies (QUADAS-2) criteria on each included study by two independent reviewers (N.N. and X.F.) and discrepancies were resolved between them. Studies that combined simple rules with another form of assessment were excluded because it was not possible to extract data on the simple rules alone, as was the case with Sayasneh et al. 8. A further exclusion was the study of Di Legge et al. 9, as it incorporated data from IOTA Phase 1b (n = 507) and Phase 2 (n = 1938), both of which are already included in the data from Timmerman 2008 and Timmerman ,10. The paper by Kaijser et al. 11 was an article describing all the IOTA studies and giving a summary. The authors included IOTA Phases 1, 1b and 2 (temporal and external validation studies) and these data are already included 2,10.Ameyeet al. 12 included patients who were already included in other studies. These were IOTA Phase 1 data patients 2 as well as Phase 2 data patients 10.Theauthorsalsowentonto exclude a significant proportion of the patients who could be classified according to simple descriptors. This study was therefore excluded. Guerriero et al. 13 used five examiners to analyze stored data. This study was excluded as it was a retrospective one and as it had different results for each of the five examiners. Ruiz de Gauna et al. 14 performed a similar study with two ultrasound operators analyzing the tumors in real time and four doing so retrospectively in order

4 506 Nunes et al. Table 1 Accuracy of International Ovarian Tumor Analysis (IOTA) simple rules (SR): comparison of findings in the original 2 and prospective validation 23 studies with data of present study Accuracy (%) (95% CI) NPV (%) (95% CI) PPV (%) (95% CI) LR (95% CI) LR + (95% CI) (%) (95% CI) (%) (95% CI) population Rules applicable 76.1% (386/507) 94.6 ( ) 90.9 ( ) 10.4 ( ) ( ) 80.9 ( ) 97.6 ( ) 92.0 ( ) Simple rules Original SR study 2 (test set) (n = 386) 77.5% (1501/1938) 92.1 ( ) 95.7 ( ) 21.3 ( ) ( ) 87.4 ( ) 97.4 ( ) 94.8 ( ) Prospective validation SR study 10 External and temporal (n = 1501) 78.2% (237/303) 96.2 ( ) 88.6 ( ) 8.46 ( ) ( ) 87.1 ( ) 96.7 ( ) 92.0 ( ) Rules applicable to population (n = 237) Premenopausal (n = 130) 80.2% (130/162) 89.7 ( ) 89.1 ( ) 8.23 ( ) ( ) 70.3 ( ) 96.8 ( ) 89.2 ( ) Postmenopausal (n = 107) 75.9% (107/141) 98.7 ( ) 87.1 ( ) 7.65 ( ) ( ) 94.9 ( ) 96.4 ( ) 95.3 ( ) SR + PR (n = 303)* 94.1 ( ) 89.9 ( ) 9.3 ( ) ( ) 88.2 ( ) 95.0 ( ) 91.7 ( ) PR (n = 66) 86.7 ( ) 94.4 ( ) 15.6 ( ) ( ) 92.9 ( ) 89.5 ( ) 90.9 ( ) SR + MA (n = 303) 97.0 ( ) 69.6 ( ) 3.2 ( ) ( ) 72.0 ( ) 96.7 ( ) 81.8 ( ) Pattern recognition Prospective validation SR study 10 Rules-applicable 91.1 ( ) 95.7 ( ) 21.0 ( ) ( ) 87.3 ( ) 97.0 ( ) 94.5 ( ) External and temporal (n = 1501) population (n = 237) Rules-applicable 97.1 ( ) 93.2 ( ) 14.2 ( ) ( ) 91.9 ( ) 97.6 ( ) 94.9 ( ) population Note: Exact confidence intervals are given. *SR+PR, simple rules and use of pattern recognition for all those in whom rules were not applicable. PR, pattern recognition used for those patients in whom rules were not applicable. SR+MA, simple rules and malignancy assumed for all those in whom rules were not applicable. LR+, positive likelihood ratio; LR, negative likelihood ratio; NPV, negative predictive value; PPV, positive predictive value. to assess interobserver agreement, and was therefore excluded as no single result was available. Prömpeler s study 15 was a summary and did not include any original data, but instead presented the data from Timmerman et al. 2. Nunes et al. 16 is a published protocol for an ongoing study and does not include data. Quality assessment The quality of each study was assessed using the QUADAS-2 tool, looking at the four domains of patient selection, index test used, reference standard or gold standard and flow and timing of the study 17. Quality characteristics were documented for the selected publications according to QUADAS-2 criteria. Verification, work-up or referral bias occurs if the decision to perform the reference test (surgery with resultant histology) is based on the result of the index test under assessment (simple rules). If the decision for surgery was based on the result of the simple-rules model, verification bias was considered to have occurred. There are various types of verification bias. Partial verification bias occurs if only a selected subset of the population who underwent the index test undergo the reference test and the decision about which subset undergoes it is dependent on the results of the test. Differential verification bias occurs when the index test result affects whether the participant receives the same reference standard test. Incorporation bias occurs if the index test forms part of the reference standard. Only prospective studies with no significant evidence of the various types of verification, selection and incorporation bias were included. To be included these patients must all have had transvaginal ultrasound scans and they should have had surgery within 120 days of performing the scan. Statistical analysis Analysis was performed using the packages metan 18,19, metareg 20,21 and metandi 22,23 in Stata 13.1 (Stata Corp., College Station, TX, USA). The sensitivity, specificity and accuracy of simple rules and pattern recognition were compared separately using McNemar s test for paired binary outcomes. These analyses are based on the 237 patients for whom the simple rules were applicable. The sensitivity, specificity and accuracy of simple rules were compared with the original and the validation IOTA studies separately using the chi-square test for independent binary outcomes 2,10. Random-effects meta-analysis was used to calculate univariate pooled estimates of sensitivity and specificity for the IOTA simple-rules tool 24. A bivariate approach was also investigated to calculate these values 25. Forest plots were constructed to summarize the results, and heterogeneity was quantified using the I 2 statistic 26. Meta-regression was used to investigate any heterogeneity present in the results and a funnel plot of the diagnostic odds ratio against study size was created to investigate the possibility of publication bias 27.

5 Meta-analysis of IOTA simple rules for ovarian cancer 507 Simple rules applied to eligible patients (n = 303) Rules applicable (n = 237) (78%) Rules not applicable (n = 66) (22%) Target condition absent: benign tumor(s) (n = 121) (51%) Target condition present: malignant tumor(s) (n = 116) (49%) Target condition absent: benign tumor(s) (n = 36) (55%) Target condition present: malignant tumor(s) (n = 30) (45%) Figure 3 Flowchart of results of study to assess patients with an adnexal tumor diagnosed on ultrasound using the International Ovarian Tumor Analysis simple rules (current study). Table 2 Histology results for false-negative diagnoses in assessment of adnexal tumors according to International Ovarian Tumor Analysis simple rules Histology Borderline 1 Dermoid cyst with 1 2-mm foci of immature cells 1 Stage III adenocarcinoma 1 Primary peritoneal cancer 1 Total 4 Table 3 Histology results for false-positive diagnoses in assessment of adnexal tumors according to International Ovarian Tumor Analysis simple rules Histology Cystadenoma/cystadenofibroma 6 Dermoid 2 Endometriosis 2 Leiomyoma 1 Actinomycosis 1 Torsion 1 Benign 1 Fibrothecoma 1 Total 15 RESULTS Five hundred and fifty-five women were initially diagnosed with adnexal tumors and assessed. Of those 555 women, 335 (60.4%) went on to have surgery. We excluded four pregnant women and a further 28 who did not have surgery within the stipulated 120 days from their last ultrasound scan, leaving 303 women for the study (Figure 1). There were no adverse events due to the conduct of the index test. The average age was 50 (range, 16 91) years and 141 women (46.5%) were postmenopausal. Thirty women had bilateral tumors, of which 11 pairs had differing features, although in only one n n case were the contralateral tumors classified differently from each other, one tumor being classified as malignant and the other as indeterminate. The tumor with the malignant classification was included in the final analysis. Of the 303 women included in the analysis, 161 (53.1%) were referred via the woman s general practitioner, from accident and emergency or from an internal referral from a non-gynecological clinic. The remaining women were referred from other hospitals (tertiary referrals). On histology there were 168 (55.4%) benign tumors, 19 (6.3%) borderline tumors, 86 (28.4%) primary invasive ovarian cancers, one (0.3%) primary tubal cancer, 22 (7.3%) metastatic tumors, four (1.3%) primary non-gynecological tumors and three (1.0%) recurrent ovarian cancers. Among women with primary invasive ovarian cancers, 27/86 (31.4%) had Stage I disease, 5/86 (5.8%) Stage II, 34/86 (39.5%) Stage III and 20/86 (23.3%) Stage IV. The simple rules were applicable in 237/303 (78.2%) tumors that were included in the final analysis. The sensitivity for the detection of malignancy in this population was 96.2% (95% CI, %) and the specificity was 88.6% (95% CI, %). Accuracy was 92.0% (95% CI, %) (Table 1). For the same population of 237 women the sensitivity of pattern recognition was 97.1% (95% CI, %) and the specificity was 93.2% (95% CI, %). The accuracy of pattern recognition was 94.9% (95% CI, %) (Table 1). As expected the prevalence of malignancy was much higher in postmenopausal patients (67.4% (95/141)) than in premenopausal patients (24.7% (40/162)). This difference was larger when we looked at the population in whom the rules were applicable (postmenopausal, 71.0% (76/107); premenopausal, 22.3% (29/130)). Of the 162 premenopausal women the rules were applicable in 80.2% (130/162) with a sensitivity of 89.7% (95% CI, %), a specificity of 89.1% (95% CI, %) and an accuracy of 89.2% (95% CI, %). Of the 141 postmenopausal women in our population the rules were applicable in 107 (75.9%). For

6 508 Nunes et al. Table 4 Studies included in meta-analysis of accuracy of International Ovarian Tumor Analysis simple rules in diagnosis of malignant adnexal masses (95% CI) (95% CI) TN (n) FN (n) FP (n) TP (n) Prevalence of malignancy (%) Benign tumors (n (%))* Malignant tumors (n (%))* Patients in whom rules are applicable (n (%))* n Timmerman (2008) (76.1) ( ) 0.91 ( ) (77.5) 542 (28.0) 1396 (72.0) ( ) 0.96 ( ) Fathallah (2011) (89.3) 14 (11.5) 108 (88.5) ( ) 0.97 ( ) (88.3) 30 (29.1) 73 (70.9) ( ) 0.87 ( ) (83.9) 74 (29.0) 181 (71.0) ( ) 0.98 ( ) Alcázar (2013) (79.4) 55 (16.2) 285 (83.8) ( ) 0.97 ( ) (78.2) 135 (44.6) 168 (55.4) ( ) 0.89 ( ) Only the first author of each study is given. *Percentage of total population. Percentage of population in whom rules are applicable. FN, false negative; FP, false positive; TN, true negative; TP, true positive. these women there was an increased sensitivity of 98.7% (95% CI, %) vs 89.7% (95% CI, %), a similar specificity of 87.1% (95% CI, %) vs 89.1% (95% CI, %) and an increased accuracy of 95.3% (95% CI, %) vs 89.2% (95% CI, %) when compared with the premenopausal women (Table 1). When pattern recognition was used on the 66 tumors that could not be classified by the simple rules, the sensitivity was lower than that of the simple-rules classification, although the specificity was much higher (86.7% (95% CI, %) and 94.4% (95% CI, %), respectively). Further analysis was done using pattern recognition as the second-line test when the rules were not applicable. The final analysis of the entire population of 303 women had a sensitivity of 94.1% (95% CI, %), a specificity of 89.9% (95% CI, %) and an accuracy of 91.7% (95% CI, %) (Table 1). If all tumors that were not classifiable by the simple rules were assumed to be malignant the sensitivity increased, to 97% (95% CI, %), while the specificity decreased, to 69.6% (95% CI, %), as expected. The overall accuracy also decreased, to 81.8% (95% CI, %). Thirty-six of 66 (54.5%) of the tumors for which the rules did not apply were benign (Figure 3). In this group of women there was a fairly equal distribution between those tumors in which neither M nor B rules applied (34/66 (51.5%)) and those in which both types of rule applied (32/66 (48.5%)). The prevalence of malignant lesions was similar in women in whom neither or both rules applied (41.2% vs 50%; P > 0.05). These malignancies were more likely to be borderline when both rules applied as opposed to when neither type applied (43.8% vs 7.1%; P < 0.05). There were four false-negative diagnoses with the simple rules: a borderline tumor, a dermoid cyst with a 2-mm area of immature cells, a primary invasive Stage III adenocarcinoma and a primary peritoneal cancer (Table 2). There were 15 false positives with a wide range of histological diagnoses (Table 3). The accuracy and sensitivity of simple-rules diagnoses were similar between our and the IOTA original and validations studies, but the specificity was significantly lower in our study than in the IOTA validation study (P < 0.001) 2,10. Meta-analysis The initial search identified 88 studies (Figure 2). After excluding duplicates (n = 18) and studies that did not include the simple-rules protocol (n = 56), 14 studies were left for detailed evaluation these full-text articles were then read fully. A further eight studies were excluded because they were using simple rules combined with another ultrasound tool 8 (n = 1), or they were a reanalysis of previously published data included here from another study and therefore of patients already included 9,11,12 (n = 3), a retrospective study looking at

7 Meta-analysis of IOTA simple rules for ovarian cancer 509 Table 5 External validation studies in meta-analysis of accuracy of International Ovarian Tumor Analysis simple rules Patients with rules applicable (n) Prevalence of malignancy (%) (95% CI) (95% CI) All women * ( ) 0.96 ( ) Fathallah (2011) ( ) 0.97 ( ) ( ) 0.87 ( ) ( ) 0.98 ( ) Alcázar (2013) ( ) 0.97 ( ) ( ) 0.89 ( ) Premenopausal women 10 * ( ) 0.97 ( ) ( ) 0.90 ( ) ( ) 1.00 ( ) Alcázar (2013) ( ) 0.97 ( ) ( ) 0.89 ( ) Postmenopausal women 10 * ( ) 0.94 ( ) ( ) 0.80 ( ) ( ) 0.93 ( ) Alcázar (2013) ( ) 1.00 ( ) ( ) 0.87 ( ) Only the first author of each study is given. *Only patients from the external center included. Fathallah et al. were unable to provide the data for pre- and postmenopausal women to us in time. Domain 4: Flow and timing Domain 3: Reference standard Domain 2: Index test Domain 1: Patient selection Proportion of studies of low, high or unclear quality with regard to domain (%) Figure 4 Quality of studies of International Ovarian Tumor Analysis simple rules included in meta-analysis as assessed by modified Quality Assessment of Diagnostic Accuracy of Studies (QUADAS-2) criteria. Risk of bias:, low;, unclear;, high. interobserver variability 13 (n = 1), a mixed prospective and retrospective assessment of interobserver variability 14 (n = 1), a summary 15 (n = 1) or a protocol 16 (n = 1). That left six studies to add to our data. All seven included datasets were from observational cross-sectional studies 2,10,28 31 (Tables 4 and 5). Quality characteristics were documented for the selected publications and are depicted in tabular and graphical formats (Table S2, Figure 4). All data were prospectively collected, and there was no evidence of any significant degree of the various types of verification bias, as the decision for surgery for all women was made independently of the simple-rules assessment, almost all women had the same reference standard and the index test did not form part of the reference standard. One study had 5/282 (1.8%) participants without a histological diagnosis (two

8 510 Nunes et al. (a) (a) Sens. (95% CI) Sens. (95% CI) Timmerman (2008) Fathallah (2011) 0.95 ( ) 0.92 ( ) 0.73 ( ) 0.91 ( ) 0.87 ( ) Fathallah (2011) 0.91 ( ) 0.73 ( ) 0.91 ( ) 0.87 ( ) 0.88 ( ) 0.88 ( ) 0.96 ( ) 0.93 ( ) 0.96 ( ) 0.92 ( ) (b) Timmerman (2008) Spec. (95% CI) 0.91 ( ) (b) Spec. (95% CI) 0.96 ( ) 0.96 ( ) Fathallah (2011) 0.97 ( ) 0.87 ( ) 0.98 ( ) 0.97 ( ) 0.89 ( ) Fathallah (2011) 0.97 ( ) 0.87 ( ) 0.98 ( ) 0.97 ( ) 0.95 ( ) 0.89 ( ) Figure 5 Forest plot of pooled sensitivity (Sens.) (a) and specificity (Spec.) (b) of seven studies of International Ovarian Tumor Analysis simple rules included in meta-analysis. Area of each shaded box is proportional to weighting of study in meta-analysis. cases of ovarian torsion untwisted at surgery and three cases of abscesses confirmed on microscopy and culture) 30 (Table S2). All the women in each study had the index test as a simple-rules assessment via a transvaginal ultrasound scan and all had histology as the reference standard. Two of the seven studies did not state whether all pregnant patients were excluded 28,29, although in one of those studies two patients were excluded, because for each of them the adnexal mass was an ectopic pregnancy 29. A variety of patients were excluded from each study. Alcázar et al. 31 and Hartman et al. 29 appeared to have high exclusion rates, but they included patients who would have been excluded at the start of other studies, such as patients who did not attend for the first scan, patients who had no mass or those who chose expectant management or had surgery at a different institution. The exclusions were therefore appropriate and without a large loss-to-follow-up rate. All studies therefore had a low risk of bias in relation to domain 1, Patient selection. In all studies, the index test was performed independently of the reference standard with a predetermined threshold for diagnosis, resulting in a low risk of bias in relation to domain 2, Index test. Only one study other than the present one declared blinding of the pathologist who performed the reference standard 28. This caused domain 3, Reference standard 0.96 ( ) Figure 6 Forest plot of pooled sensitivity (Sens.) (a) and specificity (Spec.) (b) of studies of International Ovarian Tumor Analysis simple rules included in subanalysis of externally validated studies. Area of each shaded box is proportional to weighting of study in meta-analysis. to have a large segment where the risk of bias was unclear (70%). Timing of surgery was unknown for only one study 28, whereas in all others surgery occurred within 21 days 31, 112 days 29 or 120 days 2,10,30. This therefore meant that domain 4, Flow and timing had a low risk of bias in over 80% of the studies. Details of the sensitivities, specificities, population size and prevalence of malignancy for each study are shown in Table 4. The pooled sensitivity for all seven studies was 93% (95% CI, 90 96%) (Figure 5a). The I 2 value of 32.1% suggests moderate heterogeneity amongst the studies. The pooled specificity was 95% (95% CI, 93 97%) and the I 2 value of 78.1% suggests considerable heterogeneity between the studies (Figure 5b). Very similar pooled values for sensitivity and specificity were obtained using the bivariate method (results not shown). A subanalysis was done on only the externally validated studies and the pooled sensitivity decreased slightly to 92% (95% CI, 88 96%) whereas the pooled specificity increased slightly to 96% (95% CI, 94 98%) (Figure 6). Subanalyses were done for pre- and postmenopausal women (Figures 7 and 8). was higher in

9 Meta-analysis of IOTA simple rules for ovarian cancer 511 (a) (a) Sens. (95% CI) Sens. (95% CI) 0.90 ( ) 0.91 ( ) 0.89 ( ) 0.91 ( ) 0.82 ( ) 0.90 ( ) 0.88 ( ) 0.88 ( ) 0.90 ( ) 0.99 ( ) 0.89 ( ) 0.94 ( ) (b) (b) Spec. (95% CI) Spec. (95% CI) 0.97 ( ) 0.94 ( ) 0.90 ( ) 0.80 ( ) 1.00 ( ) 0.93 ( ) 0.97 ( ) 1.00 ( ) 0.89 ( ) 0.87 ( ) 0.97 ( ) 0.94 ( ) Figure 7 Forest plot of pooled sensitivity (Sens.) (a) and specificity (Spec.) (b) of studies of International Ovarian Tumor Analysis simple rules included in subanalysis of externally validated studies in premenopausal women. Area of each shaded box is proportional to weighting of study in meta-analysis. postmenopausal women at 94% (95% CI, 89 99%) than in premenopausal women (89% (95% CI, 82 95%)), while specificity was slightly lower (94% (95% CI, 88 99%) and 97% (95% CI, 94 99%), respectively). When sensitivity was plotted against specificity, for all studies there appeared to be a decreasing correlation between sensitivity and specificity (ρ = 0.58) and both sensitivity (ρ = 0.78) and sensitivity (ρ = 0.65) appeared to be associated with prevalence (Figures 9 11). These results suggest that the sensitivity and specificity values for the IOTA simple rules depend, at least in part, on the patient population to which they are being applied. The results from using meta-regression suggest that all the heterogeneity in sensitivity, and over 50% of the heterogeneity in specificity, is due to differences in prevalence across the studies (Figures 10 and 11). A further meta-regression of sensitivity and specificity was performed on operator level but not all studies made clear the level of their operators, and for those studies in which it was known, there was not enough information to assess whether operator level was the cause of heterogeneity. Meta-regression was also used to investigate whether differences in the percentage of the Figure 8 Forest plot of pooled sensitivity (Sens.) (a) and specificity (Spec.) (b) of studies of International Ovarian Tumor Analysis simple rules included in subanalysis of externally validated studies in postmenopausal women. Area of each shaded box is proportional to weighting of study in meta-analysis. population for whom the rules were applicable was related to the observed heterogeneity. However, this percentage only explained a small amount of the heterogeneity in the sensitivity results and none of the heterogeneity in the specificity results (results not shown). We plotted diagnostic odds ratio against study size and there was no evidence of publication bias. This assessment was also done for the external validation studies alone and this confirmed no evidence of publication bias. DISCUSSION The results of our study show that, in the hands of a level-ii operator, the simple rules provide an accurate test for discriminating between benign and malignant adnexal lesions. The rules were applicable in nearly 80% of women, which is similar to the findings of the original and subsequent IOTA validation studies. The sensitivity of ovarian cancer diagnosis was not significantly different between our study and the IOTA studies. The specificity in our study, however, was significantly lower than in the IOTA validation study (P < 0.001), although it was similar to the findings of the original study (P = 0.48).

10 512 Nunes et al Timmerman (2008) Fathallah (2011) Figure 9 vs specificity of studies of International Ovarian Tumor Analysis simple rules for identification of malignant adnexal tumors (all included studies) Prevalence Figure 11 Meta-regression of specificity against prevalence of studies of International Ovarian Tumor Analysis simple rules for identification of malignant adnexal tumors (all included studies). Size of each circle is proportional to precision of the estimate Prevalence Figure 10 Meta-regression of sensitivity against prevalence of studies of International Ovarian Tumor Analysis simple rules for identification of malignant adnexal tumors (all included studies). Size of each circle is proportional to precision of the estimate. IOTA validation studies were performed by level-iii operators with experience in pattern recognition, which could explain the higher specificity in their hands. The four false-negative tumors were either unilocular, as in the case of the borderline tumor, and/or had no demonstrable blood flow, as was the case in all but the Stage III tumor. The latter tumor demonstrated acoustic shadowing, which caused it to be misclassified. These four tumors had none of the M features. The 15 false positives had a range of features that caused them to be misclassified by the simple rules. One case of actinomycosis, one of endometriomata and one fibrothecoma appeared to be irregular and solid. The other tumors were either > 100 mm and irregular multilocular solid, had more than four papillary projections or had a high color-blood-flow score. None of these patients had ascites. Pattern recognition performed by the expert operator was superior to the simple rules in cases in which the rules were applicable mainly due to significant differences in specificity (88.6% vs 93.2% (P = 0.03)). All ultrasound operators who use simple rules will need to determine what to do for those tumors that cannot be classified by this protocol. Assuming malignancy for these will improve sensitivity but at the expense of worsening specificity owing to the increased false-positive rate; overall accuracy, therefore, would deteriorate. Utilizing an expert for this population appears to be the best choice, as overall accuracy would be maintained. Our meta-analysis summarizes the currently available evidence concerning the accuracy of the simple-rules ultrasound tool in the diagnosis of ovarian cancer. It has shown that the simple rules performed well overall for the diagnosis of ovarian cancer in the hands of ultrasound operators of varying levels of expertise, with a pooled sensitivity of 93% (95% CI, 90 96%) (I 2, 32.1%) and a pooled specificity of 95% (95% CI, 93 97%) (I 2, 78.1%) when internal and external validation studies were included. The tool demonstrated a good discriminatory capacity in diagnosing ovarian cancer when the rules were applicable. The study with the lowest sensitivity and that with the lowest specificity were those with the smallest sample sizes, which may have affected their results. When we only included the external validation studies the sensitivity decreased slightly and the specificity increased slightly. The sensitivity and overall accuracy were lower in premenopausal women than in postmenopausal women. There appears to be a strong and increasing relationship between the sensitivity and the proportion of women diagnosed with ovarian cancer in the study population (Figure 10), while there was a weaker and decreasing relationship between specificity and prevalence (Figure 11). this suggests that when the rules are applicable, the sensitivity of simple rules increases and the specificity decreases with the increasing prevalence of malignancy in the study population. Daemen et al. 32 also noted this phenomenon and reported that the specificity of the test was lower in centers with a higher prevalence. They suggested

11 Meta-analysis of IOTA simple rules for ovarian cancer 513 that these centers have more borderline and complex benign cases that are harder to classify. A recently published systematic review and metaanalysis included an analysis of simple rules 33. They included all tumors unclassifiable by the simple rules as malignant and assessed data from two studies 23,30. They found the same sensitivity as we did but a lower specificity in both pre- and postmenopausal women. This is probably owing to the inclusion of all tumors assessed as indeterminate in the malignant group. When we did this with our current study, we found an increase in sensitivity and a significant fall in specificity. While the process of literature review and meta-analysis is a practical and useful way of generating a more powerful estimate of diagnostic accuracy with less random error than the individual studies, it does present some challenges and has some limitations. Primarily, the heterogeneity of the studies must be addressed because this may affect the justification for pooling the data into a single analysis. In this meta-analysis, heterogeneity was caused by diversity in study quality, nonconformity in study reporting, differences in study population characteristics and variation in the prevalence of cancer in the study populations. Not all studies were clear on the exclusion of pregnant women who were not included in any of the original IOTA studies. Possible reasons for the exclusion of pregnant women are their low risk of ovarian malignancy, increased vascularity of all tumors in pregnancy and delay in undergoing surgery. There were two studies that did not give the age range of their patients or the percentage of women who were postmenopausal, making it impossible to determine congruity of patient characteristics. The prevalence of malignancy ranged from 10% to 44% across the studies. We overcame this limitation by assessing for the influence of this heterogeneity on the results. The study sizes also demonstrated diversity, ranging from 103 to 1938 women. Additionally, operator level was not clear in at least one study, making assessment of the influence of this factor incomplete. To summarize, the simple-rules protocol is easy to use in routine clinical practice and is an accurate test for discriminating between benign and malignant ovarian lesions. The main limitation is an inability to apply the test to more than 20% of women, who would need to undergo additional testing or an ultrasound examination by an expert able to use the pattern-recognition method to determine the nature of the adnexal tumors. ACKNOWLEDGMENTS Dr Gareth Ambler received a proportion of funding from the UK Department of Health s NIHR Biomedical Research Centers funding scheme. We would like to acknowledge the assistance of Miss Aisha Lewis, a professional translator, for the translation of Fathallah et al. 28 from French to English. REFERENCES 1. Timmerman D, Valentin L, Bourne TH, Collins WP, Verrelst H, Vergote I; International Ovarian Tumor Analysis (IOTA) Group. Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) Group. Ultrasound Obstet Gynecol 2000; 16: Timmerman D, Testa AC, Bourne T, Ameye L, Jurkovic D, Van Holsbeke C, Paladini D, Van Calster B, Vergote I, Van Huffel S, Valentin L. Simple ultrasound-based rules for the diagnosis of ovarian cancer. Ultrasound Obstet Gynecol 2008; 31: Salvesen KA, Lees C, Tutschek B. Basic European ultrasound training in obstetrics and gynecology: where are we and where do we go from here? Ultrasound Obstet Gynecol 2011; 36: Valentin L. Use of morphology to characterize and manage common adnexal masses. Best Pract Res Clin Obstet Gynaecol 2004; 18: EFSUMB. Minimum training recommendations for the practice of medical ultrasound. Ultraschall Med 2006; 27: Benedet JL, Bender H, Jones H 3rd, Ngan HY, Pecorelli S. FIGO staging classifications and clinical practice guidelines in the management of gynecologic cancers. FIGO Committee on Gynecologic Oncology. Int J Gynaecol Obstet 2000; 70: Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC; Standards for Reporting of Diagnostic Accuracy. Toward complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ 2003; 326: Sayasneh A, Kaijser J, Preisler J, Johnson S, Stalder C, Husicka R, Guha S, Naji O, Abdallah Y, Raslan F, Drought A, Smith AA, Fotopoulou C, Ghaem-Maghami S, Van Calster B, Timmerman D, Bourne T. A multicenter prospective external validation of the diagnostic performance of IOTA simple descriptors and rules to characterize ovarian masses. Gynecol Oncol 2013;130: Di Legge A, Testa AC, Ameye L, Van Calster B, Lissoni AA, Leone FP, Savelli L, Franchi D, Czekierdowski A, Trio D, Van Holsbeke C, Ferrazzi E, Scambia G, Timmerman D, Valentin L. Lesion size affects diagnostic performance of IOTA logistic regression models, IOTA simple rules and risk of malignancy index in discriminating between benign and malignant adnexal masses. Ultrasound Obstet Gynecol 2012; 40: Timmerman D, Ameye L, Fischerova D, Epstein E, Melis GB, Guerriero S, Van Holsbeke C, Savelli L, Fruscio R, Lissoni AA, Testa AC, Veldman J, Vergote I, Van Huffel S, Bourne T, Valentin L. Simple ultrasound rules to distinguish between benign and malignant adnexal masses before surgery: prospective validation by IOTA Group. BMJ 2010; 341: c Kaijser J, Bourne T, Valentin L, Sayasneh A, VanHolsbeke C, Vergote I, Testa AC, Franchi D, Van Calster B, Timmerman D. Improving strategies for diagnosing ovarian cancer: a summary of the International Ovarian Tumor Analysis (IOTA) studies. Ultrasound Obstet Gynecol 2013; 41: Ameye L, Timmerman D, Valentin L, Paladini D, Zhang J, Van Holsbeke C, Lissoni AA, Savelli L, Veldman J, Testa AC, Amant F, Van Huffel S, Bourne T. Clinically oriented three-step strategy for assessment of adnexal pathology. Ultrasound Obstet Gynecol 2012; 40: Guerriero S, Saba L, Ajossa S, Peddes C, Sedda F, Piras A, Olartecoechea B, Aubá M, Alcázar JL. Assessing the reproducibility of the IOTA simple ultrasound rules for classifying adnexal masses as benign or malignant using stored 3D volumes. Eur J Obstet Gynecol Reprod Biol 2013; 171: Ruiz de Gauna B, Sanchez P, Pineda L, Utrilla-Layna J, Juez L, Alcázar JL. Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis

12 514 Nunes et al. simple rules in a real-time setting and using three-dimensional ultrasound volumes and digital clips. Ultrasound Obstet Gynecol 2014; 44: Prömpeler HJ. Ultrasonographic clarification of adnexal findings. Radiologe 2011; 51: Nunes N, Foo X, Widschwendter M, Jurkovic D. A randomised controlled trial comparing surgical intervention rates between two protocols for the management of asymptomatic adnexal tumours in postmenopausal women. BMJ Open 2012; 2: pii: e Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: Harris RJ, Bradburn M, Deeks J, Harbord RM, Altman D, Steichen T, Sterne J. METAN: Stata module for fixed and random effects meta-analysis. Boston College Department of Economics, Statistical Software Components series. Boston College Department of Economics: Boston, MA, USA, Harris RJ, Bradburn M, Deeks J, Harbord RM, Altman D, Sterne JA. Metan: fixed- and random-effects meta-analysis. Stata J 2008; 8: Harbord RM, Steichen T. METAREG: Stata module to perform meta-analysis regression. Boston College Department of Economics, Statistical Software Components series. Boston College Department of Economics: Boston, MA, USA, Harbord RM, Higgins JPT. Meta-regression in Stata. Stata J 2008; 8: Harbord RM, Whiting P, Sterne J. METANDI: Stata software for statistically rigorous meta-analysis of diagnostic accuracy studies. In First International Symposium on Methods for Evaluating Medical Tests, Birmingham, UK Harbord RM, Whiting P. Metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression. Stata J 2009; 9: DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7: Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics 2007; 8: Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med 2002; 21: Thompson SG, Higgins JP. How should meta-regression analyses be undertaken and interpreted? Stat Med 2002; 21: Fathallah K, Huchon C, Bats AS, Metzger U, Lefrère-Belda MA, Bensaid C, Lécuru F. External validation of simple ultrasound rules of Timmerman on 122 ovarian tumors. Gynecol Obstet Fertil 2011; 39: Hartman CA, Juliato CR, Sarian LO, Toledo MC, Jales RM, Morais SS, Pitta DD, Marussi EF, Derchain S. Ultrasound criteria and CA 125 as predictive variables of ovarian cancer in women with adnexal tumors. Ultrasound Obstet Gynecol 2012; 40: Sayasneh A, Wynants L, Preisler J, Kaijser J, Johnson S, Stalder C, Husicka R, Abdallah Y, Raslan F, Drought A, Smith AA, Ghaem-Maghami S, Epstein E, Van Calster B, Timmerman D, Bourne T. Multicentre external validation of IOTA prediction models and RMI by operators with varied training. Br J Cancer 2013; 108: Alcázar JL, Pascual MÁ, Olartecoechea B, Graupera B, AubáM, Ajossa S, Hereter L, Julve R, Gastón B, Peddes C, Sedda F, Piras A, Saba L, Guerriero S. IOTA simple rules for discriminating between benign and malignant adnexal masses: prospective external validation. Ultrasound Obstet Gynecol 2013; 42: Daemen A, Jurkovic D, Van Holsbeke C, Guerriero S, Testa AC, Czekierdowski A, Fruscio R, Paladini D, Neven P, Rossi A, Bourne T, De Moor B, Timmerman D. Effect of cancer prevalence on the use of risk-assessment cut-off levels and the performance of mathematical models to distinguish malignant from benign adnexal masses. Ultrasound Obstet Gynecol 2011; 37: Kaijser J, Sayasneh A, Van Hoorde K, Ghaem-Maghami S, Bourne T, Timmerman D, Van Calster B. Presurgical diagnosis of adnexal tumours using mathematical models and scoring systems: a systematic review and meta-analysis. Hum Reprod Update 2014; 20: SUPPORTING INFORMATION ON THE INTERNET The following supporting information may be found in the online version of this article: Table S1 Quality QUADAS-2 Criteria