European Commission Initiative on Breast Cancer (ECIBC) MEETING OF MONITORING SUBGROUP Ispra, 14 September 2016 (10:00-17:00) Minutes

Transcription

1 EUROPEAN COMMISSION DIRECTORATE GENERAL JOINT RESEARCH CENTRE Directorate F Health, Consumers and Reference Materials Unit F1 Health in Society European Commission Initiative on Breast Cancer (ECIBC) MEETING OF MONITORING SUBGROUP Ispra, 14 September 2016 (10:00-17:00) Minutes Document history Version Date Drafted by Comments Ahmed ELAMIN First draft proposed by RETELL Nadya DIMITROVA Minutes modified according to the comments from the participants in the meeting Nadya DIMITROVA Minutes modified according to the comments on version 1 Page: 1 of 13

2 Presentations are available on the ECIBC web hub and/or on the CIRCABC dedicated space List of Participants in Annex I The Agenda is available at The meeting was chaired by M. BROEDERS. 14/09/2016 The meeting began with a handout of Version 5 (7 Sept. 2016) of a list of questions and the comments (Annex II). Members set out to formulate the first set of PICOs based on a list of 14 potential PICO questions that had previously been sent around to members for commenting. The discussion started with clarifications regarding formulating questions in PICO format and on the potential relationship between recommendations from the monitoring subgroup and those working on screening recommendations. It was agreed to coordinate the work with the GDG Screening subgroup. Since the first set of questions from the prepared list was looking at associations between breast cancer detection ratio and breast cancer mortality, the discussion focused on how to identify the background breast cancer incidence rate, which is used as a denominator when calculating breast cancer detection ratio. The population-based screening programmes are not an appropriate source for such information, while the population-based cancer registries can provide proper estimates only during the initial years of implementation of a screening programme. In addition, breast cancer incidence is influenced by different factors, which makes it difficult to distinguish the effect of screening. Therefore, it was concluded that if comparable data on background breast cancer incidence rate is not available for each EU country, the breast cancer detection ratio is difficult to be evaluated. It was reminded that the purpose of the chapter Monitoring and Evaluation is to provide guidance on acceptable and desirable levels of performance indicators in a screening programme in order to achieve a breast cancer mortality reduction in years. The participants noted that breast cancer mortality can be influenced by other factors as well, e.g. treatment, so mortality reduction has to be evaluated carefully, because the effect of screening and treatment cannot be disentangled. However, an appropriate reference standard is needed in order to evaluate performance. Some performance measures used by established screening programmes were discussed, such as false positives rates per screening round, cumulative risk of false positive results, recalls, positive predictive value, overdiagnosis. Some of these measures are published more often than others in studies, but evaluation reports are also a valuable source in this respect, even it might be difficult to collect evidence on them. Considering these obstacles, it was admitted that the approach with formulating PICO questions might not be feasible to address this task, so the participants suggested a review of the performance data of the various screening programmes instead of using the PICO approach. Page: 2 of 13

3 They agreed that such a review would include the performance data available from any studies/evaluation reports of screening programmes that were considered to be of high quality. It was agreed that the subgroup would assess current indicators and propose new ones if appropriate. The group then reviewed and discussed the indicators in V4 of the European guidelines for quality assurance in breast cancer screening and diagnosis, section 1.9 (Epidemiological guidelines chapter) Annex III. It was noted that the indicators are not evidence-based but are based mostly on expert opinion. However, despite this, the indicators have been used by many screening programmes to assess performance. It was also noted that much has changed since the indicators were assessed and acceptable level and desirable level rates were determined, resulting in some of the levels being outdated, especially given the advances in technology and changes in the organisation of many European screening programmes. The group agreed that the next steps are: determine which indicators are still relevant and what needs to be changed or added based on results from a search for evidence. Decision for a scoping review The group asked if CCIb would be able to perform a scoping review to see what indicators were being used by other quality assessment and evaluation groups for screening and diagnosis. The group specified that the review may also identify other indicators that had not been considered by the subgroup, allowing the subgroup to have a starting list of potential indicators. The group listed potential steps: The review would look at performance indicators and surrogates being used. The scope of the review would be national /regional population-based screening programmes that have developed performance indicators, in and outside Europe. The UK, the Netherlands, Canada and Australia were suggested as countries that have good indicators based on population-based screening programmes, while Spain and Italy have good regional programmes. After the scoping review, the group would then determine a list of indicators for potential consideration; From the literature review the group would then assess whether acceptable and desirable levels could be set for performance evaluation; At another stage in the process the group would have to utilize a framework for using indirect evidence in areas where there is no direct evidence to set performance levels. Request: The group would look for guidance from the JRC/CCIb to determine if the review would be feasible instead of submitting PICOs for questions 1-7. Action: JRC and GDG will provide CCIb with a list of national / regional population-based programmes within or outside of Europe that could fall within the scope of the review. Page: 3 of 13

4 CCIb noted that the change in direction would require guidance from JRC and the group on a number of factors including on exclusion and inclusion factors for the literature, and definitions for performance indicator and outcomes, such as effectiveness, efficacy, efficiency, etc. Request: The group had asked whether IQ Healthcare can postpone the search for quality potentials in screening and diagnosis until the scoping review is done. The JRC reported that the initial reaction from IQ Healthcare was that this change was possible. The Delphi round on screening and diagnosis would then be scheduled for around May 2017.The group noted that the list of candidate indicators would then have to be ready by February CCIb might collaborate with IQ Healthcare to avoid potential overlap. The communication between CCIb and IQ Healthcare will be managed by JRC. The group suggested that by the November 2016 meeting, the monitoring subgroup would expect to have an updated list of candidate indicators to discuss and develop a protocol for defining the indicators. A pilot could be launched with a few indicators. The group would then start reviewing the literature/indicators over the next months. Discussion took place on coordination with the QASDG Indicators subgroup. Action: The group called for a combined meeting of GDG Monitoring&Evaluation and QASDG Indicators subgroup to be included in the sessions in November. The meeting should be with a few members of both groups to discuss potential overlaps. Evaluation of non-randomised studies in relation to screening studies The discussion continued with suggestions about the best study design to evaluate the long-term performance of a screening programme if it is not possible to do a randomised study. It was proposed that the ROBINS-I tool (Risk Of Bias In Non-randomised Studies - of Interventions) could be used/adapted to check how to design a study and evaluate the risk of bias. It was suggested that rather than doing a review of existing studies that the tool could be used to determine the recommended variables/domains that could be considered in developing guidance for designing a study. Action: Group to evaluate how to develop guidance using ROBINS-I. The STRATOS initiative was mentioned as another source for examining the design of non-randomised studies. Discussion of QUESTIONS 8-14 The members noted that Questions 8 and 10 are already covered by the earlier discussions. For Question 9 the group decided not to make a decision now, but will discuss questions raised by the group on intervention, outcomes and dose with all members. Questions 11 & Question 14 the group agreed that both should be developed by the group into PICOs as they are related to an independent evaluation of screening and of diagnosis respectively, and are Page: 4 of 13

5 relevant to the monitoring and evaluation chapter. However, clarification is needed on the definition of an independent evaluation process. The group has to agree about the definition of "programme evaluation", which will be provided after the meeting. Action: the group will work on clarifying the definition of an independent evaluation process/audit and then draft the two PICOs related to screening and diagnosis for questions 11 and 14. Questions the group decided to delay the work on these questions until the diagnosis group produces their questions. The group would then carry out a similar exercise. Action points Action Timeline Responsible Cross-check the eventual PICOs chosen Monitoring subgroup with the Screening and Diagnosis subgroup to ensure there are no overlaps. Group members to agree on inclusion and Monitoring subgroup exclusion criteria for the review for studies/evaluation reports to be assessed for establishing levels of performance of the relevant indicators. Draft Glossary definitions for JRC overdiagnosis and for false positive rates. Draft glossary definition of size in relation to indicators Stage II+/Total cancers screen-detected (Initial screening, Subsequent-regular screening) and Invasive cancers 10 mm/total invasive cancers screen-detected Draft Glossary definitions for prognostic factors and for predictive factors. Look for guidance from the JRC/CCIb to JRC/CCIb determine if a scoping review would be feasible instead of submitting PICOs for questions 1-7. Provide CCIb with a list of regional/national JRC/GDG screening programmes that could fall within the scope of a scoping review. Determine whether IQ Healthcare can postpone the search for evidence for performance indicators on screening and diagnosis. Consult with IQ Healthcare to see where efficiencies in the literature review could be achieved for screening and diagnosis indicators, where there is potential overlap. Request a combined QASDG indicators/gdg monitoring and evaluation meeting be included in the November sessions. The meeting should be with a few members of both groups to discuss potential overlaps. JRC to provide update on the request JRC/CCIb JRC Page: 5 of 13

6 Evaluate how to develop guidance using ROBINS-I. Clarify on the way forward on performance questions on evaluation of diagnosis. Clarify PICO 9 in terms of intervention, outcomes and dose. Clarify the definition of an independent evaluation process/audit and then draft two PICOs related to screening and diagnosis for questions 11 and 14. Monitoring subgroup/ccib JRC Monitoring subgroup Monitoring subgroup Page: 6 of 13

7 ANNEX I: List of participants Monitoring subgroup members present MIREILLE BROEDERS, LENNARTH NYSTROM, HOLGER SCHÜNEMANN, CHRIS DE WOLF, ROBERT MANSEL (ATTENDED IN THE LATE AFTERNOON) Other participants NADYA DIMITROVA (JRC), MONICA BALLESTEROS, Iberoamerican Cochrane Collaboration (CCIb); PABLO Alonso-Coello (CCIb) (attended the meeting for part of the time to discuss the proposed initial scoping review) Page: 7 of 13

8 ANNEX II: Version 5 (7 Sept. 2016) of a list of questions and the comments. VI Monitoring and evaluation of screening and diagnosis Version 5: 7 Sept We do not assume a threshold on e.g. breast cancer mortality reduction, but have agreed to phrase the PICOs as associations. This will give us the opportunity to extract a threshold from the evidence. - We will include both major benefits and harms, so we will have the opportunity to look at the balance between benefits and harms. QUESTIONS DISCUSSED AND REVISED 1. Among women aged who participate in the initial/subsequent screens 1 of a biennial screening program, what is the association between breast cancer detection ratio (i.e. cancer detection rate divided by the background 2 breast cancer incidence rate/100,000 women) and breast cancer mortality reduction 3? 2. Among women aged who participate in the initial/subsequent screens of a biennial screening program, what is the association between invasive breast cancer detection ratio (i.e. invasive cancer detection rate divided by the background invasive breast cancer incidence rate/100,000 women) and breast cancer mortality reduction? 3. Among women aged who participate in the initial/subsequent screens of a biennial screening program, what is the association between invasive interval breast cancer ratio 4 (i.e. interval cancer rate divided by the background breast cancer incidence/100,000 women) and breast cancer mortality reduction? 4. Among women aged who participate in the subsequent screens of a biennial screening program, what is the association between advanced (stage II+) breast cancer detection ratio (i.e. advanced breast cancer detection rate divided by the background advanced breast cancer incidence/100,000 women) and breast cancer mortality reduction? 5. Among women aged who participate in the subsequent screens of a biennial screening program, what is the association between small (<15mm) 5 breast cancer detection ratio (i.e. 1 We propose to use the definitions in the European Guidelines 4th edition for initial / subsequent screens. 2 Even though difficult, we propose to keep background incidence rate since there is no alternative. 3 Or should this be: a change in breast cancer mortality? 4 We propose to keep only invasive interval breast cancer ratio. 5 We propose to keep 15mm in line with the findings from Tabar et al. Page: 8 of 13

9 small breast cancer detection rate divided by the background small breast cancer incidence/100,000 women) and breast cancer mortality reduction? 6. Among women aged who participate in a biennial screening program, what is the association between breast cancer detection ratio (i.e. cancer detection rate divided by the background breast cancer incidence rate/100,000 women) and overdiagnosis 6? 7. Among women aged who participate in a biennial screening program, what is the association between breast cancer detection ratio 7 (i.e. cancer detection rate divided by the background breast cancer incidence rate/100,000 women) and the (cumulative) risk of a false-positive result? NOT DISCUSSED 8. Among women aged who participate in a biennial screening program, what proportion of double reading 8 (i.e. proportion of screening mammograms double read) is related to a breast cancer mortality reduction of >20%? 9. Among women 9 aged (or 50-74) who participate in a biennial mammography screening program, what level of image quality (defined with physical measures) and associated glandular dose level is related to an efficient/high-quality population-based screening program? (determined using outcomes selected as surrogates for overall quality such as cancer detection) 6 Do we need to specify how this is estimated, knowing that there is no consensus? 7 Is this the right determinant? Suggestions? 8 We would like to keep a PICO on double reading, but need to make sure that it is understood correctly. We do not mean to suggest that some women should have double reading whereas others do not. We could imagine that we are looking for the most efficient double reading strategy, but to word the question in that case? 9 suggestions phyisicists (Ken+ Ruben) original question Among women aged (or 50 to 74) who participate in a mammography screening program, what level of image quality (defined with physical measures) and associated glandular dose level is related to an efficient / highquality population-based screening program? (determined using outcomes selected as surrogates for overall quality such as cancer detection) "My concern about including the dose level is that no single study will easily combine dose and image quality and this may undermine the attempt to demonstrate relevance of our physical IQ measures. It would be good to discuss this with a methodologist to see if it should be split into two PICOS or if they can cope with this one as it is above. Of course once we define the desired IQ we can easily determine required doses for different types of system by combining information from different sources. If dose is included in the draft PICO at least we can then discuss how to interpret the evidence. I do not think it matters whether the programme is biennial or triennial so have deleted the frequency. Page: 9 of 13

10 10. Among women 10 aged who participate in the initial/subsequent screens of a biennial screening program, what value of <indicator> (i.e. <definition indicator>) is related to an efficient/high-quality (i.e. outcomes selected as surrogates for overall quality) population-based screening program? 11. What is the impact of an independent evaluation of outcome 11 (i.e. an external audit group independent of the administration/organization and execution of the program) vs. no independent evaluation of outcome on the quality (i.e. outcomes selected as surrogates for overall quality) of a population-based screening programs 12? 12. Among women attending diagnostic breast services with symptoms???, what level of <indicator> (i.e. <definition indicator>) is related to an efficient 13 diagnostic process? 13. Among women attending diagnostic breast services, what level of <indicator> (i.e. <definition indicator>) is related to an effective 14 diagnostic process? 14. What is the impact of an independent evaluation of outcome (i.e. an external audit group independent of the organization and execution of the breast services) vs. no independent evaluation of outcome 15 on the quality (i.e. outcomes selected as surrogates for overall quality) of the diagnostic process? Q10-12 = questions on diagnosis: For the diagnostic topics it is almost impossible to formulate the question in terms of thresholds linked to efficacy. In fact for most these topics the rationale is to avoid adverse events or reduce the unavoidable harms. This means for example that there is no a threshold for b/m ratio in open surgery that is linked with a certain outcome, but we want to minimize it at a reasonable or achievable threshold with the available technics. On the other hand this limit depends on the recommendations we will give to the 10 This PICO was meant to present a framework for including indicators like program sensitivity / (program) specificy / false-positive rate / additional imaging rate / cytology or biopsy rate with inadequate result / benign open biopsy rate / PPV of referral for assessment. Do we want to include PICOs on any of these indicators or do we just recommend monitoring? 11 We propose to limit the number of PICOs by saying the independent evaluation of outcomes/external audit group are really the same thing. Question if an institution is administering the screening program but no executing, are they considered independent? What is our definition? 12 We need to define relevant outcomes that should be assessed in the independent evaluation. 13 to be defined 14 to be defined 15 We need to define relevant outcomes that should be assessed in the independent evaluation. Page: 10 of 13

11 questions about the use of FNAC CNB VAB. Then the systematic review should answer the question which is an achievable threshold. in conclusion, questions formulated in terms of threshold for efficacy should have a strong rationale. Those on safety also, but not always safety outcomes have a rationale for a threshold different to minimization. Some indicators are linked to the diagnostic tools and pathways we are going to recommend. Suggested indicators 16 : Recall rate for further imaging/diagnostic assessment Biopsy rate (FNAC/CNB/VAB/open biopsy) Proportion of FNAC/CNB/VAB/open biopsies with insufficient results Rate of pre-operatively assured carcinomas Ratio of benign to malignant lesions diagnosed by FNAC, CNB, VAB, open biopsy Sensitivity of the diagnostic assessment Specificity of the diagnostic assessment Positive predictive value of the diagnostic assessment pt pn Time (in working days) between: screening mammography and result symptomatic mammography and result result of screening mammography and offered assessment result of diagnostic mammography and offered assessment time from biopsy to results time from biopsy to availability of biomarker results assessment and issuing of results time from surgery to results ANNEX III: Table 32 and 33 from V4 of the European guidelines for quality assurance in breast cancer screening and diagnosis, section 1.9 (Epidemiological guidelines chapter). 16 Can these be transferred into useful PICOs or should we just say these indicators should be monitored? Page: 11 of 13

12 Page: 12 of 13

13 Page: 13 of 13