Guidance producer: British Fertility Society Guidance Product: Clinical guidelines Date: 13 January 2011 Final Accreditation Report
Contents Introduction... 3 Accreditation recommendation... 3 Reapplication for accreditation... 6 Appendix A: NHS Evidence accreditation analysis... 7 Appendix B: Bibliography... 18 Appendix C: Advisory Committee members, external advisers and NHS Evidence accreditation team... 19
Introduction The NHS Evidence Accreditation Scheme recognises organisations that demonstrate high standards in producing health or social care guidance. Users of the accredited guidance can therefore have high confidence in the quality of the information. Organisations can publicly display a seal of approval called an Accreditation Mark for 3 years after their processes have been accredited. The process for accrediting producers of guidance and recommendations for practice is described in the process manual 1. Accreditation recommendation NHS Evidence has not accredited the process used by the British Fertility Society to produce clinical guidelines. Background to the guidance producer The British Fertility Society is the national professional body for those working in the reproductive medicine field. The Society s membership is multidisciplinary and includes physicians, counsellors, embryologists, andrologists, endocrinologists, nurses, and other professionals from the field of reproductive medicine. The Policy and Practice Committee authors approximately three guidelines per annum for the British Fertility Society, some of which are written in collaboration with other professional bodies including the Royal College of Obstetrics and Gynaecology. 1 http://www.evidence.nhs.uk/accreditation/documents/nhsevidenceaccredmanual.pdf
Summary The Advisory Committee considered that the processes used by the British Fertility Society to produce clinical guidelines demonstrated compliance with 15 out of the 25 accreditation criteria, did not meet one and found nine not fully met. The process guide used by the British Fertility Society to produce guidelines is titled British Fertility Society: Policy and Practice Committee Guideline Development 2009. This reflects the practice that was used for the previous 3 years to develop guidance. In 2009 the process was formalised into a written document. Although the British Fertility Society produces approximately three guidelines a year this process has not yet been used to produce new or revise existing guidance. Some aspects of the British Fertility Society s guideline development processes are clearly documented and thorough. Guidance examples demonstrate implementation of many of the elements of the formal documented process. However not fully met processes exist for patient involvement, updating guidance, how the potential for bias in guidance is accounted for along with a lack of a clear conflict of interest policy. Furthermore, the process used by the British Fertility Society does not take into account barriers to implementation. The British Fertility Society acknowledged these weaknesses in its process, and has outlined improvements that it intends to make. Although welcome, these improvements are not yet in place or implemented and therefore do not resolve any of the unmet or not fully met criteria. There are several specific areas that did not meet the required standard and where improvement is needed: Address organisational and financial barriers to implementing guidance. Provide details of the full membership (not just for lead authors) for each Guideline Development Group. Define and improve the involvement of patients through relevant patient organisations such as Infertility Network UK.
Update the process to explain how recommendations balance the health benefits, side effects and risks. Include a review date for new guidance in the process and ensure it is clearly stated in guidance. Ensure that the process requires the date of evidence search and review to be stated in guidelines. Include a section in guidelines expressly for patients and amend processes accordingly. Ensure that an audit and monitoring policy is in place. Include a conflict of interest statement in all guidelines. Ensure that the process clearly explains how bias should be addressed when reaching recommendations in guidelines. The need to make improvements is recognised by the British Fertility Society. The proposed changes should be incorporated into an updated process document, and fully implemented into all new or updated guidance. David Haslam Chair, Advisory Committee December 2010
Reapplication for accreditation Following a final where accreditation has not been granted, guidance from the non-accredited producer will still be available on the NHS Evidence site but will not be identified by the accreditation mark graphic. Guidance producers that are not accredited following the accreditation process have the opportunity to reapply from 1 year after the previous assessment. It is assumed that the organisation will have addressed any concerns highlighted in the original assessment before reapplying. The NHS Evidence team will provide detailed feedback and advice on areas where improvement is required to meet the criteria in a future application.
Appendix A: NHS Evidence accreditation analysis The Advisory Committee considered the following analysis of the guidance producer s compliance with NHS Evidence accreditation criteria, which covers six discrete domains. The full analysis leading to the accreditation is shown below. Criterion Evidence for meeting the criterion Accreditation Does the guidance producer have a policy in place and adhered to that requires them to explicitly detail: 1.1 Overall objective The 2009 process document 1 states the primary aim of guidance in the Introduction section. Specific objectives for each guidance example 2,3 are also provided. Scope and purpose 1.2 The clinical, healthcare or social questions covered The process document 1 addresses the criterion in section 3 Formulation of clinical questions. The use of the patient intervention comparison and outcome (PICO) framework is suggested to help define questions. Detailed clinical questions are specified throughout the body of guidance examples 2,3 in both instances - evidence and/or recommendation grades are shown in each case.
Criterion Evidence for meeting the criterion Accreditation 1.3 Population and/or target audience to whom the guidance applies The process document 1 explains that guidelines provide systematically developed recommendations that assist clinicians and patients in making s about appropriate treatments for specific conditions. Membership of the British Fertility Society is multidisciplinary. The process document explains that clinical questions should state both patient groups and relevant outcomes. Guidance examples 2,3 reveal the target audience and detail the relevant patient population. 1.4 Guidance includes clear recommendations in reference to specific clinical, healthcare or social circumstances The process document 1 states that, where possible, recommendations should reproduce the precision of the clinical questions. Recommendations are assessed for strength and quality of evidence, and are apparent for a range of specific clinical circumstances. Numerous clinical stages such as diagnosis, interventions and prognosis are covered within each guidance example 2,3
Criterion Evidence for meeting the criterion Accreditation Does the guidance producer have a policy in place and adhered to that means it includes: Stakeholder involvement 2.1 Individuals from all relevant stakeholder groups, including patient groups, in developing guidance It was not clear from the process manual 1 whether patient representatives are included in the Guideline Development Group for each guideline. Patient involvement through consultation with Infertility Network UK (the patients main support group) could not be confirmed from guidance examples 2,3. The British Fertility Society stated that it would in future include greater detail of the individuals in each Guideline Development Group and also involve patients through Infertility Network UK. This is an important development, which will require integrating into the process documentation and be clearly evident within future guidance produced. However, without evidence of implementation in guidance the criterion is regarded as not fully met. Not fully met
Criterion Evidence for meeting the criterion Accreditation 2.2 Patient and service user representatives and seeks patients views and preferences in developing guidance Consultation with Infertility Network UK could not be confirmed from guidance examples 2,3. No evidence of patients experiences, perspectives or expectations appears to be considered by the Guideline Development Group. Not fully met The British Fertility Society agreed to involve patients more in developing guidance via Infertility Network UK but, without evidence of implementation of this update to its process, the criterion is considered to be not fully met. The process manual 1 needs to be formally updated with the changes the British Fertility Society has proposed. 2.3 Representative intended users in developing guidance. The process document 1 and guidance examples 2,3 clearly define target users of guidance so it is apparent whether recommendations are relevant to them. The process of modification based on feedback from British Fertility Society membership prior to National Executive Committee approval and publication constitutes involvement of representative intended users.
Criterion Evidence for meeting the criterion Accreditation Does the guidance producer have a clear policy in place that: 3.1 Requires the guidance producer to use systematic methods to search for evidence and provide details of the search strategy The process document 1 covers the procedures for searching for evidence. Medical Subject Headings (MeSH) and free text keywords are employed when relevant. The process recommends that databases searched, along with search terms used, should be stated in guidance produced. Guidance examples 2,3 both state search terms utilised along with databases queried. Rigour of development 3.2 Requires the guidance producers to state the criteria and reasons for inclusion or exclusion of evidence identified by the evidence review The process manual 1 explains that evidence regarded as poor quality should be excluded from the list of studies on which recommendations will be made. It states that the number of studies included and rejected should be documented. Both examples 2,3 specify inclusion and exclusion criteria. A process is apparent with implementation demonstrated. 3.3 Describes the strengths and limitations of the body of evidence and acknowledges any areas of uncertainty The process document 1 explains that previous guidelines used the US Agency for Health Care Policy and Research (AHCPR) tool. For all new guidance produced the British Fertility Society has adopted the Scottish Intercollegiate Guidelines Network (SIGN) methodology to grade evidence. Both guidance examples 2,3 used the AHCPR grading system, and specify levels of evidence and grades of recommendation.
Criterion Evidence for meeting the criterion Accreditation 3.4 Describes the method used to arrive at recommendations (for example, a voting system or formal consensus techniques like Delphi consensus) Recommendations are usually based on value judgements of guideline outcomes according to the process document 1. Previous guidelines have been developed using the AHCPR tool. The British Fertility Society stated that the SIGN grading system will be used for new guidance. Evidence from one guidance example 2 shows recommendations are reviewed and agreed using informal consensus. 3.5 Requires the guidance producers to consider the health benefits against the side effects and risks in formulating recommendations Guidance examples 2,3 show that the health benefits are balanced against the likely side effects. However only an implied process exists and the criterion is considered to be not fully met. The British Fertility Society explained that it will amend its process to include direction about how to balance the health benefits, side effects and risks which is a positive measure. Until this is formally added into an updated process document the criterion is not fully met. Not fully met 3.6 Describes the processes of external peer review Guidance is revised in line with peer review comments prior to internal review and publication in the Human Fertility journal. Although guidance examples 2,3 do not provide evidence of implementation of the peer review process, there is a clearly described process 1.
Criterion Evidence for meeting the criterion Accreditation 3.7 Describes the process of updating guidance and maintaining and improving guidance quality The process manual 1 states that guidance is considered for review and update, and outlines a policy for scheduled revision. However no review information could be located in guidance examples 2,3 and review dates are not provided. The British Fertility Society stated that it will ensure a 5-year review date is incorporated in future guidance produced. Not fully met Does the guidance producer ensure that: Clarity and presentation 4.1 Recommendations are specific, unambiguous and clearly identifiable The process document 1 explains that recommendations need to be accurate and easy to understand in order to change practice. The process document clearly states that any ambiguity in the language used could result in confusion at implementation. The guidance examples 2,3 both provide evidence that the process is being implemented and clearly detail specific recommendations. 4.2 Different options for the management of the condition or options for intervention are clearly presented Guidance examples 2,3 confirm that options for investigations, comparison treatments and drug regimens are considered when relevant.
Criterion Evidence for meeting the criterion Accreditation 4.3 The date of search, the date of publication or last update and the proposed date for review are clearly stated The process document 1 does not specify all the date information required. The date of publication was present in guidance examples 2,3 but it was not clear when literature searches had been performed or when the scheduled dates for review were to commence. Not fully met The British Fertility Society stated that it will provide the date of search and review for each guideline produced. However, the provision of this new information needs to be formally incorporated into an updated process and evidence of implementation within guidance needs to be demonstrated. 4.4 The content of the guidance is suitable for the specified target audience. If patients or service users are part of this audience, the language should be appropriate. The process document 1 states that guidelines assist clinicians and patients in making s about appropriate treatments. However it could not been confirmed whether the content of guidance examples 2,3 is suitable for patients or carers. No separate patient versions of guidance are available. The British Fertility Society indicated that an additional paragraph specifically tailored for lay people will be included in future guidance. Not fully met
Criterion Evidence for meeting the criterion Accreditation Does the guidance producer routinely consider: 5.1 Publishing support tools to aid implementation of guidance The process document 1 states that where appropriate, algorithms can be created. These are essentially schematic summaries of recommendations support tools for integration and implementation of evidence into clinical practice. The process document explains that a Summary of Recommendations section within guidance alerts the reader to key findings which also aids implementation. Applicability No costing tools or clinical care algorithms were included in the guidance examples 2,3. However examination of additional guidance confirmed that some guidelines (such as Elective Single Embryo Transfer: Guidelines for Practice Sept 2008 ) include a summary of recommendations section, a management algorithm and specific tools to assist with implementation. 5.2 Discussion of potential organisational and financial barriers in applying its recommendations The process document 1 and guidance examples 2,3 do not address the issue of organisational or financial barriers affecting implementation of recommendations. The British Fertility Society stated that organisational and financial barriers will be considered in future guidance when required. Criterion not met
Criterion Evidence for meeting the criterion Accreditation 5.3 That their guidance is current, with review criteria for monitoring and/or audit purposes within each product. The process document 1 states that audit and implementation should be included in guidance and that the Royal College of Obstetrics and Gynaecology s Understanding Audit - Clinical Governance Advice No. 5 review criteria for monitoring/auditing is recommended. However, Its Not fully met application within guidance examples 2,3 is not apparent. 6.1 Ensure editorial independence from the funding body The British Fertility Society website reveals that the organisation is essentially two limited companies, the British Fertility Society and the British Fertility Society Educational Charity Ltd. The association funds guidance production on an in-house basis. Editorial independence 6.2 Demonstrate transparency about the funding mechanisms for its guidance The British Fertility Society website shows that the two principal sponsors are pharmaceutical companies as specified at www.britishfertilitysociety.org.uk/sponsorship/index.html Sponsors provide an educational grant for general support of the Society s educational activities. There are also six corporate members of the society which contribute to the British Fertility Society Educational Charity. The agreements are transparent.
Criterion Evidence for meeting the criterion Accreditation 6.3 Record and state any potential conflicts of interest of individuals involved in developing the recommendations One of the two guidance examples 2,3 includes a declaration of interest statement. This implies a process exists for conflict of interests despite not being defined in the process document. The British Fertility Society stated that it aims to include a conflict of interest statement in each guidance document. This is a constructive development but because the process is not explicit and it is inconsistently implemented, the criterion is considered to be not fully met. Not fully met 6.4 Take account of any potential for bias in the conclusions or recommendations of the guidance The process document 1 considers the effects of bias at a number of points along the guidance authoring route. These include methods for literature searching and selection, grading of evidence and peer review. However it was not clear whether the process in place would alleviate potential bias or offset the lack of a conflict of interest policy, especially because pharmaceutical industry sponsorship is evident in some circumstances. Guidance examples did not address the issue of bias 2,3. Not fully met 1 British Fertility Society: Policy and Practice Committee Guideline Development 2009 2 Impact of obesity on female reproductive health: British Fertility Society, Policy and Practice Guidelines, Dec 2007 3 Medical Adjuncts in IVF: Evidence for clinical practice, Mar 2009
Appendix B: Bibliography Appendix B lists the additional information taken into account in the analysis and considered by the Committee. Document name Description Location British Fertility Society: Policy and Practice Committee Guideline Development, 2009 Impact of obesity on female reproductive health: British Fertility Society, Policy and Practice Guidelines (Dec 2007) Medical Adjuncts in IVF: Evidence for clinical practice (Mar 2009) 2009 Process document Not publicly available. Clinical guideline (selected arbitrarily) Clinical guideline (selected arbitrarily) Not publicly available. Not publicly available.
Appendix C: Advisory Committee members, external advisers and NHS Evidence accreditation team NHS Evidence Advisory Committee Members The NHS Evidence Advisory Committee operates as a standing advisory committee of the Board of the National Institute for Health and Clinical Excellence (NICE). The Committee provides advice to the Institute on a framework for accrediting sources of evidence that should be recognised as trusted sources of information for the NHS. The Chair of the Committee is appointed by the Institute s Board and the meetings are conducted by the Chair or in his/her absence the vice chair. The current Chair is David Haslam. A full list of the Advisory Committee membership is available on the NICE website 2. The members have been appointed for a period of 3 years. This may be extended by mutual agreement to a further term of 3 years and up to a maximum term of office of 10 years. The s of the Committee are arrived at by a consensus of those members present. The quorum is set at 50% of committee membership. The Committee submits its recommendations to the Institute s Guidance executive which acts under delegated powers of the Institute s Board in considering and approving its recommendations. Committee members are asked to declare any interests in the guidance producer to be accredited. If it is considered that there is a conflict of interest, the member(s) is excluded from participating further in the discussions. A list of the committee members who took part in the discussions for this accreditation appears below. 2 http://www.nice.org.uk/nhsevidence/nhseac.jsp
Title Name Surname Role Organisation Ms Judy Birch Lay member Ms Lynda Cox Head of Knowledge Sharing and Learning North East Strategic Health Authority Dr Brian Fisher General Practitioner NHS Alliance (GP and national patient/public lead) Professor David Haslam National Clinical Advisor Care Quality Commission Miss Ruth Liley Dr Donal O Donoghue Assistant director of Quality Improvement National Clinical Director for Kidney Care and Consultant Renal Physician Marie Curie Cancer Care Salford Royal NHS Foundation Trust Professor Jon Nichol Dean School of Health and Related Research (ScHARR) University of Sheffield Dr Mahendra Patel Senior lecturer in pharmacy practice, school of applied sciences/consultant Pharmacist University of Huddersfield Mr Adrian Reyes- Hughes Associate Clinical Director NHS Direct Dr Karen Ritchie Ms Ann Slee Dr Mark Strong Ms Gill Swash Lead Health Services Researcher Director of Pharmacy and Medicines Management MRC Fellow, Section of Public Health Head of knowledge and Library Services NHS Quality Improvement Scotland University Hospitals Birmingham NHS Foundation Trust School of Health and Related Research (ScHARR) University of Sheffield NHS Western Cheshire
Dr Sara Twaddle Director Scottish Intercollegiate Guidelines Network External Advisers for British Fertility Society accreditation application Cheryl Harding-Trestrail, Locality Commissioning Manager (West), NHS Southampton City, Southampton, UK Mik Horswell, Consultant, Healthcare Information Management HealthSystems Consultants Limited, Wisley, Surrey, UK Victoria Wilkinson, Network Manager, Primary Care Research Network - North West (PCRN-NW), Widnes, Cheshire, UK NHS Evidence accreditation team for British Fertility Society accreditation application John Huston, Accreditation Technical Analyst, NHS Evidence, National Institute for Health and Clinical Excellence, Manchester, UK Dr Paul Chrisp, Associate Director Accreditation, NHS Evidence, National Institute for Health and Clinical Excellence, Manchester, UK