Programme of community action in the field of health ( ) 2012 ACTIVITY REPORT DELIVERABLE 8: EVALUATION REPORT

Transcription

1 Page 1 Programme of community action in the field of health ( ) 2012 ACTIVITY REPORT DELIVERABLE 8: EVALUATION REPORT

2 Page 2 Description of the deliverable An end of year external evaluation report CONTENTS EXECUTIVE SUMMARY... 3 BACKGROUND AND PURPOSE... 3 EVALUATION FRAMEWORK AND METHODOLOGY... 4 RESULTS... 4 Results of survey to patients organisations... 4 Results of survey to EPNET members following the September 2012 meeting... 5 Statistics from the website... 6 Number of new porphyria labs identified through the on-line questionnaire and personal contact Summary of the results from the 2012 EQAS... 7 External review of the European porphyria registry... 8 Number of patient organisations joining the porphyria on-line community... 8 DISCUSSION OF THE RESULTS... 8 CONCLUSIONS AND RECOMMENDATIONS... 9

3 Page 3 EXECUTIVE SUMMARY An important aspect of any project focused on improving healthcare for patients is evaluation of the effectiveness of the strategy in achieving the outcomes proposed at the start of the project. External review is particularly helpful in this context as it allows evaluation against expectations of the key stakeholders, the patients served by the project and also the porphyria Centres of Expertise (pce). Surveys of both groups have identified a need for closer working and exploring areas of mutual interest such as setting up a federation of porphyria organizations. The initial steps are already being taken, as indicated by the establishment of the porphyria community on RareConnect. Review of the EPNET website usage indicates the continued value of this service both as an information source as well as a conduit for specific porphyria related enquiries. External review of diagnostic quality through the porphyria EQAS has highlighted some specific issues, particularly in relation to complex porphyria cases, which can now be openly addressed and resolved through a formal audit process, thus furthering the objective of improved diagnostic quality. The European Porphyria Registry has been established and subject to external review of data protection requirements, data security and research governance (ethics application). The EPR will continue to undergo review as more pce s in individual member states become part of this project. The overall indication from the project evaluation is that EPNET continues to deliver on its overall objective of improving porphyria healthcare across Europe. BACKGROUND AND PURPOSE The porphyrias are uncommon diseases that are rarely encountered by most healthcare professionals. Thus there is lack of uniformity in clinical practice. EPNET aims to improve porphyria healthcare across Europe. Since 2007 EPNET has built an outstanding web resource for information about the porphyrias. Quality standards and membership criteria for porphyria centres have been established and a self funding laboratory external quality assessment scheme (EQAS) has been running since Areas of current activity that add value to EPNET s initial achievements are: Extension of the network to European countries that are not part of EPNET. During the project, the number of porphyria centres applying for participation in EPNET EQAS and activity monitoring schemes has steadily increased. Informal contacts suggest that there is scope for including additional countries using the assessment methodology developed by EPNET. Furthermore not all countries with a rare disease plan have specialist porphyria centres. Improvement of diagnostic and analytical quality of centres. Results from the laboratory EQAS show a large variation among European centres with regard to applied diagnostic strategies and analytical quality of porphyria related metabolites (Aarsand AK et al Clin Chem 2011;57: ). The goal is to minimise variation. The EPNET has set up a registry to collect clinical details on patients with acute intermittent porphyria. Information has been collected in two pilot countries: France and Sweden. In 2013 the registry will be proposed to all expert centres in the network. The registry will be used for the prospective collection of clinical data from large numbers of patients to improve current knowledge in an evidence-based manner and to facilitate future clinical trials and other healthcare projects. The purpose of this evaluation report is to describe the internal and external evaluation of the activities and progress made against the indicators and target values that were defined in the 2012 work programme.

4 Page 4 EVALUATION FRAMEWORK AND METHODOLOGY The evaluation report is led by the steering committee using external and internal sources of data and information: Sources of information: 1. Patient organisations were asked to complete a short survey following the meeting on 13 th October: 1.1. What have been the most valuable learning points of this meeting for you? 1.2. Following this meeting, what services would you like EPNET to set up for the benefit of patients? 1.3. What issues/topics would you like followed up at a future meeting? 1.4. Do you have any further feedback? 2. Participants to the EPNET board meeting were asked to complete a similar survey 3. The website is analysed, by the host organisation CYIM, on a monthly basis for number of hits and unique visitors. We also follow up on the number of enquiries to the network coming from the website. 4. A number of countries in Europe do not have a recognised laboratory and clinical porphyria service. Centres were identified through personal contact, the web and scientific societies. They were asked to complete an application form to the network. The application form can be downloaded from our website. 5. Analysis of 2011 and 2012 QA schemes allows the network to monitor progress in the quality of diagnosis 6. The patient registry has been externally reviewed by various expert agencies: data protection, ethical and scientific. 7. Work with Eurordis to identify patient groups from the porphyria on-line community. Limitations of the evaluation approach: As an operating grant is only for one year this limits the feasibility of monitoring longitudinal data. RESULTS 1. RESULTS OF SURVEY TO PATIENTS ORGANISATIONS We received the following replies from patient organisations to the survey: 1.1 What have been the most valuable learning points of this meeting for you? Understanding how EPNET can help us work better

5 Page 5 RareConnect, a true European initiative 1.2 Following this meeting, what services would you like EPNET to set up for the benefit of its members? It would be great if EPNET could help in uniting the different national patient organizations under a common umbrella to address topics of joint interest. As far as services go, it would also be great to have a central point of contact that could address different types of questions. For instance, I occasionally get questions from patients outside of Switzerland asking for medical advice - I would like to be able to provide them with a local contact who could address their questions; another example concerns utilizing the physicians' network to reach out to patients in the different countries - I have found it difficult to engage patient organizations if I reach out to them directly - Going through their MDs may mobilize them more effectively. Keep us informed and help us write news from research to our members. We need to transmit reliable and pre-digested information to our association members. Organising and financing of one meeting a year with European patients groups will be great! 1.3 What issues/topics would you like followed up at a future meeting? The topic of the international porphyria federation - Patient only or Patient/Physician? What would be the benefits? Would it replace EPNET, is it an extension, a continuation? What should its charter be? How does it differentiate itself from the national patient organizations? What roles and responsibilities do the different stakeholders have?. 2. RESULTS OF SURVEY TO EPNET MEMBERS FOLLOWING THE SEPTEMBER 2012 MEETING 56 EPNET members came to the meeting and we have received feed-back from 24 participants. They replied to the survey questions: 2.1 What have been the most valuable learning points of this meeting for you? New treatment strategies Clinical case studies Registry EQAS scheme Access to experts 2.2 Following this meeting what services would you like EPNET to set up?

6 Page 6 EQAS and molecular analysis Contact with experts Forum for complex cases Training in COE's Meetings Forming of an association bank research activities Guidelines Drug list 2.3 What topics would you like followed up at a future meeting? Guidelines for lab testing Sustainability of the network New therapies Management of recurrent patients Research Registry 3. STATISTICS FROM THE WEBSITE Forty s have been exchanged during 2012 solving enquiries from patients and healthcare professionals originating from the website. The below figure summarises the number of unique visitors to the website. It shows a median number of 301 visitors per month. 4. NUMBER OF NEW PORPHYRIA LABS IDENTIFIED THROUGH THE ON-LINE QUESTIONNAIRE AND PERSONAL CONTACT. Four new centres have joined the network in 2012: Austria, Croatia, Slovak Republic and Serbia. The following diagram shows the number of labs participating in the EQA scheme from

7 Page 7 Number of labs participating in the EQA scheme st scheme 2nd scheme SUMMARY OF THE RESULTS FROM THE 2012 EXTERNAL QUALITY ASSURANCE SCHEME (EQAS) Two sample distributions were sent out during 2012, with 33 laboratories participating, including one new European and two non-european laboratories. EQAS participants receive native samples from a patient with an unequivocal porphyria diagnosis, accompanied by a clinical case history. Participants report what parameters they would normally analyse considering the case history, analytical results with information on methodology, units and reference intervals for all available porphyria-related analyses as well as interpretation of results, the diagnosis and associated laboratory report forms. The first distribution included samples from a patient with acute intermittent porphyria (AIP). Appropriate diagnostic strategies were reported by most participants, including the analytes necessary for diagnosing and differentiating the three most common acute porphyrias (Whatley SD et al Clin Chem 2009). All laboratories performing quantitative ALA and PBG analyses reported increased urinary concentrations for these metabolites. Interlaboratory CV for PBG was 25.8% with a median concentration of 14.2 μmol/mmol creatinine. In the previous EQAS distribution that included an AIP patient, the interlaboratory CV for PBG was 65.8%, with a median PBG concentration of 7.1 μmol/mmol creatinine. Twenty-eight out of the 33 responding centres correctly reported the case as acute intermittent porphyria. One laboratory reported a suspected diagnosis of acute porphyria, three reported acute hepatic porphyria, acute porphyria, probably AIP and highly suggestive of AIP and one laboratory reported the diagnosis of hereditary coproporphyria. The laboratories that did not report the definitive diagnosis of AIP had not adequately interpreted their laboratory test results (n = 2), did not perform all the relevant laboratory analyses (n = 2) or would first have required a repeat plasma scan due to an inconclusive result for this analysis (n = 1). The AIP diagnosis was by many further supported by the finding of reduced PBGD activity. Some laboratories confirmed their diagnosis by DNA analysis, identifying a mutation in the HMBS gene. The second distribution was a challenging case with samples from a 70 years old woman with variegate porphyria and skin symptoms, but who had atypical porphyrin patterns in urine and faeces. Most laboratories reported that they would have performed the necessary analyses for diagnosing and/or excluding the relevant porphyrias for this case. There were large variations in the analytical results reported from the laboratories for many of the constituents, and in some cases this resulted in the laboratory giving the wrong diagnosis. In particular, widely varying plasma emission peak wavelengths were reported and inadequately interpreted by some centres, explaining the diversity of diagnoses reported. The number of laboratories reporting the different diagnoses were: 17/33 reported definite/possible VP or VP and PCT. However 12/33 reported definite or possible PCT and 4/33 reported no porphyria or unclear diagnosis. This data clearly show that adequate interpretation and quality of the plasma fluorescence scanning are very important to reach a correct diagnosis and to work on harmonisation of this method is urgently needed. This will be achieved through an urgent audit of laboratory methodology and a report recommending best practice diagnostic guidelines.

8 Page 8 6. EXTERNAL REVIEW OF THE EUROPEAN PORPHYRIA REGISTRY Ethics approvals and patient consent were reviewed by the BPA in September The Norwegian data legislative authority has reviewed the concept and methodology. The Haukeland university hospital has externally reviewed the security of the registry. Content and the letter of agreement has been reviewed by external experts. 7. NUMBER OF PATIENT ORGANISATIONS JOINING THE PORPHYRIA ON-LINE COMMUNITY Eleven patient organisations are known to EPNET. Four organisations participated in the 2012 meeting to define objectives. During this meeting it was agreed to set up a porphyria community: Five POs are co-founders of the porphyria connect project. The community will be officially launched in May 2013 during a meeting with all POs. DISCUSSION OF THE RESULTS Patient organisations are grateful to Eurordis and EPNET for the opportunity to launch the porphyria online community. This has proven to be an invaluable service. Furthermore the expertise offered by EPNET to PO s is important in giving expert advice, reaching out to patients and providing lay translations of recent research. It also provides an ideal opportunity to share best practice across Europe, and where appropriate worldwide with associate partners and their patient communities. The services provided by EPNET to the scientific community are numerous. Different services are more important depending on the national resources and structure. For example developed countries find

9 Page 9 particularly valuable the information on research, new treatments and the registry whilst less developed countries are very focused on the access to an expert network of diagnosticians and sharing of clinical conundrums. Meetings are an essential part of the network. The EQAS is funded on a subscription basis and has successfully grown from 19 participating centres to 33. Over time we see the quality of analysis and interpretation improving. The 2012 scheme presented a challenging atypical case in which several members produced an incorrect diagnosis. This data clearly shows that although quality is improving and laboratories are addressing any possible diagnostic issues there is still an urgent need to continue collaborating on this scheme, auditing process when major issues are identified and sharing best practice through publication and the annual EPNET Assembly. The funding secured to date has allowed the development and expansion of a European Reference Network for this group of rare inherited metabolic disorders (IMD) which traditionally have not been considered part of the larger established IMD networks. All members of EPNET feel very strongly that the services must continue and that we should secure more permanent funding for the sustainability of this highly successful ERN. CONCLUSIONS AND RECOMMENDATIONS The network was formally established in EPNET is highly successful in its objective of improving healthcare for patients with porphyria and their families wherever they live in Europe. The network of participating centres of expertise on porphyrias is large and partners enthusiastically embrace the principles of widening access to and improving quality of diagnostic and clinical services. The external quality assessment scheme (EQAS) is self-funding and all partners participate and commit to improving standards and decreasing variability. Audit and assessment of the quality of laboratory diagnosis has allowed key diagnostic criteria to be defined and services to be measured against these standards which will enhance quality. Information for patients, families and non-expert clinicians is available in a large variety of European languages. Finally, the establishment of the European porphyria registry is a major step forward in understanding some of the very complex clinical and biochemical issues related to the porphyrias. The network has engendered a momentum and enthusiasm among professionals working in this field to continue and build on what has been achieved so far this is a good indicator that the network can be sustained through the understanding that collaboration is required in these rare diseases. However without European funding the network will work with reduced activity. This will inevitably imply that less developed countries or structures will be most disadvantaged when having to find national or local funding. EPNET should look for external funding for its core activities including the registry, a project manager and travel grants for less developed countries.