1 March 2010 EMA/131300/2010 Human Medicines Development and Evaluation. 7 Westferry Circus Canary Wharf London E14 4HB United Kingdom

Transcription

1 1 March 10 EMA/131300/10 Human Medicines Development and Evaluation Experience of the CHMP Peer Review A tool for quality assurance of the evaluation of a new marketing authorisation application in the Centralised Procedure. 7 Westferry Circus Canary Wharf London E14 4HB United Kingdom Telephone +44 (0) Facsimile +44 (0) info@ema.europa.eu Website An agency of the European Union European Medicines, 10. Reproduction is authorised provided the source is acknowledged.

2 1. Introduction 1.1. Background The CHMP Peer Review process applies to the initial phase of the assessment of a new Marketing Authorisation Application (MAA). It is a process by which other members of the CHMP review the (Co-) Rapporteurs scientific evaluation, as well as the validity of the scientific/regulatory conclusions reached after the first phase of the EU regulatory assessment. In addition the secretariat product team which includes project managers from legal/regulatory, pharmacovigilance, safety and efficacy and quality sectors provides additional input to the peer review at this stage. Peer review was established at CHMP level as a quality assurance system. The peer review is performed by those CHMP members that are assigned by the Committee as Peer Reviewers The scope is to improve the quality of the day 1 List of Questions and also includes the task to judge the quality of the assessment reports from (Co-) Rapporteurs especially in relation to potential divergences in the scientific assessments made by (Co)Rapporteurs. During a CHMP meeting the Peer Reviewers are appointed from amongst the members of the CHMP (including co-opted members) or CHMP alternate members and are identified after having put their names forward on a nomination form. The Committee also decides on the number of Peer Reviewers to be assigned to this task. On Day 112 of the procedure, a teleconference is set up between (Co-) Rapporteurs, Peer Reviewers and staff to discuss and critically analyse the different objections and concerns raised in the (Co-) Rapporteur s Overview and draft List of Questions. On Day 1 the final Overview and List of Questions is adopted by the CHMP and is made available to applicants. A standard operating procedure ( is publicly available Previous work In 07 a project was set-up to evaluate the quality of the Peer Review assessment. The objective was to assess the practical implication of the pilot project. Therefore the Peer Review assessments between April 05 and March 07 were analysed retrospectively. The results showed that all applications (n = 140) assigned at least one Peer Reviewer. The majority of the reports covered all parts of the dossier, e. g. quality, non-clinical and clinical. Further a questionnaire was sent out to CHMP members in order to identify the strengths and weaknesses of the Peer Review assessment. The members of the committee were asked to answer 11 questions to rate on a 10-point scale the value of the Peer Review. The results showed that the Peer Review improves the Quality of the list of questions and facilitates the adoption of the list of questions at Day 1. The average score for agreement was 8.1, on a 10 point scale with 10 = strongly agree to 1 = strongly disagree. EMA/131300/10 Page 2/11

3 2. A study on the recent experience with the Peer Review 2.1. Objective There were three objectives for this study. Firstly, to document and characterise the issues raised by the and CHMP Peer Reviewers at day They were characterised according to the part of the dossier on which they were raised and further classified by the action they represent. Secondly the issues on the originality of CHMP Peer Reviewers comments were assessed. The third objective was to assess the impact of the comments raised by the CHMP and Peer Reviewers on the final List of Questions adopted by the CHMP Methods The data collection took place between March and May 09. It included new applications for products for which the List of Questions for adoption were discussed at CHMP monthly meetings (see Table 1). Table 1: Data set therapeutic group Number of products Oncology 5 Endocrinology 4 Anti-infective 3 CNS 3 One product was excluded from the analysis, due to the fact that more than one Peer Reviewer was nominated which would have biased the further analysis. 3 oncology products were added retrospectively due to an imbalance of the number of products per therapeutic group. A spreadsheet was generated to document the issues raised by the Peer Reviewers at day All comments made by the CHMP Peer Reviewer and the were outlined and were classified by the section in which the appeared and the part of the section and were codified by an action code. In this project the analysis of the quality part of the Peer Review was not included. The classification is shown in Table 2. The comments were classified according to the actions they represent. Table 3 shows typical actions for comments. All action codes were allocated by the same person and checked for consistency. Table 2: Classification of comments Section Part Actions Report non-clinical addition List of Questions (LoQ) clinical pharmacology deletion Product information (PI) clinical efficacy clarity clinical safety (dis-) agreement benefit-risk assessment format Risk-Management-Plan external consistency (RMP) Summary of Product Characteristics (SPC) internal consistency EMA/131300/10 Page 3/11

4 Section Part Actions Table 3: Typical actions Actions other (general, inspection, regulatory) Typical example modification depth of assessment up-/ downgrade of questions different interpretation addition deletion clarity (dis-) agreement format external consistency internal consistency modification depth of assessment up-/ downgrade of questions different interpretation Additional questions to the list of questions Question or part of a question to be deleted from the list of questions Context of the question is not clear, needs rewording The Peer Reviewer agrees or disagrees with one or both Rapporteurs. If the Peer Reviewer agrees with one Rapporteur, this means implicit disagreement with the other Rapporteur and vice versa The format of a part of the report is not in line with the template A part of the report is not in line with a guideline or previous submissions for similar products An issue that was raised in the report has not been translated into a question on the list of questions and needs to be added Merging of questions or typos or new suggestions to formulate a question A Part of the report has not been addressed adequately An other concern should be upgraded to a major objection or a major objection should be downgraded to an other concern The peer reviewer has a different point of view than the Rapporteur(s) In was then assessed whether any of the CHMP Peer Reviewer s issues had been raised by either Rapporteur or Co-Rapporteur, or any other CHMP member at day Comments were called original comments, if the comment was not raised by any other CHMP member and the Peer Reviewer did not agree with Rapporteur or Co-Rapporteur (because agreement implies that the comment was raised before). Finally the impact on final list of questions was identified by comparing the draft list of questions at day 112 for the teleconference with the final list of questions that was agreed upon by the CHMP. Statistical analysis was performed to assess whether the means of two groups are statistically different from each other. Therefore a paired T-Test was calculated where relevant and p-values are given in brackets. EMA/131300/10 Page 4/11

5 3. Summary of findings 3.1. Comparison of CHMP Peer Reviewer and comments An average of 28.9 comments was made by the CHMP Peer Reviewer and an average of 17.9 comments by the. The means of CHMP Peer Reviewer and comments differ significantly (p=0.047). The number of comments by the CHMP Peer Reviewer is distributed in a wider range than the comments. The CHMP Peer Reviewer makes between 6 and 53 comments per products, whereas the raises between 3 and 25 comments per product, but with one outlier where 64 comments were made Figure 1. Figure 1: CHMP Peer Reviewer and comments 60 No. of comments per product p = CHMP Peer Reviewer Further analysis shows, that the differences between the CHMP Peer Reviewer and the are mainly due to the difference of the count of comments made on the assessment report. The numbers differ significantly between the CHMP Peer Reviewer and the (p=0.013), whereas no significant difference is shown on the sections List of Questions and Product Information, cf. Figure 2. Figure 2: CHMP Peer Reviewer and comments by section EMA/131300/10 Page 5/11

6 50 p = p = p = No. of comments per product Assessment Report List of Questions CHMP Peer Reviewer Product Information Figure 3 shows significant differences in the part of the dossier where the comments were raised. The CHMP Peer Reviewer focused on comments on the non-clinical, clinical and benefit-risk assessment part, whereas the raised more comments on the Risk Management Plan and raises significantly more other comments, such as comments on regulatory or inspection issues. The CHMP Peer Reviewer and comments are hence complementary. Figure 3: CHMP Peer Reviewer and comments by part 50 p = p < p = p = p = p = No. of comments per product non-clinical aspects clinical aspects benefit-riskassessment RMP SPC other (regulatory/ inspection) CHMP Peer Reviewer EMA/131300/10 Page 6/11

7 As shown in Figure 4, significant differences exist between the CHMP Peer Reviewer and in the number of comments made on (dis-) agreement. The CHMP raised an average of 12.7 comments on (dis-) agreement (93% of them are agreement comments), whereas the make hardly any (dis- ) agreement comments. The vast majority, e.g. 71% of (dis-) agreement comments made by the CHMP Peer Reviewer were raised on the assessment report. That explains why both the number of comments raised on the assessment report and on the action (dis-) agreement significantly differs between CHMP Peer Reviewer and. In addition, the CHMP Peer Reviewer raised more comments on action like deletion, clarity, format, depth of assessment, up-/downgrade of questions and different interpretation. The focused more on modifications, additions, internal consistency and external consistency. However, the differences are not statistically significant. Figure 4: CHMP Peer Reviewer and comments by action (frequent comments only) 50 p = p = p = p < p = p = No. of comments per product addition clarity deletion dis-/ agreement CHMP Peer Reviewer modification consistency (internal and external) An analysis of the comments shows (Figure 5), that for the CHMP Peer Reviewer 85% of all comments consist of the top-5-comments (dis-) agreement, modification, addition, deletion, and clarity. Even 69% of the comments on the assessment report are (dis-) agreement comments. The comments consist of the top-5-comments modification, addition, deletion, clarity and consistency. The most frequent comments made by the are additions with a share of 31% and modifications with a share of 26%. The share of those comments on the list of questions is even higher, with 37% of modifications and 32% of additions on the list of questions. EMA/131300/10 Page 7/11

8 Figure 5: Share of comments by Peer Reviewer and sections 100% 90% 80% 70% 60% 50% 40% 30% remainder consistency clarity deletion addition modification (dis-) agreement % % % Assessment Report List of Questions Product Information Assessment Report List of Questions Product Information CHMP Peer Reviewer 3.2. Characterisation of the issues raised by CHMP Peer Reviewer The CHMP Peer Reviewer raises almost an equal number of comments on the list of questions with an average of 14.4 comments per product and on the assessment report, with an average of 13.2 comments per product. Only an average of 1.3 comments is raised on the product information. By looking more in depth on the comments made by the CHMP Peer Reviewer, it can be seen that 71% of the comments raised in the section assessment report are (dis-) agreement comments. Regarding the list of questions, the CHMP Peer Reviewer raised the highest number of comments on the non-clinical and clinical efficacy part. The comments on the List of Questions on the parts non-clinical, clinical and RMP are almost equally distributed within the 5 major comments (dis-) agreement, modification, addition, deletion and clarity Estimation of originality It was then assessed if any of the CHMP Peer Reviewer s issues had been raised by either Rapporteur or Co-Rapporteur, or any other CHMP member at day Comments were called original comments, if the comment was not raised by any other CHMP member and the Peer Reviewer did not agree with Rapporteur or Co-Rapporteur (because agreement implies that the comment was raised before). The other levels of originality were classified according to a scale from A to D, with A = original comment, e.g. only the Peer Reviewer raised this comment, B = at least one CHMP member raised similar comment or no CHMP member raised it and agreement with one of the Rapporteurs, C = at least one CHMP member raised similar comment and agreement with one of the Rapporteurs or no EMA/131300/10 Page 8/11

9 CHMP member raised it and agreement with both Rapporteurs and D = at least one CHMP member raised similar comment and agreement with both Rapporteurs. Figure 6 shows, the 38% of the comments are originally raised by the CHMP Peer Reviewer. The highest level of originality was found in the list of questions with 56% originality, followed by 19% originality in the assessment report and 15% originality in the product information. Concerning the actions, the lowest level of originality is found for agreement comments, with only 12% originality. This means that in 12% of the comments the Peer Reviewer disagreed with both Rapporteurs and no other CHMP member raised a similar comments. For the actions the highest level of originality was found for additions with 80% originality. It should be noted that if comments related to (dis)-agreements are excluded, the level of originality rose to 62% (from 38%). Figure 6: Level of originality of CHMP Peer Reviewer s comments (all 15 products) D: Low (5%) C: Medium Low (19%) A: High (38%) The comment has not been raised by other CHMP members. Peer Reviewer disagrees with both Rapporteurs B: Medium High (38%) 3.4. Characterisation of the issues raised by the The raised most of the comments on the section list of questions, with an average of 12.8 comments per product. Only an average of 3.9 comments per product is raised on the assessment reports and only an average of 1.1 comments per product are raised on the product information. The focused on the RMP and regulatory, inspection and general comments to the list of questions. When it comes to the actions of the comments, the makes a higher number of modifications, additions and comments on internal and external consistency. The origin of the addition comments is, in 60% of the comments, additional questions to the list of questions on the topic riskmanagement-plan. In the non-clinical and clinical part of the peer review report by the, more than 50% of the comments were modifications Impact of comments on the List of Questions The impact of the Peer Reviewer s comments is high (Figure 7). Overall, 71% of all comments made by the CHMP Peer Reviewer were adopted or partly adopted to the final list of questions. From the s comments, 70% were adopted or partly adopted. The proportion of adopted comments was similar between the CHMP Peer Reviewer and the, and there were only minor differences in the EMA/131300/10 Page 9/11

10 proportion of adopted comments regarding the actions. From the CHMP Peer Reviewer s comments 77% of the modifications and 51 % of the additions were adopted or partly to the final list of questions. From the s comments 76% of additions and 61% of modifications were adopted or partly adopted to the list of questions. The proportion of comments that had an impact on the list of questions was higher for comments that were less original. The higher the originality the least likely a comment is adopted to the list of questions. Figure 7: Impact of comments on the final List of Questions 100% 90% 80% 70% 60% 50% 40% 30% % 10% n=62 n=58 n=13 n=6 n=141 n=129 not adopted partly adopted adopted 0% CHMP Peer Reviewer 3.6. Conclusions The evaluation shows that the Peer Review is a successful tool to improve the list of question. The comments that were made by the CHMP Peer Reviewer and complement each other. With regard to the sections the CHMP Peer Reviewer raised more comments on the Assessment Reports than the, e.g. an average of 13.2 comments per product vs. 3.9 comments per product from the. Concerning the part of the assessment, the CHMP Peer Reviewer made more comments on non-clinical aspects, clinical aspects and benefit-risk-assessment. The raised more comments on the risk-management-plan, inspection and regulatory issues. With respect to the actions that were raised, the CHMP Peer Reviewer made many more disagreement or agreement comments, an average of 12.7 comments per product. Those came up mainly on the assessment report. In contrast, the made more additions to the list of questions, mainly on the risk-management-plan and on internal and external consistency. However, no differences were found for the count of comments on the summary of product characteristics. EMA/131300/10 Page 10/11

11 Since the last revision of the SOP for Peer Review in October 08, (SOP/H/3015), the product team members were given more detail to their secretarial tasks, especially with respect to improving the list of questions concerning clarity and format, consistency, regulatory and inspections issues and by proposing improvements for the SPC. The evaluation shows that the largely fulfilled this task, which is indicated by the number of comments raised on risk-management-plan, inspection and regulatory issues which is significantly higher than the number of comments by the CHMP Peer Reviewer. However, the relatively low number of comments raised on the SPC indicates that product team members may not lose sight of commenting on the product information. According to the SOP the CHMP Peer Reviewer focuses on the non-clinical and clinical scientific argumentation as well as the risk-benefit-assessment. To a great extend this is done on the assessment reports due to a great amount of disagreement and agreement comments. There is a noticeably wide range of the number of comments, especially regarding agreement comments on both the list of questions and the assessment reports. To improve the future Peer Reviews it should be clarified to which extent commenting is useful, especially regarding agreement comments. As mentioned in the SOP, Peer Review comments are most useful when they are both critical and constructive. The analysis of the originality of the CHMP Peer Reviewer s comments shows that there is a high degree of originality. By definition the agreement comments had lower the level of originality. When agreement comments were excluded, the degree of originality was higher, namely 62%. The highest level of original comments is found on the list of questions. Overall the comments made by both the CHMP Peer Reviewer and the are very valuable. More than 2/3 of the comments of the CHMP Peer Reviewer and were adopted to the final List of Questions. In conclusion, the results from the Evaluation of 07 were confirmed. The results show, that the Peer Review is a good instrument to improve the quality of the list of questions at day 1. The pilot phase of the Peer Review could therefore be terminated, and could be established as a regular exercise. Abbreviations CHMP LoQ MAA MO PI RMP SOP SPC Committee for Medicinal Products for Human Use European Medicines List of Questions Marketing Authorisation Application Major Objection Product Information Risk Management Plan Standard Operating Procedure Summary of Product Characteristics EMA/131300/10 Page 11/11