Your submission states that the adjusted indirect comparison methodology was based largely on the publication by Glenny et al

Size: px

Start display at page:

Download "Your submission states that the adjusted indirect comparison methodology was based largely on the publication by Glenny et al"

3 Clarification of the indirect comparison Your submission states that the adjusted indirect comparison methodology was based largely on the publication by Glenny et al Your methodology is as follows: An average response rate is calculated for the common comparator (MTX) across all the trials, weighted according to patient numbers. This average is referred to as the reference placebo (RP) Odds ratios are calculated for the reference placebo. The odds ratio is defined as, where Pr is the probability of response. Odds ratios are calculated for the response rates for each ACR response level (20, 50, and 70) for the trial treatment (TT) and trial placebo (TP) in a study. The formula below is applied, where adj OR is the odds ratio adjustment: The formula below is applied, where adj TT is the adjusted trial treatment response rate: Glenny et al state on page 19 that the methodology for adjusted indirect comparisons is as follows: The first methodological article explicitly discussing adjusted indirect comparisons seems to be that of Bucher and co-workers. For trials with a binary outcome they suggested combining odds ratios from separate meta-analyses, ORAB and ORAC, so that logorbc is estimated as logorab logorac, and its variance as var(log ORBC) = var(logorab) + var(logorac). From these calculations it is simple to obtain a confidence interval for logorbc and hence, by transformation, an estimate of ORBC with a confidence interval. The adjusted indirect comparison method is quite general, and this formulation is clearly a specific example of the general method described above. The approach has been used to combine trials using other effect measures, such as risk ratios, risk differences, hazard ratios or means. And page 43 as: In brief, the indirect comparison of interventions A and B was adjusted by the results of their direct comparisons with a common intervention C. Suppose TAC is the estimate of direct comparison of intervention A versus C, and TBC is the direct comparison of intervention B versus C. Then the estimate of the adjusted indirect comparison of intervention A versus B (T_AB, e.g. log relative risk, mean difference) is estimated by T_AB = TAC TBC and its variance is Var(T_AB) = Var(TAC) + Var(TBC) This adjusted method aims to overcome the potential problem of different prognostic factors between study participants in different trials, and it is valid if the relative efficacy of interventions is consistent in patients across different trials. This does not appear to correspond with your method (see above). Please clarify the following: 1) Source of this method and the rationale for not following the method in the monograph.

4 2) How the formula below is an odds ratio since it looks at one treatment arm rather than both 3) why you pooled the placebo (methotrexate) arms to form the reference placebo (RP) when this obviously destroys randomisation? Please provide the calculations and spreadsheet that were used to compile table 19.

Mr. Meindert Boyson Associate Director Single Technology Appraisals National Institute for Health and Clinical Excellence (NICE) Peter House, Oxford Street, Manchester M1 5AN Liverpool Reviews &

Appraisal Rituximab for Rheumatoid Arthritis Dear Meindert, The Liverpool Reviews and Implementation Group (LRiG) has now had an opportunity to make an initial assessment of the Roche submission for

5 Mr. Meindert Boyson Associate Director Single Technology Appraisals National Institute for Health and Clinical Excellence (NICE) Peter House, Oxford Street, Manchester M1 5AN Liverpool Reviews & Implementation Group ********* ***************** ***************** Telephone: **************** Facsimile: **************** *************** **************** 20 March 2007 Re: Single Technology Appraisal Rituximab for Rheumatoid Arthritis Dear Meindert, The Liverpool Reviews and Implementation Group (LRiG) has now had an opportunity to make an initial assessment of the Roche submission for this appraisal. In general, the quality of the submission is disappointing. The presentation is inconsistent; the majority of tables have been pasted into the document from other sources. A large number of these tables are of poor quality and difficult to read, some are incomplete and others do not match the accompanying text. The text of the document contains an over abundance of errors and does not appear to have been copy edited. Formatting of references is inconsistent in the text and in the reference list with some incomplete references. Of greater importance are concerns regarding the presentation and structure of the model that will preclude us from being able to draw any reliable inferences from the company submission. These concerns are outlined here. Model Design The economic model, submitted by Roche to support claims of cost-effectiveness for Rituximab in the treatment of severe rheumatoid arthritis, has been constructed using Microsoft Excel. It is described as a micro-simulation. This means that it involves generating 10,000 typical patients and tracing their simulated individual treatment histories when subject to two different treatment strategies (a conventional sequence of treatments vs. a sequence with rituximab used before proceeding to the conventional sequence). The implementation of the selected model structure is through the use of a complex set of Visual Basic for Applications (VBA) procedures. The programming code is quite sophisticated, but lacks detailed annotation. No description or schematic of the coding design is provided which might help another analyst to readily understand the algorithms employed. In particular, the designers have used a large number of 1

6 temporary variables with non-intuitive labels, whose function(s) are not easy to deduce. Design Faults A preliminary examination of the model and its VBA code has identified several important problems which are likely to affect the reliability of model results. However, two are particularly serious since they involve structural elements of the model and reveal a lack of methodological awareness on the part of the model developers: 1) The most significant problem arises from the manner in which random numbers are assigned during model execution to govern the response to treatments, the timing of death and other key variables in the model. Instead of allocating random numbers to patients, random numbers are allocated to treatment cycles. Since the intervention strategy involves starting the conventional treatment sequence at least one cycle later than in the comparator strategy, this means that different random numbers are used to govern the experiences of the same patient under the two strategies. This has several important consequences. The most obvious is that there is no guarantee that a patient who responds to treatment X in the intervention strategy will also respond to treatment X in the comparator strategy, or will respond to the same extent. This then has knock-on consequences for the risk of death which, when combined with a similar disparity in the random numbers governing the incidence of death, leads to a significant number cases exhibiting wild swings in estimated life expectancy (both positive and negative) at the level of individual patients - i.e. false additional survival or false loss of survival is influencing model results. 2) The generation of alternative parameter sets for probabilistic sensitivity analysis (PSA) demonstrates a misunderstanding of the difference between 1 st and 2 nd order variations. In general the parameter sets are subjected to variations which are governed by the estimated standard deviation of each variable, rather than the standard error of each estimated statistic. Additionally, for two-parameter (Weibull) distributions a strange procedure has been attempted which seems to involve sampling from the primary distribution without changes to the parameter estimates, consideration of sample size, or appeal to parameter covariance. Consequences The effect of problem (1) is that the potential benefits of the individual patient simulation are lost, and the analysis reduces to a cohort comparison. However, it is not possible to determine whether or not the cost-effectiveness results so generated are likely to be biased in any direction, or in relation to any specific aspect of the model (drug costs, other costs, life expectancy, QALYs, etc.), since the potentially large artefactual discrepancies generated are not easily predictable. Problem (2) is such a fundamental failing that the PSA results (scatterplots and CEACs) are of no practical value for decision-making, and serve only as an object lesson in the strange patterns that can be produced when PSA methodology is not applied correctly. Remedies Several additional problems have been identified which could be corrected reasonably easily by the ERG (e.g. by use of improved parameter values). However, the two problems discussed above are of a more far-reaching nature. 2

7 The complexity of the VBA coding of the model, and the lack of comprehensive model documentation make it unrealistic for the ERG to consider any sort of salvage exercise to correct problem (1) since it requires significant restructuring of parts of the model and probably the reprogramming and careful re-testing of key portions of the core model logic. Implementing a reasonably sound PSA procedure for this model (problem (2)) requires gaining access to key analytical information which derives from analysis of the original patient data for each of the two-parameter models employed (patient numbers and covariance matrix). Assistance from original study authors would need to be sought, since this information is not routinely published. Conclusions The basic structure of the submitted model is not particularly complex or elaborate. Unfortunately, its implementation has suffered from serious flaws which are immediately apparent from a preliminary assessment. These are sufficiently severe as to make it impossible for the ERG to determine whether the reported costeffectiveness results merit any degree of credibility for decision-making. It is the view of the ERG that the current Roche model cannot be rendered useable within the resources and time available to the ERG. It should be noted that none of the above comments should be taken as indicating any opinion as to whether or not the case for cost-effectiveness of Rituximab could be sustained if reliable model evidence were to be provided. The difficulty for the ERG is that the seriousness of the design problems so far identified is sufficiently great as to prevent any reliable inferences being drawn from the results presented. A number of options could be considered including but not limited to withdrawal and re-submission of the model. This however is likely to be a lengthy process and it would not be possible for the ERG to assess any new submission within the current timelines for this appraisal. In the event that a new model is made available then we have also compiled a list of issues related to other sections of the submission that would need to be addressed. These are included on the following pages. I look forward to hearing from you. Yours sincerely ************* Director 3

8 Section A: Clarifications of the effectiveness data A1) Definitions There are no definitions of Standard of Care or placebo LRiG request the definition of standard of Care and details on what constituted the placebo. A2) Inclusion /exclusion The submission includes inclusion/exclusion criteria for patients entering the pivotal phase III study (WA17042), however, there is no explanation as to why patients with ACR functional class IV disease have been excluded. Furthermore it is stated in section that the patient population consisted of RA patients who have had an inadequate response to: 1) at least one anti tumour necrosis factor (TNF inhibitor) therapy and 2) no more than five disease modifying anti-rheumatic drugs (DMARDs). This is then contradicted in section where it is stated that the maximum number of previous DMARDs was 9 in the placebo + MTX group and 8 in the rituximab + MTX group LRiG request an explanation of the rationale for excluding patients with ACR functional class IV disease and why patients who had received more than five DMARDs were included. A3) Differing sizes of patient populations The values of n change for the different analyses due to withdrawals, crossovers and pooling of different studies (as highlighted in Table 1). Not only are the reasons for these differences complex there are occasions where it is not possible to determine why certain patients were included and excluded e.g. the long term efficacy following repeated treatments. Table 1: Size of populations for each of the presented analyses Rituximab Placebo +MTX + MTX Total Primary analysis n=298 n=201 n=499 Sub-group analyses o US Vs Non-US o RF+ Vs RF- n=298 n=201 n=499 o 1 anti-tnf Vs 2 or more anti-tnfs n=298 n=201 n=499 Long-term efficacy after one course of rituximab n=308 n=209 n=517 Radiographic endpoints after 56 weeks n=277 n=186 n=463 Long-term efficacy following repeated courses n/k n/k n=281 4

9 Analysis of adverse events n=540 n=398 n=938 Analysis of infusion reactions n/k n/k n=1039* Analysis of infections n/k n/k n=1039* * For first course, n=570 for second course, n=191 for third course and n=40 for fourth course LRiG request flow diagrams for each value analysis showing the number and criteria for inclusion/exclusion at each and every entry and exit point. This will enable validity checking of the populations. A4) Indirect comparisons For purposes of indirect comparisons with studies that have the common comparator of MTX ACR20 responses for nine trials are shown and then adjusted. How were these trials identified and selected and are all relevant trials for each drug included? It is also assumed in these analyses that gold, CSA and palliative care are equal to MTX; on what is that assumption based? LRiG request details of the literature searches and inclusion/exclusion criteria of the trial selection for the indirect comparisons and an explanation as to why equivalent efficacy is assumed for gold, CSA and palliative care. A5) Missing data There are several omissions of important data. The submission does not include details of: the age range of patients, the number of fatalities due to infections in the all exposure population (Section 5.7 states there were infections some of which were fatal ), any 95% confidence intervals, and not all P values are reported. LRiG request the range of ages in the ITT population and the number of fatalities due to infection. Further questions may arise when our statistical support team have had an opportunity to complete their examination of the submission. A6) Formatting errors The formatting throughout the report is poor and in the case of Table 15 it is not possible to read all the information in the table. In addition Table 11 has no relevance to any text in the submission. LRiG request a complete copy of Table 15 and either the correct Table 11 or explanatory text. A7) General queries Radiographic endpoints are presented at 24 and 56 weeks in submission, why not at 48 weeks? LRiG request an explanation as to why radiographic endpoints at 48 weeks are not shown. 5

10 Section B: Economic Analysis B1) Early withdrawals There is no information in the submission concerning the timing of, and reason for, withdrawals in the two arms of the trial. LRiG request a detailed table of withdrawals by week and by reason (serious AE, reaction, clinical advice, patient request, lack of efficacy, etc). B2) Inter-treatment interval A mean of 307 days is quoted in the submission for the time between doses of rituximab. This is used to estimate the cost per treatment. LRiG request a detailed table AND Kaplan Meier analysis of the time to second treatment in the intervention arm of the trial, censoring all treatment terminations. We also request the provision of total numbers of first and second treatments given in the intervention arm of the trial. B3) Half cycle correction Roche claim that a half cycle correction has been applied to their model. However, it is not clear where or how this correction has been applied. LRiG request details of where the half cycle correction features in the VBA code and how this is done. B4) Inpatient resource use In the submission inpatient resource use is derived from NOAR data grouped into 6 HAQ strata. It appears that inpatient days relate to all episodes, but exclude the cost of any surgery carried out whilst in hospital. LRiG request clarification of how surgical episodes and bed days were handled. 6

1 Executive summary. Background

1 Executive summary. Background 1 Executive summary Background Rheumatoid Arthritis (RA) is the most common inflammatory polyarthropathy in the UK affecting between.5% and 1% of the population. The mainstay of RA treatment interventions