arxiv: v1 [stat.ot] 4 May 2018

Transcription

1 On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies S. Unkel 1, M. Amiri 2, N. Benda 3, J. Beyersmann 4, D. Knoerzer 5, K. Kupas 6, F. Langer 7, F. Leverkus 8, A. Loos 9, C. Ose 2, T. Proctor 10, C. Schmoor 11, C. Schwenke 12, G. Skipka 13, K. Unnebrink 14, F. Voss 15, and T. Friede 1 1 Department of Medical Statistics, University Medical Center Goettingen, Germany 2 Center for Clinical Trials, University Hospital Essen, Germany arxiv: v1 [stat.ot] 4 May Biostatistics and Special Pharmacokinetics Unit, Federal Institute for Drugs and Medical Devices, Bonn, Germany 4 Institute of Statistics, Ulm University, Germany 5 Roche Pharma AG, Grenzach, Germany 6 Bristol-Myers Squibb GmbH & Co. KGaA, München, Germany 7 Lilly Deutschland GmbH, Bad Homburg, Germany 8 Pfizer Deutschland GmbH, Berlin, Germany 9 Merck KGaA, Darmstadt, Germany 10 Institute of Medical Biometry and Informatics, University of Heidelberg, Germany 11 Clinical Trials Unit, Faculty of Medicine and Medical Center - University of Freiburg, Germany 12 Schwenke Consulting: Strategies and Solutions in Statistics (SCO:SSIS), Berlin, Germany 13 Institute for Quality and Efficiency in Health Care, Cologne, Germany 14 AbbVie Deutschland GmbH & Co. KG, Ludwigshafen, Germany 15 Boehringer Ingelheim Pharma GmbH & Co. KG, Ingelheim, Germany May 7, 2018 Correspondence should be addressed to: Steffen Unkel, Department of Medical Statistics, University Medical Center Goettingen, Humboldtallee 32, Goettingen, Germany ( steffen.unkel@med.uni-goettingen.de).

2 Abstract The analysis of adverse events (AEs) is a key component in the assessment of a drug s safety profile. Inappropriate analysis methods may result in misleading conclusions about a therapy s safety and consequently its benefit-risk ratio. The statistical analysis of AEs is complicated by the fact that the follow-up times can vary between the patients included in a clinical trial. This paper takes as its focus the analysis of AE data in the presence of varying follow-up times within the benefit assessment of therapeutic interventions. Instead of approaching this issue directly and solely from an analysis point of view, we first discuss what should be estimated in the context of safety data, leading to the concept of estimands. Although the current discussion on estimands is mainly related to efficacy evaluation, the concept is applicable to safety endpoints as well. Within the framework of estimands, we present statistical methods for analysing AEs with the focus being on the time to the occurrence of the first AE of a specific type. We give recommendations which estimators should be used for the estimands described. Furthermore, we state practical implications of the analysis of AEs in clinical trials and give an overview of examples across different indications. We also provide a review of current practices of health technology assessment (HTA) agencies with respect to the evaluation of safety data. Finally, we describe problems with meta-analyses of AE data and sketch possible solutions. Key words: Adverse events, benefit assessment, estimands, clinical trials, safety data.

3 1 1 Introduction and motivation 1.1 Setting the scene Although the current debate on estimands and their role in clinical trials appears to be mainly related to efficacy evaluation [1, 2], the concept is applicable to safety endpoints as well. Only very recently this perspective has found its way into publications such as Akacha et al. [3]. The analysis of adverse events (AEs) is a key component in the assessment of a drug s safety profile. Inappropriate analysis methods may result in misleading conclusions about a therapy s safety and consequently its benefit-risk ratio. A variety of methods are available for the analysis of AEs, but their complexity and the imposed assumptions clearly differ. The simplest methods for contingency tables [4], e.g. naïve proportions and derived effect measures such as risk differences, relative risks and odds ratios, presuppose identical follow-up times and usually ignore recurrent events and competing risks. If follow-up times are different across treatment groups, then comparisons based on simple incidence proportions produce biased results. Consideration of varying follow-up times by means of incidence densities is possible. However, the incidence density relies on a rather restrictive constant hazard assumption. Standard procedures for event times [see e.g. 5], in turn, are based on non-informative censoring and are not readily suitable for recurrent events and competing risks. For this purpose, more complex methods in the area of event time analysis do exist [see e.g. 5, Chapters 12 13]; for whose application, however, the data must in turn meet the corresponding prerequisites. The Working Group Therapeutic Research (ATF) of the German Society for Medical Informatics, Biometrics and Epidemiology (GMDS) and the Working Group Pharmaceutical Research (APF) of the German Region of the International Biometric Society (IBS-DR) have established the joint project group Analysis of adverse events in the presence of varying follow-up times in the context of benefit assessments. Members of the project group include statisticians from academia, pharmaceutical industry, HTA bodies and drug approval agencies. The group has identified the research requirements for the analysis of AEs at varying follow-up times in the benefit assessment of therapeutic interventions and

4 2 has elaborated solutions. The purpose of the present paper is to address the research gap in the analysis of AE data in the spirit of the current discussion on clinical trial estimands. Instead of approaching the problem of investigating AEs directly and solely from an analysis point of view, we discuss which quantities should be estimated in the context of safety data, leading to the concept of safety estimands. Within the framework of estimands, we present estimation functions for estimating the quantities of interest, that is, statistical methods that map the AE data to a single value. In this context, we also describe problems related to metaanalyses of AE data and sketch possible solutions. Finally, we provide a review of current practices of health technology assessment (HTA) agencies with respect to the evaluation of safety data. An AE is any unfavourable and unintended sign including an abnormal laboratory finding, symptom, or disease temporarily associated with the exposure to an investigational product, whether or not considered related to the product [6]. The term treatment emergent is often added to an AE as a modifier in order to remove manifestations of preexisting conditions from consideration [7]. Adverse events are documented by the investigator and coded with the Medical Dictionary for Regulatory Activities (MedDRA), which provides clinically validated medical terminology ( The MedDRA includes symptoms, diseases, diagnoses, investigation names and qualitative results, medical and surgical procedures, social and family history. Adverse events are coded with the lowest level terms. These are combined for the analyses to so-called preferred terms. The latest MedDRA version contains more than preferred terms. As the present paper focuses on methods for analyzing AEs, we do not report further on definitions related to AEs or standards for the collection and documentation of AE data. The remainder of the paper is organized as follows. To begin with, our work is motivated in the next subsection by giving a brief overview of examples with respect to the analysis of AE data in clinical trials in different indications. In Section 2, we provide a review of current practices of HTA agencies with respect to the evaluation of AE data. The estimand framework is established in Section 3. We discuss what should be estimated in the context of safety data both from a regulatory context and from an HTA perspective.

5 3 In Section 4, we provide statistical methods for analysing AEs with the focus being on the time to the occurrence of first AEs. We also describe some problems and their solutions for meta-analyses of AE data. In Section 5, we state some recommendations which estimators fit best to the described estimands. A discussion in Section 6 concludes the paper. 1.2 Motivation and examples Based on the experience with early benefit assessments by the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany we looked over some examples whether variable follow-up times for AEs between individual patients, treatment groups and studies are common across different indications. In general, trials with a primary time-to-event endpoint ususally have variable followup times for each individual patient. However, depending on the indication the average follow-up within the treatment groups but also between treatment groups can be very different, see Table 1. *** Insert Table 1 about here *** In oncology, study treatment is often given until disease progression only with limited follow-up time for AEs after discontinuation of trial treatment due to subsequent therapies, resulting in differential follow-up times for the different treatment arms and censored observations for the occurrence of AEs. Due to the dependency of follow-up time of AEs on progression-free survival, the treatment group with longer progression-free survival has a higher likelihood of observing an AE. In such situations, a simple comparison of incidence proportions between arms is biased in favour of the inferior treatment. In timeto-event trials with average follow-up times not considerably different between treatment groups, there can still be variable follow-up times for individual patients. Examples are large trials in cardiovascular disease, metabolism (diabetes) or respiratory disease (COPD) with cardiovascular outcomes or mortality as primary endpoint, where mortality is relatively low. Other trial designs than time-to-event trials with variable follow-up times were identified

6 4 in infectious diseases and central nervous system disorders. For example, there are several trials with planned trial length in Hepatitis C that allowed shortening the treatment time either for all patients in the experimental arm or for those in the experimental arm who achieved an early response. Therefore, the follow-up time in the arm with the experimental drug was shorter than in the control group, see Table 1. Here, the follow-up times not only differ between the treatment groups but also between studies of a drug in the same indication (e.g. Hepatitis C). Comparison of AEs between treatment groups can be undertaken via incidence proportions only for the duration of the shorter treatment. It is not possible to demonstrate a potential advantage in terms of lower AE probabilities over a longer period of time. On the other hand, safety evaluations using naïvely all AEs on treatment are biased in favour of the treatment group with shorter treatment duration. For instance, in multiple sclerosis, trials with fixed treatment duration have frequently been used, but if control patients switch to treatment in an extension study this would lead to longer follow-up in the experimental group and AE probabilities that are biased in favour of the control group. Figure 1 illustrates different scenarios of typical AE follow-up periods in clinical trials. *** Insert Figure 1 about here *** Adverse events are assessed on a regular basis at the visit at the beginning of the trial (V0) and during treatment (V1,...,Vn). The end of treatment (EoT) triggers a safety follow-up visit (Saf-FU) marking the last regular safety assessment. Adverse events occurring on treatment or during safety follow-up are analyzed as treatment-emergent AEs (TEAEs, marked by bold symbols). First occurrences of AEs (marked by triangles) are in general considered only when occurring during the TEAE period. Serious AEs may be reported spontaneously after the safety follow-up visit.

7 5 2 Current practice of regulatory and HTA agencies 2.1 Regulatory context Drug approval usually requires a confirmatory proof of efficacy in at least two well conducted randomized controlled trials followed by a benefit-risk assessment that shows that the treatment s benefit outweighs the expected risk associated with the new treatment. In contrast to the demonstration of efficacy as compared to a control, the benefit risk assessment is far less standardized with respect to properly balancing benefit and expected side effects. Nevertheless, details on clinical trial specifications with respect to the investigated population, the study duration and/or the number of patients to be studied are given in several regulatory guidelines, as in therapeutic EMA guidelines (the CHMP clinical efficacy and safety guidelines) and ICH guidelines [8, 9]. Safety assessments in drug approval are usually done with respect to the number and proportion of patients with specific AEs that occurred in the individual clinical trials, primarily focusing on the estimated probability of experiencing a given event within the predefined study duration potentially stratified in relevant subpopulations. It is based on a limited database available at the marketing authorization application where uncertainties about important risks may either prevent from authorization or imply the requirement of post-authorization safety investigations. Due to a number of different side effects, and since the absence of evidence of an increased risk is not necessarily evidence of absence of an increased risk [10], the analysis of AEs is often referred to as a signal detection problem. Study duration may be too short or different studies may have different durations leading to different rates. In addition the database from phase III studies may be insufficient to detect infrequent but serious events. In case of uncertainty the EMA s Pharmacovigilance Risk Assessment Committee (PRAC) may require that a post-authorisation safety study (PASS) be carried out after a medicine has been approved. Nevertheless, difficulties in the assessment at the time of marketing authorization are highly relevant especially in indications with a small patient population and a high unmet medical need imposing high pressure to the regulatory system. Hence, an assessment of safety often targets the probability of an AE within a subject and

8 6 a given period of time. The safety of a new treatment is assessed using all data available for this treatment. The FDA, e.g., asks for a document called integrated summary of safety (ISS) as described in U.S. Food and Drug Administration (FDA) [11]. In contrast to efficacy assessments, comparative safety assessments are difficult and the appraisal of side effects is usually done in absolute terms. Even if relative (comparative) safety assessments are given and relate to individual studies, accounting for different observational times due to study discontinuation, other events and the presence of changing individual proneness over time is difficult. Competing risks for a number of targeted adverse effects add another layer of complexity. For example, in two trials involving patients with type 2 diabetes and an elevated risk of cardiovascular disease, patients treated with canagliflozin had a lower risk of cardiovascular events than those who received placebo but a greater risk of amputation of toes, feet, or legs with canagliflozin than with placebo (6.3 vs. 3.4 participants with amputation per 1000 patient-years, corresponding to a hazard ratio (HR) of 1.97; 95% confidence interval: [1.41, 2.75]) [12]. In such a situation, the increased risk of amputation might be difficult to determine because of the much larger risk of mortality in comparison. 2.2 HTAs The international perspective The evaluation of safety is considered as an important element of health technology assessments (HTA) [13]. However, due to different approaches to HTA in different health care systems and little methodological guidance given generally [14], the integration of safety data and with it the analyses and interpretation of safety data differs considerably. Some health care systems, such as the one in England, focus on the economic value of a new technology by implementing a cost effectiveness threshold. Those are usually based on an incremental cost effectiveness ratio (ICER) or quality-adjusted life years (QALY) by comparing against the standard of care [15]. In this context data on AEs which are considered most relevant from the patients QoL and/or costing perspective are usually integrated by means of utility functions [16]. A

9 7 comprehensive review of the current practice of economic models concludes that there appears to be an implicit assumption within modelling guidance that adverse effects are very important. There appears to be a lack of clarity how they should be dealt with and considered for modelling purposes [17]. Other health care systems, such as the one in Germany, base their decision on the incremental medical benefit against the standard of care [18]. Usually data from clinical trials are used in order to evaluate the added benefit for all patient relevant endpoints separately to demonstrate an overall added benefit of the drug for the population in scope. Typically, a comprehensive description of AEs is provided in those assessments. However, no specific guidelines with respect to safety analyses are in place in countries following this approach. In the following, we discuss one specific example The current practice at IQWiG in Germany At the beginning of 2011 the early benefit assessment of new drugs was introduced in Germany with the Act on the Reform of the Market for Medicinal Products (AMNOG). The Federal Joint Committee (G-BA) generally commissions the Institute for Quality and Efficiency in Health Care (IQWiG) with this type of assessment, which examines whether a new therapy shows an added benefit (a positive patient-relevant treatment effect) over the current standard therapy. The IQWiG is required to assess the extent of added benefit on the basis of a dossier submitted by the pharmaceutical company responsible. In this assessment, the qualitative and quantitative certainty of results within the evidence presented, as well as the size of observed effects and their consistency, are appraised. The general methods of IQWiG are described in the Allgemeine Methoden [18, Version 5.0]. In accordance with 35b (1) Sentence 4 Sozialgesetzbuch (SGB) V, the following outcomes related to patient benefit are to be given appropriate consideration: increase in life expectancy, improvement in health status and quality of life, as well as reduction in disease duration and adverse effects. In the benefit assessment, all patient-relevant endpoints play a role, and for safety, particular consideration is given to serious and severe AEs as well as treatment discontinuations. In addition, AEs that are of special interest within the context of the disease or drug class considered may play a role.

10 8 For the assessment of the extent of added benefit the effect sizes are of main interest. An effect size in this context is defined as the (relative) difference between the new treatment and the appropriate comparator therapy. It is an important step to grade the qualitative certainty of the estimated effect size e.g. based on trial design, data quality and estimation method. Patients not included into the analyses and patients with incomplete data increase the risk of bias. Therefore, the following information is gathered and considered to assess the risk of bias: study design (randomized, open-label or double-blind), proportion of patients without consideration in the analyses per study arm, proportion of patients with incomplete data per study arm (censored data, lost-to-follow-up,...), reasons for censoring (informative, non-informative, competing risks), distribution of censoring times. Generally, this information is used to assess the direction (in favour of arm x) and the strength (low or high) of the risk of bias. In case of varying follow-up times, methods based on survival time analyses are preferred compared to analyses based on four-fold contingency tables [19]. To classify the risk of bias the number of the censored patients and the reasons for censoring have to be considered. Different types of censoring are possible: uninformatively censored, patients with competing risks and informatively censored, which influence the risk of bias. We will elaborate further on this issue in Subsection Estimands 3.1 Framework It is paramount to agree upon the relevant target of estimation defined by the question what would happen to a specific patient or what is the patient s risk with respect to a specific event or multiple events when treated with a given drug as compared to another drug or to not being treated at all. In the context of efficacy assessments, this concept has recently been introduced within the framework of estimands in the new draft addendum R1 to the ICH E9 guideline on statistical principles in clinical trials entitled Estimands and Sensitivity Analyses in Clinical Trials; this addendum is referring to the precise parameter or function of parameters to be estimated in situations where intercurrent events

11 9 as treatment discontinuation, death, rescue medication or switch to the other study treatment may influence subsequent measurements [20, 7]. We would like to stress the fact that the above-mentioned addendum is not yet finalized, hence our expositions in the sequel can only reflect the current state of discussion. Parts of the draft addendum are seen critically by HTA agencies [21]. Moreover, the addendum does not prescribe the use of a particular estimand in a certain situation. An analysis of a group of patients with further follow-up information after discontinuation of study medication would target real adherence to treatment. A treatment effect estimation at the end of treatment could be used to estimate the effect that occurs if all patients adhered. Whereas the former effect does account for real treatment exposure, the latter one is hypothetical. Similarly, the intake of additional medication could be accounted for or not, depending on the treatment effect or the estimand of interest. In this respect, the precise definition of treatment effects in clinical trial practice appears paramount and the related estimands have been debated in the regulatory context during the past years [22, 23]. In 2014, an ICH concept paper has been issued on Choosing Appropriate Estimands and Defining Sensitivity Analyses in Clinical Trials that clarifies the need for the above mentioned addendum to the ICH E9 [24]. Four different elements are required to describe the estimand of interest: the targeted population, the endpoint (variable), the intervention effect that describes how intercurrent events that potentially influences the endpoint are accounted for, and the summary measure that summarize the comparison of the two treatments under investigation. Whereas the nomenclature of types of estimands has been developed and changed during the last few years, currently, the following classes of estimands are discussed in the regulatory context: Treatment policy: The treatment policy estimand does not account for any intercurrent event. The treatment effect is measured irrespective of any intercurrent event, as treatment discontinuation or additional medication given. Composite: Composite estimands combine the variable of interest with the intercurrent event, e.g., by defining a treatment failure by the lack of response or

12 10 treatment discontinuation. Hypothetical: Hypothetical estimands target an effect that would occur in the overall population in a hypothetical scenario, in which no patient experienced the intercurrent event. E.g., the effect of all patients adhering to treatment constitutes a hypothetical effect when some patients in fact do not adhere to treatment. Prinicpal stratum: Principal strata are defined by the subset of patients in whom the intercurrent event occurs either under one of the treatments or under both. Since a group comparison trial cannot directly identify these patients with respect to the not-administered treatment, causal inference methods using specific assumptions would be required for the analysis. While on treatment: These estimands relate to the effect prior to the occurrence of an intercurrent event, e.g. before intake of rescue medication or the effect while being alive. Although the current discussion is related to the efficacy evaluation, the concept is applicable to safety endpoints, usually the occurrence of a specific side effect, as well. Considering the variable of interest as the time to a specific side effect, summary measures might be given after a specific period of time. Relevant intercurrent events are treatment discontinuation or switch, death or other side effects that may prevent from the event of interest. Whereas the basic idea of an estimand is not restricted to the efficacy assessment, different issues related to the large number of event types and the desired equivalence proof combined with the cautionary principle point to somewhat different difficulties. Envisaging the chances of a beneficial treatment in the sense of both, efficacy and safety, for a given patient may be seen as a concept of incorporating efficacy and safety in an estimand, but may be still difficult to be interpreted in the comparison of two treatments with different safety and efficacy patterns. In that sense, it appears sensible to go along the lines that are conceived for the efficacy assessment and clarify the precise parameters in the event analysis in the presence of other concurring events either in relation to the given treatment, the patient s condition or competing side effects.

13 Estimands in the regulatory context and in HTAs - what to aim for? The estimand framework is not specific to the clinical development of new drugs in the regulatory context but is also relevant to address needs of HTA bodies. The aim of the HTA process is the assessment of evidence as a basis for further decisions about reimbursement, pricing and market access. Estimands are supposed to focus and describe the research question in detail. As the aims of drug approval agencies and HTA bodies differ to some extent, different estimands may be of primary interest, but some overlap in secondary considerations can be expected. The following considerations focussing on HTA may also apply to the benefit risk assessment in drug approval. In the German early benefit assessment according to AMNOG, i.e. 35a SGB V, there is a need for the HTA authorities to identify the estimands, which are not necessarily the same as in the regulatory context to obtain marketing authorization. For marketing authorization the current practice, in general, is to report estimates for what could be considered while on treatment estimands to provide evidence for the safety profile of the treatment of interest. Specific information, however, on AEs occurring in a long-term follow-up regardless of study drug adherence may be requested in special cases, which may, however, be hampered by the limited observational period. Potentially diluting effects of treatment policy estimands in case of treatment discontinuation or switch may, depending on the treatment comparison and the disease, be anti-conservative for comparative safety assessments, hence favouring while on treatment estimands for regulatory purposes. Certainly, within the context of recent regulatory discussion on estimands in efficacy, a new regulatory framework also on estimands in safety is needed. HTA bodies are most interested in the treatment policy estimand, independently of occurrences like rescue therapy (e.g. rheumatoid arthritis) or subsequent therapies (e.g. oncology). However, in indications like oncology, the AEs are frequently only collected up to a certain point after last dose of study medication. These data do not support a treatment policy estimand. In such situations, the evidence for risk assessment is less strong. It may be difficult or impossible to cover the treatment policy estimand with the data usually obtained. To obtain sufficient data for a treatment policy estimand and

14 12 provide a solid basis for an early benefit assessment, the current practice of how data are collected in clinical studies needs to be changed [19]. A major challenge in oncology is the treatment change after progression of the disease, where in many cases patients enter a subsequent clinical study, e.g. in malignant melanoma where about four years ago the only treatment option was dacarbacin and most patients entered clinical trials after progression. These studies were under evaluation by the HTA bodies recently due to the time gap between study conduct and marketing authorization. However, in most studies a patient is not allowed to enter a new clinical study, if they are still participants of the prior study. As a consequence, they need to withdraw consent for the first study to enter the next. Therefore, AEs cannot be collected after progression for the first study in general. Exceptions from this practice are observed [25], following a protocol recording new onset of serious adverse events up to 90 days after last dose of study treatment and those serious AEs considered related any time after discontinuation of treatment. Another example consists of the German Society of Pediatric Oncology and Hematology (GPOH) [26], which records further follow-up data on children after end of treatment and with this provides the possibility for long-term surveillance and follow-up and late effect evaluation in paediatric oncology patients. Apart from these examples, the while on treatment estimand is used in clinical trials to avoid bias due to unbalanced withdrawals in the treatment groups. E.g., if the control treatment is a standard first line treatment and a subsequent study in second line requires the standard treatment as first line, then only patients of the control group are allowed to enter the second line study leading to unbalanced subsequent therapy along with biases in efficacy but also safety. Four different scenarios, which are displayed in Figure 2, can be distinguished to describe safety estimands in an HTA system. *** Insert Figure 2 about here *** The scenarios in Figure 2 differ according to the lengths of the planned and observed follow-up times in the study. The definition of estimands becomes increasingly complex with more pronounced differences in follow-up time due to intercurrent events. In the first two scenarios, the planned follow-up times of AEs in the study population are similar, e.g.

15 13 in trials with fixed trial lengths (see Subsection 1.2). In Scenario 2, however, differences in follow-up times are observed between the single patients within treatment groups or between treatment groups, due to e.g. high level of losses-to-follow-up in depression studies. Scenarios 3 and 4 consider studies with planned differences in follow-up times by treatment group, e.g. in oncology (see Subsection 1.2), where AE reporting stops a certain number of days after last dose of study drug and the time on study drug differs between treatment groups. In all four scenarios, HTA bodies usually aim for the treatment policy estimand. However, studies are commonly planned to collect data that are appropriate for the while on treatment estimand, which is usually the focus of regulatory agencies in the marketing authorization process. When benefit dossiers are based on the same studies, it is often not possible to provide estimates of the treatment policy estimand desired by the HTA agencies due to lack of adequate data. Hence, it is apparent that the described requirements by HTA bodies with regard to safety analyses need to be taken into account already in the planning phase of a clinical study. 4 Statistical methodologies The aim of this Section is to discuss methods for analysing AEs. We focus on the occurrence of the first AE of a specific type because the statistical considerations for the analysis of the first AE are relevant also for the analysis of recurrent AEs. We found a great variety of methods in the literature, some of which were not well defined, making it difficult to identify both estimator and estimand. Main methods which focussed on adverse events occurrence in one group were described in the literature as crude rate, incidence proportion, incidence rate, exposure-adjusted incidence rate, hazard function for adverse events, Kaplan-Meier and cumulative incidence function considering competing risks. We first describe methods of estimation within one treatment group in Subsection 4.1, and subsequently the comparison of AE occurrence between samples in Subsection 4.2. The time point of adverse event occurrence or comparison thereof will be made explicit. A major issue will be that safety data need not be completely observed over the whole study period for all patients and will possibly be right-censored. This requires the use of

16 14 time-to-event methodology [27, 28] taking different follow-up times into account, which is common in efficacy analyses, but less so for safety. In this context, it is often discussed which kind of censoring is informative, and the role of censoring will briefly be revisited in Subsection 4.3. Methods for meta-analyses of AE data are discussed in Subsection Methods of estimation within one treatment group One major common method is the crude rate, which is defined as ˆP(AE) = #AE n (writing #AE for the number of patients experiencing at least one adverse event of a specific type, see [28], [27], [29], [30], [31] and references therein), which is a correct estimator of the probability to experience at least one AE of the interesting type in case of complete data and identical follow-up times for all patients. With different follow-up times in different samples, the crude rate will estimate the AE probability at different points in time. This difficulty is resolved by considering the incidence proportion ([28] and references therein): ˆP(AE in [0, t]) = #AE in [0, t], (1) n which estimates the probability of experiencing at least one AE within some time-interval [0, t], but is again only valid for complete data over the considered time interval. In the presence of censoring, both the crude rate and the incidence proportion underestimate the AE probability [28]. The reason is that these methods in fact estimate the probability of both the AE occurrence and the non-occurrence of censoring. Some authors, e.g. Ioannidis et al. [30] and Amit et al. [32], therefore suggest to use one minus the Kaplan-Meier estimator for estimating P(AE in [0, t]), censoring time to AE by both the end of follow-up and by competing events that preclude AE occurrence such as death without a prior AE. However, this method overestimates the AE probability [28, 33]. The reason is that one minus the Kaplan-Meier estimator approximates a cumulative distribution function, implicitly assuming that eventually 100% of all patients experience the AE under consideration, possibly after death. Therefore, one minus the Kaplan- Meier estimator must not be used to estimate P(AE in [0, t]) in the presence of competing events in the presence of competing events which prevent the occurrence of the AE under

17 15 consideration. It is the Aalen-Johansen estimator [34] that generalizes the Kaplan-Meier estimator to multiple event types and nonparametrically estimates the so-called cumulative incidence function P(AE in [0, t]), accounting for both competing events [35, 36] and the usual censoring due to end of follow-up. Following Allignol et al. [28], the Aalen-Johansen estimator for the probability of AE occurrence is ˆP(T t, AE) = u t ˆP(T > u ) # AE at u # under observation, no AE before u, (2) where T is the time until AE occurrence or occurrence of a competing event and ˆP(T > u ) denotes the estimate of the probability of not experiencing an AE or the competing event just prior to time u. The interpretation of this estimator is that of a sum over the empirical probabilities of experiencing an AE at the observed event times. Here, for estimation of P(T > u ) the Kaplan-Meier method is used, because the definition of T encompasses all competing events. Time-to-event analyses are based on hazards, because in general follow-up times are incomplete. display, u In fact, the sum over the quotients on the right hand side in the previous # AE at u # under observation, no AE before u. (3) is the Nelson-Aalen estimator of the cumulative AE hazard t 0 α AE(u)du. For analysing AEs, the Nelson-Aalen estimator is key in three ways. Firstly, it enters the computation of the Aalen-Johansen estimator. Secondly, it is closely linked to the Mean Cumulative Function which is based on the Nelson-Aalen estimator and is also used in safety analyses [29]. Thirdly, the Nelson-Aalen estimator is the cumulative nonparametric counterpart of the commonly used incidence rate (or incidence density) of AEs [28, 30, 31], #AE Population time at risk, (4) which is an estimator of the AE hazard α AE (t) under a constant hazard assumption, α AE (t) = α AE for all times t. The incidence rate is popular, because its denominator accounts for varying follow-up times. Sometimes, the exposure-adjusted incidence rate

18 16 is reported by counting in the denominator only the population time during exposure to study treatment. However, it is not a probability estimator (and should better not be reported as a percentage). In fact, it is easily seen that depending on how time is measured (think of milliseconds or decades), the denominator can be made arbitrarily large or small, possibly resulting in values larger than one. In perfect analogy to the Aalen-Johansen estimator, translating incidence rates into probability statements requires incorporating competing events. For instance, the incidence rate of the competing event, #Competing events Population time at risk, (5) can be used to obtain a parametric counterpart of the Aalen-Johansen estimator. Comparing the AE incidence rate with the all-cause incidence rate, #AE Population time at risk #AE + #Competing events Population time at risk Population time at risk, population time at risk cancels out. In the case of complete data without censoring every patient either has an observed AE or an observed competing event, and we obtain the crude rate introduced earlier. We also note that both the Nelson-Aalen estimator and the incidence rate allow for AEs to be recurrent. In this situation, the translation into probability statements becomes more complex because of the more complicated recurrent events structure, but also with recurrent AEs competing events have to be taken into account. 4.2 Comparison of treatment groups When comparing two treatment groups with respect to AE occurrence, often measures like risk difference, relative risk or odds ratio of crude rates are suggested [e.g. 32]. However, if such relative measures are used in the presence of censoring and are based on biased onesample estimators as discussed above, the result of such a comparison will be biased too, but the direction of the bias is uncertain. For instance, a ratio of incidence proportions calculated from censored data will divide something too small by something too small. As a parametric analysis, the ratio of incidence rates is an appropriate estimator of the

19 17 hazard ratio under a constant hazard assumption. The obvious semi-parametric extension is to use a Cox proportional hazards model, α AE (t Z) = α AE;0 (t) exp(β AEZ), (6) where α AE;0 (t) is an unspecified baseline AE hazard, β AE is the vector of regression coefficients and Z a vector of baseline covariates including treatment group. In other words, if the only relevant covariate is treatment group, Z {0, 1}, then the ratio of the incidence rates, #AE in group 1 Population time at risk in group 1 / #AE in group 0 Population time at risk in group 0, estimates the hazard ratio exp(β AE ) under the assumption of a constant baseline hazard for adverse events, α AE;0 (t) constant. If this assumption is in doubt, any Cox regression software technically censoring the time to AE by observed competing events will yield the usual maximum partial likelihood estimator of exp(β AE ). Technically, censoring by observed competing events is in perfect analogy to calculation of the incidence rates, but, again in analogy to the incidence rates, it does not allow for probability statements. In other words, the analysis remains somewhat incomplete without consideration of the hazard of the competing event, e.g., via a second Cox model, α competing event (t Z) = α competing event;0 (t) exp(β competing event Z), which now technically censors the time to the competing event by observed AEs, see Beyersmann et al. [37] for a practical in-depth discussion. A reasonable method of choice will be a Cox regression model for the event-specific hazards. The important point is that it requires as many Cox regression models as there are event-specific hazards present. Although fitting two Cox models is straightforward from a computational perspective, the presence of two hazards is not without subtleties. We want to illustrate this using a toy example, assuming, for ease of presentation, constant hazards and the estimation of incidence rates. We consider a treatment that reduces the AE hazard by a factor of 0.5 and the competing event hazard by a factor of The estimator of P(AE group 1) in treatment group 1 then eventually becomes # AE in group 0 Population time at risk in group 0 # AE in group 0 Population time at risk in group #Competing events in group 0 Population time at risk in group 0,

20 18 which is larger than ˆP(AE group 0), although the AE hazard has been reduced. The reason is simple. In our toy example, treatment reduces both hazards, thus, delaying both events. Because the effect is larger on the competing event than on the AE, there will eventually be more AEs in treatment group 1 than in treatment group 0, such that the cumulative AE probabilities cross at some point in time. This is illustrated in Figure 3, showing the cumulative AE probabilities in group 0 and group 1 over time for the situation of constant hazards described above. *** Insert Figure 3 about here *** In group 0, both the AE hazard rate and the competing event hazard rate were set to 0.02 events per day, eventually leading to an AE probability of 1/2 in group 0 and of 2/3 in group 1. In group 1, the AE hazard is reduced by a factor of 0.5 and the competing event hazard by a factor of Because multiple hazards are present and an analysis of only one hazard does not suffice for probability statements, so-called direct approaches such as the Fine and Gray model for the so-called subdistribution hazard [38] or, easier to interpret, the proportional odds cumulative incidence function model [39] have been developed. Most popular is perhaps the Fine and Gray approach, which interprets one minus the cumulative incidence function as a survival function and fits a Cox model to the corresponding hazard, the so-called subdistribution hazard. The approach is useful in that a subdistribution hazard ratio greater (smaller) than one translates into an increase (decrease) of the cumulative incidence function but is otherwise difficult to interpret [40], because the subdistribution hazard, say λ(t Z), can be expressed as λ(t Z) = P(T > t Z) 1 P(T t, AE Z) α AE(t Z), which results in a complicated mixture of effects on the hazard scale and on probability scales. Alternatives include group comparisons based on confidence bands of the cumulative incidence functions [e.g. 41] or the proportional odds cumulative incidence function model [39] mentioned above. The latter is a generalization of the logistic regression model to binomial probabilities P(T t, AE Z) as a function of time t and in the presence of censoring.

21 Censoring: independent or informative? Our presentation so far has demonstrated that the analysis of AE occurrence in trials with varying follow-up times has to account for differences in follow-up, in particular in the form of censoring. Survival methodology should not only be used for efficacy, but also for safety analyses. However, the safety estimands of interest may still be a matter of debate. Above, we have demonstrated that the relationship between hazards and probabilities is more subtle when competing events that preclude AE occurrence are present. But even when this is accounted for, there may be a choice between, say incidence rates and exposure-adjusted incidence rates (which are also incidence rates, but with a different at-risk period). Censoring is a concept that pertains to all these aspects, but it is more subtle than may seem at first glance. So far, we have argued that more standard statistical techniques not from the field of survival analysis are inappropriate because the analysis will be about AEs that are observed rather than about AEs that the patient experiences. Next, we have found that one minus the Kaplan-Meier estimator censoring the time to AE by competing events overestimates the cumulative AE probability, but that such a censoring approach yields a valid analysis of the AE hazard. Whether or not censoring yields a valid analysis, has an impact on the estimand at hand, of course. When survival methods for AE analysis are discussed, authors often warn against informative censoring [e.g. 19], but this discussion on censoring is somewhat complicated by inconsistent terminology in the literature. Random censoring typically refers to the situation where time-to-event and time-to-censoring are independent random variables taking values in [0, ) [e.g. 42, p. 30]. This is also called independent censoring or noninformative censoring [e.g. 43] in the literature and it is neither uncommon that these last two terms are used interchangeably [e.g. 5] nor that they refer to different censoring mechanisms [e.g. 44]. This bedevils the discussion both on AE analyses and estimands, because one must first define what is meant by, e.g., the term independent censoring, because different authors may use the term differently, referring to different censoring mechanisms. In our context, one will rarely be willing to assume that the time to a certain AE and

22 20 the time to a competing event such as death (or progression) without prior AE are independent and present themselves as an example of random censoring. In fact, and more importantly, it is entirely unclear how to define time-to-ae for a patient who has died as the value of a random variable in the positive real numbers. Such a value would suggest that there is an AE after death, which is an awkward concept (to say the least), and we prefer an agnostic point of view. Above, we have found that observed occurrence of a competing event can be regarded as independent censoring in the sense that technically treating it as a censored observation allows for a correct analysis of the AE hazard. In other words, the analysis of the AE hazard does not depend on whether censoring was due to administrative closure of the study or whether it was due to a competing event. On the other hand, observed occurrence of a competing event can be regarded as informative censoring in the sense that technically treating it as a censored observation does not allow for a correct analysis of the AE probability as in a Kaplan-Meier procedure. These ideas are made rigorous in the counting process approach to survival analysis [45, 46, 42], see also Allignol et al. [28] for a non-technical account. In a nutshell, censoring by a competing event is independent censoring in that it preserves the desired form of the intensity of the counting process of the event under consideration, but it becomes independent, yet informative censoring if the target parameter is the cumulative incidence function. It is worthwhile to reflect on these concepts in oncology trials, where common endpoints are progression-free survival and overall survival. It is not uncommon that recording of AEs is stopped for patients who progress and undergo a second line treatment. Of course, these patients may still experience AEs, and it is generally assumed that the hazard of an AE after progression is different as compared to before progression. Progression then is a competing event for AE without prior progression. And progression is a competing event for death without prior progression and without prior AE. Hence, censoring by the progression event will yield a valid analysis of the hazards of the other two competing events, but any probability statement will need to account for all hazards involved. A different question, however, is what kind of censoring by progression is with respect to

23 21 AE occurrence after progression. In a way, the answer is easy: if censoring by progression events yields a valid analysis of AE occurrence before progression, but if the AE hazard after progression changes, progression cannot be independent (and, hence, not non-informative) censoring with respect to AE occurrence after progression. The argument can be made rigorously by showing that censoring by progression does not preserve the desired form of the intensity of the AE counting process, if the latter is not restricted to AEs before progression, but the bottom line is obvious: if recording of AEs is stopped for patients with diagnosed progression, inference for post-progression AEs is impossible. 4.4 Meta-analyses of adverse event data When data from more than one study are available it is not uncommon to naïvely pool the data across the studies by e.g. simply combin[ing] the numerator events and the denominators for the selected studies [47]. McEntegart [48] and later Rücker and Schumacher [49] as well as Chuang-Stein and Beltangady [50] warned of such naïve pooling as results might be biased due to Simpson s paradox. The International Conference on Harmonization (ICH) E9 states that any statistical procedures used to combine data across trials should be described in detail and that attention should be paid [...] to the proper modelling of the various sources of variation. Meta-analysis techniques allow for variation in baseline (control group) outcomes across the various studies, and random-effects meta-analysis in addition allows for variation in treatment effects across studies (so-called between-trial heterogeneity). Therefore, this type of models is appropriate to formally combine several studies in one analysis. In the context of safety analyses a number of specific problems arise [see e.g. 51], some of which will be considered in the following. Meta-analysis can be carried out using aggregated data of the individual studies or, if available, individual patient data (IPD). IPD meta-analyses have some advantages over aggregate data meta-analyses [52], in particular with time-to-event data considered in this manuscript. If time-to-event data are considered and the meta-analysis is based on published data, it is sometimes necessary to reconstruct the data by using appropriate methods [53, 54].