Systematic Reviews and Meta- Analysis in Kidney Transplantation

Systematic Reviews and Meta- Analysis in Kidney Transplantation Greg Knoll MD MSc Associate Professor of Medicine Medical Director, Kidney Transplantation University of Ottawa and The Ottawa Hospital KRESCENT Workshop November 27, 2010

Objectives To understand the importance and necessity of systematic reviews in the conduct of research To learn the components in a meta-analysis To understand and recognize the advantages and disadvantages of meta-analyses

Definitions Number of terms are used, including: Quantitative review, systematic review, meta-analysis, overview, pooled analysis

Definitions Systematic review A review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method (Cook, Sackett, and Spitzer, 1995)

Definitions Meta-analysis A statistical analysis of a collection of studies Focus is on contrasting and combining results form different studies to identify consistent patterns and sources of disagreement among results (Rothman & Greenland, 1998)

Differences between Narrative Reviews and Systematic Reviews Cook DJ et al, Ann Intern Med 1997;126:376-380

Importance of Systematic Reviews and Meta-Analyses Can provide the highest quality evidence to guide clinical care Can help resolve current controversies in the medical community (e.g. 1 positive trial, then 2 negative trials, then ) Can discover new hypotheses (i.e. from subgroups) Identify areas where additional studies is unethical Can identify effective/non-effective/harmful therapies earlier

Cumulative metaanalysis of Aprotinin vs placebo Approximately 5000 more patients unnecessarily randomized into a trial when the answer was already known Fergusson et al, Clinical Trials 2005; 2: 218 232

Limitations Identifying all relevant trials Methodological concerns Quality of included studies Statistical methods for pooling Study level data versus individual patient data Concern that all possible sources of heterogeneity have been accounted for

Components of a Systematic Review The to do list Formulate a review question Define inclusion/exclusion criteria Locate studies Select studies Assess study quality Extract data Analyze and present results Interpret results Dissemination plan

Development of a Review Protocol The how to do it list Should include all the previous elements in a study protocol format 1. Background 2. Objective(s) 3. Research question(s) 4. Selection criteria 5. Search strategy 6. Methods 7. Implications/Impact of Review

Importance of a protocol It is important to note that SR/MAs represent secondary analyses of data collected by others i.e. they are retrospective in nature Need to state a priori the question and methods Process needs to be: well-defined thorough transparent reproducible practical/efficient

Formulating a Research Question Key components of a research question Needs to specify 1. Types of patients 2. Types of therapies 3. Types of outcomes 4. Types of studies Increase precision increase focus

Formulating a Research Question 1. Types of Patients 1 St Step: define disease/condition e.g. diabetics vs Hep C pos etc 2 nd Step: define population e.g. all kidney Tx vs ECD recipients etc 3 rd Step: define setting e.g. early vs 6-months vs 1-yr post-tx etc Need to justify decisions at each step e.g. biological rationale: Does it make sense to separate diabetics from nondiabetics? First transplants from repeat transplants?

Formulating a Research Question 2. Types of Therapies What are the types of comparisons of interest? Control groups: placebo vs usual care vs active drug Intervention: drug or technique or management process e.g. ureteric stent vs no stent e.g. prednisone vs placebo e.g. outpatient vs inpatient management of rejection

Formulating a Research Question 3. Types of Outcomes Outcomes need to be clinically relevant and meaningful to the end-user e.g. acute rejection, graft survival, death Should not solely depend on whether data is available or not Your search may yield data not previously known or highlight need for more rigorous data Do not overlook harm or safety endpoints e.g. polyoma virus infection, anemia, PTDM

Formulating a Research Question 4. Types of Study Designs Interventional RCTs gold standard for establishing effectiveness/efficacy blinded vs open-label effectiveness vs efficacy Observational Case-control, cohort, cross-sectional

Formulating a Research Question Focus of Research Question has implications on: how and where to locate studies define inclusion/exclusion criteria what data to collect how to assess quality how to analyze/present/interpret results

Poorly vs Well-formulated Question Use of tacrolimus in kidney transplantation? Versus Does tacrolimus use reduce the occurrence of graft loss at 1-year in de novo kidney transplant recipients versus those receiving cyclosporine?

Inclusion/Exclusion Criteria Analogous to setting inclusion criteria in a clinical trial Inclusion criteria needs to include: Treatment Patient population Study design Diagnosis Outcomes

Inclusion/Exclusion Criteria and Study Question Should mirror one another Certainly with regards to treatment, outcome, and population Study design may be separate as this is the method chosen to answer your research question e.g. might only want to include RCTs if looking at efficacy; cohort studies if examining for adverse effects in realworld situation

Other inclusion/exclusion items can include Type of comparators Subgroups Language Years of publication Type of publication Choice among multiple sources of a study Sample size Peer vs non-peer review Methdological quality

Key points to remember A well-formulated question drives the inclusion/exclusion criteria The inclusion/exclusion criteria drives the literature search for eligible studies

Locating Studies Where do we find studies? Published data versus unpublished Published Reference databases (Medline, EMBASE, Cochrane) Conference proceedings (abstracts) Unpublished Filing cabinets industry

Locating Studies: The Literature Search The literature search of reference databases is absolutely critical to locating studies Needs to be systematic and applicable to more than one database Get help if necessary experienced librarian, methodologist, etc

Locating Studies: The Literature Search Literature search strategy needs to be sensitive Therefore, for purposes of the search, some components of the inclusion/exclusion criteria may need to be broadened or even omitted e.g. don t try to exclude studies from search based on quality, make this assessment from the full paper once retrieved This does not necessarily mean that selection criteria changes

Components of a Systematic Review The to do list Formulate a review question Define inclusion/exclusion criteria Locate studies Select studies 2 step process Assess study quality Extract data Analyze and present results Interpret results Dissemination plan

Rating Quality RCTs Randomization Allocation concealment Blinding Withdrawals/loss to follow-up Scores vs global assessment

Once all studies are rated How do you incorporate quality? Threshold for inclusion into review Could very well be a possible cause of differences between study results (heterogeneity) (methodological versus clinical) Sensitivity analysis (i.e. exclude low quality studies) Weight overall results by study quality

Data Abstraction Rationale and Purpose: Bridge between what is reported by primary investigators and what is reported by reviewer

Data Abstraction Form Serves three important functions: 1. Visual representation of study question and inclusion/exclusion criteria 2. Provides the historical record of the review process 3. Provides the data base from which analysis can be conducted

Developing An Analytical Plan Two Types of Analysis Qualitative Quantitative Varying degrees of analytical rigor e.g. a quantitative review may be simple vote counting or a meta-analysis of pooled trials

Analytical Plan Important to have a systematic approach to analysis Need to carefully consider the important issues Plan stated a priori

Analytical Plan Considerations 1. What comparisons should be made? 2. What study results should be used in each comparison? 3. Are the results similar within each comparison? 4. What is the best summary of effect for each comparison? 5. How reliable are those summaries?

Comparisons Comparison consists of treatments and outcome Data Required Trial Identification Treatment groups Number of patients enrolled/analyzed Outcomes Number of patients with outcome in each treatment group Clinical/Methodological subgroups of interest e. g. high quality vs low quality trials; diabetics only etc

Data Binary (e.g. graft loss, acute rejection) Number of pts in each treatment arm Number of pts experiencing outcome Continuous (e.g. GFR, BP, Hgb) Number of patients in each treatment arm Measure of central tendency (usually mean) Measure of dispersion (usually SD)

Summary Measures Binary Odds ratio similar to RR when outcomes rare (<20%) Relative risk Risk Difference e.g. absolute difference in acute rejection at 1-year Continuous Weighted Mean Difference e.g. GFR at 1-yr Standardized Mean difference

When to produce a summary measure of effect? When clinically relevant to do so Probably wouldn t combine trials of induction vs no induction when examining incidence of CMV infection No large discrepancies between the studies i.e. no obvious sources of heterogeneity that will bias the results

Assessment of Heterogeneity Heterogeneity Any variability among studies in a systematic review Clinical: variability in participants (pediatric vs adult), severity of disease, intervention, drug dosage etc Methodological: variability in trial design, how outcomes defined or measured (e.g. presumed vs biopsy-proven rejection), trial quality (blinding, allocation concealment etc) Visual Statistical

Assessment of Heterogeneity Outcome: Difference in GFR Hiremath et al, AJT 7: 2350, 2007

Assessment of Heterogeneity Webster et al, Polyclonal and monoclonal antibodies for treating acute rejection episodes in kidney transplant recipients

Pooling Studies vary in size Trial summary effect based on 5000 patients is assumed to be more precise than summary effect based on 50 patients Therefore, larger studies should carry more weight in the pooling To do this we weight each study based on variance Lower variance (i.e. more precise) studies weighted more heavily more precise studies usually have larger sample size [ weight = 1/standard error 2 ]

Pooling of Trials Forest Plot FK Csa Outcome = graft failure at 3-yrs Tacrolimus vs ciclosporin as primary immunosuppression for kidney transplant recipients: meta-analysis of randomised trial data. Webster et al, BMJ, 2005

Fixed vs Random Effects Fixed Effects Model: Assumes within trial variation only Assumes that all studies come from a common population and that the true treatment effect is the same (i.e. FIXED across all studies) Inference based only on studies you have located for the analysis Therefore, the pooled summary is not significantly different among the different trials Test with visual or formal heterogeneity tests

Fixed vs Random Effects Random Effects Model: Assumes random variation within the studies and between the studies Inference based on the assumption that studies you have located come from a random sample of hypothetical population of studies Adjusts study weights according to extent of variation or heterogeneity among the treatment effects of each study More conservative (i.e. wider confidence intervals) and used more often if some degree of heterogeneity exists

Problems and Limitations of Systematic Reviews Two issues have led to skepticism of systematic reviews and meta-analyses Large trials contradicting meta-analyses of smaller trials Contradicting meta-analyses

Problems and Limitations of Systematic Reviews Two key components to help ensure a good review Understanding the strengths and limitations of the clinical and methodological quality of included trials Clinical and methodological rigor of the meta-analysis itself Did they pool trials inappropriately? Did they miss important trials?

Problems and Limitations of Systematic Reviews Publication bias Time lag bias Duplicate publication bias Citation bias Language bias Outcome reporting bias All contribute to the biased inclusion of studies in a systematic review This is SELECTION BIAS at the systematic review/metaanalysis level

Publication Bias The file drawer problem Negative studies not published Researcher and Journal Problem What is the impact? Influence on benefit Detection of harm

Publication Bias: Time Lag Bias Significant positive studies published before null studies Thus, trials with positive results could dominate the literature and could introduce bias for several years

Duplicate Publication Publication of previous study results Different language Different audience Different outcome Need to be aware and protect against double-counting e.g. ACE meta-analysis, 34 potential studies; 8 (24%) turned out to be publications based on same patients Confounded by the problem of the publication of positive studies

Language Bias Inclusion of trials only published in English can lead to a biased estimate of effect Publication in non-english language journal could be dependent on whether trial is positive or not

Summary Rigorous methods exist for the conduct and reporting of systematic reviews and metaanalyses These analyses are retrospective in nature and thus have limitations that need to be addressed

Summary However, unlike traditional narrative reviews, metaanalyses produce systematic, quantitative summaries of all the available evidence Researchers in kidney transplantation should strive for an increase in the number of rigorously conducted RCTs An increase in RCTs will inevitably lead to more systematic reviews which will allow for more accurate, unbiased estimates of treatment effect