CONCORDANCE BETWEEN PRIMARY AND SECONDARY OUTCOMES IN COCHRANE REVIEWS AND CLINICAL TRIALS - JOCHEN SCHMITT -

Transcription

1 CONCORDANCE BETWEEN PRIMARY AND SECONDARY OUTCOMES IN COCHRANE REVIEWS AND CLINICAL TRIALS - JOCHEN SCHMITT - Annual Cochrane Skin Group & CSG-COUSIN Meeting 2018 Amsterdam, 15th January

2 Meta-epidemiological study Objectives I the primary aim of this study was to systematically investigate the concordance between Cochrane review outcomes and the outcomes investigated in the corresponding clinical trials I Our secondary aim was to conduct a meta-epidemiologic study to investigate whether the pooled treatment effect differs between trials with the primary outcome concordant vs discordant with the primary review outcome. 2

3 Methods I Study design Meta-epidemiologic study I Data source and data collected A random sample of 10 systematic reviews out of all 64 CSG reviews of the Cochrane Skin Group published until 01/2015 The 10 CSG systematic reviews included 242 primary studies (range 7-85) 3

4 Methods I Data extraction from reviews and trials Independently and in duplicate by two reviewers 6 review teams Christian Apfelbacher & Susanne Brandstetter Sanna Prinsen & Phyllis Spuls Jan Kottner & Elisabeth Hahnel Tobias Weberschock & Andreas Dreher Jochen Schmitt & Lena Jacobi Matthew Grainge & Esther Burden-Teh, Natasha Rogers All data accessible through Archie were used Additional information was abstracted from Cochrane Reviews Mapping of outcome domains by two reviewers (Jochen Schmitt & Toni Lange) in an iterative stepwise process 4

5 Methods Cochrane Archi data CSG COUSIN data extraction via standardised extraction forms 5 Data preparation process Data cleaning Harmonisation Standardisation

6 Results 6

7 Included Cochrane reviews Trials* Outcomes 1 Bamford 2013: Oral evening primrose oil and borage oil for eczema Bath-Hextall 2012: Dietary supplements for established atopic eczema, Chen 2013: Narrow-band ultraviolet B phototherapy versus broad-band ultraviolet B or psoralen-ultraviolet A photochemotherapy for psoriasis 4 Chi 2011: Topical interventions for genital lichen sclerosus Eekhof 2012: Interventions for ingrowing toenails Ersser 2014: Psychological and educational interventions for atopic eczema in children Kastarinen 2014: Topical anti-inflammatory agents for seborrhoeic dermatitis of the face or scalp. 8 Kwok 2012: Topical treatments for cutaneous warts Martin 2009: Interventions for pemphigus vulgaris and pemphigus foliaceus Sasse 2007: Chemoimmunotherapy versus chemotherapy for metastatic malignant melanoma * Trials included in meta-epidemiologic study

8 Results I 21 primary efficacy outcomes in 10 Cochrane reviews I For 5 of these efficacy outcomes (24%) not assessed in a single trial. I Another 6 review outcomes were assessed in less then 50% of trials. I Only 1/21 primary review outcome was reported in all corresponding trials. I For 11/21 primary review outcomes (52%) meta-analysis could not be performed because outcomes were not reported at all, imprecisely defined, or measured inappropriately in the review trials. Very low overlap between primary review outcomes and trial outcomes. 8

9 Results 10 / 10 Cochrane review teams (100%) identified outcome assessment and reporting in trials as barrier for evidence synthesis I Bamford, et al A major limitation of the current evidence is the variation in how outcomes were reported, thus, making it impossible to pool all studies measuring the same outcome in meta-analyses. I Ersser, et al We could not synthesise data from these studies because of the following factors: the heterogeneous nature of the outcome measures used, a lack of adequate data (both in quality and accessibility), and methodological weaknesses in study design. 9

10 Results definition of primary outcomes in reviews Reference Condition Primary review outcome Bamford, et al. 1 atopic eczema - Global degree of improvement in symptoms and signs as rated by participant or medical doctor - Improvement in quality of life Bath-Hextall, et al. 2 atopic eczema - Degree of long-term (over six months) control, such as reduction in number of flares or reduced need for other treatments - Changes in participant-rated or parent-rated symptoms of atopic eczema, such as pruritus (itching) or sleep loss (short-term within six weeks) Chen, et al. 3 psoriasis - Participant-rated global improvement - Percentage of participants reaching Psoriasis Area and Severity Index (PASI) 75 - Clearance rate. (Clearance was defined as no lesions of psoriasis or minimal residual activity Chi, et al. 4 lichen sclerosus (genital) - Participant-rated improvement or remission of symptoms (in terms of quality of life, pain, itching, and less pain with intercourse) - Investigator-rated global degree of improvement (in terms of pallor, purpura, hyperkeratosis, ulceration, erosion, erythema, sclerosis, and scarring). Eekhof, et al. 5 ingrowing toenails - Relief of symptoms - Regrowth (including nail spicules/nail spikes) - Recurrence Ersser, et al. 6 atopic eczema - participant-rated global assessment - Reduction in disease severity as measured by a trained assessor using an objective measure - Improvement in sleep - Improvement in quality of life (or reduction in distress of the child and parent) Cochrane reviewers often did not precicisely define primary review outcomes 10

11 Results - concordance from the perspective of the reviews Reference Condition Primary review outcome Trials specified primary review outcome as primary study outcome, secondary study outcome, unspecified Bamford, et al. 1 atopic eczema - Global degree of improvement in symptoms and signs as rated by participant or 4/23 (17%); 11/23 (48%); 12/23 (52%) medical doctor - Improvement in quality of life 0/2 (0%); 1/2 (50%); 1/2 (50%) Bath-Hextall, et al. 2 atopic eczema - Degree of long-term (over six months) control, such as reduction in number of flares 0/3 (0%); 0/3 (0%); 3/3 (100%) or reduced need for other treatments - Changes in participant-rated or parent-rated symptoms of atopic eczema, such as 0/6 (0%); 0/6 (0%); 6/6 (100%) pruritus (itching) or sleep loss (short-term within six weeks) Chen, et al. 3 psoriasis - Participant-rated global improvement NOT ASSESSED IN ANY TRIAL - Percentage of participants reaching Psoriasis Area and Severity Index (PASI) 75 0/1 (0%); 0/1 (0%); 1/1 (100%) - Clearance rate. (Clearance was defined as no lesions of psoriasis or minimal 0/7 (0%); 0/7 (0%); 7/7 (100%) residual activity Chi, et al. 4 lichen sclerosus - Participant-rated improvement or remission of symptoms (in terms of quality of life, 0/4 (0%); 1/4 (25%); 3/4 (75%) (genital) pain, itching, and less pain with intercourse) - Investigator-rated global degree of improvement (in terms of pallor, purpura, 0/6 (0%); 1/6 (17%); 5/6 (83%) hyperkeratosis, ulceration, erosion, erythema, sclerosis, and scarring). Eekhof, et al. 5 ingrowing toenails - Relief of symptoms 0/8 (0%); 0/8 (0%); 8/8 (100%) - Regrowth (including nail spicules/nail spikes) 1/2 (50%); 0/2 (0%); 1/2 (50%) - Recurrence 2/17 (12%); 0/17 (0%); 15/17 (88%) - Very low concordance between primary review and primary trial outcomes - In all 10 sets of trials majority of outcomes was not specified as primary or secondary Confirmatory validity of these trial outcomes is unclear! Meta-epidemiologic analysis could not be performed 11

12 Results concordance from the perspective of the trials On the domain level, there seems to be quite good overlap between the most frequently assessed trial outcomes and review outcomes (based on a very broad outcome domain classification system) 12

13 WHAT is measured construct 13 Boers M, Kirwan JR, Wells G, et al. Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0. Journal of clinical epidemiology 2014;67(7):

14 Results WHAT is measured construct Reference Condition Number of trials Number of outcomes Death n trials (%); n outcomes (%) Life Impact n trials (%); n outcomes (%) Pathophysiological Manifestations n trials (%); n outcomes (%) Resource Use/Economical Impact n trials (%); n outcomes (%) Bamford atopic eczema (0%); 0 (0%) 20 (77%); 74 (39%) 25 (96%); 110 (57%) 7 (27%); 8 (4%) Bath-Hextall atopic eczema (0%); 0 (0%) 6 (55%); 28 (28%) 11 (100%); 62 (63%) 3 (27%); 9 (9%) Chen psoriasis (0%); 0 (0%) 2 (15%); 2 (3%) 13 (100%); 76 (97%) 0 (0%); 0 (0%) Chi lichen sclerosus (0%); 0 (0%) 6 (86%); 13 (50%) 6 (86%); 13 (50%) 0 (0%); 0 (0%) (genital) Eekhof ingrowing toenails (0%); 0 (0%) 11 (52%); 28 (30%) 21 (100%); 57 (61%) 5 (24%); 8 (9%) Ersser atopic eczema (0%); 0 (0%) 7 (78%); 23 (57%) 7 (78%); 16 (40%) 1 (11%); 1 (2%) Kastarinen Kwok Martin Sasse 14 seborrhoeic dermatitis (face and scalp) common warts cutaneous warts (nongenital) pemphigus vulgaris and pemphigus foliaceus metastatic malignant melanoma (0%); 0 (0%) 32 (91%); 80 (38%) 35 (100%); 129 (61%) 0 (0%); 0 (0%) (0%); 0 (0%) 2 (3%); 3 (2%) 66 (94%); 145 (97%) 2 (3%); 2 (1%) (0%); 0 (0%) 1 (10%); 1 (2%) 10 (100%); 44 (75%) 6 (60%); 14 (24%) (89%);44 (32%) 3 (17%); 3 (2%) 18 (100%); 92 (66%) 0 (0%); 0 (0%)

15 Results - WHAT is measured construct Mapping of OMERACT 2.0 Filter on outcomes of all 10 sets of trials 16

16 Core Area (OMERACT 2.0) Domain Sub Domain Item Life Impact Acceptance of care Compliance / adherence Coping 17 Activity participation Daily life impact Family impact Sports activity participation Itching behavior (scratching) Skin (care) behavior Burden for mother Mother child interaction Parental QOL Patient satisfaction Treatment satisfaction Satisfaction with cosmetic outcome Perception of health Quality of life Perception of cosmetic outcome Perception of general symptoms Perception of global disease Perception of global health Single symptoms Visibility appearance Quality of life, generic Quality of life, dermatologic Quality of life, oncologic Satisfaction with treatment effectiveness Appetite/thirst Sleeping problems Perception of eczema severity Disease related sleeping problems Burning Dandruff Dryness Dyspareunia Erythema / redness Flakiness Inflammation Itching Lichenification Pain Papules Pruritus Scaling Seborrhea Soreness Stinging Unspecified dermatological symptoms

17 Core Area (OMERACT 2.0) Domain Sub Domain Item Pathophysiological Manifestations Clinical assessment Combinations of two or more signs Erythema, oedema, excoriation, lichenification Erythema, oedema, excoriation, lichenification, crusting, dryness ( ) Global disease severity Appearance of nail Global severity of eczema Presence of wart Visibility of disease signs 18 Laboratory assessment Physiological skin assessment Single signs Biomarkers Colonisation Histo pathology Bleeding Crusting Dandruff Desquamation Dryness Erythema / redness Excoriation Exudation Fissures Greasiness Infiltration Inflammation Lichenification Maceration Oedema Oozing Papulation Peeling Roughness Scaling Sebum secretion Skin texture Surface damage Ulceration Vesiculation Weeping

18 Core Area (OMERACT 2.0) Domain Sub Domain Item Death Overall survival Overall survival (all cause death) Treatment related Mortality Core Area (OMERACT 2.0) Domain Sub Domain Item Resource Use/Economical Direct costs Financial and temporal burden Impact Healtcare Treatment visitis Treatment utilization Indication for concommitant treatment Utilization of specific treatment Sick leave This outcome classification system could be starting point for dermatology specific outcome taxonomy to standardize outcome definition and reporting Also possible resource for COS groups 19

19 From the domain to the outcome Outcome 12 Primary, secondary, and other outcomes, including the specific measurement variable (eg, systolic blood pressure), analysis metric (eg, change from baseline, final value, time to event), method of aggregation (eg, median, proportion), and time point for each outcome. Explanation of the clinical relevance of chosen efficacy and harm outcomes is strongly recommended 20 Chan AW, Tetzlaff JM, Gotzsche PC, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. Bmj 2013;346:e7586.

20 Classification of outcomes 21 Zarin DA, Tse T, Williams RJ, et al. The ClinicalTrials.gov Results Database Update and Key Issues. New England Journal of Medicine 2011;364(9):

21 Hierarchy of outcome specification in dermatology Core Area Domain Sub Domain Item Conceptionalization Time of measurement WHAT to measure Construct CONCEPT to measure, e.g. time to response Harmonized Definitions Measurement instrument Statistic (effect estimate, test procedure) HOW to measure and analyze 22

22 Limitations I Definitions / classification elaborated based on abstracted outcome information I No quality assurance across teams I Inconsistent definitions of outcome, outcome domain, measurement instrument I Data base: Inconsistent extraction extent, low study quality I Random sample of only 10 Cochrane reviews I Lag of time, no trials included that were published after

23 Conclusions I Outcome assessment and reporting is a very significant, general problem in many fields in dermatology I Reviewers and trialists have to focus more on this important topic to make trials and reviews meaningful I COS just for domains is probably insufficient to solve problems I COS for measurement instrument may also be not sufficient I Taxonomy for outcomes in dermatology is recommended 24

24 Thank you for your attention Address: University hospital Carl Gustav Carus an der TU Dresden Center for Evidence-based Healthcare Fetscherstraße 74, Dresden Contact: Prof. Dr. med. Jochen Schmitt, MPH

25 Background I High relevance of standardising outcome measurement in clinical trials I Systematic overview from Stefanie Deckert All 64 CSG reviews (comprising 1566 trials), published until 01/2015 Aim: to systematically compare predefined outcomes in CSG reviews and the reporting of these outcomes in underlying trials in CSG reviews I Results 26 Figure: Schmitt J, Deckert S, Alam M, et al. Report from the kick-off meeting of the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN). Brit J Dermatol 2016;174(2):287-95

26 Critical Reflection of the process 27 I No funding Few resources, Long project length, delayed dead lines Big group inconsistent definitions of outcome, outcome domain, measurement instrument I Data base Inconsistent extraction extent low study quality Allocation of domains by two reviewer (pro: consistency, contra: missing discussion with other coautors) I Conditions Technical problems/ high requirements, Communication problems Misjudgement of the workload