How can we systematically summarize results across many studies? The invention of Meta-Analysis. An exercise in mega-silliness?

Transcription

1 How can we systematically summarize results across many studies? The invention of Meta-Analysis. An exercise in mega-silliness? Peter Wilhelm 2. and PD Dr. Peter Wilhelm, Spring

2 Overview of Today s Lecture How can we systematically summarize results across many studies? The first Meta Analyis about the efficacy of Psychotherapy (Smith, Glass, & Miller, 1980) Critical Discussion A Contemporary Meta-analysis PD Dr. Peter Wilhelm, Spring

3 The Benefits of Psychotherapy The first Meta Analysis of Psychotherapy Smith, Glass & Miller (1980) Smith, M. L., & Glass, G. V.(1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore: Johns Hopkins University Press. Glass, G. V. & Kliegl., R. M. (1983). An apology for research integration in the study of psychotherapy. Journal of Consulting and Clinical Psychology, 51(1), PD Dr. Peter Wilhelm, Spring

4 Background: The Problem Since Eysenck s (1952) critical review about efficacy of PT many RCTs had been conducted: Many non-consistent results, difficult to summarize, integrate, and draw conclusions. Glass and Kliegl (1983, p. 28) describe the situation: The outcome literature had splintered and disintegrated; hundreds of unrelated efforts seemed to defy integration into anything that might address the question of the efficacy of competing therapies; indeed, it seemed to be believed that such questions could not or ought not be addressed and that incommensurability applied not only to theories of psychotherapy but also to its practice. PD Dr. Peter Wilhelm, Spring

5 Methods to summarize and integrate findings: Narrative Literature Review Qualitative No standardized criteria Report rather selective Prototypical examples are emphasized Subjective weighting and evaluation of findings After reviewing the PT outcome literature Rachman (1971, cited from Smith, Glass, & Miller, 1980, p. 1) concluded: We do not have satisfactory evidence to support the claim that psychotherapy is effective. It would seem therefore that those psychologists and psychiatrists who advocate and/or practice psychotherapy carry the claim that psychotherapy is effective Luborsky, Singer, & Luborsky (1975, cited from Smith, Glass, & Miller, 1980, p. 3) came to a positive conclusion : a high percentage of patients who go through any of these psychotherapies gain from them. -> No difference between efficacy of behavioral and non behavioral PTs PD Dr. Peter Wilhelm, Spring

6 Background: Methods to summarize and integrate findings Summary of findings of traditional reviews (Smith et al., 1980, p. 23) PD Dr. Peter Wilhelm, Spring

7 Methods to summarize and integrate findings: Box Score Method Box Score Method Quantitative Systematic and standardized documentation of findings Count of significant and non significant findings Example from Grawe et al. (1994) Comparision of psychodynamic short term therapy and: (C)BT (1. row) Medication (2. row) Client centered PT (3. row) Non specific PT (4. row) Grawe, K., Donati, R., & Bernauer F. (1994). Psychotherapie im Wandel: Von der Konfession zur Profession. Göttingen: Hogrefe. PD Dr. Peter Wilhelm, Spring

8 Methods to summarize and integrate findings: Box Score Method Box Score Method Quantitative Systematic and standardized documentation of findings Count of significant and non significant findings Example from Grawe et al. (1994) Comparision of psychodynamic short term therapy and: (C)BT (1. row) Medication (2. row) Client centered PT (3. row) Non specific PT (4. row) Grawe, K., Donati, R., & Bernauer F. (1994). Psychotherapie im Wandel: Von der Konfession zur Profession. Göttingen: Hogrefe. PD Dr. Peter Wilhelm, Spring

9 Methods to summarize and integrate findings: Meta Analysis Meta Analysis Systematic method to summarize and quantify results reported in primary studies Provides estimates of size and robustness of interventioneffects (or any other effect of interest) Allows identification of moderators Provides systematic information about the field of research PD Dr. Peter Wilhelm, Spring

10 Goals of Smith et al. s (1980) Meta Analysis Determine state of knowledge about the effects of PT (Smith et al., 1980, p. 55): comprehensive objective unbiased PD Dr. Peter Wilhelm, Spring

11 Definition of population of studies All controlled PT-outcome studies conducted between Any form of PT: Clients with emotional or behavioral problems Treatment should ameliorate problems of clients and was psychological or behavioral Treatment was performed by psychotherapists (profession or training) PD Dr. Peter Wilhelm, Spring

12 Definition of population of studies Excluded Treatments: Pharmacological Treatment Bibliotherapy Sensitivity Training Consciousness raising groups Encounter groups Education and training programs Peer counseling Electro Shock treatment Occupational or recreational therapy PD Dr. Peter Wilhelm, Spring

13 Literature research Psychological Abstracts Previous Reviews Journals with focus on PT studies until 1977 e.g. Journal of Consulting and Clinical Psychology Colleagues Conference Reports Dissertations-Abstracts Excluded: 50 Studies without sufficient statistical data Many studies on systematic desensitisation and school counseling -> random sample 30 studies that were not coded dead-line PD Dr. Peter Wilhelm, Spring

14 Information extracted from each study Publication date Publication Type Experimentator Blinding Diagnosis Hospitalization Intelligence Client-therapist similarity Social Status Assignment to groups Drops-outs Internal Validity Allegiance of Researcher(s) Treatment classification Therapy modalities Setting Duration Experience of therapist etc. PD Dr. Peter Wilhelm, Spring

15 Information extracted from each study Internal validity 1 = low, 2 = medium, 3 = high Outcome type 1 Fear/Anxiety, 2 Self esteem, 3 global adjustment, 13 other Reactivity of outcome: Rating 1 low to 5 high Effect size (ES) calculated for each available measure = M therapy M control SD control PD Dr. Peter Wilhelm, Spring

16 Coding of a Study Example : A study of the comparative effectiveness of systematic desensitization and implosive therapy. (Willis & Edwards, 1969) Publication Date 1969 Type of Publication Training of Experimenter Blinding Diagnosis (12 Categories) Hospitalization Intelligence of clients Client-Therapist Similarity Age of client Percent male clients Journal Psychologist Experimenter was Therapist Simple phobia (mouse phobia) none Above average (inferred from college student status) High (similar education) 20 (inferred from college student status) 0% (only female clients) Recriutement By Experimenter 16 PD Dr. Peter Wilhelm, Spring

17 Group assignment of clients Group assignment of therapists Mortality Internal Validity (Rating: low, middle, high) Treatments Simultaneous Comparisions Reliability Coding of a Study Example : A study of the comparative effectiveness of systematic desensitization and implosive therapy. (Willis & Edwards, 1969) Randomization (3 Groups) Non randomization 0% Post, 4% Follow-up High because of Randomization, and low Mortality (1) Systematic Desensisitivation (SD) (2) Implosion (1) & (2) Sub class: behavioral Yes; SD, Implosion, Control group High PD Dr. Peter Wilhelm, Spring

18 desensitization and implosive therapy. (Willis & Edwards, 1969) Experimenter Allegiance No prefered treatment Modality Location Duration Experience of Therapist Results Effect Size Coding of a Study Example : A study of the comparative effectiveness of systematic Individual treatment College 3 sessions within 3 weeks 5 years Outcome-Type Anxiety: Fear Survey Schedule (FSS) Behavior Avoidance Measure (BAM) -> High reactivity of outcome measure ES of Follow-up Data not calculated ES treatment vs control group SD FSS ES=1.71 BAM ES=.07 Impl. FSS ES=2.12 BAM ES=.48 PD Dr. Peter Wilhelm, Spring

19 Classification of treatments Behavioral Verbal Operant-classical behavioral Cognitve behavioral Cognitive Dynamic Humanistic Behavior modification Cognitive behavioral Rationalemotive Psychodynamic Client-centered Implosion Eclectic behavioral Other cognitive Eclectic dynamic Gestalt Systematic Desensitization Reality Therapy Hypnotherapy Transactional Analysis Adlerian 19 PD Dr. Peter Wilhelm, Spring

20 Reliability of coding 20 Studies were coded twice Authors and Raters 92% Agreement PD Dr. Peter Wilhelm, Spring

21 Analysis of data 1. Analyses of main effects 2. Analyses with control variables Cross-classification Multiple Regression Analyses 3. Same-experiment Comparisons 21 PD Dr. Peter Wilhelm, Spring

22 Descriptive Results 475 Studies 75% PTs, 25% School psychology treatments 1766 Effect Sizes 3.7 ES per Study Therapy Duration: sessions: M=16.18 (SD=26.56) 2/3 less than 13 sessions Assessment: (0-300 weeks after treatment) 2/3 Post measures Client s gender : 47.95% Male Client s Age: M=22.91 Jahre (SD=9.02 Jahre) 22 PD Dr. Peter Wilhelm, Spring

23 Descriptive Results ᴓ ES = 0.85 (SD = 0.03) PT vs. control group ᴓ ES = 0.93 when placebo or undifferentiated counseling were excluded 9% negative ES 23 PD Dr. Peter Wilhelm, Spring

24 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

25 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

26 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

27 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

28 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

29 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

30 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

31 Descriptive Results: ES of Treatments PD Dr. Peter Wilhelm, Spring

32 Results: ES of Treatments considering diagnostic type PD Dr. Peter Wilhelm, Spring

33 Descriptive results: ES and time of Follow up 2/3 of assessments at post measurement 1/3: 4 weeks to 10 years The later the assessment the lower ES (linear and squared effect of time R =.78) PD Dr. Peter Wilhelm, Spring

34 Descriptive results: ES and duration of therapies PD Dr. Peter Wilhelm, Spring

35 Descriptive results: ES and type of therapist s training PD Dr. Peter Wilhelm, Spring

36 Descriptive results: ES and reactivity of measurement 36 PD Dr. Peter Wilhelm, Spring

37 Descriptive results: ES and type of measurement PD Dr. Peter Wilhelm, Spring

38 Descreptive results: ES and internal validity PD Dr. Peter Wilhelm, Spring

39 Descriptive results: ES and Experimenter Allegiance 39 PD Dr. Peter Wilhelm, Spring

40 Results: Correlation between study characteristics and effect size Correlation of UVs with effect sizes 40 PD Dr. Peter Wilhelm, Spring

41 Results of same experiment analysis PD Dr. Peter Wilhelm, Spring

42 Results of same experiment analysis Diffences between groups only in psychological measures, but not for less tractable measures PD Dr. Peter Wilhelm, Spring

43 Summary Meta Analysis provides a comprehensive summary Psychotherapy is efficacious In simple comparisions: (C)BTs had largest ES In controlled analysis: No reliable differences between PTs -> Confirms Dodo Bird 43 PD Dr. Peter Wilhelm, Spring

44 Summary ES depended on: Outcome-Type Anxiety measures associated with big ES, personality and performance measures with small ES Reactivity of outcome ES higher the higher reactivity Time of measurement Smaller for Follow up Experimenter Alliegence ES did not depend on: Therapy duration Internal Vailidity Patient variables 44 PD Dr. Peter Wilhelm, Spring

45 Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness Garbage in garbage out A mass of reports good, bad, and indifferent are fed into the computer in the hope that people will cease caring about the quality of the material on which the conclusions are based. (p. 517) Advocacy of low standards of judgment -> Back to a dark age of scientific psychology. Not possible to distill scientific knowledge from a compilation of studies mostly of poor design, Relying on subjective, unvalidated, judgments dissimilar with respect to nearly all vital parameters PD Dr. Peter Wilhelm, Spring

46 Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness Takes subjective reports of therapists as source of information subjective, unvalidated, and certainly unreliable clinical judgments (= unscientific) -> Severely impair findings Initial differences between patients Patients for psychoanalysis more selective than patients for behavior therapy higher intelligence, emotional resources, ego strength, etc. -> Much more likely to improve spontaneously. PD Dr. Peter Wilhelm, Spring

47 Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness In sum Eysenck (1978) sees major problem in primary studies included into the meta-analysis Only better-designed experiments than those in the literature can bring us a better understanding on the points raised Necessary: RCTs with placebo groups to study therapy-specific effects several therapists for each method I would suggest that there is no single study in existence which does not show serious weaknesses, and until these are overcome I must regretfully restate my conclusion of 1952, namely that there still is no acceptable evidence for the efficacy of psychotherapy. (p. 517) PD Dr. Peter Wilhelm, Spring

48 Smith et al. s reply to Eysenck s garbage in - garbage out argument Meta-analysis treats methodological assumptions of studies as part of an object field in itself -> A posteriori judgment of weaknesses of method Empirical status of methodological principles can only be studied when there are different studies under various methodological circumstances -> estimation of relationship between the principles and study findings. If design "flaws" are crucial, they will show a correlation with study findings expressed as effect sizes. PD Dr. Peter Wilhelm, Spring

49 Incommensurability Problem: "Mixing apples and oranges" Studies are pooled that assess the effects of different treatments with different durations for different problems or diseases with different patients, using different instruments «myth of uniformity" No distinction between persons, therapists, therapies, and pathologies etc. Which studies should be combined depends on the level of generalisation Apples and oranges can be mixed if we want to tell something about fruits Studies over which results are aggregated should be carefully selected to avoid confounding effects Reducing conceptual heterogeneity by narrowing the focus: E.g. One diagnosis, only RCT studies with direct comparisons (e.g. Cuijper et al., 2008, Barth et al. 2013). When studies vary methodologically, differences can be taking into account by treating them as moderator variables PD Dr. Peter Wilhelm, Spring

50 Aggregation of Effect Sizes: Each possible effect Size was calculated Effect size is level of analysis (N = 1766 ES from 475 Studies; 3.7 ES per Study) Ignores dependence of ES from same study Studies with many measures contribute more Solution: Aggregating ES across studies Multilevel Analysis ES are less reliable when sample size is small Weighting ES by sample size PD Dr. Peter Wilhelm, Spring

51 Representativeness of Findings Relevant studies were overseen (Rachman & Wilson, 1980=) -> Disadvantage for studies of behavioral therapy Only studies published in English were included Important information in primary studies is frequently lacking ES can t be computed Description of therapist, treatment, and patients often insufficient «Publication bias» Studies with Null findings have lower chance to get published ES estimates are positively biased (probably larger than in reality) Estimating «Fail-Save-Number» PD Dr. Peter Wilhelm, Spring

52 PD Dr. Peter Wilhelm, Spring

53 Goal Re-examine the comparative efficacy of different psychotherapeutic interventions for adult depression Using network meta-analysis (a new technique) PD Dr. Peter Wilhelm, Spring

54 Inclusion Criteria RCTs that compare effects of PT to a control condition waitlist usual care placebo compare effects two PTs to another Adults with a depressive disorder, or with elevated levels of depressive symptoms Psychotherapeutic interventions were defined as: interventions with primary focus on language based communication between a patient and a therapist or bibliotherapy supported by a therapist PD Dr. Peter Wilhelm, Spring

55 Literature Research and studies included into MA PD Dr. Peter Wilhelm, Spring

56 Literature Research and studies included into MA PD Dr. Peter Wilhelm, Spring

57 Patient Population PD Dr. Peter Wilhelm, Spring

58 Interventions and Control Conditions PD Dr. Peter Wilhelm, Spring

59 Format and Setting of Intervention PD Dr. Peter Wilhelm, Spring

60 Network Analyis PD Dr. Peter Wilhelm, Spring

61 Network Analyis Effects for non existing comparisons are estimated from available comparisons Simplified example: CBT - Placebo ES =.4 IPT - Placebo: ES =.5 CBT - Psychodynamic ES.=.1 Estimated effect IPT Psychodynamic: ES =.2 PD Dr. Peter Wilhelm, Spring

62 Results: PT vs. wait-list control group (All Studies) PD Dr. Peter Wilhelm, Spring

63 Results: PT vs. wait-list control group (only studies with medium or large N) PD Dr. Peter Wilhelm, Spring

64 Limitations Only Post Assessment, No Follow up Number of studies for several therapies was small PD Dr. Peter Wilhelm, Spring

65 Conclusions Small study effects affect the results of RCTs and should receive more attention In larger trials robust effects for cognitive-behavioural therapy, interpersonal therapy, problem-solving therapy, while effects were less robust for psychodynamic therapy supportive counselling, behavioural activation. However, effect differences between these six psychotherapeutic interventions were rather small. Overall conclusion: Different psychotherapeutic interventions for depression have comparable, moderate-to-large effects. PD Dr. Peter Wilhelm, Spring

66 Literature Barth, J., Munder, T., Gerger, H., Nüesch, E., Trelle, H. Znoj, H.-J. Jüni, P., & Cuijpers, P. (2013). Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network meta-analysis. PLoS Medicine, 10 (5), e doi: /journal.pmed Glass, G. V. & Kliegl., R. M. (1983). An apology for research integration in the study of psychotherapy. Journal of Consulting and Clinical Psychology, 51(1), Eysenck, H. J. (1978).An exercise in mega-silliness. American Psychologist, 33, 517. Smith, M. L., & Glass, G. V.(1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore: Johns Hopkins University Press. PD Dr. Peter Wilhelm, Spring