Discussion of meta-analysis on the benefits of psychotherapy

Discussion of meta-analysis on the benefits of psychotherapy Meta-analyses of psychodynamic long term psychotherapy. Science or lobbyism? Peter Wilhelm 27.4.2016

Overview of Today s Lecture Critical Discussion of the first Meta Analyis about the efficacy of Psychotherapy (Smith, Glass, & Miller, 1980) A Contemporary Meta-analysis Effectiveness of long-term psychodynamic psychotherapy

Summary of findings of Smith et al. (1980) Meta Analysis provides a comprehensive summary Psychotherapy is efficacious In simple comparisions: (C)BTs had largest ES In controlled analysis: No reliable differences between PTs -> Confirms Dodo Bird Verdict 3

Summary of findings of Smith et al. (1980) ES depended on: Outcome-Type Anxiety measures associated with big ES, Personality and performance measures with small ES Reactivity of outcome ES higher the higher reactivity Time of measurement Smaller for Follow up Experimenter Alliegence ES did not depend on: Therapy duration Internal vailidity Patient variables 4

Aggregation of Effect Sizes: Each possible effect size was calculated Effect size is level of analysis (N = 1766 ES from 475 Studies; 3.7 ES per Study) Ignores dependence of ES from same study Studies with many measures contribute more Solution: Aggregating ES across studies Multilevel Analysis ES are less reliable when sample size is small Weighting ES by sample size

Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness Selective references: Disregards Rachman (1971): Rachman reports large differences in recovery (spontaneous remission) between different types of patients Takes subjective reports of therapists as source of information subjective, unvalidated, and certainly unreliable clinical judgments (= unscientific) -> Severely Impair findings Initial differences between patients Patients for psychoanalysis more selective than patients for behavior therapy higher intelligence, emotional resources, ego strength, etc. -> Much more likely to improve spontaneously.

Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness Garbage in garbage out A mass of reports good, bad, and indifferent are fed into the computer in the hope that people will cease caring about the quality of the material on which the conclusions are based. (p. 517) Advocacy of low standards of judgment -> Back to a dark age of scientific psychology. Not possible to distill scientific knowledge from a compilation of studies mostly of poor design, Relying on subjective, unvalidated, judgments dissimilar with respect to nearly all vital parameters

Critical comments of Eysenck (1978) on first publication of findings by Smith & Glass (1977): Exercise in Mega Silliness In sum Eysenck (1978) sees major problem in primary studies included into the meta-analysis Only better-designed experiments than those in the literature can bring us a better understanding on the points raised Necessary: RCTs with placebo groups to study therapy-specific effects several therapists for each method I would suggest that there is no single study in existence which does not show serious weaknesses, and until these are overcome I must regretfully restate my conclusion of 1952, namely that there still is no acceptable evidence for the efficacy of psychotherapy. (p. 517)

Smith et al. s reply to Eysenck s garbage in - garbage out argument Meta-analysis treats methodological assumptions of studies as part of an object field in itself -> A posteriori judgment of weaknesses of method Empirical status of methodological principles can only be studied when there are different studies under various methodological circumstances -> estimation of relationship between the principles and study findings. If design "flaws" are crucial, they will show a correlation with study findings expressed as effect sizes.

Incommensurability Problem: "Mixing apples and oranges" Studies that might have differed in authors' intentions in respect to object, field, taxonomy, and methodology are compared to each other Problem: Which questions can be answered at which level of aggregation? Apples and oranges can be mixed if we want to tell something about fruits Avoiding confounding effects when results are aggregated

Uniformity Problem: «myth of uniformity" For basic analyses no distinction was made between persons, therapists, therapies, and pathologies Again a question of level of aggregation Reducing conceptual heterogeneity by narrowing the focus: E.g. One diagnosis, only RCT studies with direct comparisons (e.g. Cuijper et al., 2008, Barth et al. 2013).

Aggregation of Effect Sizes: Each possible effect size was calculated Effect size is level of analysis (N = 1766 ES from 475 Studies; 3.7 ES per Study) Ignores dependence of ES from same study Studies with many measures contribute more Solution: Aggregating ES across studies Multilevel Analysis ES are less reliable when sample size is small Weighting ES by sample size

Representativeness of Findings Relevant studies were overseen (Rachman & Wilson, 1980=) -> Disadvantage for studies of behavioral therapy Only studies published in English were included Important information in primary studies is frequently lacking ES can t be computed Description of therapist, treatment, and patients often insufficient -> Standards for reporting results in primary studies are necessary «Publication bias» Studies with findings have lower chance to get published ES estimates are positively biased (probably larger than in reality) Estimating «Fail-Save-Number»

Comparative Efficacy of Seven Psychotherapeutic Interventions for Patients with Depression: A Network Meta-Analysis Barth, J., Munder, T., Gerger, H., Nüesch, E., Trelle, H. Znoj, H.-J. Jüni, P., & Cuijpers, P. (2013). Comparative Efficacy of Seven Psychotherapeutic Interventions for Patients with Depression: A Network Meta-Analysis. Plos Medicine, 10 (5), 1-16, e1001454.

Goal Re-examine the comparative efficacy of different psychotherapeutic interventions for adult depression using network meta-analysis (a new technique)

Inclusion Criteria RCTs Compare effects of PT to a control condition Waitlist usual care Placebo or Compare effects two PTs to another Adults with a depressive disorder, or with elevated levels of depressive symptoms Psychotherapeutic interventions were defined as: interventions with primary focus on language based communication between a patient and a therapist or bibliotherapy supported by a therapist

Literature research and studies included into MA

Literature Research and studies included into MA

Patient Population

Interventions and Control Conditions

Format and Setting of Intervention

Network Analyis

Results: PT vs. wait-list control group (All Studies)

Results: PT vs. wait-list control group (only studies with medium or large N)

Limitations Only post Assessment, No Follow up Confounding effects could not be stastistically controlled simultaneously

Conclusions Small study effects affect the results of RCTs and should receive more attention In larger trials robust effects for cognitive-behavioural therapy, interpersonal therapy, problem-solving therapy, while effects were less robust for psychodynamic therapy, supportive counselling, behavioral activation. However, effect differences between these six psychotherapeutic interventions were rather small. Overall conclusion: Different psychotherapeutic interventions for depression have comparable, moderate-to-large effects.

Effectiveness of long-term psychodynamic psychotherapy Leichsenring & Rabung (2008). Leichsenring, F. & Rabung, S. (2008). Effectiveness of long-term psychodynamic psychotherapy. Journal of the American Medical Association, 300, 1551 1565. 27

Background: The Problem Uncertainty about status of psychoanalytic and psychodynamic treatments Some evidence for efficacy of short-term psychodynamic psychotherapy (STPP) for specific disorders Efficacy of long-term psychodynamic psychotherapy (LTPP) unclear For patients with complex disorders short-term PTs are insufficient multiple or chronic mental disorders patients with personality disorders -> greater deficits in social- and occupational functioning For such patients short term PT seems not be sufficient -> they probably need higher dosage -> Long term PT indicated Some studies suggest that LTPP may be helpful for these patients However, no strong evidence-based support yet. No meta analysis 28

Goals and Research Questions Is LTPP superior to other (shorter) PTs, particularly for patients with complex mental disorders? How effective is LTPP with regard to different outcome domains (e.g. target problems, personality- or social functioning)? What patient, treatment, or research factors contribute to LTPP outcome? 29

Definition of Long-term psychodynamic psychotherapy (LTPP) LTPP uses rather interpretive or supportive interventions depending on patient s needs. involves careful attention to therapist-patient interaction thoughtfully timed interpretation of transference and resistance Embedded in appreciation of therapist s contribution to the two person field Duration: at least 1 year or 50 sessions

Definition of population of studies Inclusion criteria: Studies of individual PPT meeting definition 1 year, or 50 sessions Prospective studies: before-and after or follow-up assessments Reliable outcome measures Clearly described sample of patients with mental disorders Adult patients ( 18 years); Sufficient data to allow determination of effect sizes Concomitant (eg, psychopharmacological) treatments were admissible RCTs and observational studies -> Author s claim that criteria were consistent with recent metaanalyses of PT

Literature research Studies published between 1960 and May 2008 MEDLINE, PsycINFO, Current Contents Manual searches of articles and textbooks Communication with authors and experts in the field To reduce file-drawer effect

Selection of studies

Effect size computation Between groups effect size (LTPP vs control or other PT) Point biserial correlation rp Within in group effect size M pre-treatment M post-treatment Hedges d = SD pre-treatment 34

Outcome domains effect sizes were computed separately for target problems general psychiatric symptoms broad measures of psychiatric symptoms (SCL-90) specific measures of of non target symptoms personality functioning, social functioning (Social Adjustment Scale) overall outcome (average effect across domains) assessments at termination and follow-up with longest follow-up period Intent to treat data preferred before completer data 35

Descriptive Results 23 studies 11 RCTs: LTPP vs. comparative treatments: CBT, cognitive-analytic therapy, dialectical behavioral therapy, family therapy, supportive therapy, short-term psychodynamic therapy, psychiatric treatment as usual 12 observational studies (no control group) 16 studies LTPP alone without any concomitant psychotropic medication 7 studies, patients received concomitant psychotropic medication 12 studies with treatment manuals or manual-like guidelines 1053 patients with LTPP Sessions M = 151 (SD= 155), Median = 74 Duration of therapy: M = 95 ( SD = 59) weeks, Median = 69 Follow-up period : M = 93 (SD = 65) weeks, Median = 65 36

Tests for Publication Bias Searching in the internet and contacting researchers 1 additional LTPP unpublished study was identified, but not included (group LTPP) Test for publication bias: Spearman rank correlation between effect size and sample size across studies not significant (p >.30) Fail-safenumber (Rosenthal) number of nonsignificant, unpublished, or missing studies that would need to be added to a meta-analysis in order to change results from significance to non-significance. In the total sample of studies examining LTPP alone, FSN was: 921 for overall outcome, 535 for target problems, 623 for general symptoms, and 358 for social functioning 40 for personality functioning -> No publication bias

Correlation of study quality with outcome Within-group effect sizes were correlated with Jadad scale (judging study quality with 3 items). No significant correlation (p >.28).

Within subject ES for LTPP in RCTs 39

Within subject ES for LTPP in observational studies 40

Were there differences between RCTs and observational studies? No sig. correlations between within-group ES (post) of LTPP and type of study (RCTs vs observational studies) (p >.36). No sig. differences between within-group ES of 16 controlled (including 11 RCTs and 5 observational studies) and 7 uncontrolled studies -> RCTs and observational studies were combined 41

Were there differences between LTPP and other treatments? 8 studies had sufficient data for comparison of LTPP with other treatments No sig. correlation between psychotropic medication and outcome in sample of 8 comparative studies (p >.13). -> Studies with and without and psychotropic medication were combined Number of sessions: LTPP: M = 103 (SD = 136), median = 49 Other treatments: M = 33 (SD = 28), median = 22 Sig. correlations between within-group ES (post) and treatment condition in favor of LTPP: overall outcome r=0.60 ** target outcome r=0.48 * personality functioning r=0.76 ** LTPP treatment effects for complex mental disorders (1 study excluded) were even larger 42

Were there differences between LTPP with and without psychotropic medication? LTPP alone (16 studies) produced sig. larger within group ES (post) than LTPP combined with psychotropic medication (7 studies): for target problems r = -.45* Only LTPP alone was considered for subsequent analyses of different patient groups 43

Within group ES (post) of LTPP alone for treatment of personality disorders 44

Within group ES (post) of LTPP alone for treatment of chronic mental disorders ( 1 year) 45

Within group ES (post) of LTPP alone for treatment of patients with multiple disorders 46

Exploring moderater effects Impact of 10 variables on 10 outcome variables (5 beforeafter and 5 before follow-up variables) was explored: age, sex, diagnostic group (personality disorders, chronic or multiple mental disorders, and depressive and anxiety disorders) experience of therapists (years) specific training in the applied treatment use of treatment manuals (0/1) Bonferoni adjustment (0.05/100; Alpha =.0005) No sig. correlations with outcome (p >.04) 47

Summary and Discussion by Leichsenring & Rabung LTPP was sig. superior to shorter-term methods of PT for overall outcome target problems personality functioning LTPP yielded large and stable effect sizes for: patients with personality disorders patients with multiple mental disorders patients with chronic mental disorders ES for overall outcome increased sig. between end of therapy and follow-up 48

Summary and Discussion by Leichsenring & Rabung Limited number of studies However, results were robust: Independent of age, sex, patient subgroups, experience of therapists or use of treatment manuals No publication bias Except for personality functioning more than 300 non. sig. studies needed to obtain non. sig. finding in meta analysis Some methodological shortcomings in older studies (randomization, allocation concealment, or observer bias) However, no sig. correlation between quality of study and ESs of LTPP No sig. difference between RCTs vs observational studies -> outcome data of the RCTs are representative for clinical practice -> observational studies did not over- or underestimate effects of LTPP 49

Critics of Rief & Hoffmann, 2009 Observational studies do not provide causal evidence for efficacy of LTPP In 5 of 11 RCTs number of sessions was below 50 -> Impairs generalisation to LTPP (usually > 120 sessions) Questionable treatment validity In some studies LTPP comprised improvement of affective expression or dysfunctional cognitions Additional treatment with psychotropic medication in 7 studies Validity of PT comparison condition (e.g. CBT in 2 studies) questionable Diagnosis of patients questionable no standardized diagnosis and heterogeneous patient samples in many studies Broad range of different diseases No intent to treat analyses but only completer ES were included into Meta-analysis Fail save number was calculated only for studies examining LTPP alone In these studies effects were larger -> exegerated fail save numbers Most results rely only on post but not on follow up data 50

Critics: Conclusion of Rief & Hoffmann, 2009 LTPP were considered to be to expensive and not efficicacous Leichsenring & Rabung s (2008) interpretation of findings suggest the contrary Rief & Hoffmann (2009) reassessed meta-analysis and concluded that its conclusions are not warranted. We conclude that a few of the included randomised clinical trials using psychodynamic short-term interventions truly are promising and of high quality; however these do not justify generalizations on long-term psychoanalysis, and the presented meta-analysis is clearly biased. (p. 594) Meta analysis serves to save LTPP in the health care system 51