The singer and not the song - evidencing differential clinical effectiveness and efficiency between PWPs.

Size: px

Start display at page:

Download "The singer and not the song - evidencing differential clinical effectiveness and efficiency between PWPs."

Kristian Tate
5 years ago
Views:

1 The singer and not the song - evidencing differential clinical effectiveness and efficiency between PWPs. Stephen Kellett Helen Green Michael Barkham Nick Firth & Dave Saxon, Research Associate, ScHARR

2 What I will be taking about First study trying to define whether variability exists between PWPs in terms of their outcome and what might explain this Second study looking at differentials of PWP efficiency speed of change If we have time, some new PWP competency measures

3 How do we measure therapist effects? Use of Multilevel Modelling Hierarchical Data & Random Effects Explictly models nesting or clustering - variance at level 1 and level 2 From this can calculate the size of therapist effect Can also model other influences on outcome at either level

4 The size of the therapist effects? Therapist Effects General consensus? 8-9% or around 5-10% of outcome variance explained... BUT still a lot of variability! Ignoring therapist effects risks overestimating the effect of the therapy modality. Majority of therapist effect research in psychotherapy so far focused on high intensity interventions. One previous study by York group stated no therapist effect in LI interventions.

5 Therapist effects context is important RCT = therapist intervention tightly controlled to a specific model treating a specific client group Routine practice = therapists use wide variety of approaches to treating clients with a number of different presentations/diagnoses (practice based evidence) IAPT is like a huge practice based dataset!

6 Psychological Wellbeing Practitioners Low intensity interventions at step two of IAPT - low contact/high volume basis. Emphasis placed on self-management - PWP role can be likened to a CBT self-help coach (The IAPT Programme, 2011b). Use of treatment manuals homogeneity of delivery. Furthermore, some have suggested that manualisation of the PWP role could make the development of responsive therapeutic relationships potentially difficult. Could mean that individual differences between PWPs are minimised, resulting in a minimal or undetectable therapist effect.

7 Therapist effects background Literature search identified 17 studies of therapist effects in psychotherapy with adult clients RCT = 5 Non-randomised trial = 1 ROS = 11 Used multi-level (9) or single level (8) analysis

8 Therapist effects - background 15 / 17 (88%) studies found a therapist effect 4 / 5 (80%) RCTs found a therapist effect 10 / 11 (90%) routine practice found a therapist effect 7 / 9 (78%) using MLM found a therapist effect 4 studies looked at % of variance of outcomes attributable to therapists Ranged from 0% to 8%

9 Therapist effects the implications.. Evidence that therapists do vary in their outcomes, in some cases despite adherence to and competence in specific models Suggests that despite training and working to a specific model, individual therapists do have an effect on treatment outcomes its not just the model

10 What is it that makes some therapists more effective than others the search so far Not:- Age Gender Ethnicity Years of experience Training discipline

11 Defining effective practice Difficult construct to research, therefore limited knowledge of what makes an effective therapist or LI coach Some findings/suggestions: In therapy factors Alliance, warmth, relationship skills Out of therapy factors Good emotional adjustment, self-reflective abilities, highly self-critical of performance

12 So where are we at? But...as yet, little conclusive evidence as to what makes an effective therapist and particularly what makes an effective PWP

13 Hypothesised factors Intuition Do PWPs use their gut instinct to guide intervention and is this useful!? Definition rapid combination of cognitive and affective information leading to a decision Rational intuition: conscious, measured and analytic approach Experiential intuition: affect driven, pre-conscious approach

14 Hypothesised factors Resilience Do PWPs need to be resilient to be effective? Definition abilities and characteristics that provide individuals with the skills to cope with, and bounce back from, adverse situations

15 Hypothesised factors Ego strength Do PWPs need to have high ego strength in order to be effective? Definition an ability to maintain a sense of self in the face of challenges, and an ability to manage conflicts without becoming overwhelmed

16 Research Questions 1. Do therapist effects exist within a population of PWPs? 2. If so, what factors are associated with effectiveness?

17 Method Use IAPT minimum dataset relating to participating PWPs to establish whether a therapist effect exists PWPs completed measures hypothesised to be associated with effectiveness: Resilience Ego strength Intuition

18 Method Supervisors completed measures hypothesised to be associated with effectiveness Intuition (Other measures of resilience and ego strength not given to supervisors to minimise burden on their time) Interview PWPs and their supervisors to explore how they describe going about PWP work and examine accounts of most effective PWPs

19 Method PWP participants Approached Universities of Sheffield, Nottingham & York PWPs who completed training in / 15 (60%) possible IAPT services agreed to participate 47 PWPs across the 9 services 29 (62%) agreed to participate

20 Method PWP participants 3 services unable to provide outcome data due to storage and retrieval difficulties with data Final sample: 21 PWPs 6 IAPT services across Yorkshire, Derbyshire and Nottinghamshire Data taken from when PWP started in the service to end of Feb 2011 (one service provided until end of Sept 2010)

21 PWP sample 16 female, 5 male Mean age = 30 years (range 23-52) Mean no. clients = 54 (range 8 197) All individual sessions (no group contacts) Mean no. sessions = 5 (range 2 21)

22 Client sample Data included if clients had seen a PWP at least twice and had completed PHQ-9 and GAD-7 at both sessions 1,122 clients 65% female, 35% male Mean age 41 years (range 16 92)

23 Supervisor sample Supervisors of all 21 PWPs approached 17 (81%) agreed to participate

24 Procedure Blinding procedures used to minimise bias Interviews held before any quantitative analysis was done i.e. no knowledge of effectiveness ranks or questionnaire scores while interviews were ongoing 1 st stage of interview analysis done without knowing effectiveness ranks MLM analysis of therapist effects done without knowing identity of individual PWPs in dataset

25 Results therapist effect in PWPs Outcome data analysed using MLM Accounts for hierachical nature of data Level 1: Client Level 2: Therapist Controlled for initial intake score Found that overall, PWPs accounted for almost 9% of the variance in outcome

26 Results therapist effects in PWPs 9% - so what? Consistent with literature on other types of therapist (when an effect was found, ranged from 3-8% in high intensity therapies) Suggests that despite training in specific interventions, PWPs vary in their contribution to client outcomes

27 Results PWP effectiveness Across the whole sample: Overall recovery rate: 35.4% Outcome measure Pretreatment score mean Posttreatment score mean Change score mean PHQ GAD Uncontrolled effect size

28 Results MLM residual plots

29 Results MLM residual plots

30 Results PWP ranks Residual plots used to rank PWPs in terms of effectiveness Composite measure of PHQ-9 and GAD- 7 used Examined differences between PWPs ranked 1-5 (upper quartile) and (lower quartile) Spread of services across the two quartiles

31 Results comparison of quartiles, client data Upper quartile of PWPs Lower PHQ-9 post-treatment scores Lower GAD-7 post-treatment scores Higher change scores

32 Results comparison of quartiles, PWP data Upper quartile Higher resilience No differences on other measures

33 Resilience

34 Results comparison of quartiles, supervisor data Upper quartile Lower experiential intuition No difference on rational intuition

35 Work with your buddy sat next to you Try to define what makes you resilient as a PWP What actions could you take that would increase your reliance?

36 Results qualitative data Template analysis High order themes utilised from interview schedule Lower order themes emerged from stage one analysis of interviews Compared lower order themes of upper quartile then lower quartile

37 Results qualitative: PWPs High order theme How previous experience helped Engagement with supervision to improve skills Awareness of gaps in skills or knowledge Hallmarks of clinical practice Effectiveness in working in the stepped care model Lower order theme: PWP upper quartile Knowledge of CBT principles Being prepared and organised Process supervision Medication Gaps beyond low intensity remit Communication skills Adapting interventions to the individual Understanding IAPT model and PWP role Lower order theme: PWP lower quartile Development of interpersonal skills Specific clinical questions Specific skills Knowledge of specific presentations Communication

38 Results qualitative : Supervisors High order theme Engagement with supervision to improve skills Methods of improving practice Hallmarks of clinical practice Effective components of therapeutic delivery Lower order themes: Supervisors upper quartile Openness to discussing difficulties Active supervision participant Proactive in improving practice Organisational skills Knowledge and understanding Lower order themes: Supervisors lower quartile Openness Interpersonal skills

39 Results - qualitative Upper quartile Confidence (lack of gaps in skills and knowledge, previous experience, ability to adapt interventions) Good understanding of IAPT model Openness to discussing difficulties Proactive, organised & thorough

40 Again, work with your buddy sat next to you How to increase your engagement with supervision in your role? What are the gaps in your knowledge and how you might plug these?

41 Summary & implications Research question 1: Do therapist effects exist within a population of PWPs? Yes 9% of variance in outcomes attributable to PWPs Suggests practitioners vary even after training in standardised interventions Not just what treatment is delivered, but who delivers the treatment also important

42 Implications of a PWP effect PWPs will vary in their outcomes, with some more effective than others Routinely monitor outcomes and use supervision to provide feedback and training on cases that are not improving

43 Summary and implications Research question 2: What factors are characteristic of effective PWPs? Resilience Supervisor s perception of experiential intuition Confidence Understanding of IAPT model Proactive, organised and thorough approach

44 Implications of factors associated with effective PWPs Resilience = ability to cope with adverse events PWP training/role stressful Identify those who are less resilient and provide support (environmental factors involved in resilience) Resilience building?!

45 Implications of factors associated with effective PWPs Take on board findings associated with effective PWPs to improve effectiveness overall: Some suggestions...

46 Implications: PWP Level PWPs approach their work in an organised and thorough manner Be open to using supervision to discuss difficulties Review outcome scores regularly to determine client progress

47 Implications: Supervisor Level Develop good relationships with supervisees to facilitate safety in bringing difficulties to supervision Offer support to those who have difficulties managing workload/cases Encourage use of outcome scores in supervision to identify clients who are not improving and discuss treatment strategies

48 Implication: Service Level Ensure PWP caseloads allow time for supervision preparation and monitoring of client outcomes Ensure supervisor caseloads allow time for PWP supervision and monitoring of client outcomes

49 Main criticism of the research Unmodeled service effects!

51 Now A Multilevel Modelling Analysis of PWP Efficiency; a moderator analysis

52 Aims Use a suitably powered large N single service sample to investigate the extent to which PWPs are differentially efficient (i.e. change in outcome score per session) the effect of demographic and process factors on PWP outcomes.

53 Design Quantitative cross-sectional design (3 years) Routine clinical outcome data from an IAPT service. 6,111 patients working individually with 56 PWPs. Dependent variables = PHQ-9, GAD-7, & WSAS scores Analyses predominantly = Multilevel Models and ANOVAs

54 Inclusion Criteria 2+ consecutive individual sessions with a PWP, with first and last outcome scores (data was required for all variables). PWPs were required to work with 30+ included patients. Some other criteria to improve data quality (e.g. only the first instance of PWP work per patient was used).

55 Predictive Variables Caseload (estimated in this study) Number of Sessions Treatment Ending Patient Initial Symptom Severity Patient Employment Status Patient Deprivation Patient Age and Gender

56 Table 1 Summary Sample Characteristics Final Sample Routine Practice Dataset Treatment instances PWPs Mean sessions with scores a (SD) 3.7 (1.9) 3.7 Mean patient age (SD) 41.6 (15.1) 41.6 Mean patient IMD (SD) 26.9 (18.4) 27.3 Percentage female Percentage White British 88 (N = 5750) 88 Employment Status Percentage: Employed Full-time student Retired Unemployed Full-Time Homemaker or Carer IMD = index of multiple deprivation, PWP = psychological wellbeing practitioner. a per treatment instance

57 Patients in treatment instances included and excluded from the sample were compared. Independent samples t-tests and chi-square tests found: Patients in treatment instances excluded from the sample had higher IMDs than those included. Excluded treatment instances had more sessions than included instances. significant differences were found in employment status between patients included and excluded from the sample No significant differences were found regarding patient age, gender, or proportion of White British patients. Patients excluded from the sample had higher final GAD-7 and WSAS scores than those included. No significant differences were found in initial PHQ-9, GAD-7, or WSAS scores, or in final PHQ-9 scores.

60 Results: PWP therapist effect size PWP therapist effects accounted for 6-7% of outcome variance, depending on the outcome measure. PHQ-9 = 6.4%, GAD-7 = 6.1%, and WSAS = 7.0% High commonality between models - 17 significant effects common to all three models, 7 effects common to two models, and 1 effect appearing in just one model. The depression model was the most representative.

61 Results: Factors influencing Outcome Poorer Outcomes More severe initial symptoms (but greater change) Unemployment: added approx. 1.5 points Non-completion: added approx. 4 to 5 points Better Outcomes Higher Caseload: approx. 1 point maximum difference U-shaped Curves Number of sessions: best outcomes at six to eight sessions points difference between two and six session interventions Age: best outcome around years. Approx. 1 point maximum difference

62 Predicted Final GAD-7 Scores Results: Initial Severity and Therapist Effects Initial GAD-7 Scores

63 Predicted Final PHQ-9 Scores Results: Initial Severity and Therapist Effects Initial PHQ-9 Scores

64 Predicted Final PHQ-9 Scores Results: Initial Severity and Therapist Effects Number of Sessions

65 Results: Comparing PWPs by effectiveness PWPs were categorised into three groups by outcome (i.e. effectiveness); above average, average, and below average. PWPs with above average outcomes achieved almost double the change per session than PWPs with below average outcomes. Significant differences in the number of patients achieving RCSI No significant differences in deterioration rates or completion vs. drop-out rates

66 Multi level model Residual Results: Comparing PWPs by effectiveness Therapist Rank (depression outcomes)

67 So what does and doesn t this tell us? PWP therapist effects found again and not caused by a service effect PWPs differ in terms of their efficiency and effectiveness What we don t know is why!

68 Limitations/Cautions Therapist effect size depends to some extent on available explanatory variables, which were limited by the available data. The study's inclusion criteria mean that the sample may differ to some extent from 'routine practice' patient throughput in similar services. Outcome models were constructed to model holistic intervention goals (i.e., all patients modelled on all outcomes), rather than assigning patients to outcome models by condition or target of intervention.

69 Again, work with your buddy sat next to you Discuss what you think is going on in the sessions of more efficient PWPs? Anything from your practice that you have noticed makes a difference regarding your efficiency? What makes you drift into longer treatments?

70 ~ Fin ~ Questions?

71 PWP Assessment Competency For a PWP to be graded as competent in an assessment session, the session has to score 18 (range 0-36) and the PWP must score 3 in each of the six sections (half-point scoring accepted). The competency rating tool is designed to be appropriate for assessments lasting between minutes. Assessment OSCE range = 0-36 with a pass of 18 (and an automatic fail if the risk assessment is graded as incompetent or any of the six sections scores lower than 3).

79 PWP Treatment Competency For a PWP to be graded as competent in a treatment session, the session has to score 18 overall (range 0-36) and the PWP must score 3 in each of the six sections (half-point scoring accepted). The competency rating tool is designed to be appropriate for treatment sessions lasting for around minutes. Treatment OSCE range = 0-36 with a pass of 18 (and an automatic fail if the risk assessment is not re-visited (graded as incompetent), the treatment has drift into a high intensity approach or any of the six sections scores lower than 3).

White Rose Research Online URL for this paper: Version: Accepted Version

This is a repository copy of Therapist effects and IAPT Psychological Wellbeing Practitioners (PWPs): A multilevel modelling and mixed methods analysis. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/103217/