The singer and not the song - evidencing differential clinical effectiveness and efficiency between PWPs. Stephen Kellett Helen Green Michael Barkham Nick Firth & Dave Saxon, Research Associate, ScHARR
What I will be taking about First study trying to define whether variability exists between PWPs in terms of their outcome and what might explain this Second study looking at differentials of PWP efficiency speed of change If we have time, some new PWP competency measures
How do we measure therapist effects? Use of Multilevel Modelling Hierarchical Data & Random Effects Explictly models nesting or clustering - variance at level 1 and level 2 From this can calculate the size of therapist effect Can also model other influences on outcome at either level
The size of the therapist effects? Therapist Effects General consensus? 8-9% or around 5-10% of outcome variance explained... BUT still a lot of variability! Ignoring therapist effects risks overestimating the effect of the therapy modality. Majority of therapist effect research in psychotherapy so far focused on high intensity interventions. One previous study by York group stated no therapist effect in LI interventions.
Therapist effects context is important RCT = therapist intervention tightly controlled to a specific model treating a specific client group Routine practice = therapists use wide variety of approaches to treating clients with a number of different presentations/diagnoses (practice based evidence) IAPT is like a huge practice based dataset!
Psychological Wellbeing Practitioners Low intensity interventions at step two of IAPT - low contact/high volume basis. Emphasis placed on self-management - PWP role can be likened to a CBT self-help coach (The IAPT Programme, 2011b). Use of treatment manuals homogeneity of delivery. Furthermore, some have suggested that manualisation of the PWP role could make the development of responsive therapeutic relationships potentially difficult. Could mean that individual differences between PWPs are minimised, resulting in a minimal or undetectable therapist effect.
Therapist effects background Literature search identified 17 studies of therapist effects in psychotherapy with adult clients RCT = 5 Non-randomised trial = 1 ROS = 11 Used multi-level (9) or single level (8) analysis
Therapist effects - background 15 / 17 (88%) studies found a therapist effect 4 / 5 (80%) RCTs found a therapist effect 10 / 11 (90%) routine practice found a therapist effect 7 / 9 (78%) using MLM found a therapist effect 4 studies looked at % of variance of outcomes attributable to therapists Ranged from 0% to 8%
Therapist effects the implications.. Evidence that therapists do vary in their outcomes, in some cases despite adherence to and competence in specific models Suggests that despite training and working to a specific model, individual therapists do have an effect on treatment outcomes its not just the model
What is it that makes some therapists more effective than others the search so far Not:- Age Gender Ethnicity Years of experience Training discipline
Defining effective practice Difficult construct to research, therefore limited knowledge of what makes an effective therapist or LI coach Some findings/suggestions: In therapy factors Alliance, warmth, relationship skills Out of therapy factors Good emotional adjustment, self-reflective abilities, highly self-critical of performance
So where are we at? But...as yet, little conclusive evidence as to what makes an effective therapist and particularly what makes an effective PWP
Hypothesised factors Intuition Do PWPs use their gut instinct to guide intervention and is this useful!? Definition rapid combination of cognitive and affective information leading to a decision Rational intuition: conscious, measured and analytic approach Experiential intuition: affect driven, pre-conscious approach
Hypothesised factors Resilience Do PWPs need to be resilient to be effective? Definition abilities and characteristics that provide individuals with the skills to cope with, and bounce back from, adverse situations
Hypothesised factors Ego strength Do PWPs need to have high ego strength in order to be effective? Definition an ability to maintain a sense of self in the face of challenges, and an ability to manage conflicts without becoming overwhelmed
Research Questions 1. Do therapist effects exist within a population of PWPs? 2. If so, what factors are associated with effectiveness?
Method Use IAPT minimum dataset relating to participating PWPs to establish whether a therapist effect exists PWPs completed measures hypothesised to be associated with effectiveness: Resilience Ego strength Intuition
Method Supervisors completed measures hypothesised to be associated with effectiveness Intuition (Other measures of resilience and ego strength not given to supervisors to minimise burden on their time) Interview PWPs and their supervisors to explore how they describe going about PWP work and examine accounts of most effective PWPs
Method PWP participants Approached Universities of Sheffield, Nottingham & York PWPs who completed training in 2010 9 / 15 (60%) possible IAPT services agreed to participate 47 PWPs across the 9 services 29 (62%) agreed to participate
Method PWP participants 3 services unable to provide outcome data due to storage and retrieval difficulties with data Final sample: 21 PWPs 6 IAPT services across Yorkshire, Derbyshire and Nottinghamshire Data taken from when PWP started in the service to end of Feb 2011 (one service provided until end of Sept 2010)
PWP sample 16 female, 5 male Mean age = 30 years (range 23-52) Mean no. clients = 54 (range 8 197) All individual sessions (no group contacts) Mean no. sessions = 5 (range 2 21)
Client sample Data included if clients had seen a PWP at least twice and had completed PHQ-9 and GAD-7 at both sessions 1,122 clients 65% female, 35% male Mean age 41 years (range 16 92)
Supervisor sample Supervisors of all 21 PWPs approached 17 (81%) agreed to participate
Procedure Blinding procedures used to minimise bias Interviews held before any quantitative analysis was done i.e. no knowledge of effectiveness ranks or questionnaire scores while interviews were ongoing 1 st stage of interview analysis done without knowing effectiveness ranks MLM analysis of therapist effects done without knowing identity of individual PWPs in dataset
Results therapist effect in PWPs Outcome data analysed using MLM Accounts for hierachical nature of data Level 1: Client Level 2: Therapist Controlled for initial intake score Found that overall, PWPs accounted for almost 9% of the variance in outcome
Results therapist effects in PWPs 9% - so what? Consistent with literature on other types of therapist (when an effect was found, ranged from 3-8% in high intensity therapies) Suggests that despite training in specific interventions, PWPs vary in their contribution to client outcomes
Results PWP effectiveness Across the whole sample: Overall recovery rate: 35.4% Outcome measure Pretreatment score mean Posttreatment score mean Change score mean PHQ-9 13.17 9.83 3.34 0.52 GAD-7 12.04 8.99 3.05 0.55 Uncontrolled effect size
Results MLM residual plots
Results MLM residual plots
Results PWP ranks Residual plots used to rank PWPs in terms of effectiveness Composite measure of PHQ-9 and GAD- 7 used Examined differences between PWPs ranked 1-5 (upper quartile) and 17-21 (lower quartile) Spread of services across the two quartiles
Results comparison of quartiles, client data Upper quartile of PWPs Lower PHQ-9 post-treatment scores Lower GAD-7 post-treatment scores Higher change scores
Results comparison of quartiles, PWP data Upper quartile Higher resilience No differences on other measures
Resilience
Results comparison of quartiles, supervisor data Upper quartile Lower experiential intuition No difference on rational intuition
Work with your buddy sat next to you Try to define what makes you resilient as a PWP What actions could you take that would increase your reliance?
Results qualitative data Template analysis High order themes utilised from interview schedule Lower order themes emerged from stage one analysis of interviews Compared lower order themes of upper quartile then lower quartile
Results qualitative: PWPs High order theme How previous experience helped Engagement with supervision to improve skills Awareness of gaps in skills or knowledge Hallmarks of clinical practice Effectiveness in working in the stepped care model Lower order theme: PWP upper quartile Knowledge of CBT principles Being prepared and organised Process supervision Medication Gaps beyond low intensity remit Communication skills Adapting interventions to the individual Understanding IAPT model and PWP role Lower order theme: PWP lower quartile Development of interpersonal skills Specific clinical questions Specific skills Knowledge of specific presentations Communication
Results qualitative : Supervisors High order theme Engagement with supervision to improve skills Methods of improving practice Hallmarks of clinical practice Effective components of therapeutic delivery Lower order themes: Supervisors upper quartile Openness to discussing difficulties Active supervision participant Proactive in improving practice Organisational skills Knowledge and understanding Lower order themes: Supervisors lower quartile Openness Interpersonal skills
Results - qualitative Upper quartile Confidence (lack of gaps in skills and knowledge, previous experience, ability to adapt interventions) Good understanding of IAPT model Openness to discussing difficulties Proactive, organised & thorough
Again, work with your buddy sat next to you How to increase your engagement with supervision in your role? What are the gaps in your knowledge and how you might plug these?
Summary & implications Research question 1: Do therapist effects exist within a population of PWPs? Yes 9% of variance in outcomes attributable to PWPs Suggests practitioners vary even after training in standardised interventions Not just what treatment is delivered, but who delivers the treatment also important
Implications of a PWP effect PWPs will vary in their outcomes, with some more effective than others Routinely monitor outcomes and use supervision to provide feedback and training on cases that are not improving
Summary and implications Research question 2: What factors are characteristic of effective PWPs? Resilience Supervisor s perception of experiential intuition Confidence Understanding of IAPT model Proactive, organised and thorough approach
Implications of factors associated with effective PWPs Resilience = ability to cope with adverse events PWP training/role stressful Identify those who are less resilient and provide support (environmental factors involved in resilience) Resilience building?!
Implications of factors associated with effective PWPs Take on board findings associated with effective PWPs to improve effectiveness overall: Some suggestions...
Implications: PWP Level PWPs approach their work in an organised and thorough manner Be open to using supervision to discuss difficulties Review outcome scores regularly to determine client progress
Implications: Supervisor Level Develop good relationships with supervisees to facilitate safety in bringing difficulties to supervision Offer support to those who have difficulties managing workload/cases Encourage use of outcome scores in supervision to identify clients who are not improving and discuss treatment strategies
Implication: Service Level Ensure PWP caseloads allow time for supervision preparation and monitoring of client outcomes Ensure supervisor caseloads allow time for PWP supervision and monitoring of client outcomes
Main criticism of the research Unmodeled service effects!
Now A Multilevel Modelling Analysis of PWP Efficiency; a moderator analysis
Aims Use a suitably powered large N single service sample to investigate the extent to which PWPs are differentially efficient (i.e. change in outcome score per session) the effect of demographic and process factors on PWP outcomes.
Design Quantitative cross-sectional design (3 years) Routine clinical outcome data from an IAPT service. 6,111 patients working individually with 56 PWPs. Dependent variables = PHQ-9, GAD-7, & WSAS scores Analyses predominantly = Multilevel Models and ANOVAs
Inclusion Criteria 2+ consecutive individual sessions with a PWP, with first and last outcome scores (data was required for all variables). PWPs were required to work with 30+ included patients. Some other criteria to improve data quality (e.g. only the first instance of PWP work per patient was used).
Predictive Variables Caseload (estimated in this study) Number of Sessions Treatment Ending Patient Initial Symptom Severity Patient Employment Status Patient Deprivation Patient Age and Gender
Table 1 Summary Sample Characteristics Final Sample Routine Practice Dataset Treatment instances 6111.0 7454.0 PWPs 56.0 85.0 Mean sessions with scores a (SD) 3.7 (1.9) 3.7 Mean patient age (SD) 41.6 (15.1) 41.6 Mean patient IMD (SD) 26.9 (18.4) 27.3 Percentage female 64 64 Percentage White British 88 (N = 5750) 88 Employment Status Percentage: Employed Full-time student Retired Unemployed Full-Time Homemaker or Carer 56 9 9 20 6 55 9 9 21 6 IMD = index of multiple deprivation, PWP = psychological wellbeing practitioner. a per treatment instance
Patients in treatment instances included and excluded from the sample were compared. Independent samples t-tests and chi-square tests found: Patients in treatment instances excluded from the sample had higher IMDs than those included. Excluded treatment instances had more sessions than included instances. significant differences were found in employment status between patients included and excluded from the sample No significant differences were found regarding patient age, gender, or proportion of White British patients. Patients excluded from the sample had higher final GAD-7 and WSAS scores than those included. No significant differences were found in initial PHQ-9, GAD-7, or WSAS scores, or in final PHQ-9 scores.
Results: PWP therapist effect size PWP therapist effects accounted for 6-7% of outcome variance, depending on the outcome measure. PHQ-9 = 6.4%, GAD-7 = 6.1%, and WSAS = 7.0% High commonality between models - 17 significant effects common to all three models, 7 effects common to two models, and 1 effect appearing in just one model. The depression model was the most representative.
Results: Factors influencing Outcome Poorer Outcomes More severe initial symptoms (but greater change) Unemployment: added approx. 1.5 points Non-completion: added approx. 4 to 5 points Better Outcomes Higher Caseload: approx. 1 point maximum difference U-shaped Curves Number of sessions: best outcomes at six to eight sessions - 2.1 points difference between two and six session interventions Age: best outcome around 45-50 years. Approx. 1 point maximum difference
Predicted Final GAD-7 Scores Results: Initial Severity and Therapist Effects Initial GAD-7 Scores
Predicted Final PHQ-9 Scores Results: Initial Severity and Therapist Effects Initial PHQ-9 Scores
Predicted Final PHQ-9 Scores Results: Initial Severity and Therapist Effects Number of Sessions
Results: Comparing PWPs by effectiveness PWPs were categorised into three groups by outcome (i.e. effectiveness); above average, average, and below average. PWPs with above average outcomes achieved almost double the change per session than PWPs with below average outcomes. Significant differences in the number of patients achieving RCSI No significant differences in deterioration rates or completion vs. drop-out rates
Multi level model Residual Results: Comparing PWPs by effectiveness Therapist Rank (depression outcomes)
So what does and doesn t this tell us? PWP therapist effects found again and not caused by a service effect PWPs differ in terms of their efficiency and effectiveness What we don t know is why!
Limitations/Cautions Therapist effect size depends to some extent on available explanatory variables, which were limited by the available data. The study's inclusion criteria mean that the sample may differ to some extent from 'routine practice' patient throughput in similar services. Outcome models were constructed to model holistic intervention goals (i.e., all patients modelled on all outcomes), rather than assigning patients to outcome models by condition or target of intervention.
Again, work with your buddy sat next to you Discuss what you think is going on in the sessions of more efficient PWPs? Anything from your practice that you have noticed makes a difference regarding your efficiency? What makes you drift into longer treatments?
~ Fin ~ Questions?
PWP Assessment Competency For a PWP to be graded as competent in an assessment session, the session has to score 18 (range 0-36) and the PWP must score 3 in each of the six sections (half-point scoring accepted). The competency rating tool is designed to be appropriate for assessments lasting between 30-45 minutes. Assessment OSCE range = 0-36 with a pass of 18 (and an automatic fail if the risk assessment is graded as incompetent or any of the six sections scores lower than 3).
PWP Treatment Competency For a PWP to be graded as competent in a treatment session, the session has to score 18 overall (range 0-36) and the PWP must score 3 in each of the six sections (half-point scoring accepted). The competency rating tool is designed to be appropriate for treatment sessions lasting for around 30-35 minutes. Treatment OSCE range = 0-36 with a pass of 18 (and an automatic fail if the risk assessment is not re-visited (graded as incompetent), the treatment has drift into a high intensity approach or any of the six sections scores lower than 3).