Measurement of Outcome in Specialist Healthcare Interventions for People with ID. Evidence Base, Considerations and a Method of Routine Measurement

Transcription

1 Measurement of Outcome in Specialist Healthcare Interventions for People with ID Evidence Base, Considerations and a Method of Routine Measurement

2 Outcome: a definition A characteristic of the service recipient s service experience that can reasonably be expected to change as a result of receiving the service Lyons et al., 2007

3 ID psychotherapy research Efficacy has support for CBT for anger (Hamelin et al., 2013, J Men H Res Int Dis, 6, 1.) Evidence is supportive of psychodynamic treatment (James & Stacey, 2014, Tiz L Dis Rev, 19, 1.) Other psychotherapies have small to non-existent evidence bases, but are often used But DoH now expects practitioners to produce an evidence base for what they do (DoH, 2010; Liberating the NHS: Transparency in Outcomes: a Framework for the NHS) In this context we can t base interventions on philosophy or orientation

4 Measurement of psychological change Sometimes something with inherent validity e.g. alive or dead. Often a construct & based in classical test theory, e.g. anxiety An aspect of psychological health is measured A psychological construct is measured with some degree of true measurement, some degree of error Validity refers to the extent to which a measure captures what it purports to measure A subsequent measurement tells us what change there has been (minus error)

5 Considerations Generic versus specific Constructs and error Efficacy versus effectiveness Validity, built on reliability Current recommended measures Importance of effect size A methodology for service evaluation

6 Generic Outcome measurement Health in terms of overall symptom presentation, e.g. HoNOS-LD Quality of Life, e.g. Euroquol Useful for overall evaluation of services Useless for identifying specific change factors

7 Community LD teams in Berkshire Skelly et al., 2006 Clin Psy & P w LD, 4, 3. Six community teams with 153 cases Social work, psychiatry, psychology, OT & SLT interventions. Not physiotherapy Significant effect found on the HoNOS-LD between assessment and closure No effect found on Quality of Life Effect size for HoNOS-LD total score 0.34

8 Generic Outcome in Community Teams in Berkshire 2006 HoNOS-LD Total Score (Skelly, Delicata & D Antonio, 2006) Wait 14 Intervene 13 Wait and Intervene Mean T1 Score Mean T2 Score

9 Specific outcome measurement Based in the experimental paradigm (dependent variable) Focus on a single construct, e.g. anxiety, delusions, avoidance, social engagement, etc. In efficacy studies, quasi-randomisation / waiting list control, and specification of inputs (e.g. a manualised therapy)

10 Example of specific measurement looking at efficacy: our current study The efficacy of psychodynamic psychotherapy and Cognitive Analytic Therapy in People with Intellectual Disabilities 20 cases receiving 16-session CAT 20 cases receiving psychodynamic psychotherapy of variable length Measure of therapeutic adherence (specific features of the therapy), therapy relationship, & extratherapeutic factors Waiting list control and follow up (1 year) Outcome measures include attachment security / insecurity on a reliable and valid measure

11 Researchers seldom ask Error How good is my measure? All construct-bases measurement in psychological tests contain vary amounts of error Should we state the expected error with our comparisons, as with IQ? (Morris & Fritz, 2013, The Psychol, 26, 8.)

12 Glasgow Depression Scale for People with a Learning Disability Cuthill, Gillespie & Cooper (2003: B J Psych 182) A score of 13 or above involves 96% sensitivity (Specificity 90%) with 0-13 being considered normal referenced to psychiatric interview based diagnosis. But the standard deviation is 2.94 for non-depressed people, 6.30 for depressed people. The sample is quite small (38 people with ID) percentages therefore speak to small nos. of actual people. Should effect sizes always be given with confidence intervals (Morris and Fritz, 2013)? GDS is a good measure!

13 Validity Issues Best measures have: Internal consistency Inter-rater reliability Test-retest reliability Face validity / acceptabilty Construct validity Discriminant validity Ease of use Interpretable results that translate into corporately useful information BUT The number of such measures in ID outcomes research can be counted on your fingers

14 What is the quality of available measures? McGurk and Skelly, 2013 Bull Fac P w ID, 12, 1. High quality measures are relatively few Often validated on small samples (but with slick marketing) Training sometimes glosses over reliability issues

15 Recommended: Anxiety and Depression Glasgow Anxiety Scale Glasgow Depression Scale Anxiety, Depression & Mood Scale (Hermans et al., 2012, Res Dev Dis 33, 2.) CORE-LD?

16 Recommended: Interpersonal Problems / Anger Inventory of Interpersonal Problems Novaco Anger Scales HoNOS-LD factor 2 score (Skelly & D Antonio, 2008, Clin Psy & P w LD, 6, 3.)

17 Psychosis Positive and Negative Syndrome Scale & Psychotic Symptom Rating Scales (PANSS & PSYRATS) See Hatton et al. (2005) J Int Dis Res, 49.

18 Challenging Behaviour HoNOS-LD (Total or Factor 2 Score) Behaviour Problems Inventory Challenging Behaviour Interview Maslow Ax of Needs Scale (LD) Life Experiences Checklist should not be used See Morris, Bush & Joyce (2012: DCP publications)

19 Epilepsy Glasgow Epilepsy Outcome Scale Espie et al. (2001) see Epilepsia, 42, 8.

20 Dysphagia, Profound & Multiple ID, personality disorder, Very little of any quality Barthel index for physical well being in profound & multiple ID? Abbey pain index for pain? DISDAT is widely used but we found limited validation unless audience knows differently?

21 Mean effect sizes across 18 Meta-Analyses Shedler, J. (2010) Am Psychol, March, General Psychotherapy 0.73 to 0.85 (530 studies) CBT & Beh Therapy 0.58 to 1.0 (105 studies) Antidepressant Medication 0.17 to 0.31 (74 studies) Psychodynamic Psychotherapy 0.69 to 1.80 (74 studies)

22 Methodology for Routine Service Evaluation Choose a generic measure (e.g. HoNOS-LD) Use the best available specific measure Measure at Referral, Assessment, post-intervention and at Follow-up Present results (1) for the whole service over the audit year using the generic measure and (2) for grouped complaint on the specific measure. Measure average change against effect size in the literature Alter practice accordingly

23 Clinical concern Selected recommended Measure n Effect size from example service [cohen s d] from assessment to case discharge significance Concern / comment Anxiety Glasgow Anxiety Scale <.01 Depression inc complex bereavement Glasgow Depression Scale <.01 Anger [nonphysical aggression] Aggression Self-harm Repetitive / compulsive Behaviour Traumatic reaction Vulnerability to exploitation Not categorised / measures not standardised Generic [HoNOS- LD] Novaco modified provocation scale HoNOS-LD BMR scale HoNOS-LD BMR scale ns Requires review of current practice, using practice guidance to improve outcomes < <.05 Low n HoNOS-LD ID scale 6.11 ns May not be sensitive to psychological intervention [8] but to pharmacological Life Events Scale- Revised Test of Interpersonal Competence and Personal Vulnerability < Risk ratings Methodology is not validated and so improvement only surmised. HoNOS-LD total score <.01 Suggests expected level of overall effectiveness based on national benchmarks