Development of a new patient reported outcome measure for mental health services

Transcription

1 Development of a new patient reported outcome measure for mental health services Anju Keetharuth, John Brazier, Janice Connell, Lizzie Taylor Buck, Tom Ricketts, Jill Carlton, Michael Barkham School of Health and Related Research Contact: reqol@sheffield.ac.uk Website:

2 Policy Closing the Gap, No health without mental health, NHS outcome framework, NHS payment system in MH, NICE economic evaluation Traditionally measures focused on symptoms (e.g. PHQ 9) and/or involved clinical assessment (e.g. HONoS), or too generic (EQ 5D and swemwbs) Existing measures not able to capture wide spectrum of mental health conditions and severity (Boardman et al, 2013) Measure required that captures the concerns of service users, such as those identified by Leamy et al (2011): Connectedness, Hope, Identity, Meaning and Purpose, and Empowerment (CHIME)

3 Aim: Develop a brief measure of recovery and quality of life for users of mental health services For whom? For mental health populations aged 16 and over Generic suitable across all mental health populations including common mental health problems, severe and complex and psychotic disorders (clusters 1 17) (not dementia orlearning disabilities) For primary, secondary and tertiary mental health services For self completion though will have interview and proxy versions Long version (20 items) and brief version (10 items)

4 5 stages mixed methods approach Qualitative and quantitative inputs from services users Qualitative ti inputs from clinicians i i 4 main governance groups: Stakeholder group, Advisory group, Scientific Group and Expert User Group Expert Users involved in all stages and in decision making 10/08/2016 The University of Sheffield

5 Methods used Systematic review of qualitative literature Qualitative interviews with 19 service users Framework analysis to identify themes ReQoL (QoL literature) Recovery literature Connell et al. Leamy et al. 1.Belonging and Connectedness Relationships 2. Hope Hope 3. Selfperception Identity 4. Activity Meaning 5. Autonomy, Choice and Empowerment Control 6. Well Being Well Being/Symptoms 7. Physical health

6 Methods used 1. Identified items under each theme from: Manuscripts from qualitative interviews (mainly) Existing QoL measures Existing Recovery measures 2. Applied Norman and Streiner criteria

7 Methods used Participantswere i t presented with a subset of 87 questions Commented on the questions Choose their favourite ones and rephrased items. 59 service users interviewed across 4 NHS trusts Wide range of self reported diagnoses: Schizophrenia, Depression, Bipolar, Psychosis, OCD, Anxiety, Asperger's Syndrome, PTSD, Personality Disorder, Eating disorder, Alcohol problems, Cognitive impairment, Agoraphobia, Tourette s Syn. Items falling in the following categories were eliminated: Potentially distressing Judgemental Difficult to respond to Not too relevant to everyone Difficulties with interpretation Members of Expert user Group: post it notes exercise to select items to vote for favourite items At the Scientific Group meeting, 61 items were taken forward to the Stage 4 field testing

8 Study 1 Study 2 Extending ReQoL to those aged Cross cultural study Stage 3 repeated with year olds Stage 3 repeated with Urdu and Polish speakers Face to face interviews and focus groups 17 participants Participants were asked to reflect on the relevance and wording of 61 items 2 focus groups 2 individual interviews Participants were asked to reflect on the relevance and wording of the items Findings from both studies have been considered alongside psychometric evidence

9 Methods used Recruitment took place in secondary care, primary care and voluntary sector 61 items completed by service users Mode of administration Face to face in clinics i Post Online 2261 service users recruited 12 Trusts 59% 3 GP practices 6% Casper Trial cohort 8% Online panels 23% Voluntary sector 3% CAMHS 1% Self reported main diagnosis s Depression 44% Anxiety 28% Psychotic Disorders 17% Bipolar 11% Personality disorder 6% Eating disorders 6%

10 Aim: to reduce number of items using psychometric analyses to finalise short and long versions of ReQoL Methods used Recruitment took Service users recruited Baseline: 4252 Follow up : 1237 place in secondary 20 Trusts (18 MH) 56% care, primary care and voluntary ou ysector 3 GP practices 29% 40 items Community 8% completed IAPT 6% by service users CAMHS <1% Clusters n % Non psychotic py (1 4) Mode of Non psychotic (5 8) administration Psychotic Face to face in clinics Post Self reported diagnoses Similar to Stage 4

11 Factor analysis (EFA) aims to identify underlying unobservable (latent) domains The structure of the 6 themes is not supported from the data but that does not mean they are not important The best models (CFA) are: Two factor model: positively worded items and negatively worded items Bifactor model: Global, positively worded items and negatively worded items Implications 1 primary factor with methods factors for positive and negative items (common result)

12 1. All six mental health domains identified in Stage 1 of the project are represented tdin the final two versions of ReQoL 2. Service user acceptability (based on data collected in stage 2, 3, 4 and 5) and clinician i i acceptability and usefulness (based on data collected on the 40 item pool through focus groups with clinicians) 3. Acceptable psychometric properties following the application of multiple analyses (we ensured that items areresponsive responsive and can measure improvement or deterioration in QoL)

13 Analyses Guidance/Judgement/rule Determines Notes exclusion 1.Missing data at If any item has 5% missing data, this item Yes None identified item level should be dropped 2. Factor Analyses Identify items with high residual correlations (>0.1) Select one 15 pairs identified (19 items) 3. IRT Misfitting Identify itemswith pvalues < 0.05 in at least 3 No 5 items identified d items datasets. However, do not use the item level fit to exclude items. Acknowledge the misfit but retain item in the item pool. 4. IRT Information functions Ensure that items cover the whole measurement range (i.e., intensity) by choosing items to balance maximization of information over the total range and content validity (including items from all themes) No 5.Differential Item Exclude items with DIF No None identified Functioning (age, ethnicity, gender, condition) 6. Sensitivity to Ensure selected items show change in Yes Limited change: 2 change response over time items

14

15 1. I found it difficult to get started 11. I did things I found rewarding with everyday tasks 12. I avoided things I needed to do 2. I felt able to trust others 13. I felt irritated 3. I felt unable to cope 4. I could do the things I wanted to do 5. I felt happy 14. I felt like a failure 15. I felt in control of my life 6. I thought my life was not worth 16. I felt terrified living 17. I felt anxious 7. I enjoyed what I did 18. I had problems with my sleep 8. I felt hopeful about my future 9. I felt lonely 19. I felt calm 10. I felt confident in myself 20. I found it hard to concentrate Pl d ib h i lh lth( bl ith i bilit diffi lti i f Please describe your physical health (problems with pain, mobility, difficulties caring for yourself or feeling physically unwell) over the last week

16 ReQoL will be available to the NHS at no costs Paper and online versions The ReQoL measures will be distributed by a company called Oxford University Innovation Details will appear on ReQoL website

17 1. Consistent with the themes of recovery 2. Constructed using inputs from service users and clinicians at all stages of the project including decision making to ensure relevance of items for all parties 3. Suitable for a range of mental health conditions from mild, moderate, severe to very severe 4. ReQoL is psychometrically sound robust methodology 5. Acceptable to service users and clinicians i i as it is short and simple 6. ReQoL can measure changes in QoL responsiveness has been factored in item selection 7. Can be used to generate QALYs

18 Where does ReQoL fit in the current policy context? Payment system national tariff NHS Outcomes Framework Mental health policy team (adult and young people) Five year forward view mental health Use for economic evaluation

19 Work with clinicians and service users to co design a strategy and practical tools to facilitate and enhance implementation Use of ReQoL in clinical decision making Evaluate the impact of the implementation of Evaluate the impact of the implementation of ReQoL in clinical settings

20 Work with clinicians and service users to co design a strategy and practical tools to facilitate and enhance implementation Early ReQoL implementation work with MH Trusts As part of the care planning process To contribute to recovery plans To provide information across transitions To develop recovery oriented services To produce simple graphs and other ways of providing feedback about progress

21 Thursday 13 th Westminster Workshop focus Start of an iterative co design process to ensure themeasure meets theneeds of the services using it.

22 All participants in the study, NIHR CRN Mental Health, Staff at all participating trusts, charities and GP surgeries

23 Analyses Guidance/Judgement/rule Determines Notes exclusion 1.Missing data at If any item has 5% missing data, this item Yes None identified item level should be dropped 2. Factor Analyses Identify items with high residual correlations (>0.1) Select one 15 pairs identified (19 items) 3. IRT Misfitting Identify itemswith pvalues < 0.05 in at least 3 No 5 items identified d items datasets. However, do not use the item level fit to exclude items. Acknowledge the misfit but retain item in the item pool. 4. IRT Information functions Ensure that items cover the whole measurement range (i.e., intensity) by choosing items to balance maximization of information over the total range and content validity (including items from all themes) No 5.Differential Item Exclude items with DIF No None identified Functioning (age, ethnicity, gender, condition) 6. Sensitivity to Ensure selected items show change in Yes Limited change: 2 change response over time items

24 Start with 1 item from each theme Consider whether second item per theme is needed dd and, if so, which one Post scientific group meeting: consultation process Post consultation: finalise the long and short versions Guiding principle A good measure should have the following properties: face validity content validity responsiveness (i.e., sensitivity to change) acceptable psychometric properties Basis for choice: whether h another sub theme within ihi the theme needs to be represented consideration of severity balance between positively and negatively worded items Advisory and Stakeholder group to comment on work to date Optimise the information function of the long and short versions in light of the recommendations of the Scientific Group and other groups Implementation Start with choosing 1 central item per domain Once an item is chosen, then any of the items that it is very highly correlated (residual correlation 0.1) can be considered for exclusion from the item pool Research team to propose initial i i selection and then seek recommendations from the scientific group Screencast + online consultation At the team meeting final versions were agreed

25 BELONGING & RELATIONSHIPS Top item:bel2 I felt lonely Relationships requiring a BEL3P judgement on BEL4 BEL5P other people BEL1 WB5 WB6 Relationship self BEL2 I felt lonely Do we need another item in this theme? If so, which one of: BEL4 BEL3P BEL1 BEL5P WB5 and WB6

26 Relationshipswith others

27 Recruiting organisations Freq. Percent South Staffordshire and Shropshire Healthcare Stage 5 GP Nethergreen recruitment Southern Health Birmingham Solihull MHFT GP Crookes GP Richmond CASPER Trial cohort Leeds and York FPT Solent Northumberland Tyne and Wear NHS FT Cumbria Bradford and District Care FT Dorset South thwest tlondon St George s Trust Tees, Esk and Wear Valleys 84 2 Black Country 80 2 Humber FT 79 2 Dudley and Walsall MHT 62 1 Central and North West London 55 1 Rotherham Doncaster and South Humber 55 1 Sheffield Health and Social Care Trust 44 1 Central and North West London Recovery College 42 1 South West Yorkshire FT 33 1 McPIN 32 1 My Space charity 13 0 Sheffield Children Hospital 6 0 Chesterfield Royal Hospital (Adolescents)