Can PROMs be used to make decisions about individual patients?

Transcription

1 Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Science, Institute of Musculoskeletal Science, Botnar Research Centre Can PROMs be used to make decisions about individual patients? David J Beard, Kristina K Harris, Andrew J Price

2 WHO Development ICF Domains of pain & mobility/dexterity Participation Capacity Performance Patient satisfaction ICF Impairment Kinematics Clinical (AKS) Activities Self reported (with or without emphasis) Distinguishing functional tests Pain

3 Use of PROMs Research Clinical practice Audit Longitudinal change in status effect of treatment monitoring communication Snapshot effect of treatment diagnostic/predictive shared decision making communication

4 Pathway

5 PROMS: Contemporary usage? Haves and Have Nots Individual decision making!

6 Extended Remit Not Validated Healthcare policy (threshold for intervention) Not Validated Decision making for individuals Not Validated Measurement of individual Validated Use Listed TKR population

7 Methodology Tailored to the intended use of PROM Patient engagement/ self management Communication tool Diagnostic tool Predictive tool Monitoring Effect of treatment

8 What aspects of validity for individual patient decision making 1? Random error Systematic error Reliability: stability over time when no change in true status PROM TIME

9 What aspects of validity for individual patient decision making? Clinically meaningful change What is clinically meaningful? PROM TIME

10 What aspects of validity for individual patient decision making 3? Choice of threshold/decision criteria True change in status over time when to measure how to interpret PROM TIME

11 Heterogeneity across instruments Same purpose but No common currency V V

12 PROBLEMS AHEAD! PROMS policy

13 HTA Call 2012 Introducing Standardised and Evidence based Thresholds for Hip and Knee Replacement Surgery The Arthroplasty Candidacy Help Engine (The ACHE tool) General Medicine and Sub-specialty

14 Research questions: Can clinical tools for assessment of a patient s suitability for knee or hip replacement be used to set thresholds for operation? How does the choice of threshold affect the cost effectiveness of the procedure and subsequent improvements in patient quality of life?

15 Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Science, Institute of Musculoskeletal Science, Botnar Research Centre

16 Principal Investigator: Professor Andrew Price Collaborators: University of Oxford: Professor David Beard, Professor Ray Fitzpatrick, Dr Jill Dawson, Dr Karen Barker, Professor Andrew Carr, Professor David Murray, Dr Andy Judge, Professor Nigel Arden, Professor Alistair Gray, Helen Dakin University of Bristol: Professor Ashley Blom, Dr Rachel Gooberman-Hill Peninsular Medical School: Professor Paul Dieppe Lead researcher/coordinator: Dr Kristina Harris; Health Economist: Adrian Sayers; Statistician: TBA

17 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

18

19 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

20 Work Package 1 CREATE A SHORTLIST OF SCORING SYSTEMS POTENTIALLY USEFUL FOR SELECTING CANDIDATES FOR ARTHROPLASTY SURGERY 1.1 Use published literature to identify existing scoring systems and assess evidence concerning their suitability, measurement properties and feasibility. 1.2 Use existing datasets to calculate the measurement properties of those potential candidate tools not previously evaluated.

21

22 Measurement properties The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC - Qual Life Res (2010)

23

24 Validity Reliability The degree to which a PROM measures the construct it purports to measure The degree to which the measurement is free from measurement error Responsiveness The ability of a PROM to detect change over time in the construct to be measured

25 Easy to use Valid, reliable and responsive Interpretation Is a drop of 3 points clinically important? Is an improvement of 5 points worthwhile?

26 Classical test theory (CTT)

27

28 What about clinical relevance? Great news! The change in your knee score is beyond MDC 90 I don t want to live on this planet anymore

29 Method Anchor Based Overall, how are your <hip/knee> problems now, compared to before your operation? Much better A little better About the same A little worse Much worse MIC Minimal Important Change Single group, mean change score from baseline = little better Meaningful changes for the Oxford hip and knee scores after joint replacement surgery. Beard DJ, Harris, KK et al. Journal of Clinical Epidemiology. Available online 31 October 2014, ISSN ,

30 Use MDC (as a lower bound of meaningful change) and MIC as a reference points to help interpret score changes Change NOT above measurement error and NOT important Change above measurement error, but NOT important Change above measurement error and important no change MDC MIC maximum change

31 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

32 Included Eligibility Screening Identification PRISMA 2009 Flow Diagram Records identified through OVID database searching (AMED, Embase, MEDLINE, PsycINFO n = 3774 ) Additional records identified through other sources (EconLit n = 162; PROMs Database n= 454; Dare = 0) Records after duplicates removed (n = 2887, in EndNote n=2832) Recods included following handsearch (n= 5 ) Electronic records screened (n = 3448 ) Records excluded (n = 3267 ) Articles selected on the bases of title and abstract (n = 167 ) Studies included in qualitative synthesis (n =146 ) Full-text articles excluded (Conference abstract n = 10; Abstract could not be found within OU libraries n= 10; Book n=1)

33 Included Eligibility Screening Identification PRISMA 2009 Flow Diagram Initial number of instruments mentioned (Generic n = 34) (Hip Specific n = 12), (Knee Specific n = 19), (Lower limb and other n = 70) (Total n = 135) After screening (Generic n = 26) (Hip Specific n = 7), (Knee Specific n = 8), (Lower limb and other n = 23) (Total n = 64) Instruments excluded, with main reasons, see Table 1. Final number of instruments (Generic n = 18) (Hip Specific n = 3), (Knee Specific n = 9), (Lower limb and other n = 10) (Total n = 40) Instruments excluded, with reasons, see Table 2

34 Instrument ICOAP (h/k) P4 (h/k) K10 (h/k) McGill Pain- SF (h/k) SIP (h) NEADL (h) HUI2 and HUI3 (h) EQ5D (h/k h, k) AQOL (h/k, k) SF12 (h/k) SF6D (h) SF-36 (h/k, h, k) PSI (h) WHO-QOL Bref (h/k) Reproducibility Internal consistency Validity: Content Construct Responsiveness Interpretability Floor/ceiling/precision Acceptability Feasibility not reported - no evidence in favour + some limited evidence in favour ++ some good evidence in favour +++ good evidence in favour Additionally, we will try to obtain additional information from the authors of all scores

35 WP1 Progress Literature review Calculation of measurement properties USER vote

36 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

37 Work Package Determine pre-operative threshold scores for surgery in each of the 3 shortlisted scoring systems. Calculation of general and individual pre-operative thresholds for surgery and individual capacity to benefit, after accounting for the effect of pre operative co-variables (age, gender, co-morbidities)

38 2.2 Determine the relationship between threshold levels and cost effectiveness of hip and knee arthroplasty surgery for each scoring system Economic threshold for each clinical tool Relation of cost effectiveness to the thresholds determined in 2.1

39 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

40 2.3 Establish a scoring system with thresholds for the ACHE tool. Two stage voting process: 1) Selection of the scoring system, 2) Selection of the threshold value for the selected scoring system

41 WP1 WP2: Selection of Tool WP3 Lit review & Measurement Properties A. Threshold Determination B. Healthcare Economics C. System & Threshold Selection Acceptance, Evaluation & Uptake USER Group Meeting 1 Identify potential scoring systems In candidate scores only (using WP1 cohort data) In candidate scores only (using WP1 cohort data) USER group meeting 3 Evaluation of ACHE tool impact (local dataset) 3 years Existing cohorts N=7452 knee N=4815 hip NJR/HES N = 85,215 knee N= 95,481 hip Assess: validity, reliability, responsiveness A: Threshold Score Ability to Benefit Threshold B: Threshold Score Adjustment for Covariables (prediction modelling) Output to WP2C Health Economics Cost Effectiveness Threshold Variation in cost effectiveness according to instrument, threshold and procedure Output to WP2C 1. USER Selection of scoring system 2. USER Selection of threshold Incorporation into ACHE tool User Opinion Survey in Primary Care (n=10 GP practices) EXTENDED USER GROUP CONSULTATION Consensus opinion patients, healthcare professionals, and commissioners USER Group Meeting 2 Candidate scores identified Output to WP3 Fit for purpose. Full roll out in NHS Output to WP2A & WP2B ACHE - Arthroplasty Candidacy Help Engine, Price & Beard et al (HTA programme)

42 Work Package 3 EXPLORE THE POTENTIAL IMPACT OF THE ACHE TOOL AND DETERMINE THE ACCEPTABILITY OF THIS APPROACH TO STAKEHOLDERS AND PATIENTS. 3.1 Determine the potential impact of using the ACHE tool in the NHS. Randomly selected block of 200 Hip and 200 Knee patients referred to the NOC referral for consideration for arthroplasty 3.2 Survey evaluation of potential users opinion of the ACHE tool; (GPs and patients). a) ACHE piloted on GPs who participate in cognitive interviews b) Online/postal survey of patients and public (n=174)

43 3.3 Explore the potential acceptability and feasibility of the general approach and the calculated thresholds to patients, health care practitioners and commissioners. Extended USER group consultation 10 GPs 10 patients 5 commissioners 5 orthopaedic surgeons 5 extended scope physiotherapists To identify levels of consensus about thresholds and tools and specify any challenges.

44 ACHE a tool for GPs Identifies candidates for arthroplasty

45 Summary It may be possible Measurement properties underpin most concepts Should not use for individual decision making until all pieces of jigsaw are in place Methodology Evaluate impact Value

46 Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Science, Institute of Musculoskeletal Science, Botnar Research Centre Funding Acknowledgements Biomedical Research Unit, Oxford

47 What is a construct? In psychology, non-observable variables are called constructs (or latent variables). Examples would be a person's motivation, anger, personality, intelligence, love, attachment, or fear. Constructs can be observed indirectly by measuring observable indicators or items/responses on questionnaires (or response variables ), that are related to the construct. Usually, multiple items are used to measure a construct of interest.

48 What are measurement properties? The measurement properties of a PROM relate to the data that has been collected, to determine how well it measures the construct of interest. In order to develop a good PROM, the new test is subjected to statistical analyses to ensure that it has good measurement properties.