Enhancing rigour in the validation of Patient Reported Outcome Measures (PROMs): bridging linguistic and psychometric testing

Size: px

Start display at page:

Download "Enhancing rigour in the validation of Patient Reported Outcome Measures (PROMs): bridging linguistic and psychometric testing"

Blaise Blake
5 years ago
Views:

1 Enhancing rigour in the validation of Patient Reported Outcome Measures (PROMs): bridging linguistic and psychometric testing Gwerfyl Roberts, Seren Roberts, Richard Tranter, Rhiannon Whitaker, Emma Bedson, Siobhan Tranter, Delyth Prys, Heledd Owen, Yvonne Sylvestre

Background Patient Reported Outcome Measures (PROMs) used increasingly in clinical practice and research Gaining insight from patient s perspective on aspects of health and impact of disease on

2 Background Patient Reported Outcome Measures (PROMs) used increasingly in clinical practice and research Gaining insight from patient s perspective on aspects of health and impact of disease on quality of life Role of culture and language in shaping perceptions of health and well-being PROMs translated and adapted for host of target languages Reliability and validity paramount Rigorous multi-step approach to translation endorsed But no recommendations on early assessment of reliability and validity before large-scale testing of psychometric properties 2

3 Overview of Presentation Demonstrate how a thorough and rigorous approach to validation can inform the refinement of translated outcome measures for cross-cultural research and clinical practice. Report on the development of a conceptually equivalent Welsh language version of the Beck Depression Inventory II (Beck et al 1996) Outline preliminary investigations of its psychometric properties when administered to a student and clinical population Demonstrate the importance of linking linguistic and psychometric testing through a purposeful stage which bridges the gap between translation and largescale validation 3

Beck Depression Inventory II (BDI-II) Established in the USA in 1961 and widely used for measuring the severity of depression in individuals aged 13 years and over 21-item self-report inventory

4 Beck Depression Inventory II (BDI-II) Established in the USA in 1961 and widely used for measuring the severity of depression in individuals aged 13 years and over 21-item self-report inventory relating to depressive symptoms, cognition and physical symptoms 4 point scale for each item ranging from 0-3 Total score represent levels of depression 0-13 no depression mild depression moderate depression severe depression 4

5 Beck Depression Inventory II (BDI-II) One of the PROMs recommended by the Welsh and UK Governments for screening depression in high risk populations in primary care Translated into numerous languages and psychometrically robust and appropriate for use in. e.g. Argentina, Mexico, Greece, Japan BUT No Welsh language version available Rigour in translation essential to establish equivalence across measures Performance testing crucial to establish psychometric properties, including reliability and validity 5

7 Study Design Establish a Welsh language version of BDI-II which is conceptually equivalent to the source measure and culturally relevant to the target group Undertake preliminary investigations of its psychometric properties when administered to a student and clinical population 7

8 Establishing Equivalence across PROMs Equivalence Conceptual equivalence Item equivalence Semantic equivalence Operational equivalence Measurement equivalence Definition Constructs exist, are relevant, and acceptable in target language Items represent the constructs and are relevant and acceptable in the target language The meaning attached to each item is comparable across the source and target language The format of the measure and mode of administration is relevant and acceptable in the target language The psychometric properties of the measure are comparable in both language versions 8

9 Methods [1] Linguistic Validation Adopt the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Framework for the Translation and Cultural Adaptation of PROMs (Wild et al 2005) 9

10 Step ISPOR Guidelines (Wild et al 2005) Brief description 1. Preparation Initial work carried out before the translation work begins 2. Forward translation Translation of source language of instrument into target language 3. Reconciliation Comparing and merging more than one forward translation into single forward translation. 4. Back translation Translation of new language version back into source language 5. Back translation review Comparison of back-translated version of instrument with source version to highlight and investigate discrepancies between source and reconciled translation, which is then revised in process of resolving the issues 6. Harmonization Comparison of back translations of multiple language versions with each other and source instrument to highlight discrepancies between original and derivative translations, as well as to achieve consistent approach to translation problems 7. Cognitive debriefing Testing instrument on small group of relevant patients or lay people in order to test alternative wording and check understandability, interpretation, and cultural relevance of translation 8. Review of cognitive debriefing results and finalization Comparison of patients or lay persons interpretation of translation with source version to highlight and amend discrepancies 9. Proofreading Final review of translation to highlight and correct any typographic, grammatical or other errors 10. Final report Report written at end of process documenting development of each translation 10

11 Results [1] Linguistic Validation Step 1 BDI-II S Step 2 BDI-II TV1 BDI-II TV2 77 differences Step 3 BDI-II TV3 70 differences Step 4 BDI-II B 3 amendments Step 5 BDI-II TV4 Step 7 Cognitive debriefing Step 8 BDI-II TV5 12 amendments Step 9 Proof reading Step 10 BDI-II TV6

12 Results [1] Linguistic Validation No conceptual errors identified Although different words and phrasings used by different translators and back translators Differences categorized according to synonyms, language register, grammar and other

13 Results [1] Linguistic Validation Synonyms: Forward Translations Source Forward Translation 1 Forward Translation 2 activities gweithgareddau gwneud pethau before ynghynt o r blaen beside gerllaw ger completely yn gyfangwbl yn llwyr criticize barnu beirniadu difficulty anhawster trafferth

14 Results [1] Linguistic Validation Synonyms: Back Translation Source a lot before crave food picked pleasure worthwhile much Synonym in Back Translation previously starving chosen enjoyment valuable

15 Results [1] Linguistic Validation Language Register English Welsh Formal Welsh Informal I feel sad Rwy n teimlo n drist Rydw i n teimlo n drist I can t get Nid wyf yn cael Alla i ddim cael I do not Nid wyf Dydw i ddim I don t have Nid oes gennyf Does gen i

16 Results [1] Linguistic Validation Grammar Source Welsh Abstract Noun Welsh Verb feelings teimladau teimlo punishment cosb cosbi self-dislike diffyg hoffter o r hunan diffyg hoffi fy hun selfcriticalness Hunan-feirniadaeth beirniadu fy hun thoughts meddyliau meddwl indecisivness Diffyg pendantrwydd trafferth penderfynu

17 Methods [2] Psychometric Testing Hypotheses: The Welsh BDI-II will have: 1. A two-factor structure similar to the model presented in the original manual (Beck 1996) 2. Adequate correlations with other accepted depression scales, and negative correlations with quality of life scales 17

18 Methods [2] Student Sample 115 bilingual university students Data collection during 2009 in classroom setting Administration of following measures: BDI-II (English) European Quality of Life-5 Dimensions (EQ- 5D) (Welsh) Hospital Anxiety and Depression Scale (HADS) (English) Short-Form 12-item Health Survey version 2 (SF-12 v2) (English) BDI-II (Welsh) 18

19 Methods [2] Clinical Sample 37 bilingual patients with depression Recruited between 2009 and 2010 through Folate Augmentation of Treatment Evaluation for Depression (FolATED) trial Administration of following measures: BDI-II (English) Researcher-rated Montgomery-Asberg Depression Rating Scale (English) (SF-12 v2) (English) EQ5-D (English) BDI-II (Welsh) 19

20 Ethical Approval Student study - Bangor Universality School of Healthcare Sciences Ethics Committee Clinical study - Multi-centre Research Ethics Committee for Wales 20

21 Analysis All data anonymised and analysed using PASW and AMOS for Windows (version 18.0) All statistical tests two-sided and P-values of 0.05 considered statistically significant 21

22 Characteristics of Study Samples Characteristic Cognitive testing Sample Student Clinical N Gender, n (%) Age, n (%) Female 5 (62.5) 94 (81.7) 22 (59.5) Male 3 (37.5) 21 (18.3) 15 (40.5) <17 years 1 (12.5) years 2 (25) 73 (63.5) 1 (2.7) years 2 (25) 18 (15.7) 7 (18.9) years 1 (12.5) 16 (13.9) 13 (35.1) years 2 (25) 7 (6.1) 7 (18.9) > 54 years 0 (0) 1 (0.9) 9 (24.3)

23 Results [2] Internal Consistency Characteristic Sample Student Clinical N BDI-II: Welsh α=0.90 α=0.96 BDI-II: English α=0.87 α=0.92 α=0.93 ( Beck et al 1996)

24 Results [2] Factor Analysis 2 factor structure emerging from both samples for each language version One over-riding depression-related dimension But confirmatory factor analysis of the student data revealed that the three indices did not meet the criteria for good fit (GFI=0.54, AGFI=0.47, RMR=0.06)

Results [2] Concurrent and Discriminant Validity Scale HADS: Anxiety HADS: Depression EQ-5D SF-12v2: MCS SF-12v2: PCS Student sample Welsh BDI-II.66**.07** -.69** -.74.11 English BDI-II.67**.66** -.

25 Results [2] Concurrent and Discriminant Validity Scale HADS: Anxiety HADS: Depression EQ-5D SF-12v2: MCS SF-12v2: PCS Student sample Welsh BDI-II.66**.07** -.69** English BDI-II.67**.66** -.67** -.69**.08 Clinical sample Welsh BDI-II ** -.71**.21 English BDI-II ** -.66**.22 **. Correlation is significant at the 0.01 level (2-tailed). MSC: Mental Component Summary, PSC: Physical Component Summary

26 Results [2] BDI-II Depression Scores Characteristic Sample Student Clinical N BDI-II: Welsh BDI-II: English Range Mean (SD) 5.1 (5.9)* 38.4 (11.9) Median (IQR) 3 (1 to 7) 39 ( ) Missing 4 - Range Mean (SD) 5.7 (5.5)* 37.7 (11.5) Median (IQR) 5 (1.8 to 8) 38 (28.5 to 47) Missing 1 -

27 Results [2] Bland Altman Plot for Welsh and English BDI-II (Student Sample)

% of participants reporting symptom Item-total correlation Weighted Kappa coefficient Generalised McNemar asymmetry Welsh English Welsh English Welsh English Welsh English p-value 1. Sadness 0.05 0.

28 Results [2] Item Level Analysis of the BDI-II (Student Sample) Item Mean S.D. % of participants reporting symptom Item-total correlation Weighted Kappa coefficient Generalised McNemar asymmetry Welsh English Welsh English Welsh English Welsh English p-value 1. Sadness Pessimism Past Failure Loss of Pleasure Guilty Feelings Punishment Feelings* Self-Dislike* Self - Criticalness* Suicidal Thoughts* Crying Agitation Loss of interest Indecisiveness Worthlessness Loss of Energy* Changes in Sleeping Pattern Irritability Changes in Appetite Concentration Difficulty Tiredness or Fatigue* Loss of interest in Sex

Results [2] Items and Potential Interpretations leading to Bias in the Student Sample Item Score English Statement Welsh Statement Welsh translation (lit.

29 Results [2] Items and Potential Interpretations leading to Bias in the Student Sample Item Score English Statement Welsh Statement Welsh translation (lit.) Potential interpretation Dydw i ddim yn teimlo fy mod i n cael fy nghosbi. I don t feel I am being 6. Punishment feelings 0 I don t feel I am being punished 1 I feel I may be punished Rydw i n teimlo y gallwn i gael fy nghosbi. 2 I expect to be punished Rydw i n disgwyl cael fy nghosbi. 3 I feel I am being Rydw i n teimlo fy mod punished i n cael fy nghosbi. Dydw i ddim yn meddwl am ladd fy hun. 9. Suicidal thoughts 0 I don t have any thoughts of killing myself 1 I have thoughts of killing myself but I would not carry them out Rydw i wedi meddwl am ladd fy hun ond fyddwn i byth yn gwneud. punished I feel I can be punished I expect to be punished I feel I am being punished I am not thinking of killing myself I have thought of killing myself but I would never do it 2 I would like to kill myself Fe hoffwn i ladd fy hun. I would like to kill myself 3 I would kill myself if I had the chance Byddwn i n lladd fy hun petawn i'n cael y cyfle I would kill myself if I had the chance 15. Loss of energy 0 I have as much energy as ever Mae gen i gymaint o egni ag erioed. I have as much energy as ever 1 I have less energy than I Mae gen i lai o egni nag I have less energy than I used to have oedd gen i n arfer bod used to have 2 I don t have enough Does gen i ddim digon o I don t have enough energy to do very much egni i wneud fawr o energy to do much of 3 I don t have enough energy to do anything ddim. Does gen i ddim digon o egni i wneud dim. anything I don t have enough energy to do anything Welsh translation reflects stronger punishment feelings Welsh translation reflects stronger suicidal thoughts Welsh translation reflects stronger suicidal thoughts Welsh translation reflects greater loss of energy

Discussion Results support previous findings on the psychometric properties of the BDI-II in relation to i) two-factor structure and ii) concurrent validity with other depression and quality of life

30 Discussion Results support previous findings on the psychometric properties of the BDI-II in relation to i) two-factor structure and ii) concurrent validity with other depression and quality of life measures Thus translation and early validation process relatively successful Despite high correlation, poor fit and bias led to exploration of items of concern Ambiguities in translation at lower end of scale biased response to some items This bias not observed in clinical sample because majority reported symptoms at higher end of scale

31 Recommendations Refinement of Welsh BDI-II Need for scrutiny in the translation of everyday vocabulary Importance of ensuring equivalence in scaling of translated measures Value of investigating item discrepancies through early exploratory psychometric evaluations of translated measures prior to large-scale, psychometric testing Proposal for model enhancement

32 Proposal for Model Enhancement (Wild et al 2005) Step 1 Source instrument Step 2 TV1 TV2 Step 3 TV3 Step 4 Back translation Step 5 TV4 Step 7 Cognitive debriefing Step 8 TV5 Step 9 Proof reading Step 10 STEP 11 Preliminary psychometric testing TV6 Target language instrument

Further Research and Development Linguistic and psychometric validation of suite of depression measures (BDI-II, HADS, PHQ-9) Awaiting funding for All Wales

33 Further Research and Development Linguistic and psychometric validation of suite of depression measures (BDI-II, HADS, PHQ-9) Awaiting funding for All Wales Welsh Language Measures Research Facilitator from NISCHR AHSC Research Methodology Support 2011/12 Prioritisation of further measures, facilitation of uptake

Enhancing rigour in the validation of patient reported outcome measures (PROMs): bridging linguistic and psychometric testing

Roberts et al. Health and Quality of Life Outcomes 2012, 10:64 SHORT REPORT Open Access Enhancing rigour in the validation of patient reported outcome measures (PROMs): bridging linguistic and psychometric