Plenary Session 2 Psychometric Assessment. Ralph H B Benedict, PhD, ABPP-CN Professor of Neurology and Psychiatry SUNY Buffalo

Size: px

Start display at page:

Download "Plenary Session 2 Psychometric Assessment. Ralph H B Benedict, PhD, ABPP-CN Professor of Neurology and Psychiatry SUNY Buffalo"

Jordan Brown
5 years ago
Views:

1 Plenary Session 2 Psychometric Assessment Ralph H B Benedict, PhD, ABPP-CN Professor of Neurology and Psychiatry SUNY Buffalo

2 Reliability Validity Group Discrimination, Sensitivity Validity Association with ADLs Validity Ability to Detect Clinically Meaningful Change

Cognitive Performance PRO Patient report Informant Report Composite MACFIMS CES-D-10 Depression -.45 -.64.61.

3 Cognitive Performance PRO Patient report Informant Report Composite MACFIMS CES-D-10 Depression Benedict RHB, et al (2004). Reliable Screening for Neuropsychological Impairment in MS. Multiple Sclerosis, 10:

5 100 MS Patient Volunteers from Community 100 NCs Most Sensitive Tests: Selective Reminding Test 7/24 Spatial Recall Test Paced Auditory Serial Addition Test Brief Repeatable Battery of Neuropsychological Tests for MS Selective Reminding Test 10/36 Spatial Recall Test Symbol Digit Modalities Test Paced Auditory Serial Addition Test Controlled Oral Word Association aka Verbal Fluency

6 % 22-31% % 12-19% 7-8% 13-19% 0 Language Spatial Abilities Attention Span Processing Speed Memory Executive Rao et al

7 Recommended Rao Battery: Problems No assessment of visual/spatial or executive function Psychometrics of adapted versions of SRT and 7/24 Spatial Recall Test unknown Validity of adapted version of SDMT unknown Lack of information on reliability: test-retest alternate form

8 SDMT test 1 vs test 2 r = 0.84 PASAT test1 vs test3 r = 0.70 Acceptable > 0.80 Costs of low reliability : - less confidence in accuracy in clinical settings - error variance reduces statistical power

9 Minimal Neuropsychological Assessment of MS Patients: A Consensus Approach Benedict Fischer et al. Clinical Neuropsychologist, 2002 Standardized Manual, commercially available Normative Data Published, large sample, age Adequate Range Reliability Criterion Validity Alternate Forms Practical No ceiling or floor effects. Moderate/high test-retest reliability. Discriminates MS and controls, good sensitivity & specificity. Alternate forms are equivalent and reliable. Brief, minimal equipment, minimal confound of neurol signs.

10 Test Cognitive Domain Min Paced Auditory Serial Addition Test Processing Speed/Working Memory 10 Symbol Digit Modalities Test Processing Speed/Working Memory 5 California Verbal Learning Test Learning and Memory 25 Brief Visuospatial Memory Test Learning and Memory 10 D-KEFS Sorting Test Executive Function 25 Judgment of Line Orientation Test Visual-Spatial Ability 10 Controlled Oral Word Association Test Language and Other Domains 5 Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS) Benedict et al, 2002

11 ns TestxTrial F=4.7 p<.05 T1 p<.001 T5 p=.05 Delay p<.01 p=.07 Trial F=49.7 p<.001 TestxTrial ns T1 ns T5 ns Delay ns

13 Test-Retest r CVLT2 Total Learning 0.78 CVLT2 Delayed Recall 0.89 BVMTR Total Learning 0.91 BVMTR Delayed Recall 0.85 PASAT Total Score 0.94 SDMT 0.97 COWAT 0.84

14 291 MS pts 56 controls Validity Visual Processing Speed and Memory Tests Most Sensitive in MS 59.5% impaired on 2 or more measures TEST % z< -1.5 COWAT 13.1 JLO 22.3 CVLT-II Learning 29.6 CVLT-II Delayed Recall 34.4 BVMT-R Learning 54.3 BVMT-R Delayed Recall 56.0 PASAT 3.0 ISI 27.4 PASAT 2.0 ISI 27.5 Symbol Digit Modalities Test 51.9 DKEFS Sorting Test Categories 15.8 DKEFS Sorting Test Description 25.8 Benedict RHB, et al (2006). J Inter Neuropsychol Soc 12:

15 % 22-31% % 12-19% 7-8% 13-19% 0 Language Spatial Abilities Attention Span Processing Speed Memory Executive Benedict et al MACFIMS data from 291 pts, superimposed on data adapted from Rao et al

16 Model Predicting disabled/employed status, conservatively defined (n=102) Odds* Ratio 95% Confidence Interval p Model Predicting disabled/employed status, liberally defined (n=162) Odds* Ratio 95% Confidence Interval p Depression Disease Course <0.001 CVLT2-DR BVMTR-DR ns ns PASAT ns SDMT ns ns DKEFS ns

17 Parmenter B, Testa SM, Schretlen DJ, Benedict RHB. (2010). Utility of Regressionbased Norms in Interpreting the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS). J Int Neuropsychol Soc, 16:6-16. Measure Predictor B Standard Error B T Standardized B Total R square SDMT (constant) age age sex education

19 Sample: 65 MS 46 NC 72% RR Mean EDSS = Memory Tests in Counterbalanced Order

20 Strober et al 2009: Comparing Effect Sizes Among Gold Standard Cognitive Outcomes

21 MACFIMS Rao BRNB 90 min testing 45 min testing Brief Assessment?? New Learning and Recall Memory Processing Speed and Working Memory Retention of Learned Material

22 Concurrent Validity SDMT most robust correlate with structural imaging MRI Partial r Bermel et al, 2003 BCR Benedict et al, VW Amato et al, 2004 Cortical Vol Christodoulou et al, 2004 Vent Vol Benedict et al, 2005 Temp Lobe Vol Benedict et al, 2006 BPF Benedict et al, VW Houtchens, Benedict, et al, 2007 Thalamic Vol Tekok-Kilic, Benedict, et al, 2007 R Sup Frnt GM Benedict, et al, 2009 Deep GM +0.62

23 Discriminative Validity - SDMT is most sensitive Camp et al. Brain. 1999;122: Benedict et al. JINS; 12: Benedict et al. Arch Neurol; 63: Deloire et al. JNNP. 2005;76: Deloire et al. MS. 2007;13:52-57 Huijbregts et al. Neurol. 2004; 63: Nocentini et al. MS. 2006;12:77-87 Parmenter et al. MS. 2007;13:52-57 Strober et al. MS. 2009; in press

24 SDMT in Longitudinal Work: Amato et al. Mult Scler, 2010

25 SDMT in Longitudinal Work: Amato et al. Mult Scler, 2010

26 Morrow SA, O Connor PW, Polman CH, Goodman AD, Kappos L, Lublin FD, Rudick RA, Jurgensen S, Paes D, Forrestal F, Panzara MA, Benedict RHB (2010). Evaluation of the Symbol Digit Modalities Test (SDMT) and MS Neuropsychological Screening Questionnaire (MSNQ) in Natalizumab treated MS patients over 48 weeks. Multiple Sclerosis, 16: pts, 109 sites in 21 countries Translated into 14 languages

28 Morrow SA, Jurgensen S, Forrestal, Munchauer FE, Benedict RHB. (2011). Effects of acute relapses on neuropsychological status in multiple sclerosis patients. Journal of Neurology, in press. 5% Drop 65 MS cases with confirmed relapses: age 42.9±7.8, 66.2% female, median EDSS 3.0 (range ). No significant difference on demographics or EDSS at baseline.

29 Adaptation of SDMT to the MSFC: Drake et al. Mult Scler, 2010

30 Symbol Digit Modalities Test in MSFC Test-Retest over 2 years MSFCPjnims r = 0.84; MSFCSjnims r = 0.83 MSFCPjninc r = 0.80; MSFCSjninc r = 0.81

31 BICAMS Brief International Cognitive Assessment for MS BICAMS Member Meeting September 11, 2010 Chairs: Dawn Langdon and Ralph Benedict 1. The committee will recommend a brief cognitive assessment that is optimised for small centres, with perhaps one or few staff members, who may not have neuropsychological training. 2. The recommended assessment will be constructed to maximise international use, although in the first instance suitability may only be established for certain areas of the world. 3. The committee will review relevant scientific papers and evaluate possible scales for psychometric qualities, international application, ease of administration, feasibility in the specified context and acceptability to patients. 4. A consensus opinion article will be submitted by the committee, which will comprise the objectives, a literature review, and recommendations

32 Language and Spatial Processing Controlled Oral Word Association Test Judgment of Line Orientation Test PSYCHOMETRIC AVE PRAGMATIC AVE Memory Wechsler Memory Scale Rey Auditory Verbal Learning Test California Verbal Learning Test Selective Reminding Test Brief Visuospatial Memory Test Revised 10/36 or 7/24 Spatial Recall Test PSYCHOMETRIC AVE PRAGMATIC AVE Attention, Processing Speed, Working Memory Digit Span Spatial Span Wechsler Digit Symbol or Coding Test Symbol Digit Modalities Test Paced Auditory Serial Addition Test Stroop Conflict Tasks N-Back Tasks PSYCHOMETRIC AVE PRAGMATIC AVE Higher Executive Function Wisconsin Card Sorting Test Tower Tasks, eg Tower of London DKEFS Sorting Test Raven's Progressive Matrices PSYCHOMETRIC AVE PRAGMATIC AVE

33 Table 1 Manner in which BICAMS measures meet psychometric criteria in samples with English as a first language. SDMT CVLT2 Learning BVMTR Learning Standardization Smith 1982 Teat Manual Delis 2000 Test Manual Benedict 1997 Test Manual Normalization Parmenter Peer review journal article Delis 2000 Test Manual Benedict 2005 Test Manual Reliability I: Test-Retest Benedict 2005 r = 0.91 Benedict 2005 r = 0.80 Benedict 2005 r = 0.91 Reliability II: Alternate Form Rao No or Little Data Delis Fair Benedict Good Validity I: Criterion Related Validity II: Clinically Meaningful Change Many Studies. Good Many Studies. Good Many Studies. Good Morrow Fair No or Little Data No or Little Data

Multiple Sclerosis (MS) CDE Working Group Chairman: [Ralph HB Benedict] Subgroup members: [Dawn Langdon, Lauren Krupp, Nicholas LaRocca, Steve Rao] Which psychometric

34 Multiple Sclerosis (MS) CDE Working Group Chairman: [Ralph HB Benedict] Subgroup members: [Dawn Langdon, Lauren Krupp, Nicholas LaRocca, Steve Rao] Which psychometric tests should be recommended as a brief battery ie supplemental CDE for studies focusing on cognition in MS? Should any test be recommended for the overall core MS CDE?

35 Test Domain Time CVLT2 Learning Trials 1-5 auditory/verbal memory 10 min BVMTR Learning Trials 1-3 visual/spatial memory 05 min Rao PASAT 3.0 ISI Rao SDMT processing speed and working memory 05 min processing speed and working memory 05 min DKEFS Sorting Test executive function 10 min CVLT2 Delayed Recall auditory/verbal memory 10 min BVMTR Delayed Recall visual/spatial memory 05 min COWAT executive function 05 min

36 Symbol Digit Modalities Test (SDMT) recommended for MS CDE 1. Excellent reliability 2. Excellent discriminative validity 3. Can replace the PASAT in MSFC 4. Clinically meaningful change associated with discrete raw score decline

40 Conclusions Self Report is Invalid, performance measures are and will be needed SDMT is the most cost effective and sensitive test available Should be complemented by verbal and visual memory tests Psychometric standards will be increasingly emphasized in clinical trial work

Cognitive rehabilitation: assessment. Dawn Langdon PhD

Cognitive rehabilitation: assessment Dawn Langdon PhD 1 Assessment for cognitive rehabilitation Patient context Individual Family Work Insight Mood Motivation Observation of patient performance Rehabilitation