Devils, Details, and Data: Measurement Models and Analysis Strategies for Novel Technology-Based Clinical Outcome Assessments

Transcription

1 Devils, Details, and Data: Measurement Models and Analysis Strategies for Novel Technology-Based Clinical Outcome Assessments ISCTM 2018 Autumn Meeting Robert M Bilder, UCLA Michael E. Tennenbaum Family Professor Psychiatry & Biobehavioral Sciences and Psychology David Geffen School of Medicine Semel Institute for Neuroscience and Human Behavior

2

3 New clinical outcomes assessment methods require new strategies Changes compared to old-fashioned RCTs Traditional RCT - primary endpoint was usually: A test summary score Reflecting performance across a fixed bunch of items From a single test instrument That was administered by a trained human At one point in time With results recorded on a clinical record form and Then transcribed into a database for analysis

4 New behavior sampling methods require new strategies Changes compared to oldfashioned RCTs with primary endpoint include: Dense temporal sampling Multivariate sampling Passive sampling Machine sampling More direct sampling of biological variables

5 Temporal sampling density Increased density of observations (from mobile, wearable or IOT) Sampling may occur more than 1 per second consider: 8 weeks x 7 days x 24 hours x 60 minutes x 60 seconds = 4.84M measures Analyze trajectories rather than simple changes from baseline to endpoint

6 Pros and Cons of Laboratory Assessments

7

8

9 SS Dot n-back occasion 1-day (x5) 2-day (x10) 3-day (x15) 14-day (x70) 1-occasion 1-day (x5) 2-day (x10) 3-day (x15) 14-day (x70) 1-occasion 1-day (x5) 2-day (x10) 3-day (x15) 14-day (x70) 12 items/test 2 items/test 12 items/test Are the advantages of repeated measures over time any greater than you would expect simply from having more items? Reliability (alpha) is a function of average inter-item covariance (c-bar), average item variance, and N of items. Reliability predicted from estimated c-bar is correlated with observed reliability over repeated measures (across 3 tasks x 4 time points: r =.96)

10 Multivariate sampling Single mobile device yields multiple outputs in different modalities GPS Motion Voice Video: light/dark, facial affect, oxygenation EMA GSR HR, HRV Or data may be integrated across multiple devices Smart watch or actigraphy Skin patch sensor Sleep respiration monitor EEG, EKG, etc Methods to aggregate all these data types into composite COAs under development

11 Overall, correlations were low-to-moderate with a mean of 0.37 (SD = 0.25) and a range of to 0.98 Passive sampling = direct, more objective Less censoring and bias of data related to: Compliance Effort Intent Examinee less prepared for assessment Measures less likely to be affected by expectancy biases Presumably better at overcoming placebo effects A comparison of direct versus self-report measures for assessing physical activity in adults: a systematic review; Prince et al 2008

12 Machine sampling Increased precision Probably decreased flexibility All flexibility must be programmed in advance (there is no on the fly flexibility that occurs with humans, for better or worse) Interaction monitoring still early (e.g., interactive video monitoring of engagement during assessment) Unclear impacts on human responders Tech naïve older adults vs early adopters Consider rod & frame studies

13 BUT we still face the same reliability and validity concerns Reliability Internal consistency, construct validity Test-retest reliability: stability, bias, effects of repeated measurement Inter-rater, Inter-site, Inter-national reliability At least as good as conventional measures? Criterion validity With respect to existing measures With respect to clinical outcomes At least as good as conventional measures?

14 Using IRT for co-calibration of tests and longitudinal assessment Test linking Quantify shared latent trait that both instruments measure Typically requires at least some linking or anchor items Examine differential item functioning (DIF) for anchor items Summaries include: Test characteristic curves: plot most likely score for each level of ability Test information curves: plot measurement precision at each level of ability Assumption that test characteristics are constant over time is probably wrong Regression and change score approaches all assume linearity across scale not true for virtually any test

15

16 From Crane et al 2008

17

18 Methods to Assure Equivalency General measurement invariance issues, using multiple group confirmatory factor analysis (CFA) Equal form: The number of factors and the pattern of factor-indicator relationships are identical across groups (aka configural equivalence). Equal loadings: Factor loadings are equal across groups (aka metric equivalence). Equal intercepts: When observed scores are regressed on each factor, the intercepts are equal across groups (aka scalar equivalence). Equal residual variances: The residual variances of the observed scores not accounted for by the factors are equal across groups (aka uniqueness equivalence).

19 Measurement Invariance Methods for Introducing New Methods into Clinical Trials Assessment of measurement invariance typically requires: Shared linking items across instruments that serve as anchors against which other aspects of covariance can be judged Absent linking items, comparability can be established by studying the same people with both methods. This is the conventional criterion validity approach or assessment of concurrent validity. Other strategies are possible for integrative data analysis, sometimes even without linking items and without having a shared sample: Variable network harmonization Covariance structure harmonization Factor alignment

20 Classical psychometric and network approaches to measurement invariance Psychometric model Assumes latent variable Constrains correlations Dysphoria Psychometric Major depression Insomnia Anhedonia Appetite Appetite Network model No constraints on correlations Saturated model If networks harmonize so will factor model so will composites Network Appetite Dysphoria Appetite Anhedonia Insomnia

21 Method Harmonize matching symptoms bottom up or backward-search method) 1. No initial constraint on correlations ( fully saturated model) 2. Add constraints until fit is maximized CFI: scale from worst (0) to best (1) possible fit; >.95 RMSEA: misfit per degree of freedom; <.05 SRMR: size of model residuals; <.05 Backwards search algorithm, minimizing loss function: LOSS = MAX RMSEA, SRMR, 2 1 CFI. 3. Identify and diagnose non-harmonized symptoms Content/wording differences Language differences Measurement scale/response option differences Population differences in symptom expression

22 Depression Matching symptoms Model fit: CFI=.992, RMSEA=.061, SRMR=.089 SCID N=1290 Symptom Name SCID DI-PAD Dysphoria (Depression) A52 OPCRIT37 Loss of pleasure A53 OPCRIT39 Weight loss/decreased appetite A55 OPCRIT489 Weight gain/increased appetite A56 OPCRIT501 Insomnia A58 OPCRIT4456 Excessive sleep A59 OPCRIT47 Slowed activity A62 OPCRIT24 Loss of energy or fatigue A63 OPCRIT25 Inappropriate guilt A66 OPCRIT42 Impaired Concentration A68 OPCRIT41 Suicidal ideation A72 OPCRIT43 DI-PAD N=3344

23 Depression Matching symptoms Model fit: CFI=.999, RMSEA=.032, SRMR=.038 SCID N=1290 Symptom Name SCID DI-PAD MAD r Dysphoria (Depression) A52 OPCRIT37 Loss of pleasure A53 OPCRIT Weight loss/decreased appetite A55 OPCRIT489 Weight gain/increased appetite A56 OPCRIT501 Insomnia A58 OPCRIT Excessive sleep A59 OPCRIT Slowed activity A62 OPCRIT24 Loss of energy or fatigue A63 OPCRIT25 Inappropriate guilt A66 OPCRIT42 Impaired Concentration A68 OPCRIT41 Suicidal ideation A72 OPCRIT43 DI-PAD N=3344

24 Depression Non-matching symptoms SCID N=1290 Symptom Name SCID DI-PAD Psychomotor agitation A61 Feelings of worthlessness A65 Indecisiveness A69 Recurrent thoughts of death A71 Specific plan A73 Suicide attempts A74 Altered libido OPCRIT40 Diurnal variation OPCRIT38 Residual variance Low residual variance DI-PAD N=3344 Residual correlation

25 IRT-Based Harmonization DI-PAD (Bipolar) SCID (Dutch bipolar)

26

27

28 Next, alignment proceeds as in the continuous case by minimizing the graded response model (GRM) complexity function: F GRM = p g 1 <g 2 w g1,g 2 f λ pg1,1 λ pg2,1 + p w g1,g 2 f ν pqg1,1 ν pqg2,1 g 1 <g 2 q Note the extra summation in the second term, which accounts for multiple measurement intercepts in the graded response model. After the model parameters are aligned in the factor analytic metric, the aligned IRT model parameters are given by the following transformations: a pg1,1 = 1.7 λ pg 1,1 ψ pg 2 1 λ pg1,1ψ pg d pqg1,1 = d pqg1,1 a pg1,1 α g With these modifications, the final alignment complexity function is given by F GRM = g 1 <g 2 p I 1,p I 2 w g1,g 2 f λ pg1,1 λ pg2,1 + w g1,g 2 f ν p0g1,1 ν p0g2,1 g 1 <g 2 p I 1,p I 2 As described above, measurement non-invariance is only minimized for items which appear in each pair of instruments, and only the first measurement intercept is considered.

29 ~20,000 cases with schizophrenia, schizoaffective disorder, bipolar disorder, major depressive disorder and autism spectrum disorder, relatives and controls from >10 cohorts

30

31 Many thanks! Consortium for Neuropsychiatric Phenomics (52 investigators); Investigators in current RDoC projects, and Whole Genome Sequencing in Psychiatric Disorders (WGSPD; Freimer et al.). Special thanks to Steve Reise, Max Mansolf, Annabel Vreeker, Catherine Sugar, Gerhard Helleman, and Ariana Anderson. Supported by NIH Grants: (CNP) UL1-DE019580, RL1MH083268, RL1MH083269, RL1DA024853, RL1MH083270, RL1LM009833, PL1MH083271, and PL1NS062410; (Cognitive Atlas) RO1NS061771; (Multilevel WM/RDoC) R01MH101478; (Modeling/RDoC) R03MH106922; (WGSPD) U01 MH