A Cross-sectional, Randomized, Non-interventional Methods Study to Compare Three Methods of Assessing Suicidality in Psychiatric Inpatients Eric A. Youngstrom, Ph.D., Ahmad Hameed, M.D., Michael Mitchell, B.A., Andrew J. Freeman, M.A., Anna Van Meter, M.A., Guillermo Perez Algorta, Ph.D., Alan Gelenberg, M.D., & Roger E. Meyer, M.D.
Acknowledgments Study supported by an investigator initiated award from Pfizer, Inc. to Penn State Hershey Medical Center Alan Gelenberg, MD PI
Primary objective: Compare self-report and clinical interviewbased versions of the older and current editions of the Sheehan Suicide Tracking Scale (S-STS) to the C-SSRS in the assessment of suicidal ideation/behavior in the context of the domains defined by the C-CASA and in relationship to the clinical evaluation of recently admitted psychiatric inpatients.
"'In this world nothing can be said to be certain, except death and taxes." -Benjamin Franklin, 1789 I am realizing Franklin was not a clinical trialist.
C-CASA Classification (circa 2010) Item Classification Mapping 1 Completed suicide Impossible to interview 2 Suicide Attempt CCASA #2 3 Preparatory suicidal behavior CCASA #3 4 Suicidal Ideation CCASA #4 5 Other, no deliberate self harm NA in CSSRS interview 6 Fatal, but not enough information to classify Impossible to interview 7 Self-injurious act, no suicidal intent (NSSI) CCASA #7 8 Self-injurious behavior, intention unknown Interview should not have unknown 9 Nonfatal, not enough information to classify
C-CASA Classification (August 2012)
Secondary Objectives Quantify relative ease, time and means of administration (self-report and clinical interview) Inter-rater and Intra-rater reliability (over time) of the scoring of rater administered form Patient preference re: acceptability of the instruments
Urn Randomization Stratified variables: Age (up to 24 years versus 25 and older) Psychosis (present/absent) Two arms: C-SSRS, new S-STS self report, old S-STS interview C-SSRS, old S-STS self report, new S-STS interview Order counterbalanced within each arm 6 sequences
Other design features Break between interviews Same day completion whenever possible Same interviewer Video recording for re-rating by different judge (n = 90 will be re-rated) Raters trained by Posner and by Sheehan
Demographics (N =199) Variable Descriptive Age M = 38.5 (SD = 12.4) 18 to 65 years Gender 113 female (57%) Race 157 White (79%), 37 Black (18%), 9 American Indian (5%), 6 Asian/Pacific (3%) Hispanic 26 (13%) Married 40 (20%) Education 87% with GED or higher Raters 3 interviewers, 3 different video raters Videos 60 rated (more than 90 available)
Administration Times C-SSRS > New S-STS > Old STS all p <.001 Five minutes or less (ecssrs sig faster)
Minutes to Administer C-SSRS Old STS New S-STS Minimum 3 3 3 25 th Pct 12 4.3 11 Median 16 6 15 75 th Pct 22 7 20 Maximum 57 20 40 Mean (SD) 17.8 (8.7) 6.5 (2.7) 16.2 (7.4) N = 199
Interview Length Much Longer if Endorse Suicidal Ideation
What is a good kappa? K 1.0.9.8.7.6.5.4.3.2.1.0 Excellent Good Fair Poor Cicchetti & Sparrow, 1981 Excellent Fair to Good Poor Fleiss, 1981 Almost Perfect Substantial Moderate Fair Slight Poor Landis & Koch (1977) (Excellent) Very Good Good Questionable Unacceptable Regier et al. (2012) DSM-5
Reliability Standoff Two sources of error affect each estimate: Reliability of Test A, Reliability of Test B C-SSRS Kappa Kappa New S-STS Kappa Old STS
Kappa for Lifetime Attempts (C-CASA #2).68 (.06) C-SSRS.66 (.06) New S-STS.68 (.06) Old STS
Recommended approach for missing or imperfect gold standard Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. New York: Wiley. Zhou, X.-H., Obuchowski, N. A., & McClish, D. K. (2002). Statistical methods in diagnostic medicine. New York: Wiley.
LCA Approach Put all converging measures into analysis Fit two class solution: Diagnosis+ vs. Diagnosis-, Haves vs. Have Nots Save predicted class membership Can estimate sensitivity and specificity of each measure calibrated against latent class (could also test larger number of classes, but diagnostic efficiency statistics no longer meaningful)
Latent Class Model True status, cannot measure perfectly Latent Class Drives performance on measured variables C-SSRS New S-STS Old STS
Latent Class Model Suicidal Behavior (C-CASA 2) Lifetime Kappa Sensitivity Specificity C-SSRS.82.93.94 New S-STS.86.91.85 Old STS.83.97.85 C-CASA 2 Behavior None perfect, all excellent C-SSRS New S-STS Old STS
Latent Class Model Preparatory Acts (C-CASA 3) Lifetime C-CASA 3 Prep acts Kappa C-SSRS.86 New S-STS.73 Old STS.63 Weakest agreement across the measures C-SSRS New S-STS Old STS
Latent Class Model Ideation (C-CASA 4) Lifetime Kappa Sensitivity Specificity C-SSRS.72.99.80 New S-STS.91.99 1.00 Old STS.83.99 1.00 C-CASA 4 Ideation C-SSRS New S-STS Old STS
Latent Class Model NSSI (C-CASA 7) Lifetime C-CASA 7 NSSI Kappa C-SSRS.78 New S-STS.77 Old STS.83 C-SSRS New S-STS Old STS
Kappas for Lifetime Ratings 1.0.9.8.7 K.6.5 New S-STS vs. C-SSRS Old S-STS vs. C-SSRS.4 New S-STS vs. Old S-STS.3.2.1.0 CASA-2 Attempts CASA-3 Prep CASA-4 Ideation CASA-7 NSSI
Kappas for Lifetime Ratings 1.0.9.8 K.7.6.5.4.3 New S-STS vs. C-SSRS Old S-STS vs. C-SSRS New S-STS vs. Old S-STS New S-STS vs. LCA CSSRS vs. LCA Old S-STS vs. LCA.2.1.0 CASA-2 Attempts CASA-3 Prep CASA-4 Ideation CASA-7 NSSI
Kappas for Lifetime Ratings 1.0.9.8 Excellent K.7.6.5 Good Fair.4.3 Poor.2.1.0 Attempts CASA-2 Prep CASA-3 Ideation CASA-4 NSSI CASA-7 New S-STS vs. LCA CSSRS vs. LCA Old S-STS vs. LCA
Kappas for Lifetime Ratings across 4 C-CASA categories 1.0.9.8 K.7.6.5.4.3 New S-STS vs. C-SSRS Old S-STS vs. C-SSRS New S-STS vs. Old S-STS New S-STS vs. LCA CSSRS vs. LCA Old S-STS vs. LCA.2.1.0 Attempts CASA-2 Prep CASA-3 Ideation CASA-4 NSSI CASA-7
Summary Suicidal Behavior Past Month N=73 (37%) Kappa BCa Low High Sensitivity Specificity NPV PPV CSSRS.901.835.958.904.984.947.971 New S-STS.915.852.966 1.000.937 1.000.901 Old S-STS.640.544.724.945.746.959.683 Lifetime N=152 (76%) CSSRS.818.720.908.928.938.804.979 New S-STS.859.772.940.980.854.932.955 Old S-STS.833.726.918.967.854.891.955 Median.85.75.93.96.90.94.96 All values sig p <.0005
Summary Suicidal Ideation Past Month N=180(90%) Kappa BCa Low High Sensitivity Specificity NPV PPV CSSRS.784.637.898.950 1.000.679 1.000 New S-STS.778.615.902.972.842.762.983 Old S-STS.843.672.965.789.994.938.978 Lifetime N=195(98%) CSSRS.720.219 1.000.990.800.667.995 New S-STS.907.660 1.000.995 1.000.833 1.000 Old S-STS.828.492 1.000.990 1.000.714 1.000 Median.81.63.98.98 1.00.74 1.00 All values sig p <.0005
Behavior: Kappa and 95% c.i. (BCa) K 1.0.9.8.7.6.5.4.3.2.1.0 Past Month Lifetime
Ideation: Kappa and 95% c.i. (BCa) K 1.0.9.8.7.6.5.4.3.2.1.0 Past Month Lifetime
Inter-Rater Reliability (more tape coding planned a priori) Lifetime C-SSRS (N =60) New S-STS (n =27) Old STS (n =33) CASA #2 (Attempt).893 (.060) <.0005.658 (.181).001.857 (.098) <.0005 CASA #3 (Prep).651 (.106) <.0005.257 (.178) ns.857 (.097) <.0005 CASA #4 (Ideation).551 (.226) <.0005 -- a -- a CASA #7 (NSSI).799 (.078) <.0005.703 (.137) <.0005.767 (.106) <.0005 Past Month CASA #2 (Attempt).720 (.097) <.0005.684 (.144) <.0005.937 (.062) <.0005 CASA #3 (Prep).800 (.077) <.0005.488 (.163).009.879 (.082) <.0005 CASA #4 (Ideation) 1.000 (.000) <.0005.867 (.129) <.0005 1.000 (.000) <.0005 CASA #7 (NSSI).795 (.114) <.0005.807 (.131) <.0005.841 (.155) <.0005 a Cannot estimate -- 100% ideation on cases rated
Preliminary Inter-Rater Reliability CASA 2012 Categories C-SSRS (N =60) CASA 2012 Category C-SSRS Lifetime C-SSRS Past Month #1 Passive Ideation.734 (.178).924 (.075) #2 Ideation (no method, intent, plan).703 (.160).957 (.043) #3 Ideation+Method (no intent, plan).774 (.124).926 (.052) #4 Ideation+Method+Intent (no plan).884 (.065).934 (.046) #5 Ideation+Method+Intent +Plan.770 (.081).688 (.093) #7 Attempt.893 (.060).720 (.097) #8 Interrupted Attempt.678 (.103).573 (.137) #9 Aborted Attempt.602 (.101).946 (.053) #10 Prep Acts.761 (.085).804 (.094) #11 NSSI.799 (.078).795 (.114) No completed suicides (CASA 2012 #6); all K significant p <.0005
Patient Acceptance: Majority highly satisfied 100 90 80 70 60 50 40 30 20 10 0 d =.08 d =.40 Patient Acceptance (% Max) CSSRS New S-STS CSSRS Old STS
Key Points Validity of all three versions excellent Using better model yields more accurate estimate Agreement with each other good Inter-rater reliability (video) excellent Similar patient, interviewer preferences & tolerability Multiple endpoints complicates analyses and interpretation
Limitations Did not compare CSSRS interview to E version Inpatient unit meant extremely high rate of ideation Important clinical setting & context Challenging for tests (big standard errors) Consumer Reports model generalization across clinical settings vital Need to explore moderators of accuracy Did not assess prospective prediction or treatment response
Future Directions Investigate where the signal is Brown s talk severe ideation more important than mild Start with simple models and measures Have data guide decisions to add complexity
Complexity has a cost Categories df Type I Error Rate Evidence about Model Suicidal? Yes/No 1.05 Probably oversimplified Ideation, Behavior 2.10 (Could differentiate severe ideation from mild ) C-CASA 2010, Interview 4.19 C-CASA 2012 11.43 May be too complex Broadly read C-CASA 2012 15.54 Definitely too complex Endpoints probably not all equally important clinically Cannot easily adjust Type II error too serious What are costs of false positive in drug development or post marketing?
"Everything should be made as simple as possible, but not simpler." -Albert Einstein, 1933
Thank You!
Kappa for Lifetime Preparatory Acts (C-CASA #3).59 (.06) C-SSRS.47 (.07) New S-STS.39 (.06) Old STS
Kappa for Lifetime Ideation (C-CASA #4).66 (.19) C-SSRS.60 (.19) New S-STS.76 (.13) Old STS
Kappa for Lifetime NSSI (C-CASA #7).56 (.06) C-SSRS.61 (.05) New S-STS.61 (.06) Old STS