Evidence-Based Assessment: From simple clinical judgments to statistical learning

Transcription

1 Evidence-Based Assessment: From simple clinical judgments to statistical learning Eric A. Youngstrom, PhD University of North Carolina at Chapel Hill

2

3 Looking at clinical decision-making from many angles My journey from age 18 Researcher/Teacher/Clinician/Parent Revisioning my research How to teach it better? Or transfer the flag? How to incorporate new research, more information?

4 Early 1990s Where were you? (Working on master s thesis) First modern sightings of pediatric bipolar Geller 1993 Depression Trial Wozniak 1995 JAACAP paper (ADHD sample) 1999 Papolos book

5 Evaluating Diagnostic Efficiency: Receiver Operating Characteristics (ROC) Test Scores V. High Medium Low Sensitivity False Alarm Rate (1 Specificity)

6 A Visual Comparison of Diagnostic Efficiency 1.00 ROC Curve Measure 1 clearly performs better across a wide range of scores. Source of the Curve Sensitivity Ref erence Line Measure 2 Measure 1 AUC #1 =.94 AUC #2 =.79 (Measure 1 is better, p <.0005) 1 - Specificity

7 Areas Under the Curve (AUC) Excellent:.90 + Good:.70 to.89 Fair:.60 to.69 Poor: <.60 Be suspicious! Chance:.50 (If you get a number significantly below.50, you are using a good test backwards!)

8 (come to the workshops!)

9 Bringing Bayes to clinicians LR+ = 3.9

10 Using a Nomogram Add a CBCL Test Result Box #3.1% 1% % 68% Connect dots and read post-test prob. 99% Pre-test Prob. LR+ (3.9) Likelihood Ratio 1%.1% Post-test Prob.??? Box #4

11 Is the Nomogram Worth Using? Clinical Judgment Nomogram Estimated Probability of Bipolar Diagnosis % Probability (Adding Test Result) Most tend to overdiagnose Still extreme range of opinion Frequency N = 610 clinicians, 13 sites

12 Is the Nomogram Worth Using? Clinical Judgment Nomogram Estimated Probability of Bipolar Diagnosis % Probability (Adding Test Result) Much Reduces more accurate overdiagnosis Much less range of opinion Frequency N = 610 clinicians, 13 sites

13 Statistical Learning Models Count how many buzzwords you have heard: q Data mining, q Machine learning, q Watson, q Statistical learning q big data, q Internet of Things It s not just for psychology: Netflix, Amazon, IBM, Google Turns out that most of the methods are things that we learned in grad school! Key is to have computer do the heavy work: Automate the model building and testing Bias-Variance Trade-off (~Type I versus Type II error) Use internal cross-validation to pick a model that is likely to generalize

14 IBM Watson wins on Jeopardy! Natural language, unlike chess Better approximates clinical interview Medical decision-making 14 February, 2011

15 Putting the proper pieces together at the point of impact can be life changing Patient Medications Symptoms Findings Family History History Symptoms Family History Patient History Medications Findings difficulty swallowing fever dry mouth thirst anorexia frequent urination dizziness no abdominal pain no back pain no cough no diarrhea Oral cancer Bladder cancer Hemochromatosis Purpura Graves Disease (Thyroid Autoimmune) cutaneous lupus osteoporosis hyperlipidemia frequent UTI hypothyroidism Alendronate pravastatin levothyroxine hydroxychloroquine urine dipstick: leukocyte esterase supine 120/80 mm HG heart rate: 88 bpm urine culture: E. Coli Diagnosis Models Renal Failure UTI Diabetes Influenza Hypokalemia Esophagitis Confidence Example of Watson Decision-support Kohn, 2012, IBM

16 LASSO regression

17

18 How to pick? 10-fold crossvalidation

19 Contestants There ain t no such thing! Bet the base rate Take the best screener test positive or negative? Bayes Theorem too hard Nomogram: Just connect the dots! Multilevel Likelihoods, two predictors Logistic regression 1 predictor (every score gets its own prediction) Multiple predictors LASSO (but could also do quadratic discrim, random forests.)

20 Increasing model complexity Unfair advantages:

21 The challenge: Identify cases with bipolar disorder

22 under clinically realistic conditions

23 Place your bets human versus LASSO? Criterion Statistical significance? Clinical significance? Best accuracy? Usability? Bet Both Both

24 Not much of a contest We know that regression will produce optimized weights LASSO is getting extra variables that clinician wouldn t Next questions: How much better is the statistical model? And would it work at your clinic? ßthe external validation question

25 Plot twist: There s a second clinic Academic Community

26 Academic and Community Samples: Different on almost every variable

27 Academic and Community Samples: Big differences in diagnoses

28 Score distributions on PGBI-10M

29 Round 1 results: Academic Clinic

30 PGBI-10M works in both clinics

31 Round 2 results: Community Clinic

32 Good news! PGBI & Family History Discovery! White x Anxiety But: many more predictors in Academic than Community?

33 Just when you thought it was over ROUND 3! What if we used billing diagnoses to train the model?

34 Now only $299 8/23/12

35 (Rapidly changing ethics and guidelines! January 2014)

36 June 2016 Version (with 20% discount for additional people in 2017)

37

38 Conclusions Naïve Bayesian approaches (even nomogram) would be a big step forward They generalize better than expected Can include local rates, information LASSO, etc. More accurate in training sample External validity is a big hurdle need more implementation support

39 Wikipedia: Best of the Free Assessments Write pages for free use tools that have good score psychometrics across samples Link to copies of measures Solves Awareness and Access issues Supported by grants from SCCAP, APS, SSCP, APA CODAPAR & D12

40 Free Evidence-based Assessments (and embed the interpretation) Also on Wikipedia & Wikiversity Can help us frame the feedback & suggest resources: ck

41 Thank You!