DOWNLOAD EASYCIE AND DATASET

Transcription

1 DOWNLOAD EASYCIE AND DATASET

2 EASYCIE: A DEVELOPMENT PLATFORM TO SUPPORT QUICK AND EASY, RULE-BASED CLINICAL INFORMATION EXTRACTION JIANLIN SHI, MS MD DANIELLE MOWERY MS PHD FIFTH IEEE INTERNATIONAL CONFERENCE ON HEALTHCARE INFORMATICS 2017 TUTORIAL

3 WHAT IS NATURAL LANGUAGE PROCESSING (NLP)? Natural language processing Structured data (machine interpretable) Classify Extract Clinical Texts Summarize

4 WHAT IS NATURAL LANGUAGE PROCESSING (NLP)? Classify patients for stroke Extract Summarize Mowery D, Hill B, Chapman W, Cannon-Albright Lisa, Majersik J. Development of a knowledge base to support the automatic classification of a computable ischemic stroke phenotype from electronic medical records. Neurology: Genetics; PubMed PMID: ; PubMed Central PMCID: PMC

5 WHAT IS NATURAL LANGUAGE PROCESSING (NLP)? Classify Extract identify ejection fractions Summarize The left ventricular cavity size and wall thickness appear normal. The wall motion and left ventricular systolic function appears hyperdynamic with estimated ejection fraction of 70%. There is near-cavity obliteration seen. ejection fraction = 70 percent Garvin JH, DuVall SL, South BR, et al. Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure. Journal of the American Medical Informatics Association : JAMIA. 2012;19(5):

6 WHAT IS NATURAL LANGUAGE PROCESSING (NLP)? Classify Diagnosis: myocardial infarction (MI).. Extract Summarize create an active problem list Mowery DL, Jordan P, Wiebe J, Harkema H, Dowling J, Chapman WW. Semantic Annotation of Clinical Events for Generating a Problem List. AMIA Annual Symposium Proceedings. 2013;2013:

7 WHY IS NLP SO DIFFICULT? Synonyms coughs cough burning up fever short of breath dyspnea Abbreviations/Acronyms feb febrile or february? n/v nausea/vomiting sob shortness of breath Truncations poss possible Concatenations blurredvision burred vision flus sxs flu symptoms Misspellings & typographic errors nausa nausea diahrea diarrhea Quantifications BP 140/90 hypertension Contextual descriptors no cough cough_negated childhood cough cough_historical brother has cough cough_not_patient return if cough cough_hypothetical Discourse sentences sections notes visits

8 HOW IS NLP USED IN THE CLINICAL DOMAIN? Clinical Decision Support Identifying Medline articles to support clinician information needs (Zhang et al. 2013) Quality Improvement Measuring quality of colonoscopy procedures (Harkema et al. 2011) Hospital Operations Automating the coding of medicall billing codes (Stanfill et al. 2010) Genetic Studies Supporting high throughput phenotyping (Pathak et al. 2013) Biosurvelliance Detecting Influenza from emergency department visits (Ye et al. 2014)

9 USE CASES: CONTRACEPTIVE METHODS Motivation Automatically identifying high-risk women not using contraceptive methods (e.g., intrauterine device) for counseling and reproductive planning could mitigate adverse outcomes. Lori Gawron MD MPH Needs Assessment Extract mentions of contraceptive methods and their contexts from clinical texts Shi J, Mowery D, Chapman WW, Zhang M, Sanders J, Gawron L. Extracting Intrauterine Device Usage from Clinical Texts using Natural Language Processing. ICHI (in press).

10 USE CASES: CONTRACEPTIVE METHODS Learn more about EASYCIE applied to this use case: August 25, :05pm-6:30pm Poster Session 2: #21 Jianlin Shi MD MS Shi J, Mowery D, Chapman WW, Zhang M, Sanders J, Gawron L. Extracting Intrauterine Device Usage from Clinical Texts using Natural Language Processing. ICHI (in press).

11 USE CASES: PNEUMONIA DIAGNOSIS Motivation Diagnosing patients with pneumonia can be difficult due to presentation of non-specific signs and symptoms and elusive pathogen discovery Needs Assessment Extract variables (e.g., fever, rales, worsening cough) associated with pneumonia to improve diagnostic accuracy and reduce cognitive biases. Barbara Jones MD, MSCI South BR, Mowery DL, Kramer H, Jones B, Castine M, Hillert D, Sibitsky M, Chapman WW. Assessing Visualization and Semantic Priming on Classifying Supporting, Refuting, or Uncertain Evidence for Suspected Pneumonia Case Review. (under review)

12 ROADMAP Getting back to the basics Collecting ingredients and reading the recipes Cooking with the easy button ; Multi-tasking in the kitchen Hands on exercise Borrowing a cup of sugar CONFIDENTIAL

13 Getting back to the basics CONFIDENTIAL

14 NLP PIPELINE Sentence Segmentation Tokenization Term Normalization Part-of-Speech Tagging Shallow Parsing Named Entity Recognition Assertion Classification FHx: Sister had childhood fevers. FHx : Sister had childhood fevers. Family History : Sister had childhood fever. Family History : Sister had childhood fever. JJ NN NN VBD JJ NN. Family History : Sister had childhood fever. [ NP ] : [ NP ] [ VP ]. Family History : Sister had childhood fever. UMLS code: Fever- C Family History : Sister had childhood fever. temporality = historical experiencer = nonpatient negation = affirmed

15 NLP PIPELINE Sentence Segmentation Tokenization Term Normalization Part-of-Speech Tagging Shallow Parsing Named Entity Recognition Assertion Classification FHx: Sister had childhood fevers. FHx : Sister had childhood fevers. Family History : Sister had childhood fever. Family History : Sister had childhood fever. JJ NN NN VBD JJ NN. Family History : Sister had childhood fever. [ NP ] : [ NP ] [ VP ]. Family History : Sister had childhood fever. UMLS code: Fever- C EasyCIE processed on backend Family History : Sister had childhood fever. temporality = historical experiencer = nonpatient negation = affirmed

16 USE CASE: PNEUMONIA INDICATORS

17 IDENTIFYING FEVER EXPRESSIONS FROM CLINICAL TEXT Lay terms

18 IDENTIFYING FEVER EXPRESSIONS FROM CLINICAL TEXT Morphology afebrile: a = without febrile = fever

19 IDENTIFYING FEVER EXPRESSIONS FROM CLINICAL TEXT Quantifications Units Celsius Fahrenheit Numbers Whole Decimal

20 IDENTIFYING FEVER EXPRESSIONS FROM CLINICAL TEXT Other contexts Course fever abated Hypothetical return if fever

21 Collecting ingredients and reading the recipe CONFIDENTIAL

22 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC Update rule files

23 ROADMAP POS_DOC NEG_DOC One or more positive mentions of pneumonia indicators No positive mentions of pneumonia indicators

24 ROADMAP Input corpus Define rule files POS_DOC NEG_DOC

25 DEFINE MENTION ANNOTATION LOGIC Inclusionary mention (+) Exclusionary mention (-) Assertions Negation: Affirmed/Certain complains of shortness of breath cannot rule out pneumonia likely pna Temporality: Present dx of bacter. pneumonia return if worsening fever Experiencer: Patient Pt has worsening cough Patient is febrile Value in abnormal range high fever: Temp > F low oxygen saturation: O2 saturation less than 90% Assertions Negation: Negated/Uncertain denies shortness of breath rule out pneumonia unlikely pna Temporality: Historical/Hypothetical history of bacter. pneumonia return if fever Experiencer: Non-patient brother has worsening cough roommate is febrile Value in normal range Normal or low fever: Temp <101.3 F low oxygen saturation: O2 saturation under 90% CONFIDENTIAL

26 IDENTIFY TARGETS AND MODIFIERS Finding (target): Negation(modifier): cough negated Finding (target): Negation(modifier): headache (default) affirmed scope Patient denies cough but complains of headache. trigger term termination term NegEx algorithm Courtesy: Wendy Chapman

27 IDENTIFY FEVER AND MODIFIERS Finding (target): Negation(modifier): Uncertain (modifier): Temporality (modifier): Experiencer (modifier): fever (default) affirmed (default) certain (default) present (default) patient She stated she was burning up..

28 IDENTIFY FEVER AND MODIFIERS Finding (target): Negation(modifier): Uncertain (modifier): Temporality (modifier): Experiencer (modifier): fever negated (default) certain (default) present (default) patient fever had abated..

29 IDENTIFY FEVER AND MODIFIERS Finding (target): Finding Negation(modifier): (target): Negation(modifier): Uncertain Temporality (modifier): Experiencer (modifier): fever (default) affirmed fever (default) affirmed certain (default) present (default) patient temperature of 38C..

30 IDENTIFY FEVER AND MODIFIERS Finding (target): Negation(modifier): Uncertain (modifier): Temporality (modifier): Experiencer (modifier): fever negated (default) certain (default) present (default) patient She improved and became afebrile..

31 ROADMAP Input corpus Define rule files Apply the rules POS_DOC NEG_DOC

32 DEFINE TASK LOGIC AKA THE RECIPE Is this mention an indicator of pneumonia? Is the mention inclusionary (+)? Is the mention exclusionary (-)? Mark mention Ignore mention Is there one mention of a pneumonia diagnoses? Classify document as POS_DOC Classify document as NEG_DOC Step 1: Classify indicators of pneumonia in document Step 2: Classify whether document contains 1+ indicators CONFIDENTIAL

33 DEFINE MENTION ANNOTATION LOGIC Is this mention an indicator of pneumonia? Is the mention inclusionary (+)? Is the mention exclusionary (-)? Mark mention Ignore mention Patient has fever Patient should return if febrile

34 DEFINE DOCUMENT CLASSIFICATION LOGIC Is there one mention of a pneumonia indicator? Classify document as POS_DOC Classify document as NEG_DOC Patient has PNA Pneumonia unlikely

35 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC

36 HOW WELL DOES NLP DETECT INDICATORS? NLP classified = POS_DOC = NEG_DOC Expert reviewed = POS_DOC True positive False negative = NEG_DOC False positive True negative

37 HOW WELL DOES NLP DETECT INDICATORS? NLP classified = POS_DOC = NEG_DOC Expert reviewed = POS_DOC TP: 4 FN: 3 = NEG_DOC FP: 2 TN: 1 Sensitivity (recall) = TP / TP+FN n = 4 = 57% 4+3

38 HOW WELL DOES NLP DETECT INDICATORS? NLP classified = POS_DOC = NEG_DOC Expert reviewed = POS_DOC TP: 4 FN: 3 = NEG_DOC FP: 2 TN: 1 Positive predictive value (precision) = TP / TP+FP n = 4 = 67% 4+2

39 HOW WELL DOES NLP DETECT INDICATORS? NLP classified = POS_DOC = NEG_DOC Expert reviewed = POS_DOC TP: 4 FN: 3 = NEG_DOC FP: 2 TN: 1 F1-score (harmonic mean of precision and recall) = 2 * (p * r) (p + r) = 2 x (57% x 67%) = 62% (57% + 67%)

40 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC Update rule files

41 Cooking with the easy button & Multi-tasking in the kitchen CONFIDENTIAL

42 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC Easy Clinical Information Extractor Update rule files

43 PREPARE THE PRACTICE DATASET(ALREADY DONE) MIMIC II demo dataset Consist of 4000 deceased ICU patients Select 50 encounters has ICD9 code start with: 480 Viral pneumonia 481 Pneumococcal 482 Other bacterial pneumonia 483 Pneumonia due to other specified organism 484 Pneumonia in infectious diseases classified elsewhere 485 Bronchopneumonia, organism unspecified 486 Pneumonia, organism unspecified And 50 encounters that do not have any ICD9 code above

44 PREPARE THE PRACTICE DATASET(ALREADY DONE) For demonstration purpose, sampled: 70 radiology reports, 20 discharge summaries 10 nursing notes

45 GOAL: Identify any indication of pneumonia Including signs and symptoms Diagnoses Lab tests CT findings Not include treatments (narrow the scope) Conclude whether a document has any indication or not

46 PREPARE THE GOLD STANDARD(ALREADY DONE) Two clinical annotators The 3 rd annotator solves the disagreement Split the dataset to 60 for training, 40 for testing.

47 ANNOTATION SCHEMA IND_PNEUMONIA: for any mention that indicate pneumonia Pos_Doc: for the documents that have any indication of pneumonia Neg_Doc: for the documents that do not have

48 ROADMAP Input corpus POS_DOC NEG_DOC

49 IMPORT TEXT DOCUMENTS POS_DOC NEG_DOC Click on ImportDocuments

50 IMPORT GOLD ANNOTATIONS POS_DOC NEG_DOC Click on ImportAnnotations

51 ROADMAP Input corpus Define rule files POS_DOC NEG_DOC

52 DEFINING RULES Target rules Context rules Feature inference rules Document inference rules Courtesy: Wendy Chapman

53 DEFINING TARGET RULES Most of concepts have already been included, need improve

54 DEFINE MODIFIER AND TARGET RULES FILES Click on Value to update rules for rulefile and crulefile modifiers = rulefile targets = crulefile

55 DEFINE RULEFILE (MODIFIER RULE FILE) Actual are extracted and classified: high fever Pseudos are ignored: Yellow Fever vaccination clinic Modifiers and their value sets; (d) = default value Negation = {affirmed (d), negated} -- Certainty = {certain (d), uncertain} Temporality = {present (d), historical, hypothetical} Experiencer = {patient (d), nonpatient} Rule file (modifier dictionary given) rule string direction trigger type modifier window size resolved backward actual negation 30 possible forward actual uncertain 8 in the past bidirectional actual temporality 8 mom forward actual experiencer 8 vaccination clinic backward pseudo 8

56 DEFINING CONTEXT RULES Most of concepts have already been included, need improve

57 DEFINING FEATURE INFERENCE RULES To exclude the mentions that you don t want (Done)

58 DEFINING DOCUMENT INFERENCE RULES When conclude "Pos_Doc" (Done)

59 ROADMAP Input corpus Define rule files Apply the rules POS_DOC NEG_DOC Easy Clinical Information Extractor

60 RUN EASYCIE ONE CLICK! Select RunEasyCIE

61 REVIEW AND COMPARE RESULTS ONE CLICK! Select ViewOutputinDB

62 RESULT VIEW

63 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC

64 COMPARE OUTPUTS

65 REVIEW & ANALYZE ERRORS

66 DEBUG ERRORS (1) Use a snippet to test the pipeline

67 DEBUG ERRORS (2) A step by step output display for each component

68 DEBUG ERRORS (3) Details view of all clues for final output

69 ROADMAP Input corpus Define rule files Apply the rules Compare outputs POS_DOC NEG_DOC Easy Clinical Information Extractor Update rule files

70 Hands-on exercise CONFIDENTIAL

71 Borrowing a cup of sugar CONFIDENTIAL

72 REUSE OTHERS WORK

73 AUTOMATE THE REST CONFIGURATION

74 ACKNOWLEDGEMENTS Wendy Chapman Barbara Jones Kelly Peterson

75 T h Cedar Breaks National Park a n k y o u Arches National Park Capitol Reef National Park danielle.mowery@utah.edu

76 PLEASE FILL IN THE SURVEY We highly appreciate your feedbacks: Thank you!