Memory-Augmented Active Deep Learning for Identifying Relations Between Distant Medical Concepts in Electroencephalography Reports Ramon Maldonado, BS, Travis Goodwin, PhD Sanda M. Harabagiu, PhD The University of Texas at Dallas Human Language Technology Research Institute http://www.hlt.utdallas.edu/~{ramon, travis, sanda}
Conflicts There are no conflicts of interest
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
Introduction Clinical electroencephalography (EEG) is the most important investigation in the diagnosis and management of epilepsies. As more clinical EEG becomes available, the interpretation of EEG signals can be improved by providing neurologists with results of search for patients that exhibit similar EEG characteristics. MERCuRY (Multi-modal ElectroencephalogRam patient Cohort discovery) - Goodwin & Harabagiu (2016) 1 for cohort identification
Introduction QUERY: Patients with shifting arrhythmic delta suspected of underlying cerebrovascular disease EXAMPLE RECORD: CLINICAL HISTORY: 55 y/o man admitted for change in mental status, with a past medical history of GI bleed, anemia, encephalopathy, and others. MEDICATIONS: Pantoprazole, Folic Acid, Carvedilol DESCRIPTION OF THE RECORD: The background EEG is characterized by slowing and disorganization. There is prominent shifting arrhythmic delta activity more prominent in the left mid to anterior temporal region. Photic stimulation generates scant driving. IMPRESSION: Abnormal EEG due to: 1. Marked background slowing and disorganization; 2. Some arrhythmic delta activity CLINICAL CORRELATION: These findings are supportive of a bihemispheric disturbance of cerebral function. These are nonspecific findings which can be seen in a toxic and metabolic encephalopathy and/or underlying cerebrovascular disease
Introduction Current state-of-the-art relation identification methods require arguments to be in the same sentence Our method operates on pairs of concepts from the same EEG reports not constrained to appear in the same sentence/section Accurate relations enable higher quality Medical Knowledge Embeddings (MKE) (Maldonado et al., 2017) Capture knowledge from a large corpus of EEG reports using deep learning Represent concepts and relations as low-dimensional vectors Vector representation helps resolve semantic heterogeneity from differences in terminology
Introduction EEG-RelNet Extension of RelNet from Bansal, Neelakantan, & McCallum (2017) Processes entire EEG report to model cross-sentence relations Memory-augmented neural network with abstract memory cells Concept memory cells Relation memory cells Memories implicitly model current knowledge about medical concepts and their relations in the EEG report
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
The Data EEG reports from Temple University Hospital (TUH) 25,000 reports from 15,000 patients collected over 12 years Sections: 1. Clinical History: Lists past and current medical problems, symptoms, signs, and treatments as well as significant medical events. 2. Medications 3. Introduction: depiction of the techniques used for the EEG 4. Description: a complete and objective description of the EEG, noting all observed activity, patterns, and events 5. Impression: states whether the EEG test is normal or abnormal and, if abnormal, lists the abnormalities in order of importance 6. Clinical Correlation: explains what the EEG findings mean in terms of clinical interpretation
The Data - Preprocessing Multi-Task Active Deep Learning system for concept detection & attribute classification (Maldonado et. al 2017) EEG activities, EEG events, medical problems, treatments Medical concept attributes Modality Polarity Medical Concept Type 16 EEG activity attributes (morphology, dispersal, location, etc.) 365,218 concepts with 3,062,846 attributes Normalization UMLS for medical problems and treatments 10 EEG event types EEG activities are normalized by morphology attribute
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
Memory-Augmented Active Deep Learning The purpose of the Memory-Augmented Active Deep Learning (MAADL) system is to automatically identify relations between pairs of medical concepts in EEG reports, regardless of how distant they are. The MAADL Paradigm consists of 5 steps: STEP 1: The development of an annotation schema STEP 2: Annotation of initial training data STEP 3: Design of deep learning method capable of learning from the data STEP 4: Development of sampling methods for MAADL STEP 5: Usage of the Active Learning system involving: STEP 5.a: Accepting/Editing annotations of sampled examples STEP 5.b: Re-training the deep learning model
MAADL Annotation Schema Three relation types: EVIDENCES, EVOKES, TREATMENT-FOR 1 EVIDENCES 2 EVOKES EEG event EEG activity medical problem treatment medical problem EEG event EEG activity medical problem treatment EEG activity 3 TREATMENT FOR treatment medical problem
Memory-Augmented Active Deep Learning EEG Reports Deep Learning-Based Identification of medical concepts + their attributes Automatically recognized Relations between medical concepts Active Learning Loop STEP 4 EEG Report Annotation SAMPLING STEP 1 STEP 2 Initial Training Data Manual Annotation of Relations between Medical Concepts. Relation types: EVIDENCES EVOKES TREATMENT-FOR EEG Reports with Seed Annotations of Relations between Medical Concepts STEP 3 Deep Learning-Based Recognition of Relations EEG-RelNet Relational Memories STEP 5 Validation/ Editing of Sampled Annotations of Relations in EEG Reports Re-Training Data
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
EEG-RelNet EEG-RelNet Operates on the full text of an EEG report Considers each pair of medical concepts identified in the report Detects EVIDENCES, EVOKES, and TREATMENT-FOR relations between any pair of concepts 3 modules Input Encoding Module encodes the report into concept- and sentence-level embedding vectors Dynamic Relational Memory maintains and updates 2 sets of hidden states (memories) to accumulate information about each concept and potential relation Output Module feeds the updated memories to prediction layer
EEG-RelNet Input Encoding Module Learn an embedding c i R N for each medical concept c i in the report c i = W C c, a 1 c i,, a A c i c is the normalized concept name embedding a 1 c i,, a A c i are the attribute embeddings for concept c i Learn an embedding s i R N for each sentence s i by combining word embeddings e j for each word w j in sentence s i s i = σ j m f j e j f 1, f m is a learned positional mask e j = c i if word w j corresponds to concept c i
EEG-RelNet Dynamic Relational Memory (DRM) Module To account for long-distance relations, we use the DRM Module to keep track of the interactions between the medical concepts in each report The DRM for a report with d concepts consists of: d Concept Memories which accumulate information about each concept d(d 1) Relation Memories which accumulate information about the potential relations between each pair of concepts
EEG-RelNet c Concept Memory Cell Relation Memory Cell c Concept Memory Cell c Concept Memory Cell c c Concept Memory Cell Concept Memory Cell Relation Memory Cell Relation Memory Cell Relation Memory Cell Relation Memory Cell c c Concept Memory Cell Concept Memory Cell Relational Memory c c Concept Memory Cell Concept Memory Cell Relational Memory s s s
EEG-RelNet Concept Memory h i is updated as follows: i c = s k, h i + c i i + i h i = φ W u h i + W v c i + W s s k h i h i + i c h i c i i s Relation Memory r i is updated as follows: c c ij = i j s k, r ij ij + i r ij = φ W A r ij + W B s k r ij r ij + ij r ij j ij ij s
EEG-RelNet Output Module Identifies the relation (if any) between any pair of medical concepts from the report using the updated DRM after processing each sentence in the report To determine the relation between concepts c i and c j 1. We pass the concept memories h i and h j along with the relation memory r ij to a fully connected PReLU layer: q ij = φ W q h i, h j, r ij ; 2. Then softmax: R ij = softmax φ W z q ij. 3. The predicted relation type is R ij = argmax t R t ij
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
Experimental Results To measure the impact of the EEG-RelNet architecture, we compare against: 1. EEG-RelNet_NRM: a configuration of EEG-RelNet with no relation memories (i.e. q ij = φ W q h i, h j ) 2. EEG-RelNet_NA: a configuration of EEG-RelNet that ignores medical concept attributes (i.e. c i = c) 3. A Heuristic baseline: a high-recall, rule-based method from our previous work using concept type and section information to relate concepts deterministically We use 5-fold cross validation on the full set of 140 manually annotated EEG reports selected via active learning containing 1513 relations between 3691 medical concepts
Experimental Results The performance of the MAADL system for detecting relations between medical concepts in EEG reports
Experimental Results Active Learning Learning Curves shown for the first 100 EEG reports annotated and evaluated with F 1 measure.
Experimental Results - Discussion Best performance for EVOKES relations 0.8371 vs 0.7116 (TREATMENT-FOR), 0.6939 (EVIDENCES) EVOKES always involves an EEG Activity EEG activities have a richer representation Context is often clearer for EVOKES relations Could use existing ontologies to inform medical problem and treatment concepts (e.g. TREATMENT-FOR(Lamictal, seizure)) TREATMENT-FOR relations have high recall and low precision Most common treatments (medications) have uninformative context Improves markedly as AL progresses
Outline 1. Introduction 2. The data 3. Memory-Augmented Active Deep Learning 4. EEG-RelNet for long distance relation detection 5. Experimental Results 6. Conclusion
Conclusion In this work, we presented a novel active deep learning framework for identifying relations between medical concepts in EEG reports. EEG-RelNet: a novel deep neural architecture capable of inferring relations between medical concepts not mentioned in the same sentence or section To our knowledge, this is the first method capable of detecting relations between distant medical concepts in medical narratives Updates a set of memories that accumulate information about each concept and potential relation in the EEG report Experimental results show promise Future work Incorporate outside knowledge to improve representation of medical problems/treatments Use MAADL system to improve Medical Knowledge Embeddings for epilepsy
Acknowledgements Research reported in this publication was supported by the National Human Genome Research Institute of the National Institutes of Health under award number 1U01HG008468. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Questions