Midterm project (Part 2) Due: Monday, November 5, 2018

Transcription

1 University of Pittsburgh CS3750 Advanced Topics in Machine Learning Professor Milos Hauskrecht Midterm project (Part 2) Due: Monday, November 5, 2018 The objective of the midterm project is to gain experience working with machine learning methods covered in the first half of the course. The project will be conducted on data derived from electronic health records (EHRs). You will implement (with the help of existing libraries) methods for analyzing EHRs using (1) SVD, (2) LDA, and (3) CBOW models. You will analyze the data with these models in order to demonstrate their benefits in understanding the different patient cases, their similarities and differences, and their value in supporting various prediction tasks. Teams The project is a group project. There will be 5 teams of four students each. One group has five members. The groups are posted on the course web site. The groups should work on the problems and solutions independently. The joint development of the solutions is strictly prohibited!!! Problem The data we analyze in this project consist of sequences of events logged in time that correspond to various events (or activities) related to patient care. These include lab tests, medications given and procedures done on patients. There are multiple sequences, each corresponding to a different patient. While the LDA, SVD or CBOW models were originally designed for document and text analysis one can view different data objects (in our case patient cases) as documents and the basic entities that form these objects as terms or words. Briefly, a patient case can be thought of as a document, and events related to patient care are words. One of the hypothesis to be studied in this project is that the sequences of observed events are related to the problems the patients suffer from (e.g. diseases, complications) in a similar way the documents may be represented by the different topics. Another hypothesis is that co-occurring events are semantically related and that this relation can be expressed to define meaningful semantic distances among events, much like word distances induced by the word2vec models when applied to texts. Data The data for the project are stored in the departmental database server: cs-mysql-01.cs.pitt.edu The database name is CS3750_ DB. The data are derived from MIMIC-III dataset of ICU patient cases that is distributed by physionet.org. The data in CS3750_ DB database reside in five different tables. The first table (ADMISSIONS) enumerates patient instances (or patient cases). There are 21,571 patient instances in the database. Each instance in the ADMISSION table is associated with four columns with binary entries that reflect some aspects of the case, indicating whether the patient is (1) a trauma patient, (2) was admitted due to

2 or experienced a heart-related condition, (3) developed sepsis, and (4) is a transplant patient. We use these entries for the external validation of SVD and LDA approaches. The second table (EVENTS) lists events related to labs and medications observed for individual patients and times of these events. Please note, that times of patient admissions were shifted so that the first day of every admission is January 1, All events are coded. The third table (EVENT_LIST) is the dictionary that describes the meaning of event codes. The fourth table (PROCEDURES) gives a list of special events - procedures for every patient case (these cover, for example, surgeries). These events are not logged in time. They are provided on the patient level only. Also note that one patient can have more procedures. The fifth table (PROCEDURE_LIST) defines the meaning of the procedures codes. Data access and rules for working with the data The data you will work with were prepared for the project purposes and should not be publicly distributed or copied in any way and should reside only on the servers. All of you will need to sign the agreement form outlining the rules for working with the data including a no distribution clause. Tasks Our objective is to study lower dimensional representations of patient cases that can define similarities among patients and events. We start our investigations with the Singular value decomposition (SVD) methods. Task 1. SVD analysis SVD starts from the term-document matrix formed by the bag of words (vector-based) representation of events observed for the different patients and their counts. Alternatively, it can be formed by their tf-idf scores. The SVD finds a decomposition of the term-document matrix that is restricted by the internal dimension k. The internal dimension defines the latent space for the documents/words. The matrices forming the decomposition represent patients (documents) and events (words) in the k-dimensional latent space. The similarities among patients (documents) can be calculated by transforming the documents using the right eigenvectors defining the SVD decomposition (multiplied by the matrix of singular values) and the cosine metrics. Similarly, the similarity among events (words) can be calculated using the left eigenvectors defining the SVD decomposition and the cosine metric. Part 1.a. Get target patient instances from the ADMISSION table assigned to your group (See Appendix 1) and target events from the EVENT_LIST and PROCEDURELIST tables assigned to your group (See Appendix 2). Part 1.b. Run SVD with internal dimensionality k=20 such that instances are defined by events in EVENTS and PROCEDURES tables. For every target instance, identify five instances that are most similar to it based on the SVD. Analyze the differences between the target and best matching instances in terms of: (1) the events and procedures associated with these instances, (2) four attributes/labels associated with the instances in the ADMISSION table. Do most similar instances to the targets make sense? Please explain and discuss your conclusions. Part 1.c. For every target event Identify 10 events from (EVENT_LIST or PROCEDURE_LIST) most similar to it. Research the relations between these events by searching the web. Are there any associations

3 between the events/procedures you think make or do not make sense? Please, analyze the problem and explain your conclusions. Part 1.d. In this part, our goal is to validate the usefulness of the lower dimensional representation of patient cases using labels assigned to the instances in the ADMISSION table. Propose and implement a method for validation of the lower dimensional representation x of patient instances x as defined by the SVD decomposition and their ability to predict the labels. Explain and justify your approach. Analyze the results using the proposed approach and compare it to a simple method that assumes the labels are assigned to instances randomly and without considering any instance-related information. Part 1.e. Research techniques to determine the dimensionality k of the latent space for the SVD. Report your findings. Repeat the analysis in Parts 1.b- 1.d by attempting to find a better choice of k. Compare the results of analysis for the 20-dimensional and the new optimized k-dimensional space. Discuss the results and write your conclusions. Part 1.f. In all previous analyses we used the data that consisted of events recorded in both EVENTS and PROCEDURE tables. Consider restricting the analysis only to the events recorded in the EVENTS table. Repeat the SVD analysis in Parts 1.b-1.e., but omit procedure events. Determine whether the events in the PROCEDURE table are important for defining the latent space and improve the results, that is, compare the results obtained with and without procedures in the representation of the patient case. Task 2. LDA analysis An alternative method to define and analyze a low-dimensional representation of patient cases (documents) is the Latent Dirichlet Allocation (LDA). Briefly, LDA defines a generative probabilistic model of patient cases. The lower dimensional representation of instances is represented using the distribution over topics. The similarities among documents are determined in terms of distance measures for two distributions. Two common distance choices are Jensen-Shannon (a symmetric version of the KL-divergence) and Hellinger distances. Part 2.a. Consider the same set of target patient instances from the ADMISSION table assigned to your group in Appendix 1 as used in Part 1.a, and the same set of target events from the EVENT_LIST and PROCEDURE LIST tables assigned to you in Appendix 2. Part 2.b. Run the LDA with internal dimensionality k=20 where instances are defined by events in EVENTS and PROCEDURES tables. Similarly, to Part 1.b. for every target instance find five instances that are most similar (closest) to it. Analyze the differences between the target and the best matching instances in terms of: (1) the events and procedures associated with these instances, (2) four attributes/labels associated with the instances in the ADMISSION table. Do the closest instances to the targets make sense? Compare the closest instances picked by the LDA to the best instances based on the SVD. Analyze and report the differences. Is one of them better than the other one? Discuss the results. Part 2.c. One of the components of the LDA model is the conditional probability distribution of events (words) given the topic. Report 10 most important events (words) defining individual topics based on

4 the LDA model trained on the patient data. By searching the web try to assess if the events most important for each topic are related and whether their combination makes sense. Discuss the results. Part 2.d. Explain, how would you define the similarities/distances in between the two words in the LDA model. For every target event apply the approach to identify 10 events most similar to it, similarly to Part 1.c. for the SVD. Research the relations between these events by searching the web resources. Compare the results to results in Part 1.c and discuss the differences. Part 2.e. Validate the usefulness of the lower dimensional representation of patient cases based on LDA using labels assigned to the instances in the ADMISSION table. Replicate the approach for external validation from Part 1.d. Compare the results obtained for SVD and LDA models. Discuss the results. Part 2.f. Select/pick new dimensionality k (> 20) for the topic space and repeat the analysis in Parts 2.b- 2.e. Compare the results. Report your findings. Are any topics (CPTs for words given the topic) comparable to topics based on the 20-dimensional topic model? Part 2.g Restrict the LDA analysis by considering only the events recorded in the EVENTS table. Repeat the LDA analysis with 20-dimensional topic space in Parts 2.b-2.e. but omit procedures. Analyze and compare the differences in results for Parts 2.b-2.e. Use these analyses to decide if inclusion of procedures in the representation matters. Also compare and discuss the corresponding results you have obtained for the SVD. Task 3. CBOW analysis SVD and LDA analyses work with complete patient cases (documents) to derive similarities between the patients, as well as, similarities between events. CBOW Is a method that aims to extract similarities in between words (events) only. It does this by considering local relations in between events, that is, events that co-occur close in time. As a result, the latent space is expected to capture mostly local relations among events. Part 3.a. Consider the first four target events assigned to your group in Appendix 2. Part 3.b. Run the CBOW model on the event data in the EVENTS table. (Ignore events in the PROCEDURES table). Consider the latent spaces of dimension 20 and 50. Propose a solution of how to define a local window. Part 3.c. For every target event Identify 10 events most similar to it based on the low dimensional embedding defined by the CBOW model. Research the relations between these events by searching the web. Are there any associations you think make or do not make sense? Are these different from the similarities derived in Part 1.f and Part 2.g.? Discuss the differences. Which approach do you think makes most sense for extracting meaningful relations among events. Task 4. Extensions challenge Think about the different ways of extending SVD, LDA or CBOW analyses of data in Electronic Health Records to slightly different or new problems. Pick one problem you thing is the most interesting. Clearly explain the problem and outline the idea of your solution/extension. Justify why the problem and the solution could be interesting for analysis of EHRs. Implementation and preliminary experiments supporting your idea are not necessary but if given, they would earn you extra credit.

5 Methods and libraries The project expects you to take full advantage of existing libraries implementing various machine learning methods. The ML researchers typically resort to python and python libraries. There are different libraries available to support SVD, LDA and CBOW methodologies. One library that covers all three methods we plan to experiment with is gensim library -- so looking at it, is definitely a good start. Elements machines and python. Five elements machines: Arsenic, Germanium, Hydrogen, Oxygen, and Selenium work with and have python3.5. installed. You can run it using either virtualenv or conda. If you do not use them, run python3.5 by including the version number. Timeline Monday, November 6, 2018 (noon): Project reports and programs are due Tuesday, November 7, 2018 (11:00am): Project presentations (15 minutes per group) Appendices Appendix 1. Target patient instances to be considered by the groups when analyzing the similarity between patients. Group 1: ADM_ID: 172 ADM_ID: 752 ADM_ID: 1769 ADM_ID: 2446 ADM_ID: 3857 Group 2: ADM_ID: 407 ADM_ID: 802 ADM_ID: 1868 ADM_ID: 2589 ADM_ID: 4320 Group 3: ADM_ID: 503 ADM_ID: 896 ADM_ID: 1983

6 ADM_ID: 3092 ADM_ID: 3966 Group 4: ADM_ID: 657 ADM_ID: 993 ADM_ID: 2157 ADM_ID: 3259 ADM_ID: 4074 Group 5: ADM_ID: 731 ADM_ID: 1619 ADM_ID: 2266 ADM_ID: 3775 ADM_ID: 4247 Appendix 2. Target events to be considered by the different groups when analyzing the similarity between events. Group 1: 69 Antibiotics: Vancomycin 375 Chemistry: Blood: Troponin I 638 Hematology: Blood: Platelet Count 37 Medications: Nitroprusside 1035 Operations On Valves And Septa Of Heart Group 2: 26 Medications: Furosemide 31 Medications: Norepinephrine 282 Chemistry: Blood: CK-MB Index 629 Hematology: Blood: Neutrophils 1036 Operations On Vessels Of Heart Group 3: 24 Medications: Fentanyl 10 Medications: Amiodarone 617 Hematology: Blood: Lymphocytes 278 Chemistry: Blood: Cholesterol, HDL 1055 Operations On Kidney

7 Group 4: 54 Medications: Heparin Sodium 627 Hematology: Blood: Monocytes 610 Hematology: Blood: INR(PT) 55 Medications: Labetalol 1050 Operations On Liver Group 5: 42 Medications: Vasopressin 648 Hematology: Blood: PTT 621 Hematology: Blood: MCH 286 Chemistry: Blood: Creatinine 1045 Incision, Excision, And Anastomosis Of Intestine