Deep learning approaches to medical applications

Transcription

1 Deep learning approaches to medical applications Joseph Paul Cohen Postdoctoral Fellow at Montreal Institute for Learning Algorithms Friend of the Farlow Fellow at Harvard University U.S. National Science Foundation Graduate Fellow What is Deep Learning? Image and Volume a. b. c. 3. Segmentation Counting Interpreting Genomics a. Representation

2 What is Deep Learning? Logistic Regression [200, 45, -4, 80] [1, 1, 0, 1]

3 What is Deep Learning? Logistic Regression

4 What is Deep Learning? Logistic Regressions Image credit Pascal Vincent

5 What is Deep Learning? Image credit Pascal Vincent

6 What is Deep Learning? Logistic Regressions

7

8 Automated pathology

9 Tumor Segmentation Automated detection of metatases in hematoxylin and eosin (H&E) stained whole-slide images of lymph node sections. Reduce the workload of the pathologists and the subjectivity in diagnosis! Image credit: Harvard Medical School, MIT, and EXB Research Determining the probability that an image patch is a tumour can be solved using deep learning models.

10 Image credit: Harvard Medical School, MIT, and EXB Research

11 Image credit: Harvard Medical School, MIT, and EXB Research

12 High accuracy screening and very reproducible! Image credit: Harvard Medical School, MIT, and EXB Research

13 How does it work?

14 Downsample Segmentation with Deep Learning Images by Vincent Dumoulin (UdeM) [Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation 2015] [Honari et al., Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation 2015] Upsample Image by Qikui Zhu

15 Segmentation training in progress Image credit: Lewis Fishgold and Rob Emanuele

16 How can the network learn?

17 Example No Tumor Tumor Image Patch Necrosis Given a patch predict the class of the center pixel Image credits:

18 Softmax No Tumor 3.2 Tumor 5.1 Necrosis exp normalize To predict multiple classes we project to a probability distribution

19 Simplex Tumor 0.13 No Tumor x 0.87 Tumor 0.0 Necrosis No Tumor Necrosis No Tumor Tumor Necrosis Because it is on a simplex; the correction of one term impacts all Image credits:

20 Necrosis Tumor No Tumor Infinite ways to generate the same output. No Tumor A correction of one sends gradients to others We can learn unseen classes through a process of elimination. Necrosis Tumor

21 Softmax and Cross-entropy loss No Tumor 3.2 Tumor 5.1 Necrosis exp normalize loss To predict multiple class we can project the output onto a simplex and compute the loss there.

22

23 More segmentation tasks! CT Pancreas Segmentation MRI Brain Tumor Segmentation

24 More segmentation tasks! CT Pancreas Segmentation MRI Brain Tumor Segmentation

25 More segmentation tasks! Sub-acute ischemic stroke lesion segmentation Stroke Perfusion Estimation Brain tumor segmentation Image Credit: Mohammad Havaei

26 Multi-modal MRI acquisition T1 T1C Sub-acute ischemic stroke lesion segmentation Stroke Perfusion Estimation T2 Flair Brain tumor segmentation Image Credit: Mohammad Havaei

27 Multi-modal MRI acquisition T1 Missing Missing T2 Missing T1C Flair Missing Sub-acute ischemic stroke lesion segmentation Stroke Perfusion Estimation Brain tumor segmentation Image Credit: Mohammad Havaei

28 HeMIS: Hetero-Modal Image Segmentation

29 HeMIS: Hetero-Modal Image Segmentation Missing

30 Results (BRATS) FLAIR T2W T1W T1c HeMIS Truth

31 Results (BRATS) FLAIR T2W T1W T1c HeMIS Truth

32 Results (BRATS) FLAIR T2W T1W T1c HeMIS Truth

33 Results (BRATS) FLAIR T2W T1W T1c HeMIS Truth

34 Brain tumor segmentation (BRATS2013 dataset) T1 T2 GT Edema Necrosis Non-enhanced Enhanced Training data: 220 subjects with high grade and 54 subjects with low grade tumors T1C Flair Dice Similarity [Havaei et al. HeMIS: Hetero-Modal Image Segmentation, MICCAI 2016]

35

36 Interpreting Chest X-Rays Wang, ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases, 2017

37 Convolutional Neural Net Predictions AUCs for each class of a multi-target model 32,717 patients 108,948 X-Rays Text extracted using NLP from doctors notes

38

39

40

41 Cell counting from images Complicated cell structure [Cohen et al. 2017]

42 Cell counting from images Cohen et al., "Count-ception: Counting by Fully Convolutional Redundant Counting," 2017

43 Cell counting from images Cohen et al., "Count-ception: Counting by Fully Convolutional Redundant Counting," 2017

44 [Cohen et al. 2017]

45 [Cohen et al. 2017]

46 Kaggle sea lion challenge (37th/385 place) Implemented by Robin Dinse (Universität Koblenz-Landau)

47

48 Semi-supervised Segmentation with GANs Images with segmentation labels Images without segmentation labels

49 Semi-supervised Segmentation with GANs Zhang et al., "Deep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images," 2017

50

51

52 Genomics - Gene Understanding Costanzo, Michael, et al. "The genetic landscape of a cell." Science (2010)

53 Gene understanding How can we represent different tissue types? [Trofimov et al. ICML WCB 2017]

54 Factorized Embeddings Gene expression is predicted given (Tissue, Gene) pair Genes condition Tissue Prediction Tissues condition Gene Prediction Embedding locations are updated based on function [Trofimov et al. ICML WCB 2017]

55

56 MYH6 "Muscle contraction" [uniprot.org] We can visualize an expression level for every tissue embedding

57 MYH6 What contraction" is this gene "Muscle involved in? [uniprot.org]

58 MYH6 "Muscle contraction" [uniprot.org]

59 KRT (Keratin) Whatisisthe thisprotein gene Keratin involved in? that protects epithelial cells from damage or stress

60 KRT (Keratin) Keratin is the protein that protects epithelial cells from damage or stress

61

62 Thanks! Conferences ShortScience.org ShortScience.org