MICCAI 2018 VISDMML Tutorial Image Analysis Meets Deep Learning and Visualization in Cardiology and Cardiac Surgery

Transcription

1 MICCAI 2018 VISDMML Tutorial Image Analysis Meets Deep Learning and Visualization in Cardiology and Cardiac Surgery Dr. Sandy Engelhardt Faculty of Computer Science, Mannheim University of Applied Sciences, Germany Granada, September 16th, /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 1

2 We demonstrate performance in making a referral recommendation that reaches or exceeds that of experts on a range of sight-threatening retinal diseases after training on only 14,884 scans. We show that deep learning can extract new knowledge from retinal fundus images. Using deeplearning models trained on data from 284,335 patients and validated on two independent datasets of 12,026 and 999 patients, we predicted cardiovascular risk factors not previously thought to be present or quantifiable in retinal images. 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 2

3 Deep Learning Deep learning has absolutely dominated computer vision over the last few years, achieving top scores on many tasks and their related competitions. The most popular and well known of these computer vision competitions is ImageNet (1000 object categories/classes) By 2012, ImageNet had nearly 1.3 million training images. The main challenge with such a large scale image classification task is the diversity of the images. Over the past few years, deep learning techniques have enabled rapid progress in this competition, even surpassing human performance. 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 3

4 ILSVRC challenge ImageNet Large Scale Visual Recognition Challenge 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 4

5 Popular tasks to be solved with DL 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 5

6 Challenges in Medical Image Analysis - Small data sets (few training data) - High anatomical variance - Different modalities/scanners - Large, volumetric images Scarcity of labels - Noisy data - Noisy annotations / interpretability - Unlabelled data Brain MRI with 3 mm lesions 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 6

7 Biomedical Grand Challenges since MICCAI 2007; 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 7

8 Popular Segmentation Networks 2D U-Net (2015) Ronneberger et al.; 3D U-Net (2016) Cicek et al. skipping connections Encoder Context aggregation Decoder Localization [Fig. arxiv: ] 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 8

9 2D/3D U-Net 3rd place BraTS Challenge (Brain Tumor Segmentation Challenge) 1st place STACOM ACDC Challenge (Automated Cardiac Diagnosis Challenge) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 9

10 Automated Cardiac Diagnosis Challenge (MICCAI 2017) Left Ventricle Multistructure Segmentation Right Ventricle Myocardium Abnormal Right Ventricle Classification (5 groups) Hypertrophic Cardiomyopathy Dilated Cardiomyopathy 150 patient data with a high variance (resolution, pathologies, image sequences, MR devices, etc.) Myocardial Infarction Normal Subjects 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 10

11 The data short-axis cine-mri timesteps high in-plane resolution (~0.5 mm), slice thickness of 5-10 mm slice misalignments 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 11

12 2D/3D U-Net Resampling to 1.25 x 1.25 x 10 mm Due to low z-resolution of the input, pooling and upscaling operations are carried out only in the x-y-plane. Context in the z- dimension is solely aggregated through the 3D convolutions. 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 12

13 Winning Deep Learning Approach Segmentation predictions from a 2D and a 3D model are averaged and used to extract instant and dynamic volume features, which are fed into an ensemble of classifiers for disease prediction. Isensee et al., Automatic Cardiac Disease Assessment on cine-mri via Time-Series Segmentation and Domain Specific Features, MICCAI STACOM 2017 (Winner Segmentation Challenge, Runner-Up Classification Challenge) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 13

14 Isensee et al., Automatic Cardiac Disease Assessment on cine-mri via Time-Series Segmentation and Domain Specific Features, MICCAI STACOM 2017 (Winner Segmentation Challenge, Runner-Up Classification Challenge) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 14

15 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 15

16 Can we do better? Observation I: Most deep learning is done offline. Observation II: A physician is becoming better the more patients he/she has seen. Same is true for neural networks. AI Cardiologist artificial intelligence software infrastructure 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 16

17 Artificial Intelligence inside the hospital segmentation classification AI Cardiologist correction re-train networks 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 17

18 Artificial Intelligence inside the hospital better performance for the next case AI Cardiologist captures and reproduces local standards re-train networks 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 18

19 Cardiovascular Imaging Ultrasound high temporal, low spatial resolution artefacts many variants MRI highly anisotropic voxels low temporal resolution many variants CT radiation contrast agents ability to image plaque Consider breathing and arrhythmia in cardiac imaging Quantification and visualization of motion and flow Realtime imaging Population-based analysis Enhanced analysis for congenital heart disease 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 19

20 Cardiovascular Imaging Image Processing Visualization Interpretation 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 20

21 The potential of Virtual Reality Images courtesy of Medscan Barangaroo, Sydney, Australia Volumetric rendering, Color coding 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 21

22 Mitral Valve Segmentations manual automatic quantifications coaptation zone area. Engelhardt, et al., Towards Automatic Assessment of the Mitral Valve Coaptation Zone from 4D Ultrasound., FIMH 2015, LNCS /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 22

23 Mitral valve reconstruction (ca. 250 steps) anaesthesia patient positioning native valve diagnosis and planning with ultrasound port (open or minimally invasive) cardiopulmonary bypass stitching of annuloplasty sutures annuloplasty ring sutures implantation of neo-chordae surgical correction of annulus, leaflets, papillary muscles, chordae implantation of ring prosthesis assessment on ultrasound 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 23

24 Training Mitral Valve Repair with Surgical Simulator shape material stitching and cutting properties no realistic appearance heterogeneous texture specularities blood Engelhardt, S., Sauerzapf, S., et al. (2018) Elastic Mitral Valve Silicone Replica Offer Advanced Surgical Training. In: BVM 2018; Invited to Special Issue IJCARS; conhit Nachwuchspreis 2018 Engelhardt, et al., Towards Automatic Assessment of the Mitral Valve Coaptation Zone from 4D Ultrasound., FIMH 2015, LNCS /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 24

25 Hyperrealism A novel form of Augmented Reality hyperrealistic image Spotlight talk tomorrow 5:45 p.m. Poster #M-85 temp Cycle GAN simulator with silicone phantom Engelhardt S., et al., Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation, MICCAI 2018 (straight accepted; spotlight talk) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 25

26 Reality-virtuality continuum Hyperrealism A novel form of AR Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Virtual Environment P. Milgram, et al. "Augmented Reality: A class of displays on the reality-virtuality continuum". Proceedings of Telemanipulator and Telepresence Technologies, 1994, pp /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 26

27 Unpaired CycleGAN first domain second domain first domain discriminator D Y real fake discriminator D X real fake generator generator x t G y t F x t 1 J. Y. Zhu, et al., Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE ICCV, 2017, pp /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 27

28 J. Y. Zhu, et al., Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE ICCV, 2017, pp /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 28

29 A pre-trained deep neural network making predictions on live camera input, trying to make sense of what it sees, in context of what it s seen before. Source: 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 29

30 Unpaired CycleGAN Temporal incoherent J. Y. Zhu, et al., Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE ICCV, 2017, pp /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 30

31 Unpaired CycleGAN domain transfer domain transfer Original CycleGAN (Zhu et al., 2017) J. Y. Zhu, et al., Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE ICCV, 2017, pp /09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 31

32 tempcyclegan x t 2 y t 2 x t 2 x t 1 G y t 1 D Y 1 1 F x t 1 D X x t G y t 1 D TY F x t 1 D TX y t D Y 1 x t D X Engelhardt S., et al., Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation, MICCAI 2018 (straight accepted; spotlight talk) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 32

33 tempcyclegan tempcyclegan (Engelhardt et al., 2018) Engelhardt S., et al., Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation, MICCAI 2018 (straight accepted; spotlight talk) 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 33

34 Real or Fake? 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 34

35 Thank you for your 25/09/ Hochschule Mannheim University of Applied Sciences Dr. Sandy Engelhardt Slide 35