Computational Pathology

Transcription

1 Computational Pathology In the Midst of a Revolution: How Computational Pathology is Transforming Clinical Practice and Biomedical Research Thomas J. Fuchs Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill Cornell Graduate School of Medical Sciences Director, Computational Pathology and Medical Machine Learning Lab Department of Medical Physics Department of Pathology fuchst@mskcc.org thomasfuchslab.org Disclaimer: co-founder of Paige.AI

2 Fuchs MSKCC + Weill Cornell Thomas Peter Fem Andrew Hassan Gabe Arjun Amanda

3 Background Graz, Austria Arnold Schwarzenegger 38 th Governor of California

4 Bladder Computational Pathology Kidney

5

6

7

8

9

10

11

12

13 pixel

14

15

16

17

18 1,000,000 new glass slides per MSKCC

19 1,000,000 new glass slides per MSKCC

20 1,000,000 new glass slides per MSKCC

21

22 Comp. Pathology AI Digital Pathology (Scanning, QC, P-PACS, Image Processing, ) Pathology Informatics (EHR, LIS, Barcoding, frfid,...) Wet Laboratory (Physical Slide Production, Cutting, Staining, ) Simplified Pathology Department Stack

23 Definition Computational Pathology investigates a complete probabilistic treatment of scientific and clinical workflows in general pathology, i.e. it combines experimental design, statistical pattern recognition and survival analysis within an unified framework to answer scientific and clinical questions in pathology. [Fuchs 2011]

24 Archimedes lever, 1824 Mechanic Magazine

25 25 Ubiquity of Machine Learning Self-driving Cars Knowledge Systems Computational Pathology Surveillance Cyber Security Space Exploration

26 Computer Vision Tasks in Pathology Nuclei Detection and Classification Sub-cellular level Segmentation Structure Estimation Morphology

27 Dataset Sizes: Computer Vision vs. Computational Pathology 1 Whole Slide = 100,000 x 60,000 = 6 billion pixels CIFAR-10 (32*32)*60K= million pixels All 60,000 CIFAR images fit into this box

28 Dataset Sizes: Computer Vision vs. Computational Pathology n=1 n=474 All of ImageNet 482 x 415 * 14,197,122 = 2.8 trillion pixels 474 Whole Slides 100,000 x 60,000 *474 = 2.8 trillion pixels

29 Ground Truth for Statistical Learning Labeled samples are needed for training and validation. What is the Ground Truth?

30 Past Present Future [BD2K Proposal 2014] Expert & Crowd Sourcing

31 Expert Staining Estimation

32 Intra Pathologist Evaluation 50 nuclei were repeated flipped and rotated to test the intra pathologist variability. Original Flipped & Rotated 53/250 mismatches Baseline: Intra-Pathologists classification uncertainty of ~20%

33 easy humans hard Why is Comp. Path. Challenging? expert with decades of training chess genomics Computational Pathology amenable to crowdsourcing repetitive or boring a child s play for humans easy filing manufacturing machines vision robotics hard structured machine readable data unstructured/visual data

34 Computational Pathology

35 Nucleus Based Analysis DAGM 2008

36 Applications of the Framework Spatial Processes for Hippocampal Sclerosis Original Image Detected Objects Process Intensity Pancreatic Islet Segmentation for T2 Diabetes Counting of Mouse Liver Hepatocytes Detection in IHC Stained Cell Cultures

37 Cell Nuclei Detection

38 Survival Analysis p = p = low risk high risk low risk high risk

39 FGI Grant Quantifying and Correlating Tissue Pathology with the MK-IMPACT Genotype Classic UC E-cadherin IHC Plasmacytoid Ca Plasmacytoid Ca Classic UC One tumor with two morphologies with different mutational profile (but also share 2 mutations indicating same origin).

40 A Joint Effort for Personalized Medicine Pathology Radiology Genomics Computational Pathology MSKCC Combining quantitative analyses from pathology, radiology and genomics facilitates true personalized medicine. MSKCC

41 Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 20 GLASS challenge 200 slides Camelyon Challenge 400 Slides Equivalent to

42 State-of-the-art in March slides for training 240 slides for testing binary classification

43 State-of-the-art vs. Reality in clinical practice State-of-the art datasets in pathology: tiny (~400 slides) very well curated Like training your autonomous car only on an empty parking lot. It has never seen rain, snow or a dirt road. Clinical reality: messy diverse surprising How can we ever hope to train clinical-grade models?

44 1/1/15 2/1/15 3/1/15 4/1/15 5/1/15 6/1/15 7/1/15 8/1/15 9/1/15 10/1/15 11/1/15 12/1/15 1/1/16 2/1/16 3/1/16 4/1/16 5/1/16 6/1/16 7/1/16 8/1/16 9/1/16 10/1/16 11/1/16 12/1/16 1/1/17 2/1/17 3/1/17 4/1/17 Number of Digitized Whole Slides Clinical Slide Memorial Sloan Kettering

45 1/1/15 3/1/15 5/1/15 7/1/15 9/1/15 11/1/15 1/1/16 3/1/16 5/1/16 7/1/16 9/1/16 11/1/16 1/1/17 3/1/17 5/1/17 7/1/17 9/1/17 11/1/17 1/1/18 3/1/18 5/1/18 7/1/18 9/1/18 11/1/18 Number of Digitized Whole Slides Clinical Slide Memorial Sloan Kettering ~ 1 petabyte of compressed image data Projection with current ramp-up to 40,000 slides / month

46 QC: a machine learning solution the Blur detector [Campanella et al. 2017] sharp blurred thumbnail blur mask sharp blurred blurred

47 Aperio Scanner Hamamatsu Scanner... Philips Scanner cbio Portal Consultation Portal Aperio Viewer Hamamatsu Viewer... Philips Viewer cbio Portal Viewer Consultation Viewer ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL...

48 Aperio Scanner Hamamatsu Scanner... Philips Scanner cbio Portal Consultation Portal slides.mskcc.org

49

50

51 High Per for mance Computing for Pathology Awarded Center of Excellence for GPU Computing from for our work in Pathology and cbio. GPU MSKCC s HPC Cluster 120 GPUs in total Pascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology

52 Memorial Sloan Kettering Cancer Center Deep Learning for Decision Support in Skin Cancer

53 Basal Cell Carcinoma Prediction Segmentation and Diagnosis Prediction

54 CNN Whole-slide tumor prediction

55 Convergence Curves: BCC Classification

56 Classification Error 97% Accuracy in Predicting BCC Logistic Regression Random Forest AlexNet AlexNet pretrained ResNet 18 ResNet 18 pretrained ResNet 34 pretrained

57 Generative Adversarial Networks for Large-Scale Semantic Image Retrieval

58

59 Dreaming of Cancer: A Nightmare of Cancer Samples drawn from our Generative Adversarial Network (GAN) Prostate Cancer Model Natural Images (CIFAR-10)

60 Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 20 GLASS challenge 200 slides Camelyon Challenge 400 Slides

61

62 athology rtificial ntelligence uidance ngine

63 Computational Pathology Datasets Paige.AI Prostate Biopsy Complete Diagnosis 15,000 Slides First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 20 GLASS challenge 200 slides Camelyon Challenge 400 Slides Google [Liu et al. 2017] 509 Slides

64 Changing Clinical Practice MSK-P15K Dataset We generated an unrivaled prostate biopsy dataset of 15,000 whole slides of needle biopsy cores with clinical annotation. Deep Learning We are training convolutional neural networks and generative models at scale on our HPC cluster Medical Expertise MSK is the nations leading center for prostate cancer consultation with world-renown domain experts, who annotate the data and interactively train our AI. Goal: The first ever clinical-grade Computational Pathology model Whole slides of Prostate Needle Biopsies We developed an unified slide viewer for sample annotation slides.mskcc.org

65 Fuchs MSKCC / Weill Cornell Andrew Schaumberg Thomas Fuchs Peter Schüffler Arjun Raj Rajanna Gabriele Campanella Amanda Beras Hassan Muhammad

66 MSKCC Collaborators David Klimstra Meera Hameed Victor Reuter Malcolm Pike Klaus Busam Joe Sirintrapun Hikmat Al-Ahmadie Edi Brogi Jinru Shia Oscar Lin Joseph O. Deasy Jung Hun Oh Harini Veeraraghavan Adity Apte John L. Humm

67 Thank you for your attention! Questions welcomed! Thomas J. Fuchs thomasfuchslab.org Open ML and CS positions in Manhattan