Imaging Collaboration: From Pen and Ink to Artificial Intelligence June 2, 2018

Transcription

1 Imaging Collaboration: From Pen and Ink to Artificial Intelligence June 2, 2018 Curtis P. Langlotz, MD, PhD Professor of Radiology and Biomedical Informatics Director, Center for Artificial Intelligence for Medicine & Imaging Associate Chair, Information Systems, Department of Radiology Medical Informatics Director for Radiology, Stanford Health Care

2 Samuel J. Dwyer, III, PhD

3 Investigative Radiology 10: , 1975 Multi-Planar Reconstruction

4 3D Surface Rendering 4 J Comput Assist Tomogr 5: 60-67, 1981

5 Financial Modeling of PACS Radiology 144: , 1982 Radiology 206: ,

6 PACS System Design Radiology 152: ,

7 Modeling of PACS Storage Capacity Radiology 169: ,

8 Comparison of Film to Digital Images Radiographics 12: ,

9 J Digit Imaging 9: ,

10

11 Disclosures and Acknowledgments Board of directors, Radiological Society of North America Product advisory board, Nuance Communications Shareholder and advisor, whiterabbit.ai Shareholder and advisor, Nines.ai Research support from: Google Philips Siemens GE Healthcare Paustenbach Medical AI Research Fund

12 Chapters 1. Perfect Pictures, Imperfect Prose 2. Expressing an Imaging Observation 3. Radiology Reporting Best Practices 4. A Guide to Reporting Style Practical Guide 5. Mastering Speech Recognition 6. Organizing the Radiology Report 7. The History of Radiology Reporting 8. Toward Structured Reporting 9. Standard Terminology for the Radiology Report 10. How to Think about Imaging Information History and Foundations 11. Decision Making for Diagnostic Imaging 12. The Future of Radiology Reporting 12

13 Morton, WJ & Hammer, EW The X- Ray or Photography of the Invisible and Its Value in Surgery

14 The First Radiology Reports (1896) I only got the The X-ray shows plainly that there is no stone of an appreciable size in the kidney. negative today and could not therefore report earlier The picture is not so strong as I would like but it is strong enough to differentiate the parts. 14 Courtesy New York Academy of Medicine

15 Dictation Tools Over the Last Century 15

16 83 y/o F, aortic valve replacement, CHF 18 months later

17 Radiology AI v4.0 June 2, 2018 Curtis P. Langlotz, MD, PhD Professor of Radiology and Biomedical Informatics Director, Center for Artificial Intelligence for Medicine & Imaging Associate Chair, Information Systems, Department of Radiology Medical Informatics Director for Radiology, Stanford Health Care

18 AI: Artificial Intelligence ML: Machine Learning NN: Neural Networks DL: Deep Learning Definitions AI: When computers do things that make humans seem intelligent ML: Rapid automatic construction of algorithms from data NN: Powerful form of machine learning DL: Neural networks with many layers

19 AI v1.0-v2.0: History

20 AI v1.0: AI: Feature Engineering Benign Malignant Custom programming Machine Learning AI v2.0: Benign Cancer Not Cancer Malignant 20 Algorithm icon by Sergey Novosyolov from the Noun Project

21 AI 2.0: Machine Learning spiculation microcalcification

22 Forms of Machine Learning Benign Malignant Nearest neighbor Linear support vector machine Radial support vector machine spiculation Decision tree Random forest Naïve Bayes microcalcification

23 AI v1.0: AI: Feature Engineering Benign Malignant Custom programming AI v2.0: Machine Learning Benign Cancer Not Cancer Malignant AI v3.0: Neural Networks and Deep Learning Benign Cancer Not Cancer Malignant 23

24 AI v3.0: The Present

25 Neural Network A flexible and powerful form of machine learning. Malignant Benign

26 Motivations: ImageNet 14 million images 21,841 distinct labels: 856 types of bird 993 types of tree 157 musical instruments Russakovsky O, Deng J, Su H, et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis. 2015;115(3):

27 Deep Neural Networks: Tens of Millions of Parameters

28 Learning Object Recognition Faces Cars Elephants Chairs Courtesy of Andrew Ng

29 Karpathy, Andrej & Li, Fei Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions, CVPR,

30 Data Labeling

31 Supported by Departments of: Radiology Pathology Medicine Ophthalmology Dermatology Collaborations with: Andrew Ng lab Fei Fei Li lab Chris Re lab Curt Langlotz, MD, PhD Matt Lungren, MD, MPH

32 Deep Learning Research in Radiology Images and Health Data from Patients and Organizations Image Transfer and Labeling Labeled Training Data New Deep Learning Methods Decision Support Systems Evaluation Deep Learning Explanation Methods Actionable Advice

33 Data Science Enterprise Resource Medical ImageNet EHR Images Omics Honest Broker (De-ID API) Data Commons App Ecosystem + Biobank Bench scientists Clinical researchers Students + trainees

34 Radiologist Labels 1: No Significant Abnormality 4: Possible Significant Abnormality May Need Action 9: Critical, Clinical Notified 1.5 million studies

35 Image Labeling Hassanpour, S & Langlotz, CP. Artif Intell Med 23(1):84-9, 2016.

36 The Power of Weak Labeling 1,000 Human expert labels CNN 85% accuracy 100,000 1,000 Rule-based weak labels CNN 83% 87% accuracy

37 Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li Jia-Li, David Ayman Shamma, Michael Bernstein, Li Fei-Fei A man is taking a picture behind a girl feeding an elephant.

38 Decision Support

39

40 Saliency Maps Machine learning enables systems tailored to local demographics.

41 Deep Learning to Improve MRI Image Quality High SNR ASL Synthetic ASL T2 weighted Proton density for regularized de-noising Low SNR ASL Enhao Gong John Pauly Greg Zaharchuk Stanford EE & Neuroradiology Difference map vs High SNR + Difference map vs High SNR

42 Stanford Healthcare AI Bootcamp

43 Expert-Level Chest Radiograph Interpretation Matt Lungren, MD, MPH and Andrew Ng, PhD.

44 1 Radiologist-Algorithm AUC Comparison =Statistically significant difference Radiologists Algorithm

45 AI for Global Health

46 AI to Augment Radiologists and Orthopedic Surgeons

47

48

49

50

51 Benefits of AI Throughout the Image Life Cycle Communicate timely data for a learning health system Predict positivity rate among similar patients Image reporting Test selection Select precise treatment with global reach Disease classification Image reconstruction Lower radiation dose and imaging time Computer aideddetection Image quality control Prevent/detect disease early among populations Image triage Reduce patient call-backs Decrease length of stay

52 The Role of Professional Organizations

53 Building Healthcare AI: Education Radiology trainees Refresher courses Scientific sessions Learning center Crowds cure cancer DEEP LEARNING CLASSROOM Clinical radiologists Research scientists Health IT industry

54 Building Healthcare AI: Research Radiology trainees Clinical radiologists Pediatric Bone Age Challenge Research scientists Health IT industry

55 Building Healthcare AI: Research & Education Radiology trainees Refresher courses Scientific sessions Learning center Crowds cure cancer DEEP LEARNING CLASSROOM Pediatric Bone Age Challenge Clinical radiologists Research scientists Health IT industry

56 Building Radiology AI: The Role of Professional Organizations Educate clinical users of AI algorithms Develop a robust technical workforce Convene collaborations: radiologists, scientists, industry Support development of AI use cases Assemble publicly-available training data sets Advocate for and provide research funding for AI Establish standards for AI data and algorithms Encourage balanced regulation of AI technology

57 Challenges of Machine Learning

58 Machine Learning Weakness: First Principles Courtesy of Dr Donna D'Souza, Radiopaedia.org Marcus G. Deep Learning: A Critical Appraisal.

59 Machine Learning Weakness: Inferring Abstract Concepts doi: /GII

60 Machine learning security: These are not stop signs? Eykholt et al. Robust Physical-World Attacks on Machine Learning Models. arxiv.org/abs/

61 Machine learning security: Everything is a toaster.

62 Adversarial Attacks Against Medical Deep Learning Systems Finlayson SG, Kohane IS, Beam AL. Adversarial Attacks Against Medical Deep Learning Systems

63 Radiology Example: Pneumothorax Detection Original Modified Finlayson SG, Kohane IS, Beam AL. Adversarial Attacks Agains Medical Deep Learning Systems

64 Radiology AI v4.0 #radiologyai4

65 The Future of AI in Radiology Today Radiology AI v4.0 Hype over binary detection tasks Radiologists fear systems designed to replace them Primary focus on neural network architecture Static training data sets created at great cost Private data sets from single institutions Single apps built for the average patient Systems rely solely on machine learning Peer-reviewed publication One-time regulatory approval process One-size-fits-all AI algorithms to health care organizations Recognition of task complexity Radiologists embrace systems that augment the care team Major focus on curation of high-quality labeled data Routine care creates training data for a learning health system Public challenges using multi-institutional data Multiple precision apps built for specific patient groups Systems combine machine learning and symbolic systems Open pre-print followed by peer-reviewed publication Post-approval monitoring of system performance Software and services that enable health care organizations to develop their own algorithms #radiologyai4

66

67 Thank You Curtis P. Langlotz, MD, PhD Professor of Radiology and Biomedical Informatics Director, Center for Artificial Intelligence for Medicine & Imaging Associate Chair for Information Systems Department of Radiology, Stanford University Informatics Director for Radiology Stanford Health