Developing ML Models for semantic segmentation of medical images

Transcription

1 Developing ML Models for semantic segmentation of medical images Raj Jena University of Cambridge Microsoft Research Cambridge

2 Disclosures I perform consultancy work for InnerEye team at Microsoft Research I am in receipt of funding from CRUK and MRC Opinions are my own! I am not a machine learning expert!!!

3 Radiation therapy Effective, and cost-effective anti cancer therapy Treat 156,000 patients per year in UK with curative intent Precision RT techniques should be used for 48% of UK cancer patients

4 Cancer imaging pathway Diagnose Stage Plan therapy Assess response / QR WBIC Terra 7T PET/MR unit

5 Precision radiotherapy imaging pathway Diagnose Stage Plan therapy Deliver therapy Assess response / QR Animation courtesy of Brainlab AG

6 Computer vision viewpoint Diagnosis Classification task (Commoditised ML) Glioblastoma

7 Computer vision viewpoint Stage Classification, Localisation, Object detection (Differentiated ML) UICC TNM: T2N1M1b NLP: Lung cancer with adrenal metastases

8 Segmentation in RT Planning Marking of tumour and normal tissue structures First step in RT pathway Time consuming, error prone step (2.5 hours for this patient)

9 Computer vision viewpoint Plan Semantic segmentation task (Area of unmet need) RT Planning Surgery / IR Therapy response Radiomics

10 Segmentation accuracy matters TROG study Additional of hypoxia activated radiosensitiser to radiotherapy in head & neck cancer Clear protocol for segmentation Study severely affected by consistency of segmentation

11 Semantic segmentation using decision forests

12 The goal of automatic 3D segmentation Two axial slices of the same CT scan Overlaid axial segmentations Natively 3D segmentation Bladder Prostate, seminal vesicles, rectum, bladder, left femur, right femur, skin

13 Why is voxel-wise semantic segmentation hard? The challenge 44 HU 44 HU Same HU value for different anatomies Large deformations Implants Beam-hardening artefacts Different image resolution Image noise Presence/absence of contrast medium Different patient preparation

14 patient 3 patient 2 patient 1 Our ground-truth labelled image dataset hundreds of patients axial + labels prostate seminal vesicles rectum bladder

15 Modeling context via learned neighborhood patterns Probe voxels Reference voxel

16 Deep Decision Forests for semantic segmentation - training Input CT image For each input voxel (and all its context features) Output - layer 0 M. Fiterau, A. Criminisi, S. Rota Bulo, P. Kontschierder, Deep Neural Decision Forests [Winner of the David Marr Prize]. ICCV 2015 L. Le Folgoc, A. V. Nori, S. Ancha, A. Criminisi, Lifted Auto-Context Forests for Brain Tumour Segmentation [Winner of the 2016 BrATS challenge]. MICCAI 2016.

17 Appearance context Semantic context Deep Decision Forests for semantic segmentation - training Input CT image Output - layer 1 Input - layer 0

18 Appearance context Semantic context Deep Decision Forests for semantic segmentation - training Input CT image Output - layer 2 Input - layer 1

19 Trained Deep Decision Forest model Forest layer 0 layer 0 Forest layer 1 layer 1 Input CT images Forest layer 2 layer 2 Forest layer 3 Segmentation

20 Integration at Addenbrooke s SimCT 1 Prosoma TPS 1 SimCT 2 TPS 2 TPS 3 Gateway

21 Innereye Segmentation accuracy Just accepted for publication in PMB this morning!!!

22 Innereye Segmentation speed Glioblastoma Manual segmentation time minutes Innereye ML 1 minute segmentation 3-4 minutes fix-up

23 Segmentation is the core enabling technology

24 InnerEye segmentation services Input: MR High/low grade gliomas Oedema Necrosis Longitudinal analysis Input: CTA Left/right parenchymas Collecting systems Arteries, Veins Masses Input: CT Spine Microsoft Azure InnerEye segmentation services BrainML.Glioblastoma BrainML.LowGradeGlioma HeadNeckML ThoraxML.Lungs AbdomenML.Liver AbdomenML.Kidneys PelvisML.Spine PelvisML.Prostate Medical components, paid services Input: CT Parotid glands Spinal cord Mandible Input: CT Liver Masses Input: CT Prostate Seminal vesicles Left/right femurs Rectum Bladder

25 Adaptive radiation therapy Radiotherapy normally given daily over a 5-6 week course of treatment Alterations in anatomy can occur due to Tumour shrinkage Consequential change in geometry of healthy tissues

26 5 minute re-plan? Position patient Scan patient Segment tumour and healthy tissues Recalculate treatment plan Deliver new treatment

27 Quantitative Radiology MR T1 contrast with overlaid tumor segmentation

28 Volume cc Quantitative Radiology Volume of active tumor time

29 Predicting toxicity of radiotherapy

30 Predicting toxicity of radiotherapy

31 Predictive radiomics for radiotherapy

32 Predictive radiomics for radiotherapy Baseline function data Planning CT / MRI data Daily imaging data Follow-up toxicity questionnaires InnerEye segmentation services Autoprognosis ensembles of machine learning pipelines for structured kernel learning

33 A collaborative study Study oversight Cambridge Clinical Trials Centre InnerEye segmentation services Antonio Criminisi - Innereye Autoprognosis Mihaela van der Schaar - CMIH

34 Project InnerEye Medical Imaging AI to Empower Clinicians

35