Improving Network Accuracy with Augmented Imagery Training Data

Transcription

1 Improving Network Accuracy with Augmented Imagery Training Data Ted Hromadka 1, LCDR Niels Olson 2 MD 1, 2 US Navy NMCSD Presented at GTC 2017

2 Goals Determine effectiveness of various affine transforms for augmenting medical imagery training data

3 Background prostate cancer is a significant problem for DoD US military s hospitals care for disproportionately more male patients Prostate cancer is second-leading cause of cancer death in American men Approximately 220,000 new cases per year Early screening involves a blood test for prostate-specific antigen (PSA) or a digital rectal exam (DRE) If those tests generate abnormal results, then a prostate biopsy may be required

4 Each biopsy procedure creates around 12 samples Prostate biopsy is conducted by taking core samples using a hollow needle After processing, 5 micron sections of these samples are placed on glass slides, stained, and manually interpreted by a pathologist under a microscope.

5 Analysis is very labor-intensive Digital scans are opened with custom viewing software from the microscope vendor Multiple zoom levels available up to 40x. This dataset was scanned at 20x. Pathologist will annotate cancerous regions with polygons drawn by hand with a mouse Process requires careful judgment and is susceptible to fatigue and stress factors. Polygons cannot be edited once drawn (e.g., at higher magnification).

6 Augmented data Necessary when training data set is too small to generalize Some data is more expensive to source Medical, copyrighted, classified, etc Also necessary when doing object detection, not just image classification Sensitivity and accuracy Generally supplementing original training data set with significant transforms Also synthesizing new training data from models or GANs

7 Using MNIST to demonstrate effect of rotation on CNN 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0%

8 DetectNet Released by NVIDIA with DIGITS4, last summer Based on KITTI - self-driving car computer vision project Incorporates some affine transformations as a training option detectnet_augmentation_param { crop_prob: 1.0 shift_x: 32 shift_y: 32 scale_prob: 0.4 scale_min: 0.8 scale_max: 1.2 flip_prob: 0.5 rotation_prob: 0.0 max_rotate_degree: 5.0 hue_rotation_prob: 0.8 hue_rotation: 30.0 desaturation_prob: 0.8 desaturation_max: 0.8 }

9 Test Procedure Goal controlling all other variables, determine the effect of each augmented data approach First evaluate DetectNet-supported parameters Stride (shift) Scale We have two variations on this: image zoom vs microscope zoom Flip (horizontal, vertical) Rotation, Hue, Desaturation Not relevant: Crop

10 Test Procedure Also evaluate factors beyond DetectNet Oversaturation Noise (random speckle) Artifacts Ink Scanning stripes Not evaluated: skew/shear, warping, squeeze mapping, non-affine transforms All tests use the same learning parameters and stock GoogLeNet on top of Caffe1 Cross-validation (25%) to reduce overfitting Each factor test run five times Once for each of 5 training/testing permutations (staggered 80%/20% cohorts) Accuracy results were averaged together at the end

11 Specifications System NVIDIA DevBox 1 4x NVIDIA TITAN X GPU 12GB memory per GPU 6-core Intel i7-5930k at 3.5 GHz with 64GB DDR4 Ubuntu 14.04, DIGITS 4, CUDA/cuDNN/NVCaffe DIGITS 5 rumored to support new image segmentation features

12 Specifications Imagery Source: NMCSD samples Data later shared with Cancer Moonshot, Joint Pathology Center, and commercial partners Collected, scanned, and annotated by NMCSD pathologists 202 annotated full-size color SVS images over 100K image 256px Average full size image ~ 845 MB

13 MATLAB image chipper prepared the images Split SVS into image chips of size 256x256 pixels at the 4:1 zoom level Chipper labels each image chip based on XML annotation polygons (50% inclusion rule for Caffe; more refined segmentation for TF) Chipper 2.0 also used pixel averaging and histograms to determine if chip was a blank. This could also be applied to ink smears. Original set of image chips were then augmented with a supplemental set of processed chips XML parser built on work by Andrew Janowczyk (

14 Need to handle pragmatic labeling, real-world data Training data suffers from some inaccuracies Annotation was not meant for training neural networks Not pixel-perfect Artifacts due to the scanner or tissue preparation Striping Ink

15 Side note: how accurate is the measure of accuracy? Elmore et al Breast Biopsy Concordance study found only 75% agreement between expert pathologists JAMA, 2015: If an image chip is 50% cancer and 50% other tissue, how should it be labeled? This issue already addressed in frameworks like TensorFlow (regions exactly specified in JSON), Caffe+DetectNet (regions specified in labels.txt up to 400px), other frameworks Need protocol for the confidence levels What threshold to use when network gives it a substantial but minority chance of cancer?

16 Original Naïve Model baseline Chip count Cancer: 38,080 Benign: 34,095 MATLAB None Effect Control set Overall Accuracy Precision Recall Sensitivity Specificity

17 Stride-varied Model Chip count Cancer: 35,033 Benign: 32,731 MATLAB None Effect Surprisingly similar to the control set Overall Accuracy Precision Recall Sensitivity Specificity

18 Flip Models Chip count Cancer: 76,160 Benign: 68,190 MATLAB B fliplr(a); B flipud(a); Effect in FlipH Consistent 2 point improvement. FlipV was similar. Overall Accuracy Precision Recall Sensitivity Specificity

19 Rotate-90 Model Chip count Cancer: 76,160 Benign: 68,190 MATLAB B imrotate(a, 90); Effect Slightly improved from flipping, not sure why Overall Accuracy Precision Recall Sensitivity Specificity

20 Rotate-360 Model Chip count Cancer: 152,320 Benign: 136,380 MATLAB B imrotate(a, 90); B imrotate(a, 180); B imrotate(a, 270); Effect 3-point improvement, no overfitting indicated Training loss rate decreased with validation loss Overall Accuracy Precision Recall Sensitivity Specificity

21 Hue Model Chip count Cancer: 76,160 Benign: 68,190 MATLAB H rgb2hsv(a); Effect H(:,:,1) H(:,:,1) * (rand / 50.0); B hsv2rgb(h); Insignificant Overall Accuracy Precision Recall Sensitivity Specificity

22 Saturation Model Chip count Cancer: 76,160 Benign: 68,190 MATLAB B(:,:,:) A(:,:,:) * 2; Effect Better at benign tissue De-saturation insignificant effect Overall Accuracy Precision Recall Sensitivity Specificity

23 Poisson Noise Model Chip count Cancer: 76,160 Benign: 68,190 MATLAB B imnoise(a, poisson ); Effect Promising, needs more variations (salt & pepper, etc) Overall Accuracy Precision Recall Sensitivity Specificity

24 Inked Model Chip count Cancer: 76,160 Benign: 68,190 MATLAB B ink(a); % 100-line script Effect Insignificant probably only helped with edge cases Overall Accuracy Precision Recall Sensitivity Specificity

25 Streaked Model Chip count Cancer: 76,160 Benign: 68,190 Pseudo-MATLAB Choose random column Starting from that col, add ++i to each col Do for 80 cols Effect Similar to inked version edge cases? Overall Accuracy Precision Recall Sensitivity Specificity

26 Conclusions For medical imagery Didn t help: Varying chip stride (a.k.a. translation) Hue Adding ink Adding scanner streaks Not sure: Horizontal and vertical flips Helpful: Rotations (surprising implies quantifiable orientation?) Oversaturation helped with benign tissue true identification Adding random noise was the most help Only tried Poisson, will explore other options

27 Current incarnation uses TensorFlow Shapely, TensorFlow, GoogLeNet, TensorBox Augmented with rotations + noise only Struggles with arteries, seminal vesicles Correctly labels benign tissue regions that were incorrectly included in cancer annotations

28 Next steps No signs of overfitting but seeking more data Adaptive refinement low-magnification whole slide image initially high-magnification regions of interest Try 128px chips (or higher magnification 256px chips) to reduce chance of multiple tissue types per image Ensemble classifiers for the edge cases Evaluating possible R&D deployment via HPCMP Portal on Hokule a machine

29 Thank you Ted Hromadka