Biomedical Engineering in the Metroplex: Hyperspectral Imaging for Medical Use Edward H. Livingston, MD, FACS, AGAF Professor and Chairman-Gastrointestinal and Endocrine Surgery Professor and Chairman-Graduate Program in Biomedical Engineering University of Texas Southwestern School of Medicine Contributing Editor-JAMA 3/13/2012 SURGERY@utsouthwestern.edu 1
The Problem Surgeons cannot see what they are doing when they operate Does this worry you? (it should) 3/13/2012 SURGERY@utsouthwestern.edu 2
The Problem Gallbladder Disease is one of the most frequent causes for hospitalizations and surgery 750,000 hospital admissions annually >500,000 Cholecystectomies performed annually 0.25-0.5% Bile Duct Injury Result in up to 2,500 injuries per year.
What Does The Gallbladder Do? 3/13/2012 SURGERY@utsouthwestern.edu 4
What Things Should Look Like 3/13/2012 SURGERY@utsouthwestern.edu 6
What They Really look Like 3/13/2012 SURGERY@utsouthwestern.edu 7
What Keeps Us Out Of Trouble? 5+ Years of intense training after medical school Inherent skill, experience Intuition, touch, being luckier than good. Does these characteristics meet an engineers standards for reliability, reproducibility, QC? 3/13/2012 SURGERY@utsouthwestern.edu 8
Available Technology 3/13/2012 SURGERY@utsouthwestern.edu 9
We Need Engineering Solutions Develop an imaging system that penetrates opaque tissues Identify important structures by their unique chemical composition 3/13/2012 SURGERY@utsouthwestern.edu 10
Solutions Infrared imaging-penetrates tissue Spectral Imaging-Identify the chemical spectrum for bile 3/13/2012 SURGERY@utsouthwestern.edu 11
Bile is Green 3/13/2012 SURGERY@utsouthwestern.edu 12
Hyperspectral Imaging 3/13/2012 SURGERY@utsouthwestern.edu 13
LCTF HSI 3/13/2012 SURGERY@utsouthwestern.edu 14
Figure 1. In vivo visible-reflectance hyperspectral imaging system. a, Broadband light source produces radiation, which is reflected off a subject and spectrally discriminated by a liquid crystal tunable filter (LCTF), transduced by a CCD detector, and digi... Copyright American Heart Association Zuzak K J et al. Circulation 2001;104:2905-2910
Apparent Absorbance Apparent Absorbance Apparent Absorbance Near Infrared Macroscopic Hyperspectral Imaging Gallbladder, Cystic Duct and Liver Tissue Zuzak & Livingston Cystic Duct 1.0 Gallbladder 0.8 0.6 1.0 0.8 0.6 0.4 0.2 0.0 650 750 850 950 1050 Wavelength nm 0.4 0.2 0.0 650 750 850 950 1050 Liver Wavelength nm 1.0 0.8 0.6 0.4 0.2 Principle Component Analysis 0.0 650 750 850 950 1050 Wavelength nm 60 pound swine, post mortem
Problems The system was too slow 1 minute acquisition time Liquid crystal filter was limiting Too slow for clinical use Spectral artifacts from motion 3/13/2012 SURGERY@utsouthwestern.edu 17
Solution Collaboration between UTSW and UTA Partner with TI and Elcan Develop a DLP based system to generate light of specific wavelengths at discrete time intervals Vast system performance improvement 3/13/2012 SURGERY@utsouthwestern.edu 18
DLP 3/13/2012 SURGERY@utsouthwestern.edu 19
Initial HSI 3/13/2012 SURGERY@utsouthwestern.edu 20
Image Frame # 26 of 101 (Wavelength = 480.00) Apparent Absorbance (a.u.) Apparent Absorbance (a.u.) 10-04-2011 Case 07 (Schwarz) CBD Arteriole 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 380 480 580 680 780 Wavelength (nm) 380 480 580 680 780 Wavelength (nm) 10-4-2011 CBD Arteriole 10-4-2011 CBD Arteriole
Image Frame # 26 of 101 (Wavelength = 480.00) Procedure notes state middle structure is CBD 10-18-2011 Case 08 (Schwarz)
Apparent reflectance intensity (a.u.) 1.6 Unfiltered Spectra 07-19-2011 CBD 05-26-2011 CBD 1.4 1.2 1 0.8 08-09-2011 CBD 08-30-2011 CBD-6 08-30-2011 (2) CBD 09-20-2011 CBD 10-04-2011 CBD 10-18-2011 CBD 0.6 0.4 0.2 0 380 430 480 530 580 630 680 730 780 Wavelength (nm)
Normalized apparent reflectance (a.u.) Filtered and Normalized Spectra 1 0.9 0.8 0.7 0.6 07-19-2011 CBD 05-26-2011 CBD Avg norm spectrum 08-09-2011 CBD 08-30-2011 (1) CBD-6 08-30-2011 (2) CBD 09-20-2011 CBD 10-04-2011 CBD 10-18-2011 CBD 0.5 0.4 0.3 0.2 0.1 0 380 430 480 530 580 630 680 730 780 Wavelength (nm)
Hemoglobin Least Square Solution Obtain pure substance spectral standards (saturated and destaurated Hgb)-Vector P Projection Matrix Project the observed spectra onto the saturated and desaturated Hgb planes to get the amount of Hgb 3/13/2012 SURGERY@utsouthwestern.edu 25
Apparent Absorbance A xy (λ i )= -log 10 (R xy (λ i ) / R xy (λ i ) ) Reference Spectral Components: Oxyhemoglobin and Deoxyhemoglobin 1.0 0.9 Oxyhemoglobin Deoxyhemoblobin 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 520 540 560 580 600 620 640 Wavelength (nm)
Deconvolution Example 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 520 540 560 580 600 620 640 Wavelength (nm) Measured Superposition HbO 2 Reference Hb Reference
Deconvolution Example 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 520 540 560 580 600 620 640 Wavelength (nm) Measured Superposition HbO 2 Reference Hb Reference
Deconvolution Example 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 520 540 560 580 600 620 640 Wavelength (nm) Measured Superposition HbO 2 Reference Hb Reference
DLP Hyperspectral Imaging System COMPUTER Synchronized Hardware Control Image Processing Programmable Spectral Light Source DLP Inside Focal Plane Array Tissue Sample Acquire Data Cube Digitized Hyperspectral Image Cube Visualization
Graphical User Interface (GUI) DLP Hyperspectral Imaging System GUI developed by:
Timing Diagram for DLP HSI with 126shot Method Start Acquisition (n = 1) Command OL 490 to Illuminate Spectrum n n = n+1 HQ2 Open Shutter Expose CCD 40 ms t shot = 116.3 ms Close Shutter t acquisition = 116.3 * 126 = 14 648 ms A/D conversion Save as temp00n.dat no Is n = 126? yes Process with Oxyz Algorithm in MATLAB t processing = 8 432 ms Output bitmap image T total = 23 080 ms
Timing Diagram for DLP HSI with 3shot Method Start Acquisition (n = 1) Command OL 490 to Illuminate Spectrum n n = n+1 HQ2 Open Shutter Expose CCD 1.25 ms t shot = 93.3 ms Close Shutter t acquisition = 93.3 * 3 = 280 ms A/D conversion Save as temp00n.dat no Is n = 3? yes Process with 3shot Algorithm in MATLAB t processing = 110 ms Output bitmap image T total = 390 ms
Relative % HbO2 Normalized to Control Video Rate Hyperspectral Imaging Biochemical Visualization of Human Kidney Over Time 90 70 50 30 Relative Contribution of HbO 2 Control 8 min after clamping 1 min after clamp off 10 1.2 1.1 Clamped Control Iced Cut Series1 Poly. (Series1) 1 0.9 R² = 0.3774 0.8 0.7 0.6 Clamp Off 0 10 20 30 40 50 60 70 Time after Clamping (minutes) Images from Human Partial Case 15 (12-18-09) and Plot of all Cases
Animal Study DLP Hyperspectral Imaging Picture of pig kidney immediately after arterial clamping Assuming arterial blood is shunted away from kidney by arterial clamp Invasive Lycox probe inserted into kidney Kidney
DLP Hyperspectral Imaging Kidney Hyperspectral Image discovers Pig Kidney with 2 Renal Arteries 90 80 70 60 50 40 30 20 Relative Contribution of HbO 2 -Discovery: The Kidney had two renal arteries. Only the lower pole artery was clamped, Hyperspectral Imaging immediately indicated continued perfusion of the upper Kidney pole. 10
DLP Hyperspectral Imaging Kidney Hyperspectral Image of Pig Kidney with Both Renal Arteries Clamped 90 80 70 60 50 40 30 20 Relative Contribution of HbO 2 Re-situating the clamp over both arteries led to ischemia of both upper lower poles. 10
Pig Partial Nephrectomy Trial 2 Note:- 3 shot Images showing more oxygenation in intestines as kidney becomes deoxygenated 5 min 32 sec before clamping 4 min 44 seconds before clamping Clamp on 1 minute after clamping 2 minutes after clamping 3 minutes after clamping 100 Clamp On 90 80 70 % HbO2 60 50 40 Kidney Intestine 30 20 10 0-6 -5-4 -3-2 -1 0 1 2 3 4 Time before and after Clamping in Minutes
Relative Percent HbO2 Pig Partial Nephrectomy Case 1 (Kidney and Intestines) Control 27 minutes before clamping 5 minutes after clamping 15 minutes after clamping 34 minutes after clamping 57 minutes after clamping 1 minute after clamp off 14 minutes after clamp off 120 Clamp On (100%) Clamp Off 100 80 60 Kidney Intestines 40 20 0-40 -20 0 20 40 60 80 100 Time before and after Clamp (Minutes)
Relative Percent HbO2 Human Partial Nephrectomy Case 3 Control (0 min) After Clamping (3 min) Clamped & After Icing (11 min) Clamped (15 min) Clamped (21 min) Clamped (31 min) After Clamp Off (33 min) Clamp Off (1 hr 5 min) Partial Nephrectomy 100 Clamp On Clamp On & After Icing Clamp Off 90 80 70 60 50 % HbO2 40 30 20 10 0 0 10 20 30 40 50 60 70 Time in Minutes
In Vivo, Visible Reflectance, Hyperspectral Imager Coupled to a Zeiss Neurosurgical Microscope Source Illumination Reflection Of Subject
Brain Surgery Monitoring for Preventing Post Surgery Stroke Control After Induced Ischemia 90 80 70 60 50 40 30 20 10 % HbO2 Normal Tissue Damaged Tissue
DLP Hyperspectral Microscope Monitoring Diabetic Retinopathy Double Knock Out (Apoe -/-, db/db) Mouse Model
Relative Oxyhemoglobin Double Knock Out Diabetic Mouse: Hom Apoe/ Hom db 90 Day: 78 Day: 136 70 B Parental 1: Hom Apoe/ Wildtype db 50 Day: 79 Day: 142 30 10 Parental 2: Wildtype Apoe /Hom db Day: 81 Day: 151 Data Collected Using LCTF
LCTF Hyperspectral Imaging System For Imaging Human Retina Illumination Mirror CCD Beam Splitter 0 Subject s Head rest Visible LCTF Common Center of Rotation Eye Piece Magnification Knob Light Source Joystick Intensity Control Knob Table Top
Apparent Absorbance Relative Oxyhemoglobin Contribution Apparent Absorbance % HbO 2 70 60 50 40 30 20 10 0 Microscopic Hyperspectral Imager: Human Retinal Imaging of Oxyhemoglobin Contribution P < 0.0001 Area 1 Area 2 1.0 0.8 0.6 0.4 0.2 0 530 550 570 590 Wavelength (nm) 1.0 0.8 0.6 90 80 70 60 50 40 30 20 10 Gray-scale Encoded Hyperspectral Image 0.4 0.2 0 530 550 570 590 Wavelength (nm)
Hyperspectral Fundus Camera Laptop CCD Camera LLG Spectral Light Source Fundus Camera
60 50 40 30 20 10 Need to Improve Visualization Algorithms artifacts Relative HbO 2
In Vivo Hyperspectral Imaging of Human Tissue: Spatial Variation of Percentage of HbO 2 and Surface Temperature in Response to Burn Apparent Absorbance A xy (λ i ) = -log 10 (R xy (λ i ) / R xy (λ i ) ) Bright Field 1 inch Thermal Image 1 inch 1 inch Multivariate Least Squares with respect to Oxyhemoglobin Spectra Under points and 500 540 580 620 660 Wavelength (nm)
BURNS Color Photo Chemically Encoded HbO 2 Chemically Encoded H 2 O 10 30 50 90 70 Relative Contribution of HbO 2 10 30 50 70 90 Relative Contribution of H 2 O
BURNS Post Fasciectomy Color Photo Chemically Encoded Oxyhemoglobin Chemically Encoded Water 90 80 70 60 50 40 30 20 10 Relative Contribution of HbO 2 90 80 70 60 50 40 30 20 10 Relative Contribution of H 2 O
Suture Tension Session 1 051707 Session 2 052107 Digital Image Zuzak & Livingston NIR HbO 2 Vis HbO 2 Patient 002, Monitoring Post Amputation Left Lower Limb, Foot, left BKA A A 76.5 6.5 B 84.5±2.1 A 62 3.9 B 62±1.2 B1 Percentage of HbO 2 100 90 80 70 60 50 40 30 20 10 0 100 90 80 70 60 50 40 30 A B1 20 10 A 0 Percentage of HbO 2 B1 A A 70 4.2 B 70±3.2 B2 B2 90 80 70 60 50 40 30 A 68.9 1.7 20 B 67.6±1.1 10 90 80 70 60 50 40 30 20 10
Percentage of H 2 O Percentage of H 2 O Percentage of H 2 O Monitoring Post Amputation Recovery Percentage of H 2 O Percentage of H 2 O Percentage of H 2 O 100 90 80 70 60 50 40 30 20 10 0 100 90 80 70 60 50 40 30 20 10 0 A 75.83 2.59 B 76.3±0.86 A 56.91 2.91 B 56.2±0.63 Session I A A B B 90 80 70 60 50 40 30 20 10 90 80 70 60 50 40 30 20 10 Patient 002, Amputation, Left AKA Patient 005, Amputation, Left AKA 100 90 80 70 90 80 70 60 Patient 006, Amputation, Left BKA 60 50 40 30 20 10 0 A 85.61 3.53 B 81.58±4.38 A B 50 40 30 20 10 Patient 002, 005, 006
Percentage of Oxyhemoglobin Percentage of oxyhemoglobin Hyperspectral Imaging Visible NIR 90 90 80 80 70 70 60 60 50 40 30 20 10 50 40 30 20 10 81.05 % HbO 2 85.83 % HbO 2 Data collected 10 th minute after removing the tight shoe
Diabetic Neuropathy 79.1163 +/- 2.208 71.7893 +/- 3.7673 69.8058 +/- 2.8421 63.8259 +/- 5.3345 62.0387 +/- 3.2884 57.3525 +/- 5.7005
Other Potential Applications The speed and versatility of DLP technology is making Hyperspectral Imaging practical for a wide variety of surgical and clinical applications. Neurological Surgery - Preventing Stroke After Brain Tumors. Plastic Surgery - Skin flaps, Wound Management and Burns. Surgery - Non-Invasive Laparoscopic Optical Biopsy During. Urology Tumor Removal during Kidney Surgery. Ophthalmology Monitoring Diabetic Retinopathy. Clinical Monitoring Managing Lower Limb Amputations.
Versatility 3/13/2012 SURGERY@utsouthwestern.edu 57
Fluorescence of ICG Vascular Infusion and DLP Spectral Excitation Picture of Placenta Prior to ICG infusion Placenta Control Prior to ICG infusion Vascular infusion Of ICG via cannual The DLP Illuminates tissue with predetermined spectral illumination exciting the ICG to fluoresce.
Major seed funding provided by Texas Instruments Supplemental funding provided in part by Hudson-Pen Endowment Smith Endowment Department of Energy.
BME in the Metroplex 3/13/2012 SURGERY@utsouthwestern.edu 60
1972 3/13/2012 SURGERY@utsouthwestern.edu 61
The Future 3/13/2012 SURGERY@utsouthwestern.edu 62