CRANIOMAXILLOFACIAL DEFORMITIES/COSMETIC SURGERY Measuring Upper Airway Volume: Accuracy and Reliability of Dolphin 3D Software Compared to Manual Segmentation in Craniosynostosis Patients Valerie R. de Water, BSc,* Joan K. Saridin, IR,y Frederik Bouw, MD, DDS,z Magdalena M. Murawska, MSc,x and Maarten J. Koudstaal, MD, DDS, PhDk Purpose: To test the accuracy and reliability of Dolphin 3-dimensional (3D) software airway analysis compared with manual segmentation in patients who underwent a Le Fort III osteotomy. Materials and Methods: Computed tomographic scans of 20 patients with syndromic craniosynostosis at Sophia s Children s Hospital (Rotterdam, The Netherlands) were used for airway volume measurements using Dolphin 3D. The same scans had been used for measurement using a manual segmentation method. The results of this previous study were reported in 2010. The manual segmentation measuring result was used as a gold standard. The airway was subdivided into the oropharynx and the nasal passage. A linear mixed effects statistical model was applied. Results: Dolphin 3D measurements differed from manual segmentation by 9 to 43%, depending on the observer, the time at which the measured scan was acquired (pre- or postoperative), and the airway compartment being measured. The highest accuracy for Dolphin 3D was found for measurements from postoperative scans of the nasal passage. Conclusion: The airway analysis tool of Dolphin 3D is not accurate or reliable enough to use in a Le Fort III osteotomy evaluation. When scanning properties are conditioned and measurements are standardized, accuracy and reliability may increase. Ó 2014 American Association of Oral and Maxillofacial Surgeons J Oral Maxillofac Surg 72:139-144, 2014 Patients with craniosynostosis often have anomalous airway morphology, which may lead to severe obstructive sleep apnea. 1,2 Le Fort III (LFIII) osteotomy has proved effective in increasing the airway volume by 27 to 37% and thus decreasing respiratory problems. 3-6 Upper airway volume analysis may be helpful in evaluating the results of LFIII maxillary advancement in patients *Undergraduate Medical Student, Department of Oral and Maxillofacial Surgery, Erasmus University Medical Center Rotterdam, Sophia s Children s Hospital, Rotterdam, The Netherlands. ymaxillofacial Prosthetist, Department of Oral and Maxillofacial Surgery, Erasmus University Medical Center Rotterdam, Sophia s Children s Hospital, Rotterdam, The Netherlands. zresident, Department of Oral and Maxillofacial Surgery, Erasmus University Medical Center Rotterdam, Sophia s Children s Hospital, Rotterdam, The Netherlands. xphd Student, Department of Biostatistics, Erasmus Medical Center, Rotterdam, The Netherlands. kmaxillofacial Surgeon, Department of Oral and Maxillofacial Surgery, Erasmus University Medical Center Rotterdam, Sophia s Children s Hospital, Rotterdam, The Netherlands. with craniosynostosis. In a previous study, airway volumes of patients who underwent a LFIII osteotomy were evaluated with a manual segmentation software technique. This method of measurement requires the user to delineate the airway slice by slice. The authors found a significant increase in airway volume of patients with craniosynostosis after LFIII osteotomy Address correspondence and reprint requests to Valerie R. de Water: Department of Oral and Maxillofacial Surgery, Erasmus University Medical Center Rotterdam, Sophia s Children s Hospital, Room D-230, Gravendijkwal 230, Postbus 2040, 3000 CA, Rotterdam, The Netherlands; e-mail: v.dewater@erasmusmc.nl Received April 16 2013 Accepted July 22 2013 Ó 2014 American Association of Oral and Maxillofacial Surgeons 0278-2391/13/00945-2$36.00/0 http://dx.doi.org/10.1016/j.joms.2013.07.034 139
140 MEASURING UPPER AIRWAY VOLUME USING DOLPHIN 3D and proposed to implement volume measurements in the treatment protocol. 6 However, the manual segmentation method is time consuming and thus undesirable to use as a standard method for measuring airway volumes. The Dolphin 3-dimensional (3D) software is a Digital Imaging and Communications in Medicine (DICOM) viewer that contains a tool for semiautomatic airway segmentation and volume measurement. It requires the user to initiate the calculation by selecting and limiting the specific airway part of interest. It has been qualified and used in several studies. 7-9 Few studies, however, have specifically evaluated the accuracy and reliability of the airway analysis tool of Dolphin 3D. 10,11 In these previous studies, the airway volume measurements were performed on scans acquired at fixed settings. The aim of this study was to validate the reliability and accuracy of Dolphin 3D in measuring airway volumes in patients with craniosynostosis, from several scans, when settings were not necessarily fixed. Materials and Methods This study was approved by the medical ethics review committee of the Erasmus University Medical Center (Rotterdam, The Netherlands). The guidelines as stated in the Declaration of Helsinki were followed. Computed tomographic (CT) scans measured with manual segmentation in the previous study were exported to Dolphin 3D (Dolphin Imaging 11.0, Dolphin Imaging and Management Solutions, Chatsworth, CA) in DICOM format. The images were acquired at Sophia Children s Hospital from 2003 through 2008, with FIGURE 1. Delineation of the hypopharynx and oropharynx with Dolphin 3-dimensional software. de Water et al. Measuring Upper Airway Volume Using Dolphin 3D. J Oral Maxillofac Surg 2014.
DE WATER ET AL 141 a slice thickness of 1.25 mm and different radiation doses. Volumes obtained with Dolphin 3D were compared with volumes previously measured with a manual segmentation method using MeVisLab (MeVis Medical Solutions AG, Bremen, Germany). To obtain airway volume, Dolphin 3D requires from the user to define the airway part of interest by taking 3 actions. First, the user has to restrict the volume of interest from adjacent volumes by delineating the compartment borders in the sagittal, axial, and coronal planes in a 2-dimensional view. For optimal accuracy, the observer should use all 3 planes to limit the airway. Second, the user has to place seed points in the target compartment. Seed points denote densities that represent the airway. The target airway volume will grow from these seed points. Third, a threshold value must be determined. This threshold defines a density range that will be included in the measured airway volume. The airway was subdivided into the oropharynx (compartment A) and the nasal passage (compartment B). Compartment A was defined as the airway space between the most caudal point of the hyoid bone and half the length of the uvula (Fig 1). Compartment B was defined as the airway space between half the length of the uvula and the most cranial point of the nasal passage. Compartments were defined in midsagittal views. When the hyoid bone was not captured in the scan, an alternative lower boundary was defined. The paranasal sinuses and pyriform sinus were considered dead space. For this reason, these sinuses were excluded from measurement. Compartment A was delineated and analyzed. To obtain a reliable total volume without overlapping of the 2 compartments, compartment B was delineated by dragging the lower borders of compartment A to the nasal passage. The total airway volume was calculated simply by adding the 2 compartment volumes. As a result of variations in scanning properties, the authors could not rely on fixed threshold values. The delineated airway was evaluated visually by the observer, FIGURE 2. Delineation of the nasal passage with Dolphin 3-dimensional software. The airway delineation was visually evaluated and repeatedly adjusted until it fit the authors definition. de Water et al. Measuring Upper Airway Volume Using Dolphin 3D. J Oral Maxillofac Surg 2014.
142 MEASURING UPPER AIRWAY VOLUME USING DOLPHIN 3D Table 1. AVERAGE AIRWAY VOLUMES FOR DOLPHIN THREE-DIMENSIONAL MEASUREMENTS AND MANUAL SEGMENTATION MEASUREMENTS, PER COMPARTMENT, AND PER OBSERVER Dolphin 3-Dimensional Measurements Observer 1 Observer 2 Average Manual Segmentation Compartment A (mm 2 ) Preoperative 10,090 (4,989) 7,676 (4,016) Postoperative 10,040 (4,109) 8,747 (3,895) Compartment B (mm 2 ) Preoperative 19,963 (4,152) 17,164 (3,955) Postoperative 27,351 (7,075) 24,761 (6,432) Total (mm 2 ) Preoperative 30,054 (7,922) 23,448 (9,841) 26,845 (7,716) 24,840 (6,953) Postoperative 36,574 (10,290) 31,501 (9,677) 34,448 (8,180) 33,508 (8,382) Post- vs preoperative 6,521 (7,116) 10,397 (89,404) 8,459 (8,045) 8,668 (4,678) Note: Data are presented as average (standard deviation). de Water et al. Measuring Upper Airway Volume Using Dolphin 3D. J Oral Maxillofac Surg 2014. and delineation of the airway was repeatedly adjusted until it represented the authors definition (Fig 2). STATISTICAL ANALYSIS For all measurement categories, averages and standard deviations were computed. For statistical analysis of the data, a linear mixed effects model was applied. As a response, the absolute percentage of difference between values obtained with Dolphin 3D and values obtained with the manual segmentation method was used (ie, jd3d MSj/MS, where D3D represents Dolphin 3D and MS represents manual segmentation). The model contained fixed and random effects on measurement accuracy for compartment (A, B, or total), observer, and time of measurement (pre- or postoperative). For testing fixed effects, a likelihood ratio test was used; for testing random effects, a mixture of c 2 distributions was used. Because of the small sample, the significance level was set at an a value equal to 0.1. Results Average airway volumes for Dolphin 3D and the semiautomatic segmentation program are presented in Table 1. The linear mixed effects model contained random effects for the compartment measured (A, B, total), observer, and random intercept accounting for all remaining variability within a patient. The fixed effects were the compartment (A, B, total) effect, observer effect, and time of measurement (pre- or postoperative). All these fixed effects were significant. Fixed effects are presented in Table 2. According to these results, Dolphin 3D calculated volumes showed an absolute percentage of difference of 36% from volumes computed using manual segmentation when the measurements were made of compartment A (oropharynx) on preoperative scans by observer 2. When compartment B was measured,thedifferencebetweenthemanualsegmentation method and Dolphin 3D was smaller (0.20). Dolphin 3D produced the most accurate results when measuring compartment B (nasal passage) on postoperative scans, in this case by observer 1, which produced a deviation of 0.12 from the manual segmentation method. Factors that contributed to the accuracy of Dolphin 3D were postoperative scanning, measurement of compartment B, and, in this case, measurement by observer 1. The difference between observers regarding measured volumes was significant, with a tendency for a larger percentage of bias in the assessments of observer 2 compared with observer 1. The intraobserver difference calculated for observer 1 was not significant. Table 2. FIXED EFFECTS Estimate SE t Value P Value Reference 0.36 0.06 6.35 <.0001 Compartment B 0.15 0.04 4.15 <.0001 Compartments A + B 0.12 0.03 4.13 <.0001 Postoperative 0.09 0.04 2.04 <.0001 Observer 2 0.07 0.04 1.8.06800 Note: The reference is a Dolphin 3-dimensional volume measurement of compartment A (hypo- and oropharynx) on postoperative scans by observer 1. Other effects are listed below the reference, with their deviations from the reference value. Abbreviation: SE, standard error. de Water et al. Measuring Upper Airway Volume Using Dolphin 3D. J Oral Maxillofac Surg 2014.
DE WATER ET AL 143 Discussion In this study, manual airway segmentation was used as a gold standard. This method also was used by Aboudara et al 12 who found an error of 0% to 5% compared with airway phantom volumes. Nout et al 6 found high interobserver and intraobserver correlations for manual segmentation. Dolphin 3D has been used in studies on airway changes. 7,8 According to these studies, Dolphin 3D airway analysis is an adequate tool to measure airway volumes. The accuracy and reliability of automatic airway segmentation by Dolphin 3D were tested by El and Palomo. 11 They compared 3 commercially available DICOM viewers. According to their results, Dolphin 3D airway measurements were reliable, but not accurate. Compared with manual segmentation, Dolphin 3D produced higher values for measured airway volumes. However, all these studies used scans that were acquired at similar settings. When measuring and comparing scans with different characteristics, Dolphin 3D seemed to be less accurate according to the present results. The difference of 36% was substantial, especially compared with a volume increase after LFIII advancement that varied from 27% (compartment A) to 37% (compartment B). 6 Postoperative measurements in Dolphin 3D were more accurate than preoperative measurements. This suggests an inaccuracy in Dolphin 3D measurements that has a relatively greater impact when measuring smaller volumes. An important difference between the present and other studies is that the observers who measured with Dolphin 3D were not the same as those who measured the scans using a manual segmentation technique. Another factor that may contribute to differences in measured volumes between the 2 segmentation methods is the fact that Dolphin 3D relies on manual airway delineation in a maximum of 3 planes. This is particularly problematic when excluding sinuses from the airway. The effect of excluding inactive airway space on interobserver and intraobserver variabilities has already been mentioned for manual segmentation. For semiautomatic segmentation, the effect of air-holding cavity exclusion on outcome measurements may be greater, because delineation is being performed in 3 planes instead of in numerous slices. It is impossible to exclude all paranasal sinuses in only these 3 planes without also excluding parts of the functional airway. Thus, the observer has to choose whether to include a nonfunctional airway or exclude a functional airway. This might be a cause for low measurement accuracy and low interobserver reliability. Standardization of the measurement technique and uniform instructions for observers may lead to greater accuracy and reliability. The user s control of the rendered volume is limited by the threshold scale in Dolphin 3D, because there are no absolute units. The threshold number should be adjusted to the scan characteristics to obtain the same densities as the airway. Alves et al 10 aimed to determine the most accurate threshold value for airway volume quantification. According to their results, a threshold value between 70 and 75 provided results that corresponded with their gold standard, with 73 as the most accurate threshold value. In their study, all scans were acquired at similar settings. The threshold values were determined only for images taken at 120 kv, 5 ma, and a 40-second scanning time and thus are not generally applicable. For statistical analysis of the data, a linear mixed effects model was applied. This model considered the effects of different observers, repeated measurements by the same observer, and compartments on the measurements and compared all results with the standard, in this case the manual segmentation method. Future studies should focus on a standardization of measurement techniques in semiautomatic segmentation. Further research could concentrate on possibilities to exclude aerodynamically nonfunctional air-holding cavities from the semiautomatically segmented volume. In conclusion, the Dolphin 3D airway analysis tool is not accurate enough to measure airway volumes, because there was a significant absolute difference of up to 42% between the Dolphin 3D and the manual segmentation results, which exceeded the effect of the LFIII osteotomy on airway volume. Moreover, the Dolphin 3D measurements do not have sufficient reliability, because there were significant effects of observer, compartments measured, and time of measurement (pre- or postoperative) on the measurement results. The Dolphin 3D airway measurement software tool may still be useful when applied to CT scans acquired at similar settings and for measuring larger airway volumes. References 1. Schafer ME: Upper airway obstruction and sleep disorders in children with craniofacial anomalies. Clin Plast Surg 9:555, 1982 2. Cohen MM Jr, Kreiborg S: Upper and lower airway compromise in the Apert syndrome. Am J Med Genet 44:90, 1992 3. Flores RL, Shetye PR, Zeitler D, et al: Airway changes following Le Fort III distraction osteogenesis for syndromic craniosynostosis: A clinical and cephalometric study. Plast Reconstr Surg 124: 590, 2009 4. Bannink N, Nout E, Wolvius EB, et al: Obstructive sleep apnea in children with syndromic craniosynostosis: Long-term respiratory outcome of midface advancement. Int J Oral Maxillofac Surg 39:115, 2010 5. Nout E, Bannink N, Koudstaal MJ, et al: Upper airway changes in syndromic craniosynostosis patients following midface or monobloc advancement: Correlation between volume changes and respiratory outcome. J Craniomaxillofac Surg 40:209, 2012
144 MEASURING UPPER AIRWAY VOLUME USING DOLPHIN 3D 6. Nout E, Bouw FP, Veenland JF, et al: Three-dimensional airway changes after Le Fort III advancement in syndromic craniosynostosis patients. Plast Reconstr Surg 126:564, 2010 7. Smith T, Ghoneima A, Stewart K, et al: Three-dimensional computed tomography analysis of airway volume changes after rapid maxillary expansion. Am J Orthod Dentofacial Orthop 141:618, 2012 8. Iannetti G, Polimeni A, Pagnoni M, et al: Upper airway volume after Le Fort III advancement in subjects with craniofacial malformation. J Craniofac Surg 22:351, 2011 9. Pangrazio-Kulbersh V, Wine P, Haughey M, et al: Cone beam computed tomography evaluation of changes in the naso-maxillary complex associated with two types of maxillary expanders. Angle Orthod 82:448, 2012 10. Alves MJ, Baratieri C, Mattos CT, et al: Is the airway volume being correctly analyzed? Am J Orthod Dentofacial Orthop 141:657, 2012 11. El H, Palomo JM: Measuring the airway in 3 dimensions: A reliability and accuracy study. Am J Orthod Dentofacial Orthop 137(suppl):S50e1, 2010 12. Aboudara C, Nielsen I, Huang JC, et al: Comparison of airway space with conventional lateral headfilms and 3-dimensional reconstruction from cone-beam computed tomography. Am J Orthod Dentofacial Orthop 135:468, 2009