Optical coherence tomography (OCT) is a noninvasive

Transcription

1 Coparison of Optical Coherence Toography in Diabetic Macular Edea, with and without Reading Center Manual Grading fro a Clinical Trials Perspective Ada R. Glassan, 1 Roy W. Beck, 1 David J. Browning, 2 Ronald P. Danis, 3 and Craig Kollan, 1 for the Diabetic Retinopathy Clinical Research Network Study Group 4 PURPOSE. To analyze the value of reading center error correction in autoated optical coherence toography (OCT; Stratus; Carl Zeiss Meditec, Inc., Dublin, CA) retinal thickness easureents in eyes with diabetic acular edea (DME). METHODS. OCT scans (n 6522) obtained in seven Diabetic Retinopathy Clinical Research Network (DRCR.net) studies were analyzed. The reading center evaluated whether the autoated center point easureent appeared correct, and when it did not, easured it anually with calipers. Center point standard deviation (SD) as a percentage of thickness, center point thickness, signal strength, and analysis confidence were evaluated for their association with an autoated easureent error (anual easureent needed and exceeded 12% of autoated thickness). Curves were constructed for each factor by plotting the error rate against the proportion of scans sent to the reading center. The ipact of easureent error on interpretation of clinical trial results and statistical power was also assessed. RESULTS. SD was the best predictor of an autoated easureent error. The other three variables did not augent the ability to predict an error using SD alone. Based on SD, an error rate of 5% or less could be achieved by sending only 33% of scans to the reading center (those with an SD 5%). Correcting autoated errors had no appreciable effect on the interpretation of results fro a copleted randoized trial and had little ipact on a trial s statistical power. CONCLUSIONS. In DME clinical trials, the error involved with using autoated Stratus OCT center point easureents is sufficiently sall that results are not likely to be affected if scans are not routinely sent to a reading center, provided adequate quality control easures are in place. (Invest Ophthalol Vis Sci. 2009;50: ) DOI: /iovs Fro the 1 Jaeb Center for Health Research, Tapa, Florida; 2 Charlotte Eye Ear Nose and Throat Associates, PA, Charlotte, North Carolina; and the 3 Departent of Ophthalology and Visual Sciences, University of Wisconsin, Madison, Wisconsin. 4 A current list of the Diabetic Retinopathy Clinical Research Network is available at Supported through a cooperative agreeent fro the National Eye Institute and the National Institute of Diabetes and Digestive and Kidney Diseases EY14231, EY14269, EY Subitted for publication February 14, 2008; revised May 8, 2008; accepted Noveber 20, Disclosure: A.R. Glassan, None; R.W. Beck, None; D.J. Browning, None; R.P. Danis, None; C. Kollan, None The publication costs of this article were defrayed in part by page charge payent. This article ust therefore be arked advertiseent in accordance with 18 U.S.C solely to indicate this fact. No reprints will be available. Corresponding author: Ada R. Glassan, c/o Jaeb Center for Health Research, Aberly Drive, Suite 350, Tapa, FL 33647; drcrnetx3@jaeb.org. Optical coherence toography (OCT) is a noninvasive ethod for easuring the thickness of the central retina. It has becoe a standard tool in the anageent of patients with diabetic acular edea (DME). The Stratus OCT (Carl Zeiss Meditec, Dublin, CA) provides autoated easureents of ean retinal thickness of the center point, central subfield, and each of four inner and four outer subfields (in icroeters), and total acular volue (cubic illieters). The OCT software identifies the center point as the intersection of six radial lines and reports the center point thickness (CPT) as the average of the six easureents of thickness at the center point. An SD of the six easureents is then coputed. In the DME studies conducted by the Diabetic Retinopathy Clinical Research Network (DRCR.net), OCT scans have been sent to a central reading center for evaluation of quality, orphology, and accuracy of autoated thickness easureents. DRCR.net has adopted the use of the central subfield for analysis of central retinal thickness data rather than the center point because of reduced variability in the easureent of the forer copared with the latter. 1,2 However, only CPT can be easily anually easured when the autoated easureent is incorrect. To liit issing data for scans with incorrect autoated easureents, the center point anual easureent can be used to copute a value for the central subfield, since the two correlate highly. 1,3 Since ost scans in DME have accurate autoated easureents, we questioned the cost effectiveness of sending scans to a reading center to correct errors in the autoated easureents copared with just using the autoated easureents for analysis of clinical trial data. To answer our question, we used data collected in DRCR.net protocols. METHODS The study included 6522 OCT scans at baseline and follow-up fro eyes obtained in seven DRCR.net studies of DME. OCT operators were certified by the DRCR.net Reading Center at the University of Wisconsin, Madison. Soe protocols required that central foveal edea be present in the study eyes whereas others did not. In each study, Stratus OCT was obtained after pupil dilation. Scans were 6 in length and included the six-radial-line pattern (128 scan resolution, fast acular scan option with Stratus OCT) for quantitative easures and the cross hair pattern (6 12 to 9 3 o clock) at 512 scan resolution for qualitative assessent of retinal orphology. 1 OCT operators were instructed that if the SD of CPT was greater than or equal to 10% of the CPT or if signal strength exceeded 5, the scans should be repeated and subitted only if the OCT technician believed that the scans were of adequate quality or better quality easures were not obtainable after repeat scans. Scans were sent to the DRCR.net Reading Center for evaluation. Scans with center point easureents graded as inaccurate by evaluators at the reading center were anually reeasured fro prints of the retina ap scans using digital calipers. The center point was anually easured for all scans with a center point SD 10.5%, which Investigative Ophthalology & Visual Science, February 2009, Vol. 50, No Copyright Association for Research in Vision and Ophthalology

2 IOVS, February 2009, Vol. 50, No. 2 Corrected Versus Autoated OCT Values in DME Trials 561 was coputed as the SD of the six autoated center point easureents divided by the CPT (average of the six easureents). This cutoff was selected based on the reading center s prior experience in evaluating a rando saple of iages in which anual easureent of the center point was necessary in all iages with center point SD 10.5%. For all other scans, if decentration (the center of the scan not aligned with the center of the acula) or boundary line artifacts (the instruent software incorrectly identified the internal and external liits of the retina) were identified, the CPT was anually easured unless poor scan quality precluded a easureent. For analysis, each OCT scan was categorized as having a correct or incorrect center point easureent. The easureent was considered to be correct if either the reading center deterined that a anual easureent was not needed or a anual easureent was deeed necessary but the autoated easureent was within 12% of the anual easureent. Twelve percent was selected because it approxiates the 95% confidence interval on an OCT easureent. 2 An autoated easureent error was defined as an autoated easureent that differed fro a anual easureent by 12% or ore. Autoated easureents of scans in which poor quality precluded a anual easureent were also considered to be errors. Four factors were evaluated as possible predictors of an incorrect autoated center point easureent error: SD of the center point as a percentage of the CPT (SD), signal strength, the confidence of the analysis as reported by the instruent software, and the autoated CPT. All four factors are present on or can be calculated fro the OCT 3 printout. The correlation of each factor with the others was assessed with Spearan correlation coefficients. The association of each factor with the center point autoated easureent error rate was assessed in univariate and ultivariate logistic regression odels. A cost benefit type of analysis was used to evaluate further the relative iportance of each factor in deterining whether autoated errors were present. The cost benefit analyses were perfored by calculating the nuber of scans in which an incorrect center point easureent was detected (benefit) as a function of the proportion of iages to be sent to the reading center for anual assessent (cost). Parallel analyses using receiver operating curve ethods produced siilar conclusions (data not shown). Additional cost benefit curves were constructed for SD stratified by center point thickness, where separate curves were generated for CPT 250, CPT 250 to 299, CPT 300 to 399, CPT 400 to 499, and CPT 500. To evaluate the ipact of reading center evaluation of the center point autoated easureents in a clinical trial, results fro a DRCR-.net protocol coparing two laser techniques for DME were used in which the results obtained with reading center evaluation of all scans were copared with the results that would have been obtained using just the autoated easureents. 4 For this analysis, change in CPT fro baseline to 12 onths was copared between treatent groups using repeated easures least-squares regression odels adjusted for baseline thickness and accounting for the correlated data fro subjects with two study eyes. In parallel fashion, the treatent groups were copared for change in central subfield thickness, with and without replacing invalid autoated central subfield values with values iputed fro anually easured CPT. 1 The ipact on a trial s saple size of using solely autoated center point easureents versus having a reading center evaluate the scans was assessed using the fact that saple size is proportional to the overall variance. The overall variance was calculated as the betweensubject variation plus easureent error. The percentage reduction in variance (and therefore saple size) was calculated assuing that anual easureent would give the correct thickness value (i.e., zero easureent error for scans sent to the reading center). Saple size for a hypothetical trial with 90% power, type 1 error of 5%, assuing an effect size of 50 with an SD of 150 was then coputed, varying the nuber of iages sent to the reading center. Statistical analyses were perfored with coercial software (SAS software, ver. 9.1; SAS Institute, Cary, NC). FIGURE 1. Relationship of difference in autoatic and anually graded CPT versus anually graded CPT (n 1199).

3 562 Glassan et al. IOVS, February 2009, Vol. 50, No. 2 TABLE 1. Frequency of Manual Grading for Factors Potentially Predictive of Center Point Errors* RESULTS Factor Of the 6522 OCT scans included in the analysis, 1243 (19%) required anual easureent of the center point as deterined by the reading center. For 44 (4%) of these iages, the poor quality of the scan precluded a anual easureent, and n Error Rate* n (%) Overall (14) Standard deviation of center point 0% 67 6 (9) 0% 1% (4) 1% 2% (5) 2% 3% (8) 3% 4% (8) 4% 5% (10) 5% 6% (9) 6% 7% (16) 7% 8% (17) 8% 9% (17) 9% 10% (21) 11% (61) Analysis confidence low Absent (11) Present (28) Signal strength (67) (30) (25) (17) (16) (14) (13) (12) (12) (9) (4) Center point thickness (A) (13) (B) (23) (C) (23) (D) (13) (E) (7) (F) (4) (G) (7) (H) (9) * A scan was considered to have an error when the autoated center point easureent differed fro a anual easureent by ore than 12% or the autoated easureent was deterined to need a anual easureent, but the iage quality precluded anual easureent. the autoated easureent was assued to be inaccurate. For the 1199 anually graded scans where a anual easureent was attainable, the edian difference between the autoated center point value and the anually graded value was 34 (interquartile range 2 to 65 ) and the edian absolute value of the difference was 48 (interquartile range 23 to 82 ). The autoated easureent was ore frequently greater when the retina was thinner and the anual easureent was ore frequently greater when the retina was thicker (Fig. 1). For 362 (30%) of the 1199 anually graded iages, the autoated easureent was within 12% of the anual easureent. Thus, for 5641 (86%) of the 6522 scans, the autoated center point easureent was either considered to be correct or was within 12% of the anual easureent. In 881 (14%) scans the autoated easureent was inaccurate by 12% or ore, or the iages were of a quality that precluded anual easureent. Table 1 shows the frequency of an autoated easureent error across the range of values of the four evaluated factors (SD of the center point, signal strength, analysis confidence, and CPT). The correlation between these factors was weak, ranging in absolute value fro 0.04 to 0.30 (Suppleentary Table S1, online at 560/DC1). Each of the four factors had a significant association with the autoated easureent error rate. The error rate was higher with a larger SD, when the analysis confidence variable was low, when the signal strength was low, and when CPT was low. After adjustent for the effect of the other factors, all were still significant except for the analysis confidence variable (Table 2). However, the SD of the center point explained uch ore of the variance than did the other factors. Adding the other factors to a odel with SD did not add to the predictive ability. Cost benefit curves were constructed to show the error rate relative to the proportion of scans that would need to be graded by a reading center. It can be seen that for a given percentage of scans sent to the reading center, the error rate was lower when using the SD as the flag to send than when using the other three factors (Fig. 2), and there was no additive value for these factors. If no scans had been sent to a reading center, 14% of the scans would have been considered to have an autoated easureent error and the ean center point value would have been 5 higher and the ean central subfield value 4 higher than the ean using just the autoated easureents. Retinal volue ean would be using autoated easureents and with inaccurate autoated easureents excluded. Additional cost benefit curves were constructed for SD stratified into five levels of autoated CPT: 250, 250 to 299, 300 to 399, 400 to 499, and 500. Figure 3 shows that even after adjusting for SD, the error rates were lower in eyes with thicker aculas. If no scans were sent to the reading center, 15%, 23%, 13%, 7%, and 6% of the iages TABLE 2. Potentially Predictive Factors of the Need for Manual Grade in Logistic Regression Models Univariate Model of Predictive Factor Multivariate Model Factor Estiate [OR] Estiate [OR] (95% CI) P R 2 (95% CI) P Final Model Cuulative* R 2 Ratio of SD to CPT (per 1% increase) 1.20 ( ) % 1.18 ( ) % OCT CPT (per 100 increase) 0.82 ( ) % 0.80 ( ) % Signal strength (per 1 unit increase) analysis 0.85 ( ) % 0.93 ( ) % Confidence (low vs. not low) 3.24 ( ) % 1.17 ( ) % * Cuulative r 2 obtained fro forward selection regression, calculating the r 2 after adding factors to the logistic regression odel. The nubers in this colun represent the r 2 fro four odels in which the factor in the row has been cuulatively added to the odel.

4 IOVS, February 2009, Vol. 50, No. 2 Corrected Versus Autoated OCT Values in DME Trials 563 FIGURE 2. Relationship of error rate to proportion of scans sent to the reading center, based on predictive factors of need for anual grading. The error rate is defined as the percentage of scans requiring anual grading and having an autoatic easureent differing fro the anual easureent by 12% that were not sent to the reading center. would be considered to have an autoated easureent error for the five thickness subgroups, respectively. As seen in Figure 4, based on SD, a 5% or less error rate would be achieved by sending only 33% of scans to the reading center (those with a center point SD 5%). The center point ean using the autoated easureents only (with none sent to the reading center) would be 2 higher than the ean when sending the 33% of scans with a center point SD 5%.A5%or less error rate could be achieved by sending 37%, 69%, 34%, 4%, and 1% of scans to the reading center for iages with autoated thicknesses of 250, 250 to 299, 300 to 399, 400 to 499, and 500, respectively. The distance between SD levels on each curve differs based on the CPT stratu. To illustrate the effect in a randoized trial of using only the autoated easureents with no scans evaluated for anual easureent by a reading center, data were reanalyzed fro a DRCR.net randoized trial coparing two laser regiens for DME (Table 3). 4 In the analysis using the autoated easureents, the treatent group difference in ean change ( SE) in central subfield thickness fro baseline to 1 year was (P 0.02) copared with (P 0.01) using the corrected dataset. Siilarly, the treatent group difference in change in CPT fro baseline to 1 year was (P 0.04) fro the autoated dataset copared with (P 0.01), using the corrected dataset. Data fro the DRCR.net studies can be used to assess the ipact on projected saple size in a hypothetical trial for varying proportion of scans sent to the reading center. Figure 5 deonstrates the projected saple size for a hypothetic power calculation on the difference in central subfield thickness having 90% power with an effect size of 50 assuing an SD of 150. It can be seen that the proportion of scans sent to the reading center had little ipact on the projected saple size, estiating 402 subjects if no iages were sent to the reading center and 378 subjects if all the iages were sent, an increase of only 6%. DISCUSSION In this study, we have shown that in a large randoized trial, there is little benefit derived fro sending all scans to a reading center to evaluate whether the autoated easureent of CPT is correct in eyes with DME. When all scans were sent, approxiately 86% had accurate autoated easureents. Manually easuring the CPT for the other 14% enhanced overall accuracy, but only slightly. In doing this, the ean difference in CPT differed by only 5 and the ean difference in central subfield thickness differed by only 4, copared with having all scans reviewed by a reading center. Siilar results are obtained using retinal volue as the outcoe easures. The SD of the center point is ore predictive of the probability of an error in the autoated easureents than is signal strength or the confidence assessent. Both of these easures

5 564 Glassan et al. IOVS, February 2009, Vol. 50, No. 2 FIGURE 3. Relationship of error rate by proportion of scans sent to the reading center, based on standard deviation of the center point stratified by autoated CPT. Each point on the curve denotes a separate threshold for sending scans to the reading center based on the SD of the center point. The error rate is defined as the percentage of scans requiring anual grading and having an autoatic easureent differing fro the anual easureent by 12% that were not sent to the reading center. are provided by the Stratus OCT software as easures of scan quality but neither added uch to the center point SD. Although autoated errors are ore likely for eyes with center involved DME when the fovea is thinner, retinal thickness also added little to the predictive ability of the center point SD alone. If there is a desire to reduce the error rate fro the expected 14%, a portion of the scans could be sent to the reading center. Our results indicate that the center point SD is the best easure to use in this regard and an error rate of 5% can be achieved by sending to a reading center scans with a center point SD exceeding 5%. However, this has no appreciable ipact on the error present in the ean change in CPT. Evaluation of OCT scan quality by the clinical site OCT technicians and investigators would also increase data accuracy. The SD of the center point ay still be less than 5%, even if the scan has decentration and/or boundary line errors. By repeating erroneous scans until a satisfactory quality scan is obtained for subission, operator-error issues can be corrected. Specific training regarding OCT quality assessent is required for optial clinical site perforance. Despite this caveat, it is unlikely that the frequency of erroneous scans could be reduced significantly in this study considering in the DRCR.net protocols, OCT technicians were instructed to repeat the OCT easureent if the SD was 10% or the signal strength was 6 (i.e., iages of potentially poor quality). There is a tradeoff for a clinical trial in reducing or eliinating the nuber of scans sent to a reading center. Sending scans to the reading center and correcting errors in the center point easureent will reduce the required saple size for a given statistical power. Our data suggest that the increase in saple size is likely to be sall, approxiately 6%, in a study of center-involved DME powered to assess treatent group differences in the change in central subfield thickness if no scans were sent to a reading center copared with having all scans sent to a reading center. In large trials, even a 6% increase in saple size could lead to a significant increase in study cost. Therefore, in planning a trial, the cost per additional subject recruited versus the cost to have OCT scans sent to a reading center for grading ust be considered to deterine the ost cost-effective strategy for the trial. When the retina is thicker, the likelihood of a easureent error decreases. Thus, the yield is lower in sending scans to a reading center for evaluation. This is probably due to a detection bias, where a decentered scan is ore likely to be identified if the retina has a orphologically distinct center (such as a foveal depression) which is ore likely to be observed in a thinner retina copared to a thickened retina with disorganized orphology. In deterining whether to send scans to a reading center for a protocol, there ay be less reason to send the scans for evaluation in a study in which eligibility requires substantial central retinal thickening than there would be in a protocol in which central thickening was required to be absent and the objective of the intervention was the prevention of the developent of DME. In the latter circustance, the isclassification rate of outcoe due to autoated errors could necessitate that all scans be evaluated by a reading center.

6 IOVS, February 2009, Vol. 50, No. 2 Corrected Versus Autoated OCT Values in DME Trials 565 FIGURE 4. Relationship of error rate by proportion of scans sent to the reading center, based on the SD of the center point. Each point on the curve denotes a separate threshold for sending scans to the reading center based on the SD of the center point. For exaple, using SD 5% as the criterion results in sending approxiately 33% of iages to the reading center and a corresponding error rate of 5%. The error rate is defined as the percentage of scans requiring anual grading and having an autoatic easureent differing fro the anual easureent by 12% that were not sent to the reading center. DRCR.net is using the results of this study to deterine for each protocol which scans should be sent to the reading center for evaluation and anual easureent of the center point when indicated. For protocols in which OCT-easured retinal thickness is not the priary outcoe easure and orphology grading is not required, in general either no scans are sent to the reading center or only scans exceeding a specified center point SD value or less than a specified CPT are sent to the reading center. It is iportant to recognize that these results apply to cases of DME and not to age-related acular degeneration or other acular conditions in which the autoated error rate can be uch higher. 5 8 In addition, these results apply to the software available with the Stratus OCT (Carl Zeiss Meditec, Inc.) and not necessarily to software algoriths of other OCT devices. Nevertheless, the principles followed in this analysis ay be generalized to other devices and all conditions with regard to deterining the ost efficient strategy for having a reading center evaluate retinal thickening, whether by OCT or other technology, in a clinical trial. Finally, frequency doain OCT with registration capability to fundus landarks is expected to reduce decentration errors, and better retina sectioning algoriths are expected to reduce boundary line errors in the near future. It appears that for Stratus OCT, anual grading adds little easureent accuracy to studies of DME. Manual grading of thickness easureents of frequency doain OCT iages in clinical trials of the future ay be even less iportant. Reading centers provide a echanis for asked assessent TABLE 3. Coparison of Thickness Data fro a DRCR.net Randoized Trial Baseline 1 Year Change Modified- ETDRS MMG Modified- ETDRS MMG Modified- ETDRS MMG P Center point easureent Autoatic easureents* Corrected dataset Central subfield Autoatic easureents* Corrected dataset Data are analyzed using the corrected dataset with the dataset reanalyzed using only the autoated easureents with no scans evaluated for anual easureent by the reading center. All easureents are in icroeters. * Analyzed using dataset with thickness fro the autoated easureent fro OCT achine with no scans evaluated for anual easureent by the reading center. Analyzed using the corrected dataset, which is the autoated values when the autoated values are fro a scan not needing anual grading and the anually reeasured values fro a scan in which anual grading was necessary.

7 566 Glassan et al. IOVS, February 2009, Vol. 50, No. 2 FIGURE 5. Required saple size to achieve 90% power to detect treatent effect of 50 with SD 150, varied by nuber of iages sent to the reading center. of orphology and will be iportant for quality control in clinical trials eploying only the nueric data fro the OCT output. With the coon scarcity of resources and the need to reallocate available support to other iportant aspects of clinical trials, utilization of reading center analyses where necessary, and avoidance where not of ajor ipact will perit cost savings without substantially sacrificing data accuracy. References 1. Browning DJ, Glassan AR, Aiello LP, et al. Optical coherence toography easureents and analysis ethods in optical coherence toography studies of diabetic acular edea. Ophthalology. 2008;115(8): , 1371.e1. 2. Diabetic Retinopathy Clinical Research Network. Reproducibility of acular thickness and volue using Zeiss optical coherence toography in patients with diabetic acular edea. Ophthalology. 2007;114: Diabetic Retinopathy Clinical Research Network. Relationship between optical coherence toography-easured central retinal thickness and visual acuity in diabetic acular edea. Ophthalology. 2007;114: Writing Coittee for the Diabetic Retinopathy Clinical Research Network. Coparison of the odified early treatent diabetic retinopathy study and ild acular grid laser photocoagulation strategies for diabetic acular edea. Arch Ophthalol. 2007;125: Sadda SR, Wu Z, Walsh AC, et al. Errors in retinal thickness easureents obtained by optical coherence toography. Ophthalology. 2006;113: Ray R, Stinnett SS, Jaffe GJ. Evaluation of iage artifact produced by optical coherence toography of retinal pathology. A J Ophthalol. 2005;139: Sadda SR, Joeres S, Wu Z, et al. Error correction and quantitative subanalysis of optical coherence toography data using coputerassisted grading. Invest Ophthalol Vis Sci. 2007;48: Leung CK, Chan WM, Chong KK, et al. Alignent artifacts in optical coherence toography analyzed iages. Ophthalology. 2007; 114: