Interobserver Reliability of the Teller Acuity Card Procedure in Pediatric Patients

Transcription

1 Interobserver Reliability of the Teller Acuity Card Procedure in Pediatric Patients Lisa M. Getz,* Velma Dobson,*-\- Beatriz Luna,^ and Clay Mashj Purpose. To compare interobserver agreement for Teller Acuity Card estimates of grating acuity between children with ocular or neurologic abnormalities, or both, and age-matched healthy preterm children. Methods. Subjects were 57 children, 3 to 38 months of age, who were referred for visual assessment because of diagnosed or suspected visual impairment (clinical group), and 57 healthy preterm children with no known visual or neurologic abnormalities (control group), each of whom was matched to a clinical subject, based on corrected age at the time of testing, and type of testing (monocular or binocular). Each child's grating acuity was tested by two independent observers. Results. Interobserver agreement of octave or better was found in 9% of the monocular and 96% of the binocular clinical test-retest comparisons and in 95% of the monocular and 96% of the binocular control comparisons. For estimates of interocular acuity difference, interobserver agreement of octave or better was found in 88% of clinical subjects and 88% of control subjects. Average test time was significantly longer in the clinical group (4. minutes [SD =.9] for monocular and 3.6 minutes [SD =.9] for binocular tests) than in the control group (2.5 minutes [SD = 0.9] for monocular and 2.4 minutes [SD = 0.6] for binocular tests), suggesting that children in the clinical group were more difficult to test. Conclusions. Teller Acuity Card testing conducted by experienced testers is as reliable in children with mild to severe ocular or neurologic abnormalities as it is in healthy children, even though children with abnormalities may be more difficult to test. Invest Ophthalmol VisSci. 996;37: L he acuity card procedure was developed to allow rapid assessment of grating acuity of infants and young children in clinical settings. In the procedure, the patient is shown a series of gratings composed of blackand-white stripes on a gray background. A trained tester observes the patient's eye and head movement responses to the gratings and, based on these responses, makes a judgment as to the finest grating (smallest stripes) the patient can distinguish from the gray background. Because acuity estimates obtained with the acuity From tlie Departments of * Psychiatry and f Psychology, University of Pittslrurgh, Pennsylvania. Presented in part at the AliVO Annual Meeting, May 994, Sarasota, Florida. Suppoited by National Institutes of Health grant EY Submitted for publication February 27, 995; revised July 9, 995; accepted September, 995. Proprietary interest category: C. Refmnl requests: Velma Dobson, Department of Ophthalmology, University of Arizona School of Medicine, 80 N. Campbell, Tucson, AZ card procedure are dependent on the subjective judgment of the tester and because previous studies have shown small but reliable differences among testers in their tendency to obtain higher or lower acuity values, 2 " 5 it is important to document the interobserver test-retest reliability of the acuity card procedure. Previous reports'~ 4>6 ~ 4 have indicated that the interobserver reliability of acuity card testing is similar to that of more scientifically rigorous, laboratory-based, forced-choice preferential looking procedures. 5 " 8 However, studies in which the Teller Acuity Card (TAC) procedure was used to test children with ocular or neurologic abnormalities 9 " l2cl8 generally showed poorer interobserver agreement than was found in studies of the TAC procedure in which subjects were healthy, normal infants and children. ' 4 ' ' 7 The purpose of the current research was to examine, within a single study, interobserver test-retest reliability of healthy children and children with ocular 80 Investigative Ophthalmology & Visual Science, January 996, Vol. 37, No. Copyright Association for Research in Vision and Ophthalmology

2 Reliability of Teller Acuity Card in Patients 8 or neurologic abnormalities. The goal was to determine whether the lower interobserver reliability seen in studies in which abnormal subjects were tested was caused by differences in the reliability of testers in working with normal versus abnormal children or to procedural differences across studies. METHODS Subjects Subjects were 4 children between 3 and 38 months of age (mean = 3.8 months, SD = 9.2). Half of these children (the clinical group) were referred for visual acuity assessment because of diagnosed or suspected visual impairment. Included in the clinical group were children with unilateral or bilateral cataract or aphakia (n = 8), cortical visual impairment andor brain abnormalities, including encephalopathy, seizures, hydrocephalus, and brain trauma (n = 8), optic nerve abnormality (n = 4), chromosomal abnormality (n = 4), albinism (n = 2), nystagmus (n - 2), strabismus (n = 2), coloboma (n = 2), and other abnormalities (n = 5), including macular scar, hemangioma, glaucoma, iridectomy, retinal reattachment (one case of each). Forty-six children in the clinical group were full-term, and were preterm (gestational age ^36 weeks). The remaining 57 children (the control group) were healthy children whose gestational ages were ^36 weeks and who were participants in a follow-up study of infants treated in a neonatal intensive care unit. Healthy preterm children were used as control subjects because of their availability and because of difficulties in recruiting full-term subjects in our research setting. Previous research has indicated that interobserver reliability in healthy preterm children is similar to that in full-term children. 23 Children in the control group had no perinatal complications known to affect visual development, e.g., retinopathy of prematurity, periventricular leukomalacia, severe intraventricular hemorrhage (grade HI or IV), encephalopathy, or seizures, and no known visual abnormalities, e.g. strabismus or nystagmus, as determined by an eye examination by a pediatric ophthalmologist (26 subjects) or parental report (3 subjects). Each child in the control group was matched to a clinical subject, based on type of testing (monocular or binocular), and corrected age at the time of testing (±3 weeks for children younger than 6 months, ±4 weeks for children 6 to 2 months of age, and ±8 weeks for children older than year of age). Apparatus and Procedure The tenets of the Declaration of Helsinki were followed, and the study was approved by the human subjects committees of the University of Pittsburgh and Magee-Women's Hospital. Informed written consent was obtained from the child's parent before testing. The apparatus consisted of a set of 7 TACs and a TAC stage through which the cards are presented (Vistech, Dayton, OH). Fifteen cards contained a 2.5 X 2.5-cm patch of black-and-white grating, one was a blank gray card, and one, the Low Vision card, contained a 25 X 23-cm patch of 2.2-cm wide stripes. In 98% of the tests, subjects were tested by one of three observers who had more than three years of experience using the TAC procedure. Two percent of tests were conducted by two additional observers, each of whom had more than one year of experience with TAC testing. In 2 (36.8%) of the 57 matched pairs, the same two observers tested both the clinical and the control member of the pair. This was not possible for the remaining pairs because of scheduling constraints. After informed consent was obtained from the parent, the child was seated on the parent's lap or alone in front of the stage. Initially, testing was attempted at 38 cm for children younger than 7 months of age and at 55 cm for those between 7 months and 3 years. Children with low vision often were tested without the stage, at a nearer test distance (9 or 9.5 cm). Children with nystagmus that interfered with the observer's ability to judge whether the child was looking to the left or to the right side of the card were tested outside the stage, using vertical presentation of the cards. 9 Vertical presentation of the cards allowed the observer to use up-down eye and head movements, rather than horizontal eye and head movements, as an indicator of whether the child could see the gratings on each acuity card. 9 ' For each child in the clinical group, grating acuity was assessed monocularly or binocularly, according to the request of the referring physician and the level of cooperation of the child for monocular testing. The matched control subject was tested in the same manner as the clinical subject to whom he or she was matched. When monocular testing was conducted, the first observer tested one eye. This was followed by the second observer's test of that same eye. The second observer then tested the other eye, after which the first observer tested that eye. Observers were masked to each other's acuity results until all testing had been completed. The testing procedure used was that recommended in the TAC manual. 20 Testing begins with a specific "start card," based on a child's age, which is followed by presentation of cards containing gratings that increase in spatial frequency in approximately half-octave steps. The observer is masked to the location, but not the spatial frequency, of the grating on

3 82 Investigative Ophthalmology 8c Visual Science, January 996, Vol. 37, No. each card. Acuity is estimated as the finest grating that the observer judges the child can see, based on the child's looking behavior in response to several presentations of each card. Test time was measured from the time the the first card was presented until the time that observer arrived at an acuity estimate. FIRST EYE SECOND EYE Data Analysis Acuity scores were converted to log values. Interobserver differences for the eye tested first, the eye tested second, and for binocular acuity results were calculated to the nearest octave (the approximate difference in spatial frequency between adjacent acuity cards). For children tested monocularly, interocular acuity difference (IAD) was calculated as the signed difference, to the nearest octave, between the acuity of the right eye and the acuity of the left eye, converted to octaves. Interobserver difference in IAD was calculated as the difference, in octaves, between the IADs obtained by the two observers, with the sign of the difference maintained in the calculation ( if the acuity of the right eye was better than the acuity of the left eye; -I- if the left eye had the better acuity). For example, if the first observer found octave better acuity in the right eye than the left eye, and the second observer found octave better acuity in the left eye than the right eye, then the interobserver difference in IAD was octave. RESULTS Figure compares interobserver agreement for each matched pair of clinical and control subjects. Separate plots show results for subjects tested monocularly (the eye tested first, the eye tested second, and interocular acuity difference) and for subjects tested binocularly. Interocular acuity difference data are not available for all subjects tested monocularly because not all subjects tested monocularly completed testing of both eyes. Across all four plots, 38 points lie on the unity line, indicating equal interobserver agreement for the clinical and control member of a pair. In 37 pairs (points above the line), a smaller interobserver difference was found for the control than for the clinical member of the pair, and in 32 pairs (points below the line), a smaller interobserver difference was found for the clinical member of the pair. Chi-square analysis indicated no difference in the proportion of points lying above the unity line and the proportion of points lying below the unity line, across the four test conditions. Chi-square analysis also indicated that there was no difference in the proportion of data points lying above versus lying below the line, for pairs in which the clinical member had an average acuity score more than octave below that of the matched control subject < o i * - 76 V V IAD V V v - BINOCULAR CONTROL INTEROBSERVER DIFFERENCE (OCTAVES) FIGURE l. Comparison of interobserver differences for matched pairs of subjects in the clinical and control groups, for the eye tested first (n = 32), the eye tested second (n = 26), interocular acuity difference (n = 24), and binocular acuity (n = 25). Points lying on the diagonal line indicate that interobserver agreement was identical for the clinical and the control members of a pair. Points lying above the line are from pairs in which agreement was better in tests of the control than of the clinical subject, and points below the line are from pairs in which interobserver agreement was better in tests of the clinical than of the control subject. Filled symbols indicate pairs in which the clinical member had a significant acuity deficit, and open symbols indicate pairs in which the acuity of the clinical member of the pair was no more dian octave worse than that of the control member of the pair. (filled symbols), in comparison to pairs in which the clinical and control members had similar acuity scores (open symbols). These data suggest that interobserver reliability is as good for subjects with reduced acuity as for subjects with normal or near-normal acuity. From Figure, it is not possible to determine whether the reliability of estimates of IAD was worse in children with large IADs than in children with litde difference in acuity between eyes. To examine this issue, IAD estimates for observer versus observer 2 were plotted in Figure 2 for the clinical group. There were four subjects (filled symbols) for whom one or both observers found an IAD greater than one octave. In one (25%) of these subjects, observers disagreed by more than octave in their estimation of IAD. Among the 20 subjects for whom neither observer found an IAD greater than octave (open symbols), there were two subjects (0%) in whom observers dif-

4 Reliability of Teller Acuity Card in Patients 83 RE. BETTER "3 Q < LLJ.E SI L.E. BETTER ' ' CLINICAL GROUP v y ' y RE. BETTER OBSERVER IAD (difference in octaves) FIGURE 2. Interobserver agreement for estimation of interocular acuity difference for the 24 members of the clinical group who had monocular acuity assessed in each eye. Filled symbols indicate children in whom one or both observers indicated an interocular acuity difference greater than octave. Solid line indicates perfect agreement between observers' estimates of interocular acuity difference. Area between dashed lines indicates interobserver agreement of octave or better. fered by more than octave in their estimation of IAD. These numbers are too small to determine whether interobserver agreement was worse among children with large IADs than among children with small IADs. Figure 3 plots group data from the clinical and - - the control subjects, summarizing the percentage of first eye, second eye, IAD, and binocular comparisons in which interobserver agreement differed by 0,,, and > octave. For monocular results, 9.4% of clinical test-retest pairs and 94.8% of control pairs showed agreement of octave or better, and 77.6% of the clinical and 79.3% of the control pairs showed agreement of octave or better. For interobserver agreement in estimation of IAD, 87.5% of both clinical and control test-retest pairs showed agreement of octave or better, and 70.8% of both the clinical and control test-retest pairs showed agreement of octave or better. In binocular testing, 96% of the clinical and 96% of the control interobserver test-retest pairs showed agreement of octave or better, and 76% of the clinical and 72% of the control pairs showed agreement of octave or better. Chi-square analysis showed no differences in the proportion of 0,,, or > octave differences between observers for the clinical versus control groups, for the eye tested first, the eye tested second, estimates of IAD, or binocular tests. Thus, the interobserver agreement for children with ocular or neurologic abnormalities is similar to the interobserver agreement for healthy children who were born before term, and there is no clear evidence that interobserver agreement is lower in children with reduced acuity than in children with normal acuity. To examine observer differences, matched t- tests were conducted comparing acuity results for pairwise combinations of the three observers who performed 98% of the acuity tests. For the same CO 80 ill hi CO 70 FIGURE 3. Percentage of clinical and control test-retest pairs in which the difference in acuity estimates between the two observers differed by 0,,.0, or >.0 octave for data from monocular tests (eye tested first and eye tested second), estimates of within-subject interocular acuity difference, and binocular tests. Numbers above each bar indicate the number of test-retest pairs included in that bar. LL. O O Cd ill a ' 4 '' i CLINCON ST EYE CLINCON 2ND EYE IAD BINOC INTEROBSERVER DIFFERENCE : > OCTAVE.0 OCTAVE OCTAVE 0 OCTAVE

5 84 Investigative Ophthalmology & Visual Science, January 996, Vol. 37, No. DISCUSSION 5.0 ~ 4.0 D 3.0 H CO MONOCULAR ST EYE MONOCULAR 2ND EYE BINOCULAR FIGURE 4. Mean test duration for subjects in the clinical and the control groups for monocular (eye tested first and eye tested second) and binocular acuity tests. Bars indicate ± SEM. subjects, scores obtained by observer 3 averaged 0.4 octave lower (worse) than those of observer (^8 = 5.3, P < 0.00) and 0.3 octave lower than those of observer 2 (h$ = 4.7, P < 0.00). Scores for observer 2 averaged 0.2 octave lower than those for observer in tests of the same subjects, but the difference was not significant. These data confirm the small but reliable difference among observers that has been reported previously. 2 " 5 Mean test times for monocular and binocular tests for clinical and control groups are shown in Figure 4. Across all test conditions, mean test time was 4.0 minutes for the clinical group and 2.5 minutes for the control group. Ninety-five percent of tests of the clinical group were completed in 8 minutes or less, whereas 95% of tests of the control group were completed in 4 minutes or less, -tests indicated that test times were significandy different between clinical and control groups for the eye tested first (^7 = 3.80, P < 0.00), the eye tested second (fe, = 3.70, P < 0.0), and for binocular tests (fc, = 2.88, P < 0.0). Test times did not differ between the 5% of tests in which the "start card" recommended based on the child's age was below threshold versus the 85% of tests in which the "start card" was above threshold, suggesting that the lengthier test times in the clinical group resulted from difficulty of testing, not from the fact that the initial card shown was below the subject's acuity threshold. The current study provides the first within-study comparison of interobserver test-re test reliability of the TAC procedure in infants and young children with and without ocular or neurologic abnormalities. Results indicated no differences in interobserver reliability between children with mild to severe ocular or neurologic abnormalities and age-matched healthy preterm children without ocular or neurologic abnormalities (Figs., 3) and no differences in interobserver reliability between children with significantly reduced acuities and those with near-normal acuity (Figs., 2). Thus, although children with ocular or neurologic abnormalities may be difficult to test because of abnormal eye movements and looking patterns 9 and may require more time to test than children without abnormalities (Fig. 4), our data suggest that the reliability of the TAC procedure in the hands of experienced observers is as high in children with mild to severe ocular and neurologic abnormalities as it is in children without such abnormalities. Tables and 2 compare interobserver test-re test results from the current study with data from previous reports of interobserver test-retest reliability for binocular and monocular acuity card testing, respectively. Percentages of test-retest comparisons showing agreement of octave or better and octave or better in the current study are similar to those reported previously for normal children, but somewhat higher than those reported for children with perinatal complications tested in our laboratory. 2 ' 3 The lower test-retest agreement reported previously from our laboratory is probably caused by the use of a variant of the TAC procedure in which the observer is masked to the spatial frequency of the gratings used. This variant of the TAC procedure can lead to increased variability of acuity results from a bias introduced by the particular spatial frequencies of the cards used during testing (e.g., observers may expect to show a certain number of cards before reaching threshold, which may lead to better acuity scores when the first card presented contains a high spatial frequency grating than when the first card presented contains a low spatial frequency grating). 3 Interobserver agreement in the current study is considerably higher than that reported for children with severe neurologic abnormalities tested by Hertz and colleagues. 9 " 2 The lower interobserver test-retest reliability found by them may reflect difficulties in testing children who have extremely poor mental, motor, and visual capabilities. Table 3 compares data from the current study with data from two previous studies 3 ' 8 that examined interobserver test-retest reliability for IAD. In all

6 Reliability of Teller Acuity Card in Patients 85 TABLE l. Interobserver Agreement in Acuity Card Studies: Tests of Binocular Acuity Study Step Size () Age N* 0.0 (%) = (%) (%) Normal subjects McDonald et al McDonald et al 7 McDonald et al 6 Heersema and van Hof-van Duin 3 Getz et al (healthy preterm infants) Abnormal subjects Preston et al 8 (visual abnormality) Hertz and Rosenberg" (cerebral palsy) Hertz 9 (mentally retarded) Hertz et al 0 (mentally retarded, cortically visually impaired) Dobson et al 2 (NICU-treated) Hertz and Rosenberg 2 (cerebral palsy) Mash et al 3 (NICU-treated) Getz et al (ocular-neurologic abnormalities) months -2 months 8-36 months -4 years 3-38 months 2-8 months 2-7 years 8-7 years 2-2 years -7-3 days -8 years months 3-38 months f * Number of interobserver test pairs. f Percent of tests that differed by no more than 0.3 octave. three studies, agreement for estimation of IAD was poorer than it was for either monocular or binocular acuity estimation in the same study. This reflects the fact that between-observer comparisons of IAD are based on four acuity tests (one per eye for each of the two observers) and their associated variability, whereas between-observer comparisons of monocular or binocular acuity estimates are based on only two tests (one per observer). It is important to note that in the current study, all interobserver comparisons are conducted as within-subjects comparisons. Thus, the finding that interobserver agreement for estimation of IAD is lower than interobserver agreement for a single acuity test does not contradict a previous report that between-subject variability is less for IAD than for single estimates of acuity. 7 Results of the current study confirm previous reports 2 " 5 that some observers are more conservative than others in their estimation of acuity. The fact that acuity estimates of observer 3 averaged 0.4 and 0.3 octave lower than those of the two other primary observers, respectively, undoubtedly contributed to the large proportion of interobserver differences of octave (Fig. 3). However, this magnitude of difference between observers is much smaller than the - to 2- octave width of the normal range of acuities in infants and toddlers 42 and, therefore, should have little influence on the effectiveness of the TAC procedure in TABLE 2. Interobserver Agreement in Acuity Card Studies: Tests of Monocular Acuity Study Step Size () Age (months) N* 0.0 ^ ^.0 Normal subjects McDonald et al 7 McDonald et al 6 Mayer et al 4 Getz et al (healthy preterms) Abnormal subjects Preston et al 8 (visual abnormality) Dobson et al 2 (NICU-treated) Mash et al 3 (NICU-treated) Getz et al (ocular-neurologic abnormalities) Number of interobserver test pairs.

7 86 Investigative Ophthalmology 8c Visual Science, January 996, Vol. 37, No. TABLE 3. Interobserver Agreement in Acuity Card Studies: Interocular Acuity Difference Study Step Size () Age (months) N* 0.0 Normal subjects Getz et al (healthy preterm infants) Abnormal subjects Mash et al 3 (NICU-treated) Preston et al 8 (visual abnormality) Getz et al (ocular-neurologic abnormalities) v f t * Number of interobserver test pairs. f Percentages were identical in the clinical and the healthy preterm groups. clinical settings. This rinding does suggest, however, that, before testing patients, each new observer should do enough "practice" tests on normal subjects to determine whether his or her scores tend to fall at the high or the low end of the normal range. In addition, small (< octave) changes in acuity across visits should be interpreted with caution, especially if the acuity tests were performed by two observers, one of whom tends to find higher acuity scores than the other. In summary, the results of the current study suggest that the TAC procedure is as reliable for assessment of children in a pediatric eye clinic as it is for healthy children without ocular or neurologic abnormalities. In both populations, more than 90% of testretest comparisons produced acuity scores that differed by no more than octave. These results support the use of the TAC procedure in patient populations. Key Words acuity cards, children, grating acuity, infants, interobserver reliability Acknowledgments The authors thank Albert Biglan, Kenneth Cheng, Milton Pettapiece, Mark Scher, and the staff of the Western Pennsylvania School for Blind Children for referral of children in the clinical group. They also thank the staff of the Magee- Women's Hospital Neonatal Intensive Care Unit for referral of children in the control group. References. McDonald MA, Dobson V, Sebris SL, Baitch L, Varner D, Teller DY. The acuity card procedure: A rapid test of infant acuity. Invest Ophthalmol Vis Sri. 985; 26: Dobson V, Carpenter NA, Bonvalot K, Bossier J. The acuity card procedure: Interobserver agreement in infants with perinatal complications. Clin Vision Sri. 990; 6: Mash C, Dobson V, Carpenter N. Interobserver agreement for measurement of grating acuity and interocular acuity differences with the Teller Acuity Card procedure. Vision Res. 994; 35: Mayer DL, Beiser AS, Warner AF, Pratt EM, Raye KN, Lang JM. Monocular acuity norms for the Teller Acuity Cards between ages one month and four years. Invest Ophthalmol Vis Sri. 995; 36: Quinn GE, Berlin JA, James M. The Teller Acuity Card procedure: Three testers in a clinical setting. Ophthalmology. 993; 00: McDonald MA, Ankrum C, Preston K, Sebris SL, Dobson V. Monocular and binocular acuity estimation in 8- to 36-month-olds: Acuity Card results. AmJ Optom PhysiolOpt. 986;63: McDonald MA, Sebris SL, Mohn G, Teller DY, Dobson V. Monocular acuity in normal infants: The Acuity Card procedure. Am J Optom Physiol Opt. 986;63: Preston KL, McDonald MA, Sebris SL, Dobson V, Teller DY. Validation of the Acuity Card procedure for assessment of infants with ocular disorders. Ophthalmology. 987; 94: Hertz BG. Use of the Acuity Card method to test retarded children in special schools. Child Care Health Dev. 988; 4: Hertz BG, Rosenberg J, Sjo O, Warburg M. Acuity card testing of patients with cerebral visual impairment. Dev Med Child Neurol. 988;30: Hertz BG, Rosenberg J. Acuity card testing of spastic children: Preliminary results. JPediatr Ophthalmol Strab. 988;25: Hertz BG, Rosenberg J. Effect of mental retardation and motor disability on testing with visual acuity cards. Dev Med Child Neurol. 992;34: Heersema DJ, van Hof-van Duin J. Age norms for visual acuity in toddlers using the acuity card procedure. Clin Vision Sri. 99O;5: Marx MS, Werner P, Cohen-Mansfield J, Hartmann EE. Visual acuity estimates in noncommunicative elderly persons. Invest Ophthalmol Vis Sri. 990; 3: Atkinson J, Braddick O, Pimm-Smith E. 'Preferential looking' for monocular and binocular acuity testing of infants. Br J Ophthalmol. 982; 66:

8 Reliability of Teller Acuity Card in Patients Birch EE. Infant interocular acuity differences and binocular vision. Vision Res. 985; 25: Birch EE, Hale LA. Criteria for monocular acuity deficit in infancy and early childhood. Invest Ophthalmol Vis Set. 988; 29: Maurer D, Lewis TL, Brent HP. The effects of deprivation on human visual development: Studies of children treated for cataracts. In: Morrison FJ, Lord CE, Keating DP, eds. Applied Developmental Psychology. San Diego: Academic Press; 989: Trueb L, Evans J, Hammel A, Bartholomew P, Dobson V. Assessing visual acuity of visually impaired children using the Teller acuity ard procedure. Am Orthoptic J. 992;42: Teller Acuity Card Manual. Dayton, OH: Vistech, Inc Salomao SR, Ventura DF. Large sample population age norms for visual acuities obtained with Vistech- Teller acuity cards. Invest Ophthalmol Vis Sri. 995; 36: