Supplemental Information

Transcription

1 ARTICLE Supplemental Information This section contains additional detail about alternative scores that were considered for the M-CHAT-R/F (see Supplemental Appendix), as well as additional information about the performance of the tool with subsets of the data. Alternative Scores In addition to total score, several scores consisting of selected items were evaluated. Discriminant function analysis on the first 600 participants in the original M-CHAT study 15 derived the original critical score. Once the critical score was found to be ineffective in increasing sensitivity above and beyond total score, discriminant function analysis was applied to the larger M-CHAT sample 12 and the 7 items that best discriminated ASD from all other cases in this larger sample were used to create the Best7 score (M-CHAT-R items 1, 2, 3, 7, 8, 9, and 10; see Supplemental Tables 5 and 6. At the onset of the current M-CHAT-R/F study, a child screened positive on the M- CHAT-R/F based on a threshold of 3 on total score or a threshold of 2 on the Best7 score. The goals of the current study were twofold: (1) to investigate whether sensitivity for Best7 was adequate, in which case it could be used as the sole threshold, discarding the total score, and (2) to investigate whether Best7 used in conjunction with total score increased sensitivity compared with total score alone. In June 2011, preliminary analyses indicated that the Best7 did not have adequate sensitivity, nor did it identify a sufficient number of cases missed by the total score to warrant its continued use. At this time, the GSU site undertook an exploratory study of an expanded set of best items, called Best14 (M-CHAT-R items 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, and 18; see Supplemental Tables 5 and 6. Children in this substudy screened positive if they showed risk on 2 of the Best14 items or Total3. Reliability of Best Scores When only the 14 best (Best14) items were included, internal consistency was adequate (Cronbach s a5 0.69). However, Cronbach s a was poor when only the 7 best (Best7) items were included (0.52). Optimal Scoring Receiver operating characteristic curves indicated that the threshold of 2 for Best7 and for Best14 yielded the optimal balanceofsensitivityandspecificity (see Supplemental Table 5. The sensitivity of Best7 was significantly reduced compared with the sensitivity of Total3 (1-tailed McNemar s test,p,.001) (see Supplemental Table 6). Furthermore, no cases were detected by Best7 and not also detected by Total3. Best14 was examined in the subsample subsequent to its development (n ); only 2 additional ASD cases were detected by using a combination of Best14 and Total3 on the initial screen, compared with Total3 alone, and Best14 did not reduce the false-positive rate compared with Total3. Therefore, the most parsimonious scoring for the initial M-CHAT-R is Total3, and for M-CHAT-R/F is Total2. Psychometric Properties of Subsamples Comparison of Subsets of the Sample Several factors in the current study mayinfluencethewayascreeningtool performs, thereby affecting its psychometric properties. To evaluate these differences, the sample was divided by gender (male, female), age (younger versus older toddlers), and site (UConn versus GSU); and the psychometric properties were directly compared for each variable (see Supplemental Table 7). Sex Among the children diagnosed with ASD, there were 89 boys and 34 girls (72% boys). This ratio is slightly different from the 4:1 sex ratio reported in the literature. 2 However, the male-tofemale ratio in toddlers is less studied, and more girls may be related to the likelihood that many of the ASD cases that are not detected until children are older (eg, Asperger disorder) show the greatest gender differences in prevalence. Sensitivity was not significantly different in boys (0.87) than in girls (0.82) (x 2 [1, n 5 123] , P 5.989). However, although specificity was very high for both subgroups (0.99), there was a significant difference (x 2 [1, n ] , P 5.034). PPV and negative predictive value (NPV) were not significantly different by sex (P..05), although there were trends toward higher PPV in boys and higher NPV in girls. The significant difference in specificity indicates that further research on successful screening in girls is warranted. Younger/Older Toddlers On the basis of previous research using the original M-CHAT, which found higher PPV in older toddlers compared with younger toddlers, 29 the current sample was split into PEDIATRICS Volume 133, Number 1, January 2014 SI1

2 younger (14 19 months) and older (20 30 months) subgroups. Different from the previous literature, we found that PPV was not significantly different for younger toddlers (0.42) compared with older toddlers (0.52) (x 2 [1, n 5 221] , P 5.079), although the direction of difference was as previously reported. Nor were other psychometric properties significantly different between the younger and older subgroups (P..10). Site Sensitivity was not significantly higher in the UConn sample (0.94) compared with the GSU sample (0.82) (x 2 [1, n 5 123] , P 5.067). However, specificity was significantly higher in the GSU sample (0.995) than in the UConn sample (0.989) (x 2 [1, n ] , P,.001), and PPV was higher in the GSU sample (0.58) than in the UConn sample (0.35) (x 2 [1, n 5 221] , P 5.001). One of the strategies to find cases of ASD missed by the M-CHAT-R/Fasked doctors to flag children with possible ASD. There was high variability among pediatric offices regarding the implementation of this strategy. Some offices never flagged any cases, whereas others flagged a large number of cases. The frequency of physicians noting ASD concerns was higher in the metro-atlanta site (32 flags total; 0.32% of screens) compared with the Connecticut site (13 flags total; 0.21% of screens), although this difference was not significant (x 2 [1, n ] , P 5.21). An additional strategy at the GSU site was to invite astratified random sample of toddlers who screened negative either on the initial M-CHAT-R or on the follow-up to attend a second in-person screening session using the STAT. Twenty children screened positive on the STAT and attended an evaluation; 7 of these cases were diagnosed with ASD. SUPPLEMENTAL REFERENCE 29. Pandey J, Verbalis A, Robins DL, et al. Screening for autism in older and younger toddlers with the Modified Checklist for Autism in Toddlers. Autism. 2008;12(5): SI2 ROBINS et al

3 ARTICLE SUPPLEMENTAL TABLE 5 Psychometric Properties for Thresholds of Alternative Scores on Initial Screening Cutoff Score Total Score Best7 Score Best14 Score Sensitivity Specificity Sensitivity Specificity Sensitivity Specificity AUC , no value is available; AUC, area under the curve; Best7, score derived from the 7 items that best discriminated ASD from nonasd (M-CHAT-R items 1, 2, 3, 7, 8, 9, 10) cutoff score of $2 on the Best 7 items; Best14, score derived from the 14 items that best discriminated ASD from nonasd (M-CHAT-R items 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18) cutoff score of $2 on the Best 14 items. SUPPLEMENTAL TABLE 6 Psychometric Properties of M-CHAT-R and M-CHAT-R/F Alternative Scores TP (hit) FN (miss) FP TN Sensitivity (95% CI) Specificity (95% CI) PPV NPV LR1 LR2 Initial M-CHAT-R scoring Best7 a , ( ) ( ) Best7 or Total3 b ( ) ( ) Best14 c ( ) ( ) Best14 or Total ( ) ( ) Total ( ) ( ) M-CHAT-R/F scoring Best7 or Total3 w/follow-up d Best7 or Total ( ) ( ) Best7 or Total3 w/follow-up Total2 e ( ) ( ) Best14 or Total3 w/follow-up Best14 or Total3 c ( ) ( ) Best14 or Total3 w/follow-up Total ( ) ( ) Total3 w/follow-up Total ( ) ( ) Total3 w/follow-up Total ( ) ( ) FN, false-negative cases; FP, false-positive cases; LR1, likelihood ratio of positive screen; LR2, likelihood ratio of negative screen; NPV, negative predictive value; TN, true-negative cases, presumed based on screening negative without other indicators of ASD risk; TP, true-positive cases. a Best7: classification as TP, FN, FP, or TN based on the threshold of 2 on the Best7 score. b Total3: classification as TP, FN, FP, or TN based on the threshold of 3 on the total score. c Best14: classification as TP, FN, FP, or TN based on the threshold of 2 on the Best14 score. Note that sample is smaller for thresholds involving Best14, given that it was developed during the current study and only implemented in 1 site. d w/follow-up: these scores are represented by the threshold(s) applied to the initial screening with the additional threshold(s) applied to the tollow-up. e Total2: classification as TP, FN, FP, or TN based on the threshold of 2 on the follow-up total score. SUPPLEMENTAL TABLE 7 M-CHAT-R/F Scoring of Total3 With Follow-up Total2 Across Subsamples n TP (hit) FN (miss) FP TN Sensitivity (95% CI) Specificity (95% CI) PPV NPV LR1 LR2 By sex Male ( ) ( ) Female ( ) ( ) x 2 Comparing psychometrics by gender * By age Younger a ( ) ( ) Older b ( ) ( ) x 2 Comparing psychometrics by age By site UConn ( ) ( ) GSU x 2 Comparing psychometrics by site ** ** 5.497* * P,.05, **P,.01. FN, false-negative cases; FP, false-positive cases; LR1, likelihood ratio of positive screen; LR2, likelihood ratio of negative screen; NPV, negative predictive value; TN, truenegative cases, presumed based on screening negative without other indicators of ASD risk; TP, true-positive cases. a Younger toddlers ranged from Mos b Older toddlers ranged from Mos. PEDIATRICS Volume 133, Number 1, January 2014 SI3

4 SI4 ROBINS et al

5 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI5

6 SI6 ROBINS et al

7 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI7

8 SI8 ROBINS et al

9 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI9

10 SI10 ROBINS et al

11 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI11

12 SI12 ROBINS et al

13 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI13

14 SI14 ROBINS et al

15 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI15

16 SI16 ROBINS et al

17 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI17

18 SI18 ROBINS et al

19 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI19

20 SI20 ROBINS et al

21 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI21

22 SI22 ROBINS et al

23 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI23

24 SI24 ROBINS et al

25 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI25

26 SI26 ROBINS et al

27 ARTICLE PEDIATRICS Volume 133, Number 1, January 2014 SI27

28 SI28 ROBINS et al