Early detection of emotional and behavioural problems Oxford, DME Diabetic 0742-3071 Blackwell 20 Original Emotional UK Article article Medicine Publishing Science well-being: Ltd, Ltd. screening 2003 using CHQ and BASC F. J. Cameron al. in children with diabetes: the validity of the Child Health Questionnaire as a screening instrument F. J. Cameron*, D. Smidts, K. Hesketh, M. Wake and E. A. Northam *Department of Endocrinology and Diabetes, Department of Psychology, and Centre for Community Child Health, Royal Children s Hospital, Melbourne, Australia Accepted 19 February 2003 Summary Aims To assess the validity of the Child Health Questionnaire (CHQ) as a screening tool for detecting at risk emotional and behavioural maladjustment in children with diabetes, using the Behaviour Assessment System for Children (BASC) as a gold standard measure. Methods CHQ and BASC were administered to 103 parents of children with Type 1 diabetes, aged 7 12 years. Sub-scales of the two measures were compared using Pearson s bivariate correlations. CHQ sensitivity and specificity cut-points were optimized against the BASC borderline category using receiver operating characteristic curves. Results The BASC Externalizing Problems scale correlated strongly with CHQ Behaviour, Global Behaviour, Mental Health, Family Activities and Family Cohesion scales (r-values 0.68, 0.54, 0.51, 0.59, and 0.42, respectively). BASC Internalizing Problems scale correlated strongly with CHQ Behaviour, Mental Health and Family Cohesion scales (r-values 0.40, 0.43 and 0.45, respectively). Using receiver operating characteristic curve analysis, the CHQ Mental Health scale most effectively identified children classified as borderline on the BASC Internalizing Problems scale (sensitivity 87%, specificity 78%), while the CHQ Global Behaviour scale most effectively identified children classified as borderline on the BASC Externalizing Problems scale (sensitivity 73%, specificity 82%). Conclusions Significant correlations were seen between the CHQ Global Behaviour and Mental Health scales and the BASC Externalizing and Internalizing scales, respectively. Sequential use of the CHQ, as a screening tool, followed by an established mental health measure such as the BASC, may help identify children with diabetes at risk for chronic maladjustment and poor health outcomes. Diabet. Med. 20, 646 650 (2003) Keywords CHQ, BASC, maladjustment, children, screening Correspondence to: Dr F J Cameron, Deputy Director, Department of Endocrinology and Diabetes, Royal Children s Hospital, Flemington Road, Parkville, 3052, Australia. E-mail: fergus.cameron@rch.org.au Introduction Clinical and empirical evidence suggest that, for children with Type 1 diabetes, the period of adolescence is frequently accompanied by an increase in psychological symptoms [1] and a deterioration in metabolic control [2]. Chronic poor metabolic control is associated with an elevated risk of complications of 646
Original article 647 Type 1 diabetes, such as nephropathy, retinopathy and vascular disease [3]. In addition, poorly controlled diabetes is associated with subtle neuropsychological deficits which may reduce academic achievement, career options and life-style choices [4,5]. The deterioration seen in both psychological well being and metabolic control during adolescence is linked, with non-compliance with the treatment regimen (insulin omission, binge eating, poor clinic attendance, etc.) possibly acting as a mediating variable [1,6]. Research in our own [7] and other centres [8] supports the notion that early adjustment to Type 1 diabetes is predictive of longer-term outcome, with early psychological difficulties leading to ongoing maladjustment, reduced treatment compliance and poorer mental and clinical health outcomes. Early identification of children at risk of moving along the trajectory of psychological maladjustment reduced treatment compliance poor metabolic control could facilitate optimal use of scarce resources, as well as reducing the psychological and health morbidity of individual children and their families. However, it is neither possible nor necessary to offer individual mental health clinical services to all children and families. One approach to early identification would be the routine use of short but sensitive screening measures for early detection of emotional problems in the diabetes clinic [9]. The difficulties inherent in such screening programmes in diabetic populations have been previously acknowledged [3]. The Child Health Questionnaire (CHQ) [10] has high reliability and validity when used to assess group functional health status in clinical populations of children with chronic illness, including Type 1 diabetes [11,12]. However, the sensitivity of the CHQ to screen for clinically significant levels of mental health disturbance in individual children is unknown. The purpose of this study was to assess the validity of the CHQ as a screening tool to identify children at risk of emotional and behavioural maladjustment, using the Behaviour Assessment System for Children (BASC) [13] as a gold standard measure of clinically significant psychological disturbance in children. Patients and methods Subjects The patient database within the Department of Endocrinology and Diabetes, Royal Children s Hospital (RCH), Melbourne, was used to identify children aged 7 12 years with Type 1 diabetes who had an outpatient clinic visit scheduled between March and June 2001. All such children and their mothers were the subjects for this study. Non-English speaking or reading mothers and their children were excluded. The RCH Diabetes Clinic provides services to approximately 60% of Victorian children with diabetes. These children live within a large, sociodemographically diverse region comprising approximately two-thirds of Melbourne (population 3.4 million), Australia. Children attend the clinic at least once every 3 months. The study was approved by the Royal Children s Hospital Ethics in Human Research Committee. Procedures An invitation to participate in the study was mailed to eligible families in the month prior to the child s next clinic visit. The research assistant (DS) telephoned families a week later to answer questions and clarify participants willingness to take part in the project. Families willing to participate were asked to attend the clinic 45 min before their scheduled appointment to sign consent forms and to complete two questionnaires, the BASC and the CHQ (see Measures). The questionnaires were administered in counterbalanced order. Measures The BASC is a standardized, validated, parent-report, 137-item instrument which assesses emotional and behavioural problems in children, in addition to adaptive and social skills [13]. The BASC has been standardized on a demographically representative sample of 2130 children aged 6 11 years. It has shown high internal consistency (coefficient alpha reliability is 0.93 for the Externalizing, 0.89 for the Internalizing scales and 0.94 for the Total Behaviour Symptoms Index, respectively). Test retest reliability is very high over intervals of 2 8 weeks with r-values of 0.91 and 0.94 for the Externalizing and Internalizing scales, respectively. Scale scores correlate highly (< 0.7) with corresponding scale scores on other similar standardized instruments of child behaviour (e.g. The Child Behaviour Checklist, Achenbach, 1991). Sample items are: Has friends who are in trouble ; hits other children (externalizing); worries about things that can t be changed, is easily upset (internalizing). Parents are asked to respond on a 4-point scale: Never, Sometimes, Often, Almost Always. Validity indexes are incorporated in the measure to minimize the distortion of rater or social desirability bias. The measure is computer scored and sub-scale and global scores are expressed as T scores to facilitate comparison across gender and age. T scores have a mean of 50 with a standard deviation of 10. Subjects who score > 60 are designated as being at risk or borderline and subjects who score > 70 are designated as having a clinically significant disturbance in the relevant sub-scale. Sub-scale scores are used to assign clinical diagnoses in accordance with DSM-IV criteria. The BASC is an individually administered instrument and requires formal psychological interpretation. Whilst the BASC takes only 15 20 min to complete, it requires 1 2 h of computer scoring and interpretation by an appropriately trained psychologist. It is thus not suitable as a general screening tool within an outpatient clinic and would require substantial resources to enable routine administration as part of regular clinical services. The CHQ PF-50 is a standardized, validated, parent-report instrument that assesses functional health status in children aged 5 18 years, covering domains of physical, emotional, social and mental health. It has 13 single and multi-item scales, eight of which were included for this study: Behaviour, Global Behaviour, Mental Health, Self Esteem, Parent Impact-Time, Parent Impact-Emotional, Family Functioning, and Family Cohesion. Five scales covering the physical functioning domain were excluded as physical health was not a focus of this study and these scales were not expected to predict emotional and behavioural problems identified by the BASC. The excluded
648 Emotional well-being: screening using CHQ and BASC F. J. Cameron et al. scales contained a total of 20 items; parents in this study completed 30 CHQ items. Scale scores are transformed to a range of 0 (worse health) to 100 (better health). The CHQ has been adapted and validated for use with Australian children [11]. The Australian Authorized Adaptation has demonstrated good validity on a number of parameters and good internal consistency when used with parents of children with diabetes, and parents of children in the general community [11,12,14]. The CHQ is simple to administer and can be scored manually or using simple computer-based scoring. It requires no formal training to administer, score or interpret. It is feasible for use in a busy clinical setting [12], and most parents are able to complete these sub-scales within 5 min. Statistical analyses Criterion validity of the CHQ Pearson s bivariate correlations were calculated between the relevant subscales of the CHQ (Behaviour, Mental Health and Self Esteem) and the gold standard instrument (BASC) to measure the strength of association between the two instruments. CHQ as a screening instrument Receiver operating characteristic curves (ROC curves) were plotted for each of the CHQ scales against each of the BASC scales. ROC curves plot the true positive rate (sensitivity) against the false positive rate (specificity) for a series of cut-points to illustrate the trade off between sensitivity and specificity, allowing the cut-point with the best balance between sensitivity and specificity to be identified [15,16]. ROC curves were calculated against borderline cut-points (T score = 60) for the BASC as the gold standard instrument. The small number of children classified as having clinically significant disturbance on the BASC precluded calculation of ROC curves against clinical cut-points. We considered sensitivity and specificity values should each be at least 75% to be of potential value as a practical tool [17,18]. All analyses were performed using the statistical package SPSS for Windows version 10.0.7 [19]. Results One hundred and three parents of children with diabetes participated in the study (75% response; 44% male). Underlying diabetes control (mean of HbA 1c at the time of study and HbA 1c 3 months prior) was identical for participants and non-participants (8.2% in both groups). Five participants were excluded from analyses due to incomplete data. The BASC and CHQ were significantly correlated on all scales except BASC Internalizing Problems with CHQ Parent Impact Time. As expected, because high scores on the BASC indicate psychopathology while high scores on the CHQ indicate better functioning, all correlations were inverse (negative). High correlations were found for BASC Externalizing Problems and CHQ Behaviour (r = 0.68), CHQ Global Behaviour (r = 0.54), CHQ Mental Health (r = 0.51), CHQ Family Activities (r = 0.59) and CHQ Family Cohesion (r = 0.42), and for BASC Internalizing Problems and CHQ Behaviour (r = 0.40), CHQ Mental Health (r = 0.43) and CHQ Family Cohesion (r = 0.45). Seven children were classified as having clinically significant internalizing problems using the clinical criterion (T score > 70) on the BASC Internalizing Problems scale. This number increased to 23 when the borderline criterion was used (T score > 60). Six children were classified as having externalizing problems using the clinical criterion on the BASC Externalizing Problems scale. This number increased to 15 when the borderline criterion was used. To increase power, the borderline cut-points were therefore used for all subsequent analyses. ROC curves were generated to assess the ability of each CHQ scale to identify high scores on the BASC Internalizing and BASC Externalizing scales, respectively. The cut-off points on each of the CHQ scales, and their respective sensitivity and specificity values, are presented in Table 1. The CHQ Mental Health scale most effectively identified cases classified as borderline on the BASC Internalizing Problems scale (sensitivity 87%, specificity 78%). The CHQ Global Behaviour scale most effectively identified cases classified as borderline on the Table 1 CHQ scale cut-off points at which sensitivity and specificity optimized for identifying borderline cases on the BASC Internalizing Problems and BASC Externalizing Problems Scales BASC Internalizing BASC Externalizing CHQ sub-scale Cut-off point Sensitivity Specificity Cut-off point Sensitivity Specificity Behaviour 74 76 77 60 53 66 73 74 Global Behaviour 73 39 77 73 73 82 Mental Health 73 87 78 78 73 53 Self Esteem 77 74 68 85 79 42 Parent Impact Time 83 77 64 72 83 73 75 57 Parent Impact Emotional 63 71 70 71 61 54 63 71 79 65 Family Activities 65 73 70 48 65 73 95 66 Family Cohesion 73 85 68 68 73 85 60 64 PsS 45 75 82 43 49 64 59 85
Original article 649 Figure 1 ROC curve for CHQ Mental Health Scale s prediction of borderline cases on the BASC Internalizing Problems Scale. Figure 2 ROC curve for CHQ Global Behaviour Scale s prediction of borderline cases on the BASC Externalizing Problems Scale. BASC Externalizing Problems scale (sensitivity 73%, specificity 82%). The ROC curves for these two scales are presented in Figs 1 and 2. The ROC curves show the sensitivity plotted on the y-axis and 1-specificity plotted on the x-axis, thus the specificity is higher towards the left of the x-axis. The point on the curve closest to the top left-hand corner represents the best trade-off between sensitivity and specificity. Discussion Our group has previously reported that, in many patients, diabetic control during the critical years of adolescence and early adulthood is determined by control during late childhood [2]. Therefore, interventions to prevent deterioration of control during adolescence may be most effective in middle childhood, before maladaptive lifestyle practices become entrenched. Screening tools may help clinicians identify at risk patients who exhibit borderline levels of maladjustment to their diabetes. As with any screening process, such tools must be simple, quick to administer and have criterion validity when compared with a gold standard assessment tool. We have previously shown that the CHQ fulfils the first two of these criteria in a diabetes clinic setting [12,20]. In addition to this, we have shown that clinicians should not rely solely upon measures of metabolic control as contemporaneous, surrogate markers of underlying quality of life or functional health [12]. This study has now extended our previous work by showing a high correlation between the Global Behaviour and Mental Health sub-scales of the CHQ and the Externalizing and Internalizing Problem scales of the BASC, a reliable gold standard for identifying emotional and behavioural problems in children with chronic illness [13]. The CHQ Global Behaviour scale is a single item asking parent s to rate their child s overall behaviour relative to other children of the same age, whereas the BASC Externalizing Scale is based on a composite and detailed profile of the child s symptoms of hyperactivity, aggression and conduct problems. The CHQ Global Behaviour item shows high-scale internal consistency within the multi-item Behaviour scale [11,12,14]. The CHQ Mental Health scale comprises five items, asking parents to evaluate their child s moods based upon observed emotions within the child, compared with the BASC s Internalizing scale which measures specific symptoms of anxiety, depression and somatization. Intuitively, the CHQ Global Behaviour and Mental Health scales appear to measure similar constructs to the BASC Externalizing and Internalizing scales, respectively. However, the strength of correlations and the high sensitivity and specificity of these CHQ scales to identify children classified as borderline on the BASC scales, given the brevity of the CHQ scales, is notable. The choice of acceptable sensitivity and specificity levels for a screening tool will vary depending on the purpose of screening, the resources available to follow up those identified as at risk by the screening, the likelihood of harm resulting from false negative screens, and opportunities for ongoing surveillance. The sensitivity and specificity levels achieved here by these brief CHQ items well exceed 70%; although this means that a proportion of children at borderline risk will not be identified at a single clinic visit, these children return 3- monthly and therefore have regular opportunities for ongoing monitoring. These findings cannot be generalized to families not literate in English, though in practice we find that most families from a variety of ethnic backgrounds are competent in English and cope more than adequately with these measures. The three CHQ sub-scales that correlated significantly with the BASC Internalizing Problems scale also correlated significantly with the BASC Externalizing Problems scale. This is not surprising, as children with internalizing problems often have externalizing problems as well; within the BASC itself, the
650 Emotional well-being: screening using CHQ and BASC F. J. Cameron et al. Internalizing and Externalizing scales typically show moderate correlations. The cumulative behavioural, functional and mental health research in our clinic population indicates that several CHQ scales could be useful as a routine tool to screen all children as they pass through the diabetes clinic for latent or early emotional and behavioural difficulties. Patients identified by the CHQ could then be further and more definitively investigated by discussion with child and parents and a formal mental health instrument such as the BASC. We anticipate that those patients whose difficulties are supported by the BASC will be those most likely to benefit from early support and intervention. Sequential use of the CHQ followed by a formal mental health instrument such as the BASC could allow a practical process of identification, which might then facilitate the rational adoption of targeted psychosocial intervention programs in children at risk for chronic maladjustment and poor health outcome. References 1 Hanson CL, Henggeler SW, Burghen GA. Model of associations between psychosocial variables and health-outcome measures of adolescents with IDDM. Diabetes Care 1987; 10: 752 758. 2 Dabaghdao P, Vidmar S, Cameron FJ. Deteriorating diabetic control through adolescence do the origins lie in childhood? Diabet Med 2001; 18: 889 894. 3 The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. New Engl J Med 1993; 329: 977 986. 4 Northam EA, Anderson PJ, Werther GA, Warne GL, Adler RG, Andrewes D. Neuropsychological complications of IDDM in children with 2 years after disease onset. Diabet Care 1998; 3: 379 84. 5 Northam EA, Anderson PJ, Jacobs R, Hughes M, Warne GL, Werther GA. Neuropsychological profiles of children with type 1 diabetes 6 years after disease onset. Diabet Care 2001; 9: 1541 1546. 6 Becker DJ. Management of insulin-dependent diabetes mellitus in children and adolescents. Curr Opin Pediatr 1991; 3: 710 723. 7 Northam E, Anderson P, Werther G, Adler R, Warne G. Psychosocial and family functioning in children with insulin dependent diabetes at diagnosis and one year later. J Pediatr Psychol, 1996; 25: 699 717. 8 Kovacs M, Makerji M, Iyengar S, Drash A. Psychiatric disorder and metabolic control among youths with IDDM. Diabetes Care 1996; 19: 318 323. 9 Eiser C. Children s quality of life measures. Arch Dis Child 1997; 77: 350 354. 10 Landgraf JM, Abetz L, Ware JE. The Child Health Questionnaire (CHQ): A User s Manual. Boston: The Health Institute, New England. Medical Centre, 1996. 11 Waters E, Salmon L, Wake M. The parent-form Child Health Questionnaire in Australia: comparison of reliability, validity, structure, and norms. J Pediatr Psychol 2000; 25: 381 91. 12 Wake M, Hesketh K, Cameron FJ. The Child Health Questionnaire in children with diabetes: cross sectional survey of parent and adolescent-reported functional health status. Diabet Med 2000; 17: 700 707. 13 Reynolds CR, Kamphaus RW. Behaviour Assessment System for Children Manual. Circle Pines, MN: American Guidance Service, Inc, 1992. 14 Waters E, Salmon L, Wake M, Hesketh K, Wright M. The Child Health Questionnaire in Australia: reliability, validity and population means. Aust NZ J Public Health 2000; 24: 207 210. 15 Metz CE. Basic principle of ROC analysis. Semin Nucl Med 1978; 8: 283 298. 16 Empson MB. Statistics in the pathology laboratory: characteristics of diagnostic tests. Pathology 2001; 33: 93 95. 17 Muir Gray JA. Evidenced-Based Healthcare (2nd edn.). Oxford: Churchill Livingstone, 2001. 18 Squires J, Nickel RE, Eisert D. Early detection of developmental problems: strategies for monitoring young chidlren in the practice setting. J Dev Behav Pediatr 1996; 17: 420 427. 19 SPSS (statistical software). Windows version. Release 10.0.7. Chicago: SPPS Inc, 2000. 20 Cameron FJ, Clarke C, Hesketh K, White EL, Boyce DF, Dalton VL et al. Regional and urban Victorian diabetic youth: clinical and quality of life outcomes. J Paed Child Health 2002; 38: 593 596.