The predictive value of the Pediatric Symptom Checklist in 5 year-old Austrian children

Transcription

1 The predictive value of the Pediatric Symptom Checklist in 5 year-old Austrian children Leonhard Thun-Hohenstein, MD 1,2, Stefanie Herzog, MS 1 1 Institute for Research and Education in Child & Adolescent Neuropsychiatry (KNIFFF), Salzburg 2 Child & Adolescent Psychiatry, Dep.of Psychiatry I, Paracelsus Private Medical University Christian Dopplerklinik, Salzburg Acknowledgement: This study was funded by Grünes Kreuz für Vorsorgemedizin Österreich Corresponding Author Leonhard Thun-Hohenstein, MD, Child & Adolescent Psychiatry, Dep.of Psychiatry I, Paracelsus Private Medical University, Ignaz Harrerstr.79, 5020 Salzburg 1

2 Abstract In this investigation the predictive value of the Pediatric Symptom Checklist (PSC) in relation to the Child Behaviour Checklist (CBCL) was studied in preschool children. 30 nursery schools in the city and province of Salzburg participated in the study and a total of 179 correctly completed questionnaires (82 male and 97 female children) were analysed. Questionnaires were completed by the parents. Data were analysed using descriptive statistics (SPSS 11.0). Predictive validity was determined by nonparametric correlations and calculations of sensitivity and specificity, as well as a ROC analysis. The mean PSC at 10.74±6.8 was found to be below international and also Austrian values for this age-group and was also found to differ significantly between girls (9.33±6.2) and boys (12.4±7.1; p<0.01). The recommended international cut-off point at 24 is too high. CBCL analysis resulted in 12.6 % impaired (t-value between 60 and 64) and 8.2 % (t-value>64) pathological children. PSC sensitivity in relation to the CBCL total score was 15.7 % and specificity 98.5 %. The cut-off was optimized with the help of ROC analysis and optimum specificity (80.2%) and sensitivity (81.1%) were obtained at a cut-off value of This cut-off would allow 24.6 % to be detected as impaired, which is somewhat above the frequency (20.2 %) found by CBCL. Conclusion: To summarize, the PSC is a valid psychosocial screening instrument at least for this age group and thus applicable also for German-speaking countries. Keywords: psychosocial screening, Pediatric Symptom Checkllist, preschool children 2

3 Introduction Psychologically impaired children and adolescents have increasingly come to the fore as a field of medical activity. Recording psychological dysfunction (frequency about 15%) and mental disorders (frequency about 5%) has become a significant part of the work of paediatricians or general practitioners in relation to prognosis and chronification [18]. The main focus should lie on early detection and correspondingly early initiation of adequate intervention. Detection rates with paediatric and general practices lie between 4 to 7% [2]. In Austria, an examination within the scope of the mother-child pass was introduced for 5-year-old children [12], which is aimed at the early detection of school underachievement. However, and predominantly due to the lack of data for Austria, no use was made of standardized instruments and thus, examination of psychological dysfunction is not included. The Pediatric Symptom Checklist (PSC) has been used for screening psychological dysfunction in the American region. Jellinek et al. studied this first in 1986 [6] in children 6 to 12 years old, and today data are available for all ages groups in the range of 4 to 18 years [7, 13, 14]. PSC validitiy and reliability has been successfully reproduced several times [8, 10, 15, 18], including an own study [19] in Austrian preschool children. The predictive value of PSC regarding different diagnostic instruments is generally good, but still varies depending on social status and cultural background [10, 15, 18]. The aim of this study is to investigate the predictive value of the German version of PSC in a population of Austrian preschool children. Methods This study focusing on playgroups within the province of Salzburg was planned as a cross-sectional follow-up to an Austrian-wide study (20). 15 nursery-schools in the city of Salzburg and 15 nursery-schools in the province of Salzburg were approached. The rate of participation was 93 %. Nursery-school teachers were asked to distribute the questionnaires for completion by those parents whose children were between 4.5 and 5.5 years old. A total of 973 questionnaires were distributed to nursery-school teachers, of which 318 (32.7 %) were returned. If four or more items remain unanswered the questionnaire is regarded as invalid [5]. Therefore only 179 (18.3 %) were included % of these children were from the city, and 46.4 % from the province of Salzburg. 87 % of the questionnaires were completed by the mother, 3

4 % by the father, and 7.2% by both parents. In one case the sheets were filled in by an unknown person. Social class Allocation to social class was determined using the father s profession and maternal educational level and split into 6 categories [17]. Respective categories were added, divided by 2 and therefrom the mean socio-economic status (SES) estimated for each family. The lowest possible score (= highest SES) is therefore 1, and the highest possible score 6 (= lowest SES). The mean level of the collective was 3.5±1.3 (range 1-6), i.e. corresponding to an average occupational level of a qualified non-manual employee. Instruments of investigation Pediatric Symptom Checklist (PSC) The PSC was taken from a publication by Jellinek [8], translated by the first author, and handed over to a translation bureau for retranslation. The PSC consists of 35 items, offering the answer categories occurring never, sometimes and frequently, which are allocated 0, 1 or 2 points, respectively. Values obtained for separate items are added up to a total score. Internationally, a value of >28 in children 6 to 16 years old is regarded as cut-off value. For children four to five years old a PSC cut-off value of 24 is recommended [15]. From an Austrian study a cut-off value of 23 points was determined [19].. PSC validity has been examined by several studies and has been described as sufficiently valid. Our analysis of the Austrian-wide data revealed an internal validity with Cronbach s Alpha of 0.86 and factorial analysis resulted in 11 factors which were able to explain 55 % of the variance [19]. Child Behavior Checklist 4-18 The parent questionnaire on the behaviour of children and adolescents (CBCL/4-18) is the German language version of the Child Behavior Checklist for ages 4-18 [3]. In its first part the questionnaire records parental judgement on psychosocial competencies, and in its second part, judgement on behavioural disorders, emotional disorders and somatic complaints of children and adolescents aged 4 to 18 years. Items from the first part can be combined to form three competence scales (activities, social competence and school). The items from the second part of the questionnaire 4

5 were used to generate eight problem scales. The scales Social Withdrawal; Somatic Complaints; and Anxious/Depressive are combined to form the superordinate scale Internalising Dysfunctions. The scales Dissocial Behaviour and Aggressive Behaviour are used to form the superordinate scale Externalising Dysfunctions. The three remaining scales named Social Problems; Schizoid/Obsessive and Attention problems were not assigned to a superordinate scale. The total score comprises 118 items. CBCL predictive value from the age of five years for later psychopathological conditions and mental disorders is well documented [11]. Statistics Both test procedures were analysed using descriptive methods, and group comparisons were carried out using t-tests und Ä 2 -tests. Non-parametric correlations between PSC total score and the complete CBCL scale as well as subscales, sensitivity and specificity were calculated in order to determine the predictive value. In addition, ROC analyses were carried out to determine the ideal cut-off for optimum specificity and sensitivity. Single factors determined by factor analysis (Varimax rotated factor analysis) were subjected to contents analysis using non-parametric correlations with the CBCL total score. Calculations were carried out using SPSS 11.0 statistical software. Subjects Questionnaires from 179 children were suitable to be included in the final evaluation, thereof 82 (45.8 %) were boys and 97 (54.2 %) girls, at a mean age of 5.01Å0.3 years (range 4.5 Ç 5.5 years). Results PSC Table 1 shows the results (mean, sd, range) as well as percentiles of the PSC total score for all children, and for girls and boys separately. When compared with the internationally accepted cut-off at 24 points, 6 children from our collective (m=4, w=2; 3.4 %) are found within the ÉimpairedÑ range. When compared with the Austrian cutoff at 23 points (90 th percentile) 9 children (5.0 %) from our collective are found to be above it. 5

6 Reliability: internal validity (Cronbach s Alpha ) was 0.87; the mean for single itemtotal score correlations was r=0.409 (range 0.182* 0.65***). Factor analysis: eleven factors, which explained a variance of 67 %. Subject-specific grouping of items onto factors is identical with that of the Austrian-wide survey [19]. CBCL Table 2 shows means, standard deviations and ranges for the complete CBCL scales, the clinical subscales, and also separately for girls and boys. 8.2 % of the children were found to be within the pathological range (T-value > 64) und 12.6 % within the impaired range (T-value between 60 und 64). Overall, therefore, 20.8 % had to be rated as impaired or pathological. 4,4 % were impaired and 7.1 % pathological on the internalising scale, and 14.2 % and 9.8 % on the externalising scale. Influence of social class on the results of PSC and CBCL No significant correlations were found between the PSC total score, the CBCL (total, internalising, externalising) and the socio-economic status. Influence of sex As visible from Tables 1 and 2 significant differences, to the disadvantage of the boys, are found for PSC and CBCL total scores, as well as the CBCL subscales Externalising, Social Problems, Attention and Aggression. Correlations and predictive value In Table 3 the correlations between the PSC total score, the overall CBCL scales and the clinical scales are listed. At a cut-off value of 23 (Austrian 90th percentile (20)) a specificity of 98.5 and a sensitivity of 15.7 is calculated for the prediction of the CBCL Total Problem Score (> T 60). Setting the cut-off to the 90 th percentile of this study (=20), a sensitivity of 52.9 and a specificity of 82.1 is obtained. ROC analysis Despite the very good correlations the obtained specificity and sensitivity appear unsatisfactory, therefore it was attempted to determine the optimum cut-off by calculating the area-under-the-curve for the Total Problem Score (T-value<64, T>60), 6

7 the internalising and externalising subscales, as well as the optimised cut-off points (see Figure 1). Table 4 contains statistical data regarding the ROC analysis. Discussion The PSC mean value for this voluntary group lies clearly below the Austrian data [19] and also below that found in most international studies [15]. In some studies, such as Pagano s [15], similarly low means were obtained. In a recent European study by Reijneveld and coworkers [16] a similar mean (11.6±8.1) was reported in schoolaged children. The reasons for these comparatively divergent results are frequently discussed as related to study group socio-economic structure or ethnic distribution [10, 15]. The present study is representative for Austria regarding the socio-economic situation and no influence of the socio-economic status on the results could be demonstrated. Similarly, the cut-off point corresponding to the 90th percentile lies below international and national data. Likewise, the cut-off sensitivity and specificity for this study was primarily low and therefore the cut-off was optimized to 15.5 using ROC analysis. These results correspond well to the orders of magnitude found in the small number of studies available for this age group [4, 9, 15]. For this age group Gardner et al. [4] calculated PSC sensitivity, also using ROC analysis, and obtained, for a cut-off at 15, a sensitivity of 82.1 and specificity of 81, which corresponds approximately to our results. Jutte et al. [10] likewise found however for older children a low sensitivity at a cut-off of 24 (7.4 %) and were able to obtain a specificity of 94 and a sensitivity of 74 after ROC analysis and cut-off optimization to 12. In the recent European study [16] sensibility and specifity obtained by ROCanalysis were found to be optimal for cut off of 22 for school aged children. The requirement of having to recalculate the cut-off for each new collective in order to increase specificity and sensitivity does present a certain difficulty in using the PSC as a screening instrument. Nevertheless, the PSC shows strong correlations with the CBCL scales, both with the Total Problem scale and the externalising and internalising behavior scales, as well as the clinical subscales. In their study, Simonian and Tarnowksi [18] documented sufficient PSC validity regarding the prediction of CBCL scores, but also recommend cut-off adaptation by ROC analysis. Dependence on the cut-off chosen is evident regarding the detection rate for impaired children. The internationally agreed cut-off at 24 would only allow a detection rate of 3.4 % of the children in our study group. With the help of the 90 th 7

8 percentile (=23) taken from the Austrian-wide study [19] 5 % would be covered, which would be clearly too low for the expected frequency of psychological disorders. When using the cut-off at 15.5 as determined by ROC analysis 44 children (24.6 %) would be included as impaired (> CBCL T-value 60) and this result lies only somewhat above the expected frequencies (20.8 % using CBCL). This result is in close agreement with international studies [15, 20]. Limitations Our study group is not a representative sample, because it resembles a purely voluntary selection. Therefore we are unable to ascertain whether possibly non- German-speaking parents and parents with difficult children had decided not to participate. To summarize, the PSC is a highly suitable and reliable screening instrument for 5- year-old children if the respective cut-offs are taken into account. For other agegroups normative surveys are however still missing for the German-speaking region. In the future, PSC might be included in the mother-child pass examination at the age of five years, in order to assist the early detection of psychologically impaired children in Austria and therefore contribute to early treatment and health service cost reduction [1]. Reference List 1. Bernal, P., Estroff, D. B., Aboudarham, J. F., Murphy, M. J, Keller, A., and Jellinek, M. S. (2000) Psychosocial Morbidity: The ecomomic burden in a pediatric health maintenance organisation sample. Arch Pediatr Adolesc Med 154: Costello, E. J. and Shugart, M. A. (1992) Above and below the threshold: severity of psychiatric symptoms and functional impairment in a pediatric sample. Pediatrics 90: Döpfner, M., Plück, J., Bölte, S., Lenz, K., Melchers, P., and Heim, K. (1998) Elternfragebogen über das Verhalten von Kindern und Jugendlichen: Deutsche Bearbeitung der Child Behavior Checklist/4-18.2nd; Manual: 4. Gardner, W., Pajer, K. A., Kelleher, K., Scholle, S. H., and Wassermann, R. C. (2002) Child sex differences in primary care clinicians mental health 8

9 care of children and adolescents. Arch Pediatr Adolesc Med 156: Jellinek, M. S. (1998) Approach to the behavior problems of children and adolescents Jellinek, M. S., Murphy, M., and Burns, B. J. (1986) Brief psychosocial screening in outpatient pediatric practice. J Pediatr 109: Jellinek, M. S., Murphy, M., Pagano, M. E., Comer, D., and Kelleher, K. (1999) Use of the Pediatric Symptom Checklist (PSC) to creen for psychological problems in pediatric primary care: a national feasability study. Arch Pediatr Adolesc Med 153: Jellinek, M. S., Murphy, M., Robinson, J., Feins, A., Lamb, S., and Fenton, T. (1988) Pediatric Symptom Checklist: screening school-age children for psychosocial dysfunction. J Pediatr 112: Jellinek, M. S., Murphy, M. J, Little, M., Pagano, M. E., Comer, D., and Kelleher, K. (1999) Use of the pediatric symptom Checklist to screen for psychosocial problems in pediatric primary care. Arch Pediatr Adolesc Med 153: Jutte, D. P., Burgos, A., Mendoza, F., Ford, C. B., and Huffman, L. C. (2003) Use of the Pediatric Symptom Checklist in a low-income, Mexican- American population. Arch Pediatr Adolesc Med 157: Kroes, M., Kalff, A. C., Steyaert, J., Kessels, A. G. H., Feron, F. J. M., Hendriksen, J. G. M., van Zeben, T. M. C. B., Troost, J., Jolles, J., and Vles, J. S. H. (2002) A Longitudinal community study : do psychosocial risk factors and Child Behavior Checklist scores at 5 years of age predict psychiatric diagnoses at a later age. J Am Acad Child Adolesc Psychiatry 41: Kurz, R., Fally-Kausek, R., Haferl, B., and Mauritz, D. (2006) Mutter-Kind- Pass.24: 13. Little, M., Jellinek, M. S., and Murphy, M. J (1994) Screening four and five year old children for psychosocial dysfunction: a preliminary study with the Pediatric Symptom Checklist. J Dev Behav Ped 15: Murphy, M. J, Reede, J., Jellinek, M. S., and Bishop, S. J. (1992) Screening for psychosocial dysfunction in inner-city children: further validation of the Pediatric Symptom Checklist. J Am Acad Child Adolesc Psychiatry 31: Pagano, M. E., Murphy, M. J, Pedersen, M., Mosbacher, D., Crist-Whitzel, J., Jordan, P., Rodas, C., and Jellinek, M. S. (1996) Screening for psychosocial problems in 4-5-year-olds during routine EPSDT examinations: validity and reliability in an Mexican sample. Clin Pediatr 35:

10 Reijneveld, S. A., Vogels, A. G. C., Hoekstra, F., and Crone, M. R. (2006) Use of the Pediatric Symptom Checklist for the detection of psychosocial problems in preventive healthcare. BMC Public Health 6: Schlack, H. (1995) Sozialpädiatrie Gesundheit.Krankheit. Lebenswelten Simonian, S. J. and Tarnowski, K. J (2001) Utility of the Pediatric Symptom Chekclist fo behavoral screening of disadvantaged children. Child Psychiatry Hum Dev 31: Thun-Hohenstein, L., Herzog, S., Seim, G., and Schreuer, M. (2006) The Pediatric Symptom Checklist - a screening instrument for psychiatric disorders. Neuropsychiatrie 20: Wildman, B. G., Kinsman, A. M., and Smucker, W. D. (2000) Use of child reports of daily functionung to facilitate identification of psychosocial problems in children. Arch Fam Med 9:

11 Figure 1. Receiver-Operator Characteristic (ROC) curve for all possible cut-off points of the Austrian version of the PSC, using the CBCL-Toal Problem Score (T-value >60; T-value > 64) 11

12 CBCL Total Problem Scale T > 60 1,0 ROC Curve,8,5 Sensitivity,3 0,0 0,0,3,5,8 1,0 1 - Specificity Diagonal segments are produced by ties. CBCL Total ProblemScale T > 64 ROC Curve 1,0,8,5 Sensitivity,3 0,0 0,0,3,5,8 1,0 1 - Specificity Diagonal segments are produced by ties. CBCL Internalising Behavior Scale T > 60 ROC Curve 1,0,8,5 Sensitivity,3 0,0 0,0,3,5,8 1,0 1 - Specificity Diagonal segments are produced by ties. 12

13 Sensitivity and Specifity for several cut-off points; optimal cut-off bold. Cut-off Sensitivity Specifity Sensitivity and Specifity for several cut-off points; optimal cut-off bold. Cut-off Sensitivity Specifity Sensitivity and Specifity for several cut-off points; optimal cut-off bold. Cut-off Sensitivity Specifity CBCL-Externalising Behavior Scale T > 60 13

14 1,0 ROC Curve,8,5 Sensitivity,3 0,0 0,0,3,5,8 1,0 1 - Specificity Diagonal segments are produced by ties. Sensitivity and Specifity for several cut-off points; optimal cut-off bold. Cut-off Sensitivity Specifity

15 Table 1 Means, standard deviations, range and percentile values of the PSC for the whole group and girls and boys PSC total score mean±s.d. all girls boys 10.74± ± ±7.1 ** range th percentile th percentile th percentile th percentile th percentile th percentile th percentile ** p=

16 Table 2 Mean value, standard deviations (SD) of CBCL: total score, internalising, externalising and clinical scales overall, girls and boys, as well as significance level. Questionnaire all boys girls mean SD mean SD mean SD p- value CBCL overall score CBCL externalising score CBCL internalising score n.s. CBCL social withdrawal n.s. CBCL somatic complaints n.s. CBCL anxiety/depression n.s. CBCL social problems CBCL schizoid, obsessive n.s. CBCL attention disorder CBCL delinquency n.s. 373 CBCL aggression

17 Table 3 Non-parametric correlations between PSC total score and CBCL total score, internalising and externalising scale as well as clinical subscales. PSC total score r= CBCL overall score 0.72 *** CBCL externalising score 0.66 *** CBCL internalising score 0.51 *** CBCL social withdrawal 0.38 *** CBCL somatic complaints 0.36 *** CBCL anxiety/depression 0.43 *** CBCL social problems 0.49 *** CBCL schizoid, obsessive 0.31 *** CBCL attention disorder 0.58 *** CBCL delinquency 0.49 *** CBCL aggression 0.64 *** *** p<

18 Table 4 Statistical determination of area-under-the-curve/roc analysis of the predictive value of PSC for CBCL total Probem Scale and subscales. area std. error asymptotic sig. asymptotic 95% confidence interval CBCL scale lower bound upper bound Total Problem Scale T> Total Problem Scale T> Internalising Behavior Scale Externalising Behavior Scale