Author's response to reviews Title: Reliability and validity of the adolescent stress questionnaire in a sample of European adolescents - the HELENA study Authors: Tineke De Vriendt (tineke.devriendt@ugent.be) Els Clays (els.clays@ugent.be) Luis A Moreno (lmoreno@unizar.es) Patrick Bergman (patrick.bergman@gu.se) Germán Vicente-Rodriguez (gervicen@unizar.es) Eniko Nagy (eniko.e.nagy@aok.pte.hu) Sabine Dietrich (sabine.dietrich@meduniwien.ac.at) Yannis Manios (manios@hua.gr) Stefaan De Henauw (stefaan.dehenauw@ugent.be) Version: 3 Date: 3 May 2011 Author's response to reviews: see over
Ghent, May 3 rd, 2011 To the editor of BMC Public Health Dear editor, On behalf of all co-authors, I would like to thank the editorial board of BMC Public Health for the revision of the manuscript entitled Reliability and validity of the adolescent stress questionnaire in a sample of European adolescents the HELENA study for potential publication in BMC Public Health. The authors also acknowledge the efforts of the reviewers in revising this manuscript. We are pleased to see that the reviewers have appreciated the considerable work that was done in addressing all the comments and that two out of three reviewers have no major comments left. Concerning the third reviewer, the authors do appreciate his point of view in his remaining comments and some analyses were changed as suggested by the reviewer. A detailed description of this revision process is attached below. All authors have read and approved the revised manuscript. We are at your and the reviewer s disposal for further clarifications if necessary. We kindly ask you to consider this revised manuscript for publication in BMC Public Health. Looking forward to a reply from the editorial board, please accept our best regards. Yours sincerely, Tineke de Vriendt Corresponding author Ghent University, Department of Public Health De Pintelaan 185, 2BlokA B-9000 Ghent Belgium Tel: +32 9 332 36 25 Fax: +32 9 332 49 94 email: tineke.devriendt@ugent.be
Answer to the reviewer s comments REVIEWER 1 Reviewer's report Title: Reliability and validity of the adolescent stress questionnaire in a sample of European adolescents - the HELENA study Version: 2 Date: 23 March 2011 Reviewer: Martin Teicher Reviewer's report: Major Compulsory Revisions This revision is improved and provides a more conservative presentation of the results. Indeed, this revision does a good job pointing out the weaknesses of the instrument. The major question is whether or not the instrument has sufficient criterion validity to be useful. Gender differences, which supposedly provide the strongest support, do not necessarily indicate criterion validity. An alternative explanation is that the instrument is inherently gender-biased. This concern is similar to the author s supposition that differences between adolescents in the participating European cities may reflect cultural-biases. The ASQ's failure to show age, pubertal status or financial affluence affects (and to a lesser degree associations with cortisol) raise serious concerns about what the scale measures and it's fidelity. There should be sufficient power with a sample of this size to detect even small effects. The key strengths of the scale are: (1) that 50% of the stress component scales, and the overall summary score, have Cronbach alpha > 0.8; (2) that the results provide modest confirmation that the factor structure of the translated scale corresponds to the factor structure previously reported by Byrne et al, though this model fails dramatically by chi-square criteria, and (3) that there are gender differences on the summary score and most subscales. These limited strengths are counterbalanced by insufficient test-retest reliability on almost all subscales, and inability to detect age, pubertal status and financial effects. These weaknesses raise doubts about the author s contention that this is a promising instrument. An alternative perspective that cannot be ruled out by the available data, is that this is potentially a gender-biased, and culturally-biased instrument with insufficient test-retest reliability, and inadequate sensitivity to detect age, pubertal and family affluence affects on adolescent stress. This alternative possibility should be presented. The authors responded well to the previous recommendation that the data be analyzed in a way that can detect quadratic effects in addition to linear effects. However, they did not respond to the suggestion to used mixed effect models. ANCOVA analyses have become an increasingly antiquated approach, and while they are suitable in many situations they are not well-suited in this case. Data of this type are most appropriately analyzed using hierarchical mixed effect models in which subjects are nested within cities. Failure to analyze the data in this way violates the independence assumption of the ANCOVA design, as their data shows that subjects nested within the same city are more similar in their scores than subjects in different cities. It would be very valuable to ascertain whether there are detectable effects of
age, pubertal status and financial affluence using hierarchical mixed effect models, as the failure to detect such effects calls the value of this scale into question. The authors may be better served by reevaluating the scale using item response theory rather than classical psychometric test theory (though this would be a separate study and paper). IRT analysis on a subsample may enable them to better capitalize on their efforts by developing a new scale from a subset of questions that show greater criterion validity, and which can then be verified in the remaining sample. Level of interest: An article whose findings are important to those with closely related research interests Quality of written English: Needs some language corrections before being published Statistical review: Yes, and I have assessed the statistics in my report. Declaration of competing interests: I declare that I have no competing interests. Answer: We first want to thank the reviewer for acknowledging the efforts we have made during the first revision and for taking time to evaluate the revised version of this paper. We understand the point of view saying that the lack of effects of the socio-demographic characteristics on the variability in the ASQ scores provides insufficient support to the validity of the ASQ. Since the data were nested in cities, you make a good point that performing ANCOVAs violates the assumption of independency and thus hierarchical linear models (HLM) were a better alternative. Initially we did not have the intention to use HLM because the sample size of the HELENA study was not calculated for investigating cross-cultural differences. However, based on your suggestion we have now performed HLM with the cities as groups to reveal contributions of the socio-demographic characteristics to the ASQ scores. Despite the few groups (six), we felt confident in performing HLM since we had a large average sample size across groups (n=190) and since no group-level predictor variables were considered. We also performed the Restricted Maximum Likelihood method, which can provide reasonable variance estimates with even few groups (Hox J. Multilevel Analysis. Techniques and Applications. Lawrence Erlbaum Associates, 2002: New Jersey). This new statistical analysis is now mentioned in the Methods-section: Page 16, line 17: Contributions of socio-demographic characteristics (gender, age, pubertal stage, SES) on the variability in ASQ scale scores and the summary score were investigated using Hierarchical Linear Models (HLM) with Restricted Maximum Likelihood Estimation [22], whereby city was specified as subject grouping variable, and gender, age, pubertal stage and SES as fixed parameters. Results of the HLM, investigating the independent contributions of socio-demographic characteristics to the variance in ASQ scores, indicated that gender (girls had higher scores than boys) and pubertal stage (post-pubertal adolescents had higher stress scores than those still in pubertal development) systematically contributed to the variance of the majority of ASQ scales and the summary score. However, age was less convincingly a significant negative predictor of the ASQ scales (only for two scales). Overall, these results indicated slightly stronger predictive values of the investigated socio-demographic characteristics, when compared to the ANCOVAS
performed, and confirm the concern made by the reviewer. These new results are now mentioned in the manuscript: Page 18, line 17: The independent contributions of socio-demographic variables to the variance in ASQ scale scores and the summary score were investigated using HLM (Table 3 (see additional file: Table 3)). Intercept-only models demonstrated that city as grouping variable had a proportional explained variance of 6.7% for the summary score, while for the separate scales these proportional explained variances varied from 1.3% to 10.9%. Results of the HLM showed that mainly gender and pubertal stage systematically contributed to the variance of the majority of ASQ scales and the summary score: boys and those in a pre-, beginning-, mid-, or advanced pubertal stage (stage I to IV) reported lower stress scores compared to girls or those in a post-pubertal stage (stage V), respectively. Age and SES in general contributed to a lesser extent to the variance in ASQ scale scores and the summary score: age was only a significant negative predictor for stress from teacher interaction, peer pressure, and the summary score, while SES was a negative predictor for stress from teacher interaction and financial pressure. We would like to challenge the reviewer s remark that the gender differences in ASQ scores may indicate that the ASQ is probably gender-biased. It is documented that gender differences in stress experiences are common (Matud, 2004 (see reference list of paper)) and that they are also present in adolescents (Rudolph K. 2002). In addition, the gender difference was also clearly found in the original study from Byrne et al., suggesting similar functioning of the ASQ in our population and confirming a known-groups discriminant validity of the ASQ. The differences in stress experience across the adolescents pubertal development was previously documented in literature (ref) and this is now also mentioned more in detail in the discussion section. The fact that our results confirm these previous findings gives support to the discriminant validity of the ASQ. Age was observed to be no systematic predictor of the ASQ scores, suggesting that the ASQ functions similarly across the whole age range (12.5-17.5 years). This is not in agreement with observations of Byrne et al., where bivariate correlations with age were weak positive for five out of ten scales (significant correlations varied from 0.12 to 0.35) (Byrne et al., 2007). Also, no contributions of the SES on the variability in ASQ scores were revealed but this was already discussed in the discussion section. Based on these new results, the authors have reconsidered their discussion section and reformulated their conclusion to reflect more the Results and Discussion section of the paper. The reviewer was correct by saying that the previous conclusion was too positive by saying that the ASQ was a promising tool. Page 20, line 14 (Discussion): The present study demonstrated that adolescent girls experienced higher stress levels than boys, confirming the gender differences in the ASQ as presented by Byrne et al. [6], and confirming previously reported gender differences in adults [26] and adolescents [27]. In addition, the present study demonstrated that pubertal stage was also a predictor for the ASQ, with the adolescents that are already in a post-pubertal stage of development having higher scores than those still in a pre-pubertal stage or in full pubertal development. This is in line with the hypothesis of Dahl and Gunnar that pubertal development is the driving force behind increasing stress sensitivity during adolescence [28] and with findings from Sumter et al. that the biological stress sensitivity increased with pubertal status [29]. The differential
functioning of the ASQ according to the adolescents gender and pubertal stage give strength to the validity of the ASQ. Age was no systematic predictor of the ASQ scores, suggesting that the ASQ functions similarly across the whole age range (12.5-17.5 years). This is not in agreement with observations of Byrne et al., where bivariate correlations with age were weak positive for five out of ten scales (significant correlations varied from 0.12 to 0.35) [6]. This might be indicative of differential scales functioning across cultures (see below). Page 25, line 22 (Conclusions): The present study demonstrated an acceptable internal reliability and scale construct of the ASQ when it was implemented in a European adolescent sample. In addition, significant independent contributions of gender and pubertal stage to the variance in the ASQ were established. These strengths were counterbalanced by a poor test-retest reliability and criterion validity of the ASQ, probably because of methodological shortcomings, and failure to demonstrate effects of socio-economic status. Based on these shortcomings, the utility of the ASQ within an European adolescent sample is uncertain and further research concerning these aspects is required. In addition, supplementing the ASQ with questions concerning previous (severe) lifetime stressful events is recommended in order to have a more complete picture of adolescent chronic stress exposure. The abstract of the manuscript was also modified in agreement with above-mentioned changes. To shortly come back to the criterion validity (association of the ASQ scores with BWSF cortisol), the study sample was drastically reduced, with an average of 33 cases per city, which resulted in serious loss of power when performing MLA or when controlling for city. Therefore, we decided to keep with the linear regression analysis controlling only for pubertal stage, since it was our main intention to investigate the pure criterion validity of the ASQ when compared with the cortisol values among this reduced study sample. To respond to the last remark, reevaluation of the scale constructs using the items response theory is indeed pertinent (and we acknowledge the reviewer for this since we might consider to do so), but this concerns a topic for a separate study and paper, as already indicated by the reviewer himself.
REVIEWER 2 Reviewer's report Title: Reliability and validity of the adolescent stress questionnaire in a sample of European adolescents - the HELENA study Version: 2 Date: 18 March 2011 Reviewer: Soraya Seedat Reviewer's report: Discretionary revision: I would suggest chnaging 'image' on line 18, page 25 to 'picture'. Level of interest: An article of importance in its field Quality of written English: Acceptable Statistical review: No, the manuscript does not need to be seen by a statistician. Declaration of competing interests: None. Answer: The authors sincerely want to thank the reviewer for her efforts to review the revised version of our manuscript and to acknowledge the importance of this manuscript. The writing suggestion is modified accordingly (see page 23, line 2).
REVIEWER 3 Reviewer's report Title: Reliability and validity of the adolescent stress questionnaire in a sample of European adolescents - the HELENA study Version: 2 Date: 16 March 2011 Reviewer: Silvina Berra Reviewer's report: I appreciate and thank the work made by authors answering my observations and trying to achieve almost all my suggestions. The improvement is evident and it does more clear the manuscript. I observe one unique point which might be even considered as a discretionary revision. The cross-country adaptation of the instrument is not completely informed here and I had not access to the cited article in the short time available to make this report. On the base of what is explained here, it seems that the equivalence between local versions was not tested by the recommended methods (e.g. cognitive interviews to analyse if the expected meaning underline responses in each culture). I would like to know the conclusions of authors about this point. Level of interest: An article whose findings are important to those with closely related research interests Quality of written English: Acceptable Statistical review: Yes, but I do not feel adequately qualified to assess the statistics. Declaration of competing interests: I declare that I have no competing interests. Answer: You make a pertinent remark by saying that the equivalence between local versions was not tested by recommended methods such as cognitive interviews or questionnaire appraisal tools. We established a standardized procedure of translation and backtranslation by independent professional interpreters/translators and the backtranslated version was compared with the original English version to ensure semantic equivalence between the translated versions and the original English version. If not equivalent, the translators were advised to reconsider the translation. The translators were also advised to use a language which is comprehensible for a wide range of adolescents and to check the comprehensibility in some adolescents. We aimed at using local versions with a high linguistic equivalence. This whole process was coordinated by one person. However, explaining this whole procedure in detail would take a separate paper. It could be that when performing for instance a cognitive interview according to standardized procedures, we would find differences in the cognitive appraisal of the questionnaire items between the different centers. This could be addressed by further culturally adapting the questionnaire, but then one gets six versions which could be slightly different, making pooling of the data even more difficult. Our intention was to have an image of the level of perceived stress among European adolescents and not to compare between different centers or cultures, since the sample size was not calculated for investigating cross-cultural differences. This also explains why further culturally adapting the questionnaire was not a priority in our study.