Validity in Psychiatry Maj-Britt Posserud, MD, PhD Child and Adolescent Mental Health Haukeland University Hospital, Bergen, Norway
Validity validity is the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world. Source: Wikipedia
Reliability and Validity Nevit Dilmen.
Various forms of validity Construct validity the degree to which a test measures what it claims, or purports, to be measuring. Cronbach & Meehl (1955)» Group differences» Correlation matrices and factor analysis» Predictive value Content validity the extent to which a measure represents all facets of a given social construct Criterion validity is a measure of how well one variable or set of variables predicts an outcome
Validation 1. Define «gold standard» Reference for construct validity 2. Construct your measure content validity, face validity 3. Apply your measure Internal validity, reliability (cronbachs alpha) Factor analysis - construct/content validity 4. Compare to other measures/outcomes Measurement invariance (CFA, SEM) Convergent validity w gold standard Discriminant validity (differentiating groups) Prediction (criterion validity)
The validity of categories «Diagnostic categories defined by their syndromes should be regarded as valid only if they have been shown to be discrete entities with natural boundaries that separate them from other disorders» «valid meaning well-founded sound against which no objection can fairly be brought» «At present there is little evidence that most contemporary psychiatric diagnoses are valid. This does not mean, though, that most psychiatric diagnoses are not useful concepts. In fact, many of them are invaluable. But, because utility often varies with the context, statements about utility must always be related to context, including who is using the diagnosis, in what circumstances, and for what purposes.» «If no detectable discontinuities in symptoms are found in large tracts of the territory of psychiatric disorder, it is likely that, sooner or later, our existing typology will be abandoned and replaced by a dimensional classification» Am J Psych, 2003. Robert Kendell, M.D., Assen Jablensky, M.D.
ADHD symptoms Parental report in the BCS
ADHD DSM-IV ADHD symptoms: «present to a degree that is maladaptive and inconsistent with developmental level» «often fails to», «often loses», «often fidgets» «clear evidence of clinically significant (distress or) impairment» «not occur exclusively during» «not better accounted for by»
Where is the deviance?
Impairment Intrinsic requirement for diagnosis However, the same degree of social ineptitude will lead to varying degrees of impairment depending on context The diagnosis is not constant! Although vital for diagnosis, unspecific and not related to the underlying taxon or trait
Important concepts in screening/criterion validity: Sensitivity Screen positive sick/sick Specificity Screen negative healthy/healthy Positive predictive value Screen positive sick/screen positive Negative predictive value Screen negative healthy/screen negative Diagnostic Odds Ratio DOR Correctly classified/incorrectly classified Reciever Operating Curves
Factor analysis PCA principal component analysis Extracts all variance, including error variance EFA exploratory factor analysis CFA confirmatory factor analysis A structure is decided beforehand data is «fitted» to model» Measure of model fit SEM structural equation modelling Latent variables may also be independent variables
Latent traits, SEM and measurement error
Measurement invariance & item response theory Construct validity A measure may measure difference thing in different places In psychiatry we generally measure latent traits Factor analysis and structural equation modelling Item response theory The probability that a person with a given trait will score on a particular item
SDQ - the world s most available questionnaire! African languages: Afrikaans Arabic (+ D) Amharic Chichewa Igbo Somali Swahili Xhosa Yoruba Zulu Asian languages: Azeri (+ D) Bengali (+D) Chinese (+D) Dari Farsi Georgian Gujarati Hindi Hmong Indonesian Kannada Malayalam Pashto Thai.
SDQ in 8 countries: UK, Norway, Bangladesh, India, Yemen, Brazil, Russia
parent SDQ symptoms scores were so high in rural Yemen that the population mean of non-disordered children was comparable to that of children with a disorder in Britain. The converse was true of the final notably anomalous population the 26 Indian children with a disorder.
Conclusions: The result was that none of these SDQ caseness indicators could be used to make meaningful estimates of prevalence across the non-british samples. population-specific norms may be needed when estimating prevalence the SDQ shows large cross-cultural reporting effects and cannot be assumed a priori to be a valid method for comparing prevalences cross nationally without recourse to population-specific norms. hihglights the importance of using local cultural and linguistic knowledge when reading the DAWBA transcripts and interpreting responses to structured questions. We therefore recommend that questionnaires are only used in crosscultural comparisons when their cross-cultural equivalence has been empirically demonstrated. Doing so may require detailed diagnostic measurements that employ local and contextual knowledge in order to provide population-specific reference points for judging the performance of brief questionnaire measures.
SDQ Congo
Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among school children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry, 14(4), 208-215.
1-28% 2 9.6% 3 6.7% 4 5.2% 5 4.9% 3.55 8.69 13.61 16.69 24.75 5.58 7.38 12.64 18.64 22.59 2.80 10.80 15.80 21.48 25.69 6.74 11.67 14.52 19.48 23.70 1.67 4.75 9.77 Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among school children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry, 14(4), 208-215. 17.68 20.72
Caci, H., Morin, A. J., & Tran, A. (2015). Investigation of a bifactor model of the Strengths and Difficulties Questionnaire. European child & adolescent psychiatry, 1-11.
Receiver Operating Curves AUC Area Under the Curve
ADHD bifactor analysis One general factor Hyperactivity ADHD Impulsivity and inattention also load on separate domains ICD-10 organisation