Background 6/24/2014. Validity Testing in Pediatric Populations. Michael Kirkwood, PhD, ABPP/CN. Conflict of Interest Statement

Size: px

Start display at page:

Download "Background 6/24/2014. Validity Testing in Pediatric Populations. Michael Kirkwood, PhD, ABPP/CN. Conflict of Interest Statement"

Simon Stevenson
6 years ago
Views:

Validity Testing in Pediatric Populations Michael Kirkwood, PhD, ABPP/CN Background Board Certified Clinical Neuropsychologist at Children s Hospital Colorado Exclusively pediatric-focused Patient

neuropsychological/cognitive exam Using validity tests clinically for last 10 years Researching about for last 7 years Interest in pediatric validity testing more broadly has grown considerably over

1 Validity Testing in Pediatric Populations Michael Kirkwood, PhD, ABPP/CN Background Board Certified Clinical Neuropsychologist at Children s Hospital Colorado Exclusively pediatric-focused Patient work nearly all clinical Minimal forensic work No disability work My interest in validity testing arose out of concerns that kids being seen clinically were not providing adequate effort on neuropsychological/cognitive exam Using validity tests clinically for last 10 years Researching about for last 7 years Interest in pediatric validity testing more broadly has grown considerably over the last 10 years but still pales in comparison to that seen in adults. Conflict of Interest Statement No relevant conflicts of interest Employment: Children s Hospital Colorado Consulting: Have served with no payment on CDC Pediatric Expert Panel and the /Concussion Work Group, NINDS Common Data Elements Project. Have received payment for consulting to individual neuropsychologists and for work on several other projects. Stock ownership: No relevant investments. Research support: Principal investigator, co-investigator, or collaborator on pediatric TBI grants funded by numerous organizations including CDC, NIDRR, NIMH, Colorado TBI Trust Fund, Children s Hospital. Honoraria: Have received honoraria and expense reimbursement from multiple institutions and professional scientific bodies for presenting pediatric work at meetings, conferences, and symposia. Potential conflict of interest Royalties: I will receive royalties on the sale of a book on pediatric validity testing that I m editing (publication: 2015). Work from this book will be mentioned today. 1

2 Objective Methods to Detect Invalid Data Performance Validity Tests (PVTs) Used to detect inadequate effort or noncredible performance during testing Symptom Validity Tests (SVTs) Used to detect noncredible responding during selfreport measures Stand-Alone Tests Embedded Indicators Larrabee (2012). Performance validity and symptom validity in neuropsychological assessment. Journal of International Neuropsychological Society, 18, * Adapted from Kirkwood (2015). Introduction and rationale for incorporating validity measures into pediatric practice. In M.W. Kirkwood (Ed.). Validity Testing in the Assessment of Children and Adolescents. Guilford Press. Lots of reasons individuals display noncredible presentations during a healthcare exam Deceptive behavior underlies many Arguments I ve heard why this doesn t matter for cognitive assessment with children: kids aren t sophisticated enough to deceive kids wouldn t deceive a doctor or if they did it would be obvious kids wouldn t think to malinger 2

Deception is not unique to humans, as also occurs widely in plants and nonhuman animals Exists because it has had advantages from an evolutionary perspective and has evolved under strong natural

years) Developmental Trends in Childhood Deception < 2 Minimal evidence to suggest deceptive behavior occurs. 2-5 Across preschool years, deception increases, primarily to deny transgressions.

3 Deception is not unique to humans, as also occurs widely in plants and nonhuman animals Exists because it has had advantages from an evolutionary perspective and has evolved under strong natural selection pressures VIDEO CLIP 8 Childhood Deception Numerous studies indicate deception is common in childhood, especially in school-aged years Great deal of experimental work on topic Age (in years) Developmental Trends in Childhood Deception < 2 Minimal evidence to suggest deceptive behavior occurs. 2-5 Across preschool years, deception increases, primarily to deny transgressions. Fairly basic and unsophisticated lies Deceit grows more common. Underlying skills needed for more sophisticated deception improve. Kids better able to maintain consistency following lies to avoid self-incrimination More and more sophisticated lies, consistent with ongoing maturation of higher order cognitive abilities. Likely indistinguishable from adult deception in later teen years. Adapted from: * Peterson & Peterson (2015). Development of deception. In M.W. Kirkwood (Ed.). Validity Testing in the Assessment of Children and Adolescents. Guilford Press. * Salekin, Kubak, & Lee (2008). Deception in children and adolescents. In R. Rogers (Ed). Clinical Assessment of Malingering and Deception. 3

4 Individual Pediatric Case Reports Source Condition Child Age Lu & Boone (2002) Moderate TBI 9 yo Henry (2005) 8 yo Flaro, Green, & Blaskewitz (2007) Criminal charges 12 yo McCaffrey & Lynch (2008) TBI 13 yo Flaro & Boone (2008) 16 yo Kirkwood et al. (2010) 16 yo Chafetz & Prentkowski (2011) Medically unexplained symptoms Medically unexplained symptoms Social Security disability determination 8 yo 15 yo 13 yo 16 yo 11 yo 9 yo 11 General Pediatric Clinical Case Series Source Population N Age PVT % Noncredible Presentation Donders (2005) Mixed Neuro TOMM 2% Carone (2008) MacAllister et al (2009) Green et al. (2010) Green et al. (2010) Kirk et al. (2011) Moderate- Severe Brain Injury 38 (mean: 11.8) MSVT 5% Epilepsy TOMM 3% Mixed Neuro/Dev Mixed Neuro/Dev Mixed Neuro/Dev 380 WMT 5% 265 MSVT 3% TOMM 4% Brooks (2012) Mixed Neuro VSVT 5% 12 4

5 Pediatric Case Series: & Social Security Disability Source Population N Age PVT % Noncredible Presentation Children s Hospital Colorado Kirkwood & Kirk (2010); Kirkwood et al. (2011); Kirkwood et al. (2012); Kirkwood et al. (2013); Baker et al. (2013); Green et al. (2014); Kirk et al. (2014); Kirkwood et al. (2014) (clinical) total 8 17 MSVT + TOMM Rey FIT Various embedded measure 12 19% Araujo et al. (2014) Chafetz et al. (2007); Chafetz (2008) (clinical) Social Security Disability Claimaints (independent) RDS Digit Span TOMM MSVT 20% 48 60% (30% PVT chance level or below) 13 Why such a (relatively) high rate? Pediatric clinical cases In contrast to adult mild TBI, compensation-seeking is not the driving force in most cases (at least clinical). More typically relates to other secondary gain issues (e.g., getting out of school or homework) or psychosocial factors (e.g., increased attention) Of course, when using single PVT, will see potential false positives (i.e., no identified reason for noncredible effort and plausible data otherwise) False positives minimized through the use of multiple PVTs Note: false negatives occur too (i.e., pass validity tests but other evidence of noncredible presentation) Pediatric Social Security Disability case series Financial compensation likely strong motivator Malingering by proxy thought to be a frequent cause in explaining children s noncredible presentations MBP: Children responding noncredibly because caregivers are directing or pressuring them to. 5

Historically, reliance on subjective judgment to determine validity in pediatric evaluations Mary appeared to put forth her best effort on all tasks.

6 Historically, reliance on subjective judgment to determine validity in pediatric evaluations Mary appeared to put forth her best effort on all tasks. The results are therefore considered a reliable and valid representation of her cognitive functioning. Objective instrumentation has allowed us to move away from subjective judgments in vast majority of other domains (e.g., attention, language, memory, mood). Why should validity be different? Imagine with intelligence. Mary appeared to have below average intelligence. The results therefore indicate that she has an intellectual disability (aka, mental retardation). Problems with relying only on subjective judgment to identify invalid data Lots of literature suggests that relying on clinical judgment alone is fraught with potential for error Ziskin & Faust (1988); Faust (1988); Dawes (1994); Garb (1998) Objective methodology has clear potential of reducing error Even in our experienced clinical group, we do not rely on subjective judgment alone because we know how many classification errors would result Stand-Alone PVTs Investigated in Pediatric Populations General pediatric PVT reviews Kirkwood (2012). Overview of tests and techniques to detect negative response bias in children. In Sherman & Brooks (Eds.). (2012). Pediatric Forensic Neuropsychology. DeRight & Carone (2013). Assessment of effort in children: A systematic review. Child Neuropsychology. Kirkwood (2015). Review of PVTs and SVTs in children. In Kirkwood (Ed.). Validity Testing in the Assessment of Children and Adolescents. Guilford Press. 6

7 Embedded PVTs and Symptom Validity Tests Extensive literature in adult populations (Boone, 2007; Larrabee, 2007) Embedded indicators much smaller amount of research in pediatric populations, with mixed findings or single investigative groups examining Digit Span or Reliable Digit Span Blaskewitz et al., 2008; Kirkwood et al., 2011; Welsh et al., 2012; Harrison & Armstrong, 2014 CVLT-C Recognition Discriminability Baker et al., 2013 Symptom Validity Scale (SVS) for low functioning individuals Chafetz et al., 2007 Automatized Sequences Kirkwood et al., 2013 Even less work for symptom validity tests in context of cognitive or health-related complaints Conclusion: Neither embedded indicators nor SVTs can yet be recommended for widespread use in pediatric populations Implications of PVT Failure for Interpreting Other Data During a Cognitive Exam Multiple studies with adults have found that PVTs explain more variance in neuropsychological test scores (IQ, memory, etc.) than brain injury severity, education, age Green et al., 2001; Constantinou et al., 2005; Green, 2007; Lange et al., 2010; Meyer et al., 2011 In these samples (mostly compensation-seeking), ~50% test variance explained by PVT results Until recently, no pediatric studies: similar effects? 7

Support for idea MSVT measures effort rather than ability - No background or injury-related variable differentiated those who passed from those who failed - PVT results explained ~40% of the variance

8 Support for idea MSVT measures effort rather than ability - No background or injury-related variable differentiated those who passed from those who failed - PVT results explained ~40% of the variance across the test battery Implications of PVT Failure for Clinical Management Case of SL Demographics & Injury - 14 yo male sustained concussion in football game - Normal neurologic exam and head CT in Emergency Department - Lots of plausible symptoms in first days after injury Initial Course - Seen at 3 & 6 weeks postinjury for cognitive evaluations by a psychologist who documented severe deficits in memory & response speed from TBI - No validity tests used - Deficits used as justification to restrict from school in the name of cognitive rest Subsequent Course - Seen at 11 weeks postinjury when still not trending better and ongoing restriction from school - Noncredible effort on exam, with failure on multiple validity tests - His motivation judged to be a strong dislike of school - Not recognizing invalid cognitive results resulted in clear mismanagement 8

9 Implications of PVT Failure for Broader Systems Horner et al. (2014) recently found that PVT failure associated with increased healthcare utilization (ED visits, more inpatient services) in an adult VA sample Perhaps related to inaccurate diagnoses and inappropriate clinical management for these patients No pediatric studies on topic but we see in our program and are collecting data now Social Security Administration In 2011, for malingered mental disorders in adults, estimated cost to SSA was $20.02 billion Chafetz & Underhill (2013). Estimated costs of malingered disability in Social Security Disability examinations. ACN, 22, In 2011, for malingered mental disorders in children, estimated cost to SSA was $2.13 billion. Chafetz (2015). Disability: SSI exams for children. In In M.W. Kirkwood (Ed.). Validity Testing in the Assessment of Children and Adolescents. Guilford Press. Michael Kirkwood, Ph.D., ABPP/CN Children s Hospital Colorado Department of Rehabilitation Medicine Michael.Kirkwood@childrenscolorado.org

Background 5/1/2015. Performance Validity Testing and Management of Prolonged Recovery (in Youth) Acknowledgements. Michael W. Kirkwood, PhD, ABPP/CN

Background 5/1/2015. Performance Validity Testing and Management of Prolonged Recovery (in Youth) Acknowledgements. Michael W. Kirkwood, PhD, ABPP/CN Performance Validity Testing and Management of Prolonged Recovery (in Youth) Michael W. Kirkwood, PhD, ABPP/CN Background Clinical neuropsychologist at Children s Hospital Colorado Founded Concussion Program