Jessica Mazza, MSPH. 25th Annual Children s Mental Health Research & Policy Conference March 5, 2012

Transcription

1 Jessica Mazza, MSPH University of Illinois at Chicago; Chestnut Health Systems 25th Annual Children s Mental Health Research & Policy Conference March 5, 2012

2 Outline: Introduction: Externalizing Disorders Behavioral Complexity Scale (BCS) Psychometric Validation (using Rasch) Prevalence & Co-morbidity Analysis Longitudinal Methods & Results Trauma Factors GVS Longitudinal Item Response Analysis Conclusions Implications & Suggestions for Future Research

3 What are externalizing disorders? Disorder of behavioral control / disruption Attention-deficit/hyperactivity disorder (ADHD) Oppositional Defiant Disorder Conduct disorder Nearly 50% of all child/adolescent psych referrals in the US are for an externalizing disorder. Increasing number of adults being treated for externalizing disorder.

4 Attention Deficit Hyperactivity Disorder Prevalence 6-9% in children/adolescents ~ 5% for adults Two clusters of symptoms: Inattentive Hard time paying attention, easily distracted, lose things frequently Hyperactivity/Impulsivity Running around, interrupting conversations, unable to stay seated

5 Oppositional Defiant Disorder Symptoms include: Arguing with adults Losing one s temper Deliberately annoying other people Considered a precursor to conduct disorder Also thought to be a more mild form of conduct disorder Cannot be diagnosed with both ODD & CD simultaneously

6 Conduct Disorder Typically considered a childhood disorder Median onset of 12 years Severe behavioral symptoms Destruction of property Severe violation of behavioral rules Truancy, running away from home, arson

7 Conceptual Framework of BCS

8 Measuring Externalizing Disorders Child Behavioral Checklist (CBCL) Screens for externalizing symptoms, but not diagnostic criteria Ratings scales for ADHD Conners Rating Scale, Brown Rating Scale No rating scales exist for conduct disorder Only lengthy diagnostic interviews (Diagnostic Interview for Children; Kiddie-Schedule for Affective Disorder)

9 Need for externalizing scale: High co-occurrence of conduct disorder & ADHD (estimates of 30-50%) Relatively high prevalence of externalizing behaviors in children and adults Appropriate to have a measure to screen for all externalizing disorders at once Externalizing behaviors are strongly linked to adolescents and adults entering substance abuse treatment

10 Behavioral Complexity Scale (BCS) Subsection of the Global Assessment of Individual Needs (GAIN), administered to people entering substance abuse treatment. The GAIN is an evidenced-based chemical dependency assessment that supports a number of substance abuse treatment practices, including initial screening; brief interventions, and referrals to ancillary services. Self-report, structured interview of past-year symptoms related to externalizing disorders Attention deficit, hyperactivity/impulsivity, and conduct disorder. 33 items, all DSM-IV-TR diagnostic criteria for the above disorders; additional item for recency

11 Item Stem/Response Categories During the past 12 months, have you done the following things two or more times? Response option: Yes or No

12 BCS Subscales Inattentive Disorder Scale (IDS) 9 items Hyperactivity/Impulsivity Scale (HIS) 9 items Conduct Disorder Scale (CDS) 15 items

13 Inattentive Disorder Scale 1. Made mistakes because you were not paying attention. Y N 2. Had a hard time paying attention at school, work, or home. Y N 3. Had a hard time listening to instructions at school, work, or home. Y N 4. Not followed instructions or finished your assignments. Y N 5. Had a hard time staying organized or getting everything done. Y N 6. Avoided things that took too much effort, like school or paperwork. Y N 7. Lost things that you needed for school, work, or home. Y N 8. Been unable to pay attention when other things were going on. Y N 9. Been forgetful or absentminded. Y N

14 Hyperactivity Impulsivity Scale 10. Fidgeted or had a hard time keeping your hands or feet still when you were supposed Y N to. 11. Been unable to stay in a seat or where you were supposed to stay. Y N 12. Felt restless or the need to run around our climb on things. Y N 13. Gotten in trouble for being too loud when you were playing or relaxing. Y N 14. Felt like you were always on the go or driven by a motor. Y N 15. Talked too much or had others complain you talked too much. Y N 16. Gave answers before the other person finished asking the question. Y N 17. Had a hard time waiting for your turn. Y N 18. Interrupted or butted into other people s conversations or games. Y N

15 Conduct Disorder Scale 19. Been a bully or threatened other people. Y N 20. Started physical fights with other people. Y N 21. Used a weapon in fights. Y N 22. Been physically cruel to other people. Y N 23. Been physically cruel to animals. Y N 24. Taken a purse, money, or other things from a person by force. Y N 25. Forced someone to have sex with you when they did not want to. Y N 26. Set fires. Y N 27. Broken windows or destroyed property. Y N 28. Taken money or things from a house, building, or car. Y N 29. Lied or conned to get things you wanted or to avoid having to do something. Y N 30. Taken things from a store or written bad checks to buy things. Y N 31. Stayed out at night later than your parents or partner wanted. Y N 32. Run away from home (partner) for at least one night. Y N 33. Skipped work or school. Y N

16 Diagnostic Criteria All symptom counts are based on DSM-IV-TR criteria. 6 items on IDS endorsed consider ADHD- Inattentive type 6 items on HIS endorsed consider ADHD- Hyperactive/Impulsive Type 3 items on CDS endorsed consider conduct disorder

17 Application of the BCS It can be used by clinicians and substance use/mental health professionals to help diagnose and monitor externalizing disorders for individuals entering and receiving substance abuse treatment. It can help inform research practices that support the improvement of substance abuse/mental health service delivery for individuals with co-occurring externalizing disorders and substance use problems.

18 Rasch Analysis of the BCS Summary Statistics Dimensionality Wright Map Item/Person Fit Statistics DIF Analysis

19 Rasch Dichotomous Model The dichotomous model estimates the probability that a respondent will choose a particular response category for an item as: Pni ln 1 P ni = B n D i where ln is the natural logarithm, P ni is the probability of respondent n endorsing item i, 1-P ni is the probability of respondent n not endorsing item i, B n is the person measure of respondent n, D i is the difficulty of item i.

20 Rasch - Sample & Methodology Sample 7,435 cases from 77 substance abuse treatment studies Mean age was 19 years old (SD = 8.9) 67% male 45% Caucasian, 26% AA, 10.8% Hispanic, 17.7% mixed/other Methods Data were analyzed using Winsteps version

21 Summary Statistics (Non- Extreme Persons) SUMMARY OF 5532 MEASURED (NON-EXTREME) PERSONS RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.49 TRUE SD 1.25 SEPARATION 2.54 PERSON RELIABILITY.87 MODEL RMSE.47 TRUE SD 1.26 SEPARATION 2.67 PERSON RELIABILITY.88 S.E. OF PERSON MEAN = MAXIMUM EXTREME SCORE: 1 PERSONS MINIMUM EXTREME SCORE: 1856 PERSONS LACKING RESPONSES: 46 PERSONS VALID RESPONSES: 99.7%

22 Summary Statistics (All Persons) SUMMARY OF 7389 MEASURED (EXTREME AND NON-EXTREME) PERSONS RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE 1.02 TRUE SD 2.22 SEPARATION 2.18 PERSON RELIABILITY.83 MODEL RMSE 1.01 TRUE SD 2.23 SEPARATION 2.20 PERSON RELIABILITY.83 S.E. OF PERSON MEAN = PERSON RAW SCORE-TO-MEASURE CORRELATION =.96 (approximate due to missing data) CRONBACH ALPHA (KR-20) PERSON RAW SCORE RELIABILITY =.94 (approximate due to missing data)

23 Summary Statistics (Items) SUMMARY OF 33 MEASURED (NON-EXTREME) ITEMS RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.04 TRUE SD 1.55 SEPARATION ITEM RELIABILITY 1.00 MODEL RMSE.04 TRUE SD 1.55 SEPARATION ITEM RELIABILITY 1.00 S.E. OF ITEM MEAN = UMEAN=.000 USCALE=1.000 ITEM RAW SCORE-TO-MEASURE CORRELATION = -.96 (approximate due to missing data) DATA POINTS. LOG-LIKELIHOOD CHI-SQUARE: with d.f. p=1.0000

24 Dimensionality Analysis Table of STANDARDIZED RESIDUAL variance (in Eigenvalue units) -- Empirical -- Modeled Total raw variance in observations = % 100.0% Raw variance explained by measures = % 37.7% Raw variance explained by persons = % 18.6% Raw Variance explained by items = % 19.1% Raw unexplained variance (total) = % 100.0% Unexplned variance in 1st contrast = % 9.1% Unexplned variance in 2nd contrast = % 7.0% Unexplned variance in 3rd contrast = % 4.9% Unexplned variance in 4th contrast = % 4.5% Unexplned variance in 5th contrast = % 3.8% * >30% is considered a moderate measurement dimension, >40% strong

25 Dimensionality (Simulated) Table of STANDARDIZED RESIDUAL variance (in Eigenvalue units) -- Empirical -- Modeled Total raw variance in observations = % 100.0% Raw variance explained by measures = % 38.1% Raw variance explained by persons = % 16.3% Raw Variance explained by items = % 21.8% Raw unexplained variance (total) = % 100.0% 61.9% Unexplned variance in 1st contrast = % 3.5% Unexplned variance in 2nd contrast = % 3.4%

26 6. + FORCEDSEX TRUNCATED +.. T 3 + CRULANML... TAKMONEY T 2. + SETFIRES. WEAPON. S.. 1.# +.# S DSTROYDP PHYSCRUL TAKEFROM.# RESTLESS STOLSTOR.# RUNAWAYO STARTFIG TALKTOOM.### BULLY FELTONTH STAYSEAT 0.## +M.## LOUDPLAY WAIT.# INTERRUP.# M FIDGET.### -1.# + ANSWRB4Q LISTNINS LOSTTHIN.# AVOIDEFF LIEDCONN SKIPSCHO STAYORG UNABLEPA.# ABSENTMI PAYATTNS.# S FOLLOWIN.# S -2.# + STAYOUT2..#. MISTAKES EACH '#' IS TRUNCATED -5.############ + <less> <frequ> Wright Map Red Items: CDS Green Items: HIS Purple Items: IDS

27 Infit/Outfit

28 7 ITEMS 6 Less Measures More FORCEDSEX CRULANML TAKMONEY SETFIRES WEAPON PHYSCRUL DSTROYDP TAKEFROM RESTLESS STOLSTOR STARTFIG TALKTOOM RUNAWAYO STAYSEAT FELTONTH BULLY LOUDPLAY WAIT FIDGET INTERRUP LISTNINS LOSTTHIN AVOIDEFF STAYORG ANSWRB4Q UNABLEPA LIEDCONN SKIPSCHO PAYATTNS ABSENTMI FOLLOWIN STAYOUT2 MISTAKES Overfit Infit Mean-square Underfit

29 Misfitting Response Strings MOST MISFITTING RESPONSE STRINGS ITEM OUTMNSQ PERSON high SKIPSCHO 1.38 A WEAPON 1.33 B CRULANML 1.31 C STAYOUT D TALKTOOM 1.21 E RUNAWAYO 1.21 F ANSWRB4Q 1.17 G FORCEDSEX 1.17 H STOLSTOR 1.17 I STARTFIG 1.08 J SETFIRES 1.05 K INTERRUP 1.05 L FELTONTH 1.04 M DSTROYDP 1.03 N BULLY 1.02 O TAKEFROM 1.02 P PHYSCRUL 1.01 Q MISTAKES 1.00 p LOSTTHIN.97 o TAKMONEY.94 n STAYORG.99 m LIEDCONN.94 l LOUDPLAY.94 k FOLLOWIN.91 j RESTLESS.91 i FIDGET.90 h low

30 Unexpected Responses

31 Misfits

32 Differential Item Functioning Youth vs. adults Easier for youth to endorse: 4. Not followed instructions or not finished your assignments. (IDS) 26. Set fires. (CDS) 27. Broken windows or destroyed property. (CDS) 33. Skipped work or school. (CDS) Easier for adults to endorse: 9. Been forgetful or absentminded. (IDS) 14. Felt like you were always on the go or driven by a motor. (HIS) 25. Forced someone to have sex with you when they did not want to. (CDS) No significant DIF by gender or by drug of choice. Since youth endorsed two serious items much more than did adults (26 & 27), they will tend to have somewhat higher measures. Some more serious and more age-appropriate conduct disorder items could be created for adults.

33 DIF (by age) ITEM DIF by Age Group (<18 vs. >18) DIF Measure (diff.)

34 DIF (by gender) ITEM DIF Measure (diff.) Male Female

35 Conclusions: Rasch Analysis The BCS is a unidimensional measure and functioned well in this sample of persons being assessed for substance abuse. A dimensional measure could be used as an alternative and, perhaps, as an improvement, to categorical diagnoses Using the Rasch item hierarchy, develop cut-off scores based on the severity calibrations of the items when the person score patterns fit the model.

36 Conclusions: Rasch Analysis Some DIF exists for age and gender, and could be further investigated. The results suggest the possibility of some important age differences that, if adjusted for, may reduce the possibility of bias in comparing the measures of youth vs. adults.

37 Prevalence and Comorbidity Analysis 6 items on IDS endorsed consider ADHD- Inattentive type 6 items on HIS endorsed consider ADHD- Hyperactive/Impulsive Type 3 items on CDS endorsed consider conduct disorder

38 Diagnoses of BCS Number of Cases IDS Only HIS Only ADHD-Combined Only CD Only CD & IDS CD & HIS CD & Combined Diagnosis

39 CD Comorbidity Rates Diagnosis Number of Cases CD only CD & IDS CD & HIS CD & Combined 996 (26.7% of CD) 1351 (36% of CD) 153 (4.1% of CD) 1225 (36% of CD) All CD (N) 3725 *Does not include those with only the 3 low-level items (4.9% of CD)

40 CD Comorbidity Rates Number of Cases 33% 4% 27% CD only CD & IDS only CD & HIS only CD & Combined 36%

41 Low-severity CDS items 3 CDS items were low-severity 29. Lied or conned to get things you wanted or to avoid having to do something. 31. Stayed out at night later than your parents or partner wanted. 33. Skipped work or school. Having just the 3 low-level CD items: 183/7435 = 2.5% Of CD diagnoses: 183/3725 = 4.9%

42 Conclusions: CD Symptom Counts Three CDS items are placed low on the measure (StayOut2Late, SkipSchool, and LiedConned) and endorsement of these 3 items alone not necessarily conduct disorder. Caveat: Nothing can replace clinical judgment.

43 Longitudinal Analysis Hierarchical Linear Model N=975 Investigation of effect of traumagenic factors (GVS) on improvement of BCS symptoms over time

44 Sample Description VARIABLE NAME N MEAN SD MINIMUM MAXIMUM MALE WHITE BLACK HISPANIC ALCOHOL MARIJUAN AGEATINT Data collected from 6 sites: intake, 3 mos, 6 mos, 9mos, 12 mos

45 Rasch Application of HLM Level-1 Model: log 1 p ij P ij = β 0 j + β1 j 1 j + β 2 j 2 j +... and the Level-2 Model: β β 0 j β = γ 1 j ( k 1) = 00 = γ γ + u 10 0 j ( k 1)0 X X ( k 1) j ( k 1) j In this formula, p ij is the probability that person j answers item i correctly, and X ij is the ith indicator variable for person j (values 1 when the item is for the ith item, and 0 otherwise). The coefficient β oj is the intercept term, and β 1j is associated with X ij. The term u oj is a random component of β oj and is distributed as N(0, τ). U 0j is assumed to have a normal distribution. + β X 45

46 Selection of Victimization Indicators Anyone ever hurt you Anyone forced you to have sex Emotional abuse Before 18 Prolonged abuse Multiple people Trusted one People didn t believe you Worried someone will abuse you emotionally Got help for trauma Currently worried about being victimized Sex, emotional, physical attack Any days of victimization in the past 90 days

47 Selection of Victimization Indicators Recently attacked Prolonged abuse Emotional abuse Anyone ever hurt you Anyone forced you to have sex Before 18 Multiple people Trusted one People didn t believe you Worried someone will abuse you emotionally Got help for trauma Currently worried about being victimized Sex, emotional, physical attack

48

49 Reliability Estimates Random level-1 coefficient Reliability estimate INTRCPT1, B OBS, B

50 Final estimation of fixed effects: (Unit-specific model) Standard Approx. Fixed Effect Coefficient Error T-ratio d.f. P-value For INTRCPT1, B0 INTRCPT2, G VEMOTABU, G VPROLONG, G VWHENLAS, G BLACK, G MARIJUAN, G AGEATINT, G For OBS slope, B1 INTRCPT2, G VWHENLAS, G MALE, G AGEATINT, G

51 Odds Ratios Odds Confidence Fixed Effect Coefficient Ratio Interval For For INTRCPT1, B0 INTRCPT2, G (0.234,0.304) VEMOTABU, G (1.045,1.463) VPROLONG, G (1.102,1.398) VWHENLAS, G (1.220,1.410) BLACK, G (0.373,0.570) MARIJUAN, G (0.598,0.884) AGEATINT, G (0.775,0.931) OBS slope, B1 INTRCPT2, G (0.798,0.822) VWHENLAS, G (1.007,1.026) MALE, G (1.019,1.075) AGEATINT, G (0.954,0.979)

52 Final Estimation of Variance Final estimation of variance components: Random Effect Standard Variance df Chi-square P-value Deviation Component INTRCPT1, U OBS slope, U

53 Discussion Replication of item hierarchy Good reliability and evidence for validity Implications for DSM-IV Trauma Significant predictor of externalizing behaviors Explore items for change in time by trauma (potential DIF) Results suggest that substance treatment decreases externalizing behaviors, especially for CD items

54 Suggestions for future research Reproduce analysis in a different sample Mixed ages (to allow youth vs. adult comparisons) More females Substance of choice Any others?

55 Thank you! Upcoming article in Psychological Assessment: Dimensionality, Hierarchical Structure, Age Generalizability, and Criterion Validity of the GAIN s Behavioral Complexity Scale More information about the GAIN: Acknowledgements Mike Dennis, PhD Ken Conrad, PhD Karen Conrad, PhD Barth Riley, PhD Rod Funk, BS Mark Stein, PhD

56

57

58 Mistakes StayOut Instruct Forget PayAtten AvoidEff PayAttnThingsGoOn StayOrg LiedConned AnswerB4Qx HardListen SkipSchool Lost Things Interrupt Fidget WaitTurn TooLoud Bully StaySeat Onthego BadCheck TalkMuch RunAway Fights Restless CrulPpl TakeHouse DstryPrprty Weapon SetFires TakeMoney CrulAnml ForcedSex

59 Cognitive Interview Findings Some items hard for adults to endorse: set fires run away from home for at least one night stayed out at night later than your parents or partner wanted had a hard time waiting for your turn Problem wording: driven by a motor been a bully written bad checks Some items considered double-barreled : not followed instructions or finished your assignments items had a hard time staying organized or getting everything done