Culture Bias in Clinical Assessment: Using New Metrics to Address Thorny Problems in Practice and Research

Transcription

1 Cltre Bias in Clinical Assessment: Using New Metrics to Address Thorny Problems in Practice and Research MICHAEL CANUTE LAMBERT 1 GEORGE T. ROWAN 2 FREDRICK HICKLING 3 MAUREEN SAMMS VAUGHAN 3 1 The niversity of North Carolina at Chapel Hill 2 Michigan State niversity 3 University of the West Indies

2 Agenda Brief focs on diverse grops their challenges and strengths Measrement of fnctioning in diverse grops and concerns associated with historical and contemporary trends A look at measrement eqivalence within and across diverse grops, as well as corrective efforts Focs on content and cltral validity for diverse grops a thorny isse and how or efforts address this Some examples of how we created cltre specific measres for different grops Introdction to item response theory (IRT) and how we se it to permit cltrally appropriate and nbiased cross-grop assessment and research The complexity of IRT and its strengths

3 Challenges Diverse Grops Face Persons from diverse grops often face tremendos challenges whether they live in indstrialized nations sch as the U.S. or in lower and middle income nations Regardless of challenges they face, sch grops develop immense behavioral and emotional strengths Challenges experienced cold also contribte to stress and stress-related problems that are also grop-specific

4 Measring Fnctioning of Diverse Grops in Research and Intervention Fnctioning in diverse grops has been historically compared with persons of Eropean heritage Sch stdies se measres from North America or Erope

5 Why shold we be concerned? Or ethical principles as researchers and practitioners reqire that or clinical and research findings do not create bias for diverse grops When diverse grops are compared (e.g., Latinos or African Americans with Ero-Americans) they shold reflect tre differences or similarities between sch grops (Lambert, Fergson, & Rowan, 2016)

6 Eqivalence Langage eqivalence (often achieved throgh translation and back-translation) Conceptal Eqivalence Psychometric Eqivalence Configral Metric Scalar

7 Typical Corrective Efforts Examining factorial eqivalence (often stops here) Scalar and metric invariance stdies sometimes done Sch efforts represent a step in the right direction bt also problematic Ignores content, cltral, and conceptal validity for diverse grops

8 Content and Cltral Validity A Thorny Isse Content validity is the pillar on which other forms of validity (inclding constrct validity) rests Rarely if ever achieved withot cltral validity Specific qestions to ask as one evalates cltral validity Measres withot appropriate content and cltral validity for diverse grops can adversely impact them by yielding inaccrate research and clinical findings especially in research where grops are compared Validly assessing, stdying, and nderstanding fnctioning in diverse poplations reqire measres that are designed to be cltre specific to sch poplations and their psychometric properties shold be estimated for these poplations

9 Addressing the Challenges of Cltral Validity Cross-cltral behavioral scientists typically look at this in terms of etic vs. emic approaches Etic theories, theoretical models and measres of constrcts from sch models are applied niversally across cltral grops Emic theories, theoretical models and measres of constrcts from sch models are applied specifically to each cltral grops More recently researchers have noted that either approach is inadeqate and proposed integrating the two approaches Hardin, Robitschek, Flores, Navarro, & Ashton (2015)

10 Or Approach A combination of the joint emic and etic approach Creating measres that reflect cltral validity for each grop stdied Using new metrics to permit cltral specific measrement yet also allow nbiased measrement

11 Diversity in Specific Socioethnic Gropings Acknowledge that diversity exists even within specific ethnic grops (e.g., African American, Latino) yet common cltral characteristics and experiences are common within diverse grops (e.g., history of oppression, cltre). We arge that persons within any grops of color cold have psychological syndromes and strength dimensions that might be similar to those established for other grops Posit that individals might present their behavioral and emotional strengths and problems in a manner that is inconsistent with content depicted in items from Eropean or North American measres Constrcts for certain grops might also be configred differently from those of others

12 Or Emic Measrement Approach SOME EXAMPLES

13 The Caribbean Symptom Checklist Addressing cltral and content validity concerns items derived from data acqired from two sorces Focs grops that asked the broad qestion regarding behavioral and emotional difficlties do adlts in the general Caribbean poplation observe in their fellow citizens Stdying the presenting problems from clinic records of more than 500 adlts in more than a dozen mental health facilities (i.e., 12 otpatient and 3 inpatient facilities) Qalitative data derived from above sorces examined and themes created Test items written from sch themes

14 Conceptal Validity Identification of factors sing Lenga et al. (2001) recommendation of both rational and empirical approaches Rational approach, where 11 clinicians (i.e., three psychologists, seven psychiatrists, and one clinical social worker) groped CSC items. According to broad categorical dimensions of the DSM-IV- TR labeled Antisocial/Aggressive (setting fires, stoning people) Anxiety (heavy Arms/legs, beating/shooting sensation in head), Attention Problems (troble finishing tasks, restless), Depression (feeling dead inside, head feels heavy) Hypomania/Mania (too religios, excessive singing), and Psychosis (believe obeahed, ganja tea, rolling sensation in head) Ensring essential nidimensionality, confirmatory condcted on each dimension showed good data to model fit ( i.e., CFI and TLI 0.9, RMSEA 0.05) Frthermore, fll information factor analysis and bifactor analysis showed that items on this mltidimensional models selected by clinicians met criteria for conditional independence

15 The Behavioral and Emotional Assessment for Children of Caribbean Heritage (BEACCH) Yoth Self Report

16 Development of the BEACCH Developed with similar methodology as sed with CSC Emerged from work on the Jamaica Yoth Checklist (JYC) JYC comprised of items from the Child Behavior Checklist and items derived from presenting problems listed in clinic records of well over 600 yoth referred for clinical service BEACCH contains only items on the JYC that match those on the Behavioral Assessment of African Heritage (BACAH) and items derived from presenting problems in clinic records of more than 600 yoth

17 Derivation of Dimensions for BEACH From an early stdy where a dozen child psychiatrists and child psychiatry residents from UWI department of psychiatry and from a child gidance clinic examined items and sorted them nder categories similar to the CSC Since significant overlap exist between items on the BEACCH and CSC, item groping compared with those of CSC to frther calibrate gropings across the two measres permit se in longitdinal stdies Note that the BEACCH has three forms Parent-Report for ages 6 18 (BEACCH-P) Teacher-Report for ages 5 18 (BEACCH-T) Adolescent Self-Report for ages 11 to 18 (BEACH-A)

18 The Behavioral Assessment for Children of African Heritage A CULTURALLY VALID MEASURE FOR AFRICAN AMERICAN YOUTH

19

20 The BACAH Measres The Behavioral Assessment for Children of African Heritage Measres (BACAH) (Lambert et al., 2005; 2016) developed to address some of the problems presented in the previos slides For BACAH measres were developed (i.e. parentteacher-, and self-report measres as well as an interview schedle) Parent report assesses ages 4 to 18 Teacher report assesses from 5-18 Self report assesses Interview schedle assesses 6-10 Syndromes derived in similar fashion to those of the CSC and BEACCH Normed entirely on reports from 1,465 parents, teachers, and adolescents who reported on Black yoth fnctioning

21 Using Etic-designed Measres in Unbiased Cross-Grop (emic) Assessment? THE BACAH RESILIENCE SCALE AS AN EXAMPLE

22 Cross-Informant and Cross- Ethnic Grop Assessment Cross-ethnic grop child/adolescent assessment exact heavy brden on researcher as gold standard is cross-informant assessment Mltiformat assessment across socio ethnic grops reqires eqivalence in scores across informants and ethnicities (note that types of items asked of each informant type vary in content and composition of factor soltions might also vary across informants) Or recent stdy on eqivalence of the BACAH Resilience scales across African American and Jamaican Adolescents demonstrate eqivalence is possible (Lambert et al., 2016) How did we do this? We sed item response theory linking to place different and similar sets of items across informants and socioethnic grops on the same scale and metric

23 Introdcing Item Response Theory Defined IRT addresses the probability that informants respond affirmatively to items that match the adolescent s trait (labeled θ) IRT reqires that researchers ensre that certain assmptions are met Appropriate dimensionality (IRT fll information factor analysis) Conditional independence (IRT bifactor model) Most appropriate IRT model is selected IRT sed to identify items with significant differential item fnctioning (DIF) i.e., lack of invariance at the item level Link item parameter estimates across grops redcing bias in cross-grop assessment

24 IRT Models One parameter (1PL) model (similar to Rasch) assmes identical discrimination parameter (a) and only location parameters (b) reflecting the level of fnction items measre are estimated 2PL model permits both a and b parameters to vary (Samejima s graded model, a variant of 2PL model applied to Likert scale items sed in or stdies) 3 PL model most often sed in edcational/ achievement testing and incldes a c (gessing) parameter estimate where the probability of individals responding correctly to an item that measres higher than trait levels they possess

25 Note on a and b Parameter Estimates a parameter estimate similar to a factor loading where higher a parameter estimate reflects greater discrimination (sally ranges form 0-3 and higher a parameter) b parameter estimate are standardized scores, similar to the intercept in CFA and ranges from negative to positive infinity (typically ranges from 3 to 3) Also known as bondary parameter estimates, where one less than the nmber points on a scale for items are estimated Note that IRT can accommodate mixed response formats and several different types of models can be estimated for different grops of items in a single rn

26 Examples of ICC with Good Vs. Poor Discrimination Item Characteristic Crve a =2.66 b = Item Characteristic Crve a = b = Probability Probability b θ b θ

27 Item response Fnction for BACAH ODD Scale item 126. Uncooperative

28 IRT Differential Item Fnctioning (DIF) IRT DIF is especially applicable to testing invariance at the item level, where significant DIF reflects absence of invariance Testing for DIF in the a parameter estimate tests for each item is a test of metric invariance (significant DIF called nonniform DIF since response to an item for a grop is higher at one end of the continm and lower at the other end) Significant DIF for b parameter estimates is reflective of scalar invariance also called location DIF indicating that scores for one grop are niformly higher than that of the other grop Note that identifying items withot significant DIF are essential in placing scales from different grops on an identical metric

29 IRT Linking Derived from edcational testing, where one might pdate or extend test withot having to collect data on all old and all new items Many forms of linking exists bt the test characteristic crve method is or choice (finds linking items by identifying items withot significant DIF) Linking steps for BACAH Resilience scale across African American and Jamaican yoth parent and self-reports Identify identical items across ethnic and informant grops Obtain data from each nationality X informant grops Condct confirmatory fll information IRT factor analyses for each ethnic X informant grop Identify common items across each and se IRT analyses to identify items withot significant DIF across varios pairs of for grops Constrain items withot significant DIF and freely estimate item parameters across grops whose data are placed adjacent to each other.

30 Theoretical Cross-Grop Linking of Items Withot Significant DIF First Item Grop Second Item Grop Third Item Grop Forth Item Grop A A A B B B B C D E E E F F F G G H H H I J J K K L L L M N O O Fifth Item Grop

31

32

33 Discssion IRT is a grop of modern measrement modeling procedres that are considered to otperform traditional methods We have shown that IRT is capable of permitting cltre and informant specific measrement yet permit cross-grop comparisons with redced measrement bias

34 Why is IRT Infreqently Applied IRT very complex since most analyses occr at the item level and at least two and most often three or more parameters are estimated for each item IRT software applications complex to learn and to institte often reqiring se of at least five different software programs to arrive at reslts sch as those presented here Apart from M-Pls, I know of no other existing widely sed software packages that inclde IRT analyses So why go throgh all of this troble? Easily lends itself to compterized adaptive testing that can shorten administration time by at least ½ CAT proven to provide identical or even more accrate test reslts than traditional testing Permits easy se different sets of items on any dimension, as screening forms

35 Conclsion Researchers who condct cross-ethnic grop work bear the brden of proof that findings from their stdies have minimal measrement bias across grops Mlti-informant measrement for children and adolescents makes job more complex Frther research needed for the mltiple behavioral and emotional strengths and problems scales Procedres demonstrated in this presentation can provide a fondation for scaffolding frther research