TRIBAL EARLY CHILDHOOD RESEARCH CENTER Centers for American Indian and Alaska Native Health University of Colorado Anschutz Medical Campus

Transcription

1 Call in number Use access code # Use this toll-free number instead of the number provided by GoToWebinar 1 TRIBAL EARLY CHILDHOOD RESEARCH CENTER Centers for American Indian and Alaska Native Health University of Colorado Anschutz Medical Campus MIECHV Tribal Grantee Webinar April 5, 2012

2 2 Guidance for Adapting Measures from the TRC Compendium for Use in Specific American Indian / Alaska Native Communities Doug Novins Nancy Rumbaugh Whitesell Michelle Sarche

3 Before we start 3 Questions Opportunities to ask built in throughout the presentation Raise your virtual hand Type in questions Or ask questions verbally Phones on mute (when not asking a question) not on hold

4 4 Website redux

5 5 Login/Registration Registration information required Name Affiliation Address Phone Type of user Tribal MIECHV Grantee Project Director Tribal MIECHV Grantee Project Staff Tribal MIECHV Evaluator HS/EHS Director HS/EHS Teacher HS/EHS Program Staff Other Tribal Early Childhood Program Staff Tribal Early Childhood Researcher Other Terms of Use

6 6

7 Goal of this webinar 7 Describe strategies for review and adaptation of measures for use with American Indian and Alaska Native children and families What is adaptation? When is adaptation needed? What can be adapted? How should adaptation happen? What are some examples of adaptations? Verifying adaptations When adaptation is not possible

8 8 What is adaptation?

9 What is adaptation? 9 Measures developed for use with other populations may not work for American Indian/Alaska Native children and families The way the measure is administered may not be appropriate The way that questions are worded may not make sense The way that the measure is scored may not be not appropriate Important questions may be missing When this is the case, the measure may need to be changed (adapted) so that it can be used with your population of interest

10 10 When is adaptation needed?

11 When is adaptation needed? 11 When a measure doesn t include important questions, ask questions in a way that doesn t make sense, etc., the measure may not be reliable or valid for your population Validity measuring what you intend to measure Reliability measuring it consistently A measure can be reliable but not valid, but an unreliable measure cannot be valid It is important to remember that even though there are problems with some measures, many measures work quite well across populations and contexts

12 When is adaptation needed? 12 Some measures are not reliable across populations Questions get at the right thing, but are difficult for participants to understand or have a different meaning based on local use In a typical day, how often do you In the past week, I felt anxious Questions may ask about things that participants are uncomfortable answering questions because of cultural norms Asking an elder about very personal information Asking about bad things that have happened in the past, like certain traumatic events or losses

13 When is adaptation needed? 13 Some measures are not valid across populations The questions they ask are not a marker of what it is that you re trying to measure High verbal engagement as indicator of good parenting (or conversely, low verbal engagement as indicator of bad parenting) Questions that ask about important things are missing entirely A measure for family relationships may be focused on the quality of nuclear family relationships and not ask about the quality of extended family relationships A measure of early childhood social skills may not include questions about respect for elders or being observant

14 When is adaptation needed? 14 Questions may mean something else entirely A measure of stressful childhood experiences: Did child live with a relative for more than a week? Yes interpreted as a sign of early childhood stress In many Native contexts, this is normative and not a sign of stress Yes should be interpreted as indicating a close relationship with his/her relatives (important to successful child development) Indicators for many important constructs can vary culturally and this variation needs to be considered in how that construct is measured

15 When is adaptation needed? 15 When there are questions about how well a measure works for AI/AN children and families, more work is needed to determine whether adaptation is necessary Maybe the measure has never been used with this population Maybe it has been used, but not carefully evaluated so no clear understanding of whether it works well or not May need some preliminary review to determine the need for adaptation

16 16

17 What can be adapted? 17 Process Content Interpretation

18 Adaptation of Process 18 Process How or where the measure is administered, or who administers it, may need to be adapted, e.g.: trained lay assessors from within the tribal community v. outsiders example American Indian epidemiology survey assessment protocol that involves animated conversation to engage child may need to be tempered to reflect cultural protocols for more reserved interaction consideration of cultural protocols for what is appropriate to discuss with whom, based on gender, age, etc.

19 Adaptation of Content 19 Content What the questions ask about may need to be changed, e.g.: activities parents do with their children may need to include cultural events and traditions local language use, anxious definition of family negatively-worded questions questions about bad things in the past or bad potential outcomes in the future may be taboo

20 Adaptation of Interpretation 20 Interpretation Scores on measures may need to be interpreted within the cultural context Understanding the process and content of measures within the cultural context is critical, even more measures that are used as is (un-adapted), e.g.: My mother often praises me item in parent-child warmth scale

21 Un-adaptable Measures 21 Standardized and norm-referenced measures may be difficult to adapt Standardized measures depend on strict protocols for administering the measure the kinds of questions that are asked, the way the questions are asked, and how the overall measure is scored (e.g. the CLASS), and may also depend on those who administer the measure to be deemed reliable Norm-referenced tests are standardized but also norm-referenced with respect to how the scores are interpreted (e.g. how a child performs on a language test such as the PPVT is compared to other children his or her age and s/he is given a standard score that places him/her average, below average, above average, etc. with respect to his/her peers) These kinds of measures particularly those that are also proprietary may not be amenable to adaptation

22 What do with un-adaptable measures 22 Temper the interpretation of standardized or norm-referenced tests within the context of tribal communities. Consider supplementing standardized or norm-referenced test with other measures to provide a fuller/more accurate picture of outcomes in your community. Mixed methods including qualitative data may be particularly helpful here Interviewers notes Observations Debriefing interviews Focus groups Examples CBCL and BASC (Michigan State)

23 23

24 24 How should adaptation happen?

25 Who should be involved? Your Home Visitation Project Team Grantees, evaluation partners Community advisory board Purpose Oversee the measure adaptation process Share ideas for what might need to be adapted Develop materials for focus group reviews and field testing Review the results of focus groups and field testing Draft and approve the final measures

26 Who should be involved? Community Focus Groups Facilitator (researcher/evaluator) 6-8 individuals (per group) who bring their unique perspectives on the capabilities and limitations of a measure local teachers, child care or other service providers, parents, elders, etc., who can help review measures for local applicability and appropriateness Purpose Review items and measures (comprehensibility, acceptability, relevance) Help generate new items/measures

27 Who should be involved? Expert consultants early childhood care providers early childhood researchers community/cultural experts (e.g., elders) measure developers Purpose Develop new items and/or new measures to assess important constructs not covered by existing measures, e.g.: Culturally-grounded measure of young children s social competence Measure of early foundations of cultural identity development

28 28

29 What is the process? Project Team Review Project leaders sit down with evaluation partners to go through potential measures Item by item review of measure What does this item mean to me? Is the intended meaning easily understood? How would I ask this question of someone else? Might I encounter resistance in obtaining an answer to this question? How could I rephrase the question to reduce such resistance?

30 What is the process? Project Team Review (continued) Identify items that seem incongruent with the community context Such items can result in unreliability and/or invalidity Identify items that are culturally inappropriate Cultural taboos Mismatch between cultural ways and assessment process Take the time to do this step carefully and thoroughly Identify measures that may need adaptation

31 What is the process? Community Focus Group Review Provides broader input Same process as Project Team Review, but on a limited basis Targeted questions about targeted items/measures Broader sampling of interpretations of items Broader net for identifying problematic items/measures

32 What is the process? Measure Refinement (Project Team) Reviews results of focus group reviews item by item and decide whether to: 1. Include original items as is 2. Include items but modify to improve comprehensibility, acceptability, and relevance 3. Eliminate items altogether If the review of all items leads to option 1, no need for adaptation If review leads to option 2 or option 3 for at least some items, adaptation is needed We ll look at some examples next...

33 33

34 34 What are some examples of adaptations?

35 Adaptation Examples 35 Examples of Option 1 = Include original items as is Decision that available measures/items are likely to work just fine in your community Emergency room visits Infant growth markers (height, weight) Most items on the Peabody Picture Vocabulary Test (PPVT) example next page from old version of PPVT)

36 36 Show me hand

37 Adaptation Examples 37 Examples of Option 2 = Include items but modify to improve comprehensibility, acceptability, and relevance Decision that available measures/items are likely to work just fine in your community with some modification in language or context NOTE: Change items only if intended meaning is likely, or very likely, to be misunderstood with the original wording

38 Adaptation Examples Example: Substituting single words within an item spirit in item of standard measure of depression (CIDI) spirit may be very offensive in certain cultures, especially those in which witchcraft is an important cultural construct Adaptation Substituted feeling low for low spirits

39 Adaptation Examples Example: Deconstructing compound questions Has there ever been a time when you thought about your own death, someone else s death, or about death in general? (CIDI) Confusing - should they answer yes if any of the possibilities were true or only if all were true? Adaptation Broke this item into 3 separate questions Has there ever been a time when you thought about your own death? Has there ever been a time when you thought about someone else s death? Has there ever been a time when you thought about death in general?

40 Adaptation Examples Example: Inserting an introduction before a question or set of questions Standard measure of substance use includes questions about use of Peyote, classified as a hallucinogen Ceremonial use of Peyote in some Native traditions/ceremonies should not be included as abuse of a drug Adaptation Separated questions about Peyote from questions about other hallucinogens Added an introduction Peyote has an important role in the spiritual lives of many Indian people. However, like prescription drugs, there may be times that it is misused. Have you ever used peyote on your own, other than for religious purposes?

41 Adaptation Examples Example: Including alternate phrasing of an item CIDI substance abuse/dependence question - Have you ever felt such a strong desire or urge to use [substance(s)] that you could not resist it or could not think of anything else? Focus group suggested that this question would be hard for some participants to understand Adaptation Added alternate phrasing that didn t change meaning of item Have you ever wanted or needed [substance(s)] so badly that you couldn t turn it down or think of anything else?

42 Adaptation Examples 42 Recognition of some mismatches between standard measures and your context Emergency room visits may not be a clear cut indicator, depending on available services and distance to ER in your community Language development in bilingual/native-speaking families or in remote reservation community may be underestimated by standard items on the PPVT (example next slide)

43 43 Show me helicopter

44 Adaptation Examples 44 Examples of Option 3 = Eliminate items altogether Extremely rare decision Usually reserved for items that are considered to be insulting or taboo I expect my health to get worse. PPVT (next slide); cultural norms may influence the way children view the pictures in the PPVT

45 Show me accident 45

46 46 Verifying Adaptations

47 Verifying Adaptations 47 Maintain reliability and validity Validity measuring what you intend to measure Reliability measuring it consistently 1. Pre-test adaptations before using them in your evaluation pilot test to make sure they work as you intend, and that reliability and validity are maintained 2. Confirm reliability and validity in your evaluation data before proceeding to interpretation

48 48 When Adaptation is Not Possible

49 When Adaptation is Not Possible 49 Sometimes review will suggest a need for adaptation, but adaptation is not possible: norm-referenced measures some benchmark indicators time and resources needed for careful adaptation are scarce INFORMED INTERPRETATION WILL BE CRITICAL Explore reliability and validity in your evaluation data before proceeding to interpretation

50 50

51 Beyond Adapting Measures Adapting Constructs Sometimes there is no existing measure available to review and adapt What you need to measure with your children and your communities may not be relevant or as central in other communities Community mindedness What you need to measure may be fundamentally different in your communities than in other populations Spirituality Developing new measures to assess new constructs is very labor intensive and time consuming work

52 Beyond Adapting Measures Adapting Constructs First, you will need to clearly articulate what it is you need to measure: Project team discussions, conversations with community members (focus groups), consultation with experts in culture and/or child development Example: Foundations of healthy cultural identity in early childhood Once the construct is clearly articulated: Project team and expert panel work together to draft items Measure development process (pilot testing, developmental administration, psychometric analyses, measure refinement

53 Beyond Adapting Measures Adapting Constructs Don t Panic This level of construct articulation and measure development is beyond the scope of your work with the MIECHV grants BUT in identifying and articulating such constructs, you can provide guidance to the TRC about important constructs and needed measures informing future measure development efforts on a broader scope

54 54

55 Summary

56 Summary It is possible to identify, adapt, and develop culturally appropriate, reliable, and valid measures for use with American Indian people Such development efforts require combining the wisdom of American Indian communities with the specific skills offered by early childhood researchers

57 Summary 57 In making adaptations, work closely with community members and with researchers with expertise in measure development, early childhood, parenting, etc.

58 Summary 58 It is critical to maintain the reliability and the validity of the measure and to verify both before proceeding to use an adapted measure. Using a measure that is adapted to be culturally appropriate but is unreliable or invalid will: Result in data that are un-interpretable Undermine your ability to accurately evaluate outcomes

59 59

60 TRC Contact Info 60 Doug Novins, M.D. Director of Consultation, Collaboration, & Community Director of Research-to-Practice Michelle Sarche, Ph.D. Director of Research Training & Information Dissemination Nancy Rumbaugh Whitesell, Ph.D. Director of Research & Measurement