Measuring impact of a gender sensitisation program among children: Lessons from data collection in Haryana, India

Transcription

1 Measuring impact of a gender sensitisation program among children: Lessons from data collection in Haryana, India Tarun Jain Indian School of Business, Hyderabad, India September 2, 2014 Tarun Jain (ISB) Measuring impact Sep / 65

2 Gender attitudes Extensive discrimination against women throughout life-cycle However, traditional economic incentives do not appear make major impact Gender attitudes among adults tend to be very stable Small window for changing gender attitudes among adolescents Can a school-based gender sensitisation program change attitudes? Designed and implemented by Breakthrough 150 treatment schools, 164 control schools in 4 districts in Haryana Targeting students in 7th, 8th and 9th grades in government schools Program from 2014 to 2016 academic years Tarun Jain (ISB) Measuring impact Sep / 65

3 Gender attitudes Extensive discrimination against women throughout life-cycle However, traditional economic incentives do not appear make major impact Gender attitudes among adults tend to be very stable Small window for changing gender attitudes among adolescents Can a school-based gender sensitisation program change attitudes? Designed and implemented by Breakthrough 150 treatment schools, 164 control schools in 4 districts in Haryana Targeting students in 7th, 8th and 9th grades in government schools Program from 2014 to 2016 academic years Tarun Jain (ISB) Measuring impact Sep / 65

4 Study location Tarun Jain (ISB) Measuring impact Sep / 65

5 Study location Tarun Jain (ISB) Measuring impact Sep / 65

6 Measuring attitudes Randomized control trial (RCT) design requires measuring baseline attitudes How should we measure gender attitudes among thousands of students? What is a good way to measure attitudes among children? Appropriate tests for year olds Robust to context Reliable with consistent responses Avoiding social desirability bias Implementation challenges - scope, institutions, geography, personnel, time and expense Tarun Jain (ISB) Measuring impact Sep / 65

7 Measuring attitudes Randomized control trial (RCT) design requires measuring baseline attitudes How should we measure gender attitudes among thousands of students? What is a good way to measure attitudes among children? Appropriate tests for year olds Robust to context Reliable with consistent responses Avoiding social desirability bias Implementation challenges - scope, institutions, geography, personnel, time and expense Tarun Jain (ISB) Measuring impact Sep / 65

8 Measuring attitudes Randomized control trial (RCT) design requires measuring baseline attitudes How should we measure gender attitudes among thousands of students? What is a good way to measure attitudes among children? Appropriate tests for year olds Robust to context Reliable with consistent responses Avoiding social desirability bias Implementation challenges - scope, institutions, geography, personnel, time and expense Tarun Jain (ISB) Measuring impact Sep / 65

9 Haryana school Tarun Jain (ISB) Measuring impact Sep / 65

10 Haryana school classroom Tarun Jain (ISB) Measuring impact Sep / 65

11 The Main Challenge in Measurement Getting Accuracy and Precision More accurate!! More precise Tarun Jain (ISB) Measuring impact Sep / 65

12 Biased and Unbiased are used to describe Accuracy Biased On average, we get the wrong answer Unbiased On average, we get the right answer Tarun Jain (ISB) Measuring impact Sep / 65

13 Noisy and Precise are used to describe Precision Noisy Random error causes answer to bounce around Precise Measures of the same thing cluster together Tarun Jain (ISB) Measuring impact Sep / 65

14 Tradeoffs between measurement precision and accuracy More accurate!! More precise Tarun Jain (ISB) Measuring impact Sep / 65

15 Tradeoffs between measurement precision and accuracy A surveyor doesn t follow the exact phrase of the question: Surveyor: You feel unsecure in this neighborhood, right? Respondent: Well, sometimes yes and sometimes, well, no Surveyor: So, is more of a yes, right? Respondent: mmm...well Surveyor: Ok, thanks. Next question. Does this introduce noise or bias? Tarun Jain (ISB) Measuring impact Sep / 65

16 Error in Accuracy In theory: How well does the indicator map to the outcome? (e.g. IQ test intelligence) In practice, are you getting unbiased answers? Respondent bias Recall bias Anchoring bias Social desirability bias Framing effect / Neutrality Tarun Jain (ISB) Measuring impact Sep / 65

17 Error in Precision In theory: The measure is unbiased, but not necessarily accurate In practice, are you getting precise answers? Length, fatigue How much did you spend on broccoli yesterday? (as a measure of annual broccoli spending) Ambiguous wording (definitions, relationships, recall period, units of question) For example, Definition of terms household, income Recall period/units of question Type of answer - Open/Closed Choice of options for closed questions Likert (i.e. Strongly disagree, disagree, neither agree nor disagree,...) Rankings Quantitative scales. Numbers or bins. Tarun Jain (ISB) Measuring impact Sep / 65

18 Random Error Mistakes in estimation or recall Question wording Surveyor training/quality Data entry Length, fatigue of survey Tarun Jain (ISB) Measuring impact Sep / 65

19 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

20 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

21 How much juice did you consume last month? a. Less than 2 liters b. 2 to 5 liters c. 6 to 10 liters d. More than 11 liters Tarun Jain (ISB) Measuring impact Sep / 65

22 1. Things people do not know very well What Anything to estimate, particularly across time. Prone to recall error and poor estimation Examples: distance to health center, profit, consumption, income, plot size Strategies Frame the question in a way people are likely to know and remember. Consistency checks How much did you spend in the last week on x? How much did you spend in the last 4 weeks on x? Multiple measurements of same indicator How many minutes does it take to walk to the health center? How many kilometers away is the health center? Tarun Jain (ISB) Measuring impact Sep / 65

23 How many glasses of juice did you consume yesterday? a. 0 b. 1 to 3 c. 4 to 6 d. More than 6 Tarun Jain (ISB) Measuring impact Sep / 65

24 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

25 How frequently do you yell at your spouse? a. Daily b. Several times per week c. Once per week d. Once per month e. Never Tarun Jain (ISB) Measuring impact Sep / 65

26 2. Things people do not want to talk about What Anything socially risky or something painful Examples: sexual activity, alcohol and drug use, domestic violence, conduct during wartime, mental health Strategies Dont start with the hard stuff! Ensure comfort and privacy of respondent Consider asking question in third person Tarun Jain (ISB) Measuring impact Sep / 65

27 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

28 I feel more empowered now than last year a. Strongly disagree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly agree Tarun Jain (ISB) Measuring impact Sep / 65

29 3. Abstract concepts What Potentially the most challenging and interesting type of difficult-to-measure indicators Examples: empowerment, bargaining power, social cohesion, risk aversion Three key steps when measuring abstract concepts Define what you mean by your abstract concept Choose the outcome that you want to serve as the measurement of your concept Design a good question to measure that outcome Often choice between choosing a self-reported measure and a behavioral measure both can add value! Tarun Jain (ISB) Measuring impact Sep / 65

30 I am involved in the decision to send my child to private vs public school a. Strongly disagree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly agree Tarun Jain (ISB) Measuring impact Sep / 65

31 Perceptions and Attitudes Ask directly How effective is the leader of your village? (ineffective, somewhat effective, effective, very... ) Indirect approaches often have better accuracy Listen to a Vignette (Male v. Female) Revealed preference (For example, voting behavior) Implicit Association Tests Tarun Jain (ISB) Measuring impact Sep / 65

32 Implicit Association Tests People simplify the world for efficiency Use thumb rules to draw connections May not even be aware themselves For some important outcomes, may be worth trying to measure these indirectly Implicit Association Tests are one technique Tarun Jain (ISB) Measuring impact Sep / 65

33 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

34 What proportion of individuals of race X are denied jobs due to racial bias? a. No one b. 1 to 20% c. 21 to 40% d. 41 to 60% e. More than 60% Tarun Jain (ISB) Measuring impact Sep / 65

35 Things that are not (always) directly observable What: You may want to measure outcomes that you cant ask directly about or directly observe Examples: Corruption, fraud, discrimination Strategies Do a literature review to borrow other peoples clever ideas Audit studies, e.g. Rajasthan police experiment to register cases, Delhi Drivers Licenses, Delhi doctors Multiple sources of data, e.g. inputs of funds vs. outputs received by recipients, pollution reports by different parties Tarun Jain (ISB) Measuring impact Sep / 65

36 Hard to measure concepts 1 Things people do not know very well 2 Things people do not want to talk about 3 Abstract concepts 4 Things that are not (always) directly observable 5 Things that are best directly observed Tarun Jain (ISB) Measuring impact Sep / 65

37 Things that are best directly observed What: Behavioral preferences are more believable than anything that is said Example: Asking principal about school attendance vs. measuring yourself Strategies Develop detailed protocols Ensure data collection of behavioral measures is consistent across all participants Tarun Jain (ISB) Measuring impact Sep / 65

38 Evaluating a gender sensitisation program in India: Three ways to measure attitudes 1 Vignettes 2 Direct attitude questions 3 Implicit Association Test Tarun Jain (ISB) Measuring impact Sep / 65

39 Vignette There exists a lower middle class family a village. The family consists of three children, their parents and grandparents. Among the three children the two elder ones Rakhi and Rajat are twins. They have a younger brother Ramesh. Rakhi and Rajat have just passed their HSC exams with 80% marks. Both of them have aspirations to go to the nearby town and study in a good college. But that will require them to stay in a hostel in the town independently. The family has got just enough money to send only one of their two children to the town. They also have another younger son to take care of. Finally their father decides that Rajat should continue his studies whereas Rakhi will stay in the home and help her mother in the household chores and eventually get married. Tarun Jain (ISB) Measuring impact Sep / 65

40 Vignette responses Agrees with sending Rajat(boy) to school Agrees with sending Rajat(boy) to school Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

41 Vignette responses Who would you send to school: Rajat Who would you send to school: Rajat Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

42 Vignette responses Who would you send to school: Rakhi Who would you send to school: Rakhi Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

43 Vignette responses If Rakhi was a better student, send: Rajat If Rakhi was a better student, send: Rajat Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

44 Vignette responses Agrees staying alone in the town is not safe for Rakhi Agrees staying alone in the town is not safe for Rakhi Jhajjar Sonipat Rohtak Panipat 0.57 Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

45 Vignette responses Agrees Rakhi needs to be married off as she is 18 years old Agrees Rakhi needs to be married off as she is 18 years old Jhajjar Sonipat Rohtak Panipat 0.63 Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

46 Direct questions on gender attitudes Measured on a five point scale A woman s most important role is being a good homemaker A man should have the final word about decisions in his home A woman should tolerate violence to keep her family together Wives should be less educated than their husbands Boys should get more opportunities/resources for education than girls Parents should give dowry for their girl s marriage Men and women should get equal opportunities in all spheres of life - education For the most part, it is easier to be a man than to be a woman Girls should be allowed to study as far as they want Daughters should have a similar right to inherited property as sons It would be a good idea to elect a woman as the village Sarpanch Tarun Jain (ISB) Measuring impact Sep / 65

47 Survey Operations Tarun Jain (ISB) Measuring impact Sep / 65

48 Five-point scale 1. बह त ठ क बह त ख श 2. ठ क ख श 3. न ठ क न गलत न ख श न द ख 4. गलत द ख 5. बह त गलत बह त द ख Tarun Jain (ISB) Measuring impact Sep / 65

49 Survey Operations Tarun Jain (ISB) Measuring impact Sep / 65

50 Results Girls should be allowed to study as far as they want Girls should be allowed to study as far as they want Jhajjar Sonipat Rohtak Panipat 0.95 Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

51 Results Wives should be less educated than their husbands Wives should be less educated than their husbands Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

52 Implicit Association Test (IAT) Standard computer-based test to detect the strength of a person s automatic association between mental representations of objects (concepts) in memory For example, how quickly do we associate women with humanities and men with engineering relative to men with humanities and women and engineering? Categorization task done quickly Software measures the speed of response Helps overcome social desirability bias in measuring attitudes Extremely sensitive to context (words, images, language etc.) No IAT for measuring gender attitudes among year olds in government schools in Haryana Tarun Jain (ISB) Measuring impact Sep / 65

53 Implicit Association Test (IAT) Standard computer-based test to detect the strength of a person s automatic association between mental representations of objects (concepts) in memory For example, how quickly do we associate women with humanities and men with engineering relative to men with humanities and women and engineering? Categorization task done quickly Software measures the speed of response Helps overcome social desirability bias in measuring attitudes Extremely sensitive to context (words, images, language etc.) No IAT for measuring gender attitudes among year olds in government schools in Haryana Tarun Jain (ISB) Measuring impact Sep / 65

54 Implicit Association Test (IAT) Standard computer-based test to detect the strength of a person s automatic association between mental representations of objects (concepts) in memory For example, how quickly do we associate women with humanities and men with engineering relative to men with humanities and women and engineering? Categorization task done quickly Software measures the speed of response Helps overcome social desirability bias in measuring attitudes Extremely sensitive to context (words, images, language etc.) No IAT for measuring gender attitudes among year olds in government schools in Haryana Tarun Jain (ISB) Measuring impact Sep / 65

55 !"#$%&# &)*+&,-#%+$-&*.,&-"&/.-+0"123+&.,&+$56.2$+%&7+6"8 & 9$.:56+,&";&2:.0+,&.$%&8"1%, & Tarun Jain (ISB) Measuring impact Sep / 65 &

56 Tarun Jain (ISB) Measuring impact Sep / 65

57 Tarun Jain (ISB) Measuring impact Sep / 65

58 Tarun Jain (ISB) Measuring impact Sep / 65

59 Tarun Jain (ISB) Measuring impact Sep / 65 &

60 !+,-.)& )01')23,.'-3)1#2)3+)4#3'5+678')#2)'$%&#7-'.)9'&+: )) ) Tarun Jain (ISB) Measuring impact Sep / 65 )

61 Tarun Jain (ISB) Measuring impact Sep / 65

62 Tarun Jain (ISB) Measuring impact Sep / 65

63 Tarun Jain (ISB) Measuring impact Sep / 65

64 Tarun Jain (ISB) Measuring impact Sep / 65 )

65 Taking the Implicit Association Test Tarun Jain (ISB) Measuring impact Sep / 65

66 IAT D-measure D measure (Implicit preference for boys) D measure (Implicit preference for boys) Jhajjar Sonipat Rohtak Panipat Boy Girl Tarun Jain (ISB) Measuring impact Sep / 65

67 Correlation between IAT and Direct attitude questions Gender index (from direct attitude questions) D measure * D measure x Girl 0.053*** Observations 5375 Tarun Jain (ISB) Measuring impact Sep / 65

68 Takeaways from this session Measurement matters! Both accuracy and precision are important for good measurement Tradeoffs between accuracy and precision Recognize and work around measurement challenges Creativity, reading literature are important for better measurement Tarun Jain (ISB) Measuring impact Sep / 65

69 Thank you and stay in touch! Assistant Professor of Economics Indian School of Business Hyderabad, India Phone: (40) Web: Twitter: LinkedIn: Tarun Jain (ISB) Measuring impact Sep / 65