Outcomes, Indicators and Measuring Impact

Transcription

1 J-PAL Africa Executive Education Course Outcomes, Indicators and Measuring Impact Lecture 2, January 23, 2012 Isaac M. Mbiti Southern Methodist University

2 Course Overview 1. Why evaluate? What is evaluation? 2. Outcomes, indicators and measuring impact 3. Impact evaluation why randomize 4. How to randomize 5. Sampling and Sample Size 6. Analysis and inference 7. RCTs: Start to Finish

3 Lecture Overview Setting & some background information on interventions Goals of Measurement Practical issues Design decisions Human subjects

4 Background: Motivation Youth unemployment is a major economic and social problem in Africa Youth account for 60% of the unemployed and 72% of adolescents live on less the $2/day (World Bank, 2009) In Kenya: Young adults currently entering the labor force (individuals aged 15-29) account for 30% of the Kenyan population. Survey data shows approx 21% of youths aged are unemployed, and a further 25% are neither in school nor working (Source 2005 KIHBS) Youth underemployment has been linked to violence during the 2007 post-election crisis

5 Background: Study Sample Participants recruited from the Kenya Life Panel Survey (KLPS) sample Random sample of participants from Primary School Deworming Project (Miguel and Kremer, 2004) and Girls (Merit) Scholarship Program (Kremer, Miguel and Thornton, 2009) Representative longitudinal dataset All originally from Busia District and a majority still located in the district; but extended to people relocated to urban areas of Kenya as well (Kisumu, Nairobi, Mombasa)

6 Background: Intervention Design KLPS sample individuals were invited to meetings A random half of meetings included an information intervention Participants completed an application to the program Among eligible applicants (2,163), 50% were awarded no voucher, 25% were awarded a restricted public-only voucher, 25% were awarded an unrestricted voucher Voucher value was about $460, enough to cover all or nearly all fees (but not covering incidentals like transport and boarding)

7 10,767 youth from KLPS invited to info meeting 2,705 attend on of 70 meetings 35 meetings (50%) given info treatment Randomization Process 2,163 attend 2nd meeting They submit letter of support and course preferences Individuals stratified in 36 groups Gender x PSDP/GSP x Info treat x course

8 Background: Voucher Intervention Information treatment = Yes Information treatment = No Unrestricted (public or private) Voucher Type Restricted (public only) None ` Total ,108

9 Background: Information Intervention Mincerian returns to vocational education were presented in 35 of 70 initial program meetings, based on standard cross-sectional analysis using existing KLPS data Returns highlighted the higher earnings associated with certain trades, including maledominated trades over female-dominated trades More subjective persuasive elements included a video about a couple female auto mechanics and a discussion about the benefits of vocational education and the idea of women taking up maledominated trades

10 Background: Research Project Goals Examine the impacts of vocational training using a randomized evaluation design coupled with a unique panel dataset The labor market returns to vocational training Differential returns to attending private vocational training centers The spatial distribution of returns to educational investments Demand for vocational training Heterogeneous treatment effects by gender, cognitive ability, household wealth, ruralurban

11 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

12 Relevance How do we ensure we measure the relevant outcomes?

13 Relevance: Drawing the chain of causality We want to answer more than: how effective is the intervention? We also want to answer: why it is effective? We want to draw the link inputs intermediary outcomes primary outcomes Defining and measuring intermediate outcomes will enrich our understanding of the program, reinforce our conclusions, and make it easier to draw general lessons 13

14 Relevance: The Critical Role of Theory 1. Map out a theory of change 2. Use the theory to generate hypotheses that you can test in your project 3. What (theoretical) final outcomes and indicators are needed to demonstrate the validity of hypothesis 4. What (theoretical) intermediary indicators are needed to distinguish various hypotheses?

15 Relevance: Defining key hypotheses What might be examples of a few key hypotheses to test in the following? Malaria bed-nets Conditional cash transfer Micro-credit Which variables, or combinations of variables, might you use to test these key hypotheses? 15

16 Relevance: An Example using Vocational Training What are the possible chain of outcomes for the vocational training program? Information? Vouchers?

17 Relevance: Theory of Change for voucher treatment Youth would like to enroll in vocational training but they are poor and credit constrained enrollment is low Providing vouchers of any kind (i.e. scholarships) will allow youth to circumvent credit constraints and enroll large increases in enrollment

18 Relevance: Theory of Change for voucher treatment Recall there are 2 types of vouchers: Public school only Public or private school What is the theory of change here?

19 Relevance: Theory of Change for information treatment Youth do not know the returns to vocational training (maybe voc-ed is stigmatized) Youth also don t know which trades are lucrative Providing information could boost demand for vocational training and change course selection.

20 Relevance: Hypothesis from Vocational Training intervention What variables should we try to obtain to better our understanding of the program? Intermediary? Final?

21 Relevance to Reality Our theory and hypothesis helps us define the set of outcomes and variables Often these are basic indicators which are relatively straightforward (e.g. height and weight) In many cases it is difficult to translate theoretical measures into real variables Cognitive ability, non-cognitive ability Risk aversion, impatience, selfishness Happiness, Income, savings,

22 Relevance to Reality Good news: often no need to reinvent the wheel. Bad news: often not clear if such measures are implementable in your setting and context. Need to be properly adapted to context.

23 Hints on outcomes and indicators Choose those with a reasonable chance of being moved within the evaluation timeline Chose those that are not too difficult to collect and measure Chose those that occur with enough frequency to detect an impact given your sample size

24 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

25 Sources of Measures First task is to define your study sample Who should be in your sample? How representative is sample? (representative of who?) Sampling issues depend on research project Voc-ed we our sample was drawn from youth who were in the KLPS panel data Pros? Cons?

26 Sources of Measures Where do you obtain such data? Two main sources Administrative data Survey data

27 Sources of Measures Administrative data: Advantages? Often cheap, fast, easy and clean (although you would be surprised how often there are errors) Disadvantages? Data may not be relevant Variables of interest may not be in the data The data may not correspond to the relevant time period May be difficult to get permission May not come in the format you want May be aggregated,

28 Sources of Measures Survey data (collected by you) Advantages: Control- you can ensure its relevant and comprehensive and collected at the right time Disadvantages: Heavy time investment required Management, designing, piloting and refining Expensive

29 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

30 Reliability and Accuracy Regardless of data sources we need to make sure that the data is reliable and accurate. Recall the cliché: Junk in -Junk out Unreliable data false or misleading conclusions

31 Reliability Administrative data: General perception is that this source should be reliable but it is often full of errors Important to understand HOW the data is collected and the CONTEXT automated data collection generally high quality (e.g. M-pesa transfer records, phone records) Important to think or ask whether the data could be manipulated for someone s gain E.g. school enrollment records when schools get per capita student grant

32 Reliability Ensuring reliability in survey data collection is very important and time consuming Probably the most important part is Piloting

33 Piloting Good surveys are developed by trial and error Often it s good to start with a very basic set of questions asked in an open-ended way More of a qualitative or focus group style Over time, the lessons from this can be refined into a survey Good to find out that respondents don t understand certain issues before it s too late 33

34 Piloting Good piloting will almost always raise questions that weren t thought of before Sometimes the research design can be changed slightly to capture these issues 34

35 Piloting As the survey becomes more formalized, working through the details becomes important Phrasing of questions Skip codes ( If no, skip to. Translation issues 35

36 Reliability: Paper, Phone, PDA, Tablet or Netbook With prices of electronics falling more surveys are collected via PDAs etc Regardless of data collection method it is necessary to implement relevant protocol and procedures to ensure data quality.

37 Reliability: Paper Survey The paper survey process: Survey printed on paper filled in by enumerator data entry electronic dataset Where can this go wrong?

38 Reliability: Paper Survey The paper survey process: Survey printed on paper filled in by enumerator data entry electronic dataset Where can this go wrong? Everywhere!

39 Reliability: Paper surveys Include ID number on all pages in case pages get separated The survey should look nice- easier for the enumerator to fill it out Minimize complicated skip patterns: i.e. don t have instructions like if the answer to the question 5 pages ago was `no, skip this question. Minimize data entry errors: double entry and reconcile differences 39

40 Reliability: Electronic Survey The electronic survey process: Paper survey created programmed into netbook filled in by enumerator electronic dataset Where can this go wrong?

41 Reliability: Electronic Survey The electronic survey process: Paper survey created programmed into netbook filled in by enumerator electronic dataset Where can this go wrong? Everywhere

42 Reliability: Electronic surveys Follow paper design tips Thoroughly test the electronic version before launch (esp. skip patterns etc) Make sure it is easy to fill out (stylus vs. typing) Program logical consistency checks A male should not be a mother, daughter etc Test these checks Protocol to transfer data from pda/laptop to central database efficiently and routinely.

43 Reliability: Electronic Vs Paper Trade off that should be evaluated on case by case Electronic surveys: Huge upfront investment (time etc) Makes more sense for surveys with many respondents Less flexible (harder to change things on the go) Faster- (no need to do data entry) Less error (if programmed well) Risky for enumerators to carry?

44 Reliability: Survey Design Though it s hard to come up with a real set of rules for designing a survey, there are a few important bullet points which would be good to keep in mind to think about Some apply to paper based surveys on 44

45 Reliability: General Tips Collect accurate tracking information Directions to home, GPS location, etc. It might be a different person doing the followup Get correct mobile phone number including that of parents, siblings and friends Surveys should be clear and should not leave room for interpretation by the enumerator. 45

46 Reliability: Surveys Training enumerators in these procedures is essential. Create manuals for all survey instruments. Good to do regular back checking resurveying respondents to make sure that they were actually interviewed Re-survey sample of respondents on a random basis 46

47 Reliability: General Tips Enumerators have to understand why these questions are being asked This is especially true for questions that aren t as simple For example, we might be asking about a respondent s assets and she ll say she doesn t have a TV even though the enumerator can see it right there 47

48 Reliability: General Tips A lot of times it s easy for a respondent to always say I didn t do something because I don t have enough money even though there s actually a deeper reason as well Important for the enumerator to understand what we re trying to get at with all these questions 48

49 Reliability: The Right respondent It is important to ensure that you obtain the information from the right respondent Sometimes this is simple: in voc-ed example the individual youth would be the right person Suppose you are collecting household data or farm data Women in the household may know more about children Men may know more about household assets. 49

50 Survey Design: Social desirability bias Similarly, people might feel badly to tell the truth about some socially undesirable subjects How many drinks did you have last week? Do you always use a condom when having sex? We call this social desirability bias Almost 99% of voc-ed sample wants to start their own business 50

51 Reliability: Manipulation of Data Often occurs when there is something valuable at stake CCT eligibility on the basis of a poverty score CCT payouts on the basis of attendance Teacher incentives on the basis of attendance measured by head-teacher Voc-ed voucher winners selling their vouchers Can lead to false conclusions

52 Reliability: Manipulation Various strategies can be used to circumvent this problem CCTs and enrollment and teacher attendance: unannounced spot checks can be used Voc-ed: took pictures of students and also did spot checks Lesson: measures should be designed to be free of manipulation In some cases we can use the cross-checks to measure the extent of manipulation

53 Reliability: Recall issues Some variables are easier to remember than others e.g. birth of a child Often have to collect data that involves recall where things aren t so easy: Time use, consumption, health, finances, exam scores How to ensure reliability?

54 Reliability: Recall Issues Easiest: shorten the recall period One strategy is to do high frequency short surveys where you ask respondents to recall events from prior day. Diaries: e.g. time-use or financial diaries Can be difficult to implement- respondents don t always fill them out (or fill them out when you ask for them) Administrative data: e.g. for test scores One survey on nutrition had enumerator live with family and weighed each individuals meals prior to eating and the amount wasted

55 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

56 Innocuous The act of measurement can actually influence behavior of respondents Live in enumerator measuring each person s food Financial diaries If treatment and control group respond differently to measurement then it would be problematic

57 Innocuous Recall the random spot check method for school attendance in CCTs Baird et al: compare UCTs and CCTs Outcome of interest is school attendance Suppose they conducted random spot checks could this affect integrity of their research?

58 Innocuous Recall the random spot check method for school attendance in CCTs Baird et al: compare UCTs and CCTs Outcome of interest is school attendance Suppose they conducted random spot checks could this affect integrity of their research? YES: if the spot checks send a signal to UCT group that they should go to school

59 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

60 Feasibility Budget adequately Things go wrong Exchange rate movements Enumerator downtime Resurveys needed New opportunities

61 Feasibility Financial resources tradeoff between sample size and amount of information obtained from each household Human resource capacity of organization implementing the survey research coordinators, interviewers, data entry staff Willingness and ability of respondents to provide desired information

62 Feasibility As with any job, managing surveyors creates its own problems Sometimes it s possible to contract out survey work to a firm which specializes in it Less hassle, but less control over data quality Important to find a really competent firm 62

63 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

64 Timely When should you collect outcomes and variables? Related to theory and hypotheses Crucial question is how long do you need for intervention effects to materialize?

65 Research design: Baseline Although you can sometimes avoid collecting baseline data it is very risky to skip this step. Sample might not be balanced, especially if sample isn t very big Having a baseline allows the researcher to control for baseline characteristics E.g. initial test scores are very important to have in education Baseline allows you to look at effects for subgroups Also contributes to reliability 65

66 Research design : Follow-up The amount of data that needs to be collected at end line really depends on the type of experiment that is being run Sometimes take-up (not adoption or some other similar outcome) itself is the outcome of interest Voc-ed: measure enrollment each semester, completion, test performance, and then labor market outcomes 66

67 Research design : Follow-up Other times, getting follow-up data is crucial Theory suggests that voc-ed impacts may take time to materialize This required multiple follow-ups over several years and also tracking of individuals Need to plan appropriately for appropriate timing of the follow-up is also key Will it take some time for the intervention to take off? Will people migrate? 67

68 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

69 Human subjects An oft-neglected part of research is human subject approvals Respondents do not have to answer any questions and their rights have to be respected above all else It is important to remember that human subjects research is meant to be designed such that respondents are volunteering their time to the survey They can t be induced to answer questions they don t want to with punishments (or huge rewards) Treating subjects with dignity / complying with all review panels MUST come before the research

70 Human subjects Check what approvals needed Country IRBs Research permits University IRBs Permissions National government Local authorities Relevant ministry Remember this takes time!

71 Consent Respondents must be read or given a consent form which explicitly tells them what will be asked of them, how much time it will take, what the risks are, what the rewards are Contact information to complain if something goes wrong Usually need written consent, though sometimes oral is allowed This can be intimidating to people but is a necessary part of the process and a good discipline device for the researcher too

72 Keys to Successful Measurement Relevance Appropriate source Reliability and Accuracy Innocuous Feasible Timely Ethical

73