Randomized Evaluations in Agriculture January 9 11, 2012

Transcription

1 Randomized Evaluations in Agriculture January 9 11, 2012 Abdul Latif Jameel Poverty Action Lab and Agricultural Technology adoption Initiative training at CIMMYT

2

3 Table of Contents Agenda...3 Biographies of J-PAL and ATAI Lecturers...5 List of Participants...7 Group Presentation Guide..9 Course Material Group Work: Drafting Theory of Change on Adoption and Impact...11 Case Study 1: How to Randomize...13 Case Study 2: Threats to Experimental Integrity...17 Exercise 1: Mechanics of Randomization...23 Exercise 2: Sample Size and Power...31 Group Presentation Template...39 Checklist for Reviewing Randomized Evaluations of Social Programs...41 Impact Evaluation Glossary

4

5 Randomized Evaluation in Agriculture January 9-11, 2012 Day 1 9:00-9:10 Welcome and introduction Bekele Shiferaw 9:10-9:20 Intro: Introduction to ATAI and J-PAL Rachel Glennerster 9:20-9:30 Official opening -Thomas Lumpkin, Director General, CIMMYT 9:30-10:45 Lecture: Using randomized evaluations to test adoption constraints Rachel Glennerster COFFEE BREAK 11:00-12:00 Lecture: From Adoption to Impact Jonathan Robinson 12:00-1:00 Group Work: Drafting Theory of Change on Adoption and Impact LUNCH - CIMMYT Guest House 2:00-3:00 Lecture: Alternative Strategies for Randomizing Programs Jenny Aker COFFEE BREAK 3:15-4:30 Group Work: Randomization Design - Case study: Fertilizer and BlueSpoon 4:30-6:00 Group Work: Randomization Design - Exercise: Sampling and mechanics of randomization (excel and stata) 6:15-8:00 Welcome dinner CIMMYT Guest House Day 2 9:00-9:30 Recap: from Day 1 Jonathan Robinson 9:30-10:45 Lecture: Power and Sample size for Clustered Randomized Trials Rachel Glennerster COFFEE BREAK 11:00-12:00 Group Work: Power calculations - Exercise: Calculate sample size for complex designs 12:00-1:00 Group Work: Maximizing power within a budget constraint - Exercise: Determine sample with a budget constraint LUNCH - CIMMYT Guest House 2:00-3:00 Lecture: Managing and Minimizing Threats to Analysis Jenny Aker 3

6 COFFEE BREAK 3:15-4:30 Group Work: Managing Threats - Case study: TNS agronomy training 4:30-5:30 Non-randomized approaches for impact evaluation Girma Tesfahun 5:30 6:00 Group Work: Project Work - Work on evaluation design Day 3 9:00-9:30 Lecture: Common Pitfalls Jenny Aker 9:30-10:45 Lecture: Randomized Evaluation: Start-to-finish Jonathan Robinson COFFEE BREAK 11:00-12:00 Non-randomized approaches for impact evaluation Vijesh Krishna 12:00-1:00 Group Work: Project Work Work on evaluation design LUNCH - CIMMYT Guest House 2:00-3:00 Group Work: Project Work (cont.) Work on evaluation design COFFEE BREAK 3:15-4:30 Group Presentations 4:30-4:45 Wrap up 4

7 J-PAL Lecturers Jenny Aker Assistant Professor of Economics Tufts University Jenny C. Aker is an Assistant Professor of Economics at the Fletcher School and Department of Economics at Tufts University. Jenny works on economic development in Africa, with a primary focus on the impact of information and information technology on development outcomes, particularly in the areas of agriculture, agricultural marketing and education; the relationship between shocks and agricultural food market performance; the determinants of agricultural technology adoption; and impact evaluations of NGO and World Bank projects. Rachel Glennerster Executive Director J-PAL Global Rachel Glennerster is Executive Director of J-PAL. Her current research includes randomized evaluations of community driven development in Sierra Leone, empowerment of adolescent girls in Bangladesh, and health, education, and microfinance in India. She oversees J-PAL s work to translate research findings into policy action and helped establish Deworm the World of which she is a board member. Jonathan Robinson Assistant Professor of Economics University of California, Santa Cruz Jonathan Robinson is an assistant professor of economics at the University of California, Santa Cruz. His research is primarily focused in sub-saharan Africa, and includes studies of how individuals cope with risk, a project to understand why farmers do not adopt potentially profitable agricultural technologies, and several studies of small businesses in Kenya. His current work includes evaluations of various strategies to improve health outcomes in poor countries.. 5

8

9 List of Participants # CIMMYT Based in 1 Asfaw Negassa ETHIOPIA 2 Bekele Shiferaw KENYA 3 Damaris López MEXICO 4 Dave Hodson ETHIOPIA 5 Federico Carrión MEXICO 6 Frederick Rossi BANGLADESH 7 Girma Tesfaye Kassie ZIMBABWE 8 Hugo De Groote KENYA 9 Jaleta Moti Debello ETHIOPIA 10 Javier Becerril MEXICO 11 Jon Hellin MEXICO 12 Kindie Tesfaye ETHIOPIA 13 Laura Donnet MEXICO 14 Menale Kassie Berresaw KENYA 15 Mulugetta Mekuria ZIMBABWE 16 Olaf Erenstein ETHIOPIA 17 Sika Gbegbelegbe KENYA 18 Surabhi Mittal INDIA 19 Tina Beuchelt MEXICO 20 Vijesh Krishna INDIA Training Team 21 Angela Ambroz USA 22 Ben Jaques USA 23 Bryan Plummer USA 24 Emilio Chavez MEXICO 25 Jenny Aker USA 26 Jon Robinson USA 27 Rachel Glennerster USA 28 Sam Bazzi USA 7

10

11 Group Presentation Participants will be placed into 3-5 person groups which will work through the design process for a randomized evaluation of an intervention that considers technology adoption or final impact. Groups will be aided in this project by both the faculty and teaching assistants with the work culminating in presentations at the end of the week. The goal of the group presentations is to consolidate and apply the knowledge of the lectures. We encourage groups to work on projects that are relevant to participants organizations. All groups will present on Wednesday. The 15-minute presentation is followed by a 15- minute question-answer session led by J-PAL affiliates and staff. We provide groups with template slides for their presentation (see next page). While the groups do not need to follow this exactly, the presentation should have no more than 9 slides (including title slide, excluding appendix) and should include the following topics: Brief project background Theory of change Evaluation question Outcomes Evaluation design Data and sample size Potential validity threats and how to manage them Dissemination strategy of results Please time yourself and do not exceed the allotted time. We have only a limited amount of time for these presentations, so we will follow a strict timeline to be fair to all groups. 9

12

13 Drafting Theory of Change on Adoption and Impact Assignment: On Wednesday, your group will present the design of a randomized evaluation to the rest of the participants. During this hour, your team must agree on which project you will collaborate, and draft the project s theory of change. In later sessions, you will think more specifically about the research design and randomization strategy. This hour is divided into 5 sections: 1. Project Introductions (20 minutes) Each member of the group should discuss a research project on which s/he is working or think would be interesting for the group to use. Justify why this project should be chosen, ideally offering an informal needs assessment to justify the program. After hearing options from each of the group members, choose one project. For this exercise, please think about issues of adoption and impact in particular. 2. Theory of Change (15 minutes) Construct a logical framework or theory of change for the team s chosen intervention. What are the inputs, outputs, intermediate outcomes, and impact? Where does adoption fit into this framework? 3. Indicators (15 minutes) What are some indicators used to measure stages or outcomes identified in the log-frame? 4. What might go wrong? (8 minutes) What can go wrong with the implementation of or concept behind the intervention? How can this be measured? Think about the process evaluation of this program. 5. Unintended Consequences (2 minutes) What unintended consequences might come of this intervention? How can these be measured? You can also think of this as being part of the impact evaluation of the program. Background: According to Rossi, Freeman, and Lipsy, 1 a comprehensive program evaluation is made up of five elements: (1) Needs assessment, which is conducted to identify the key policy issues where social indicators are lagging, as well as the potential sources of the problem. Ideally, an intervention is conceived after need is established. (2) Program theory assessment is an umbrella term used to describe the process of drawing up the blueprints of an intervention. More familiar than the term, program theory assessment, are the specific examples it is meant to encapsulate: logical framework, results framework, theory of change, etc. Upon implementation, a (3) process evaluation can be conducted to ensure that the services are being delivered and that the program is being run efficiently. Distinguishing 1 Evaluation: A systematic approach 11

14 it from monitoring, process evaluation is usually seen as an external activity meant to report on implementation, rather than used for day-to-day management. (4) Impact evaluation will test the causal impact of the program on the most important outcomes i.e. establish how the program changed the lives of those in its catchment area. Once all these different evaluations are completed, (5) a costeffectiveness of cost-benefit analysis can be conducted to test whether the costs associated to the program are sufficiently outweighed by its benefits. This will inform whether it is worth scaling it up to a wider population. 12

15 TRANSLATING RESEARCH INTO ACTION Case 3: Wome Case 1: How to Randomize Examining Barriers to Fertilizer Use in Kenya This case study is based on the paper "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya." by Esther Duflo (MIT), Michael Kremer (Harvard), and Jonathan Robinson (UCSC) J-PAL thanks the authors for allowing us to use their paper 13

16 Key Vocabulary 1. Level of Randomization: the level of observation (ex. individual, household, school, village) at which treatment and comparison groups are randomly assigned. Overview Adoption of fertilizer in much of sub-saharan Africa is quite low even though there are clear benefits to its use. 1 In 2009, a group of researchers with Innovations for Poverty Action in Kenya began a project to investigate three possible ways to increase fertilizer adoption among subsistence farmers in western Kenya. At the time of harvest, farmers were provided with coupons for discounted fertilizer to be purchased within the following few weeks. Researchers encouraged farmers to form cooperatives to share information about their farming. Lastly, measuring spoons were provided to farmers to allow them to apply the appropriate about of fertilizer on their plants. How would these interventions affect farmers uptake of the right amount of fertilizer? What experimental designs could test the impact of this intervention on fertilizer adoption? Which of these interventions is primarily responsible for improved fertilizer use? Problem Agricultural productivity in Sub-Saharan Africa has stagnated over the past decades: although total output has risen, food production has not kept up with the increase in Africa s population. The number of chronically undernourished people in Africa has increased to 200 million in When used correctly, chemical fertilizer can substantially raise agricultural yields, yet usage of fertilizer remains low. Many reasons exist for this underinvestment in fertilizer by smallholder farmer. Some past studies have suggested that usage is low because of: 1. Time inconsistent preferences : Farmers have difficulty saving harvest income to purchase fertilizer for the next growing season. At harvest time, farmers may have the cash on-hand that could be used to purchase fertilizer for the following growing season, but they have hard time holding on to that money until it is time to buy the fertilizer. Research has shown that if farmers are able to purchase fertilizer for the next season at the time of harvest they are much more likely to use fertilizer Lack of information: Farmers have limited information on the benefits of using fertilizer properly. Though there is some awareness about fertilizer, farmers are not aware of the appropriate application techniques and amounts of fertilizer necessary to maximize their profitability. 3. Lack of knowledge sharing: Farmers do not pass knowledge about fertilizer use to each other. Even if a farmer in a community is a user of fertilizer, he or she does not tend to share this information with fellow farmers. Randomized Evaluation 1 (The World Bank 2008) 2 (Harsch n.d.) 3 (Duflo, Kremer and Robinson n.d.) 14

17 Attempting to understand these barriers to fertilizer adoption, researchers in western Kenya devised a research project to investigate these questions. The project, Examining Barriers to Fertilizer Use in Kenya, focused on small-scale subsistence farmers in rural Western Kenya, many of whom grow maize as their staple crop. All farmers in this population are extremely poor, earning on the order of $1 per day. Previous research in this area has shown that when used correctly, top dressing fertilizer can increase yields by about 48%, amounting to a 36% rate of return on this investment over just a few months. However, only 40% of sampled farmers in the Busia district of Western Kenya report ever having used fertilizer. 4 In 2009, Innovations for Poverty Action in Kenya (IPAK) began the implementation and evaluation of this two year project. IPAK oversaw the implementation of three interventions described below for 20,000 subsistence farmers in rural western Kenya. Interventions The researchers used three interventions to examine the barriers above. The first intervention, designed to address farmers difficulties with time preferences, IPAK distributed small, time-limited discounts, which were valid within a three week window right after harvest, redeemable at a local shop. Farmers received coupons for a discount of about 15% of the price of fertilizer, for up to 25 kilograms. With this coupon farmers would have greater incentive to use their earnings from harvest to purchase fertilizer for the next season s crop. The second intervention included efforts to catalyze the establishment of farmers networks. Groups of farmers were encouraged to form farmers networks with their friends and neighbors to talk about fertilizer and agricultural practices. The researchers organized the groups and coordinated the first few meetings, but did not provide any direct information to the groups. The logic behind these efforts is that if farmers have an established network to communicate about farming practices, perhaps information about fertilizer can spread more swiftly. In the third intervention, IPAK supplied measuring spoons to farmers so that they could apply the appropriate amount of fertilizer to their plants. 5 The research team visited farmers and provided them with ½ teaspoon measuring spoons, as well as information about the returns to using ½ teaspoon of fertilizer per plant. To enable diffusion of this technology to others in the community, the spoons were made available in nearby fertilizer shops to other farmers for a nominal fee. In addition, when distributing the measuring spoons, the farmers were given vouchers for spoons which they could give to their friends. Addressing the research questions through experimental design Different randomization strategies may be used to answer different questions. What strategies could be used to evaluate the following questions? How would you design the study? Who would be in the treatment and control groups, and how would they be randomly assigned to these groups? 4 (Duflo, Kremer and Robinson n.d.) 5 (Duflo, Kremer and Robinson n.d.) 15

18 Discussion Topic 1: Testing the effectiveness of coupons 1. What is the relative effectiveness of coupons? Discussion Topic 2: Testing the effectiveness of social networks 1. What is the effect of forming cooperatives? 2. What is the effect of supplying measuring spoons? Discussion Topic 3: Addressing all questions with a single evaluation 1. Could a single evaluation explore all of these issues at once? 2. What randomization strategy could do so? 3. What do think about the time-specific aspect of the coupon? How would you design a project to disentangle the different attributes of the coupon? Bibliography Duflo, Esther, Michael Kremer, and Jonathan Robinson. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya." n.d. Harsch, Ernest. Agriculture: Africa s Engine for Growth. n.d. The World Bank. World Development Report 2008: Agriculture for Development. Washington, DC: The World Bank,

19 TRANSLATING RESEARCH INTO ACTION This case study is based on a current study by Esther Duflo and Tavneet Suri. J-PAL thanks the authors for allowing us to use their project 17

20 Key Vocabulary 1. Equivalence: groups are identical on all baseline characteristics, both observable and unobservable. Ensured by randomization. 2. Attrition: the process of individuals joining in or dropping out of either the treatment or comparison group over the course of the study. 3. Attrition Bias: statistical bias which occurs when individuals systematically join in or drop out of either the treatment or the comparison group for reasons related to the treatment. 4. Partial Compliance: individuals do not comply with their assignment (to treatment or comparison). Also termed "diffusion" or "contamination." 5. Intention to Treat: the measured impact of a program that includes all data from participants in the groups to which they were randomized, regardless of whether they actually received the treatment. Intention-to-treat analysis prevents bias caused by the loss of participants, which may disrupt the baseline equivalence established by randomization and which may reflect non-adherence to the protocol. 6. Treatment on the Treated: the measured impact of a program that includes only the data for participants who actually received the treatment. 7. Externality: an indirect cost or benefit incurred by individuals who did not directly receive the treatment. Also termed "spillover." In 2010, the Technoserve (TNS) Coffee Initiative partnered with J-PAL researchers to conduct a randomized evaluation on their coffee agronomy-training program in Nyarubaka sector in southern Rwanda. Technoserve carried out their regular recruitment sign-up processes across all 27 villages in the sector and registered 1600 coffee farmers who were interested in attending the monthly training modules. The study design for the evaluation then required that this pool of farmers be split into treatment and control groups, meaning those who would participate in the training, and those who wouldn t (for now they would be trained in later phases). The trainings in Nyarubaka included 800 coffee farmers, randomly selected from the pool of Randomization ensures that the treatment and comparison groups are equivalent at the beginning, mitigating concern for selection bias. But it cannot ensure that they remain comparable until the end of the program. Nor can it ensure that people comply with the treatment, or even the non-treatment, that they were assigned. Life also goes on after the randomization: other events besides the program happen between initial randomization and the end-line data collection. These events can reintroduce selection bias; they diminish the validity of the impact estimates and are threats to the integrity of the experiment. How can common threats to experimental integrity be managed? 18

21 Evaluation design The experiment as planned As previously mentioned, the agronomy training evaluation consisted of 1600 farmers, half of which attended monthly training sessions, and the other half did not. In addition, there was a census done of the entire sector to show us which households were coffee farmers and which ones were not. The census showed that there were 5400 households in Nyarubaka non-coffee farming households and 3000 coffee farming households (1600 of which were already in our sample). Each month a Technoserve farmer trainer would gather the farmers assigned to his/her group and conduct a training module on farming practices (e.g. weeding, pruning, bookkeeping, etc). The farmers were taught the best practices by using a practice plot so they could see and do exactly what the instructor was explaining. To think about: How can we be certain that the control group farmers did not attend the training too? What can be done to reduce this risk? Since we have a census for Nyarubaka, how might this be helpful in at least controlling for or documenting any spillovers? Think about what can be done at the trainings themselves. What type of data might you need/want to try to control for any spillovers in this case? What were other forms or opportunities for agronomy training in the area? Threats to integrity of the planned experiment Discussion Topic 1: Threats to experimental integrity Randomization ensures that the groups are equivalent, and therefore comparable, at the beginning of the program. The impact is then estimated as the difference between the average outcome of the treatment group and the average outcome of the comparison group, both at the end of the program. To be able to say that the program caused the impact, you need to be able to say that the program was the only difference between the treatment and comparison groups over the course of the evaluation. 1. What does it mean to say that the groups are equivalent at the start of the program? 2. Can you check if the groups are equivalent at the beginning of the program? How? 3. Other than the program s direct and indirect impacts, what can happen over the course of the evaluation (after conducting the random assignment) to make the groups non-equivalent? 4. How does non-equivalence at the end threaten the integrity of the experiment? 5. In the Technoserve agronomy training example, why is it useful to randomly select from the farmers who signed up for the Technoserve training program, rather than amongst all the coffee farmers in the sector? 19

22 Managing attrition when the groups do not remain equivalent Attrition is when people join or drop out of the sample both treatment and comparison groups over the course of the experiment. One common example in clinical trials is when people die; so common indeed that attrition is sometimes called experimental mortality. Discussion Topic 2: Managing Attrition You are looking at how much farmers adopt the recommendations and techniques from the agronomy trainings. Using a stylized example, let s divide adoption of the techniques as follows: Full adoption = score of 2 Partial adoption = score of 1 No adoption = score of 0 Let s assume that there are 1800 farmers: 900 treatment farmers who receive the training and 900 comparison farmers who do not receive the training. After you randomize and collect some baseline data, you determine that the treatment and comparison groups are equivalent, meaning farmers from each of the three categories are equally represented in both groups. Suppose protocol compliance is 100 percent: all farmers who are in the treatment go to the training and none of the farmers in the comparison attend the training. Let s Farmers who attend all agronomy trainings end up with full adoption, scoring a 2. Let s assume that there was a drought during this period, and those who adopted best-practices managed to protect their crops against damage. However, the farmers who have adoption level 0 see most of their crops perish, and members of the household enter the migrant labor market to generate additional income. The number of farmers in each treatment group, and each adoption category is shown for both the pre-adoption and post-adoption. Pre-adoption Post-adoption Adoption Level Treatment Comparison Treatment Comparison Dropped out Total farmers in the sample a. At program end, what is the average adoption for the treatment group? b. At program end, what is the average adoption for the comparison group? c. What is the difference? d. Is this outcome difference an accurate estimate of the impact of the program? Why or why not? e. If it is not accurate, does it overestimate or underestimate the impact? f. How can we get a better estimate of the program s impact? 2. Besides level of adoption, the Technoserve agronomy training evaluation also looked at outcome measures such as yields and farm labor. a. Would differential attrition (i.e. differences in drop-outs between treatment and comparison groups) bias either of these outcomes? How? b. Would the impacts on these final outcome measures be underestimated or overestimated? 20

23 3. In the Technoserve agronomy evaluation, identify some other causes for attrition in the Treatment group and the Control groups? What can be done to mitigate these? 4. You may know of other research designs to measure impact, such as nonexperimental or quasi-experimental methodologies (eg. pre-post, differencein-difference, regression discontinuity, instrumental variables (IV), etc) a. Is the threat of attrition unique to randomized evaluations? Managing partial compliance when the treatment does not actually get treated or the comparison gets treated Some people assigned to the treatment may in the end not actually get treated. In an after-school tutoring program, for example, some children assigned to receive tutoring may simply not show up for tutoring. And the others assigned to the comparison may obtain access to the treatment, either from the program or from another provider. Or comparison group children may get extra help from the teachers or acquire program materials and methods from their classmates. In any of these scenarios, people are not complying with their assignment in the planned experiment. This is called partial compliance or diffusion or, less benignly, contamination. In contrast to carefully-controlled lab experiments, diffusion is ubiquitous in social programs. After all, life goes on, people will be people, and you have no control over what they decide to do over the course of the experiment. All you can do is plan your experiment and offer them treatments. How, then, can you deal with the complications that arise from partial compliance? Discussion Topic 3: Managing partial compliance Suppose that farmers who have adoption level 0 are too risk averse to adopt the techniques they learn at the training. Famers believe that there is no way for them to adopt the techniques that are described in early trainings and stop attending. Consequently, none of the treatment farmers with adoption level 0 increased their adoption and remained at level 0 at the end of the program. No one assigned to comparison had attended the trainings. All the farmers in the sample at the beginning of the program were followed up. PreadoptionPreadoptio Postadoption n Adoption Level Treatment Comparison Treatment Comparison Total farmers in the sample Calculate the impact estimate based on the original group assignments. a. Is this an unbiased measure of the effect of the program? b. In what ways is it useful and in what ways is it not as useful? You are interested in learning the effect of treatment on those actually treated ( treatment on the treated (TOT) estimate). 2. Five of your colleagues are passing by your desk; they all agree that you should calculate the effect of the treatment using only the 10,000 farmers who attended the training. a. Is this advice sound? Why or why not? 21

24 3. Another colleague says that it s not a good idea to drop the farmers who stopped attending the trainings entirely; you should use them but consider them as part of the control group. a. Is this advice sound? Why or why not? 4. Another colleague suggests that you use the compliance rates, the proportion of people in each group that did or did not comply with their treatment assignment. You should divide the intention to treat estimate by the difference in the treatment ratios (i.e. proportions of each experimental group that received the treatment). a. Is this advice sound? Why or why not? Managing spillovers when the comparison, itself untreated, benefits from the treatment being treated People assigned to the control group may benefit indirectly from those receiving treatment. For example, a program that distributes insecticide-treated nets may reduce malaria transmission in the community, indirectly benefiting those who themselves do not sleep under a net. Such effects are called externalities or spillovers. Discussion Topic 4: Managing spillovers In the Technoserve agronomy training evaluation, randomization was at the farmer level, meaning that while one farmer might have been selected to be in the training, his neighbor didn t have the same fortunes during the randomization process. Depending on the evaluation and the nature of the program, it might be more challenging to prevent spillovers of agronomic knowledge between friends, than it is for delivering hard tangible objects in farmers hands, like a weighing scale or calendar to maintain harvest records. 1. How do you imagine spillovers might occur in agronomy training? 2. What types of mechanisms can you think of that could be used to reduce or manage spillovers? Measuring Spillovers Discussion Topic 5: Measuring spillovers 1. Can you think of ways to design the experiment explicitly to measure the spillovers of the agronomy training? 22

25 Exercise 1: The mechanics of random assignment using MS Excel Part 1: simple randomization Like most spreadsheet programs MS Excel has a random number generator function. Say we had a list of schools and wanted to assign half to treatment and half to control (1) We have all our list of schools. 23

26 (2) Assign a random number to each school: The function RAND () is Excel s random number generator. To use it, in Column C, type in the following = RAND() in each cell adjacent to every name. Or you can type this function in the top row (row 2) and simply copy and paste to the entire column, or click and drag. Typing = RAND() puts a 15-digit random number between 0 and 1 in the cell. 24

27 (3) Copy the cells in Colum C, then paste the values over the same cells The function, =RAND() will re-randomize each time you make any changes to any other part of the spreadsheet. Excel does this because it recalculates all values with any change to any cell. (You can also induce recalculation, and hence re-randomization, by pressing the key F9.) This can be confusing, however. Once we ve generated our column of random numbers, we do not need to re-randomize. We already have a clean column of random values. To stop excel from recalculating, you can replace the functions in this column with the values. To do this, highlight all values in Column C. Then right-click anywhere in the highlighted column, and choose Copy. Then right click anywhere in that column and chose Paste Special. The Paste Special window will appear. Click on Values. 25

28 (4) Sort the columns in either descending or ascending order of column C: Highlight columns A, B, and C. In the data tab, and press the Sort button: A Sort box will pop up. In the Sort by column, select random #. Click OK. Doing this sorts the list by the random number in ascending or descending order, whichever you chose. 26

29 There! You have a randomly sorted list. (5) Sort the columns in either descending or ascending order of column C: Because your list is randomly sorted, it is completely random whether schools are in the top half of the list, or the bottom half. Therefore, if you assign the top half to the treatment group and the bottom half to the control group, your schools have been randomly assigned. In column D, type T for the first half of the rows (rows 2-61). For the second half of the rows (rows ), type C 27

30 Re-sort your list back in order of school id. You ll see that your schools have been randomly assigned to treatment and control groups 28

31 Part 2: stratified randomization Stratification is the process of dividing a sample into groups, and then randomly assigning individuals within each group to the treatment and control. The reasons for doing this are rather technical. One reason for stratifying is that it ensures subgroups are balanced, making it easier to perform certain subgroup analyses. For example, if you want to test the effectiveness on a new education program separately for schools where children are taught in Hindi versus schools where children are taught in Gujarati, you can stratify by language of instruction and ensure that there are an equal number schools of each language type in the treatment and control groups. (1) We have all our list of schools and potential strata. Mechanically, the only difference in random sorting is that instead of simply sorting by the random number, you would first sort by language, and then the random number. Obviously, the first step is to ensure you have the variables by which you hope to stratify. (2) Sort by strata and then by random number Assuming you have all the variables you need: in the data tab, click Sort. The Sort window will pop up. Sort by Language. Press the button, Add Level. Then select, Random #. 29

32 (3) Assign Treatment Control Status for each group. Within each group of languages, type T for the first half of the rows, and C for the second half. 30

33 Exercise 2: Sample Size and Power Key Vocabulary: 1. Power: the likelihood that, when the program has an effect, one will be able to distinguish the effect from zero given the sample size. 2. Significance: the likelihood that the measured effect did not occur by chance. Statistical tests are performed to determine whether one group (e.g. the experimental group) is different from another group (e.g. comparison group) on the measurable outcome variables used in the evaluation. 3. Standard Deviation: a standardized measure of the variation of a sample population from its mean on a given characteristic/outcome. Mathematically, the square root of the variance. 4. Standardized Effect Size: a standardized measure of the [expected] magnitude of the effect of a program. 5. Cluster: the level of observation at which a sample size is measured. Generally, observations which are highly correlated with each other should be clustered and the sample size should be measured at this clustered level. 6. Intra-cluster Correlation Coefficient: a measure of the correlation between observations within a cluster; i.e. the level of correlation in drinking water source for individuals in a household. Sample size calculations In this exercise, we ll use a new example to explore the same issues of sample size and power: Promoting Agricultural Technology Adoption in Rwanda. In this example, we re interested in measuring the effect of a treatment (agronomy training) on outcomes measured at the household level: specifically, productivity. However, the randomization of trainings was done at the village level. It could be that our outcome of interest is correlated for farmers in the same village, for reasons that have nothing to do with the training itself. For example, all the farmers in a village may be affected by their original shared knowledge, by whether their land is especially fertile or not, or whether their weather patterns are helpful; these factors mean that when one farmer in the village does particularly well for this reason, all the farmers in that village probably also do better which might have nothing to do with the training. Therefore, if we sample 100 households from 10 randomly selected villages, that sample is less representative of the population of villages in a district than if we selected 100 random households from the whole population of villages, and therefore absorbs less variance. In effect, we have a smaller sample size than we think. This will lead to more noise in our sample, and hence larger standard error than in the usual case of independent sampling. However, sampling some households in fewer villages may be cheaper since travel costs between villages are likely greater than travel costs within villages. When planning both the sample size and the best way to sample classrooms, we need to take both statistical and budgetary issues into account into account. This exercise will help you understand how to do that. Should you sample every household in just a few villages? Should you sample a few households from many villages? How do you decide? We will work through these questions by determining the sample size that allows us to detect a specific effect with at least 80% power. Remember power is the likelihood that when the treatment has an effect you will be able to distinguish it from zero in your sample. 31

34 In this example, clusters refer to clusters of households in other words, a village. This exercise shows you how the power of your sample changes with the number of clusters, the size of the clusters, the size of the treatment effect and the Intraclass Correlation Coefficient. We will use a software program developed by Steve Raudebush (with funding from the William T. Grant Foundation). You can find additional resources on clustered designs on their web site. Section 1: Using the OD Software (Windows PCs only) First, download the OD software from the website (a software manual is also available): When you open it, you will see a screen which looks like the one below. Select the menu option Design to see the primary menu. Select the option Cluster Randomized Trials with personlevel outcomes, Cluster Randomized Trials, and then Treatment at level 2. You ll see several options to generate graphs; choose Power vs. Total number of clusters (J). A new window will appear: Select α (alpha). You ll see it is already set to for a 95% significance level. First let s assume we want to survey only 50 households per village. How many villages do you need to go to in order to have a statistically significant answer? Click on n, which represents the number of households per village. Since we are surveying only 50 households per village, fill in n(1) with 50 and click OK. Now we have to determine δ (delta), the standard effect size (the effect size divided by the standard deviation of the variable of interest). Assume we are interested in detecting whether there is an increase of 10% in coffee cherry production. (Or more accurately, are uninterested in an effect less than 10%.) Our baseline survey indicated that the average production is 104 KG of coffee cherries, with a standard deviation of 109 KG. We want to detect an effect size of 10% of 104, which is We divide 10.4 by the standard deviation to get δ equal to 10.49/109, or

35 Select δ from the menu. In the dialogue box that appears there is a prefilled value of for delta(1). Change the value to 0.096, and change the value of delta(2) to empty. Select OK. Finally we need to choose ρ (rho), which is the intra-cluster correlation. ρ tells us how strongly the outcomes are correlated for units within the same cluster. If households from the same village were clones (no variation) and all produced the exact same amount of coffee cherries, then ρ would equal 1. If, on the other hand, households from the same villages are in fact independent and there was no differences between villages, then ρ will equal 0. You have determined in your pilot study that ρ is Fill in rho(1) to 0.034, and set rho(2) to be empty. You should see a graph similar to the one below. You ll notice that your x axis isn t long enough to allow you to see what number of clusters would give you 80% power. Click on the button to set your x axis maximum to 500. Then, you can click on the graph with your mouse to see the exact power and number of clusters for a particular point. 33

36 Exercise 3.1: How many villages are needed to achieve 80% power? 90% power? Now you have seen how many clusters you need for 80% power, sampling 50 households per village. Suppose instead that you only have the ability to go to 150 villages, due to budget constraints. Exercise 3.2: Given a constraint of 150 villages, how many households per village are needed to achieve 80% power? 90% power? Choose different values for n to see how your graph changes. Finally, let s see how the Intraclass Correlation Coefficient (ρ) changes power of a given sample. Leave rho(1) to be 0.03 but for comparison change rho(2) to You should see a graph like the one below. The solid blue curve is the one with the parameters you ve set - based on your pretesting estimates of the effect of agronomy training on farmer productivity. The blue dashed curve is there for comparison to see how much power you would get from your sample if ρ were zero. Look carefully at the graph. 34

37 Exercise 3.3: How does the power of the sample change with the Intraclass Correlation Coefficient (ρ)? To take a look at some of the other menu options, close the graph by clicking on the in the top right hand corner of the inner window. Select the Cluster Randomized Trial menu again. 35

38 Exercise 3.4: Try generating graphs for how power changes with cluster size (n), intra-class correlation (ρ) and effect size (δ). You will have to re-enter your pre-test parameters each time you open a new graph. Section 2: Using Stata For this section, we ll be using Stata. Stata is a powerful data analysis software. Often, complicated tasks things which would take several steps in another software, such as Optimal Design or Excel take only a single line of code in Stata. The problem, then, is knowing which Stata code to use. For computing sample size and power, the relevant Stata code is: sampsi and sampclus. The first command, sampsi, is default in Stata. The second one, sampclus, is an additional.ado file you ll need to download that before beginning this exercise. To get familiar with sampsi and sampclus, try typing in help sampsi and help sampclus. (To download sampclus, you can either type findit sampclus into Stata, and then download the file which appears. Also, we have included the.ado file itself in your jump drives. Check where your Stata grabs its.ado files from with the adopath command and either move sampclus.ado to one of those folders, or tell Stata to look in your jump drive as well with adopath++ location of your jump drive.) The remaining instructions for this exercise can be found in the pre-prepared.do file, CIMMYT_Power.do, which will be in your jump drives (Folder > Folder > XXX). Section 3: Using Excel 36

39 The same data which was used in Stata is available for our Excel exercise. We ll be conducting the same power calculations for the same sample so, ideally, we should be getting the same results. The benefits of the Excel exercise are that it builds the power analysis from the ground up: you manually calculate each part of the Section 4: Working within a budget constraint [DRAFT TEXT] A typical constraint to sample size is the budget - and this may sound discouraging, given that other aspects of power (the intra-class correlation coefficient, effect size and original summary statistics) are also beyond the researcher s control. Indeed, it may sometimes appear that power is a direct consequence of funds available. That said, it s not a one-to-one relationship. A number of factors impact the funds needed. One example: some villages may be further away and hard to reach. And perhaps the villages which are easier to reach are also similar in other characteristics. So we would pick up some variance by spending a little extra to visit the more remote villages as well. What about variables? Some measures are cheaper to collect such as a household survey and some are more expensive such as checking water quality. Researchers must identify (and prioritize) the data collection instruments which will minimize cost. The remaining instructions for this exercise can be found in the Excel file, CIMMYT_MX_budget.xls, which will be in your jump drives (Folder > Folder > XXX). General tips However, researchers can positively impact their power by The looser the level of significance we impose, the more likely we are to reject the null, i.e. the higher the power: but also the more likely we are to make false positive (type II) errors. The higher the MDE, the higher the power. The lower the variance of the underlying population, the lower the variance of the estimated effect size and the higher the power. The larger the sample size, the lower the variance of our estimate effect and the higher the power. The more evenly the sample is distributed between treatment and comparison, the higher the power. Individual-level randomization is more powerful than group-level randomization given the same sample size. The more outcomes are correlated within groups in a group-level randomization, the less power. 37

40 38

41 Group Presentation Template

42

43 Checklist For Reviewing a Randomized Controlled Trial of a Social Program or Project, To Assess Whether It Produced Valid Evidence

44 Updated February 2010 This publication was produced by the Coalition for Evidence-Based Policy, with funding support from the William T. Grant Foundation, Edna McConnell Clark Foundation, and Jerry Lee Foundation. This publication is in the public domain. Authorization to reproduce it in whole or in part for educational purposes is granted. We welcome comments and suggestions on this document 2

45 Checklist For Reviewing a Randomized Controlled Trial of a Social Program or Project, To Assess Whether It Produced Valid Evidence This is a checklist of key items to look for in reading the results of a randomized controlled trial of a social program, project, or strategy ( intervention ), to assess whether it produced valid evidence on the intervention s effectiveness. This checklist closely tracks guidance from both the U.S. Office of Management and Budget (OMB) and the U.S. Education Department s Institute of Education Sciences (IES) 1 ; however, the views expressed herein do not necessarily reflect the views of OMB or IES. This checklist limits itself to key items, and does not try to address all contingencies that may affect the validity of a study s results. It is meant to aid not substitute for good judgment, which may be needed for example to gauge whether a deviation from one or more checklist items is serious enough to undermine the study s findings. A brief appendix addresses how many well-conducted randomized controlled trials are needed to produce strong evidence that an intervention is effective. Checklist for overall study design Random assignment was conducted at the appropriate level either groups (e.g., classrooms, housing projects), or individuals (e.g., students, housing tenants), or both. Random assignment of individuals is usually the most efficient and least expensive approach. However, it may be necessary to randomly assign groups instead of, or in addition to, individuals in order to evaluate (i) interventions that may have sizeable spillover effects on nonparticipants, and (ii) interventions that are delivered to whole groups such as classrooms, housing projects, or communities. (See reference 2 for additional detail. 2 ) The study had an adequate sample size one large enough to detect meaningful effects of the intervention. Whether the sample is sufficiently large depends on specific features of the intervention, the sample population, and the study design, as discussed elsewhere. 3 Here are two items that can help you judge whether the study you re reading had an adequate sample size: If the study found that the intervention produced statistically-significant effects (as discussed later in this checklist), then you can probably assume that the sample was large enough. If the study found that the intervention did not produce statistically-significant effects, the study report should include an analysis showing that the sample was large enough to detect meaningful effects of the intervention. (Such an analysis is known as a power analysis. 4 ) Reference 5 contains illustrative examples of sample sizes from well-conducted randomized controlled trials conducted in various areas of social policy. 5 3

46 Checklist to ensure that the intervention and control groups remained equivalent during the study The study report shows that the intervention and control groups were highly similar in key characteristics prior to the intervention (e.g., demographics, behavior). If the study asked sample members to consent to study participation, they provided such consent before learning whether they were assigned to the intervention versus control group. If they provided consent afterward, their knowledge of which group they are in could have affected their decision on whether to consent, thus undermining the equivalence of the two groups. Few or no control group members participated in the intervention, or otherwise benefited from it (i.e., there was minimal cross-over or contamination of controls). The study collected outcome data in the same way, and at the same time, from intervention and control group members. The study obtained outcome data for a high proportion of the sample members originally randomized (i.e., the study had low sample attrition ). As a general guideline, the studies should obtain outcome data for at least 80 percent of the sample members originally randomized, including members assigned to the intervention group who did not participate in or complete the intervention. Furthermore, the follow-up rate should be approximately the same for the intervention and the control groups. The study report should include an analysis showing that sample attrition (if any) did not undermine the equivalence of the intervention and control groups. The study, in estimating the effects of the intervention, kept sample members in the original group to which they were randomly assigned. This even applies to: Intervention group members who failed to participate in or complete the intervention (retaining them in the intervention group is consistent with an intention-to-treat approach); and Control group members who may have participated in or benefited from the intervention (i.e., cross-overs, or contaminated members of the control group). 6 Checklist for the study s outcome measures The study used valid outcome measures i.e., outcome measures that are highly correlated with the true outcomes that the intervention seeks to affect. For example: Tests that the study used to measure outcomes (e.g., tests of academic achievement or psychological well-being) are ones whose ability to measure true outcomes is well-established. 4