Sampling and Power Calculations East Asia Regional Impact Evaluation Workshop Seoul, South Korea

Transcription

1 Sampling and Power Calculations East Asia Regional Impact Evaluation Workshop Seoul, South Korea Jacobus Cilliers, World Bank

2 Overview What is sampling and why is it necessary? Why does sample size matter? How large is large enough?

3 WHAT IS SAMPLING AND WHY IS IT NECESSARY?

4 Drawing a smaller sample. Use this to infer characteristics of whole population 1. Target Population 2. Evaluation sample Same characteristics

5 from your target population Whole Population Target Population INELIGIBLE

6 Example: Generasi Fund in Indonesia Target Population Random Sample Random Assignment Poorest subdistricts in 5 provinces 300 subdistricts 150 subdistricts 150 subdistricts

7 External and Internal Validity Random Assignment Target Population Random Sample External Validity Internal Validity

8 Drawing a random sample from two groups does not make them comparable. Random Sample is NOT the Same as Random Assignment! Participants Non Participants

9 WHY DOES SAMPLE SIZE MATTER?

10 Not all (stick)children are created equal Different heights of children aged 5

11 Test: are children taller because of the program? Different heights of children aged 5

12 Small Sample Chance? 1.4m 1.3m 1.3m. Not enough confidence that the difference we observe will also be observed in the population as a whole

13 Chance? 1.4m 1.3m 1.3m

14 Statistical Significance Confidence that result we observe is not due to chance. Normally want to be 95% confident With larger sample, more confidence

15 Statistical Significance Not statistically significant 95% confident Control Treatment

16 More confident with larger sample Statistically significant 95% confident Treatment Control

17 Power Probability of detecting a statistically significant result when it is there. Normally aim for 80%. I.e. 20% risk of not detecting and effect even though it is there. Need a larger sample for more power!

18 POWER CALCULATIONS HOW LARGE IS LARGE ENOUGH?

19 Depends on the Population, Program, and level of Randomization 1. How Similar or Different is the population? 2. How Large an impact do you expect? 3. Which Unit of Randomization: cluster or individual?

20 1. How similar or different is the population? ALL THE SAME ALL VERY DIFFERENT

21 Test: Did height Increase? SIMILAR DIFFERENT

22 With similar population, less likely due to chance if observe difference 1.4m 1.3m Without the program, everyone is 1.3m. 1.3m So if we observe people are 1.4m, we know it is because of the program

23 2. How large an impact do you expect? If you expect the impact of the program to be large, then you will be able to detect the impact, even with a smaller sample. Because a large difference is unlikely due to chance. Conversely, if you have a large sample, then you will be able to detect even a small change. DANGER: DON T MAKE UNREALISTIC ASSUMPTIONS ABOUT EFFECT SIZE!!

24 Who is Taller?

25 Who is Taller?

26 3. Level of Randomization You will need a larger sample if you randomize at a cluster (village/school/clinic) level You want to increase the number of clusters, not the number of individuals per cluster

27 Similar Pupil Performance in Same School SCHOOL A SCHOOL B

28 Chance? A A B B

29 Increasing number of pupils per school doesn t help a lot. Pupils within the same school will still be similar to each other, because the school-level factors remain the same. Increase power by more if increase number of schools

30 Depends on intra-cluster correlation (ρ) Degree of how similar people are within a cluster. If ρ=1 All individuals within the same cluster are exactly the same. Increasing number of individuals no impact on power If ρ=0 As if performing individual level randomization.

31 Example More Clusters The following two studies have the same power: 80 clusters, 20 individuals per cluster 40 clusters, 1067 individuals per cluster That s 1,800 individuals compared to 42,680! Assume intra-cluster correlation of 0.05

32 Example cluster vs individual The following two studies have the same power: Individual level: 393 in treatment and 393 in control Cluster level: 80 clusters, 20 individuals per cluster (1,600 individuals in treatment and 1,600 in control) Assume intra-cluster correlation of 0.05

33 HOW TO INCREASE POWER?

34 How to Increase with same sample size? Stratify Control for all factors (e.g. gender, age, occupation) Baseline data DANGER: DON T MAKE UNREALISTIC ASSUMPTIONS ABOUT EFFECT SIZE!!

35 Stratify by District

36 Stratify by District

37 Stratify by District

38 Stratify by District

39 Appendix Summary of technical terms α Significance Level The probability of falsely detecting an impact when there is none. Conventionally set at 5%. The smaller it is, the larger sample size required to get the same power. Power The probability of detecting an effect when it truly exists. Normally aim for 80%. ρ Intra-cluster correlation Degree of how similar people are within a cluster. If this is high, then need more clusters to get the same power. MDE Minimal detectable effect size. The size you want to detect. If you expect a very large impact, then you need a smaller sample to reach the same power. Take-up

40 POWER CALCULATIONS TEST

41 Program A: you expect a large impact of 20% Program B: you expect a small impact of 5% For both to have the same power, which study will need a larger sample? 50% 50% A. Program A B. Program B Program A Program B

42 Government wants to evaluate a Conditional Cash Transfer Program. Which evaluation method has larger power? A. Randomly assign some eligible individuals to receive the transfer and some not. B. Randomly assign some communities to receive the program and some communities not. 50% 50% Randomly assign some eligib.. Randomly assign some co...

43 You are evaluating a program that provides HIV/AIDS awareness training in schools. The evaluation is currently in 40 schools (20 treatment and 20 control), with 20 pupils being surveyed in each school. A member of your academic team is afraid that the sample is not sufficiently powered. 50% 50% What will increase power by more? A. Increase number of clusters from 40 to 50. B. Increase number of pupils per school from 20 to 30. Increase number of cluster... Increase number of pupils...

44 APPENDIX

45 Rules of Thumb Individual: 100 treatment and control individuals is rarely enough. 500 treatment and 500 control is often enough Cluster 10 treatment and control clusters is rarely enough 50 treatment and 50 control, with 15 individuals per cluster, is often enough DISCLAIMER: depends on: standard deviation, intra-cluster correlation, minimum expected effect size, correlation between baseline and endline measures, etc