Sample size
Outline 1.What is sampling? 2.How large of a sample size do we need? 3.How can we increase statistical power?
Sampling
Why do we sample? Usually we cannot gather data from entire target population. Random sampling allows you to infer characteristics of the target group from the smaller sample.
Random sampling Target group Sample The sample has the same average characteristics as the target group.
Random assignment
Do we have to do any sampling for an impact evaluation? Does the evaluation sample have to be sampled from a larger group?
Do we have to sample for an impact evaluation? No Reall e saple he it s ot feasile to collect data on a large group. If evaluation sample is already small If obtaining and analyzing data for all is not expensive No need to sample
Not always necessary to sample, but.. We need to worry about the size of this group
Why should we care about the size of the evaluation sample? Let s thik aout soethig sipler tha program impact. What is the average height of women in Nepal? Surey : Let s ask 5 people Surey : Let s ask, people
Which survey will give us an answer closer to the true average? Height Asking 5 people Number of responses 140-144cm 1 145-149cm 2 150-154cm 1 155-159cm 0 160-164cm 1 Height Asking 1000 people Number of responses 140-144cm 70 145-149cm 150 150-154cm 650 155-159cm 125 160-164cm 5
Larger sample sizes will yield a program impact closer to the true impact.
Larger sample sizes help minimize statistical errors Type 1 error You say there is a progra ipat he there really is t one. Can minimize this with choice of confidence interval (95%-99%) Type 2 error There really is a program impact but you cannot detect it. Very dependent on sample size
Larger sample sizes help to maximize statistical power POWER The likelihood you detect an effect when there is one. Typically want this to be more than 80%. Thus, want to keep Type 2 error=20%
What happens if statistical power is too low? We cannot distinguish a large effect from an effect of zero Your boss: What was the impact of that project that you led? You: Ummm it ight have ireased ioe y 200%. It ight ot have doe aythig. I a t really tell fro the data. Your boss: How much did you spend on this evaluation? You: [You should run away at this point].
Sample size
How large is large? 1. Probability of Type 1 and Type 2 errors. 2. How similar or different is the population? 3. How large of an impact is expected? 4. What is the unit of randomization? 1) ( 1 ) ( 4 2 2 2 / 2 H D z z N
2. How similar or different is the population? Less underlying variance easier to detect difference can have lower sample size
3. How large of an impact is expected? If impact is expected to be large, can detect it with a smaller sample. Large difference unlikely to be due to chance Need a large sample to detect small differences
How many times do you need to look to see who is taller?
Choosing an expected effect size Choose the lowest value you would consider a success Consider a flagship education program to increase access to schools Would a 5 percent enrolment increase be considered a success?
4. What is the unit of randomization? Larger samples needed if unit of randomization is a cluster (village, school, clinic) Goal is to maximize number of clusters, not number of individuals per cluster
Clustering creates similarity
Power increases when number of schools increases Pupils in same school will be similar because school-level factors are similar Sampling more pupils does not provide more information Akin to asking the same person about average height of women in Nepal multiple times
Measuring the degree of similarity from clustering 4 2 ( z z ) 2 / 2 Intra-cluster correlation N 1 2 D If ρ= All individuals within the same cluster are exactly the same. Increasing number of individuals no impact on power If ρ= As if performing individual level randomization. ( H 1)
More clusters the better The following two studies have the same power: 80 clusters, 20 individuals per cluster 40 clusters, 1067 individuals per cluster [intra-cluster correlation=0.05] That s 1,600 individuals compared to 42,680!
Randomizing individuals vs. clusters The following two studies have the same power: Individual level: 393 in treatment and 393 in control Cluster level: 80 clusters each for treatment and control, 20 individuals per cluster [1,600 individuals in treatment and 1,600 in control]
How can we increase statistical power?
What do we have control over? N 1. Sample size 2. Expected effect size 3. Number of units per cluster 4. Variance Stratification Controlling for many factors Baseline data 4 2 ( z / 2 D 2 z ) 2 1 ( H 1)
THANK YOU