Evaluation Team: Field Coordinators: Advisors: Evaluation Website:

Transcription

1

2 Evaluation Team: Field Coordinators: Advisors: Evaluation Website:

3 Goals Provide a rigorous and detailed assessment of the impacts of NSP on economic outcomes and local governance; Provide evidence to inform decisions made by Government of Afghanistan, donors, and NGOs concerning the future of NSP; Create knowledge which can inform the design of other local governance and development programs, both in Afghanistan and in other post-conflict settings

4 Relevance NSP-RIE represents rare opportunity to rigorously observe impact of an attempt to effect local institutional change and of an attempt to use CDD mechanisms to improve development outcomes; NSP-RIE will answer questions such as: - How does NSP affect local institutional structures? - Does NSP build trust among villagers? - Does NSP increase female participation in governance? - Does NSP increase access to services? - Does NSP alter levels and diversity of village consumption and production?

5 Core Hypotheses Hypotheses reflect dual function of NSP to impact local governance structures and village economies: 1. Economic Questions: - Access of villagers to essential services and infrastructure; - Levels, diversity, and composition of consumption and assets; - Levels, diversity, and composition of production; - Female participation in economic activities. 2. Institutional Outcomes: - Structures and function of local governance; - Participation of women and non-elite males in governance; - Interpersonal trust and community relations; - Attitudes towards central government. NSP-RIE will also examine interactions between NSP and pre-existing conditions and test ability of NSP to meet exante project preferences.

6 Sub-Treatment Interventions Designed to test potential refinements in NSP to improve impact, program satisfaction etc.: Two sub-treatment interventions (STIs): 1. Election Type: Cluster vs. At-Large Elections 2. Sub-Project Selection Procedure: Meeting vs. Referendum Impact of STIs assessed using core NSP-RIE outcomes of interest and program-specific outcomes, such as: - Characteristics of elected CDC Members; - Coincidence of NSP sub-project with sub-project(s) preferred ex-ante by different village groups; - Community involvement in sub-project implementation and monitoring; - Quality of completed sub-projects; - Levels of community satisfaction with CDC and sub-project. Data for STI study to be provided by core NSP-RIE surveys, as well as separate CDC election, sub-project selection, sub-project implementation, and sub-project completion monitoring and data supplied by FPs.

7

8 Design Structured to meet state-of-the-art, without disrupting implementation of NSP or exposing implementers to additional risks: Design: Evaluation methodology designed in cooperation with and vetted by professors from Harvard University and MIT, including senior members of MIT Poverty Action Lab; Sample: 500 evaluation villages in 10 districts in 6 provinces; Treatment Group: 250 NSP evaluation villages randomly selected from 500 evaluation villages to participate in NSP during ; Control Group: 250 non-nsp evaluation villages will receive NSP after final follow-up survey in 2009; Data: Series of 3 male & female, household & focus group surveys (1 baseline survey and 2 follow-ups) to be administered in 500 evaluation villages over 2-3 years; Estimation Strategy: 1-year and 2-year impact of NSP estimated by comparing differences (or differences-in-differences) in outcomes of interest between treatment and control groups.

9 10 Evaluation Districts Balkh Khost Wa Firing Gulran Adraskan Chist-e Sharif Farsi Daulina Sang Takht Hisarak Sherzad

10 Treatment Assignment 25 matched-pairs of villages formed in each of 10 sample districts using multivariate matching: Attrition Proof: Protects inferences from loss of sample units; Interaction Effects: Permits inferences over interaction of program effects with underlying conditions. Treated assigned to one unit in matched-pair using random number generator: No Selection Bias: Negligible differences between units selected to receive program and those selected not to receive program; Facilitates Transparent Estimation: Program effects are the difference between outcomes in 250 NSP evaluation villages and the 250 non-nsp evaluation villages. The data will speak for itself No Role for Publication Bias : Very few ways through which researchers can manufacture results that don t actually exist; Threat of Replication: Findings of study can (and will) be checked by external researchers using data and code published on evaluation website.

11 Progress & Timeline Baseline Survey, CDC Elections & Sub-Project Selection completed in 10 districts; Preparations for follow-up underway: Evaluation team has partnered with Vulnerability Analysis Unit (VAU) to collect data for baseline and follow-up surveys; Baseline survey of 13,000 people in 500 villages completed in Sept. 2007; CDC election and sub-project selection monitoring completed in July 2008 (1,500 people in 125 villages interviewed for each); First follow-up survey tentatively planned for Nov Apr. 2009; Second follow-up survey tentatively planned for Sept. Nov. 2009; Report on sub-treatment interventions expected by December; Survey instruments for 1 st Follow-Up Survey currently being prepared.

12

13 Access to Electricity 34% 36% 16% 16% 12% 13% 8% 2% 2% 2% Adraskan (Herat) Balkh (Balkh) Chisht-e Sharif (Herat) Daulina (Ghor) Farsi (Herat) Gulran (Herat) Hisarak (Nangarhar) Khost Wa Firing (Baghlan) Sang Takht (Daykundi) Sherzad (Nangarhar)

14 Access to Safe Drinking Water 49% 32% 25% 22% 15% 15% 16% 13% 7% 2% Adraskan (Herat) Balkh (Balkh) Chisht-e Sharif (Herat) Daulina (Ghor) Farsi (Herat) Gulran (Herat) Hisarak (Nangarhar) Khost Wa Firing (Baghlan) Sang Takht (Daykundi) Sherzad (Nangarhar)

15 Preferred Project 45% Male Household Female Individual Male Shura 40% 35% 30% 25% 20% 15% 10% 5% 0% Drinking Water Schools Health Facilities Roads or Bridges Irrigation Electricity Agricultural Inputs Courses Microfinance Programs Community Building

16 Incidence of Borrowing 87% 63% 60% 66% 53% 48% 31% 28% 20% 21% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

17 Reason for Borrowing 4% 9% 4% 5% 4% 45% Buy Food Medical Reason Loan Repayment House Construction Wedding Purchase Ag, Inputs Other 29%

18 Female Role in Governance 100% 90% 80% 70% 60% 50% 40% Separate Committee Main Shura No Membership 30% 20% 10% 0% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

19 Support for Female Shura Membership 100% Male Household Women Male Shura 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

20 NSP Awareness 90% Male Household Women Male Shura 80% 70% 60% 50% 40% 30% 20% 10% 0% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

21 Consent to Taxation 100% Male Household Male Shura 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

22 Appropriate Recipient of Tax Revenue 100% 90% 80% 70% 60% 50% 40% Individuals Village Shura or Leaders Subnational Authorities Central Government 30% 20% 10% 0% Adraskan Balkh Chisht-e Sharif Daulina Farsi Gulran Hisarak Khost Wa Firing Sang Takht Sherzad

23 Trust in Political Figures 100% 90% 80% 70% 60% 50% 40% 30% Themselves Some All 20% 10% 0%

24

25 What is the purpose of the CDC? Help villagers 73% Undertake development projects 52% Resolve disputes between villagers 21% Serve as a village council 14% Other 10%

26 How did you decide who to vote for? 11% 10% 22% Education Religious Virtue Candidate's Honesty Commitment to Community Other 28% 29%

27 Who organized the CDC election? MRRD 46% NSP 46% FP / NGO 37% Government of Afg. 28% Foreigners 3% Shura 1% District / Prov. Govn. 0% Village Leaders 0%

28 Is CDC a part of the Government? 10% No Yes 91%

29

30 Seeking feedback (& monitors)... - Evaluation Team is currently designing first follow-up survey... Current QoIs include: (1) Access to services; (2) Consumption; (3) Production; (4) Female Participation in Economic Activities and Governance; (5) Structures and Functions of Village Governance; (6) Interpersonal Trust and Community Relations; (7) Attitudes towards Central Government. - Are the additional questions of interest on which NSP should be evaluated? - What metric should be used to judge the success or failure of NSP? - What questions should villagers be asked to determine whether NSP is having an impact relative to villages without NSP? - In addition, involvement in ensuring the integrity of data collection during follow-up surveys is most welcome.

31