The debate about the Randomized Controlled Trial (RCT) in impact evaluation:

Transcription

1 an institute of the The debate about the Randomized Controlled Trial (RCT) in impact evaluation: A reflection on the professionalization in the field of evaluation Rahel Kahlert, Ph.D. Ludwig Boltzmann Institut Health Promotion Research EES, 2 October 2014 LBIHPR: A-1020 Vienna, Untere Donaustraße 47, Austria office@lbihpr.lbg.ac.at FAX - 50 in cooperation with: Co-funded by:

2 2 Topics from two perspectives: RCT as translational approach in the interdisciplinary context RCT as cultural demarcation in the evaluation discipline in the professional debate 2

3 Data used Review of grey literature E.g., evaluation guidelines Evaluation reports Training materials Electronic lists of evaluation communities Primary data collection Interviews with evaluation experts Surveys of evaluation professionals Documents of evaluation theories Literature from other disciplines (agricultural, medical, development) 3

4 Methodological approach Historical-critical approach in the hermeneutic tradition Origins: Wilhelm Dilthey: Studies Toward the Foundation of the Human Sciences (1924; 1970) Hans-Georg Gadamer: Truth and Method (1960, 2004) Central ideas: Distinction between the natural (explaining) sciences and the social (understanding/hermeneutic) sciences Iterative approach from a proto-understanding to a refined understanding of a concept 4

5 1. Context Pressure of funders to demonstrate impact Demand "to improve the quality of aid and its impact on development (The Paris Declaration on Aid Effectiveness) But how? How should impact be evaluated? 5 5

6 2. What is impact? Long-term effect Net effect Most definitions fall somewhere in-between the continuum Ask where the definitions of the different stakeholders fall, e.g., of the Funding institution Field operations team your own definition Original and actual program theory Is there a conflict between definitions of the different stakeholders? 6 6

7 3. How to evaluate impact? Two traditions, depending on impact definition: Long-term effect Net effect Best method to evaluate impact: Multi-disciplinary and multi-methods approach Experimental designs/ Randomized controlled trials (RCT) 7 7

8 Impact as long-term effects (left side) Comprehensive definition in the OECD DAC Glossary long-term effects produced by a development intervention positive and negative primary and secondary directly and indirectly intended and unintended economic, socio-cultural, institutional, environmental, technological etc. 8 8

9 Impact in the logic model Source: Kellogg Logic Model Development Guide (2004) Note: Long-term dimension of impact No clear methodological choice Context, non-linear processes, unintended consequences need to be added 9

10 Impact as net effect (right side) Economic tradition of defining impact Comparing the outcomes of an intervention with what would have happened without the intervention (=counterfactual) Source: World Bank Research Group 10 10

11 Evaluating impact via RCT Using a counterfactual ( observed intervention impact (counterfactual situation) Baseline measure Outcome measure 11 11

12 Randomized experiments as hard evidence When we talk of hard evidence, we will therefore have in mind evidence from a randomized experiment, or, failing that, evidence from a true natural experiment, in which an accident of history creates a setting that mimics a randomized trial. (Banerjee 2007, with respect to development evaluation) 12

13 Example of methods hierarchy: Maryland Scale of Scientific Methods (for evaluating crime prevention) 13

14 RCT debate case study: NONIE Assumption: nothing works in development due to bad evaluations Solution: guidelines on rigorous impact evaluation Challenges: Disagreement about what is best method and evidence Methodological versus political stakes Kahlert, 1 LBIHPR

15 Decision tree (with RCT as starting point) Source: NONIE Guidance for Impact 15

16 Critiques of RCTs The critiques can be organized into three categories: 1. Critique is based on a misunderstanding of RCTs and is therefore not justified. 2. Critique is applicable to the large-scale federal RCTs in the 1970s and 1980s, but improvements have been made since. 3. Critiques address true limitations of RCTs, but which researchers attempt to address. 16

17 Critique 1: misunderstanding RCTs create an artificial environment (1). RCTs do not contextualize program (1). 17

18 Critiques 2: new generation of RCTs RCTs require too much standardization and control (2). Untreated control group is often not a policy option (2). 18

19 Critiques 3: True limitations, but could still be addressed RCTs do not explain why a program is effective (3). RCTs have low external validity (3). RCTs are only an ideal type (3). RCTs cannot be implemented flawlessly (attrition, contamination, not triple-blind; 3). 19

20 an institute of the Importing Randomized Controlled Trials (RCTs) from drug trials to public program evaluation: Pitfalls, policy implications and recommendations The debate about the Randomized Controlled Trial (RCT) in impact evaluation: A reflection on the professionalization in the field of evaluation Rahel Kahlert, Ph.D. Ludwig Boltzmann Institut Health Promotion Research EES, 2 October 2014 LBIHPR: A-1020 Vienna, Untere Donaustraße 47, Austria office@lbihpr.lbg.ac.at FAX - 50 in cooperation with: Co-funded by: