The promise of RBF to reach the Health MDGs and the Evidence Gap: How Impact Evaluation Can Inform Policy Dialogue African Development Bank Agnes Soucat, MD, Ph.D Director Department of Human Development 1
Progress towards the health MDGs is inadequate 25 Number of children dying before the age of 5 20 4.1 Africa Asia Other 15 10 5 0 13.5 3.6 10.9 2.7 8.3 7 5.1 4.1 2.9 3.2 3.5 4.1 4.6 4.9 1.8 1960 1970 1980 1990 2000 2005 2015 with achievement of 1.4 1.1 0.1 2.2 2 0.8 3 5.1 2015 with current 2 Trend
Percent Treated fever among children under 5 by wealth quintile 50 45 46.2 40 36.5 35 30 25 26.1 28.7 31.4 20 15 10 5 0 Lowest Second Middle Fourth Highest Wealth Quintile Source: Demographic and Health Surveys 3
Most mortality causes still avoidable with low cost interventions Cause-specific proportional mortality in the Africa region Neonatal HIV/AIDS 2% 6% 21% Diarrhoeal diseases 21% Measles 7% Malaria Respiratory infections 17% 17% Injuries 4% Others 4
Billions of US$ 2009-15 The Health MDGs are within reach. Progress towards MDGs, Marginal Cost, and Fiscal Space 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% $151.0 $89.0 $160.0 $140.0 $120.0 $100.0 $80.0 40.0% 30.0% 20.0% 10.0% $48.0 $60.0 $40.0 $20.0 0.0% $- Comprehensive Medium Focussed MDG 1 MDG 4 MDG 5 MDG 6 MDG7 Source : high Level Task Force on Innovative Financing, 2009 5
Percent of total health expenditure And money is here Africa is growing at 5% a year and external aid is an important source of health spending in Sub-Saharan Africa External aid as % of total health spending (2002) 20 18 16 14 12 10 8 6 4 2 0 East Asia & Pacific Eastern Europe & Central Asia Latin America & the Caribbean Middle East & North Africa South Asia Sub-Saharan Africa Region 6
But more money does not always give better results But increasing public spending is also not enough * Percent deviation from rate predicted by GDP per capita Source: Spending and GDP from World Development Indicators database. Under-5 mortality from Unicef 72002
Countries use well-designed policies to achieve growth and human development outcomes Health, Education, Poverty But * 8
But, what looks good on paper seems to break down in practice Leakage of Funds Bad policy Poor budget handling Primary education Low quality services Provider incentives unclear, absenteeism Hard to monitor, users helpless Sub-optimal spending (Big salary bills but insufficient commodities) Financing problems Information & monitoring Local govt. incentives skewed Local capacity issues Lack of demand Externalities Community norms Budget constraints Intra-household behavior 9
Public Money Benefits Richer Groups Expenditure incidence Health Source: Filmer 2003b 10
Public Money Does Not Reach the Frontline Nonwage funds not reaching health services: Evidence from PETS (%) Health Country Mean Chad 2004 95 Senegal 2003 60 Cameroon 2004 70 Rwanda 2003 40 Source: World Bank 11
And the Results Chain to achieving the MDGs has missing links Health Systems Outcomes Upstream diagnostic Country Status Report on Health, Poverty, Health Systems and the MDGs Policy dialogue One country, one costed plan for country supported by all partners (IHP+) Financing Programs Focused funds:,malaria HIV/AIDS, Tuberculosis etc. Problem specific and health system focused operations (eg: SWAPs, PBS, PRSC) Joint Bank-Partners Programs (eg: Resultsbased financing) Health Systems Health Financing Human Resources For Health Supply Chain and Pharmaceuticals Governance and Service Delivery Infrastructure and ICT Immunization, ORT ARI treatment, Exclusive Breastfeeding, Vitamin A supplementation, ITNs, Antimalarial treatment, Attended Deliveries, Antenatal Care, Contraception, etc Multisectoral Outcomes Female Education Access to water Income School Health Health Outcomes Nutrition (MDG1) Child health (MDG4) Maternal Health and mortality (MDG5) Malaria,AIDS, TB, prevalence and mortality (MDG6) Financial protection Control of Non Communicable Diseases Control of Neglected Tropical Diseases 12
the missing middle problem. Financing Health systems Health systems Outcomes MDGs Focused funds:,malaria HIV/AIDS, Tuberculosis etc. Problem specific and health system focused operations (eg: SWAPs, PBS, PRSC) Joint Bank- Partners Programs (eg: Results-based financing) Human Resources For Health Supply chain Governance and Service Delivery Public finance and health insurance Infrastructure and ICT Other outcomes Education Water Social Protection Immunization ORT ARI treatment Exclusive breastfeeding Vitamin A supplementation ITNs Antimalarial treatment Attended Deliveries Antenatal Care Contraception etc Other sectors Outcomes Access to safe water Female education Income Nutrition (MDG1) Child health (MDG4) Maternal Health and mortality (MDG5) Malaria,AIDS, TB, prevalence and mortality (MDG6) Financial protection Non Communicable Diseases 13
What do we know and don t know We know that high impact interventions lead to improved health outcomes.. We do not know how to best produce demand and effective coverage with these interventions in specific contexts.what economists call the production function We know the inputs.. But not the incentives, institutional and management arrangements The missing middle or black box.. 14
When we do not know there are consequences Government s policies are shooting in the dark Public money is not used efficiently.. Aid effectiveness is low.. Ideology prevails.. 15
An example of an ideology driven debate user fees Proponents: user fees provide incentives to providers, reduce absenteeism, improve quality, improve accountability to users/clients Opponents: user fees reduce access and utilization, and are a burden on the poor s households budgets Evidence? 16
An example of an ideology driven debate user fees Hundreds of publications on the topic.. Yet less than 5 rigorous studies based on population surveys of service use and randomized control group Conclusions of these studies are mixed.. 17
100 20 40 60 80 0 Uganda: Utilization of Curative Services Trending Upwards Utilization rates in Uganda, 1996-2006 User fees eliminated Overall Private Government NGO 1996 1998 2000 2002 2004 2006 Year Source: UBOS (various years) The World Bank Public Expenditure Review 2008 The World 18 Bank
% Uganda: little effect of user fees abolition on MDGs related High Impact Interventions 100 90 80 70 60 50 40 30 20 10 0 2000 2006 % delivered in a health facility TOTAL DPT3 (%) Currently Using any modern FP method (%) % U5 who slept under an ITN the past night Source: DHS 2000 and 19
The promise of Results Based Financing.. Setting Evidence Based Policies : The Rwanda example 20
U5MR per 1,000 Rwanda is back on track to reach the MDGs Under five mortality trends with MDG target for 2015 250 1990 level 200 MDG target for 2015 150 100 Observed Trends since 1998 50 0 1999 2001 2003 2005 2007 2009 2011 2013 2015 Trends required to reach the 2015 target 21
Increase in utilization of high impact services Trends in assistance at delivery : Years 2000, 2005, 2007 Percentage (%) of women delivered by a health professional 22
Average number per month Rwanda : Intake of Family planning tripled in three years.. Family Planning, Modern Methods, Users at the End of the Month Average Per Health Center per Month 700 640 600 500 R 2 = 0.9784 400 300 200 175 100 0 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 2006 2007 23
Performance-based Financing (PBF) Developed after extensive piloting from 2001-2005 Objectives Focus on maternal and child health as well as communicable diseases (MDGs 4 & 5) Increase quantity and quality of health services provided Increase health worker motivation Financial incentives to providers to see more patients and provide higher quality of care Operates through contracts between Government Public and Private Health facilities providing services 24 24
Table 1: Output Indicators (U s) and Unit Payments for PBF Formula OUTPUT INDICATORS Amount paid per unit (US$) Visit Indicators: Number of 1 curative care visits 0.18 2 first prenatal care visits 0.09 3 women who completed 4 prenatal care visits 0.37 4 first time family planning visits (new contraceptive users) 1.83 5 contraceptive resupply visits 0.18 6 deliveries in the facility 4.59 7 child (0-59 months) preventive care visits 0.18 Content of care indicators: Number of 8 women who received tetanus vaccine during prenatal care 0.46 9 women who received malaria prophylaxis during prenatal care 10 at risk pregnancies referred to hospital for delivery 1.83 11 emergency transfers to hospital for obstetric care 4.59 12 children who completed vaccinations (child preventive care) 0.92 13 malnourished children referred for treatment 1.83 14 other emergency referrals 1.83 0.46 25
Evaluation Questions: Did PBF Increase the quantity of contracted health services delivered? Improve the quality of contracted health services provided? 26
Researcher & Policy Maker Collaboration Research Team Paulin Basinga, National University of Rwanda Paul Gertler, UC Berkeley Jennifer Sturdy, World Bank and UC Berkeley Christel Vermeersch, World Bank Policy Counterpart Team A collaboration between the Rwanda Ministry of Health, CNLS, and SPH, the INSP in Mexico, UC Berkeley and the World Bank Agnes Binagwaho, Rwanda MOH and CNLS Louis Rusa, Rwanda Rwanda MOH Claude Sekabaraga, Rwanda MOH Agnes Soucat, World Bank 27
Acknowledgments Funding by: World Bank Government of Rwanda (PHRD grant) Bank-Netherlands Partnership Program (BNPP) ESRC/DFID GDN 28
Evaluation Design During decentralization, phased rollout at district level Identified districts without PBF in 2005 Group districts into similar pairs based on population density & livelihoods Randomly assign one to treatment and other to control MOH reallocated some districts to treatment With decentralization, some new districts had PBF in an area of the new district must be treatment 29 29
Isolating the incentive effect PBF Performance incentives Additional resources Compensate control facilities with equal resources Average of what treatments receive Not linked to performance Money allocated by the health center management 30
Sample: Panel 165 Facilities 2006-08 2145 households in catchment areas Random sample of 14 per clinic 31
Log Expenditures Year Treatment Control Difference P-Value 15.812 15.612 0.200 2006 0.418 (1.042) (1.007) 0.241 16.906 16.989-0.083 2008 0.568 (0.71) (1.08) (0.14) Randomization balanced baseline Follow-up balanced, so difference in followup outcomes due to incentives not resources 32
A dramatic increase of utilization of services in Rwanda Rates of Assisted deliveries increased in both treatment and control groups P4P Begins treatment facilities 33
0.7 Impact on 4+ Prenatal Visits and Facility Delivery 0.6 0.5 0.4 0.3 2006 2008 No PBF 2008 PBF 0.2 0.1 0 4+ prenatal vists Facility Delivery 34
Proportion of of institutional deliveries Delivery at the health facility increased overall in Rwanda, but 7% more in PBF facilities. 60.0 55.6 50.0 49.7 7.3 % increase due to PBF 40.0 36.3 30.0 34.9 Baseline (2006) Follow up (2008) Control facilities Treatment (PBF facilities) 35 35
0.5 Impact of PBF on Probability of Child Preventive Care Visit in Last 4 Weeks 0.45 0.4 0.35 0.3 0.25 0.2 2006 2008 NO PBF 2008 PBF 0.15 0.1 0.05 0 0-23 Months 24-47 Months 36
37
Quality Conceptual Framework What They Do: (Quality) Production Possibility Frontier What They Know (Ability/Technology) 38
Goal: Use Pay for Performance to Close Productivity Gap Production Possibility Frontier What They Do Productivity Gap Conditional on Ability Actual Performance Ability/Technology 39
Prenatal Competency & Quality Provider knowledge/competency Standardized vignette presented to provider Compare answers to Rwandan CPG Measure of ability/knowledge Process quality Patient exit interview of clinical services provided Clinical content of care Provider effort 40
Standardized Prenatal effort score In the last years, PBF has increased prenatal care quality significantly 0.20 0.15 0.10 0.05 0.00-0.05-0.10-0.15 0.15 0-0.10-0.13 Baseline (2006) Follow up (2008) 15 % Standard deviation increase due to PBF Control facilities Treatment (PBF facilities) 41 41
.2.3.4.5 Kernel Non parametric regression practice-competency at baseline Control facilities Treatment facilities.3.4.5.6.7.8 Competence 42
.2.3.4.5 Kernel Non parametric regression practice-competency at follow up Control facilities Treatment facilities.3.4.5.6.7 Competence 43
Results Summary Balanced at baseline Expenditures same, so isolate incentives Impact on utilization Delivery & Child prevention, but not prenatal Impact on prenatal quality Bigger for better doctors Reduced child morbidity & Taller children Effect sizes bigger than most other interventions 44
Discussion Rwanda is back on track towards the health MDGs because of many different factors including Strong political leadership Micro-Insurance (Mutuelles) Autonomy and Fiscal Decentralization (Imihigo) HIV services and earmarked funding PBF effect seen despite many other national level intervention: possible bigger effect in other countries 45 45
Average number per month Beware the pre-post trap : the example of Family Planning in Rwanda Family Planning, Modern Methods, Users at the End of the Month Average Per Health Center per Month 700 640 600 500 R 2 = 0.9784 400 300 200 175 100 0 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 2006 2007 46
Conclusions Reaching the MDGs or at least making a major dent towards reaching them- is possible even within the next five years.. Results Based financing can be a powerful way to address the problems of the missing middle and inject incentives into the implementation black box Only because of the rigorous impact evaluation conducted in Rwanda can we conclude that RBF played a role in the increased utilization of services Impact E valuation should systematically be nested into major policy interventions 47