Impact Evaluation of Project Ujjwal

Transcription

1 Impact Evaluation of Project Ujjwal Improved Family Planning and Reproductive Health Services in India Mid-Term Survey Report Oxford Policy Management - New Delhi July 2015

2 Acknowledgements The analysis of data and writing of the report has been undertaken by Oxford Policy Management Limited (OPM). OPM also collected the qualitative data. The quantitative survey was undertaken by Sigma Research and Consulting, who were selected and contracted by Futures Group (the lead of the consortium implementing Project Ujjwal). The roles and responsibilities of the team members were as follows: The project was managed by Tom Newton-Lewis. Development of survey research tools was undertaken by Mehjabeen Jagmag, Pooja Singh, Prakash Nayak, Purava Joshi, Rael Sarmeen and Vanika Grover at the OPM New Delhi office Quantitative data collection and data entry were undertaken by Sigma Research and Consulting (Dilip Parida, SVSRK Bharadwaj, Tilak Mukherji and UV Somayajulu). Quality assurance and management of fieldwork were undertaken by OPM s Prabal Vikram Singh. Contributions to the design, analysis and writing of the report were made by the following staff at OPM s New Delhi office: Mahika Shishodia, Mehjabeen Jagmag, Nayan Kumar, Pooja Singh, Prabal Vikram Singh, Prakash Nayak, Purava Joshi, Rael Sarmeen, Ruhi Saith, Tom Newton- Lewis and Vanika Grover. Additional quantitative research support was provided by Ridhima Gupta. Additional qualitative research support was provided by Shilpa Maiya. Specialist technical support and inputs were provided by OPM s Oxford office: Michele Binci, Nouria Brikci, Robert Greener and Tomas Lievens; and OPM s Nepal office: Sanjaya Acharya. The team acknowledges the contributions of the evaluation reference group (Loveleen Johri, Ramesh Bhat and V. Selvaraju) for providing quality assurance for the entire survey process and report content. The team also thanks DFID (Rashmi Kukreja, Sudipta Mondal and Sudha Menon) and the Futures Group (Amit Bhanot, Hanimi Reddy, Nidhi Tikku and Tanya Liberhan) for their valuable comments and inputs. 1 Oxford Policy Management New Delhi

3 Executive summary 1. Introduction This report presents the findings of data collected in the second year of Project Ujjwal. The data from this first post-implementation survey/mid-term survey contributes to the evaluation of Project Ujjwal and is to be used to a) report on the current status of implementation and b) provide a base against which the impact of the project (between the collection of this data and the end line, when the second survey is to be undertaken) is assessed. Project Ujjwal is a three-year project (April 2013 to March 2016) funded by Department for International Development (DFID). It is being implemented by a consortium led by Futures Group and including Hindustan Latex Family Planning Promotion Trust (HLFPPT), the Public Health Foundation of India (PHFI) and the Johns Hopkins Bloomberg School of Public Health Center for Communication Programs (JHUCCP). It aims at reducing unwanted maternal and infant deaths in Bihar and Odisha through increasing the provision and uptake of family planning services as well as services for safe abortions. This is to be done by using a total marketing approach that is, drawing on the private sector (social franchise and social marketing) to address gaps in services and also through a range of public private partnership (PPP) approaches involving government. The combination of supply- and demand-side interventions is as follows: Social franchise (SF): Increased choice of sites providing quality clinical family planning/reproductive health (FP/RH) services with a focus on clinic-based services in rural and underserved areas (with the creation of 280 Social Franchise Clinics) by inviting existing private clinics to join the newly created Ujjwal franchise Social marketing (SM): Increased choice of social marketing sites providing quality FP/RH products, with a focus on rural and underserved areas (with the creation of 18,000 Social Marketing Outlets) by inviting existing traditional outlets (for example chemist shops) to join the Ujjwal franchise as well as involving non-traditional outlets (like paan and bindi shops) Capacity building and training (CB and Training): Building FP/RH capacity of providers, providing training and mentoring support, and facilitating improved implementation of the Pre- Conception and Pre-Natal Diagnostic Techniques (PCPNDT) Act as well as undertaking quality assurance Demand generation: Generating demand through a) supply-led demand creation increasing awareness about improvements in supply (resulting from the above supply-side interventions) by Ujjwal Saathis (trained field-level workers attached to the franchised clinics), and b) consumer-led demand creation focussing on improving knowledge and overcoming barriers to FP uptake, and addressing gender norms through communication (for example, mass media) and community outreach for delaying and spacing births (rather than focussing just on limiting) A limitation of this study is that in the absence of baseline data, a comparison of the mid-term and end line surveys will not capture the full impact of the project. However, given the findings at the population level at mid-term as described below, it is expected that any change captured between the mid-term and end line surveys will capture the majority of the project impact. In the sections that follow in this executive summary, the theory of change underpinning Project Ujjwal s interventions and detailing the inputs, outputs, intermediate outcomes, outcomes and impacts, is shown first (Section 2). Methods of analysis to be used for the evaluation are summarised next (Section 3), followed by an outline of the data (quantitative and qualitative) collected (Section 4). Findings from the analysis of the data are presented in Section 5 in three subsections. In subsection 5.1, the progress of implementation of the four interventions mentioned above is presented, focussing on the inputs and intended outputs of each. This is followed in subsection 5.2 by a presentation of the effect of the interventions as assessed by intermediate Oxford Policy Management New Delhi 2

4 outcomes and final outcomes. Note that while each intervention has specific inputs and outputs, they all result in common intermediate outcomes (related to uptake of family planning methods and services and safe abortions) and outcomes (related to the contraceptive prevalence rate [CPR] and unmet need). In subsection 5.3, an assessment of Project Ujjwal with regard to equity and inclusion is provided. The executive summary ends with implications of the survey findings (in section 6) for the design and implementation of Project Ujjwal between now and the end line. 2. Theory of change As shown in Figure 1Figure 2, Project Ujjwal has four interventions (mentioned above): three directed towards addressing the supply side (SF, SM, and CB and training) and one on the demand side (demand generation both supplier and consumer-led). In the figure, the stages in the conversion of the inputs (from these interventions) to the impact are shown. The assumptions made with regard to the conversions of the different stages are also outlined. It is expected that taken together the inputs of the supply-side interventions will result in the outputs of improved supply (availability, range, price and quality) of family planning products and services. Similarly, the inputs of the demand-generation interventions are expected to result in the outputs of improved awareness and knowledge as well as favourable attitudes to family planning amongst the target population. The resulting increased demand for family planning products and services is expected to be met by Ujjwal s supply-side interventions. This is expected to result in the intermediate outcomes of changes in practice reflected by a) increase in uptake of Ujjwal FP products and services (including safe abortions). It is assumed that this is an absolute increase and not just a substitution effect (that is, new users, not existing users who are just changing providers) and b) increase in uptake from other providers (like for example the public sector). Combined together this would thus result in overall outcomes (at the population level) of increased CPR and decreased unmet need. A combination of the quantitative data from household surveys and facility surveys, along with the qualitative data collected from some individual users as well as providers, should allow us to assess the changes in CPR and unmet need, and the contribution of uptake of Ujjwal s products and services to this. The outcomes will translate into an impact measured by health indicators like maternal mortality rate (MMR) and total fertility rate (these are not, however, measured by the data collected for this report). While inputs and outputs are specific to each intervention (as detailed in the TOC related to individual interventions in the respective chapters), the intermediate outcomes and outcomes are common to all the interventions. Oxford Policy Management New Delhi 3

5 Figure 1: Theory of change Project Ujjwal Oxford Policy Management New Delhi 4

6 3. Methodology The overall evaluation to be conducted at end line involves an assessment of the impact of Project Ujjwal on the target outcomes (contraceptive prevalence rate and unmet need) and also an assessment of impact of the project on less quantifiable dimensions (like attitudes and gender norms). Assumptions as well as inputs, outputs, intermediate outcomes and outcomes along the theory of change are to be assessed. This, therefore, requires the use of a mixed-methods approach, including both quantitative and qualitative data analyses with data collected during the mid-term survey providing the basis against which data will be compared at end line. The impact evaluation methodology involves use of a regression analysis applied to panel data (that is, data from the same households and individuals collected now and at end line) so that a range of observable and non-observable factors can be controlled for and any significant impact of variables associated with the Ujjwal interventions can be isolated to the extent possible. The regression analysis will also help understand the influence played by a large range of factors, including for instance household and individual characteristics, on the project s target outcomes. This method, which will be applied when the end line data is collected, was chosen in preference to other quasiexperimental methods of analysis, since the implementation did not allow for separation into clear control and treatment groups. The qualitative evaluation involves the use of a framework analysis approach to identify the themes emerging from the data, such as to allow the undertaking of a) a diagnostic review of the servicedelivery chain (medical practitioners, outlet owners and health workers who are part of the project), and b) a detailed study of the demand for the intervention or lack thereof. Whilst the impact will be evaluated at end line, the data from the current survey has been collected and analysed in preparation for the analysis at end line. The findings from this analysis are presented in this report. In order to ensure a consistent mixed-methods approach, the findings have been presented after undertaking a reiterative integration of the findings from the quantitative and qualitative analyses, rather than presenting them separately. 4. Data collection The quantitative data was collected between May and September The surveys included: a) a secondary sampling unit (SSU) survey covering 600 villages or urban wards, b) a household survey covering 14,344 households and including 16,791 women (in the years age group) and 7,929 men (who are husbands of eligible women), and c) a supply-side survey including 61 Social Franchise Clinics, 629 Social Marketing Outlets and 616 health workers. Being representative at the state level, the household survey (data from the same households is to be collected at the end line) allows an assessment of changes in the outcome indicators of CPR and unmet need at the state level, between the current survey and the end line. The qualitative data was collected post the quantitative surveys, from four districts in Odisha and four districts in Bihar. Fieldwork spanned four weeks in each state, being conducted between mid- November to the first week of December in Odisha and from the last week of November to mid- December in Bihar. Investigating the supply side, a total of 15 Ujjwal Saathis, 15 SM Outlet staff, 16 SF Clinic staff and 7 members of the District Quality Assurance Committee (DQAC) were interviewed across Bihar and Odisha. With regard to the intended beneficiaries of the programme, a total of 57 in-depth interviews and 15 focus group discussions were carried out with women, and 45 in-depth interviews and 16 focus group discussions with men across the two states. Oxford Policy Management New Delhi 5

7 More on methodology and data collection See Chapter 3 for a full description of the methodology and data collection, including: A discussion of the quantitative methodology (with details of the evaluation strategy, data collection and sampling strategy) Details of the qualitative methodology and data collection A description of the limitations and the mitigation strategies employed 5. Findings The size, socio-demographic landscape and political situation in Bihar and Odisha are quite different. It is not surprising, therefore, that both states at the time of the survey showed differences in the health indicators including those related to fertility and family planning. Data has therefore been analysed state-wise as well as overall. Wherever there is substantial divergence in the findings for Odisha and Bihar, figures for both are presented else, overall values are given. Findings related to inputs and outputs specific to each intervention are presented first in section 5.1. This is followed in section 5.2 by the findings related to intermediate outcomes and outcomes, which are common to all interventions. In section 5.3 findings related to equity and inclusion with regard to the overall programme are presented. 5.1 Findings: Progress of implementation As of July 2014, Project Ujjwal had established 308 SF Clinics, against the target of 280; trained 5,784 Ujjwal Saathis against the target of 6,000; and established 32,008 SM Outlets against the target of 18,000. The 61 SF Clinics that had been surveyed had, on average, joined Ujjwal nine months before the survey, and the 562 SM Outlets surveyed had joined Ujjwal 11 months prior to the survey. As per the July 2014 Annual Review, trainings were undertaken for more than 600 private health providers on FP/RH, followed by regular sessions and medical audits for clinic quality assurance in collaboration with the Federation of Obstetric and Gynaecological Societies of India (FOGSI). Project Ujjwal s consumer-led demand creation activities prior to or around the time of data collection consisted of activities such as mass-media campaigns (conducted in February and March 2014) and community-level folk performances (from March to July 2014), meant to generate awareness about the various family planning methods and inspire behaviour change by influencing attitudes towards family planning. Additional rounds of activities were also to be conducted later. Given the number of SF Clinics and SM Outlets inducted into the network, the average duration of time these were inducted into the programme, and the trainings and communication-related activities that had been undertaken (mentioned above), it can be expected that the franchise clinics and outlets would be reasonably operationalised at the time of the survey. Some progress would also be expected with regard to outputs (an increase in availability of family planning products with a shift towards spacing methods, effective training, increased knowledge and impact on attitudes to family planning) by both supply side and demand side interventions. Survey findings related to assessing such operationalisation and progress are presented below Assessing inputs and outputs Inputs and outputs related to Intervention One (SF Clinics): Oxford Policy Management New Delhi 6

8 Of the services offered by the SF Clinics, family planning counselling was the most commonly availed of, followed by injectables and IUCDs; this was more so in Odisha (where an average of 49 counselling sessions were provided by the SF Clinics in the month before the survey, compared to 14 in Bihar). Other services like male sterilisation, female sterilisation and abortions were offered at less than 40% of the clinics. With regard to FP products, 82% of the facilities had received products from the Ujjwal network by the time of the mid-term survey. The most commonly stocked products were OCPs, male condoms and pregnancy test cards. However, no product was stocked by more than 40% of the SF Clinics. With respect to ancillary services, the SF Clinics in Odisha performed much better than those in Bihar. Eighty-four per cent of Ujjwal clinics in Odisha had organised fixed day service (FDS), compared with only 4% in Bihar. The provision of injectables and female laparoscopic sterilisation was a big focus 93% and 86% SF Clinics, respectively, offered these services during the FDS. Eighty-four per cent of clinics in Odisha had an Ujjwal Saathi, compared to 38% in Bihar (overall 62%). The vast majority of Ujjwal Saathis reported providing family planning counselling to households (83% of the Saathis) 1 and referring clients to Ujjwal facilities (70%). In Odisha, 32% of Ujjwal Saathis reported that they were stocking family planning products, compared to 44% in Bihar. The qualitative data collected during the post-implementation study suggested that several clinics saw value in their affiliation with Ujjwal. A large number of doctors in both Bihar and Odisha, however, expressed their unhappiness over delayed payments and non-payments for activities that they had conducted for Ujjwal, and also over lack of receipt of products and services they had expected from the project. More on Intervention One See Chapter 4 for a full description of Intervention One, including: A description of the theory of Change for Intervention One that guided the evaluation design, data collection and analysis An assessment of the general details of SF Clinics, including the stock of FP products from SF Clinics, and the ancillary services A discussion of the reasons for demand for the products/services offered at the SF Clinics, based on the qualitative interviews Inputs and outputs related to Intervention Two (SM Outlets): The most commonly stocked product in the SM Outlets across both states was male condoms (stocked at 78% of the SM Outlets), followed by OCPs (23% of the SM Outlets). However, with regard to being supplied with products by Ujjwal, the situation was unlike that of the SF Clinics, where more than three-fourths of the sample had received products from the network. Only about half of the SM Outlets had received FP products from Ujjwal by the time of the survey. 1 Note the discrepancy with the results of the household-level survey (described under findings for Intervention Three) showing that 0.2% of both women (41 women) and men (13 men) said they had discussed family planning with an Ujjwal Saathi. Oxford Policy Management New Delhi 7

9 For SM Outlets, there was little perceptible impact on product offerings after joining Ujjwal, except in the case of male condoms 51% of the outlets reported that they stocked male condoms before joining Ujjwal, as against 78% stocking them at the time of the interview (that is, after joining the network). If taken at face value, this suggests a possible rise of 27 percentage points. The products that were most sold at the SM Outlets were male condoms, OCPs, IUCDs and sanitary napkins. The sales were noticeably higher in Odisha as compared to Bihar. Both men and women felt that it would be more appropriate if men purchased contraceptives, mostly due to societal norms. This points to the possibility that much of the customer demand could be driven by men. Thus, it appears that not only did women have limited access to these sources of contraceptives due to social norms, but also that this layer of service delivery in Project Ujjwal (SM Outlets) had limited interaction with and influence on women. In the qualitative data sample, across both states many outlet (traditional and non-traditional) owners expressed the view that though Ujjwal products seemed to be in demand in their area, they often faced stock shortages that were a hindrance to meeting this demand. The shortage in stock was expressed as a major factor causing dissatisfaction, especially when customers developed a preference for a brand supplied by Ujjwal and the outlet then ran out of stock. More on Intervention Two See Chapter 5 for a full description of Intervention Two, including: A description of the theory of change for Intervention Two that guided the evaluation design, data collection and analysis An assessment of the general details of SM Outlets, including the stock of FP products from SM Outlets, and whether joining Ujjwal had an impact on this Inputs and outputs related to Intervention Three (capacity building and training): Under Intervention Three, Ujjwal provides training on FP/RH issues to health service providers the staff at SF Clinics and SM Outlets as well as Ujjwal Saathis. Almost 79% of the SF Clinics reported that their staff had received training on FP/RH issues. However, this figure was only 0.7% for the staff at SM Outlets. Among the SF Clinics that had training sessions, 44% of the total staff (full-time and part-time staff) received training. In all, 62% of the clinics reported that an Ujjwal Saathi was linked to the facility. Out of the clinics with an Ujjwal Saathi linked to them, 84% reported that the Saathis had received FP/RH training from Ujjwal. The data collected from Ujjwal Saathis showed that 71% of them reported having received training from Ujjwal. In contrast, in both states a very low proportion had received any interpersonal communication (IPC) training (7% for Odisha and 2% for Bihar), although interpersonal communication was to play an important role. Though a majority of the Saathis had received training, the data showed inadequacies in several aspects of their knowledge for example, a greater proportion of health workers (44%) reported that they would recommend OCPs/IUCDs as a limiting method, compared to those who would recommend female sterilisation (28%). Less than half the Ujjwal Saathis interviewed (44%) reported knowing the side effects of unsafe abortions. Qualitative studies, too, revealed that Ujjwal Saathis harboured several misconceptions about pills. Oxford Policy Management New Delhi 8

10 In qualitative interviews, doctors and other medical staff at SF Clinics reported having benefited from the training. Of the few respondents who felt that they did not benefit, it was because they had received similar training in the past or they felt that the compensation for attending the training was inadequate to cover the loss of work at the clinic. Respondents also said that they would like more practical training and information on permanent sterilisation methods and the use of new surgical instruments, postpartum IUCDs and injectables. Staff at SM Outlets who had not received any training also expressed the need for some training or information related to the products supplied to them. Under Project Ujjwal, attempts are being made to implement quality assurance (QA) standards. Fifty-six per cent of the SF Clinics surveyed reported that they were visited by a QA officer as part of the Ujjwal programme. The percentage of SF Clinics visited by such an officer was much lower for Odisha (34%) compared to Bihar (79%). The District Quality Assurance Committees (DQACs) were not familiar with Project Ujjwal and empanelment of SF Clinics on government programmes appears to face challenges with inadequate interest on the part of the clinics as well as the government. Although a majority (79%) of the SF Clinics in Bihar had been visited by a QA officer, and almost 90% said they followed guidelines related to family planning and safe abortion services, qualitative data suggestive of malpractices providing incorrect information to clients, discriminating against HIV-positive patients, etc. was largely obtained from Bihar. More on Intervention Three See Chapter 6 for a full description of Intervention Three, including: A description of the theory of change for Intervention Three that guided the evaluation design, data collection and analysis An assessment of the training and capacity building conducted for the clinic-based staff and Ujjwal Saathis A discussion of abortion services provided under Project Ujjwal A description of the quality-assurance standards implemented under the Project Inputs and outputs related to Intervention Four (demand creation): Under Intervention Four, the project aims to increase the awareness and uptake of family planning products and services provided by Ujjwal s interventions through various demandcreation activities. The data analysed showed that there was limited rollout of these interventions by the time of the survey. To examine the extent to which the knowledge of supply-side changes was transmitted to the target population, the survey asked men and women about their knowledge of Ujjwal s interventions. Of the 16,265 women and 7,336 men surveyed, 0.4% of women and 0.5% of men were aware of an Ujjwal facility, and less than 0.1% of the respondents in the sample had visited an Ujjwal clinic. Around 0.1% of women and 0.5% of men had visited an Ujjwal FDS, and 0.2% of men and women had discussed a family planning issue with an Ujjwal Saathi. Less than 1% of the respondents had heard of the Ujjwal helpline (0.1% [11] women and 0.4% [33] men). Of those who had heard of the Ujjwal helpline, 13% (two) of women and 10% (three) of men had used it. Oxford Policy Management New Delhi 9

11 The population-level data showed that 2% of the women and 3% of the men surveyed had heard of an Ujjwal SM Outlet. Only one woman from the sample had visited an SM Outlet, and she reported not having bought any products during her visit. Six men had visited an SM Outlet; three of them bought male condoms and three did not buy any product. Intervention Four also aims to improve the quality of FP counselling by strengthening the interpersonal communication and counselling skills of Ujjwal Saathis and SF Clinic staff. However, of the 546 Saathis interviewed during the survey, only 5% reported that they had received training in IPC from Ujjwal. The population-level data collected showed that 0.2% of both women (41 women) and men (13 men) said they had discussed family planning with an Ujjwal Saathi. Of them, 12 women and 3 men recollected seeing a short film/interview on family planning on a mobile phone, shown to them by an Ujjwal Saathi. Among the respondents who saw these films/interviews, nine women and one man found these to be helpful in making FPrelated decisions. A low proportion recollected folk performances being organised in their village by Ujjwal (none of the men, and 1.5% of the women). Amongst the female respondents who said that a folk performance was organised in their village (111 women), one respondent from Odisha said that the performance was organised by Ujjwal. This one respondent also attended the performance and obtained FP product information or counselling at the event. Additionally, only 0.4% of women and 1% of men recollected a mobile van visiting their village for health and family planning promotion in the three months preceding the survey. In the qualitative survey, a smaller sample of men and women in villages were asked if they had heard about or attended folk performances organised by Project Ujjwal. While most of the respondents replied in the negative, there were a few married men in the sample who said that they had heard about Ujjwal folk performances being organised, but were unable to attend as they were busy with their work at the time. This finding also related to the convenience of the timing and venue of these events, not only for the organisers but also more importantly for the target population. The qualitative data revealed that many supply-side functionaries and demand-side respondents felt that while Project Ujjwal was organising folk performances and other activities that were beneficial, more awareness needed to be generated about the occurrence of these events through better promotion. They also felt that existing platforms such as gram sabhas and selfhelp group (SHG) meetings could be utilised as a means to reach a wider range of people. With regard to accessing information related to family planning, both quantitative and qualitative data revealed that women s access to information sources differed from those of men owing to social norms and attitudes around discussion of such topics. The primary source of information about family planning was through the television for women (for 23%) and through wall hoardings for men (24%). From among newspapers/magazines, radio and television, the least accessed medium by female respondents was the radio; only 1% said they heard the radio almost every day, whereas 24% said they watched the television almost every day. This finding was similar for male respondents but with the proportion of men (14%) who reported seeing FP messages in newspapers/magazines being twice that for women. Women were also found to often access information through traditional networks (for example, family members). The data also seemed to suggest that outlets, especially TOs such as medical stores, were generally seen as a credible source of knowledge and information, more so for new users or those who had no specific product in mind. However, with little or no formal training, the nature of advice and counsel provided by these outlets was quite variable, and was influenced by personal perceptions, misconceptions and biases. Oxford Policy Management New Delhi 10

12 The quantitative surveys also asked men and women about their awareness of, and attitudes towards, use of family planning methods. The methods that saw high levels of awareness were female sterilisation (98% women and 98% men were aware of it), male sterilisation (75% women and 87% men), OCPs (75% women and 68% men), and male condoms (61% women and 90% men). On the other hand, modern methods that saw low levels of awareness were female condoms (3% women and 9% men), ECPs (13% women and 24% men), and injectables (39% women and 37% men). A substantial proportion (41%) of female respondents (of those who said they were aware of female sterilisation) said that they were currently not using this method but were willing to. The willingness to use other methods was much lower (between 3 and 10% for other methods). More on Intervention Four See Chapter 7 for a full description of Intervention Four, including: A description of the theory of change for Intervention Four that guided the evaluation design, data collection and analysis A discussion of findings related to the four core demand-generation inputs of the project creation of smart couples, smart providers, smart communities, and a smart environment An assessment of the knowledge and attitudes of the surveyed men and women towards family planning 5.2 Findings: Assessing the effects of interventions Inputs and outputs related to each of the four interventions have been discussed above. The effects that these translate into are assessed by intermediate outcomes and outcomes. As mentioned earlier, the former relate to changes in practice reflected by a) increase in uptake/sales of Ujjwal FP products and services and b) increase in uptake from other providers (like for example the public sector). For this evaluation, data has only been collected with regard to a). The combined effects of a) an b) are captured by the overall outcomes (at the population level) of increased CPR and decreased unmet need. Findings related to the inputs and outputs suggested reasonable progress in operationalisation. In the absence of a baseline, findings related to intermediate outcomes and outcomes presented below are largely providing the figures against which to assess the impact of the project between the undertaking of this survey and the end line Intermediate outcomes Practice Forty per cent of the surveyed women had used a family planning method at least once. This estimate was greater for modern methods (37% women) than for traditional methods (9% women). For those women who had discontinued any method, the most common reasons were that they did not feel any need to use it (33%) and that they wanted more children (22%). Of the ever users, 82% of the women were currently using a family planning method. The primary decision to currently use a specific FP method was usually made jointly by the couple. In general, female respondents who were willing to adopt family planning expressed a greater preference towards limiting (41% of those aware of female sterilisation reported willingness to use it) than spacing methods (between 3% and 10% of those aware of the different spacing Oxford Policy Management New Delhi 11

13 methods expressed willingness to use them). Male respondents, on the other hand, indicated a greater preference towards traditional methods of family planning than modern ones. Acceptance of male methods of sterilisation was negligible compared to acceptance of female methods of limiting. The data also showed that the mother-in-law or the mother had an influence over FP practices, with a marginally bigger influence than the spouse. When asked whether they preferred private or public facilities for FP services, more than threefourths of the respondents reported they would rather visit government/public facilities than private facilities to undergo female sterilisation, male sterilisation and IUCD insertions. Amongst those who said they used OCPs/ECPs, injectables and condoms, a greater proportion reported using private facilities (rather than government/public facilities). The qualitative data revealed that no one single factor drove preference for government or private facilities, nor was there a clear preference for one over the other. For the respondents who preferred going to government facilities, the main reason was affordability. Many women and men felt that the quality of services provided by private facilities, especially in terms of quality of counsel provided by doctors, was better, though this entailed higher expenditure. Sales and related In order to measure changes in uptake of FP products the magnitude of the sale of FP products and services by the various Ujjwal inputs (SF Clinics, SM Outlets and Ujjwal Saathis) was recorded. The products that saw the highest quantity of sales from the SF Clinics, according to the quantitative data, were OCPs, male condoms, IUCDs and MTP kits. OCPs were particularly popular in Bihar, with an average of 110 units being sold by the SF Clinics in the month before the survey, compared to 45 units for Odisha. In the context of sales, the qualitative interviews recorded a high customer demand for procedures related to female sterilisation. The data also revealed the low demand for male sterilisation especially in Bihar, with the number of men opting for the procedure in Odisha (an average of 14) being nine times that for Bihar (an average of two) in the month preceding the survey 2. In qualitative interviews, doctors revealed how their personal preferences for methods and products shaped the counsel they provided. Besides, their recommendations depended not only on the number but also on the gender of the children. The products that were most sold at the SM Outlets were male condoms, OCPs, IUCDs and sanitary napkins. The sales were noticeably higher in Odisha as compared to Bihar. Both men and women felt that it would be more appropriate if men purchased contraceptives, mostly due to societal norms. This points to the possibility that much of the customer demand could be driven by men. Thus, it appears that not only did women have limited access to these sources of contraceptives due to social norms, but also that this layer of service delivery in Project Ujjwal (SM Outlets) had limited interaction with and influence on women. In the qualitative data sample, across both states many outlet (traditional and non-traditional) owners expressed the view that though Ujjwal products seemed to be in demand in their area, they often faced stock shortages that were a hindrance to meeting this demand. The shortage in stock was expressed as a major factor causing dissatisfaction, especially when customers developed a preference for a brand supplied by Ujjwal and the outlet then ran out of stock. 2 The raw numbers are 13.7 and 1.5; thus nine times Oxford Policy Management New Delhi 12

14 The sales from the SF clinics and outlets were converted into CYPs using standard multipliers. It was calculated that over two years of implementation, at current rates of sales, the 280 clinics would generate 780,516 CYPs; the 4,269 Ujjwal Saathis would generate 935,140 CYPs; and the 18,000 SM Outlets would generate 614,717 CYPs. Note that this includes the CYPs that would have been generated in any case in the absence of Project Ujjwal except with regard to nontraditional outlets that were earlier not selling FP products. The provision of services for safe abortion is an important aspect of SF Clinics as well. The quantitative surveys conducted for this study indicated that of the women who had had at least one abortion, 31% said they had undergone their last abortion at a private facility (55% in Bihar and 23% in Odisha), whereas 34% women said they undergone their abortion at a government facility (14% in Bihar and 40% in Odisha). Forty-three per cent of the SF Clinics reported that non-ujjwal facilities had referred couples to them for abortion; this statistic for Bihar (59%) was about double of that in Odisha (28%). These SF Clinics had received an average of three such referrals in the month before the survey. The most common reasons for receiving abortion referrals from non-ujjwal sources was that the SF Clinics were safe and reliable (according to 92% of the SF Clinics that had received such referrals), that the services provided were of good quality (77%), that the services were cheap (58%), and that the Ujjwal brand was trusted (35%). More on intermediate outcomes See Chapter 8 for a full description of the intermediate outcomes, including: Profiles of FP products currently being used by the surveyed men and women Details of the reasons for discontinuing family planning A discussion of the respondents preference for private or public service providers An integrated assessment of the findings surrounding the knowledge, attitude and practice of the respondents regarding family planning Details of sales of FP products and services by SF Clinics and SF outlets A discussion of the factors influencing quantity sold from the SM Outlets, based on the qualitative interviews Calculations of the couple years of protection generated by SF Clinics, Ujjwal Saathis and SM Outlets A discussion of the abortion services provided by SF Clinics Outcomes The CPR as calculated from the survey data was 36% for Bihar and 52% for Odisha. The unmet need was 24% and 20%, respectively. The data was disaggregated to understand the mix of family planning methods used that resulted in these values. This will allow an assessment at end line of the contribution of the project to changing the mix of methods (and the extent of shift, if any, from limiting to spacing methods). The major contribution to the CPR was by female sterilisation, the predominant family planning method used in the two states (28% of women in Bihar and 27% in Odisha), in both rural and urban areas. In Bihar, besides female sterilisation, there were almost no other methods reported, with the exception of condoms (used by 4% of men in urban areas) and withdrawal (again used by 4% men in urban areas). In Odisha, use of withdrawal was higher than in Bihar (11% and 2% Oxford Policy Management New Delhi 13

15 men in rural and urban areas respectively reported practising the method in Odisha and Bihar, respectively), driving the vast majority of the difference between the state-level CPR rates. The analysis of CPR undertaken by age, by method, by number of children, and by gender of children suggested that there were strong norms around family size, with limiting methods being used in substantial numbers after two children in Odisha and three children in Bihar, and almost no contraceptive methods being used amongst couples with no children. Use of spacing methods was much higher in Odisha than in Bihar after the first birth 35% and 9% of the women in Odisha and Bihar, respectively, reported the use of spacing methods after the first birth. The findings suggested strong gendered norms around family size, with substantially higher rates of contraception use among women who had at least one boy but no girl, than among those with at least one girl but no boy. This was the case in both states but more so in Bihar. Women in Bihar with at least one boy but no girl were 27 times more likely to use limiting methods than those with at least one girl but no boy. The comparative figure was three times in Odisha. The difficulty in contemplating female sterilisation if a son had not been born (and also getting familial and spousal acceptance) was clearly expressed by respondents and was more explicit in Bihar than in Odisha. Unmet need, which identified the groups that the project could benefit quickly from targeting, was found to be lower among the higher wealth quintiles and higher in the rural areas (for both states). In Bihar, there were particularly low rates of demand realisation among young women, rural households and Muslims. In Odisha, there were similar contours, although Muslims had relatively high rates of demand realisation and it was other religious minorities and Scheduled Tribes who struggled to realise their demand. With regard to spacing and limiting methods, the unmet need was higher for both types of methods in Bihar as compared to Odisha. The unmet need for spacing was higher for women with no child or one child. Unmet need for limiting was higher than that for spacing after two children in both states. More on outcomes See Chapter 9 for a full description of the outcomes, including: A detailed analysis of the contraceptive prevalence rate for modern and traditional methods, and disaggregated by age of respondent, number and gender of children, and socio-economic background A description of the findings from the qualitative studies about respondents perceptions of various family planning concepts such as ideal family size and the sex of the child Calculations of unmet need, and an integrated analysis of CPR and unmet need 5.3 Findings: Assessing equity and inclusion Analysis of the mid-term survey data with regard to socio-economic equity and inclusion of underserved areas highlighted the following: Both SF Clinics and SM Outlets were found disproportionately in areas with a higher proportion of people from the highest wealth quintile, a lower proportion of people from the lowest wealth quintile, and a lower proportion of Scheduled Tribes. A majority of the villages and wards (88%) reported that the SM Outlet was more than 10 km away, and only 4% had an outlet within 5 km. Oxford Policy Management New Delhi 14

16 SF Clinics were found disproportionately in areas where existing levels of CPR through modern methods was higher than average, and use of traditional methods was lower than average in Bihar (this was statistically not significant in Odisha). SM Outlets were found disproportionately in areas with higher existing levels of overall and modern CPR in both states. Thus, the data suggests that by the time of the survey, Project Ujjwal was not yet able to focus on underserved areas. This makes intuitive sense, as private sector providers would be more likely to cluster in areas where the demand for products and services was higher and there was an ability to pay for private services. More on assessing equity and inclusion See Chapter 10.3 for an assessment of Project Ujjwal s focus on equity and inclusion, including: The testing of differences in social and wealth characteristics between SSUs that were within 10 km of the SF Clinic/SM Outlet and those that were more than 10 km away The testing of differences in CPR between SSUs that were within 10 km of the SF Clinic/SM Outlet and those that were more than 10 km away 6. Conclusions relevant to programme design The mid-term survey findings suggest that there has been reasonable operationalisation of Project Ujjwal at the time of the survey. With regard to assessing changes at the population level, the percentages of those surveyed who were aware of Project Ujjwal s activities or visited their clinics and outlets were found to be quite low. Since the various activities related to demand generation had only just begun taking off at the time of data collection, it may be the case that population impact will increase and findings at end line will allow a better assessment of impact. Further, given the population base of around 140 million including 30 million women of the reproductive age group across both states, it might be argued that small percentages would still translate into large absolute numbers. Even so, given the low percentages suggested by the current data and drawing on the findings of the analysis, certain measures are mentioned below that may help Project Ujjwal improve its impact in Bihar and Odisha. 1. Improving the supply chain: A regular and consistent supply of products to clinics (18% had received no Ujjwal products), outlets (about 50% had not received any Ujjwal products) and Ujjwal Saathis (less than 50% stocked any FP products) would increase confidence among providers in promoting Ujjwal products and services. 2. Increasing focus on training: The project needs to focus on refresher training for Ujjwal Saathis and SF Clinic staff alike to enable the project to bridge the existing knowledge gap, especially on appropriate methods to limit and space. In addition, the provision of training to SF Clinic staff, who currently obtain their information from informal sources, could improve their ability to counsel customers. 3. Targeting need to get impact: Ujjwal has a greater scope for impact in Bihar over Odisha as fewer couples in Bihar were able to realise demand for spacing, compared to Odisha. According to the survey data, the project could have its greatest impact on couples with one child/two children who need greater support to realise their demand for spacing methods. Oxford Policy Management New Delhi 15

17 4. Targeting messages: Men largely purchased contraceptive products such as OCPs and condoms. Women hesitated in approaching outlets and facilities to buy contraceptives, either because they did not travel out of the house or because they were hesitant to ask male outlet owners for these products. More women obtained contraceptives from female community health workers. To maximise take-up of products and encourage method mix, Ujjwal can have messages at outlets such as to target male users and support alternative local female sources for women. 5. Choosing the right medium: The most popular medium in disseminating information among women was the television, and among men it was through wall hoardings. Radio had negligible listenership in both Bihar and Odisha. Apart from taking note of the most popular mediums, the programme would need more than mass-media messages to encourage its target population of young couples to discuss family planning-related messages, as they took negligible action after viewing a message. Viewing family planning messages through mass media only encouraged older age groups to take any action. 6. Increasing awareness and changing attitudes: The project would do well to focus on increasing awareness about methods that respondents were less aware about (including ECPs, female condoms and LAM, and to a lesser extent IUCDs and injectables) methods for which respondents showed high levels of unawareness. For methods such as male condoms, male sterilisation, IUCDs and OCPs, a sizeable proportion said they were aware of the method but would not be willing to use it. For such methods, the focus should be more on changing attitudes. The state-disaggregated data showed that in general there needed to be more of a focus on awareness in Bihar and attitudes in Odisha. 7. Cementing relationships with the DQAC: Greater trust in private clinics, and knowledge of Ujjwal, would be required to forge a stronger PPP. At present, DQAC members interviewed had not heard of Ujjwal and were uncertain of the reliability of private clinics in general. The unwillingness to partner with private clinics was greater in Bihar than Odisha at the time of the study. 8. Increasing service to underserved areas: While involving the private sector in these areas is difficult (given the low density there), Ujjwal s activities linking with the public sector in underserved areas (for example, outreach activities, activities at public sector facilities, etc.) may need to be emphasised. Thus, as the programme continues to roll out in the states of Bihar and Odisha and expand its coverage, the implementers can attempt to ensure that activities reach the poorer and disadvantaged areas. Additionally, focussing the project s behavioural change activities (namely, Intervention Four) in the areas that showed high CPR through traditional methods would help a move to modern methods. 9. Long-term strategies: Long-term strategies which may be beyond the scope of Project Ujjwal in the short time left will be required to address some specific groups in both states. Given cultural mores, there is low responsiveness to using contraceptives after marriage to delay having children. At this stage, it appears that a strategy towards delaying the age of marriage may be more fruitful. Similarly, son preference and reluctance to use limiting methods if at least one son had not been born was high in both states, but much more so in Bihar. Under such circumstances, interventions towards gender equity and changing attitudes are required for acceptance of family planning methods and services. Oxford Policy Management New Delhi 16

18 10. State-specific suggestions: Given the differences in some findings across the two states, some state-specific suggestions are as follows: Data suggested that at the time of survey, implementation especially with regard to Interventions One and Two was lagging behind in Bihar (with lower number of Ujjwal Saathis attached to SF Clinics, lower organisation of FDS, lower sales at SF Outlets, and need for greater vigilance for malpractice). Implementation thus needs to be accelerated in Bihar. With regard to intervention Three, while fewer clinics in Odisha had QA visits compared to Odisha, the audits that were conducted were more comprehensive in Odisha. Thus while the scale of coverage with regard to QA needs to be increased in Odisha, it is the depth of assessment that needs to be improved in Bihar. The Ujjwal project staff can, in the time left, capitalise on FP products and services that have an edge in each state; for example, OCPs were comparatively more popular in Bihar, while the incidence of male sterilisation in Odisha at more than 13% was nine times that in Bihar. Besides, men in Odisha seemed responsive to the idea of family planning as use of withdrawal was also comparatively quite high, suggesting that a switch to modern methods like condoms and male sterilisation may be easier in Odisha. Strategies towards these may, therefore, be quite fruitful. Government services for undertaking abortions seem comparatively more popular in Odisha than in Bihar. Project Ujjwal could explore the facilities for safe abortions by government providers and if these are reasonably provided for in Odisha, the focus of the SF Clinics could be on provision of other FP services. On the other slide, the reasons for the lower use of abortion facilities in the private sector may need to be explored and addressed in the SF Clinics in Odisha. It is interesting that majority of SF Clinics in Odisha followed Government protocols while a majority of SF Clinics in Bihar followed Ujjwal protocols. The possibility of building further on the existing government protocols rather than altering to Ujjwal protocols could be explored in Odisha. According to the survey data, the project could have its greatest impact on couples with one child/two children who need greater support to realise their demand for spacing methods (across both states) - given that there is low demand (and low unmet need) in both states amongst couples with no children. For Ujjwal, however, there seems to be greater scope for impact in Bihar over Odisha as fewer couples in Bihar were able to realise demand for spacing compared to Odisha. There are minor differences between the groups showing relatively higher unmet need in the two states with these being rural households, younger women and Muslims in Bihar, and Scheduled Tribes and other religious minorities in Odisha. This may have implications in the design of activities to be undertaken by Ujjwal Saathis as well as the design of BCC and IPC, and geographic availability of goods and services planned by Project Ujjwal. Oxford Policy Management New Delhi 17

19 Table of contents Acknowledgements 1 Executive summary 2 Table of contents 18 List of tables 22 List of figures 24 How to read the tables and analyses 25 List of abbreviations 26 1 Introduction Background to Project Ujjwal Outline of Project Ujjwal Evaluation: Background and key questions Key users of the report Structure of the report 35 2 Theory of change Problem diagnosis Theory of change Data collection in line with the theory of change 40 3 Methodology Mixed-methods methodology Quantitative methodology Estimating impact Evaluation strategy Taking into account other related programmes Quantitative data collection Sampling strategy Power calculations and sample size Post data collection: Validity checks at entry, cleaning and analysis Qualitative methodology Qualitative sampling strategy Tools employed for qualitative data collection Data analysis Data-related limitations and mitigation strategies Ethical considerations Equity and inclusion Cost-effectiveness analysis and process evaluation 56 4 Intervention One: Social franchising 57 Oxford Policy Management New Delhi 18

20 4.1 Overview and TOC Findings FP products and service delivery offering: Stock Perceptions of Ujjwal Ancillary services Conclusions 68 5 Intervention Two: Social marketing Overview and TOC Findings Social Marketing Outlets Family planning products offering: Stock of products Overall stock maintained Changes in stock after Project Ujjwal Perceptions of Project Ujjwal Perceptions of benefits received Perceptions of challenges faced Perceptions of improvement in services Conclusions 78 6 Intervention Three: Capacity building and quality assurance Overview and TOC Findings Training and capacity building Training sessions Issues covered during training Increase in knowledge and understanding Abortion services PCPNDT Act Perceptions of training Quality assurance Guidelines and protocols Quality assurance (QA) visits Government involvement Conclusions and recommendations 90 7 Intervention Four: Demand generation Overview and TOC Findings Smart couples Smart providers Smart communities 96 Oxford Policy Management New Delhi 19

21 7.2.4 Smart environment Knowledge and attitudes Knowledge of FP products/services and RH concepts Attitudes towards family planning Intra-household decision making Conclusions Intermediate outcomes Practice of family planning Profiles of FP products being used currently Provider preferences public versus private Integrating knowledge, attitudes and practice Sales of FP products and services: at SF Clinics Family planning products: Sales at SM Outlets Overall product-specific quantity sold Exploring factors influencing quantity sold insights from the qualitative survey Preferences for source of contraceptives TOs versus NTOs CYPs generated CYPs generated at the SF Clinics and by Ujjwal Saathis CYPs generated by SM Outlets Abortion Conclusions Outcomes: CPR, unmet need and abortions Introduction Target CPR: Analysing modern versus traditional methods By age By number of live children By sex of children By socio-economic groups CPR composition: Method mix By number of live children CPR: Perceptions Unmet Need Putting CPR and unmet need together Newlyweds and the need to delay Conclusions Implementation status Introduction 154 Oxford Policy Management New Delhi 20

22 10.2 Status of the project interventions Beneficiary-level awareness of Ujjwal Beneficiary-level uptake of Ujjwal Conclusions: Putting it together Assessing equity and inclusion Testing the differences in social and wealth characteristics Testing differences in the CPR of treatment and control SSUs Conclusions Conclusions of relevance to programme design 172 References 175 Oxford Policy Management New Delhi 21

23 List of tables More on methodology and data collection 6 More on Intervention One 7 More on Intervention Two 8 More on Intervention Three 9 More on Intervention Four 11 More on intermediate outcomes 13 More on outcomes 14 More on assessing equity and inclusion 15 Table 1.1 Project Ujjwal key users of the report 34 Table 3.1 Sample size by location and level 44 Table 4.1 Transmission mechanism 59 Table 4.2 Social Franchise Clinics 61 Table 4.3 Ancillary services in SF Clinics 67 Table 4.4 Ujjwal Saathi s Role 68 Table 4.5 Ujjwal Saathis receipt of products 68 Table 5.1 SM Outlets: General details 73 Table 5.2 SM Outlets: Distance from SSU 74 Table 7.1 Actions taken after viewing FP messages on mass media 100 Table 8.1 Reasons for discontinuing an FP method 110 Table 8.2 Overall: Women and men: Profile of currently used FP methods 112 Table 8.3 Overall: Women and men: Price details of currently used FP methods 112 Table 8.4 CYPs generated by SF Clinics 124 Table 8.5 CYPs generated by Ujjwal Saathis 125 Table 8.6 CYPs generated by SM Outlets 126 Table 8.7 SF Clinics: Abortions and sex-determination services 127 Table 9.1 Annual Health Survey Table 9.2 Ujjwal Mid-Term Survey Table 9.3 Sex preferences 137 Table 9.4 Project Ujjwal data: CPR distribution by method for Bihar 139 Table 9.5 Project Ujjwal data: CPR distribution by method for Odisha 140 Table 9.6 Use of methods (%) 141 Table 9.7 Method-wise use by wealth quintile and type of contraceptive provider 142 Table 9.8 Method-wise use by wealth quintile and type of contraceptive provider 143 Table 9.9 Method-wise use by wealth quintile and type of contraceptive provider 144 Table 9.10 Method-wise use by wealth quintile and type of contraceptive provider 145 Table 10.1 SF Clinics: Stock and training 155 Table 10.2 SM Outlets: Family planning and reproductive healthcare training 155 Table 10.3 SF Clinics: Quality assurance and capacity building 156 Table 10.4 Ujjwal Saathis: Training and stock of FP products 157 Table 10.5 Population-level data: Awareness of SF Clinics and ancillary services 159 Table 10.6 Population-level data: Awareness of SM Outlets 160 Oxford Policy Management New Delhi 22

24 Table 10.7 Population-level data: Uptake of Ujjwal interventions 161 Table 10.8 Wald tests for SSUs within/beyond 10 km of SF Clinic 164 Table 10.9 SSU/Household: Wald tests for SSUs with and without an SM Outlet 166 Table SSU/Household: Wald tests for SSUs within/beyond 10 km of SM Outlet 167 Table CPR for women: Wald tests for SSUs within/beyond 10 km of SF Outlet 168 Table CPR for women: Wald tests for SSUs within/beyond 10 km of SM Outlet 169 Table CPR for women: Wald tests for SSUs with and without an SM Outlet 170 Oxford Policy Management New Delhi 23

25 List of figures Figure 1: Theory of change Project Ujjwal 4 Figure 2: Theory of change Project Ujjwal 39 Figure 3: Sampling strategy 47 Figure 4: Intervention One theory of change 57 Figure 5: FP services available in Ujjwal SF Clinics 62 Figure 6: FP products stocked in Ujjwal SF Clinics 63 Figure 7: Intervention Two theory of change 70 Figure 8: FP products stocked by SM Outlets 74 Figure 9: FP products stocked by TOs and NTOs 75 Figure 10: Receipt of FP products from Project Ujjwal and products stocked 76 Figure 11: Joining Ujjwal and FP products stocked 76 Figure 12 Intervention Three theory of change 79 Figure 13: Issues covered during training of staff at SF Clinics 82 Figure 14: Issues covered during training of Ujjwal Saathis 83 Figure 15: Intervention Four theory of change 92 Figure 16: Access to mass media 98 Figure 17: Sources of information about family planning (for women) 99 Figure 18: Sources of information about family planning (for men) 100 Figure 19: Awareness of FP products and services (Bihar) 104 Figure 20: Awareness of FP products and services (Odisha) 104 Figure 21: Willingness to use FP methods (Bihar) 106 Figure 22: Willingness to use FP methods (Odisha) 106 Figure 23: Reasons for not willing to use female FP methods 107 Figure 24: Reasons for not willing to use male FP methods 107 Figure 25: Perceived acceptance of FP methods 108 Figure 26: Awareness, attitudes and use with regard to family planning 116 Figure 27: Quantity of services provided by SF Clinics 117 Figure 28: Quantity of products sold by SF Clinics 119 Figure 29: Average quantity of FP sold stocked by SM Outlets 120 Figure 30: Place of last abortion 128 Figure 31: Contraceptive prevalence rate 136 Figure 32: CPR by number of live births 136 Figure 33: CPR disaggregation by modern methods 138 Figure 34: CPR disaggregation by traditional methods 138 Figure 35: CPR composition: Method mix 139 Figure 36: Limiting and spacing methods by number of children 140 Figure 37: Comparing ideal family size and actual number of children 146 Figure 38: Total unmet need (disaggregated) 148 Figure 39: Unmet need by number of children 149 Figure 40: Current use and unmet need 150 Figure 41: Proportion of demand realised 150 Figure 42: Causal process of Project Ujjwal 154 Figure 43: Funnel of attrition for Project Ujjwal (women) 162 Figure 44: Funnel of attrition for Project Ujjwal (men) 162 Oxford Policy Management New Delhi 24

26 How to read the tables and analyses Statistics and analyses The descriptive statistics presented in the tables in this report are, unless mentioned otherwise, percentages rounded off (up if five or more and down if less than five) to one decimal point. When these statistics are referred to in the text, then they are rounded off to the nearest whole number (up if 0.5 or more and down if less than 0.5). Also, if a statistic is between 0 and 1 (for example, 0.4 or 0.5), then one decimal point is retained in the text as well. In cases where the indicator is not a percentage (for example, if a mean value is presented), the unit of measurement has been specified. The descriptive statistics presented in most of the tables are a simple disaggregation by categories of interest (for example, age groups and wealth quintiles) pertaining to the same analytical group of observations. A two-sample test of the equality of the means is not necessary for the summary statistics of different groups belonging to the same population that are presented in the descriptive tables. In Chapter 10, however, a comparison of two different groups (samples) rather than a disaggregation within the same group is shown. Quasi-treatment and quasi-control groups (distance less than and more than 10 km, respectively, from the Ujjwal network sites) are compared to assess the equity and inclusion of the interventions. The analysis here is meant to show us not only whether observable means for the two groups are different, but also whether the difference is statistically significant. For such tables, therefore, details on whether the difference is statistically significant based on the results of a Wald test have been provided. n values In all tables of the report, the n value is presented in the cell immediately to the right of each statistic. This indicates the un-weighted number of observations in the sample on which that indicator is based. This gives an indication of how certain we can be about the estimate in question. The more respondents that answer a question, the greater is the reliability of the sample mean as a reflection of the population mean. Standard errors For indicators represented by mean values, the standard error is presented in the cell immediately below the statistic, in parentheses. The standard error gives us an idea of how close or far the sample mean (derived from the sample) is from the population mean (the population from which the sample is derived). This gives us an indication of how representative the value of the mean reported for our sample is. Sampling weights In order to obtain estimates of key indicators that were representative for Project Ujjwal intervention areas, the units of the survey data (HH, men and women) were analysed using sampling weights. Therefore, in the tables, the estimates are weighted averages. The implication of this is: the result of an un-weighted average would be different from the result obtained through a weighted average. Oxford Policy Management New Delhi 25

27 List of abbreviations ACMO ADMO AHS AIDS ANM APL ASHA AWW BBC BCC BMGF BPL CAPI CEA CEB CPR CYP D&C DEFT DFID DHS DLHS DPM DQAC EAG ECP Additional chief medical officer Additional district medical officer Annual health survey Acquired immune deficiency syndrome Auxiliary nurse midwife Above poverty line Accredited social health activist Anganwadi worker British Broadcasting Corporation Behavioural change communication Bill and Melinda Gates Foundation Below poverty line Computer-assisted personal interviewing Cost-effectiveness analysis Census enumeration block Contraceptive prevalence rate Couple years of protection Dilation and curettage Design effect Department for International Development Demographic and Health Surveys District-Level Household and Facility Survey District programme manager District Quality Assurance Committee Empowered Action Group Emergency contraceptive pill Oxford Policy Management New Delhi 26

28 EE FDS FGD FP FP/RH FOGSI FSU GPS GOI HH HIV HLFPPT ICC ICT ID IDI IEC ITC IMR IPC IUCD IUD JHUCCP JHPIEGO JSK KAP Entertainment education Fixed day services Focus group discussion Family planning Family planning/reproductive health Federation of Obstetric and Gynaecological Societies of India First-stage sampling units Global positioning system Government of India Household Human immunodeficiency virus Hindustan Latex Family Planning Promotion Trust Intra-cluster correlation Information communication technology Identification In-depth interview Information education and communication Inter-temporal correlation Infant mortality rate Interpersonal communication Intrauterine contraceptive device Intrauterine device Johns Hopkins University Center for Communication Programs Johns Hopkins Program for International Education in Gynaecology and Obstetrics Jansankhya Sthirata Kosh Knowledge, attitudes and practices Oxford Policy Management New Delhi 27

29 KII LAM MDE MKBKSH MMA MMR MRP MSI MTP NFHS NGO NHM NRHM NTO OBC OCP OPM PAPI PCPNDT Act PFI PHC PIP PPIUCD PPP PPS PRACHAR QA Key-informant interview Lactational amenorrhoea method Minimum detectable effect Main Kuch Bhi Kar Sakti Hoon Medical methods of abortion Maternal mortality ratio Maximum retail price Marie Stopes International Medical termination of pregnancy National Family Health Survey Non-governmental organization National Health Mission National Rural Health Mission Non-traditional outlet Other backward classes Oral contraceptive pill Oxford Policy Management Paper-and-pencil interviewing Pre-Conception and Pre-Natal Diagnostic Techniques Act Population Foundation of India Primary health centre Programme implementation plan Post-partum intrauterine contraceptive device Public private partnership Probability proportional to size Promoting Change in Reproductive Behaviour Quality assurance Oxford Policy Management New Delhi 28

30 QAC QC RH RMP RQAC SC SF SHG SM SQAC SRS SSU ST STI TAST TMST TO TOC TSU VHND UNFPA Quality assurance committee Quality control Reproductive health Rural medical practitioners Regional Quality Assurance Committee Scheduled Caste Social Franchise Self-help group Social Marketing State Quality Assurance Committee Sample registration system Secondary sampling unit Scheduled Tribe Sexually transmitted infections Technical Assistance and Support Team Technical and Management Support Traditional outlet Theory of change Third-stage sampling unit Village Health & Nutrition Day United Nations Population Fund Oxford Policy Management New Delhi 29

31 1 Introduction This report presents the findings from the analysis of data collected following the beginning of implementation of Project Ujjwal. This data is to be used to a) report on the status of implementation, and b) provide a base against which the impact of the project (between the collection of this data and the end line when the second survey is undertaken) is assessed. In the current chapter, Section 1.1 provides a background to Project Ujjwal and Section 1.2 highlights key aspects of the project. In Section 1.3 the background to and overview of the evaluation, which this report is part of, are provided, followed by a section on key users of the report (Section 1.4). Section 1.5 provides the structure of the rest of the report. 1.1 Background to Project Ujjwal In December 2010, the Department for International Development (DFID) published its Global Framework for Results for improving reproductive, maternal and newborn health in the developing world, and set out the following objectives to be achieved by : Save the lives of at least 50,000 women during pregnancy and childbirth and 250,000 newborn babies by 2015 Enable at least 10 million more women to use modern methods of family planning by 2015, contributing to a wider global goal of 100 million new users Prevent more than five million unintended pregnancies Support at least two million safe deliveries, ensuring long-lasting improvements in quality maternity services, particularly for the poorest India is one of DFID s priority countries for achieving these results, given the magnitude of potential gains. To support these objectives, DFID India has committed in its operational plan that an additional 3.4 million women will use modern family planning methods, and 4,400 fewer mothers will die during pregnancy and delivery, to be achieved through its programmes in the three low-income states of Madhya Pradesh (MP), Odisha and Bihar. A DFID-supported multi-country programme, Preventing Maternal Death through Unwanted Pregnancy, is already being implemented in Madhya Pradesh. Project Ujjwal is a new initiative in Bihar and Odisha, and as in MP, is supporting the expanded provision of family planning and reproductive health services through private and nongovernmental channels, as well as attempting to foster stronger interaction between public and private health sectors. Such approaches involving social franchise/social marketing in other parts of the country include: Project PEHEL of Population Services International (PSI) in the states of Uttar Pradesh (UP), Rajasthan and Delhi; Project Matrika of Pathfinder International and World Health Partners in Uttar Pradesh; and the Government of India s own social marketing of family planning products that began as far back as 1968 and is now being implemented through HLFPPT (the not-for-profit organisation set up by HLL Lifecare Limited). Currently, a number of projects using a range of approaches towards family planning and reproductive health are being implemented in Bihar and Odisha. Their presence would have to be taken into account during the evaluation of Project Ujjwal. Some of these projects are: 3 DFID (2010) 30

32 DKT India s 4 flagship social franchising programme, Surya, operated by Janani in Bihar, Jharkhand and Uttar Pradesh. Surya, considered a classic social franchising model, provides family planning products and services through DKT franchise clinics as well as rural medical practitioners. DKT also partners with the government to strengthen service delivery at public health facilities Ananya in Bihar funded by the Bill and Melinda Gates Foundation (BMGF) including family planning as one of nine focus areas Investment in the revitalisation of post-partum family planning services after institutional deliveries in public facilities by Johns Hopkins Program for International Education in Gynaecology and Obstetrics (JHPIEGO) in both Bihar and Odisha, funded by the Norway India Partnership Initiative (which is being separately evaluated by OPM) Pathfinder International s Promoting Change in Reproductive Behaviour (PRACHAR) programme in Bihar The national-level Adarsh Dampati Yojana (Ideal Couple Scheme) being implemented by the Bihar state government to promote delaying and spacing (by providing incentives to couples who have adopted planned-parenthood behaviours) The state FP cell in Odisha created under the Directorate of Family Welfare (and technically supported by the United Nations Population Fund) to implement all FP activities under the National Rural Health Mission (NRHM). It invests in demand generation and awareness generation for FP through Kalyani and youth clubs. The state has also instituted performancebased awards for districts and providers to motivate them to reinstate focus on FP 1.2 Outline of Project Ujjwal Objective: The objective of Project Ujjwal is to improve contraceptive prevalence and prevent unintended pregnancies, especially among young, low parity, poor and socially excluded women. The project proposes a total marketing approach involving the private sector (for example, social marketing and social franchising) as well as public private partnerships. Targets: As per the revised Implementation Plan (March 2013) 5 of Project Ujjwal, it targets to: Reach 780,000 new users through various interventions for family planning services 6 Reach 300,000 women for safe abortion services Strengthen 18,000 Social Marketing Outlets and 280 Social Franchise Clinics so as to generate around three million couple years of protection (CYPs 7 ) 8 Contribute to increasing the contraceptive prevalence rate by 10 percentage points from the levels in 2012, in both Bihar (increase from 32% to 42%) and Odisha (increase from 38% to 48%) by A private provider of family planning and reproductive health products and services in the developing world 5 DFID (2013) 6 This was altered in the updated November 2014 log frame to 810,000 new users to account for an extension of the project from March 2015 to Please see Annex A for a definition of the term CYP. 8 The additional number of CYPs to be generated by the end of the project in 2016, was changed in the updated log frame to 3.5 million. 9 The figures for 2012 were estimated as projections using DLHS ( ) data, assuming a continuation of the same trajectory. The log frame was later updated and figures from AHS ( ) were used for the base year of 2012 (37% 31

33 Avert 1,700 maternal deaths and 37,000 infant deaths in the project period and beyond. Interventions: As detailed in the log frame, the project is aiming to achieve these outcomes and impacts through four main project outputs /interventions: Social franchise: Increased choice of sites providing quality clinical family planning/reproductive health (FP/RH) services with a focus on clinic-based services in rural areas (with the creation of 280 Social Franchise Outlets) 10 Social marketing: Increased choice of social marketing sites providing quality FP/RH products, with a focus on rural areas (with the creation of 18,000 Social Marketing Outlets) 11 Capacity building and training 12 : Building FP/RH capacity of providers, providing training and mentoring support, and facilitating improved implementation of the Pre-Conception and Pre- Natal Diagnostic Techniques (PCPNDT) Act Demand generation: Generating demand, overcoming barriers to FP uptake, and addressing gender norms through communication and community outreach (through Ujjwal Saathis trained field-level workers attached to the franchised clinics) for delaying and spacing births As mentioned above, the project log frame refers to four project outputs : a) social franchise, b) social marketing, c) capacity building and training, and d) demand generation. The term interventions has been used in this report to refer to these outputs. This is to prevent confusion with the term outputs in the theory of change (TOC), which refers to the interim stage between inputs and outcomes. Also, please note that while the term intervention refers to the broad strategy for example, social franchise the term inputs used in the TOC details the actual elements comprising the intervention for example, setting up Social Franchise Clinics, having minicamps, etc. Implementation partners: Project Ujjwal is being implemented by a consortium led by the Futures Group and including the Hindustan Latex Family Planning Promotion Trust (HLFPPT), the Public Health Foundation of India (PHFI) and the Johns Hopkins University Center for Communication Programs (JHUCCP). Futures Group as the lead partner provides overall programme management and assists state governments in partnering with the private sector for improving access to healthcare. HLFPPT makes available, through their distribution network, socially marketed brands of contraceptives at subsidised prices. PHFI is undertaking the training as well as leading the quality assurance. JHUCCP is leading the strategic behavioural change communication (BCC) and community outreach. Timescale: Project Ujjwal began in April 2013 and was initially for two years, but it has now been extended until March Data collection for an evaluation of the project was planned at baseline prior to the start of the project, but this did not happen. Data was eventually collected between May and December Data collection for the end term report is planned for for Bihar and 46% for Odisha). In the updated log frame, the CYP targets were also changed to 43% in Bihar and 52% in Odisha by the end of the project in These are to be pre-existing clinics that are made part of the Ujjwal franchise (that is, empanelled or made part of Ujjwal, on agreement to certain conditions). 11 For social marketing, some outlets were to be pre-existing outlets already selling family planning products (for example, pharmacies) and made part of Ujjwal s social marketing network. Some additional outlets that were hitherto not selling such products (that is, non-traditional outlets like bindi and paan shops) were also to be included. 12 The staffing complement of a clinic is the sum total of the clinic s full-time and part-time staff. 32

34 2013 April: Project Ujjwal begins 2014 May to December: Mid-term survey 2016 March: Project ends. End-term survey is scheduled Geographic focus: The project is focused on two states: Odisha and Bihar (demographic details and details of specific family planning-related indicators for the states are given in Annex Q, available online). Initially, the idea was to focus the two main interventions (social franchising and social marketing) on the districts with low performance in uptake of family planning and reproductive health products and services. However, during the course of implementation, this strategy was changed and all interventions are now being implemented across all districts in both states. 1.3 Evaluation: Background and key questions OPM was included in the consortium as the evaluation (including cost-effectiveness analysis [CEA]) partner, with no role in the implementation of the project. DFID wanted further assurance of independence and therefore set up an independent Evaluation Reference group. It was agreed that OPM would report to DFID (who have responsibility for approving deliverables) and the independent Evaluation Reference Group and not to the implementing consortium. Therefore, the current arrangement whereby OPM undertakes the IE and the CEA, and the main contractor does the project monitoring and implementation oversight was decided upon. There is thus a firm Chinese wall. The current arrangement which ensures independence was presented in the Inception report approved by SEQAS. Given that the project began in March 2013 and data collection started in May 2014, this report is not a baseline report but is referred to as a mid-term survey report. It will serve as the basis against which data collected at the end line of the project will be compared. While implementation had not been completed by the time data collection was undertaken, and it is also expected that interventions would take some time to begin having an impact at the population level, it is a limitation of the evaluation that it will not be able to capture the full impact of the project. It does, however, offer the opportunity to provide some feedback that may be of relevance during the rest of the implementation and contribute towards helping the project achieve its targets. As per the Terms of Reference (see Annex B and also as mentioned in the Inception Report), the key questions to be addressed in the evaluation of Project Ujjwal are as follows: What is the attributable impact of the interventions on family planning uptake and outcomes? This includes: o To what extent do the supply-side interventions (280 Social Franchise Clinics, 18,000 Social Marketing Outlets, and some other smaller-scale PPP approaches) increase the uptake of family planning services and products, and to what extent is this uptake a net increase of new users or a substitution by existing users? To what extent does this change the method mix as well as overall uptake? How does this differ for the three types of family planning needs (delay, limiting and spacing)? How does it affect continuation rates? To what extent does it change client satisfaction rates? o To what extent does the project work increase access to and uptake of safe abortion services? 33

35 o o o To what extent do the behavioural change communication (BCC) activities (both IPC and mass media) impact on individual and household decision making and behaviour? How does the transmission mechanism work? What is the impact of Project Ujjwal as a package, as individual interventions and as combinations of interventions? How does this impact differ for different groups, especially hard-to-reach groups (e.g. based on age, social exclusion status, geographical isolation, poverty status), the poorest 40%, young girls (15 19 years) and young couples (19 24), as they are part of DFID s global FP evaluation priorities? What is the cost-effectiveness of these interventions? Should the governments of Bihar and Odisha consider continuing or scaling up the intervention and, if so, in what format? In addition compliance with OECD DAC criteria is also to be explored. The mid-term survey data was collected in 2014, as detailed later in the report. The data for the end line will be collected in 2016 such that the collection and analysis are undertaken a few months after Ujjwal s support for its field interventions ends. This should allow for an assessment of the impact as well as sustainability of the interventions. 1.4 Key users of the report This report serves as the main reference for the study design and the methods being undertaken towards the evaluation, as also the point of comparison for the data to be collected at the end line. It is expected that the report will also be useful to various key stakeholders, as delineated in Table 1.1. Table 1.1 Project Ujjwal key users of the report Key stakeholder Purpose Key targets Implementation team Government Providing inputs into the design of the programme as it moves ahead The data provides an input to the state governments of Bihar and Odisha on the current status in the respective states with regard to family planning, as well as wider inputs to state and central governments especially with regard to the demand-side factors, which could influence the design of programmes All consortium members Ministry and Departments of Health and Family Welfare National and state health systems/resource centres National Health Mission secretariat and state health societies Other national stakeholders, international research community and implementing agencies Evidence sharing for DFID and other stakeholders including civil society; providing a status report on the current situation in the two states Public, private and nongovernment implementers of family planning services and products, including umbrella organisations like FOGSCI DFID and other donor partners who work in this area 34

36 Table 1.1 Project Ujjwal key users of the report Key stakeholder Purpose Key targets 1.5 Structure of the report (especially the United States Agency for International Development [USAID] and the BMGF) Knowledge hubs including the World Bank, the Center for Global Development (CGD) and the International Initiative for Impact Evaluation (3ie) The rest of the report is structured as follows: In Chapter 2, the theory of change of Project Ujjwal is analysed, including a review of the underlying problem diagnosis. Chapter 3 presents the evaluation approach and elaborates on the mixed-methods methodology. Chapters 4, 5, 6 and 7 present findings related to the TOC of each of the four project interventions up to the outputs level (as the inputs and outputs are specific to each intervention). This is followed by two chapters presenting the findings for the intermediate outcomes (Chapter 8) and outcomes (Chapter 9), which are common to all interventions. Chapter 10 presents reflections on Project Ujjwal s focus on equity and inclusion based on the data findings. Chapter 11 concludes. To enable a comparison with data at the end line for each state separately (given the sociodemographic differences as well as the different starting points of the contraceptive prevalence rate [CPR] and unmet need in Bihar and Odisha), data has been analysed and results presented statewise as well as overall, through the entire report. The annexures of this report include: Annex A: Definition of key indicators Annex B: Terms of Reference Annex C: Evaluation framework Annex D: Detailed quantitative evaluation methodology Annex E: Detailed qualitative evaluation methodology Annex F: Profiles of the sampled women and men Annex G: Intervention One: Social Franchise Clinic profile Annex H: Intervention Two: Social Marketing Outlet profile Annex I: Intervention Three: Capacity building and quality assurance Annex J: Intervention Four: Mass media and IPC Annex K: Intermediate outcome indicators Annex L: Outcome indicators Annex M: Triangulation with AHS data Annex N: Data: Ownership and storage Annex O: Bibliography The online annexures include: Annex P: Cost-effectiveness analysis 35

37 Annex Q: State profiles Annex R: Sampled SSU profile Annex S: Household profile Annex T: Intervention One: Social Franchise Clinic profile (Part II) Annex U: Intervention Two: Social Marketing Outlet profile (Part II) Annex V: Intervention Three: Capacity building and quality assurance (Part II) Annex W: Intervention Four: Mass media and IPC (Part II) Annex X: Process evaluation 36

38 2 Theory of change Project Ujjwal comprises interventions directed towards bringing about improvements in CPR and unmet need. The status of these FP indicators in Bihar and Odisha and the possible factors underlying unsatisfactory levels of indicators that Project Ujjwal s interventions are directed towards are discussed first. This is followed by a discussion of the theory of change underlying the interventions. 2.1 Problem diagnosis The Business Case 13 for Project Ujjwal draws on existing information to make a case for the project as well as guide the development of the particular interventions. Secondary figures for family planning-related indicators are presented in Table Q.2 in Annex Q (online). The table shows that Bihar and Odisha are lagging behind the average levels for India with regard to values for CPR and unmet need (District-Level Household and Facility Survey [DLHS-3] for ). CPR for Bihar and Odisha are 32% and 47%, respectively, compared to the all-india average of 54%. The same data shows that these values are even lower when the assessment is restricted to modern contraceptive methods. While looking at the mix of the methods used, female sterilisation is the most prevalent method in both the states (25% in Bihar and 26% in Odisha), suggesting low demand for spacing methods. The unmet need for family planning is high in both the states, whether it is total unmet need or that for spacing or limiting methods. According to DLHS 3, the unmet need for contraceptives in Bihar is 37%, and in Odisha it is 24%. Further, the Business Case identifies unmet need as being the greatest in the years age group, in the less educated and poorest households. The low CPR and high rate of unmet need have been approached by Project Ujjwal as: a) a supplyside issue, with the project addressing possible gaps in family planning products and services (range, quantity, quality and price); and b) a demand-side issue, with the project addressing possible lack of awareness and unfavourable attitudes towards family planning. On the supply side, efforts are being made through the National Health Mission to improve family planning and reproductive health services. According to the State Programme Implementation Plans (State PIPs, ), with only 55% doctors in place in Bihar and 65% in Odisha, the biggest challenge in the public sector is the availability and continuation of doctors. Besides, for accessing limiting methods (that is, female sterilisation), most women first come in contact with the public sector only after having given birth and usually after several births (although this is changing with higher institutional deliveries). Thus, younger women who could benefit greatly from spacing and women with unwanted pregnancies looking for safe abortion services are often neglected. Project Ujjwal proposes drawing on the private sector s services to address the needs of this vulnerable group. Given the challenges faced, especially in reaching young, low-parity women for provision of spacing methods and safe abortion services from the public sector, Project Ujjwal suggests complementing the public sector with the private sector. A total market approach is adopted wherein the use of family planning products and services within each wealth quintile is to be increased. It is expected that those who can afford will use the social franchise facilities; by accrediting franchise facilities into government schemes, the poorest will also be able to access government subsidies and benefits here. 13 DFID (2012) 37

39 This is to be accompanied by interventions on the demand side to address existing misconceptions, increase knowledge, and change attitudes around family planning and other reproductive health issues. Although this is anticipated to be challenging, Project Ujjwal attempts to tackle social and cultural barriers surrounding the demand for family planning. As mentioned in the Business Case, it is expected that communication and social mobilisation campaigns will be used to expand demand for services. Such initiatives are expected to especially trigger the uptake of spacing methods (especially intrauterine devices [IUDs] and injectables) and safe, legal abortion care. 2.2 Theory of change A theory of change (TOC) for Project Ujjwal, drawing on the above problem diagnosis, has been retrospectively constructed in the course of developing the methodology for the evaluation. It draws on the broad TOC in the Business Case and on the log frame in the Implementation Plan. Comments and inputs into this have also been obtained from the implementing agency. As shown in Figure 2, Project Ujjwal has four interventions: three directed towards addressing the supply side and one the demand side. The supply-side interventions include: Setting up of a social franchise for provision of family planning products and services (Ujjwal Clinics) Social marketing of family planning products (Ujjwal Outlets) Provision of training to Ujjwal clinic staff and the implementation of quality standards by Ujjwal providers It is expected that taken together the interventions will result in the outputs of improved supply (availability, range, price and quality) of family planning products and services. It has been assumed that private sector clinics would be interested in becoming part of the franchise. Similarly, it has been assumed that traditional (for example, pharmacists) and non-traditional (for example, paan shops) outlets would be interested in undertaking social marketing. The demand-side interventions include: Supplier-led demand creation (through health workers) leading to increase in the awareness of target-group customers (especially young, low-parity women) towards the improvements in supply Customer-led demand creation through the use of mass media as well as community outreach and interpersonal communication to increase the knowledge of the target audience, as also to inform the broader community about family planning concepts, products and services. This will also address attitudes and gender norms that form barriers to family planning. It has been assumed that the messages will reach the targeted audience of young, low-parity women in particular, especially in the context of using newer spacing methods. Initially the focus of the interventions was to be on selected underserved districts, but it was later extended to cover all districts in both states. The focus, however, remains on rural areas. 38

40 Figure 2: Theory of change Project Ujjwal 39