THIRD PARTY EVALUATION OF NACP PROJECTS

Size: px

Start display at page:

Download "THIRD PARTY EVALUATION OF NACP PROJECTS"

Charity Little
5 years ago
Views:

1 THIRD PARTY EVALUATION OF NACP PROJECTS EXECUTIVE SUMMARY Introduction 1. SoSec consulting services carried out process evaluation under a contract from the National AIDS Control Program, Pakistan. The evaluation parameters covered the assessment of: (i) preventive package of services provided to three high-risk population groups at five sites i.e. female sex workers in cities of Lahore and Karachi, men having sex with men in Lahore and jail inmates at Karachi, Hyderabad and Sukkur; (ii) quality of services for sexually transmitted infections (STIs), on the basis of syndromic management approach, in randomly selected 57 public sector hospitals in provinces and special areas; and (iii) screening of blood and blood products against HIV including quality assurance standards from a ample of 63 randomly selected public and private sector blood banks from provinces and special areas. SoSec consulting services, in consultation with relevant stakeholders, developed comprehensive research tools to undertake the evaluation. The field researchers administered assessment tools during August October, 2005 and data collection was completed over a period of 10-weeks. The present evaluation is Round-1 in the series of five annual evaluations. I PREVENTIVE SERVICES FOR HIGH-RISK POPULATION GROUPS 1.1 Findings good news 2. Consumer satisfaction. Consumers have posed their confidence on the services they received from the NGO managed projects. The exit interviews revealed that 14% clients from the high-risk population were very satisfied and the remaining satisfied with the services they received from the service providers for the treatment of STIs. They described the provider attitude as passionate, all prescribed medicines and condoms were provided along with instructions for use of medicines and repeat visit if not cured, and gave education on how STIs spread. However, advice on the treatment of partner/s was less regular. 3. Focus group discussions. The participants of FGDs appreciated the benefits and value of the services they received through NGO projects. The major highlights included: (i) training of some members as peer educators: (ii) services for the treatment of STIs including general ailments; (iii) provision of free of charge condoms; and (vi) referral for treatment and voluntary counselling and testing services. However, FGD participant explained that there are challenges to condom use by the clients, but they prefer to use condoms or occasionally insist or refuse nonusers. 4. Enabling environment. At all project sites reasonable efforts have been made in creating enabling environment, to implement project activities, by organizing advocacy sessions with a variety of local organizations. 5. Counselling. Counsellors had good knowledge of preventive measures that could protect high-risk groups from contracting STIs including HIV and field researchers were satisfied with the overall counselling approach. 6. Generally, the clinics were reasonably equipped with necessary supplies for managing STI patients. Further, privacy arrangements for patient s examination were found highly satisfactory at all the NGO clinics visited. 1.2 Findings challenges i

2 7. The implementing NGOs, within the context of local culture, are on the learning cure. Implementation arrangements are plagued with a series of challenges that have a direct bearing on the quality of services. The main challenges are summarized below. 8. Monitoring system. Each project has developed a recording and reporting system based on its understanding. Further, none of the projects have so far developed a monitoring system to track the quality and efficiency of the services in line with the project objectives. 9. Training material. Besides communication skills, peer education s curricular material for FSWs project in Karachi and MSM project in Lahore was thin in terms of topics. Similarly, NGO projects are not using uniform material for VCT training in spite of the fact that NACP has already developed a pre-test manual for VCT services. 10. Revised mapping of high-risk groups in the project sites is over due which should include: (i) the new high-risk population sites identified; (ii) registered FSWs or MSM who have left the project area; (iii) new FSWs or MSM who have settled in the project area; and (iv) information about street-based FSWs. 11. Data analysis and its use in planning. The services statistics were generally collected but not consolidated and analyzed in terms of distribution of STI patients by age, diagnoses, repeat visits, non-response to treatment, linking with lab. tests, re-infections, and use of patients data for procurement of STI drugs. 12. Waste management standard. In none of the NGO-clinics SOP for waste management was available. The NGOs have not put in place any system for disinfecting hazardous and infectious waste materials. The method of final disposal was dumping in the municipal waste, but burial through landfill by the clinic for jail inmates. 13. Essential drugs and supplies for managing STI patients. None of the NGO-clinics have provided STI drugs in line with the National STD Management Guidelines. In summary, 3-4 antibiotics were not supplied at the clinics, besides some stock-outs. 14. Management of STIs. The service providers in the NGO clinics were not properly skilled in the management of STIs. Further, the NGOs were not training and coordinating with the private sector health service providers in their project areas for institutionalizing the management of STIs on syndromic approach. 15. Services by Peer Educators. The knowledge and skills of peer educators was less than desirable in many areas. Their strengths included the knowledge in areas like: (i) ways by which HIV infection spreads and how spread of infection can be prevented; (ii) safer methods of sex; and (iii) persons at higher risk of contracting HIV infection. 16. Peer educators are expected to offer 7-8 type of services to their peers, while most of them were trained in 3-4 topics except those working for FSWs in Karachi. Further, only 1/3 rd interviewed peer educators identified them as the source for their peers to obtain condoms. Peer educators from FSWs project, Lahore were less familiar with STI symptoms reflection of their skills to identify potential STI patients for referral. NGOs have not formalized arrangements with peer educators for the referral of suspected STI cases to health facilities other than the clinic/s run by NGOs themselves. This gap was due to the fact that NGOs had not mapped health facilities in their project areas for training the GPs in the management of STIs on syndromic approach and then keeping an ongoing liaison with them. Peer educators don t have a standard format for the maintenance of daily logbook of their field activities. ii

3 17. Peer educators faced the following major challenges: (i) police raids sometime created embarrassing situation; (ii) Late night work e.g. following street girls in FSWs project, Karachi; (iii) daily working with peers; and (iv) issue related to remuneration. 1.3 Way forward. 18. Possible options for furthering the quality of and access to services need to be explored. Some of the options for consideration are summarized below: Organization of stakeholders workshop to review the evaluation report and develop a detailed plan of action for addressing the challenges. Implementing NGOs to jointly organize review meetings at agreed intervals to share best practices and challenges as part of ongoing learning. Develop a monitoring system including the software that responds to measure the project objectives over time. Work back from the monitoring system to develop reporting and recording system that becomes the basis of the quarterly monitoring reports. Adapt existing training materials and manuals for quality training and skill development of staff and support each other in skill development activities. II STI SERVICES AT PUBLIC SECTOR HOSPITALS 2.1 Findings 19. Counselling of STI patients is not well organized, generally because of lack of counselling skills of staff providing STI services, absence of a qualified counsellor, and the volume of outpatients in the clinics other than those having STI symptoms. 20. The referral of suspected HIV/AIDS cases to VCT centres was more regularly practiced in Sindh province followed by NWFP and not-at-all in Balochistan. 21. Record keeping of STI statistics. Recording syndromic diagnosis, and age and occupation of the patient on the outpatient register was not a common practice at the outpatient clinics. Collection of statistics of STI patients, analysis of data and its use for the procurement of STI drugs was also uncommon. Only seven clinics stated consolidating data of STI patients, five formally analyzed patients data and four clinics in Sindh used the analyzed information to guide the procurement of STI drugs. 22. Essential drugs for the treatment of STIs. Field researchers could not assess stock-outs of STI drugs as inventory of drugs was kept in the main pharmacy of the hospitals as drugs were purchased for the hospital patients as a whole and not separately for the outpatient clinics providing services to the STI patients. 23. Essential supplies related to STI services. Sixty percent outpatient clinics were well equipped with necessary supplies. Essential supplies like gloves, hand lens, condoms and educational material were less commonly available. 24. Privacy for the patients examination. Thirty five (35) percent of outpatient clinics had provided privacy for the examination of STI patients, mostly in the teaching hospitals. 25. Training. Twelve or 21% of doctors working in the sample clinics confirmed that they had been trained in the management of STIs on syndromic approach. The most recently trained doctor received training two years back. iii

4 26. Management protocols. The STIs management protocols, algorithms or charts at sample outpatient clinics were generally not available. 27. Knowledge of STI diseases. Respondent doctors working in the outpatients clinics had good idea of common STI diseases. The respondent doctors from Punjab province and AJK/FANA had good knowledge of common STI symptoms while those from Balochistan province had inadequate knowledge. 28. The awareness among prescribing doctors of the advantages of syndromic management approach for the treatment of STIs was limited, especially in the provinces of Balochistan and NWFP. 29. Overall diagnostic and case management skills of the respondent doctors on syndromic approach were quite inadequate and this observation is in line with the extent of training provided to the doctors on syndromic management of STIs. Doctors from the province of Punjab and AJK/FANA had much better diagnostic and management skills than the other provinces. 30. Waste management standards. Segregation of waste generated at STI outpatient clinics was an uncommon practice. In none of the hospitals, SOP for waste management was available. Thirteen hospitals (23%) were disinfecting hazardous and infectious materials as part of the overall hospital waste management system and did not specifically relate to STI outpatient clinic. 31. Consumer satisfaction. 33% of the clients were either very satisfied or satisfied with the quality of services they received. 32. Composite ranking score. The overall quality of services was found highly satisfactory at two clinics of teaching hospitals and satisfactory at eight clinics - four teaching hospitals and four district hospitals. The quality of services was found lowest at the clinics assessed in NWFP and Balochistan. 2.2 Way Forward 33. Options to enhance access and quality of STI services at public sector hospitals are presented below for the consideration of national and provincial AIDS Control Programs: Negotiation with provincial and district governments to designate STI outpatient clinic/s at each district or tertiary care hospital on the pattern of Sindh province and that adopt a policy that all other outpatient clinics in respective hospital refer suspected STI cases to the designated STI clinic/s. Institutionalize in-service training program/s for doctors and paramedics working at STI outpatient clinics in the management of STIs on syndromic approach and in counselling techniques along with organization of regular follow-up courses to impart skills to new staff inducted in the STIs clinics as a consequence of staff transfers. Periodically organize training of doctors working at STI clinics in the analysis of STI data and related laboratory data and then its use in the procurement decisions of STI drugs. Organize a system of provision of adequate supply of BCC material and protocols on syndromic management to the STI outpatient clinics. There is a need to standardise list of essential STI drugs, related supplies and equipment and plan developed on annual basis to make up deficiencies, besides ensuring regular supply of STI drugs to enhance credibility of public sector hospitals in the provision of quality STI services. iv

5 Rigorously enforce referral of suspected HIV/AIDS cases from STI clinics to VCT centres for improving the knowledge, attitude and skills of the persons having risky behaviour. Introduce standard recording and reporting system at STI clinics (including the culture of entering on the outpatient register the age, sex, occupation and syndromic diagnosis). III BLOOD BANK SERVICES 3.1 Standard Operating Procedures 34. Application of Standard operating procedures (SOPs) was assessed for anti-hiv testing and four SOPs related to quality assurance comprising: (i) optimum storage of kits, reagents and consumables; (ii) preventive maintenance of equipment; (iii) calibration of equipment; and (iv) incident reporting. 35. The assessment results indicate that the concept of use of SOPs has not been introduced in the surveyed blood banks. Staff of every blood bank, based on their knowledge and skills, used unwritten OPs. Such unwritten procedures didn t permit assessing and comparing the standards used by different blood banks and even of the same blood bank overtime. 36. Since almost all blood banks were providing services without the application of SOPs the review objectively assessed technical contents of services in the following areas and made rating in terms of adequate with regard to: (i) scope and application of unwritten operating procedures (OPs); (ii) responsibility for the results; (iii) reference for comparing results; (iv) required material related to two OPs i.e. anti-hiv testing and optimum storage of kits/reagents/consumable ; (v) procedures specific to each OP e.g. procedures covering principle, method, validation and interpretation in case anti-hiv testing; (vi) documentation; and (vii) staff training. 37. Scope and application of unwritten OPs was found adequate only in about 33% of the blood banks, and falling much below the average in case of incident reporting (18%), calibration of equipment (22%) and preventive maintenance of equipment (27%). 38. Assigning the responsibility for implementing various unwritten OPs was little more clear than the scope and application of OPs. The assignment for responsibility was found adequate in 57% of blood banks However, adequacy of responsibility for calibration of equipment and incident reporting was low. 39. The knowledge of reference documents for comparing results, in general, was highly inadequate. The relatively better knowledge of reference documents for anti-hiv test and optimum storage of kits/reagents/consumable was because of the fact that many respondents referred to the literature that comes with the test kits. The respondents, on the other hand, had no clue of reference documents for the OPs for (i) calibration of equipment and (ii) mechanisms for correction and prevention of errors and incidents. 40. Blood banks had better knowledge of materials required for anti-hiv testing and optimum storage of kits/reagent/consumable. Staff from 65% of blood banks had adequate knowledge of required materials. 41. Knowledge of procedures to implement the OPs was also found quite low particularly for preventive maintenance of equipment, calibration of equipment and incident reporting and was in the range of 19%, 11% and 10% respectively. v

6 42. Documentation required for OPs e.g. in terms of expiry date of kits, reactive tests, maintenance of records and files, incidents, etc was found more adequate for anti-hiv testing and optimum storage of kits/reagents/consumable and was at the level of 60% and 37% respectively. However, documentation for OPs on preventive maintenance of equipment, calibrations of equipment and incidents reporting was highly inadequate; and adequate level of documentation was rated in the range of 8-18%. 43. Training of blood bank staff in the application of OPs had hardly been undertaken. This was particularly true for preventive maintenance of equipment either through external system or in-house arrangements, calibration of equipment, and incident reporting. Nearly ½ to 1/3rd staff had adequate training in anti-hiv testing and optimum storage of kits/reagents/consumable. Staff training in Balochistan and AJK/FANA appeared to be non-existing. 3.2 Implementation of universal precautions. 44. Universal precautions are a set of precautions designed to prevent transmission of HIV and other blood borne pathogens while providing healthcare services. For review purposes, universal precaution covered the areas described below. 45. Use of apron in the working environment was uncommon and observed, on an average, in nearly 20% of blood banks, far more common in private sector followed by blood banks in the teaching hospitals, and least common in NWFP, AJK/FANA and THQ hospitals. 46. Use of gloves while working with the blood donors was practiced in about 40% of blood banks, more commonly practiced in the private sector blood banks and those in Punjab province. 47. Staff training. In 74% of the blood banks, most of the technical staff had received training in epidemiology, mode of transmission and prevention of blood borne infections, except in AJK/FANA where none of the staff was trained. 48. Hand washing practice was almost universal. However, hand washing after removal of gloves was less commonly practiced (in about 25% of blood banks) than washing by convenience or necessity. 49. Cleaning of working table. 80% of blood banks confirmed the practice of cleaning the table top whenever it was spoiled with blood and blood products, while blood banks from Sindh, NWFP, AJK/FANA and private sector were universally following this practice. 50. Availability of disinfectant and its use. On an average, 80% of blood banks had disinfectant in their stock. However, 22 (35%) blood banks were cleaning the top of the working table with a substance other than disinfectant. 51. Disposal of discarded blood and blood products. 25% of the blood banks were disposing off discarded blood and blood products in a drain connected with sewer while the remaining were using other methods of disposal e.g. land filling or dumping, burning/ incineration and disposal with municipal waste. 52. Method of disposal of sharps and needles. On an average, 20.6% of blood banks had been using punctured resistant containers for the collection of used needles and this practice was more commonly followed by private sectors blood banks (70%) and those from Sindh province (60%). 53. Quantitative assessment shows that universal precautions were better practiced in the private sector blood banks (65% score) followed by those in the teaching hospitals (60%). vi

7 Universal precautions were also better implemented by blood banks in Punjab province (62% score) followed by those in Sindh province (59% score). 3.3 Assessment of waste management standards 54. Collection of blood bank waste. Use of containers for separately collecting kitchen, hazardous and infectious wastes was found an uncommon practice. Only two blood banks in Punjab province were collecting three types of wastes in separate containers of standardized colors. 55. In 57% blood banks, the disinfection of hazardous and infectious material was not practiced. Incineration was the common method of treating hazardous and infectious material in 17 or 27% blood banks while autoclave, microwaving and treatment with sodium hypochlorite were less common methods followed in six, three and one blood bank respectively. Blood banks were more commonly disposing off the waste by dumping in the municipal waste (87.3%) than by landfill (eight or 12.7%). 3.4 Screening of donated blood for HIV before transfusion 56. Method of screening blood for HIV. Fifteen blood banks had ELISA equipment, 44 were using rapid qualitative assay for the detection of antibodies to human immunodeficiency virus as an initial screening test, and four blood banks were not screening blood or blood products for HIV. 57. Screening of blood for HIV before transfusion. In 2004, 47 blood banks (75%) screened all donated blood for HIV, 10 (16%) had partially screened donated blood because of shortage of HIV test kits, two blood banks confirmed screening of blood for HIV but not maintained any record (3%), and four (6%) were not screening donated blood for HIV. 58. Stock-outs of HIV test kits. Unfortunately record keeping was incomplete in most of the blood banks and record maintenance pattern was also variable. Therefore, stock-outs of HIV test kits could not be meaningfully analyzed. 3.5 Status of blood bank supplies for HIV testing. 59. Many blood banks were without essential equipment like refrigerator, centrifuge machine, water bath, incubator and timer. Glassware and material were also in short supply in many blood banks, especially in Balochistan and NWFP. Stock position of reagents, on the day of visit, was not very encouraging especially the availability of sodium hypochlorite, sulphuric acid, alcohol and distilled water. Test kits were found out of stock in 12 blood banks. 3.6 Way forward. 60. AIDS Control Programs are encouraged to arrange technical assistance for developing standards in the following areas: Standard operating procedures including those for quality assurance Standard protocols for enforcement of universal precautions Development and implementation of waste management standards Development and implementation of standard recording and reporting system to monitor donor screening and stock-out of various test kits Enforcement of uniform policy for screening of blood and blood products vii

8 8

Tool B Facilities, Supplies & Equipment

Tool B Facilities, Supplies & Equipment Voluntary Medical Male Circumcision (VMMC) Tool B Facilities, Supplies & Equipment External Quality Assurance (EQA) Site Name: Reviewer s Name: Date: This Tool (B)