Evaluation report of the Imp.Ac.T. Project

Transcription

1 The Rainbow Group Foundation (FRG), Amsterdam, the Netherlands Evaluation report of the Imp.Ac.T. Project [Improving Access to TB and HIV Testing for marginalized groups] Katrin Schiffer & Jenneke van Ditzhuijzen

2 Copyrights 2013 Copyrights remains with the author(s) and the publisher Authors: Katrin Schiffer & Jenneke van Ditzhuijzen Publishers: De Regenboog Groep Correlation Network Postbus EW Amsterdam The Netherlands Tel.: Fax.: Imp.Ac.T. has been co funded by European Commission under the programme of community action in the field of health and Zon Mw in the Netherlands. Neither the European Commission nor any person acting on its behalf is liable for any use of information contained in this publication. 2

3 Preface This is the Evaluation Report of the European Project Imp.Ac.T. Improving access tohiv/tb Testing for marginalised groups ( ). The Imp.Ac.T. project is the result of the work of many people; in particular, those that did the testing and the data collection for the project. The Imp.Ac.T. project was coordinated by Villa Maraini, developed and implemented by all project partners (Villa Maraini in Rome, Gruppo Abele in Turin, Odyseus in Bratislava, Sananim in Prague), and evaluated by the Rainbow Group Foundation. The Imp.Ac.T. project has been co-funded by the Health Programme of the European Commission. In this project, we (the evaluators) chose to give regular feedback of our findings to the project coordinators. Since the Imp.Ac.T. project was a pilot project and tools were not yet developed, it was important to contribute to the process in order to identify problems or bottlenecks in an early stage. Rather than waiting until the end of the project period with our feedback, we shared preliminary results with the project leaders in a systematic manner. The evaluators also introduced tools for monitoring, which could be used by the project leaders. We chose this approach because the benefits of continuous monitoring, evaluation and improvement of the project outweighed the benefits of a completely independent objective assessment of project integrity. We hope that in a future stage, this project will be repeated in other cities; then it could (and should) be evaluated by an independent evaluator. We thank all our colleagues of the Imp.Ac.T. project for their warm and good cooperation. Evaluating this project was an intense journey on rocky roads with lots of bumps and pitfalls. It was not always easy. However, we all made it, and we are proud that so many drug users have been tested. We hope that the road, which is now paved with guidelines and advice, will inspire others to continue building it. We also thank the people who advised us, in particular, Lucas Wiessing (EMCDDA, Lisbon, Portugal), Dagmar Hedrich (EMCDDA, Lisbon, Portugal), Rob van Hest (Dept of TB control, Municipal Health Service Rotterdam, Netherlands), Enrico Girardi (Spallanzani, Rome, Italy), Niels Krummacher (independent trainer, Netherlands), Jessica Baars (IVO, Rotterdam, Netherlands) and Richard Braam (CVO, Utrecht, Netherlands). Amsterdam, January 2013 Katrin Schiffer and Jenneke van Ditzhuijzen 3

4 Contents Executive summary... 5 List of acronyms Project partners General introduction Description of the project Project description Primary objectives Secondary objectives Project outline Process Evaluation Process evaluation objectives Process evaluation methods Process Evaluation results Supporting tools Project integrity Motivation and support Data collection and impact evaluation Introduction and research objectives Data collection methods Gathering epidemiological data on baseline situation Focus groups with target group members Questionnaires Database Response monitoring Data collection results Response Quantitative results from the questionnaires Qualitative results from the focus groups Lessons learned and conclusions Literature

5 Executive summary Project description In the Imp.Ac.T. project, outreach work has been used to promote a new kind of providerinitiated counselling and TB and HIV testing. Rapid HIV and TB tests has been offered to problem drug users (PDU s) in street units, drop-in centres and on the streets in 4 European cities: Rome, Turin, Prague and Bratislava. In each city HIV rapid tests and TB screenings were done in outreach facilities, as well as TB tests to those people symptomatic or in close relations with TB positive persons. The people who were tested positive with HIV and/or TB were referred to collaborating clinics, for diagnosis confirmation and treatment. The general objective of this project was to improve the access to HIV and TB testing, prevention, treatment and care for PDU s in different European countries. PDU s have specific needs and encounter specific challenges in order to get tested and treated. Specific objectives were: To develop a framework and model (guidelines) to improve the effectiveness of HIV and TB testing among PDU s. To increase the percentage of PDU s having access to HIV and TB testing. To promote treatment for people living with HIV and TB. To promote healthier ways of life and risk reduction among PDU s. To assess the effectiveness of street HIV and TB testing in terms of proportion of new infections identified. Evaluation questions The evaluation of the Imp.Ac.T. project consisted of two parts: a process evaluation and an evaluation of the possible impact of the intervention in terms of behaviour change and epidemiology (if possible). The main questions guiding the process evaluation were: Was the project carried out according to the plan (research protocol)? What was changed/ adapted, and why? Were all the goals reached? Which barriers were encountered and how were they solved? How was the intervention received by clients and social workers? How were the different stages of the project experienced? What went well, what could have been improved? (in terms of: project coordination, cooperation and communication, motivation and support, implementation, dissemination, data collection, etc.) What are the most important lessons to be learned? The main questions guiding the impact 1 evaluation were: Is there a change in the indicators (experienced access to HIV and TB testing and treatment, testing behaviour and attitudes, risk-related behaviour, knowledge on TB 1 Formally, we cannot speak of an effect evaluation or an impact evaluation; but since there is no adequate alternative term, we continue to use this terminology in this report. 5

6 and HIV infection and prevention) that can be attributed to the intervention, when comparing baseline to post-intervention measurement? Is the intervention experienced as effective? The main categories of variables for the data collection were: Sociodemographic variables Drug use, active/recent/ever IDU Imprisonment Sexual risk behaviours HIV testing behaviour TB testing behaviour METHODS Process evaluation methods The logical framework approach was used to define the objectives, indicators and means of verification, in order to be able to measure level of achievement of the goals. In addition, a milestones table and activity lists were developed. Progress was measured with progress monitoring forms, however, halfway the project it was decided to substitute these forms by evaluation interviews (using skype), since the forms were not informative enough. Meetings and training sessions were evaluated using online survey tools (combining open and structured questions), as well as evaluation report forms. Main variables for meeting evaluations were: purpose of meeting attained, participant roles and tasks, meeting quality/ productivity and level of success. Data collection and impact evaluation methods At the outset of the project, an overview was made of already available epidemiological and demographical data (baseline information), for the target group in the participating cities. A qualitative pre- and post-intervention measurement - in the form of focus groups - was chosen as the most appropriate method to show behavioural change and changes in experiences. Two questionnaires were developed within the Imp.Ac.T. project. The first one was the main questionnaire which was administered the first time a participant was tested. The second one was a follow up questionnaire which was administered whenever participants came back for follow up testing. Questionnaire data and testing details and results were registered in two databases (one general database, one from an online survey tool in Prague). In addition, a response monitoring form was used to register some demographical details of responders and non-responders as well as reasons for nonresponse. RESULTS Process evaluation results To a large extent, the project was carried out according to the protocol. The most important adaptations included (1) a lowering of the target sample sizes form 1,000 tests per city to 700 tests in Rome, Turin and Prague and 300 tests in Bratislava; (2) a change in TB testing methodology from latent TB testing using TST to active TB testing using sputum (only for those that were symptomatic according to the screening); and (3) the target group definition 6

7 was broadened twice, from IDU to PDU to clients of the low threshold services, even though PDU s remained the main target group for the research. The primary project objectives were largely reached: A framework and model for improvement of effective HIV and TB Testing among PDU s was developed; the percentage of PDU s having access to HIV and TB testing was increased; the effectiveness of street HIV and TB testing in terms of proportion of new infections identified, was assessed, treatment for people living with HIV and TB was encouraged, but not achieved due to difficulties in follow up, healthier ways of life and risk reduction among PDU s were promoted in the counseling, however this did not lead to a risk reduction. Changing behavior is a slow process that is usually not done in one or a few sessions. Main barriers that were encountered, were (1) unforeseen circumstances, such as the closing down of services which decreased response; (2) follow up of reactive participants was very difficult; (3) communication between project partners was not always efficient or supportive/ stimulating, (4) problems with switching roles between social worker and interviewer. These barriers were addressed by (1) finding new outreach locations; (2) providing incentives to increase response; (3) setting up regular contact between partners by using an group and regular skype conferences; (4) encouraging regular exchange in the local teams, to be aware of potential problems; and (5) organise a second training event dedicated to the motivation of respondents as well as the conflicting roles. After taking these measures, the response increased and communication between partners was experienced as more positive. Of course there is always room for more improvement: communication regarding financial issues could have been improved and deadlines could have been respected more by the partners. Data collection and impact evaluation results In total, 4,855 persons have been approached, of which 2,352 have been interviewed, resulting in a total response rate of 48%. Most participants were male, (73%), age (40%), often homeless (44%), unemployed (66%) and had mostly fulfilled a secondary education (50%). Only 13% was migrant (Bratislava 3% Prague 6%, Rome 16% and Turin 25%. We found 20 HIV reactive cases (9 in Rome, 9 in Turin and 1 in Bratislava and 1 in Prague, of which 5 were confirmed positive and 1 turned out to be a false positive (the one in Prague). Assuming that the other cases are no false positives, 19 cases of HIV have been detected (of which 15 active IDU s). No real prevalence or incidence can be calculated, but this diagnostic testing prevalence or incidence, the newly diagnosed cases within the active IDU-group reached 1%. If this would reflect real incidence in terms of percentage of new infections within our target group (and there are good reasons to do so), the data regarding HIV in Italy are quite alarming. No case of active TB has been detected, only one case of latent TB has been discovered among the HIV positive persons. We do not know whether TB is not a big problem in the four participating cities, or if our screening and/or testing method was not adequate enough. Drug use patterns of active drug use (in the last 4 weeks) were quite different for the four cities. In Rome, heroin was by far the most popular drug (83%), and other drug use was low. In Turin, the pattern of drug use was more diffuse. Cocaine was the most used drug (77%, often injected), followed by one footstep by heroin (72%). The difference between Rome and 7

8 Turin in drug use might be partly related to higher methadone use in Turin, which could explain the low use of heroin. In Prague, heroin use was the lowest, only 15%. However, a lot of other drugs were used in Prague, with pervitin (metamphetamines) as the most used drug (78%), which was mostly injected (75%). In Bratislava, drug use was mainly centered around pervitin (95%) and heroin (95%). As far as other risk factors, 60% had ever shared needles (from 38% in Rome to 77% in Bratislava), and 13% in the last 4 weeks (from 10% in Rome to 28% in Bratislava). At follow up 15% was still sharing needles. Of the participants, 52% had ever been in prison and 25% had injected drugs in prison. Sex workers and their clients used condoms about half the time to mostly (m=3.6); casual partners occasionally to about half the time (m=2.7). Based on the pre-intervention and post-intervention focus group results, we did not find evidence that experienced access to HIV and TB testing had changed. Both for TB and HIV barriers in access to testing were mostly attributed internally, to themselves (fear, underestimation of risk, indifference, other priorities), and not externally to the health care facilities. To a lesser extent, discrimination because of their drug use or migrant status was mentioned. Knowledge on TB was often sketchy and low ( a disease from the past ), knowledge on HIV was a lot richer in most cities ( we know more than non-users ). However, knowledge, attitudes and behaviours of both Imp.Ac.T. participants and non-participating target group members was more or less the same before and after the implementation. In the follow up questionnaire, it was also asked if risk behaviour had changed; 60% answered not at all. Clients evaluated the project as very positive; it was effective in the sense that people were both tested and educated where they would have normally delayed this or not gotten it at all. They were satisfied and happy about the services, although many of them considered Hepatitis C as more important, which might give an idea for future actions in the field. CONCLUSIONS Lessons learned In hindsight, we believe it would have been better if the baseline assessment would have been done more thoroughly before the start of the project. It was assumed that TB is a major problem among DU s, based on general European findings that proportions of TB-HIV coinfections range from 2 to 15% in the EU. However, in the participating cities TB did not seem to be a big problem among the target group. The baseline assessment did not provide conclusive information, but data from a pilot project like Imp.A.c.T. could provide more insight into TB prevalence among drug users. The involvement of collaborating local partners at an early stage is essential. Additional partners can be the clinical centres, health services, laboratories and other low threshold services, which might cooperate with the implementing organisations. Involving these additional partners in an early stage could improve the methodology of the project, the practical collaboration on the local level, as well as the commitment of these partners to reach the planned objectives and results. Furthermore, it will contribute to facilitate the access of clients to treatment and care, namely the follow-up, which proved to be problematic in most of the participating cities. The development of a detailed and comprehensive methodology and study protocol is essential as well. The methodology should be based on the review of existing evidence and 8

9 should take into account specific needs of the target population as well as potential limitations of the service providers, which provide the testing. In addition, it is important to be aware of legal boundaries and to set up a supportive local network, which is involved in the planning and implementation of the project. To ensure an effective and measurable intervention it is essential to develop and establish regular monitoring tools. The monitoring tools were complemented by evaluation interviews, organised after 1 year and after two years. This additional method provided useful information on the progress of the project and gave insight about perceived barriers and problems. The training of staff members was an important part during the preparation phase of Imp.Ac.T. All partners contributed actively to the content of the training manual. The experience within Imp.Ac.T. has shown that it is extremely important to organise regular team meetings, to monitor the needs of the workers and to offer regular and continuous training. Some workers experienced problems in combining their work as social worker with the role as interviewer and data collector. We have described several ways to deal with this role conflict. Apart from monitoring social workers experiences, listening to them, and guiding them; it is also important that their work for the research part is acknowledged, appreciated and rewarded. The recruitment was carried out by the outreach workers in low-threshold settings. Although it has not been investigated, it might have been useful to involve DU s as peers for the recruitment and to see if this approach would have increased the final sample size and the motivation of PDU's to enroll in the project. Existing evidence could plead for such an approach in the future. Three partners decided to give incentives (phone cards and food tickets) to increase the motivation of DU s to enroll in the project. All of them reported an increased motivation among the participants, and this was also visible in the response rates of these cities, which leads to the assumption that incentives could work well for this particular target group. The TB testing methodology was a point of critique. None of the discussed and suggested methods seemed to be ideal. The question remains whether the right choices were made in developing TB testing methodology. Regarding HIV testing, the use of the HIV rapid test worked well, but it is recommended to use other methods for confirmatory testing as well, such as the blood analysis on the spot or the use of a second rapid test for confirmation In this way, clients do not need to come back for follow up or be referred to other health centres, with the risk to get lost for follow up. Ensuring follow up has been one of the major challenges during the project. Therefore, one of the most important recommendations should be to cooperate more closely with other local partners. It is not enough to involve them during the course of the activities. They should also be part of the planning and preparation phase to become more committed to the project and the target group. The data collection should be embedded in the regular services of the partner organisations. It was an important project principle that the research activities should not intervene with the practical work too much, but rather be complementing to it as a logical extension of the regular activities, providing research data. 9

10 The combination of research and practice worked well, although it has not always been unproblematic. This again pleads for decent training and cooperation among staff members. It is also advisable to have a researcher at each location (partner organisation), who could support workers and supervise the data collection more closely. Within Imp.Ac.T. several data collection tools were used. The data of the questionnaires were recorded in an online database, which could be accessed by all partners (only their own data could be modified). The online survey tool which has been used by the Czech Republic only, enabled the workers to fill in the data directly at the spot, without having double work and risking loss of data. The Response Monitoring Form was considered to be very time-consuming and some workers experienced it as quite disturbing for the overall working process. The tool has clear benefits, such as knowing the response rate, demographics of non-responders and reasons for refusal. However. these benefits might not outweigh the possible harm (resistance to data collection) of the extra workload and interference with the primary social work process. In future projects like these, benefits and possible harms should be discussed and the use of a tool like this should be carefully deliberated. Final conclusions: the impact of Imp.Ac.T. The Imp.Ac.T. Project provided about 2600 tests (first and follow up), interviewed and tested nearly 2350 people, of which nearly 2200 problematic drug users (PDU's). This on itself is already a success and provides us with new and vital information on the situation of DU's in Europe, the HIV and TB prevalence and incidence rate among DU's and the (perceived) barriers for DU's to access HIV and TB testing, treatment and care. The specific approach within Imp.Ac.T. has been that low-threshold services and to be more specific social workers, psychologists and nurses/doctors were responsible for the implementation of activities, which included the recruitment, the pre- and post test counselling, the interviewing and recording of data as well as the TB and HIV testing procedure. This was a unique approach which offered the opportunity to reach a large number of target group members and collect a rich dataset. Imp.Ac.T. has been a valuable example of a combination of a street intervention and data collection. The fact that interviews were done by social workers sometimes improved the relationship with clients, but was also difficult in terms of role switching. It is evident that Imp.Ac.T. managed to test a large number of DU s on HIV and TB. It can also be assumed that the provision of low-threshold testing has been useful and effective for hard-to reach groups: almost 20% of the people tested in the framework of the Imp.Ac.T. project were never tested before for HIV. 10

11 List of acronyms AB BR CR DU EAHC ECDC EMCDDA FG FRG GA HCV HIV IDU Imp.Ac.T. LFA OD PDU PR RMS/ RMF RO SA TB TU TST VM WP Advisory Board Bratislava Czech Republic Drug User Executive Agency for Health and Consumers of the European Commission European Centre for Disease Prevention and Control European Monitoring Centre for Drugs and Drug Addiction Focus group The Rainbow Group Foundation (Stichting De Regenboog Groep) Gruppo Abele Hepatitis C Virus Human Immunodeficiency Virus Injecting Drug User Improving Access to TB and HIV Testing for marginalized groups Logical Framework Approach Odyseus Problem Drug User (EMCDDA definition) Prague Response Monitoring Sheet/ Response Monitoring Form Rome Sananim Tuberculosis Turin Tuberculin Skin Test (also known as Mantoux or PPD test) Villa Maraini Work package 11

12 Project partners Villa Maraini, Rome, Italy: project coordination and implementation Laura Ceccarini (project leader) Nadia Gasbarini (project officer) Philippe Garcia (project financial administrator) Rita Sista (psychologist) Gruppo Abele, Turin, Italy: project implementation Lorenzo Camoletto (contact person) Massimo Carroci Raffaella de Vincentiis Odyseus, Bratislava, Slovakia: project implementation Katarina Jiresova (contact person) and colleagues Sananim, Prague, Czech Republic: project implementation Jiri Richter (contact person) Tomas Vejrych Jakub Minarik The Rainbow Group Foundation, Amsterdam, the Netherlands: project evaluation Katrin Schiffer Jenneke van Ditzhuijzen 12

13 1. General introduction The evaluation of the Imp.Ac.T. project consisted of two parts: a process evaluation and an evaluation of the possible impact of the intervention in terms of behaviour change. Since a formal or real effect evaluation (with pre-and post-intervention measurement and control group) was not possible, we chose to make use of qualitative methods such as focus groups. These methods have the positive side-effect that they make use of the knowledge of experts (in this case, drug users), and lead to discussion, knowledge transfer, and can contribute to empowerment of the people involved. Furthermore, the Rainbow Group Foundation was involved in the work package dedicated to data collection ( assessment of main determinants ). The process evaluation was coordinated by Katrin Schiffer, and the data collection/ impact evaluation by Jenneke van Ditzhuijzen, who was subcontracted by the Rainbow Group Foundation as an independent researcher. The main questions guiding the process evaluation were: Was the project carried out according to the plan (research protocol)? What was changed/ adapted, and why? Were all the goals reached? Which barriers were encountered and how were they solved? How was the intervention received by clients and social workers? How were the different stages of the project experienced? What went well, what could have been improved? (in terms of: project coordination, cooperation and communication, motivation and support, implementation, dissemination, data collection, etc.) What are the most important lessons to be learned? The main questions guiding the impact 2 evaluation were: Is there a change in the indicators (experienced access to HIV and TB testing and treatment, testing behaviour and attitudes, risk-related behaviour, knowledge on TB and HIV infection and prevention) that can be attributed to the intervention, when comparing baseline to post-intervention measurement? Is the intervention experienced as effective? The main categories of variables for the data collection were: Sociodemographic variables Drug use, active/recent/ever IDU Imprisonment Sexual risk behaviours HIV testing behaviour TB testing behaviour In the second chapter of the evaluation report, a short description of the project and its specific components and objectives will be provided. The third chapter will be dedicated to the process evaluation. The fourth chapter is about the data collection (both quantitative and qualitative methods and results). In the last chapter we will draw the main conclusions and describe the lessons to be learned. Annexes with more detailed information can be found in the supplement of the report. 2 Even though formally we cannot speak of an effect evaluation or an impact evaluation, we continue to use this term in this report. 13

14 2. Description of the project 2.1 Project description In this pilot project, outreach work has been used to promote a new kind of providerinitiated counselling and TB and HIV testing. Rapid HIV and TB tests has been offered to drug users in street units, drop-in centres offering low-threshold drug services (needle exchange, substitution treatment) and on the streets (as part of outreach work) in 4 European cities: Rome, Turin, Prague and Bratislava. In each city HIV rapid tests were available and TBtests were provided to those people who were symptomatic or high-risk (HIV-positive or in close relations with TB positive persons). The people tested positive with HIV and/or TB were referred to collaborating clinics, for diagnosis confirmation and treatment. A common methodology has been developed, both for the implementation of the testing and the assessment of its effectiveness. The common methodology consisted of specific guidelines for outreach work and access to testing, the testing itself, the pre and post test counselling, and the referral to treatment and care. It is highly important that the developed strategies were tailored to the needs of the target groups. It is not only difficult to reach the target groups in the first place, but it could also be difficult to approach them for testing without evoking adverse reactions. Therefore it was made clear to the clients that testing is completely voluntarily and confidential. Individuals in these target groups are often already marginalized and may therefore be more susceptible to (and afraid of) coercion and discrimination upon disclosure of HIV or TB status (possibly also within their own population). It should be guaranteed that conditions under which testing is done are in accordance with human rights and ethical principles regarding confidentiality. 2.2 Primary objectives The general objective of this project was to improve the access to HIV and TB testing, prevention, treatment and care for DU s in different European countries. DU s have specific needs and encounter specific challenges in order to get tested and treated. Specific objectives were: To develop a framework and model (guidelines) to improve the effectiveness of HIV and TB testing among DU s. To increase the percentage of DU s having access to HIV and TB testing. To promote treatment for people living with HIV and TB. To promote healthier ways of life and risk reduction among DU s. To assess the effectiveness of street HIV and TB testing in terms of proportion of new infections identified. 14

15 2.3 Secondary objectives The development of guidelines will be useful for improvement and promotion of a wider access to testing and care for hard to reach risk groups across Europe. Better access to testing and treatment will also influence the public health sector in general. Since more people will get tested and treated, this could lead to a decreased transmission among the whole population. Knowing one s own TB/HIV status can have an important impact on transmission-related behaviours and health-promoting behaviours, especially if adequate and targeted information is provided. Other secondary objectives were: To contribute to universal access to HIV prevention, testing, treatment and care To reduce the gap between drug users/migrants and health care services To reduce inequalities in the access to treatment. 2.4 Project outline The project consisted of three phases: 1. Development of common tools for street HIV and TB testing among DU s. Methods and means: a. Organisation of workshops/seminars in order to (1) exchange experience and information among the project partners, and (2) analyze weaknesses and challenges of current HIV and TB testing strategies. b. Development of training courses for multi-disciplinary staff (doctors, social workers, psychologists) working in low-threshold services for DU s. c. Identification of specific indicators/determinants for monitoring and reporting of new diagnosis among these target groups. 2. Implementation of HIV and TB rapid tests in low-threshold facilities for DU s. Methods and means: a. Training sessions for the staff of the partner organisations in the four different locations. b. Data collection: clients are requested to fill in a short questionnaire for collecting data on lifestyles, behaviours, health conditions, history of HIV and TB testing. The questionnaire will also include questions about the presence of clinical symptoms of TB in order to identify suspected cases. On the basis of the information collected through this questionnaire, individuals will be offered the option of (free) testing or provided risk reduction counselling in case they do not want to be tested. c. The actual implementation: providing the tests (1000 tests per city), the counselling and the referral to follow up confirmation of results and treatment facilities. d. Follow-up: A follow-up appointment will be scheduled at dedicated health care services for those individuals that are tested HIV-positive for confirmatory diagnosis. TB positive subjects will be referred to clinical services for further control. e. Treatment: All collaborating clinics offer HIV and TB treatment therapy. Within this project only treatment entry will be measured, since the project period is too short to effectively measure adherence. 3. Analysis and assessment of effectiveness of the intervention, as well as 15

16 dissemination of the results. Methods and means: a. Analysis of all collected demographical, epidemiological, clinical and laboratory data from each partner (number of persons tested, number of HIV and TB-positive, number of persons received treatment after testing). b. Assessment of short-term impact of the intervention (as far as this is possible within the current set-up without control group and without quantitative pre-intervention measurement). c. Publication of guidelines for best practices dissemination. d. Organisation of a final conference for presenting project results and outcomes. The workpackages of the project were structured as followed: WP 1: Management and coordination of the project WP 2: Dissemination WP 3: Evaluation of the project WP 4: Development of common methods and tools for HIV and TB testing of DU's/migrants in low-threshold services WP 5: HIV/TB street testing and low threshold services testing for DU's/migrants WP 6: Assessment of main determinants of late and early diagnosis and entry into care for risk groups. Scope of the evaluation It is important to note that not everything could be evaluated. In the beginning of the project, ambitions were high. The evaluators commented that not all indicators could be measured within the scope and timeframe of the project. For example, main determinants of late or early diagnosis of TB and/or HIV cannot be assessed when the HIV or TB positive subsample is small, entry into care (follow up) of these small subsamples is virtually impossible to measure in this relatively short project period. Following consultation with the EAHC project officer, it was decided by the project leading organisation to keep the indicators unchanged. 16

17 3. Process Evaluation 3.1 Process evaluation objectives A process evaluation is a systematic assessment of the implementation of a programme or project as a whole in order to find out whether the project is carried out according to plan, which barriers were encountered (if any) and whether and how the intervention can be improved in the future. The three main categories of research questions for the process evaluation are: Project plan (content): Was the project carried out according to the plan/ protocol (project-integrity)? Were the preconditions met, milestones reached, activities carried out? Were participants selected on the basis of screening criteria? Were the instruments (questionnaires and other tools) user friendly and accurate in terms of information retrieval? In what way was the quality of the intervention secured on the longer term? Was the cooperation between the partners and with third parties satisfactory? Which barriers in the implementation were encountered? How were these problems solved? Motivation and support: How was the intervention received by the target group members? Are participants motivated to enrol in the project? What are reasons for the target group members to enrol in the study or not to participate (response and nonresponse)? What are opinions of the partners and different stakeholders and key informants on the method and tools? Lessons to be learned: Which barriers, bottlenecks or problems were identified (foreseen and unforeseen)? Which solutions were chosen or which or changes have been made to the intervention, in order to prevent these problems in the future? Which lessons can be learned? Involvement of the evaluators In this project, the process evaluation addressed all three main phases of the project: development, implementation and analysis. Since this is a pilot project and tools are not yet developed, there was regular feedback on the process in order to identify problems or bottlenecks in an early stage. Rather than waiting until the end of the project period, the evaluators shared preliminary results with the project leaders in a systematic manner. In this way possible problems could be tackled in an early stage. In order to evaluate the process, the evaluators also introduced tools for monitoring. These tools might be useful for project management objectives as well. This means that the process evaluators were, up to a certain extent, involved in the process of development and not entirely independent. In this pilot project, the benefits of continuous monitoring, evaluation and improvement of the process outweigh the benefits of a completely independent and objective assessment of project integrity. In a follow up stage of this project, it would be advisable to do an independent process evaluation. 17

18 3.2 Process evaluation methods Logical Framework Matrix Because the intervention was not yet (fully) developed at the outset of the project, objectives have been stated at a fairly abstract level. In the process of development of the project the objectives should become more specific, measurable, attainable, realistic and time-oriented ( SMART ), in order to be able to measure level of achievement of objectives (impact evaluation). The Logical Framework Approach (LFA, or logframe method ; e.g., Horstman et al., 2002; UNAIDS, 2007) is an internationally widely used and appreciated method that can aid this process. The logframe matrix clarifies the intervention logic because one needs to identify the objectively verifiable indicators (expected changes in variables) and what the means or operationalisations are to verify this (methods or sources of information). In the Imp.Ac.T. project, a Logframe Matrix was developed, by describing for each objective the general activities that were needed to reach that objective, the indicators which would tell us when the activity was fulfilled to sufficient extent, and the means of verification; that is, the source of data or information. Milestones table and activities/output lists per work package In order to have a good overview of the project period in terms of reaching objectives or producing deliverables, a milestones table will be developed for the project as a whole. In addition, all activities and the corresponding outputs per work package and per project partner will be listed. In these tables, the following will be registered by each partner: activity activity period responsible person(s) output/ deliverables (note that not all activities have measurable outputs) finalizing date (deadline) level of success comments Progress monitoring forms Every three months, the progress was assessed. All partners are asked to complete a standardised periodical progress reporting format. In this report, the activities and outputs/ deliverables for the reporting period are listed (according to the activities list mentioned above). The partners were asked to report the level of progress (or success) of the activities and outputs/ deliverables, as well as barriers and problems they might have encountered. Another part of this progress report was dedicated to cooperation with project partners, collaborating partners and others. Lastly, it was assessed what the major achievements and barriers were in the reporting period, and which risks or barriers were possibly expected in the next period. An example progress report form can be found in Annex I. Evaluation interviews After half a year of progress reporting, it became apparent that the progress reporting was too time-consuming and not informative enough for evaluation purposes. It was decided to stop using the progress reporting forms and instead, have qualitative interviews with all the partners halfway the project and at the end of the project. Topic lists for these interviews can be found in Annex II. In order to monitor progress during the implementation phase, the 18

19 project coordinator decided to have regular skype meetings with the project steering committee (a representative of each partner organisation). Focus groups with target group members and project workers For the impact evaluation (see next chapter) interviews and focus groups were held in order to assess the situation at the outset of the implementation and after the implementation had taken place. In these interviews and focus groups, also some process evaluation topics were added, in order to get more in-depth information on the level of motivation and support of the different stakeholders at the outset of the project and how the project has been received in the four cities. The topic list of the focus groups with clients can be found in Annex IX. Meeting evaluation questionnaires Every meeting, workshop or training was evaluated using an online survey tool (Survey Monkey). Main categories of items are (to be adapted according to the purpose of each meeting): purpose of meeting attained participant roles and tasks quality of the meeting (in terms of level of participation, decision making, productivity, organisation, et cetera) level of success The questionnaires were distributed on paper or sent out digitally to the participants of each meeting. The data were analysed by the external evaluator and a short report was sent to the project management team (Villa Maraini). The questions of the meeting evaluation questionnaires are listed in Annex IV. 3.3 Process Evaluation results Supporting tools After consultation of a Dutch expert (Richard Braam, CVO) in the field of Logical Framework Approach, a logframe matrix was developed for the Imp.Ac.T. project during the development stage of the project. At the end of the project, the indicators mentioned in the Logical Framework were assessed. The assessment of the indicators was also added to the logframe matrix. The result of this is displayed in Table 1. Experiences with using the Logical Framework were mixed. It was helpful to create the matrix because all the indicators and activities were more clearly defined this way. However, the Logframe Matrix was not very useful for monitoring progress. The activity lists and milestones table were much more user friendly as progress monitoring tools. Furthermore, we also experienced that some of the indicators were formulated too broad, whereas others were very detailed, especially those indicators on which we could not collect enough data. The Logframe matrix also did not cover all activities, for example, experience sharing meetings were held, but not part of the Logframe Matrix. The milestones table and activity lists were created as an overview of milestones and activities for each work package separately. The Milestones Table is displayed in Table 2, an example of an activity list for one work package can be found in Annex III. 19

20 Table 1. Logical Framework Matrix with activities, indicators, means of verification and short description of results/assessment Specific objective 1: To develop a framework and model to improve the effectiveness of HIV and TB testing and counseling among (P)DU's Activities Indicators Means of verification Results/Assessment Organisation/coordination of the workflow and the project activities aimed at development of the intervention 3 meetings with associated and collaborating partners dedicated to the development of the intervention, 2 experience sharing meetings with associated and collaborating partners, dedicated to the supervision and evaluation of the intervention. Participation of at least 8 participants to each meeting, including various collaborating and all associated partner organisations. Meeting reports Participants lists Meeting evaluation report Achieved: 1 kick-off meeting (October 2010, Rome) and 2 following meetings (January 2011 in Amsterdam and March 2011 in Turin) were dedicated to the development of tools. 2 Experience sharing meetings (October 2011 in Bratislava and February 2012 in Prague) were dedicated to the supervision and evaluation of the intervention. To each meeting around 3 persons per partner organisation (around 12 in total) and 1 or 2 project evaluators were present. Collaborating partners did not participate to these meetings. Meeting evaluation reports show that main purposes of each meeting were attained. Discussion and agreement on the overall study algorithm and the content of the training Audio conferences/skype calls Consultations with the Advisory Board Final Study Protocol and Training Manual finalized in March 2011 to be used by all implementing organisations Meeting reports Participants lists Meeting evaluation report Draft and Final Study Protocol Draft and Final Training Manual Regular audio or skype conferences were held in which agreements were made. Reports of these meetings were made by VM. An Advisory Board with international experts was set up. Some members of the Advisory Board were consulted regularly and contributed to the development of tools. Other members of the AB only provided feedback to the study protocol. Final Study Protocol (with study algorithm) finalized in M5, Final Training Manual ( concept) finalized in M6, final version adapted after implementation and printed, presented at Final Conference (Rome, November 2012). Training the operational staff in all implementing cities Organisation of at least 1 training session per implementing organisation targeting operational staff in March 2011 Training Manual Training Feedback Report Organisation of 2 training sessions per organisation Development of a monitorig system to analyse and evaluate collected data Presentation and dissemination of the activities of the project Participation of at least 10 operational staff members in total for all implementing organisations in March A standardised reporting system (data base) for monitoring and evaluating HIV/TB testing provision in low-threshold facilities operational until March 2011 Organisation of 4 national and 1 European Conference to present and disseminate the outcomes of the project to be held in Participation of at least 60 external stakeholders to 4 national and 1 European conference (in 2011 and 2012). Follow-Up Data Base Conference Reports Participants Lists Participation of 104 staff members Database has been developed by Villa Maraini, Sananim decided to use an online survey tool and a separate database. Both databases were merged for data-analysis. 3 national (VM and GA organised a common conference in Rome) 1 Final Conference organised Participation of 340 persons to the 3 national conferences (Prague: 150; Bratislava: 128; Rome: 62) Participation of 105 persons to the European Conference in Rome 20

21 Specific objective 2: To increase the percentage of (P)DU's having access to HIV and TB testing. Activities Indicators Means of verification Results/Assessment Assess the current local situation in regard to HIV and TB testing, care and treatment Setting up the organisational preconditions for providing rapid HIV and TB testing Baseline measurement to be finalised in april 2011 One new low-threshold service for HIV/TB testing per implementing city, providing services from May 2011 February A cooperation agreement with local collaboration partners before april 2011, stipulating all relevant details. Ordering 1000 TB and 1000 HIV tests per implementing city before april Reports on the current situation in regard HIV and TB testing, care and treatment in all implementing cities Low-threshold service for HIV and TB testing in all implementing cities. Collaboration agreement between associated and collaborating partners. Availability of testing material in all implementing cities. finalised One to three new low threshold services established per city, providing HIV/TB testing collaboration agreements set up and signed in all implementing cities 1000 tests ordered (and received) per city Providing rapid HIV en TB testing for (P)DU's in all implementing cities Number of eligible (P)DU's for HIV and TB testing in all implementing cities. Maximum of 1000 HIV tests being provided to (P)DU's in all implementing cities from May 2011 until Feb Maximum of 1000 TB clinical screenings (including sputum samples) being provided to (P)DU's in all implementing cities from May 2011 until Feb For all indicators: Baseline measurement table Response Non-response form Questionnaire data Test results Number of PDU s per partner city estimates available. Response analysis performed. Data collected. Data collected, indicator attained. Minimum of 250 (P)DU's tested for HIV in all of the implementing cities from March 2011 until Feb Data collected, indicator attained in 3 out of 4 cities, in Bratislava 181 PDU s were tested (and 100 non-pdu s). Minimum of 250 clinical TB screenings provided in all 4 implementing cities from May 2011 until Feb Data collected, indicator attained. All (P)DU's with clinical TB symptoms are tested on TB in all implementing cities from May 2011 until Feb Data collected, indicator attained, however in Prague PU s with TB symptoms were referred to a clinic for x-ray (sputum not possible). Increase of (P)DU's tested for HIV and screened for TB once per year in all implementing cities compared to the year before. Percentage first-time tested individuals calculated, however so far not possible to relate to baseline data Significant increase of (P)DU's tested for HIV every 3 months in all implementing cities compared to the year before. See above Significant increase of (P)DU's tested for TB at least every 6 months (compared to the year before). See above Incidence of tested (P)DU's never tested before in all implementing cities. Percentage (P)DU s never tested before calculated (real incidence not possible) Minimum of 80% of the TB tested patients receiving their results after the first testing. Not achieved: most TB tested patients did not come back for collection of second sputum sample. However, for TB screening, 100% of participants 21