Cancer Control Council Evaluation and Monitoring Framework

Transcription

1 Cancer Control Council Evaluation and Monitoring Framework

2 Table of contents 1. Purpose Background Clarification of key tasks International evaluation and monitoring frameworks Cancer Control Council Evaluation and Monitoring Framework Engaging stakeholders Describing the programme Focusing the evaluation Gathering credible evidence Justifying conclusions Ensuring use and lessons learned References...8 Cancer Control Council Evaluation and Monitoring Framework i

3 1. Purpose This evaluation and monitoring framework has been developed to provide the conceptual framework for the Cancer Control Council s evaluation and monitoring work. 2. Background The New Zealand Cancer Control Strategy, launched in 2003, provides a high level framework for reducing the incidence and impact of cancer in New Zealand and reducing inequalities with respect to cancer. The six goals of the Strategy are to: 1. reduce the incidence of cancer through primary prevention 2. ensure effective screening and early detection to reduce cancer incidence and mortality 3. ensure effective diagnosis and treatment to reduce cancer morbidity and mortality 4. improve the quality of life for those with cancer, their family and whanau through support, rehabilitation and palliative care 5. improve the delivery of services across the continuum of cancer control through effective planning, co-ordination and integration of resources and activity, monitoring and evaluation 6. improve the effectiveness of cancer control in New Zealand through research and surveillance. The New Zealand Cancer Control Strategy Action Plan outlines how the objectives of the Strategy will be achieved by means of specific objectives and individual actions. The Cancer Control Council of New Zealand was appointed in 2005, under Section 11 of the New Zealand Public Health and Disability Act 2000, to provide an independent, sustainable focus for cancer control. The Council's key objective is to lead the sector to successfully implement the New Zealand Cancer Control Strategy. The Council has five key tasks, as outlined in its terms of reference: monitor and review implementation of the Cancer Control Strategy provide independent strategic advice to the Minister of Health, the Director-General of Health, DHBs and NGOs on matters related to cancer control foster collaboration and co-operation between bodies involved in cancer control foster and support best practice in, and an evidence-based approach to, improvements in the effectiveness of cancer control establish and maintain linkages with overseas cancer control agencies. This evaluation and monitoring framework describes the Council s approach to the first key task. Evaluation and monitoring of the Cancer Control Strategy, Action Plan and cancer control in New Zealand more generally are a core part of the Council s annual work programme. They will provide the basis for its annual report and inform its independent strategic advice to the Minister of Health. Cancer Control Council Evaluation and Monitoring Framework 1

4 The Council s principles for evaluation and monitoring are: being independent of providers and funders of services as far as possible using available data sources wherever possible, to maximise efficiency, reduce duplication and control compliance burden providing timely measurement of progress and outcomes of the implementation of the Cancer Control Strategy. 3. Clarification of key tasks The Council's key tasks are to evaluate, monitor and review implementation of the Cancer Control Strategy. These terms require some clarification, as they fall within a wide range of evaluation activities. Evaluation The Cancer Control Strategy Action Plan states that: Evaluation is required to determine whether the Action Plan is achieving its overall purpose of reducing the incidence and impact of cancer and reducing inequalities. It is also required to ensure that the various actions prescribed in the plan are meeting their expected outcomes, and that the milestones for each action are being achieved (Cancer Control Taskforce 2005). Evaluation implies judgment by one body of another. To ensure and maintain co-operation, the purposes of any evaluation project and the proposed reportback mechanisms should be clarified well in advance, in consultation with stakeholders. Evaluation refers to a wide range of activities, which can include formative, process, output and outcome evaluations of programmes. Formative evaluation refers to an evaluation in the early stages of implementation, where the evaluation itself contributes to implementation. Process evaluation refers to the evaluation of systems and processes once a programme is up and running. Output evaluation refers to evaluating those things that result from the execution of a programme, but might not be termed outcomes at this stage. 1 Outcome evaluation, sometimes referred to as summative evaluation, assesses whether the outcomes or results of a programme are consistent with the desired outcomes. An outcome evaluation study asks questions about whether the programme worked, and if so, how well and for whom it worked. A wider evaluation perspective offers the ability to influence and improve programme implementation before changes in outputs and outcomes can be detected. 1 The differentiation between outputs and outcomes is not always clear. As an example, the uptake of fresh fruit in schools might be thought of as an output and, to use an extreme example, changes in colorectal cancer rates in 50 years time might be considered the outcome of interest. Cancer Control Council Evaluation and Monitoring Framework 2

5 Monitoring Monitoring and review are particular activities that contribute to overall evaluation. Monitoring can be seen as assessing progress against agreed performance indicators and milestones. The Cancer Control Strategy Action Plan provides the structure for this monitoring: Monitoring overall progress towards achieving the purposes of the Strategy (reducing incidence, impact and inequalities) will be an important role of the Cancer Control Council and involve routine data collection and analysis by the New Zealand Health Information Service. This will require improvements in the use, efficiency and scope of national data collection and reporting (Cancer Control Taskforce 2005). Monitoring implies ongoing collection and analysis of data. Review Review implies comprehensive consideration of monitoring and evaluation activities, best practice and literature. It is a longer term action, and will... provide a basis for any major changes in the implementation plan. Periodic review will help to ensure that actions are achieving the desired effects. New objectives can be added or substituted as research identifies more effective ways of reducing both the incidence and impact of cancer and health inequalities (Cancer Control Taskforce 2005). 4. International evaluation and monitoring frameworks Several countries have had comprehensive cancer control strategies in place for up to 10 years, eg the Scandinavian countries making up the Nordic Cancer Union, Australia, the United States and the United Kingdom. However, the evaluation and monitoring of comprehensive cancer control strategies is a relatively recent concept internationally. For example, a recent report from Australia refers to little or no evaluation of uptake or impact of cancer control plans (Anderiesz et al 2006). In the United Kingdom, the National Health Service (NHS) published a Three Year Progress Report on the NHS Cancer Plan (Department of Health 2003). This report provides a useful and easy to read update on the progress of a national cancer control strategy. It also demonstrates the utility of monitoring cancer mortality rates even after three years, although whether progress can be attributed to a three-year plan is uncertain. However, this progress report is superficial in some respects. For example, it is very positive overall, which may reflect selective reporting. It also does not convey lessons learned and things that did not go well. Medline and Google searches indicate a United States-based body of work on cancer control frameworks. The National Center for Chronic Disease Prevention and Health Promotion (CDC) website provides a useful overview of Cancer Prevention and Control including evaluation. 2 A recent (2005) supplement to the journal Cancer Causes and 2 Cancer Control Council Evaluation and Monitoring Framework 3

6 Control is dedicated to cancer control implementation and evaluation, including a paper focused on evaluation (Rochester et al 2005). A common theme in this US-based work is the relative infancy of the evaluation of comprehensive cancer control programmes and therefore the need to draw on generic evaluation frameworks often associated with health promotion (Centers for Disease Control and Prevention 1999; Green and Kreuter 1999). Another common theme is that of scale; many evaluations are of single projects rather than comprehensive cancer control programmes. Rochester et al provide a useful evaluation framework for cancer control, based on a generic evaluation framework. They describe the following six steps: 1. Engaging stakeholders 2. Describing the programme 3. Focusing the evaluation 4. Gathering credible evidence 5. Justifying conclusions 6. Ensuring use and sharing lessons learned These six steps have been used to structure the Cancer Control Council s evaluation and monitoring framework. 5. Cancer Control Council Evaluation and Monitoring Framework 5.1. Engaging stakeholders The New Zealand Cancer Control Strategy involves multiple stakeholders, many of whom are actively involved in the implementation of the Strategy and Action Plan. The purpose of any evaluation is to improve a programme s implementation and effectiveness. This will undoubtedly involve some recommended changes in stakeholder practice and therefore stakeholders are a key audience for the evaluation. Having the agreement of these stakeholders to the evaluation questions and monitoring framework is therefore essential. The Council can assume that there is already some implicit agreement with key stakeholders by way of the Action Plan. But it is likely that a more specific engagement with the Council's evaluation and monitoring will be needed. Another key task of the Council, as stated in its terms of reference, is to foster collaboration and co-operation between bodies involved in cancer control. The Cancer Control Collaborative, to be established by the Council, will help in engaging and informing stakeholders. It should also provide a useful mechanism for feeding into, and then disseminating, the analysis of the Council s evaluation and monitoring processes Describing the programme For the purposes of evaluation, a programme can be the entire cancer control programme itself or specific initiatives or interventions within the plan. However the programme is defined, evaluation will benefit from a clear, succinct description. This ensures all concerned share an appropriate level of understanding about the cancer control programme s objectives, implementation strategies and intended effects. This understanding will foster strategic thinking about the implementation and help focus the evaluation through common understandings. Cancer Control Council Evaluation and Monitoring Framework 4

7 The description should include aspects of the programme that support individual and community actions, the group or groups that need to take action to accomplish the intended outcomes, specific actions that should be taken and clearly-defined measurable outcomes the programme is trying to achieve. It should also incorporate important relationships between activities and outcomes, and describe unique features of comprehensive cancer control such as partnerships, the infrastructure needed to ensure joint action and activities that may influence individuals, the community and region. In the New Zealand context, the Cancer Control Strategy and Action Plan form the basis for this description and provide a logical understanding of pathways Focusing the evaluation To focus the evaluation, it is necessary to identify the most appropriate questions to ask. Deciding the mix of questions for a specific evaluation is a balancing act between utility, questions that will produce useful results, and feasibility, questions that can be answered at or by a specific time. Three starter questions can help to identify the most useful or feasible questions for the evaluation: 1. What is the stated purpose of the evaluation? 2. Who are the intended users of the results? 3. How will these results be used? It is anticipated that the mix of questions and issues in an evaluation should change and evolve as programmes mature and needs vary (Rochester et al 2005). It is important to maintain strong lines of communication and consultation with key stakeholders in order to decide or adapt the focus and resolve differences as to where that focus should be. It is useful to break down the Council's evaluation role in monitoring and reviewing implementation of the Cancer Control Strategy into three levels: 1. Strategy level 2. Action Plan level 3. Individual action level Strategy level Review at Strategy level describes the Council s role in the five-yearly process of undertaking a global review of progress with the Strategy and generating the next Action Plan (for ). This review process will also involve independent external and international input. The Council will plan for such a review to start in 2008, considering that the present Action Plan for took 18 months to develop. Review at Strategy level also describes the Council s role in monitoring cancer control in New Zealand more generally. This big picture review is not limited to the specific actions listed in the Action Plan. It refers to the largely quantitative exercise of determining trends in a range of cancer-relevant indicators, including monitoring incidence, mortality, risk factors, Cancer Control Council Evaluation and Monitoring Framework 5

8 health service provision and treatment outcomes. Account will also need to be taken of other strategies in the health sector, eg cardiovascular disease risk factors overlap with cancer risk factors, and funding of the Strategy. Action Plan level Monitoring at Action Plan level describes the Council s role in generating data and analysis that will inform the five-yearly review and allow the Council to report annually to the Minister of Health and stakeholders on the progress and effectiveness of the Action Plan. Monitoring at Action Plan level is a form of process evaluation. Its key purpose is to provide timely input that can improve how the Strategy and Action Plan are being implemented. To be feasible it is critical to select only a few high priority questions or issues each year. Selecting these questions or issues will involve consultation with key stakeholders, including the Minister and Ministry of Health, DHBs, NGOs and other organisations as appropriate. Individual action level Monitoring at individual action level describes the Council s role in measuring the progress of the specific actions in the Action Plan. The Action Plan is divided into objectives, outcomes and specific actions which address the six key goals set out in the Strategy. The Action Plan comprises 112 outcomes and approximately 240 specific actions, which range from establishing national standards, guidelines and working groups to implementing programmes and undertaking research. Initially, given the scale of the Action Plan, the focus of this monitoring will be on those actions already being implemented. It is however also important to comment on actions with no or inadequate progress. As the evaluation and monitoring framework develops, increasing emphasis will be placed on routine monitoring by the Council. This will be particularly important in ensuring that the monitoring does not focus exclusively on Ministry of Health-led actions, but also those actions to be undertaken by other key stakeholders who contribute to cancer control. These include NGOs, primary and public health providers, funders, researchers, professional groups, tertiary institutes and community and support groups, including Maori and Pacific organisations Gathering credible evidence The Action Plan recognises that monitoring overall progress towards achieving the goals of the Cancer Control Strategy will require improvements in the use, efficiency and scope of national data collection and reporting. There will be a role for the Cancer Control Council in advocating for improved data collection and statistical precision, which is essential for it to carry out its evaluations and monitoring activities. In addition it must have independent data gathering and analytical capacity to independently explore actions of particular interest and provide additional evidence for its strategic advice. To avoid duplication, the Council will investigate areas where the Ministry of Health collects Cancer Control Council Evaluation and Monitoring Framework 6

9 data. This will call for co-ordination with the various Ministry of Health-based groups involved in the implementation of the Strategy, eg the Cancer Control Implementation Steering Group, New Zealand Cancer Treatment Working Party and its working groups, and the Principal Advisor Cancer Control, and collecting data, eg National Screening Unit, Public Health Intelligence and New Zealand Health Information Service. That said, the Council s annual and other evaluation and monitoring reports must be independent of these agencies and collate and interpret all evidence in one document, even if other parties are responsible for most of the evaluation and monitoring activity. The Council must also find a means of accessing data for cancer control activities in NGOs and the community at large. Ideally the Cancer Control Collaborative, once established, will help in developing an information base and disseminating information Justifying conclusions Conclusions are justified when there is a clear linkage between the questions posed, the design chosen, data reviewed and results presented through a process that allows a broad review and acceptance of findings, interpretations and conclusions. When stakeholders and users are engaged in interpreting findings and developing conclusions and next steps, the rights and values of individuals and communities are honoured and respected, and the results are more relevant to their needs and interests. The following are important issues: success of the effort, ie have measures of success been accomplished? alternative explanations for the results achieved and the outcomes accomplished significance of the results in terms of action, eg where is attention needed, are midcourse corrections required, what is not working? are actions recommended that are consistent with conclusions (Rochester et al 2005)? 5.6. Ensuring use and lessons learned To ensure this evaluation and monitoring framework is used, it is important to plan for dissemination of information at an early stage. Consulting key stakeholders on the framework, with particular focus on the priority evaluation questions will set the scene for feedback and use later on. Without the involvement of stakeholders, the evaluation could overlook key elements of the programme, which could lead to its findings being ignored or resisted (WHO 2002). The Council will play a leading role in promoting, organising and contributing to gatherings that disseminate information. The Cancer Control Collaborative will also play an important role in ensuring the use of this framework. Cancer Control Council Evaluation and Monitoring Framework 7

10 6. References Anderiesz C, Elwood M, Hill DJ Cancer control policy in Australia. Australia and New Zealand Health Policy 3: Cancer Control Taskforce The New Zealand Cancer Control Strategy Action Plan Wellington: Ministry of Health. Centers for Disease Control and Prevention Framework for programme evaluation in public health. Morbidity and Mortality Weekly Report 48 (No. RR-11). Department of Health The NHS Cancer Plan. Three Year Progress Report - Maintaining the Momentum. London: Department of Health. Green L, Kreuter M Health Promotion Planning: An Education and Ecological Approach. Mountain View, CA: Mayfield Publishing Company. Minister of Health The New Zealand Cancer Control Strategy. Wellington: Ministry of Health and the New Zealand Cancer Control Trust. Rochester P, Chapel T, Black B, Bucher J, Housemann R The evaluation of comprehensive cancer control efforts: useful techniques and unique requirements. Cancer Causes and Control 16 (Suppl. 1): WHO National Cancer Control Programmes: Policies and Managerial Guidelines (2nd Ed.). Geneva: World Health Organization. Cancer Control Council Evaluation and Monitoring Framework 8