Evaluation Tool Reviewing and evaluating your health and social care worker flu campaign The aim of this guide is to help you reflect and evaluate this year s health and/or social care worker influenza vaccination campaign, to identify and address any areas that may require further work and also to identify areas of success. This reflection can help in planning future campaigns. Why evaluate your campaign? Evaluation is a vital element of your flu campaign. Well planned evaluation helps identify what worked and what didn t work so well. This will help you identify good practice to share, and also inform planning for a stronger campaign next year. Sometimes, one small change can have a big impact and evaluation can help identify that change and the size of the impact. Evaluation of your local flu campaign should aim to: Gather information from your team - this demonstrates you value their opinion, and can provide a valuable insight Identify key stakeholders Gather information from key stakeholders - this should include patients/service users and can be enlightening I dentify and highlight good work that has been carried out by your team(s) Identify interventions that seem to have been successful and also consider if they can be improved Identify interventions that were less successful and consider if they can be improved Identify awareness raising resources, and consider if they have delivered the messages as planned to the target group Identify opportunities to strengthen the campaign going forward Inform planning Estimate resources needed for a strong campaign next season The information may be gathered in a variety of ways. A campaign assessment checklist may help when collecting this information (appendix 1). The information you gather as part of your evaluation can be useful in planning your next campaign and also to inform business cases if required.
Evaluation Tool Getting started Consider: What you want to measure What/when/how are you going to measure Who you are going to ask to contribute How you will share your findings Some suggestions of potential outputs/outcomes would be: Vaccine uptake by clinical area, directorate and professional group Number of and percentage of front line staff protected vs unprotected Vaccine uptake in areas with a peer vaccinator vs without one Some questions that may help you identify valuable detail behind the figures might include: Have we done anything different this year? If yes: What did we do this year that was different to previous years? Did this appear to have any effect? If yes, what effects have been identified (positive and negative)? If no; why not? What feedback have you had from your staff? What are the main points arising? What feedback have you had from service users/ patients? What are the main points arising? A campaign assessment checklist may help when collecting this information (appendix 1). Examples of staff questionnaires to help gather information about staff opinion are included in appendix 2 and appendix 3. How you gather this information is also a consideration. You could use a traditional questionnaire (appendix 3), hold focus groups, or be more innovative and creative with gathering information (appendix 4). Displaying a staff opinion dartboard (appendix 4) in areas where staff congregate such as the canteen, staff room or staff changing facilities is another way to collect staff opinion. What to do with the information you collect The information collected should be analysed and reflected on. It will become the building blocks for the following year s campaign and to develop areas that may require investment. Separate your findings into individual departments, directorates, staff groups and/or sites as this will help identify differences and may direct you to areas of good practice as well as help highlight areas where more support may be needed next season. Using your findings It can be helpful to use this information to encourage competition between teams, staff groups, and departments within your organisation and also to benchmark against other similar organisations. You can learn from those with higher vaccine uptake rates, identifying and sharing good practice. Another way to sort your findings is into themes, as some issues may be organisation wide. Use the information you have gathered in this evaluation to inform your end of flu season report. Remember to include positive and negative, qualitative and quantitative data and reflect on how this can influence your recommendations on planning a stronger campaign.
Appendix 1. A campaign assessment tool Initial Questions Flu vaccine uptake % in all staff Flu vaccine uptake % in front line* staff Are any staff groups lower than others? Do we know why? Is the policy to vaccinate all staff or front line staff only How was the number of front line staff calculated? Date planning the campaign started Do you have a documented flu plan? Does the organisation have an aspirational vaccine uptake target? If so - what is it? Was the planning robust and complete? How was the flu vaccine promoted to staff? Email, social media etc Were communications colleagues included in sharing messages? 2015/16 2016/17 Action 2017/18 campaign * A definition of front line staff can be found in The Green Book (Immunisation against infectious disease) Chapter 19 influenza.https://www.gov.uk/government/uploads/ system/uploads/attachment_data/file/456568/2904394_green_book_chapter_19_v10_0.pdf
Who was involved in planning the campaign? Staff-side representation 2015/16 2016/17 Action 2017/18 campaign Occupational health Communications Volunteers within the organisation Estates and facilities Board-level representation Human resources Infection control Peer vaccinators IT Other
How are challenges and successes identified? Via staff? 2015/16 2016/17 Action 2017/18 campaign Through the flu team? At the Board? Using evaluation forms? Using social media polls? Other Tackling the challenges What challenges have been identified? What are we doing, as a team, about these challenges?
Acknowledging and sharing successes What successes were identified? 2015/16 2016/17 Action 2017/18 campaign How were successes celebrated and/or shared? Incentives Was there an incentive scheme? If yes - describe If yes - how was impact measured and how is this being evaluated? Would you recommend an incentive scheme for next year? (describe)
How was vaccination provided for staff? Arranged clinics in OH department Arranged OH clinics in other venues Drop in OH clinics 2015/16 2016/17 Action 2017/18 campaign Roving clinics Peer vaccinators On request Through line managers and at key gatherings (such as mandatory training) Other (describe)
If you have peer vaccinators... Who are the peer vaccinators? 2015/16 2016/17 Action 2017/18 campaign How is impact measured and did they have an impact? Did staff know who the peer vaccinators were? If not, how could they be identified next year? In Summary... What went well? What didn t go so well? What lessons were learnt? What will be done differently next season?
Appendix 2. Staff Flu Campaign Template Questionnaire 1. Did you have a flu vaccination this year? Yes/No 2. If yes, why? (tick all that apply). If no - go to question 4. To protect myself from getting flu To protect against spreading flu To protect my family from getting flu To protect my patients from getting flu I get the vaccination every year - I don t know why I felt compelled Other (please describe) 3. Where did you receive your flu vaccination? (tick one only) By a peer vaccinator In department/office/ward In occupational health department by appointment In occupational health department at a drop in clinic In a staff area (canteen etc) At my GP surgery At a community pharmacy Other (please describe)
4. If you answered NO to question 1, why didn t you have a flu vaccine this year? (tick all that apply) I wasn t offered a flu vaccination I didn t know I was entitled to it I didn t know how to get it I was offered it but the time/date was inconvenient I think I might be allergic to one of its components I didn t want to experience side effects from the vaccine I don t believe the vaccine is effective I don t believe the vaccine is safe Other (please describe) 5. How likely are you to have a flu vaccination next year? (tick one only) I will definitely have it I will probably have it I might have it I probably won t have it I definitely won t have it Undecided
6. How did you hear about the flu vaccination this year? Colleague in my department/team Occupational Health Department News item on the intranet (NHS Wales) Information on the internet including social media such as facebook, twitter etc Email I didn t this questionnaire is the first I ve heard of it Other (please describe) 7. What would you do differently to increase flu vaccine uptake among your colleagues/staff? Comment
Appendix 3. Brief staff questionnaire NHS staff are key to patient safety and we would like to ask you about the flu vaccination 1. Have you had or are you going to have your flu vaccination? Yes No Not sure 2. Please tell us why you made this decision? 3. If you are not going to have your vaccination, or are not sure, what would encourage you to have it?
Appendix 4. Staff opinion dartboard tool For a quick and visual way for staff to give feedback you could use the dartboard tool. On a piece of flipchart paper create the diagram below and display it in areas with high staff footfall like the canteen. The heading can be changed to gather information to meet your evaluation aims. Using a pen or small sticky dots, staff can place a mark within each of the four areas of the dart board. The closer the mark is to the centre, the better they rate that aspect of the campaign. It might be an interesting exercise to place several of these dartboards in different departments to gather a range of information. This feedback can be used to help tailor next season s campaign. Missed the mark Vaccine availability Met expectations Clinic location Exceeded expectations Flu campaign communications Peer vaccinator knowledge PHW030 04.17