To view an archived recording of this presentation please click the following link:

Transcription

1 To view an archived recording of this presentation please click the following link: Please scroll down this file to view a copy of the slides from the session.

2 Helpful tips when viewing the recording: The default presentation format includes showing the event index. To close the events index, please click on the following icon and hit close If you prefer to view the presentation in full screen mode, please click on the following icon in the top right hand corner of the share screen PublicHealthOntario.ca

3 Evaluation of Toronto's pilot EMS surveillance system used during the 2015 Pan/Parapan Am Games Amanda Shane PHO Rounds: Epidemiology Thursday, August 18, 2016

4 Learning objectives 1. Describe Toronto s EMS surveillance system 2. Explain the strengths and weaknesses of this system, and dispatch data in general 3. Identify important considerations to make when developing a surveillance system using dispatch data 2

5 Outline Background Implementation Evaluation methods Results Recommendations 3

6 Background The 2015 Pan/Parapan Am Games were a multi-jurisdictional mass gathering sporting event occurred across 10 health units provincially-coordinated surveillance plan/response TPH developed a local surveillance plan for the event, informed by provincial surveillance recommendations Goal of the event surveillance was the early detection of public health threats in order to carry-out public health action 4

7 Context for project Planning and preparations started in 2013 Test run with World Pride in June 2014 TPH involved in Provincial PPAG Surveillance Operationalization WG TPH s planning included a gap analysis of surveillance data sources 5

8 TPH s PPAG data sources 6

9 Surveillance gap Location-based data minimal Not timely in current form (iphis exposure data) Important during mass events Geography plays a key role Large # of people aggregating at small # of places Timely geo-based data can facilitate public health intervention for highly mobile population 7

10 EMS data for surveillance what we know TPH previously explored use of dispatch data World Youth Day (2002) Heat-related illness WorldPride (2014) Literature on usefulness of EMS systems limited (n=4 studies) and mixed Search conducted by PIDAC (2012) and updated by TPH (2015) 3 papers focused on ILI; 1 on CO poisoning Some other HUs have explored/are interested in exploring use of data for PH surveillance 8

11 Opportunity knocked calls are geocoded and a possible source of location-based health data Field Epidemiologist joined TPH in September 2014 for two year placement EMS partnered with us for WP and was open to continued collaboration EMS surveillance system pilot for the PPAG 9

12 Purpose of the EMS project EMS project Develop and pilot a new surveillance system during the PPAG that fills a known surveillance gap; and through evaluation, determine whether and how it adds value to TPH s existing complement of surveillance systems 10

13 Background Implementation Evaluation methods Results Recommendations 11

14 Development process 12

15 System description Syndromic surveillance system Relies on real-time access to web-based platform, FirstWatch FirstWatch is used by the Toronto Paramedic Services to capture dispatch data Near real-time data collection at several points in time Purpose of the EMS surveillance system: to enable early detection of PH threats during mass gathering events in Toronto by identifying spatial and temporal clusters of PH syndromes associated with mass gathering events in order to prompt further public health investigation 13

16 14

17 15

18 System description 16

19 System description 17

20 System description 18

21 System description Both communicable and non-communicable conditions (n=5 syndromes) Heat-related illness, physical injury and overdose/substance abuse Respiratory illness and gastrointestinal illness Two types of signals City-wide (signal algorithm) Venue-based (sentinel) 19

22 City-wide alerts 20

23 Venue-based alert 21

24 Investigation and reporting Surveillance procedure developed Epidemiologist logged into the system twice/day during PPAG (7 or 8*am and 2pm) * weekends 22

25 Refer to program/ims 23

26 Evaluation purpose and scope Purpose To determine whether the system added value to existing TPH surveillance activities To make recommendations to inform future use of the system Scope Based on data collected during: Development period (January to June 28, 2015) Pilot period (June 29 to August 28, 2015) Syndrome validity at an individual level out of scope 24

27 Learning objectives check-in 1. Describe Toronto s EMS surveillance system 2. Explain the strengths and weaknesses of this system, and dispatch data in general 3. Identify important considerations to make when developing a surveillance system using dispatch data 25

28 Background Implementation Evaluation methods Results Recommendations 26

29 Methods US CDC s Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks Assessed 10 attributes Objective 1: system performance Timeliness Validity Data quality Usefulness Objective 2: user experience Flexibility Acceptability Portability Stability Simplicity Cost 27

30 Methods Mixed methods evaluation approach Prospective data collection Focus group (facilitated by Meg Gassanov) Survey Evaluation questions Provided in hand-out and detailed in final report Evaluation period Development period (January 1 to June 28, 2015) Pilot period (June 29 to August 28, 2015) 28

31 Background Implementation Evaluation methods Results Recommendations 29

32

33 Number of signals Results 5 city-wide signals 98 venue-based signals HRI Injury Overdose Respiratory Gastro 0 HRI Injury Overdose Respiratory Gastro 31

34 Key findings of the evaluation 32

35 Attributes that performed well Data quality (completeness) All calls had valid geographic coordinates Completeness for age and sex varied by syndrome Variable Gastrointestinal Heat-related Overdose/sub- (n [%]) illness illness stance abuse (n=29) (n=339) (n=20) Valid geographic coordinates 29 (100) 339 (100) 20 (100) Gender 29 (100) 280 (83) 18 (90) Age 29 (100) 249 (74) 16 (80) 33

36 Attributes that performed well Flexibility FirstWatch application is flexible Development period: 8 days for change requests Pilot period: 1 day for change requests Changes were standard for their system No additional time/resources required to make the changes This turnaround time was typical 34

37 Attributes that performed well Acceptability # data requests TPS met within deadline (30 mins): 0/1 Focus group and survey results suggest the system was acceptable to TPH users, TPS 35

38 Attributes that performed well Stability System operation consistent throughout pilot period Only risks to stability are reliance on internet connection and TPS infrastructure FirstWatch has mechanisms in place to ensure stable system access One primary and three live back-ups with auto-failover and Five Nines uptime for mission critical features (= availability % of the time; downtime of 6.05 seconds per week) TPH had contingency plans in place to manage connection issues 36

39 Attributes that performed well Cost No additional cost to TPS or FirstWatch Direct cost to TPH low (~0.1 FTE to manage system) 37

40 Attributes with uncertain results Timeliness Syndrome (no. alerts) Measure EMS System (hh:mm) ACES (hh:mm) Heat-related illness T 1 Mean: 11:07 No comparator (n=3) T 2 N/A available Gastrointestinal illness T 1 12:44 8:51 (n=1) T 2 N/A N/A Overdose/substance T 1 0:38 7:03 abuse (n=1) T 2 N/A N/A Overall (median) T 1 10:30 7:26 T 2 N/A N/A 38

41 Attributes with uncertain results Validity Could only be measured for 2 syndromes Specificity and PVN performed well Small sample size (n=2 alerts); interpret with caution Measure Gastrointestinal illness Overdose/substance abuse Sensitivity (%) 0/2 = 0% 1/10 = 10% Specificity (%) 58/59 = 98% 50/50 = 100% Predictive value positive (%) 0/1 = 0% 1/1 = 100% Predictive value negative (%) 58/60 = 97% 50/60 = 83% 39

42 Attributes with uncertain results Simplicity What is required to maintain the system? 1. Time required to modify the system (# meetings x duration) = 5 hours 2. Time required for training: 1 hour per epi Ease of use (platform and epi investigation) Intuitive layout and navigation Investigating venue-based alerts was simple City-wide alerts were time-consuming and limited data were available to characterize them 40

43 Attributes with uncertain results Portability FirstWatch has been used for PH surveillance in other jurisdictions which suggests portability Difficult to implement an EMS system with similar resources and upfront cost without the infrastructure in place 41

44 Attributes with uncertain results Usefulness Public health action Gastro (n=1) Gastro (venue) HRI (n=3) Overdose (n=1) (n=1) Review of 1 (100) 1 (100) 3 (100) 1 (100) complementary systems Data request from TPS 0 (0) 0 (0) 1 (33) 0 (0) Include in PPAG 0 (0) 0 (0) 1 (33) 1 (100) surveillance report Forward to program area 0 (0) 0 (0) 0 (0) 0 (0) Identified alerts that were not captured by other systems 42

45 Attributes with uncertain results Usefulness (cont d) focus group Value in venue-based alerts Only system that provide geo-specific info Increased confidence that nothing was missed Timely source of information May be more useful during MG than routine scenarios Included syndromes could be revised 43

46 Key findings of the evaluation 44

47 Learning objectives check-in 1. Describe Toronto s EMS surveillance system 2. Explain the strengths and weaknesses of this system, and dispatch data in general 3. Identify important considerations to make when developing a surveillance system using dispatch data 45

48 Limitations 46

49 Limitations Pilot period was short opportunity for epis to investigate alerts/interact with system; may impact user experience results Pilot period did not coincide with any major public health events Validity assessment difficult Assessment of relative usefulness difficult Complementary and gold standard systems not evaluated dispatch data vs. FirstWatch system 47

50 Background Implementation Evaluation methods Results Recommendations 48

51 Recommendations: for TPH pilot system Discontinue use in current form Use the system during future mass events Explore use of EMS data for retrospective analyses Validate syndromes Disseminate system design and evaluation results 49

52 Recommendations: for surveillance using dispatch data (i.e. lessons learned) Clearly stated objectives at both event and system levels + response component resulting from signal Sentinel, location-based alerting was key Level of activity was manageable Use of existing platform and syndromes necessary to meet timelines Training, buy-in required 50

53 Summary TPH successfully implemented a new surveillance system during the PPAG that aimed to fill a surveillance gap Evaluation suggests system may be useful for venue-based surveillance during mass gathering events Opportunity to collaborate with key City partner Next steps Implement recommendations Publish evaluation results 51

54 Learning objectives review 1. Describe Toronto s EMS surveillance system 2. Explain the strengths and weaknesses of this system, and dispatch data in general 3. Identify important considerations to make when developing a surveillance system using dispatch data 52

55 Acknowledgements Effie Gournis Anne Arthur Les Shulman The CDSU team Chris Olynyk François-William Tremblay CFEP field epis 53

56 Amanda Shane A/Senior Epidemiologist Centre for Immunization and Respiratory Infections Diseases Public Health Agency of Canada Effie Gournis Associate Director Integration, Information and Surveillance Toronto Public Health 54

57 References Berger M, Shiau R and JM Weintraub (2006). Review of syndromic surveillance: implications for waterborne disease detection. J Epidemiol Community Health. 60(6): FirstWatch: What we do. Available from: [Accessed: ]. Ontario Agency for Health Protection and Promotion, Provincial Infectious Diseases Advisory Committee. Syndromic Surveillance Discussion Paper. Toronto, ON: Queen s Printer for Ontario; Bassil KL, Henry B, Rea E, Varia M and D Cole (2005). Public Health Surveillance for World Youth Day --- Toronto, Canada, MMWR Abstracts. 54(Suppl);183. Images: all images included in this presentation were labelled for non-commercial re-use 55