Performance of a syndromic system for influenza based on the activity of general practitioners, France

Transcription

1 Journal of Public Health Vol. 31, No. 2, pp doi: /pubmed/fdp020 Advance Access Publication 5 March 2009 Performance of a syndromic system for influenza based on the activity of general practitioners, France G. Gault 1, S. Larrieu 1, C. Durand 2, L. Josseran 2, B. Jouves 3, L. Filleul 1 1 Regional Epidemiology Unit Aquitaine, Espace Rodesse, 103 bis rue de Belleville, BP 922, Bordeaux Cedex, France 2 Institute of Public Health Surveillance, Saint-Maurice, France 3 SOS Médecins Bordeaux, Bordeaux, France Address correspondence to G. Gault, gaelle.gault@sante.gouv.fr ABSTRACT Background In France, as in other industrialized countries, syndromic surveillance systems for the early detection of illnesses have proliferated, but few validation studies on these systems performances exist. In Bordeaux, a south-western city in France, a system using a network of general practitioners house calls, such as SOS Médecins, provided local health data used to guide health service response, in particular in case of flu-like pandemic. We explored the capacity of SOS Médecins system to identify and follow influenza outbreaks using data from the Sentinel network, considered as being a gold standard for tracking seasonal influenza in France. Methods Data from SOS Médecins were analysed and compared with data from the Sentinel network. The sensitivity and specificity of SOS Médecins system were evaluated for different simulated thresholds. Results A relationship between the number of visits for influenza from SOS Médecins and the number of influenza cases from the Sentinel network was observed; data from the two systems were highly correlated. We showed the capacity of SOS Médecins system to identify outbreaks with a sensitivity and specificity of 93%. Conclusion The sensitivity and specificity of SOS Médecins for early outbreak detection showed the value of these data in monitoring influenza activity. Keywords general practitioners, influenza, sensitivity, syndromic surveillance Introduction Surveillance systems for the early detection of outbreaks related to biologic terrorism and infectious disease have been proliferating for a few years, 1 in order to detect events that can improve morbidity and mortality. The term syndromic surveillance refers to systems providing data on the health condition of populations, such as emergency activity, passive reporting illnesses or chief complaints. Such surveillance systems are useful for following the size and spread of outbreaks, based on real-time health data, providing immediate analysis and feedback to those charged with investigating and following-up potential outbreaks. 2 5 This approach can also contribute to earlier identification when the frequency of a disease increases, usually a few days before the laboratory test results are available. 6,7 This early detection of unusual health events is then reported to public health agencies to help decision-making and set up appropriate rapid responses. 3,8 Thus, with the permanent threat of a flu-like pandemic, prevention and action plans elaborated by the health authorities require the implementation of surveillance systems to follow real-time trends of the situation and quantify the sanitary impact. 9 In France, the Sentinel network (French Communicable Diseases Computer Network), composed of more than 1200 doctors spread over the whole territory, is considered as the reference system for infectious diseases monitoring at the national level. 10 At the regional level, however, additional surveillance systems, based on local data intended to enhance analyses on the local sanitary situation, are necessary. Such G. Gault, Epidemiologist S. Larrieu, Epidemiologist C. Durand, Epidemiologist L. Josseran, Epidemiologist B. Jouves, General Practitioner L. Filleul, Epidemiologist 286 # The Author 2009, Published by Oxford University Press on behalf of Faculty of Public Health. All rights reserved.

2 PERFORMANCE OF A SYNDROMIC SYSTEM FOR INFLUENZA 287 systems were set up following the August 2003 heat wave in France. 11 The French Institute for Public Health Surveillance (Institut de veille sanitaire, InVS) developed a syndromic surveillance system based on the activity of emergency health professionals through an organization called SOS Médecins, which is the first emergency network of general practitioners (GPs) and healthcare in France. Its major objectives are to follow-up morbidity and mortality in the population and detect unusual health events and outbreaks. Influenza, which is also followed in France by a Sentinel network, is one of the diseases monitored through SOS Médecins. 12 The Sentinel network is commonly considered as a reference in terms of tracking seasonal influenza, 13 giving the opportunity to evaluate the value of the syndromic system based on the data from SOS Médecins. In addition to traditional surveillance systems based on the activity of healthcare professionals, such as clinicians and microbiologists, syndromic surveillance systems must be considered as complementary. Monitoring based only on syndromic surveillance systems has proved to be insufficient, even if their potential value in detecting naturally occurring illnesses (influenza, gastroenteritis, etc.) have been described in the scientific literature. 14 Furthermore, little is known of the capacity of such systems to detect outbreaks in terms of sensitivity and specificity. 1,15,16 More evidence-based research on the performance and added value of non-specific and new sources of health information is needed. 17 According to the Centers for Disease Control recommendations, 18 it is necessary to measure the performance of public health surveillance systems in terms of outbreak detection if the relative value of different approaches is to be established. Likewise, information needed to improve their efficacy for detecting outbreaks at the earliest stages has to be provided. To assess outbreak detection capacities of a system, the measurement of outbreak detection validity and timeliness is necessary. Part of this evaluation is presented here by determining the sensitivity and the specificity of a syndromic system for influenza, based on the activity of GPs. Our aim was to evaluate SOS Médecins capacity to identify and follow influenza outbreaks, using data from the Sentinel network and to establish whether data from the SOS Médecins was useful as an efficient regional public health strategy for early warning and following influenza activity. Methods We used a set of data based on the SOS Médecins and the Sentinel network, known as the gold standard to identify influenza outbreaks. Different thresholds were tested, based on variations in SOS Médecins activity and we constructed a receiver-operating characteristic (ROC) curve to determine the best threshold of outbreak detection with data from the SOS Médecins. Data sources SOS Médecins is a French network of GPs, spread throughout the country, responding to private house calls 24 h a day, 7daysaweek. 19 In the urban area of Bordeaux, a city in south-western France, the local team of SOS Médecins accounts for 60 GPs, who report on a daily basis and for each visit, socio-demographic and medical data coded according to the International Classification of Primary Care (ICPC-2). 20,21 The Sentinel network is composed of 1260 volunteer GPs, representing about 2% of the total number of GPs, although this percentage varies from one region to the other. 22 This national surveillance system provides clinical data for 14 health indicators, including influenza-like illness (ILI). A weekly report, called Sentiweb-Hebdo, presents the epidemiologic situation of seasonal outbreaks at the regional level together with national predictions. 23 Data analysis The study covered the period from 5 April 1999 (week 13) to 1 April 2007 (week 13), i.e. 417 weeks. The weekly numbers of visits for influenza syndromes performed by SOS Médecins Bordeaux were used as well as the number of ILI cases extrapolated in the Aquitaine region by the Sentinel network. 13 Indicators An indicator of the influenza syndrome was defined for each surveillance system for maximal comparability. For SOS Médecins, five diagnoses coded by ICPC-2 were gathered with the GPs approval: influenza (R80.02), flu-like symptoms (R80.01), fever and febrile symptoms (A03) and virosis (A77). In the Sentinel network, ILI was defined as a sudden fever (.398C or.1028f), accompanied with myalgia and respiratory symptoms. 22 Correlation analysis First, the two sets of data were compared graphically with a 4-week moving average. A correlation coefficient was then estimated between these two time series. Sensitivity and specificity analysis To estimate the capacity of SOS Médecins to identify influenza outbreaks in terms of sensitivity and specificity, we identified epidemic periods in the Aquitaine region that corresponded to the Sentinel system threshold used at the

3 288 JOURNAL OF PUBLIC HEALTH Fig. 1 Four-week moving averages of the number of visits for influenza syndromes by SOS Médecins in the area of Bordeaux and the number of influenza cases reported by the Sentinel network in the Aquitaine region, 5 April 1999 to 1 April 2007, France. The dark grey line indicates the moving average of the weekly number of visits for influenza syndromes realized by SOS Médecins Bordeaux. The light grey line indicates the moving average of the weekly number of cases extrapoled declared by the Sentinel network. national level (in number of cases for inhabitants) reported for the population in Aquitaine. 24 A week was considered as epidemic when this threshold was twice exceeded sequentially and the epidemic week was the second one overtaking the threshold. 25 These epidemic periods were then compared with those identified through SOS Médecins, simulating different thresholds to identify the best one in terms of sensitivity and specificity. A threshold corresponded to the number of weekly visits for influenza syndromes, different thresholds were tested, between 50 and 600 weekly visits for influenza syndromes. Regarding data from SOS Médecins, a week was also considered as epidemic when the threshold was twice exceeded sequentially and the epidemic week was the second one overtaking the threshold. Sensitivity and specificity were determined for each of these thresholds. An ROC curve was constructed reporting sensitivity versus (1 - specificity) and the area under the ROC curve was estimated by the trapezoidal method. Results Over the study period, from week 14 of 1999 to week 13 of 2007, cases of ILI were reported by the Sentinel network and visits for influenza syndromes, hence 7% of total activity were performed by the SOS Médecins Bordeaux GPs, with an average of 51 visits per day in winter for febrile symptoms (52.9%), influenza (30.4%), fever (13.1%), flu-like symptoms (2%) and virosis (1.6%). Figure 1 shows the weekly evolution of the number of cases identified through the two data sources with different scales, considering the SOS Médecins diagnosed cases in the department of Gironde alone, whereas the Sentinel network data concerned the whole Aquitaine region. The curves showed similar seasonal trends, with epidemic peaks at the same periods. Furthermore, these data presented a high coefficient of correlation of Over the study period, 46 weeks (11%) were identified as epidemic when using the data and the epidemic threshold defined by the Sentinel network. Assuming that these 46 weeks were actual epidemic periods and the 371 others were non-epidemic, the capacity to detect these weeks in terms of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) was determined with the SOS Médecins data according to different simulated thresholds (Table 1). ROC analysis showed that the devised classification system for determining an epidemic period had acceptable accuracy with an area under the curve of The threshold with the best performances was identified at 230 weekly visits for influenza syndromes (Fig. 2). At this threshold, the sensitivity and specificity were 93%, the system allowed detection of 93% of the epidemic weeks reported by the Sentinel network and identified 93% non-epidemics weeks, causing few false alarms. Among 27 false non-epidemics weeks, 21 were identified just before the beginning of epidemic period determined by the Sentinel network and 6 just after the end. When the weekly number of visits for influenza syndromes twice exceeded the threshold, the probability that

4 PERFORMANCE OF A SYNDROMIC SYSTEM FOR INFLUENZA 289 Table 1 Sensitivity (Se), specificity (Sp), PPV and NPV of SOS Médecins system in the detection of influenza outbreaks (gold standard: Sentinel network), 5 April 1999 to 1 April 2007, France Simulated threshold Outbreaks No outbreaks Predictive value Number of epidemic weeks identified Se (%) Number of non-epidemic weeks identified Sp (%) PPV (%) NPV (%) the second week was a real epidemic week was 61%. The NPV was 99%, when the threshold was not exceeded; the probability of not being an epidemic week was 99%. Over the study period, eight epidemic periods were identified from both the SOS Médecins and the Sentinel network. SOS Médecins could have identified these epidemics periods about 2.5 weeks before the Sentinel network. Thus, with a threshold fixed at 230 weekly visits, the durations of epidemics identified by SOS Médecins were in mean about 3 weeks longer than those identified by the Sentinel network (Table 2). Discussion Main findings This study shows the value of data from the SOS Médecins system in the monitoring of influenza activity and identification of influenza outbreaks, considering that the weekly trends of GPs visits for influenza syndromes showed Fig. 2 ROC curve of influenza syndromes indicator related to SOS Médecins. Gold standard: Sentinel network, 5 April 1999 to 1 April 2007, France. ROC space is defined by 1 - specificity and sensitivity as x and y axes, respectively, which depicts the relative trade-offs between true positive and false positive. The diagonal line divides the ROC space in areas of good or bad classification. Points above the diagonal line indicate good classification results, whereas points below the line indicate wrong results. Each point on the ROC plot represents a sensitivity/specificity pair corresponding to a particular threshold. The area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. a seasonal trend that was concordant with epidemic peaks during each winter. A relationship was observed between the number of visits for influenza from SOS Médecins and the number of influenza cases from the Sentinel network; data from both systems were highly correlated (coefficient of correlation: 0.87). Peaks of increasing visits for influenza syndromes and cases declared by the Sentinel network occurred over the same period, suggesting similar epidemic trends in the whole region. If systems with inadequate specificity may have frequent false alarms, systems with inadequate sensitivity may fail to detect health events. This could result in potential public health crises in terms of morbidity and mortality when unusual situations are not detected early enough to take effective preventive measures. 1 As shown by the good performance of data from the SOS Médecins system, this study illustrates how useful these data are in monitoring influenza activity for early outbreaks detection. With an epidemic threshold based on 230 visits for influenza syndromes per week, epidemic periods could be relatively well identified with a sensitivity and specificity of 93%. However, the epidemic threshold estimated for SOS Médecins was fixed over time, in opposition to the Sentinel network that used the Serfling method. 25 Nevertheless, the objective of this study

5 290 JOURNAL OF PUBLIC HEALTH Table 2 Epidemic period and duration from the two data sources: Sentinel network and SOS Médecins (threshold fixed at 230 weekly visits), for each season 1999/2000 to 2006/2007, France Season Sentinel network SOS Médecins Epidemic period (year-week) Duration (in weeks) Epidemic period (year-week) Duration (in weeks) 1999/ to to / to to / to to / to to / to to / to to / to to / to to was not to determine an epidemic threshold, but rather to show that the SOS Médecins was capable of identifying and following influenza outbreaks. Moreover, from 1999 to 2007, eight epidemic periods had been identified by the system based on the activity of SOS Médecins Bordeaux about 2.5 weeks before the Sentinel network detected them. It is difficult to know if this shift was due to an earlier detection of outbreaks from SOS Médecins, due to the lack of specificity of the indicator that generated false alarms or due to the different population, since SOS Médecins only concerns the Bordeaux area, where outbreaks could occur earlier. However, the fact that all our false-positive epidemic weeks occur at the very beginning or end of the epidemic periods are in favour of a better detection rather than false alarms. Cooper et al. 26 have found identical results in a study based on syndromic data, which provided advanced warning of influenza circulating in the community compared with ILI clinical incidence. What is already known on this topic? Influenza outbreaks occur each winter and are responsible for increased general practice consultation rates, hospital admissions 27 and substantial mortality. 28 The results of this study demonstrate the value of syndromic surveillance, previously validated 29 notably through a study on the accuracy of data in the Electronic Surveillance System for the Early Notification of Community-based Epidemics. 30 What does this study add? In France, surveillance of infectious disease, such as ILI, has been performed for many years through syndromic surveillance using different data sources, such as emergency departments, the Sentinel network or the Regional Group for the surveillance of influenza (Groupes régionaux d Observation de la Grippe, Grog), 31 which provide information on the dissemination of ILI virus in the community. However, the Sentinel and the Grog networks provide weekly regional data and do not monitor outbreaks at local or daily levels. Inversely, a surveillance system, such as the one developed in Bordeaux with SOS Médecins, is a realtime source of information available to follow the influenza activity locally. The capacity of SOS Médecins system to detect epidemic periods and monitor the morbidity and health impacts of influenza has to be checked. Furthermore, if an influenza pandemic involving human transmission of the virus occurred in the future, it would generate warnings in most public health services. Should this happen, GPs would need to be in the forefront of the pandemic response This syndromic system has allowed the availability of information on the spread of influenza cases for regional health authorities. The good sensitivity and specificity of this system for early detection of influenza outbreaks suggest the efficacy of these data in monitoring influenza activity at the local level. Although the choice of the threshold for the early detection of epidemic periods should be properly considered, the interest in monitoring the indicator of influenza syndromes from SOS Médecins at our local level seems obvious in the current context of a pandemic risk, in particular with regard to the control of drug stocks, and the coordination of the different local health authorities in case of widespread epidemics. Now that the value of the system has been established, further analyses could be initiated to improve thresholds, for

6 PERFORMANCE OF A SYNDROMIC SYSTEM FOR INFLUENZA 291 example integrating factors that should be taken into account to propose a threshold based on epidemic variations and inter-epidemic periods and working on a daily threshold. Indeed, for this study, we had to work on weekly data since information from the Sentinel network is given on this basis; however, data from SOS Médecins system are available and analysed on a daily basis, and working on the elaboration of a daily threshold is therefore possible. As information on the city where each visit takes place is available, one of the perspectives to improve the value of these data could be to integrate a routine spatio-temporal analysis to follow the epidemic spread. In any case, this system allowed the detection of influenza outbreaks at a local level and could also contribute to characterizing the population affected by the epidemics, since individual data are available for each visit, unlike data from the Sentinel system. To follow the epidemics, it is therefore necessary to confront the data of both systems. The advantage of the Sentinel network is to have a national and regional vision of the spread of influenza outbreaks, whereas the system based on SOS Médecins allows the identification and describe influenza activity locally, and to inform local health actors. The SOS Médecins system, in addition to mortality and hospitalization data, increases the performances of the regional syndromic surveillance system, following real-time influenza activity. This study shows that influenza syndromes data from SOS Médecins Bordeaux may be a valuable data source of information for real-time syndromic surveillance, which could prove to be a very useful tool for decisionmaking processes on a regional health and warning system. Limitations of this study In France, there is no clear consensus on the most sensitive, specific or timely data for influenza surveillance. Moreover, it was difficult to have a real reference standard for surveillance data to be compared with, since few systems provide numerical data on the sensitivity and specificity of the system. 1 Even if data from Sentinel network did not concern the department of Gironde, but the whole Aquitaine region, which constitutes a limit in our study, influenza data from both of sources (SOS Médecins and Sentinel network) were highly correlated. Syndromic surveillance is based on syndromic groupings of data; chief complaints can be grouped into syndromes, even in case of substantial variations in coding practices between health professionals and institutions. 8 In our study, we used two indicators of influenza syndromes: one based on a clinical definition for the Sentinel network and another created for SOS Médecins, gathering ICPC-2 codes of fives diagnoses. This could explain why a background noise on the weekly variations of the number of visits for influenza syndromes was observed, decreasing the specificity of the indicator. However, this background noise may be constant during all the study period, and this difference between the constructed indicators should therefore not induce a bias for comparing both data sources. Ritzwoller et al. 5 have shown such a difference between a syndromic system and a Sentinel provider, namely by explaining that syndromic surveillance of influenza could capture visits, which the provider did not define as ILI. Acknowledgements This work was supported by association of SOS Médecins Bordeaux. We would also like to thank participating GPs of SOS Médecins Bordeaux, particularly Franck Couvy and Louis Rouxel, who have contributed to developing this syndromic surveillance system. We thank Farida Mihoud for reading the article. References 1 Bravata DM, McDonald KM, Smith WM et al. Systematic review: surveillance systems for early detection of bioterrorism-related diseases. Ann Intern Med 2004;140(11): Doroshenko A, Cooper D, Smith G et al. Evaluation of syndromic surveillance based on National Health Service Direct derived data England and Wales. MMWR Morb Mortal Wkly Rep 2005;54(Suppl): Henning KJ. What is syndromic surveillance? MMWR Morb Mortal Wkly Rep 2004;53(Suppl): Miller B, Kassenborg H, Dunsmuir W et al. Syndromic surveillance for influenza like illness in ambulatory care network. Emerg Infect Dis 2004;10(10): Ritzwoller DP, Kleinman K, Palen T et al. Comparison of syndromic surveillance and a sentinel provider system in detecting an influenza outbreak Denver, Colorado, MMWR Morb Mortal Wkly Rep 2005;54(Suppl): Lazarus R, Kleinman KP, Dashevsky I et al. Using automated medical records for rapid identification of illness syndromes (syndromic surveillance): the example of lower respiratory infection. BMC Public Health 2001;1:9. 7 Zheng W, Aitken R, Muscatello DJ et al. Potential for early warning of viral influenza activity in the community by monitoring clinical diagnoses of influenza in hospital emergency departments. BMC Public Health 2007;7(147): Mandl KD, Overhage JM, Wagner MM et al. Implementing syndromic surveillance: a practical guide informed by the early experience. J Am Med Inform Assoc 2004;11(2):

7 292 JOURNAL OF PUBLIC HEALTH 9 WHO. National Pandemic Influenza Preparedness Plans in France. (13 May 2008, date last accessed). 10 Valleron AJ, Bouvet E, Garnerin P et al. A computer network for the surveillance of communicable diseases: the French experiment. Am J Public Health 1986;76(11): Josseran L, Gailhard I, Nicolau J et al. Organisation expérimentale d un nouveau système de veille sanitaire, France, Bull Epidemiol Hebd 2005;27 28: Josseran L, Nicolau J, Caillere N et al. Syndromic surveillance based on emergency department activity and crude mortality: two examples. Euro Surveill 2006;11(12): Flahault A, Boussard E, Vibert JF et al. Sentiweb remains efficient tool for nationwide surveillance of disease. BMJ 1997;314(7091): Marsden-Haug N, Foster VB, Gould PL et al. Code-based syndromic surveillance for influenzalike illness by International Classification of Diseases, Ninth Revision. Emerg Infect Dis 2007;13(2): Kaufman Z, Wong WK, Peled-Leviatan T et al. Evaluation of a syndromic surveillance system using the WSARE algorithm for early detection of an unusual, localized summer outbreak of influenza B: implications for bioterrorism surveillance. Isr Med Assoc J 2007;9(1): Smith GE, Cooper DL, Loveridge P et al. A national syndromic surveillance system for England and Wales using calls to a telephone helpline. Euro Surveill 2006;11(12): Desenclos JC. Are there new and old ways to track infectious diseases hazards and outbreaks? Euro Surveill 2006;11(12): Buehler JW, Hopkins RS, Overhage JM et al. Framework for evaluating public health surveillance systems for early detection of outbreaks: recommendations from the CDC Working Group. MMWR Recomm Rep 2004;53(RR-5): SOS Medecins France. default.htm (22 January 2008, date last accessed). 20 Bentsen BG. International classification of primary care. Scand J Prim Health Care 1986;4(1): Flamand C, Larrieu S, Couvy F et al. Validation of a syndromic surveillance system using a General Practitioner s House Calls Network. Euro Surveill 2008;13(25):pii: Sentinel network. php?lang=en (15 May 2008, date last accessed). 23 Sentinel network. SentiWeb hebdo No org/ (18 April 2008, date last accessed). 24 French National Institute or Statistics and Economic Studies: Census of Population in March insee.fr/fr/st_ana/r72/popallpop1pop1ar72fr.html (13 May 2008, date last accessed). 25 Viboud C, Boelle PY, Carrat F et al. Prediction of the spread of influenza epidemics by the method of analogues. Am J Epidemiol 2003;158(10): Cooper DL, Verlander NQ, Elliot AJ et al. Can syndromic thresholds provide early warning of national influenza outbreaks? J Public Health (Oxf) 2009;31(1): Fleming DM. The contribution of influenza to combined acute respiratory infections, hospital admissions, and deaths in winter. Commun Dis Public Health 2000;3(1): Thompson WW, Shay DK, Weintraub E et al. Mortality associated with influenza and respiratory syncytial virus in the United States. JAMA 2003;289(2): Bourgeois FT, Olson KL, Brownstein JS et al. Validation of syndromic surveillance for respiratory infections. Ann Emerg Med 2006;47(3): Betancourt JA, Hakre S, Polyak CS et al. Evaluation of ICD-9 codes for syndromic surveillance in the electronic surveillance system for the early notification of community-based epidemics. Mil Med 2007;172(4): Desenclos J-C. Le renforcement et l évolution de la surveillance de la grippe en France: une nécessaire coordination avec un meilleur relais régional. Bull Epidemiol Hebd 2007;39 40: Collins N, Litt J, Moore M et al. General practice: professional preparation for a pandemic. Med J Aust 2006;185(10 Suppl): S Shaw KA, Chilcott A, Hansen E et al. The GP s response to pandemic influenza: a qualitative study. Fam Pract 2006;23(3): Patel MS, Phillips CB, Pearce C et al. General practice and pandemic influenza: a framework for planning and comparison of plans in five countries. PLoS ONE 2008;3(5):e2269.