FORECASTING HOSPITAL EMERGENCY DEPARTMENT VISITS FOR RESPIRATORY ILLNESS USING ONTARIO S TELEHEALTH SYSTEM

Transcription

1 FORECASTING HOSPITAL EMERGENCY DEPARTMENT VISITS FOR RESPIRATORY ILLNESS USING ONTARIO S TELEHEALTH SYSTEM An Application of Real-Time Syndromic Surveillance to Forecasting Health Services Demand by ALEXANDER GORDON PERRY A thesis submitted to the Department of Community Health and Epidemiology in conformity with the requirements for the degree of Master of Science Queen s University Kingston, Ontario, Canada August 2009 Copyright Alexander Gordon Perry, 2009

2 Abstract Background: Respiratory illnesses can have a substantial impact on population health and burden hospitals in terms of patient load. Advance warnings of the spread of such illness could inform public health interventions and help hospitals manage patient services. Previous research showed that calls for respiratory complaints to Telehealth Ontario are correlated up to two weeks in advance with emergency department visits for respiratory illness at the provincial level. Objectives: This thesis examined whether Telehealth Ontario calls for respiratory complaints could be used to accurately forecast the daily and weekly number of emergency department visits for respiratory illness at the health unit level for each of the 36 health units in Ontario up to 14 days in advance in the context of a real-time syndromic surveillance system. The forecasting abilities of three different time series modeling techniques were compared. Methods: The thesis used hospital emergency department visit data from the National Ambulatory Care Reporting System database and Telehealth Ontario call data and from June 1, 2004 to March 31, Parallel Cascade Identification (PCI), Fast Orthogonal Search (FOS), and Numerical Methods for Subspace State Space System Identification (N4SID) algorithms were used to create prediction models for the daily number of emergency department visits using Telehealth call counts and holiday/weekends as predictors. Prediction models were constructed using the first year of the study data and i

3 their accuracy was measured over the second year of data. Factors associated with prediction accuracy were examined. Results: Forecast error varied widely across health units. Prediction error increased with lead time and lower call-to-visits ratio. Compared with N4SID, PCI and FOS had significantly lower forecast error. Forecasts of the weekly aggregate number of visits showed little evidence of ability to accurately flag corresponding actual increases. However, when visits were aggregated over a four day period, increases could be flagged more accurately than chance in six of the 36 health units accounting for approximately half of the Ontario population. Conclusions: This thesis suggests that Telehealth Ontario data collected by a real-time syndromic surveillance system could play a role in forecasting health services demand for respiratory illness. ii

4 Acknowledgements This project was unique and challenging because it combined elements of Epidemiology and Engineering. The following individuals and organizations deserve recognition for their roles in this project: Dr. Kieran Moore, Adam van Dijk, and the other members of the Queen s Public Health Informatics (QPHI) team for their advice and for providing the resources necessary to carry out the project Dr. Will Pickett whose open-mindedness and willingness to supervise this crossdisciplinary project made it possible Dr. Michael Korenberg of the Department of Electrical and Computer Engineering for his insightful suggestions and for agreeing to supervise a project outside his home department in addition to the many other projects with which he is involved Dr. Miu Lam for his advice on statistical aspects of the project The Kingston General Hospital for its financial support through the KGH Scholarship Don McGuinness for his advice and help with ICD code translation Dr. Linda Levesque for her advice and support Finally, I would like to thank my grandfather, Dr. V. R. Perry, for his enthusiasm in my return to school to study Epidemiology iii

5 Table of Contents Abstract... i Acknowledgements... iii Table of Contents... iv List of Acronyms and Abbreviations... vi List of Symbols... vii List of Tables... viii List of Figures... x Chapter 1 Introduction Background Real-Time Syndromic Surveillance Applications of Syndromic Surveillance Study Objectives... 3 Chapter 2 Literature Review and Study Rationale Previous Research on the Telehealth Ontario Call-Emergency Department Visit Relationship for Respiratory Illness Time Series Forecasting Previous Research on Health Service Demand Forecasting Gaps in Existing Knowledge Study Rationale Conceptual Framework Addressing Gaps in Knowledge...16 Chapter 3 Study Design and Methods Study Population, Setting, and Design Data Sources and Ethics Approval Definitions Emergency Department Visits: the NACRS Database Coverage and Data Quality Inclusion/Exclusion Telehealth Ontario Calls Coverage and Data Quality Inclusion/Exclusion Confounders Geographic Grouping of Telehealth Calls and Emergency Visits Analytic Techniques for Establishing the Relationship between Calls and Visits Background Numerical Algorithms for Subspace State Space System Identification...33 iv

6 3.8.3 Fast Orthogonal Search Parallel Cascade Identification Model Implementation Measures...47 Chapter 4 Results Summary Statistics of Telehealth Ontario Calls and Emergency Department Visits by Health Unit Plots of Daily Calls and Daily Visits over Study Period Qualitative Forecast Assessment Quantitative Forecast Assessment Ability to Predict Increases Increases in Emergency Visits Aggregated over a Seven Day Window Increases in Emergency Visits Aggregated over Four Day Windows Chapter 5 Discussion and Conclusions Summary of Key Findings Forecast Accuracy Usefulness of Telehealth Ontario Calls versus Knowledge of Upcoming Holidays and Weekends to Predict Future Visits for Respiratory Illness Comparison of Forecasting Methods Results in the Context of the Existing Literature Study Strengths Study Limitations Application of Results and Implications for Future Research References Appendices APPENDIX A: Ethics Approval APPENDIX B: Ability of Forecasts to Predict Increases in Emergency Department Visits APPENDIX C: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits for All Ontario Health Units v

7 List of Acronyms and Abbreviations Acronym/Abbreviation ARIMA ARX ARMAX AUROC CIHI ED FOS FN FP FSA FWER GARCH ICD LN MA MAPE MCC MSE N4SID NACRS NHS NPV PCI PEM PHLS PHU PPV QPHI RMS ROC RSV Sn Sp SS TN TP UK Definition AutoRegressive Integrated Moving Average AutoRegressive with Exogenous Input AutoRegressive Moving Average with Exogenous Input Area Under the Receiver Operating Characteristic Canadian Institute of Health Information Emergency Department Fast Orthogonal Search False Negative False Positive Forward Sortation Area Family-Wise Error Rate Generalized Autoregressive Conditional Heterokedasticity International Classification of Disease Codes Linear Nonlinear Moving Average Mean Absolute Percentage Error or Mean Absolute Prediction Error Matthew s Correlation Coefficient Mean Square Error Numerical Methods for Subspace State Space System Identification National Ambulatory Care Reporting System National Health Service Negative Predictive Value Parallel Cascade Identification Prediction Error Method Public Health Laboratory Service Public Health Unit Positive Predictive Value Queen s Public Health Informatics Group Root Mean Square Receiver Operating Characteristic Respiratory Syncytial Virus Sensitivity Specificity Subspace True Negative True Positive United Kingdom vi

8 List of Symbols Note: The following list is not exhaustive and provides a reference only to symbols found in the body of the text with no associated equation. Symbols used in equations are defined immediately following the equation. Symbol C c m j 0 j 1 j 2 K k p m M N n w 1 (n) w 2 (n) w m (n) u 1 (n) u 2 (n) v y (n) y(n) z(n) Definition Number of candidate terms in the Fast Orthogonal Search model Candidate term in the Fast Orthogonal Search difference equation model Number of y factors in a term p m of the Fast Orthogonal Search difference equation model Number of u 1 factors in a term p m of the Fast Orthogonal Search difference equation model Number of u 2 factors in a term p m of the Fast Orthogonal Search difference equation model Kalman gain matrix Time index shift m th term in the Fast Orthogonal Search difference equation model Number of terms in the Fast Orthogonal Search difference equation model Sample size/total number of time values in a time series Time index Error in Telehealth Ontario calls at time index n Error in Holidays/Weekends at time index n Orthogonal basis function for the set of terms in the Fast Orthogonal Search difference equation model Telehealth Ontario calls time series at time index n Indicator variable time series for holidays/weekends at time index n Error in Emergency department visits time series at time index n Actual emergency department visits time series at time index n Predicted emergency department visits time series at time index n vii

9 List of Tables Table 1: Literature on Forecasting Health Services Demand ( )... 9 Table 2: ICD-10CA Codes Used to Identify Emergency Visits for Respiratory Complaints from the NACRS Data Set...22 Table 3: Guidelines Used to Identify Calls for Respiratory Complaints from the Telehealth Ontario Data Set...24 Table 4: NACRS Fields Used in Analysis of Emergency Department Visits...25 Table 5: Telehealth Ontario Call Database Fields Used in Analysis...27 Table 6: Structure Choices Required for each Type of Prediction Model...43 Table 7: Total Telehealth Ontario Calls and Emergency Department Visits for Respiratory Complaints by Health Unit over Study Period...64 Table 8: Summary Statistics of Daily Telehealth Ontario Call and Emergency Department Visit Activity for Respiratory Complaints by Health Unit over Study Period...65 Table 9: Ratio of the Median Number of Daily Telehealth Ontario Calls to Median Number of Daily Hospital Emergency Department Visits by Health Unit...66 Table 10: Ages of Individuals Telehealth Ontario Calls were Concerning and Ages of Emergency Department Visit Patients by Health Unit...67 Table 11: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the (Approximate) City of Toronto Health Unit over the Validation Dataset...86 Table 12: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the (Approximate) Grey Bruce Health Unit over the Validation Dataset...86 Table 13: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number of Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) City of Toronto Health Unit over the Validation Dataset...87 Table 14: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number of Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) Grey Bruce Health Unit over the Validation Dataset...87 Table 15: %MSE (MAPE) for 0-Day-Ahead Forecasts of Hospital Emergency Department Visits for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset...88 Table 16: %MSE (MAPE) for 5-Day-Ahead Forecasts of Hospital Emergency Department Visits for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset...89 Table 17: %MSE (MAPE) for 8-Day-Ahead Forecasts of Hospital Emergency Department Visits for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset...90 Table 18: %MSE (MAPE) for 11-Day-Ahead Forecasts of Hospital Emergency Department Visits for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset...91 Table 19: %MSE (MAPE) for 14-Day-Ahead Forecasts of Hospital Emergency Department Visits for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset...92 viii

10 Table 20: Parameter Estimates for the Multilevel Regression Model of Transformed %MSE, MSE T...95 Table 21: Health Units where Forecasts Show Ability to Discriminate between Increases and Decreases in the Aggregate Number of Visits over the Next Four Days Table 22: Health Units where Forecasts Show Ability to Predict 10% Nominal Increases in the Aggregate Number of Visits over the Next Four Days ix

11 List of Figures Figure 1: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario Calls and Emergency Department Visits at the Individual Level...15 Figure 2: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario Calls and Emergency Department Visits at the Population Level...16 Figure 3: Inclusion/Exclusion of Hospital Emergency Department Visits for Respiratory Complaints...26 Figure 4: Inclusion/Exclusion of Telehealth Ontario Calls for Respiratory Complaints...28 Figure 5: The Dynamic Relationship between Calls and Visits Time Series Framed as a System Identification Problem...32 Figure 6: Prediction of Aggregate Hospital Visits over a period of 1-7 Days in the Future (1 Window Ahead) and a period of 8-14 Days in the Future (2 Windows Ahead)...49 Figure 7: Ability to Predict Important Increases in Visits over a Seven-Day Window...55 Figure 8: Threshold used for Flagging an Important Increases in the Number of Emergency Department Visits...57 Figure 9: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls for Respiratory Complaints for the Approximate City of Toronto Health Unit from June 1, 2004 to March 31, Figure 10: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls for Respiratory Complaints for the Approximate Grey Bruce Health Unit from June 1, 2004 to March 31, Figure 11: Zero-Day Ahead Emergency Department Visit Forecast for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting Methods)...72 Figure 12: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto Health Unit...73 Figure 13: Zero-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting Methods)..74 Figure 14: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce Health Unit...75 Figure 15: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting Methods)...76 Figure 16: Forecasting Errors (Predicted - Actual) for Five-Day Ahead Emergency Department Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto Health Unit...77 x

12 Figure 17: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting Methods)..78 Figure 18: Forecasting Errors (Predicted - Actual) for Five-Day Ahead Emergency Department Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce Health Unit...79 Figure 19: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of Actual Visits for the (Approximate) City of Toronto Health Unit...81 Figure 20: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of Actual Visits for the (Approximate) City of Toronto Health Unit...82 Figure 21: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of Actual Visits for the (Approximate) Grey Bruce Health Unit...83 Figure 22: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of Actual Visits for the (Approximate) Grey Bruce Health Unit...84 Figure 23: Regression Model Estimates for the %MSE versus Prediction Lead for Each Forecasting Method for a Ratio of Median Daily Number of Calls to Median Daily Number of Visits of Figure 24: Regression Model Estimates for the %MSE versus Prediction Lead for Each Forecasting Method for a Ratio of Median Daily Number of Calls to Median Daily Number of Visits of Figure 25: Regression Model Estimates for the %MSE versus Prediction Lead for Each Forecasting Method for a Ratio of Median Daily Number of Calls to Median Daily Number of Visits of Figure 26: Plot Illustrating Analyses of PCI-Predicted versus Actual Sequence of Increases/Decreases in Emergency Department Visits One Week in Advance for the City of Toronto Health Unit xi

13 Chapter 1 Introduction 1.1 Background This thesis investigates the use of a nursing telephone help line, Telehealth Ontario, as a source of real-time data for syndromic surveillance of respiratory illness in Ontario. This builds on past research by members of the Queen s Public Health Informatics (QPHI) group (1-5). Specifically, it investigates a practical application of syndromic surveillance using Telehealth Ontario to predict demand for hospital emergency department services for the treatment of respiratory illness Real-Time Syndromic Surveillance In the context of public health, surveillance is the continuous monitoring of the occurrence and distribution of disease in a population and it involves the collection, analysis, interpretation and dissemination of information for this purpose (6). Timeliness, sensitivity, and specificity of detected events are key characteristics of an effective surveillance system (7)(8)(9). Timeliness can pose one of the greatest challenges to surveillance as gathering and assimilating the information from its various sources can be slow (7)(3). Threats of an influenza pandemic and bioterrorism, and events such water contamination in Walkerton, Ontario, and North Battleford, Saskatchewan, and the SARS outbreak in Hong Kong and Toronto, have generated interest in the development of more timely surveillance systems(10-13)(9)(14). 1

14 Syndromic surveillance systems rely on the detection of clinical case features or health behaviours that are discernable before confirmed diagnoses are made... and exploit the fact that ill persons may exhibit behavioural patterns, symptoms, signs, or laboratory findings that can be tracked through a variety of sources (15). This approach combined with real-time automated data collection and anomaly detection methods has given rise to real-time syndromic surveillance systems. The strength of these systems is that they address the issue of timeliness (7)(9). Ideally, real-time syndromic surveillance systems collect data that are leading indicators of disease, provide good coverage of the target population, accurately reflect the level of disease in the target population, and are readily available from electronic sources. Examples of such data sources include calls to nursing help lines, over-the-counter drug sales, emergency medical services dispatch, and emergency department triage information (7)(5). Real-time syndromic surveillance systems automatically integrate and process these data into syndrome categories, scan the resulting time series for unusual numbers of events, and provide rapid dissemination of any anomalies to the appropriate individuals (7,16)(5)(17)(15) Applications of Syndromic Surveillance Early detection of respiratory illnesses, including influenza, has obvious benefits to public health and the health of individuals. Timely warning of increased illness in a population could be used in planning public health interventions to prevent further spread of disease such as vaccination (18), especially vaccination in vulnerable populations which fall short of national targets (19), and screening of health care staff (20). Less obvious, by giving 2

15 an estimate of the prevalence of respiratory illness, these systems could also help facilitate clinical health professionals diagnostic and treatment decisions by providing a measure of the pre-test probability of respiratory illness. Influenza can be difficult to diagnose and knowing the prevalence of disease can substantially increase the utility of a set of symptoms (21). Early disease detection could also have benefits to the health system. Respiratory illness can place a significant burden on hospitals. It is a leading cause of hospitalization in Canada (22), accounts for 12-16% of emergency department visits in Canada (23,24), and has been linked to emergency department overcrowding (25,26)(21). Canada is not alone in this predicament. In fact, it has been recommended that the British National Health Service (NHS) use its disease surveillance systems and its telephone nursing line, the NHS Direct system, to anticipate sudden increases in hospital admissions in winter months, of which respiratory infections are a major factor (27). Anticipating increased visits could help hospitals better manage patient load and reduce wait times for emergency services (28). Doing so might also help improve the efficiency of hospital spending by reducing demand uncertainty (29). 1.2 Study Objectives The objective of this thesis was to examine whether calls to a nursing helpline, Telehealth Ontario, could be used to generate accurate forecasts for the number of emergency department visits for respiratory illness for each of the 36 health regions (health units) in 3

16 Ontario. This tests the hypothesis that Telehealth Ontario calls are a leading indicator of emergency department visits for respiratory illness. The accuracy of the forecasts provides a measure of the degree of association between Telehealth Ontario calls and emergency visits for different lead times. This thesis compared the accuracy of three methods for generating emergency department visit forecasts from the Telehealth calls time series. Nonlinearity in the temporal relationship between calls and visits was considered. Two of the methods used were capable of modeling nonlinearities while the third was not. The forecasting methods were applied in such a way that they could be deployed as part of a real-time syndromic surveillance system: the forecasts used only data that would be available to such a realtime system in making predictions. By using this approach, it was hoped that the study results would have practical significance and real-life application. The three modeling techniques compared have never, to my knowledge, been previously applied to health services demand forecasting or in the context of syndromic surveillance. These methods were developed primarily for time series analysis and modeling in the context of engineering. They represent a progression from the ARIMA (Autoregressive Integrated Moving Average) models used previously in research on forecasting health services demand. Two of them were novel non-linear techniques: Parallel Cascade Identification (PCI) and Fast Orthogonal Search (FOS); and one of them a wellestablished and widely-used linear technique: Numerical Methods for Subspace State Space System Identification (abbreviated in the literature as N4SID or 4SID pronounced forsid (30)). 4

17 Chapter 2 Literature Review and Study Rationale 2.1 Previous Research on the Telehealth Ontario Call- Emergency Department Visit Relationship for Respiratory Illness Telehealth Ontario is a 24-hour, 7 days-a-week, free telephone helpline providing health advice from trained Registered Nurses in English, French, and with translation support available in other languages to callers across Ontario (31)(5). Telehealth receives an average of 2700 calls each day that are captured in a central database. Advice offered includes self-care, referral to physician, referral to the hospital emergency department (ED), and immediate referral to 911 emergency services (1,4,5). Previous research by the QPHI group has characterized Telehealth Ontario calls and emergency department visits for respiratory complaints between mid-2004 and mid At the provincial level, the majority of calls to Telehealth occurred during January and March, on weekends, and in the late afternoon or evening (4). Compared to the hospital emergency visit demographic, Telehealth calls for respiratory symptoms over-represent children 0-4 years old and under-represent older age groups (5-17 years old, years old, and older than 65 years old). Specifically, ages 0-4 represent approximately 49% of calls but only 24% of visits, while individuals older than 65 years represent approximately only 3% of calls but 11% of visits (1). Intensities of both emergency department use and Telehealth Ontario use is known to vary widely across Ontario based on geographic location (3). 5

18 One previous approach to the assessment of data sources for real-time syndromic surveillance used by several researchers involves cross-correlation analyses of the candidate data source, after applying a syndromic mapping, with a gold-standard measure of the outcome or disease being monitored, such as laboratory results (1,32,33). The syndromic mapping classifies events in the candidate data source into syndrome categories. The goal is to evaluate the strength of the correlation between the time series of events assigned to a specific syndrome in the candidate data source and the goldstandard measure of outcome, and to determine how far in advance this correlation is significant. In this way, one can assess the candidate data source as a leading indicator of disease and the usefulness of the syndromic mapping. This type of analysis has been carried out for Telehealth Ontario for monitoring respiratory illness by the QPHI group (1) based on methods developed by an earlier study (32). The Telehealth calls time series was compared to the emergency department visits time series obtained from the National Ambulatory Care Reporting System (NACRS) database for respiratory complaints. Telehealth Ontario calls for respiratory syndrome were identified according to set of call guidelines developed by QPHI (the syndromic mapping). Emergency department visits for respiratory illness were identified by ICD- 10CA codes (International Classification of Disease Codes Revision 10 Canadian Enhancement) for reason for visit. A number of steps to remove the effects of confounding created by repeating patterns in the time series data, in particular weekends which are associated both with increased call(4) and emergency visit (23) activity, were required before assessing the cross-correlation of the time series. To do this, an ARIMA (AutoRegressive Integrated Moving Average) model was fit to the time series to remove 6

19 autocorrelation, including that generated by weekends. Fitting an ARIMA model requires stationary time series, which was achieved through differencing (34)(35). These steps were performed for both time series. Cross-correlation was then performed on the residuals and statistical significance was assessed for the different lags (1). This study concluded that, at the provincial level, Telehealth Ontario calls for respiratory complaints were significantly correlated with emergency department visits for respiratory illness, with strong correlations at zero lag and weak correlations at lags of 15 days (1). 2.2 Time Series Forecasting Forecasting health service demand can be done on a long- or short-term basis. Whereas short-term forecasting can rely exclusively on time series analyses (34), longer-term forecasting must account for other factors such as change in the age structure of the population, the development of alternate forms of care, new procedures, and many other factors that short-term forecasting assumes remain constant (36). Generally, short-term forecasting methods can take three approaches: i) univariate timeseries forecasting methods that rely on past values of a time series to predict its future values, ii) causal models that use the relationship between the variable to be forecast and one or several independent variables to make the forecast, or iii) a combination of both (34). When only a univariate approach is taken, it is assumed that past values of the time series contain information that allow future values to be determined. This thesis is 7

20 concerned with short-term forecasting and employs a causal (as defined above) approach: Telehealth Ontario calls were assumed to be a leading indicator of visits. The influence of holidays/weekends on visits was also accounted for. A popular approach to time series modeling and forecasting involves the use of ARIMA (AutoRegressive Integrated Moving Average) models. ARIMA models can take either a univariate or a causal form. If a causal form is chosen, an exogenous input is used and the model is sometimes referred to as an ARMAX (AutoRegressive Moving Average with exogenous input(s)) model. 2.3 Previous Research on Health Service Demand Forecasting A Medline search for studies in peer reviewed journals from July 2008 dating back to 1996 using subject headings Forecasting, Hospitalization/ or Patient Admission and Health Services Needs and Demand and a Google search (also for studies dating back to 1996) of the world-wide web using the same search terms were performed to determine what methods had been used previously to create short-term health services demand forecasts. Only studies using time series methods to forecast health care contact were included. Five such studies were identified. The study objectives relevant to forecasting and the forecasting methods used are summarized in Table 1. 8

21 Table 1: Literature on Forecasting Health Services Demand ( ) Author, Date Study Objective Relevant to Forecasting Analytic Techniques Used Abdel-Aal, 1998 (37) Forecast monthly patient volume of a primary health care clinic in Saudi Arabia Univariate time-series forecasting using ARIMA models and ad-hoc extrapolation Diaz, 2001(38) Forecast emergency admissions for organic disease, circulatory disease, and respiratory disease in a Madrid hospital using ARIMA models with environmental variables as exogenous inputs environmental variables Jones, 2002 (28) Forecast daily bed occupancy and emergency admissions in an acute hospital in UK ARIMA with exogenous inputs and GARCH (Generalized Autoregressive Conditional Heteroskedasticity) to forecast volatility Reis, 2003 (39) Generated forecasts for number of emergency department visits in order to establish an expected number of visits that could be used in Trimmed mean seasonal model combined with univariate ARIMA models statistical tests for outbreaks in syndromic surveillance Upshur, 2005(40) Examined seasonality and predictability of hospital admissions for various health outcomes in Ontario Regression techniques using sinusoidal terms and spectral analysis 9

22 Abdel-Aal et al.(37) fit a univariate ARIMA model to 108 months of monthly patient visit volume data for a primary care clinic using univariate Box-Jenkins methods. This model was used to forecast visits over the following 24 month period. The clinic served a population of 13,000 and no particular age range or patient population details were discussed. The visits data showed a very regular repeating pattern with increasing trend in the monthly visits. Visits ranged from approximately 400 to 850 patients over the 11 year study period. The study found that the ARIMA models had a forecasting accuracy with a mean absolute percentage error of 1.86% and a maximum absolute percentage error of 4.23% over the last two years of data. Because the visit pattern was so regular, this study also considered a simple ad-hoc extrapolation method for generating forecasts (referred to as extrapolating the growth curve) which involved using values of past visits multiplied by a factor determined using the ratio of past visits indicating anticipated growth. This ad-hoc method produced more accurate forecasts with mean absolute percentage error of 0.55% and a maximum absolute percentage error of 1.17% over the last two years of data. Diaz et al. (38) used an ARIMA model with exogenous inputs including levels four air pollutants, air temperature, humidity, and day of week in order to establish a relationship between air pollutants and daily hospital admissions for total organic-disease, circulatory disease, and respiratory system disease for a single teaching hospital in Madrid over a 1004 day period. Specific details on the demographic characteristics of the patients visiting the hospital examined in the study were not given, but 13% of the Madrid population is over 65. While the authors did not explicitly attempt to forecast with the model, they did suggest that the model might be used to detect variations in the number 10

23 of hospital admissions well in advance and thereby ensure optimal management and allocation of hospital health care resources. A mean error of 15% is reported for the ability to accurately model daily hospital admissions, although it was not clear whether this measurement was made over a separate validation data set or over the set used to fit the model. The lack of precise description of the methods used and the models developed in this study made it difficult to interpret the results. Jones et al. (28) used 2182 days of hospital emergency admissions data in an attempt to build time series models for forecast emergency admissions and bed occupancy in a 540 bed hospital in the Britain serving what the authors describe as an older than average population. The study examined the relationship between the Public Health Laboratory Service s (PHLS) influenza surveillance program estimate of new influenza and influenza-like illness cases and emergency admissions and bed occupancy. Both admissions and bed occupancy were found to be correlated with these estimates up to two weeks in advance. It also examined the relationship between outside temperature and bed occupancy and admissions and found that temperature was related to current bed occupancy but not to admissions. The authors used knowledge of these relationships to attempt to build predictive models of bed occupancy and admissions. To build and test models for bed occupancy and admissions (each treated separately), the available data was divided into 10 segments and an ARIMA model was fit for each segment. The next 32 days was used to assess the model performance. ARIMA models for bed occupancy incorporating exogenous inputs for temperature and the PHLS influenza surveillance program were developed. Errors were calculated as percentages relative to the mean and standard deviations of visits: the mean number (and standard deviation) of daily occupied 11

24 beds was (standard deviation of 32.48) and the mean number of daily admissions was (standard deviation of 13.39). These models had root mean square (RMS) error of 23 beds (5.2%) (standard deviation 4.2) compared with 15.1 (3.4%) (standard deviation 2.09) when no exogenous inputs were included. They noted that the forecasts were poor during times of bed crisis. A simple moving average (MA) model was used to predict admissions and tested in the same way. This model was found to have an RMS error of 12.6 (standard deviation of 2.5) or 24% relative to a mean of admissions. The authors note that using the mean level of admissions to forecast future admissions was almost as good as the moving average model. This study also examined forecasting volatility in admissions and bed occupancy. The authors suggest that future research should consider nonlinearity as it may improve forecasting. Reis et al. (39) attempted to find a systematic method for forecasting the expected number of daily emergency department visits for respiratory complaints in order to be able to reliably detect abnormal visit patterns for the purpose of surveillance. The forecasting models used to generate the expected number of visits incorporated both a trimmed seasonal model and an ARIMA model. The trimmed seasonal model generated the expected number of visits using a sum of the overall mean, a mean for day of week, and a mean for day of year. The authors fit an ARIMA model to the residuals of this trimmed model and the actual time series. Combining these two models improved overall fit. Models were constructed using 2775 days of data and validated over a period of 730 days. The study found a mean absolute percentage error (MAPE) of 9.37% for prediction of overall visits and an MAPE of 27.54% for respiratory visits. This study also investigated the ability to detect outbreaks using a scheme that looked at the difference between the 12

25 expected number of visits forecast by the developed model and actual visits; however, these results are not relevant to the current study. Upshur et al. (40) developed a regression model including sinusoidal terms of monthly hospital admissions for 52 of the most common admission diagnoses for all of Ontario for the period from April 1988 to December The first 148 monthly observations for each series were used to fit the models and the last 12 were used to assess the adequacy of fit. The only measurement of fit provided by the authors is the number of samples in the 12 month validation set that fell outside the 95% confidence interval which was not specified. 2.4 Gaps in Existing Knowledge Based on the research reviewed above, the following gaps are noted: 1) Although the QPHI group has investigated the relationship between Telehealth calls and emergency visits in Ontario at the provincial level, the fact that Telehealth Ontario calls and emergency department visits each vary in intensity by region suggests the callvisits relationship may also vary by geographical region. A preliminary evaluation of the relationship between calls and visits at the health unit level was done by creating plots of weekly calls and visits (3), but the relationship has not been quantitatively evaluated. 13

26 2) The cross-correlation analyses used in past studies to measure the association between a data source for syndromic surveillance and the outcome it was intended to monitor (32,33), including that performed for Telehealth Ontario calls and emergency department visits for respiratory illness at the provincial level (1), ignore the possibility that there may be important information in the calls time-series about visits in the form of a nonlinear relationship. 3) To be useful, knowledge of the relationship between Telehealth Ontario calls and emergency department visits must eventually lead to practical applications. However, to date, studies of the Telehealth calls/emergency department visits relationship have not addressed how information about the call-visits relationship might be applied to public health and clinical practice. 4) While it has been suggested that respiratory illness has an impact on demand for hospital services and that surveillance systems, including telephone help lines, might be used to anticipate demand for these services, there have been few attempts to study how this can be done. Of the literature reviewed, Jones et al. (28) was the best attempt at this specific task. The forecasts obtained by this study for admissions and bed occupancy were not as good as the authors had hoped, and they suggested that nonlinear relationships might be investigated in order to improve results. The study was done at the level of a single hospital and it is unknown whether better or worse results might have been achieved if a larger number of hospitals had been included. The study was performed in the UK and it is not clear how results might differ in Canada. 14

27 2.5 Study Rationale Conceptual Framework Figure 1 and Figure 2 illustrate a hypothetical framework for the temporal relationship between Telehealth Ontario calls and Emergency Department visits for respiratory illness. This relationship can be thought of at two levels: an individual level and a population level. Figure 1 presents a framework at the individual level. Individuals are infected with a respiratory pathogen. The onset of symptoms occurs after some incubation period. Symptoms cause individuals to initiate some sort of behaviour, in this case a call to Telehealth Ontario, which precedes seeking care at the emergency department. Figure 1: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario Calls and Emergency Department Visits at the Individual Level Exposure and Infection Onset of Symptoms Initiation of Behaviour Seek Care Time Call to Telehealth Ontario Delay 1 Emergency Department Visit Although the delay (labeled Delay 1 in Figure 1) between a call to Telehealth and a visit to the emergency department may be short for a given individual (hours or a single day), there may be a longer delay between when some individuals make calls and other members of the population seek care ( Delay 2 in Figure 2). Telehealth Ontario calls for 15

28 respiratory complaints are primarily for younger individuals (1,4). There is evidence that younger individuals start to use health services for respiratory illness sooner than older individuals (41), meaning that the delay between the majority of calls may occur before the majority of visits. Figure 2: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario Calls and Emergency Department Visits at the Population Level Infection of Younger Individuals Infection of Older Individuals Telehealth Ontario Calls from Younger Individuals Emergency Visits from Older Individuals Delay 2 Time Research also indicates that health care contact for the 0-4 year old age group showed the highest seasonal variability in rates (41). Since the majority of calls to Telehealth are from this age group, this might mean that Telehealth calls have good signal-to-noise properties, where signal is defined as the difference in means between the time when there is respiratory illness present to when there is not, and noise is defined as the standard deviation during the time there is no illness present (42) Addressing Gaps in Knowledge The objectives of this study address the knowledge gaps summarized in section

29 1) This study examines the Telehealth call/emergency visit relationship at the health unit level which has not been formally done. Because some health interventions may be coordinated at the health unit level, health units are involved in monitoring infectious disease, and there is geographic variability in the intensity of emergency department and Telehealth Ontario use, it would be helpful to make the assessment of the relationship between Telehealth Ontario calls and emergency department visits for respiratory illness at the health unit level. 2) This study uses methods that are capable of capturing a nonlinear relationship between calls and visits. Furthermore, because three methods of modeling the call-visits relationship are compared, two of which are capable of modeling nonlinear relationships and a third that is not, the results of the study may demonstrate the potential importance of accounting for nonlinearity in these models. If methods capable of modeling nonlinearity perform better than those that do not, the difference might be attributable to important nonlinearity in the relationship captured in the time series models. 3) By measuring the ability of calls to forecast visits, this study examines a practical application of the calls-visits relationship. Currently, there is no published research investigating practical application of the known relationship between Telehealth Ontario calls and emergency department visits for respiratory illness. Although the forecasting of respiratory illness using surveillance information has been suggested in the literature, it appears that few studies have been done to examine its feasibility. Furthermore, past 17

30 studies do not attempt to use nonlinear relationships to generate forecasts, but it has been suggested that doing so might be of value (28). 4) Finally, the literature has recognized the need for more integration between the areas of health services research and informatics in order to improve health care delivery (43). This thesis attempts to bring new approaches to Epidemiology. It suggests a new application for syndromic surveillance systems in forecasting health services demand. 18

31 Chapter 3 Study Design and Methods 3.1 Study Population, Setting, and Design This study examined Telehealth Ontario calls and emergency department visits for respiratory complaints for all of Ontario from June 1, 2004 to March 31, 2006 (669 days). Time-series analyses of the relationship between calls and visits were carried out at the health unit level for each of the 36 health units in Ontario. Forecasting models were constructed using roughly half of the approximately two years of time series data, and then validated on the remaining data. Individual Telehealth calls were not linked to corresponding individual emergency department visits. 3.2 Data Sources and Ethics Approval Hospital emergency department (ED) visits for respiratory complaints were obtained from the Canadian Institute of Health Information (CIHI) National Ambulatory Care Reporting System (NACRS) database using data from the fiscal years and (44,45). All institutions in Ontario providing hospital care are mandated by the Ontario Ministry of Health and Long Term care to submit emergency data to CIHI on a yearly basis (45). Telehealth Ontario calls were obtained from Clinidata Inc. which was contracted by the Ontario Ministry of Health and Long-Term Care to administer the Telehealth Ontario nursing call line over the study period. 19

32 Ethics approval for the project was obtained from the Queen s University Ethics Review Board in accordance with the Tri-Council Policy Statement on the Ethical Conduct of Research Involving Humans (refer to Appendix A for a copy of the ethics approval). 3.3 Definitions Respiratory illness was defined as sickness caused by respiratory pathogens. Pathogens responsible for the majority of respiratory illness screened for by laboratory tests in Canada include respiratory syncytial virus (RSV), parainfluenza viruses, adenoviruses, influenza A and influenza B (46). Specific definitions used in identifying emergency visits and Telehealth Ontario calls are as follows: Emergency Department Visits: Using a gold-standard of laboratory test results for respiratory pathogens, a study by Marsden-Haug et. al. (42) developed a set of International Classification of Disease version 9 (ICD-9) codes for use in syndromic surveillance that were highly correlated with respiratory illness. These ICD-9 codes were translated to ICD-10CA codes, the Canadian enhancement to the ICD-10 codes published by the World Health Organization, using a conversion file and by matching definitions (47)(48). Emergency department visits for respiratory illness were identified from the NACRS database using this set of ICD10-CA codes. Both the ICD-9 codes developed by Marsden-Haug et al. and the corresponding ICD10-CA codes are given in the third column of Table 2. This set of ICD10-CA codes is similar to that used by van Dijk et al. 20

33 (1,3) in a previous study of the Telehealth Ontario call emergency department visit relationship discussed in section

34 Table 2: ICD-10CA Codes Used to Identify Emergency Visits for Respiratory Complaints from the NACRS Data Set ICD9 Codes Developed by Marsden-Haug et al. Corresponding ICD10-CA Codes ICD9 Code ICD9 Description ICD10-CA Code ICD10-CA Description Unspecified viral and chlamydial infections B34.9 Viral infection, unspecified Unspecified otitis media H66.9 Otitis media, unspecified 460 Acute nasopharyngitis [common cold] J00 Acute nasopharyngitis (common cold) Acute sinusitis, unspecified J01.9 Acute sinusitis, unspecified Acute upper respiratory infections of multiple or unspecified sites J06.8 Other acute upper respiratory infections of multiple sites Acute upper respiratory infections of multiple or unspecified sites J39.9 Disease of upper respiratory tract, unspecified J06.9 Acute upper respiratory infection, unspecified Acute bronchitis J20.0 Acute bronchitis due to mycoplasma pneumoniae J20.1 Acute bronchitis due to haemophilus influenzae J20.2 Acute bronchitis due to streptococcus J20.3 Acute bronchitis due to coxsackievirus J20.4 Acute bronchitis due to parainfluenza virus J20.5 Acute bronchitis due to respiratory syncytial virus J20.6 Acute bronchitis due to rhinovirus J20.7 Acute bronchitis due to echovirus J20.8 Acute bronchitis due to other specified organisms J20.9 Acute bronchitis, unspecified 486 Pneumonia, organism unspecified J18.8 Other pneumonia, organism unspecified J18.9 Pneumonia, unspecified Influenza w/ pneumonia J10.0 Influenza with pneumonia, influenza virus identified J11.0 Influenza with pneumonia, virus not identified Influenza w/ other respiratory manifestations J10.1 Influenza with other respiratory manifestations, influenza virus identified J11.1 Influenza with other respiratory manifestations,virus not identified Influenza w/ other manifestations J10.8 Influenza with other manifestations, influenza virus identified J11.8 Influenza with other manifestations, virus not identified 490 Bronchitis, not specified as acute or chronic J40 Bronchitis, not specified as acute or chronic Fever (general symptoms, pyrexia of unknown origin) R50.0 Fever with chills R50.1 Persistent fever 22