Optimizing drug-dose alerts using commercial software throughout an integrated health care system

Transcription

1 Journal of the American Medical Informatics Association, 24(6), 2017, doi: /jamia/ocx031 Advance Access Publication Date: 24 April 2017 Case Report Case Report Optimizing drug-dose alerts using commercial software throughout an integrated health care system Salim M Saiyed, 1,2,3 Peter J Greco, 4,5,6 Glenn Fernandes, 7 and David C Kaelber 4,5,6,8,9 1 Division of Health Sciences Informatics, Johns Hopkins School of Medicine, Baltimore, MD, USA, 2 Department of Family Medicine, St. Joseph s Hospital and Medical Center, Phoenix, AZ, USA, 3 CaroMont Health, Gastonia, NC, USA, 4 Department of Information Services, the MetroHealth System, Case Western Reserve University, Cleveland, OH, USA, 5 Department of Internal Medicine, the MetroHealth System, Case Western Reserve University, 6 Department of Pediatrics, the MetroHealth System, Case Western Reserve University, 7 Department of Pharmacy, the MetroHealth System, Case Western Reserve University, 8 Department of Epidemiology and Biostatistics, the MetroHealth System, Case Western Reserve University, and 9 Center for Clinical Informatics Research and Education, the MetroHealth System, Case Western Reserve University Corresponding Author: David Kaelber. david.kaelber@case.edu. Phone: Received 21 September 2016; Revised 2 February 2017; Accepted 11 March 2017 ABSTRACT All default electronic health record and drug reference database vendor drug-dose alerting recommendations (single dose, daily dose, dose frequency, and dose duration) were silently turned on in inpatient, outpatient, and emergency department areas for pediatric-only and nonpediatric-only populations. Drug-dose alerts were evaluated during a 3-month period. Drug-dose alerts fired on 12% of orders ( / ). System-level and drug-specific strategies to decrease drug-dose alerts were analyzed. System-level strategies included: (1) turning off all minimum drug-dosing alerts, (2) turning off all incomplete information drug-dosing alerts, (3) increasing the maximum single-dose drug-dose alert threshold to 125%, (4) increasing the daily dose maximum drug-dose alert threshold to 125%, and (5) increasing the dose frequency drug-dose alert threshold to more than 2 doses per day above initial threshold. Drug-specific strategies included changing drug-specific maximum single and maximum daily drug-dose alerting parameters for the top 22 drug categories by alert frequency. System-level approaches decreased alerting to 5% (46 988/ ) and drug-specific approaches decreased alerts to 3% (25 455/ ). Drug-dose alerts varied between care settings and patient populations. Key words: clinical decision support, drug-dose alerts, optimizing, EHR BACKGROUND Computerized physician order entry (CPOE) presents many opportunities to improve medication-related patient safety. Drug-drug, drugallergy, drug-diagnosis, and drug-dose alerts are potential areas in which to reduce medication errors and have been identified by Leap- Frog and others as CPOE best practices. 1 However, CPOE drug alerts are overridden 49% 96% of the time. 2 Poorly designed alert systems with low specificity can be an obstacle to patient safety while causing unintended consequences for clinicians. 3 Furthermore, barriers to adoption of meaningful clinical decision support include lack of resources and the expertise to develop, implement, and evaluate these systems. 4 Improving CPOE drug-dose alerts to enhance specificity and sensitivity, reducing false-positive alerts and thereby creating more effective drug-dose alerts, is necessary to realize the full potential of CPOE for patient safety. To date, few studies have looked at categorizing CPOE drug-dose alerts and described strategies to optimize commercially available CPOE alerts using commercially available drug information in an integrated health care delivery network encompassing different health care settings and different patient populations. 5 Our study aims were (1) to determine the initial frequency of drug-dose alerts using our commercial electronic health record (EHR) and commercial VC The Author Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please journals.permissions@oup.com 1149

2 1150 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 6 Table 1. Drug-dosing alerts by category, care setting, and pediatric-only clinical settings Drug-dosing alert category Baseline drug-dose alerts, % ED, % IP, % OP, % Pediatric, % Nonpediatric only populations, % (n) (n) (n) (n) (n) (n) Below minimum daily dose (24 508) (1684) (12 922) (9902) (1787) (23 814) Below minimum frequency (10 330) (718) (5163) (4449) (559) (9772) Exceeded maximum duration (4972) (245) (816) (3911) (208) (4764) Exceeded maximum frequency (16 566) (2840) (9143) (4583) (634) (1593) Exceeded maximum daily dose (24 183) (3662) (14 177) (6344) (2307) (2187) Exceeded maximum single dose (23 539) (4594) (12 760) (6171) (2,206) (21 333) Total ( ) (13 743) (54 981) (35 371) (7701) (96 397) ED, emergency department; IP, inpatient; OP, outpatient. third-party drug information and (2) to develop and evaluate generalizable strategies to optimize drug-dose alerting. METHODS The MetroHealth System (MHS) is a safety-net, integrated, academic health care system affiliated with Case Western Reserve University in Northeast Ohio. MHS includes a 542-bed level-1 trauma center hospital, 4 emergency departments (EDs), and over 2 dozen outpatient clinics. MHS employs over 500 primary-care and specialty-care physicians and over 350 physicians in training. MHS has more than inpatient hospitalizations, over 1 million outpatient visits, and over ED visits annually. MHS is a Healthcare Information Management and Systems Society Electronic Medical Record Adoption Model Stage 7 health care system in both its hospital and ambulatory settings and a Healthcare Information Management and Systems Society enterprise Davies Award winner. MHS began using the Epic EHR (EpicCare, Epic Systems, Verona WI, USA), including CPOE in All visit information is recorded in the EHR, including all orders and prescriptions. All drug-dose alerts were turned on silently, not shown to end users but tracked and analyzed. The Medi-Span VR drug dosing database provided the data for drug-dose decision support. Medi-Span and our EHR vendor provided recommended default settings for drug-dose decision support alerts. The drug-dosing alerts do not have tiers of alerts, such as high, medium, and low risk, as other drug-drug interactions alerts do. These recommended default settings can be adjusted by the customer. The Medi-Span drug dosing database includes data specifying recommended minimum and maximum single dose, daily dose, dose frequency, dose duration, and maximum lifetime dose amounts specific to route of administration, indication, and dose type. We examined all silent drug-dose alerts during 3 months in The initial drug-dose alert setup used default EHR systemlevel settings for drug-dose alerts and the standard drug information vendor values. Each drug-dose alert record included medication name, dosage, route, alert type and percentage difference from the dose rule, patient department, ordering location, ordering provider, and alert ID. We also obtained the total number of medication orders placed each month. Alerts were categorized by type of alert: Below minimum daily dose limit Below minimum dose frequency Exceeding maximum single dose limit Exceeding maximum daily dose limit Exceeding maximum dose frequency Exceeding maximum dose duration We also categorized drug-dose alerts by care setting (ED, inpatient, or outpatient, based on ordering location) and pediatric-only versus nonpediatric-only, based on department where the order was entered (Table 1). The drug-dose data were manually analyzed to determine clinically appropriate, generalizable strategies to reduce clinically falsepositive drug-dose alerts. A 2-step approach was used, employing system-level strategies followed by drug-specific approaches. System-level strategies included changing default settings controlling the firing of drug-dose alerts, either turning off whole categories of alerts or changing their firing thresholds. We incrementally increased the thresholds to evaluate the corresponding impact on the percent reduction in alerts. The research team (3 practicing physicians internist, family medicine physician, and combined internist and pediatrician and a practicing registered clinical pharmacist) picked final thresholds to balance decrease in presumed falsepositive alerts (ie, nonclinically significant dose increases) with potential increase in patient risk (ie, clinically significant dose increases). Individual drugs were identified that comprised a large portion of alerts by volume (any drug causing >1% of all drug-dose alerts). Drug-specific approaches included changes to the drug-dose firing thresholds for specific drugs. These strategies were developed by the authors for clinical relevance and relatively ease of implementation and maintenance. To estimate the true drug-overdose rate, we identified the number of self-reported drug-dose errors in the patient safety network reporting system throughout our health care system during the study period. Microsoft Excel 2010 was used to calculate and summarize the drug-dosing alerts in each category and produce descriptive statistics. This study did not undergo Institutional Review Board review because the data contained no protected health information and the study was performed for operational purposes. RESULTS During the 3-month study period, medication orders were entered through CPOE throughout MHS. Silent drug-dose alerts

3 Journal of the American Medical Informatics Association, 2017, Vol. 24, No Table 2. Frequency of drug-dosing alerts among maximum drugdose alert categories Type of Alert % (n) Single drug-dose amount alerts Exceeds maximum single dose limit by 1% 5% 9 (2188) Exceeds maximum single dose limit by 6% 10% 1 (309) Exceeds maximum single dose limit by 11% 15% 2 (344) Exceeds maximum single dose limit by 16% 20% 1 (292) Exceeds maximum single dose limit by 21% 25% 4 (903) Exceeds maximum single dose limit by 26% 99% 16 (3838) Exceeds maximum single dose limit by 100% 37 (8592) Exceeds maximum single dose limit by >100% 30 (7073) Subtotal 100 (23 539) Daily drug-dose amount alerts Exceeds maximum daily dose limit by 1% 5% 3 (711) Exceeds maximum daily dose limit by 16% 10% 1 (210) Exceeds maximum daily dose limit by 11% 15% 4 (852) Exceeds maximum daily dose limit by 16% 20% 3 (771) Exceeds maximum daily dose limit by 21% 25% 2 (587) Exceeds maximum daily dose limit by 26% 99% 32 (7671) Exceeds maximum daily dose limit by 100% 25 (6162) Exceeds maximum daily dose limit by >100% 30 (7219) Subtotal 100 (24 183) Daily drug-dose frequency alerts Frequency exceeded by 1 dose/day 44 (7247) Frequency exceeded by 2 doses/day 17 (2886) Frequency exceeded >2 doses/day 39 (6433) Subtotal 100 (16 566) fired on 12% ( ) of these orders. For comparison, during the same 3-month period, drug-drug interaction and drug-allergy alerts, both turned on in our system for end users to see, fired on 8% (66 206) and 3% (27 464) of medication orders, respectively. Incomplete information alerts indicating possible drug-dose alerts occurred on 11% (91 060) of these orders ( rule not found and missing weight ). Examples included missing weights for weight-based medications or instances where rules could not be applied due to the system s inability to calculate dose, such as titratable intravenous medication indicating a dose range. For alerts with complete information, the types of drug-dosing alerts and their frequency by clinical area and patient population are shown in Table 1. At the system level, the paramount concern was patient safety, as opposed to drug efficacy, so we turned off all drug-dose minimum alerts. Removing minimum daily dose and minimum frequency alerts removed 24% (24 508) and 10% (10 330) of all alerts. Maximum single and maximum daily drug-dose alerts accounted for the 2 largest categories of alerts, making up 46% of all the dosing alerts (23% [23 539] and 23% [24 183], respectively). Table 2 depicts the breakdown of maximum drug-dose alerts by overdose magnitude. Based on drug-dose alert analysis and clinical judgment, we increased maximum single and maximum daily drug-dose thresholds to 125%, reducing these alerts by 13% (3131) and 17% (4036), respectively. Changing drug-dose frequency alerts to more than 2 doses per day above the recommended drug-dose frequency reduced these alerts by 61% (10 133). After system-level configuration changes, individual drugs were identified based on frequency of alerts triggered. We identified the 22 most frequently alerting drugs (any drug that caused 1% or more of total drug-dose alerts after system-level changes occurred), which caused 21% of all maximum single and maximum daily drug-dose alerts ( and 9072, respectively) (Table 3). The top drugs, simethicone and proton-pump inhibitors, had no alerts after drugspecific dose threshold modifications. Single-dose vancomycin had no alerts, and daily dose had 9 alerts after drug-specific configuration changes. Maximum daily and maximum single drug-dose threshold drug-specific modifications reduced these alerts by approximately 21% (21 533). The 6 system-level interventions resulted in a reduction of the alert rate from 12% to 5% (46 988). Medication-specific changes to the top 22 alerting drugs decreased drug-dose alerts from 5% to 3% (25 455) (Table 4). During the 3-month study period, outpatient (87%), ED (10%), and 7152 inpatient (3%) visits occurred. At baseline, without any drug-dose alert reduction strategies implemented, this led to approximately 0.16 alerts per outpatient, 7.69 alerts per inpatient, and 0.53 alerts per ED patient. In subpopulation analysis, drug-dose alerts were most prevalent for inpatients, followed by patients in the ED, with the lowest incidence of drug-dose alerts among outpatients. Drug-dose alerts were more prevalent in pediatric-only populations then among other populations (Table 1). During the study period, 32 drug-dosing errors (24 wrong dose, 7 wrong frequency, and 1 wrong duration) were manually reported through MHS s patient safety network reporting system, which likely underreports the true number of drug-dose patient safety issues. We analyzed the details of reported errors and found that 3 of those errors would have been intercepted if the drug-dose alerts were displayed to ordering users. Even after our drug-dose optimizations, drug-dose alerts would have been displayed to ordering users in those 3 cases. The other 29 reported drug-dose errors were related to drug administration and not ordering, and so would not have been affected by drug-dose ordering decision support. DISCUSSION Out-of-the-box CPOE drug-dosing alerts based on a commercially available EHR and a commercially available drug information reference produces high (12%) drug-dose alerting rates across multiple care settings and patient populations. However, approaches that focus on patient safety related drug-dose alerts that could potentially cause significant patient harm can significantly reduce baseline drug-dose alerts (decreased by almost 80%). Suppressing alerts can be safe for irrelevant alerts and help reduce alert fatigue. 6 The framework presented here addresses system-level drug-alert configuration changes, as well as drug-specific configuration changes focusing on a limited number of high-drug-dose alerting drugs. These changes can be relatively quickly implemented and are easy to maintain, leveraging the value of a commercial drug information database to provide and maintain comprehensive drug information. 7 Others have reported a high incidence of drug-dose alerts and their relative ineffectiveness. Our observations are consistent with previous reports showing considerable volumes of dosing alerts Our observations are also consistent with previous reports of top alerting drugs. 9 Low specificity and sensitivity, alert overload, workflow interruptions, and perceived relevance to improving quality have all been cited as reasons for ignoring drug-dose alerts. 2 Drugdose alerts are also complex because of the thousands of drugs involved and the wide variety of dose ranges and issues, such as prn (as needed) medications in short-term situations where the theoretic daily dose would be high but would never be reached in actual daily dose values (such as sublingual nitroglycerin every 5 min). 11

4 1152 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 6 Table 3. Optimized individual drug-dose alerts for high drug-dose alerting frequency medications Drug name Baseline max single-dose Baseline max daily dose Baseline single-dose limit Single drug-dose optimization Baseline daily dose limit Daily drug-dose optimization Aluminum and magnesium 3 (2992) 1 (1199) 20 mg 30 mg 60 mg 180 mg hydroxide (simethicone) Acetaminophen 2 (1966) 1 (1430) Weight-based 1000 mg Weight-based 4000 mg PPI 1 1 (1466) 1 (1751) 20 mg 80 mg 20 mg 120 mg Vancomycin 1 (1128) 0.5 (556) 1000 mg 2000 mg 2000 (weight-based) 3000 mg Diatrizoate 0.6 (665) 0 (0) 90 ml 1000 ml 90 ml 1000 ml meglumine-sodium (contrast) Albuterol (inhaler, 0.6 (598) 0.5 (539) 4 mg or 2 puffs 15 mg or 8 puffs 8 puffs or 30 mg 48 puffs or 288 mg solutions) Prednisolone, prednisone, 0.6 (575) 0.3 (309) Varied 1000 mg IV or 100 mg PO Varied 3000 mg IV or 100 mg PO triamcinolone Lorazepam 0.5 (540) 1 (444) Weight-based 30 mg Weight-based 96 mg Sodium polystyrene 0.4 (376) 0 (2) 15 g 40 g or 30 mmol 60 g 180 g or 30 mmol Zolpidem 0.3 (336) 0.3 (327) 5 mg 10 mg 5 mg 10 mg Amoxicillin 0.3 (309) 0.2 (193) Weight-based 2000 mg Weight-based 4000 mg H2 blockers (297) 0.2 (226) 75 mg 300 mg 150 mg 600 mg Ibuprofen 0.3 (297) 0.5 (566) Weight-based 800 mg Weight-based 3600 mg Insulin 0.3 (294) 0.3 (349) Varied 100 units Varied 300 units Enoxaparin 0.2 (255) 0.2 (166) 40 mg 180 mg Weight-based 300 mg Ipratropium 0.2 (253) 0.4 (387) 0.5 mg or 2 puffs 1.5 mg or 4 puffs 2 mg or 12 puffs 6 mg or 24 puffs Eye/ear drops (151) 0.2 (218) Varied Turned off Varied Turned off Statins (138) 0.1 (119) Drug-based 80 mg Drug-based 80 mg Heparin 0.1 (120) 0 (6) Weight-based units units units Ferrous sulfate 0.1 (103) 0.05 (47) Varied 440 mg Varied 975 mg Ondansetron 0.05 (55) 0.08 (80) 4 mg 32 mg 12 mg 48 mg Morphine 0.03 (33) 0.2 (158) 30 mg PO 40 mg PO 30 mg PO or 180 mg IV 80 mg PO or 360 mg IV Total 12 (12 461) 9 (9072) 1 PPI, proton pump inhibitor (omeprazole, esomeprazole, pantoprazole, lansoprazole) 2 H2 blockers (ranitidine, famotidine, cimetidine) 3 Eye/ear drops (antibiotics and steroids) 4 Statins (atrovastatin, pravastatin, rosuvastatin, simvastatin) PO, oral; IV, intravenous Additionally, commercial drug information vendors may be very conservative in their alerting threshold data due to medical-legal reasons as well as to follow market trends. 7,12 Low-specificity alerts can lead to erroneous alert handling. 13 When optimized, non drug-dose drug alerting (duplicate drug class, drug-drug, drug-lab, drug-disease, and drug-pregnancy) has achieved high rates of acceptance. 14 Turning off the maximum duration alerts accounted for 5% of all drug-dose alerts (4972/ ). Most maximum drug-dose duration alerts were for antibiotics, steroids, opioids, and benzodiazepines. While these medications are typically for relatively short-term use, many clinical examples exist where patients are appropriately placed on these medications for chronic use with typically no or minimal adverse effects. For maximum single-dose and maximum daily drug-dose alerts, clinically there is not a discrete cutoff between therapeutic and toxic drug-dose windows, and therapeutic ranges vary by patient for known and unknown factors, with most medications having a therapeutic index (lethal dose for 50% of the population divided by minimum effective dose for 50% of the population) of at least 2 and many times 10 or more for typical patients. 15 Medications with a therapeutic index <2 are already considered narrow therapeutic index medications and typically require additional side effect/toxicity monitoring regardless of the dose being prescribed. 16 Therefore, we felt comfortable that increasing the maximum single-dose and maximum daily drug-dose alerts by 25% would not significantly risk false-negative alerts, but would significantly decrease clinically insignificant false-positive alerts. The 25% increase before alerting also allows for standard clinically appropriate overdosing in situations such as weight-based dosing of medications where small rounding up is needed to provide a practical measurement to dose, or in situations where the patient has developed some tolerance to a medication. Although there could be potential patient safety issues with increasing drug-dose frequency alerting by 2 doses per day for certain medications in isolation, all drug-dose alerts are additive. Therefore, allowing an increase of 2 doses per day in the drug-dose frequency alert would still cause a maximum daily drug-dose alert to fire if the added 2 doses exceeded the maximum daily drug-dose alert threshold. After making systemlevel changes, a relatively small number of drugs (<2 dozen) make up a large percentage (46%) of the remaining drug-dose alerts. We found that in the subcategories below minimum frequency, maximum frequency, maximum daily dose, and single dose, inpatients had a higher incidence of alerts compared to outpatients and

5 Journal of the American Medical Informatics Association, 2017, Vol. 24, No Table 4. Impact of system-level and drug-level drug-dose alert optimization Drug-dose alert category Baseline drug-dose Optimization of drug-dose Optimized drug-dose alerts per order Optimized drug-dose alerts per 100 orders Decrease in drug-dose alerting, % System-level drug-dose alerts Minimum drug-dose daily 24 (24 508) 0 (0) dose alerts (removed) Minimum drug-dose frequency 10 (10 330) 0 (0) alerts (removed) Maximum drug-dose duration 5 (4972) 0 (0) alerts (removed) Maximum drug-dose single-dose 23 (23 539) 42 (19 503) alerts (increased to 125% of threshold) Maximum drug-dose daily dose alerts 23 (24 183) 45 (21 052) (increased to 125% of threshold) Maximum drug-dose dose frequency alerts 16 (16 566) 14 (6433) (increased to more than 2 doses/day of threshold) Subtotal system-level drug-dose alerts 100 ( ) 100 (46 988) Drug-level drug-dose alerts Maximum drug-dose single-dose alerts 58 (12 461) 0 (0) a 0 a 0 a >>99 (top 22 individual dose adjustment customized) Maximum drug-dose daily dose alerts 42 (9072) 0 (0) a 0 a 0 a >99 (top 22 individual dose adjustment customized) Subtotal individual drug-dose alerts 100 (21 533) 0 (0) a 0 a 0 a >99 Total drug-dose alerts 100 ( ) 24 (25 455) a Approximate value. the ED across all drug-dose alerts. The subcategories below minimum daily dose and maximum duration alerts were more common in outpatient areas. Our observation is different than that of the Del Beccaro et al. 5 study in a pediatric population, where they reported a higher percentage of dosing alerts in the outpatient than the inpatient setting. Our rate of pediatric drug-dose alerts is likely underreported due to our limitation of looking by EHR department. While most pediatric patients are seen in pediatric-only areas, other areas such as the ED and family medicine and internal medicine pediatrics clinics also see pediatric populations. Del Beccaro et al., in their pediatric population, showed a higher alert rate for dose range (24%) than our rate of 7%. 5 Some of our top individual drugs with the highest alerting frequency are similar to the top drug alerts reported by Del Beccaro. The study also demonstrates the absolute value of turning on drug-dose alerts (and all other alerts) silently to end users first. This allows true in silico testing, analysis, and modification of alert algorithms without any impact on end users. As shown here, detailed analysis of silent alert data can relatively quickly lead to modification of alerting parameters that can greatly change alert rates, increasing their effectiveness. Alerts should only be shown to end users after optimization has occurred based on data. This study has limitations. First of all, we did not have any objective way to assess whether the proposed alerting changes, compared to the baseline alerting, could have led to any increased potential for patient harm (ie, increase false-negative alerting in any clinically significant way). However, the ordering provider is still always responsible for drugs ordered, and the significant decrease in alerts should lead to increased alert acceptance and therefore improve overall patient safety. Additionally, we did not measure the actual acceptance rate of the baseline alerting or the alerting based on the system-level and drug-level strategies described. CPOE drug-dose alerts using commercial EHRs and commercial drug information can be optimized using a combination of systemlevel and limited drug-specific approaches that are relatively easy to implement and maintain. These approaches significantly decrease drug-dose alerts while maintaining such alerts for potentially clinically significant drug overdoses, which should increase drug-dose alert acceptance and improve patient safety. FUNDING This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector. COMPETING INTERESTS The authors have no competing interests to declare. CONTRIBUTORS All authors made substantial contributions to the manuscript. SS and DK served as the lead authors, conducting data analysis and leading manuscript preparation and writing. PG and GF provided substantial guidance, feedback, and edits during the research and editing process. All authors approved this work. REFERENCES 1. The Leapfrog Group. CPOE Evaluation Tool. talsurvey.org/cpoe-evaluation-tool/. Accessed July 21, van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:

6 1154 Journal of the American Medical Informatics Association, 2017, Vol. 24, No Glassman PA, Simon B, Belperio P, Lanto A. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care. 2002;40(12): Phansalkar S, Wright A, Kuperman GJ, et al. Towards meaningful medication-related clinical decision support: recommendations for an initial implementation. Appl Clin Inform. 2011;2: Beccaro MA, Villanueva R, Knudson KM, Harvey EM, Langle JM, Paul W. Decision support alerts for medication ordering in a computerized provider order entry (CPOE) system: a systematic approach to decrease alerts. Appl Clin Inform. 2010;1: van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc. 2008;15: Reichley RM, Seaton TL, Resetar E, et al. Implementing a commercial rule base as a medication order safety net. J Am Med Inform Assoc. 2005;12: Resetar E, Reichley RM, Noirot LA, Doherty JA, Dunagan WC, Bailey TC. Strategies for reducing nuisance alerts in a dose checking application. Proc AMIA Symp. 2005: Resetar E, Reichley RM, Noirot LA, Dunagan WC, Bailey TC. Implementing daily dosing rules using a commercial rule base. AMIA Annu Symp Proc. 2006: van der Sijs H, Mulder A, van Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf. 2009;18: Stutman HR, Fineman R, Meyer K, Jones D. Optimizing the acceptance of medication-based alerts by physicians during CPOE implementation in a community hospital environment. Proc AMIA Symp. 2007: Gardner RM, Evans RS. Using computer technology to detect, measure, and prevent adverse drug events. J Am Med Inform Assoc. 2004;11(6): van der Sijs H, van Gelder T, Vulto A, Berg M, Aarts J. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform. 2010;79: Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13: Schulz M, Schmoldt, A. Therapeutic and toxic blood concentrations of more than 800 drugs and other xenobiotics. Pharmazie. 2003;58(7): Tamargo J, Heuzey JL, Mabo P. Narrow therapeutic index drugs: a clinical pharmacological consideration of flecainide. Eur J Clin Pharmacol. 2015;71(5):