Building a Robust Conceptual Foundation for Defining and Measuring Diagnostic Errors

Size: px

Start display at page:

Download "Building a Robust Conceptual Foundation for Defining and Measuring Diagnostic Errors"

Joella Charles
6 years ago
Views:

1 1 Building a Robust Conceptual Foundation for Defining and Measuring Diagnostic Errors Hardeep Singh, MD MPH Houston Veterans Affairs Center for Innovations in Quality, Effectiveness & Safety Michael E. DeBakey VA Medical Center Baylor College of Medicine

2 2 Where Do We Start? Measurement is the first step to improvement But, we cannot measure what we cannot define Experts still debating definitions of diagnosis o lack of standards, agreed upon definitions for most concepts involving diagnosis Operational definitions of diagnostic error harder o often confused with screening and treatment errors Basic science of diagnostic errors evolving o harm often involves relatively common conditions Outpatients (including ER) fertile environments Singh et al BMJQS 2014 Singh et al JAMA Intern Med 2013 Singh & Weingart Adv Health Sci Educ 2009

3 3 What are Diagnostic Errors? Case analysis reveals evidence of a missed opportunity to make a correct or timely diagnosis o something different could have been done to make the correct diagnosis earlier o may result from cognitive and/or system factors, or o more blatant provider factors (accountability/negligence) Missed opportunity is framed within the context of an evolving diagnostic process o Accounts for uncertainty Opportunity could be missed by the provider, care team, system, and/or patient Singh Jt Comm J Qual Patient Saf 2014

4 A Model for Defining Diagnostic Error MISSED OPPORTUNITIES HARM (from delayed or wrong treatment/test) NO MISSED OPPORTUNITIES A B C D Missed opportunities in diagnosis due to system and/or

4 4 A Model for Defining Diagnostic Error MISSED OPPORTUNITIES HARM (from delayed or wrong treatment/test) NO MISSED OPPORTUNITIES A B C D Missed opportunities in diagnosis due to system and/or cognitive factors Preventable diagnostic harm Delayed/wrong diagnosis associated with patient harm but no clear evidence of missed opportunities Delayed/wrong diagnosis but no clear evidence of missed opportunities Adapted from Singh Jt Comm J Qual Patient Saf 2014

5 5 Missed Opportunities-based Definition Used in multiple studies, thousands of record-reviews Accounts for broad stakeholder perspectives: patients, clinicians, health systems & consistent with safety/qi Useful to advance the basic science of diagnostic error o Avoids overestimating/overcalling: all diagnoses are delayed/wrong at some point in time Universal definition needed for all settings and stakeholders (consumers, providers, researchers/qi) However, specific diseases might require specific operational criteria for what is a missed opportunity o E.g.: lung cancer - 7 day delay on abnormal chest x-ray Singh et al BMJQS 2012 Singh et al Am J Gastroenterol 2009 Singh et al J Clin Oncol 2010 Singh et al Arch Intern Med 2007

6 6 Foundation for Rigorous Measurement Few valid and reliable data sources Error measurement must reflect real-world practice o more than just what s in the doctors head o systems, team members, and patients, all inevitably influence clinicians thought processes Structure - complex adaptive sociotechnical system - technological and non-technological dimensions Process - diagnosis evolves in distributed dimensions beyond the doctors visit Outcomes - safe (correct and timely) diagnosis vs. missed/delayed/wrong/over diagnosis; but should also account for patient and care outcomes Singh BMJQS 2013 Sarkar et al BMJQS 2012

7 Framework for Measurement and Reduction of Diagnostic Errors 7 Sociotechnical Work System* Diagnostic Process Dimensions Changes in policy and practice to reduce preventable harm from missed, delayed, wrong or over diagnosis Patient-provider encounter & initial diagnostic assessment Follow-up and tracking of diagnostic information Patient Diagnostic test performance & interpretation Subspecialty consultation/ referral issues Measurement of diagnostic errors Reliable Valid Retrospective Prospective Collective mindfulness Organizational learning Improved calibration Better measurement tools and definitions Safer Diagnosis Improved value of health care Improved Patient Outcomes Feedback for improvement * Includes 8 technological and non-technological dimensions Sittig & Singh Qual Saf Health Care 2010 Singh et al BMJQS 2012 Singh & Weingart Adv Health Sci Educ 2009 Weick et al Research in Organizational Behavior 1999

8 Methods of Retrospective Measurements Signals from administrative data are too weak o If validated, could provide clues on possible missed opportunities that warrant additional clinical evaluation o Current evidence insufficient for rigorous measurement Stronger signals to bolster error measurement o Try more autopsies: virtual autopsies in selected cases? o Try high-risk cohorts (cancer ~ 1/3 rd delays; abnl results) o Provider surveys (30% missed results with care delays) o Incident reports from providers Not covered comprehensively by AHRQ Common Formats ER experience with MD champion and QI team; in reality, organizations don t provide protected time o Future potential of incident reports from patients? Singh et al Pediatrics 2010 Singh et al JAMIA 2012 Okafor et al DEM 2014 Singh et al Qual Manag Health Care

9 9 Triggers: Promises and Challenges Algorithms to select high-risk patient records for further reviews to look for missed opportunities o should leverage clinical (EHR) data Application retrospective or prospective surveillance Diagnostic missed opportunities found so far: o patient-provider encounter related breakdowns common, o lack of timely recognition/follow-up of predefined diagnostic clues such as abnormal test results Documentation-dependent; not easy to identify contributory factors esp. precise cognitive origins Challenges of lack of definitions & standards Murphy et al BMJQS 2013 Singh & Thomas AHRQ Special Report 2009

10 10 Time for Triggered Reviews? Multiple reviewers; time investment o Institutions/practices have too many competing priorities o Will it give bang for the buck outside of research? Different mental models for what's an error o High level of disagreement on presence/absence of error o Higher reliability with explicit criteria: failure to act on predefined red-flag signs/symptoms or abnormal tests Diagnostic Error Evaluation Tool: ongoing effort to convert subjective error judgment to objective criteria to facilitate error determination Singh et al BMJQS 2012 Singh et al Am J Gastroenterol 2009 Singh BMJQS 2013

11 11 More Prospective Measurements Direct observations resource intensive Simulations/vignettes: generalizability issues Gain insights from safety huddles, active monitoring of patient complaints and peer-review data Checklist-type tool for Proactive Risk Assessment at organizational or practice level o Self-assessment of specific recommended practices to ensure safer diagnosis o Prototype: Office of the National Coordinator (ONC) Sponsored Safety Assurance Factors for EHR Resilience Project (SAFER Guides) Meeks et al BMJQS 2014 Singh BMJQS 2013 Singh et al BMC Med Inform Decis Mak

12 Collective Mindfulness for 12 Diagnostic Safety Organizations and practices need to o Be preoccupied with diagnostic errors: they are unaddressed by current safety measures o Gather intelligence related to diagnostic safety through any retrospective and prospective surveillance method (ideally multiple methods) leverage existing safety/qi infrastructure: risk managers o Learn, improve and evaluate both intended and unintended consequences of interventions Critical to inform good measures and solutions based on science vs. belief Weick et al Research in Organizational Behavior 1999 Sittig & Singh JHRM 2013

13 13 Reflections on Measurement Measurement ready for QI purposes (e.g. specific triggers with more refinement, test results follow-up, possibly others) Not ready for public reporting, performance measurement or penalties Still need more evidence and research in measurement o data, standards and operational definitions o sharp-end outcome measures for errors o blunt-end measures for systems diagnostic performance Need to go beyond the VA and few other institutions! o Others should start measuring for transparency Smith et al BMJQS 2013 Wachter Diagnosis 2014 Singh et al In preparation Giardina et al Health Affairs 2013

14 Health IT & Diagnostic Safety 14 Few leveraging health IT to measure diagnostic safety o Our measurement revealed e-communication breakdowns o Innovation slow due to focus on meaningful use o Copy/paste issues & absent reflections/differential diagnosis Measuring diagnostic safety in EHR settings: o Current EHRs provide inadequate cognitive support for provider or team situational awareness o Potential to interfere with provider cognition (info overload) o Data display issues leading to ambiguity/missed findings o ipatient and effect on critical thinking skills? Singh et al BMJQS 2012 Singh et al Arch Int Med 2009 and Singh et al AJM 2010 Sittig & Singh NEJM 2012 Giardina et al Health Affairs 2013 Singh et al JGIM 2011 Singh et al JAMA Intern Med 2013

15 15 Feedback on Diagnostic Performance Could improve provider calibration (i.e. alignment between diagnostic accuracy and confidence in that accuracy) Could improve system calibration (i.e. alignment between safety measurement and reality) Diagnostic performance = individual + system performance so need both provider-centric and system-centric approaches Preliminary experiences with provider feedback o Uncertain responsiveness: Examples from work on secure s and phone calls about missed test results o Will be challenging to integrate into workflow o Only ½ wanted EHR alerts-related performance feedback Meyer et al JAMA Intern Med 2013 Singh & Sittig J Gen Intern Med 2014 Singh et al JAMIA 2012 Singh & Graber JAMA 2011 Meyer et al DEM 2014

16 16 Unknowns in Formal Feedback First, robust definitions and measurements critical Unknown attributes; not easy to borrow lessons o Content: Signal strength (unplanned hospitalization vs. preventable diagnostic adverse event?) o Delivery: Methods verbal vs. written? o Timing: Delayed when event is clear or more real-time when details evolving or unavailable? o Quality: Unambiguous, non-threatening, non-punitive? Shared accountability beyond the clinician/s involved Unintended consequences need to be monitored o More testing/treatment could occur

17 17 Institute of Medicine & Funding Agencies Department of Veterans Affairs Agency for Health Care Research & Quality National Institute of Health Multidisciplinary team at Houston-based VA Health Services Research Center of Innovation Baylor College of Medicine

18 18 References: Slides 2-7 Slide 2 Singh et al BMJQS 2014 Singh et al JAMA Intern Med 2013 Singh & Weingart Adv Health Sci Educ 2009 Slide 3 & 4 Singh Jt Comm J Qual Patient Saf 2014 Slide 5 Singh et al BMJQS 2012 Singh et al Am J Gastroenterol Singh et al J Clin Oncol Singh et al Arch Intern Med Slide 6 Singh BMJQS 2013 Sarkar et al BMJQS 2012 Slide 7 Sittig & Singh Qual Saf Health Care 2010 Singh et al BMJQS 2012 Singh & Weingart Adv Health Sci Educ 2009 Weick et al Research in Organizational Behavior 1999 Singh H, Meyer AN, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf Singh H, Giardina TD, Meyer AN, Forjuoh SN, Reis MD, Thomas EJ. Types and Origins of Diagnostic Errors in Primary Care Settings. JAMA Intern Med 2013;1-8. Singh H, Weingart SN. Diagnostic errors in ambulatory care: dimensions and preventive strategies. Adv Health Sci Educ Theory Pract 2009;14 Suppl 1: Singh H. Editorial: Helping Health Care Organizations to Define Diagnostic Errors as Missed Opportunities in Diagnosis. Joint Commission Journal on Quality and Patient Safety 2014;40: Singh H, Giardina TD, Forjuoh SN et al. Electronic health record-based surveillance of diagnostic errors in primary care. BMJ Qual Saf 2012;21: Singh H, Daci K, Petersen L et al. Missed opportunities to initiate endoscopic evaluation for colorectal cancer diagnosis. Am J Gastroenterol 2009;104: Singh H, Hirani K, Kadiyala H et al. Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study. J Clin Oncol 2010;28: Singh H, Thomas E, Khan MM, Petersen L. Identifying diagnostic errors in primary care using an electronic screening algorithm. Arch Intern Med 2007;167: Singh H. Diagnostic errors: moving beyond 'no respect' and getting ready for prime time. BMJ Qual Saf 2013;22: Sarkar U, Bonacum D, Strull W et al. Challenges of making a diagnosis in the outpatient setting: a multisite survey of primary care physicians. BMJ Qual Saf 2012;21: Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care Oct;19 Suppl 3:i Singh H. Diagnostic errors: moving beyond 'no respect' and getting ready for prime time. BMJ Qual Saf 2013;22: Singh H, Weingart SN. Diagnostic errors in ambulatory care: dimensions and preventive strategies. Adv Health Sci Educ Theory Pract 2009;14 Suppl 1: Weick KE, Sutcliffe KM, Obstfeld D. Organizing for High Reliability: Processes of Collective Mindfulness. In: Sutton RS, Staw BM, eds. Research in Organizational Behavior. Vol 1 ed. Stanford: Jai Press; 1999;

19 19 References: Slides 8-11 Slide 8 Singh et al Pediatrics 2010 Singh et al JAMIA 2012 Okafor et al DEM 2014 Singh et al Qual Manag Health Care 2012 Slide 9 Murphy et al BMJQS 2013 Singh Thomas AHRQ Special Report 2009 Slide 10 Singh et al BMJQS 2012 Singh et al Am J Gastroenterol Singh BMJQS 2013 Slide 11 Meeks et al BMJQS 2014 Singh BMJQS 2013 Singh et al BMC Med Inf Singh H, Thomas EJ, Wilson L et al. Errors of diagnosis in pediatric practice: a multisite survey. Pediatrics 2010;126: Singh H, Spitzmueller C, Petersen NJ et al. Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc Okafor N, Payne VL, Singh H, et al. Using Voluntary Physician Reporting To Learn From Diagnostic Errors in Emergency Medicine. Diagnostic Error in Medicine 7th International Conference, Singh H, Khan R, Giardina TD et al. Postreferral colonoscopy delays in diagnosis of colorectal cancer: a mixed-methods analysis. Qual Manag Health Care 2012;21: Murphy DR, Laxmisan A, Reis BA et al. Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf 2014;23:8-16. Singh H, Thomas E. Diagnostic event triggers: Current state of science and future directions. AHRQ Special Report Singh H, Giardina TD, Forjuoh SN et al. Electronic health record-based surveillance of diagnostic errors in primary care. BMJ Qual Saf 2012;21: Singh H, Daci K, Petersen L et al. Missed opportunities to initiate endoscopic evaluation for colorectal cancer diagnosis. Am J Gastroenterol 2009;104: Singh H. Diagnostic errors: moving beyond 'no respect' and getting ready for prime time. BMJ Qual Saf 2013;22: Meeks D, Meyer ND, Rose B, Walker YN, Singh H. Exploring New Avenues to Assess the Sharp End of Patient Safety: An Analysis of Nationally Aggregated Peer Review Data. BMJ Qual Saf 2014 (In Press) Singh H. Diagnostic errors: moving beyond 'no respect' and getting ready for prime time. BMJ Qual Saf 2013;22: Singh H, Wilson L, Petersen L et al. Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication. BMC Med Inf and Decision Making 2009;9. Safety Assurance Factors for EHR Resilience (SAFER) Guides. Office of the National Coordinator for Health Information Technology (ONC) [serial online] 2014; Accessed August 5, 2014.

20 20 References: Slides Slide 12 Weick et al Research in Organizational Behavior 1999 Sittig & Singh JHRM 2013 Slide 13 Smith et al BMJQS 2013 Wachter Diagnosis 2014 Giardina et al Health Affairs 2013 Slide 14 Singh et al BMJQS 2012 Singh et al Arch Int Med 2009 Singh et al AJM 2010 Sittig & Singh NEJM 2012 Giardina et al Health Affairs 2013 Singh et al JGIM 2011 Singh et al JAMA Intern Med 2013 Slide 15 Meyer et al JAMA Intern Med 2013 Singh & Sittig J Gen Intern Med 2014 Singh et al JAMIA 2012 Singh & Graber JAMA 2011 Weick KE, Sutcliffe KM, Obstfeld D. Organizing for High Reliability: Processes of Collective Mindfulness. In: Sutton RS, Staw BM, eds. Research in Organizational Behavior. Vol 1 ed. Stanford: Jai Press; 1999; Sittig DF, Singh H. A red-flag-based approach to risk management of EHR-related safety concerns. J Healthc Risk Manag 2013;33: Smith MW, Davis GT, Murphy DR, Laxmisan A, Singh H. Resilient actions in the diagnostic process and system performance. BMJ Qual Saf 2013;22: Wachter RM. Diagnostic errors: central to patient safety, yet still in the periphery of safety's radar screen. Diagnosis 2014;1: Giardina TD, King BJ, Ignaczak A, Singh H et al. Root Cause Analysis Reports Help Identify Common Factors in Delayed Diagnosis and Treatment of Outpatients. Health Aff 2013;32. Singh H, Giardina TD, Forjuoh SN et al. Electronic health record-based surveillance of diagnostic errors in primary care. BMJ Qual Saf 2012;21: Singh H, Mani S, Espadas D, et al. Prescription errors and outcomes related to inconsistent information transmitted through computerized order entry: a prospective study. Arch Intern Med 2009;169: Singh H, Thomas EJ, Sittig DF et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med 2010;123: Sittig DF, Singh H. Electronic health records and national patient-safety goals. N Engl J Med 2012;367: Giardina TD, King BJ, Ignaczak A et al. Root Cause Analysis Reports Help Identify Common Factors in Delayed Diagnosis and Treatment of Outpatients. Health Aff 2013;32. Singh H, Esquivel A, Sittig DF et al. Follow-up Actions on Electronic Referral Communication in a Multispecialty Outpatient Setting. J Gen Intern Med 2011;26: Singh H, Giardina TD, Meyer AN, Forjuoh SN, Reis MD, Thomas EJ. Types and Origins of Diagnostic Errors in Primary Care Settings. JAMA Intern Med 2013;1-8. Meyer AN, Payne VL, Meeks DW, Rao R, Singh H. Physicians' Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study. JAMA Intern Med Singh H, Sittig DF. Were My Diagnosis and Treatment Correct? No News is Not Necessarily Good News. J Gen Intern Med 2014;29: Singh H, Spitzmueller C, Petersen NJ et al. Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc Singh H, Graber M. Reducing diagnostic error through medical home-based primary care reform. JAMA 2010;304:

INSIGHTS ON MEASUREMENT AND IMPROVEMENT OF DIAGNOSTIC ERRORS IN HEALTH CARE

INSIGHTS ON MEASUREMENT AND IMPROVEMENT OF DIAGNOSTIC ERRORS IN HEALTH CARE HARDEEP SINGH, MD, MPH HOUSTON VETERANS AFFAIRS CENTER FOR INNOVATIONS IN QUALITY, EFFECTIVENESS & SAFETY MICHAEL E. DEBAKEY