Data Quality and Medical Record Abstraction in the Veterans Health Administration s External Peer Review Program

Size: px

Start display at page:

Download "Data Quality and Medical Record Abstraction in the Veterans Health Administration s External Peer Review Program"

Laurence Goodwin
6 years ago
Views:

1 Data Quality and Medical Record Abstraction in the Veterans Health Administration s External Peer Review Program (Practice-oriented paper) James H. Forsythe, PhD Epidemiologist, West Virginia Medical Institute Jonathan B. Perlin, MD, PhD Chief, VHA Office of Quality and Performance John Brehm, MD Chief Medical Officer, West Virginia Medical Institute Executive Summary Under the Veterans Health Administration s External Peer Review Program, the West Virginia Medical Institute (WVMI) conducts monthly medical record abstractions in over 150 VA Medical Centers throughout the United States and Puerto Rico. The abstractions are performed by approximately 90 highly trained abstractors and are used to assess VHA clinical performance for: in-patient and out-patient encounters, JCAHO ORYX measures, and ad hoc studies on topics such as management of low back pain, spinal chord injury, and diabetic foot care. To help improve the validity and reliability of the abstracted medical data, WVMI has implemented a multi-method approach to monitoring abstracted data quality. The approach includes five major components: Bi-weekly computer-aided screening to detect anomalous performance (e.g., leading and terminal digit distributions of continuous variables); On-site interrater reliability assessments and calculation of prevalence adjusted Kappa agreement between abstractors and supervising Network Coordinators; Random and special assignment audits by one or more trained auditors; Analyses using SAS Enterprise Miner (including runs and randomness testing, hierarchal modeling (decision tree and cluster analysis) and neural network programming for assessing performance; Statistical process control for tracking and trending performance of abstractors, VAMCs, and items over time. In addition, WVMI has created web-enabled feedback capabilities so that key administrators can rapidly access and report on performance impacting data quality. This paper will outline the data quality techniques and results that have enhanced the use of medical record data for assessing clinical performance throughout the VHA system. 362

2 Data Quality and Medical Record Abstraction in the Veterans Health Administration s External Peer Review Program James H. Forsythe, Ph.D. West Virginia Medical Institute Jonathan B. Perlin, M.D., Ph.D. Chief, Office of Performance and Quality Veterans Health Administration John G. Brehm, M.D. West Virginia Medical Institute West Virginia Medical Institute WVMI is one of 37 designated Peer Review Organizations serving Medicare and Medicaid beneficiaries Staffed by 200 employees located in six offices in WV, VA, DE,& MD WVMI conducts medical record review for the Veterans Health Administration and the Department of Defense Veterans Health Administration External Peer Review Program EPRP began in 1992; WVMI has been prime contractor for both of the 5 year cycles EPRP assesses clinical guideline performance using third party medical record abstraction EPRP is used for comparing performance among VHA hospitals, clinics, and across the 22 administrative regions Medical Record Review WVMI has a Nation-wide network of 95 certified medical record abstractors Records are abstracted throughout the year at 170 hospitals In FY 2001 over 350,000 records were abstracted in hospitals, out-patient clinics, and other care delivery settings Records are transmitted electronically to Charleston, WV and compiled and analyzed for quarterly reports Objectives for Abstractor Monitoring and Data Quality Assessment Measure abstractor performance and detect anomalous behavior Use real-time surveillance & analytical techniques to more quickly identify and correct substandard abstractor performance Rule out abstractor error and focus on other sources of variation Use surveillance for quality control and quality improvement Techniques used to Build the Monitoring and Assessment Model Data Entry Error Detection Leading & Terminal Digit Analysis Pattern Analysis Cluster Analysis AI-aided Profiling 363

3 E-1 Uploaded IRRA Data Uploaded Abstractor Data WVMI EPRP II Data Stores Bi-weekly Computeraided Data Screening Bi-weekly IQC Report to EPRP II Staff Statistical Outliers? Outliers are flagged and logged for Scientific Staff End of trend testing End of trend testing Uploaded Auditor Data Data Passes Stage 1 Assessments Data Sent to Oracle Dbs A-2 QIC Hx Flag records for possible auditing Test variable rates against history of responses Does the rate differ significantly from history of responses QIC*VAMC Hx Does the rate differ significantly from history of responses Audit Process Scientfic Staff Conducts Statistical Analysis Statistically Significant? tfiy AC & NC Assess s Needed? VAMC Hx Flag subsequent dataset on outliers to track next review period Data Passes Level 2 Assessments Send tification to AC and IT Data Released to Oracle Dbs A-2 Data Passes Stage 2 Significance Testing tification & Check off Data Released to Oracle Dbs A-2 Identify variable for analysis Conduct Communicate Results Analyze Data from NC visit or Audit Patterns of Poor Abstraction? Implement Corrective Action Log Test variable rates for responses against a group standard (IQC Data/DEER) Does the rate fall within the limit set by the standard Flag for possible auditing Audit Process Accept Data Passes Level 3 Data Release Check-off Data Released to Store Table A-2 End of range testing Track Performance Assess Outcome Performance Improved? Complete Log Generate Report Quartelry Report New / Termination c:\eprp\2001 data quality process v01.vsd Proceedings of the Sixth International Conference on Information Quality Steps in Monitoring and Improving Abstractor Performance Abstractor Monitoring and Data Quality Assessment Flow Diagram Screen up-loaded medical records Identify abstractors (and records) with unexpected results Analyze results to determine source and extent of the anomalous performance Conduct interventions and field audit where needed Use results for quality improvement training Stage 1: Data Screening Stage 2 : Significance Testing Stage 3 s Stage 4: Tracking & Outcomes Stage 1 Real-time Screening Techniques Data Entry Error Rates IQC Data Screening Process Leading & Terminal digit analysis Disease discrepancy rates Diabetic abnormal foot rates % Dates filled Do not review rates Examples of Data Entry Error Reduction Terminal Digit Analysis: Continuous Variables 4Q98 3Q00 HT Max Min WT Max 2, Min BPs Max Min BPd Max Min 1 32 HbA1c Max Min Terminal Digits Across Quarters 1Q Q Q Q 2000 Base Q Q Q 2001 Term 0 Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 Term 8 Term 9 364

4 Terminal Digit Analysis: Continuous Variable Digit Distributions Blood Pressure TDA: Fabricated Data? 30 Terminal Digits Proportions 85.00% BP Terminal Digit Analysis FY % 65.00% Percent Systolic Diastolic HBa1c Weight (lbs) Height (in) Percent of All Digits 55.00% 45.00% 35.00% 25.00% 15.00% % % Terminal Digit Terminal Digits Blood Pressure TDA: Abstractor #2--Same VAMC/Similar Results Interrater Reliability Assessment Blood Pressure Terminal Digit BP Terminal Digit Analysis FY99 10 BP Terminal Digit Analysis & IRRA Percent of All Digits Percent of All Digits Terminal Digits Terminal Digits Terminal Digit Analysis: BP Terminal Digit Distribution for VAMC 672 Data Screening Abstractor Anomaly Report VAMC 672 BP Terminal Digits 3rd Quarter Anomaly Report Number of Anomalies QIC 5 Area Q Q Q Q Q Q Q Q 2000 Baseline 1Q Q Q 2001 Term 0 Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 Term 8 Term Abnormal Foot Exams: Pnumovac Contraindicated: HTN Discrepancy: DM Discrepancy: COPD Number of 3 Anomalies QIC Area 204 Abnormal Foot Exams: Hospice/Terminal: Pnumovac Contraindicated: 185 Pnumovac Contraindicated: HTN Discrepancy: COPD Discrepancy: 138 Terminal Digit 0: Pnumovac Contraindicated: Do not review: 128 Pnumovac Contraindicated: Do not review: COPD Discrepancy: 102 Terminal Digit 0: Pnumovac Contraindicated: HTN Discrepancy: 365

5 Abstractor Outlier Report SPC Analysis of Increase in Depression Screening rd Quarter - QIC Outlier Report: Key Performance Indicators for QIC 199 variable level description Chg from history Chg from previous Q Q Q Q BL HXASCVD 1 extreme increase significant increase 26% 17% 0.4% 4.3% 2 extreme decrease significant decrease 74% 83% 100% 96% TOBSTATUS 1 Current user increased increased 19% 19% 18% 18% 2 Former user increased increased 22% 21% 17% 19% 3 Denies current use/ no further info decreased decreased 22% 22% 29% 22% 4 use in past 7 years increased increased 35% 31% 28% 28% 5 documentation extreme decrease significant decrease 2.3% 6.2% 8.8% 13% Techniques used to Assess Data Reliability Interrater Reliability Assessment Intrarater Reliability Assessment False Negative & False Positive Rates Service/Clinical Indicator Date Variance Item Reliability Assessment Interrater Reliability Assessments IRRAs occur between abstractors and either their field supervisor or an auditor Attempt to interrate between 20 and 25 records Calculation of agreement using weighted percent agreement and Kappa beyond chance agreement Abstractors (and items) yielding low Kappa agreement (<.85) are identified for QI training Problems with Kappa in Contexts of High Goal Attainment Prevalence of an observed trait: 100% Agreement that a service was provided = Kappa Score Example: % agreement = 100 Kappa can not be calculated One disagreement can yield a Kappa Score of Zero 95%+ Agreement that a service was/was not provided can yield a zero or negative Kappa Score Example: % agreement = 95 Kappa = 0 366

6 Run SAS report, charting, and SPC procedures with ODS HTM L output Send HTM L output to specified network location for sta ff re vie w w ith w e b browser HTM L Identify variables for analysis Calculate frequency distributions of responses for each variable level (by QIC,VAMC,QIC*VAMC) Run test of proportion SAS macro on each variable for specified tim e fram e. (Test against com plete history of scores and previous quarter) Append results into a single dataset and keep outlier in fo rm a tio n Transpose data for reporting/ exporting Paper and HTML reporting Paper Paper Export data to MS Access to generate printed reports Print SAS output charts Distribute reports for staff review Proceedings of the Sixth International Conference on Information Quality High agreement yielding a negative Kappa 90%+ Agreement that a service was/was not provided can yield a zero or negative Kappa Score Example 5: % agreement = 90 Kappa = -.05 High agreement yielding an unacceptable Kappa Score 95% Agreement that a service was/was not provided can yield a low Kappa Score Example: % agreement = 95 Kappa =.64 Interrater Reliability Prevalence Adjusted Item Scores: 4Q99 Auditing Process Flow Diagram Item Kappa Scores Fourth Quarter Average Item kappa score: Kappa Score (adjusted and weighted) Items 3Q01 Abstractor Assessment & Audit Kappas Abstractor Assessment Audit Kappas #Records Kappas #Records Overall: Overall: Abstractor Performance with Increases in Record Volume Over an 18 month period, the number of required abstractions nearly tripled WVMI increased the number of abstractors from 35 to 95 How has performance been impacted with increases in record volume? How has adding items to the instrument impacted medical record abstraction? 367

7 Total Number of Clinical Guideline & Performance Improvement Records Abstracted Total Number of Instrument Items in the CGPI Records Abstracted Number of Instrument Items Q Q Q Q 2000 Baseline 1Q Q Q Q1998 1Q1999 2Q1999 3Q1999 4Q1999 1Q Q Q Q 2000 Baseline 1Q Q Q 2001 Percent of Items Filled By Quarter Percentage of Items Filled By Abstractor 6 Percent of Items Filled by Quarter 6 Average Percent of Items Filled by Abstractor 2000 to 3Q Q Q Q Q Q Q Q Outlier Rates: HTN Outlier Rates: COPD HTN Discrepancy Outlier Rate COPD Discrepancy Outlier Rate 1 1 3Q Q 2000 Baseline 1Q Q Q Q Q 2000 Baseline 1Q Q Q

8 Outlier Rates: Diabetes Outlier Rates: Abnormal Foot Diabetes Discrepancy Outlier Rate Abnormal Foot Outlier Rate 1 1 3Q Q 2000 Baseline 1Q Q Q Q Q 2000 Baseline 1Q Q Q 2001 Current Status of the Assessment and Data Quality Model Demonstrated ability to detect negligent or fabricated data Rate of agreement among abstractors is approximately 90% Agreement rates are impacted by quality in, and types of, record keeping (paper, electronic, and both together) and, the item needing abstraction 369

NHSN Tips for CMS Hospital IQR Program: MRSA Bacteremia and CDI LabID Healthcare Personnel Influenza Vaccination

NHSN Tips for CMS Hospital IQR Program: MRSA Bacteremia and CDI LabID Healthcare Personnel Influenza Vaccination Maggie Dudeck, MPH, CPH Epidemiologist National Provider Education Webcast May 1, 2013 National