S10: Data Quality Assessment and Control Framework for Secondary Use of Healthcare Data

S10: Data Quality Assessment and Control Framework for Secondary Use of Healthcare Data Presented by: Meredith Nahm Zozus Duke University October 7, 2014 1:00 pm - 1:45 pm 1

What do we mean by Secondary Use? 1 o Use: data generated or used in care of a patient 2 o Use: all uses other than that for which the data were originally collected, e.g.: Quality improvement Public health Performance measurement Research

Topics Kinds of data generated in healthcare Secondary use-cases Observational research and characterization Hypothesis driven research Two categories of DQA DQA for the Health Care Systems Research Collaboratory Accuracy assessment examples Accuracy of EHR data via Chart Review Accuracy of EHR data via asking patients 3

Kinds of Data in Healthcare Nature of the phenomena Anatomic or pathologic e.g., fracture, lesion Physiologic or functional e.g., ejection fraction, creatinine clearance, blood pressure Patients symptomatic experiences e.g., pain, nausea, dizziness Patients behaviors or functioning e.g., food intake, Activities of daily living Method of ascertainment Asking patients Observing patient behavior, anatomy, function or images Measurement Synthesis, interpretation by a trained clinician 4

What Happens to Data Nahm, M., Johnson, C., Johnson, T., Zhang, J., What Happens to Data Before Secondary Use? AMIA 2010. Synthesis of four published models.

Example Data Problems Records receive but never viewed Inaccurate statements Charting on the wrong patient Assumed Diagnoses from off-label drug use Problems at the registration desk Have to fill in something Only have a limited number of fields / space / time Local workflow / practices impacting what gets charted and how All of these problems and more are present in routine healthcare data 6

Arguments Against 2 o Use All medical record information should be regarded as suspect; much of it is fiction. - Burnum, 1989 First law: Data shall be used only for the purpose for which they were collected. Collateral: If no purpose was defined prior to the collection of the data, then the data should not be used. - van der Lei, 1991 Burnum IF The misinformation era: fall of the medical record. Ann Intern M 1989; 110: 482-4. van der Lei, J., Use and Abuse of Computer- Stored Medical Records. Meth. Inform. Med.. Vol. 30. No.2. 1991

Tipping Point for 2 o Use in Research Many unanswered questions can be solved with 2 o data We can t afford randomized controlled trials (RCT) for all of these RCTs have evolved several significant inefficiencies that double or triple the cost National initiatives built on 2 o use Patient Centered Outcomes Research network (PCORnet) Healthcare Systems Research Collaboratory FDA MiniSentinel Clinical & Translational Science Awards (CTSA) 8

Not Just Claims Data 9

Special attention on Health Care Systems Research Collaboratory DQAC Framework FRAMEWORKS 10

Two / Three Varieties Predicated on a Common Data Model (tools, IR, no accuracy asmt.) OMOP observational & CER MiniSentinel Post market surveillance of marketed therapeutics PCORnet PCOR & CER Space in the Middle for metadata driven frameworks (healthcare is not there yet requires documenting metadata first) Promise = Tools + no IR! However no accuracy asmt. Assumes NO common data model (no tools, no IR, accuracy asmt.) Healthcare Systems Research Collaboratory Pragmatic Clinical Trials 11

Data Quality Assessment Recommendations v1.0 Initiated by 1. Inventory of data sources and data quality assessment plans proposed in first round UH2 applications, 2. Demonstration project grant review criteria requiring data validation The work presented here was funded by the Health Care Systems Research Collaboratory Coordinating Center grant number 1U54AT007748-01 through the National Center for Complementary & Alternative Medicine, a center of the National Institutes of Health.

Guiding Principles Need to demonstrate that data are capable of supporting research conclusions Should not assume use of a common data model for individual research projects Recommendations should be practical and reasonably achievable Ouch! 13

Recommendations 1 Three key data quality dimensions to be measured 2 Description of formal of assessments 3 Formal impact assessment 4 Reporting data quality assessment with research results 14

Recommendation 1 Accuracy, completeness, and consistency be formally assessed for data elements used in subject identification, outcome measures, and important covariates. Why? These are most impactful on the ability of data to support research conclusions. 15

Recommendation 2 Specifics for measuring accuracy, completeness and consistency Completeness assessment recommendation: four-part completeness assessment. Same column and data value completeness measures can be employed for monitoring completeness during the study. The completeness assessment applies to both prospectively collected and secondary use data. Additional requirements suggested by the Good Clinical Data Management Practices (GCDMP) document, such as onscreen prompts for missing data where appropriate, apply to data collected prospectively for a study. 16

rsely correlated with data accuracy, then percent missing may be an indicator of l racy. hierarchy of sources for comparison shown in Figure 1 provides a list of po parisons ranging (from bottom to top) from those that are achievable in every situ provide less information about true data accuracy, to the ideal but rarely achievable provides an actual data error rate. This hierarchy simplifies the selection of sourc parison: where more than one source for comparison exists, the highest pra parison in the list should be used. Recommendation 2 cont. The highest practical accuracy assessment in the hierarchy should be used Comparison o t a so urce of truth Comparison to an independent measurement Accuracy Comparison to independently managed data Comparison to an upstream data source Comparison to a known standard Comparison to valid values Comparison to validated indicators Comparison to aggregate statistics Partial accuracy Discrepancy detection Gestalt 17

Recommendation 2 cont. Consistency assessment recommendation: Identification of: a) areas where differences in clinical documentation, data collection, or data handling may exist between individuals, units, facilities, sites, or assessors, or over time and b) measures to assess consistency and monitor it throughout the project. A systematic approach to identifying candidate consistency assessments should be used. Such an approach will likely be based on review of available data sources, accompanied by an approach for systematically identifying and evaluating the likelihood and impact of possible inconsistencies. This recommendation applies to both prospectively collected data and secondary use data. 18

Recommendation 3 Impact assessment recommendation: Use of completeness, accuracy, and consistency assessment results by the project statistician to test sensitivity of the analyses to anticipated or identified data quality problems, including a plan for reassessing based on results of data quality monitoring throughout the project. 19

Recommendation 4 Data quality assessments should be reported with research results. Note: there is ongoing work content and format standard for reporting dta quality assessments with research results (Michael Kahn PI). Funded by a PCORI contract. 20

A little more about ACCURACY 21

ard the bottom identify only data discrepancies, i.e., items that may or may not repr ctual error. For example, if it has been shown that a percentage of missing valu rsely correlated with data accuracy, then percent missing may be an indicator of l racy. Accuracy Asmt. Hierarchy hierarchy of sources for comparison shown in Figure 1 provides a list of po parisons ranging (from bottom to top) from those that are achievable in every situ provide less information about true data accuracy, to the ideal but rarely achievable provides an actual data error rate. This hierarchy simplifies the selection of sourc Chart review, parison: where more than one source for comparison exists, the highest pra Medical Record Abstraction parison in the list should be used. Comparison o t a so urce of truth Comparison to an independent measurement Comparison to independently managed data Comparison to an upstream data source Comparison to a known standard Comparison to valid values Comparison to validated indicators Comparison to aggregate statistics Accuracy Partial accuracy Discrepancy detection Gestalt 22

Chart Review Use Cases 1. Data acquisition 2. Phenotype validation validation of data elements + logic/algorithm to extract data from electronic sources 23

Issues with Chart Review Can be subjective Often poorly documented Often poorly controlled Methods infrequently reported Associated with HIGH error rates 24

Chart Review System 292 Unique factors that impact the accuracy of data abstracted from Medical Records 25

Abysmal Reporting of Chart Review Methods a Category includes validation of administrative data, performance measures, or indicators (18); data quality assessment (11); and questionnaire validation (1). 26

MURDOCK Data Quality Study Problem Statement Availability and completeness of the EHR data have not been assessed The accuracy of 1) the self-report data and 2) the EHR data have not been assessed We do not know what type of data discrepancies there are, what data elements are impacted, or how the data discrepancies are otherwise distributed Need to know PPV and NPV These should be assessed to support continuing use of the data. 27

Data Collected in Both EHR and Self-report 34 Medical conditions Medications 8 Procedures Hospitalizations Smoking status Data collection points Self-report: Baseline and annual Follow-up EHR: Longitudinally, quaterly 28

29 Literature Not much help

Study Comparisons Ultimate plan: At the end of the study, will have the Gold standard and can use it to develop and validate indicators of EHR Data Quality. 30 Goal: to make accuracy assessment more attainable!

Comparison Strategy Writing phenotypes for self-report and EHR Three-way classification Self Report Yes Don t know No EHR Confirmatory Suggestive but uncertain No evidence of condition For the 2x2 table, will drop middle categories and analyze separately 31

Study Structure % n = 12,000 n =100 % % 32

Other Sources for Accuracy Asmt. Registry data (electronic compare) Health plan / Medicare data (electronic compare) Inpatient versus outpatient Database for a research study Recorded or double/repeated observer encounters Partial/subset assessment versus whole dataset An exercise in creative opportunicity! 33