Administrative database research has unique characteristics that can risk biased results

Transcription

1 Journal of Clinical Epidemiology 65 (2012) 126e131 REVIEW ARTICLE Administrative database research has unique characteristics that can risk biased results Carl van Walraven a,b,c, *, Peter Austin d a Department of Medicine, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Ontario K1Y 4E9, Canada b ICES@uOttawa, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Ontario, Canada c Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Ontario, Canada d Institute for Clinical Evaluative Sciences, University of Toronto, Toronto, Ontario, Canada Accepted 2 August 2011; Published online 9 November 2011 Abstract Objective: The provision of health care frequently creates digitized datadsuch as physician service claims, medication prescription records, and hospitalization abstractsdthat can be used to conduct studies termed administrative database research. While most guidelines for assessing the validity of observational studies apply to administrative database research, the unique data source and analytical opportunities for these studies create risks that can make them uninterpretable or bias their results. Study Design: Nonsystematic review. Results: The risks of uninterpretable or biased results can be minimized by; providing a robust description of the data tables used, focusing on both why and how they were created; measuring and reporting the accuracy of diagnostic and procedural codes used; distinguishing between clinical significance and statistical significance; properly accounting for any time-dependent nature of variables; and analyzing clustered data properly to explore its influence on study outcomes. Conclusion: This article reviewed these five issues as they pertain to administrative database research to help maximize the utility of these studies for both readers and writers. Ó 2012 Elsevier Inc. All rights reserved. Keywords: Administrative database research; Bias; Diagnostic codes; Clinical significance; Time-dependent bias; Multilevel analysis 1. Introduction Health care provision is becoming increasingly digitized. In most jurisdictions, patient visits are logged in registration systems. The dates of physician visits, laboratory tests, and radiological investigations are recorded in physician claims databases. The diagnoses, procedures, and simple outcomes of visits to emergency departments or admissions to hospital are documented in hospitalization databases. Each of these systems leaves a trail of digital information that describes (to varying degrees of detail) a patient s course through a health care system. These data can be used to conduct research studies that can be termed administrative database research. As in other types of observational research, the overarching goal of administrative database research is the * Corresponding author. Department of Medicine, Ottawa Hospital Research Institute, F Carling Avenue, University of Ottawa, Ottawa, Ontario, Canada K1Y 4E9. Tel.: (613) ; fax: (613) address: carlv@ohri.ca (C. van Walraven). description of a particular measure (or variable) with or without its relationship to another measure. Many of the guidelines that are available to assess the internal validity of observational research [1] apply to administrative database research. However, these studies have several unique issues that also need to be addressed by the writer and evaluated by the reader to establish their internal validity [2]. If these are not addressed or considered, potential threats to the validity of administrative database research may persist. In this article, we discuss five issues that likely should be considered whenever administrative database research is written or read. 2. Description of the data sets used for study In studies using primary data collection, the methods section describes steps taken to collect the data used to create the study analytical data set. Key issues here include the sampling frame and sampling methods as well as the inclusion and exclusion criteria. This information helps readers understand which people were considered for inclusion in the study /$ - see front matter Ó 2012 Elsevier Inc. All rights reserved. doi: /j.jclinepi

2 C. van Walraven, P. Austin / Journal of Clinical Epidemiology 65 (2012) 126e What is new? The administration of health care frequently creates digitized datadsuch as physician service claims, medication prescriptions, and hospitalization abstractsdthat can be used to conduct studies termed administrative database research. While most guidelines to assess the validity of observational studies apply to administrative database research, its unique data source and analytical opportunities create risks that can make the study uninterpretable or biased. Some of these risks can be minimized by providing a robust description of the data tables used in the study describing both how and why they were created; accurately measuring and reporting the accuracy of diagnostic and procedural codes used in the study; distinguishing between clinical and statistical significance; properly accounting for timedependent nature of postbaseline variables; and analyzing clustered data properly to explore its influence on study outcomes. This article reviews each of these aspects regarding administrative database research for both readers and writers to help maximize the utility of administrative data for research. and has important implications for determining the internal and, especially, external validity of the study. In administrative database research, analysts retrieve their study data from an administrative data set rather than a population. Therefore, a description of those administrative data sets is essential for readers to understand the study cohort [2]. The description of an administrative data set starts by explaining why it was created. By definition, these data sets are created for reasons other than research, and these reasons can strongly influence the completeness and accuracy of the data set. Consider a physician claims data set in which each row represents a billable service. Such a data set is created to pay physicians for their activities. For physicians being paid on a fee-for-service basis, one would expect the data set to completely capture their activities (because physicians will not get paid without submitting a claim). In contrast, one would expect the database to be less representative for physicians who are paid through other mechanisms that are not reliant on physician claims. In general, data sets are significantly more complete when those responsible for supplying its information benefit in some way from providing those data. This example highlights that the reasons why an administrative data set was created is relevant to the research conducted with those data. After describing why an administrative data set was created, the writer should focus on how the data set was created. This should clearly identify the population whose data will end up in the data set and highlight populations (potentially relevant to the study at hand) who are excluded from the data set. These descriptions should give readers a clear understanding of the people who were candidates for the study, what each row in the data set represents (eg, a person, hospitalization, medication prescribed), and the key columns (or variables) from the data set used in the study. Describing the steps taken to create the administrative data set will let readers infer the potential for biased or missing information. For example, an administrative data set of all hospital laboratory tests that is created as a download from the laboratory information system would be expected to be highly accurate (because the data do not require any modification) and complete (assuming that all laboratory tests are captured by the laboratory information system). As a result, one would be confident that the data in this administrative data set would accurately and completely reflect the laboratory activity in that hospital. Contrast this situation with an administrative data set, in which diagnoses are represented with codes, used to measure the prevalence of disease X in hospital. For disease X to be captured in such a data set, several steps must be successfully and sequentially completed: (1) the physician must recognize and diagnose the disease; (2) the physician must legibly document disease X in the chart; (3) this documentation must be recognized and correctly interpreted by the health records abstractor (HRA); and (4) the HRA must identify the proper code for disease X. Any break in this chain of processes will result in a true case getting omitted from the data set. In addition, any mistake in this chain when processing another disease could result in the data set mistakenly identifying disease X. In general, the accuracy of specific information in an administrative data set decreases as the steps required to get that information into the data set becomes more numerous, burdensome, or complex. Finally, two other pieces of informationdif applicabled help support the validity of using a data set for research. Any data quality checks routinely conducted on the data set should be cited. These could range from the use of logical checks (eg, identifying cases of men with a procedural code for cesarean delivery) to random chart reabstractions for gauging the reliability of information in the data set. In addition, citing publications of previous research conducted using the same data set reassures readers about its utility for research. 3. Reporting diagnostic and procedural code accuracy using meaningful statistics Administrative data use codes to identify diagnoses or procedures that are often used for research studies. In a systematic review of administrative database research [3], we found that 76% of administrative database studies used diagnostic or procedural codes to define patient cohorts, exposures, or outcomes. HRAs (or, occasionally, physicians) review health records to identify diagnoses and procedures that have been documented therein. They then use standard

3 128 C. van Walraven, P. Austin / Journal of Clinical Epidemiology 65 (2012) 126e131 coding systems to substitute the diagnosis or procedure with a code. The accuracy of the code is influenced by two issues: the validity of the diagnosis or procedure and the association of the code with the documented diagnosis or procedure. The first factor reflects the accuracy of diagnoses or (very occasionally) procedures that are documented in health records. One would expect that diagnostic accuracy may vary between physicians; a diagnosis of hypertrophic cardiomyopathy from a cardiologist s record may be more accurate than that from a family physician. Diagnostic accuracy may also vary between locales; difficult diagnoses that require extensive imaging, invasive investigations, and input from experienced consultants may be more accurate coming from a tertiary referral center than a community hospital. Despite its importance, the issue about validity of documented diagnoses or procedures is infrequently discussed in administrative database research. The association of a code with the documented diagnosis or procedure is more commonly considered. Analytical variables defined by diagnostic or procedural codes are, in essence, surrogate measures of the disease or procedure they represent. To have a chance of being valid, a surrogate measure must be statistically associated with the true entity [4e6]. Therefore, any analysis using diagnostic or procedural codes should measure the association between the code and the real variable. Without this association, one cannot quantify misclassification arising from the use of codes or the amount of subsequent bias. The design used in code validation studies will influence the reliability of code accuracy measures. Three validation study designs are commonly used. 1. Ecological studies compare statistics (eg, disease incidence or event rates) that are measured using the code with other, presumably more reliable, methods. Ecological studies are a crude gauge of code accuracy because they are not conducted at the individual-patient level and are susceptible to ecological bias [7]. 2. Reabstraction studies repeat the medical record abstraction process and compare the code status of the two processes to generate agreement statistics. The code abstraction process is susceptible to both missed cases (when physicians do not accurately diagnose the disease or document it in the chart; when abstractors do not recognize the documentation or assign the incorrect code) and misidentified cases (when physicians document? myocardial infarction in the chart, which is misinterpreted by the abstractors and coded as a true myocardial infarction). Therefore, these studies can produce unreliable accuracy statistics. This is especially true if the second abstractor knows the code status of the original review. 3. Gold standard studies compare the code with a gold standard determination of the condition s status. This can be measured with various methods: (1) a set of standard clinical and laboratory criteria required for disease status, (2) a panel review to consensually determine disease status, or (3) a second data set that contains an accurate measure of disease status (eg, a population-based disease registry). Gold standard studies are by far the strongest validation of codes, but their reliability depends on the criteria and methods used to determine gold standard status for individual patients. In summary, the methods used to measure code accuracy should always be reported whenever they are cited. Readers should be wary of code accuracy statistics that do not mention the validation study design. The statistics used to report code accuracy can be deceptive. We believe that the most understandable statistic is the probability that someone with (or without) a code actually has (or does not have) the condition. This statistic is inherently understood by readers and lets them gauge the impact of code inaccuracy on the study. Positive predictive value (PPV) is the most commonly used statistic to report code accuracy. However, PPV can be deceptive because its value varies significantly by disease prevalence even when code accuracy remains the same. Brenner and Gefeller [8] illustrated how PPVs vary extensively by disease prevalence. Their simulation study showed that PPV for a binary measure (such as the presence or absence of a particular code) dropped from 95% to 60% when disease prevalence decreased from 90% to 20% (even when code accuracy did not change). Therefore, the probability that someone with a particular code truly has the disease decreases when disease prevalence decreases. This is an important issue because most validation studies are done using samples consisting entirely of people with the code. Disease prevalence in these samples will invariably be higher than in samples containing a mixture of people with and without the code. Therefore, disease prevalence is lower in database study populations than in validation studies that measured PPV only. For example, we found that disease prevalence in validation studies that only reported PPV averaged 98% [3]. Because disease prevalence in study populations will be much lower, the probability of someone with the code truly having that disease will be less than the PPV reported in the validation study. Using sensitivity and specificity to measure code accuracy decreases deviation of code accuracy between study and validation populations for two reasons. Sensitivity and specificity vary less with changes in disease prevalence [8]. And, when compared with validation studies measuring PPV, those measuring sensitivity and specificity have a disease prevalence that is closer to clinical study populations [3]. However, sensitivity and specificity are not directly informative because these statistics do not communicate the probability that people with (or without) the code truly have (or do not have) the disease. This issue can be addressed by

4 C. van Walraven, P. Austin / Journal of Clinical Epidemiology 65 (2012) 126e combining sensitivity and specificity to calculate positive and negative likelihood ratios [9]. These can be combined with the baseline odds of disease to calculate the probability that someone with a code truly has the disease using this equation: O LR þ O LR þ þ 1 O Here, O is the odds of disease in the study sample and LR þ is the positive likelihood ratio of the code. Using the negative likelihood ratio in this equation returns the probability that someone without the code truly has the disease. The strength of this method is the way it accounts for disease prevalence when estimating the probability that someone with a code truly has the disease. To measure disease prevalence in the study population, a gold standard method would ideally be used. However, disease prevalence can also be estimated by the code s prevalence if its sensitivity is accurately measured [10,11]. These calculations, however, can produce a negative prevalence when disease prevalence is low [12]. 4. Statistical significance vs. clinical significance The issue of statistical significance vs. clinical significance is not unique to administrative database research. However, these studies often have very large sample sizes, thereby highlighting this issue and making it a recurrent theme in such studies. Table 1 illustrates the influence that study sample size can have on P-values for statistical testing. In this example, two equally sized groups have a very similar baseline prevalence of a binary trait (49.9% vs. 50.1%). Data in the table show that the P-value for the comparison of these proportions meets traditional significance levels (ie, a of 0.05) when the total sample size exceeds 250,000. This example highlights that the tension between statistical significance and clinical significance is prominent in Table 1. Influence of sample size on P-values when comparing two proportions Total sample size Standard normal deviate P-value , , , , , ,000, ,000, For different total sample sizes (first column), this table presents the standard normal deviate (z-value, second column) and associated P-value (third column) between proportions in two independent samples (0.499 and 0.501) of equal size. administrative database research. Writers should avoid interpreting the importance of differences based solely on P-values because small differences do not become more meaningful with additional zeroes in the P-value. Readers should pay special attention to the values of absolute and relative differences between groups, rather than the P-value of those differences, to gauge the importance of these differences. The use of confidence intervals (CIs) should help distinguish between clinical significance and statistical significance. CIs are usually more informative when comparing two populations because they are generated around the absolute or relative difference between those populations. This forces the writer and reader to actually note the differences between the populations, thereby highlighting clinical significance rather than statistical significance. 5. Time-dependent bias Patient-level variables can change value during observation. Such time-dependent variables can be termed baseline immeasurable if their value cannot be determined at baseline. Biased conclusions can occur when these variables are analyzed as if their values were known at the start of patient observation. In this situation, a patient s outcome will influence the value of their time-dependent variable. Consider a binomial (0/1) time-dependent covariate indicating the presence or absence of a particular exposure (eg, prescription of a new medication or undergoing a particular surgery). People having an outcome early in their observation are less likely to have this exposure. As a result, outcome rates in people without the time-dependent exposure are higher than in people with the exposure. This will make the time-dependent exposure appear protective. This has been termed survivor treatment selection bias [13], immortal-time bias [14], and timedependent bias [15]. Consider the example of all elderly patients discharged from an Ontario hospital in 2004 with a primary diagnosis of hip fracture (n 5 5,614). We linked to the provincial drug database to determine if (and when) these people received a prescription for laxatives or antiflatulents. These patients were followed to death or December 31, 2009 (median observation time, 4.2 years; interquartile range, 1.1e5.4), with an overall death rate of 17.2 deaths per 100 personyears observation. A survival analysis that ignores the time-dependent nature of prescribing laxatives and antiflatulents concludes that getting such a prescription is associated with a significantly decreased risk of death (hazard ratio, 0.84; 95% CI, 0.78e0.90). This suggests that people were 16% less likely to die if they were prescribed laxatives or antiflatulents. These results are implausible and result from those dying early never having the chance to get a prescription for laxatives or antiflatulents.

5 130 C. van Walraven, P. Austin / Journal of Clinical Epidemiology 65 (2012) 126e131 Researchers can start to address this problem by analyzing these exposures as time-dependent covariates [16]. Such analyses allow these variables to change value over time so that its correct value is used to calculate the hazard ratio whenever an outcome occurs. In the aforementioned example, a proper time-dependent analysis shows that prescription of laxatives or antiflatulents is associated with a significantly increased risk of death (hazard ratio, 1.93; 95% CI, 1.79e2.08). It should be noted, however, that the use of time-dependent analyses is not a complete panacea for potential biases resulting from time-dependent covariates. This is because factors influencing the interpretation of such models including the temporal form of, and causal relationships between, associations linking the covariate and outcome must be considered [17]. Time-dependent bias is not unique to administrative database research. However, administrative database research certainly accounts for a substantial amount of timedependent bias in the scientific literature. In a systematic review of time-dependent bias in highly cited general medical journals [15], we found 52 studies that were susceptible to time-dependent bias; 14 of these (26.9%) were administrative database research. We believe that time-dependent bias is more likely to occur in administrative database research, as opposed to cohort studies using primary data collection, because the post baseline status of covariates is more easily measured with administrative databases than with primary data collection. Because administrative database research is more likely to have postbaseline covariates in the analytical model, they are more likely to have timedependent bias. 6. Accounting for clustering Study samples derived from health administrative data are frequently subject to clustering. For example, consider a study that consists of patients hospitalized with an acute myocardial infarction (AMI) who were treated by physicians who practice within hospitals [18]. This study consists of data having a three-level structure of AMI patients nested within physicians nested within hospitals. Researchers using health administrative data are frequently interested in determining the association between patient outcomes and characteristics of the patient, provider, and facility. In the aforementioned study, researchers may be interested in estimating the association between patient outcomes (patient mortality and receipt of a prescription for a specific medication) and physician training (cardiologist vs. general internist vs. family physician), physician annual volume of AMI patients, hospital type (academic vs. community), and hospital annual volume of AMI patients. Most applied health researchers are familiar with the use of regression models to determine the association between patient characteristics and outcomes. For instance, logistic regression can be used to determine the association between patient characteristics and receipt of a statin prescription at hospital discharge. Conventional regression models assume that subjects are independent of one another. With clustered data, however, outcomes for subjects within the same cluster are often more similar than those of subjects in a different cluster, even after accounting for subject characteristics [19]. In the aforementioned example, one could anticipate that statin use may vary across physicians independently of patient characteristics. Some physicians will have a high propensity to prescribe statins at discharge, whereas others may have more conservative prescribing practices. Multilevel (also known as hierarchical, random effects, or mixed effects) regression models allow one to account for the clustering of patients within health care providers and facilities when analyzing clustered data [19e22]. In our example, these models accomplish this by incorporating provider-specific and facility-specific random effects. This allows patient outcomes to be correlated within health care providers and facilities. Failure to use analytical methods that account for clustering can result in misleading conclusions. In a prior study, we contrasted results from data consisting of AMI patients nested within treating physicians nested within hospitals with and without the use of multilevel modeling [23]. We noted that the 95% CIs for hospital effects were much wider when multilevel logistic models compared with conventional logistic regression models, that ignored clustering, were used. Furthermore, substantially fewer statistically significant associations between patient outcomes and hospital characteristics were found when multilevel regression models compared with conventional regression models were used. When evaluating research using health administrative data, readers need to carefully assess whether the statistical methods accounted for any clustering that may have been present in the data. 7. Summary Administrative database research can offer extensive opportunities for health-related scientific studies. In this article, we discussed five issues that we believe are especially prominent in administrative database research. It is important that writers of administrative database research address and clarify these issues to avoid confusion and misinformation in readers. References [1] van Walraven C, Hebert PC. A reader s guide to the evaluation of prognostic studies. Postgrad Med J 1996;72:6e11. [2] Schneeweiss S. Understanding secondary databases: a commentary on Sources of bias for health state characteristics in secondary databases. J Clin Epidemiol 2007;60:648e50.

6 C. van Walraven, P. Austin / Journal of Clinical Epidemiology 65 (2012) 126e [3] van Walraven C, Bennett C, Forster AJ. Administrative database research infrequently uses validated diagnostic or procedural codes. J Clin Epidemiol 2011;64:1054e9. [4] Lesko L, Atkinson A Jr. Use of biomarkers and surrogate endpoints in drug development and regulatory decision making: criteria, validation, strategies. Annu Rev Pharmacol Toxicol 2001;41: 347e66. [5] Lin DY, Fleming TR, De Gruttola V. Estimating the proportion of treatment effect explained by a surrogate marker. Stat Med 1997;16:1515e27. [6] Freedman LS, Graubard BI, Schatzkin A. Statistical validation of intermediate endpoints for chronic diseases. Stat Med 1992;11: 167e78. [7] Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI; Anti- Lymphocyte Antibody Induction Therapy Study Group. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Stat Med 2002;21:371e87. [8] Brenner H, Gefeller O. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat Med 1997;16:981e91. [9] Sackett DL, Haynes RB, Guyatt GH, Tugwell P. The interpretation of diagnostic data. Clinical epidemiology. A basic science for clinical medicine. Boston, MA: Little, Brown; 1991:69e152. [10] Rogan WJ, Gladen B. Estimating prevalence from the results of a screening test. Am J Epidemiol 1978;107:71e6. [11] Rahme E, Joseph L. Estimating the prevalence of a rare disease: adjusted maximum likelihood. The Statistician 1998;47:149e58. [12] Berkvens D, Speybroeck N, Praet N, Adel A, Lesaffre E. Estimating disease prevalence in a Bayesian framework using probabilistic constraints. Epidemiology 2006;17:145e53. [13] Glesby MJ, Hoover DR. Survivor treatment selection bias in observational studies: examples from the AIDS literature. Ann Intern Med 1996;124:999e1005. [14] Suissa S, Garbe E. Primer: administrative health databases in observational studies of drug effectsdadvantages and disadvantages. [review]. Nat Clin Pract Rheumatol 2007;3:725e32. [15] van Walraven C, Davis D, Forster AJ, Wells GA. Time-dependent bias due to improper analytical methodology is common in prominent medical journals. J Clin Epidemiol 2004;57:672e82. [16] Cox DR. Regression models and life tables. J R Stat Soc Series B Stat Methodol 1972;34:187e220. [17] Fisher LD, Lin DY. Time-dependent covariates in the Cox proportional-hazards regression model. Ann Rev Public Health 1999;20:145e57. [18] Tu JV, Austin PC, Chan BT. Relationship between annual volume of patients treated by admitting physician and mortality after acute myocardial infarction. JAMA 2001;285:3116e22. [19] Snijders PJ, Boskers R. Statistical treatment of clustered data. Multilevel analysis: an introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage Publications, Inc.; 1999:13e37. [20] Austin PC, Goel V, van Walraven C. An introduction to multilevel regression models. Can J Public Health 2001;92:150e4. [21] Raudenbush SW, Bryk AS. Hierarchical linear models: applications and data analysis methods. Thousand Oaks, CA: Sage Publications, Inc.; [22] Goldstein H. Multilevel statistical models. 2nd ed. London, UK: Edward Arnold; [23] Austin PC, Tu JV, Alter DA. Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003;145:27e35.