Clinical research and epidemiological studies of heart failure. Original Article

Transcription

1 Original Article Classification of Acute Decompensated Heart Failure An Automated Algorithm Compared With a Physician Reviewer Panel: The Atherosclerosis Risk in Communities Study Laura R. Loehr, MD, PhD*; Sunil K. Agarwal, MD, PhD, MPH*; Chris Baggett, PhD; Lisa M. Wruck, PhD; Patricia P. Chang, MD, MHS; Scott D. Solomon, MD; Eyal Shahar, MD, MPH; Hanyu Ni, PhD, MPH; Wayne D. Rosamond, PhD; Gerardo Heiss, MD, PhD Downloaded from by guest on July 5, 2018 Background An algorithm to classify heart failure (HF) end points inclusive of contemporary measures of biomarkers and echocardiography was recently proposed by an international expert panel. Our objective was to assess agreement of HF classification by this contemporaneous algorithm with that by a standardized physician reviewer panel, when applied to data abstracted from community-based hospital records. Methods and Results During , all hospitalizations were identified from 4 US communities under surveillance as part of the Atherosclerosis Risk in Communities (ARIC) study. Potential HF hospitalizations were sampled by International Classification of Diseases discharge codes and demographics from men and women aged 55 years. The HF classification algorithm was automated and applied to 2729 (n= weighted hospitalizations) hospitalizations in which either brain natriuretic peptide measures or ejection fraction were documented (mean age, 75 years). There were 1403 (54%; n=7534 weighted) events classified as acute decompensated HF by the automated algorithm, and 1748 (68%; n=9276 weighted) such events by the ARIC reviewer panel. The chance-corrected agreement between acute decompensated HF by physician reviewer panel and the automated algorithm was moderate (κ=0.39). Sensitivity and specificity of the automated algorithm with ARIC reviewer panel as the referent standard were 0.68 (95% confidence interval, ) and 0.75 (95% confidence interval, ), respectively. Conclusions Although the automated classification improved efficiency and decreased costs, its accuracy in classifying HF hospitalizations was modest compared with a standardized physician reviewer panel. (Circ Heart Fail. 2013;6: ) Key Words: ARIC BNP classification ejection fraction heart failure Clinical research and epidemiological studies of heart failure (HF) have been hindered by the lack of a consensus definition of HF as a recurrent event or end point that is valid, repeatable, and cost-effective. 1 4 The pleomorphic nature of the HF syndrome contributes to the difficulty in defining and classifying HF. HF manifestations can be vague, as well as shared with other conditions that are often comorbid with HF, such as respiratory and renal disease. 5 Thus, the current gold standard for HF classification is expert review of medical records and adjudication, 6,7 although classification of HF by an expert reviewer panel is subject to more misclassification than for events, such as myocardial infarction and stroke. A standardized and repeatable event review by a reviewer panel is expensive and time consuming and thus not practical for most studies, and further difficulties include the use of diverse classification schema. Although the Framingham, modified Boston, and National Health and Nutrition Examination Survey classification schemas are widely used, their relevance to contemporary classifications of HF events is questionable 4 because most extant HF classification schema were created before the clinical use of biomarkers and cardiac imaging in HF diagnosis and care. Furthermore, they largely do not consider whether an HF event is new or decompensated. 2 Editorial see p 621 Clinical Perspective on p 726 To develop a contemporary definition of HF, an international group of cardiovascular clinical trialists (CCTs), biostatisticians, National Institutes of Health scientists, Received November 15, 2012; accepted April 29, From the Departments of Epidemiology (L.R.L., W.D.R., G.H.), Biostatistics (L.M.W.), and Medicine (P.P.C.), University of North Carolina at Chapel Hill, NC; Department of Medicine, Johns Hopkins University, Baltimore, MD (S.K.A.); Health Sciences Division, RTI International, Research Triangle Park, NC (C.B.); Department of Cardiovascular Medicine, Brigham and Women s Hospital, Harvard Medical School, Boston, MA (S.D.S.); Epidemiology Branch, National Heart, Lung, and Blood Institute, National Institute of Health, Bethesda, MD (H.N.); and Epidemiology and Biostatistics Division, University of Arizona, Tucson, AZ (E.S.). *Drs Loehr and Agarwal are joint first authors. The online-only Data Supplement is available at /-/DC1. Correspondence to Laura R. Loehr, MD, PhD, Department of Epidemiology, University of North Carolina, 137 E Franklin St, Suite 306, Chapel Hill, NC. lloehr@ .unc.edu 2013 American Heart Association, Inc. Circ Heart Fail is available at DOI: /CIRCHEARTFAILURE

2 720 Circ Heart Fail July 2013 Downloaded from by guest on July 5, 2018 Table 1. Cardiovascular Clinical Trialists Definitions of Heart Failure for a New Diagnosis, New Event, or Recurrent Event, Followed by the Adapted Automated Algorithm Used in This Study (Adapted Based on the Column 2) New Onset HF as a Diagnosis HF as a New Event* HF as a Recurrent Event Adapted Automated Algorithm History of HF No No Yes Yes or no HF signs and +/ * Yes Yes Yes symptoms Treatment Yes* Yes Yes Yes for HF Imaging and biomarkers Yes Yes No Yes HF indicates heart failure. *For new onset HF as a diagnosis, treatment must be for HF symptoms, but there is no requirement that there be 2 symptoms present as with the other 2 categories. Modified to include all 4 categories of HF as a new event (column 2) to include those with or without a prior history of HF documented in the medical. Outpatient visits are not included in the study sample. regulators, and pharmaceutical industry scientists published recommendations for an updated classification of HF for clinical trials and observational studies of HF. 4 Extending prior HF classifications, biomarker and echocardiographic information was included, and the distinction between 3 types of HF events was emphasized (Table 1). The 3 types of HF events include those with a new diagnosis, a new event without prior HF, or a new event with history of HF. The first 2 groups largely differ by severity and setting of presentation. Henceforth, we refer to this expert panel as the CCT Workshop 4 and to their proposed HF event definition as the HF algorithm. As far as we know, this algorithm has yet to be implemented or evaluated; therefore, we operationalized and automated a modified version of the HF algorithm proposed by the CCT for hospitalized events of HF regardless of history of HF. We examined its performance characteristics on data abstracted by trained personnel from medical records of a population-based sample of HF hospitalizations in 4 US communities. Hospitalizations included all men and women aged 55 years with International Classification of Diseases (ICD) coded discharge diagnoses related to HF during in these areas. 8 We tested the concordance of this automated HF classification algorithm with an established panel of standardized physician reviewers of the Atherosclerosis Risk in Communities (ARIC) study. 8 Methods Automated Classification of HF To examine the applicability and usefulness of the HF algorithm in population settings, we ascertained the HF classification criteria items from hospital medical records and thus applicable to clinical research studies using electronic health records or epidemiological surveillance studies. Accordingly, we deferred classification of HF according to history of HF and instead examined performance of a modified version of the HF algorithm that does not consider HF history. We modified the criteria identified by the CCT as HF as a new event, to achieve wider interest and applicability (Table 1). If a classification algorithm performs sufficiently well, the distinction of events according to their prevalent or incident nature is typically done as an analytic step and not as an event classification category. Furthermore, we did not include death caused by HF. See Methods section in the online-only Data Supplement for details. Given that the purpose of this study is to test the automated algorithm in the real-world setting of hospital medical records, there was neither an echocardiogram reading center nor a central laboratory for the measurement of brain natriuretic peptide (BNP). Hospital records that made no reference to measures of either ejection fraction or BNP/N-terminal-pro- BNP were considered not indicative of HF for the missing measure. Records missing both BNP and ejection fraction measures were excluded to preserve the validity of the comparison. Study Population The ARIC study has conducted population-based retrospective surveillance for coronary heart disease since HF has been a target for community surveillance in ARIC since 2005, based on a sample of hospital discharges in 4 geographically defined areas in the United States, for all residents aged 55 years. 9 Because ARIC began automatically classifying some of the eligible hospitalizations in 2008, we limit this analysis to The 4 ARIC study areas are the city of Jackson, MS; Washington County, MD; 8 northwestern suburbs of Minneapolis, MN; and Forsyth County, NC. In 2005, these 4 regions had an overall population of , aged 55 years. Nonblack and nonwhite race groups are excluded because of small numbers. The institutional review boards from each study site (Wake Forest University, University of Minnesota, Johns Hopkins University, and the University of Mississippi) approved the ARIC study. Ascertainment of Hospitalizations for HF Annually, lists of hospital discharges containing a code from a target list of ICD-Ninth Revision-Clinical Modification (ICD-9-CM) codes were obtained from the hospitals in the 4 ARIC communities (31 hospitals in 2005). See Table I in the online-only Data Supplement for a list of targeted HF ICD-9-CM codes. For 91% of the sample, a 428 for congestive HF was listed as one of the codes. For all community residents, aged 55 years, hospitalizations were sampled using stratified probabilistic sampling by HF ICD-9-CM code, age, sex, race, and area of residence in the community. Sampling probabilities by strata were selected to optimize variance estimates for HF event rates within strata, and based on the prespecified maximum number of events planned for data abstraction. 9 Results are weighted for these sampling probabilities to maintain population estimates for the distribution of ICD codes and other factors that may affect concordance. Abstraction and Classification of HF Events Medical records were abstracted by trained study personnel following a standardized protocol. Each record was first abstracted to answer 6 screening questions for acute decompensated HF (ADHF); if any of the answers were positive a full abstraction ensued. The 6 screening items included mention of any of the following: increasing or new onset shortness of breath, peripheral edema, paroxysmal dyspnea, orthopnea, hypoxia, or HF as a cause for hospitalization. Of all records with a HF ICD code, 36% did not meet the screening criteria and were not abstracted in full and were not included in these analyses. A separate analysis examined the effect of this efficiency-based screening in a subset of 797 medical records, based on a full data abstraction for medical records that would have been screened out. We found that 48% (n=386) had either BNP or a measure of ejection fraction and thus would have qualified for the analysis. Of the 386 medical records with biomarker or imaging information, 11.7% were found to have definite or possible ADHF per ARIC reviewer panel. In comparison, 68% of the records fully abstracted for this study had definite or possible HF by ARIC reviewer panel. Thus, screening before full record abstraction was effective in yielding a low number of false negatives. Full record abstraction using the HF abstraction form comprehensively incorporated the pertinent elements for classification of HF, and history of comorbid conditions as described previously (abstraction form available at 9 A computer-based classification was applied to the abstracted data to arrive at the appropriate classification for the CCT automated

3 Loehr et al Automated Algorithm for Heart Failure 721 Table 2. Adapted Automated Cardiovascular Clinical Trialists Algorithm for a Hospitalized Event of Acute Decompensated Heart Failure (Either New or Recurrent) Downloaded from by guest on July 5, 2018 Signs and symptoms, presence of 2 HF signs, or symptoms among the following Shortness of breath or dyspnea on exertion, orthopnea, paroxysmal nocturnal dyspnea, fatigue or reduced exercise tolerance, pulmonary edema, rales, peripheral edema, JVD, S3, hepatojugular reflux, altered hemodynamics, and cardiomegaly Treatment Initiation or increase in treatment with loop diuretics or intravenous vasoactive agents. The automated algorithm s criteria specify that this treatment should be specifically for the above symptoms; however, our abstraction only confirms that such treatment was provided during this hospitalization Biomarkers and imaging, 1 of the following 1. Elevated BNP ( 400 ng/l*) or elevated NT-pro-BNP using age-defined cut points Or 2. LVEF <40% Or 3. Moderately elevated BNP ( ng/l) or NT-pro-BNP (defined as less than age cut points) and documentation of LVEF <40% or diastolic dysfunction All 3 criteria elements must be met to define a heart failure event. BNP indicates brain natriuretic peptide; HF, heart failure; JVD, jugular venous distention; LVEF, left ventricular ejection fraction; and NT, N-terminal. *SI units shown of ng/l=pg/ml. Elevated NT-pro-BNP defined as: if <50 y then 450 ng/l; if y then 900 ng/l; and if 75 y then 1800 ng/l. Moderately elevated NT pro-bnp defined with 300 ng/l as the bottom cut point for all age groups: if <50 y then ng/l; if y then ng/l; and if 75 y then ng/l. algorithm (Tables 1 and 2). Secondarily, conventional HF criteria (Framingham, 10 Boston, 11 National Health and Nutrition Examination Survey, 12 and Gothenburg) 13 were also defined from abstracted data (results are presented in the online-only Data Supplement). 9 Eligible hospitalizations were independently reviewed by 1 or 2 trained physician reviewer(s) with resolution of disagreements by an adjudicator. Physicians followed ARIC HF classification guidelines when evaluating medical records, and applied judgment to arrive at a classification of definite ADHF, possible ADHF, chronic HF, HF unlikely, or unclassifiable. 9 Here, definite and possible ADHF have been combined into a single category of ADHF present, and the other 3 categories have been combined as ADHF absent. Classification of HF Events in ARIC The ARIC classification guidelines have been described. 9 Classification of definite ADHF required clear evidence of HF with active decompensation and the presence of HF with certainty as to the cause of the presentation. Possible ADHF included criteria similar to definite ADHF, without as much certainty that HF is the cause of the presentation. A classification of chronic HF applied to a history of HF that was not decompensated. Statistical Analysis All estimates were weighted to account for the sampling design and to maintain the population distribution of ICD codes and other factors that may affect concordance. We cannot reliably link hospitalizations to identify repeat events; therefore, all hospitalizations are assumed to be independent. The positive and negative agreement, the κ coefficient, and the prevalence- and bias-adjusted κ were calculated relating the automated algorithm to the ARIC reviewer panel classifications. The prevalence- and bias-adjusted κ were calculated because the prevalences of positive and negative tests were not balanced, which can result in a κ with low reliability even when observed agreement is good. 14,15 Measures of validity were calculated for the components of the automated algorithm individually and for the schema overall. Positive and negative predictive values were calculated for several different disease prevalences. 16 Formulas are specified in table footnotes. Results There were 2729 sampled hospitalizations eligible for review during , which resulted in events after applying weights to account for sampling fractions. The tables and their discussion refer to the weighted number of events. Of these, 10.5% (n=1630 weighted) were missing BNP measures Table 3. Hospitalizations* During Among Residents Aged 55 Years From 4 US Communities, Identified as Possible Heart Failure, According to the Automated Heart Failure Classification Algorithm and the Atherosclerosis Risk in Communities Reviewer Panel ARIC Panel HF Classification Numbers Before Exclusion, n* (%) Automated Algorithm Numbers After Exclusion, Due to Missing BNP and Ejection Fraction, n* (%) ADHF Present, n* (%) ADHF Absent, n* (%) ADHF present Definite HF 6888 (45%) 6813 (49%) 5211 (69%) 1602 (25%) Possible HF 2864 (19%) 2563 (19%) 1200 (16%) 1363 (22%) ADHF absent Chronic HF 2005 (13%) 1738 (13%) 464 (6%) 1274 (20%) Not HF 2389 (15%) 1811 (13%) 358 (5%) 1453 (23%) Unclassifiable 1338 (8%) 929 (7%) 301 (4%) 629 (10%) Total (100%) (100%) 7534 (100%) 6321 (100%) ADHF indicates acute decompensated heart failure; BNP, brain natriuretic peptide; and HF, heart failure. *All numbers were weighted to account for sampling fractions. Overall 10.5% excluded because of missing values for BNP or ejection fraction.

4 722 Circ Heart Fail July 2013 Downloaded from by guest on July 5, 2018 and ejection fraction and thus were excluded, leaving a sample of for this analysis. Of those classified as ADHF by the automated algorithm, 85% (69%+16%) were classified as definite or possible ADHF by the ARIC reviewer panel (Table 3, with unweighted numbers in Table II in the online-only Data Supplement). Of those classified as not having ADHF by the automated algorithm, 47% were classified as ADHF and 20% as chronic HF by ARIC panel review. Overall, characteristics of patients with ADHF per the ARIC reviewer panel and the automated algorithm did not differ appreciably (Table III in the online-only Data Supplement). In each group the mean age was 75 years, with 51% to 52% women, and 28% to 30% blacks. Hypertension (83%) and diabetes mellitus (46% to 48%) were common for both groups. Table 4 shows the characteristics of those classified with agreement and disagreement when comparing the automated algorithm with the ARIC reviewer panel. The overt differences between groups were few, but informative. The frequency of end stage renal disease was highest (34%) in those without ADHF by both criteria, and then next highest (21%) for those with ADHF per the automated algorithm, and not by ARIC reviewer panel. The mean levels of BNP and N-terminal-pro-BNP were visibly lower in the group classified as ADHF absent by the automated algorithm but present according to the ARIC reviewer panel. Those given diuretics were more likely to be classified with agreement as ADHF present (85% of those correctly classified, as compared with 55% to 69% for those misclassified). Table 5 shows measures of test validity calculated for the automated algorithm and its components, compared with the ARIC reviewer panel as a referent. The sensitivity was 0.68 and specificity 0.75 for the automated algorithm overall, with a positive predictive value of 0.85 and negative predictive value of The prevalence of ADHF was 68% in this enriched sample of hospitalized events. Because predictive values differ according to prevalences, we calculated predictive values for lower disease prevalences (eg, for a prevalence of HF in the sample of 25%, the positive predictive value=0.48 and negative predictive Table 4. Characteristics of Hospitalizations According to Agreement Between the Automated Heart Failure Classification Algorithm and the Atherosclerosis Risk in Communities Reviewer Panel for the Classification of Acute Decompensated Heart Failure for Hospitalizations Identified as Eligible for Review as Possible Heart Failure Disagreement Agreement ARIC reviewer panel ADHF present ADHF absent ADHF present ADHF absent Automated algorithm ADHF absent ADHF present ADHF present ADHF absent Weighted* number n=2966 n=1123 n=6411 n=3356 Demographics (%, unless stated) Age, mean (SD), y 75 (24) 76 (23) 75 (24) 73 (21) Race, black Women Teaching hospital Comorbidities, % Coronary heart disease Diabetes mellitus Hypertension COPD End stage renal disease Atrial fibrillation Heart block or other bradycardia HF signs and symptoms, % 2 HF signs and symptoms Biomarkers and imaging BNP level, ng/l, mean (SD) 376 (1400) 1084 (2764) 1846 (6849) 178 (614) NT-pro-BNP level, ng/l, mean (SD) 3075 (15 896) 7544 (19 492) (21 162) 725 (1730) Ejection fraction, %, mean (SD) 46 (35) 51 (30) 39 (38) 53 (28) Ejection fraction <40% Treatment, % Diuretics (at admission or during) Intravenous inotropes Intravenous diuretics Intravenous diuretics or inotropes ADHF indicates acute decompensated heart failure; BNP, brain natriuretic peptide; COPD, chronic obstructive pulmonary disease; HF, heart failure; and NT, N-terminal. *All numbers were weighted to account for sampling fraction. Hospitalizations were screened and include only those that mention 1 of 6 signs or symptoms of ADHF.

5 Loehr et al Automated Algorithm for Heart Failure 723 Table 5. Sensitivity, Specificity, Positive and Negative Predictive Values, and Likelihood Ratios Positive and Negative for the Modified Automated Algorithm s Classification and Its Components, Compared With a Referent Standard of Acute Decompensated Heart Failure (Definite or Possible) Classification by Atherosclerosis Risk in Communities Reviewer Downloaded from by guest on July 5, 2018 Elements of Automated HF Algorithm Weighted* Number in Analysis Sensitivity Specificity value=0.88). As for the individual components of the algorithm, notably, elevated BNP or N-terminal-pro-BNP taken in isolation showed comparable levels of validity to the algorithm overall (a sensitivity of 0.78 and specificity of 0.64), although this represents a smaller group (81% of the sampled hospital records) with nonmissing biomarkers. In Table 6 (also in Table IV in the online-only Data Supplement), the agreement and validity statistics for ADHF by the ARIC reviewer panel were compared with the automated algorithm. The prevalence- and bias-adjusted of κ does not suggest a large influence of internal imbalance in these data on the κ statistic. Discussion We assessed the applicability and classification properties of an algorithm proposed for the classification of HF end points in clinical trials or observational studies, which incorporates diagnostic tools routinely used in current medical practice. We evaluated the performance of this algorithm in the setting of hospitalizations sampled from a large number of hospitals from 4 regions of the United States that participate in a National Heart, Lung, and Blood Institute sponsored epidemiology study of HF. The evaluation of an automated HF algorithm that incorporates biomarkers and echocardiographic measures is novel in the context of a large, populationbased sample of hospitalizations and is notable for its scope and generalizability. In addition to signs and symptoms as elements of the HF syndrome, the availability of echocardiographic imaging and biomarker information abstracted from records generated in the course of routine medical care indicate that an application of an automated algorithm is feasible under these circumstances and was successful. We found that 89.5% of hospital medical records sampled during the period contained either BNP/N-terminal-pro-BNP or Positive Predictive Value Negative Predictive Value Likelihood Ratio Positive Likelihood Ratio Negative HF signs and symptoms ( 2) Intravenous diuretics or inotropes Diastolic dysfunction Systolic dysfunction High BNP or NT-pro-BNP Automated algorithm Formulas used in calculations: sensitivity=a/a+c; specificity=d/b+d; PPV=a/(a+b); NPV=d/c+d. Using a 2 2 table with ARIC as the gold standard, a to d are defined as follows: a=+aric,+trialist; b= ARIC,+automated algorithm; c=+aric, automated algorithm; and d= ARIC, automated algorithm. Positive likelihood ratio=(sensitivity)/(1 specificity)=tp/fp. Negative likelihood ratio=(1 sensitivity)/(specificity)=fn/tn. BNP indicates brain natriuretic peptide; FN, false negative; FP, false positive; HF, heart failure; NPV, negative predictive value; NT, N-terminal; PPV, positive predictive value; TN, true negative; and TP, true positive. *All numbers were weighted to account for sampling fractions. The positive and negative predictive values vary as a function of disease prevalence in the population. The prevalence of HF by definite or acute decompensated HF by ARIC reviewer panel=9376/13 854=0.68. To calculate the PPV and NPV for populations with different disease prevalences, use the following formulas. Formulas: PPV=(sensitivity prevalence)/{(sensitivity prevalence)+[(1 specificity) (1 prevalence)]}; NPV=(specificity 1 prevalence)/[(1 sensitivity) prevalence]+[specificity (1 prevalence)]. Thus for a prevalence of 0.5, the PPV=0.73 and NPV=0.71; and for a prevalence of 0.4, the PPV=0.64 and the NPV=0.78. echocardiographic measures suitable for use in applying this algorithm. Furthermore, by adding detail and some modifications to the definitions published by Zannad et al, 4 we were able to operationalize an algorithmic definition for HF. The ability to apply an automated algorithm to real-world settings and electronic health record highlights the potential efficiencies in the classification of HF events for research and administrative applications based on hospital medical records. To our knowledge this is the first study to assess the classification performance of this algorithm and its validity relative to a standardized HF classification method by a panel of physician reviewers. Because HF is a clinical syndrome for which there is no consensus definition, its classification is difficult. Additional complexity is added by the episodes of acute decompensation that characterize HF. This study focused on an accurate and reproducible algorithmic classification of ADHF. On the basis of (weighted) hospitalizations sampled during from all hospitals that serve the residents of 4 regions in the United States, we found modest agreement between ADHF defined by the automated algorithm and by the ARIC reviewer panel (κ of 0.39, prevalence- and bias-adjusted κ of 0.41). Chance-adjusted agreement, as measured by Cohen s κ, was slightly higher here than agreement between existing HF criteria and the ARIC reviewer panel as shown in a prior publication (Framingham K=0.32, Modified Boston K=0.18). 9 Given that the ARIC HF panel reviewers considered BNP measures and echocardiography findings in classifying HF events, we expected the automated algorithm (which includes criteria elements for these measures) to have better agreement with ARIC s ADHF than the other schema considered which do not consider these measures. In addition, existing HF schemas do not distinguish ADHF from chronic HF, whereas the automated algorithm and the ARIC reviewer classification do.

6 724 Circ Heart Fail July 2013 Downloaded from by guest on July 5, 2018 Table 6. Measures of Agreement and Validity for the Classification of Acute Decompensated Heart Failure Using the Automated Algorithm as Compared With a Referent Standard of the Atherosclerosis Risk in Communities Reviewer panel (n=13855*) ARIC Reviewer Panel ADHF Present, n ADHF Absent, n Automated algorithm, ADHF present Automated Algorithm, ADHF absent Measures of agreement Positive agreement 0.76 Negative agreement 0.62 Cohen s κ 0.39 (0.38, 0.41) Prevalence- and bias-adjusted κ 0.41 Prevalence index 0.22 Bias index 0.13 Positive agreement 0.76 Negative agreement 0.62 Cohen s κ 0.39 (0.38, 0.41) Prevalence- and bias-adjusted κ 0.41 Prevalence index 0.22 Bias index 0.13 Measures of validity with ARIC Reviewer Panel as referent standard Sensitivity 0.68 (0.67, 0.69) Specificity 0.75 (0.74, 0.76) Positive predictive value 0.85 (0.84, 0.86) Negative predictive value 0.53 (0.52, 0.54) Likelihood ratio positive 2.72 (2.59, 2.87) Likelihood ratio negative 0.43 (0.41, 0.44) Formulas used in calculations: using a 2 2 table with ARIC as the gold standard, a to d are defined as follows: a=+aric,+automated algorithm; b= ARIC,+automated algorithm; c=+aric, automated algorithm; and d= ARIC, automated algorithm. Sensitivity=a/a+c; specificity=d/b+d; PPV=a/(a+b); and NPV=d/c+d. Positive likelihood ratio=(sensitivity)/(1 specificity)=tp/fp. Negative likelihood ratio=(1 sensitivity)/(specificity)=fn/tn. PABAK formula=2 (observed agreement 1). The observed agreement=(a+d)/n. The prevalence of ADHF by ARIC=9376/13 854=0.68. ADHF indicates acute decompensated heart failure; ARIC, atherosclerosis risk in communities; FN, false negative; FP, false positive; NPV, negative predictive value; PABAK, prevalence- and bias-adjusted κ; and PPV, positive predictive value; TN, true negative; and TP, true positive. *All numbers were weighted to account for sampling fractions. Using ARIC s classification of ADHF as a referent standard, we found a sensitivity of 0.68 (95% confidence interval, ) for the automated algorithm (ie, 68% of those with ADHF by the ARIC reviewer panel [reference standard] were also found to have ADHF per automated algorithm). Thus, 32% of those with ADHF were missed as false negatives applying the automated algorithm. Specificity was estimated as 0.75 (95% confidence interval, ), implying that 75% of those who did not have ADHF by the reference standard also were found not to have ADHF by the automated algorithm (true negatives), and 25% of those without HF were false positives according to the automated algorithm. Using the ARIC reviewer panel as the referent standard, the automated algorithm performed with higher specificity, and lower sensitivity compared with other commonly used HF classification schema (Table V in the online-only Data Supplement). The automated algorithm did not perform at higher validity on both sensitivity and specificity when compared with the existing criteria, thus the relative value of sensitivity and specificity, and the cost of each type of misclassification need to be considered in the particular setting for which the classification of HF events is needed. The varied settings in which HF classification may be applied include the identification of potential participants with HF for a clinical trial, the identification of HF as an adverse events, and casefinding efforts that search through large databanks of electronic medical records. Overall the automated algorithm had a better balance of sensitivity and specificity than any 1 individual component. The overall balance between sensitivity and specificity for the automated algorithm was closest to that for the BNP levels as an individual criterion element, although the biomarkers achieved higher sensitivity than specificity. Of note, hospital records containing BNP measures may reflect a different spectrum of disease or patient population than the overall sample of hospitalized residents of these study areas. Because biomarkers and echocardiographic measurements may be performed differentially in clinical settings, the automated algorithm is not likely to perform as well in circumstances where these measures would not be obtained routinely. Furthermore, our focus is the performance of this algorithm in real-world settings, and thus we did not limit the analysis to those with both measures. We would expect different results in a population that had both biomarkers and echocardiogram measures performed during a hospitalization, but both of these measures are not routinely obtained in HF hospitalizations. Our study material included only hospitalizations lasting 24 hours. This may be a limitation in that milder cases of ADHF that could have been managed in the emergency department or admitted overnight to observation care would mostly be missed here. It is, therefore, unknown how the automated HF classification algorithm performs on data that include milder forms of ADHF. An additional limitation is that eligible hospital records with a qualifying ICD code for HF were abstracted only in part when the record did not include reference to increasing, or new onset shortness of breath, peripheral edema, paroxysmal dyspnea, orthopnea, hypoxia, or documentation that the reason for the event was HF. Across all hospitals included in this study, 36% of medical records did not meet the above screening criteria and were not abstracted in full. A calibration substudy of hospital records that did not meet these inclusion criteria and were fully abstracted (n=797) found that only 48% of those records that were screened out would have met criteria for classification using this automated algorithm, and that only 11.7% of those were identified as ADHF by ARIC. Thus, the impact of the criterion to select hospital records eligible for full abstraction on the results reported here is, therefore, quite small. Finally, as expected in the setting of communitybased hospitals, the biomarker assays and echocardiography measurements were not interpreted in a central reading center

7 Loehr et al Automated Algorithm for Heart Failure 725 Downloaded from by guest on July 5, 2018 or laboratory; therefore, some (unmeasured) variability is to be expected. Further studies should assess this algorithm in other settings, such as in a clinical trial, in which the goal is usually to define HF end points of ADHF in those known to have HF. Although centrally analyzed biomarker levels and centrally read imaging may be available from most clinical trials at baseline, it is relevant to note that many large multicenter clinical trials with HF hospitalization as an end point are also dependent on the clinical infrastructure for imaging and biomarker measures, in place of a centralized processing of these measures. 4 Among the strengths of this report are the novelty of the application of an automated algorithm for the classification of HF in a population-based setting and the rigorous evaluation of its performance characteristics in contemporaneous hospital-based practice. Additional strengths include the use of a large database of hospital records sampled to represent hospitalizations among the residents of 4 regions and their abstraction by trained study personnel following a standardized protocol. Because as of yet, there is no agreed-on gold standard to classify HF, a systematic physician review and classification according to standardized criteria represents the best available gold standard. Our reliance on a comprehensive and standardized protocol for the classification of ADHF that included a panel of calibrated physician reviewers adds strengths to the information reported here. In conclusion, we were able to apply an algorithm recommended by an international panel of experts for the classification of HF to medical records sampled from diverse hospitals in geographically well-defined areas in the United States, and to automate this algorithm efficiently for use on data abstracted from records by trained personnel. The validity (accuracy) of the automated algorithm for ADHF was moderate at best compared with the classification of ADHF by ARIC s reviewer panel, although the agreement and specificity for the automated algorithm were greater than for the commonly used HF criteria that do not account for contemporary measures of BNP or echocardiography (Table V in the online-only Data Supplement). The development of HF classification criteria that agree with the highest reference standard of physician reviewer classification and their evaluation in the setting of medical practice are priorities for clinical and population-based research. If such an algorithm is used to classify all hospital admissions rather than those with high prior odds of HF as done in this study, then the concordance will be extremely high, as most records will not have HF by either criterion. Diastolic dysfunction, a common finding in the elderly population without HF, does not contribute much to the ability to classify HF. Unlike with systolic dysfunction, the CCT algorithm requires that those with diastolic dysfunction must also have moderately elevated biomarkers to meet criteria for ADHF. It is possible that uniform measurements of diastolic parameters, which are not often reported in clinical echocardiograms, and research to define the appropriate set of parameters to best define diastolic dysfunction may improve its use for classification. The ability to classify ADHF with an up-to-date, automated classification algorithm and evaluate its performance characteristics is a critical step toward the establishment and standardized application of consensus criteria for HF. Advantages derived from their use would apply to the use of large medical records database resources, as well as efficiencies in time and costs. Acknowledgments We thank the staff and participants of the Atherosclerosis Risk in Communities study for their important contributions. Sources of Funding The Atherosclerosis Risk in Communities Study is performed as a collaborative study supported by National Heart, Lung, and Blood Institute contracts (HHSN C, HHSN C, HHSN C, HHSN C, HHSN C, HHSN C, HHSN C, and HHSN C). None. Disclosures References 1. Vasan RS, Levy D. Defining diastolic heart failure: a call for standardized diagnostic criteria. Circulation. 2000;101: Mosterd A, Deckers JW, Hoes AW, Nederpel A, Smeets A, Linker DT, Grobbee DE. Classification of heart failure in population based research: an assessment of six heart failure scores. Eur J Epidemiol. 1997;13: Di Bari M, Pozzi C, Cavallini MC, Innocenti F, Baldereschi G, De Alfieri W, Antonini E, Pini R, Masotti G, Marchionni N. The diagnosis of heart failure in the community. Comparative validation of four sets of criteria in unselected older adults: the ICARe Dicomano Study. J Am Coll Cardiol. 2004;44: Zannad F, Stough WG, Pitt B, Cleland JG, Adams KF, Geller NL, Torp- Pedersen C, Kirwan BA, Follath F. Heart failure as an endpoint in heart failure and non-heart failure cardiovascular clinical trials: the need for a consensus definition. Eur Heart J. 2008;29: Rutten FH, Cramer MJ, Lammers JW, Grobbee DE, Hoes AW. Heart failure and chronic obstructive pulmonary disease: An ignored combination? Eur J Heart Fail. 2006;8: Heckbert SR, Kooperberg C, Safford MM, Psaty BM, Hsia J, McTiernan A, Gaziano JM, Frishman WH, Curb JD. Comparison of self-report, hospital discharge codes, and adjudication of cardiovascular events in the Women s Health Initiative. Am J Epidemiol. 2004;160: Schellenbaum GD, Heckbert SR, Smith NL, Rea TD, Lumley T, Kitzman DW, Roger VL, Taylor HA, Psaty BM. Congestive heart failure incidence and prognosis: case identification using central adjudication versus hospital discharge diagnoses. Ann Epidemiol. 2006;16: White AD, Folsom AR, Chambless LE, Sharret AR, Yang K, Conwill D, Higgins M, Williams OD, Tyroler HA. Community surveillance of coronary heart disease in the Atherosclerosis Risk in Communities (ARIC) Study: methods and initial two years experience. J Clin Epidemiol. 1996;49: Rosamond WD, Chang PP, Baggett C, Johnson A, Bertoni AG, Shahar E, Deswal A, Heiss G, Chambless LE. Classification of heart failure in the atherosclerosis risk in communities (ARIC) study: a comparison of diagnostic criteria. Circ Heart Fail. 2012;5: Ho KK, Anderson KM, Kannel WB, Grossman W, Levy D. Survival after the onset of congestive heart failure in Framingham Heart Study subjects. Circulation. 1993;88: Carlson KJ, Lee DC, Goroll AH, Leahy M, Johnson RA. An analysis of physicians reasons for prescribing long-term digitalis therapy in outpatients. J Chronic Dis. 1985;38: Schocken DD, Arrieta MI, Leaverton PE, Ross EA. Prevalence and mortality rate of congestive heart failure in the United States. J Am Coll Cardiol. 1992;20: Eriksson H, Caidahl K, Larsson B, Ohlson LO, Welin L, Wilhelmsen L, Svärdsudd K. Cardiac and pulmonary causes of dyspnoea validation of

8 726 Circ Heart Fail July 2013 a scoring test for clinical-epidemiological use: the Study of Men Born in Eur Heart J. 1987;8: Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol. 1993;46: Cunningham M. More than just the kappa coefficient: A program to fully characterize inter-rater reliability between two raters. Proceedings of the sas global forum 2009 conference. Cary, NC: SAS Institute Inc.; Altman DG, Bland JM. Diagnostic tests 2: Predictive values. BMJ. 1994;309:102. Downloaded from by guest on July 5, 2018 CLINICAL PERSPECTIVE Clinical research and epidemiologic studies of heart failure (HF) are hindered by the lack of a consensus definition of HF as an endpoint that is valid, repeatable, and cost-effective. While classification of the HF syndrome is challenging, a standardized and repeatable event review by a physician reviewer panel is costly and not practical in most settings. Existing HF classification criteria are widely used; however, they were created prior to the clinical use of biomarkers and cardiac imaging in HF diagnosis and care. An algorithm to classify HF endpoints for research was recently proposed by an international expert panel and includes contemporary measures of biomarkers (BNP, beta natriuretic peptides) and echocardiography. Our objective was to assess agreement of HF classification by this contemporaneous algorithm with that by a standardized physician reviewer panel from the Atherosclerosis Risk in Communities (ARIC) study. Data were abstracted by trained personnel from medical records of a population-based sample of HF hospitalizations in 4 areas of the United States. Hospitalizations included all men and women aged 55 years and older with ICD-coded discharge diagnoses related to HF during 2005 to The ability to classify acute decompensated heart failure with an up-to-date, automated classification algorithm would reduce research costs and enable the utilization of large medical records databases. Although the automated classification improved efficiency and decreased costs, its accuracy in classifying HF hospitalizations was modest compared to a standardized physician reviewer panel.

9 Downloaded from by guest on July 5, 2018 Classification of Acute Decompensated Heart Failure: An Automated Algorithm Compared With a Physician Reviewer Panel: The Atherosclerosis Risk in Communities Study Laura R. Loehr, Sunil K. Agarwal, Chris Baggett, Lisa M. Wruck, Patricia P. Chang, Scott D. Solomon, Eyal Shahar, Hanyu Ni, Wayne D. Rosamond and Gerardo Heiss Circ Heart Fail. 2013;6: ; originally published online May 6, 2013; doi: /CIRCHEARTFAILURE Circulation: Heart Failure is published by the American Heart Association, 7272 Greenville Avenue, Dallas, TX Copyright 2013 American Heart Association, Inc. All rights reserved. Print ISSN: Online ISSN: The online version of this article, along with updated information and services, is located on the World Wide Web at: Data Supplement (unedited) at: Permissions: Requests for permissions to reproduce figures, tables, or portions of articles originally published in Circulation: Heart Failure can be obtained via RightsLink, a service of the Copyright Clearance Center, not the Editorial Office. Once the online version of the published article for which permission is being requested is located, click Request Permissions in the middle column of the Web page under Services. Further information about this process is available in the Permissions and Rights Question and Answer document. Reprints: Information about reprints can be found online at: Subscriptions: Information about subscribing to Circulation: Heart Failure is online at:

10 Supplemental Material Supplementary Methods The HF abstraction form is available on the ARIC website ( We adapted the category called Heart Failure as a New Event as shown in Table 1 to any event of ADHF, regardless of prior history of HF. Several specifications were required in order to implement the automated algorithm in our setting. First, we specified a definition for altered hemodynamics from the list of signs and symptoms of the algorithm (SBP > 180 mmhg or < 90 mmhg, DBP > 100 mmhg or HR > 110 bpm or < 40 bpm). Second, because medical records rarely document whether treatment was indicated specifically for the symptoms of HF, the automated algorithm treatment criterion was operationalized as therapy with diuretics or inotropes. Third, we specified a bottom cut point for moderate elevations of NT pro-bnp associated with diastolic dysfunction (300 pg/ml for all age-groups). In addition, we defined diastolic dysfunction as documented in the medical record from a prior echocardiogram, from the current echocardiogram, or by a physician and thus did not require the specific criteria recommended in the algorithm (ie E/A > 1).

11 Supplemental Table 1. ICD-9 codes that were eligible for probabilistic sampling Rheumatic heart disease Hypertensive heart disease-malignant with congestive heart failure Hypertensive heart disease-benign with congestive heart failure Unspecified hypertensive heart disease with congestive heart failure Hypertensive heart disease and renal failure-unspecified with congestive heart and renal failure Acute cor pulmonale Chronic pulmonary heart disease, unspecified Other primary cardiomyopathies 428 Congestive heart failure Acute edema of lung, unspecified Dyspnea and respiratory abnormalities

12 Supplemental Table 2. Unweighted number of hospitalizations* during among residents ages 55 years and older of four communities in the U.S., identified as eligible for review as possible heart failure, according to the automated HF classification algorithm and the ARIC reviewer panel Automated algorithm ARIC panel HF Numbers before Numbers after exclusion due to ADHF present, ADHF absent classification exclusion, missing BNP and ejection fraction, N * (%) N * (%) N * (%) N * (%) ADHF present Definite HF 1,276 (41%) 1,263 (46%) 972 (69%) 291 (22%) Possible HF 532 (17%) 483 (18%) 220 (16%) 263 (20%) ADHF absent Chronic HF 398 (13%) 350 (13%) 85 (6%) 265 (20%) Not HF 626 (15%) 452 (17%) 73 (5%) 379 (29%) Unclassifiable 266 (9%) 181 (7%) 53 (4%) 128 (10%) Total 3,098 (100%) 2,729 (100%) 1,403 (100%) 1,326 (100%) * The numbers reported here are unweighted and therefore do not reflect the population distribution defined for this analysis. Overall 12% of the unweighted numbers were excluded due to missing values for BNP or ejection fraction

13 Supplemental Table 3. Characteristics of hospitalizations, by presence of Acute Decompensated Heart Failure (ADHF) according to Cardiovascular Clinical Trailists (CCT) algorithm and ARIC classification Characteristics Demographics (%, unless stated) ARIC Community Hospitalized ADHF by CCT algorithm (N= 7,534)* Definite or Possible Hospitalized ADHF by ARIC classification (N= 9,377)* Jackson, Miss Forsyth County, NC Minneapolis, MN Washington County, MD Age, mean (SD) 75 (24) 75 (24) African-American Women Teaching Hospital Comorbidities (%) Coronary heart disease 47 47

14 Diabetes mellitus Hypertension COPD End stage Renal Disease Atrial fibrillation Heart block or other bradycardia 6 5 HF Signs and Symptoms (%) 2 HF signs and symptoms Biomarkers and Imaging BNP level, mean (SD) 1,736 (6,450) 1,517 (6,213) -NT-proBNP level, mean (SD) 9,751 (20,951) 9,347 (21,063) BNP > 100 (%) Ejection Fraction, mean (SD) 40 (38) 41 (38) Ejection Fraction < 40% Treatment (%) Diuretics IV inotropes 10 9 IV diuretics 81 80

15 IV diuretics or inotropes *All numbers were weighted to account for sampling fractions