Short and Precise Patient Self-Assessment of Heart Failure Symptoms Using a Computerized Adaptive Test

Transcription

1 Short and Precise Patient Self-Assessment of Heart Failure Symptoms Using a Computerized Adaptive Test Matthias Rose, MD, PhD; Milena Anatchkova, PhD; Jason Fletcher, PhD; Arthur E. Blank, PhD; Jakob Bjørner, MD, PhD; Bernd Löwe, MD, PhD; Thomas S. Rector, PhD; John E. Ware, PhD Background Assessment of dyspnea, fatigue, and physical disability is fundamental to the monitoring of patients with heart failure (HF). A plethora of patient-reported measures exist, but most are too burdensome or imprecise to be useful in clinical practice. New techniques used for computer adaptive tests (CATs) may be able to address these problems. The purpose of this study was to build a CAT for patients with HF. Methods and Results Item banks of 74 queries ( items ) were developed to assess self-reported physical disability, fatigue, and dyspnea. All queries were administered to 658 adults with HF to build 3 item banks. The resulting HF-CAT was administered to 100 patients with ancillary HF (New York Heart Association I, 11%; II, 53%; III and IV, 36%). In addition, the physical function and vitality domains of the SF-36 Health Survey questionnaire, an established shortness-of-breath scale, and the Minnesota Living with Heart Failure Questionnaire were applied. The HF-CAT assessment took 3:09 1:52 minutes to complete and score. All HF-CAT scales demonstrated good construct validity through high correlations with the corresponding SF-36 Health Survey physical function (r 0.87), vitality (r 0.85), and shortness-of-breath (r 0.84) scales. Simulation studies showed a more precise measurement of all HF-CAT scales over a larger range than comparable static tools. The HF-CAT scales identified significant differences between patients classified by the New York Heart Association symptom criteria, similar to the Minnesota Living with Heart Failure Questionnaire. Conclusions A new CAT for patients with HF was built using modern psychometric methods. Initial results demonstrate its potential to increase the feasibility and precision of patient self-assessments of symptoms of HF with minimized respondent burden. Clinical Trial Registration URL: Unique identifier: 1R43HL (Circ Heart Fail. 2012;5: ) Key Words: heart failure patient-reported outcomes computer adaptive tests The cardinal manifestations of heart failure (HF) are dyspnea and fatigue, limited tolerance of physical activity, fluid retention, pulmonary congestion, and peripheral edema. Therefore, HF is a clinical diagnosis that is largely based on physical examination and a careful history about typical subjective symptoms in the presence of cardiac dysfunction. 1 A patient-centered measurement approach is particularly important in HF, to provide clinicians with tools to help them to monitor the syndrome, to compare improvements under different forms of therapy, and to identify risk of deterioration. The New York Heart Association (NYHA) classification has been used for this purpose, but it is being criticized for its questionable reliability 2,3 and is rarely used outside clinical studies or specialized units. Clinical Perspective on p 339 Generally, patient self-assessments have been the more reliable assessments of subjective symptoms, which is one reason for a growing interest in subjective health status measures from the scientific community, clinical practitioners, and industry. 4,5 Self-assessed symptoms are used to predict declines in health status of patients with HF, 6 total expenses for HF care, 7 hospitalization, or even mortality. 8,9 Their widespread use has been recommended to increase quality of care, 10 and 30% of all new drug developments use patient-reported outcomes (PROs) as their primary or coprimary end point. 11 However, with traditional methods, a comprehensive and reliable static measure is likely to be long and timeconsuming to administer and score. If questionnaire data need Received June 28, 2011; accepted March 20, From the Department of Quantitative Health Sciences, University of Massachusetts, Worcester, MA (M.R., M.A., J.E.W.); the Department of Psychosomatic Medicine, Charité University Medicine, Berlin, Germany (M.R.); Department of Psychosomatic Medicine, University Medical Center Hamburg-Eppendorf and Schön Klinik Hamburg-Eilbek, Hamburg, Germany (M.R., B.L.); the Department of Family and Social Medicine, Albert Einstein College of Medicine, Bronx, NY (J.F., A.E.B.); 3i QualityMetric, Lincoln, RI (J.B.); Veterans Affairs Medical Center and Department of Medicine, University of Minnesota, Minneapolis, MN (T.S.R.); and John Ware Research Group, Incorporated, Worcester, MA (J.E.W.). Correspondence to Matthias Rose, Department of Psychosomatic Medicine, Charité University Medicine Berlin, Charitéplatz 1, Berlin, Germany. matthias.rose@umassmed.edu 2012 American Heart Association, Inc. Circ Heart Fail is available at DOI: /CIRCHEARTFAILURE

2 332 Circ Heart Fail May 2012 to be analyzed manually, assessments become cost prohibitive for use in routine clinical practice, and individual patient reports cannot be provided in a timely manner. Short forms limit the respondent burden but often show more ceiling or floor effects and lack the precision required at the individual patient level. 12,13 Measurement precision to guide individual decision making must be substantially higher than for group comparisons, because true change must be separated from measurement error for every single assessment. 13 For example, if a CI of 95% is required, a traditional tool with good psychometric properties for group comparisons (eg, Cronbach 0.80) would only allow for interpretation of score differences of almost 1 SD when used for an individual. 14 Moreover, classic psychometric methods cannot be used to determine the measurement precision for an individual measurement. As a result, none of the existing tools has become a standard measure in clinical practice. 15,16 Enhancing the precision, accessibility, and interpretability of PRO measures could make HF management more efficient and effective in meeting patient care needs. With the presented study, we apply computerized adaptive testing (CAT) methods, a measurement technology 17 that is used widely in educational testing. 18 We aimed to build a system that will allow routine, comprehensive assessment of pathognomonic symptoms. The use of CAT techniques also promises to provide more precise measures, with fewer items, and an effective resolution to the classic conflict between practicality and precision faced by traditional measurement methods. 12 The CATs tailor each assessment to the individual s status on what is being measured, applying only items that are most appropriate for her or his current health status. Responses to each CAT item direct the choice of the following CAT item toward the most informative for this particular assessment. A patient indicating higher levels of disability within the first questions would only be asked about this level of ability. Omitting the use of uninformative items not relevant for a given functional limitation focuses the assessment, decreases the respondent burden, and increases the measurement precision achievable with a given number of items. The CATs select the items out of a larger item bank representing the entire range of the construct being measured. Most of the item banks are built on the principles of the Item-Response Theory (IRT). The National Institutes of Health are intensively promoting the use of these methods to develop a comprehensive Patient-Reported Outcomes Measurement Information System (PROMIS) as part of their roadmap initiatives ( The authors of this article are part of the PROMIS initiative, which aims to provide a standard assessment for generic health status measures in the near future. 19 The goal of this study was to develop CATs for dyspnea, fatigue, and physical function for the assessment of patients with HF and to evaluate their acceptability, precision, and validity. Methods Development of the Items After review of the relevant literature, we developed a set of 74 patient questions (items) covering the 3 primary physical impairments commonly reported by patients with HF: physical function/ Figure 1. The heart failure (HF) computer adaptive test (CAT) patient interface and examples for 1 item of each bank. disability (24 items), dyspnea (30 items), and vitality/fatigue (20 items). The queries were designed to be short enough to fit on a portable telephone screen for home assessments (Figure 1). Items were selected to represent the entire continuum of each aspect of HF from no to severe impairment. All 3 item banks have been scored in the direction that higher scores indicate more impairment (ie, physical disability, fatigue, and dyspnea). The item bank development was performed separately for each of the 3 domains of physical function, dyspnea, and fatigue, following the same procedures as described in previous studies. 20,21 After the item banks were developed, we used them as a basis for a CAT. A new software solution was developed to work on a Personal Digital Assistant. The CAT logic can be set to stop after the measurement reaches a particular precision or after a maximum of items is administered. For this study phase, the CAT was set to assess each of the 3 different domains with an SE 3.3 (corresponding to a reliability of Cronbach 0.90 for samples with an SD of 10) or a maximum of 7 items per scale. Participants The data for the CAT item bank development sample were collected via the Internet from English-speaking adults with HF. All respondents were recruited by YouGov. YouGov uses a method called sample matching for the selection of study samples from pools of opt-in respondents. 22 Sample matching starts with an enumeration of the target population. For patient recruitments, the target population is all adults with similar sociodemographic characteristics, such as patients with a particular condition, as enumerated in consumer databases (eg, maintained by Acxiom, Experian, and InfoUSA). Then, a random sample is drawn from the target population. Finally,

3 Rose et al Heart Failure CAT 333 for each member of the target sample, a matching member of the Internet pool of opt-in respondents is selected, resulting in a matched sample. Matching was based on age, sex, and race. The resulting matched sample has similar characteristics to the target population and will have similar properties to a true random sample. For this study, adults were approached until the target number of patients with HF was enrolled. All newly developed items were administered randomly. The same data collection method and vendor were used for many similar projects, including an National Institutes of Health roadmap initiative for the development of generic PRO tools ( To ensure a sufficient distribution of responses for the item parameter estimation, we used a quota of one third of patients with minor, medium, and severe impairment based on 1 screening question describing the level of impairment analogous to the NYHA classification (I, II, and III). To help ensure the quality of the data, we applied the following exclusion criteria: (a) average answering time per item was 5 seconds, (b) subjects who did not indicate they had HF and 1 underlying cause for HF, (c) subjects who did not indicate that the HF diagnosis was given by a physician, (d) last visit to a physician was 6 months ago, or (e) current medication did not indicate at least 1 drug used for the treatment of HF (diuretics, angiotensinconverting enzyme inhibitor or angiotensin II receptor blocker, -blockers, or digoxin). To examine the characteristics of the HF-CAT, different simulation studies were conducted, as previously described. 20,23 These analyses are based on the real data provided for all items in the bank by the patients in the online survey. Only small subsets of those item responses are used to estimate the patient score for the CAT simulation (in IRT terms, called score ). The quality of the items in the bank defines the precision of the score at different ranges. The test information curve identifies floor and ceiling effects and if the measurement range of the tool fits to the symptoms of the sample. To illustrate this for the HF-CAT, the precision of the score estimate was plotted as a function of the patient scores. 20 To evaluate the construct validity of the HF-CAT, items from the following established tools were also included in the data collection: the SF-36 Health Survey scales for physical functioning and vitality, 24 4 items from the Medical Health Outcomes Survey to assess shortness of breath, 25 and the Minnesota Living with Heart Failure Questionnaire 26 (MLHFQ, 21 items) as a legacy tool for measuring HF as indicated by patients perceptions of its overall effects on their lives. A separate sample of 100 consecutive participants was recruited for the validity test conducted at the HF clinic of the Montefiore Medical Center, Bronx, NY. The clinic was selected because it usually does not use PRO assessments and predominantly serves a low-income diverse population. We considered this environment as particularly challenging to test a new technology, assuming relatively low health literacy levels. In addition, we believed that an evaluation of psychometric properties would be more relevant in a less educated sample, because the validity of the IRT assumptions has been already evaluated in the developmental sample, which was affluent and well educated (Table 1). Patients with previously diagnosed HF were invited to participate in the study. Consenting participants were asked to complete the actual HF-CAT on a hand-held computer (personal digital assistant [PDA]) and a series of paper-and-pencil assessments, including sociodemographic questions, the MLHFQ, and a survey evaluation of the experience with the HF-CAT. All participants completed both instruments. Participants were randomly assigned to 1 of 2 groups within a crossover design, in which the order of presentation of the HF-CAT assessment and the MLHFQ was counterbalanced. Patients were placed in the waiting area and asked to follow the standard instructions provided for each measure. Medical information, including NYHA class, was extracted from the medical files. The NYHA class is determined routinely for all patients at every visit at the Montefiore Medical Center Heart Failure Clinic based on the clinical assessment of the treating physician. The NYHA class was determined without knowledge of the results of Table 1. Characteristics of the Samples HF-CAT Development Evaluation (IB Sample) (MMC Sample) Characteristics (N 658) (N 100) Age, y* 60 (13) 58 (12) Time with HF, y* 8.8 (7.9) 4.6 (4.5) Family status Living in partnership Living alone Female sex Hispanic or Latino ethnicity 4 35 Race White African American 3 46 Other 4 35 Education 8th grade Some high school 3 21 High school graduate Some college College graduate Postgraduate 20 5 Household income, $ Prefer not to answer 9 37 Employment status Student Working at a paying job Retired Laid off or unemployed 3 2 A full-time homemaker 7 9 Other Data are given as percentage of each group unless otherwise indicated. HF indicates heart failure; CAT, computer adaptive test; IB, item bank; MMC, Montefiore Medical Center. *Data are given as mean (SD). patient self-assessments. Patients gave written informed consent and received a $25 incentive for their participation in the study. Results Samples After applying the inclusion and exclusion criteria, the final item bank development sample consisted of 658 participants, aged years (49% female), who had experienced HF for years (Table 1). Patients reported the following conditions in addition to their HF: 43%, coronary heart disease; 42%, previous heart attacks; 18%, cardiomyopathy; 14%, valvular heart disease; 5.2%, rheumatic fever; 60%,

4 334 Circ Heart Fail May 2012 hypertension; 31%, arrhythmias; and 40%, diabetes. Alcohol abuse was reported by 5.9% of patients. The Montefiore Medical Center clinical sample (n 100) was predominantly male (62%), with a mean age of 58 years. The sample was diverse, including mostly African American patients and many Hispanics. One third of the population had a comparatively low household income. The severity of their HF symptoms, assessed by the NYHA classification, was 11% in class I, 53% in class II, and 36% in class III or IV. HF-CAT Development Item Banks Development In the final calibrated item banks, there were 21 items assessing physical disability, 20 items assessing fatigue, and 29 items in the dyspnea bank with satisfactory item fit (Table 2). Most informative (ie, with a high discrimination parameter: slope ) was the item asking about the ability to run errands, an item referring to a feeling of being worn out, and the item asking if the patient will be short of breath walking from one room to another. Simulation Studies The precision of every score estimate can be displayed as a function of the level of function or the severity of the symptoms. The results of the simulation studies showed that a highly precise score (comparable to an internal consistency of 0.90) can be estimated with 5 items for each domain over a range of nearly 3 SDs (Figure 2, left). The concordance between the results of the CATs and the entire item bank was good for all of the constructs, as illustrated by the extremely high correlations (r ), showing that the 5-item CAT can essentially capture the information provided by the entire bank. As expected, there were high correlations between the simulated CAT scale scores and the corresponding SF-36 Health Survey s physical function (r 0.87) and vitality (r 0.84) scales, as well as the static shortness of breath measurement (r 0.83). Compared with all legacy tools, the HF-CAT provides a more precise measurement over a larger measurement range (Figure 2, right). For physical disability, a similar measurement precision, such as with the SF-36 physical function scale, can be achieved with 1 2 the number of items (Figure 2, top left). HF-CAT Evaluation Respondent Burden On average, 4 to 5 items were administered for the assessment of physical disability, fatigue, and dyspnea to achieve the predefined level of precision (Table 3). The average time for administration of the entire HF-CAT with all 3 domains was 3 minutes (3 2 minutes). Validity We used the MLHFQ to help evaluate the constructs of the HF-CAT and the NYHA class to evaluate its discriminative validity (Table 3). The mean MLHFQ score of the sample was 38 25, and the mean scores of the HF-CAT were for physical disability, for fatigue, and for dyspnea. There were no order effects for any measure. The HF-CAT scales for physical disability, fatigue, and dyspnea correlated significantly with the MLHFQ total score (r 0.71, r 0.63, and r 0.68, respectively). A general linear model was used to evaluate the ability of the HF-CAT scales to statistically differentiate patients with different levels of symptom severity, as measured by the clinician s NYHA classification (Table 3). The main effects for all the measures were significant, with similar discriminative ability (Eta 2, F values) for the HF-CAT physical disability and dyspnea scales and the MLHFQ scale. User Experience Because this study took place in a low-income, less educated, minority population, we were particularly interested in the subjective user experience with a computer assessment. Of the patients, 98% found the HF-CAT assessment overall very easy or easy, 100% thought it was very easy or easy to follow the instructions, and 95% said it was very easy or easy to read the questions on the screen. In addition, of the patients, 98% judged the time for the assessment as just right, and 90% considered the questions as relevant; 98% were willing to use the device again on the next visit. Discussion For the first time, to our knowledge, we applied computerized adaptive testing methods to develop and evaluate an ultrashort assessment system for patients with HF (HF-CAT) in clinical practice. The tool allows routine, comprehensive assessment of 3 primary problems that are commonly experienced by patients with HF. If the emotional or social impact of the disease is of additional interest, further tools (eg, from the PROMIS) need to be added for a comprehensive coverage of the health-related quality-of-life construct. Feasibility The feasibility of the HF-CAT in its PDA version was evaluated in a low-income, low-educated, minority population in the Bronx. The HF-CAT is a practical and wellaccepted tool. Nevertheless, it was tested under study conditions, and participants might have been biased receiving an incentive for their participation. To our knowledge, only 1 report about the acceptance of CATs within clinical practice settings is available. A similar CAT, also being displayed on a PDA, has been in routine clinical use since Patients answering this CAT also report a high acceptability. Almost all of the 423 consecutive patients considered the handling as easy and believed that the use of the PDA made sense. 27 Several other studies report about the reception of CATs under study conditions. Most patients in a feasibility test of a pain CAT found the CAT application to be useful, relevant, of appropriate length, and easy to complete. 28 Similarly, most respondents in a feasibility study of an asthma impact CAT found it easy to complete and of appropriate length. 29 The results of a feasibility test of a diabetes CAT gave somewhat mixed results. Although both English- and Spanish-speaking participants agreed that a paper-and-pencil assessment was more burdensome than a CAT, the Spanish-speaking participants preferred the paper tool and were more willing to complete a paper tool in the future. 30

5 Rose et al Heart Failure CAT 335 Table 2. IRT Item Parameters: HF-CAT Item Banks Thresholds Parameter Slope Mean Physical disability Exercising hard for 0.5 h (1) Doing 1hofphysical labor (1) Walking up a steep hill (1) Rearranging furniture at home (1) Doing chores (1) Doing daily physical activities (2) Climbing up a flight of stairs (1) Doing daily physical activities (1) Carrying 2 bags of groceries (1) Walking on flat ground (1) * Preparing a meal (1) * Walking 100 yards (1) Standing up from a chair (1) * Running errands and shopping (1) Dressing myself (1) * Taking a tub bath (1) Getting from one room to another (1) * Standing up from a bed (1) * Getting on and off the toilet (1) * Making the bed (1) Putting a trash bag outside (1) Fatigue Full of energy (3) Strong and vital (3) Fresh and rested (3) Lively (3) Active (3) Full of life (3) Tired (3) Fatigued (3) Sluggish (3) Worn out (3) Run down (3) Wide awake (3) As if I have no energy left (3) Spent (3) Exhausted (3) Weary (3) Weak (3) Save my energy (3) Sleepy all day (3) Jaded (3) Dyspnea Running a short distance makes me short of breath (3) Exercising hard for 0.5 h makes me short of breath Talking while walking up a hill will make me short of breath (3) An hour of physical labor makes me short of breath (4) My breathing problems limit my ability to exercise as much as I would like (3) (Continued)

6 336 Circ Heart Fail May 2012 Table 2. Continued Thresholds Parameter Slope Mean Talking while walking up a flight of stairs makes me short of breath (3) During a typical day, I feel short of breath (4) Doing chores, such as vacuuming or yard work, makes me short of breath (3) Climbing up 1 flight of stairs makes me short of breath (3) Going outside for a walk makes me short of breath (3) Walking 100 yards makes me short of breath (3) Walking up a hill makes me short of breath (4) Carrying groceries makes me short of breath (3) Talking while walking makes me short of breath (3) Running errands makes me short of breath (3) Taking a bath makes me short of breath (4) Dressing myself makes me short of breath (4) Preparing a meal makes me short of breath (4) Singing or humming makes me short of breath (4) Speaking in a group makes me short of breath (4) I feel short of breath when I sit and rest (4) * Talking at noisy places makes me short of breath (4) Walking from one room to another makes me short of breath (4) Talking to someone makes me short of breath (4) Talking on the telephone makes me short of breath (4) Getting off the bed makes me short of breath (4) Going to the toilet makes me short of breath (4) Lying down flat makes me short of breath (3) Standing up from a chair makes me short of breath (4) The table is ordered by the mean threshold value. Response options are as follows: 1, easy/hard/impossible; 2, no difficulty/a little bit of difficulty/some difficulty/a lot of difficulty/cannot do because of my health; 3, not at all/somewhat/very much; 4, not at all/a little bit/quite a lot/cannot do; and 5, not at all/a little bit/quite a lot. The IRT item bank parameters are developed as usual on a 0 1 metric, with 0 representing the scaling sample mean with an SD of 1. For easier interpretability, estimated patient scores are transformed linear to a metric later. The slope parameter is also called the discrimination parameter. Higher slope parameters indicate a better discrimination, which makes the item more valuable (ie, informative for the score estimation: eg, the capability to run errands is more informative to determine the physical disability of a patient than her or his ability to put the trash outside the house ). The thresholds of an item show at which score level a particular response option is the most likely to be endorsed. For the item running errands, the threshold separates the response easy from hard and the threshold separates the response hard from impossible. If a patient scores 3 SDs higher than the population mean, she or he is most likely to answer the item running errands is with impossible because her or his score is higher than the threshold of If her or his level of disability is only 1.5 SDs higher than the US population mean, she or he is likely to endorse hard because the score is between the thresholds and The mean threshold illustrates the position of the item on the metric, which can be seen as item difficulty in traditional terms. The table is sorted by the mean threshold. IRT indicates Item-Response Theory; HF, heart failure; CAT, computer adaptive test. *The 2 highest response options were collapsed for the item parameter estimation, the presentation of response options for the patient remains the same. Respondent Burden One important contribution of the CAT technology will be to reduce the respondent burden without compromising the precision and validity of the assessment, by tailoring each assessment to the patient s condition. This advantage was previously demonstrated in a simulation study of the Activities of Daily Living CAT, which found that the CAT provided similar results to a static version while reducing the number of items administered by 50%. 31 Results from other studies indicate that scores similar to those obtained with full-length item banks (ranging in length from items) can be achieved through much shorter CATs when measuring functional status, mental health status, 21,27,35,36 or the impact of conditions, such as headache, 23,37 diabetes, 30 chronic pain, 28 and asthma. 29 Most actual CAT applications used between 5 and 7 items to measure 1 construct. The present HF-CAT applied between 4 and 5 items per scale, and the average total time for the entire assessment and scoring was 3 minutes (ie, 1 minute per scale, which could be applied individually). The assessment time of the MLHFQ electronically measured in a previous study was 4 2 minutes, 38 and time to administer the Kansas City Cardiomyopathy Questionnaire, another common tool for the assessment of patients with HF, is reported to be 4 to 6 minutes without scoring. 39 In summary, the HF-CAT provides a precise measure over a large measurement range with minimal respondent burden. As far as it is known today, it seems that CATs offer an effective resolution to the classic conflict between practicality and precision faced by traditional measurement technology. 12

7 Rose et al Heart Failure CAT 337 Figure 2. Measurement precision in relation to measurement range. The x axis shows the patient score. In Item-Response Theory (IRT) terminology, this score is referred to as the score. To make the heart failure (HF) computer adaptive test (CAT) and the legacy tools comparable, both instruments are scored on the same metric, as determined by the developed item banks. The y axis shows the 95% CI of the patient score; the smaller the y value, the higher the precision of the score. The dotted lines show CIs that would be comparable to an internal constancy of Cronbach 0.80, 0.90, and 0.95, for illustrative purposes. Validity Studies of CAT applications in diseases, such as depression 27,35 or headache, 40 have shown that their measurement advantages can transfer to increased validity in identifying differences between groups known to differ in clinical characteristics, compared with static tools. The 3 scales of the HF-CAT also discriminated between groups of patients of different NYHA classification equally and a legacy tool measuring the impact of HF, using 4 times more items. These initial results show that the HF-CAT has the potential to provide a valid, highly relevant assessment of patients with HF. Serial Measurements For the assessment of patients with HF, we believe it is important to assess the health status of the patient at the point of care and at the patient s home. Because many elderly patients do not have access to the Internet or are not familiar with its use, one way to do so is the use of a smart phone and/or interactive voice recognition. Most established tools include items that are not suitable to be used over the telephone. The IRT methods allow using much simpler items over the telephone and more comprehensive items at the physician s office, and scoring both assessments on the same measurement metric. This allows having a smart phone administer the HF-CAT at the patient s home and having the same patient answering the more comprehensive PROMIS- CAT on a tablet PC at the physician s office. The IRT-based measurements of health outcomes are independent of the particular items being administered and of the test administrator. The same value for the same domain yields the same interpretation, whereas results from different traditional tools

8 338 Circ Heart Fail May 2012 Table 3. Score Differences Between Different NYHA Classes NYHA Class* Variable No. of Items I(n 11) II (n 53) III/IV (n 36) Eta 2 F Value P Value RV (95% CI) Physical disability (6.2) 58.9 (8.6) 62.6 (7.4) ( ) Fatigue (6.9) 52.0 (7.6) 55.4 (9.4) ( ) Dyspnea (14.4) 53.8 (12.7) 59.8 (11.7) ( ) MLHFQ (14.8) 38.3 (25.3) 44.9 (22.9) The theta values of the computer adaptive test (CAT) scales are scored on a T-distribution. The MLHFQ scores are summary scores ranging from 0 to 105. All analyses were controlled for the order of administration as a confounding variable. A bootstrap analysis was used to determine the CIs. NYHA indicates New York Heart Association; RV, relative validity (heart failure CAT scale F values/f value for the MLHFQ sum scale); MLHFQ, Minnesota Living with Heart Failure Questionnaire. *Data are given as mean (SD). cannot be compared directly, making serial health status monitoring less practicable. Limitations Despite many encouraging findings with recent CAT developments, several issues still need to be addressed. Within this study, we have only used outpatients to evaluate the HF- CAT, which limits the generalizability to less severely disabled patients. However, one of the most relevant advantages of CATs is that they can essentially eliminate floor and ceiling effects by applying items tailored to the test taker. Our simulation studies have shown that the current item bank covers 3 SDs above the population mean, which is where a hospitalized population of patients with HF usually scores. We did not evaluate the test-retest reliability for the HF-CAT. Similarly, we have not used the HF-CAT in an intervention study to test its responsiveness to treatments. However, several studies have reported on the ability of other CATs to detect change. For example, in a telephone study of 540 patients with headache, a CAT for headache impact was more responsive to self-evaluated changes of headache impact than a corresponding 54-item bank. 23 In a longitudinal, prospective cohort study of 94 patients discharged from inpatient rehabilitation, the CAT version of the Activity Measure for Post-Acute Care was comparable in responsiveness to the 66-item static version. 41 Similarly, in a series of articles, Hart et al 33,34 report on the results of validation studies of condition-specific CATs, using large data sets from patients receiving rehabilitation services across multiple US clinics. Summary In summary, we have developed a promising method to measure patient-reported dyspnea, fatigue, and physical function for use in the care of patients with HF. This new measure is part of a rapidly growing number of new assessment tools using the advantages of item response theory and computerized adaptive test techniques, 16,19,42 with some of them being used in clinical practice already. 27,43 However, whether these encouraging improvements in measurement will transfer to improved care and ultimately health of patients with HF warrants further studies. Sources of Funding This study was supported in part by grant 1 R43 HL from the National Institutes of Health/National Heart, Lung, and Blood Institute (Dr Rose). Disclosures The HF-CAT software was developed by QualityMetric Inc. Dr. J. Bjørner is an employee of this company. References 1. Hunt SA, Baker DW, Chin MH, Cinquegrani MP, Feldman AM, Francis GS, Ganiats TG, Goldstein S, Gregoratos G, Jessup ML, Noble RJ, Packer M, Silver MA, Stevenson LW, Gibbons RJ, Antman EM, Alpert JS, Faxon DP, Fuster V, Jacobs AK, Hiratzka LF, Russell RO, Smith SC Jr. ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2001;38: Bennett JA, Riegel B, Bittner V, Nichols J. Validity and reliability of the NYHA classes for measuring research outcomes in patients with cardiac disease. Heart Lung. 2002;31: Goldman L, Hashimoto B, Cook EF, Loscalzo A. Comparative reproducibility and validity of systems for assessing cardiovascular functional class: advantages of a new specific activity scale. Circulation. 1981;64: Lett HS, Blumenthal JA, Babyak MA, Sherwood A, Strauman T, Robins C, Newman MF. Depression as a risk factor for coronary artery disease: evidence, mechanisms, and treatment. Psychosom Med. 2004;66: Konstam V, Moser DK, De Jong MJ. Depression and anxiety in heart failure. J Card Fail. 2005;11: Rumsfeld JS, Havranek E, Masoudi FA, Peterson ED, Jones P, Tooley JF, Krumholz HM, Spertus JA. Depressive symptoms are the strongest predictors of short-term declines in health status in patients with heart failure. J Am Coll Cardiol. 2003;42: Sullivan M, Simon G, Spertus J, Russo J. Depression-related costs in heart failure care. Arch Intern Med. 2002;162: Rumsfeld JS, Jones PG, Whooley MA, Sullivan MD, Pitt B, Weintraub WS, Spertus JA. Depression predicts mortality and hospitalization in patients with myocardial infarction complicated by heart failure. Am Heart J. 2005;150: Junger J, Schellberg D, Muller-Tasch T, Raupp G, Zugck C, Haunstetter A, Zipfel S, Herzog W, Haass M. Depression increasingly predicts mortality in the course of congestive heart failure. Eur J Heart Fail. 2005; 7: Cleary PD, Edgman-Levitan S. Health care quality: incorporating consumer perspectives. JAMA. 1997;278: Burke L. FDA Perspectives on IRT/CAT: DIA Workshop on Advances in Health Outcomes Measurement: Exploring the Current State and the Future Applications of Item Response Theory, Item Banks, and Computer-Adaptive Testing. Bethesda, MD: conference lecture; McHorney CA, Cohen AS. Equating health status measures with item response theory: illustrations with functional status items. Med Care. 2000;38:II43 II Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res. 2009;18: McHorney CA, Tarlov AR. Individual-patient monitoring in clinical practice: are available health status surveys adequate? Qual Life Res. 1995;4: Rector TS. A conceptual model of quality of life in relation to heart failure. J Card Fail. 2005;11:

9 Rose et al Heart Failure CAT Garin O, Ferrer M, Pont A, Rue M, Kotzeva A, Wiklund I, Van GE, Alonso J. Disease-specific health-related quality of life questionnaires for heart failure: a systematic review with meta-analyses. Qual Life Res. 2009;18: Bjorner JB, Chang CH, Thissen D, Reeve BB. Developing tailored instruments: item banking and computerized adaptive assessment. Qual Life Res. 2007;16(suppl 1): Wainer H, Dorans NJ, Eignor D, Flaugher R, Green BF, Mislevy RJ, Steinberg L, Thissen D. Computerized Adaptive Testing: A Primer. Mahwah, NJ: Lawrence Erlbaum Associates; Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, Ader D, Fries JF, Bruce B, Rose M. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45:S3 S Rose M, Bjorner JB, Becker J, Fries JF, Ware JE. Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2008;61: Fliege H, Becker J, Walter OB, Bjorner JB, Klapp BF, Rose M. Development of a computer-adaptive test for depression (D-CAT). Qual Life Res. 2005;14: Rubin DB. Matched Sampling for Causal Effects. New York, NY: Cambridge University Press; Ware JE Jr, Kosinski M, Bjorner JB, Bayliss MS, Batenhorst A, Dahlof CG, Tepper S, Dowson A. Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Qual Life Res. 2003;12: Ware JE Jr, Dewey J. How to Score Version Two of the SF-36 Health Survey. Lincoln, RI: QualityMetric Inc; National Committee for Quality Assurance. Specifications for the Medicare Health Outcomes Survey: HEDIS. 6. Washington, DC: National Committee for Quality Assurance; Rector T, Cohn J. Patients self-assessment of their congestive heart failure, part 2: content, reliability and validity of a new measure, the Minnesota Living with Heart Failure questionnaire. Heart Failure. 1987; 3: Fliege H, Becker J, Walter OB, Rose M, Bjorner JB, Klapp BF. Evaluation of a computer-adaptive test for the assessment of depression (D-CAT) in clinical application. Int J Methods Psychiatr Res. 2009;18: Anatchkova MD, Saris-Baglama RN, Kosinski M, Bjorner JB. Development and preliminary testing of a computerized adaptive assessment of chronic pain. J Pain. 2009;10: Turner-Bowker DM, Saris-Baglama RN, Anatchkova M, Mosen DM. A computerized asthma outcomes measure is feasible for disease management. Am J Pharm Benefits. 2010;2: Schwartz C, Welch G, Santiago-Kelley P, Bode R, Sun X. Computerized adaptive testing of diabetes impact: a feasibility study of Hispanics and non-hispanics in an active clinic population. Qual Life Res. 2006;15: Chien TW, Wu HM, Wang WC, Castillo RV, Chou W. Reduction in patient burdens with graphical computerized adaptive testing on the ADL scale: tool development and simulation. Health Qual Life Outcomes. 2009;7: Haley SM, Gandek B, Siebens H, Black-Schaffer RM, Sinclair SJ, Tao W, Coster WJ, Ni P, Jette AM. Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II: participation outcomes. Arch Phys Med Rehabil. 2008;89: Hart DL, Wang YC, Stratford PW, Mioduski JE. Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function. J Clin Epidemiol. 2008;61: Hart DL, Werneke MW, Wang YC, Stratford PW, Mioduski JE. Computerized adaptive test for patients with lumbar spine impairments produced valid and responsive measures of function. Spine (Phila Pa 1976). 2010;35: Gibbons RD, Weiss DJ, Kupfer DJ, Frank E, Fagiolini A, Grochocinski VJ, Bhaumik DK, Stover A, Bock RD, Immekus JC. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatr Serv. 2008;59: Walter OB, Becker J, Bjorner JB, Fliege H, Klapp BF, Rose M. Development and evaluation of a computer adaptive test for Anxiety (A-CAT). Qual Life Res. 2007;16(suppl 1): Bayliss MS, Dewey JE, Dunlap I, Batenhorst AS, Cady R, Diamond ML, Sheftell F. A study of the feasibility of Internet administration of a computerized health survey: the headache impact test (HIT). Qual Life Res. 2003;12: Bennett SJ, Oldridge NB, Eckert GJ, Embree JL, Browning S, Hou N, Chui M, Deer M, Murray MD. Comparison of quality of life measures in heart failure. Nurs Res. 2003;52: Green CP, Porter CB, Bresnahan DR, Spertus JA. Development and evaluation of the Kansas City Cardiomyopathy Questionnaire: a new health status measure for heart failure. J Am Coll Cardiol. 2000;35: Martin M, Kosinski M, Bjorner JB, Ware JE Jr, Maclean R, Li T. Item response theory methods can improve the measurement of physical function by combining the modified health assessment questionnaire and the SF-36 physical function scale. Qual Life Res. 2007;16: Haley SM, Fragala-Pinkham M, Ni P. Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme. Clin Rehabil. 2006;20: Ruo B, Choi SW, Baker DW, Grady KL, Cella D. Development and validation of a computer adaptive test for measuring dyspnea in heart failure. J Card Fail. 2010;16: Becker J, Fliege H, Kocalevent RD, Bjorner JB, Rose M, Walter OB, Klapp BF. Functioning and validity of a Computerized Adaptive Test to measure anxiety (A-CAT). Depress Anxiety. 2008;25:E182 E194. CLINICAL PERSPECTIVE Patient-reported outcome measures can assist clinicians in monitoring the effectiveness of their treatment of patients with heart failure (HF), comparing improvements under different forms of therapy, and identifying risk of deterioration. There are several questionnaires available to measure typical HF symptoms. However, established questionnaires are often long or too imprecise for individual decision making. New computer adaptive tests (CATs) promise to provide more precise measures, with fewer items, and an effective resolution to the classic conflict between practicality and precision faced by traditional measurement methods. The CATs tailor each assessment to the individual s status on what is being measured, applying only items that are most appropriate for her or his current health status. We have developed a CAT for the assessment of 3 typical HF symptoms (HF-CAT): dyspnea, fatigue, and physical function. In-clinic tests confirmed the expected CAT advantages, including shorter surveys by eliminating questions not relevant to each patient, equal or better enumerations over a wide range of scores, and surveys that did not require a test administrator. In summary, we have developed a promising method to measure patient-reported symptoms for use in the care of patients with HF. This new measure is part of a rapidly growing number of new assessment tools using the advantages of item response theory and computerized adaptive test techniques.