WHICH OUTCOME MEASURES SHOULD BE USED IN RHEUMATOID ARTHRITIS CLINICAL TRIALS?

Transcription

1 1568 ARTHRITIS & RHEUMATISM Vol. 38, No. 11, November 1995, pp , American College of Rheumatology WHICH OUTCOME MEASURES SHOULD BE USED IN RHEUMATOID ARTHRITIS CLINICAL TRIALS? Clinical and Quality-of-Life Measures Responsiveness to Treatment in a Randomized Controlled Trial RACHELLE BUCHBINDER, CLAIRE BOMBARDIER, MARIA YEUNG, and PETER TUGWELL Objective. To determine the discriminant validity of the core set of outcome measures proposed by the American College of Rheumatology (ACR) and the Outcome Measures in Clinical Trials (OMERACT) conference committee to be used in clinical trials of rheumatoid arthritis (RA). Methods. Utilizing data from a multicenter randomized double-blind clinical trial of low-dose cyclosporine and placebo in RA, we estimated the relative efficiency (RE) of measures to detect a treatment effect (relative to tender joint count, which was assigned a value of 1). Four pain measures (10-cm visual analog scale [VAS], 5-point categorical scale, Health Assessment Questionnaire [HAQ] pain index, Arthritis Impact Measurement Scales [AIMS] pain score) and 3 qualityof-life measures (Problem Elicitation Technique [PET], HAQ, AIMS) were compared. Results. Physician and patient global measures were the most responsive instruments, although neither was statistically superior to tender joint count. Swollen joint count, grip strength, pain measured on a 10-cm VAS, and functional status as measured by the PET and HAQ were all of intermediate responsiveness. Morning stiffness, 5-point pain scale, and erythrocyte sedimentation rate were the least responsive instruments. Rachelle Buchbinder, MBBS, MSc, FRACP: Wellesley Hospital Research Institute, University of Toronto, Toronto, Ontario, Canada (currently with Monash University, Melbourne, Australia); Claire Bombardier, MD, FRCP(C), Maria Yeung, MA(stats), MSc(stats): Wellesley Hospital Research Institute, University of Toronto; Peter Tugwell, MD, MSc, FRCP(C): University of Ottawa, Ottawa, Ontario, Canada. Address reprint requests to Rachelle Buchbinder, MBBS, MSc, FRACP, Department of Social and Preventive Medicine, Monash Medical School, Alfred Hospital, Commercial Road, Prahran Victoria 3181, Australia. Submitted for publication March 1, 1995; accepted in revised form June 6, Conclusion. This study provides further evidence to support the core set of outcome measures proposed by the ACR and OMERACT. The rheumatology community is striving for a more standardized approach in choosing end points for the escalating number of clinical trials of therapies in rheumatoid arthritis (RA). End points may quantify objective physical signs such as the number of tender or swollen joints, symptoms such as the duration of morning stiffness or level of pain, or functional status or overall health as perceived by the patient or the physician. Recommendations of a core set of outcomes to be used in clinical trials of RA have been prepared by the American College of Rheumatology (ACR) (1) and an international conference committee (Outcome Measures in Rheumatoid Arthritis Clinical Trials [OMERACT]) (2). These recommendations were formed after review of the evidence, from previous trials, that specific measures meet required criteria for evaluative instruments (i.e., those designed to examine change over time [3]). The ACR committee has also recommended specific methods for assessing each outcome measure (1). The purpose of this investigation was, first, to confirm the discriminant validity of the core set of outcome measures recommended by the ACR and to further examine the sensitivity to change of disease activity measures in RA clinical trials. Sensitivity to change in the context of an RA clinical trial, which usually compares a new therapy with either placebo or a standard treatment, may be studied by examining an instrument s ability to detect a treatment effect, i.e., to detect a difference in change with interventions that have differing efficacies. Utilizing the data from a

2 OUTCOME MEASURES IN RA CLINICAL TRIALS 1569 multicenter randomized double-blind clinical trial of low-dose cyclosporine and placebo in RA (4), we estimated and compared the ability of the instruments used in that study to detect a treatment effect. The trial showed a significant benefit of cyclosporine over placebo in all clinical measures examined. In particular, cyclosporine-treated patients had 8.2 (23%) fewer active joints in the tender joint count, 24% less pain (measured on a visual analog scale [VAS]), and 16% better functional status (measured by the Problem Elicitation Technique [PET]), compared with placebotreated patients (4). Second, we examined the ACR recommendations of which methods should be utilized for assessing each outcome measure. In particular, we compared the discriminant validity of the 4 methods for assessing patient pain that were utilized in the cyclosporine study, as well as quality-of-life measures: the Health Assessment Questionnaire (HAQ) (9, the Arthritis Impact Measurement Scales (AIMS) (6), and the PET, a modification of the McMaster Toronto Arthritis Patient Preference Disability Questionnaire (MAC- TAR) (7). The PET was administered to all patients, the HAQ was administered to patients at 3 of the 6 centers in the trial (HAQ subgroup), and the AIMS was administered to patients at the remaining 3 centers (AIMS subgroup). We also examined the stability of the estimate of responsiveness of instruments calculated for the study group as a whole, versus calculations for each of the subgroups. PATIENTS AND METHODS The study design, patients, outcome measures, and statistical methods have been described in detail previously (4). Six clinical centers across Canada participated in a 6-month, double-blind, randomized trial comparing oral cyclosporine and placebo in 144 patients with severe RA. The 144 patients were allocated to receive cyclosporine or placebo according to a prearranged randomization plan, generated separately for each center. Patients who met the following entry criteria were eligible: disease that met ACR (formerly, the American Rheumatism Association) criteria for RA (8); age >18 years; 26 actively inflamed joints (tender or swollen); and at least 2 of the following criteria: 29 joints tender on pressure, 145 minutes of morning stiffness, and an erythrocyte sedimentation rate () 228 mmihour. In addition, patients had to have had previous unsuccessful (due to toxicity or lack of efficacy) treatment with aspirin and other nonsteroidal antiinflammatory drugs (NSAIDs), and either gold salts or D-penicillamine (or both) (4). Outcome measures. Outcome assessments were conducted, under blinded conditions, by an independent clinical evaluator at each center. All evaluators were trained prior to the study; the training included a standardized method for conducting the tender joint count. The following outcome measures were compared: tender joint count (number of clinically active joints as determined by pain on passive motion or tenderness on pressure, assessed in 61 diarthrodial joints [9]); tender joint score (summation of scores for each of the 61 joints assessed for pain on passive motion or tenderness on pressure, graded on a scale of 0 = none, 1 = positive response on questioning, 2 = spontaneous response elicited, 3 = withdrawal by patient on examination); swollen joint count (number of swollen joints, assessed in 26 joints); duration of morning stiffness in minutes, measured using a patient questionnaire regarding the day preceding the clinic visit (patients were asked to record the time of awakening, time of arising, and time of cessation of morning stiffness, and the duration was measured as the time from awakening); grip strength, measured using a mercury column sphygmomanometer with standard grip bag which registers a minimum of 250 mm Hg (the system was inflated to 20 mm Hg before measuring grip strength; 3 readings were recorded for each hand [alternating hands], and grip strength was recorded as the mean of the 6 measurements); (Westergren method); patient assessment ofjoint pain, using a 10-cm VAS (a 10-cm vertical line labeled no pain at the bottom [scored 01 and maximum imaginable pain at the top [scored 101 [lo]) and using a 5-point categorical pain rating scale (1 = none, 2 = mild, 3 = moderate, 4 = severe, 5 = very severe); physician global efficacy assessment of the patient s condition at the time of study withdrawal as compared with study enrollment, using a 5-point scale (1 = marked improvement, 2 = moderate improvement, 3 = same, 4 = moderate worsening, 5 = marked worsening); patient global efficacy assessment of his or her condition at the time of study withdrawal as compared with study enrollment, using the same 5-point scale; and physician assessment of disease activity, using a 5-point scale (1 = asymptomatic, 2 = mild, 3 = moderate, 4 = severe, 5 = very severe). Three functional status instruments were administered, as noted above. The PET was administered to all patients (n = 144). The HAQ subgroup consisted of 78 patients, and the AIMS subgroup comprised the remaining 66 patients. The PET is a preference disability questionnaire modified from the MACTAR (7), in which patients are asked to identify problems related to their arthritis that they would most like to see improve as a result of therapy. Problems may be spontaneously volunteered by the patients or elicited with the use of a series of open-ended questions asking about problems in areas of self-care, mobility, role activities, leisure activities, social activities, emotion, communication, sleep, and appearance. Previous work has demonstrated that many items identified as important by the patient do not appear in conventional questionnaires, and that even the most frequently identified disabilities are either not mentioned at all or classified as unimportant by many other patients (7). To augment the responsiveness of the MAC- TAR, a measure of the magnitude of each problem was incorporated into the PET instrument. Patients are asked to identify the level of difficulty, severity, or frequency that they experience with each problem, on a 7-point scale (e.g.,

3 1570 BUCHBINDER ET AL 0 = no difficulty to 7 = unable to do for the level of difficulty scale for self-care problems). They are asked to identify the level of importance of each problem on a 7-point scale (0 = not at all important to 7 = most important ), and then they are asked to rank their problems in order of importance with respect to each other. The PET also assesses patient overall global health by use of a 10-point scale (1 = worst possible health to 10 = perfect health ). Each problem is scored by multiplying the magnitude of the problem by the importance of the problem. The PET score is derived by summing the scores for the 5 most important problems. At the 6-month followup visit or at the time of discontinuation of the study medication, the list of problems, their degree of difficulty/severity/frequency, level of importance, and ranking was reviewed, and any new problems identified. The PET requires a trained interviewer for administration and takes approximately 20 minutes to complete. It was administered to all patients in a standardized manner by the same trained, blinded interviewer at each center at baseline and at 6 months (or earlier if the study medication was discontinued). The HAQ is a short (10-minute) self-administered instrument that asks 2 or 3 fixed items in 8 areas of daily life, i.e., dressing and grooming, arising, eating, walking, hygiene, reach, grip, and activities (5). For each item, a score of 0 (no difficulty), 1 (some difficulty), 2 (much difficulty or with assistance), or 3 (unable to perform) is assigned. Any activity that requires assistance from another individual or requires the use of an assistive device receives a score of 2. The highest score for each of the 8 areas is summed (range 0-24) and divided by 8 to yield, on a continuous scale, a functional disability index between 0 and 3 (3 = worst). A second component of the HAQ is a measure of pain which is calculated from a 15-cm horizontal VAS, with no pain at one end (scored 0) and very severe pain at the other (scored 100). A score is determined based on the location of the respondent s mark. The distance from 0 to the mark, in cm, is measured using a metric rule and multiplied by 0.2 to obtain a value from 0 to 3.0 (3.0 = worst). Previous studies have demonstrated that the HAQ has good reliability, validity, and sensitivity to change in RA clinical trials (5,11,12). The AIMS is a longer (15-20-minute) self-administered instrument consisting of 69 fixed items (6). Nine scales assess mobility (ability to move about the community), physical activity (lower extremity function), dexterity (hand function), household activities (routine household tasks), activities of daily living (basic self-care tasks), social activity (interaction with friends and family), anxiety, depression, and pain. These 9 scales are summarized into 3 broad components: physical disability (including mobility, physical activity, dexterity, household activities, and activities of daily living), psychological status (including depression and anxiety), and pain. All AIMS scale and component scores are standardized to a &lo range, where higher numbers represent poorer health status. The pain scale of the AIMS consists of 4 items asking, During the last month, how often have you had severe pain from your arthritis? ; How would you describe the arthritis pain you usually have? ; How long has your morning stiffness usually lasted from the time you wake up? ; and How often have you had pain in two or more joints at the same time? Each of the questions has 6 response items, e.g., for the first question, the possible responses range from always (scored 6) to never (scored 1). The pain score, like the other component scores, is derived by summation of the scores for the 4 items, standardized to give a value between 0 and 10 (i.e., by multiplying the raw score by ), where higher numbers represent more pain. Previous studies have demonstrated that the AIMS has good reliability, validity, and sensitivity to change in RA clinical trials (6,13-15). The HAQ and AIMS were administered twice, at baseline and at 6 months (or earlier if the study medication was discontinued). Statistical analysis. Comparison of ability to detect a treatment effect. For each outcome measure, several calculations were performed, as described below. These calculations were performed for the study population as a whole and for the HAQ and AIMS subgroups. The subscript t denotes the active treatment group (i.e., the cyclosporine group), the subscript - c denotes the control group (i.e., the placebo group), denotes the mean difference between the 6-month (or final) visit and baseline scores for a measure, n is the number of patients, and SD is the standard deviation. The observed treatment effect was calculated as the difference between the mean change from the beginning to the end of the study in the cyclosporine group (4) versus the placebo group (a,), measured in the clinical units of the scale, i.e., & - a,. The relative percent improvement in the cyclosporine group compared with the placebo group was calculated, i.e., I(&/baseline score in cyclosporin group - a,/ baseline score in placebo group)l x 100. The standard deviation of mean change in scores in the placebo group (S%) and cyclosporine group (SD,) was recorded as the SD of the calculated mean change in scores from baseline to 6-month (or final) visit in each group. The pooled standard deviation (PSD) was a pooling of the standard deviations of the calculated mean change in scores from baseline to 6-month (or final) visit in each group, i.e.,d((n, - 1)SDat2 + (n, - 1) SDat2/(n, - n, - 2). The standardized effect size (SES) was calculated as the ratio of the treatment effect (a, - a,) to the PSD of these differences, i.e., (a, - a,)/psd. Dividing the estimated treatment effect by the variability (or noise) of that estimate results in a standardized effect size, which then enables direct comparison between competing measures. If one standardized effect size is larger and therefore better than another, this could be because the numerator (the treatment effect) is larger, the denominator (the variance of the treatment effect) is smaller, or some combination of the two. The relative efficiency to detect a difference among treatment groups (RE) was calculated in relation to tender joint count for each instrument, by taking the square of the ratio of the SES of the instrument to the SES of the tender joint count, i.e., (SESinstrurnen,/SESjoi ~ Tender joint count is a traditional end point of clinical trials in RA and is known to be sensitive to change (16-18). It has also been used previously as a referent measure (17,19). An RE of > 1 would imply that the instrument is more efficient than tender joint count in detecting a treatment effect. The RE can be directly compared between instruments. We also used the PET as the referent measure to evaluate the RE of the HAQ disability index and AIMS

4 OUTCOME MEASURES IN RA CLINICAL TRIALS 1571 physical function scale, and used the 10-cm VAS pain scale as the referent measure to evaluate the relative efficiencies of the HAQ, AIMS, and 5-point categorical pain scales. We compared these with the values obtained using the tender joint count as the referent measure. The standardized effect sizes were formally compared via an approximate Z test to determine whether they differed significantly from tender joint count, using the equation (z = I(SES,,,trument - SESjoint count)/v)2(l/nt + l/nj (1- Irl)l where n, = number in treatment group, nc = number in placebo group, and r = observed pooled within-group correlation between the two outcome measures. It is difficult to compare treatment effects across outcome measures because of different units. A rough indication of the relative magnitude of the treatment effects across measures can be provided by the relative treatment effect. This is the ratio of the treatment effect to the mean change in score between 6-month (or final) visit and baseline in the placebo group, i.e., (a, - a,)&. Similarly, a rough indication of the relative magnitude of the variance (or noise) across different measures can be provided by the coefficient of variation. This is calculated by dividing the pooled standard deviation by the mean change in score between 6-month (or final) visit and baseline in the placebo group, i.e., (PSD/d,). Assessment of the degree of homogeneity of treatment groups and HAQ and AIMS subgroups. In order to assess treatment group homogeneity, comparisons between the 2 treatment groups in the study population as a whole, with respect to patient characteristics and evaluative measures at baseline, have been made and reported previously (4). The 2 groups were comparable for all variables at baseline (4). Similar comparisons between treatment groups were also made within the HAQ and AIMS subgroups. Subgroup homogeneity was assessed by comparing the mean values of pretreatment patient characteristics and evaluative measures in the HAQ group versus the AIMS group. Continuous variables were compared using t-tests, and Pearson s chisquare tests were used to compare percentages. Graphic description of the data. Scatterplots of individual patients changes in tender joint count versus the change in each of the quality-of-life measures (PET for the whole study group, HAQ disability index and AIMS physical function score for each of the 2 subgroups) were generated. This was done in order to get a sense of the degree to which the changes in quality-of-life health status measures mirror (or are concordant with) the change in tender joint count. RESULTS Patients included in the analysis. Of the 144 patients enrolled in the study, 1 I2 (62 in the cyclosporine group and 50 in the placebo group) completed the 6-month treatment regimen. Patients who discontinued the study drug (details reported previously) continued to be evaluated monthly except for 2 patients, 1 of whom died (cyclosporine group) and 1 of whom dropped out of the study before the first followup (placebo group). For the purposes of the relative efficiency analysis, the remaining 142 patients were analyzed. Comparison of the outcome measures in terms of ability to detect a treatment effect. The relative efficiency of the various outcome measures is shown graphically in Figure 1. The measures relative efficiency to detect a treatment effect in comparison with tender joint count (RE tender joint count = 1) varied between 0.01 and The global measures all tended to perform well compared with tender joint count, although none of them was statistically different from tender joint count by Z statistic estimation. Both and pain assessed on a 5-point scale were statistically inferior to tender joint count in their ability to detect a treatment effect. While the RE of morning stiffness was only 0.23, this was not significantly different compared with tender joint count. A summary of the calculations performed is displayed in Table 1. For the sake of clarity, not all of the calculations described in Patients and Methods are shown. The most efficient measure in detecting a treatment effect was physician global efficacy assessed by a 5-point categorical scale (RE = 1.68). It had a large standardized effect size (-0.79). Although its relative treatment effect (-0.28) appeared small compared with other measures, there was not much noise in the measurement compared with other measures (CV = 0.35). The physician global disease activity measure had a similar relative efficiency (1.32) and standardized effect size (-0.70) when compared with the physician global efficacy measure. However, in contrast to the efficacy scale, this could be attributed to both a larger relative treatment effect (5.36) and a larger amount of noise (CV = -7.64). The was the least efficient measure (RE = 0.01). It had a small effect size (SES = 0.07), with both a small relative treatment effect (-0.38) and a moderate amount of noise (CV = -5.66) compared with other measures. While grip strength had the highest relative treatment effect (12.06), it showed the largest amount of noise (CV = 23.30), making it less sensitive. There was a comparatively small amount of noise in the PET measurement (CV = -2.16). The standardized effect sizes (ratio of effect size to variance of the effect size) were of similar magnitude for the remaining outcome measures. Comparison of the 4 pain and 3 quality-of-life measures in terms of ability to detect a treatment effect. The relative efficiencies of pain measured by the 10-cm VAS, 5-point categorical scale, HAQ pain scale, and

5 1572 BUCHBINDER ET AL morning stiffness = * Relative Efficiency ( = 1) Figure 1. Relative efficiency (compared with tender joint count) of the outcome measures analyzed. * = P < 0.05 versus tender joint count. = erythrocyte sedimentation rate; VAS = visual analog scale; Pt = patient; PET = Problem Elicitation Technique. AIMS pain scale are shown in Figure 2. Overall, there pared with tender joint count (less sensitive). The was a wide variation in the relative efficiencies of the 4 relative efficiency of the 10-cm VAS for pain was pain measures, although only the relative efficiency of similar for both the study group as a whole (RE = 0.45) the 5-point pain scale when estimated for the study and when calculated for each subgroup (RE lhaql = population as a whole was significantly different corn- 0.49, RE [AIMS] = 0.38). However, the relative Table 1. Relative efficiency of various outcome measures to detect a treatment effect: all patients* Observed Relative treatment Relative % Z statistic treatment Outcome measure effect? improvement$ SESB RE7 (PI# effect* * CVti Tender joint score (0.73) Swollen joint count (0.33) Grip strength (0.62) Morning stiffness (0.12) (0.01) Pain, 10-cm VAS (0.23) Pain, 5-point scale (0.03) Physician-rated global efficacy (0.21) Patient-rated global efficacy (0.76) Physician-rated global disease activity (0.57) PET (0.15) * SES = standardized effect size; RE = relative efficiency; CV = coefficient of variation; = erythrocyte sedimentation rate; VAS = visual analog scale; PET = Problem Elicitation Technqiue. See Patients and Methods for details.? Difference in the mean change between beginning and end of study in the cyclosporine group (a,) and the placebo group (a,), measured in the clinical units of the scale, i.e., at - a,. t Percent improvement in the cyclosporine group in excess of improvement in the placebo group, i.e., [(JJbaseline score in cyclosporine group - ajbaseline score in placebo group)] x Ratio of the treatment effect (a,- 4) to the pooled standard deviation (PSD) of these differences, i.e., (a,- &/PSD (see Patients and Methods for formula of PSD). 1T Relative efficiency to detect a difference among treatment groups with respect to joint count, i.e., (SESinStmment/SESjOint count)z. # Formal comparison of standardized effect sizes by an approximate Z test to determine if signficantly different from joint count, i.e., [(~E~i,,t,,,,I - S E S count~/q2(~/nt ~ ~ ~ ~ ~ + l/nc)(l - I~III. ** Relative magnitude of the treatment effect across different measures, i.e., (a, - tt Relative magnitude of the variance (or noise) across different measures, i.e., (PSDA,).

6 OUTCOME MEASURES IN RA CLINICAL TRIALS 1573 pain 10 cm VAS pain 5 pt scale AIMS psychological Figure 2. Relative efficiency (compared with tender joint count) of pain and quality-of-life measures in the total study population and in the Health Assessment Questionnaire (HAQ) and Arthritis Impact Measurement Scales (AIMS) subgroups. * = P < 0.05 versus tenderjoint count. VAS = visual analog scale; PET = Problem Elicitation Technique. efficiency of the 5-point categorical rating scale vaned between 0.18 for the study group as a whole, 0.46 for the HAQ subgroup, and 0.04 for the AIMS subgroup. Within the HAQ subgroup, the relative efficiency of the HAQ pain index was 0.31, and within the AIMS subgroup, the relative efficiency of the AIMS pain score was None of the functional status measures performed better than tender joint count as assessed by their relative efficiencies. The estimated relative efficiency of the PET was similar both for the study group as a whole (RE = 0.33) and for each subgroup (RE [HAQ] = 0.39, RE [AIMS] = 0.26). Within the HAQ subgroup, the relative efficiency of the HAQ disability index was 0.58, and within the AIMS subgroup, the Table 2. Relative efficiencies of the functional status measures using the PET and the tender ioint count as referent measures* Outcome measure PET AIMS subgroup I PET AIMS physical function score HAQ subgroup 1 relative efficiency of the AIMS physical function score was 0.04 and the relative efficiency of AIMS psychological status score was Calculation of the relative efficiencies of the functional status measures and pain scales using the PET or 10-cm VAS pain scale rather than the tender joint count as a referent measure yielded similar results. These are displayed in Tables 2 and 3. Relative efficiency of outcome measures calculated for the study group as a whole versus the HAQ and AIMS subgroups. Figure 3 displays the relative efficiency for the remaining measures, by subgroup. The relative efficiencies of the swollen joint count, grip Table 3. Relative efficiencies of the pain scales using the 10-cm VAS pain scale and the tender joint count as referent measures* Referent measure Referent measure Outcome measure Pain, 10 cm VAS AIMS subgroup Pain, 10-cm VAS 1 Pain, 5-point scale 0.11 AIMS pain score 3.66 HAQ subgroup * PET = Problem Elicitation Technique; AIMS = Arthritis Impact Measurement Scales; HAQ = Health Assessment Questionnaire. * VAS = visual analog scale; AIMS = Arthritis Impact Measurement Scales; HAQ = Health Assessment Questionnaire.

7 1574 BUCHBINDER ET AL Swollen joint count Grip strength Morning stiffness Dr global efficacy Pt global efficacy Dr disease activity Relative Efficiency ( = 1) UHAQ (n=77) Figure 3. Relative efficiency (compared with tender joint count) of outcome measures in the total study population and in the Health Assessment Questionnaire (HAQ) and Arthritis Impact Measurement Scales (AIMS) subgroups. = erythrocyte sedimentation rate; F t = patient. * = P < 0.05 versus tender joint count. strength, and global measures varied widely when measured in the HAQ and AIMS subgroups. In particular, they appeared much more responsive in the AIMS subgroup compared with the HAQ subgroup. For example, the RE of physician global efficacy was 3.24 in the AIMS subgroup, compared with 0.83 when measured in the HAQ subgroup. Homogeneity of the AIMS and HAQ subgroups and of treatment groups within these subgroups. Table 4 compares baseline characteristics between patients in the HAQ and AIMS subgroups, and patient characteristics by treatment group within each of the subgroups. When the overall means for each subgroup were compared, significantly more patients in the HAQ cohort were in functional class I11 (20). Within each subgroup, the cyclosporine-treated and placebotreated patients appeared to be comparable for each of the measures. Table 5 presents the evaluative measures at baseline for each cohort. With respect to baseline evaluative measures within the HAQ subgroup, there were no apparent differences between the cyclosporine-treated and placebo-treated patients. Within the AIMS subgroup, however, there were higher (worse) scores in the placebo group for both the PET score and the AIMS physical function score. When the overall mean values of evaluative measures at baseline were compared between the HAQ and AIMS cohorts, the mean scores for tender joint count, tender joint score, swollen joint count, and PET score were significantly Table 4. Patient characteristics at baseline in the HAQ and AIMS subgroups; overall and by treatment group Mean i SEM age, years % female % white Mean? SEM disease duration, years % taken 2 1 second-line drug % in functional class I11 % with subcutaneous nodules % with keratoconjunctivitis sicca ~ ~ HAQ subgroup (n = 77) AIMS subgroup (n = 65) C yclosporine Placebo C yclosporine Placebo Overall (n = 38) (n = 39) Overall (n = 32) (n = 33) 53.3 f f f t t % t f ~~ ~ ~ ~~ ~ ~~ * P = 0.001, overall Health Assessment Questionnaire (HAQ) subgroup versus overall Arthritis Impact Measurement Scales (AIMS) subgroup.

8 OUTCOME MEASURES IN RA CLINICAL TRIALS 1575 Table 5. Evaluative measures at baseline in the HAQ and AIMS subgroups, overall and by treatment group* HAQ subgroup (n = 77) AIMS subgroup (n = 65) Possible range Cyclosporine Placebo Cyclosporine Placebo (worst-best) Overall (n = 38) (n = 39) Overall (n = 32) (n = 33) f f2.1 Tender joint score $ f 3.1% Swollen joint count f f f Grip strength, rnm Hg f f f 7.7, mm/hour f f 5.3 Morning stiffness, minutes Allday f Pain, 10-cm VAS f f f Pain, 5-point scale f f f 0.2 Physician-rated disease activity PET f t t # 3.6 f # HAQ disability index ? HAQ pain index f AIMS physical function score f f 0.3** 4.8 -t 0.4** AIMS psychological status score * AIMS pain score * Except for possible range, values are the mean 2 SEM. = erythrocyte sedimentation rate; VAS = visual analog scale. See Table 2 for other definitions. t P = , overall HAQ subgroup versus overall AIMS subgroup. $ P = 0.000, overall HAQ subgroup versus overall AIMS subgroup. 5 P = 0.001, overall HAQ subgroup versus overall AIMS subgroup. 1 P = , overall HAQ subgroup versus overall AIMS subgroup. # P = 0.01, cyclosporine-treated verus placebo-treated patients. ** P = 0.03, cyclosporine-treated versus placebo-treated patients. higher in the AIMS cohort, suggesting that this subgroup overall may have had more active disease than the HAQ subgroup at baseline. Changes in individual patients. Figures 4A, B, and C show scatterplots of change in tenderjoint count versus change in PET, HAQ disability index, and AIMS physical score, respectively. The findings help explain the results of the estimation of relative efficiency for the quality-of-life instruments. Both the PET and the HAQ disability score changes (Figures 4A and B) showed a trend toward concordance with respect to tenderjoint count (i.e., most points were in the upper right or lower left quadrants of the plots). The correlation coefficient between the change in tender joint count and change in PET score was 0.44 overall, 0.48 for the active treatment group, and 0.35 for the placebo group. The correlation coefficient between the change in tender joint count and change in HAQ disability index was 0.41 overall, 0.54 for the active treatment group, and 0.21 for the placebo group. The pattern of scatter had a different appearance with regard to the change in AIMS physical score relative to the change in joint score (Figure 4C). Scatter points were more evenly distributed throughout all 4 quadrants. The correlation coefficients for change in tender joint count and change in AIMS physical function component were 0.30 overall, 0.36 for the active treatment group, and 0.26 for the placebo group. DISCUSSION This study provides additional data concerning the responsiveness of outcome measures from a multicenter randomized controlled trial comparing cyclosporine with placebo. Our overall analysis indicated that both physician and patient global measures were the most responsive instruments, although neither was statistically superior to tender joint count. Swollen joint count, grip strength, pain measured on a 10-cm VAS, and functional status as measured by the PET all demonstrated intermediate responsiveness. Morning stiffness, pain measured on a 5-point categorical scale, and estimation were the least responsive instruments, although only the latter 2 were significantly different from tender joint count. Both the ACR and the OMERACT conference committee have proposed a core set of outcome measures to be included in all rheumatoid arthritis clinical trials (1,2). The outcome measures proposed include tender joint count, swollen joint count, patient pain assessment, patient and physician global assessments,

9 1576 BUCHBINDER ET AL B.".. # 3.., -100 Change in Weighted PET Smre Change in HAQ Disabiliiy Index *" c Figure 4. Change in tender joint count versus change in weighted PET score (A), HAQ disability index (B), and AIMS physical function component (C), by treatment group (cyclosporine [O] or placebo [O]). For each treatment group, values in the lower left quadrant represent patients who showed improvement in both tender joint count and quality of life, values in the upper right quadrant represent patients who had worsening with respect to both of these measures, and values in either the upper left or lower right quadrant represent patients who showed improvement in one measure but worsening in the other. See Figure 2 for definitions. Change in AIMS Physical Fundion Component patient-assessed physical function, and level of an acute-phase reactant. These were chosen by first examining the currently available evidence that these measures meet all fundamental criteria for validity. These criteria include construct validity, i.e., disease activity measures should change in the same direction as the patient changes clinically; content validity, i.e., all aspects of clinical change should be captured by a set of outcome measures (not necessarily by each one); face validity, i.e., these outcome measures should appear sensible; criterion validity, i.e., the ability of outcome measures to predict a long-term outcome of importance (examined by reviewing the association of outcome measures with 3 long-term end points: disability, death, and radiologic damage); and discriminant validity, i.e., an outcome measure's responsiveness or sensitivity to change. The reliability of measures was also reviewed, and finally, each measure was examined to ensure contribution of significant, nonduplicated information. The core set proposed, as well as the methods suggested for assessing each outcome, reflect the set of measures which together optimally fulfill the above criteria. For example, an outcome measure may demonstrate good discriminant validity, i.e., be highly responsive to change, but lack criterion validity, i.e., not be correlated with any long-term end points of RA; it would hence be excluded from consideration for the core set. While measures of outcome should meet all measurement criteria, discriminant validity, or an instrument's sensitivity to change, is of particular importance in planning clinical trials. The more responsive an instrument, the smaller numbers of patients are required to achieve the same power; alternatively, the more responsive an instrument, the more power is achieved with the same sample size (21). The ACR utilized evidence from previous studies in RA which have examined the responsiveness of instruments (17,18,22,23) as well as further analysis of data from Cooperative Systematic Studies of the Rheumatic Diseases trials (16) and 2 trials of NSAIDs (24).

10 OUTCOME MEASURES IN RA CLINICAL TRIALS 1577 Table 6. Sensitivity to change (most sensitive to least sensitive) of disease activity measures in RA clinical trials* Authors, year (ref.); type of study Anderson et al, 1989 Bombardier et al, 1991 Gotzsche, 1990 (18); Tugwell et al, 1992 (16); 3 clinical trials of (17); 1 large trial of 130 RA NSAID trials (19); 1 large trial of Present study; 1 large DMARDs (CSSRD) auranofin? (meta-analysis) methotrexate trial of cyclosporine ~ Swollen joint count Physician global assessment of disease activity Tender joint score Platelet count Swollen joint score Grip strength Patient global assessment Pain (1-5 scale) Morning stiffness Hemoglobin Functional class PIP circumference Walk time Measures not evaluated Functional status measures Radiograph Patient s global assessment Overall health (6 days) Keitel index Pain (10-cm line) and tender joint count Swollen joint count and Quality of Well- Being Grip strength and Rand current health Walk time Morning stiffness Joint scores Laboratory other than Physician global assessment Radiograph Patient global assessment Pain Ritchie Articular Index Morning stiffness Swollen joint count Grip strength Walk time Digital joint size Functional status Radiograph Physician global assessment of disease activity MACTAR Pain (10-cm line) Joint swelling score Patient assessment of disease activity Joint paidtenderness score Joint swelling count Lee index McMaster Health Index and joint tenderness count PIP circumference Grip strength Morning stiffness Walk time Laboratory measures Radiograph Physician global efficacy assessment Physician global assessment of disease activity Patient global efficacy assessment Tender joint score Grip strength Swollen joint count Pain (10-cm line) PET Morning stiffness Pain point scale) Laboratory other than Radiograph * Measures are listed in descending order of sensitivity to change. Data in the first 3 columns are from Table 3 in ref. 1. RA = rheumatoid arthritis; DMARDS = disease-modifying antirheumatic drugs; CSSRD = Cooperative Systematic Studies of the Rheumatic Diseases group; NSAID = nonsteroidal antiinflammatory drug; = erythrocyte sedimentation rate; PIP = proximal interphalangeal joint; MACTAR = McMaster Toronto Arthritis Patient Preference Disability Questionnaire; PET = Problem Elicitation Technique. t Multiple measures of pain and health status were used; only those most sensitive to change are listed. Table 6 compares the disease activity measures used in various RA clinical studies in terms of their sensitivity to change. This table is modified from Felson et a1 (l), incorporating the data from the present study as well as one further analysis from the results of a trial comparing methotrexate with placebo (19). These studies all determined responsiveness of outcome measures by calculations of effect size. In only one previous study, however, was there a formal statistical comparison of the responsiveness of instruments to determine whether there was a statistically significant difference between them (17). We consider this preferable to making empirical judgments as to what constitutes high, moderate, or low sensitivity and what constitutes significant differences between measures. All measures that had high or intermediate responsiveness in our analysis, with the exception of grip strength, were included in the core set of outcome measures proposed by the ACR. However, previous studies have found grip strength to be relatively insensitive (17,18,22,23), although Bombardier et a1 found the RE of grip strength with respect to tender joint count to be 0.83 in a randomized controlled trial of auranofin and placebo (17), comparable with our estimate of Grip strength has been shown to be a predictor of subsequent physical disability (criterion validity) (25-27) and is also reliable, especially if assessments are standardized (28). Felson et a1 found the sensitivity of grip strength to be highly dependent upon disease duration (29), and this variable is also likely to be more sensitive in patients who have more severe hand disease. It was excluded from the ACR core set, however, on the basis of its relative insensitivity compared with other measures. The relative insensitivity of morning stiffness in detecting a treatment effect is in accordance with the

11 1578 BUCHBINDER ET AL results of previous studies (17,18,19,22), and morning stiffness was excluded from the ACR core set (1). The responsiveness of in previous studies has been inconsistent. While it was sensitive in 2 studies which examined responsiveness in trials of methotrexate, intramuscular and oral gold, penicillamine, and auranofin (17,22), it was relatively insensitive in a metaanalysis of 130 trials of NSAIDs (18). Perhaps this may be related to the biologic effect of the therapies on and may explain its poor performance in our study. Different acute-phase reactants may be preferred in different studies, e.g., C-reactive protein performs better than in studies of cyclosporine. Further work is needed to determine how this should be taken into account when measuring the responsiveness of outcome measures. The proposed core set of outcome measures includes a physician global assessment. Two physician global assessment measures were used in the present study: a transition scale (physician global efficacy assessment), comparing the patient s condition at study entry versus study end utilizing a 5-point scale from marked improvement to marked worsening, and an assessment of disease state by the physician at both points in time, again utilizing a 5-point scale from asymptomatic to very severe. The relative efficiencies of the two were similar, although the former had both a small relative treatment effect and small coefficient of variation and the latter had a somewhat larger value for both. Further study is required to determine which of these measures may be more useful. The subgroup analysis revealed discrepancies in estimates of responsiveness for some of the outcome measures assessed. This raises several important issues about the generalizability of the results of studies examining the attributes of evaluative instruments. First, can the results of an estimation of responsiveness of an outcome measure be extrapolated to other patient groups? Further analysis of our data showed that patients in the AIMS subgroup had more active disease at baseline. One possible explanation for the instruments appearing more responsive in this subgroup may have been the fact that these patients had the greatest potential to improve. Felson et a1 explored whether outcome measures were equally sensitive to change in different subgroups of patients participating in several clinical trials and found that tender joint count was a more sensitive outcome measure in patients with a high, as opposed to moderate or low, initial tender joint count (I). A second issue is whether the results of an estimation of responsiveness of an outcome measure can be extrapolated to other settings. Hawley and Wolfe examined the sensitivity of outcome measures on clinic patients followed up over time and concluded that variables that are most effective in controlled clinical trials are not effective as long-term measures of RA outcome (30). These results need to be confirmed in another independent dataset before clinicians accept the notion that clinical trial end points are not useful in clinic patients. Other issues include the question of whether the results can be extrapolated to other therapies (e.g., lack of effect of cyclosporine in reducing the ) and other investigators (e.g., variation in techniques of measurement, reliability, and accuracy of scoring). While none of the 3 quality-of-life measures was statistically significantly different from tender joint count in the ability to detect a treatment effect, the AIMS physical function score appeared to be the least responsive. This contrasts with the findings of previous studies examining the responsiveness of the AIMS to short-term clinical changes, in which this instrument was found to closely parallel improvements demonstrated by traditional clinical measures (13,22). The AIMS has been revised since our study was performed, however, and the revisions would seem to be particularly focused on improving its responsiveness. This will require further study. We have followed the guidelines of Pocock (3 1) in providing visual descriptions of our data. We believe the scatterplots make a valuable contribution to our understanding of the results. Small or subtle, but clinically important, results may be overlooked by a purely numerical analysis. In addition, visual display techniques help to explain the results of statistical analyses and contribute to the reader s understanding of the results. For example, the poor correlation between change in AIMS physical function score and tender joint count is evident upon examining the scatterplot and explains the poor relative efficiency of AIMS physical function demonstrated in this study. The differing pattern of scatter in comparison with the other plots suggests that the AIMS physical function score may be measuring an attribute different from that measured by the HAQ disability index and PET. The relative efficiencies of outcome measures in the present study were very similar to the findings in a double-blind randomized trial of met hotrexate and placebo (19). However, the MACTAR, in comparison with both traditional clinical end points and standard-

12 OUTCOME MEASURES IN RA CLINICAL TRIALS 1579 item health status questionnaires, performed better than the PET in the present study. This can probably be explained by the fact that the MACTAR looks only at improvement of problems over time and does not consider either a worsening of problems or the development of new problems. On further analysis of the PET data, there also seemed to be substantial variation in the execution of the instrument between centers/ investigators. The PET has been revised since our study was performed, and these revisions have been mainly directed toward standardizing its administration. In addition, both an interviewers manual and a troubleshooting guide have been developed. While it is not evident by this study, we believe that the revisions to the MACTAR, by incorporating both a measure of the magnitude of each problem and assessing any change in this magnitude (whether better or worse), will enhance the responsiveness of this instrument. In summary, this study provides further evidence to support the core set of outcome measures chosen by the ACR and OMERACT. In addition, it highlights the importance of testing the statistical significance of conclusions drawn from studying the behavior of outcome measures-in our overall analysis, apart from and pain measured on a 5-point categorical scale, none of the measures were significantly different from tender joint count in their ability to detect a treatment effect REFERENCES Felson DT, Anderson JJ, Boers M, Bombardier C, Chernoff M, Fried B, Furst D, Goldsmith C, Kieszak S, Lightfoot R, Paulus H, Tugwell P, Weinblatt M, Widmark R, Williams HJ, Wolfe F: The American College of Rheumatology preliminary core set of disease activity measures for rheumatoid arthritis clinical trials. Arthritis Rheum 36: , 1993 Tugwell P, Boers M, for the OMERACT Committee: Developing consensus on preliminary core efficacy endpoints for rheumatoid arthritis clinical trials. J Rheumatol , 1993 Kirshner B, Guyatt G: A methodological framework for assessing health indices. J Chron Dis 38:27-36, 1985 Tugwell P, Bombardier C, Gent M, Bennett KJ, Benson WG, Carette S, Chalmers A, Esdaile JM, Klinkhoff AV, Kraag GR, Ludwin D, Roberts RS: Low-dose cyclosporin versus placebo in patients with rheumatoid arthritis. Lancet 335: , 1990 Fries JF, Spitz PW, Kraines RG, Holman HR: Measurement of patient outcome in arthritis. Arthritis Rheum 23: , 1980 Meenan RF, Gertman PM, Mason JH: Measuring health status in arthritis: the Arthritis Impact Measurement Scales. Arthritis Rheum 23: , 1980 Tugwell P, Bombardier C, Buchanan WW, Goldsmith CH, Grace E, Hanna B: The MACTAR Patient Preference Disability Questionnaire: an individualized functional priority approach for assessing improvement in physical disability in clinical trials in rheumatoid arthritis. J Rheumatol 14:44&451, Arnett FC, Edworthy SM, Bloch DA, McShane DJ, Fries JF, Cooper NS, Healey LA, Kaplan SR, Liang MH, Luthra HS, Medsger TA Jr, Mitchell DM, Neustadt DH, Pinals RS, Schaller JG, Sharp JT, Wilder RL, Hunder GG: The American Rheumatism Association 1987 revised criteria for the classification of rheumatoid arthritis. Arthritis Rheum 31: , Klinkhoff A, Bellamy N, Bombardier C, Carette S, Chalmers A, Esdaile JM, Goldsmith C, Tugwell P, Smythe HA, Buchanan WW: An experiment in reducing observer variability of the examination of joint tenderness. J Rheumatol E , Huskisson EC: Measurement of pain. J Rheumatol 5: , I982 If. Fries JF, Spitz PW, Young DY: The dimensions of health outcomes: the Health Assessment Questionnaire, disability and pain scales. J Rheumatol 9: , Bombardier C, Ware J, Russell IJ, Larson MG, Chalmers A, Read JL: Auranofin therapy and quality of life in patients with rheumatoid arthritis: results of a multicenter trial. Am J Med 81: , Meenan RF, Anderson JJ, Kazis LE, Egger MJ, Altz-Smith M, Samuelson CO Jr, Willkens RF, Solsky MA, Hayes SP, Blocka KL, Weinstein A, Guttadauria M, Kaplan SB, Klippel J: Outcome assessment in clinical trials: evidence for the sensitivity of a health status measure. Arthritis Rheum 27: , I Meenan RF, Gertman PM, Mason JH, Dunaif R: The Arthritis Impact Measurement Scales: further investigations of a health status measure. Arthritis Rheum 25: , Liang MH, Larson MG, Cullen KE, Schwartz JA: Comparative measurement efficiency and sensitivity of five health status instruments for arthritis research. Arthritis Rheum 28: , Anderson JJ, Felson DT, Meenan RF, Williams HJ: Which traditional measures should be used in rheumatoid arthritis clinical trials? Arthritis Rheum 32: , Bombardier C, Raboud J, and the Auranofin Cooperating Group: a comparison of health-related quality-of-life measures for rheumatoid arthritis research. Control Clin Trials 12:243S- 2568, Gotzsche PC: Sensitivity of effect variables in rheumatoid arthritis: a meta-analysis of 130 placebo-controlled NSAID trials. J Clin Epidemiol 43: , Tugwell P, Bombardier C, Buchanan WW, : Impact of methotrexate upon quality of life in rheumatoid arthritis assessed by traditional standard items and individualized patient preference health status questionnaires. Arch Intern Med 150:59-62, Steinbrocker 0, Traeger CH, Batterman RC: Therapeutic criteria in rheumatoid arthritis. JAMA 140: , Guyatt G, Walter S, Norman G: Measuring change over time: assessing the usefulness of evaluative inst&mentst J Chron Dis , Anderson JJ, Firschein HE, Meenan RF: Sensitivity of a health status measure to short-term clinical changes in arthritis. Arthritis Rheum 32: , Wolfe F, Hawley DJ, Cathey MA: Clinical and health status measures over time: prognosis and outcome assessment in rheumatoid arthritis. J Rheumatol 18: , Kolodny AL: Two double-blind trials of diclofenac sodium with aspirin and with naproxen in the treatment of patients with rheumatoid arthritis. J Rheumatol 15: , Spiegel JS, Paulus HE, Ward NB, Speigel TM, Leake B, Kane RL: What are we measuring? An examination of walk time and grip strength. J Rheumatol 14:8&86, Van der Heijde DMFM, van Riel PLCM, van Rijswijk MH, van de Putte LBA: Influence of prognostic features on the final outcome in rheumatoid arthritis: a review of the literature. Semin Arthritis Rheum 17:28&292, 1988

13 1580 BUCHBINDER ET AL 27. Wolfe F, Cathey MA: The assessment and prediction of functional disability in rheumatoid arthritis. J Rheumatol 18: , Bellamy N, Anastassiades TP, Buchanan WW, Davis P, Lee P, McCain GA, Wells GA, Campbell J: Rheumatoid arthritis antirheumatic drug trials. I. Effects of standardization procedures on observer dependent outcome measures. J Rheumatol 18: , Felson DT, Anderson JJ, Meenan RF: The comparative efficacy and toxicity of second-line drugs in rheumatoid arthritis: results of two metaanalyses. Arthritis Rheum 33:144%1461, Hawley DJ, Wolfe F: Sensitivity to change of the Health Assessment Questionnaire (HAQ) and other clinical and health status measures in rheumatoid arthritis: results of short term clinical trials and observational studies versus long term observational studies. Arthritis Care Res 5: , Pocock SJ: A perspective on the role of quality-of-life assessment in clinical trials. Control CIin Trials 12:257S-265S, 1991