Evaluating the Effectiveness of Stroke Rehabilitation: Choosing a Discriminative Measure

92 Evaluating the Effectiveness of Stroke Rehabilitation: Choosing a Discriminative Measure Kim A. Brock, PhD, Patricia A. Goldie, PhD, Kenneth M. Greenwood, PhD ABSTRACT. Brock KA, Goldie PA, Greenwood KM. Evaluating the effectiveness of stroke rehabilitation: choosing a discriminative measure. Arch Phys Med Rehabil 2002;83: 92-9. Objective: To evaluate the discriminative ability of several measures of physical disability used to determine quality of outcome for poststroke rehabilitation. Design: A comparative study, using Rasch analysis, of the discriminative ability of functional status and mobility measures in rehabilitation patients with stroke. Setting: A 26-bed rehabilitation unit, on site of a tertiary teaching hospital in Melbourne, Australia. Participants: A consecutive sample of 106 patients with acute stroke admitted for rehabilitation. Interventions: Not applicable. Main Outcome Measures: Rasch analysis of the motor subscale of the FIM instrument, Motor Assessment Scale, Functional Ambulation Classification, gait velocity, and gait endurance. Results: The more difficult items of the FIM motor scale adequately discriminated among higher functioning patients. The gait velocity measure further distinguished 9% of the sample, who functioned at a higher level than could be indicated by FIM motor subscale. The other measures did not add levels of discrimination to that provided by the FIM motor. Ability estimates provided by Rasch analysis of the FIM motor scale were a more accurate indication of ability than raw scores. Raw scores underestimated change in ability observed at higher levels of ability. Conclusion: Rasch estimates of the FIM motor subscale provide a discriminative measure for evaluating outcomes and change in ability achieved in stroke rehabilitation. Key Words: Cerebrovascular disorders; Disability evaluation; Outcome assessment (health care); Rehabilitation. 2002 by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation The search for appropriate outcome measures has been a focus of rehabilitation research for decades. Introducing benchmarking processes into health care, which facilitates comparisons of resource use and outcomes, has intensified From the Physiotherapy Department, St Vincent s Hospital, Fitzroy, Melbourne, Australia (Brock); and Schools of Physiotherapy (Goldie) and Psychological Science (Greenwood), La Trobe University, Bundoora, Victoria, Australia. Accepted in revised form March 7, 2001. Presented at the Joint Annual Scientific Meeting of the Australasian Faculty of Rehabilitation Medicine, the British Society of Rehabilitation Medicine, and the International Association of Health Professionals in Rehabilitation, May 27-30, 1998, Sydney, Australia. No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the author(s) or upon any organization with which the author(s) is/are associated. Reprint requests to Kim Brock, PhD, Physiotherapy Dept, St Vincent s Hospital, Victoria Pde, Fitzroy, 3065 Melbourne, Australia, e-mail: brockka@svhm.org.au. 0003-9993/02/8301-6469$35.00/0 doi:10.1053/apmr.2002.27348 discussion about valid methods for outcome evaluation. Without evidence of outcomes, measures of resource use may be used to justify spending reductions when the true need is to maximize health benefits for the money spent. 1 Benchmarking processes for analyzing costs of delivering health care are more advanced than processes for determining outcomes of health care. Although this difference exists, changes in service delivery may be driven mainly by cost factors, with insufficient regard to the quality of outcomes associated with the changes. A major issue in pursuing outcome-based benchmarking is the adequacy of the tool used to measure outcome. Tools used for this purpose must be reliable, valid, and discriminative. Examining discriminative ability is important to ensure that a chosen outcome measure is able to differentiate within the patient group and to identify meaningful differences in patients abilities. 2 A discriminative scale will have items that span the range of patient ability, with appropriately spaced intervals. Failure to differentiate among patients at higher levels of function may produce inaccurate evaluations of the effectiveness of rehabilitation units. Functional status measures such as the FIM instrument, 3 which assess the assistance required for a person to complete daily living tasks, are the most widely used outcome measures. Concerns have been expressed about possible ceiling effects of these ordinal disability scales. 4-8 Dodds et al 4 argue that the FIM does not take into account elements such as speed, ease, and quality of task performance. Johnston et al 9 argue that ordinal functional status measures are responsive to improvement in a limited part of the range only, with both floor and ceiling effects. For patients with less disability, Johnston suggests that measures of speed and endurance may be important. In light of these comments, it may be useful to compare the discrimination of the FIM motor subscale with other measures in the physical domain that include aspects such as speed and quality of task performance. The present study considered outcome measures that reflect mobility function. In the rehabilitation of motor function, much emphasis is placed on the quality and ease of walking and of motor tasks such as moving from sitting to standing. This study investigates whether the motor subscale of the FIM adequately represents mobility function after rehabilitation. The appropriateness of mathematically manipulating ordinal values of functional status measures, such as summing scores and calculating change values, has also come under question in the literature. 10-12 Item response theory, using Rasch analysis, has been used to investigate the properties of ordinal scales, including the FIM. With Rasch analysis, ordinal scales are converted into interval measures by estimating the increase in ability for each level of the scale, for each of the items. 12,13 Several studies have shown the validity of developing Rasch estimates of ability from the FIM. 14-16 Linacre et al 15 showed that the steps between scores on the FIM motor scale are not equal in level of difficulty. A change in ability at the top end of the scale was of greater magnitude when converted into Rasch estimates than a similar change in ability toward the middle of the scale. A 10-point change in raw data scores from 80 to 90, when converted to Rasch estimates, indicates 4 times as much

CHOOSING A DISCRIMINATIVE MEASURE, Brock 93 change as a change in raw scores from 40 to 50. This finding indicates that summed raw scores cannot be considered to be an interval scale for the FIM motor subscale. However, the rehabilitation field seems slow to adopt the use of Rasch estimates, and summed raw scores remain the more frequently reported indicator of outcome. The purpose of the present study was to compare several outcome measures, using Rasch analysis, to distinguish which measure or combination of measures is the most discriminative for assessing physical disability at the end of intensive rehabilitation after stroke. METHODS Measures of Physical Ability We examined 3 ordinal scales: the 13 items that make up the motor component of the FIM motor subscale 3 (FIM motor), the Motor Assessment Scale 17 (MAS), and the Functional Ambulation Classification 18 (FAC). The FIM motor assesses the level of assistance required to perform various activities of daily living. Extensive investigations of the FIM s reliability and validity have provided evidence of its interrater and test-retest reliability, 19 internal consistency, 20,21 concurrent validity, 22,23 and predictive validity. 23-26 The MAS 17 assesses 5 functional motor tasks (supine to side lie, supine to sitting, sitting, sit to stand, walking) by using criteria that address the quality of performance as well as the level of assistance required. The instrument s quality aspects include symmetry, control, timing of movement, and use of the affected side. The upper limb components of the MAS were not considered in the present study. Interrater reliability, 17,27,28 concurrent validity, 28,29 and predictive validity 30 have been shown for the MAS. The FAC 18 addresses walking ability relevant to community ambulation, such as ability to walk on rough ground, ramps, and over curbs. Limited reliability and validity evaluations have been undertaken, including interrater reliability 31 and concurrent validity with other gait measures. 18,31,32 We included 2 interval scaled measures of walking ability gait velocity and endurance in the present study, to compare the discriminative ability of ordinal scales and specific interval measures. Gait velocity is a widely used measure of function at the end of rehabilitation. Few stroke patients achieve normal gait velocity, with a recent study finding that 83% of stroke patients were still impaired in terms of gait speed at 3 months poststroke. 33 Studies have provided evidence of interrater reliability, 18,34,35 test-retest reliability, 18,35,36 concurrent validity, 18,31,32,37-41 and predictive validity. 42 Previous studies 36,43,44 have shown gait velocity to be a discriminative measure. Few functional status measures include an assessment of endurance beyond the 50-meter requirement of the FIM. An audit of stroke outcomes undertaken by Hill et al 45 showed that only 15% of patients could walk more than 500 meters at discharge. To date, there are no simple tests of endurance with established reliability and validity. We included a standardized test of endurance in the present study, requiring the subject to walk laps of a 50-meter circuit. If the areas of function represented by the 4 ordinal scales and 2 interval measures are shown to be measuring the same domain (physical disability), then Rasch analysis could be used to compare difficulty levels of items in the disability scales and the gait measures. This comparison would then permit one to identify the measures that most adequately differentiate between various levels of ability at the higher end of the spectrum of functional ability, observed at the end of rehabilitation. A key advantage to using Rasch analysis in the evaluation of rating scales is the potential to include measures that do not cover the whole spectrum of ability but are more sensitive to a portion of the range. 46 This provides a method for integrating measures, inclusive of measures that are not as discriminative but cover the range of ability, and measures that discriminate well, but only in a portion of the range. Few studies have used Rasch analysis to undertake comparative analysis of the discriminative ability of scales. Fisher et al 47 used it to compare the discriminative abilities of the FIM motor and 22 items from the motor competency components of the Patient Evaluation Conference System, including items assessing housework and meal preparation. Grimby et al 48 combined the physical activities of the FIM with the Instrumental Activity Measure (IAM). The IAM assesses domestic tasks, such as cleaning and shopping, and community mobility. The aim of using Rasch analysis to investigate outcome measures in the present study was to establish which measure, or combination of measures, would yield a single ability score for each patient that most accurately reflects mobility outcome. This score could represent either functional level at outcome or change in function between admission and discharge. Having established a discriminative measure, comparisons of outcome in relation to resource use could then be undertaken with greater confidence. At the time of the study (1993 1995), the funding of rehabilitation services in Victoria was provided by block grants, allowing rehabilitation teams to set individual goals and programs for each patient, without being restricted by specific requirements of funding agencies. This approach permitted wide variation in length of stay (LOS) in the rehabilitation unit at St Vincent s Hospital, to achieve high levels of outcome, with a large proportion of discharges to independent living in the community. During the study, few alternatives to inpatient care were available for intensive rehabilitation services in Victoria. Almost all patients in the present study had stopped participating in intensive rehabilitation at discharge; some patients had prolonged inpatient stays to achieve sufficient function to return home. Therefore, in this study, discharge from inpatient rehabilitation adequately represents the end point of intensive rehabilitation. Subjects All patients admitted to the St Vincent s Hospital Rehabilitation Unit from 1993 to 1995 with a primary diagnosis of stroke, infarct, or hemorrhage were included in the study. Patients were excluded if they suffered another major incident at onset of stroke (eg, fracture, amputation), or had cerebrovascular accidents related to trauma. Patients were selected for admission to the rehabilitation unit according to criteria that included potential to return to some form of independent living and the ability to cope with an intensive rehabilitation program. The study was approved by the ethics committees of St Vincent s Hospital and the La Trobe University Faculty of Health Sciences. One hundred six patients were included in the study. The patients had an average age standard deviation of 68.7 11.3 years (range, 36 91yr). Sixty-seven (63%) were men. Forty-nine patients (46%) had right-hemisphere lesions, 48 (45%) left-hemisphere lesions, and 9 patients (9%) had bilateral lesions. Eighteen patients (17%) had hemorrhagic strokes and 87 patients (82%) had infarcts. The pathology was unknown for 1 patient. Before admission, 27% of patients lived alone, 49% lived with a spouse, 17% with their family, and 7% lived in special accommodation houses or hostels. Table 1 shows median time from onset of stroke to admission to reha-

94 CHOOSING A DISCRIMINATIVE MEASURE, Brock Table 1: Hospital LOS and Functional Status of Cohort Median 25th Percentile 75th Percentile Range LOS (d) Onset to admission 11 8 15.5 0 52 Rehabilitation (d) 28 1668 3 274 Functional status (summed raw scores) Admission FIM motor 67 51 80 18 91 Discharge FIM motor 8681.5 89 32 91 bilitation, LOS in rehabilitation, and median admission and discharge FIM motor scores. At the end of rehabilitation, 89% of patients returned home, 7% to semi-independent living, and 4% to nursing home care. Instruments The FIM and MAS were assessed at admission and discharge. Additional mobility variables of gait speed, endurance, and rating on the FAC were assessed at discharge. Procedure Patients were assessed at admission and at discharge from rehabilitation. The FIM motor scores were evaluated according to the FIM guidelines by rehabilitation team members considering the patient s performance over a 24-hour period. During the study period, training and examination of FIM reliability was performed at least twice yearly, and at least 10 staff on the Rehabilitation Unit achieved satisfactory reliability, as measured by the Uniform Data System for Medical Rehabilitation examination process, at any 1 time. The MAS was assessed by 1 of 3 physiotherapist raters, all of whom had undergone reliability testing on the MAS through the School of Physiotherapy, University of Sydney, achieving satisfactory reliability scores of above 80%. Walking speed was assessed by the treating physiotherapist as unassisted, self-selected walking speed over the central 6 meters of a 10-meter walkway. The use of aids and splints customarily worn was allowed and recorded. The endurance test required the patient to walk laps of a 50-meter circuit outdoors. The patients were asked to walk as many laps as you can, without tiring yourself, to a maximum of 10. The FAC was rated by the treating physiotherapist. Statistical Analysis Discriminative ability was evaluated by examining the spread of scores for ceiling effects and through Rasch analysis. The use of Rasch analysis is dependent on the underlying variations in behavior being dominated by 1 dimension. 12 Before subjecting the data to Rasch analysis, we applied principal components analysis 49,50 to establish the unidimensionality of the measures used, for the measures individually and for combined data sets. At the end of intensive rehabilitation, we calculated item estimates (level of difficulty of items) and case estimates (level of ability of persons tested) by using Rasch analysis for each functional measure. These estimates are measured as logits, mathematically defined units that are constant from 1 end of a continuum to the other. Comparing estimates allowed us to compare both the difficulty level of items on a scale and the ability level of patients. Rasch analysis also provides fit statistics, which indicate how the observed ratings vary from those predicted by the model. These fit statistics show the estimate s internal validity and indicate the unidimensionality of the model. 51 The stability of the measures over time was investigated to determine whether the relative level of difficulty of the items remained constant when assessed at different points in time. 15 This attribute is essential to the validity of a measure used for quantifying change. The stability of FIM motor scores has been shown in several studies. 15,16,52,53 The stability of the measures used in the present study was examined by using intraclass correlation coefficients (ICCs), as recommended by Chang and Chan. 53 Rasch analysis does not provide case estimates for perfect scores. If a considerable portion of the sample achieve a perfect score, one must determine the level of ability this score indicates. Otherwise, this group s achievements are not considered when assessing change. Case estimates for perfect scores on the FIM motor and MAS were established by adding a more difficult item from another measure to the functional scale and performing another Rasch analysis. The items on the functional status measures that discriminated among higher level performances were then compared by entering them into a further Rasch analysis. The purpose of which was to identify the functional status measure, or combination of measures, that most adequately discriminated among higher level performances. Data analysis was performed on a personal computer, using the computer software packages SPSS, version 6.0, a and QUEST. b RESULTS Spread of Scores The measures were examined for ceiling effects at discharge. Table 2 shows the proportion of patients achieving the highest score for each of the measures and on the hardest item of the measure. The FAC had the strongest ceiling effect (table 2). The endurance test revealed that 39% of subjects were able to walk 500 meters on outdoor surfaces without a rest. In contrast, the FIM motor subscale had 16% and the MAS had 25% of persons scoring the highest score. Gait velocity has no ceiling effect. However, to facilitate comparison, the gait velocity variable was grouped into 8 classes, with the fastest walkers being those who walked at more than 70m/min. This cutoff was chosen on the basis of a study 54 investigating the gait velocity required to safely cross signalled road intersections in Melbourne. Patients with a selfselected gait velocity of 70m/min have sufficient gait velocity to safely cross almost all signalled intersections in Melbourne, at their normal walking speed. Nine percent of patients achieved this optimum speed. This test showed the least ceiling effect. Although the FAC and endurance showed large ceiling effects, these single-item tests may yield useful information in combination with other tests. The validity of combining gait Table 2: Percentage of Patient s Achieving the Highest Score for Each Measure Total (%) Most Difficult Item (%) FIM motor 1629 MAS 25 35 Gait velocity (grouped) 9 Endurance 39 FAC 46

CHOOSING A DISCRIMINATIVE MEASURE, Brock 95 endurance, the FAC, and gait velocity into a variable representing gait function was explored. Principal Components Analysis Principal components analysis of FIM motor scores showed that the scale closely approached a unidimensional scale: 2 factors had eigenvalues above 1.0 (8.6, 1.2). The items of bowel function and eating loaded most strongly onto the second factor. Principal components analysis of MAS scores revealed 1 factor with an eigenvalue above 1.0. We entered the 3 items assessing gait (gait velocity, gait endurance, FAC score) into principal components analysis to establish their unidimensionality. One factor was revealed with an eigenvalue above 1.0. Because all the measures met the criteria of unidimensionality to a reasonable degree, the measures were further examined with Rasch analysis. Fig 1. Item threshold estimates for FIM motor scores at discharge. Abbreviations: Eat, eating; Groom, grooming; Bath, bathing; Dress up, dressing the upper body; Dress low, dressing the lower body; Toilet, toileting; Bed tr, bed transfers; Toilet tr, toilet transfers; Tub tr, tub transfers. Fig 2. Summed raw scores and Rasch estimates of FIM motor scores at discharge. (Rasch estimates and summed raw scores were rescaled to range between 0 and 100.) Rasch Analysis FIM motor. Rasch analysis was performed on the discharge FIM motor scores. One item, bladder function, showed serious misfit, as indicated by the infit statistic (2.61). Figure 1 shows the estimated difficulty of each level of the 13 items. Item levels that are scored similarly in the FIM motor may have markedly different weightings in the Rasch analysis. The FIM motor item estimates obtained in this study were compared with those quoted in the study by Linacre et al 15 to examine how closely our small sample related to the very large sample of Linacre. Correlation of combined admission and discharge estimates yielded a Pearson s r correlation of.87, indicating substantial similarity between the item estimates. Sixteen percent of patients had a perfect score on the FIM motor. Review of case estimates revealed only 5 cases with infit t values beyond 2.0 or 2.0 that seriously misfit the model. The mean case estimate was 2.34 1.56, indicating that the sample was biased toward cases with higher scores, as would be expected in the discharge scores of a selected rehabilitation sample. The item separation reliability 51 was.85. Although the Rasch modeling shows some areas of concern, the overall model fit justified our continuing with the analysis. Calculation of the ICC of each item s level of difficulty at 2 weeks poststroke and at discharge revealed an ICC of.87. Chang and Chan 53 stipulated a cutoff of an ICC value of.90 as an indication of the appropriateness of producing generalized item difficulty estimates from scores at both occasions. Although the present value falls short of that.90 threshold, it is close to the specified level. Because 3 previous studies 16,52,53 found the FIM motor s stability to be acceptable, we combined the admission and discharge scores in the present study. The item thresholds developed on the combined data set are used later in discussion of Rasch transformation of the FIM motor. As previously discussed, Rasch techniques do not provide an estimate for perfect or zero scores. This is a problem for comparison of summed raw scores and Rasch estimates and for the calculation of change. A value for perfect scores was ascertained by adding another variable to the Rasch analysis. The only variable in the study that had less ceiling effect than the FIM motor was gait velocity (grouped as previously discussed). We performed a further Rasch analysis with the FIM motor variables and gait velocity. Before this, we assessed the unidimensionality of the combined FIM motor and gait velocity by using principal components analysis. Two factors were revealed with eigenvalues of 7.1 and 1.7, with the gait and transfer items loading heavily on the first factor and eating and grooming loading heavily onto the second factor. The fit statistics of the Rasch analysis showed an infit mean square value of 1.4 for gait velocity. Although the fit of gait velocity to the FIM scale was not optimum, it was considered reasonable to continue this analysis because it was the most accurate means of obtaining an estimate for perfect scores on the FIM motor. The difference in Rasch case estimates between cases with raw scores of 90 and 91 revealed that the step size between 90 and 91 in raw scores was equivalent to 11% of the Rasch estimates. This finding indicates that a large improvement in ability is required to move from a score of 90 to 91, compared with an improvement of 1 point at other points of the scale. Calculating a Rasch estimate for perfect FIM scores enables more accurate comparison of Rasch estimates and summed raw scores. The relationship between the summed raw scores and Rasch estimates at discharge displayed an ogival relationship at the upper end (fig 2). The correlation between summed raw scores and Rasch estimates was.92 at admission and.83 at discharge. Rasch estimates and summed raw scores were compared as measures of change. The correlation between change as summed raw scores and Rasch estimates was.71 (fig 3). Figure 4 shows that for those subjects with low admission scores, summed raw scores tended to overstate the degree of change in ability, and for subjects with high admission scores, the summed raw scores tended to understate the degree of change in ability.

96 CHOOSING A DISCRIMINATIVE MEASURE, Brock Fig 3. Summed raw scores and Rasch estimates for change in FIM motor scores. Motor Assessment Scale. Rasch analysis was performed on the discharge MAS scores. One item, sitting, had an infit mean square value above 1.3 (1.45). Figure 5 shows the estimated difficulty of each level of the 5 items. As with the FIM motor, levels of items that are scored similarly in the MAS may have markedly different weightings in the Rasch analysis. Review of case estimates revealed only 2 cases with infit t values beyond 2.0 or 2.0. The mean case estimate was 1.94 1.60. The item separation reliability was.70. Calculation of the ICC of the estimates at 2 weeks poststroke and discharge revealed an ICC of.77. This ICC is considerably lower than the cutoff of.90, stipulated by Chang and Chan 53 as an indication of the appropriateness of producing generalized item difficulty estimates from scores at both occasions. The lack of stability of the thresholds of the items over time precludes combining admission and discharge data into a single data set, as was performed for the FIM motor. This finding therefore, decreases the validity of the MAS Rasch estimates being used to measure change. For the present analysis, the MAS was used as a measure of ability at discharge only. To obtain a value for subjects who had a perfect score, we added another variable to the Rasch analysis. Gait velocity was the variable of choice, having less ceiling effect than the MAS. A further Rasch analysis was performed with the MAS and gait velocity. Before this analysis, the unidimensionality of the MAS with gait velocity was explored with principal components analysis. The results were very similar to the results for Fig 5. Item threshold estimates for the MAS at discharge. the MAS alone, with only 1 factor having an eigenvalue above 1.0. A Rasch analysis was performed, adding gait velocity to the MAS. The fit statistics of the Rasch analysis showed an infit mean square value of.87 for gait velocity. The difference in Rasch case estimates between cases scoring 29 and 30 on the MAS was calculated and converted into a percentage of the available range of Rasch estimates. Thus, it was revealed that the difference between 95 and 100 in raw scores was equivalent to 17% of the range of Rasch estimates. Cases with perfect scores were recoded to reflect this relationship, with a case estimate of 4.89. Rasch estimates were then recoded to range between 0 and 100 to facilitate comparison with summed raw scores. The correlation between summed raw MAS scores and Rasch estimates at discharge was.92. Although the correlation is relatively high, summed raw scores understate the level of ability for the cases with the highest MAS scores. Gait measures. Rasch analysis was performed on the gait measures. The infit mean square values of the 3 items indicate that no items misfitted the model. Figure 6 shows the estimated difficulty of each level of the 3 items. Ten cases had perfect scores at discharge. Review of case estimates revealed only 3 cases with infit t values beyond 2.0 or 2.0. The mean case estimate was 1.43 1.83. The item separation reliability was.83. Comparison of the Discriminative Properties of the Functional Status Measures The discriminative properties of selected items from the FIM motor, the MAS, and the gait measures were compared by Fig 4. Summed raw scores and Rasch estimates of change in FIM motor scores for randomly selected cases. (Cases are ordered according to initial FIM motor score.) Fig 6. Item threshold estimates for gait measures.

CHOOSING A DISCRIMINATIVE MEASURE, Brock 97 Table 3: Items Chosen for Comparative Analysis Measure FIM motor MAS Gait Item Stairs Bathing Tub transfer Walking Walking Sitting Velocity Endurance using Rasch analysis. Because the sample size of this study precluded including too many items within any 1 analysis, we considered only those items of the measures that had discriminated most effectively at higher levels of ability. We identified the items with high-item thresholds in the Rasch analysis for each measure and entered them into a combined Rasch analysis. The items included in the combined Rasch analysis are shown in table 3. Principal components analysis of the items revealed 2 factors with eigenvalues above 1.0. These factors had eigenvalues of 6.31 and 1.29, explaining 57.4% and 11.8% of the variance. Although the scale revealed 2 factors greater than 1, scree testing showed that the second factor represented the point at which a line drawn through the points changed direction. Subsequently, we continued with Rasch analysis because the scale closely approaches a unidimensional scale. Rasch analysis was performed on the combined gait and functional status measures at discharge. Two items were found to misfit the model, MAS sitting (1.44) and endurance (1.39). Two patients had perfect scores at discharge. Review of case estimates revealed 9 cases with infit t values beyond 2.0 or 2.0. The mean case estimate was 2.46 2.35. The item separation reliability was.90. Figure 7 shows the item threshold estimates for each of the items and the number of cases at the upper scoring levels of each scale. Gait velocity showed the highest level of difficulty, at 4.84 logits (see fig 7). The next most difficult item thresholds, at 3.46 and 3.33 logits, were the next level down for walking velocity and the highest level of the FIM stairs item. The FIM motor items of stairs and walking provided a good spread of item thresholds at higher levels of ability. Table 4 Fig 7. Item threshold estimates for the combined measures. For the higher levels of each item, the number of patients with scores at each level are provided, to facilitate comparing the discriminative abilities of items at the upper end of the spectrum of ability. Table 4: Distribution of Scores for FIM Motor Walking and Stairs FIM Walking FIM Stairs n* 6 7 7 29 7 22 6619 6 5 6 5 5 5 5 4 0 4or 4 4 or 4 9 * For simplification, this table does not include 11 patients who had more than a 2-point difference between the FIM walking and FIM stairs item. shows the distribution of scores for these 2 FIM motor items, showing the capacity of these items to discriminate within the sample. The FIM motor items of bathing and tub transfer also differentiated among cases at this level. The MAS walking item (highest level threshold at 2.90 logits) discriminated nearly as well as the FIM motor, but the other items of the MAS contributed little to the discriminative ability of the scale. The endurance variable s highest threshold level was 2.52 logits, well below that of other variables. DISCUSSION Examining the spread of scores for the measures, we found wide variation in the ceiling effects. Those for endurance and the FAC showed marked ceiling effects, with 39% and 46% of subjects achieving the highest scores. The ceiling effect of the FIM motor was much less pronounced, with only 16% achieving a perfect score. The MAS was shown to have a mild ceiling effect, with 25% achieving a perfect score at discharge. The gait velocity variable showed minimal ceiling effects with only 9% of patients achieving the highest level. In stroke rehabilitation, a small proportion of patients recover with very little residual disability. Arguably, a ceiling effect of 16% of cases, to use the FIM motor as an example, is an acceptable level because there may be little need to discriminate among patients making a near perfect recovery. It is also possible that the disability in these cases fell in other domains, such as cognition and/or communication. If that is so, then there is little point in attempting to increase the discriminatory power of a measure of physical disability. However, examination of ceiling effects alone is insufficient to show the suitability of a measure for evaluating higher level outcomes. A measure may have a low ceiling effect, with inclusion of a difficult item. The step size between the most difficult and the next level down the scale is equally important. If large gaps exist between the upper levels, the measure may not discriminate adequately among the population. Rasch analysis investigates the size of the steps. Rasch analysis showed that the FIM motor items represented varying levels of difficulty. Step size tended to increase with level of difficulty (see fig 1), and the step between 90 and 91 represented 11% of the scale. This value is similar to that provided by Fiedler, 55 in which the difference between 90 and 91 for summed raw scores represented 10% of the range of Rasch estimates. The relationship between the summed raw scores and Rasch estimates at discharge displayed a nonlinear relationship, as previously shown by Linacre et al. 15 The upper levels of the FIM motor require greater steps in level of ability to achieve the next level than are required at the lower levels. The correlation between summed raw scores and Rasch estimates for

98 CHOOSING A DISCRIMINATIVE MEASURE, Brock change was.71. Change in the lower part of the disability spectrum tends to be overstated by the summed raw scores and change in the upper part of the disability spectrum tends to be understated by the summed raw scores. Summed raw scores and Rasch estimates are not readily comparable for calculating change. To accurately represent level of ability or change in ability, Rasch estimates are preferable to summed raw scores. Rasch analysis of the MAS at discharge showed acceptable model fit. For this scale, the difference between the perfect score and the next level down occupied 17% of the range of Rasch case estimates. Summed raw scores and Rasch estimates were compared for MAS at discharge, yielding a correlation of.92. We found a nonlinear relationship for scores in the upper end of the scale. The use of Rasch case estimates, in preference to summed raw scores, is recommended for the MAS at discharge to evaluate accurately ability level. Comparing item thresholds for admission and discharge scores on the MAS revealed an ICC of.77. This correlation was well below the cutoff of.90 recommended by Chang and Chan. 53 This finding suggests that it is inadvisable to calculate change scores from the MAS because the difficulty level of the items did not show stability over time. This is a major problem for the use of this scale. Further development of the scale, identifying and rectifying those item levels that do not maintain a constant level of difficulty at admission and discharge, may improve the stability of the MAS. Rasch analysis of gait velocity, endurance, and the FAC at discharge showed acceptable model fit. The case estimates developed provide a composite score representing the patient s walking ability in terms of speed, endurance, level of assistance required, and ability to walk on stairs, inclines, and nonlevel surfaces (FAC). Comparing items from the FIM, MAS, and the gait measures by using Rasch analysis, we found that gait velocity had the strongest discriminative ability among patients with higher level abilities. However, this single item does not extend to lower levels of ability. The item is not appropriate for patients who cannot walk without assistance at discharge from rehabilitation. The next item to discriminate well among high-level cases was FIM stairs. This item requires patients to go up and down 12 to 14 stairs, without use of an aid or stair rail, safely and in a timely manner. That the FIM motor scale achieved the third highest level of difficulty argues against a ceiling effect for this scale. This finding adds weight to the contention by Linacre 15 that the ceiling effect of the FIM is apparent rather than real, and that use of Rasch converted scores clarifies the true magnitude of the raw score differences. The 4 FIM motor items of stairs, walking, bathing, and tub transfer discriminated well throughout the spectrum of ability at discharge. Although the MAS item of walking was a good discriminator, the other variables of the MAS were less useful. The threshold level of MAS walking was lower than the threshold level of the FIM stairs item. This is curious because both items require the patient to go up and down 12 stairs without a rail or a walking aid, with a specific time limit applied to the MAS. The main difference between the tests is that the MAS item is tested on a single occasion, whereas the FIM is rated on 24-hour performance, with safety as a consideration. Patients may successfully complete the MAS test under supervision, though their therapist may not rate them as safe on stairs without supervision. At the time of data collection in this study (1993 1995), most rehabilitation services in Australia were delivered in the inpatient setting. Long hospital LOS for the more severely disabled stroke patients were not uncommon. In recent years, other options for intensive rehabilitation services have been developed, such as same day rehabilitation (an intensive outpatient rehabilitation program) and home-based rehabilitation. Although the present data set is composed solely of inpatient data, it is arguable that outcome measurement should be performed at discharge from intensive rehabilitation, no matter how those services are delivered. CONCLUSION The FIM motor subscale proved to be the most suitable measure for evaluating mobility outcomes, as either level of ability at discharge, or as change in ability from admission to discharge. Comparison of measures did not support the need to add additional items to the FIM motor to prevent ceiling effects. Rasch analysis of the FIM motor showed that summed raw scores did not show the characteristics of interval measures. Small increments of change in raw scores at the higher levels of the scale, which may be interpreted as indicating little progress, actually denote far more substantial change. For an accurate indication of ability level, Rasch estimates should be used, particularly for ability level at discharge and for the calculation of change scores. The use of the conversion table of summed raw scores to Rasch estimates of ability calculated by Fiedler 55 provides a simple and valid means of accurately comparing levels of ability at the end of intensive rehabilitation. The present study supports the use of the FIM motor, with Rasch-transformed item difficulty and patient ability levels as a discriminative outcome measure of physical disability. The use of this outcome evaluation model would permit benchmarking of stroke rehabilitation services according to standards of outcome, as well as according to costs incurred in stroke rehabilitation. References 1. Hungate RW. Purchaser quality measures: progressing from wants to needs. Jt Comm J Qual Improv 1994;20:381-6. 2. American Education Research Association and National Council on Research in Education. Standards for educational and psychological tests. Washington (DC): American Psychological Association; 1985. 3. Center for Functional Assessment Research and the Uniform Data System for Medical Rehabilitation. Guide for use of the Uniform Data Set for Medical Rehabilitation including the Functional Independence Measure (FIM) version 3.1. Buffalo (NY): State Univ New York; 1990. 4. Dodds TA, Martin DP, Stolov WC, Deyo RA. A validation of the Functional Independence Measurement and its performance among rehabilitation inpatients. Arch Phys Med Rehabil 1993;74: 531-6. 5. Nissen MI. Emphasis placed by Cedar Court Physical Rehabilitation Hospital on the functional independence measure as the overall index for successful rehabilitation. Aust Clin Rev 1989;9: 36-8. 6. Pollock C, Freemantle N, Sheldon T, Song F. Methodological difficulties in rehabilitation research. Clin Rehabil 1993;7:63-72. 7. Tsuji T, Sonada S, Domen K, Saitoh E, Liu M, Chino N. ADL structure for stroke patients in Japan based on the Functional Independence Measure. Am J Phys Med Rehabil 1995;74:432-8. 8. Wade DT, Skilbeck CE, Hewer RL. Predicting Barthel ADL score at 6 months after an acute stroke. Arch Phys Med Rehabil 1983; 64:24-8. 9. Johnston MV, Findley TW, DeLuca J, Katz RT. Research in physical medicine and rehabilitation: XII. Measurement tools with application to brain injury. Am J Phys Med Rehabil 1991;70:40-56. 10. Merbitz C, Morris J, Grip JC. Ordinal scales and foundations of misinference. Arch Phys Med Rehabil 1989;70:308-12. 11. Silverstein B, Fisher WP, Kilgore KM, Harley JP, Harvey RF. Applying psychometric criteria to functional assessment in med-

CHOOSING A DISCRIMINATIVE MEASURE, Brock 99 ical rehabilitation: II. Defining interval measures. Arch Phys Med Rehabil 1992;73:507-18. 12. Wright BD, Linacre JM. Observations are always ordinal; measurements, however, must be interval. Arch Phys Med Rehabil 1989;70:857-60. 13. Fisher AG, Bryze KA, Granger CV, et al. Applications of conjoint measurement to the development of functional assessments. Int J Educ Res 1994;21:579-93. 14. Heinemann AW, Linacre JM, Wright BD, Hamilton BB, Granger C. Relationships between impairment and physical disability as measured by the Functional Independence Measure. Arch Phys Med Rehabil 1993;74:566-73. 15. Linacre JM, Heinemann AW, Wright BD, Granger CV, Hamilton BB. The structure and stability of the Functional Independence Measure. Arch Phys Med Rehabil 1994;75:127-32. 16. Wright BD, Linacre JM, Heinemann AW. Measuring functional status in rehabilitation. Phys Med Rehabil North Am 1993;4:475-91. 17. Carr JH, Shepherd RB, Nordholm L, Lynne D. Investigation of a new motor assessment scale for stroke patients. Phys Ther 1985; 65:175-9. 18. Holden MK, Gill KM, Magliozzi MR, Nathan J, Piehj-Baker L. Clinical gait assessment in the neurologically impaired: reliability and meaningfulness. Phys Ther 1986;64:1530-9. 19. Ottenbacher KJ, Hsu Y, Granger CV, Fiedler RC. The reliability of the Functional Independence Measure: a quantitative review. Arch Phys Med Rehabil 1996;77:1226-32. 20. Stineman MG, Jette A, Fiedler R, Granger C. Impairment-specific dimensions within the Functional Independence Measure. Arch Phys Med Rehabil 1997;78:636-43. 21. Stineman MG, Shea JA, Jette A, et al. The Functional Independence Measure: tests of scaling assumptions, structure, and reliability across 20 diverse impairment categories. Arch Phys Med Rehabil 1996;77:1101-8. 22. Granger CV, Cotter AC, Hamilton BB, Fiedler RC. Functional assessment scales: a study of persons after stroke. Arch Phys Med Rehabil 1993;74:133-8. 23. Oczkowski WJ, Barreca S. The Functional Independence Measure: its use to identify rehabilitation needs in stroke survivors. Arch Phys Med Rehabil 1993;74:1291-4. 24. Heinemann AW, Linacre JM, Wright BD, Hamilton BB, Granger C. Prediction of rehabilitation outcomes with disability measures. Arch Phys Med Rehabil 1994;75:133-43. 25. Stineman MG, Escarce JJ, Goin JE, Hamilton BB, Granger DV, Williams SV. A case-mix classification system for medical rehabilitation. Med Care 1994;32:366-79. 26. Stineman MG, Goin JE, Granger CV, Fiedler R, Williams SV. Discharge motor FIM function related groups. Arch Phys Med Rehabil 1997;78:980-5. 27. Loewen SC, Anderson BA. Reliability of the Modified Motor Assessment Scale and the Barthel Index. Phys Ther 1988;68: 1077-81. 28. Poole JL, Whitney SL. Motor assessment scale for stroke patients: concurrent validity and interrater reliability. Arch Phys Med Rehabil 1988;69:195-7. 29. Malouin F, Pichard L, Bonneau C, Durand A, Corriveau D. Evaluating motor recovery early after stroke: comparison of the Fugl-Meyer assessment and the Motor Assessment Scale. Arch Phys Med Rehabil 1994;75:1206-12. 30. Loewen SC, Anderson BA. Predictors of stroke outcome using objective measurement scales. Stroke 1990;21:78-81. 31. Holden MK, Gill KM, Magliozzi MR. Gait assessment for neurologically impaired patients. Phys Ther 1984;66:35-40. 32. Roth EJ, Merbitz CT, Grip JC, et al. The timer-logger-communicator gait monitor: recording temporal gait parameters using a portable computerized device. Int Disabil Stud 1990;12:10-6. 33. Mayo NE, Wood-Dauphinee SA, Gordon C, Higgins J, McEwen S, Salbach N. Disablement following stroke. Disabil Rehabil 1999;21:258-68. 34. Evans M, Goldie P, Hill K. Systematic and random error in repeated measurements of temporal and distance parameters of gait following stroke. Arch Phys Med Rehabil 1997;78:725-9. 35. Wade DT, Wood VA, Heller A, Maggs J, Hewer RL. Walking after stroke. Scand J Rehabil Med 1987;19:25-30. 36. Collen FM, Wade DT, Bradshaw CM. Mobility after stroke: reliability of measures of impairment and disability. Int Disabil Stud 1990;12:6-9. 37. Bohannon RW. Gait performance of hemiparetic stroke patients: selected variables. Arch Phys Med Rehabil 1987;68:777-81. 38. Brandstater ME, de Bruin H, Gowland C, Clark BM. Hemiplegic gait: analysis of temporal variables. Arch Phys Med Rehabil 1983;64:583-7. 39. Corcoran PJ, Jebsen RH, Brengelmann GL, Simons BC. Effects of plastic and metal leg braces on speed and energy cost of hemiparetic ambulation. Arch Phys Med Rehabil 1970;51:69-77. 40. Dettman MA, Linder MT, Sepic SB. Relationships among walking performance, postural stability, and functional assessments of the hemiplegic patient. Am J Phys Med Rehabil 1987;66:77-90. 41. Lehmann JF, Condon SM, Price R, delateur BJ. Gait abnormalities in hemiplegia: their correction by ankle foot orthoses. Arch Phys Med Rehabil 1987;68:763-71. 42. Friedman PJ. Gait recovery after hemiplegic stroke. Int Disabil Stud 1991;12:119-22. 43. Richards CL, Malouin F, Dumas F, Tardif D. Gait velocity as an outcome measure of locomotor recovery after stroke. In: Craik RL, Oatis CA, editors. Gait analysis: theory and application. St Louis: Mosby; 1995. p 355-64. 44. Goldie PA, Matyas TA, Evans OM. Deficit and change in gait velocity during rehabilitation after stroke. Arch Phys Med Rehabil 1996;77:1074-82. 45. Hill K, Ellis P, Bernhardt J, Maggs P, Hull S. Balance and mobility outcomes for stroke patients: a comprehensive audit. Aust J Physiother 1997;43:173-80. 46. Velozo CA, Kielhofner G, Lai J. The use of Rasch analysis to produce scale-free measurement of functional ability. Am J Occup Ther 1999;53:83-90. 47. Fisher WP, Harvey RF, Taylor P, Kilgore KM, Kelly CK. Rehabits: a common language of functional assessment. Arch Phys Med Rehabil 1995;76:113-22. 48. Grimby G, Andren E, Daving Y, Wright B. Dependence and perceived difficulty in daily activities in community living stroke survivors two years after stroke. Stroke 1998;29:1843-9. 49. Silverstein B, Kilgore KM, Fisher WP, Harley JP, Harvey RF. Applying psychometric criteria to functional assessment in medical rehabilitation: 1. Exploring unidimensionality. Arch Phys Med Rehabil 1991;72:631-7. 50. Weiss DJ, Yoes ME. Item response theory. In: Hambleton RK, Zaal JN, editors. Advances in educational and psychological testing: theory and applications. Boston: Kluwer Academic; 1991. p 69-95. 51. Wright BD, Masters GN. Rating scale analysis. Chicago: Mesa Pr; 1982. 52. Grimby G, Gudjonsson G, Rodhe M, Sunnerhagen KS, Sundh V, Ostensson ML. The Functional Independence Measure in Sweden: experience for outcome measurement in rehabilitation medicine. Scand J Rehabil Med 1996;28:52-62. 53. Chang W-C, Chan C. Rasch analysis for outcomes measures: some methodological considerations. Arch Phys Med Rehabil 1995;76:934-9. 54. McGinley J. Criteria for community ambulation: implications for stroke rehabilitation [dissertation]. Melbourne (Aust): La Trobe Univ; 1991. 55. Fiedler R. The Rasch measurement model. UDS Update 1993;7: 1-10. Suppliers a. SPSS Inc, 233 S Wacker Dr, 11th F1, Chicago, IL 60606. b. Adams R. Khoo S. QUEST: the interactive test analysis system. Melbourne (Aust): Australian Council for Educational Research; 1994.