STROKE IS CURRENTLY among the major causes of

Transcription

1 977 Establishing the Reliability of Mobility Milestones as an Outcome Measure for Stroke Gillian D. Baer, MSc, MCSP, Mark T. Smith, BSc, GradDipPhys, MCSP, Philip J. Rowe, PhD, Lindsay Masterton, MCSP ABSTRACT. Baer GD, Smith MT, Rowe PJ, Masterton L. Establishing the reliability of mobility milestones as an outcome measure for stroke. Arch Phys Med Rehabil 2003;84: Objective: To establish intrarater, interrater, and test-retest reliability of a standardized measure of mobility, mobility milestones, incorporating sitting, standing, and walking ability. Design: Repeated-measures reliability study by using video data of patients with stroke. Setting: Physiotherapy and rehabilitation departments in Scotland. Participants: Forty physiotherapists recruited from within the Lothian region: 20 senior physiotherapists with at least 3 years of experience working with neurologic patients and 20 staff grade physiotherapists with less than 12 months of experience working with neurologic patients. Intervention: Videotape comprising 40 clips (36 original clips, 4 repeated clips) of stroke patients of differing levels of ability attempting the mobility milestones was produced. After a short training session in the interpretation and application of the mobility milestones, each physiotherapist viewed the tape separately and scored whether the milestone had been achieved or not. This was repeated at a separate test session 2 weeks later. Main Outcome Measure: Score for each mobility milestone. Results: Kappa statistics were used to determine interrater reliability and showed good (.61.80) to very good ( ) reliability for 3 of 4 milestones. Intraclass correlation coefficients (ICCs) were used to determine intrarater reliability of the 4 repeated clips and showed 75% of all subjects had high (ICC 2, ) reliability. The ICC 2,1 for test-retest reliability showed a similar pattern, with 70% of subjects showing good (.81.90) or high ( ) reliability. Conclusions: The mobility milestones showed favorable levels of reliability when used by experienced or novice physiotherapists. The milestones can be adopted as a simple clinical outcome measure for use with stroke. Further research is required to establish reliability levels when the measure is used by different rehabilitation professionals. Key Words: Cerebrovascular accident; Equilibrium; Outcome assessment (health care); Rehabilitation; Reproducibility of results. From the Department of Physiotherapy, Queen Margaret University College, Edinburgh (Baer, Rowe); Department of Physiotherapy, Royal Victoria Hospital, Edinburgh (Smith); and Department of Physiotherapy, West Lothian Community Rehabilitation Service, West Lothian (Masterton), Scotland, UK. Supported by the Chief Scientist Office of the Scottish Executive (grant no. K/OPR/15/11/F8). No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the authors(s) or upon any organization with which the author(s) is/are associated. Reprint requests to Gillian Baer, MSc, MCSP, Dept of Physiotherapy, Queen Margaret University College, Leith Campus, Duke St, Edinburgh EH6 8HF, Scotland, UK, gbaer@qmuc.ac.uk /03/ $30.00/0 doi: /s (03) by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation STROKE IS CURRENTLY among the major causes of disability and handicap in the Western world 1 and is the most common cause of disability in adults in Scotland. 2 General stroke incidence is reported at around 1 per 5000 persons. Recovery of functional ability after stroke is variable, with between 30% and 60% of people reported as having a degree of dependence in some activities of daily living. 3 It has been established that stroke survivors do not form an homogeneous entity. Discrete subclassifications may have differing potential for recovery and reach optimum levels of recovery at different times. 4-6 Many outcome measures exist for the routine assessment of stroke, including the Barthel Index 7 or the FIM instrument. 8 These outcome measures can be used to show gross changes over time in patients overall performance. In stroke rehabilitation, however, many of these existing measures can be criticized for their lack of sensitivity to functional outcome and their inability to show improvements that may be expected after successful rehabilitation. With an increasing need to show clinical effectiveness through the delivery of evidence-based practice, more specific outcome measures are being developed to investigate recovery from stroke and to establish the effectiveness of intervention. There is recent evidence that stroke care within a designated stroke unit is the optimal management approach 9 resulting in a reduced length of stay, 10 reduced dependency, 9,11,12 and earlier restoration of some aspects of physical function. 13 A simple outcome measure that is sensitive to physical recovery profiles in stroke rehabilitation may therefore provide an appropriate tool in the therapeutic setting for evaluating and optimizing physical outcome after stroke. It is essential that any new outcome measure undergo extensive testing to establish its particular strengths and weaknesses. Essential characteristics of any outcome measure include reliability, validity, sensitivity, and communicability. 14,15 If outcome measures are to be adopted widely, they must also be easy to use in a clinical environment. 16,17 The mobility milestones for stroke were first published in and are summarized in table 1. They measure days to achieve 4 simple functional mobility tasks. The intention was to develop a tool that could be adopted as an integral part of routine clinical practice and one that would provide a quick, simple, and standardized outcome measure of mobility in stroke in a way that would be meaningful to the rehabilitation team. 6 The mobility milestones were developed partly pragmatically from observational clinical experiences. These experiences indicated that certain achievements during recovery from stroke seemed integral to acquiring a degree of functional independence. For example, the ability to sit is crucial for safety, because upperbody dressing is difficult if the sound arm is required to maintain sitting. The ability to take 10 steps may indicate a progression toward independent walking and allow planning of a home discharge assessment. The mobility mile-

2 978 MOBILITY MILESTONES FOR STROKE, Baer Table 1: Standardized Protocols for Mobility Milestones 6 1-Minute Sitting Balance (a period of unsupported sitting on a bed or plinth without a backrest for longer than 1min) Hips, knees, and ankles should be positioned at 90, with both feet flat on the floor. The upper limbs should rest passively. 10-Second Standing Balance (a period of unsupported standing for longer than 10s) The weight should be evenly distributed between both feet in coronal and sagittal planes. Physical help is permissible in making the transition from sitting to standing, but during the timing period no help should be given. 10 Independent Steps (independent walking of 10 steps or more [5 left, 5 right]) This may conclude a period of supported walking or evolve from a 10-second stand, but during the milestone no physical help is allowed. Verbal cueing is permissible and a walking aid may be used where appropriate. 10-Meter Walk (a walk over a measured distance of 10m) Subject commences walking from standing and is instructed to walk to a point distant to the end of the walkway. The walk starts at the beginning of the first step and finishes as the subject crosses the mark indicating the end of the 10m. The assessor should walk beside the patient on the affected side. Verbal instruction should be limited to I should like you to walk to the end of this room at a speed which is comfortable for you. If appropriate, a walking aid may be used but verbal cueing should be avoided. stones were developed because a need existed to identify more subtle functional changes indicative of progress in rehabilitation yet not detectable by a standardized measure such as the Barthel Index. The aim of the present study was to establish the interrater, intrarater, and test-retest reliability of the mobility milestones when used by physiotherapists with varying levels of experience in neurologic rehabilitation. We defined the different types of reliability as follows: (1) intrarater reliability: the consistency in scoring by an individual rater during a single test session, (2) interrater reliability: the consistency in scoring between a group of raters during a single test session, and (3) test-retest reliability: the consistency in scoring by an individual rater over 2 separate testing sessions. METHODS were required to view videotape data of stroke subjects attempting to achieve the mobility milestones and to rate either yes (mobility milestone achieved) or no (mobility milestone not achieved). The study received ethics approval from the Lothian Research Ethics Committee and all subjects (both stroke patients and physiotherapists) were required to sign informed consent forms. Videotape Data of Stroke Subjects Nineteen stroke subjects at various stages of recovery were asked to attempt the mobility milestones, and a video recording was made of each attempt. The video recordings of passed and failed milestone attempts were edited into a master tape of 40 milestone clips. The tape consisted of 9 original clips of each milestone, plus a single repeated clip for each milestone, making a total of 10 clips of 1-minute sitting, 10 clips of 10-second standing, 10 clips of 10 steps, and 10 clips of a 10-m walk. The repeated clips were included to establish intrarater reliability on 1 occasion for each milestone. The master tape was compiled to present the 40 clips of mobility milestones in random order. The final tape took approximately 45 minutes to view. Participants Forty physiotherapists from the Lothian region were recruited to view the video. Twenty senior physiotherapists and 20 staff grade physiotherapists were recruited. Inclusion criteria stated that the senior physiotherapists must have worked a minimum of 3 years in neurology and the staff grade physiotherapists must have worked less than a year in neurology. Each physiotherapist gave informed consent and rated the videoclips on 2 occasions separated by 2 weeks. Rater Training A standardized training session, lasting approximately 30 minutes, in the use of the mobility milestones was conducted. The training consisted of issuing standardized descriptions of the mobility milestones (see table 1) and a stopwatch. An explanation was given of how and why the mobility milestones were developed and how the descriptors were used, allowing the subjects to read the descriptors and ask any questions. This introduction was followed by a viewing of a training video. The video consisted of a further 21 assorted videoclips of patients attempting the mobility milestones. The training video did not show data from subjects included in the main video. After training, the physiotherapists were again allowed to ask the researcher for clarification of any issues. Data Collection After the training session, the physiotherapists undertook their first data viewing session. Each physiotherapist viewed the master video independently in 1 session and scored each videoclip. Standardized equipment was used. It consisted of a television monitor with 21-in screen and videocassette recorder with slow-motion playback and freeze-frame facility. A standardized score sheet was provided for the physiotherapists to mark whether the patients had achieved the milestone. A Cronus memory 100 stopwatch a was issued to enable time for the sitting and standing milestones to be verified. Subjects were free to stop the tape between each videoclip, and to rewind the tape if they were unsure of patient performance on any milestone. This process of data collection was repeated at a subsequent session 2 weeks later. Data Analysis The data were analyzed by using Microsoft Excel 97. b Initial analysis allowed calculation of percentage agreement to provide an overall impression of whether the physiotherapists were in agreement with themselves and also with the judgment of the originators of the milestones. Percentage agreement may give an elevated impression of levels of agreement due to the possibility of 50% agreement by chance. Interrater reliability was therefore calculated by using the Cohen statistic because this statistic indicates the amount of agreement beyond that expected by chance. 18 An intraclass correlation coefficient (ICC) for intrarater reliability (for the repeated clips during a

3 MOBILITY MILESTONES FOR STROKE, Baer 979 Table 2: Percentage Agreement Between and Milestone Originators That Mobility Milestones Were Achieved Mobility Milestone 1-Minute Sitting Balance (%) single-test session) and test-retest reliability (all videoclips viewed 2wk apart) was calculated. 19 Because all videoclips were rated by all the physiotherapists, and because we wanted to generalize the findings to a wider population, we used the ICC 2,1 form, which permits separation of variance attributable to an individual. 20 RESULTS Participant Characteristics Forty physiotherapists (20 senior, 20 staff grade) were recruited for the study. The 20 senior physiotherapists had a mean standard deviation of years of experience in neurology, and the 20 staff grade physiotherapists had a mean experience of months working in neurology. All 40 physiotherapists evaluated all 40 videoclips on 2 occasions; therefore, data from 1600 videoclips were available on both occasions. The mean time taken between test 1 and the retest was 14 days for the senior physiotherapists and 15.3 days for the staff physiotherapists. Levels of Agreement The percentage of all 40 physiotherapists agreeing with the 2 originators of the milestones 6 that a mobility milestone had been achieved (or not achieved) was calculated. These data are presented in table 2. For each clip, a large majority of the physiotherapists were in agreement. Interrater Reliability Kappa statistics for interrater reliability (consistency in scoring between a group of raters during a single-test session) were calculated for all 4 mobility milestones together and for each Table 3: Kappa Statistics for Interrater Levels of Agreement All 10-Second Standing Balance (%) Senior 10 Steps (%) Staff Test 1 All milestones min sitting s standing steps m walk Test 2 All milestones min sitting s standing steps m walk Meter Walk (%) Test Test Table 4: Distribution of ICC 2,1 Values for Intrarater Reliability ICC Value %ofall % of Senior % of Staff individually for both test sessions. Kappa statistics were also calculated for the 20 senior physiotherapists and the 20 staff physiotherapists. The results are given in table 3. Intrarater Reliability Intrarater reliability (consistency in scoring during a single test session) was calculated using the data from the 8 repeated videoclips (4 clips on 2 occasions). An ICC 2,1 was calculated from the 8 videoclips for each physiotherapist, and the percentage distribution was subsequently determined. The findings are shown in table 4. Test-Retest Reliability Test-retest reliability (consistency in scoring over 2 separate testing sessions) was calculated from the data from the 36 single videoclips scored on 2 occasions, 2 weeks apart. An ICC 2,1 was used for each physiotherapist and the percentage distribution determined. The findings are shown in table 5. DISCUSSION The percentage agreement on the mobility milestones between the physiotherapists ratings and our ratings was very high overall. This agreement indicates that a high level of consistency was achieved when assessing all 4 mobility milestones. Further, this level of consistency was similar on both occasions. With the statistics used to establish interrater reliability beyond that expected purely by chance, interrater reliability of physiotherapists assessing all the mobility milestones could be classified as good (.61 80) or very good ( ) with the exception of the senior physiotherapists interpretation of 10 steps during test 2 and a consistent assessment of the 10-second standing milestone, which was moderate (.41.60) or fair (.21.40). 18 A similar profile of reliability was shown by both the senior and the staff grade physiotherapists. Interestingly, the staff physiotherapists showed slightly higher levels of reliability than the senior physiotherapists in assessing the milestones overall and in more than half of the individual milestones. The reason for this is unclear, but it may be that after the training, the staff physiotherapists were more accepting of the descriptors than the senior physiotherapists, Table 5: Distribution of ICC 2,1 Values for Test-Retest Reliability ICC Value %ofall % of Senior % of Staff Grade

4 980 MOBILITY MILESTONES FOR STROKE, Baer some of whom expressed some dissatisfaction in that they wanted the standardized descriptor to include other characteristics. Another interesting point is that the 10-second standing milestone received only moderate or fair levels of reliability. Informal feedback and comments included on the scoring sheet from the physiotherapy subjects indicated that it was hard to determine whether even distribution of weight was achieved from video data, hence, there was some uncertainty in assigning a score. To establish intrarater and test-retest reliability, the convention suggested by Youdas et al 21 was followed. This convention assigns values between.91 to 1.0 as high,.81 to.90 as good,.71 to.80 as fair, and values less than.70 as poor. By using these labels, three quarters of the physiotherapists showed high intrarater reliability, with only 3 subjects showing poor intrarater reliability. Similarly for test-retest reliability, 28 of the sample showed good or high test-retest reliability, with only 3 subjects showing poor reliability. The results would seem to indicate acceptably high levels of interrater, intrarater, and test-retest reliability at all levels within this sample of physiotherapists. But how generalizable should the results be considered? The physiotherapists showed a wide range of expertise from no neurologic experience to 25 years working in neurologic rehabilitation. Further, they represented a large sample size in terms of reliability studies It is notoriously difficult to calculate power statistics for reliability studies, and, to date, we could find no published reliability reports that cited power statistics. It has been suggested that, for data analyzed by using ICCs, for a power greater than.95 with a probability of.05, approximately 20 subjects (videoclips) and 2 raters (physiotherapists) would be required. 26,27 It is also proposed that these numbers would need to be increased to around 30 subjects and 6 raters when using the statistic. 26,27 Thus, the sample size and methodology used in the present study would appear to have sufficient power to be generalizable to a wider population of physiotherapists. Some caution should be shown in relation to the intrarater reliability results, however, because only a limited number of videoclips were available. The reliability of the mobility milestones perform comparably with other studies investigating the reliability of existing functional outcome measures using various methods of administration. The Frenchay Activities Index, administered by interview, has fair to moderate agreement ( range,.26.52) for 6 of the 15 items, with good agreement ( range,.64.80) for the remaining 9 items. 28 Van der Lee et al 29 found median weighted kappas greater than.92 when establishing the reliability of the Action Research Arm Test by repeated viewing of video data. Agreement between 2 raters assessing patients on the same day has shown variable results. The Barthel Index showed fair to moderate agreement ( range,.27.58) in 8 of 10 items, with the remaining 2 items showing good agreement ( range,.63.68) when used by a nurse and a nonclinical researcher. 30 Rehabilitation professionals found good to high agreement in two thirds of the FIM FAM items ( range,.67.95), with only 1 of the remaining items showing a low agreement of less than We believe that the mobility milestones have the advantage over many other measures of physical outcome in that they are quick and simple to use (subjects can be tested in 5min), are reliable, have clinical relevance, and can be communicated in a way that informs ongoing multidisciplinary stroke management. We have also discussed 6 the use of mobility milestones as predictors of goal achievement. Although the present study has established reliability by using video data, it is unlikely that data collected in the clinical situation would be vastly different; in fact, it may show greater levels of reliability. Having established reliability, we believe that further research could be conducted by using the mobility milestones as a measure to quantify the effects of different rehabilitation interventions on functional outcome. CONCLUSION The mobility milestones showed acceptable levels of interrater, intrarater, and test-retest reliability. These levels of reliability were shown by both senior physiotherapists with substantial experience treating patients with neurologic disabilities and by staff grade physiotherapists with little experience in neurologic rehabilitation. The findings offer encouraging support for the argument to adopt simple outcome measures such as the mobility milestones for routine clinical use. Future work in relation to the mobility milestones will be needed to establish levels of reliability for other health care professionals drawn from an international sample. References 1. Khaw KT. Epidemiology of stroke. J Neurol Neurosurg Psychiatry 1996;61: Scottish Intercollegiate Guidelines Network (SIGN). Management of patients with stroke: IV rehabilitation, prevention management of complications and discharge planning. Edinburgh: SIGN; Guideline No Duncan PW, Goldstein LB, Matchar D, Divine GW, Feussner J. Measurement of motor recovery after stroke. Outcome assessment and sample size requirements. Stroke 1992;23: Reding MJ, Potes E. Rehabilitation outcome following initial unilateral hemispheric stroke: life table analysis approach. Stroke 1988;19: Bamford J, Sandercock P, Dennis M, Burn J, Warlow C. Classification and natural history of clinically identifiable subtypes of cerebral infarction. Lancet 1991;337: Smith MT, Baer GD. Achievement of simple mobility milestones following stroke. Arch Phys Med Rehabil 1999;80: Mahoney FI, Barthel DW. Functional evaluation: the Barthel Index. Md State Med J 1965;14: Keith RA, Granger C, Hamilton BB, Sherwin FS. The functional independence measure: a new tool for rehabilitation. Adv Clin Rehabil 1987;1: Organised inpatient (stroke unit) care for stroke. Stroke Unit Trialists Collaboration. Cochrane Database Syst Rev 2000;(2): CD Kalra L. The influence of stroke unit rehabilitation on functional recovery from stroke. Stroke 1994;25: Indredavik B, Slordahl SA, Bakke F, Rokseth R, Haheim LL. Stroke unit treatment. Long-term effects. Stroke 1997;28: Indredavik B, Bakke F, Slordahl SA, Rokseth R, Haheim LL. Stroke unit treatment improves long-term quality of life: a randomized controlled trial. Stroke 1998;29: Logan PA, Ahern J, Gladman JR, Lincoln NB. A randomized trial of enhanced Social Services occupational therapy for stroke services. Clin Rehabil 1997;11: Cole B, Finch E, Gowland C, Mayo N. Physical rehabilitation outcome measures. Toronto: Canadian Physiotherapy Association; Wade DT. Measurement in neurological rehabilitation. Oxford: Oxford Univ Pr; Greenhalgh J, Long AF, Brettle AJ, Grant MJ. Reviewing and selecting outcome measures for use in routine practice. J Eval Clin Pract 1998;4: Kendall N. Developing outcome assessments: a step by step approach. N Z J Physiother 1997;Dec: Brennan P, Silman A. Statistical methods for assessing observer variability in clinical measures. BMJ 1992;304: Shrout P, Fleiss J. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420-8.

5 MOBILITY MILESTONES FOR STROKE, Baer Rankin G, Stokes M. Reliability of assessment tools in rehabilitation: an illustration of appropriate statistical analyses. Clin Rehabil 1998;12: Youdas JW, Carey JR, Garrett TR. Reliability of measurements of cervical spine range of motion comparison of 3 methods. Phys Ther 1991;71: Hughes K, Bell F. Visual assessment of hemiplegic gait following stroke. Arch Phys Med Rehabil 1994;75: Daley K, Mayo N, Wood-Dauphinee S. Reliability of scores on the Stroke Rehabilitation Assessment of Movement (STREAM) measure. Phys Ther 1999;79:8-19; quiz Kopp B, Kunkel A, Flor H, et al. The Arm Motor Ability Test: reliability, validity and sensitivity to change of an instrument. Arch Phys Med Rehabil 1997;78: Lord S, Halligan P, Wade D. Visual gait analysis: the development of a clinical assessment and scale. Clin Rehabil 1998;12: Donner A, Eliasziw M. Sample size requirements for reliability studies. Stat Med 1987;6: Cohen J. Statistical power analysis for the behavioral sciences. New York: Academic Pr; Piercy M, Carter J, Mant J, Wade DT. Inter-rater reliability of the Frenchay Activities Index in patients with stroke. Clin Rehabil 2000;14: van der Lee JH, de Groot V, Beckerman H, Wagenaar RC, Lankhorst GJ, Bouter LM. The intra- and interrater reliability of the action research arm test: a practical test of upper extremity function in patients with stroke. Arch Phys Med Rehabil 2001; 82: Richards SH, Peters TJ, Coast J, Gunnell DJ, Darlow M, Pounsford J. Inter-rater reliability of the Barthel ADL Index: how does a researcher compare to a nurse. Clin Rehabil 2000;14: McPherson KM, Pentland B, Cudmore SF, Prescott RJ. An interrater reliability study of the Functional Assessment Measure (FIM FAM). Disabil Rehabil 1996;18: Suppliers a. Cronus Precision Products, 2895 Northwestern Pkwy, Santa Clara, CA b. Microsoft Corp, One Microsoft Way, Redmond, WA