LACK OF STANDARDIZATION of intervention and testing

Transcription

1 ORIGINAL ARTICLE A Method for Standardizing Procedures in Rehabilitation: Use in the Extremity Constraint Induced Therapy Evaluation Multisite Randomized Controlled Trial David M. Morris, PhD, PT, Edward Taub, PhD, David M. Macrina, PhD, Edwin W. Cook, PhD, Brian F. Geiger, EdD 663 ABSTRACT. Morris DM, Taub E, Macrina DM, Cook EW, Geiger BF. A method for standardizing procedures in rehabilitation: use in the Extremity Constraint Induced Therapy Evaluation multisite randomized controlled trial. Arch Phys Med Rehabil 2009;90: Objective: An enduring problem in the field of rehabilitation has been the lack of standardization in the protocols of treatments and tests. To develop a process evaluation method to standardize the administration of rehabilitation procedures used in the Extremity Constraint Induced Therapy Evaluation (EXCITE) Trial, a randomized controlled trial of upper-extremity constraint-induced therapy implemented across 7 sites. Design: Process evaluation. Setting: Research laboratory. Participants: Convenience sample or research personnel. Interventions: Not applicable. Main Outcome Measures: Checklist scoring sheets were developed to rate videotapes using systematic application of prescribed steps for each of 5 procedures across 3 time periods. Time periods were immediately after training, and 1 and 2 years later. A performance score of at least 90% was required before individual research personnel were allowed to participate in the trial. Results: Overall performance scores ranged from 85.8% to 95% of performance items correctly executed. There was a significant improvement in standard performance of procedures between the first time period (immediately after training) and each of the subsequent time periods for all but 1 procedure. The scoring of standardized performance when carried out with routine participant testing and training did not differ significantly from scoring from videotaped sessions submitted for standardization rating for 2 of the procedures, suggesting adequate validity of scoring from videotape. Conclusions: The present method was successful in assessing protocol fidelity for the EXCITE research personnel and represents 1 means of addressing the longstanding problem in rehabilitation of the lack of standardization in administering different treatments and tests. Key Words: Methods; Process assessment (Health Care); Rehabilitation; Research design by the American Congress of Rehabilitation Medicine LACK OF STANDARDIZATION of intervention and testing procedures has been one of the longstanding serious problems in rehabilitation, especially in research, but in clinical practice as well. 1 In both cases, there is often no clear delineation of what was done with the patient. Challenges to operationalizing rehabilitation interventions include the dynamic nature of interactions between clients and clinicians and opportunities for errors when administering complex rehabilitation protocols with multiple treatment components. In addition, clinicians and investigators often use a particular name to designate a procedure they carry out that may be widely at variance with what other professionals perform and understand by procedures characterized by that same name. This often makes the comparison of results among studies problematic. It is necessary to minimize threats to validity when conducting a multisite RCT. 2,3 By its nature, a successful multisite RCT requires systematic performance of all treatment and testing procedures across different sites. It is likely that clinicians will vary in their performance of procedures without careful attention to training and supervision of practice. However, for rehabilitation, the problem is much greater in scope than implementing a valid RCT and impacts almost all aspects of clinical practice. What, for example, is meant by therapeutic exercise or facilitation in precise terms? 1 The first 2 of the present authors were involved in planning and setting up the EXCITE trial of CI therapy, the first multisite upper extremity RCT of a rehabilitation therapy for stroke in the United States. 4 The training center for the trial conducted a 9-day workshop to train personnel at all locations in the standardized practice of all project procedures. 5-9 The purpose was to achieve uniformity of protocol delivery among the 7 From the Departments of Physical Therapy (Morris), Psychology (Taub, Cook), and Human Studies (Macrina, Geiger), and Center for Educational Accountability (Geiger), University of Alabama at Birmingham; and Research Service, Birmingham Veterans Affairs Medical Center (Taub), Birmingham, AL. Presented in part to the American Congress of Rehabilitation/American Society of Neurorehabilitation, September 8 11, 2004, Jacksonville, FL. Supported by National Institutes of Health (grant nos. RO1 HD and HD 34273). No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated. Reprint requests to David M. Morris, PhD, PT, Dept of Physical Therapy, University of Alabama at Birmingham, RMSB 360, rd Ave S, Birmingham, AL , morrisd@uab.edu /09/ $36.00/0 doi: /j.apmr AAUT ANOVA CI EXCITE MAL OT PT RCT WMFT List of Abbreviations Actual Amount of Use Test analysis of variance constraint induced Extremity Constraint Induced Therapy Evaluation Motor Activity Log occupational therapist physical therapist randomized controlled trial Wolf Motor Function Test

2 664 REHABILITATION PROCEDURES STANDARDIZATION, Morris sites (1 site was split between 2 different institutions). The method described herein was devised and implemented to maintain the fidelity of the procedures after the end of the workshop and in ensuing years. Process evaluation is defined as activities related to the identification and definition of the procedures executed in a project and the assessment of the conformity of research personnel to a specific protocol Use of process evaluation reduces the likelihood of a type III error in the research process that is, drawing conclusions about the outcomes from an intervention when, in fact, the program was not implemented according to its intended design. 13 Few articles have been published describing a systematic process evaluation monitoring adherence for neurorehabilitation interventions. The Post-Stroke Rehabilitation and Outcomes Study 14,15 was an 11-site study designed to determine the extent to which rehabilitation professionals followed stroke rehabilitation guidelines developed by the U.S. Agency for Healthcare Research and Quality, 16 making use of medical chart reviews. With this system, there was no direct observation of clinician behavior. Dobkin et al 17 described a personnel training and process evaluation procedure for an RCT of body weight supported treadmill training during inpatient rehabilitation after incomplete spinal cord injury. The training center provided a 1-week course for therapists from each of the 4 participating clinical sites, followed by onsite training by training center staff at each of the sites for at least 2 additional therapists and 2 therapist assistants. Intervention sessions were videotaped during a practice period and then assessed by the project s central training group using checklists. Project staff members were required to meet a competency score of 80% or higher on the checklists before they were allowed to enroll participants. New therapists were also required to complete training certification. It is not possible to tell from descriptions of this process the extent to which there was a strong subjective element in the ratings. The aims of this study were to determine (1) the level of standardized performance for selected testing and training procedures across 3 years of the EXCITE Project, and (2) the consistency of standardized performance among the EXCITE sites. METHODS Participants The standardization rating process was used with all testers and trainers participating in the EXCITE trial. Research personnel differed in terms of educational preparation and experience with rehabilitation examination and intervention delivery. Personnel included licensed PTs and OTs, and in some cases, nonlicensed participants who were supervised by licensed PTs and OTs. Funding limitations prohibited selection of only highly trained clinicians. This is the case in virtually any multisite study. This standardization of procedures method is an attempt to reduce the influence of this factor. Procedure The standardization process began with a 9-day training program held at the University of Alabama at Birmingham in August 2000 that was designed to familiarize all personnel with the EXCITE protocol. The 9-day workshop included didactic material of the research and conceptual basis of the CI therapy intervention, 5 detailed descriptions of each of the 5 main project testing and treatment procedures, and hands-on practice of procedures with patients. The 5 procedures whose standardized performance had to be maintained were the MAL, WMFT, AAUT, and 2 systematic motor training procedures: shaping and task practice without shaping. Personnel joining the project after the 9-day training program learned their assigned procedures through supervised practice with personnel at their sites who had attended the training program and through the use of the manual of procedures. A checklist had been devised for each procedure specifying each step to be performed in sequential order. Long checklists were developed for the 3 tests evaluated (MAL, WMFT, AAUT) to prompt training center personnel to focus attention on each item separately. Checklists for shaping and task practice were shorter and more subjective, because their components vary among patients performing them. When used in the EXCITE research protocol, shaping and task practice activities were selected from a large bank of tasks, and then tailored to be as good as possible for each patient s specific motor impairment. Clinicians exercised professional judgment about the level of intensity of shaping and task practice required during therapy; however, each checklist detailed specific steps that had to be performed to minimize unintended variation in procedure. Four of the aforementioned procedures were employed in the first CI therapy study, 5 while the fifth procedure, the AAUT test, was introduced later. 18 The tests and procedures scored for standard administration are described in detail elsewhere: the MAL, WMFT, and task practice, 6,8,19 and shaping. 20 Measures Videotapes were requested from all team members at each of the EXCITE sites across 3 time points: after initial training and during the second and third years of subject enrollment. Additionally, videotapes were required from new testers and trainers joining the project throughout the trial. Videotapes submitted for standardization rating included an entire testing session for each of 3 procedures performed by individual testers and 2 entire task practice and shaping activities executed by each trainer. The EXCITE trial was carried out in a single-blind fashion. Testing and treatment were carried out by different individuals except in the case of the AAUT test. This exception was necessary because of its nature as a method of measuring the spontaneous use of the more affected arm during the introductory segment of the first training period. All tapes were rated by the senior author. During each rating session, the rater observed the videotaped sessions and compared performance with a list of items on the administration rating sheet for the appropriate protocol procedure to determine whether each performance item was correctly executed. Observation of correct execution resulted in a check in the yes column and skipping or incorrectly performing a performance item resulted in a check in the no column of the scoring sheet. Figures 1 and 2 provide an example of a portion of the WMFT and shaping administration rating sheets, respectively. The WMFT, MAL, AAUT, shaping, and task practice administration rating forms included 619, 261, 72, 72, and 14 performance items for review, respectively. Total scores reflected the percent of performance items receiving a yes rating. All personnel were required to achieve a score equal to or greater than 90% of items correctly executed. Failure to meet this criterion required the tester or trainer to withdraw from the project and to resubmit standardization videotapes for rating until the 90% or higher criterion was achieved. This criterion score was selected by personnel at the training center. Perfect scores (ie, 100% correct execution) appeared to be an unreasonable expectation, and 90% seemed sufficiently stringent for relatively uniform execution of study procedures. Feedback on the fidelity of performance was pro-

3 REHABILITATION PROCEDURES STANDARDIZATION, Morris 665 Fig 1. Example of a section of the WMFT Administration Rating Sheet. vided to the site from which it originated, regardless of whether the desired score of 90% accurate performance was achieved. Data Analysis A repeated-measures ANOVA was used to examine mean performance scores achieved at all EXCITE sites for each time period and each of the training and testing procedures followed by a planned comparison analysis ( level.05). Mean performance scores on procedures from each of the research sites were also compared with scores at the University of Alabama at Birmingham research site using a Dunnett test. 21 RESULTS A total of 310 videotapes were reviewed and scored at the training center during the 3-year standardization period. The mean standardization scores for all videotaped procedures was 90%. Table 1 lists the scores for all periods combined for each site and for each procedure separately. All sites scored within an acceptable range after repeating therapeutic procedures with initial scores below the 90% accuracy rate. Combined standardization scores from all sites for the WMFT, MAL, AAUT, and shaping and task practice procedures were 87%, 91%, 86%, 95%, and 87%, respectively. These figures include initial scores before error feedback was provided and performance corrected. It was apparently more difficult to achieve the criterion for accuracy for some test items and treatment procedures than others. Considering items on individual tests, 9 of 18 WMFT tasks, 7 of 30 MAL tasks, and 5 of 10 AAUT tasks were performed correctly less than 90% of the time. In treatment procedures, 3 of 9 shaping components and 2 of 7 task practice components were performed correctly less than 90% of the time, although most of the item standardization scores were above 80%. Those cases in which scores were below 90% suggest items that should be emphasized in future personnel training efforts, discarded, or reformulated. Figure 3 presents the mean performance scores on initial videotaped submissions for all sites combined across the different testing and training sessions presented by time period. A repeated-measures ANOVA indicated that there was a difference in the level of standardized performance for each of the tests and training procedures among the 3 time frames except for task practice, which stayed the same. Planned comparisons using the Holm procedure 22 revealed that the difference was an improvement in standardized performance after the first time period for 4 of 5 procedures, possibly suggesting the value of the feedback procedure. There was also a significant difference in standardized performance for the MAL between time periods 2 and 3. Table 2 displays the percent of videotapes from all sites and by each procedure that did not meet the 90% criterion score. Those not meeting the criterion score were required to resubmit subsequent videotapes until the criterion score was met. Improvement in standardization over time was also demonstrated by the progressive decrease in the percent of initially submitted videotaped sessions scoring below the 90% criterion across time period 1 to time period 3 (see table 2). A significant difference in the mean standardized performance among the research sites was observed for each of the procedures. There were significant differences among sites achieving the criterion performance score. In only 2 cases, standardization scores of initial videotapes were below 80% (75.2% and 73.4%). DISCUSSION Standardization of procedures is a major unmet need for both research and clinical practice. There is a wide variation in the

4 666 REHABILITATION PROCEDURES STANDARDIZATION, Morris Fig 2. Example of a section of the Shaping Administration Rating Sheet. protocol of almost all treatments and tests identified by the same name. Differences in administrations may affect treatment outcomes. This is a serious impediment to documenting actual effectiveness and establishing the rehabilitation field on a sound evidential basis. The present study indicates that a standardization rating procedure after a training workshop where the target procedures are explained in detail and guided hands-on practice can be an effective way of achieving this aim. A process evaluation and correction procedure such as the one employed here is clearly critical for the conduct of an RCT The aim of this approach is to reduce variation in the execution of protocol procedures among research personnel and across research sites. In this study, we used a criterion of 90% adherence to 5 treatment and testing protocols and found that standardized performance remained stable within acceptable limits over a 3-year period. Although no models for monitoring intervention fidelity in rehabilitation RCTs have been published, reports Table 1: Mean Percent Standardization Score, SD, and Sample Size by Research Site for All Procedures for All Time Periods Combined Site WMFT MAL AAUT SH TP Mean Training center (7) (5) (4) (12) (12) (40) Site (6) (5) (9) (7) (6) (33) Site (6) (4) (5) (15) (15) (45) Site (8) (9) (11) (25) (29) (82) Site (14) (15) (4) (8) (9) (50) Site (10) (9) (14) (13) (14) (60) Mean (51) (47) (47) (80) (85) Abbreviation: SH, shaping; TP, task practice.

5 REHABILITATION PROCEDURES STANDARDIZATION, Morris 667 Fig 3. Mean standardization scores for different time points at all sites combined for each of the tests and treatment procedures. Abbreviation: TP, task practice. * Significantly different from time 1 (P<.05). Significantly different from times 1 and 2 (P<.05). from other fields of study (eg, public health, psychology) do exist Comparison of our fidelity scores (ie, approximately 90% correct performance) to those of other published process evaluation of complex interventions suggests that fidelity for our study was good. For example, in the studies cited, the range of mean fidelity was 60% to 87%. More generally, procedures such as those employed here should be of value for future rehabilitation research and clinical efforts in standardizing clinical protocols across different institutions and among therapists at the same facility. An important principle in the construction of a standardization rating instrument is to develop a checklist made up of objective performance items presented in sequence. This strategy would reduce reliance on the subjective evaluation of raters and would more specifically identify performance strengths and areas in need of remediation. This is preferred to more generalized expert opinion about the adequacy of the performance of personnel. In the present study, overall rating scores of 3 of the 5 standardized protocols across sites and time periods were below the minimally acceptable 90%. However, the rating scores for these 3 procedures were between 85.8% and 90%. Investigators at the training center provided feedback about individual performances below the 90% correct criterion to clinicians, who were required to submit videotaped procedures after additional practice and before continuing work with project activities. Considering final performance rather than initial videotaped submissions, all tests were above the 90% criterion. The standardization rating process was also helpful to identify performance items that were especially prone to error and required emphasis by way of further training, elimination, or reformulation. 12 When examined by time period, results revealed that greater variation in performance of protocol procedures occurred during the start-up phase of the project. These findings suggest that the 9-day training program and extensive written instructions for each procedure were insufficient to assure optimal standardization of the protocol. Because all personnel at sites other than the University of Alabama at Birmingham were previously unfamiliar with most of the procedures, it is not surprising that the first timeframe would be most vulnerable to least good standardization. Moreover, competing duties associated with establishing a new research site (eg, purchasing equipment, recruiting participants for the first time, setting up the laboratory) may have distracted personnel from learning the procedures adequately. However, the differences were not excessively large, and the difference between standardization scores for the first time period and the best subsequent score did not exceed 10% in any single case (see fig 3). In addition, standardized performance improved over time. This suggests that the standardization rating process and the feedback provided to those rated influenced standardization across sites, and more generally supports the value of the standardization rating process as part of the training procedure for RCTs. Without these quality assurance efforts, continued deviation from the procedure protocols could have increased and seriously challenged the entire EXCITE Trial effort. Data gathered from the standardization rating procedure could also be helpful in interpreting results from a given site in an RCT. A poor administration rating score for any particular procedure from a site could raise questions about the validity of that data. Study Limitations Because of financial constraints and a number of other practical considerations, such as keeping the EXCITE Trial on schedule, a number of procedural additions to the rating standardization process that were considered desirable were not carried out. These included submission and scoring of performance videotapes more often than once a year and more frequent onsite visits, especially surprise visits, to determine the validity of the scoring of performance videotapes and thus enhance process evaluation procedures. The consideration that more frequent on-site visits should be made was prompted by the possibility that trial personnel might select the best of several tapes for submission and thereby introduce a selection bias in the scoring process. Although decreasing the time interval between performance videotape submissions might have improved the scores further, the degree of standardized performance was adequate and above the 90% minimum decided prior to trial inception. The standardization ratings, as conducted in this trial, were very time-intensive for 1 person. Recent work in our laboratory demonstrated acceptable interrater reliability (r.70) for the standardization process used in this project. Therefore, a team of raters could be assembled to reduce this burden after interrater reliability was established. CONCLUSIONS Deviations from an intended research protocol can result in erroneous conclusions from data gathered, especially in a multisite RCT. Process evaluation procedures such as the standardization scoring method described here address this issue. Thus, the results from this study can be important for (1) the interpretation of the results of a multisite RCT, (2) the design of future multisite RCTs in rehabilitation research, and (3) per- Table 2: Percent of Videotaped Sessions Scoring Below 90% by Procedure and Time Period Period Procedure WMFT (%) MAL (%) AAUT (%) Shaping (%) TP (%) Abbreviation: TP, task practice.

6 668 REHABILITATION PROCEDURES STANDARDIZATION, Morris haps most importantly, the standardized duplication of performance of rehabilitation procedures in both research and clinical practice. Acknowledgments: We thank the participating EXCITE sites of Emory University, Atlanta, GA; University of Alabama at Birmingham, Birmingham, AL; University of Southern California, Los Angeles, CA; University of North Carolina at Chapel Hill, Chapel Hill, NC; Wake Forest University, Winston Salem, NC; Ohio State University, Columbus, OH; and University of Florida Gainesville, Gainesville, FL. We also thank Washington University, St. Louis, MO, which served as the Data Management Center. References 1. Whyte J, Hart T. It s more than a black box; it s a Russian doll: defining rehabilitation treatments. Am J Phys Med Rehabil 2003; 82: Fuhrer M. Overview of clinical trials in medical rehabilitation: impetuses, challenges, and needed future directions. Am J Phys Med Rehabil 2003;82:S Whyte J. Clinical trials in rehabilitation: what are the obstacles? Am J Phys Med Rehabil 2003;82:S Wolf SL, Winstein CJ, Miller JP, et al. Effect of constraintinduced movement therapy on upper extremity function 3 to 9 months after stroke: the EXCITE randomized clinical trial. JAMA 2006;296: Taub E, Miller NE, Novack TA, et al. Technique to improve chronic motor deficit after stroke. Arch Phys Med Rehabil 1993; 74: Taub E, Uswatte G, King DK, Morris DM, Crago JE, Chaterjee A. A placebo controlled trial of constraint induced movement therapy for upper extremity after stroke. Stroke 2006;37: Taub E. Harnessing brain plasticity through behavioral techniques to produce new treatments in neurorehabilitation. Am Psychol 2004;59: Morris DM, Crago JE, DeLuca SC, Pidikiti RD, Taub E. Constraint-induced movement therapy for motor recovery after stroke. NeuroRehabilitation 1997;9: Morris DM, Taub E, Mark VW. Constraint-induced movement therapy: characterizing the intervention protocol. Eur Medicophys 2006;42: Rossi P, Freeman H. Evaluation: a systematic approach. 5th ed. Newbury Park: Sage; Windsor R, Baranowski T, Clark N, Cutter G. Evaluation of health education and disease prevention programs. Mountain View: Mayfield; Linnan L, Steckler A. Process evaluation for public health interventions and research: and overview. In: Steckler A, Linnan L, editors. Process evaluation for public health interventions and research. San Francisco: Jossey-Bass; Scanlon J, Horst P, Nay J. Evaluability assessment: avoiding type III and IV errors. In: Gilbert G, Conklin P, editors. Evaluation management: a source book of readings. Charlottesville: US Civil Service Commission; LeClair BJ, Reker DM, Duncan PW, Horner RD, Hoenig H. Stroke care: a method for measuring compliance with AHCPR guidelines. Am J Phys Med Rehabil 2001;80: Duncan PW, Horner RD, Reker DM, et al. Adherence to postacute rehabilitation guidelines is associated with functional recovery in stroke. Stroke 2004;33: Gresham GE, Duncan PW, Stason WB, et al. Post-stroke rehabilitation: assessment, referral and patient management: clinical practice guideline. Rockville: Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research; Dobkin BH, Apple D, Barbeau H, et al. Methods for a randomized trial of weight-supported treadmill training versus conventional training for walking during inpatient rehabilitation after traumatic spinal cord injury. Neurorehabil Neural Repair 2003;17: Taub E, Uswatte G, Pidikiti R. Constraint-induced movement therapy: a new family of techniques with broad application to physical rehabilitation a clinical review. J Rehabil Res Dev 1999;36: Taub E, Uswatte G, Mark VW, Morris DM. The learned nonuse phenomenon: implications for rehabilitation. Eur Medicophys 2006;42: Taub E, Burgio L, Miller NE, et al. An operant approach to overcoming learned nonuse after CNS damage in monkeys and man: the role of shaping. J Exp Anal Behav 1994;61: Dunnett CW. A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 50;1955: Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6: Helitzer D, Yoon SJ, Wallerstein N, Dow y Garcia-Velarde L. The role of process evaluation in the training of facilitators for an adolescent health education program. J School Health 2000;70: Rosecrans AM, Gittelsohn LS, Ho SB, Harris M. Process evaluation of a multi-institutional community-based program for diabetes prevention among first nations. Health Educ Res 2008;23: Sanchez V, Steckler A, Nitirat P, Hllfors D, Cho H, Brodish P. Fidelity of implementation in a treatment effectiveness trial of reconnecting youth. Health Educ Res 2007;22: