THE EFFECTIVENESS of rehabilitation services is best

Transcription

1 649 Short-Form Activity Measure for Post-Acute Care Stephen M. Haley, PhD, PT, Patricia L. Andres, MS, PT, Wendy J. Coster, PhD, OTR, Mark Kosinski, MA, Pengsheng Ni, MD, MPH, Alan M. Jette, PhD, MPH, PT ABSTRACT. Haley SM, Andres PL, Coster WJ, Kosinski M, Ni P, Jette AM. Short-form Activity Measure for Post-Acute Care. Arch Phys Med Rehabil 2004;85: Objective: To develop a comprehensive set of short forms using item response theory (IRT) and item pooling procedures for the purpose of monitoring postacute care functional recovery. Design: Prospective study. Setting: Six postacute health care networks in the greater Boston area, including inpatient acute rehabilitation, transitional care units, home care, and outpatient services. Participants: A convenience sample of 485 adult volunteers who were currently receiving skilled rehabilitation services. Interventions: Not applicable. Main Outcome Measures: We developed a set of 6 short forms across 3 activity domains from new items and items from existing postacute care instruments. Results: Inpatient- and community-based short forms were developed for each of 3 activity domains: physical & movement, applied cognition, and personal care & instrumental. s were selected for inclusion on the short forms to maximize content coverage and information value of items across the range of content and to minimize ceiling and floor effects. We were able to match the distribution of sample scores with very good item precision for 1 of the constructs (physical & movement); the other 2 domains (personal care & instrumental, applied cognition) were more challenging because of the variability in patient recovery and ceiling effects. Conclusions: ITR methods and item pooling procedures were valuable in developing paired sets of short-form instruments for inpatient and community rehabilitation that provided estimates of functioning along a common metric for use across postacute care settings. Key Words: Activities of daily living; Outcomes research; Rehabilitation by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation From the Research and Training Center on Measuring Rehabilitation Outcomes, Center for Rehabilitation Effectiveness, Sargent College of Health and Rehabilitation Sciences, Boston University, Boston, MA (Haley, Andres, Coster, Ni, Jette); and QualityMetric Inc, Lincoln, RI (Kosinski). Supported in part by the National Institute on Disability and Rehabilitation Research (grant no. H133B990005), the National Institute of Child Health and Human Development (grant no. R01 HD43568), and the Agency for Healthcare Research and Quality. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the funders. No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the author(s) or on any organization with which the author(s) is/are associated. Reprint requests to Stephen M. Haley, PhD, PT, Research and Training Center on Measuring Rehabilitation Outcomes, Center for Rehabilitation Effectiveness, Sargent College of Health and Rehabilitation Sciences, Boston University, 635 Commonwealth Ave, Boston, MA 02215, smhaley@bu.edu /04/ $30.00/0 doi: /j.apmr THE EFFECTIVENESS of rehabilitation services is best understood through repeated measurements of functional outcomes with a consistent, valid, and reliable assessment system. Within postacute care, there is a growing demand for outcome assessment systems that can monitor interventions and programs on an ongoing basis across diagnostic groups and across the continuum of care settings. 1 A well-accepted approach for addressing this need is the development of a clinical monitoring system designed for outcomes management. 2 As highlighted in the long-range research plan of the National Institute on Disability and Rehabilitation Research, 3 such clinical monitoring systems can provide important information on the outcomes of services to direct ongoing process and outcome improvements. 4-6 Rehabilitation programs that serve patients across multiple service settings are hampered by a paucity of practical outcome instruments to monitor the effectiveness of interventions from hospital to community reentry. 7-9 There are, however, several well-respected outcome-monitoring systems currently in use in postacute care appropriate for a single setting. Some of the most prominent examples include the Uniform Data System for Medical Rehabilitation, 10 Minimal Data Set 11 (MDS), and the Standardized Outcome and Assessment Information Set for Home Health Care 12 (OASIS). However, for the purpose of monitoring outcomes across service settings, existing postacute care monitoring systems have several important shortcomings. These include poor measurement breadth, lack of precision across different patient ability levels, and, in some cases, limited feasibility. It is difficult for 1 instrument to cover the number of items necessary to reflect the full range of ability levels of persons in postacute care settings. Ceiling effects are common when disability assessments that are primarily developed for 1 postacute care setting are used in another, such as when the FIM instrument is applied to outpatient populations. 13 The assessment challenge is to identify an optimal set of functional items that is relevant for the diverse group of patients who receive postacute care services across different settings yet is feasible for use in the current health care environment. A variety of issues are encountered when trying to meet this challenge. Data from functional assessments may be compromised or not even collected if the response burden on clinicians or patients is too high. Thus, brevity is desired whenever possible in designing an outcome system. 14 If not developed properly, static clinical forms or interviews with a fixed number of items may not capture every patient s range of functional status and recovery when applied in different care settings. Clearly, the widespread adoption of short forms such as the Medical Outcomes Study 8- Short-Form Health Survey 15 (SF-8) and the SF in a variety of community-based health care services in the past decade underscores the importance of practical considerations in determining whether patient-based instruments will be used to monitor outcomes. Clinical programs often choose feasibility and low response burden over comprehensiveness of content. Several solutions to this outcome-assessment quandary have been suggested, including selecting the best items from a comprehensive item pool.

2 650 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley An item pool is a collection of items that represent a range of performance or difficulty levels of a particular functional domain. 17 pools are developed by equating outcome items from different sources so that they can be meaningfully compared on a common underlying metric. response theory (IRT) methods can been used to calibrate items from existing instruments onto a common scale, thus developing a structure and order of domain-specific items. 18,19 Once the structure and ordering of items is determined, items can then be selected to create short forms based on a number of criteria, including comprehensiveness of content, item fit to the construct, item precision, correlation to the total item score, and practical considerations of length. In this article, we describe the use of item pooling and IRT methods to develop paired sets of short-form instruments created for outcome assessment across inpatient and community settings. We chose to use the World Health Organization s International Classification of Functioning, Disability and Health 20 (ICF) as a framework for constructing an activity item pool for monitoring postacute care services. Within the ICF framework, activity is defined as the execution of a task or action by an individual. 21 In previous work, 21 we identified 3 separate activity domains for postacute care functioning: (1) movement & physical (body position and transfers, moving around, bending and lifting), (2) personal & instrumental (personal care, home skills); and (3) applied cognition (speaking and understanding, use of telephone, interpersonal skills, use of print information). We applied this framework to develop activity item pools for each of these 3 functional domains, which we refer to collectively as the Activity Measure for Post-Acute Care (AM-PAC). AM-PAC is designed as a comprehensive set of item pools from which paired sets of short-form instruments can be developed. In building the item pools, we drew on core functional items that were used in establishing the 3-domain (physical & movement, personal care & instrumental, applied cognition) activity model, tested new items (supplementary) based on ICF activity categories that were important for community functioning, added activity items that addressed the use of assistive technology, and cocalibrated these newly constructed items with items from existing outcome measures used in postacute care settings. Once developed, these item pools could be used to create short-form instruments that can be applied across different postacute care settings. In this article, we describe the content, psychometric properties, and floor and ceiling effects of the short forms, as well as estimates of how the short-form items match person abilities from people in inpatient and community rehabilitation settings. METHODS Participants This study included a sample of 485 participants who were receiving rehabilitation services from 6 health provider networks in the greater Boston area. Patients were recruited from (1) inpatient (199 from acute inpatient rehabilitation, 90 from transitional care units) and (2) community settings (90 from ambulatory services, 106 from home care). Eligibility criteria included age of 18 years or older, currently receiving skilled rehabilitation services (physical, occupational, or speech therapy), and ability to speak English. The sample was stratified to include approximately equal numbers of subjects in 3 major patient groups: (1) 33.2% with neurologic disorders (eg, stroke, multiple sclerosis, Parkinson s disease, brain injury, spinal cord injury, neuropathy), (2) 28.4% with musculoskeletal disorders (eg, fractures, joint replacements, orthopedic surgery, joint or muscular pain), and (3) 38.4% with medically complex disorders (eg, debility resulting from illness, cardiopulmonary conditions, postsurgical recovery). To assure good representation of levels of functional severity, recruitment was also stratified to yield a wide distribution of subjects representing 3 distinct severity levels slight (35.9%), moderate (44.1%), and severe (20.0%) based on scores from an adapted Modified Rankin Scale. 22 The sample was heterogeneous and reflects the racial and ethnic distribution of the greater Boston metropolitan population. The sample contained a wide age range (19 100y; mean age, 62.7y). Most subjects were women (58.8%), white (81.6%), and nonmarried (61.1%). Almost half (48.7%) had only a high school education or less. The wide range of onset from initial injury or illness ( y) characterizes different stages of recovery within both inpatient and community settings. The SF-8 15 data indicated that physical functioning of the overall sample (mean standard deviation [SD], ) was below the US population norms (mean, 50 10), although mental functioning (mean, ) was consistent with US population norms (mean, 50 10). Instrumentation A core set of 58 newly constructed items was chosen for inclusion in the AM-PAC item pool based on a comprehensive review of items from existing instruments, the ICF framework, a review by 10 measurement and content experts, and suggestions solicited from several focus groups of people with disabilities. These core items served as the common items administered to all subjects (common-item test equating design). 23 The 58 core activity items included 15 physical functioning, 14 self-care, 12 daily routine, 11 communication, and 6 interpersonal interaction items. Activity questions were phrased, How much difficulty do you currently have (without help from another person or device) with the following activities...? A polytomous response choice included none, a little, somewhat, a lot, and cannot do. We framed the activity questions in a general fashion without specific attribution to health, medical conditions, or disabling factors. In addition to these core items, to which all subjects responded, another group of items (supplementary) were derived from ICF content to administer to people receiving services in community settings (home care, outpatient). The addition of community-based items was important because there is a predominance of basic items and a paucity of community-focused items in existing functional outcome instruments. The supplemental item set included 52 items that focused on communityrelevant activities, such as walking outdoors, transferring to a car, and instrumental activities at home. A third set of activity questions (devices) was administered to people who used assistive technology for movement (25 items) or communication (8 items). This set of items included use of a device in the definition of the activity, such as How difficult is it currently to walk several blocks outdoors with the assist of a cane or walking device? A final set comprised items from existing standardized instruments in postacute care. These included the FIM 10 and the Minimum Data Set Post Acute Care 24 (MDS-PAC) for inpatient rehabilitation settings, the MDS 11 used in long-term care settings, and the OASIS 12 for home care. Only items from these instruments that met the ICF definition of activity were included in the analyses. We included 16 FIM items (excluding bowel and bladder management), 22 MDS items (self-performance physical functioning, selected cognitive activities), 25 MDS-PAC items (self-performance functional status, selected cognitive activities), and 17 OASIS items (activities of daily

3 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley 651 living, instrumental activities of daily living). Because no consistent data were available from charts in the outpatient setting, we administered the 10 physical functioning items (PF-10) of the SF to all individuals receiving outpatient services. In accordance with FIM scoring rules, items that were not administered are scored as total dependence. Both the MDS and MDS-PAC have a response option of activity did not occur. We converted these codes to the lowest score for that item, namely, total dependence. We reasoned that the most likely explanation for the activity not occurring was that the item could not be performed, an assumption that others have made when comparing functional instruments. 26 Data Collection Procedures Data on core, supplementary, and assistive technology questions were collected via subject or proxy interviews. The ability of a subject to take part in the interview self-report was assessed by a treating clinician who determined whether the respondent could (1) understand the interview questions, (2) sustain attention during an interview, and (3) reliably respond to the questions. If the answer to any of these screening questions was no, then the interview was completed by either a clinician or a family member proxy participant. A proxy participant completed fewer than 3% of the interviews. We found no differences in item calibrations, whether or not we included proxy data; therefore, proxy responses are included in these analyses. Each interview ( 45 60min) was conducted in a quiet space, either in an inpatient setting, an outpatient facility, or the participant s home. The interview was fully scripted, with standard instructions and an answer card to help subjects identify the desired response choice. The order of presentation of test sections was randomly assigned, to mitigate the possibility of large portions of missing data in any 1 section because of respondent fatigue. For the items in the applied cognition domain, we developed a screening procedure consisting of 6 difficult items. If a subject identified difficulty on any of these 6 items, all additional items were administered. However, if the subject had no difficulty on any of the screener items, then the remainder of the items were skipped and scored as no difficulty. This procedure was used to avoid asking applied cognition questions that clearly were not applicable to subjects who were without functional deficits in communication, print information, or use of the telephone. Twenty-nine percent of the entire sample were able to answer the initial questions with no difficulty and were given a no difficulty rating for all the items. Interviewers, who were experienced rehabilitation clinicians (mean clinical experience, 5.7y), received training and quality assurance from an initial 3-hour training session, a protocol manual, supervision by the research project staff on all first interviews, and acceptable completion of a procedural checklist and interrater reliability on a subsequent interview. A project staff member accompanied interviewers on approximately every tenth interview, to check on adherence to study protocol and to ensure overall quality control. Data on items from existing instruments (eg, FIM, OASIS, MDS) were recorded from retrospective chart review. Interviews were timed to coincide with administration of standard outcome instruments. For example, the FIM is administered in most facilities within 3 days of admission and before discharge in the inpatient rehabilitation setting. Thus, the subject interview was arranged to take place within 3 days of admission or at discharge, so that the FIM information was collected close to the time of the interview. Likewise, interviews of subjects in transitional care units were scheduled to coincide with the MDS assessment. Subjects receiving homecare therapy were interviewed either near to admission or to discharge, and the OASIS assessment was performed in close proximity to the subject interview. Because the MDS-PAC was not routinely used in any of our data collection sites, interviewers were trained to collect these data at the same time as the subject interview. The PF-10 items were also administered at the time of the interview by data collections for subjects in outpatient programs. All subjects, regardless of setting, were administered the SF-8 15 items to establish a normative description of the sample. Analyses Analyses were conducted in 2 stages. First, we developed separate item pools representing activity in the 3 domains. As reported elsewhere for personal care & instrumental activity items, 27 we conducted a series of 1-parameter Rasch partialcredit analyses for each dimension-specific item pool, to test the dimensionality of each scale and to estimate item parameters. 28,29 We used a method of concurrent calibration, 30 which involves estimating item and ability parameters in the subject groups simultaneously. This approach treats items not taken by subjects as missing data, because the Rasch program uses information available to estimate person and item parameters. A method to evaluate the success of an item pool is to estimate the extent to which the people and item locations (estimates) are spread apart, relative to their respective standard errors (SEs). A separation statistic indicates the extent to which the SD of the person or item estimates are larger than the measurement error associated with those same estimates. 28 In addition, a reliability statistic, similar to a traditional internal consistency statistic, is computed. The closer it is to 1.0, the higher the reliability of the separation statistic. Second, we selected 10 items from each domain to be included in the short forms for inpatient and community settings. Information on 6 short forms, 1 for each of the AM-PAC domains (physical & movement, personal care & instrumental, applied cognition) for each of 2 postacute settings (inpatient, community) is reported here. We have also developed an alternative physical & movement short form for persons who use a wheelchair, but in this article we report only on the ambulation physical & movement short forms. We used a variety of criteria to select short-form items, including maximizing content coverage, maximizing precision across the full functional range, and minimizing ceiling and floor effects. We used test information functions to help select a subset of items from the full-item pools that provided the best match of measurement precision to the ability levels of the sample in each functional domain and setting (inpatient, community). Information function indicates in which range of functioning an item or item set is most useful for distinguishing among individuals. information functions are related to the location and shape of the item characteristic curve, which describes the probabilities of responding to particular response options of an item. Using item information functions, we can identify whether a given item is precise at any level of the activity domain; higher item information means more precision. information values are continuous over the ability scale, and they can be summed and displayed graphically to form test information functions (TIFs). To determine the best item set for the short forms using the TIFs, we added and removed items iteratively, so that a subset of items was identified that most closely fit the levels of precision needed to match the person ability scores. TIFs are approximately equal to the expected value of the inverse of the squared SE of the ability estimate; thus, the smaller the SE of measurement, the

4 652 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley Source and Sample Table 1: Source of s for AM-PAC Pools Movement & Physical AM-PAC Domain (no. of items) Personal Care & Instrumental Applied Cognition Core set (N 485) Supplemental (community) (n 289) Assistive technology (N 485) Postacute care instruments MDS (n 91) MDS-PAC (n 86) FIM (n 108) OASIS (n 103) SF-36 (PF-10) (n 82) Total available Total retained* *See text for explanation of removed items. Total s more information or precision a subset of items provides at any 1 point along each activity scale. Because our analytic goal was to match as closely as possible item precision with the ability of the target population during inpatient or community recovery, we calculated the short-form TIFs with the average person scores (in logits) and ranges of 2 SDs. For the applied cognition scales, we used the scores on only the patients with neurologic problems, because there was a large ceiling effect with the entire sample. To quantify the match between the TIFs, we calculated the area under the TIFs that corresponded to the range of the sample ability estimates ( 2 SDs of scores on each activity domain). For each final short form, we calculated floor and ceiling effects and compared those with the floor and ceiling effects of the full-item pool. Floor and ceiling effects were calculated based on the percentages of persons obtaining minimum and maximum possible scores. The scores on the short forms and the long forms are based on a common cocalibrated item pool for each activity dimension and are based on a 1-parameter Rasch model. To gain an estimate of bandwidth, or content coverage, 34 of each short form, we compared the range from the lowest to the highest item-step estimates of the short forms to the complete item pools. In polytomous item response models using partial-credit analyses, the difference between the lowest and the highest category step-estimate calibration reflects the full functional range of the set of items. 35 For the final short forms, we report the average item calibrations of each short form as estimates of each item s difficulty along the continuum. We also report the informationweighted fit statistic (infit z standard), which is standardized to approximate a mean of zero and an SD of 1. The infit statistic of 2 or more was used as the criterion for deciding whether an item or person fit statistic was unexpected according to the predicted model. Values greater than 2 indicate potential departures from unidimensionality, whereas values less than 2 indicate potential violations of local independence. RESULTS AM-PAC Pools Table 1 summarizes the source of items retained in each domain-specific pool. Even though we had considerable item redundancy in each domain item pool, we sought to keep as many items in the pool as appropriate (based on item goodness of fit properties and content analyses). 18 All available items were retained in the physical & movement item pool, 2 items (cut and file toenails, use an ATM machine) were removed from the personal care & instrumental item pool, and 10 items were removed from the applied cognition item pool. These included 8 items on assistive technology for communication (only applicable to 6 subjects; thus the items were dropped) and controlling temper and coping with noise in the environment because of poor item fit. Figure 1 summarizes the range of item content of each item pool, as indicated by scale estimates of the lowest (left column), highest (right column), and average (center column) rating category calibration for each item. These functional content ranges are placed on the same scale as person scores for each item pool. Person separation statistics ranged from 5.26 for physical & mobility domain to 2.17 for the applied cognition domain. separation statistics, which reflect variation among item difficulties, ranged from 9.48 to 3.15 (table 2). Based on our experience with these types of items and persons, 36 these statistics are satisfactory in their representation of the spread (which we want to be wide in terms of coverage) and precision (which we want to be high in terms of small standard errors of estimates) for the person measures and item estimates of the activity item pools. Content of AM-PAC Short Forms Physical & movement. The inpatient short form included 2 bed items (mobility, sitting), 1 item on use of the bathroom or bedpan, 2 chair transfer items, 2 standing items (reaching, bending), 2 hospital walking items, and 1 stairs item. The items are arranged in order of average item difficulty (easy to difficult) in table 3. Two items had larger than expected positive infit scores (sit up in bed, reach up while standing) but were retained in the short forms because of content relevance. We included 1 item from the ICF supplemental item set (transfer from low chair/furniture) because it gave us better content range in the inpatient short form. We retained 1 PF-10 item (bending, kneeling, stooping), because it was a challenging item; however, we removed attribution to health care status in the rating scale, and the time frame of the item was within the past 24 hours rather than within the past month. In cases in which the original item was administered as a clinician report, we altered the wording on the short forms to allow for an alternative self-report or patient interview format. Particularly for use in acute care settings, in which many patients do not have the opportunity to get out of bed, we added the response choice not performed activity in inpatient facility. The community short form included 2 basic bed and transfer items, 1 bending item, 4 walking items (home, building,

5 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley 653 Fig 1. AM-PAC item pool and person score ranges. Note that item calibrations are provided for the lowest and highest scale points for each item, along with the average item calibrations. Person range refers to summary scores along each of the activity scales. Table 2: Person and Separation Statistics for AM-PAC Pools AM-PAC domain Person Separation Index Reliability Separation Index Reliability Movement & physical Personal care & instrumental Applied cognition blocks, 1 mile), 1 outdoor stairs using no handrail item, and 2 items that involve a range of moderate and vigorous activities. We retained the item sit up in bed in the community form, even though the positive fit was higher than expected, because it helped cover content for persons who are slow in recovering, and it allowed us to have at least 1 similar item between the 2 physical & mobility short forms. Having a similar item in each AM-PAC domain short form made the comparative scoring between the inpatient and community short forms more accurate. Two of the ambulation items (walk in home, walk inside building) used paired questions to estimate short-form scoring. For example, if a person used a device, responses to difficulty with and without the device were used to estimate the shortform score. Although we attempted to avoid this if possible, we did change the bed-to-chair transfer item to a 6-point scale, because we obtained a better distribution of scale responses than with a 7-point scale. Personal care & instrumental. The inpatient short form included 3 eating items (2 use of utensils, 1 removing wrapper), 4 dressing items (2 specific activities, 2 global assistance), 1 bathing item, 1 grooming item, and a combined bathing and dressing item. The items are arranged in order of average item difficulty (easy to difficult) in table 4. All items had acceptable item fit scores, except for put on and take off socks. This item was retained because it was relatively difficult and helped expand the range of content at the higher end of the scale. In the final short form, we have attempted to clarify this item to improve scale fit. We included 1 item from the ICF supplemental item set ( removing wrappings from small objects ) because almost all patients encounter this task when eating meals in the hospital. In cases in which the original item was administered in a clinician-report format (FIM and MDS-PAC items), we altered the wording on the short forms to allow for an alternative self-report or patient interview format. Particularly for use in acute care settings, where many patients do not have the opportunity to get dressed in street clothes, we added the response have not performed activity in the dressing items. The community short form included 3 advanced hand skill items (use of kitchen utensils, removing lid from jar, filing fingernails), 4 dressing and bathing items of varying difficulty, and 3 instrumental items (laundry, housekeeping, putting away clothes and dishes). We retained the items housekeeping, laundry, and socks and shoes, because of the important content and the coverage needed for relatively advanced recovery. Four of the items in this short form came from the ICF supplemental community item set. In cases in which the original item was administered in a clinician-report format (FIM, MDS-PAC, and OASIS items), we altered the wording on the short forms to adapt to an alternative self-report or patient interview format. Applied cognition. The inpatient short form included 1 basic comprehension item, 2 expressive communication items, 2 phone-use items, 2 reading items, 1 complicated written task, 1 memory item related to daily experiences, and 1 problemsolving item. The items are arranged in order of average item difficulty (easy to difficult) in table 5. Two items originating from the FIM instrument had unexpected positive infit scores (memory of daily routines, problem solving of complex tasks) but were retained in the short forms based on the importance of the content. These items were changed to a 6-point scale, because we obtained a better distribution of response categories. In cases in which the original item was administered as a clinician report (FIM, MDS-PAC), we altered the wording on the short forms to allow for an alternative self-report or patient interview format. We did not include the response choice have not performed activity in inpatient facility, because we rea-

6 654 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley Table 3: AM-PAC Physical & Movement Short Forms Source Instrument Average Location Infit z SD Comments on Short-Form Response Choices Short form: inpatient ambulator 1. Use of bathroom/bedpan OASIS point assistance scale* 2. Bed mobility MDS-PAC point assistance scale* 3. Bed and chair transfers MDS-PAC point assistance scale* 4. Sit up in bed ICF core point difficulty scale 5. Walk in hospital room MDS point assistance scale* 6. Reach up to high object while standing ICF core point difficulty scale 7. Walk in hospital facility off unit MDS point assistance scale* 8. Transfer from low chair/furniture ICF suppl point difficulty scale 9. Walk up and down stairs MDS-PAC point assistance scale* 10. Bending, kneeling, stooping SF point limitations scale; attribution to health conditions removed; time frame changed to current status Short form: community ambulator 1. Bed to chair transfer FIM Changed from 7- to 6-point scale* 2. Sit up in bed ICF core point difficulty scale 3. Walk in home with/without device ICF suppl/at point difficulty scale; if device is used, both with and without device options administered 4. Walk inside building with/without device ICF suppl/at point difficulty scale; if device is used, both with and without device options administered 5. Bending to pick up clothes from the floor ICF core point difficulty scale 6. Walk blocks ICF suppl point difficulty scale 7. Up and down stairs with no handrail ICF suppl point difficulty scale 8. Brisk one-mile walk ICF suppl point difficulty scale 9. Moderate activities (moving a table, golf) SF point limitations scale; attribution to health conditions removed; time frame changed to current status 10. Vigorous activities SF point limitations scale; attribution to health conditions removed; time frame changed to current status Abbreviation: AT, assistive technology; suppl, supplemental item set. *Reformatted rating scale categories for self-report. Additional response choice added: not performed activity in inpatient facility. Revised item definition to improve item fit. soned that all inpatients had an opportunity to use the telephone and to read and fill out forms. The community short form included 1 expressive item, 2 phone items, 2 money-management items (count out change, review financial statements), 2 complex tasks requiring following multiple directions (follow instructions for new home appliance, fill out long form), and other complex tasks such as problem solving, preparation of a complex meal, and planning for home tasks. We retained the meal-preparation item in the community form, even though the positive fit was higher than expected, because it helped cover content for persons who were advanced in applied cognitive skills; however, we altered the item description to improve scale fit. Two items (fill out a long written form, problem solving of complex tasks) were similar for the inpatient and community short forms, to allow better linking of scoring between the 2 applied cognition short forms. Test Information Functions and Sample Scores Figures 2 through 4 display the summed TIFs of the 10 items in each short form (inpatient, community) displayed with the mean and 2 SD distributions of scores for persons in the inpatient and community samples. For example, in figure 2, note that the peaks of the information function curves are approximately at the midpoint of the sample score distributions for each physical & movement short form. The inpatient short form provides more measurement precision at the lower end of the ability range (during early recovery), whereas the community short form provides more measurement precision at the higher ability range. Thus, the items selected for each short form match the sample for which they were intended. As noted in figure 2, 2 SDs of the inpatient sample scores on the physical & movement domain corresponds to 77.7% of the area under TIF, and 2 SDs of the community sample scores on the physical & movement domain corresponds to 72.3% of the area under TIF. The match between TIFs and person scores is somewhat less for the personal care & instrumental inpatient (58.5%) and community (46.5%) forms and is even less for the applied cognition inpatient (45%) and community (49%) domain. Figures 3 and 4 also show that measurement precision is expected to be relatively poor in the higher ability ranges of the personal care & instrumental and applied cognition domains. Further, the short forms for both inpatient and community settings for personal care & instrumental and applied cognition can be expected to perform about the same (only slight improvement in precision for the community short forms at higher ability levels), even though the items differ in setting short forms.

7 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley 655 Table 4: AM-PAC Personal Care & Instrumental Short Forms Source Instrument Average Location Infit z SD Comments on Short-Form Response Choices Short form: inpatient 1. Eating utensils: spoon and fork ICF core point difficulty scale 2. Grooming FIM point assistance scale* 3. Upper-body dressing FIM point assistance scale* 4. Remove wrappings from small objects ICF suppl point difficulty scale 5. Lower-body dressing MDS-PAC point difficulty scale 6. Bathing or dressing SF point limitations scale; attribution to health conditions removed; time frame changed to current status 7. Bath/shower/sponge bath MDS-PAC point assistance scale* 8. Cutting meat or vegetables ICF core point difficulty scale 9. Put on and take off pants ICF core point difficulty scale 10. Put on and take off socks ICF core point difficulty scale Short form: community 1. Upper-body dressing FIM point assistance scale* 2. Bathing or dressing SF point limitations scale; attribution to health conditions removed; time frame changed to current status 3. Trimming fingernails ICF suppl point difficulty scale 4. Bath/shower/sponge bath MDS-PAC point assistance scale* 5. Use of kitchen utensils ICF suppl point difficulty scale 6. Put away dishes and laundry ICF suppl point difficulty scale 7. Put on/off shoes and socks ICF core point difficulty scale 8. Remove lid from unopened jar ICF suppl point difficulty scale 9. Housekeeping OASIS point assistance scale* 10. Laundry OASIS point assistance scale* *Reformatted rating scale categories for self-report. Additional response choice added: not performed activity in inpatient facility. Revised item definition to improve item fit. Floor, Ceiling, and Coverage Range Results of the analysis of floor and ceiling effects of the short forms are summarized in table 6. For the physical & movement and personal care & instrumental domains, the short forms have approximately a 1% to 5% floor effect and a 7% to 10% ceiling effect. These are marginal increases from the floor and ceiling effects noted for the full item pools. The floor effect for the applied cognition domain is essentially the same as the full item pool (59 items); however, the ceiling effects rise from 4% to 7% with the full item pool to 17% to 20% for the short forms for patients with neurologic diagnoses. The content coverage range of each short form is listed in table 6. Note that ceiling effects persist on short forms, even though the item that is most difficult (representing coverage to 100) is included in each short form. The coverage range of each short form does minimize floor effects on any of the scales. DISCUSSION The results show that general Rasch methods can be an effective methodology for building short forms tailored to postacute patients, to obtain estimates of functioning along a series of broad functional metrics. Using paired sets of short forms allows us to estimate and compare scores for patients who answer different sets of relevant items as they receive services across postacute care settings. The setting-specific short forms that we have developed here are an attempt to provide the most relevant items to persons at each stage of functional recovery. We have used the 1-parameter Rasch model at this point in our research because it seems to provide stable estimates of item difficulty requiring fewer cases (in this sample, 500) than 2-parameter models. 35 Static short forms are a compromise between content comprehensiveness and precision. 34 In this study, we were successful in developing short forms that provided a broad range of coverage for each setting, particularly for the physical & movement and personal & instrumental domains. As noted in table 6, the range of coverage in the inpatient physical & movement short form was between approximately 1% and 80%, whereas the community form covered about 15% to 100% of the range. With these overlapping ranges, we had very little change in ceiling and floor effects going from the full item pool (101 items) to the short forms. However, more item development is needed in the personal care & instrumental short forms, to raise the ceiling for those who make a strong recovery in the community setting. We also had difficulty minimizing ceiling effects in the area of applied cognition. Because of our criterion for subject inclusion (able to reliably answer about daily events), we often eliminated many participants who might have scored lower in this area. Clearly, this is an important area of work that needs much more attention and future research. We found that the approach of matching TIFs to ability of the target population in each setting was very useful in selecting items to include in each short form. This process goes beyond just matching item difficulty with person scores. Although item difficulty and precision are related, they are not always the same, particularly with polytomous scales. Our ability to better match information function to the ability level

8 656 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley Table 5: AM-PAC Applied Cognition Short Forms Source Instrument Average Location Infit z SD Comments on Short-Form Response Choices Short form: inpatient 1. Understanding familiar people during conversations ICF core point difficulty scale 2. Describe an event that has happened ICF core point difficulty scale 3. Assistance in use of telephone MDS-PAC point difficulty scale* 4. Requesting information on phone ICF core point difficulty scale 5. Reading simple material (eg, hospital menu) ICF core point difficulty scale 6. Carry on a conversation in a group/noisy environment ICF core point difficulty scale 7. Read newspaper or magazine ICF core point difficulty scale 8. Problem solving of complex tasks FIM Changed to 6-point scale* 9. Memory of daily routines FIM Changed to 6-point scale* 10. Fill out long written form ICF core point difficulty scale Short form: community 1. Phone use OASIS point difficulty scale 2. Explaining how to do something ICF core point difficulty scale 3. Count out correct money ICF suppl point difficulty scale 4. Look up phone number or address ICF core point difficulty scale 5. Problem solving of complex tasks FIM Changed to 6-point scale* 6. Plan and manage daily routine ICF suppl point difficulty scale 7. Check finances ICF suppl point difficulty scale 8. Read and follow complex instructions ICF core point difficulty scale 9. Fill out long written form ICF core point difficulty scale 10. Meal preparation MDS-PAC point assistance scale* *Reformatted rating scale categories for self-report. Revised item definition to improve item fit. of each target group was due in part to the greater number of items we had available from the physical & movement item pool ( 50% more) than from the other 2 domains. The smaller number of items in the personal care & instrumental and applied cognitive domains also affected the separation in ability levels of the inpatient and community samples. Recent work has suggested the existence of 3 separate domains of functioning for activity assessment in postacute care, 21 whereas the current ICF structure suggests that there may be up to 8 domains of functional activity. 20 However, using multiple domains to assess function is somewhat contrary to recent work that promotes 1 underlying dimension of function assessment. For example, McHorney 37 recently linked a wide range of items into a single scale of functioning that included items such as balance checkbook, remember to turn off appliances, and remember people s names, along with physical functioning items (walk half block, climb down 20 steps). Hart and Wright 38 report on a new index of physical functional health status, with items such as pain intensity and bladder control. Although items from different domains may fit statistically along a 1-dimensional scale, reliance on this criterion suggests that methodologic unidimensionality is considered more important than conceptual unidimensionality. This assumption deserves much more critical analysis. 35,39 In the community physical & movement short form, we have incorporated specific ambulation items that include a device Fig 2. Paired test information functions for inpatient and community short forms for the AM-PAC physical & mobility domain. Abbreviation: pauc, percentage area under the curve.

9 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley 657 Fig 3. Paired test information functions for inpatient and community short forms for the AM-PAC personal care & instrumental domain. and no device option. In other work, 40 we have observed very different item calibrations for self-reported ambulation items with and without a device using a difficulty response choice. This difference occurs primarily with ambulation items within the home and inside buildings. calibrations for more advanced ambulation items with and without a device in the community (eg, walk a brisk mile) are essentially the same; thus, use of devices appears not to matter once ambulation progresses to a certain point. Based on our results, we recommend that ambulation with and without a device (for less advanced ambulation items) should be treated as different items. Thus, in our short forms, only those persons who use a device are asked to respond to the additional device questions. For those using a device, this gives us a better estimate of their overall physical and movement performance than if we just asked them about ambulation without a device. Our strategy in developing the item pools was to include items from existing instruments that continue to serve as useful outcome monitoring instruments. One of the advantages of doing this is that we can link prospective studies to earlier work and have the ability to score new instruments in terms of the metric used in existing instruments. However, because existing instruments use a variety of evaluative criteria (ie, assistance, limitations), we had to introduce multiple response sets into the item pool. Earlier work has shown that items from existing measures with different response sets can be successfully merged into a single scale. Rather than making the scale more complex, Gill et al 44 suggest that difficulty and dependence provide complementary information that together can depict the continuum of function more fully than either response scale alone. In developing item pools, it is often easy to focus on item content and not to consider the importance of the response Fig 4. Paired test information functions for inpatient and community short forms for the AM-PAC applied cognition domain.

10 658 OUTCOMES RESEARCH, REHABILITATION, AND PSYCHOMETRICS, Haley Short Form n Table 6: Floor, Ceiling, and Coverage Range of AM-PAC Short Forms Full Pool (101 items) Physical & Movement (ambulator) Floor, n (%) Ceiling, n (%) Short Forms (10 items) Full Pool (101 items) Short Forms (10 items) Percentiles of Full- Pool Range Inpatient (0.7%) 13 (4.5%) 0 (0%) 21 (7.3%) 1 79 Community (0%) 6 (3.1%) 0 (0%) 4 (2.0%) Short Form n Full- Pool (62 items) Personal Care & Instrumental Short Forms (10 items) Full- Pool (62 items) Short Forms (10 items) Percentiles of Full- Pool Range Inpatient (0%) 0 5 (1.7%) 20 (6.9%) 1 63 Community (0.5%) 3 (1.5%) 8 (4.0%) 20 (10.2%) Short Form n Full Pool (59 items) Applied Cognition* Short Forms (10 items) Full- Pool (59 items) Short Forms (10 items) Percentiles of Full- Pool Range Inpatient 76 0 (0%) 0 3 (4.0%) 15 (19.7%) 1 76 Community 54 0 (0%) 1 (1.9%) 4 (7.4%) 9 (16.7%) *Sample with neurologic diagnoses only for each setting (inpatient, community). categories. 35 To retain the character and meaning of the items from the original source, we have been reluctant to make substantial changes in the response sets with our current sample sizes for items from existing instruments. However, we have suggested some general changes in some rating scale categories in the deployment of the short forms for future applications (tables 3 5), as others 45 have found that collapsing categories can improve item fit and can reduce disordered rating scale categories. We also have adapted a number of the clinicianbased items so that they would be more amenable to an alternative self-report or patient interview format. A new prospective study is underway that will examine the properties of the AM-PAC short forms in a new sample. A potential limitation of our study is the combining of item calibration estimates using both clinician and patient selfreports. In a subset of the sample, Andres et al 46 found very acceptable agreement between clinician report and patient selfreport (intraclass correlation coefficient [ICC].90 for physical & mobility, ICC.86 for personal & instrumental, ICC.68 for applied cognition). We believe that these results suggest that data from multiple respondents on at least 2 of the 3 activity constructs (applied cognition is lower partially because of decreased variability of overall scores) can be combined with minimal loss of accuracy in building item calibrations. However, the effects of using multiple respondents for building item pools or for collecting data across settings need to be examined carefully in future studies. Although many methods and analytic techniques are available to develop short forms, such as linear regression, factor analysis, interitem correlations, item total correlations, comparing items to external criteria, and using patients ratings of importance, we have chosen a strategy of linking items together in a common pool and using IRT analyses to select the most relevant items for each care setting. We have used TIFs, assessment of person and item fit, Rasch content-based interpretations, and Rasch partial-credit models to create a paired set of linked short forms. These methods have also created efficient and comprehensive short forms to measure headache impact. 47 One decided disadvantage of the short forms over the entire item pool is the loss of precision or confidence around any single score. For example, in the physical & movement domain, 95% confidence intervals (CIs) for midrange scores (where precision is often the best) on the inpatient short forms are slightly greater than 1.00; 95% CIs in the midrange of the community short form is on the order of.80. This is in contrast with the 95% CIs in the midrange scores of the entire item pool, which are on the order of.34 ( 2 3 times less). Thus, the improvement in efficiency does result in a loss of precision in the estimation of any individual score. Similar item pooling and Rasch methods can be used to develop computer-adaptive testing (CAT) applications. 18,48 CAT methodology uses a computer interface for the person (or a computerized interview/clinician report) that is tailored to the unique ability level of that person. CAT applications require (1) a large set of items in any one functional area (item pools), (2) items that consistently scale along a dimension of low to high functional proficiency, and (3) rules guiding starting, stopping, and scoring procedures. The CAT application can achieve good efficiency without the dramatic loss in individual score precision that is seen with the short forms. The development of these short forms is a transitional step for those postacute settings in which CAT applications are neither feasible nor likely to be accepted. If developed in concert, short forms can be developed with the same items and with the same underlying metrics as CAT applications, thus making the eventual transformation from static short forms to CAT applications easier and more acceptable. CONCLUSIONS Paired sets of 10-item short forms were developed for the assessment of postacute activity functioning across inpatient and community settings. The short forms cover 3 distinct activity domains: physical & movement, personal & instrumental, and applied cognition. The short forms were developed from large item pools that included items from existing instruments used in current postacute settings and from new items based on an ICF framework. We demonstrated how information functions can be used to select items to gain precision in relation to the ability levels of the intended samples for each setting. Future work will examine how these short forms perform in practice to monitor functional recovery over time in a