Test-Retest Reliability of an Abbreviated Self-Report Overall Health Status Measure

Transcription

1 Test-Retest Reliability of an Abbreviated Self-Report Overall Health Status Measure Dennis L. Hart, PT, PhD 1 Journal of Orthopaedic & Sports Physical Therapy Study Design: Test-retest reliability study. Objective: To assess test-retest reliability and estimate minimal detectable change of an overall measure and 2 summary measures of patient self-report of health status. Background: Change in patient self-report of health status is a common outcome measure following rehabilitation. Because collection of health status data takes time and clinicians are required to be productive, selected items from reliable instruments were used to form a new, abbreviated instrument of health status relevant to patients in outpatient rehabilitation. There are no test-retest reliability statistics of these health status measures in this population. Methods and Measures: A convenience sample of 71 patients (mean age ± SD, 41.9 ± 17.9 years; age range, years; sex, 35% male), with a variety of orthopaedic diagnoses, seeking rehabilitation in 2 outpatient facilities, volunteered. Patients completed health status questionnaires at initial evaluation and at 24 to 72 hours following evaluation. Intraclass correlation coefficients (ICC 2,1 ) were used to estimate test-retest reliability and to estimate measurement error and minimal detectable changes. Results: ICCs with 1-sided lower limit 95% confidence intervals (CI) of the Overall Health Status measure and the Physical and Mental Component Summary measures for patients with chronic symptoms were 0.92 (0.85), 0.82 (0.68), and 0.85 (0.74), respectively. Minimal detectable changes (90% CI) were ±12 (scale range, 100), ±9 (scale range, 60), and ±9 (scale range, 60) scale points, respectively, for the same measures. Conclusions: Results support the test-retest reliability of the Overall Health Status measure and summary measures for patients with chronic symptoms and demonstrate ability of the Overall Health Status and Physical Summary Scale measures to detect improvement of patient self-report of health status within the first few days of rehabilitation. J Orthop Sports Phys Ther 2003;33: Key Words: minimal detectable change, questionnaire, reliability, SF-12, SF-36Patient self-report of health status provides a measure of perceived function that is amenable to rehabilitation clinical practice and research. 17 Although there is no well-accepted criterion standard for measuring health status, 2 health constructs of the MOS SF ,37 are the principal generic measures to which other self-report measures are commonly compared. 2,11,14,16,17,19 The SF-36 was designed to assess well-being and functional abilities via patient self-report in person or by telephone interview by quantifying physical and psychosocial constructs. 37 The SF-36 is a multifaceted health-related quality-of-life tool that quantifies 8 health constructs, including general health, physical functioning, role functioning, bodily pain, mental health, emotional functioning, vitality, 1 Director of Research and Consulting Services, Focus on Therapeutic Outcomes, Inc., White Stone, VA , USA. Address all correspondence and reprint requests to Dennis L. Hart, Focus on Therapeutic Outcomes, Inc., 551 Yopps Cove Road, White Stone, VA dsailhart@rivnet.net The Focus on Therapeutic Outcomes, Inc., Institutional Review Board for the Protection of Human Subjects reviewed and approved this study. and social functioning. 20,21,37-38 These constructs can be condensed into 2 summary measures: the Physical Component Summary (SF-36 PCS) and the Mental Component Summary (SF-36 MCS) measures. 36 Reliability and validity of the 8 health construct and 2 summary measures have been supported in groups of patients with a variety of medical conditions and functional deficits ,36-38 Because collecting health status data takes time, developers continue to downsize health status instruments in an effort to reduce the administrative burden associated with data collection. 34 Developers of health status instruments are challenged by competing interests: the use of many items to enhance comprehensiveness of content and precision of scores versus the use of a reduced number of items to reduce administrative burden while maintaining acceptable measure precision. 34 One example of an effort to reduce respondent burden while maintaining adequate measure precision of the original instrument is the SF-12, 34 with its component summary measures (SF-12 PCS and MCS), 35 which were developed from a subset of SF-36 items. Test-retest reliability and validity of the SF-12 PCS and MCS measures are good, and the SF-12 PCS and MCS data explained 90% of the variance of the parent SF-36 PCS and MCS scores. 34 These findings demonstrate that shorter instruments can 734 Journal of Orthopaedic & Sports Physical Therapy

2 be condensed from larger instruments for reduced administrative burden while maintaining adequate measure precision, reliability, and validity. 34 Another way to develop efficient outcomes tools is to place items from reliable and valid instruments into a new instrument. When items from different instruments are embedded in a new instrument, even when they retain their original wording, the psychometrics of the new measures must be reassessed. A new outcomes instrument using a subset of SF-36 items plus several items pertinent to patients with upper-extremity impairment has been developed. 12 The new instrument was developed because it reduced the administrative burden of data collection and was more specific to patients seeking outpatient rehabilitation than the SF-36. Administrative burden was reduced because patients answered 12 fewer items than they would in the SF-36. The new instrument, as compared to the SF-36, was more specific to patients receiving outpatient rehabilitation because there were an expanded number of physical functioning items, including some physical functioning items pertinent to patients with upper-extremity impairments. Data from patients receiving rehabilitation have shown varying degrees of responsiveness for SF-36 subscales, depending on the patient sample. 38 For this paper, responsiveness to clinical change is operationally defined as the ability of a scale to detect change. 18 For example, in 2 studies of patients receiving rehabilitation in outpatient facilities, 3 subscales (the Physical Functioning, Role Physical, and Bodily Pain scales) were most responsive to clinical change In another study of patients receiving outpatient rehabilitation, 6 the Bodily Pain and Physical Functioning SF-36 subscales were responsive to varying levels, depending on impairment. In a study of patients seen in orthopaedic back pain clinics in a tertiary hospital, most SF-36 subscales were responsive, but most patients in the sample received spinal surgery, or facet joint or epidural injections. Less than 20% of patients received physical therapy. 33 In a study of older patients with chronic osteoarthritis recruited from a rheumatology clinic where they were undergoing inpatient rehabilitation, 1 pain subscales were more responsive than functional SF-36 subscales. From these studies, although improvement in physical functioning and pain are the major goals of most patients seeking outpatient rehabilitation, a limited number of psychosocial items seem warranted for this population. 12,26 The SF-36 was designed for ambulatory adults, regardless of medical condition or impairment, 37 and has limited applicability and reduced responsiveness 6 for patients with upper-extremity functional deficits. The new instrument contains 3 items designed to improve responsiveness for patients with functional deficits secondary to upper-extremity impairments. 9 The 3 upper-extremity items were shown to improve content and construct validity of the SF-36 Physical Functioning scale 38 for patients receiving rehabilitation for upper-extremity impairments. 9 The SF-12 has reduced administrative burden compared to the SF-36, 34 which is attractive. However, in a study of patients with low back pain receiving outpatient rehabilitation, scores from the SF-36 Physical Functioning scale, the SF-36 PCS, and the SF-12 PCS were equally sensitive to clinical change for groups of patients; however, the SF-12 PCS scores did not adequately predict SF-36 PCS scores for individual patients. 25 This finding suggests that more items for physical functioning beyond the SF-12 are warranted for patients seeking outpatient rehabilitation, if measures of change for individual patients are of interest. Twenty-four items, as previously described, 12 were selected for the new instrument because all or subsets of the items have been shown to be responsive to clinical change for patients seeking rehabilitation in outpatient centers, allowed calculation of Physical and Mental Component Summary measures, 35 or have improved content and construct validity of the physical functioning items in patients with upperextremity impairments, 9 as compared to those of the SF-36 PF The intended purpose of the new instrument is to measure change in functional health status in patients seeking outpatient rehabilitation. From the new instrument, measures of overall functional health status and summary scores of physical and mental functioning can be generated at intake, throughout treatment and discharge, so that change in functional health status can be assessed over the treatment episode using any of the 3 measures. The Physical and Mental Functioning Summary scores in the new instrument are equivalent to those of the SF-12 PCS and MCS Responsiveness has been published for the 24-item and the summary measures (Table 1) for patients receiving rehabilitation. While the 24-item and physical summary measures have moderate to large 4 responsiveness to clinical change, responsiveness of mental functioning, as measured by the SF-12 MCS, 34 is typically less (effect sizes ). 12 Data exist supporting construct validity of the new 24-item measure, 24 and the Physical and Mental Component Summary scores for patients receiving rehabilitation. 10,12 In a large sample of patients with lumbar impairments receiving physical therapy (n ), patients treated by therapists with advanced manual training reported more gain (riskadjusted general linear models) in health status measured by the 24-item measure, as compared to patients treated by therapists with different advanced clinical training. 24 In another study, patients (n = 266) with a variety of impairments, who were all RESEARCH REPORT J Orthop Sports Phys Ther Volume 33 Number 12 December

3 TABLE 1. Representative responsiveness 18 data for patients receiving outpatient rehabilitation. Measure Study Impairment Statistic Overall Health Status 12 Mellion 24 Lumbar 0.83 ES, 0.87 SRM SF-12 PCS 35 Mellion 24 Lumbar 0.86 ES, 0.79 SRM SF-36 Physical Functioning Scale 38 Mellion 24 Lumbar 0.69 ES, 0.75 SRM Hart et al 13 Wrist or hand ~0.60 ES Jette and Jette 16 Lumbar or cervical ~0.70 ES Jette and Jette 15 Knee ~0.80 ES SF-36 Role Physical Scale 38 Jette and Jette 16 Lumbar or cervical ~0.40 ES Jette and Jette 15 Knee ~0.50 ES SF-12 Role Physical Scale 34 Hart et al 13 Wrist or hand ~1.0 ES SF-36 Bodily Pain Scale 38 Jette and Jette 16 Lumbar or cervical ~0.80 ES Jette and Jette 15 Knee ~0.90 ES Hart et al 13 Wrist or hand ~1.1 ES Abbreviations: ES, effect size 4 ; SRM, standardized response mean 30 ; PCS, SF-12 Physical Component Summary scale Journal of Orthopaedic & Sports Physical Therapy receiving workers compensation benefits, reported more improvement in SF-12 PCS scores when in acute work rehabilitation as compared to those who received work conditioning; and patients with acute symptoms reported more gain in SF-12 PCS scores as compared to patients with more chronic symptoms. 12 In another large sample of patients (n ) receiving outpatient rehabilitation, there was more gain in SF-12 PCS and MCS scores over a rehabilitation episode for those who exercised regularly before rehabilitation, as compared to those who did not exercise. 10 Internal consistency of the items in constructs with 2 or more items has been reported ( = ), 10,12 but no data have been published describing test-retest reliability of the measures from the 24-item instrument. Therefore, the purpose of this study was to assess the test-retest reliability of the Overall Health Status measure and the Physical and Mental Component Summary measures of the 24-item instrument. The reliability coefficients will be used to estimate measurement error and minimal detectable change for each of the 3 measures. If the new instrument is reliable, the reduced administrative burden may justify its use over the longer SF-36 for patients seeking outpatient rehabilitation. METHODS Subjects Subjects representing a sample of convenience were 71 consecutive patients referred to 2 physical therapy clinics participating with Focus on Therapeutic Outcomes, Inc., in 1999 (sample 1, n = 33; sample 2, n = 38). Patients had a variety of orthopaedic impairments (Table 2). Because diagnoses for patients with spinal syndromes (51% of the sample) have not been shown to be valid, 28 the body parts treated are listed (Table 2). Representative diagnoses by ICD-9-CM code 8 included disorders of joints, joint pain, spondylosis, spondylolisthesis, intervertebral disc disorders of cervical or lumbar spines, cervical or lumbar spinal stenosis, cervicalgia, lumbago, sciatica, backache, adhesive capsulitis of the shoulder, bicipital tenosynovitis, and sprain/strain of the neck, lumbar spine, ankle, or wrist. Patients were 14 years or older, 37 read English, and signed informed consent forms. The Focus on Therapeutic Outcomes, Inc., Institutional Review Board for the Protection of Human Subjects approved the study. Outcomes Measures Twenty-four health status questions (Appendix), which have been previously described, 12 were used to calculate 3 health status measures: an Overall Health Status measure and 2 component summary measures: SF-12 PCS and MCS scales Items included were all 12 items from the SF ; 8 additional items from the SF-36 PF not included in the SF-12 items; 3 physical functioning questions pertinent to patients with upper-extremity impairments 9 ; and 1 additional item from the SF-36 Bodily Pain scale 38 not included in the SF-12 items. In the final tool, 13 items represented the physical functioning construct (PF- 13) originally described by Hart. 9,12 The new physical functioning items pertain to lifting overhead to a cabinet, gripping or opening a can, and handling small items such as a pen or coins. Response categories and item introductions were identical to the SF-36 PF The internal consistency of the 13 physical functioning items has been reported for patients who were participating in an industrial rehabilitation program and receiving workers compensation benefits (n = 266): intake ( =.85) and discharge ( =.89). 12 Results from that same study supported known groups construct validity of the PF-13, where patients receiving acute work rehabilitation reported greater improvement in physical functioning as compared to those receiving work conditioning. 12 The 24 health status questions, which originate from the SF-36, 38 SF-12, 35 and clinical input, 9 can be used to calculate an overall measure of health status. Because the entire set of questions from the SF-12 are 736 J Orthop Sports Phys Ther Volume 33 Number 12 December 2003

4 TABLE 2. Demographic characteristics of patients. Characteristic All Patients (n = 71) Chronic Symptoms (n = 18) Age (y) Mean * SD Range Sex Male 25 6 Female Time since onset (wk) Site of problem Cervical spine 18 4 Lumbar spine 18 7 Thoracic spine 2 1 Hip 3 Thigh 2 Knee 9 1 Ankle 2 1 Foot 1 1 Shoulder 9 2 Elbow 4 1 Hand 3 Surgery None Missing 2 1 Time to retest (h) Type of referring physician General practitioner 45 8 Orthopedic surgeon 22 6 Neurologist 3 3 Podiatrist 1 1 Health transition Same or better than a year ago Worse than a year ago 26 7 * Patients with chronic symptoms were older than patients with less than chronic symptoms (t df = 29.4 = 3.7, P =.001) Chi-square test not significant for patients with chronic symptoms compared to patients with less than chronic symptoms. Chi-square test significant for patients with chronic symptoms compared to patients with less than chronic symptoms (see text). embedded within the total set of questions, the SF-12 PCS and MCS can be calculated. 35 To calculate the Overall Health Status measure, the response category selected by the patient for each of the 24 questions was transformed into a score from 0 to 100, following published algorithms. 9,38 For example, response categories for the moderate activities item are yes, limited a lot, yes, limited a little, and no, not limited at all, and their respective transformed scores are 0, 50, and A score of 0 represented low functioning and 100 represented high functioning. If there were 2 or more questions per construct, transformed question scores were averaged to create 1 score for each of the 8 constructs: general health (1 item), 35 physical functioning (13 items), 9,38 bodily pain (2 items), 38 role functioning (2 items), 35 mental health (2 items), 35 role emotional (2 items), 35 vitality (1 item), 35 and social functioning (1 item). 35 The Overall Health Status measure (0-100) was calculated by averaging the 8 construct scores. To calculate the PCS and MCS, responses to appropriate questions were selected and published algorithms 35 were followed. Component summary measures represent weighted composite scores that are scaled to have a mean of 50 and a standard deviation of 10 for the general population of the United States. 35 Higher scores represent higher levels of functioning. Score ranges of 10 to 70 have been published. 35 Procedures Support staff administered the health status questionnaire immediately preceding initial evaluation as part of routine outcomes assessment procedures. Following initial evaluation, and possibly initial treatment, if the therapist wanted the patient to return for another visit, the patient was directed to the office staff to schedule an appointment. After the appointment was scheduled, the patient was approached by support staff and asked if he/she would like to participate in the study. Staff explained the study to the patient, and if the patient agreed to participate, the patient provided signed consent. When the patient returned for the second appointment, the questionnaire was readministered by support staff prior to treatment. In this way, selection bias by the treating therapist was minimized. The time between administering the first and second questionnaires was 1 to 3 days (Table 2). Because the patients needed to complete the retest questionnaires within 24 to 72 hours, only new patients who were evaluated on Monday, Tuesday, or Wednesday, and who were scheduled for more visits, were solicited. Data Analysis Descriptive statistics were used to characterize patients. Differences between patients with chronic symptoms (time since onset more than 24 weeks) and patients without chronic symptoms (time since onset less than or equal to 24 weeks) were assessed with 2-group t tests or chi-square statistics with standardized deviates (ie, [observed expected]/[square root of expected]). There is no standard cut point for chronicity of symptoms. Although some may argue that other factors, such as surgical intervention or severity of symptoms, affect the definition of chronicity of symptoms, confirmable, generally accepted, RESEARCH REPORT J Orthop Sports Phys Ther Volume 33 Number 12 December

5 reliable, and valid operational definitions have not been advanced. The cut point of 24 weeks was selected for chronicity of symptoms to improve the probability that the symptoms were stable. ICC 2,1, which uses a 2-way ANOVA, was used to estimate test-retest reliability with 1-sided lower-limit 95% confidence intervals. 27 Only 1-sided lower-limit confidence intervals were calculated to provide worst case values of agreement estimates. 2,27 Because many patients, with acute musculoskeletal impairments are expected to change rapidly, intraclass correlation coefficients (ICCs) were calculated for the Overall Health Status measure and the 2 summary measures for the entire sample of 68 patients with complete data and for 18 patients with chronic symptoms ( 24 weeks postonset) as determined by their time since onset. To test whether measures had changed over testing time, a patient-specific change (readministration minus initial) for each measure was calculated. The difference in the change scores compared to 0 (Ho: mean = 0) was tested with 1-sample t tests with Bonferroni adjustments and retrospective power (power = 1 ) analyses 4 for all patients, regardless of time since onset, and for patients with symptoms greater than 24 weeks. Following a process previously described, 2,31 the standard error of measurement (SEM) with 1-sided upper 95% confidence interval (CI) limits 32 was estimated twice using the reliability estimate for the entire sample and for the sample of patients with chronic complaints. The SEM estimate was calculated by multiplying the sample SD and the square root of 1 minus the reliability estimate (ICC). 23 The SEMs were used to estimate the minimal detectable change (MDC) at the 90% CI by multiplying the SEM by the z value for the 90% confidence level (z = 1.65) and multiplying this value by the square root of 2. 2,22,31 A 90% CI was selected because it is common in the literature and will be used to compare current results directly to previous findings. 2,29 The alpha level was set at.05 for these analyses. TABLE 3. RESULTS Descriptive Statistics Patient characteristics are in Table 2. Patients with chronic symptoms ( 24 weeks) as compared to patients with less-than-chronic symptoms ( 24 weeks) received more surgery (Chi-square = 19.4, df =4,P =.001, standardized deviate 2.2 for 1 surgery and 2.1 for 2 surgical procedures). There were more patients with chronic symptoms referred to physical therapy by neurologists (Chi-square = 13.2, df =3,P =.004, standardized deviate 2.6 for neurologists) compared to patients with less than chronic symptoms. Descriptive statistics for the health status measure and the component summary measures are in Table 3. Reliability and Minimal Detectable Change Three patients did not have complete data for calculation of reliability estimates. Test-retest reliability estimates ranged between 0.82 and 0.92, and were almost identical between the entire sample and patients with chronic symptoms (Table 4). The SEMs ranged from 4 to 6 points for all measures. Estimates of minimal detectable change were ±12 and ±14 Overall Health Status measure points on a 100-point scale for the patients with chronic symptoms and the entire sample, respectively (90% CI). Estimates of minimal detectable change for summary measures were ±9 points (90% CI). Possible range of SF-12 summary measures is not intuitively obvious because SF-12 data are transformed using a t score transformation. 35 Combined range of the PCS and MCS (18-71 [Table 3]), is not dissimilar to published ranges of approximately 60 SF-12 points. 35 Change Over Testing Time Health status, as measured by the Overall Health Status and Physical Summary measures, improved over the time between testing administrations for the Descriptive data for Overall Health Status measure and Physical and Mental Component Summary measures. Measure Sample Size Mean SD Median Minimum Maximum Skewness (SE) Kurtosis (SE) Overall health Initial (0.29) 0.95 (0.56) Retest (0.29) 0.88 (0.57) PCS* Initial (0.29) 0.57 (0.56) Retest (0.29) 0.56 (0.57) MCS* Initial (0.29) 0.44 (0.56) Retest (0.29) 0.51 (0.57) Abbreviations: PCS, Physical Component Summary measure 35 ; MCS, Mental Component Summary measure. 35 *Component summary scores are scaled to have a mean of 50 and SD of 10 and are standardized to have the same mean as SF-36 summary scores in the general population of the United States J Orthop Sports Phys Ther Volume 33 Number 12 December 2003

6 TABLE 4. Reliability coefficients and measurement properties of the Overall Health Status measure and Physical and Mental Component Summary measures. Measure ICC 2,1 * SEM MDC Overall Health Status Entire sample (n = 68) 0.90 (0.84) 6 (7.1) 14 Patients with chronic 0.92 (0.85) 5 (7.6) 12 PCS Entire sample (n = 68) 0.82 (0.72) 4 (4.6) 9 Patients with chronic 0.82 (0.68) 4 (5.8) 9 MCS Entire sample (n = 68) 0.85 (0.77) 4 (4.7) 9 Patients with chronic 0.85 (0.74) 4 (5.9) 9 Abbreviations: PCS, Physical Component Summary measure 35 ; MCS, Mental Component Summary measure 35 ; ICC, intraclass correlation coefficient 27 ; SEM, standard error of measurement 2 ; MDC, minimal detectable change. 2 *Lower 1-sided 95% CI (confidence interval) shown in parentheses. 27 Upper 1-sided 95% CI shown in parentheses. 27 Journal of Orthopaedic & Sports Physical Therapy TABLE 5. Change in Overall Health Status measure and Physical and Mental Component Summary measures over testing times. Measure Mean (SD) t Value* P Power Overall Health Status Entire sample (n = 68) 3.4 (7.9) Patients with chronic 2.6 (7.2) PCS Entire sample (n = 68) 2.0 (5.2) Patients with chronic 1.3 (5.7) MCS Entire sample (n = 68) 0.9 (5.6) Patients with chronic 0.2 (6.0) Abbreviations: PCS, Physical Component Summary measure 35 ; MCS, Mental Component Summary measure. 35 *1-sample t test with Ho: mean = 0. Bonferroni-adjusted probability. Power, 1 4. entire sample, but not for those patients with chronic symptoms (Table 5). The Mental Summary measure did not change over time, regardless of patient sample. DISCUSSION Results support test-retest reliability of the Overall Health Status measure and the Physical and Mental Component Summary measures. These results were not unexpected because previously published reliability coefficients were good for the summary measures and for questions used to calculate the Overall Health Status measure. 38 Ware et al 34 have reported test-retest reliability of the SF-12 PCS and MCS to be 0.89 and 0.76, respectively. In that study, data for test-retest reliability estimates were calculated from repeated administrations of the SF-36, in which the 12 questions used to calculate the SF-12 PCS and MCS were embedded. Subjects came from 2 subsets of people: the general United States population participating in the National Survey of Functional Health Status, and 1 from the United Kingdom general population. 3 The 2 testing administrations were taken 2 weeks apart and reliability coefficients were calculated from product-moment correlation coefficients. 34 In the current study, test-retest reliability coefficients (ICC 2,1 ) 27 of the PCS and MCS administered as 12 questions embedded in the outcomes tool were 0.82 and 0.85, respectively (Table 4). Results of these studies support internal consistency and test-retest reliability coefficients reported from administration of the SF ,37-38 and SF-12, and from administration of the same questions embedded in the 24-item outcomes tool. Results presented here add to the body of literature that quantifies reliability of Physical and Mental Component Summary measures and presents test-retest reliability of a new composite measure of health status. Results also support the concept that items from instruments with reliable measures can produce reli- RESEARCH REPORT J Orthop Sports Phys Ther Volume 33 Number 12 December

7 able measures when those items are embedded in different instruments, although reliability of the new measures should be tested, not taken for granted. Current results enhance understanding of error associated with measures of patient self-report of health status. The MDC represents an estimate of reliability of a scale in terms of measurement error in the same units as the original scale, above which a change in scale measurement must be for the change to represent more than measurement error for an individual patient. 32 The MDC for the Overall Health Status measure was ±12 out of 100 health status scale points, or 12% of the scale range. MDCs for the PCS and MCS were ±9 summary scale points, which equates to 15% of the scale range using 60 as the range for summary scores. 35 This is not dissimilar to other health status measures. Estimated MDC of the Lower Extremity Functional Scale (LEFS) is ±9 out of 80 LEFS scale points, or 11% of the scale range. 2 As another comparison, standard error of measurements published for the Back Pain Functional Scale (BPFS) (SEM = 3.9) and the Roland-Morris Low Back Pain Questionnaire (RMQ) (SEM = 2.1) 29 equate to MDC estimates of ±9 out of 60 BPFS scale points and ±5 out of 24 RMQ scale points. These MDCs represent 15% and 20% of the BPFS and RMQ scale ranges, respectively. All MDCs were for 90% CIs. As more researchers publish error estimates, clinicians and researchers will have more data from which to compare outcomes instruments. Results allow comparison of 2 reliability coefficients (ICC and SEM, and a group assessment of change over time [t test]). The ICC is a reliability coefficient that assesses measurement agreement 27 and provides information about a measure s ability to differentiate among patients 32 through estimates of true patient variance divided by total patient variance. 32 The SEM expresses measurement error in the same units as the original measurement 32 without being influenced by variability among patients. 32 Measures studied here demonstrated adequate test-retest reliability as assessed by the ICC and SEM statistics (Table 4): the ICC for assessing measurement agreement influenced by patient variance, and the SEM for assessing measurement error not influenced by variability among patients. Including both estimates of reliability strengthens the interpretation of results for all health status measures. Assessment of whether measures changed over test-retest times (t test results) represents a group assessment, not patient-specific single-point-in-time reliability like the ICCs and SEMs. The mean differences over testing times (Table 5) were small but significant. The Overall Health Status mean difference for the entire sample was the largest (3 units), but Physical Summary measure difference (2 units) was similar demonstrating the 24-item and physical summary scale measures were sensitive to small changes over the short testing period. Time intervals should be small enough to not have true clinical differences occur because of natural history or effects of treatment and evaluation, but long enough to reduce the influence of recall of previous responses to questions. Results of t tests demonstrated groups of patients without chronic symptoms changed over time, as assessed with the Overall Functional Health Status measure and Physical Summary score compared to groups of patients with chronic symptoms. Therefore, findings support the nonchronic sample changed in health status over the testing time, which is clinically logical. Previous studies reported good responsiveness of the Overall Health Status measure 24 and PCS in patients receiving outpatient rehabilitation. The fact that the measures demonstrated adequate patient-specific single-point-in-time reliability, even with the nonchronic sample improves over testing times, supports that the measures are reliable. There are important differences between comparisons of estimates of measurement error for groups of patients versus individual patients, which should be considered when interpreting these findings. 7 First, patient-specific point estimates (ICCs and SEMs), commonly are associated with larger measurement errors compared to group analyses of change scores. 7 Second, although the ICC, SEM, and the t test of test-retest differences are related, they express different information about the reliability of the measures. The ICC is influenced by variation in subjects, number of replications, and testing times as well as error. The SEMs are affected by measurement error. 32 The t test on measure differences was influenced by error and variances in subjects, testing times, and score differences. For measures of health status to be useful as research and clinical tools, measures must not only be reliable; they must be valid, responsive to clinical change occurring over time, 2,5,18,29 and efficient. Results of this study support good reliability, while results of previous studies support construct validity 10-12,24 and responsiveness 10-12,24 of the new 24-item measure and the Physical and Mental Component Summary measures for patients seeking outpatient rehabilitation. The 24-item instrument and the summary scales, as compared to the SF-36, have reduced administrative burden. Direct comparative studies between the SF-36 and the 24-item instrument are needed to assess differences in responsiveness of all constructs for specific patient samples in rehabilitation. Limitations The new 24-item instrument had items specific for patients with upper-extremity impairments, but the sample did not permit analyses specific to those patients. The new instrument contained more items specific to physical functioning, as compared to the 740 J Orthop Sports Phys Ther Volume 33 Number 12 December 2003

8 SF-36 physical functioning scale or the SF-12 physical summary scale. The largest ICCs were estimated for the 24-item and physical summary measures. Although these data tend to support the ability of the 24-item instrument to differentiate change in physical functioning better than the SF-12 physical summary score, further study with stronger statistical designs is needed to study the discriminant validity of these instruments. Whether the additional physical and psychosocial items in the larger instrument are warranted awaits future study. Chronicity of symptoms was operationally defined as 24 weeks of symptom duration without any recognition of the effect of other defining characteristics (ie, surgical history or severity of symptoms). Controlling for these characteristics in a reliable and valid manner may affect reliability, validity, and responsiveness of the outcomes instruments. Future studies should explore larger sample sizes with a varied set of cut points or combination of characteristics defining chronicity or stability of symptoms. CONCLUSION Test-retest reliability was reported for 3 measures of patient self-report of health status that can be estimated using a new 24-item outcomes instrument. Results support reliability of the measures, demonstrate error estimates comparable to recent outcomes tools, and demonstrate differences between point estimates of population values and group assessments. Results support good reliability coefficients can be obtained while reducing administrative burden associated with data collection when questions from reliable patient self-report of health status questionnaires are embedded in different outcomes tools. ACKNOWLEDGMENTS The author would like to thank Mr. Al Amato, PT, MBA, and Ms. Mary Jo Marino, PT, for their data collection efforts, and Mr. Paul W. Stratford, PT, MSc, for his manuscript review, statistical oversight, and recommendations. REFERENCES 1. Angst F, Aeschlimann A, Steiner W, Stucki G. Responsiveness of the WOMAC osteoarthritis index as compared with the SF-36 in patients with osteoarthritis of the legs undergoing a comprehensive rehabilitation intervention. Ann Rheum Dis. 2001;60: Binkley JM, Stratford PW, Lott SA, Riddle DL. The Lower Extremity Functional Scale (LEFS): scale development, measurement properties, and clinical application. North American Orthopaedic Rehabilitation Research Network. Phys Ther. 1999;79: Brazier JE, Harper R, Jones NM, et al. Validating the SF-36 health survey questionnaire: new outcome measure for primary care. BMJ. 1992;305: Cohen J. Statistical Power Analysis for the Behavior Sciences. New York, NY: Academic Press, Inc.; Deyo RA, Centor RM. Assessing the responsiveness of functional scales to clinical change: an analogy to diagnostic test performance. J Chronic Dis. 1986;39: Di Fabio RP, Boissonnault W. Physical therapy and health-related outcomes for patients with common orthopaedic diagnoses. J Orthop Sports Phys Ther. 1998;27: Goldsmith CH, Boers M, Bombardier C, Tugwell P. Criteria for clinically important changes in outcomes: development, scoring and evaluation of rheumatoid arthritis patient and trial profiles. OMERACT Committee. J Rheumatol. 1993;20: Hart AC. Premier ICD-9-CM Code Book. Reston, VA: Anthony Publishing; Hart DL. Assessment of unidimensionality of physical functioning in patients receiving therapy in acute, orthopedic outpatient centers. J Outcome Meas. 2000;4: Hart DL, Dobrzykowski EA. Impact of exercise history on health status outcomes in patients with musculoskeletal impairments. In: C. Schunk eds. Orthopaedic Physical Therapy Clinics of North America, Functional Outcomes. Philadelphia, PA: WB Saunders Company; 2000: Hart DL. Orthotics and Prosthetics National Office Outcomes Tool (OPOT): initial reliability and validity assessment for lower extremity prosthetics. J Prosthet Orthot. 1999;11: Hart DL. The power of outcomes: FOTO Industrial Outcomes Tool-Initial assessment. Work. 2001;16: Hart DL, Tepper S, Lieberman D. Changes in health status for persons with wrist or hand impairments receiving occupational therapy or physical therapy. Am J Occup Ther. 2001;55: Hawker G, Melfi C, Paul J, Green R, Bombardier C. Comparison of a generic (SF-36) and a disease specific (WOMAC) (Western Ontario and McMaster Universities Osteoarthritis Index) instrument in the measurement of outcomes after knee replacement surgery. J Rheumatol. 1995;22: Jette DU, Jette AM. Physical therapy and health outcomes in patients with knee impairments. Phys Ther. 1996;76: Jette DU, Jette AM. Physical therapy and health outcomes in patients with spinal impairments. Phys Ther. 1996;76: ; discussion Jette AM. Using health-related quality of life measures in physical therapy outcomes research. Phys Ther. 1993;73: Kirshner B, Guyatt G. A methodological framework for assessing health indices. J Chronic Dis. 1985;38: Martin DP, Engelberg R, Agel J, Swiontkowski MF. Comparison of the Musculoskeletal Function Assessment questionnaire with the Short Form-36, the Western Ontario and McMaster Universities Osteoarthritis Index, and the Sickness Impact Profile health-status measures. J Bone Joint Surg Am. 1997;79: McHorney CA, Ware JE, Jr., Lu JF, Sherbourne CD. The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Med Care. 1994;32: McHorney CA, Ware JE, Jr., Raczek AE. The MOS 36-Item Short-Form Health Survey (SF-36): II. Psycho- RESEARCH REPORT J Orthop Sports Phys Ther Volume 33 Number 12 December

9 metric and clinical tests of validity in measuring physical and mental health constructs. Med Care. 1993;31: Ottenbacher KJ, Johnson MB, Hojem M. The significance of clinical change and clinical change of significance: issues and methods. Am J Occup Ther. 1988;42: Portney LG, Watkins MP. Foundations of Clinical Research. Upper Saddle River, NJ: Prentice Hall Health; Resnik Mellion L. Clinician and organizational factors associated with outcomes effectiveness in the treatment of patients with low back pain [dissertation]. Nova Southeastern University, Ft. Lauderdale, Riddle DL, Lee KT, Stratford PW. Use of SF-36 and SF-12 health status measures: a quantitative comparison for groups versus individual patients. Med Care. 2001;39: Riddle DL, Stratford PW. Use of generic versus regionspecific functional status measures on patients with cervical spine disorders. Phys Ther. 1998;78: Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86: Spitzer WO. Diagnosis of the problem (the problem of diagnosis): scientific approach to the assessment and management of activity-related spinal disorders-a monograph for clinicians: report of the Quebec Task Force on Spinal Disorders. Spine. 1987;12:S16-S Stratford PW, Binkley JM, Riddle DL. Development and initial validation of the back pain functional scale. Spine. 2000;25: Stratford PW, Binkley JM, Riddle DL. Health status measures: strategies and analytic methods for assessing change scores. Phys Ther. 1996;76: Stratford PW, Binkley JM, Solomon P, Finch E, Gill C, Moreland J. Defining the minimum level of detectable change for the Roland-Morris questionnaire. Phys Ther. 1996;76: ; discussion Stratford PW, Goldsmith CH. Use of the standard error as a reliability index of interest: an applied example using elbow flexor strength data. Phys Ther. 1997;77: Taylor SJ, Taylor AE, Foy MA, Fogg AJ. Responsiveness of common outcome measures for patients with low back pain. Spine. 1999;24: Ware JE, Jr., Kosinski M, Keller SD. A 12-Item Short- Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34: Ware JE, Jr., Kosinski M, Keller SD. SF-12: How to Score the SF-12 Physical and Mental Health Summary Scales. Boston, MA: The Health Institute, New England Medical Center; Ware JE, Jr., Kosinski M, Keller SD. SF-36 Physical and Mental Health Summary Scales: A User s Manual. Boston, MA: The Health Institute, New England Medical Center; Ware JE, Jr., Sherbourne CD. The MOS 36-item shortform health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992;30: Ware JE, Jr., Snow KK, Kosinski M, Gandek B. SF-36 Health Survey Manual and Interpretation Guide. Boston, MA: The Health Institute, New England Medical Center; J Orthop Sports Phys Ther Volume 33 Number 12 December 2003

10 Appendix Journal of Orthopaedic & Sports Physical Therapy HEALTH STATUS QUESTIONS 12 Each question includes the response categories below and the transformed response values in parentheses. 9,38 General Health Construct (1 item) In general, would you say your health is *,38 : Excellent (100) Very good (75) Good (50) Fair (25) Poor (0) Physical Functioning Construct (13 items) 12,38 The following items are about activities you might do during a typical day. Does your health now limit you in these activities? If so, how much? Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports,38 3. Moderate activities, such as moving a table, pushing a vacuum cleaner, bowling, or playing golf *,38 4. Lifting or carrying groceries,38 5. Climbing several flights of stairs *,38 6. Climbing one flight of stairs,38 7. Bending, kneeling, or stooping,38 8. Walking more than a mile,38 9. Walking several hundred yards, Walking one hundred yards, Bathing or dressing yourself, Lifting overhead to a cabinet,9 13. Gripping or opening a can,9 14. Handling of small items such as a pen or coins,9 Role Physical Construct (2 items) 35 During the past week, have you had any of the following problems with your work or other regular daily activities as a result of your physical health? Accomplished less than you would like *,38 Yes (0) No (100) 16. Were limited in the kind of work or other activities *,38 Yes (0) No (100) Role Emotional Construct (2 items) 35 During the past week, have you had any of the following problems with your work or other regular daily activities as a result of your emotional problems (such as feeling depressed or anxious)? Accomplished less than you would like,38 Yes (0) No (100) 18. Didn t do work or other activities as carefully as usual,38 Yes (0) No (100) RESEARCH REPORT J Orthop Sports Phys Ther Volume 33 Number 12 December

11 Bodily Pain Construct (2 items) How much bodily pain have you had during the past week?,38 None (100) Very mild (80) Mild (60) Moderate (40) Severe (20) Very severe (0) 20. During the past week, how much did pain interfere with your normal work (including both work outside the home and housework)? *,38 Not at all (100) A little bit (75) Moderately (50) Quite a bit (25) Extremely (0) Mental Health Construct (2 items) 35 These questions are about how you feel and how things have been with you during the past week. For each question, please give the one answer that comes closest to the way you have been feeling. How much of the time during the past week Have you felt calm and peaceful?,38 All of the time (100) Most of the time (80) A good bit of the time (60) Some of the time (40) A little of the time (20) None of the time (0) 22. Have you felt downhearted and depressed?,38 All of the time (0) Most of the time (20) A good bit of the time (40) Some of the time (60) A little of the time (80) None of the time (100) Vitality Construct (1 item) 35 These questions are about how you feel and how things have been with you during the past week. For each question, please give the one answer that comes closest to the way you have been feeling. How much of the time during the past week Did you have a lot of energy?,38 All of the time (100) Most of the time (80) A good bit of the time (60) Some of the time (40) A little of the time (20) None of the time (0) Social Functioning Construct (1 item) During the past week, how much of the time has your physical health or emotional problems interfered with your social activities (like visiting with friends, relatives, etc.)?,38 All of the time (0) Most of the time (25) Some of the time (50) A little of the time (75) None of the time (100) * Used to calculate SF-12 PCS 35 and 24-item Overall Health Status measure 12 Used to calculate 24-item Overall Health Status measure 12 Used to calculate SF-12 MCS 35 and 24-item Overall Health Status measure J Orthop Sports Phys Ther Volume 33 Number 12 December 2003