Shoulder disorders affect

Transcription

1 The Penn Shoulder Score: Reliability and Validity Brian G. Leggin, PT, MS, OCS 1 Lori A. Michener, PT, PhD, ATC, SCS 2 Michael A. Shaffer, PT, MS, ATC, OCS 3 Susan K. Brenneman, PT, PhD 4 Joseph P. Iannotti, MD, PhD 5 Gerald R. Williams Jr, MD 6 Journal of Orthopaedic & Sports Physical Therapy Study Design: Psychometric evaluation of a cross-sectional survey. Objectives: The purpose of this study was to examine the psychometric properties of reliability and validity of the Penn Shoulder Score (PSS). Background: Shoulder outcome measures are used to assess patient self-report levels of pain, satisfaction, and function. The PSS is a 100-point shoulder-specific self-report questionnaire consisting of 3 subscales of pain, satisfaction, and function. This scale has been utilized in the literature. However, the measurement properties of reliability and validity, including responsiveness, of the PSS subscales and overall scale need to be established. Methods and Measures: Patients (n = 40) with shoulder disorders undergoing a course of outpatient physical therapy completed the PSS at initial visit and again within 72 hours to assess test-retest reliability. The Constant Shoulder Score (CSS) and the American Shoulder and Elbow Surgeons Shoulder Score (ASES) were also completed at the initial visit and compared to the PSS to assess convergent construct validity. A separate cohort of patients (n = 109) completed the PSS at initial visit and 4 weeks later. These scores were used to assess internal consistency and responsiveness. Results: Reliability analysis revealed a test-retest ICC 2,1 of 0.94 (95% CI, ). Internal consistency analysis revealed a Cronbach alpha of The standard error of measurement (SEM) was ± 8.5 scale points (based on a 90% CI) and the minimal detectable change (MDC) was ± 12.1 scale points (based on a 90% CI). The minimal clinically important difference (MCID) for improvement was 11.4 points. Pearson product moment correlation coefficients between the PSS and the CSS and ASES were 0.85 and 0.87, respectively. Responsiveness analysis revealed an effect size of 1.01 and a standardized response mean of Conclusions: This study has demonstrated that the PSS is a reliable and valid measure for reporting outcome of patients with various shoulder disorders. J Orthop Sports Phys Ther 2006;36: Clinical Specialist, University of Pennsylvania Medical Center, Presbyterian, Philadelphia, PA. 2 Assistant Professor, Virginia Commonwealth University, Medical College of Virginia Campus, Richmond, VA. 3 Clinical Specialist, University of Iowa Hospitals and Clinics, Iowa City, IA. 4 Adjunct Faculty, Department of Physical Therapy, Arcadia University, Glenside, PA. 5 Chairman, Department of Orthopaedics, Cleveland Clinic Foundation, Cleveland, OH. 6 Professor, Department of Orthopaedics, University of Pennsylvania Health System, Philadelphia, PA. This project was partially supported by a grant from the APTA Section on Research, Clinical Demonstration Project. The Institutional Review Board of the University of Pennsylvania approved the protocol for this study. Address correspondence to Brian Leggin, University of Pennsylvania Medical Center, Presbyterian, Medical Office Building, Suite 110, 39th and Market Streets, Philadelphia, PA brian.leggin@uphs.upenn.edu Key Words: outcome assessment, psychometrics, reliability, validity Shoulder disorders affect 7% to 27% of the general population and account for 1% to 2.5% of patients presenting to general medical practitioners. 38,61 Up to 53% of these patients are referred to physical therapy. 61 Consumers of healthcare are becoming more careful when it comes to making decisions regarding their care. Therefore providers must document efficacy of treatment, compare results of different forms of treatment, and compare their results to those reported by others. For these types of comparisons to exist, a standardized and widely used system must be available. Moreover, a chosen outcome tool must be proven reliable and valid. 32 Several generic and regionspecific outcome tools have been developed and used to document outcomes of patients with shoulder pathologies. 11,17,21,36,37,41,44,47,48, 52,58,62 Generic health status measures are designed to be applicable across a broad spectrum of diseases, conditions, and demographic and cultural subgroups. 5 They can be used to compare the 138 Journal of Orthopaedic & Sports Physical Therapy

2 impact of particular disease conditions on the overall health of the individual. Region- or condition-specific self-report measures are intended to assess disability and clinically important changes in disability within a specific group. Condition-specific questionnaires of the shoulder have demonstrated greater responsiveness than generic measures in patients with shoulder disorders. 4,39 Although, several shoulder self-report outcome tools are available, no one tool has been widely accepted and utilized. 22,51 The lack of universal acceptance could be attributed to several factors, including the wording of questions, specific content of the scale, allocation of point values, and lack of or inadequate measurement properties. 9 Shoulder self-report tools should measure components of pain, patient satisfaction, and function. 22 Many tools contain these components, but do not adequately assess each dimension. Several tools contain only 1 pain scale, which is not specific to arm position or level of activity. 11,17,47 Patients with shoulder disorders frequently complain of pain that varies depending upon the demands placed on the shoulder. The patient s satisfaction with the function of the shoulder is a dimension that is scored in only 1 existing shoulder scale. 17 This is an important dimension to consider because a patient may achieve what can be considered a good or excellent score, but remain unsatisfied with their ability to use the shoulder. Function and disability items are included in most, if not all, available shoulder self-report measures. However, the items included in this dimension differ in each scale. In addition, the available response options and scoring in many scales does not consider whether the patient performed the activity prior to their injury or surgery. 11,17,47 Therefore, a patient will be penalized points on a questionnaire if the patient does not or has never engaged in a particular activity. The Shoulder Pain and Disability Index (SPADI) allows patients to respond not applicable to up to 2 of the 8 items of its disability subscale. 48 However, this scale does not adequately measure occupational and recreational disability. 28 The psychometric properties of clinical measurement tools should be established so that they may be used with confidence in patient care or research. 8,24,26,32,55,60 Tools should demonstrate reliability by yielding consistent results in a group of clinically stable patients (repeatability) and demonstrate that the items on the questionnaire are strongly related to each other and therefore measure a single characteristic (internal consistency). 26,32,44 Validity refers to the extent to which a questionnaire measures what it is intended to measure and the meaningfulness of that questionnaire s score. 8,32 Responsiveness or longitudinal validity must also be established to demonstrate the ability of a measure to detect change over time. 19,55 For many of the existing shoulder scales, these properties have been examined. The Penn Shoulder Score (PSS) is a conditionspecific self-report measure that first became available in It is a 100-point scale that consists of 3 subscales, including pain, satisfaction, and function. The pain subscale consists of 3 pain items that address pain at rest, with normal activities and with strenuous activities (Appendix). All are based on a 10-point numeric rating scale with end points of no pain and worst possible pain. Points are awarded for each item by subtracting the number circled from the maximum of 10. Therefore, a patient is awarded 30 points for complete absence of pain. If a patient is not able to use the arm for normal or strenuous activities, 0 points are scored for that item. Patient satisfaction with shoulder function is also assessed with a 10-point numeric rating scale. The endpoints are not satisfied and very satisfied. A maximum of 10 points for this section indicates that the patient is very satisfied with the current level of function of their shoulder. The function subsection is based on a sum of 20 items, each with a 4-point Likert scale. The response options include 0 (can t do at all), 1 (much difficulty), 2 (with some difficulty), and 3 (no difficulty). A patient is awarded 60 points if all activities can be performed without difficulty. Because some items in this subscale may not be applicable to all patients, the response option did not do before injury is available. For scoring purposes, the total possible points for the function subscale is reduced by 3 when this option is circled. Scoring is based on a percentage of the total possible points. For example, a patient has a total score for the function subsection of 27 points. He or she responded did not do before injury for 2 items. Therefore, the total possible points would be 54 (60 6). The final function score would be calculated: 27 54=0.5,then0.5 60=patient s function subscale score of 30 points. The total PSS maximum score of 100 indicates high function, low pain, and high satisfaction with the function of the shoulder. The PSS can be used in the aggregate or each subscale individually. Most patients complete the scale in less than 10 minutes, and the clinician can typically calculate the final scores in less than 2 minutes. 43 Two previous studies examined specific attributes of the PSS. Cook and colleagues 12 examined the error associated with the function subscale and compared it to other shoulder scales at differing levels of function. Michener and colleagues 44 demonstrated convergent construct validity of the American Shoulder and Elbow Surgeons (ASES) score by reporting correlation with the PSS of The PSS has also been used as an outcome measure in 2 clinical trials involving patients with shoulder impingement syndrome and rotator cuff repair. 40,50 RESEARCH REPORT J Orthop Sports Phys Ther Volume 36 Number 3 March

3 TABLE 1. Characteristics of the patients in cohorts 1 and 2. Cohort 1 (n = 40) Cohort 2 (n = 178) n % n % Shoulder disorder Impingement/tendonitis Rotator cuff tear Instability Adhesive capsulitis/ frozen shoulder Proximal humerus fracture Acromioclavicular joint arthritis Glenohumeral joint arthritis Time since injury or surgery 1 mo mo mo mo Postoperative Yes No Dominant hand injured Yes No Work injury Yes No Regular sport participation Yes No For the PSS to be used with confidence when making decisions regarding patient care, a more rigorous examination of its measurement properties is needed. To date, the error associated with the function subscale of the PSS and the relationship of its overall score with the ASES score have been examined. 12,44 Therefore, there is a need for further study to examine the measurement properties of reliability and validity, including responsiveness, of the PSS subscales and overall scale. 5 The purpose of this study was to assess the psychometric properties of the PSS, including test-retest reliability, internal consistency, and validity. By demonstrating acceptable measurement properties, this scale can be used to make confident and accurate judgments regarding the status of individual and groups of patients. METHOD Subjects The Institutional Review Board of the University of Pennsylvania approved this study. Two separate cohorts of patients undergoing a course of outpatient physical therapy gave their informed consent to participate in this study. Cohort 1 (n = 40), 22 men (mean ± SD age, 42.4 ± 11.7 years) and 18 women (mean ± SD age, 54.8 ± 17.1 years) with a variety of shoulder disorders (Table 1), was used to determine test-retest reliability and convergent construct validity. Cohort 2 (n = 178), also with a variety of shoulder pathologies, was asked to complete the PSS at initial evaluation and 4 weeks later (Table 1). Internal consistency was calculated from the questionnaires obtained at the initial evaluation. One hundred nine (61%) patients completed the PSS at both the initial evaluation and 4 weeks later (Table 2). The data from 56 males and 53 females (mean ± SD age, 49.1 ± 15.3 years) were used to determine longitudinal validity of the PSS (Table 2). Sixty-nine patients completed the PSS at the initial visit, but failed to do so at follow-up. Of these 69 patients, 48 (70%) completed at least 4 weeks of physical therapy care and achieved their physical therapy goals. Eighteen of the 48 patients were given the PSS at 4-weeks, but the forms were not returned. The remaining patients (30) were not given the form at 4 weeks. Instrumentation In addition to the PSS, 2 other shoulder outcome tools, the ASES and Constant Shoulder Score (CSS) were used in this study. The ASES is a 100-point scale that consists of 2 dimensions: pain and activities of daily living. 47 There is 1 pain scale worth 50 points and 10 activities-of-daily-living questions also worth 50 points. 47 This scale was chosen because its measurement properties have been established and it was constructed by and endorsed by the American Shoulder and Elbow Surgeons. 44,47 Patients can complete the questionnaire in less than 5 minutes. 44 Constant and Murley described the CSS in It is a 100-point scoring system in which 35 points are derived from the patient s report of pain and function. The remaining 65 points are allocated from measurement of range of motion and strength. Testretest reliability of the pain scale and error estimates of the CSS total score have been reported. 10,13 The patient can complete the self-report section of the CSS in approximately 5 minutes and the clinician can complete the range of motion and strength portion in approximately 10 minutes. This scale has been used internationally for reporting outcome of various shoulder conditions. 34 In addition, the European Shoulder and Elbow Society requires results of clinical data to be reported using the CSS at its meetings. 33 Procedures To assess test-retest reliability, patients in cohort 1 were administered the PSS at the initial physical 140 J Orthop Sports Phys Ther Volume 36 Number 3 March 2006

4 TABLE 2. Characteristics of the patients in cohort 2 of those who completed the study and those who did not complete the study. Patients Who Completed Study (n = 109) Patients Who Did Not Complete Study (n = 69) Chi-Square Statistic n % n % P Value Shoulder disorder.22 Impingement/tendonitis Rotator cuff tear Instability Adhesive capsulitis/frozen shoulder Proximal humerus fracture Acromioclavicular joint arthritis Glenohumeral joint arthritis Time since injury or surgery.24 1 mo mo mo mo Postoperative.43 Yes No Dominant hand injured.54 Yes No Work injury.67 Yes No Regular sport participation.61 Yes No therapy evaluation and a second time within 72 hours of the initial assessment. Convergent construct validity is concerned with the extent to which a measure relates to other similar measures in a manner that is consistent with theoretically derived hypotheses concerning the constructs that are being measured. 32 Convergent validity was assessed with cohort 1 by determining the relationship between the PSS and 2 other self-report shoulder scales, the CSS and the ASES. 11,47 Longitudinal validity or responsiveness was assessed to determine the ability of the PSS to distinguish groups of patients whose status has improved, deteriorated, and remained stable. 19,55 Patients in cohort 2 completed the PSS at the initial physical therapy evaluation and again 4 weeks later. All patients received physical therapy treatment within that 4-week period. After 4 weeks, patients were asked to assess whether they believed their shoulder function had changed since the first visit. They were asked to assess their perceived level of change on a 5-point global rating of change scale with the following response options: much worse, slightly worse, stayed the same, slightly better, much better. 3,44 This question was used as the criterion for change in this study. Data Analysis Reliability To generalize results to other raters, intraclass correlation coefficient (ICC 2,1 ) was used to determine test-retest reliability of the PSS. 54 A coefficient alpha was used to determine internal consistency of the PSS collected from the initial visit of patients in cohorts 1 and Internal consistency is important for 3 reasons. 19 First, it refers to the extent which all items on a scale measures the same entity. 19,46,53 Second, it provides an index of a scale s ability to differentiate among clients at an instant in time. 19 Third, the internal consistency coefficient is used to calculate the standard error of measurement (SEM) associated with a measure s score at an instant in time. 19 Error and Change The error associated with a single application of the PSS was analyzed using the SEM. The SEM provides an estimate of how reliably a scale estimates an individual s true score, that is, the score that would be obtained for the person if the scale measured perfectly, without error. 16,42,49 The SEM is a representation of measurement error expressed in the same units as the original measurement. 6 SEM can be calculated by using either the Cronbach alpha statistic or the ICC. 19 Whereas error RESEARCH REPORT J Orthop Sports Phys Ther Volume 36 Number 3 March

5 TABLE 3. Penn shoulder score descriptive statistics for patients in cohort 2. Patients Who Completed Study (n = 109) Patients Who Did Not Complete Study (n = 69) Mean SD Min Max Mean SD Min Max Initial score* week score * t test comparison of initial scores for patients who did and did not complete the study (P =.40). Journal of Orthopaedic & Sports Physical Therapy estimates based on Cronbach s alpha are specific to an instant in time, error estimates derived from test-retest reliability can be generalized over a time interval equal to that applied in the test-retest reliability study. 19 Because it is likely that a clinician will be more interested in the error associated with an instant in time rather than over a short time interval, the SEM was calculated based on the alpha coefficient with the formula: SEM = SD 1 Cronbach. The SEM carries with it 68% confidence bounds. To achieve 90% confidence level, the SEM was multiplied by the z value associated with the 90% confidence level (z = 1.65). The error associated with multiple applications of the PSS was determined by calculating the minimal detectable change (MDC). This was calculated by using the formula: MDC = SD 1 ICC value The 90% CI was calculated for the MDC, which is the statistically minimal amount of change required in a PSS score to be 90% confident that true change has occurred. The minimal clinically important difference (MCID) is the smallest difference in a score that is considered to be worthwhile or important. 27 Nine different methods have been used to calculate the MCID for an instrument. 2,63 We chose a commonly used approach to MCID calculation in which the patient rates his/her change on a global-rating-ofchange scale. 2,29 The MCID was calculated by taking the mean change score of everyone who reported being slightly better. 2,29 Calculating single error and change estimates assumes that a scale measures all levels of outcome equally. Cook and colleagues 11 have demonstrated that the function subscale of 4 shoulder scales measured the midranges of function better than high and low shoulder function. Therefore, the conditional SEM, MDC, and MCID were calculated for those patients who scored 0 to 25, 26 to 50, 51 to 75, and 76 to 100 on the overall PSS. Validity To assess convergent construct validity, Pearson correlation coefficients were calculated to determine the relationships between the PSS, CSS, and ASES. Pearson correlation coefficients were also calculated to establish construct validity by determining the relationship of each subscale of the PSS at both initial and 4-week visits. Longitudinal validity or responsiveness of the PSS was determined by calculating both the effect size (ES) and standardized response mean (SRM). 30,35 The effect size is calculated by subtracting each patient s initial score from the final score to obtain a change score. The mean change score is then divided by the standard deviation of the initial scores. The SRM is calculated by dividing the mean of the patients score changes by the standard deviation of the change scores. It is suggested that an absolute value of 0.8 or greater for both methods indicates excellent responsiveness of the scale. 46 RESULTS Descriptive Statistics Table 3 provides the descriptive statistics for the PSS for those subjects who completed the questionnaire at initial visit and at 4 weeks. Reliability Cohort 1 (n = 40) demonstrated test-retest reliability of ICC 2,1 = 0.94 (95% CI, ). For each subscale, test-retest reliability was: pain ICC 2,1 = 0.88 (95% CI, ), satisfaction ICC 2,1 = 0.93 (95% CI, ), and function ICC 2,1 = 0.93 (95% CI, ). Table 4 provides the test-retest reliability of each pain and function item. The internal consistency analysis of the PSS revealed a Cronbach alpha of.93. Validity Pearson product moment correlation coefficients between the PSS and the CSS and the ASES were r = 0.85 and 0.87, respectively. Correlations of each subscale of the PSS to the total score at initial and 4-week visits are presented in Table 5. Patients in cohort 2 (n = 109) completed the PSS at both the initial evaluation and 4 weeks later, and were used to evaluate longitudinal validity. Descriptive statistics of the PSS total score and each of the subscales of those patients who rated their change on the global rating of change scale as slightly better or much better (n = 104) is presented in Table J Orthop Sports Phys Ther Volume 36 Number 3 March 2006

6 TABLE 4. Test-retest reliability (ICC 2,1 ) of the pain, satisfaction, and function subscales (and their individual items) of the Penn Shoulder Score. Subscales/Items ICC (95% CI) Pain subscale 0.88 ( ) Pain at rest 0.84 ( ) Pain with normal activities 0.82 ( ) Pain with strenuous activities 0.78 ( ) Satisfaction subscale 0.93 ( ) Function subscale 0.93 ( ) Reach small of back to tuck in shirt 0.70 ( ) Wash middle of back/hook bra 0.87 ( ) Perform necessary toileting 0.95 ( ) Wash back of opposite shoulder 0.80 ( ) Comb hair 0.87 ( ) Place hand behind head 0.72 ( ) Dress self 0.91 ( ) Sleep on affected side 0.97 ( ) Open a door 0.77 ( ) Carry a bag of groceries 0.81 ( ) Carry a briefcase/small suitcase 0.80 ( ) Place a soup can on shelf at shoulder 0.89 ( ) level Place a gallon container on shelf at 0.82 ( ) shoulder level Reach a shelf overhead 0.71 ( ) Place a soup can on a shelf overhead 0.87 ( ) Place a gallon container on a shelf 0.82 ( ) overhead Perform usual sport/hobby 0.89 ( ) Perform household chores 0.94 ( ) Throw overhand/swim/overhead 0.94 ( ) racquet sports Work full time at your regular job 0.96 ( ) TABLE 5. Correlation coefficient of each subscale to total score. Subscale Initial Visit (n = 109) Penn Shoulder Score 4-wk Visit (n = 109) Pain Satisfaction Function Effect size and standardized response mean of the PSS total score and subscales ranged from 0.80 to 1.27 (Table 6). Error and Change The error associated with the total PSS at a given point in time (SEM) is ±8.5 scale points (based on a 90% CI). The MDC at the 90% confidence level is ±12.1 scale points. The SEM (based on a 90% CI) for the individual subscales ranged from 1.3 to 6.1 scale points, and the MDC (based on a 90% CI) ranged from 1.8 to 8.6 scale points (Table 7). The MCID for patients who reported slight improvement is 11.4 scale points. The SEM, MDC, and MCID for various score ranges and the overall score are presented in Table 8. DISCUSSION Several shoulder outcome scoring tools are available. However, each tool contains deficiencies that prevent universal acceptance and utilization. 22,51 For an outcome tool to be used with confidence when making decisions regarding patient care, measurement properties of reliability and validity must be established. 8,24,26,32,44,55,60 The results of this study indicate that the PSS appears to be reliable and valid for use in patients with a variety of shoulder disorders. The PSS demonstrated excellent test-retest reliability. 46 Table 9 shows that a higher level of test-retest reliability has been reported for the Shoulder Rating Questionnaire and the Simple Shoulder Test (0.97 and 0.99), whereas others have demonstrated a lower level (range, ). 3,10,41,44,48 Internal consistency of the PSS ( =.93) indicates that the items within the scale measure the same construct. 46 For scales that are used as research tools to compare groups, Cronbach alpha values of.7 to.8 are regarded as satisfactory. 7 For clinical application, higher Cronbach alpha values are needed. The minimum desired value for clinical application has been reported to be.90, with =.95 more desirable. 7 Internal consistency of the PSS appears to be equal or better than some of the existing shoulder scales (range, ) (Table 9). 43,44,48,49 Self-report condition-specific outcome measures should address the areas of principal concern to the TABLE 6. Mean ± SD, effect size, and standardized response mean for the Penn Shoulder Score at initial visit and at 4 weeks for those subjects who reported improvement. Initial Score (n = 104) 4-wk Score Change Score Effect Size Standardized Response Mean Pain 16.7 ± ± ± Satisfaction 3.2 ± ± ± Function 29.5 ± ± ± Total score 49.4 ± ± ± RESEARCH REPORT J Orthop Sports Phys Ther Volume 36 Number 3 March

7 patient regarding his or her functional status. 44 These include aspects of pain, satisfaction, activities of daily living, as well as work and recreational activities. 22,44 The PSS evaluates these aspects, thus providing evidence for the face validity of this scale. Convergent construct validity of the PSS was supported by strong relationships, with 2 other shoulder outcome tools (r = 0.85 and 0.87). This is consistent with previous reports on the relationship among shoulder self-assessment measures. 4,44 A high positive correlation coefficient is desired when the scales being examined are, in theory, similar in nature. 43 The degree to which the instruments measure the same trait or quantity of a variable reflects the degree of convergent construct validity. 15 Pain is an important parameter that has been addressed by several shoulder scales and measured by various manners. 11,14,47,48,52 Many shoulder scales include only 1 pain question, which is not specific to activity or arm position. The PSS includes 3 separate numeric rating scales for the patient to assess pain associated with different levels of activity. Numeric rating scales have been demonstrated to be the most reliable among both literate and illiterate subjects. 18 The pain subscale of the PSS demonstrated excellent reliability (ICC = 0.88). Table 9 reveals that this value is similar to that reported for the pain subscales of other shoulder instruments (range, ). 13,44 The patient s satisfaction with the ability to use the shoulder is an important construct to measure for both the patient and clinician. 22 The PSS includes a satisfaction item in which the patient is able to rate satisfaction with their current level of shoulder function on a 10-point numeric rating scale. This item demonstrated excellent test-retest reliability (ICC = 0.93). To our knowledge, the UCLA Shoulder Score is the only other shoulder scale to address patient satisfaction. 17 This scale contains 1 item that dichotomizes the patient s response into satisfied and better and not satisfied and worse. 13,17 Limiting patient responses to these 2 options allows for only a gross estimate of the patient s level of satisfaction. 13 Cook and colleagues 13 reported an ICC value of 0.79 for this item in patients post shoulder surgery. However, they were unable to calculate a valid reliability statistic for a group of patients without surgery, partly because 8 of 24 respondents changed their satisfaction rating between the 2 test administrations. 13 The numeric rating scale of patient satisfaction appears to have yielded better reliability than the dichotomous response option of the UCLA scale. The function subsection demonstrated excellent test-retest reliability (ICC = 0.93). Table 9 shows that this reliability value is comparable to or better than that reported for the function subscale of other shoulder scales (range, ). 13,44 It is unclear from our data whether the response option did not TABLE 7. Standard error of measurement (SEM) and minimal detectable change (MDC) of each subscale of the Penn Shoulder Score. Both values were calculated based on a 90% confidence interval. SEM MDC Pain subscale Pain at rest Pain with normal activities Pain with strenuous activities Satisfaction subscale Function subscale TABLE 8. Standard error of measurement (SEM), minimal detectable change (MDC), and minimal clinically important difference (MCID) of the Penn Shoulder Score for various score ranges and overall score. SEM and MDC were calculated based on a 90% confidence interval. SEM* MDC* MCID ± SD 0-25 (n = 26) ± 12.3 (n = 5) (n = 67) ± 8.0 (n = 19) (n = 72) ± 9.1 (n = 21) (n = 13) ± 4.8 (n = 4) Overall score (n = 178) ± 9.5 (n = 49) * Calculated from all patients in cohort 2. Calculated only for those patients who reported becoming slightly better. do before injury contributed to or detracted from the reliability of this section. Both the effect size (1.01) and standardized response mean (1.27) were used to assess longitudinal validity or responsiveness of the PSS. Values greater than 1.0 are considered large, indicating good responsiveness. 44,46,57 Table 9 reveals that the standardized response mean for other shoulder scales has ranged from 0.59 to It should be noted that any comparison of shoulder scale responsiveness can only be inferred unless a head-to-head comparison of shoulder-specific measures is performed. 44 The PSS can be used by clinicians to make decisions regarding individual patients at initial evaluation, intermediate progress, and at the completion of a course of care. Our results indicate that the SEM of the PSS is ±8.5 scale points (based on a 90% CI) and the MDC is ±12.1 scale points (based on a 90% CI). A clinician can be 90% confident that an initial PSS score of 60 points actually falls within ±8.5 points. To determine whether true change has occurred in an individual patient s PSS score, the MDC can be used. For example, if that same patient who scored 60 points at the initial evaluation scores 75 points during a reassessment 4 weeks later, the clinician can be reasonably confident that the patient has demonstrated true improvement, because the change of 15 points is greater than the MDC value of To 144 J Orthop Sports Phys Ther Volume 36 Number 3 March 2006

8 TABLE 9. Characteristics and psychometric properties of shoulder scales. Scale Penn Shoulder Score (PSS) Dimensions (% of Total Score) Test-Retest Reliability (ICC) Internal Consistency Error Estimate Responsiveness Pain, 30% Satisfaction, 10% Function, 60% Range, Pain, 0.88 Satisfaction, 0.93 Function, 0.93 Total, 0.94 =.93 SEM (90% CI) Pain, ±3.8 Satisfaction, ±1.3 Function, ±6.1 Total, ±8.5 MDC (90% CI) Pain, ±5.2 Satisfaction, ±1.8 Function, ±8.6 Total, ±12.1 MCID, 11.4 SRM: 1.27 ES: 1.01 American Shoulder and Elbow Surgeons (ASES) 47 Pain, 50% Function, 50% Range, Pain, Function, Total, = SEM (90% CI) Pain, ± Function, ± Total, ± MDC (90% CI) Pain, ± Function, ± Total, ± MCID, SRM: 1.54, ES: Journal of Orthopaedic & Sports Physical Therapy Constant Shoulder Score 11 Pain, 15% Function, 20% Clinician assessment (ROM and strength), 65% Score range, Disabilities of the Arm, Shoulder, and Hand (DASH) 41 Shoulder Pain and Disability Index (SPADI) 48 Symptoms, 16.7% Disability, 83.3% Optional, sports, performing arts, or work module Score range, Pain, 50% Disability, 50% Score range, Shoulder Scale (UCLA) 17 Pain, 29% Function, 29% Satisfaction, 14% ROM, 14% Strength, 14% Score range, 0-35 Simple Shoulder Test (SST) 37 Function, 100% Score range, 0-12 Shoulder Rating Questionnaire 36 Global assessment, 15% Pain, 40% Daily activities, 20% Recreational/ athletic activities, 15% Work, 10% Total, Not tested SEM (95% CI), ± SRM: Total Score, = SEM, 7.6 (90% CI) 41 SRM: 1.13, 44 MDC, 12.8 (90% CI) Pain, ( ) 13 Disability, ( ) 13 Total, ( ) 13 Pain, Function, Satisfaction,* Pain, = Disability, = Total, = Not tested SEM (95% CI) 48 Pain, ±15.3 Disability, ±11.3 Total, ±9.3 MDC, not calculated SEM, not calculated MDC, not calculated SRM: 1.23, SRM: not calculated ES: not calculated ICC, = SEM, ± SRM: not calculated ES: not calculated Kappa = = SEM, not calculated MDC, not calculated SRM: RESEARCH REPORT Abbreviations: ES, effect size; ICC, intraclass correlation coefficient; MCID, minimal clinically important difference; MDC, minimal detectable change; SEM, standard error of measurement; SRM, standardized response mean. * Patients post surgery. Converted score range J Orthop Sports Phys Ther Volume 36 Number 3 March

9 further help with clinical decision making, we have calculated the SEM and MDC values for each of the subscales of the PSS in addition to the total score (Table 7). The reported SEM for other self-report scales presented in Table 9 are of similar magnitude as the PSS. The MDC for the PSS is also similar or slightly less than for other shoulder scales presented in Table 9. Cook and colleagues 12 calculated the measurement error of the function subscale of 4 shoulder scales for different levels of shoulder function. The 4 shoulder scores used in this study were the SST and the function subscales of the ASES, the SPADI, and the PSS. These investigators found that all scales measured the middle ranges of function with lower error, as compared to the high and low function levels. The PSS and SPADI measured function more consistently throughout all ranges than the ASES and SST. 12 The error associated with the PSS was nearly equal to the SPADI at middle-range scores, but was better than the other scales for high and low scores. 12 These authors concluded that the error associated with the function subscale of these self-report measures could vary among different levels of functioning. We calculated the SEM and MDC of the PSS total score at 4 different score ranges. Our results differed from that of Cook and colleagues 12 in that the error was lower at the low and high levels of function than at the mid ranges. There are 2 possible explanations for these conflicting results. First, we calculated error for the total score of the PSS, which included the pain and satisfaction subscales in addition to function. Therefore, pain and satisfaction may have aided the improved error estimates at the low and high functioning levels. Second, there were fewer patients at the extremes of function levels. This may have biased the results. Ceiling and floor effects do exist when patients score at the extremes of an outcome measure. 6 In the case of a ceiling effect, there is a restricted range for improvement because patients begin at the highest level on the scale. Conversely, in the case of a floor effect, there is a restricted range for deterioration in functional status. In this study, 1 patient scored 9, 1 patient scored 8, and 1 patient scored 92 at the initial visit. Because the MDC of the PSS is ±12.1, it appears that there is a slight potential for both ceiling and floor effects to exist for a small number of individuals. It is generally believed that small differences in self-report outcome tools may be statistically significant yet clinically unimportant. 27 Therefore, outcomes research is faced with the challenge of interpretability of the scores. 2,25,59 The concept of the MCID has been proposed to refer to the smallest difference in a score that is considered to be worthwhile or important. 27 Only 5 of the 109 (4.6%) patients in our study reported that they had stayed the same or become worse. Therefore, we were only able to calculate the MCID for those patients who said that they had become slightly better. The MCID for the overall PSS is 11.4 points and is to be used to interpret improvement only. The MCID of the PSS is slightly less than the MDC. This has been demonstrated with other self-report scales, indicating that the amount of change a patient perceives as important can be less than the amount deemed to be statistically significant. 20 A limitation of these results is the relatively small sample size available for this calculation. A larger sample size of patients who reported slight improvement may yield a slightly different MCID. Many studies have shown that retrospective report of change is associated with a larger prospective change for those with more room for change. 27 For example, Baker et al 1 found that self-reported physical health decrements for those who said they were somewhat worse ranged from 26 points (on a0to 100 scale) for people with high (81-100) to +3 points for people with low (0-20) baseline physical health. The results of our study support this theory in that the MCID for those patients who said they were slightly better ranged from 24.3 points for people with low (0-25) to 3.5 for people with high (76-100) baseline shoulder scores (Table 8). This phenomenon may reflect ceiling and floor effects (less change will be observed for people who are close to the top of the scale and are improving or near the bottom of the scale and declining), but it also may represent differences in the meaning of the scale along the underlying continuum. 27 We used the patient s global rating of change as the criterion for change in this study. Beaton, 3 Gummesson, 23 Heald, 28 Michener, 44 and Williams 64 have used the same or a similar scale as the standard for change in their studies to assess functional status measures. The use of a global rating of change has been questioned. 45,55 The reliability and validity of this global rating of change have not been established and a patient s recall of his or her previous condition may be inaccurate or biased. 20,44,45,55 Both self-report outcome tools and the global rating of change are measures that involve a patient s judgment; therefore, the errors of the global rating of change and the self-assessment scale are most likely correlated. 20,44,55 This potentially makes the global rating a biased measure of change. 44 However, several studies have provided evidence that the use of a global rating of change has the ability to differentiate changes in clinical status over time. 3,20,23,28,44 We were disappointed with the relatively high attrition rate (39%) in cohort 2. This may have biased our results toward patients who improved rather than those who remained the same or became 146 J Orthop Sports Phys Ther Volume 36 Number 3 March 2006

10 worse. To determine whether differences existed between patients who completed the study (n = 109) and patients who completed forms only during the initial visit (n = 69), a chi-square test was performed on the variables of shoulder disorder, duration of symptoms, surgical status, hand dominance, work injury status, and sport participation, and a t test was performed for shoulder score at the initial visit. No differences were demonstrated. In addition, a majority (70%) of the patients who did not complete the PSS at 4 weeks completed their course of physical therapy care and were either not given the PSS by their therapist or did not return the form. An interview with the therapists involved in the study revealed that some of the therapists simply forgot to administer the PSS at the 4-week mark. There is no explanation for why patients failed to return the form. Binkley (67%) and Heald (62%) have reported higher attrition rates in their studies aimed at validating outcome tools. 6,28 A vast majority of the patients who did complete the PSS on both occasions reported that they improved at the 4-week mark (104/109). Therefore, there were an insufficient number of patients to calculate responsiveness for patients who did not improve. This result prevented a more robust assessment of responsiveness to determine if the PSS has the ability to remain stable in unimproved patients. The PSS was developed to measure outcome of patients with a wide variety of shoulder disorders. The goal of this study was to establish measurement properties of this outcome measurement tool. Future research should concentrate on discriminant and divergent validity of the PSS as well as measurement properties in individual shoulder pathologies. Headto-head comparisons of the performance of shoulder outcome measures in patients with a specific shoulder disorder would help the clinician choose the best outcome tool for a specific disorder. A study utilizing item response theory to determine whether the weighting of items of the PSS is appropriate may also be useful. CONCLUSION This study has provided evidence for the use of the PSS as a region-specific shoulder measure for reporting outcome of patients with various shoulder disorders. Clinicians treating patients with similar shoulder diagnoses to those in this study can apply the measurement properties of the subscales and total score as presented in this study. The individual subscales and the PSS total score can be considered a reliable and valid measure that can be used confidently to assess outcome of both individuals and groups of patients with shoulder disorders. REFERENCES 1. Baker DW, Hays RD, Brook RH. Understanding changes in health status. Is the floor phenomenon merely the last step of the staircase? Med Care. 1997;35: Beaton DE, Boers M, Wells GA. Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research. Curr Opin Rheumatol. 2002;14: Beaton DE, Richards RR. Assessing the reliability and responsiveness of 5 shoulder questionnaires. J Shoulder Elbow Surg. 1998;7: Beaton DE, Richards RR. Measuring function of the shoulder. A cross-sectional comparison of five questionnaires. J Bone Joint Surg Am. 1996;78: Binkley JM. Functional outcome measures for the shoulder. In: Butler DS, Jones MA, eds. Evaluation and Treatment of the Shoulder: An Integration of the Guide to Physical Therapist Practice. Philadelphia, PA: F.A. Davis Co; Binkley JM, Stratford PW, Lott SA, Riddle DL. The Lower Extremity Functional Scale (LEFS): scale development, measurement properties, and clinical application. North American Orthopaedic Rehabilitation Research Network. Phys Ther. 1999;79: Bland JM, Altman DG. Cronbach s alpha. BMJ. 1997;314: Bombardier C, Tugwell P. Methodological considerations in functional assessment. J Rheumatol Suppl. 1987;14 Suppl 15: Bot SD, Terwee CB, van der Windt DA, Bouter LM, Dekker J, de Vet HC. Clinimetric evaluation of shoulder disability questionnaires: a systematic review of the literature. Ann Rheum Dis. 2004;63: Conboy VB, Morris RW, Kiss J, Carr AJ. An evaluation of the Constant-Murley shoulder assessment. J Bone Joint Surg Br. 1996;78: Constant CR, Murley AH. A clinical method of functional assessment of the shoulder. Clin Orthop Relat Res. 1987; Cook KF, Gartsman GM, Roddey TS, Olson SL. The measurement level and trait-specific reliability of 4 scales of shoulder functioning: an empiric investigation. Arch Phys Med Rehabil. 2001;82: Cook KF, Roddey TS, Olson SL, Gartsman GM, Valenzuela FF, Hanten WP. Reliability by surgical status of self-reported outcomes in patients who have shoulder pathologies. J Orthop Sports Phys Ther. 2002;32: Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16: Currier D. Elements of Research in Physical Therapy. 3rd ed. Baltimore, MD: Williams & Wilkins; Diamond JJ. A practical application of reliability theory to family practice research. Fam Pract Res J. 1991;11: Ellman H, Hanker G, Bayer M. Repair of the rotator cuff. End-result study of factors influencing reconstruction. J Bone Joint Surg Am. 1986;68: Ferraz MB, Quaresma MR, Aquino LR, Atra E, Tugwell P, Goldsmith CH. Reliability of pain scales in the assessment of literate and illiterate patients with rheumatoid arthritis. J Rheumatol. 1990;17: Finch E, Brooks D, Stratford P, Mayo N. Physical Rehabilitation Outcome Measures: A Guide to Enhanced Clinical Decision Making. Hamilton, Ontario: Canadian Physiotherapy Association; RESEARCH REPORT J Orthop Sports Phys Ther Volume 36 Number 3 March

11 20. Fritz JM, Irrgang JJ. A comparison of a modified Oswestry Low Back Pain Disability Questionnaire and the Quebec Back Pain Disability Scale. Phys Ther. 2001;81: Gartsman GM, Brinker MR, Khan M, Karahan M. Self-assessment of general health status in patients with five common shoulder conditions. J Shoulder Elbow Surg. 1998;7: Gerber C. Integrated scoring systems for the functional assessment of the shoulder. In: Masten F, Fu F, Hawkins R, eds. The Shoulder: A Balance of Mobility and Stability. Rosemont, IL: American Academy of Orthopaedic Surgeons; Gummesson C, Atroshi I, Ekdahl C. The disabilities of the arm, shoulder and hand (DASH) outcome questionnaire: longitudinal construct validity and measuring self-rated health change after surgery. BMC Musculoskelet Disord. 2003;4: Guyatt GH, Bombardier C, Tugwell PX. Measuring disease-specific quality of life in clinical trials. CMAJ. 1986;134: Guyatt GH, Cook DJ. Health status, quality of life, and the individual. JAMA. 1994;272: Guyatt GH, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987;40: Hays RD, Woolley JM. The concept of clinically meaningful difference in health-related quality-of-life research. How meaningful is it? Pharmacoeconomics. 2000;18: Heald SL, Riddle DL, Lamb RL. The shoulder pain and disability index: the construct validity and responsiveness of a region-specific disability measure. Phys Ther. 1997;77: Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10: Kazis LE, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Med Care. 1989;27:S Kirkley A, Griffin S, McLintock H, Ng L. The development and evaluation of a disease-specific quality of life measurement tool for shoulder instability. The Western Ontario Shoulder Instability Index (WOSI). Am J Sports Med. 1998;26: Kirshner B, Guyatt G. A methodological framework for assessing health indices. J Chronic Dis. 1985;38: Kuhn JE, Blasier RB. Measuring outcomes in shoulder arthroplasty. Semin Arthroplasty. 1995;6: Leggin BG, Lannotti J. Shoulder outcome measurement. In: Lannotti JP, Williams GR, eds. Disorders of the Shoulder: Diagnosis and Management. Philadelphia, PA: Lippincott, Williams & Wilkins; 1999: Liang MH, Fossel AH, Larson MG. Comparisons of five health status instruments for orthopedic evaluation. Med Care. 1990;28: L Insalata JC, Warren RF, Cohen SB, Altchek DW, Peterson MG. A self-administered questionnaire for assessment of symptoms and function of the shoulder. J Bone Joint Surg Am. 1997;79: Lippitt SB, Harryman DT, Matsen FA. A practical tool for evaluating function: the simple shoulder test. In: Matsen FA, Fu FH, Hawkins RJ, eds. The Shoulder: A Balance of Mobility and Stability. Rosemont, IL: American Academy of Orthopaedic Surgeons; Luime JJ, Koes BW, Hendriksen IJ, et al. Prevalence and incidence of shoulder pain in the general population; a systematic review. Scand J Rheumatol. 2004;33: MacDermid JC, Richards RS, Donner A, Bellamy N, Roth JH. Responsiveness of the short form-36, disability of the arm, shoulder, and hand questionnaire, patientrated wrist evaluation, and physical impairment measurements in evaluating recovery after a distal radius fracture. J Hand Surg [Am]. 2000;25: McClure PW, Bialker J, Neff N, Williams G, Karduna A. Shoulder function and 3-dimensional kinematics in people with shoulder impingement syndrome before and after a 6-week exercise program. Phys Ther. 2004;84: McConnell S, Beaton D, Bombardier C. The DASH Outcome Measure User s Manual. Toronto, Canada: Institute for Work & Health; McHorney CA, Tarlov AR. Individual-patient monitoring in clinical practice: are available health status surveys adequate? Qual Life Res. 1995;4: Michener LA, Leggin BG. A review of self-report scales for the assessment of functional limitation and disability of the shoulder. J Hand Ther. 2001;14: Michener LA, McClure PW, Sennett BJ. American Shoulder and Elbow Surgeons Standardized Shoulder Assessment Form, patient self-report section: reliability, validity, and responsiveness. J Shoulder Elbow Surg. 2002;11: Norman GR, Stratford P, Regehr G. Methodological problems in the retrospective computation of responsiveness to change: the lesson of Cronbach. J Clin Epidemiol. 1997;50: Portney L, Watkins M. Foundations of Clinical Research: Applications to Practice. Norwalk, CT: Appleton & Lange; Richards RR, An KN, Bigliani LU, et al. A standardized method for the assessment of shoulder function. J Shoulder Elbow Surg. 1994;3: Roach KE, Budiman-Mak E, Songsiridej N, Lertratanakul Y. Development of a shoulder pain and disability index. Arthritis Care Res. 1991;4: Roddey TS, Olson SL, Cook KF, Gartsman GM, Hanten W. Comparison of the University of California-Los Angeles Shoulder Scale and the Simple Shoulder Test with the shoulder pain and disability index: singleadministration reliability and validity. Phys Ther. 2000;80: Roddey TS, Olson SL, Gartsman GM, Hanten WP, Cook KF. A randomized controlled trial comparing 2 instructional approaches to home exercise instruction following arthroscopic full-thickness rotator cuff repair surgery. J Orthop Sports Phys Ther. 2002;32: Romeo AA, Bach BR, Jr., O Halloran KL. Scoring systems for shoulder conditions. Am J Sports Med. 1996;24: Rowe CR, Patel D, Southmayd WW. The Bankart procedure: a long-term end-result study. J Bone Joint Surg Am. 1978;60: Shelley SI. Research Methods in Nursing and Health. Boston, MA: Little, Brown & Co; Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86: Stratford PW, Binkley FM, Riddle DL. Health status measures: strategies and analytic methods for assessing change scores. Phys Ther. 1996;76: Stratford PW, Binkley J, Solomon P, Finch E, Gill C, Moreland J. Defining the minimum level of detectable change for the Roland-Morris questionnaire. Phys Ther. 1996;76: ; discussion Stucki G, Liang MH, Fossel AH, Katz JN. Relative responsiveness of condition-specific and generic health status measures in degenerative lumbar spinal stenosis. J Clin Epidemiol. 1995;48: J Orthop Sports Phys Ther Volume 36 Number 3 March 2006

12 58. Tarlov AR, Ware JE, Jr., Greenfield S, Nelson EC, Perrin E, Zubkoff M. The Medical Outcomes Study. An application of methods for monitoring the results of medical care. JAMA. 1989;262: Testa MA. Interpretation of quality-of-life outcomes: issues that affect magnitude and meaning. Med Care. 2000;38: Tugwell P, Bombardier C. A methodologic framework for developing and selecting endpoints in clinical trials. J Rheumatol. 1982;9: van der Windt DA, Koes BW, de Jong BA, Bouter LM. Shoulder disorders in general practice: incidence, patient characteristics, and management. Ann Rheum Dis. 1995;54: Ware JE, Jr., Sherbourne CD. The MOS 36-item shortform health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992;30: Wells G, Beaton D, Shea B, et al. Minimal clinically important differences: review of methods. J Rheumatol. 2001;28: Williams JW, Jr., Holleman DR, Jr., Simel DL. Measuring shoulder function with the Shoulder Pain and Disability Index. J Rheumatol. 1995;22: Journal of Orthopaedic & Sports Physical Therapy RESEARCH REPORT J Orthop Sports Phys Ther Volume 36 Number 3 March

13 Appendix A Journal of Orthopaedic & Sports Physical Therapy 150 J Orthop Sports Phys Ther Volume 36 Number 3 March 2006